support glibc version as part of the target #2847

andrewrk · 2019-07-07T21:36:30Z

This pull request obsoletes #2509. That PR adequately describes the problem, so go read that description to understand the context for this pull request.

This strategy, rather than building map files and dummy .so files for every target architecture that we have to ship and having the target glibc hardcoded in Zig repository tree, allows the Zig or C programmer to choose the minimum required glibc version as part of the target. Zig then generates the map files and dummy .so files on the fly depending on the target arch, abi, and glibc version.

If we were to bundle all the .abilist files from glibc directly, it would be a total of 2.4 MiB of install size (409 KiB gzipped) that we would have to add to the zig distribution. Instead, this new strategy does some preprocessing, and still leaves a readable text file format, but that takes up 190 KiB of install size (20 KiB gzipped). That's a small price to pay for being able to target any glibc version on the fly!

Updating to a new glibc version is as simple as pointing the tool directly at the unzipped glibc source code of the release you want to update to. It's going to look at all the .abilist files.

zig build-exe ../tools/update_glibc.zig  && ./update_glibc ~/downloads/glibc/ ../

This will update these files:

libc/glibc/vers.txt (442 bytes)
libc/glibc/fns.txt (51 KB)
libc/glibc/abi.txt (139 KB)

There is a new CLI parameter, -target-glibc which is allowed when the target is glibc. There is a new section to zig targets. Here's the relevant snippet on my computer:

Available glibc versions:
  2.0
  2.1
  2.1.1
  2.1.2
  2.1.3
  2.2
  2.2.1
  2.2.2
  2.2.3
  2.2.4
  2.2.5
  2.2.6
  2.3
  2.3.2
  2.3.3
  2.3.4
  2.4
  2.5
  2.6
  2.7
  2.8
  2.9
  2.10
  2.11
  2.12
  2.13
  2.14
  2.15
  2.16
  2.17
  2.18
  2.19
  2.22
  2.23
  2.24
  2.25
  2.26
  2.27 (native)
  2.28
  2.29

The native glibc version is detected by the zig compiler looking at its own dynamic libraries, identifying the one that is libc, and reading the symlink to determine the glibc version. The static binaries shipped on the download page will need another heuristic in order to correctly identify the system glibc version. For now they will choose 2.17, which is also the default when cross compiling. Here's the relevant TODO comment:

        // TODO There is still more we could do to detect the native glibc version. For example,
        // we could look at the ELF file of `/usr/bin/env`, find `libc.so.6`, and then `readlink`
        // to find out the glibc version. This is relevant for the static zig builds distributed
        // on the download page, since the above detection based on zig's own dynamic linking
        // will not work.

There is a new CLI parameter, --version-map which works the same way as in #2509.

Zig now has another interesting ability, which is that it could emit a compile error when trying to use a glibc function that does not exist on the target. See #397 for a related issue. That is not implemented in this PR, but the information is there.

This PR exposes the glibc version in @import("builtin") which the standard library could potentially use to determine whether it is safe to call a given function.

@LemonBoy what do you think?

This is the beginning of supporting minimum GLIBC version as part of the target. See #2509 for the motivation. The dummy libc zig files are removed. A future commit will build them on-the-fly, using the generated text files generated by the new tool, which are checked into source control and distributed along with zig. These generated text files are, together, 142KB (20KB gzipped). Compare that to a naive bundling of the .abilist files, which would be 2.2MiB (375KB gzipped). This is based on glibc 2.29.

Fixes glibc_version being set to garbage. I've made this mistake before so this is an attempt to prevent future bugs. Zig doesn't have zero-initialization, so are we being a hypocrite by using this C feature? No, because C doesn't have the feature that forces you to initialize all fields. That would have prevented this bug every single time.

daurnimator · 2019-07-08T07:23:00Z

libc/glibc/vers.txt (442 bytes)

libc/glibc/fns.txt (51 KB)

libc/glibc/abi.txt (139 KB)

As some future work, could we have these compressed?

daurnimator · 2019-07-08T07:28:03Z

tools/update_glibc.zig

+};
+const Function = struct {
+ name: []const u8, // example: "puts"
+ lib: []const u8, // example: "c"


Has it ever happended that a function got moved between libraries? e.g. from libm into libc? Could we handle it if it happened in future?

I don't know if it's happened but it's trivially handled by this script. This script parses the .abilist files from glibc source, which describe all symbols. So if a function got moved then the .abilist file in glibc would get updated and this tool would pick up the changes, no problem. This PR actually makes zig more resilient to this compared to master branch.

But wouldn't it then be in one library for an old release and in a different library for a new release? What would appear in fns.txt? multiple entries for one function name?

I see the question now. Technically some functions are in multiple libraries, and there are some straightforward changes that could be done to respect that. However, when zig links against glibc we always link all the libraries. I'm pretty sure it doesn't matter which library it says a function came from, because at runtime the dynamic linker will find the correct one. If we run into trouble, no problem, I'll do the change to make it support functions in multiple libs.

@daurnimator yes, many time functions got moved from librt to libc

daurnimator · 2019-07-08T07:31:47Z

tools/update_glibc.zig

+ } else if (b_next == null) {
+ return false;
+ }
+ const a_int = fmt.parseInt(u64, a_next.?, 10) catch unreachable;


Should this be catch @panic()? Is there anything to prevent someone compiling this in release-fast mode?

Although I didn't intend for this code to ever be run in non-debug mode, it probably should do as you say, just to set a good example for people who are reading zig source and expecting it to be idiomatic. Same goes for the zig build system implementation.

Or, there is another option...

comptime{ assert(builtin.mode == .Debug); // this tool is coded for debug mode only }

Yep either of those would be solutions.

If you lead by example you might think of new syntax to make the "right" thing easier to do and the "bad" thing harder to do.

andrewrk · 2019-07-08T15:36:14Z

As some future work, could we have these compressed?

Sure, that should work just fine. The decompression code could even be in .zig code since this feature does not depend on itself.

it was causing a problem on the CI

andrewrk mentioned this pull request Jul 7, 2019

Add glibc versioned symbols #2509

Closed

andrewrk added 5 commits July 7, 2019 17:56

ability to target any glibc version

7ccf780

add -target-glibc to cli help and zig build

336ddb8

remove debug log

3e36cad

expose glibc version in builtin

56d8185

andrewrk force-pushed the glibc-abi-versioning branch from 099ee58 to 56d8185 Compare July 7, 2019 21:56

fix not setting a target glibc version on non-linux

f047827

andrewrk force-pushed the glibc-abi-versioning branch from 37da996 to f047827 Compare July 8, 2019 03:22

daurnimator reviewed Jul 8, 2019

View reviewed changes

avoid dependency on linux/limits.h header

201033d

it was causing a problem on the CI

This was referenced Jul 8, 2019

deflate implementation in standard library #213

Closed

xz decompressor #2851

Closed

andrewrk merged commit 4953e84 into master Jul 8, 2019

andrewrk deleted the glibc-abi-versioning branch July 8, 2019 17:51

andrewrk mentioned this pull request Jul 8, 2019

introduce operating system version ranges as part of the target #1907

Closed

daurnimator mentioned this pull request Sep 8, 2020

Distribute glibc ABI data compressed #6283

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support glibc version as part of the target #2847

support glibc version as part of the target #2847

andrewrk commented Jul 7, 2019 •

edited

Loading

daurnimator commented Jul 8, 2019

daurnimator Jul 8, 2019

andrewrk Jul 8, 2019 •

edited

Loading

daurnimator Jul 8, 2019 •

edited

Loading

andrewrk Jul 8, 2019

shawnl Aug 5, 2019

daurnimator Jul 8, 2019

andrewrk Jul 8, 2019

daurnimator Jul 8, 2019

andrewrk commented Jul 8, 2019

support glibc version as part of the target #2847

support glibc version as part of the target #2847

Conversation

andrewrk commented Jul 7, 2019 • edited Loading

daurnimator commented Jul 8, 2019

daurnimator Jul 8, 2019

Choose a reason for hiding this comment

andrewrk Jul 8, 2019 • edited Loading

Choose a reason for hiding this comment

daurnimator Jul 8, 2019 • edited Loading

Choose a reason for hiding this comment

andrewrk Jul 8, 2019

Choose a reason for hiding this comment

shawnl Aug 5, 2019

Choose a reason for hiding this comment

daurnimator Jul 8, 2019

Choose a reason for hiding this comment

andrewrk Jul 8, 2019

Choose a reason for hiding this comment

daurnimator Jul 8, 2019

Choose a reason for hiding this comment

andrewrk commented Jul 8, 2019

andrewrk commented Jul 7, 2019 •

edited

Loading

andrewrk Jul 8, 2019 •

edited

Loading

daurnimator Jul 8, 2019 •

edited

Loading