[Wheel-builders] Heads-up re: new kernel configurations breaking the manylinux docker image

[Nathaniel Smith]

Hi all,

Just ran into this, and it took me a bit to track down, so wanted to
let people know.

The "vsyscall" mechanism is a clever hack/trick that Linux uses to
speed up some syscalls (most notably gettimeofday), and involves the
kernel injecting some code in processes' memory maps, that glibc then
knows to call. How does glibc know how to call it? In the old days,
the answer was simple: the kernel always injected it at the same fixed
address, so glibc was hard-coded to just "know" that e.g. gettimeofday
was at 0xffffffffff600400.

In these less trusting times, these hard-coded addresses are
considered a security risk (they violate ASLR etc.), so they were
deprecated a long time ago. The kernel now has configuration options
to control whether it injects this code at the fixed address. If you
flip the options to disable the old vsyscall stuff, then it makes your
system more secure... but, of course, if you try to run an old binary
that blindly uses the hardcoded addresses then it will segfault as
soon as it tries to call gettimeofday.

Debian has recently flipped the switch to disable this on their
kernels, so if you're running a recent Debian testing or unstable
(kernel 4.8 or better), then your experience of the manylinux_x86_64
docker image is now... not so great:

~$ docker run -it --rm quay.io/pypa/manylinux1_x86_64 /bin/bash
~$

Yeah, it just... silently exits when trying to do almost anything. One
clue as to what's going on is that the exit code is 139:

~$ echo $?
139

which means "segfault". I also get this in my kernel log (dmesg),
which is a little more informative:

[1060150.319215] bash[18225] vsyscall attempted with vsyscall=none
ip:ffffffffff600400 cs:33 sp:7ffff5521ea8 ax:ffffffffff600400
si:7ffff5522f80 di:0
[1060150.319216] bash[18225]: segfault at ffffffffff600400 ip
ffffffffff600400 sp 00007ffff5521ea8 error 15

This affects any docker image that uses glibc 2.13 or earlier. This
includes both CentOS 5 and CentOS 6.

The workaround is to reboot and add the option 'vsyscall=emulate' to
the kernel command line.

So far I guess it's only Arch and Debian testing/unstable doing this,
but I'd expect we may see this popping up with other distributors in
the future.

Bug report of this happening with Arch:
  https://github.com/CentOS/sig-cloud-instance-images/issues/62
Debian bug report:
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=847154
Docker bug report:
  https://github.com/docker/docker/issues/28705

-n

-- 
Nathaniel J. Smith -- https://vorpus.org