[Distutils] CentOS5 is EOL, impact on manylinux1?

[Nathaniel Smith]

On Fri, Apr 7, 2017 at 10:41 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 7 April 2017 at 06:32, Nathaniel Smith <njs at pobox.com> wrote:
>> On Wed, Apr 5, 2017 at 2:37 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> And as it turns out, not only are we able to get bare metal machines
>>> to run our VMs on (rather than messing about with nested virt
>>> support), but the CentOS boxes are pre-cached on the local network, so
>>> they should be pretty quick to download (we haven't actually got our
>>> CI up and running yet, so I won't be able to vouch for that personally
>>> until we do).
>>
>> I think bare metal access only matters for running x86 with hardware
>> accelerated virtualization? If we want to emulate a totally different
>> architecture like ppc64le then IIUC qemu does that as a regular
>> user-space program. You definitely can run qemu in this mode on
>> travis-ci, e.g. check out all the virtual architectures that rust
>> builds on:
>>
>>    https://travis-ci.org/rust-lang/rust/
>>
>> So I think travis-ci and centos-ci are equivalent on this axis – if we
>> want to go the qemu route, then either works, and if we don't, then
>> neither works?
>
> My understanding of the problem is that Travis can't actually run the
> 64-bit CentOS VMs fast enough to get build jobs done in a reasonable
> time period (due to the resource constraints imposed on the test VMs),
> whereas the CentOS systems have more resources available to them
> (since they have the whole machine to themselves).
>
> "Try it and see" would be the best way to confirm whether or not
> qemu-ppc64le-on-Travis works, though.

Ah, good point – Travis imposes a 1 hour time limit on builds, and for
manylinux1 we're at about half that already with all the supported
versions of Python... though some of that is compensating for centos 5
brokenness (building our own openssl) and some of it is Python
versions we could probably drop at this point (2.6 at least, since
there will never be a version of pip that supports both python 2.6 and
manylinux2+!).

>> For me the big question is whether emulation is actually a good idea.
>> When rust announced their plans I remember seeing some skepticism
>> about whether one can really trust emulated machines for this kind of
>> use case, though I can't find it again now... but it's definitely true
>> that all the major distributions go to great lengths to use real
>> hardware in their build farms, despite the obvious drawbacks (not just
>> in terms of maintenance, but also the ongoing pain of using tiny
>> little arm and mips machines that take dozens of hours to build
>> things). They know a whole lot more about this than I do so I assume
>> they have *some* reason :-).
>
> It's one thing using emulation for a service being provided for free
> in a community project, something else entirely to be doing it for
> commercial or commercially-sponsored software.

Oh sure. I actually don't know which case this is -- does anyone care
about ppc64le who isn't working with some big IBM support contract?
We've had at least one @ibm address in one of these threads...

> One particular problem with it is that any use of profile-guided
> optimisation would be optimising for the performance characteristics
> of the emulator, not those of the real hardware (that's also a problem
> for cross-compilation: as far as I am aware, you *can't* readily use
> profile guided optimisation in the cross-compilation case).

I thought PGO uses profiling to observe branches and variable values,
which emulation shouldn't effect? Fortunately this part really doesn't
matter much to us anyway though -- nothing in the docker image gets
shipped to end-users. The important thing would be how the package
distributors are running the image to do *their* builds, and that's
not our problem.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org