[Distutils] draft PEP: manylinux1

[Nathaniel Smith]

On Sun, Jan 24, 2016 at 8:17 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 25 January 2016 at 10:23, Donald Stufft <donald at stufft.io> wrote:
>>
>>> On Jan 24, 2016, at 7:21 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>>
>>> Maybe we need wheel-builders-sig? Their mandate would be to hash out
>>> things like how to build binary-libraries-wrapped-up-in-wheels, share
>>> knowledge about the minutiae of linker behavior on different platforms
>>> (oh god there's so much minutiae), maintain tools like delocate and
>>> auditwheel (and whatever the equivalent will be for windows... and do
>>> we really need 3 different tools?), collect knowledge from where it's
>>> scattered now and put it into the guide at packaging.python.org [1],
>>> etc.? It seems a bit outside distutils-sig's focus in practice, since
>>> this would all be about third-party tools and individual package
>>> authors as opposed to distutils-sig's focus on writing
>>> interoperability PEPs and maintaining the core python.org-affiliated
>>> infrastructure like PyPI / setuptools / pip.
>>
>> I’m not about to tell someone they *have* to use distutils-sig, but I
>> think at this point it’s perfectly reasonable to discuss things of
>> this nature here as well.
>
> I agree with this - while I wouldn't call any aspect of software
> distribution easy, binary compatibility is one of the hardest, and we
> want *more* knowledge sharing in that regard, rather than continue to
> maintain the divide between "folks who care about binary
> compatibility" and "folks who wish everyone would just stop creating
> extension modules already because pure Python modules are so much
> easier to deal with" :)
>
> Keeping the discussions centralised also creates more opportunities
> for serendipitous collaboration, where folks notice that they can
> potentially help out with what someone else is working on.

I'm definitely in favor of centralising this kind of knowledge and
initiative upstream with the core Python community. The whole
manylinux concept falls into this category of "hey let's move take
this experience with scientific packages and port it upstream" (and
btw if/when PEP 513 is accepted we should perhaps consider moving the
related tools into the pypa/ namespace?).

I guess my concern, though, is that distutils-sig has historically
been a rather contentious place. It's definitely not as bad these days
as its old reputation would suggest (much thanks to everyone who's
helped make that happen!). But I guess some amount of this is
intrinsic to its nature: the core infrastructure like distutils, pip,
PyPI, interoperability PEPs, is stuff that people with wildly varying
backgrounds and goals all *have* to use, and so we're kinda all
trapped here together. In a way, this kind of contention is a good
thing, because it forces this core infrastructure to do a better job
of serving all its many stakeholders. But OTOH I'm not sure it's very
conducive to making quick collaborative progress on focused topics
that don't need broad community consensus. (I'm not sure where
day-to-day discussion of routine changes to pip and warehouse and
setuptools is happening, but I do notice that it doesn't seem to be
here. And similarly there's a reason that the initial hashing out of
the manylinux stuff happened off-list -- if only because you didn't
need to see the blow-by-blow of our futzing about making docker work
;-).)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org