[Distutils] PEP 517 again

[cyanrave at gmail.com]

Great discussions on this PEP over the proposal, here and in the archives. I've just now gotten plugged in to the distutils distro, at a seemingly perfect time... somewhat new to Python packaging and still learning.

Many of the issues observed by users in the mail chain are reasons why we had marching orders at the organization I work for to internally build all whls, on the platforms we support by using 'pip wheel' with specific flags. One of these flags is to pull source only (pip download, no-binary) and direct 'pip wheel' to build with that source dir.

For the caching issue we ended up using Docker, spinning up a new instance per build with the platforms we support, running the build, and uploading those artifacts to our internal repo for pulls by the users, then thrashing that container wholly. In a similar way, I see the 'venv' requirement of 517 mirroring this process.

The presupposition of this pipeline we built is that metadata and source repo structures are trusted to produce accurate binaries... the more I learn about how projects / packages have set up their build instructions, the more horrified I become in the variance of the implementations out in the wild.

517 and 518 to me seems like a way to reign in some of these variances and formalize them, especially around setup.py and metadata... Is that correct or am I way off base with the discussion and proposal so far?

Best regards,
C. Schmautz

> On Sep 2, 2017, at 12:42 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> 
>> On Sep 1, 2017, at 2:30 PM, Chris Barker <Chris.Barker at noaa.gov> wrote:
>> 
>>> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> 
>>> since it
>>> doesn't reliably distinguish between "this cached wheel was downloaded
>>> from a repository" and "this wheel was generated locally with a
>>> particular version of Python".
>> 
>> It shouldn't have to. sigh.
>> 
>>>  PEP 517 deliberately doesn't let
>>> frontends do that as part of the initial build process (instead, if
>>> they want to adjust the tags, they need to do it as a post-processing
>>> step).
>>> 
>>> Since PEP 517 breaks the current workaround for the caching scheme
>>> being inaccurate, the most suitable response is to instead fix pip's
>>> caching scheme to use a two tier local cache:
>> 
>> I'm still confused -- if setuptools ( invoked  by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools?
>> 
>> If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has?
>> 
>> 
> 
> 
> Someone building a wheel for distribution is likely intimately aware of that project, and can take care to ensure that the wheel is built in such a way that it is giving people the most optimal behavior. Pip is auto building wheels without human intervention, and as such there is nobody there to make sure that we’re not accidentally creating a too-broad wheel, so we want to ensure that we have some mechanism in place for not re-using the wheel across boundaries that might cause issues.
> 
> 
>> 
>>> we also have plenty of PyPI users that
>>> explicitly *opt out* of using publisher-provided pre-built binaries.
>>> While Linux distributions are the most common example (see [1] for
>>> Fedora's policy, for example), we're not the only ones that have that
>>> kind of rule in place.
>> 
>> But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them.
>> 
>> Condo-forge, for example, almost always builds from source -- sometimes an sdist 
>> from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort).
> 
> 
> Pip supports more systems than Conda does, and we do that by relying on auto building support. Pip supports systems that don’t have a wheel compatibility tag defined for them, and for which we’re unlikely to ever have any wheels published for (much less wide spread). It’s pretty easy to cover Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long.
> 
> Pip works in all these situations, and it does so by relying on building from source.
> 
> 
>> 
>> Do the Linux distros use pip to build their packages?
> 
> 
> Not that I am aware of.
> 
>> 
>> I tried to do that with conda-packages, and failed due to pip's caching  behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds.
> 
> 
> Adding —no-cache-dir disables all of pip’s caching.
> 
> —
> Donald Stufft
> 
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170902/ec15abfe/attachment.html>