[Distutils] PEP 517 again

[Nick Coghlan]

On 1 September 2017 at 07:29, Chris Barker <chris.barker at noaa.gov> wrote:
> But I do think we should be clear about what is a hack for backward
> compatibility, and what is part of the designed functionality.

Right, and I think it's pretty clear that the problem xoviat raised is
with the way pip's implicit local artifact cache works, since it
doesn't reliably distinguish between "this cached wheel was downloaded
from a repository" and "this wheel was generated locally with a
particular version of Python".

Currently pip is working around that problem by overriding the
compatibility tags produced by setup.py to ensure the generated wheel
files are interpreter specific, but PEP 517 deliberately doesn't let
frontends do that as part of the initial build process (instead, if
they want to adjust the tags, they need to do it as a post-processing
step).

Since PEP 517 breaks the current workaround for the caching scheme
being inaccurate, the most suitable response is to instead fix pip's
caching scheme to use a two tier local cache:

1. Wheel download cache (if this gets a hit, don't even check the package repo)
2. Check the package repo for a wheel file (and if you find one, add
it to the download cache)
3. Local build cache (make this interpreter implementation specific)
4. Build a local wheel that's specific to the running interpreter implementation

Independently of that though, it's also important to note that pip
probably *won't* switch to invoking legacy setup.py files through the
PEP 517 backend - instead, it will only do that if there's a
pyproject.toml file that says "Use the setuptools PEP 517 backend". In
such cases, it's reasonable to expect that the project will also be
using environment markers appropriately, and not putting conditional
dependency installation logic directly into setup.py.

Autobuilding support definitely *won't* be removed from pip, as there
will not only always be a large number of projects that don't provide
their own prebuilt wheel files, we also have plenty of PyPI users that
explicitly *opt out* of using publisher-provided pre-built binaries.
While Linux distributions are the most common example (see [1] for
Fedora's policy, for example), we're not the only ones that have that
kind of rule in place.

Cheers,
Nick.

[1] https://fedoraproject.org/wiki/Packaging:Guidelines#No_inclusion_of_pre-built_binaries_or_libraries

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia