[Distutils] PEP 517 again

[Chris Barker]

On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncoghlan at gmail.com> wrote:

since it
> doesn't reliably distinguish between "this cached wheel was downloaded
> from a repository" and "this wheel was generated locally with a
> particular version of Python".


It shouldn't have to. sigh.

>
 PEP 517 deliberately doesn't let
> frontends do that as part of the initial build process (instead, if
> they want to adjust the tags, they need to do it as a post-processing
> step).
>
> Since PEP 517 breaks the current workaround for the caching scheme
> being inaccurate, the most suitable response is to instead fix pip's
> caching scheme to use a two tier local cache:


I'm still confused -- if setuptools ( invoked  by pip) is producing
incorrectly named wheels -- surely that's a bug-fix/workaround that should
go into setuptools?

If the build is being run by pip, then doesn't setuptools have all the info
about the system that pip has?

I'm trying to think through the use case presented the other day:

A package can be build either for all python's or for cpython in
particular. That selection is (currently) done at build time, by custom
code in setup.py.

In this case, the build system SHOULD be able to build the proper wheel,
and label it correctly.  But there might need to be a way to tell the build
system which you want. That logic isn't in existing setup.py files, but
there is only so much you can do!

Then there is the caching problem. If pip finds a pure-python wheel that
matches the spec in the cache, it won't know to go looking for a cpython
specific one. I can't see how separating the downloaded  cache from the
locally built cache will solve that problem:

If a pure-python wheel was already downloaded, then that'll be in the
downloaded cache anyway.

I think the core problem here is having, for instance, pypy and cpython
sharing a wheel cache. Is it really that important to support that?

If so, this is a total kludge, but maybe the pip cache could keep a
record-- I looked for a cpython specific wheel, and it doesn't exist, so I
never need to look again.

Or -- maybe more kludgy -- every time pip puts a  wheel in the cache via
cpython, it makes a link with the cpython name.

Independently of that though, it's also important to note that pip
> probably *won't* switch to invoking legacy setup.py files through the
> PEP 517 backend - instead, it will only do that if there's a
> pyproject.toml file that says "Use the setuptools PEP 517 backend".


That's probably a good way to keep backward compatibility while moving
forward. Which means pip can have two modes -- legacy mode and pep517 mode.

Which may encourage me to stop being such a noodge -- As long as we're
talking legacy mode, we can be as hacky as we need to be.

Autobuilding support definitely *won't* be removed from pip,


I'm not suggesting it should be (and I think I may be the only one that
might have been) However, I do think auto-building support does not have to
support all the complex use cases.

we also have plenty of PyPI users that
> explicitly *opt out* of using publisher-provided pre-built binaries.
> While Linux distributions are the most common example (see [1] for
> Fedora's policy, for example), we're not the only ones that have that
> kind of rule in place.


But this is an argument for why pypi should host sdists, and the build
tools should build sdists, but not why pip should auto-build them.

>
Condo-forge, for example, almost always builds from source -- sometimes an
sdist
from pypi, sometimes a source distribution from github or wherever the
package is hosted. And sometimes from a git tag ( last resort).

Do the Linux distros use pip to build their packages?

I tried to do that with conda-packages, and failed due to pip's caching
 behavior-- it probably would have worked fine in production, but when I
was developing the build script, I couldn't reliably get pip to ignore
cached wheels from previous experimental builds.

So I ( and the entire community, as far as I can tell) stuck with calling
setuptools directly - via setup.py, but with setuptools-specific flags)

As pip is a package management tool, it makes sense that other package
managers wouldn't need to use it.

NOTE: I do use pip to build conda packages from wheels in a couple cases --
but that's considered a work around for my pathetic build skills -- and
even then, I don't have pip download the wheel -- that's conda-build's job.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170901/ad3571e1/attachment.html>