On Thu, Aug 31, 2017 at 9:04 AM, Donald Stufft <donald@stufft.io> wrote:

I'm a bit confused -- are we talking about the backwards compatible path to the future -- or the end-game?

Pip purposely overrides what setuptools tags the wheel with in order to make it extremely specific to the Python it is currently being run on. Now, a big part of why it needs to do this, is because a lot of setup.py were not written with wheel in mind, so they produce broken wheels by default.

right -- just so we're clear that this is hack to maintain backward compatibility. Though that seems like a hack that setuptools should have, not pip.

One such example is a setup.py that produces a pure Python wheel for PyPy, and a C-extension wheel for Python. If someone installs that project on PyPy first and they only have a sdist available on PyPI, then pip will cache a pure Python wheel for that project, and by default (without this bit to force it) that wheel would be acceptable for CPython too and would have been used instead of building a wheel with a compiled extension for CPython.

Frankly, I'm not sure that pip should be sharing a cache between PyPy and cPython anyway.. (I've had other issues with pip's fairly aggressive caching in the past...) But if it does, then pip should always look to see if there is a more "specific" version of a package available anyway. But I see your point -- if there is a cached pure-python wheel, pip can go look on PyPi for a cpython wheel, but if it isn't there, and there is only an sdist, it would have to download and build it to find out if there is a cPython version. Which of course, would then happen (and be wasted effort) essentially all the time, as most packages are pure python. But the proper solution to this is for packages that have both a pure python and cpython version available to put the wheels up on pypi. After all, this is the unusual case. In fact, I see PEP 517 as a step toward more wheels, and fewer sdists for distribution. -CHB

You could argue that the new crop of build backends should handle this better— and hopefully you’re correct— but I think that the two desires are at odd with one another. Most build backends are going to want to, by default, tag the wheel with as general of a tag as they think will work given the information available to them, whereas with pip’s internal wheel cache, we generally want that wheel to be as specific to the current version of Python as possible.

Another possible solution is to stop using the wheel tagging format to encode this information for pip’s internal wheel cache and to encode it into the path, so that instead of having PyPy and CPython share a cache directory, they each have their own. That isn’t an unreasonable mechanism for doing that and which one gets used is an implementation detail of pip.

— Donald Stufft

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov