conda environment and running a shell script to install the python
conda-build doesn't produce or consume wheels. It works by creating a clean package
into that environment
To be really clear -- conda build doesn't directly use ANY build/install system. All files produced by the install
script are then put into a tar file, which is conda's version of a wheel.
right -- conda-build simply runs the script it is told to run (by the recipe), and then looks and sees what files were installed. In theory, one could build a conda package directly out of a binary wheel -- after all, as mentioned, they are actually pretty similar, but most folks would probably just use pip inside their conda-build recipe. In fact, that is already being done -- most python packages are built with "setup.py install", but some are built by using pip to install a wheel. But I didn't think that the goal of PEP 517 was to ensconce pip as the only way to install a package in Python. So if a build system can install (like setuptools currently does), then conda recipes may well call it directly. I don't think this really has any bearing on the the PEP, though -- As I understand it, the goal of the PEP is to formalize the interface between build systems and package managers -- and that can only be a good thing for other package managers. NOTE: For the most part, folks want to use a pip-compatible install in conda-build, because users tend to also directly use pip to install stuff in conda environments e.g. if there is not a conda package available. If a conda package were built with, say, plain distutils, then pip won't find the meta data it's looking for, and might falsely see an unresolved dependency. Personally, I've always avoided pip installing packages in conda -- preferring to make conda packages for all the deps I need, but conda-pip compatibility has come a long way, so it does work fine for many users. But it would be nice if packages didn't NEED to be pip installed -- but I think that's an issue with the wheel spec (that may well be solved). -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 29 August 2017 at 19:45, Chris Barker <chris.barker@noaa.gov> wrote:
But it would be nice if packages didn't NEED to be pip installed -- but I think that's an issue with the wheel spec (that may well be solved).
Wheels don't need pip to install them - the spec has always been clear on how to "manually" install a wheel (unzip it and move a few things around), and the wheel package itself has a simple "wheel install" command. If you want to create wrapper scripts for entry points, you need to do a bit more work, but distlib offers that functionality (that's what pip uses). One point about PEP 517 is that it provides a standard way to ask *any* build system to make a wheel. So it's in theory possible to install any package without pip (of course, if you try to handle all the various corner cases, extra quirks, etc that come up in the real world, you end up reinventing pip ;-)). Paul
For me, using NotImplemented is a misuse of the singleton since I know what it's meant to be used for (and so I cringe every time I hear it brought up as a solution).
I hate to reiterate too much, but I think that you are correct here: NotImplemented is not Pythonic. It's a complete hack that I would think people would like to quarantine to the limited use case of binary operations. Even with the poorly contextualized email that Guido received, I think it was clear that he thought it wasn't appropriate outside of that use case either (I don't think better context would have changed that). Personally, I was willing to overlook that for the sake of a speedy resolution. But reading the email chain, it seems that only Thomas and Daniel are currently in favor of using NotImplemented, which is a shift from the earlier situation. So now it seems that NotImplemented is not likely to be used. 2017-08-29 14:07 GMT-05:00 Paul Moore <p.f.moore@gmail.com>:
On 29 August 2017 at 19:45, Chris Barker <chris.barker@noaa.gov> wrote:
But it would be nice if packages didn't NEED to be pip installed -- but I think that's an issue with the wheel spec (that may well be solved).
Wheels don't need pip to install them - the spec has always been clear on how to "manually" install a wheel (unzip it and move a few things around), and the wheel package itself has a simple "wheel install" command.
If you want to create wrapper scripts for entry points, you need to do a bit more work, but distlib offers that functionality (that's what pip uses).
One point about PEP 517 is that it provides a standard way to ask *any* build system to make a wheel. So it's in theory possible to install any package without pip (of course, if you try to handle all the various corner cases, extra quirks, etc that come up in the real world, you end up reinventing pip ;-)).
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Hi, On Tue, Aug 29, 2017 at 8:30 PM, xoviat <xoviat@gmail.com> wrote:
For me, using NotImplemented is a misuse of the singleton since I know what it's meant to be used for (and so I cringe every time I hear it brought up as a solution).
I hate to reiterate too much, but I think that you are correct here: NotImplemented is not Pythonic. It's a complete hack that I would think people would like to quarantine to the limited use case of binary operations. Even with the poorly contextualized email that Guido received, I think it was clear that he thought it wasn't appropriate outside of that use case either (I don't think better context would have changed that).
Personally, I was willing to overlook that for the sake of a speedy resolution. But reading the email chain, it seems that only Thomas and Daniel are currently in favor of using NotImplemented, which is a shift from the earlier situation. So now it seems that NotImplemented is not likely to be used.
Just for the record, I thought Nathaniel's NotImplemented suggestion was a good one, and his explanation of why, was clear and convincing, but he's already accepted the alternative, of a custom error, so I don't think there's much point on going back over that. Also just for the record, I'd like to warmly thank y'all for your great patience in this long discussion. Cheers, Matthew
So while I was working on pip, I noticed a parameter called 'autobuilding' in wheel.py. What this does (I think) is that if pip decides that it needs to be enabled (which is an implementation detail not relevant here), it adds the following to the "setup.py bdist_wheel" call: ["--python-tag", python_tag] This switches the output of the bdist_wheel filename to include something like "cp36" rather than "py36," which indicates the Python implementation. The question is this: this behavior isn't captured (I think) in the build_wheel function, possibly (probably) for good reason. So that means once everything goes through the PEP 517 protocol, pip will not be able to let setuptools know about this information. Which means that we need to decide: Should the PEP 517 backend for setuptools assume "cp36" in python_tag OR Should the PEP 517 backend for setuptools assume "py36" in python_tag 2017-08-29 18:51 GMT-05:00 Matthew Brett <matthew.brett@gmail.com>:
Hi,
On Tue, Aug 29, 2017 at 8:30 PM, xoviat <xoviat@gmail.com> wrote:
For me, using NotImplemented is a misuse of the singleton since I know what it's meant to be used for (and so I cringe every time I hear it brought up as a solution).
I hate to reiterate too much, but I think that you are correct here: NotImplemented is not Pythonic. It's a complete hack that I would think people would like to quarantine to the limited use case of binary operations. Even with the poorly contextualized email that Guido received, I think it was clear that he thought it wasn't appropriate outside of that use case either (I don't think better context would have changed that).
Personally, I was willing to overlook that for the sake of a speedy resolution. But reading the email chain, it seems that only Thomas and Daniel are currently in favor of using NotImplemented, which is a shift from the earlier situation. So now it seems that NotImplemented is not likely to be used.
Just for the record, I thought Nathaniel's NotImplemented suggestion was a good one, and his explanation of why, was clear and convincing, but he's already accepted the alternative, of a custom error, so I don't think there's much point on going back over that.
Also just for the record, I'd like to warmly thank y'all for your great patience in this long discussion.
Cheers,
Matthew
This appears to be related to https://github.com/pypa/pip/pull/3225, where pip would generate overbroad wheel names in the cache. So to proceed on this issue, setuptools will assume that generated wheels are valid for the specific python implementation only (eg. wheels generated for CPython will not be valid for pypy). Again, let me repeat that: wheels generated using setuptools are valid for CPython only if build on CPython. This is not the current setuptools behavior but will be for all setuptools build backend calls (I assume legacy will remain the same). Jacob, Paul, Donald, is everyone okay with that? And does everyone understand the reason for this? 2017-08-30 18:29 GMT-05:00 xoviat <xoviat@gmail.com>:
So while I was working on pip, I noticed a parameter called 'autobuilding' in wheel.py. What this does (I think) is that if pip decides that it needs to be enabled (which is an implementation detail not relevant here), it adds the following to the "setup.py bdist_wheel" call:
["--python-tag", python_tag]
This switches the output of the bdist_wheel filename to include something like "cp36" rather than "py36," which indicates the Python implementation. The question is this: this behavior isn't captured (I think) in the build_wheel function, possibly (probably) for good reason. So that means once everything goes through the PEP 517 protocol, pip will not be able to let setuptools know about this information. Which means that we need to decide:
Should the PEP 517 backend for setuptools assume "cp36" in python_tag OR Should the PEP 517 backend for setuptools assume "py36" in python_tag
2017-08-29 18:51 GMT-05:00 Matthew Brett <matthew.brett@gmail.com>:
Hi,
For me, using NotImplemented is a misuse of the singleton since I know what it's meant to be used for (and so I cringe every time I hear it brought up as a solution).
I hate to reiterate too much, but I think that you are correct here: NotImplemented is not Pythonic. It's a complete hack that I would think people would like to quarantine to the limited use case of binary operations. Even with the poorly contextualized email that Guido received, I think it was clear that he thought it wasn't appropriate outside of
case either (I don't think better context would have changed that).
Personally, I was willing to overlook that for the sake of a speedy resolution. But reading the email chain, it seems that only Thomas and Daniel are currently in favor of using NotImplemented, which is a shift from the earlier situation. So now it seems that NotImplemented is not
On Tue, Aug 29, 2017 at 8:30 PM, xoviat <xoviat@gmail.com> wrote: that use likely to
be used.
Just for the record, I thought Nathaniel's NotImplemented suggestion was a good one, and his explanation of why, was clear and convincing, but he's already accepted the alternative, of a custom error, so I don't think there's much point on going back over that.
Also just for the record, I'd like to warmly thank y'all for your great patience in this long discussion.
Cheers,
Matthew
On 31 August 2017 at 14:22, xoviat <xoviat@gmail.com> wrote:
This appears to be related to https://github.com/pypa/pip/pull/3225, where pip would generate overbroad wheel names in the cache. So to proceed on this issue, setuptools will assume that generated wheels are valid for the specific python implementation only (eg. wheels generated for CPython will not be valid for pypy).
Again, let me repeat that: wheels generated using setuptools are valid for CPython only if build on CPython. This is not the current setuptools behavior but will be for all setuptools build backend calls (I assume legacy will remain the same).
While I do think your proposal would work (on the assumption that folks aren't use "pip wheel" to generate their wheel files for upload), an alternative approach with a lower risk of unintended side effects would be for *pip* to either rename the autobuilt file before adding it to the cache, or else to adjust its caching strategy a bit to internally separate a shared wheel download cache from per-interpreter-compatibility-tag caches for locally built wheel files. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 31 August 2017 at 14:22, xoviat <xoviat@gmail.com> wrote:
Again, let me repeat that: wheels generated using setuptools are valid for CPython only if build on CPython. This is not the current setuptools behavior but will be for all setuptools build backend calls (I assume legacy will remain the same).
While I do think your proposal would work (on the assumption that folks aren't use "pip wheel" to generate their wheel files for upload),
I use 'pip wheel' to generate wheel files for upload... (I like to generate an sdist and then build a wheel from that, and 'pip wheel sdist.tar.gz' is more convenient than manually unpacking and running bdist_wheel. )
an alternative approach with a lower risk of unintended side effects would be for *pip* to either rename the autobuilt file before adding it to the cache, or else to adjust its caching strategy a bit to internally separate a shared wheel download cache from per-interpreter-compatibility-tag caches for locally built wheel files.
+1 -n -- Nathaniel J. Smith -- https://vorpus.org
Will do. Better to hash this out sooner than later. On Aug 31, 2017 12:36 AM, "Nathaniel Smith" <njs@pobox.com> wrote:
On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 31 August 2017 at 14:22, xoviat <xoviat@gmail.com> wrote:
Again, let me repeat that: wheels generated using setuptools are valid for CPython only if build on CPython. This is not the current setuptools behavior but will be for all setuptools build backend calls (I assume legacy will remain the same).
While I do think your proposal would work (on the assumption that folks aren't use "pip wheel" to generate their wheel files for upload),
I use 'pip wheel' to generate wheel files for upload... (I like to generate an sdist and then build a wheel from that, and 'pip wheel sdist.tar.gz' is more convenient than manually unpacking and running bdist_wheel. )
an alternative approach with a lower risk of unintended side effects would be for *pip* to either rename the autobuilt file before adding it to the cache, or else to adjust its caching strategy a bit to internally separate a shared wheel download cache from per-interpreter-compatibility-tag caches for locally built wheel files.
+1
-n
-- Nathaniel J. Smith -- https://vorpus.org
On 31 August 2017 at 06:36, Nathaniel Smith <njs@pobox.com> wrote:
On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 31 August 2017 at 14:22, xoviat <xoviat@gmail.com> wrote:
Again, let me repeat that: wheels generated using setuptools are valid for CPython only if build on CPython. This is not the current setuptools behavior but will be for all setuptools build backend calls (I assume legacy will remain the same).
While I do think your proposal would work (on the assumption that folks aren't use "pip wheel" to generate their wheel files for upload),
I use 'pip wheel' to generate wheel files for upload... (I like to generate an sdist and then build a wheel from that, and 'pip wheel sdist.tar.gz' is more convenient than manually unpacking and running bdist_wheel. )
Agreed. The wheels built by "pip wheel" mustn't change (and that means that neither pip nor the setuptools backend should not change the tags it applies to wheels by default). The fact that pip artificially restricts wheels it builds as part of its internal use to be platform-specific is a workaround for an underlying problem, which is that it cannot guarantee that an internally-built wheel will be the same as one that the project later publishes for that version (it's a variation on the wheel equivalency problem we've been discussing). Rather than artificially modifying the compatibility tags, pip should cache internally-built wheels differently, so that wheels published by the upstream project have priority over pip-built wheels. That's a change that purely affects pip and is unrelated to PEP 517 directly, but it's something pip probably needs to do to prepare for switching to PEP 517. (xoviat - sorry I didn't think of this when commenting on the PR - it was Nick's suggestion here that made the situation clearer to me).
an alternative approach with a lower risk of unintended side effects would be for *pip* to either rename the autobuilt file before adding it to the cache, or else to adjust its caching strategy a bit to internally separate a shared wheel download cache from per-interpreter-compatibility-tag caches for locally built wheel files.
+1
Yes. Renaming is a hack, to be honest, and improving the caching strategy would be my preference. Paul
that neither pip nor the setuptools backend should not change the tags it applies to wheels by default).
I'm a bit confused -- are we talking about the backwards compatible path to the future -- or the end-game? In short -- I'm sure we'll have to do some hacky stuff to keep backwards compatibility, but the end game should be a clean separation of concerns : Pip (or any front end) should never "build a wheel", and it certainly shouldn't have to know enough about what's in a wheel to be re-naming it for generic python vs cpython. The package manager should manage the package, not built it, or change it. Surely the build system should know how to correctly name the wheel it builds. As to using pip to build wheels -- there is good reason to do that now, but in s post PEP 517 world, one would call the build system directly to build a wheel-- after all, all pip should be doing is calling the build system anyway. -CHB
I'm not against renaming a wheel to adjust the tags, but it is a strange caching strategy. wheel is due for a 'retag' feature that does a more rigorous job of changing tags. The build system should name its wheels correctly. Normally the pyNN tag would be used for a pure distribution and the cpNN tag when there are extensions. If you want to affect the tagging when called through PEP 517, edit the source's pyproject.toml / setup.cfg / setup.py. It will probably always make sense to use pip to build wheels when you want to build wheels for a project and all its dependencies. Then you can keep these wheels in a folder for repeatable offline deployment. On Thu, Aug 31, 2017 at 11:48 AM Chris Barker - NOAA Federal < chris.barker@noaa.gov> wrote:
that neither pip nor the setuptools backend should not change the tags it applies to wheels by default).
I'm a bit confused -- are we talking about the backwards compatible path to the future -- or the end-game?
In short -- I'm sure we'll have to do some hacky stuff to keep backwards compatibility, but the end game should be a clean separation of concerns :
Pip (or any front end) should never "build a wheel", and it certainly shouldn't have to know enough about what's in a wheel to be re-naming it for generic python vs cpython.
The package manager should manage the package, not built it, or change it.
Surely the build system should know how to correctly name the wheel it builds.
As to using pip to build wheels -- there is good reason to do that now, but in s post PEP 517 world, one would call the build system directly to build a wheel-- after all, all pip should be doing is calling the build system anyway.
-CHB _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 31 August 2017 at 16:41, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
As to using pip to build wheels -- there is good reason to do that now, but in s post PEP 517 world, one would call the build system directly to build a wheel-- after all, all pip should be doing is calling the build system anyway.
I disagree - "pip wheel" will still be useful, for example to obtain a wheel from PyPI either by downloading or download & build. Also just to have a unified interface that works regardless of the project backend - if a project switches from setuptools to flit, for example, it would be good if deployment and test scripts didn't have to change. So I still expect pip wheel to be useful in a post-PEP 517 world. Paul
On Aug 31, 2017, at 11:41 AM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
that neither pip nor the setuptools backend should not change the tags it applies to wheels by default).
I'm a bit confused -- are we talking about the backwards compatible path to the future -- or the end-game?
In short -- I'm sure we'll have to do some hacky stuff to keep backwards compatibility, but the end game should be a clean separation of concerns :
Pip (or any front end) should never "build a wheel", and it certainly shouldn't have to know enough about what's in a wheel to be re-naming it for generic python vs cpython.
The package manager should manage the package, not built it, or change it.
Surely the build system should know how to correctly name the wheel it builds.
As to using pip to build wheels -- there is good reason to do that now, but in s post PEP 517 world, one would call the build system directly to build a wheel-- after all, all pip should be doing is calling the build system anyway.
Pip purposely overrides what setuptools tags the wheel with in order to make it extremely specific to the Python it is currently being run on. Now, a big part of why it needs to do this, is because a lot of setup.py were not written with wheel in mind, so they produce broken wheels by default. One such example is a setup.py that produces a pure Python wheel for PyPy, and a C-extension wheel for Python. If someone installs that project on PyPy first and they only have a sdist available on PyPI, then pip will cache a pure Python wheel for that project, and by default (without this bit to force it) that wheel would be acceptable for CPython too and would have been used instead of building a wheel with a compiled extension for CPython. You could argue that the new crop of build backends should handle this better— and hopefully you’re correct— but I think that the two desires are at odd with one another. Most build backends are going to want to, by default, tag the wheel with as general of a tag as they think will work given the information available to them, whereas with pip’s internal wheel cache, we generally want that wheel to be as specific to the current version of Python as possible. Another possible solution is to stop using the wheel tagging format to encode this information for pip’s internal wheel cache and to encode it into the path, so that instead of having PyPy and CPython share a cache directory, they each have their own. That isn’t an unreasonable mechanism for doing that and which one gets used is an implementation detail of pip. — Donald Stufft
On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
The package manager should manage the package, not built it, or change it.
Surely the build system should know how to correctly name the wheel it builds.
It's probably worth mentioning the specific problem that motivated pip to start doing this. It used to be standard, and is still quite common, for setup.py scripts to contain stuff like: install_requires = [...] if sys.version_info < (3, 4): install_requires += [...] if platform.python_implementation() == "PyPy": install_requires += [...] setup(..., install_requires=install_requires) This kind of logic in setup.py worked fine in the old days when all you did was 'setup.py install', but then wheels came along and retroactively turned lots of setup.py scripts from working into broken. The problem is that with this kind of setup.py, setuptools has *no idea* that the install_requires you gave it would have been different if you had run setup.py with a different version of Python, so when it has to assign Python tags to a built wheel it guesses wrong and uses ones that are too general. The right way to do this is to use PEP 508 environment markers: https://www.python.org/dev/peps/pep-0508/#environment-markers or the non-standard extras hack: https://wheel.readthedocs.io/en/latest/#defining-conditional-dependencies Both of these let you export the whole requirements-choosing logic into the wheel metadata, so that it can be evaluated at install time instead of build time. But it will take a while for existing setup.py files transition to using those, and in the mean time pip can't assume that a random wheel generated by 'setup.py bdist_wheel' has accurate Python tags. Hopefully new legacy-free backends will get this right from the start. For example flit makes it impossible to get this wrong. -n -- Nathaniel J. Smith -- https://vorpus.org
Don't be afraid of extras. They have not been PEP'd but are required for a useful Python packaging system. install_requires is just a shorthand for extras with the "" empty string as their name. Too bad we couldn't agree on ; vs : as the delimiter though. On Thu, Aug 31, 2017 at 2:04 PM Nathaniel Smith <njs@pobox.com> wrote:
On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
The package manager should manage the package, not built it, or change it.
Surely the build system should know how to correctly name the wheel it builds.
It's probably worth mentioning the specific problem that motivated pip to start doing this.
It used to be standard, and is still quite common, for setup.py scripts to contain stuff like:
install_requires = [...] if sys.version_info < (3, 4): install_requires += [...] if platform.python_implementation() == "PyPy": install_requires += [...]
setup(..., install_requires=install_requires)
This kind of logic in setup.py worked fine in the old days when all you did was 'setup.py install', but then wheels came along and retroactively turned lots of setup.py scripts from working into broken. The problem is that with this kind of setup.py, setuptools has *no idea* that the install_requires you gave it would have been different if you had run setup.py with a different version of Python, so when it has to assign Python tags to a built wheel it guesses wrong and uses ones that are too general.
The right way to do this is to use PEP 508 environment markers: https://www.python.org/dev/peps/pep-0508/#environment-markers or the non-standard extras hack:
https://wheel.readthedocs.io/en/latest/#defining-conditional-dependencies Both of these let you export the whole requirements-choosing logic into the wheel metadata, so that it can be evaluated at install time instead of build time.
But it will take a while for existing setup.py files transition to using those, and in the mean time pip can't assume that a random wheel generated by 'setup.py bdist_wheel' has accurate Python tags.
Hopefully new legacy-free backends will get this right from the start. For example flit makes it impossible to get this wrong.
-n
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
There are two points that I would note here: 1. setuptools handling of environment markers in wheels was actually broken until recently. 2. Even now, people still prefer runtime detection of environment markers to work around pip's upgrade behavior and old setuptools versions. There needs to be a public effort to change this behavior. For #2, it's really not a clean solution, but it would be more effective to scan the bytecode of setup.py and then warn users that this behavior is deprecated. I expect that you would get a lot of complaints from package authors but that is the point. 2017-08-31 13:03 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
The package manager should manage the package, not built it, or change it.
Surely the build system should know how to correctly name the wheel it builds.
It's probably worth mentioning the specific problem that motivated pip to start doing this.
It used to be standard, and is still quite common, for setup.py scripts to contain stuff like:
install_requires = [...] if sys.version_info < (3, 4): install_requires += [...] if platform.python_implementation() == "PyPy": install_requires += [...]
setup(..., install_requires=install_requires)
This kind of logic in setup.py worked fine in the old days when all you did was 'setup.py install', but then wheels came along and retroactively turned lots of setup.py scripts from working into broken. The problem is that with this kind of setup.py, setuptools has *no idea* that the install_requires you gave it would have been different if you had run setup.py with a different version of Python, so when it has to assign Python tags to a built wheel it guesses wrong and uses ones that are too general.
The right way to do this is to use PEP 508 environment markers: https://www.python.org/dev/peps/pep-0508/#environment-markers or the non-standard extras hack: https://wheel.readthedocs.io/en/latest/#defining- conditional-dependencies Both of these let you export the whole requirements-choosing logic into the wheel metadata, so that it can be evaluated at install time instead of build time.
But it will take a while for existing setup.py files transition to using those, and in the mean time pip can't assume that a random wheel generated by 'setup.py bdist_wheel' has accurate Python tags.
Hopefully new legacy-free backends will get this right from the start. For example flit makes it impossible to get this wrong.
-n
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith <njs@pobox.com> wrote:
Surely the build system should know how to correctly name the wheel it builds.
It's probably worth mentioning the specific problem that motivated pip to start doing this.
It used to be standard, and is still quite common, for setup.py scripts to contain stuff like:
install_requires = [...] if sys.version_info < (3, 4): install_requires += [...] if platform.python_implementation() == "PyPy": install_requires += [...]
setup(..., install_requires=install_requires)
This kind of logic in setup.py worked fine in the old days when all you did was 'setup.py install', but then wheels came along
And indeed, setuptools originally used easy_install, which was part of setuptools... and
retroactively turned lots of setup.py scripts from working into broken. The problem is that with this kind of setup.py,
But it will take a while for existing setup.py files transition to using those, and in the mean time pip can't assume that a random wheel generated by 'setup.py bdist_wheel' has accurate Python tags.
This was my original point -- I understand that we want "pip install" to continue to work for, hopefully, everything it works for now. But I do think we should be clear about what is a hack for backward compatibility, and what is part of the designed functionality. Sorry to be poking at all this from the fringes (Not having been all that involved in a very long discussion), it's just that the whole distutils--setuptools--pip--distribute--setuptools--pip stack has a LOT of legacy cruft, and I'm concerned that the efforts for backward compatibility may end up leading us to another poorly de-coupled design.
Hopefully new legacy-free backends will get this right from the start.
exactly -- let's keep the "backward compatibility hack" labels clear! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
I don't buy the argument that pip needs to keep separate caches for different implementations. According to the wheel specification, a wheel is valid for the Python implementations that it specifies. This is therefore a blatant violation by setuptools of an existing specification that, in theory, should be ignored by pip. That said, setuptools is currently the de facto build system, meaning that we cannot simply break the status quo. So here's my proposal: 1. For now, wheels are renamed if and only if the backend is setuptools. 2. In the future, we introduce a new setup() function that is imported from a different module path that requires compliance with the specification. 3. Once the vast majority of projects are using the new setup function, we can adjust the pep 517 setuptools build backend to use the most restrictive tag for the old function 4. We can finally remove autobuilding from pip. One thing to keep in mind is that there are quite a few projects on pypa with pure python source distributions uploaded that will not be updated and people may still desire to use. We want pip to be able to still build and install them. 2017-08-31 16:29 GMT-05:00 Chris Barker <chris.barker@noaa.gov>:
On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith <njs@pobox.com> wrote:
Surely the build system should know how to correctly name the wheel it builds.
It's probably worth mentioning the specific problem that motivated pip to start doing this.
It used to be standard, and is still quite common, for setup.py scripts to contain stuff like:
install_requires = [...] if sys.version_info < (3, 4): install_requires += [...] if platform.python_implementation() == "PyPy": install_requires += [...]
setup(..., install_requires=install_requires)
This kind of logic in setup.py worked fine in the old days when all you did was 'setup.py install', but then wheels came along
And indeed, setuptools originally used easy_install, which was part of setuptools...
and
retroactively turned lots of setup.py scripts from working into broken. The problem is that with this kind of setup.py,
But it will take a while for existing setup.py files transition to using those, and in the mean time pip can't assume that a random wheel generated by 'setup.py bdist_wheel' has accurate Python tags.
This was my original point -- I understand that we want "pip install" to continue to work for, hopefully, everything it works for now.
But I do think we should be clear about what is a hack for backward compatibility, and what is part of the designed functionality.
Sorry to be poking at all this from the fringes (Not having been all that involved in a very long discussion), it's just that the whole
distutils--setuptools--pip--distribute--setuptools--pip
stack has a LOT of legacy cruft, and I'm concerned that the efforts for backward compatibility may end up leading us to another poorly de-coupled design.
Hopefully new legacy-free backends will get this right from the start.
exactly -- let's keep the "backward compatibility hack" labels clear!
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
One thing to keep in mind is that there are quite a few projects on pypa with pure python source distributions uploaded that will not be updated and people may still desire to use. We want pip to be able to still build and install them. That is the challenge! But the motivating use case here was a package with a c extension that was optionally built. That's the rare case. Regular old pure-python packages are the usual case, and a lot easier. -CHB 2017-08-31 16:29 GMT-05:00 Chris Barker <chris.barker@noaa.gov>:
On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith <njs@pobox.com> wrote:
Surely the build system should know how to correctly name the wheel it builds.
It's probably worth mentioning the specific problem that motivated pip to start doing this.
It used to be standard, and is still quite common, for setup.py scripts to contain stuff like:
install_requires = [...] if sys.version_info < (3, 4): install_requires += [...] if platform.python_implementation() == "PyPy": install_requires += [...]
setup(..., install_requires=install_requires)
This kind of logic in setup.py worked fine in the old days when all you did was 'setup.py install', but then wheels came along
And indeed, setuptools originally used easy_install, which was part of setuptools...
and
retroactively turned lots of setup.py scripts from working into broken. The problem is that with this kind of setup.py,
But it will take a while for existing setup.py files transition to using those, and in the mean time pip can't assume that a random wheel generated by 'setup.py bdist_wheel' has accurate Python tags.
This was my original point -- I understand that we want "pip install" to continue to work for, hopefully, everything it works for now.
But I do think we should be clear about what is a hack for backward compatibility, and what is part of the designed functionality.
Sorry to be poking at all this from the fringes (Not having been all that involved in a very long discussion), it's just that the whole
distutils--setuptools--pip--distribute--setuptools--pip
stack has a LOT of legacy cruft, and I'm concerned that the efforts for backward compatibility may end up leading us to another poorly de-coupled design.
Hopefully new legacy-free backends will get this right from the start.
exactly -- let's keep the "backward compatibility hack" labels clear!
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 1 September 2017 at 07:29, Chris Barker <chris.barker@noaa.gov> wrote:
But I do think we should be clear about what is a hack for backward compatibility, and what is part of the designed functionality.
Right, and I think it's pretty clear that the problem xoviat raised is with the way pip's implicit local artifact cache works, since it doesn't reliably distinguish between "this cached wheel was downloaded from a repository" and "this wheel was generated locally with a particular version of Python". Currently pip is working around that problem by overriding the compatibility tags produced by setup.py to ensure the generated wheel files are interpreter specific, but PEP 517 deliberately doesn't let frontends do that as part of the initial build process (instead, if they want to adjust the tags, they need to do it as a post-processing step). Since PEP 517 breaks the current workaround for the caching scheme being inaccurate, the most suitable response is to instead fix pip's caching scheme to use a two tier local cache: 1. Wheel download cache (if this gets a hit, don't even check the package repo) 2. Check the package repo for a wheel file (and if you find one, add it to the download cache) 3. Local build cache (make this interpreter implementation specific) 4. Build a local wheel that's specific to the running interpreter implementation Independently of that though, it's also important to note that pip probably *won't* switch to invoking legacy setup.py files through the PEP 517 backend - instead, it will only do that if there's a pyproject.toml file that says "Use the setuptools PEP 517 backend". In such cases, it's reasonable to expect that the project will also be using environment markers appropriately, and not putting conditional dependency installation logic directly into setup.py. Autobuilding support definitely *won't* be removed from pip, as there will not only always be a large number of projects that don't provide their own prebuilt wheel files, we also have plenty of PyPI users that explicitly *opt out* of using publisher-provided pre-built binaries. While Linux distributions are the most common example (see [1] for Fedora's policy, for example), we're not the only ones that have that kind of rule in place. Cheers, Nick. [1] https://fedoraproject.org/wiki/Packaging:Guidelines#No_inclusion_of_pre-buil... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sep 1, 2017, at 12:23 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
1. Wheel download cache (if this gets a hit, don't even check the package repo) 2. Check the package repo for a wheel file (and if you find one, add it to the download cache) 3. Local build cache (make this interpreter implementation specific) 4. Build a local wheel that's specific to the running interpreter implementation
This is basically what exists now. We just force a compatibility tag to handle (4).
On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncoghlan@gmail.com> wrote: since it
doesn't reliably distinguish between "this cached wheel was downloaded from a repository" and "this wheel was generated locally with a particular version of Python".
It shouldn't have to. sigh.
PEP 517 deliberately doesn't let
frontends do that as part of the initial build process (instead, if they want to adjust the tags, they need to do it as a post-processing step).
Since PEP 517 breaks the current workaround for the caching scheme being inaccurate, the most suitable response is to instead fix pip's caching scheme to use a two tier local cache:
I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools? If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has? I'm trying to think through the use case presented the other day: A package can be build either for all python's or for cpython in particular. That selection is (currently) done at build time, by custom code in setup.py. In this case, the build system SHOULD be able to build the proper wheel, and label it correctly. But there might need to be a way to tell the build system which you want. That logic isn't in existing setup.py files, but there is only so much you can do! Then there is the caching problem. If pip finds a pure-python wheel that matches the spec in the cache, it won't know to go looking for a cpython specific one. I can't see how separating the downloaded cache from the locally built cache will solve that problem: If a pure-python wheel was already downloaded, then that'll be in the downloaded cache anyway. I think the core problem here is having, for instance, pypy and cpython sharing a wheel cache. Is it really that important to support that? If so, this is a total kludge, but maybe the pip cache could keep a record-- I looked for a cpython specific wheel, and it doesn't exist, so I never need to look again. Or -- maybe more kludgy -- every time pip puts a wheel in the cache via cpython, it makes a link with the cpython name. Independently of that though, it's also important to note that pip
probably *won't* switch to invoking legacy setup.py files through the PEP 517 backend - instead, it will only do that if there's a pyproject.toml file that says "Use the setuptools PEP 517 backend".
That's probably a good way to keep backward compatibility while moving forward. Which means pip can have two modes -- legacy mode and pep517 mode. Which may encourage me to stop being such a noodge -- As long as we're talking legacy mode, we can be as hacky as we need to be. Autobuilding support definitely *won't* be removed from pip, I'm not suggesting it should be (and I think I may be the only one that might have been) However, I do think auto-building support does not have to support all the complex use cases. we also have plenty of PyPI users that
explicitly *opt out* of using publisher-provided pre-built binaries. While Linux distributions are the most common example (see [1] for Fedora's policy, for example), we're not the only ones that have that kind of rule in place.
But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them.
Condo-forge, for example, almost always builds from source -- sometimes an sdist from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort). Do the Linux distros use pip to build their packages? I tried to do that with conda-packages, and failed due to pip's caching behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds. So I ( and the entire community, as far as I can tell) stuck with calling setuptools directly - via setup.py, but with setuptools-specific flags) As pip is a package management tool, it makes sense that other package managers wouldn't need to use it. NOTE: I do use pip to build conda packages from wheels in a couple cases -- but that's considered a work around for my pathetic build skills -- and even then, I don't have pip download the wheel -- that's conda-build's job. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
I don't think this is a big deal. Compatibility is a guess, wheel tags do not and are not intended to guarantee compatibility especially when the wheel was built locally for caching purposes and not intentionally published. Anything can happen during a build. PEP 517 won't change this. So right now pip has an odd way of segregating its wheel cache per interpreter that involves obscure setup.py arguments. That can be fixed without re-tagging or asking the build system to re-tag wheels or otherwise interacting with the new PEP.
On Fri, Sep 1, 2017 at 11:30 AM, Chris Barker <chris.barker@noaa.gov> wrote:
I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools?
If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has?
Some setup.py files are written by project authors who want to use them to generate wheels for uploading to PyPI, so they're carefully written to generate good wheels, the wheels go through QA, etc. They rely on setuptools's current (and fairly sensible) defaults for how to tag wheels, and if those go wrong, then they take the responsibility for fixing things. Other setup.py files were written by project authors who never considered any possibility outside of 'setup.py install', and haven't changed since. For them, setuptools's defaults are not so great, but the authors don't care, because they never guaranteed that it would work. Setuptools can't tell which kind of setup.py is calling it. But pip can make a pretty good guess: if it found a wheel on pypi, then someone had to have uploaded it, and it's that person's job to make sure the tags are correct. OTOH if it's taking some random setup.py file it found in an sdist, then it could be the second type, so better to play it safe and use a more restrictive wheel tag. It's a bit quirky and annoying, but that's life. And in the grand scheme of things this isn't a big deal. The only program that has to care about this is pip, and pip can always change to a different heuristic if the situation changes. -n -- Nathaniel J. Smith -- https://vorpus.org
On 1 September 2017 at 19:30, Chris Barker <chris.barker@noaa.gov> wrote:
I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools?
If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has?
Simple answer - for historical reasons, and because it's essentially baked into the design of distutils, setuptools cannot guarantee that wheels created by two different Python interpreters on a given machine will NOT be:given the same tags and yet not usable by both. The bug isn't directly in setuptools - it's typically because the project is doing unsafe things in setup.py. This is known to happen in real life - pip has had bug reports for it. In practice, it's sufficiently difficult to debug the situation, and sufficiently hard to get fixes made in all of the upstream projects that might need fixing, that we need to do something in pip. The solution used in pip (which is a heuristic based on the bug reports we've seen) is to segregate wheels that we build internally by the Python interpreter that was used in the build. We don't do that for downloaded wheels, as we trust that they have been packaged correctly, with suitable tags. This sort of "it's really a problem further up the chain, but pip can't afford to assume it'll get fixed there" problem is typical of the reason the pp developers have been uncomfortable with the "trust the backend" approach in the PEP - but they are very much about legacy issues with the design of distutils that setuptools inherits, and I imagine that any new build systems and backends will tend to avoid such problems in their design. So yes, to address another characterisation of this issue that was mentioned, it's a workaround for legacy issues. But that legacy will be with us for a long time to come, in practice. Sorry - history lesson over ;-) Paul
On Sep 1, 2017, at 2:30 PM, Chris Barker <Chris.Barker@noaa.gov> wrote:
On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:
since it doesn't reliably distinguish between "this cached wheel was downloaded from a repository" and "this wheel was generated locally with a particular version of Python".
It shouldn't have to. sigh.
PEP 517 deliberately doesn't let frontends do that as part of the initial build process (instead, if they want to adjust the tags, they need to do it as a post-processing step).
Since PEP 517 breaks the current workaround for the caching scheme being inaccurate, the most suitable response is to instead fix pip's caching scheme to use a two tier local cache:
I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools?
If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has?
Someone building a wheel for distribution is likely intimately aware of that project, and can take care to ensure that the wheel is built in such a way that it is giving people the most optimal behavior. Pip is auto building wheels without human intervention, and as such there is nobody there to make sure that we’re not accidentally creating a too-broad wheel, so we want to ensure that we have some mechanism in place for not re-using the wheel across boundaries that might cause issues.
we also have plenty of PyPI users that explicitly *opt out* of using publisher-provided pre-built binaries. While Linux distributions are the most common example (see [1] for Fedora's policy, for example), we're not the only ones that have that kind of rule in place.
But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them.
Condo-forge, for example, almost always builds from source -- sometimes an sdist from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort).
Pip supports more systems than Conda does, and we do that by relying on auto building support. Pip supports systems that don’t have a wheel compatibility tag defined for them, and for which we’re unlikely to ever have any wheels published for (much less wide spread). It’s pretty easy to cover Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long. Pip works in all these situations, and it does so by relying on building from source.
Do the Linux distros use pip to build their packages?
Not that I am aware of.
I tried to do that with conda-packages, and failed due to pip's caching behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds.
Adding —no-cache-dir disables all of pip’s caching. — Donald Stufft
Great discussions on this PEP over the proposal, here and in the archives. I've just now gotten plugged in to the distutils distro, at a seemingly perfect time... somewhat new to Python packaging and still learning. Many of the issues observed by users in the mail chain are reasons why we had marching orders at the organization I work for to internally build all whls, on the platforms we support by using 'pip wheel' with specific flags. One of these flags is to pull source only (pip download, no-binary) and direct 'pip wheel' to build with that source dir. For the caching issue we ended up using Docker, spinning up a new instance per build with the platforms we support, running the build, and uploading those artifacts to our internal repo for pulls by the users, then thrashing that container wholly. In a similar way, I see the 'venv' requirement of 517 mirroring this process. The presupposition of this pipeline we built is that metadata and source repo structures are trusted to produce accurate binaries... the more I learn about how projects / packages have set up their build instructions, the more horrified I become in the variance of the implementations out in the wild. 517 and 518 to me seems like a way to reign in some of these variances and formalize them, especially around setup.py and metadata... Is that correct or am I way off base with the discussion and proposal so far? Best regards, C. Schmautz
On Sep 2, 2017, at 12:42 PM, Donald Stufft <donald@stufft.io> wrote:
On Sep 1, 2017, at 2:30 PM, Chris Barker <Chris.Barker@noaa.gov> wrote:
On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
since it doesn't reliably distinguish between "this cached wheel was downloaded from a repository" and "this wheel was generated locally with a particular version of Python".
It shouldn't have to. sigh.
PEP 517 deliberately doesn't let frontends do that as part of the initial build process (instead, if they want to adjust the tags, they need to do it as a post-processing step).
Since PEP 517 breaks the current workaround for the caching scheme being inaccurate, the most suitable response is to instead fix pip's caching scheme to use a two tier local cache:
I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools?
If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has?
Someone building a wheel for distribution is likely intimately aware of that project, and can take care to ensure that the wheel is built in such a way that it is giving people the most optimal behavior. Pip is auto building wheels without human intervention, and as such there is nobody there to make sure that we’re not accidentally creating a too-broad wheel, so we want to ensure that we have some mechanism in place for not re-using the wheel across boundaries that might cause issues.
we also have plenty of PyPI users that explicitly *opt out* of using publisher-provided pre-built binaries. While Linux distributions are the most common example (see [1] for Fedora's policy, for example), we're not the only ones that have that kind of rule in place.
But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them.
Condo-forge, for example, almost always builds from source -- sometimes an sdist from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort).
Pip supports more systems than Conda does, and we do that by relying on auto building support. Pip supports systems that don’t have a wheel compatibility tag defined for them, and for which we’re unlikely to ever have any wheels published for (much less wide spread). It’s pretty easy to cover Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long.
Pip works in all these situations, and it does so by relying on building from source.
Do the Linux distros use pip to build their packages?
Not that I am aware of.
I tried to do that with conda-packages, and failed due to pip's caching behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds.
Adding —no-cache-dir disables all of pip’s caching.
— Donald Stufft
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
One more issue that has come up is that "--no-user-cfg" seems to be passed to the egg_info invocation if the "isolated" parameter is enabled. I don't understand what this does, but it is again not defined in the PEP 517 interface. Should we always pass this parameter or should we never pass it? 2017-09-02 14:42 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Sep 1, 2017, at 2:30 PM, Chris Barker <Chris.Barker@noaa.gov> wrote:
On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
since it
doesn't reliably distinguish between "this cached wheel was downloaded from a repository" and "this wheel was generated locally with a particular version of Python".
It shouldn't have to. sigh.
PEP 517 deliberately doesn't let
frontends do that as part of the initial build process (instead, if they want to adjust the tags, they need to do it as a post-processing step).
Since PEP 517 breaks the current workaround for the caching scheme being inaccurate, the most suitable response is to instead fix pip's caching scheme to use a two tier local cache:
I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools?
If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has?
Someone building a wheel for distribution is likely intimately aware of that project, and can take care to ensure that the wheel is built in such a way that it is giving people the most optimal behavior. Pip is auto building wheels without human intervention, and as such there is nobody there to make sure that we’re not accidentally creating a too-broad wheel, so we want to ensure that we have some mechanism in place for not re-using the wheel across boundaries that might cause issues.
we also have plenty of PyPI users that
explicitly *opt out* of using publisher-provided pre-built binaries. While Linux distributions are the most common example (see [1] for Fedora's policy, for example), we're not the only ones that have that kind of rule in place.
But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them.
Condo-forge, for example, almost always builds from source -- sometimes an sdist from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort).
Pip supports more systems than Conda does, and we do that by relying on auto building support. Pip supports systems that don’t have a wheel compatibility tag defined for them, and for which we’re unlikely to ever have any wheels published for (much less wide spread). It’s pretty easy to cover Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long.
Pip works in all these situations, and it does so by relying on building from source.
Do the Linux distros use pip to build their packages?
Not that I am aware of.
I tried to do that with conda-packages, and failed due to pip's caching behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds.
Adding —no-cache-dir disables all of pip’s caching.
— Donald Stufft
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Donald, This was your work in https://github.com/pypa/pip/pull/2169. Unfortunately the comments were quite sparse. 2017-09-02 18:25 GMT-05:00 xoviat <xoviat@gmail.com>:
One more issue that has come up is that "--no-user-cfg" seems to be passed to the egg_info invocation if the "isolated" parameter is enabled. I don't understand what this does, but it is again not defined in the PEP 517 interface. Should we always pass this parameter or should we never pass it?
2017-09-02 14:42 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Sep 1, 2017, at 2:30 PM, Chris Barker <Chris.Barker@noaa.gov> wrote:
On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
since it
doesn't reliably distinguish between "this cached wheel was downloaded from a repository" and "this wheel was generated locally with a particular version of Python".
It shouldn't have to. sigh.
PEP 517 deliberately doesn't let
frontends do that as part of the initial build process (instead, if they want to adjust the tags, they need to do it as a post-processing step).
Since PEP 517 breaks the current workaround for the caching scheme being inaccurate, the most suitable response is to instead fix pip's caching scheme to use a two tier local cache:
I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools?
If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has?
Someone building a wheel for distribution is likely intimately aware of that project, and can take care to ensure that the wheel is built in such a way that it is giving people the most optimal behavior. Pip is auto building wheels without human intervention, and as such there is nobody there to make sure that we’re not accidentally creating a too-broad wheel, so we want to ensure that we have some mechanism in place for not re-using the wheel across boundaries that might cause issues.
we also have plenty of PyPI users that
explicitly *opt out* of using publisher-provided pre-built binaries. While Linux distributions are the most common example (see [1] for Fedora's policy, for example), we're not the only ones that have that kind of rule in place.
But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them.
Condo-forge, for example, almost always builds from source -- sometimes an sdist from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort).
Pip supports more systems than Conda does, and we do that by relying on auto building support. Pip supports systems that don’t have a wheel compatibility tag defined for them, and for which we’re unlikely to ever have any wheels published for (much less wide spread). It’s pretty easy to cover Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long.
Pip works in all these situations, and it does so by relying on building from source.
Do the Linux distros use pip to build their packages?
Not that I am aware of.
I tried to do that with conda-packages, and failed due to pip's caching behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds.
Adding —no-cache-dir disables all of pip’s caching.
— Donald Stufft
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Whatever it was, removing it seems to have had no effect on the tests. I will remove it unless someone has an objection. 2017-09-02 18:26 GMT-05:00 xoviat <xoviat@gmail.com>:
Donald,
This was your work in https://github.com/pypa/pip/pull/2169. Unfortunately the comments were quite sparse.
2017-09-02 18:25 GMT-05:00 xoviat <xoviat@gmail.com>:
One more issue that has come up is that "--no-user-cfg" seems to be passed to the egg_info invocation if the "isolated" parameter is enabled. I don't understand what this does, but it is again not defined in the PEP 517 interface. Should we always pass this parameter or should we never pass it?
2017-09-02 14:42 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Sep 1, 2017, at 2:30 PM, Chris Barker <Chris.Barker@noaa.gov> wrote:
On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
since it
doesn't reliably distinguish between "this cached wheel was downloaded from a repository" and "this wheel was generated locally with a particular version of Python".
It shouldn't have to. sigh.
PEP 517 deliberately doesn't let
frontends do that as part of the initial build process (instead, if they want to adjust the tags, they need to do it as a post-processing step).
Since PEP 517 breaks the current workaround for the caching scheme being inaccurate, the most suitable response is to instead fix pip's caching scheme to use a two tier local cache:
I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools?
If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has?
Someone building a wheel for distribution is likely intimately aware of that project, and can take care to ensure that the wheel is built in such a way that it is giving people the most optimal behavior. Pip is auto building wheels without human intervention, and as such there is nobody there to make sure that we’re not accidentally creating a too-broad wheel, so we want to ensure that we have some mechanism in place for not re-using the wheel across boundaries that might cause issues.
we also have plenty of PyPI users that
explicitly *opt out* of using publisher-provided pre-built binaries. While Linux distributions are the most common example (see [1] for Fedora's policy, for example), we're not the only ones that have that kind of rule in place.
But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them.
Condo-forge, for example, almost always builds from source -- sometimes an sdist from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort).
Pip supports more systems than Conda does, and we do that by relying on auto building support. Pip supports systems that don’t have a wheel compatibility tag defined for them, and for which we’re unlikely to ever have any wheels published for (much less wide spread). It’s pretty easy to cover Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long.
Pip works in all these situations, and it does so by relying on building from source.
Do the Linux distros use pip to build their packages?
Not that I am aware of.
I tried to do that with conda-packages, and failed due to pip's caching behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds.
Adding —no-cache-dir disables all of pip’s caching.
— Donald Stufft
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Sat, Sep 2, 2017 at 5:17 PM xoviat <xoviat@gmail.com> wrote:
Whatever it was, removing it seems to have had no effect on the tests. I will remove it unless someone has an objection.
Just FYI, I wouldn't take the tests still passing as a major signal. I've noticed there are even common code paths / functional scenarios that aren't under test. --Chris
2017-09-02 18:26 GMT-05:00 xoviat <xoviat@gmail.com>:
Donald,
This was your work in https://github.com/pypa/pip/pull/2169. Unfortunately the comments were quite sparse.
2017-09-02 18:25 GMT-05:00 xoviat <xoviat@gmail.com>:
One more issue that has come up is that "--no-user-cfg" seems to be passed to the egg_info invocation if the "isolated" parameter is enabled. I don't understand what this does, but it is again not defined in the PEP 517 interface. Should we always pass this parameter or should we never pass it?
2017-09-02 14:42 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Sep 1, 2017, at 2:30 PM, Chris Barker <Chris.Barker@noaa.gov> wrote:
On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
since it
doesn't reliably distinguish between "this cached wheel was downloaded from a repository" and "this wheel was generated locally with a particular version of Python".
It shouldn't have to. sigh.
PEP 517 deliberately doesn't let
frontends do that as part of the initial build process (instead, if they want to adjust the tags, they need to do it as a post-processing step).
Since PEP 517 breaks the current workaround for the caching scheme being inaccurate, the most suitable response is to instead fix pip's caching scheme to use a two tier local cache:
I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools?
If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has?
Someone building a wheel for distribution is likely intimately aware of that project, and can take care to ensure that the wheel is built in such a way that it is giving people the most optimal behavior. Pip is auto building wheels without human intervention, and as such there is nobody there to make sure that we’re not accidentally creating a too-broad wheel, so we want to ensure that we have some mechanism in place for not re-using the wheel across boundaries that might cause issues.
we also have plenty of PyPI users that
explicitly *opt out* of using publisher-provided pre-built binaries. While Linux distributions are the most common example (see [1] for Fedora's policy, for example), we're not the only ones that have that kind of rule in place.
But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them.
Condo-forge, for example, almost always builds from source -- sometimes an sdist from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort).
Pip supports more systems than Conda does, and we do that by relying on auto building support. Pip supports systems that don’t have a wheel compatibility tag defined for them, and for which we’re unlikely to ever have any wheels published for (much less wide spread). It’s pretty easy to cover Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long.
Pip works in all these situations, and it does so by relying on building from source.
Do the Linux distros use pip to build their packages?
Not that I am aware of.
I tried to do that with conda-packages, and failed due to pip's caching behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds.
Adding —no-cache-dir disables all of pip’s caching.
— Donald Stufft
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Just an update for everyone here: 1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info rather than egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much more efficient release system than pip, I anticipate having a release of setuptools first and then we can switch pip over to requiring a newer setuptools via PEP 518. 2017-09-02 19:51 GMT-05:00 Chris Jerdonek <chris.jerdonek@gmail.com>:
On Sat, Sep 2, 2017 at 5:17 PM xoviat <xoviat@gmail.com> wrote:
Whatever it was, removing it seems to have had no effect on the tests. I will remove it unless someone has an objection.
Just FYI, I wouldn't take the tests still passing as a major signal. I've noticed there are even common code paths / functional scenarios that aren't under test.
--Chris
2017-09-02 18:26 GMT-05:00 xoviat <xoviat@gmail.com>:
Donald,
This was your work in https://github.com/pypa/pip/pull/2169. Unfortunately the comments were quite sparse.
2017-09-02 18:25 GMT-05:00 xoviat <xoviat@gmail.com>:
One more issue that has come up is that "--no-user-cfg" seems to be passed to the egg_info invocation if the "isolated" parameter is enabled. I don't understand what this does, but it is again not defined in the PEP 517 interface. Should we always pass this parameter or should we never pass it?
2017-09-02 14:42 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Sep 1, 2017, at 2:30 PM, Chris Barker <Chris.Barker@noaa.gov> wrote:
On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
since it
doesn't reliably distinguish between "this cached wheel was downloaded from a repository" and "this wheel was generated locally with a particular version of Python".
It shouldn't have to. sigh.
PEP 517 deliberately doesn't let
frontends do that as part of the initial build process (instead, if they want to adjust the tags, they need to do it as a post-processing step).
Since PEP 517 breaks the current workaround for the caching scheme being inaccurate, the most suitable response is to instead fix pip's caching scheme to use a two tier local cache:
I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools?
If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has?
Someone building a wheel for distribution is likely intimately aware of that project, and can take care to ensure that the wheel is built in such a way that it is giving people the most optimal behavior. Pip is auto building wheels without human intervention, and as such there is nobody there to make sure that we’re not accidentally creating a too-broad wheel, so we want to ensure that we have some mechanism in place for not re-using the wheel across boundaries that might cause issues.
we also have plenty of PyPI users that
explicitly *opt out* of using publisher-provided pre-built binaries. While Linux distributions are the most common example (see [1] for Fedora's policy, for example), we're not the only ones that have that kind of rule in place.
But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them.
Condo-forge, for example, almost always builds from source -- sometimes an sdist from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort).
Pip supports more systems than Conda does, and we do that by relying on auto building support. Pip supports systems that don’t have a wheel compatibility tag defined for them, and for which we’re unlikely to ever have any wheels published for (much less wide spread). It’s pretty easy to cover Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long.
Pip works in all these situations, and it does so by relying on building from source.
Do the Linux distros use pip to build their packages?
Not that I am aware of.
I tried to do that with conda-packages, and failed due to pip's caching behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds.
Adding —no-cache-dir disables all of pip’s caching.
— Donald Stufft
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote:
Just an update for everyone here:
1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info rather than egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much more efficient release system than pip, I anticipate having a release of setuptools first and then we can switch pip over to requiring a newer setuptools via PEP 518.
I don't think pip actually has any use for the PEP 517 prepare_wheel_metadata hook right now though? Historically 'setup.py egg-info' was needed to kluge around unwanted behavior in 'setup.py install', but with a PEP 517 backend that's irrelevant because 'setup.py install' is never used. And in the future when pip has a real resolver, then prepare_wheel_metadata should allow some optimizations. But right now, prepare_wheel_metadata is completely useless AFAIK. So why is 'setup.py dist_info' a blocker for things? -n -- Nathaniel J. Smith -- https://vorpus.org
Wheel has always implemented dist info by converting from egg info. It is necessary to invoke all of the egg info writers provided by setuptools or tons of packages will break. But so far dist info has only been generated as part of building a wheel. On Sun, Sep 3, 2017, 22:01 Nathaniel Smith <njs@pobox.com> wrote:
Just an update for everyone here:
1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info rather
On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote: than
egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much more efficient release system than pip, I anticipate having a release of setuptools first and then we can switch pip over to requiring a newer setuptools via PEP 518.
I don't think pip actually has any use for the PEP 517 prepare_wheel_metadata hook right now though? Historically 'setup.py egg-info' was needed to kluge around unwanted behavior in 'setup.py install', but with a PEP 517 backend that's irrelevant because 'setup.py install' is never used. And in the future when pip has a real resolver, then prepare_wheel_metadata should allow some optimizations. But right now, prepare_wheel_metadata is completely useless AFAIK.
So why is 'setup.py dist_info' a blocker for things?
-n
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Nathaniel: Pip requires egg_info to discover dependencies of source distributions so that it can build wheels all at once after downloading the requirements. I need to move pip off of egg_info as soon as possible and dist_info is required to do that. 2017-09-03 21:00 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
Just an update for everyone here:
1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info rather
On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote: than
egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much more efficient release system than pip, I anticipate having a release of setuptools first and then we can switch pip over to requiring a newer setuptools via PEP 518.
I don't think pip actually has any use for the PEP 517 prepare_wheel_metadata hook right now though? Historically 'setup.py egg-info' was needed to kluge around unwanted behavior in 'setup.py install', but with a PEP 517 backend that's irrelevant because 'setup.py install' is never used. And in the future when pip has a real resolver, then prepare_wheel_metadata should allow some optimizations. But right now, prepare_wheel_metadata is completely useless AFAIK.
So why is 'setup.py dist_info' a blocker for things?
-n
-- Nathaniel J. Smith -- https://vorpus.org
Also if someone with pip write access could please discuss and hopefully merge my initial PR on pip, I would very much appreciate it. Paul seems to be short on time. 2017-09-04 19:09 GMT-05:00 xoviat <xoviat@gmail.com>:
Nathaniel:
Pip requires egg_info to discover dependencies of source distributions so that it can build wheels all at once after downloading the requirements. I need to move pip off of egg_info as soon as possible and dist_info is required to do that.
2017-09-03 21:00 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
Just an update for everyone here:
1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info rather
On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote: than
egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much more efficient release system than pip, I anticipate having a release of setuptools first and then we can switch pip over to requiring a newer setuptools via PEP 518.
I don't think pip actually has any use for the PEP 517 prepare_wheel_metadata hook right now though? Historically 'setup.py egg-info' was needed to kluge around unwanted behavior in 'setup.py install', but with a PEP 517 backend that's irrelevant because 'setup.py install' is never used. And in the future when pip has a real resolver, then prepare_wheel_metadata should allow some optimizations. But right now, prepare_wheel_metadata is completely useless AFAIK.
So why is 'setup.py dist_info' a blocker for things?
-n
-- Nathaniel J. Smith -- https://vorpus.org
On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xoviat@gmail.com> wrote:
Nathaniel:
Pip requires egg_info to discover dependencies of source distributions so that it can build wheels all at once after downloading the requirements. I need to move pip off of egg_info as soon as possible and dist_info is required to do that.
"Requires" is a strong word -- AFAIK this is just a historical artifact. I don't really know what you're talking about in the second sentence. The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook. But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything. (In fact I can't see any reason why pip would ever call it before the resolver lands.) So either (a) there's some other reason you want a dist_info command, (b) there's some reason I'm missing why prepare_wheel_metadata matters, or (c) one of us is misunderstanding something :-). -n
2017-09-03 21:00 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote:
Just an update for everyone here:
1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info rather than egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much more efficient release system than pip, I anticipate having a release of setuptools first and then we can switch pip over to requiring a newer setuptools via PEP 518.
I don't think pip actually has any use for the PEP 517 prepare_wheel_metadata hook right now though? Historically 'setup.py egg-info' was needed to kluge around unwanted behavior in 'setup.py install', but with a PEP 517 backend that's irrelevant because 'setup.py install' is never used. And in the future when pip has a real resolver, then prepare_wheel_metadata should allow some optimizations. But right now, prepare_wheel_metadata is completely useless AFAIK.
So why is 'setup.py dist_info' a blocker for things?
-n
-- Nathaniel J. Smith -- https://vorpus.org
-- Nathaniel J. Smith -- https://vorpus.org
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook.
Yes. That is absolutely correct.
But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything.
The simplest way to start on this issue is to replace egg_info in pip with prepare_metadata_for_build_wheel. It is absolutely a historical artifact but I need to work on one issue at a time. The next issue will be replacing egg_info in pip with prepare_metadata_for_build_wheel. 2017-09-04 19:34 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xoviat@gmail.com> wrote:
Nathaniel:
Pip requires egg_info to discover dependencies of source distributions so that it can build wheels all at once after downloading the requirements. I need to move pip off of egg_info as soon as possible and dist_info is required to do that.
"Requires" is a strong word -- AFAIK this is just a historical artifact. I don't really know what you're talking about in the second sentence.
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook. But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything. (In fact I can't see any reason why pip would ever call it before the resolver lands.) So either (a) there's some other reason you want a dist_info command, (b) there's some reason I'm missing why prepare_wheel_metadata matters, or (c) one of us is misunderstanding something :-).
-n
2017-09-03 21:00 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote:
Just an update for everyone here:
1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info
rather
than egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much more efficient release system than pip, I anticipate having a release of setuptools first and then we can switch pip over to requiring a newer setuptools via PEP 518.
I don't think pip actually has any use for the PEP 517 prepare_wheel_metadata hook right now though? Historically 'setup.py egg-info' was needed to kluge around unwanted behavior in 'setup.py install', but with a PEP 517 backend that's irrelevant because 'setup.py install' is never used. And in the future when pip has a real resolver, then prepare_wheel_metadata should allow some optimizations. But right now, prepare_wheel_metadata is completely useless AFAIK.
So why is 'setup.py dist_info' a blocker for things?
-n
-- Nathaniel J. Smith -- https://vorpus.org
-- Nathaniel J. Smith -- https://vorpus.org
In any case, we're going to need this for prepare_metadata, so the question you should ask is: what are the reasons for *not* merging this? I haven't heard any so far but that doesn't mean that they don't exist. If there are none, then I don't see why we cannot merge my wheel PR and do a release. 2017-09-04 19:51 GMT-05:00 xoviat <xoviat@gmail.com>:
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook.
Yes. That is absolutely correct.
But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything.
The simplest way to start on this issue is to replace egg_info in pip with prepare_metadata_for_build_wheel. It is absolutely a historical artifact but I need to work on one issue at a time. The next issue will be replacing egg_info in pip with prepare_metadata_for_build_wheel.
2017-09-04 19:34 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xoviat@gmail.com> wrote:
Nathaniel:
Pip requires egg_info to discover dependencies of source distributions so that it can build wheels all at once after downloading the requirements. I need to move pip off of egg_info as soon as possible and dist_info is required to do that.
"Requires" is a strong word -- AFAIK this is just a historical artifact. I don't really know what you're talking about in the second sentence.
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook. But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything. (In fact I can't see any reason why pip would ever call it before the resolver lands.) So either (a) there's some other reason you want a dist_info command, (b) there's some reason I'm missing why prepare_wheel_metadata matters, or (c) one of us is misunderstanding something :-).
-n
2017-09-03 21:00 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote:
Just an update for everyone here:
1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info
rather
than egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much more efficient release system than pip, I anticipate having a release of setuptools first and then we can switch pip over to requiring a newer setuptools via PEP 518.
I don't think pip actually has any use for the PEP 517 prepare_wheel_metadata hook right now though? Historically 'setup.py egg-info' was needed to kluge around unwanted behavior in 'setup.py install', but with a PEP 517 backend that's irrelevant because 'setup.py install' is never used. And in the future when pip has a real resolver, then prepare_wheel_metadata should allow some optimizations. But right now, prepare_wheel_metadata is completely useless AFAIK.
So why is 'setup.py dist_info' a blocker for things?
-n
-- Nathaniel J. Smith -- https://vorpus.org
-- Nathaniel J. Smith -- https://vorpus.org
On Mon, Sep 4, 2017 at 6:08 PM, xoviat <xoviat@gmail.com> wrote:
In any case, we're going to need this for prepare_metadata, so the question you should ask is: what are the reasons for *not* merging this? I haven't heard any so far but that doesn't mean that they don't exist. If there are none, then I don't see why we cannot merge my wheel PR and do a release.
FYI, you're not the only one waiting for PR's to be merged. There are 56 other PR's, some of which have already been approved by repo members with commit access. I would try to be a little more patient. Otherwise, everyone can be emailing the list saying the same thing and asking why their PR isn't being merged. --Chris
2017-09-04 19:51 GMT-05:00 xoviat <xoviat@gmail.com>:
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook.
Yes. That is absolutely correct.
But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything.
The simplest way to start on this issue is to replace egg_info in pip with prepare_metadata_for_build_wheel. It is absolutely a historical artifact but I need to work on one issue at a time. The next issue will be replacing egg_info in pip with prepare_metadata_for_build_wheel.
2017-09-04 19:34 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xoviat@gmail.com> wrote:
Nathaniel:
Pip requires egg_info to discover dependencies of source distributions so that it can build wheels all at once after downloading the requirements. I need to move pip off of egg_info as soon as possible and dist_info is required to do that.
"Requires" is a strong word -- AFAIK this is just a historical artifact. I don't really know what you're talking about in the second sentence.
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook. But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything. (In fact I can't see any reason why pip would ever call it before the resolver lands.) So either (a) there's some other reason you want a dist_info command, (b) there's some reason I'm missing why prepare_wheel_metadata matters, or (c) one of us is misunderstanding something :-).
-n
2017-09-03 21:00 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote:
Just an update for everyone here:
1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info rather than egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much more efficient release system than pip, I anticipate having a release of setuptools first and then we can switch pip over to requiring a newer setuptools via PEP 518.
I don't think pip actually has any use for the PEP 517 prepare_wheel_metadata hook right now though? Historically 'setup.py egg-info' was needed to kluge around unwanted behavior in 'setup.py install', but with a PEP 517 backend that's irrelevant because 'setup.py install' is never used. And in the future when pip has a real resolver, then prepare_wheel_metadata should allow some optimizations. But right now, prepare_wheel_metadata is completely useless AFAIK.
So why is 'setup.py dist_info' a blocker for things?
-n
-- Nathaniel J. Smith -- https://vorpus.org
-- Nathaniel J. Smith -- https://vorpus.org
The PR that I am taking about is not for pip but for the wheel project. On Sep 4, 2017 8:19 PM, "Chris Jerdonek" <chris.jerdonek@gmail.com> wrote:
On Mon, Sep 4, 2017 at 6:08 PM, xoviat <xoviat@gmail.com> wrote:
In any case, we're going to need this for prepare_metadata, so the question you should ask is: what are the reasons for *not* merging this? I haven't heard any so far but that doesn't mean that they don't exist. If there are none, then I don't see why we cannot merge my wheel PR and do a release.
FYI, you're not the only one waiting for PR's to be merged. There are 56 other PR's, some of which have already been approved by repo members with commit access. I would try to be a little more patient. Otherwise, everyone can be emailing the list saying the same thing and asking why their PR isn't being merged.
--Chris
2017-09-04 19:51 GMT-05:00 xoviat <xoviat@gmail.com>:
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook.
Yes. That is absolutely correct.
But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything.
The simplest way to start on this issue is to replace egg_info in pip
with
prepare_metadata_for_build_wheel. It is absolutely a historical artifact but I need to work on one issue at a time. The next issue will be replacing egg_info in pip with prepare_metadata_for_build_wheel.
2017-09-04 19:34 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xoviat@gmail.com> wrote:
Nathaniel:
Pip requires egg_info to discover dependencies of source
distributions
so that it can build wheels all at once after downloading the requirements. I need to move pip off of egg_info as soon as possible and dist_info is required to do that.
"Requires" is a strong word -- AFAIK this is just a historical artifact. I don't really know what you're talking about in the second sentence.
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook. But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything. (In fact I can't see any reason why pip would ever call it before the resolver lands.) So either (a) there's some other reason you want a dist_info command, (b) there's some reason I'm missing why prepare_wheel_metadata matters, or (c) one of us is misunderstanding something :-).
-n
2017-09-03 21:00 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote: > Just an update for everyone here: > > 1. We're currently waiting on the implementation of the
'dist_info"
> command > in the wheel project. > 2. Once that is done we can switch pip over to reading dist-info > rather > than > egg_info. > 3. Then we can move the backend over to setuptools. Because Jacob > has a > much > more efficient release system than pip, I anticipate having a > release of > setuptools first and then we can switch pip over to requiring a > newer > setuptools via PEP 518.
I don't think pip actually has any use for the PEP 517 prepare_wheel_metadata hook right now though? Historically 'setup.py egg-info' was needed to kluge around unwanted behavior in 'setup.py install', but with a PEP 517 backend that's irrelevant because 'setup.py install' is never used. And in the future when pip has a real resolver, then prepare_wheel_metadata should allow some optimizations. But right now, prepare_wheel_metadata is completely useless AFAIK.
So why is 'setup.py dist_info' a blocker for things?
-n
-- Nathaniel J. Smith -- https://vorpus.org
-- Nathaniel J. Smith -- https://vorpus.org
On Mon, Sep 4, 2017 at 6:41 PM, xoviat <xoviat@gmail.com> wrote:
The PR that I am taking about is not for pip but for the wheel project.
Okay, well you started the thread asking something similar for your pip PR (see below). I'm sure similar considerations hold for the wheel project. On Mon, Sep 4, 2017 at 5:11 PM, xoviat <xoviat@gmail.com> wrote:
Also if someone with pip write access could please discuss and hopefully merge my initial PR on pip, I would very much appreciate it. Paul seems to be short on time.
--Chris
On Sep 4, 2017 8:19 PM, "Chris Jerdonek" <chris.jerdonek@gmail.com> wrote:
On Mon, Sep 4, 2017 at 6:08 PM, xoviat <xoviat@gmail.com> wrote:
In any case, we're going to need this for prepare_metadata, so the question you should ask is: what are the reasons for *not* merging this? I haven't heard any so far but that doesn't mean that they don't exist. If there are none, then I don't see why we cannot merge my wheel PR and do a release.
FYI, you're not the only one waiting for PR's to be merged. There are 56 other PR's, some of which have already been approved by repo members with commit access. I would try to be a little more patient. Otherwise, everyone can be emailing the list saying the same thing and asking why their PR isn't being merged.
--Chris
2017-09-04 19:51 GMT-05:00 xoviat <xoviat@gmail.com>:
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook.
Yes. That is absolutely correct.
But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything.
The simplest way to start on this issue is to replace egg_info in pip with prepare_metadata_for_build_wheel. It is absolutely a historical artifact but I need to work on one issue at a time. The next issue will be replacing egg_info in pip with prepare_metadata_for_build_wheel.
2017-09-04 19:34 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xoviat@gmail.com> wrote:
Nathaniel:
Pip requires egg_info to discover dependencies of source distributions so that it can build wheels all at once after downloading the requirements. I need to move pip off of egg_info as soon as possible and dist_info is required to do that.
"Requires" is a strong word -- AFAIK this is just a historical artifact. I don't really know what you're talking about in the second sentence.
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook. But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything. (In fact I can't see any reason why pip would ever call it before the resolver lands.) So either (a) there's some other reason you want a dist_info command, (b) there's some reason I'm missing why prepare_wheel_metadata matters, or (c) one of us is misunderstanding something :-).
-n
2017-09-03 21:00 GMT-05:00 Nathaniel Smith <njs@pobox.com>: > > On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote: > > Just an update for everyone here: > > > > 1. We're currently waiting on the implementation of the > > 'dist_info" > > command > > in the wheel project. > > 2. Once that is done we can switch pip over to reading dist-info > > rather > > than > > egg_info. > > 3. Then we can move the backend over to setuptools. Because Jacob > > has a > > much > > more efficient release system than pip, I anticipate having a > > release of > > setuptools first and then we can switch pip over to requiring a > > newer > > setuptools via PEP 518. > > I don't think pip actually has any use for the PEP 517 > prepare_wheel_metadata hook right now though? Historically > 'setup.py > egg-info' was needed to kluge around unwanted behavior in 'setup.py > install', but with a PEP 517 backend that's irrelevant because > 'setup.py install' is never used. And in the future when pip has a > real resolver, then prepare_wheel_metadata should allow some > optimizations. But right now, prepare_wheel_metadata is completely > useless AFAIK. > > So why is 'setup.py dist_info' a blocker for things? > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org
-- Nathaniel J. Smith -- https://vorpus.org
Supposedly there is some meeting tomorrow concerning the wheel project that will determine the fate of dist_info. So that is why I bought it up. On Sep 4, 2017 9:00 PM, "Chris Jerdonek" <chris.jerdonek@gmail.com> wrote:
On Mon, Sep 4, 2017 at 6:41 PM, xoviat <xoviat@gmail.com> wrote:
The PR that I am taking about is not for pip but for the wheel project.
Okay, well you started the thread asking something similar for your pip PR (see below). I'm sure similar considerations hold for the wheel project.
On Mon, Sep 4, 2017 at 5:11 PM, xoviat <xoviat@gmail.com> wrote:
Also if someone with pip write access could please discuss and hopefully merge my initial PR on pip, I would very much appreciate it. Paul seems to be short on time.
--Chris
On Sep 4, 2017 8:19 PM, "Chris Jerdonek" <chris.jerdonek@gmail.com>
wrote:
On Mon, Sep 4, 2017 at 6:08 PM, xoviat <xoviat@gmail.com> wrote:
In any case, we're going to need this for prepare_metadata, so the question you should ask is: what are the reasons for *not* merging this? I haven't heard any so far but that doesn't mean that they don't exist. If there are none, then I don't see why we cannot merge my wheel PR and do a
release.
FYI, you're not the only one waiting for PR's to be merged. There are 56 other PR's, some of which have already been approved by repo members with commit access. I would try to be a little more patient. Otherwise, everyone can be emailing the list saying the same thing and asking why their PR isn't being merged.
--Chris
2017-09-04 19:51 GMT-05:00 xoviat <xoviat@gmail.com>:
The only reason I can think of that setuptools would need a
dist_info
command would be to implement the PEP 517 prepare_wheel_metadata hook.
Yes. That is absolutely correct.
But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything.
The simplest way to start on this issue is to replace egg_info in pip with prepare_metadata_for_build_wheel. It is absolutely a historical artifact but I need to work on one issue at a time. The next issue will be replacing egg_info in pip with prepare_metadata_for_build_wheel.
2017-09-04 19:34 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xoviat@gmail.com> wrote: > Nathaniel: > > Pip requires egg_info to discover dependencies of source > distributions > so > that it can build wheels all at once after downloading the > requirements. I > need to move pip off of egg_info as soon as possible and dist_info > is > required to do that.
"Requires" is a strong word -- AFAIK this is just a historical artifact. I don't really know what you're talking about in the
second
sentence.
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook. But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything. (In fact I can't see any reason why pip would ever call it before the resolver lands.) So either (a) there's some other reason you want a dist_info command, (b) there's some reason I'm missing why prepare_wheel_metadata matters, or (c) one of us is misunderstanding something :-).
-n
> 2017-09-03 21:00 GMT-05:00 Nathaniel Smith <njs@pobox.com>: >> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote: >> > Just an update for everyone here: >> > >> > 1. We're currently waiting on the implementation of the >> > 'dist_info" >> > command >> > in the wheel project. >> > 2. Once that is done we can switch pip over to reading dist-info >> > rather >> > than >> > egg_info. >> > 3. Then we can move the backend over to setuptools. Because Jacob >> > has a >> > much >> > more efficient release system than pip, I anticipate having a >> > release of >> > setuptools first and then we can switch pip over to requiring a >> > newer >> > setuptools via PEP 518. >> >> I don't think pip actually has any use for the PEP 517 >> prepare_wheel_metadata hook right now though? Historically >> 'setup.py >> egg-info' was needed to kluge around unwanted behavior in 'setup.py >> install', but with a PEP 517 backend that's irrelevant because >> 'setup.py install' is never used. And in the future when pip has a >> real resolver, then prepare_wheel_metadata should allow some >> optimizations. But right now, prepare_wheel_metadata is completely >> useless AFAIK. >> >> So why is 'setup.py dist_info' a blocker for things? >> >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org > >
-- Nathaniel J. Smith -- https://vorpus.org
On Mon, Sep 4, 2017 at 5:51 PM, xoviat <xoviat@gmail.com> wrote:
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook.
Yes. That is absolutely correct.
But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything.
The simplest way to start on this issue is to replace egg_info in pip with prepare_metadata_for_build_wheel. It is absolutely a historical artifact but I need to work on one issue at a time. The next issue will be replacing egg_info in pip with prepare_metadata_for_build_wheel.
This still doesn't make sense to me. To support PEP 517, pip HAS to support backends that don't provide prepare_metadata_for_build_wheel. You will have to handle this before PEP 517 support can ship. So what's your plan for after you replace egg_info with prepare_metadata_for_build_wheel? Turning around and deleting the code you just wrote? -n
2017-09-04 19:34 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Mon, Sep 4, 2017 at 5:09 PM, xoviat <xoviat@gmail.com> wrote:
Nathaniel:
Pip requires egg_info to discover dependencies of source distributions so that it can build wheels all at once after downloading the requirements. I need to move pip off of egg_info as soon as possible and dist_info is required to do that.
"Requires" is a strong word -- AFAIK this is just a historical artifact. I don't really know what you're talking about in the second sentence.
The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook. But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything. (In fact I can't see any reason why pip would ever call it before the resolver lands.) So either (a) there's some other reason you want a dist_info command, (b) there's some reason I'm missing why prepare_wheel_metadata matters, or (c) one of us is misunderstanding something :-).
-n
2017-09-03 21:00 GMT-05:00 Nathaniel Smith <njs@pobox.com>:
On Sun, Sep 3, 2017 at 11:14 AM, xoviat <xoviat@gmail.com> wrote:
Just an update for everyone here:
1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info rather than egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much more efficient release system than pip, I anticipate having a release of setuptools first and then we can switch pip over to requiring a newer setuptools via PEP 518.
I don't think pip actually has any use for the PEP 517 prepare_wheel_metadata hook right now though? Historically 'setup.py egg-info' was needed to kluge around unwanted behavior in 'setup.py install', but with a PEP 517 backend that's irrelevant because 'setup.py install' is never used. And in the future when pip has a real resolver, then prepare_wheel_metadata should allow some optimizations. But right now, prepare_wheel_metadata is completely useless AFAIK.
So why is 'setup.py dist_info' a blocker for things?
-n
-- Nathaniel J. Smith -- https://vorpus.org
-- Nathaniel J. Smith -- https://vorpus.org
-- Nathaniel J. Smith -- https://vorpus.org
On 3 September 2017 at 05:42, Donald Stufft <donald@stufft.io> wrote:
On Sep 1, 2017, at 2:30 PM, Chris Barker <Chris.Barker@noaa.gov> wrote:
Do the Linux distros use pip to build their packages?
Not that I am aware of.
Fedora's build macros for Python projects currently rely on running setup.py directly, but we've been considering switching to pip instead since 2013 or so. PEP 517 is likely to provide the impetus to switch from "maybe we should do that" to "we need to do that, at least if setup.py is missing, and potentially always, so we get more consistent installation metadata" Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (11)
-
Chris Barker -
Chris Barker - NOAA Federal
-
Chris Jerdonek -
cyanrave＠gmail.com -
Daniel Holth -
Donald Stufft -
Matthew Brett -
Nathaniel Smith -
Nick Coghlan -
Paul Moore -
xoviat