Using Wheel with zipimport
I noticed yesterday that Nick made a commit to PEP427 that Wheels were designed to be compatible with zipimport so that you could directly add them to sys.path (http://hg.python.org/peps/rev/811e34eda04c). I was not aware that this was the case, and had thought otherwise, and have been telling people that although Wheels could technically be used that way that doing so was considered unsupported. I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it. One of the major problems with Egg was that it confused the difference between a distribution format and a runtime installation format. I believe that if we have this as a supported option, that ultimately it will lead towards the same kind of problems that Egg had trying to fulfill two roles at once. Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules). If a single file importable item for a project is something that end users want, then an additional format should be developed that does not concern itself with the ability to unpack and install the project and instead solves the problems that come with trying to stick arbitrary things in a zipped file and import them. So yea, I think that the goals of an importable format and an distribution format are not completely overlapped, and if Wheel tries to be both, then both sides of that will suffer with a less useful and less suitable ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation). The key problems with eggs in relation to this were: - easy_install preferring to install as eggs by default - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically. All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the "Root-is-purelib" setting, rather than requiring separate purelib and platlib subdirectories). Cheers, Nick.
On Jan 28, 2014, at 6:38 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation).
The key problems with eggs in relation to this were: - easy_install preferring to install as eggs by default - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation
Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically.
All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the "Root-is-purelib" setting, rather than requiring separate purelib and platlib subdirectories).
Cheers, Nick.
Regardless if it was or wasn't an accident, I believe it was a mistake. Supporting it officially at all means that we have limitations on what we can do to make Wheel a better format. I had hopes that Wheel could be made more generic than it currently is, but because of the fact that directly adding them to sys.path is supported that makes it much much more awkward to do so. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft <donald@stufft.io> wrote:
On Jan 28, 2014, at 6:38 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation).
The key problems with eggs in relation to this were: - easy_install preferring to install as eggs by default - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation
Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically.
All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the "Root-is-purelib" setting, rather than requiring separate purelib and platlib subdirectories).
Cheers, Nick.
Regardless if it was or wasn't an accident, I believe it was a mistake. Supporting it officially at all means that we have limitations on what we can do to make Wheel a better format. I had hopes that Wheel could be made more generic than it currently is, but because of the fact that directly adding them to sys.path is supported that makes it much much more awkward to do so.
Hey, as long as they are zipfiles that don't totally scramble the layout of the internal Python code you can add them to sys.path. Did you know you can even add subpaths of a zipfile to sys.path? </mind blown> I'm opposed to making wheel generic as in "package PostgreSQL itself" generic. There are other ways to do that.
On 29 Jan 2014 11:19, "Daniel Holth" <dholth@gmail.com> wrote:
On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft <donald@stufft.io> wrote:
On Jan 28, 2014, at 6:38 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile
directly
on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation).
The key problems with eggs in relation to this were: - easy_install preferring to install as eggs by default - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation
Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically.
All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the "Root-is-purelib" setting, rather than requiring separate purelib and platlib subdirectories).
Cheers, Nick.
Regardless if it was or wasn't an accident, I believe it was a mistake. Supporting it officially at all means that we have limitations on what we can do to make Wheel a better format. I had hopes that Wheel could be made more generic than it currently is, but because of the fact that directly adding them to sys.path is supported that makes it much much more awkward to do so.
Hey, as long as they are zipfiles that don't totally scramble the layout of the internal Python code you can add them to sys.path. Did you know you can even add subpaths of a zipfile to sys.path? </mind blown>
Yep, the only requirement is "there will be a non empty subset of valid wheel files that can be used directly with zipimport". Wheels that include C extensions, or otherwise need to be unpacked to disk in order to work correctly aren't in that subset, and running directly from a wheel does prevent bytecode caching, but those are both OK - "supported when practical" isn't the same thing as "recommended". Cheers, Nick.
I'm opposed to making wheel generic as in "package PostgreSQL itself" generic. There are other ways to do that.
On Jan 28, 2014, at 11:54 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 29 Jan 2014 11:19, "Daniel Holth" <dholth@gmail.com> wrote:
On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft <donald@stufft.io> wrote:
On Jan 28, 2014, at 6:38 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implementation).
The key problems with eggs in relation to this were: - easy_install preferring to install as eggs by default - setuptools install a global site hook that added every installed egg to sys.path for every application run in that Python installation
Neither of those applies to wheels - pip always unpacks them when installing, and if you want to add one to sys.path you have to do it manually, it doesn't happen automatically.
All the new note in the PEP is clarifying is that it *isn't* an accident that the wheel format is zipimport compatible for pure Python wheels, we deliberately designed it that way (hence the "Root-is-purelib" setting, rather than requiring separate purelib and platlib subdirectories).
Cheers, Nick.
Regardless if it was or wasn't an accident, I believe it was a mistake. Supporting it officially at all means that we have limitations on what we can do to make Wheel a better format. I had hopes that Wheel could be made more generic than it currently is, but because of the fact that directly adding them to sys.path is supported that makes it much much more awkward to do so.
Hey, as long as they are zipfiles that don't totally scramble the layout of the internal Python code you can add them to sys.path. Did you know you can even add subpaths of a zipfile to sys.path? </mind blown>
Yep, the only requirement is "there will be a non empty subset of valid wheel files that can be used directly with zipimport".
Wheels that include C extensions, or otherwise need to be unpacked to disk in order to work correctly aren't in that subset, and running directly from a wheel does prevent bytecode caching, but those are both OK - "supported when practical" isn't the same thing as "recommended".
Cheers, Nick.
I'm opposed to making wheel generic as in "package PostgreSQL itself" generic. There are other ways to do that.
And yet on another read through of PEP427 the first item in “Comparison to Egg” is "Wheel is an installation format; egg is importable.”. The only other mention of importing any of them in that PEP is the section you just added saying that Wheels are designed to be importable. The original text of the PEP does not provide any evidence that Wheels were meant to be importable and instead makes it a point to call that a difference from Eggs. Further more there is a comment from Daniel on python-dev [1] which states that “Jim Fulton is right that weird failures are a characteristic of zipped eggs, so one of the #1 requests for setuptools is how to prohibit zipping from ever happening. This is an important reason why wheel is billed as an installation format -- fewer users with pitchforks. It's very cool that it works though. Debugging is slightly easier than it was in the old days because pdb can now read the source code from the zip.” Which further shows that at the time it was “cool that it worked” but that the “weird failures” being a reason that Wheel was an installation format. I believe that adding this “feature” to the PEP ex post facto is a bad idea. Had the PEP contained anything to indicate that Wheels were intended to be importable as part of the design philosophy I would have spoken out about it then instead it’s been added after it was accepted with no discussion that I can see. If I missed where this discussion happened during the PEP please direct me to it because I’ve just spent 15 minutes googling attempting to find any information on it, and all I’ve been able to find is Vinay’s experiment with Wheel.mount(), You mentioning the possibility of using Wheels in the sense of Multi Version Installs (which ends up talking about a Wheel inspired directory layout instead), and the thread I mentioned above. Not that it’s entirely relevant because really regardless of if it was discussed if it wasn’t encoded in the PEP than people following along by watching the PEPs had no ability to step in to speak for or against it. [1] https://groups.google.com/d/msg/dev-python/BS28mF7mb6c/o8jOo1NcousJ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 January 2014 05:52, Donald Stufft <donald@stufft.io> wrote:
Which further shows that at the time it was "cool that it worked" but that the "weird failures" being a reason that Wheel was an installation format.
Having "cool" features in a format is not a bad thing. Zip imports are the "cool " feature here - they have been used for lots of really impressive things (like py2exe, for a less controversial example). But they have limitations (which become "weird failures" when people try to use the feature without knowing or catering for the limitations) and it's the responsibility of the user to understand and address those limitations - or not use the feature if the limitations are too great for them.
I believe that adding this "feature" to the PEP ex post facto is a bad idea. Had the PEP contained anything to indicate that Wheels were intended to be importable as part of the design philosophy I would have spoken out about it then instead it's been added after it was accepted with no discussion that I can see.
As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the "wheel install" command. I do *not* recommend using wheels as a runtime format if the package in the wheel hits any of the limitations of zipimport (C extensions, data files that are not or cannot be accessed via pkgutil.package_data, etc). I also would not recommend using wheels except in *very* specialised circumstances (where you have the wheel already and cannot easily just install it before use - pip, ensurepip, virtualenv and similar distribution-oriented environments are the key use cases here). But I do think the feature is useful and should be supported. Paul.
On Jan 29, 2014, at 4:23 AM, Paul Moore <p.f.moore@gmail.com> wrote:
As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the "wheel install" command.
I just read every version of the PEP that has ever existed in Mercurial and no version besides Nick’s most recent contains any text about the importability of Wheels besides that one of the differences of Wheel and Egg is that Wheel is an installation format and Egg is importable. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Wed, Jan 29, 2014 at 3:41 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 29, 2014, at 4:23 AM, Paul Moore <p.f.moore@gmail.com> wrote:
As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the "wheel install" command.
I just read every version of the PEP that has ever existed in Mercurial and no version besides Nick's most recent contains any text about the importability of Wheels besides that one of the differences of Wheel and Egg is that Wheel is an installation format and Egg is importable.
FWIW, Nick clarified this aspect of wheels on this list at least once -- back in Sep 2012 when the PEP was first introduced. See this email: https://mail.python.org/pipermail/distutils-sig/2012-September/018959.html "It's also not quite true that the contents of a wheel aren't importable - since they're still just a zipfile, they can still be added to an import path in the same manner as egg files. It's just not *recommended* to use them that way, as the format is designed primarily for use in distribution rather than runtime imports." --Chris
That doesn't really speak to the fact they were designed for that. If I read that it looks like some commenting that they are importable (which as it stands they are) and not someone calling it a supported feature of the format. It even states that the format is designed primarily for distribution. Given that the PEPs text had always been contrary to the position that wheels were designed to be importable I don't think you can draw any other conclusion.
On Jan 29, 2014, at 6:52 AM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Wed, Jan 29, 2014 at 3:41 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 29, 2014, at 4:23 AM, Paul Moore <p.f.moore@gmail.com> wrote:
As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the "wheel install" command.
I just read every version of the PEP that has ever existed in Mercurial and no version besides Nick's most recent contains any text about the importability of Wheels besides that one of the differences of Wheel and Egg is that Wheel is an installation format and Egg is importable.
FWIW, Nick clarified this aspect of wheels on this list at least once -- back in Sep 2012 when the PEP was first introduced. See this email:
https://mail.python.org/pipermail/distutils-sig/2012-September/018959.html
"It's also not quite true that the contents of a wheel aren't importable - since they're still just a zipfile, they can still be added to an import path in the same manner as egg files. It's just not *recommended* to use them that way, as the format is designed primarily for use in distribution rather than runtime imports."
--Chris
On 29 January 2014 11:41, Donald Stufft <donald@stufft.io> wrote:
On Jan 29, 2014, at 4:23 AM, Paul Moore <p.f.moore@gmail.com> wrote:
As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the "wheel install" command.
I just read every version of the PEP that has ever existed in Mercurial and no version besides Nick's most recent contains any text about the importability of Wheels besides that one of the differences of Wheel and Egg is that Wheel is an installation format and Egg is importable.
Apologies. It was something Daniel pointed out a few times very early on - I hadn't realised it wasn't in the spec directly. What is in the spec - which effectively constrains the format to *allowing* (rather than encouraging) direct import - are the facts that wheels are zip format, and that one of purelib/platlib is at the root. The concept of separating "unpack" and "spread" and the comment "Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages" doesn't leave any room for wheels *not* to be importable in the majority of pure-python, no package data, cases. Debating how we present this in later versions of the wheel spec is fine. But deliberately making wheels not importable would break backward compatibility in a way that would have other, likely more serious, implications. Nevertheless, I understand your concerns, and I think we should be very careful not to let people get the impression that this is in any way similar to "importable eggs", which had a very bad press. Paul
On Jan 29, 2014, at 6:59 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 January 2014 11:41, Donald Stufft <donald@stufft.io> wrote:
On Jan 29, 2014, at 4:23 AM, Paul Moore <p.f.moore@gmail.com> wrote:
As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel world using the "wheel install" command.
I just read every version of the PEP that has ever existed in Mercurial and no version besides Nick's most recent contains any text about the importability of Wheels besides that one of the differences of Wheel and Egg is that Wheel is an installation format and Egg is importable.
Apologies. It was something Daniel pointed out a few times very early on - I hadn't realised it wasn't in the spec directly.
What is in the spec - which effectively constrains the format to *allowing* (rather than encouraging) direct import - are the facts that wheels are zip format, and that one of purelib/platlib is at the root. The concept of separating "unpack" and "spread" and the comment "Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages" doesn't leave any room for wheels *not* to be importable in the majority of pure-python, no package data, cases.
Debating how we present this in later versions of the wheel spec is fine. But deliberately making wheels not importable would break backward compatibility in a way that would have other, likely more serious, implications.
Nevertheless, I understand your concerns, and I think we should be very careful not to let people get the impression that this is in any way similar to "importable eggs", which had a very bad press.
Paul
I read that statement differently, in that it doesn’t guarantee that the files would be at the *root* of the Wheel, just that you could unpack a Wheel into a site-packages directory using unzip and have it be “installed”. I can see how it could be read otherwise though. In either case this is something that is more able to be removed in later versions of Wheel because unpacking by hand is something I don’t believe will ever be commonplace, and any installer that doesn’t check it’s Wheel version before doing things with the Wheel is wrong. But more importantly, while I’m against officially supporting importable Wheels because of various reasons, my biggest problem is that this was added in without any chance for discussion. I don’t think anyone can call me a casual observer of distutils-sig or the various PEP processes and this was the first time that I had heard of Wheels being importable as anything other than a sometimes useful hack that wasn’t a design goal but instead just an accident of implementation. I followed the PEP closely trying to make sure that we weren’t going to add a supported feature that locked us into any corners while the format was still new and this is something I would have fought against had it been in the original PEP. Nick may have, in his head, considered this to be a feature of Wheel all along and not an implementation detail, however that is not what the PEP stated. I don’t believe that as BDFL-Delegate for packaging issues Nick should be able to add supported features to an already accepted PEP without discussion especially when the existing text of that PEP is directly contrary to that feature being part of the PEP. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 January 2014 12:18, Donald Stufft <donald@stufft.io> wrote:
But more importantly, while I'm against officially supporting importable Wheels because of various reasons, my biggest problem is that this was added in without any chance for discussion.
Fair point. I'm sure Nick did just think of it as clarification - I know I did - but the process point is reasonable. I'm happy to leave off arguing with your position till the point comes for proper debate ;-) Paul
On Jan 29, 2014, at 7:43 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 January 2014 12:18, Donald Stufft <donald@stufft.io> wrote:
But more importantly, while I'm against officially supporting importable Wheels because of various reasons, my biggest problem is that this was added in without any chance for discussion.
Fair point. I'm sure Nick did just think of it as clarification - I know I did - but the process point is reasonable. I'm happy to leave off arguing with your position till the point comes for proper debate ;-)
Paul
Let me be explicit that I don’t think Nick is doing anything nefarious here, I do believe that he probably did think of it as a clarification. I’m concerned with the fact that the PEP did not support this when it was accepted and contained language that, to me at least, is contrary to the fact that this was supported, and then the new text was addd to an already accepted PEP giving it a new feature without any chance for discussion or disagreement. If I wasn’t anal about reading every email that hits my inbox *and* I wasn’t on python-checkins I never would have noticed this addition until it was likely too late to do anything about it because it had been documented as part of the PEP long enough that people started to depend on it. So hopefully nobody thinks I’m calling Nick a bad actor in this case. Also to be specific, what I believe should be done is that the change should be reverted, and if it is desired that Wheels officially support zip import then in the next version of the Wheel spec it should be added when everyone has a chance to properly discuss it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 January 2014 22:57, Donald Stufft <donald@stufft.io> wrote:
Also to be specific, what I believe should be done is that the change should be reverted, and if it is desired that Wheels officially support zip import then in the next version of the Wheel spec it should be added when everyone has a chance to properly discuss it.
Whether you like it or not, the wheel spec *does* already support being used that way, and changing it *would* be a backwards incompatible change to the format (and hence not acceptable without a compelling justification, and there isn't one that wouldn't equally well apply to other powerful but potentially confusing features like metaclasses and monkeypatching). I am also not willing to allow the perception that the wheel format does not offer a strict superset of the feature of the egg format to persist until we have a defined wheel 1.1 spec (which we don't even have a volunteer to work on yet). So, rather than removing it, I have instead updated the note in PEP 427 to read: ============================= Is it possible to import Python code directly from a wheel file? Yes, the wheel format is deliberately designed to be compatible with Python's support for importing from zip files, ensuring that it provides a superset of the functionality provided by the preceding egg format. However, this is generally not a *recommended* approach to using wheel files, as importing from a zip file rather than an ordinary filesystem directory imposes a number of additional constraints that will often break the assumptions of Python developers (for example, C extensions cannot be imported directly from a zip archive, and the ``__file__`` attribute no longer refers to an ordinary filesystem path, but to a combination path that includes both the location of the zip archive on the filesystem and the relative path to the module inside the archive). Like metaclasses and metapath importers, if you're not sure if you need to take advantage of this feature, you almost certainly don't need it. ============================= Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 29, 2014, at 8:14 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 29 January 2014 22:57, Donald Stufft <donald@stufft.io> wrote:
Also to be specific, what I believe should be done is that the change should be reverted, and if it is desired that Wheels officially support zip import then in the next version of the Wheel spec it should be added when everyone has a chance to properly discuss it.
Whether you like it or not, the wheel spec *does* already support being used that way, and changing it *would* be a backwards incompatible change to the format (and hence not acceptable without a compelling justification, and there isn't one that wouldn't equally well apply to other powerful but potentially confusing features like metaclasses and monkeypatching).
I am also not willing to allow the perception that the wheel format does not offer a strict superset of the feature of the egg format to persist until we have a defined wheel 1.1 spec (which we don't even have a volunteer to work on yet).
So, rather than removing it, I have instead updated the note in PEP 427 to read:
============================= Is it possible to import Python code directly from a wheel file? Yes, the wheel format is deliberately designed to be compatible with Python's support for importing from zip files, ensuring that it provides a superset of the functionality provided by the preceding egg format.
However, this is generally not a *recommended* approach to using wheel files, as importing from a zip file rather than an ordinary filesystem directory imposes a number of additional constraints that will often break the assumptions of Python developers (for example, C extensions cannot be imported directly from a zip archive, and the ``__file__`` attribute no longer refers to an ordinary filesystem path, but to a combination path that includes both the location of the zip archive on the filesystem and the relative path to the module inside the archive).
Like metaclasses and metapath importers, if you're not sure if you need to take advantage of this feature, you almost certainly don't need it. =============================
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
So basically even though the text of the PEP specifically points out that a difference of Wheel and Egg is that Eggs are importable it somehow supports that? Can you point to a single line in the PEP that supports this besides the ones you’ve added? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 January 2014 23:16, Donald Stufft <donald@stufft.io> wrote:
So basically even though the text of the PEP specifically points out that a difference of Wheel and Egg is that Eggs are importable it somehow supports that? Can you point to a single line in the PEP that supports this besides the ones you've added?
I added the clarification based on the facts that: 1. We discussed this extensively before PEP 427 was approved, and this was an accepted feature of the design 2. Root-is-purelib only makes sense in the context of supporting the feature 3. Both ensurepip and virtualenv rely on the feature 4. PEP 453 explicitly documents ensurepip's reliance on the feature, with no caveat about this being unsupported in the spec 5. I wouldn't have accepted PEP 427 if wheels didn't provide a strict superset of the features provided by eggs We make mistakes, and things that were discussed and agreed don't always get properly captured in the corresponding PEPs. When that happens, it's a judgement call as to whether properly documenting that is a new feature requiring a new PEP, or merely a clarification to the existing one. For standard library PEPs, we often don't do either - we just fix the implementation without going back for another round of PEP discussions (for smaller tweaks, sometimes we don't even go back to python-dev and instead just resolve things on the tracker). In this case, as BDFL-Delegate, I decided it was a case that merely called for clarification, because I *know* what spec I accepted, and it was the one where wheels offer all the features that eggs do and more. I added the new text specifically because people like Armin Ronacher and yourself had gained an idea from the PEP text that emphatically does *not* align with the design discussions that occurred prior to the acceptance of the PEP. Now, if you'd like to campaign to *remove* this support, then explain your rationale, and make the case for why you think providing the feature is so dangerous that removing it is worth breaking backwards compatibility over. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 29, 2014, at 8:31 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 29 January 2014 23:16, Donald Stufft <donald@stufft.io> wrote:
So basically even though the text of the PEP specifically points out that a difference of Wheel and Egg is that Eggs are importable it somehow supports that? Can you point to a single line in the PEP that supports this besides the ones you've added?
I added the clarification based on the facts that:
1. We discussed this extensively before PEP 427 was approved, and this was an accepted feature of the design 2. Root-is-purelib only makes sense in the context of supporting the feature 3. Both ensurepip and virtualenv rely on the feature 4. PEP 453 explicitly documents ensurepip's reliance on the feature, with no caveat about this being unsupported in the spec 5. I wouldn't have accepted PEP 427 if wheels didn't provide a strict superset of the features provided by eggs
We make mistakes, and things that were discussed and agreed don't always get properly captured in the corresponding PEPs.
When that happens, it's a judgement call as to whether properly documenting that is a new feature requiring a new PEP, or merely a clarification to the existing one. For standard library PEPs, we often don't do either - we just fix the implementation without going back for another round of PEP discussions (for smaller tweaks, sometimes we don't even go back to python-dev and instead just resolve things on the tracker).
In this case, as BDFL-Delegate, I decided it was a case that merely called for clarification, because I *know* what spec I accepted, and it was the one where wheels offer all the features that eggs do and more. I added the new text specifically because people like Armin Ronacher and yourself had gained an idea from the PEP text that emphatically does *not* align with the design discussions that occurred prior to the acceptance of the PEP.
Now, if you'd like to campaign to *remove* this support, then explain your rationale, and make the case for why you think providing the feature is so dangerous that removing it is worth breaking backwards compatibility over.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
This was supposedly extensively discussed prior to PEP427 being accepted yet I have no memory of this being discussed, am unable to find any discussion of it (other than one offs saying it’s possible but not a core feature), and you’ve been apparently unwilling to point to any discussion. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it.
I know that people have had bad experiences in the past with eggs, for the reasons Nick outlined in his response. However, you don't say *with specifics* why you think putting wheels on sys.path is a bad idea - I don't think it's convincing enough just to hand-wavingly reference problems with the egg format. When this topic came up before, I asked for specific failure modes which were causing concern, but I never got a response IIRC. One can't say that having the same packaging and importable formats is inherently bad, since Java shows otherwise. So if there are specific problems, they should be identified.
Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules).
I don't yet see a technical impediment here. For example, distlib's wheel implementation uses a "mount" method to add a wheel to sys.path. This uses the tags to check for compatibility - there's no "figuring out" that the user has to do. If additional metadata about C extensions is available (which it is, if the wheel is built from source using distil), then those extensions are made available for import, too. I realise that's an extension to the current spec, but then no one is forcing any one to use it. Regards, Vinay Sajip
On Jan 28, 2014, at 11:14 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Donald Stufft <donald <at> stufft.io> writes:
I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it.
I know that people have had bad experiences in the past with eggs, for the reasons Nick outlined in his response. However, you don't say *with specifics* why you think putting wheels on sys.path is a bad idea - I don't think it's convincing enough just to hand-wavingly reference problems with the egg format.
When this topic came up before, I asked for specific failure modes which were causing concern, but I never got a response IIRC.
One can't say that having the same packaging and importable formats is inherently bad, since Java shows otherwise. So if there are specific problems, they should be identified.
Last time this topic came up Jim Fulton did point things out. 1. That unpacking into the home directory is problematic because users that run services often don’t have home directories. This you waved away by saying that the home directory isn’t a required place, to which Jim responded that unpacking them anywhere else was likely to have access control / security issues and “generally cause pain”. Which I wholly agree with, intact Mitre has assigned CVE’s to applications that did switch their Egg Cache to directories other than HOME. 2. Zipped Eggs are more difficult to work with for debugging purposes, which you responded saying that Wheels are a “deployment format” (which I don’t believe is supported by the PEP at all) and Jim responded with his experience in developing against things that got installed as zipped Eggs and the pain that has personally caused him. 3. Zipped imports (In Jim’s experience) are slower than unpacked imports, which you replied (quite literally) "Caveat emptor”. Jim’s response was that this had been tried with Egg and had been found to be more pain than it was worth. In his exact words: "It's been tried with eggs. This is not new ground. Encouraging people to do this is going to cause pain and resentment. I think one of the reasons there's so much (IMO mostly irrational) hate for eggs is that people think you can only used zipped eggs, and zipped eggs cause pain and agony.” He closes up with, "Importing from zipped eggs has proved itself to be an anti pattern. Buildout (as of buildout 2) always unzips eggs.” So you have pip that doesn’t use zipped Eggs, buildout which made sure in buildout 2 (presumably drawing from the lessons of buildout 1) to always unzip eggs, and easy_install which does support zipped eggs (sometimes, if it thinks that particular Egg will support it). Why do we insist upon repeating the mistakes of the past?
Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules).
I don't yet see a technical impediment here. For example, distlib's wheel implementation uses a "mount" method to add a wheel to sys.path. This uses the tags to check for compatibility - there's no "figuring out" that the user has to do. If additional metadata about C extensions is available (which it is, if the wheel is built from source using distil), then those extensions are made available for import, too. I realise that's an extension to the current spec, but then no one is forcing any one to use it.
Zipped Eggs have a long long history of causing very weird failures conditions, you say your Wheel.mount checks for compatibility, does it also check that the library to be installed isn’t using __file__ shenanigans instead of pkgutil.get_data()? easy_install tried to do this with varying degrees of success, I’ve never seen much except pain from zipped Eggs.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On the one hand, it's easy to see the allure of a zipimport-able format. Grab a file, import it, use functionality straight away. It has an attraction of self-containedness. On the other hand, python's dynamic nature means that things are not as simple as Java jar's as Donald points out. I agree that wheels shouldn't officially support this feature wholesale. However, perhaps they could do so conditionally? E.g. designate some mechanism by which it is possible to inspect a wheel and determine that it intends to explicitly be zipimport compatible. A prime candidate would be that the wheel is of a purelib in nature, another would be to add a top-level file or metadata indicating such (e.g. '.zipimportable'). But other wise I agree that we should explicitly seek to avoid any language that could lead users to expect this functionality of wheels wholesale in any way, even though they happen to be regular zip files. On 29 Jan 2014 17:09, "Donald Stufft" <donald@stufft.io> wrote:
On Jan 28, 2014, at 11:14 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Donald Stufft <donald <at> stufft.io> writes:
I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it.
I know that people have had bad experiences in the past with eggs, for the reasons Nick outlined in his response. However, you don't say *with specifics* why you think putting wheels on sys.path is a bad idea - I don't think it's convincing enough just to hand-wavingly reference problems with the egg format.
When this topic came up before, I asked for specific failure modes which were causing concern, but I never got a response IIRC.
One can't say that having the same packaging and importable formats is inherently bad, since Java shows otherwise. So if there are specific problems, they should be identified.
Last time this topic came up Jim Fulton did point things out.
1. That unpacking into the home directory is problematic because users that run services often don't have home directories. This you waved away by saying that the home directory isn't a required place, to which Jim responded that unpacking them anywhere else was likely to have access control / security issues and "generally cause pain". Which I wholly agree with, intact Mitre has assigned CVE's to applications that did switch their Egg Cache to directories other than HOME.
2. Zipped Eggs are more difficult to work with for debugging purposes, which you responded saying that Wheels are a "deployment format" (which I don't believe is supported by the PEP at all) and Jim responded with his experience in developing against things that got installed as zipped Eggs and the pain that has personally caused him.
3. Zipped imports (In Jim's experience) are slower than unpacked imports, which you replied (quite literally) "Caveat emptor". Jim's response was that this had been tried with Egg and had been found to be more pain than it was worth. In his exact words:
"It's been tried with eggs. This is not new ground. Encouraging people to do this is going to cause pain and resentment.
I think one of the reasons there's so much (IMO mostly irrational) hate for eggs is that people think you can only used zipped eggs, and zipped eggs cause pain and agony."
He closes up with,
"Importing from zipped eggs has proved itself to be an anti pattern.
Buildout (as of buildout 2) always unzips eggs."
So you have pip that doesn't use zipped Eggs, buildout which made sure in buildout 2 (presumably drawing from the lessons of buildout 1) to always unzip eggs, and easy_install which does support zipped eggs (sometimes, if it thinks that particular Egg will support it).
Why do we insist upon repeating the mistakes of the past?
Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules).
I don't yet see a technical impediment here. For example, distlib's wheel implementation uses a "mount" method to add a wheel to sys.path. This
uses
the tags to check for compatibility - there's no "figuring out" that the user has to do. If additional metadata about C extensions is available (which it is, if the wheel is built from source using distil), then those extensions are made available for import, too. I realise that's an extension to the current spec, but then no one is forcing any one to use it.
Zipped Eggs have a long long history of causing very weird failures conditions, you say your Wheel.mount checks for compatibility, does it also check that the library to be installed isn't using __file__ shenanigans instead of pkgutil.get_data()? easy_install tried to do this with varying degrees of success, I've never seen much except pain from zipped Eggs.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
All of those arguments are against *recommending* directly importing from wheels. Yes, there are lots of problems with running directly from a zip archive, which is why the fact that easy_install *actively encourages* potentially inexperienced users to run things that way is so problematic. However, for people that are comfortable with the import system and the limitations of direct zip imports, it's a very useful feature. I wouldn't have accepted PEP 427 without the zipimport compatibility that meant a developer *could* use it as a direct replacement for eggs if they really wanted to. Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format's capabilities). I clarified PEP 427 specifically because Armin Ronacher's wheel article showed that he was unaware of this *deliberately included* feature of the wheel format. People are free not to like it - the default tools deliberately make it less convenient to run things that way, and that's as it should be. However, it's not a super-secret capability only to be used by us to implement things like ensurepip - it's a defined capability of the format that if your software is capable of running correctly from a zip archive in the first place, then that archive can be also be a valid wheel file. Running directly from a wheel is a power tool - that's a reason to put "handle with care" warnings on it, not to refuse to support a feature that was deliberately designed into the format. You can do a lot more damage with a badly written meta-importer, yet we have no intention of deprecating that capability either. Even *.pth files have turned out to have a valid use case for sharing packages between virtual environments. We have lots of features like that elsewhere in Python - when people ask about metaclasses, the first reaction is "You probably don't want to use them". However, sometimes developers *do* need them, and that's why the feature exists. Most of the time developers won't want to make use of the zipimport compatibility of wheel files, either, but advanced use cases like ensurepip are exactly why the capability exists. I can make the new note in the PEP more explicit that while this is a supported use case that ensures the feature set provided by wheels is a strict superset of that provided by eggs, that's not the same thing as *recommending* that wheels be used that way. Cheers, Nick.
On Jan 29, 2014, at 2:32 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
All of those arguments are against *recommending* directly importing from wheels. Yes, there are lots of problems with running directly from a zip archive, which is why the fact that easy_install *actively encourages* potentially inexperienced users to run things that way is so problematic.
However, for people that are comfortable with the import system and the limitations of direct zip imports, it's a very useful feature. I wouldn't have accepted PEP 427 without the zipimport compatibility that meant a developer *could* use it as a direct replacement for eggs if they really wanted to.
Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format's capabilities).
I clarified PEP 427 specifically because Armin Ronacher's wheel article showed that he was unaware of this *deliberately included* feature of the wheel format. People are free not to like it - the default tools deliberately make it less convenient to run things that way, and that's as it should be. However, it's not a super-secret capability only to be used by us to implement things like ensurepip - it's a defined capability of the format that if your software is capable of running correctly from a zip archive in the first place, then that archive can be also be a valid wheel file.
Running directly from a wheel is a power tool - that's a reason to put "handle with care" warnings on it, not to refuse to support a feature that was deliberately designed into the format. You can do a lot more damage with a badly written meta-importer, yet we have no intention of deprecating that capability either. Even *.pth files have turned out to have a valid use case for sharing packages between virtual environments.
We have lots of features like that elsewhere in Python - when people ask about metaclasses, the first reaction is "You probably don't want to use them". However, sometimes developers *do* need them, and that's why the feature exists. Most of the time developers won't want to make use of the zipimport compatibility of wheel files, either, but advanced use cases like ensurepip are exactly why the capability exists.
I can make the new note in the PEP more explicit that while this is a supported use case that ensures the feature set provided by wheels is a strict superset of that provided by eggs, that's not the same thing as *recommending* that wheels be used that way.
Cheers, Nick.
I don’t think it particularly matters wether you would have accepted the PEP without that ability or not. You *did* accept the PEP when the text of the PEP directly pointed out that one of the differences of Wheel and Egg were that “Wheel is an installation format, Egg is importable” which points to that fact that Wheel was not designed to be importable. As far as I can tell you’ve added this “feature” to the PEP by fiat without any chance for anyone to be able to discuss it after it’s already been accepted. I’m well aware that we cannot prevent Wheels from being imported, but that’s another thing all together from directly supporting it. For one thing by supporting it we forever lock ourselves into dumping what would be in site-packages directly in the root of the Wheel because adding the .whl file directly to sys.path is now a documented feature. This behavior has personally caused me annoyance and one of the things I was hoping on doing in a Wheel 1.2 was revise the layout a bit to make installation simpler and make inspecting a Wheel file easier. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jan 29, 2014, at 6:47 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 29, 2014, at 2:32 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
All of those arguments are against *recommending* directly importing from wheels. Yes, there are lots of problems with running directly from a zip archive, which is why the fact that easy_install *actively encourages* potentially inexperienced users to run things that way is so problematic.
However, for people that are comfortable with the import system and the limitations of direct zip imports, it's a very useful feature. I wouldn't have accepted PEP 427 without the zipimport compatibility that meant a developer *could* use it as a direct replacement for eggs if they really wanted to.
Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format's capabilities).
I clarified PEP 427 specifically because Armin Ronacher's wheel article showed that he was unaware of this *deliberately included* feature of the wheel format. People are free not to like it - the default tools deliberately make it less convenient to run things that way, and that's as it should be. However, it's not a super-secret capability only to be used by us to implement things like ensurepip - it's a defined capability of the format that if your software is capable of running correctly from a zip archive in the first place, then that archive can be also be a valid wheel file.
Running directly from a wheel is a power tool - that's a reason to put "handle with care" warnings on it, not to refuse to support a feature that was deliberately designed into the format. You can do a lot more damage with a badly written meta-importer, yet we have no intention of deprecating that capability either. Even *.pth files have turned out to have a valid use case for sharing packages between virtual environments.
We have lots of features like that elsewhere in Python - when people ask about metaclasses, the first reaction is "You probably don't want to use them". However, sometimes developers *do* need them, and that's why the feature exists. Most of the time developers won't want to make use of the zipimport compatibility of wheel files, either, but advanced use cases like ensurepip are exactly why the capability exists.
I can make the new note in the PEP more explicit that while this is a supported use case that ensures the feature set provided by wheels is a strict superset of that provided by eggs, that's not the same thing as *recommending* that wheels be used that way.
Cheers, Nick.
I don’t think it particularly matters wether you would have accepted the PEP without that ability or not. You *did* accept the PEP when the text of the PEP directly pointed out that one of the differences of Wheel and Egg were that “Wheel is an installation format, Egg is importable” which points to that fact that Wheel was not designed to be importable. As far as I can tell you’ve added this “feature” to the PEP by fiat without any chance for anyone to be able to discuss it after it’s already been accepted.
I’m well aware that we cannot prevent Wheels from being imported, but that’s another thing all together from directly supporting it. For one thing by supporting it we forever lock ourselves into dumping what would be in site-packages directly in the root of the Wheel because adding the .whl file directly to sys.path is now a documented feature. This behavior has personally caused me annoyance and one of the things I was hoping on doing in a Wheel 1.2 was revise the layout a bit to make installation simpler and make inspecting a Wheel file easier.
As a follow up, even if this becomes something we document and actually support, then it should be done in the next version of Wheel when people have the chance to discuss it. It should not be added ex post facto by a committer to the PEP repo just because they want Wheel to have it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 January 2014 21:50, Donald Stufft <donald@stufft.io> wrote:
As a follow up, even if this becomes something we document and actually support, then it should be done in the next version of Wheel when people have the chance to discuss it. It should not be added ex post facto by a committer to the PEP repo just because they want Wheel to have it.
It was discussed during the approval process for the wheel format, I approved it on the basis that wheels provided a superset of the functionality of eggs. I merely missed the fact it wasn't properly recorded in the PEP (which is now rectified). You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 29, 2014, at 7:58 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 29 January 2014 21:50, Donald Stufft <donald@stufft.io> wrote:
As a follow up, even if this becomes something we document and actually support, then it should be done in the next version of Wheel when people have the chance to discuss it. It should not be added ex post facto by a committer to the PEP repo just because they want Wheel to have it.
It was discussed during the approval process for the wheel format, I approved it on the basis that wheels provided a superset of the functionality of eggs. I merely missed the fact it wasn't properly recorded in the PEP (which is now rectified).
You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Can you point to this discussion? Because I cannot find it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 January 2014 12:58, Nick Coghlan <ncoghlan@gmail.com> wrote:
You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it.
Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a "special exception" here, nor do I want it to rely on accidents of the implementation. So from my POV, pypa is already shipping code that relies on this behaviour being by design. Paul
On 29 January 2014 23:10, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 January 2014 12:58, Nick Coghlan <ncoghlan@gmail.com> wrote:
You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it.
Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a "special exception" here, nor do I want it to rely on accidents of the implementation.
So from my POV, pypa is already shipping code that relies on this behaviour being by design.
Not just PyPA - this feature is at the heart of how the ensurepip module in the standard library works. This approach is explicitly documented in PEP 453: http://www.python.org/dev/peps/pep-0453/#implementation-strategy Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 29, 2014, at 8:17 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 29 January 2014 23:10, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 January 2014 12:58, Nick Coghlan <ncoghlan@gmail.com> wrote:
You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it.
Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a "special exception" here, nor do I want it to rely on accidents of the implementation.
So from my POV, pypa is already shipping code that relies on this behaviour being by design.
Not just PyPA - this feature is at the heart of how the ensurepip module in the standard library works.
This approach is explicitly documented in PEP 453: http://www.python.org/dev/peps/pep-0453/#implementation-strategy
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
There is a very important distinction. Both of those are designed to work with very specific Wheels and relied on, until this change, private, undocumented, and undiscussed features of Wheel. It’s not unusual for the Python standard library to make use of internal APIs to make it’s features work nor is it unusual for Virtualenv to do that either because of the nature of it bolting things on to Python that Python wasn’t designed to do. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jan 29, 2014, at 8:10 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 January 2014 12:58, Nick Coghlan <ncoghlan@gmail.com> wrote:
You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it.
Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a "special exception" here, nor do I want it to rely on accidents of the implementation.
So from my POV, pypa is already shipping code that relies on this behaviour being by design.
Paul
Here’s Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 January 2014 23:31, Donald Stufft <donald@stufft.io> wrote:
Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature.
https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html
Donald, I was *there*. I know what we discussed, and I know what PEP I approved. Trying to play gotcha with links *isn't going to work*. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 29, 2014, at 8:32 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 29 January 2014 23:31, Donald Stufft <donald@stufft.io> wrote:
Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature.
https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html
Donald, I was *there*. I know what we discussed, and I know what PEP I approved. Trying to play gotcha with links *isn't going to work*.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
If I’m wrong, then by all means show me where it was discussed so I can admit I was wrong. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jan 29, 2014, at 8:34 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 29, 2014, at 8:32 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 29 January 2014 23:31, Donald Stufft <donald@stufft.io> wrote:
Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature.
https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html
Donald, I was *there*. I know what we discussed, and I know what PEP I approved. Trying to play gotcha with links *isn't going to work*.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
If I’m wrong, then by all means show me where it was discussed so I can admit I was wrong.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Ironically, in another thread [1] on distutils-sig we have Daniel Holth, the author of the PEP stating: The main reason the packaging format is not explicitly importable is simply because Python hasn't had "jar-like" deployment for as long or as consistently as Java. So while Java code universally uses a "get resource" API to get stuff on the classpath, a a lot of Python code will try to open a file. C extensions can't be loaded from inside zip files. And it's a lot harder to edit .py files once they are zipped up, unlike Java where only the compiled and non-editable .class files are zipped. ZIP import is a great feature but wheels aren't really designed for it. It's more reliable that way. [1] https://mail.python.org/pipermail/distutils-sig/2014-January/023554.html ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 January 2014 23:34, Donald Stufft <donald@stufft.io> wrote:
If I'm wrong, then by all means show me where it was discussed so I can admit I was wrong.
You have the burden of proof backwards there. You're the one asking me to break backwards compatibility, and to let people continue to believe wheels are only a partial replacement for eggs - it's up to you to make the case that waiting until wheel 1.1 (which, again, has nobody committed to writing it and a completely unspecified timeline) is a superior approach to clarifying something I thought was already documented when I accepted the PEP (and is certainly inherent in the design of the format). The zipimport compatibility didn't need to be discussed much because it was there in Daniel's original wheel design - there was never any proposal to use a zipimport *incompatible* approach, so nobody had to campaign in favour of zipimport compatibility. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 29, 2014, at 8:46 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 29 January 2014 23:34, Donald Stufft <donald@stufft.io> wrote:
If I'm wrong, then by all means show me where it was discussed so I can admit I was wrong.
You have the burden of proof backwards there. You're the one asking me to break backwards compatibility, and to let people continue to believe wheels are only a partial replacement for eggs - it's up to you to make the case that waiting until wheel 1.1 (which, again, has nobody committed to writing it and a completely unspecified timeline) is a superior approach to clarifying something I thought was already documented when I accepted the PEP (and is certainly inherent in the design of the format).
The zipimport compatibility didn't need to be discussed much because it was there in Daniel's original wheel design - there was never any proposal to use a zipimport *incompatible* approach, so nobody had to campaign in favour of zipimport compatibility.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
So what did you mean when you said “We discussed it extensively before PEP 427 was approved” if you’re now saying that it wasn’t discussed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 January 2014 23:48, Donald Stufft <donald@stufft.io> wrote:
So what did you mean when you said "We discussed it extensively before PEP 427 was approved" if you're now saying that it wasn't discussed.
"Explicitly" would be a better word: https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html Like I said, that particular aspect wasn't controversial, so while it was noted a few times (a few other examples of which you found), it was the overall discussions that were extensive. Both Daniel and I knew the zipimport compatibility for packages that were themselves zip compatible was a deliberate feature, so it was a surprise to me when Armin Ronacher said in his recent article that it wasn't supported (and hence the clarification). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 29, 2014, at 8:59 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 29 January 2014 23:48, Donald Stufft <donald@stufft.io> wrote:
So what did you mean when you said "We discussed it extensively before PEP 427 was approved" if you're now saying that it wasn't discussed.
"Explicitly" would be a better word: https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html
Like I said, that particular aspect wasn't controversial, so while it was noted a few times (a few other examples of which you found), it was the overall discussions that were extensive. Both Daniel and I knew the zipimport compatibility for packages that were themselves zip compatible was a deliberate feature, so it was a surprise to me when Armin Ronacher said in his recent article that it wasn't supported (and hence the clarification).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
That also mentions the fact that he was considering allowing other compression algorithms which would make it incompatible with zip import. I don’t agree that this points to any promise that Wheels are themselves importable as I can guarantee I poured over that spec over and over making sure we weren’t adding something from Egg that had been problematic in the past. I have no plans to support adding Wheels to sys.path in a way that is supported for end users to do that, in any tool or project I work on. But whatever, I feel like you’ve placed yourself above the PEP process and are adding features by fiat instead of through the process. Obviously you feel different and this discussion has gone toxic. So carry on. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 30/01/14 00:59, Nick Coghlan wrote:
On 29 January 2014 23:48, Donald Stufft <donald@stufft.io> wrote:
So what did you mean when you said "We discussed it extensively before PEP 427 was approved" if you're now saying that it wasn't discussed. "Explicitly" would be a better word: https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html
Like I said, that particular aspect wasn't controversial, so while it was noted a few times (a few other examples of which you found), it was the overall discussions that were extensive. Both Daniel and I knew the zipimport compatibility for packages that were themselves zip compatible was a deliberate feature, so it was a surprise to me when Armin Ronacher said in his recent article that it wasn't supported (and hence the clarification).
Cheers, Nick.
In My Humble Opinion, that just isn't good enough. I am sure that everyone on this list has a passionate interest in making sure that the decisions for "python packaging" are as right as possible, and we know there are processes to follow that help us achieve that. One of those is that one should always be explicit about what features and functionality anything supports, which means that all features and functionality can be discussed, agreed upon, improved, etc, before it becomes something that everyone wilfully agrees to support. Wheels are either *documented* to be a binary format that *officially supports* zipimport-ability (fully, conditionally, or whatever), or the default answer is, like any other functionality not explicitly mentioned, no they are not. That's how you get a standard where issues can be discussed without tension. Two people agreeing to themselves that this is an uncontroversial implicit feature is not part of the documentation process, and whether or not they are the creators and acceptors of such standards, it leaves others out of the process until this late stage of the game. The unfortunate disagreements are exactly what can happen when things aren't made explicit from the start and are what are preventable when they are. "Docs or it doesn't exist" is popular developer mantra for a reason. Even as mostly an kibitzer on this group I hope it can learn from and avoid this sort of discussion into the future. -- Matt Iversen PGP: 0xc046e8a874522973 // 2F04 3DCC D6E6 D5AC D262 2E0B C046 E8A8 7452 2973
On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 29, 2014, at 8:10 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 January 2014 12:58, Nick Coghlan <ncoghlan@gmail.com> wrote:
You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it.
Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a "special exception" here, nor do I want it to rely on accidents of the implementation.
So from my POV, pypa is already shipping code that relies on this behaviour being by design.
Paul
Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature.
https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html
I designed it very intentionally to be compatible with zip import. Otherwise wheels would probably not even be zip files (like pypm packages which are tar) and would achieve greater compression. It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so.
On Jan 29, 2014, at 8:43 AM, Daniel Holth <dholth@gmail.com> wrote:
On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 29, 2014, at 8:10 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 January 2014 12:58, Nick Coghlan <ncoghlan@gmail.com> wrote:
You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it.
Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a "special exception" here, nor do I want it to rely on accidents of the implementation.
So from my POV, pypa is already shipping code that relies on this behaviour being by design.
Paul
Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature.
https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html
I designed it very intentionally to be compatible with zip import. Otherwise wheels would probably not even be zip files (like pypm packages which are tar) and would achieve greater compression.
It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so.
Then why did you tell someone on this very list a few hours ago that Wheels were not really designed to be zip imported. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Wed, Jan 29, 2014 at 8:44 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 29, 2014, at 8:43 AM, Daniel Holth <dholth@gmail.com> wrote:
On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 29, 2014, at 8:10 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 January 2014 12:58, Nick Coghlan <ncoghlan@gmail.com> wrote:
You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it.
Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a "special exception" here, nor do I want it to rely on accidents of the implementation.
So from my POV, pypa is already shipping code that relies on this behaviour being by design.
Paul
Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature.
https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html
I designed it very intentionally to be compatible with zip import. Otherwise wheels would probably not even be zip files (like pypm packages which are tar) and would achieve greater compression.
It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so.
Then why did you tell someone on this very list a few hours ago that Wheels were not really designed to be zip imported.
It's shorter.
It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so.
What *exactly* have we learned from eggs? We've learned that sys.path manipulation under the hood is bad, because sys.path is important to system behaviour and because the ramifications of changing it under the hood are unpredictable. But we don't let that lead to a mindset that prohibits *any* sys.path manipulation - it's vital in many situations to be able to manipulate sys.path in a controlled way. What *else* have we learned from eggs? If we leave to one side the sys.path manipulation issues, what else has been problematic about the egg format? Details seem hard to find, though there's lots of talk in general about "pain" and "weirdness". I'm asking because I really want to understand at a more detailed level what those problems are, because I would be interested in trying to solve them, or minimise their impact. That might sound naïve to some - but there it is. Regards, Vinay Sajip
On Jan 29, 2014, at 9:25 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so.
What *exactly* have we learned from eggs? We've learned that sys.path manipulation under the hood is bad, because sys.path is important to system behaviour and because the ramifications of changing it under the hood are unpredictable.
But we don't let that lead to a mindset that prohibits *any* sys.path manipulation - it's vital in many situations to be able to manipulate sys.path in a controlled way.
What *else* have we learned from eggs? If we leave to one side the sys.path manipulation issues, what else has been problematic about the egg format? Details seem hard to find, though there's lots of talk in general about "pain" and "weirdness". I'm asking because I really want to understand at a more detailed level what those problems are, because I would be interested in trying to solve them, or minimise their impact. That might sound naïve to some - but there it is.
Regards,
Vinay Sajip
It’s hard to pin down because the failure modes of zipped eggs are nebulous themselves. For instance take pip. I just recently redid the get-pip.py installer to use a zip file (not a Wheel or Egg) that contained pip and add that zip file straight to sys.path instead of unzipping it. One of the failure modes was that it was suddenly unable to validate TLS certificates. The reason why, was because it bundles it’s own cacerts.pem which it passes to the ssl library to validate the connection. In this case it’s impossible as far as I can tell to use pkgutil.get_data directly because the ssl library does not directly support pkgutil.get_data nor does it support passing the certificates as a string or other in memory. That’s really the biggest problem with strictly pure python Zipped Eggs. That the failure modes are ill defined and depend greatly on the application/library itself. For some you’ll get exceptions, for others some features will just stop working, for even others you’ll just get subtle bugs that only happen under zip import. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Wed, Jan 29, 2014 at 9:40 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 29, 2014, at 9:25 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so.
What *exactly* have we learned from eggs? We've learned that sys.path manipulation under the hood is bad, because sys.path is important to system behaviour and because the ramifications of changing it under the hood are unpredictable.
But we don't let that lead to a mindset that prohibits *any* sys.path manipulation - it's vital in many situations to be able to manipulate sys.path in a controlled way.
What *else* have we learned from eggs? If we leave to one side the sys.path manipulation issues, what else has been problematic about the egg format? Details seem hard to find, though there's lots of talk in general about "pain" and "weirdness". I'm asking because I really want to understand at a more detailed level what those problems are, because I would be interested in trying to solve them, or minimise their impact. That might sound naïve to some - but there it is.
Regards,
Vinay Sajip
It’s hard to pin down because the failure modes of zipped eggs are nebulous themselves.
For instance take pip. I just recently redid the get-pip.py installer to use a zip file (not a Wheel or Egg) that contained pip and add that zip file straight to sys.path instead of unzipping it. One of the failure modes was that it was suddenly unable to validate TLS certificates. The reason why, was because it bundles it’s own cacerts.pem which it passes to the ssl library to validate the connection. In this case it’s impossible as far as I can tell to use pkgutil.get_data directly because the ssl library does not directly support pkgutil.get_data nor does it support passing the certificates as a string or other in memory.
That’s really the biggest problem with strictly pure python Zipped Eggs. That the failure modes are ill defined and depend greatly on the application/library itself. For some you’ll get exceptions, for others some features will just stop working, for even others you’ll just get subtle bugs that only happen under zip import.
Just a heads-up that if I ever get around to re-implementing zipfile importing using importlib (http://bugs.python.org/issue17630) I will ask this list for feature feedback to make sure I don't miss something that's critical.
-------------------------------------------- On Wed, 29/1/14, Donald Stufft <donald@stufft.io> wrote:
It’s hard to pin down because the failure modes of zipped eggs are nebulous themselves.
For instance take pip. I just recently redid the get-pip.py installer to use a zip file (not a Wheel or Egg) that contained pip and add that zip file straight to sys.path instead of unzipping it. One of the failure modes was that it was suddenly unable to validate TLS certificates. The reason why, was because it bundles it’s own cacerts.pem which it passes to the ssl library to validate the connection. In this case it’s impossible as far as I can tell to use pkgutil.get_data directly because the ssl library does not directly support pkgutil.get_data nor does it support passing the certificates as a string or other in memory.
That’s really the biggest problem with strictly pure python Zipped Eggs.
If that's all, I'm not overly worried as long as there's a mountability flag on wheels. I looked at running pip from a zip a while ago, the cacerts thing was easy to diagnose and fix - a five-or-ten-minute job. (I'm not trying to come across as a show-off.) Here's the code I added as a fix in pip/locations.py (no doubt it could be improved): def _check_for_zip(): import zipimport global default_cert_path try: if isinstance(__loader__, zipimport.zipimporter): # running from a zip file. Save data to a handy location. data = __loader__.get_data(default_cert_path) destdir = os.path.expanduser('~/.pip') if not os.path.isdir(destdir): os.mkdir(destdir) default_cert_path = os.path.join(destdir, 'cacert.pem') with open(default_cert_path, 'wb') as f: f.write(data) except NameError: # __loader__ not defined, so not in zip pass _check_for_zip() del _check_for_zip Of course, pip wasn't originally designed to run from zip, so one would expect to find these kinds of issues. OTOH, if importable zips were more commonplace, then presumably people would think about these kinds of issues more when writing their code. Of course, in the above case, using pkgutil wouldn't have worked, and there are bound to be other similar cases, but I still don't see any real reason to fear the consequences of importable wheels. Regards, Vinay Sajip
On Jan 29, 2014, at 10:00 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
-------------------------------------------- On Wed, 29/1/14, Donald Stufft <donald@stufft.io> wrote:
It’s hard to pin down because the failure modes of zipped eggs are nebulous themselves.
For instance take pip. I just recently redid the get-pip.py installer to use a zip file (not a Wheel or Egg) that contained pip and add that zip file straight to sys.path instead of unzipping it. One of the failure modes was that it was suddenly unable to validate TLS certificates. The reason why, was because it bundles it’s own cacerts.pem which it passes to the ssl library to validate the connection. In this case it’s impossible as far as I can tell to use pkgutil.get_data directly because the ssl library does not directly support pkgutil.get_data nor does it support passing the certificates as a string or other in memory.
That’s really the biggest problem with strictly pure python Zipped Eggs.
If that's all, I'm not overly worried as long as there's a mountability flag on wheels. I looked at running pip from a zip a while ago, the cacerts thing was easy to diagnose and fix - a five-or-ten-minute job. (I'm not trying to come across as a show-off.) Here's the code I added as a fix in pip/locations.py (no doubt it could be improved):
def _check_for_zip(): import zipimport global default_cert_path try: if isinstance(__loader__, zipimport.zipimporter): # running from a zip file. Save data to a handy location. data = __loader__.get_data(default_cert_path) destdir = os.path.expanduser('~/.pip') if not os.path.isdir(destdir): os.mkdir(destdir) default_cert_path = os.path.join(destdir, 'cacert.pem') with open(default_cert_path, 'wb') as f: f.write(data) except NameError: # __loader__ not defined, so not in zip pass
_check_for_zip() del _check_for_zip
Of course, pip wasn't originally designed to run from zip, so one would expect to find these kinds of issues. OTOH, if importable zips were more commonplace, then presumably people would think about these kinds of issues more when writing their code. Of course, in the above case, using pkgutil wouldn't have worked, and there are bound to be other similar cases, but I still don't see any real reason to fear the consequences of importable wheels.
Regards,
Vinay Sajip
FWIW I'm not particularly that upset about this feature itself. I think it's a bad idea and I think the way it locks the format into a particular method is a poor trade off and that there is possibly a better form for this feature, but if I had engaged the process argued my side, and then lost I could deal with that. What I'm upset about is that I can find no support that the *PEP* text (not the intentions of the PEP) contains explicit support for this feature (as in, it exists because we want it to work and we think people should use it) and little to no evidence that the fact that this was (apparently) a design goal was properly communicated (which includes multiple statements by various people including Daniel that it wasn't really designed for this, though Daniel is apparently recanting that meaning). According to PEP1 a PEP must "be a clear and complete description of the proposed enhancement". I believe that what was proposed for discussion, if it was indeed intended that Wheels should be generally importable by adding them to sys path, was not clear about the fact that this was part of the proposal (and in fact contains text that reads contrary to that intent), and because the proposal and the resulting discussion was unclear, that people, such as myself, were not given the chance to discuss this particular aspect of the proposal. I feel that as BDFL delegate Nick is abusing the fact that he ultimately accepted the PEP to circumvent the lack of a clear proposal of a particular design goal (albeit likely unwittingly) to include this part of the proposal without discussion. *This* is what I have a problem with. The PEP process exists to gather everyone's input on a proposal and that ability was not afforded to people who do not believe that this particular feature should be supported. This is why I started this thread (which has someone gone off the rails trying to discuss whether or not this is a good feature or not which is immaterial to the actual issue), because I believe that this change should be reverted, and if this is something that Nick, Daniel, Vinay, or anyone else feels is important then it should be proposed as part of Wheel 1.1 and everyone should be given a fair chance to discuss it before it is accepted. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 30 Jan 2014 00:28, "Vinay Sajip" <vinay_sajip@yahoo.co.uk> wrote:
It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so.
What *exactly* have we learned from eggs? We've learned that sys.path manipulation under the hood is bad, because sys.path is important to
system
behaviour and because the ramifications of changing it under the hood are unpredictable.
But we don't let that lead to a mindset that prohibits *any* sys.path manipulation - it's vital in many situations to be able to manipulate sys.path in a controlled way.
What *else* have we learned from eggs? If we leave to one side the sys.path manipulation issues, what else has been problematic about the egg format? Details seem hard to find, though there's lots of talk in general about "pain" and "weirdness". I'm asking because I really want to understand at a more detailed level what those problems are, because I would be interested in trying to solve them, or minimise their impact. That might sound naïve to some - but there it is.
I went through this with Chris McDonough back when packaging was dropped from 3.3, and he really helped me focus on what I found to be the two closely related core problems: - implicit sys.path manipulation - installing as eggs by default That was due to easy_install defaults being chosen for the Chandler use case rather than installing into a shared environment, but the egg format (and zip imports in general) still ended up being tainted by association. More recently, I've become concerned about the way setuptools/easy_install assumes that an egg is zip import compatible in the absence of clear markers that it isn't. pip's wheel support avoids those issues, because the normal workflows just install things normally. It takes special effort to go to the trouble of keeping a wheel file *as* a wheel and then putting it on sys.path later. distlib.mount already has the virtue of always being opt-in from the user point of view, and the C extension support is opt-in from the publisher point of view, but it would be improved by needing the publisher to opt in to declaring support for zipimport (or meta imports in general), rather than letting end users deal with debugging obscure errors from code that assumes it has been fully installed. Cheers, Nick.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
I went through this with Chris McDonough back when packaging was dropped from 3.3, and he really helped me focus on what I found to be the two closely related core problems:
- implicit sys.path manipulation - installing as eggs by default
That was due to easy_install defaults being chosen for the Chandler use case rather than installing into a shared environment, but the egg format (and zip imports in general) still ended up being tainted by association.
But now that we have identified that it was "by association", we can hopefully make more informed decisions and not throw the baby out with the bath-water.
pip's wheel support avoids those issues, because the normal workflows just install things normally. It takes special effort to go to the trouble of keeping a wheel file *as* a wheel and then putting it on sys.path later.
distlib.mount already has the virtue of always being opt-in from the user point of view, and the C extension support is opt-in from the publisher point of view, but it would be improved by needing the publisher to opt in to declaring support for zipimport (or meta imports in general), rather than letting end users deal with debugging obscure errors from code that assumes it has been fully installed.
That's a measured response to a specific concern: it makes sense to codify the publisher's intent in the wheel (with a sensible default) and to have distlib's Wheel.mount() take into account the publisher's intent when considering whether to allow mounting of a wheel to take place. Regards, Vinay Sajip
On 29 January 2014 14:25, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
What *else* have we learned from eggs?
That designing modules to support zipimport correctly is non-trivial. And that assuming that things work unless told otherwise is a bad default behaviour. That packaging solutions should not depend on runtime support modules. How you package your code should not imply runtime dependencies. That package installation utilities should not dabble in sys.path manipulation. The import path is the user's responsibility. Paul
-------------------------------------------- On Wed, 29/1/14, Paul Moore <p.f.moore@gmail.com> wrote:
That designing modules to support zipimport correctly is non-trivial.
It's not trivial, but it's not especially hard, either. Mostly, it's about remembering to consider zipimport, since that hasn't been a mainstream way of deploying Python software.
And that assuming that things work unless told otherwise is a bad default behaviour.
Assumptions in general are bad, if they are made without due consideration. That's not especially egg-related, is it?
That packaging solutions should not depend on runtime support modules. How you package your code should not imply runtime dependencies.
Do you mean pkg_resources here? Certainly, distlib/distil don't expect to be present in stuff that they install.
That package installation utilities should not dabble in sys.path manipulation. The import path is the user's responsibility.
User as in developer (rather than end user). Right, and distlib's wheel code does no sys.path manipulation unless explicitly asked to. Regards, Vinay Sajip
On Wed, Jan 29, 2014 at 5:22 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk>wrote:
-------------------------------------------- On Wed, 29/1/14, Paul Moore <p.f.moore@gmail.com> wrote:
That package installation utilities should not dabble in sys.path manipulation. The import path is the user's responsibility.
User as in developer (rather than end user). Right, and distlib's wheel code does no sys.path manipulation unless explicitly asked to.
Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever reason, that should be supported. Setuptools inserting stuff to sys.path that come before PYTHONPATH entries is quite annoying. Ralf
-------------------------------------------- On Thu, 30/1/14, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever reason, that should be supported. Setuptools inserting stuff to sys.path that come before PYTHONPATH entries is quite annoying.
If tool developers want to offer end users the option to control how they work with sys.path, that's up to them. For example, once the details are worked out, the distil tool will probably get a --mountable option for the package command, which will write metadata into the built wheel indicating whether the wheel is addable to sys.path or not (based on the builder's knowledge of the wheel's contents). Distlib, when asked to mount a wheel (add it to sys.path) will check the mountability metadata and honour the wheel publisher's intent. Regards, Vinay Sajip
On Jan 30, 2014, at 1:09 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
-------------------------------------------- On Thu, 30/1/14, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever reason, that should be supported. Setuptools inserting stuff to sys.path that come before PYTHONPATH entries is quite annoying.
If tool developers want to offer end users the option to control how they work with sys.path, that's up to them. For example, once the details are worked out, the distil tool will probably get a --mountable option for the package command, which will write metadata into the built wheel indicating whether the wheel is addable to sys.path or not (based on the builder's knowledge of the wheel's contents). Distlib, when asked to mount a wheel (add it to sys.path) will check the mountability metadata and honour the wheel publisher's intent.
For everyone following along, the PEP has been updated. http://hg.python.org/peps/rev/26983acc9c11 If anyone has comments on the next text you can find it at http://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-cod... I hope we can discuss further changes as a group before they are pushed live. --Noah
On Jan 30, 2014, at 1:21 PM, Noah Kantrowitz <noah@coderanger.net> wrote:
On Jan 30, 2014, at 1:09 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
-------------------------------------------- On Thu, 30/1/14, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever reason, that should be supported. Setuptools inserting stuff to sys.path that come before PYTHONPATH entries is quite annoying.
If tool developers want to offer end users the option to control how they work with sys.path, that's up to them. For example, once the details are worked out, the distil tool will probably get a --mountable option for the package command, which will write metadata into the built wheel indicating whether the wheel is addable to sys.path or not (based on the builder's knowledge of the wheel's contents). Distlib, when asked to mount a wheel (add it to sys.path) will check the mountability metadata and honour the wheel publisher's intent.
For everyone following along, the PEP has been updated. http://hg.python.org/peps/rev/26983acc9c11
If anyone has comments on the next text you can find it at http://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-cod...
I hope we can discuss further changes as a group before they are pushed live.
--Noah
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
The updated text is fine with me as it at least accurately documents the fact that using that feature is fraught with peril and reads more of a documentation of the behavior than anything else. So thanks for the final (I hope?) update Nick, and sorry this thread got all ragey. On the plus side this spurred me into trying to put the thoughts I’ve had for a newer Wheel format down into a PEP and I think it’s turning out well so far :] ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Thanks for pointing that out Noah - I was planning to come back and check if that wording was considered more acceptable. On 31 Jan 2014 07:12, "Donald Stufft" <donald@stufft.io> wrote:
The updated text is fine with me as it at least accurately documents the
fact that using that
feature is fraught with peril and reads more of a documentation of the behavior than anything else.
Aye, clearly explaining "yes, you *can*, but that doesn't mean you *should*" was always my primary goal with the new FAQ entry. Until you commented on the original update, I didn't realise there might be a perception that this egg-replacement use case didn't need to be taken into account in designing future iterations of the wheel format, and it was only after I went back and watched the PyCon 2013 panel video that I realised we had likely run afoul of the "conference curse" where a shared offline understanding hadn't been properly captured in the online materials (it's also possible Daniel and I discussed it privately back before the wheel spec became a PEP, but I didn't dig back that far into my email archives).
So thanks for the final (I hope?) update Nick, and sorry this thread got
all ragey. Hah, if you want to see ragey, look up the epic PEP 343 threads between me and PJE before Guido broke the deadlock by proposing to just remove the cause of the argument entirely (the implicit dynamic context manager support that was in the originally accepted with statement design turned out to be thoroughly confusing because it made it too hard to come up with consistent terminology for the new concepts involved). I know all too well how disconcerting it can be to find that someone else has a significantly different interpretation of a spec that appeared to be crystal clear (especially a case like this where the written spec *was* clear, but missing a critical detail). I'm also nowhere near hypocritical enough to berate anyone *else* for passionately objecting to a course of action that appears to lead places we don't want to go :)
On the plus side this spurred me into trying to put the thoughts I've had for a newer Wheel format down into a PEP and I think it's turning out well so far :]
Huzzah! Turning caremad into PEPs can be surprisingly cathartic (cf. PEP 462). Cheers, Nick.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 29 January 2014 13:31, Donald Stufft <donald@stufft.io> wrote:
Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature.
https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html
That's quoted out of context. It was in a thread about enabling importing C extensions from zipfiles, which I'm against. I don't always write emails as if they were documents to be saved for posterity. Sometimes I even say things that are contradictory, or not right. Heck, I'm not the authority on wheels, so what I'm saying is at best opinion. Sorry if I confused you or gave you the wrong impression, but honestly I don't have the time to review and vet my emails any more than I currently do :-( Paul
Nick Coghlan wrote:
Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format's capabilities).
Would it help if zipimportable wheels were given a different extension? The format could be exactly the same (except perhaps for a flag in the metadata) but users would be able to immediately tell whether a particular wheel was zipimportable without having to run a tool. We would need a suitably catchy name for them too. Zippy wheels? Fast wheels? Hotwheels? -- Greg
On 30 Jan 2014 07:50, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:
Nick Coghlan wrote:
Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format's capabilities).
Would it help if zipimportable wheels were given a different extension? The format could be exactly the same (except perhaps for a flag in the metadata) but users would be able to immediately tell whether a particular wheel was zipimportable without having to run a tool.
We would need a suitably catchy name for them too. Zippy wheels? Fast wheels? Hotwheels?
1.1 will likely have a flag in WHEEL (off by default), and distlib.mount will then refuse to add wheels that don't have it set (including 1.0 wheels) to sys.path. Cheers, Nick.
-- Greg
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Donald Stufft <donald <at> stufft.io> writes:
1. That unpacking into the home directory is problematic because users that run services often don’t have home directories. This you waved away by saying that the home directory isn’t a required place, to which Jim responded that unpacking them anywhere else was likely to have access control / security issues and “generally cause pain”.
$HOME is just a sensible default location for caches, but as long as other locations can be used in situations where there's no HOME directory, I don't see how that's "waving away". If a service has no access to writeable disk locations at all, I can see how that might pose a problem, but that hardly seems like a use case which should dictate policies for every other scenario. OTOH "anywhere else was likely to have access control / security issues and 'generally cause pain'" is light on details.
wholly agree with, intact Mitre has assigned CVE’s to applications that did switch their Egg Cache to directories other than HOME.
Are you referring to the situation where users of setuptools set PYTHON_EGG_CACHE to a world-writeable location? If so, the problem would appear to be with the using code, not setuptools itself. AFAICT it was felt that while CVEs could be assigned against individual applications that did this, and setuptools itself was not assigned a CVE. If a HOME-less application uses an alternative cache location which isn't world-writeable, does a security issue arise?
2. Zipped Eggs are more difficult to work with for debugging purposes, which you responded saying that Wheels are a “deployment format” (which I don’t believe is supported by the PEP at all) and Jim responded with his experience in developing against things that got installed as zipped Eggs and the pain that has personally caused him.
3. Zipped imports (In Jim’s experience) are slower than unpacked imports, which you replied (quite literally) "Caveat emptor”. Jim’s response was that this had been tried with Egg and had been found to be more pain than it was worth. In his exact words:
"It's been tried with eggs. This is not new ground. Encouraging people to do this is going to cause pain and resentment.
I think one of the reasons there's so much (IMO mostly irrational) hate for eggs is that people think you can only used zipped eggs, and zipped eggs cause pain and agony.”
He closes up with,
"Importing from zipped eggs has proved itself to be an anti pattern.
Buildout (as of buildout 2) always unzips eggs.”
So you have pip that doesn’t use zipped Eggs, buildout which made sure in buildout 2 (presumably drawing from the lessons of buildout 1) to always unzip eggs, and easy_install which does support zipped eggs (sometimes, if it thinks that particular Egg will support it).
Why do we insist upon repeating the mistakes of the past?
Because there are scenarios where that functionality is desirable, and the downsides (such as greater difficulty in debugging, no caching of byte code, lower performance of imports) are tolerable. Just because something can cause problems in some scenarios is no reason for a wholesale ban. C makes segfaulting easy, but that doesn't mean that C extensions are right out ;-) Python is generally a "consenting adults" kind of system. An important difference from the setuptools/easy_install/zipped eggs experience is that nothing happens automatically - a specific action has to be taken to add wheels to sys.path, whereas in the zipped eggs case, sys.path manipulation happens under the hood. Since you don't know it's happening, failures are bound to seem weird - "it's nothing I've done, I'm sure!" Making something possible != recommending its use wholesale.
Zipped Eggs have a long long history of causing very weird failures
conditions,
you say your Wheel.mount checks for compatibility, does it also check that the library to be installed isn’t using __file__ shenanigans instead of pkgutil.get_data()? easy_install tried to do this with varying degrees of success, I’ve never seen much except pain from zipped Eggs.
Weird failure conditions are only weird because they haven't been investigated to find the exact cause. There's a lot of software around that can't run from zips because of __file__ shenanigans - that doesn't negate the very real usefulness of zipimport for other software. If I want to write software that runs from zip, I could take care to use pkgutil rather than __file__. Presumably, others can do the same. Regards, Vinay Sajip
On 29 January 2014 04:14, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
When this topic came up before, I asked for specific failure modes which were causing concern, but I never got a response IIRC.
From what I recall, the topic came up before in relation to distlib's wheel mount functionality.
My specific concerns about that API (none of which have been addressed, AFAIK, but most of which are somewhat non-specific, I concede, for reasons see below) are: 1. It does *not* just use the fact that wheels are importable. It goes beyond that and *extracts* C extensions to make them importable, too. That is a workaround for a known and accepted limitation of zipimport, and as a workaround, it has issues. If C extensions in zipfiles could work reliably, this should go into zipimport itself, and *not* into 3rd party code. Then everyone would benefit. 2. It makes what should be a rare use case, to be used only when the code in the wheel has been carefully checked to work from a zipfile, seem like a common and straightforward operation. (The "attractive nuisance" argument). I believe that people using this API will typically *not* check the code, and will blame the wheel format, or distlib, when their application does not work as expected. 3. It is no easier than sys.path.insert(0, wheelname). All it adds over that is compatibility checking and the ability to import C extensions (see above on why I think that's a bad thing). As for compatibility checking, I'd prefer a distlib.wheel.check_compatibility API that people could call *before* manually inserting the wheel onto sys.path. That's a better separation of concerns, in my view. I can't give specific examples of "failure modes" because I don't use the wheel mount functions, nor do I typically add wheels to sys.path. When I did (in virtualenv) I hit a number of issues, but all of these were ones I fixed in user code (either in virtualenv itself, or in pip, which was the wheel I was importing). So you could reasonably dismiss these as "not related to the mount API" to which all I can say is that if I'd been able to do wheel.mount(wheelname) I would likely have put less thought into whether what I was doing was a good idea - and *that's* what I think is the bad aspect of the API. Ultimately, I'll just never use the distlib mount functionality, and I'll recommend not using it when (if) people ask. But I'd rather it were not there to prompt the question. I hope this helps, Paul
Paul Moore <p.f.moore <at> gmail.com> writes:
1. It does *not* just use the fact that wheels are importable. It goes beyond that and *extracts* C extensions to make them importable, too. That is a workaround for a known and accepted limitation of zipimport, and as a workaround, it has issues. If C extensions in zipfiles could work reliably, this should go into zipimport itself, and *not* into 3rd party code. Then everyone would benefit.
(a) The mount method does not do the extraction of C extensions *unconditionally* - only when there is suitable indication in the metadata. (b) What are these issues to which you refer? As I said in my other response, I don't see the zipped-egg problem as the same, because that sys.path manipulation is under the hood/not under the developer's control. (c) Why can't third-party code break new ground, which may or may not prove fertile? If considered beneficial, it could always be added to zipimport at a later date. That's how a lot of functionality has entered the stdlib, after all.
2. It makes what should be a rare use case, to be used only when the code in the wheel has been carefully checked to work from a zipfile, seem like a common and straightforward operation. (The "attractive nuisance" argument). I believe that people using this API will typically *not* check the code, and will blame the wheel format, or distlib, when their application does not work as expected.
Doesn't the same argument apply to zipimport? Any code which is zipimported shouldn't use __file__ manipulation to access package resources, for example. Are you saying that you also feel wheels should never be importable, since one can't guarantee whether any code in general will work correctly from a zip? And yet, we have PEP 441, which is intended to encourage use of zipimport. Do we say, "no point - there's bound to be people out there who'll use __file__ instead of pkgutil"?
3. It is no easier than sys.path.insert(0, wheelname). All it adds over that is compatibility checking and the ability to import C extensions (see above on why I think that's a bad thing). As for compatibility checking, I'd prefer a distlib.wheel.check_compatibility API that people could call *before* manually inserting the wheel onto sys.path. That's a better separation of concerns, in my view.
There's no problem with providing a compatibility check API, but the other POV is that people might forget to call that API, so it seems to make sense to call it from the mount method too. And I don't see why you say "all it adds" - surely the check is important to prevent a certain class of "weird things" happening.
I can't give specific examples of "failure modes" because I don't use the wheel mount functions, nor do I typically add wheels to sys.path. When I did (in virtualenv) I hit a number of issues, but all of these were ones I fixed in user code (either in virtualenv itself, or in pip, which was the wheel I was importing). So you could reasonably
When I looked into running pip from a zip, it was clear that the issues were related to pip's use of __file__ and such, and nothing to do with shortcomings in the wheel format or zipimport.
dismiss these as "not related to the mount API" to which all I can say is that if I'd been able to do wheel.mount(wheelname) I would likely have put less thought into whether what I was doing was a good idea - and *that's* what I think is the bad aspect of the API.
Again, this feels like saying "zipimport is bad because people use __file__ instead of pkgutil". Having any API available doesn't absolve people from the responsibility of thinking about what they're doing and what the implications of using that API are.
Ultimately, I'll just never use the distlib mount functionality, and I'll recommend not using it when (if) people ask. But I'd rather it were not there to prompt the question.
You don't want it to be there, even if it might be useful to others, just because it isn't useful to you? It's not as if distlib is forcing that functionality on anyone. Regards, Vinay Sajip
On 29 January 2014 20:36, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Paul Moore <p.f.moore <at> gmail.com> writes:
You don't want it to be there, even if it might be useful to others, just because it isn't useful to you? It's not as if distlib is forcing that functionality on anyone.
I believe Paul's concern is with anything that suggests that arbitrary *third party* code can be run from wheel files, when the reality is that it is fairly easy to accidentally write code that assumes it is installed on the filesystem in a way that isn't easy for a quick scan of the files in the zip archive to detect (especially since the PEP 376 installation database PEP doesn't include any support for arbitrary metapath importers). By contrast PEP 441 is a *distribution* utility - the creator of the application is expected to ensure that doing so actually works correctly before publishing their app that way, just as we would expect py2exe, py2app and cx-freeze users to do. With the "reference implementation" position that distlib is likely to occupy in a post-PEP-426/440/459 world, though, there's an additional legitimate concern about allowing end users to easily distinguish between "this API is fully supported by the PyPA as part of the reference implementation for metadata 2.0" and "this is an experimental packaging related API that may or may not be useful in general, and some members of the PyPA may still have grave reservations about it". At the moment, distlib contains both kinds of API, and it confuses *us*, let alone anyone else that isn't closely following along on distutils-sig. As long as distlib is serving the dual role of providing both "the reference implementation for metadata 2.0" and "some experimental packaging related APIs", we're going to get concerns like this one arising. If there was a clear way to distinguish them (ideally with a separate project for either the reference implementation or the experimental stuff, but even a distinct namespace within the distlib project would help a great deal), I suspect there would be less concern. In the specific case of distlib.mount, if it's eventually combined with a metadata extension like "distlib.mount" which packages must export in order for the command to allow them to be automatically used that way, then I don't see anything wrong with it *in general* - it's a natural extension of the setuptools "zip_safe" flag, but with the ability to include additional details (like whether or not there are C extensions that need to be automatically extracted). Note that this goes further than the current EXTENSIONS approach - this proposal would be akin to *requiring* an empty EXTENSIONS file, and/or the setuptools zip_safe flag in order to allow mounting of even the pure Python wheel. Such a conservative approach is also the antithesis of the setuptools "attempt to guess": if the package publisher doesn't explicitly opt in to zip support, then distlib.mount would assume that it is *not* supported (but may provide an API for the caller to override that, like "assume_zip_safe=True" or "force=True"). However, like Paul, I have some concerns about a still experimental API like that being in the metadata 2.0 reference implementation, since that will likely end up having to deal with stdlib-like levels of backwards compatibility requirements, and removing experimental APIs that we later decided we weren't happy with could prove problematic. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan <ncoghlan <at> gmail.com> writes:
I believe Paul's concern is with anything that suggests that arbitrary *third party* code can be run from wheel files, when the reality is that it is fairly easy to accidentally write code that assumes it is installed on the filesystem in a way that isn't easy for a quick scan of the files in the zip archive to detect (especially since the PEP 376 installation database PEP doesn't include any support for arbitrary metapath importers).
That is a valid concern, but no one is suggesting that arbitrary third party code can run from wheel files, just as zipimport makes no guarantees about zipped code working.
By contrast PEP 441 is a *distribution* utility - the creator of the application is expected to ensure that doing so actually works correctly before publishing their app that way, just as we would expect py2exe, py2app and cx-freeze users to do.
True, but there's no reason why wheels couldn't have some metadata indicating that this diligence has been exercised by the wheel creator.
With the "reference implementation" position that distlib is likely to occupy in a post-PEP-426/440/459 world, though, there's an additional legitimate concern about allowing end users to easily distinguish between "this API is fully supported by the PyPA as part of the reference implementation for metadata 2.0" and "this is an experimental packaging related API that may or may not be useful in general, and some members of the PyPA may still have grave reservations about it".
At the moment, distlib contains both kinds of API, and it confuses *us*, let alone anyone else that isn't closely following along on distutils-sig. As long as distlib is serving the dual role of providing both "the reference implementation for metadata 2.0" and "some experimental packaging related APIs", we're going to get concerns like this one arising. If there was a clear way to distinguish them (ideally with a separate project for either the reference implementation or the experimental stuff, but even a distinct namespace within the distlib project would help a great deal), I suspect there would be less concern.
These are social concerns perhaps more than technical concerns, and to me they lack specificity. Of course some of the APIs in distlib are new and untried-except-by-me, but the way to allay concerns is to focus on specifics, force out the details of the concerns and then see how best they can be addressed. This is not doable with "zipped-eggs-were-bad" rhetoric. Details generally help to identify what the real problem is. For example, Donald raised the spectre of security vulnerabilities with his mention of Mitre and CVEs, but there were no specifics beyond that. I found a discussion where someone had set PYTHON_EGG_CACHE to /tmp. I can certainly see the negative security implications of that, but the finger was pointed at the using applications rather than setuptools. Even though setuptools specifically added code as a remedy to warn when the env var pointed to a world-writeable directory, this was seen as trying to be helpful rather than patching a vulnerability. Of course, if I've misunderstood something in that discussion or missed some other security issue, then some pointers would help move the discussion along.
In the specific case of distlib.mount, if it's eventually combined with a metadata extension like "distlib.mount" which packages must export in order for the command to allow them to be automatically used that way, then I don't see anything wrong with it *in general* - it's a natural extension of the setuptools "zip_safe" flag, but with the ability to include additional details (like whether or not there are C extensions that need to be automatically extracted).
Are you talking just about adding wheels to sys.path, or do you mean the extension-extraction stuff? Note that distlib's Wheel.mount does a compatibility check and addition to sys.path, which I feel is not especially controversial and better than just adding to sys.path, which user code can now do, anyway. But nothing else happens, unless specific metadata is provided in the wheel to enable it. While it's not specifically a "distlib.mount" export, there is a facility to ask for extensions to be extracted, and the in absence of metadata asking for this, no extraction occurs.
goes further than the current EXTENSIONS approach - this proposal would be akin to *requiring* an empty EXTENSIONS file, and/or the setuptools zip_safe flag in order to allow mounting of even the pure Python wheel. Such a conservative approach is also the antithesis of the setuptools "attempt to guess": if the package publisher doesn't explicitly opt in to zip support, then distlib.mount would assume that it is *not* supported (but may provide an API for the caller to override that, like "assume_zip_safe=True" or "force=True").
I have no problem with adding wheel metadata to allow/disallow even adding to sys.path - it's effectively just like another step in the compatibility check. It would make most sense to place this in the WHEEL metadata, rather than pydist.json or similar, since it relates to the contents of a particular wheel rather than the distribution in general.
However, like Paul, I have some concerns about a still experimental API like that being in the metadata 2.0 reference implementation, since that will likely end up having to deal with stdlib-like levels of backwards compatibility requirements, and removing experimental APIs that we later decided we weren't happy with could prove problematic.
But we're talking about the Python 3.5 time-frame here, and 3.4 isn't even out yet. ISTM there is plenty of time to get these sorts of issues ironed out. While I tend to favour backward compatibility wherever possible, distlib is nowhere near 1.0, and so distlib users (a small number, from what I can see) could expect some API breakage if there's no sensible alternative. Regards, Vinay Sajip
On 29 January 2014 23:59, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Nick Coghlan <ncoghlan <at> gmail.com> writes:
goes further than the current EXTENSIONS approach - this proposal would be akin to *requiring* an empty EXTENSIONS file, and/or the setuptools zip_safe flag in order to allow mounting of even the pure Python wheel. Such a conservative approach is also the antithesis of the setuptools "attempt to guess": if the package publisher doesn't explicitly opt in to zip support, then distlib.mount would assume that it is *not* supported (but may provide an API for the caller to override that, like "assume_zip_safe=True" or "force=True").
I have no problem with adding wheel metadata to allow/disallow even adding to sys.path - it's effectively just like another step in the compatibility check. It would make most sense to place this in the WHEEL metadata, rather than pydist.json or similar, since it relates to the contents of a particular wheel rather than the distribution in general.
Oh, an "Import-compatible" flag in the WHEEL metadata? Yeah, I like it. Noted as an issue for wheel 1.1: https://bitbucket.org/pypa/pypi-metadata-formats/issues?component=Wheel
However, like Paul, I have some concerns about a still experimental API like that being in the metadata 2.0 reference implementation, since that will likely end up having to deal with stdlib-like levels of backwards compatibility requirements, and removing experimental APIs that we later decided we weren't happy with could prove problematic.
But we're talking about the Python 3.5 time-frame here, and 3.4 isn't even out yet. ISTM there is plenty of time to get these sorts of issues ironed out. While I tend to favour backward compatibility wherever possible, distlib is nowhere near 1.0, and so distlib users (a small number, from what I can see) could expect some API breakage if there's no sensible alternative.
Yeah, I think I'm just nervous because sitting down and properly reviewing the distlib API is somewhere in the middle of a gigantic todo list :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 29, 2014, at 8:59 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Nick Coghlan <ncoghlan <at> gmail.com> writes:
I believe Paul's concern is with anything that suggests that arbitrary *third party* code can be run from wheel files, when the reality is that it is fairly easy to accidentally write code that assumes it is installed on the filesystem in a way that isn't easy for a quick scan of the files in the zip archive to detect (especially since the PEP 376 installation database PEP doesn't include any support for arbitrary metapath importers).
That is a valid concern, but no one is suggesting that arbitrary third party code can run from wheel files, just as zipimport makes no guarantees about zipped code working.
By contrast PEP 441 is a *distribution* utility - the creator of the application is expected to ensure that doing so actually works correctly before publishing their app that way, just as we would expect py2exe, py2app and cx-freeze users to do.
True, but there's no reason why wheels couldn't have some metadata indicating that this diligence has been exercised by the wheel creator.
With the "reference implementation" position that distlib is likely to occupy in a post-PEP-426/440/459 world, though, there's an additional legitimate concern about allowing end users to easily distinguish between "this API is fully supported by the PyPA as part of the reference implementation for metadata 2.0" and "this is an experimental packaging related API that may or may not be useful in general, and some members of the PyPA may still have grave reservations about it".
At the moment, distlib contains both kinds of API, and it confuses *us*, let alone anyone else that isn't closely following along on distutils-sig. As long as distlib is serving the dual role of providing both "the reference implementation for metadata 2.0" and "some experimental packaging related APIs", we're going to get concerns like this one arising. If there was a clear way to distinguish them (ideally with a separate project for either the reference implementation or the experimental stuff, but even a distinct namespace within the distlib project would help a great deal), I suspect there would be less concern.
These are social concerns perhaps more than technical concerns, and to me they lack specificity. Of course some of the APIs in distlib are new and untried-except-by-me, but the way to allay concerns is to focus on specifics, force out the details of the concerns and then see how best they can be addressed. This is not doable with "zipped-eggs-were-bad" rhetoric. Details generally help to identify what the real problem is. For example, Donald raised the spectre of security vulnerabilities with his mention of Mitre and CVEs, but there were no specifics beyond that. I found a discussion where someone had set PYTHON_EGG_CACHE to /tmp. I can certainly see the negative security implications of that, but the finger was pointed at the using applications rather than setuptools. Even though setuptools specifically added code as a remedy to warn when the env var pointed to a world-writeable directory, this was seen as trying to be helpful rather than patching a vulnerability. Of course, if I've misunderstood something in that discussion or missed some other security issue, then some pointers would help move the discussion along.
Mitre’s rules for CVEs are not entirely obvious to people who are not familiar with them. Generally if the feature *can* be used securely or there was no evidence that the author intended that the code be secure they will not issue a CVE. The issue is that the feature makes a very attractive footgun for people using it to do the wrong thing and have it be a very bad idea.
In the specific case of distlib.mount, if it's eventually combined with a metadata extension like "distlib.mount" which packages must export in order for the command to allow them to be automatically used that way, then I don't see anything wrong with it *in general* - it's a natural extension of the setuptools "zip_safe" flag, but with the ability to include additional details (like whether or not there are C extensions that need to be automatically extracted).
Are you talking just about adding wheels to sys.path, or do you mean the extension-extraction stuff? Note that distlib's Wheel.mount does a compatibility check and addition to sys.path, which I feel is not especially controversial and better than just adding to sys.path, which user code can now do, anyway. But nothing else happens, unless specific metadata is provided in the wheel to enable it. While it's not specifically a "distlib.mount" export, there is a facility to ask for extensions to be extracted, and the in absence of metadata asking for this, no extraction occurs.
goes further than the current EXTENSIONS approach - this proposal would be akin to *requiring* an empty EXTENSIONS file, and/or the setuptools zip_safe flag in order to allow mounting of even the pure Python wheel. Such a conservative approach is also the antithesis of the setuptools "attempt to guess": if the package publisher doesn't explicitly opt in to zip support, then distlib.mount would assume that it is *not* supported (but may provide an API for the caller to override that, like "assume_zip_safe=True" or "force=True").
I have no problem with adding wheel metadata to allow/disallow even adding to sys.path - it's effectively just like another step in the compatibility check. It would make most sense to place this in the WHEEL metadata, rather than pydist.json or similar, since it relates to the contents of a particular wheel rather than the distribution in general.
However, like Paul, I have some concerns about a still experimental API like that being in the metadata 2.0 reference implementation, since that will likely end up having to deal with stdlib-like levels of backwards compatibility requirements, and removing experimental APIs that we later decided we weren't happy with could prove problematic.
But we're talking about the Python 3.5 time-frame here, and 3.4 isn't even out yet. ISTM there is plenty of time to get these sorts of issues ironed out. While I tend to favour backward compatibility wherever possible, distlib is nowhere near 1.0, and so distlib users (a small number, from what I can see) could expect some API breakage if there's no sensible alternative.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-------------------------------------------- On Wed, 29/1/14, Donald Stufft <donald@stufft.io> wrote:
Mitre’s rules for CVEs are not entirely obvious to people who are not familiar with them. Generally if the feature *can* be used securely or there was no evidence that the author intended that the code be secure they will not issue a CVE. The issue is that the feature makes a very attractive footgun for people using it to do the wrong thing and have it be a very bad idea.
So, was a CVE issued against setuptools? My understanding is that it wasn't - have I misunderstood? tool = setuptools footgun = configurability of egg cache using PYTHON_EGG_CACHE trigger = setuptools user sets PYTHON_EGG_CACHE to a world writeable directory shot = malicious user replaces eggs in the cache with malicious code BTW when no HOME directory is available, distlib uses tempfile.mkdtemp() which IIUC provides a directory with permissions of 0700, which should be safe from tampering. Do you see a security problem with this? Regards, Vinay Sajip
On 29 January 2014 10:36, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Paul Moore <p.f.moore <at> gmail.com> writes:
1. It does *not* just use the fact that wheels are importable. It goes beyond that and *extracts* C extensions to make them importable, too. That is a workaround for a known and accepted limitation of zipimport, and as a workaround, it has issues. If C extensions in zipfiles could work reliably, this should go into zipimport itself, and *not* into 3rd party code. Then everyone would benefit.
(a) The mount method does not do the extraction of C extensions *unconditionally* - only when there is suitable indication in the metadata.
Suitable indication of what, exactly? It's not possible (at an OS level) to link to a DLL in a zipfile, so if there's a DLL in there and it's used, it needs to be extracted. I don't really see much point hairsplitting over whether that counts as "unconditional" or not.
(b) What are these issues to which you refer? As I said in my other response, I don't see the zipped-egg problem as the same, because that sys.path manipulation is under the hood/not under the developer's control.
I did point out that I knew I was being non-specific. But I know people have reported problems in the past with attempts (and there have been many) to do this. Getting cleanup right is one area. Picking a suitable location to extract to is another. But conceded, this is anecdotal. Feel free to ignore it if that's what you want to do.
(c) Why can't third-party code break new ground, which may or may not prove fertile? If considered beneficial, it could always be added to zipimport at a later date. That's how a lot of functionality has entered the stdlib, after all.
This is *not* new ground. Gordon McMillan's importer did it years ago, back when zipimport was first implemented. Eggs did it in pkg_resources. I think py2exe explicitly decided *not* to do it because of known issues with Gordon McMillan's implementation (feel free to call this anecdotal again if you want - but you may be able to find the details with some research). If this was really something that had never been tried before, I'd be happy with it as an "experimental" API. But it's neither marked as experimental, nor is it new. Your implementation *may* have found some new approach - but from your descriptions, I'm not sure it has, and you seem not to have been aware of or researched the previous attempts, so I'm guessing you're not aware of the prior art here.
2. It makes what should be a rare use case, to be used only when the code in the wheel has been carefully checked to work from a zipfile, seem like a common and straightforward operation. (The "attractive nuisance" argument). I believe that people using this API will typically *not* check the code, and will blame the wheel format, or distlib, when their application does not work as expected.
Doesn't the same argument apply to zipimport? Any code which is zipimported shouldn't use __file__ manipulation to access package resources, for example. Are you saying that you also feel wheels should never be importable, since one can't guarantee whether any code in general will work correctly from a zip? And yet, we have PEP 441, which is intended to encourage use of zipimport. Do we say, "no point - there's bound to be people out there who'll use __file__ instead of pkgutil"?
No, I'm saying that hiding the subtleties may be doing people a disservice. That's why I described it as an "attractive nuisance". I'm fully aware that this is a judgement call, and one person's convenience API is another's disaster waiting to happen. Again, I'me fine with your opinion not matching mine.
3. It is no easier than sys.path.insert(0, wheelname). All it adds over that is compatibility checking and the ability to import C extensions (see above on why I think that's a bad thing). As for compatibility checking, I'd prefer a distlib.wheel.check_compatibility API that people could call *before* manually inserting the wheel onto sys.path. That's a better separation of concerns, in my view.
There's no problem with providing a compatibility check API, but the other POV is that people might forget to call that API, so it seems to make sense to call it from the mount method too. And I don't see why you say "all it adds" - surely the check is important to prevent a certain class of "weird things" happening.
OK, I misspoke. All that mount adds over a compatibility API is a sys.path.insert call (*if* you agree with my point that the C extension stuff shouldn't be used - and I know you don't). Breaking mount down: 1. Compatibility check - I support this and would like a separate API 2. sys.path modification - trivial for anyone who knows enough to be using this API 3. C extension support - we'll have to agree to disagree here
I can't give specific examples of "failure modes" because I don't use the wheel mount functions, nor do I typically add wheels to sys.path. When I did (in virtualenv) I hit a number of issues, but all of these were ones I fixed in user code (either in virtualenv itself, or in pip, which was the wheel I was importing). So you could reasonably
When I looked into running pip from a zip, it was clear that the issues were related to pip's use of __file__ and such, and nothing to do with shortcomings in the wheel format or zipimport.
You're agreeing with me here - I just *said* you can reasonably dismiss these as not related to the mount API.
dismiss these as "not related to the mount API" to which all I can say is that if I'd been able to do wheel.mount(wheelname) I would likely have put less thought into whether what I was doing was a good idea - and *that's* what I think is the bad aspect of the API.
Again, this feels like saying "zipimport is bad because people use __file__ instead of pkgutil". Having any API available doesn't absolve people from the responsibility of thinking about what they're doing and what the implications of using that API are.
You're missing my point - zipimport is not bad because people use __file__ - pkgutil is *good* because it gives people a means of getting round some of the limitations of zipimport. But it's all layers. Convenience APIs are *abstractions*. They make things *convenience* - and one aspect of that convenience is being able to avoid bothering about some of the details. My concern is that the mount API makes it too easy for people not to think about whether what they are mounting is mountable. If the API documentation clearly stated what was mountable then this would not be an issue (but the docs would be huge and scary). But the API wouldn't look as attractive, either.
Ultimately, I'll just never use the distlib mount functionality, and I'll recommend not using it when (if) people ask. But I'd rather it were not there to prompt the question.
You don't want it to be there, even if it might be useful to others, just because it isn't useful to you? It's not as if distlib is forcing that functionality on anyone.
No, no, no. Maybe you asked for facts and I gave opinions. If so I apologise. But what I'm saying is that I feel that the API might encourage people to make mistakes they would not otherwise have considered, and that's a shame. If the API is there, personally I'll just never use it. So I don't *care* as such. Let's just agree to disagree. It's a pretty small point in the overall picture. Cheers, Paul.
participants (12)
-
Brett Cannon -
Chris Jerdonek -
Daniel Holth -
Donald Stufft -
Greg Ewing -
Matthew Iversen -
Matthew Iversen -
Nick Coghlan -
Noah Kantrowitz -
Paul Moore -
Ralf Gommers -
Vinay Sajip