wheels on sys.path clarification (reboot)
I have clearly done a bad job so far of explaining the clarification in PEP 427, so here's a new attempt that relies solely on the PEP text and the way the import system works, rather than the fact that the discussions around the PEP show that the import system compatibility was a deliberate feature that I took into account when accepting the PEP. Here is the key quote from PEP 427: """Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool while preserving enough information to spread its contents out onto their final paths at any later time.""" That feature (which *is* explicitly documented in the accepted PEP), ensures that there will always be a subset of wheel files which can be used without needing a customised installer - just unzip them into a directory on sys.path and go. Not all wheels will work that way, and there are some downsides when you do it (e.g. launch scripts won't be created and bytecode files won't be compiled with elevated permissions if that is needed). However, it's explicitly covered in the PEP and is an apparently non-controversial design goal for the format. Once you have that "can be installed just by unzipping the archive" feature, then it is *an inherent property of how zipimport works* that there will also be an even smaller subset of wheels that will work correctly without even unpacking them first - you can just add the archive directly to sys.path and let zipimport do its thing. Aside from deliberately choosing a zipimport incompatible archive format (which we didn't do), you cannot design a format that meets the first requirement without inevitably also supporting the latter behaviour. However, this fact isn't obvious (as has become eminently clear to me since reading Armin Ronacher's recent wheel article), as not everyone is intimately with the subtleties of the import system. That is why I added the new text to the PEP - to make this capability clear without needing to make that connection independently. I do now plan to reword the new FAQ entry to make it clear that the import compatibility is mainly a side effect of the "no specialised installer needed" feature, but hopefully the above helps make it clear why this compatibility is already an inherent feature of the accepted PEP rather than something that can be discussed and accepted independently in a new version - it is pointing out a non-obvious consequence of an existing capability, not actually adding anything new. Regards, Nick.
On Jan 29, 2014, at 5:24 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I have clearly done a bad job so far of explaining the clarification in PEP 427, so here's a new attempt that relies solely on the PEP text and the way the import system works, rather than the fact that the discussions around the PEP show that the import system compatibility was a deliberate feature that I took into account when accepting the PEP.
Here is the key quote from PEP 427:
"""Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool while preserving enough information to spread its contents out onto their final paths at any later time."""
That feature (which *is* explicitly documented in the accepted PEP), ensures that there will always be a subset of wheel files which can be used without needing a customised installer - just unzip them into a directory on sys.path and go. Not all wheels will work that way, and there are some downsides when you do it (e.g. launch scripts won't be created and bytecode files won't be compiled with elevated permissions if that is needed). However, it's explicitly covered in the PEP and is an apparently non-controversial design goal for the format.
Once you have that "can be installed just by unzipping the archive" feature, then it is *an inherent property of how zipimport works* that there will also be an even smaller subset of wheels that will work correctly without even unpacking them first - you can just add the archive directly to sys.path and let zipimport do its thing.
Aside from deliberately choosing a zipimport incompatible archive format (which we didn't do), you cannot design a format that meets the first requirement without inevitably also supporting the latter behaviour. However, this fact isn't obvious (as has become eminently clear to me since reading Armin Ronacher's recent wheel article), as not everyone is intimately with the subtleties of the import system. That is why I added the new text to the PEP - to make this capability clear without needing to make that connection independently.
I do now plan to reword the new FAQ entry to make it clear that the import compatibility is mainly a side effect of the "no specialised installer needed" feature, but hopefully the above helps make it clear why this compatibility is already an inherent feature of the accepted PEP rather than something that can be discussed and accepted independently in a new version - it is pointing out a non-obvious consequence of an existing capability, not actually adding anything new.
Regards, Nick.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
I understand what you're saying. My point is there is a difference between supported because we didn't go out of our way to break it, and supported because it's an API contract we intend to honor. The first one is fine, because it's an implementation detail and anyone relying on it cannot expect it to have backwards compatibility concerns. It's the second one that concerned me. By including this in the PEP we've moved the fact that Wheel's are directly importable by adding them to the sys.path from an implementation detail to a documented feature (maybe documented is a better word than supported here) which means we've made some level of promise that this is going to continue working in the future. It's that second part that I have a problem with. It means that any change to Wheels in the future must be weighed against the fact that we can't break compatibility with zip import without incrementing to a new major rev of the Wheel format. It means that we're promising that this feature will work. It's essentially the difference between naming a function "_foo" instead of "foo". Yes you can use a function named "_foo" just fine, but there's no promise that it won't break in the future if you rely on that. Again let me stress that this is separate from the issue of *if* we should officially support zip importing Wheels as a feature. I have reservations about that but I would not be angry if that made it into the spec *after* discussion. What has me upset is that this "API" promise is being added without discussion with the rationale of "We didn't actively attempt to break it so now it's an officially supported feature”. I'm upset that the ability to discuss and home and perhaps exclude a certain API promise was kept from the population at large because of the fact that the format didn't go out it's way to prevent itself from being used that way. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
But that's what I'm saying, there are only three ways to break this behaviour: 1. Changing the wheel format in such a way that we drop support for being able to install simple wheel files without a specialised installer 2. Break zipimport itself to explicitly disallow wheel files 3. Switch to a zipimport incompatible compression scheme The first two aren't going to happen, which leaves only the third. You appear to be saying that you would like to reserve the right to switch to a zipimport incompatible compression format in future versions of the wheel spec. If you're *not* saying that, then what independent design decision is there to be discussed that makes the new FAQ anything other than a clarification of the status quo? The rest of the behaviour is inherent in the "no specialised installer needed" feature. People saying "I didn't realise that the current design implied zipimport compatibility" is *why* I added the clarification, so it's not a compelling argument in convincing me that the clarification wasn't needed or is inappropriate. Regards, Nick.
On Jan 29, 2014, at 5:59 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
But that's what I'm saying, there are only three ways to break this behaviour:
1. Changing the wheel format in such a way that we drop support for being able to install simple wheel files without a specialised installer 2. Break zipimport itself to explicitly disallow wheel files 3. Switch to a zipimport incompatible compression scheme
The first two aren't going to happen, which leaves only the third.
You appear to be saying that you would like to reserve the right to switch to a zipimport incompatible compression format in future versions of the wheel spec. If you're *not* saying that, then what independent design decision is there to be discussed that makes the new FAQ anything other than a clarification of the status quo? The rest of the behaviour is inherent in the "no specialised installer needed" feature.
Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official "support" for zip import means that the change would have to be weighed against also breaking that backwards compatibility.
People saying "I didn't realise that the current design implied zipimport compatibility" is *why* I added the clarification, so it's not a compelling argument in convincing me that the clarification wasn't needed or is inappropriate.
Regards, Nick.
I completely realized that the current design implied zip import compatibility, but implicit features do not make features that we intend to support going into the future. You're making the decision for us that we're not going to be making these kinds of changes and even then it's really not about whether or not this change can be removed or not. For instance, pip supports installing from Wheels how long until we get a bug report asking for the ability to install zipped Wheels? If we refuse to accept it are we violating the spec? If we implement the spec as it stands are we handing the users potential footguns? It's more than just backwards compatibility concerns as well. I care about providing the end users of these tools comprehensive solutions to the problems they solve. By simply tacking a documented feature on after the PEP has been accepted you're also removing the ability of this sig to arrive at such a comprehensive solution. Instead what you get is "Well you can add a Wheel directly to sys.path, except when you can't". This is user hostile. Essentially a feature is more than the raw ability to do something, it also includes documentation, support features, support infrastructure, etc. You've removed the chance for this sig to participate in defining how we're going to present this feature, what APIs around it we want to provide etc. Python is a very dynamic language, just because you *can* do something doesn't mean it's supported to actually do it. The PEP process exists so that important decisions can be discussed and honed. I want this change reverted and deferred to 1.1 so that we can follow the PEP process and have this discussion. These seems like the most reasonable thing to do and the thing that is most in spirit of the PEP process. It may be that ultimately zip importable Wheels are added as a feature, hopefully with the proper infrastructure to handle it sanely, but it is *wrong* to make the decision that this a feature that Wheels officially support with first having that discussion. Finally I don't understand why reverting is such a controversial change with you. As far as I can tell your rationale for adding this change is that it would be difficult to remove support for it. In that case, and if it's truly a thing that this sig wants to support as a desired feature, then I'm sure the proponents of this particular feature will have no trouble getting it formally accepted as part of Wheel 1.1 whereas inversely if it ends up being a thing that this sig does *not* want to support, "removing" that feature is much harder than "adding" it. The dialog is essentially being forced to be "Why should we break this for people who depended on our documented feature" which is a much harder thing to justify than "Why should we formally define this feature". ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jan 29, 2014, at 2:59 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
But that's what I'm saying, there are only three ways to break this behaviour:
1. Changing the wheel format in such a way that we drop support for being able to install simple wheel files without a specialised installer 2. Break zipimport itself to explicitly disallow wheel files 3. Switch to a zipimport incompatible compression scheme
The first two aren't going to happen, which leaves only the third.
You appear to be saying that you would like to reserve the right to switch to a zipimport incompatible compression format in future versions of the wheel spec. If you're *not* saying that, then what independent design decision is there to be discussed that makes the new FAQ anything other than a clarification of the status quo? The rest of the behaviour is inherent in the "no specialised installer needed" feature.
People saying "I didn't realise that the current design implied zipimport compatibility" is *why* I added the clarification, so it's not a compelling argument in convincing me that the clarification wasn't needed or is inappropriate.
If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as "well it works right now" without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior. --Noah
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/29/2014 06:55 PM, Noah Kantrowitz wrote:
If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as "well it works right now" without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior.
Nick's point in this thread is that zip-importability is a *necessary corrolary* (not an implementation detaiL) of the "no special installers" design choice. - - Given that there is a non-empty set of wheels which can be unpacked to a filesystem-directory tree in a directory on sys.path, and that some of those wheels are already known not to otherwise break zip- importability, it is a logical necessity that such wheels can be put onto sys.path without unpacking. We already have existence proof for this in software being released *by the folks who made these specs*. Noting is as such is the *point* of Nick's change. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLp2hIACgkQ+gerLs4ltQ6VGACgjnYRbgVSs8ceTXNeWTH0zCls yHwAn0/nyDUMRjNl7ARi0bVtkBOeO1nJ =2pvt -----END PGP SIGNATURE-----
On Jan 29, 2014, at 8:50 PM, Tres Seaver <tseaver@palladion.com> wrote:
Signed PGP part On 01/29/2014 06:55 PM, Noah Kantrowitz wrote:
If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as "well it works right now" without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior.
Nick's point in this thread is that zip-importability is a *necessary corrolary* (not an implementation detaiL) of the "no special installers" design choice.
No, thats a side effect of various other technologies that are beyond the scope of that spec. The spec should either say "this is undefined behavior" or commit that it is part of the spec. We can't pick this middle ground of "well right now we just get it for free because of X, Y, and Z" because that doesn't express to the reader the true semantics of is this something they can use. Don't assume the reader has as much information as we do about the pros and cons, and about how the implementations work. --Noah
On Jan 29, 2014, at 11:50 PM, Tres Seaver <tseaver@palladion.com> wrote:
Signed PGP part On 01/29/2014 06:55 PM, Noah Kantrowitz wrote:
If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as "well it works right now" without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior.
Nick's point in this thread is that zip-importability is a *necessary corrolary* (not an implementation detaiL) of the "no special installers" design choice.
- Given that there is a non-empty set of wheels which can be unpacked to a filesystem-directory tree in a directory on sys.path, and that some of those wheels are already known not to otherwise break zip- importability, it is a logical necessity that such wheels can be put onto sys.path without unpacking. We already have existence proof for this in software being released *by the folks who made these specs*.
Noting is as such is the *point* of Nick's change.
Tres. -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
I can't believe folks are unable to differentiate between the difference of "It's possible to do so because we didn't actively attempt to prevent it" and "This is a documented goal of the format and thus must be considered as part of the backward compatibility contract that the format has whenever changes are made to the format". The first option is, as far as I can tell, what the PEP read as and the discussion, that occurred in public at least, read as. However since the change Nick made he's shifted it from the first to the second type of "supports". At this point I can only imagine people are being willfully ignorant because they want this particular feature and don't want to have it available to be discussed as per the PEP process and are actively attempting to sidestep the process. I'm very explicitly trying not to argue for or against this feature, although I believe it a bad idea, but instead that before we commit to promising that Wheels will be zip importable by simply adding them to sys.path that we *need* to discuss it and figure out what the contract we're providing for that is. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I can't believe folks are unable to differentiate between the difference of "It's possible to do so because we didn't actively attempt to prevent it" and "This is a documented goal of the format and thus must be considered as
On 30 Jan 2014 15:27, "Donald Stufft" <donald@stufft.io> wrote: part of
the backward compatibility contract that the format has whenever changes are made to the format".
The first option is, as far as I can tell, what the PEP read as and the discussion, that occurred in public at least, read as. However since the change Nick made he's shifted it from the first to the second type of "supports".
At this point I can only imagine people are being willfully ignorant because they want this particular feature and don't want to have it available to be discussed as per the PEP process and are actively attempting to sidestep
process. I'm very explicitly trying not to argue for or against this feature, although I believe it a bad idea, but instead that before we commit to promising that Wheels will be zip importable by simply adding them to sys.path that we *need* to discuss it and figure out what the contract we're
Donald, there are multiple design decisions in the PEP which share a common theme of zipimport compatibility, as well as an explicit statement from the PEP author that the zipimport compatibility was intentional ( https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html). The flaw was that we never explicitly noted that rationale in the PEP, and so people got the wrong idea both that this capability wasn't available and also that it might be something that didn't need to be taken into account in the future evolution of the format. This has resulted in two conflicting messages being presented to the community in the time since the wheel format was approved: if someone asked if using wheels like eggs was supported, the answer would have depended on who was giving it. I've certainly been telling people it was deliberately designed to offer a superset of egg functionality *because it was*. Daniel wrote it that way, the capability was noted several times during the design discussions, I approved it that way and *not once* did anyone suggest declaring that advanced users should not take advantage of the zipimport compatibility that is a natural consequence of the design. For example, at the PyCon US 2014 packaging panel last year, I state explicitly that you can import things from wheel files without installing them, and not one of the other panelists or attendees at the session raised any objection (neither at the time, nor privately to me after the session): https://www.youtube.com/watch?v=ePFWp3oSfyU#t=15m40 This particular genie got out of the bottle a long time ago, so my new FAQ in the PEP was just bringing it up to speed with promises that have already been made. the providing
for that is.
The problem with believing that we still have a choice about this topic is twofold: - firstly, depending on who they have spoken to users may *already* have been told it is supported (which includes everyone at the packaging panel last year and those who watched that video since, as well as everyone that directly asked either me or Daniel about it) - secondly, when a particular behaviour is an inevitable consequence of other design decisions, then users are justified in assuming that the resulting use case is supported unless it is *explicitly* disclaimed as unsupported (the comparison with eggs in PEP 427 would be a very thin reed indeed to hang a backwards compatibility break on) You're free to tell us (and the users we have communicated our intent to directly) that you would *prefer* if this was not a supported usage model, but there's a significant asymmetry here: - those of us who have always interpreted the wheel format as specifying a functional superset of the egg format (which includes both Daniel as the PEP author and me as the BDFL-Delegate that accepted it), want to ensure that this feature is properly taken into account in any future evolution of the format (including wheel 1.1) - you would like to retain the option of *not* honouring that promise, solely because we left out that detail from the PEP itself, even though we always intended it to be that way, made multiple references to the capability in discussions prior to acceptance and have told multiple people (including the attendees at the packaging panel at PyCon US 2013) that we intended it to be that way I can apologise for not reviewing PEP 427 sufficiently and failing to realise that this design assumption was not correctly captured, and so people that would have preferred to explicitly disclaim support for this feature didn't realise they needed to. However, I can also point out that all that raising such objections would have done is to ensure that a clause along these lines was added to the PEP in February 2013, rather than in January 2014, as both Daniel and I consider this a supported feature of the wheel format. That said, I will also note that the wheel *format* supporting being used this way is a very different question from *pip* supporting using them this way. pip is an installer - it takes wheels and sdists and adds them to a Python installation or virtual environment with full PEP 376 metadata. It's entirely appropriate to declare "uninstalled wheels" to be out of pip's scope, and I'd fully back any decision by the pip team to take that stance. wheel is just a file format - not every tool that supports a format for one use case needs to support *all* the use cases of that format. If people want to write tools to make it easier to run directly from a wheelhouse, they can do that - that's a very different use case from installation, one that gets back closer to the original easy_install model and one that would be better addressed by tools dedicated specifically to the task. Regards, Nick.
On Jan 30, 2014, at 6:43 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 30 Jan 2014 15:27, "Donald Stufft" <donald@stufft.io> wrote:
I can't believe folks are unable to differentiate between the difference of "It's possible to do so because we didn't actively attempt to prevent it" and "This is a documented goal of the format and thus must be considered as part of the backward compatibility contract that the format has whenever changes are made to the format".
Donald, there are multiple design decisions in the PEP which share a common theme of zipimport compatibility, as well as an explicit statement from the PEP author that the zipimport compatibility was intentional (https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html). The flaw was that we never explicitly noted that rationale in the PEP, and so people got the wrong idea both that this capability wasn't available and also that it might be something that didn't need to be taken into account in the future evolution of the format.
This has resulted in two conflicting messages being presented to the community in the time since the wheel format was approved: if someone asked if using wheels like eggs was supported, the answer would have depended on who was giving it.
I've certainly been telling people it was deliberately designed to offer a superset of egg functionality *because it was*. Daniel wrote it that way, the capability was noted several times during the design discussions, I approved it that way and *not once* did anyone suggest declaring that advanced users should not take advantage of the zipimport compatibility that is a natural consequence of the design.
For example, at the PyCon US 2014 packaging panel last year, I state explicitly that you can import things from wheel files without installing them, and not one of the other panelists or attendees at the session raised any objection (neither at the time, nor privately to me after the session): https://www.youtube.com/watch?v=ePFWp3oSfyU#t=15m40
This particular genie got out of the bottle a long time ago, so my new FAQ in the PEP was just bringing it up to speed with promises that have already been made.
The first option is, as far as I can tell, what the PEP read as and the discussion, that occurred in public at least, read as. However since the change Nick made he's shifted it from the first to the second type of "supports".
At this point I can only imagine people are being willfully ignorant because they want this particular feature and don't want to have it available to be discussed as per the PEP process and are actively attempting to sidestep the process. I'm very explicitly trying not to argue for or against this feature, although I believe it a bad idea, but instead that before we commit to promising that Wheels will be zip importable by simply adding them to sys.path that we *need* to discuss it and figure out what the contract we're providing for that is.
The problem with believing that we still have a choice about this topic is twofold:
- firstly, depending on who they have spoken to users may *already* have been told it is supported (which includes everyone at the packaging panel last year and those who watched that video since, as well as everyone that directly asked either me or Daniel about it) - secondly, when a particular behaviour is an inevitable consequence of other design decisions, then users are justified in assuming that the resulting use case is supported unless it is *explicitly* disclaimed as unsupported (the comparison with eggs in PEP 427 would be a very thin reed indeed to hang a backwards compatibility break on) You're free to tell us (and the users we have communicated our intent to directly) that you would *prefer* if this was not a supported usage model, but there's a significant asymmetry here:
- those of us who have always interpreted the wheel format as specifying a functional superset of the egg format (which includes both Daniel as the PEP author and me as the BDFL-Delegate that accepted it), want to ensure that this feature is properly taken into account in any future evolution of the format (including wheel 1.1) - you would like to retain the option of *not* honouring that promise, solely because we left out that detail from the PEP itself, even though we always intended it to be that way, made multiple references to the capability in discussions prior to acceptance and have told multiple people (including the attendees at the packaging panel at PyCon US 2013) that we intended it to be that way I can apologise for not reviewing PEP 427 sufficiently and failing to realise that this design assumption was not correctly captured, and so people that would have preferred to explicitly disclaim support for this feature didn't realise they needed to. However, I can also point out that all that raising such objections would have done is to ensure that a clause along these lines was added to the PEP in February 2013, rather than in January 2014, as both Daniel and I consider this a supported feature of the wheel format.
That said, I will also note that the wheel *format* supporting being used this way is a very different question from *pip* supporting using them this way. pip is an installer - it takes wheels and sdists and adds them to a Python installation or virtual environment with full PEP 376 metadata. It's entirely appropriate to declare "uninstalled wheels" to be out of pip's scope, and I'd fully back any decision by the pip team to take that stance. wheel is just a file format - not every tool that supports a format for one use case needs to support *all* the use cases of that format. If people want to write tools to make it easier to run directly from a wheelhouse, they can do that - that's a very different use case from installation, one that gets back closer to the original easy_install model and one that would be better addressed by tools dedicated specifically to the task.
Regards, Nick.
Yea I've tried to respond to this about 5 times without being a dick and I can't. So whatever I give up. Now I need to go figure out how to make my projects bomb out under zip import since Wheel has managed to be worse than Egg and not even provide a way to disclaim support for it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jan 30, 2014, at 9:30 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 30, 2014, at 6:43 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 30 Jan 2014 15:27, "Donald Stufft" <donald@stufft.io> wrote:
I can't believe folks are unable to differentiate between the difference of "It's possible to do so because we didn't actively attempt to prevent it" and "This is a documented goal of the format and thus must be considered as part of the backward compatibility contract that the format has whenever changes are made to the format".
Donald, there are multiple design decisions in the PEP which share a common theme of zipimport compatibility, as well as an explicit statement from the PEP author that the zipimport compatibility was intentional (https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html). The flaw was that we never explicitly noted that rationale in the PEP, and so people got the wrong idea both that this capability wasn't available and also that it might be something that didn't need to be taken into account in the future evolution of the format.
This has resulted in two conflicting messages being presented to the community in the time since the wheel format was approved: if someone asked if using wheels like eggs was supported, the answer would have depended on who was giving it.
I've certainly been telling people it was deliberately designed to offer a superset of egg functionality *because it was*. Daniel wrote it that way, the capability was noted several times during the design discussions, I approved it that way and *not once* did anyone suggest declaring that advanced users should not take advantage of the zipimport compatibility that is a natural consequence of the design.
For example, at the PyCon US 2014 packaging panel last year, I state explicitly that you can import things from wheel files without installing them, and not one of the other panelists or attendees at the session raised any objection (neither at the time, nor privately to me after the session): https://www.youtube.com/watch?v=ePFWp3oSfyU#t=15m40
This particular genie got out of the bottle a long time ago, so my new FAQ in the PEP was just bringing it up to speed with promises that have already been made.
The first option is, as far as I can tell, what the PEP read as and the discussion, that occurred in public at least, read as. However since the change Nick made he's shifted it from the first to the second type of "supports".
At this point I can only imagine people are being willfully ignorant because they want this particular feature and don't want to have it available to be discussed as per the PEP process and are actively attempting to sidestep the process. I'm very explicitly trying not to argue for or against this feature, although I believe it a bad idea, but instead that before we commit to promising that Wheels will be zip importable by simply adding them to sys.path that we *need* to discuss it and figure out what the contract we're providing for that is.
The problem with believing that we still have a choice about this topic is twofold:
- firstly, depending on who they have spoken to users may *already* have been told it is supported (which includes everyone at the packaging panel last year and those who watched that video since, as well as everyone that directly asked either me or Daniel about it) - secondly, when a particular behaviour is an inevitable consequence of other design decisions, then users are justified in assuming that the resulting use case is supported unless it is *explicitly* disclaimed as unsupported (the comparison with eggs in PEP 427 would be a very thin reed indeed to hang a backwards compatibility break on) You're free to tell us (and the users we have communicated our intent to directly) that you would *prefer* if this was not a supported usage model, but there's a significant asymmetry here:
- those of us who have always interpreted the wheel format as specifying a functional superset of the egg format (which includes both Daniel as the PEP author and me as the BDFL-Delegate that accepted it), want to ensure that this feature is properly taken into account in any future evolution of the format (including wheel 1.1) - you would like to retain the option of *not* honouring that promise, solely because we left out that detail from the PEP itself, even though we always intended it to be that way, made multiple references to the capability in discussions prior to acceptance and have told multiple people (including the attendees at the packaging panel at PyCon US 2013) that we intended it to be that way I can apologise for not reviewing PEP 427 sufficiently and failing to realise that this design assumption was not correctly captured, and so people that would have preferred to explicitly disclaim support for this feature didn't realise they needed to. However, I can also point out that all that raising such objections would have done is to ensure that a clause along these lines was added to the PEP in February 2013, rather than in January 2014, as both Daniel and I consider this a supported feature of the wheel format.
That said, I will also note that the wheel *format* supporting being used this way is a very different question from *pip* supporting using them this way. pip is an installer - it takes wheels and sdists and adds them to a Python installation or virtual environment with full PEP 376 metadata. It's entirely appropriate to declare "uninstalled wheels" to be out of pip's scope, and I'd fully back any decision by the pip team to take that stance. wheel is just a file format - not every tool that supports a format for one use case needs to support *all* the use cases of that format. If people want to write tools to make it easier to run directly from a wheelhouse, they can do that - that's a very different use case from installation, one that gets back closer to the original easy_install model and one that would be better addressed by tools dedicated specifically to the task.
Regards, Nick.
Yea I've tried to respond to this about 5 times without being a dick and I can't. So whatever I give up. Now I need to go figure out how to make my projects bomb out under zip import since Wheel has managed to be worse than Egg and not even provide a way to disclaim support for it.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here’s how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 30 January 2014 15:03, Donald Stufft <donald@stufft.io> wrote:
For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270.
Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear. Paul
On Jan 30, 2014, at 10:12 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 January 2014 15:03, Donald Stufft <donald@stufft.io> wrote:
For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270.
Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear.
Paul
Sure, using a library via zip import when it wasn’t designed to be used as such is an anti pattern. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/30/2014 10:23 AM, Donald Stufft wrote:
On Jan 30, 2014, at 10:12 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 January 2014 15:03, Donald Stufft <donald@stufft.io> wrote:
For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270.
Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear.
Paul
Sure, using a library via zip import when it wasn’t designed to be used as such is an anti pattern.
The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path. There are any number of possible importers that would fail due to these 2 cases. I've personally written one that loads modules from a database. It should respect this flag, too. Which is a long way of saying: I think calling it "zip_safe" is a misnomer. The flag is really conveying "I don't need to be on a filesystem". Whether there should be 2 different flags for the two different problems (C extension and __file__), I can't say. They do seem to go hand-in-hand. Eric. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= =6tIo -----END PGP SIGNATURE-----
I see you've noticed wheel was released in an imperfect state. Let's add a Zip-Safe flag to the WHEEL file, values of "true" and "false" same as Root-Is-Purelib. Daniel On Thu, Jan 30, 2014 at 10:44 AM, Eric V. Smith <eric@trueblade.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 01/30/2014 10:23 AM, Donald Stufft wrote:
On Jan 30, 2014, at 10:12 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 January 2014 15:03, Donald Stufft <donald@stufft.io> wrote:
For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270.
Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear.
Paul
Sure, using a library via zip import when it wasn't designed to be used as such is an anti pattern.
The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path.
There are any number of possible importers that would fail due to these 2 cases. I've personally written one that loads modules from a database. It should respect this flag, too.
Which is a long way of saying: I think calling it "zip_safe" is a misnomer. The flag is really conveying "I don't need to be on a filesystem".
Whether there should be 2 different flags for the two different problems (C extension and __file__), I can't say. They do seem to go hand-in-hand.
Eric.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= =6tIo -----END PGP SIGNATURE----- _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Just an FYI, I've begun compiling notes and plan on proposing PEPs that will be aimed at supplanting PEP 425 and PEP 427 with the goal of nailing down undefined behavior, including missing functionality, and removing misfeatures. On Jan 30, 2014, at 10:56 AM, Daniel Holth <dholth@gmail.com> wrote:
I see you've noticed wheel was released in an imperfect state.
Let's add a Zip-Safe flag to the WHEEL file, values of "true" and "false" same as Root-Is-Purelib.
Daniel
On Thu, Jan 30, 2014 at 10:44 AM, Eric V. Smith <eric@trueblade.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 01/30/2014 10:23 AM, Donald Stufft wrote:
On Jan 30, 2014, at 10:12 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 January 2014 15:03, Donald Stufft <donald@stufft.io> wrote:
For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270.
Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear.
Paul
Sure, using a library via zip import when it wasn't designed to be used as such is an anti pattern.
The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path.
There are any number of possible importers that would fail due to these 2 cases. I've personally written one that loads modules from a database. It should respect this flag, too.
Which is a long way of saying: I think calling it "zip_safe" is a misnomer. The flag is really conveying "I don't need to be on a filesystem".
Whether there should be 2 different flags for the two different problems (C extension and __file__), I can't say. They do seem to go hand-in-hand.
Eric.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= =6tIo -----END PGP SIGNATURE----- _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 31 Jan 2014 02:22, "Donald Stufft" <donald@stufft.io> wrote:
Just an FYI, I've begun compiling notes and plan on proposing PEPs that
will be
aimed at supplanting PEP 425 and PEP 427 with the goal of nailing down undefined behavior, including missing functionality, and removing misfeatures.
Please file issues on the pypi-metadata-formats repo, that's one of the things it is for. I already created one requesting a flag to indicate whether or not a package supports meta-imports: https://bitbucket.org/pypa/pypi-metadata-formats/issue/17/add-wheel-metadata... (along with one for the linux distro tagging and a more controversial one to support more flexibility in the final installation layout) Cheers, Nick.
On Jan 30, 2014, at 10:56 AM, Daniel Holth <dholth@gmail.com> wrote:
I see you've noticed wheel was released in an imperfect state.
Let's add a Zip-Safe flag to the WHEEL file, values of "true" and "false" same as Root-Is-Purelib.
Daniel
On Thu, Jan 30, 2014 at 10:44 AM, Eric V. Smith <eric@trueblade.com>
wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 01/30/2014 10:23 AM, Donald Stufft wrote:
On Jan 30, 2014, at 10:12 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 January 2014 15:03, Donald Stufft <donald@stufft.io> wrote:
For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270.
Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear.
Paul
Sure, using a library via zip import when it wasn't designed to be used as such is an anti pattern.
The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path.
There are any number of possible importers that would fail due to these 2 cases. I've personally written one that loads modules from a database. It should respect this flag, too.
Which is a long way of saying: I think calling it "zip_safe" is a misnomer. The flag is really conveying "I don't need to be on a filesystem".
Whether there should be 2 different flags for the two different problems (C extension and __file__), I can't say. They do seem to go hand-in-hand.
Eric.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= =6tIo -----END PGP SIGNATURE----- _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-------------------------------------------- On Thu, 30/1/14, Eric V. Smith <eric@trueblade.com> wrote:
The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path.
Software can run from zip even if some things need filesystem access, so it's not as clear-cut as __file__ having to be a filesystem path. It's probably safe to assume that any C extension will need to be on a filesystem path, at least on the mainstream platforms, since that's how their run-time loaders work. For example, both pip and distil use cacert.pem to verify SSL hostnames, and the SSL module's API requires provision of a filesystem path for the .pem file. The solution is of course to use __file__ directly when running from the file system, and extracting cacert.pem to a cache location, and using that when running from zip. My view is that there are probably two different assertions: one is "will run from wheel" (I say wheel rather than zip, since wheel implies a place to put that metadata) and the other is "The C extensions are mandatory". If this flag is set, the wheel is asserted to be not worth mounting without the extensions. If clear, it means that C extensions, if present, are optional (for example, an accelerator). If there are extensions, there would need to be separate metadata giving the details so that extraction could happen (since we should refuse the temptation to guess by merely looking at file extensions of entries in the archive). Regards, Vinay Sajip
On 30 Jan 2014, at 17:27, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
-------------------------------------------- On Thu, 30/1/14, Eric V. Smith <eric@trueblade.com> wrote:
The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path.
[…]
My view is that there are probably two different assertions: one is "will run from wheel" (I say wheel rather than zip, since wheel implies a place to put that metadata) and the other is "The C extensions are mandatory”.
It’s not just wheels, the same information can be used by application bundlers like py2exe and py2app to recognise if a particular component can be included in a zipfile. Py2app currently includes all python code in a zip file within an application bundle, unless a python package is on a manually maintained list of software that won’t work that way. Ronald
On Thu, 30/1/14, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
It’s not just wheels, the same information can be used by application bundlers like py2exe and py2app to recognise if a particular component can be included in a zipfile.
Fair point - I missed that use case. Regards, Vinay Sajip
On Jan 29, 2014, at 11:50 PM, Tres Seaver <tseaver@palladion.com> wrote:
Signed PGP part On 01/29/2014 06:55 PM, Noah Kantrowitz wrote:
If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as "well it works right now" without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior.
Nick's point in this thread is that zip-importability is a *necessary corrolary* (not an implementation detaiL) of the "no special installers" design choice.
- Given that there is a non-empty set of wheels which can be unpacked to a filesystem-directory tree in a directory on sys.path, and that some of those wheels are already known not to otherwise break zip- importability, it is a logical necessity that such wheels can be put onto sys.path without unpacking. We already have existence proof for this in software being released *by the folks who made these specs*.
Noting is as such is the *point* of Nick's change.
Tres. -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Also it’s entirely inaccurate that Zip import must be maintained in order to support using unzip as a simple installer for simple Wheels. For instance if Wheels instead used directories at the root named after sysconfig scheme names then with your standard unzip tool you could simple do:: unzip example.whl 'purelib/*' Which is every bit as simple and not requiring a specialized installer but ends up breaking zip import compatibility (Techincally you'd just need to add the example.whl/purelib to the sys.path). However this is a change that wasn't off the table in the original PEP, it is now in the PEP-by-fiat because we now have to maintain zip import compatibility. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
participants (9)
-
Daniel Holth -
Donald Stufft -
Eric V. Smith -
Nick Coghlan -
Noah Kantrowitz -
Paul Moore -
Ronald Oussoren -
Tres Seaver -
Vinay Sajip