But that's what I'm saying, there are only three ways to break this behaviour: 1. Changing the wheel format in such a way that we drop support for being able to install simple wheel files without a specialised installer 2. Break zipimport itself to explicitly disallow wheel files 3. Switch to a zipimport incompatible compression scheme The first two aren't going to happen, which leaves only the third. You appear to be saying that you would like to reserve the right to switch to a zipimport incompatible compression format in future versions of the wheel spec. If you're *not* saying that, then what independent design decision is there to be discussed that makes the new FAQ anything other than a clarification of the status quo? The rest of the behaviour is inherent in the "no specialised installer needed" feature. People saying "I didn't realise that the current design implied zipimport compatibility" is *why* I added the clarification, so it's not a compelling argument in convincing me that the clarification wasn't needed or is inappropriate. Regards, Nick.

On Jan 29, 2014, at 2:59 PM, Nick Coghlan wrote:

But that's what I'm saying, there are only three ways to break this behaviour:

1. Changing the wheel format in such a way that we drop support for being able to install simple wheel files without a specialised installer 2. Break zipimport itself to explicitly disallow wheel files 3. Switch to a zipimport incompatible compression scheme

The first two aren't going to happen, which leaves only the third.

You appear to be saying that you would like to reserve the right to switch to a zipimport incompatible compression format in future versions of the wheel spec. If you're *not* saying that, then what independent design decision is there to be discussed that makes the new FAQ anything other than a clarification of the status quo? The rest of the behaviour is inherent in the "no specialised installer needed" feature.

People saying "I didn't realise that the current design implied zipimport compatibility" is *why* I added the clarification, so it's not a compelling argument in convincing me that the clarification wasn't needed or is inappropriate.

If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as "well it works right now" without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior. --Noah

On Jan 29, 2014, at 8:50 PM, Tres Seaver wrote:

Signed PGP part On 01/29/2014 06:55 PM, Noah Kantrowitz wrote:

...

If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as "well it works right now" without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior.

Nick's point in this thread is that zip-importability is a *necessary corrolary* (not an implementation detaiL) of the "no special installers" design choice.

No, thats a side effect of various other technologies that are beyond the scope of that spec. The spec should either say "this is undefined behavior" or commit that it is part of the spec. We can't pick this middle ground of "well right now we just get it for free because of X, Y, and Z" because that doesn't express to the reader the true semantics of is this something they can use. Don't assume the reader has as much information as we do about the pros and cons, and about how the implementations work. --Noah

I can't believe folks are unable to differentiate between the difference of "It's possible to do so because we didn't actively attempt to prevent it" and "This is a documented goal of the format and thus must be considered as

On 30 Jan 2014 15:27, "Donald Stufft" wrote: part of

the backward compatibility contract that the format has whenever changes are made to the format".

The first option is, as far as I can tell, what the PEP read as and the discussion, that occurred in public at least, read as. However since the change Nick made he's shifted it from the first to the second type of "supports".

At this point I can only imagine people are being willfully ignorant because they want this particular feature and don't want to have it available to be discussed as per the PEP process and are actively attempting to sidestep

process. I'm very explicitly trying not to argue for or against this feature, although I believe it a bad idea, but instead that before we commit to promising that Wheels will be zip importable by simply adding them to sys.path that we *need* to discuss it and figure out what the contract we're

Donald, there are multiple design decisions in the PEP which share a common theme of zipimport compatibility, as well as an explicit statement from the PEP author that the zipimport compatibility was intentional ( https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html). The flaw was that we never explicitly noted that rationale in the PEP, and so people got the wrong idea both that this capability wasn't available and also that it might be something that didn't need to be taken into account in the future evolution of the format. This has resulted in two conflicting messages being presented to the community in the time since the wheel format was approved: if someone asked if using wheels like eggs was supported, the answer would have depended on who was giving it. I've certainly been telling people it was deliberately designed to offer a superset of egg functionality *because it was*. Daniel wrote it that way, the capability was noted several times during the design discussions, I approved it that way and *not once* did anyone suggest declaring that advanced users should not take advantage of the zipimport compatibility that is a natural consequence of the design. For example, at the PyCon US 2014 packaging panel last year, I state explicitly that you can import things from wheel files without installing them, and not one of the other panelists or attendees at the session raised any objection (neither at the time, nor privately to me after the session): https://www.youtube.com/watch?v=ePFWp3oSfyU#t=15m40 This particular genie got out of the bottle a long time ago, so my new FAQ in the PEP was just bringing it up to speed with promises that have already been made. the providing

for that is.

The problem with believing that we still have a choice about this topic is twofold: - firstly, depending on who they have spoken to users may *already* have been told it is supported (which includes everyone at the packaging panel last year and those who watched that video since, as well as everyone that directly asked either me or Daniel about it) - secondly, when a particular behaviour is an inevitable consequence of other design decisions, then users are justified in assuming that the resulting use case is supported unless it is *explicitly* disclaimed as unsupported (the comparison with eggs in PEP 427 would be a very thin reed indeed to hang a backwards compatibility break on) You're free to tell us (and the users we have communicated our intent to directly) that you would *prefer* if this was not a supported usage model, but there's a significant asymmetry here: - those of us who have always interpreted the wheel format as specifying a functional superset of the egg format (which includes both Daniel as the PEP author and me as the BDFL-Delegate that accepted it), want to ensure that this feature is properly taken into account in any future evolution of the format (including wheel 1.1) - you would like to retain the option of *not* honouring that promise, solely because we left out that detail from the PEP itself, even though we always intended it to be that way, made multiple references to the capability in discussions prior to acceptance and have told multiple people (including the attendees at the packaging panel at PyCon US 2013) that we intended it to be that way I can apologise for not reviewing PEP 427 sufficiently and failing to realise that this design assumption was not correctly captured, and so people that would have preferred to explicitly disclaim support for this feature didn't realise they needed to. However, I can also point out that all that raising such objections would have done is to ensure that a clause along these lines was added to the PEP in February 2013, rather than in January 2014, as both Daniel and I consider this a supported feature of the wheel format. That said, I will also note that the wheel *format* supporting being used this way is a very different question from *pip* supporting using them this way. pip is an installer - it takes wheels and sdists and adds them to a Python installation or virtual environment with full PEP 376 metadata. It's entirely appropriate to declare "uninstalled wheels" to be out of pip's scope, and I'd fully back any decision by the pip team to take that stance. wheel is just a file format - not every tool that supports a format for one use case needs to support *all* the use cases of that format. If people want to write tools to make it easier to run directly from a wheelhouse, they can do that - that's a very different use case from installation, one that gets back closer to the original easy_install model and one that would be better addressed by tools dedicated specifically to the task. Regards, Nick.

On Jan 30, 2014, at 9:30 AM, Donald Stufft wrote:

On Jan 30, 2014, at 6:43 AM, Nick Coghlan wrote:

...

On 30 Jan 2014 15:27, "Donald Stufft" wrote:

...

I can't believe folks are unable to differentiate between the difference of "It's possible to do so because we didn't actively attempt to prevent it" and "This is a documented goal of the format and thus must be considered as part of the backward compatibility contract that the format has whenever changes are made to the format".

Donald, there are multiple design decisions in the PEP which share a common theme of zipimport compatibility, as well as an explicit statement from the PEP author that the zipimport compatibility was intentional (https://mail.python.org/pipermail/distutils-sig/2012-September/018960.html). The flaw was that we never explicitly noted that rationale in the PEP, and so people got the wrong idea both that this capability wasn't available and also that it might be something that didn't need to be taken into account in the future evolution of the format.

This has resulted in two conflicting messages being presented to the community in the time since the wheel format was approved: if someone asked if using wheels like eggs was supported, the answer would have depended on who was giving it.

I've certainly been telling people it was deliberately designed to offer a superset of egg functionality *because it was*. Daniel wrote it that way, the capability was noted several times during the design discussions, I approved it that way and *not once* did anyone suggest declaring that advanced users should not take advantage of the zipimport compatibility that is a natural consequence of the design.

For example, at the PyCon US 2014 packaging panel last year, I state explicitly that you can import things from wheel files without installing them, and not one of the other panelists or attendees at the session raised any objection (neither at the time, nor privately to me after the session): https://www.youtube.com/watch?v=ePFWp3oSfyU#t=15m40

This particular genie got out of the bottle a long time ago, so my new FAQ in the PEP was just bringing it up to speed with promises that have already been made.

...

The first option is, as far as I can tell, what the PEP read as and the discussion, that occurred in public at least, read as. However since the change Nick made he's shifted it from the first to the second type of "supports".

At this point I can only imagine people are being willfully ignorant because they want this particular feature and don't want to have it available to be discussed as per the PEP process and are actively attempting to sidestep the process. I'm very explicitly trying not to argue for or against this feature, although I believe it a bad idea, but instead that before we commit to promising that Wheels will be zip importable by simply adding them to sys.path that we *need* to discuss it and figure out what the contract we're providing for that is.

The problem with believing that we still have a choice about this topic is twofold:

- firstly, depending on who they have spoken to users may *already* have been told it is supported (which includes everyone at the packaging panel last year and those who watched that video since, as well as everyone that directly asked either me or Daniel about it) - secondly, when a particular behaviour is an inevitable consequence of other design decisions, then users are justified in assuming that the resulting use case is supported unless it is *explicitly* disclaimed as unsupported (the comparison with eggs in PEP 427 would be a very thin reed indeed to hang a backwards compatibility break on) You're free to tell us (and the users we have communicated our intent to directly) that you would *prefer* if this was not a supported usage model, but there's a significant asymmetry here:

- those of us who have always interpreted the wheel format as specifying a functional superset of the egg format (which includes both Daniel as the PEP author and me as the BDFL-Delegate that accepted it), want to ensure that this feature is properly taken into account in any future evolution of the format (including wheel 1.1) - you would like to retain the option of *not* honouring that promise, solely because we left out that detail from the PEP itself, even though we always intended it to be that way, made multiple references to the capability in discussions prior to acceptance and have told multiple people (including the attendees at the packaging panel at PyCon US 2013) that we intended it to be that way I can apologise for not reviewing PEP 427 sufficiently and failing to realise that this design assumption was not correctly captured, and so people that would have preferred to explicitly disclaim support for this feature didn't realise they needed to. However, I can also point out that all that raising such objections would have done is to ensure that a clause along these lines was added to the PEP in February 2013, rather than in January 2014, as both Daniel and I consider this a supported feature of the wheel format.

That said, I will also note that the wheel *format* supporting being used this way is a very different question from *pip* supporting using them this way. pip is an installer - it takes wheels and sdists and adds them to a Python installation or virtual environment with full PEP 376 metadata. It's entirely appropriate to declare "uninstalled wheels" to be out of pip's scope, and I'd fully back any decision by the pip team to take that stance. wheel is just a file format - not every tool that supports a format for one use case needs to support *all* the use cases of that format. If people want to write tools to make it easier to run directly from a wheelhouse, they can do that - that's a very different use case from installation, one that gets back closer to the original easy_install model and one that would be better addressed by tools dedicated specifically to the task.

Regards, Nick.

Yea I've tried to respond to this about 5 times without being a dick and I can't. So whatever I give up. Now I need to go figure out how to make my projects bomb out under zip import since Wheel has managed to be worse than Egg and not even provide a way to disclaim support for it.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here’s how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Jan 30, 2014, at 10:12 AM, Paul Moore wrote:

On 30 January 2014 15:03, Donald Stufft wrote:

...

For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270.

Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear.

Paul

Sure, using a library via zip import when it wasn’t designed to be used as such is an anti pattern. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

I see you've noticed wheel was released in an imperfect state. Let's add a Zip-Safe flag to the WHEEL file, values of "true" and "false" same as Root-Is-Purelib. Daniel On Thu, Jan 30, 2014 at 10:44 AM, Eric V. Smith wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 01/30/2014 10:23 AM, Donald Stufft wrote:

...

On Jan 30, 2014, at 10:12 AM, Paul Moore wrote:

...

On 30 January 2014 15:03, Donald Stufft wrote:

...

For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270.

Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear.

Paul

Sure, using a library via zip import when it wasn't designed to be used as such is an anti pattern.

The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path.

There are any number of possible importers that would fail due to these 2 cases. I've personally written one that loads modules from a database. It should respect this flag, too.

Which is a long way of saying: I think calling it "zip_safe" is a misnomer. The flag is really conveying "I don't need to be on a filesystem".

Whether there should be 2 different flags for the two different problems (C extension and __file__), I can't say. They do seem to go hand-in-hand.

Eric.

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= =6tIo -----END PGP SIGNATURE----- _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 31 Jan 2014 02:22, "Donald Stufft" wrote:

Just an FYI, I've begun compiling notes and plan on proposing PEPs that

will be

aimed at supplanting PEP 425 and PEP 427 with the goal of nailing down undefined behavior, including missing functionality, and removing misfeatures.

Please file issues on the pypi-metadata-formats repo, that's one of the things it is for. I already created one requesting a flag to indicate whether or not a package supports meta-imports: https://bitbucket.org/pypa/pypi-metadata-formats/issue/17/add-wheel-metadata... (along with one for the linux distro tagging and a more controversial one to support more flexibility in the final installation layout) Cheers, Nick.

On Jan 30, 2014, at 10:56 AM, Daniel Holth wrote:

...

I see you've noticed wheel was released in an imperfect state.

Let's add a Zip-Safe flag to the WHEEL file, values of "true" and "false" same as Root-Is-Purelib.

Daniel

On Thu, Jan 30, 2014 at 10:44 AM, Eric V. Smith

wrote:

...

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 01/30/2014 10:23 AM, Donald Stufft wrote:

...

On Jan 30, 2014, at 10:12 AM, Paul Moore wrote:

...

On 30 January 2014 15:03, Donald Stufft wrote:

...

For those following along at home who have no interest in supporting zipped Wheels who want to disclaim support for it, here's how you do zip_safe=False for Wheels: https://gist.github.com/dstufft/8710270.

Note that this will also prevent use in py2exe, cx_Freeze and any similar bundling technologies. Presumably that's what you want, as any issues are identical, but it's worth being clear.

Paul

Sure, using a library via zip import when it wasn't designed to be used as such is an anti pattern.

The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path.

There are any number of possible importers that would fail due to these 2 cases. I've personally written one that loads modules from a database. It should respect this flag, too.

Which is a long way of saying: I think calling it "zip_safe" is a misnomer. The flag is really conveying "I don't need to be on a filesystem".

Whether there should be 2 different flags for the two different problems (C extension and __file__), I can't say. They do seem to go hand-in-hand.

Eric.

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS6nNFAAoJENxauZFcKtNxgYgH/RG1OKUDEgg9NSR2XH/9Uuw6 N+MrbQtPPErcyQS3OWCpat4wHiJgOy+oyJR8E3fbJBJpV72csJZBhC0jghiy1fnO l1T72084cri7X4LzfApF84N35czaCU1V1f3/ju9cpMqO5DKJMjeHu7RFdvcHq7hv a5/7/zwxPeOpl/wuQe3YODT0x+IQjQsQsvhj1S2m7zHtPQUYlYSvhuTOkKE1snqD /t5ryU2+HoywKPlITU6dkHEb6/cN8ZDFnbi5gUWXh2URGic6I52A/mfQwtItr0eN 1GnySV6Mbbgdwa7UznhXKnIuCLwqmB6D8zOVNBtOPXKMMuQuKL1IRT1aNppuS8Y= =6tIo -----END PGP SIGNATURE----- _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

---

Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 30 Jan 2014, at 17:27, Vinay Sajip wrote:

-------------------------------------------- On Thu, 30/1/14, Eric V. Smith wrote:

...

The flag really needs to convey 2 meanings: - - There are some C extensions here that can't be loaded unless they live on a real filesystem path. - - There is some code (maybe in this package, maybe in another package that uses this one) that uses this package's __file__, and that code won't work unless __file__ points to a real filesystem path.

[…]

My view is that there are probably two different assertions: one is "will run from wheel" (I say wheel rather than zip, since wheel implies a place to put that metadata) and the other is "The C extensions are mandatory”.

It’s not just wheels, the same information can be used by application bundlers like py2exe and py2app to recognise if a particular component can be included in a zipfile. Py2app currently includes all python code in a zip file within an application bundle, unless a python package is on a manually maintained list of software that won’t work that way. Ronald

On Jan 29, 2014, at 11:50 PM, Tres Seaver wrote:

Signed PGP part On 01/29/2014 06:55 PM, Noah Kantrowitz wrote:

...

If you are going to document this, and it is not going to be explicitly supported by the spec (it isn't), the _only_ logical thing is to document that this is undefined behavior and while it works now, people should not depend on it. Under no circumstance should we document this as "well it works right now" without guidance about the fact that it isn't part of the spec and is _not_ a candidate for future design decisions. If someone would like to propose amending the spec that can happen separately, but as it stands right now this is nothing but convenient, undefined behavior.

Nick's point in this thread is that zip-importability is a *necessary corrolary* (not an implementation detaiL) of the "no special installers" design choice.

- Given that there is a non-empty set of wheels which can be unpacked to a filesystem-directory tree in a directory on sys.path, and that some of those wheels are already known not to otherwise break zip- importability, it is a logical necessity that such wheels can be put onto sys.path without unpacking. We already have existence proof for this in software being released *by the folks who made these specs*.

Noting is as such is the *point* of Nick's change.

Tres. -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Also it’s entirely inaccurate that Zip import must be maintained in order to support using unzip as a simple installer for simple Wheels. For instance if Wheels instead used directories at the root named after sysconfig scheme names then with your standard unzip tool you could simple do:: unzip example.whl 'purelib/*' Which is every bit as simple and not requiring a specialized installer but ends up breaking zip import compatibility (Techincally you'd just need to add the example.whl/purelib to the sys.path). However this is a change that wasn't off the table in the original PEP, it is now in the PEP-by-fiat because we now have to maintain zip import compatibility. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA