Changing the subject to clearly focus the discussion.
On 30 January 2014 11:57, Vinay Sajip <vinay_sajip(a)yahoo.co.uk> wrote:
> If you have other reasons for your -1, I'd like to hear them.
OK. Note that this is not, in my view, an issue with wheels, but
rather about zipfiles on sys.path, and (deliberate) design limitations
of the module loader and zipimport implementations.
First of all, it is not possible to load a DLL into a process' memory
[2, 3] unless it is stored as a file in the filesystem. So any attempt
to import a C extension from a zipfile must, by necessity, involve
extracting that DLL to the filesystem. That's where I see the
problems. None are deal-breaking issues, but they consist of a number
of niggling issues that cumulatively chip away at the reliability of
the concept until the end result has enough corner cases and risks to
make it unacceptable (depending on your tolerance for risks - there's
a definite judgement call involved).
The issues I can see are: 
1. You need to choose a location to put the extracted file. On Windows
in particular, there is no guaranteed-available filesystem location
that can be used without risk. Some accounts have no home directory,
some (locked down) users have no permissions anywhere but very
specific places, even TEMP may not be usable if there's an aggressive
housekeeping routine in place - but TEMP is probably the best choice
of a bad lot.
2. There are race conditions to consider. If the extraction is not
completely isolated per-process, what if 2 processes want to use
different versions of the same DLL? How will these be distinguished?
 So to avoid corner cases you have to assume only the one process
uses a given extracted DLL.
3. Clean-up is an issue. How will the extracted files be removed? You
can't unload the DLLs from Python, and you can't delete open files in
Windows. So do you simply leave the files lying round? Or do you do
some sort of atexit dance to run a separate process after the Python
process terminates which will do the cleanup? What happens to that
process when virus checkers hold the file open? Leaving the files
around is probably the most robust answer, but it's not exactly
As I've said elsewhere, these are fundamental issues with importing
DLLs from zipfiles, and have no direct relationship to wheels. The
only place where having a wheel rather than a general zipfile makes a
difference is that a wheel *might* at some point contain metadata that
allows the wheel to claim that it's "OK" to load its contents from a
zipfile. But my points above are not something that the author of the
C extension can address, so there's no way that I can see that an
extension author can justifiably set that flag.
So: as wheels don't give any additional reliability over any other
zipfile, I don't see this (loading C extensions) as a wheel-related
feature. Ideally, if these problems can be solved, the solution should
be included in the core zipimport module so that all users can
benefit. If there are still issues to iron out and experience to be
gained, a 3rd party "enhanced zip importer" module would be a
reasonable test-bed for the solution. A 3rd party solution could also
be appropriate if the caveats and/or limitations were generally
acceptable, but sufficient to prohibit stdlib inclusion. The wheel
mount API could, if you wanted, look for the existence of that
enhanced zipimport module and use it when appropriate, but baking the
feature into wheel mount just limits your user base (and hence your
audience for raising bug reports, etc) needlessly.
I hope this explains my reasoning in sufficient detail.
FINAL DISCLAIMER: I have no objection to this feature being provided
per se, any more than I object to the existence of (say) Zope. Just
because I'm not a member of the target audience doesn't mean that it's
not a feature that some might benefit from. All I'm trying to do here
is offer my input as someone who was involved in the initial
implementation of zipimport, and who has kept an interested eye on how
it has been used in the 11 years since its introduction - and in
particular how people have tried to overcome the limitations we felt
we had to impose when designing it. Ultimately, I would be overjoyed
if someone could find a solution to this issue (in much the same way
as I'm delighted by what Brett has done with importlib).
 Historical footnote - I was directly involved with the design of
PEP 302 and the zipimport implementation, and we made a deliberate
choice to only look at pure Python files, because the platform issues
around C extensions were "too hard".
 I'm talking from a Windows perspective here. I do not have
sufficient low-level knowledge of Unix to comment on that case. I
suspect that the issues are similar but I defer to the platform
 There is, I believe, code "out there" on the internet to map a DLL
image into a process based purely in memory, but I think it's a fairly
gross hack. I have a suspicion that someone - possibly Thomas Heller -
experimented with it at one time, but never came up with a viable
implementation. There's also TCL's tclkit technology, which *may*
support binary extensions, and may be worth a look, but TCL has
virtual filesystem support built in quite deep in the core, so how it
works may not be applicable to Python.
 I'm suggesting answers to the questions I'm raising here. The
answers *may* be wrong - I've never tried to design a robust solution
to this issue - but I believe the questions are the important point
here. Please don't focus on why my suggested approach is wrong - I
know it is!
 To be fair, this is where the wheel metadata might help in
distinguishing. But consider development and testing, where repeated
test runs would not typically have different versions, but the user
might well want to test whether running from zip still works. So wheel
metadata helps, but isn't a complete solution. And compile data is
probably just as good, so let's keep assuming we are looking at a
general zipimport facility.
Is there a way to create a wheel that contains only python code, but can only be installed on particular platforms?
In particular, I’m looking for a way to create a wheel for py2app that can only be installed on OSX because py2app cannot “cross-compile” application bundles.
I noticed yesterday that Nick made a commit to PEP427 that Wheels were designed
to be compatible with zipimport so that you could directly add them to
sys.path (http://hg.python.org/peps/rev/811e34eda04c). I was not aware that
this was the case, and had thought otherwise, and have been telling people that
although Wheels could technically be used that way that doing so was considered
I think we need to reconsider this. I cannot stress how badly an idea I think
it is for Wheels to concern itself with the ability to add the Wheel to
sys.path and import it. One of the major problems with Egg was that it confused
the difference between a distribution format and a runtime installation format.
I believe that if we have this as a supported option, that ultimately it will
lead towards the same kind of problems that Egg had trying to fulfill two roles
Further more it won't even work accurately for all Wheels (as Nick's edit says)
so we require that people wanting to do this figure out if their Wheel can or
can not be added to the sys.path (which isn't as simple as looking at the tags
because a platform specific Wheel may only contain optional C modules).
If a single file importable item for a project is something that end users
want, then an additional format should be developed that does not concern
itself with the ability to unpack and install the project and instead solves
the problems that come with trying to stick arbitrary things in a zipped file
and import them.
So yea, I think that the goals of an importable format and an distribution
format are not completely overlapped, and if Wheel tries to be both, then both
sides of that will suffer with a less useful and less suitable
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I have clearly done a bad job so far of explaining the clarification in PEP
427, so here's a new attempt that relies solely on the PEP text and the way
the import system works, rather than the fact that the discussions around
the PEP show that the import system compatibility was a deliberate feature
that I took into account when accepting the PEP.
Here is the key quote from PEP 427:
"""Although a specialized installer is recommended, a wheel file may be
installed by simply unpacking into site-packages with the standard 'unzip'
tool while preserving enough information to spread its contents out onto
their final paths at any later time."""
That feature (which *is* explicitly documented in the accepted PEP),
ensures that there will always be a subset of wheel files which can be used
without needing a customised installer - just unzip them into a directory
on sys.path and go. Not all wheels will work that way, and there are some
downsides when you do it (e.g. launch scripts won't be created and
bytecode files won't be compiled with elevated permissions if that is
needed). However, it's explicitly covered in the PEP and is an apparently
non-controversial design goal for the format.
Once you have that "can be installed just by unzipping the archive"
feature, then it is *an inherent property of how zipimport works* that
there will also be an even smaller subset of wheels that will work
correctly without even unpacking them first - you can just add the archive
directly to sys.path and let zipimport do its thing.
Aside from deliberately choosing a zipimport incompatible archive format
(which we didn't do), you cannot design a format that meets the first
requirement without inevitably also supporting the latter behaviour.
However, this fact isn't obvious (as has become eminently clear to me since
reading Armin Ronacher's recent wheel article), as not everyone is
intimately with the subtleties of the import system. That is why I added
the new text to the PEP - to make this capability clear without needing to
make that connection independently.
I do now plan to reword the new FAQ entry to make it clear that the import
compatibility is mainly a side effect of the "no specialised installer
needed" feature, but hopefully the above helps make it clear why this
compatibility is already an inherent feature of the accepted PEP rather
than something that can be discussed and accepted independently in a new
version - it is pointing out a non-obvious consequence of an existing
capability, not actually adding anything new.
>Eh, I think both 1 and 3 are things that are possibly reasonable to happen and
>they are both things that I've contemplated as things to bring forward in
>using xz as an alternative compression format. Even if #1 would need a major
>revision of Wheel to happen adding official "support" for zip import means that
>the change would have to be weighed against also breaking that backwards
Eh, please don’t break it!! Improve! Keep the ability to add wheels to python path.
I don’t care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;)
I know, i’m probably alone in the desert…l;)
I searched this list for PEX and didn't find any hits, so I thought some people here might be interested in the PEX software from Twitter:
is a 15-minute presentation by Brian Wickman about it and other topics. It uses zipimport and is used to do packaging of Python software in Twitter across multiple platforms, so it could have some relevance to recent discussions we've been having about wheels, binary compatibility and the like. Brian has posted here (re. PEP 425 tags most recently) but not about PEX AFAICT. I don't yet know if there's anything particularly revolutionary about the tech (I haven't dug into it yet) but of course it's interesting because of Twitter's size and the usage in production.
After having used and built bdist_wininst installers for years (well, I
should say decades) I have recently started playing with pip.
In theory, it is a grat solution. Respects virtual environments
(although I haven't tested them yet), allows to freeze to a requirements
file, and so on.
It would allow to freeze and then commit a requirements file into our
version control system, add a 'pip install -r requirements.txt' to our
build scripts, and any user in out company as well as the build server
would automatically have installed all the required stuff. (We have 8
developers working on our source code, plus a build server that does
However, does it work in real life?
There are no wheels for important packages like pywin32, numpy, lxml,
at least not on PyPi. 'pip install' using the source packages fails to
build for most but the simplest stuff; probably because distutils has
been extended too much, or because header files or libraries are
missing, or whatever...
Is there a solution to this? I've seen that the wheel tool can convert
bdist_wininst installers into wheels - does this work for the packages
I mentioned above? Do we have to build or convert to wheel those
packages, and setup a central place where we store them and make them
available to our developers?
BTW: It seems the pip 1.4.1 doesn't download the wheel built with the
most recent setuptools/wheel version, instead it downloads and tries
to build from the source package.
Here are some of the problems with eggs that I tried to solve:
- Cannot be unzipped on top of each other due to the EGG-INFO
directory. Wheels repeat the package name in the dist-info directory
and are more like their installed layout, hopefully taking all the
mystery out of the format.
- The egg naming scheme doesn't let you distinguish between Python
interpreter implementations (not an important issue at the time).
- They include .pyc and so are always [at least somewhat] Python
version specific. Wheels don't.
- Eggs do not have categories of files. Wheel uses subdirectories of
the .data/ directory, and are therefore a complete replacement for
most other kinds of bdist_x
- And of course wheels use .dist-info which is also something that is
newer than eggs, and generally try to bring the useful packaging PEP
work into reality.
I remember dealing with the zipped eggs hassle, and I remember having
to wait too long for binary packages to be uploaded to pypi as eggs
when a new Python version was released.
I'm reading up and done all those peps:
425 - compaibility tags
426 - metadata 2.0
427 - wheel binary format 1.0
440 - version identification and dependency specifications
and trying to make sense from them. Well, they make sense to me,
but to what do they apply?
Example: I can now build, upload, and install a wheel for my distribution.
I have imported setuptools in my setup.py script,
and have run 'setup.py bdist_wheel'.
The resulting file is named 'py2exe-0.9.0-py34-none-any.whl',
python tag 'py34', abi tag 'none', platform tag 'any'
What this file really is: it is a library that only works on
CPython 3.3 and 3.4, win32 and win_amd64. So I would have expected
a filename like py2exe-0.9.0-py33.py34-none-win.whl or similar.
Hoever, the only thing that I found which did change the filename
is this section in the setup.cfg file (which is not appopriate for
So, the question is:
Are the specifications in the PEPs above implemented somewhere
or are they only 'specifications'?
There are very detailed descriptions how to specify requirements
in the metadata (in pep426), something like
"supports_environments": ["python_version >= '3.3'
and 'win' in sys_platform"]
Where can I specify this? Does distutils support it? Do
I have to manually edit some pdist.json file, or do I have
to rename the created .whl?
I just pushed a new draft of PEP 440 live:
Relative to the version I posted just before Christmas, the main
significant change is the inclusion of version epochs as part of the
In 99.9% of cases, those won't be needed and can be ignored. However,
I'm swayed by the fact that at least Debian and Fedora found them
useful enough to add to their versioning schemes, and they're a
relatively elegant solution to the "How do I switch from date based
versioning to semantic versioning without renaming my project?"
There are also two clarifications:
- I changed the pseudo-syntax for the release segment from N[.N]+ to
N(.N)* to make it clearer that single value version numbers are
- I point out in the rationale section that "-" makes local version
identifiers more readable in isolation, and the existing escaping
rules in PEP 427 take care of converting them to hyphens to avoid
ambiguity in wheel filenames. (Paul, you can consider this a belated
reply to your post from last week...)
With this update, I believe PEP 440 is functionally complete.
However, there are three additional requirements that need to be met
before I'm willing to declare it accepted:
- a reference implementation (I'm hoping Vinay will be willing to step
up to the plate with distlib again on that front)
- rerunning the compatibility analysis against the PyPI of early 2014
- surviving distutils-sig review without too much grumbling or virtual
tomato throwing ;)
I know it's a rather complex spec, but robust and comprehensive
versioning is a rather complex problem :)
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia