Mailman 3 January 2014 - Distutils-SIG

Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot))
by Paul Moore 31 Jan '14

31 Jan '14

Changing the subject to clearly focus the discussion. On 30 January 2014 11:57, Vinay Sajip <vinay_sajip(a)yahoo.co.uk> wrote: > If you have other reasons for your -1, I'd like to hear them. OK. Note that this is not, in my view, an issue with wheels, but rather about zipfiles on sys.path, and (deliberate) design limitations of the module loader and zipimport implementations.[1] First of all, it is not possible to load a DLL into a process' memory [2, 3] unless it is stored as a file in the filesystem. So any attempt to import a C extension from a zipfile must, by necessity, involve extracting that DLL to the filesystem. That's where I see the problems. None are deal-breaking issues, but they consist of a number of niggling issues that cumulatively chip away at the reliability of the concept until the end result has enough corner cases and risks to make it unacceptable (depending on your tolerance for risks - there's a definite judgement call involved). The issues I can see are: [4] 1. You need to choose a location to put the extracted file. On Windows in particular, there is no guaranteed-available filesystem location that can be used without risk. Some accounts have no home directory, some (locked down) users have no permissions anywhere but very specific places, even TEMP may not be usable if there's an aggressive housekeeping routine in place - but TEMP is probably the best choice of a bad lot. 2. There are race conditions to consider. If the extraction is not completely isolated per-process, what if 2 processes want to use different versions of the same DLL? How will these be distinguished? [5] So to avoid corner cases you have to assume only the one process uses a given extracted DLL. 3. Clean-up is an issue. How will the extracted files be removed? You can't unload the DLLs from Python, and you can't delete open files in Windows. So do you simply leave the files lying round? Or do you do some sort of atexit dance to run a separate process after the Python process terminates which will do the cleanup? What happens to that process when virus checkers hold the file open? Leaving the files around is probably the most robust answer, but it's not exactly friendly. As I've said elsewhere, these are fundamental issues with importing DLLs from zipfiles, and have no direct relationship to wheels. The only place where having a wheel rather than a general zipfile makes a difference is that a wheel *might* at some point contain metadata that allows the wheel to claim that it's "OK" to load its contents from a zipfile. But my points above are not something that the author of the C extension can address, so there's no way that I can see that an extension author can justifiably set that flag. So: as wheels don't give any additional reliability over any other zipfile, I don't see this (loading C extensions) as a wheel-related feature. Ideally, if these problems can be solved, the solution should be included in the core zipimport module so that all users can benefit. If there are still issues to iron out and experience to be gained, a 3rd party "enhanced zip importer" module would be a reasonable test-bed for the solution. A 3rd party solution could also be appropriate if the caveats and/or limitations were generally acceptable, but sufficient to prohibit stdlib inclusion. The wheel mount API could, if you wanted, look for the existence of that enhanced zipimport module and use it when appropriate, but baking the feature into wheel mount just limits your user base (and hence your audience for raising bug reports, etc) needlessly. I hope this explains my reasoning in sufficient detail. FINAL DISCLAIMER: I have no objection to this feature being provided per se, any more than I object to the existence of (say) Zope. Just because I'm not a member of the target audience doesn't mean that it's not a feature that some might benefit from. All I'm trying to do here is offer my input as someone who was involved in the initial implementation of zipimport, and who has kept an interested eye on how it has been used in the 11 years since its introduction - and in particular how people have tried to overcome the limitations we felt we had to impose when designing it. Ultimately, I would be overjoyed if someone could find a solution to this issue (in much the same way as I'm delighted by what Brett has done with importlib). Paul Footnotes: [1] Historical footnote - I was directly involved with the design of PEP 302 and the zipimport implementation, and we made a deliberate choice to only look at pure Python files, because the platform issues around C extensions were "too hard". [2] I'm talking from a Windows perspective here. I do not have sufficient low-level knowledge of Unix to comment on that case. I suspect that the issues are similar but I defer to the platform experts. [3] There is, I believe, code "out there" on the internet to map a DLL image into a process based purely in memory, but I think it's a fairly gross hack. I have a suspicion that someone - possibly Thomas Heller - experimented with it at one time, but never came up with a viable implementation. There's also TCL's tclkit technology, which *may* support binary extensions, and may be worth a look, but TCL has virtual filesystem support built in quite deep in the core, so how it works may not be applicable to Python. [4] I'm suggesting answers to the questions I'm raising here. The answers *may* be wrong - I've never tried to design a robust solution to this issue - but I believe the questions are the important point here. Please don't focus on why my suggested approach is wrong - I know it is! [5] To be fair, this is where the wheel metadata might help in distinguishing. But consider development and testing, where repeated test runs would not typically have different versions, but the user might well want to test whether running from zip still works. So wheel metadata helps, but isn't a complete solution. And compile data is probably just as good, so let's keep assuming we are looking at a general zipimport facility.

4 7

platform specific python only wheels
by Ronald Oussoren 31 Jan '14

31 Jan '14

Hi, Is there a way to create a wheel that contains only python code, but can only be installed on particular platforms? In particular, I’m looking for a way to create a wheel for py2app that can only be installed on OSX because py2app cannot “cross-compile” application bundles. Ronald

4 8

Using Wheel with zipimport
by Donald Stufft 31 Jan '14

31 Jan '14

I noticed yesterday that Nick made a commit to PEP427 that Wheels were designed to be compatible with zipimport so that you could directly add them to sys.path (http://hg.python.org/peps/rev/811e34eda04c). I was not aware that this was the case, and had thought otherwise, and have been telling people that although Wheels could technically be used that way that doing so was considered unsupported. I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it. One of the major problems with Egg was that it confused the difference between a distribution format and a runtime installation format. I believe that if we have this as a supported option, that ultimately it will lead towards the same kind of problems that Egg had trying to fulfill two roles at once. Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules). If a single file importable item for a project is something that end users want, then an additional format should be developed that does not concern itself with the ability to unpack and install the project and instead solves the problems that come with trying to stick arbitrary things in a zipped file and import them. So yea, I think that the goals of an importable format and an distribution format are not completely overlapped, and if Wheel tries to be both, then both sides of that will suffer with a less useful and less suitable ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

12 66

wheels on sys.path clarification (reboot)
by Nick Coghlan 31 Jan '14

31 Jan '14

I have clearly done a bad job so far of explaining the clarification in PEP 427, so here's a new attempt that relies solely on the PEP text and the way the import system works, rather than the fact that the discussions around the PEP show that the import system compatibility was a deliberate feature that I took into account when accepting the PEP. Here is the key quote from PEP 427: """Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool while preserving enough information to spread its contents out onto their final paths at any later time.""" That feature (which *is* explicitly documented in the accepted PEP), ensures that there will always be a subset of wheel files which can be used without needing a customised installer - just unzip them into a directory on sys.path and go. Not all wheels will work that way, and there are some downsides when you do it (e.g. launch scripts won't be created and bytecode files won't be compiled with elevated permissions if that is needed). However, it's explicitly covered in the PEP and is an apparently non-controversial design goal for the format. Once you have that "can be installed just by unzipping the archive" feature, then it is *an inherent property of how zipimport works* that there will also be an even smaller subset of wheels that will work correctly without even unpacking them first - you can just add the archive directly to sys.path and let zipimport do its thing. Aside from deliberately choosing a zipimport incompatible archive format (which we didn't do), you cannot design a format that meets the first requirement without inevitably also supporting the latter behaviour. However, this fact isn't obvious (as has become eminently clear to me since reading Armin Ronacher's recent wheel article), as not everyone is intimately with the subtleties of the import system. That is why I added the new text to the PEP - to make this capability clear without needing to make that connection independently. I do now plan to reword the new FAQ entry to make it clear that the import compatibility is mainly a side effect of the "no specialised installer needed" feature, but hopefully the above helps make it clear why this compatibility is already an inherent feature of the accepted PEP rather than something that can be discussed and accepted independently in a new version - it is pointing out a non-obvious consequence of an existing capability, not actually adding anything new. Regards, Nick.

9 20

wheels on sys.path clarification (reboot)
by Evgeny Sazhin 31 Jan '14

31 Jan '14

>Eh, I think both 1 and 3 are things that are possibly reasonable to happen and >they are both things that I've contemplated as things to bring forward in >using xz as an alternative compression format. Even if #1 would need a major >revision of Wheel to happen adding official "support" for zip import means that >the change would have to be weighed against also breaking that backwards >compatibility. Eh, please don’t break it!! Improve! Keep the ability to add wheels to python path. I don’t care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i’m probably alone in the desert…l;)

6 21

PEX - Twitter's multi-platform executable archive format for Python
by Vinay Sajip 30 Jan '14

30 Jan '14

I searched this list for PEX and didn't find any hits, so I thought some people here might be interested in the PEX software from Twitter: https://www.youtube.com/watch?v=NmpnGhRwsu0 is a 15-minute presentation by Brian Wickman about it and other topics. It uses zipimport and is used to do packaging of Python software in Twitter across multiple platforms, so it could have some relevance to recent discussions we've been having about wheels, binary compatibility and the like. Brian has posted here (re. PEP 425 tags most recently) but not about PEX AFAICT. I don't yet know if there's anything particularly revolutionary about the tech (I haven't dug into it yet) but of course it's interesting because of Twitter's size and the usage in production. Regards, Vinay Sajip

1 0

pip on windows experience
by Thomas Heller 30 Jan '14

30 Jan '14

After having used and built bdist_wininst installers for years (well, I should say decades) I have recently started playing with pip. In theory, it is a grat solution. Respects virtual environments (although I haven't tested them yet), allows to freeze to a requirements file, and so on. It would allow to freeze and then commit a requirements file into our version control system, add a 'pip install -r requirements.txt' to our build scripts, and any user in out company as well as the build server would automatically have installed all the required stuff. (We have 8 developers working on our source code, plus a build server that does automatic builds). However, does it work in real life? There are no wheels for important packages like pywin32, numpy, lxml, at least not on PyPi. 'pip install' using the source packages fails to build for most but the simplest stuff; probably because distutils has been extended too much, or because header files or libraries are missing, or whatever... Is there a solution to this? I've seen that the wheel tool can convert bdist_wininst installers into wheels - does this work for the packages I mentioned above? Do we have to build or convert to wheel those packages, and setup a central place where we store them and make them available to our developers? BTW: It seems the pip 1.4.1 doesn't download the wheel built with the most recent setuptools/wheel version, instead it downloads and tries to build from the source package. Thanks, Thomas

11 38

problems with eggs?
by Daniel Holth 29 Jan '14

29 Jan '14

Here are some of the problems with eggs that I tried to solve: - Cannot be unzipped on top of each other due to the EGG-INFO directory. Wheels repeat the package name in the dist-info directory and are more like their installed layout, hopefully taking all the mystery out of the format. - The egg naming scheme doesn't let you distinguish between Python interpreter implementations (not an important issue at the time). - They include .pyc and so are always [at least somewhat] Python version specific. Wheels don't. - Eggs do not have categories of files. Wheel uses subdirectories of the .data/ directory, and are therefore a complete replacement for most other kinds of bdist_x - And of course wheels use .dist-info which is also something that is newer than eggs, and generally try to bring the useful packaging PEP work into reality. I remember dealing with the zipped eggs hassle, and I remember having to wait too long for binary packages to be uploaded to pypi as eggs when a new Python version was released.

3 2

wheels, metadata
by Thomas Heller 29 Jan '14

29 Jan '14

I'm reading up and done all those peps: 425 - compaibility tags 426 - metadata 2.0 427 - wheel binary format 1.0 440 - version identification and dependency specifications and trying to make sense from them. Well, they make sense to me, but to what do they apply? Example: I can now build, upload, and install a wheel for my distribution. I have imported setuptools in my setup.py script, and have run 'setup.py bdist_wheel'. The resulting file is named 'py2exe-0.9.0-py34-none-any.whl', which means: python tag 'py34', abi tag 'none', platform tag 'any' What this file really is: it is a library that only works on CPython 3.3 and 3.4, win32 and win_amd64. So I would have expected a filename like py2exe-0.9.0-py33.py34-none-win.whl or similar. Hoever, the only thing that I found which did change the filename is this section in the setup.cfg file (which is not appopriate for this distribution): [wheel] universal=1 So, the question is: Are the specifications in the PEPs above implemented somewhere or are they only 'specifications'? There are very detailed descriptions how to specify requirements in the metadata (in pep426), something like "supports_environments": ["python_version >= '3.3' and 'win' in sys_platform"] Where can I specify this? Does distutils support it? Do I have to manually edit some pdist.json file, or do I have to rename the created .whl? Thomas

5 9

PEP 440 (versioning) nearing completion (I hope)
by Nick Coghlan 29 Jan '14

29 Jan '14

I just pushed a new draft of PEP 440 live: http://www.python.org/dev/peps/pep-0440/ Relative to the version I posted just before Christmas, the main significant change is the inclusion of version epochs as part of the spec. In 99.9% of cases, those won't be needed and can be ignored. However, I'm swayed by the fact that at least Debian and Fedora found them useful enough to add to their versioning schemes, and they're a relatively elegant solution to the "How do I switch from date based versioning to semantic versioning without renaming my project?" problem. There are also two clarifications: - I changed the pseudo-syntax for the release segment from N[.N]+ to N(.N)* to make it clearer that single value version numbers are acceptable - I point out in the rationale section that "-" makes local version identifiers more readable in isolation, and the existing escaping rules in PEP 427 take care of converting them to hyphens to avoid ambiguity in wheel filenames. (Paul, you can consider this a belated reply to your post from last week...) With this update, I believe PEP 440 is functionally complete. However, there are three additional requirements that need to be met before I'm willing to declare it accepted: - a reference implementation (I'm hoping Vinay will be willing to step up to the plate with distlib again on that front) - rerunning the compatibility analysis against the PyPI of early 2014 - surviving distutils-sig review without too much grumbling or virtual tomato throwing ;) I know it's a rather complex spec, but robust and comprehensive versioning is a rather complex problem :) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1