Thanks. Most of "wheel" is just implementing the specs we have, like metadata 1.2 and a lot of tarek et al work on packaging. The point of the implementation is that we can immediately benefit from a documented install format without having to convert everyone over to the same build system. The build and install phases can be decoupled as they should be. The resource categories in .data come from the distutils2/packaging setup.cfg specification. They are mostly for including man pages, include files and so forth. If it doesn't go directly in site packages, it goes there. The wheel spec doesn't know what they are for, it only tells you where to archive them. I'm convinced that data that the code really needs at runtime should continue to go right next to the .py files as usual. Time will tell as to the utility of .data. I think .data will make Debian maintainers happy. The new setup command bdist_wheel does exactly the task of converting egg-info to dist-info. It just doesn't understand prebuilt eggs. The tripartite compatibility tag python version - abi - architecture is what I need the most advice with. I want to anticipate a multi implementation world where the abi is not tied to the version of the python language used, and I want to be able to put several architectures of shared libraries in the same .whl when those shared libraries have unique file names. Daniel Holth On Jul 1, 2012, at 2:00 PM, PJ Eby <pje@telecommunity.com> wrote:

On Sun, Jul 1, 2012 at 1:52 AM, Daniel Holth <dholth@gmail.com> wrote: I got tired of waiting for lxml to compile over and over again, so I invented a binary packaging format called 'wheel' (of cheese) that uses Metadata 1.2 and .dist-info directories instead of .egg-info, patched pkg_resources.py to be able to load .dist-info directories, implemented "python setup.py bdist_wheel", and patched pip to be able to install .whl files.

The gist of the spec is that it is a zip file with the .whl extension containing the contents of 'purelib' for a distribution, plus a Name-1.0.dist-info/ directory with the metadata files, plus Name-1.0.data/subdir directories for scripts, platlib, packaging's "categories", ...

My specification so far is at https://docs.google.com/document/d/1mWPyvoeiqCrAy4UPNnvaz7Cgrqm4s_cfaTauAeJW... and an lxml compiled for linux-x86-64 is at https://docs.google.com/open?id=0BxHz5bC4iN5TN0VWTFNrZGtCbWs

http://bitbucket.org/dholth/distribute http://github.com/dholth/pip http://pypi.python.org/pypi/wheel

Perhaps it will be useful. The implementation is still pretty rough, for example it does not check the architecture while installing, but it could be a handy way to speed up repeated virtualenv builds.

While I haven't studied the spec in detail yet, I just want to say "hurray!" for somebody Just Doing Something About It. From a feature perspective, the only thing missing (IMO) would be the ability for pip to consume .egg and win32.exe's in order to provide a migration path. Win32.exe->egg conversion code exists in easy_install, and converting eggs to wheels should be straightforward as well, despite the mixed metaphor. ;-)

Btw, looking at the packaging docs cited in the spec, I'm confused about something: it's not clear to me what resource category most data files would be placed in, if they are actually application code. For example, a ZCML spec file is not editable and would thus not properly go into "config". ZPT files (or for that matter, most other HTML templating language files) aren't data in any meaningful sense, either. So what *is* the correct category for such *code* files which are part of the library or application, yet which are not Python modules? The doc provides no guidance on this point.

On Sun, Jul 1, 2012 at 4:50 PM, PJ Eby <pje@telecommunity.com> wrote:

On Sun, Jul 1, 2012 at 2:26 PM, Daniel Holth <dholth@gmail.com> wrote:

...

I'm convinced that data that the code really needs at runtime should continue to go right next to the .py files as usual. Time will tell as to the utility of .data. I think .data will make Debian maintainers happy.

The new setup command bdist_wheel does exactly the task of converting egg-info to dist-info. It just doesn't understand prebuilt eggs.

Well, if you do that conversion on the EGG-INFO directory of an .egg file, then you'll have a wheel. At least, if I understand your spec correctly. (Since .egg contains just the code and static files.) All you need to do is get the egg's platform info and Python version from its filename; everything else is in .egg-info. The full spec is here:

http://peak.telecommunity.com/DevCenter/EggFormats

But basically, it's just going to be just convert EGG-INFO to Projectname.dist-info. The actual layout of the rest of the zipfile is essentially unchanged.

Yes. You have to use Metadata 1.2 (list requirements with Requires-Dist:) and PKG-INFO has been renamed to METADATA by the distutils-sig for whatever reason. The WHEEL file lets an installer complain when the next version of the format is invented. Once it is installed into site-packages it is not a wheel anymore, it is just a PEP-compliant Python distribution.

(Heck, add .whl support to pkg_resources, and wheels will have all the advantages of eggs as well. Though, for that to work under 3.x some of the stub file stuff would have to be changed as well.)

The pkg_resources in the patched distribute this requires already understands the unpacked format. This would be a good small project.

On 2 July 2012 14:17, Daniel Holth <dholth@gmail.com> wrote:

Shell script to build a directory full of wheels and then install them. https://gist.github.com/3033186

Is wheel for Python 2? I'm trying to build using 3.3a2, and I get:

.\Scripts\pip install -e hg+https://bitbucket.org/dholth/wheel#egg=wheel -e hg+https://bitbucket.org/dholth/distribute# egg=distribute -e git+https://github.com/dholth/pip.git#egg=pip Obtaining wheel from hg+https://bitbucket.org/dholth/wheel#egg=wheel Updating d:\data\wheeltest\src\wheel clone Running setup.py egg_info for package wheel

Obtaining distribute from hg+https://bitbucket.org/dholth/distribute#egg=distribute Cloning hg https://bitbucket.org/dholth/distribute to d:\data\wheeltest\src\distribute Running setup.py egg_info for package distribute Traceback (most recent call last): File "<string>", line 3, in <module> File "setuptools\__init__.py", line 2, in <module> from setuptools.extension import Extension, Library File "setuptools\extension.py", line 5, in <module> from setuptools.dist import _get_unpatched File "setuptools\dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax Complete output from command python setup.py egg_info: Traceback (most recent call last): File "<string>", line 3, in <module> File "setuptools\__init__.py", line 2, in <module> from setuptools.extension import Extension, Library File "setuptools\extension.py", line 5, in <module> from setuptools.dist import _get_unpatched File "setuptools\dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax ---------------------------------------- Command python setup.py egg_info failed with error code 1 Storing complete log in D:\Documents and Settings\UK03306\Application Data\pip\pip.log That's Python 2 syntax, so the error is correct. But why is Python 3 finding a Python 2 format file? Paul.

...

Is wheel for Python 2? I'm trying to build using 3.3a2, and I get:

At the moment, yes. I have yet to set up my tox environment to test in both, but https://bitbucket.org/dholth/cpython contains a (slightly older) version of it for 'packaging'.

Oops, that's wrong. It just contains some of the precursor work that was ported over to distribute.

Actually, I've hit a number of other issues when I try Python 2, largely related to (a) the fact that I'm on Windows, so you can't install pip with itself (can't overwrite the exe), and (b) I don't have a workable git (so I had to download the patched pip manually) and (c) it's a develop build, so I can't delete the patched pip once it's installed (this is mainly because I don't have any familiarity with develop installs, so I don't know how to change the command to get a "normal" install - the obvious approach of changing "develop" to "install" didn't work...). Maybe the Python 3 problem wasn't python 3 as such.

It is always a pain to use an installer to replace itself. Sorry.

I'll try to get a workable version on Windows when I have some time. (But I'd still like to know if it's intended to work on Python 3 - if not, I probably won't bother for now).

Double oops. I have pushed bdist_wheel to my cpython fork, but that doesn't take care of the installer. I am definitely interested in porting the distribute version over to Python 3. I just haven't gotten to it.

Looks like your biggest Py3 issue is that distribute uses 2to3. "pip install -e" of course does not. It works much better otherwise. I put in some tiny changes to the other bits to make them work, and I can now install Pyramid from wheels on Py3.2. On Mon, Jul 2, 2012 at 11:59 AM, Paul Moore <p.f.moore@gmail.com> wrote:

Hmm, I'm a little confused as to how the bits hang together here. Maybe some clarification would help. How close is the following?

pip - your patch for this is to recognise wheel files and use distribute to install them distribute - your patch for this is to add support for installing wheel files (and to add a bdist_wheel command for building them???) wheel - support library for the above, provides functions to work with wheel files

Is that close?

Assuming it is, I'm not sure I follow what you say in the paragraph above - what is the cpython patch you mention?

No longer important, just the first thing I tried. It is a bdist_wheel for distutils2 / packaging. It wound up being much more fun to write a setuptools/distribute plugin, so I finished there. The most important and most complete patch is probably pkg_resources.py (part of setuptools / distribute) to recognize the .dist-info directories and the Provides-Extra: extension. This is just PEP 386 and is not wheel specific. If 'markerlib' is installed it will interpret conditional requirements; markerlib should become standard/required. The distribution 'wheel' is the bdist_wheel setuptools plugin. It is also where a command line tool to manipulate wheel files would go. pip recognizes wheels and installs them itself. https://github.com/dholth/pip/blob/develop/pip/req.py#L759 It doesn't yet +x scripts or update .dist-info/RECORD to keep track of what it should uninstall.

Based on my interpretation of what you say above, the things I see outstanding are:

1. wheel discovery and download from PyPI or local archives (so pip install x will find and download x.whl before/as well as an x sdist) 2. wheel upload to PyPI (probably needs PyPI changes to support the new format)

Also, we will be moving to urlsafe-b64encoded SHA256 instead of MD5 as the hash function.

3. converters to allow people to convert eggs and bdist_wininst binaries to wheel format (otherwise you have a chicken-and-egg problem in that wheels are only good for speeding up installs, rather than for installing things you can't build for yourself)

Sure, although IMO egg2wheel is more of a roundabout way to get around the "pip doesn't support eggs" issue. Hopefully in most cases the maintainer or "someone with a compiler" will run bdist_wheel for you when a new version comes out.

(1) and (2) are longer-term requirements, and (3) may or may not be a key requirement for you - in any case, others who do need the facility could develop these independently. So it seems to me like you're nearly there (assuming the pip and distribute maintainers accept your patches).

Paul.

...

It doesn't yet +x scripts or update .dist-info/RECORD to keep track of what it should uninstall.

So pip remove won't work right on installed wheels? That would probably be my #1 issue, in practice, then.

Very easy to fix pip remove though, just another elif: in pip.req uninstall(). I also intend to define an entry point for post-install hooks generally.

But certainly converters are an interim measure. (Just an important one IMO).

You can use the issue tracker on http://bitbucket.org/dholth/wheel to help build a roadmap

Daniel Holth wrote:

On Wed, Aug 15, 2012 at 11:55 AM, M.-A. Lemburg <mal@egenix.com> wrote:

...

You might also want to take a look at the "prebuilt" binary format which we have been using for several years now, e.g.

http://www.egenix.com/products/python/mxBase/

The idea is a little different from what you describe, but works well: we essentially take a snapshot of the package after it was built and then put everything into a ZIP file.

As a result, you can pick up where distutils left off after the build and continue the installation on the target machine.

Thanks, that is fascinating. Wheel goes a little further, and zips up an installation of the package (with a particular set of installation paths), and it doesn't include setup.py. This is important because I want to be able to install into a Python that has no distutils at all.

Ok, that's a different set of requirements then. We wanted to have most of the distutils commands and options working in order to stay flexible during the actual installation process. Since the prebuilt packages are also compatible to the standard "python setup.py install" dance after you've unzipped them, any installer using this approach will just work as well. We've tested pip and both the install and uninstall commands work fine out of the box as a result.

...

The prebuilt files can also be installed and uninstalled with pip if you reference the files directly.

The only feature missing is support in pip for finding and downloading the right prebuilt archive from an index server. pip currently defaults to downloading the Windows builds, because it checks for the "highest" version available (for some meaning of high ;-)).

Which makes me think: would it be possible to make pip more clever with respect to platform version strings ?

I do plan to implement this with some help from my friends, at least as defined by PEP 425 (in progress; http://hg.python.org/peps/file/tip/pep-0425.txt ). I think it will be necessary to introduce the concept of picking the best package from a set of candidates with the same version, instead of just picking the highest version as pip does now.

Great. Please let me know when there's something available to test. Since we currently only use the version strings for human consumption, their are easy to change to whatever standard format will get adopted. The only requirements that we'd have for such a version tag format (based on our experience with the prebuilt formats) is that it includes: * a platform string with enough information to determine basic binary compatibility (ie. OS, architecture, perhaps also OS version depending on OS) * Python version, since the byte code changes with every release * Unicode variant; at least for those Python versions that need it, e.g. non-Windows builds, Python 2.x, 3.0-3.2. There's an interesting problem we faced with pure-Python builds: on Windows, many distutils commands store their internal path data in the OS format, not in the standard distutils format (which uses the Unix variant with os.sep == '/'). As a result, pure-Python builds only work cross-platform with the prebuilt format if they are created on a Unix host, since Windows works fine with the forward slash os.sep at the API level. As a result, we had to put the platform string on pure-Python builds created on Windows platforms.

...

Here's are some examples of such a version strings as detected by pip:

3.2.4.win32-py2.7.prebuilt 3.2.4.win-amd64-py2.7.prebuilt 3.2.4.linux-x86_64-py2.7_ucs4.prebuilt 3.2.4.linux-x86_64-py2.7_ucs2.prebuilt 3.2.4.linux-i686-py2.7_ucs4.prebuilt 3.2.4.linux-i686-py2.7_ucs2.prebuilt

The code for bdist_prebuilt is in the mxSetup.py that comes with egenix-mx-base, in case you want to take a look.

I will take a look.

Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 16 2012)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2012-08-25: FrOSCon, St. Augustin, Germany ... 9 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/