[Distutils] the 'wheel' binary package format

[M.-A. Lemburg]

Daniel Holth wrote:
> On Wed, Aug 15, 2012 at 11:55 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> You might also want to take a look at the "prebuilt" binary
>> format which we have been using for several years now, e.g.
>>
>> http://www.egenix.com/products/python/mxBase/
>>
>> The idea is a little different from what you describe, but works
>> well: we essentially take a snapshot of the package after it was
>> built and then put everything into a ZIP file.
>>
>> As a result, you can pick up where distutils left off after the
>> build and continue the installation on the target machine.
> 
> Thanks, that is fascinating. Wheel goes a little further, and zips up
> an installation of the package (with a particular set of installation
> paths), and it doesn't include setup.py. This is important because I
> want to be able to install into a Python that has no distutils at all.

Ok, that's a different set of requirements then. We wanted
to have most of the distutils commands and options working
in order to stay flexible during the actual installation process.

Since the prebuilt packages are also compatible to the standard
"python setup.py install" dance after you've unzipped them,
any installer using this approach will just work as well.

We've tested pip and both the install and uninstall commands
work fine out of the box as a result.

>> The prebuilt files can also be installed and uninstalled with pip
>> if you reference the files directly.
>>
>> The only feature missing is support in pip for finding and downloading
>> the right prebuilt archive from an index server. pip currently defaults
>> to downloading the Windows builds, because it checks for the "highest"
>> version available (for some meaning of high ;-)).
>>
>> Which makes me think: would it be possible to make pip more clever
>> with respect to platform version strings ?
> 
> I do plan to implement this with some help from my friends, at least
> as defined by PEP 425 (in progress;
> http://hg.python.org/peps/file/tip/pep-0425.txt ). I think it will be
> necessary to introduce the concept of picking the best package from a
> set of candidates with the same version, instead of just picking the
> highest version as pip does now.

Great. Please let me know when there's something available to
test. Since we currently only use the version strings for human
consumption, their are easy to change to whatever standard format
will get adopted.

The only requirements that we'd have for such a version tag format
(based on our experience with the prebuilt formats) is that it
includes:

 * a platform string with enough information to determine
   basic binary compatibility (ie. OS, architecture, perhaps
   also OS version depending on OS)
 * Python version, since the byte code changes with every release
 * Unicode variant; at least for those Python versions that need
   it, e.g. non-Windows builds, Python 2.x, 3.0-3.2.

There's an interesting problem we faced with pure-Python builds: on
Windows, many distutils commands store their internal path data in
the OS format, not in the standard distutils format (which uses
the Unix variant with os.sep == '/'). As a result, pure-Python builds
only work cross-platform with the prebuilt format if they are
created on a Unix host, since Windows works fine with the forward
slash os.sep at the API level. As a result, we had to put the
platform string on pure-Python builds created on Windows platforms.

>> Here's are some examples of such a version strings as detected by
>> pip:
>>
>> 3.2.4.win32-py2.7.prebuilt
>> 3.2.4.win-amd64-py2.7.prebuilt
>> 3.2.4.linux-x86_64-py2.7_ucs4.prebuilt
>> 3.2.4.linux-x86_64-py2.7_ucs2.prebuilt
>> 3.2.4.linux-i686-py2.7_ucs4.prebuilt
>> 3.2.4.linux-i686-py2.7_ucs2.prebuilt
>>
>> The code for bdist_prebuilt is in the mxSetup.py that comes with
>> egenix-mx-base, in case you want to take a look.
> 
> I will take a look.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 16 2012)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2012-08-25: FrOSCon, St. Augustin, Germany ...              9 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/