[Distutils] Platform naming standardization

[M.-A. Lemburg]

Nate Coraor wrote:
> Hi,
> 
> I'm involved with a software project that makes extensive use of third
> party modules with C extensions.  To ease installation, we pre-build
> them for some popular platforms and provide a distribution system for
> the app to fetch them.
> 
> setuptools uses distutils.util.get_platform() to decide whether an egg
> found on the path is compatible, with some special extra magic added in
> for Mac OS X.  Unfortunately, get_platform() as implemented does not
> provide enough information to ensure binary compatibility.

I'm not sure whether fixing get_platform() is the right approach -
you are mostly talking about different ways to build Python. This
API is used (and misused) in a lot of ways, so it's probably
better to just create a new API for returning a combined platform
and Python version string that correctly identifies the various
possible combinations.

All the required information is available via the platform
module APIs.

Some comments:

> Among the platforms we use, these are the issues:
> 
> 1) On all platforms, interpreters compiled with UCS2 and UCS4 are binary
> incompatible, so modules compiled on one will fail to import on the
> other.  We work around this by appending -ucsX to the platform string,
> based on the value of sys.maxunicode.

Indeed. This is a major Python property completely ignored by setuptools.

> 2) On OS X, the modification to the value returned by
> pkg_resources.get_platform() isn't correct for fat version of Python
> 2.5, as detailed in setuptools issue 19.  To solve that, we're using the
> patch I submitted to the issue (with a couple recent changes).

I think that using "fat" in the version string is wrong for
Mac OS X, since there are many ways to build fat binaries.

Instead, the version string should include the details of
all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.

> 3) On Solaris (and likely other UNIXes where 32 and 64 bit user spaces
> coexist), no distinction is made between a 32 bit and 64 bit
> interpreter, and they are not binary compatible.  We work around this by
> checking sys.maxint, and right now in testing I'm appending -32 or -64
> to the platform string (but before -ucsX).  I haven't settled on this,
> though, since I have a feeling maybe it should be part of the arch
> (sun4u_32, i86pc_64, etc.) or something like 'solaris32' and 'solaris64'
> instead of 'solaris'.

Both the processor type and 32/64-bit type are important to
have in the string.

> 4) Also on Solaris, the OS is binary compatible with older releases, so
> Solaris binary distributors will often build on the oldest reasonable
> release.  This is not possible with setuptools now, although extending
> pkg_resources.compatible_platforms() in pretty much the same manner as
> is done for OS X solves this (and I can provide a patch).

Ideally, distutils or platform should provide both an API
to create a standard Python and platform identification string
as well as an API to check the runtime values against a given
string read from e.g. an egg index, ie.

platform.get_distribution_platform()
platform.compare_distribution_platform()

> It's not even crucial to me that these be fixed, but before I continue
> to hack up the platform string, I wanted to ask the SIG to address these
> issues and hopefully decide on a standard.  That way, I can at least
> implement patches in my app that will be compatible with whatever (if
> anything) is decided.

Since most of the issues are with setuptools and/or distribute,
it's probably better to ask there.

Python eggs with C extensions don't really work well on Unix
platforms due to the above restrictions.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 13 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/