Platform naming standardization
Hi, I'm involved with a software project that makes extensive use of third party modules with C extensions. To ease installation, we pre-build them for some popular platforms and provide a distribution system for the app to fetch them. setuptools uses distutils.util.get_platform() to decide whether an egg found on the path is compatible, with some special extra magic added in for Mac OS X. Unfortunately, get_platform() as implemented does not provide enough information to ensure binary compatibility. Among the platforms we use, these are the issues: 1) On all platforms, interpreters compiled with UCS2 and UCS4 are binary incompatible, so modules compiled on one will fail to import on the other. We work around this by appending -ucsX to the platform string, based on the value of sys.maxunicode. 2) On OS X, the modification to the value returned by pkg_resources.get_platform() isn't correct for fat version of Python 2.5, as detailed in setuptools issue 19. To solve that, we're using the patch I submitted to the issue (with a couple recent changes). 3) On Solaris (and likely other UNIXes where 32 and 64 bit user spaces coexist), no distinction is made between a 32 bit and 64 bit interpreter, and they are not binary compatible. We work around this by checking sys.maxint, and right now in testing I'm appending -32 or -64 to the platform string (but before -ucsX). I haven't settled on this, though, since I have a feeling maybe it should be part of the arch (sun4u_32, i86pc_64, etc.) or something like 'solaris32' and 'solaris64' instead of 'solaris'. 4) Also on Solaris, the OS is binary compatible with older releases, so Solaris binary distributors will often build on the oldest reasonable release. This is not possible with setuptools now, although extending pkg_resources.compatible_platforms() in pretty much the same manner as is done for OS X solves this (and I can provide a patch). It's not even crucial to me that these be fixed, but before I continue to hack up the platform string, I wanted to ask the SIG to address these issues and hopefully decide on a standard. That way, I can at least implement patches in my app that will be compatible with whatever (if anything) is decided. Thanks, --nate
Hi,
I'm involved with a software project that makes extensive use of third party modules with C extensions...
setuptools uses distutils.util.get_platform() to ... .. .. It's not even crucial to me that these be fixed, but before I continue to hack up the platform string, I wanted to ask the SIG to address these issues and hopefully decide on a standard. That way, I can at least implement patches in my app that will be compatible with whatever (if anything) is decided.
I think you are going to have to hack for now.. In fact, everyone is just having a good old hack at it.. distutils started as a hack.. and it is maybe getting cleaned up now - but it was a hack for such a long time. If you hack.. you'll be in good company. It's what everyone is doing. You're on the right list. :-)
To ease installation, we pre-build them for some popular platforms and provide a distribution system for the app to fetch them.
very modern.. Can you divulge the name of your project? I'd like to look at your hacking.. David
On Tue, Jan 12, 2010 at 7:08 PM, Nate Coraor
Hi,
I'm involved with a software project that makes extensive use of third party modules with C extensions. To ease installation, we pre-build them for some popular platforms and provide a distribution system for the app to fetch them.
setuptools uses distutils.util.get_platform() to decide whether an egg found on the path is compatible, with some special extra magic added in for Mac OS X. Unfortunately, get_platform() as implemented does not provide enough information to ensure binary compatibility.
I think that this has to be discussed at Distutils/stdlib level. Notice that I am currently in the process of moving get_platform() out of Distutils, so it leaves in its own module in the stdlib. [..]
It's not even crucial to me that these be fixed, but before I continue to hack up the platform string, I wanted to ask the SIG to address these issues and hopefully decide on a standard. That way, I can at least implement patches in my app that will be compatible with whatever (if anything) is decided.
Nothing has been decided yet on this. If you provide a patch for get_platform() with a rational, we can start the discussion here and work on a standard, we can then bring it up at python-dev once its ready. Regards Tarek
Hi Tarek, i386 wrote:
setuptools uses distutils.util.get_platform() to decide whether an egg found .. Unfortunately, get_platform() .. compatibility.
I think that this has to be discussed at Distutils/stdlib level. Notice that I am currently in the process of moving get_platform() out of Distutils, so it leaves in its own module in the stdlib.
But that only affects future users.
Nothing has been decided yet on this. If you provide a patch for get_platform() with a rational, we can start the discussion here and work on a standard, we can then bring it up at python-dev once its ready.
I know you can patch python, that's easy for you. Not only that but you are good at it. The point Nate is making though is about better identifying platforms to make build/installation decisions. This use to be simple in metadata.. take a look... ----------------------------------- Metadata-Version: 1.0 Name: tiddlyweb Version: 0.9.96 Summary: An optionally headless, extensible RESTful datastore for tiddlers: bits of stuff. Home-page: http://pypi.python.org/pypi/tiddlyweb Author: Chris Dent Author-email: x@xxxxx License: UNKNOWN Description: TiddlyWeb Home Site: http://tiddlyweb.com/ See also: http://tiddlywiki.org/wiki/TiddlyWeb Platform: Posix; MacOS X; Windows ----------------------------------- What Nate is requesting is specialised build requirements on those particular operating systems. In an easy to do manner. Without resorting to hacks. My thoughts are starting to change for 2010. I want Santa to bring me a new design for a build client for Christmas. I want it web based, I want to add my files. Do a build with a nice graphic. Maybe this new build tool won't even work under python 2.x but only 3. I don't know. I really wish you would start work on a new build client so that we can all jump in and help you. Just being realistic here. Patching distutils can only go so far. Hacking with system api's can only go so far. David
Tarek Ziadé wrote:
It's not even crucial to me that these be fixed, but before I continue to hack up the platform string, I wanted to ask the SIG to address these issues and hopefully decide on a standard. That way, I can at least implement patches in my app that will be compatible with whatever (if anything) is decided.
Nothing has been decided yet on this. If you provide a patch for get_platform() with a rational, we can start the discussion here and work on a standard, we can then bring it up at python-dev once its ready.
I was holding off on creating such a patch until I had some input on what how the platform string should be constructed. And indeed, from other messages in this thread it sounds like hacking up the platform string should not even be the way to do this. I would be happy to provide something, though, if I should go ahead and come up with something that seems reasonable to me. --nate
Nate wrote:
It's not even crucial to me that these be fixed, but before I continue to hack up the platform string, I wanted to ask the SIG to address these issues and hopefully decide on a standard. That way, I can at least implement patches in my app that will be compatible with whatever (if anything) is decided.
Some discussion on ways to achieve different builds was done last year. Lots of things were discussed and a few decisions. I fear the wrong decision about platform naming standardization is going to be made with PEP-345. That is, abandoning traditional description via metadata, and replacing it with the ability to embed python sub-system hacks. It's cheap. And disrespects prior Python Metadata art. Already in Python Metadata, there's a wealth of platform information that could be used and extended to make decisions on platform conditional builds. Thus helping to solve your particular problem. For example, look for the "Platform:" keyword below.. --------------------------------------------------- Metadata-Version: 1.0 Name: tiddlyweb Version: 0.9.96 bits of stuff. Home-page: http://pypi.python.org/pypi/tiddlyweb Author: Chris Dent Description: TiddlyWeb Home Site: http://tiddlyweb.com/ See also: http://tiddlywiki.org/wiki/TiddlyWeb For installation instructions see http://tiddlyweb.peermore.com/ .... Platform: Posix; MacOS X; Windows --------------------------------------------------- Those constants could be extended or reused to do conditional builds. For example, in a setup.cfg, specify: [Build Posix] command=buildit.sh [Build MacOS X] command=buildosx.sh [Build Windows] command=buildw.bat Unfortunately, the guys who make the decisions on the PEPs don't seem to want to extend this way. Perhaps because they (wrongly) think it might lead to a bikeshed. I can't see how it would. This to me (existing art), makes sense:
Platform: Posix; MacOS X; Windows
Not using these terms and resorting to i386 for mac and x86 or i686 for Windows is really strange. We had everything right there in front of us before. Anyway... I looked through Galaxy, and there is no setup.py file. So I think an installation could be totally achievable with via a modified Metadata file looking like this: ----------------------------------------------------
Name: Galaxy Home-page: http://bitbucket.org/galaxy/ Source-Repository: hg:http://bitbucket.org/galaxy/galaxy-dist/ Platform: Posix; MacOS X
and a setup.cfg looking like this: ---------------------------------------------------- [postinstall] script = run.sh ---------------------------------------------------- I wish I could say that I had it working today. I don't unfortunately. I've been trying to learn the mercurial api for checking it out etc etc. I'm a humble hacker doing this for my own enjoyment. And getting paid 50 ruppies from work too. But a whole install of a package as complex as Galaxy impwo could quite easily be done by pushing all the hard work back to mercurial. Then chaining of to your run.sh at the end. My hacking has started on http://bitbucket.org/djlyon/original-setup.py-with-metadata/src/tip/install_... So there's some way to go... Regards David
On Wed, Jan 13, 2010 at 3:08 AM, Nate Coraor
It's not even crucial to me that these be fixed, but before I continue to hack up the platform string, I wanted to ask the SIG to address these issues and hopefully decide on a standard. That way, I can at least implement patches in my app that will be compatible with whatever (if anything) is decided.
I think the only reasonable approach today is to admit that python does not have a well defined ABI (but see PEP 384), and only target well known binary distributions. On mac os x, only target the python from python.org, etc... IMHO, all the attempts at distinguishing between UCS2 vs UCS4, etc... are flawed, because that's only the tip of the iceberg, and it quickly gives a big number of combinations. Cramming information into get_platform to pretend it gives some kind of ABI guarantee does not sound very robust. cheers, David
On Wed, Jan 13, 2010 at 2:01 AM, David Cournapeau
On Wed, Jan 13, 2010 at 3:08 AM, Nate Coraor
wrote: It's not even crucial to me that these be fixed, but before I continue to hack up the platform string, I wanted to ask the SIG to address these issues and hopefully decide on a standard. That way, I can at least implement patches in my app that will be compatible with whatever (if anything) is decided.
I think the only reasonable approach today is to admit that python does not have a well defined ABI (but see PEP 384), and only target well known binary distributions. On mac os x, only target the python from python.org, etc...
IMHO, all the attempts at distinguishing between UCS2 vs UCS4, etc... are flawed, because that's only the tip of the iceberg, and it quickly gives a big number of combinations. Cramming information into get_platform to pretend it gives some kind of ABI guarantee does not sound very robust.
Besides a well-defined ABI, if usc2/usc4 + 32/64 bits distinction on some platforms already fixes a numbers of use cases, I think could worth it for 2.7/3.2
On Wed, Jan 13, 2010 at 10:18 AM, Tarek Ziadé
Besides a well-defined ABI, if usc2/usc4 + 32/64 bits distinction on some platforms already fixes a numbers of use cases, I think could worth it for 2.7/3.2
What bothers me is that get_platform is the wrong function for this - it is also likely that changing its behavior may break existing extensions and build scripts (as get_platform is used to generate some "distributions" names). If a function to qualify an ABI on a specific platform is needed, it should be a new function IMHO. cheers, David
David Cournapeau wrote:
If a function to qualify an ABI on a specific platform is needed, it should be a new function IMHO.
+1, but I do think that this new function should provide the level of detail that Nate is asking for. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Wed, Jan 13, 2010 at 7:17 AM, David Cournapeau
On Wed, Jan 13, 2010 at 10:18 AM, Tarek Ziadé
wrote: Besides a well-defined ABI, if usc2/usc4 + 32/64 bits distinction on some platforms already fixes a numbers of use cases, I think could worth it for 2.7/3.2
What bothers me is that get_platform is the wrong function for this - it is also likely that changing its behavior may break existing extensions and build scripts (as get_platform is used to generate some "distributions" names).
If a function to qualify an ABI on a specific platform is needed, it should be a new function IMHO.
I had a backward compatible mode in mind, maybe with a boolean option like "detailed". Because at the end, If i understand the need correctly, it will be used where get_platform() is used today
David
-- Tarek Ziadé | http://ziade.org
On Wed, Jan 13, 2010 at 5:47 PM, Tarek Ziadé
Because at the end, If i understand the need correctly, it will be used where get_platform() is used today
I don't think that's true - get_platform only returns the platform. It has been used for ABI check by accident, and if you look at the current distutils documentation, nothings hints about ABI compatibility issues. Certainly, the *concepts* of platform (as described in distutils doc) and ABI have nothing to do with each other. As for what the hypothetical function, it should at least contain the python version, UCS 2 vs 4, debug vs release vs other special builds (which are all incompatible with each other, at least on some platforms), the C runtime you link against, the compiler, etc... Just to cite what causes trouble in numpy's case - and I don't see why any sufficiently complicated C extension would not have the same kind of issues. cheers, David
Nate Coraor wrote:
Hi,
I'm involved with a software project that makes extensive use of third party modules with C extensions. To ease installation, we pre-build them for some popular platforms and provide a distribution system for the app to fetch them.
setuptools uses distutils.util.get_platform() to decide whether an egg found on the path is compatible, with some special extra magic added in for Mac OS X. Unfortunately, get_platform() as implemented does not provide enough information to ensure binary compatibility.
I'm not sure whether fixing get_platform() is the right approach - you are mostly talking about different ways to build Python. This API is used (and misused) in a lot of ways, so it's probably better to just create a new API for returning a combined platform and Python version string that correctly identifies the various possible combinations. All the required information is available via the platform module APIs. Some comments:
Among the platforms we use, these are the issues:
1) On all platforms, interpreters compiled with UCS2 and UCS4 are binary incompatible, so modules compiled on one will fail to import on the other. We work around this by appending -ucsX to the platform string, based on the value of sys.maxunicode.
Indeed. This is a major Python property completely ignored by setuptools.
2) On OS X, the modification to the value returned by pkg_resources.get_platform() isn't correct for fat version of Python 2.5, as detailed in setuptools issue 19. To solve that, we're using the patch I submitted to the issue (with a couple recent changes).
I think that using "fat" in the version string is wrong for Mac OS X, since there are many ways to build fat binaries. Instead, the version string should include the details of all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.
3) On Solaris (and likely other UNIXes where 32 and 64 bit user spaces coexist), no distinction is made between a 32 bit and 64 bit interpreter, and they are not binary compatible. We work around this by checking sys.maxint, and right now in testing I'm appending -32 or -64 to the platform string (but before -ucsX). I haven't settled on this, though, since I have a feeling maybe it should be part of the arch (sun4u_32, i86pc_64, etc.) or something like 'solaris32' and 'solaris64' instead of 'solaris'.
Both the processor type and 32/64-bit type are important to have in the string.
4) Also on Solaris, the OS is binary compatible with older releases, so Solaris binary distributors will often build on the oldest reasonable release. This is not possible with setuptools now, although extending pkg_resources.compatible_platforms() in pretty much the same manner as is done for OS X solves this (and I can provide a patch).
Ideally, distutils or platform should provide both an API to create a standard Python and platform identification string as well as an API to check the runtime values against a given string read from e.g. an egg index, ie. platform.get_distribution_platform() platform.compare_distribution_platform()
It's not even crucial to me that these be fixed, but before I continue to hack up the platform string, I wanted to ask the SIG to address these issues and hopefully decide on a standard. That way, I can at least implement patches in my app that will be compatible with whatever (if anything) is decided.
Since most of the issues are with setuptools and/or distribute, it's probably better to ask there. Python eggs with C extensions don't really work well on Unix platforms due to the above restrictions. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 13 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Wednesday, January 13, 2010, at 04:15PM, "M.-A. Lemburg"
2) On OS X, the modification to the value returned by pkg_resources.get_platform() isn't correct for fat version of Python 2.5, as detailed in setuptools issue 19. To solve that, we're using the patch I submitted to the issue (with a couple recent changes).
I think that using "fat" in the version string is wrong for Mac OS X, since there are many ways to build fat binaries.
Instead, the version string should include the details of all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.
Maybe in the long run, but for now "fat" has a well-defined meaning for distutils: fat == ppc + x86_64. There is also a number of other variants, as described in the documentation for distutils. Ronald
Ronald Oussoren wrote:
On Wednesday, January 13, 2010, at 04:15PM, "M.-A. Lemburg"
wrote: 2) On OS X, the modification to the value returned by pkg_resources.get_platform() isn't correct for fat version of Python 2.5, as detailed in setuptools issue 19. To solve that, we're using the patch I submitted to the issue (with a couple recent changes).
I think that using "fat" in the version string is wrong for Mac OS X, since there are many ways to build fat binaries.
Instead, the version string should include the details of all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.
Maybe in the long run, but for now "fat" has a well-defined meaning for distutils: fat == ppc + x86_64. There is also a number of other variants, as described in the documentation for distutils.
I think you meant: fat == ppc + i386. However, it's also possible to build binaries with ppc, i386 and x86_64 - as are shipped with Mac OS X 10.6, so "fat" is not really well-defined and could lead to trying to install 32-bit software for a 64-bit build of Python. IMHO, it's better to list the actually supported architectures as e.g. "darwin-i386-ppc" or "darwin-i386-ppc-x86_64". -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 13 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 13 Jan, 2010, at 18:41, M.-A. Lemburg wrote:
Ronald Oussoren wrote:
On Wednesday, January 13, 2010, at 04:15PM, "M.-A. Lemburg"
wrote: 2) On OS X, the modification to the value returned by pkg_resources.get_platform() isn't correct for fat version of Python 2.5, as detailed in setuptools issue 19. To solve that, we're using the patch I submitted to the issue (with a couple recent changes).
I think that using "fat" in the version string is wrong for Mac OS X, since there are many ways to build fat binaries.
Instead, the version string should include the details of all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.
Maybe in the long run, but for now "fat" has a well-defined meaning for distutils: fat == ppc + x86_64. There is also a number of other variants, as described in the documentation for distutils.
I think you meant: fat == ppc + i386.
Thats right.
However, it's also possible to build binaries with ppc, i386 and x86_64 - as are shipped with Mac OS X 10.6, so "fat" is not really well-defined and could lead to trying to install 32-bit software for a 64-bit build of Python.
"fat" is well-defined for distutils, see the definition of get_platform at http://docs.python.org/distutils/apiref.html. For distutils "fat" is always a universal binary with architectures i386 and ppc, with alternate names for other variants.
IMHO, it's better to list the actually supported architectures as e.g. "darwin-i386-ppc" or "darwin-i386-ppc-x86_64".
That is not how distutils currently works. I also object to "darwin" as a prefix, the platform is named "macosx". Darwin is the opensource unix variant used in OSX and that name shouldn't be interchangeably with macosx. I'm also unhappy that sys.platform is "darwin" on OSX, but it's probably too late to change that. Ronald
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Source (#1, Jan 13 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Ronald Oussoren wrote:
On 13 Jan, 2010, at 18:41, M.-A. Lemburg wrote:
Ronald Oussoren wrote:
On Wednesday, January 13, 2010, at 04:15PM, "M.-A. Lemburg"
wrote: 2) On OS X, the modification to the value returned by pkg_resources.get_platform() isn't correct for fat version of Python 2.5, as detailed in setuptools issue 19. To solve that, we're using the patch I submitted to the issue (with a couple recent changes).
I think that using "fat" in the version string is wrong for Mac OS X, since there are many ways to build fat binaries.
Instead, the version string should include the details of all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.
Maybe in the long run, but for now "fat" has a well-defined meaning for distutils: fat == ppc + x86_64. There is also a number of other variants, as described in the documentation for distutils.
I think you meant: fat == ppc + i386.
Thats right.
However, it's also possible to build binaries with ppc, i386 and x86_64 - as are shipped with Mac OS X 10.6, so "fat" is not really well-defined and could lead to trying to install 32-bit software for a 64-bit build of Python.
"fat" is well-defined for distutils, see the definition of get_platform at http://docs.python.org/distutils/apiref.html.
For distutils "fat" is always a universal binary with architectures i386 and ppc, with alternate names for other variants.
Thanks for pointing that out, however, I don't think that creating aliases for combinations of various different architectures is a good idea. It's better to make the included architectures explicit and use this logic for all platforms, not just Mac OS X.
IMHO, it's better to list the actually supported architectures as e.g. "darwin-i386-ppc" or "darwin-i386-ppc-x86_64".
That is not how distutils currently works. I also object to "darwin" as a prefix, the platform is named "macosx". Darwin is the opensource unix variant used in OSX and that name shouldn't be interchangeably with macosx. I'm also unhappy that sys.platform is "darwin" on OSX, but it's probably too late to change that.
"Darwin" is what Mac OS X itself returns as "uname -s" and that's generally what's being used for sys.platform on Unix systems (configure sets MACHDEP which then gets transmogrified into PLATFORM which then is fed to Py_GetPlatform() which then gets exposed as sys.platform - just wrote that down here, since I just spent half an hour trying to find the definition of PLATFORM...). I agree, though, that the marketing names of the OSes are somewhat more intuitive :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 13 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 13 Jan, 2010, at 23:44, M.-A. Lemburg wrote:
Ronald Oussoren wrote:
On 13 Jan, 2010, at 18:41, M.-A. Lemburg wrote:
Ronald Oussoren wrote:
On Wednesday, January 13, 2010, at 04:15PM, "M.-A. Lemburg"
wrote: 2) On OS X, the modification to the value returned by pkg_resources.get_platform() isn't correct for fat version of Python 2.5, as detailed in setuptools issue 19. To solve that, we're using the patch I submitted to the issue (with a couple recent changes).
I think that using "fat" in the version string is wrong for Mac OS X, since there are many ways to build fat binaries.
Instead, the version string should include the details of all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.
Maybe in the long run, but for now "fat" has a well-defined meaning for distutils: fat == ppc + x86_64. There is also a number of other variants, as described in the documentation for distutils.
I think you meant: fat == ppc + i386.
Thats right.
However, it's also possible to build binaries with ppc, i386 and x86_64 - as are shipped with Mac OS X 10.6, so "fat" is not really well-defined and could lead to trying to install 32-bit software for a 64-bit build of Python.
"fat" is well-defined for distutils, see the definition of get_platform at http://docs.python.org/distutils/apiref.html.
For distutils "fat" is always a universal binary with architectures i386 and ppc, with alternate names for other variants.
Thanks for pointing that out, however, I don't think that creating aliases for combinations of various different architectures is a good idea.
It's better to make the included architectures explicit and use this logic for all platforms, not just Mac OS X.
I would probably have done that, knowing what I know now. Hashing out the details on what combinations of architectures are valid during installation will be fun though ;-). That is, if my python says its machine is "i386,x86_64" is it then acceptable to install an "i386" binary, an "i386,x86_64" binary, and "i386,ppc, x86_64" binary? Ronald
Ronald Oussoren wrote:
On 13 Jan, 2010, at 23:44, M.-A. Lemburg wrote:
Ronald Oussoren wrote:
On 13 Jan, 2010, at 18:41, M.-A. Lemburg wrote:
Ronald Oussoren wrote:
On Wednesday, January 13, 2010, at 04:15PM, "M.-A. Lemburg"
wrote: > 2) On OS X, the modification to the value returned by > pkg_resources.get_platform() isn't correct for fat version of Python > 2.5, as detailed in setuptools issue 19. To solve that, we're using the > patch I submitted to the issue (with a couple recent changes).
I think that using "fat" in the version string is wrong for Mac OS X, since there are many ways to build fat binaries.
Instead, the version string should include the details of all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.
Maybe in the long run, but for now "fat" has a well-defined meaning for distutils: fat == ppc + x86_64. There is also a number of other variants, as described in the documentation for distutils.
I think you meant: fat == ppc + i386.
Thats right.
However, it's also possible to build binaries with ppc, i386 and x86_64 - as are shipped with Mac OS X 10.6, so "fat" is not really well-defined and could lead to trying to install 32-bit software for a 64-bit build of Python.
"fat" is well-defined for distutils, see the definition of get_platform at http://docs.python.org/distutils/apiref.html.
For distutils "fat" is always a universal binary with architectures i386 and ppc, with alternate names for other variants.
Thanks for pointing that out, however, I don't think that creating aliases for combinations of various different architectures is a good idea.
It's better to make the included architectures explicit and use this logic for all platforms, not just Mac OS X.
I would probably have done that, knowing what I know now.
Hashing out the details on what combinations of architectures are valid during installation will be fun though ;-). That is, if my python says its machine is "i386,x86_64" is it then acceptable to install an "i386" binary, an "i386,x86_64" binary, and "i386,ppc, x86_64" binary?
The point is that even though your Python binary may say it's "i386,x86_64", the version you run your application with will either be "i386" or "x86_64" (depending on the OS environment settings). Now let's say you're running the "i386" version. As long as all installed components provide the "i386" part you should be fine. In your example all components provide the "i386" part, so all of them can be installed. With the aliases, this kind of detection is also possible, but only after mapping the aliases back to the combination of included architecture names. In a few years, we'll probably only see "x86_64" binaries for Mac OS, but until then, package installers will have to be able to work out the problem of finding installable distribution files among the available ones. BTW: With Python 2.6, if you build using the x86_64 version of Python, distutils will still use the "macosx-10.5-i386" platform identifier. Should I file a bug for this ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 14 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 14 Jan, 2010, at 10:36, M.-A. Lemburg wrote:
Ronald Oussoren wrote:
On 13 Jan, 2010, at 23:44, M.-A. Lemburg wrote:
Ronald Oussoren wrote:
On 13 Jan, 2010, at 18:41, M.-A. Lemburg wrote:
Ronald Oussoren wrote:
On Wednesday, January 13, 2010, at 04:15PM, "M.-A. Lemburg"
wrote: > >> 2) On OS X, the modification to the value returned by >> pkg_resources.get_platform() isn't correct for fat version of Python >> 2.5, as detailed in setuptools issue 19. To solve that, we're using the >> patch I submitted to the issue (with a couple recent changes). > > I think that using "fat" in the version string is wrong for > Mac OS X, since there are many ways to build fat binaries. > > Instead, the version string should include the details of > all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.
Maybe in the long run, but for now "fat" has a well-defined meaning for distutils: fat == ppc + x86_64. There is also a number of other variants, as described in the documentation for distutils.
I think you meant: fat == ppc + i386.
Thats right.
However, it's also possible to build binaries with ppc, i386 and x86_64 - as are shipped with Mac OS X 10.6, so "fat" is not really well-defined and could lead to trying to install 32-bit software for a 64-bit build of Python.
"fat" is well-defined for distutils, see the definition of get_platform at http://docs.python.org/distutils/apiref.html.
For distutils "fat" is always a universal binary with architectures i386 and ppc, with alternate names for other variants.
Thanks for pointing that out, however, I don't think that creating aliases for combinations of various different architectures is a good idea.
It's better to make the included architectures explicit and use this logic for all platforms, not just Mac OS X.
I would probably have done that, knowing what I know now.
Hashing out the details on what combinations of architectures are valid during installation will be fun though ;-). That is, if my python says its machine is "i386,x86_64" is it then acceptable to install an "i386" binary, an "i386,x86_64" binary, and "i386,ppc, x86_64" binary?
The point is that even though your Python binary may say it's "i386,x86_64", the version you run your application with will either be "i386" or "x86_64" (depending on the OS environment settings).
Now let's say you're running the "i386" version. As long as all installed components provide the "i386" part you should be fine. In your example all components provide the "i386" part, so all of them can be installed.
I don't agree, "easy_install somepackage" should install a component that supports at least all architectures supported by the Python binary. Otherwise you might install a package and have problems later when you try to use it. An example of this is a recent 64-bit capable machine with older versions of Tkinter or wxPython: on those systems python will run as a 64-bit binary by default, but you sometimes have to run python in 32-bit mode to be able to use Tkinter. It would be very annoying and possibly confusing when I install a library and end up not being able to use it sometimes. I also regularly build standalone app bundles on one machine and run them on other machines, those should also included all components in all supported architectures.
With the aliases, this kind of detection is also possible, but only after mapping the aliases back to the combination of included architecture names.
In a few years, we'll probably only see "x86_64" binaries for Mac OS, but until then, package installers will have to be able to work out the problem of finding installable distribution files among the available ones.
I agree, at least until the next new thing comes along (such as something arm-based).
BTW: With Python 2.6, if you build using the x86_64 version of Python, distutils will still use the "macosx-10.5-i386" platform identifier. Should I file a bug for this ?
That should be fixed in the repository, distutils assumed that the "uname -m" reflects the correct architecture and that is not true for a single-architecture build with default compiler flags on OSX 10.6. Ronald
Ronald Oussoren wrote:
["fat" binaries on Mac OS X]
It's better to make the included architectures explicit and use this logic for all platforms, not just Mac OS X.
I would probably have done that, knowing what I know now.
Hashing out the details on what combinations of architectures are valid during installation will be fun though ;-). That is, if my python says its machine is "i386,x86_64" is it then acceptable to install an "i386" binary, an "i386,x86_64" binary, and "i386,ppc, x86_64" binary?
The point is that even though your Python binary may say it's "i386,x86_64", the version you run your application with will either be "i386" or "x86_64" (depending on the OS environment settings).
Now let's say you're running the "i386" version. As long as all installed components provide the "i386" part you should be fine. In your example all components provide the "i386" part, so all of them can be installed.
I don't agree, "easy_install somepackage" should install a component that supports at least all architectures supported by the Python binary. Otherwise you might install a package and have problems later when you try to use it.
An example of this is a recent 64-bit capable machine with older versions of Tkinter or wxPython: on those systems python will run as a 64-bit binary by default, but you sometimes have to run python in 32-bit mode to be able to use Tkinter. It would be very annoying and possibly confusing when I install a library and end up not being able to use it sometimes.
I also regularly build standalone app bundles on one machine and run them on other machines, those should also included all components in all supported architectures.
A package manager should of course try to install the best match per default and best match should probably mean: use the binary that supports all architecture parts supported by the currently running Python binary. However, it is well possible that some binary packages do not come in all combinations supported by the Python binary, but do come in the variant currently running. In such a case, the user should still be able to install the binary package - perhaps with a warning that the installed version won't run on some other supported variants. With the current aliasing of architecture combinations this won't be possible, so the user won't be able to install an ppc/i386 egg, even if she only ever uses the i386 part of a ppc/i386/x64_86 Python binary. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 18 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, 14 Jan 2010 10:36:46 +0100, "M.-A. Lemburg"
> Instead, the version string should include the details of > all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.
Hashing out the details on what combinations of architectures are valid during installation will be fun though ;-). That is, if my python says its machine is "i386,x86_64" is it then acceptable to install an "i386" binary, an "i386,x86_64" binary, and "i386,ppc, x86_64" binary?
The point is that even though your Python binary may say it's "i386,x86_64", the version you run your application with will either be "i386" or "x86_64" (depending on the OS environment settings).
That's right.
Now let's say you're running the "i386" version. As long as all installed components provide the "i386" part you should be fine. In your example all components provide the "i386" part, so all of them can be installed.
The main point here though is being able to trigger the correct build for non-standard configuration, no matter what the platform. We're having the same issue on Windows as it is running the same processors.
In a few years, we'll probably only see "x86_64" binaries for Mac OS, but until then, package installers will have to be able to work out the problem of finding installable distribution files among the available ones.
We'll see what if anything gets changed/fixed. I'm concerned about the excessive use of code-freeze-spray being used in 2010. That's where you take 5-10 year old code and just freeze it into place. I'd really like to see distutils create packages and run in a browser window. That's my wish to santa. Having a 64-bit processor and a 10 year old command line interface on distutils, seems somewhat wrong in this millenia. There really is so much to do.. we could make a really fresh start with Python 3 and make it really different and better than Python 2.x I'd love to see a plan or a roadmap for packaging for 2010.. David
David Lyon wrote:
On Thu, 14 Jan 2010 10:36:46 +0100, "M.-A. Lemburg"
wrote: >> Instead, the version string should include the details of >> all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'.
Hashing out the details on what combinations of architectures are valid during installation will be fun though ;-). That is, if my python says its machine is "i386,x86_64" is it then acceptable to install an "i386" binary, an "i386,x86_64" binary, and "i386,ppc, x86_64" binary? The point is that even though your Python binary may say it's "i386,x86_64", the version you run your application with will either be "i386" or "x86_64" (depending on the OS environment settings).
That's right.
Now let's say you're running the "i386" version. As long as all installed components provide the "i386" part you should be fine. In your example all components provide the "i386" part, so all of them can be installed.
The main point here though is being able to trigger the correct build for non-standard configuration, no matter what the platform. We're having the same issue on Windows as it is running the same processors.
In a few years, we'll probably only see "x86_64" binaries for Mac OS, but until then, package installers will have to be able to work out the problem of finding installable distribution files among the available ones.
We'll see what if anything gets changed/fixed.
I'm concerned about the excessive use of code-freeze-spray being used in 2010. That's where you take 5-10 year old code and just freeze it into place.
I'd really like to see distutils create packages and run in a browser window. That's my wish to santa.
Having a 64-bit processor and a 10 year old command line interface on distutils, seems somewhat wrong in this millenia.
There really is so much to do.. we could make a really fresh start with Python 3 and make it really different and better than Python 2.x
I'd love to see a plan or a roadmap for packaging for 2010..
I'd hate to see this only end up in 3. We'll probably be supporting 2.x for a long time. Are we any closer to a plan? =) I just found another: (g)libc version. --nate
David
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Nate wrote:
I'd hate to see this only end up in 3. We'll probably be supporting 2.x for a long time.
Are we any closer to a plan? =) I just found another: (g)libc version.
Well everyday we get closer to seeing the roadmap.. I can give an official answer on that. See: http://us.pycon.org/2010/conference/talks/?search=packaging#63.%20%20The%20s... Here my friend Tarek will speak on the distutils revamp that I have been suggesting. If you can get along, it will be an extremely interesting session.. David
participants (7)
-
Chris Withers
-
David Cournapeau
-
David Lyon
-
M.-A. Lemburg
-
Nate Coraor
-
Ronald Oussoren
-
Tarek Ziadé