[setuptools] setuptools and fat binaries
Hi, Apple supports fat binaries on Mac OS X (they call them universal binaries), that is binaries that contain executable code for multiple architectures. In released version of the os this can be used to build binaries and libraries that support both PPC and PPC64 architectures, which isn't used very much, and in the future this will be the mechanism to support both the current PPC based systems and the to-be-introduced intel systems with a single binary. I'm currently playing around with Python on an Intel developer system (my goal is a patch that will make it easy to build a fat/universal build of Python on Mac OS X), and I wondering how setuptools (and specifically eggs) can support universal binaries. The reason I ask is because of eggs that contain extensions, whose name contains the architecture, such as aem-0.10.0-py2.4-macosx-10.4-ppc.egg. Just using two eggs, one for each architecture, would work but seems wasteful and would make it harder to create self fat application bundles. Ronald
On Jan 4, 2006, at 12:51 PM, Ronald Oussoren wrote:
Apple supports fat binaries on Mac OS X (they call them universal binaries), that is binaries that contain executable code for multiple architectures. In released version of the os this can be used to build binaries and libraries that support both PPC and PPC64 architectures, which isn't used very much, and in the future this will be the mechanism to support both the current PPC based systems and the to-be-introduced intel systems with a single binary.
I'm currently playing around with Python on an Intel developer system (my goal is a patch that will make it easy to build a fat/universal build of Python on Mac OS X), and I wondering how setuptools (and specifically eggs) can support universal binaries. The reason I ask is because of eggs that contain extensions, whose name contains the architecture, such as aem-0.10.0-py2.4-macosx-10.4-ppc.egg. Just using two eggs, one for each architecture, would work but seems wasteful and would make it harder to create self fat application bundles.
Using two or three eggs doesn't really make it any harder to create an application bundle, py2app needs to change to support eggs anyway and having a separate arch egg would work fine. It might be better that way because it's easy to see which packages support which architectures and it's easy for someone who has the relevant platform to built, test, and upload. py2app could even merge arch eggs together to make them fat at build time. The only thing that's off is that easy_install would need to fetch eggs for all architectures... maybe a pydistutils.cfg that says you want to fetch "ppc ppc64 x86". -bob
On 4-jan-2006, at 22:26, Bob Ippolito wrote:
On Jan 4, 2006, at 12:51 PM, Ronald Oussoren wrote:
Apple supports fat binaries on Mac OS X (they call them universal binaries), that is binaries that contain executable code for multiple architectures. In released version of the os this can be used to build binaries and libraries that support both PPC and PPC64 architectures, which isn't used very much, and in the future this will be the mechanism to support both the current PPC based systems and the to-be-introduced intel systems with a single binary.
I'm currently playing around with Python on an Intel developer system (my goal is a patch that will make it easy to build a fat/universal build of Python on Mac OS X), and I wondering how setuptools (and specifically eggs) can support universal binaries. The reason I ask is because of eggs that contain extensions, whose name contains the architecture, such as aem-0.10.0-py2.4-macosx-10.4-ppc.egg. Just using two eggs, one for each architecture, would work but seems wasteful and would make it harder to create self fat application bundles.
Using two or three eggs doesn't really make it any harder to create an application bundle, py2app needs to change to support eggs anyway and having a separate arch egg would work fine. It might be better that way because it's easy to see which packages support which architectures and it's easy for someone who has the relevant platform to built, test, and upload.
Someone, somewhere has to do the work, either setuptools or py2app. I'd like to see fat eggs to avoid duplication of all non-executable stuff (python files and data files). I'd guess that in most cases just using '-arch i386 -arch ppc' in the extra_compile_args and extra_link_args would enable you to build a fat version of an extension. Building a fat egg that works on OSX 10.3 and forward might be harder, but even there the simple solution might work most of the time (I haven't put much thought into that yet).
py2app could even merge arch eggs together to make them fat at build time.
If it would do that setuptools would still require some support for fat eggs, or do you mean that py2app would merge the two variants and then have a symlink from one architecture variant to the other?
The only thing that's off is that easy_install would need to fetch eggs for all architectures... maybe a pydistutils.cfg that says you want to fetch "ppc ppc64 x86".
-bob
On Jan 4, 2006, at 1:38 PM, Ronald Oussoren wrote:
On 4-jan-2006, at 22:26, Bob Ippolito wrote:
On Jan 4, 2006, at 12:51 PM, Ronald Oussoren wrote:
Apple supports fat binaries on Mac OS X (they call them universal binaries), that is binaries that contain executable code for multiple architectures. In released version of the os this can be used to build binaries and libraries that support both PPC and PPC64 architectures, which isn't used very much, and in the future this will be the mechanism to support both the current PPC based systems and the to-be-introduced intel systems with a single binary.
I'm currently playing around with Python on an Intel developer system (my goal is a patch that will make it easy to build a fat/universal build of Python on Mac OS X), and I wondering how setuptools (and specifically eggs) can support universal binaries. The reason I ask is because of eggs that contain extensions, whose name contains the architecture, such as aem-0.10.0-py2.4-macosx-10.4-ppc.egg. Just using two eggs, one for each architecture, would work but seems wasteful and would make it harder to create self fat application bundles.
Using two or three eggs doesn't really make it any harder to create an application bundle, py2app needs to change to support eggs anyway and having a separate arch egg would work fine. It might be better that way because it's easy to see which packages support which architectures and it's easy for someone who has the relevant platform to built, test, and upload.
Someone, somewhere has to do the work, either setuptools or py2app. I'd like to see fat eggs to avoid duplication of all non-executable stuff (python files and data files).
Work has to be done in both places, either way.
I'd guess that in most cases just using '-arch i386 -arch ppc' in the extra_compile_args and extra_link_args would enable you to build a fat version of an extension. Building a fat egg that works on OSX 10.3 and forward might be harder, but even there the simple solution might work most of the time (I haven't put much thought into that yet).
Aren't there still pyconfig.h issues with doing a fat build like that? What about getting ppc64 support in a fat build?
py2app could even merge arch eggs together to make them fat at build time.
If it would do that setuptools would still require some support for fat eggs, or do you mean that py2app would merge the two variants and then have a symlink from one architecture variant to the other?
There are lots of ways it could work. It doesn't really need any support of fat eggs, it could just remove the platform spec. -bob
On 4-jan-2006, at 22:48, Bob Ippolito wrote:
On Jan 4, 2006, at 1:38 PM, Ronald Oussoren wrote:
On 4-jan-2006, at 22:26, Bob Ippolito wrote:
On Jan 4, 2006, at 12:51 PM, Ronald Oussoren wrote:
Apple supports fat binaries on Mac OS X (they call them universal binaries), that is binaries that contain executable code for multiple architectures. In released version of the os this can be used to build binaries and libraries that support both PPC and PPC64 architectures, which isn't used very much, and in the future this will be the mechanism to support both the current PPC based systems and the to-be-introduced intel systems with a single binary.
I'm currently playing around with Python on an Intel developer system (my goal is a patch that will make it easy to build a fat/universal build of Python on Mac OS X), and I wondering how setuptools (and specifically eggs) can support universal binaries. The reason I ask is because of eggs that contain extensions, whose name contains the architecture, such as aem-0.10.0-py2.4-macosx-10.4-ppc.egg. Just using two eggs, one for each architecture, would work but seems wasteful and would make it harder to create self fat application bundles.
Using two or three eggs doesn't really make it any harder to create an application bundle, py2app needs to change to support eggs anyway and having a separate arch egg would work fine. It might be better that way because it's easy to see which packages support which architectures and it's easy for someone who has the relevant platform to built, test, and upload.
Someone, somewhere has to do the work, either setuptools or py2app. I'd like to see fat eggs to avoid duplication of all non- executable stuff (python files and data files).
Work has to be done in both places, either way.
I'd guess that in most cases just using '-arch i386 -arch ppc' in the extra_compile_args and extra_link_args would enable you to build a fat version of an extension. Building a fat egg that works on OSX 10.3 and forward might be harder, but even there the simple solution might work most of the time (I haven't put much thought into that yet).
Aren't there still pyconfig.h issues with doing a fat build like that? What about getting ppc64 support in a fat build?
My current hack is to include macosx-config.h at the end of pyconfig.h and to redefine all processor specific defines in that file. That seems to allow me to build a fat framework using 'OPT=- arch i386 -arch ppc LDFLAGS=-arch i386 -arch ppc'. Well, actually that and a patch to Makefile.pre.in to call libtool in the right way. But, I haven't tested the result on a PPC box yet, I'm hunting down i386 specific bugs first (probably byte-order related). Adding ppc64 support would require 3 seperate builds + a merge because not all extensions build on the ppc64 flavor, the entire Carbon package cannot be build for ppc64. I don't have a G5 box, and probably would work on this even if I had. Making it possible to build a 10.3 capable python binary on 10.4 is more useful.
py2app could even merge arch eggs together to make them fat at build time.
If it would do that setuptools would still require some support for fat eggs, or do you mean that py2app would merge the two variants and then have a symlink from one architecture variant to the other?
There are lots of ways it could work. It doesn't really need any support of fat eggs, it could just remove the platform spec.
-bob
P.S. the ruckus(sp?) about a compiler warning getpath.c over on python-dev is quite funny given the amount of warnings I'm getting in other parts of the build, although most of those are in Carbon package.
participants (2)
-
Bob Ippolito -
Ronald Oussoren