Which Python to use for Mac binaries
Hi, Currently the NumPy binaries are built using the pavement.py script, which uses the following Pythons: MPKG_PYTHON = { "2.5": ["/Library/Frameworks/Python.framework/Versions/2.5/bin/python"], "2.6": ["/Library/Frameworks/Python.framework/Versions/2.6/bin/python"], "2.7": ["/Library/Frameworks/Python.framework/Versions/2.7/bin/python"], "3.1": ["/Library/Frameworks/Python.framework/Versions/3.1/bin/python3"], "3.2": ["/Library/Frameworks/Python.framework/Versions/3.2/bin/python3"], "3.3": ["/Library/Frameworks/Python.framework/Versions/3.3/bin/python3"], } So for example I can easily create the 2.6 binary if that Python is pre-installed on the Mac box that I am using. On one of the Mac boxes that I am using, the 2.7 is missing, so are 3.1, 3.2 and 3.3. So I was thinking of updating my Fabric fab file to automatically install all Pythons from source and build against that, just like I do for Wine. Which exact Python do we need to use on Mac? Do we need to use the binary installer from python.org? Or can I install it from source? Finally, for which Python versions should we provide binary installers for Mac? For reference, the 1.6.2 had installers for 2.5, 2.6 and 2.7 only for OS X 10.3. There is only 2.7 version for OS X 10.6. Also, what is the meaning of the following piece of code in pavement.py: def _build_mpkg(pyver): # account for differences between Python 2.7.1 versions from python.org if os.environ.get('MACOSX_DEPLOYMENT_TARGET', None) == "10.6": ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch x86_64 -Wl,-search_paths_first" else: ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first" ldflags += " -L%s" % os.path.join(os.path.dirname(__file__), "build") if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) In particular, the last line gets executed and it then fails with: paver dmg -p 2.6 ---> pavement.dmg ---> pavement.clean LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first -Lbuild' /Library/Frameworks/Python.framework/Versions/2.6/bin/python setupegg.py bdist_mpkg Traceback (most recent call last): File "setupegg.py", line 17, in <module> from setuptools import setup ImportError: No module named setuptools The reason is (I think) that if the Python binary is called explicitly with /Library/Frameworks/Python.framework/Versions/2.6/bin/python, then the paths are not setup properly in virtualenv, and thus setuptools (which is only installed in virtualenv, but not in system Python) fails to import. The solution is to simply apply this patch: diff --git a/pavement.py b/pavement.py index e693016..0c637f8 100644 --- a/pavement.py +++ b/pavement.py @@ -449,7 +449,7 @@ def _build_mpkg(pyver): if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: - sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) + sh("python setupegg.py bdist_mpkg") @task def simple_dmg(): and then things work. So an obvious question is --- why do we need to fiddle with LDFLAGS and paths to the exact Python version? Here is a proposed simpler version of the build_mpkg() function: def _build_mpkg(pyver): sh("python setupegg.py bdist_mpkg") Thanks for any tips. Ondrej
On Sun, Jan 6, 2013 at 3:21 AM, Ondřej Čertík
Hi,
Currently the NumPy binaries are built using the pavement.py script, which uses the following Pythons:
MPKG_PYTHON = { "2.5": ["/Library/Frameworks/Python.framework/Versions/2.5/bin/python"], "2.6": ["/Library/Frameworks/Python.framework/Versions/2.6/bin/python"], "2.7": ["/Library/Frameworks/Python.framework/Versions/2.7/bin/python"], "3.1": ["/Library/Frameworks/Python.framework/Versions/3.1/bin/python3"], "3.2": ["/Library/Frameworks/Python.framework/Versions/3.2/bin/python3"], "3.3": ["/Library/Frameworks/Python.framework/Versions/3.3/bin/python3"], }
So for example I can easily create the 2.6 binary if that Python is pre-installed on the Mac box that I am using. On one of the Mac boxes that I am using, the 2.7 is missing, so are 3.1, 3.2 and 3.3. So I was thinking of updating my Fabric fab file to automatically install all Pythons from source and build against that, just like I do for Wine.
Which exact Python do we need to use on Mac? Do we need to use the binary installer from python.org?
Yes, the one from python.org.
Or can I install it from source? Finally, for which Python versions should we provide binary installers for Mac? For reference, the 1.6.2 had installers for 2.5, 2.6 and 2.7 only for OS X 10.3. There is only 2.7 version for OS X 10.6.
The provided installers and naming scheme should match what's done for Python itself on python.org. The 10.3 installers for 2.5, 2.6 and 2.7 should be compiled on OS X 10.5. This is kind of hard to come by these days, but Vincent Davis maintains a build machine for numpy and scipy. That's already set up correctly, so all you have to do is connect to it via ssh, check out v.17.0 in ~/Code/numpy, check in release.sh that the section for OS X 10.6 is disabled and for 10.5 enabled and run it. OS X 10.6 broke support for previous versions in some subtle ways, so even when using the 10.4 SDK numpy compiled on 10.6 won't run on 10.5. As long as we're supporting 10.5 you therefore need to compile on it. The 10.7 --> 10.6 support hasn't been checked, but I wouldn't trust it. I have a 10.6 machine, so I can compile those binaries if needed.
Also, what is the meaning of the following piece of code in pavement.py:
def _build_mpkg(pyver): # account for differences between Python 2.7.1 versions from python.org if os.environ.get('MACOSX_DEPLOYMENT_TARGET', None) == "10.6": ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch x86_64 -Wl,-search_paths_first" else: ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first" ldflags += " -L%s" % os.path.join(os.path.dirname(__file__), "build")
The 10.6 binaries support only Intel Macs, both 32-bit and 64-bit. The 10.3 binaries support PPC Macs and 32-bit Intel. That's what the above does. Note that we simply follow the choice made by the Python release managers here.
if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver])))
This is necessary because in Python 2.5, distutils asks for "gcc" instead of "gcc-4.0", so you may get the wrong one without CC=gcc-4.0. From Python 2.6 on this was fixed.
In particular, the last line gets executed and it then fails with:
paver dmg -p 2.6 ---> pavement.dmg ---> pavement.clean LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first -Lbuild' /Library/Frameworks/Python.framework/Versions/2.6/bin/python setupegg.py bdist_mpkg Traceback (most recent call last): File "setupegg.py", line 17, in <module> from setuptools import setup ImportError: No module named setuptools
The reason is (I think) that if the Python binary is called explicitly with /Library/Frameworks/Python.framework/Versions/2.6/bin/python, then the paths are not setup properly in virtualenv, and thus setuptools (which is only installed in virtualenv, but not in system Python) fails to import. The solution is to simply apply this patch:
Avoid using system Python for anything. The first thing to do on any new OS X system is install Python some other way, preferably from python.org.
diff --git a/pavement.py b/pavement.py index e693016..0c637f8 100644 --- a/pavement.py +++ b/pavement.py @@ -449,7 +449,7 @@ def _build_mpkg(pyver): if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: - sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) + sh("python setupegg.py bdist_mpkg")
This doesn't work unless using virtualenvs, you're just throwing away the version selection here. If you can support virtualenvs in addition to python.org pythons, that would be useful. But being able to build binaries when needed simply by "paver dmg -p 2.x" is quite useful.
@task def simple_dmg():
and then things work. So an obvious question is --- why do we need to fiddle with LDFLAGS and paths to the exact Python version? Here is a proposed simpler version of the build_mpkg() function:
def _build_mpkg(pyver): sh("python setupegg.py bdist_mpkg")
Thanks for any tips.
Did you see the release.sh script? Some of the answers to your questions were already documented there, and it should do the job out of the box. Last note: bdist_mpkg is unmaintained and doesn't support Python 3.x. Most recent version is at: https://github.com/matthew-brett/bdist_mpkg, for previous versions numpy releases I've used that at commit e81a58a471https://github.com/rgommers/bdist_mpkg/commit/e81a58a47120522fc0e9374d2d2cf5... If we want 3.x binaries, then we should fix that or (preferably) build binaries with Bento. Bento has grown support for mpkg's; I'm not sure how robust that is. Cheers, Ralf
On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers
Which exact Python do we need to use on Mac? Do we need to use the binary installer from python.org?
Yes, the one from python.org.
Or can I install it from source?
you could install from source using the same method that the python.org binaries are built -- there is a script with the source to do that, though I'm not sure what the point of that would be.
The 10.3 installers for 2.5, 2.6 and 2.7 should be compiled on OS X 10.5.
It would be great to continue support for that, though I wonder how many people still need it -- I don't think Apple supports 10.5 anymore, for instance.
The 10.7 --> 10.6 support hasn't been checked, but I wouldn't trust it. I have a 10.6 machine, so I can compile those binaries if needed.
That would be better, but it would also be nice to check how building on 10.7 works.
Avoid using system Python for anything. The first thing to do on any new OS X system is install Python some other way, preferably from python.org.
+1
Last note: bdist_mpkg is unmaintained and doesn't support Python 3.x. Most recent version is at: https://github.com/matthew-brett/bdist_mpkg, for previous versions numpy releases I've used that at commit e81a58a471
There has been recent discussion on the pythonmac list about this -- some waffling about how important it is -- though I think it would be good to keep it up to date.
If we want 3.x binaries, then we should fix that or (preferably) build binaries with Bento. Bento has grown support for mpkg's; I'm not sure how robust that is.
So maybe bento is a better route than bdist_mpkg -- this is worth discussion on teh pythonmac list. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Hi,
On Sun, Jan 6, 2013 at 10:40 PM, Chris Barker - NOAA Federal
On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers
wrote: Which exact Python do we need to use on Mac? Do we need to use the binary installer from python.org?
Yes, the one from python.org.
Or can I install it from source?
you could install from source using the same method that the python.org binaries are built -- there is a script with the source to do that, though I'm not sure what the point of that would be.
The 10.3 installers for 2.5, 2.6 and 2.7 should be compiled on OS X 10.5.
It would be great to continue support for that, though I wonder how many people still need it -- I don't think Apple supports 10.5 anymore, for instance.
The 10.7 --> 10.6 support hasn't been checked, but I wouldn't trust it. I have a 10.6 machine, so I can compile those binaries if needed.
That would be better, but it would also be nice to check how building on 10.7 works.
Avoid using system Python for anything. The first thing to do on any new OS X system is install Python some other way, preferably from python.org.
+1
Last note: bdist_mpkg is unmaintained and doesn't support Python 3.x. Most recent version is at: https://github.com/matthew-brett/bdist_mpkg, for previous versions numpy releases I've used that at commit e81a58a471
There has been recent discussion on the pythonmac list about this -- some waffling about how important it is -- though I think it would be good to keep it up to date.
I updated my fork of bdist_mpkg with Python 3k support. It doesn't have any tests that I could see, but I've run it on python 2.6 and 3.2 and 3.3 on one of my packages as a first pass.
If we want 3.x binaries, then we should fix that or (preferably) build binaries with Bento. Bento has grown support for mpkg's; I'm not sure how robust that is.
So maybe bento is a better route than bdist_mpkg -- this is worth discussion on teh pythonmac list.
David - can you give a status update on that? Cheers, Matthew
On Mon, Jan 7, 2013 at 5:31 AM, Matthew Brett
I updated my fork of bdist_mpkg with Python 3k support. It doesn't have any tests that I could see, but I've run it on python 2.6 and 3.2 and 3.3 on one of my packages as a first pass.
Have you been in communication with Ronald Oussoren about this? I'm sure he'd be interested in bringing into the "official"repository. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Hi,
On Mon, Jan 7, 2013 at 5:48 PM, Chris Barker - NOAA Federal
On Mon, Jan 7, 2013 at 5:31 AM, Matthew Brett
wrote: I updated my fork of bdist_mpkg with Python 3k support. It doesn't have any tests that I could see, but I've run it on python 2.6 and 3.2 and 3.3 on one of my packages as a first pass.
Have you been in communication with Ronald Oussoren about this? I'm sure he'd be interested in bringing into the "official"repository.
I just emailed him, thanks for the suggestion. Best, Matthew
On Mon, Jan 7, 2013 at 7:31 AM, Matthew Brett
Hi,
On Sun, Jan 6, 2013 at 10:40 PM, Chris Barker - NOAA Federal
wrote: On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers
wrote: Which exact Python do we need to use on Mac? Do we need to use the binary installer from python.org?
Yes, the one from python.org.
Or can I install it from source?
you could install from source using the same method that the python.org binaries are built -- there is a script with the source to do that, though I'm not sure what the point of that would be.
The 10.3 installers for 2.5, 2.6 and 2.7 should be compiled on OS X 10.5.
It would be great to continue support for that, though I wonder how many people still need it -- I don't think Apple supports 10.5 anymore, for instance.
The 10.7 --> 10.6 support hasn't been checked, but I wouldn't trust it. I have a 10.6 machine, so I can compile those binaries if needed.
That would be better, but it would also be nice to check how building on 10.7 works.
Avoid using system Python for anything. The first thing to do on any new OS X system is install Python some other way, preferably from python.org.
+1
Last note: bdist_mpkg is unmaintained and doesn't support Python 3.x. Most recent version is at: https://github.com/matthew-brett/bdist_mpkg, for previous versions numpy releases I've used that at commit e81a58a471
There has been recent discussion on the pythonmac list about this -- some waffling about how important it is -- though I think it would be good to keep it up to date.
I updated my fork of bdist_mpkg with Python 3k support. It doesn't have any tests that I could see, but I've run it on python 2.6 and 3.2 and 3.3 on one of my packages as a first pass.
If we want 3.x binaries, then we should fix that or (preferably) build binaries with Bento. Bento has grown support for mpkg's; I'm not sure how robust that is.
So maybe bento is a better route than bdist_mpkg -- this is worth discussion on teh pythonmac list.
David - can you give a status update on that?
It is more a starting point than anything else, and barely tested. I would advise against using it ATM. thanks, David
On Sun, Jan 6, 2013 at 2:40 PM, Chris Barker - NOAA Federal
On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers
wrote: Which exact Python do we need to use on Mac? Do we need to use the binary installer from python.org?
Yes, the one from python.org.
Or can I install it from source?
you could install from source using the same method that the python.org binaries are built -- there is a script with the source to do that, though I'm not sure what the point of that would be.
Is it possible to install the dmg images without root access from the command line? I know how to access the contents: $ hdiutil attach python-2.7.3-macosx10.6.dmg $ ls /Volumes/Python\ 2.7.3/ Build.txt License.txt Python.mpkg ReadMe.txt But I am not currently sure what to do with it. The Python.mpkg directory seems to contain the sources. I have access to Vincent's computer, as suggested by Ralf and it is already setup, so I am using it. But I am not able (so far) to replicate the setup there so that I can create the binaries on any other Mac computer, which makes me feel really uneasy. By replicating the setup, at least once (preferably automated) would make me understand things much better. If possible, I would prefer to just use a command line (ssh) to do all that. (So that's maybe building from source is the only option.) Ondrej
On Mon, Jan 7, 2013 at 6:09 PM, Ondřej Čertík
Is it possible to install the dmg images without root access from the command line?
I've never tried, but it looks like you can: http://www.commandlinefu.com/commands/view/2031/install-an-mpkg-from-the-com...
But I am not currently sure what to do with it. The Python.mpkg directory seems to contain the sources.
yup -- that's where everything is. the "installer" command should be able to unpack it.
By replicating the setup, at least once (preferably automated) would make me understand things much better. If possible, I would prefer to just use a command line (ssh) to do all that. (So that's maybe building from source is the only option.)
If you ndo need to build from source, see this message for a bit more info: http://mail.python.org/pipermail/pythonmac-sig/2012-October/023742.html there are a few prerequisites you need to install first... Either way, you should be able to build a start-to-finish build script. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Mon, Jan 7, 2013 at 8:41 PM, Chris Barker - NOAA Federal
On Mon, Jan 7, 2013 at 6:09 PM, Ondřej Čertík
wrote: Is it possible to install the dmg images without root access from the command line?
I've never tried, but it looks like you can:
http://www.commandlinefu.com/commands/view/2031/install-an-mpkg-from-the-com...
This requires root access. Without sudo, I get: $ installer -pkg /Volumes/Python\ 2.7.3/Python.mpkg/ -target ondrej installer: This package requires authentication to install. and since I don't have root access, it doesn't work. So one way around it would be to install python from source, that shouldn't require root access.
But I am not currently sure what to do with it. The Python.mpkg directory seems to contain the sources.
yup -- that's where everything is. the "installer" command should be able to unpack it.
Ok.
By replicating the setup, at least once (preferably automated) would make me understand things much better. If possible, I would prefer to just use a command line (ssh) to do all that. (So that's maybe building from source is the only option.)
If you ndo need to build from source, see this message for a bit more info:
http://mail.python.org/pipermail/pythonmac-sig/2012-October/023742.html
there are a few prerequisites you need to install first...
Either way, you should be able to build a start-to-finish build script.
Yes, that would be my goal eventually. Without root access. But right now, I am not even sure it's possible. So for now I'll simply use already pre-configured box. Ondrej
On Mon, Jan 7, 2013 at 10:23 PM, Ondřej Čertík
http://www.commandlinefu.com/commands/view/2031/install-an-mpkg-from-the-com...
This requires root access. Without sudo, I get:
$ installer -pkg /Volumes/Python\ 2.7.3/Python.mpkg/ -target ondrej installer: This package requires authentication to install.
and since I don't have root access, it doesn't work.
So one way around it would be to install python from source, that shouldn't require root access.
hmm -- this all may be a trick -- both the *.mpkg and the standard build put everything in /Library/Frameworks/Python -- which is where it belongs. Bu tif you need root access to write there, then there is a problem. I'm sure a non-root build could put everything in the users' home directory, then packages built against that would have their paths messed up. What's odd is that I'm pretty sure I've been able to point+click install those without sudo...(I could recall incorrectly). This would be a good question for the pythonmac list -- low traffic, but there are some very smart and helpful folks there: http://mail.python.org/mailman/listinfo/pythonmac-sig
But I am not currently sure what to do with it. The Python.mpkg directory seems to contain the sources.
It should be possible to unpack a mpkg by hand, but it contains both the contents, and various instal scripts, so that seems like a really ugly solution. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Tue, Jan 8, 2013 at 8:45 AM, Chris Barker - NOAA Federal
On Mon, Jan 7, 2013 at 10:23 PM, Ondřej Čertík
wrote: http://www.commandlinefu.com/commands/view/2031/install-an-mpkg-from-the-com...
This requires root access. Without sudo, I get:
$ installer -pkg /Volumes/Python\ 2.7.3/Python.mpkg/ -target ondrej installer: This package requires authentication to install.
and since I don't have root access, it doesn't work.
So one way around it would be to install python from source, that shouldn't require root access.
hmm -- this all may be a trick -- both the *.mpkg and the standard build put everything in /Library/Frameworks/Python -- which is where it belongs. Bu tif you need root access to write there, then there is a problem. I'm sure a non-root build could put everything in the users' home directory, then packages built against that would have their paths messed up.
Right.
What's odd is that I'm pretty sure I've been able to point+click install those without sudo...(I could recall incorrectly).
This would be a good question for the pythonmac list -- low traffic, but there are some very smart and helpful folks there:
http://mail.python.org/mailman/listinfo/pythonmac-sig
But I am not currently sure what to do with it. The Python.mpkg directory seems to contain the sources.
It should be possible to unpack a mpkg by hand, but it contains both the contents, and various instal scripts, so that seems like a really ugly solution.
Yep. In the meantime, the hard drive on Vincent's box failed, so he reinstalled the box completely. Also he explained to me a lot of Mac things over the phone, so I think I now understand what is going on with the dmg. As such, I have updated my instructions in my release helper repo: https://github.com/certik/numpy-vendor by the following paragraph: """ First prepare the Mac build box as follows: * Install Python 2.5, 2.6, 2.7 from python.org using the dmg disk image * Install setuptools and bdist_mpkg into all these Pythons * Install Paver into the default Python Tip: Add the /Library/Frameworks/Python.framework directory into git and commit after each installation of any package or Python. That way you can easily remove temporary installations. """ And you need sudo access to do those. If your user is an admin, then it can do it, otherwise it can't. So one can only use a Mac, which has the above setup installed. With that, my Fabfile can then do the rest. So I just built the following binaries: numpy-1.7.0rc1-py2.5-python.org-macosx10.3.dmg numpy-1.7.0rc1-py2.6-python.org-macosx10.3.dmg numpy-1.7.0rc1-py2.7-python.org-macosx10.3.dmg and uploaded to: https://sourceforge.net/projects/numpy/files/NumPy/1.7.0rc1/ So I think we are all set here. Ralf, would you be willing to build the final binary on 10.6? I don't think you have to do it for this rc1, but I am going to release rc2 now and for that it would be nice to have it. Ondrej
On Thu, Jan 10, 2013 at 3:55 AM, Ondřej Čertík
On Tue, Jan 8, 2013 at 8:45 AM, Chris Barker - NOAA Federal
wrote: On Mon, Jan 7, 2013 at 10:23 PM, Ondřej Čertík
wrote: http://www.commandlinefu.com/commands/view/2031/install-an-mpkg-from-the-com...
This requires root access. Without sudo, I get:
$ installer -pkg /Volumes/Python\ 2.7.3/Python.mpkg/ -target ondrej installer: This package requires authentication to install.
and since I don't have root access, it doesn't work.
So one way around it would be to install python from source, that shouldn't require root access.
hmm -- this all may be a trick -- both the *.mpkg and the standard build put everything in /Library/Frameworks/Python -- which is where it belongs. Bu tif you need root access to write there, then there is a problem. I'm sure a non-root build could put everything in the users' home directory, then packages built against that would have their paths messed up.
Right.
What's odd is that I'm pretty sure I've been able to point+click install those without sudo...(I could recall incorrectly).
This would be a good question for the pythonmac list -- low traffic, but there are some very smart and helpful folks there:
http://mail.python.org/mailman/listinfo/pythonmac-sig
But I am not currently sure what to do with it. The Python.mpkg directory seems to contain the sources.
It should be possible to unpack a mpkg by hand, but it contains both the contents, and various instal scripts, so that seems like a really ugly solution.
Yep.
In the meantime, the hard drive on Vincent's box failed, so he reinstalled the box completely. Also he explained to me a lot of Mac things over the phone, so I think I now understand what is going on with the dmg.
As such, I have updated my instructions in my release helper repo:
https://github.com/certik/numpy-vendor
by the following paragraph:
""" First prepare the Mac build box as follows:
* Install Python 2.5, 2.6, 2.7 from python.org using the dmg disk image * Install setuptools and bdist_mpkg into all these Pythons * Install Paver into the default Python
Tip: Add the /Library/Frameworks/Python.framework directory into git and commit after each installation of any package or Python. That way you can easily remove temporary installations. """
And you need sudo access to do those. If your user is an admin, then it can do it, otherwise it can't. So one can only use a Mac, which has the above setup installed. With that, my Fabfile can then do the rest.
So I just built the following binaries:
numpy-1.7.0rc1-py2.5-python.org-macosx10.3.dmg numpy-1.7.0rc1-py2.6-python.org-macosx10.3.dmg numpy-1.7.0rc1-py2.7-python.org-macosx10.3.dmg
and uploaded to:
https://sourceforge.net/projects/numpy/files/NumPy/1.7.0rc1/
So I think we are all set here. Ralf, would you be willing to build the final binary on 10.6? I don't think you have to do it for this rc1, but I am going to release rc2 now and for that it would be nice to have it.
Sure, no problem. For the part that needs to be built on 10.6 that is. Vincent's box still has 10.5, right? Ralf
Ondřej, Vincent, and Ralf (and others..) Thank you so much for doing all this -- it's a great service to the MacPython community. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers
On Sun, Jan 6, 2013 at 3:21 AM, Ondřej Čertík
wrote: Hi,
Currently the NumPy binaries are built using the pavement.py script, which uses the following Pythons:
MPKG_PYTHON = { "2.5": ["/Library/Frameworks/Python.framework/Versions/2.5/bin/python"], "2.6": ["/Library/Frameworks/Python.framework/Versions/2.6/bin/python"], "2.7": ["/Library/Frameworks/Python.framework/Versions/2.7/bin/python"], "3.1": ["/Library/Frameworks/Python.framework/Versions/3.1/bin/python3"], "3.2": ["/Library/Frameworks/Python.framework/Versions/3.2/bin/python3"], "3.3": ["/Library/Frameworks/Python.framework/Versions/3.3/bin/python3"], }
So for example I can easily create the 2.6 binary if that Python is pre-installed on the Mac box that I am using. On one of the Mac boxes that I am using, the 2.7 is missing, so are 3.1, 3.2 and 3.3. So I was thinking of updating my Fabric fab file to automatically install all Pythons from source and build against that, just like I do for Wine.
Which exact Python do we need to use on Mac? Do we need to use the binary installer from python.org?
Yes, the one from python.org.
Or can I install it from source? Finally, for which Python versions should we provide binary installers for Mac? For reference, the 1.6.2 had installers for 2.5, 2.6 and 2.7 only for OS X 10.3. There is only 2.7 version for OS X 10.6.
The provided installers and naming scheme should match what's done for Python itself on python.org.
The 10.3 installers for 2.5, 2.6 and 2.7 should be compiled on OS X 10.5. This is kind of hard to come by these days, but Vincent Davis maintains a build machine for numpy and scipy. That's already set up correctly, so all you have to do is connect to it via ssh, check out v.17.0 in ~/Code/numpy, check in release.sh that the section for OS X 10.6 is disabled and for 10.5 enabled and run it.
OS X 10.6 broke support for previous versions in some subtle ways, so even when using the 10.4 SDK numpy compiled on 10.6 won't run on 10.5. As long as we're supporting 10.5 you therefore need to compile on it.
The 10.7 --> 10.6 support hasn't been checked, but I wouldn't trust it. I have a 10.6 machine, so I can compile those binaries if needed.
Also, what is the meaning of the following piece of code in pavement.py:
def _build_mpkg(pyver): # account for differences between Python 2.7.1 versions from python.org if os.environ.get('MACOSX_DEPLOYMENT_TARGET', None) == "10.6": ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch x86_64 -Wl,-search_paths_first" else: ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first" ldflags += " -L%s" % os.path.join(os.path.dirname(__file__), "build")
The 10.6 binaries support only Intel Macs, both 32-bit and 64-bit. The 10.3 binaries support PPC Macs and 32-bit Intel. That's what the above does. Note that we simply follow the choice made by the Python release managers here.
if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver])))
This is necessary because in Python 2.5, distutils asks for "gcc" instead of "gcc-4.0", so you may get the wrong one without CC=gcc-4.0. From Python 2.6 on this was fixed.
In particular, the last line gets executed and it then fails with:
paver dmg -p 2.6 ---> pavement.dmg ---> pavement.clean LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first -Lbuild' /Library/Frameworks/Python.framework/Versions/2.6/bin/python setupegg.py bdist_mpkg Traceback (most recent call last): File "setupegg.py", line 17, in <module> from setuptools import setup ImportError: No module named setuptools
The reason is (I think) that if the Python binary is called explicitly with /Library/Frameworks/Python.framework/Versions/2.6/bin/python, then the paths are not setup properly in virtualenv, and thus setuptools (which is only installed in virtualenv, but not in system Python) fails to import. The solution is to simply apply this patch:
Avoid using system Python for anything. The first thing to do on any new OS X system is install Python some other way, preferably from python.org.
diff --git a/pavement.py b/pavement.py index e693016..0c637f8 100644 --- a/pavement.py +++ b/pavement.py @@ -449,7 +449,7 @@ def _build_mpkg(pyver): if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: - sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) + sh("python setupegg.py bdist_mpkg")
This doesn't work unless using virtualenvs, you're just throwing away the version selection here. If you can support virtualenvs in addition to python.org pythons, that would be useful. But being able to build binaries when needed simply by "paver dmg -p 2.x" is quite useful.
Absolutely. I was following the release.sh in the numpy git repository, which contains: paver bootstrap source bootstrap/bin/activate python setupsconsegg.py install paver pdf paver dmg -p 2.7 So it is using the virtualenv and it works on Vincent's computer, but it doesn't work on my other computer. I wanted to make the steps somehow reproducible. I started adding the commands needed to setup the Mac (any Mac) into my Fabfile here: https://github.com/certik/numpy-vendor/blob/master/fabfile.py#L98 but I run into the issues above. Of course, I'll try to just use Vincent's computer, but I would feel much better if the numpy release process for Mac didn't depend on one particular computer, but rather could be quite easily reproduced on any Mac OS X of the right version. Ondrej
On Tue, Jan 8, 2013 at 3:12 AM, Ondřej Čertík
On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers
wrote: On Sun, Jan 6, 2013 at 3:21 AM, Ondřej Čertík
wrote: Hi,
Currently the NumPy binaries are built using the pavement.py script, which uses the following Pythons:
MPKG_PYTHON = { "2.5": ["/Library/Frameworks/Python.framework/Versions/2.5/bin/python"], "2.6": ["/Library/Frameworks/Python.framework/Versions/2.6/bin/python"], "2.7": ["/Library/Frameworks/Python.framework/Versions/2.7/bin/python"], "3.1": ["/Library/Frameworks/Python.framework/Versions/3.1/bin/python3"], "3.2": ["/Library/Frameworks/Python.framework/Versions/3.2/bin/python3"], "3.3": ["/Library/Frameworks/Python.framework/Versions/3.3/bin/python3"], }
So for example I can easily create the 2.6 binary if that Python is pre-installed on the Mac box that I am using. On one of the Mac boxes that I am using, the 2.7 is missing, so are 3.1, 3.2 and 3.3. So I was thinking of updating my Fabric fab file to automatically install all Pythons from source and build against that, just like I do for Wine.
Which exact Python do we need to use on Mac? Do we need to use the binary installer from python.org?
Yes, the one from python.org.
Or can I install it from source? Finally, for which Python versions should we provide binary installers for Mac? For reference, the 1.6.2 had installers for 2.5, 2.6 and 2.7 only for OS X 10.3. There is only 2.7 version for OS X 10.6.
The provided installers and naming scheme should match what's done for Python itself on python.org.
The 10.3 installers for 2.5, 2.6 and 2.7 should be compiled on OS X 10.5. This is kind of hard to come by these days, but Vincent Davis maintains a build machine for numpy and scipy. That's already set up correctly, so
all
you have to do is connect to it via ssh, check out v.17.0 in ~/Code/numpy, check in release.sh that the section for OS X 10.6 is disabled and for 10.5 enabled and run it.
OS X 10.6 broke support for previous versions in some subtle ways, so even when using the 10.4 SDK numpy compiled on 10.6 won't run on 10.5. As long as we're supporting 10.5 you therefore need to compile on it.
The 10.7 --> 10.6 support hasn't been checked, but I wouldn't trust it. I have a 10.6 machine, so I can compile those binaries if needed.
Also, what is the meaning of the following piece of code in pavement.py:
def _build_mpkg(pyver): # account for differences between Python 2.7.1 versions from python.org if os.environ.get('MACOSX_DEPLOYMENT_TARGET', None) == "10.6": ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch x86_64 -Wl,-search_paths_first" else: ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first" ldflags += " -L%s" % os.path.join(os.path.dirname(__file__),
"build")
The 10.6 binaries support only Intel Macs, both 32-bit and 64-bit. The 10.3 binaries support PPC Macs and 32-bit Intel. That's what the above does. Note that we simply follow the choice made by the Python release managers here.
if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver])))
This is necessary because in Python 2.5, distutils asks for "gcc" instead of "gcc-4.0", so you may get the wrong one without CC=gcc-4.0. From Python 2.6 on this was fixed.
In particular, the last line gets executed and it then fails with:
paver dmg -p 2.6 ---> pavement.dmg ---> pavement.clean LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first -Lbuild' /Library/Frameworks/Python.framework/Versions/2.6/bin/python setupegg.py bdist_mpkg Traceback (most recent call last): File "setupegg.py", line 17, in <module> from setuptools import setup ImportError: No module named setuptools
The reason is (I think) that if the Python binary is called explicitly with /Library/Frameworks/Python.framework/Versions/2.6/bin/python, then the paths are not setup properly in virtualenv, and thus setuptools (which is only installed in virtualenv, but not in system Python) fails to import. The solution is to simply apply this patch:
Avoid using system Python for anything. The first thing to do on any new OS X system is install Python some other way, preferably from python.org.
diff --git a/pavement.py b/pavement.py index e693016..0c637f8 100644 --- a/pavement.py +++ b/pavement.py @@ -449,7 +449,7 @@ def _build_mpkg(pyver): if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: - sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) + sh("python setupegg.py bdist_mpkg")
This doesn't work unless using virtualenvs, you're just throwing away the version selection here. If you can support virtualenvs in addition to python.org pythons, that would be useful. But being able to build binaries when needed simply by "paver dmg -p 2.x" is quite useful.
Absolutely. I was following the release.sh in the numpy git repository, which contains:
paver bootstrap source bootstrap/bin/activate python setupsconsegg.py install paver pdf paver dmg -p 2.7
So it is using the virtualenv and it works on Vincent's computer, but it doesn't work on my other computer.
Note that it's only using a virtualenv for this one step (building the docs). This is because building the docs requires installing numpy first to be able to extract the docstrings.
I wanted to make the steps somehow reproducible. I started adding the commands needed to setup the Mac (any Mac) into my Fabfile here:
https://github.com/certik/numpy-vendor/blob/master/fabfile.py#L98
but I run into the issues above.
Of course, I'll try to just use Vincent's computer, but I would feel much better if the numpy release process for Mac didn't depend on one particular computer, but rather could be quite easily reproduced on any Mac OS X of the right version.
It doesn't depend on that one computer of course, it takes only a few minutes to set up a new Mac. But yes, currently it does require admin rights to install a framework Python. Ralf
On Mon, Jan 7, 2013 at 11:36 PM, Ralf Gommers
On Tue, Jan 8, 2013 at 3:12 AM, Ondřej Čertík
wrote: On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers
wrote: On Sun, Jan 6, 2013 at 3:21 AM, Ondřej Čertík
wrote: Hi,
Currently the NumPy binaries are built using the pavement.py script, which uses the following Pythons:
MPKG_PYTHON = { "2.5": ["/Library/Frameworks/Python.framework/Versions/2.5/bin/python"], "2.6": ["/Library/Frameworks/Python.framework/Versions/2.6/bin/python"], "2.7": ["/Library/Frameworks/Python.framework/Versions/2.7/bin/python"], "3.1": ["/Library/Frameworks/Python.framework/Versions/3.1/bin/python3"], "3.2": ["/Library/Frameworks/Python.framework/Versions/3.2/bin/python3"], "3.3": ["/Library/Frameworks/Python.framework/Versions/3.3/bin/python3"], }
So for example I can easily create the 2.6 binary if that Python is pre-installed on the Mac box that I am using. On one of the Mac boxes that I am using, the 2.7 is missing, so are 3.1, 3.2 and 3.3. So I was thinking of updating my Fabric fab file to automatically install all Pythons from source and build against that, just like I do for Wine.
Which exact Python do we need to use on Mac? Do we need to use the binary installer from python.org?
Yes, the one from python.org.
Or can I install it from source? Finally, for which Python versions should we provide binary installers for Mac? For reference, the 1.6.2 had installers for 2.5, 2.6 and 2.7 only for OS X 10.3. There is only 2.7 version for OS X 10.6.
The provided installers and naming scheme should match what's done for Python itself on python.org.
The 10.3 installers for 2.5, 2.6 and 2.7 should be compiled on OS X 10.5. This is kind of hard to come by these days, but Vincent Davis maintains a build machine for numpy and scipy. That's already set up correctly, so all you have to do is connect to it via ssh, check out v.17.0 in ~/Code/numpy, check in release.sh that the section for OS X 10.6 is disabled and for 10.5 enabled and run it.
OS X 10.6 broke support for previous versions in some subtle ways, so even when using the 10.4 SDK numpy compiled on 10.6 won't run on 10.5. As long as we're supporting 10.5 you therefore need to compile on it.
The 10.7 --> 10.6 support hasn't been checked, but I wouldn't trust it. I have a 10.6 machine, so I can compile those binaries if needed.
Also, what is the meaning of the following piece of code in pavement.py:
def _build_mpkg(pyver): # account for differences between Python 2.7.1 versions from python.org if os.environ.get('MACOSX_DEPLOYMENT_TARGET', None) == "10.6": ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch x86_64 -Wl,-search_paths_first" else: ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first" ldflags += " -L%s" % os.path.join(os.path.dirname(__file__), "build")
The 10.6 binaries support only Intel Macs, both 32-bit and 64-bit. The 10.3 binaries support PPC Macs and 32-bit Intel. That's what the above does. Note that we simply follow the choice made by the Python release managers here.
if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver])))
This is necessary because in Python 2.5, distutils asks for "gcc" instead of "gcc-4.0", so you may get the wrong one without CC=gcc-4.0. From Python 2.6 on this was fixed.
In particular, the last line gets executed and it then fails with:
paver dmg -p 2.6 ---> pavement.dmg ---> pavement.clean LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first -Lbuild' /Library/Frameworks/Python.framework/Versions/2.6/bin/python setupegg.py bdist_mpkg Traceback (most recent call last): File "setupegg.py", line 17, in <module> from setuptools import setup ImportError: No module named setuptools
The reason is (I think) that if the Python binary is called explicitly with /Library/Frameworks/Python.framework/Versions/2.6/bin/python, then the paths are not setup properly in virtualenv, and thus setuptools (which is only installed in virtualenv, but not in system Python) fails to import. The solution is to simply apply this patch:
Avoid using system Python for anything. The first thing to do on any new OS X system is install Python some other way, preferably from python.org.
diff --git a/pavement.py b/pavement.py index e693016..0c637f8 100644 --- a/pavement.py +++ b/pavement.py @@ -449,7 +449,7 @@ def _build_mpkg(pyver): if pyver == "2.5": sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) else: - sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, " ".join(MPKG_PYTHON[pyver]))) + sh("python setupegg.py bdist_mpkg")
This doesn't work unless using virtualenvs, you're just throwing away the version selection here. If you can support virtualenvs in addition to python.org pythons, that would be useful. But being able to build binaries when needed simply by "paver dmg -p 2.x" is quite useful.
Absolutely. I was following the release.sh in the numpy git repository, which contains:
paver bootstrap source bootstrap/bin/activate python setupsconsegg.py install paver pdf paver dmg -p 2.7
So it is using the virtualenv and it works on Vincent's computer, but it doesn't work on my other computer.
Note that it's only using a virtualenv for this one step (building the docs). This is because building the docs requires installing numpy first to be able to extract the docstrings.
Ah, I missed this important part. Since I generate the pdf files in linux, I can just copy them on Mac and thus don't need any of the virtualenv part.
I wanted to make the steps somehow reproducible. I started adding the commands needed to setup the Mac (any Mac) into my Fabfile here:
https://github.com/certik/numpy-vendor/blob/master/fabfile.py#L98
but I run into the issues above.
Of course, I'll try to just use Vincent's computer, but I would feel much better if the numpy release process for Mac didn't depend on one particular computer, but rather could be quite easily reproduced on any Mac OS X of the right version.
It doesn't depend on that one computer of course, it takes only a few minutes to set up a new Mac. But yes, currently it does require admin rights to install a framework Python.
Ok, that's what I wanted to know. Thanks, Ondrej
participants (5)
-
Chris Barker - NOAA Federal
-
David Cournapeau
-
Matthew Brett
-
Ondřej Čertík
-
Ralf Gommers