[Numpy-discussion] Which Python to use for Mac binaries

[Ondřej Čertík]

On Sun, Jan 6, 2013 at 2:04 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
>
> On Sun, Jan 6, 2013 at 3:21 AM, Ondřej Čertík <ondrej.certik at gmail.com>
> wrote:
>>
>> Hi,
>>
>> Currently the NumPy binaries are built using the pavement.py script,
>> which uses the following Pythons:
>>
>> MPKG_PYTHON = {
>>         "2.5":
>> ["/Library/Frameworks/Python.framework/Versions/2.5/bin/python"],
>>         "2.6":
>> ["/Library/Frameworks/Python.framework/Versions/2.6/bin/python"],
>>         "2.7":
>> ["/Library/Frameworks/Python.framework/Versions/2.7/bin/python"],
>>         "3.1":
>> ["/Library/Frameworks/Python.framework/Versions/3.1/bin/python3"],
>>         "3.2":
>> ["/Library/Frameworks/Python.framework/Versions/3.2/bin/python3"],
>>         "3.3":
>> ["/Library/Frameworks/Python.framework/Versions/3.3/bin/python3"],
>> }
>>
>> So for example I can easily create the 2.6 binary if that Python is
>> pre-installed on the Mac box that I am using.
>> On one of the Mac boxes that I am using, the 2.7 is missing, so are
>> 3.1, 3.2 and 3.3. So I was thinking
>> of updating my Fabric fab file to automatically install all Pythons
>> from source and build against that, just like I do for Wine.
>>
>> Which exact Python do we need to use on Mac? Do we need to use the
>> binary installer from python.org?
>
>
> Yes, the one from python.org.
>
>>
>> Or can I install it from source? Finally, for which Python versions
>> should we provide binary installers for Mac?
>> For reference, the 1.6.2 had installers for 2.5, 2.6 and 2.7 only for
>> OS X 10.3. There is only 2.7 version for OS X 10.6.
>
>
> The provided installers and naming scheme should match what's done for
> Python itself on python.org.
>
> The 10.3 installers for 2.5, 2.6 and 2.7 should be compiled on OS X 10.5.
> This is kind of hard to come by these days, but Vincent Davis maintains a
> build machine for numpy and scipy. That's already set up correctly, so all
> you have to do is connect to it via ssh, check out v.17.0 in ~/Code/numpy,
> check in release.sh that the section for OS X 10.6 is disabled and for 10.5
> enabled and run it.
>
> OS X 10.6 broke support for previous versions in some subtle ways, so even
> when using the 10.4 SDK numpy compiled on 10.6 won't run on 10.5. As long as
> we're supporting 10.5 you therefore need to compile on it.
>
> The 10.7 --> 10.6 support hasn't been checked, but I wouldn't trust it. I
> have a 10.6 machine, so I can compile those binaries if needed.
>
>>
>> Also, what is the meaning of the following piece of code in pavement.py:
>>
>> def _build_mpkg(pyver):
>>     # account for differences between Python 2.7.1 versions from
>> python.org
>>     if os.environ.get('MACOSX_DEPLOYMENT_TARGET', None) == "10.6":
>>         ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch
>> x86_64 -Wl,-search_paths_first"
>>     else:
>>         ldflags = "-undefined dynamic_lookup -bundle -arch i386 -arch
>> ppc -Wl,-search_paths_first"
>>     ldflags += " -L%s" % os.path.join(os.path.dirname(__file__), "build")
>
>
> The 10.6 binaries support only Intel Macs, both 32-bit and 64-bit. The 10.3
> binaries support PPC Macs and 32-bit Intel. That's what the above does. Note
> that we simply follow the choice made by the Python release managers here.
>
>>
>>     if pyver == "2.5":
>>         sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" %
>> (ldflags, " ".join(MPKG_PYTHON[pyver])))
>>     else:
>>         sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, "
>> ".join(MPKG_PYTHON[pyver])))
>
>
> This is necessary because in Python 2.5, distutils asks for "gcc" instead of
> "gcc-4.0", so you may get the wrong one without CC=gcc-4.0. From Python 2.6
> on this was fixed.
>
>>
>> In particular, the last line gets executed and it then fails with:
>>
>> paver dmg -p 2.6
>> ---> pavement.dmg
>> ---> pavement.clean
>> LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc
>> -Wl,-search_paths_first -Lbuild'
>> /Library/Frameworks/Python.framework/Versions/2.6/bin/python
>> setupegg.py bdist_mpkg
>> Traceback (most recent call last):
>>   File "setupegg.py", line 17, in <module>
>>     from setuptools import setup
>> ImportError: No module named setuptools
>>
>>
>> The reason is (I think) that if the Python binary is called explicitly
>> with /Library/Frameworks/Python.framework/Versions/2.6/bin/python,
>> then the paths are not setup properly in virtualenv, and thus
>> setuptools (which is only installed in virtualenv, but not in system
>> Python) fails to import. The solution is to simply apply this patch:
>
>
> Avoid using system Python for anything. The first thing to do on any new OS
> X system is install Python some other way, preferably from python.org.
>
>>
>> diff --git a/pavement.py b/pavement.py
>> index e693016..0c637f8 100644
>> --- a/pavement.py
>> +++ b/pavement.py
>> @@ -449,7 +449,7 @@ def _build_mpkg(pyver):
>>      if pyver == "2.5":
>>          sh("CC=gcc-4.0 LDFLAGS='%s' %s setupegg.py bdist_mpkg" %
>> (ldflags, " ".join(MPKG_PYTHON[pyver])))
>>      else:
>> -        sh("LDFLAGS='%s' %s setupegg.py bdist_mpkg" % (ldflags, "
>> ".join(MPKG_PYTHON[pyver])))
>> +        sh("python setupegg.py bdist_mpkg")
>
>
> This doesn't work unless using virtualenvs, you're just throwing away the
> version selection here. If you can support virtualenvs in addition to
> python.org pythons, that would be useful. But being able to build binaries
> when needed simply by "paver dmg -p 2.x" is quite useful.


Absolutely. I was following the release.sh in the numpy git
repository, which contains:

paver bootstrap
source bootstrap/bin/activate
python setupsconsegg.py install
paver pdf
paver dmg -p 2.7

So it is using the virtualenv and it works on Vincent's computer, but
it doesn't work on my
other computer.

I wanted to make the steps somehow reproducible. I started adding the
commands needed to setup the Mac (any Mac)
into my Fabfile here:

https://github.com/certik/numpy-vendor/blob/master/fabfile.py#L98

but I run into the issues above.

Of course, I'll try to just use Vincent's computer, but I would feel
much better if the numpy release process for Mac didn't depend on one
particular computer, but rather could be quite easily reproduced on
any Mac OS X of the right version.

Ondrej