Hi Robert, Thanks for the report. On Sun, Mar 29, 2009 at 12:10 AM, Robert Pyle wrote:

Hi all,

On Mar 28, 2009, at 9:26 AM, David Cournapeau wrote:

...

I am pleased to announce the release of the rc1 for numpy 1.3.0. You can find source tarballs and installers for both Mac OS X and Windows on the sourceforge page:

https://sourceforge.net/projects/numpy/ https://sourceforge.net/projects/numpy/

I have a PPC Mac, dual G5, running 10.5.6.

The Mac OS X installer (numpy-1.3.0rc1-py2.5-macosx10.5.dmg) did not work for me.  It said none of my disks were suitable for installation.

Hm, strange, I have never encountered this problem. To be sure I understand, you could open/mount the .dmg, but the .pkg refuses to install ?

 The last time around, numpy-1.3.0b1-py2.5- macosx10.5.dmg persisted in installing itself into the system python rather than the Enthought distribution that I use, so I installed that version from the source tarball.

I am afraid there is nothing I can do here - the installer can only work with the system python I believe (or more exactly the python version I built the package against). Maybe people more familiar with bdist_mpkg could prove me wrong ? cheers, David

Robert Pyle wrote:

Yes. When it gets to "Select a Destination", I would expect my boot disk to get the green arrow as the installation target, but it (and the other three disks) have the exclamation point in the red circle. Same thing happened on my MacBook Pro (Intel) with its one disk.

Now that I think about it, maybe this is due to the lack of a python interpreter from python.org on your side. Did you install any other python besides the one included in EPD ? If that's the problem, we should at least mention somewhere that the python from python.org is needed. cheers, David

On Sun, Mar 29, 2009 at 7:43 AM, Robert Pyle wrote:

Hi David,

On Mar 29, 2009, at 4:03 AM, David Cournapeau wrote:

...

Robert Pyle wrote:

...

Yes.  When it gets to "Select a Destination", I would expect my boot disk to get the green arrow as the installation target, but it (and the other three disks) have the exclamation point in the red circle. Same thing happened on my MacBook Pro (Intel) with its one disk.

Now that I think about it, maybe this is due to the lack of a python interpreter from python.org on your side. Did you install any other python besides the one included in EPD ? If that's the problem, we should at least mention somewhere that the python from python.org is needed.

Okay, I just installed 2.6.1 from python.org, and it is now the version that starts when I type "python" to Terminal.  I still cannot install numpy-1.3.0rc1 from the OS X installer, numpy-1.3.0rc1-py2.5- macosx10.5.dmg

Yes, you can't install a python 2.5 package on python 2.6. It is almost like installing from sources is actually easier than from an installer on mac os x... I can relatively easily provide a 2.6 installer, though. cheers, David

On Mon, Mar 30, 2009 at 3:36 AM, Robert Pyle wrote:

I just installed 2.5.4 from python.org, and the OS X installer still doesn't work.  This is on a PPC G5; I haven't tried it on my Intel MacBook Pro.

I think I got it. To build numpy, I use virtualenv to make a "bootstrap" environment, but then the corresponding python path get embedded in the .mpkg - so unless you have your python interpreter in exactly the same path as my bootstrap (which is very unlikely), it won't run at all. This would also explain why I never saw the problem. I will prepare a new binary, cheers, David

Hi David, I decided to change the Subject line to be more apropos. On Mar 30, 2009, at 3:41 AM, David Cournapeau wrote:

David Cournapeau wrote:

...

On Mon, Mar 30, 2009 at 3:36 AM, Robert Pyle wrote:

...

I just installed 2.5.4 from python.org, and the OS X installer still doesn't work. This is on a PPC G5; I haven't tried it on my Intel MacBook Pro.

Could you try this one ?

http://www.ar.media.kyoto-u.ac.jp/members/david/archives/numpy/numpy-1.3.0rc...

This one installs, but only in /Library/Python/2.5/site-packages/, that is, for Apple's system python. This happened when `which python` pointed to either EPD python or python.org's 2.5.4.

If it does not work, getting the /var/tmp/install.log would be helpful (the few last lines),

/var/tmp/ had a bunch of stuff in it, but no file named install.log. Perhaps that's because the installation succeeded?

Bob

David Cournapeau wrote:

On Mon, Mar 30, 2009 at 11:06 PM, Robert Pyle wrote:

...

This one installs, but only in /Library/Python/2.5/site-packages/, that is, for Apple's system python. This happened when `which python` pointed to either EPD python or python.org's 2.5.4.

Yes, what your default python is does not matter: I don't know the details, but it looks like the mac os x installer only looks whether a python binary exists in /System/Library/..., that is the one I used to build the package. You can see this in the Info.plist inside the .mpkg.

Well, this is the big question: what python(s) should be provide binaries for -- I think if you're only going to do one, it should be the python.org build, so that you can support 10.4, and 10.5 and everyone can use it. There are ways to build an installer that puts it in a place that both can find it -- wxPython does this -- but I'm not so sure that's a good idea. One of the key questions is how one should think of Apple's Python. They are using it for some system tools, so we really shouldn't break it. If you upgrade the numpy it comes with, there is some chance that you could break something. Also, Apple has not (and likely won't) upgrade their Python. I know I happened to run into a bug and needed a newer 2.5, so I'd rather have the control. A few years ago the MacPython community (as represented by the members of the pythonmac list) decided that the python.org build was that one that we should all target for binaries. That consensus has weakened with 10.5, as Apple did provide a Python that is fairly up to date and almost fully functional, but I think it's still a lot easier on everyone if we just stick with the python.org build as the one to target for binaries. That being said, it shouldn't be hard to build separate binaries for each python -- they would be identical except for where they get installed, and if they are clearly marked for downloading, there shouldn't be too much confusion. -Chris

David Cournapeau wrote:

I don't really care, as long as there is only one. Maintaining binaries for every python out there is too time consuming. Given that mac os X is the easiest platform to build numpy/scipy on,

I assume you meant NOT the easiest? ;-)

that's not something i am interested in.

quite understandable.

...

There are ways to build an installer that puts it in a place that both can find it -- wxPython does this -- but I'm not so sure that's a good idea.

there is the problem of compatibility. I am not sure whether Apple python and python.org are ABI compatible

In theory, yes, and in practice, it seems to be working for wxPython. However, I agree that it's a bit risky. I'm at the PyCon MacPython sprint as we type -- and apparently Apple's is linked with the 10.5 sdk, whereas python.org's is linked against the 10.3 sdk -- so there could be issues.

I will thus build binaries against python.org binaries (I still have to find a way to guarantee this in the build script, but that should not be too difficult).

Hardcoding the path to python should work: PYTHON=/Library/Frameworks/Python.framework/Versions/2.5/bin/python

My experience is that every choice presented to the user makes for more problem. And that just takes too much time. I prefer spending time making a few good installers rather than many half baked.

I agree -- and most packages I use seem to supporting python.org exclusively for binaries.

Ideally, we should have something which could install on every python version, but oh well,

well, I guess that's the promise of easy_install -- but someone would have to build all the binary eggs... and there were weird issues with universal eggs on the mac that I understand have been fixed in 2.6, but not 2.5 Thanks for all your work on this, -Chris

David Cournapeau wrote:

On Tue, Mar 31, 2009 at 2:10 AM, Chris Barker wrote:

...

I assume you meant NOT the easiest? ;-)

Actually, no, I meant it :) It has gcc, which is the best supported compiler by numpy and scipy, there is almost no problem with g77, and the optimized blas/lapack is provided by the OS vendor, meaning on ABI issue, weird atlas build errors, etc... It is almost impossible to get the build wrong on mac os x once you get the right fortran compiler.

I see -- well that's good news. I've found the Universal library requirements to be a pain sometimes, and it probably would be here if Apple wasn't giving us lapack/blas.

I am almost certain there are issues in some configurations, in particular x86_64.

Well, neither Apple nor python.org's builds are 64 bit anyway at this point. There is talk of quad (i386,and ppc_64 i86_64) builds the the future, though.

...

...

I will thus build binaries against python.org binaries (I still have to find a way to guarantee this in the build script, but that should not be too difficult). Hardcoding the path to python should work:

PYTHON=/Library/Frameworks/Python.framework/Versions/2.5/bin/python

Well, yes, but you can't really control this in the bdist_mpkg command.

bdist_mpkg should do "the right thing" if it's run with the right python. So you need to make sure you run: /Library/Frameworks/Python.framework/Versions/2.5/bin/bdist_mpkg Rather than whatever one happens to be found on your PATH.

Also, my current paver file uses virtualenv to build a isolated numpy - that's what breaks the .mpkg, but I like this approach for building, so I would like to keep it as much as possible.

Well, maybe we need to hack bdist_mpkg to support this, we're pretty sure that it is possible. I want o make sure I understand what you want: Do you want to be able to build numpy in a virtualenv, and then build a mpkg that will install into the users regular Framework? Do you want to be able to build a mpkg that users can install into the virtualenv of their choice? Both? Of course, easy_install can do that, when it works!

There are numerous problems with eggs (or more precisely, with "easy" install), which I am just not interested in getting into.

me neither --

In particular, it often breaks the user system - fixing it is easy for developers/"power users", but is a PITA for normal users. As long as easy_install is broken, I don't want to use it.

We were just talking about some of that last night -- we really need a "easy_uninstall" for instance. I'm going to poke into bdist_mpkg a bit right now. By the way, for the libgfortran issue, while statically linking it may be the best option, it wouldn't be too hard to have the mpkg include and install /usr/local/lib/ligfortran.dylib (or whatever). -Chris

David Cournapeau wrote:

Chris Barker wrote:

...

Well, neither Apple nor python.org's builds are 64 bit anyway at this point. There is talk of quad (i386,and ppc_64 i86_64) builds the the future, though.

Yes, but that's something that has to should be supported sooner rather than later.

It does, but we don't need a binary installer for a python that doesn't have a binary installer.

...

Well, maybe we need to hack bdist_mpkg to support this, we're pretty sure that it is possible.

Yes - more exactly, there should be a way to guarantee that if I create a virtual env from a given python interpreter, I can target a .mpkg to this python interpreter.

Hmmm -- I don't know virtualenv enough to know what the virtualenv knows about how it was created... However, I'm not sure you need to do what your saying here. I imagine this workflow: set up a virtualenv for, say numpy x.y.rc-z play around with it, get everything to build, etc. with plain old setup.py build, setup.py install, etc. Once you are happy, run: /Library/Frameworks/Python.framework/Versions/2.5/bin/bdist_mpkg (or the 2.6 equivalent, etc) I THINK you'd get a .mpkg that was all set for the user to install in their Framework python. As long as you don't run the installer, you won't end up with it in your virtualenv. Or is this what you've tried and has failed for you? By the way, if you run bdist_mpkg from a version installed into your virtualenv, you will get an installer that will install into your virtualenv, whit the path hard coded, so really useless.

Installing is necessary to build the doc correctly, and I don't want to mess my system with setuptools stuff.

ah -- maybe that's the issue then -- darn. Are the docs included in the .mpkg? Do they need to be built for that?

I have started writing something to organize my thought a bit better - I can keep you posted if you are interested).

yes, I am.

...

By the way, for the libgfortran issue, while statically linking it may be the best option, it wouldn't be too hard to have the mpkg include and install /usr/local/lib/ligfortran.dylib (or whatever).

I don't think it is a good idea: it would overwrite existing libgfortran.dylib, which would cause a lot of issues because libgfortran and gfortran have to be consistent. I know I would be very pissed if after installing a software, some unrelated software would be broken or worse overwritten.

True. In that case we could put the dylib somewhere obscure: /usr/local/lib/scipy1.6/lib/ or even: /Library/Frameworks/Python.framework/Versions/2.5/lib/ But using static linking is probably better. Actually, and I betray my ignorance here, but IIUC: - There are a bunch of different scipy extensions that use libgfortran - Many of them are built more-or-less separately - So each of them would get their own copy of the static libgfortran - Just how many separate copies of libgfortran is that? - Enough to care? - How big is libgfortran? This is making me think solving the dynamic linking problem makes sense. Also, would it break anything if the libgfortran installed were properly versioned: libgfortran.a.b.c Isn't that the point of versioned libs? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

David Cournapeau wrote:

Christopher Barker wrote:

...

It does, but we don't need a binary installer for a python that doesn't have a binary installer.

Yes, not now - but I would prefer avoiding to have to change the process again when time comes. It may not look like it, but enabling a working process which works well on all platforms including windows took me several days to work properly.

I'm not surprised it took that long -- that sounds short to me! Anyway, If there are new python builds that are 64-bit (quad?) you wont' have to change much -- "only" make sure that the libs you are linking to are 64 bit. I suppose you could try to get a quad-universal gfortran.a now, but I'd wait 'till you need it.

...

However, I'm not sure you need to do what your saying here. I imagine this workflow:

set up a virtualenv for, say numpy x.y.rc-z

play around with it, get everything to build, etc. with plain old setup.py build, setup.py install, etc.

Once you are happy, run:

/Library/Frameworks/Python.framework/Versions/2.5/bin/bdist_mpkg

This means building the same thing twice.

does it? I just did test: I set up a virtualenv with nothing in it. I used that environment to do: setup.py build setup.py install and got an apparently working numpy in my virtualenv. Then I ran: /Library/Frameworks/Python.framework/Versions/2.5/bin/bdist_mpkg It did a lot, but I don't THINK is re-compiled everthing. I think the trick is that eveything it built was still in the build dir -- and it is the saem python, even though it's not living in the same place. I got what seems to be a functioning Universal Installer for the python.org python. Having said that, it looks like it may be very easy to hack a package built in the virtualenv to install in the right place. In: dist/numpy-1.3.0rc1-py2.5-macosx10.4.mpkg/Contents/Packages/ there are two mpkgs. IN each of those, there is: Contents/Info.plist which is an xml file. In there, there is: <key>IFPkgFlagDefaultLocation</key> which should be set to: <string>/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages</string> If it was built in a virtualenv, that would be the virtualenv path. It's a hack, but you could post-process the mpkg to change that value after building.

The docs are included in the .dmg, and yes, the doc needs to be built from the same installation (or more exactly the same source).

In my tests, that seems to work fine.

...

True. In that case we could put the dylib somewhere obscure:

/usr/local/lib/scipy1.6/lib/

Hm, that's strange - why /usr/local/lib ? It is outside the scipy installation.

I'm not sure, really, except that that's where Robin Dunn puts stuff for wxPython -- I think one reason may be that he can then point to it from more than one Python installation -- for instance Apple's and python.orgs. And it could help with building from a virtualenv.

...

or even:

/Library/Frameworks/Python.framework/Versions/2.5/lib/

That's potentially dangerous: since this directory is likely to be in LIBDIR, it means libgfortran will be taken there or from /usr/local/lib if the user builds numpy/scipy after installing numpy. If it is incompatible with the user gfortran, it will lead to weird issues, hard to debug.

showing what a pain all of this is! Of course, you could put it in: /Library/Frameworks/Python.framework/Versions/2.5/lib/site_packages/scipy/lib/ or something that only scipy would know about.

If we install something like libgfortran, it should be installed privately - but dynamically linking against private libraries is hard, because that's very platform dependent

yup -- on the Mac, it could work well, 'cause the paths to the libs are hard-coded when linked, so you WILL get the right one -- if it's there! Tough this could lead to trouble when you want to build from a virtualenv, and install to the system location. mocaholib gives you tools to re-write the locations, but someone would have to write that code.

gfortran hello.f -> a.out is 8 kb gfortran hello.f -static-libgfortran -> a.out is 130 kb libgfortran.a -> ~ 1.3Mb

And thinking about it: mac os x rather encourage big binaries - "fat binary", so I am not sure it is a big concern.

so I guess we're back to static linking -- I guess that's why it's generally "recommended" for distributing binaries.

...

Also, would it break anything if the libgfortran installed were properly versioned: libgfortran.a.b.c

versioned libraries only make sense for shared libraries,I think.

right -- I meant for the dynamic lib option.

On Linux, the static library is not even publicly available (it is in /usr/lib/gcc/4.3.3). I wonder whether the mac os x gfortran binary did not make a mistake there, actually.

where are you getting that? I'm about to go on vacation, but maybe I could try a few things... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Christopher Barker wrote:

I'm not surprised it took that long -- that sounds short to me!

Anyway, If there are new python builds that are 64-bit (quad?) you wont' have to change much -- "only" make sure that the libs you are linking to are 64 bit. I suppose you could try to get a quad-universal gfortran.a now, but I'd wait 'till you need it.

My main concern was about making sure I always built against the same python interpreter. For now, I have hardcoded the python executable to use for the mpkg, but that's not good enough.

It did a lot, but I don't THINK is re-compiled everthing. I think the trick is that eveything it built was still in the build dir -- and it is the saem python, even though it's not living in the same place.

Well, yes, it could be a totally different, incompatible python that it would still do as you said - distutils cannot be trusted at all to do the right thing here, it has no real dependency system. The problem is to make sure that bdist_mpkg and the installed numpy in virtual env are built against the same python binary: - it makes sure the build is actually compatible - it also gives a sanity check about 'runnability' of the binary - at least, numpy can be imported (for the doc), and by building from scratch, I lose this. Of course, the real solution would be to automatically test for the built numpy - I unfortunately did not have the time to do this correctly,

<key>IFPkgFlagDefaultLocation</key>

which should be set to:

<string>/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages</string>

If it was built in a virtualenv, that would be the virtualenv path.

It's a hack, but you could post-process the mpkg to change that value after building.

This has the same problem - it works as long as the virtual python and the 'python targetted by the binary installer are the same'.

showing what a pain all of this is! Of course, you could put it in:

/Library/Frameworks/Python.framework/Versions/2.5/lib/site_packages/scipy/lib/

or something that only scipy would know about.

That's the only correct solution I can see, yes.

...

On Linux, the static library is not even publicly available (it is in /usr/lib/gcc/4.3.3). I wonder whether the mac os x gfortran binary did not make a mistake there, actually.

where are you getting that? I'm about to go on vacation, but maybe I could try a few things...

On linux, libgfortran.a is /usr/lib/gcc/i486-linux-gnu/4.3/libgfortran.a -> this is private On mac OS X, it is /usr/local/lib -> this is public Exactly the kind of 'small' things which end up with quite some headache when you care about reliability and repeatability. As a related problem, I am not a big fan about the Apple way of building fat binaries, I much prefer the approach one build/arch and merging the binaries after build with lipo. This is more compatible with all autoconf projects out there - and it means using/updating the compilers is easy (building multi-arch binaries the 'apple' way is really a pain - the scripts used by R, to build the universal gfortran, is quite complicated). I think this above problem would not have happened with a pristine gfortran built from sources, cheers, David

On Mar 30, 2009, at 2:56 AM, David Cournapeau wrote:

On Mon, Mar 30, 2009 at 3:36 AM, Robert Pyle wrote:

...

I just installed 2.5.4 from python.org, and the OS X installer still doesn't work. This is on a PPC G5; I haven't tried it on my Intel MacBook Pro.

I think I got it. To build numpy, I use virtualenv to make a "bootstrap" environment, but then the corresponding python path get embedded in the .mpkg - so unless you have your python interpreter in exactly the same path as my bootstrap (which is very unlikely), it won't run at all. This would also explain why I never saw the problem.

This is exactly the problem. This is the error message that you get when running the .dmg and no hard drives are available for selection. You cannot install numpy 1.3.0rc1 on this volume. numpy requires /Users/david/src/dsp/numpy/1.3.x/bootstrap Python 2.5 to install.

I will prepare a new binary,

Any idea when a new binary will be available on sourceforge.net? Cheers Tommy

On Mon, Mar 30, 2009 at 10:28 AM, Mark Sienkiewicz wrote:

Numpy 1.3.0 rc1 fails this self-test on Solaris.

====================================================================== FAIL: Test find_duplicates ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/ra/pyssg/2.5.1/numpy/lib/tests/test_recfunctions.py", line 163, in test_find_duplicates assert_equal(test[0], a[control]) File "/usr/stsci/pyssgdev/2.5.1/numpy/ma/testutils.py", line 121, in assert_equal return assert_array_equal(actual, desired, err_msg) File "/usr/stsci/pyssgdev/2.5.1/numpy/ma/testutils.py", line 193, in assert_array_equal header='Arrays are not equal') File "/usr/stsci/pyssgdev/2.5.1/numpy/ma/testutils.py", line 186, in assert_array_compare verbose=verbose, header=header) File "/usr/stsci/pyssgdev/2.5.1/numpy/testing/utils.py", line 395, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal

(mismatch 50.0%) x: array([(1, (2.0, 'B')), (2, (2.0, 'B')), (2, (2.0, 'B')), (1, (2.0, 'B'))], dtype=[('A', '>i4'), ('B', [('BA', '>f8'), ('BB', '|S1')])]) y: array([(2, (2.0, 'B')), (1, (2.0, 'B')), (2, (2.0, 'B')), (1, (2.0, 'B'))], dtype=[('A', '>i4'), ('B', [('BA', '>f8'), ('BB', '|S1')])])

----------------------------------------------------------------------

The software I am using:

NumPy version 1.3.0rc1 Python version 2.5.1 (r251:54863, Jun 4 2008, 15:48:19) [C] nose version 0.10.4

These are new (two months old) tests. Hmm, they are also marked as known failures on win32. I wonder why they fail there and not on linux? I think you should open a ticket for this. Chuck

On Mon, Mar 30, 2009 at 3:04 PM, Mark Sienkiewicz wrote:

...

...

====================================================================== FAIL: Test find_duplicates ---------------------------------------------------------------------- ... AssertionError: Arrays are not equal

(mismatch 50.0%) x: array([(1, (2.0, 'B')), (2, (2.0, 'B')), (2, (2.0, 'B')), (1, (2.0, 'B'))], dtype=[('A', '>i4'), ('B', [('BA', '>f8'), ('BB', '|S1')])]) y: array([(2, (2.0, 'B')), (1, (2.0, 'B')), (2, (2.0, 'B')), (1, (2.0, 'B'))], dtype=[('A', '>i4'), ('B', [('BA', '>f8'), ('BB', '|S1')])])

----------------------------------------------------------------------

These are new (two months old) tests. Hmm, they are also marked as known failures on win32. I wonder why they fail there and not on linux? I think you should open a ticket for this.

I'm not sure how old the test is, but I see that it has been failing since Feb 1. (That is the earliest report I have online at the moment.)

The ticket is http://projects.scipy.org/numpy/ticket/1039 . I added this specific failure mode to the ticket today.

It does not surprise me at all when the trunk is broken on solaris. I'm mentioning it on the list because I see it is still broken in the release candidate. I assume somebody would want to either fix the problem or remove the non-working feature from the release.

I'm guessing that it is the test that needs fixing. And maybe the windows problem is related. Chuck

Pauli Virtanen wrote:

Probably they are both related to unspecified sort order for the duplicates. There were some sort-order ignoring missing in the test.

I think the test is now fixed in trunk:

http://projects.scipy.org/numpy/changeset/6827

The test passes in 1.4.0.dev6827. Tested on Solaris 8, Mac OSX 10.4 (Tiger) on x86 and ppc, and both 32 and 64 bit Red Hat Enterprise, all with Python 2.5.1. Thanks for fixing this. Mark S.