how to use numpy-vendor
Hi, How should one use the "vendor" repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README. I've played with OpenBlas and managed to compile numpy with it on linux, following the tutorial [1] and it works, so at least on linux it's clear to me. In wine, since all the binaries are .a files, the only way to check that it works is to install it and check that things like eigh() are much faster. Is there some other way? On linux using openblas as an .so library, I just do "ldd" and all is clear. Ondrej [1] http://www.der-schnorz.de/2012/06/optimized-linear-algebra-and-numpyscipy/
Hi Ondrej, On Tue, Aug 14, 2012 at 5:34 AM, Ondřej Čertík <ondrej.certik@gmail.com> wrote:
Hi,
How should one use the "vendor" repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README.
There is no information on vendor: that's a repo I set up to avoid polluting the main repo with all the binary stuff that used to be in SVN. The principle is to put binaries used to *build* numpy, but we don't put anything there for end-users. What binaries do you need to put there ? Numpy binaries are usually put on sourceforge (although I would be more than happy to have a suggestion for a better way because uploading on sourceforge is the very definition of pain). David
On Tue, Aug 14, 2012 at 11:06 AM, David Cournapeau <cournape@gmail.com> wrote:
Hi Ondrej,
On Tue, Aug 14, 2012 at 5:34 AM, Ondřej Čertík <ondrej.certik@gmail.com> wrote:
Hi,
How should one use the "vendor" repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README.
There is no information on vendor: that's a repo I set up to avoid polluting the main repo with all the binary stuff that used to be in SVN. The principle is to put binaries used to *build* numpy, but we don't put anything there for end-users.
What binaries do you need to put there ? Numpy binaries are usually put on sourceforge (although I would be more than happy to have a suggestion for a better way because uploading on sourceforge is the very definition of pain).
I think he's asking how to use the binaries in numpy-vendor to build a release version of numpy. -n
On Tue, Aug 14, 2012 at 11:22 AM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Aug 14, 2012 at 11:06 AM, David Cournapeau <cournape@gmail.com> wrote:
Hi Ondrej,
On Tue, Aug 14, 2012 at 5:34 AM, Ondřej Čertík <ondrej.certik@gmail.com> wrote:
Hi,
How should one use the "vendor" repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README.
There is no information on vendor: that's a repo I set up to avoid polluting the main repo with all the binary stuff that used to be in SVN. The principle is to put binaries used to *build* numpy, but we don't put anything there for end-users.
What binaries do you need to put there ? Numpy binaries are usually put on sourceforge (although I would be more than happy to have a suggestion for a better way because uploading on sourceforge is the very definition of pain).
I think he's asking how to use the binaries in numpy-vendor to build a release version of numpy.
Hm, good point, I don't know why I read putting .wine stuff into vendor instead of the opposite. Anyway, the way to use the binaries is to put them in some known location, e.g. C:\local ($WINEPREFIX/drive_c/local for wine), and copy the nosse/sse2/sse3 directories in there. For example: C:\local\lib\yop\nosse C:\local\lib\yop\sse2 ... This is then referred through env by the pavement script (see https://github.com/numpy/numpy/blob/master/pavement.py#L143). Renaming yop to atlas would be a good idea, don't know why I let that non-descriptive name in there. Manually, you can just do something like "ATLAS=C:\local\lib\yop\sse2 python setup.py build", but being careful about how env variables are passed between shell and wine (don't remember the details). Note that the nosse is not ATLAS, but straight netlib libs, which is why in that case you need to use BLAS=... LAPACK=... I would strongly suggest not to use openblas for this release, because of all the issues related to CPU tuning. We could certainly update a bit what we have in there, but building windows binaries is big enough of a pain, that you don't want to do everything at once I think, especially testing/building blas on windows is very time consuming. David
On Tue, Aug 14, 2012 at 3:43 AM, David Cournapeau <cournape@gmail.com> wrote:
On Tue, Aug 14, 2012 at 11:22 AM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Aug 14, 2012 at 11:06 AM, David Cournapeau <cournape@gmail.com> wrote:
Hi Ondrej,
On Tue, Aug 14, 2012 at 5:34 AM, Ondřej Čertík <ondrej.certik@gmail.com> wrote:
Hi,
How should one use the "vendor" repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README.
There is no information on vendor: that's a repo I set up to avoid polluting the main repo with all the binary stuff that used to be in SVN. The principle is to put binaries used to *build* numpy, but we don't put anything there for end-users.
What binaries do you need to put there ? Numpy binaries are usually put on sourceforge (although I would be more than happy to have a suggestion for a better way because uploading on sourceforge is the very definition of pain).
I think he's asking how to use the binaries in numpy-vendor to build a release version of numpy.
Yes.
Hm, good point, I don't know why I read putting .wine stuff into vendor instead of the opposite.
Anyway, the way to use the binaries is to put them in some known location, e.g. C:\local ($WINEPREFIX/drive_c/local for wine), and copy the nosse/sse2/sse3 directories in there. For example:
C:\local\lib\yop\nosse C:\local\lib\yop\sse2 ...
This is then referred through env by the pavement script (see https://github.com/numpy/numpy/blob/master/pavement.py#L143). Renaming yop to atlas would be a good idea, don't know why I let that non-descriptive name in there.
I'll send a PR. Got it, thanks for your help. I'll also send a PR to the "vendor" repository, so that it's clear how to actually use it with NumPy.
Manually, you can just do something like "ATLAS=C:\local\lib\yop\sse2 python setup.py build", but being careful about how env variables are passed between shell and wine (don't remember the details). Note that
Right.
the nosse is not ATLAS, but straight netlib libs, which is why in that case you need to use BLAS=... LAPACK=...
I would strongly suggest not to use openblas for this release, because of all the issues related to CPU tuning. We could certainly update a bit what we have in there, but building windows binaries is big enough of a pain, that you don't want to do everything at once I think, especially testing/building blas on windows is very time consuming.
Absolutely, I don't plan to use nor ship openblase. My apologies for the confusion. I was just using it on linux to understand how to make numpy use it (with the ATLAS, BLAS and LAPACK env variables). Ondrej
On Tue, Aug 14, 2012 at 7:26 AM, Ondřej Čertík <ondrej.certik@gmail.com> wrote:
On Tue, Aug 14, 2012 at 3:43 AM, David Cournapeau <cournape@gmail.com> wrote: [...]
This is then referred through env by the pavement script (see https://github.com/numpy/numpy/blob/master/pavement.py#L143). Renaming yop to atlas would be a good idea, don't know why I let that non-descriptive name in there.
I'll send a PR:
Hi, I've uploaded the binaries here: https://sourceforge.net/projects/numpy/files/NumPy/1.7.0beta/ The only thing that's missing are Mac binaries, otherwise everything else is there. Here is the full log from the build (you have to click on "View Raw" as the log is long): https://gist.github.com/3352057 What is the best way to test, that they indeed were built with the atlas from the "vendor" repository? The relevant part of the log says: [127.0.0.1:2222] out: lapack_info: [127.0.0.1:2222] out: FOUND: [127.0.0.1:2222] out: libraries = ['lapack'] [127.0.0.1:2222] out: library_dirs = ['C:\\local\\lib\\yop\\nosse'] [127.0.0.1:2222] out: language = f77 [127.0.0.1:2222] out: FOUND: [127.0.0.1:2222] out: libraries = ['lapack', 'blas'] [127.0.0.1:2222] out: library_dirs = ['C:\\local\\lib\\yop\\nosse'] [127.0.0.1:2222] out: define_macros = [('NO_ATLAS_INFO', 1)] [127.0.0.1:2222] out: language = f77 and [127.0.0.1:2222] out: FOUND: [127.0.0.1:2222] out: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] [127.0.0.1:2222] out: library_dirs = ['C:\\local\\lib\\yop\\sse2'] [127.0.0.1:2222] out: language = f77 [127.0.0.1:2222] out: define_macros = [('NO_ATLAS_INFO', -1)] and [127.0.0.1:2222] out: FOUND: [127.0.0.1:2222] out: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] [127.0.0.1:2222] out: library_dirs = ['C:\\local\\lib\\yop\\sse3'] [127.0.0.1:2222] out: language = f77 [127.0.0.1:2222] out: define_macros = [('NO_ATLAS_INFO', -1)] So that seems that it found it, correct? Ondrej
On 08/14/2012 06:34 AM, Ondřej Čertík wrote:
Hi,
How should one use the "vendor" repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README.
I've played with OpenBlas and managed to compile numpy with it on linux, following the tutorial [1] and it works, so at least on linux it's clear to me.
One thing to be aware of with OpenBlas is that it is *very* tuned to the CPU at hand. As in, every CPU has hand-coded *assembly* and the makefile more or less probes for which specific CPU generation from which vendor you have and compiles and link the corresponding assembly file. So you may have to take some care that you don't compile and ship a version that breaks if you don't have SSE3 installed etc... (Unless OpenBlas has changed recently. I'm not saying I'm right, I'm saying it should be looked into.) Dag
In wine, since all the binaries are .a files, the only way to check that it works is to install it and check that things like eigh() are much faster. Is there some other way? On linux using openblas as an .so library, I just do "ldd" and all is clear.
Ondrej
[1] http://www.der-schnorz.de/2012/06/optimized-linear-algebra-and-numpyscipy/ _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Tue, Aug 14, 2012 at 6:33 AM, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no> wrote:
On 08/14/2012 06:34 AM, Ondřej Čertík wrote:
Hi,
How should one use the "vendor" repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README.
I've played with OpenBlas and managed to compile numpy with it on linux, following the tutorial [1] and it works, so at least on linux it's clear to me.
One thing to be aware of with OpenBlas is that it is *very* tuned to the CPU at hand. As in, every CPU has hand-coded *assembly* and the makefile more or less probes for which specific CPU generation from which vendor you have and compiles and link the corresponding assembly file. So you may have to take some care that you don't compile and ship a version that breaks if you don't have SSE3 installed etc...
(Unless OpenBlas has changed recently. I'm not saying I'm right, I'm saying it should be looked into.)
OpenBlas as the option to build all version and select at run time the right one. MKL do the same. But I never tested it. So I don't know how well it work. The other option would be to force an older CPU that support only sse2. ATLAS do that by default. OpenBLAS select the best one for the computer where it is being built by default. Fred
On Tue, Aug 14, 2012 at 6:32 AM, Frédéric Bastien <nouiz@nouiz.org> wrote:
On Tue, Aug 14, 2012 at 6:33 AM, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no> wrote:
On 08/14/2012 06:34 AM, Ondřej Čertík wrote:
Hi,
How should one use the "vendor" repository (https://github.com/numpy/vendor) in Wine? Should I put the binaries into .wine/drive_c/Python25/libs/, or somewhere else? I've search all mailinglists and I didn't find any information on it. I vaguely remember that somebody mentioned it somewhere, but I am not able to find it. Once I understand it, I'll send a PR updating the README.
I've played with OpenBlas and managed to compile numpy with it on linux, following the tutorial [1] and it works, so at least on linux it's clear to me.
One thing to be aware of with OpenBlas is that it is *very* tuned to the CPU at hand. As in, every CPU has hand-coded *assembly* and the makefile more or less probes for which specific CPU generation from which vendor you have and compiles and link the corresponding assembly file. So you may have to take some care that you don't compile and ship a version that breaks if you don't have SSE3 installed etc...
(Unless OpenBlas has changed recently. I'm not saying I'm right, I'm saying it should be looked into.)
OpenBlas as the option to build all version and select at run time the right one. MKL do the same.
But I never tested it. So I don't know how well it work. The other option would be to force an older CPU that support only sse2. ATLAS do that by default. OpenBLAS select the best one for the computer where it is being built by default.
For the record, I don't plan to ship openblas, I was just playing with it to make sure I understand how things work. Ondrej
participants (5)
-
Dag Sverre Seljebotn -
David Cournapeau -
Frédéric Bastien -
Nathaniel Smith -
Ondřej Čertík