[ANN] numscons 0.4.1. Building numpy with MS compiler + g77 works !
Hi, I've just finished a new release of numscons. This version is the first one to be able to compile numpy using MS compilers and g77 compiler on windows, which was the last "big" platform not supported by numscons. This also means I can now refocus on building scipy with numscons, so that the whole numpy/scipy framework can be built with numscons on all platforms. This release is available through pypi (just use easy_install numscons, or easy_install -U numscons to upgrade a previous install), as well as on launchpad: https://launchpad.net/numpy.scons.support/0.4/0.4.1 cheers, David
Hello, On Feb 18, 2008 9:30 PM, David Cournapeau <cournape@gmail.com> wrote:
Hi,
I've just finished a new release of numscons. This version is the first one to be able to compile numpy using MS compilers and g77 compiler on windows, which was the last "big" platform not supported by numscons.
Good stuff. I noticed that the Launchpad page says: "I decided not to support dynamic linking against 3rd party dll. Because of intrinsics windows limitations, it is impossible to do it in a reliable way without putting too much burden on the maintainer." I might note that I had problems with NumPy and other C code crashing randomly when statically linked against MKL 9.1 on Win32 (probably when using multiple cores). Dynamically linking fixed the problem. I haven't tested with MKL 10 yet. Cheers, Albert
Good stuff. I noticed that the Launchpad page says:
"I decided not to support dynamic linking against 3rd party dll. Because of intrinsics windows limitations, it is impossible to do it in a reliable way without putting too much burden on the maintainer."
I might note that I had problems with NumPy and other C code crashing randomly when statically linked against MKL 9.1 on Win32 (probably when using multiple cores). Dynamically linking fixed the problem. I haven't tested with MKL 10 yet.
If you use an installed ATLAS/MKL/... library, I don't know where is the problem wit linking with them :| Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher
Matthieu Brucher wrote:
If you use an installed ATLAS/MKL/... library, I don't know where is the problem wit linking with them :|
Atlas is not a problem, because if you know how to build a dll for ATLAS, you know how to handle environment variable problems. It is not a purely technical problem, but a combination of both cultural and technical. I know that if I tell you, Matthieu, to put you library path in your path, you won't tell me it does not work if the PATH is changed or something similar. But that's not how most windows users work in my experience. As for the problem: if you use PATH for dll, you need: - to use PATH inside scons, that is polluting the build environment with a variable you cannot control at all. - to add the path of the used libraries into the user environment. I can imagine many things going wrong in this scenario. And guess who a user will complain to if something is messed up ? :) cheers, David
2008/2/19, David Cournapeau <david@ar.media.kyoto-u.ac.jp>:
Matthieu Brucher wrote:
If you use an installed ATLAS/MKL/... library, I don't know where is the problem wit linking with them :|
Atlas is not a problem, because if you know how to build a dll for ATLAS, you know how to handle environment variable problems. It is not a purely technical problem, but a combination of both cultural and technical. I know that if I tell you, Matthieu, to put you library path in your path, you won't tell me it does not work if the PATH is changed or something similar. But that's not how most windows users work in my experience.
Now that you provide an installer for Atlas, it may become the same problem as MKL, can't it ? As for the problem: if you use PATH for dll, you need:
- to use PATH inside scons, that is polluting the build environment with a variable you cannot control at all. - to add the path of the used libraries into the user environment. I can imagine many things going wrong in this scenario. And guess who a user will complain to if something is messed up ? :)
Yes, they will complain to you for sure, but I can't see where dynamic libraries can become a problem : - are they numpy dlls ? is so there is a problem, I agree with you - are they external dlls ? If so, the installer must have set up everything so that everything works, but this is where people bark at the wrong people : the users of the dll and not the installers. You seem to indicate that the problem is the last dlls, am I correct ? Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher
Matthieu Brucher wrote:
Now that you provide an installer for Atlas, it may become the same problem as MKL, can't it ?
It is exactly the same problem, yes. Right now, my installer does not modify the environment at all (like MKL or ACML, actually), and you have to do it manually (add PATH, or put in system32).
Yes, they will complain to you for sure, but I can't see where dynamic libraries can become a problem : - are they numpy dlls ? is so there is a problem, I agree with you
numpy 'dll' (they are technically dll, but are named .pyd on recent version of windows) are mandatory, and is not a problem: it is handled by python (whose import mechanism knows how to import dll not in the path env var).
- are they external dlls ? If so, the installer must have set up everything so that everything works, but this is where people bark at the wrong people : the users of the dll and not the installers. Well, neither ACML or MKL does it, at least the versions I looked into. You have to launch a special cmd.exe (like for VS2005) which import the variables.
cheers, David
Hello, On Feb 19, 2008 11:34 AM, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
Matthieu Brucher wrote:
Now that you provide an installer for Atlas, it may become the same problem as MKL, can't it ?
It is exactly the same problem, yes. Right now, my installer does not modify the environment at all (like MKL or ACML, actually), and you have to do it manually (add PATH, or put in system32).
Have you tried installing the DLLs to C:\Python2x or to the same directory as the numpy .pyd? As far as I know, this should work. Obviously there are some issues: multiple modules might be linked to MKL, in which case you would be duplicating a lot of files, but hey, so it goes. Ideally, all the modules should be using one module to interact with native BLAS and LAPACK. In my opinion, modifying the PATH or installing to System32 are not ways to properly deploy DLLs on Windows. Cheers, Albert
Albert Strasheim wrote:
Have you tried installing the DLLs to C:\Python2x or to the same directory as the numpy .pyd? As far as I know, this should work.
Yes, it does, I think I tried it. But this mean duplicating dll, and more worrying, filesystem manipulations, which I don't like much (windows having this extremely annoying 'feature' of locking files by default, which is another thing I would have to care about).
Obviously there are some issues: multiple modules might be linked to MKL, in which case you would be duplicating a lot of files, but hey, so it goes. Ideally, all the modules should be using one module to interact with native BLAS and LAPACK.
In my opinion, modifying the PATH or installing to System32 are not ways to properly deploy DLLs on Windows.
I agree, but once you've said modifying PATH and system32 are not the right way, and there is no other way to do it, what's your conclusion :) cheers, David
Yes, it does, I think I tried it.
Strange that it worked for you, it didn't for me :| Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher
It is exactly the same problem, yes. Right now, my installer does not modify the environment at all (like MKL or ACML, actually), and you have to do it manually (add PATH, or put in system32).
Have you tried installing the DLLs to C:\Python2x or to the same directory as the numpy .pyd? As far as I know, this should work.
The first solution may indeed work (perhaps in C:\Python2x\Lib as well ?) but the second does not. I tried it several times (same with Linux BTW), the library search is then the default one, not Python's one. Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher
Now that you provide an installer for Atlas, it may become the same problem as MKL, can't it ?
It is exactly the same problem, yes. Right now, my installer does not modify the environment at all (like MKL or ACML, actually), and you have to do it manually (add PATH, or put in system32).
OK ;)
Yes, they will complain to you for sure, but I can't see where dynamic libraries can become a problem : - are they numpy dlls ? is so there is a problem, I agree with you
numpy 'dll' (they are technically dll, but are named .pyd on recent version of windows) are mandatory, and is not a problem: it is handled by python (whose import mechanism knows how to import dll not in the path env var).
Yes, that works as long as there are no "real" dll that were built at the same time.
- are they external dlls ? If so, the installer must have set up
everything so that everything works, but this is where people bark at the wrong people : the users of the dll and not the installers.
Well, neither ACML or MKL does it, at least the versions I looked into. You have to launch a special cmd.exe (like for VS2005) which import the variables.
That's almost stupid... I understand now what you meant :| Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher
Albert Strasheim wrote:
Good stuff. I noticed that the Launchpad page says:
"I decided not to support dynamic linking against 3rd party dll. Because of intrinsics windows limitations, it is impossible to do it in a reliable way without putting too much burden on the maintainer."
I might note that I had problems with NumPy and other C code crashing randomly when statically linked against MKL 9.1 on Win32 (probably when using multiple cores). Dynamically linking fixed the problem. I haven't tested with MKL 10 yet.
Yes, it is a problem, at least it is advertised as such on mkl release notes (static vs dll). I could use PATH and get away with it, but I am reluctant to do that, because it is really fragile. If I could be sure that nobody would complain about it when it does not work, then I would do it in a minute (codewise, it is trivial; actually, the development version of numscons does it, because it is useful for debugging). But by experience with all my other packages, windows installation is always what get me the most emails, because windows is more fragile, and its users less aware of technical issues. I am okay spending time on code, much less spending time on a platform which I do not use, on a project which I am working for free. So it really boils down to either someone pays me to do it, or someone tells me how to do it reliably (I would of course do something which works reliably, if anyone can explain to me how to). cheers, David
participants (4)
-
Albert Strasheim -
David Cournapeau -
David Cournapeau -
Matthieu Brucher