Mailman 3 NumPy and Python 2.6 on Windows - NumPy-Discussion

newer
Should object arrays have a buffer...

NumPy and Python 2.6 on Windows

Lenard Lindstrom

27 Dec 2008 27 Dec '08

8:05 p.m.

Hi everyone, I build the Pygame dependencies for Windows. With the next Pygame release, 1.9.0, we would like to include Python 2.6 support. As you already know, Pygame has NumPy bindings. Though NumPy is not required, it is a useful addition. I understand NumPy is built with MinGW on Windows, which I use to with Pygame and its dependencies. I know the problems linking against msvcr90.dll. I am willing to offer what advice I can to get NumPy up and running for Python 2.6. You are welcome to use the Pygame build tools if they will help. I also have a Python 2.6 build of NumPy 1.2.1 for Pygame testing. http://www3.telus.net/len_l/pygame/numpy-1.2.1.win32-py2.6.msi md5sum: b791f5c4b620da21f779b53252b5932e *numpy-1.2.1.win32-py2.6.msi Lenard -- Lenard Lindstrom

Show replies by date

Robert Kern

27 Dec 27 Dec

9:47 p.m.

On Sat, Dec 27, 2008 at 15:05, Lenard Lindstrom wrote:

...

Hi everyone,

I build the Pygame dependencies for Windows. With the next Pygame release, 1.9.0, we would like to include Python 2.6 support. As you already know, Pygame has NumPy bindings. Though NumPy is not required, it is a useful addition. I understand NumPy is built with MinGW on Windows, which I use to with Pygame and its dependencies. I know the problems linking against msvcr90.dll. I am willing to offer what advice I can to get NumPy up and running for Python 2.6.

Yes, please. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

David Cournapeau

28 Dec 28 Dec

2:10 a.m.

Hi Lenard, On Sun, Dec 28, 2008 at 5:05 AM, Lenard Lindstrom wrote:

...

Hi everyone,

I build the Pygame dependencies for Windows. With the next Pygame release, 1.9.0, we would like to include Python 2.6 support. As you already know, Pygame has NumPy bindings. Though NumPy is not required, it is a useful addition. I understand NumPy is built with MinGW on Windows, which I use to with Pygame and its dependencies. I know the problems linking against msvcr90.dll. I am willing to offer what advice I can to get NumPy up and running for Python 2.6.

Thanks. I think I have covered most problems concerning python 2.6 and windows in the trunk (upcoming 1.3): - linking against msvcr90.dll - generating manifest for running code snippets (with mingw) - fix some bugs with python 2.6 msvc support (in particular http://bugs.python.org/issue4702) You are welcome to test the trunk to see if that fixes everything. I don't think everything can be fixed for 1.2.2, because the changes are not all trivial (much revamp C99 math support, in particular). Unfortunately, I have been working on some formatting issues which were more difficult than previously thought, and it was time to go to sleep before I actually fixed the problem, so the trunk may be broken ATM. I will fix this now, cheers, David

Lenard Lindstrom

2:55 a.m.

David Cournapeau wrote:

...

Hi Lenard,

On Sun, Dec 28, 2008 at 5:05 AM, Lenard Lindstrom wrote:

...
Hi everyone,

I build the Pygame dependencies for Windows. With the next Pygame release, 1.9.0, we would like to include Python 2.6 support. As you already know, Pygame has NumPy bindings. Though NumPy is not required, it is a useful addition. I understand NumPy is built with MinGW on Windows, which I use to with Pygame and its dependencies. I know the problems linking against msvcr90.dll. I am willing to offer what advice I can to get NumPy up and running for Python 2.6.

Thanks. I think I have covered most problems concerning python 2.6 and windows in the trunk (upcoming 1.3):

- linking against msvcr90.dll - generating manifest for running code snippets (with mingw) - fix some bugs with python 2.6 msvc support (in particular http://bugs.python.org/issue4702)

You are welcome to test the trunk to see if that fixes everything. I don't think everything can be fixed for 1.2.2, because the changes are not all trivial (much revamp C99 math support, in particular).

Unfortunately, I have been working on some formatting issues which were more difficult than previously thought, and it was time to go to sleep before I actually fixed the problem, so the trunk may be broken ATM. I will fix this now,

It looks like you have a handle on the problem. How did you get around the problems with the incomplete libmsvcr90.a import library? I have custom import libraries which you can use if needed. Lenard -- Lenard Lindstrom

David Cournapeau

2:58 a.m.

Lenard Lindstrom wrote:

...

David Cournapeau wrote:

...
Hi Lenard,

On Sun, Dec 28, 2008 at 5:05 AM, Lenard Lindstrom wrote:

...
Hi everyone,

I build the Pygame dependencies for Windows. With the next Pygame release, 1.9.0, we would like to include Python 2.6 support. As you already know, Pygame has NumPy bindings. Though NumPy is not required, it is a useful addition. I understand NumPy is built with MinGW on Windows, which I use to with Pygame and its dependencies. I know the problems linking against msvcr90.dll. I am willing to offer what advice I can to get NumPy up and running for Python 2.6.

Thanks. I think I have covered most problems concerning python 2.6 and windows in the trunk (upcoming 1.3):

- linking against msvcr90.dll - generating manifest for running code snippets (with mingw) - fix some bugs with python 2.6 msvc support (in particular http://bugs.python.org/issue4702)

You are welcome to test the trunk to see if that fixes everything. I don't think everything can be fixed for 1.2.2, because the changes are not all trivial (much revamp C99 math support, in particular).

Unfortunately, I have been working on some formatting issues which were more difficult than previously thought, and it was time to go to sleep before I actually fixed the problem, so the trunk may be broken ATM. I will fix this now,

It looks like you have a handle on the problem. How did you get around the problems with the incomplete libmsvcr90.a import library? I have custom import libraries which you can use if needed.

Do you mean on xp 32 bits or 64 bits ? For the later, I have yet to submit patchs to the mingw-w64 project - the whole libmsvcr90.a is missing, actually. For 32 bits, I simply got around it by changing the missing functions in numpy itself - if we are talking about the same thing, that is missing time functions for random. You can look at revisions r6080, r6076, r6074, r6073, r6072,r6070,r6069,r6029,r6028, the final patch is as follow: diff --git a/numpy/random/mtrand/randomkit.c b/numpy/random/mtrand/randomkit.c index 56f52c0..0fbc40d 100644 --- a/numpy/random/mtrand/randomkit.c +++ b/numpy/random/mtrand/randomkit.c @@ -64,18 +64,33 @@ /* static char const rcsid[] = "@(#) $Jeannot: randomkit.c,v 1.28 2005/07/21 22:14:09 js Exp $"; */ - #include #include #include #include -#include #include #include #ifdef _WIN32 /* Windows */ +/* XXX: we have to use this ugly defined(__GNUC__) because it is not easy to + * detect the compiler used in distutils itself */ +#if (defined(__GNUC__) && defined(NPY_NEEDS_MINGW_TIME_WORKAROUND)) +/* FIXME: ideally, we should set this to the real version of MSVCRT. We need + * something higher than 0x601 to enable _ftime64 and co */ +#define __MSVCRT_VERSION__ 0x0700 +#include #include +/* mingw msvcr lib import wrongly export _ftime, which does not exist in the + * actual msvc runtime for version >= 8; we make it an alias to _ftime64, which + * is available in those versions of the runtime + */ +#define _FTIME(x) _ftime64((x)) +#else +#include +#include +#define _FTIME(x) _ftime((x)) +#endif #ifndef RK_NO_WINCRYPT /* Windows crypto */ #ifndef _WIN32_WINNT @@ -86,6 +101,7 @@ #endif #else /* Unix */ +#include #include #include #endif @@ -167,7 +183,7 @@ rk_error rk_randomseed(rk_state *state) rk_seed(rk_hash(getpid()) ^ rk_hash(tv.tv_sec) ^ rk_hash(tv.tv_usec) ^ rk_hash(clock()), state); #else - _ftime(&tv); + _FTIME(&tv); rk_seed(rk_hash(tv.time) ^ rk_hash(tv.millitm) ^ rk_hash(clock()), state); #endif diff --git a/numpy/random/setup.py b/numpy/random/setup.py index e7955db..dde3119 100644 --- a/numpy/random/setup.py +++ b/numpy/random/setup.py @@ -1,13 +1,19 @@ -from os.path import join, split +from os.path import join, split, dirname +import os import sys +from distutils.dep_util import newer +from distutils.msvccompiler import get_build_version as get_msvc_build_version -def msvc_version(): - """Return the msvc version used to build the running python, None if not - built with MSVC.""" - msc_pos = sys.version.find('MSC v.') - if msc_pos != -1: - return sys.version[msc_pos+6:msc_pos+10] - return None +def needs_mingw_ftime_workaround(): + # We need the mingw workaround for _ftime if the msvc runtime version is + # 7.1 or above and we build with mingw ... + # ... but we can't easily detect compiler version outside distutils command + # context, so we will need to detect in randomkit whether we build with gcc + msver = get_msvc_build_version() + if msver and msver >= 8: + return True + + return False def configuration(parent_package='',top_path=None): from numpy.distutils.misc_util import Configuration, get_mathlibs @@ -22,6 +28,10 @@ def configuration(parent_package='',top_path=None): ext.libraries.extend(libs) return None + defs = [] + if needs_mingw_ftime_workaround(): + defs.append(("NPY_NEEDS_MINGW_TIME_WORKAROUND", None)) + libs = [] # Configure mtrand config.add_extension('mtrand', @@ -32,7 +42,8 @@ def configuration(parent_package='',top_path=None): depends = [join('mtrand','*.h'), join('mtrand','*.pyx'), join('mtrand','*.pxi'), - ] + ], + define_macros = defs, ) config.add_data_files(('.', join('mtrand', 'randomkit.h'))) David

Lenard Lindstrom

3:50 a.m.

...

Lenard Lindstrom wrote:

...
David Cournapeau wrote:

...
Hi Lenard,

On Sun, Dec 28, 2008 at 5:05 AM, Lenard Lindstrom wrote:

...
Hi everyone,

[snip]

...
I am willing to offer what advice I can to get NumPy up and running for Python 2.6.

Thanks. I think I have covered most problems concerning python 2.6 and windows in the trunk (upcoming 1.3)[.]

[snip]

...
It looks like you have a handle on the problem. How did you get around the problems with the incomplete libmsvcr90.a import library? I have custom import libraries which you can use if needed.

Do you mean on xp 32 bits or 64 bits ? For the later, I have yet to submit patchs to the mingw-w64 project - the whole libmsvcr90.a is missing, actually. For 32 bits, I simply got around it by changing the missing functions in numpy itself - if we are talking about the same thing, that is missing time functions for random. Yes, the _ftime function, which is an inlined function in VC 2008 that calls _ftime64. I have to build a lot of dependencies for Pygame so I want to avoid patching code when possible. Instead I have a custom

David Cournapeau wrote: libmsvcr90.a that has stub functions for the various time functions. It lets me create static libraries that link to both msvcr71.dll and msvcr90.dll. No manifest files required. And no patches to MinGW. -- Lenard Lindstrom

David Cournapeau

3:48 a.m.

Lenard Lindstrom wrote:

...

David Cournapeau wrote:

...
Lenard Lindstrom wrote:

...
David Cournapeau wrote:

...
Hi Lenard,

On Sun, Dec 28, 2008 at 5:05 AM, Lenard Lindstrom wrote:

...
Hi everyone,

[snip]

...
...
...
...
I am willing to offer what advice I can to get NumPy up and running for Python 2.6.

Thanks. I think I have covered most problems concerning python 2.6 and windows in the trunk (upcoming 1.3)[.]

[snip]

...
...
...
It looks like you have a handle on the problem. How did you get around the problems with the incomplete libmsvcr90.a import library? I have custom import libraries which you can use if needed.

Do you mean on xp 32 bits or 64 bits ? For the later, I have yet to submit patchs to the mingw-w64 project - the whole libmsvcr90.a is missing, actually. For 32 bits, I simply got around it by changing the missing functions in numpy itself - if we are talking about the same thing, that is missing time functions for random.

Yes, the _ftime function, which is an inlined function in VC 2008 that calls _ftime64. I have to build a lot of dependencies for Pygame so I want to avoid patching code when possible.

I understand you don't want to patch the sources. The above fix is in the trunk, though - and I don't feel like backporting those fixes in the 1.2.x branch, because it would be a lot of work.

...

Instead I have a custom libmsvcr90.a that has stub functions for the various time functions. It lets me create static libraries that link to both msvcr71.dll and msvcr90.dll. No manifest files required. And no patches to MinGW.

Manifests are needed for any executable linking against msvcr90.dll, whether you build with mingw or VS: this is required by windows itself to be able to load msvcr90.dll at all (the dreadful Side by Side assembly stuff). This is a totally independent issue of the _ftime thing, and AFAIK, there is no way around it - except installing msvcrt90.dll in system32 yourself, which is obviously a very bad idea. Patching mingw is necessary for 64 bits support, since their headers are missing some math functions - no patch is needed for 32 bits. cheers, David

Lenard Lindstrom

4:34 a.m.

David Cournapeau wrote:

...

Lenard Lindstrom wrote:

...
David Cournapeau wrote:

...
Do you mean on xp 32 bits or 64 bits ? For the later, I have yet to submit patchs to the mingw-w64 project - the whole libmsvcr90.a is missing, actually. For 32 bits, I simply got around it by changing the missing functions in numpy itself - if we are talking about the same thing, that is missing time functions for random.

Yes, the _ftime function, which is an inlined function in VC 2008 that calls _ftime64. I have to build a lot of dependencies for Pygame so I want to avoid patching code when possible.

I understand you don't want to patch the sources. The above fix is in the trunk, though - and I don't feel like backporting those fixes in the 1.2.x branch, because it would be a lot of work.

Sorry for the confusion. I meant that I don't like patching SDL and such. I build these in msys using the "configure; make install" incantation, so can't easily use magic like manifest files. Instead I link the DLLs against msvcr71.dll, making sure there are also static libraries, then create the msvcr90.dll linked DLL's from the static libraries. This trick can also be used with the NumPy dependencies Blas and fftw. Actually it is easier, since they are statically linked into NumPy.

...

...
Instead I have a custom libmsvcr90.a that has stub functions for the various time functions. It lets me create static libraries that link to both msvcr71.dll and msvcr90.dll. No manifest files required. And no patches to MinGW.

Manifests are needed for any executable linking against msvcr90.dll, whether you build with mingw or VS: this is required by windows itself to be able to load msvcr90.dll at all (the dreadful Side by Side assembly stuff). This is a totally independent issue of the _ftime thing, and AFAIK, there is no way around it - except installing msvcrt90.dll in system32 yourself, which is obviously a very bad idea.

Patching mingw is necessary for 64 bits support, since their headers are missing some math functions - no patch is needed for 32 bits.

Yes, I've had my run in with manifest files. I avoid them by linking test programs against msvcrt or msvcr71 instead. A manifest is not needed for a DLL, luckily, as it uses the DLL libraries loaded by its host program. And I've had no luck using msvcr90.dll outside an SxS assembly. It needs a manifest file wherever it is, and I have had yet to writing a working manifest for a private assembly. So the Python developers' solution of copying msvcr90.dll into the Python directory is of no help. Lenard -- Lenard Lindstrom

David Cournapeau

4:45 a.m.

Lenard Lindstrom wrote:

...

Sorry for the confusion. I meant that I don't like patching SDL and such. I build these in msys using the "configure; make install" incantation, so can't easily use magic like manifest files.

I don't know about SDL :) numpy needs manifests at the configuration stage, because it build code snippet which it needs to run, and this requires manifest AFAIK, if you link against msvcr90.dll. I did not want to link with another dll at the configuration stage, because this could lead to subtle issues.

...

Instead I link the DLLs against msvcr71.dll, making sure there are also static libraries, then create the msvcr90.dll linked DLL's from the static libraries. This trick can also be used with the NumPy dependencies Blas and fftw. Actually it is easier, since they are statically linked into NumPy.

Note that numpy does not depend on fftw. My solution to this was to generate the manifest file, which was relatively easy to do since we control our build process in python (look at numpy/distutils/mingw32compiler.py). To run a program 'locally' (one which is not installed), having the manifest in the same directory as the .exe is enough, so we only need to generate it. The main difficulty is to make sure you are using the same version of the dll as python: this feature has been integrated in python 2.6.1. For 2.6.0, I just assume it is the same as the official python binary.

...

Yes, I've had my run in with manifest files. I avoid them by linking test programs against msvcrt or msvcr71 instead. A manifest is not needed for a DLL, luckily, as it uses the DLL libraries loaded by its host program.

Yes, that's my understanding too. Since this is of course undocumented, we can only guess.

...

And I've had no luck using msvcr90.dll outside an SxS assembly. It needs a manifest file wherever it is, and I have had yet to writing a working manifest for a private assembly. So the Python developers' solution of copying msvcr90.dll into the Python directory is of no help.

This may be of some interest for you: http://cournape.wordpress.com/2008/09/02/how-to-embed-a-manifest-into-a-dll-... that's a summary of my own findings about manifest using only open source tools. All the necessary code, including the manifest template is in the mingw32compiler.py file: http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/distutils/mingw32c... cheers, David

Lenard Lindstrom

6:33 a.m.

David Cournapeau wrote:

...

Lenard Lindstrom wrote:

...
Sorry for the confusion. I meant that I don't like patching SDL and such. I build these in msys using the "configure; make install" incantation, so can't easily use magic like manifest files.

I don't know about SDL :) numpy needs manifests at the configuration stage, because it build code snippet which it needs to run, and this requires manifest AFAIK, if you link against msvcr90.dll. I did not want to link with another dll at the configuration stage, because this could lead to subtle issues.

Yes, it is best to have configuration programs link to msvcr90.dll when possible. Pygame configuration is relatively straight forward. No test programs are used. But all the dependencies, such as SDL, are built with Msys, and the Unix configuration shell scripts are used when possible. These scripts do create small test programs. It would be possible provide manifest files in this case, but only after determining the names and locations of all the test programs generated. For now it is simpler to just use a less fussy C runtime during configuration, then link in msvcr90 later.

...

...
Instead I link the DLLs against msvcr71.dll, making sure there are also static libraries, then create the msvcr90.dll linked DLL's from the static libraries. This trick can also be used with the NumPy dependencies Blas and fftw. Actually it is easier, since they are statically linked into NumPy.

Note that numpy does not depend on fftw.

My solution to this was to generate the manifest file, which was relatively easy to do since we control our build process in python (look at numpy/distutils/mingw32compiler.py). To run a program 'locally' (one which is not installed), having the manifest in the same directory as the .exe is enough, so we only need to generate it. The main difficulty is to make sure you are using the same version of the dll as python: this feature has been integrated in python 2.6.1. For 2.6.0, I just assume it is the same as the official python binary.

A manifest file will work as long as the public key doesn't change. Or is the key provided by the developer rather than Microsoft's build tools?

...

...
Yes, I've had my run in with manifest files. I avoid them by linking test programs against msvcrt or msvcr71 instead. A manifest is not needed for a DLL, luckily, as it uses the DLL libraries loaded by its host program.

Yes, that's my understanding too. Since this is of course undocumented, we can only guess.

Well, this is what I've found for Python and Pygame extension modules anyway. And it makes a certain amount of sense if manifests exist to prevent library conflicts. A dynamic library and main program using different C runtime instances would be a problem.

...

...
And I've had no luck using msvcr90.dll outside an SxS assembly. It needs a manifest file wherever it is, and I have had yet to writing a working manifest for a private assembly. So the Python developers' solution of copying msvcr90.dll into the Python directory is of no help.

This may be of some interest for you:

http://cournape.wordpress.com/2008/09/02/how-to-embed-a-manifest-into-a-dll-...

that's a summary of my own findings about manifest using only open source tools. All the necessary code, including the manifest template is in the mingw32compiler.py file:

http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/distutils/mingw32c...

Ok, you have given me some more things to consider. Thanks for the discussion. I will keep you suggestions in mind. Just to clear up any confusion pygame.org is not intending to build and distribute NumPy with Pygame. NumPy was required to run the full Pygame test suite under Python 2.6. My custom NumPy build will disappear once scipy.org releases its Python 2.6 build. Lenard -- Lenard Lindstrom

Lenard Lindstrom

4:06 a.m.

Lenard Lindstrom wrote:

...

David Cournapeau wrote:

...
Do you mean on xp 32 bits or 64 bits ? For the later, I have yet to submit patchs to the mingw-w64 project - the whole libmsvcr90.a is missing, actually. For 32 bits, I simply got around it by changing the missing functions in numpy itself - if we are talking about the same thing, that is missing time functions for random.

Yes, the _ftime function, which is an inlined function in VC 2008 that calls _ftime64. I have to build a lot of dependencies for Pygame so I want to avoid patching code when possible. Instead I have a custom libmsvcr90.a that has stub functions for the various time functions. It lets me create static libraries that link to both msvcr71.dll and msvcr90.dll. No manifest files required. And no patches to MinGW.

It just occurred to me: -D_ftime=_ftime64. I will have to see if this works with gmtime in the png library. Thanks for the advice. Lenard -- Lenard Lindstrom

David Cournapeau

3:04 a.m.

David Cournapeau wrote:

...

Hi Lenard,

On Sun, Dec 28, 2008 at 5:05 AM, Lenard Lindstrom wrote:

...
Hi everyone,

I build the Pygame dependencies for Windows. With the next Pygame release, 1.9.0, we would like to include Python 2.6 support. As you already know, Pygame has NumPy bindings. Though NumPy is not required, it is a useful addition. I understand NumPy is built with MinGW on Windows, which I use to with Pygame and its dependencies. I know the problems linking against msvcr90.dll. I am willing to offer what advice I can to get NumPy up and running for Python 2.6.

Thanks. I think I have covered most problems concerning python 2.6 and windows in the trunk (upcoming 1.3):

- linking against msvcr90.dll - generating manifest for running code snippets (with mingw) - fix some bugs with python 2.6 msvc support (in particular http://bugs.python.org/issue4702)

You are welcome to test the trunk to see if that fixes everything. I don't think everything can be fixed for 1.2.2, because the changes are not all trivial (much revamp C99 math support, in particular).

Unfortunately, I have been working on some formatting issues which were more difficult than previously thought, and it was time to go to sleep before I actually fixed the problem, so the trunk may be broken ATM. I will fix this now,

I have reverted the buggy changes, so the trunk should be usable again, and contain all the fixes so far for mingw + python 2.6 support (including mingw-w64). cheers, David

5591

Age (days ago)

5592

Last active (days ago)

List overview

Download

11 comments

4 participants

participants (4)

David Cournapeau
David Cournapeau
Lenard Lindstrom
Robert Kern

NumPy and Python 2.6 on Windows

Lenard Lindstrom

Robert Kern

David Cournapeau

Lenard Lindstrom

David Cournapeau

Lenard Lindstrom

David Cournapeau

Lenard Lindstrom

David Cournapeau

Lenard Lindstrom

Lenard Lindstrom

David Cournapeau

tags

participants (4)