Martin v. Löwis wrote:
That's good to know, but somewhat unrelated to the issue under discussion. While it is clear from your message that you can build python, as well as python extensions with MingW, it is not that clear whether extensions build with mingw will work in the standard 2.4 distribution, or whether you can use the standard 2.4 distribution to build extensions with mingw.
Martin it is somewhat related. :) But seriously, I did mention that which you note, and this is partly the reason why I am following this thread. But you are right. We must resolve that issue as it pertains to the pyMinGW patched & MinGW compiled Python, if you would kindly allow me to. And so I propose the following: [1] Can someone please email me [*] the officially distributed Python24.dll and python.exe, (I do not wish to install the full version yet; if this is too much to ask, then nevermind, I'll download the whole distribution), or alternatively, and more conveniently [2] Can someone who has the official Python 2.4 download the sample extension [**] created using the pyMinGW patched & MinGW compiled Python 2.4 and SWIG? And see if it works? Sources are in the zip file whose details appears bellow. Regards, Khalid [*] Email to this address please: abkhd[-AT-]earth.co.jp [**] pymingw.zip (5.35 KB): Location: http://jove.prohosting.com/iwave/misc/pymingw.zip MD5: b4c1a2ebcb8a00fde89f6efe102d983f -------------------------- Contents in KB: -------------------------- _PYMINGW PYD 9.216 13/12/04 1:00 _pymingw.pyd PYMINGW PY 1.115 13/12/04 1:00 pymingw.py PYMINGW I 278 13/12/04 1:00 pymingw.i EXAMPLE C 848 13/12/04 0:23 example.c SETUP PY 192 13/12/04 0:59 setup.py _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
On Sun, 12 Dec 2004 23:19:55 +0000, A.B., Khalid <abkhd@hotmail.com> wrote:
[2] Can someone who has the official Python 2.4 download the sample extension [**] created using the pyMinGW patched & MinGW compiled Python 2.4 and SWIG? And see if it works? Sources are in the zip file whose details appears bellow.
I tried this out, and from some basic uses, it seems to work OK. However, the PYD file references msvcrt.dll, which suggests that there will be issues in more complex cases. The biggest problem with CRT compatibility issues is that (AFAIK) no-one has actually been able to trigger a *real* error, all of the problems are still theoretical. I have done some fairly extensive analysis of what could go wrong, and I suspect that there are very few genuine cases where data from one CRT could end up being manipulated by another - but this doesn't mean that the problems aren't real, just VERY hard to locate :-( My current technique for checking an extension is compatible with Python 2.4 is to run objdump -p (from the mingw distribution - use dumpbin /imports from MSVC) and review the import table. If any symbols are referenced from msvcrt.dll, you need to convince yourself that they are used solely by the mingw runtime startup code. For added certainty, add a dummy function to your C code which references these symbols, rebuild and confirm that they are now satisfied from msvcr71. (For total guarantees, you need to get Martin to validate your analysis, as he seems to be the only person here who *really* understands the potential issues :-)) If you want to build extensions compatible with the standard Python 2.4 build, you need to add the -lmsvcr71 flag to the build (link step). Hope this helps, Paul.
Paul Moore <p.f.moore@gmail.com> writes:
The biggest problem with CRT compatibility issues is that (AFAIK) no-one has actually been able to trigger a *real* error, all of the problems are still theoretical.
There have been problems with the bdist_wininst exe-stub linking to the wrong CRT dll, but I don't remember the exact details. And recently I played with bindings to OpenGL's glut32.dll - glut calls exit() from internal C code. If linked with the wrong CRT, this will do nothing instead of terminating the process. Thomas
Thomas Heller wrote:
And recently I played with bindings to OpenGL's glut32.dll - glut calls exit() from internal C code. If linked with the wrong CRT, this will do nothing instead of terminating the process.
Interesting. Looking at the code of exit(), it is clear that the wrong atexit handlers will be called if you use the wrong CRT (i.e. only one set of atexit handlers will be called, whereas you might have multiple lists of atexit handlers); see atonexit.c. It is not so clear why you could possibly get out of exit() without actually exiting. The only possible code path I see in the msvcr71.dll is when mscoree.dll is in the address space, in which case the CRT will invoke CorExitProcess (!). In all other cases, it will invoke ExitProcess, which, to my understanding, will terminate the process unconditionally. Regards, Martin
Paul Moore wrote:
My current technique for checking an extension is compatible with Python 2.4 is to run objdump -p (from the mingw distribution - use dumpbin /imports from MSVC) and review the import table. If any symbols are referenced from msvcrt.dll, you need to convince yourself that they are used solely by the mingw runtime startup code.
I forgot the details of your analysis, but I think you are right. However, I would feel more comfortable if only a single CRT was used from an extension module. As for creating a case that crashes if you mix CRTs: Just try PyRun_SimpleFile in the extension, with a file that you fopen'ed in the extension. Satisfy this fopen from, say, msvcrt.dll, and load the extension into Python 2.4. Regards, Martin
On Mon, 13 Dec 2004 23:17:51 +0100, Martin v. Löwis <martin@v.loewis.de> wrote:
I forgot the details of your analysis, but I think you are right. However, I would feel more comfortable if only a single CRT was used from an extension module.
Agreed. But to some extent I'm equally uncomfortable with the idea that the mingw developers added support for -lmsvcr71 and botched it badly enough for things like this not to work :-)
As for creating a case that crashes if you mix CRTs: Just try PyRun_SimpleFile in the extension, with a file that you fopen'ed in the extension. Satisfy this fopen from, say, msvcrt.dll, and load the extension into Python 2.4.
Sorry, I wasn't specific enough. Certainly, if you create a FILE* from one CRT, then pass it to another CRT, this will fail. But my point is, if your extension uses fopen() then the link command generated by distutils will ensure that the symbol is satisfied from msvcr71.dll. So your scenario cannot occur. The only symbols which can be satisfied from msvcrt in a mingw-compiled extension (assuming that distutils is used, or the -lmsvcr71 flag is otherwise added) are those referenced in the mingw startup code, but not in user code. As the startup code will never reference Python APIs, CRT data should never be shared between incompatible CRTs. Your point about atexit (from another email) may be an issue. However, again we have the fact that user C code is *always* satisfied from msvcr71, so any exit handlers added in C or Python will be registered with the msvcr71 CRT, and so, as the main python.exe uses msvcr71, will be executed. I don't see any signs that the mingw runtime for a DLL uses atexit (this by experiment, I haven't time at the moment to review the mingw sources) and so I don't believe that the startup code has issues here, either. Thanks for your comments. Your support for the viability of building extensions using mingw is important to me, so if you still have any concerns, let me know and I will do my best to address them. Paul.
Paul Moore wrote:
Thanks for your comments. Your support for the viability of building extensions using mingw is important to me, so if you still have any concerns, let me know and I will do my best to address them.
I understand that one still needs to build libpython24.a in order to use this process. As I have said, I'd happily ship that file with the 2.4.1 MSI, unless the release manager tells me that this would an unacceptable new feature, and as long as somebody provides a fully automatic build process integrated into msi.py; for that build process, it is ok to assume that a cygwin installation is in c:\cygwin. So if this would be useful (which I don't know for sure), I still need a volunteer to contribute the appropriate magic. Regards, Martin
On Tue, 14 Dec 2004 23:48:41 +0100, Martin v. Löwis <martin@v.loewis.de> wrote:
I understand that one still needs to build libpython24.a in order to use this process. As I have said, I'd happily ship that file with the 2.4.1 MSI, unless the release manager tells me that this would an unacceptable new feature, and as long as somebody provides a fully automatic build process integrated into msi.py; for that build process, it is ok to assume that a cygwin installation is in c:\cygwin.
So if this would be useful (which I don't know for sure), I still need a volunteer to contribute the appropriate magic.
I am willing to work on this, if you wouldn't mind me asking dumb questions about the build process :-) However, I don't have VS.NET, so if automation requires integration into the VS project stuff, I can't help. For a starter, what steps do you actually take to build a release? I assume that the first step is to build Python, by clicking on "build" in VS.NET. Once you have that, is there a single script you run? If there's a document describing the release process somewhere, feel free to point me at it. Thanks, Paul.
Paul Moore wrote:
For a starter, what steps do you actually take to build a release? I assume that the first step is to build Python, by clicking on "build" in VS.NET.
Yes. You can skip this step by just putting all the .pyds, dlls, and .exes into the PCbuild directory. The packaging will try to pick them up from there (and proceed if some are missing, like Tcl likely will).
Once you have that, is there a single script you run?
Yes. Invoke Tools\msi\msi.py, using a Python that has pythonwin (i.e. COM interopability). The only tricky part is that you need cabarc.exe, which is included in VC, and in the platform SDK.
If there's a document describing the release process somewhere, feel free to point me at it.
Tools/msi/README.txt Regards, Martin
On Wed, 15 Dec 2004 22:57:00 +0100, Martin v. Löwis <martin@v.loewis.de> wrote:
Paul Moore wrote:
For a starter, what steps do you actually take to build a release? I assume that the first step is to build Python, by clicking on "build" in VS.NET.
Yes. You can skip this step by just putting all the .pyds, dlls, and .exes into the PCbuild directory. The packaging will try to pick them up from there (and proceed if some are missing, like Tcl likely will).
Once you have that, is there a single script you run?
Yes. Invoke Tools\msi\msi.py, using a Python that has pythonwin (i.e. COM interopability). The only tricky part is that you need cabarc.exe, which is included in VC, and in the platform SDK.
OK, I've got a copy of the Python sources, and had a look. The change needed to msi.py to include libpythonXX.a in the installer looks simple. But I'm less sure as to where to build the file. It seems to me that msi.py is not the right place - that's focused on building the installer, not building the files to be installed. On the other hand, including it in the build process is a nuisance, as well, as it would add a dependency on mingw (or cygwin) to the MSVC build process. My feeling is that building libpythonXX.a should be a separate step, handled by a dedicated script, which gets run before msi.py. What do others (particularly Martin) think? Should I keep the steps separate, and focused on one task each, or should I incorporate the build of libpythonXX.a into msi.py, so that the installer build still requires just one step? Paul.
Paul Moore wrote:
OK, I've got a copy of the Python sources, and had a look. The change needed to msi.py to include libpythonXX.a in the installer looks simple. But I'm less sure as to where to build the file. It seems to me that msi.py is not the right place - that's focused on building the installer, not building the files to be installed.
Don't worry about this: there is already quite some building going on in msi.py. If you look at the CVS copy of Tools/msi, you find that it now has a nmake makefile to build msisupport.dll, which replaces the VB scripts. It also extracts msvcr71.dll from the merge module (.msm) each time it is invoked. So having yet another build process would be just fine; adding it to the makefile would have the added advantage that nmake will compare time stamps for us (if it is easier to do in Python than in nmake, that would be a reason not to use nmake, though).
On the other hand, including it in the build process is a nuisance, as well, as it would add a dependency on mingw (or cygwin) to the MSVC build process.
That is definitely undesirable. Lots of people build Python using the project files, and only few need the packaging to work.
My feeling is that building libpythonXX.a should be a separate step, handled by a dedicated script, which gets run before msi.py.
Making it separate is fine, as long as msi.py invokes/calls it.
What do others (particularly Martin) think? Should I keep the steps separate, and focused on one task each, or should I incorporate the build of libpythonXX.a into msi.py, so that the installer build still requires just one step?
Having the entire process involve as few manual steps as possible is definitely an important goal. Keeping it modular (in terms of splitting it into several files) is also a good idea, and one which does not conflict at all with the "fully automatic" goal. msi.py supports a config.py which allows to add customization. Putting cygwin_dir = r"C:\cygwin" into msi.py might be appropriate, with an option to set cygwin_dir to None, to indicate that cygwin should not be used in the build process. (similar to the way have_tcl already works). Regards, Martin
participants (4)
-
"Martin v. Löwis" -
A.B., Khalid -
Paul Moore -
Thomas Heller