Python 2.4 and MSVC 7.1 - extension compatibility
One of the issues with moving to VC7 for the Python Windows build is the fact that extensions (or embedding apps) built with MSVC 6 "won't work" with it. But I've never actually managed to provoke a crash (not that I've tried *very* hard, mind). And thinking about the issue a little, I wonder whether there is a real problem here at all... A quick check of the DLLs in my Windows\System32 directory shows a lot which use MSVCRT. Including OLEAUT32, which is one of the key OLE/COM DLLs. So, presumably, this means that DLLs built with MSVCRT *can* be used by applications built with VC7. And a Python extension is just a DLL, with an unusual extension. So, the question is - is there really any problem with Python 2.4, built with MSVC7, using an extension DLL built with MSVC6? The areas where I can see a potential issue are where CRT objects are shared between the extension, and the Python runtime: 1. Memory allocated by the extension CRT, and freed by Python 2. FILE* objects passed from one CRT to the other 3. Any others? Memory allocation "should" be OK, if extension writers follow the rules in the Python/C API manual, section 9.1 - "To avoid memory corruption, extension writers should never try to operate on Python objects with the functions exported by the C library: malloc(), calloc(), realloc() and free()". And no-one ever writes extensions which violate the rules :-) There are a series of APIs in the "Very High Level Layer" (PyRun_* and friends), and some in the PyMarshal series, which pass FILE* objects about. I wouldn't expect these to be common in extensions. So, a question - would it be worth documenting which APIs are *not* supported in extensions built with an incompatible CRT to the one Python was built with? And if this was done, would it be OK to state "if you don't use these APIs, you don't have to build your extension or embedding application with the same CRT as that of Python"? If it is deemed acceptable to make these sort of guarantees, I'll volunteer to search out the relevant APIs, and draft a section for the Python/C API manual. But I don't want to spend the time and effort, if it's still not possible to say that it's safe to use MSVCRT in those cases. Paul. -- This signature intentionally left blank
Paul Moore wrote:
So, the question is - is there really any problem with Python 2.4, built with MSVC7, using an extension DLL built with MSVC6?
Just try and see for yourself. I saw the omniORB IDL compiler crash because of that.
Memory allocation "should" be OK, if extension writers follow the rules in the Python/C API manual, section 9.1 - "To avoid memory corruption, extension writers should never try to operate on Python objects with the functions exported by the C library: malloc(), calloc(), realloc() and free()". And no-one ever writes extensions which violate the rules :-)
This is not sufficient. For example, using the s# argument parser means the extension module needs to call PyMem_Free on the resulting memory block, allocated by getargs. According to the documentation, the extension module can safely call PyMem_FREE instead, which would cause heap confusion.
So, a question - would it be worth documenting which APIs are *not* supported in extensions built with an incompatible CRT to the one Python was built with? And if this was done, would it be OK to state "if you don't use these APIs, you don't have to build your extension or embedding application with the same CRT as that of Python"?
I would not make such a statement. It is *never* OK to mix CRTs, even if one cannot construct a case where this causes problems. That said, people will be using other compilers to create extensions if they find it works for them. They need to perform their own testing, and once they are satisfied, they will ship the code no matter what the documentation says. Regards, Martin
"Martin v. Löwis" <martin@v.loewis.de> writes:
Paul Moore wrote:
So, the question is - is there really any problem with Python 2.4, built with MSVC7, using an extension DLL built with MSVC6?
Just try and see for yourself. I saw the omniORB IDL compiler crash because of that.
Hmm. In itself, that's not conclusive, as it's possible that omniORB uses the "dangerous" APIs I mention below. But your other arguments *are* sufficient, so having a specific example is very helpful. I've seen too many "well, it seems to work OK for me" comments to be entirely comfortable with the fact that I had managed to find no evidence of crashes myself.
Memory allocation "should" be OK,
[...]
This is not sufficient. For example, using the s# argument parser means the extension module needs to call PyMem_Free on the resulting memory block, allocated by getargs. According to the documentation, the extension module can safely call PyMem_FREE instead, which would cause heap confusion.
OK. That's a documented usage which requires using the same C runtime. It could be argued that the documentation could be changed to "tighten up" the requirements, but that way lies madness :-)
So, a question - would it be worth documenting which APIs are *not* supported in extensions built with an incompatible CRT to the one Python was built with? And if this was done, would it be OK to state "if you don't use these APIs, you don't have to build your extension or embedding application with the same CRT as that of Python"?
I would not make such a statement. It is *never* OK to mix CRTs, even if one cannot construct a case where this causes problems.
Thanks. I guess my only reservation was that I wasn't 100% clear on what constitutes "mixing", given that, for example, Python, built with MSVC7, can load win32api.pyd (from win32all), built with MSVC7, which uses oleaut32.dll (MS system file) which in turn uses msvcrt.dll. That is expected to work, so we are, in some sense, not "mixing" CRTs in this case. I'm still not entirely clear, but I'm happy to accept your statement on the matter.
That said, people will be using other compilers to create extensions if they find it works for them. They need to perform their own testing, and once they are satisfied, they will ship the code no matter what the documentation says.
I know :-( And given the fact that there's nothing in a binary distribution to make it obvious what compiler was used to build the extension, there's the risk that binary distributions will be shipped with subtle problems like this. [Checks] On the other hand, i've just seen that the (MSVC7) Python 2.4 distutils refuses to let me build an extension with MSVC 6. So maybe things aren't as bad as all that - if you get a distutils-built Windows binary, you can be pretty happy that it uses a compatible CRT. Thanks for the comments. Most of this is quite probably FUD, and in reality everything will go smoothly. I just shocked myself this evening by going through my Python environment, and realising how many binary extensions I rely on (many of which I don't have the facilities to build for myself, even if I had the time...) Paul. -- This signature intentionally left blank
participants (2)
-
"Martin v. Löwis" -
Paul Moore