Supporting Third Party Modules (was The other Py2.4 issue)
Raymond:
Acceptance for Py2.4 partially hinges on how quickly third party apps have their binaries updated.
I wonder if there is anything we can do to help.
One good way of helping out is to provide an dynamic loading function that third party modules could access the basic python functions such as PyArgParseTuple, PyString_AsString etc regardless of which python the user is running. This would be similar to the COM approach. You can load all the function pointers into a struct and then call them. Third party modules would link against this DLL independent of which python is being used. Chui
Chui G. Tey wrote:
One good way of helping out is to provide an dynamic loading function that third party modules could access the basic python functions such as PyArgParseTuple, PyString_AsString etc regardless of which python the user is running. This would be similar to the COM approach. You can load all the function pointers into a struct and then call them.
Third party modules would link against this DLL independent of which python is being used.
I believe this is not implementable: How can the DLL know which Python DLL to use? Regards, Martin
On Dec 11, 2004, at 3:39 AM, Martin v. Löwis wrote:
Chui G. Tey wrote:
One good way of helping out is to provide an dynamic loading function that third party modules could access the basic python functions such as PyArgParseTuple, PyString_AsString etc regardless of which python the user is running. This would be similar to the COM approach. You can load all the function pointers into a struct and then call them. Third party modules would link against this DLL independent of which python is being used.
I believe this is not implementable: How can the DLL know which Python DLL to use?
Well for py2app on Mac OS X, I wrote an executable stub that chooses a Python runtime from an XML file, looks up and binds a few symbols from it dynamically, and then starts doing stuff. Works for 2.3 (as long as it's not compiled debug or trace refs) and 2.4 (universally, because it has exported functions for incref and decref). For extension modules, this makes much less sense, but I wanted to be able to bootstrap Python programs with a precompiled executable that I didn't have to maintain across several versions. -bob
Bob Ippolito wrote:
I believe this is not implementable: How can the DLL know which Python DLL to use?
Well for py2app on Mac OS X, I wrote an executable stub that chooses a Python runtime from an XML file, looks up and binds a few symbols from it dynamically, and then starts doing stuff.
While that would work, I think this is inappropriate for this specific issue: we want to write extension modules which are independent of the Python version, and might even be used with multiple Python installations on the same system. In that case, adding configuration files won't work, as each usage of the extension might require a different Python DLL. Regards, Martin
On Dec 12, 2004, at 11:02 AM, Martin v. Löwis wrote:
Bob Ippolito wrote:
I believe this is not implementable: How can the DLL know which Python DLL to use? Well for py2app on Mac OS X, I wrote an executable stub that chooses a Python runtime from an XML file, looks up and binds a few symbols from it dynamically, and then starts doing stuff.
While that would work, I think this is inappropriate for this specific issue: we want to write extension modules which are independent of the Python version, and might even be used with multiple Python installations on the same system. In that case, adding configuration files won't work, as each usage of the extension might require a different Python DLL.
Yes, of course, I was talking about the executable, not extensions. On Mac OS X 10.3+, the linker flag -undefined dynamic_lookup allows extensions to link to no Python whatsoever. The extensions will just find the symbols it needs from some other image already loaded into the process at runtime. If it weren't for the "forced" ABI incompatibility, we'd already have extensions that work cross-Python-major-version (assuming they used a safe subset of functions and structures). -bob
Bob Ippolito wrote:
Yes, of course, I was talking about the executable, not extensions. On Mac OS X 10.3+, the linker flag -undefined dynamic_lookup allows extensions to link to no Python whatsoever.
It's the same on SysV ELF shared libraries, and in most other unices.
The extensions will just find the symbols it needs from some other image already loaded into the process at runtime. If it weren't for the "forced" ABI incompatibility, we'd already have extensions that work cross-Python-major-version (assuming they used a safe subset of functions and structures).
Are you talking about a forced ABI incompatibility, beyond the Windows issue of linking with a specific pythonxy.dll? On Unix, you certainly can have extensions across Python major versions. Regards, Martin
On Dec 12, 2004, at 1:09 PM, Martin v. Löwis wrote:
Bob Ippolito wrote:
Yes, of course, I was talking about the executable, not extensions. On Mac OS X 10.3+, the linker flag -undefined dynamic_lookup allows extensions to link to no Python whatsoever.
It's the same on SysV ELF shared libraries, and in most other unices.
The extensions will just find the symbols it needs from some other image already loaded into the process at runtime. If it weren't for the "forced" ABI incompatibility, we'd already have extensions that work cross-Python-major-version (assuming they used a safe subset of functions and structures).
Are you talking about a forced ABI incompatibility, beyond the Windows issue of linking with a specific pythonxy.dll?
On Unix, you certainly can have extensions across Python major versions.
I was talking about PYTHON_API_VERSION, hence "forced" in quotes. Nobody likes to see an ugly error message. -bob
Bob Ippolito wrote:
I was talking about PYTHON_API_VERSION, hence "forced" in quotes. Nobody likes to see an ugly error message.
Ah, that... If PYTHON_API_VERSION changes, it is because there is a real incompatibility, somewhere. It is not automatically bumped with each release. So it is safe to ignore the ugly error message only if you have not used any of the API that became incompatible. Regards, Martin
Martin v. Löwis wrote:
Third party modules would link against this DLL independent of which python is being used.
I believe this is not implementable: How can the DLL know which Python DLL to use?
the Python interpreter could initialize this DLL, using some suitable mechanism. alternatively, it could figure out what EXE you used to start the program in the first place, and figure out what Python DLL that EXE uses. my ExeMaker tool can do that, for example: http://effbot.org/zone/exemaker.htm (see the #!python.exe entry in the table) </F>
Fredrik Lundh wrote:
the Python interpreter could initialize this DLL, using some suitable mechanism.
What suitable mechanism could that be? Suppose the Python script imports foo.pyd, which in turn is linked with pythonany.dll, which in turn is supposed to find the right Python DLL. Now, python24.dll runs LoadLibary for foo.pyd, but has no control over pythonany.dll. How could the suitable mechanism work?
alternatively, it could figure out what EXE you used to start the program in the first place, and figure out what Python DLL that EXE uses. my ExeMaker tool can do that, for example:
http://effbot.org/zone/exemaker.htm
(see the #!python.exe entry in the table)
That requires a module handle for the EXE, right? How do you get that handle? Out of curiosity: How come that the hModule is the start address of the image? On what Windows versions does that work? Is that documented? Regards, Martin
participants (4)
-
"Martin v. Löwis" -
Bob Ippolito -
Chui G. Tey -
Fredrik Lundh