future-proofing vector tables for python APIs: binary-module interoperability
On Fri, Jan 23, 2009 at 10:48 PM, Roumen Petrov <bugtrack@roumenpetrov.info> wrote:
python.exe (say, the official one) loads python25.dll. Then, an import is made of a ming-wine extension, say foo.pyd, which is linked with libpython2.5.dll, which then gets loaded. Voila, you have two interpreters in memory, with different type objects, memory heaps, and so on.
ok, there's a solution for that - the gist of the solution is already implemented in things like Apache Runtime and Apache2 (modules), and is an extremely common standard technique implemented in OS kernels. the "old school" name for it is "vector tables".
[SNIP] Did you think that this will escape python MSVC from "Assembly hell" ?
let me think about that.... write some things down, i might have an answer at the end :) but it would certainly mean that there would be both a future-proof path for binary modules from either msvc-compiled _or_ mingw-compiled 2.5, 2.6, 2.7 etc. to work with 2.5, 2.6, 2.7, 2.8 etc. _without_ a recompile. [forwards-future-proof-compatibility _is_ possible, but... it's a bit more... complicated. backwards-compatibility is easy]. what you do is you make sure that the vector-table is always and only "extended" - added to - never "removed from" or altered. if one function turns out to be a screw-up (inadequate, not enough parameters), you do NOT change its function parameters, you add an "Ex" version - or an "Ex1" version. just like microsoft does. [.... now you know _why_ they do that "ridiculous" thing of adding FunctionEx1 FunctionEx2 and if you look at the MSHTML specification i think they go up to _six_ revisions of the same function in one case!] to detect revisions of the vector-table you use a "negotiation" tactic. you add a bit-field at the beginning of the struct, and each bit expresses a "new revision" indicating that the vector table has been extended (and so needs to be typecast to a different struct - exactly the same as is done with PyObject being typecast to different structs). the first _function_ in the vector-table is one which the module must call (in its initXXXX()) to pass in the "version number" of the module, to the python runtime. just in case someone needs to know. but for the most part, the initiation - of function call-out - is done _from_ modules, so each and every module will never try to call something beyond what it understands. but basically, not only is this technique nothing new - it's in use in Apache RunTime, FreeDCE, the NT Kernel, the Linux Kernel - but also it's actually _already_ in use in one form in the way that python objects are typecast from PyObject to other types of structs! the difference is that a bit-field would make detection of revisions a bit easier but to be honest you could just as easily make it an int and increase the revision number. .... ok, i've thought about your question, and i think it might [save us from assembly hell]. what you would likely have to do is compile _individual modules_ with assemblies, should they need them. for example, the msvcrt module would obviously have to be.... hey, that'd be interesting, how about having different linked versions of the msvcrt module? coool :) in the mingw builds, it's not necessary to link in PC/msvcrtmodule.o into the python dll - so (and this confused the hell out of me for a minute so i had to do find . -name "msvcrt*") you end up with a Modules/msvcrt.pyd. surely, that should be the _only_ dll which gets _specifically_ linked against msvcr71.dll (or 90, or... whatever) and it would be even _better_ if that then got _named_ msvcr71.pyd, msvcr90.pyd etc. i'll do an experiment, later, to confirm that this actually _does_ work - i.e. creating an msvcr80.pyd with "mingw gcc -specs=msvcr80". the neat thing is that if it works, you wouldn't need to _force_ people to link to the python dll or the python exe with msvcr90 or any other version. and the mingw built python.exe or python dll would be interchangeable, as it would be _specific modules_ that required specific versions of the msvc runtime. l.
Luke Kenneth Casson Leighton wrote:
On Fri, Jan 23, 2009 at 10:48 PM, Roumen Petrov [SNIP] but it would certainly mean that there would be both a future-proof path for binary modules from either msvc-compiled _or_ mingw-compiled 2.5, 2.6, 2.7 etc. to work with 2.5, 2.6, 2.7, 2.8 etc. _without_ a recompile. [forwards-future-proof-compatibility _is_ possible, but... it's a bit more... complicated. backwards-compatibility is easy].
what you do is you make sure that the vector-table is always and only "extended" - added to - never "removed from" or altered. if one function turns out to be a screw-up (inadequate, not enough parameters), you do NOT change its function parameters, you add an "Ex" version - or an "Ex1" version. [SNIP] but basically, not only is this technique nothing new - it's in use in Apache RunTime, FreeDCE, the NT Kernel, the Linux Kernel - but also it's actually _already_ in use in one form in the way that python objects are typecast from PyObject to other types of structs! the difference is that a bit-field would make detection of revisions a bit easier but to be honest you could just as easily make it an int and increase the revision number.
surely, that should be the _only_ dll which gets _specifically_ linked against msvcr71.dll (or 90, or... whatever) and it would be even _better_ if that then got _named_ msvcr71.pyd, msvcr90.pyd etc. [SNIP] Yes it is enough to encapsulate memory allocation and file functions into python shared library. The python provide memory allocation functions, but not all modules use them. File functions are hiden by
This look like simple RCP implemantation. If I remember well SUN-RPC assign number to program, function, version. [SNIP] posixmodule and python modules can't use them. Roumen
but basically, not only is this technique nothing new - it's in use in Apache RunTime, FreeDCE, the NT Kernel, the Linux Kernel - but also
This look like simple RPC implemantation.
yep.
If I remember well SUN-RPC assign number to program, function, version.
yep - i forgot about that one: yes, that's another example. this is pretty basic well-understood, well-documented techniques that virtually every large project that requires isolation between components (and an upgrade path) ends up using in one form or another. the only fly in the ointment will be that putting pointers to PyType_String etc. etc. into a vector table (struct), you end up with an extra de-ref overhead, which is often an argument utilised to do away with vector tables. but - tough: the decision involves getting away from "Hell" to something that makes everyone's lives that much easier, it's an easy decision to make.
surely, that should be the _only_ dll which gets _specifically_ linked against msvcr71.dll (or 90, or... whatever) and it would be even _better_ if that then got _named_ msvcr71.pyd, msvcr90.pyd etc.
[SNIP] Yes it is enough to encapsulate memory allocation and file functions into python shared library. The python provide memory allocation functions, but not all modules use them. File functions are hiden by posixmodule and python modules can't use them.
except ... posixmodule gets renamed to ntmodule .... oh, i see what you mean: python modules aren't allowed _direct_ access to msvcrtNN's file functions, they have to go via posixmodule-renamed-to-ntmodule. so it's still ok. l.
[SNIP] Yes it is enough to encapsulate memory allocation and file functions into python shared library. The python provide memory allocation functions, but not all modules use them. File functions are hiden by posixmodule and python modules can't use them.
except ... posixmodule gets renamed to ntmodule .... oh, i see what you mean: python modules aren't allowed _direct_ access to msvcrtNN's file functions, they have to go via posixmodule-renamed-to-ntmodule.
.... thinking about this some more... posixmodule.c is linked (by default) into pythonNN.dll, thus making pythonNN.dll totally dependent on a version of msvcrt. decoupling posixmodule.c from pythonNN.dll leaves the possibility to make python independent of msvcrt versioning. it would need to be a custom-compiled .pyd module, due to the early dependency. i'll see if this is possible. l.
decoupling posixmodule.c from pythonNN.dll leaves the possibility to make python independent of msvcrt versioning.
it would need to be a custom-compiled .pyd module, due to the early dependency.
i'll see if this is possible.
i'd added PyExc_OSError, for example, as data exported from dlls. i'm finding that this causes.... problems :) so when posixmodule.c is a module (nt.pyd), doing this works: PyAPI_FUNC(PyObject *) PyErr_GetPyExc_OSError(void) { return (PyObject*)PyExc_OSError; } and thus oserr = PyErr_GetPyExc_OSError(); Py_INCREF(oserr); PyModule_AddObject(m, "error", oserr) but doing _direct_ access to PyExc_OSError fails miserably. i'll try to track down why (am adding __cdecl to PyAPI_DATA to see if that helps). l.
Luke Kenneth Casson Leighton wrote:
[SNIP] Yes it is enough to encapsulate memory allocation and file functions into python shared library. The python provide memory allocation functions, but not all modules use them. File functions are hiden by posixmodule and python modules can't use them. except ... posixmodule gets renamed to ntmodule .... oh, i see what you mean: python modules aren't allowed _direct_ access to msvcrtNN's file functions, they have to go via posixmodule-renamed-to-ntmodule.
.... thinking about this some more... posixmodule.c is linked (by default) into pythonNN.dll, thus making pythonNN.dll totally dependent on a version of msvcrt.
This is not problem. If python*.dll hide msvcrt and other modules depend directly from python*.dll I expect issue to be resolved. i.e. python*.dll to be "portable runtime interface".
decoupling posixmodule.c from pythonNN.dll leaves the possibility to make python independent of msvcrt versioning.
it would need to be a custom-compiled .pyd module, due to the early dependency.
i'll see if this is possible.
Румен
On Sun, Jan 25, 2009 at 3:55 PM, Roumen Petrov <bugtrack@roumenpetrov.info> wrote:
Luke Kenneth Casson Leighton wrote:
[SNIP] Yes it is enough to encapsulate memory allocation and file functions into python shared library. The python provide memory allocation functions, but not all modules use them. File functions are hiden by posixmodule and python modules can't use them.
except ... posixmodule gets renamed to ntmodule .... oh, i see what you mean: python modules aren't allowed _direct_ access to msvcrtNN's file functions, they have to go via posixmodule-renamed-to-ntmodule.
.... thinking about this some more... posixmodule.c is linked (by default) into pythonNN.dll, thus making pythonNN.dll totally dependent on a version of msvcrt.
This is not problem. If python*.dll hide msvcrt and other modules depend directly from python*.dll I expect issue to be resolved. i.e. python*.dll to be "portable runtime interface".
yehhhh :) well - it looks like i am having success with removing all references to data e.g. Py_NoneStruct, all of the PyExc_*Warning and PyExc_*Error, all of the Py*_Types and more. i'm making sure that macros are heavily used - so that on systems where data _can_ be accessed through dynamic shared objects, it's done so. #if defined(MS_WINDOWS) || defined(__MINGW32__) /* Define macros for conveniently creating "getter" functions, * to avoid restrictions on dlls being unable to access data. * see #5056 */ /* use these for data that is already a pointer */ #define PyAPI_GETHDR(type, obj) \ PyAPI_FUNC(type) _Py_Get_##obj(void); #define PyAPI_GETIMPL(type, obj) \ PyAPI_FUNC(type) _Py_Get_##obj(void) { return (type)obj; } #define _PYGET(obj) _Py_Get_##obj() /* use these for data where a pointer (&) to the object is returned * e.g. no longer #define Py_None (&Py_NoneStruct) but * #define Py_None _PTGETPTR(Py_NoneStruct) */ #define PyAPI_GETHDRPTR(type, obj) \ PyAPI_FUNC(type) _Py_Get_##obj(void); #define PyAPI_GETIMPLPTR(type, obj) \ PyAPI_FUNC(type) _Py_Get_##obj(void) { return (type)&obj; } #define _PYGETPTR(obj) _Py_Get_##obj() #else /* on systems where data can be accessed directly in shared modules, * as an optimisation, return the object itself, directly. */ #define PyAPI_GETFNIMPL(obj) ; #define PyAPI_GETHDR(type, obj) ; #define PyAPI_GETIMPL(type, obj) ; #define PyAPI_GETIMPLPTR(type, obj) ; #define _PYGET(obj) (obj) #define _PYGETPTR(obj) (&obj) #endif /* defined(MS_WINDOWS) || defined(__MINGW32__)*/ as you can see from the Py_None example, on non-dll-based systems, you get... wow, a macro which returns... exactly what was returned before. zero impact. on windows, you end up defining, creating and using a "getter" function. two types. one which returns the object, the other returns a pointer to the object. l.
participants (2)
-
Luke Kenneth Casson Leighton -
Roumen Petrov