[Python-Dev] future-proofing vector tables for python APIs: binary-module interoperability

Luke Kenneth Casson Leighton lkcl at lkcl.net
Sat Jan 24 12:15:45 CET 2009


On Fri, Jan 23, 2009 at 10:48 PM, Roumen Petrov
<bugtrack at roumenpetrov.info> wrote:

>>> python.exe (say, the official one) loads
>>> python25.dll. Then, an import is made of a ming-wine extension, say
>>> foo.pyd, which is linked with libpython2.5.dll, which then gets loaded.
>>> Voila, you have two interpreters in memory, with different type objects,
>>> memory heaps, and so on.
>>
>>  ok, there's a solution for that - the gist of the solution is already
>> implemented in things like Apache Runtime and Apache2 (modules), and
>> is an extremely common standard technique implemented in OS kernels.
>> the "old school" name for it is "vector tables".
>>
> [SNIP] Did you think that this will escape python MSVC from "Assembly hell"
> ?


let me think about that.... write some things down, i might have an
answer at the end :)


but it would certainly mean that there would be both a future-proof
path for binary modules from either msvc-compiled _or_ mingw-compiled
2.5, 2.6, 2.7 etc. to work with 2.5, 2.6, 2.7, 2.8 etc. _without_ a
recompile.  [forwards-future-proof-compatibility _is_ possible, but...
it's a bit more... complicated.  backwards-compatibility is easy].

what you do is you make sure that the vector-table is always and only
"extended" - added to - never "removed from" or altered.  if one
function turns out to be a screw-up (inadequate, not enough
parameters), you do NOT change its function parameters, you add an
"Ex" version - or an "Ex1" version.

just like microsoft does. [.... now you know _why_ they do that
"ridiculous" thing of adding FunctionEx1 FunctionEx2 and if you look
at the MSHTML specification i think they go up to _six_ revisions of
the same function in one case!]

to detect revisions of the vector-table you use a "negotiation"
tactic.  you add a bit-field at the beginning of the struct, and each
bit expresses a "new revision" indicating that the vector table has
been extended (and so needs to be typecast to a different struct -
exactly the same as is done with PyObject being typecast to different
structs).  the first _function_ in the vector-table is one which the
module must call (in its initXXXX()) to pass in the "version number"
of the module, to the python runtime.  just in case someone needs to
know.

but for the most part, the initiation - of function call-out - is done
_from_ modules, so each and every module will never try to call
something beyond what it understands.

but basically, not only is this technique nothing new - it's in use in
Apache RunTime, FreeDCE, the NT Kernel, the Linux Kernel - but also
it's actually _already_ in use in one form in the way that python
objects are typecast from PyObject to other types of structs!  the
difference is that a bit-field would make detection of revisions a bit
easier but to be honest you could just as easily make it an int and
increase the revision number.



.... ok, i've thought about your question, and i think it might [save
us from assembly hell].

what you would likely have to do is compile _individual modules_ with
assemblies, should they need them.  for example, the msvcrt module
would obviously have to be.... hey, that'd be interesting, how about
having different linked versions of the msvcrt module?

coool :)

in the mingw builds, it's not necessary to link in PC/msvcrtmodule.o
into the python dll - so (and this confused the hell out of me for a
minute so i had to do find . -name "msvcrt*") you end up with a
Modules/msvcrt.pyd.

surely, that should be the _only_ dll which gets _specifically_ linked
against msvcr71.dll (or 90, or... whatever) and it would be even
_better_ if that then got _named_ msvcr71.pyd, msvcr90.pyd etc.

i'll do an experiment, later, to confirm that this actually _does_
work - i.e. creating an msvcr80.pyd with "mingw gcc -specs=msvcr80".

the neat thing is that if it works, you wouldn't need to _force_
people to link to the python dll or the python exe with msvcr90 or any
other version.

and the mingw built python.exe or python dll would be interchangeable,
as it would be _specific modules_ that required specific versions of
the msvc runtime.

l.


More information about the Python-Dev mailing list