[Cython] Shared Cython runtime

[Stefan Behnel]

Nikita Nemkin, 09.04.2013 18:30:
> On Tue, 09 Apr 2013 21:21:25 +0600, Stefan Behnel wrote:
>> Oh, and even simpler, you can just stick multiple Cython modules with their
>> module init functions into one big shared library and have the "main" init
>> function of that module call the others. That will stick them into
>> sys.modules as a side effect, so it's not really different from going
>> through Python's import machinery for each module separately (just way more
>> efficient - at least if you really need all of the modules...).
> 
> That's exactly what I meant.
> 
>> There may be glitches with stuff like "__file__" and friends, but that
>> should not be any worse than the current way of workings.
> 
> I know of two "glitches":
>
> 1) Py_InitModule4 expects qualified module name to be provided via
>    the _Py_PackageContext global, in order to initialize new module's
> __name__.
>    Of course __name__ can also be set manually afterwards.

Cython could easily do that. It needs to know the FQMN at compile time
anyway, so adding a C compile time switch that sets _Py_PackageContext
before calling that function is trivial. (Would also have to store and
restore the old value to support recursive imports, obviously)


> 2) "Top level" init function does not have access to it's own __file__,
>    but it has to initialize submodules' __file__ somehow.
>    My solution is to query the current shared library name directly
>    from the OS (GetModuleFileName() on Windows, dladdr() on everything else).

Yep:

http://bugs.python.org/issue13429

Sadly, the new module init mechanism in Py3 wasn't designed to address
these issues.


> I'm interested in implementing this feature someday. For now, doing it
> manually is good enough.

I assume you're aware of the cython_freeze tool? That currently only works
for embedding Python in a Cython compiled executable. Supporting this also
(automatically) for a build where you have multiple .pyx files in a single
distutils Extension() would be great.

So, please do. And don't wait too long. Anything that can be automated
should be.

Stefan