[Cython] Shared Cython runtime

[Robert Bradshaw]

On Tue, Apr 9, 2013 at 8:21 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 09.04.2013 17:12:
>> mark florisson, 09.04.2013 16:51:
>>> On 9 April 2013 15:47, Stefan Behnel wrote:
>>>
>>>> mark florisson, 09.04.2013 16:32:
>>>>> On 9 April 2013 14:55, Nikita Nemkin wrote:
>>>>>> One alternative for code reuse in large Cython projects
>>>>>> could be packaging multiple modules into one shared library.
>>>>>
>>>>> We have 'include'! :) Seriously though, that wouldn't work well with the
>>>>> import mechanism, and probably not for C compile time either.
>>>>
>>>> CPython has support for it. It's actually quite easy, you basically just
>>>> register all module init functions explicitly at init time and let CPython
>>>> do the rest.
>>>>
>>>> So, compiling a package into one big Cython module with shared internal
>>>> types and utility code should be quite simple and would IMHO be a very cool
>>>> feature.
>>>
>>> I was unaware of that, it'd be a nice feature then (though still not imho
>>> the right solution to a runtime system). How does this work? Feel free to
>>> point me at some documentation.
>>
>> It's the inittab mechanism, as described starting here:
>>
>> http://docs.python.org/2/c-api/import.html#PyImport_AppendInittab
>
> Oh, and even simpler, you can just stick multiple Cython modules with their
> module init functions into one big shared library and have the "main" init
> function of that module call the others. That will stick them into
> sys.modules as a side effect, so it's not really different from going
> through Python's import machinery for each module separately (just way more
> efficient - at least if you really need all of the modules...).

Yep. We could even create stub .so files for the included ones that
did nothing but import the big one to avoid having to worry about
importing things in the right order.

The big question is what API to specify that things should be linked
together. An option to cythonize(...) could be used, but for some
projects (e.g. Sage) one would want more control over which modules
get bundled together rather than all-or-nothing.

> There may be glitches with stuff like "__file__" and friends, but that
> should not be any worse than the current way of workings.

This can probably be manipulated by Cython, though it's unclear what
the best value would be.

- Robert