[Cython] Shared Cython runtime

[Nikita Nemkin]

On Wed, 10 Apr 2013 11:24:33 +0600, Stefan Behnel <stefan_ml at behnel.de>  
wrote:

> Nikita Nemkin, 10.04.2013 06:22:
>> The stubs don't have to be shared libraries, pure python would work
>> just fine (and reduce the complexity).
>
> Sure. That's actually the most common way to do it. Most C modules in
> CPython's stdlib have a Python wrapper script, for example. It's also not
> uncommon to add a certain amount of functionality in that script. That
> makes even just "having it there" a feature for future development.

Cython's ability to create self-contained binaries is an important feature.

For bundles of unrelated modules, stubs seem to be the only option,
but whole package bundles can do without them.

> Maybe compiling whole packages isn't all that a good idea after all,  
> unless at least most of the modules in that package are actually required
> for the core functionality.

In-binary import is quite fast. No extra file access, no parsing.
Capsule indirection can be eliminated, some other things optimized...

As an option, PEP 302 (Import hooks) provides an elegant way to implement
delayed imports for whole package bundles.

> The problem is not which value to use. The problem is that the shared
> library pathname is not known at init time, except inside of the loader,
> and it doesn't tell us about it. In the specific case of a meta-module,  
> it won't even be set before *all* init functions of the submodules were
> called and the main init function terminates.
>
> Which is a really bad situation, because it means that even though the  
> main
> module will eventually have its __file__ set by the loader, it has no way
> to propagate it to the submodules anymore. So they won't have their
> __file__ property set at all.
>
> Just in case someone missed it, here's the bug URL once again:
>
> http://bugs.python.org/issue13429
>
> Cython can fix up stuff related to the FQMN and sys.modules (and it does
> that already), but can't do anything about the file path. That makes  
> things like resource loading rather annoying.
>
> BTW, here's also the python-dev discussion on the topic:
>
> http://thread.gmane.org/gmane.comp.python.devel/135764
>
> Maybe that discussion should be revived and this use case added to it.

Well, here is the workaround Cython can use:
https://gist.github.com/nnemkin/5352088
(Warning: I haven't fully tested it yet).

Best regards,
Nikita Nemkin