[Cython] Multiple modules in one compilation unit

Lisandro Dalcin dalcinl at gmail.com
Thu Mar 3 05:38:09 CET 2011


On 2 March 2011 21:01, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Stefan Behnel wrote:
>>
>> you'd call "cython" on a package and it would output a directory with a
>> single __init__.so that contains the modules compiled from all .pyx/.py
>> files in that package. Importing the package would then trigger an import of
>> that __init__.so, which in turn will execute code in its init__init__()
>> function to register the other modules.
>
> I don't think it even has to be a directory with an __init__,
> it could just be an ordinary .so file with the name of the
> package.
>
> I just tried an experiment in Python:
>
> # onefilepackage.py
> import new, sys
> blarg = new.module("blarg")
> blarg.thing = "This is the thing"
> sys.modules["onefilepackage.blarg"] = blarg
>
> and two different ways of importing it:
>
>>>> from onefilepackage import blarg
>>>> blarg
> <module 'blarg' (built-in)>
>>>> blarg.thing
> 'This is the thing'
>
>>>> import onefilepackage.blarg
>>>> onefilepackage.blarg.thing
> 'This is the thing'
>

I'm hacking around these lines. However, I'm working to maintain
different modules in different C compilation units, in order to
workaround the obvious issue with duplicated global C symbols.

> So assuming the same thing works with a .so instead of a .py,
> all you need to do is emit a .so whose init function stuffs
> appropriate entries into sys.modules to make it look like
> a package.
>

However, the import machinery does not treat .so the same as *.pyx.
For example, I have a problem with Python 3. For .py modules, before
the module code starts to execute, the matching entries in sys.modules
is already there. The same happens in Python 2 for .so modules, as
Py_InitModule() add the entry in sys.modules early. However, in Python
3 that is not te case (and only for the .so, for .py is the same as in
Py2), the import machinery adds the entry later, after the
finalization of the module init function. I'm tempted to workaround
this by setting the entry in sys.modules right after the call to
PyModule_Create() ... What do you think about this? Any potential
issue?


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169


More information about the cython-devel mailing list