[Python-Dev] nested extension modules?

[David Abrahams]

From: "Guido van Rossum" <guido@python.org>

> > Using the source (Luke), I was trying to figure out the best way to add
a
> > nested submodule from within an extension module. I noticed that the
module
> > initialization code will set the module name from the package context
(if
> > set), altogether discarding any name passed explicitly:
> >
> > [modsupport.c: Py_InitModule4()]
> >
> >  ...
> >  if (_Py_PackageContext != NULL) {
> >   char *p = strrchr(_Py_PackageContext, '.');
> >   if (p != NULL && strcmp(name, p+1) == 0) {
> >    name = _Py_PackageContext;
> >    _Py_PackageContext = NULL;
> >   }
> >  }
> >
> > This _Py_PackageContext is set up from within
_PyImport_LoadDynamicModule
> > [importdl.c:]
> >
> >  ...
> >  oldcontext = _Py_PackageContext;
> >  _Py_PackageContext = packagecontext;
> >  (*p)();
> >  _Py_PackageContext = oldcontext;
> >
> > IIUC, this means that when an extension module is loaded as part of a
> > package, any submodules I create my calling Py_InitModule<whatever>
will
> > come out with the same name.
> >
> > Questions:
> >
> > a. Have I got the analysis right?
>
> Not quite, if I understand what you're saying.  The package context,
> despite its name, is not the package name, but the full name of the
> *module*, when the shared library is found inside a package.

I think I understood that part.

> If, e.g., a package directory P contains an extension module file
> E.so, the package context is set to "P.E".  The initE() function is
> supposed to call Py_InitModule4() with "E" as the module name.
> Py_InitModule4() then sees that this is the last component of the
> package context, and changes the module name to "P.E".

Yeah, that's what I expected.

> It also nulls out the package context.

Oops! I missed that part. Maybe that makes my problem imaginary, except
that you go on to say...

> The checkin comment I made back in 1997 explains this:
>
>     Fix importing of shared libraries from inside packages.
>     This is a bit of a hack: when the shared library is loaded, the
>     module name is "package.module", but the module calls
>     Py_InitModule*() with just "module" for the name.  The shared
>     library loader squirrels away the true name of the module in
>     _Py_PackageContext, and Py_InitModule*() will substitute this (if
>     the name actually matches).
>
> > b. Is there a more-sanctioned way around this other than touching
> > _Py_PackageContext (which seems to be intended to be private)
>
> I think using _Py_PackageContext is your only hope.  If you contribute
> some docs for it we'll gladly add them to the API docs.

Hmm, my only hope for what? What I was worried about was that if I tried to
create a nested sub-extension module from within my extension module by
calling Py_InitModuleXXX() directly, its name would be forced to be the
same as that of the outer extension module. Since you pointed out that
_Py_PackageContext was being nulled out, I don't think that's much of an
issue. What issues /do/ I need to be aware of when doing this?

Thanks,
Dave


-----------------------------------------------------------
           David Abrahams * Boost Consulting
dave@boost-consulting.com * http://www.boost-consulting.com