[Python-Dev] advice needed: best approach to enabling "metamodules"?

[Nathaniel Smith]

On Sun, Nov 30, 2014 at 10:14 PM, Mark Shannon <mark at hotpy.org> wrote:
> Hi,
>
> This discussion has been going on for a while, but no one has questioned the
> basic premise. Does this needs any change to the language or interpreter?
>
> I believe it does not. I'm modified your original metamodule.py to not use
> ctypes and support reloading:
> https://gist.github.com/markshannon/1868e7e6115d70ce6e76

Interesting approach!

As written, your code will blow up on any python < 3.4, because when
old_module gets deallocated it'll wipe the module dict clean. And I
guess even on >=3.4, this might still happen if old_module somehow
manages to get itself into a reference loop before getting
deallocated. (Hopefully not, but what a nightmare to debug if it did.)
However, both of these issues can be fixed by stashing a reference to
old_module somewhere in new_module.

The __class__ = ModuleType trick is super-clever but makes me
irrationally uncomfortable. I know that this is documented as a valid
method of fooling isinstance(), but I didn't know that until your
yesterday, and the idea of objects where type(foo) is not
foo.__class__ strikes me as somewhat blasphemous. Maybe this is all
fine though.

The pseudo-module objects generated this way will still won't pass
PyModule_Check, so in theory this could produce behavioural
differences. I can't name any specific places where this will break
things, though. From a quick skim of the CPython source, a few
observations: It means the PyModule_* API functions won't work (e.g.
PyModule_GetDict); maybe these aren't used enough to matter. It looks
like the __reduce__ methods on "method objects"
(Objects/methodobject.c) have a special check for ->m_self being a
module object, and won't pickle correctly if ->m_self ends up pointing
to one of these pseudo-modules. I have no idea how one ends up with a
method whose ->m_self points to a module object, though -- maybe it
never actually happens. PyImport_Cleanup treats module objects
differently from non-module objects during shutdown.

I guess it also has the mild limitation that it doesn't work with
extension modules, but eh. Mostly I'd be nervous about the two points
above.

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org