[Python-Dev] advice needed: best approach to enabling "metamodules"?

[Nathaniel Smith]

On Mon, Dec 1, 2014 at 1:27 AM, Guido van Rossum <guido at python.org> wrote:
> Nathaniel, did you look at Brett's LazyLoader? It overcomes the subclass
> issue by using a module loader that makes all modules instances of a
> (trivial) Module subclass. I'm sure this approach can be backported as far
> as you need to go.

The problem is that by the time your package's code starts running,
it's too late to install such a loader. Brett's strategy works well
for lazy-loading submodules (e.g., making it so 'import numpy' makes
'numpy.testing' available, but without the speed hit of importing it
immediately), but it doesn't help if you want to actually hook
attribute access on your top-level package (e.g., making 'numpy.foo'
trigger a DeprecationWarning -- we have a lot of stupid exported
constants that we can never get rid of because our rules say that we
have to deprecate things before removing them).

Or maybe you're suggesting that we define a trivial heap-allocated
subclass of PyModule_Type and use that everywhere, as a
quick-and-dirty way to enable __class__ assignment? (E.g., return it
from PyModule_New?) I considered this before but hesitated b/c it
could potentially break backwards compatibility -- e.g. if code A
creates a PyModule_Type object directly without going through
PyModule_New, and then code B checks whether the resulting object is a
module by doing isinstance(x, type(sys)), this will break. (type(sys)
is a pretty common way to get a handle to ModuleType -- in fact both
types.py and importlib use it.) So in my mind I sorta lumped it in
with my Option 2, "minor compatibility break". OTOH maybe anyone who
creates a module object without going through PyModule_New deserves
whatever they get.

-n

> On Sun, Nov 30, 2014 at 5:02 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Mon, Dec 1, 2014 at 12:59 AM, Nathaniel Smith <njs at pobox.com> wrote:
>> > On Sun, Nov 30, 2014 at 10:14 PM, Mark Shannon <mark at hotpy.org> wrote:
>> >> Hi,
>> >>
>> >> This discussion has been going on for a while, but no one has
>> >> questioned the
>> >> basic premise. Does this needs any change to the language or
>> >> interpreter?
>> >>
>> >> I believe it does not. I'm modified your original metamodule.py to not
>> >> use
>> >> ctypes and support reloading:
>> >> https://gist.github.com/markshannon/1868e7e6115d70ce6e76
>> >
>> > Interesting approach!
>> >
>> > As written, your code will blow up on any python < 3.4, because when
>> > old_module gets deallocated it'll wipe the module dict clean. And I
>> > guess even on >=3.4, this might still happen if old_module somehow
>> > manages to get itself into a reference loop before getting
>> > deallocated. (Hopefully not, but what a nightmare to debug if it did.)
>> > However, both of these issues can be fixed by stashing a reference to
>> > old_module somewhere in new_module.
>> >
>> > The __class__ = ModuleType trick is super-clever but makes me
>> > irrationally uncomfortable. I know that this is documented as a valid
>> > method of fooling isinstance(), but I didn't know that until your
>> > yesterday, and the idea of objects where type(foo) is not
>> > foo.__class__ strikes me as somewhat blasphemous. Maybe this is all
>> > fine though.
>> >
>> > The pseudo-module objects generated this way will still won't pass
>> > PyModule_Check, so in theory this could produce behavioural
>> > differences. I can't name any specific places where this will break
>> > things, though. From a quick skim of the CPython source, a few
>> > observations: It means the PyModule_* API functions won't work (e.g.
>> > PyModule_GetDict); maybe these aren't used enough to matter. It looks
>> > like the __reduce__ methods on "method objects"
>> > (Objects/methodobject.c) have a special check for ->m_self being a
>> > module object, and won't pickle correctly if ->m_self ends up pointing
>> > to one of these pseudo-modules. I have no idea how one ends up with a
>> > method whose ->m_self points to a module object, though -- maybe it
>> > never actually happens. PyImport_Cleanup treats module objects
>> > differently from non-module objects during shutdown.
>>
>> Actually, there is one showstopper here -- in the first version where
>> reload() uses isinstance() is actually 3.4. Before that you need a
>> real module subtype for reload to work. But this is in principle
>> workaroundable by using subclassing + ctypes on old versions of python
>> and the __class__ = hack on new versions.
>>
>> > I guess it also has the mild limitation that it doesn't work with
>> > extension modules, but eh. Mostly I'd be nervous about the two points
>> > above.
>> >
>> > -n
>> >
>> > --
>> > Nathaniel J. Smith
>> > Postdoctoral researcher - Informatics - University of Edinburgh
>> > http://vorpus.org
>>
>>
>>
>> --
>> Nathaniel J. Smith
>> Postdoctoral researcher - Informatics - University of Edinburgh
>> http://vorpus.org
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)



-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org