[Python-ideas] Implicit submodule imports

[Andrew Barnert]

On Sep 25, 2014, at 18:02, Nathaniel Smith <njs at pobox.com> wrote:

> On Thu, Sep 25, 2014 at 11:31 PM, Greg Ewing
> <greg.ewing at canterbury.ac.nz> wrote:
>> Nathaniel Smith wrote:
>>> 
>>> They are really really hard
>>> to do cleanly, and you risk all kinds of breakage in edge-cases (e.g.
>>> try reload()'ing a module that's been replaced by an object).
>> 
>> One small thing that might help is to allow the
>> __class__ of a module to be reassigned to a
>> subclass of the module type. That would allow
>> a module to be given custom behaviours, while
>> remaining a real module object so that reload()
>> etc. continue to work.
> 
> Heh, I was actually just pondering whether it would be opening too big
> a can of worms to suggest this myself. This is the best design I
> managed to come up with last time I looked at it, though in existing
> versions of python it requires ctypes hackitude to accomplish the
> __class__ reassignment. (The advantages of this approach are that (1)
> you get to use the full class machinery to define your "metamodule",
> (2) any existing references to the module get transformed in-place, so
> you don't have to worry about ending up with a mixture of old and new
> instances existing in the same program, (3) by subclassing and
> avoiding copying you automatically support all the subtleties and
> internal fields of actual module objects in a forward- and
> backward-compatible way.)

When I tried this a year or two ago, I did I with an import hook that allows you to specify metaclass=absolute.qualified.spam in any comment that comes before any non-comment lines, so you actually construct the module object as a subclass instance rather than re-classing it.

In theory that seems a lot cleaner. In practice it's a weird way to specify your type; it only works if the import-hooking module and the module that defines your type have already been imported and otherwise silently does the wrong thing; and my implementation was pretty hideous.

Is there a cleaner version of that we could do if we were modifying the normal import machinery instead of hooking it, and if it didn't have to work pre-3.4, and if it were part of the language instead of a hack?

IIRC (too hard to check from my phone on the train), a module is built by calling exec with a new global dict and then calling the module constructor with that dict, so it's just a matter of something like:

    cls = g.get('__metamodule__', module)
    if not issubclass(cls, module):
        raise TypeError('metamodule
{} is not a module type'.format(cls))
    mod = cls(name, doc, g)
    # etc.

Then you could import the module subclass and assign it to __metamodule__ from inside, rather than needing to pre-import stuff, and you'd get perfectly understandable errors, and so on.

It seems less hacky and more flexible than re-classing the module after construction, for the same reason metaclasses and, for that matter, normal class constructors are better than reclassing after the fact.

Of course I could be misremembering how modules are constructed, in which case... Never mind.

> 
> This would work today, and would solve all these problems, except for
> the following code in Objects/typeobject.c:object_set_class:
> 
>    if (!(newto->tp_flags & Py_TPFLAGS_HEAPTYPE) ||
>        !(oldto->tp_flags & Py_TPFLAGS_HEAPTYPE))
>    {
>        PyErr_Format(PyExc_TypeError,
>                     "__class__ assignment: only for heap types");
>        return -1;
>    }
>    if (compatible_for_assignment(oldto, newto, "__class__")) {
>        Py_INCREF(newto);
>        Py_TYPE(self) = newto;
>        Py_DECREF(oldto);
>        return 0;
>    }
> 
> The builtin "module" type is not a HEAPTYPE, so if we try to do
> mymodule.__class__ = mysubclass, then the !(oldto->tp_flags &
> Py_TPFLAGS_HEAPTYPE) check gets triggered and the assignment fails.
> 
> This code has been around forever, but I don't know why. AFAIK we
> could replace the above with
> 
>    if (compatible_for_assignment(oldto, newto, "__class__")) {
>        if (newto->tp_flags & Py_TPFLAGS_HEAPTYPE) {
>            Py_INCREF(newto);
>        }
>        Py_TYPE(self) = newto;
>        if (oldto->tp_flags & Py_TPFLAGS_HEAPTYPE) {
>           Py_DECREF(oldto);
>        }
>        return 0;
>    }
> 
> and everything would just work, but I could well be missing something?
> Is there some dragon lurking inside Python's memory management or is
> this just an ancient overabundance of caution?
> 
> -n
> 
> -- 
> Nathaniel J. Smith
> Postdoctoral researcher - Informatics - University of Edinburgh
> http://vorpus.org
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/