[Import-SIG] Thoughts on cleaner reloading support

[Nick Coghlan]

On 19 September 2013 15:34, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Mon, Sep 2, 2013 at 7:53 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> For a while, I was thinking we could design the import system to "just
>> figure it out", but now I'm thinking a selection of read/write
>> properties on spec objects may make more sense:
>>
>>     allow_reload
>>     allow_unload
>>     allow_reimport
>>     allow_subinterpreter_import
>>
>> These would all default to True, but loaders and modules could
>> selectively turn them off.
>
> 2 things:
>
> 1. These make more sense to me on the loader (though perhaps exposed on the
> spec).

They can't go on the loader, since even pure Python modules can
violate them. For example, if your Python module does the following
unconditionally, then it no longer supports in-place reloading:

    import sys
    sys.modules[__name__] = obj

So "allow" may be the wrong prefix. "handles" is probably better:

    handles_reload
    handles_unload
    handles_reimport
    handles_subinterpreter_import

For backwards compatibility, these would all default to True, but
loaders and modules would now have the ability to opt-out.

This should probably go hand-in-hand with a new "post import hooks"
PEP, since an "atimport" hook with the following signature might be
useful to allow other modules to properly handle reloading of
dependencies:

    @importlib.atimport("foo")
    def handle_foo_import(mod, reloaded):
        # If you need to do something when the module is destroyed:
        #    weakref.finalize(mod, my_callback)

Also, something neat about weakref.finalize, is that it means Python
effectively supports module destructors!

    import weakref, sys
    mod = sys.modules[__name__]
    def del_this():
        # implicit reference to the module globals from the function body
    weakref.finalize(mod, del_this)

Modules already support weak references in 3.4, and weakref.finalize
is new in 3.4 as well.

> 2. These (and other related attributes) may be easier to digest if bundled
> into a LoaderCapabilities named tuple.

Aside from it needing to vary by module rather than by loader, I'd
also be OK with:

    can_handle.reload
    can_handle.unload
    can_handle.reimport
    can_handle.subinterpreter_import

> For these attributes to be useful to the import system we would have to have
> several other module registries akin to sys.modules (or one registry that
> manages the extra info).

I was thinking it would be strictly warnings based, so you could still
*try* these things, you'd just have to deal with the consequences.

>> They would also be advisory
>
>
> Ah, here's where you were talking about "advisory" APIs. :)
>
>> rather than enforced via all possible
>> import state manipulation mechanisms. New functions in importlib.util
>> could provide easier alternatives to directly manipulating
>> sys.modules:
>>
>> - importlib.util.reload (replacement for imp.reload that checks the
>> spec allows reloading)
>> - importlib.util.unload (replacement for "del
>> sys.modules[module.__name__]" that checks the spec allows unloading,
>> and also unloads all child modules)
>> - importlib.util.reimport (replacement for
>> test.support.import_fresh_module that checks the spec of any existing
>> sys.module entry allows reimporting a parallel copy)
>
>
> At moments like this I keep thinking about PEP 406... :)

Yup, that's definitely relevant. It would probably be good to sort out
the thread-local context version of PEP 406 for 3.5 :)

>> One of these is not like the others... aside from the existing
>> extension module specific mechanism defined in PEP 3121, I'm not sure
>> we can devise a general *loader* level API to force imports for a
>> particular name to fail in a subinterpreter. So this concern probably
>> needs to be ignored in favour of a possible future C API level
>> solution.
>
> Again, great write-up.  I think you nailed it.

Yeah, the deep dive with Stefan into the extension loader
implementation greatly clarified my thinking on a lot of things :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia