[Python-Dev] Unique loader per module

[Nick Coghlan]

On 3 January 2018 at 06:35, Barry Warsaw <barry at python.org> wrote:
> Brett doesn’t like this, for several reasons (quoting):
>
> 1. redundant API in all cases where the loader is unique to the module
> 2. the memory savings of sharing a loader is small
> 3. it's implementation complexity/overhead for an optimization case.
>
> The second solution, and the one Brett prefers, is to reimplement zip importer to not use a shared loader.  This may not be that difficult, if for example we were to use a delegate loader wrapping a shared loader.
>
> The bigger problem IMHO is two-fold:
>
> 1. It would be backward incompatible.  If there’s any code out there expecting a shared loader in zipimport, it would break
> 2. More problematic is that we’d have to impose an additional requirement on loaders - that they always be unique per module, contradicting the advice in PEP 302

We added module.__spec__.loader_state as part of PEP 451 precisely so
shared loaders had a place to store per-module state without have to
switch to a unique-loader-per-module model.

I think the main reason you're seeing a problem here is because
ResourceReader has currently been designed to be implemented directly
by loaders, rather than being a subcomponent that you can request
*from* a loader.

If you instead had an indirection API (that could optionally return
self in the case of non-shared loaders), you'd keep the current
resource reader method signatures, but the way you'd access the itself
would be:

    resources = module.__spec__.loader.get_resource_reader(module)
    # resources implements the ResourceReader ABC

For actual use, the loader protocol could be hidden behind a helper function:

    resources = importlib_resources.get_resource_reader(module)

For a shared loader, get_resource_reader(module) would return a new
*non*-shared resource reader (perhaps caching it in
__spec__.loader_state).

For a non-shared loader, get_resource_reader(module) would just return self.

In both cases, we'd recommend that loaders ensure "self is
module.__spec__.loader" as part of their get_resource_reader()
implementation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia