[Import-SIG] PEP proposal: Per-Module Import Path

[Brett Cannon]

[SNIP to stop Mailman from holding up the email for moderation because of
size]

 A Module Attribute to Expose Contributing Ref Files
>>> ---------------------------------------------
>>>
>>> Knowing the origin of a module is important when tracking down problems,
>>> particularly import-related ones.  Currently, that entails looking at
>>> `<module>.__file__` and `<module.__package__>.__path__` (or `sys.path`).
>>>
>>> With this PEP there can be a chain of ref files in between the currently
>>> available path and a module's __file__.  Having access to that list of ref
>>> files is important in order to determine why one file was selected over
>>> another as the origin for the module.  When an unexpected file gets used
>>> for one of your imports, you'll care about this!
>>>
>>> In order to facilitate that, modules will have a new attribute:
>>> `__indirect__`.  It will be a tuple comprised of the chain of ref files, in
>>> order, used to locate the module's __file__.  An empty tuple or with one
>>> item will be the most common case.  An empty tuple indicates that no ref
>>> files were used to locate the module.
>>>
>>
>> This complicates things even further. How are you going to pass this info
>> along a call chain through find_loader()? Are we going to have to add
>> find_loader3() to support this (nasty side-effect of using tuples instead
>> of types.SimpleNamespace for the return value)? Some magic second value or
>> type from find_loader() which flags the values in the iterable are from a
>> .ref file and not any other possible place? This requires an API change and
>> there isn't any mention of how that would look or work.
>>
>
> This is the big open question in my mind.  I suppose having find_loader()
> return a SimpleNamespace would help.  Then the indirect path we aggregate
> in find_loader() could be passed as a new argument to loaders (when
> instantiated in either FileFinder.find_loader() or in
> PathFinder.find_module().
>
> Here are the options I see, some more realistic than others:
>
> 1. Build __indirect__ after the fact (in init_module_attrs()?).
> 2. Change FileFinder.find_loader() to return a types.SimpleNamespace
> instance.
>

You can't do that; it would change the method signature in a way that would
break code automatically unpacking the tuple. I was lamenting the fact that
no one thought to use types.SimpleNamespace in the first place, not
suggesting it now be used.


> 3. Change FileFinder.find_loader() to return a namedtuple subclass with an
> extra "loader" attribute.
>

Only if it also subclassed dict or types.SimpleNamespace so that it could
be documented that in Python 4 the tuple usage will be removed but the new,
alternative access approach would continue to work. Plus something in
importlib.util to help construct this monstrosity of an object so people
future-proof their code.


> 4. Piggy-back the indirect path on the loader returned by
> FileFinder.find_loader() in an "_indirect" attribute (or in the loader spot
> in the case of namespace packages).
>

Doesn't that tie this very tightly to FileFinder and not allowing
alernative finders to participate?


> 5. Something along the lines of Nick's IndirectReference.
>

I would avoid having to do any change that requires an isinstance check.
New code can use getattr() (like with a namedtuple hybrid) w/o issue since
that's a common way of dealing with API expansion, but having changing
types in the return is something even Guido has said he doesn't care for.

6. Wrap the loader in a proxy that also sets __indirect__ when
> load_module() is called.
>

Ew.


> 7. Totally refactor the import system so that ModuleSpec objects are
> passed to metapath finders rather than (name, path) and simply store the
> indirect path on the spec (which is used directly to load the module rather
> than the loader).
>
>
Yeah, that ain't going to happen for backwards-compatibility reasons unless
you're ready to make this new API work in a fully compatible way with the
current API.


> 4 feels too much like a hack, particularly when we have other options.  7
> would need a PEP of its own (forthcoming <wink>).
>
>>
> I see 2 as the best one.  Is it really too late to change the return type
> of FileFinder.find_loader()? If we simply can't bear the backward
> compatibility risk (no matter how small <wink>),
>

We unfortunately can't. It would require a new method which as a stub would
call the old API to return the proper object (which is fine if you can come
up with a reasonable name).


> I'd advocate for one of 1, 3, 5, or 6.
>

I would try 1 and 3.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20130801/48ac52ca/attachment.html>