[Python-Dev] making the import machinery explicit

[Eric Snow]

On Sat, Apr 14, 2012 at 2:03 PM, Brett Cannon <brett at python.org> wrote:
> To start off, what I am about to propose was brought up at the PyCon
> language summit and the whole room agreed with what I want to do here, so I
> honestly don't expect much of an argument (famous last words).
>
> In the "ancient" import.c days, a lot of import's stuff was hidden deep in
> the C code and in no way exposed to the user. But with importlib finishing
> PEP 302's phase 2 plans of getting imoprt to be properly refactored to use
> importers, path hooks, etc., this need no longer be the case.
>
> So what I propose to do is stop having import have any kind of implicit
> machinery. This means sys.meta_path gets a path finder that does the heavy
> lifting for import and sys.path_hooks gets a hook which provides a default
> finder. As of right now those two pieces of machinery are entirely implicit
> in importlib and can't be modified, stopped, etc.
>
> If this happens, what changes? First, more of importlib will get publicly
> exposed (e.g. the meta path finder would become public instead of private
> like it is along with everything else that is publicly exposed). Second,
> import itself technically becomes much simpler since it really then is about
> resolving module names, traversing sys.meta_path, and then handling fromlist
> w/ everything else coming from how the path finder and path hook work.
>
> What also changes is that sys.meta_path and sys.path_hooks cannot be blindly
> reset w/o blowing out import. I doubt anyone is even touching those
> attributes in the common case, and the few that do can easily just stop
> wiping out those two lists. If people really care we can do a warning in 3.3
> if they are found to be empty and then fall back to old semantics, but I
> honestly don't see this being an issue as backwards-compatibility would just
> require being more careful of what you delete (which I have been warning
> people to do for years now) which is a minor code change which falls in line
> with what goes along with any new Python version.
>
> And lastly, sticking None in sys.path_importer_cache would no longer mean
> "do the implicit thing" and instead would mean the same as NullImporter does
> now (which also means import can put None into sys.path_importer_cache
> instead of NullImporter): no finder is available for an entry on sys.path
> when None is found. Once again, I don't see anyone explicitly sticking None
> into sys.path_importer_cache, and if they are they can easily stick what
> will be the newly exposed finder in there instead. The more common case
> would be people wiping out all entries of NullImporter so as to have a new
> sys.path_hook entry take effect. That code would instead need to clear out
> None on top of NullImporter as well (in Python 3.2 and earlier this would
> just be a performance loss, not a semantic change). So this too could change
> in Python 3.3 as long as people update their code like they do with any
> other new version of Python.
>
> In summary, I want no more magic "behind the curtain" for Python 3.3 and
> import: sys.meta_path and sys.path_hooks contain what they should and if
> they are emptied then imports will fail and None in sys.path_importer_cache
> means "no finder" instead of "use magical, implicit stuff".

This is great, Brett.  About sys.meta_path and sys.path_hooks, I see
only one potential backwards-compatibility problem.

Those implicit hooks were fallbacks, effectively always at the end of
the list, no matter how you manipulated the them.  Code that appended
onto those lists would now have to insert the importers/finders in the
right way.  Otherwise the default hooks would be tried first, which
has a good chance of being the wrong thing.

That concern aside, I'm a big +1 on your proposal.

-eric