[Python-Dev] Created branch for PEP 302 phase 2 work (in C)

[Paul Moore]

On 10/2/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> Just FYI, it's not possible (so far as I know) to implement phase 2 while
> maintaining backward compatibility with existing 2.x code.  So this work
> shouldn't go back to the 2.x trunk without discussion of those issues.

While that's a fair point, we need to be clear what compatibility
issues there are. The built in import mechanisms aren't well
documented, so it's not a black-and-white situation. An unqualified
statement "there are issues" isn't much help on its own...

> Essentially, I abandoned trying to do the phase 2 work for Python 2.5
> because there's too much code in the field that depends on the current
> order of when special/built-in imports are processed vs. when PEP 302
> imports are processed.

Can you say what that code is, and who we should be talking to to
understand their issues? If not, how do we find such code? Presumably,
you've got a lot of feedback through your work on setuptools/eggs - do
you have a record of who might participate in a discussion?

> Thus, instead of adding new PEP 302 APIs (like
> get_loader) to 'imp', I added them to 'pkgutil'.

How does that help? Where the code goes doesn't seem likely to make
much difference...

> There are, I believe,
> some notes in that module's source regarding what the ordering issues are
> w/meta_path vs. the way import works now.

The only notes I could see in pkgutil.py refer to special locations
like the Windows registry, and refer to the fact that they will be
searched after path entries, not before (for reasons I couldn't quite
follow, but that's likely because I only read the comments fairly
quickly). But if the whole mechanism is moved to sys.meta_path (which
is what Phase 2 is about) surely it's possible to choose the ordering
just by the order the importers go on sys.meta_path?

> That having been said, we could possibly have a transition for 2.6, but
> everybody who's written any PEP 302 emulation code (outside of pkgutil
> itself) would have to adapt their code somewhat.

I don't really see how we're going to address that other than by
implementing it, and waiting for people with issues to speak up.
Highlighting the changes early is good, as it avoids a mid-beta rush
of people "suddenly" finding issues, but I doubt we'll do much better
than that.

> I'm surprised, however, that you think working on this in C is going to be
> *less* time than it would take to simply replace __import__ with a Python
> function that reimplements PEP 302...

That I do agree with. There's a bootstrapping issue (you can't import
the Python module that does all this without using a C-coded import
mechanism) but that should be resolvable.

>  especially since pkgutil contains a
> whole lot of the code you'd need, e.g.:

Yes, I'm quite surprised at how much has appeared in pkgutil. The
"what's new" entry is very terse, and the module documentation itself
hasn't been updated to mention the new stuff. That's a shame, as it
looks very useful (and as you say, could form a substantial part of
this change if we were coding it in Python).

Paul.