[Python-Dev] PEP 402: Simplified Package Layout and Partitioning

[Glyph Lefkowitz]

On Aug 12, 2011, at 11:24 AM, P.J. Eby wrote:

> That is, the above code hardocdes a variety of assumptions about the import system that haven't been true since Python 2.3.

Thanks for this feedback.  I honestly did not realize how old and creaky this code had gotten.  It was originally developed for Python 2.4 and it certainly shows its age.  Practically speaking, the code is correct for the bundled importers, and paths and zipfiles are all we've cared about thus far.

> (For example, it assumes that the contents of sys.path strings have inspectable semantics, that the contents of __file__ can tell you things about the module-ness or package-ness of a module object, etc.)

Unfortunately, the primary goal of this code is to do something impossible - walk the module hierarchy without importing any code.  So some heuristics are necessary.  Upon further reflection, PEP 402 _will_ make dealing with namespace packages from this code considerably easier: we won't need to do AST analysis to look for a __path__ attribute or anything gross like that improve correctness; we can just look in various directories on sys.path and accurately predict what __path__ will be synthesized to be.

However, the isPackage() method can and should be looking at the module if it's already loaded, and not always guessing based on paths.  The whole reason there's an 'importPackages' flag to walk() is that some applications of this code care more about accuracy than others, so it tries to be as correct as it can be.

(Of course this is still wrong for the case where a __path__ is dynamically constructed by user code, but there's only so well one can do at that.)

> If you want to fully support PEP 302, you might want to consider making this a wrapper over the corresponding pkgutil APIs (available since Python 2.5) that do roughly the same things, but which delegate all path string inspection to importer objects and allow extensible delegation for importers that don't support the optional methods involved.

This code still needs to support Python 2.4, but I will make a note of this for future reference.

> (Of course, if the pkgutil APIs are missing something you need, perhaps you could propose additions.)

>> Now it seems like pure virtual packages are going to introduce a new type of special case into the hierarchy which have neither .pathEntry nor .filePath objects.
> 
> The problem is that your API's notion that these things exist as coherent concepts was never really a valid assumption in the first place.  .pth files and namespace packages already meant that the idea of a package coming from a single path entry made no sense.  And namespace packages installed by setuptools' system packaging mode *don't have a __file__ attribute* today...  heck they don't have __init__ modules, either.

The fact that getModule('sys') breaks is reason enough to re-visit some of these design decisions.

> So, adding virtual packages isn't actually going to change anything, except perhaps by making these scenarios more common.

In that case, I guess it's a good thing; these bugs should be dealt with.  Thanks for pointing them out.  My opinion of PEP 402 has been completely reversed - although I'd still like to see a section about the module system from a library/tools author point of view rather than a time-traveling perl user's narrative :).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20110812/43f1b847/attachment.html>