[Import-SIG] My objections to implicit package directories

[Barry Warsaw]

On Mar 12, 2012, at 08:49 PM, Guido van Rossum wrote:

>*If* there is a foo.py or a foo/__init__.py anywhere along sys.path,
>the *current* rules apply. That is, of one of these occurs on an
>earlier sys.path entry, it wins; if both of these occur together on
>the same sys.path entry, foo/__init__.py wins. (We discovered that the
>latter disambiguation must prefer the directory, not just for
>backwards compatibility, but also to make relative imports in
>subpackages work right. This is probably the biggest deviation from
>PEP 402.) And in this case all those foo/ directories *without* a
>__init__.py in them are completely ignored, even if they come before
>either foo.py or foo/__init__.py on sys.path. (If __init__.py wants to
>manipulate its own __path__, that's fine.)
>
>*Only* if *neither* foo.py *nor* foo/__init__.py is found *anywhere*
>along sys.path do we take all directories foo/ along sys.path together
>and combine them into a namespace package. If there are no foo/
>directories at all, the import fails. If there is exactly one foo/, it
>acts like a classic package with an empty __init__.py. We avoid having
>to do two scans of sys.path by collecting info about __init__.py-less
>foo/ directories during the same scan where we look for foo.py and
>foo/__init__.py; but we collect it in a separate variable. (It occurs
>to me that this may not be trivial when PEP-302-style finders are
>involved. That's a detail that will to be figured out later.)
>
>So the only backwards incompatibility is that "import foo" may succeed
>where it previously failed if there is a directory foo/ somewhere on
>sys.path but no foo.py and no foo/__init__.py anywhere. I don't think
>this is a big deal.

In my sleep-deprived thinking about this, I notice that like my previous
objection to .pyp directory names, this will make it difficult for vendors
like Debian to support namespace packages when providing both Python 3.2 and
3.3.  However, unlike the current PEP 382, I see an easy way out during the
transition until we can ignore anything < 3.3.

While we still support 3.2 and 3.3, our package installation helpers will
simply continue to write __init__.py files into namespace package directories,
just as it does now.  Since we (generally, assuming nothing insane like extra
non-__path__ hacking code is in the upstream namespace __init__.py file)
generally substitute a consistently written __init__.py file for namespace
packages, containing only the standard __path__ hackery, this will continue to
work just as it does today even after this new PEP lands.  IOW, backward
compatibility FTW!

Once we can drop 3.2, our helper simply removes those namespace __init__.py
files, and all is happy in a 3.3-and-beyond world.

Cheers,
-Barry