[Import-SIG] New PEP draft: "Simplified Package Layout and Partitioning"

[P.J. Eby]

At 12:17 PM 7/18/2011 -0400, Barry Warsaw wrote:
>1. Sometimes, packages can have non-importable data directories,
>    e.g. foo/test/data.  Where foo.test would be an importable subpackage,
>    foo.test.data should not be.  Today we can just omit the __init__.py from
>    foo/test/data.  Under the proposed regime there would IIUC, be no way to
>    prevent foo.test.data from being a subpackage.  It's entirely 
> possible that
>    foo/test/data would have .py files in it which would themselves be
>    importable.  Is this a bad thing?

Why would it be?

>If so, do we need some mechanism to
>    prevent recursion into some subdirectories?

You could rename the subdirectory, I suppose.



>2. The __file__ issue.  My gut tells me that pure virtual modules should have
>    None as their __file__.  It seems wrong to use anything else, and your
>    "accidentally work" observation is not calming. ;)

Heh.  ;-)


>    The inability to use __file__ to find data files is somewhat troubling
>    though.  Let's say we want to find the foo/test/data subdir above, and
>    `foo` is pure-virtual, while `test` is an __init__.py-less package.
>
>    I'm fine not being able to use foo.__file__, but I will probably want to
>    use `os.path.join(foo.test.__file__, 'data')`.

Currently, you'd actually join to the dirname() of the __file__, not 
the plain file.  Thus, putting a directory name with a trailing '/' 
in __file__ would then make the current incantation work for that 
case, as long as you were fine with looking in the *first* directory 
where the file was.

However, I'm not as keen on that as a general solution, simply 
because if you add a 'foo/test.py', then the __file__ will change 
such that a different incantation is required to find the directory.


>   Will that work?  What would
>    foo.test's __file__ be?  The `foo/test` directory perhaps?  Of 
> course there
>    could be multiple `foo/test` directories, so this is probably why your
>    suggesting to search foo.test.__path__ instead.
>
>    I'd actually be okay with that, *if* pkg_resources will be updated to
>    handle this case.  In general, we've been recommending people use
>    pkg_resources anyway (wasn't there a push to move part of this 
> package into
>    the stdlib?).

pkg_resources says not to use a namespace package as your target for 
a lookup, but instead to always use a self-contained package or a 
module that's adjacent to what you're looking for, for this very 
reason.  There's really no change here.


>I'll read up on the rest of the thread now, but I think the PEP holds up well
>and makes a convincing argument.  I think it's certainly worthy of posting to
>python-dev to see if anybody else can shoot holes in it, or come up with
>useful solutions to open questions.  I'll be very interested to see Guido's
>reaction to it. :)

Me too.  ;-)