[Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Eric V. Smith]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/22/2015 08:31 PM, Donald Stufft wrote:
> 
>> On Apr 22, 2015, at 5:25 PM, Chris Barker <chris.barker at noaa.gov 
>> <mailto:chris.barker at noaa.gov>> wrote:
>> 
>> A note from the peanut gallery:
>> 
>> I  like the idea of namepace packages, but every time I've tried
>> to use them, I've been stymied -- maybe this PEP will solve that,
>> but...
>> 
>> First -  the issues:
>> 
>> - It somehow seems like a lot of work, details to get right, and 
>> more-than-one-way-to-do-it. But maybe that's all pre- PEP 420
>> 
>> - Last time I tried, I couldn't get them to work with "setup.py
>> develop"
>> 
>> But at the core of this -- Why does it have to be so hard? It
>> seems very simple to me -- what am I missing?
>> 
>> What are namespace packages? To me, they are a package that
>> serves no other purpose than to provide a single namespace in
>> which to put other packages. This makes a lot of sense if you
>> have a bunch of related packages where users may only require
>> one, or a couple, but not all. And you want to be able to
>> maintain them, and version control them independently.
>> 
>> But it seem to get this, all we need is:
>> 
>> 1) A directory with the top-level name
>> 
>> 2) It has an (empty)  __init__.py (so it is a python package)
>> 
>> 3) It has other directories in it -- each of these are regular
>> old python packages -- the ONLY difference is that they are
>> installed under that name
>> 
>> That's it. Done. Now all we need is a way to install these things
>> -- well that's easy, each sub-package installs itself just like
>> it would, maybe overwriting the top level directory name and the
>> __init__.py, if another sub-package has already installed it. But
>> that's OK, because the name is by definition the same, and the
>> __init__ is empty.
>> 
>> This seems SO SIMPLE. No declaring things all over the place, no 
>> dynamic path manipulation, nothing unusual at all, except the
>> ability to install a module into a dir without clobbering what
>> might already be in that dir.
>> 
>> What am I missing?
> 
> 
> Prior to PEP 420 you needed the dynamic path stuff because
> sometimes your namespace package is split across multiple locations
> on sys.path. Relying on the filesystem as you mentioned only works
> if you install every single namespace package into the same
> directory.
> 
> PEP 420 more or less solves all of the problems with namespace
> packages, other than it’s a Python 3 only feature so most people
> aren’t going to be willing to depend on it.

Right. The problem is that there are 2 ways to install namespace
packages. Say you have a namespace package foo, with 2 "portions"
named bar and baz (see:
https://www.python.org/dev/peps/pep-0420/#terminology). There are 2
ways to install these 2 portions:

1. in 2 different directories on sys.path, say /somewhere/foo-bar and
/somewhere/foo-baz.

2. in a single /somewhere/foo directory on sys.path, with the files
unioned on top of each other, say /somewhere/foo/bar and
/somewhere/foo/baz.

The problem that PEP 420 tries to solve is: what do I do with
foo/__init__.py? Prior to PEP 420, you'd include a copy in both
portions. This foo/__init__.py would contain the magic incantations to
call extend_path(), or whatever similar thing setuptools supports.

This works great for scenario 1 above. You can install the bar portion
or that baz portion, or both, and everything works fine. Note that all
copies of /somewhere/foo/*/__init__.py have the same contents.

The problem is in the second scenario. This is the scenario bdist_rpm
supports. But which RPM owns /somewhere/foo/__init__.py if both
portions are installed? It's the same file in terms of its contents,
and you'd like them to just overlay each other (with either one
winning), but with RPM and other package managers you can't do this,
because you'd have 2 different packages both owning the same file.

So what PEP 420 does is just delete foo/__init__.py. It's never
needed, by any package. Now you can install all namespace packages
with either scenario 1 or 2 above, and everything just works.

Eric.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)

iQEcBAEBAgAGBQJVOE+pAAoJENxauZFcKtNxEf0H/jgUwkdj0q6CsxMC2UPPQg0o
grjhL2FMVfbqhy74aJ0stJhBQ6+fSFR09b6LJ8va3Ql2iJLyXQVX0Kedhts9Hjud
zpfMxpQdxeKE41QjCjeua5hQjTFpqQofxMmcwmjoDOB89Tn+30K1gatPJ4xzjTRc
Lek5UT4yaZTS6mil61vdPUZMSWbxppJQBI0/EUmK9ps4vW3OfVxHMJK0AbvUbRnN
oXRW+NlEhXL2FtMdaoApQQL1STmsH0+RrRY+XlgGbT2G1LbXNviWMLaookUKwLJd
9V4SjS/dgepO9hqL7glSZU7/3THOpF5548gwzmPKuq65bYkx0XzqjUQzaWE6Guo=
=+J5a
-----END PGP SIGNATURE-----