[Distutils] Experimental non-empty namespace package support

[Phillip J. Eby]

I've just checked in experimental support for lazy, non-empty namespace 
packages.  And what is that?, you might well ask.

Well, it seems that livinglogic.de distributes certain Python packages 
using a distutils kludge that allows a kind of crude namespace package to 
exist, without using pkg_resources.  Specifically, their 'll-core' project 
distributes the 'll' package, and various other projects such as 'll-color' 
distribute modules for the 'll' package, but *without including an 
__init__.py*.  This allows the distutils to install the modules without 
overwriting or duplicating the single __init__.py file.

This is an interesting approach to addressing namespace packages in a 
pre-setuptools world, but until now it hasn't been really usable with 
setuptools, for several reasons:

   * pkg_resources only loaded one __init__ file for a given package

   * pkg_resources couldn't import modules from a zipfile or directory with 
no __init__.py

   * pkg_resources automatically imports registered namespace packages as 
soon as they're discovered, which is not a big deal for empty namespace 
packages, but could easily become quite problematic for ones like 'll' that 
contain actual code.

So, here's the solution:

    * pkg_resources now loads all __init__.py's from all the distributions 
that provide contents for the module.

    * the bdist_egg command automatically generates a dummy __init__.py for 
packages that don't have them.  The __init__.py file contains the single line:

      __import__('pkg_resources').declare_namespace(__name__)

      This ensures that the other package contents are importable, and that 
as soon as an import of the package occurs, it gets registered as a 
namespace package.

    * projects that use this approach to namespace packaging must *NOT* 
pass a 'namespace_packages' argument to setup(), because that would cause 
the package to be imported even when it isn't necessary.  They must also 
put a 'declare_namespace()' call in all of their __init__.py's, to ensure 
that the package will become a namespace no matter what the order of 
distributions on sys.path is.

Because of the complexity of this approach, I do not recommend it for new 
projects.  Namespace packages should not have code in any of their 
__init__.py files, as it eliminates all of these headaches.  So, this is an 
experimental feature intended to facilitate backward compatibility 
only.  Don't expect it to show up in the documentation any time soon!

Anyway, I was able to get this to work for ll-core and ll-color, as long as 
I added a 'declare_namespace()' call like the one shown above to ll-core's 
__init__.py, and made them both use setuptools.  I would appreciate any 
feedback that folks have on this.

By the way, because SourceForge CVS updating is slow, it may be a few hours 
before you can check out the version of setuptools that supports 
this.  Make sure you have pkg_resources.py revision 1.69 or higher, and 
setuptools/command/bdist_egg.py revision 1.28 or higher, if you want to 
experiment.  Thanks.