[Distutils] Confusion about the effect of eggs on import

[Phillip J. Eby]

At 05:30 PM 2/9/2007 -0600, skip at pobox.com wrote:
>(I'm sending this to distutils because I have this vague notion that the
>problem might have something to do with setuptools or eggs on sys.path.)

Their mere presence on sys.path doesn't do anything, if you're not 
importing pkg_resources or importing anything from inside the eggs.

However, matplotlib imports pkg_resources and is a (broken) namespace package.


>We have a weird issue trying to import matplotlib at work.

I investigated a little, and here's what's happening.  Matplotlib declares 
'matplotlib.toolkits' as a namespace package, but you can't make a child 
package a namespace package without the parent being one too.  A namespace 
package combines all packages on sys.path with the same name into a single 
package.

So here's what's happening.  When setuptools is on sys.path, then importing 
matplotlib causes it to declare its namespace, which merges all copies of 
matplotlib on sys.path into a single super-package -- and invokes all the 
__init__.py's.

When setuptools is not on sys.path, the attempt to declare the namespace 
fails (it's wrapped in a try/except in matplotlib/__init__.py), so nothing 
else happens.

There are, I think, two problems here.  One, is that setuptools shouldn't 
be executing multiple __init__.py's for a package, but for backward 
compatibility reasons this isn't being dropped until 0.7a1.  Two, is that 
setuptools currently allows you to declare a child namespace package (like 
'matplotlib.toolkits') without explicitly declaring the parent to be a 
namespace package.  So, sometimes people declare a subpackage without 
realizing the parent will also have to be treated as a namespace package.

As a result, they end up thinking that they can include initialization code 
in the parent package, when in fact there are many circumstances where it 
simply won't work.  In this case, matplotlib/__init__.py contains 
executable code, which is a no-no for a namespace package.

My suggestion would be that matplotlib use a matplotlib_toolkits namespace 
package, rather than attempting to keep matplotlib.toolkits, since it 
appears they are relying on a substantial amount of code living in the 
__init__.py.

In short, the problem is a matplotlib bug, but in fairness it's probably 
due to the sketchy documentation surrounding the proper care and feeding of 
namespace packages, coupled with the implicit declaration of namespace 
packages.  All of the problems are explained in the setuptools 
documentation, but unfortunately that's not the same as anybody being able 
to figure out that the problems will apply to THEM.  ;)