[Distutils] namespace packages

[David Cournapeau]

On Sat, Apr 24, 2010 at 1:32 AM, P.J. Eby <pje at telecommunity.com> wrote:

>
> If you don't mind trying a simple test for me, would you patch your
> pkg_resources to comment out this loop:
>
>            for pkg in self._get_metadata('namespace_packages.txt'):
>                if pkg in sys.modules: declare_namespace(pkg)

That looks much better. It is roughly half the time (450 ms -> 250
ms). I had a simple test set with a directory containing N empty
*.egg-info directory, and the import time was proportional to N, now
it does not matter anymore.


> This change is not backward compatible with some older packages (from years
> ago) that were not declaring their namespace packages correctly, but it has
> been announced for some time (with warnings) that such packages will not
> work with setuptools 0.7.
>
> (By the way, in case you're thinking this change would only affect namespace
> packages, and you don't have any, what's happening is that the
> _get_metadata() call forces a check for the *existence* of
> namespace_packages.txt in every .egg-info or .egg/EGG-INFO on your path,
> whether the file actually exists or not.  In the case of zipped eggs, this
> check is just looking in a dictionary; for actual files/directories, this is
> a stat call.)

Yes, that's exactly what I was seeing in the strace output. Is there a
design document or something else decribing how the namespace
mechanism works for setuptools ? I would like to support namespace
package in my own packaging project, but it is not clear to me what
needs to be done on my side of things.

David