[Distutils] namespace packages

[Tarek Ziadé]

On Fri, Apr 23, 2010 at 10:30 AM, David Cournapeau <cournape at gmail.com> wrote:
[..]
> This all sounds complicated. pkg_resources is already complicated
> enough (the other big reason why I don't use it in any of my
> packages). Without a clear specification of what pkg_resources or its
> successor is doing, caching and C-based implementation sound like
> premature optimization to me. For example, pkg_resources does a lot of
> things which seem quite orthogonal to me. While scanning every package
> may be needed for namespace package support the "setuptools" way, this
> is almost never useful for locating resources at runtime, but you pay
> the price in both cases.

I am not sure what you are defining as "complicated". While pkg_resources
is hard to read and it's a project on its own with many other
features, the use case
we are talking about here is dead simple:

  scan all sys.path entries to look for .egg and .egg-info files/directories.

It's setuptools' "live" installed packages database index. It has to
be recalculated
at every run because sys.path and the directories it points may change.

That's the basic feature it needs for a broad range of features, and
pkg_util will need
it as well in a near future in the stdlib. It's the de-facto way to
find out what's installed (see PEP 376)

We have an implementation
(http://bitbucket.org/tarek/pep376/src/tip/pkgutil.py) that is being
moved
to distutils2, and that will probably land in the stdlib if PEP 376 is accepted.

Now, about the C part, what I am saying is that I don't know if we
will be able to reduce
the numbers of I/O to read the content of directories if we use C, but
I do know that string comparisons
will be faster in C.

Hey, I barely did any C since college, I am going to give it a shot
and see if it goes faster :)
I am just scared that this can be quite complex when I take a look at
posixpath.c.. any C expert here that want to help ?

Regards
Tarek

-- 
Tarek Ziadé | http://ziade.org