[Distutils] Adding entry points into Distutils ?

Noah Gift noah.gift at gmail.com
Tue May 5 11:20:13 CEST 2009


On Tue, May 5, 2009 at 8:55 PM, Tarek Ziadé <ziade.tarek at gmail.com> wrote:
> 2009/5/5 P.J. Eby <pje at telecommunity.com>:
>> At 06:57 PM 5/4/2009 -0500, Ian Bicking wrote:
>>>
>>> * I'm uncomfortable with the way entry points are scanned.  I haven't
>>> looked close enough to back it up with numbers, but I think there's a
>>> noticeable performance degradation when the number of installed packages
>>> becomes large.  (Given the algorithm this would be expected.)
>>
>> It's linear in the number of entry_points.txt files, yes, but in most apps
>> it should occur at most once, since pkg_resources has a single WorkingSet
>> object holding Distribution objects which cache their entry point data upon
>> first access.
>>
>> There are all sorts of ways you could make different tradeoffs, but this
>> particular set of tradeoffs was optimized for a single-application
>> environment, rather than a massive global shared site-packages where there
>> are plugins for every application on the system.  It was also optimized for
>> the zipimport case, because you can tell whether a project has entry points
>> from its cached zip directory, that's needed at startup anyhow.
>>
>
> Would it make sense then to maintain an global index of all entry
> points, that would be updated
> upon installation / uninstallation (if it's added like we wrote in PEP 376)
> rather than scanning the paths everytime ?

I think so.  In an environment like the one I work in, a massive NFS
infrastructure with thousands of hosts, scanning upon startup is
painful.  I think writing the package list out to a file is the way to
go, and then scanning that file upon startup.  While you are at it, I
think someone also needs to look at some of things that are happening
up startup like looking for .pyc .so etc.  Unfortunately, file servers
are stupid and penalize you each time you stat a file that isn't
there.  I would like an expert mode that allowed me control exactly
what was looked at when the Python interpreter launched.  I blogged
about it here:

http://artificialcode.blogspot.com/2009/04/short-circuiting-python-module-lookup.html

If you run the python with strace you will see some interesting
things.  This is something that I feel is in dire need of optimization
and makes Python look pretty bad...

>
> we could have one index file per site-package-like directory, and
> discover index files
> rather than all directories / zip files.

I again like the expert mode idea, in which you could literally say,
"dammit, this is where stuff is and you aren't looking anywhere
else...I really mean it...I know what I am doing..".
>
> Extra paths added in sys.path would be omited but I don't see it as a problem
>
>
>
> --
> Tarek Ziadé | http://ziade.org
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Cheers,

Noah


More information about the Distutils-SIG mailing list