[Distutils] Performance implications of having large numbers of eggs?

Rick Ratzel rlratzel at enthought.com
Thu Jun 28 18:05:03 CEST 2007


>    Date: Thu, 28 Jun 2007 10:47:43 -0500
>    From: Dave Peterson <dpeterson at enthought.com>
> 
>    Has anyone done any investigation into the performance implications of 
>    having large numbers of eggs installed?  Is there any sort of 
>    performance hit? 
> 
> 
>    It seems to me that having a really large path might slow down imports a 
>    bit, though I suspect this is in C code so probably not a significant 
>    problem.  It also seems like there might be some startup penalties due 
>    to the overhead of setting up the path when using eggs, but this is a 
>    one-time cost during python startup, so probably not too bad either.

   Another option to avoid a startup penalty is to have all eggs installed with
-m (not in the easy-install.pth file, or "deactivated") and have code require()
the specific dependency.  This has the obvious disadvantage of having to change
a fair amount of code.  But, the advantages are only adding the eggs that are
needed to the path when they're needed (instead of every egg in every PYTHONPATH
dir), and your code will be sure that it's using the version that it's
compatible with.

   I should mention that I don't have any metrics on the startup penalty, so a
change like this may not be worth it if you're only trying to improve that.

> 
>    I'm asking because we're in the process to switching our open-source 
>    Enthought Tool Suite library to a distribution of components via eggs 
>    and we're having some internal debate as to whether we need to minimize 
>    the number of eggs or not.   It definitely seems nice to have smaller 
>    subsets of functionality -- from the point of being able to make things 
>    stable,  managing their APIs, managing cross-component dependencies, and 
>    from the user update size viewpoint.  But are we paying a performance 
>    penalty for going too small in scope with our eggs?
> 
> 
>    -- Dave
> 

-- 
Rick Ratzel - Enthought, Inc.
515 Congress Avenue, Suite 2100 - Austin, Texas 78701
512-536-1057 x229 - Fax: 512-536-1059
http://www.enthought.com



More information about the Distutils-SIG mailing list