[Distutils] buildout prepends eggs to sys.path, performance issues

[Jim Fulton]

On Tue, Apr 29, 2014 at 7:33 AM, Jan Van Hees <jakkevanhees at gmail.com> wrote:
> Hi list,
>
>
>
> A bit of context: we’re running buildout 2.0.1, building zope and plone
> applications, zope 2.13.10, combined with plone.recipe.zope2instance. This
> question only impacts the performance of the startup of an instance, the the
> performance of a running instance. But starting an instance is something
> developpers do quite often on a days development.
>
>
>
> While debugging some instance startup performance issues, I came across the
> following.
>
> The buildout Scripts prepends all the eggs to the system path, before the
> python path.
>
>
>
> In our setup this causes quite some delays, because imports from standard
> python modules, also try to find that module in every eggs directory, before
> it can find in in the default python location (because the eggs are
> prepended).

...

You make a good performance argument.

> As far as I’ve always understood, the default procedure working with paths
> should be to append, unless you have a good reason. The good reason in this
> case, that I see, could be that you want to prepend certain packages and
> that way make sure you use your version instead of what’s present in
> site-packages.

Yes, exactly.

> Now my expectation would be that that is a fairly limited set
> of packages that need to be prepended.

But buildout can't know what they are.

> If there are many, options like
> virtualenv exist to avoid taking site-packages at all (that’s what we do
> btw).

Buildout doesn't require virtualenv.

I've always prepended appliocation paths for predicability.

I'm not interesated in changing this behavior of buildout, at
least not as a default.

I'll give this some thought, but some possible ideas:

- Provide an option to install the eggs for a part in a single
  directory.

  collective.recipe.omlette does this, although it doesn't seem to be
  used for run-time package lookup afaict from the docs.  You might
  try using this.

- Provide an option to append, rather than prepend eggs.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton