[Distutils] Are there any plans to move to pip/wheels in buildout?

Jim Fulton jim at zope.com
Tue Dec 2 16:04:44 CET 2014

On Tue, Dec 2, 2014 at 7:53 AM, Reinout van Rees <reinout at vanrees.org> wrote:
> On 01-12-14 14:55, Piotr Dobrogost wrote:
>> Are there any plans to move from easy_install/eggs to pip/wheels in
>> buildout?
> I asked that too, once. I think Jim Fulton said that there was a mismatch
> between pip's and buildout's goal. If I remember correctly the problem is
> this:
> - Pip installs everything into (normally) a virtualenv. So your have ONE
> project with ONE single set of packages.
> - Buildout allows different combinations of packages per part. So your
> sphinx documentation buildout doesn't have to have your webserver packages
> in its sys.path.
> Jim, is that a fair summary? (Of course, this only looks at the python
> package part of what buildout does, not at the whole recipe mechanism).

Yes, but ...

> Something I don't understand here is that there *is* a single version list.

Right, although explicit requirements in a part can override the
versions list.  In practice, most projects use the same versions for

In the past, this has been an important requirement for us (ZC). We've
even had projects that used different versions of Python for different
parts, although ultimately, this "requirement" didn't justify the

Something to keep in mind (I know you know this) is that buildout is
used to assemble applications with many parts that may be separate
applications and may not all be written in Python.

> And so it feels like it doesn't really matter that much if you just lump all
> the packages together (like pip does).

In practice, for most projects this is true.

> I could imagine a buildout that just uses pip every time it has to install a
> python package. Feeding pip the version list, of course. And then puts the
> buildout.cfg mechanism on top of it.

I only use versions lists for applications. I almost never pin
versions for libraries.

Another important goal of buildout is that, given a buildout
configuration, two buildouts run at the same time get equivalent
results regardless of their ages and histories.

For example, I create a buildout on Monday and it gets version 42 of
package A. On a different machine, I checkout and update the buildout
to add package B. It gets version 9 of package B and version 43 of
package A. If I go back to the other machine, and rerun the buildout,
I'll get versions 43 and 9 for package A and B.  This was impossible
to achieve with easy_install, which is what drove me down to
setuptools.  I don't know how difficult this would be with pip. Given
that pip had largely the same goals as easy_install, I'm not
optimistic.  It's worth looking into.

There's also the fact that pip doesn't provide an API other than the
command line or the "command" interface which is essentially the same
things plus ... distutils.  I'm not opposed to trying to create a
version of buildout on top of pip+virtualenv, but I don't think that's
a very good architecture and I don't really expect it to turn out
well. If no standard API emerges, we may have no choice.  Maybe we
should bet on distlib, but with pip not using it...

> The alternative is, like Marius said in his reply, to use virtualenv + pip +
> makefiles or so.

That sounds like a terrible idea to me. On multiple levels.

> And you'd have to add something that replaces the recipes.

Well, people use chef and puppet and similar tools to configure docker
containers.  Then there are Java-based tools, like ant, maven, gradle,
sbt...  The recipe pattern is well established.

> For that, I really like buildout :-)

Me too. As long as I'm using Python, I'll maintain buildout if only
for my own use.  I think recipe-based automation systems make a lot of
sense, and I prefer to write recipes, like most other things, in


Jim Fulton

More information about the Distutils-SIG mailing list