On Fri, Feb 08, 2008 at 09:27:00AM -0500, Tres Seaver wrote:

Chris Withers wrote:

...

Hi All,

What are the alternatives to zc.buildout?

I'm wondering before I plough into zc.buildout because I'm guessing other big projects have run into the problem of pinning certain versions of eggs, building "instance homes" for projects, using a mixture of "public" eggs and those that are private and for a specific project, etc and otherwise working round limitations in setuptools...

The major competitor for mindshare is probably Ian Bicking's 'virtualenv': it makes using 'easy_install' *really* easy, because you are working in a sandbox.

And then there are people who combine zc.buildout with virtualenv. They solve slightly different problems: virtualenv isolates a sandbox from unknown and possibly broken stuff in your system-wide Python installation; zc.buildout automates downloading and building dependencies (which can be anything from pure Python packages to C libraries). Marius Gedminas -- 99 little bugs in the code, 99 bugs in the code, fix one bug, compile it again... 101 little bugs in the code....

Martin Aspeli wrote:

Tres Seaver wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Chris Withers wrote:

...

Hi All,

The standard installs for repoze.zope2, repoze.grok, repoze.plone, etc. all assume a virtualenv, and add scripts which turn the vanilla virtualenv into an "instance."

The problem I have with virtualenv - or rather, with raw setuptools based solutions - is that it's difficult to uninstall eggs. With buildout, once you remove the requirement/egg line and re-run buildout, the egg is no longer installed.

I usually edit site-packages/easy_install.pth and delete the line that points to the egg. And either delete the egg, or not.

That said, I like virtualenv and use it a lot. I just think it solves a slightly different use case from the "repeatable configuration management system" one that buildout tries to solve.

It does indeed handle uninstall which lots of other things don't. - C

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin Aspeli wrote:

Tres Seaver wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Chris Withers wrote:

...

Hi All,

What are the alternatives to zc.buildout?

I'm wondering before I plough into zc.buildout because I'm guessing other big projects have run into the problem of pinning certain versions of eggs, building "instance homes" for projects, using a mixture of "public" eggs and those that are private and for a specific project, etc and otherwise working round limitations in setuptools... The major competitor for mindshare is probably Ian Bicking's 'virtualenv': it makes using 'easy_install' *really* easy, because you are working in a sandbox.

The standard installs for repoze.zope2, repoze.grok, repoze.plone, etc. all assume a virtualenv, and add scripts which turn the vanilla virtualenv into an "instance."

The problem I have with virtualenv - or rather, with raw setuptools based solutions - is that it's difficult to uninstall eggs. With buildout, once you remove the requirement/egg line and re-run buildout, the egg is no longer installed.

My major beef with zc.buildout is perhaps actually a problem with the recipes nearly everybody uses: they blow away hand-edited stuff without warning. In particular, changes to things like the zope.conf file get zapped, because buildout (or the recipe) thinks that the file is its own personal property.

That said, I like virtualenv and use it a lot. I just think it solves a slightly different use case from the "repeatable configuration management system" one that buildout tries to solve.

My sense is that zc.buildout's focus on "production deployment" usecass is the source of a lot of my frustration with it as a developer: I expect to "inhabit" the environment it creates, which is completely irrelvant in a production rollout. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHrf/Q+gerLs4ltQ4RAuWmAJ9BkaMqg5xnantzpinhSEc/IF1KSQCePtFo dA/0O/68QFhZ4KFUlflwW9Y= =PA70 -----END PGP SIGNATURE-----

Chris McDonough wrote:

Martin Aspeli wrote:

...

Chris McDonough wrote:

...

...

Tres Seaver wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Chris Withers wrote:

...

Hi All,

I usually edit site-packages/easy_install.pth and delete the line that points to

Martin Aspeli wrote: the egg. And either delete the egg, or not. That file's fairly cryptic

It's one path per line. Doesn't get too much simpler than that. .pth files have been around forever (almost literally, they have been around since I started using Python anyway, 7 years ago) and are documented pretty well in the Python docs:

http://docs.python.org/lib/module-site.html

...

and the chances of screwing something up there I think are higher than in the more user-oriented buildout.cfg.

True, at least if you trust the recipe authors. Although deleting a line isn't very hard.

Knowing which line to delete may be. ;)

...

Also, does deleting egg A in easy_install.pth cause any eggs that were installed as dependencies of A but are no longer needed to go as well?

No.

Right. Which means that there are lots of eggs there that you won't know whether you need to delete or not and some very bad things may happen if you delete the wrong one (or bad things may be happening already due to a dependency that got installed by accident and that you need to delete). Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book

On Feb 9, 2008, at 2:32 PM, Tres Seaver wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Martin Aspeli wrote:

...

Tres Seaver wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Chris Withers wrote:

...

Hi All,

What are the alternatives to zc.buildout?

I'm wondering before I plough into zc.buildout because I'm guessing other big projects have run into the problem of pinning certain versions of eggs, building "instance homes" for projects, using a mixture of "public" eggs and those that are private and for a specific project, etc and otherwise working round limitations in setuptools... The major competitor for mindshare is probably Ian Bicking's 'virtualenv': it makes using 'easy_install' *really* easy, because you are working in a sandbox.

The standard installs for repoze.zope2, repoze.grok, repoze.plone, etc. all assume a virtualenv, and add scripts which turn the vanilla virtualenv into an "instance."

The problem I have with virtualenv - or rather, with raw setuptools based solutions - is that it's difficult to uninstall eggs. With buildout, once you remove the requirement/egg line and re-run buildout, the egg is no longer installed.

My major beef with zc.buildout is perhaps actually a problem with the recipes nearly everybody uses: they blow away hand-edited stuff without warning. In particular, changes to things like the zope.conf file get zapped, because buildout (or the recipe) thinks that the file is its own personal property.

Which it is. Part of using buildout is understanding that anything built by buildout is controlled by it. I often make temporary hacks to created files, knowing that the changes will be thrown away on the next build. If you want to make a permanent change, you need to update the buildout configuration. There *can* be exceptions to this. We have some recipes that manage checkouts. These don't remove checkouts to avoid losing user data. Similarly, the zc.recipes.filestorage recipe doesn't remove data directories it creates on uninstall.

...

That said, I like virtualenv and use it a lot. I just think it solves a slightly different use case from the "repeatable configuration management system" one that buildout tries to solve.

My sense is that zc.buildout's focus on "production deployment" usecass is the source of a lot of my frustration with it as a developer: I expect to "inhabit" the environment it creates, which is completely irrelvant in a production rollout.

I like buildout for development be because I don't have to "inhabit" the created artifacts. I (can) have a single high-level configuration that lets me control a variety of subsystems in one place. I'm sure this is a matter of taste. Jim -- Jim Fulton Zope Corporation

On Feb 10, 2008, at 2:01 PM, Noah Gift wrote:

In reference to this rather long discussion. I would love to get an interview or case study on buildout in the scope of package management for Python for the book I am co-authoring on Python For *nix Systems Administration. If there is an official ambassador, I would love to talk to that person, to make sure I do it justice. That particular chapter will cover eggs, virtualenv, and buildout, so I pretty excited about it, although is the toughest chapter in the book so far.

Cool! Obviously, you can work with me on this. One thing to understand about buildout is that it is oriented toward distributing applications vs easy_install/setuptools with is aimed more at distributing libraries. At ZC, we use buildout to create development environments. We also use buildout to create self-contained source-releases and rpms for deploying out *applications* to out deployment environments in ways that facilitate *operation* by experienced Unix system administrators. Jim -- Jim Fulton Zope Corporation

Thanks guys. I have sent you both emails offline. On Feb 10, 2008, at 3:33 PM, zooko wrote:

On Feb 10, 2008, at 12:14 PM, Jim Fulton wrote:

...

On Feb 10, 2008, at 2:01 PM, Noah Gift wrote:

...

In reference to this rather long discussion. I would love to get an interview or case study on buildout in the scope of package management for Python for the book I am co-authoring on Python For *nix Systems Administration.

Cool!

Yeah!

...

One thing to understand about buildout is that it is oriented toward distributing applications vs easy_install/setuptools with is aimed more at distributing libraries.

For comparison, at http://allmydata.org, we are using eggs plus py2exe, py2app, etc. to build applications to deploy to end users, but we are using Debian packages to deploy applications to system administrators.

Our set of tools is working well enough, but there are clearly places where they aren't mature yet. For example we don't use an automated tool to produce .deb's from our setup.py or from our .egg's, but instead have a separately-maintained Debian build. Likewise py2exe and py2app don't naturally manage our own code (which is built by setuptools) and our dependencies (which are mostly but not all packaged as eggs) as smoothly as we would like, but they work.

Regards,

Zooko

Noah Gift / http://noahgift.com [Python+Grok+AJAX+Mashup]

Tres Seaver wrote:

My major beef with zc.buildout is perhaps actually a problem with the recipes nearly everybody uses: they blow away hand-edited stuff without warning. In particular, changes to things like the zope.conf file get zapped, because buildout (or the recipe) thinks that the file is its own personal property.

This was also something that drove me nuts. It's too bad this is still the case. We have a build tool ourselves, very similar in scope to zc.buildout, though I don't really have intentions at this time to make it a legitimate project for other people to use. But, for the record: http://www.openplans.org/projects/fassembler/ One of the core parts of it is filemaker, which does most of the interaction with the system: https://svn.openplans.org/svn/fassembler/trunk/fassembler/filemaker.py It goes to great length to notice changes, even if they aren't due to edits; I find it's nice to know what exactly is going on. It's also got a bit of support for detecting why things changed, by saving the template, and probably will grow support for detecting user changes so some things can be overwritten without warning. With these build things I don't really care who owns what, since that's mostly an abstract concept that the build user won't know and shouldn't really have to know. (filemaker was generally based on the code from paster create) I also pay a lot of attention to logging, as I hate noisy output and of course not enough output is also a problem. I can't remember how zc.buildout acts in that respect. Ian

zooko wrote:

On Feb 12, 2008, at 9:35 PM, Ian Bicking wrote:

...

I don't know that many people have used this (or know about it), but I added a command to buildutils (http://knowledgetap.com/hg/buildutils/) to do "python setup.py bundle", which takes a package and all its dependencies and puts them together, with a script that adds all the dependencies. It's like what zc.buildout does mechanically, but intended to be used more like py2exe. I think it'd fit the model of managing Python commands and scripts pretty well.

That's interesting. What we do is have our setup.py look in a directory named "misc/dependencies/" and add any tarballs it finds therein to the dependency_links:

http://allmydata.org/trac/tahoe/browser/setup.py#L93

That's similar to bundle; bundle just calls easy_install to install all the dependencies into a particular directory, then adds "site.addsitedir(dependency_dir)" to the top of any scripts. But as a result you don't have to call setup.py on the host. Figuring out the location of dependency_dir is less than perfect. Both relative and absolute filenames have their problems. Ian