zc.buildout usage in the real world
Hi,
I'm currently working on a project based on django, together with a dozen of external libraries. Often I am in one of these scenarios:
- I discover bugs in an external libraries, and i fix them, but the upstream author is busy and I need to deploy the fix ASAP. So I produce a patch. - I have to install a particular revision of an external library from git/svn/hg
The approach I've used so far has been setting up a private pypi-compatible repository, and recreate packages which fit my needs. But it really feels like killing an ant with a sledgehammer. I don't understand if there is a better way to:
- Patch packages before installing - Fetch packages via git/svn/hg or via direct http link
I'm currently using the *great* minitage.recipe.scripts recipe to do all of this, and infact i can do something like: ---------------------------- [complex-deps] interpreter = python recipe = minitage.recipe.scripts urls = http://media.djangoproject.com/releases/1.1.1/Django-1.1.1.tar.gz http://github.com/robhudson/django-debug-toolbar/commit/f36b72655349cf3b8e91...
Django-1.1.1-patches = ${buildout:directory}/patches/django-admin-fixes.diff debug_toolbar-patches = ${buildout:directory}/patches/debug_toolbar_sqlpanel.diff -----------------------------
Which is very very handy, it handles all my use case in a compact and nice way. The minitage.recipe.scripts is nice but is not widely used, so I'm a bit worried about its future and possible breakages. You know - when you work on a large project you have to think hard about your dependencies. It's like getting married.
So the question is: Are there alternative solutions? Am I the only one with such needs?
Thank you
Alex Rades wrote:
The approach I've used so far has been setting up a private pypi-compatible repository, and recreate packages which fit my needs. But it really feels like killing an ant with a sledgehammer. I don't understand if there is a better way to:
- Patch packages before installing
- Fetch packages via git/svn/hg or via direct http link
What I do is check out the offending package, tweak the version in their setup.py to include the identifier of the revision I checked out, then do "python setup.py sdist" and put that sdist into my "egg server".
If I needed to apply patches that the packager won't accept (hasn't happened yet!), I'd do an export of the upstream package into my source control system, apply the patches and then do the process I've already described.
My "egg server" is just an authentication-required folder on an https web server, I use lovely.buildouthttp and a find-links to make this work.
This is actually even quicker and simpler than it looks.
[complex-deps] interpreter = python recipe = minitage.recipe.scripts urls = http://media.djangoproject.com/releases/1.1.1/Django-1.1.1.tar.gz http://github.com/robhudson/django-debug-toolbar/commit/f36b72655349cf3b8e91...
Django-1.1.1-patches = ${buildout:directory}/patches/django-admin-fixes.diff debug_toolbar-patches = ${buildout:directory}/patches/debug_toolbar_sqlpanel.diff
This kind of patching makes me nervous...
Which is very very handy, it handles all my use case in a compact and nice way. The minitage.recipe.scripts is nice but is not widely used,
...but then minitage as a whole makes me nervous, I just don't trust it, but I'm afraid I couldn't tell you why. Gut feeling...
Chris
Chris Withers a écrit :
Alex Rades wrote:
This is actually even quicker and simpler than it looks.
[complex-deps] interpreter = python recipe = minitage.recipe.scripts urls = http://media.djangoproject.com/releases/1.1.1/Django-1.1.1.tar.gz http://github.com/robhudson/django-debug-toolbar/commit/f36b72655349cf3b8e91... Django-1.1.1-patches = ${buildout:directory}/patches/django-admin-fixes.diff debug_toolbar-patches = ${buildout:directory}/patches/debug_toolbar_sqlpanel.diff
This kind of patching makes me nervous...
why ? patches are a common way to 'patch' code.
Which is very very handy, it handles all my use case in a compact and nice way. The minitage.recipe.scripts is nice but is not widely used,
...but then minitage as a whole makes me nervous, I just don't trust it, but I'm afraid I couldn't tell you why. Gut feeling...
Do not spread your personal fud, Chris. Moreover without arguments.
Chris
kiorky wrote:
${buildout:directory}/patches/debug_toolbar_sqlpanel.diff
This kind of patching makes me nervous...
why ? patches are a common way to 'patch' code.
Maybe 10 years ago. With a DVCS, or even a non-d VCS for that matter, why would you need to apply patches like this, which would need to be re-done with every release of the software you're patching, when you can let the (D)VCS do the hard work and cut your own sdist that has any patching done already the way you want it?
...but then minitage as a whole makes me nervous, I just don't trust it, but I'm afraid I couldn't tell you why. Gut feeling...
Do not spread your personal fud, Chris. Moreover without arguments.
*shrugs* Everyone's entitled to their own opinion...
Chris
Chris Withers a écrit :
kiorky wrote:
${buildout:directory}/patches/debug_toolbar_sqlpanel.diff
This kind of patching makes me nervous...
why ? patches are a common way to 'patch' code.
Maybe 10 years ago. With a DVCS, or even a non-d VCS for that matter, why would you need to apply patches like this, which would need to be re-done with every release of the software you're patching, when you can let the (D)VCS do the hard work and cut your own sdist that has any patching done already the way you want it?
Just let everybody do what he want like he want. I prefer to use forks, but sometime, it is just too much for a single patch. More over if the patch is already distributed as-is an i just need to use it without modifications.
[part] Foo-patch=/patch
is quickier that
SCM clone / sed "s/version/specialversion/g" -i setup.py patch < patch SCM ci -m azer SCM push /somewhere python setup.py sdist upload /somewhere
verify the url make it indexed somehow
[part] eggs=myegg
And making special dists everywhere make deployments tedious and add to you the job to keep your house clean and to know what is where. With a lot of packages, it is not that simple as integrating a single patch applied to a buildout which will generate under the hood an egg with a special name which will be never used even if you drop it in a shared cache unless you use the same patch in another project (minitage.recipe generates an egg with a special version, when you decide to install it with a patch, that s allow to use both the official egg and patched eggs in the same sharedcache).
...but then minitage as a whole makes me nervous, I just don't trust it, but I'm afraid I couldn't tell you why. Gut feeling...
Do not spread your personal fud, Chris. Moreover without arguments.
*shrugs* Everyone's entitled to their own opinion...
The fact that you are not knowing or understanding what minitage is or what it does or allow is not a reason to say something may not be a good choice to think on.
More over minitage "are": - paster templates (minitage.paste.*) - collection of recipes which also integrate system dependencies (making CFLAGS and LDFLAGS pointing to your dependencies, for example.) (minitage.recipe.*). - a meta-package-manager with goal to isolate dependencies and projects from the system and install them in a well know layout (minitage.core).
And there is http://minitage.org + Pypi pages as a documentation.
Chris
mr.developer is a good recipe that could help with this, check it out:
http://pypi.python.org/pypi/mr.developer
The buildout below will auto-checkout django-debug-toolbar from your fork when buildout is run (replace 'myapp' with your apps name of course, and have django-debug-toolbar in the install_requires field):
[buildout] parts = app develop = . extensions = mr.developer auto-checkout = django-debug-toolbar
[sources] django-debug-toolbar = git git://github.com/robhudson/django-debug-toolbar.git
[app] recipe = zc.recipe.egg eggs = myapp
Alex Rades a écrit :
Hi,
Which is very very handy, it handles all my use case in a compact and nice way. The minitage.recipe.scripts is nice but is not widely used,
But supported by a company which will do not switch for another recipes. We deploy on various systems, that's why we created minitage both as sysadmins and developers efforts.
so I'm a bit worried about its future and possible breakages.
We have a tracker, mails & irc and we don't like to have spending bugs for a long time ;)
You know
- when you work on a large project you have to think hard about your
dependencies. It's like getting married.
If minitage is dead, all our current deployments (lot of projects) there will be in bad shape. Just a word to say that you ll have years to see it dead.
So the question is: Are there alternative solutions? Am I the only one with such needs?
This is not only the way to do, and i told you the alternatives like doing with mr.developer. That depend on your feelings and habits and use cases.
I like very much both my recipes and mr.developer, When i need to apply a small change, mostly for upstream approval, i go for minitage.recipe.scripts. Or for a single patch without hacking, (ZODDB + RelStorage, for example), i go also for minitage.recipe.scripts. But for more deeper hacking, i'd go for mr.developer.*
After that, there are much more things in minitage.recipe.* that zc.equivalent, that i can't do without forking them and that's that i ve done that, some years ago. Now their code is too much different, and it was not a primary goal too, to merge back. Specially for minitage integration, even if you are not tied with minitage by using only the recipes, i don't think the code integrating it belongs to any zc.recipe.*.
Thank you _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
participants (4)
-
Alex Rades -
Chris Withers -
Kevin Teague -
kiorky