buildouts upgrades in production
Hello We are happily using zc.buildout to deploy our Python softwares (mainly zope) in production. The offline mode is very helpfull to create a media with a prepared buildout to deploy the thing on a server that has no web access (this is most of the time the case i guess) Now I am working on the best ways to prepare upgrades for the production and I try to find a way to avoid recreating a big tarball again just for a few changes. My guess is that a snapshot of the download-cache and eggs directory is enough, so a diff can be made to "upgrade" a production server with a tarball of the two directories and the new .cfg But I was wondering if someone had similar experiences already, and maybe a simple, better way ? Thanks Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tarek Ziadé wrote:
Hello
We are happily using zc.buildout to deploy our Python softwares (mainly zope) in production.
The offline mode is very helpfull to create a media with a prepared buildout to deploy the thing on a server that has no web access (this is most of the time the case i guess)
Now I am working on the best ways to prepare upgrades for the production and I try to find a way to avoid recreating a big tarball again just for a few changes.
My guess is that a snapshot of the download-cache and eggs directory is enough, so a diff can be made to "upgrade" a production server with a tarball of the two directories and the new .cfg
But I was wondering if someone had similar experiences already, and maybe a simple, better way ?
Thanks
I would avoid "upgrade in place" scenarios for production use: roll an entirely new deployment into a separate directory, use repozo to dump and reload the ZODB, bring up the new version, test, then switch the port numbers. Just my USD 0.02, Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHYBZQ+gerLs4ltQ4RAtWXAKDLmOBq5ZcDFnVbMoknbAZW+cIXTgCgvZlo PFELpT7lPY87E0En4WGCOHI= =5FER -----END PGP SIGNATURE-----
Tres Seaver wrote:
I would avoid "upgrade in place" scenarios for production use: roll an entirely new deployment into a separate directory, use repozo to dump and reload the ZODB, bring up the new version, test, then switch the port numbers.
What I forgot to mention is that the .fs files where backed up and set back after update, but I didn't thought about using repozo, thanks ! For the upgrade in place, i guess that is just a matter of backing it up before rolling the update, because in most case our updates are done like this: 1/ an apache directive switches users to a admisitrative page outside Zope 2/ the zope directory is backed up in some place 3/ it is updated then tested 4/ the apache is switched back -- View this message in context: http://www.nabble.com/buildouts-upgrades-in-production-tp14296760p14312704.h... Sent from the Python - distutils-sig mailing list archive at Nabble.com.
On Dec 11, 2007, at 12:11 PM, Tarek Ziadé wrote:
We are happily using zc.buildout to deploy our Python softwares (mainly zope) in production.
Cool.
The offline mode is very helpfull to create a media with a prepared buildout to deploy the thing on a server that has no web access (this is most of the time the case i guess)
Are you aware of zc.sourcerelease? http://pypi.python.org/pypi/zc.sourcerelease This automates creating buildout-based self-contained source releases that are installed without network access. These source releases can also be used as the basis for system packaging. We use them with a trivial spec file to create RPMs.
Now I am working on the best ways to prepare upgrades for the production and I try to find a way to avoid recreating a big tarball again just for a few changes.
IMO, the benefit doesn't justify the extra complexity. IMO, deployments should involve as little thought and thus risk as possible. Our RPMs typically install in a few seconds, which is fast enough for me. :)
My guess is that a snapshot of the download-cache and eggs directory is enough, so a diff can be made to "upgrade" a production server with a tarball of the two directories and the new .cfg
But I was wondering if someone had similar experiences already, and maybe a simple, better way ?
There are ways to use a download cache to speed creation of a source release. If you're interested, I could expand on this. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
Are you aware of zc.sourcerelease?
No ! thanks for mentioning it. We did a similar tool that creates tarballs over buildouts (http://pypi.python.org/pypi/iw.releaser) it also works at eggs level to release them, then control that there's no more "develop" section in the buildout, builds it and create a tarball with offline options. (and does svn tags) -> I need to drop the tarball part in favor of zc.sourcerelease I guess Jim Fulton wrote:
Now I am working on the best ways to prepare upgrades for the production and I try to find a way to avoid recreating a big tarball again just for a few changes.
IMO, the benefit doesn't justify the extra complexity. IMO, deployments should involve as little thought and thus risk as possible. Our RPMs typically install in a few seconds, which is fast enough for me. :)
I agree, but in some cases, my fellows that does upgrades from SSH or Terminal Server are angry because they have to upload a big tarball that often weight > 100 megas just for two changes in some python files. They also argue that sometimes it can be a pain with some firewalls rules. So my idea was to try to make their life easier in that case. The current work consists of creating buildout releases in svn branches after the buildout is created, with a small '.snapshot' file in the folder. It's a shorcut to avoid a complex scanning of .installed.cfg for instance. It lists all folders and files (non recursive) that where collected in the eggs/ folder and the download-cache (and its dist) folders in a flat list. On the next release, the tool knows how to build a small tarball by comparing what is has collected with the .snapshot of the previous release. (work in progress) Jim Fulton wrote:
My guess is that a snapshot of the download-cache and eggs directory is enough, so a diff can be made to "upgrade" a production server with a tarball of the two directories and the new .cfg
But I was wondering if someone had similar experiences already, and maybe a simple, better way ?
There are ways to use a download cache to speed creation of a source release. If you're interested, I could expand on this.
I'd love to learn more, please :) ++ Tarek -- View this message in context: http://www.nabble.com/buildouts-upgrades-in-production-tp14296760p14312913.h... Sent from the Python - distutils-sig mailing list archive at Nabble.com.
On Dec 13, 2007, at 4:47 AM, Tarek Ziadé wrote:
Jim Fulton wrote:
Are you aware of zc.sourcerelease?
No ! thanks for mentioning it. We did a similar tool that creates tarballs over buildouts (http://pypi.python.org/pypi/iw.releaser)
it also works at eggs level to release them, then control that there's no more "develop" section in the buildout, builds it and create a tarball with offline options. (and does svn tags)
-> I need to drop the tarball part in favor of zc.sourcerelease I guess
I doubt it. It looks like iw.releaser and zc.sourcerelease are aimed at different, if related, problems. AFAICT, iw.releaser automates some aspects of releasing individual Python distributions. zc.sourcerelease automated releasing course-grained applications, rather individual Python distributions. It assembles necessary Python distributions (and-configure-make-make-install-style distributions) into a single distribution with a Python install script that allows the application to be built and installed from a single download.
Jim Fulton wrote:
Now I am working on the best ways to prepare upgrades for the production and I try to find a way to avoid recreating a big tarball again just for a few changes.
IMO, the benefit doesn't justify the extra complexity. IMO, deployments should involve as little thought and thus risk as possible. Our RPMs typically install in a few seconds, which is fast enough for me. :)
I agree, but in some cases, my fellows that does upgrades from SSH or Terminal Server are angry because they have to upload a big tarball that often weight > 100 megas just for two changes in some python files. They also argue that sometimes it can be a pain with some firewalls rules.
/me shrugs. ...
Jim Fulton wrote:
My guess is that a snapshot of the download-cache and eggs directory is enough, so a diff can be made to "upgrade" a production server with a tarball of the two directories and the new .cfg
But I was wondering if someone had similar experiences already, and maybe a simple, better way ?
There are ways to use a download cache to speed creation of a source release. If you're interested, I could expand on this.
I'd love to learn more, please :)
I'd like to automate this in the future, but, for now, the following trick works if you know that all of the distributions you need are already in your download cache: buildout-source-release \ svn+ssh://some_repo_url release.cfg \ buildout:index=/home/jim/.buildout/download-cache/dist \ buildout:find-links= \ buildout:extensions= The source-release script takes a URL, usually a subversion URL, for the source to be released and the name of a release configuration file. You can also add additional buildout command-line options. Here, I've specified my download cache as an index. Obviously, you'd need to use a different path. I've also set the find-links and extensions options to empty strings so they aren't used. If you don't use the options in your configuration file, then you don't need to use this and the above would be simplified to: buildout-source-release \ svn+ssh://some_repo_url release.cfg \ buildout:index=/home/jim/.buildout/download-cache/dist With this, the source-release script doesn't have to spend time downloading anything. It does still need to build everything to decide what has to be included. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
-> I need to drop the tarball part in favor of zc.sourcerelease I guess
I doubt it. It looks like iw.releaser and zc.sourcerelease are aimed at different, if related, problems. AFAICT, iw.releaser automates some aspects of releasing individual Python distributions.
iw.releaser consists of 3 parts (the doc is outdated) : - an entry point for setuptools to add a release command for eggs. This helps developer release one package. but it is just a tool to help releasing. - a project_release script, that create a svn branch for a buildout - a project_deploy that runs bootstrap, bin/buildout then creates a tarball this is the part that does like zc.sourcerelease (see https://ingeniweb.svn.sourceforge.net/svnroot/ingeniweb/iw.releaser/trunk/iw...) afaik Jim Fulton wrote:
I'd like to automate this in the future, but, for now, the following trick works if you know that all of the distributions you need are already in your download cache:
buildout-source-release \ svn+ssh://some_repo_url release.cfg \ buildout:index=/home/jim/.buildout/download-cache/dist \ buildout:find-links= \ buildout:extensions=
The source-release script takes a URL, usually a subversion URL, for the source to be released and the name of a release configuration file. You can also add additional buildout command-line options. Here, I've specified my download cache as an index. Obviously, you'd need to use a different path. I've also set the find-links and extensions options to empty strings so they aren't used. If you don't use the options in your configuration file, then you don't need to use this and the above would be simplified to:
buildout-source-release \ svn+ssh://some_repo_url release.cfg \ buildout:index=/home/jim/.buildout/download-cache/dist
With this, the source-release script doesn't have to spend time downloading anything. It does still need to build everything to decide what has to be included.
this is neat ! Tarek -- View this message in context: http://www.nabble.com/buildouts-upgrades-in-production-tp14296760p14320894.h... Sent from the Python - distutils-sig mailing list archive at Nabble.com.
participants (3)
-
Jim Fulton -
Tarek Ziadé -
Tres Seaver