[Distutils] buildouts upgrades in production

[Jim Fulton]

On Dec 13, 2007, at 4:47 AM, Tarek Ziadé wrote:

>
>
> Jim Fulton wrote:
>>
>> Are you aware of zc.sourcerelease?
>>
>>   http://pypi.python.org/pypi/zc.sourcerelease
>>
>
> No ! thanks for mentioning it. We did a similar tool that creates  
> tarballs
> over buildouts (http://pypi.python.org/pypi/iw.releaser)
>
> it also works at eggs level to release them, then control
> that there's no more "develop" section in the buildout, builds
> it and create a tarball with offline options. (and does svn tags)
>
> -> I need to drop the tarball part in favor of zc.sourcerelease I  
> guess

I doubt it.  It looks like iw.releaser and zc.sourcerelease are aimed  
at different, if related, problems.  AFAICT, iw.releaser automates  
some aspects of releasing individual Python distributions.

zc.sourcerelease automated releasing course-grained applications,  
rather individual Python distributions. It assembles necessary Python  
distributions (and-configure-make-make-install-style distributions)  
into a single distribution with a Python install script that allows  
the application to be built and installed from a single download.

> Jim Fulton wrote:
>>
>>> Now I am working on the best ways to prepare upgrades for the
>>> production and I try to find a way to avoid recreating a big tarball
>>> again just for a few changes.
>>
>> IMO, the benefit doesn't justify the extra complexity.  IMO,
>> deployments should involve as little thought and thus risk as
>> possible.  Our RPMs typically install in a few seconds, which is fast
>> enough for me. :)
>>
>
> I agree, but in some cases, my fellows that does upgrades from
> SSH or Terminal Server are angry because they have to upload a big
> tarball that often weight > 100 megas just for two changes in some
> python files. They also argue that sometimes it can be a pain with
> some firewalls rules.

/me shrugs.

...

> Jim Fulton wrote:
>>
>>> My guess is that a snapshot of the download-cache and eggs directory
>>> is enough, so a diff can be made to "upgrade" a production server
>>> with a tarball of the two directories and the new .cfg
>>>
>>> But I was wondering if someone had similar experiences already, and
>>> maybe a simple, better way ?
>>
>>
>> There are ways to use a download cache to speed creation of a source
>> release.  If you're interested, I could expand on this.
>>
>
> I'd love to learn more, please :)


I'd like to automate this in the future, but, for now, the following  
trick works if you know that all of the distributions you need are  
already in your download cache:

   buildout-source-release \
     svn+ssh://some_repo_url  release.cfg \
     buildout:index=/home/jim/.buildout/download-cache/dist \
     buildout:find-links= \
     buildout:extensions=

The source-release script takes a URL, usually a subversion URL, for  
the source to be released and the name of a release configuration  
file.  You can also add additional buildout command-line options.   
Here, I've specified my download cache as an index.  Obviously, you'd  
need to use a different path. I've also set the find-links and  
extensions options to empty strings so they aren't used.  If you don't  
use the options in your configuration file, then you don't need to use  
this and the above would be simplified to:

   buildout-source-release \
     svn+ssh://some_repo_url  release.cfg \
     buildout:index=/home/jim/.buildout/download-cache/dist

With this, the source-release script doesn't have to spend time  
downloading anything. It does still need to build everything to decide  
what has to be included.

Jim

--
Jim Fulton
Zope Corporation