[python-uk] CI and deployment (was Re: Suggestions / best practices for deployment)

[John Lee]

Speaking of CI and deployment -- Does anybody know of an open source 
implementation of the idea described in this blog post?  Maybe somebody 
has one and could open-source it?

http://googletesting.blogspot.co.uk/2011/06/testing-at-speed-and-scale-of-google.html


We're not all google, but overkill nicely implemented is sometimes just 
convenient and powerful.

In fact, contrary to what you might expect, I think some environments 
where development is smaller scale need this more than some "big product" 
development does: for example, companies that might do web app development 
for multiple clients, where you have a fair amount of code you reuse and 
multiple pieces of work going on at the same time.  In that situation -- 
unlike a big monolithic product -- you might want to keep some flexibility 
about which versions of code projects work together so that old apps can 
easily take advantage of new code when needed.  This seems like a good way 
to test that your dependency declarations correspond to reality, and of 
course to detect that a change to code project A has broken code project B 
earlier rather than later (even though production is pinned to old 
releases, you probably want to find out about the breakage early because 
it might get picked up later on in production).

VMs and deployment remind me of this because I think I recall a certain 
ex-colleague, as part of a scheme like this, wanting to build VMs to check 
whether system build inputs have changed.  Something like that...

Also, one "overkill" use I have in mind involves testing against fakes, 
when your application depends on some "heavy" systems that are slow and 
flaky to build and run -- probably built as a VM or a set of VMS.  The 
tests for the heavy system would depend on both the real system and the 
fake, and tests of applications would depend on the fake.  When the real 
system changes, the VMs get rebuilt and both the fake and the real thing 
automatically get retested.  If any of those tests fail and somebody fixes 
the fake to match reality, the application tests get automatically re-run 
against the fixed fake.


John

On Wed, 15 May 2013, Stestagg wrote:

> To add a slightly different angle to this, whatever deployment solution you
> use, make sure it is fully automated, and then hook it into you CI system.
>
> Deployment, disaster recovery etc. are made so much simpler if you're
> thinking about it from the start rather than just before release.  One way
> to enforce this is to start running the process regularly.
>
> My personal preference for this is to use snapshotted virtualbox VMs to
> deploy the service/app to a clean OS install, and the run the tests, after
> every commit (or on a schedule).  This way, you always know what
> dependencies you need, and you're forced to make things as seamless as
> possible.
>
> The same goes for testing upgrades.  Set up a VM with version X and set up
> a job to:  Upgrade > Test > Check > Rollback regularly.
>
> Thanks
>
> Steve
>
>
>
>
> On Wed, May 15, 2013 at 12:19 PM, Muhammad Rahman <
> muhammad.rahman at tangentlabs.co.uk> wrote:
>
>>  Hi Harry,
>>
>> I would use salt(http://docs.saltstack.com/index.html) for provisioning
>> and keep the python specific package to fabric to deploy using pip.
>>
>> Even though salt is able to deploy pip requirements but I think this
>> separation is important.
>>
>> Mustafiz.
>>
>>
>> On 15/05/2013 11:34, George Hickman wrote:
>>
>>  Hi Harry,
>>
>>  I use two methods for deployment - Heroku and Ubuntu VPSs.
>>
>>  Heroku is really simple to get going but gets expensive if you want to
>> run a serious production app. However it's probably what I'd recommend for
>> beginners since so much is done for you and they have first class python
>> support.
>>
>>  For VPS deploys I use Nginx, Daemontools (for the envdir package),
>> Supervisor, Gunicorn and Postgres (with Django). I usually deploy this
>> setup with a mixture of Salt to build the VPS and Fabric to deploy the code.
>>
>>  George
>>
>>
>> On Wed, May 15, 2013 at 10:57 AM, Harry Percival <harry.percival at gmail.com
>>> wrote:
>>
>>>  Dear UK Python chums,
>>>
>>>  some of you probably know I'm writing a book about TDD for O'Reilly.
>>> I'm looking for some help with the (first) chapter on deployment.
>>>
>>> http://www.obeythetestinggoat.com/what-to-say-about-deployment.html
>>>
>>>  What do you use for deployment?  Do you have any kind of automated
>>> scripts? How do you manage virtualenvs, the database, apache/uwsgi
>>> config... What do you think might work as a sort of "best practice lite"
>>> for a simple site for beginners?  (django, sqlite database, static files)
>>>
>>> --
>>> ------------------------------
>>> Harry J.W. Percival
>>> ------------------------------
>>> Twitter: @hjwp
>>> Mobile:  +44 (0) 78877 02511 <%2B44%20%280%29%2078877%2002511>
>>> Skype:         harry.percival
>>>
>>> _______________________________________________
>>> python-uk mailing list
>>> python-uk at python.org
>>> http://mail.python.org/mailman/listinfo/python-uk
>>>
>>>
>>
>>
>> _______________________________________________
>> python-uk mailing listpython-uk at python.orghttp://mail.python.org/mailman/listinfo/python-uk
>>
>>
>> --
>>
>> Muhammad Rahman
>> Developer
>>
>> Tangent Labs
>> 84 - 86 Great Portland Street
>> London, W1W 7NR
>>
>> T:  +44 (0)207 462 6150 (Office)
>> F:  +44 (0)207 462 6111
>> W:  www.tangentlabs.co.uk
>>
>>
>> _______________________________________________
>> python-uk mailing list
>> python-uk at python.org
>> http://mail.python.org/mailman/listinfo/python-uk
>>
>>
>