[python-uk] Suggestions / best practices for deployment

Mon May 20 12:17:33 CEST 2013

On Sat, 2013-05-18 at 17:41 +0100, Andy Robinson wrote:
> Daniel, who are you disagreeing with?   Everyone here agrees on
> automation, I think.
> 
> - Andy

As a small user (one or two servers, one or two packages to deploy):

I started by doing a lot of typing of commands, got bored with that,
moved on to command line scripts and I now use a bunch of (python)
scripts, some fabric based, to manage the server farm api and the ssh
stuff. I think the point is that the underlying process is always going
to involve ssh and some server farm api and it is the automation of all
of that where the cornucopia of choice (?) is displayed. 

>From a teaching perspective (that's what the book is about, I guess) I'd
recognise the basics of ssh and so on and then build up from there. The
basic principles behind testing a fabric deploy should be extendable to
other environments, I'd have thought?

I must say, I'd love to see something about testing bash scripts, and,
perhaps, where to draw the line (if at all) between server provisioning
and ci.

Cheers

Mike S.
> 
> 
> 
> On 18 May 2013 12:24, Daniel Pope <lord.mauve at gmail.com> wrote:
> > I thoroughly disagree with those who are saying that for small
> > installations, it's less time-consuming to do things manually. A
> > deployment/provisioning system gives you reproducibility - an executable
> > record of how to re-create a server configuration or re-run a deployment
> > without missing anything. The number of times I've spent 20 minutes
> > wondering what I've missed all add up - that is why it automation breaks
> > even so rapidly.
> >
> > Of course, you (or your employers) might have other reasons to want
> > automation:
> >
> > - so that your deployment/provisioning is subject to automated testing,
> > version control, code review, etc
> > - so that you can build replicas of production servers for development,
> > testing or disaster recovery
> > - so that you can smash and rebuild a compromised or faulty machine without
> > wasting time
> > - so that you can deploy a dozen times a day and get features or fixes into
> > the hands of your users
> >
> >
> > On 17 May 2013 15:39, M.-A. Lemburg <mal at egenix.com> wrote:
> >>
> >> On 16.05.2013 17:46, Andy Robinson wrote:
> >> > Speaking as a relatively obsolete dinosaur, I would suggest that if
> >> > you are going to discuss specific deployment practices, you start with
> >> > the most fundamental ones:  SSH, the unix shell and so on.
> >> >
> >> > We have had issues over the years with people coming in and
> >> > introducing sexy new deployment tools, but ultimately they all just
> >> > run unix commands.  Anyone managing a web application in the
> >> > non-microsoft world is ultimately depending on this.  Some key skills
> >> > (assuming a Linux/Mac/Unix-ish environment):
> >> > - know about SSH keys and logging into remote machines
> >> > - know the basics of at least one command line editor (e.g. vi)
> >> > - basic shell knowledge:  environment variables, testing for existence
> >> > of files and directories etc
> >> > - how to interact with your database from the command line, if you use
> >> > one (including dump and restore)
> >> > - how your web server works: starting, stopping, configuration files,
> >> > where log files live and how it talks to Python
> >> >
> >> > Fabric may be useful if you want to control half a dozen machines from
> >> > your desktop, and it might add a lot of value if you want to control a
> >> > hundred of them.  But to update one server, you deploy by logging into
> >> > it and then running commands or short scripts.
> >> >
> >> > For example, we have a 'demo site' we rebuild pretty often that uses
> >> > Django, MYSQL, Celery and a few other things.  It runs on plain
> >> > vanilla Ubuntu machines we build ourselves.  The sequence is...
> >> >
> >> > 1. Log in via SSH
> >> > 2. CD to correct directory
> >> > 3. activate virtual environment
> >> > 4. stop any celery worker processes
> >> > 5. stop web server processes (* in our setup, we leave Apache running)
> >> > 6. pull latest code from mercurial - both the app, and 3-4 libraries
> >> > it depends on
> >> > 7. run a management command to rebuild the database
> >> > 8. run a smallish in-place test suite
> >> > 9. restart celery workers
> >> > 10.restart web server
> >> > 11. log out
> >> >
> >> > All of this after the login and CD can be handled by a shell script on
> >> > the path of the server, so you can just run a command called something
> >> > like
> >> >   ./update_server
> >> >
> >> > More realistically, we tend to end up with a management shell script
> >> > called 'server' with a bunch of commands/arguments like 'stop / start
> >> > / restart / update-code-in-staging / copy-live-data-to-staging /
> >> > run-health-checks / swap-live-and-staging' and so on.  SSH can execute
> >> > remote commands like this just fine with the right arguments, if
> >> > actually logging in is too tedious.
> >> >
> >> > Production sites are complex and all different.  You might want to do
> >> > instantaneous swaps from live to staging (and be able to back out fast
> >> > if stuff goes wrong); to switch DNS so the world is looking at another
> >> > server while you update one; you might have large databases to copy or
> >> > migrate that need significant time; it may or may not be acceptable to
> >> > lose sessions and have downtime; and so on.
> >> >
> >> >
> >> > It takes less time to learn the fundamentals than you will spend
> >> > debugging why your fancy new deployment tool stopped working after
> >> > some Python dependency upgrade somewhere.   And it is less likely that
> >> > your new hires will disagree if you stick with the lowest common
> >> > denominator.
> >>
> >> Fully agreed.
> >>
> >> The new devops tools are nice when it comes to managing farms
> >> of VMs, where each VM is setup in more or less the same way,
> >> and you want to reduce repetitive tasks, but for a typical
> >> setup with just a few VMs/servers it'll take you longer to
> >> write and (most importantly) test your devops scripts than
> >> it would to write a few scripts that you can run via SSH on
> >> the servers.
> >>
> >> No matter how smart you make your devops scripts, Murphy's law
> >> is inevitably going to strike and humans are so much better at
> >> parsing weird intermittent error messages than machines ...
> >> still, after all these years :-)
> >>
> >> --
> >> Marc-Andre Lemburg
> >> eGenix.com
> >>
> >> Professional Python Services directly from the Source  (#1, May 17 2013)
> >> >>> Python Projects, Consulting and Support ...   http://www.egenix.com/
> >> >>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
> >> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> >> ________________________________________________________________________
> >> 2013-05-07: Released mxODBC Zope DA 2.1.2 ...     http://egenix.com/go46
> >> 2013-05-06: Released mxODBC 3.2.3 ...             http://egenix.com/go45
> >>
> >> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
> >>
> >>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
> >>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
> >>            Registered at Amtsgericht Duesseldorf: HRB 46611
> >>                http://www.egenix.com/company/contact/
> >> _______________________________________________
> >> python-uk mailing list
> >> python-uk at python.org
> >> http://mail.python.org/mailman/listinfo/python-uk
> >
> >
> >
> > _______________________________________________
> > python-uk mailing list
> > python-uk at python.org
> > http://mail.python.org/mailman/listinfo/python-uk
> >
> 
> 
> 