[python-uk] Suggestions / best practices for deployment

MikedePlume mike at mikedeplume.com
Mon May 20 16:40:28 CEST 2013


On Mon, 2013-05-20 at 14:51 +0100, Harry Percival wrote:
> Interesting points both.  I think it's insightful to think about the
> line beween provisioning and deployment... The idea that deployment
> shouldn't require root permissions is a good stake in the ground.
> 
> 
> In the book I've decided I'm going to take people through spinning up
> a server with Apache, and configuring two virtualhosts with mod_wsgi,
> one for staging and one for production.   We negotiate the gotchas,
> including setting up static files, using absolute paths and getting
> the permissions right for the (sqlite) database, etc.
> 
> 
> Then, deployment is a matter of updating the source code, updating any
> static files, and bouncing apache.
> 
> 
> Provisioning:
> - spinning up a server
> - installing apache
> - configuring apache
> 
> Deployment:
> -updating source code
> -updating static files
> -migrate database
> -restart apache

> For those, the root/non-root perms thing holds pretty well, apart from
> the "restart apache" bit...  And, up to a certain point, what I talk
> about under "provisioning" is stuff that might vary from platform to
> platform, but stuff under "deployment" should be more universal...

Interesting that the subject veers towards system design. I use FCGI
myself and the restart can be entirely non-root.  Provisioning then
includes adding the FCGI reference into the server config.

> For automating, I think they can be two separate scripts...  Although
> we might keep "spinning up a server" manual... 
> 
> What about testing?  Obviously both scripts will be tested indirectly
> via my functional tests (which I'm planning to run against staging,
> and they will show up any, eg, CSS problems)... But I've been pretty
> hot in the book on unit testing everything too.  At PA, our deploy
> scripts aren't unit tested, and as a result they're a bit of a mess.
> 
Yup. That's where I'm coming from. But maybe there is a reasonable
cut-off? If a deployment script does not have too many paths (which
sounds reasonable) then one test (the just-do-it one) may be sufficient?

> But, unit testing fabric scripts?  Won't it end up being a series of
> mocks?  I find those kinds of tests a bit pointless...  then again,
> maybe if we'd had mocky unit tests for all our deploy scripts since
> the beginning, perhaps they'd be better structured now...  Thoughts?

It _is_ possible to test deployment scripts (against a VM perhaps) to
check that directories are created and so on, but I've tried to make
sure that pretty much all of the files (config or other set-up, and
static files) are part of the deployment itself and will be copies of
whatever happened in the testing environment (perhaps, maybe, up to a
point ... ) so testing might just boil down to "Oh looky here - there's
a web site". But that might be a bit simplistic.

Testing _will_ be required if the deployment updates a database - or is
that part of migration?
> 
> Also, question for django people:  static files, as collected by
> "manage.py collectstatic":  in the repo, or not?
> 
> 
> 
> 
> On 20 May 2013 13:03, Daniel Pope <lord.mauve at gmail.com> wrote:
>         It doesn't need to be SSH-based, there are orchestration
>         systems that are RPC or pubsub based (eg. Jenkins,
>         MCollective).
>         
>         As per the line between provisioning and deployment I've
>         argued in the past that provisioning should be for state that
>         spans many minor releases of an application. I also believe
>         it's preferable to require that deployment cannot use root
>         permissions, as this is less likely to impact independent
>         services hosted on the same machine. It also allows better
>         auditability - it prevents deployments making changes that are
>         rightly the domain of the provisioning system.
>         
>         On 20 May 2013 11:17, "MikedePlume" <mike at mikedeplume.com>
>         wrote:
>                 On Sat, 2013-05-18 at 17:41 +0100, Andy Robinson
>                 wrote:
>                 > Daniel, who are you disagreeing with?   Everyone
>                 here agrees on
>                 > automation, I think.
>                 >
>                 > - Andy
>                 
>                 As a small user (one or two servers, one or two
>                 packages to deploy):
>                 
>                 I started by doing a lot of typing of commands, got
>                 bored with that,
>                 moved on to command line scripts and I now use a bunch
>                 of (python)
>                 scripts, some fabric based, to manage the server farm
>                 api and the ssh
>                 stuff. I think the point is that the underlying
>                 process is always going
>                 to involve ssh and some server farm api and it is the
>                 automation of all
>                 of that where the cornucopia of choice (?) is
>                 displayed.
>                 
>                 From a teaching perspective (that's what the book is
>                 about, I guess) I'd
>                 recognise the basics of ssh and so on and then build
>                 up from there. The
>                 basic principles behind testing a fabric deploy should
>                 be extendable to
>                 other environments, I'd have thought?
>                 
>                 I must say, I'd love to see something about testing
>                 bash scripts, and,
>                 perhaps, where to draw the line (if at all) between
>                 server provisioning
>                 and ci.
>                 
>                 Cheers
>                 
>                 Mike S.
>                 >
>                 >
>                 >
>                 > On 18 May 2013 12:24, Daniel Pope
>                 <lord.mauve at gmail.com> wrote:
>                 > > I thoroughly disagree with those who are saying
>                 that for small
>                 > > installations, it's less time-consuming to do
>                 things manually. A
>                 > > deployment/provisioning system gives you
>                 reproducibility - an executable
>                 > > record of how to re-create a server configuration
>                 or re-run a deployment
>                 > > without missing anything. The number of times I've
>                 spent 20 minutes
>                 > > wondering what I've missed all add up - that is
>                 why it automation breaks
>                 > > even so rapidly.
>                 > >
>                 > > Of course, you (or your employers) might have
>                 other reasons to want
>                 > > automation:
>                 > >
>                 > > - so that your deployment/provisioning is subject
>                 to automated testing,
>                 > > version control, code review, etc
>                 > > - so that you can build replicas of production
>                 servers for development,
>                 > > testing or disaster recovery
>                 > > - so that you can smash and rebuild a compromised
>                 or faulty machine without
>                 > > wasting time
>                 > > - so that you can deploy a dozen times a day and
>                 get features or fixes into
>                 > > the hands of your users
>                 > >
>                 > >
>                 > > On 17 May 2013 15:39, M.-A. Lemburg
>                 <mal at egenix.com> wrote:
>                 > >>
>                 > >> On 16.05.2013 17:46, Andy Robinson wrote:
>                 > >> > Speaking as a relatively obsolete dinosaur, I
>                 would suggest that if
>                 > >> > you are going to discuss specific deployment
>                 practices, you start with
>                 > >> > the most fundamental ones:  SSH, the unix shell
>                 and so on.
>                 > >> >
>                 > >> > We have had issues over the years with people
>                 coming in and
>                 > >> > introducing sexy new deployment tools, but
>                 ultimately they all just
>                 > >> > run unix commands.  Anyone managing a web
>                 application in the
>                 > >> > non-microsoft world is ultimately depending on
>                 this.  Some key skills
>                 > >> > (assuming a Linux/Mac/Unix-ish environment):
>                 > >> > - know about SSH keys and logging into remote
>                 machines
>                 > >> > - know the basics of at least one command line
>                 editor (e.g. vi)
>                 > >> > - basic shell knowledge:  environment
>                 variables, testing for existence
>                 > >> > of files and directories etc
>                 > >> > - how to interact with your database from the
>                 command line, if you use
>                 > >> > one (including dump and restore)
>                 > >> > - how your web server works: starting,
>                 stopping, configuration files,
>                 > >> > where log files live and how it talks to Python
>                 > >> >
>                 > >> > Fabric may be useful if you want to control
>                 half a dozen machines from
>                 > >> > your desktop, and it might add a lot of value
>                 if you want to control a
>                 > >> > hundred of them.  But to update one server, you
>                 deploy by logging into
>                 > >> > it and then running commands or short scripts.
>                 > >> >
>                 > >> > For example, we have a 'demo site' we rebuild
>                 pretty often that uses
>                 > >> > Django, MYSQL, Celery and a few other things.
>                  It runs on plain
>                 > >> > vanilla Ubuntu machines we build ourselves.
>                  The sequence is...
>                 > >> >
>                 > >> > 1. Log in via SSH
>                 > >> > 2. CD to correct directory
>                 > >> > 3. activate virtual environment
>                 > >> > 4. stop any celery worker processes
>                 > >> > 5. stop web server processes (* in our setup,
>                 we leave Apache running)
>                 > >> > 6. pull latest code from mercurial - both the
>                 app, and 3-4 libraries
>                 > >> > it depends on
>                 > >> > 7. run a management command to rebuild the
>                 database
>                 > >> > 8. run a smallish in-place test suite
>                 > >> > 9. restart celery workers
>                 > >> > 10.restart web server
>                 > >> > 11. log out
>                 > >> >
>                 > >> > All of this after the login and CD can be
>                 handled by a shell script on
>                 > >> > the path of the server, so you can just run a
>                 command called something
>                 > >> > like
>                 > >> >   ./update_server
>                 > >> >
>                 > >> > More realistically, we tend to end up with a
>                 management shell script
>                 > >> > called 'server' with a bunch of
>                 commands/arguments like 'stop / start
>                 > >> > / restart / update-code-in-staging /
>                 copy-live-data-to-staging /
>                 > >> > run-health-checks / swap-live-and-staging' and
>                 so on.  SSH can execute
>                 > >> > remote commands like this just fine with the
>                 right arguments, if
>                 > >> > actually logging in is too tedious.
>                 > >> >
>                 > >> > Production sites are complex and all
>                 different.  You might want to do
>                 > >> > instantaneous swaps from live to staging (and
>                 be able to back out fast
>                 > >> > if stuff goes wrong); to switch DNS so the
>                 world is looking at another
>                 > >> > server while you update one; you might have
>                 large databases to copy or
>                 > >> > migrate that need significant time; it may or
>                 may not be acceptable to
>                 > >> > lose sessions and have downtime; and so on.
>                 > >> >
>                 > >> >
>                 > >> > It takes less time to learn the fundamentals
>                 than you will spend
>                 > >> > debugging why your fancy new deployment tool
>                 stopped working after
>                 > >> > some Python dependency upgrade somewhere.   And
>                 it is less likely that
>                 > >> > your new hires will disagree if you stick with
>                 the lowest common
>                 > >> > denominator.
>                 > >>
>                 > >> Fully agreed.
>                 > >>
>                 > >> The new devops tools are nice when it comes to
>                 managing farms
>                 > >> of VMs, where each VM is setup in more or less
>                 the same way,
>                 > >> and you want to reduce repetitive tasks, but for
>                 a typical
>                 > >> setup with just a few VMs/servers it'll take you
>                 longer to
>                 > >> write and (most importantly) test your devops
>                 scripts than
>                 > >> it would to write a few scripts that you can run
>                 via SSH on
>                 > >> the servers.
>                 > >>
>                 > >> No matter how smart you make your devops scripts,
>                 Murphy's law
>                 > >> is inevitably going to strike and humans are so
>                 much better at
>                 > >> parsing weird intermittent error messages than
>                 machines ...
>                 > >> still, after all these years :-)
>                 > >>
>                 > >> --
>                 > >> Marc-Andre Lemburg
>                 > >> eGenix.com
>                 > >>
>                 > >> Professional Python Services directly from the
>                 Source  (#1, May 17 2013)
>                 > >> >>> Python Projects, Consulting and Support ...
>                 http://www.egenix.com/
>                 > >> >>> mxODBC.Zope/Plone.Database.Adapter ...
>                 http://zope.egenix.com/
>                 > >> >>> mxODBC, mxDateTime, mxTextTools ...
>                  http://python.egenix.com/
>                 > >>
>                 ________________________________________________________________________
>                 > >> 2013-05-07: Released mxODBC Zope DA 2.1.2 ...
>                 http://egenix.com/go46
>                 > >> 2013-05-06: Released mxODBC 3.2.3 ...
>                 http://egenix.com/go45
>                 > >>
>                 > >> ::::: Try our mxODBC.Connect Python Database
>                 Interface for free ! ::::::
>                 > >>
>                 > >>    eGenix.com Software, Skills and Services GmbH
>                  Pastor-Loeh-Str.48
>                 > >>     D-40764 Langenfeld, Germany. CEO Dipl.-Math.
>                 Marc-Andre Lemburg
>                 > >>            Registered at Amtsgericht Duesseldorf:
>                 HRB 46611
>                 > >>
>                  http://www.egenix.com/company/contact/
>                 > >> _______________________________________________
>                 > >> python-uk mailing list
>                 > >> python-uk at python.org
>                 > >> http://mail.python.org/mailman/listinfo/python-uk
>                 > >
>                 > >
>                 > >
>                 > > _______________________________________________
>                 > > python-uk mailing list
>                 > > python-uk at python.org
>                 > > http://mail.python.org/mailman/listinfo/python-uk
>                 > >
>                 >
>                 >
>                 >
>                 
>                 
>                 _______________________________________________
>                 python-uk mailing list
>                 python-uk at python.org
>                 http://mail.python.org/mailman/listinfo/python-uk
>         
>         _______________________________________________
>         python-uk mailing list
>         python-uk at python.org
>         http://mail.python.org/mailman/listinfo/python-uk
>         
> 
> 
> 
> -- 
> ------------------------------
> Harry J.W. Percival
> ------------------------------
> Twitter: @hjwp
> Mobile:  +44 (0) 78877 02511
> Skype:         harry.percival
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> http://mail.python.org/mailman/listinfo/python-uk




More information about the python-uk mailing list