[Web-SIG] Preferred set of WSGI servers

Shannon -jj Behrens jjinux at gmail.com
Fri Apr 29 20:30:57 CEST 2005


The reloading problem is a tough one, and Aquarium went through a lot
of iterations before it came to a "good solution".  Now days, I have a
property which tells what types of modules I'm interested in
reloading.  Then, if any module is stale, I reload all of those
modules.  This takes care of cases of the parent being stale,
requiring the child to be reloaded.  Modules that aren't subject to
reloading aren't reloaded because they probably have important state. 
I have very few of those, and they require a restart if they change. 
Fortunately, that is rare.
I'm more than happy to share code.  It's rather small.

Best Regards,
-jj

On 4/28/05, Rene Dudfield <renesd at gmail.com> wrote:
> On 4/29/05, Ian Bicking <ianb at colorstudy.com> wrote:
> > Here's the full set of WSGI-enabled servers I'd like to see (and in
> > some way encorporate into Paste, of course):
> >
> > * CGI gateway (done: not sure of canonical location)
> > * Simple threaded HTTP server (done: wsgiutils)
> > * mod_python (?)
> > * ISAPI (done: http://isapi-wsgi.python-hosting.com/)
> > * Zope product (not done)
> >
> > Those are all rather specific to certain environments.  Then I'd like
> > to see two other servers: multi-process and threaded.  Obviously the
> > two imply very different architectures.  Well, maybe not, but it seems
> > like it to me.
> >
> > Right now we have several servers with different purposes: SCGI (one
> > for forking and threaded each), FastCGI, HTTP, etc.  I don't really
> > like that, I feel protocol support should be a separate concern.  I
> > would like for the two "preferred" deployment servers to support this
> > set of protocols:
> >
> > * HTTP
> > * HTTPS
> > * FastCGI
> > * SCGI
> >
> > Support for other protocols like FTP doesn't hurt, of course, but
> > their exact relation to WSGI is unclear.  Right now servers seem to be
> > FastCGI-ish (which includes SCGI) *or* HTTP.  I don't think this
> > protocol choice should restrict the server choices.  I think
> > FastCGI-ish setups can be easier to configure an integrate; other
> > people may disagree, but I don't see any technical reason not to
> > include all these protocols, so why not make everyone happy?  (Note
> > that I'm personally not concerned about the fancier features of
> > FastCGI specifically, like starting processes and doing pooling as
> > part of the protocol; I'm only interested in the delegation of
> > requests to different processes.)
> >
> > Twisted is an obvious contender for the threaded server.  However, to
> > my knowledge it does not support FastCGI or SCGI (as a client).  Also,
> > WSGI support is only available for twisted.web2, which doesn't have a
> > release yet.  Lastly, the installation isn't as easy as it could be --
> > it requires Zope interfaces, which includes C extensions.  So, there's
> > some work to be done there.  To their credit, with the recent split of
> > Twisted into smaller packages they are moving in the right direction.
> > Medusa is a simpler alternative; it's not as actively maintained, but
> > there aren't as many packaging or dependency issues.
> >
> 
> Twisted reverse proxied by apache2 is my server of choice at the
> moment for wsgi.  As it can nicely run multiple wsgi apps.  I use
> apache2 instead of lighttpd now, because it supports compression, and
> caching of dynamic content.
> 
> With twisted I can also do other things in the background quite
> easily.  Like making http requests, sending/serving email, etc etc.
> 
> There are still a few bugs in twisted.web2 as it is not released.
> However there are a few active contributors to web2 who are quick to
> help out with any problems.
> 
> Here is an example of using twisted to serve two wsgi apps with web2.
> http://www.madecollective.com/~rene/two_wsgi.py
> 
> 
> > For the forking server, I really have no idea.  There's several out
> > there, but I don't know about all the features that are desirable or
> > implemented.  Several of the servers don't speak HTTP.  I can't make
> > any specific criticisms, because I don't even know what to look at.
> >
> 
> http://www.saddi.com/software/py-lib/  with his flup package contains
> experimental forking/preforking wsgi servers.  Have you seen these?
> 
> The above also contains an XMLRPC wsgi server.  So you can easily
> serve xmlrpc on with wsgi too :)
> 
> 
> > Lingering in the background of these server discussions are issues
> > like user isolation and module reloading (and/or server restarting).
> >
> > Thoughts, reactions?
> >
> 
> Another nice thing about twisted is that it has server restarting in
> there.  It also has module reloading.  Allthough I don't use either.
> 
> I think a separate process which monitors things and sends the app
> restart, reload, signals is a good idea.  Then in development you can
> have the monitor send it a restart signal as soon as code/files
> change.
> 
> I use this bash script on debian linux to watch a directory and then
> do stuff if any of my files change.  It uses rsync, which is really
> fast.  It doubles as an automatic incremental backup service.  With
> putting files that have changed in separate time stamped directories.
> It can be easily modified to save the backups on another machine too.
> 
> watcher.sh
> 
> #!/bin/bash
> DIR_TO_WATCH=pretendpaper/pp
> BACKUP_DIR=/tmp/
> CHANGED_DIR=/tmp/changed/
> export THE_DATE=`date +%Y_%m_%d_%H_%M_%s`
> # you need to make sure the changed dir is writable, with good permissions.
> #  if you use /tmp uncomment these lines.
> #mkdir -p $CHANGED_DIR
> #chmod 755 $CHANGED_DIR
> 
> NUM_CHANGED=`rsync --delete --backup
> --backup-dir=$CHANGED_DIR/$THE_DATE --progress -v -a $DIR_TO_WATCH
> $BACKUP_DIR | grep "100%" | wc -l`
> 
> if [ "$NUM_CHANGED" = "0" ]; then
>     echo "nothing changed"
> else
>     echo "something changed"
>     # Put your commands here.  eg. rebuild cheetah templates, restart
> server, flush caches etc.
> fi
> 
> Then I use   `watch -n 2 ./watcher.sh`   in an xterm.  Which makes it
> run every two seconds.
> 
> User isolation:
> ----------------------
> 
> Running different processes for different users is the best way I
> think.  That way you can use operating system level resource
> constraints and auditing.  On pretendpaper.com I have given users with
> different roles separate processes.  Each process also uses a
> different database user, to add in another layer of security.  So
> basic members database user does not have access to
> select/delete/update tables, the process serving their requests should
> not be able to access them either.
> 
> On linux(and other unixes) with ulimit you can set each group of users
> having a different amount of available memory and cpu usage.  eg
> ulimit -v 50000 will limit it to running 50 megabytes of virtual
> memory.
> 
> Cheers,
> 
> Rene Dudfield
> 
> e: "@".join( reversed(['madecollective.com', 'rene']) )
> w: http://www.madecollective.com/
> w: http://www.pretendpaper.com/
> For creative, and technical services contact us @ made.
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/jjinux%40gmail.com
> 


-- 
I have decided to switch to Gmail, but messages to my Yahoo account will
still get through.


More information about the Web-SIG mailing list