Python web development, really

[Ian Bicking]

On Fri, 2003-01-24 at 10:08, Jacob Smullyan wrote:
> Ian Bicking <ianb at colorstudy.com> writes:
> 
> > This data can be used for caching.  IMHO the convenience of this caching
> > makes increasing performance easier in Webware than, say, SkunkWeb
> > (which uses multiple processes, and so makes data sharing harder). 
> > SkunkWeb claims the multiple processes mean that it can better utilize
> > multi-processor systems because it is not pinned by the global
> > interpreter lock.  This is true, but there are many ways to increase
> > performance, and I believe easy data sharing and caching is more of a
> > benefit than multiple processors would be.  But anyway, I digress...
> 
> I'm a developer for SkunkWeb, and would make a different value
> judgement, although I have no factual disagreement with Mr. Bicking.
> I would agree that SkunkWeb's multiple-process design does not
> necesarrily translate into a performance advantage in all scenarios,
> and the published SkunkWeb propaganda may make too much of this design
> decision (although I think it was an important factor for the
> applications for which it was initially designed).  There is a cost to
> be paid in the heaviness of the application (which doesn't matter
> much) and the relative difficulty in doing some kinds of resource
> sharing that would be simpler or perhaps more efficient in a
> multi-threaded context.

I'm sorry, I was kind of randomly poking randomly at SkunkWeb, a product
for which I really have no antipathy.  But anyway...

I guess to me performance is something that can *always* be improved
through a huge number of different techniques.  The real limits come
from the amount of effort you can put into those techniques.  So
theoretical limits aren't as important as the ease of incremental
improvements.  There aren't any theoretical limits either, just limits
within certain bounds.  SkunkWeb posits a situation where there's
multiple processors, and the application is bound by Python's execution
speed, which is not that general of a situation.

> There are some pluses to a multi-process design beyond transcending
> the limitations of the global interpreter lock.  You never ever have
> to worry about thread sychronization in writing for SkunkWeb, which
> will make some developers' lives easier.  

I don't see this as a real advantage.  To the degree you share data and
resources you have to worry about concurrency issues.  You may tend to
share data more often with threads, and so have to worry about
concurrency issues more often, but that just mitigates the advantages of
sharing.  I don't think it makes multiple processes easier than threads.

Well, I suppose you can use globals in SkunkWeb in a way you couldn't in
Webware.  Perhaps it's because I've always avoided globals, or perhaps
because I've internalized Webware's model, but I don't really desire to
use globals except when they are truly "global", i.e., shared.

> You don't have to worry
> about crashing the whole application server.  It just fits with
> SkunkWeb's rather open, un-boxed-in aesthetic; it is an easy design to
> live with, for some folks.  Let's just say, without starting a
> religious war about threads (I'm agnostic!) that a multi-process
> alternative is nice to have.

Certainly there can be issues in a single-process model.  So far many of
them have come from buggy C modules, but there's other problems of
corruption as well.  I must admit I remain a little wary of letting the
Webware server sit for months at a time -- cruft may start filling up
the cache, or other problems so far unforeseen may occur.  This is a
problem that has always made me nervous of any stateful long-running
application, Webware included.

But I feel the problems are more connected with controlling persistence
then threads in particular.  The partitioning required for multiple
processes tends to formalize some of that persistence, and I think the
advantage is as much one of enforced discipline as it is a quality of
the underlying technique.

> As for caching, SkunkWeb is I believe very strong in this area, as it
> does have a very useful component cache, which stores component output
> on disk; and this cache can be controlled with considerable finesse,
> and furthermore with ease.  While access to a disk cache is not going
> to be as fast as access to memory, just as with the GIL lock issue,
> there are other ways of improving performance.  The convenience and
> controllability of the SkunkWeb cache is a huge advantage, as it can
> be utilized to the fullest with virtually no development effort.

That is a definite advantage of SkunkWeb which Webware does not share. 
We don't have any real abstract representation of the output in Webware
-- the output is almost opaque to the application server -- so it's not
clear where caching would go in.  Cheetah has some significant caching
features, if you use that.  Any caching done more directly in Webware is
usually from a pre-HTML perspective, caching calculated data and complex
objects, not necessarily the direct output.

-- 
Ian Bicking           Colorstudy Web Development
ianb at colorstudy.com   http://www.colorstudy.com
PGP: gpg --keyserver pgp.mit.edu --recv-keys 0x9B9E28B7
4869 N Talman Ave, Chicago, IL 60625 / (773) 275-7241