[Baypiggies] Web Crawler/Backend Engineer - San Francisco, CA

[Stephen McInerney]

Guys, 

 

Since this is a discussion of broad interest, why not have a night

on head-to-head comparison of event-based frameworks and threading?

(Jim is our next free date March?)

 

Should I ask for a +1 if anyone agrees?

 

Stephen
 


Date: Thu, 4 Feb 2010 09:25:43 -0800
From: alecf at flett.org
To: rich at noir.com
CC: baypiggies at python.org
Subject: Re: [Baypiggies] Web Crawler/Backend Engineer - San Francisco, CA





What experience do folks have with event based frameworks in python?






On the HTTP server side, Twisted is by far the most pervasive, but Tornado, which came out last year, has had a lot of positive responses, and I've heard of more and more people using it. In general the trade off with these is you have is that you have to think more about parallelism even during a single http request because your control flow changes dramatically - learn about Deferreds to learn more. 


The bonus is that you get that parallelism automatically, very cheaply, once you buy into the framework... but the drawback is that that because your control flow is no longer stack-based, WSGI generally goes out the window... so it's virtually impossible to reuse all those great WSGI middleware packages out there.


Stackless has an excellent solution for this which is to make it look like your control flow follows the stack, and thus you can actually use WSGI middleware.


Why is the "automatic parallelism" I mentioned above important? At least where I'm working, a single incoming HTTP request is fulfilled by the appserver making several (2-100) internal HTTP requests to internal services. If *those* requests can be made in parallel, then you can really get very high throughput. 


Our solution was actually to run Twisted inside of Pylons, but that's a longer discussion...


Alec
 
I was thinking I was going to have to write my own, but I stumbled upon "circuits" today and then read through some of the "twisted" doc and was surprised to find that it closely mirrored my own thoughts.

Are there others?  Are any of the implementations definitive either positively or negatively?

--rich
_______________________________________________
Baypiggies mailing list
Baypiggies at python.org
To change your subscription options or unsubscribe:
http://mail.python.org/mailman/listinfo/baypiggies

 		 	   		  
_________________________________________________________________
Hotmail: Trusted email with powerful SPAM protection.
http://clk.atdmt.com/GBL/go/201469227/direct/01/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20100204/0a93616f/attachment.htm>