[Chicago] Status of wsgi

[Daniel Peters]

Jordan, that urbanfresco site is really, really cool.  Many thanks for that.

On Thu, Oct 11, 2012 at 6:35 PM, Tal Liron <tal.liron at threecrickets.com>wrote:

>  I think your app is a great use case for Tornado, and a great topic for a
> ChiPy talk. ;)
>
> But I think it's also pretty clearly not a typical "web" (=REST)
> application. Which is really the myth I'm trying to dispel: if you are
> doing typical "web," async servers are of no particular use to you, whether
> for serving files or serving your own generated HTML entities. You yourself
> are immune to this particular myth, but I think the mythical formula
> "async=scalability" (or, worse, "async=fast") is pervasive among those
> developing typical web applications.
>
> I also readily agree that threading is not critical for many use cases:
> running multiple processes can be just fine, as in your case. But this
> assumes that every "worker" is independent and would not benefit from
> collaboration, especially via shared memory states. If you were doing, say,
> video streaming, there would be a lot of benefit in having threads
> coordinate and reuse memory, with a smart load balancer that knows to send
> clients watching the same video stream to the same process. Again, Python's
> abhorrence of threads is often quite perfect and simple. Still, it pays to
> know when it's just not the best tool for the job.
>
>
> On 10/11/2012 06:23 PM, Jordan Bettis wrote:
>
> On 10/11/2012 03:00 PM, Tal Liron wrote:
>
>
>  My life mission is to dispel this myth (especially because I used to
> believe it myself).
>
>  I'm not entirely sure what what myth you're dispelling.
>
> I suppose it's entirely possible that a carefully built synchronous
> thread model could be more efficient than an asynchronous epoll worker
> process model. I'd think the details would depend on the relative
> amounts of bookkeeping and overhead.
>
> Seems irrelevant when discussing Python though, because you'll either be
> spinning up and down a ton of interpreter instances or you'll be eaten
> by a g(il)rue.
>
> The criticism that tornado is poor at serving files to slow clients also
> kinda silly given that nobody does that and I think it's stated pretty
> early in the tornado documentation that it's not a good use case.
>
> I was never particularly interested in tornado until my latest project.
> I chose it because my old way of doing things fell down, not because
> reddit said it was the bee's knees or something.
>
> Anyway, my project has a map tile server builtin to it:http://urbanfresco.com
>
> Originally I just did the tile generation in the worker threads. But the
> thing fell apart with only one user if there were a lot of tiles to be
> generated, because all the workers would devote themselves to creating
> tiles and the site would go unresponsive.
>
> I reworked it so that the mapserver was a tornado requesthandler and the
> rest of the site was stuck inside a tornado wsgi handler. I then
> separated out the tilemaking to a set of daemons that the mapserver
> could send jobs to.
>
> If the tile is in the cache, the mapserver serves it directly, if it's
> not, then the map server requests the tilemaker generate it, with a
> callback, and then goes back to serving requests until it's done. I have
> nginx in front of the whole lot, like so:
>
> client <--> nginx --> static files
>               |
>               \-> tornado daemons --> mapserver --> tilemaker daemons
>                       /                   \
>           wsgi app <-/                     \-> tile cache
>
> I suppose I could have made a faster system by doing the whole thing as
> a multi-threaded C++ app... but given the comparative amount of work
> involved I'm very happy with the results of using tornado.
>
> The tornado and tile maker instances are started and stopped using a
> script and pid files, just like normal unix daemons. Static files
> including uploaded photos are served directly by nginx. I am serving the
> tile images through python, so I suppose a slow client could tie up my
> tornado processes. I was thinking that nginx would buffer the tiles for
> me but I suppose I should make sure of that.
>
> At any rate, my interest in tornado had nothing to do with slow clients.
> I have nginx for that. My interest was in separating tile generation out
> so it was no longer using up all my workers.
>
> _______________________________________________
> Chicago mailing listChicago at python.orghttp://mail.python.org/mailman/listinfo/chicago
>
>
>
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/chicago/attachments/20121011/f17265e1/attachment.html>