[Baypiggies] Web Crawler/Backend Engineer - San Francisco, CA
damonmc at gmail.com
Fri Feb 5 02:07:39 CET 2010
I don't agree with all the presenter's conclusions, but this
video<http://blip.tv/file/2232349>mentions a bunch of
Cogen, Multitask, Chiral, Eventlet, Weightless, Fibra, Stackless Python,
Twisted, Concurrence, Circuits.Web, Greenlet, Candygram, Papyros, Parley.
(I copied this list from 34:10)
On Thu, Feb 4, 2010 at 11:24 AM, Jeff Kunce <jjkunce at gmail.com> wrote:
> OK. It's not a simple messaging problem, and it's not primarily an
> async IO problem. If it were me, I'd spend a few hours researching
> Twisted. Read some of the online reviews/tutorials about it, then dig
> into the real thing. It's a pretty impressive framework. The most
> common complaint is that it has its own (somewhat "twisted")
> culture/terminology that makes it hard to learn.
> Spread is cool. I've tried several times to find an excuse to use it :)
> -- Jeff
> On Thu, Feb 4, 2010 at 10:28 AM, K. Richard Pixley <rich at noir.com> wrote:
> > Jeff Kunce wrote:
> > Actually, I'm wondering about the scope of the original question about
> > an event-based framework:
> > - Just wanting objects to communicate with events? Something simple
> > like circuits is good.
> > - async IO? asyncore, medusa et al are tried and true
> > - A comprehensive framework with everything included (even peanut
> > butter and jelly)? Twisted
> > The scope really matters. If you just want to send simple events
> > between objects in your app, you could write your own circuits-like
> > framework before you could even figure out how to read the
> > documentation for Twisted.
> > I'm building a moderately complex automated builder based primarily on
> > incremental builds. Multiple servers, each parallel in their own right,
> > building multiple code branches against multiple target machines. So
> > got three or four distinct levels of parallelism going on in the builds
> > already. And there are several types of checking going on, including
> > precommit speculative testing, post commit "continuous builds", a form of
> > partial building used for dependency checking, and a sort of
> > checking, (essentially branched recurrences of the pre-commit checker).
> > builder monitors a source code repository building when necessary as well
> > taking speculative build requests from users and building those.
> > Most of this parallelism doesn't happen in python. The python pieces of
> > builder coordinate the various builders, partition and aggregate very
> > level parallelism, manage working directories, and alerting mechanisms.
> > My plan is a distributed, dynamic, asynchronous, fault tolerant
> > based on virtual synchrony, (
> > - probably the spread toolkit http://spread.org).
> > ext-js grid widget), and my plan is to produce their data using pylons,
> > perhaps tg2, which pulls data out of the virtual synchrony "cloud",
> > via spread, but potentially via traditional socket calls, maybe even
> > jsonrpc.
> > So the "daemon"/cloud part needs to be capable of managing distributed,
> > shared state, including the work queue, through spread, as well as
> > a few subtasks like builds. I'm not big on threads, and they don't
> > help much in the virtual synchrony paradigm anyway.
> > It's not a huge project, but I'm expecting it to take me at least a few
> > months to implement.
> > --rich
> > _______________________________________________
> > Baypiggies mailing list
> > Baypiggies at python.org
> > To change your subscription options or unsubscribe:
> > http://mail.python.org/mailman/listinfo/baypiggies
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Baypiggies