[BangPypers] HarvestMan with eventlets

Anand Balachandran Pillai abpillai at gmail.com
Thu Dec 27 07:56:49 CET 2007


Gr8. Have you written any code in stackless ? If you are familiar with
it, we could plan for a meeting in the new year (reminiscent of good
old days) where we can discuss coroutines,
stackless and eventlets.

What say ?

Regards

--Anand

On Dec 27, 2007 12:09 PM, gnuyoga <gnuyoga at gmail.com> wrote:
>
> Anand Balachandran Pillai wrote:
> > Hi,
> >
> >      Continuing from my previous post, the context of which was eventlet,
> > here is the reason why I was asking about it.
> >
> > I have been planning to rewrite HarvestMan using a non-blocking approach
> > rather than the current design of achieving concurrency using multiple
> > threads (which do not scale very well in Python).
> >
> > I have been evaluating twisted for sometime for doing this, but I have not
> > been able to wrap my brain around twisted. Also the fact that twisted
> > requires an explicit event loop call is not very good and makes it difficult
> > to write an application which has its own separate event loop.
> >
> > Eventlets look promising because it is implemented using coroutines without
> > an explicit event loop call, instead the event loop is run transparently when
> > required. Also due to its design, it can be easily merged with Stackless, which
> > I think it is already on the way to.
> >
> > Moreover, the sample code shows a crawler example, so there it is!
> >
> > I am looking at creating another implementation of HarvestMan based on
> > eventlets rather than threads. This will run parallel to the regular HarvestMan.
> > The plan is to write a crawler that scales very well w.r.t the number
> > of requests
> > and websites.
> >
> > Let me know if someone would like to be part of this.
> >
> > Thanks
> >
> >
> + 1
>
> am a g8 fan of stackless. will read about eventlet and get back 2 u.
>
> - sree
>
>
>
> --
> http://picasaweb.google.com/gnuyoga
>
> Each soul is potentially divine. The goal is to manifest the divine by controlling nature, internal or external. Do this by work or worship or psychic control or philosophy by one or more, or all of these and be free.
>
> _______________________________________________
> BangPypers mailing list
> BangPypers at python.org
> http://mail.python.org/mailman/listinfo/bangpypers
>



-- 
-Anand


More information about the BangPypers mailing list