[Baypiggies] Web Crawler/Backend Engineer - San Francisco, CA

Jeff Enderwick jeff.enderwick at gmail.com
Wed Feb 3 19:36:57 CET 2010


I think you insult "goto" with your statements below :-). I too have had
good experiences with event-based frameworks. You can avoid all that thread
pool stuff, and your state ends up in normal data structures that can be
easily dumped and displayed (versus some thread's stack frame).



On Wed, Feb 3, 2010 at 9:33 AM, K. Richard Pixley <rich at noir.com> wrote:

> Jeremy Fishman wrote:
>
>> My experience tells me something slightly different about Python:
>>  * for parallel processing, use multiple processes, ideally one per
>> processor
>>  * for responsive I/O with an intuitive model, use threads
>>  * for high throughput I/O, use asynchronous I/O and/or an event framework
>>
> Even apart from python, that's pretty much my experience across the board
> with other languages and other systems.
>
> Multiple heavy weight processes have the advantage that they can be spread
> across kernel instances which allows for hardware expansion, allocating
> varying degrees of hardware power to different processes, etc.
>
> I'll add that threads are generally less portable, much harder to debug,
> (debuggers never work quite right), much harder to test, (those silly
> nondeterminisms!), and that it's just as easy to architect an event
> framework around asynchronous I/O which is just as intuitive as a threaded
> model.  (I'm doing this now for a python project).
>
> I'm pretty down on threads.  I think of them as a nasty unprofessionalism
> akin to "goto" and symbolic links.  I think it's possible to write
> reasonable code with them.  I just haven't found the environment where that
> was easier or more straightforward than doing so with async I/O or multiple
> heavy weight processes.
>
> --rich
> _______________________________________________
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:
> http://mail.python.org/mailman/listinfo/baypiggies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20100203/74e78724/attachment.htm>


More information about the Baypiggies mailing list