[Baypiggies] Web Crawler/Backend Engineer - San Francisco, CA

[K. Richard Pixley]

Jeremy Fishman wrote:
> My experience tells me something slightly different about Python:
>   * for parallel processing, use multiple processes, ideally one per 
> processor
>   * for responsive I/O with an intuitive model, use threads
>   * for high throughput I/O, use asynchronous I/O and/or an event 
> framework
Even apart from python, that's pretty much my experience across the 
board with other languages and other systems.

Multiple heavy weight processes have the advantage that they can be 
spread across kernel instances which allows for hardware expansion, 
allocating varying degrees of hardware power to different processes, etc.

I'll add that threads are generally less portable, much harder to debug, 
(debuggers never work quite right), much harder to test, (those silly 
nondeterminisms!), and that it's just as easy to architect an event 
framework around asynchronous I/O which is just as intuitive as a 
threaded model.  (I'm doing this now for a python project).

I'm pretty down on threads.  I think of them as a nasty 
unprofessionalism akin to "goto" and symbolic links.  I think it's 
possible to write reasonable code with them.  I just haven't found the 
environment where that was easier or more straightforward than doing so 
with async I/O or multiple heavy weight processes.

--rich