[Baypiggies] Web Crawler/Backend Engineer - San Francisco, CA
K. Richard Pixley
rich at noir.com
Wed Feb 3 18:33:57 CET 2010
Jeremy Fishman wrote:
> My experience tells me something slightly different about Python:
> * for parallel processing, use multiple processes, ideally one per
> * for responsive I/O with an intuitive model, use threads
> * for high throughput I/O, use asynchronous I/O and/or an event
Even apart from python, that's pretty much my experience across the
board with other languages and other systems.
Multiple heavy weight processes have the advantage that they can be
spread across kernel instances which allows for hardware expansion,
allocating varying degrees of hardware power to different processes, etc.
I'll add that threads are generally less portable, much harder to debug,
(debuggers never work quite right), much harder to test, (those silly
nondeterminisms!), and that it's just as easy to architect an event
framework around asynchronous I/O which is just as intuitive as a
threaded model. (I'm doing this now for a python project).
I'm pretty down on threads. I think of them as a nasty
unprofessionalism akin to "goto" and symbolic links. I think it's
possible to write reasonable code with them. I just haven't found the
environment where that was easier or more straightforward than doing so
with async I/O or multiple heavy weight processes.
More information about the Baypiggies