[concurrency] Inside the Python GIL
aahz at pythoncraft.com
Fri Jun 12 19:50:00 CEST 2009
On Fri, Jun 12, 2009, Jeremy Hylton wrote:
> I'm not sure I understand how to distinguish between I/O bound threads
> and CPU bound threads. If you've got a relatively simple
> multi-threaded application like an HTTP fetcher with a thread pool
> fetching a lot of urls, you're probably going to end up having more
> than one thread with input to process at any instant. There's a ton
> of Python code that executes when that happens. You've got a urllib
> addinfourl wrapper, a httplib HTTPResponse (with read & _safe_read)
> and a socket _fileobject. Heaven help you if you are using readline.
> So I could image even this trivial I/O bound program having lots of
> CPU contention.
You could imagine, but have you tested it? ;-) Back in the 1.5.2 days,
I helped write a web crawler where the sweet spot was around twenty or
thirty threads. That clearly indicates a significant I/O bottleneck.
Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/
"Many customs in this life persist because they ease friction and promote
productivity as a result of universal agreement, and whether they are
precisely the optimal choices is much less important." --Henry Spencer
More information about the concurrency-sig