[concurrency] Inside the Python GIL

Aahz aahz at pythoncraft.com
Fri Jun 12 19:50:00 CEST 2009


On Fri, Jun 12, 2009, Jeremy Hylton wrote:
>
> I'm not sure I understand how to distinguish between I/O bound threads
> and CPU bound threads.  If you've got a relatively simple
> multi-threaded application like an HTTP fetcher with a thread pool
> fetching a lot of urls, you're probably going to end up having more
> than one thread  with input to process at any instant.  There's a ton
> of Python code that executes when that happens.  You've got a urllib
> addinfourl wrapper, a httplib HTTPResponse (with read & _safe_read)
> and a socket _fileobject.  Heaven help you if you are using readline.
> So I could image even this trivial I/O bound program having lots of
> CPU contention.

You could imagine, but have you tested it?  ;-)  Back in the 1.5.2 days,
I helped write a web crawler where the sweet spot was around twenty or
thirty threads.  That clearly indicates a significant I/O bottleneck.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"Many customs in this life persist because they ease friction and promote
productivity as a result of universal agreement, and whether they are
precisely the optimal choices is much less important." --Henry Spencer


More information about the concurrency-sig mailing list