[Python-ideas] non-blocking buffered I/O

[Kristján Valur Jónsson]

> -----Original Message-----
> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf
> Of Guido van Rossum
> Sent: 31. október 2012 15:01
> To: Kristján Valur Jónsson
> Cc: python-ideas at python.org
> Subject: Re: [Python-ideas] non-blocking buffered I/O
> 
> Modern CPUs are black boxes full of magic. I'm not too surprised that running
> Python code on multiple threads incurs some kind of overhead that keeping
> the Python interpreter in one thread avoids.
> 
Ah, but I forgot to mention one weird thing:
If we used a pool of threads for the callbacks, and pre-initalized those threads with python states, and then acquired the GIL using 
PyEval_RestoreThread(), then this overhead went away.
It was only the dynamic tread state acquired using PyGilState_Ensure() that caused cpu overhead.
Using the fixed pool was not acceptable in the long run, in particular we din't want to complicate things to another level by adding a thread pool manger to the whole thing when the OS is fully capable of providing an external callback thread.

I regret not spending more time on this and to be able to provide an actual performance analysis and fix.  Instead I have to be that weird old man in the tavern uttering inscrutable warnings that no young adventurer pays any attention to :)

K