Evgeniy Khramtsov <xramtsov@gmail.com> wrote:
Mike Klaas пиÑеÑ:
I'm not sure how having python execute code at an arbitrary time would _reduce_ race conditions and/or deadlocks. And if you want to make it safe by executing code that shares no variables or resources, then it is no less safe to use threads, due to the GIL.
Ok. And what about a huge thread overhead? Just try to start 10-50k threading timers :)
That is a foolish task in any language, and is why heap-based schedulers exist. You can use a Queue as synchronization between timer producers and the scheduler, and another Queue as synchronization between the scheduler and your threadpool (5-10 should be fine and have minimal overhead). Also 10-50k threads, in just about any language or runtime (except using C in solaris on SPARC, which can handle that pretty well), is way too much to handle. With thread stack sizes as they are, I wouldn't be surprised if you were running into the limit of your main memory size, and you would certianly be destroying the effectiveness of most caches. One thing that came to mind is that perhaps the "overhead" you are experiencing when using threads isn't context switch time, but is the fact that you have thousands of threads waiting on signals from the OS to continue, or even that Python switches threads after a certain number of bytecodes have been interpreted (if they are not waiting on wait syscalls). Python 2.5 sets the interval at 100 bytecodes (see sys.[get|set]checkinterval()), which is either 1 million or 5 million bytecodes with your load of 10k-50k. For most operations, this would be a palpable delay of up to a couple seconds before a thread gets back to doing what it was doing. Seriously, threadpool with a scheduler. - Josiah