Non-blocking (asynchronous) timer without thread?
The question to python core developers: Is there any plans to implement non-blocking timer like a threading.Timer() but without thread? Some interpreted languages (like Tcl or Erlang) have such functionality, so I think it would be a great feature in Python :) The main goal is to prevent threads overhead and problems with race conditions and deadlocks. Thanks in advance!
On 12/22/06, Evgeniy Khramtsov <xramtsov@gmail.com> wrote:
The question to python core developers: Is there any plans to implement non-blocking timer like a threading.Timer() but without thread? Some interpreted languages (like Tcl or Erlang) have such functionality, so I think it would be a great feature in Python :)
The main goal is to prevent threads overhead and problems with race conditions and deadlocks.
I'm not sure how having python execute code at an arbitrary time would _reduce_ race conditions and/or deadlocks. And if you want to make it safe by executing code that shares no variables or resources, then it is no less safe to use threads, due to the GIL. If you can write you application in an event-driven way, Twisted might be able to do what you are looking for. cheers, -Mike
Mike Klaas пишет:
I'm not sure how having python execute code at an arbitrary time would _reduce_ race conditions and/or deadlocks. And if you want to make it safe by executing code that shares no variables or resources, then it is no less safe to use threads, due to the GIL.
Ok. And what about a huge thread overhead? Just try to start 10-50k threading timers :)
If you can write you application in an event-driven way, Twisted might be able to do what you are looking for.
I don't like an idea of Twisted: you want the banana, but get the whole gorilla as well :)
On 12/23/06, Evgeniy Khramtsov <xramtsov@gmail.com> wrote:
Mike Klaas пишет:
I'm not sure how having python execute code at an arbitrary time would _reduce_ race conditions and/or deadlocks. And if you want to make it safe by executing code that shares no variables or resources, then it is no less safe to use threads, due to the GIL.
Ok. And what about a huge thread overhead? Just try to start 10-50k threading timers :)
If you can write you application in an event-driven way, Twisted might be able to do what you are looking for.
I don't like an idea of Twisted: you want the banana, but get the whole gorilla as well :)
Well you simply can't do what you propose without writing code in the style of Twisted or with interpreter modifications or evil stack slicing such as with stackless or greenlet. If you aren't willing to choose any of those then you'll have to live without that functionality or use another language (though I can't think of any usable ones that actually safely do what you're asking). It should be relatively efficient to do what you want with a thread pool (one thread that manages all of the timers, and worker threads to execute the timer callbacks). FWIW, Erlang doesn't have that functionality. You can wait on messages with a timeout, but there are no interrupts. You do have cheap and isolated processes instead of expensive shared state threads, though. Writing Erlang/OTP code is actually a lot closer to writing Twisted style code than it is to other styles of concurrency (that you'd find in Python). It's just that Erlang/OTP has better support for concurrency oriented programming than Python does (across the board; syntax, interpreter, convention and libraries). -bob
Bob Ippolito пишет:
Well you simply can't do what you propose without writing code in the style of Twisted or with interpreter modifications or evil stack slicing such as with stackless or greenlet. If you aren't willing to choose any of those then you'll have to live without that functionality or use another language (though I can't think of any usable ones that actually safely do what you're asking). It should be relatively efficient to do what you want with a thread pool (one thread that manages all of the timers, and worker threads to execute the timer callbacks).
FWIW, Erlang doesn't have that functionality. You can wait on messages with a timeout, but there are no interrupts. You do have cheap and isolated processes instead of expensive shared state threads, though. Writing Erlang/OTP code is actually a lot closer to writing Twisted style code than it is to other styles of concurrency (that you'd find in Python). It's just that Erlang/OTP has better support for concurrency oriented programming than Python does (across the board; syntax, interpreter, convention and libraries).
Ok. I see your point. It looks like Python was not the right choice for my task :( Thanks.
Evgeniy Khramtsov <xramtsov@gmail.com> wrote:
Mike Klaas пиÑеÑ:
I'm not sure how having python execute code at an arbitrary time would _reduce_ race conditions and/or deadlocks. And if you want to make it safe by executing code that shares no variables or resources, then it is no less safe to use threads, due to the GIL.
Ok. And what about a huge thread overhead? Just try to start 10-50k threading timers :)
That is a foolish task in any language, and is why heap-based schedulers exist. You can use a Queue as synchronization between timer producers and the scheduler, and another Queue as synchronization between the scheduler and your threadpool (5-10 should be fine and have minimal overhead). Also 10-50k threads, in just about any language or runtime (except using C in solaris on SPARC, which can handle that pretty well), is way too much to handle. With thread stack sizes as they are, I wouldn't be surprised if you were running into the limit of your main memory size, and you would certianly be destroying the effectiveness of most caches. One thing that came to mind is that perhaps the "overhead" you are experiencing when using threads isn't context switch time, but is the fact that you have thousands of threads waiting on signals from the OS to continue, or even that Python switches threads after a certain number of bytecodes have been interpreted (if they are not waiting on wait syscalls). Python 2.5 sets the interval at 100 bytecodes (see sys.[get|set]checkinterval()), which is either 1 million or 5 million bytecodes with your load of 10k-50k. For most operations, this would be a palpable delay of up to a couple seconds before a thread gets back to doing what it was doing. Seriously, threadpool with a scheduler. - Josiah
Evgeniy Khramtsov <xramtsov@gmail.com> wrote:
The question to python core developers: Is there any plans to implement non-blocking timer like a threading.Timer() but without thread? Some interpreted languages (like Tcl or Erlang) have such functionality, so I think it would be a great feature in Python :)
The main goal is to prevent threads overhead and problems with race conditions and deadlocks.
How would you propose a non-threaded timer work? Would it involve the writing of a scheduler and the execution of scheduled tasks in a scheduler? If so, you can implement it today (I would suggest using the heapq module, and/or the pair heap I posted to this list a few weeks ago), or you can use a previously existing mainloop for handling events (wx.App().MainLoop() works well, you just need to write a wx.EvtHandler subclass). Heck, you could even use Twisted's event loop to handle it. If you were really crazy, you could even use exceptions to signal that an event was ready... import time import sys _scheduled = [] def hook(*args): if _scheduled and time.time() >= _scheduled[0][0]: raise Exception("Scheduled Event Pending!") sys.settrace(hook) Aside from using a library that does it for you, or hooking into native message hooks (I'm sure that Windows has a "notify this process in X seconds" message), I personally don't see a way of making what you want happen work in Python, or even *how* you would signal to the only thread in your application, "hey, there is this other task that should happen" that is substantially different from a periodic "if time.time() >= scheduled" check. - Josiah
Josiah Carlson пишет:
writing of a scheduler and the execution of scheduled tasks in a scheduler? If so, you can implement it today (I would suggest using the heapq module, and/or the pair heap I posted to this list a few weeks ago), or you can use a previously existing mainloop for handling events (wx.App().MainLoop() works well, you just need to write a wx.EvtHandler subclass).
I did a dirty hack in the asyncore.py module, and now my asyncore objects have methods for timers (self.start_timer, self.cancel_timer, etc.) But I think that timer scheduling is a job for a python interpreter and it must be hidden from an end programmer. Regards.
participants (5)
-
"Martin v. Löwis" -
Bob Ippolito -
Evgeniy Khramtsov -
Josiah Carlson -
Mike Klaas