[Twisted-Python] accurate periodic call
Hello all, I was astonished to find out that looping call period depends on the system time by default. The periodic tick can even stall for a long time, if the system time jumps backwards during program execution. It turned out that this is in fact a python problem (not providing a monotonic time, at least not for posix). I urgently need accurate periodic call in my program and I've found the solution below that seems to be working. I kindly ask you for your comments: - Is this monotonic_time implementation OK from python perspective? - Is monkey patch to the reactor OK or is there any other solution more appropriate in this case (I do not want to patch each looping call, but once in the application)? - Does this patch have any negative influence to the rest of the reactor? - How would you implement a periodic function call in twisted application (as accurate as possible)? - Any chance to see something implemented inside twisted and/or python, so that applications don't need this kind of tricks? Thanks a lot for your comments. Zoran #! /usr/bin/env python from twisted.internet import task from twisted.internet import reactor import os import time import ctypes # python MONOTONIC time, borrowed here # http://stackoverflow.com/questions/1205722/how-do-i-get-monotonic-time-durat... if os.name == 'posix': CLOCK_MONOTONIC = 1 # see <linux/time.h> class timespec(ctypes.Structure): _fields_ = [ ('tv_sec', ctypes.c_long), ('tv_nsec', ctypes.c_long) ] librt = ctypes.CDLL('librt.so.1', use_errno=True) clock_gettime = librt.clock_gettime clock_gettime.argtypes = [ctypes.c_int, ctypes.POINTER(timespec)] def monotonic_time(): t = timespec() if clock_gettime(CLOCK_MONOTONIC, ctypes.pointer(t)) != 0: errno_ = ctypes.get_errno() raise OSError(errno_, os.strerror(errno_)) return t.tv_sec + t.tv_nsec * 1e-9 # monkey patch the reactor reactor.seconds = monotonic_time # TODO: check for other platforms!! else: monotonic_time = time.time def tick(): """This function is suppose to execute once a second, regardless of the system time.""" print 'tick', monotonic_time() loop = task.LoopingCall(tick) loop.start(1.0) reactor.run()
On 02/19/2012 11:49 AM, Zoran Bošnjak wrote:
Hello all, I was astonished to find out that looping call period depends on the system time by default. The periodic tick can even stall for a long time, if the system time jumps backwards during program execution. It turned out that this is in fact a python problem (not providing a monotonic time, at least not for posix).
I urgently need accurate periodic call in my program and I've found the solution below that seems to be working. I kindly ask you for your comments: - Is this monotonic_time implementation OK from python perspective? I didn't read it in detail, but it seems vaguely plausible; overriding reactor.seconds is the thing to do. - Is monkey patch to the reactor OK or is there any other solution more appropriate in this case (I do not want to patch each looping call, but once in the application)? That's probably the easiest in this situation. - Does this patch have any negative influence to the rest of the reactor? Everything is *supposed* to use reactor.seconds - grep for time.time and see if it shows up anywhere in twisted.internet. - How would you implement a periodic function call in twisted application (as accurate as possible)? LoopingCall if you had this patch. - Any chance to see something implemented inside twisted and/or python, so that applications don't need this kind of tricks?
See http://twistedmatrix.com/trac/ticket/2424 - using ctypes is probably easier than the C extension the included patch provides.
On Mon, Feb 20, 2012 at 12:14 PM, Itamar Turner-Trauring < itamar@itamarst.org> wrote:
Hello all, I was astonished to find out that looping call period depends on the system time by default. The periodic tick can even stall for a long time, if the system time jumps backwards during program execution. It turned out
On 02/19/2012 11:49 AM, Zoran Bošnjak wrote: that this is in fact a python problem (not providing a monotonic time, at least not for posix).
I urgently need accurate periodic call in my program and I've found the
solution below that seems to be working. I kindly ask you for your comments:
- Is this monotonic_time implementation OK from python perspective? I didn't read it in detail, but it seems vaguely plausible; overriding reactor.seconds is the thing to do. - Is monkey patch to the reactor OK or is there any other solution more appropriate in this case (I do not want to patch each looping call, but once in the application)? That's probably the easiest in this situation. - Does this patch have any negative influence to the rest of the reactor? Everything is *supposed* to use reactor.seconds - grep for time.time and see if it shows up anywhere in twisted.internet. - How would you implement a periodic function call in twisted application (as accurate as possible)? LoopingCall if you had this patch. - Any chance to see something implemented inside twisted and/or python, so that applications don't need this kind of tricks?
See http://twistedmatrix.com/trac/ticket/2424 - using ctypes is probably easier than the C extension the included patch provides.
_______________________________________________
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Does it work on windows??
On 20/02/12 04:14, Itamar Turner-Trauring wrote:
I find that ticket disturbing reading. The original description seems to claim there is a logic bug in LoopingCall that is triggered when the clock goes back, but subsequent discussion is all about the pros and cons of monotonic clocks. Is there such a bug, and if so should there be a separate ticket for it? Independent of monotonic clocks, it seems clear that LoopingCall stopping and never restarting is bad. Being called with the wrong interval is tedious, but far less serious, surely?
On 12:09 pm, p.mayers@imperial.ac.uk wrote:
On 20/02/12 04:14, Itamar Turner-Trauring wrote:
I find that ticket disturbing reading.
The original description seems to claim there is a logic bug in LoopingCall that is triggered when the clock goes back, but subsequent discussion is all about the pros and cons of monotonic clocks.
Is there such a bug, and if so should there be a separate ticket for it?
There is no such bug, as far as I am aware. Jean-Paul
On 20/02/12 14:29, exarkun@twistedmatrix.com wrote:
On 12:09 pm, p.mayers@imperial.ac.uk wrote:
On 20/02/12 04:14, Itamar Turner-Trauring wrote:
I find that ticket disturbing reading.
The original description seems to claim there is a logic bug in LoopingCall that is triggered when the clock goes back, but subsequent discussion is all about the pros and cons of monotonic clocks.
Is there such a bug, and if so should there be a separate ticket for it?
There is no such bug, as far as I am aware.
Phew! In which case, it's not disturbing. Thanks for the clarification. Slightly confusing that the OP seemed to experience such a bug, but it was 6 years ago I guess...
On Feb 20, 2012, at 12:52 PM, Phil Mayers wrote:
On 20/02/12 14:29, exarkun@twistedmatrix.com wrote:
On 12:09 pm, p.mayers@imperial.ac.uk wrote:
On 20/02/12 04:14, Itamar Turner-Trauring wrote:
I find that ticket disturbing reading.
The original description seems to claim there is a logic bug in LoopingCall that is triggered when the clock goes back, but subsequent discussion is all about the pros and cons of monotonic clocks.
Is there such a bug, and if so should there be a separate ticket for it?
There is no such bug, as far as I am aware.
Phew! In which case, it's not disturbing. Thanks for the clarification.
Slightly confusing that the OP seemed to experience such a bug, but it was 6 years ago I guess...
Well, it depends on how you define a "bug". LoopingCall's internal state remains consistent, but if you set your clock backwards, LoopingCall won't fire your callback again until the system clock catches up to where it previously was. Any new LoopingCalls that are created will work fine, although they will probably report an incorrect skipCount if you set the clock forward again. This can be quite dramatic if you change the date, of course, but if you're talking about adjustments like NTP slew it's unlikely you'll notice it. -glyph
On 20/02/12 20:32, Glyph wrote:
Well, it depends on how you define a "bug". LoopingCall's internal state remains consistent, but if you set your clock backwards, LoopingCall won't fire your callback again until the system clock catches up to where it previously was. Any new LoopingCalls that are created will work fine, although they will probably report an incorrect skipCount if you set the clock forward again.
Ah. This doesn't jump out from the original ticket text; the way I read the text, it implies the LoopingCall stops being *scheduled* if the clock goes backwards. If I'm understanding it correctly, the problem is actually that the callLater for the next run doesn't fire when expected, because it's scheduled in the "pre-change" timeframe (now the future). As opposed to being anything in the LoopingCall code? I realise this is tricky to solve, but I'll note it's not impossible for really REALLY big clock skews to happen. For example: recently we had a server kernel panic and need a cold reboot. The machine booted and read it's time from the CMOS clock, which was way WAY out. A minute or two after the machine had booted, NTP slewed the clock back by months... Fortunately this box doesn't run any Twisted code that a) starts on boot and b) makes use of LoopingCall. [I know this is horrible, but a lot of Unix distros don't sync system to hardware clock periodically - usually on shutdown only - and a lot of distros don't block boot until NTP has set the clock] I realise it's an edge case; but at the very least, it is probably worth updating the LoopingCall/IReactorTime.callLater documentation to make this behaviour clear, because I certainly didn't appreciate this could occur. It's pretty obvious in hindsight, though.
On 11:05 am, p.mayers@imperial.ac.uk wrote:
On 20/02/12 20:32, Glyph wrote:
Well, it depends on how you define a "bug". LoopingCall's internal state remains consistent, but if you set your clock backwards, LoopingCall won't fire your callback again until the system clock catches up to where it previously was. Any new LoopingCalls that are created will work fine, although they will probably report an incorrect skipCount if you set the clock forward again.
Ah. This doesn't jump out from the original ticket text; the way I read the text, it implies the LoopingCall stops being *scheduled* if the clock goes backwards.
If I'm understanding it correctly, the problem is actually that the callLater for the next run doesn't fire when expected, because it's scheduled in the "pre-change" timeframe (now the future). As opposed to being anything in the LoopingCall code?
I realise this is tricky to solve, but I'll note it's not impossible for really REALLY big clock skews to happen. For example: recently we had a server kernel panic and need a cold reboot. The machine booted and read it's time from the CMOS clock, which was way WAY out. A minute or two after the machine had booted, NTP slewed the clock back by months... Fortunately this box doesn't run any Twisted code that a) starts on boot and b) makes use of LoopingCall.
I've fixed this problem before by ensuring ntp gets to run before other time-sensitive services get to start. Jean-Paul
[I know this is horrible, but a lot of Unix distros don't sync system to hardware clock periodically - usually on shutdown only - and a lot of distros don't block boot until NTP has set the clock]
I realise it's an edge case; but at the very least, it is probably worth updating the LoopingCall/IReactorTime.callLater documentation to make this behaviour clear, because I certainly didn't appreciate this could occur. It's pretty obvious in hindsight, though.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Tue, Feb 21, 2012 at 1:05 PM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
I realise this is tricky to solve, but I'll note it's not impossible for really REALLY big clock skews to happen. For example: recently we had a server kernel panic and need a cold reboot. The machine booted and read it's time from the CMOS clock, which was way WAY out. A minute or two after the machine had booted, NTP slewed the clock back by months...
This is kind of a nitpick, but I think it's a fairly important one: "slewing" the clock refers to a process where the clock frequency is adjusted to make it run faster or slower in order to catch up or lose a few seconds, without any discontinuities in the clock. This is a relatively slow process, so "slewing" the clock by months would take millennia; instead, beyond a certain threshold (I think a few seconds?), ntp will always step the clock, not slew it, which is when applications start running into difficulty as stepping introduces a discontinuity in the clock. -- mithrandi, i Ainil en-Balandor, a faer Ambar
On 02/22/2012 03:20 AM, Tristan Seligmann wrote:
On Tue, Feb 21, 2012 at 1:05 PM, Phil Mayers<p.mayers@imperial.ac.uk> wrote:
I realise this is tricky to solve, but I'll note it's not impossible for really REALLY big clock skews to happen. For example: recently we had a server kernel panic and need a cold reboot. The machine booted and read it's time from the CMOS clock, which was way WAY out. A minute or two after the machine had booted, NTP slewed the clock back by months...
This is kind of a nitpick, but I think it's a fairly important one: "slewing" the clock refers to a process where the clock frequency is
Sorry, you're quite right. Stepping, not slewing. Brain hiccup.
On Sun, 19 Feb 2012 16:49:23 +0000 Zoran Bošnjak <Zoran.Bosnjak@sloveniacontrol.si> wrote:
Hello all, I was astonished to find out that looping call period depends on the system time by default. The periodic tick can even stall for a long time, if the system time jumps backwards during program execution. It turned out that this is in fact a python problem (not providing a monotonic time, at least not for posix).
I urgently need accurate periodic call in my program and I've found the solution below that seems to be working. I kindly ask you for your comments: - Is this monotonic_time implementation OK from python perspective?
For the record: http://docs.python.org/dev/library/time.html#time.clock_gettime http://docs.python.org/dev/library/time.html#time.monotonic You'll have to fallback on the normal time when clock_gettime() fails (which can happen if there is a mismatch between glibc version and kernel version, for example). clock_gettime() is a POSIX standard so you probably shouldn't use a Linux-specific magic value for CLOCK_MONOTONIC. Regards Antoine.
Hello all, Thanks for all your comments. I have filed a ticket regarding this issue: http://twistedmatrix.com/trac/ticket/5552 I hope it will be accepted as a problem. regards, Zoran ________________________________________ From: twisted-python-bounces@twistedmatrix.com [twisted-python-bounces@twistedmatrix.com] on behalf of Zoran Bošnjak [Zoran.Bosnjak@sloveniacontrol.si] Sent: Sunday, February 19, 2012 5:49 PM To: twisted-python@twistedmatrix.com Subject: [Twisted-Python] accurate periodic call Hello all, I was astonished to find out that looping call period depends on the system time by default. The periodic tick can even stall for a long time, if the system time jumps backwards during program execution. It turned out that this is in fact a python problem (not providing a monotonic time, at least not for posix). I urgently need accurate periodic call in my program and I've found the solution below that seems to be working. I kindly ask you for your comments: - Is this monotonic_time implementation OK from python perspective? - Is monkey patch to the reactor OK or is there any other solution more appropriate in this case (I do not want to patch each looping call, but once in the application)? - Does this patch have any negative influence to the rest of the reactor? - How would you implement a periodic function call in twisted application (as accurate as possible)? - Any chance to see something implemented inside twisted and/or python, so that applications don't need this kind of tricks? Thanks a lot for your comments. Zoran #! /usr/bin/env python from twisted.internet import task from twisted.internet import reactor import os import time import ctypes # python MONOTONIC time, borrowed here # http://stackoverflow.com/questions/1205722/how-do-i-get-monotonic-time-durat... if os.name == 'posix': CLOCK_MONOTONIC = 1 # see <linux/time.h> class timespec(ctypes.Structure): _fields_ = [ ('tv_sec', ctypes.c_long), ('tv_nsec', ctypes.c_long) ] librt = ctypes.CDLL('librt.so.1', use_errno=True) clock_gettime = librt.clock_gettime clock_gettime.argtypes = [ctypes.c_int, ctypes.POINTER(timespec)] def monotonic_time(): t = timespec() if clock_gettime(CLOCK_MONOTONIC, ctypes.pointer(t)) != 0: errno_ = ctypes.get_errno() raise OSError(errno_, os.strerror(errno_)) return t.tv_sec + t.tv_nsec * 1e-9 # monkey patch the reactor reactor.seconds = monotonic_time # TODO: check for other platforms!! else: monotonic_time = time.time def tick(): """This function is suppose to execute once a second, regardless of the system time.""" print 'tick', monotonic_time() loop = task.LoopingCall(tick) loop.start(1.0) reactor.run() _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 11:57 am, zoran.bosnjak@sloveniacontrol.si wrote:
Hello all, Thanks for all your comments.
I have filed a ticket regarding this issue: http://twistedmatrix.com/trac/ticket/5552
I hope it will be accepted as a problem.
Hi Zoran, Thanks for the follow-up. We're already tracking this feature, so I've marked your ticket as a duplicate of the existing ticket: http://twistedmatrix.com/trac/ticket/2424 Jean-Paul
participants (8)
-
Antoine Pitrou -
exarkun＠twistedmatrix.com -
gelin yan -
Glyph -
Itamar Turner-Trauring -
Phil Mayers -
Tristan Seligmann -
Zoran Bošnjak