[Python-Dev] PEP 418: Add monotonic clock

[Guido van Rossum]

On Wed, Mar 28, 2012 at 8:08 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Thu, Mar 29, 2012 at 12:42 AM, Guido van Rossum <guido at python.org> wrote:
>> As I said, I think the caching idea is bad. We may have to settle for
>> semantics that are less than perfect -- presumably if you are doing
>> benchmarking you just have to throw away a bad result that happened to
>> be affected by a clock anomaly, and if you are using timeouts, retries
>> are already part of life.
>
> I agree caching doesn't solve the problems that are solved by an OS
> level monotonic clock, but falling back to an unmodifided time.time()
> result instead doesn't solve those problems either. Falling back to
> time.time() just gives you the status quo: time may jump forwards or
> backwards by an arbitrary amount between any two calls. Cached
> monotonicity just changes the anomalous modes to be time jumping
> forwards, or time standing still for an extended period of time. The
> only thing the caching provides is that it becomes a reasonable
> fallback for a function called time.monotonic() - it *is* a monotonic
> clock that meets the formal contract of the function, it's just
> nowhere near as good or effective as one the OS can provide.

TBH, I don't think like this focus on monotonicity as the most
important feature.

> Forward jumping anomalies aren't as harmful, are very hard to detect
> in the first place and behave the same regardless of the presence of
> caching, so the interesting case to look at is the difference in
> failure modes when the system clock jumps backwards.

Agreed.

> For benchmarking, a caching clock will produce a zero result instead
> of a negative result. Zeros aren't quite as obviously broken as
> negative numbers when benchmarking, but they're still sufficiently
> suspicious that most benchmarking activities will flag them as
> anomalous. If the jump back was sufficiently small that the subsequent
> call still produces a higher value than the original call, then
> behaviour reverts to being identical.

So for benchmarking we don't care about jumps, really, and the caching
version is slightly less useful.

> For timeouts, setting the clock back means your operation will take
> longer to time out than you expected. This problem will occur
> regardless of whether you were using cached monotonicity (such that
> time stands still) or the system clock (such that time actually goes
> backwards). In either case, your deadline will never be reached until
> the backwards jump has been cancelled out by the subsequent passage of
> time.

Where in the stdlib do we actually calculate timeouts instead of using
the timeouts built into the OS (e.g. select())?

I think it would be nice if we could somehow use the *same* clock as
the OS uses to implement timeouts.

> I want the standard library to be able to replace its time.time()
> calls with time.monotonic().

Where in the stdlib? (I'm aware of threading.py. Any other places?)

> The only way we can do that without
> breaking cross-platform compatibility is if time.monotonic() is
> guaranteed to exist, even when the platform only provides time.time().
> A dumb caching fallback implementation based on time.time() is the
> easiest way to achieve that withou making a complete mockery of the
> "monotonic()" name.

Yeah, so maybe it's a bad name. :-)

> There is then a *different* use case, which is 3.3+ only code which
> wants to fail noisily when there's no OS level monotonic support - the
> application developer really does want to fail *immediately* if
> there's no OS level monotonic clock available, instead of crossing
> your fingers and hoping you don't hit a clock adjustment glitch
> (crossing your fingers has, I'll point out, been the *only* option for
> all previous versions of Python, so it clearly can't be *that* scary a
> prospect).
>
> So, rather than making time.monotonic() something that the *standard
> library can't use*, I'd prefer to address that second use case by
> exposing the OS level monotonic clock as time.os_monotonic() only when
> it's available. That way, the natural transition for old time.time()
> based code is to time.monotonic() (with no cross-platform support
> implications), but time.os_monotonic() also becomes available for the
> stricter use cases.

I'd be happier if the fallback function didn't try to guarantee things
the underlying clock can't guarantee. I.e. I like the idea of having a
function that uses some accurate OS clock if one exists but falls back
to time.time() if not; I don't like the idea of that new function
trying to interpret the value of time.time() in any way. Applications
that need the OS clock's guarantees can call it directly. We could
also offer something where you can introspect the properties of the
clock (or clocks) so that an app can choose the best clock depending
on its needs.

To summarize my problem with the caching idea: take a simple timeout
loop such as found in several places in threading.py.

def wait_for(delta, eps):
  # Wait for delta seconds, sleeping eps seconds at a time
  deadline = now() + delta
  while now() < deadline:
    sleep(eps)

If the now() clock jumps backward after the initial call, we end up
waiting too long -- until either the clock jumps forward again or
until we've made up the difference. If the now() clock jumps forward
after the initial call, we end up waiting less time, which is probably
not such a big problem (though it might).

But now consider a caching clock, and consider that the system clock
made a jump backwards *before* this function is called. The cache
prevents us from seeing it, so the initial call to now() returns the
highest clock value seen so far. And until the system clock has caught
up with that, now() will return the same value over and over -- so WE
STILL WAIT TOO  LONG.

My conclusion: you can't win this game by forcing the clock to return
a monotonic value. A better approach might be to compute how many
sleep(eps) calls we're expected to make, and to limit the loop to that
-- although sleep() doesn't make any guarantees either about sleeping
too short or too long. Basically, if you do sleep(1) and find that
your clock didn't move (enough), you can't tell the difference between
a short sleep and a clock that jumped back. And if your clock moved to
much, you still don't know if the problem was with sleep() or with
your clock.

-- 
--Guido van Rossum (python.org/~guido)