[Python-Dev] Use QueryPerformanceCounter() for time.monotonic() and/or time.highres()?

[Victor Stinner]

2012/3/31 Guido van Rossum <guido at python.org>:
> Given the amount of disagreement I sense, I think we'll need to wait
> for more people to chime in.

I hope that the PEP now gives enough data to help to choose the best API.

>> Hum, I read that you can expect a drift between QPC and GetTickCount.
>> I don't have exact numbers. There are also issues with power-saving.
>> When the CPU frequency changes, the frequency of the TSC is also
>> impacted on some processors (others provide a TSC at a fixed
>> frequency).
>
> And how does that impact teh various use cases? (There still isn't
> enough about use cases in the PEP. Thank you very much though for
> adding all those details about the various platform clocks.)

I didn't find a good source, but it looks like QueryPerformanceCounter
behaves badly on system suspend/resume, whereas GetTickCount is
reliable and has a well defined behaviour: "The elapsed time retrieved
by GetTickCount or GetTickCount64 includes time the system spends in
sleep or hibernation." So QPC is not really monotonic nor steady. It
may be even worse than the system clock (especially in virtual
machines).

Impact: A timeout may be 42 seconds shorter than the requested
duration if is uses QPC. For a scheduler, a task would be scheduled at
the right moment.

>> I prefer use cases:
>>
>>  * time.highres() should be used for profiling and benchmarking
>>  * time.monotonic() should be used for a scheduler and timeout
>
> So if I want to schedule something a week in the future, what should I
> use? monotonic() or time()? And can you explain the reason for your
> answer? What about an hour or a day? Or a month or a year?

You should use time.monotonic() because time.monotonic() has a steady
rate and so is reliable for long duration (at least more reliable than
time.highres()).

In practice, time.monotonic() and time.highres() are only different on
Windows. If I understood correctly, the Windows kernel uses something
like GetTickCount whch has an accuracy of 1 ms in the best case. So
Python only needs a clock with a similar accuracy for timeout and
schedulers.

time.highres() (QPC) rate is only steady during a short duration: it
is not an issue for a benchmark because you usually rerun the same
test at least 3 times and keep the minimum. It looks like QPC bugs are
only unexpected forward jumps (no backward jump), so taking the
minimum would workaround these issues. The hibernation issue should
not affect benchmarking/profiling.

>> time.monotonic() rate should be as steady as possible, its stability
>> is more important than its accuracy. time.highres() should provide the
>> clock with the best accuracy. Both clocks may have an undefined
>> starting point.
>
> So what kind of drift is acceptable for each?

I don't know.

>> I disagree. Python must provide a truly monotonic clock, even it is
>> not truly monotonic by default.
>
> Why?

See Zooko Wilcox-O'Hearn's email:
http://mail.python.org/pipermail/python-dev/2012-March/118147.html

>> I suggest this API: time.monotonic(fallback=True).
>
> But this goes against the idea of "A keyword argument that gets passed
> as a constant in the caller is usually poor API."

Right :-) After changing (again) the PEP, I realized that it
reintroduces the time.monotonic(fallback=False) raising
NotImplementedError case and I really don't like this exception here.

> And it would encourage the creation of trivial lambdas just to call
> the timer with this flag, since a universal API is a timer function
> that takes no arguments.

My bet is that fallback=True is an uncommon use case.

>> Sometimes, it does matter which exact OS clock is used.
>
> When is that?

If you need to know more properties of the clock,. For example, Python
may be unable to give the accuracy of the clock, but you may get it
differently if you know which clock function is used.

>> Should I include such new function in the PEP, or can it be discussed
>> (later) in a separated thread?
>
> FWIW, I prefer a single thread devoted to all aspects of this PEP.

Let's propose an API:

time.clock_info() ->
{'time': {'function': 'gettimeofday', 'resolution': 1e-6, 'monotonic': False},
 'monotonic': {'function': 'clock_gettime(CLOCK_MONOTONIC)',
'resolution': 1e-9, 'monotonic': True},
 'highres': {'function': 'clock_gettime(CLOCK_MONOTONIC)',
'resolution': 1e-9, 'monotonic': True}}

or

time.clock_info('time') -> {'function': 'gettimeofday', 'resolution':
1e-6, 'monotonic': False}
time.clock_info('monotonic') -> {'function':
'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic':
True}
time.clock_info( 'highres') -> {'function':
'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic':
True}

or

time.clock_info(time.time) -> {'function': 'gettimeofday',
'resolution': 1e-6, 'monotonic': False}
time.clock_info(time.monotonic) -> {'function':
'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic':
True}
time.clock_info( time.highres) -> {'function':
'clock_gettime(CLOCK_MONOTONIC)', 'resolution': 1e-9, 'monotonic':
True}

"clock_" prefix in "time.clock_xxx" name is used by POSIX clock_xxx
functions, another prefix may be used instead.

>> I don't want to only provide time.monotonic() which would fallback if
>> no monotonic clock is available, because we would be unable to tell if
>> the clock is monotonic or not.
>
> There could be a separate inquiry API.

Well, the PEP mentions something like that in the "One function, no
flag" section.

>> See the implementation in the PEP: the
>> "try monotonic or fallback" function may use a different clock at each
>> call, and so may be monotonic at startup and then becomes
>> non-monotonic.
>
> No, it should never switch implementations once it has chosen. That's
> just asking for trouble. If it can become non-monotonic it just isn't
> monotonic. Again, Cameron Simpson's idea might help here.

Correct. In practice, one call to time.monotonic() is enough to know
if next calls will fail or not, and so you don't have to poll regulary
to check if the clock becomes non-monotonic. So always fallback and
providing something to check if time.monotonic() is or not monotonic
is be an acceptable solution. I agree that it is less ugly than the
time.monotonic(fallback=True) API.

Victor