[Python-Dev] [RFC] PEP 418: Add monotonic time, performance counter and process time functions
Victor Stinner
victor.stinner at gmail.com
Sun Apr 15 17:15:15 CEST 2012
Hi,
Here is a simplified version of the first draft of the PEP 418. The
full version can be read online.
http://www.python.org/dev/peps/pep-0418/
The implementation of the PEP can be found in this issue:
http://bugs.python.org/issue14428
I post a simplified version for readability and to focus on changes
introduced by the PEP. Removed sections: Existing Functions,
Deprecated Function, Glossary, Hardware clocks, Operating system time
functions, System Standby, Links.
---
PEP: 418
Title: Add monotonic time, performance counter and process time functions
Version: f2bb3f74298a
Last-Modified: 2012-04-15 17:06:07 +0200 (Sun, 15 Apr 2012)
Author: Cameron Simpson <cs at zip.com.au>, Jim Jewett
<jimjjewett at gmail.com>, Victor Stinner <victor.stinner at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 26-March-2012
Python-Version: 3.3
Abstract
========
This PEP proposes to add ``time.get_clock_info(name)``,
``time.monotonic()``, ``time.perf_counter()`` and
``time.process_time()`` functions to Python 3.3.
Rationale
=========
If a program uses the system time to schedule events or to implement
a timeout, it will not run events at the right moment or stop the
timeout too early or too late when the system time is set manually or
adjusted automatically by NTP. A monotonic clock should be used
instead to not be affected by system time updates:
``time.monotonic()``.
To measure the performance of a function, ``time.clock()`` can be used
but it is very different on Windows and on Unix. On Windows,
``time.clock()`` includes time elapsed during sleep, whereas it does
not on Unix. ``time.clock()`` precision is very good on Windows, but
very bad on Unix. The new ``time.perf_counter()`` function should be
used instead to always get the most precise performance counter with a
portable behaviour (ex: include time spend during sleep).
To measure CPU time, Python does not provide directly a portable
function. ``time.clock()`` can be used on Unix, but it has a bad
precision. ``resource.getrusage()`` can also be used on Unix, but it
requires to get fields of a structure and compute the sum of time
spent in kernel space and user space. The new ``time.process_time()``
function acts as a portable counter that always measures CPU time
(doesn't include time elapsed during sleep) and has the best available
precision.
Each operating system implements clocks and performance counters
differently, and it is useful to know exactly which function is used
and some properties of the clock like its resolution and its
precision. The new ``time.get_clock_info()`` function gives access to
all available information of each Python time function.
New functions:
* ``time.monotonic()``: timeout and scheduling, not affected by system
clock updates
* ``time.perf_counter()``: benchmarking, most precise clock for short
period
* ``time.process_time()``: profiling, CPU time of the process
Users of new functions:
* time.monotonic(): concurrent.futures, multiprocessing, queue, subprocess,
telnet and threading modules to implement timeout
* time.perf_counter(): trace and timeit modules, pybench program
* time.process_time(): profile module
* time.get_clock_info(): pybench program to display information about the
timer like the precision or the resolution
The ``time.clock()`` function is deprecated because it is not
portable: it behaves differently depending on the operating system.
``time.perf_counter()`` or ``time.process_time()`` should be used
instead, depending on your requirements. ``time.clock()`` is marked as
deprecated but is not planned for removal.
Python functions
================
New Functions
-------------
time.get_clock_info(name)
^^^^^^^^^^^^^^^^^^^^^^^^^
Get information on the specified clock. Supported clock names:
* ``"clock"``: ``time.clock()``
* ``"monotonic"``: ``time.monotonic()``
* ``"perf_counter"``: ``time.perf_counter()``
* ``"process_time"``: ``time.process_time()``
* ``"time"``: ``time.time()``
Return a dictionary with the following keys:
* Mandatory keys:
* ``"implementation"`` (str): name of the underlying operating system
function. Examples: ``"QueryPerformanceCounter()"``,
``"clock_gettime(CLOCK_REALTIME)"``.
* ``"resolution"`` (float): resolution in seconds of the clock.
* ``"is_monotonic"`` (bool): True if the clock cannot go backward.
* Optional keys:
* ``"precision"`` (float): precision in seconds of the clock
reported by the operating system.
* ``"is_adjusted"`` (bool): True if the clock is adjusted (e.g. by a
NTP daemon).
time.monotonic()
^^^^^^^^^^^^^^^^
Monotonic clock, i.e. cannot go backward. It is not affected by system
clock updates. The reference point of the returned value is
undefined, so that only the difference between the results of
consecutive calls is valid and is a number of seconds.
On Windows versions older than Vista, ``time.monotonic()`` detects
``GetTickCount()`` integer overflow (32 bits, roll-over after 49.7
days): it increases a delta by 2\ :sup:`32` each time than an overflow
is detected. The delta is stored in the process-local state and so
the value of ``time.monotonic()`` may be different in two Python
processes running for more than 49 days. On more recent versions of
Windows and on other operating systems, ``time.monotonic()`` is
system-wide.
Availability: Windows, Mac OS X, Unix, Solaris. Not available on
GNU/Hurd.
Pseudo-code [#pseudo]_::
if os.name == 'nt':
# GetTickCount64() requires Windows Vista, Server 2008 or later
if hasattr(time, '_GetTickCount64'):
def monotonic():
return _time.GetTickCount64() * 1e-3
else:
def monotonic():
ticks = _time.GetTickCount()
if ticks < monotonic.last:
# Integer overflow detected
monotonic.delta += 2**32
monotonic.last = ticks
return (ticks + monotonic.delta) * 1e-3
monotonic.last = 0
monotonic.delta = 0
elif os.name == 'mac':
def monotonic():
if monotonic.factor is None:
factor = _time.mach_timebase_info()
monotonic.factor = timebase[0] / timebase[1]
return _time.mach_absolute_time() * monotonic.factor
monotonic.factor = None
elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_HIGHRES"):
def monotonic():
return time.clock_gettime(time.CLOCK_HIGHRES)
elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_MONOTONIC"):
def monotonic():
return time.clock_gettime(time.CLOCK_MONOTONIC)
On Windows, ``QueryPerformanceCounter()`` is not used even though it
has a better precision than ``GetTickCount()``. It is not reliable
and has too many issues.
time.perf_counter()
^^^^^^^^^^^^^^^^^^^
Performance counter with the highest available precision to measure a
duration. It does include time elapsed during sleep and is
system-wide. The reference point of the returned value is undefined,
so that only the difference between the results of consecutive calls
is valid and is a number of seconds.
Pseudo-code::
def perf_counter():
if perf_counter.use_performance_counter:
if perf_counter.performance_frequency is None:
try:
perf_counter.performance_frequency =
_time.QueryPerformanceFrequency()
except OSError:
# QueryPerformanceFrequency() fails if the installed
# hardware does not support a high-resolution performance
# counter
perf_counter.use_performance_counter = False
else:
return _time.QueryPerformanceCounter() /
perf_counter.performance_frequency
else:
return _time.QueryPerformanceCounter() /
perf_counter.performance_frequency
if perf_counter.use_monotonic:
# The monotonic clock is preferred over the system time
try:
return time.monotonic()
except OSError:
perf_counter.use_monotonic = False
return time.time()
perf_counter.use_performance_counter = (os.name == 'nt')
if perf_counter.use_performance_counter:
perf_counter.performance_frequency = None
perf_counter.use_monotonic = hasattr(time, 'monotonic')
time.process_time()
^^^^^^^^^^^^^^^^^^^
Sum of the system and user CPU time of the current process. It does
not include time elapsed during sleep. It is process-wide by
definition. The reference point of the returned value is undefined,
so that only the difference between the results of consecutive calls
is valid.
It is available on all platforms.
Pseudo-code [#pseudo]_::
if os.name == 'nt':
def process_time():
handle = win32process.GetCurrentProcess()
process_times = win32process.GetProcessTimes(handle)
return (process_times['UserTime'] +
process_times['KernelTime']) * 1e-7
else:
import os
try:
import resource
except ImportError:
has_resource = False
else:
has_resource = True
def process_time():
if process_time.use_process_cputime:
try:
return time.clock_gettime(time.CLOCK_PROCESS_CPUTIME_ID)
except OSError:
process_time.use_process_cputime = False
if process_time.use_getrusage:
try:
usage = resource.getrusage(resource.RUSAGE_SELF)
return usage[0] + usage[1]
except OSError:
process_time.use_getrusage = False
if process_time.use_times:
try:
times = os.times()
return times[0] + times[1]
except OSError:
process_time.use_getrusage = False
return _time.clock()
process_time.use_process_cputime = (
hasattr(time, 'clock_gettime')
and hasattr(time, 'CLOCK_PROCESS_CPUTIME_ID'))
process_time.use_getrusage = has_resource
# On OS/2, only the 5th field of os.times() is set, others are zeros
process_time.use_times = (hasattr(os, 'times') and os.name != 'os2')
Alternatives: API design
========================
Other names for time.monotonic()
--------------------------------
* time.counter()
* time.metronomic()
* time.seconds()
* time.steady(): "steady" is ambiguous: it means different things to
different people. For example, on Linux, CLOCK_MONOTONIC is
adjusted. If we uses the real time as the reference clock, we may
say that CLOCK_MONOTONIC is steady. But CLOCK_MONOTONIC gets
suspended on system suspend, whereas real time includes any time
spent in suspend.
* time.timeout_clock()
* time.wallclock(): time.monotonic() is not the system time aka the
"wall clock", but a monotonic clock with an unspecified starting
point.
The name "time.try_monotonic()" was also proposed for an older
proposition of time.monotonic() which was falling back to the system
time when no monotonic clock was available.
Other names for time.perf_counter()
-----------------------------------
* time.hires()
* time.highres()
* time.timer()
Only expose operating system clocks
-----------------------------------
To not have to define high-level clocks, which is a difficult task, a
simpler approach is to only expose operating system clocks.
time.clock_gettime() and related clock identifiers were already added
to Python 3.3 for example.
time.monotonic(): Fallback to system time
-----------------------------------------
If no monotonic clock is available, time.monotonic() falls back to the
system time.
Issues:
* It is hard to define correctly such function in the documentation:
is it monotonic? Is it steady? Is it adjusted?
* Some user want to decide what to do when no monotonic clock is
available: use another clock, display an error, or do something
else?
Different APIs were proposed to define such function.
One function with a flag: time.monotonic(fallback=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* time.monotonic(fallback=True) falls back to the system time if no
monotonic clock is available or if the monotonic clock failed.
* time.monotonic(fallback=False) raises OSError if monotonic clock
fails and NotImplementedError if the system does not provide a
monotonic clock
A keyword argument that gets passed as a constant in the caller is
usually poor API.
Raising NotImplementedError for a function is something uncommon in
Python and should be avoided.
One time.monotonic() function, no flag
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
time.monotonic() returns (time: float, is_monotonic: bool).
An alternative is to use a function attribute:
time.monotonic.is_monotonic. The attribute value would be None before
the first call to time.monotonic().
Choosing the clock from a list of constraints
---------------------------------------------
The PEP as proposed offers a few new clocks, but their guarentees
are deliberately loose in order to offer useful clocks on different
platforms. This inherently embeds policy in the calls, and the
caller must thus choose a policy.
The "choose a clock" approach suggests an additional API to let
callers implement their own policy if necessary
by making most platform clocks available and letting the caller pick
amongst them.
The PEP's suggested clocks are still expected to be available for the common
simple use cases.
To do this two facilities are needed:
an enumeration of clocks, and metadata on the clocks to enable the user to
evaluate their suitability.
The primary interface is a function make simple choices easy:
the caller can use ``time.get_clock(*flags)`` with some combination of flags.
This include at least:
* time.MONOTONIC: clock cannot go backward
* time.STEADY: clock rate is steady
* time.ADJUSTED: clock may be adjusted, for example by NTP
* time.HIGHRES: clock with the highest precision
It returns a clock object with a .now() method returning the current time.
The clock object is annotated with metadata describing the clock feature set;
its .flags field will contain at least all the requested flags.
time.get_clock() returns None if no matching clock is found and so calls can
be chained using the or operator. Example of a simple policy decision::
T = get_clock(MONOTONIC) or get_clock(STEADY) or get_clock()
t = T.now()
The available clocks always at least include a wrapper for ``time.time()``,
so a final call with no flags can always be used to obtain a working clock.
Example of flags of system clocks:
* QueryPerformanceCounter: MONOTONIC | HIGHRES
* GetTickCount: MONOTONIC | STEADY
* CLOCK_MONOTONIC: MONOTONIC | STEADY (or only MONOTONIC on Linux)
* CLOCK_MONOTONIC_RAW: MONOTONIC | STEADY
* gettimeofday(): (no flag)
The clock objects contain other metadata including the clock flags
with additional feature flags above those listed above, the name
of the underlying OS facility, and clock precisions.
``time.get_clock()`` still chooses a single clock; an enumeration
facility is also required.
The most obvious method is to offer ``time.get_clocks()`` with the
same signature as ``time.get_clock()``, but returning a sequence
of all clocks matching the requested flags.
Requesting no flags would thus enumerate all available clocks,
allowing the caller to make an arbitrary choice amongst them based
on their metadata.
Example partial implementation:
`clockutils.py <http://hg.python.org/peps/file/tip/pep-0418/clockutils.py>`_.
Working around operating system bugs?
-------------------------------------
Should Python ensure manually that a monotonic clock is truly
monotonic by computing the maximum with the clock value and the
previous value?
Since it's relatively straightforward to cache the last value returned
using a static variable, it might be interesting to use this to make
sure that the values returned are indeed monotonic.
* Virtual machines provide less reliable clocks.
* QueryPerformanceCounter() has known bugs (only one is not fixed yet)
Python may only work around a specific known operating system bug:
`KB274323`_ contains a code example to workaround the bug (use
GetTickCount() to detect QueryPerformanceCounter() leap).
Issues of a hacked monotonic function:
* if the clock is accidentally set forward by an hour and then back
again, you wouldn't have a useful clock for an hour
* the cache is not shared between processes so different processes
wouldn't see the same clock value
More information about the Python-Dev
mailing list