
Hi, I started to write the PEP 418 to clarify the notions of monotonic and steady clocks. The PEP is a draft and everyone is invited to contribute! http://www.python.org/dev/peps/pep-0418/ http://hg.python.org/peps/file/tip/pep-0418.txt Victor

time.steady() doesn't fit the benchmarking use case: it looks like we have to decide between stability and clock resolution. QueryPerformanceCounter() has a good resolution for benchmarking, but it is not monotonic and so GetTickCount64() would be used for time.steady(). GetTickCount64() is monotonic but has only a resolution of 1 millisecond. We might add a third new function which provides the most accurate clock with or without a known starting point. We cannot use QueryPerformanceCounter() to enhance time.time() resolution because it has an unknown starting point. Victor

On 3/26/2012 7:32 PM, Victor Stinner wrote:
I started to write the PEP 418 to clarify the notions of monotonic and steady clocks.
""" time.steady This clock advances at a steady rate relative to real time. It may be adjusted. """ Please do not call this "steady". If the clock can be adjusted, then it is not "steady" by any acceptable definition. I cannot fathom the utility of this function other than as a function that provides an automatic fallback from "time.monotonic()". More importantly: this definition of "steady" is in conflict with the C++0x definition of "steady" that is where you sourced this named from![1] """ time.steady(strict=False) falls back to another clock if no monotonic clock is not available or does not work, but it does never fail. """ As I say above, that is so far away from what "steady" implies that this is a misnomer. What you are describing is a best-effort clock, which sounds a lot more like the C++0x "high resolution" clock. """ time.steady(strict=True) raises OSError if monotonic clock fails or NotImplementedError if the system does not provide a monotonic clock """ What is the utility of "strict=True"? If I wanted that mode of operation, then why would I not just try to use "time.monotonic()" directly? At worst, it generates an "AttributeError" (although that is not clear from your PEP). What is the use case for "strict=True" that is not covered by your "time.monotonic()"? If you want to define new clocks, then I wish you would use the same definitions that C++0x is using. That is: system_clock = wall clock time monotonic_clock = always goes forward but can be adjusted steady_clock = always goes forward and cannot be adjusted high_resolution_clock = steady_clock || system_clock Straying from that is only going to create confusion. Besides that, the one use case for "time.steady()" that you give (benchmarking) is better served by a clock that follows the C++0x definition. As well, certain kinds of scheduling/timeouts would be better implemented with the C++0x definition for "steady" rather than the "monotonic" one and vice-versa. Rather, it seems you have a particular use-case in mind and have settled on calling that a "steady" clock despite how it belies its name. [1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3128.html#time.cloc... """ Objects of class steady_clock represent clocks for which values of time_point advance at a steady rate relative to real time. That is, the clock may not be adjusted. """ -- Scott Dial scott@scottdial.com

Note that the C++ standard deprecated monotonic_clock once they realized that there is absolutely no point in having a clock that jumps forward but not back, and that none of the operating systems implement such a thing -- instead they all implement a clock which doesn't jump in either direction. http://stackoverflow.com/questions/6777278/what-is-the-rationale-for-renamin... In other words, yes! +1! The C++ standards folks just went through the process that we're now going through, and if we do it right we'll end up at the same place they are: http://en.cppreference.com/w/cpp/chrono/system_clock """ system_clock represents the system-wide real time wall clock. It may not be monotonic: on most systems, the system time can be adjusted at any moment. It is the only clock that has the ability to map its time points to C time, and, therefore, to be displayed. steady_clock: monotonic clock that will never be adjusted high_resolution_clock: the clock with the shortest tick period available """ Note that we don't really have the option of providing a clock which is "monotonic but not steady" in the sense of "can jump forward but not back". It is a misunderstanding (doubtless due to the confusing name "monotonic") to think that such a thing is offered by the underlying platforms. We can choose to *call* it "monotonic", following POSIX instead of calling it "steady", following C++. Regards, Zooko

So does anyone care to dig into the libstd++/boost/windoze implementation to see how they each did steady_clock?

On 3/26/2012 10:59 PM, Matt Joiner wrote:
So does anyone care to dig into the libstd++/boost/windoze implementation to see how they each did steady_clock?
The Boost implementation can be summarized as: system_clock: mac = gettimeofday posix = clock_gettime(CLOCK_REALTIME) win = GetSystemTimeAsFileTime steady_clock: mac = mach_absolute_time posix = clock_gettime(CLOCK_MONOTONIC) win = QueryPerformanceCounter high_resolution_clock: * = { steady_clock, if available system_clock, otherwise } Whether or not these implementations meet the specification is an exercise left to the reader.. -- Scott Dial scott@scottdial.com

On Mar 26, 2012, at 10:26 PM, Zooko Wilcox-O'Hearn wrote:
This is why I don't like the C++ terminology, because it seems to me that the C++ standard makes incorrect assertions about platform behavior, and apparently they standardized it without actually checking on platform capabilities. The clock does jump forward when the system suspends. At least some existing implementations of steady_clock in C++ already have this problem, and I think they all might. I don't think they can fully fix it without kernel changes, either. On linux, see discussion of a possible CLOCK_BOOTTIME in the future. The only current way I know of to figure out how long the system has been asleep is to look at the wall clock and compare, and we've already gone over the problems with relying on the wall clock. Plus, libstdc++ gives you no portable way to get informed about system power management events, so you can't fix it even if you know about this problem, natch. Time with respect to power management state changes is something that the PEP should address fully, for each platform. On the other hand, hopefully you aren't controlling your python-based CNC laser welder from a laptop that you are closing the lid on while the beam is in operation. Not that the PEP shouldn't address it, but maybe it should just address it to say "you're on your own" and refer to a few platform-specific resources for correcting this type of discrepancy. (<https://developer.apple.com/library/mac/#qa/qa1340/_index.html>, <http://msdn.microsoft.com/en-us/library/aa394362.aspx>, <http://upower.freedesktop.org/docs/UPower.html#UPower::Sleeping>). -glyph

On Mar 27, 2012, at 3:17 AM, Glyph wrote:
I don't think they can fully fix it without kernel changes
I got really curious about this and went and did some research. With some really platform-specific hackery on every platform, you can mostly figure it out; completely on OS X and Windows, although (as far as I can tell) only partially on Linux and FreeBSD. I'm not sure if it's possible to make use of these facilities without a Twisted-style event-loop though. If anybody's interested, I recorded the results of my research in a comment on the Twisted ticket for this: <http://twistedmatrix.com/trac/ticket/2424#comment:26>. -glyph

Reading this discussion, my conclusion is that not only us are confused, but everyone is. I think therefore, that the way forward is to only expose underlying API functions, and pretty much have no intelligence at all. At a higher level, we have two different "desires" here. You may want a monotonic clock, or you may not care. You may want high resolution, or you might not care. Which one is more important is something only you know. Therefore, we must have, at the minimum, a function that returns the highest resolution monotonic clock possible, as well as a function that returns the highest resolution system/wall clock possible. We also need ways to figure out what the resolution is of these clocks. In addition to that, you may have the requirement that the monotonic clock also should not be able to jump forward, but if I understand things correctly, most current OS's will not guarantee this. You may also have the requirement that the clock not only does not jump forward, but that it doesn't go faster or slower. Some clock implementations will speed up or slow down the monotonic clock, without jumps, to sync up with the wall clock. It seems only Unix provides a monotonic clock (CLOCK_MONOTONIC_RAW) that does not get adjusted at all. Now between all these requirements, only you know which one is more important? Do you primarily want a raw monotonic clock, and secondarily high resolution, or is the resolution more important than it being monotonic? (Because if you need a high resolution, you are usually measuring small timeframes, and the clock is more unlikely to be adjusted, for example). Since there is no obvious "A is better than B that is better than C" we first of all have to expose the underlying API's somehow, to allow people to make their own decisions. Secondly, since apparently not only python-dev, but many others as well, are a bit confused on this complex issue, I'm not sure we can provide any high-level functions that makes a best choice. As such the proposed time.monotonic() to get the monotonic clock on the various systems makes a lot of sense to me. It should get the highest resolution available on the system. Get GetTickCount64() of available on Windows, else GetTickCount(). The function could have a raw=False parameter to select between clock_gettime(CLOCK_MONOTONIC) and clock_gettime(CLOCK_MONOTONIC_RAW) on Unix, and it would get mach_absolute_time() on OS X. If no monotonic clock is available, it should raise an error.The same if you pass in raw=True and there is no monotonic clock that has no adjustments available. In the same vein, time.time() should provide the highest resolution system clock/wall clock available. We also need functions or attributes to get the resolution of these clocks. But a time.steady() that tries to get a "best case" doesn't make sense at this time, as apparently nobody knows what a best case is, or what it should be called, except that it should apparently not be called steady(). Since monotonic() raises an error if there is no monotonic clock available, implementing your own fallback is trivial in any case. //Lennart

On Tue, Mar 27, 2012 at 7:03 PM, Lennart Regebro <regebro@gmail.com> wrote:
+1 from me to Lennart's suggestion of mostly just exposing time.monotonic() without trying to get too clever. Applications that need a truly precise time source should *never* be reading it from the host OS (one fairly good solution can be to read your time directly from an attached GPS device). However, I think Victor's right to point out that the *standard library* needs to have a fallback to maintain backwards compatibility if time.monotonic() isn't available, and it seems silly to implement the same fallback logic in every module where we manipulate timeouts. As I concur with others that time.steady() is a thoroughly misleading name for this concept, I suggest we encapsulate the "time.monotic if available, time.time otherwise" handling as a "time.try_monotic()" function. That's simple clear and explicit: try_monotic() tries to use the monotic clock if it can, but falls back to time.time() rather than failing entirely if no monotonic clock is available. This is essential for backwards compatibility when migrating any current use of time.time() over to time.monotic(). Yes the monotonic clock is *better* for many use cases (including timeouts), but you'll usually be OK with the non-monotonic clock, too (particularly if that's what you were using anyway in earlier versions). After all, we've survived this long using time.time() for our timeout calculations, and bugs associated with clock adjustments are a rather rare occurrence. Third party libraries that need to support earlier Python versions can then implementation their own fallback logic (since they couldn't rely on time.try_monotonic being available either). The 3.3 time module would then be left with three interfaces: time.time() # Highest resolution timer available time.monotonic() # I'm not yet convinced we need the "raw" parameter but don't much mind either way time.try_monotonic() # Monotonic is preferred, but non-monotonic presents a tolerable risk Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2012-03-27, at 9:23 AM, Nick Coghlan wrote:
time.try_monotonic() # Monotonic is preferred, but non-monotonic presents a tolerable risk
This function seems unnecessary. It's easy to implement it when required in your application, hence I don't think it is worth adding to the stdlib. - Yury

I renamed time.steady() to time.try_monotonic() in the PEP. It's a temporary name until we decide what to do with this function.
How about get rid of it? Also monotonic should either not exist if it's not available, or always guarantee a (artificially) monotonic value. Finding out that something is already known to not work shouldn't require a call and a faked OSError.

On 27/03/2012 20:18, Matt Joiner wrote:
Also monotonic should either not exist if it's not available, or always guarantee a (artificially) monotonic value.
What do you mean by "(artificially) monotonic value"? Should Python workaround OS bugs by always returning the maximum value of the clock?
Finding out that something is already known to not work shouldn't require a call and a faked OSError.
What do you mean? time.monotonic() is not implemented if the OS doesn't provide any monotonic clock (e.g. if clock_gettime(CLOCK_MONOTONIC) is missing on UNIX). OSError is only raised when the OS returns an error. There is no such "faked OSError". Victor

Matt, we need the fallback behaviour in the stdlib so we can gracefully degrade the stdlib's *own* timeout handling back to the 3.2 status quo when there is no monotic clock available. It is *not* acceptable for the Python 3.3 stdlib to only work on platforms that provide a monotonic clock. Since duplicating that logic in every module that handles timeouts would be silly, it makes sense to provide an obvious way to do it in the time module. -- Sent from my phone, thus the relative brevity :)

I don't think that Python should workaround OS issues, but document them correctly. I started with this sentence for time.monotonic(): "The monotonic clock may stop while the system is suspended." I don't know exactly how clocks behave with system suspend. Tell me if you have more information.
I will read these links and maybe add them to the PEP. Victor

I mentioned the strict=True API in the PEP just to list all propositions, but the PEP only proposes time.monotonic() and time.try_monotonic(), no the flags API.
At worst, it generates an "AttributeError" (although that is not clear from your PEP).
I tried to mention when a function is always available or not always available. Is it better in the last version of the PEP?
I tried to follow these names in the PEP. I don't propose steady_clock because I don't see exactly which clocks would be used to implement it, nor if we need to provide monotonic *and* steady clocks. What do you think?
I added a time.hires() clock to the PEP for the benchmarking/profiling use case. This function is not always available and so a program has to fallback manually to another clock. I don't think that it is an issue: Python programs already have to choose between time.clock() and time.time() depending on the OS (e.g. timeit module and pybench program).
Sorry, I don't understand. Which kind of scheduling/timeouts? The PEP is still a draft (work-in-progress). Victor

On 27/03/2012 18:45, Victor Stinner wrote:
It is this always-having-to-manually-fallback-depending-on-os that I was hoping your new functionality would avoid. Is time.try_monotonic() suitable for this usecase? Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Scott wrote: << The Boost implementation can be summarized as: system_clock: mac = gettimeofday posix = clock_gettime(CLOCK_REALTIME) win = GetSystemTimeAsFileTime steady_clock: mac = mach_absolute_time posix = clock_gettime(CLOCK_MONOTONIC) win = QueryPerformanceCounter high_resolution_clock: * = { steady_clock, if available system_clock, otherwise } >> I read again the doc of the QElapsedTimer class of the Qt library. So Qt and Boost agree to say that QueryPerformanceCounter() *is* monotonic. I was confused because of a bug found in 2006 in Windows XP on multicore processors. QueryPerformanceCounter() gave a different value on each core. The bug was fixed in Windows and is known as KB896256 (I already added a link to the bug in the PEP).
If QueryPerformanceCounter() is monotonic, the API can be simplified to: * time.time() = system clock * time.monotonic() = monotonic clock * time.hires() = monotonic clock or fallback to system clock time.hires() definition is exactly what I was trying to implement with "time.steady(strict=True)" / "time.try_monotonic()". -- Scott> monotonic_clock = always goes forward but can be adjusted Scott> steady_clock = always goes forward and cannot be adjusted I don't know if the monotonic clock should be called time.monotonic() or time.steady(). The clock speed can be adjusted by NTP, at least on Linux < 2.6.28. I don't know if other clocks used by my time.monotonic() proposition can be adjusted or not. If I understand correctly, time.steady() cannot be implemented using CLOCK_MONOTONIC on Linux because CLOCK_MONOTONIC can be adjusted? Does it really matter if a monotonic speed is adjusted? Victor

On Mar 28, 2012 8:38 AM, "Victor Stinner" <victor.stinner@gmail.com> wrote:
and Boost agree to say that QueryPerformanceCounter() *is* monotonic.
I was confused because of a bug found in 2006 in Windows XP on multicore
processors. QueryPerformanceCounter() gave a different value on each core. The bug was fixed in Windows and is known as KB896256 (I already added a link to the bug in the PEP).
"time.steady(strict=True)" / "time.try_monotonic()".
time.steady(). The clock speed can be adjusted by NTP, at least on Linux < 2.6.28. Monotonic. It's still monotonic if it is adjusted forward, and that's okay.
I don't know if other clocks used by my time.monotonic() proposition can
be adjusted or not.
If I understand correctly, time.steady() cannot be implemented using
CLOCK_MONOTONIC on Linux because CLOCK_MONOTONIC can be adjusted?
http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com

On Wed, Mar 28, 2012 at 10:36 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
Please don't call the fallback version "hires" as it suggests it may be higher resolution than time.time() and that's completely the wrong idea. If we're simplifying the idea to only promising a monotonic clock (i.e. will never go backwards within a given process, but may produce the same value for an indefinite period, and may jump forwards by arbitrarily large amounts), then we're back to being able to enforce monotonicity even if the underlying clock jumps backwards due to system clock adjustments. Specifically: time.time() = system clock time._monotonic() = system level monotonic clock (if it exists) time.monotonic() = clock based on either time._monotonic() (if available) or time.time() (if not) that enforces monotonicity of returned values. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Georg Brandl wrote:
Precisely. I always read "hires" as the verb hires (as in "he hires a car to go on holiday") rather than HIgh RESolution. -1 on hires, it's a horrible name. And misleading as well, because on Linux, it isn't any more high res than time.time(). +1 on Nick's suggestion of try_monotonic. It is clear and obvious and doesn't mislead. I don't have an opinion as to what the implementation of try_monotonic should be. Whether it should fall back to time.time, time.clock, or something else, I don't know. But it is a clear and obvious solution for the use-case of "I prefer the monotonic clock, if it is available, otherwise I'll take my chances with a best-effect clock." -- Steven

On Wed, 28 Mar 2012 23:05:59 +1100, Steven D'Aprano <steve@pearwood.info> wrote:
+1 on Nick's suggestion of try_monotonic. It is clear and obvious and doesn't mislead.
How about "monotonicest". (No, this is not really a serious suggestion.) However, time.steadiest might actually work. --David

Why would it be a wrong idea? On Windows, time.monotonic() frequency is at least 1 MHz (can be GHz if it uses your CPU TSC) whereas time.time() is only updated each millisecond at the best case (each 15 ms by default if I remember correctly). On UNIX, CLOCK_MONOTONIC has the same theorical resolution than CLOCK_REALTIME (1 nanosecond thanks to the timespec structure) and I expect the same accuracy. On Mac, I don't know if mach_absolute_time() is more or as accurate than time.time(). time.hires() uses time.monotonic() if available, so if time.monotonic() has an higher resolution than time.time(), time.hires() can also be called a high-resolution clock. In practice, time.monotonic() is available on all modern platforms.
I don't know any monotonic clock jumping "forwards by arbitrarily large amounts". Linux can change CLOCK_MONOTONIC speed, but NTP doesn't "jump".
Do you know a monotonic clock that goes backward? If yes, Python might workaround the clock bug directly in time.monotonic(). But I would prefer to *not* workaround OS bugs. Victor

On Wed, Mar 28, 2012 at 6:40 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
If I understood Glyph's explanation correctly, then if your application is running in a VM and the VM is getting its clock data from the underlying hypervisor, then suspending and resuming the VM may result in forward jumping of the monotonic clocks in the guest OS. I believe suspending and hibernating may cause similar problems for even a non-virtualised OS that is getting its time data from a real-time clock chip that keeps running even when the main CPU goes to sleep. (If I *misunderstood* Glyph's explanation, then he may have only been talking about the latter case) Monotonicity is fairly easy to guarantee - you just remember the last value you returned and ensure you never return a lower value than that for the lifetime of the process. The only complication is thread synchronisation, and the GIL (or a dedicated lock for Jython/IronPython) can deal with that. Steadiness, on the other hand, requires a real world time reference and is thus really the domain of specialised hardware like atomic clocks and GPS units rather than software that can be suspended and resumed later without changing its internal state. There's a reason comms station operators pay substantial chunks of money for time & frequency reference devices [1]. This is why I now think we only need one new clock function: time.monotonic(). It will be the system monotonic clock if one is available, otherwise it will be our own equivalent wrapper around time.time() that just caches the last value returned to ensure the result never goes backwards. With time.monotonic() guaranteed to always be available, there's no need for a separate function that falls back to an unconditioned time.time() result. Regards, Nick. [1] For example: http://www.symmetricom.com/products/gps-solutions/gps-time-frequency-receive... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Mar 28, 2012 at 7:36 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
As I said, I think the caching idea is bad. We may have to settle for semantics that are less than perfect -- presumably if you are doing benchmarking you just have to throw away a bad result that happened to be affected by a clock anomaly, and if you are using timeouts, retries are already part of life.
I would love to have only one new clock function in 3.3.
-- --Guido van Rossum (python.org/~guido)

On Thu, Mar 29, 2012 at 12:42 AM, Guido van Rossum <guido@python.org> wrote:
I agree caching doesn't solve the problems that are solved by an OS level monotonic clock, but falling back to an unmodifided time.time() result instead doesn't solve those problems either. Falling back to time.time() just gives you the status quo: time may jump forwards or backwards by an arbitrary amount between any two calls. Cached monotonicity just changes the anomalous modes to be time jumping forwards, or time standing still for an extended period of time. The only thing the caching provides is that it becomes a reasonable fallback for a function called time.monotonic() - it *is* a monotonic clock that meets the formal contract of the function, it's just nowhere near as good or effective as one the OS can provide. Forward jumping anomalies aren't as harmful, are very hard to detect in the first place and behave the same regardless of the presence of caching, so the interesting case to look at is the difference in failure modes when the system clock jumps backwards. For benchmarking, a caching clock will produce a zero result instead of a negative result. Zeros aren't quite as obviously broken as negative numbers when benchmarking, but they're still sufficiently suspicious that most benchmarking activities will flag them as anomalous. If the jump back was sufficiently small that the subsequent call still produces a higher value than the original call, then behaviour reverts to being identical. For timeouts, setting the clock back means your operation will take longer to time out than you expected. This problem will occur regardless of whether you were using cached monotonicity (such that time stands still) or the system clock (such that time actually goes backwards). In either case, your deadline will never be reached until the backwards jump has been cancelled out by the subsequent passage of time. I want the standard library to be able to replace its time.time() calls with time.monotonic(). The only way we can do that without breaking cross-platform compatibility is if time.monotonic() is guaranteed to exist, even when the platform only provides time.time(). A dumb caching fallback implementation based on time.time() is the easiest way to achieve that withou making a complete mockery of the "monotonic()" name. There is then a *different* use case, which is 3.3+ only code which wants to fail noisily when there's no OS level monotonic support - the application developer really does want to fail *immediately* if there's no OS level monotonic clock available, instead of crossing your fingers and hoping you don't hit a clock adjustment glitch (crossing your fingers has, I'll point out, been the *only* option for all previous versions of Python, so it clearly can't be *that* scary a prospect). So, rather than making time.monotonic() something that the *standard library can't use*, I'd prefer to address that second use case by exposing the OS level monotonic clock as time.os_monotonic() only when it's available. That way, the natural transition for old time.time() based code is to time.monotonic() (with no cross-platform support implications), but time.os_monotonic() also becomes available for the stricter use cases. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Mar 28, 2012 at 8:08 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
TBH, I don't think like this focus on monotonicity as the most important feature.
Agreed.
So for benchmarking we don't care about jumps, really, and the caching version is slightly less useful.
Where in the stdlib do we actually calculate timeouts instead of using the timeouts built into the OS (e.g. select())? I think it would be nice if we could somehow use the *same* clock as the OS uses to implement timeouts.
I want the standard library to be able to replace its time.time() calls with time.monotonic().
Where in the stdlib? (I'm aware of threading.py. Any other places?)
Yeah, so maybe it's a bad name. :-)
I'd be happier if the fallback function didn't try to guarantee things the underlying clock can't guarantee. I.e. I like the idea of having a function that uses some accurate OS clock if one exists but falls back to time.time() if not; I don't like the idea of that new function trying to interpret the value of time.time() in any way. Applications that need the OS clock's guarantees can call it directly. We could also offer something where you can introspect the properties of the clock (or clocks) so that an app can choose the best clock depending on its needs. To summarize my problem with the caching idea: take a simple timeout loop such as found in several places in threading.py. def wait_for(delta, eps): # Wait for delta seconds, sleeping eps seconds at a time deadline = now() + delta while now() < deadline: sleep(eps) If the now() clock jumps backward after the initial call, we end up waiting too long -- until either the clock jumps forward again or until we've made up the difference. If the now() clock jumps forward after the initial call, we end up waiting less time, which is probably not such a big problem (though it might). But now consider a caching clock, and consider that the system clock made a jump backwards *before* this function is called. The cache prevents us from seeing it, so the initial call to now() returns the highest clock value seen so far. And until the system clock has caught up with that, now() will return the same value over and over -- so WE STILL WAIT TOO LONG. My conclusion: you can't win this game by forcing the clock to return a monotonic value. A better approach might be to compute how many sleep(eps) calls we're expected to make, and to limit the loop to that -- although sleep() doesn't make any guarantees either about sleeping too short or too long. Basically, if you do sleep(1) and find that your clock didn't move (enough), you can't tell the difference between a short sleep and a clock that jumped back. And if your clock moved to much, you still don't know if the problem was with sleep() or with your clock. -- --Guido van Rossum (python.org/~guido)

time.timeout_clock? Everyone knows what that will be for and we won't have to make silly theoretical claims about its properties and expected uses. If no one else looks before I next get to a PC I'll dig up the clock/timing source used for select and friends, and find any corresponding syscall that retrieves it for Linux.

On Thu, Mar 29, 2012 at 1:47 AM, Guido van Rossum <guido@python.org> wrote:
Where in the stdlib? (I'm aware of threading.py. Any other places?)
Victor had at least one other example. multiprocessing, maybe? I believe the test suite may still have a few instances as well.
Ouch. OK, I'm convinced the caching fallback is worse than just falling back to time.time() directly, which means the naming problem needs to be handled another way.
With your point about the problem with the naive caching mechanism acknowledged, I think we can safely assign time.monotonic() as the name of the OS provided monotonic clock. That means choosing a name for the version that falls back to time() if monotonic() isn't available so it can be safely substituted for time.time() without having to worry about platform compatibility implications. I don't like Victor's current "hires" (because it doesn't hint at the fallback behaviour, may actually be the same res as time.time() and reads like an unrelated English word). My own suggestion of "try_monotic()" would get the job done, but is hardly going to win any API beauty contests. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Mar 29, 2012 at 1:14 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
What's wrong with "time.time()" again? As documented in http://docs.python.org/py3k/library/time.html it makes no guarantees, and specifically there is *no* guarantee that it will ever behave *badly*<wink/>. Of course, we'll have to guarantee that, if a badly-behaved clock is available, users can get access to it, so call that time._time().

I'm not sure I understand your suggestion correctly, but replacing time.time() by time.monotonic() with fallback won't work, because time.monotonic() isn't wall-clock time: it can very well use an arbitrary reference point (most likely system start-up time). As for the hires() function, since there's no guarantee whatsoever that it does provide a better resolution than time.time(), this would be really misleading IMHO.

Where in the stdlib do we actually calculate timeouts instead of using the timeouts built into the OS (e.g. select())?
At least in threading and queue modules. The common use case is to retry a function with a timeout if the syscall was interrupted by a signal (EINTR error). The socket module and _threading.Lock.acquire() implement such retry loop using the system clock. They should use a monotonic clock instead.
I think it would be nice if we could somehow use the *same* clock as the OS uses to implement timeouts.
On Linux, nanosleep() uses CLOCK_MONOTONIC whereas POSIX suggests CLOCK_REALTIME. Some functions allow to choose the clock, like pthread locks or clock_nanosleep().
We may workaround some OS known bugs like: http://support.microsoft.com/?id=274323 The link contains an example how to workaround the bug. The idea of the workaround is to use two different monotonic clocks to detect leaps, with one trusted clock (GetTickCount) and one untrusted clock having an higher resolution (QueryPerformanceCounter). I don't think that the same algorithm is applicable on other OSes because other OSes usually only provide one monotonic clock, sometimes though different API. Victor

On 2012-03-28, at 10:36 AM, Nick Coghlan wrote:
As I said in my previous mail - I don't think we should ever do that. Time may jump back and forth, and with your approach it will result in monotonic() being completely unusable. If time jumps back for N minutes, or years, that leads to completely broken expectations for timeouts for N minutes or years correspondingly (and that's just the timeouts case, I'm sure that there are much more critical time-related use-cases.) If monotonic() will utilize such hack, you add nothing usable in stdlib. Every serious framework or library will have to re-implement it using only OS-level functions, and *FAIL* if the OS doesn't support monotonic time. Fail, because such framework can't guarantee that it will work correctly. So I think time module should have only one new function: monotonic(), and this function should be only available if OS provides the underlying functionality. No need for steady(), try_monotonic() and other hacks. Each module can decide if its dependancy on monotonic is critical or not, and if it is not, you can always have: try: from time import monotonic as _time except ImportError: from time import time as _time That's how lots of code is written these days, like using 'epoll' if available, and fallback to 'select' if not. Why don't you try to abstract differences between them in the standard library? So I see no point in adding some loose abstractions to the stdlib now. - Yury

On 3/27/2012 8:36 PM, Victor Stinner wrote:
You are right that CLOCK_MONOTONIC can be adjusted, so the Boost implementation is wrong. I'm not sure that CLOCK_MONOTONIC_RAW is right either due to suspend -- there doesn't appear to be a POSIX or Linux clock that is defined that meets the "steady" definition. I am not familiar enough with Windows or Mac to know for certain whether the Boost implementation has the correct behaviors either. With that in mind, it's certainly better that we just provide time.monotonic() for now. If platform support becomes available, then we can expose that as it becomes available in the future. In other words, at this time, I don't think "time.steady()" can be implemented faithfully for any platform so lets just not have it at all. In that case, I don't think time.try_monotonic() is really needed because we can emulate "time.monotonic()" in software if the platform is deficient. I can't imagine a scenario where you would ask for a monotonic clock and would rather have an error than have Python fill in the gap with an emulation. -- Scott Dial scott@scottdial.com

The term "adjusted" should be clarified. A clock can be adjusted by setting its counter (e.g. setting the system date and time) or by changing temporary its frequency (to go faster or slower). Linux only adjusts CLOCK_MONOTONIC frequency but the clock is monotonic because it always goes forward. The monotonic property can be described as: t1=time.monotonic() t2=time.monotonic() assert t2 >= t1
time.hires() is needed when the OS doesn't provide any monotonic clock and because time.monotonic() must not use the system clock (which can jump backward). As I wrote, I don't think that Python should workaround OS bugs. If the OS monotonic clock is not monotonic, the OS should be fixed.
Sorry, I don't understand what you mean with "fill in the gap with an emulation". You would like to implement a monotonic clock based on the system clock? Victor

On 3/28/2012 4:48 AM, Victor Stinner wrote:
I agree. The point I was making is that implication of "steady" is that (t2-t1) is the same (given that t2 and t1 occur in time at the same relative moments), which is a guarantee that I don't see any platform providing currently. Any clock that can be "adjusted" in any manner is not going to meet the "steady" criterion.
I sympathize with this, but if the idea is that the Python stdlib should use time.monotonic() for scheduling, then it needs to always be available. Otherwise, we are not going to use it ourselves, and what sort of example is that to set?
If "time.monotonic()" is only sometimes available, then I don't see the added clock being anything more than an amusement. (In this case, I'd rather just use clock_gettime() and friends directly, because I have to be platform aware anyways.) What developers want is a timer that is useful for scheduling things to happen after predictable interval in the future, so we should give them that to the best of our ability. -- Scott Dial scott@scottdial.com

Victor, I have completely lost track of the details of this discussion. Could you (with help from others who contributed) try to compile a table showing, for each platform (Windows/Mac/Linux/BSD) which clocks (or variations) we are considering, and for each of those: - a link for the reference documentation - what their typical accuracy is (barring jumps) - what they do when the "civil" time is made to jump (forward or back) by the user - how they are affected by small tweaks to the civil time by NTP - what they do if the system is suspended and resumed - whether they can be shared between processes running on the same machine - whether they may fail or be unsupported under some circumstances I have a feeling that if I saw such a table it would be much easier to decide. I assume much of this has already been said at one point in this thread, but it's impossible to have an overview at the moment. If someone has more questions they'd like to see answered please add to the list. -- --Guido van Rossum (python.org/~guido)

On Wed, Mar 28, 2012 at 8:56 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
Completely unintuitive and unnecessary. With the GIL taking care of synchronisation issues, we can easily coerce time.time() into being a monotonic clock by the simple expedient of saving the last returned value: def _make_monotic: try: # Use underlying system monotonic clock if we can return _monotonic except NameError: _tick = time() def monotic(): _new_tick = time() if _new_tick > _tick: _tick = _new_tick return _tick monotonic = _make_monotonic() Monotonicity of the result is thus ensured, even when using time.time() as a fallback. If using the system monotonic clock to get greater precision is acceptable for an application, then forcing monotonicity shouldn't be a problem either. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Mar 29, 2012 at 12:27 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
You have to keep in mind the alternative here: falling back to an *unconditioned* time.time() value (which is the status quo, and necessary to preserve backwards compatibility). That will break just as badly in that scenario and is precisely the reason that the OS level monotonic functionality is desirable in the first place. I'd be quite happy with a solution that made the OS level monotonic clock part of the public API, with the caveat that it may not be available. Then the necessary trio of functions would be: time.time(): existing system clock, always available time.os_monotonic(): OS level monotonic clock, not always available time.monotonic(): always available, same as os_monotonic if it exists, otherwise uses a time() based emulation that may not be consistent across processes and may "mark time" for extended periods if the underlying OS clock is forced to jump back a long way. I think that naming scheme is more elegant than using monotonic() for the OS level monotonicity and try_monotonic() for the fallback version, but I'd be OK with the latter approach, too. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2012-03-28, at 10:45 AM, Nick Coghlan wrote:
Well, my argumentation is that you either have some code that depends on monotonic time and can't work without it, or you have a code that can work with any time (and only precision matters). Maybe I'm wrong.
I still don't like this 'emulation' idea. Smells bad for standard lib. Big -1 on this approach. - Yury

On Thu, Mar 29, 2012 at 1:02 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
You're wrong. The primary use case for the new time.monotonic() function is to replace *existing* uses of time.time() in the standard library (mostly related to timeouts) that are currently vulnerable to clock adjustment related bugs. This real, concrete use case has been lost in some of the abstract theoretical discussions that have been going on this thread. We can't lose sight of the fact that using a system clock that is vulnerable to clock adjustment bugs to handle timeouts and benchmarking in Python has worked just fine for 20+ years. Using a monotonic clock instead is *better*, but it's far from essential, since clock adjustments that are big enough and poorly timed enough to cause real problems are fortunately a very rare occurrence. So, the primary use case is that we want to replace many of the time.time() calls in the standard library with time.monotonic() calls. To avoid backwards compatibility problems in the cross-platform support, that means time.monotonic() *must be available on every platform that currently provides time.time()*. This is why Victor's original proposal was that time.monotonic() simply fall back to time.time() if there was no OS level monotonic clock available. The intended use cases are using time.time() *right now* and have been doing so for years, so it is clearly an acceptable fallback for those cases. People (rightly, in my opinion) objected to the idea of time.monotonic() failing to guarantee monotonicity, thus the proposal to enforce at least a basic level of monotonicity through caching of the last returned value. I agree completely that this dumb caching solution doesn't solve any of the original problems with time.time() that make a time.monotonic() function desirable, but it isn't meant to. It's only meant to provide graceful degradation to something that is *no worse than the current behaviour when using time.time() in Python 3.2* while still respecting the property of monotonicity for the new API. Yes, it's an ugly hack, but it is a necessary fallback to avoid accidental regressions in our cross-platform support. For the major platforms (i.e. *nix, Mac OS X, Windows), there *will* be an OS level monotonic clock available, thus using time.monotonic() will have the desired effect of protecting from clocks being adjusted backwards. For other platforms, the behaviour (and vulnerabilities) will be essentially unchanged from the Python 3.2 approach (i.e. using time.time() with no monotonicity guarantees at all). However, some 3.3+ applications may want to be stricter about their behaviour and either bail out completely or fall back to an unfiltered time.time() call if an OS-level monotonic clock is not available. For those, it makes sense to expose time.os_monotonic() directly (and only if it is available), thus allowing those developers to make up their own mind instead of accepting the cross-platform fallback in time.monotonic(). Yes, you can get the exact same effect with the "monotonic()" and "try_monotonic()" naming scheme, but why force the standard library (and anyone else wanting to upgrade from time.time() without harming cross-platform support) to use such an ugly name when the "os_monotonic" and "monotonic" naming scheme provides a much neater alternative? Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2012-03-28, at 11:35 AM, Nick Coghlan wrote:
OK. I got your point. And also I've just realized what I dislike about the way you want to implement the fallback. The main problem is that I treat the situation when time jumps backward as an exception, because, again, if you have timeouts you may get those timeouts to never be executed. So let's make the "try_monotonic()" function (or whatever name will be chosen) this way (your original code edited): def _make_monotic(): try: # Use underlying system monotonic clock if we can return _monotonic except NameError: _tick = time() def monotic(): nonlocal _time _new_tick = time() if _new_tick <= _tick: raise RuntimeError('time was adjusted backward') _tick = _new_tick return _new_tick return monotonic try_monotonic = _make_monotonic() At least this approach tries to follow some of the python's zen. - Yury

On Wed, Mar 28, 2012 at 7:17 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
That's a pretty obvious trick. But why don't the kernels do this if monotonicity is so important? I'm sure there are also downsides, e.g. if the clock is accidentally set forward by an hour and then back again, you wouldn't have a useful clock for an hour. And the cache is not shared between processes so different processes wouldn't see the same clock value (I presume that most of these clocks have state in the kernel that isn't bound to any particular process -- AFAIK only clock() does that, and only on Unixy systems). -- --Guido van Rossum (python.org/~guido)

On 3/28/2012 10:29 AM, Guido van Rossum wrote:
What makes you think that isn't already true? I don't know what platforms that CPython compiles for that *won't* have one of the aforementioned functions available that provide a *real* monotonic clock. Surely, any platform that doesn't didn't recognize the need for it, or they would just provide a monotonic clock. That is to say, if you are a POSIX compliant system, then there is no reason to break gettimeofday() and friends when you can just implement CLOCK_MONOTONIC proper (even if it's just a trick like Nick's). I think the PEP should enumerate what platforms that CPython supports that will not benefit from a real monotonic clock. I think the number of platforms will be such a minority that the emulation makes sense. Practicality beats purity, and all. -- Scott Dial scott@scottdial.com

Nick Coghlan wrote:
Here's a version that doesn't suffer from the flaw of returning a long stream of constant values when the system clock jumps backwards a significant amount: class MockTime: def __init__(self): self.ticks = [1, 2, 3, 4, 2, 3, 4, 5, 7, 3, 4, 6, 7, 8, 8, 9] self.i = -1 def __call__(self): self.i += 1 return self.ticks[self.i] time = MockTime() _prev = _prev_raw = 0 def monotonic(): global _prev, _prev_raw raw = time() delta = max(0, raw - _prev_raw) _prev_raw = raw _prev += delta return _prev And in use:
[monotonic() for i in range(16)] [1, 2, 3, 4, 4, 5, 6, 7, 9, 9, 10, 12, 13, 14, 14, 15]
Time: [1, 2, 3, 4, 2, 3, 4, 5, 7, 3, 4, 6, 7, 8, 8, 9] Nick: [1, 2, 3, 4, 4, 4, 4, 5, 7, 7, 7, 7, 7, 8, 8, 9] Mine: [1, 2, 3, 4, 4, 5, 6, 7, 9, 9, 10, 12, 13, 14, 14, 15] Mine will get ahead of the system clock each time it jumps back, but it's a lot closer to the ideal of a *strictly* monotonically increasing clock. Assuming that the system clock will never jump backwards twice in a row, the double-caching version will never have more than two constant values in a row. -- Steven

On Wed, Mar 28, 2012 at 12:56, Victor Stinner <victor.stinner@gmail.com> wrote:
There is time.hires() if you need a monotonic clock with a fallback to the system clock.
Does this primarily give a high resolution clock, or primarily a monotonic clock? That's not clear from either the name, or the PEP. //Lennart

Does this primarily give a high resolution clock, or primarily a monotonic clock? That's not clear from either the name, or the PEP.
I expect a better resolution from time.monotonic() than time.time(). I don't have exact numbers right now, but I began to document each OS clock in the PEP. Victor

On Wed, Mar 28, 2012 at 23:40, Victor Stinner <victor.stinner@gmail.com> wrote:
Sure. And for me that means that time.hires() would give a high resolution version of time.time(). Ie, not monotonic, but wall clock. The question then is why time.time() doesn't give that resolution from the start. It seems to me we need three functions: One to get the wall clock, one to get a monotonic clock, and one that falls back if no monotonic clock is available. Both time.time() and time.monotonic() should give the highest resolution possible. As such, time.hires() seems pointless. //Lennart

The overview of the different monotonic clocks was interesting, because only one of them is adjusted by NTP, and that's the unix CLOCK_MONOTONIC. Hence we don't need a raw=False flag, which I previously suggested, we only need to not use CLOCK_MONOTONIC (which the PEP psuedo-code indeed also does not do, so that's all good). That means I think the PEP is fine now, if we rename highres(). time.time() already gets the highest resolution clock it can. Hence a highres() is confusing as the name implies that it returns a higher resolution clock than time.time(). And the name does not in any way indicate that the returned clock might be monotonic. try_monotonic() seems the obvious choice, since that's what it actually does. //Lennart

On Fri, Mar 30, 2012 at 12:01 PM, Lennart Regebro <regebro@gmail.com> wrote:
Right on.
That means I think the PEP is fine now, if we rename highres(). time.time() already gets the highest resolution clock it can.
No, time.time() is the clock that can be mapped to and from "civil time". (Adjustments by NTP and the user notwithstanding.) The other clocks have a variable epoch and do not necessarily tick with a constant rate (e.g. they may not tick at all while the system is suspended).
I am still unhappy with the two names, but I'm glad that we're this close. We need two new names; one for an OS-provided clock that is "monotonic" or "steady" or whatever you want to call it, but which may not exist on all systems (some platforms don't have it, some host may not have it even though the platform generally does have it). The other name is for a clock that's one or the other; it should be the OS-provided clock if it exists, otherwise time.time(). Most code should probably use this one, so perhaps its name should be the shorter one. C++ calls these steady_clock and high_resolution_clock, respectively. But it also calls the civil time clock system_clock, so perhaps we shouldn't feel to bound by it (except that we *shouldn't* call something steady if it isn't). I still think the name "monotonic" give the wrong impression; I would be happy calling it steady. But for the other, I'm still at a loss, and that name is the most important one. We can't call it steady because it isn't always. highres or hires sounds awkward; try_monotonic or try_steady are even more awkward. I looked in an online thesaurus and here's a list of what it gave: Big Ben, alarm, chroniker, chronograph, chronometer, digital watch, hourglass, metronome, pendulum, stopwatch, sundial, tattler, tick-tock, ticker, timekeeper, timemarker, timepiece, timer, turnip, watch I wonder if something with tick would work? (Even though it returns a float. :-) If all else fails, I'd go with turnip. -- --Guido van Rossum (python.org/~guido)

On Fri, 30 Mar 2012 12:40:25 -0700, Guido van Rossum <guido@python.org> wrote:
We could call it "alice"[*]: sometimes it goes fast, sometimes it goes slow, sometimes it even goes backward, but it does try to tell you when you are late. --David [*] 'whiterabbit' would be more descriptive, but that's longer than turnip.

On 2012-03-30, at 3:40 PM, Guido van Rossum wrote:
I still think the name "monotonic" give the wrong impression; I would be happy calling it steady.
Simple google search comparison shows that people ask about 'monotonic' clock in python, not 'steady'. How about following Nick's (if I recall correctly) proposal of calling the OS function - '_monotonic', and a python wrapper - 'monotonic'? And one more question: what do you think about introducing a special check, that will ensure that our python implementation of 'monotonic', in case of fallback to 'time.time()', raises an exception if time suddenly goes backward? - Yury

Guido van Rossum wrote:
"hires" is a real English word, the present tense verb for engaging the service or labour of someone or something in return for payment, as in "he hires a gardener to mow the lawn". Can we please eliminate it from consideration? It is driving me slowly crazy every time I see it used as an abbreviation for high resolution.
I can't tell if you are being serious or not. For the record, "turnip" in this sense is archaic slang for a thick pocket watch. -- Steven

On Sat, Mar 31, 2012 at 02:26, Steven D'Aprano <steve@pearwood.info> wrote:
If I understand this correctly, the most common use for this function is when to time things. It will give you the best source available for timers, but it doesn't guarantee that it is steady or monotonic or high resolution or anything. It is also not the time, as it's not reliable as a wall-clock. So, how about time.timer()? //Lennart

On Sat, Mar 31, 2012 at 8:27 AM, Lennart Regebro <regebro@gmail.com> wrote:
So, how about time.timer()?
That seems like a bad idea; it would be too easy to confuse with (or misspell as) time.time(). Out of the big synonym list Guido posted, I rather like time.stopwatch() - it makes it more explicit that the purpose of the function is to measure intervals, rather identifying absolute points in time. Cheers, Nadeem

On 28Mar2012 23:40, Victor Stinner <victor.stinner@gmail.com> wrote: | > Does this primarily give a high resolution clock, or primarily a | > monotonic clock? That's not clear from either the name, or the PEP. | | I expect a better resolution from time.monotonic() than time.time(). I | don't have exact numbers right now, but I began to document each OS | clock in the PEP. I wish to raise an alternative to these set-in-stone policy-in-the-library choices, and an alternative to any proposal that does fallback in a function whose name suggests otherwise. Off in another thread on PEP 418 I suggested a cleaner approach to offering clocks to the user: let the user ask! My (just two!) posts on this are here: http://www.mail-archive.com/python-dev@python.org/msg66174.html http://www.mail-archive.com/python-dev@python.org/msg66179.html The second post is more important as it fleshes out my reasons for considering this appraoch better. I've just finished sketching out a skeleton here: https://bitbucket.org/cameron_simpson/css/src/fb476fcdcfce/lib/python/cs/clo... In short: - take Victor's hard work on system clocks and classifying thm by feature set - tabulate access to them in a list of clock objects - base access class goes (example user call): # get a clock object - often a singleton under the hood T = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_STEADY|T_HIRES) # what kind of clock did I get? print T.flags # get the current time now = T.now - offer monotonic() and/or steady() etc as convenience functions calling get_clock() in a fashion like the above example - don't try to guess the user's use case ahead of time This removes policy from the library functions and makes it both simple and obvious in the user's calling code, and also makes it possible for the user to inspect the clock and find out what quality/flavour of clock they got. Please have a glance through the code, especially the top and botth bits; it is only 89 lines long and includes (presently) just a simple object for time.time() and (importantly for the bikeshedding) an example synthetic clock to give a monotonic caching clock from another non-monotonic clock (default, again, time.time() in this prototype). Suitably fleshed out with access to the various system clocks, this can offer all the current bikeshedding in a simple interface and without constraining user choices to "what we thought of, or what we thought likely". Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Availability: Samples Q1/97 Volume H2/97 So, it's vapor right now, but if you want to sell vapor in 1997 you better had damn fast vapor then... - Burkhard Neidecker-Lutz on the DEC Alpha 21264, October 1996

I've just finished sketching out a skeleton here:
https://bitbucket.org/cameron_simpson/css/src/fb476fcdcfce/lib/python/cs/clo...
get_clock() returns None if no clock has the requested flags, whereas I expected an exception (LookupError or NotImplementError?). get_clock() doesn't remember if a clock works or not (if it raises an OSError) and does not fallback to the next clock on error. See "pseudo-codes" in the PEP 418. The idea of flags attached to each clock is interesting, but I don't like the need of different list of clocks. Should I use MONTONIC_CLOCKS or HIRES_CLOCKS when I would like a monotonic and high-resolution clock? It would be simpler to have only one global and *private* list. If you have only one list of clocks, how do sort the list to get QueryPerformanceCounter when the user asks for highres and GetTickCount when the user asks for monotonic? The "if clock.flags & flags == flags:" test in get_clock() is maybe not enough. I suppose that we would have the following flags for Windows functions: QueryPerformanceCounter.flags = T_HIRES GetTickCount.flags = T_MONOTONIC | T_STEADY (or maybe QueryPerformanceCounter.flags = T_HIRES | T_MONOTONIC ?) monotonic_clock() should maybe try to get a clock using the following list of conditions: - T_MONOTONIC | T_STEADY - T_MONOTONIC | T_HIGHRES - T_MONOTONIC The T_HIGHRES flag in unclear, even in the PEP. According to the PEP, any monotonic clock is considered as a "high-resolution" clock. Do you agree? So we would have: GetTickCount.flags = T_MONOTONIC | T_STEADY | T_HIGHRES Even if GetTickCount has only an accuracy of 15 ms :-/ Can list please give the list of flags of each clocks listed in the PEP? Only clocks used for time.time, time.monotonic and time.highres (not process and thread clocks nor QueryUnbiasedInterruptTime).
The API looks much more complex than the API proposed in PEP 418 just to get the time. You have to call a function to get a function, and then call the function, instead of just calling a function directly. Instead of returning an object with a now() method, I would prefer to get directly the function getting time, and another function to get "metadata" of the clock.
I'm not sure that users understand correctly differences between all these clocks and are able to use your API correctly. How should I combinese these 3 flags (T_HIRES, T_MONOTONIC and T_STEADY)? Can I use any combinaison? Which flags are "portable"? Or should I always use an explicit fallback to ensure getting a clock on any platform? Could you please update your code according to my remarks? I will try to integrate it into the PEP. A PEP should list all alternatives! Victor

On 4/2/2012 4:37 AM, Victor Stinner wrote:
If there are more than two clocks, with different characteristics, no API is going to be both simple to use and fast to call. If there are more than two clocks, with different characteristics, then having an API to get the right API to call to get a time seems very natural to me. One thing I don't like about the idea of fallback being buried under some API is that the efficiency of that API on each call must be less than the efficiency of directly calling an API to get a single clock's time. For frequently called high resolution clocks, this is more burdensome than infrequently called clocks.... yet those seem to be the ones for which fallbacks are proposed, because of potential unavailability. Having properties on each of various different clock functions is cumbersome... the user code must know about each clock, how to obtain the properties, and then how to choose one for use... And how will one be chosen for use? Under the assumption that all return some sort of timestamp and take no parameters, a local name will be assigned to the clock of interest: if ...: myTime = os.monotonous elif ...: myTime = os.evenhigherres ... elif ...: myTime = time. time so that myTime can be use throughout. Cameron's API hides all the names of the clocks, and instead offers to do the conditional logic for you, and the resultant API returned can be directly assigned to myTime, and the logic for choosing a clock deals only with the properties of the clock, not the names of the APIs, which is a nice abstraction. There would not even be a need to document the actual names of the APIs for each individual clock, except that probably some folks would want to directly code them, especially if they are not interested in cross-platform work. The only thing I'm not so sure about: can the properties be described by flags? Might it not be better to have an API that allows specification of minimum resolution, in terms of fractional seconds? Perhaps other properties suffice as flags, but perhaps not resolution.

On Tue, Apr 3, 2012 at 3:44 AM, Glenn Linderman <v+python@g.nevcal.com> wrote:
No, that's a misunderstanding of the fallback mechanism. The fallback happens when the time module is initialised, not on every call. Once the appropriate clock has been selected during module initialisation, it is invoked directly at call time. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 4/2/2012 2:40 PM, Nick Coghlan wrote:
I would hope that is how the fallback mechanism would be coded, but I'm pretty sure I've seen other comments in this thread that implied otherwise. But please don't ask me to find them, this thread is huge. Glenn

On 02Apr2012 14:59, Glenn Linderman <v+python@g.nevcal.com> wrote: | On 4/2/2012 2:40 PM, Nick Coghlan wrote: | > On Tue, Apr 3, 2012 at 3:44 AM, Glenn Linderman<v+python@g.nevcal.com> wrote: | >> > One thing I don't like about the idea of fallback being buried under some | >> > API is that the efficiency of that API on each call must be less than the | >> > efficiency of directly calling an API to get a single clock's time. | > No, that's a misunderstanding of the fallback mechanism. The fallback | > happens when the time module is initialised, not on every call. Once | > the appropriate clock has been selected during module initialisation, | > it is invoked directly at call time. | | I would hope that is how the fallback mechanism would be coded, but I'm | pretty sure I've seen other comments in this thread that implied | otherwise. But please don't ask me to find them, this thread is huge. The idea of falling back to different clocks on the fly on different calls got a bit of a rejection I thought. A recipe for clock inconsitency whatever the failings of the current clock. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ We need a taxonomy for 'printing-that-is-no-longer-printing.' - overhead by WIRED at the Intelligent Printing conference Oct2006

On 02Apr2012 10:44, Glenn Linderman <v+python@g.nevcal.com> wrote: | On 4/2/2012 4:37 AM, Victor Stinner wrote: | > The API looks much more complex than the API proposed in PEP 418 just | > to get the time. You have to call a function to get a function, and | > then call the function, instead of just calling a function directly. | > | > Instead of returning an object with a now() method, I would prefer to | > get directly the function getting time, and another function to get | > "metadata" of the clock. | | If there are more than two clocks, with different characteristics, no | API is going to be both simple to use and fast to call. | | If there are more than two clocks, with different characteristics, then | having an API to get the right API to call to get a time seems very | natural to me. It is, though Victor's point about offering the very easy to use API is valid. The new code has the "flat" monotonic() et al calls as well. | One thing I don't like about the idea of fallback being buried under | some API is that the efficiency of that API on each call must be less | than the efficiency of directly calling an API to get a single clock's | time. For frequently called high resolution clocks, this is more | burdensome than infrequently called clocks.... yet those seem to be the | ones for which fallbacks are proposed, because of potential unavailability. I hadn't thought about that, but it isn't actually a big deal. The overhead isn't zero, but in order to always use the _same_ clock to return hires() (for example) the library has to cache the clock lookup anyway. Current clockutils.py skeleton here: https://bitbucket.org/cameron_simpson/css/src/91848af8663b/lib/python/cs/clo... does so. | The only thing I'm not so sure about: can the properties be described by | flags? Might it not be better to have an API that allows specification | of minimum resolution, in terms of fractional seconds? Perhaps other | properties suffice as flags, but perhaps not resolution. It sounds nice, but there are some difficulties. Firstly, the (currently just 3) flags were chosen to map to the three features sought (in various cobinations) for clocks. Once you start requesting precision (a totally reasonable desire, BTW) you also want to request degree of slew (since "steady" is a tunabe term) and so forth. And what for clocks that have variable precision (I'm imaging here a clock which really is a float, and for large times (in the far future) can't return the sane resolution because of the size of a float. The concern is valid though. I could imagine beefing up the clock object metadata with .epoch (can be None!), precision (function of float width versus clock return value epislon), epsilon (your fraction of a second parameter). Of course, for some clocks any of these might be None. Then the truly concerned user iterates over the available clocks with the desired coarse flags, inspecting each closely for precision or whatever. Easy enough to tweak get_clock() to take an optional all_clocks=False parameter to return all matching clocks in an iterable instead of (the first match or None). Or the user could reach directly for one of the clock lists. cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Craft, n. A fool's substitute for brains. - The Devil's Dictionary

On 02Apr2012 13:37, Victor Stinner <victor.stinner@gmail.com> wrote: | > I've just finished sketching out a skeleton here: | > https://bitbucket.org/cameron_simpson/css/src/fb476fcdcfce/lib/python/cs/clo... | | get_clock() returns None if no clock has the requested flags, whereas | I expected an exception (LookupError or NotImplementError?). That is deliberate. People can easily write fallback like this: clock = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_MONOTONIC) With exceptions one gets a complicated try/except/else chain that is much harder to read. With a second fallback the try/except gets even worse. If one wants an exception it is easy to follow up with: if not clock: raise RunTimeError("no suitable clocks on offer on this platform") | get_clock() doesn't remember if a clock works or not (if it raises an | OSError) and does not fallback to the next clock on error. See | "pseudo-codes" in the PEP 418. I presume the available clocks are all deduced from the platform. Your pseudo code checks for OSError at fetch-the-clock time. I expect that to occur once when the module is loaded, purely to populate the table of avaiable platform clocks. If you are concerned about clocks being available/unavailable at different times (unplugging the GPS peripheral? just guessing here) that will have to raise OSError during the now() call (assuming the clock even exposes the failure; IMO it should when now() is called). | The idea of flags attached to each clock is interesting, but I don't | like the need of different list of clocks. There's no need, just quality of implementation for the monotonic()/hires() convenience calls, which express the (hoped to be common) policy of what clock to offer for each. We've just had pages upon pages of discussion about what clock to offer for the rather bald monotonic() (et al) calls. The ordering of the MONTONIC_CLOCKS list would express the result of that discussion, in that the "better" clocks come first. | Should I use | MONTONIC_CLOCKS or HIRES_CLOCKS when I would like a monotonic and | high-resolution clock? Note that you don't need to provide a clock list at all; get_clock(0 will use ALL_CLOCKS by default, and hires() and monotonic() should each have their own default list. I'll put in montonic() and montonic_clock(clocklist=MONOTONIC_CLOCKS) into the skeleton to make this clear; I see I've omitted them. Regarding the choice itself: as the _caller_ (not the library author), you must decide what you want most. You're already planning offering monotonic() and hires() calls without my proposal! Taking your query "Should I use MONTONIC_CLOCKS or HIRES_CLOCKS when I would like a monotonic and high-resolution clock" is _already_ a problem. Of course you must call monotonic() or hires() first under the current scheme, and must answer this question anyway. Do you prefer hires? Use it first! No preference? Then the question does not matter. If I, as the caller, have a preference then it is obvious what to use. If I do not have a preference then I can just call get_clock() with both flags and then arbitrarily fall back to hires() or monotonic() if that does not work. | It would be simpler to have only one global and | *private* list. No. No no no no no! The whole point is to let the user be _able_ to control the choices to a fair degree without platform special knowledge. The lists are deliberately _optional_ parameters and anyway hidden in the hires() and monotonic() convenince functions; the user does not need to care about them. But the picky user may! The lists align exactly one to one with the feature flags, so there is no special knowledge present here that is not already implicit in publishing the feature flags. | If you have only one list of clocks, how do sort the list to get | QueryPerformanceCounter when the user asks for highres and | GetTickCount when the user asks for monotonic? This is exactly why there are supposed to be different lists. You have just argued against your objection above. | The "if clock.flags & | flags == flags:" test in get_clock() is maybe not enough. I suppose | that we would have the following flags for Windows functions: | | QueryPerformanceCounter.flags = T_HIRES | GetTickCount.flags = T_MONOTONIC | T_STEADY | | (or maybe QueryPerformanceCounter.flags = T_HIRES | T_MONOTONIC ?) Obviously these depend on the clock characteristics. Is QueryPerformanceCounter monotonic? | monotonic_clock() should maybe try to get a clock using the following | list of conditions: | - T_MONOTONIC | T_STEADY | - T_MONOTONIC | T_HIGHRES | - T_MONOTONIC Sure, seems reasonable. That is library internal policy _for the convenince monotonic() function()_. | The T_HIGHRES flag in unclear, even in the PEP. According to the PEP, | any monotonic clock is considered as a "high-resolution" clock. Do you | agree? Not particularly. I easily can imagine a clock with one second resolution hich was monotonic. I would not expect it to have the T_HIRES flag. Example: a synthetic monotonic clock based on a V7 UNIX time() call. But, if it _happens_ that all the monotonic clocks are also hires, so be it. That would be an empirical outcome, not policy. | So we would have: | | GetTickCount.flags = T_MONOTONIC | T_STEADY | T_HIGHRES | | Even if GetTickCount has only an accuracy of 15 ms :-/ T_HIGHRES is a quality call, surely? If 15ms is too sloppy for a "high resolution, the is should _not_ have the T_HIRES flag. | Can list please give the list of flags of each clocks listed in the | PEP? Only clocks used for time.time, time.monotonic and time.highres | (not process and thread clocks nor QueryUnbiasedInterruptTime). | | > # get a clock object - often a singleton under the hood | > T = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_STEADY|T_HIRES) | > # what kind of clock did I get? | > print T.flags | > # get the current time | > now = T.now | | The API looks much more complex than the API proposed in PEP 418 just | to get the time. You have to call a function to get a function, and | then call the function, instead of just calling a function directly. One could have a flat interface as in the PEP, but then the results are not inspectable; the user cannot find out what specific clock, or even kind of clock, was used for the result returned. Unless you want to subclass float for the return values. You could return instances of a float with meta information pointing at the clock used to provide it. I'm -0.5 on that idea. Another advantage of returning a clock object is that it avoids the difficulty of switching implementations behind the user's back, an issue raised in the discussion and rightly rejected as a bad occurence. If the user is handed a clock object, _they_ keep the "current clock in use" state by by having the object reference. | Instead of returning an object with a now() method, I would prefer to | get directly the function getting time, and another function to get | "metadata" of the clock. Then they're disconnected. How do I know the get-metadata call accesses the clock I just used? Only by having library internal global state. I agree some people probably want the flat "get me the time" call, and have no real objection to such existing. But I strongly object to not giving the user control over what they use, and the current API offers no control. | > This removes policy from the library functions and makes it both simple | > and obvious in the user's calling code, and also makes it possible for | > the user to inspect the clock and find out what quality/flavour of clock | > they got. | | I'm not sure that users understand correctly differences between all | these clocks and are able to use your API correctly. How should I | combinese these 3 flags (T_HIRES, T_MONOTONIC and T_STEADY)? Can I use | any combinaison? Of course. Just as with web searches, too many flags may get you an empty result on some platforms, hence the need to fall back. But the _nature_ of the fallback should be in the user's hands. The hires() et al calls can of course offer convenient presupplied fallback according to the preconceptions of the library authors, hopefully well tuned to common users' needs. But if should not be the only mode offered, because you don't know the user's needs. | Which flags are "portable"? Or should I always use an explicit | fallback to ensure getting a clock on any platform? All the flag are portable, but if the platform doesn't supply a clock with the requested flags, even if there's only one flag, the correct result is "None" for the clock offered. Note you can supply no flags! You can always fall all the way back to 0 for the flags; in the skeleton provided that will get you UNIXClock, which is a wrapper for the existing time.time(). In fact, I'll make the flags parameter also optional for get_clock(), defaulting to 0, to make that easy. That becomes your totally portable call:-) | Could you please update your code according to my remarks? I will try | to integrate it into the PEP. A PEP should list all alternatives! Surely. The only updates I can see are to provide the flat interface (instead of via clock-object indirection) and the missing hires_clock() and monotonic_clock() convenience methods. I'll do that. Followup post shortly with new code URL. Would you propose other specific additions? Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Whatever is not nailed down is mine. What I can pry loose is not nailed down. - Collis P. Huntingdon

On 03Apr2012 07:38, I wrote: | On 02Apr2012 13:37, Victor Stinner <victor.stinner@gmail.com> wrote: | | Could you please update your code according to my remarks? I will try | | to integrate it into the PEP. A PEP should list all alternatives! New code here: https://bitbucket.org/cameron_simpson/css/src/91848af8663b/lib/python/cs/clo... Diff: https://bitbucket.org/cameron_simpson/css/changeset/91848af8663b Changelog: updates based on suggestions from Victor Stinner: "flat" API calls to get time directly, make now() a method instead of a property, default flags for get_clock(), adjust hr_clock() to hires_clock(0 for consistency. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Q: How does a hacker fix a function which doesn't work for all of the elements in its domain? A: He changes the domain. - Rich Wareham <rjw57@hermes.cam.ac.uk>

On 03Apr2012 07:51, I wrote: | Changelog: updates based on suggestions from Victor Stinner: "flat" API | calls to get time directly, make now() a method instead of a property, | default flags for get_clock(), adjust hr_clock() to hires_clock(0 for | consistency. BTW, I'd also happily change T_HIRES to HIRES and so forth. They're hard to type and read at present. The prefix is a hangover from old C coding habits, with no namespaces. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ If you don't live on the edge, you're taking up too much space. - t-shirt

On 03Apr2012 07:38, I wrote: | On 02Apr2012 13:37, Victor Stinner <victor.stinner@gmail.com> wrote: | | Should I use | | MONTONIC_CLOCKS or HIRES_CLOCKS when I would like a monotonic and | | high-resolution clock? | | Note that you don't need to provide a clock list at all; get_clock(0 | will use ALL_CLOCKS by default, and hires() and monotonic() should each | have their own default list. [...] | | It would be simpler to have only one global and | | *private* list. [...] | The whole point is to let the user be _able_ to control the choices to a | fair degree without platform special knowledge. On some reflection I may lean a little more Victor's way here: I am still very much of the opinion that there should be multiple clock lists so that hires() can offer the better hires clocks first and so forth. However, perhaps I misunderstood and he was asking if he needed to name a list to get a hires clock etc. This intent is not to need to, via the convenience functions. Accordingly, maybe the list names needn't be published, and may complicate the published interface even though they're one to one with the flags. It would certainly up the ante slightly f we added more flags some time later. (For example, I think any synthetic clocks such as the caching example in the skeleton should probably have a SYNTHETIC flag. You might never ask for it, but you should be able to check for it. (I personally suspect some of the OS clocks are themselves synthetic, but no matter...) The flip side of this of course is that if the list names are private then the get_clock() and hires() etc functions almost mandatorially need the optional all_clocks=False parameter mooted in a sibling post; the really picky user needs a way to iterate over the available clocks to make a fine grained decision. On example would be to ask for monotonic clocks but omit synthetic ones (there's a synthetic clock in the skeleton though I don't partiularly expect one in reality - that really is better in a broader "*utils" module; I also do NOT want to get into complicated parameters to say these flags but not _those_ flags and so forth for other metadata. And again, an external module offering synthetic clocks could easily want to be able to fetch the existing and augument the list with its own, then use that with the get_clock() interfaces. So in short I think: - there should be, internally at least, multiple lists for quality of returned result - there should be a way to iterate over the available clocks, probably via an all_clocks paramater instead of a public list name Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ There is hopeful symbolism in the fact that flags do not wave in a vacuum. - Arthur C. Clarke

I like the aim of letting the user control what clock it get, but I find this API pretty horrible:
clock = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_MONOTONIC)
Just my 2 groszy. //Lennart

On 03Apr2012 07:51, Lennart Regebro <regebro@gmail.com> wrote: | I like the aim of letting the user control what clock it get, but I | find this API pretty horrible: | | > clock = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_MONOTONIC) FWIW, the leading "T_" is now gone, so it would now read: clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) If the symbol names are not the horribleness, can you qualify what API you would like more? -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ We had the experience, but missed the meaning. - T.S. Eliot

On 03/04/2012 07:03, Cameron Simpson wrote:
I reckon the API is ok given that you don't have to supply the flags, correct? A small point but I'm with (I think) Terry Reedy and Steven D'Aprano in that hires is an English word, could you please substitute highres and HIGHRES, thanks. -- Cheers. Mark Lawrence.

On 03Apr2012 09:03, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote: | On 03/04/2012 07:03, Cameron Simpson wrote: | > On 03Apr2012 07:51, Lennart Regebro<regebro@gmail.com> wrote: | > | I like the aim of letting the user control what clock it get, but I | > | find this API pretty horrible: | > | | > |> clock = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_MONOTONIC) | > | > FWIW, the leading "T_" is now gone, so it would now read: | > | > clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) | > | > If the symbol names are not the horribleness, can you qualify what API | > you would like more? | | I reckon the API is ok given that you don't have to supply the flags, | correct? That's right. And if the monotonic() or monotonic_clock() functions (or the hires* versions if suitable) do what you want you don't even need that. You only need the "or" style to choose your own fallback according to your own criteria. | A small point but I'm with (I think) Terry Reedy and Steven D'Aprano in | that hires is an English word, could you please substitute highres and | HIGHRES, thanks. I have the same issue and would be happy to do it. Victor et al, how do you feel about this? People have been saying "hires" throughout the threads I think, but I for one would be slightly happier with "highres". Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ I bested him in an Open Season of scouring-people's-postings-looking-for- spelling-errors. - kevin@rotag.mi.org (Kevin Darcy)

On Tue, Apr 3, 2012 at 08:03, Cameron Simpson <cs@zip.com.au> wrote:
Well, get_clock(monotonic=True, highres=True) would be a vast improvement over get_clock(MONOTONIC|HIRES). I also think it should raise an error if not found. The clarity and easy of use of the API is much more important than how much you can do in one line. //Lennart

Lennart Regebro wrote:
Allowing get_clock(True, True)? Ick. My nomination would be get_clock(MONOTONIC, HIGHRES) -- easier on the eyes with no |.
What's unclear about returning None if no clocks match? Cheers, ~Ethan~

On 03Apr2012 09:07, Ethan Furman <ethan@stoneleaf.us> wrote: | Lennart Regebro wrote: | > On Tue, Apr 3, 2012 at 08:03, Cameron Simpson <cs@zip.com.au> wrote: | >> clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) | >> | >> If the symbol names are not the horribleness, can you qualify what API | >> you would like more? | > | > Well, get_clock(monotonic=True, highres=True) would be a vast | > improvement over get_clock(MONOTONIC|HIRES). | | Allowing get_clock(True, True)? Ick. My nomination would be | get_clock(MONOTONIC, HIGHRES) -- easier on the eyes with no |. get_clock already has two arguments - you can optionally hand it a clock list - that's used by monotonic_clock() and hires_clock(). Have a quick glance at: https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.p... (I finally found out how to point at the latest revision on BitBucket; it's not obvious from the web interface itself.) | > I also think it should | > raise an error if not found. The clarity and easy of use of the API is | > much more important than how much you can do in one line. How much you can do _clearly_ in one line is a useful metric. | What's unclear about returning None if no clocks match? The return of None is very deliberate. I _want_ user specified fallback to be concise and easy. The example: clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) seems to satisfy both these criteria to my eye. Raising an exception makes user fallback a royal PITA, with a horrible try/except cascade needed. Exceptions are all very well when there is just one thing to do: parse this or fail, divide this by that or fail. If fact they're the very image of "do this one thing or FAIL". They are not such a good match for do this thing or that thing or this other thing. When you want a simple linear cascade of choices, Python's short circuiting "or" operator is a very useful thing. Having an obsession with exceptions is IMO unhealthy. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Because of its special customs, crossposting between alt.peeves and normal newsgroups is discouraged. - Cameron Spitzer

Cameron Simpson wrote:
get_clock already has two arguments - you can optionally hand it a clock list - that's used by monotonic_clock() and hires_clock().
def get_clock(*flags, *, clocklist=None): ''' Return a Clock based on the supplied `flags`. The returned clock shall have all the requested flags. If no clock matches, return None. ''' wanted = 0 for flag in flags: wanted |= flag if clocklist is None: clocklist = ALL_CLOCKS for clock in clocklist: if clock.flags & wanted == wanted: return clock.factory() return None Would need to make *flags change to the other *_clock functions.
Have a quick glance at:
https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.p...
Thanks.
Which would become: clock = get_clock(MONOTONIC, HIGHRES) or get_clock(MONOTONIC) +1 to returning None
Another +1. ~Ethan~

On 03Apr2012 15:08, Ethan Furman <ethan@stoneleaf.us> wrote: | Cameron Simpson wrote: | > get_clock already has two arguments - you can optionally hand it a clock | > list - that's used by monotonic_clock() and hires_clock(). | | def get_clock(*flags, *, clocklist=None): I presume that bare "*," is a typo. Both my python2 and python3 commands reject it. [...] | wanted = 0 | for flag in flags: | wanted |= flag [...] I could do this. I think I'm -0 on it, because it doesn't seem more expressive to my eye than the straight make-a-bitmask "|" form. Other opinions? | Would need to make *flags change to the other *_clock functions. Yep. | > The return of None is very deliberate. I _want_ user specified fallback | > to be concise and easy. The example: | > clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) | | Which would become: | clock = get_clock(MONOTONIC, HIGHRES) or get_clock(MONOTONIC) | | +1 to returning None | | > Exceptions are all very well when there is just one thing to do: parse | > this or fail, divide this by that or fail. If fact they're the very | > image of "do this one thing or FAIL". They are not such a good match for do | > this thing or that thing or this other thing. Another thought that occurred in the shower was that get_clock() et al are inquiry functions, and returning None is very sensible there. monotonic() et al are direct use functions, which should raise an exception if unavailable so that code like: t0 = monotonic() ....... t1 = monotonic() does not become littered with checks for special values like None. I consider this additional reason to return None from get_clock(). Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ DON'T DRINK SOAP! DILUTE DILUTE! OK! - on the label of Dr. Bronner's Castile Soap

On Wed, Apr 4, 2012 at 9:38 AM, Cameron Simpson <cs@zip.com.au> wrote:
Yes. I've been mostly staying out of the PEP 418 clock discussion (there are enough oars in there already), but numeric flags are unnecessarily hard to debug. Use strings as your constants unless there's a compelling reason not to. Seeing "('MONOTONIC', 'HIGHRES')" in a debugger or exception message is a lot more informative than seeing "3". Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Apr 3, 2012 at 18:07, Ethan Furman <ethan@stoneleaf.us> wrote:
What's unclear about returning None if no clocks match?
Nothing, but having to check error values on return functions are not what you typically do in Python. Usually, Python functions that fail raise an error. Please don't force Python users to write pseudo-C code in Python. //Lennart

Lennart Regebro wrote:
You mean like the dict.get() function? --> repr({}.get('missing')) 'None' Plus, failure mode is based on intent: if the intent is "Give a clock no matter what", then yes, an exception when that's not possible is the way to go. But if the intent is "Give me a clock that matches this criteria" then returning None is perfectly reasonable. ~Ethan~

Georg Brandl wrote:
Also not a very good example -- if 'missing' was there with a value of None the two situations could not be distinguished with the one call. At any rate, the point is that there is nothing inherently wrong nor unPythonic about a function returning None instead of raising an exception. ~Ethan~

On 04Apr2012 19:47, Georg Brandl <g.brandl@gmx.net> wrote: | Am 04.04.2012 18:18, schrieb Ethan Furman: | > Lennart Regebro wrote: | >> On Tue, Apr 3, 2012 at 18:07, Ethan Furman <ethan@stoneleaf.us> wrote: | >>> What's unclear about returning None if no clocks match? | >> | >> Nothing, but having to check error values on return functions are not | >> what you typically do in Python. Usually, Python functions that fail | >> raise an error. Please don't force Python users to write pseudo-C code | >> in Python. | > | > You mean like the dict.get() function? | > | > --> repr({}.get('missing')) | > 'None' | | Strawman: this is not a failure. And neither is get_clock() returning None. get_clock() is an inquiry function, and None is a legitimate response when no clock is satisfactory, just as a dict has no key for a get(). Conversely, monotonic() ("gimme the time!") and indeed time() should raise an exception if there is no clock. They're, for want of a word, "live" functions you would routinely embed in a calculation. So not so much a straw man as a relevant illuminating example. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ A crash reduces your expensive computer to a simple stone. - Haiku Error Messages http://www.salonmagazine.com/21st/chal/1998/02/10chal2.html

On Wed, Apr 04, 2012 at 05:47:16PM +0200, Lennart Regebro wrote:
Absolutely. "Errors should never pass silently."
Please don't force Python users to write pseudo-C code in Python.
+1. Pythonic equivalent of "get_clock(THIS) or get_clok(THAT)" is for flag in (THIS, THAT): try: clock = get_clock(flag) except: pass else: break else: raise ValueError('Cannot get clock, tried THIS and THAT') Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

Oleg Broytman wrote:
Again, what's the /intent/? No matching clocks does not have to be an error.
Wow -- you'd rather write nine lines of code instead of three? clock = get_clock(THIS) or get_clock(THAT) if clock is None: raise ValueError('Cannot get clock, tried THIS and THAT') ~Ethan~

On Wed, Apr 04, 2012 at 11:03:02AM -0700, Ethan Furman wrote:
Yes - to force people to write the last two lines. Without forcing most programmers will skip them. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

Oleg Broytman wrote:
Forced? I do not use Python to be forced to use one style of programming over another. And it's not like returning None will allow some clock calls to work but not others -- as soon as they try to use it, it will raise an exception. ~Ethan~

On Wed, Apr 04, 2012 at 12:52:00PM -0700, Ethan Furman wrote:
Then it's strange you are using Python with its strict syntax (case-sensitivity, forced indents), ubiquitous exceptions, limited syntax of lambdas and absence of code blocks (read - forced functions), etc.
There is a philosophical distinction between EAFP and LBYL. I am mostly proponent of LBYL. Well, I am partially retreat. "Errors should never pass silently. Unless explicitly silenced." get_clock(FLAG, on_error=None) could return None. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Thu, Apr 5, 2012 at 8:05 AM, Oleg Broytman <phd@phdru.name> wrote:
I still don't see what's erroneous about returning None when asked for an object that is documented to possibly not exist, ever, in some implementations. Isn't that precisely why None exists?

On Thu, Apr 05, 2012 at 10:06:38PM +0900, Stephen J. Turnbull wrote:
Why doesn't open() return None for a non-existing file? or socket.gethostbyname() for a non-existing name? Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Thu, Apr 5, 2012 at 10:34 PM, Oleg Broytman <phd@phdru.name> wrote:
Why doesn't open() return None for a non-existing file? or socket.gethostbyname() for a non-existing name?
That's not an answer to my question, because those calls have very important use cases where the user knows the object exists (and in fact in some cases open() will create it for him), so that failure to exist is indeed a (user) error (such as a misspelling). I find it hard to imagine use cases where "file = open(thisfile) or open(thatfile)" makes sense. Not even for the case where thisfile == 'script.pyc' and thatfile == 'script.py'. The point of the proposed get_clock(), OTOH, is to ask if an object with certain characteristics exists, and the fact that it returns the clock rather than True if found is a matter of practical convenience. Precisely because "clock = get_clock(best) or get_clock(better) or get_clock(acceptable)" does make sense.

On Thu, Apr 05, 2012 at 11:45:06PM +0900, Stephen J. Turnbull wrote:
Counterexamples - any configuration file: a program looks for its config at $HOME and not finding it there looks in /etc. So config = open('~/.someprogram.config') or open('/etc/someprogram/config') would make sense. The absence of any of these files is not an error at all - the program just starts with default configuration. So if the resulting config in the code above would be None - it's still would be ok. But Python doesn't allow that. Some configuration files are constructed by combining a number of user-defined and system-defined files. E.g., the mailcap database. It should be something like combined_database = [db for db in ( open('/etc/mailcap'), open('/usr/etc/mailcap'), open('/usr/local/etc/mailcap'), open('~/.mailcap'), ) if db] But no way - open() raises IOError, not return None. And I think it is the right way. Those who want to write the code similar to the examples above - explicitly suppress exceptions by writing wrappers.
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Thu, 05 Apr 2012 19:22:17 +0400, Oleg Broytman <phd@phdru.name> wrote:
Ah, but the actual code in the mimetypes module (whose list is even longer) looks like this: for file in files: if os.path.isfile(file): db.read(file) That is, Python provides a query function that doesn't raise an error. Do you really think we need to add a third clock function (the query function) that just returns True or False? Maybe we do, if actually creating the clock could raise an error even if exists, as is the case for 'open'. (But unless I'm confused none of this has anything to do with Victor's PEP as currently proposed :) --David

On Thu, Apr 05, 2012 at 11:38:13AM -0400, R. David Murray wrote:
May be we do. Depends on the usage patterns. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Thu, Apr 05, 2012 at 07:22:17PM +0400, Oleg Broytman wrote:
A counterexample with gethostbyname - a list of proxies. It's not an error if some or even all proxies in the list are down - one just connect to the first that's up. So a chain like proxy_addr = gethostbyname(FIRST) or gethostbyname(SECOND) would make sense. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Fri, Apr 6, 2012 at 12:22 AM, Oleg Broytman <phd@phdru.name> wrote:
Note, implicit existential quantifier.
Counterexamples
Not an argument against an existential quantifier.
But Python doesn't allow [use of conditional constructs when opening a series of files, one must trap exceptions].
True. Python needs to make a choice, and the existence of important cases where the user knows that the object (file) exists makes it plausible that the user would prefer an Exception. Also, open() is intended to be a fairly thin wrapper over the OS facility, and often the OS terms a missing file an "error". I might have chosen to implement a 'None' return if I had designed open(), but I can't get too upset about raising an Exception as it actually does. What I want to know is why you're willing to assert that absence of a clock of a particular configuration is an Exception, when that absence clearly documented to be a common case? I don't find your analogies to be plausible. They seem to come down to "sometimes in Python we've made choices that impose extra work on some use cases, so we should impose extra work on this use case too." But that surely isn't what you mean.

On Thu, Apr 5, 2012 at 21:57, Stephen J. Turnbull <stephen@xemacs.org> wrote:
One fundamental difference is that there are many reasons one might fail to open a file. It may not exist. It may not have permissions allowing the request. It may be locked. If open() returned None, this information would have to be retrievable through another function. However since it returns an exception, that information is already wrapped up in the exception object, should you choose to catch it, and likely to be logged otherwise. In the case of the clocks, I'm assuming the only reason you would fail to get a clock is because it isn't provided by hardware and/or OS. You don't have to worry about transient scenarios on multi-user systems where another user has locked the clock. Thus the exception cannot tell you anything more than None tells you. (Of course, if my assumption is wrong, I'm not sure whether my reasoning still applies.) -- Michael Urman

On Fri, Apr 06, 2012 at 11:57:20AM +0900, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
An error or not an error depends on how people will use the API. I usually don't like error codes -- people tend to ignore them or check lazily. If some library would do (get_clock(THIS) or get_clock(THAT)).clock() I want to get a clearly defined and documented clock-related error, not some vague "AttributeError: 'NoneType' object has no attribute 'clock'". Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

Oleg Broytman wrote:
I come from assembly -- 'a' and 'A' are *not* the same. indents -- I already used them; finding a language that gave them the same importance I did was incredible. exceptions -- Python uses them, true, but I don't have to in my own code (I do, but that's besides the point). lambdas -- they work just fine for my needs. etc.
It's only an error if it's documented that way and, more importantly, thought of that way. The re module is a good example: if it can't find what you're looking for it returns None -- it does *not* raise a NotFound exception. I see get_clock() the same way: I need a clock that does xyz... None? Okay, there isn't one. ~Ethan~

On Thu, Apr 05, 2012 at 11:56:00AM -0700, Ethan Furman wrote:
But open() raises IOError. ''.find('a') returns -1 but ''.index('a') raises ValueError. So we can argue in circles both ways, there are too many arguments pro and contra. Python is just too inconsistent to be consistently argued over. ;-) Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On 06Apr2012 00:15, Oleg Broytman <phd@phdru.name> wrote: | So we can argue in circles both ways, there are too many arguments | pro and contra. Python is just too inconsistent to be consistently | argued over. ;-) Bah! I think these threads demonstrate that we can consistently argue over Python for weeks per topic, sometimes months and years. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Sam Jones <samjones@leo.unm.edu> on the Nine Types of User: Frying Pan/Fire Tactician - "It didn't work with the data set we had, so I fed in my aunt's recipe for key lime pie." Advantages: Will usually fix error. Disadvantages: 'Fix' is defined VERY loosely here. Symptoms: A tendancy to delete lines that get errors instead of fixing them. Real Case: One user complained that their program executed, but didn't do anything. The scon looked at it for twenty minutes before realizing that they'd commented out EVERY LINE. The user said, "Well, that was the only way I could get it to compile."

On 05Apr2012 03:05, Oleg Broytman <phd@phdru.name> wrote: | On Wed, Apr 04, 2012 at 12:52:00PM -0700, Ethan Furman wrote: | > Forced? I do not use Python to be forced to use one style of | > programming over another. | | Then it's strange you are using Python with its strict syntax | (case-sensitivity, forced indents), ubiquitous exceptions, limited | syntax of lambdas and absence of code blocks (read - forced functions), | etc. But exceptions are NOT ubiquitous, nor should they be. They're a very popular and often apt way to handle certain circumstances, that's all. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ On the one hand I knew that programs could have a compelling and deep logical beauty, on the other hand I was forced to admit that most programs are presented in a way fit for mechanical execution, but even if of any beauty at all, totally unfit for human appreciation. - Edsger W. Dijkstra

Oleg Broytman wrote:
You're not my real Dad! You can't tell me what to do! *wink* This level of paternalism is unnecessary. It's not your job to "force" programmers to do anything. If people skip the test for None, they will get an exception as soon as they try to use None as an exception, and then they will fix their broken code. Although I don't like the get_clock() API, I don't think this argument against it is a good one. Exceptions are the *usual* error-handling mechanism in Python, but they are not the *only* mechanism, there are others, and it is perfectly okay to use non-exception based failures when appropriate. This is one such example. "Return None on failure" is how re.match() and re.search() work, and it is a good design for when you have multiple fallbacks on failure. result = re.match(spam, s) or re.match(ham, s) or re.match(eggs, s) if result is None: raise ValueError('could not find spam, ham or eggs') This is a *much* better design than nested tries: try: result = re.match(spam, s) except ValueError: try: result = re.match(ham, s) except ValueError: try: result = re.match(eggs, s) except ValueError: raise ValueError('could not find spam, ham or eggs') Wow. Now *that* is ugly code. There's nothing elegant or Pythonic about being forced to write that out of a misplaced sense of purity. -- Steven

On 05Apr2012 08:50, Steven D'Aprano <steve@pearwood.info> wrote: | Although I don't like the get_clock() API, I don't think this argument against | it is a good one. Just to divert briefly; you said in another post you didn't like the API and (also/because?) it didn't help discoverability. My core objective was to allow users to query for clocks, and ideally enumerate and inspect all clocks. Without the caller having platform specific knowledge. Allowing for the sake of discussion that this is desirable, what would you propose as an API instead of get_clock() (and its friend, get_clocks() for enumeration, that I should stuff into the code). Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Q: How many user support people does it take to change a light bulb? A: We have an exact copy of the light bulb here and it seems to be working fine. Can you tell me what kind of system you have?

Cameron Simpson wrote:
Clocks *are* platform specific -- not just in their availability, but also in the fine details of their semantics and behaviour. I don't think we can or should try to gloss over this. If people are making decisions about timers without knowledge of what their platform supports, they're probably making poor decisions. Even the venerable time.time() and time.clock() differ between Linux and Windows.
The old ways are the best. We don't have math.get_trig() and math.get_trigs() functions for querying trigonometric functions, we just expose the functions directly. I think the way to enumerate and inspect all clocks is with the tried and true Python introspection tools that people use on all other functions: * use dir(time) to see a list of names available in the module * use help(time) to read their help * read the Fine Manual to find out more * use try... except... to detect the existence of a clock There's nothing special about clocks that needs anything more than this. get_clock() looks like a factory function, but it actually isn't. It just selects from a small number of pre-existing clocks. We should just expose those pre-existing clocks directly. I don't see any advantage in adding that extra level of indirection or the addition of all this complexity: * a function get_clock() to select a clock * a function get_clocks() to enumerate all the clocks * another function for querying the properties of a clock All those functions accomplish is to increase the complexity of the API, the documentation and the implementation. It's one more special case for the user to learn: "To find out what functions are available, use dir(module), except for clocks, where you have to use time.get_clocks()." Blah. Another problem with get_clock() -- it will be an attractive nuisance for the sort of person who cares about symmetry and completeness. You will have a steady trickle of "feature requests" from users who are surprised that not every combination of features is supported. Out of the eight or sixteen or thirty-two potential clocks that get_clock() tempts the user with, only three or five will actually exist. The only advantage of get_clock is that you don't need to know the *name* of a platform clock in order to use it, you can describe it with a series of flags or enums. But in practice, that's not an advantage, that's actually a disadvantage. Consider: "Which clock should I use for such-and-such a task, foo or bar?" versus "Which clock should I use for such-and-such a task, get_clock(spam, eggs, cheese) or get_clock(ham, eggs, truffles)?" The mere mechanics of talking about these clocks will suffer because they aren't named. -- Steven

On 06Apr2012 20:25, Steven D'Aprano <steve@pearwood.info> wrote: | Cameron Simpson wrote: | > My core objective was to allow users to query for clocks, and ideally | > enumerate and inspect all clocks. Without the caller having platform | > specific knowledge. | | Clocks *are* platform specific -- not just in their availability, but also in | the fine details of their semantics and behaviour. I don't think we can or | should try to gloss over this. This is why get_clock() returns a clock object, which can have metadata exposing such details. Up to and including the name of the platform specific library/system-call at its core. The issue with monotonic() on its own is that the guarentees in the doco will have to be fairly loose. That prevents the user learning about "fine details of their semantics and behaviour". Glossing over this stuff is exactly what offering _only_ a few genericly characterised clock names (monotonic() et al) does. | If people are making decisions about timers | without knowledge of what their platform supports, they're probably making | poor decisions. Even the venerable time.time() and time.clock() differ between | Linux and Windows. time.clock() does, as (you?) clearly demonstrated elsewhere. time.time()? (Aside from precision?) | > Allowing for the sake of discussion that this is desirable, what would | > you propose as an API instead of get_clock() (and its friend, get_clocks() | > for enumeration, that I should stuff into the code). | | The old ways are the best. We don't have math.get_trig() and math.get_trigs() | functions for querying trigonometric functions, we just expose the functions | directly. | | I think the way to enumerate and inspect all clocks is with the tried and true | Python introspection tools that people use on all other functions: | | * use dir(time) to see a list of names available in the module So, they see "monotonic". Does that tell them much about fine details? | * use help(time) to read their help Useful only to humans, not programs. | * read the Fine Manual to find out more Useful only to humans, not programs. | * use try... except... to detect the existence of a clock Useful only for a fixed list of defined name. Works fine for monotonic, highres, steady or whatever. And I would be ok with the module presenting these only where available and concealing them otherwise, thus raising AttributeError. Or ImportError ("from time import monotonic"). | There's nothing special about clocks that needs anything more than this. This I think is false. In fact, I think your own statement at the start about glossing over fine details goes against this. If I ask for a highres clock, I might well care _how_ precise it was. If I ask for a steady clock, I might well care how large its slews were. If I ask for a monotonic clock, I might well want to know if it tracks wall clock time (even if by magic) or elapsed system run time (eg time that stops increasing if the system is suspended, whereas wallclocks do not). Example: a wallclock is nice for log timestamps. A system run time clock is nice for profiling. They're both monotonic in some domain. | get_clock() looks like a factory function, but it actually isn't. It just | selects from a small number of pre-existing clocks. That number may still be a few. Victor's made it clear that Windows has a choice of possible highres clocks, UNIX clock_getres() offers several possible clock behaviours and an indication that a platform may have several clocks embodying a subset of these, and may indeed offer more clocks. | We should just expose | those pre-existing clocks directly. But exposing them _purely_ _by_ _name_ means inventing names for every single platform clock, and knowing those names per platform. time.clock() is a fine example where the name tells you nearly nothing about the clock behaviour. If the user cares about fine detail as you suggest they need to know their platform and have _external_ knowledge of the platform specifics; they can't inspect from inside the program. | I don't see any advantage in adding that | extra level of indirection or the addition of all this complexity: | * a function get_clock() to select a clock | * a function get_clocks() to enumerate all the clocks These are only two functions because the next alternative seemed an all_clocks= mode parameter, which changed the signature of the function return. Another alternative is the public lists-of-clocks. The point it to be able to enumerate all available clocks for consideration of their properties; get_clock() provides a simple way to coarsely say "a clock like _this_" for the common instances of "this". | * another function for querying the properties of a clock No, that's why you get a clock object back. You can examine it directly for defined metadata names (epoch, precision, underlying-os-clock-name, etc). In exactly the fashion you appear to want for the top level offerings: by knowing the metadata property names. | All those functions accomplish is to increase the complexity of the API, the | documentation and the implementation. It's one more special case for the user | to learn: | | "To find out what functions are available, use dir(module), except for clocks, | where you have to use time.get_clocks()." But dir(module) _will_ list monotonic et al anyway, and possibly matching public clock list names. get_clock() is only for when you want to dig around more flexibly. | Another problem with get_clock() -- it will be an attractive nuisance for the | sort of person who cares about symmetry and completeness. You will have a | steady trickle of "feature requests" from users who are surprised that not | every combination of features is supported. Out of the eight or sixteen or | thirty-two potential clocks that get_clock() tempts the user with, only three | or five will actually exist. And the optional "clocklist" parameter addresses such feaping creaturism by providing a hook for _other_ modules to offer a clock list. Such as a list of syntheic clocks with cool (or insane:-) properties. Without burdening the time module. | The only advantage of get_clock is that you don't need to know the *name* of a | platform clock in order to use it, you can describe it with a series of flags | or enums. But in practice, that's not an advantage, that's actually a | disadvantage. Consider: | | "Which clock should I use for such-and-such a task, foo or bar?" What's your list of foo, bah? Again, I'm not talking about removing monotonic et al. I'm talking about exposing the alternatives for when the chosen-by-the-module monotonic doesn't fit. | versus | "Which clock should I use for such-and-such a task, get_clock(spam, eggs, | cheese) or get_clock(ham, eggs, truffles)?" One hopes the user knows the task. Then they can specify cheese or truffles. Again, only if they feel they need to because the bare monotonic et al don't fit, or was too vague. | The mere mechanics of talking about these clocks will suffer because they | aren't named. But they _can_ be named! get_clock() is for when you don't know or care their names, only their behaviours! And also for when an available clock _wasn't_ one returned by the monotonic et al names. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ I do not trust thee, Cage from Hell, / The reason why I cannot tell, / But this I know, and know full well: / I do not trust thee, Cage from Hell. - Leigh Ann Hussey, leighann@sybase.com, DoD#5913

On 4/6/2012 4:11 PM, Cameron Simpson wrote:
Another alternative is the public lists-of-clocks.
After watching this thread with amusement and frustration, amusement because it is so big, and so many people have so many different opinions, frustration, because it seems that few of the clocks that are available are anywhere near ideal for any particular stated characteristic, and because none of the APIs presented provide a way for the user to specify the details of the characteristics of the desired clock, I think this idea of a list-of-clocks sounds better and better. Hopefully, for each system, the characteristics of each clock can be discovered, and fully characterized in available metadata for the clock... tick rate, or list of tick rates maximum variation of tick rate precision maximum "helicopter drop" jump delta monotonicity frequency of rollover or None base epoch value or None behavior during system sleep, hibernate, suspend, shutdown, battery failure, flood, wartime events, and acts of God. These last two may have values that are long prose texts full of political or religious rhetoric, such as the content of this thread :) any other characteristics I forgot to mention Of course, it is not clear that all of these characteristics can be determined based on OS/Version; hardware vendors may have different implementations. There should be a way to add new clock objects to the list, given a set of characteristics, and an API to retrieve them, at least by installing a submodule that provides access to an additional clock.

On 06Apr2012 17:30, Glenn Linderman <v+python@g.nevcal.com> wrote: | On 4/6/2012 4:11 PM, Cameron Simpson wrote: | > Another alternative is the public lists-of-clocks. | | After watching this thread with amusement and frustration, amusement | because it is so big, and so many people have so many different | opinions, frustration, because it seems that few of the clocks that are | available are anywhere near ideal for any particular stated | characteristic, My partner has occasionally opined that most Prolog programs simply result in "*** NO ***". We could optimise for that and simplify the implementation enormously. It would also let us provide very strong guarrentees about the offered clocks on the basis that no suitable clock would ever provided:-) | and because none of the APIs presented provide a way for | the user to specify the details of the characteristics of the desired | clock, I think this idea of a list-of-clocks sounds better and better. | | Hopefully, for each system, the characteristics of each clock can be | discovered, and fully characterized in available metadata for the clock... Victor has asked me to do that for my skeleton, based on the tables he has assembled. I'll see what i can do there... | Of course, it is not clear that all of these characteristics can be | determined based on OS/Version; hardware vendors may have different | implementations. If you can look up the kernel revision you can do fairly well. In principle. | There should be a way to add new clock objects to the list, given a set | of characteristics, and an API to retrieve them, at least by installing | a submodule that provides access to an additional clock. Returning to seriousness, the get_clock() call admits a clocklist. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Principles have no real force except when one is well fed. - Mark Twain

Lennart Regebro wrote:
That's a matter of opinion. I'm not particularly fond of this get_clock idea, but of the two examples given, I much prefer the first of these: get_clock(MONOTONIC|HIRES) get_clock(monotonic=True, highres=True) and not just because it is shorter. The API is crying out for enum arguments, not a series of named flags. But frankly I think this get_clock API sucks. At some earlier part of this thread, somebody listed three or four potential characteristics of clocks. If we offer these as parameters to get_clock(), that means there's eight or sixteen different clocks that the user can potentially ask for. Do we really offer sixteen different clocks? Or even eight? I doubt it -- there's probably only two or three. So the majority of potential clocks don't exist. With get_clock, discoverability is hurt. How does the caller know what clocks are available? How can she look for documentation for them? A simple, obvious, discoverable API is best. If we offer three clocks, we have three named functions. If some of these clocks aren't available on some platform, and we can't emulate them, then simply don't have that named function available on that platform. That's easy to discover: trying to use that clock will give a NameError or AttributeError, and the caller can then fall back on an alternative, or fail, whichever is appropriate. -- Steven

On Wed, Apr 4, 2012 at 9:04 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
If I were looking at that in documentation, my automatic guess would be that the only thing that matters is whether the argument compares-as-true or not. So get_clock(monotonic="yes") would be the same as =True, and =False wouldn't be. And get_clock(monotonic="No, you idiot, I want one that ISN'T") would... be stupid. But it'd still function :) Chris Angelico

On Wed, Apr 4, 2012 at 13:04, Victor Stinner <victor.stinner@gmail.com> wrote:
It depends if the option supports other values. But as I understood, the keyword value must always be True.
Or False, obviously. Which would also be default. //Lennart

2012/4/4 Lennart Regebro <regebro@gmail.com>:
Ok for the default, but what happens if the caller sets an option to False? Does get_clock(monotonic=False) return a non-monotonic clock? (I guess no, but it may be confusing.) Victor

On Thu, Apr 5, 2012 at 01:10, Victor Stinner <victor.stinner@gmail.com> wrote:
Good point, but the same does for using flags. If you don't pass in the MONOTONIC flag, what happens? Only reading the documentation will tell you. As such this, if anything, is an indication that the get_clock() API isn't ideal in any incarnation. //Lennart

On 05Apr2012 10:21, Lennart Regebro <regebro@gmail.com> wrote: | On Thu, Apr 5, 2012 at 01:10, Victor Stinner <victor.stinner@gmail.com> wrote: | > Ok for the default, but what happens if the caller sets an option to | > False? Does get_clock(monotonic=False) return a non-monotonic clock? | > (I guess no, but it may be confusing.) This is where the bitmap approach can be less confusing - the docstring says "The returned clock shall have all the requested flags". It is at least very predictable. | Good point, but the same does for using flags. Only notionally. With a keyword argument the (lazy non doc reading) caller can imagine the default is None, and True and False specify concrete position and negative requirements. Not the case with a bitmask, which only has two states per feature, not three (or arbitrarily many, depending how nasty one wants to be - I could play devil's advocate and ask for monotonic=0.7 and demand a competivtive evaluation of relative merits:-) | If you don't pass in | the MONOTONIC flag, what happens? Only reading the documentation will | tell you. Gah! ALL functions are like that! How often do we see questions about max() or split() etc that a close reading of the docs obviate? | As such this, if anything, is an indication that the | get_clock() API isn't ideal in any incarnation. It's not meant to be ideal. I find that word almost useless in its overuse. get_clock() is meant to be _very_ _often_ _useful_ and easy to use for expressing simple fallback when the PEP418 monotonic() et al calls don't fit. For the truly arbitrary case the caller needs to be able to enumerate all the available clocks and make their own totally ad hoc decision. My current example code offers both public lock list names and get_clocks() (just like get_clock() in signature, but returning all matches instead of just the first one). Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ From the EXUP mailing list <exup-brotherhood@Majordomo.net> ... Wayne Girdlestone <WayneG@mega.co.za>: WG> Let's say there are no Yamaha's or Kawa's in the world. Stevey Racer <ssturm@co.la.ca.us>: SR> sriw - so then you are saying that Revelations (from the Bible) has come SR> true and Hell is now on Earth. WG> Your choice for you new bike is either a new '98 fuel injected SRAD, or a WG> new '98 Fireblade. SR> sriw -The devil's minions - full of temptation but never fulfilling their SR> promise.

Le 06/04/2012 00:17, Cameron Simpson a écrit :
By the way, I removed ("deferred") the time.highres() function from the PEP, and I try to avoid the term "steady" because no OS clock respect the definition of "steady" (especially in corner cases as system suspend/resume). So which flags do you want to support? (only "monotonic"?) Basically, get_clock("monotonic") should give time.monotonic() whereas get_clock() gives time.time()? Victor

On 06Apr2012 00:27, Victor Stinner <victor.stinner@gmail.com> wrote: | Le 06/04/2012 00:17, Cameron Simpson a écrit : | > This is where the bitmap approach can be less confusing - the docstring | > says "The returned clock shall have all the requested flags". It is at | > least very predictable. | | By the way, I removed ("deferred") the time.highres() function from the | PEP, Chuckle; was not the whole PEP for a high res clock? | and I try to avoid the term "steady" because no OS clock respect | the definition of "steady" (especially in corner cases as system | suspend/resume). I can think of definitions of "steady" that I personally would accept, and they'd accept that suspend/resume would be concealed (I guess I would usually want - purely myself here - a clock representing system run time; I'd go for time.time() for wall clock). | So which flags do you want to support? (only "monotonic"?) I'd stay with my current list, with metadata in the clock objects indicating what _flavour_ of "steady" or "high res" they present. | Basically, get_clock("monotonic") should give time.monotonic() whereas If time.monotonic() never falls back to a non-monotonic source, yes. | get_clock() gives time.time()? Might in theory give something better, but time.time() would always be a valid result of nothing else seemed better to the module author. I imagine in practice that time.time() might always use the "best" clock absent special requirements. So you'd probably get what particular clock used to implement time.time(), yes. (I realise this has interesting implications for the list orders; time.time() would come _first_, but providing feature flags to get_clock() can cause it not to be chosen when it doesn't match.) This a reason why I think we should present (even privately only) all the system clocks for a platform. Then you _can_ still offer highres() and steady() with detailed qualifications in the docs as to what considerations went into acepting a clock as highres or steady, and therefore why some users may find them unsatisfactory i.e. under what sort of circumstances/requirements they may not suit. Any of the montonic()/highres()/steady() represent policy decisions by the module author; it is just that monotonic() is easier to qualify than the others: "never goes backwards in return value". Even though VMs and system suspend can add depth to the arguments. It _is_ useful for people to be able to reach for highres() or steady() a lot of the time; they do, though, need to be able to decide if that's sensible. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ I thought back to other headaches from my past and sneered at their ineffectiveness. - Harry Harrison

On 06Apr2012 08:51, I wrote: | On 06Apr2012 00:27, Victor Stinner <victor.stinner@gmail.com> wrote: | | By the way, I removed ("deferred") the time.highres() function from the | | PEP, | | Chuckle; was not the whole PEP for a high res clock? Gah. I see it was for montonic, not high res. Sorry. [...] | I can think of definitions of "steady" that I personally would accept, | and they'd accept that suspend/resume would be concealed (I guess I | would usually want - purely myself here - a clock representing system | run time; I'd go for time.time() for wall clock). | | | So which flags do you want to support? (only "monotonic"?) | | I'd stay with my current list, with metadata in the clock objects | indicating what _flavour_ of "steady" or "high res" they present. On reflection, since I have historically presumed time.time() on UNIX mapped to "man 2 time", I clearly think that time.time() is wall clock time, and may jump when the sysadmin notices it is incorrect (of course this is often mediated by NTP, which in turn is usually mediated by some ntpd using adjtime(), which slews instead of jumping). But it might jump. (I'm intending to jump a wayard VM today, in fact:-) So guess I expect time.time() to be only usually steady. And usually monotonic. So having neither flag. Do I want a WALLCLOCK flag? Meaning a clock that is supposed to be real world time (did I see REALTIME in one of your examples?), and may be almost arbirarily corrected to be made real world if it is wrong. Maybe. +0 on that I think. Basicly I'm distinguishing here between a clock used for timestamps, for example in log entries, and a clock used for measuring elapsed system run time, for example in benchmarking. I would want to log entries to match what a clock on the wall should say. So I think I'm _still_ for the three original flags I suggested (monotonic, high res, steady) and expect time.time() to not necessarily meet any of them. But to meet a hypothetical WALLCLOCK flag. Regarding UNIX time(2) (or POSIX time(3)), POSIX says: The time() function shall return the value of time in seconds since the Epoch. and the epoch is a date. So UNIX time() should be a wall clock. Python "help(time.time)" says: Return the current time in seconds since the Epoch. So I think it should also be a wall clock by that same reasoning. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Uh, this is only temporary...unless it works. - Red Green

On Fri, Apr 6, 2012 at 00:17, Cameron Simpson <cs@zip.com.au> wrote:
Gah! ALL functions are like that! How often do we see questions about max() or split() etc that a close reading of the docs obviate?
My point exactly. //Lennart

I don't know who started this, but the PEP 418 threads have altogether too much snarkiness and not enough content. It's bad enough that we're bikeshedding so intensely; we don't need clever comebacks in triplicate to every out-of-context argument. --Guido On Fri, Apr 6, 2012 at 2:26 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
-- --Guido van Rossum (python.org/~guido)

On 04Apr2012 09:53, Steven D'Aprano <steve@pearwood.info> wrote: | Lennart Regebro wrote: | > On Tue, Apr 3, 2012 at 08:03, Cameron Simpson <cs@zip.com.au> wrote: | >> clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) | >> If the symbol names are not the horribleness, can you qualify what API | >> you would like more? | > | > Well, get_clock(monotonic=True, highres=True) would be a vast | > improvement over get_clock(MONOTONIC|HIRES).[...] | | That's a matter of opinion. I'm not particularly fond of this get_clock idea, | but of the two examples given, I much prefer the first of these: | | get_clock(MONOTONIC|HIRES) | get_clock(monotonic=True, highres=True) | | and not just because it is shorter. The API is crying out for enum arguments, | not a series of named flags. Enums would be ok with me. I went with a bitmask because it is natural to me and very simple. But anything symbolicly expression will do. | But frankly I think this get_clock API sucks. At some earlier part of this | thread, somebody listed three or four potential characteristics of clocks. If | we offer these as parameters to get_clock(), that means there's eight or | sixteen different clocks that the user can potentially ask for. Do we really | offer sixteen different clocks? Or even eight? I doubt it -- there's probably | only two or three. So the majority of potential clocks don't exist. That's not the point. I think we should offer all the platform system clocks, suitably described. That there are up to 8 or 16 flag combinations is irrelevant; no user is going to try them all. A user will have requirements for their clock. They ask for them either blandly via get_clock() or (for example considering monotonic most important) via monotonic_clock(). In the latter case, the supported clocks can be considered in a more apt order via a different internal clock list. | With get_clock, discoverability is hurt. No, because the other calls still exist. (In my proposal. I see Victor's characterised this as either/or in the PEP, never my intent.) | How does the caller know what clocks | are available? I would definitely want either: - the module clock lists available via public names, for example as in my sample clockutils.py code (ALL_CLOCKS, MONTONIC_CLOCKS etc) or via some map (eg clocks['monotonic']). - a get_clocks() function to return matching clocks, like get_clock() but not stopping on the first match - an all_clocks=False parameter to get_clock() to get an iterable of the suitable clocks | How can she look for documentation for them? There is good text in the PEP. That could be easily moved into the module doco in a "clocks" section. Since my clocks proposal wraps clocks in an object, they _can_ have nice class names and good docstrings and more metadata in the object (possibilities including .epoch, .precision, .is_steady() methods, .os_clock_name (eg "QueryPerformanceCounter"), etc). | A simple, obvious, discoverable API is best. If we offer three clocks, we have | three named functions. If some of these clocks aren't available on some | platform, and we can't emulate them, then simply don't have that named | function available on that platform. That's easy to discover: trying to use | that clock will give a NameError or AttributeError, and the caller can then | fall back on an alternative, or fail, whichever is appropriate. And I hate this. Because many platforms offer several OS clocks. The time module SHOULD NOT dictate what clocks you get to play with, and you should not need to have platform specific knowledge to look for a clock with your desired characteristics. If you just want montonic() and trust the module authors' policy decisions you can go with monotonic(), have it do AttributeError if unavailable and never worry about discoverability or the inspectable object layer. Many will probaby be happy with that. But without get_clock() or something like it, there is no discoverability and not ability for a user to decide their own clock choice policy. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Your modesty is typically human, so I will overlook it. - a Klingon

Why not passing a a list of set of flags? Example: haypo_steady = get_clock(MONOTONIC|STEADY, STEADY, MONOTONIC, REALTIME) # try to get a monotonic and steady clock, # or fallback to a steady clock, # or fallback to a monotonic clock, # or fallback to the system clock haypo_perf_counter = get_clock(HIGHRES, MONOTONIC|STEADY, STEADY, MONOTONIC, REALTIME) # try to get a high-resolution clock # or fallback to a monotonic and steady clock, # or fallback to a steady clock, # or fallback to a monotonic clock, # or fallback to the system clock On Windows, haypo_steady should give GetTickCount (MONOTONIC|STEADY) and haypo_perf_counter should give QueryPerformanceCounter (MONOTONIC|HIGHRES). Hum, I'm not sure that haypo_highres uses the same clocks than time.perf_counter() in the PEP.
And if don't read the doc carefuly and forget the test, you can a "NoneType object is not callable" error.
It's better to avoid unnecessary system calls at startup (when the time module is loaded), but you may defer the creation of the clock list, or at least of the flags of each clock.
A list of clocks and a function are maybe redundant. Why not only providing a function?
My PEP starts with use cases: it proposes one clock per use case. There is no "If you need a monotonic, steady and high-resolution clock ..." use case. The "highres" name was confusing, I just replaced it with time.perf_counter() (thanks Antoine for the name!). time.perf_counter() should be used for benchmarking and profiling.
I mean having to choose the flags *and* the list of clocks is hard. I would prefer to only have to choose flags or only the list of clocks. The example was maybe not the best one.
You can solve this issue with only one list of clocks if you use the right set of flags.
So what is the minimum resolution and/or accuracy of the HIGHRES flag?
A full implementation would help to decide which API is the best one. "Full" implementation: - define all convinience function - define all list of clocks - define flags of all clocks listed in the PEP 418: clocks used in the pseudo-code of time.steady and time.perf_counter, and maybe also time.time Victor

On 04Apr2012 01:45, Victor Stinner <victor.stinner@gmail.com> wrote: | > | get_clock() returns None if no clock has the requested flags, whereas | > | I expected an exception (LookupError or NotImplementError?). | > | > That is deliberate. People can easily write fallback like this: | > | > clock = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_MONOTONIC) | | Why not passing a a list of set of flags? Example: | | haypo_steady = get_clock(MONOTONIC|STEADY, STEADY, MONOTONIC, REALTIME) | # try to get a monotonic and steady clock, | # or fallback to a steady clock, | # or fallback to a monotonic clock, | # or fallback to the system clock That's interesting. Ethan Furman suggested multiple arguments to be combined, whereas yours bundles multiple search criteria in one call. While it uses a bitmask as mine does, this may get cumbersome if we went with Nick's "use strings!" suggestion. | haypo_perf_counter = get_clock(HIGHRES, MONOTONIC|STEADY, STEADY, | MONOTONIC, REALTIME) | # try to get a high-resolution clock | # or fallback to a monotonic and steady clock, | # or fallback to a steady clock, | # or fallback to a monotonic clock, | # or fallback to the system clock | | On Windows, haypo_steady should give GetTickCount (MONOTONIC|STEADY) | and haypo_perf_counter should give QueryPerformanceCounter | (MONOTONIC|HIGHRES). Sounds ok to me. I am not familiar with the Windows counters and am happy to take your word for it. | Hum, I'm not sure that haypo_highres uses the same clocks than | time.perf_counter() in the PEP. | | > If one wants an exception it is easy to follow up with: | > if not clock: | > raise RunTimeError("no suitable clocks on offer on this platform") | | And if don't read the doc carefuly and forget the test, you can a | "NoneType object is not callable" error. Excellent! An exception either way! Win win! | > | get_clock() doesn't remember if a clock works or not (if it raises an | > | OSError) and does not fallback to the next clock on error. See | > | "pseudo-codes" in the PEP 418. | > | > I presume the available clocks are all deduced from the platform. Your | > pseudo code checks for OSError at fetch-the-clock time. I expect that | > to occur once when the module is loaded, purely to populate the table | > of avaiable platform clocks. | | It's better to avoid unnecessary system calls at startup (when the | time module is loaded), but you may defer the creation of the clock | list, or at least of the flags of each clock. Yes indeed. I think this should be deferred until use. | > Note that you don't need to provide a clock list at all; get_clock(0 | > will use ALL_CLOCKS by default, and hires() and monotonic() should each | > have their own default list. | | A list of clocks and a function are maybe redundant. Why not only | providing a function? Only because the function currently only returns one clock. The picky user may want to peruse all the clocks inspecting other metadata (precision etc) than the coarse flag requirements. There should be a way to enumerate the available clock implementation; in my other recent post I suggest either lists (as current), a get_clocks() function, or a mode parameter to get_clock() such as _all_clocks, defaulting to False. | > Regarding the choice itself: as the _caller_ (not the library author), | > you must decide what you want most. You're already planning offering | > monotonic() and hires() calls without my proposal! | | My PEP starts with use cases: it proposes one clock per use case. | There is no "If you need a monotonic, steady and high-resolution clock | ..." use case. Yes. but this is my exact objection to the "just provide hires() and steady() and/or monotonic()" API; the discussion to date is littered with "I can't imagine wanting to do X" style remarks. We should not be trying to enumerate the user case space exhaustively. I'm entirely in favour of your list of use cases and the approach of providing hires() et al to cover the thought-common use cases. But I feel we really _must_ provide a way for the user with a not-thought-of use case to make an arbitrary decision. get_clock() provides a simple cut at the "gimme a suitable clock" approach, with the lists or other "get me an enumeration of the available clocks" mechanism for totally ad hoc perusal if the need arises. This is also my perhaps unstated concern with Guido's "the more I think about it, the more I believe these functions should have very loose guarantees, and instead just cater to common use cases -- availability of a timer with minimal fuss is usually more important than the guarantees" http://www.mail-archive.com/python-dev@python.org/msg66173.html The easy to use hires() etc must make very loose guarentees or they will be useless too often. That looseness is fine in some ways - it provides availability on many platforms (all?) and discourages the user from hoping for too much and thus writing fragile code. But it also PREVENTS the user from obtaining a really good clock if it is available (where "good" means their partiuclar weirdo feature requirements). So I think there should be both - the easy and simple calls, and a mecahnism for providing all clocks for the user to chose with arbitrary criteria and fallback. | The "highres" name was confusing, I just replaced it with | time.perf_counter() (thanks Antoine for the name!). | time.perf_counter() should be used for benchmarking and profiling. I've been wondering; do we distinguish between clocks and counters. In my mind a clock or timer has a very linear relationship with "real time", the wall clock. A counter, by comparison, may measure CPU cycles or kernel timeslice ticks or python opcode counts or any number of other time-like resource consumption things. I've been presuming we're concerned here with "clocks" and not counters. | > Taking your query "Should | > I use MONTONIC_CLOCKS or HIRES_CLOCKS when I would like a monotonic and | > high-resolution clock" is _already_ a problem. Of course you must call | > monotonic() or hires() first under the current scheme, and must answer this | > question anyway. Do you prefer hires? Use it first! No preference? Then the | > question does not matter. | | I mean having to choose the flags *and* the list of clocks is hard. I | would prefer to only have to choose flags or only the list of clocks. | The example was maybe not the best one. Yah; I think I made a followup post where I realised you may have meant this. The use of default arguments is meant to make it easy to use flags and/or lists or even neither (which for get_clock() at least would always get you a clock because a wrapper for time.time() is always provided). In my mind, usually just flags of course. | > | If you have only one list of clocks, how do sort the list to get | > | QueryPerformanceCounter when the user asks for highres and | > | GetTickCount when the user asks for monotonic? | > | > This is exactly why there are supposed to be different lists. | > You have just argued against your objection above. | | You can solve this issue with only one list of clocks if you use the | right set of flags. No you can't, not in general. If there are multiple clocks honouring those flags you only ever get the first one in the list. The point of the MONOTONIC_CLOCKS list etc is that the lists may be differently ordered to provide quality of clock within that domain. Suppose I ask for steady_clock(MONOTONIC); I would probably prefer a more-steady clock over a more-precise/hires clock. And converse desires when asking for hires_clock(). So different list orders, at least in principle. If it turns out empirically that this isn't the case then all the names can in fact refer to the same list. But offering only one list _name_ prevents offering these nuances when other platforms/clocks come in the future. | > | So we would have: | > | | > | GetTickCount.flags = T_MONOTONIC | T_STEADY | T_HIGHRES | > | | > | Even if GetTickCount has only an accuracy of 15 ms :-/ | > | > T_HIGHRES is a quality call, surely? If 15ms is too sloppy for a "high | > resolution, the is should _not_ have the T_HIRES flag. | | So what is the minimum resolution and/or accuracy of the HIGHRES flag? No idea. But you must in principle have one in mind to offer the hires() call at all in the PEP, or be prepared to merely offer the most hires clock available regardless. In this latter case you would always mark that clock as having the HIRES flag and the problem is solved. It would be good to have metadata to indicate how hires a partiulcar clock is. | > | Could you please update your code according to my remarks? I will try | > | to integrate it into the PEP. A PEP should list all alternatives! | > | > Surely. | > | > The only updates I can see are to provide the flat interface | > (instead of via clock-object indirection) and the missing hires_clock() | > and monotonic_clock() convenience methods. | | A full implementation would help to decide which API is the best one. | "Full" implementation: | | - define all convinience function | - define all list of clocks Ok. My current code is here, BTW: https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.p... (Finally found a revision independent URL on bitbucket.) | - define flags of all clocks listed in the PEP 418: clocks used in | the pseudo-code of time.steady and time.perf_counter, and maybe also | time.time I'll make one. It will take a little while. Will post again when ready. At present the code compiles and runs (albeit with no platform specific clocks:-) This table may require fictitious code. Should still compile I guess... Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ The right to be heard does not include the right to be taken seriously. - Hubert Horatio Humphrey

Victor et al, Just an update note: I've started marking up clocks with attributes; not yet complete and I still need to make a small C extension to present the system clocks to Python space (which means learning to do that, too). But you can glance over the start on it here: https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.p... Several feature flags and some properties for qualifying clocks. Still needed stuff includes: C access to clocks, .accuracy being actual clock precision versus the resolution of the units in the underlying OS call, a __repr__ and/or __str__ to decode feature bitmaps into useful strings, .is_*() __getattr__ method to resolve against flags by name or maybe has_flag(str), etc. On 07Apr2012 01:16, Victor Stinner <victor.stinner@gmail.com> wrote: | > | This is the original reason for the original defect (issue 10278) | > | unix' clock() doesn't actually provide a clock in this sense, | > | it provides a resource usage metric. | > | > Yeah:-( Its help says "Return the CPU time or real time since [...]". | > Two very different things, as demonstrated. I suppose neither goes | > backwards, but this seems like a classic example of the "useless | > monotonic clock" against which Greg Ewing railed. | > | > And why? For one thing, because one can't inspect its metadata to find | > out what it does. | | Should I add another key to the result of | time.get_clock_info('clock')? How can we define "clock on Windows" | (almost monotonic and steady clock) vs "clock on UNIX" (CPU time) with | a flag or a value? For clocks I'm going with two feature flags: WALLCLOCK and RUNTIME. The former indicates a clock that tries to stay in synch with real world time, and would still advance when the system is suspended or idle; it would almost certainly need to "step" over a suspend. The latter means system run time; it accrues only while the system is up. Neither is CPU usage (so a time.sleep(10) would add 10 seconds to both). I think resource usage is not a "clock". We could characterise such timers and counters with a lot of the same metrics we like to use with clocks, but I do not think they should be returned by a system purporting to return clocks or clock values. On 07Apr2012 14:22, I wrote: | On 06Apr2012 17:30, Glenn Linderman <v+python@g.nevcal.com> wrote: | | Hopefully, for each system, the characteristics of each clock can be | | discovered, and fully characterized in available metadata for the clock... | | Victor has asked me to do that for my skeleton, based on the tables he | has assembled. I'll see what i can do there... I've started on this, see above. Victor Stinner <victor.stinner@gmail.com> wrote: | | - define flags of all clocks listed in the PEP 418: clocks used in | | the pseudo-code of time.steady and time.perf_counter, and maybe also | | time.time | | I'll make one. It will take a little while. Will post again when ready. So, new code to glance over as evidence of good faith, if not speed:-( Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Life is uncertain. Eat dessert first. - Jim Blandy

FWIW, I'm not sure you're the right person to drive time PEPs. You don't seem to have come into it with much knowledge of time, and it's taken several repetitions for you to take corrections into account in both this discussion and the Decimal/datetime representation PEP. On Mon, Mar 26, 2012 at 4:32 PM, Victor Stinner <victor.stinner@gmail.com> wrote:

On Tue, Mar 27, 2012 at 3:51 PM, Jeffrey Yasskin <jyasskin@gmail.com> wrote:
The main things required to be a PEP champion are passion and a willingness to listen to expert feedback and change course in response. If someone lacks the former, they will lose steam and their PEP will eventually be abandoned. If they don't listen to expert feedback, then their PEP will ultimately be rejected (sometimes a PEP will be rejected anyway as a poor fit for the language *despite* being responsive to feedback, but that's no slight to the PEP author). Victor has shown himself to be quite capable of handling those aspects of the PEP process, and the topics he has recently applied himself to are ones where it is worthwhile having a good answer in the standard library for Python 3.3. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I started to write the PEP 418 to clarify the notions of monotonic and steady clocks.
I replaced time.steady() by time.try_monotonic(). I misunderstood "may not" in the C++ doc: I understood it as "it may be adjusted by NTP", whereas it means "it cannot be adjusted". Sorry for the confusion. I added a time.hires() clock because time.monotonic() and time.try_monotonic() are not the best clocks for profiling or benchmarking. For example, on Windows, time.hires() uses QueryPerformanceCounter() whereas time.monotonic() and time.try_monotonic() uses GetTickCount[64](). I added the pseudo-code of each function. I hope that it is easier to understand than a long text. Victor

time.steady() doesn't fit the benchmarking use case: it looks like we have to decide between stability and clock resolution. QueryPerformanceCounter() has a good resolution for benchmarking, but it is not monotonic and so GetTickCount64() would be used for time.steady(). GetTickCount64() is monotonic but has only a resolution of 1 millisecond. We might add a third new function which provides the most accurate clock with or without a known starting point. We cannot use QueryPerformanceCounter() to enhance time.time() resolution because it has an unknown starting point. Victor

On 3/26/2012 7:32 PM, Victor Stinner wrote:
I started to write the PEP 418 to clarify the notions of monotonic and steady clocks.
""" time.steady This clock advances at a steady rate relative to real time. It may be adjusted. """ Please do not call this "steady". If the clock can be adjusted, then it is not "steady" by any acceptable definition. I cannot fathom the utility of this function other than as a function that provides an automatic fallback from "time.monotonic()". More importantly: this definition of "steady" is in conflict with the C++0x definition of "steady" that is where you sourced this named from![1] """ time.steady(strict=False) falls back to another clock if no monotonic clock is not available or does not work, but it does never fail. """ As I say above, that is so far away from what "steady" implies that this is a misnomer. What you are describing is a best-effort clock, which sounds a lot more like the C++0x "high resolution" clock. """ time.steady(strict=True) raises OSError if monotonic clock fails or NotImplementedError if the system does not provide a monotonic clock """ What is the utility of "strict=True"? If I wanted that mode of operation, then why would I not just try to use "time.monotonic()" directly? At worst, it generates an "AttributeError" (although that is not clear from your PEP). What is the use case for "strict=True" that is not covered by your "time.monotonic()"? If you want to define new clocks, then I wish you would use the same definitions that C++0x is using. That is: system_clock = wall clock time monotonic_clock = always goes forward but can be adjusted steady_clock = always goes forward and cannot be adjusted high_resolution_clock = steady_clock || system_clock Straying from that is only going to create confusion. Besides that, the one use case for "time.steady()" that you give (benchmarking) is better served by a clock that follows the C++0x definition. As well, certain kinds of scheduling/timeouts would be better implemented with the C++0x definition for "steady" rather than the "monotonic" one and vice-versa. Rather, it seems you have a particular use-case in mind and have settled on calling that a "steady" clock despite how it belies its name. [1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3128.html#time.cloc... """ Objects of class steady_clock represent clocks for which values of time_point advance at a steady rate relative to real time. That is, the clock may not be adjusted. """ -- Scott Dial scott@scottdial.com

Note that the C++ standard deprecated monotonic_clock once they realized that there is absolutely no point in having a clock that jumps forward but not back, and that none of the operating systems implement such a thing -- instead they all implement a clock which doesn't jump in either direction. http://stackoverflow.com/questions/6777278/what-is-the-rationale-for-renamin... In other words, yes! +1! The C++ standards folks just went through the process that we're now going through, and if we do it right we'll end up at the same place they are: http://en.cppreference.com/w/cpp/chrono/system_clock """ system_clock represents the system-wide real time wall clock. It may not be monotonic: on most systems, the system time can be adjusted at any moment. It is the only clock that has the ability to map its time points to C time, and, therefore, to be displayed. steady_clock: monotonic clock that will never be adjusted high_resolution_clock: the clock with the shortest tick period available """ Note that we don't really have the option of providing a clock which is "monotonic but not steady" in the sense of "can jump forward but not back". It is a misunderstanding (doubtless due to the confusing name "monotonic") to think that such a thing is offered by the underlying platforms. We can choose to *call* it "monotonic", following POSIX instead of calling it "steady", following C++. Regards, Zooko

So does anyone care to dig into the libstd++/boost/windoze implementation to see how they each did steady_clock?

On 3/26/2012 10:59 PM, Matt Joiner wrote:
So does anyone care to dig into the libstd++/boost/windoze implementation to see how they each did steady_clock?
The Boost implementation can be summarized as: system_clock: mac = gettimeofday posix = clock_gettime(CLOCK_REALTIME) win = GetSystemTimeAsFileTime steady_clock: mac = mach_absolute_time posix = clock_gettime(CLOCK_MONOTONIC) win = QueryPerformanceCounter high_resolution_clock: * = { steady_clock, if available system_clock, otherwise } Whether or not these implementations meet the specification is an exercise left to the reader.. -- Scott Dial scott@scottdial.com

On Mar 26, 2012, at 10:26 PM, Zooko Wilcox-O'Hearn wrote:
This is why I don't like the C++ terminology, because it seems to me that the C++ standard makes incorrect assertions about platform behavior, and apparently they standardized it without actually checking on platform capabilities. The clock does jump forward when the system suspends. At least some existing implementations of steady_clock in C++ already have this problem, and I think they all might. I don't think they can fully fix it without kernel changes, either. On linux, see discussion of a possible CLOCK_BOOTTIME in the future. The only current way I know of to figure out how long the system has been asleep is to look at the wall clock and compare, and we've already gone over the problems with relying on the wall clock. Plus, libstdc++ gives you no portable way to get informed about system power management events, so you can't fix it even if you know about this problem, natch. Time with respect to power management state changes is something that the PEP should address fully, for each platform. On the other hand, hopefully you aren't controlling your python-based CNC laser welder from a laptop that you are closing the lid on while the beam is in operation. Not that the PEP shouldn't address it, but maybe it should just address it to say "you're on your own" and refer to a few platform-specific resources for correcting this type of discrepancy. (<https://developer.apple.com/library/mac/#qa/qa1340/_index.html>, <http://msdn.microsoft.com/en-us/library/aa394362.aspx>, <http://upower.freedesktop.org/docs/UPower.html#UPower::Sleeping>). -glyph

On Mar 27, 2012, at 3:17 AM, Glyph wrote:
I don't think they can fully fix it without kernel changes
I got really curious about this and went and did some research. With some really platform-specific hackery on every platform, you can mostly figure it out; completely on OS X and Windows, although (as far as I can tell) only partially on Linux and FreeBSD. I'm not sure if it's possible to make use of these facilities without a Twisted-style event-loop though. If anybody's interested, I recorded the results of my research in a comment on the Twisted ticket for this: <http://twistedmatrix.com/trac/ticket/2424#comment:26>. -glyph

Reading this discussion, my conclusion is that not only us are confused, but everyone is. I think therefore, that the way forward is to only expose underlying API functions, and pretty much have no intelligence at all. At a higher level, we have two different "desires" here. You may want a monotonic clock, or you may not care. You may want high resolution, or you might not care. Which one is more important is something only you know. Therefore, we must have, at the minimum, a function that returns the highest resolution monotonic clock possible, as well as a function that returns the highest resolution system/wall clock possible. We also need ways to figure out what the resolution is of these clocks. In addition to that, you may have the requirement that the monotonic clock also should not be able to jump forward, but if I understand things correctly, most current OS's will not guarantee this. You may also have the requirement that the clock not only does not jump forward, but that it doesn't go faster or slower. Some clock implementations will speed up or slow down the monotonic clock, without jumps, to sync up with the wall clock. It seems only Unix provides a monotonic clock (CLOCK_MONOTONIC_RAW) that does not get adjusted at all. Now between all these requirements, only you know which one is more important? Do you primarily want a raw monotonic clock, and secondarily high resolution, or is the resolution more important than it being monotonic? (Because if you need a high resolution, you are usually measuring small timeframes, and the clock is more unlikely to be adjusted, for example). Since there is no obvious "A is better than B that is better than C" we first of all have to expose the underlying API's somehow, to allow people to make their own decisions. Secondly, since apparently not only python-dev, but many others as well, are a bit confused on this complex issue, I'm not sure we can provide any high-level functions that makes a best choice. As such the proposed time.monotonic() to get the monotonic clock on the various systems makes a lot of sense to me. It should get the highest resolution available on the system. Get GetTickCount64() of available on Windows, else GetTickCount(). The function could have a raw=False parameter to select between clock_gettime(CLOCK_MONOTONIC) and clock_gettime(CLOCK_MONOTONIC_RAW) on Unix, and it would get mach_absolute_time() on OS X. If no monotonic clock is available, it should raise an error.The same if you pass in raw=True and there is no monotonic clock that has no adjustments available. In the same vein, time.time() should provide the highest resolution system clock/wall clock available. We also need functions or attributes to get the resolution of these clocks. But a time.steady() that tries to get a "best case" doesn't make sense at this time, as apparently nobody knows what a best case is, or what it should be called, except that it should apparently not be called steady(). Since monotonic() raises an error if there is no monotonic clock available, implementing your own fallback is trivial in any case. //Lennart

On Tue, Mar 27, 2012 at 7:03 PM, Lennart Regebro <regebro@gmail.com> wrote:
+1 from me to Lennart's suggestion of mostly just exposing time.monotonic() without trying to get too clever. Applications that need a truly precise time source should *never* be reading it from the host OS (one fairly good solution can be to read your time directly from an attached GPS device). However, I think Victor's right to point out that the *standard library* needs to have a fallback to maintain backwards compatibility if time.monotonic() isn't available, and it seems silly to implement the same fallback logic in every module where we manipulate timeouts. As I concur with others that time.steady() is a thoroughly misleading name for this concept, I suggest we encapsulate the "time.monotic if available, time.time otherwise" handling as a "time.try_monotic()" function. That's simple clear and explicit: try_monotic() tries to use the monotic clock if it can, but falls back to time.time() rather than failing entirely if no monotonic clock is available. This is essential for backwards compatibility when migrating any current use of time.time() over to time.monotic(). Yes the monotonic clock is *better* for many use cases (including timeouts), but you'll usually be OK with the non-monotonic clock, too (particularly if that's what you were using anyway in earlier versions). After all, we've survived this long using time.time() for our timeout calculations, and bugs associated with clock adjustments are a rather rare occurrence. Third party libraries that need to support earlier Python versions can then implementation their own fallback logic (since they couldn't rely on time.try_monotonic being available either). The 3.3 time module would then be left with three interfaces: time.time() # Highest resolution timer available time.monotonic() # I'm not yet convinced we need the "raw" parameter but don't much mind either way time.try_monotonic() # Monotonic is preferred, but non-monotonic presents a tolerable risk Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2012-03-27, at 9:23 AM, Nick Coghlan wrote:
time.try_monotonic() # Monotonic is preferred, but non-monotonic presents a tolerable risk
This function seems unnecessary. It's easy to implement it when required in your application, hence I don't think it is worth adding to the stdlib. - Yury

I renamed time.steady() to time.try_monotonic() in the PEP. It's a temporary name until we decide what to do with this function.
How about get rid of it? Also monotonic should either not exist if it's not available, or always guarantee a (artificially) monotonic value. Finding out that something is already known to not work shouldn't require a call and a faked OSError.

On 27/03/2012 20:18, Matt Joiner wrote:
Also monotonic should either not exist if it's not available, or always guarantee a (artificially) monotonic value.
What do you mean by "(artificially) monotonic value"? Should Python workaround OS bugs by always returning the maximum value of the clock?
Finding out that something is already known to not work shouldn't require a call and a faked OSError.
What do you mean? time.monotonic() is not implemented if the OS doesn't provide any monotonic clock (e.g. if clock_gettime(CLOCK_MONOTONIC) is missing on UNIX). OSError is only raised when the OS returns an error. There is no such "faked OSError". Victor

Matt, we need the fallback behaviour in the stdlib so we can gracefully degrade the stdlib's *own* timeout handling back to the 3.2 status quo when there is no monotic clock available. It is *not* acceptable for the Python 3.3 stdlib to only work on platforms that provide a monotonic clock. Since duplicating that logic in every module that handles timeouts would be silly, it makes sense to provide an obvious way to do it in the time module. -- Sent from my phone, thus the relative brevity :)

I don't think that Python should workaround OS issues, but document them correctly. I started with this sentence for time.monotonic(): "The monotonic clock may stop while the system is suspended." I don't know exactly how clocks behave with system suspend. Tell me if you have more information.
I will read these links and maybe add them to the PEP. Victor

I mentioned the strict=True API in the PEP just to list all propositions, but the PEP only proposes time.monotonic() and time.try_monotonic(), no the flags API.
At worst, it generates an "AttributeError" (although that is not clear from your PEP).
I tried to mention when a function is always available or not always available. Is it better in the last version of the PEP?
I tried to follow these names in the PEP. I don't propose steady_clock because I don't see exactly which clocks would be used to implement it, nor if we need to provide monotonic *and* steady clocks. What do you think?
I added a time.hires() clock to the PEP for the benchmarking/profiling use case. This function is not always available and so a program has to fallback manually to another clock. I don't think that it is an issue: Python programs already have to choose between time.clock() and time.time() depending on the OS (e.g. timeit module and pybench program).
Sorry, I don't understand. Which kind of scheduling/timeouts? The PEP is still a draft (work-in-progress). Victor

On 27/03/2012 18:45, Victor Stinner wrote:
It is this always-having-to-manually-fallback-depending-on-os that I was hoping your new functionality would avoid. Is time.try_monotonic() suitable for this usecase? Michael
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Scott wrote: << The Boost implementation can be summarized as: system_clock: mac = gettimeofday posix = clock_gettime(CLOCK_REALTIME) win = GetSystemTimeAsFileTime steady_clock: mac = mach_absolute_time posix = clock_gettime(CLOCK_MONOTONIC) win = QueryPerformanceCounter high_resolution_clock: * = { steady_clock, if available system_clock, otherwise } >> I read again the doc of the QElapsedTimer class of the Qt library. So Qt and Boost agree to say that QueryPerformanceCounter() *is* monotonic. I was confused because of a bug found in 2006 in Windows XP on multicore processors. QueryPerformanceCounter() gave a different value on each core. The bug was fixed in Windows and is known as KB896256 (I already added a link to the bug in the PEP).
If QueryPerformanceCounter() is monotonic, the API can be simplified to: * time.time() = system clock * time.monotonic() = monotonic clock * time.hires() = monotonic clock or fallback to system clock time.hires() definition is exactly what I was trying to implement with "time.steady(strict=True)" / "time.try_monotonic()". -- Scott> monotonic_clock = always goes forward but can be adjusted Scott> steady_clock = always goes forward and cannot be adjusted I don't know if the monotonic clock should be called time.monotonic() or time.steady(). The clock speed can be adjusted by NTP, at least on Linux < 2.6.28. I don't know if other clocks used by my time.monotonic() proposition can be adjusted or not. If I understand correctly, time.steady() cannot be implemented using CLOCK_MONOTONIC on Linux because CLOCK_MONOTONIC can be adjusted? Does it really matter if a monotonic speed is adjusted? Victor

On Mar 28, 2012 8:38 AM, "Victor Stinner" <victor.stinner@gmail.com> wrote:
and Boost agree to say that QueryPerformanceCounter() *is* monotonic.
I was confused because of a bug found in 2006 in Windows XP on multicore
processors. QueryPerformanceCounter() gave a different value on each core. The bug was fixed in Windows and is known as KB896256 (I already added a link to the bug in the PEP).
"time.steady(strict=True)" / "time.try_monotonic()".
time.steady(). The clock speed can be adjusted by NTP, at least on Linux < 2.6.28. Monotonic. It's still monotonic if it is adjusted forward, and that's okay.
I don't know if other clocks used by my time.monotonic() proposition can
be adjusted or not.
If I understand correctly, time.steady() cannot be implemented using
CLOCK_MONOTONIC on Linux because CLOCK_MONOTONIC can be adjusted?
http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com

On Wed, Mar 28, 2012 at 10:36 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
Please don't call the fallback version "hires" as it suggests it may be higher resolution than time.time() and that's completely the wrong idea. If we're simplifying the idea to only promising a monotonic clock (i.e. will never go backwards within a given process, but may produce the same value for an indefinite period, and may jump forwards by arbitrarily large amounts), then we're back to being able to enforce monotonicity even if the underlying clock jumps backwards due to system clock adjustments. Specifically: time.time() = system clock time._monotonic() = system level monotonic clock (if it exists) time.monotonic() = clock based on either time._monotonic() (if available) or time.time() (if not) that enforces monotonicity of returned values. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Georg Brandl wrote:
Precisely. I always read "hires" as the verb hires (as in "he hires a car to go on holiday") rather than HIgh RESolution. -1 on hires, it's a horrible name. And misleading as well, because on Linux, it isn't any more high res than time.time(). +1 on Nick's suggestion of try_monotonic. It is clear and obvious and doesn't mislead. I don't have an opinion as to what the implementation of try_monotonic should be. Whether it should fall back to time.time, time.clock, or something else, I don't know. But it is a clear and obvious solution for the use-case of "I prefer the monotonic clock, if it is available, otherwise I'll take my chances with a best-effect clock." -- Steven

On Wed, 28 Mar 2012 23:05:59 +1100, Steven D'Aprano <steve@pearwood.info> wrote:
+1 on Nick's suggestion of try_monotonic. It is clear and obvious and doesn't mislead.
How about "monotonicest". (No, this is not really a serious suggestion.) However, time.steadiest might actually work. --David

Why would it be a wrong idea? On Windows, time.monotonic() frequency is at least 1 MHz (can be GHz if it uses your CPU TSC) whereas time.time() is only updated each millisecond at the best case (each 15 ms by default if I remember correctly). On UNIX, CLOCK_MONOTONIC has the same theorical resolution than CLOCK_REALTIME (1 nanosecond thanks to the timespec structure) and I expect the same accuracy. On Mac, I don't know if mach_absolute_time() is more or as accurate than time.time(). time.hires() uses time.monotonic() if available, so if time.monotonic() has an higher resolution than time.time(), time.hires() can also be called a high-resolution clock. In practice, time.monotonic() is available on all modern platforms.
I don't know any monotonic clock jumping "forwards by arbitrarily large amounts". Linux can change CLOCK_MONOTONIC speed, but NTP doesn't "jump".
Do you know a monotonic clock that goes backward? If yes, Python might workaround the clock bug directly in time.monotonic(). But I would prefer to *not* workaround OS bugs. Victor

On Wed, Mar 28, 2012 at 6:40 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
If I understood Glyph's explanation correctly, then if your application is running in a VM and the VM is getting its clock data from the underlying hypervisor, then suspending and resuming the VM may result in forward jumping of the monotonic clocks in the guest OS. I believe suspending and hibernating may cause similar problems for even a non-virtualised OS that is getting its time data from a real-time clock chip that keeps running even when the main CPU goes to sleep. (If I *misunderstood* Glyph's explanation, then he may have only been talking about the latter case) Monotonicity is fairly easy to guarantee - you just remember the last value you returned and ensure you never return a lower value than that for the lifetime of the process. The only complication is thread synchronisation, and the GIL (or a dedicated lock for Jython/IronPython) can deal with that. Steadiness, on the other hand, requires a real world time reference and is thus really the domain of specialised hardware like atomic clocks and GPS units rather than software that can be suspended and resumed later without changing its internal state. There's a reason comms station operators pay substantial chunks of money for time & frequency reference devices [1]. This is why I now think we only need one new clock function: time.monotonic(). It will be the system monotonic clock if one is available, otherwise it will be our own equivalent wrapper around time.time() that just caches the last value returned to ensure the result never goes backwards. With time.monotonic() guaranteed to always be available, there's no need for a separate function that falls back to an unconditioned time.time() result. Regards, Nick. [1] For example: http://www.symmetricom.com/products/gps-solutions/gps-time-frequency-receive... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Mar 28, 2012 at 7:36 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
As I said, I think the caching idea is bad. We may have to settle for semantics that are less than perfect -- presumably if you are doing benchmarking you just have to throw away a bad result that happened to be affected by a clock anomaly, and if you are using timeouts, retries are already part of life.
I would love to have only one new clock function in 3.3.
-- --Guido van Rossum (python.org/~guido)

On Thu, Mar 29, 2012 at 12:42 AM, Guido van Rossum <guido@python.org> wrote:
I agree caching doesn't solve the problems that are solved by an OS level monotonic clock, but falling back to an unmodifided time.time() result instead doesn't solve those problems either. Falling back to time.time() just gives you the status quo: time may jump forwards or backwards by an arbitrary amount between any two calls. Cached monotonicity just changes the anomalous modes to be time jumping forwards, or time standing still for an extended period of time. The only thing the caching provides is that it becomes a reasonable fallback for a function called time.monotonic() - it *is* a monotonic clock that meets the formal contract of the function, it's just nowhere near as good or effective as one the OS can provide. Forward jumping anomalies aren't as harmful, are very hard to detect in the first place and behave the same regardless of the presence of caching, so the interesting case to look at is the difference in failure modes when the system clock jumps backwards. For benchmarking, a caching clock will produce a zero result instead of a negative result. Zeros aren't quite as obviously broken as negative numbers when benchmarking, but they're still sufficiently suspicious that most benchmarking activities will flag them as anomalous. If the jump back was sufficiently small that the subsequent call still produces a higher value than the original call, then behaviour reverts to being identical. For timeouts, setting the clock back means your operation will take longer to time out than you expected. This problem will occur regardless of whether you were using cached monotonicity (such that time stands still) or the system clock (such that time actually goes backwards). In either case, your deadline will never be reached until the backwards jump has been cancelled out by the subsequent passage of time. I want the standard library to be able to replace its time.time() calls with time.monotonic(). The only way we can do that without breaking cross-platform compatibility is if time.monotonic() is guaranteed to exist, even when the platform only provides time.time(). A dumb caching fallback implementation based on time.time() is the easiest way to achieve that withou making a complete mockery of the "monotonic()" name. There is then a *different* use case, which is 3.3+ only code which wants to fail noisily when there's no OS level monotonic support - the application developer really does want to fail *immediately* if there's no OS level monotonic clock available, instead of crossing your fingers and hoping you don't hit a clock adjustment glitch (crossing your fingers has, I'll point out, been the *only* option for all previous versions of Python, so it clearly can't be *that* scary a prospect). So, rather than making time.monotonic() something that the *standard library can't use*, I'd prefer to address that second use case by exposing the OS level monotonic clock as time.os_monotonic() only when it's available. That way, the natural transition for old time.time() based code is to time.monotonic() (with no cross-platform support implications), but time.os_monotonic() also becomes available for the stricter use cases. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Mar 28, 2012 at 8:08 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
TBH, I don't think like this focus on monotonicity as the most important feature.
Agreed.
So for benchmarking we don't care about jumps, really, and the caching version is slightly less useful.
Where in the stdlib do we actually calculate timeouts instead of using the timeouts built into the OS (e.g. select())? I think it would be nice if we could somehow use the *same* clock as the OS uses to implement timeouts.
I want the standard library to be able to replace its time.time() calls with time.monotonic().
Where in the stdlib? (I'm aware of threading.py. Any other places?)
Yeah, so maybe it's a bad name. :-)
I'd be happier if the fallback function didn't try to guarantee things the underlying clock can't guarantee. I.e. I like the idea of having a function that uses some accurate OS clock if one exists but falls back to time.time() if not; I don't like the idea of that new function trying to interpret the value of time.time() in any way. Applications that need the OS clock's guarantees can call it directly. We could also offer something where you can introspect the properties of the clock (or clocks) so that an app can choose the best clock depending on its needs. To summarize my problem with the caching idea: take a simple timeout loop such as found in several places in threading.py. def wait_for(delta, eps): # Wait for delta seconds, sleeping eps seconds at a time deadline = now() + delta while now() < deadline: sleep(eps) If the now() clock jumps backward after the initial call, we end up waiting too long -- until either the clock jumps forward again or until we've made up the difference. If the now() clock jumps forward after the initial call, we end up waiting less time, which is probably not such a big problem (though it might). But now consider a caching clock, and consider that the system clock made a jump backwards *before* this function is called. The cache prevents us from seeing it, so the initial call to now() returns the highest clock value seen so far. And until the system clock has caught up with that, now() will return the same value over and over -- so WE STILL WAIT TOO LONG. My conclusion: you can't win this game by forcing the clock to return a monotonic value. A better approach might be to compute how many sleep(eps) calls we're expected to make, and to limit the loop to that -- although sleep() doesn't make any guarantees either about sleeping too short or too long. Basically, if you do sleep(1) and find that your clock didn't move (enough), you can't tell the difference between a short sleep and a clock that jumped back. And if your clock moved to much, you still don't know if the problem was with sleep() or with your clock. -- --Guido van Rossum (python.org/~guido)

time.timeout_clock? Everyone knows what that will be for and we won't have to make silly theoretical claims about its properties and expected uses. If no one else looks before I next get to a PC I'll dig up the clock/timing source used for select and friends, and find any corresponding syscall that retrieves it for Linux.

On Thu, Mar 29, 2012 at 1:47 AM, Guido van Rossum <guido@python.org> wrote:
Where in the stdlib? (I'm aware of threading.py. Any other places?)
Victor had at least one other example. multiprocessing, maybe? I believe the test suite may still have a few instances as well.
Ouch. OK, I'm convinced the caching fallback is worse than just falling back to time.time() directly, which means the naming problem needs to be handled another way.
With your point about the problem with the naive caching mechanism acknowledged, I think we can safely assign time.monotonic() as the name of the OS provided monotonic clock. That means choosing a name for the version that falls back to time() if monotonic() isn't available so it can be safely substituted for time.time() without having to worry about platform compatibility implications. I don't like Victor's current "hires" (because it doesn't hint at the fallback behaviour, may actually be the same res as time.time() and reads like an unrelated English word). My own suggestion of "try_monotic()" would get the job done, but is hardly going to win any API beauty contests. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Mar 29, 2012 at 1:14 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
What's wrong with "time.time()" again? As documented in http://docs.python.org/py3k/library/time.html it makes no guarantees, and specifically there is *no* guarantee that it will ever behave *badly*<wink/>. Of course, we'll have to guarantee that, if a badly-behaved clock is available, users can get access to it, so call that time._time().

I'm not sure I understand your suggestion correctly, but replacing time.time() by time.monotonic() with fallback won't work, because time.monotonic() isn't wall-clock time: it can very well use an arbitrary reference point (most likely system start-up time). As for the hires() function, since there's no guarantee whatsoever that it does provide a better resolution than time.time(), this would be really misleading IMHO.

Where in the stdlib do we actually calculate timeouts instead of using the timeouts built into the OS (e.g. select())?
At least in threading and queue modules. The common use case is to retry a function with a timeout if the syscall was interrupted by a signal (EINTR error). The socket module and _threading.Lock.acquire() implement such retry loop using the system clock. They should use a monotonic clock instead.
I think it would be nice if we could somehow use the *same* clock as the OS uses to implement timeouts.
On Linux, nanosleep() uses CLOCK_MONOTONIC whereas POSIX suggests CLOCK_REALTIME. Some functions allow to choose the clock, like pthread locks or clock_nanosleep().
We may workaround some OS known bugs like: http://support.microsoft.com/?id=274323 The link contains an example how to workaround the bug. The idea of the workaround is to use two different monotonic clocks to detect leaps, with one trusted clock (GetTickCount) and one untrusted clock having an higher resolution (QueryPerformanceCounter). I don't think that the same algorithm is applicable on other OSes because other OSes usually only provide one monotonic clock, sometimes though different API. Victor

On 2012-03-28, at 10:36 AM, Nick Coghlan wrote:
As I said in my previous mail - I don't think we should ever do that. Time may jump back and forth, and with your approach it will result in monotonic() being completely unusable. If time jumps back for N minutes, or years, that leads to completely broken expectations for timeouts for N minutes or years correspondingly (and that's just the timeouts case, I'm sure that there are much more critical time-related use-cases.) If monotonic() will utilize such hack, you add nothing usable in stdlib. Every serious framework or library will have to re-implement it using only OS-level functions, and *FAIL* if the OS doesn't support monotonic time. Fail, because such framework can't guarantee that it will work correctly. So I think time module should have only one new function: monotonic(), and this function should be only available if OS provides the underlying functionality. No need for steady(), try_monotonic() and other hacks. Each module can decide if its dependancy on monotonic is critical or not, and if it is not, you can always have: try: from time import monotonic as _time except ImportError: from time import time as _time That's how lots of code is written these days, like using 'epoll' if available, and fallback to 'select' if not. Why don't you try to abstract differences between them in the standard library? So I see no point in adding some loose abstractions to the stdlib now. - Yury

On 3/27/2012 8:36 PM, Victor Stinner wrote:
You are right that CLOCK_MONOTONIC can be adjusted, so the Boost implementation is wrong. I'm not sure that CLOCK_MONOTONIC_RAW is right either due to suspend -- there doesn't appear to be a POSIX or Linux clock that is defined that meets the "steady" definition. I am not familiar enough with Windows or Mac to know for certain whether the Boost implementation has the correct behaviors either. With that in mind, it's certainly better that we just provide time.monotonic() for now. If platform support becomes available, then we can expose that as it becomes available in the future. In other words, at this time, I don't think "time.steady()" can be implemented faithfully for any platform so lets just not have it at all. In that case, I don't think time.try_monotonic() is really needed because we can emulate "time.monotonic()" in software if the platform is deficient. I can't imagine a scenario where you would ask for a monotonic clock and would rather have an error than have Python fill in the gap with an emulation. -- Scott Dial scott@scottdial.com

The term "adjusted" should be clarified. A clock can be adjusted by setting its counter (e.g. setting the system date and time) or by changing temporary its frequency (to go faster or slower). Linux only adjusts CLOCK_MONOTONIC frequency but the clock is monotonic because it always goes forward. The monotonic property can be described as: t1=time.monotonic() t2=time.monotonic() assert t2 >= t1
time.hires() is needed when the OS doesn't provide any monotonic clock and because time.monotonic() must not use the system clock (which can jump backward). As I wrote, I don't think that Python should workaround OS bugs. If the OS monotonic clock is not monotonic, the OS should be fixed.
Sorry, I don't understand what you mean with "fill in the gap with an emulation". You would like to implement a monotonic clock based on the system clock? Victor

On 3/28/2012 4:48 AM, Victor Stinner wrote:
I agree. The point I was making is that implication of "steady" is that (t2-t1) is the same (given that t2 and t1 occur in time at the same relative moments), which is a guarantee that I don't see any platform providing currently. Any clock that can be "adjusted" in any manner is not going to meet the "steady" criterion.
I sympathize with this, but if the idea is that the Python stdlib should use time.monotonic() for scheduling, then it needs to always be available. Otherwise, we are not going to use it ourselves, and what sort of example is that to set?
If "time.monotonic()" is only sometimes available, then I don't see the added clock being anything more than an amusement. (In this case, I'd rather just use clock_gettime() and friends directly, because I have to be platform aware anyways.) What developers want is a timer that is useful for scheduling things to happen after predictable interval in the future, so we should give them that to the best of our ability. -- Scott Dial scott@scottdial.com

Victor, I have completely lost track of the details of this discussion. Could you (with help from others who contributed) try to compile a table showing, for each platform (Windows/Mac/Linux/BSD) which clocks (or variations) we are considering, and for each of those: - a link for the reference documentation - what their typical accuracy is (barring jumps) - what they do when the "civil" time is made to jump (forward or back) by the user - how they are affected by small tweaks to the civil time by NTP - what they do if the system is suspended and resumed - whether they can be shared between processes running on the same machine - whether they may fail or be unsupported under some circumstances I have a feeling that if I saw such a table it would be much easier to decide. I assume much of this has already been said at one point in this thread, but it's impossible to have an overview at the moment. If someone has more questions they'd like to see answered please add to the list. -- --Guido van Rossum (python.org/~guido)

On Wed, Mar 28, 2012 at 8:56 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
Completely unintuitive and unnecessary. With the GIL taking care of synchronisation issues, we can easily coerce time.time() into being a monotonic clock by the simple expedient of saving the last returned value: def _make_monotic: try: # Use underlying system monotonic clock if we can return _monotonic except NameError: _tick = time() def monotic(): _new_tick = time() if _new_tick > _tick: _tick = _new_tick return _tick monotonic = _make_monotonic() Monotonicity of the result is thus ensured, even when using time.time() as a fallback. If using the system monotonic clock to get greater precision is acceptable for an application, then forcing monotonicity shouldn't be a problem either. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Mar 29, 2012 at 12:27 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
You have to keep in mind the alternative here: falling back to an *unconditioned* time.time() value (which is the status quo, and necessary to preserve backwards compatibility). That will break just as badly in that scenario and is precisely the reason that the OS level monotonic functionality is desirable in the first place. I'd be quite happy with a solution that made the OS level monotonic clock part of the public API, with the caveat that it may not be available. Then the necessary trio of functions would be: time.time(): existing system clock, always available time.os_monotonic(): OS level monotonic clock, not always available time.monotonic(): always available, same as os_monotonic if it exists, otherwise uses a time() based emulation that may not be consistent across processes and may "mark time" for extended periods if the underlying OS clock is forced to jump back a long way. I think that naming scheme is more elegant than using monotonic() for the OS level monotonicity and try_monotonic() for the fallback version, but I'd be OK with the latter approach, too. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2012-03-28, at 10:45 AM, Nick Coghlan wrote:
Well, my argumentation is that you either have some code that depends on monotonic time and can't work without it, or you have a code that can work with any time (and only precision matters). Maybe I'm wrong.
I still don't like this 'emulation' idea. Smells bad for standard lib. Big -1 on this approach. - Yury

On Thu, Mar 29, 2012 at 1:02 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
You're wrong. The primary use case for the new time.monotonic() function is to replace *existing* uses of time.time() in the standard library (mostly related to timeouts) that are currently vulnerable to clock adjustment related bugs. This real, concrete use case has been lost in some of the abstract theoretical discussions that have been going on this thread. We can't lose sight of the fact that using a system clock that is vulnerable to clock adjustment bugs to handle timeouts and benchmarking in Python has worked just fine for 20+ years. Using a monotonic clock instead is *better*, but it's far from essential, since clock adjustments that are big enough and poorly timed enough to cause real problems are fortunately a very rare occurrence. So, the primary use case is that we want to replace many of the time.time() calls in the standard library with time.monotonic() calls. To avoid backwards compatibility problems in the cross-platform support, that means time.monotonic() *must be available on every platform that currently provides time.time()*. This is why Victor's original proposal was that time.monotonic() simply fall back to time.time() if there was no OS level monotonic clock available. The intended use cases are using time.time() *right now* and have been doing so for years, so it is clearly an acceptable fallback for those cases. People (rightly, in my opinion) objected to the idea of time.monotonic() failing to guarantee monotonicity, thus the proposal to enforce at least a basic level of monotonicity through caching of the last returned value. I agree completely that this dumb caching solution doesn't solve any of the original problems with time.time() that make a time.monotonic() function desirable, but it isn't meant to. It's only meant to provide graceful degradation to something that is *no worse than the current behaviour when using time.time() in Python 3.2* while still respecting the property of monotonicity for the new API. Yes, it's an ugly hack, but it is a necessary fallback to avoid accidental regressions in our cross-platform support. For the major platforms (i.e. *nix, Mac OS X, Windows), there *will* be an OS level monotonic clock available, thus using time.monotonic() will have the desired effect of protecting from clocks being adjusted backwards. For other platforms, the behaviour (and vulnerabilities) will be essentially unchanged from the Python 3.2 approach (i.e. using time.time() with no monotonicity guarantees at all). However, some 3.3+ applications may want to be stricter about their behaviour and either bail out completely or fall back to an unfiltered time.time() call if an OS-level monotonic clock is not available. For those, it makes sense to expose time.os_monotonic() directly (and only if it is available), thus allowing those developers to make up their own mind instead of accepting the cross-platform fallback in time.monotonic(). Yes, you can get the exact same effect with the "monotonic()" and "try_monotonic()" naming scheme, but why force the standard library (and anyone else wanting to upgrade from time.time() without harming cross-platform support) to use such an ugly name when the "os_monotonic" and "monotonic" naming scheme provides a much neater alternative? Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2012-03-28, at 11:35 AM, Nick Coghlan wrote:
OK. I got your point. And also I've just realized what I dislike about the way you want to implement the fallback. The main problem is that I treat the situation when time jumps backward as an exception, because, again, if you have timeouts you may get those timeouts to never be executed. So let's make the "try_monotonic()" function (or whatever name will be chosen) this way (your original code edited): def _make_monotic(): try: # Use underlying system monotonic clock if we can return _monotonic except NameError: _tick = time() def monotic(): nonlocal _time _new_tick = time() if _new_tick <= _tick: raise RuntimeError('time was adjusted backward') _tick = _new_tick return _new_tick return monotonic try_monotonic = _make_monotonic() At least this approach tries to follow some of the python's zen. - Yury

On Wed, Mar 28, 2012 at 7:17 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
That's a pretty obvious trick. But why don't the kernels do this if monotonicity is so important? I'm sure there are also downsides, e.g. if the clock is accidentally set forward by an hour and then back again, you wouldn't have a useful clock for an hour. And the cache is not shared between processes so different processes wouldn't see the same clock value (I presume that most of these clocks have state in the kernel that isn't bound to any particular process -- AFAIK only clock() does that, and only on Unixy systems). -- --Guido van Rossum (python.org/~guido)

On 3/28/2012 10:29 AM, Guido van Rossum wrote:
What makes you think that isn't already true? I don't know what platforms that CPython compiles for that *won't* have one of the aforementioned functions available that provide a *real* monotonic clock. Surely, any platform that doesn't didn't recognize the need for it, or they would just provide a monotonic clock. That is to say, if you are a POSIX compliant system, then there is no reason to break gettimeofday() and friends when you can just implement CLOCK_MONOTONIC proper (even if it's just a trick like Nick's). I think the PEP should enumerate what platforms that CPython supports that will not benefit from a real monotonic clock. I think the number of platforms will be such a minority that the emulation makes sense. Practicality beats purity, and all. -- Scott Dial scott@scottdial.com

Nick Coghlan wrote:
Here's a version that doesn't suffer from the flaw of returning a long stream of constant values when the system clock jumps backwards a significant amount: class MockTime: def __init__(self): self.ticks = [1, 2, 3, 4, 2, 3, 4, 5, 7, 3, 4, 6, 7, 8, 8, 9] self.i = -1 def __call__(self): self.i += 1 return self.ticks[self.i] time = MockTime() _prev = _prev_raw = 0 def monotonic(): global _prev, _prev_raw raw = time() delta = max(0, raw - _prev_raw) _prev_raw = raw _prev += delta return _prev And in use:
[monotonic() for i in range(16)] [1, 2, 3, 4, 4, 5, 6, 7, 9, 9, 10, 12, 13, 14, 14, 15]
Time: [1, 2, 3, 4, 2, 3, 4, 5, 7, 3, 4, 6, 7, 8, 8, 9] Nick: [1, 2, 3, 4, 4, 4, 4, 5, 7, 7, 7, 7, 7, 8, 8, 9] Mine: [1, 2, 3, 4, 4, 5, 6, 7, 9, 9, 10, 12, 13, 14, 14, 15] Mine will get ahead of the system clock each time it jumps back, but it's a lot closer to the ideal of a *strictly* monotonically increasing clock. Assuming that the system clock will never jump backwards twice in a row, the double-caching version will never have more than two constant values in a row. -- Steven

On Wed, Mar 28, 2012 at 12:56, Victor Stinner <victor.stinner@gmail.com> wrote:
There is time.hires() if you need a monotonic clock with a fallback to the system clock.
Does this primarily give a high resolution clock, or primarily a monotonic clock? That's not clear from either the name, or the PEP. //Lennart

Does this primarily give a high resolution clock, or primarily a monotonic clock? That's not clear from either the name, or the PEP.
I expect a better resolution from time.monotonic() than time.time(). I don't have exact numbers right now, but I began to document each OS clock in the PEP. Victor

On Wed, Mar 28, 2012 at 23:40, Victor Stinner <victor.stinner@gmail.com> wrote:
Sure. And for me that means that time.hires() would give a high resolution version of time.time(). Ie, not monotonic, but wall clock. The question then is why time.time() doesn't give that resolution from the start. It seems to me we need three functions: One to get the wall clock, one to get a monotonic clock, and one that falls back if no monotonic clock is available. Both time.time() and time.monotonic() should give the highest resolution possible. As such, time.hires() seems pointless. //Lennart

The overview of the different monotonic clocks was interesting, because only one of them is adjusted by NTP, and that's the unix CLOCK_MONOTONIC. Hence we don't need a raw=False flag, which I previously suggested, we only need to not use CLOCK_MONOTONIC (which the PEP psuedo-code indeed also does not do, so that's all good). That means I think the PEP is fine now, if we rename highres(). time.time() already gets the highest resolution clock it can. Hence a highres() is confusing as the name implies that it returns a higher resolution clock than time.time(). And the name does not in any way indicate that the returned clock might be monotonic. try_monotonic() seems the obvious choice, since that's what it actually does. //Lennart

On Fri, Mar 30, 2012 at 12:01 PM, Lennart Regebro <regebro@gmail.com> wrote:
Right on.
That means I think the PEP is fine now, if we rename highres(). time.time() already gets the highest resolution clock it can.
No, time.time() is the clock that can be mapped to and from "civil time". (Adjustments by NTP and the user notwithstanding.) The other clocks have a variable epoch and do not necessarily tick with a constant rate (e.g. they may not tick at all while the system is suspended).
I am still unhappy with the two names, but I'm glad that we're this close. We need two new names; one for an OS-provided clock that is "monotonic" or "steady" or whatever you want to call it, but which may not exist on all systems (some platforms don't have it, some host may not have it even though the platform generally does have it). The other name is for a clock that's one or the other; it should be the OS-provided clock if it exists, otherwise time.time(). Most code should probably use this one, so perhaps its name should be the shorter one. C++ calls these steady_clock and high_resolution_clock, respectively. But it also calls the civil time clock system_clock, so perhaps we shouldn't feel to bound by it (except that we *shouldn't* call something steady if it isn't). I still think the name "monotonic" give the wrong impression; I would be happy calling it steady. But for the other, I'm still at a loss, and that name is the most important one. We can't call it steady because it isn't always. highres or hires sounds awkward; try_monotonic or try_steady are even more awkward. I looked in an online thesaurus and here's a list of what it gave: Big Ben, alarm, chroniker, chronograph, chronometer, digital watch, hourglass, metronome, pendulum, stopwatch, sundial, tattler, tick-tock, ticker, timekeeper, timemarker, timepiece, timer, turnip, watch I wonder if something with tick would work? (Even though it returns a float. :-) If all else fails, I'd go with turnip. -- --Guido van Rossum (python.org/~guido)

On Fri, 30 Mar 2012 12:40:25 -0700, Guido van Rossum <guido@python.org> wrote:
We could call it "alice"[*]: sometimes it goes fast, sometimes it goes slow, sometimes it even goes backward, but it does try to tell you when you are late. --David [*] 'whiterabbit' would be more descriptive, but that's longer than turnip.

On 2012-03-30, at 3:40 PM, Guido van Rossum wrote:
I still think the name "monotonic" give the wrong impression; I would be happy calling it steady.
Simple google search comparison shows that people ask about 'monotonic' clock in python, not 'steady'. How about following Nick's (if I recall correctly) proposal of calling the OS function - '_monotonic', and a python wrapper - 'monotonic'? And one more question: what do you think about introducing a special check, that will ensure that our python implementation of 'monotonic', in case of fallback to 'time.time()', raises an exception if time suddenly goes backward? - Yury

Guido van Rossum wrote:
"hires" is a real English word, the present tense verb for engaging the service or labour of someone or something in return for payment, as in "he hires a gardener to mow the lawn". Can we please eliminate it from consideration? It is driving me slowly crazy every time I see it used as an abbreviation for high resolution.
I can't tell if you are being serious or not. For the record, "turnip" in this sense is archaic slang for a thick pocket watch. -- Steven

On Sat, Mar 31, 2012 at 02:26, Steven D'Aprano <steve@pearwood.info> wrote:
If I understand this correctly, the most common use for this function is when to time things. It will give you the best source available for timers, but it doesn't guarantee that it is steady or monotonic or high resolution or anything. It is also not the time, as it's not reliable as a wall-clock. So, how about time.timer()? //Lennart

On Sat, Mar 31, 2012 at 8:27 AM, Lennart Regebro <regebro@gmail.com> wrote:
So, how about time.timer()?
That seems like a bad idea; it would be too easy to confuse with (or misspell as) time.time(). Out of the big synonym list Guido posted, I rather like time.stopwatch() - it makes it more explicit that the purpose of the function is to measure intervals, rather identifying absolute points in time. Cheers, Nadeem

On 28Mar2012 23:40, Victor Stinner <victor.stinner@gmail.com> wrote: | > Does this primarily give a high resolution clock, or primarily a | > monotonic clock? That's not clear from either the name, or the PEP. | | I expect a better resolution from time.monotonic() than time.time(). I | don't have exact numbers right now, but I began to document each OS | clock in the PEP. I wish to raise an alternative to these set-in-stone policy-in-the-library choices, and an alternative to any proposal that does fallback in a function whose name suggests otherwise. Off in another thread on PEP 418 I suggested a cleaner approach to offering clocks to the user: let the user ask! My (just two!) posts on this are here: http://www.mail-archive.com/python-dev@python.org/msg66174.html http://www.mail-archive.com/python-dev@python.org/msg66179.html The second post is more important as it fleshes out my reasons for considering this appraoch better. I've just finished sketching out a skeleton here: https://bitbucket.org/cameron_simpson/css/src/fb476fcdcfce/lib/python/cs/clo... In short: - take Victor's hard work on system clocks and classifying thm by feature set - tabulate access to them in a list of clock objects - base access class goes (example user call): # get a clock object - often a singleton under the hood T = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_STEADY|T_HIRES) # what kind of clock did I get? print T.flags # get the current time now = T.now - offer monotonic() and/or steady() etc as convenience functions calling get_clock() in a fashion like the above example - don't try to guess the user's use case ahead of time This removes policy from the library functions and makes it both simple and obvious in the user's calling code, and also makes it possible for the user to inspect the clock and find out what quality/flavour of clock they got. Please have a glance through the code, especially the top and botth bits; it is only 89 lines long and includes (presently) just a simple object for time.time() and (importantly for the bikeshedding) an example synthetic clock to give a monotonic caching clock from another non-monotonic clock (default, again, time.time() in this prototype). Suitably fleshed out with access to the various system clocks, this can offer all the current bikeshedding in a simple interface and without constraining user choices to "what we thought of, or what we thought likely". Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Availability: Samples Q1/97 Volume H2/97 So, it's vapor right now, but if you want to sell vapor in 1997 you better had damn fast vapor then... - Burkhard Neidecker-Lutz on the DEC Alpha 21264, October 1996

I've just finished sketching out a skeleton here:
https://bitbucket.org/cameron_simpson/css/src/fb476fcdcfce/lib/python/cs/clo...
get_clock() returns None if no clock has the requested flags, whereas I expected an exception (LookupError or NotImplementError?). get_clock() doesn't remember if a clock works or not (if it raises an OSError) and does not fallback to the next clock on error. See "pseudo-codes" in the PEP 418. The idea of flags attached to each clock is interesting, but I don't like the need of different list of clocks. Should I use MONTONIC_CLOCKS or HIRES_CLOCKS when I would like a monotonic and high-resolution clock? It would be simpler to have only one global and *private* list. If you have only one list of clocks, how do sort the list to get QueryPerformanceCounter when the user asks for highres and GetTickCount when the user asks for monotonic? The "if clock.flags & flags == flags:" test in get_clock() is maybe not enough. I suppose that we would have the following flags for Windows functions: QueryPerformanceCounter.flags = T_HIRES GetTickCount.flags = T_MONOTONIC | T_STEADY (or maybe QueryPerformanceCounter.flags = T_HIRES | T_MONOTONIC ?) monotonic_clock() should maybe try to get a clock using the following list of conditions: - T_MONOTONIC | T_STEADY - T_MONOTONIC | T_HIGHRES - T_MONOTONIC The T_HIGHRES flag in unclear, even in the PEP. According to the PEP, any monotonic clock is considered as a "high-resolution" clock. Do you agree? So we would have: GetTickCount.flags = T_MONOTONIC | T_STEADY | T_HIGHRES Even if GetTickCount has only an accuracy of 15 ms :-/ Can list please give the list of flags of each clocks listed in the PEP? Only clocks used for time.time, time.monotonic and time.highres (not process and thread clocks nor QueryUnbiasedInterruptTime).
The API looks much more complex than the API proposed in PEP 418 just to get the time. You have to call a function to get a function, and then call the function, instead of just calling a function directly. Instead of returning an object with a now() method, I would prefer to get directly the function getting time, and another function to get "metadata" of the clock.
I'm not sure that users understand correctly differences between all these clocks and are able to use your API correctly. How should I combinese these 3 flags (T_HIRES, T_MONOTONIC and T_STEADY)? Can I use any combinaison? Which flags are "portable"? Or should I always use an explicit fallback to ensure getting a clock on any platform? Could you please update your code according to my remarks? I will try to integrate it into the PEP. A PEP should list all alternatives! Victor

On 4/2/2012 4:37 AM, Victor Stinner wrote:
If there are more than two clocks, with different characteristics, no API is going to be both simple to use and fast to call. If there are more than two clocks, with different characteristics, then having an API to get the right API to call to get a time seems very natural to me. One thing I don't like about the idea of fallback being buried under some API is that the efficiency of that API on each call must be less than the efficiency of directly calling an API to get a single clock's time. For frequently called high resolution clocks, this is more burdensome than infrequently called clocks.... yet those seem to be the ones for which fallbacks are proposed, because of potential unavailability. Having properties on each of various different clock functions is cumbersome... the user code must know about each clock, how to obtain the properties, and then how to choose one for use... And how will one be chosen for use? Under the assumption that all return some sort of timestamp and take no parameters, a local name will be assigned to the clock of interest: if ...: myTime = os.monotonous elif ...: myTime = os.evenhigherres ... elif ...: myTime = time. time so that myTime can be use throughout. Cameron's API hides all the names of the clocks, and instead offers to do the conditional logic for you, and the resultant API returned can be directly assigned to myTime, and the logic for choosing a clock deals only with the properties of the clock, not the names of the APIs, which is a nice abstraction. There would not even be a need to document the actual names of the APIs for each individual clock, except that probably some folks would want to directly code them, especially if they are not interested in cross-platform work. The only thing I'm not so sure about: can the properties be described by flags? Might it not be better to have an API that allows specification of minimum resolution, in terms of fractional seconds? Perhaps other properties suffice as flags, but perhaps not resolution.

On Tue, Apr 3, 2012 at 3:44 AM, Glenn Linderman <v+python@g.nevcal.com> wrote:
No, that's a misunderstanding of the fallback mechanism. The fallback happens when the time module is initialised, not on every call. Once the appropriate clock has been selected during module initialisation, it is invoked directly at call time. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 4/2/2012 2:40 PM, Nick Coghlan wrote:
I would hope that is how the fallback mechanism would be coded, but I'm pretty sure I've seen other comments in this thread that implied otherwise. But please don't ask me to find them, this thread is huge. Glenn

On 02Apr2012 14:59, Glenn Linderman <v+python@g.nevcal.com> wrote: | On 4/2/2012 2:40 PM, Nick Coghlan wrote: | > On Tue, Apr 3, 2012 at 3:44 AM, Glenn Linderman<v+python@g.nevcal.com> wrote: | >> > One thing I don't like about the idea of fallback being buried under some | >> > API is that the efficiency of that API on each call must be less than the | >> > efficiency of directly calling an API to get a single clock's time. | > No, that's a misunderstanding of the fallback mechanism. The fallback | > happens when the time module is initialised, not on every call. Once | > the appropriate clock has been selected during module initialisation, | > it is invoked directly at call time. | | I would hope that is how the fallback mechanism would be coded, but I'm | pretty sure I've seen other comments in this thread that implied | otherwise. But please don't ask me to find them, this thread is huge. The idea of falling back to different clocks on the fly on different calls got a bit of a rejection I thought. A recipe for clock inconsitency whatever the failings of the current clock. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ We need a taxonomy for 'printing-that-is-no-longer-printing.' - overhead by WIRED at the Intelligent Printing conference Oct2006

On 02Apr2012 10:44, Glenn Linderman <v+python@g.nevcal.com> wrote: | On 4/2/2012 4:37 AM, Victor Stinner wrote: | > The API looks much more complex than the API proposed in PEP 418 just | > to get the time. You have to call a function to get a function, and | > then call the function, instead of just calling a function directly. | > | > Instead of returning an object with a now() method, I would prefer to | > get directly the function getting time, and another function to get | > "metadata" of the clock. | | If there are more than two clocks, with different characteristics, no | API is going to be both simple to use and fast to call. | | If there are more than two clocks, with different characteristics, then | having an API to get the right API to call to get a time seems very | natural to me. It is, though Victor's point about offering the very easy to use API is valid. The new code has the "flat" monotonic() et al calls as well. | One thing I don't like about the idea of fallback being buried under | some API is that the efficiency of that API on each call must be less | than the efficiency of directly calling an API to get a single clock's | time. For frequently called high resolution clocks, this is more | burdensome than infrequently called clocks.... yet those seem to be the | ones for which fallbacks are proposed, because of potential unavailability. I hadn't thought about that, but it isn't actually a big deal. The overhead isn't zero, but in order to always use the _same_ clock to return hires() (for example) the library has to cache the clock lookup anyway. Current clockutils.py skeleton here: https://bitbucket.org/cameron_simpson/css/src/91848af8663b/lib/python/cs/clo... does so. | The only thing I'm not so sure about: can the properties be described by | flags? Might it not be better to have an API that allows specification | of minimum resolution, in terms of fractional seconds? Perhaps other | properties suffice as flags, but perhaps not resolution. It sounds nice, but there are some difficulties. Firstly, the (currently just 3) flags were chosen to map to the three features sought (in various cobinations) for clocks. Once you start requesting precision (a totally reasonable desire, BTW) you also want to request degree of slew (since "steady" is a tunabe term) and so forth. And what for clocks that have variable precision (I'm imaging here a clock which really is a float, and for large times (in the far future) can't return the sane resolution because of the size of a float. The concern is valid though. I could imagine beefing up the clock object metadata with .epoch (can be None!), precision (function of float width versus clock return value epislon), epsilon (your fraction of a second parameter). Of course, for some clocks any of these might be None. Then the truly concerned user iterates over the available clocks with the desired coarse flags, inspecting each closely for precision or whatever. Easy enough to tweak get_clock() to take an optional all_clocks=False parameter to return all matching clocks in an iterable instead of (the first match or None). Or the user could reach directly for one of the clock lists. cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Craft, n. A fool's substitute for brains. - The Devil's Dictionary

On 02Apr2012 13:37, Victor Stinner <victor.stinner@gmail.com> wrote: | > I've just finished sketching out a skeleton here: | > https://bitbucket.org/cameron_simpson/css/src/fb476fcdcfce/lib/python/cs/clo... | | get_clock() returns None if no clock has the requested flags, whereas | I expected an exception (LookupError or NotImplementError?). That is deliberate. People can easily write fallback like this: clock = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_MONOTONIC) With exceptions one gets a complicated try/except/else chain that is much harder to read. With a second fallback the try/except gets even worse. If one wants an exception it is easy to follow up with: if not clock: raise RunTimeError("no suitable clocks on offer on this platform") | get_clock() doesn't remember if a clock works or not (if it raises an | OSError) and does not fallback to the next clock on error. See | "pseudo-codes" in the PEP 418. I presume the available clocks are all deduced from the platform. Your pseudo code checks for OSError at fetch-the-clock time. I expect that to occur once when the module is loaded, purely to populate the table of avaiable platform clocks. If you are concerned about clocks being available/unavailable at different times (unplugging the GPS peripheral? just guessing here) that will have to raise OSError during the now() call (assuming the clock even exposes the failure; IMO it should when now() is called). | The idea of flags attached to each clock is interesting, but I don't | like the need of different list of clocks. There's no need, just quality of implementation for the monotonic()/hires() convenience calls, which express the (hoped to be common) policy of what clock to offer for each. We've just had pages upon pages of discussion about what clock to offer for the rather bald monotonic() (et al) calls. The ordering of the MONTONIC_CLOCKS list would express the result of that discussion, in that the "better" clocks come first. | Should I use | MONTONIC_CLOCKS or HIRES_CLOCKS when I would like a monotonic and | high-resolution clock? Note that you don't need to provide a clock list at all; get_clock(0 will use ALL_CLOCKS by default, and hires() and monotonic() should each have their own default list. I'll put in montonic() and montonic_clock(clocklist=MONOTONIC_CLOCKS) into the skeleton to make this clear; I see I've omitted them. Regarding the choice itself: as the _caller_ (not the library author), you must decide what you want most. You're already planning offering monotonic() and hires() calls without my proposal! Taking your query "Should I use MONTONIC_CLOCKS or HIRES_CLOCKS when I would like a monotonic and high-resolution clock" is _already_ a problem. Of course you must call monotonic() or hires() first under the current scheme, and must answer this question anyway. Do you prefer hires? Use it first! No preference? Then the question does not matter. If I, as the caller, have a preference then it is obvious what to use. If I do not have a preference then I can just call get_clock() with both flags and then arbitrarily fall back to hires() or monotonic() if that does not work. | It would be simpler to have only one global and | *private* list. No. No no no no no! The whole point is to let the user be _able_ to control the choices to a fair degree without platform special knowledge. The lists are deliberately _optional_ parameters and anyway hidden in the hires() and monotonic() convenince functions; the user does not need to care about them. But the picky user may! The lists align exactly one to one with the feature flags, so there is no special knowledge present here that is not already implicit in publishing the feature flags. | If you have only one list of clocks, how do sort the list to get | QueryPerformanceCounter when the user asks for highres and | GetTickCount when the user asks for monotonic? This is exactly why there are supposed to be different lists. You have just argued against your objection above. | The "if clock.flags & | flags == flags:" test in get_clock() is maybe not enough. I suppose | that we would have the following flags for Windows functions: | | QueryPerformanceCounter.flags = T_HIRES | GetTickCount.flags = T_MONOTONIC | T_STEADY | | (or maybe QueryPerformanceCounter.flags = T_HIRES | T_MONOTONIC ?) Obviously these depend on the clock characteristics. Is QueryPerformanceCounter monotonic? | monotonic_clock() should maybe try to get a clock using the following | list of conditions: | - T_MONOTONIC | T_STEADY | - T_MONOTONIC | T_HIGHRES | - T_MONOTONIC Sure, seems reasonable. That is library internal policy _for the convenince monotonic() function()_. | The T_HIGHRES flag in unclear, even in the PEP. According to the PEP, | any monotonic clock is considered as a "high-resolution" clock. Do you | agree? Not particularly. I easily can imagine a clock with one second resolution hich was monotonic. I would not expect it to have the T_HIRES flag. Example: a synthetic monotonic clock based on a V7 UNIX time() call. But, if it _happens_ that all the monotonic clocks are also hires, so be it. That would be an empirical outcome, not policy. | So we would have: | | GetTickCount.flags = T_MONOTONIC | T_STEADY | T_HIGHRES | | Even if GetTickCount has only an accuracy of 15 ms :-/ T_HIGHRES is a quality call, surely? If 15ms is too sloppy for a "high resolution, the is should _not_ have the T_HIRES flag. | Can list please give the list of flags of each clocks listed in the | PEP? Only clocks used for time.time, time.monotonic and time.highres | (not process and thread clocks nor QueryUnbiasedInterruptTime). | | > # get a clock object - often a singleton under the hood | > T = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_STEADY|T_HIRES) | > # what kind of clock did I get? | > print T.flags | > # get the current time | > now = T.now | | The API looks much more complex than the API proposed in PEP 418 just | to get the time. You have to call a function to get a function, and | then call the function, instead of just calling a function directly. One could have a flat interface as in the PEP, but then the results are not inspectable; the user cannot find out what specific clock, or even kind of clock, was used for the result returned. Unless you want to subclass float for the return values. You could return instances of a float with meta information pointing at the clock used to provide it. I'm -0.5 on that idea. Another advantage of returning a clock object is that it avoids the difficulty of switching implementations behind the user's back, an issue raised in the discussion and rightly rejected as a bad occurence. If the user is handed a clock object, _they_ keep the "current clock in use" state by by having the object reference. | Instead of returning an object with a now() method, I would prefer to | get directly the function getting time, and another function to get | "metadata" of the clock. Then they're disconnected. How do I know the get-metadata call accesses the clock I just used? Only by having library internal global state. I agree some people probably want the flat "get me the time" call, and have no real objection to such existing. But I strongly object to not giving the user control over what they use, and the current API offers no control. | > This removes policy from the library functions and makes it both simple | > and obvious in the user's calling code, and also makes it possible for | > the user to inspect the clock and find out what quality/flavour of clock | > they got. | | I'm not sure that users understand correctly differences between all | these clocks and are able to use your API correctly. How should I | combinese these 3 flags (T_HIRES, T_MONOTONIC and T_STEADY)? Can I use | any combinaison? Of course. Just as with web searches, too many flags may get you an empty result on some platforms, hence the need to fall back. But the _nature_ of the fallback should be in the user's hands. The hires() et al calls can of course offer convenient presupplied fallback according to the preconceptions of the library authors, hopefully well tuned to common users' needs. But if should not be the only mode offered, because you don't know the user's needs. | Which flags are "portable"? Or should I always use an explicit | fallback to ensure getting a clock on any platform? All the flag are portable, but if the platform doesn't supply a clock with the requested flags, even if there's only one flag, the correct result is "None" for the clock offered. Note you can supply no flags! You can always fall all the way back to 0 for the flags; in the skeleton provided that will get you UNIXClock, which is a wrapper for the existing time.time(). In fact, I'll make the flags parameter also optional for get_clock(), defaulting to 0, to make that easy. That becomes your totally portable call:-) | Could you please update your code according to my remarks? I will try | to integrate it into the PEP. A PEP should list all alternatives! Surely. The only updates I can see are to provide the flat interface (instead of via clock-object indirection) and the missing hires_clock() and monotonic_clock() convenience methods. I'll do that. Followup post shortly with new code URL. Would you propose other specific additions? Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Whatever is not nailed down is mine. What I can pry loose is not nailed down. - Collis P. Huntingdon

On 03Apr2012 07:38, I wrote: | On 02Apr2012 13:37, Victor Stinner <victor.stinner@gmail.com> wrote: | | Could you please update your code according to my remarks? I will try | | to integrate it into the PEP. A PEP should list all alternatives! New code here: https://bitbucket.org/cameron_simpson/css/src/91848af8663b/lib/python/cs/clo... Diff: https://bitbucket.org/cameron_simpson/css/changeset/91848af8663b Changelog: updates based on suggestions from Victor Stinner: "flat" API calls to get time directly, make now() a method instead of a property, default flags for get_clock(), adjust hr_clock() to hires_clock(0 for consistency. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Q: How does a hacker fix a function which doesn't work for all of the elements in its domain? A: He changes the domain. - Rich Wareham <rjw57@hermes.cam.ac.uk>

On 03Apr2012 07:51, I wrote: | Changelog: updates based on suggestions from Victor Stinner: "flat" API | calls to get time directly, make now() a method instead of a property, | default flags for get_clock(), adjust hr_clock() to hires_clock(0 for | consistency. BTW, I'd also happily change T_HIRES to HIRES and so forth. They're hard to type and read at present. The prefix is a hangover from old C coding habits, with no namespaces. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ If you don't live on the edge, you're taking up too much space. - t-shirt

On 03Apr2012 07:38, I wrote: | On 02Apr2012 13:37, Victor Stinner <victor.stinner@gmail.com> wrote: | | Should I use | | MONTONIC_CLOCKS or HIRES_CLOCKS when I would like a monotonic and | | high-resolution clock? | | Note that you don't need to provide a clock list at all; get_clock(0 | will use ALL_CLOCKS by default, and hires() and monotonic() should each | have their own default list. [...] | | It would be simpler to have only one global and | | *private* list. [...] | The whole point is to let the user be _able_ to control the choices to a | fair degree without platform special knowledge. On some reflection I may lean a little more Victor's way here: I am still very much of the opinion that there should be multiple clock lists so that hires() can offer the better hires clocks first and so forth. However, perhaps I misunderstood and he was asking if he needed to name a list to get a hires clock etc. This intent is not to need to, via the convenience functions. Accordingly, maybe the list names needn't be published, and may complicate the published interface even though they're one to one with the flags. It would certainly up the ante slightly f we added more flags some time later. (For example, I think any synthetic clocks such as the caching example in the skeleton should probably have a SYNTHETIC flag. You might never ask for it, but you should be able to check for it. (I personally suspect some of the OS clocks are themselves synthetic, but no matter...) The flip side of this of course is that if the list names are private then the get_clock() and hires() etc functions almost mandatorially need the optional all_clocks=False parameter mooted in a sibling post; the really picky user needs a way to iterate over the available clocks to make a fine grained decision. On example would be to ask for monotonic clocks but omit synthetic ones (there's a synthetic clock in the skeleton though I don't partiularly expect one in reality - that really is better in a broader "*utils" module; I also do NOT want to get into complicated parameters to say these flags but not _those_ flags and so forth for other metadata. And again, an external module offering synthetic clocks could easily want to be able to fetch the existing and augument the list with its own, then use that with the get_clock() interfaces. So in short I think: - there should be, internally at least, multiple lists for quality of returned result - there should be a way to iterate over the available clocks, probably via an all_clocks paramater instead of a public list name Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ There is hopeful symbolism in the fact that flags do not wave in a vacuum. - Arthur C. Clarke

I like the aim of letting the user control what clock it get, but I find this API pretty horrible:
clock = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_MONOTONIC)
Just my 2 groszy. //Lennart

On 03Apr2012 07:51, Lennart Regebro <regebro@gmail.com> wrote: | I like the aim of letting the user control what clock it get, but I | find this API pretty horrible: | | > clock = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_MONOTONIC) FWIW, the leading "T_" is now gone, so it would now read: clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) If the symbol names are not the horribleness, can you qualify what API you would like more? -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ We had the experience, but missed the meaning. - T.S. Eliot

On 03/04/2012 07:03, Cameron Simpson wrote:
I reckon the API is ok given that you don't have to supply the flags, correct? A small point but I'm with (I think) Terry Reedy and Steven D'Aprano in that hires is an English word, could you please substitute highres and HIGHRES, thanks. -- Cheers. Mark Lawrence.

On 03Apr2012 09:03, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote: | On 03/04/2012 07:03, Cameron Simpson wrote: | > On 03Apr2012 07:51, Lennart Regebro<regebro@gmail.com> wrote: | > | I like the aim of letting the user control what clock it get, but I | > | find this API pretty horrible: | > | | > |> clock = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_MONOTONIC) | > | > FWIW, the leading "T_" is now gone, so it would now read: | > | > clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) | > | > If the symbol names are not the horribleness, can you qualify what API | > you would like more? | | I reckon the API is ok given that you don't have to supply the flags, | correct? That's right. And if the monotonic() or monotonic_clock() functions (or the hires* versions if suitable) do what you want you don't even need that. You only need the "or" style to choose your own fallback according to your own criteria. | A small point but I'm with (I think) Terry Reedy and Steven D'Aprano in | that hires is an English word, could you please substitute highres and | HIGHRES, thanks. I have the same issue and would be happy to do it. Victor et al, how do you feel about this? People have been saying "hires" throughout the threads I think, but I for one would be slightly happier with "highres". Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ I bested him in an Open Season of scouring-people's-postings-looking-for- spelling-errors. - kevin@rotag.mi.org (Kevin Darcy)

On Tue, Apr 3, 2012 at 08:03, Cameron Simpson <cs@zip.com.au> wrote:
Well, get_clock(monotonic=True, highres=True) would be a vast improvement over get_clock(MONOTONIC|HIRES). I also think it should raise an error if not found. The clarity and easy of use of the API is much more important than how much you can do in one line. //Lennart

Lennart Regebro wrote:
Allowing get_clock(True, True)? Ick. My nomination would be get_clock(MONOTONIC, HIGHRES) -- easier on the eyes with no |.
What's unclear about returning None if no clocks match? Cheers, ~Ethan~

On 03Apr2012 09:07, Ethan Furman <ethan@stoneleaf.us> wrote: | Lennart Regebro wrote: | > On Tue, Apr 3, 2012 at 08:03, Cameron Simpson <cs@zip.com.au> wrote: | >> clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) | >> | >> If the symbol names are not the horribleness, can you qualify what API | >> you would like more? | > | > Well, get_clock(monotonic=True, highres=True) would be a vast | > improvement over get_clock(MONOTONIC|HIRES). | | Allowing get_clock(True, True)? Ick. My nomination would be | get_clock(MONOTONIC, HIGHRES) -- easier on the eyes with no |. get_clock already has two arguments - you can optionally hand it a clock list - that's used by monotonic_clock() and hires_clock(). Have a quick glance at: https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.p... (I finally found out how to point at the latest revision on BitBucket; it's not obvious from the web interface itself.) | > I also think it should | > raise an error if not found. The clarity and easy of use of the API is | > much more important than how much you can do in one line. How much you can do _clearly_ in one line is a useful metric. | What's unclear about returning None if no clocks match? The return of None is very deliberate. I _want_ user specified fallback to be concise and easy. The example: clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) seems to satisfy both these criteria to my eye. Raising an exception makes user fallback a royal PITA, with a horrible try/except cascade needed. Exceptions are all very well when there is just one thing to do: parse this or fail, divide this by that or fail. If fact they're the very image of "do this one thing or FAIL". They are not such a good match for do this thing or that thing or this other thing. When you want a simple linear cascade of choices, Python's short circuiting "or" operator is a very useful thing. Having an obsession with exceptions is IMO unhealthy. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Because of its special customs, crossposting between alt.peeves and normal newsgroups is discouraged. - Cameron Spitzer

Cameron Simpson wrote:
get_clock already has two arguments - you can optionally hand it a clock list - that's used by monotonic_clock() and hires_clock().
def get_clock(*flags, *, clocklist=None): ''' Return a Clock based on the supplied `flags`. The returned clock shall have all the requested flags. If no clock matches, return None. ''' wanted = 0 for flag in flags: wanted |= flag if clocklist is None: clocklist = ALL_CLOCKS for clock in clocklist: if clock.flags & wanted == wanted: return clock.factory() return None Would need to make *flags change to the other *_clock functions.
Have a quick glance at:
https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.p...
Thanks.
Which would become: clock = get_clock(MONOTONIC, HIGHRES) or get_clock(MONOTONIC) +1 to returning None
Another +1. ~Ethan~

On 03Apr2012 15:08, Ethan Furman <ethan@stoneleaf.us> wrote: | Cameron Simpson wrote: | > get_clock already has two arguments - you can optionally hand it a clock | > list - that's used by monotonic_clock() and hires_clock(). | | def get_clock(*flags, *, clocklist=None): I presume that bare "*," is a typo. Both my python2 and python3 commands reject it. [...] | wanted = 0 | for flag in flags: | wanted |= flag [...] I could do this. I think I'm -0 on it, because it doesn't seem more expressive to my eye than the straight make-a-bitmask "|" form. Other opinions? | Would need to make *flags change to the other *_clock functions. Yep. | > The return of None is very deliberate. I _want_ user specified fallback | > to be concise and easy. The example: | > clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) | | Which would become: | clock = get_clock(MONOTONIC, HIGHRES) or get_clock(MONOTONIC) | | +1 to returning None | | > Exceptions are all very well when there is just one thing to do: parse | > this or fail, divide this by that or fail. If fact they're the very | > image of "do this one thing or FAIL". They are not such a good match for do | > this thing or that thing or this other thing. Another thought that occurred in the shower was that get_clock() et al are inquiry functions, and returning None is very sensible there. monotonic() et al are direct use functions, which should raise an exception if unavailable so that code like: t0 = monotonic() ....... t1 = monotonic() does not become littered with checks for special values like None. I consider this additional reason to return None from get_clock(). Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ DON'T DRINK SOAP! DILUTE DILUTE! OK! - on the label of Dr. Bronner's Castile Soap

On Wed, Apr 4, 2012 at 9:38 AM, Cameron Simpson <cs@zip.com.au> wrote:
Yes. I've been mostly staying out of the PEP 418 clock discussion (there are enough oars in there already), but numeric flags are unnecessarily hard to debug. Use strings as your constants unless there's a compelling reason not to. Seeing "('MONOTONIC', 'HIGHRES')" in a debugger or exception message is a lot more informative than seeing "3". Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Apr 3, 2012 at 18:07, Ethan Furman <ethan@stoneleaf.us> wrote:
What's unclear about returning None if no clocks match?
Nothing, but having to check error values on return functions are not what you typically do in Python. Usually, Python functions that fail raise an error. Please don't force Python users to write pseudo-C code in Python. //Lennart

Lennart Regebro wrote:
You mean like the dict.get() function? --> repr({}.get('missing')) 'None' Plus, failure mode is based on intent: if the intent is "Give a clock no matter what", then yes, an exception when that's not possible is the way to go. But if the intent is "Give me a clock that matches this criteria" then returning None is perfectly reasonable. ~Ethan~

Georg Brandl wrote:
Also not a very good example -- if 'missing' was there with a value of None the two situations could not be distinguished with the one call. At any rate, the point is that there is nothing inherently wrong nor unPythonic about a function returning None instead of raising an exception. ~Ethan~

On 04Apr2012 19:47, Georg Brandl <g.brandl@gmx.net> wrote: | Am 04.04.2012 18:18, schrieb Ethan Furman: | > Lennart Regebro wrote: | >> On Tue, Apr 3, 2012 at 18:07, Ethan Furman <ethan@stoneleaf.us> wrote: | >>> What's unclear about returning None if no clocks match? | >> | >> Nothing, but having to check error values on return functions are not | >> what you typically do in Python. Usually, Python functions that fail | >> raise an error. Please don't force Python users to write pseudo-C code | >> in Python. | > | > You mean like the dict.get() function? | > | > --> repr({}.get('missing')) | > 'None' | | Strawman: this is not a failure. And neither is get_clock() returning None. get_clock() is an inquiry function, and None is a legitimate response when no clock is satisfactory, just as a dict has no key for a get(). Conversely, monotonic() ("gimme the time!") and indeed time() should raise an exception if there is no clock. They're, for want of a word, "live" functions you would routinely embed in a calculation. So not so much a straw man as a relevant illuminating example. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ A crash reduces your expensive computer to a simple stone. - Haiku Error Messages http://www.salonmagazine.com/21st/chal/1998/02/10chal2.html

On Wed, Apr 04, 2012 at 05:47:16PM +0200, Lennart Regebro wrote:
Absolutely. "Errors should never pass silently."
Please don't force Python users to write pseudo-C code in Python.
+1. Pythonic equivalent of "get_clock(THIS) or get_clok(THAT)" is for flag in (THIS, THAT): try: clock = get_clock(flag) except: pass else: break else: raise ValueError('Cannot get clock, tried THIS and THAT') Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

Oleg Broytman wrote:
Again, what's the /intent/? No matching clocks does not have to be an error.
Wow -- you'd rather write nine lines of code instead of three? clock = get_clock(THIS) or get_clock(THAT) if clock is None: raise ValueError('Cannot get clock, tried THIS and THAT') ~Ethan~

On Wed, Apr 04, 2012 at 11:03:02AM -0700, Ethan Furman wrote:
Yes - to force people to write the last two lines. Without forcing most programmers will skip them. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

Oleg Broytman wrote:
Forced? I do not use Python to be forced to use one style of programming over another. And it's not like returning None will allow some clock calls to work but not others -- as soon as they try to use it, it will raise an exception. ~Ethan~

On Wed, Apr 04, 2012 at 12:52:00PM -0700, Ethan Furman wrote:
Then it's strange you are using Python with its strict syntax (case-sensitivity, forced indents), ubiquitous exceptions, limited syntax of lambdas and absence of code blocks (read - forced functions), etc.
There is a philosophical distinction between EAFP and LBYL. I am mostly proponent of LBYL. Well, I am partially retreat. "Errors should never pass silently. Unless explicitly silenced." get_clock(FLAG, on_error=None) could return None. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Thu, Apr 5, 2012 at 8:05 AM, Oleg Broytman <phd@phdru.name> wrote:
I still don't see what's erroneous about returning None when asked for an object that is documented to possibly not exist, ever, in some implementations. Isn't that precisely why None exists?

On Thu, Apr 05, 2012 at 10:06:38PM +0900, Stephen J. Turnbull wrote:
Why doesn't open() return None for a non-existing file? or socket.gethostbyname() for a non-existing name? Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Thu, Apr 5, 2012 at 10:34 PM, Oleg Broytman <phd@phdru.name> wrote:
Why doesn't open() return None for a non-existing file? or socket.gethostbyname() for a non-existing name?
That's not an answer to my question, because those calls have very important use cases where the user knows the object exists (and in fact in some cases open() will create it for him), so that failure to exist is indeed a (user) error (such as a misspelling). I find it hard to imagine use cases where "file = open(thisfile) or open(thatfile)" makes sense. Not even for the case where thisfile == 'script.pyc' and thatfile == 'script.py'. The point of the proposed get_clock(), OTOH, is to ask if an object with certain characteristics exists, and the fact that it returns the clock rather than True if found is a matter of practical convenience. Precisely because "clock = get_clock(best) or get_clock(better) or get_clock(acceptable)" does make sense.

On Thu, Apr 05, 2012 at 11:45:06PM +0900, Stephen J. Turnbull wrote:
Counterexamples - any configuration file: a program looks for its config at $HOME and not finding it there looks in /etc. So config = open('~/.someprogram.config') or open('/etc/someprogram/config') would make sense. The absence of any of these files is not an error at all - the program just starts with default configuration. So if the resulting config in the code above would be None - it's still would be ok. But Python doesn't allow that. Some configuration files are constructed by combining a number of user-defined and system-defined files. E.g., the mailcap database. It should be something like combined_database = [db for db in ( open('/etc/mailcap'), open('/usr/etc/mailcap'), open('/usr/local/etc/mailcap'), open('~/.mailcap'), ) if db] But no way - open() raises IOError, not return None. And I think it is the right way. Those who want to write the code similar to the examples above - explicitly suppress exceptions by writing wrappers.
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Thu, 05 Apr 2012 19:22:17 +0400, Oleg Broytman <phd@phdru.name> wrote:
Ah, but the actual code in the mimetypes module (whose list is even longer) looks like this: for file in files: if os.path.isfile(file): db.read(file) That is, Python provides a query function that doesn't raise an error. Do you really think we need to add a third clock function (the query function) that just returns True or False? Maybe we do, if actually creating the clock could raise an error even if exists, as is the case for 'open'. (But unless I'm confused none of this has anything to do with Victor's PEP as currently proposed :) --David

On Thu, Apr 05, 2012 at 11:38:13AM -0400, R. David Murray wrote:
May be we do. Depends on the usage patterns. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Thu, Apr 05, 2012 at 07:22:17PM +0400, Oleg Broytman wrote:
A counterexample with gethostbyname - a list of proxies. It's not an error if some or even all proxies in the list are down - one just connect to the first that's up. So a chain like proxy_addr = gethostbyname(FIRST) or gethostbyname(SECOND) would make sense. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On Fri, Apr 6, 2012 at 12:22 AM, Oleg Broytman <phd@phdru.name> wrote:
Note, implicit existential quantifier.
Counterexamples
Not an argument against an existential quantifier.
But Python doesn't allow [use of conditional constructs when opening a series of files, one must trap exceptions].
True. Python needs to make a choice, and the existence of important cases where the user knows that the object (file) exists makes it plausible that the user would prefer an Exception. Also, open() is intended to be a fairly thin wrapper over the OS facility, and often the OS terms a missing file an "error". I might have chosen to implement a 'None' return if I had designed open(), but I can't get too upset about raising an Exception as it actually does. What I want to know is why you're willing to assert that absence of a clock of a particular configuration is an Exception, when that absence clearly documented to be a common case? I don't find your analogies to be plausible. They seem to come down to "sometimes in Python we've made choices that impose extra work on some use cases, so we should impose extra work on this use case too." But that surely isn't what you mean.

On Thu, Apr 5, 2012 at 21:57, Stephen J. Turnbull <stephen@xemacs.org> wrote:
One fundamental difference is that there are many reasons one might fail to open a file. It may not exist. It may not have permissions allowing the request. It may be locked. If open() returned None, this information would have to be retrievable through another function. However since it returns an exception, that information is already wrapped up in the exception object, should you choose to catch it, and likely to be logged otherwise. In the case of the clocks, I'm assuming the only reason you would fail to get a clock is because it isn't provided by hardware and/or OS. You don't have to worry about transient scenarios on multi-user systems where another user has locked the clock. Thus the exception cannot tell you anything more than None tells you. (Of course, if my assumption is wrong, I'm not sure whether my reasoning still applies.) -- Michael Urman

On Fri, Apr 06, 2012 at 11:57:20AM +0900, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
An error or not an error depends on how people will use the API. I usually don't like error codes -- people tend to ignore them or check lazily. If some library would do (get_clock(THIS) or get_clock(THAT)).clock() I want to get a clearly defined and documented clock-related error, not some vague "AttributeError: 'NoneType' object has no attribute 'clock'". Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

Oleg Broytman wrote:
I come from assembly -- 'a' and 'A' are *not* the same. indents -- I already used them; finding a language that gave them the same importance I did was incredible. exceptions -- Python uses them, true, but I don't have to in my own code (I do, but that's besides the point). lambdas -- they work just fine for my needs. etc.
It's only an error if it's documented that way and, more importantly, thought of that way. The re module is a good example: if it can't find what you're looking for it returns None -- it does *not* raise a NotFound exception. I see get_clock() the same way: I need a clock that does xyz... None? Okay, there isn't one. ~Ethan~

On Thu, Apr 05, 2012 at 11:56:00AM -0700, Ethan Furman wrote:
But open() raises IOError. ''.find('a') returns -1 but ''.index('a') raises ValueError. So we can argue in circles both ways, there are too many arguments pro and contra. Python is just too inconsistent to be consistently argued over. ;-) Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On 06Apr2012 00:15, Oleg Broytman <phd@phdru.name> wrote: | So we can argue in circles both ways, there are too many arguments | pro and contra. Python is just too inconsistent to be consistently | argued over. ;-) Bah! I think these threads demonstrate that we can consistently argue over Python for weeks per topic, sometimes months and years. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Sam Jones <samjones@leo.unm.edu> on the Nine Types of User: Frying Pan/Fire Tactician - "It didn't work with the data set we had, so I fed in my aunt's recipe for key lime pie." Advantages: Will usually fix error. Disadvantages: 'Fix' is defined VERY loosely here. Symptoms: A tendancy to delete lines that get errors instead of fixing them. Real Case: One user complained that their program executed, but didn't do anything. The scon looked at it for twenty minutes before realizing that they'd commented out EVERY LINE. The user said, "Well, that was the only way I could get it to compile."

On 05Apr2012 03:05, Oleg Broytman <phd@phdru.name> wrote: | On Wed, Apr 04, 2012 at 12:52:00PM -0700, Ethan Furman wrote: | > Forced? I do not use Python to be forced to use one style of | > programming over another. | | Then it's strange you are using Python with its strict syntax | (case-sensitivity, forced indents), ubiquitous exceptions, limited | syntax of lambdas and absence of code blocks (read - forced functions), | etc. But exceptions are NOT ubiquitous, nor should they be. They're a very popular and often apt way to handle certain circumstances, that's all. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ On the one hand I knew that programs could have a compelling and deep logical beauty, on the other hand I was forced to admit that most programs are presented in a way fit for mechanical execution, but even if of any beauty at all, totally unfit for human appreciation. - Edsger W. Dijkstra

Oleg Broytman wrote:
You're not my real Dad! You can't tell me what to do! *wink* This level of paternalism is unnecessary. It's not your job to "force" programmers to do anything. If people skip the test for None, they will get an exception as soon as they try to use None as an exception, and then they will fix their broken code. Although I don't like the get_clock() API, I don't think this argument against it is a good one. Exceptions are the *usual* error-handling mechanism in Python, but they are not the *only* mechanism, there are others, and it is perfectly okay to use non-exception based failures when appropriate. This is one such example. "Return None on failure" is how re.match() and re.search() work, and it is a good design for when you have multiple fallbacks on failure. result = re.match(spam, s) or re.match(ham, s) or re.match(eggs, s) if result is None: raise ValueError('could not find spam, ham or eggs') This is a *much* better design than nested tries: try: result = re.match(spam, s) except ValueError: try: result = re.match(ham, s) except ValueError: try: result = re.match(eggs, s) except ValueError: raise ValueError('could not find spam, ham or eggs') Wow. Now *that* is ugly code. There's nothing elegant or Pythonic about being forced to write that out of a misplaced sense of purity. -- Steven

On 05Apr2012 08:50, Steven D'Aprano <steve@pearwood.info> wrote: | Although I don't like the get_clock() API, I don't think this argument against | it is a good one. Just to divert briefly; you said in another post you didn't like the API and (also/because?) it didn't help discoverability. My core objective was to allow users to query for clocks, and ideally enumerate and inspect all clocks. Without the caller having platform specific knowledge. Allowing for the sake of discussion that this is desirable, what would you propose as an API instead of get_clock() (and its friend, get_clocks() for enumeration, that I should stuff into the code). Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Q: How many user support people does it take to change a light bulb? A: We have an exact copy of the light bulb here and it seems to be working fine. Can you tell me what kind of system you have?

Cameron Simpson wrote:
Clocks *are* platform specific -- not just in their availability, but also in the fine details of their semantics and behaviour. I don't think we can or should try to gloss over this. If people are making decisions about timers without knowledge of what their platform supports, they're probably making poor decisions. Even the venerable time.time() and time.clock() differ between Linux and Windows.
The old ways are the best. We don't have math.get_trig() and math.get_trigs() functions for querying trigonometric functions, we just expose the functions directly. I think the way to enumerate and inspect all clocks is with the tried and true Python introspection tools that people use on all other functions: * use dir(time) to see a list of names available in the module * use help(time) to read their help * read the Fine Manual to find out more * use try... except... to detect the existence of a clock There's nothing special about clocks that needs anything more than this. get_clock() looks like a factory function, but it actually isn't. It just selects from a small number of pre-existing clocks. We should just expose those pre-existing clocks directly. I don't see any advantage in adding that extra level of indirection or the addition of all this complexity: * a function get_clock() to select a clock * a function get_clocks() to enumerate all the clocks * another function for querying the properties of a clock All those functions accomplish is to increase the complexity of the API, the documentation and the implementation. It's one more special case for the user to learn: "To find out what functions are available, use dir(module), except for clocks, where you have to use time.get_clocks()." Blah. Another problem with get_clock() -- it will be an attractive nuisance for the sort of person who cares about symmetry and completeness. You will have a steady trickle of "feature requests" from users who are surprised that not every combination of features is supported. Out of the eight or sixteen or thirty-two potential clocks that get_clock() tempts the user with, only three or five will actually exist. The only advantage of get_clock is that you don't need to know the *name* of a platform clock in order to use it, you can describe it with a series of flags or enums. But in practice, that's not an advantage, that's actually a disadvantage. Consider: "Which clock should I use for such-and-such a task, foo or bar?" versus "Which clock should I use for such-and-such a task, get_clock(spam, eggs, cheese) or get_clock(ham, eggs, truffles)?" The mere mechanics of talking about these clocks will suffer because they aren't named. -- Steven

On 06Apr2012 20:25, Steven D'Aprano <steve@pearwood.info> wrote: | Cameron Simpson wrote: | > My core objective was to allow users to query for clocks, and ideally | > enumerate and inspect all clocks. Without the caller having platform | > specific knowledge. | | Clocks *are* platform specific -- not just in their availability, but also in | the fine details of their semantics and behaviour. I don't think we can or | should try to gloss over this. This is why get_clock() returns a clock object, which can have metadata exposing such details. Up to and including the name of the platform specific library/system-call at its core. The issue with monotonic() on its own is that the guarentees in the doco will have to be fairly loose. That prevents the user learning about "fine details of their semantics and behaviour". Glossing over this stuff is exactly what offering _only_ a few genericly characterised clock names (monotonic() et al) does. | If people are making decisions about timers | without knowledge of what their platform supports, they're probably making | poor decisions. Even the venerable time.time() and time.clock() differ between | Linux and Windows. time.clock() does, as (you?) clearly demonstrated elsewhere. time.time()? (Aside from precision?) | > Allowing for the sake of discussion that this is desirable, what would | > you propose as an API instead of get_clock() (and its friend, get_clocks() | > for enumeration, that I should stuff into the code). | | The old ways are the best. We don't have math.get_trig() and math.get_trigs() | functions for querying trigonometric functions, we just expose the functions | directly. | | I think the way to enumerate and inspect all clocks is with the tried and true | Python introspection tools that people use on all other functions: | | * use dir(time) to see a list of names available in the module So, they see "monotonic". Does that tell them much about fine details? | * use help(time) to read their help Useful only to humans, not programs. | * read the Fine Manual to find out more Useful only to humans, not programs. | * use try... except... to detect the existence of a clock Useful only for a fixed list of defined name. Works fine for monotonic, highres, steady or whatever. And I would be ok with the module presenting these only where available and concealing them otherwise, thus raising AttributeError. Or ImportError ("from time import monotonic"). | There's nothing special about clocks that needs anything more than this. This I think is false. In fact, I think your own statement at the start about glossing over fine details goes against this. If I ask for a highres clock, I might well care _how_ precise it was. If I ask for a steady clock, I might well care how large its slews were. If I ask for a monotonic clock, I might well want to know if it tracks wall clock time (even if by magic) or elapsed system run time (eg time that stops increasing if the system is suspended, whereas wallclocks do not). Example: a wallclock is nice for log timestamps. A system run time clock is nice for profiling. They're both monotonic in some domain. | get_clock() looks like a factory function, but it actually isn't. It just | selects from a small number of pre-existing clocks. That number may still be a few. Victor's made it clear that Windows has a choice of possible highres clocks, UNIX clock_getres() offers several possible clock behaviours and an indication that a platform may have several clocks embodying a subset of these, and may indeed offer more clocks. | We should just expose | those pre-existing clocks directly. But exposing them _purely_ _by_ _name_ means inventing names for every single platform clock, and knowing those names per platform. time.clock() is a fine example where the name tells you nearly nothing about the clock behaviour. If the user cares about fine detail as you suggest they need to know their platform and have _external_ knowledge of the platform specifics; they can't inspect from inside the program. | I don't see any advantage in adding that | extra level of indirection or the addition of all this complexity: | * a function get_clock() to select a clock | * a function get_clocks() to enumerate all the clocks These are only two functions because the next alternative seemed an all_clocks= mode parameter, which changed the signature of the function return. Another alternative is the public lists-of-clocks. The point it to be able to enumerate all available clocks for consideration of their properties; get_clock() provides a simple way to coarsely say "a clock like _this_" for the common instances of "this". | * another function for querying the properties of a clock No, that's why you get a clock object back. You can examine it directly for defined metadata names (epoch, precision, underlying-os-clock-name, etc). In exactly the fashion you appear to want for the top level offerings: by knowing the metadata property names. | All those functions accomplish is to increase the complexity of the API, the | documentation and the implementation. It's one more special case for the user | to learn: | | "To find out what functions are available, use dir(module), except for clocks, | where you have to use time.get_clocks()." But dir(module) _will_ list monotonic et al anyway, and possibly matching public clock list names. get_clock() is only for when you want to dig around more flexibly. | Another problem with get_clock() -- it will be an attractive nuisance for the | sort of person who cares about symmetry and completeness. You will have a | steady trickle of "feature requests" from users who are surprised that not | every combination of features is supported. Out of the eight or sixteen or | thirty-two potential clocks that get_clock() tempts the user with, only three | or five will actually exist. And the optional "clocklist" parameter addresses such feaping creaturism by providing a hook for _other_ modules to offer a clock list. Such as a list of syntheic clocks with cool (or insane:-) properties. Without burdening the time module. | The only advantage of get_clock is that you don't need to know the *name* of a | platform clock in order to use it, you can describe it with a series of flags | or enums. But in practice, that's not an advantage, that's actually a | disadvantage. Consider: | | "Which clock should I use for such-and-such a task, foo or bar?" What's your list of foo, bah? Again, I'm not talking about removing monotonic et al. I'm talking about exposing the alternatives for when the chosen-by-the-module monotonic doesn't fit. | versus | "Which clock should I use for such-and-such a task, get_clock(spam, eggs, | cheese) or get_clock(ham, eggs, truffles)?" One hopes the user knows the task. Then they can specify cheese or truffles. Again, only if they feel they need to because the bare monotonic et al don't fit, or was too vague. | The mere mechanics of talking about these clocks will suffer because they | aren't named. But they _can_ be named! get_clock() is for when you don't know or care their names, only their behaviours! And also for when an available clock _wasn't_ one returned by the monotonic et al names. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ I do not trust thee, Cage from Hell, / The reason why I cannot tell, / But this I know, and know full well: / I do not trust thee, Cage from Hell. - Leigh Ann Hussey, leighann@sybase.com, DoD#5913

On 4/6/2012 4:11 PM, Cameron Simpson wrote:
Another alternative is the public lists-of-clocks.
After watching this thread with amusement and frustration, amusement because it is so big, and so many people have so many different opinions, frustration, because it seems that few of the clocks that are available are anywhere near ideal for any particular stated characteristic, and because none of the APIs presented provide a way for the user to specify the details of the characteristics of the desired clock, I think this idea of a list-of-clocks sounds better and better. Hopefully, for each system, the characteristics of each clock can be discovered, and fully characterized in available metadata for the clock... tick rate, or list of tick rates maximum variation of tick rate precision maximum "helicopter drop" jump delta monotonicity frequency of rollover or None base epoch value or None behavior during system sleep, hibernate, suspend, shutdown, battery failure, flood, wartime events, and acts of God. These last two may have values that are long prose texts full of political or religious rhetoric, such as the content of this thread :) any other characteristics I forgot to mention Of course, it is not clear that all of these characteristics can be determined based on OS/Version; hardware vendors may have different implementations. There should be a way to add new clock objects to the list, given a set of characteristics, and an API to retrieve them, at least by installing a submodule that provides access to an additional clock.

On 06Apr2012 17:30, Glenn Linderman <v+python@g.nevcal.com> wrote: | On 4/6/2012 4:11 PM, Cameron Simpson wrote: | > Another alternative is the public lists-of-clocks. | | After watching this thread with amusement and frustration, amusement | because it is so big, and so many people have so many different | opinions, frustration, because it seems that few of the clocks that are | available are anywhere near ideal for any particular stated | characteristic, My partner has occasionally opined that most Prolog programs simply result in "*** NO ***". We could optimise for that and simplify the implementation enormously. It would also let us provide very strong guarrentees about the offered clocks on the basis that no suitable clock would ever provided:-) | and because none of the APIs presented provide a way for | the user to specify the details of the characteristics of the desired | clock, I think this idea of a list-of-clocks sounds better and better. | | Hopefully, for each system, the characteristics of each clock can be | discovered, and fully characterized in available metadata for the clock... Victor has asked me to do that for my skeleton, based on the tables he has assembled. I'll see what i can do there... | Of course, it is not clear that all of these characteristics can be | determined based on OS/Version; hardware vendors may have different | implementations. If you can look up the kernel revision you can do fairly well. In principle. | There should be a way to add new clock objects to the list, given a set | of characteristics, and an API to retrieve them, at least by installing | a submodule that provides access to an additional clock. Returning to seriousness, the get_clock() call admits a clocklist. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Principles have no real force except when one is well fed. - Mark Twain

Lennart Regebro wrote:
That's a matter of opinion. I'm not particularly fond of this get_clock idea, but of the two examples given, I much prefer the first of these: get_clock(MONOTONIC|HIRES) get_clock(monotonic=True, highres=True) and not just because it is shorter. The API is crying out for enum arguments, not a series of named flags. But frankly I think this get_clock API sucks. At some earlier part of this thread, somebody listed three or four potential characteristics of clocks. If we offer these as parameters to get_clock(), that means there's eight or sixteen different clocks that the user can potentially ask for. Do we really offer sixteen different clocks? Or even eight? I doubt it -- there's probably only two or three. So the majority of potential clocks don't exist. With get_clock, discoverability is hurt. How does the caller know what clocks are available? How can she look for documentation for them? A simple, obvious, discoverable API is best. If we offer three clocks, we have three named functions. If some of these clocks aren't available on some platform, and we can't emulate them, then simply don't have that named function available on that platform. That's easy to discover: trying to use that clock will give a NameError or AttributeError, and the caller can then fall back on an alternative, or fail, whichever is appropriate. -- Steven

On Wed, Apr 4, 2012 at 9:04 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
If I were looking at that in documentation, my automatic guess would be that the only thing that matters is whether the argument compares-as-true or not. So get_clock(monotonic="yes") would be the same as =True, and =False wouldn't be. And get_clock(monotonic="No, you idiot, I want one that ISN'T") would... be stupid. But it'd still function :) Chris Angelico

On Wed, Apr 4, 2012 at 13:04, Victor Stinner <victor.stinner@gmail.com> wrote:
It depends if the option supports other values. But as I understood, the keyword value must always be True.
Or False, obviously. Which would also be default. //Lennart

2012/4/4 Lennart Regebro <regebro@gmail.com>:
Ok for the default, but what happens if the caller sets an option to False? Does get_clock(monotonic=False) return a non-monotonic clock? (I guess no, but it may be confusing.) Victor

On Thu, Apr 5, 2012 at 01:10, Victor Stinner <victor.stinner@gmail.com> wrote:
Good point, but the same does for using flags. If you don't pass in the MONOTONIC flag, what happens? Only reading the documentation will tell you. As such this, if anything, is an indication that the get_clock() API isn't ideal in any incarnation. //Lennart

On 05Apr2012 10:21, Lennart Regebro <regebro@gmail.com> wrote: | On Thu, Apr 5, 2012 at 01:10, Victor Stinner <victor.stinner@gmail.com> wrote: | > Ok for the default, but what happens if the caller sets an option to | > False? Does get_clock(monotonic=False) return a non-monotonic clock? | > (I guess no, but it may be confusing.) This is where the bitmap approach can be less confusing - the docstring says "The returned clock shall have all the requested flags". It is at least very predictable. | Good point, but the same does for using flags. Only notionally. With a keyword argument the (lazy non doc reading) caller can imagine the default is None, and True and False specify concrete position and negative requirements. Not the case with a bitmask, which only has two states per feature, not three (or arbitrarily many, depending how nasty one wants to be - I could play devil's advocate and ask for monotonic=0.7 and demand a competivtive evaluation of relative merits:-) | If you don't pass in | the MONOTONIC flag, what happens? Only reading the documentation will | tell you. Gah! ALL functions are like that! How often do we see questions about max() or split() etc that a close reading of the docs obviate? | As such this, if anything, is an indication that the | get_clock() API isn't ideal in any incarnation. It's not meant to be ideal. I find that word almost useless in its overuse. get_clock() is meant to be _very_ _often_ _useful_ and easy to use for expressing simple fallback when the PEP418 monotonic() et al calls don't fit. For the truly arbitrary case the caller needs to be able to enumerate all the available clocks and make their own totally ad hoc decision. My current example code offers both public lock list names and get_clocks() (just like get_clock() in signature, but returning all matches instead of just the first one). Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ From the EXUP mailing list <exup-brotherhood@Majordomo.net> ... Wayne Girdlestone <WayneG@mega.co.za>: WG> Let's say there are no Yamaha's or Kawa's in the world. Stevey Racer <ssturm@co.la.ca.us>: SR> sriw - so then you are saying that Revelations (from the Bible) has come SR> true and Hell is now on Earth. WG> Your choice for you new bike is either a new '98 fuel injected SRAD, or a WG> new '98 Fireblade. SR> sriw -The devil's minions - full of temptation but never fulfilling their SR> promise.

Le 06/04/2012 00:17, Cameron Simpson a écrit :
By the way, I removed ("deferred") the time.highres() function from the PEP, and I try to avoid the term "steady" because no OS clock respect the definition of "steady" (especially in corner cases as system suspend/resume). So which flags do you want to support? (only "monotonic"?) Basically, get_clock("monotonic") should give time.monotonic() whereas get_clock() gives time.time()? Victor

On 06Apr2012 00:27, Victor Stinner <victor.stinner@gmail.com> wrote: | Le 06/04/2012 00:17, Cameron Simpson a écrit : | > This is where the bitmap approach can be less confusing - the docstring | > says "The returned clock shall have all the requested flags". It is at | > least very predictable. | | By the way, I removed ("deferred") the time.highres() function from the | PEP, Chuckle; was not the whole PEP for a high res clock? | and I try to avoid the term "steady" because no OS clock respect | the definition of "steady" (especially in corner cases as system | suspend/resume). I can think of definitions of "steady" that I personally would accept, and they'd accept that suspend/resume would be concealed (I guess I would usually want - purely myself here - a clock representing system run time; I'd go for time.time() for wall clock). | So which flags do you want to support? (only "monotonic"?) I'd stay with my current list, with metadata in the clock objects indicating what _flavour_ of "steady" or "high res" they present. | Basically, get_clock("monotonic") should give time.monotonic() whereas If time.monotonic() never falls back to a non-monotonic source, yes. | get_clock() gives time.time()? Might in theory give something better, but time.time() would always be a valid result of nothing else seemed better to the module author. I imagine in practice that time.time() might always use the "best" clock absent special requirements. So you'd probably get what particular clock used to implement time.time(), yes. (I realise this has interesting implications for the list orders; time.time() would come _first_, but providing feature flags to get_clock() can cause it not to be chosen when it doesn't match.) This a reason why I think we should present (even privately only) all the system clocks for a platform. Then you _can_ still offer highres() and steady() with detailed qualifications in the docs as to what considerations went into acepting a clock as highres or steady, and therefore why some users may find them unsatisfactory i.e. under what sort of circumstances/requirements they may not suit. Any of the montonic()/highres()/steady() represent policy decisions by the module author; it is just that monotonic() is easier to qualify than the others: "never goes backwards in return value". Even though VMs and system suspend can add depth to the arguments. It _is_ useful for people to be able to reach for highres() or steady() a lot of the time; they do, though, need to be able to decide if that's sensible. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ I thought back to other headaches from my past and sneered at their ineffectiveness. - Harry Harrison

On 06Apr2012 08:51, I wrote: | On 06Apr2012 00:27, Victor Stinner <victor.stinner@gmail.com> wrote: | | By the way, I removed ("deferred") the time.highres() function from the | | PEP, | | Chuckle; was not the whole PEP for a high res clock? Gah. I see it was for montonic, not high res. Sorry. [...] | I can think of definitions of "steady" that I personally would accept, | and they'd accept that suspend/resume would be concealed (I guess I | would usually want - purely myself here - a clock representing system | run time; I'd go for time.time() for wall clock). | | | So which flags do you want to support? (only "monotonic"?) | | I'd stay with my current list, with metadata in the clock objects | indicating what _flavour_ of "steady" or "high res" they present. On reflection, since I have historically presumed time.time() on UNIX mapped to "man 2 time", I clearly think that time.time() is wall clock time, and may jump when the sysadmin notices it is incorrect (of course this is often mediated by NTP, which in turn is usually mediated by some ntpd using adjtime(), which slews instead of jumping). But it might jump. (I'm intending to jump a wayard VM today, in fact:-) So guess I expect time.time() to be only usually steady. And usually monotonic. So having neither flag. Do I want a WALLCLOCK flag? Meaning a clock that is supposed to be real world time (did I see REALTIME in one of your examples?), and may be almost arbirarily corrected to be made real world if it is wrong. Maybe. +0 on that I think. Basicly I'm distinguishing here between a clock used for timestamps, for example in log entries, and a clock used for measuring elapsed system run time, for example in benchmarking. I would want to log entries to match what a clock on the wall should say. So I think I'm _still_ for the three original flags I suggested (monotonic, high res, steady) and expect time.time() to not necessarily meet any of them. But to meet a hypothetical WALLCLOCK flag. Regarding UNIX time(2) (or POSIX time(3)), POSIX says: The time() function shall return the value of time in seconds since the Epoch. and the epoch is a date. So UNIX time() should be a wall clock. Python "help(time.time)" says: Return the current time in seconds since the Epoch. So I think it should also be a wall clock by that same reasoning. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Uh, this is only temporary...unless it works. - Red Green

On Fri, Apr 6, 2012 at 00:17, Cameron Simpson <cs@zip.com.au> wrote:
Gah! ALL functions are like that! How often do we see questions about max() or split() etc that a close reading of the docs obviate?
My point exactly. //Lennart

I don't know who started this, but the PEP 418 threads have altogether too much snarkiness and not enough content. It's bad enough that we're bikeshedding so intensely; we don't need clever comebacks in triplicate to every out-of-context argument. --Guido On Fri, Apr 6, 2012 at 2:26 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
-- --Guido van Rossum (python.org/~guido)

On 04Apr2012 09:53, Steven D'Aprano <steve@pearwood.info> wrote: | Lennart Regebro wrote: | > On Tue, Apr 3, 2012 at 08:03, Cameron Simpson <cs@zip.com.au> wrote: | >> clock = get_clock(MONOTONIC|HIRES) or get_clock(MONOTONIC) | >> If the symbol names are not the horribleness, can you qualify what API | >> you would like more? | > | > Well, get_clock(monotonic=True, highres=True) would be a vast | > improvement over get_clock(MONOTONIC|HIRES).[...] | | That's a matter of opinion. I'm not particularly fond of this get_clock idea, | but of the two examples given, I much prefer the first of these: | | get_clock(MONOTONIC|HIRES) | get_clock(monotonic=True, highres=True) | | and not just because it is shorter. The API is crying out for enum arguments, | not a series of named flags. Enums would be ok with me. I went with a bitmask because it is natural to me and very simple. But anything symbolicly expression will do. | But frankly I think this get_clock API sucks. At some earlier part of this | thread, somebody listed three or four potential characteristics of clocks. If | we offer these as parameters to get_clock(), that means there's eight or | sixteen different clocks that the user can potentially ask for. Do we really | offer sixteen different clocks? Or even eight? I doubt it -- there's probably | only two or three. So the majority of potential clocks don't exist. That's not the point. I think we should offer all the platform system clocks, suitably described. That there are up to 8 or 16 flag combinations is irrelevant; no user is going to try them all. A user will have requirements for their clock. They ask for them either blandly via get_clock() or (for example considering monotonic most important) via monotonic_clock(). In the latter case, the supported clocks can be considered in a more apt order via a different internal clock list. | With get_clock, discoverability is hurt. No, because the other calls still exist. (In my proposal. I see Victor's characterised this as either/or in the PEP, never my intent.) | How does the caller know what clocks | are available? I would definitely want either: - the module clock lists available via public names, for example as in my sample clockutils.py code (ALL_CLOCKS, MONTONIC_CLOCKS etc) or via some map (eg clocks['monotonic']). - a get_clocks() function to return matching clocks, like get_clock() but not stopping on the first match - an all_clocks=False parameter to get_clock() to get an iterable of the suitable clocks | How can she look for documentation for them? There is good text in the PEP. That could be easily moved into the module doco in a "clocks" section. Since my clocks proposal wraps clocks in an object, they _can_ have nice class names and good docstrings and more metadata in the object (possibilities including .epoch, .precision, .is_steady() methods, .os_clock_name (eg "QueryPerformanceCounter"), etc). | A simple, obvious, discoverable API is best. If we offer three clocks, we have | three named functions. If some of these clocks aren't available on some | platform, and we can't emulate them, then simply don't have that named | function available on that platform. That's easy to discover: trying to use | that clock will give a NameError or AttributeError, and the caller can then | fall back on an alternative, or fail, whichever is appropriate. And I hate this. Because many platforms offer several OS clocks. The time module SHOULD NOT dictate what clocks you get to play with, and you should not need to have platform specific knowledge to look for a clock with your desired characteristics. If you just want montonic() and trust the module authors' policy decisions you can go with monotonic(), have it do AttributeError if unavailable and never worry about discoverability or the inspectable object layer. Many will probaby be happy with that. But without get_clock() or something like it, there is no discoverability and not ability for a user to decide their own clock choice policy. -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Your modesty is typically human, so I will overlook it. - a Klingon

Why not passing a a list of set of flags? Example: haypo_steady = get_clock(MONOTONIC|STEADY, STEADY, MONOTONIC, REALTIME) # try to get a monotonic and steady clock, # or fallback to a steady clock, # or fallback to a monotonic clock, # or fallback to the system clock haypo_perf_counter = get_clock(HIGHRES, MONOTONIC|STEADY, STEADY, MONOTONIC, REALTIME) # try to get a high-resolution clock # or fallback to a monotonic and steady clock, # or fallback to a steady clock, # or fallback to a monotonic clock, # or fallback to the system clock On Windows, haypo_steady should give GetTickCount (MONOTONIC|STEADY) and haypo_perf_counter should give QueryPerformanceCounter (MONOTONIC|HIGHRES). Hum, I'm not sure that haypo_highres uses the same clocks than time.perf_counter() in the PEP.
And if don't read the doc carefuly and forget the test, you can a "NoneType object is not callable" error.
It's better to avoid unnecessary system calls at startup (when the time module is loaded), but you may defer the creation of the clock list, or at least of the flags of each clock.
A list of clocks and a function are maybe redundant. Why not only providing a function?
My PEP starts with use cases: it proposes one clock per use case. There is no "If you need a monotonic, steady and high-resolution clock ..." use case. The "highres" name was confusing, I just replaced it with time.perf_counter() (thanks Antoine for the name!). time.perf_counter() should be used for benchmarking and profiling.
I mean having to choose the flags *and* the list of clocks is hard. I would prefer to only have to choose flags or only the list of clocks. The example was maybe not the best one.
You can solve this issue with only one list of clocks if you use the right set of flags.
So what is the minimum resolution and/or accuracy of the HIGHRES flag?
A full implementation would help to decide which API is the best one. "Full" implementation: - define all convinience function - define all list of clocks - define flags of all clocks listed in the PEP 418: clocks used in the pseudo-code of time.steady and time.perf_counter, and maybe also time.time Victor

On 04Apr2012 01:45, Victor Stinner <victor.stinner@gmail.com> wrote: | > | get_clock() returns None if no clock has the requested flags, whereas | > | I expected an exception (LookupError or NotImplementError?). | > | > That is deliberate. People can easily write fallback like this: | > | > clock = get_clock(T_MONOTONIC|T_HIRES) or get_clock(T_MONOTONIC) | | Why not passing a a list of set of flags? Example: | | haypo_steady = get_clock(MONOTONIC|STEADY, STEADY, MONOTONIC, REALTIME) | # try to get a monotonic and steady clock, | # or fallback to a steady clock, | # or fallback to a monotonic clock, | # or fallback to the system clock That's interesting. Ethan Furman suggested multiple arguments to be combined, whereas yours bundles multiple search criteria in one call. While it uses a bitmask as mine does, this may get cumbersome if we went with Nick's "use strings!" suggestion. | haypo_perf_counter = get_clock(HIGHRES, MONOTONIC|STEADY, STEADY, | MONOTONIC, REALTIME) | # try to get a high-resolution clock | # or fallback to a monotonic and steady clock, | # or fallback to a steady clock, | # or fallback to a monotonic clock, | # or fallback to the system clock | | On Windows, haypo_steady should give GetTickCount (MONOTONIC|STEADY) | and haypo_perf_counter should give QueryPerformanceCounter | (MONOTONIC|HIGHRES). Sounds ok to me. I am not familiar with the Windows counters and am happy to take your word for it. | Hum, I'm not sure that haypo_highres uses the same clocks than | time.perf_counter() in the PEP. | | > If one wants an exception it is easy to follow up with: | > if not clock: | > raise RunTimeError("no suitable clocks on offer on this platform") | | And if don't read the doc carefuly and forget the test, you can a | "NoneType object is not callable" error. Excellent! An exception either way! Win win! | > | get_clock() doesn't remember if a clock works or not (if it raises an | > | OSError) and does not fallback to the next clock on error. See | > | "pseudo-codes" in the PEP 418. | > | > I presume the available clocks are all deduced from the platform. Your | > pseudo code checks for OSError at fetch-the-clock time. I expect that | > to occur once when the module is loaded, purely to populate the table | > of avaiable platform clocks. | | It's better to avoid unnecessary system calls at startup (when the | time module is loaded), but you may defer the creation of the clock | list, or at least of the flags of each clock. Yes indeed. I think this should be deferred until use. | > Note that you don't need to provide a clock list at all; get_clock(0 | > will use ALL_CLOCKS by default, and hires() and monotonic() should each | > have their own default list. | | A list of clocks and a function are maybe redundant. Why not only | providing a function? Only because the function currently only returns one clock. The picky user may want to peruse all the clocks inspecting other metadata (precision etc) than the coarse flag requirements. There should be a way to enumerate the available clock implementation; in my other recent post I suggest either lists (as current), a get_clocks() function, or a mode parameter to get_clock() such as _all_clocks, defaulting to False. | > Regarding the choice itself: as the _caller_ (not the library author), | > you must decide what you want most. You're already planning offering | > monotonic() and hires() calls without my proposal! | | My PEP starts with use cases: it proposes one clock per use case. | There is no "If you need a monotonic, steady and high-resolution clock | ..." use case. Yes. but this is my exact objection to the "just provide hires() and steady() and/or monotonic()" API; the discussion to date is littered with "I can't imagine wanting to do X" style remarks. We should not be trying to enumerate the user case space exhaustively. I'm entirely in favour of your list of use cases and the approach of providing hires() et al to cover the thought-common use cases. But I feel we really _must_ provide a way for the user with a not-thought-of use case to make an arbitrary decision. get_clock() provides a simple cut at the "gimme a suitable clock" approach, with the lists or other "get me an enumeration of the available clocks" mechanism for totally ad hoc perusal if the need arises. This is also my perhaps unstated concern with Guido's "the more I think about it, the more I believe these functions should have very loose guarantees, and instead just cater to common use cases -- availability of a timer with minimal fuss is usually more important than the guarantees" http://www.mail-archive.com/python-dev@python.org/msg66173.html The easy to use hires() etc must make very loose guarentees or they will be useless too often. That looseness is fine in some ways - it provides availability on many platforms (all?) and discourages the user from hoping for too much and thus writing fragile code. But it also PREVENTS the user from obtaining a really good clock if it is available (where "good" means their partiuclar weirdo feature requirements). So I think there should be both - the easy and simple calls, and a mecahnism for providing all clocks for the user to chose with arbitrary criteria and fallback. | The "highres" name was confusing, I just replaced it with | time.perf_counter() (thanks Antoine for the name!). | time.perf_counter() should be used for benchmarking and profiling. I've been wondering; do we distinguish between clocks and counters. In my mind a clock or timer has a very linear relationship with "real time", the wall clock. A counter, by comparison, may measure CPU cycles or kernel timeslice ticks or python opcode counts or any number of other time-like resource consumption things. I've been presuming we're concerned here with "clocks" and not counters. | > Taking your query "Should | > I use MONTONIC_CLOCKS or HIRES_CLOCKS when I would like a monotonic and | > high-resolution clock" is _already_ a problem. Of course you must call | > monotonic() or hires() first under the current scheme, and must answer this | > question anyway. Do you prefer hires? Use it first! No preference? Then the | > question does not matter. | | I mean having to choose the flags *and* the list of clocks is hard. I | would prefer to only have to choose flags or only the list of clocks. | The example was maybe not the best one. Yah; I think I made a followup post where I realised you may have meant this. The use of default arguments is meant to make it easy to use flags and/or lists or even neither (which for get_clock() at least would always get you a clock because a wrapper for time.time() is always provided). In my mind, usually just flags of course. | > | If you have only one list of clocks, how do sort the list to get | > | QueryPerformanceCounter when the user asks for highres and | > | GetTickCount when the user asks for monotonic? | > | > This is exactly why there are supposed to be different lists. | > You have just argued against your objection above. | | You can solve this issue with only one list of clocks if you use the | right set of flags. No you can't, not in general. If there are multiple clocks honouring those flags you only ever get the first one in the list. The point of the MONOTONIC_CLOCKS list etc is that the lists may be differently ordered to provide quality of clock within that domain. Suppose I ask for steady_clock(MONOTONIC); I would probably prefer a more-steady clock over a more-precise/hires clock. And converse desires when asking for hires_clock(). So different list orders, at least in principle. If it turns out empirically that this isn't the case then all the names can in fact refer to the same list. But offering only one list _name_ prevents offering these nuances when other platforms/clocks come in the future. | > | So we would have: | > | | > | GetTickCount.flags = T_MONOTONIC | T_STEADY | T_HIGHRES | > | | > | Even if GetTickCount has only an accuracy of 15 ms :-/ | > | > T_HIGHRES is a quality call, surely? If 15ms is too sloppy for a "high | > resolution, the is should _not_ have the T_HIRES flag. | | So what is the minimum resolution and/or accuracy of the HIGHRES flag? No idea. But you must in principle have one in mind to offer the hires() call at all in the PEP, or be prepared to merely offer the most hires clock available regardless. In this latter case you would always mark that clock as having the HIRES flag and the problem is solved. It would be good to have metadata to indicate how hires a partiulcar clock is. | > | Could you please update your code according to my remarks? I will try | > | to integrate it into the PEP. A PEP should list all alternatives! | > | > Surely. | > | > The only updates I can see are to provide the flat interface | > (instead of via clock-object indirection) and the missing hires_clock() | > and monotonic_clock() convenience methods. | | A full implementation would help to decide which API is the best one. | "Full" implementation: | | - define all convinience function | - define all list of clocks Ok. My current code is here, BTW: https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.p... (Finally found a revision independent URL on bitbucket.) | - define flags of all clocks listed in the PEP 418: clocks used in | the pseudo-code of time.steady and time.perf_counter, and maybe also | time.time I'll make one. It will take a little while. Will post again when ready. At present the code compiles and runs (albeit with no platform specific clocks:-) This table may require fictitious code. Should still compile I guess... Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ The right to be heard does not include the right to be taken seriously. - Hubert Horatio Humphrey

Victor et al, Just an update note: I've started marking up clocks with attributes; not yet complete and I still need to make a small C extension to present the system clocks to Python space (which means learning to do that, too). But you can glance over the start on it here: https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.p... Several feature flags and some properties for qualifying clocks. Still needed stuff includes: C access to clocks, .accuracy being actual clock precision versus the resolution of the units in the underlying OS call, a __repr__ and/or __str__ to decode feature bitmaps into useful strings, .is_*() __getattr__ method to resolve against flags by name or maybe has_flag(str), etc. On 07Apr2012 01:16, Victor Stinner <victor.stinner@gmail.com> wrote: | > | This is the original reason for the original defect (issue 10278) | > | unix' clock() doesn't actually provide a clock in this sense, | > | it provides a resource usage metric. | > | > Yeah:-( Its help says "Return the CPU time or real time since [...]". | > Two very different things, as demonstrated. I suppose neither goes | > backwards, but this seems like a classic example of the "useless | > monotonic clock" against which Greg Ewing railed. | > | > And why? For one thing, because one can't inspect its metadata to find | > out what it does. | | Should I add another key to the result of | time.get_clock_info('clock')? How can we define "clock on Windows" | (almost monotonic and steady clock) vs "clock on UNIX" (CPU time) with | a flag or a value? For clocks I'm going with two feature flags: WALLCLOCK and RUNTIME. The former indicates a clock that tries to stay in synch with real world time, and would still advance when the system is suspended or idle; it would almost certainly need to "step" over a suspend. The latter means system run time; it accrues only while the system is up. Neither is CPU usage (so a time.sleep(10) would add 10 seconds to both). I think resource usage is not a "clock". We could characterise such timers and counters with a lot of the same metrics we like to use with clocks, but I do not think they should be returned by a system purporting to return clocks or clock values. On 07Apr2012 14:22, I wrote: | On 06Apr2012 17:30, Glenn Linderman <v+python@g.nevcal.com> wrote: | | Hopefully, for each system, the characteristics of each clock can be | | discovered, and fully characterized in available metadata for the clock... | | Victor has asked me to do that for my skeleton, based on the tables he | has assembled. I'll see what i can do there... I've started on this, see above. Victor Stinner <victor.stinner@gmail.com> wrote: | | - define flags of all clocks listed in the PEP 418: clocks used in | | the pseudo-code of time.steady and time.perf_counter, and maybe also | | time.time | | I'll make one. It will take a little while. Will post again when ready. So, new code to glance over as evidence of good faith, if not speed:-( Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Life is uncertain. Eat dessert first. - Jim Blandy

FWIW, I'm not sure you're the right person to drive time PEPs. You don't seem to have come into it with much knowledge of time, and it's taken several repetitions for you to take corrections into account in both this discussion and the Decimal/datetime representation PEP. On Mon, Mar 26, 2012 at 4:32 PM, Victor Stinner <victor.stinner@gmail.com> wrote:

On Tue, Mar 27, 2012 at 3:51 PM, Jeffrey Yasskin <jyasskin@gmail.com> wrote:
The main things required to be a PEP champion are passion and a willingness to listen to expert feedback and change course in response. If someone lacks the former, they will lose steam and their PEP will eventually be abandoned. If they don't listen to expert feedback, then their PEP will ultimately be rejected (sometimes a PEP will be rejected anyway as a poor fit for the language *despite* being responsive to feedback, but that's no slight to the PEP author). Victor has shown himself to be quite capable of handling those aspects of the PEP process, and the topics he has recently applied himself to are ones where it is worthwhile having a good answer in the standard library for Python 3.3. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I started to write the PEP 418 to clarify the notions of monotonic and steady clocks.
I replaced time.steady() by time.try_monotonic(). I misunderstood "may not" in the C++ doc: I understood it as "it may be adjusted by NTP", whereas it means "it cannot be adjusted". Sorry for the confusion. I added a time.hires() clock because time.monotonic() and time.try_monotonic() are not the best clocks for profiling or benchmarking. For example, on Windows, time.hires() uses QueryPerformanceCounter() whereas time.monotonic() and time.try_monotonic() uses GetTickCount[64](). I added the pseudo-code of each function. I hope that it is easier to understand than a long text. Victor
participants (30)
-
Antoine Pitrou
-
Cameron Simpson
-
Charles-François Natali
-
Chris Angelico
-
Ethan Furman
-
Georg Brandl
-
Glenn Linderman
-
Glyph
-
Greg Ewing
-
Guido van Rossum
-
Jeffrey Yasskin
-
Jonathan French
-
Larry Hastings
-
Lennart Regebro
-
Mark Lawrence
-
Matt Joiner
-
Michael Foord
-
Michael Urman
-
Nadeem Vawda
-
Nick Coghlan
-
Oleg Broytman
-
Paul Moore
-
R. David Murray
-
Scott Dial
-
Stephen J. Turnbull
-
Steven D'Aprano
-
Terry Reedy
-
Victor Stinner
-
Yury Selivanov
-
Zooko Wilcox-O'Hearn