datetime.timedelta total_microseconds
In a recent code review, the following snippet was called out as reinventing the wheel: _MICROSECONDS_PER_SECOND = 1000000 def _timedelta_to_microseconds(delta): return int(delta.total_seconds() * _MICROSECONDS_PER_SECOND) The reviewer thought that there must already exist a standard library function that fulfills this functionality. After we had both satisfied ourselves that we hadn't simply missed something in the documentation, we decided that we had better raise the issue with a wider audience. Does this functionality already exist within the standard library? If not, would a datetime.timedelta.total_microseconds function be a reasonable addition? I would be happy to submit a patch for such a thing. Richard Belleville
Looks like timedelta has a microseconds property. Would this work for your needs? In [12]: d Out[12]: datetime.timedelta(0, 3, 398407) In [13]: d.microseconds Out[13]: 398407 On Wed, Feb 13, 2019 at 9:08 PM Richard Belleville via Python-Dev < python-dev@python.org> wrote:
In a recent code review, the following snippet was called out as reinventing the wheel:
_MICROSECONDS_PER_SECOND = 1000000
def _timedelta_to_microseconds(delta): return int(delta.total_seconds() * _MICROSECONDS_PER_SECOND)
The reviewer thought that there must already exist a standard library function that fulfills this functionality. After we had both satisfied ourselves that we hadn't simply missed something in the documentation, we decided that we had better raise the issue with a wider audience.
Does this functionality already exist within the standard library? If not, would a datetime.timedelta.total_microseconds function be a reasonable addition? I would be happy to submit a patch for such a thing.
Richard Belleville _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/tahafut%40gmail.com
Oops. That isn't the TOTAL microseconds, but just the microseconds portion. Sorry for the confusion. On Wed, Feb 13, 2019 at 9:23 PM Henry Chen <tahafut@gmail.com> wrote:
Looks like timedelta has a microseconds property. Would this work for your needs?
In [12]: d Out[12]: datetime.timedelta(0, 3, 398407)
In [13]: d.microseconds Out[13]: 398407
On Wed, Feb 13, 2019 at 9:08 PM Richard Belleville via Python-Dev < python-dev@python.org> wrote:
In a recent code review, the following snippet was called out as reinventing the wheel:
_MICROSECONDS_PER_SECOND = 1000000
def _timedelta_to_microseconds(delta): return int(delta.total_seconds() * _MICROSECONDS_PER_SECOND)
The reviewer thought that there must already exist a standard library function that fulfills this functionality. After we had both satisfied ourselves that we hadn't simply missed something in the documentation, we decided that we had better raise the issue with a wider audience.
Does this functionality already exist within the standard library? If not, would a datetime.timedelta.total_microseconds function be a reasonable addition? I would be happy to submit a patch for such a thing.
Richard Belleville _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/tahafut%40gmail.com
I don't think it's totally unreasonable to have other total_X() methods, where X would be days, hours, minutes and microseconds, but it also doesn't seem like a pressing need to me. I think the biggest argument against it is that they are all trivial to implement as necessary, because they're just unit conversions that involve multiplication or division by constants, which is nowhere near as complicated to implement as the original `total_seconds` method. Here's the issue where total_seconds() was implemented, it doesn't seem like there was any discussion of other total methods until after the issue was closed: https://bugs.python.org/issue5788 I think the main issue is how "thick" we want the timedelta class to be. With separate methods for every unit, we have to maintain and document 5 methods instead of 1, though the methods are trivial and the documentation could maybe be shared. If I had a time machine, I'd probably recommend an interface something like this: def total_duration(self, units='seconds'): return self._total_seconds() * _SECONDS_PER_UNIT[units] I suppose it would be possible to move to that interface today, though I think it would be mildly confusing to have two functions that do the same thing (total_seconds and total_duration), which may not be worth it considering that these functions are a pretty minor convenience. Best, Paul On 2/14/19 12:05 AM, Richard Belleville via Python-Dev wrote:
In a recent code review, the following snippet was called out as reinventing the wheel:
_MICROSECONDS_PER_SECOND = 1000000
def _timedelta_to_microseconds(delta): return int(delta.total_seconds() * _MICROSECONDS_PER_SECOND)
The reviewer thought that there must already exist a standard library function that fulfills this functionality. After we had both satisfied ourselves that we hadn't simply missed something in the documentation, we decided that we had better raise the issue with a wider audience.
Does this functionality already exist within the standard library? If not, would a datetime.timedelta.total_microseconds function be a reasonable addition? I would be happy to submit a patch for such a thing.
Richard Belleville
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io
On Thu, Feb 14, 2019 at 9:07 AM Paul Ganssle <paul@ganssle.io> wrote:
I don't think it's totally unreasonable to have other total_X() methods, where X would be days, hours, minutes and microseconds
I do. I was against adding the total_seconds() method to begin with because the same effect can be achieved with delta / timedelta(seconds=1) this is easily generalized to delta / timedelta(X=1) where X can be days, hours, minutes or microseconds.
Ah yes, good point, I forgot about this because IIRC it's not supported in Python 2.7, so it's not a particularly common idiom in polyglot library code. Obviously any new methods would be Python 3-only, so there's no benefit to adding them. Best, Paul On 2/14/19 1:12 PM, Alexander Belopolsky wrote:
On Thu, Feb 14, 2019 at 9:07 AM Paul Ganssle <paul@ganssle.io <mailto:paul@ganssle.io>> wrote:
I don't think it's totally unreasonable to have other total_X() methods, where X would be days, hours, minutes and microseconds
I do. I was against adding the total_seconds() method to begin with because the same effect can be achieved with
delta / timedelta(seconds=1)
this is easily generalized to
delta / timedelta(X=1)
where X can be days, hours, minutes or microseconds.
On Fri, 15 Feb 2019 at 04:15, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Thu, Feb 14, 2019 at 9:07 AM Paul Ganssle <paul@ganssle.io> wrote:
I don't think it's totally unreasonable to have other total_X() methods, where X would be days, hours, minutes and microseconds
I do. I was against adding the total_seconds() method to begin with because the same effect can be achieved with
delta / timedelta(seconds=1)
this is easily generalized to
delta / timedelta(X=1)
where X can be days, hours, minutes or microseconds.
As someone who reads date/time manipulation code far more often then he writes it, it's immediately obvious to me what "delta.total_seconds()" is doing, while "some_var / some_other_var" could be doing anything. So for the sake of those us that aren't as well versed in how time delta division works, it seems to me that adding: def total_duration(td, interval=timedelta(seconds=1)): return td / interval as a module level helper function would make a lot of sense. (This is a variant on Paul's helper function that accepts the divisor as a specifically named argument with a default value, rather than creating it on every call) Cheers, Nick. P.S. Why a function rather than a method? Mostly because this feels like "len() for timedelta objects" to me, but also because as a helper function, the docs can easily describe how to add it as a utility function for older versions. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I am definitely sympathetic to the idea of it being more readable, but I feel like this adds some unnecessary bloat to the interface when "divide the value by the units" is not at all uncommon. Plus, if you add a total_duration that by default does the same thing as total_seconds, you now have three functions that do exactly the same thing: - td / timedelta(seconds=1) - td.total_seconds() - total_duration(td) If it's just for the purposes of readability, you can also do this: from operator import truediv as total_duration # (timedelta, interval) I think if we add such a function, it will essentially be just a slower version of something that already exists. I suspect the main reason the "divide the timedelta by the interval" thing isn't a common enough idiom that people see it all the time is that it's only supported in Python 3. As more code drops Python 2, I think the "td / interval" idiom will hopefully become common enough that it will obviate the need for a total_duration function. That said, if people feel very strongly that a total_duration function would be useful, maybe the best thing to do would be for me to add it to dateutil.utils? In that case it would at least be available in Python 2, so people who find it more readable /and/ people still writing polyglot code would be able to use it, without the standard library unnecessarily providing two ways to do the exact same thing. On 2/16/19 11:59 AM, Nick Coghlan wrote:
On Fri, 15 Feb 2019 at 04:15, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Thu, Feb 14, 2019 at 9:07 AM Paul Ganssle <paul@ganssle.io> wrote:
I don't think it's totally unreasonable to have other total_X() methods, where X would be days, hours, minutes and microseconds I do. I was against adding the total_seconds() method to begin with because the same effect can be achieved with
delta / timedelta(seconds=1)
this is easily generalized to
delta / timedelta(X=1)
where X can be days, hours, minutes or microseconds.
As someone who reads date/time manipulation code far more often then he writes it, it's immediately obvious to me what "delta.total_seconds()" is doing, while "some_var / some_other_var" could be doing anything.
So for the sake of those us that aren't as well versed in how time delta division works, it seems to me that adding:
def total_duration(td, interval=timedelta(seconds=1)): return td / interval
as a module level helper function would make a lot of sense. (This is a variant on Paul's helper function that accepts the divisor as a specifically named argument with a default value, rather than creating it on every call)
Cheers, Nick.
P.S. Why a function rather than a method? Mostly because this feels like "len() for timedelta objects" to me, but also because as a helper function, the docs can easily describe how to add it as a utility function for older versions.
On Sun, 17 Feb 2019 at 03:20, Paul Ganssle <paul@ganssle.io> wrote:
I think if we add such a function, it will essentially be just a slower version of something that already exists. I suspect the main reason the "divide the timedelta by the interval" thing isn't a common enough idiom that people see it all the time is that it's only supported in Python 3. As more code drops Python 2, I think the "td / interval" idiom will hopefully become common enough that it will obviate the need for a total_duration function.
And personally, the total_seconds() case has always been enough for me.
That said, if people feel very strongly that a total_duration function would be useful, maybe the best thing to do would be for me to add it to dateutil.utils? In that case it would at least be available in Python 2, so people who find it more readable and people still writing polyglot code would be able to use it, without the standard library unnecessarily providing two ways to do the exact same thing.
I'm now thinking a slight documentation improvement would have addressed my own confusion (and I suspect the OPs as well): * In the "Supported Operations" section of https://docs.python.org/3/library/datetime.html#timedelta-objects, change "Division (3) of t2 by t3." to "Division (3) of overall duration t2 by interval unit t3." * In the total_seconds() documentation, add a sentence "For interval units other than seconds, use the division form directly (e.g. `td / timedelta(microseconds=1)`)" Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
+1 on the improved docs solution: no new code to maintain and big return on investment in preventing future bugs / confusion :) On Sat, Feb 16, 2019 at 9:40 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
I think if we add such a function, it will essentially be just a slower version of something that already exists. I suspect the main reason the "divide the timedelta by the interval" thing isn't a common enough idiom
On Sun, 17 Feb 2019 at 03:20, Paul Ganssle <paul@ganssle.io> wrote: that people see it all the time is that it's only supported in Python 3. As more code drops Python 2, I think the "td / interval" idiom will hopefully become common enough that it will obviate the need for a total_duration function.
And personally, the total_seconds() case has always been enough for me.
That said, if people feel very strongly that a total_duration function would be useful, maybe the best thing to do would be for me to add it to dateutil.utils? In that case it would at least be available in Python 2, so people who find it more readable and people still writing polyglot code would be able to use it, without the standard library unnecessarily providing two ways to do the exact same thing.
I'm now thinking a slight documentation improvement would have addressed my own confusion (and I suspect the OPs as well):
* In the "Supported Operations" section of https://docs.python.org/3/library/datetime.html#timedelta-objects, change "Division (3) of t2 by t3." to "Division (3) of overall duration t2 by interval unit t3." * In the total_seconds() documentation, add a sentence "For interval units other than seconds, use the division form directly (e.g. `td / timedelta(microseconds=1)`)"
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/tahafut%40gmail.com
while "some_var / some_other_var" could be doing anything.
"At an elementary level the division of two natural numbers is – among other possible interpretations – the process of calculating the number of times one number is contained within another one." -- <https://en.wikipedia.org/wiki/Division_(mathematics)> The process of figuring out how many seconds fit into a given interval is called division by a second. I am afraid people who get confused by timedelta / timedelta division know too much about Python where / can indeed mean anything including e.g. joining filesystem paths.
The TimeDelta class is derived from the Time class and shares many of its
You could specify the return value type annotations in the docstring of a convert()/to_unit() method, or for each to_unit() method. Are they all floats? div and floordiv are not going away. Without reading the docs, I, too, wouldn't have guessed that division by the desired unit is the correct way. In terms of performance, a convert()/to_unit() method must do either a hash lookup or multiple comparisons to present a more useful exception than that returned by initializing timedelta with something like nanoseconds=. FWIW, astropy.time.TimeDelta supports sub-nanosecond precision and has a .to() method for changing units/quantities. http://docs.astropy.org/en/stable/time/#time-deltas properties. One difference is that the time scale has to be one for which one day is exactly 86400 seconds. Hence, the scale cannot be UTC.
The available time formats are:
Format Class sec TimeDeltaSec jd TimeDeltaJD datetime TimeDeltaDatetime
http://docs.astropy.org/en/stable/api/astropy.time.TimeDelta.html#astropy.ti... http://docs.astropy.org/en/stable/_modules/astropy/time/core.html#TimeDelta.... On Thursday, February 28, 2019, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
while "some_var / some_other_var" could be doing anything.
"At an elementary level the division of two natural numbers is – among other possible interpretations – the process of calculating the number of times one number is contained within another one."
-- <https://en.wikipedia.org/wiki/Division_(mathematics)>
The process of figuring out how many seconds fit into a given interval is called division by a second.
I am afraid people who get confused by timedelta / timedelta division know too much about Python where / can indeed mean anything including e.g. joining filesystem paths.
A function with "microseconds" in the name IMO misleadingly suggests that it has something closer to microsecond accuracy than a 1-second granularity. Rob Cliffe On 14/02/2019 05:05:54, Richard Belleville via Python-Dev wrote:
In a recent code review, the following snippet was called out as reinventing the wheel:
_MICROSECONDS_PER_SECOND = 1000000
def _timedelta_to_microseconds(delta): return int(delta.total_seconds() * _MICROSECONDS_PER_SECOND)
The reviewer thought that there must already exist a standard library function that fulfills this functionality. After we had both satisfied ourselves that we hadn't simply missed something in the documentation, we decided that we had better raise the issue with a wider audience.
Does this functionality already exist within the standard library? If not, would a datetime.timedelta.total_microseconds function be a reasonable addition? I would be happy to submit a patch for such a thing.
Richard Belleville
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rob.cliffe%40btinternet.c...
On Fri, Feb 15, 2019 at 11:58 AM Rob Cliffe via Python-Dev < python-dev@python.org> wrote:
A function with "microseconds" in the name IMO misleadingly suggests that it has something closer to microsecond accuracy than a 1-second granularity.
it sure does, but `delta.total_seconds()` is a float, so ms accuracy is preserved. However, if you DO want a "timedelta_to_microseconds" function, it really should use the microseconds field in the timedelta object. I haven't thought it through, but it makes me nervous to convert to floating point, and then back again -- for some large values of timedelta some precision may be lost. Also: _MICROSECONDS_PER_SECOND = 1000000 really? why in the world would you define a constant for something that simple that can never change? (and probably isn't used in more than one place anyway As Alexander pointed out the canonical way to spell this would be: delta / timedelta(microseconds=1) but I think that is less than obvious to the usual user, so I think a: delta.total_microseconds() would be a reasonable addition. I know I use .totalseconds() quite a bit, and would not want to have to spell it: delta / timedelta(seconds=1) (and can't do that in py2 anyway) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
I'm still with Alexander on this. I see functions like total_X as basically putting one of the arguments directly in the function name - it should be `total_duration(units)`, not `total_units()`, because all of those functions do the same thing and only differ in the units they use. But Alexander's approach of "divide it by the base unit" is /even more general/ than this, because it allows you to use non-traditional units like weeks (timedelta(days=7)) or "two-day periods" or whatever you want. If you use this idiom a lot and want a simple "calculate the total" function, this should suffice: def total_duration(td, *args, **kwargs): return td / timedelta(*args, **kwargs) Then you can spell "x.total_microseconds()" as: total_duration(x, microseconds=1) Or you can write it like this: def total_duration(td, units='seconds'): return td / timedelta(**{units: 1}) In which case it would be spelled: total_duration(x, units='microseconds') I don't see there being any compelling reason to add a bunch of methods for a marginal (and I'd say arguable) gain in aesthetics. On 2/15/19 4:48 PM, Chris Barker via Python-Dev wrote:
On Fri, Feb 15, 2019 at 11:58 AM Rob Cliffe via Python-Dev <python-dev@python.org <mailto:python-dev@python.org>> wrote:
A function with "microseconds" in the name IMO misleadingly suggests that it has something closer to microsecond accuracy than a 1-second granularity.
it sure does, but `delta.total_seconds()` is a float, so ms accuracy is preserved.
However, if you DO want a "timedelta_to_microseconds" function, it really should use the microseconds field in the timedelta object. I haven't thought it through, but it makes me nervous to convert to floating point, and then back again -- for some large values of timedelta some precision may be lost.
Also:
_MICROSECONDS_PER_SECOND = 1000000
really? why in the world would you define a constant for something that simple that can never change? (and probably isn't used in more than one place anyway As Alexander pointed out the canonical way to spell this would be:
delta / timedelta(microseconds=1)
but I think that is less than obvious to the usual user, so I think a:
delta.total_microseconds()
would be a reasonable addition.
I know I use .totalseconds() quite a bit, and would not want to have to spell it:
delta / timedelta(seconds=1)
(and can't do that in py2 anyway)
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov <mailto:Chris.Barker@noaa.gov>
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io
Indeed there is a potential loss of precision: _timedelta_to_microseconds(timedelta(0, 1, 1)) returns 1000000 where conversion function is defined according to the initial message in this thread On Fri, Feb 15, 2019 at 2:29 PM Paul Ganssle <paul@ganssle.io> wrote:
I'm still with Alexander on this. I see functions like total_X as basically putting one of the arguments directly in the function name - it should be `total_duration(units)`, not `total_units()`, because all of those functions do the same thing and only differ in the units they use.
But Alexander's approach of "divide it by the base unit" is *even more general* than this, because it allows you to use non-traditional units like weeks (timedelta(days=7)) or "two-day periods" or whatever you want. If you use this idiom a lot and want a simple "calculate the total" function, this should suffice:
def total_duration(td, *args, **kwargs): return td / timedelta(*args, **kwargs)
Then you can spell "x.total_microseconds()" as:
total_duration(x, microseconds=1)
Or you can write it like this:
def total_duration(td, units='seconds'): return td / timedelta(**{units: 1})
In which case it would be spelled:
total_duration(x, units='microseconds')
I don't see there being any compelling reason to add a bunch of methods for a marginal (and I'd say arguable) gain in aesthetics. On 2/15/19 4:48 PM, Chris Barker via Python-Dev wrote:
On Fri, Feb 15, 2019 at 11:58 AM Rob Cliffe via Python-Dev < python-dev@python.org> wrote:
A function with "microseconds" in the name IMO misleadingly suggests that it has something closer to microsecond accuracy than a 1-second granularity.
it sure does, but `delta.total_seconds()` is a float, so ms accuracy is preserved.
However, if you DO want a "timedelta_to_microseconds" function, it really should use the microseconds field in the timedelta object. I haven't thought it through, but it makes me nervous to convert to floating point, and then back again -- for some large values of timedelta some precision may be lost.
Also:
_MICROSECONDS_PER_SECOND = 1000000
really? why in the world would you define a constant for something that simple that can never change? (and probably isn't used in more than one place anyway
As Alexander pointed out the canonical way to spell this would be:
delta / timedelta(microseconds=1)
but I think that is less than obvious to the usual user, so I think a:
delta.total_microseconds()
would be a reasonable addition.
I know I use .totalseconds() quite a bit, and would not want to have to spell it:
delta / timedelta(seconds=1)
(and can't do that in py2 anyway)
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
_______________________________________________ Python-Dev mailing listPython-Dev@python.orghttps://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/tahafut%40gmail.com
This thread petered out, seemingly with a consensus that we should update the docs -- is anyone doing that? But anyway, I'd like to offer a counterpoint:
From the OP, it is clear that:
* Folks have a need for timedeltas to be converted to numbers values, with units other than seconds (milliseconds, at least). * If left to their own devices, they may well do it wrong (Or at least not as right as they should. So: it would be good to provide a correct, simple, intuitive, and discoverable way to do that. timedelta.total_seconds() Provides that for seconds, but there is no equivalent for other units. a_time_delta / timedelta(microseconds=1) Is now possible in py3, and has been proposed as the canonical way to convert to specific time units. However, while it does provide a correct[1] way to do it, it is: - not very simple - not very intuitive. - not the least bit discoverable Each of these in turn: simple: ===== compare duration = a_timedelta.total_seconds() to duration = a_timedelta / datetime.timedelta(seconds=1) Keep in mind that the timedelta object may have been generated by another module somehow, so the coder that gets it and wants to turn it into a number of seconds (or milliseconds, or ...) needs to import datetime and reference the timedelta object. And if they are converting to a plain number, it's probably because they don't want to be working with timedeltas at all anyway. So no, not so simple. intuitive: ====== A casual user reading the first will very likely know what it means -- a casual user reading the second line will need to think about it carefully, and probably have to go read the datetime docs, or at least do some experiments to make sure it does what they think it does. Granted, a comment: duration = a_timedelta / datetime.timedelta(seconds=1) # convert to seconds would help a lot, but if you need a comment to explain a line of code this simple, then it's not intuitive. A couple more data points: -- I am a physical scientist, I work with unitted quantities all the time (both in code and in other contexts). It never dawned on me to use this approach to convert to seconds or milliseconds, or ... Granted, I still rely on python2 for a fair bit of my work, but still, I had to scratch my head when it was proposed on this thread. -- There are a number of physical unit libraries in Python, and as far as I know, none of them let you do this to create a unitless value in a particular unit. "pint" for example: https://pint.readthedocs.io/en/latest/ In pint, you can create objects with units, including time: In [50]: timespan = 2 * ureg.day In [51]: print(timespan) 2 day But if you divide a value of days by a value in seconds, you don't get a unitless seconds per day: In [54]: unitless = timespan / (1 * ureg.second) In [55]: print(unitless) 2.0 day / second Though pint does know it is dimensionless: In [56]: unitless.dimensionless Out[56]: True And you can reduce it to a dimensionless object: In [57]: unitless.to_reduced_units() Out[57]: 172800.0 <Unit('dimensionless')> And there is your seconds value. But the "right" way to get a given pint object of time into particular units is to convert, and then, if you want a plain number, get the magnitude: In [53]: print(timespan.to('second').magnitude) 172800.0 So no -- dividing a datetime by another datetime with the value you want is not intuitive: not to a physical scientist, not to a user of other physical quantities libraries -- is it intuitive to anyone other than someone that was involved in python datetime development?? Discoverable: ========== It is clearly not discoverable -- the OP didn't find it, and no one other than Alexander found it on this thread (I'm pretty sure). That could be made much better with docs, but we all know no one reads docs -- I'm willing to bet that even if we better document it, folks will still be writing utility functions like the OP posted. And (this is also a doc issue) -- I wanted to know what the options were for units we could specify to the datetime constructor, so I used the nifty iPython ?, and got: In [59]: timedelta? Init signature: timedelta(self, /, *args, **kwargs) Docstring: Difference between two datetime values. Gosh, that's helpful! (we really need to fix that regardless of this thread). And someone earlier mentioned "weeks" without realizing that is was already supported: In [60]: timedelta(weeks=1) Out[60]: datetime.timedelta(days=7) So we have a real discoverability problem -- we really need to fix that. On the other hand, if we add a few convenience methods, we will have a way for folks to do this that is: correct simple intuitive discoverable And we really don't need to add many. Looking at the docs ('cause the docstring is useless), I see that the timedelta takes: datetime.timedelta(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0) So at most, we could have: .total_microseconds() .total_seconds() .total_minutes() .total_hours() .total_days() .total_weeks() Yes, I know that that's more code to write, maintain and document, but really, it will not take long to write, will take less work to document than the doc improvements we need anyway, and hardly any extra maintenance burden. (and the utility of weeks and minutes is questionable) BTW, why are these methods, rather than properties? Another option, is, of course, to add a: .to_unit() method that takes the above as strings. -- but unless we ar going to support a bunch more units than those six, I'd rather see the methods. And what would those others be? fortnights? I've made my case -- and maybe we won't do this. But let's please at least update the docstring of timedleta! -CHB [1] is it the best possible for microseconds? it returns a float, which can only carry 15 or so digits, which is "only" 300 or so years. So microsecond precision is lost for timedeltas representing longer than that -- does that matter ??? -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 2/26/2019 7:03 PM, Chris Barker via Python-Dev wrote:
So: it would be good to provide a correct, simple, intuitive, and discoverable way to do that.
timedelta.total_seconds()
To me, total_x implies that there is a summation of multiple timedeltas, and there is not. So not intuitive to me. (Neither us current obscure option). It is also not obvious is answer is rounded to nearest second or not.
So at most, we could have:
.total_microseconds() .total_seconds() .total_minutes() .total_hours() .total_days() .total_weeks()
I am also not enthusiastic about multiple methods doing essentially the same thing. I might prefer one method, .convert? with an argument specifying the conversion unit, 'microseconds', 'seconds', ... . I think this is in python-ideas territory. -- Terry Jan Reedy
On Tue, Feb 26, 2019 at 10:20 PM Terry Reedy <tjreedy@udel.edu> wrote:
To me, total_x implies that there is a summation of multiple timedeltas, and there is not.
Do you believe this is a particularly dominant perception? I don't, but specific backgrounds probably play into this heavily. I'd expect to total a bunch of timedelta values using sum([d0, d1, ..., dn]). Given we already have total_seconds(), it's not clear avoiding additional methods is meaningful, unless we're going to deprecate total_seconds(). Not really a win in my book. I'd rather stick with the existing pattern, if anything even needs to be done. I'm quite happy to use d.total_seconds() * 1000000 as long as the accuracy is sufficient. Someone with more floating point expertise can probably think of a reason that's not good enough, in which case... an appropriate method wouldn't be poorly named as total_microseconds.
I might prefer one method, .convert? with an argument specifying the conversion unit, 'microseconds', 'seconds', ... .
Using a function that takes a units indicator (as d.convert(units='microseconds')) seems like a poor choice; most uses will hard-code exactly one value for the units, rather than passing in a variable. Getting a more specific name seems reasonable.
It is also not obvious is answer is rounded to nearest second or not.
No, but that's a problem we have now with total_seconds(). Best handled by maintaining the pattern and documenting the behavior. While fractional microseconds aren't a thing with timedelta values now (and probably not in any near future), it seems good to keep these floats so things stay consistent if we can ever get better clocks. :-) -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
On Tue, Feb 26, 2019 at 7:22 PM Terry Reedy <tjreedy@udel.edu> wrote:
timedelta.total_seconds()
To me, total_x implies that there is a summation of multiple timedeltas, and there is not. So not intuitive to me.
THAT was a discussion for when it was added -- I can't say it's my favorite name either. But at least a quick glance at the docstring will address that. Hmm -- that one could use a little enhancement, too: In [3]: td.total_seconds? Docstring: Total seconds in the duration. But at least it COULD be made clear in a docstring :-)
(Neither us current obscure option). It is also not obvious is answer is rounded to nearest second or not.
also could be made clear in a docstring -- the full docs say: """ timedelta.total_seconds() Return the total number of seconds contained in the duration. Equivalent to td / timedelta(seconds=1). Note that for very large time intervals (greater than 270 years on most platforms) this method will lose microsecond accuracy. """ That last point indicates that it is not rounded -- and a quick check will note that it returns a float -- but those docs could be improved to make it clear that a float is returned. (i.e. that is WHY it loses microsecond accuracy). But anyway, you are nevert going to know all the intricacies of a method by its name -- you need to know about the rounding and all if you are trying to decide if you are going to us the method, but if you read it, you probably have a pretty good idea what it does. Anyway -- "I don't really like the name much" is really the answer to a different question (I don't like it either, but it's what's there)
So at most, we could have:
.total_microseconds() .total_seconds() .total_minutes() .total_hours() .total_days() .total_weeks()
I am also not enthusiastic about multiple methods doing essentially the same thing. I might prefer one method, .convert? with an argument specifying the conversion unit, 'microseconds', 'seconds', ... .
yup -- that's been proposed, and is still on the table as far as I'm concerned. Though I'm not sure I like "convert", but would rather, say "in_unit" or "to_unit". But any of these would address the intuitive and discoverability issue -- very few folks would have any trouble guessing what any of: a_datetime.convert("hours") a_datetime.to_unit("hours") a_datetime.total_hours() mean -- at least generally, even if they con't know if they are getting a float or an integer result. And as for discoverable, any of these as methods on a dt object would probably be checked out if you were looking for such a thing.
I think this is in python-ideas territory.
I'd rather not :-) -- have you SEEN the length of the thread on how long the line limiet should be in PEP8. Frankly, I think the arguments have been made, and the options on the table -- if this were a PEP, it would be time for a pronouncement -- or at least one of: - not going to happen or - maybe happen but we need to iron out the details I've lost track of where we are with the governance model, so is there a way to get a clear idea for whether it's time to drop this? BTW, I don't think this is worth a PEP, but if I get a go-ahead, and it needs to be written up, I'd be willing to do so. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 2/26/19 7:03 PM, Chris Barker via Python-Dev wrote:
This thread petered out, seemingly with a consensus that we should update the docs -- is anyone doing that?
I don't think anyone is, I've filed a BPO bug for it: https://bugs.python.org/issue3613
-- I am a physical scientist, I work with unitted quantities all the time (both in code and in other contexts). It never dawned on me to use this approach to convert to seconds or milliseconds, or ... Granted, I still rely on python2 for a fair bit of my work, but still, I had to scratch my head when it was proposed on this thread.
As another data point, I also have a background in the physical sciences, and I actually do find it quite intuitive. The first time I saw this idiom I took it to heart immediately and only stopped using it because many of the libraries I maintain still support Python 2. It seemed pretty obvious that I had a `timedelta` object that represents times, and dividing it by a base value would give me the number of times the "unit" timedelta fits into the "value" timedelta. Seeing the code `timedelta(days=7) / timedelta(days=1)`, I think most people could confidently say that that should return 7, there's really no ambiguity about it.
-- There are a number of physical unit libraries in Python, and as far as I know, none of them let you do this to create a unitless value in a particular unit. "pint" for example:
https://pint.readthedocs.io/en/latest/
...
And you can reduce it to a dimensionless object:
In [57]: unitless.to_reduced_units() Out[57]: 172800.0 <Unit('dimensionless')>
I think the analogy with pint's unit-handling behavior is not completely appropriate, because `timedelta` values are not in /specific/ units at all, they are just in abstract duration units. It makes sense to consider "seconds / day" as a specific dimensionless unit in the same sense that "percent" and "ppb" make sense as specific dimensionless units, and so that would be the natural behavior I would expect for unit-handling code. For timedelta, we don't have it as a value in specific units, so it's not clear what the "ratio unit" would be. What we're looking at with timedelta division is really more "how many <unit>s are there in this duration".
So no -- dividing a datetime by another datetime with the value you want is not intuitive: not to a physical scientist, not to a user of other physical quantities libraries -- is it intuitive to anyone other than someone that was involved in python datetime development??
Just to clarify, I am involved in Python datetime development now, but I have only been involved in Python OSS for the last 4-5 years. I remember finding it intuitive when I (likely working as a physicist at the time) first saw it used.
Discoverable: ==========
I agree that it is not discoverable (which is unfortunate), but you could say the same thing of /all/ operators. There's no tab-completion that will tell you that `3.4 / 1` is a valid operation or that (3,) + (4,) will work, but we don't generally recommend adding methods for those things. I do think the discoverability is hindered by the existence of the total_seconds method, because the fact that total_seconds exists makes you think that it is the correct way to get the number of seconds that a timedelta represents, and that you should be looking for other analogous methods as the "correct" way to do this, when in fact we have a simpler, less ambiguous (for example, it's not obvious whether the methods would truncate or not, whereas __truediv__ and __floordiv__ gives the division operation pretty clear semantics) and more general way to do things. I think it's too late to /remove/ `total_seconds()`, but I don't think we should be compounding the problem by bloating the API with a bunch of other methods, per my earlier arguments.
We should also consider that before long datetimes are frequently stored to nanosecond resolution (two fields where this is significant are finance and physics, and the latter would probably appreciate femtosedonds as well). So maybe an external library that layers on top of datetime might be preferable? Kind regards Steve Holden On Wed, Feb 27, 2019 at 3:13 PM Paul Ganssle <paul@ganssle.io> wrote:
On 2/26/19 7:03 PM, Chris Barker via Python-Dev wrote:
This thread petered out, seemingly with a consensus that we should update the docs -- is anyone doing that?
I don't think anyone is, I've filed a BPO bug for it: https://bugs.python.org/issue3613
-- I am a physical scientist, I work with unitted quantities all the time (both in code and in other contexts). It never dawned on me to use this approach to convert to seconds or milliseconds, or ... Granted, I still rely on python2 for a fair bit of my work, but still, I had to scratch my head when it was proposed on this thread.
As another data point, I also have a background in the physical sciences, and I actually do find it quite intuitive. The first time I saw this idiom I took it to heart immediately and only stopped using it because many of the libraries I maintain still support Python 2.
It seemed pretty obvious that I had a `timedelta` object that represents times, and dividing it by a base value would give me the number of times the "unit" timedelta fits into the "value" timedelta. Seeing the code `timedelta(days=7) / timedelta(days=1)`, I think most people could confidently say that that should return 7, there's really no ambiguity about it.
-- There are a number of physical unit libraries in Python, and as far as I know, none of them let you do this to create a unitless value in a particular unit. "pint" for example:
https://pint.readthedocs.io/en/latest/
...
And you can reduce it to a dimensionless object:
In [57]: unitless.to_reduced_units()
Out[57]: 172800.0 <Unit('dimensionless')>
I think the analogy with pint's unit-handling behavior is not completely appropriate, because `timedelta` values are not in *specific* units at all, they are just in abstract duration units. It makes sense to consider "seconds / day" as a specific dimensionless unit in the same sense that "percent" and "ppb" make sense as specific dimensionless units, and so that would be the natural behavior I would expect for unit-handling code.
For timedelta, we don't have it as a value in specific units, so it's not clear what the "ratio unit" would be. What we're looking at with timedelta division is really more "how many <unit>s are there in this duration".
So no -- dividing a datetime by another datetime with the value you want is not intuitive: not to a physical scientist, not to a user of other physical quantities libraries -- is it intuitive to anyone other than someone that was involved in python datetime development??
Just to clarify, I am involved in Python datetime development now, but I have only been involved in Python OSS for the last 4-5 years. I remember finding it intuitive when I (likely working as a physicist at the time) first saw it used.
Discoverable: ==========
I agree that it is not discoverable (which is unfortunate), but you could say the same thing of *all* operators. There's no tab-completion that will tell you that `3.4 / 1` is a valid operation or that (3,) + (4,) will work, but we don't generally recommend adding methods for those things.
I do think the discoverability is hindered by the existence of the total_seconds method, because the fact that total_seconds exists makes you think that it is the correct way to get the number of seconds that a timedelta represents, and that you should be looking for other analogous methods as the "correct" way to do this, when in fact we have a simpler, less ambiguous (for example, it's not obvious whether the methods would truncate or not, whereas __truediv__ and __floordiv__ gives the division operation pretty clear semantics) and more general way to do things.
I think it's too late to *remove* `total_seconds()`, but I don't think we should be compounding the problem by bloating the API with a bunch of other methods, per my earlier arguments. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com
On Wed, Feb 27, 2019 at 7:15 AM Paul Ganssle <paul@ganssle.io> wrote:
As another data point, I also have a background in the physical sciences, and I actually do find it quite intuitive. The first time I saw this idiom I took it to heart immediately and only stopped using it because many of the libraries I maintain still support Python 2.
we know have two contrasting data points -- it's settled! But I'd like to know -- I too will probably remember and use this idiom now that I know it (if not take it to heart ...) -- but would you hvae figured it out on your own?
Seeing the code `timedelta(days=7) / timedelta(days=1)`, I think most people could confidently say that that should return 7, there's really no ambiguity about it.
in that example absolutely -- but this is the one I'm worried about: an_arbitrary_dateitme_returned_from_a_lib_you_didn't_write / datetime.timedelta(days=7) not so obvious.
I think the analogy with pint's unit-handling behavior is not completely appropriate, because `timedelta` values are not in *specific* units at all, they are just in abstract duration units.
agreed -- but it is a use-case that is worth looking into -- folks will either have no experience with units, or with a lib like that one (there are more, and I think they all handle it a similar way) And if you have a pint object representing time that you do not know the units of, the way to get it into the units you want is to call teh to() method: a_time_in_arbitrary_units.to('second') That is an obvious and familiar API.
I agree that it is not discoverable (which is unfortunate), but you could say the same thing of *all* operators.
sure, but most operators are really widely used.
There's no tab-completion that will tell you that `3.4 / 1` is a valid operation
well, THAT one had better be obvious :-)
or that (3,) + (4,) will work, but we don't generally recommend adding methods for those things.
and that IS a source of confusion, but at least for the built-in types (and ones that conform to the primary ABCs), people are going to learn it pretty early. But conversion of a datetime object into particular units is a pretty specialized operation. Also, in most (all?) cases with builtin (and most of the standard library) the math operators return the same type (or a conceptional same type: int / int => float -- all numbers)
I do think the discoverability is hindered by the existence of the total_seconds method, because the fact that total_seconds exists makes you think that it is the correct way to get the number of seconds that a timedelta represents, and that you should be looking for other analogous methods as the "correct" way to do this,
well, yes. I for one, was very happy to have found it when I did :-) -- of course that was py2, when it WAS the only easy way to do it.
when in fact we have a simpler, less ambiguous (for example, it's not obvious whether the methods would truncate or not, whereas __truediv__ and __floordiv__ gives the division operation pretty clear semantics)
I don't think so, I was actually a touch surprised that: a_timedelta / timedelta(microseconds=1) yielded a float that may have lost precision, when the timedetla type stores ms precision, and python ints can handle any size int. it seems __floordiv__ does preserve precision. Which is kinda ironic -- after all, it's called "ture division" and "floor division", not "float division" and "integer division" My point being -- if you care about the details, you're going to have to dig deeper no matter what method is used. I think it's too late to *remove* `total_seconds()`, but I don't think we
should be compounding the problem by bloating the API with a bunch of other methods, per my earlier arguments.
I'm not a big fan of bloated APIs -- but this is not a lot of methods, with a clear naming pattern, on an API without a huge number of methods (seven non-dunder attributes by my count) already.
From a practicality beats purity standpoint -- a few new methods (or a single conversion method) is less pure and "correct", but also a lot more usable.
-CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Did we ever hear back from the OP as to whether they were using py2 or 3? If they were unable to find timedelta division in py3 -- that's a pretty good case that we need something else. The OP: """ On Wed, Feb 13, 2019 at 9:10 PM Richard Belleville via Python-Dev < python-dev@python.org> wrote:
In a recent code review, the following snippet was called out as reinventing the wheel:
_MICROSECONDS_PER_SECOND = 1000000
def _timedelta_to_microseconds(delta): return int(delta.total_seconds() * _MICROSECONDS_PER_SECOND)
The reviewer thought that there must already exist a standard library function that fulfills this functionality. After we had both satisfied ourselves that we hadn't simply missed something in the documentation, we decided that we had better raise the issue with a wider audience.
""" -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Sorry for the slow response. Timedelta division is quite a nice solution to the problem. However, since we're maintaining a python version agnostic library at least until 2020, we need a solution that works in python 2 as well. For the moment, we've left the code as in the original snippet. Richard Belleville On Wed, Feb 27, 2019 at 2:56 PM Chris Barker <chris.barker@noaa.gov> wrote:
Did we ever hear back from the OP as to whether they were using py2 or 3?
If they were unable to find timedelta division in py3 -- that's a pretty good case that we need something else.
The OP: """ On Wed, Feb 13, 2019 at 9:10 PM Richard Belleville via Python-Dev < python-dev@python.org> wrote:
In a recent code review, the following snippet was called out as reinventing the wheel:
_MICROSECONDS_PER_SECOND = 1000000
def _timedelta_to_microseconds(delta): return int(delta.total_seconds() * _MICROSECONDS_PER_SECOND)
The reviewer thought that there must already exist a standard library function that fulfills this functionality. After we had both satisfied ourselves that we hadn't simply missed something in the documentation, we decided that we had better raise the issue with a wider audience.
"""
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
On Wed, Feb 27, 2019 at 3:04 PM Richard Belleville <rbellevi@google.com> wrote:
Timedelta division is quite a nice solution to the problem. However, since we're maintaining a python version agnostic library at least until 2020, we need a solution that works in python 2 as well.
So you were limited to a py2 solution. But di you poke around in py3 before posting? (you should have, python-dev is really about active development, i.e. python 3 -- but that's not the point here)
For the moment, we've left the code as in the original snippet.
If you care about microsecond precision for large timeseltas, you may want to improve that. I *think* this is the "correct" way to do it: def timedelta_to_microseconds(td): return td.microseconds + td.seconds * 1000 + td.days * 86400000 (hardly tested) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
participants (12)
-
Alexander Belopolsky -
Chris Barker -
Fred Drake -
Henry Chen -
INADA Naoki -
Nick Coghlan -
Paul Ganssle -
Richard Belleville -
Rob Cliffe -
Steve Holden -
Terry Reedy -
Wes Turner