Mailman 3 July 2015 - Datetime-SIG

Re: [Datetime-SIG] Datetime arithmetic proposal
by Tim Peters 29 Jul '15

29 Jul '15

[Chris Barker] >> 1) timespan since an epoch: -- i.e seconds since Jan 1, 1970 00:00. This is >> how the datetime object keeps it internally. [Alexander Belopolsky <alexander.belopolsky(a)gmail.com>] > No, it is not. You are probably confusing stdlib datetime with > numpy.datetime64. > The stdlib datetime keeps broken down year, month, day, hour, minute, second > and microsecond values. It's interesting (well, maybe to someone ;-) ) to note that this division into named attributes is simply for efficiency in extracting commonly accessed values. Conceptually, it's really just storing the integer number of microseconds since the start of "year number 1" of the proleptic Gregorian calendar. All those named fields can be viewed as just representing that integer in a very messy mixed-radix notation. That's not just academic. For example, if you stare at the Python code for the implementation of datetime + timedelta (datetime.__add__), you'll see that it converts the datetime _to_ a timedelta first (which is another way of representing an integer number of microseconds). Of course that conversion has to be done "by hand" since it makes no sense in the user-visible model. But, internally,datetimes, dates, times, and timedeltas are all sometimes viewed as just complicated ways of spelling "some number of microseconds". It's cute ;-)

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Alexander Belopolsky 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 10:24 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > [Chris Barker <chris.barker(a)noaa.gov>] >> what the heck would that mean? the timedeltas already do that -- what they >> don't do is calendar maniplutaions -- things like "this same time tomorrow" >> -- or tomorrow morning at 7:00am. > > We've been through this before, and it's been discussed literally > dozens of times on python-dev recently. So I refuse to use any "word > salad" to try to describe anything anymore - nobody can hear anything > anyone else says ;-) Maybe the following session log (ran from the root of the source distribution) will get the message across: >>> with open('Doc/includes/tzinfo-examples.py') as f: ... exec(f.read()) ... >>> t = datetime(2014,11,1,12,tzinfo=Eastern) >>> t.strftime('%c %Z %z') 'Sat Nov 1 12:00:00 2014 EDT -0400' >>> t += timedelta(1) >>> t.strftime('%c %Z %z') 'Sun Nov 2 12:00:00 2014 EST -0500' This is unmodified Python 3.4.3 running an official documentation example.

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Tim Peters 29 Jul '15

29 Jul '15

[Felipe Ochoa] >> ... >> B. Implement `.get_offset_time(self, hours, minutes, seconds, ms)` and >> `.compute_duration(self, other_dt)` methods for "aware arithmetic." (I.e., >> congruent to 'convert to UTC, increment, convert back') [Chris Barker <chris.barker(a)noaa.gov>] > what the heck would that mean? the timedeltas already do that -- what they > don't do is calendar maniplutaions -- things like "this same time tomorrow" > -- or tomorrow morning at 7:00am. We've been through this before, and it's been discussed literally dozens of times on python-dev recently. So I refuse to use any "word salad" to try to describe anything anymore - nobody can hear anything anyone else says ;-) Start with a datetime object representing day 23 in some month and some year, and time 4:56. Add timedelta(hours=24) to that, or add timedelta(days=1) to that, or (there are many other possible spellings of the same thing) and you will ALWAYS get back a timedelta object representing day 24 in the same month, the same year, the same time (4:56), and with the same tzinfo member. It makes no difference which year, no difference which month, and no difference which tzinfo member you started with. You always get back "tomorrow at the same time (as shown on the local clock)". And, yes, the two _may_ be 23 hours apart, or 24 hours, or 25 hours, or even (e.g.,) 23 hours and 15 minutes apart as measured by a stopwatch, depending on the rules implemented by the tzinfo member. The tzinfo member is never consulted about anything in a datetime + timedelta operation. The tzinfo member is ignored (apart from being copied to the result datetime). The arithmetic is done exactly the same as for a naive datetime object. I think you're calling that a "calendar operation". Other people call it "naive datetime arithmetic", some call it "broken", Guido calls it a feature of his "naive time" design, some call it the "a day is always 24 hours rule", and some call it the "s day is not always 24 hours rule". That's why it's impossible to understand anyone without an excruciatingly detailed example ;-) In any case, that's what Python has always done. It won't change, not even in the imaginary "backward compatibility is irrelevant" Python 4 (because Guido still favors this design). That's for the builtin datetime module and its already-existing objects and operations. There's no backward-compatibility constraint on anything new, of course.

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Chris Barker 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 7:04 PM, Alexander Belopolsky < alexander.belopolsky(a)gmail.com> wrote: > On Tue, Jul 28, 2015 at 9:54 PM, Chris Barker <chris.barker(a)noaa.gov> > wrote: > > 1) timespan since an epoch: -- i.e seconds since Jan 1, 1970 00:00. This > is > > how the datetime object keeps it internally. > > No, it is not. You are probably confusing stdlib datetime with > numpy.datetime64. > or who knows what else! I've been involved with far toomany datetime conversations lately! > The stdlib datetime keeps broken down year, month, day, hour, minute, > second > and microsecond values. > Thanks for the clarification -- that would make calendar manipulations easier, I suppose. And I can see why anyone would even consider storing in particular time zone format. Sorry to confuse things. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Alexander Belopolsky 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 9:54 PM, Chris Barker <chris.barker(a)noaa.gov> wrote: > 1) timespan since an epoch: -- i.e seconds since Jan 1, 1970 00:00. This is > how the datetime object keeps it internally. No, it is not. You are probably confusing stdlib datetime with numpy.datetime64. The stdlib datetime keeps broken down year, month, day, hour, minute, second and microsecond values.

1 0

Re: [Datetime-SIG] What's are the issues?
by Chris Barker 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 6:43 PM, Chris Angelico <rosuav(a)gmail.com> wrote: What if the existing timedelta is left as-is, and a new calendardelta > type is introduced? Instead of folding everything down to a simple > microsecond count, it would retain the exact values it was given, sure -- that would be one way to do it -- maybe good way. But it would be a lot more complicated than just keeping days. Lots to hash out for API and functionality at the corner cases (the old what's one year after Feb 29th? kinds of questions) > When you add a > calendardelta to an aware datetime, it performs what you're describing > as calendar arithmetic, actually, no reason you couldn't do calendar manipulations on naive datetimes, too (they wouldn't get any ugly DST issues, but otherwise the same). One reason I don't like calling the arithmetic naive or not.... > adding > calendardelta(days=1) would never affect the hour/minute/second shown, > unless it would otherwise create an illegal time. exactly -- all those corner cases would need to be worked out. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Chris Barker 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 5:15 PM, Felipe Ochoa <felipe.nospam.ochoa(a)gmail.com > wrote: > 1. We cannot break backwards-compatibility (This point has been beaten to > death) > OK then -- then timedelta is NOT going to be fundamentally re-denied as something else altogether -- good! > 2. There are valid use-cases for wanting datetimes with and without > timezone info, and for doing naive and aware math. All 3 legal combinations > of the two (excluding naive dt with aware deltas :-)) have valid use-cases > sure -- but the datetime module does not have to support them all, and they don't all have to be supported with the same objects. semantics again -- there is no such thing as naive or aware math -- there are timespan computations and calendar manipulations.... > 3. [new point] If I'm a new user looking in the library for a solution to > my problem, my use-case is likely to appear "obviously correct", and I'll > be shocked if Python doesn't support it > True for all possible libraries :-) -- though I suppose we have a special obligation 'cause we claim that "batteries are included" > 4. [new point] The non-trivial nature of all 3 possibilities makes it > advantageous to include in the standard library > There are a lot of non-trivial, and fairly common things that are not included in the stdlib. But I'm all for including calender manipulations in the stlib -- just not for altering what timedelta means, and trying to be realistic that that's actually a big project, with lots of ugly corner cases and API decisions. And there is nothing about that in the PEP, by the way. As far as I can see, the PEP is about including ready to go, standard tzinfo object in the stdlib -- whereas when dateime was introduced, it provided a mechanism, but no implimentation. This is about providing an implementation. A. Keep the current mechanics as-is. `+` and `-` result in "naive" datetime > arithmetic > *sigh* about the wording, but yes. > B. Implement `.get_offset_time(self, hours, minutes, seconds, ms)` and > `.compute_duration(self, other_dt)` methods for "aware arithmetic." (I.e., > congruent to 'convert to UTC, increment, convert back') > what the heck would that mean? the timedeltas already do that -- what they don't do is calendar maniplutaions -- things like "this same time tomorrow" -- or tomorrow morning at 7:00am. Maybe this will help people conceptualize: timedeltas represent what is measured by a stopwatch. datetimes represent what is represented by a calendar and a watch. If you start your stopwatch at a particular date indicated by your calendar, and particular time indicated by your watch, and run it for 24 hours (one day) you can then look at your calendar and watch and see what the datetime is -- and if you crossed a DST border, it may not be the _same_ time on your watch. And if you start your stopwatch at 2:00pm one day, and look at it again at 2:pm the next day, it will say 24 hours, unless you crossed a DST boundary, in which case it will say 23 or 25 hours -- but would that surprise anyone? > C. Come up with a new name (or clearly define naive/aware arithmetic) to > disambiguate the two > sure -- I'll vote to not use the term "aware arithmetic" -- it's not really arithmetic -- "calendar operations"? and the datetimes can be naive, but the operation is neither naive nor aware -- a time span is a time span is a time span.... Another thought here that may help: you can "encode" a datetime in at least two ways: 1) timespan since an epoch: -- i.e seconds since Jan 1, 1970 00:00. This is how the datetime object keeps it internally. 2) Calendar time in some calendar and time zone: year, month, day, hour, min, sec. The "magic" of the datetime object all happens in the conversion between the two. (side note -- timedeltas have no calendar concept at all) most time math is most easily done in encoding (1) -- that's (one reason) why it's used most of the time. But if you want to do calendar manipluations, you are better off working in the calendar encoding -- "two days from now" is much easier to do there. So I think when people are thinking about "time zone aware math" what they really are thinking about is calendar encoded math. By the way, is anyone proposing to add Calendar manipulations to the stdlib? I understand that folks want that, and apparently some people got confused that better time zone support meant that -- but is anyone proposing to do it? If anyone is, then it's probably a topic for Python-ideas and an entirely different PEP. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

Re: [Datetime-SIG] What's are the issues?
by Chris Barker - NOAA Federal 29 Jul '15

29 Jul '15

1. Datetime arithmetic >> a) datetime +/- timedelta >> ii. Aware (with tzinfo) datetime objects > > If the math is done in UTC, then there is no issue, yes? the timedelta is > added or subtracted, and the result has the same tzinfo object. If I've understood correctly, this is one of the bigger points of contention. Specifically, the following operation is quite polarizing: `datetime(2015, 3, 7, 13, 0, tzinfo=zoneinfo('America/New_York') ) + timedelta(days=1)` It raises the following two questions: * Should `days=1` be interpreted the same as `hours=24`? Yes, absolutely! * If it is interpreted the same, should the addition be "naive" (not adjusting for the jump from 1:59 to 3:00) or "aware" If it is 24 hours, then there's no question. So the question really is: should the datetime module provide what I've been calling "calendar operations" ? Yes, that would be very useful, but it's a heck of a lot more complicated than just redefine what a timedelta(days=1) means. And particularly with backward compatibility in mind, it definitely should not be done with that API. If people really are THAT prone to confusion that a day means 24 hours, then we should consider deprecating "days" as a valid unit for a timedelta. Current semantics call for the answers to be "yes" and "naive" Here are the semantics again -- there is nothing naive about subtracting and adding 24 hours. The problem here is that people are conflating a "day" meaning 24 hours and a "day" meaning a numbered entity on a calendar. The two are related, but totally different in meaning. And sorry to get pedantic here, but backwards compatibility aside, I, for one, would be very surprised if a timedelta(days=1) ever meant anything other than 24 hours depending on context! b) datetime - datetime > > ii. When both have same tzinfo > > Just do the math. Store as UTC makes it easy. return a timedelta. This is another contentious issue. What do you want `datetime(2015, 3, 8, 13, tzinfo=zoneinfo('America/New_York')) - datetime(2015, 3, 7, 13, tzinfo=zoneinfo('America/New_York'))` to be? There's a camp for `timedelta(seconds=24 * 60 * 60)` and a Assuming I've got the DST thing right here -- god no! In that time zone, those time stamps are NOT 24 hrs apart! You'd better not mess with that!!!! I'm going to indulge for a minute here: IF there was such a thing as a calendar_delta object of some sort, then you might want one with a value of one day there. But certainly not anything with a value of 24 *3600 seconds! Does anyone really want that? Note that this is s good argument for what calendar operations are so complex. -Chris

2 1

Re: [Datetime-SIG] What's are the issues?
by Tim Peters 29 Jul '15

29 Jul '15

[Felipe Ochoa <felipe.nospam.ochoa(a)gmail.com>] > ... > I'm less clear on the issue here, but Isaac seemed to disagree with the > algorithm in datetime.py > (https://hg.python.org/cpython/file/v3.5.0b1/Lib/datetime.py#l1935) > [http://code.activestate.com/lists/python-dev/137300/] The algorithm is fine, so far as it goes (with which Isaac will agree). "So how far does it go?" ;-) It handles DST transitions correctly in all known timezones(*). But, as the docs have always pointed out, it (specifically .fromutc(), which is at the base of all relevant calculations) doesn't even try to handle transitions due to _other_ causes. The only other such cause I'm aware of is when a timezone's _base_ offset from UTC changes (not a temporary DST transition, the timezone's "standard" offset changes permanently (meaning until someone decides to change it again)). Many timezones have suffered such a fundamental redefinition at least once. There is no pattern whatsoever to when such things occur - seems it's driven purely by local politics. And for any given timezone it typically happens at most once across its lifetime to date. However, while rare, such changes do still happen, and a "wholly correct" implementation needs to cater to them too (meaning a more general .fromutc() implementation needs to be crafted, and the docs updated to match the new scheme). >From there, Isaac went on to define an approach to a more general method, which could obviously handle that kind of thing too. How practical that approach can be made remains unknown, because nobody yet (that I know of) has tried to produce corresponding code. The mathematical abstractions in his writeup appear far more general than actually needed for any timezone yet invented (excepting the absurd "RIyadh Solar" timezone experiment, which was never actually used by anyone in the real world); stronger assumptions could probably allow more efficient implementation. (*) By "timezone" I mean whatever rules a specific tzinfo instance implements. I have a wholly Python-centric view of everything ;-)

1 0

Datetime arithmetic proposal
by Felipe Ochoa 29 Jul '15

29 Jul '15

My viewpoint (specifically on point 1 from my other email): 1. We cannot break backwards-compatibility (This point has been beaten to death) 2. There are valid use-cases for wanting datetimes with and without timezone info, and for doing naive and aware math. All 3 legal combinations of the two (excluding naive dt with aware deltas :-)) have valid use-cases 3. [new point] If I'm a new user looking in the library for a solution to my problem, my use-case is likely to appear "obviously correct", and I'll be shocked if Python doesn't support it 4. [new point] The non-trivial nature of all 3 possibilities makes it advantageous to include in the standard library With that said, my proposal would be to: A. Keep the current mechanics as-is. `+` and `-` result in "naive" datetime arithmetic B. Implement `.get_offset_time(self, hours, minutes, seconds, ms)` and `.compute_duration(self, other_dt)` methods for "aware arithmetic." (I.e., congruent to 'convert to UTC, increment, convert back') C. Come up with a new name (or clearly define naive/aware arithmetic) to disambiguate the two D. Document these differences clearly Downsides: * clunkyness of doing aware arithmetic * `aware_dt + delta` can result in a nonexistent or ambiguous time * `aware_dt_1 - aware_dt_2` behaves differently depending on the underlying timezones * [ fill in the blank -- I'm sure I'm missing many more ] -Felipe

1 0