Mailman 3 July 2015 - Datetime-SIG

Re: [Datetime-SIG] Clearing up terminology
by Tim Peters 31 Jul '15

31 Jul '15

.... [Tim] >>>> [explains that strict/timeline arithmetic _is_ currently done when >>>> subtracting or comparing two aware datetimes] [ijs] > But only when the aware datetimes have different time zones! Oh, fudge - yes. My mistake! I overlooked that in my explanation. And, to be clear, two datetime objects x and y "have the same timezone" in _this_ context means: x.tzinfo is y.tzinfo That is, iff they share a common tzinfo object. In that case they are indeed subtracted (ditto compared) using classic arithmetic Given that classic arithmetic is used whenever it can be partly justified in under 4 pages of dense text, there's simply not a compelling reason to break from the naive-time model in that case. So I take back everything I said about your specific example: > tz = timezone('America/Chicago') > datetime(2013, 11, 3, 1, 30, tzinfo=tz, is_dst=True) - datetime(2013, 11, 3, 1, 0, tzinfo=tz, is_dst=False) Assuming this isn't some tzinfo implementation that, e.g., sometimes magically replaces instances bound to datetimes, then both datetimes share a common tzinfo member, the result won't change. There are other cases that _will_ change, when the tzinfo members are different objects. If you want code similar to this that does give a timeline arithmetic result, you'll need to use whatever spelling of "use timeline arithmetic" is implemented (if any). > It's precisely because of this behavior that pytz has one fixed-offset > DstTzInfo object for each offset attained over a time zone's history, > so that two datetimes in the same time zone but with different offsets > will have different tzinfo members and therefore be converted to UTC > before being subtracted. Both painful and clever. While I don't personally care about timezone transition anomalies (I've never written an app that cared, nor expect I ever will), if I _did_ I think it's bloody obvious that "the right thing" to do in Python's current state is to stick to UTC for every operation apart from conversions for input or output. Classic arithmetic and timeline arithmetic give the same results in UTC (or any other eternally-fixed-offset timezone). Insert "Doctor! Doctor! It hurts when I do this!" "So stop doing it" ;-) > This also means that addition leaves the datetime with the same fixed > offset DstTzInfo object, so that calls to .utcoffset() and .dst() return the > wrong result until the datetime object is normalized, which can be > accomplished either with tz.normalize(dt) or with dt.astimezone(tz) >, except that the second way doesn't work if tz is dt.tzinfo (as opposed > to one of the other DstTzInfo objects corresponding to tz). I won't do the "Doctor! Doctor" joke more than once in a single message ;-) > So anyone who is already using pytz is using timedeltas as > durations anyway, it's just that they and Stuart Bishop have > to jump through hoops to accomplish it. Which, come to think > of it, might mean that the backward compatibility problem is > overrated-- the behavior could be changed for aware datetimes > only, and .normalize() could be added as a no-op. I believe Stuart also strongly recommends sticking to UTC (at least for people who do care about transitions). I conclude he went to all the trouble primarily because he found it to be an irresistibly bizarre technical puzzle. I'm only surprised that attitude is apparently contagious ;-) > ... > I'm not clear on how narrowly you mean "this case" Since I retracted what I said, doesn't matter. > What about > > tz = timezone('America/Chicago') > dt = datetime(2013, 11, 3, tzinfo=tz) > [[(dt.astimezone(timezone.utc) + timedelta(hours=n1)).astimezone(tz) > - dt.astimezone(timezone.utc) + timedelta(hours=n2)).astimezone(tz) for n2 in range(5)] for n1 in range(5)] > > ? I think the project would be better served if you directed this ingenuity and energy into creating unit tests, or at least a persistent document that grows over time, or a wiki entry ... What do _you_ think should happen? I agree that getting edge cases right is important, but since I strongly doubt I'll be working on them it's of little real use to invite me to think about them. At heart, they're just not interesting to me - and mailing lists are black holes. After I finish answering, this will all be lost in time ;-) Briefly as I can, because nobody yet knows what will and won't be implemented, or when each piece that is implemented will land: - Conversions have always done the best job they could in all cases; conversions are _outside_ the "naive time" model. - But conversions inherit utcoffset()'s limitations. Provided utcoffset's current limitations are repaired, conversion should do the correct thing in all cases. It cannot today. - Since you start in Chicago standard time, I expect that (assuming it's implemented) some spelling.of is_dst will record "no, it's not DST" in your dt object. That will never change across the duration of the program. There is no spelling of is_dst today. - The two instances of `dt.astimezone(timezone.utc)` could be hoisted out of the nested loops and computed just once: it's a loop invariant. That value should already be computed correctly (it's converting a not-near-an-edge-case Chicago daylight time to UTC, right? easy). - And there doesn't appear to be anything tricky about the two additions.. Their left-hand sides are the same loop invariant, and then a varying number of hours are added to that invariant. Finally, the additions are performed _in_ UTC, where classic and timeline arithmetic produce the same results. Nothing changes, and all that should also already be done correctly - You've contrived for the additions to create two UTC time spellings that will convert to the same time spelling in America/Chicago (the "repeated hour" at the end of Chicago DST). Both now and regardless of what may change, both such UTC spellings will convert to 1:00:00 Chicago time, via astimezone(tz). - The top-level subtraction will continue to use classic arithmetic because both operands share the same tzindo object. Unless some spelling of "use timeline arithmetic all the time" is implemented that's somehow triggered by code outside the fragment shown. So I don't expect any change to the _displayed_ results here. _If_ the top-level subtraction were done with timeline arithmetic instead, then I would expect changes, provided an .is_dst equivalent is implemented. In that case, some astimezone(tz) intermediate results would change, not in the Chicago HH:MM:SS results delivered, but that some of the Chicago 01:00:00 results would have is_dst False and others True. Classic arithmetic won't even see that, but timeline subtraction would take those differences into account. Enough? Did I even stumble into the issue you were _really_ wondering about? ;-) > ... > ...I might have had some comments of that nature in my code until fairly recently. Though they > weren't as angry as the comments about getting pytz and dateutil.tz to interoperate. Hear about the guy who went to the doctor and complained "Doctor! Doctor! It hurts when I do this!"? Na, me neither ;-) > Or the comments about the device that tried to store sub-second resolution data in > Excel format (days since 1899-12-30)... to five decimal places. I _am_ an expert on floating-point trivia. In case you were wondering, the person who wrote the software for that device was ... optimistic ;-)

1 0

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Ethan Furman 31 Jul '15

31 Jul '15

On 07/30/2015 06:04 PM, Tim Peters wrote: > I like Alexander's new tzstrict (subclass of tzinfo) idea best so far. > Flags invariably create nasty problems, like: It's growing on me, too. The flag idea was per-instance, though, not global (and not changeable once an instance was created), so most of those horrible nightmare situations wouldn't happen. And, yeah, I remember that Guido doesn't like flags. ;) -- ~Ethan~

2 1

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Ethan Furman 31 Jul '15

31 Jul '15

On 07/30/2015 04:50 PM, Alexander Belopolsky wrote: > I think we can change the behavior datetime.__add__ and > datetime.__sub__ if we do it based on the type of the object attached > as tzinfo. Yup, it's really the datetime.__xxx__ methods that need to do something different, so we can base that off a flag to the datetime, or the type of the tzinfo. I suppose the key question is which will make it more obvious that "strict" rules are in affect? -- ~Ethan~

2 1

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Alexander Belopolsky 31 Jul '15

31 Jul '15

On Thu, Jul 30, 2015 at 8:36 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > Speaking of which, the current tzinfo API has no way to ask "is this > an ambiguous time?" I was hoping that we would agree on the name of the flag before someone asks this question. :-) With my proposal, a naive datetime t is ambiguous in timezone tz if tz.utcoffset(t) < tz.utcoffset(t.replace(first=False)) > or "is this an invalid (missing) time?" I was hoping to sneak in a rule that for an invalid time t tz.utcoffset(t) > tz.utcoffset(t.replace(first=False)) (I really don't want tz.utcoffset(t) to ever raise an exception) and of course, for most of the times tz.utcoffset(t) == tz.utcoffset(t.replace(first=False)) > The most > important new question callers will want to resolve is "what should > `first` (aka is_dst) be now?". I want most callers to be able to get away with not knowing that `first` exists and consistently get the earlier time from an ambiguous input and some "nearby" time from an invalid input. A careful application will have to call tz.utcoffset() with both values of the flag and either warn about the default choice or ask the user for an additional input.

1 0

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Tim Peters 31 Jul '15

31 Jul '15

[Alexander Belopolsky <alexander.belopolsky(a)gmail.com>] > I think we can change the behavior datetime.__add__ and > datetime.__sub__ if we do it based on the type of the object attached > as tzinfo. For example, if we introduce a new abstract subclass of > the tzinfo class called say "tzstrict", then we > can be sure that there is no tzinfo implementation in the wild that > inherits from tzstrict (we fully control the datetime module > namespace.) Now, we can make datetime.__add__ and __sub__ check the > type of the tzinfo member on its argument(s) and apply the new rules > if one of them is an instance of tzstrict. It's a happy idea :-) > With these changes in place, users who want new aware datetime > arithmetic will use timezone implementations that inherit from > tzstrict and legacy applications can continue using existing timezone > libraries. And we can spend the next decade arguing what kind of > timezone implementation belongs to the standard library. :-) Or fixing all the bugs in the recommended way to write tzstrict instances ;-) Speaking of which, the current tzinfo API has no way to ask "is this an ambiguous time?" or "is this an invalid (missing) time?" The most important new question callers will want to resolve is "what should `first` (aka is_dst) be now?". The only (at best indirectly) relevant things we can ask it now are "what's the total UTC offset at the local-time instant I represent?", and likewise "what's the DST adjustment component of the total UTC offset at (etc)?:" And there's no way at all now to ask (not even indirectly) whether the standard UTC offset has changed. So it's unclear to me how arithmetic (or anything else) can use the current tzinfo API efficiently to set the `first` (aka is_dst) flag correctly in all cases. Perhaps tzstrict could grow a helpful new method or two to make this easy for callers? The logic is bound to be annoying enough that we'd want to concentrate it in tzstrict (rather then replicate it all over call sites). But no concrete suggestions from me at this time.

1 0

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Alexander Belopolsky 31 Jul '15

31 Jul '15

On Thu, Jul 30, 2015 at 5:25 AM, Guido van Rossum <guido(a)python.org> wrote: > Even if I wouldn't design it that way today (which honestly I haven't > decided whether I would or not) I don't want to change anything about the > observable behavior[1] that would give a different outcome, no matter how > much more "correct" you believe a different outcome would be. The > deprecation time would have be measured in decades. > > The road to improvement that I encourage everyone to explore: add an is_dst > flag (with a different name) and create new timedelta-ish classes that > implement "human" arithmetic (similar to what pytz offers) and "strict" > arithmetic (similar to [NJ]odaTime, IIUC). I think we can change the behavior datetime.__add__ and datetime.__sub__ if we do it based on the type of the object attached as tzinfo. For example, if we introduce a new abstract subclass of the tzinfo class called say "tzstrict", then we can be sure that there is no tzinfo implementation in the wild that inherits from tzstrict (we fully control the datetime module namespace.) Now, we can make datetime.__add__ and __sub__ check the type of the tzinfo member on its argument(s) and apply the new rules if one of them is an instance of tzstrict. With these changes in place, users who want new aware datetime arithmetic will use timezone implementations that inherit from tzstrict and legacy applications can continue using existing timezone libraries. And we can spend the next decade arguing what kind of timezone implementation belongs to the standard library. :-)

1 0

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Ethan Furman 31 Jul '15

31 Jul '15

On 07/30/2015 03:10 PM, Chris Barker wrote: > By the way -- which __add__ actually does the implementation? datetime's or timedelta's ? class timedelta: def __add__(self, other): if isinstance(other, timedelta): # for CPython compatibility, we cannot use # our __class__ here, but need a real timedelta return timedelta(self._days + other._days, self._seconds + other._seconds, self._microseconds + other._microseconds) return NotImplemented timedelta only adds directly to other timedeltas; so datetime is doing the actual addition. -- ~Ethan~

2 1

Re: [Datetime-SIG] naive DateTime, aware DateTime, precise DateTIme
by Ethan Furman 31 Jul '15

31 Jul '15

[redirecting back to list] On 07/30/2015 11:49 AM, Skip Montanaro wrote: > On Thu, Jul 30, 2015 at 1:33 PM, Ethan Furman <ethan(a)stoneleaf.us> wrote: >> Instead of making a new timedelta object, which, after all, represents >> exactly what it says, why don't we make a new DateTime and Time that do the >> duration arithmetic? >> >> Anyone who wants that behaviour has to explicitly use the type so there >> would be no backwards compatibility issues with the existing DateTime, Time, >> or timedelta objects. > > I'm sorry. I must surely have missed something. I freely admit having > mentally elided the datetime-related discussions on python-dev. Under > what circumstances (other than perhaps the grey areas around the > change in or out of daily savings, or intervals crossing leap seconds) > do the current types not perform as expected? Are there test cases > missing from the test suite which would fail using the current > implementation? It is exactly those grey areas that have caused all the hullabaloo, and that's where 24 hours isn't (through no fault of timedelta). Using the example code from the datetime docs [1] as the base and going forward: --> my_time = datetime(2015, 3, 7, 13, 30, tzinfo=Pacific) --> my_time.strftime('%H:%M %z') '13:30 -0800' --> (my_time + 24 * HOUR).strftime('%H:%M %z') '13:30 -0700' The actual duration of time between those two instances is not 24 HOURS, but the blame for that lies with datetime, not timedelta. So my thought is to either make an entirely new type, or have a keyword argument, with the effect of a datetime that does account for dst switches: # HYPOTHETICAL CODE, DOES NOT CURRENTLY WORK --> my_time = datetime(2015, 3, 7, 13, 30, tzinfo=Pacific, precise=True) --> my_time.strftime('%H:%M %z') '13:30 -0800' --> (my_time + 24 * HOUR).strftime('%H:%M %z') '14:30 -0700' # assuming I did the math right in my head ;) I must admit, though, I'm not sure how often such a class would be used -- if the application is important enough to be exact with the durations (nuclear power plants, flying planes, etc.) then I would think a time zone such as UTC would be used just to avoid any possible confusion around those dst switches. -- ~Ethan~ [1] https://docs.python.org/3/library/datetime.html#datetime.tzinfo

2 1

Re: [Datetime-SIG] Clearing up terminology
by ISAAC J SCHWABACHER 31 Jul '15

31 Jul '15

[Tim] > >> ....I personally would have preferred that "aware" datetime objects > >> use strict/timeline arithmetic instead, but it's not a big deal to me either > >> way. > > [ijs] > > I thought I understood what you were saying until now, but through > > this entire discussion I have been under the impression that you felt > > the exact opposite of that last sentence (except for the not being a > > big deal part). > > Ah. but my "preferred" above is past tense. That's how I felt at the > time - and would still feel if we were starting over from scratch. > Throughout the vast bulk of _this_ entire discussion we've been > talking about the present and foreseeable future (in which "starting > over from scratch" doesn't exist). Now backward compatibility, and > explaining how & why it's possible to live happily enough with the > status quo,, are far more important than that one of Tim's mild > preferences was frustrated a dozen years ago ;-) And this is the first time you've mentioned that. :) > >> [explains that strict/timeline arithmetic _is_ currently done when > >> subtracting or comparing two aware datetimes] [ijs] But only when the aware datetimes have different time zones! It's precisely because of this behavior that pytz has one fixed-offset DstTzInfo object for each offset attained over a time zone's history, so that two datetimes in the same time zone but with different offsets will have different tzinfo members and therefore be converted to UTC before being subtracted. This also means that addition leaves the datetime with the same fixed offset DstTzInfo object, so that calls to .utcoffset() and .dst() return the wrong result until the datetime object is normalized, which can be accomplished either with tz.normalize(dt) or with dt.astimezone(tz), except that the second way doesn't work if tz is dt.tzinfo (as opposed to one of the other DstTzInfo objects corresponding to tz). So anyone who is already using pytz is using timedeltas as durations anyway, it's just that they and Stuart Bishop have to jump through hoops to accomplish it. Which, come to think of it, might mean that the backward compatibility problem is overrated-- the behavior could be changed for aware datetimes only, and .normalize() could be added as a no-op. > > So once we have an is_dst bit, what will be the result of > > > > tz = timezone('America/Chicago') > > datetime(2013, 11, 3, 1, 30, tzinfo=tz, is_dst=True) - datetime(2013, 11, 3, 1, 0, tzinfo=tz, is_dst=False) > > > > ? > > I assume you already know, so I too will leave it as an exercise for > the reader ;-) > > > How can it be consistent with current behavior and be anything other > > than timedelta(0, 1800), even though the first time preceded the second? > > The result will change. The intended and documented behavior has > always been to follow "strict arithmetic" rules in this case. So that > it currently doesn't is, and always has been, a bug. Not a bug in > subtraction itself, it's a bug in .utcoffset(), which couldn't > possibly get all cases right before the is_dst flag was added. The > instant utcoffset() is repaired, subtraction and comparison will be > too. I'm not clear on how narrowly you mean "this case". What about tz = timezone('America/Chicago') dt = datetime(2013, 11, 3, tzinfo=tz) [[(dt.astimezone(timezone.utc) + timedelta(hours=n1)).astimezone(tz) - dt.astimezone(timezone.utc) + timedelta(hours=n2)).astimezone(tz) for n2 in range(5)] for n1 in range(5)] ? > Of course it's possible that the change will break somebody's > currently working code. But that's true of all bug fixes. I _might_ > care if anyone could convince me that it has any real chance of > breaking code except for some timezone wonk's collection of edge cases > peppered with > > # EVIL PYTHON!!! HAVE THEY NO SHAME!!!!!??????!!!!!! > > comments ;-) ...I might have had some comments of that nature in my code until fairly recently. Though they weren't as angry as the comments about getting pytz and dateutil.tz to interoperate. Or the comments about the device that tried to store sub-second resolution data in Excel format (days since 1899-12-30)... to five decimal places. ijs

1 0

Re: [Datetime-SIG] Clearing up terminology
by Tim Peters 30 Jul '15

30 Jul '15

[Tim] >> ....I personally would have preferred that "aware" datetime objects >> use strict/timeline arithmetic instead, but it's not a big deal to me either >> way. [[ijs] > I thought I understood what you were saying until now, but through > this entire discussion I have been under the impression that you felt > the exact opposite of that last sentence (except for the not being a > big deal part). Ah. but my "preferred" above is past tense. That's how I felt at the time - and would still feel if we were starting over from scratch. Throughout the vast bulk of _this_ entire discussion we've been talking about the present and foreseeable future (in which "starting over from scratch" doesn't exist). Now backward compatibility, and explaining how & why it's possible to live happily enough with the status quo,, are far more important than that one of Tim's mild preferences was frustrated a dozen years ago ;-) >> [explains that strict/timeline arithmetic _is_ currently done when >> subtracting or comparing two aware datetimes] [[ijs] > So once we have an is_dst bit, what will be the result of > > tz = timezone('America/Chicago') > datetime(2013, 11, 3, 1, 30, tzinfo=tz, is_dst=True) - datetime(2013, 11, 3, 1, 0, tzinfo=tz, is_dst=False) > > ? I assume you already know, so I too will leave it as an exercise for the reader ;-) > How can it be consistent with current behavior and be anything other > than timedelta(0, 1800), even though the first time preceded the second? The result will change. The intended and documented behavior has always been to follow "strict arithmetic" rules in this case. So that it currently doesn't is, and always has been, a bug. Not a bug in subtraction itself, it's a bug in .utcoffset(), which couldn't possibly get all cases right before the is_dst flag was added. The instant utcoffset() is repaired, subtraction and comparison will be too. Of course it's possible that the change will break somebody's currently working code. But that's true of all bug fixes. I _might_ care if anyone could convince me that it has any real chance of breaking code except for some timezone wonk's collection of edge cases peppered with # EVIL PYTHON!!! HAVE THEY NO SHAME!!!!!??????!!!!!! comments ;-)

1 0

2022

2021

2020

2019

2018

2017

2016

2015

Datetime-SIG July 2015