Mailman 3 August 2015 - Datetime-SIG

Re: [Datetime-SIG] Another round on error-checking
by Tim Peters Sept. 1, 2015

Sept. 1, 2015

[Alex] >> After some thought, I believe the way to fix the implementation is what I >> suggested at first: reset fold to 0 before calling utcoffset() in __hash__. >> A rare hash collision is a small price to pay for having datetimes with >> different timezones in the same dictionary. [Tim] > Ya, I can live with that. In effect, we give up on converting to UTC > correctly for purposes of computing hash(), but only in rare cases. > hash() doesn't really care, and it remains true that datetime equality > (which does care) still implies hash equality. The later and earlier > of ambiguous times will simply land on the same hash chain. Nope, you wore me out prematurely ;-) Consider datetimes dt1 and dt2 representing the earlier & later of an ambiguous time in their common zone (whatever it may be - doesn't matter). Then all fields are identical except for `fold`. Assume __hash__ forces `fold` to 0 before obtaining the UTC offset. Then we have: dt1 == dt2 hash(dt1) == hash(dt2) Fine so far as it goes. Now do: u1 = dt1.astimezone(timezone.utc) u2 = dt2.astimezone(timezone.utc) At this point we have: u1 == dt1 == dt2 == u2 and u1 < u2 hash(dt1) == hash(dt2) == hash(u1) (Parenthetically, note that despite the chain of equalities in the first of those lines, we do _not_ have u1 == u2 - transitivity fails, which is a bit of a wart by itself.) Since u1 == dt1, and hash(u1) == hash(dt1), no problem there either. But u1 isn't at all the same as u2, so hash(u2) can be the same as hash(u1) only by (unlikely) accident. hash(u2) is off in a world of its own. Therefore hash(dt2) can be the same as hash(u2) only by (the same unlikely) accident, despite that dt2 == u2. So, in all, __hash__ forcing fold=0 at the start hides the problem for ambiguous times in the same zone, but doesn't really touch the problem for cross-zone equivalent spellings of such times (not even if one of the zones is UTC, which is likely the most important case). One way to fix that is to have datetime.__hash__() _always_ return, say, 0 ;-)

1 0

Re: [Datetime-SIG] Another round on error-checking
by Alexander Belopolsky Aug. 31, 2015

Aug. 31, 2015

On Mon, Aug 31, 2015 at 5:15 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > > Now, please, can we not start discussing how __hash__ should behave if > > utcoffset() raises a MissingTimeError? > > __hash__ is a tail. I still want to get people thinking about the > dog. Really. Let's give people some space to look at it from a > higher level? Implementation details aren't users' problems, and I > want to be (more) sure we can live with the high-level model. This forum may not be inclusive enough for this. People in this group know too much! I plan to cross some t's and dot some i's in the PEP text this week and complete the "big rename" in the reference implementation. After that, I will be ready to present the PEP for a final round on Python-Dev. Granted, Python-Dev is also a rather elite group, but I hope we will find a few people there who have used time.mktime(dt.timetuple()) in their programs and never thought there was anything wrong with their code.

1 0

Re: [Datetime-SIG] Another round on error-checking
by Alexander Belopolsky Aug. 31, 2015

Aug. 31, 2015

On Mon, Aug 31, 2015 at 2:53 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > But "continuing to never raise an exception" does not imply > "continuing to work as intended". All such code needs to be carefully > audited in a PEP-495 world to ensure that the problem-case return > values work as intended with the old code and the new PEP 495 behavior > _if_ a 495-compliant tzinfo object is used. > My design goals with respect to aware datetime objects were (1) objects with pre-PEP tzinfo should work exactly the same as they did before (2) objects with post-PEP tzinfo may produce different results when used with pre-PEP code, but these differences should be limited to the problem cases (gaps and folds) where new behavior can be defended as a bug fix. I find it unacceptable for a program that switched to post-PEP tzinfo to crash several years after that because it was not sufficiently well tested on problem times. In most applications, confusing the first and the seconds 01:30 AM is a forgivable error. Applications that cannot tolerate it should not use local times or should be carefully audited for PEP 495 compliance. However, even applications that can tolerate a random choice between one 01:30AM and another usually cannot tolerate a server crash at 02:45 AM.

1 0

Re: [Datetime-SIG] Another round on error-checking
by Tim Peters Aug. 31, 2015

Aug. 31, 2015

[Alex] > After some thought, I believe the way to fix the implementation is what I > suggested at first: reset fold to 0 before calling utcoffset() in __hash__. > A rare hash collision is a small price to pay for having datetimes with > different timezones in the same dictionary. Ya, I can live with that. In effect, we give up on converting to UTC correctly for purposes of computing hash(), but only in rare cases. hash() doesn't really care, and it remains true that datetime equality (which does care) still implies hash equality. The later and earlier of ambiguous times will simply land on the same hash chain. > Now, please, can we not start discussing how __hash__ should behave if > utcoffset() raises a MissingTimeError? __hash__ is a tail. I still want to get people thinking about the dog. Really. Let's give people some space to look at it from a higher level? Implementation details aren't users' problems, and I want to be (more) sure we can live with the high-level model.

1 0

Re: [Datetime-SIG] Another round on error-checking
by Alexander Belopolsky Aug. 31, 2015

Aug. 31, 2015

On Mon, Aug 31, 2015 at 4:38 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > [Tim] > >> The easiest way out of this particular puzzle is, I believe, to say > >> that two datetimes identical except for `fold` do _not_ compare equal. > >> `fold` breaks the tie in the obvious way (the one with fold==1 is > >> "greater"). > > [Alex] > > I am afraid you are right, but proving that we will not break naive (fold > > unaware) programs will be harder in this case. Let me think some more > > about this. > > > > Meanwhile, would you see any problem with not(x - y) not implying x == y? > > Which is another puzzle :-( It's very intentional now that > > dt1 == dt2 if and only if dt1 - dt2 == timedelta(0) > > Here's a related puzzle, if comparison used `fold` to break ties: > > y = x + timedelta(0) > > If x had first=1, y will have fold=0, and then x != y. > > In all, maybe it's better to leave __hash__ slightly broken. > After some thought, I believe the way to fix the implementation is what I suggested at first: reset fold to 0 before calling utcoffset() in __hash__. A rare hash collision is a small price to pay for having datetimes with different timezones in the same dictionary. Now, please, can we not start discussing how __hash__ should behave if utcoffset() raises a MissingTimeError?

1 0

Re: [Datetime-SIG] Another round on error-checking
by Tim Peters Aug. 31, 2015

Aug. 31, 2015

[Tim] >> The easiest way out of this particular puzzle is, I believe, to say >> that two datetimes identical except for `fold` do _not_ compare equal. >> `fold` breaks the tie in the obvious way (the one with fold==1 is >> "greater"). [Alex] > I am afraid you are right, but proving that we will not break naive (fold > unaware) programs will be harder in this case. Let me think some more > about this. > > Meanwhile, would you see any problem with not(x - y) not implying x == y? Which is another puzzle :-( It's very intentional now that dt1 == dt2 if and only if dt1 - dt2 == timedelta(0) Here's a related puzzle, if comparison used `fold` to break ties: y = x + timedelta(0) If x had first=1, y will have fold=0, and then x != y. In all, maybe it's better to leave __hash__ slightly broken.

1 0

Re: [Datetime-SIG] Another round on error-checking
by Alexander Belopolsky Aug. 31, 2015

Aug. 31, 2015

On Mon, Aug 31, 2015 at 4:17 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > The easiest way out of this particular puzzle is, I believe, to say > that two datetimes identical except for `fold` do _not_ compare equal. > `fold` breaks the tie in the obvious way (the one with fold==1 is > "greater"). > I am afraid you are right, but proving that we will not break naive (fold unaware) programs will be harder in this case. Let me think some more about this. Meanwhile, would you see any problem with not(x - y) not implying x == y?

1 0

Re: [Datetime-SIG] Another round on error-checking
by Tim Peters Aug. 31, 2015

Aug. 31, 2015

>> def __hash__(self): >> if self._hashcode == -1: >> tzoff = self.utcoffset() >> if tzoff is None: >> self._hashcode = >> hash(self.replace(first=True)._getstate()[0]) >> else: >> days = _ymd2ord(self.year, self.month, self.day) >> seconds = self.hour * 3600 + self.minute * 60 + >> self.second >> self._hashcode = hash(timedelta(days, seconds, >> self.microsecond) - tzoff) >> return self._hashcode >> >> So it's the case that two datetimes that compare true may have >> different hashes, when they represent the earlier and later times in a >> fold. I didn't say "it's a puzzle" lightly ;-) [Alex] > Yes, it looks like I have a bug there, but isn't fixing it just a matter of > moving self.replace(first=True) up two lines? Is there a bigger puzzle? > Certainly x == y ⇒ hash(x) == hash(y) is the implication that I intend to > preserve in all cases. Yes, there's a bigger puzzle: datetimes expressed in different timezones can also compare equal. Conceptually, they're converted to UTC before comparison - and so also, to maintain the crucial hash invariant, before being hashed. That can't work right without using their actual UTC offsets (i.e,, `first` can't be ignored for interzone equality, but would be ignored for hashes if forcing `first` to 1 were done before extracting the offset). The real problem here is that this stuff just barely managed to work from the start ;-) In effect, for the purpose of hashing, _all_ datetimes are converted to UTC first now. That didn't interfere with the "naive time" view before because all possible insanities were blithely ignored in all contexts before. The easiest way out of this particular puzzle is, I believe, to say that two datetimes identical except for `fold` do _not_ compare equal. `fold` breaks the tie in the obvious way (the one with fold==1 is "greater"). Then __hash__ can continue using the real UTC offset, just as before. If a user doesn't force `fold` to 1, then no existing code will change behavior, at least until they start using 495 tzinfos. Then `fold=1` can start appearing "by magic" via .fromutc().

1 0

Re: [Datetime-SIG] Another round on error-checking
by Alexander Belopolsky Aug. 31, 2015

Aug. 31, 2015

On Mon, Aug 31, 2015 at 3:50 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > But passing fold=1 in a > non-fold/gap case is almost certainly a flat-out programmer error, > _except_ that PEP 495 requires never ever complaining about it, and to > the contrary _requires_ it to be done if the user wants to find out > whether their datetime is any way odd. > A programmer who knows about fold does not need as much hand-holding as the one who does not. My customer is the ignorant programmer. > C'mon, admit it - _part_ of it makes you cringe too, if even just a little > ;-) > What makes me cringe is the problem domain where I am required to implement an inverse to a non-monotonic function. And that's on top of the world where clock hands can move backwards.

1 0

Re: [Datetime-SIG] Another round on error-checking
by Tim Peters Aug. 31, 2015

Aug. 31, 2015

>>> def __hash__(self): >>> if self._hashcode == -1: >>> tzoff = self.utcoffset() >>> if tzoff is None: >>> self._hashcode = >>> hash(self.replace(first=True)._getstate()[0]) >>> else: >>> days = _ymd2ord(self.year, self.month, self.day) >>> seconds = self.hour * 3600 + self.minute * 60 + >>> self.second >>> self._hashcode = hash(timedelta(days, seconds, >>> self.microsecond) - tzoff) >>> return self._hashcode > ... > I think I admitted defeat too soon. Can you present a specific case where > "two datetimes that compare true have different hashes"? Two aware datetimes in a single zone representing the earlier and later ambiguous time. All fields (including tzinfo) are identical except for fold. "==" says True. But `tzoff` differs between them, so the code above passes different values to `hash()`. It's not guaranteed that the hashes differ, but it's very likely they differ. > There may be some subtlety due to the fact that we ignore tzinfo in == > if it is the same for both sides, That's why they compare equal in this case. > but when we compute hash(), we don't know what's on the other > side. It is hard to tell without a specific example. I thought I got it > right when I wrote the code above, but it is possible I missed some case. It's a puzzle ;-)

1 0