Mailman 3 August 2015 - Datetime-SIG

Re: [Datetime-SIG] Another round on error-checking
by Alexander Belopolsky 31 Aug '15

31 Aug '15

On Mon, Aug 31, 2015 at 3:33 PM, Alexander Belopolsky < alexander.belopolsky(a)gmail.com> wrote: > On Mon, Aug 31, 2015 at 3:20 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > >> def __hash__(self): >> if self._hashcode == -1: >> tzoff = self.utcoffset() >> if tzoff is None: >> self._hashcode = >> hash(self.replace(first=True)._getstate()[0]) >> else: >> days = _ymd2ord(self.year, self.month, self.day) >> seconds = self.hour * 3600 + self.minute * 60 + >> self.second >> self._hashcode = hash(timedelta(days, seconds, >> self.microsecond) - tzoff) >> return self._hashcode >> >> So it's the case that two datetimes that compare true may have >> different hashes, when they represent the earlier and later times in a >> fold. I didn't say "it's a puzzle" lightly ;-) >> > > Yes, it looks like I have a bug there, but isn't fixing it just a matter > of moving self.replace(first=True) up two lines? Is there a bigger > puzzle? Certainly x == y ⇒ hash(x) == hash(y) is the implication that I > intend to preserve in all cases. I think I admitted defeat too soon. Can you present a specific case where "two datetimes that compare true have different hashes"? There may be some subtlety due to the fact that we ignore tzinfo in == if it is the same for both sides, but when we compute hash(), we don't know what's on the other side. It is hard to tell without a specific example. I thought I got it right when I wrote the code above, but it is possible I missed some case.

1 0

Re: [Datetime-SIG] Another round on error-checking
by Tim Peters 31 Aug '15

31 Aug '15

[Tim] >> At a high level, I'm questioning the "_never_ raise an exception" PEP >> 495 behavior. It grates. "Errors should never pass silently" and >> such. [Alex][ > But these are not errors! That depends on the application. As W Kahan quipped when devising the IEEE-754 exception model, they're called "exceptions" because _whatever_ you do someone will take exception to it ;-) > As I mentioned before, it was bad PR on my part > to call datetimes in the gaps or with ignorable fold=1 "invalid." I should > have called them "denormalized." It doesn't matter what they're called _except_ for "PR purposes". I'm not trying to do PR here. > I believe many aware datetime manipulation algorithms can benefit from > having denormalized instances as intermediate values and being able > to call .utcoffset() and friends on such instances. That's why, e.g., C's mktime allows "insane" values for the day, etc. Python's internal C datetime implementation does too. It's also why Paul Eggert patched glibc's mktime to "pick one" for a gap case.. But that's all C, working at a level so low everything is painful. datetime isn't intended to be infinitely painful ;-) > My primary use case is the "naive scheduler" which gives you no means to > schedule anything with fold=1 and if you give it 02:45 AM in the gap it will > silently take it for 03:45 AM. As long as it displays the correct time in > every reminder, I don't care that it did to chastise me for not knowing > about the DST gap. I don't dispute that, in a gap case, picking the time that "would have displayed" had the user moved the clock forward is most useful most often. If that's believed to be overwhelmingly the case, that at least argues for opt-in exceptions. But passing fold=1 in a non-fold/gap case is almost certainly a flat-out programmer error, _except_ that PEP 495 requires never ever complaining about it, and to the contrary _requires_ it to be done if the user wants to find out whether their datetime is any way odd. C'mon, admit it - _part_ of it makes you cringe too, if even just a little ;-)

1 0

Re: [Datetime-SIG] Another round on error-checking
by Alexander Belopolsky 31 Aug '15

31 Aug '15

On Mon, Aug 31, 2015 at 3:20 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > def __hash__(self): > if self._hashcode == -1: > tzoff = self.utcoffset() > if tzoff is None: > self._hashcode = > hash(self.replace(first=True)._getstate()[0]) > else: > days = _ymd2ord(self.year, self.month, self.day) > seconds = self.hour * 3600 + self.minute * 60 + self.second > self._hashcode = hash(timedelta(days, seconds, > self.microsecond) - tzoff) > return self._hashcode > > So it's the case that two datetimes that compare true may have > different hashes, when they represent the earlier and later times in a > fold. I didn't say "it's a puzzle" lightly ;-) > Yes, it looks like I have a bug there, but isn't fixing it just a matter of moving self.replace(first=True) up two lines? Is there a bigger puzzle? Certainly x == y ⇒ hash(x) == hash(y) is the implication that I intend to preserve in all cases.

1 0

Re: [Datetime-SIG] Another round on error-checking
by Alexander Belopolsky 31 Aug '15

31 Aug '15

On Mon, Aug 31, 2015 at 3:07 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > At a high level, I'm questioning the "_never_ raise an exception" PEP > 495 behavior. It grates. "Errors should never pass silently" and > such. > But these are not errors! As I mentioned before, it was bad PR on my part to call datetimes in the gaps or with ignorable fold=1 "invalid." I should have called them "denormalized." I believe many aware datetime manipulation algorithms can benefit from having denormalized instances as intermediate values and being able to call .utcoffset() and friends on such instances. My primary use case is the "naive scheduler" which gives you no means to schedule anything with fold=1 and if you give it 02:45 AM in the gap it will silently take it for 03:45 AM. As long as it displays the correct time in every reminder, I don't care that it did to chastise me for not knowing about the DST gap.

1 0

Re: [Datetime-SIG] Another round on error-checking
by Tim Peters 31 Aug '15

31 Aug '15

[Tim] >> BTW, it just occurred to me that PEP 495 has already broken >> datetime.__hash__. ... [Alex] > The PEP did not and in the reference implementation, I am careful to reset > fold/first before computing the hash: > > https://github.com/abalkin/cpython/blob/issue24773/Lib/datetime.py#L1178 But you're pointing to time.__hash__ there. I'm talking about datetime.__hash__. You replace `first` there too, but _only_ if .utcoffset() returns None: def __hash__(self): if self._hashcode == -1: tzoff = self.utcoffset() if tzoff is None: self._hashcode = hash(self.replace(first=True)._getstate()[0]) else: days = _ymd2ord(self.year, self.month, self.day) seconds = self.hour * 3600 + self.minute * 60 + self.second self._hashcode = hash(timedelta(days, seconds, self.microsecond) - tzoff) return self._hashcode So it's the case that two datetimes that compare true may have different hashes, when they represent the earlier and later times in a fold. I didn't say "it's a puzzle" lightly ;-)

1 0

Re: [Datetime-SIG] Another round on error-checking
by Alexander Belopolsky 31 Aug '15

31 Aug '15

On Mon, Aug 31, 2015 at 2:53 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > BTW, it just occurred to me that PEP 495 has already broken > datetime.__hash__. ... > The PEP did not and in the reference implementation, I am careful to reset fold/first before computing the hash: https://github.com/abalkin/cpython/blob/issue24773/Lib/datetime.py#L1178

1 0

Re: [Datetime-SIG] Another round on error-checking
by Carl Meyer 31 Aug '15

31 Aug '15

On 08/31/2015 12:58 PM, Alexander Belopolsky wrote: > On Mon, Aug 31, 2015 at 2:38 PM, Carl Meyer <carl(a)oddbird.net > <mailto:carl@oddbird.net>> wrote: > [Tim Peters] > > > Nope. There's nothing here about, e.g., messing with datetime > > constructors, .replace(), .combine() ... "naive time" is left alone > > here. It's only timezone-specific operations targeted here, which are > > all implemented _by_ tzinfo objects. Not by datetime itself. > > There wasn't any of that stuff (messing with constructors, or replace, > or combine, or naive time) in what Alex and I were discussing in the > other thread, either. Just the idea of having `utcoffset()` raise an > error if it hit an ambiguity. > > > I think the main difference between Tim's current proposal and what was > previously discussed is that all older proposals somehow required a > third value for fold. Yes, that's true. That's because (unless I'm misunderstanding) Tim is suggesting something far more audacious than I had considered: making "raise an error on ambiguity" the default behavior, instead of an opt-in choice. The extra value for `fold` was just the opt-in mechanism. Carl

2 1

Re: [Datetime-SIG] Another round on error-checking
by Carl Meyer 31 Aug '15

31 Aug '15

On 08/31/2015 12:34 PM, Tim Peters wrote: > [Carl Meyer <carl(a)oddbird.net>] >> That's pretty much what I proposed in the first invalid-time-checking >> thread. Alex didn't like it because `utcoffset()` is called from so many >> different places: >> https://mail.python.org/pipermail/datetime-sig/2015-August/000499.html > > That is a potential problem. > >> AFAICT, you are re-proposing the same solution you characterized several >> times earlier as "spraying errors all over the place" and "going nowhere >> fast." :-) > > Nope. There's nothing here about, e.g., messing with datetime > constructors, .replace(), .combine() ... "naive time" is left alone > here. It's only timezone-specific operations targeted here, which are > all implemented _by_ tzinfo objects. Not by datetime itself. There wasn't any of that stuff (messing with constructors, or replace, or combine, or naive time) in what Alex and I were discussing in the other thread, either. Just the idea of having `utcoffset()` raise an error if it hit an ambiguity. Carl

3 2

Re: [Datetime-SIG] Another round on error-checking
by Tim Peters 31 Aug '15

31 Aug '15

[Tim] >> PEP 495 specifies resolving such cases by magic, in essentially >> arbitrary (from the user's point of view) ways. This isn't for >> backward compatibility, because 495-compliant tzinfos don't currently >> exist [Alex] > .. but tzinfo.fromutc() [1] does exist I know - I wrote it ;-) > and the first thing happening in that code is an unguarded call > to utcoffset() on a UTC datetime with a transplanted tzinfo. [2] A > trick like this have been valid in datetime since its introduction > and has been used in numerous places since. But "continuing to never raise an exception" does not imply "continuing to work as intended". All such code needs to be carefully audited in a PEP-495 world to ensure that the problem-case return values work as intended with the old code and the new PEP 495 behavior _if_ a 495-compliant tzinfo object is used. > You know that with a little effort universal .fromutc() can be rewritten to be > PEP 495 compliant. > > If you think allowing datetime.utcoffset() to raise an exception is an > option, let's first see how one can rewrite universal tzinfo.fromutc() for > that design. Even though I don't think reusing a universal .fromutc() is a > good option for tzinfo implementers, it gives an example of simple but > non-trivial datetime manipulation code which PEP 495 is designed not to > break. Eh,. _If_ .utcoffet() could raise exceptions, then the obvious way would be to catch the exceptions and, in each exceptional case, implement whichever behavior is most appropriate in context. That could be used to implement any desired behavior whatsoever. > [1]: https://hg.python.org/cpython/file/v3.5.0rc2/Lib/datetime.py#l957 > [2]: https://hg.python.org/cpython/file/v3.5.0rc2/Lib/datetime.py#l965 BTW, it just occurred to me that PEP 495 has already broken datetime.__hash__. That is, the fundamental invariant a hash implementation must satisfy is: A == B implies hash(A) == hash(B) But, under 495, equality of datetimes ignores `fold`, but .utcoffset() does not, and def __hash__(self): tzoff = self.utcoffset() if tzoff is None: ... else: ... self._hashcode = hash(timedelta(days, seconds, self.microsecond) - tzoff) So two datetimes in a fold, one with fold=0 and the other with fold=1, will compare equal but almost certainly have different hashes. That's a real puzzle. But the kind of error-checking I'm discussing doesn't make that _worse_ ;-) WRT the current __hash__ problem, perhaps it's enough for the PEP to note that ... in some number of cases including hash(), `fold` does a make difference ;-)

1 0

Re: [Datetime-SIG] Another round on error-checking
by Carl Meyer 31 Aug '15

31 Aug '15

On 08/31/2015 11:58 AM, Tim Peters wrote: > I think I'd rather acknowledge that problem cases exist in a direct > and straightforward way, by adding a new tzinfo (say).classify() > method. For example, .classify() could return a > > (kind, detail) > > 2-tuple. FWIW, regardless of the question of `utcoffset()` raising exceptions, the addition of this `classify()` method alone would resolve every use case I've ever had for pytz's `is_dst=None` strict ambiguity handling. Carl

1 0