Mailman 3 August 2015 - Datetime-SIG

Re: [Datetime-SIG] Another round on error-checking
by Carl Meyer 31 Aug '15

31 Aug '15

Hi Tim, On 08/31/2015 11:58 AM, Tim Peters wrote: > I've been playing with what it would take to wrap zoneinfo efficiently > in a post-495 world. When I got to .utcoffset(), I just cringed when > trying to implement the "in the face of ambiguity and/or > impossibility, make stuff up ;-) ", parts. > > The pytz folks have been enthusiastic about pytz's approach. Alas, > it's a poor fit to datetime's design, because pytz strives to make it > appear that "naive time" doesn't exist at all for datetimes with a > tzinfo. But in the design, they do. Regardless of whether a tzinfo > is present, a datetime is intended to be viewed as working in naive > time. "Missing" and "ambiguous" times plain don't exist in naive > time, so it's unnatural to check for them all over the place. > > It's when a timezone-*specific* operation is attempted that the user > is explicitly moving out of naive time (not merely when a tzinfo is > attached). So, in my view, *that's* when to check. .utcoffset() is > the primary such place (whether called directly or indirectly). That's pretty much what I proposed in the first invalid-time-checking thread. Alex didn't like it because `utcoffset()` is called from so many different places: https://mail.python.org/pipermail/datetime-sig/2015-August/000499.html AFAICT, you are re-proposing the same solution you characterized several times earlier as "spraying errors all over the place" and "going nowhere fast." :-) Carl

2 1

Re: [Datetime-SIG] Another round on error-checking
by Alexander Belopolsky 31 Aug '15

31 Aug '15

On Mon, Aug 31, 2015 at 1:58 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > I think I'd rather acknowledge that problem cases exist in a direct > and straightforward way, by adding a new tzinfo (say).classify() > method. For example, .classify() could return a > > (kind, detail) > > 2-tuple. > > - kind==DTKIND_NORMAL. > Not an exceptional case. > detail is None. > > - kind==DTKIND_FOLD_NORMAL. > The datetime is in a fold, and its `fold` value is sane. > detail is the datetime's `fold` value (0 or 1). > > - kind==DTKIND_FOLD_INVALID. > The datetime does not have `fold==0`, but the datetime is not in a fold. > detail is the datetime's `fold` value (whatever it may be). > > - kind==DTKIND_GAP. > The datetime is in a gap. > detail is a (d1, d2) 2-tuple, where `d1` and `d2` are > timedeltas such that (in classic arithmetic): > datetime - d1 is the closest earlier non-gap time > datetime + d2 is the closest later non-gap time > > Users can call that directly when they like. > I have no objection to this method as long as a default implementation in terms of a PEP 495 compliant .utcoffset() is provided in the basic datetime.tzinfo. Not a bad idea for another PEP. :-)

1 0

Re: [Datetime-SIG] Another round on error-checking
by Alexander Belopolsky 31 Aug '15

31 Aug '15

On Mon, Aug 31, 2015 at 1:58 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > PEP 495 specifies resolving such cases by magic, in essentially > arbitrary (from the user's point of view) ways. This isn't for > backward compatibility, because 495-compliant tzinfos don't currently > exist > .. but tzinfo.fromutc() [1] does exist and the first thing happening in that code is an unguarded call to utcoffset() on a UTC datetime with a transplanted tzinfo. [2] A trick like this have been valid in datetime since its introduction and has been used in numerous places since. You know that with a little effort universal .fromutc() can be rewritten to be PEP 495 compliant. If you think allowing datetime.utcoffset() to raise an exception is an option, let's first see how one can rewrite universal tzinfo.fromutc() for that design. Even though I don't think reusing a universal .fromutc() is a good option for tzinfo implementers, it gives an example of simple but non-trivial datetime manipulation code which PEP 495 is designed not to break. [1]: https://hg.python.org/cpython/file/v3.5.0rc2/Lib/datetime.py#l957 [2]: https://hg.python.org/cpython/file/v3.5.0rc2/Lib/datetime.py#l965

1 0

Another round on error-checking
by Tim Peters 31 Aug '15

31 Aug '15

I've been playing with what it would take to wrap zoneinfo efficiently in a post-495 world. When I got to .utcoffset(), I just cringed when trying to implement the "in the face of ambiguity and/or impossibility, make stuff up ;-) ", parts. The pytz folks have been enthusiastic about pytz's approach. Alas, it's a poor fit to datetime's design, because pytz strives to make it appear that "naive time" doesn't exist at all for datetimes with a tzinfo. But in the design, they do. Regardless of whether a tzinfo is present, a datetime is intended to be viewed as working in naive time. "Missing" and "ambiguous" times plain don't exist in naive time, so it's unnatural to check for them all over the place. It's when a timezone-*specific* operation is attempted that the user is explicitly moving out of naive time (not merely when a tzinfo is attached). So, in my view, *that's* when to check. .utcoffset() is the primary such place (whether called directly or indirectly). At that point, two kinds of "meaningless" times pop into existence: 1. fold != 0 when the datetime isn't actually in a fold. 2. The datetime is in a gap. There is no UTC time that maps back to such cases, so there is no possible timedelta .utcoffset() can return that's wholly justifiable. PEP 495 specifies resolving such cases by magic, in essentially arbitrary (from the user's point of view) ways. This isn't for backward compatibility, because 495-compliant tzinfos don't currently exist(*). It's more that 495 gives users no other way to determine whether a datetime _is_ "a problem case" other than by calling .utcoffset() twice with different values for `fold`, and then making .utcoffset() return carefully chosen (but arbitrary from the user's POV) problem-case results sufficient to classify the datetime from the two .utcoffset() results. I think I'd rather acknowledge that problem cases exist in a direct and straightforward way, by adding a new tzinfo (say).classify() method. For example, .classify() could return a (kind, detail) 2-tuple. - kind==DTKIND_NORMAL. Not an exceptional case. detail is None. - kind==DTKIND_FOLD_NORMAL. The datetime is in a fold, and its `fold` value is sane. detail is the datetime's `fold` value (0 or 1). - kind==DTKIND_FOLD_INVALID. The datetime does not have `fold==0`, but the datetime is not in a fold. detail is the datetime's `fold` value (whatever it may be). - kind==DTKIND_GAP. The datetime is in a gap. detail is a (d1, d2) 2-tuple, where `d1` and `d2` are timedeltas such that (in classic arithmetic): datetime - d1 is the closest earlier non-gap time datetime + d2 is the closest later non-gap time Users can call that directly when they like. .utcoffset() (and other appropriate timezone-specific methods) would raise exceptions in the DTKIND_FOLD_INVALID and DTKIND_GAP cases, with the same exception detail as `classify()` returns. This would, of course, require major rewriting of the PEP. So Alex will hate it ;-) But, leaving aside how much design pain it would cause, is it "the right" (or "a righter") thing to do? That's what I'm more concerned about. In any case, since this is _a_ view of error checking that hasn't been mentioned at all before, it's worth putting it out in public. BTW, I don't expect pytz to like it. In Python's datetime design, timeline arithmetic should be done in UTC (or via timestamps) instead. The scheme above intends to catch errors _when_ converting to UTC, leaving naive time alone until (if ever) the user does explicitly invoke a timezone operation. (*) WRT backward compatibility, there are other non-obvious cases after 495 tzinfos do exist. LIke datetime.__hash__() calling .utcoffset(). It would be desirable that people living in naive time (despite attaching tzinfos) not need to worry about exceptions in cases like that when using a 495 tzinfo. In the kind of scheme above, one way around that is changing __hash__ (which could resolve problem cases in any way that works best for _its_ purposes). Another way is adding optional `check` Boolean arguments to various methods, defaulting to False, in which case the current 495 "make stuff up" results would be returned. But I'm trying to take a higher-level view of "what's right" in _this_ msg ;-)

1 0

Re: [Datetime-SIG] A local timezone class
by Akira Li 31 Aug '15

31 Aug '15

Guido van Rossum <guido(a)python.org> writes: > I've written a simple class that implements a local timezone tzinfo object, > deferring to what the time module exposes about the local zone. > > https://gist.github.com/gvanrossum/ef201fe313719305c4c7 > > There are two variations: one for systems that support tm_gmtoff and > tm_zone, one for systems without those. > > Output of the test program: > > BetterLocalTimeZone > Fri Aug 28 15:50:22 2015 PDT (-0700) > Wed Feb 24 15:50:22 2016 PST (-0800) > > I'm not entirely sure why we didn't add this to the stdlib ages ago. (Maybe > out of a sense of perfectionism, since time.localtime() may be wrong for > dates in the past or future where different DST rules or a different > standard offset apply? But why would we care, if we're fine with the time > module's behavior?) If tm_gmtoff, tm_zone attributes are not available then time module does not provide a way to get correct tzname [1]. In particular, datetime.astimezone() may fail to return a correct tzname due to that [2] [1] http://bugs.python.org/issue22798 [2] https://mail.python.org/pipermail/datetime-sig/2015-August/000471.html

2 1

Re: [Datetime-SIG] PEP-431/495
by Akira Li 31 Aug '15

31 Aug '15

Tim Peters <tim.peters(a)gmail.com> writes: > [Akira Li <4kir4.1i(a)gmail.com>] > ... >> then arithmetic "as though via conversion to utc time" is *equally valid* as >> the arithmetic "as though it is a timezone-naive datetime object".aa > > __add__ can only mean one of them. And it already does. It _could_ > have meant the other, but it doesn't. > In fact, it does mean the other: https://mail.python.org/pipermail/datetime-sig/2015-August/000545.html stdlib is consistent with pytz (utc arithmetic). dateutil (naive) produces different results. ... >> Timezones is a very complicated >> topic -- no solution works in the general case.a > > ? Timezone transitions are mathematically trivial. They're just full > of lumps (irregularities). > Irregularity is another word for complexity. Timezone rules are defined by politicians. Even if they appear to be simpletons; it is still hard to create a math equation that will predict the rules. Instead, the *database* approach with regular updates works well in practice.

1 0

Re: [Datetime-SIG] Trivial vs easy: .utcoffset()
by Tim Peters 30 Aug '15

30 Aug '15

[Tim] > Easy idea: after loading a tzfile, create and store (on the tzinfo > instance) a list of every unique total UTC offset in the zone's > history, ordered by most recent to least. .utcoffset() then only > needs to march through that list once, seeing whether the input minus > the current offset (from the list) converts back to the input via > .fromutc(). If none do, that's an internal error. Not quite. "Converts back" has to include reproducing the `fold` value too. If a user gives a datetime with fold=1 where it doesn't make sense,.fromutc() can't reproduce it. Likewise if a user gives a datetime in a gap. With enough extra annoying code, useful error messages for those could be produced. I wouldn't want to _guess_ the intent in either of those cases. It's an internal error if neither of those apply.

1 0

Re: [Datetime-SIG] Trivial vs easy: .utcoffset()
by Tim Peters 30 Aug '15

30 Aug '15

[Tim[ > I haven't stared at mktime() in anger. Is there anything it could be > _told_ about the local zone that could ease its job? ... Easy idea: after loading a tzfile, create and store (on the tzinfo instance) a list of every unique total UTC offset in the zone's history, ordered by most recent to least. .utcoffset() then only needs to march through that list once, seeing whether the input minus the current offset (from the list) converts back to the input via .fromutc(). If none do, that's an internal error. That should usually get out in one or two tries (the zone's most recent "standard" and "daylight" total offsets, in some order, will usually be tried first). For historical dates, who cares - they're only going to appear in test cases anyway, and even then can't require .more .fromutc() calls than there are unique UTC offsets in the zone's history. For zones entirely defined by a POSIX TZ rule, the list would contain at most two entries. Nailed it ;-) Pickles are always a puzzle. The list of unique offsets may well change in the future, so it should really be recomputed from scratch when unpickling.

1 0

Re: [Datetime-SIG] Trivial vs easy: .utcoffset()
by Tim Peters 29 Aug '15

29 Aug '15

[Alex] >> The reason I think Python API is superior is because with exception >> of leap seconds, all transitions in Olson database are given in local >> time in the raw files. [Tim] > Well, they don't have to be in the plain text data files, that's just > the default. And overwhelmingly most common. There are also ways to > say "but this time is in the zone's 'standard time` regardless of > is_dst", and "but this time is in UTC". ... Here's a list of all the string values appearing in Rule-line "AT" columns, each followed by the number of times it appears. The ones ending with "s" mean "standard time, not wall-clock time", and the ones ending with "u" mean "UTC, not wall-clock time". All others are the default wall-clock time (which can be spelled with an explicit "w" suffix, but never is): 0 1 0:00 755 0:00s 74 0:01 12 1:00 61 1:00s 40 1:00u 14 2:00 389 2:00s 277 2:30 7 2:45s 9 3:00 51 3:00s 3 3:00u 16 3:30 26 4:00 4 4:00u 12 5:00 2 9:00 1 11:00 1 12:00 2 22:00 1 22:00s 10 23:00 41 23:00s 105 23:00u 7 23:30 1 23:30s 1 24:00 20 I was surprised to see often "s" gets used. For example, quite often in France rules before 1940 (in the "europe" file). You can even ask me what the difference is between "0:00 tomorrow" and "24:00 today" ;-) There really is a reason for that. Some crazy scheme switching at the end of the last Thursday of some month. zoneinfo has a way to spell, e.g., "the last Thursday of March", but not a way to spell "the day after the last Thursday of March", and "the day after the last Thursday of March" isn't always "the last Friday of March". So you spell the switch as "24:00 on the last Thursday of March". Why on Earth would anyone switch DST during the work week? That one is tougher. If the rules are changed every year, people get annoyed. But if you never change the rules, then eventually DST will switch during some important stretch of the Islamic calendar (which is lunar, not solar, and has years about 11 days shorter than solar years). So the scheme above was apparently calculated to ensure Egypt wouldn't switch DST near the start of Ramadan for at least a decade. But then they'll have to change the rules. But not really, because they already changed them again ;-) Note that it can't work to define DST in regular rules related to the Islamic calendar instead, because "the reason" for DST is to adjust to physical realities related to the solar calendar. Then again, looks like the Egyptians are catching on to that there's no real "reason" to mess with DST at all: http://www.timeanddate.com/news/time/egypt-cancels-dst-2015.html What I remain unclear about is why anyone would imagine any of this is Python's problem ;-)

1 0

Re: [Datetime-SIG] Trivial vs easy: .utcoffset()
by Alexander Belopolsky 29 Aug '15

29 Aug '15

> On Aug 29, 2015, at 2:50 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > > So how can .utcoffset() be computed efficiently in a zoneinfo world > using "hybrid" tzinfo classes (tzinfos that are smart enough to figure > out the appropriate offset all on their own)? As I am learning more about Olson/IANA database, I am more and more convinced that Python approach is better than that of UNIX. The Python approach is to provide effectively local to UTC mapping via utcoffset() while UNIX approach is to provide UTC to local mapping via the localtime() function. Python then supplies fromutc() which is real simple in regular cases and I think implementable in a general case while UNIX supplies its mktime which is a poke six times and hope it is enough mess. The reason I think Python API is superior is because with exception of leap seconds, all transitions in Olson database are given in local time in the raw files. The raw files then get "compiled" so that localtime() can be implemented efficiently and Olson never supplies his own mktime as far as I can tell. A familiar example where DST rules are simpler when formulated in local time are the US rules. In local time, all three (or four?) US zones have exactly the same rule - fall-back at 2am on first Sunday in November and spring-forward at the 2am on the second Sunday in March. Expressed in UTC, the transitions will be all at a different hour and may not even happen on the same day. I think the future of TZ support in Python is to come up with some automatic way to translate from raw Olson files to utcoffset()/dst()/tzname() implementations and invent some clever fromutc() algorithm that will correctly "invert" y = x - x.utcoffset() in all cases. For the later task, I have a solution in my prototype branch, but it requires up to six calls to utcoffset() which may indeed be the best on can do and it is not a coincidence that the number of calls in the worst case is the same as the number of pokes in mktime.

2 1