Mailman 3 August 2015 - Datetime-SIG

Re: [Datetime-SIG] PEP 495: Conversion from naive to aware
by Guido van Rossum 29 Aug '15

29 Aug '15

OK, I've pondered this some more, and in the end I think it is a reasonable proposal. Note that there are many subtleties in various other methods already; e.g. the distinction between timetuple() and utctimetuple(), or between fromtimestamp() and utcfromtimestamp(), is not as straightforward as their names might suggest. So I think it's better not to add more utc*() functions. If I had to do the design over I might have made a deeper distinction between aware and naive datetimes (e.g. aware being a subclass of naive) but such a redesign is not on the table. Another (mostly unrelated) observation about datetime: even an aware datetime does not really represent an instant in time, unless its timezone has a fixed offset. For example, suppose I create a datetime representing noon, June 3rd, US/Eastern time in 2020. My best guess is that this will correspond to the instant 16:00 in UTC on that date. But I can't really tell. Maybe by then the campaign against DST has finally succeeded, and noon on that date in US/Eastern is actually 17:00 UTC. But if I store this date in a database and read it back in the year 2020, I want it to be read back as noon US/Eastern (June 3rd 2020), not as "whatever time in US/Eastern corresponding to 16:00 UTC on that date". Now you may wonder why I prefer it this way, and I'll just let you guess -- but I need this for my use case. If your use case needs to record the instant, you should use UTC or some other fixed offset zone. (And yes, I fundamentally consider pytz's behavior here a bug, and datetime's behavior a feature.) On Fri, Aug 28, 2015 at 4:29 PM, Alexander Belopolsky < alexander.belopolsky(a)gmail.com> wrote: > While PEP 495 is primarily about improving the tzinfo interface, it > proposes one feature that will be visible to datetime users who don't write > their own tzinfo implementations or use any from a third party. This > feature is described in the section "Conversion from Naive to Aware" [1] > and will enable users to call astimezone() method on naive instances to > convert them to aware. > > The PEP proposes that naive datetimes should be presumed > by astimezone() to be in system local timezone similarly to the way they > are treated in the timestamp() method now. > > This feature has been criticized because "Treating a naive datetime as > being a local time is as senseless as treating it as a UTC time - it's > arbitrary." [2] > > The proposed changes to astimezone() will make it discriminate between the > two arbitrary choices. Converting a naive datetime to say UTC timezone > assuming it is local will become > > >>> dt.astimezone(timezone.utc) > > but the same conversion assuming dt is already in UTC is done by > > >>> dt.replace(tzinfo=timezone.utc) > > The asymmetry is more visible in when the task is to imbue a naive > datetime with an appropriate local fixed offset timezone: > > >>> dt.astimezone() # assuming dt is local > > vs. > > >>> dt.replace(tzinfo=timezone.utc).astimezone() # assuming dt is UTC > > As an alternative to an implicit local timezone, we can provide an > explicit timezone.local implementation and tell users to write > > >>> dt.replace(tzinfo=timezone.local).astimezone() # assuming dt is local > > making working with implicitly local and implicitly UTC naive instances > equally inconvenient. > > Furthermore, implicit local time zone logic is already implemented with > respect to the target timezone: to convert to a local (fixed offset) > timezone astimezone() is called without arguments, but any other timezone > needs to be provided explicitly, including timezone.utc. > > IMO, if nondiscrimination between between UTC and local is important, I > would rather add utcastimezone() and utctimestamp() methods that will work > like their namesakes but assume that the instances they are invoked on are > in UTC. > > Please discuss. > > > > > [1]: > https://www.python.org/dev/peps/pep-0495/#conversion-from-naive-to-aware > [2]: > https://mail.python.org/pipermail/datetime-sig/2015-August/000464.html > > _______________________________________________ > Datetime-SIG mailing list > Datetime-SIG(a)python.org > https://mail.python.org/mailman/listinfo/datetime-sig > The PSF Code of Conduct applies to this mailing list: > https://www.python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido)

1 0

Trivial vs easy: .utcoffset()
by Tim Peters 29 Aug '15

29 Aug '15

Timezone conversion is mathematically trivial, but that doesn't mean it's obvious or easy. Details can really bite. tzinfo supplies .utcoffset(), which made converting _to_ UTC dead obvious. But how to convert _from_ UTC remained clear as mud. The default .fromutc() "gets it right" (as far as is possible without a fold/is_dst flag), but _only_ handles DST transitions that strictly alternate between "on" (although with a DST adjustment that may change each time) and "off" (a DST adjustment of exactly 0). Nothing fancier than that; e.g., no base offset changes. It was amazingly annoying to craft an efficient, correct (so far as it goes) implementation of just that much. Even then, hand-written tzinfo implementations had to express DST transition points in "standard time" for it to always work, instead of in natural local wall-clock times (end-of-DST is where that makes a difference). As Alex noted elsewhere, unlike the hand-written .utcoffset() implementations shown in the Python docs, most timezone sources (chiefly Olson - zoneinfo) effectively supply a .fromutc() implementation instead. Which makes converting from UTC dead obvious, but - surprise ;-) - leaves how to convert _to_ UTC (how to implement .utcoffset()) clear as mud instead. In a zoneinfo world, referring back to Guido's diagram a local datetime is staring at a chart with _no_ visible diagonal lines when looking right from the Y (local) axis; they're only visible when looking up from the X (UTC) axis. The hand-written tzinfo classes in the Python docs had the opposite problem, but implicitly left it to the default .fromutc() to figure out the invisible part so "the problem" isn't apparent in the docs. Stewart noted before that always using fixed-offset classes in pytz effectively supplies the missing is_dst bit, but it does more than just that: it effectively stores the datetime's current UTC offset too. The transition charts a pytz tzinfo sees always have a single, continuous diagonal line, visible from both axes. Easy peasy. In return, any operation on the datetime object that creates a new datetime but just copies the original tzinfo into the result may end up with a tzinfo that's no longer correct (lying about the UTC offset that's _appropriate_ for the new date and time). Hence the need to call .normalize() all over the place. If .normalize() were applied magically instead by Python internals, that need would go away, but then timeline arithmetic is "the natural" result - it's unclear to me that classic arithmetic _could_ be implemented if the result of every relevant operation added a "convert to UTC and back again, to get the appropriate current UTC offset" step at the end (not to mention how much slower much code would become). So how can .utcoffset() be computed efficiently in a zoneinfo world using "hybrid" tzinfo classes (tzinfos that are smart enough to figure out the appropriate offset all on their own)? It's like re-inventing the default .fromutc() all over again, but in the other direction in a much lumpier world. Of course there are many ideas. Rather than drone on about them, I'd like to put the puzzle out there in case a correct "duh - it's obvious, you moron" reply is just waiting for an invitation - but do note the "correct" ;-) BTW, after 15 minutes I wasn't able to convince myself I understood what dateutil's zoneinfo-wrapping's .utcoffset() was doing; and if I don't understand what it's doing, there's no way I can guess whether it's always correct. One obvious idea for a zoneinfo exhaustive-list-of-transitions-in-UTC world: precompute another exhaustive list of transitions, but expressed in local time (including "fold") mapping to the correct UTC offset at each point. That could pretty obviously work, but is essentially a way of implementing "poke and hope" in a simple, uniform way (via binary search). There's also that exhaustive lists of transition points is a doomed approach over time. zoneinfo supplies them through 2037 for the benefit of legacy clients, but they expect modern clients to use a POSIX TZ rule (stored in version 2 tzfiles) too. pytz and dateutil both ship with version 2 (or maybe version 3) tzfiles, but neither goes beyond using the version 1 exhaustive-list portion of tzfiles. So more fun is waiting there ;-)

1 0

Re: [Datetime-SIG] A local timezone class
by Guido van Rossum 28 Aug '15

28 Aug '15

On Fri, Aug 28, 2015 at 5:26 PM, Alexander Belopolsky < alexander.belopolsky(a)gmail.com> wrote: > > On Fri, Aug 28, 2015 at 8:06 PM, Guido van Rossum <guido(a)python.org> > wrote: > >> .. >> So, again, why have we been shy of adding this to the stdlib? We did >> (eventually) add the fixed offset classes. >> > > I may be wrong, but I thought it was due to unresolvable DST fold issues > that are mooted by PEP 495. (At least that was the reason I did not push > for LocalTimezone as hard as I did for the fixed offset class.) > Again, that sounds like perfection being the enemy of the good. :-( Those issues are resolved by the demo class however mktime() resolves them when tm_isdst is set to -1. Outside the fold it should work fine, and in the fold it should work as well as can be expected. :-) I am definitely +1 on having LocalTimezone (or timezone.local) in the > datetime module, but there is one reason why I would rather see it as a > part of a larger Olson TZ database-size offering. > > If we just add LocalTimezone, pickling a datetime instance on one system > and reading it on another with a different TZ will result in changing the > time value. > OK, that's definitely a concern. > On the other hand, if we have a complete TZ database, then LocalTimezone > can simply become an alias for Zoneinfo(os.environ['TZ']) and there will be > no problem sharing aware datetime objects between systems that use the same > version of the database. > (But can a TZ database ever be considered complete? :-) I think the best way out of this will be for there to be two different local timezone objects: one that always defers to what the time module reveals about local time, another that is always tied to the local zone as it is recorded in the tz database. The two may differ even if they agree about the tz name; the tz database may have historical details that are missing from the implementation consulted by mktime() and localtime() (e.g. on Windows, or using certain encodings of the TZ environment variable on certain systems). -- --Guido van Rossum (python.org/~guido)

1 0

Re: [Datetime-SIG] A local timezone class
by Alexander Belopolsky 28 Aug '15

28 Aug '15

On Fri, Aug 28, 2015 at 8:26 PM, Alexander Belopolsky < alexander.belopolsky(a)gmail.com> wrote: > On Fri, Aug 28, 2015 at 8:06 PM, Guido van Rossum <guido(a)python.org> > wrote: > >> .. >> So, again, why have we been shy of adding this to the stdlib? We did >> (eventually) add the fixed offset classes. >> > > I may be wrong, but I thought it was due to unresolvable DST fold issues > that are mooted by PEP 495. (At least that was the reason I did not push > for LocalTimezone as hard as I did for the fixed offset class.) > See <http://bugs.python.org/issue5094#msg107072>.

1 0

Re: [Datetime-SIG] A local timezone class
by Alexander Belopolsky 28 Aug '15

28 Aug '15

On Fri, Aug 28, 2015 at 8:06 PM, Guido van Rossum <guido(a)python.org> wrote: > .. > So, again, why have we been shy of adding this to the stdlib? We did > (eventually) add the fixed offset classes. > I may be wrong, but I thought it was due to unresolvable DST fold issues that are mooted by PEP 495. (At least that was the reason I did not push for LocalTimezone as hard as I did for the fixed offset class.) I am definitely +1 on having LocalTimezone (or timezone.local) in the datetime module, but there is one reason why I would rather see it as a part of a larger Olson TZ database-size offering. If we just add LocalTimezone, pickling a datetime instance on one system and reading it on another with a different TZ will result in changing the time value. On the other hand, if we have a complete TZ database, then LocalTimezone can simply become an alias for Zoneinfo(os.environ['TZ']) and there will be no problem sharing aware datetime objects between systems that use the same version of the database.

1 0

Re: [Datetime-SIG] A local timezone class
by Guido van Rossum 28 Aug '15

28 Aug '15

On Fri, Aug 28, 2015 at 4:58 PM, Alexander Belopolsky < alexander.belopolsky(a)gmail.com> wrote: > > On Fri, Aug 28, 2015 at 7:51 PM, Guido van Rossum <guido(a)python.org> > wrote: > >> On Fri, Aug 28, 2015 at 6:53 PM, Guido van Rossum <guido(a)python.org> >>> wrote: >>> >>>> I'm not entirely sure why we didn't add this to the stdlib ages ago. >>> >>> >> But (as my base class shows) implementing the required API using only the >> tm_isdst flag and the timezone-related attributes of the time module >> (tzname, timezone, altzone) a satisfactory implementation can be obtained, >> and that API has been stable since the beginning of time. >> > > To be fair, we did ship something very similar, but it was only accessible > to users who read the library manual: < > https://hg.python.org/cpython/file/v3.5.0rc2/Doc/includes/tzinfo-examples.p…>. > :-) > So, again, why have we been shy of adding this to the stdlib? We did (eventually) add the fixed offset classes. (And before you say "for the same reason we didn't add the USTimeZone class", the reasons can't be the same -- the latter would require maintenance whenever the US changes its DST rules, while LocalTime is oblivious to all that.) -- --Guido van Rossum (python.org/~guido)

1 0

Re: [Datetime-SIG] A local timezone class
by Alexander Belopolsky 28 Aug '15

28 Aug '15

On Fri, Aug 28, 2015 at 7:51 PM, Guido van Rossum <guido(a)python.org> wrote: > On Fri, Aug 28, 2015 at 6:53 PM, Guido van Rossum <guido(a)python.org> >> wrote: >> >>> I'm not entirely sure why we didn't add this to the stdlib ages ago. >> >> >> > But (as my base class shows) implementing the required API using only the > tm_isdst flag and the timezone-related attributes of the time module > (tzname, timezone, altzone) a satisfactory implementation can be obtained, > and that API has been stable since the beginning of time. > To be fair, we did ship something very similar, but it was only accessible to users who read the library manual: < https://hg.python.org/cpython/file/v3.5.0rc2/Doc/includes/tzinfo-examples.p…>. :-)

1 0

Re: [Datetime-SIG] A local timezone class
by Guido van Rossum 28 Aug '15

28 Aug '15

On Fri, Aug 28, 2015 at 4:37 PM, Alexander Belopolsky < alexander.belopolsky(a)gmail.com> wrote: > > On Fri, Aug 28, 2015 at 6:53 PM, Guido van Rossum <guido(a)python.org> > wrote: > >> I'm not entirely sure why we didn't add this to the stdlib ages ago. > > > Maybe because tm_gmtoff was not added to time.struct_time until Python 3.3 > [1] and was not available from the most of the C libraries at the time? > > [1]: https://docs.python.org/3/library/time.html#time.struct_time > But (as my base class shows) implementing the required API using only the tm_isdst flag and the timezone-related attributes of the time module (tzname, timezone, altzone) a satisfactory implementation can be obtained, and that API has been stable since the beginning of time. (Well, at least since 1993: https://hg.python.org/cpython-fullhistory/rev/6ee380349c84 .) -- --Guido van Rossum (python.org/~guido)

1 0

Re: [Datetime-SIG] A local timezone class
by Alexander Belopolsky 28 Aug '15

28 Aug '15

On Fri, Aug 28, 2015 at 6:53 PM, Guido van Rossum <guido(a)python.org> wrote: > I'm not entirely sure why we didn't add this to the stdlib ages ago. Maybe because tm_gmtoff was not added to time.struct_time until Python 3.3 [1] and was not available from the most of the C libraries at the time? [1]: https://docs.python.org/3/library/time.html#time.struct_time

1 0

PEP 495: Conversion from naive to aware
by Alexander Belopolsky 28 Aug '15

28 Aug '15

While PEP 495 is primarily about improving the tzinfo interface, it proposes one feature that will be visible to datetime users who don't write their own tzinfo implementations or use any from a third party. This feature is described in the section "Conversion from Naive to Aware" [1] and will enable users to call astimezone() method on naive instances to convert them to aware. The PEP proposes that naive datetimes should be presumed by astimezone() to be in system local timezone similarly to the way they are treated in the timestamp() method now. This feature has been criticized because "Treating a naive datetime as being a local time is as senseless as treating it as a UTC time - it's arbitrary." [2] The proposed changes to astimezone() will make it discriminate between the two arbitrary choices. Converting a naive datetime to say UTC timezone assuming it is local will become >>> dt.astimezone(timezone.utc) but the same conversion assuming dt is already in UTC is done by >>> dt.replace(tzinfo=timezone.utc) The asymmetry is more visible in when the task is to imbue a naive datetime with an appropriate local fixed offset timezone: >>> dt.astimezone() # assuming dt is local vs. >>> dt.replace(tzinfo=timezone.utc).astimezone() # assuming dt is UTC As an alternative to an implicit local timezone, we can provide an explicit timezone.local implementation and tell users to write >>> dt.replace(tzinfo=timezone.local).astimezone() # assuming dt is local making working with implicitly local and implicitly UTC naive instances equally inconvenient. Furthermore, implicit local time zone logic is already implemented with respect to the target timezone: to convert to a local (fixed offset) timezone astimezone() is called without arguments, but any other timezone needs to be provided explicitly, including timezone.utc. IMO, if nondiscrimination between between UTC and local is important, I would rather add utcastimezone() and utctimestamp() methods that will work like their namesakes but assume that the instances they are invoked on are in UTC. Please discuss. [1]: https://www.python.org/dev/peps/pep-0495/#conversion-from-naive-to-aware [2]: https://mail.python.org/pipermail/datetime-sig/2015-August/000464.html

1 0