Mailman 3 August 2015 - Datetime-SIG

Re: [Datetime-SIG] Conversion vs arithmetic (was Re: Is EDT a timezone? Was: PEP-0500)
by Tim Peters Aug. 25, 2015

Aug. 25, 2015

[Tim] >> ... >> In any case, there is no active PEP at the moment proposing to add any >> new timezone implementations to Python. One step at a time. [ijs] > PEP 431 did. Hence my "active" PEP. Lennart withdrew 431. > But I agree, PEP 495 is a small and straightforward step in the > right direction. Though it's not looking like the bikeshed is going > to get painted my color. :) What if we renamed "fold" to "ijs"? We aim to please ;-) > ... > I assume pytz is committed for backward compatibility to timeline > arithmetic for datetime subtraction, which means it can't afford to > change its "one tz instance per offset" design as long as datetime > arithmetic ignores the time zone when it's identical. For backward compatibility, and for consistency with classic datetime + timedelta addition, the default datetime subtraction won't change. They both have to use the same kind of arithmetic for the obvious identities to hold. But nothing in PEP 495 precludes offering optional timeline arithmetic later - arithmetic just isn't in 495's scope. > There's no hook for pytz to hang its behavior on, and PEP 495 > doesn't give it one. I don't grasp why people keep bringing up issues PEP 495 never intended to address when talking _about_ 495. PEP 495 is _not_ a proposed substitute for PEP 431, and Alexander never claimed it was. It's a different path entirely, prudently (IMO) proposing a relatively small change to get things moving in a useful direction. Even if all related activity stops with 495, fixing conversions alone is worth some effort. Not even appealing to "naive time" can really excuse screwing up conversions _between_ naive times ;-) > On the upside, it does look like there's a way to foil the dreaded > self check in datetime_astimezone and make dt.astimezone(tz) > work as intended. I can pretty much guarantee it always worked as intended ;-) If you mean that 495 will make it work when dt is naive, that's marginal to me. Treating a naive datetime as being a local time is as senseless as treating it as a UTC time - it's arbitrary. I'd rather datetime.timezone grew a name for a tzinfo representing the system timezone, so "local time" became as easy to spell explicitly (when desired) as "utc" has become already. But that's out of scope for 495 too.

1 0

Re: [Datetime-SIG] PEP-431/495
by Stuart Bishop Aug. 25, 2015

Aug. 25, 2015

On 25 August 2015 at 01:02, Guido van Rossum <guido(a)python.org> wrote: > I'm still confused about what makes Stuart believe a tzinfo database must > also change the arithmetic rules (especially since Gustavo's dateutils > apparently gets along quite well without this). A tzinfo database does not care about the arithmetic rules. The database provided by pytz is designed in such a way that you get correct timeline arithmetic out of Python's datetime library, by exposing each period in the timezone definition as distinct fixed offset tzinfo instances. >>> from datetime import datetime, timezone, timedelta >>> from pytz.reference import Eastern as reftz # Tim's reference implementation >>> import pytz >>> dt = datetime(2004, 4, 4, 6, 59, tzinfo=timezone.utc) >>> str(dt.astimezone(pytz.timezone('US/Eastern'))) '2004-04-04 01:59:00-05:00' >>> str(dt.astimezone(reftz)) '2004-04-04 01:59:00-05:00' >>> str(dt.astimezone(pytz.timezone('US/Eastern')) + timedelta(minutes=1)) '2004-04-04 02:00:00-05:00' >>> str(dt.astimezone(reftz) + timedelta(minutes=1)) '2004-04-04 02:00:00-04:00' So I think pytz's current implementation is entwined with the arithmetic style. But the tzfile loader in pytz could be reworked to present tzinfo instances suitable for classic arithmetic if you want it in stdlib, or the parser out of dateutils used. >> "Classic behaviour as you describe it is a bug. It sounds ok when you >> state it as 'add one day to today and you get the same time tomorrow'. >> It does not sound ok when you state it as 'add one second to now and >> you will normally get now + 1 second, but sometimes you will get an >> instant further in the future, and sometimes you will get an instant >> in the past'." > > That sounds tautological to me -- "it's a bug because I find it buggy". I call it a bug because adding 1 second to a moment in time and having it incremented by an entire hour is extremely surprising behaviour. This surprising behaviour also makes it impossible to reliably add an absolute value to a timezone aware datetime. While people can understand that adding a relative time period such as 1 day might get you the same time tomorrow, it makes absolutely no sense that adding an absolute time period like 1 second or 24 hours does not give you an absolute answer. It is unfortunate that in this case the datetime library interprets a timedelta as a relative period when it is documented as being an absolute period. It maybe documented behaviour, but the vast majority will consider it a bug and will consider Python's datetime handling broken. At the moment, hardly anybody is aware of the issue. I'd never even really considered it myself - I was just trying to get my library to spit the right answers out. > Maybe the underlying reason is that to me, even a datetime with tzinfo does > not refer to an instant -- it refers to something that's displayed on a > clock. To me, arithmetic is a way of moving the hands of the clock. In this definition, the current arithmetic fails. If you have Tim's clock or a phone which displays the utcoffset, there are positions you cannot move the clock hands to. In a fold you can only set the hands to either the pre or post DST position, but not both. But you can set the hands to invalid positions if you want to confuse people. If, however, you stop ignoring the tzinfo you certainly can consider arithmetic as a way of moving the hands of the clock. Add one minute, and the minute hand goes from 59 to 0 and the hour hand increments by one. Add one hour and the hour hand decrements and the utcoffset is changed. All valid positions can be reached, and invalid ones helpfully avoided. -- Stuart Bishop <stuart(a)stuartbishop.net> http://www.stuartbishop.net/

1 0

Re: [Datetime-SIG] PEP-431/495
by Tim Peters Aug. 25, 2015

Aug. 25, 2015

[Stuart Bishop] I'm skipping to the end here to maintain some hope of temporal continuity with the "Conversion vs arithmetic" thread I already spun off: > For amusement, here is how you can add an hour and end up exactly > where you started. Careful you do your conversions at the right time, > or the dst transition might eat your data (this example performed by a > professional stuntman and should not be attempted at home): > > >>> from pytz.reference import Eastern > >>> dt = datetime(2004, 4, 4, 1, 0, 0, tzinfo=Eastern) > >>> str(dt.astimezone(timezone.utc)) > '2004-04-04 06:00:00+00:00' > >>> str((dt + timedelta(hours=1)).astimezone(timezone.utc)) > '2004-04-04 06:00:00+00:00' The missing bit there: from datetime import datetime, timezone, timedelta at the start. For those who don't know (like me ;-) ), pytz.reference.Eastern appears to be a copy/paste from the first version of the datetime docs, showing a very simple implementation of US daylight rules at the time it was written (transitions at 2am local time on the first Sunday of April and the last Sunday of October). Eastern's standard offset is -5 (and daylight -4). So you're creating a gap time at DST start, via classic arithmetic, and marveling ;-) at the senseless output. The original dt is "in standard time", and correctly maps to the UTC hour 5 hours later. Adding an hour creates 2am standard time, which doesn't exist on the local clock (which jumps from 1:59:59 to 3:00:00). But since 2 is >= 2, it's considered to be in daylight time, and offset -4 maps to the same 6am UTC. In timeline arithmetic, the addition would have jumped to 3am, and offset -4 would map to 7am UTC. It would be a mistake to believe it wasn't all known a dozen years ago that stuff like that would happen. We were acutely aware of it. Regardless, classic arithmetic was a deliberate design decision. I don't think it would help to keep repeating why, not any more than it would help to keep showing examples where timeline and classic arithmetic give different results. Everyone already knows the latter, and everyone already knows some people hate it ;-) Why I wanted to isolate this part was for the _conversion_ question, which is a source of surprises divorced from arithmetic. Like so: u = datetime(2004, 10, 31, 5, 27, tzinfo=timezone.utc) print(u) print(u.astimezone(Eastern).astimezone(timezone.utc)) There I contrived to create a UTC time in the fold at Eastern DST end (in the same year as your example). Then roundtrip to Eastern and back The output: 2004-10-31 05:27:00+00:00 2004-10-31 06:27:00+00:00 So that's how you can do nothing at all yet end up an hour away ;-) In that case, `u` maps to 1:27am Eastern daylight time, which is ambiguous in Eastern. Because it's impossible now to record that the earlier of the ambiguous hours was picked, on the trip back it's considered to be in standard time, so maps to the next UTC hour. _This_ part has nothing to do with timeline arithmetic, and (as explained in the "Conversion vs arithmetic" spinoff) will fix itself by magic as soon as `Eastern` and `timezone.utc` tzinfos implementing PEP 495 exist. Conversion surprises have nothing to do with arithmetic; they're solely due to the lack of a disambiguation bit.

1 0

Re: [Datetime-SIG] Conversion vs arithmetic (was Re: Is EDT a timezone? Was: PEP-0500)
by Carl Meyer Aug. 25, 2015

Aug. 25, 2015

On 08/24/2015 05:50 PM, Tim Peters wrote: > [ijs] >>>>> I see how PEP 495 makes it possible to convert datetimes correctly >>>>> in all cases, > > [Tim] >>>> Whew! You may well be only the second person to grasp this - thanks :-) > > [ijs] >>> Wait... was this ever in contention? > > [Carl Meyer] >> I don't think so. I'm not sure what Tim is referring to here; I haven't >> seen any messages in this thread that indicated a lack of understanding >> on this point. > > I've lost track of how often I've explained it. That suggests it's > not being understood. Start with the message at the top of _this_ > (renamed) thread, which was posted today, so it shouldn't fade > entirely from memory for at least another 37 minutes ;-) Fair enough :-) I can't seem to find Stuart's message that you were replying to, but my (sometimes failing) memory thought it was more of a design justification of pytz (in a non-PEP-495 world) than a critique of PEP 495. In any case, since I've argued several times that naive arithmetic on aware datetimes is wrong, I'll be crystal clear: I don't think there's any question that it is possible to make conversions unambiguous without changing how datetime arithmetic works, and that PEP 495 achieves that much. My only quibbles about PEP 495 are minor ones: naming, API convenience for "strict mode", and the fact that it won't help me, because until we do change the arithmetic, I'll keep on using pytz and its fixed-offset tzinfo objects :-) Carl

2 1

Re: [Datetime-SIG] Conversion vs arithmetic (was Re: Is EDT a timezone? Was: PEP-0500)
by Tim Peters Aug. 25, 2015

Aug. 25, 2015

[Guido] > Sounds to me like Tim and Stuart disagree on the definition of conversion, > though. I don't think so. The pytz docs contain a variant of the word "conversion" 6 times, each one followed by examples without any arithmetic but with calls to .astimezone(). Contrarily, the examples showing arithmetic are not preceded by text calling them any variant of "conversions". So, in all, it's more likely I had no real idea what "Conversion was the larger issue, and to do that correctly the arithmetic needed to be fixed" meant to begin with ;-)

1 0

Re: [Datetime-SIG] Conversion vs arithmetic (was Re: Is EDT a timezone? Was: PEP-0500)
by Alexander Belopolsky Aug. 25, 2015

Aug. 25, 2015

On Mon, Aug 24, 2015 at 8:07 PM, Guido van Rossum <guido(a)python.org> wrote: > Sounds to me like Tim and Stuart disagree on the definition of conversion, > though. That's not surprising given that Tim's .fromutc() [1] *does not* change .tzinfo (so arguably it is not a timezone conversion) but *does* involve arithmetic (because ignoring the DST nonsense, it is just dt + dt.utcoffset()). Fortunately, Tim has established that there are only two people in the world that need to understand any of this, but Stuart is one of them. :-) [1]: https://hg.python.org/cpython/file/v3.5.0rc1/Lib/datetime.py#l957

1 0

Re: [Datetime-SIG] Conversion vs arithmetic (was Re: Is EDT a timezone? Was: PEP-0500)
by Guido van Rossum Aug. 25, 2015

Aug. 25, 2015

Sounds to me like Tim and Stuart disagree on the definition of conversion, though. On Mon, Aug 24, 2015 at 5:00 PM, Alexander Belopolsky < alexander.belopolsky(a)gmail.com> wrote: > > On Mon, Aug 24, 2015 at 7:55 PM, Carl Meyer <carl(a)oddbird.net> wrote: > >> But you're changing the subject. That exercise requires far more >> detailed knowledge of the tzinfo APIs than is required to understand >> that a single added disambiguating bit is enough to handle timezone >> conversions, even in the presence of (two-layered) folds and gaps. >> > > Sorry for that. Meanwhile, however Tim has responded and the point that > both of us think is misunderstood is that contrary to Stuart's assertion > below and at the top of this thread, all conversion issues can be addressed > without any change to arithmetic. (Moreover, addressing them with classic > arithmetic is easier.) > > [Stuart Bishop <stuart(a)stuartbishop.net>] > > ... > > Conversion was the larger issue, and to do that correctly the > > arithmetic needed to be fixed. > > _______________________________________________ > Datetime-SIG mailing list > Datetime-SIG(a)python.org > https://mail.python.org/mailman/listinfo/datetime-sig > The PSF Code of Conduct applies to this mailing list: > https://www.python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido)

1 0

Re: [Datetime-SIG] Conversion vs arithmetic (was Re: Is EDT a timezone? Was: PEP-0500)
by Carl Meyer Aug. 25, 2015

Aug. 25, 2015

On 08/24/2015 05:23 PM, Alexander Belopolsky wrote: > > On Mon, Aug 24, 2015 at 7:01 PM, Carl Meyer <carl(a)oddbird.net > <mailto:carl@oddbird.net>> wrote: > > On 08/24/2015 03:58 PM, ISAAC J SCHWABACHER wrote: > > [ijs] > >>> I see how PEP 495 makes it possible to convert datetimes correctly > >>> in all cases, > >> > > [Tim Peters] > >> Whew! You may well be only the second person to grasp this - thanks :-) > > > > Wait... was this ever in contention? > > I don't think so. I'm not sure what Tim is referring to here; I haven't > seen any messages in this thread that indicated a lack of understanding > on this point. > > > I cannot speak for Tim, but here is an exercise you can try to check > your grasp of PEP 495: given PEP-compliant utcoffset() and dst() methods > for the current rules in the US/Eastern timezone, write a PEP-compliant > fromutc(). You can start from the current datetime.py implementation. [1] > > [1]: https://hg.python.org/cpython/file/v3.5.0rc1/Lib/datetime.py#l957 Looks fun; I'll give it a try later tonight. But you're changing the subject. That exercise requires far more detailed knowledge of the tzinfo APIs than is required to understand that a single added disambiguating bit is enough to handle timezone conversions, even in the presence of (two-layered) folds and gaps. I haven't seen anyone object to PEP 495 on the grounds that it doesn't provide enough information to handle timezone conversions (well, except in the as-yet-hypothetical case of a three-layered fold, which could be handled by your latest proposal to make `ltdf` an integer rather than a boolean). I've seen a lot of discussion of the name of the flag, and a couple people express a desire to have a simpler built-in spelling of "raise an exception on invalid/ambiguous time" than "convert twice with two different flag values and compare the results". Carl

2 1

Re: [Datetime-SIG] Conversion vs arithmetic (was Re: Is EDT a timezone? Was: PEP-0500)
by Carl Meyer Aug. 24, 2015

Aug. 24, 2015

On 08/24/2015 03:58 PM, ISAAC J SCHWABACHER wrote: > [ijs] >>> I see how PEP 495 makes it possible to convert datetimes correctly >>> in all cases, >> > [Tim Peters] >> Whew! You may well be only the second person to grasp this - thanks :-) > > Wait... was this ever in contention? I don't think so. I'm not sure what Tim is referring to here; I haven't seen any messages in this thread that indicated a lack of understanding on this point. Carl

3 2

Re: [Datetime-SIG] pytz vs. PEP 495 Was: PEP-431/495
by Ethan Furman Aug. 24, 2015

Aug. 24, 2015

On 08/24/2015 03:18 PM, Alexander Belopolsky wrote: > On Mon, Aug 24, 2015 at 5:58 PM, Ethan Furman wrote: > > try: > some_date = get_a_datetime_from_somewhere(ltdf=None) > except AmbiguousTimeError, NonExistentTimeError as e: > log(weird_time_error, e) > some_date = use_default(e.datetime) > > .. >> Writing functions to work around short-comings is not how I like to spend >> my time (see my contention above about library code). > > > Neither do I. That's why in my code you are more likely to see > > some_date = get_a_datetime_from_somewhere() > > instead of the wall of text that you presented. As you can imagine, I will > not be happy facing a prospect of adding try:.. except's everywhere to port > my code to Python 3.6. Why would you have to? I am not suggesting that None be the default, simply that it be available. -- ~Ethan~

1 0