Mailman 3 August 2015 - Datetime-SIG

Re: [Datetime-SIG] PEP-495 - Strict Invalid Time Checking
by Stuart Bishop Aug. 25, 2015

Aug. 25, 2015

On 26 August 2015 at 01:42, Alexander Belopolsky <alexander.belopolsky(a)gmail.com> wrote: > Speaking of DBAs, how would she feel if a system zoneinfo upgrade made her > database unreadable? A zoneinfo upgrade can easily create new gaps and > folds even in the past if someone at IANA decides that they found a better > source of historical information. Speaking as a DBA, I know that my timestamps with timezones are converted and stored as UTC time on entry so this doesn't happen. You use timestamp with timezone to store an absolute time. It does, however, mean that revisions to the IANA database may change the result when it is rendered back in its original timezone. If you don't want that, you actually want a timestamp without timezone (naive), and maybe a physical location (because the local government may declare that this year they will central time instead of mountain time). -- Stuart Bishop <stuart(a)stuartbishop.net> http://www.stuartbishop.net/

1 0

Re: [Datetime-SIG] PEP-495 - Strict Invalid Time Checking
by Alexander Belopolsky Aug. 25, 2015

Aug. 25, 2015

On Tue, Aug 25, 2015 at 1:56 PM, Stuart Bishop <stuart(a)stuartbishop.net> wrote: > What would be even nicer is if users didn't have to use localize at all: > > >> datetime.now(tz=pytz.timezone('US/Eastern')) > This is certainly one of the main goals of PEP 495. Note that datetime.now() will never produce an invalid datetime and will always set the fold attribute correctly in the (two-)fold case if pytz.timezone('US/Eastern') follows PEP guidelines. Therefore there is no need for the third fold value in this case.

1 0

Re: [Datetime-SIG] PEP-495 - Strict Invalid Time Checking
by Alexander Belopolsky Aug. 25, 2015

Aug. 25, 2015

On Tue, Aug 25, 2015 at 1:56 PM, Stuart Bishop <stuart(a)stuartbishop.net> wrote: > On 25 August 2015 at 21:51, Alexander Belopolsky > <alexander.belopolsky(a)gmail.com> wrote: > > > On Tue, Aug 25, 2015 at 9:30 AM, Stuart Bishop <stuart(a)stuartbishop.net> > > wrote: > >> > >> As mentioned elsewhere, pytz requires strict checking to remain > >> backwards compatible. > > > > Can you provide the specific examples where strict checking is required? > > Systems where it is better to fail than continue with incorrect > results. For example, ingesting transaction logs. It is more desirable > for the script parsing the log files to fail with a traceback than to > feed incorrect results into the rest of the system, where some poor > DBA is going to have to repair the cascade of damage months or years > later. > Speaking of DBAs, how would she feel if a system zoneinfo upgrade made her database unreadable? A zoneinfo upgrade can easily create new gaps and folds even in the past if someone at IANA decides that they found a better source of historical information.

1 0

Re: [Datetime-SIG] Conversion vs arithmetic (was Re: Is EDT a timezone? Was: PEP-0500)
by ISAAC J SCHWABACHER Aug. 25, 2015

Aug. 25, 2015

[ijs] > > But I agree, PEP 495 is a small and straightforward step in the > > right direction. Though it's not looking like the bikeshed is going > > to get painted my color. :) [Tim] > What if we renamed "fold" to "ijs"? We aim to please ;-) Heh. I argued that pandas should call it "ambiguous" and allow it the values True | False | 'NaT' | 'raise' | 'infer', where the boolean values had the same sense as if the flag were "is_dst" or "first" and 'NaT' and 'infer' were important for the vectorized case. And I convinced them, and they painted the bikeshed my color, and now I have doubts and wonder whether the flag should simply be called "is_dst" since that's what everyone else calls it (so that must be how they think of it, right?) and accompanied by a stern corner-case warning in the docs. Which is to say that I think it's important that the flag's name be grokkable. "is_dst" is grokkable. "first" is grokkable, once you get past the point of "first what?". "fold" is not grokkable. It's also not discoverable, which is also a problem for "first"-- the name of the field doesn't tell you what problem it solves. Nobody is going to be confused about what "is_dst" means (which is great until you get to a corner case where it doesn't mean that). So I'm torn between "is_dst" and "first", but I really don't like "fold". And that's all I have to say about that. > > ... > > I assume pytz is committed for backward compatibility to timeline > > arithmetic for datetime subtraction, which means it can't afford to > > change its "one tz instance per offset" design as long as datetime > > arithmetic ignores the time zone when it's identical. > > For backward compatibility, and for consistency with classic datetime > + timedelta addition, the default datetime subtraction won't change. > They both have to use the same kind of arithmetic for the obvious > identities to hold. But nothing in PEP 495 precludes offering > optional timeline arithmetic later - arithmetic just isn't in 495's > scope. > > > There's no hook for pytz to hang its behavior on, and PEP 495 > > doesn't give it one. > > I don't grasp why people keep bringing up issues PEP 495 never > intended to address when talking _about_ 495. PEP 495 is _not_ a > proposed substitute for PEP 431, and Alexander never claimed it was. > It's a different path entirely, prudently (IMO) proposing a relatively > small change to get things moving in a useful direction. Even if all > related activity stops with 495, fixing conversions alone is worth > some effort. Not even appealing to "naive time" can really excuse > screwing up conversions _between_ naive times ;-) I brought it up because I'm looking at PEP 495 through the lens of "how much simpler does this allow pytz's API to become?" Support for timeline arithmetic is a major constraint on pytz's design, and it could hamper that module's ability to make use of the gains from PEP 495-- in which case I would argue that the scope of the PEP should be expanded at least enough to permit pytz to partake. I initially thought this would be necessary, but then reread the relevant parts of _datetimemodule.c and came to the conclusion that it's not. But I admit I did not motivate the connection in my post. > > On the upside, it does look like there's a way to foil the dreaded > > self check in datetime_astimezone and make dt.astimezone(tz) > > work as intended. > > I can pretty much guarantee it always worked as intended ;-) As Stuart intended, when tz is a pytz.DstTzInfo object and in the corner case that dt is a non-normalized datetime in the same time zone (i.e., dt.tzinfo.utcoffset() returns the wrong offset). Jeez, how could that not be obvious? ;D > If you > mean that 495 will make it work when dt is naive, that's marginal to > me. Treating a naive datetime as being a local time is as senseless > as treating it as a UTC time - it's arbitrary. I'd rather > datetime.timezone grew a name for a tzinfo representing the system > timezone, so "local time" became as easy to spell explicitly (when > desired) as "utc" has become already. But that's out of scope for 495 > too. That's not what I mean; I agree with you here. ijs

1 0

Re: [Datetime-SIG] PEP-495 - Strict Invalid Time Checking
by Stuart Bishop Aug. 25, 2015

Aug. 25, 2015

On 25 August 2015 at 21:51, Alexander Belopolsky <alexander.belopolsky(a)gmail.com> wrote: > On Tue, Aug 25, 2015 at 9:30 AM, Stuart Bishop <stuart(a)stuartbishop.net> > wrote: >> >> As mentioned elsewhere, pytz requires strict checking to remain >> backwards compatible. > > Can you provide the specific examples where strict checking is required? Systems where it is better to fail than continue with incorrect results. For example, ingesting transaction logs. It is more desirable for the script parsing the log files to fail with a traceback than to feed incorrect results into the rest of the system, where some poor DBA is going to have to repair the cascade of damage months or years later. Yes, pytz already has a disambiguation solution. I would love to be able to deprecate it, and encourage users to use stdlib as much as possible. > Since pytz already has a disambiguation solution, PEP 495 is not as > indispensable for it as it is for datetime or dateutil. However, there is > one case I can think of where pytz will benefit: with the PEP, it will be > possible to make say Eastern.localize(datetime.now()) work correctly at all > times. If for backward compatibility, you want to continue raising > AmbiguousTimeError during one hour each year, I am sure you will figure out > how to make Eastern.localize(datetime.now(), isdst=None). (Hint: Don't > change anything in this branch of your code.) Yes, that is nice. What would be even nicer is if users didn't have to use localize at all: >> datetime.now(tz=pytz.timezone('US/Eastern')) > I assume you refer to "Another suggestion was to use first=-1 or first=None > to indicate that > the program truly has no means to deal with the folds and gaps and > dt.utcoffset() should raise an error whenever dt represents an > ambiguous or missing local time." > > It looks like you want to make it impossible to construct invalid dt > instances. In other words, you want to make dt.replace(fold=-1) or > dt.replace(tzinfo=Eastern) raise an error under certain circumstances. Is > this right? Right. When a user requests that exceptions are raised, it becomes impossible to construct invalid dt instances. This does require calling pytz Python code from the datetime constructor, which you discuss below. >> pytz users need to optionally have exceptions raised >> when they try to construct an invalid or ambiguous datetime instance, > > This is a legitimate need, by why does it need to be done in datetime rather > than in pytz itself? You already ignore the datetime(..., tzinfo=...) > constructor and require your users to call localize() instead. What stops > you from providing a function strict_datetime() that will perform any checks > that you or your users desire? My goal is to have no pytz specific API, or at least minimize it and make it unnecessary for the most common use cases. There will be much less confusion, and it will be easier for large projects to migrate to stdlib if it meets their needs. >> directly via __new__ or indirectly with something like dt.replace(). > > __new__ and .replace() are low level methods called in many performance > critical places. We cannot afford to call arbitrary python code in those > methods. This seems to be the crux of the issue. The datetime constructor needs to call into the tzinfo implementation, in the same way as converting from utc time invokes tzinfo.fromutc(dt). If it did this, pytz could swap in the correct fixed offset tzinfo and, if requested, perhaps raise an exception. The localize method would be gone entirely, the biggest half of pytz' problematic API. The datetimes filtered through pytz would always be valid, and dt.utcoffset() will never raise an exception causing confusing failures. Is the overhead on calling a method on the tzinfo that bad? If the tzinfo implementation does not override it, it should still be fast. pytz users are already paying the overhead in the form of the localize method (which seems about 20x slower than just using datetime.now() unwrapped, 20usec vs 2usec according to timeit. But if you cared, you would be using time.time() at 0.08usec). Is there some way of me providing a hook that doesn't cause major slowdowns for non pytz users? For what its worth, the only complaints I've ever had about performance with pytz have been about how long it took to import the package. I'm not sure what sort of application instantiates so many timezone aware datetime instances that constructor overhead becomes noticeable. All examples I can come up with that create timestamps so rapidly would never add timezone information, and would be using a float or long for further optimization in any case. >> If these methods are called with first=None, it will be passed through >> to dt.utcoffset() and it may raise an exception. > > This part I don't understand. If __new__ raises an exception - you will > have no instance to "be passed through to dt.utcoffset()." Sorry. I'm mixing up tzinfo.utcoffset() and dt.utcoffset() here. > .astimezone() does not and need not "allow the user to specify which side > of the fold in the target zone." As long as it knows how to interpret the > time that it is given (disambiguate the fold and "normalize" the gap) it > should be able to set the fold=1 attribute correctly in the result if it > happen to fall into the repeated hour and should never produce a time that > is in the gap of the target zone. Yes. I got this backwards sorry. -- Stuart Bishop <stuart(a)stuartbishop.net> http://www.stuartbishop.net/

1 0

Re: [Datetime-SIG] time module issues Was: PEP-431/495
by Alexander Belopolsky Aug. 25, 2015

Aug. 25, 2015

I am changing the subject line because neither of the PEPs mentioned in the original subject propose any changes to the time module. It is also likely that this discussion is off-topic on the Datetime-SIG mailing list and we should continue it on the relevant bug tracker issues. On Tue, Aug 25, 2015 at 11:47 AM, Akira Li <4kir4.1i(a)gmail.com> wrote: > > Alexander Belopolsky <alexander.belopolsky(a)gmail.com> writes: > > >> On Aug 25, 2015, at 7:44 AM, Akira Li <4kir4.1i(a)gmail.com> wrote: > >> > >> note: stdlib variant datetime.now(timezone.utc).astimezone() may fail if it > >> uses time.timezone, time.tzname internally [3,4,5] when tm_gmtoff > >> tm_zone are not available on a given platform. > > > > If this actually happens on any supported platform - please file a bug > > report. What we do in this case is not as simplistic as you describe. > > Bug-driven development is probably not the best strategy for a datetime > library ;) Tests can't catch all bugs. I've found out that astimezone() > may fail by *reading* its source and trying to *understand* what it does. I agree, but once you've read the code and see any logical errors, you should be able to construct a test case demonstrating wrong behavior. > > Here's the part from datetime.py [1] that computes the local timezone if > tm_gmtoff or tm_zone are not available: > > # Compute UTC offset and compare with the value implied > # by tm_isdst. If the values match, use the zone name > # implied by tm_isdst. > delta = local - datetime(*_time.gmtime(ts)[:6]) > dst = _time.daylight and localtm.tm_isdst > 0 > gmtoff = -(_time.altzone if dst else _time.timezone) > if delta == timedelta(seconds=gmtoff): > tz = timezone(delta, _time.tzname[dst]) > else: > tz = timezone(delta) > > Here's its C equivalent [2]. > > Python issues that I've linked in the previous message [3,4,5] demonstrate > that time.timezone and time.tzname may have wrong values and therefore > the result *tz* may have a wrong tzname. To summarize for those who will not follow the links: [3] Is a closed "No obvious and correct way to get the time zone offset" issue. It was superseded by <http://bugs.python.org/issue9527> which in turn was closed by implementing the argument-less .astimezone() method. [4] and [5] are time module issues. > > Here's an example inspired by > "incorrect time.timezone value" Python issue [4]: > > >>> from datetime import datetime, timezone > >>> from email.utils import parsedate_to_datetime > >>> import tzlocal # to get local timezone as pytz timezone > >>> d = parsedate_to_datetime("Tue, 28 Oct 2013 14:27:54 +0000") > >>> # expected (TZ=Europe/Moscow) > ... > >>> d.astimezone(tzlocal.get_localzone()).strftime('%Z%z') > 'MSK+0400' > >>> # got > ... > >>> d.astimezone().strftime('%Z%z') > 'UTC+04:00+0400' > I don't understand why you keep presenting a mix of pytz, email.utils and something called "tzlocal" and then claim that the unexpected behavior indicates a problem in the datetime module? It could as well be in any of the three other modules that you use or in the way you combine them. If you want to parse the string "Tue, 28 Oct 2013 14:27:54 +0000" and convert it to Moscow time, here is how you do it using the datetime module: >>> import os; os.environ['TZ'] = 'Europe/Moscow' >>> from datetime import datetime >>> d = datetime.strptime("Tue, 28 Oct 2013 14:27:54 +0000", "%a, %d %b %Y %H:%M:%S %z") >>> d.astimezone().strftime("%F %T %Z%z") '2013-10-28 18:27:54 MSK+0400' Does this code behave differently on your system? If it does - please file a bug report. > > 'UTC+04:00' instead of 'MSK' is not a major issue. I don't consider it a > bug because without access to the tz database stdlib can't do much > better, there always be cases when it breaks. It is quite possible that that such cases exist, but you have not demonstrated one. > > I just use pytz instead which does provide access to the tz database. This will always be your option as it is your option to use just the datetime module. In both cases you can write correct code if you follow the reference manual or buggy code if you don't. An almost sure way to write buggy code is to use one library manual to write code using another. > > > [1] https://github.com/python/cpython/blob/fced0e12fc510e4a6158628695774ccfd023… > [2] https://github.com/python/cpython/blob/fced0e12fc510e4a6158628695774ccfd023… > [3] http://bugs.python.org/issue1647654 > [4] http://bugs.python.org/issue22752 > [5] http://bugs.python.org/issue22798

1 0

Re: [Datetime-SIG] PEP-431/495
by Akira Li Aug. 25, 2015

Aug. 25, 2015

Stuart Bishop <stuart(a)stuartbishop.net> writes: > On 25 August 2015 at 13:39, Akira Li <4kir4.1i(a)gmail.com> wrote: > >> As far as I know utc -> local timezone *conversions* do not work during >> DST transitions in dateutil [1] while pytz manages just fine. >> >> Let's not do, whatever dateutil does here. It would be a regression. > > dateutils does not work at the moment. With the addition of the new > flag proposed by PEP-495 it should be a simple update to make > dateutils work. Similarly, depending on the semantics of the updated > datetime constructors it may be possible to drop pytz' localize method > and have it work better. It is not clear why one need a disambiguation flag if there is no ambiguity in utc -> local timezone conversions but it is great that PEP-495 might help fix the dateutil bug [1] [1] https://github.com/dateutil/dateutil/issues/112 Should it also help fix datetime.now(tz.tzlocal()) [2] bug -- getting the current time in the local timezone as an aware datetime object? [2] https://github.com/dateutil/dateutil/issues/57 note: stdlib variant datetime.now(timezone.utc).astimezone() may fail if it uses time.timezone, time.tzname internally [3,4,5] when tm_gmtoff tm_zone are not available on a given platform. [3] http://bugs.python.org/issue1647654 [4] http://bugs.python.org/issue22752 [5] http://bugs.python.org/issue22798 pytz variant datetime.now(tzlocal.get_localzone()) works even during DST transitions. If PEP-495 implies that dateutil type of timezone implementation might be adapted in stdlib; it might be worth mentioning to what types of bugs it leads to.

2 2

Re: [Datetime-SIG] PEP-495 - Strict Invalid Time Checking
by Alexander Belopolsky Aug. 25, 2015

Aug. 25, 2015

On Tue, Aug 25, 2015 at 9:30 AM, Stuart Bishop <stuart(a)stuartbishop.net> wrote: > As mentioned elsewhere, pytz requires strict checking to remain > backwards compatible. > Can you provide the specific examples where strict checking is required? Since pytz already has a disambiguation solution, PEP 495 is not as indispensable for it as it is for datetime or dateutil. However, there is one case I can think of where pytz will benefit: with the PEP, it will be possible to make say Eastern.localize(datetime.now()) work correctly at all times. If for backward compatibility, you want to continue raising AmbiguousTimeError during one hour each year, I am sure you will figure out how to make Eastern.localize(datetime.now(), isdst=None). (Hint: Don't change anything in this branch of your code.) > The description above does not match the desired > behaviour though. > I assume you refer to "Another suggestion was to use first=-1 or first=None to indicate that the program truly has no means to deal with the folds and gaps and dt.utcoffset() should raise an error whenever dt represents an ambiguous or missing local time." It looks like you want to make it impossible to construct invalid dt instances. In other words, you want to make dt.replace(fold=-1) or dt.replace(tzinfo=Eastern) raise an error under certain circumstances. Is this right? > pytz users need to optionally have exceptions raised > when they try to construct an invalid or ambiguous datetime instance, > This is a legitimate need, by why does it need to be done in datetime rather than in pytz itself? You already ignore the datetime(..., tzinfo=...) constructor and require your users to call localize() instead. What stops you from providing a function strict_datetime() that will perform any checks that you or your users desire? > directly via __new__ or indirectly with something like dt.replace(). > __new__ and .replace() are low level methods called in many performance critical places. We cannot afford to call arbitrary python code in those methods. > If these methods are called with first=None, it will be passed through > to dt.utcoffset() and it may raise an exception. > This part I don't understand. If __new__ raises an exception - you will have no instance to "be passed through to dt.utcoffset()." > dt.utcoffset() will > only ever raise an exception when the user has explicitly requested > construction of a datetime with strict checking. > If you allow constructing instances with failing .utcoffset(), these instances will make innocent-looking code capable of raising an error. For example dt1 in {dt2} will raise the same error as dt2.utcoffset() regardless of what dt1 is. You will have similar problems with dt1 == dt2, dictionary lookups, list searches and so on. > It will never raise > an exception in normal operation, including arithmetic, and could not > cause problems in the situations you cite. > If I have two instances dt1 and dt2 with different .tzinfo and dt1.utcoffset() raises an exception. What dt1 - dt2 should return? or dt1 == dt2? or hash(dt1)? > > This would require the first argument to be available in all the > methods that can construct datetime instances, including things not > mentioned in the PEP like dt.astimezone() > .astimezone() is mentioned in the PEP [1], but I should probably add explicit discussion of how it should handle invalid times. In my view, the behavior of .astimezone() follows straight from that of .utcoffset(), but it may not be obvious from the PEP alone. > (Where I think it might be > using the first flag from the original dt instance, rather than > allowing the user to specify which side of the fold in the target > zone). > .astimezone() does not and need not "allow the user to specify which side of the fold in the target zone." As long as it knows how to interpret the time that it is given (disambiguate the fold and "normalize" the gap) it should be able to set the fold=1 attribute correctly in the result if it happen to fall into the repeated hour and should never produce a time that is in the gap of the target zone. [1]: https://www.python.org/dev/peps/pep-0495/#conversion-from-naive-to-aware

1 0

PEP-495 - Strict Invalid Time Checking
by Stuart Bishop Aug. 25, 2015

Aug. 25, 2015

There is a point I'd like to clarify in the current PEP: """ Strict Invalid Time Checking Another suggestion was to use first=-1 or first=None to indicate that the program truly has no means to deal with the folds and gaps and dt.utcoffset() should raise an error whenever dt represents an ambiguous or missing local time. The main problem with this proposal, is that dt.utcoffset() is used internally in situations where raising an error is not an option: for example, in dictionary lookups or list/set membership checks. So strict gap/fold checking behavior would need to be controlled by a separate flag, say dt.utcoffset(raise_on_gap=True, raise_on_fold=False) . """ As mentioned elsewhere, pytz requires strict checking to remain backwards compatible. The description above does not match the desired behaviour though. pytz users need to optionally have exceptions raised when they try to construct an invalid or ambiguous datetime instance, directly via __new__ or indirectly with something like dt.replace(). If these methods are called with first=None, it will be passed through to dt.utcoffset() and it may raise an exception. dt.utcoffset() will only ever raise an exception when the user has explicitly requested construction of a datetime with strict checking. It will never raise an exception in normal operation, including arithmetic, and could not cause problems in the situations you cite. This would require the first argument to be available in all the methods that can construct datetime instances, including things not mentioned in the PEP like dt.astimezone() (Where I think it might be using the first flag from the original dt instance, rather than allowing the user to specify which side of the fold in the target zone). -- Stuart Bishop <stuart(a)stuartbishop.net> http://www.stuartbishop.net/

1 0

Re: [Datetime-SIG] PEP-431/495
by Akira Li Aug. 25, 2015

Aug. 25, 2015

Stuart Bishop <stuart(a)stuartbishop.net> writes: > On 25 August 2015 at 01:02, Guido van Rossum <guido(a)python.org> wrote: > >> I'm still confused about what makes Stuart believe a tzinfo database must >> also change the arithmetic rules (especially since Gustavo's dateutils >> apparently gets along quite well without this). > > A tzinfo database does not care about the arithmetic rules. > > The database provided by pytz is designed in such a way that you get > correct timeline arithmetic out of Python's datetime library, by > exposing each period in the timezone definition as distinct fixed > offset tzinfo instances. > >>>> from datetime import datetime, timezone, timedelta >>>> from pytz.reference import Eastern as reftz # Tim's reference implementation >>>> import pytz >>>> dt = datetime(2004, 4, 4, 6, 59, tzinfo=timezone.utc) >>>> str(dt.astimezone(pytz.timezone('US/Eastern'))) > '2004-04-04 01:59:00-05:00' >>>> str(dt.astimezone(reftz)) > '2004-04-04 01:59:00-05:00' >>>> str(dt.astimezone(pytz.timezone('US/Eastern')) + timedelta(minutes=1)) > '2004-04-04 02:00:00-05:00' >>>> str(dt.astimezone(reftz) + timedelta(minutes=1)) > '2004-04-04 02:00:00-04:00' > > So I think pytz's current implementation is entwined with the > arithmetic style. But the tzfile loader in pytz could be reworked to > present tzinfo instances suitable for classic arithmetic if you want > it in stdlib, or the parser out of dateutils used. As far as I know utc -> local timezone *conversions* do not work during DST transitions in dateutil [1] while pytz manages just fine. Let's not do, whatever dateutil does here. It would be a regression. The result does not depend on the datetime implementation details. Given a specific tz db version the utc -> local conversion is not a matter of interpretation for recent dates. pytz is the only Python library that does timezone *conversions* correctly for almost all timezones from the tz database in a +/- decade time range. If pep-495 does not help pytz with timezone conversions then what is the point of pep-495? [1] https://github.com/dateutil/dateutil/issues/112

2 1