Mailman 3 August 2015 - Datetime-SIG

Re: [Datetime-SIG] PEP-431/495
by Alexander Belopolsky Aug. 27, 2015

Aug. 27, 2015

On Thu, Aug 27, 2015 at 12:25 PM, Stuart Bishop <stuart(a)stuartbishop.net> wrote: > I think people are more aware of daylight savings time than you > imagine, given the nagging by the media on how to adjust their clocks. > I am afraid this is a very US-centric view. Try to explain the concept to someone from a country that does not mess with the clocks.

1 0

Re: [Datetime-SIG] PEP-431/495
by Stuart Bishop Aug. 27, 2015

Aug. 27, 2015

On 27 August 2015 at 22:00, Alexander Belopolsky <alexander.belopolsky(a)gmail.com> wrote: > I have no idea where you get tzlocal from, but I can assure you that there > is no such thing in the standard library. Consider tzlocal as part of pytz. It is Lennart's work, designed to work with pytz and would have been part of it except I felt it better to keep the scope narrowly defined. tzlocal.get_localzone() will load the correct dst aware local IANA zoneinfo file wherever it is defined on the platform, and I believe also gets you a working local dst aware tzinfo on Windows. -- Stuart Bishop <stuart(a)stuartbishop.net> http://www.stuartbishop.net/

1 0

Re: [Datetime-SIG] PEP-431/495
by Stuart Bishop Aug. 27, 2015

Aug. 27, 2015

On 27 August 2015 at 21:05, Alexander Belopolsky <alexander.belopolsky(a)gmail.com> wrote: > > On Thu, Aug 27, 2015 at 9:12 AM, Stuart Bishop <stuart(a)stuartbishop.net> > wrote: >> >> But, really, I hate all the options for the flag name. I lean towards >> is_dst mainly because people are used to it. > > Ironically, you created one of the reasons I did not want any mention of > "dst" in the new attribute: the way pytz's localize() uses is_dst is so > different from the way mktime() uses tm_isdst, that creating a third set of > rules for a variable with a similar name was out of the question. > > If you have time, please re-read the answer to "Why not call the new flag > 'isdst'?" [1] in the PEP and let me know if it needs any improvements. > > [1]: > https://www.python.org/dev/peps/pep-0495/#why-not-call-the-new-flag-isdst In my experience, the first conversation would actually read: Alice: Hey, lets watch the meteor shower at 1:30am Bob: Is that 1:30am daylight savings time? Alice: Hmm... yes, the paper says daylight savings time. I think people are more aware of daylight savings time than you imagine, given the nagging by the media on how to adjust their clocks. Most of the argument against isdst also applies to first/fold. I think you are more likely to know if you want dst or not than you are to know if you want the first or second period. Most of my input comes from other computer systems, and will give me an offset. This is more easily mapped to the is_dst flag than the fold flag. If you have the dst flag, then you need details about the timezone to calculate the fold flag. In all cases, if you chose not to or are unable to specify an explicit flag you are forced to guess and will get the incorrect answer 50% of the time. While most of the time you will just guess with either approach, I think you are more likely to get disambiguation information from a computer in the form of an offset or is_dst flag than you are to get disambiguation information from a human. I have billions of rows of data in my databases. If I'm dealing with input from computer systems, such as my databases, I will have billions of conversions to do and the overhead of calculating the fold flag is billions of times worse than calculating the isdst flag from the non-bulk human input. Yes, there is a subtle difference between how tm_isdst is handled and how an is_dst flag in the datetime module should be handled, in that tm_isdst will force the use of the dst timezone or not even if it would normally not be applied to that period. Personally, I find the behaviour close enough that the similar names would be a benefit. Yes, the dst() method returning the dst offset and the is_dst flag are have similar names, which is fine as they have similar concepts. The proposed type checking would ensure that any misinterpretation was quickly picked up, as attempting to pass timedelta(0) as the value of is_dst would raise an exception. So I think the argument against is_dst is weak. That said, I can't make a strong argument in favour of it either. I still lean in favour of is_dst myself, but I'm only leaning. -- Stuart Bishop <stuart(a)stuartbishop.net> http://www.stuartbishop.net/

1 0

Re: [Datetime-SIG] PEP-431/495
by Akira Li Aug. 27, 2015

Aug. 27, 2015

Stuart Bishop <stuart(a)stuartbishop.net> writes: ... > the complexities and slowdowns of timeline arithmetic. While not > changing the behaviour of datetime at all, we could get cats and dogs > living together by just clarifying what it actually is. An *observation*: the local timezone -- the only timezone with a variable utc offset in _stdlib -- behaves like pytz_: # start with the same utc time utc_time = datetime(2015, 10, 25, 1, tzinfo=timezone.utc) stdlib_time = utc_time.astimezone() # stdlib local time pytz_time = utc_time.astimezone(tzlocal.get_localzone()) dateutil_time = utc_time.astimezone(dateutil.tz.tzlocal()) All times are consistent so far. Perform the same operations: >>> stdlib_time - timedelta(seconds=1) datetime.datetime(2015, 10, 25, 1, 59, 59, tzinfo=datetime.timezone(datetime.timedelta(0, 3600), 'CET')) >>> pytz_time - timedelta(seconds=1) datetime.datetime(2015, 10, 25, 1, 59, 59, tzinfo=<DstTzInfo 'Europe/Paris' CET+1:00:00 STD>) >>> dateutil_time - timedelta(seconds=1) datetime.datetime(2015, 10, 25, 1, 59, 59, tzinfo=tzlocal()) Get different results. The times are the same but utc offset are different: >>> (stdlib_time - timedelta(seconds=1)).utcoffset() datetime.timedelta(0, 3600) >>> (pytz_time - timedelta(seconds=1)).utcoffset() datetime.timedelta(0, 3600) >>> (dateutil_time - timedelta(seconds=1)).utcoffset() #XXX different datetime.timedelta(0, 7200) It is expected for arithmetic in the presense of DST transitions. Here's the standard fix: >>> (stdlib_time - timedelta(seconds=1)).astimezone() datetime.datetime(2015, 10, 25, 2, 59, 59, tzinfo=datetime.timezone(datetime.timedelta(0, 7200), 'CEST')) >>> (pytz_time - timedelta(seconds=1)).astimezone(tzlocal.get_localzone()) datetime.datetime(2015, 10, 25, 2, 59, 59, tzinfo=<DstTzInfo 'Europe/Paris' CEST+2:00:00 DST>) >>> (dateutil_time - timedelta(seconds=1)).astimezone(dateutil.tz.tzlocal()) #XXX different datetime.datetime(2015, 10, 25, 1, 59, 59, tzinfo=tzlocal()) >>> datetime(2015, 10, 25, 1, 59, 59, tzinfo=dateutil.tz.tzlocal()).strftime('%Z%z') 'CEST+0200' Now the all timezones are the same but the times are different. pytz recommends normalize() method instead of astimezone() here: >>> tzlocal.get_localzone().normalize(pytz_time - timedelta(seconds=1)) datetime.datetime(2015, 10, 25, 2, 59, 59, tzinfo=<DstTzInfo 'Europe/Paris' CEST+2:00:00 DST>) The result is the same in this case. It is a _fact_. It is how python behaves now on my platform. To try it yourself, add this at the top: import os import time from datetime import datetime, timezone, timedelta os.environ['TZ'] = 'Europe/Paris' time.tzset() import dateutil.tz import tzlocal # get local timezone as pytz tzinfo

2 2

Re: [Datetime-SIG] PEP-431/495
by Alexander Belopolsky Aug. 27, 2015

Aug. 27, 2015

On Thu, Aug 27, 2015 at 9:12 AM, Stuart Bishop <stuart(a)stuartbishop.net> wrote: > > But, really, I hate all the options for the flag name. I lean towards > is_dst mainly because people are used to it. Ironically, you created one of the reasons I did not want any mention of "dst" in the new attribute: the way pytz's localize() uses is_dst is so different from the way mktime() uses tm_isdst, that creating a third set of rules for a variable with a similar name was out of the question. If you have time, please re-read the answer to "Why not call the new flag 'isdst'?" [1] in the PEP and let me know if it needs any improvements. [1]: https://www.python.org/dev/peps/pep-0495/#why-not-call-the-new-flag-isdst

1 0

Re: [Datetime-SIG] PEP-495 - Strict Invalid Time Checking
by Ethan Furman Aug. 27, 2015

Aug. 27, 2015

On 08/25/2015 02:22 PM, Alexander Belopolsky wrote: > On Tue, Aug 25, 2015 at 5:04 PM, Carl Meyer wrote: >> >> I can't imagine how raising an exception on invalid times only if a >> non-default sentinel value is given for a flag that is _new in PEP 495_ >> could possibly break 4000 lines of existing datetime tests. > > OK, so datetime module itself will never set fold=-1. In the list below, can you mark the methods that need to be patched to check the fold attribute in your preferred design: If `fold=None` is present /when attempting to create a datetime/ that is ambiguous or invalid, an exception is raised /at that moment/ meaning that a datetime with `fold=None` /will never exist/. `fold=None` /will not be the default/. > datetime.__add__ no > datetime.__eq__ no > datetime.__format__ no > datetime.__ge__ no > datetime.__gt__ no > datetime.__hash__ no > datetime.__le__ no > datetime.__lt__ no > datetime.__ne__ no > datetime.__new__ YES > datetime.__radd__ no > datetime.__reduce__ no > datetime.__reduce_ex__ no > datetime.__repr__ no > datetime.__rsub__ no > datetime.__str__ no > datetime.astimezone no > datetime.combine YES (could have a time instance with fold=None) > datetime.ctime no > datetime.date no > datetime.dst no > datetime.fromordinal no > datetime.fromtimestamp no > datetime.isocalendar no > datetime.isoformat no > datetime.isoweekday no > datetime.now no > datetime.replace YES > datetime.strftime no > datetime.strptime only if `fold=` is allowed > datetime.time no > datetime.timestamp no > datetime.timetuple no > datetime.timetz no > datetime.today no > datetime.toordinal no > datetime.tzname no > datetime.utcfromtimestamp no > datetime.utcnow no > datetime.utcoffset no > datetime.utctimetuple no > datetime.weekday no -- ~Ethan~

2 1

Re: [Datetime-SIG] PEP-431/495
by Stuart Bishop Aug. 27, 2015

Aug. 27, 2015

On 26 August 2015 at 15:16, Tim Peters <tim.peters(a)gmail.com> wrote: >> In this case, I'm not fussed if the datetime instance has a 2 state or >> 3 state flag. This is different to the various constructors which I >> think need a 3 state flag in their arguments. True, False, None. > > As things seem to have progressed later, mapping pytz's explicit time > checking into a more magical scheme sprayed all over the datetime > internals is not straightforward, So, as I concluded elsewhere, that > may or may not be done someday, but it's out of scope for PEP 495. > I'm a fan of making progress ("now is better than never", where the > latter was PEP 431's fate waiting for perfection on all counts). Yup. >> Classic behaviour as you describe it is a bug. > > Believe me, you won't get anywhere with that approach. > > - Classic arithmetic is the only kind that makes good sense in the > "naive time" model, which _is_ datetime's model. [... sensible stuff trimmed ...] > That said, I would have preferred it if Python's datetime had used > classic arithmetic only for naive datetimes. I feared it might be > endlessly confusing if an aware datetime used classic arithmetic too. > I'm not sure about "endlessly" now, but it has come up more than once > ;-) Far too late to change now, though. I'm wondering if it is worth formalizing this (post-PEP-495,or maybe some choice wording changes made in the docs). Would it work if we introduced a new type, datetimetz? We would have a time, with a tzinfo because it might be useful later, a naive time, with a tzinfo because it is useful for rendering and conversions, and a datetimetz with all the complexities and slowdowns of timeline arithmetic. While not changing the behaviour of datetime at all, we could get cats and dogs living together by just clarifying what it actually is. >> If you use pytz tzinfo instances, adding 1 second always adds one second > > But only in the _model_ you have in mind: real-life clocks showing > real-life civil time suffer from leap seconds too. You can laugh that > off in _your_ apps (and I can too ;-) ), but for other apps it's dead > serious. If our underlying platforms that we needed to work with supported it, I'd probably be in favour of leap seconds. I doubt that would ever happen - there are more palatable workarounds. >> and adding 1 day always adds 24 hours. > > That's also true of classic arithmetic. The meanings of "day" and "24 > hours" also depend on the model in use. I think in my view, as soon as you go to the bother of adding a tzinfo instance to the datetime you are making a statement about the expected behaviour; that the simpler classic arithmetic no longer applies and the more complex model needs to be used. > def dt_add(dt, td): > return dt.tzinfo.fromutc(dt + (td - dt.utcoffset())) > > There you go: "timeline" datetime + timedelta arithmetic about as > efficiently as possible in pure Python. Note that _if_ the default > changed to timeline arithmetic, this code would no longer work. The > "+" there requires classic arithmetic to get the right result. Change > the default, this code would break too. I find it hard to imagine I'm > the only person in the world who has code similarly taking advantage > of what Python actually does. > > Example: I see. What I don't like about this approach is the developers need to be aware that they need to call it, and that dt + timedelta(hours=24) may not work. Of course, developers will not be aware or have done more than skim the docs until after their guests have all died of salmonella poisoning from the undercooked Turkey. Its one of the reasons I'm wondering if something more in your face like the datetimetz proposal above would be an improvement. Stop making me hungry dammit. >> However... this also means the new flag on the datetime instances is >> largely irrelevant to pytz. pytz' API will need to remain the same. > > My hope was that 495 alone would at least spare pytz's users from > needing to do a `.normalize()` dance after `.astimezone()` anymore. > Although I'm not clear on why it's needed even now. Instead of one tzinfo instance, there are dozens for your timezone. The datetime implementation does not give pytz the opportunity to choose which one is used when constructing the datetime, so localize is needed to sort that. Similarly, arithmetic does not always give pytz the opportunity to choose which one is used after crossing a timezone boundary, so normalize is needed to sort that out. While the results of the timeline arithmetic are unambiguous and obvious, they are arguably incorrect until normalize puts things right. > See the "PEP-495 - Strict Invalid Time Checking" thread for more. > There seems to be increasing "feature creep" here. Rewriting vast > swaths of datetime internals to cater to this is at best impractical, > especially compared to supplying a "check this datetime" function > users who care can call when they care. Nevertheless, it's a suitable > subject for a different PEP. I don't want to bog 495 down with it. > If it had _stopped_ with asking for an optional check in the datetime > constructor, it may have been implemented already ;-) Yup. I think I'm after hooks to replace localize on construction and normalize after arithmetic, so users don't have to be relied on to do this explicitly. This doesn't need to happen now, and I fully understand this could be considered fast path and the overhead unacceptable. >> The important bit here for pytz is that tzinfo.fromutc() may return a >> datetime with a different tzinfo instance. > > Sorry, didn't follow that. Of course you can write your .fromutc() to > return anything you want. I can't find what concerned me any more. I think there was some wording along the lines of 'the result will be used to initialize the first flag'. What I'm reading now on fromutc() though looks fine, so I think I was mixed up. >>>> - My argument in favour of 'is_dst' over 'first' is that this is what >>>> we have in the data we are trying to load. You commonly have >>>> a timestamp with a timezone abbreviation and/or offset. This can >>>> easily be converted to an is_dst flag. > >>> You mean by using platform C library functions (albeit perhaps wrapped >>> by Python)? > > I really missed an answer to that ;-) I think all the data we have access to, including from platform C library functions, uses the is_dst flag or is simpler to map to the is_dst flag. The C library as exposed by the time.struct_time gives you is_dst. Mapping that to first/fold means first doing doing two conversions and determining which one comes first. Similarly, when loading your JSON file or examining email headers you need to load in a string like '2004-04-04 02:30:00 EDT-05:00'. Its simple to use a lookup table to map the abbreviation + offset to an is_dst flag. Its harder to map it to first/fold because they are swapped around in April and October. And there can be more than two transitions in a year, so if you need to support that your going to need to do the lookup, construct a couple of instances, and compare to work out if EDT or EST comes first that month in that year. But, really, I hate all the options for the flag name. I lean towards is_dst mainly because people are used to it. > But there's every reason to be optimistic: even someone as old and > in-the-way as me doesn't find any of this particularly confusing ;-) I may be old, but at least I'm not as old as Tim ;) -- Stuart Bishop <stuart(a)stuartbishop.net> http://www.stuartbishop.net/

1 0

Re: [Datetime-SIG] PEP-431/495
by Stuart Bishop Aug. 27, 2015

Aug. 27, 2015

On 27 August 2015 at 08:37, Tim Peters <tim.peters(a)gmail.com> wrote: > I don't know whether Stuart mucked with arithmetic because he believed > that was necessary in order to get conversions to work correctly (if > so, he was mistaken), or whether the effects on arithmetic were just a > _consequence_ of using fixed-offset classes all the time (that's "a > natural" outcome of using only fixed-offset classes - it would take > extra effort to _stop_ it - classic and timeline arithmetic are the > same thing in any eternally-fixed-offset timezone) . He said, in an > earlier message, that conversion was his primary concern. But maybe > we're all using the same words with different meanings. The fixed-offset classes and sorting arithmetic were the only way to get things round tripping. Take a datetime. Convert it to another timezone. Add one hour to both. Compare. The results were inconsistent. You would only get correct results with fixed offset timezones, because the builtin arithmetic ignored the timezone, because there was no is_dst flag and without it it is impossible to get correct results. The burden was left on tzinfo implementations to deal with the problem. You could have naive times and do arithmetic correctly, or you could have zone aware times and do conversions correctly, but to do both developers had to always convert to and from utc to do the arithmetic. And developers being lazy creatures wouldn't bother because it would normally work, or even always work in their particular timezone, and systems would crash at 4am killing innocent kittens. And this was a problem with my tzinfo implementation, because the only way you could possibly experience the problem was by using my tzinfo implementation. Python had avoided this clearly documented problem by not supplying any tzinfo implementations, even though it would have been easy to create a 'local' one using the information already exposed in the time module, and I'd always assumed that fixing it was a requirement of adding timezone implementations to the standard library. So I fixed it. Drunk on my own cleverness and relative youth, it never occurred to me that it was possible to rationalize the existing behaviour with a straight face, where after going to all the effort of constructing and adding a tzinfo to your datetime it would sit there entirely ignored by Python, except for conversion operations, consistently giving you answers that are demonstrably incorrect using most modern timekeeping systems. I'm still not capable of conjuring up such a monumental rationalization ;) >>> My hope was that 495 alone would at least spare pytz's users from >>> needing to do a `.normalize()` dance after `.astimezone()` anymore. >>> Although I'm not clear on why it's needed even now. > >> As far as I know, normalize() is not necessary after astimezone() even >> now >> https://answers.launchpad.net/pytz/+question/249229 > > That agrees with my best guess, but my knowledge of pytz is shallow. > If it's correct that the .normalize() dance isn't needed here, it > would be nice if Stuart plainly said so on that page, and - of course > - changed the docs to stop saying it _is_ required. And then it's > also the case that I don't see any benefit to pytz from PEP 495 alone. > :-( Yeah, I'm putting off answering that one because I'm not sure if I'll get the answer right. People sometimes think I actually know what I'm doing. I'll have a look after I get the overdue pytz release out. >> replace() is just a shortcut for a constructor. > > Yet pytz does nothing to check .replace() results, right? It has no opportunity to check the results at the moment, nor does it have the opportunity to swap in the correct fixed offset tzinfo. It just gets rudely shoved into a datetime instance without consent. So replace gives you what it gives you, and you have to sort it out with a call to normalize, at which point you realize you have a timezone definition from 1910 in play and learn to always construct your localized datetime instances with tzinfo.localize(). Such a lovely user experience. -- Stuart Bishop <stuart(a)stuartbishop.net> http://www.stuartbishop.net/

1 0

Re: [Datetime-SIG] DST explained visually
by Akira Li Aug. 27, 2015

Aug. 27, 2015

Tim Peters <tim.peters(a)gmail.com> writes: ... > So we're switching to a boolean flag again, named solidbefore? The > great advantage to that would be easy of finding via Google. While > there a hundred times more hits on "python which" than on "python > fold", there are still over 800,000 hits on the latter. There only 7 > on > > python "solidbefore" > > although you need the quotes to cut it back from some thousands ;-) The first hit on google for *python datetime fold* is PEP 495.

1 0

Re: [Datetime-SIG] PEP-431/495
by Alexander Belopolsky Aug. 27, 2015

Aug. 27, 2015

[Akira Li] > Even if datetime's or replace()'s *first* parameter would be 3-state > None|True|False; the internal flag can still be 2-state True|False [Tim Peters] > So please take this to the following thread instead (but > after reading all of it first ;-) ): > > PEP-495 - Strict Invalid Time Checking Actually a follow-up thread "Strict Invalid Time Checking: an idea for another PEP" may be a better (and shorter!) fit.

1 0