Mailman 3 September 2015 - Datetime-SIG

Re: [Datetime-SIG] IANA TZ database statistics
by Tim Peters Sept. 24, 2015

Sept. 24, 2015

[Alex] > ... > This is not about new tzinfos. This is about implementing PEP 495's > .astimezone(). Ah. You realize that's the first time that's been mentioned in this thread? It's been a total mystery until now ;-) > ... > This is not about wrapping IANA's tzdist. This is about implementing PEP > 495 features using POSIX APIs. Specifically which features? Do you just mean .astimezone() treating a naive datetime as being in the system zone, and the absence of any argument implying the system zone? Or more than just that? >> ... >> In that case, "best" is returning what the IANA database >> says should be returned in all cases. > Which version of IANA database? If it's still relevant, the only version any user cares about: the one that happens to be installed on their machine ;-) > ... > I don't want to try to figure out how to access tzfiles in a portable way. > We need another PEP for this because I don't see any better solution than to > repackage IANA files as a pip-installable package. Such PEP should probably > be discussed on distutils-sig first. Sorry, since this thread started by presenting statistics about the contents of the IANA database, I three-quarters assumed that _was_ what this was about. I agree that needs a whole different PEP. I also agree figuring out the system zone's rules is a puzzle using POSIX. Note that Gustavo gave up on trying to use mktime() in dateutil's tzlocal class. You could say time.timezone and time.altzone define the only two (or one, if time.daylight is 0) possible total UTC offsets, and assume that's always been, and always will be, the case. But I don't think even `altzone` is actually required by POSIX - it's of little help :-(

1 0

Re: [Datetime-SIG] IANA TZ database statistics
by Alexander Belopolsky Sept. 24, 2015

Sept. 24, 2015

> [Alex] > > My code ignores "version 2" data, so I include only transitions > > that fall within 32-bit time_t (1900 to 2038 range). > [Tim] > From staring at zic.c, .. I get a pounding headache. :-( > looks like (so far) the data in the version 2 > section is identical to that in the version 1 section, except written > out in wider data formats. The pretty clear intent is that they never > intend to generate explicit transitions beyond 2037 in any version, > I compiled the latest Github version on my Mac and I get $ /usr/local/etc/zdump -V 'America/New_York'| tail -4 America/New_York Sun Mar 8 06:59:59 2499 UT = Sun Mar 8 01:59:59 2499 EST isdst=0 gmtoff=-18000 America/New_York Sun Mar 8 07:00:00 2499 UT = Sun Mar 8 03:00:00 2499 EDT isdst=1 gmtoff=-14400 America/New_York Sun Nov 1 05:59:59 2499 UT = Sun Nov 1 01:59:59 2499 EDT isdst=1 gmtoff=-14400 America/New_York Sun Nov 1 06:00:00 2499 UT = Sun Nov 1 01:00:00 2499 EST isdst=0 gmtoff=-18000 > until it's after 2037 in the real world and they need to do so because > a POSIX TZ rule can't handle some new goofy exception (and version 2 > also contains a POSIX TZ rule at the end, when possible). What they do is a so-called 400-year hack: since the Gregorian calendar repeats itself every 400 years, any regular calendar-based rule will generate transitions with a 400-year period. This observation allows them to generate 400+ years of explicit transitions through 2499 and extent that through eternity by periodicity. > Then they'll need to add new transitions in the version 2 section only > (version 1 data formats are too narrow to record them). > > They already do that for transitions both before EPOCH - 2**31 seconds and after EPOCH + 2**31 seconds. $ /usr/local/etc/zdump -V 'America/New_York'| head -2 America/New_York Sun Nov 18 16:59:59 1883 UT = Sun Nov 18 12:03:57 1883 LMT isdst=0 gmtoff=-17762 America/New_York Sun Nov 18 17:00:00 1883 UT = Sun Nov 18 12:00:00 1883 EST isdst=0 gmtoff=-18000 [Alex] > I am considering making the cut-off at 1970 and assume 1970 standard > time for all times before that year. > [Tim] > Is there a real need for a "high performance" tzinfo? This is not about new tzinfos. This is about implementing PEP 495's .astimezone(). > That is, who cares? ;-) I do. :-) > It would sure be _surprising_ if a Python wrapping of > zoneinfo returned different results than native Linux tools wrapping > the same thing. > This is not about wrapping IANA's tzdist. This is about implementing PEP 495 features using POSIX APIs. > [Alex] > I think this is best we can do on Windows > [Tim] > Of course not, if by "best" we mean "gets the same answers everyone > else gets". In that case, "best" is returning what the IANA database > says should be returned in all cases. > Which version of IANA database? > [Alex] > where (IIRC) mktime does not work for times before epoch. (What about > > localtime?) > [Tim] > >>> time.localtime(-1) .. OSError: [Errno 22] Invalid argument > That's what I thought. > > Which is another meaning for "best": avoid flaky C library functions > altogether. > > >>> time.localtime(1e11) .. > OSError: [Errno 22] Invalid argument > I don't want to try to figure out how to access tzfiles in a portable way. We need another PEP for this because I don't see any better solution than to repackage IANA files as a pip-installable package. Such PEP should probably be discussed on distutils-sig first.

1 0

Re: [Datetime-SIG] IANA TZ database statistics
by Tim Peters Sept. 24, 2015

Sept. 24, 2015

[Alex] > ... > If we can make some simplified assumptions about transition locations and > sizes, we can avoid a binary search over seconds to locate the transitions > via POSIX localtime/mktime APIs. BTW, "the obvious" way to almost always avoid binary search is for a tzinfo to remember the index of the last transition it had to use, then next time start a linear search from there. It should usually succeed in 1 or 2 tries. Programs in real life don't jump around across all possible times at random. To be truly insane, it could meld linear search with binary search, like Python's listsort.c's "gallop" functions. I'd say that's far more trouble than it's worth in this context, though. Simple linear search with a search finger (index saved across searches) should do fine.

1 0

Re: [Datetime-SIG] IANA TZ database statistics
by Random832 Sept. 24, 2015

Sept. 24, 2015

On Wed, Sep 23, 2015, at 20:47, Tim Peters wrote: > [Alex] > > I added a method to datetimetester [1] to compute some overall statistics on > > tzfiles. My code ignores "version 2" data, so I include only transitions > > that fall within 32-bit time_t (1900 to 2038 range). > > From staring at zic.c, looks like (so far) the data in the version 2 > section is identical to that in the version 1 section, except written > out in wider data formats. The pretty clear intent is that they never > intend to generate explicit transitions beyond 2037 in any version, They do have transitions for before 1901, though. > > I think this is best we can do on Windows > > where (IIRC) mktime does not work for times before epoch. (What about > > localtime?) Windows has its own mechanism for storing timezone information, not tzdata, but no, none of the MSVCRT functions work for times before 1970.

1 0

Re: [Datetime-SIG] IANA TZ database statistics
by Tim Peters Sept. 24, 2015

Sept. 24, 2015

[Alex] > I added a method to datetimetester [1] to compute some overall statistics on > tzfiles. My code ignores "version 2" data, so I include only transitions > that fall within 32-bit time_t (1900 to 2038 range). >From staring at zic.c, looks like (so far) the data in the version 2 section is identical to that in the version 1 section, except written out in wider data formats. The pretty clear intent is that they never intend to generate explicit transitions beyond 2037 in any version, until it's after 2037 in the real world and they need to do so because a POSIX TZ rule can't handle some new goofy exception (and version 2 also contains a POSIX TZ rule at the end, when possible). Then they'll need to add new transitions in the version 2 section only (version 1 data formats are too narrow to record them). > I am considering making the cut-off at 1970 and assume 1970 standard > time for all times before that year. Is there a real need for a "high performance" tzinfo? That is, who cares? ;-) It would sure be _surprising_ if a Python wrapping of zoneinfo returned different results than native Linux tools wrapping the same thing. > I think this is best we can do on Windows Of course not, if by "best" we mean "gets the same answers everyone else gets". In that case, "best" is returning what the IANA database says should be returned in all cases. > where (IIRC) mktime does not work for times before epoch. (What about > localtime?) Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32 >>> import time >>> time.localtime(0) time.struct_time(tm_year=1969, tm_mon=12, tm_mday=31, tm_hour=18, tm_min=0, tm_sec=0, tm_wday=2, tm_yday=365, tm_isdst=0) >>> time.localtime(-1) Traceback (most recent call last): File "<pyshell#2>", line 1, in <module> time.localtime(-1) OSError: [Errno 22] Invalid argument Which is another meaning for "best": avoid flaky C library functions altogether. >>> epoch = datetime(1970, 1, 1) >>> epoch + timedelta(seconds=1e11) datetime.datetime(5138, 11, 16, 9, 46, 40) >>> import time >>> time.localtime(1e11) Traceback (most recent call last): File "<pyshell#13>", line 1, in <module> time.localtime(1e11) OSError: [Errno 22] Invalid argument

1 0

Re: [Datetime-SIG] IANA TZ database statistics
by Chris Angelico Sept. 23, 2015

Sept. 23, 2015

On Thu, Sep 24, 2015 at 8:44 AM, Alexander Belopolsky <alexander.belopolsky(a)gmail.com> wrote: > In any case, TZ data before 1970 is highly suspect so we will probably do > our users a favor by assuming standard time and letting those with > historical timeseries figure out the transitions by themselves. Yeah. Originally I made a boolean to suppress pre-1970 data, before settling on the arbitrary starting year option. I expect that 1970 will be the most common year to use as the base. FWIW: https://github.com/abalkin/cpython/pull/1 ChrisA

1 0

Re: [Datetime-SIG] IANA TZ database statistics
by Alexander Belopolsky Sept. 23, 2015

Sept. 23, 2015

On Wed, Sep 23, 2015 at 6:16 PM, Chris Angelico <rosuav(a)gmail.com> wrote: > A lot of the small numbers are going to be when different places > adopted standard time, and such. To get a better handle on what's > happening _now_, I added an option [1] to your stats function for a > starting year: > > >>> ZoneInfo.stats(start_year=1970) > Number of zones: 1790 = 46266 (gaps) + 46130 (folds) + 843 (zeros) > Min gap: 0:15:00 at 1985-12-31 18:30:13 in > right/Asia/Kathmandu > Max gap: 1 day, 0:00:00 at 2011-12-30 10:00:24 in > right/Pacific/Apia > Min fold: 0:30:00 at 2037-04-04 15:00:26 > in right/Australia/Lord_Howe > Max fold: 3:00:00 at 2012-02-21 17:00:24 > in right/Antarctica/Casey > >>> ZoneInfo.stats() > Number of zones: 1790 = 58914 (gaps) + 58777 (folds) + 1363 (zeros) > Min gap: 0:00:16 at 1935-01-01 03:40:52 in > posix/America/Paramaribo > Max gap: 1 day, 0:00:00 at 2011-12-30 10:00:24 in > right/Pacific/Apia > Min fold: 0:01:31 at 1932-01-01 03:58:29 in > posix/America/Barbados > Max fold: 10:00:00 at 1952-01-13 14:00:00 in > posix/Antarctica/DumontDUrville > It looks like you've got double counts because you included both "posix" and "right" tzfiles in the search. (I don't think the data that I actually read is different between the two sets.) > I'm not sure whether this actually helps anything or not, but hey, cool > stats :) > If we can make some simplified assumptions about transition locations and sizes, we can avoid a binary search over seconds to locate the transitions via POSIX localtime/mktime APIs. I am considering making the cut-off at 1970 and assume 1970 standard time for all times before that year. I think this is best we can do on Windows where (IIRC) mktime does not work for times before epoch. (What about localtime?) In any case, TZ data before 1970 is highly suspect so we will probably do our users a favor by assuming standard time and letting those with historical timeseries figure out the transitions by themselves.

1 0

Re: [Datetime-SIG] IANA TZ database statistics
by Chris Angelico Sept. 23, 2015

Sept. 23, 2015

On Thu, Sep 24, 2015 at 7:00 AM, Alexander Belopolsky <alexander.belopolsky(a)gmail.com> wrote: > Here are the results for the default Mac OSX system files and the most > recent Github version of tz [2]: > >>>> from datetimetester import * >>>> ZoneInfo.stats() > Number of zones: 584 > Number of transitions: 38510 = 19058 (gaps) + 19008 (folds) + 444 (zeros) > Min gap: 0:00:16 at 1935-01-01 03:40:52 in > America/Paramaribo > Max gap: 1 day, 0:00:00 at 2011-12-30 10:00:00 in Pacific/Apia > Min fold: 0:01:31 at 1932-01-01 03:58:29 in America/Barbados > Max fold: 10:00:00 at 1952-01-13 14:00:00 in > Antarctica/DumontDUrville >>>> ZoneInfo.zoneroot = '/usr/local/etc/zoneinfo' >>>> ZoneInfo.stats() > Number of zones: 585 > Number of transitions: 39018 = 19434 (gaps) + 19131 (folds) + 453 (zeros) > Min gap: 0:00:04 at 1914-01-01 04:00:04 in America/Manaus > Max gap: 1 day, 0:00:00 at 2011-12-30 11:00:00 in Pacific/Fakaofo > Min fold: 0:00:10 at 1906-06-30 16:53:20 in Asia/Ho_Chi_Minh > Max fold: 23:00:00 at 1969-09-30 13:00:00 in Kwajalein Neat! (Is that meant to be "from test.datetimetester import *", or was I loading this up the wrong way? Anyway, not significant.) A lot of the small numbers are going to be when different places adopted standard time, and such. To get a better handle on what's happening _now_, I added an option [1] to your stats function for a starting year: >>> ZoneInfo.stats(start_year=1970) Number of zones: 1790 = 46266 (gaps) + 46130 (folds) + 843 (zeros) Min gap: 0:15:00 at 1985-12-31 18:30:13 in right/Asia/Kathmandu Max gap: 1 day, 0:00:00 at 2011-12-30 10:00:24 in right/Pacific/Apia Min fold: 0:30:00 at 2037-04-04 15:00:26 in right/Australia/Lord_Howe Max fold: 3:00:00 at 2012-02-21 17:00:24 in right/Antarctica/Casey >>> ZoneInfo.stats() Number of zones: 1790 = 58914 (gaps) + 58777 (folds) + 1363 (zeros) Min gap: 0:00:16 at 1935-01-01 03:40:52 in posix/America/Paramaribo Max gap: 1 day, 0:00:00 at 2011-12-30 10:00:24 in right/Pacific/Apia Min fold: 0:01:31 at 1932-01-01 03:58:29 in posix/America/Barbados Max fold: 10:00:00 at 1952-01-13 14:00:00 in posix/Antarctica/DumontDUrville I'm not sure whether this actually helps anything or not, but hey, cool stats :) ChrisA [1] https://github.com/Rosuav/cpython/commit/ed51575f7ffe7ba98bfad58a43602cb8f7…

1 0

IANA TZ database statistics
by Alexander Belopolsky Sept. 23, 2015

Sept. 23, 2015

I added a method to datetimetester [1] to compute some overall statistics on tzfiles. My code ignores "version 2" data, so I include only transitions that fall within 32-bit time_t (1900 to 2038 range). Here are the results for the default Mac OSX system files and the most recent Github version of tz [2]: >>> from datetimetester import * >>> ZoneInfo.stats() Number of zones: 584 Number of transitions: 38510 = 19058 (gaps) + 19008 (folds) + 444 (zeros) Min gap: 0:00:16 at 1935-01-01 03:40:52 in America/Paramaribo Max gap: 1 day, 0:00:00 at 2011-12-30 10:00:00 in Pacific/Apia Min fold: 0:01:31 at 1932-01-01 03:58:29 in America/Barbados Max fold: 10:00:00 at 1952-01-13 14:00:00 in Antarctica/DumontDUrville >>> ZoneInfo.zoneroot = '/usr/local/etc/zoneinfo' >>> ZoneInfo.stats() Number of zones: 585 Number of transitions: 39018 = 19434 (gaps) + 19131 (folds) + 453 (zeros) Min gap: 0:00:04 at 1914-01-01 04:00:04 in America/Manaus Max gap: 1 day, 0:00:00 at 2011-12-30 11:00:00 in Pacific/Fakaofo Min fold: 0:00:10 at 1906-06-30 16:53:20 in Asia/Ho_Chi_Minh Max fold: 23:00:00 at 1969-09-30 13:00:00 in Kwajalein [1]: https://github.com/abalkin/cpython/commit/fa4f8055ac6723d4d0940ea141e05f931… [2]: https://github.com/eggert/tz

1 0

Re: [Datetime-SIG] Computing .dst() as a timedelta
by M.-A. Lemburg Sept. 23, 2015

Sept. 23, 2015

I think I only got part of my tongue-in-cheek suggestion across :-) The idea was to drop local time altogether and instead use UTC everywhere. Wall clocks would all show UTC. Instead of switching time zones, you'd adapt your schedule as needed and this could be as flexible as you want. People would just have to get used to having dinner at e.g. 03:00 UTC instead of 8pm [add some timezone here] and you would be able to enjoy sunset at 10:00 UTC in some places. Ain't going to happen, but it would allow people to gain back some more freedom in scheduling their lives. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 23 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ 2015-09-14: Released mxODBC Plone/Zope DA 2.2.3 http://egenix.com/go84 2015-09-26: Python Meeting Duesseldorf Sprint 2015 3 days to go 2015-10-21: Python Meeting Duesseldorf ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

2 1