use gmtime(0) epoch in functions that use mktime()
Python uses "seconds since the epoch" term to describe time.time() return value. POSIX term is "seconds since the Epoch" (notice the capitalization) where Epoch is 1970-01-01 00:00:00+00:00. C99 term is "calendar time" -- the encoding of the calendar time returned by the time() function is unspecified. Python documentation defines `epoch` as: The :dfn:`epoch` is the point where the time starts. On January 1st of that year, at 0 hours, the "time since the epoch" is zero. For Unix, the epoch is 1970. To find out what the epoch is, look at ``gmtime(0)``. time module documentation specifies calendar.timegm() as the inverse of time.gmtime() while timegm() uses the fixed 1970 Epoch instead of gmtime(0) epoch. datetime.astimezone() (local_timezone()) passes Unix timestamp [1970] to time.localtime() that may expect timestamp with different epoch [gmtime(0)]. email.util.mktime_tz() uses both mktime() [gmtime(0)] and timegm() [1970]. mailbox.py uses both time.time() [gmtime(0)] and timegm() [1970]. http.cookiejar uses both EPOCH_YEAR=1970 and datetime.utcfromtimestamp() [gmtime(0) epoch] for "seconds since epoch". It seems 1970 Epoch is used for file times on Windows (os.stat()) but os.path.getatime() refers to "seconds since epoch" [gmtime(0) epoch]. date{,time}.{,utc}fromtimestamp(), datetime.timestamp() docs equates "POSIX timestamp" [1970 Epoch] and time.time()'s returned value [gmtime(0) epoch]. datetime.timestamp() is inconsistent if gmtime(0) is not 1970. It uses mktime() for naive datetime objects [gmtime(0) epoch]. But it uses POSIX Epoch for aware datetime objects. Correct me if I'm wrong here. --- Possible fixes: Say in the `epoch` definition that stdlib doesn't support gmtime(0).tm_year != 1970. OR don't use mktime() if 1970 Epoch is used e.g., create an aware datetime object in the local timezone instead and use it to compute the result with 1970 Epoch. OR add *analog* of TZ=UTC time.mktime() and use it in stdlib where necessary. Looking at previous attempts (e.g., [1], [2]) to implement timegm(), the problem seems over-constrained. A different name could be used, to avoid wrong expectations e.g., datetime could use `(aware_datetime_object - gmtime0_epoch) // sec` [1] http://bugs.python.org/issue6280, [2] http://bugs.python.org/issue1667546 OR set EPOCH_YEAR=gmtime(0).tm_year instead of 1970 in calendar.timegm(). It may break backward compatibility if there is a system with non-1970 epoch. Deal on a case-by-case basis with other places where 1970 Epoch is used. And drop "POSIX timestamp" [1970 Epoch] and use "seconds since the epoch" [gmtime(0) epoch] in the datetime documentation. Change internal EPOCH year accordingly. What is Python-ideas opinion about it? -- Akira
There used to be systems with a different notion of epoch. Are there still such systems around? OSX has the UNIX epoch -- what's it gmtime(0) on Windows? On Sat, Sep 6, 2014 at 6:53 AM, Akira Li <4kir4.1i@gmail.com> wrote:
Python uses "seconds since the epoch" term to describe time.time() return value. POSIX term is "seconds since the Epoch" (notice the capitalization) where Epoch is 1970-01-01 00:00:00+00:00. C99 term is "calendar time" -- the encoding of the calendar time returned by the time() function is unspecified.
Python documentation defines `epoch` as:
The :dfn:`epoch` is the point where the time starts. On January 1st of that year, at 0 hours, the "time since the epoch" is zero. For Unix, the epoch is 1970. To find out what the epoch is, look at ``gmtime(0)``.
time module documentation specifies calendar.timegm() as the inverse of time.gmtime() while timegm() uses the fixed 1970 Epoch instead of gmtime(0) epoch.
datetime.astimezone() (local_timezone()) passes Unix timestamp [1970] to time.localtime() that may expect timestamp with different epoch [gmtime(0)].
email.util.mktime_tz() uses both mktime() [gmtime(0)] and timegm() [1970].
mailbox.py uses both time.time() [gmtime(0)] and timegm() [1970].
http.cookiejar uses both EPOCH_YEAR=1970 and datetime.utcfromtimestamp() [gmtime(0) epoch] for "seconds since epoch".
It seems 1970 Epoch is used for file times on Windows (os.stat()) but os.path.getatime() refers to "seconds since epoch" [gmtime(0) epoch].
date{,time}.{,utc}fromtimestamp(), datetime.timestamp() docs equates "POSIX timestamp" [1970 Epoch] and time.time()'s returned value [gmtime(0) epoch].
datetime.timestamp() is inconsistent if gmtime(0) is not 1970. It uses mktime() for naive datetime objects [gmtime(0) epoch]. But it uses POSIX Epoch for aware datetime objects.
Correct me if I'm wrong here.
--- Possible fixes:
Say in the `epoch` definition that stdlib doesn't support gmtime(0).tm_year != 1970.
OR don't use mktime() if 1970 Epoch is used e.g., create an aware datetime object in the local timezone instead and use it to compute the result with 1970 Epoch.
OR add *analog* of TZ=UTC time.mktime() and use it in stdlib where necessary. Looking at previous attempts (e.g., [1], [2]) to implement timegm(), the problem seems over-constrained. A different name could be used, to avoid wrong expectations e.g., datetime could use `(aware_datetime_object - gmtime0_epoch) // sec`
[1] http://bugs.python.org/issue6280, [2] http://bugs.python.org/issue1667546
OR set EPOCH_YEAR=gmtime(0).tm_year instead of 1970 in calendar.timegm(). It may break backward compatibility if there is a system with non-1970 epoch. Deal on a case-by-case basis with other places where 1970 Epoch is used. And drop "POSIX timestamp" [1970 Epoch] and use "seconds since the epoch" [gmtime(0) epoch] in the datetime documentation. Change internal EPOCH year accordingly.
What is Python-ideas opinion about it?
-- Akira _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
On 06/09/2014 18:06, Guido van Rossum wrote:
There used to be systems with a different notion of epoch. Are there still such systems around? OSX has the UNIX epoch -- what's it gmtime(0) on Windows?
time.gmtime(0) time.struct_time(tm_year=1970, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=3, tm_yday=1, tm_isdst=0)
-- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence
On 2014-09-06 18:06, Guido van Rossum wrote:
There used to be systems with a different notion of epoch. Are there still such systems around? OSX has the UNIX epoch -- what's it gmtime(0) on Windows?
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64 bit (AM D64)] on win32 Type "help", "copyright", "credits" or "license" for more information.
import time time.gmtime(0) time.struct_time(tm_year=1970, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=0, tm_sec= 0, tm_wday=3, tm_yday=1, tm_isdst=0)
[snip]
On Sat, Sep 6, 2014, at 13:06, Guido van Rossum wrote:
There used to be systems with a different notion of epoch. Are there still such systems around? OSX has the UNIX epoch -- what's it gmtime(0) on Windows?
If you call the time.h functions from the CRT library, they use 1970 (and always have). Windows has _other_ functions that use a different epoch (1600, if I remember correctly), but they're native win32 functions not called directly by python. The wrinkle you get on windows is that most of the functions don't work with *negative* time_t values (or struct tm values representing dates before 1970), and/or some other arbitrary cutoff dates. In particular, time.localtime gives an OSError on negative values, but time.gmtime gives an OSError on values below -43200, and both give errors if passed a value representing a year above 2999, and strftime does not accept years above 9999. But, as I've advocated before, there's no fundamental reason that python should chain itself to the C library's underlying implementation rather than defining its own time functions that do always use an epoch of 1970, and handle leap seconds consistently, have unlimited range, etc.
On Sat, Sep 6, 2014 at 2:56 PM, <random832@fastmail.us> wrote:
On Sat, Sep 6, 2014, at 13:06, Guido van Rossum wrote:
There used to be systems with a different notion of epoch. Are there still such systems around? OSX has the UNIX epoch -- what's it gmtime(0) on Windows?
If you call the time.h functions from the CRT library, they use 1970 (and always have). Windows has _other_ functions that use a different epoch (1600, if I remember correctly), but they're native win32 functions not called directly by python.
The wrinkle you get on windows is that most of the functions don't work with *negative* time_t values (or struct tm values representing dates before 1970), and/or some other arbitrary cutoff dates. In particular, time.localtime gives an OSError on negative values, but time.gmtime gives an OSError on values below -43200, and both give errors if passed a value representing a year above 2999, and strftime does not accept years above 9999.
But, as I've advocated before, there's no fundamental reason that python should chain itself to the C library's underlying implementation rather than defining its own time functions that do always use an epoch of 1970, and handle leap seconds consistently, have unlimited range, etc.
I'm fine with that, as long as "handle leap seconds consistently" means "pretend they don't exist" (which is necessary for compatibility with POSIX). But it sounds like a big coding project. Fixing the docs to be consistent and correctly describe the current implementation may be simpler. -- --Guido van Rossum (python.org/~guido)
On 09/06/2014 03:19 PM, Guido van Rossum wrote:
On Sat, Sep 6, 2014 at 2:56 PM, random832 wrote:
But, as I've advocated before, there's no fundamental reason that python should chain itself to the C library's underlying implementation rather than defining its own time functions that do always use an epoch of 1970, and handle leap seconds consistently, have unlimited range, etc.
I'm fine with that, as long as "handle leap seconds consistently" means "pretend they don't exist" (which is necessary for compatibility with POSIX). But it sounds like a big coding project. Fixing the docs to be consistent and correctly describe the current implementation may be simpler.
Or at least fix the docs until the new project is done, and then we can fix them again. :) -- ~Ethan~
Guido van Rossum <guido@python.org> writes:
On Sat, Sep 6, 2014 at 2:56 PM, <random832@fastmail.us> wrote:
On Sat, Sep 6, 2014, at 13:06, Guido van Rossum wrote:
There used to be systems with a different notion of epoch. Are there still such systems around? OSX has the UNIX epoch -- what's it gmtime(0) on Windows?
If you call the time.h functions from the CRT library, they use 1970 (and always have). Windows has _other_ functions that use a different epoch (1600, if I remember correctly), but they're native win32 functions not called directly by python.
The wrinkle you get on windows is that most of the functions don't work with *negative* time_t values (or struct tm values representing dates before 1970), and/or some other arbitrary cutoff dates. In particular, time.localtime gives an OSError on negative values, but time.gmtime gives an OSError on values below -43200, and both give errors if passed a value representing a year above 2999, and strftime does not accept years above 9999.
But, as I've advocated before, there's no fundamental reason that python should chain itself to the C library's underlying implementation rather than defining its own time functions that do always use an epoch of 1970, and handle leap seconds consistently, have unlimited range, etc.
I'm fine with that, as long as "handle leap seconds consistently" means "pretend they don't exist" (which is necessary for compatibility with POSIX). But it sounds like a big coding project. Fixing the docs to be consistent and correctly describe the current implementation may be simpler.
datetime doesn't support leap seconds. That part is already implemented ;) I've submitted a documentation patch that explicitly mentions the assumption about 1970 epoch -- https://bugs.python.org/issue22356 Does it make sense to submit other patches that only matter if the epoch is not 1970? --- Unrelated: it is possible to convert between POSIX time and UTC without knowing leap seconds: posix_time = (utc_time - datetime(1970, 1, 1)) // timedelta(seconds=1) It is the exact relation. TAI is one bisect() call away from UTC if the list of leap seconds (such as provided by the tz database (used in pep 431)) is known. It is available on all modern OSes (including Windows [1]). [1] http://www.iana.org/time-zones/repository/tz-link.html -- Akira
On Sat, Sep 6, 2014, at 18:19, Guido van Rossum wrote:
I'm fine with that, as long as "handle leap seconds consistently" means "pretend they don't exist" (which is necessary for compatibility with POSIX).
Consistently pretend they don't exist. AFAIK you're more likely to encounter a system using the so-called "right" timezones in tzdata (and therefore _not_ pretending that leap seconds don't exist) than one which doesn't have an epoch of 1970. In which case you would need to use "time2posix" and "posix2time" when calling any platform-specific functions that use time_t (with the user-visible python side, of course, being the non-leap-second posix timestamps) I did an inventory of names in the time module, by whether they can be implemented platform-independently or not: Depends on system-dependent ways of getting the current time: clock clock_getres clock_gettime clock_settime get_clock_info monotonic perf_counter process_time time Various default values (gmtime etc), can be implemented in terms of time Depends on system-dependent ways of getting the local timezone: ctime, can be implemented in terms of localtime localtime mktime tzset, can be used to set constants: timezone tzname altzone daylight Otherwise system-dependent: sleep Can be implemented in a platform-independent or pure python way: asctime gmtime calendar.timegm strftime (except %z %Z and locale) strptime (except %Z and locale) struct_time The list of theoretically platform-independent functions is, as it turns out, depressingly small. It might also be worthwhile to make a windows-specific implementation of some of the platform-dependent functions, rather than one relying on the C library (for example, localtime only has a range of 1970 to 2199, whereas SystemTimeToTzSpecificLocalTime has a range of 1601 to 30828.) But it would have the issue of not having the C library's somewhat obscure support of part of the POSIX TZ standard. (However, a full implementation of POSIX TZ could be done in a platform-independent way).
random832@fastmail.us writes:
On Sat, Sep 6, 2014, at 13:06, Guido van Rossum wrote:
There used to be systems with a different notion of epoch. Are there still such systems around? OSX has the UNIX epoch -- what's it gmtime(0) on Windows?
If you call the time.h functions from the CRT library, they use 1970 (and always have). Windows has _other_ functions that use a different epoch (1600, if I remember correctly), but they're native win32 functions not called directly by python.
python does call these functions, look at the code that uses FILETIME type. For example, os.utime() uses the hardcoded 1970 Epoch on Windows and converts it to FILETIME (1601 epoch in 100s of nanoseconds) to call SetFileTime(). In this case, using 1970 Epoch is justified. gmtime(0) is documented as "midnight (00:00:00), January 1, 1970, coordinated universal time (UTC)" on Windows. -- Akira
participants (6)
-
Akira Li -
Ethan Furman -
Guido van Rossum -
Mark Lawrence -
MRAB -
random832＠fastmail.us