Hi!
I've been working a lot with date/time variables lately, and maybe it's just me, and maybe I just don't get it yet, but it sure doesn't feel like there's one-and-only-one elegant, obvious way to work with them. Feels like working with strings in C.
So I was thinking, have date and time literals ever been seriously discussed anywhere in these discussion lists? (If so, I apologize in advance for reviving this subject.)
Is there any chance we could ever see date/time literals in Python? Perhaps as a top-level abstraction itself, or as a subclass of numbers.Number, or even as common float or int (by adding attributes regarding specific properties of a number's date/time interpretation, e.g. "<2010-12-31>.incr_month(2) == <2011-02-28>" -- just an example though, i don't really think this exact method would be practical).
If we adopt a standard like ISO 8601, then we automatically get an unambiguous one-to-one relationship between (date+time) and (amount of time elapsed from <epoch>), over a continuous, infinite domain (towards +inf and -inf). Very often the "time elapsed" value is what we really want (or at least the absolute difference between two values, which is of course easier to find in this form), and besides being more compact and flexible, no functionality is lost.
I guess the most controversial point here would be the meaning of arithmetic operations between dates (or the very validity of such operations).
Another (not so obvious) problem is leap seconds (btw, who the hell invented that?!..). It's a weird and complicated problem, but I think it's not impossible to turn this "complicated" into just "complex", keeping the tricky concepts buried inside the implementation.
Anyway, here's a couple of suggestions for a syntax:
2010.05.29 + 20.06.17 2010.05.29d + 20.06.17t
(just like complex literals)
So, what do you guys think? I'd love to hear others' opinions, even if it's just for me to understand what I got wrong and see the obvious way that's already there :)
PS: Just for fun, is there a standard way I could experiment with creating new types of literals?
-- Marcos --
On Sat, May 29, 2010 at 6:05 PM, Marcos Bonci marcos.bonci@gmail.com wrote:
Hi! I've been working a lot with date/time variables lately, and maybe it's just me, and maybe I just don't get it yet, but it sure doesn't feel like there's one-and-only-one elegant, obvious way to work with them. Feels like working with strings in C.
Reading the rest of your post, it sounds like you're unaware of the fairly nice API of the `datetime` std lib module: http://docs.python.org/library/datetime.html
IMO, it's pretty nice to work with; although it is annoying that some operations can only be done using the more primitive `time` or `calendar` modules. Patches are probably welcome; perhaps I should consider trying to write one.
So I was thinking, have date and time literals ever been seriously discussed anywhere in these discussion lists? (If so, I apologize in advance for reviving this subject.) Is there any chance we could ever see date/time literals in Python?
Well, the datetime module does appear to already be written in C, so that'd be one less hurdle. I wasn't able to find a prior post on the exact subject; perhaps someone else will.
Perhaps as a top-level abstraction itself, or as a subclass of numbers.Number, or even as common float or int (by adding attributes regarding specific properties of a number's date/time interpretation, e.g. "<2010-12-31>.incr_month(2) == <2011-02-28>" -- just an example though, i don't really think this exact method would be practical).
The `datetime` module already provides nice and more typesafe abstractions; purposefully conflating date/times with plain numbers is not a good idea, imo.
<snip>
Very often the "time elapsed" value is what we really want (or at least the absolute difference between two values, which is of course easier to find in this form), and besides being more compact and flexible, no functionality is lost.
datetime.timedelta: http://docs.python.org/library/datetime.html#timedelta-objects
I guess the most controversial point here would be the meaning of arithmetic operations between dates (or the very validity of such operations).
Already defined and accepted as valid: datetime.date, "Supported operations" http://docs.python.org/library/datetime.html#date-objects
<snip>
So, what do you guys think? I'd love to hear others' opinions, even if it's just for me to understand what I got wrong and see the obvious way that's already there
I think the classes in the `datetime` module are the nice "missing" API that you're looking for, and that there's not enough of a gain to justify complicating the language by adding literals for them and making them built-in types.
Cheers, Chris -- http://blog.rebertia.com
Hey, thanks for your response, Chris!
Reading the rest of your post, it sounds like you're unaware of the
fairly nice API of the `datetime` std lib module:
Yes, you're (partially) right. I did know it existed, but I guess I forgot about it and got used to the wrong modules. This one does seem to have most (if not all) of what I was talking about, and does look like "the obvious way".
But there are some limitations I don't understand the reason for. Why aren't negative years allowed? (That ISO regulation I mentioned earlier "says it's ok".) And why does `toordinal` return an int (ignoring the "time" part of a `datetime`)?
And I still fail to see any disadvantage in having date and time literals (probably as shortcuts for the already existing datetime types, or a future version of them).
But maybe it's all just me putting my nose where it doesn't belong :)
-- Marcos --
On 29 May 2010 22:49, Chris Rebert pyideas@rebertia.com wrote:
On Sat, May 29, 2010 at 6:05 PM, Marcos Bonci marcos.bonci@gmail.com wrote:
Hi! I've been working a lot with date/time variables lately, and maybe it's
just
me, and maybe I just don't get it yet, but it sure doesn't feel like there's one-and-only-one elegant, obvious way to work with them. Feels like working with strings
in
C.
Reading the rest of your post, it sounds like you're unaware of the fairly nice API of the `datetime` std lib module: http://docs.python.org/library/datetime.html
IMO, it's pretty nice to work with; although it is annoying that some operations can only be done using the more primitive `time` or `calendar` modules. Patches are probably welcome; perhaps I should consider trying to write one.
So I was thinking, have date and time literals ever been seriously discussed anywhere in these discussion lists? (If so, I apologize in advance for reviving
this
subject.) Is there any chance we could ever see date/time literals in Python?
Well, the datetime module does appear to already be written in C, so that'd be one less hurdle. I wasn't able to find a prior post on the exact subject; perhaps someone else will.
Perhaps as a top-level abstraction itself, or as a subclass of numbers.Number, or even as common float or int (by adding attributes regarding specific properties of a number's
date/time
interpretation, e.g. "<2010-12-31>.incr_month(2) == <2011-02-28>" -- just an example
though,
i don't really think this exact method would be practical).
The `datetime` module already provides nice and more typesafe abstractions; purposefully conflating date/times with plain numbers is not a good idea, imo.
<snip> > Very often the "time elapsed" value > is what we really > want (or at least the absolute difference between two values, which is of > course easier to find > in this form), and besides being more compact and flexible, no functionality > is lost.
datetime.timedelta: http://docs.python.org/library/datetime.html#timedelta-objects
I guess the most controversial point here would be the meaning of
arithmetic
operations between dates (or the very validity of such operations).
Already defined and accepted as valid: datetime.date, "Supported operations" http://docs.python.org/library/datetime.html#date-objects
<snip> > So, what do you guys think? I'd love to hear others' opinions, even if it's > just for me > to understand what I got wrong and see the obvious way that's already there
I think the classes in the `datetime` module are the nice "missing" API that you're looking for, and that there's not enough of a gain to justify complicating the language by adding literals for them and making them built-in types.
Cheers, Chris -- http://blog.rebertia.com
On 30/05/10 13:10, Marcos Bonci wrote:
And I still fail to see any disadvantage in having date and time literals (probably as shortcuts for the already existing datetime types, or a future version of them).
The same disadvantage that any new syntax has: it increases the size of the language, making it harder to learn and harder to implement.
Why make date/time literals a special case, when they can be dealt with quite adequately as a standard module? (the existing module may have some odd quirks, but that's a far cry from needing to be given literal syntax).
Cheers, Nick.
On 2010-05-30, at 03:49 , Chris Rebert wrote:
On Sat, May 29, 2010 at 6:05 PM, Marcos Bonci marcos.bonci@gmail.com wrote:
Hi! I've been working a lot with date/time variables lately, and maybe it's just me, and maybe I just don't get it yet, but it sure doesn't feel like there's one-and-only-one elegant, obvious way to work with them. Feels like working with strings in C.
Reading the rest of your post, it sounds like you're unaware of the fairly nice API of the `datetime` std lib module: http://docs.python.org/library/datetime.html
IMO, it's pretty nice to work with; although it is annoying that some operations can only be done using the more primitive `time` or `calendar` modules. Patches are probably welcome; perhaps I should consider trying to write one.
datetime does have a bunch of issues and limitations which I believe soon become harmful when doing serious date/calendaring works (which I don't claim to do, but I've seen colleagues in serious trouble due to both personal lack of knowledge in the field and issues with datetime itself): it only supports a gregoriany calendar for instance, it's horrendous in dealing with timezones, some methods are pretty much broken, constructor refuses "24" as an hour value, blows up on positive leap seconds)…
See the following resources for issues with datetime: http://blog.mfabrik.com/2008/06/30/relativity-of-time-shortcomings-in-python... http://www.enricozini.org/2009/debian/using-python-datetime/
Libraries built over datetime: http://labix.org/python-dateutil http://pytz.sourceforge.net/ [1]
[1] note that as mentioned in the first blog post, datetime apparently has issues re querrying tz objects, so it's generally suggested to work with utc-only datetime objects and let the timezone stuff to direct pytz calls.
From Masklinn's post in his blog:
Sadly it seems that Java got it right: having one class (Datehttp://java.sun.com/j2se/1.5.0/docs/api/java/util/Date.html) that stores times in UTC seconds relative to Unix Epoch, and subclasses of abstract Calendarhttp://java.sun.com/j2se/1.5.0/docs/api/java/util/Calendar.html class that deal with getting and setting individual components and date arithmetic in a localized way would indeed be the best long-term solution.
That's exactly what I was talking about. Date/time is something apparently near-trivial (it's something everyone deals with every day), but awfully complicated when looked at in detail. This can be very deceiving for the inexperienced/inattentive programmer, and very inconvenient/time-wasting for the attentive one.
I like Python because I can usually rely on its "magic" and just care about what's elegant and readable (or else I would be using C). Judging from your personal experience, the magic looks more like illusionism when it comes to dates. Maybe this matter deserves closer attention in the context of the language/stdlibs than it has received so far? After all it's time we're talking about, it's not like it's something unimportant or rarely used (and it's something I would definitely expect to be able to rely on Python for, since pretty much everything else Python is just awesome).
There's just too many ways to do it, and none of them seem to be good enough.
-- Marcos --
On 30 May 2010 05:28, Masklinn masklinn@masklinn.net wrote:
On 2010-05-30, at 03:49 , Chris Rebert wrote:
On Sat, May 29, 2010 at 6:05 PM, Marcos Bonci marcos.bonci@gmail.com
wrote:
Hi! I've been working a lot with date/time variables lately, and maybe it's
just
me, and maybe I just don't get it yet, but it sure doesn't feel like there's one-and-only-one elegant, obvious way to work with them. Feels like working with strings
in
C.
Reading the rest of your post, it sounds like you're unaware of the fairly nice API of the `datetime` std lib module: http://docs.python.org/library/datetime.html
IMO, it's pretty nice to work with; although it is annoying that some operations can only be done using the more primitive `time` or `calendar` modules. Patches are probably welcome; perhaps I should consider trying to write one.
datetime does have a bunch of issues and limitations which I believe soon become harmful when doing serious date/calendaring works (which I don't claim to do, but I've seen colleagues in serious trouble due to both personal lack of knowledge in the field and issues with datetime itself): it only supports a gregoriany calendar for instance, it's horrendous in dealing with timezones, some methods are pretty much broken, constructor refuses "24" as an hour value, blows up on positive leap seconds)…
See the following resources for issues with datetime:
http://blog.mfabrik.com/2008/06/30/relativity-of-time-shortcomings-in-python... http://www.enricozini.org/2009/debian/using-python-datetime/
Libraries built over datetime: http://labix.org/python-dateutil http://pytz.sourceforge.net/ [1]
[1] note that as mentioned in the first blog post, datetime apparently has issues re querrying tz objects, so it's generally suggested to work with utc-only datetime objects and let the timezone stuff to direct pytz calls.
On 2010-05-30, at 12:39 , Marcos Bonci wrote:
From Masklinn's post in his blog:
It's very nice of you to attribute such a great post to me, but I just linked to it. I'm not the author of any of the resources I linked to.
I like Python because I can usually rely on its "magic" and just care about what's elegant and readable (or else I would be using C). Judging from your personal experience, the magic looks more like illusionism when it comes to dates. Maybe this matter deserves closer attention in the context of the language/stdlibs than it has received so far? After all it's time we're talking about, it's not like it's something unimportant or rarely used (and it's something I would definitely expect to be able to rely on Python for, since pretty much everything else Python is just awesome).
One of the issues from what I understand is that you pretty much need to be a specialist in the time/date domain (or have one available at all times) to get this stuff right. I'm guessing the Python community doesn't have any involved/available, least of which at the core level.
Furthermore if you want to see what's usually considered a best-of-breed in the java world, look not at the standard library but at joda time [1]. In fact, with Java 7 the current Date/Calendar API should be replaced by one strongly inspired by Joda (and created by its author) and influenced by a few other APIs of the Java world. See JSR 310 for details [2], though note that the inclusion in Java 7 apparently isn't certain yet due to delays in the JSR 310 process [3].
For a (probably long outdated) overview of what JSR 310 would provide, see [4]
[1] http://joda-time.sourceforge.net/ [2] http://jcp.org/en/jsr/detail?id=310 [3] http://tech.puredanger.com/java7 [4] http://today.java.net/pub/a/today/2008/09/18/jsr-310-new-java-date-time-api....
It's very nice of you to attribute such a great post to me, but I just linked to it. I'm not the author of any of the resources I linked to.
Oops, lack-of-sleep in action!.. Being myself an example of the inattentive guy I was talking about, I guess...
One of the issues from what I understand is that you pretty much need to be a specialist in the time/date domain (or have one available at all times) to get this stuff right. I'm guessing the Python community doesn't have any involved/available, least of which at the core level.
Furthermore if you want to see what's usually considered a best-of-breed in the java world, look not at the standard library but at joda time [1]. In fact, with Java 7 the current Date/Calendar API should be replaced by one strongly inspired by Joda (and created by its author) and influenced by a few other APIs of the Java world. See JSR 310 for details [2], though note that the inclusion in Java 7 apparently isn't certain yet due to delays in the JSR 310 process [3].
For a (probably long outdated) overview of what JSR 310 would provide, see [4]
[1] http://joda-time.sourceforge.net/ [2] http://jcp.org/en/jsr/detail?id=310 [3] http://tech.puredanger.com/java7 [4] http://today.java.net/pub/a/today/2008/09/18/jsr-310-new-java-date-time-api....
Since it's all open source stuff, shouldn't it be possible to just "clone" most of it? Or is there some Java-specific thing that makes it hard to translate?
-- Marcos --
On Sun, May 30, 2010 at 4:06 AM, Marcos Bonci marcos.bonci@gmail.com wrote:
It's very nice of you to attribute such a great post to me, but I just linked to it. I'm not the author of any of the resources I linked to.
Oops, lack-of-sleep in action!.. Being myself an example of the inattentive guy I was talking about, I guess...
One of the issues from what I understand is that you pretty much need to be a specialist in the time/date domain (or have one available at all times) to get this stuff right. I'm guessing the Python community doesn't have any involved/available, least of which at the core level.
Furthermore if you want to see what's usually considered a best-of-breed in the java world, look not at the standard library but at joda time [1]. In fact, with Java 7 the current Date/Calendar API should be replaced by one strongly inspired by Joda (and created by its author) and influenced by a few other APIs of the Java world. See JSR 310 for details [2], though note that the inclusion in Java 7 apparently isn't certain yet due to delays in the JSR 310 process [3].
For a (probably long outdated) overview of what JSR 310 would provide, see [4]
[1] http://joda-time.sourceforge.net/ [2] http://jcp.org/en/jsr/detail?id=310 [3] http://tech.puredanger.com/java7 [4] http://today.java.net/pub/a/today/2008/09/18/jsr-310-new-java-date-time-api....
Since it's all open source stuff, shouldn't it be possible to just "clone" most of it? Or is there some Java-specific thing that makes it hard to translate?
Doubtful, but anyhow, I hear mxDateTime (http://www.egenix.com/products/python/mxBase/mxDateTime/ ) is the current and extant Python gold standard. Not quite as nimble as Joda (though not many applications need to deal with e.g. Coptic dates anyway), but still a significant improvement over `datetime`.
Cheers, Chris -- http://blog.rebertia.com
On 2010-05-30, at 13:17 , Chris Rebert wrote:
On Sun, May 30, 2010 at 4:06 AM, Marcos Bonci marcos.bonci@gmail.com wrote:
It's very nice of you to attribute such a great post to me, but I just linked to it. I'm not the author of any of the resources I linked to.
Oops, lack-of-sleep in action!.. Being myself an example of the inattentive guy I was talking about, I guess...
One of the issues from what I understand is that you pretty much need to be a specialist in the time/date domain (or have one available at all times) to get this stuff right. I'm guessing the Python community doesn't have any involved/available, least of which at the core level.
Furthermore if you want to see what's usually considered a best-of-breed in the java world, look not at the standard library but at joda time [1]. In fact, with Java 7 the current Date/Calendar API should be replaced by one strongly inspired by Joda (and created by its author) and influenced by a few other APIs of the Java world. See JSR 310 for details [2], though note that the inclusion in Java 7 apparently isn't certain yet due to delays in the JSR 310 process [3].
For a (probably long outdated) overview of what JSR 310 would provide, see [4]
[1] http://joda-time.sourceforge.net/ [2] http://jcp.org/en/jsr/detail?id=310 [3] http://tech.puredanger.com/java7 [4] http://today.java.net/pub/a/today/2008/09/18/jsr-310-new-java-date-time-api....
Since it's all open source stuff, shouldn't it be possible to just "clone" most of it? Or is there some Java-specific thing that makes it hard to translate?
Doubtful, but anyhow, I hear mxDateTime (http://www.egenix.com/products/python/mxBase/mxDateTime/ ) is the current and extant Python gold standard. Not quite as nimble as Joda (though not many applications need to deal with e.g. Coptic dates anyway), but still a significant improvement over `datetime`.
Used to be, but these days I believe most people use datetime + dateutil instead. mx.DateTime is actually an improvement of what came before datetime (raw date and time modules) and datetime was inspired by mx.DateTime. Most of the mx.DateTime stuff missing from datetime is in dateutil, and mx.DateTime is very heavyweight, a pain to install and has other issues. Mostly a complete lack of support in implementations other than CPython.
A significant limitation of datetime which mx doesn't have, though, is that mx.DateTime can deal with BCE dates (< year 1).
I'm sure M.A. Lemburg will disagree and fix the mistakes in my post if he passes by though.
Anyway personally, my biggest issues with mx.DateTime are that it mandates the installation of the rest of the mx distro (the package is actually egenix-mx-base) and that it generally refuses to install, short of a bare `python setup.py install` (eqsy_install or pip install egenix-mx-base didn't ever work for me)
On Sun, May 30, 2010 at 9:28 AM, Masklinn masklinn@masklinn.net wrote:
datetime does have a bunch of issues and limitations which I believe soon become harmful when doing serious date/calendaring works (which I don't claim to do, but I've seen colleagues in serious trouble due to both personal lack of knowledge in the field and issues with datetime itself): it only supports a gregoriany calendar for instance, it's horrendous in dealing with timezones, some methods are pretty much broken, constructor refuses "24" as an hour value, blows up on positive leap seconds)…
Are there tracker issues open for all these problems? It not, please would you consider opening some? The datetime module has recently been getting a lot more attention than it used to, thanks largely to the efforts of Alexander Belopolsky, so I think opening relevant tracker issues would be worthwhile.
Some of the issues you mention look like easy fixes (e.g., allowing positive leap seconds, allowing '24:00:00' as a valid time). The API problems for timezones look a little bit more serious.
What are the use-cases for non-Gregorian calendars, and why do you think the datetime module should support them? This seems like a specialist need to me.
And which methods are 'pretty much broken'?
If you want to see progress on these issues, please do open some bugtracker issues. Or if open issues already exist, it might be worth pinging them.
Mark
On Tue, Jun 1, 2010 at 2:00 AM, Mark Dickinson dickinsm@gmail.com wrote:
On Sun, May 30, 2010 at 9:28 AM, Masklinn masklinn@masklinn.net wrote:
datetime does have a bunch of issues and limitations which I believe soon become harmful when doing serious date/calendaring works (which I don't claim to do, but I've seen colleagues in serious trouble due to both personal lack of knowledge in the field and issues with datetime itself): it only supports a gregoriany calendar for instance, it's horrendous in dealing with timezones, some methods are pretty much broken, constructor refuses "24" as an hour value, blows up on positive leap seconds)…
Are there tracker issues open for all these problems? It not, please would you consider opening some? The datetime module has recently been getting a lot more attention than it used to, thanks largely to the efforts of Alexander Belopolsky, so I think opening relevant tracker issues would be worthwhile.
Some of the issues you mention look like easy fixes (e.g., allowing positive leap seconds, allowing '24:00:00' as a valid time).
Whoa, the datetime module was explicitly designed not to support leap seconds. This matches the POSIX standard for timestamps, which, although commonly explained as "seconds since 1/1/1970 UTC" doesn't count leap seconds either (it would make the conversions between timestamps and date/time objects horribly complicated since leap seconds are not determined by an algorithm). This is all intentional, since leap seconds are designed to be ignorable by most people except a few communities like astronomers, who have their own clocks.
The API problems for timezones look a little bit more serious.
Isn't the main problem that no timezone implementations are provided by the standard library? There is a reason for that too (although we should at least have UTC in the stdlib).
What are the use-cases for non-Gregorian calendars, and why do you think the datetime module should support them? This seems like a specialist need to me.
I believe the main use case is compatibility with Java, which does support other calendars. Not a big motivation for me. :-)
And which methods are 'pretty much broken'?
If you want to see progress on these issues, please do open some bugtracker issues. Or if open issues already exist, it might be worth pinging them.
In general I would hesitate about attempts to "fix" "problems" with the datetime module that were carefully considered API properties when the design was first made.
The only problems that I currently take seriously are issues with dates before 1900, which IIRC stem from reliance on C stdlib functions for manipulating time structs.
On Tue, Jun 1, 2010 at 3:08 PM, Guido van Rossum guido@python.org wrote:
On Tue, Jun 1, 2010 at 2:00 AM, Mark Dickinson dickinsm@gmail.com wrote:
Some of the issues you mention look like easy fixes (e.g., allowing positive leap seconds, allowing '24:00:00' as a valid time).
Whoa, the datetime module was explicitly designed not to support leap seconds. This matches the POSIX standard for timestamps, which, although commonly explained as "seconds since 1/1/1970 UTC" doesn't count leap seconds either (it would make the conversions between timestamps and date/time objects horribly complicated since leap seconds are not determined by an algorithm). This is all intentional, since leap seconds are designed to be ignorable by most people except a few communities like astronomers, who have their own clocks.
Yes, I understand these issues: UTC is not POSIX time. By 'support for leap seconds', all I meant (and all I was assuming Masklinn meant) was that it would be helpful for e.g., datetime.datetime(1985, 6, 30, 23, 59, 60) to be accepted, rather producing a ValueError as it currently does:
datetime.datetime(1985, 6, 30, 23, 59, 60)
Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: second must be in 0..59
As per the POSIX standard (IIUC), that would be immediately converted to datetime.datetime(1985, 7, 1, 0, 0, 0) internally. So the datetime object itself wouldn't support leap seconds, and would continue to use POSIX time; only the constructor would support leap seconds.
Similar comments apply to accepting a time of 24:00:00 (and converting it internally to 00:00:00 on the following day).
Mark
On Tue, Jun 1, 2010 at 10:41 AM, Mark Dickinson dickinsm@gmail.com wrote: ..
As per the POSIX standard (IIUC), [datetime(1985, 6, 30, 23, 59, 60)] would be immediately converted to datetime.datetime(1985, 7, 1, 0, 0, 0) internally. So the datetime object itself wouldn't support leap seconds, and would continue to use POSIX time; only the constructor would support leap seconds.
It is my understanding that POSIX mandates that mktime() function normalizes the tm structure and therefore converts (1985, 6, 30, 23, 59, 60, ...) to (1985, 7, 1, 0, 0, 0, ...). It is not quite accurate to say that tm structure is converted "immediately". It is perfectly legal to pass around non-normalized tm structures and have for example utc2gps() function that would produce different values for Y-M-D 23:59:60 and T-M-[D+1] 00:00:00.
I would prefer a similar behavior for datetime constructor:
datetime(1985, 6, 30, 23, 59, 60).second
60
datetime(1985, 6, 30, 23, 59, 60).timetuple()
(1985, 6, 30, 23, 59, 60, ...)
but
datetime(1985, 6, 30, 23, 59, 60) - datetime(1985, 7, 1, 0, 0, 0)
datetime.timedelta(0)
On Tue, Jun 1, 2010 at 8:23 AM, Alexander Belopolsky alexander.belopolsky@gmail.com wrote:
On Tue, Jun 1, 2010 at 10:41 AM, Mark Dickinson dickinsm@gmail.com wrote: ..
As per the POSIX standard (IIUC), [datetime(1985, 6, 30, 23, 59, 60)] would be immediately converted to datetime.datetime(1985, 7, 1, 0, 0, 0) internally. So the datetime object itself wouldn't support leap seconds, and would continue to use POSIX time; only the constructor would support leap seconds.
It is my understanding that POSIX mandates that mktime() function normalizes the tm structure and therefore converts (1985, 6, 30, 23, 59, 60, ...) to (1985, 7, 1, 0, 0, 0, ...). It is not quite accurate to say that tm structure is converted "immediately". It is perfectly legal to pass around non-normalized tm structures and have for example utc2gps() function that would produce different values for Y-M-D 23:59:60 and T-M-[D+1] 00:00:00.
I would prefer a similar behavior for datetime constructor:
datetime(1985, 6, 30, 23, 59, 60).second
60
datetime(1985, 6, 30, 23, 59, 60).timetuple()
(1985, 6, 30, 23, 59, 60, ...)
but
datetime(1985, 6, 30, 23, 59, 60) - datetime(1985, 7, 1, 0, 0, 0)
datetime.timedelta(0)
I expect this will cause a lot of subtle issues. E.g. What should comparison of an unnormalized datetime value to an equivalent normalized datetime value yield? How far will you go? Is datetime.datetime(2010, 6, 1, 36, 0, 0) a way of spelling datetime.datetime(2010, 6, 2, 12, 0 0) ? How do you force normalization? Won't it break apps if the .seconds attribute can be out of range or if normalization calls need to be inserted?
The datetime module was written with "commercial" and everyday use in mind. In such use, there is no need to carry leap seconds around.
On 1 June 2010 13:17, Guido van Rossum guido@python.org wrote:
On Tue, Jun 1, 2010 at 8:23 AM, Alexander Belopolsky alexander.belopolsky@gmail.com wrote:
On Tue, Jun 1, 2010 at 10:41 AM, Mark Dickinson dickinsm@gmail.com
wrote:
..
As per the POSIX standard (IIUC), [datetime(1985, 6, 30, 23, 59, 60)]
would be
immediately converted to datetime.datetime(1985, 7, 1, 0, 0, 0) internally. So the datetime object itself wouldn't support leap seconds, and would continue to use POSIX time; only the constructor would support leap seconds.
It is my understanding that POSIX mandates that mktime() function normalizes the tm structure and therefore converts (1985, 6, 30, 23, 59, 60, ...) to (1985, 7, 1, 0, 0, 0, ...). It is not quite accurate to say that tm structure is converted "immediately". It is perfectly legal to pass around non-normalized tm structures and have for example utc2gps() function that would produce different values for Y-M-D 23:59:60 and T-M-[D+1] 00:00:00.
I would prefer a similar behavior for datetime constructor:
datetime(1985, 6, 30, 23, 59, 60).second
60
datetime(1985, 6, 30, 23, 59, 60).timetuple()
(1985, 6, 30, 23, 59, 60, ...)
but
datetime(1985, 6, 30, 23, 59, 60) - datetime(1985, 7, 1, 0, 0, 0)
datetime.timedelta(0)
I expect this will cause a lot of subtle issues. E.g. What should comparison of an unnormalized datetime value to an equivalent normalized datetime value yield? How far will you go? Is datetime.datetime(2010, 6, 1, 36, 0, 0) a way of spelling datetime.datetime(2010, 6, 2, 12, 0 0) ? How do you force normalization? Won't it break apps if the .seconds attribute can be out of range or if normalization calls need to be inserted?
The datetime module was written with "commercial" and everyday use in mind. In such use, there is no need to carry leap seconds around.
If this is really a design choice, then I guess my suggestions about date+time literals and a unique/"official" date+time interpretation as number really aren't good ideas. (Although I can't see why a precise/scientific approach wouldn't be better than a commercial one, as commercial applications often rely on precise standards.)
But I still don't understand why datetime.datetime.toordinal returns an int that truncates time information. Is this deliberate?
-- --Guido van Rossum (python.org/~guido)
-- Marcos --
[Marcos Bonci]
... But I still don't understand why datetime.datetime.toordinal returns an int that truncates time information. Is this deliberate?
That it does exactly what it's documented to do is a clue about that ;-)
As the module docs say, the notion of "ordinal" was deliberately defined in this way:
This matches the definition of the “proleptic Gregorian” calendar in Dershowitz and Reingold’s book "Calendrical Calculations", where it’s the base calendar for all computations. See the book for algorithms for converting between proleptic Gregorian ordinals and many other calendar systems.
That's the primary use case we had in mind for date & datetime ordinals. Indeed, the meaning of "ordinal" is "an integer indicating position in a sequence".
On Tue, Jun 1, 2010 at 12:17 PM, Guido van Rossum guido@python.org wrote: ..
I expect this will cause a lot of subtle issues.
I will try to answer to those.
E.g. What should comparison of an unnormalized datetime value to an equivalent normalized datetime value yield?
I am not proposing supporting arbitrary unnormalized datetime values, only to allow seconds (0 - 60). I am not proposing any notion of "equivalent datetime" objects either. POSIX will require that for t1 = datetime(1985, 6, 30, 23, 59, 60) and t2 =datetime(1985, 7, 1, 0, 0, 0) time.mktime(t1.timetuple()) == time.mktime(t2.timetuple()), but this does not mean that t1 and t2 should compare equal.
It is a more subtle issue, what difference t1 - t2 should produce. I think it can be defined as difference in corresponding POSIX times.
How far will you go? Is datetime.datetime(2010, 6, 1, 36, 0, 0) a way of spelling datetime.datetime(2010, 6, 2, 12, 0 0) ?
I would not go any further than extending seconds to 0-60 range which is common to many modern standards.
How do you force normalization?
Normalization is never forced. A round trip through POSIX timestamp will naturally produce normalized datetime objects.
Won't it break apps if the .seconds attribute can be out of range or if normalization calls need to be inserted?
Many standards require that seconds range be 0-60. Applications that obtain time from timetuples should already be prepared to handle this range to be POSIX compliant. Note that I do not propose changing internal sources of datetime objects such as datetime.now() to return dt.seconds == 60. Therefore all extended range times will originate outside of the datetime library. Current application should already validate such sources before passing them to datetime library. Of course an application that relies on constructor throwing an exception for validation and then asserts that seconds < 60 will break, but this can be addressed by proper deprecation schedule. Maybe even starting with enabling extended seconds range with a from __future__ import.
On Tue, Jun 1, 2010 at 11:10 AM, Alexander Belopolsky alexander.belopolsky@gmail.com wrote:
On Tue, Jun 1, 2010 at 12:17 PM, Guido van Rossum guido@python.org wrote: ..
I expect this will cause a lot of subtle issues.
I will try to answer to those.
E.g. What should comparison of an unnormalized datetime value to an equivalent normalized datetime value yield?
I am not proposing supporting arbitrary unnormalized datetime values, only to allow seconds (0 - 60). I am not proposing any notion of "equivalent datetime" objects either. POSIX will require that for t1 = datetime(1985, 6, 30, 23, 59, 60) and t2 =datetime(1985, 7, 1, 0, 0, 0) time.mktime(t1.timetuple()) == time.mktime(t2.timetuple()), but this does not mean that t1 and t2 should compare equal.
It is a more subtle issue, what difference t1 - t2 should produce. I think it can be defined as difference in corresponding POSIX times.
But consistency within the API is out the window. Currently datetimes are linearly ordered like numbers and if a difference is zero the two values are the same.
I think it would be safer for your use case to either store the tuple or the string representation, if you really need to represent a leap second.
Also note that there will be no validation possible for future datetimes (and for past dates it would require an up-to-date leap second database).
How far will you go? Is datetime.datetime(2010, 6, 1, 36, 0, 0) a way of spelling datetime.datetime(2010, 6, 2, 12, 0 0) ?
I would not go any further than extending seconds to 0-60 range which is common to many modern standards.
That's good.
How do you force normalization?
Normalization is never forced. A round trip through POSIX timestamp will naturally produce normalized datetime objects.
Well code that for whatever reason wants normalized timestamps only will have to know about this method to force normalization, so it would be a backwards incompatibility (since currently one can assume that *all* datetime objects are normalized).
Won't it break apps if the .seconds attribute can be out of range or if normalization calls need to be inserted?
Many standards require that seconds range be 0-60. Applications that obtain time from timetuples should already be prepared to handle this range to be POSIX compliant. Note that I do not propose changing internal sources of datetime objects such as datetime.now() to return dt.seconds == 60.Therefore all extended range times will originate outside of the datetime library. Current application should already validate such sources before passing them to datetime library. Of course an application that relies on constructor throwing an exception for validation and then asserts that seconds < 60 will break, but this can be addressed by proper deprecation schedule. Maybe even starting with enabling extended seconds range with a from __future__ import.
I see nothing but trouble down this road.
Also: http://en.wikipedia.org/wiki/Leap_second#Proposal_to_abolish_leap_seconds
[and later] On Tue, Jun 1, 2010 at 11:36 AM, Alexander Belopolsky alexander.belopolsky@gmail.com wrote:
On Tue, Jun 1, 2010 at 1:44 PM, MRAB python@mrabarnett.plus.com wrote: ..
but
datetime(1985, 6, 30, 23, 59, 60) - datetime(1985, 7, 1, 0, 0, 0)
datetime.timedelta(0)
Actually, that's wrong because there was a leap second. The clock went:
1985-06-30 23:59:59 1985-06-30 23:59:60 1985-07-01 00:00:00
The following year, however, it went:
1986-06-30 23:59:59 1986-07-01 00:00:00
It is only wrong if you expect datetime difference to reflect the actual duration between the corresponding UTC events.
What on earth do you mean by *actual duration*? Most datetimes are derived from clocks that aren't accurate to a second even.
The datetime library does not do it even for dates.
For example, on my system
date(1752, 9, 14) - date(1752, 9, 2)
datetime.timedelta(12)
even though calendar application on the same machine shows that September 14 was the day after September 2 in 1752.
And here you are mixing topics completely -- calendar reform is a completely different topic from leap seconds.
$ cal 9 1752 September 1752 Su Mo Tu We Th Fr Sa 1 2 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
This was a deliberate design choice to implement proleptic calendar rather than a historically more accurate variant. Similarly I see nothing wrong with datetime difference not capturing leap seconds. An application interested in leap seconds effects, however should still be able to use the basic datetime object and define its own duration functions.
You haven't proven this need at all, and your reference to calendar reform (which by the way didn't happen in the same year or even century everywhere) makes it weaker still.
I've put my foot down against leap seconds once before (when datetime was introduced) and I will do it again.
On Tue, Jun 1, 2010 at 11:40 PM, Guido van Rossum guido@python.org wrote:
Also note that there will be no validation possible for future datetimes (and for past dates it would require an up-to-date leap second database).
It's even worse than that :(. Complete validation would also require timezone knowledge, because leap seconds happen at the same instant the world over: e.g., the leap second that occurred at 23:59:60 UTC on 31st December 2008 occurred at 19:29:60 local time in Caracas. So for naive datetime objects validation is going to be difficult. Given that timezone offsets can be an arbitrary number of minutes, the only reasonable options as far as I can see would be either *always* to accept seconds in the range 0-60, or *always* restrict the range to 0-59, as now. (Well, you could only accept seconds=60 for timestamps within 24 hours of midnight on Jun 30 or Dec 31st, but that's fairly horrible. :-)
I'm still not convinced that incorrectly accepting some invalid UTC times is worse than incorrectly rejecting some (rare) valid UTC times, but I'll let it drop for now.
On Wed, Jun 2, 2010 at 11:50 AM, Mark Dickinson dickinsm@gmail.com wrote:
within 24 hours of midnight on Jun 30 or Dec 31st, but that's fairly
To avoid ambiguity, that should read "within 24 hours of 24:00:00 on Jun 30th or Dec 31st", of course. :)
-- Mark
Mark Dickinson wrote:
On Tue, Jun 1, 2010 at 11:40 PM, Guido van Rossum guido@python.org wrote:
Also note that there will be no validation possible for future datetimes (and for past dates it would require an up-to-date leap second database).
It's even worse than that :(. Complete validation would also require timezone knowledge, because leap seconds happen at the same instant the world over: e.g., the leap second that occurred at 23:59:60 UTC on 31st December 2008 occurred at 19:29:60 local time in Caracas. So for naive datetime objects validation is going to be difficult. Given that timezone offsets can be an arbitrary number of minutes, the only reasonable options as far as I can see would be either *always* to accept seconds in the range 0-60, or *always* restrict the range to 0-59, as now. (Well, you could only accept seconds=60 for timestamps within 24 hours of midnight on Jun 30 or Dec 31st, but that's fairly horrible. :-)
I'm still not convinced that incorrectly accepting some invalid UTC times is worse than incorrectly rejecting some (rare) valid UTC times, but I'll let it drop for now.
You can use mxDateTime to store such values. I added support for storing leap seconds long ago, but only for the UTC variants, not for arbitrary time zones. This was mainly done to support those values when using mxDateTime as storage container rather than for calculations (those are all POSIX conform, ie. omit leap seconds).
Note that most C libs nowadays only support the POSIX interpretation of time_t values. Those don't include leap seconds:
DateTime(1986,12,31,23,59,59).gmticks()
536457599.0
DateTime(1986,12,31,23,59,60).gmticks()
536457600.0
DateTime(1987,1,1,0,0,0).gmticks()
536457600.0
with leap seconds, you'd get 536457612 for DateTime(1986,12,31,23,59,59).gmticks().
As a result, conversion to time_t will be lossy.
IIRC, the BSDs were the last to switch off leap second support, but could be mistaken.
Alexander Belopolsky wrote:
On Tue, Jun 1, 2010 at 10:41 AM, Mark Dickinson dickinsm@gmail.com wrote: ..
As per the POSIX standard (IIUC), [datetime(1985, 6, 30, 23, 59, 60)] would be immediately converted to datetime.datetime(1985, 7, 1, 0, 0, 0) internally. So the datetime object itself wouldn't support leap seconds, and would continue to use POSIX time; only the constructor would support leap seconds.
It is my understanding that POSIX mandates that mktime() function normalizes the tm structure and therefore converts (1985, 6, 30, 23, 59, 60, ...) to (1985, 7, 1, 0, 0, 0, ...). It is not quite accurate to say that tm structure is converted "immediately". It is perfectly legal to pass around non-normalized tm structures and have for example utc2gps() function that would produce different values for Y-M-D 23:59:60 and T-M-[D+1] 00:00:00.
I would prefer a similar behavior for datetime constructor:
datetime(1985, 6, 30, 23, 59, 60).second
60
datetime(1985, 6, 30, 23, 59, 60).timetuple()
(1985, 6, 30, 23, 59, 60, ...)
but
datetime(1985, 6, 30, 23, 59, 60) - datetime(1985, 7, 1, 0, 0, 0)
datetime.timedelta(0)
Actually, that's wrong because there was a leap second. The clock went:
1985-06-30 23:59:59 1985-06-30 23:59:60 1985-07-01 00:00:00
The following year, however, it went:
1986-06-30 23:59:59 1986-07-01 00:00:00
On Tue, Jun 1, 2010 at 1:44 PM, MRAB python@mrabarnett.plus.com wrote: ..
but
datetime(1985, 6, 30, 23, 59, 60) - datetime(1985, 7, 1, 0, 0, 0)
datetime.timedelta(0)
Actually, that's wrong because there was a leap second. The clock went:
1985-06-30 23:59:59 1985-06-30 23:59:60 1985-07-01 00:00:00
The following year, however, it went:
1986-06-30 23:59:59 1986-07-01 00:00:00
It is only wrong if you expect datetime difference to reflect the actual duration between the corresponding UTC events. The datetime library does not do it even for dates.
For example, on my system
date(1752, 9, 14) - date(1752, 9, 2)
datetime.timedelta(12)
even though calendar application on the same machine shows that September 14 was the day after September 2 in 1752.
$ cal 9 1752 September 1752 Su Mo Tu We Th Fr Sa 1 2 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
This was a deliberate design choice to implement proleptic calendar rather than a historically more accurate variant. Similarly I see nothing wrong with datetime difference not capturing leap seconds. An application interested in leap seconds effects, however should still be able to use the basic datetime object and define its own duration functions.
On Tue, Jun 1, 2010 at 7:41 AM, Mark Dickinson dickinsm@gmail.com wrote:
On Tue, Jun 1, 2010 at 3:08 PM, Guido van Rossum guido@python.org wrote:
On Tue, Jun 1, 2010 at 2:00 AM, Mark Dickinson dickinsm@gmail.com wrote:
Some of the issues you mention look like easy fixes (e.g., allowing positive leap seconds, allowing '24:00:00' as a valid time).
Whoa, the datetime module was explicitly designed not to support leap seconds. This matches the POSIX standard for timestamps, which, although commonly explained as "seconds since 1/1/1970 UTC" doesn't count leap seconds either (it would make the conversions between timestamps and date/time objects horribly complicated since leap seconds are not determined by an algorithm). This is all intentional, since leap seconds are designed to be ignorable by most people except a few communities like astronomers, who have their own clocks.
Yes, I understand these issues: UTC is not POSIX time. By 'support for leap seconds', all I meant (and all I was assuming Masklinn meant) was that it would be helpful for e.g., datetime.datetime(1985, 6, 30, 23, 59, 60) to be accepted, rather producing a ValueError as it currently does:
datetime.datetime(1985, 6, 30, 23, 59, 60)
Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: second must be in 0..59
As per the POSIX standard (IIUC), that would be immediately converted to datetime.datetime(1985, 7, 1, 0, 0, 0) internally. So the datetime object itself wouldn't support leap seconds, and would continue to use POSIX time; only the constructor would support leap seconds.
Similar comments apply to accepting a time of 24:00:00 (and converting it internally to 00:00:00 on the following day).
What's the use case for these relaxations in argument range checking? I'd say they are more confusing, since they might lead one to suspect that leap seconds are in fact supported.
On Tue, Jun 1, 2010 at 5:07 PM, Guido van Rossum guido@python.org wrote:
What's the use case for these relaxations in argument range checking? I'd say they are more confusing, since they might lead one to suspect that leap seconds are in fact supported.
For the first, it would prevent tuples corresponding to valid UTC times (or local times) causing an exception in the datetime constructor. I don't have any specific use-cases, but it's not hard to imagine passing a tuple from some external UTC-supporting source to datetime.datetime.
The second relaxation (allowing 24:00:00) comes from ISO 8601, but I don't really know how widespread its use is. I admit I don't find this one particularly convincing; perhaps Masklinn can expand on why it's useful.
Mark
On Tue, Jun 1, 2010 at 9:28 AM, Mark Dickinson dickinsm@gmail.com wrote:
On Tue, Jun 1, 2010 at 5:07 PM, Guido van Rossum guido@python.org wrote:
What's the use case for these relaxations in argument range checking? I'd say they are more confusing, since they might lead one to suspect that leap seconds are in fact supported.
For the first, it would prevent tuples corresponding to valid UTC times (or local times) causing an exception in the datetime constructor. I don't have any specific use-cases, but it's not hard to imagine passing a tuple from some external UTC-supporting source to datetime.datetime.
Imagined use cases are just that.
The second relaxation (allowing 24:00:00) comes from ISO 8601, but I don't really know how widespread its use is. I admit I don't find this one particularly convincing; perhaps Masklinn can expand on why it's useful.
This I can understand, but more for output than for input. It is useful to specify the end time of an event (e.g. a party) ending at midnight as ending at 24:00 on a given date rather than at 00:00 on the next day, since that might confuse humans.
On Tue, Jun 1, 2010 at 12:36 PM, Guido van Rossum guido@python.org wrote:
On Tue, Jun 1, 2010 at 9:28 AM, Mark Dickinson dickinsm@gmail.com wrote:
..
For the first, it would prevent tuples corresponding to valid UTC times (or local times) causing an exception in the datetime constructor. I don't have any specific use-cases, but it's not hard to imagine passing a tuple from some external UTC-supporting source to datetime.datetime.
Imagined use cases are just that.
Developers writing generic libraries have to deal with imagined use cases all the time. If I write an rfc3339 timestamp parser, I cannot ignore the fact that XXXX-12-31T23:59:60Z is a valid timestamp. If I do, I cannot claim that my parser implements rfc3339. An application that uses python datetime objects to represent time may crash parsing logs produced in December 2008 on the systems that keeps time in UTC.
If all my application does is to read timestamps from some source, store them in the database and display them on a later date, I don't want to worry that it will crash when presented with 23:59:60.
Of course, allowing leap seconds in time/datetime constructor may be a way to delay detection of a bug. An application may accept XXXX-12-31T23:59:60Z, but later rely on the fact that dt1-dt2 == timedelta(0) implies dt1 == dt2. Such issues, if exist, can be addressed by the application without replacing datetime object as a means of storing timestamps. On the other hand the current restriction in the constructor makes datetime fundamentally incompatible with a number of standards.
PS: I believe systems capable of producing 23:59:60 in timestamps are already more common than those that don't use IEEE standard for floating point values. Nevertheless, CPython contains a lot of code designed to deal with imagined deviations from IEEE 754.
On 02/06/10 03:28, Alexander Belopolsky wrote:
Developers writing generic libraries have to deal with imagined use cases all the time. If I write an rfc3339 timestamp parser, I cannot ignore the fact that XXXX-12-31T23:59:60Z is a valid timestamp. If I do, I cannot claim that my parser implements rfc3339. An application that uses python datetime objects to represent time may crash parsing logs produced in December 2008 on the systems that keeps time in UTC.
If all my application does is to read timestamps from some source, store them in the database and display them on a later date, I don't want to worry that it will crash when presented with 23:59:60.
Of course, allowing leap seconds in time/datetime constructor may be a way to delay detection of a bug. An application may accept XXXX-12-31T23:59:60Z, but later rely on the fact that dt1-dt2 == timedelta(0) implies dt1 == dt2. Such issues, if exist, can be addressed by the application without replacing datetime object as a means of storing timestamps. On the other hand the current restriction in the constructor makes datetime fundamentally incompatible with a number of standards.
The case for allowing a "60" value for seconds in the datetime constructor seems reasonable to me (i.e. prevent leap seconds from breaking date parsing), but I don't see the use case for delaying normalisation to a valid POSIX time.
If the constructor just converts the 60 to a zero and adds 1 minute immediately, then the chance of subtle breakages would be minimal and the current ValueError would be replaced by a far more graceful behaviour.
(Allowing 2400 hours just seems plain odd to me, but if it was adopted I'd suggest immediate normalisation be similarly applied in that case as well).
Cheers, Nick.
On Wed, Jun 2, 2010 at 5:45 AM, Nick Coghlan ncoghlan@gmail.com wrote:
On 02/06/10 03:28, Alexander Belopolsky wrote:
Developers writing generic libraries have to deal with imagined use cases all the time. If I write an rfc3339 timestamp parser, I cannot ignore the fact that XXXX-12-31T23:59:60Z is a valid timestamp. If I do, I cannot claim that my parser implements rfc3339. An application that uses python datetime objects to represent time may crash parsing logs produced in December 2008 on the systems that keeps time in UTC.
If all my application does is to read timestamps from some source, store them in the database and display them on a later date, I don't want to worry that it will crash when presented with 23:59:60.
Of course, allowing leap seconds in time/datetime constructor may be a way to delay detection of a bug. An application may accept XXXX-12-31T23:59:60Z, but later rely on the fact that dt1-dt2 == timedelta(0) implies dt1 == dt2. Such issues, if exist, can be addressed by the application without replacing datetime object as a means of storing timestamps. On the other hand the current restriction in the constructor makes datetime fundamentally incompatible with a number of standards.
The case for allowing a "60" value for seconds in the datetime constructor seems reasonable to me (i.e. prevent leap seconds from breaking date parsing), but I don't see the use case for delaying normalisation to a valid POSIX time.
If the constructor just converts the 60 to a zero and adds 1 minute immediately, then the chance of subtle breakages would be minimal and the current ValueError would be replaced by a far more graceful behaviour.
(Allowing 2400 hours just seems plain odd to me, but if it was adopted I'd suggest immediate normalisation be similarly applied in that case as well).
I'd be okay with immediate normalization in both of these cases as well. Immediate normalization cuts off all concerns about unnormalized datetimes, consistent comparisons, etc.
Nick Coghlan wrote:
The case for allowing a "60" value for seconds in the datetime constructor seems reasonable to me (i.e. prevent leap seconds from breaking date parsing), but I don't see the use case for delaying normalisation to a valid POSIX time.
If the constructor just converts the 60 to a zero and adds 1 minute immediately, then the chance of subtle breakages would be minimal and the current ValueError would be replaced by a far more graceful behaviour.
I think this is the best we can do and not get sucked into supporting leap seconds.
(Allowing 2400 hours just seems plain odd to me, but if it was adopted I'd suggest immediate normalisation be similarly applied in that case as well).
I'm not as concerned about this, but I've had occasions where it would have been handy.
On Tue, Jun 1, 2010 at 10:08 AM, Guido van Rossum guido@python.org wrote: ..
Some of the issues you mention look like easy fixes (e.g., allowing positive leap seconds, allowing '24:00:00' as a valid time).
Whoa, the datetime module was explicitly designed not to support leap seconds. This matches the POSIX standard for timestamps, which, although commonly explained as "seconds since 1/1/1970 UTC" doesn't count leap seconds either (it would make the conversions between timestamps and date/time objects horribly complicated since leap seconds are not determined by an algorithm). This is all intentional, since leap seconds are designed to be ignorable by most people except a few communities like astronomers, who have their own clocks.
The POSIX standard was heavily influenced by the desire to preserve last century's existing practices. Its notable that even the leap year rule was initially specified incorrectly and only fixed in 2001 version.
Here is the post that I find intriguing: http://www.mail-archive.com/leapsecs@rom.usno.navy.mil/msg00109.html
An excerpt:
""" In addition these "glued to the table" cards, there were a number of unfortunate attitudes:
"Don't confuse people with UTC. Everyone uses GMT and knows what it means".
"Lets not complicate things by worrying about the fact that the year 2100 is not a leap year."
"You mean the year 2000 is, but 2100 is not a leap year?"
"Everyone knows there are only 60 seconds in a minute."
"I'm lucky if my system's clock is accurate to the minute, so I could care less about sometime as small as a leap second".
"It takes hours, sometime days, for my EMail message to reach most people. Why should I worry about something as small as a second?"
"What matters to me is just that POSIX systems produce the same ctime(3) string (i.e., Wed Jun 30 21:49:08 1993\n") when given the same time(2) time-stamp."
"SI? TAI? UT1? I'm having trouble with using UTC instead of good old GMT!". """
Systems that are aware of leap seconds are not that uncommon. BSD derivatives including Mac OS X have time2posix() and posix2time() functions. NTP distributes leap seconds notifications. Any system that takes time from a GPS source needs to make a leap second translation.
I think what Mark meant by "easy fixes" was not leap second aware timestamp to datetime and back translations or datetime arithmetics, but instead just the ability to store 23:59:60 in time/datetime object. This would allow leap second aware applications to use standard objects to store time and implement arithmetics as correction to the standard datetime arithmetics. This is much easier than to reimplement the entire datetime module from scratch.
On Tue, Jun 1, 2010 at 8:07 AM, Alexander Belopolsky alexander.belopolsky@gmail.com wrote:
On Tue, Jun 1, 2010 at 10:08 AM, Guido van Rossum guido@python.org wrote: ..
Some of the issues you mention look like easy fixes (e.g., allowing positive leap seconds, allowing '24:00:00' as a valid time).
Whoa, the datetime module was explicitly designed not to support leap seconds. This matches the POSIX standard for timestamps, which, although commonly explained as "seconds since 1/1/1970 UTC" doesn't count leap seconds either (it would make the conversions between timestamps and date/time objects horribly complicated since leap seconds are not determined by an algorithm). This is all intentional, since leap seconds are designed to be ignorable by most people except a few communities like astronomers, who have their own clocks.
The POSIX standard was heavily influenced by the desire to preserve last century's existing practices.
I don't expect this century's practices will change much. Show me a a labor contract with a provision to pay for work during leap seconds and I might change my mind.
Its notable that even the leap year rule was initially specified incorrectly and only fixed in 2001 version.
I don't see how that's an argument for supporting leap seconds. The change here is really about the expectation of the lifetime of software systems, not unlike what caused Y2K.
Here is the post that I find intriguing: http://www.mail-archive.com/leapsecs@rom.usno.navy.mil/msg00109.html
A rants by someone with a grudge.
An excerpt:
""" In addition these "glued to the table" cards, there were a number of unfortunate attitudes:
"Don't confuse people with UTC. Everyone uses GMT and knows what it means".
"Lets not complicate things by worrying about the fact that the year 2100 is not a leap year."
"You mean the year 2000 is, but 2100 is not a leap year?"
"Everyone knows there are only 60 seconds in a minute."
"I'm lucky if my system's clock is accurate to the minute, so I could care less about sometime as small as a leap second".
"It takes hours, sometime days, for my EMail message to reach most people. Why should I worry about something as small as a second?"
"What matters to me is just that POSIX systems produce the same ctime(3) string (i.e., Wed Jun 30 21:49:08 1993\n") when given the same time(2) time-stamp."
"SI? TAI? UT1? I'm having trouble with using UTC instead of good old GMT!". """
He throws ripe and green comments together in a way to make it sound as if not knowing the Gregorian leap year rule is of the same magnitude as not caring about leap seconds or TAI.
Systems that are aware of leap seconds are not that uncommon. BSD derivatives including Mac OS X have time2posix() and posix2time() functions. NTP distributes leap seconds notifications. Any system that takes time from a GPS source needs to make a leap second translation.
That's a solved problem though, is it? The accounting for leap seconds properly belongs in the layers closest to NTP / GPS.
All other software running in the typical computer (even Google's servers) uses interfaces that use POSIX timestamps (albeit often with fractions of a second supportd) or something logically equivalent.
I think what Mark meant by "easy fixes" was not leap second aware timestamp to datetime and back translations or datetime arithmetics, but instead just the ability to store 23:59:60 in time/datetime object. This would allow leap second aware applications to use standard objects to store time and implement arithmetics as correction to the standard datetime arithmetics. This is much easier than to reimplement the entire datetime module from scratch.
Tell us about the use case.
Note that if you're talking about times in the future (a very useful use case for the datetime module) you *can't* account for leap seconds since it is not known (far) ahead when they will be.
To me, the single most irritating problem with the Python support for date/time is the lack of support for time-zone understanding. This breaks down into two major issues, %z and lack of a standard time-zone table.
First, let's say I have to parse a Medusa log file, which contains time stamps in the form "DD/Mon/YYYY:HH:MM:SS [+|-]HHMM", e.g. "31/May/2010:07:10:04 -0800". What I'd like to write is
tm = time.mktime(time.strptime(timestamp, "%d/%b/%Y:%H:%M:%S %z"))
which is what I'd do if I was writing in C. But no! The Python _strptime module doesn't support "%z". So instead, I have to pick the timestamp apart and do things separately and remember that "-0800" isn't octal, and also isn't the same as -800, and remember whether to add or subtract it. This seems insane. So, IMO, support for %z should be added to Lib/_strptime.py. We need a patch.
Secondly, we really need concrete subclasses of tzinfo, and some sort of mapping. Lots of people have spent lots of time trying to figure out this cryptic hint in datetime: "The datetime module does not supply any concrete subclasses of tzinfo." I'm not sure whether pytz is the best ideas, or what I use, the "zoneinfo" module from python-dateutil. With that, I still have to add the Windows timezone names, using the table at http://unicode.org/repos/cldr/trunk/common/supplemental/windowsZones.xml, because the code in python-dateutil only works with Windows timezone names when running on Windows.
Bill
On Tue, Jun 1, 2010 at 2:32 PM, Bill Janssen janssen@parc.com wrote:
To me, the single most irritating problem with the Python support for date/time is the lack of support for time-zone understanding.
There are two related issues on the tracker:
http://bugs.python.org/issue5094 "datetime lacks concrete tzinfo impl. for UTC" http://bugs.python.org/issue6641 "strptime doesn't support %z format ?"
On Tue, Jun 1, 2010 at 11:32, Bill Janssen janssen@parc.com wrote:
To me, the single most irritating problem with the Python support for date/time is the lack of support for time-zone understanding. This breaks down into two major issues, %z and lack of a standard time-zone table.
First, let's say I have to parse a Medusa log file, which contains time stamps in the form "DD/Mon/YYYY:HH:MM:SS [+|-]HHMM", e.g. "31/May/2010:07:10:04 -0800". What I'd like to write is
tm = time.mktime(time.strptime(timestamp, "%d/%b/%Y:%H:%M:%S %z"))
which is what I'd do if I was writing in C. But no! The Python _strptime module doesn't support "%z". So instead, I have to pick the timestamp apart and do things separately and remember that "-0800" isn't octal, and also isn't the same as -800, and remember whether to add or subtract it. This seems insane. So, IMO, support for %z should be added to Lib/_strptime.py. We need a patch.
Secondly, we really need concrete subclasses of tzinfo, and some sort of mapping. Lots of people have spent lots of time trying to figure out this cryptic hint in datetime: "The datetime module does not supply any concrete subclasses of tzinfo." I'm not sure whether pytz is the best ideas, or what I use, the "zoneinfo" module from python-dateutil. With that, I still have to add the Windows timezone names, using the table at http://unicode.org/repos/cldr/trunk/common/supplemental/windowsZones.xml, because the code in python-dateutil only works with Windows timezone names when running on Windows.
First of all, there will never be a timezone table in the stdlib, period. This has been brought up before and is always shot down because python-dev does not want to have to keep track of timezone changes. pytz and other modules fit that bill fine.
Now if you want UTC, that's different. Alexander already linked to an issue that is discussing that end. The current proposal is to provide a generic class that creates fixed UTC-offset timezones, with an instance for UTC set on the datetime module.
If you get that class in, you could then patch _strptime to support the %z directive so as to return a timezone that had a set UTC-offset. Not optimal, but it's something. Otherwise you would need to patch _strptime to simply consume the number which I don't think anyone wants.
Brett Cannon brett@python.org wrote:
First of all, there will never be a timezone table in the stdlib, period. This has been brought up before and is always shot down because python-dev does not want to have to keep track of timezone changes. pytz and other modules fit that bill fine.
Sure, sure. Though I'm not sure that it has to be "in" the standard library to be part of the standard library. Past time for CPython to start thinking about on-demand data, pulled dynamically from "the cloud", with a static version for backup. Just a thought...
Now if you want UTC, that's different. Alexander already linked to an issue that is discussing that end. The current proposal is to provide a generic class that creates fixed UTC-offset timezones, with an instance for UTC set on the datetime module.
Yes, I've been following that. Very promising.
If you get that class in, you could then patch _strptime to support the %z directive so as to return a timezone that had a set UTC-offset. Not optimal, but it's something.
Yes, exactly.
Otherwise you would need to patch _strptime to simply consume the number which I don't think anyone wants.
No.
Bill
On Tue, Jun 1, 2010 at 18:18, Bill Janssen janssen@parc.com wrote:
Brett Cannon brett@python.org wrote:
First of all, there will never be a timezone table in the stdlib, period. This has been brought up before and is always shot down because python-dev does not want to have to keep track of timezone changes. pytz and other modules fit that bill fine.
Sure, sure. Though I'm not sure that it has to be "in" the standard library to be part of the standard library. Past time for CPython to start thinking about on-demand data, pulled dynamically from "the cloud", with a static version for backup. Just a thought...
Now if you want UTC, that's different. Alexander already linked to an issue that is discussing that end. The current proposal is to provide a generic class that creates fixed UTC-offset timezones, with an instance for UTC set on the datetime module.
Yes, I've been following that. Very promising.
Just need a patch. =)
If you get that class in, you could then patch _strptime to support the %z directive so as to return a timezone that had a set UTC-offset. Not optimal, but it's something.
Yes, exactly.
Then that's fine. Get the fixed offset timezone in and then get a patch for this and I don't see resistance.
On 06/02/10 10:50, Brett Cannon wrote:
On Tue, Jun 1, 2010 at 11:32, Bill Janssen janssen@parc.com wrote:
To me, the single most irritating problem with the Python support for date/time is the lack of support for time-zone understanding. This breaks down into two major issues, %z and lack of a standard time-zone table.
First, let's say I have to parse a Medusa log file, which contains time stamps in the form "DD/Mon/YYYY:HH:MM:SS [+|-]HHMM", e.g. "31/May/2010:07:10:04 -0800". What I'd like to write is
tm = time.mktime(time.strptime(timestamp, "%d/%b/%Y:%H:%M:%S %z"))
which is what I'd do if I was writing in C. But no! The Python _strptime module doesn't support "%z". So instead, I have to pick the timestamp apart and do things separately and remember that "-0800" isn't octal, and also isn't the same as -800, and remember whether to add or subtract it. This seems insane. So, IMO, support for %z should be added to Lib/_strptime.py. We need a patch.
Secondly, we really need concrete subclasses of tzinfo, and some sort of mapping. Lots of people have spent lots of time trying to figure out this cryptic hint in datetime: "The datetime module does not supply any concrete subclasses of tzinfo." I'm not sure whether pytz is the best ideas, or what I use, the "zoneinfo" module from python-dateutil. With that, I still have to add the Windows timezone names, using the table at http://unicode.org/repos/cldr/trunk/common/supplemental/windowsZones.xml, because the code in python-dateutil only works with Windows timezone names when running on Windows.
First of all, there will never be a timezone table in the stdlib, period. This has been brought up before and is always shot down because python-dev does not want to have to keep track of timezone changes. pytz and other modules fit that bill fine.
Has a module to pull timezone data from the OS been proposed before?
Has a module to pull timezone data from the OS been proposed before?
Yes.
"We really could use a module that accesses the entire timezone database but that's even more platform specific." (Guido van Rossum, 2007-03-06, http://bugs.python.org/issue1647654#msg31139).
participants (14)
-
Alexander Belopolsky -
Bill Janssen -
Brett Cannon -
Chris Rebert -
Eric Smith -
Guido van Rossum -
Lie Ryan -
M.-A. Lemburg -
Marcos Bonci -
Mark Dickinson -
Masklinn -
MRAB -
Nick Coghlan -
Tim Peters