Mailman 3 July 2015 - Datetime-SIG

The BDFL's take
by Guido van Rossum 29 Jul '15

29 Jul '15

[I posted this to python-ideas, but then found that this SIG was created. So let's discuss it here.] Honestly, I think some of these proposals are way too ambitious and will drown in a sea of bike-shedding. Here's what I think is reasonable: - Keep the time module; POSIX timestamps represented as floats are still very useful (and ubiquitous in other APIs), and it would take decades to deprecate. - Keep the datetime module; it has its uses (and even fans), and again, deprecating would take decades. - Add a few convenience APIs to one or both of the above for better conversion between various forms of datetime and POSIX timestamps (I think there are a few missing APIs in this area). - Design a new module with a separate API for "human time manipulation". This is where operations belong like "add 1 month" or "add 1 day" with heuristics for edge cases (e.g. months of different lengths due to calendar conventions, and the occasional short or long day due to DST). Possibly this can borrow some API and/or implementation from pytz. Possibly it could be made available as a 3rd party module on PyPI first, to get it user-tested in quick iterations (quicker than core Python releases, anyway). But that doesn't preclude also writing a PEP. This module needs to interface somehow with the datetime module (and possibly also with the time module) in ways that need some serious thinking -- that's where I expect the hard part of writing the PEP will be. (Maybe a "human timedelta" type would be useful here? Eventually this might even be added to the datetime module.) There are a few additional issues which I'm not sure are included in this discussion or not: - A unified API for accessing an up-to-date timezone database. This has been discussed in the past, and we couldn't find a solution that satisfied all requirements (the main problem being different platform expectations IIRC), but it's probably worth it trying again. - Nanosecond precision? I'm not against adding this to the datetime module, as long as backward compatibility is somehow preserved, especially for pickles (maybe nanoseconds default to zero unless explicitly requested or specified). - An "is_dst" flag for datetime objects. Again, this should default to "backwards compatible" (== "don't care", the -1 value in struct tm) so that pickles are two-way compatible with Python 3.5 and before. -- --Guido van Rossum (python.org/~guido)

1 0

Re: [Datetime-SIG] What's are the issues?
by Chris Angelico 29 Jul '15

29 Jul '15

On Wed, Jul 29, 2015 at 1:13 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > [Chris Angelico <rosuav(a)gmail.com>] >> What if the existing timedelta is left as-is, > > That much isn't a "what if" - that much is a requirement ;-) Heh, sure. Vagaries of the English language sometimes result in an apparent suggestion like this :) >> and a new calendardelta type is introduced? Instead of folding >> everything down to a simple microsecond count, it would retain >> the exact values it was given, >> so calendardelta(days=1) != calendardelta(seconds=86400). >> When you add a calendardelta to an aware datetime, it performs >> what you're describing as calendar arithmetic > > That's what the _current_ timedelta already does (things like "same > time tomorrow", "same time in 25000 weeks", and "same time plus an > hour 57 days ago" are all trivially done already by datetime + > timedelta operations). Except that it isn't possible to distinguish "add 86400 seconds" from "add 1 day". It's fine for adding to a naive datetime, as they're equivalent, but they differ with aware datetimes - hence the notion of having a different way to add to them. The semantics would be that adding any unit to an aware datetime *will not change* any unit of finer resolution, unless the result would otherwise be impossible to represent. So when you add 1 day to 9AM today, you get 9AM tomorrow; but when you add 24 hours to 9AM today, you might get 10AM tomorrow, because one hour got snipped out of the intervening night. Similarly, adding 1 year to Feb 23rd 2012 will result in Feb 23rd 2013 (has added 366 days), but adding 1 year to Feb 23rd 2013 will add only 365 days; this is a concept that the current timedelta simply can't handle, as it doesn't work in any unit larger than the day. There is one edge case that I'm aware of, and that's where the simple calculation results in an invalid timestamp - either because the clock jumps forward (DST switch) or because you would land on a leap day. In each case, it's possible only when you're moving in a unit greater than the one that's being adjusted - you can't land on Feb 29th 2015 by moving X days, because it'll count days right past it, so you must have been moving months or years. When you move forward, the obvious thing is to jump forward by a distance exactly equal to the gap - one extra day to land on March 1st, or one extra hour to land at 3AM - but I'm not sure what the best solution is when moving backward. Are there any other nasty edge cases? ChrisA

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Alexander Belopolsky 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 10:45 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > But, > internally,datetimes, dates, times, and timedeltas are all sometimes > viewed as just complicated ways of spelling "some number of > microseconds". While obvious when you look from the inside of the implementation, this is something that datetime library users will never understand. That's why over my strong objections, timedelta grew its total_seconds() method which is nothing else but x / timedeta(seconds=1). After that, I gave up and committed datetime.timestamp() as a shortcut for (x - EPOCH) / timedeta(seconds=1).

1 0

Re: [Datetime-SIG] What's are the issues?
by Tim Peters 29 Jul '15

29 Jul '15

[Chris Angelico <rosuav(a)gmail.com>] > What if the existing timedelta is left as-is, That much isn't a "what if" - that much is a requirement ;-) > and a new calendardelta type is introduced? Instead of folding > everything down to a simple microsecond count, it would retain > the exact values it was given, > so calendardelta(days=1) != calendardelta(seconds=86400). > When you add a calendardelta to an aware datetime, it performs > what you're describing as calendar arithmetic That's what the _current_ timedelta already does (things like "same time tomorrow", "same time in 25000 weeks", and "same time plus an hour 57 days ago" are all trivially done already by datetime + timedelta operations). What Lennart is on about is that there's no direct way to spell "the other" kind of arithmetic now (operationally defined by: convert to UTC first, then do the "naive arithmetic", and finally convert back to the original timezone). But it's also straightforward to write a simple function to do that, and not every simple function has to be built in. > and it will be the most plausibly sane behaviour for the exact unit used; so, > for instance, adding calendardelta(days=1) would never affect the hour/minute/second > shown, unless it would otherwise create an illegal time. Would that serve? Note that timedelta doesn't accept any units for which there are any questions about what a "calendar operation" "should do" in Guido's "naive time" design. That's also an intentional part of the datetime design. So. e.g., you can ask a timedelta for some number of days or weeks, but _not_ for some number of months or years. "Why not years?" is because there's one case where what to do isn't 100% obvious to everyone: a year starting from a leap year's 29 Feb. "Why not a month?" has many more questionable cases. In any case, you should look at other Python date-&-time libraries (like mxDateTime and dateutil) because there are already approaches to richer "calendar options" fully thought out.

1 0

Re: [Datetime-SIG] What's are the issues?
by Chris Angelico 29 Jul '15

29 Jul '15

On Wed, Jul 29, 2015 at 12:02 PM, Chris Barker <chris.barker(a)noaa.gov> wrote: > On Tue, Jul 28, 2015 at 6:43 PM, Chris Angelico <rosuav(a)gmail.com> wrote: > >> What if the existing timedelta is left as-is, and a new calendardelta >> type is introduced? Instead of folding everything down to a simple >> microsecond count, it would retain the exact values it was given, > > > sure -- that would be one way to do it -- maybe good way. But it would be a > lot more complicated than just keeping days. Lots to hash out for API and > functionality at the corner cases (the old what's one year after Feb 29th? > kinds of questions) I'd say "one year after Feb 29th" is exactly the same question as "one day after 2:30AM before a DST switch", and has the same answer (add another day, or another hour). >> When you add a >> calendardelta to an aware datetime, it performs what you're describing >> as calendar arithmetic, > > actually, no reason you couldn't do calendar manipulations on naive > datetimes, too (they wouldn't get any ugly DST issues, but otherwise the > same). One reason I don't like calling the arithmetic naive or not.... Yes, but when you do them on naive datetimes, they'll be indistinguishable from the existing behaviour, and thus not a compelling argument for a new timedelta variant. I agree though - it wouldn't be an error to add a calendardelta to a naive datetime. >> adding >> calendardelta(days=1) would never affect the hour/minute/second shown, >> unless it would otherwise create an illegal time. > > > exactly -- all those corner cases would need to be worked out. Simplest definition would be what we want here. Jump forward by another unit of whatever one has disappeared. What if we're moving backward? If you subtract a year from Feb 29th 2012, do you get Feb 28th 2011 or Mar 1st 2011? But I think adding a year should produce Mar 1st 2013. ChrisA

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Tim Peters 29 Jul '15

29 Jul '15

[Chris Barker] >> 1) timespan since an epoch: -- i.e seconds since Jan 1, 1970 00:00. This is >> how the datetime object keeps it internally. [Alexander Belopolsky <alexander.belopolsky(a)gmail.com>] > No, it is not. You are probably confusing stdlib datetime with > numpy.datetime64. > The stdlib datetime keeps broken down year, month, day, hour, minute, second > and microsecond values. It's interesting (well, maybe to someone ;-) ) to note that this division into named attributes is simply for efficiency in extracting commonly accessed values. Conceptually, it's really just storing the integer number of microseconds since the start of "year number 1" of the proleptic Gregorian calendar. All those named fields can be viewed as just representing that integer in a very messy mixed-radix notation. That's not just academic. For example, if you stare at the Python code for the implementation of datetime + timedelta (datetime.__add__), you'll see that it converts the datetime _to_ a timedelta first (which is another way of representing an integer number of microseconds). Of course that conversion has to be done "by hand" since it makes no sense in the user-visible model. But, internally,datetimes, dates, times, and timedeltas are all sometimes viewed as just complicated ways of spelling "some number of microseconds". It's cute ;-)

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Alexander Belopolsky 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 10:24 PM, Tim Peters <tim.peters(a)gmail.com> wrote: > [Chris Barker <chris.barker(a)noaa.gov>] >> what the heck would that mean? the timedeltas already do that -- what they >> don't do is calendar maniplutaions -- things like "this same time tomorrow" >> -- or tomorrow morning at 7:00am. > > We've been through this before, and it's been discussed literally > dozens of times on python-dev recently. So I refuse to use any "word > salad" to try to describe anything anymore - nobody can hear anything > anyone else says ;-) Maybe the following session log (ran from the root of the source distribution) will get the message across: >>> with open('Doc/includes/tzinfo-examples.py') as f: ... exec(f.read()) ... >>> t = datetime(2014,11,1,12,tzinfo=Eastern) >>> t.strftime('%c %Z %z') 'Sat Nov 1 12:00:00 2014 EDT -0400' >>> t += timedelta(1) >>> t.strftime('%c %Z %z') 'Sun Nov 2 12:00:00 2014 EST -0500' This is unmodified Python 3.4.3 running an official documentation example.

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Tim Peters 29 Jul '15

29 Jul '15

[Felipe Ochoa] >> ... >> B. Implement `.get_offset_time(self, hours, minutes, seconds, ms)` and >> `.compute_duration(self, other_dt)` methods for "aware arithmetic." (I.e., >> congruent to 'convert to UTC, increment, convert back') [Chris Barker <chris.barker(a)noaa.gov>] > what the heck would that mean? the timedeltas already do that -- what they > don't do is calendar maniplutaions -- things like "this same time tomorrow" > -- or tomorrow morning at 7:00am. We've been through this before, and it's been discussed literally dozens of times on python-dev recently. So I refuse to use any "word salad" to try to describe anything anymore - nobody can hear anything anyone else says ;-) Start with a datetime object representing day 23 in some month and some year, and time 4:56. Add timedelta(hours=24) to that, or add timedelta(days=1) to that, or (there are many other possible spellings of the same thing) and you will ALWAYS get back a timedelta object representing day 24 in the same month, the same year, the same time (4:56), and with the same tzinfo member. It makes no difference which year, no difference which month, and no difference which tzinfo member you started with. You always get back "tomorrow at the same time (as shown on the local clock)". And, yes, the two _may_ be 23 hours apart, or 24 hours, or 25 hours, or even (e.g.,) 23 hours and 15 minutes apart as measured by a stopwatch, depending on the rules implemented by the tzinfo member. The tzinfo member is never consulted about anything in a datetime + timedelta operation. The tzinfo member is ignored (apart from being copied to the result datetime). The arithmetic is done exactly the same as for a naive datetime object. I think you're calling that a "calendar operation". Other people call it "naive datetime arithmetic", some call it "broken", Guido calls it a feature of his "naive time" design, some call it the "a day is always 24 hours rule", and some call it the "s day is not always 24 hours rule". That's why it's impossible to understand anyone without an excruciatingly detailed example ;-) In any case, that's what Python has always done. It won't change, not even in the imaginary "backward compatibility is irrelevant" Python 4 (because Guido still favors this design). That's for the builtin datetime module and its already-existing objects and operations. There's no backward-compatibility constraint on anything new, of course.

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Chris Barker 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 7:04 PM, Alexander Belopolsky < alexander.belopolsky(a)gmail.com> wrote: > On Tue, Jul 28, 2015 at 9:54 PM, Chris Barker <chris.barker(a)noaa.gov> > wrote: > > 1) timespan since an epoch: -- i.e seconds since Jan 1, 1970 00:00. This > is > > how the datetime object keeps it internally. > > No, it is not. You are probably confusing stdlib datetime with > numpy.datetime64. > or who knows what else! I've been involved with far toomany datetime conversations lately! > The stdlib datetime keeps broken down year, month, day, hour, minute, > second > and microsecond values. > Thanks for the clarification -- that would make calendar manipulations easier, I suppose. And I can see why anyone would even consider storing in particular time zone format. Sorry to confuse things. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Alexander Belopolsky 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 9:54 PM, Chris Barker <chris.barker(a)noaa.gov> wrote: > 1) timespan since an epoch: -- i.e seconds since Jan 1, 1970 00:00. This is > how the datetime object keeps it internally. No, it is not. You are probably confusing stdlib datetime with numpy.datetime64. The stdlib datetime keeps broken down year, month, day, hour, minute, second and microsecond values.

1 0

Re: [Datetime-SIG] What's are the issues?
by Chris Barker 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 6:43 PM, Chris Angelico <rosuav(a)gmail.com> wrote: What if the existing timedelta is left as-is, and a new calendardelta > type is introduced? Instead of folding everything down to a simple > microsecond count, it would retain the exact values it was given, sure -- that would be one way to do it -- maybe good way. But it would be a lot more complicated than just keeping days. Lots to hash out for API and functionality at the corner cases (the old what's one year after Feb 29th? kinds of questions) > When you add a > calendardelta to an aware datetime, it performs what you're describing > as calendar arithmetic, actually, no reason you couldn't do calendar manipulations on naive datetimes, too (they wouldn't get any ugly DST issues, but otherwise the same). One reason I don't like calling the arithmetic naive or not.... > adding > calendardelta(days=1) would never affect the hour/minute/second shown, > unless it would otherwise create an illegal time. exactly -- all those corner cases would need to be worked out. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

Re: [Datetime-SIG] Datetime arithmetic proposal
by Chris Barker 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 5:15 PM, Felipe Ochoa <felipe.nospam.ochoa(a)gmail.com > wrote: > 1. We cannot break backwards-compatibility (This point has been beaten to > death) > OK then -- then timedelta is NOT going to be fundamentally re-denied as something else altogether -- good! > 2. There are valid use-cases for wanting datetimes with and without > timezone info, and for doing naive and aware math. All 3 legal combinations > of the two (excluding naive dt with aware deltas :-)) have valid use-cases > sure -- but the datetime module does not have to support them all, and they don't all have to be supported with the same objects. semantics again -- there is no such thing as naive or aware math -- there are timespan computations and calendar manipulations.... > 3. [new point] If I'm a new user looking in the library for a solution to > my problem, my use-case is likely to appear "obviously correct", and I'll > be shocked if Python doesn't support it > True for all possible libraries :-) -- though I suppose we have a special obligation 'cause we claim that "batteries are included" > 4. [new point] The non-trivial nature of all 3 possibilities makes it > advantageous to include in the standard library > There are a lot of non-trivial, and fairly common things that are not included in the stdlib. But I'm all for including calender manipulations in the stlib -- just not for altering what timedelta means, and trying to be realistic that that's actually a big project, with lots of ugly corner cases and API decisions. And there is nothing about that in the PEP, by the way. As far as I can see, the PEP is about including ready to go, standard tzinfo object in the stdlib -- whereas when dateime was introduced, it provided a mechanism, but no implimentation. This is about providing an implementation. A. Keep the current mechanics as-is. `+` and `-` result in "naive" datetime > arithmetic > *sigh* about the wording, but yes. > B. Implement `.get_offset_time(self, hours, minutes, seconds, ms)` and > `.compute_duration(self, other_dt)` methods for "aware arithmetic." (I.e., > congruent to 'convert to UTC, increment, convert back') > what the heck would that mean? the timedeltas already do that -- what they don't do is calendar maniplutaions -- things like "this same time tomorrow" -- or tomorrow morning at 7:00am. Maybe this will help people conceptualize: timedeltas represent what is measured by a stopwatch. datetimes represent what is represented by a calendar and a watch. If you start your stopwatch at a particular date indicated by your calendar, and particular time indicated by your watch, and run it for 24 hours (one day) you can then look at your calendar and watch and see what the datetime is -- and if you crossed a DST border, it may not be the _same_ time on your watch. And if you start your stopwatch at 2:00pm one day, and look at it again at 2:pm the next day, it will say 24 hours, unless you crossed a DST boundary, in which case it will say 23 or 25 hours -- but would that surprise anyone? > C. Come up with a new name (or clearly define naive/aware arithmetic) to > disambiguate the two > sure -- I'll vote to not use the term "aware arithmetic" -- it's not really arithmetic -- "calendar operations"? and the datetimes can be naive, but the operation is neither naive nor aware -- a time span is a time span is a time span.... Another thought here that may help: you can "encode" a datetime in at least two ways: 1) timespan since an epoch: -- i.e seconds since Jan 1, 1970 00:00. This is how the datetime object keeps it internally. 2) Calendar time in some calendar and time zone: year, month, day, hour, min, sec. The "magic" of the datetime object all happens in the conversion between the two. (side note -- timedeltas have no calendar concept at all) most time math is most easily done in encoding (1) -- that's (one reason) why it's used most of the time. But if you want to do calendar manipluations, you are better off working in the calendar encoding -- "two days from now" is much easier to do there. So I think when people are thinking about "time zone aware math" what they really are thinking about is calendar encoded math. By the way, is anyone proposing to add Calendar manipulations to the stdlib? I understand that folks want that, and apparently some people got confused that better time zone support meant that -- but is anyone proposing to do it? If anyone is, then it's probably a topic for Python-ideas and an entirely different PEP. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

Re: [Datetime-SIG] What's are the issues?
by Chris Barker - NOAA Federal 29 Jul '15

29 Jul '15

1. Datetime arithmetic >> a) datetime +/- timedelta >> ii. Aware (with tzinfo) datetime objects > > If the math is done in UTC, then there is no issue, yes? the timedelta is > added or subtracted, and the result has the same tzinfo object. If I've understood correctly, this is one of the bigger points of contention. Specifically, the following operation is quite polarizing: `datetime(2015, 3, 7, 13, 0, tzinfo=zoneinfo('America/New_York') ) + timedelta(days=1)` It raises the following two questions: * Should `days=1` be interpreted the same as `hours=24`? Yes, absolutely! * If it is interpreted the same, should the addition be "naive" (not adjusting for the jump from 1:59 to 3:00) or "aware" If it is 24 hours, then there's no question. So the question really is: should the datetime module provide what I've been calling "calendar operations" ? Yes, that would be very useful, but it's a heck of a lot more complicated than just redefine what a timedelta(days=1) means. And particularly with backward compatibility in mind, it definitely should not be done with that API. If people really are THAT prone to confusion that a day means 24 hours, then we should consider deprecating "days" as a valid unit for a timedelta. Current semantics call for the answers to be "yes" and "naive" Here are the semantics again -- there is nothing naive about subtracting and adding 24 hours. The problem here is that people are conflating a "day" meaning 24 hours and a "day" meaning a numbered entity on a calendar. The two are related, but totally different in meaning. And sorry to get pedantic here, but backwards compatibility aside, I, for one, would be very surprised if a timedelta(days=1) ever meant anything other than 24 hours depending on context! b) datetime - datetime > > ii. When both have same tzinfo > > Just do the math. Store as UTC makes it easy. return a timedelta. This is another contentious issue. What do you want `datetime(2015, 3, 8, 13, tzinfo=zoneinfo('America/New_York')) - datetime(2015, 3, 7, 13, tzinfo=zoneinfo('America/New_York'))` to be? There's a camp for `timedelta(seconds=24 * 60 * 60)` and a Assuming I've got the DST thing right here -- god no! In that time zone, those time stamps are NOT 24 hrs apart! You'd better not mess with that!!!! I'm going to indulge for a minute here: IF there was such a thing as a calendar_delta object of some sort, then you might want one with a value of one day there. But certainly not anything with a value of 24 *3600 seconds! Does anyone really want that? Note that this is s good argument for what calendar operations are so complex. -Chris

2 1

Re: [Datetime-SIG] What's are the issues?
by Tim Peters 29 Jul '15

29 Jul '15

[Felipe Ochoa <felipe.nospam.ochoa(a)gmail.com>] > ... > I'm less clear on the issue here, but Isaac seemed to disagree with the > algorithm in datetime.py > (https://hg.python.org/cpython/file/v3.5.0b1/Lib/datetime.py#l1935) > [http://code.activestate.com/lists/python-dev/137300/] The algorithm is fine, so far as it goes (with which Isaac will agree). "So how far does it go?" ;-) It handles DST transitions correctly in all known timezones(*). But, as the docs have always pointed out, it (specifically .fromutc(), which is at the base of all relevant calculations) doesn't even try to handle transitions due to _other_ causes. The only other such cause I'm aware of is when a timezone's _base_ offset from UTC changes (not a temporary DST transition, the timezone's "standard" offset changes permanently (meaning until someone decides to change it again)). Many timezones have suffered such a fundamental redefinition at least once. There is no pattern whatsoever to when such things occur - seems it's driven purely by local politics. And for any given timezone it typically happens at most once across its lifetime to date. However, while rare, such changes do still happen, and a "wholly correct" implementation needs to cater to them too (meaning a more general .fromutc() implementation needs to be crafted, and the docs updated to match the new scheme). >From there, Isaac went on to define an approach to a more general method, which could obviously handle that kind of thing too. How practical that approach can be made remains unknown, because nobody yet (that I know of) has tried to produce corresponding code. The mathematical abstractions in his writeup appear far more general than actually needed for any timezone yet invented (excepting the absurd "RIyadh Solar" timezone experiment, which was never actually used by anyone in the real world); stronger assumptions could probably allow more efficient implementation. (*) By "timezone" I mean whatever rules a specific tzinfo instance implements. I have a wholly Python-centric view of everything ;-)

1 0

Datetime arithmetic proposal
by Felipe Ochoa 29 Jul '15

29 Jul '15

My viewpoint (specifically on point 1 from my other email): 1. We cannot break backwards-compatibility (This point has been beaten to death) 2. There are valid use-cases for wanting datetimes with and without timezone info, and for doing naive and aware math. All 3 legal combinations of the two (excluding naive dt with aware deltas :-)) have valid use-cases 3. [new point] If I'm a new user looking in the library for a solution to my problem, my use-case is likely to appear "obviously correct", and I'll be shocked if Python doesn't support it 4. [new point] The non-trivial nature of all 3 possibilities makes it advantageous to include in the standard library With that said, my proposal would be to: A. Keep the current mechanics as-is. `+` and `-` result in "naive" datetime arithmetic B. Implement `.get_offset_time(self, hours, minutes, seconds, ms)` and `.compute_duration(self, other_dt)` methods for "aware arithmetic." (I.e., congruent to 'convert to UTC, increment, convert back') C. Come up with a new name (or clearly define naive/aware arithmetic) to disambiguate the two D. Document these differences clearly Downsides: * clunkyness of doing aware arithmetic * `aware_dt + delta` can result in a nonexistent or ambiguous time * `aware_dt_1 - aware_dt_2` behaves differently depending on the underlying timezones * [ fill in the blank -- I'm sure I'm missing many more ] -Felipe

1 0

Re: [Datetime-SIG] What's are the issues?
by Felipe Ochoa 29 Jul '15

29 Jul '15

In the interest of continuing to lay the foundation for discussion, let me build on Chris's recap before giving my viewpoint on the issue: 1. Datetime arithmetic >> a) datetime +/- timedelta >> ii. Aware (with tzinfo) datetime objects > > If the math is done in UTC, then there is no issue, yes? the timedelta is > added or subtracted, and the result has the same tzinfo object. If I've understood correctly, this is one of the bigger points of contention. Specifically, the following operation is quite polarizing: `datetime(2015, 3, 7, 13, 0, tzinfo=zoneinfo('America/New_York') ) + timedelta(days=1)` It raises the following two questions: * Should `days=1` be interpreted the same as `hours=24`? * If it is interpreted the same, should the addition be "naive" (not adjusting for the jump from 1:59 to 3:00) or "aware" Current semantics call for the answers to be "yes" and "naive" > b) datetime - datetime > > ii. When both have same tzinfo > > Just do the math. Store as UTC makes it easy. return a timedelta. This is another contentious issue. What do you want `datetime(2015, 3, 8, 13, tzinfo=zoneinfo('America/New_York')) - datetime(2015, 3, 7, 13, tzinfo=zoneinfo('America/New_York'))` to be? There's a camp for `timedelta(seconds=24 * 60 * 60)` and a camp for `timedelta(seconds=23 * 60 * 60)`, though I think everyone agrees that `(dt + delta) - dt == delta` is necessary. 2. Inter-timezone conversion > > b) algorithm specification > > what part of the algorithm? seems pretty straightforward to me (the > algorithm, anyway) I'm less clear on the issue here, but Isaac seemed to disagree with the algorithm in datetime.py ( https://hg.python.org/cpython/file/v3.5.0b1/Lib/datetime.py#l1935) [ http://code.activestate.com/lists/python-dev/137300/]

1 0

Re: [Datetime-SIG] What's are the issues?
by Alexander Belopolsky 29 Jul '15

29 Jul '15

On Tue, Jul 28, 2015 at 5:38 PM, Felipe Ochoa <felipe.nospam.ochoa(a)gmail.com> wrote: > > Here is my best shot at categorizing the different outstanding design issues: > I would like to make sure that everyone is aware that we are not designing in a vacuum. Unlike Python 2, Python 3 already has a decent amount of timezone support. In fact, with some small exceptions, I believe Python 3 datetime has a fairly complete support for UTC and system local timezones. For the items below, I will try to briefly describe the status quo > 1. Datetime arithmetic > a) datetime +/- timedelta > i. Naive (no tzinfo) datetime objects This has been supported from the very beginning: >>> t = datetime.now() >>> t datetime.datetime(2015, 7, 28, 19, 18, 59, 699588) >>> t + timedelta(1) datetime.datetime(2015, 7, 29, 19, 18, 59, 699588) (no surprises here) > ii. Aware (with tzinfo) datetime objects This works fine with the builtin timezone instances: >>> t = datetime(2014,11,1,16,tzinfo=timezone.utc).astimezone() >>> t.strftime("%c %Z %z") 'Sat Nov 1 12:00:00 2014 EDT -0400' >>> u = t + timedelta(1) >>> u.strftime("%c %Z %z") 'Sun Nov 2 12:00:00 2014 EDT -0400' This may be a little surprising to those expecting to see DST to end, but the legal time is one method call away: >>> u.astimezone().strftime("%c %Z %z") 'Sun Nov 2 11:00:00 2014 EST -0500' > b) datetime - datetime > i. When both are naive (Trust me: this part works!) > ii. When both have same tzinfo >>> u - t datetime.timedelta(1) (Works.) > iii. When the two have different tzinfos >>> u - datetime(2014,11,1,16,tzinfo=timezone.utc) datetime.timedelta(1) (Works as well.) > 2. Inter-timezone conversion UTC to local just works. Local to UTC "kind of" works: >>> t = datetime.now() >>> u = datetime.fromtimestamp(t.timestamp(), timezone.utc) >>> u.strftime("%c %Z") 'Tue Jul 28 23:40:38 2015 UTC+00:00' > a) adding an isdst (or which) attribute > b) algorithm specification > > I think 1ai, 2bi, and 1biii are uncontroversial at this point, and 2a we have consensus on the need but not the name >

1 0

Re: [Datetime-SIG] What's are the issues?
by Ethan Furman 28 Jul '15

28 Jul '15

On 07/28/2015 01:58 PM, Chris Barker wrote: > Now that we have a datetime list, maybe it's time to re-center ourselves. > > I know I got all caught up in the semantics and details, but I just now gave the PEP another read-through, and, well, it looks good to me. > > What are the issues still to be resolved? I think the biggest issue is that Lennart was trying to change the way datetimes work, which a. would be a compatibility break b. is not clear from the PEP I think a lot of confusion comes from the necessity of using timedeltas as a time measurement device, even when we aren't really using them that way -- meaning when we add a timedelta of 7 days to go from Jun 27 09:00 to Jul 4 09:00 we (usually) don't care if that is exactly 7 * 24 hours later. I think the ideal solution would be to have tz-aware datetimes able to switch their timezone as needed, but that would also be a backwards-compatibility break -- not to mention the need to be able to specify whether one wants the "calendar" operation (7am is still 7am) or the actual duration operation (7am may be 6am or 8am if a clock change occurs). On the gripping hand we already have the calendar semantics, so perhaps just a whole new type of datetime would be best? -- ~Ethan~

1 0

Re: [Datetime-SIG] What's are the issues?
by Chris Barker 28 Jul '15

28 Jul '15

On Tue, Jul 28, 2015 at 2:38 PM, Felipe Ochoa <felipe.nospam.ochoa(a)gmail.com > wrote: > Here is my best shot at categorizing the different outstanding design > issues: > great, thanks! My thoughts: (some obvious) NOTE: I'm assuming that timedelta is the same as it is now -- i.e. "naive", or simply representing a timespan nothing t do with calendars. 1. Datetime arithmetic > a) datetime +/- timedelta > i. Naive (no tzinfo) datetime objects > same as now > ii. Aware (with tzinfo) datetime objects > If the math is done in UTC, then there is no issue, yes? the timedelta is added or subtracted, and the result has the same tzinfo object. > b) datetime - datetime > i. When both are naive > same as now. > ii. When both have same tzinfo > Just do the math. Store as UTC makes it easy. return a timedelta. > iii. When the two have different tzinfos > Do the math in UTC, return a timedelta. > 2. Inter-timezone conversion > a) adding an isdst (or which) attribute > yup -- and I've bike-shedeed enough without getting into what to call it! > b) algorithm specification > what part of the algorithm? seems pretty straightforward to me (the algorithm, anyway) So really a lot of bike-shedding on pyton-dev ;-) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

Re: [Datetime-SIG] What's are the issues?
by Felipe Ochoa 28 Jul '15

28 Jul '15

Here is my best shot at categorizing the different outstanding design issues: 1. Datetime arithmetic a) datetime +/- timedelta i. Naive (no tzinfo) datetime objects ii. Aware (with tzinfo) datetime objects b) datetime - datetime i. When both are naive ii. When both have same tzinfo iii. When the two have different tzinfos 2. Inter-timezone conversion a) adding an isdst (or which) attribute b) algorithm specification I think 1ai, 2bi, and 1biii are uncontroversial at this point, and 2a we have consensus on the need but not the name On 28 July 2015 at 15:58, Chris Barker <chris.barker(a)noaa.gov> wrote: > Hi folks, > > Now that we have a datetime list, maybe it's time to rre-center ourselves. > > I know I got all caught up in the semantics an details, but I just now > gave the PEP another read-through, and, well, it looks good to me. > > What are the issues still to be resolved? > > -Chris > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker(a)noaa.gov > > _______________________________________________ > Datetime-SIG mailing list > Datetime-SIG(a)python.org > https://mail.python.org/mailman/listinfo/datetime-sig > >

1 0

What's are the issues?
by Chris Barker 28 Jul '15

28 Jul '15

Hi folks, Now that we have a datetime list, maybe it's time to rre-center ourselves. I know I got all caught up in the semantics an details, but I just now gave the PEP another read-through, and, well, it looks good to me. What are the issues still to be resolved? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

Calendar vs timespan calculations...
by Chris Barker 28 Jul '15

28 Jul '15

Moved from python-dev to the datetime SIG TL;DR: There are are two types of operations on datetimes: adding subtracting actual time units (multiples of seconds) and calendar operations: things like "this time the next day". I think the the datetime module should only support the former -- i.e. the same old timedelta we have. In that context, storing datetimes in UTC, doing all operations there, and converting back makes the most sense. But for calendar manipulations, they need to be done in a time-zone aware way. If you're really interested, I got quite carried away below.... On Mon, Jul 27, 2015 at 10:03 PM, Lennart Regebro <regebro(a)gmail.com> wrote: > On Tue, Jul 28, 2015 at 12:03 AM, Tim Peters <tim.peters(a)gmail.com> wrote: > > timedelta objects only store days, seconds, and microseconds, > > Except that they don't actually store days. They store 24 hour > periods, which, because of timezones changing, is not the same thing. > yes, it is ;-) And as you have repeated > many times now, the datetime module's arithmetic is "naive" ie, it > assumes that one day is always 24 hours. The problem with that > assumption is that it isn't true. > it's not an assumption, it's a definition. We have a serious semantic problem here -- and I _think_ it's the source of almost all the discussion in this thread -- I'm not sure what to do about it though. Tim understands this stuff, Lennart understands this stuff, I'm pretty sure I understand this stuff -- I'm not sure about everyone else on this discussion, but they probably all understand it, too -- the only thing any of us doesn't understand is what the heck anyone is talking about! Maybe use Tim's approach of "naive" -- so a naive day is exactly 24 hours period end of story -- this is the definition. But what do we call what I've been trying to refer to as "calendar" operations? A summary for (maybe) some clarity: We have this nifty model of a continuous time axis -- moving along at a steady rate from the beginning of the Universe to the end of the Universe. Modulo relativity, it works pretty well. Then we have units of time spans: in SI units it's seconds. then a bunch of other units that are clearly defined in terms of seconds: minutes (60 s) hours (60 min) days (24 hrs). And of course milliseconds and microseconds. Then we have Calendars: this is the year, month, day, etc. we are all familiar with (and the hours in that day 4:00 o'clock pm, etc) -- confusing here is that we use the same word for "day" and "hour" as part of a calendar descriptions AND also as timespan units -- but they are NOT the same thing (yes, they are related). Calendars are how we map a nice human understandable (and historically based) naem onto the theoretically time axis. Being both human-oriented and with a lot of historical baggage, calendar naming is designed to fit more or less with relationship between the earth and the sun. So we want the Solstices to fall around the same date every year, for instance, and we want 1200 hrs to be around the middle of the day. This is why it all gets ugly, because the various celestial phenomena aren't nice integers multiples of each-other (hence the need for leap years) and even constant (hence the need for leap seconds). And, of course, the earth is round, and the sun's relative position to each point on earth is different, hence the need for time zones. (then add political differences for it to get really ugly. So: I think there are more or less two types of manipulations one might need: What is currently supported by the datetime module, and I think Tim is referring to as "naive" time operations: adding, subtracting, units of time along the theoretical time axis. This is actually really simple math -- as Tim points out all the timedelta object really is is a fancy integer. Then there are what I am calling "calendar" operations -- these are operations that only make sense with a calendar (and, in fact a timezone, I think). this is operations like: "the same time two days later" -- this is not the same as moving two days (48 hrs) along the time axis -- it simply is not. It is a shame that we use the name "day" to refer to both 24 hours along the time axis and enumeration of sunrises in a month -- or, the thing we use on calendars. There has been a lot of chatter about "tomorrow" or "adding three days", as these bring up the ugly DST issues, but once you add one calendaring operation, people will want (and they should) more: next month, next year, or even uglier, next business day, etc. These are very useful things, but I argue they belong, as a unit, in a separate package -- maybe for potential inclusion in the standard library, but I don't think that's on the table now. (and doesn't dateutil support many of these?) Now on to time zones: the datetime package is useful because it not only supports time-axis arithmetic (really pretty trivial -- it's just integer arithmetic), but it supports translating from time-axis units (microseconds since some epoch) to Calendar units: year, month, day, hour, minute, seconds, microseconds (Using the (proleptic?) Gregorian Calendar -- note that different Calendars are a whole other ball of wax! This is where the magic is (or really where the ugly code is) , and where tiem zones come in. UTC time is the calendar time at one longitude on the earth. (ignoring leap seconds for the moment) it is relatively simple: continuous, etc, no DST, no changing definitions, etc. It is a useful reference system for this reason. math, and all that, are easier in this zone. By default, Python datetime objects are "naive" -- meaning they know nothing of timezones or daylight savings -- they can convert back and forth between the internal representation (time span since an epoch) and human calendar times (Gregorian, anyway). It turns out that you can use naive datetimes as UTC time -- there really is no difference, until you want to convert to a different time zone -- with UTC, you may be able to do that, with naive, you can't do that unless you specify what time zone the time is, and then it's no longer naive ;-) But how should pyton handle time zones? Given all the ugliness of DST and changing time zones, and all that, UTC is the lingua franca of time -- time zones are defined by how the are offset from UTC, and in UTC (as in naive), math is relatively easy. So the best way to handle time zones is to store and manipulate everything in UTC, and then convert to/from the calendar representation using the time zone, when the user needs (or provides). I've been trying to figure out what all the confusion, discussion has been about, and I think it's this: If you want to do a Calendar operation, like "this time tomorrow", then THAT is best done in a timezone aware way -- in particular, a DST-aware way. i.e. from 12:00 one day to 12:00 the next day will generally be 24 hours, but might be 23 or 25 hours if crossing a DST transition. But if we aren't supporting those operations, we don't need to worry about that now. -Chris > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0