Mailman 3 August 2015 - Datetime-SIG

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Łukasz Rekucki 02 Aug '15

02 Aug '15

On Sunday, August 2, 2015, Alexander Belopolsky < alexander.belopolsky(a)gmail.com <javascript:_e(%7B%7D,'cvml','alexander.belopolsky(a)gmail.com');>> wrote: > On Sun, Aug 2, 2015 at 9:46 AM, Guido van Rossum <guido(a)python.org> wrote: > > There's a simpler reason for ignoring leap seconds in datetime: Python's > > wall clock is mappable to POSIX timestamps, which also ignore leap > seconds > > (the reason being Tim's long explanation :-). > > Note that if you combine several recently made proposals, you can have > a fully backward-compatible solution for leap seconds. I don't have much time to respond now, but please don't add leap seconds. People are actively working to get rid of them entirely and they are never useful on the business level which this module is aimed at. > You can > simply use time(23, 59, ss, us, first=False) to stand for times in the > 23:60 minute and delegate arithmetics to the tzinfo object: > > def __add__(self, delta): > try: > add = self.tzinfo.add > except AttributeError: > # current logic > else: > return add(self, delta) > > Same for `__sub__` and possibly `strftime`, `isoformat`, etc. if you > want time(23, 59, ss, us, first=False) to be printed as 23:60:ss.us. > > Note that Olson database contains information about leap seconds, so > once you have access to that and a way to spell 23:60 (as the repeated > 23:59!) you can easily implement "correct" UTC timezone. > > On the other hand, I would like to proceed in baby steps. Let's > first implement the means to disambiguate repeated times and various > improvements that it will allow to the existing datetime > functionality. Meanwhile the developers of timezone libraries will > hopefully embrace the new feature and improve their offerings. > > Hopefully, by the time we are ready to distribute a full TZ database > with Python, the problem of distribution will be solved by IETF [1]. > If their solution (as expected) includes leap second information, we > can find a way to give Python users access to it in one way or > another. > > [1]: https://tools.ietf.org/html/draft-ietf-tzdist-service-11 > _______________________________________________ > Datetime-SIG mailing list > Datetime-SIG(a)python.org > https://mail.python.org/mailman/listinfo/datetime-sig > The PSF Code of Conduct applies to this mailing list: > https://www.python.org/psf/codeofconduct/ > -- Łukasz Rekucki

1 0

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Tim Peters 02 Aug '15

02 Aug '15

[Guido] > There's a simpler reason for ignoring leap seconds in datetime: Python's > wall clock is mappable to POSIX timestamps, which also ignore leap seconds > (the reason being Tim's long explanation :-). Ya, but I wanted to give some reasons that make actual sense ;-) Because when time wonks get agitated about this, it's just like American politics: both sides dig in and endlessly repeat the same talking points with ever-increasing volume. "Because POSIX said so" was smashed by the ever-affable Daniel Bernstein a long time ago: http://cr.yp.to/proto/utctai.html ... The main obstacle is POSIX. POSIX is a ``standard'' designed by a vendor consortium several years ago to eliminate progress and protect the installed base. The behavior of the broken localtime() libraries was documented and turned into a POSIX requirement. Fortunately, the POSIX rules are so outrageously dumb---for example, they require that 2100 be a leap year, contradicting the Gregorian calendar---that no self-respecting engineer would obey them. See? You've been paid off by vendors to eliminate progress, to protect their ill-gotten gains. datetime is just another tool of capitalist pigs to ensure they increase their profits at the expense of the people. Just thought I'd put that out there first ;-) boldly-speaking-truth-to-power-since-2015-ly y'rs - tmi

1 0

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Alexander Belopolsky 02 Aug '15

02 Aug '15

On Sun, Aug 2, 2015 at 9:46 AM, Guido van Rossum <guido(a)python.org> wrote: > There's a simpler reason for ignoring leap seconds in datetime: Python's > wall clock is mappable to POSIX timestamps, which also ignore leap seconds > (the reason being Tim's long explanation :-). Note that if you combine several recently made proposals, you can have a fully backward-compatible solution for leap seconds. You can simply use time(23, 59, ss, us, first=False) to stand for times in the 23:60 minute and delegate arithmetics to the tzinfo object: def __add__(self, delta): try: add = self.tzinfo.add except AttributeError: # current logic else: return add(self, delta) Same for `__sub__` and possibly `strftime`, `isoformat`, etc. if you want time(23, 59, ss, us, first=False) to be printed as 23:60:ss.us. Note that Olson database contains information about leap seconds, so once you have access to that and a way to spell 23:60 (as the repeated 23:59!) you can easily implement "correct" UTC timezone. On the other hand, I would like to proceed in baby steps. Let's first implement the means to disambiguate repeated times and various improvements that it will allow to the existing datetime functionality. Meanwhile the developers of timezone libraries will hopefully embrace the new feature and improve their offerings. Hopefully, by the time we are ready to distribute a full TZ database with Python, the problem of distribution will be solved by IETF [1]. If their solution (as expected) includes leap second information, we can find a way to give Python users access to it in one way or another. [1]: https://tools.ietf.org/html/draft-ietf-tzdist-service-11

1 0

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Guido van Rossum 02 Aug '15

02 Aug '15

There's a simpler reason for ignoring leap seconds in datetime: Python's wall clock is mappable to POSIX timestamps, which also ignore leap seconds (the reason being Tim's long explanation :-). On Sun, Aug 2, 2015 at 11:11 AM, Tim Peters <tim.peters(a)gmail.com> wrote: > [Carl Meyer <carl(a)oddbird.net>] > > Sorry for the delay - there's just too much here to keep up with. I > enjoyed and appreciated your essay, and will take time to give you > straight answers about the "most important" question you asked. Of > course I don't speak for Guido. Although I often do ;-) > > > ... > > In order to defend the current model as coherent, one has to discard one > > of the following points, and (despite having read every message in all > > the related threads), I am still not clear precisely which one of these > > Tim et al consider untrue or expendable: > > > > 1) A datetime with a tzinfo member that carries both a timezone and a > > specific UTC offset within that timezone (e.g. a pytz timezone instance) > > corresponds precisely and unambiguously to a single instant in > > astronomical time (as well as carrying additional information). > > datetime had no intent to support "astronomical time" in any way, > shape or form. It's no coincidence that, in Guido's first message > about "naive time": > > https://mail.python.org/pipermail/python-dev/2002-March/020648.html > > he talked about "for most *business* uses of date and time". datetime > development was suggested & funded by Zope Corporation, which mostly > works to meet other businesses' "content management" needs. The use > cases collected were overwhelmingly from the commercial business > world. > > Astronomical time systems weren't on the table. In this respect, it's > important to realize that while Python 3.2 finally supplied a concrete > instance (of a tzinfo subclass) as "the standard" UTC timezone object > (datetime.timezone.utc), that's still just an approximation: it > wholly ignores that real-life UTC suffers from leap seconds added (or, > perhaps some day also removed) at various times. Subtract two > datetimes in `utc`, and the duration returned may be off from real > life, but whether and by how much can only be determined by looking up > the history of leap second adjustments (made to real-life UTC). > > Those who suspect "Noda Time" is what they really want should note > that it ignores leap seconds too. As they say on their site, "We want > to solve the 99% case. > Noda Time doesn't support leap seconds, relativity or various other > subtleties around time lines." Although in the Zope community (which > mostly drove Python's datetime requirements), it was more the 99.997% > case ;-) > > If an astronomical union had funded the project instead ... > > > > 2) A timedelta object is clearly a Duration, not a Period, because > > timedelta(days=1), timedelta(hours=24), and timedelta(seconds=86400) > > result in indistinguishable objects. I think this point is > > uncontroversial; Tim has said several times that a timedelta is just a > > complicated representation of an integer number of microseconds. That's > > a Duration. > > That's my view, yes. Although these are "naive time" microseconds > too, with eternally fixed relation to all of naive time seconds, > minutes, hours, days and weeks. In real-life UTC, you can't even say > how long a minute is in seconds - "it depends". > > > > 3) If one has two datetime objects that each unambiguously correspond to > > an instant in real time, and one subtracts them and gets back an object > > which represents a Duration in real microseconds, the only reasonable > > content for that Duration is the elapsed microseconds in real time > between > > the two instants. > > Since there's no accounting for leap seconds, this cannot always be > true using tzinfo objects approximating real-life UTC, or any timezone > defined as offsetting real-life UTC. Which is all of 'em ;-) > > So what's the hangup with leap seconds? They're of no use to business > applications, but would introduce irregularities business logic is > ill-prepared to deal with. Same as DST transitions, leap-second > adjustments can create missing and ambiguous times on a local clock. > But unlike DST transitions, which occur in each jurisdiction at a time > picked to be minimally visible in the jurisdiction (wee hour on a > weekend), leap-second adjustments occur at a fixed UTC time, which is > usually "in the middle of the work day" in _some_ jurisdictions. For > that reason, when a leap second was inserted this year, some major > financial markets across the world - normally open at the adjustment > time! - shut down temporarily rather than risk a cascade of software > disasters: > > http://money.cnn.com/2015/06/29/technology/leap-second/ > > I'm glad they did. Example: The order in which trades are executed > (based on timestamps with sub-second resolution) can have legal > consequences. For example, a big customer calls a broker and tells > them to buy a million shares of Apple stock. The broker thinks "good > idea!". He tells his software to place the customer buy order, then > wait a millisecond, then send an order to buy a thousand shares for > his own account. That's legal. If the orders are placed in the > opposite order, it's illegal and the broker could go to jail ("front > running", placing his order first _knowing_ that a large order will > soon follow; the large order will certainly drive the stock price up, > benefiting the broker who bought before the thoroughly predictable > rise). > > Inserting a leap second causes the local clock to "repeat a second" in > its idea of time (just as "inserting an hour" at the end of DST causes > local clocks to repeat an hour) - or to blow up. A repeated second > could cause the orders in the example above to _appear_ to have > arrived in "the other" order. Even if the system time services report > a time like 13:59:60.000 (instead of repeating 13:59:59.000), lots of > software never expected to see such a thing. Who knows what may > happen? > > So I doubt datetime will ever use "real UTC". It's pretty horrid! > For another example, what will the UTC calendar date and time be 300 > million seconds from now? That's simply impossible to compute for > real UTC, not even in theory. Saying how many seconds away it will be > is trivial (300 million!), but the physical processes causing leap > second adjustments to UTC are chaotic - nobody can predict how many > leap second adjustments will be made to UTC over the next 300 million > seconds, or when, so there's no way to know what the UTC calendar date > and time will be then. It _can_ affect the calendar date-and-time > even for times just half a year in the future . Unless the definition > of UTC is changed yet again (dead serious proposals for which are > pending, supported by most participating countries): > > > https://en.wikipedia.org/wiki/Leap_second#Proposal_to_abolish_leap_seconds > > That page is also interesting for its account of various software > problems known to have been caused so far by leap-second adjustments. > > Anyway, under "real UTC" today, you could get an excellent > approximation of "real time durations" by subtracting, but would have > to accept that there is no fixed mapping between UTC timeline points > and calendar notations except for datetimes no later than about 3 > months from now (best I can tell, "they" don't promise to give more > than 3 month notice before the next leap second adjustment). > > Finally, I have to note the irony in asking anything about "real time" > ;-) What does "real time" mean? The most accurate clocks we have are > atomic clocks, but even when two are made as identically as possible - > even if we made two that kept _perfect_ time forever - they will > _appear_ to run at slightly different rates when placed at different > locations on Earth. That's at least due to gravitational time > dilation: relativistic effects matter at currently achievable > resolutions. As a result, current TAI time (the astonishingly uniform > "atomic time" measure from which today's definition of UTC is derived) > can't be known _as_ it happens: it's the output of an algorithm > (which consumes time!) that collects "elapsed seconds" from hundreds > of ultra-stable clocks around the globe, and averages them in a way to > make a highly informed, provably excellent guess at what they would > have said had they all been flawless, all at mean sea level altitude, > and all at 0 degrees Kelvin. This computed "TAI time" is out of date > by the time it's known, and typically disagrees (slightly) with most > of the clocks feeding into it. > > So the best measure of "real time" we have is a product of human > ingenuity. The closer to "plain old unadulterated real time as it > exists in nature" you want to get, the more contrived & bogglingly > complex the means needed to achieve it ;-) > > Everyone is settling for an approximation, because that's the best > that can be done. Naive time starts and stops with what most people > "already know". > > When UTC started mucking with leap seconds (it didn't always), the > computing world should have embraced TAI internally instead. TAI > suffers no adjustments of any kind, ever - it's just the running total > of SI seconds since the start of the TAI epoch, as determined by the > best clocks on Earth. In fact, it's very close to Python's "naive > time"! TAI uses the propleptic Gregorian calendar too (albeit > starting at a different epoch than year 1), and the TAI "day" is also > defined to be exactly 86400 SI seconds. The difference is that TAI's > Gregorian calendar will, over time, become unboundedly out of synch > with UTC's Gregorian calendar, as leap seconds pile up in the latter. > So far they're only 36 seconds out of synch. > > > ... > > To be clear, I'm not arguing that this behavior can now be changed in > > the existing library objects in a backwards-incompatible way. But > > accepting that it is lacking in internal coherence (rather than just > > being an "alternative and equally good model") would be useful in > > clarifying what kind of an implementation we actually want (IMO, > > something very much like JodaTime/NodaTime). And then can we figure out > > how to get there from here. > > I mentioned Noda Time before. Just looked up Joda-Time, and: > > http://joda-time.sourceforge.net/faq.html > > """ > Joda-Time does not support leap seconds. Leap seconds can be supported > by writing a new, specialized chronology, or by making a few > enhancements to the existing ZonedChronology class. In either case, > future versions of Joda-Time will not enable leap seconds by default. > Most applications have no need for it, and it might have additional > performance costs. > """ > > There's a pattern here: "almost all" people want nothing to do with > leap seconds, not even time library developers. That doesn't mean > they're right. But it doesn't mean they're wrong either ;-) Without > leap seconds, they're all approximating real-life UTC, and in the same > way Python's `utc` is. > _______________________________________________ > Datetime-SIG mailing list > Datetime-SIG(a)python.org > https://mail.python.org/mailman/listinfo/datetime-sig > The PSF Code of Conduct applies to this mailing list: > https://www.python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido)

1 0

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Carl Meyer 02 Aug '15

02 Aug '15

On 07/30/2015 10:46 AM, Chris Barker wrote: > There is "correct" and incorrect, but I"m not arguing that anything is > incorrect about the current behavior -- I thought a timedelta was a > duration, but I was wrong, it is a Period in units of days (I think!), > and sure it apparently does that right. No, it isn't a Period, and it doesn't "do that right". The current behavior _is_ incorrect (or at least lacking in internal coherence), and I don't think we can get clarity on what we want unless we acknowledge that. The discussion keeps getting sidetracked on the red herring of whether there are use cases for period arithmetic with "one day" defined as "same local time next day" as a period unit. Of course there are such use cases (and timedelta(1) can satisfy those use cases), but the current timedelta (as it behaves in arithmetic with tz-aware datetimes) is a hybrid that is not coherent considered either as a Period or a Duration. "Satisfying some use cases sometimes" is not sufficient for correctness; conceptual coherency matters too. Timedelta can only be considered a "Period object in units of days" if one accepts that the things it calls "hours", "minutes", and "seconds" are not really hours, minutes, or seconds, but rather fractional units of the "day-as-same-time-next-day" period that often (but not always) correspond to real hours, minutes, and seconds. I don't think this is a tenable explanation (and no one has attempted it), but you've just shown that it's one possible conclusion from defending the current model. Tim's valiant efforts notwithstanding, I don't think there is any coherent conceptual model that justifies the current behavior of timedelta. The _implementation_ can easily be explained of course (and Tim has done so very clearly, many times - I'd summarize it as "all arithmetic temporarily pretends all datetimes are naive, and then blindly reattaches the original tzinfo member"), but in terms of the underlying concepts, it makes no sense. In order to defend the current model as coherent, one has to discard one of the following points, and (despite having read every message in all the related threads), I am still not clear precisely which one of these Tim et al consider untrue or expendable: 1) A datetime with a tzinfo member that carries both a timezone and a specific UTC offset within that timezone (e.g. a pytz timezone instance) corresponds precisely and unambiguously to a single instant in astronomical time (as well as carrying additional information). 2) A timedelta object is clearly a Duration, not a Period, because timedelta(days=1), timedelta(hours=24), and timedelta(seconds=86400) result in indistinguishable objects. I think this point is uncontroversial; Tim has said several times that a timedelta is just a complicated representation of an integer number of microseconds. That's a Duration. 3) If one has two datetime objects that each unambiguously correspond to an instant in real time, and one subtracts them and gets back an object which represents a Duration in real microseconds, the only reasonable content for that Duration is the elapsed microseconds in real time between the two instants. Much virtual ink has been spilled over whether the behavior of "datetime + timedelta(days=1)" is correct, but this is an intentionally muddying case to consider, because there _are_ two perfectly reasonable interpretations of "add one day to a datetime". It's just that one of those interpretations (the Period one), which has been used to justify the current model, is inconsistent with _everything else_ about the behavior and implementation of timedelta. To be clear, I'm not arguing that this behavior can now be changed in the existing library objects in a backwards-incompatible way. But accepting that it is lacking in internal coherence (rather than just being an "alternative and equally good model") would be useful in clarifying what kind of an implementation we actually want (IMO, something very much like JodaTime/NodaTime). And then can we figure out how to get there from here. Carl

4 3

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Alexander Belopolsky 02 Aug '15

02 Aug '15

On Sat, Aug 1, 2015 at 1:16 AM, Tim Peters <tim.peters(a)gmail.com> wrote: >> A careful application will have to call tz.utcoffset() with both values of the >> flag and either warn about the default choice or ask the user for an >> additional input. > > As above, how can one programatically pick a valid default when faced > with a missing time? Suppose, on UTC time u=X, local clocks are advanced d > 0 units. Then the function L(u) that maps UTC time u to local time, can be written as L(u) = u + o + d * 1[u >= X] where o is the UTC offset before the transition and 1[] is the (Knuth?) indicator function. Let L0(u) = u + o and L1(u) = u + o + d. My proposal is that when t is between X + o and X + o + d and therefore t = L(u) has no solution, we should offer solutions to t = L0(u) and t = L1(t) instead. (These solutions are, BTW, t - o and t - o - d.) With the notation introduced so far, my "extended local-to-global function" xG(t, first=True) can be written as def xG(t, first=True): if X + o <= t < X + o + d: if first: return t - o else: return t - o - d .. # handle other times Note that xG(t, first=True) > xG(t, first=False) is the deliberate choice that makes it dead easy to detect that the returned values are not solutions of t = L(u).

1 0

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Ethan Furman 02 Aug '15

02 Aug '15

On 08/01/2015 02:01 PM, Alexander Belopolsky wrote: > On Sat, Aug 1, 2015 at 4:51 PM, Ethan Furman wrote: >> I just also think that part our "gold standard" is refusing the temptation >> to guess what the result should have been when handed a non-strict dt that >> is either ambiguous or non-existent. > > Ethan, you responded to a joke, but snipped the substantive argument. My apologies. I don't have much input about the actual algorithms to use, and, quite frankly, still find DST irritating and confusing. It wasn't until a couple years ago that somebody taught me "fall back, spring forward" so I could at least keep straight which way the clock was going to shift. So really my involvement here is as a user of these things -- and as a user I would be greatly frustrated with being able to hand an ambiguous/non-existent date/time to a strict dt/tzinfo and not get an error because sooner or later the guess would be wrong and I'd be wondering what happened. -- ~Ethan~

2 1

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Ethan Furman 02 Aug '15

02 Aug '15

On 08/01/2015 12:58 PM, Alexander Belopolsky wrote: > On Sat, Aug 1, 2015 at 3:40 PM, Ethan Furman wrote: >> If this new 'strict' support still has buggy behaviour around time shifts, >> why are we bothering? > > Because the new 'strict' way of dealing with timezones and datetime > arithmetics will become our "gold standard" and we will not tolerate > buggy behavior even in the corner cases? I have no doubt that adding and subtracting strict dt's will behave appropriately. I just also think that part our "gold standard" is refusing the temptation to guess what the result should have been when handed a non-strict dt that is either ambiguous or non-existent. -- ~Ethan~

2 1

Re: [Datetime-SIG] Calendar vs timespan calculations...
by Ethan Furman 02 Aug '15

02 Aug '15

On 08/01/2015 12:09 PM, Alexander Belopolsky wrote: > On Sat, Aug 1, 2015 at 2:34 PM, Ethan Furman <ethan(a)stoneleaf.us> wrote: >>> My goal here is to minimize the impact on the programs that are >>> already written and deployed. >> >> >> Why is this even a problem? Already written programs will not be using the >> new strict tzinfo. > > They will if their tzinfo provider switches to tzstrict. It is not our problem if a third-party library makes backwards incompatible changes. We will still be supporting all three types in the stlib: naive, aware, and strict. If this new 'strict' support still has buggy behaviour around time shifts, why are we bothering? -- ~Ethan~

2 1

Re: [Datetime-SIG] Local time disambiguation proposal
by Alexander Belopolsky 02 Aug '15

02 Aug '15

On Thu, Jul 30, 2015 at 12:43 PM, Alexander Belopolsky <alexander.belopolsky(a)gmail.com> wrote: > I would like to make a specific proposal on a narrow topic of how to > handle ambiguities inherent in representing time instances in common > time zones. I have now posted a first draft of the reference implementation at the bug tracker: Issue: http://bugs.python.org/issue24773 Patch: http://bugs.python.org/file40094/ltdf.patch I am also working on converting my write-up [1] for publication as a PEP. Please note that I have made substantial changes since my initial post, so if you comment on the proposal please make sure that you've read the latest version. [1]: https://github.com/abalkin/ltdf/blob/master/README.rst

1 0