Tim Peters tim.peters@gmail.com writes:
[Akira Li 4kir4.1i@gmail.com]
...
Nobody has said some apps don't need reliable conversions (to the contrary, that's the primary _point_ of PEP 495). Nobody has said some apps don't need timeline arithmetic - although I have said it's poor practice to even _try_ to do timeline arithmetic if an app isn't working in UTC or with naive datetimes. If an app is following best practice (UTC or naive datetimes), then timeline arithmetic is what they _always_ get (it's the same thing as classic arithmetic in those contexts).
I agree on the best practices here. I would prefer that __add__ would be forbidden for local timezones unless they have a fixed utc offset. But it might be too late for that now.
If __add__ is allowed for timezone-aware datetime objects then arithmetic "as though via conversion to utc time" is *equally valid* as the arithmetic "as though it is a timezone-naive datetime object".
Non-pytz timezones make mistake on the order of an hour regularly. It is *three orders of magnitude larger* than a second. It is a different class of errors. The code that can't handle ~1s errors over short period of time should use time.monotonic() anyway.
Apps that care about leap seconds _should_ be using TAI. Apps that want timeline arithmetic _should_ be using UTC. Unfortunately, people shoot themselves in the feet all the time. Python can't stop that. But it doesn't have to _cater_ to poor practices either.
By your logic: Apps that care about timezone-naive arithmetic _should_ be using naive datetime objects.
I agree it is a poor practice to perform arithmetic on localized time. But as long as such arithmetic *is* allowed then it *is* ambiguous what type of arithmetic should be used. There is no *one obvious* way here.
... dateutil doesn't work during DST transitions but PEP 495 might allow to fix it.
I don't know what "doesn't work" means, precisely. There are certain behaviors that do and don't work as you might hope. For example, even the stupidest possible tzinfo implementation that follows the docs today has no problem converting from UTC to local time across DST transitions - the default .fromutc() was designed to ensure that conversion in _that_ direction mimics the local clock in all cases (including skipping a local hour at DST start, and repeating a local hour at DST end - where "hour" really means "whole number of minutes"). What's impossible now (outside of pytz) is converting ambiguous local times _back_ to UTC in all cases. PEP 495 will repair that - that's its primary point. There's no "might" about it. But, for that to be of use to dateutil users, dateutil will need to change its tzinfo implementation to meet 495's new tzinfo requirements.
I've linked to a couple of dateutil bugs previously in PEP-431/495 thread [1]
I was surprised as you that dateutil .fromutc() appears to be broken.
I use "might" because I haven't read dateutil code. I can't be sure e.g., what backward-compatibility concerns might prevent PEP 495 fix its issues with an ambigous local time. Timezones is a very complicated topic -- no solution works in the general case.
As I understand, outside of DST transitions if dates are unique valid local times; dateutil uses "same time tomorrow":
(d_with_dateutil_tzinfo + DAY == d.tzinfo.localize(d.replace(tzinfo=None) + DAY, is_dst=None))
while pytz uses "+24 hours":
dt_add(d_with_dateutil_tzinfo, DAY) == d + DAY
where dt_add() is defined below. The equility works but (d + DAY) may have a wrong tzinfo object if the arithmetic crosses DST boundaries (but it has correct timestamp/utc time anyway). d.tzinfo.normalize(d + DAY) should be used to get the correct tzinfo e.g. for displaying the result.
Both types of operations should be supported.
If you're saying that classic and timeline arithmetic both have legitimate uses, sure. Nobody has said otherwise. If you're trying to say more than just that, sorry, I missed the point.
Yes, it is exactly my point.
The only my objection that timezone-naive arithmetic is somehow superior for localized times. Though I don't mind it as long as timezone conversions would work.
...
If dateutil can be fixed to work correctly using the disambiguation flag then its behavior is preferable because it eliminates localize, normalize calls
Then you get classic arithmetic. Which is not only fine by me, I believe it's the only realistic outcome for the reasons explained just above.
The key word here is "If". *If* it works; great. It is still possible to perform both types of arithmetic as the examples above demonstrate.
...
If people forget localize() then tzinfo is not attached and an exception is raised later. It is like mixing bytes and Unicode: if you forget decode() then an exception is raised later.
AFAICT, pytz can't enforce anything. You don't _need_ to call localize() to get _a_ datetime. From scanning message boards, e.g., I see it's a common mistake for new pytz users to use datetime.datetime(..., tzinfo=...;) directly, not using localize() at all, despite the very clear instructions in the docs that they must _not_ do that...
I had this (incorrect as I've now realized) picture in mind:
naive = datetime.strptime(...) aware = tz.localize(naive, is_dst=None)
If localize() is forgotten then we would get a naive object that would raise an exception in tzinfo-related methods later.
Yes. datetime(..., tzinfo=...) is a common error and I don't see how *without a performance hit* pytz can't fix it with PEP 495 *alone*. But it *can* be fixed if the performance is not a concern (perhaps with a slight change in the pickle semantics if relevant tzdata has changed).
...That can be a real problem for modules fighting basic design warts: newcomers are lost at first, and even experts can have trouble inter-operating with code _outside_ what typically becomes an increasingly self-contained world (e.g., Isaac cheerfully complained earlier about his pains trying to get pytz and dateutil to work together).
I agree, the usability is a real issue (for newcomers and experts).
But dateutil doesn't work [1] in cases where pytz does work and therefore if people use dateutil now; the correctness is not their primary concern and it should be much easier to combine the two libraries if you don't actually need correct answers.
[1] https://mail.python.org/pipermail/datetime-sig/2015-August/000467.html
[Akira Li 4kir4.1i@gmail.com] ...
I agree on the best practices here. I would prefer that __add__ would be forbidden for local timezones unless they have a fixed utc offset. But it might be too late for that now.
It's many years too late to change anything about datetime arithmetic in any backward-incompatible way.
If __add__ is allowed for timezone-aware datetime objects
It already is.
then arithmetic "as though via conversion to utc time" is *equally valid* as the arithmetic "as though it is a timezone-naive datetime object".aa
__add__ can only mean one of them. And it already does. It _could_ have meant the other, but it doesn't.
...
Apps that care about leap seconds _should_ be using TAI. Apps that want timeline arithmetic _should_ be using UTC. Unfortunately, people shoot themselves in the feet all the time. Python can't stop that. But it doesn't have to _cater_ to poor practices either.
By your logic: Apps that care about timezone-naive arithmetic _should_ be using naive datetime objects.
As I've said several times before, that's indeed what I would have _preferred_. But it was only a mild preference. Since timeline arithmetic is far better done in UTC anyway, I'm "happy enough" with aware datetimes using classic arithmetic. Indeed, my own code uses that frequently, and so does datetime's implementation. Several examples of that have been given in other messages.
I agree it is a poor practice to perform arithmetic on localized time. But as long as such arithmetic *is* allowed then it *is* ambiguous what type of arithmetic should be used. There is no *one obvious* way here.
Sure. But this isn't a case of "in the face of ambiguity refuse the temptation to guess". It's a case of "in the face of ambiguity, pick one, document the choice, and move on". Similarly, when printing a floating point number, there are _many_ equally valid choices for how many digits to display after the decimal point. That's no argument for refusing to print floats. Python has made different choices about that over time. If you don't like Python's default choice, with enough extra work you can force any number of digits you like. And if you don't like classic arithmetic, with enough extra work you can get any other kind of datetime arithmetic you like.
... dateutil doesn't work during DST transitions but PEP 495 might allow to fix it.
...
I've linked to a couple of dateutil bugs previously in PEP-431/495 thread [1]
I was surprised as you that dateutil .fromutc() appears to be broken.
I use "might" because I haven't read dateutil code. I can't be sure e.g., what backward-compatibility concerns might prevent PEP 495 fix its issues with an ambigous local time.
Sorry, I haven't been (and won't be) making time to stare at bugs in other packages. I probably know less about them than you do anyway. Python has always had tests for proper DST transition UTC->local conversions for the kind of tzinfo classes the docs suggest writing. But do note that "proper" in this context _only_ means "mimics the local clock" (gets the repeated or missing YYYY-MM-DD HH:MM:SS behaviors right). Without a disambiguation flag, it's flatly impossible to always get the zone name and/or UTC offset right for ambiguous local-clock hours.
Timezones is a very complicated topic -- no solution works in the general case.a
? Timezone transitions are mathematically trivial. They're just full of lumps (irregularities).
... The only my objection that timezone-naive arithmetic is somehow superior for localized times.
It's not only superior, it's essential for some purposes. For other purposes, it''s worse than useless.
Though I don't mind it as long as timezone conversions would work.
Good! That's precisely what PEP 495 intends to make possible.
...
If dateutil can be fixed to work correctly using the disambiguation flag then its behavior is preferable because it eliminates localize, normalize calls
... The key word here is "If". *If* it works; great.
I can't speak for dateutil or its author. But we're not doing brain surgery here. Conversion just isn't a deep problem. The only non-trivial cases involve ambiguous times. C fixed that long before Python existed with is_dst, although granted that mktime() is notoriously flaky across platforms in edge cases.
It is still possible to perform both types of arithmetic as the examples above demonstrate.
That's always been possible - except for errors in timeline arithmetic inherited from the rare failing conversion cases. 495 allows to repair all failures.
... [more about pytz and dateutil] .;..
Please don't take offense at my chopping this. It's only that I need to make _some_ time tonight for 495 ;-)
-
Akira Li -
Tim Peters