[Datetime-SIG] PEP-431/495

Akira Li 4kir4.1i at gmail.com
Thu Aug 27 13:33:13 CEST 2015

Tim Peters <tim.peters at gmail.com> writes:

> [Akira Li <4kir4.1i at gmail.com>]

> Nobody has said some apps don't need reliable conversions (to the
> contrary, that's the primary _point_ of PEP 495).  Nobody has said
> some apps don't need timeline arithmetic - although I have said it's
> poor practice to even _try_ to do timeline arithmetic if an app isn't
> working in UTC or with naive datetimes.  If an app is following best
> practice (UTC or naive datetimes), then timeline arithmetic is what
> they _always_ get (it's the same thing as classic arithmetic in those
> contexts).

I agree on the best practices here. I would prefer that __add__ would be
forbidden for local timezones unless they have a fixed utc offset. But
it might be too late for that now.

If __add__ is allowed for timezone-aware datetime objects then
arithmetic "as though via conversion to utc time" is *equally valid* as
the arithmetic "as though it is a timezone-naive datetime object".

>> Non-pytz timezones make mistake on the order of an hour regularly.
>> It is *three orders of magnitude larger* than a second. It is a different
>> class of errors. The code that can't handle ~1s errors over short period
>>  of time should use time.monotonic() anyway.
> Apps that care about leap seconds _should_ be using TAI.  Apps that
> want timeline arithmetic _should_ be using UTC.  Unfortunately, people
> shoot themselves in the feet all the time.  Python can't stop that.
> But it doesn't have to _cater_ to poor practices either.

By your logic: Apps that care about timezone-naive arithmetic _should_
be using naive datetime objects.

I agree it is a poor practice to perform arithmetic on localized time.
But as long as such arithmetic *is* allowed then it *is* ambiguous what
type of arithmetic should be used. There is no *one obvious* way here.

>> ...
>> dateutil doesn't work during DST transitions but PEP 495 might allow to
>> fix it.
> I don't know what "doesn't work" means, precisely.  There are certain
> behaviors that do and don't work as you might hope.  For example, even
> the stupidest possible tzinfo implementation that follows the docs
> today has no problem converting from UTC to local time across DST
> transitions - the default .fromutc() was designed to ensure that
> conversion in _that_ direction mimics the local clock in all cases
> (including skipping a local hour at DST start, and repeating a local
> hour at DST end - where "hour" really means "whole number of
> minutes").  What's impossible now (outside of pytz) is converting
> ambiguous local times _back_ to UTC in all cases.  PEP 495 will repair
> that - that's its primary point.  There's no "might" about it.  But,
> for that to be of use to dateutil users, dateutil will need to change
> its tzinfo implementation to meet 495's new tzinfo requirements.

I've linked to a couple of dateutil bugs previously in PEP-431/495
thread [1]

I was surprised as you that dateutil .fromutc() appears to be broken.

I use "might" because I haven't read dateutil code. I can't be sure
e.g., what backward-compatibility concerns might prevent PEP 495 fix its
issues with an ambigous local time. Timezones is a very complicated
topic -- no solution works in the general case.

>> As I understand, outside of DST transitions if dates are unique valid
>> local times; dateutil uses "same time tomorrow":
>>   (d_with_dateutil_tzinfo + DAY ==
>>    d.tzinfo.localize(d.replace(tzinfo=None) + DAY, is_dst=None))
>> while pytz uses "+24 hours":
>>    dt_add(d_with_dateutil_tzinfo, DAY) == d + DAY
>> where dt_add() is defined below. The equility works but (d + DAY) may
>> have a wrong tzinfo object if the arithmetic crosses DST boundaries (but
>> it has correct timestamp/utc time anyway).  d.tzinfo.normalize(d + DAY)
>> should be used to get the correct tzinfo e.g. for displaying the result.
>> Both types of operations should be supported.
> If you're saying that classic and timeline arithmetic both have
> legitimate uses, sure.  Nobody has said otherwise.  If you're trying
> to say more than just that, sorry, I missed the point.

Yes, it is exactly my point.

The only my objection that timezone-naive arithmetic is somehow superior
for localized times. Though I don't mind it as long as timezone
conversions would work.

>> If dateutil can be fixed to work correctly using the disambiguation flag
>> then its behavior is preferable because it eliminates localize,
>> normalize calls
> Then you get classic arithmetic.  Which is not only fine by me, I
> believe it's the only realistic outcome for the reasons explained just
> above.

The key word here is "If". *If* it works; great. It is still possible to
perform both types of arithmetic as the examples above demonstrate.

>> If people forget localize() then tzinfo is not attached and an exception
>> is raised later. It is like mixing bytes and Unicode: if you forget
>> decode() then an exception is raised later.
> AFAICT, pytz can't enforce anything.  You don't _need_ to call
> localize() to get _a_ datetime.  From scanning message boards, e.g., I
> see it's a common mistake for new pytz users to use
> datetime.datetime(..., tzinfo=...;) directly, not using localize() at
> all, despite the very clear instructions in the docs that they must
> _not_ do that...

I had this (incorrect as I've now realized) picture in mind:

  naive = datetime.strptime(...)
  aware = tz.localize(naive, is_dst=None)

If localize() is forgotten then we would get a naive object that would
raise an exception in tzinfo-related methods later.

Yes. datetime(..., tzinfo=...) is a common error and I don't see how
*without a performance hit* pytz can't fix it with PEP 495 *alone*. But
it *can* be fixed if the performance is not a concern (perhaps with a
slight change in the pickle semantics if relevant tzdata has changed).

> ...That can be a real problem for modules fighting basic
> design warts:  newcomers are lost at first, and even experts can have
> trouble inter-operating with code _outside_ what typically becomes an
> increasingly self-contained world (e.g., Isaac cheerfully complained
> earlier about his pains trying to get pytz and dateutil to work
> together).

I agree, the usability is a real issue (for newcomers and experts).

But dateutil doesn't work [1] in cases where pytz does work and
therefore if people use dateutil now; the correctness is not their
primary concern and it should be much easier to combine the two
libraries if you don't actually need correct answers.

[1] https://mail.python.org/pipermail/datetime-sig/2015-August/000467.html

More information about the Datetime-SIG mailing list