[Python-Dev] Status on PEP-431 Timezones

[Tim Peters]

[Tim]
>> The Python docs also are quite clear about that all arithmetic within
>> a single timezone is "naive".  That was intentional.  The _intended_
>> way to do "aware" arithmetic was always to convert to UTC, do the
>> arithmetic, then convert back.

[Lennart]
> We can't explicitly implement incorrect timezone aware arithmetic and
> then expect people to not use it.

Python didn't implement timezone-aware arithmetic at all within a
single time zone.  Read what I wrote just above.  It implements naive
arithmetic within a single time zone.

> We can make the arithmetic correct,

The naive arithmetic within a timezone is already correct, by its own
internal criteria.  It's also useful (see the original discussions, or
Paul Moore's recent brief account).  That it's not the arithmetic you
want doesn't make it "incorrect", it makes it different from what you
want.  That's fine - you're allowed to want anything ;-)  But it's a
dozen years too late to change that decision.  Maybe for Python 4.

> and we can raise an error when doing tz-aware arithmetic in a
> non-fixed timezone.

Sorry, I don't know what that means.  Under any plausible
interpretation, I don't see any need to raise an exception.

> But having an implementation we know is incorrect

You really have to get over insisting it's incorrect.  It's
functioning exactly the way it was intended to function.  It's
_different_ from what you favor.  Note that I'm not calling what you
favor "incorrect".  It's different.  Both kinds of arithmetic are
useful for different purposes, although I still agree with Guido's
original belief that the current arithmetic is most useful most often
for most programmers.

> and telling people "don't do that" doesn't seem like a good solution
> here.

We don't tell people "don't do that".  It's perfectly usable exactly
as-is for many applications.  Not all.  For those applications needing
the other kind of arithmetic, the convert-to/from-UTC dance was the
intended solution.

> Why do we even have timezone aware datetimes if we don't intend them
> for usage?

They are intended for usage.  But a single way of using them is not
suitable for all possible applications.

>> ...
>> Python's datetime never intended to support that directly.

> I think it should.

Ya, I picked that up ;-)  I don't, but it's too late to break backward
compatibility regardless.

> It's expected that it supports it,

By some people, yes.  Not by all.

> and there is no real reason not to support it.

Backward compatibility is a gigantic reason to continue with the
status quo.  See Paul Moore's post for a start on why naive arithmetic
was picked to begin with.

> The timezone handling becomes complicated if you base yourself on
> localtime, and simple if you base yourself on UTC.

That's an implementation detail unrelated (in principle) to how
arithmetic works.  Although as a practical matter it cuts both ways:
naive local-time arithmetic is complicated if the internal time is
stored in UTC, but simple if stored in local time.

> As you agree, we recommend to people to use UTC at all times,

I recommend people don't use tzinfo at all if they can avoid it.
Beyond that, there are many attractions to using UTC, and to
explicitly use UTC.  Not all applications need to care, though.

> and only use timezones for input and output. Well, what I'm now
> proposing is to take that recommendation to heart, and change
> datetime's implementation so it does exactly that.

Suppose I'm correct in my belief that there's scant chance of getting
approval for changing the default datetime arithmetic in Python 3 (or
Python 2).  Would you still be keen to replace the internals with UTC
format?  Note that there are many consequences to that implementation
detail.  For example, it was an explicit requirement of the datetime
design that the month, day, hour, minute and second components be very
cheap to extract.  If you have to do conversion every time one is
accessed, it's much slower; if you cache the "local time" components
separately, the memory burden increases.  Etc.

> I saw the previous mention of "pure" vs "practical", and that is often
> a concern. Here it clearly is not. This is a choice between impure,
> complicated and impractical, and pure, simple and practical.

There is nothing in the datetime world simpler than naive arithmetic
;-)  "Practical" is relevant to a specific application's specific
needs, and neither kind of arithmetic is "practical" for all
applications.  Guido believed naive arithmetic is most practical
overall.  But even believing that too, datetime certainly "should be"
beefed up to solve the _other_ problems:  like resolving ambiguous
times, and supporting the full range of zoneinfo possibilities

>> Is it the case that pytz also "fails" in the cases your attempts "fail"?

> No, that is not the case. And if you wonder why I just don't do it
> like pytz does it, it's because that leads to infinite recursion, much
> as discussions on this mailing list does. ;-) And this is because we
> need to normalize the datetime after arithmatic, but normalizing is
> arithmetics.

If I could talk you out of trying to "fix" the arithmetic, all those
headaches would go away and you could make swift progress on what
remains.  But I can't, so I won't try again ;-)

>> Ah, but it already happens that way

> No, in fact it does not.

Yes, it does.  But rather than repeat it all again, go back to the
original message.  I quoted the relevant Python docs in full, and
they're telling the truth.  They're talking about what happens _in a
single time zone_.  You go on to mix time zones again, and that's
_not_ what the docs are talking about.  Obviously, two times "an hour
apart" on a local clock may or may not be an hour apart in UTC, and
vice versa.

The real problem here, I suspect, is that I keep talking about naive
arithmetic, and you keep reading it as "an utterly broken
implementation of what I really want".  But neither what Python does,
_nor_ what you want, is "correct" or "incorrect".  They're different.
They're each consistent within their own view of the world.

> ...
> It's not a question of changing datetime arithmetic per se. The PEP
> does indeed mean it has to be changed, but only to support ambiguous
> and non-existent times.

The latter have nothing to do with arithmetic.  For example, in US
Eastern 2:37 AM doesn't exist on the local clock on the day DST
begins, and 1:48 AM is ambiguous on the day DST ends.  See?  Not one
word about arithmetic.  Those are just plain facts about how US
Eastern works, so datetime could address them even if no user-visible
datetime arithmetic of any kind were supported.  Naive arithmetic
doesn't care about them, but they're still potential issues when
converting to/from other time zones (and language-supplied conversions
have nothing to do with user-visible arithmetic semantics either).

> It's helpful to me to understand, which I hadn't done before, that
> this was never intended to work. That helps me argue for changing
> datetimes internal implementation, once I get time to do that. (I'm
> currently moving, renovating a new house, trying fix up a garden that
> has been neglected for years, and insanely, write my own code editor,
> all at the same time, so it won't be anytime soon).

If I were you, I'd work on the code editor.  That sounds like the most
fun, and moving sounds like the least fun ;-)

> The "changing arithmetic" discussion is a red herring.

I'm afraid it's crucial:  the prospect of breaking programs (by
changing arithmetic) that have worked for a dozen+ years is a major
problem.  If, e.g., some program "adds a week" to a datetime to
schedule a new meeting "at the same time" next week, and that's
suddenly off by an hour (one way or the other) because a DST
transition just happened to occur across that week, people would
rightfully scream bloody murder.  Many cases like that "just work"
_because_ single-zone arithmetic is naive.  With "your kind" of
arithmetic, conceptually trivial cases like that become a real pain in
the ass to program correctly (not a coincidence:  they're conceptually
trivial because "adding a week" is, to virtually everyone's mind, an
instance of naive datetime arithmetic - they're certainly not thinking
anything akin to "what I really want is to convert to UTC, add 168
hours, and convert back again", and they don't want the _result_ of
doing that either).

So if you're determined to change arithmetic, it will have to be done
in a backward-compatible way (and I briefly sketched such a way in an
earlier message today).

> Now my wife insist I help her pack, so this is the end of this
> discussion for me. If i continue it will be only as a part of
> discussing how we change how datetime works internally.

OK.  But do try to get some sleep too!  Sleep is good :-)