On Apr 8, 2015, at 12:37, Carl Meyer wrote:

...

Anyone interested in a session on this, mail me and we'll set up a time and place!

I'm interested in the topic, and would probably attend a BoF at PyCon.

I'm of a similar mind.

On 9 April 2015 at 04:38, Lennart Regebro wrote:

...

So is adopting pytz's expanded API into the stdlib really a big problem?

Maybe, maybe not. But that API is also needlessly complicated, precisely because it's working around the limitations of datetime.tzinfo. In the PEP I remove those limitations but keep the simpler API. With a solution based on how pytz does it, I don't think that's possible.

Speaking at the pytz author and maintainer, I can categorically state that pytz's extended API sucks.

...

Is this really adequate? pytz's implementation handles far more than "is DST or not", it also correctly handles historical timezone changes. How would those be handled under this proposal?

Those would still be handled. The flag is only to flag if it's DST or not in a timestamp that is otherwise ambiguous.

Yeah. I look forward to datetime growing an isdst flag, which if nothing else means they can round trip with the 9-tuples used by the time module and a simple LocalTz class written that uses time.mktime and friends. I'd have done it myself if I remembered any C. At that point, the most disgusting parts of pytz can be torn out, giving it the standard documented API and I'm more than happy to do that bit. -- Stuart Bishop http://www.stuartbishop.net/

OK, so I realized another thing today, and that is that arithmetic doesn't necessarily round trip. For example, 2002-10-27 01:00 US/Eastern comes both in DST and STD. But 2002-10-27 01:00 US/Eastern STD minus two days is 2002-10-25 01:00 US/Eastern DST However, 2002-10-25 01:00 US/Eastern DST plus two days is 2002-10-27 01:00 US/Eastern, but it is ambiguous if you want DST or not DST. And you can't pass in a is_dst flag to __add__, so the arithmatic must just pick one, and the sensible one is to keep to the same DST. That means that: tz = get_timezone('US/Eastern') dt = datetime(2002, 10, 27, 1, 0, tz=tz, is_dst=False) dt2 = dt - 420 + 420 assert dt == dt2 Will fail, which will be unexpected for most people. I think there is no way around this, but I thought I should flag for it. This is a good reason to do all your date time arithmetic in UTC. //Lennart

Yeah, I just realized this. As long as you use timedelta, the difference is of course not one day, but 24 hours. That solves the problem, but it is surprising in other ways. In US/Eastern datetime.datetime(2002, 10, 27, 1, 0) - datetime.timedelta(1) needs to become datetime.datetime(2002, 10, 26, 2, 0) (Note the hour change) I was thinking in calendrial arithmetic, which the datetime module doesn't need to care about. On Wed, Apr 15, 2015 at 12:59 AM, Stuart Bishop wrote:

On 14 April 2015 at 21:04, Lennart Regebro wrote:

...

OK, so I realized another thing today, and that is that arithmetic doesn't necessarily round trip.

For example, 2002-10-27 01:00 US/Eastern comes both in DST and STD.

But 2002-10-27 01:00 US/Eastern STD minus two days is 2002-10-25 01:00 US/Eastern DST However, 2002-10-25 01:00 US/Eastern DST plus two days is 2002-10-27 01:00 US/Eastern, but it is ambiguous if you want DST or not DST. And you can't pass in a is_dst flag to __add__, so the arithmatic must just pick one, and the sensible one is to keep to the same DST.

...

...

...

import pytz from datetime import datetime, timedelta tz = pytz.timezone('US/Eastern') dt = tz.localize(datetime(2002, 10, 27, 1, 0), is_dst=False) dt2 = tz.normalize(dt - timedelta(days=2) + timedelta(days=2)) dt == dt2 True

tz.normalize(dt - timedelta(days=2)) datetime.datetime(2002, 10, 25, 2, 0, tzinfo=) tz.normalize(tz.normalize(dt - timedelta(days=2)) + timedelta(days=2)) datetime.datetime(2002, 10, 27, 1, 0, tzinfo=)

2002-10-27 01:00 US/Eastern is_dst=0 is after the DST transition (EST). Subtracting 48 hours from it crosses the DST boundary and should give you 2002-10-27 02:00 US/Eastern is_dst=1, prior to the DST transition (EDT). Adding 48 hours again goes past 2002-10-27 01:00 EDT, crosses the DST boundary, and gives you back 2002-10-27 01:00 EST.

-- Stuart Bishop http://www.stuartbishop.net/

On Thu, Apr 16, 2015 at 1:00 AM, Lennart Regebro wrote:

So because of this, perhaps we actually *should* change the internal representation to UTC, because that makes the issues I'm fighting with now so much simpler. (I'm currently trying to get arithmetic to do the right thing in all cases, which is crazy complicated).

If I understand you correctly, then, an aware datetime would represent a unique instant in time (modulo relativity), coupled with some metadata stating what civil timezone it should be understood in terms of. This is the same as a PostgreSQL "timestamp with time zone" field, and IMO is a pretty reliable way to do things. So count me as +1 for this proposal. Bikeshed: Would arithmetic be based on UTC time or Unix time? It'd be more logical to describe it as "adding six hours means adding six hours to the UTC time", but it'd look extremely odd when there's a leap second. ChrisA

On Wed, Apr 15, 2015 at 11:43 AM, Lennart Regebro wrote:

On Wed, Apr 15, 2015 at 11:10 AM, Chris Angelico wrote:

...

Bikeshed: Would arithmetic be based on UTC time or Unix time? It'd be more logical to describe it as "adding six hours means adding six hours to the UTC time", but it'd look extremely odd when there's a leap second.

It would ignore leap seconds. If you want to call that unix time or not is a matter of opinion. Hm. I guess the internal representation *could* be EPOCH + offset, and local times could be calculated properties, which could be cached (or possibly calculated at creation).

In any case there wold probably need to be a PEP on that, and that means PEP 431 wouldn't make it into 3.5, unless somebody smarter than me want to take a shot at it. //Lennart

On 15 April 2015 at 17:00, Lennart Regebro wrote:

OK, so I just had a realization.

Because we want some internal flag to tell if the datetime is in DST or not, the datetime pickle format will change. And the datetime pickle format changing is the biggest reason I had against changing the internal representation to UTC.

So because of this, perhaps we actually *should* change the internal representation to UTC, because that makes the issues I'm fighting with now so much simpler. (I'm currently trying to get arithmetic to do the right thing in all cases, which is crazy complicated).

Huh. I didn't think you would need to change any arithmetic (but haven't looked at this for quite some time). You can already add or subtract timedeltas to timezone aware datetime instances. The problem with the existing implementation is the tzinfo instance does not have enough information to do correct conversions when the time is ambiguous, so it has to guess. With the addition of the is_dst hint to the datetime instance, it will no longer need to guess. Arithmetic remains 'add the timedelta to the naive datetime, and then punt it to the tzinfo to make any necessary adjustments' and I thought this would not need to be changed at all.

We can add support to unpickle previous datetimes, but we won't be able to add forwards compatibility, meaning that pickles saved in Python 3.5 will not be unpicklable in Python 3.4.

I don't think this can be avoided entirely. Any ideas I can come up with that might help are worse than requiring devs to convert their datetimes to strings in the rare case they need their 3.5 pickles read with 3.4. -- Stuart Bishop http://www.stuartbishop.net/

On 15 April 2015 at 21:51, Lennart Regebro wrote:

On Wed, Apr 15, 2015 at 3:23 PM, Stuart Bishop wrote:

Just punting it to tzinfo to make adjustments, ie effectively just doing what normalize() does creates infinite recursion as there is more arithmetic in there, so it's not quite that simple.

This sounds familiar. Its infinite recursion if the tzinfo does its calculations using localized datetimes. If the tzinfo is stripped for the calculations, there is no tzinfo to recurse into. At least this was how I hoped it would work, and it sucks if it doesn't. You could be right that using the UTC representation internally for datetimes with a tzinfo makes the most sense. -- Stuart Bishop http://www.stuartbishop.net/

Lennart Regebro writes:

OK, so I realized another thing today, and that is that arithmetic doesn't necessarily round trip.

For example, 2002-10-27 01:00 US/Eastern comes both in DST and STD.

But 2002-10-27 01:00 US/Eastern STD minus two days is 2002-10-25 01:00 US/Eastern DST

"two days" is ambiguous here. It is incorrect if you mean 48 hours (the difference is 49 hours): #!/usr/bin/env python3 from datetime import datetime, timedelta import pytz tz = pytz.timezone('US/Eastern') then_isdst = False # STD then = tz.localize(datetime(2002, 10, 27, 1), is_dst=then_isdst) now = tz.localize(datetime(2002, 10, 25, 1), is_dst=None) # no utc transition print((then - now) // timedelta(hours=1)) # -> 49

However, 2002-10-25 01:00 US/Eastern DST plus two days is 2002-10-27 01:00 US/Eastern, but it is ambiguous if you want DST or not DST.

It is not ambiguous if you know what "two days" *in your particular application* should mean (`day+2` vs. +48h exactly): print(tz.localize(now.replace(tzinfo=None) + timedelta(2), is_dst=then_isdst)) # -> 2002-10-27 01:00:00-05:00 # +49h print(tz.normalize(now + timedelta(2))) # +48h # -> 2002-10-27 01:00:00-04:00 Here's a simple mental model that can be used for date arithmetics: - naive datetime + timedelta(2) == "same time, elapsed hours unknown" - aware utc datetime + timedelta(2) == "same time, +48h" - aware datetime with timezone that may have different utc offsets at different times + timedelta(2) == "unknown time, +48h" "unknown" means that you can't tell without knowning the specific timezone. It ignores leap seconds. The 3rd case behaves *as if* the calculations are performed using these steps (the actual implementation may be different): 1. convert an aware datetime object to utc (dt.astimezone(pytz.utc)) 2. do the simple arithmetics using utc time 3. convert the result to the original pytz timezone (utc_dt.astimezone(tz)) you don't need `.localize()`, `.normalize()` calls here.

And you can't pass in a is_dst flag to __add__, so the arithmatic must just pick one, and the sensible one is to keep to the same DST.

That means that:

tz = get_timezone('US/Eastern') dt = datetimedatetime(2002, 10, 27, 1, 0, tz=tz, is_dst=False) dt2 = dt - 420 + 420 assert dt == dt2

Will fail, which will be unexpected for most people.

I think there is no way around this, but I thought I should flag for it. This is a good reason to do all your date time arithmetic in UTC.

//Lennart

It won't fail: from datetime import datetime, timedelta import pytz tz = pytz.timezone('US/Eastern') dt = tz.localize(datetime(2002, 10, 27, 1), is_dst=False) delta = timedelta(seconds=420) assert dt == tz.normalize(tz.normalize(dt - delta) + delta) The only reason `tz.normalize()` is used so that tzinfo would be correct for the resulting datetime object; it does not affect the comparison otherwise: assert dt == (dt - delta + delta) #XXX tzinfo may be incorrect assert dt == tz.normalize(dt - delta + delta) # correct tzinfo for the final result

On Tue, Apr 14, 2015 at 4:52 PM, Alexander Belopolsky wrote:

For those who were not at the conference, can someone summarize the post-PyCon status of this PEP?

Is Barry still the "BDFL-Delegate"? Is there an updated draft? Should this discussion move to python-ideas?

There is no status change except that it will need updates, but I'm waiting with that until I know what updates are needed, and that means an implementation is needed first. //Lennart