Mailman 3 should utcnow and utcfromtimestamp return tz-aware? - Datetime-SIG

should utcnow and utcfromtimestamp return tz-aware?

Brock Mendel

16 Apr 2019 16 Apr '19

3:01 a.m.

This has come up in pandas.Timestamp (which subclasses datetime). The issue is internal consistency and round-trips. Intuitively, we would expect: ``` a = datetime.utcnow() b = a.timestamp() c = datetime.utcfromtimestamp(b) assert a == b ``` but this fails. (Note: it works for `datetime.now()` with `datetime.fromtimestamp()`, though that is partially because of weird-to-me behavior of `timestamp()` for naive datetimes). It would succeed if utcnow and utcfromtimestamp returned tz-aware datetime objects. Thoughts?

Attachments:

attachment.htm (text/html — 691 bytes)

Show replies by date

Paul Ganssle

16 Apr 16 Apr

3:23 a.m.

utcnow() and utcfromtimestamp are semi-deprecated functions /because/ they do not return tz-aware datetime objects and because in all functions that require a time zone offset (like `timestamp`), naive datetimes are treated as *local* times. If you want to use tz-aware datetime objects (which will have the correct behavior), you should simply pass a time zone to the `tz` parameter of `now()` and `fromtimestamp`, respectively, so: ``` from datetime import datetime, timezone a = datetime.now(tz=timezone.utc) b = a.timestamp() c = datetime.fromtimestamp(b, tz=timezone.utc) assert a == c ``` I will note that because of the way aware datetime comparison semantics works, `a == c` will be true no matter what `tz` object you pass to `fromtimestamp`. If I rewrite your example with equivalent commands that do /not/ use `utcnow`, you will hopefully see how it works: ``` from datetime import datetime, timezone from dateutil.tz import tzlocal a = datetime.now(tz=timezone.utc).replace(tzinfo=None) b = a.replace(tzinfo=tzlocal()).timestamp() c = datetime.fromtimestamp(b, tz=timezone.utc).replace(tzinfo=None) assert a == c ``` As you can see, in `a` you discard the time zone information associated with the datetime, and so in `b` it is implicitly taken to be "local time", shifting it by however many hours from UTC your local time is. I will answer the hashability question in a separate e-mail, but I don't see how this has much to do with hashability. P.S. Accidentally sent this to just Brock - Brock, sorry you are getting this twice. On 4/15/19 8:01 PM, Brock Mendel wrote:

...

This has come up in pandas.Timestamp (which subclasses datetime). The issue is internal consistency and round-trips. Intuitively, we would expect:

``` a = datetime.utcnow() b = a.timestamp() c = datetime.utcfromtimestamp(b) assert a == b ```

but this fails. (Note: it works for `datetime.now()` with `datetime.fromtimestamp()`, though that is partially because of weird-to-me behavior of `timestamp()` for naive datetimes).

It would succeed if utcnow and utcfromtimestamp returned tz-aware datetime objects. Thoughts?

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

Brock Mendel

5:32 a.m.

...

utcnow() and utcfromtimestamp are semi-deprecated functions *because* they do not return tz-aware datetime objects

Is this documented somewhere? Should that be reflected in the docs https://docs.python.org/3/library/datetime.html? Is there a game-plan to make it actually-deprecated instead of semi-deprecated? On Mon, Apr 15, 2019 at 5:23 PM Paul Ganssle wrote:

...

utcnow() and utcfromtimestamp are semi-deprecated functions *because* they do not return tz-aware datetime objects and because in all functions that require a time zone offset (like `timestamp`), naive datetimes are treated as *local* times.

If you want to use tz-aware datetime objects (which will have the correct behavior), you should simply pass a time zone to the `tz` parameter of `now()` and `fromtimestamp`, respectively, so:

``` from datetime import datetime, timezone a = datetime.now(tz=timezone.utc) b = a.timestamp() c = datetime.fromtimestamp(b, tz=timezone.utc) assert a == c ```

I will note that because of the way aware datetime comparison semantics works, `a == c` will be true no matter what `tz` object you pass to `fromtimestamp`.

If I rewrite your example with equivalent commands that do *not* use `utcnow`, you will hopefully see how it works:

``` from datetime import datetime, timezone from dateutil.tz import tzlocal a = datetime.now(tz=timezone.utc).replace(tzinfo=None) b = a.replace(tzinfo=tzlocal()).timestamp() c = datetime.fromtimestamp(b, tz=timezone.utc).replace(tzinfo=None) assert a == c ```

As you can see, in `a` you discard the time zone information associated with the datetime, and so in `b` it is implicitly taken to be "local time", shifting it by however many hours from UTC your local time is.

I will answer the hashability question in a separate e-mail, but I don't see how this has much to do with hashability.

P.S. Accidentally sent this to just Brock - Brock, sorry you are getting this twice.

On 4/15/19 8:01 PM, Brock Mendel wrote:

This has come up in pandas.Timestamp (which subclasses datetime). The issue is internal consistency and round-trips. Intuitively, we would expect:

``` a = datetime.utcnow() b = a.timestamp() c = datetime.utcfromtimestamp(b) assert a == b ```

but this fails. (Note: it works for `datetime.now()` with `datetime.fromtimestamp()`, though that is partially because of weird-to-me behavior of `timestamp()` for naive datetimes).

It would succeed if utcnow and utcfromtimestamp returned tz-aware datetime objects. Thoughts?

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.orghttps://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

Paul Ganssle

5:23 p.m.

I may be over-stating the degree of consensus about the deprecation here. I'm not sure too many people have discussed it, but there is very little reason to ever use them, because: 1. The reason they existed in the first place was as a convenience when there were no concrete time zone types in the standard library. 2. Adding the `tz` argument to `now` and `timestamp` has provided a more general solution to the problem anyway. 3. They create objects that are inconsistent with the way Python treats naive datetimes, since naive datetimes are supposed to be abstract representations of a time, and if they need to be treated as a concrete time, they are taken to be /local/ times. The one advantage `datetime.utcnow()` has is that it is somewhat faster than `datetime.now(tz=timezone.utc)` (I clock it at 168ns vs 300ns), but getting the current time in UTC is not likely to be the bottleneck in any sort of "hot loop" situation where 300ns is going to make the difference. I think deprecating them would be nice, but it would break backwards compatibility to actually /remove/ them, so I'm maybe +0 on that. I'd say at the very least documenting that these things have a good replacement would be a good idea. Maybe we should make a bpo issue (and possibly PR) for this. Best, Paul On 4/15/19 10:32 PM, Brock Mendel wrote:

...

...
utcnow() and utcfromtimestamp are semi-deprecated functions /because/ they do not return tz-aware datetime objects

Is this documented somewhere? Should that be reflected in the docs https://docs.python.org/3/library/datetime.html? Is there a game-plan to make it actually-deprecated instead of semi-deprecated?

On Mon, Apr 15, 2019 at 5:23 PM Paul Ganssle mailto:paul@ganssle.io> wrote:

utcnow() and utcfromtimestamp are semi-deprecated functions /because/ they do not return tz-aware datetime objects and because in all functions that require a time zone offset (like `timestamp`), naive datetimes are treated as *local* times.

If you want to use tz-aware datetime objects (which will have the correct behavior), you should simply pass a time zone to the `tz` parameter of `now()` and `fromtimestamp`, respectively, so:

``` from datetime import datetime, timezone a = datetime.now(tz=timezone.utc) b = a.timestamp() c = datetime.fromtimestamp(b, tz=timezone.utc) assert a == c ```

I will note that because of the way aware datetime comparison semantics works, `a == c` will be true no matter what `tz` object you pass to `fromtimestamp`.

If I rewrite your example with equivalent commands that do /not/ use `utcnow`, you will hopefully see how it works:

``` from datetime import datetime, timezone from dateutil.tz http://dateutil.tz import tzlocal

a = datetime.now(tz=timezone.utc).replace(tzinfo=None) b = a.replace(tzinfo=tzlocal()).timestamp() c = datetime.fromtimestamp(b, tz=timezone.utc).replace(tzinfo=None) assert a == c ```

As you can see, in `a` you discard the time zone information associated with the datetime, and so in `b` it is implicitly taken to be "local time", shifting it by however many hours from UTC your local time is.

I will answer the hashability question in a separate e-mail, but I don't see how this has much to do with hashability.

P.S. Accidentally sent this to just Brock - Brock, sorry you are getting this twice.

On 4/15/19 8:01 PM, Brock Mendel wrote:

...
This has come up in pandas.Timestamp (which subclasses datetime). The issue is internal consistency and round-trips. Intuitively, we would expect:

``` a = datetime.utcnow() b = a.timestamp() c = datetime.utcfromtimestamp(b) assert a == b ```

but this fails. (Note: it works for `datetime.now()` with `datetime.fromtimestamp()`, though that is partially because of weird-to-me behavior of `timestamp()` for naive datetimes).

It would succeed if utcnow and utcfromtimestamp returned tz-aware datetime objects. Thoughts?

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org mailto:datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org mailto:datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org mailto:datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org mailto:datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

Chris Barker - NOAA Federal

17 Apr 17 Apr

2:42 a.m.

1. The reason they existed in the first place was as a convenience when there were no concrete time zone types in the standard library. Also because the machine has two datetime you might want to work with: UTC and whatever time zone the locale thinks it’s in. And either way, you may want a naive datetime. 2. Adding the `tz` argument to `now` and `timestamp` has provided a more general solution to the problem anyway. Yup 3. They create objects that are inconsistent with the way Python treats naive datetimes, since naive datetimes are supposed to be abstract representations of a time, and if they need to be treated as a concrete time, they are taken to be *local* times. “Local” is as poorly defined, and more likely to be misinterpreted, as “naive”. Local to the machine the code is running on? Local to the user? Local to the data the user is working with? The one advantage `datetime.utcnow()` has is that it is somewhat faster than `datetime.now(tz=timezone.utc)` Using a naive datetime for UTC is still very useful, I’d like to keep an easy, backward compatible way to get it. -CHB

Paul Ganssle

5:56 p.m.

On 4/16/19 7:42 PM, Chris Barker - NOAA Federal wrote:

...

...
1. The reason they existed in the first place was as a convenience when there were no concrete time zone types in the standard library.

Also because the machine has two datetime you might want to work with: UTC and whatever time zone the locale thinks it’s in. And either way, you may want a naive datetime.

There are many more types of datetime you may want to work with. I don't see any particularly compelling reason to use naive datetimes to represent UTC when there is very little support for that in the language itself, and there is direct support for using /aware/ time zones for this.

...

“Local” is as poorly defined, and more likely to be misinterpreted, as “naive”. Local to the machine the code is running on? Local to the user? Local to the data the user is working with?

In this case it is /not/ poorly defined, it specifically means the context of the system time. I was not saying that I think naive datetimes /should/ represent local time, I'm saying that whenever a naive datetime is treated as representing a specific time (as opposed to an abstract datetime for calendrical operations), it is interpreted as a system-local time. So if your local time is set to America/New_York, then you get: In [7]: dt = datetime(2019, 4, 15, 12).astimezone(timezone.utc) In [8]: print(dt) 2019-04-15 16:00:00+00:00 This is the bug at the root of Brock's original issue, and it's one reason why using naive datetimes to represent a specific time zone is somewhat perilous.

...

...
The one advantage `datetime.utcnow()` has is that it is somewhat faster than `datetime.now(tz=timezone.utc)`

Using a naive datetime for UTC is still very useful, I’d like to keep an easy, backward compatible way to get it.

Like I said, I don't really feel strongly about breaking backwards compatibility to doing this, but I do dispute that it is "very useful". It creates an implicit assumption about what the datetime represents that is /contrary/ to what the language itself assumes /and/ there's already a perfectly good mechanism for making those assumptions explicit - attaching the timezone.utc object to the datetime. Best, Paul

Chris Barker

22 Apr 22 Apr

8 p.m.

I'm not sure there is anything left on the table to decide here, but... On Wed, Apr 17, 2019 at 7:56 AM Paul Ganssle wrote:

...

“Local” is as poorly defined, and more likely to be misinterpreted, as “naive”. Local to the machine the code is running on? Local to the user? Local to the data the user is working with?

In this case it is *not* poorly defined, it specifically means the context of the system time. I was not saying that I think naive datetimes *should* represent local time, I'm saying that whenever a naive datetime is treated as representing a specific time (as opposed to an abstract datetime for calendrical operations), it is interpreted as a system-local time.

That is dangerous (or at least annoying) and not useful. In these modern days of networked computing the system-local time is highly unlikely to be relevant, an highly likely to be wrong for the use case at hand. Example: when numpy.datetime64 was introduced, the author made the very reasonable assumption, that based on ISO 8601: "If no UTC relation information is given with a time representation, the time is assumed to be in local time." - then a ISO string without a offset is given is local, and it applied the machine local timezone to create a UTC time. This was a pain in the #$%$ and not the least bit useful. For anything. It required a bunch of hacks and work arounds, and was ultimately changed. I really like the concept of naive datetimes, it makes the point -- this thing known nothing about time zones. period, end of story. It does not represent "local time" because that is a very poorly defined concept. However, there are lots and lots of applications that punt on timezones (because they are a nightmare), and rather, assume that all teh datetime data in a a given application is in the same time zone, whatever that may be. So the offset needs to be checked/corrected on I/O -- kind of like text encodings :-). It happens that UTC has no DST or other ugliness, to naive datetimes can be used quite well to represent UTC -- whether a datetime is UTC or naive is only relevant when you want to convert to another timezone -- the use-case for naive datetimes is for when the timezone handling happens away from the datetime object (at the applications I/O boundary) -- and in this use case, using naive datetimes of rUTC is very useful. The point is that naive datetimes are, well, naive, so none of the rest of the datetime machinery should every make ANY assumptions about what timezone it might represent. But if it is going to, then UTC is the only one that is at all useful or reliable.

...

So if your local time is set to America/New_York, then you get:

In [7]: dt = datetime(2019, 4, 15, 12).astimezone(timezone.utc)

In [8]: print(dt)

2019-04-15 16:00:00+00:00

This is the whole problem: "your local time" is poorly defined -- what if I'm in New York, but running on a JupyterLab system in Los Angeles? If I had noticed tha astimezone did this, when it was introduced, I would have suggested that we NOT do that. But now it's there, I see the docstring: In [17]: dt.astimezone? Docstring: tz -> convert to local time in new timezone tz Type: builtin_function_or_method I'd like to see that expanded a bit -- maybe: tz -> convert to local time of the system on which python is running in new timezone tz I see that the full docs are pretty clear about this -- but even then, some omre caution would be in order. "If self is naive, it is presumed to represent time in the system timezone." I 'd like to see that up front and center. If we has it to do again, I'd probably make it raise an exception with a naive datetime. oh well. Like I said, I don't really feel strongly about breaking backwards

...

compatibility to doing this, but I do dispute that it is "very useful". It creates an implicit assumption about what the datetime represents that is *contrary* to what the language itself assumes *and* there's already a perfectly good mechanism for making those assumptions explicit - attaching the timezone.utc object to the datetime.

and then taking it off again if you want a naive datetime? Looking back at the OP's example: a = datetime.utcnow() b = a.timestamp() c = datetime.utcfromtimestamp(b) assert a == b fails. I would suggest that the "problem" here is timestamp() : datetime.timestamp() Return POSIX timestamp corresponding to the datetime instance. The return value is a float similar to that returned by time.time(). Naive datetime instances are assumed to represent local time There's that pesky local time thing again. I see a big note in there making my point. Anyway, we're all for backward compatibility, so I'm not sure there's much to be done here -- though updating the docs to provide more caution might be in order. If nothing else, every time the term "local time" is used, something like "the timezone set in the locale of the computer on which the code is running", or a less wordy version, would be good. -CHB

...

Best, Paul _______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Alexander Belopolsky

8:54 p.m.

...

what if I'm in New York, but running on a JupyterLab system in Los Angeles?

In this case, it would be the responsibility of the JupyterLab designers to set the timezone of the kernel to something useful to the user. In the case of a single-user kernel that would be the user's timezone, in the case of multi-user kernels - most likely UTC or something that the users can agree upon. The location of the JupyterLab server should never be a consideration. Setting the timezone is currently more complicated than it should be: import sys, time sys.environ['TZ'] = 'America/New_York' time.tzset() but I don't think we can improve that before python gets an integrated timezone database. For now, this is what you should do to make sure a remote system uses the timezone that you want.

Brock Mendel

28 Dec 28 Dec

3:33 a.m.

Now that zoneinfo is in the stdlib, I'd like to revisit the idea of either a) making utcnow/utcfromtimestamp return tzaware or b) deprecating in favor of explicitly passing tzinfos to now/fromtimestamp. Thoughts? On Mon, Apr 22, 2019 at 10:54 AM Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

...

...
what if I'm in New York, but running on a JupyterLab system in Los Angeles?

In this case, it would be the responsibility of the JupyterLab designers to set the timezone of the kernel to something useful to the user. In the case of a single-user kernel that would be the user's timezone, in the case of multi-user kernels - most likely UTC or something that the users can agree upon. The location of the JupyterLab server should never be a consideration.

Setting the timezone is currently more complicated than it should be:

import sys, time sys.environ['TZ'] = 'America/New_York' time.tzset()

but I don't think we can improve that before python gets an integrated timezone database. For now, this is what you should do to make sure a remote system uses the timezone that you want. _______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

Paul Ganssle

5:24 a.m.

I don't really think zoneinfo being in the stdlib has much bearing on the question of utcnow/utctimestamp. We already had a UTC object in the standard library, and could have made this change any time, but chose not to because it would break backwards compatibility. I'm definitely more in favor of deprecating it than I am of changing the behavior, which would be a major breaking change and one that is hard to warn about. Removing utcnow would probably be a lot of work for a lot of people, but many people are probably using it wrong anyway so that may be a net benefit. That said, there is at least one use case I'm concerned about where the replacement is not great: If you want a datetime in UTC that you /immediately format/, attaching the `UTC` object rather than using `utcnow` would lead to a performance regression without any actual benefit (and sometimes it's actually a bit of a pain). For example: print(datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")) print(datetime.utcnow().isoformat()) would have to turn into: print(datetime.now(UTC).strftime("%Y-%m-%d %H:%M:%S")) print(datetime.now(UTC).replace(tzinfo=None).isoformat()) The first of these is slower but no safer than the `utcnow` version, and the second of these actually involves stripping the UTC object from the result of `now` just to avoid `isoformat`'s behavior where it automatically appends the time zone offset to aware datetimes. I haven't looked at it, but it may be that we can special-case `datetime.timezone.utc` in the `now()` and `fromtimestamp()` constructors so that `datetime.now(timezone.utc)` is just as fast as `datetime.utcnow()`, in which case the first example is not a huge deal, and maybe the second performance regression isn't so bad that it is worth leaving around footguns like this. It'd be nice to put some numbers on that. Best, Paul On 12/27/21 19:33, Brock Mendel wrote:

...

Now that zoneinfo is in the stdlib, I'd like to revisit the idea of either a) making utcnow/utcfromtimestamp return tzaware or b) deprecating in favor of explicitly passing tzinfos to now/fromtimestamp.

Thoughts?

On Mon, Apr 22, 2019 at 10:54 AM Alexander Belopolsky wrote:

> what if I'm in New York, but running on a JupyterLab system in Los Angeles?

In this case, it would be the responsibility of the JupyterLab designers to set the timezone of the kernel to something useful to the user. In the case of a single-user kernel that would be the user's timezone, in the case of multi-user kernels - most likely UTC or something that the users can agree upon. The location of the JupyterLab server should never be a consideration.

Setting the timezone is currently more complicated than it should be:

import sys, time sys.environ['TZ'] = 'America/New_York' time.tzset()

but I don't think we can improve that before python gets an integrated timezone database. For now, this is what you should do to make sure a remote system uses the timezone that you want. _______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

_______________________________________________ Datetime-SIG mailing list --datetime-sig@python.org To unsubscribe send an email todatetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list:https://www.python.org/psf/codeofconduct/

Oren Tirosh

30 Dec 30 Dec

1:19 p.m.

Since changing datetime would break compatibility, I experimented with a module called adatetime ('a' for aware) where timezone-naive objects are not allowed. The class methods adatetime.datetime.fromtimestamp(t) and adatetime.datetime.utcfromtimestamp(t) produces objects that compare as equal. One is in the local timezone and the other is in UTC but they represent the same point in time. With the standard datetime module this results in unequal naive objects. There is no such thing as a local time_t in POSIX. Internal representation is always UTC and broken-down human time (struct tm) always has an associated timezone. No mixups. The python datetime module supports a non-UTC int or float representation with the methods .fromtimestamp(), .utcfromtimestamp() and .timestamp(). In these methods the time in seconds may be implicitly treated or generated as local time. I consider this the "Original Sin" of an otherwise very well-designed module. I experimented with various alternatives for how adatetime should behave when tzinfo is not provided: using either UTC or the local timezone as default or always require it to be passed explicitly. I think the best default is UTC, with a singleton named 'local' that can be passed as tzinfo to specify using the local timezone (same behavior as adding .astimezone(None)). For code that always passes an explicit tzinfo this makes adatetime almost 100% compatible to the standard datetime. The only difference is the behavior of .fromtimestsamp() / .utcfromtimestamp() being aware by default. To make it fully compatible would require either adding some new method for creating aware datetime to both datetime and adatetime or removing .utcfromtimestamp() from adatetime and making adatetime's .fromtimestamp() tzinfo argument not optional. On Tue, 28 Dec 2021 at 04:24, Paul Ganssle wrote:

...

I don't really think zoneinfo being in the stdlib has much bearing on the question of utcnow/utctimestamp. We already had a UTC object in the standard library, and could have made this change any time, but chose not to because it would break backwards compatibility.

I'm definitely more in favor of deprecating it than I am of changing the behavior, which would be a major breaking change and one that is hard to warn about. Removing utcnow would probably be a lot of work for a lot of people, but many people are probably using it wrong anyway so that may be a net benefit.

That said, there is at least one use case I'm concerned about where the replacement is not great:

If you want a datetime in UTC that you *immediately format*, attaching the `UTC` object rather than using `utcnow` would lead to a performance regression without any actual benefit (and sometimes it's actually a bit of a pain). For example:

print(datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")) print(datetime.utcnow().isoformat())

would have to turn into:

print(datetime.now(UTC).strftime("%Y-%m-%d %H:%M:%S")) print(datetime.now(UTC).replace(tzinfo=None).isoformat())

The first of these is slower but no safer than the `utcnow` version, and the second of these actually involves stripping the UTC object from the result of `now` just to avoid `isoformat`'s behavior where it automatically appends the time zone offset to aware datetimes.

I haven't looked at it, but it may be that we can special-case `datetime.timezone.utc` in the `now()` and `fromtimestamp()` constructors so that `datetime.now(timezone.utc)` is just as fast as `datetime.utcnow()`, in which case the first example is not a huge deal, and maybe the second performance regression isn't so bad that it is worth leaving around footguns like this. It'd be nice to put some numbers on that.

Best, Paul On 12/27/21 19:33, Brock Mendel wrote:

Now that zoneinfo is in the stdlib, I'd like to revisit the idea of either a) making utcnow/utcfromtimestamp return tzaware or b) deprecating in favor of explicitly passing tzinfos to now/fromtimestamp.

Thoughts?

On Mon, Apr 22, 2019 at 10:54 AM Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

...
...
what if I'm in New York, but running on a JupyterLab system in Los Angeles?

In this case, it would be the responsibility of the JupyterLab designers to set the timezone of the kernel to something useful to the user. In the case of a single-user kernel that would be the user's timezone, in the case of multi-user kernels - most likely UTC or something that the users can agree upon. The location of the JupyterLab server should never be a consideration.

Setting the timezone is currently more complicated than it should be:

import sys, time sys.environ['TZ'] = 'America/New_York' time.tzset()

but I don't think we can improve that before python gets an integrated timezone database. For now, this is what you should do to make sure a remote system uses the timezone that you want. _______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.orghttps://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

Brock Mendel

7:22 p.m.

FWIW the proximate reason for bumping this thread is an issue reported to pandas caused by a user confused by datetime.utcnow().timestamp() behavior https://github.com/pandas-dev/pandas/issues/45054 I guess there's a couple of empirical questions here: 1) How common is this type of confusion? 2) How common is the utcnow().strftime() or utcnow().isoformat() usage where changing/deprecating would make a real perf/brevity difference? On Thu, Dec 30, 2021 at 2:19 AM Oren Tirosh wrote:

...

Since changing datetime would break compatibility, I experimented with a module called adatetime ('a' for aware) where timezone-naive objects are not allowed.

The class methods adatetime.datetime.fromtimestamp(t) and adatetime.datetime.utcfromtimestamp(t) produces objects that compare as equal. One is in the local timezone and the other is in UTC but they represent the same point in time. With the standard datetime module this results in unequal naive objects.

There is no such thing as a local time_t in POSIX. Internal representation is always UTC and broken-down human time (struct tm) always has an associated timezone. No mixups. The python datetime module supports a non-UTC int or float representation with the methods .fromtimestamp(), .utcfromtimestamp() and .timestamp(). In these methods the time in seconds may be implicitly treated or generated as local time. I consider this the "Original Sin" of an otherwise very well-designed module.

I experimented with various alternatives for how adatetime should behave when tzinfo is not provided: using either UTC or the local timezone as default or always require it to be passed explicitly. I think the best default is UTC, with a singleton named 'local' that can be passed as tzinfo to specify using the local timezone (same behavior as adding .astimezone(None)). For code that always passes an explicit tzinfo this makes adatetime almost 100% compatible to the standard datetime. The only difference is the behavior of .fromtimestsamp() / .utcfromtimestamp() being aware by default. To make it fully compatible would require either adding some new method for creating aware datetime to both datetime and adatetime or removing .utcfromtimestamp() from adatetime and making adatetime's .fromtimestamp() tzinfo argument not optional.

On Tue, 28 Dec 2021 at 04:24, Paul Ganssle wrote:

...
I don't really think zoneinfo being in the stdlib has much bearing on the question of utcnow/utctimestamp. We already had a UTC object in the standard library, and could have made this change any time, but chose not to because it would break backwards compatibility.

I'm definitely more in favor of deprecating it than I am of changing the behavior, which would be a major breaking change and one that is hard to warn about. Removing utcnow would probably be a lot of work for a lot of people, but many people are probably using it wrong anyway so that may be a net benefit.

That said, there is at least one use case I'm concerned about where the replacement is not great:

If you want a datetime in UTC that you *immediately format*, attaching the `UTC` object rather than using `utcnow` would lead to a performance regression without any actual benefit (and sometimes it's actually a bit of a pain). For example:

print(datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")) print(datetime.utcnow().isoformat())

would have to turn into:

print(datetime.now(UTC).strftime("%Y-%m-%d %H:%M:%S")) print(datetime.now(UTC).replace(tzinfo=None).isoformat())

The first of these is slower but no safer than the `utcnow` version, and the second of these actually involves stripping the UTC object from the result of `now` just to avoid `isoformat`'s behavior where it automatically appends the time zone offset to aware datetimes.

I haven't looked at it, but it may be that we can special-case `datetime.timezone.utc` in the `now()` and `fromtimestamp()` constructors so that `datetime.now(timezone.utc)` is just as fast as `datetime.utcnow()`, in which case the first example is not a huge deal, and maybe the second performance regression isn't so bad that it is worth leaving around footguns like this. It'd be nice to put some numbers on that.

Best, Paul On 12/27/21 19:33, Brock Mendel wrote:

Now that zoneinfo is in the stdlib, I'd like to revisit the idea of either a) making utcnow/utcfromtimestamp return tzaware or b) deprecating in favor of explicitly passing tzinfos to now/fromtimestamp.

Thoughts?

On Mon, Apr 22, 2019 at 10:54 AM Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

...
...
what if I'm in New York, but running on a JupyterLab system in Los Angeles?

In this case, it would be the responsibility of the JupyterLab designers to set the timezone of the kernel to something useful to the user. In the case of a single-user kernel that would be the user's timezone, in the case of multi-user kernels - most likely UTC or something that the users can agree upon. The location of the JupyterLab server should never be a consideration.

Setting the timezone is currently more complicated than it should be:

import sys, time sys.environ['TZ'] = 'America/New_York' time.tzset()

but I don't think we can improve that before python gets an integrated timezone database. For now, this is what you should do to make sure a remote system uses the timezone that you want. _______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.orghttps://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

Oren Tirosh

8:32 p.m.

In over 20 years of using python I have found this issue, in its various manifestations, to be the biggest source of confusion for people with datetime module. On Thu, 30 Dec 2021 at 18:22 Brock Mendel wrote:

...

FWIW the proximate reason for bumping this thread is an issue reported to pandas caused by a user confused by datetime.utcnow().timestamp() behavior https://github.com/pandas-dev/pandas/issues/45054

I guess there's a couple of empirical questions here: 1) How common is this type of confusion? 2) How common is the utcnow().strftime() or utcnow().isoformat() usage where changing/deprecating would make a real perf/brevity difference?

On Thu, Dec 30, 2021 at 2:19 AM Oren Tirosh wrote:

...
Since changing datetime would break compatibility, I experimented with a module called adatetime ('a' for aware) where timezone-naive objects are not allowed.

The class methods adatetime.datetime.fromtimestamp(t) and adatetime.datetime.utcfromtimestamp(t) produces objects that compare as equal. One is in the local timezone and the other is in UTC but they represent the same point in time. With the standard datetime module this results in unequal naive objects.

There is no such thing as a local time_t in POSIX. Internal representation is always UTC and broken-down human time (struct tm) always has an associated timezone. No mixups. The python datetime module supports a non-UTC int or float representation with the methods .fromtimestamp(), .utcfromtimestamp() and .timestamp(). In these methods the time in seconds may be implicitly treated or generated as local time. I consider this the "Original Sin" of an otherwise very well-designed module.

I experimented with various alternatives for how adatetime should behave when tzinfo is not provided: using either UTC or the local timezone as default or always require it to be passed explicitly. I think the best default is UTC, with a singleton named 'local' that can be passed as tzinfo to specify using the local timezone (same behavior as adding .astimezone(None)). For code that always passes an explicit tzinfo this makes adatetime almost 100% compatible to the standard datetime. The only difference is the behavior of .fromtimestsamp() / .utcfromtimestamp() being aware by default. To make it fully compatible would require either adding some new method for creating aware datetime to both datetime and adatetime or removing .utcfromtimestamp() from adatetime and making adatetime's .fromtimestamp() tzinfo argument not optional.

On Tue, 28 Dec 2021 at 04:24, Paul Ganssle wrote:

...
I don't really think zoneinfo being in the stdlib has much bearing on the question of utcnow/utctimestamp. We already had a UTC object in the standard library, and could have made this change any time, but chose not to because it would break backwards compatibility.

I'm definitely more in favor of deprecating it than I am of changing the behavior, which would be a major breaking change and one that is hard to warn about. Removing utcnow would probably be a lot of work for a lot of people, but many people are probably using it wrong anyway so that may be a net benefit.

That said, there is at least one use case I'm concerned about where the replacement is not great:

If you want a datetime in UTC that you *immediately format*, attaching the `UTC` object rather than using `utcnow` would lead to a performance regression without any actual benefit (and sometimes it's actually a bit of a pain). For example:

print(datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")) print(datetime.utcnow().isoformat())

would have to turn into:

print(datetime.now(UTC).strftime("%Y-%m-%d %H:%M:%S")) print(datetime.now(UTC).replace(tzinfo=None).isoformat())

The first of these is slower but no safer than the `utcnow` version, and the second of these actually involves stripping the UTC object from the result of `now` just to avoid `isoformat`'s behavior where it automatically appends the time zone offset to aware datetimes.

I haven't looked at it, but it may be that we can special-case `datetime.timezone.utc` in the `now()` and `fromtimestamp()` constructors so that `datetime.now(timezone.utc)` is just as fast as `datetime.utcnow()`, in which case the first example is not a huge deal, and maybe the second performance regression isn't so bad that it is worth leaving around footguns like this. It'd be nice to put some numbers on that.

Best, Paul On 12/27/21 19:33, Brock Mendel wrote:

Now that zoneinfo is in the stdlib, I'd like to revisit the idea of either a) making utcnow/utcfromtimestamp return tzaware or b) deprecating in favor of explicitly passing tzinfos to now/fromtimestamp.

Thoughts?

On Mon, Apr 22, 2019 at 10:54 AM Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

...
...
what if I'm in New York, but running on a JupyterLab system in Los Angeles?

In this case, it would be the responsibility of the JupyterLab designers to set the timezone of the kernel to something useful to the user. In the case of a single-user kernel that would be the user's timezone, in the case of multi-user kernels - most likely UTC or something that the users can agree upon. The location of the JupyterLab server should never be a consideration.

Setting the timezone is currently more complicated than it should be:

import sys, time sys.environ['TZ'] = 'America/New_York' time.tzset()

but I don't think we can improve that before python gets an integrated timezone database. For now, this is what you should do to make sure a remote system uses the timezone that you want. _______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.orghttps://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

_______________________________________________ Datetime-SIG mailing list -- datetime-sig@python.org To unsubscribe send an email to datetime-sig-leave@python.org https://mail.python.org/mailman3/lists/datetime-sig.python.org/ The PSF Code of Conduct applies to this mailing list: https://www.python.org/psf/codeofconduct/

849

Age (days ago)

1838

Last active (days ago)

List overview

Download

12 comments

7 participants

participants (7)

Alexander Belopolsky
Brock Mendel
Chris Barker
Chris Barker - NOAA Federal
Oren Tirosh
Paul Ganssle
Paul Ganssle

should utcnow and utcfromtimestamp return tz-aware?

tags

participants (7)