Hi,
I actually arrived at this by first trying to use pandas.Timestamp and getting very frustrated about it. With pandas, I get:
>>> pd.Timestamp.now()
Timestamp('2020-11-06 09:45:24.249851')
I find the whole notion of a "timezone naive timestamp" to be nearly meaningless. A timestamp should mean a moment in time (as the current numpy documentation defines very well). A "naive timestamp" doesn't mean anything. It's exactly like a "unit naive length". I can have a Length type which just takes a number, and be very happy that it works both if my "unit zone" is inches or centimeters. So "Length(3)" will mean 3 cm in most of the world and 3 inches in the US. But then, if I get "Length(3)" from someone, I can't be sure what length it refers to.
So currently, this happens with pandas timestamps:
>>> os.environ['TZ'] = 'UTC'; time.tzset()
... t0 = pd.Timestamp.now()
... time.sleep(1)
... os.environ['TZ'] = 'EST-5'; time.tzset()
... t1 = pd.Timestamp.now()
... t1 - t0
Timedelta('0 days 05:00:01.001583')
This is not just theoretical - I actually need to work with data from several devices, each in its own time zone. And I need to know that I won't get such meaningless results.
And you can even get something like this:
>>> t0 = pd.Timestamp.now()
... time.sleep(10)
... t1 = pd.Timestamp.now()
... t1 - t0
Timedelta('0 days 01:00:10.001583')
if the first measurement happened to be in winter time and the second measurement happened to be in daylight saving time.
The solution is simple, and is what datetime64 used to do before the change - have a type that just represents a moment in time. It's not "in UTC" - it just stores the number of seconds that passed since an agreed moment in time (which is usually 1970-01-01 02:00+0200, which is more commonly referred to as 1970-01-01 00:00Z - it's the exact same moment).
I think it would make things clearer if I'll mention that there are operations that are not dealing with timestamps. For example, it's meaningless to ask what is the year of a timestamp - it may depend on the time zone. These are always *human* related questions, that depend on certain human conventions. We can call them "calendar questions". For these types of questions, a type that includes both a timestamp and a timezone offset (in minutes from UTC) can be useful. Some questions even require full timezone information, meaning a function that defines what's the timezone offset for each moment. However, I don't think numpy should deal with those calendar issues. As a very simple example, even for "timestamp+offset" types, it's not clear how to compare them - should values with the same timestamp and different offsets be considered equal or not? And in virtually all of my data analysis, this calendar aspect has nothing to do with the questions I'm trying to answer.
I have a suggestion. Instead of changing datetime64 (which I consider to be ill-defined, but never mind), add a new type called "timestamp64". It will have the exact same behavior as datetime64 had before the change, except that its only allowed units will be seconds, milliseconds, microseconds and nanoseconds. Removing the longer units will make it clear that it doesn't deal with calendar and dates. Also, all the business day functionality will not be applicable to timestamp64. In order to get calendar information (such as the year) from timestamp64, you will have to manually convert it to python's datetime (or to np.datetime64) with an explicit timezone (utc, local, an offset, or a timezone object).
What do you think?
Thanks,
Noam