Francesc, The datetime proposal is very impressive in its depth and thought. For me as well as many other people this would be a massive improvement to numpy and allow numpy to get a foothold in areas like econometrics where R/S is now dominant. I had one question regarding casting of strings: I think it would be ideal if things like the following worked:

...

...

series = numpy.array(['1970-02-01','1970-09-01'], dtype = 'datetime64[D]') series == '1970-02-01' [True, False]

I view this as similar to:

...

...

series = numpy.array([1,2,3], dtype=float) series == 2 [False,True,False]

1. However it does numpy recognizes that an int is comparable with a float and does the float cast. I think you want the same behavior between strings that parse into dates and date arrays. Some might object that the relationship between string and date is more tenuous than float and int, which is true, but having used my own homespun date array numpy extension for over a year, I've found that the first thing I did was wrap it into an object that handles these string->date translations elegantly and that made it infinately more usable from an ipython session. 2. Even more important to me, however, is the issue of date parsing. The mx library does many things badly but it does do a great job of parsing dates of many formats. When you parse '1/1/95' or 1995-01-01' it knows that you mean 19950101 which is really nice. I believe the scipy timeseries code for parsing dates is based on it. I would highly suggest starting with that level of functionality. The one major issue with it is an uninterpretable date doesn't throw an error but becomes whatever date is right now. That is obviously unfavorable. 3. Finally my current implementation uses floats uses nan to represent an invalid date. When you assign an element of an date array to None it uses nan as the value. When you assign a real date it puts in the equivalent floating point value. I have found this to be hugely beneficial and just wanted to float the idea of reserving a value to indicate the floating point equivalent of nan. People might prefer masked arrays as a solution, but I just wanted to float the idea. Forgive me if any of this has already been covered. There has been a lot of volume on this subject and I've tried to read it all diligently but may have missed a point or two. --Tom

Tom Denniston (el 2008-07-29 a les 12:21:39 -0500) va dir::

[...] I think it would be ideal if things like the following worked:

...

...

...

series = numpy.array(['1970-02-01','1970-09-01'], dtype = 'datetime64[D]') series == '1970-02-01' [True, False]

I view this as similar to:

...

...

...

series = numpy.array([1,2,3], dtype=float) series == 2 [False,True,False]

1. However it does numpy recognizes that an int is comparable with a float and does the float cast. I think you want the same behavior between strings that parse into dates and date arrays. Some might object that the relationship between string and date is more tenuous than float and int, which is true, but having used my own homespun date array numpy extension for over a year, I've found that the first thing I did was wrap it into an object that handles these string->date translations elegantly and that made it infinately more usable from an ipython session.

That may be feasible as long as there is a very clear rule for what time units you get given a string. For instance, '1970' could yield years and '1970-03-12T12:00' minutes, but then we don't have a way of creating a time in business days... However, it looks interesting. Any more people interested in this behaviour?

2. Even more important to me, however, is the issue of date parsing. The mx library does many things badly but it does do a great job of parsing dates of many formats. When you parse '1/1/95' or 1995-01-01' it knows that you mean 19950101 which is really nice. I believe the scipy timeseries code for parsing dates is based on it. I would highly suggest starting with that level of functionality. The one major issue with it is an uninterpretable date doesn't throw an error but becomes whatever date is right now. That is obviously unfavorable.

Umm, that may get quite complex. E.g. does '1/2/95' refer to February the 1st. or January the 2nd.? There are sooo many date formats and standards that maybe using an external parser code (like mx, TimeSeries or even datetime/strptime) for them would be preferable. I think the ISO 8601 is enough for a basic, well defined time string support. At least to start with.

3. Finally my current implementation uses floats uses nan to represent an invalid date. When you assign an element of an date array to None it uses nan as the value. When you assign a real date it puts in the equivalent floating point value. I have found this to be hugely beneficial and just wanted to float the idea of reserving a value to indicate the floating point equivalent of nan. People might prefer masked arrays as a solution, but I just wanted to float the idea. [...]

Good news! Our next proposal includes a "Not a Time" value which came around due to the impossibility of converting some times into business days. Stay tuned. However I should point out that the NaT value isn't as powerful as the floating-point NaN, since the former is completely lacking of any sense to hardware, and patching that in all cases would make computations quite slower. Using floating point values doesn't look like an option anymore, since they don't have a fixed precision given a time unit. Cheers, :: Ivan Vilata i Balaguer @ Intellectual Monopoly hinders Innovation! @ http://www.selidor.net/ @ http://www.nosoftwarepatents.com/ @

On Tuesday 29 July 2008 14:08:28 Francesc Alted wrote:

A Tuesday 29 July 2008, David Huard escrigué:

Hmm, the idea of the ``.add()`` and ``.subtract()`` methods is tempting, but I not sure it is a good idea to add new methods to the ndarray object that are meant to be used with just the date/time dtype.

I'm afraid that I'm -1 here.

I fully agree with Francesc, .add and .subtract will be quite confusing. About inplace conversions, the right-end (other) is cast to the type of the left end (self) by default following the basic rule of casting when there's no ambiguity and raising an exception otherwise ?

On Tuesday 29 July 2008 15:14:13 Ivan Vilata i Balaguer wrote:

Pierre GM (el 2008-07-29 a les 12:38:19 -0400) va dir::

...

...

Relative time versus relative time ----------------------------------

This case would be the same than the previous one (absolute vs absolute). Our proposal is to forbid this operation if the time units of the operands are different.

Mmh, less sure on this one. Can't we use a hierarchy of time units, and force to the lowest ?

For example:

...

...

...

numpy.ones(3, dtype="t8[Y]") + 3*numpy.ones(3, dtype="t8[M]") array([15,15,15], dtype="t8['M']")

I agree that adding ns to years makes no sense, but ns to s ? min to hr or days ? In short: systematically raising an exception looks a bit too drastic. There are some simple unambiguous cases that sould be allowed (Y+M, Y+Q, M+Q, H+D...)

Do you mean using the most precise unit for operations with "near enough", different units? I see the point, but what makes me doubt about it is giving the user the false impression that the most precise unit is *always* expected. I'd rather spare the user as many surprises as possible, by simplifying rules in favour of explicitness (but that may be debated).

Let me rephrase: Adding different relative time units should be allowed when there's no ambiguity on the output: For example, a relative year timedelta is always 12 month timedeltas, or 4 quarter timedeltas. In that case, I should be able to do:

...

...

numpy.ones(3, dtype="t8[Y]") + 3*numpy.ones(3, dtype="t8[M]") array([15,15,15], dtype="t8['M']") numpy.ones(3, dtype="t8[Y]") + 3*numpy.ones(3, dtype="t8[Q]") array([7,7,7], dtype="t8['Q']")

Similarly: * an hour is always 3600s, so I could add relative s/ms/us/ns timedeltas to hour timedeltas, and get the result in s/ms/us/ns. * A day is always 24h, so I could add relative hours and days timedeltas and get an hour timedelta * A week is always 7d, so W+D -> D However: * We can't tell beforehand how much days are in any month, so adding relative days and months would raise an exception. * Same thing with weeks and months/quarters/years There'll be only a limited number of time units, therefore a limited number of potential combinations between time units. It'd be just a matter of listing which ones are allowed and which ones will raise an exception.

...

...

Note: we refused to use the ``.astype()`` method because of the additional 'time_reference' parameter that will sound strange for other typical uses of ``.astype()``.

A method would be really, really helpful, though... [...]

Yay, but what doesn't seem to fit for me is that the method would only have sense to time values.

Well, what about a .tounit(new_unit, reference=None) ? By default, the reference would be None and default to the POSIX epoch. We could also go for .totunit (for to time unit)

NumPy is pretty orthogonal in that every method and attribute applies to every type. However, if "units" were to be adopted by NumPy, the method would fit in well. In fact, we are thinking of adding a ``unit`` attribute to dtypes to support time units (being ``None`` for normal NumPy types). But full unit support in NumPy looks so far away that I'm not sure to adopt the method.

Thanks for the insights. Cheers,

A Wednesday 30 July 2008, Ivan Vilata i Balaguer escrigué:

Pierre GM (el 2008-07-29 a les 15:47:52 -0400) va dir::

...

On Tuesday 29 July 2008 15:14:13 Ivan Vilata i Balaguer wrote:

...

Pierre GM (el 2008-07-29 a les 12:38:19 -0400) va dir::

...

...

Relative time versus relative time ----------------------------------

This case would be the same than the previous one (absolute vs absolute). Our proposal is to forbid this operation if the time units of the operands are different.

Mmh, less sure on this one. Can't we use a hierarchy of time units, and force to the lowest ?

For example:

...

...

>numpy.ones(3, dtype="t8[Y]") + 3*numpy.ones(3, > dtype="t8[M]") array([15,15,15], dtype="t8['M']")

I agree that adding ns to years makes no sense, but ns to s ? min to hr or days ? In short: systematically raising an exception looks a bit too drastic. There are some simple unambiguous cases that sould be allowed (Y+M, Y+Q, M+Q, H+D...)

Do you mean using the most precise unit for operations with "near enough", different units? I see the point, but what makes me doubt about it is giving the user the false impression that the most precise unit is *always* expected. I'd rather spare the user as many surprises as possible, by simplifying rules in favour of explicitness (but that may be debated).

Let me rephrase: Adding different relative time units should be allowed when there's no ambiguity on the output: For example, a relative year timedelta is always 12 month timedeltas, or 4

quarter timedeltas. In that case, I should be able to do:

...

...

...

numpy.ones(3, dtype="t8[Y]") + 3*numpy.ones(3, dtype="t8[M]")

array([15,15,15], dtype="t8['M']")

...

...

...

numpy.ones(3, dtype="t8[Y]") + 3*numpy.ones(3, dtype="t8[Q]")

array([7,7,7], dtype="t8['Q']")

Similarly: * an hour is always 3600s, so I could add relative s/ms/us/ns timedeltas to hour timedeltas, and get the result in s/ms/us/ns. * A day is always 24h, so I could add relative hours and days timedeltas and get an hour timedelta * A week is always 7d, so W+D -> D

However: * We can't tell beforehand how much days are in any month, so adding relative days and months would raise an exception. * Same thing with weeks and months/quarters/years

There'll be only a limited number of time units, therefore a limited number of potential combinations between time units. It'd be just a matter of listing which ones are allowed and which ones will raise an exception.

That's "keep the precision" over "keep the range". At first Francesc and I opted for "keep the range" because that's what NumPy does, e.g. when operating an int64 with an uint64. Then, since we weren't sure about what the best choice would be for the majority of users, we decided upon letting (or forcing) the user to be explicit. However, the use of time units and integer values is precisely intended to "keep the precision", and overflow won't be so frequent given the correct time unit and the span of uint64, so you may be right in the end. :)

Well, I do think that the "keep the precision" rule can be a quite sensible approach for this case, so I am in favor to it. Also, the Pierre suggestion of allowing automatic castings for all the time units except when the 'Y'ear and 'M'onth are involved makes a lot of sense too. I'll adopt these for the third version of the proposal then.

...

...

...

...

Note: we refused to use the ``.astype()`` method because of the additional 'time_reference' parameter that will sound strange for other typical uses of ``.astype()``.

A method would be really, really helpful, though... [...]

Yay, but what doesn't seem to fit for me is that the method would only have sense to time values.

Well, what about a .tounit(new_unit, reference=None) ? By default, the reference would be None and default to the POSIX epoch. We could also go for .totunit (for to time unit)

Yes, that'd be the signature for a method. The ``reference`` argument shouldn't be allowed for ``datetime64`` values (absolute times, no ambiguities) but it should be mandatory for ``timedelta64`` ones. Sorry, but I can't see the use of having a default reference, unless one wanted to work with Epoch-based deltas, which looks like an extremely particular case. Could you please show me a use case for having a reference defaulting to the POSIX epoch?

Yeah, I agree with Ivan in that a default reference time makes little sense for general relative times. IMO, and provided that we will be allowing an implicit casting for most of time units for relative vs relative and in absolute vs relative, the use of forced casting will not be as frequent, and that a function would be enough. Having said that, I still see the merit of method for some situations, so I'll mention that in the third proposal as a possible improvement. -- Francesc Alted

A Wednesday 30 July 2008, Pierre GM escrigué:

On Wednesday 30 July 2008 06:35:32 Francesc Alted wrote:

...

A Wednesday 30 July 2008, Ivan Vilata i Balaguer escrigué:

...

Pierre GM (el 2008-07-29 a les 15:47:52 -0400) va dir::

...

On Tuesday 29 July 2008 15:14:13 Ivan Vilata i Balaguer wrote:

...

Pierre GM (el 2008-07-29 a les 12:38:19 -0400) va dir::

[Pierre]

...

...

...

Well, what about a .tounit(new_unit, reference=None) ? By default, the reference would be None and default to the POSIX epoch. We could also go for .totunit (for to time unit)

[Ivan]

...

...

Yes, that'd be the signature for a method. The ``reference`` argument shouldn't be allowed for ``datetime64`` values (absolute times, no ambiguities) but it should be mandatory for ``timedelta64`` ones. Sorry, but I can't see the use of having a default reference, unless one wanted to work with Epoch-based deltas, which looks like an extremely particular case. Could you please show me a use case for having a reference defaulting to the POSIX epoch?

[Francesc]

...

Yeah, I agree with Ivan in that a default reference time makes little sense for general relative times. IMO, and provided that we will be allowing an implicit casting for most of time units for relative vs relative and in absolute vs relative, the use of forced casting will not be as frequent, and that a function would be enough. Having said that, I still see the merit of method for some situations, so I'll mention that in the third proposal as a possible improvement.

In my mind, .tounit(*args) should be available for both relative (timedeltas) and absolute (datetime) times.

Well, what we are proposing is that the conversion time unit method for absolute times would be '.astype()' because its semantics is respected in this case. The problem is with relative times, and only with conversions between years or months and the rest of time units. This is why I propose the adoption of just a humble function for this cases. Introducing a method (.tounit()) for the ndarray object that only is useful for the date/time types seems a bit too much to my eyes (but I can be wrong, indeed).

I agree that for relative times, a default reference is meaningless. However, for absolute times, there's only one possible reference, the POSIX epoch, right ?

That's correct.

Now, what format do you consider for this reference ?

Whatever that can be converted into a datetime64 scalar. Some examples: ref = '2001-04-01' ref = datetime.datetime(2001, 4, 1)

Moreover, could you give some more examples of interaction between datetime and timedelta ?

In the second proposal there are some examples of this interaction and I'm populating the third proposal with more examples yet. Just wait a bit (maybe a couple of hours) to see the new proposal. Cheers, -- Francesc Alted

A Wednesday 30 July 2008, Pierre GM escrigué:

...

...

Now, what format do you consider for this reference ?

Whatever that can be converted into a datetime64 scalar. Some examples:

ref = '2001-04-01' ref = datetime.datetime(2001, 4, 1)

Er, should I see ref as having a 'day' unit or 'business day' unit in that case? I know that 'business days' spoil the game, but Matt really needs them, so...

OK. I was wrong. Of course you need to specify the resolution, so the reference *should* be a NumPy scalar: ref = numpy.datetime64('2001-04-01', unit="B") # 'B'usiness days

...

...

Moreover, could you give some more examples of interaction between datetime and timedelta ?

In the second proposal there are some examples of this interaction and I'm populating the third proposal with more examples yet. Just wait a bit (maybe a couple of hours) to see the new proposal.

OK, with pleasure. It's just that I have trouble understanding the meaning of something like t2 = numpy.ones(5, dtype="datetime64[s]")

That's five times one second after the epoch, right ? But in what circumstances would you need t2 ?

I'm not sure I follow you. This is just an example so as to produce an array of time objects quickly. In general, you should also be able to produce the same result by doing: t2 = numpy.array(['1970-01-01T00:00:05', '1970-01-01T00:00:05', '1970-01-01T00:00:05', '1970-01-01T00:00:05', '1970-01-01T00:00:05', dtype="datetime64[s]") which is more visual, but has the drawback that it's just too long for documenting purposes. When you don't need the values for some examples, conciseness is a virtue. -- Francesc Alted

A Wednesday 30 July 2008, Pierre GM escrigué:

Which brings me to another question: datetime64 and timedelta64 are just dtypes, therefore they don't impose any restriction (in terms of uniqueness of elements, ordering of the elements...) on the underlying ndarray, right ?

That's right. Perhaps this is the reason why you got mystified about the numpy.ones(5, dtype="datetime64[s]") thing. -- Francesc Alted

A Wednesday 30 July 2008, Tom Denniston escrigué:

When people are refering to busienss days are you talking about weekdays or are you saying weekday non-holidays?

Plain weekdays. Taking in account holidays for all the world round would be certainly much more complex than timezones, which neither are being considered in this proposal. -- Francesc Alted

A Wednesday 30 July 2008, Tom Denniston escrigué:

If it's really just weekdays why not call it that instead of using a term like business days that (quite confusingly) suggests holidays are handled properly?

Well, we were adopting the name from the TimeSeries package. Perhaps the authors can answer this better than me.

Also, I view the timezone and holiday issues as totally seperate. I would definately NOT recommend basing holidays on a timezone because holidays are totally unrelated to timezones. Usually when you deal with holidays, because they vary by application and country and change over time, you provide a calendar as an outside input. That would be very useful if that were allowed, but might make the implementation rather complex.

Yeah, I agree in that timezone and holiday issues as totally separate issues. I only wanted to stress out that the implementation of these things is *complex*, and that this was the reason to not consider them. -- Francesc Alted

Yes this all makes a lot of sense. I would propose changing the name from business days to weekdays though. Does anyone object wih that? On 7/30/08, Ivan Vilata i Balaguer wrote:

Tom Denniston (el 2008-07-30 a les 13:12:45 -0500) va dir::

...

If it's really just weekdays why not call it that instead of using a term like business days that (quite confusingly) suggests holidays are handled properly?

Yes, that may be a better term. I guess we didn't choose that because we aren't native English speakers, and because TimeSeries was already using the other term.

...

Also, I view the timezone and holiday issues as totally seperate. I would definately NOT recommend basing holidays on a timezone because holidays are totally unrelated to timezones. Usually when you deal with holidays, because they vary by application and country and change over time, you provide a calendar as an outside input. That would be very useful if that were allowed, but might make the implementation rather complex.

I think that what Francesc was trying to say is that taking holidays into account would be way too difficult to be implemented. Timezones were just an example of another (unrelated) feature we left out due to its complexity.

...

On 7/30/08, Francesc Alted wrote:

...

A Wednesday 30 July 2008, Tom Denniston escrigué:

...

When people are refering to busienss days are you talking about weekdays or are you saying weekday non-holidays?

Plain weekdays. Taking in account holidays for all the world round would be certainly much more complex than timezones, which neither are being considered in this proposal.

::

Ivan Vilata i Balaguer @ Intellectual Monopoly hinders Innovation! @ http://www.selidor.net/ @ http://www.nosoftwarepatents.com/ @

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)

iQCVAwUBSJC1HYB9xakSvZSBAQIqGAP9FES0aN1ioJWL9NsBggBCmdxdA0d973nr 0dP00xdaq9CeGfNa78NOxzphxsL3kiKR4t6eDE1y3DwyhFV9+X9B+w/pFOcZAuRX fIlkRHOiQn0SODf287LwAsSab2dKgL+HpiJjAa45QFMDNUUmz7KCa2HKrBZSdL5y rMfxlbKwAwA= =Vt/d -----END PGP SIGNATURE-----

_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

A Thursday 31 July 2008, Matt Knox escrigué:

While on the topic of FAME... being a financial analyst, I really am quite fond of the multitude of quarterly frequencies we have in the timeseries package (with different year end points) because they are very useful when doing things like "calenderizing" earnings from companies with different fiscal year ends. These frequencies are included in FAME, which makes sense since it targets financial users. I know Pierre likes them too for working with different seasons. I think it would be ok to leave them out of an initial implementation, but it might be worth keeping in mind during the design phase about how the dtype could be extended to incorporate such things.

Well, introducing a quarter should not be difficult. We just wanted to keep the set of supported time units under a minimum (the list is already quite large). We thought that the quarter fits better as a 'derived' time unit, similarly as biweekly, semester or biyearly (to name just a few examples). However, if quarters are found to be much more important than other derived time units, they can go into the proposal too.

...

...

As forbidding operations among absolute/absolute and relative/relative types can be unacceptable in many situations, we are proposing an explicit casting mechanism so that the user can inform about the desired time unit of the outcome. For this, a new NumPy function, called, say, ``numpy.change_unit()`` (this name is for the purposes of the discussion and can be changed) will be provided. The signature for the function will be:

change_unit(time_object, new_unit, reference)

where 'time_object' is the time object whose unit is to be changed, 'new_unit' is the desired new time unit, and 'reference' is an absolute date that will be used to allow the conversion of relative times in case of using time units with an uncertain number of smaller time units (relative years or months cannot be expressed in days). For

example, that would allow to do:

...

...

...

numpy.change_unit( numpy.array([1,2], 'T[Y]'), 'T[d]' )

array([365, 731], dtype="datetime64[d]")

If I understand you correctly, this is very close to the "asfreq" method of the Date/DateArray/TimeSeries classes in the timeseries module. One key element missing here (from my point of view anyway) is an equivalent of the 'relation' parameter in the asfreq method in the timeseries module. This is only used when converting from a lower frequency to a higher frequency (eg. annual to daily). For example...

...

...

...

a = ts.Date(freq='Annual', year=2007) a.asfreq('Daily', 'START')

...

...

...

a.asfreq('Daily', 'END')

This is another one of those things that I use all the time. Now whether it belongs in the core dtype, or some extension module I'm not sure... but it's an important feature in the timeseries module.

I agree that such a 'relation' parameter in the proposed 'change_timeunit' could be handy in many situations. It should be applicable only to absolute times though. With this, the signature for the function would be: change_timeunit(time_object, new_unit, relation, reference) where 'relation' only can be used with absolute times and 'reference' only with relative times. Who knows, perhaps in the future one can find a way to implement such a 'change_timeunit' function as methods without disturbing too much the method schema of the ndarray objects. Cheers, -- Francesc Alted

A Thursday 31 July 2008, Alan G Isaac escrigué:

...

A Thursday 31 July 2008, Matt Knox escrigué:

...

While on the topic of FAME... being a financial analyst, I really am quite fond of the multitude of quarterly frequencies we have in the timeseries package (with different year end points) because they are very useful when doing things like "calenderizing" earnings from companies with different fiscal year ends.

On Thu, 31 Jul 2008, Francesc Alted apparently wrote:

...

Well, introducing a quarter should not be difficult. We just wanted to keep the set of supported time units under a minimum (the list is already quite large). We thought that the quarter fits better as a 'derived' time unit, similarly as biweekly, semester or biyearly (to name just a few examples). However, if quarters are found to be much more important than other derived time units, they can go into the proposal too.

Quarterly frequency is probably the most analyzed frequency in macroeconometrics.

Widely used macroeconometrics packages (e.g., EViews) traditionally support only three explicit frequencies: annual, quarterly, and monthly.

I see. However, I forgot to mention that another reason for not including the quarters is that they normally need flexibility to be defined as starting in *any* month of the year. As we don't wanted to provide an ``origin`` metadata in the proposal (things got too complex already, as you can see in the third proposal that I sent to this list yesterday), then the usefulness of such an 'unflexible' quarters would be rather limited. So, in the end, I think it is best to avoid them for the dtype (and add support for them in the ``Date`` class). Cheers, -- Francesc Alted