Mailman 3 July 2008 - NumPy-Discussion

The date/time dtype and the casting issue
by Francesc Alted July 31, 2008

July 31, 2008

Hi, During the making of the date/time proposals and the subsequent discussions in this list, we have changed a couple of times our point of view about the way how the castings would work between different date/time types and the different time units (previously called resolutions). So I'd like to expose this issue in detail here, and give yet another new proposal about this, so as to gather feedback from the community before consolidating it in the final date/time proposal. Casting proposal for date/time types ==================================== The operations among the proposed date/time types can be divided in three groups: * Absolute time versus relative time * Absolute time versus absolute time * Relative time versus relative time Now, here are our considerations for each case: Absolute time versus relative time ---------------------------------- We think that in this case the absolute time should have priority for determining the time unit of the outcome. That would represent what the people wants to do most of the times. For example, this would allow to do: >>> series = numpy.array(['1970-01-01', '1970-02-01', '1970-09-01'], dtype='datetime64[D]') >>> series2 = series + numpy.timedelta(1, 'Y') # Add 2 relative years >>> series2 array(['1972-01-01', '1972-02-01', '1972-09-01'], dtype='datetime64[D]') # the 'D'ay time unit has been chosen Absolute time versus absolute time ---------------------------------- When operating (basically, only the substraction will be allowed) two absolute times with different unit times, we are proposing that the outcome would be to raise an exception. This is because the ranges and timespans of the different time units can be very different, and it is not clear at all what time unit will be preferred for the user. For example, this should be allowed: >>> numpy.ones(3, dtype="T8[Y]") - numpy.zeros(3, dtype="T8[Y]") array([1, 1, 1], dtype="timedelta64[Y]") But the next should not: >>> numpy.ones(3, dtype="T8[Y]") - numpy.zeros(3, dtype="T8[ns]") raise numpy.IncompatibleUnitError # what unit to choose? Relative time versus relative time ---------------------------------- This case would be the same than the previous one (absolute vs absolute). Our proposal is to forbid this operation if the time units of the operands are different. For example, this should be allowed: >>> numpy.ones(3, dtype="t8[Y]") + 3*numpy.ones(3, dtype="t8[Y]") array([4, 4, 4], dtype="timedelta64[Y]") But the next should not: >>> numpy.ones(3, dtype="t8[Y]") + numpy.zeros(3, dtype="t8[fs]") raise numpy.IncompatibleUnitError # what unit to choose? Introducing a time casting function ----------------------------------- As forbidding operations among absolute/absolute and relative/relative types can be unacceptable in many situations, we are proposing an explicit casting mechanism so that the user can inform about the desired time unit of the outcome. For this, a new NumPy function, called, say, ``numpy.change_unit()`` (this name is for the purposes of the discussion and can be changed) will be provided. The signature for the function will be: change_unit(time_object, new_unit, reference) where 'time_object' is the time object whose unit is to be changed, 'new_unit' is the desired new time unit, and 'reference' is an absolute date that will be used to allow the conversion of relative times in case of using time units with an uncertain number of smaller time units (relative years or months cannot be expressed in days). For example, that would allow to do: >>> numpy.change_unit( numpy.array([1,2], 'T[Y]'), 'T[d]' ) array([365, 731], dtype="datetime64[d]") or: >>> ref = numpy.datetime64('1971', 'T[Y]') >>> numpy.change_unit( numpy.array([1,2], 't[Y]'), 't[d]', ref ) array([366, 365], dtype="timedelta64[d]") Note: we refused to use the ``.astype()`` method because of the additional 'time_reference' parameter that will sound strange for other typical uses of ``.astype()``. Opinions? -- Francesc Alted

7 29

[ANNOUNCE] Traits 3.0 has been released
by Dave Peterson July 31, 2008

July 31, 2008

Hello, I am very pleased to announce that Traits 3.0 has just been released! All Traits projects have been registered with PyPi (aka The Cheeseshop) and each project's listing on PyPi currently includes a source tarball. In the near future, we will also upload binary eggs for Windows and Mac OS X platforms. Installation of Traits 3.0 is now as simple as: easy_install Traits The Traits projects include: http://pypi.python.org/pypi?:action=display&name=Traits&version=3.0.0 http://pypi.python.org/pypi?:action=display&name=TraitsGUI&version=3.0.0 http://pypi.python.org/pypi?:action=display&name=TraitsBackendQt&version=3.… http://pypi.python.org/pypi?:action=display&name=TraitsBackendWX&version=3.… The Traits project is at the center of all Enthought Tool Suite development and has changed the mental model used at Enthought for programming in the already extremely efficient Python programming language. We encourage everyone to join us in enjoying the productivity gains from using such a powerful approach. The Traits project allows Python programmers to use a special kind of type definition called a trait, which gives object attributes some additional characteristics: * Initialization: A trait has a default value, which is automatically set as the initial value of an attribute before its first use in a program. * Validation: A trait attribute's type is explicitly declared. The type is evident in the code, and only values that meet a programmer-specified set of criteria (i.e., the trait definition) can be assigned to that attribute. * Delegation: The value of a trait attribute can be contained either in the defining object or in another object delegated to by the trait. * Notification: Setting the value of a trait attribute can notify other parts of the program that the value has changed. * Visualization: User interfaces that allow a user to interactively modify the value of a trait attribute can be automatically constructed using the trait's definition. (This feature requires that a supported GUI toolkit be installed. If this feature is not used, the Traits project does not otherwise require GUI support.) A class can freely mix trait-based attributes with normal Python attributes, or can opt to allow the use of only a fixed or open set of trait attributes within the class. Trait attributes defined by a classs are automatically inherited by any subclass derived from the class. -- Dave

3 2

Example of numpy cov() not correct?
by Anthony Kong July 31, 2008

July 31, 2008

Hi, all, I am trying out the example here (http://www.scipy.org/Numpy_Example_List_With_Doc#cov) >>> from numpy import * ... >>> T = array([1.3, 4.5, 2.8, 3.9]) >>> P = array([2.7, 8.7, 4.7, 8.2]) >>> cov(T,P) The answer is supposed to be 3.9541666666666657 The result I got is instead a cov matrix array([[ 1.97583333, 3.95416667], [ 3.95416667, 8.22916667]]) So, I just wanna confirm this particular example may be no longer correct. I am using python 2.4.3 with numpy 1.1.0 on MS win Cheers, Anthony NOTICE This e-mail and any attachments are confidential and may contain copyright material of Macquarie Group Limited or third parties. If you are not the intended recipient of this email you should not read, print, re-transmit, store or act in reliance on this e-mail or any attachments, and should destroy all copies of them. Macquarie Group Limited does not guarantee the integrity of any emails or any attached files. The views or opinions expressed are the author's own and may not reflect the views or opinions of Macquarie Group Limited.

4 4

NumPy won't work with distutils loaded?
by Dag Sverre Seljebotn July 31, 2008

July 31, 2008

While developing a testcase for NumPy in Cython, I had the problem demonstrated below. Essentially this means that I couldn't run testcases in our Cython testing framework (though I will work around it). Is this some known restriction in how NumPy can be used, is it work in progress, a problem with my environment/Ubuntu, or should I file a bug report? This was on an up-to-date Ubunty Hardy 64-bit i386 system (with distro numpy, Python and distutils). Py> import numpy is ok, however (note the uppercase UnixCCompiler in dir(...)): Py> import distutils.unixccompiler Py> dir(distutils.unixccompiler) ['CCompiler', 'CompileError', 'DistutilsExecError', 'LibError', 'LinkError', 'NoneType', 'StringType', 'UnixCCompiler', '__builtins__', '__doc__', '__file__', '__name__', '__revision__', '_darwin_compiler_fixup', 'copy', 'gen_lib_options', 'gen_preprocess_options', 'log', 'newer', 'os', 'sys', 'sysconfig'] Py> import numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.5/site-packages/numpy/__init__.py", line 37, in <module> import testing File "/usr/lib/python2.5/site-packages/numpy/testing/__init__.py", line 3, in <module> from numpytest import * File "/usr/lib/python2.5/site-packages/numpy/testing/numpytest.py", line 19, in <module> from numpy.distutils.exec_command import splitcmdline File "/usr/lib/python2.5/site-packages/numpy/distutils/__init__.py", line 6, in <module> import ccompiler File "/usr/lib/python2.5/site-packages/numpy/distutils/ccompiler.py", line 393, in <module> setattr(getattr(_m, _cc+'compiler'), 'gen_lib_options', AttributeError: 'module' object has no attribute 'unixccompiler' -- Dag Sverre

1 1

labeling decorators
by Fernando Perez July 30, 2008

July 30, 2008

Howdy (esp. Alan McIntyre): I've been using numpy's decorators a lot, many thanks to Matthew B and Alan for this code! Here's a snippet to auto-generate labeling decorators that might come in handy to avoid repetition in creating decos like @slow & friends. It's doctested as well as validating things a bit, feel free to use it if you find it useful. Cheers, f ### def make_label_dec(label,ds=None): """Factory function to create a decorator that applies one or more labels. :Parameters: label : string or sequence One or more labels that will be applied by the decorator to the functions it decorates. Labels are attributes of the decorated function with their value set to True. :Keywords: ds : string An optional docstring for the resulting decorator. If not given, a default docstring is auto-generated. :Returns: A decorator. :Examples: A simple labeling decorator: >>> slow = make_label_dec('slow') >>> print slow.__doc__ Labels a test as 'slow' And one that uses multiple labels and a custom docstring: >>> rare = make_label_dec(['slow','hard'], ... "Mix labels 'slow' and 'hard' for rare tests.") >>> print rare.__doc__ Mix labels 'slow' and 'hard' for rare tests. Now, let's test using this one: >>> @rare ... def f(): pass ... >>> >>> f.slow True >>> f.hard True """ if isinstance(label,basestring): labels = [label] else: labels = label # Validate that the given label(s) are OK for use in setattr() by doing a # dry run on a dummy function. tmp = lambda : None for label in labels: setattr(tmp,label,True) # This is the actual decorator we'll return def decor(f): for label in labels: setattr(f,label,True) return f # Apply the user's docstring if ds is None: ds = "Labels a test as %r" % label decor.__doc__ = ds return decor

1 0

Suppressing skipped tests
by Stéfan van der Walt July 30, 2008

July 30, 2008

Alan, If others agree, could we suppress the output of skipped tests unless specifically requested? They clutter the output, and makes it more difficult to see which "real" tests fail. Stéfan

2 2

Core dump during numpy.test()
by James Turner July 30, 2008

July 30, 2008

I have built NumPy 1.1.0 on RedHat Enterprise 3 (Linux 2.4.21 with gcc 3.2.3 and glibc 2.3.2) and Python 2.5.1. When I run numpy.test() I get a core dump, as follows. I haven't noticed any special errors during the build. Should I post the entire terminal output from "python setup.py install"? Maybe as an attachment? Let me know if I can provide any more info. Thanks a lot, James. --- [iraf@sbfirf01 DRSetupScripts]$ python Python 2.5.1 (r251:54863, Jul 28 2008, 19:08:11) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-20)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Numpy is installed in /astro/iraf/i686/gempylocal/lib/python2.5/site-packages/numpy Numpy version 1.1.0 Python version 2.5.1 (r251:54863, Jul 28 2008, 19:08:11) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-20)] Found 2/2 tests for numpy.core.tests.test_ufunc Found 143/143 tests for numpy.core.tests.test_regression Found 63/63 tests for numpy.core.tests.test_unicode Found 7/7 tests for numpy.core.tests.test_scalarmath Found 3/3 tests for numpy.core.tests.test_errstate Found 16/16 tests for numpy.core.tests.test_umath Found 12/12 tests for numpy.core.tests.test_records Found 70/70 tests for numpy.core.tests.test_numeric Found 18/18 tests for numpy.core.tests.test_defmatrix Found 36/36 tests for numpy.core.tests.test_numerictypes Found 286/286 tests for numpy.core.tests.test_multiarray Found 3/3 tests for numpy.core.tests.test_memmap Found 4/4 tests for numpy.distutils.tests.test_fcompiler_gnu Found 5/5 tests for numpy.distutils.tests.test_misc_util Found 2/2 tests for numpy.fft.tests.test_fftpack Found 3/3 tests for numpy.fft.tests.test_helper Found 15/15 tests for numpy.lib.tests.test_twodim_base Found 1/1 tests for numpy.lib.tests.test_regression Found 4/4 tests for numpy.lib.tests.test_polynomial Found 43/43 tests for numpy.lib.tests.test_type_check Found 1/1 tests for numpy.lib.tests.test_financial Found 1/1 tests for numpy.lib.tests.test_machar Found 53/53 tests for numpy.lib.tests.test_function_base Found 6/6 tests for numpy.lib.tests.test_index_tricks Found 15/15 tests for numpy.lib.tests.test_io Found 10/10 tests for numpy.lib.tests.test_arraysetops Found 1/1 tests for numpy.lib.tests.test_ufunclike Found 5/5 tests for numpy.lib.tests.test_getlimits Found 24/24 tests for numpy.lib.tests.test__datasource Found 49/49 tests for numpy.lib.tests.test_shape_base Found 3/3 tests for numpy.linalg.tests.test_regression Found 89/89 tests for numpy.linalg.tests.test_linalg Found 36/36 tests for numpy.ma.tests.test_old_ma Found 94/94 tests for numpy.ma.tests.test_core Found 15/15 tests for numpy.ma.tests.test_extras Found 17/17 tests for numpy.ma.tests.test_mrecords Found 4/4 tests for numpy.ma.tests.test_subclassing Found 7/7 tests for numpy.tests.test_random Found 16/16 tests for numpy.testing.tests.test_utils Found 5/5 tests for numpy.tests.test_ctypeslib ..................................................................................................................................................................................................................................................................................................Floating exception (core dumped)

5 18

A (third) proposal for implementing some date/time types in NumPy
by Francesc Alted July 30, 2008

July 30, 2008

Hi, After several weeks of gathering and pondering through very valuable feedback, we are happy to release the third (and final?) version of the proposal for the addition of the date/time types in NumPy. The bad news is that, due to a series of circumstances (apparently not related on how this job was being done) Enthought is no longer funding this project. However, we decided to go ahead and publish this in the hope that it could be useful in the future for someone brave enough to willing to do an implementation. So, while I'll be glad to answer the doubts or questions about this proposal, it should be clear that we (Ivan Vilata and myself) are not going to proceed with the implementation phase (unless Enthought would find fundings for sponsoring this in the future). Finally, I'd like to thank everybody that has been involved in the making of this proposal. Without their advices, suggestions and criticism we probably have never been able to realize all the requeriments and intrincacies that need a dtype for allowing a date/time manipulation which is adequate for numerical purposes. Cheers, -- Francesc Alted ==================================================================== A (third) proposal for implementing some date/time types in NumPy ==================================================================== :Author: Francesc Alted i Abad :Contact: faltet(a)pytables.com :Author: Ivan Vilata i Balaguer :Contact: ivan(a)selidor.net :Date: 2008-07-30 Executive summary ================= A date/time mark is something very handy to have in many fields where one has to deal with data sets. While Python has several modules that define a date/time type (like the integrated ``datetime`` [1]_ or ``mx.DateTime`` [2]_), NumPy has a lack of them. In this document, we are proposing the addition of a series of date/time types to fill this gap. The requirements for the proposed types are two-folded: 1) they have to be fast to operate with and 2) they have to be as compatible as possible with the existing ``datetime`` module that comes with Python. Types proposed ============== To start with, it is virtually impossible to come up with a single date/time type that fills the needs of every case of use. So, after pondering about different possibilities, we have stuck with *two* different types, namely ``datetime64`` and ``timedelta64`` (these names are preliminary and can be changed), that can have different time units so as to cover different needs. .. Important:: the time unit is conceived here as metadata that *complements* a date/time dtype, *without changing the base type*. It provides information about the *meaning* of the stored numbers, not about their *structure*. Now follows a detailed description of the proposed types. ``datetime64`` -------------- It represents a time that is absolute (i.e. not relative). It is implemented internally as an ``int64`` type. The internal epoch is the POSIX epoch (see [3]_). Like POSIX, the representation of a date doesn't take leap seconds into account. In time unit *conversions* and time *representations* (but not in other time computations), the value -2**63 (0x8000000000000000) is interpreted as an invalid or unknown date, *Not a Time* or *NaT*. See the section on time unit conversions for more information. Time units ~~~~~~~~~~ It accepts different time units, each of them implying a different time span. The table below describes the time units supported with their corresponding time spans. ======== ================ ========================== Time unit Time span (years) ------------------------- -------------------------- Code Meaning ======== ================ ========================== Y year [9.2e18 BC, 9.2e18 AC] M month [7.6e17 BC, 7.6e17 AC] W week [1.7e17 BC, 1.7e17 AC] B business day [3.5e16 BC, 3.5e16 AC] D day [2.5e16 BC, 2.5e16 AC] h hour [1.0e15 BC, 1.0e15 AC] m minute [1.7e13 BC, 1.7e13 AC] s second [ 2.9e9 BC, 2.9e9 AC] ms millisecond [ 2.9e6 BC, 2.9e6 AC] us microsecond [290301 BC, 294241 AC] ns nanosecond [ 1678 AC, 2262 AC] ======== ================ ========================== The value of an absolute date is thus *an integer number of units of the chosen time unit* passed since the internal epoch. When working with business days, Saturdays and Sundays are simply ignored from the count (i.e. day 3 in business days is not Saturday 1970-01-03, but Monday 1970-01-05). Building a ``datetime64`` dtype ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The proposed ways to specify the time unit in the dtype constructor are: Using the long string notation:: dtype('datetime64[us]') Using the short string notation:: dtype('T8[us]') Note that a time unit should always be specified, as there is not a default. Setting and getting values ~~~~~~~~~~~~~~~~~~~~~~~~~~ The objects with this dtype can be set in a series of ways:: t = numpy.ones(3, dtype='T8[s]') t[0] = 1199164176 # assign to July 30th, 2008 at 17:31:00 t[1] = datetime.datetime(2008, 7, 30, 17, 31, 01) # with datetime module t[2] = '2008-07-30T17:31:02' # with ISO 8601 And can be get in different ways too:: str(t[0]) --> 2008-07-30T17:31:00 repr(t[1]) --> datetime64(1199164177, 's') str(t[0].item()) --> 2008-07-30 17:31:00 # datetime module object repr(t[0].item()) --> datetime.datetime(2008, 7, 30, 17, 31) # idem str(t) --> [2008-07-30T17:31:00 2008-07-30T17:31:01 2008-07-30T17:31:02] repr(t) --> array([1199164176, 1199164177, 1199164178], dtype='datetime64[s]') Comparisons ~~~~~~~~~~~ The comparisons will be supported too:: numpy.array(['1980'], 'T8[Y]') == numpy.array(['1979'], 'T8[Y]') --> [False] or by applying broadcasting:: numpy.array(['1979', '1980'], 'T8[Y]') == numpy.datetime64 ('1980', 'Y') --> [False, True] The next should work too:: numpy.array(['1979', '1980'], 'T8[Y]') == '1980-01-01' --> [False, True] because the right hand expression can be broadcasted into an array of 2 elements of dtype 'T8[Y]'. Compatibility issues ~~~~~~~~~~~~~~~~~~~~ This will be fully compatible with the ``datetime`` class of the ``datetime`` module of Python only when using a time unit of microseconds. For other time units, the conversion process will loose precision or will overflow as needed. The conversion from/to a ``datetime`` object doesn't take leap seconds into account. ``timedelta64`` --------------- It represents a time that is relative (i.e. not absolute). It is implemented internally as an ``int64`` type. In time unit *conversions* and time *representations* (but not in other time computations), the value -2**63 (0x8000000000000000) is interpreted as an invalid or unknown time, *Not a Time* or *NaT*. See the section on time unit conversions for more information. Time units ~~~~~~~~~~ It accepts different time units, each of them implying a different time span. The table below describes the time units supported with their corresponding time spans. ======== ================ ========================== Time unit Time span ------------------------- -------------------------- Code Meaning ======== ================ ========================== Y year +- 9.2e18 years M month +- 7.6e17 years W week +- 1.7e17 years B business day +- 3.5e16 years D day +- 2.5e16 years h hour +- 1.0e15 years m minute +- 1.7e13 years s second +- 2.9e12 years ms millisecond +- 2.9e9 years us microsecond +- 2.9e6 years ns nanosecond +- 292 years ps picosecond +- 106 days fs femtosecond +- 2.6 hours as attosecond +- 9.2 seconds ======== ================ ========================== The value of a time delta is thus *an integer number of units of the chosen time unit*. Building a ``timedelta64`` dtype ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The proposed ways to specify the time unit in the dtype constructor are: Using the long string notation:: dtype('timedelta64[us]') Using the short string notation:: dtype('t8[us]') Note that a time unit should always be specified, as there is not a default. Setting and getting values ~~~~~~~~~~~~~~~~~~~~~~~~~~ The objects with this dtype can be set in a series of ways:: t = numpy.ones(3, dtype='t8[ms]') t[0] = 12 # assign to 12 ms t[1] = datetime.timedelta(0, 0, 13000) # 13 ms t[2] = '0:00:00.014' # 14 ms And can be get in different ways too:: str(t[0]) --> 0:00:00.012 repr(t[1]) --> timedelta64(13, 'ms') str(t[0].item()) --> 0:00:00.012000 # datetime module object repr(t[0].item()) --> datetime.timedelta(0, 0, 12000) # idem str(t) --> [0:00:00.012 0:00:00.014 0:00:00.014] repr(t) --> array([12, 13, 14], dtype="timedelta64[ms]") Comparisons ~~~~~~~~~~~ The comparisons will be supported too:: numpy.array([12, 13, 14], 't8[ms]') == numpy.array([12, 13, 13], 't8 [ms]') --> [True, True, False] or by applying broadcasting:: numpy.array([12, 13, 14], 't8[ms]') == numpy.timedelta64(13, 'ms') --> [False, True, False] The next should work too:: numpy.array([12, 13, 14], 't8[ms]') == '0:00:00.012' --> [True, False, False] because the right hand expression can be broadcasted into an array of 3 elements of dtype 't8[ms]'. Compatibility issues ~~~~~~~~~~~~~~~~~~~~ This will be fully compatible with the ``timedelta`` class of the ``datetime`` module of Python only when using a time unit of microseconds. For other units, the conversion process will loose precision or will overflow as needed. Examples of use =============== Here it is an example of use for the ``datetime64``:: In [5]: numpy.datetime64(42, 'us') Out[5]: datetime64(42, 'us') In [6]: print numpy.datetime64(42, 'us') 1970-01-01T00:00:00.000042 # representation in ISO 8601 format In [7]: print numpy.datetime64(367.7, 'D') # decimal part is lost 1971-01-02 # still ISO 8601 format In [8]: numpy.datetime('2008-07-18T12:23:18', 'm') # from ISO 8601 Out[8]: datetime64(20273063, 'm') In [9]: print numpy.datetime('2008-07-18T12:23:18', 'm') Out[9]: 2008-07-18T12:23 In [10]: t = numpy.zeros(5, dtype="datetime64[ms]") In [11]: t[0] = datetime.datetime.now() # setter in action In [12]: print t [2008-07-16T13:39:25.315 1970-01-01T00:00:00.000 1970-01-01T00:00:00.000 1970-01-01T00:00:00.000 1970-01-01T00:00:00.000] In [13]: repr(t) Out[13]: array([267859210457, 0, 0, 0, 0], dtype="datetime64[ms]") In [14]: t[0].item() # getter in action Out[14]: datetime.datetime(2008, 7, 16, 13, 39, 25, 315000) In [15]: print t.dtype dtype('datetime64[ms]') And here it goes an example of use for the ``timedelta64``:: In [5]: numpy.timedelta64(10, 'us') Out[5]: timedelta64(10, 'us') In [6]: print numpy.timedelta64(10, 'us') 0:00:00.000010 In [7]: print numpy.timedelta64(3600.2, 'm') # decimal part is lost 2 days, 12:00 In [8]: t1 = numpy.zeros(5, dtype="datetime64[ms]") In [9]: t2 = numpy.ones(5, dtype="datetime64[ms]") In [10]: t = t2 - t1 In [11]: t[0] = datetime.timedelta(0, 24) # setter in action In [12]: print t [0:00:24.000 0:00:01.000 0:00:01.000 0:00:01.000 0:00:01.000] In [13]: print repr(t) Out[13]: array([24000, 1, 1, 1, 1], dtype="timedelta64[ms]") In [14]: t[0].item() # getter in action Out[14]: datetime.timedelta(0, 24) In [15]: print t.dtype dtype('timedelta64[s]') Operating with date/time arrays =============================== ``datetime64`` vs ``datetime64`` -------------------------------- The only arithmetic operation allowed between absolute dates is the subtraction:: In [10]: numpy.ones(3, "T8[s]") - numpy.zeros(3, "T8[s]") Out[10]: array([1, 1, 1], dtype=timedelta64[s]) But not other operations:: In [11]: numpy.ones(3, "T8[s]") + numpy.zeros(3, "T8[s]") TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and 'numpy.ndarray' Comparisons between absolute dates are allowed. Casting rules ~~~~~~~~~~~~~ When operating (basically, only the subtraction will be allowed) two absolute times with different unit times, the outcome would be to raise an exception. This is because the ranges and time-spans of the different time units can be very different, and it is not clear at all what time unit will be preferred for the user. For example, this should be allowed:: >>> numpy.ones(3, dtype="T8[Y]") - numpy.zeros(3, dtype="T8[Y]") array([1, 1, 1], dtype="timedelta64[Y]") But the next should not:: >>> numpy.ones(3, dtype="T8[Y]") - numpy.zeros(3, dtype="T8[ns]") raise numpy.IncompatibleUnitError # what unit to choose? ``datetime64`` vs ``timedelta64`` --------------------------------- It will be possible to add and subtract relative times from absolute dates:: In [10]: numpy.zeros(5, "T8[Y]") + numpy.ones(5, "t8[Y]") Out[10]: array([1971, 1971, 1971, 1971, 1971], dtype=datetime64[Y]) In [11]: numpy.ones(5, "T8[Y]") - 2 * numpy.ones(5, "t8[Y]") Out[11]: array([1969, 1969, 1969, 1969, 1969], dtype=datetime64[Y]) But not other operations:: In [12]: numpy.ones(5, "T8[Y]") * numpy.ones(5, "t8[Y]") TypeError: unsupported operand type(s) for *: 'numpy.ndarray' and 'numpy.ndarray' Casting rules ~~~~~~~~~~~~~ In this case the absolute time should have priority for determining the time unit of the outcome. That would represent what the people wants to do most of the times. For example, this would allow to do:: >>> series = numpy.array(['1970-01-01', '1970-02-01', '1970-09-01'], dtype='datetime64[D]') >>> series2 = series + numpy.timedelta(1, 'Y') # Add 2 relative years >>> series2 array(['1972-01-01', '1972-02-01', '1972-09-01'], dtype='datetime64[D]') # the 'D'ay time unit has been chosen ``timedelta64`` vs ``timedelta64`` ---------------------------------- Finally, it will be possible to operate with relative times as if they were regular int64 dtypes *as long as* the result can be converted back into a ``timedelta64``:: In [10]: numpy.ones(3, 't8[us]') Out[10]: array([1, 1, 1], dtype="timedelta64[us]") In [11]: (numpy.ones(3, 't8[M]') + 2) ** 3 Out[11]: array([27, 27, 27], dtype="timedelta64[M]") But:: In [12]: numpy.ones(5, 't8') + 1j TypeError: the result cannot be converted into a ``timedelta64`` Casting rules ~~~~~~~~~~~~~ When combining two ``timedelta64`` dtypes with different time units the outcome will be the shorter of both ("keep the precision" rule). For example:: In [10]: numpy.ones(3, 't8[s]') + numpy.ones(3, 't8[m]') Out[10]: array([61, 61, 61], dtype="timedelta64[s]") However, due to the impossibility to know the exact duration of a relative year or a relative month, when these time units appear in one of the operands, the operation will not be allowed:: In [11]: numpy.ones(3, 't8[Y]') + numpy.ones(3, 't8[D]') raise numpy.IncompatibleUnitError # how to convert relative years to days? In order to being able to perform the above operation a new NumPy function, called ``change_timeunit`` is proposed. Its signature will be:: change_timeunit(time_object, new_unit, reference) where 'time_object' is the time object whose unit is to be changed, 'new_unit' is the desired new time unit, and 'reference' is an absolute date (NumPy datetime64 scalar) that will be used to allow the conversion of relative times in case of using time units with an uncertain number of smaller time units (relative years or months cannot be expressed in days). With this, the above operation can be done as follows:: In [10]: t_years = numpy.ones(3, 't8[Y]') In [11]: t_days = numpy.change_timeunit(t_years, 'D', '2001-01-01') In [12]: t_days + numpy.ones(3, 't8[D]') Out[12]: array([366, 366, 366], dtype="timedelta64[D]") dtype vs time units conversions =============================== For changing the date/time dtype of an existing array, we propose to use the ``.astype()`` method. This will be mainly useful for changing time units. For example, for absolute dates:: In[10]: t1 = numpy.zeros(5, dtype="datetime64[s]") In[11]: print t1 [1970-01-01T00:00:00 1970-01-01T00:00:00 1970-01-01T00:00:00 1970-01-01T00:00:00 1970-01-01T00:00:00] In[12]: print t1.astype('datetime64[D]') [1970-01-01 1970-01-01 1970-01-01 1970-01-01 1970-01-01] For relative times:: In[10]: t1 = numpy.ones(5, dtype="timedelta64[s]") In[11]: print t1 [1 1 1 1 1] In[12]: print t1.astype('timedelta64[ms]') [1000 1000 1000 1000 1000] Changing directly from/to relative to/from absolute dtypes will not be supported:: In[13]: numpy.zeros(5, dtype="datetime64[s]").astype('timedelta64') TypeError: data type cannot be converted to the desired type Business days have the peculiarity that they do not cover a continuous line of time (they have gaps at weekends). Thus, when converting from any ordinary time to business days, it can happen that the original time is not representable. In that case, the result of the conversion is *Not a Time* (*NaT*):: In[10]: t1 = numpy.arange(5, dtype="datetime64[D]") In[11]: print t1 [1970-01-01 1970-01-02 1970-01-03 1970-01-04 1970-01-05] In[12]: t2 = t1.astype("datetime64[B]") In[13]: print t2 # 1970 begins in a Thursday [1970-01-01 1970-01-02 NaT NaT 1970-01-05] When converting back to ordinary days, NaT values are left untouched (this happens in all time unit conversions):: In[14]: t3 = t2.astype("datetime64[D]") In[13]: print t3 [1970-01-01 1970-01-02 NaT NaT 1970-01-05] Final considerations ==================== Why the ``origin`` metadata disappeared --------------------------------------- During the discussion of the date/time dtypes in the NumPy list, the idea of having an ``origin`` metadata that complemented the definition of the absolute ``datetime64`` was initially found to be useful. However, after thinking more about this, we found that the combination of an absolute ``datetime64`` with a relative ``timedelta64`` does offer the same functionality while removing the need for the additional ``origin`` metadata. This is why we have removed it from this proposal. Operations with mixed time units -------------------------------- Whenever an operation between two time values of the same dtype with the same unit is accepted, the same operation with time values of different units should be possible (e.g. adding a time delta in seconds and one in microseconds), resulting in an adequate time unit. The exact semantics of this kind of operations is defined int the "Casting rules" subsections of the "Operating with date/time arrays" section. Due to the peculiarities of business days, it is most probable that operations mixing business days with other time units will not be allowed. Why there is not a ``quarter`` time unit? ----------------------------------------- This proposal tries to focus on the most common used set of time units to operate with, and the ``quarter`` can be considered more of a derived unit. Besides, the use of a ``quarter`` normally requires that it can start at whatever month of the year, and as we are not including support for a time ``origin`` metadata, this is not a viable venue here. Finally, if we were to add the ``quarter`` then people should expect to find a ``biweekly``, ``semester`` or ``biyearly`` just to put some examples of other derived units, and we find this a bit too overwhelming for this proposal purposes. .. [1] http://docs.python.org/lib/module-datetime.html .. [2] http://www.egenix.com/products/python/mxBase/mxDateTime .. [3] http://en.wikipedia.org/wiki/Unix_time .. Local Variables: .. mode: rst .. coding: utf-8 .. fill-column: 72 .. End:

1 0

Memmap and other read/write operations
by Thorsten Kranz July 30, 2008

July 30, 2008

Hi there, I have a question concerning numpy.memmap. I'm working with a binary format, consisting of a header of certain size (1024 byte) in the beginning and a 2d-float32 array afterwards. I would like to open the array-part using a memmap-object using mm = n.memmap("test.dat",dtype=n.float32,offset=1024,shape=(2000,32),mode="r+") This works fine. My question is now, if I can in the mean time securely open the file for custom writing by using f = open("test.dat", "r+") or will there be problems? Is there another possibility to do custom writing to the header-part? Thanks, Thorsten

1 0

Re: [Numpy-discussion] FFT usage / consistency
by Felix Richter July 30, 2008

July 30, 2008

Hi all, Stefan, thank you very much for your quick answer. This was an obvious silly mistake. Now the first function does what it should. Still, me and my colleagues can't make any sense of what happens in the second example. The re-transformed function is identical to the original one, but the Fourier-transform doesn't have anything to do with what it should be mathematically. Could you or someone else please have a look at it? Thanks again, Felix

6 10