Mailman 3 July 2008 - NumPy-Discussion

Why are all fortran compilers looked for when --fcompiler=something is given ?
by David Cournapeau 29 Jul '08

29 Jul '08

Hi, While building numpy in wine, I got some errors in distutils when initializing the fortran compilers. I build numpy with: wine python setup.py build -c mingw32 --fcompiler=gnu95 And I got an error in load_all_fcompiler_classes when it tries to load the Compaq visual compiler. I don't think it has anything to do with wine, but rather that in python 2.6, running MSVCCompiler().initialize() can raise an IOError (python 2.6 uses a new method to lookd for msvc compilers, based on the existence of a bat file, which I don't have on my wine installation; I can check this on Windows, but that would be ackward since I would need to uninstall visual studio first...). I could catch the exception in the CompaqVisualCompiler, but I don't understand why this class is loaded at all. It also explains a warning I never quite understand before about "one should fix me in fcompiler/compaq.py" on windows whereas I have never used this compiler. Is this something we should "fix" ? Or just let alone to avoid breaking anything ? cheers, David

1 0

Numpy 1.1.0 (+ PIL 1.1.6) crashes on large datasets
by Christoph Rackwitz 29 Jul '08

29 Jul '08

Hi, I've managed to crash numpy+PIL when feeding it rather large images. Please see the URL for a test image, script, and gdb stack trace. This crashes on my box (Windows XP SP3) as well as on a linux box (the gdb trace I've been provided with) and a Mac. Windows reports the crash to happen in "multiarray.pyd"; the stack trace mentions the equivalent file. Unfortunately, I don't know how to fix this. Can I help somehow? -- Chris [1] http://cracki.ath.cx:10081/pub/numpy-pil-crash/

2 1

Operation over multiple axes? (Or: Partial flattening?)
by Hans Meine 29 Jul '08

29 Jul '08

Hi, with a multidimensional array (say, 4-dimensional), I often want to project this onto one single dimension, i.e.. let "dat" be a 4D array, I am interested in dat.sum(0).sum(0).sum(0) # equals dat.sum(2).sum(1).sum(0) However, creating intermediate results looks more expensive than necessary; I would actually like to say dat.sum((0,1,2)) One way to achieve this is partial flattening, which I did like this: dat.reshape((numpy.prod(dat.shape[:3]), dat.shape[3])).sum(0) Is there a more elegant way to do this? Ciao, / / .o. /--/ ..o / / ANS ooo

3 4

why isn't libfftw.a being accessed?
by Mathew Yeates 29 Jul '08

29 Jul '08

Hi In my site.cfg I have [DEFAULT] library_dirs = /home/ossetest/lib64:/home/ossetest/lib include_dirs = /home/ossetest/include [fftw] libraries = fftw3 but libfftw3.a isn't being accesed. ls -lu ~/lib/libfftw3.a -rw-r--r-- 1 ossetest ossetest 1572628 Jul 26 15:02 /home/ossetest/lib/libfftw3.a anybody know why? Mathew

2 1

Re: [Numpy-discussion] No Copy Reduce Operations
by Luis Pedro Coelho 29 Jul '08

29 Jul '08

Travis E. Oliphant wrote: > Your approach using C++ templates is interesting, and I'm very glad for > your explanation and your releasing of the code as open source. I'm > not prepared to start using C++ in NumPy, however, so your code will > have to serve as an example only. I will keep this as a separate package. I will write back one I have put it up somewhere. If this is not going into numpy, then I will do things a little differently, namely, I will have a very simple python layer which cleans up the arguments before calling the C++ implementation. bye, Luis

1 0

Recent work for branch cuts / C99 complex maths: problems on mingw
by David Cournapeau 29 Jul '08

29 Jul '08

Hi, I was away during the discussion on the updated complex functions using C99, and I've noticed it breaks some tests on windows (with mingw; I have not tried with Visual Studio, but it is likely to make things even worse given C support from MS compilers): http://scipy.org/scipy/numpy/ticket/865 Were those changes backported to 1.1.x ? If so, I would consider this as a release blocker, cheers, David

3 8

svn numpy selftests fail on Solaris
by Christopher Hanley 29 Jul '08

29 Jul '08

This has apparently been occurring for a few days. My apologizes but I have been away on vacation. FAILED (failures=5) Running unit tests for numpy NumPy version 1.2.0.dev5565 NumPy is installed in /usr/ra/pyssg/2.5.1/numpy Python version 2.5.1 (r251:54863, Jun 4 2008, 15:48:19) [C] nose version 0.10.0 ctypes is not available on this python: skipping the test (import error was: ctypes is not available.) No distutils available, skipping test. errors: failures: (Test(test_umath.TestC99.test_cacos(<ufunc 'arccos'>, (1.0, NaN), (NaN, NaN), 'invalid-optional')), 'Traceback (most recent call last):\n File "/usr/stsci/pyssgdev/2.5.1/nose/case.py", line 202, in runTest\n self.test(*self.arg)\n File "/usr/ra/pyssg/2.5.1/numpy/core/tests/test_umath.py", line 393, in _check\n assert got == expected, (got, expected)\nAssertionError: (\'(-NaN, -NaN)\', \'(NaN, NaN)\')\n') (Test(test_umath.TestC99.test_cacos(<ufunc 'arccos'>, (NaN, 1.0), (NaN, NaN), 'invalid-optional')), 'Traceback (most recent call last):\n File "/usr/stsci/pyssgdev/2.5.1/nose/case.py", line 202, in runTest\n self.test(*self.arg)\n File "/usr/ra/pyssg/2.5.1/numpy/core/tests/test_umath.py", line 393, in _check\n assert got == expected, (got, expected)\nAssertionError: (\'(-NaN, -NaN)\', \'(NaN, NaN)\')\n') (Test(test_umath.TestC99.test_cacosh(<ufunc 'arccosh'>, (1.0, NaN), (NaN, NaN), 'invalid-optional')), 'Traceback (most recent call last):\n File "/usr/stsci/pyssgdev/2.5.1/nose/case.py", line 202, in runTest\n self.test(*self.arg)\n File "/usr/ra/pyssg/2.5.1/numpy/core/tests/test_umath.py", line 393, in _check\n assert got == expected, (got, expected)\nAssertionError: (\'(NaN, -NaN)\', \'(NaN, NaN)\')\n') (Test(test_umath.TestC99.test_casinh(<ufunc 'arcsinh'>, (NaN, 1.0), (NaN, NaN), 'invalid-optional')), 'Traceback (most recent call last):\n File "/usr/stsci/pyssgdev/2.5.1/nose/case.py", line 202, in runTest\n self.test(*self.arg)\n File "/usr/ra/pyssg/2.5.1/numpy/core/tests/test_umath.py", line 393, in _check\n assert got == expected, (got, expected)\nAssertionError: (\'(NaN, -NaN)\', \'(NaN, NaN)\')\n') (Test(test_umath.TestC99.test_clog(<ufunc 'log'>, (-0.0, -0.0), (-Infinity, 3.1415926535897931), 'divide')), 'Traceback (most recent call last):\n File "/usr/stsci/pyssgdev/2.5.1/nose/case.py", line 202, in runTest\n self.test(*self.arg)\n File "/usr/ra/pyssg/2.5.1/numpy/core/tests/test_umath.py", line 393, in _check\n assert got == expected, (got, expected)\nAssertionError: (\'(-Infinity, 0.0)\', \'(-Infinity, 3.1415926535897931)\')\n') -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338

2 1

RFC: A (second) proposal for implementing some date/time types in NumPy
by Francesc Alted 29 Jul '08

29 Jul '08

Hi, After tons of excellent feedback received for our first proposal about the date/time types in NumPy Ivan and me have had another brainstorming session and ended with a new proposal for your consideration. While this one does not reap all and every of the suggestions you have made, we think that it does represent a fair balance between capabilities and simplicity and that it can be a solid and efficient basis for build-up more date/time niceties on top of it (read a full-fledged ``DateTime`` array class). Although the proposal is not complete, the essentials are there. So, please read on. We will be glad to hear your opinions. Thanks! -- Francesc Alted ==================================================================== A (second) proposal for implementing some date/time types in NumPy ==================================================================== :Author: Francesc Alted i Abad :Contact: faltet(a)pytables.com :Author: Ivan Vilata i Balaguer :Contact: ivan(a)selidor.net :Date: 2008-07-16 Executive summary ================= A date/time mark is something very handy to have in many fields where one has to deal with data sets. While Python has several modules that define a date/time type (like the integrated ``datetime`` [1]_ or ``mx.DateTime`` [2]_), NumPy has a lack of them. In this document, we are proposing the addition of a series of date/time types to fill this gap. The requirements for the proposed types are two-folded: 1) they have to be fast to operate with and 2) they have to be as compatible as possible with the existing ``datetime`` module that comes with Python. Types proposed ============== To start with, it is virtually impossible to come up with a single date/time type that fills the needs of every case of use. So, after pondering about different possibilities, we have stick with *two* different types, namely ``datetime64`` and ``timedelta64`` (these names are preliminary and can be changed), that can have different resolutions so as to cover different needs. **Important note:** the resolution is conceived here as a metadata that *complements* a date/time dtype, *without changing the base type*. Now it goes a detailed description of the proposed types. ``datetime64`` -------------- It represents a time that is absolute (i.e. not relative). It is implemented internally as an ``int64`` type. The internal epoch is POSIX epoch (see [3]_). Resolution ~~~~~~~~~~ It accepts different resolutions and for each of these resolutions, it will support different time spans. The table below describes the resolutions supported with its corresponding time spans. +----------------------+----------------------------------+ | Resolution | Time span (years) | +----------------------+----------------------------------+ | Code | Meaning | | +======================+==================================+ | Y | year | [9.2e18 BC, 9.2e18 AC] | | Q | quarter | [3.0e18 BC, 3.0e18 AC] | | M | month | [7.6e17 BC, 7.6e17 AC] | | W | week | [1.7e17 BC, 1.7e17 AC] | | d | day | [2.5e16 BC, 2.5e16 AC] | | h | hour | [1.0e15 BC, 1.0e15 AC] | | m | minute | [1.7e13 BC, 1.7e13 AC] | | s | second | [ 2.9e9 BC, 2.9e9 AC] | | ms | millisecond | [ 2.9e6 BC, 2.9e6 AC] | | us | microsecond | [290301 BC, 294241 AC] | | ns | nanosecond | [ 1678 AC, 2262 AC] | +----------------------+----------------------------------+ Building a ``datetime64`` dtype ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The proposed way to specify the resolution in the dtype constructor is: Using parameters in the constructor:: dtype('datetime64', res="us") # the default res. is microseconds Using the long string notation:: dtype('datetime64[us]') # equivalent to dtype('datetime64') Using the short string notation:: dtype('T8[us]') # equivalent to dtype('T8') Compatibility issues ~~~~~~~~~~~~~~~~~~~~ This will be fully compatible with the ``datetime`` class of the ``datetime`` module of Python only when using a resolution of microseconds. For other resolutions, the conversion process will loose precision or will overflow as needed. ``timedelta64`` --------------- It represents a time that is relative (i.e. not absolute). It is implemented internally as an ``int64`` type. Resolution ~~~~~~~~~~ It accepts different resolutions and for each of these resolutions, it will support different time spans. The table below describes the resolutions supported with its corresponding time spans. +----------------------+--------------------------+ | Resolution | Time span | +----------------------+--------------------------+ | Code | Meaning | | +======================+==========================+ | W | week | +- 1.7e17 years | | D | day | +- 2.5e16 years | | h | hour | +- 1.0e15 years | | m | minute | +- 1.7e13 years | | s | second | +- 2.9e12 years | | ms | millisecond | +- 2.9e9 years | | us | microsecond | +- 2.9e6 years | | ns | nanosecond | +- 292 years | | ps | picosecond | +- 106 days | | fs | femtosecond | +- 2.6 hours | | as | attosecond | +- 9.2 seconds | +----------------------+--------------------------+ Building a ``timedelta64`` dtype ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The proposed way to specify the resolution in the dtype constructor is: Using parameters in the constructor:: dtype('timedelta64', res="us") # the default res. is microseconds Using the long string notation:: dtype('timedelta64[us]') # equivalent to dtype('datetime64') Using the short string notation:: dtype('t8[us]') # equivalent to dtype('t8') Compatibility issues ~~~~~~~~~~~~~~~~~~~~ This will be fully compatible with the ``timedelta`` class of the ``datetime`` module of Python only when using a resolution of microseconds. For other resolutions, the conversion process will loose precision or will overflow as needed. Example of use ============== Here it is an example of use for the ``datetime64``:: In [10]: t = numpy.zeros(5, dtype="datetime64[ms]") In [11]: t[0] = datetime.datetime.now() # setter in action In [12]: t[0] Out[12]: '2008-07-16T13:39:25.315' # representation in ISO 8601 format In [13]: print t [2008-07-16T13:39:25.315 1970-01-01T00:00:00.0 1970-01-01T00:00:00.0 1970-01-01T00:00:00.0 1970-01-01T00:00:00.0] In [14]: t[0].item() # getter in action Out[14]: datetime.datetime(2008, 7, 16, 13, 39, 25, 315000) In [15]: print t.dtype datetime64[ms] And here it goes an example of use for the ``timedelta64``:: In [8]: t1 = numpy.zeros(5, dtype="datetime64[s]") In [9]: t2 = numpy.ones(5, dtype="datetime64[s]") In [10]: t = t2 - t1 In [11]: t[0] = 24 # setter in action (setting to 24 seconds) In [12]: t[0] Out[12]: 24 # representation as an int64 In [13]: print t [24 1 1 1 1] In [14]: t[0].item() # getter in action Out[14]: datetime.timedelta(0, 24) In [15]: print t.dtype timedelta64[s] Operating with date/time arrays =============================== ``datetime64`` vs ``datetime64`` -------------------------------- The only operation allowed between absolute dates is the subtraction:: In [10]: numpy.ones(5, "T8") - numpy.zeros(5, "T8") Out[10]: array([1, 1, 1, 1, 1], dtype=timedelta64[us]) But not other operations:: In [11]: numpy.ones(5, "T8") + numpy.zeros(5, "T8") TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and 'numpy.ndarray' ``datetime64`` vs ``timedelta64`` --------------------------------- It will be possible to add and subtract relative times from absolute dates:: In [10]: numpy.zeros(5, "T8[Y]") + numpy.ones(5, "t8[Y]") Out[10]: array([1971, 1971, 1971, 1971, 1971], dtype=datetime64[Y]) In [11]: numpy.ones(5, "T8[Y]") - 2 * numpy.ones(5, "t8[Y]") Out[11]: array([1969, 1969, 1969, 1969, 1969], dtype=datetime64[Y]) But not other operations:: In [12]: numpy.ones(5, "T8[Y]") * numpy.ones(5, "t8[Y]") TypeError: unsupported operand type(s) for *: 'numpy.ndarray' and 'numpy.ndarray' ``timedelta64`` vs anything --------------------------- Finally, it will be possible to operate with relative times as if they were regular int64 dtypes *as long as* the result can be converted back into a ``timedelta64``:: In [10]: numpy.ones(5, 't8') Out[10]: array([1, 1, 1, 1, 1], dtype=timedelta64[us]) In [11]: (numpy.ones(5, 't8[M]') + 2) ** 3 Out[11]: array([27, 27, 27, 27, 27], dtype=timedelta64[M]) But:: In [12]: numpy.ones(5, 't8') + 1j TypeError: The result cannot be converted into a ``timedelta64`` dtype/resolution conversions ============================ For changing the date/time dtype of an existing array, we propose to use the ``.astype()`` method. This will be mainly useful for changing resolutions. For example, for absolute dates:: In[10]: t1 = numpy.zeros(5, dtype="datetime64[s]") In[11]: print t1 [1970-01-01T00:00:00 1970-01-01T00:00:00 1970-01-01T00:00:00 1970-01-01T00:00:00 1970-01-01T00:00:00] In[12]: print t1.astype('datetime64[d]') [1970-01-01 1970-01-01 1970-01-01 1970-01-01 1970-01-01] For relative times:: In[10]: t1 = numpy.ones(5, dtype="timedelta64[s]") In[11]: print t1 [1 1 1 1 1] In[12]: print t1.astype('timedelta64[ms]') [1000 1000 1000 1000 1000] Changing directly from/to relative to/from absolute dtypes will not be supported:: In[13]: numpy.zeros(5, dtype="datetime64[s]").astype('timedelta64') TypeError: data type cannot be converted to the desired type Final considerations ==================== Why the ``origin`` metadata disappeared --------------------------------------- During the discussion of the date/time dtypes in the NumPy list, the idea of having an ``origin`` metadata that complemented the definition of the absolute ``datetime64`` was initially found to be useful. However, after thinking more about this, Ivan and me find that the combination of an absolute ``datetime64`` with a relative ``timedelta64`` does offer the same functionality while removing the need for the additional ``origin`` metadata. This is why we have removed it from this proposal. Resolution and dtype issues --------------------------- The date/time dtype's resolution metadata cannot be used in general as part of typical dtype usage. For example, in:: numpy.zeros(5, dtype=numpy.datetime64) we have to found yet a sensible way to pass the resolution. Perhaps the next would work:: numpy.zeros(5, dtype=numpy.datetime64(res='Y')) but we are not sure if this would collide with the spirit of the NumPy dtypes. At any rate, one can always do:: numpy.zeros(5, dtype=numpy.dtype('datetime64', res='Y')) BTW, prior to all of this, one should also elucidate whether:: numpy.dtype('datetime64', res='Y') or:: numpy.dtype('datetime64[Y]') numpy.dtype('T8[Y]') would be a consistent way to instantiate a dtype in NumPy. We do really think that could be a good way, but we would need to hear the opinion of the expert. Travis? .. [1] http://docs.python.org/lib/module-datetime.html .. [2] http://www.egenix.com/products/python/mxBase/mxDateTime .. [3] http://en.wikipedia.org/wiki/Unix_time .. Local Variables: .. mode: rst .. coding: utf-8 .. fill-column: 72 .. End:

6 12

asarray issue with type codes
by Damian Eads 29 Jul '08

29 Jul '08

Hi there, I ran into a little problem in some type checking code for a C extension I'm writing. I construct X as a C-long array and then I cast it to a C-int array Y, however the type code does not change. However, when I try constructing the array from scratch as a C-int, I get the right type code (ie 5). I assumed that when X gets casted to a C-int, no copying should occur but a new array view should be constructed with the C-int type code. What's wrong with this logic? Also note that casting from a C-long (type code 7) to a double to a C-int returns an array with the right type code, although a double copy occurs. Damian # Construct X as a C-long. In [16]: X=numpy.zeros((10,10),dtype='l') # Now cast X to a C-int. In [17]: Y=numpy.asarray(X, dtype='i') # Check X and Y's data type; they are the same. In [18]: X.dtype Out[18]: dtype('int32') In [19]: Y.dtype Out[19]: dtype('int32') # Their type codes are the same. In [20]: X.dtype.num Out[20]: 7 In [21]: Y.dtype.num Out[21]: 7 # Constructing with dtype='i', gives the right type code. In [22]: Z=numpy.zeros((10,10),dtype='i') In [23]: Z.dtype Out[23]: dtype('int32') In [24]: Z.dtype.num Out[24]: 5

2 1

Ashigabou Repository atlas vs yum blas/lapack
by jb 28 Jul '08

28 Jul '08

Hello: I'm hoping someone can straighten me out. I have a 64 bit fedora 8 quad core machine and can install blas and lapack from the yum repository. With these, numpy installs fine and finds blas and lapack. I also tried removing the yum blas/lapack libs and installing atlas via the instructions given on the scipy site for Ashigabou Repository. Atlas built fine from source and after installing the new rpm there were two files in the lib64/atlas/sse2 folder: libblas.so.3.0 and liblapack.so.3.0. However, when I try to install numpy, it cannot find any blas, lapack, or atlas, even though my site.cfg file has: [DEFAULT] library_dirs = /usr/lib64:/usr/lib64/atlas/sse2 [blas_opt] libraries = f77blas, cblas, atlas [lapack_opt] libraries = lapack, f77blas, cblas, atlas [atlas] library_dirs = /usr/lib64/atlas/sse2 atlas_libs = lapack, blas Using LD_LIBRARY_PATH=/usr/lib64/atlas/sse2 before installing numpy does not make a difference. My questions are, are the yum versions of lapack/blas just as good as the one built from Ashigabou source, and if not, why would numpy not be able to find the Ashigabou blas and lapack files (even though its looking in the right directory)? Thanks.

2 1