Mailman 3 February 2008 - NumPy-Discussion

[F2PY]: Allocatable Arrays
by Andrea Gavana 04 Feb '08

04 Feb '08

Hi All, I sent a couple of messages to f2py mailing list, but it seems like my problem has no simple solution so I thought to ask for some suggestions here. Basically, I read some huge unformatted binary files which contain time-step data from a reservoir simulation. I don't know the dimensions (i.e., lengths) of the vectors I am going to read, and I find out this information only when I start reading the file. So, I thought it would be nice to do something like: 1) Declare outputVector as allocatable; 2) Start reading the file; 3) Find the outputVector dimension and allocate it; 4) Read the data in the outputVector; 5) Return this outputVector. It works when I compile it and build it in Fortran as an executable (defining a "main" program in my f90 module), but it bombs when I try to use it from Python with the error: C:\Documents and Settings\gavana\Desktop\ECLIPSEReader>prova.py Traceback (most recent call last): File "C:\Documents and Settings\gavana\Desktop\ECLIPSEReader\prova.py", line 3, in <module> inteHead, propertyNames, propertyTypes, propertyNumbers = ECLIPSEReader.init.readinspec("OPT_INJ.INSPEC") ValueError: failed to create intent(cache|hide)|optional array-- must have defined dimensions but got (-1,) So, I have tried with a suggestion given in the f2py mailing list, and I found out that this routine works: MODULE DUMMY IMPLICIT NONE ! Ok, so I want an allocatable array as output real(8), allocatable :: realOutput(:) CONTAINS subroutine AllocateDummy(dummyInput) implicit none save ! dummyInput is *not* used, it's here just as ! an example integer, intent(in) :: dummyInput ! Allocate and build the output array allocate(realOutput(10)) realOutput(1:10) = 0.0 realOutput(3) = 3.0 realOutput(7) = 7.0 deallocate(realOutput) return end subroutine AllocateDummy END MODULE DUMMY But this one doesn't work: MODULE DUMMY IMPLICIT NONE ! Ok, so I want an allocatable array as output real(8), allocatable :: realOutput(:) integer, allocatable :: inteOutput(:) CONTAINS subroutine AllocateDummy(dummyInput) implicit none save ! dummyInput is *not* used, it's here just as ! an example integer, intent(in) :: dummyInput ! Allocate and build the output array allocate(realOutput(10)) allocate(inteOutput(20)) realOutput(1:10) = 0.0 realOutput(3) = 3.0 realOutput(7) = 7.0 inteOutput(10) = 2 deallocate(realOutput) deallocate(inteOutput) return end subroutine AllocateDummy END MODULE DUMMY The difference between the 2 scripts, is just that in the second one I want 2 allocatable arrays instead of 1. When I compile it with f2py, I get this warning from getarrdims: Building modules... Building module "dummy"... Constructing F90 module support for "dummy"... Variables: realoutput inteoutput getarrdims:warning: assumed shape array, using 0 instead of ':' getarrdims:warning: assumed shape array, using 0 instead of ':' Constructing wrapper function "dummy.allocatedummy"... allocatedummy(dummyinput) Which is not present if I compile script number 1. Actually, if I run script 2, I can't access anymore the 2 variables realoutput and inteoutput (they are not there), while with script 1 I can easily access realoutput by writing dummy.dummy.realoutput. I can't actually see any big difference between the 2 scripts... am I missing something? This is Windows XP, Python 2.5, numpy 1.0.3.1, Compaq Visual Fortran 6.6, MS Visual Studio .NET 2003. Thank you for all your suggestions, I am at loss. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/

4 9

numpy.asarray( iterator )
by Sebastian Haase 04 Feb '08

04 Feb '08

Hi, Can this be changed: If I have a list L the usual N.asarray( L ) works well -- however I just discovered that N.asarray( reversed( L ) ) breaks my code.... Apparently reversed( L ) returns an iterator object, and N.asarray( reversed( L ) ) (called arrY in my function) results in: (Pdb) p arrY array(<listreverseiterator object at 0x1a508a90>, dtype=object) (Pdb) p arrY.shape () Comments ? How about letting asarray call fromiter when it sees that the argument is a iterator !? Thanks, Sebastian Haase

2 1

Re: [Numpy-discussion] histogramdd memory needs
by Lars Friedrich 04 Feb '08

04 Feb '08

Hi, > 2) Is there a way to use another algorithm (at the cost of performance) >> > that uses less memory during calculation so that I can generate bigger >> > histograms? > > > You could work through your array block by block. Simply fix the range and > generate an histogram for each slice of 100k data and sum them up at the > end. Thank you for your answer. I sliced the (original) data into blocks. However, when I do this, I need at least twice the memory for the whole histogram (one for the temporary result and one for accumulating the total result). Assuming my histogram has a size of (280**3)*8 = 176 (megabytes) this does not help, I think. What I will try next is to compute smaller parts of the big histogram and combine them at the end. (Slice the histogram into blocks) Is it this, that you were recommending? Lars

2 1

Unexpected behavior with numpy array
by Damian Eads 04 Feb '08

04 Feb '08

Good day, Reversing a 1-dimensional array in numpy is simple, A = A[:,:,-1] . However A is a new array referring to the old one and is no longer contiguous. While trying to reverse an array in place and keep it contiguous, I encountered some weird behavior. The reason for keeping it contiguous is the array must be passed to an old C function I have, which expects the buffer to be in row major order and contiguous. I am using lots of memory so I want to minimize copying and allocation of new arrays. >>> A=numpy.arange(0,10) >>> A array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> A[::-1] array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0]) >>> A[:] = A[::-1] >>> A array([9, 8, 7, 6, 5, 5, 6, 7, 8, 9]) >>> Is there any way to perform assignments of the following form X[sliceI] = expression involving X with dimensions that are compatible with the left hand side without causing the odd behavior I mentioned. If not, it might be helpful to throw an exception when both the LHS and the RHS of an assignment reference an array slice of the same variable? On similar note, does the assignment A = A * B create a new array with a new buffer to hold the result of A * B, and assign A to refer to the new array? Thanks very much! Damian

5 9

[ANN] Blas-Lapack superpack: click only blas/lapack installation (first alpha)
by David Cournapeau 02 Feb '08

02 Feb '08

Hi, I started working on an easy installer for blas/lapack. The idea is that you would use this installer so that building numpy and scipy from source is easy on windows (32 bits for now). It would give blas/lapack compiled correctly, with optional atlas optimized version. http://www.ar.media.kyoto-u.ac.jp/members/david/archives/blas_lapack_superp… How to use ========== Run the setup.exe, click yes all the way. Add the installed dll in your path, or add the path where the dll are installed in your PATH. main features: ============== - Click only, easy installation of blas and lapack libraries (including atlas if supported, see below). - Install atlas *only if your cpu supports it*: that's the main feature, actually. The installer detects your cpu, and install ATLAS only if an ATLAS matching your CPU is found (only SSE3 supported for now, but other arch, included 3dnow and co, can easily be added depending on people help to provide the built ATLAS). What can you do with it: ======================== - compile numpy and scipy without wrong SSE problem, without bothering about compiling netlib BLAS/LAPACK, etc... - compile numpy wo any fortran compiler, VS only (no need for mingw, etc... thanks to VS import libraries + DLL). - use the installed lapack to build an optimized ATLAS for your architecture (using both gnu compilers and proprietary compilers should be possible). More details ============ - built with mingw g77 from linux (dll, unix-style static archives and def) - import libraries built with VS 2003. This means you can compile numpy wo any fortran compiler, in particular, no need for mingw. I don't know if this is compatible with other versions of VS, though. - Only SSE3 and above will get ATLAS for now. This is because compiling ATLAS on windows is a PITA, and I don't want to spend time on this, so if you want something else, you will have to provide me the atlas binary first. But having atlas for sse, sse2, 3dnow, etc... is entirely possible. - I do not register the DLL yet, because I am not sure yet how to do it in a safe way (thanks MS for a totally broken handling of shared libraries, BTW) - I do not guarantee that the built atlas is optimal. ATLAS performances depend on many parameters, not just sse/sse2/sse3 (size of L1/L2/L3 cache are significant, for example), and again, I cannot build many different libraries. - The installer is built using nsis. - The whole process of making the installer is not 100 % automatic yet, but I intend to make it so, and put the necessary scripts somewhere so people can improve it if they want. This is alpha software, and because it is an installer, it can screw up your computer if I did something wrong. However, I barely touch the registry, and only install files in one directory, so the chances are pretty minimal (that's why also you have to put dll manually in a path where they will be found manually: at some point, this will be done by the installer, but that's by far the most dangerous thing, so I prefer avoiding it for now). cheers, David

1 0

Call to mrecords users
by Pierre GM 01 Feb '08

01 Feb '08

All, I just commited some updates on mrecords (numpy.maskedarray branch). Overall, it's a reorganization/simplification of the code. Regular masked arrays can already recognize named fields, but the mask works only at the record level (ie, all the fields of one record are masked). In comparison, masked record arrays (mrecarrays) permit the masking of individual fields. The conversion between a masked array of records and a mrecarray is as easy as for the classical ndarray: just use a .view(mrecarray). I'd be grateful if the current users of mrecords could give the new version a try, and let me know whether everything works seamlessly or if new modifications need to be implemented. Thanks a lot in advance. PS: that goes as well for users of trecords (in scikits.timeseries): the package has been updated to match the recent modifications on mrecords

1 0

Can not update a submatrix
by Nadav Horesh 01 Feb '08

01 Feb '08

In the following piece of code: >>> import numpy as N >>> R = N.arange(9).reshape(3,3) >>> ax = [1,2] >>> R array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >>> R[ax,:][:,ax] = 100 >>> R array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) Why R is not updated? I was expecting: >>> R array([[0, 1, 2], [3, 100, 100], [6, 100, 100]]) I am using numpy 1.04 on gentoo-linux/amd64 Nadav

7 12

histogramdd memory needs
by Lars Friedrich 01 Feb '08

01 Feb '08

Hello, I use numpy.histogramdd to compute three dimensional histograms with a total number of bins in the order of 1e7. It is clear to me, that such a histogram will take a lot of memory. For a dtype=N.float64, it will take roughly 80 megabytes. However, I have the feeling that during the histogram calculation, much more memory is needed. For example, when I have data.shape = (8e6, 3) and do a numpy.histogramdd(d, 280), I expect a histogram size of (280**3)*8 = 176 megabytes, but during histogram calculation the memory need of pythonw.exe in the Windows Task Manager increases up to 687 megabytes over the level before histogram calculation. When the calculation is done, the mem usage drops down to the expected value. I assume this is due to the internal way, numpy.histogramdd works. However, when I need to calculate even bigger histograms, I cannot do it this way. So I have the following questions: 1) How can I tell histogramdd to use another dtype than float64? My bins will be very little populated so an int16 should be sufficient. Without normalization, a Integer dtype makes more sense to me. 2) Is there a way to use another algorithm (at the cost of performance) that uses less memory during calculation so that I can generate bigger histograms? My numpy version is '1.0.4.dev3937' Thanks, Lars -- Dipl.-Ing. Lars Friedrich Photonic Measurement Technology Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-Köhler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: lars.friedrich(a)imtek.de

2 1

searchsorted bug
by James Philbin 01 Feb '08

01 Feb '08

Hi, The following gives the wrong answer: In [2]: A = array(['a','aa','b']) In [3]: B = array(['d','e']) In [4]: A.searchsorted(B) Out[4]: array([3, 0]) The answer should be [3,3]. I've come across this while trying to come up with an ismember function which works for strings (setmember1d doesn't seems to assume numerical arrays). Thanks, James

11 26