Mailman 3 December 2007 - NumPy-Discussion

resize in old Numeric
by Christian Meesters Dec. 13, 2007

Dec. 13, 2007

Hi, For compatibility reasons (work with Biopython) I would like to use to Numeric in some code of mine. (Of course, I could make a little detour converting into numpy.array, but first I wonder whether somebody might know a solution for the problem with Numeric.) I need to flatten a simple 3x3 array and try to do that with resize. First time works, next time it doesn't. Here is an example: In [1]: from Numeric import * In [2]: a = array([[ 0.50622966, -0.54764389, -0.66619644], [ 0.44338279, -0.4973092, 0.7457228 ],[-0.73969131, -0.67288704, -0.00893311]]) In [3]: a.resize((9,)) Out[3]: array([ 0.50622966, -0.54764389, -0.66619644, 0.44338279, -0.4973092 , 0.7457228 , -0.73969131, -0.67288704, -0.00893311]) In [4]: a.resize((9,)) --------------------------------------------------------------------------- <type 'exceptions.ValueError'> Traceback (most recent call last) /home/cm/Documents/projects/SAXS/Eury/MC_rigid/<ipython console> in <module>() <type 'exceptions.ValueError'>: cannot resize an array that has been referenced or is referencing another array in this way. Use the resize function. Unfortunately, when applying resize on the actual array I have, I get a similar, yet slightly different error: Traceback (most recent call last): File "sym_get.py", line 37, in <module> s.inquire_relation() File "/usr/lib/python2.5/site-packages/mc_saxs/mc_rigid.py", line 915, in inquire_relation print rotation, rotation.resize((9,)) ValueError: resize only works on contiguous arrays The arrays 'a' and 'rotation' are identical. Any ideas, what's going on or how to solve this? Numeric version is: 24.2 TIA Christian

4 4

Re: [Numpy-discussion] numpy large arrays?
by Søren Dyrsting Dec. 13, 2007

Dec. 13, 2007

Thanks guys. Stefan: > I would guess that this is due to memory fragmentation. Timothy: > Your code is doing a lot of extra work and creating a lot of temporaries. > I'd clean it up before I did anything else. You suggestions turned out to be two sides of the same problem since the data amounts in the task is too close to the ceiling of memory. The approach: "First make it work. Then optimize" apparently isn't suitable here. Thanks /Søren

1 0

Re: [Numpy-discussion] [SciPy-user] maskedarray
by Travis E. Oliphant Dec. 13, 2007

Dec. 13, 2007

Pierre GM wrote: > On Tuesday 11 December 2007 15:48:01 Tom Johnson wrote: > >> What is the status regarding 'maskedarray'? When will this become >> part (replace ma) of the standard distribution? Also, what is the >> recommended way to import sandbox modules so that code changes are >> minimized when (if) the package becomes part of the standard >> distribution. >> > > > Mmh, no answer from our leaders ? Well, for now maskedarray lives in the scipy > sandbox, but it probably could be transferred to the numpy sandbox. There's > still no particular decision I'm aware of about putting it the official > release. More feedback is probably needed from users. > I apologize for not chiming in here. It has been difficult to keep up with all the lists on which I should ostensibly know what is being discussed as I have changed jobs and am settling in to new responsibilites. It seems to me that it is time to move the scipy.sandbox.maskedarray implementation over to numpy.ma. I'm a little concerned about doing this before 1.1, but if the API has not changed, then it should be fine. If there is no opposition, then it seems that this can move forward before 1.0.5 which I would like to see in January. For the future 1.1 release (late 2008), I would like to see (in no particular priority order): 1) The MA made a C-subclass 2) The matrix object made a C-subclass (for speed). 3) The sparse matrix structure (but not all the algorithms) brought over into NumPy with the indexing matching the matrix indexing. 4) Weave brought over into NumPy The additions should be created in such a way that they can be disabled should so that perhaps a numpy-lite could be produced as well. Or, perhaps better separate packages like numpy-devel, numpy-compat, numpy-sparse could be created. I'm looking forward to some good discussions over the weekend. -Travis O.

2 1

linalg.lapack_lite has undefined symbols
by Glen W. Mabey Dec. 12, 2007

Dec. 12, 2007

Hello, I am converting my build scripts to use the Intel Compiler Suite, using today's svn code, on RHEL 4.0. Building numpy succeeds, but importing it fails: ImportError: /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: dgesdd_ And sure enough: [gmabey@hydra Diamondback]$ ldd /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so libpython2.5.so.1.0 => /usr/local/lib/libpython2.5.so.1.0 (0x0000002a9592a000) libimf.so => /usr/local/stow/intel_cce-10.1.008/lib/libimf.so (0x0000002a95c4e000) libsvml.so => /usr/local/stow/intel_cce-10.1.008/lib/libsvml.so (0x0000002a95fb0000) libm.so.6 => /lib64/tls/libm.so.6 (0x0000002a96138000) libintlc.so.5 => /usr/local/stow/intel_cce-10.1.008/lib/libintlc.so.5 (0x0000002a962be000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000002a963f7000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a96505000) libdl.so.2 => /lib64/libdl.so.2 (0x0000002a9673a000) libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a9683d000) libutil.so.1 => /lib64/libutil.so.1 (0x0000002a96953000) /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000) [gmabey@hydra Diamondback]$ nm /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so | grep " U " | head U dgeev_ U dgelsd_ U dgeqrf_ U dgesdd_ U dorgqr_ U dsyevd_ U exit@@GLIBC_2.2.5 U floor U for_stop_core U for_write_seq_fmt I'm using ATLAS 3.8.0 also compiled with ICS, but the undefined symbol is in lapack_lite.so . Any suggestions on where to go from here? Thank you, Glen Mabey

1 0

numpy large arrays?
by Søren Dyrsting Dec. 12, 2007

Dec. 12, 2007

Hi all I need to perform computations involving large arrays. A lot of rows and no more than e.g. 34 columns. My first choice is python/numpy because I'm already used to code in matlab. However I'm experiencing memory problems even though there is still 500 MB available (2 GB total). I have cooked down my code to following meaningless code snip. This code share some of the same structure and calls as my real program and shows the same behaviour. ******************************************************** import numpy as N import scipy as S def stress(): x = S.randn(200000,80) for i in range(8): print "%(0)d" % {"0": i} s = N.dot(x.T, x) sd = N.array([s.diagonal()]) r = N.dot(N.ones((N.size(x,0),1),'d'), sd) x = x + r x = x / 1.01 ******************************************************** To different symptoms depending how big x are: 1) the program becomes extremely slow after a few iterations. 2) if the size of x is increased a little the program fails with the message "MemoryError" for example at line 'x = x + r', but different places in the code depending on the matrice size and which computer I'm testing. This might also occur after several iterations, not just during the first pass. I'm using Windows XP, ActivePython 2.5.1.1, NumPy 1.0.4, SciPy 0.6.0. - Is there an error under the hood in NumPy? - Am I balancing on the edge of the performance of Python/NumPy and should consider other environments. Fortran, C, BLAS, LAPACK e.t.c. - Am I misusing NumPy? Changing coding style will be a good workaround and even perform on larger datasets without errors? Thanks in advance /Søren

3 2

ndarray.clip only with lower or upper values?
by Hans Meine Dec. 11, 2007

Dec. 11, 2007

Hi again, I noticed that clip() needs two parameters, but wouldn't it be nice and straightforward to just pass min= or max= as keyword arg? In [2]: a = arange(10) In [3]: a.clip(min = 2, max = 5) Out[3]: array([2, 2, 2, 3, 4, 5, 5, 5, 5, 5]) In [4]: a.clip(min = 2) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/meine/<ipython console> TypeError: function takes at least 2 arguments (1 given) (I could simulate that by passing max = maximum_value_of(a.dtype), if that existed, see my other mail.) Ciao, / / /--/ / / ANS

3 5

Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
by Ray Schumacher Dec. 11, 2007

Dec. 11, 2007

At 02:32 AM 12/11/2007, you wrote: >If so I'd be happy to contribute part of the purchase price, >and I assume others would too. > >What's more, I *have* an old PIII at home. The main company I consult for is set to buy the Intel compiler and FFT lib for Windows, for the express purpose of compiling Python, numpy, and the fastest FFT for each CPU, for new products. I develop on both an old dual PIII and a new Core Duo, and there is also every other flavor in the shop. As I read it, the binaries should be distributable. The catch is, I have never compiled Python or numpy - I had seen a poster who offered help. When the company actually purchases the product I'd be glad to do it on 2-3 targets if someone can assist with the parameters. We have one consultant here who has done it on Linux. Ray Schumacher Congitive Vision 8580 Production Ave., Suite B San Diego, CA 92121 858.578.2778 http://cognitivevision.com/ -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.503 / Virus Database: 269.17.0/1180 - Release Date: 12/10/2007 2:51 PM

1 0

Minimum and maximum values of numpy datatypes?
by Hans Meine Dec. 11, 2007

Dec. 11, 2007

Hi! Is there a way to query the minimum and maximum values of the numpy datatypes? E.g. numpy.uint8.max == 255, numpy.uint8.min == 0 (these attributes exist, but they are functions, obviously for technical reasons). Ciao, / / /--/ / / ANS

3 3

"NaN"
by Jean-Luc Régnier Dec. 10, 2007

Dec. 10, 2007

Hello, I switched from numarray to numpy and I have now some "NaN" in my matrix. What that means ? None a numeric ? regards Jean-Luc REGNIER ACR Mimarlik Ltd. Sti Savas Cad. 26/B Sirinyali ANTALYA, TURKEY Tel. & Fax: 0090-(0).242.316.08.09 GSM: 0090-0.532.303.36.21 http://www.acrmim.com --------------------------------- Ne gardez plus qu'une seule adresse mail ! Copiez vos mails vers Yahoo! Mail

2 2

data transit
by Renato Serodio Dec. 10, 2007

Dec. 10, 2007

Hello all, I'm developing a custom computational application which I chose to base in Numpy. Already quite in love with Python, and with proprietary things making me increasingly sick (through forced exposure to stupid errors I can't correct), Numpy was the way to go. Now, it is in times like this that I regret not going for graduate studies in computation - I'm a bit locked in the old paradigms that my [fortran] generation learn. Since my application is only vaguely required to be 'generic', I had to dive into the wonderful world of computer science - a previous post in this group led to some very interesting solutions for the application, which, while doing nothing, is capable of doing everything :) A bit of context: the application is supposed to process telemetry, outputing some chart, alarm, etc. Raw data is obtained through plugin-like objects, which provide a uniform interface to distinct sources. The processing routines are objects as well, but operate on data as if they were functions (sort of sin(x)). This way, I don't need to define anything other than the interfaces - the core remains flexible. I came to a problem, though, while trying to define some structure for data transit. At first I imagined I could keep both raw data and results inside the same object; unfortunately, if I want to use these results in a second stage, my flexibility is rather impaired. Then I thought about getting raw data into an object, passing that to the processing core, and finally storing its output in another object. While this has the advantage of clearing raw data out of memory as soon as I finish chewing it, I seem to lose the relation between raw and result data sets - which I have to maintain somewhere else. Yet another issue crops up, in relation to very large data sets. If there's not enough memory to cope with the data set, one either relies on swapping or changes the algorithm - and in this case having 'inteligent' data objects allows good, textbook encapsulation. My question is thus, does anyone have experience or could point to literature/code where related problems are addressed? I understand that my application may be suffering from excessive 'generality', but certainly this problem has surfaced elsewhere. Looking forward to your answers, Renato

2 2