Mailman 3 October 2010 - NumPy-Discussion

Re: [Numpy-discussion] ndarray of object dtype
by Ioan Ferencik Oct. 6, 2010

Oct. 6, 2010

Thanks Robert, that worked just great. I was wandering whether anyone has experience with calling Fortran 90 functions that return derived types and cast them to C/Python objects. I have found excellent reference in a couple of websites, however all examples deal with returning at best 2D arrays of fundamental/basic types. When it comes to derived types/structures I have following dilemma: should I use Fortran subroutines, allocate memory in C and pass the structures as arguments or should I use functions, allocate memory in Fortran and return the data to C/Python. I should mention I have little experience in C/Fortran. All thoughts are welcome. Quoting Robert Kern <robert.kern(a)gmail.com>: > On Mon, Oct 4, 2010 at 00:41, Ioan Ferencik <ioan.ferencik(a)tkk.fi> wrote: >> Hello list, >> >> I am trying to pass elements of a custom defined with C API. >> I have successfully wrapped a type around a C struct. >> I intent to create a list with these objects and process it using >> numpy C API. >> So i create an array: >> >> array = (PyArrayObject *)PyArray_ContiguousFromObject(input, >> PyArray_OBJECT, 0, 0); >> >> to my understanding each el. in this array is a pointer to my type so >> a cast to this type >> should work. >> >> >> this is my custom type >> >> typedef struct val{ >> PyObject_HEAD >> float q; >> float wl; >> int cssid; >> int br; >> }hm1dval; >> >> I am passing only one element for testing purposes. >> >> in python >> a = [hm1d1.hm1dval() for i in range(0,1)] >> for c in a: >> c.set_values(q=3.0, wl=2.5, cssid=6, br=7) >> >> >> So following code should be valid in C: >> >> hm1dval s* = PyArray_DATA(array); >> >> but the members are 0 after casting in spite they were set previously. > > The elements of a dtype=object array are PyObject* pointers. In C, an > array of pointers would look like this: > > hm1dval **s = PyArray_DATA(array); > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Ioan Ferencik PhD student Aalto University School of Science and Technology Faculty Of Civil and Env. Engineering Lahti Center Tel: +358505122707

1 0

numpy installation
by P Sanjey Oct. 4, 2010

Oct. 4, 2010

I couldn't intially install numpy on python2.4.......installing then importing resulted in the following (please read below before answering thanks) >>> import numpy Traceback (most recent call last): File "<stdin>", line 1, in ? File "/cluster/lib/python2.4/site-packages/numpy/__init__.py", line 147, in ? import fft File "/cluster/lib/python2.4/site-packages/numpy/fft/__init__.py", line 4, in ? from fftpack import * File "/cluster/lib/python2.4/site-packages/numpy/fft/fftpack.py", line 39, in ? import fftpack_lite as fftpack ImportError: /cluster/lib/python2.4/site-packages/numpy/fft/fftpack_lite.so: undefined symbol: vmldSinCos2 The problem was sorted by modifying the following files..... site.cfg [DEFAULT] libraries = svml, mkl, vml library_dirs = /usr/local/lib, /opt/intel/cc/9.1/lib, /opt/intel/fc/9.1/lib include_dirs = /usr/local/include, /opt/intel/cc/9.1/include, /opt/intel/fc/9.1/include numpy-1.4.1/numpy/distutils/intelccompiler.py cc_exe = 'icc -O2 -g -fomit-frame-pointer -mcpu=pentium4 -march=pentium4 -msse2 -axWN -Wall -lsvml -L/opt/intel/cc/9.1/lib -I/opt/intel/cc/9.1/include' Now I want to install numpy on the server but have the same problem When I looked in the intel folder in the server there was only on directory-licences NO cc directory was present to add to the site.cfg file in the server Can I simply find them online and move them into intel directory? I don''t want to mess the server but don't really understand all this-if someone could help me it will be much appreciated

1 0

ndarray of object dtype
by Ioan Ferencik Oct. 4, 2010

Oct. 4, 2010

Hello list, I am trying to pass elements of a custom defined with C API. I have successfully wrapped a type around a C struct. I intent to create a list with these objects and process it using numpy C API. So i create an array: array = (PyArrayObject *)PyArray_ContiguousFromObject(input, PyArray_OBJECT, 0, 0); to my understanding each el. in this array is a pointer to my type so a cast to this type should work. this is my custom type typedef struct val{ PyObject_HEAD float q; float wl; int cssid; int br; }hm1dval; I am passing only one element for testing purposes. in python a = [hm1d1.hm1dval() for i in range(0,1)] for c in a: c.set_values(q=3.0, wl=2.5, cssid=6, br=7) So following code should be valid in C: hm1dval s* = PyArray_DATA(array); but the members are 0 after casting in spite they were set previously. also using strides it should be possible to traverse the array and get the individual objects. Regards. Ioan Ferencik PhD student Aalto University School of Science and Technology Faculty Of Civil and Env. Engineering Lahti Center Tel: +358505122707

2 1

can I mapping a np.darray class with a text file instead of reading the file in to mem?
by kee chen Oct. 4, 2010

Oct. 4, 2010

Dear All, I have memory problem in reading data from text file to a np.darray. It is because I have low mem on my pc and the data is too big. Te data is stored as 3 cols text and may have 10000000 records look like this 0.64984279 0.587856227 0.827348652 0.33463377 0.210916859 0.608797746 0.230265156 0.390278562 0.186308355 0.431187207 0.127007937 0.949673389 ... 10000000 LINES OMITTED HERE ... 0.150027782 0.800999655 0.551508963 0.255163742 0.785462049 0.015694154 After googled, I found 3 ways may solve this problem: 1.hardware upgrade(upgrade memory, upgrade arch to x64 ..... ) 2. filter the data before processing 3. using pytable However , I am trying to think another possibility - the mem-time trade-off. Can I design a class inherit from the np.darray then make it mapping with the text file? It may works in such a way, inside of this class only maintain a row object and total row ID a.k.a the rows of the file. the row mapping may look like this: an row object <--- bind---> row ID in text file <--- bind---> function row_eader() Wen np function be applied on this object, the actual date is from function row_eader(actual row ID). I have no idea how to code it then may I get support here to design such a class? Thanks! Rgs, KC

2 1

what is the best way to do a statistical mode operation?
by Gordon Wrigley Oct. 3, 2010

Oct. 3, 2010

I have an array of uint8's that has a shape of X*Y*Z*8, I would like to calculate modes along the 8 axis so that I end up with an array that has the shape X*Y*Z and is full of modes. I'm having problems finding a good way of doing this. My attempts at solving this using bincount or histogram produce an intermediate array that is 32x the size of my input data and somewhat larger than I have the memory to deal with. Can anyone suggest a good way to produce modes over sets of 8 bytes? Also in the instance where there are multiple modes for a particular set I'm happy for it to pick any one arbitrarily. G

3 4

NPZ format
by Francesc Alted Oct. 2, 2010

Oct. 2, 2010

Hi, I'm going to give a seminar about serialization, and I'd like to describe the .npy format. I noticed that there is a variant of it called .npz that can pack several arrays in one single file. However, .npz does not use compression at all and I'm wondering what's the reason. I suppose that this is because you don't want to loose the possibility to memmap saved arrays, but can someone confirm this? Thanks, -- Francesc Alted

3 6

Return values stats.mstats.linregress
by Gökhan Sever Oct. 1, 2010

Oct. 1, 2010

Hello, mstats.linregress returns 6 values. I don't see this documented from the function docstring. I know 0.91... is r. What is masked_array return here? I[29]: stats.mstats.linregress(np.ma.hstack(all_measured[0::6]), np.ma.hstack(all_predicted[0::6])) O[29]: (2.6309756058562122, -358.84572340482669, 0.91388013590912054, masked_array(data = 8.93881546632e-73, mask = False, fill_value = 1e+20) , 241.88030264140224, 8.2881035327089627) When I run with mstats: I[30]: stats.linregress(np.ma.hstack(all_measured[0::6]), np.ma.hstack(all_predicted[0::6])) O[30]: (2.5731434548927012, -325.98658013998079, 0.90619871728013224, 1.4284868970559915e-69, 0.089246202128717186) Any idea also why the returned p-value is so large from the mstats version. Shouldn't p-values lie in between [0,1) ? Thanks. -- Gökhan

1 0

[ANN] python-blosc 1.0.1, a wrapper for the Blosc compression library
by Francesc Alted Oct. 1, 2010

Oct. 1, 2010

==================================================== Announcing python-blosc 1.0.1 A Python wrapper for the Blosc compression library ==================================================== What is it? =========== Blosc (http://blosc.pytables.org) is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. python-blosc is a Python package that wraps it. What is new? ============ Everything. This is the first public version of the Python wrapper for Blosc (1.1.1). It supports Python 2.6, 2.7 and 3.1. The API is very simple and it loosely follows that of the zlib module. There are two basic functions, `compress()` and `decompress()`, as well as two additional calls specific for compressing NumPy arrays, namely `pack_array()` and `unpack_array`. There are also utilities for changing dynamically the number of threads used or to release resources when you are not going to need blosc for a while. Basic Usage =========== >>> import numpy as np >>> a = np.linspace(0, 100, 1e7) >>> bytes_array = a.tostring() >>> import blosc >>> bpacked = blosc.compress(bytes_array, typesize=8) >>> bytes_array2 = blosc.decompress(bpacked) >>> print(bytes_array == bytes_array2) True More examples are available on python-blosc wiki page: http://github.com/FrancescAlted/python-blosc/wiki Documentation ============= Please refer to docstrings. Start by the main package: >>> import blosc >>> help(blosc) and ask for more docstrings in the referenced functions. Download sources ================ Go to: http://github.com/FrancescAlted/python-blosc and download the most recent release from here. Blosc is distributed using the MIT license, see LICENSES/BLOSC.txt for details. Mailing list ============ There is an official mailing list for Blosc at: blosc(a)googlegroups.com http://groups.google.es/group/blosc ---- **Enjoy data!** -- Francesc Alted

1 0

constant shaded triangle mesh in mayavi2
by Geoffrey Irving Oct. 1, 2010

Oct. 1, 2010

Hello, I'm not sure where the correct place to ask questions about Mayavi, so feel free to redirect me elsewhere. I have a triangle mesh with a bunch of data on each face. The only color-relevant argument to triangular_mesh I know about is scalars, which is one value per vertex. Is there a way to set color per triangle, ideally with control over rgb separately so I can visualize three different fields at once? Thanks, Geoffrey

2 7

ndarray: subclassing a subclass looses custom attribute
by Sebastian Haase Oct. 1, 2010

Oct. 1, 2010

Hi, I'm trying to add a 'meta' attribute to ndarray to keep track of image data filenames and resolution etc. Following the excellent document http://docs.scipy.org/doc/numpy/user/basics.subclassing.html this worked right away. However, I had done this before for some specific image-file-types: those would add there own attribute to ndarray array (e.g. arr.Mrc) Now if I call the new ndarray_meta on my ndarray_with_mrc I loose the `Mrc` attribute, leaving only the new `meta` attribute. My code is essentially a verbatim copy of http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#simple-example… What can I do ? Thanks, Sebastian Haase

2 5