Mailman 3 December 2010 - NumPy-Discussion

fromrecords yields "ValueError: invalid itemsize in generic type tuple"
by Rajat Banerjee 29 Dec '10

29 Dec '10

Hi All, I have been using Numpy for a while with great success. I left my little project for a little while (http://web.mit.edu/stardev/cluster/) and now some of my code is broken. I have some Numpy code to create graphs of activity on a cluster with matplotlib. It ran just fine in July / August 2010, but has since stopped working. I have updated numpy on my machine, I think. In [2]: np.version.version Out[2]: '1.5.1' My call to np.rec.fromrecords() is throwing this exception: File "/home/rajat/Envs/StarCluster/lib/python2.6/site-packages/numpy/core/records.py", line 607, in fromrecords descr = sb.dtype((record, dtype)) ValueError: invalid itemsize in generic type tuple Here is the code with some irrelevant stuff stripped: for line in file: a = [datetime.strptime(parts[0], '%Y-%m-%d %H:%M:%S.%f'), int(parts[1]), int(parts[2]), int(parts[3]), int(parts[4]), int(parts[5]), int(parts[6]), float(parts[7])] list.append(a) file.close() names = ['dt', 'hosts', 'running_jobs', 'queued_jobs',\ 'slots', 'avg_duration', 'avg_wait', 'avg_load'] descriptor = {'names': ('dt,hosts,running_jobs,queued_jobs,slots,avg_duration,avg_wait,avg_load'),\ 'formats' : ('S20','u','u','u','u','u','u','f')} self.records = np.rec.fromrecords(list,','.join(names)) #used to work #self.records = np.rec.fromrecords(list, dtype=descriptor) #new attempt Here is one "line" from the array "list": >>> parts (8) = ['2010-12-07 03:09:46.855712', '2', '2', '177', '2', '86', '370', '1.05']. Neither of those np.rec.fromrecords() calls works. I've tried both separately. They both throw the exact same exception, ValueError: invalid itemsize in generic type tuple Can anybody help me? Am I doing something dumb? Thank you. Rajat Banerjee Masters Candidate, Computer Science, Harvard University

2 1

Strange problem with h5py and numpy
by Johannes Korn 28 Dec '10

28 Dec '10

Hi, I have a strange problem with h5py or with numpy. I try to read a bunch of hdf files in a loop. The problem is that I get an error at the second file because the file handle is of type <Closed HDF5 file> It seems the file is opened and instantaneous closed again. Meanwhile I found the root of the evil: First I read the contents of a dataset to the numpy array tmp. In the next step I paste this array into a bigger one "alb_c1_tmp". The shape of tmp is (651, 1701). Everything works well if I omit the pasting. alb_c1_tmp = zeros([3712,3712]) c1_eu = File(filename_ch1_eu,mode='r') print c1_eu tmp = c1_eu['AL-SP-BH'][:] #Source of the evil alb_c1_tmp[ 3012:3663, 462:2163 ] = tmp c1_eu.close() The code as above crashes always when the second file is read. However originally I had the boundaries of alb_c1_tmp set dynamically from attributes of the hdf files (they are identical for all files), this code crashed after a random number of files. Here´s the line that caused the randomly delayed crashes: alb_c1_tmp[ 1855 + c1_eu.attrs['LOFF'] - c1_eu.attrs['NL'] : 1855 + c1_eu.attrs['LOFF'] , 1855 + c1_eu.attrs['COFF'] - c1_eu.attrs['NC'] : 1855 + c1_eu.attrs['COFF'] ] = tmp Is it a bug or did I something wrong? Python 2.6.5 (r265:79063, Oct 28 2010, 20:56:56) Numpy-Version is probably 1.5.0 h5py 1.3.0 Happens under 32bit as well as under 64bit SuSE-release 11.3 Kind regards! Johannes

2 3

How to call import_array() properly?
by Bruce Sherwood 27 Dec '10

27 Dec '10

In my Python code I have import cvisual cvisual.init_numpy() and in my C++ code I have void init_numpy() { import_array(); } import_array() in numpy/core/include/numpy/_multiarray_api.h is a macro: #if PY_VERSION_HEX >= 0x03000000 #define NUMPY_IMPORT_ARRAY_RETVAL NULL #else #define NUMPY_IMPORT_ARRAY_RETVAL #endif #define import_array() {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import"); return NUMPY_IMPORT_ARRAY_RETVAL; } } Note that for Python 3 there is a change so that the macro returns NULL, whereas for Python 2 it returned nothing. On Windows and Mac, this works fine for Python 2, and it works fine for a with Python 3, but with either Microsoft Visual Studio 2008 or 2010 this fails for Python 3 with the message "'void' function returning a value", presumably due to the return NULL for Python 3, something that doesn't bother the Mac. So my dumb question is, how should I call import_array() from my routine init_numpy() to get around this problem? I have found a workaround, consisting of defining init_numpy to be of type int on Windows with Python 3, but this seems like an odd kludge, since it isn't needed on the Mac, and I think it's also not needed on Linux. Bruce Sherwood

2 5

Installing on CentOS 5 claims invalid Python installation
by Roger Martin 27 Dec '10

27 Dec '10

Hi, NumPy looks like the way to get computation done in Python. Now I'm going through the learning curve of installing the module into different linux OS's and Python versions. An extra need is to install google code's h5py http://code.google.com/p/h5py/ which depends on numpy. In trying a number of Python versions the 2.x's are yielding the message " invalid Python installation" --------------- raise DistutilsPlatformError(my_msg) distutils.errors.DistutilsPlatformError: invalid Python installation: unable to open /home/roger/Python-2.6.6/dist/lib/python2.6/config/Makefile (No such file or directory) --------------- From reading on the web it appears a Python-2.x.x-devel version is needed. Yet no search combination comes back with where to get such a thing(note: I need user installs/builds for security reasons). Where are Python versions compatible with numpy? Building Python-2.6.6 Python-2.7.1(fails to build) Python3.2beta2 numpy1.5.1 invalid Python installation NA success h5py1.3.1 needs numpy NA fails To start I need just one successful combination but will need more cases depending on users of a new integration project. Interestingly your numpy 1.5.1's setup is in good shape to build with Python3.2 yet I need to allow older versions for people's systems not ready to upgrade that far.

3 3

How construct custom slice
by Mario Moura 27 Dec '10

27 Dec '10

Hi Folks a = np.zeros((4,3,5,55,5),dtype='|S8') myLen = 4 # here I use myLen = len(something) li = [3,2,4] # li from a list.append(something) sl = slice(0,myLen) tmpIndex = tuple(li) + sl + 4 # <== Here my problem a[tmpIndex] # So What I want is: fillMe = np.array(['foo','bar','hello','world']) # But I cant contruct by hand like this a[3,2,4,:4,4] = fillMe a Again. I need construct custom slice from here tmpIndex = tuple(li) + sl + 4 a[tmpIndex] Who can help me? Best Regards Mario Moura

2 2

ANN: carray 0.3 released
by Francesc Alted 24 Dec '10

24 Dec '10

===================== Announcing carray 0.3 ===================== What's new ========== A lot of stuff. The most outstanding feature in this version is the introduction of a `ctable` object. A `ctable` is similar to a structured array in NumPy, but instead of storing the data row-wise, it uses a column-wise arrangement. This allows for much better performance for very wide tables, which is one of the scenarios where a `ctable` makes more sense. Of course, as `ctable` is based on `carray` objects, it inherits all its niceties (like on-the-flight compression and fast iterators). Also, the `carray` object itself has received many improvements, like new constructors (arange(), fromiter(), zeros(), ones(), fill()), iterators (where(), wheretrue()) or resize mehtods (resize(), trim()). Most of these also work with the new `ctable`. Besides, Numexpr is supported now (but it is optional) in order to carry out stunningly fast queries on `ctable` objects. For example, doing a query on a table with one million rows and one thousand columns can be up to 2x faster than using a plain structured array, and up to 20x faster than using SQLite (using the ":memory:" backend and indexing). See 'bench/ctable-query.py' for details. Finally, binaries for Windows (both 32-bit and 64-bit) are provided. For more detailed info, see the release notes in: https://github.com/FrancescAlted/carray/wiki/Release-0.3 What it is ========== carray is a container for numerical data that can be compressed in-memory. The compression process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. Having data compressed in-memory can reduce the stress of the memory subsystem. The net result is that carray operations may be faster than using a traditional ndarray object from NumPy. carray also supports fully 64-bit addressing (both in UNIX and Windows). Below, a carray with 1 trillion of rows has been created (7.3 TB total), filled with zeros, modified some positions, and finally, summed-up:: >>> %time b = ca.zeros(1e12) CPU times: user 54.76 s, sys: 0.03 s, total: 54.79 s Wall time: 55.23 s >>> %time b[[1, 1e9, 1e10, 1e11, 1e12-1]] = (1,2,3,4,5) CPU times: user 2.08 s, sys: 0.00 s, total: 2.08 s Wall time: 2.09 s >>> b carray((1000000000000,), float64) nbytes: 7450.58 GB; cbytes: 2.27 GB; ratio: 3275.35 cparams := cparams(clevel=5, shuffle=True) [0.0, 1.0, 0.0, ..., 0.0, 0.0, 5.0] >>> %time b.sum() CPU times: user 10.08 s, sys: 0.00 s, total: 10.08 s Wall time: 10.15 s 15.0 ['%time' is a magic function provided by the IPyhton shell] Please note that the example above is provided for demonstration purposes only. Do not try to run this at home unless you have more than 3 GB of RAM available, or you will get into trouble. Resources ========= Visit the main carray site repository at: http://github.com/FrancescAlted/carray You can download a source package from: http://carray.pytables.org/download Manual: http://carray.pytables.org/manual Home of Blosc compressor: http://blosc.pytables.org User's mail list: carray(a)googlegroups.com http://groups.google.com/group/carray Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- Enjoy! -- Francesc Alted

2 2

numpy for jython
by Thunemann, Paul Z 24 Dec '10

24 Dec '10

I'd be very interested in hearing more about a numpy port to Java and Jython. If anyone has more info about how to get involved please let me know. -Zack

5 4

Re: [Numpy-discussion] numpy for jython
by Thunemann, Paul Z 23 Dec '10

23 Dec '10

If the refactor separates numpy from the Cpython objects and results in a clean C or C++ api, then porting to Java is still a chore but it's doable. I've used JNI and SWIG extensively to port math libraries and could get involved but I don't know who else might be working on this (if anyone). -Zack

1 0

How to control column for pretty print line wrap of ndarrays
by Ian Stokes-Rees 22 Dec '10

22 Dec '10

Like most people these days, I have multiple 24" monitors. I don't need "print" of ndarrays to wrap after 72 columns. Is there some way to change this? TIA Ian CURRENT: [ NaN NaN NaN NaN NaN 5.1882094 1.19646584]] DESIRED: [ NaN NaN NaN NaN NaN 5.1882094 1.19646584]] (Although your mail client mail mangle it...)

2 1

counting non-zero entries in an ndarray
by Ian Stokes-Rees 22 Dec '10

22 Dec '10

What is the most efficient way to do the Matlab equivalent of nnz(M) (nnz = number-of-non-zeros function)? I've tried Google, but no luck. My assumption is that something like a != 0 will be used, but I'm not sure then how to "count" the number of "True" entries. TIA. Ian

4 4