Mailman 3 May 2010 - NumPy-Discussion

saving object array to ascii file
by Maria Liukis May 7, 2010

May 7, 2010

Hello Everybody, Sorry if it's a trivial question. I'm trying to find out if there is a way to save object array to ascii file. numpy.savetxt() in NumPy V1.3.0 doesn't seem to work: >>> import numpy as np >>> obj_arr = np.zeros((2,), dtype=np.object) >>> obj_arr[0] = np.array([[1,2,3], [4,5,6], [7,8,9]]) >>> obj_arr[1] = np.array([[10,11], [12,13]]) >>> obj_arr array([[[1 2 3] [4 5 6] [7 8 9]], [[10 11] [12 13]]], dtype=object) >>> np.savetxt('obj_array.dat', obj_arr) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/lib/io.py", line 636, in savetxt fh.write(format % tuple(row) + '\n') TypeError: float argument required >>> scipy.io.savemat() supports Matlab format of the object array, but I could not find any documentation on ASCII file format for object arrays. Thanks in advance, Masha -------------------- liukis(a)usc.edu

2 1

Decision tree-like algorithm on numpy arrays
by Martin Raspaud May 7, 2010

May 7, 2010

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, I have an old c-extension I want to remove from my code to the benefit of numpy, but it looks kind of tricky to me. Here is the thing: I have a number of arrays of the same shape. On these arrays, I run a sequence of tests, leading to a kind of decision tree. In the end, based on these tests, I get a number of result arrays where, based on the tests, each element gets a value. The way to do this in an efficient way with numpy is quite unclear to me. My first thought would be: result_array1 = np.where(some_test_on(array1), np.where(some_test_on(array2), 1, 2), np.where(some_test_on(array3, array4), np.where(some_test_on(array5), 3, 4), 4)) result_array2 = np.where(some_test_on(array1), np.where(some_test_on(array2), True, True), np.where(some_test_on(array3, array4), np.where(some_test_on(array5), True, False), True)) etc... but that means running the same tests several times, which is not acceptable if the tests are lengthy. In order to avoid this problem I could also have some mask based on each test: mask1 = some_test_on(array1) mask2 = some_test_on(array2[mask1]) mask3 = some_test_on(array3[!mask1], array4[!mask1]) mask4 = some_test_on(array5[!mask1][mask3]) result_array1[mask1][mask2] = 1 result_array1[mask1][!mask2] = 2 result_array1[!mask1][mask3][mask4] = 3 result_array1[!mask1][mask3][!mask4] = 4 result_array1[!mask1][!mask3] = 4 result_array2[mask1][mask2] = True result_array2[mask1][!mask2] = True result_array2[!mask1][mask3][mask4] = True result_array2[!mask1][mask3][!mask4] = False result_array2[!mask1][!mask3] = True etc... but that looks a bit clumsy to me... The way it was done in the C-extension was to run the decision tree on each element sequentially, but I have the feeling that would not be very efficient with numpy (although I know I can't beat pure C code, I would like to have comparable times). Does any of you wise people have an opinion on this ? Thanks, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJL4ma5AAoJEBdvyODiyJI4SpgH/i0bb7PH8oTu481NRuYmbi40 VwJrOCdfSo6CauLBiIdxBZV2Hksbu2iDu5GEKJNUObf9bM7N+LK+qMwaBq1M5hF+ 47yNczSEUaxshBHzUFQMlS9XEtZewhYZGepkH1oThIQbSD2IbM6fWkVj+EJRwwJ5 2Ia4p1GIdLGMZ3loaWevvCmz8kjppX7Feei0hEP28+HIiWq/qmUlccYZm/ThZcFE 6ROEKtkepKsf3vOfpuS5Hr6U1Hb4mo7u9SmUcOvlCby6q/TbVtwAZjpRQB4qKEjm DRj9EvyWBnINgr3tKVN2Cida1El8Ki9jBjhx2GxLsy78pNKqZMI9UC/iM8cehYQ= =hUa2 -----END PGP SIGNATURE-----

2 3

Brooken Toolchain (Mac 10.6)
by Patrick Marsh May 7, 2010

May 7, 2010

Greetings, Per my previous email, I'm trying to setup the release process for Numpy on my MacBook Pro. When trying to build Numpy 1.4.1r3 with Python 2.5.4 I get a broken toolchain error (below). I do not get this error when trying to build Numpy with Python 2.6.5 - and there is nothing fundamentally different (that I know of) between my Python 2.5.4 and Python 2.6.5 environments. To address a previous suggestion I received offline, I do have sufficient permissions (I've even tried using sudo, just to make sure) and I do have setuptools installed. Any help, would be appreciated. Patrick ======================================================================== compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c' gcc: _configtest.c cc1: error: unrecognized command line option "-Wno-long-double" cc1: error: unrecognized command line option "-Wno-long-double" lipo: can't figure out the architecture type of: /var/tmp//ccjTUva4.out cc1: error: unrecognized command line option "-Wno-long-double" cc1: error: unrecognized command line option "-Wno-long-double" lipo: can't figure out the architecture type of: /var/tmp//ccjTUva4.out failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 187, in <module> setup_package() File "setup.py", line 180, in setup_package configuration=configuration ) File "/Users/pmarsh/git/numpy.release/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 657, in get_mathlib_info raise RuntimeError("Broken toolchain: cannot link a simple C program") RuntimeError: Broken toolchain: cannot link a simple C program -- Patrick Marsh Ph.D. Student / NSSL Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com

4 7

Remove duplicate columns
by T J May 7, 2010

May 7, 2010

Hi, Is there a way to sort the columns in an array? I need to sort it so that I can easily go through and keep only the unique columns. ndarray.sort(axis=1) doesn't do what I want as it destroys the relative ordering between the various columns. For example, I would like: [[2,1,3], [3,5,1], [0,3,1]] to go to: [[1,2,3], [5,3,1], [3,0,1]] (swap the first and second columns). So I want to treat the columns as objects and sort them. I can do this if I convert to a python list, but I was hoping to avoid doing that because I ultimately need to do element-wise bitwise operations. Thanks!

4 6

trouble with bool_
by Alan G Isaac May 6, 2010

May 6, 2010

What information exactly is `isnan` supposed to communicate? Put another way, given that it raises NotImplemented for unknown types, and that bool(NotImplemented) is True, is there a reason by it cannot return a Python bool (which seems more useful)? Thanks, Alan Isaac > >> np.isnan(np.nan) True > >> np.isnan(np.nan) is True False > >> type(np.isnan(np.nan)) <type 'numpy.bool_'> > >> np.isnan('') NotImplemented > >> bool(_) True > >> if np.isnan(''): print "Uh oh" ... Uh oh

1 0

SciPy 2010: Bioinformatic & Parallel/cloud talks announced...& register now!
by Amenity Applewhite May 6, 2010

May 6, 2010

Hello! Things are moving quickly in preparation for SciPy 2010: Last week we announced the General Conference schedule (http://conference.scipy.org/scipy2010/schedule.html ), Tuesday we announced our student sponsorship recipients (http://conference.scipy.org/scipy2010/student.html ) and now we're ready to tell you give you a look at the talks we have lined up for our Bioinformatics and Parallel Processing /Cloud Computing tracks. ===Parallel Processing & Cloud Computing track=== We really appreciate Brian and Ken's work organizing the papers for this specialized track. And of course, thanks to everyone who submitted a paper. There has been a great deal of interest in this set of talks — and word on the street is that Brian may even have a HPC tutorial up his sleeve... * StarCluster - NumPy/SciPy Computing in the Cloud- Justin Riley * pomsets: workflow management for your cloud- Michael J Pan * Getting Down with Big Data Jared Flatow, Anita Lillie, Ville Tuulos * StarFlow: A Cloud-Enables Python Workflow Engine for Scientific Analysis Pipelines Elaine Angelino, Dan Yamins, Margo Seltzer * A Programmatic Interface for Particle Plasma Simulation in Python, and Early Backend Results with PyCUDA Min Ragan-Kelley * Parallel Computing with IPython: an Application to Air Pollution Modeling B.E. Granger, J.G. Hemann * Astronomy App in the Cloud using Google Geo APIs and Python App Engine Shawn Shen ===Bioinformatics track=== Once again, we are indebted to Glen Otero, from Dell, for putting together the Bioinformatics track. He received some fantastic papers and we're really looking forward to these presentations: * Protein Folding with Python on Supercomputers Jan H. Meinke * Can Python Save Next-Generation Sequencing? * The Use of Galaxy for the Research and the Teaching of Genomics Roy Weckiewicz, Jim Hu, and Rodolfo Aramayo ===Early registration ends next Monday=== That's right: Only a few days left before rates increase! Think of all the BBQ and breakfast tacos you can buy with that $50-$100 you'll save by registering early. If that doesn't convince you, consider: -Cheap flights to Austin- Buy your tickets now for some very nice prices: $275 from Chicago, $330 from San Francisco, $380 from New York City, $810 from London...(prices from Kayak.com) -Convenient & affordable hotel- We got an fantastic deal for on-site accommodations at the AT&T Conference Center. Pay only $89/night for single occupancy or $105/ night for double occupancy. It will be great to have everyone staying in the same spot. Once you register, you'll get a code to book your hotel reservation. The discounted rate will be applied automatically. https://conference.scipy.org/scipy2010/accommodation.html No car necessary to get to the conference... and see Austin! An airport bus (http://capmetro.org/riding/current_schedules/maps/rt100_sb.pdf ) runs straight to and from the AT&T center, so you won't have to rent a car at all. Plus, the UT campus area is in walking distance to a number of great restaurants and activities. For any longer trips you'd like to make Austin has a great public bus system. Not to mention all of the mind-blowing things you'll learn and outstanding people you'll meet and catch up with. So what are you waiting for? Register: https://conference.scipy.org/scipy2010/registration.html Best, The SciPy 2010 Team @SciPy2010 on Twitter

1 0

PY_ARRAY_UNIQUE_SYMBOL is too far reaching?
by Austin Bingham May 6, 2010

May 6, 2010

Hi everyone, I've recently been developing a python module and C++ library in parallel, with core functionality in python and C++ largely just layered on top of the python (with boost.python.) In some cases, however, for performance reasons, the C++ API "reaches into" the python code via the C API, and this tends to happen very often with numpy-related code. As a result, I'm using the numpy C API a lot in my C++ library. To make a long story short, I'm finding that there are many places where I need to include numpy headers in my own headers (e.g. when a template class uses part of the numpy API.) If the symbol I want in my header is in ndarrayobject.h, it seems that I'm obligated to define PY_ARRAY_UNIQUE_SYMBOL because that file includes __multiarray_api.h. However, defining that macro in a header file seems like a bad idea because of potential conflicts with headers from other libraries. As a motivating example, I have a header which implements a type-mapping template for numpy types. It maps at compile time between the NPY_TYPES enum and actual C++ types. To get the NPY_TYPES definitions, I have to include arrayobject.h. Even though my header doesn't actually use the symbols that PY_ARRAY_UNIQUE_SYMBOL influences, it seems that I need to define it; even without it being defined, __multiarray_api.h leaves a PyArray_API definition in my header. So, am I missing something obvious? Or is there no way to access symbols like NPY_TYPES without pulling in the PyArray_API functions? If there isn't, would it be possible to restructure the headers so that the symbols affected by PY_ARRAY_UNIQUE_SYMBOL are separated from the others? Thanks. Austin

5 8

embedding/extending numpy:import_array() problem
by tino May 5, 2010

May 5, 2010

Hi guys, I've a C++ code that calls a python script. The python script, in turn, should use C modules to retrieve data (arrays). I'm having some problem with import_array. I've placed the import_array() just after the initialization of the interpreter and after PyInit_module Py_Initialize(); Py_InitModule("py_c_if", PY_C_IF_Methods); import_array(); This are inside a function called main_PYinit,called within the main C function. When I tryed to compile, I got the at the line of import_array the error "function must return a value". If I change the __multiarray_api.h and make import_array() a function with void instead of #define, the compiltion succeed. However, when then I try to use some API function I get problems.In particular I used this code: char *camera_buffer = cam_data; npy_int dim[] = {rows,cols}; /* whatever the size of your data, in C-order */ PyObject *myarray; double *array_buffer; myarray = PyArray_SimpleNew(2, &dim, NPY_INT); Py_INCREF(myarray); array_buffer = (double *)PyArray_DATA(myarray); for (int i=0; i<rows*cols; i++) *array_buffer++ = (double) *camera_buffer++; return PyArray_Return(myarray); And I goit this error: .\EyeSeeMain.cpp(242) : error C2664: 'PyObject *(PyTypeObject *,int,npy_intp *,int,npy_intp *,void *,int,int,PyObject *)' : cannot convert parameter 2 from 'npy_intp *' to 'int' for all the paramters of PyArray_SimpleNew. It seems that the complier did ot see the definition of PyAray_SimpleNew... Any help is grealy appreciated. Cheers L

1 0

Improvement of performance
by gerardob May 5, 2010

May 5, 2010

Hello, I have written a very simple code that computes the gradient by finite differences of any general function. Keeping the same idea, I would like modify the code using numpy to make it faster. Any ideas? Thanks. def grad_finite_dif(self,x,user_data = None): assert len(x) == self.number_variables points=[] for j in range(self.number_variables): points.append(x.copy()) points[len(points)-1][j]=points[len(points)-1][j]+0.0000001 delta_f = [] counter=0 for j in range(self.number_variables): delta_f.append((self.eval(points[counter])-self.eval(x))/0.0000001) counter = counter + 1 return array(delta_f) -- View this message in context: http://old.nabble.com/Improvement-of-performance-tp28452458p28452458.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

5 8

proposing a "beware of [as]matrix()" warning
by David Warde-Farley May 4, 2010

May 4, 2010

Trying to debug code written by an undergrad working for a colleague of mine who ported code over from MATLAB, I am seeing an ugly melange of matrix objects and ndarrays that are interacting poorly with each other and various functions in SciPy/other libraries. In particular there was a custom minimizer function that contained a line "a * b", that was receiving an Nx1 "matrix" and a N-length array and computing an outer product. Hence the unexpected 6 GB of memory usage and weird results... We've had this discussion before and it seems that the matrix class isn't going anywhere (I *really* wish it would at least be banished from the top-level namespace), but it has its adherents for pedagogical reasons. Could we at least consider putting a gigantic warning on all the functions for creating matrix objects (matrix, mat, asmatrix, etc.) that they may not behave quite so predictably in some situations and should be avoided when writing nontrivial code? There are already such warnings scattered about on SciPy.org but the situation is so bad, in my opinion (bad from a programming perspective and bad from a new user perspective, asking "why doesn't this work? why doesn't that work? why is this language/library/etc. so stupid, inconsistent, etc.?") that the situation warrants steering people still further away from the matrix object. I apologize for ranting, but it pains me when people give up on Python/NumPy because they can't figure out inconsistencies that aren't really there for a good reason. IMHO, of course. David David

16 32