Mailman 3 August 2010 - NumPy-Discussion

curious behavior with indexed assignment
by Nathaniel Smith 26 Aug '10

26 Aug '10

So on IRC #scipy just now, Carlos Scheidegger pointed out some behavior that I at least find very surprising: >>> a = np.zeros(10) >>> a[:] = np.arange(3) ValueError: shape mismatch: objects cannot be broadcast to a single shape Okay, that's what we expected. But if we use explicit indices... >>> a[np.arange(10)] = np.arange(3) >>> a array([ 0., 1., 2., 0., 1., 2., 0., 1., 2., 0.]) Also works with boolean masks: >>> a = np.zeros(10) >>> a[np.ones(10, dtype=bool)] = np.arange(3) >>> a array([ 0., 1., 2., 0., 1., 2., 0., 1., 2., 0.]) And if the array being assigned is too short, then the rvalue gets silently truncated: >>> a = np.zeros(10) >>> a[np.arange(5)] = np.arange(10) >>> a array([ 0., 1., 2., 3., 4., 0., 0., 0., 0., 0.]) Shouldn't these all give errors? This "recycling rule" behavior (as R would call it) seems totally inconsistent with usual broadcasting rules, but I can't seem to find any documentation or anything on it. (And indeed, I tried reproducing it with 1-d slices of a 2-d array, and just got "array is not broadcastable to the correct shape" exceptions.) Is this a bug or intentional? (Tested with self-build Numpy 1.4.1 on Python 2.6.5.) -- Nathaniel

2 1

genfromtxt usage
by Antoine Dechaume 25 Aug '10

25 Aug '10

Hello, I am trying to read a file with a variable number of values on each lines, using genfromtxt and missing_values or filling_values arguments. The usage of those arguments is not clear in the documentation, if what I am trying to do is possible, how could I do it? Thanks, Antoine.

2 1

ANN: Cython 0.13 released!
by Dag Sverre Seljebotn 25 Aug '10

25 Aug '10

Forwarding; Crag Citro wrote: It is with *great* pleasure that I email to announce the release of Cython version 0.13! This release sets another milestone on the path towards Python compatibility and brings major new features and improvements for the usability of the Cython language. Download it here: http://cython.org/release/Cython-0.13.tar.gz == New Features == * Closures are fully supported for Python functions. Cython supports inner functions and lambda expressions. Generators and generator expressions are __not__ supported in this release. * Proper C++ support. Cython knows about C++ classes, templates and overloaded function signatures, so that Cython code can interact with them in a straight forward way. * Type inference is enabled by default for safe C types (e.g. double, bint, C++ classes) and known extension types. This reduces the need for explicit type declarations and can improve the performance of untyped code in some cases. There is also a verbose compile mode for testing the impact on user code. * Cython's for-in-loop can iterate over C arrays and sliced pointers. The type of the loop variable will be inferred automatically in this case. * The Py_UNICODE integer type for Unicode code points is fully supported, including for-loops and 'in' tests on unicode strings. It coerces from and to single character unicode strings. Note that untyped for-loop variables will automatically be inferred as Py_UNICODE when iterating over a unicode string. In most cases, this will be much more efficient than yielding sliced string objects, but can also have a negative performance impact when the variable is used in a Python context multiple times, so that it needs to coerce to a unicode string object more than once. If this happens, typing the loop variable as unicode or object will help. * The built-in functions any(), all(), sum(), list(), set() and dict() are inlined as plain `for` loops when called on generator expressions. Note that generator expressions are not generally supported apart from this feature. Also, tuple(genexpr) is not currently supported - use tuple([listcomp]) instead. * More shipped standard library declarations. The python_* and stdlib/stdio .pxd files have been deprecated in favor of clib.* and cpython[.*] and may get removed in a future release. == Python compatibility == * Pure Python mode no longer disallows non-Python keywords like 'cdef', 'include' or 'cimport'. It also no longer recognises syntax extensions like the for-from loop. * Parsing has improved for Python 3 syntax in Python code, although not all features are correctly supported. The missing Python 3 features are being worked on for the next release. * from __future__ import print_function is supported in Python 2.6 and later. Note that there is currently no emulation for earlier Python versions, so code that uses print() with this future import will require at least Python 2.6. * New compiler directive language_level (valid values: 2 or 3) with corresponding command line options -2 and -3 requests source code compatibility with Python 2.x or Python 3.x respectively. Language level 3 currently enforces unicode literals for unprefixed string literals, enables the print function (requires Python 2.6 or later) and keeps loop variables in list comprehensions from leaking. * Loop variables in set/dict comprehensions no longer leak into the surrounding scope (following Python 2.7). List comprehensions are unchanged in language level 2. == Incompatible changes == * The availability of type inference by default means that Cython will also infer the type of pointers on assignments. Previously, code like this cdef char* s = ... untyped_variable = s would convert the char* to a Python bytes string and assign that. This is no longer the case and no coercion will happen in the example above. The correct way of doing this is through an explicit cast or by typing the target variable, i.e. cdef char* s = ... untyped_variable1 = <bytes>s untyped_variable2 = <object>s cdef object py_object = s cdef bytes bytes_string = s * bool is no longer a valid type name by default. The problem is that it's not clear whether bool should refer to the Python type or the C++ type, and expecting one and finding the other has already led to several hard-to-find bugs. Both types are available for importing: you can use from cpython cimport bool for the Python bool type, and from libcpp cimport bool for the C++ type. == Contributors == Many people contributed to this release, including: * David Barnett * Stefan Behnel * Chuck Blake * Robert Bradshaw * Craig Citro * Bryan Cole * Lisandro Dalcin * Eric Firing * Danilo Freitas * Christoph Gohlke * Dag Sverre Seljebotn * Kurt Smith * Erik Tollerud * Carl Witty -cc _______________________________________________ Cython-dev mailing list Cython-dev(a)codespeak.net http://codespeak.net/mailman/listinfo/cython-dev

1 0

mingw-w64 tutorial ?
by Sebastian Haase 24 Aug '10

24 Aug '10

Hi, this is somewhat OT for this list, but since I know that David and many others here have lot's of experience compiling C extensions I thought I could just ask: Looking at http://sourceforge.net/projects/mingw-w64/files/ I did not know (even after reading the FAQ) which file to download and how things would eventually work. I have a 64bit windows 7 installed, and got many precompiled packages for amd64 Python 2.7 from http://www.lfd.uci.edu/~gohlke/pythonlibs/ (thanks to Christoph Gohlke for all the work) But now I have some C++ extensions on my own, and know how build them using cygwin -- but that would only produce 32bit modules and should be unusable. So, the question is if someone has or knows of some tutorial about how to go about this - step by step. This info could maybe even go the scipy wiki.... Thanks, Sebastian Haase

5 14

Evaluating performance of f2py extensions with gprof, why spending time _gfortran_compare_string
by Åsmund Hjulstad 24 Aug '10

24 Aug '10

I am calling a few functions in a fortran library. All parameters are short (longest array of 20 elements), and I do three calls to the fortran library pr iteration. According to the python profiler (running the script as %run -p in ipython), all time is spent in the python extension. I built the extension with options -pg -O , ran a test script, and evaluated the output with gprof <libraryname>.py -b with the following output: Flat profile: Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls Ts/call Ts/call name 41.64 5.03 5.03 _gfortran_compare_string 27.40 8.34 3.31 rdxhmx_ 19.21 10.66 2.32 phimix_ 5.88 11.37 0.71 phifeq_ 2.32 11.65 0.28 phihmx_ 0.66 11.73 0.08 phiderv_ [...] and this call graph: Call graph granularity: each sample hit covers 4 byte(s) for 0.08% of 11.83 seconds index % time self children called name <spontaneous> [1] 42.9 5.07 0.00 _gfortran_compare_string [1] ----------------------------------------------- What can this mean? Executing a simple test program, exercising many of the same methods, I don't see any _gfortran_compare_string in the output. Suggestions most welcome. -- Åsmund Hjulstad, asmund(a)hjulstad.com

1 1

Datarray sprint July 28, 2010
by Keith Goodman 24 Aug '10

24 Aug '10

Join us for a datarray sprint on July 28. Several of us will meet at UC Berkeley from 2pm (Pacific Time) until our fingers bleed from typing or until 6 or 7pm, whichever comes first. If you can't be there in person, grab an item from the issue tracker or create your own. I'm told some of us will hang out on IRC #scipy, so we can coordinate there. A datarray is a subclass of a Numpy array that lets you label the axes and label the elements along each axis. Our target is to get this into Numpy. We are currently in the prototype stage, so this is a good time to jump in. code: http://github.com/fperez/datarray issue tracker: http://github.com/fperez/datarray/issues

4 6

further to my previous numpy installation problems
by martin djokovic 23 Aug '10

23 Aug '10

First of all sorry about not posting as a reply in the above thread-for some reason it does not seem to work So have to post as a new question-sorry about that. Ok so got a tar file installed as root, then did LOCATE NUMPY All most all the files found with numpy were in /usr/local/lib/python2.4/site-packages/numpy Which is where I installed numpy......... Both root and user are set to use python 2.4 However the following were also listed- I as root can remove them but I am scared that that might mess something else-should I remove them and try again? I am specially concerned about the redhat/SOURCES and also the first entry (which I did remove before installing this time but has come back) /usr/lib/python2.4/site-packages/numpy /usr/local/share/pymol/ext/lib/python2.4/site-packages/Numeric/_numpy.so /usr/share/doc/numpy-1.0.3 /usr/src/redhat/SOURCES/numpy-1.0.1-f2py.patch /usr/src/redhat/SOURCES/numpy-1.0.3-2.tar.gz /usr/src/redhat/SOURCES/numpy-1.0.3-setup.patch /usr/src/redhat/SPECS/numpy.spec /scratch/numpy-1.0.4 /usr/bin/f2py.numpy /usr/lib/python2.4/site-packages/Numeric/_numpy.so /usr/local/biopython-1.53/numpy /usr/local/lib/python2.4/site-packages/Numeric/_numpy.so

3 2

DataArray usage question + bug?
by Skipper Seabold 23 Aug '10

23 Aug '10

I have some new "typical" data that I'm trying out DataArray with, and I'm trying to get my head around it again. Is this the best way to hold data organized by, say, household and time (week number). I guess what I'm asking is do I understand the concept of axes and ticks correctly? It seems to be what I want, but my earlier understanding of how I would go about this was a bit different. import numpy as np from datarray.datarray import DataArray ddd = np.empty((52*5,5)) # Household ID ddd[:,0] = np.repeat([1111,2222,3333,4444,5555], 52) # Week Number ddd[:,1] = np.tile(range(1,53), 5) # Some Categorical Variable ddd[:,2] = np.tile(range(1,6), 52) # Some Variable ddd[:,3] = np.random.uniform(0,5.0, size=52*5) # Some Other Variable ddd[:,4] = np.random.uniform(-5,0., size=52*5) # Create axes labels and ticks hhold_ax = 'households', np.unique(ddd[:,0]).tolist() time_ax = 'week', np.unique(ddd[:,1]).tolist() var_ax = 'variables', ['some_dummy', 'some_var', 'other_var'] darr = DataArray(ddd[:,2:].reshape(5,52,-1), [hhold_ax, time_ax, var_ax]) It might be nice to have some convenience functions that will do the reshape on the original 2d data if they don't already exist, so end-users don't have to think about it. As for the bug report. If I don't tolist() the ticks above there is an error. I can file a bug report if it's warranted. <snip> --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /home/skipper/school/RA/<ipython console> in <module>() /usr/local/lib/python2.6/dist-packages/datarray/datarray.pyc in __new__(cls, data, labels, dtype, copy) 604 axes.append(Axis(label, i, arr, ticks=ticks)) 605 --> 606 _set_axes(arr, axes) 607 608 # validate the axes /usr/local/lib/python2.6/dist-packages/datarray/datarray.pyc in _set_axes(dest, in_axes) 475 # Create the containers for various axis-related info 476 for ax in in_axes: --> 477 new_ax = ax._copy(parent_arr=dest) 478 axes.append(new_ax) 479 if hasattr(ax_holder, ax.name): /usr/local/lib/python2.6/dist-packages/datarray/datarray.pyc in _copy(self, **kwargs) 113 ticks = kwargs.pop('ticks', copy.copy(self.ticks)) 114 ax.ticks = ticks --> 115 if ticks and len(ticks) != len(self.ticks): 116 ax._tick_dict = dict( zip(ticks, xrange( len(ticks) )) ) 117 else: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() Skipper

2 1

Using numpy's flatten_dtype with structured dtypes that have titles
by Emma Willemsma 23 Aug '10

23 Aug '10

Hi, I originally posted this question on stackoverflow, and the user who responded to me suggested that I bring this issue to the attention of the numpy developers. I am working with structured arrays to store experimental data. I'm using titles to store information about my fields, in this case the units of measure. When I call numpy.lib.io.flatten_dtype() on my dtype, I get: ValueError: too many values to unpack File "c:\Python25\Lib\site-packages\numpy\lib\_iotools.py", line 78, in flatten_dtype (typ, _) = ndtype.fields[field] I wouldn't really care, except that numpy.genfromtxt() calls numpy.lib.io.flatten_dtype(), and I need to be able to import my data from text files. I'm wondering if I've done something wrong. Is flatten_dtype() not meant to support titles? Is there a work-around for genfromtxt()? Here's a snippet of my code: import numpy fname = "C:\\Somefile.txt" dtype = numpy.dtype([(("Amps","Current"),"f8"),(("Volts","Voltage"),"f8")]) myarray = numpy.genfromtxt(fname,dtype) The user who responded suggested the following monkey-patch: import numpy import numpy.lib.io def flatten_dtype(ndtype): """ Unpack a structured data-type. """ names = ndtype.names if names is None: return [ndtype] else: types = [] for field in names: typ_fields = ndtype.fields[field] flat_dt = flatten_dtype(typ_fields[0]) types.extend(flat_dt) return types numpy.lib.io.flatten_dtype=flatten_dtype Is this something that could be changed in numpy? Thanks! Emma

3 5

numpy installation problems
by martin djokovic 23 Aug '10

23 Aug '10

Hello everyone, I have an old machine with Fedora 6 installed -numpy was installed but when I tried to install scipy it did not work so I gave up but that messed up my already installed numpy and now I get the following error message ******************************************** >>> import numpy Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/lib/python2.4/site-packages/numpy/__init__.py", line 44, in ? import fft File "/usr/local/lib/python2.4/site-packages/numpy/fft/__init__.py", line 4, in ? from fftpack import * File "/usr/local/lib/python2.4/site-packages/numpy/fft/fftpack.py", line 28, in ? import fftpack_lite as fftpack ImportError: /usr/local/lib/python2.4/site-packages/numpy/fft/fftpack_lite.so: undefined symbol: vmldCos2 ********************************************** I am not very conversant with Linux systems but from googling I am guessing this is to do with the C compiler-sould I reinstall something? Your help would be much appreciated. Thanks

4 3