Mailman 3 June 2010 - NumPy-Discussion

numpy.load raising IOError but EOFError expected
by Ruben Salvador 01 Jul '10

01 Jul '10

Hi there, I have a .npy file built by succesively adding results from different test runs of an algorithm. Each time it's run, I save a numpy.array using numpy.save as follows: fn = 'file.npy' f = open(fn, 'a+b') np.save(f, arr) f.close() When I try to read the file with the following code, for a file containing 3 array saves (a is there to show when the error arises): f = open(fn, 'rb') arr = np.load(f) a = 0 try: while True: print a a += 1 arr = np.vstack((arr, np.load(f))) except EOFError: pass f.close() I get the following output: 0 1 2 Traceback (most recent call last): File "./proc_stat.py", line 32, in <module> arr = np.vstack((arr, np.load(f))) File "/usr/lib/python2.5/site-packages/numpy/lib/io.py", line 201, in load "Failed to interpret file %s as a pickle" % repr(file) IOError: Failed to interpret file <open file '/home/rsalvador/trabajo/research/phd/devel/evowv/retest/avgfit_random_sigma_normal_Flp_20G.npy', mode 'rb' at 0x174ca08> as a pickle Using IOError in the except makes the code work, but this way I am masking other possible sources of error. I have tried with a file containing 3 dumps from numpy.save (this is, 3 arrays saved). As shown, the error is raised when trying to read a fourth time (since EOFError is not raised). Therefore, is this a bug? Shouldn't EOFError be raised instead of IOError? Or am I missunderstanding something? If this is not a bug, how can I detect the EOF to stop reading (I expect a way for this to work without tweaking the code with saving first in the file the number of dumps done)? Thanks in advance! -- Rubén Salvador PhD student @ Centro de Electrónica Industrial (CEI) http://www.cei.upm.es Blog: http://aesatcei.wordpress.com

6 11

building numpy against Cray xt-libsci
by Amir 01 Jul '10

01 Jul '10

I am trying to build numpy against Cray's xt-libsci library on a Cray XT5. I am getting an error I am hoping for hints on how to resolve: In [1]: import numpy <snip> 20 isfinite, size 21 from numpy.lib import triu ---> 22 from numpy.linalg import lapack_lite 23 from numpy.matrixlib.defmatrix import matrix_power 24 ImportError: /opt/xt-libsci/10.4.0/gnu/lib/libsci.so: undefined symbol: fftw_version These are the symbols in libsci: % nm /opt/xt-libsci/10.4.0/gnu/lib/libsci.so | grep fftw_version 00000000010f9a30 B __crafft_internal__crafft_fftw_version_num U fftw_version 00000000005aa8a4 T get_fftw_version I first built numpy with no custom site.cfg file. It built correctly and all tests ran. But it was too slow. Then I tried building numpy against libsci, which has BLAS, LAPACK, FFTW3 among other things. I had to build a libcblas.a from the netlib src as libsci does not have cblas (using gcc, gfortran 4.3.3). Here is my site.cfg, accumulated from several nice tutorials on how to build numpy on these machines, which for some reason don't work for me. The instructions were based on numpy 1.2. [blas] blas_libs = cblas library_dirs = /global/homes/amir/local/lib [lapack] lapack_libs = sci library_dirs = /opt/xt-libsci/10.4.0/gnu/lib [blas_opt] blas_libs = cblas, sci libraries = cblas, sci [lapack_opt] libraries = sci [fftw] libraries = fftw3 Here is what is linked to lapack_lite.so: % ldd ./numpy/linalg/lapack_lite.so libsci.so => /opt/xt-libsci/10.4.0/gnu/lib/libsci.so (0x00002b4493325000) libgfortran.so.3 => /opt/gcc/4.3.3/snos/lib64/libgfortran.so.3 (0x00002b44a4579000) libm.so.6 => /lib64/libm.so.6 (0x00002b44a4770000) libgcc_s.so.1 => /opt/gcc/4.3.3/snos/lib64/libgcc_s.so.1 (0x00002b44a48c6000) libc.so.6 => /lib64/libc.so.6 (0x00002b44a49dd000) /lib64/ld-linux-x86-64.so.2 (0x0000555555554000) Thanks, Amir.

3 2

PATCH: arange leaks references to provided dtype
by Lisandro Dalcin 01 Jul '10

01 Jul '10

Sorry, no time to make a proper bug report in the tracker. This bug have been annoying me from about 3 years, today I finally discovered the actual issue!!. The code below shows that arange leaks references to dtype (actually, PyArray_Descr instances). import sys import numpy as np t = np.dtype('d') while 1: a = np.arange(1,dtype=t) print (sys.getrefcount(t)) I see this in numpy 1.3.0 on py2.6 (Fedora 12). This is also present in numpy svn/trunk on py2.7 and 3.1 The trivial patch below seems to fix this refcount bug: Index: numpy/core/src/multiarray/multiarraymodule.c =================================================================== --- numpy/core/src/multiarray/multiarraymodule.c (revision 8467) +++ numpy/core/src/multiarray/multiarraymodule.c (working copy) @@ -1863,7 +1863,7 @@ static PyObject * array_arange(PyObject *NPY_UNUSED(ignored), PyObject *args, PyObject *kws) { - PyObject *o_start = NULL, *o_stop = NULL, *o_step = NULL; + PyObject *o_start = NULL, *o_stop = NULL, *o_step = NULL, *range; static char *kwd[]= {"start", "stop", "step", "dtype", NULL}; PyArray_Descr *typecode = NULL; @@ -1873,7 +1873,9 @@ Py_XDECREF(typecode); return NULL; } - return PyArray_ArangeObj(o_start, o_stop, o_step, typecode); + range = PyArray_ArangeObj(o_start, o_stop, o_step, typecode); + Py_XDECREF(typecode); + return range; } /*NUMPY_API -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

2 1

construction of object arrays
by Neal Becker 30 Jun '10

30 Jun '10

What are ways to construct object arrays? I want an array of objects, each element default constructed of a particular object type. Say my object is class A. I want a multi-dimensional array, each element constructed as A(). Right now, I use np.empty ((a,b,c...), dtype=object) and then iterate over each dimension, assigning elements: array[a,b,c...] = A() Any better suggestions?

2 1

Operations on a masked array of a scalar
by Martin Janousek 29 Jun '10

29 Jun '10

Hi, I have a Python code working on MaskedArray which can produce arrays of a number of different shapes and ranks. In an extreme case I get "an array" constructed from a scalar: x=numpy.ma.array(333.,mask=False) Until the recent upgrade to Numpy 1.4.1 it has been possible to do operations on such an object, like x+=2. or x==444. but now I have the following problem: >>> numpy.ma.array(333.,mask=False)==1. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/apps/python/2.6/lib/python2.6/site-packages/numpy/ma/core.py", line 3567, in __eq__ check._mask = self._mask AttributeError: 'numpy.bool_' object has no attribute '_mask' The problem seems to be only with __eq__ and __ne__ operators, others like __gt__ are ok: >>> numpy.ma.array(333.,mask=False)>1. masked_array(data = True, mask = False, fill_value = True) For ndarray it works fine as well: >>> numpy.array(333.)==1. False So did I find a bug or working with a "scalar array" is considered dangerous and as such it should be avoided? In any case the behaviour of Numpy 1.4.1 compared to 1.2.1 has changed in this aspect. Many thanks for any comment. Martin Janousek

2 1

[ANN] NLopt, a nonlinear optimization library
by Steven G. Johnson 29 Jun '10

29 Jun '10

The NLopt library, available from http://ab-initio.mit.edu/nlopt provides a common interface for a large number of algorithms for both global and local nonlinear optimizations, both with and without gradient information, and including both bound constraints and nonlinear equality/inequality constraints. NLopt is written in C, but now includes a Python interface (as well as interfaces for C++, Fortran, Matlab, Octave, and Guile). It is free software under the GNU LGPL. Regards, Steven G. Johnson

2 4

Decorator to access intermediate results in a function/algorithm
by Pietro Berkes 28 Jun '10

28 Jun '10

Dear everybody, This message belongs only marginally to a numpy-related mailing list, but I thought it might be of interest here since it addresses what I believe is a common pattern in scientific development. My apologies if that is not the case... The code can be found at http://github.com/pberkes/persistent_locals and requires the byteplay library (http://code.google.com/p/byteplay/). The problem ========= In scientific development, functions often represent complex data processing algorithm that transform input data into a desired output. Internally, the function typically requires several intermediate results to be computed and stored in local variables. As a simple toy example, consider the following function, that takes three arguments and returns True if the sum of the arguments is smaller than their product: def is_sum_lt_prod(a,b,c): sum = a+b+c prod = a*b*c return sum<prod A frequently occurring problem is that the developer/final user may need to access the intermediate results at a later stage, in order to analyze the detailed behavior of the algorithm, for debugging, or to write more comprehensive tests. A possible solution would be to re-define the function and return the needed internal variables, but this would break the existing code. A better solution would be to add a keyword argument to return more information: def is_sum_lt_prod(a,b,c, internals=False): sum = a+b+c prod = a*b*c if internals: return sum<prod, sum, prod else: return sum<prod This would keep the existing code intact, but only moves the problem to later stages of the development. If successively the developer needs access to even more local variables, the code has to be modified again, and part of the code is broken. Moreover, this style leads to ugly code like res, _, _, _, var1, _, var3 = f(x) where most of the returned values are irrelevant. Proposed solution ============= The proposed solution consists in a decorator that makes the local variables accessible from a function attribute, 'locals'. For example: @persistent_locals def is_sum_lt_prod(a,b,c): sum = a+b+c prod = a*b*c return sum<prod After calling the function (e.g. is_sum_lt_prod(2,1,2), which returns False) we can analyze the intermediate results as is_sum_lt_prod.locals -> {'a': 2, 'b': 1, 'c': 2, 'prod': 4, 'sum': 5} This style is cleaner, is consistent with the principle of identifying the value returned by a function as the output of an algorithm, and is robust to changes in the needs of the researcher. Note that the local variables are saved even in case of an exception, which turns out to be quite useful for debugging. How it works ========= The local variables in the inner scope of a function are not easily accessible. One solution (which I have not tried) may be to use tracing code like the one used in a debugger. This, however, would have a considerable cost in time. The proposed approach is to wrap the function in a callable object, and modify its bytecode by adding an external try...finally statement as follows: def f(self, *args, **kwargs): try: ... old code ... finally: self.locals = locals().copy() del self.locals['self'] The reason for wrapping the function in a class, instead of saving the locals in a function attribute directly, is that there are all sorts of complications in referring to itself from within a function. For example, referring to the attribute as f.locals results in the bytecode looking for the name 'f' in the namespace, and therefore moving the function, e.g. with g = f del f would break 'g'. There are even more problems for functions defined in a closure. I tried modfying f.func_globals with a custom dictionary which keeps a reference to f.func_globals, adding a static element to 'f', but this does not work as the Python interpreter does not call the func_globals dictionary with Python calls but directly with PyDict_GetItem (see http://osdir.com/ml/python.ideas/2007-11/msg00092.html). It is thus impossible to re-define __getitem__ to return 'f' as needed. Ideally, one would like to define a new closure for the function with a cell variable containing the reference, but this is impossible at present as far as I can tell. An alternative solution (see persistent_locals_with_kwarg in deco.py) is to change the signature of the function with an additional keyword argument f(arg1, arg2, _self_ref=f). However, this approach breaks functions that define an *args argument. Cost ==== The increase in execution time of the decorated function is minimal. Given its domain of application, most of the functions will take a significant amount of time to complete, making the cost the decoration negligible: import time def f(x): time.sleep(0.5) return 2*x df = deco.persistent_locals(f) %timeit f(1) 10 loops, best of 3: 501 ms per loop %timeit df(1) 10 loops, best of 3: 502 ms per loop Conclusion ======== The problem of accessing the intermediate results in an algorithm is a recurrent one in my research, and this decorator turned out to be quite useful in several occasions, and made some of the code much cleaner. Hopefully, it will be useful in other contexts as well! All the best, Pietro Berkes

2 2

Is numpy ignoring CFLAGS?
by Dr. David Kirkby 28 Jun '10

28 Jun '10

On some 64-bit platforms, which include, but is not limited to: * Some version of OS X (I don't know what versions or processors) * Solaris on SPARC processors. * Solaris on x86 processors. * OpenSolaris on SPARC processors. * OpenSolaris on x86 processors. * HP-UX on PA-RISC processors. the default is to build 32-bit objects, but 64-bit objects can be created if needed. This is usually done via adding the -m64 flag when compiling with GCC or SunStudio, though the flag will be different with HP's compiler. Numpy is used as part of Sage, but it would appear that adding -m64 to CFLAGS will not work. A comment in the script used to build numpy shows: # numpy's distutils is buggy and runs a conftest without # taking CFLAGS into account. With 64 bit OSX this results # in *boom* it then goes on to copy a file called gcc_fake, which is basically a script which gets renamed to gcc, but includes the -m64 flag. We are using numpy-1.3.0. Is this a known bug? If not, can I submit it to a bug database? Better still, does anyone have a patch to resolve it - I hate the idea of making I've now changed the build method in Sage so it does not only work on OS X, and does not hard-code the path to gcc. I have: if [ "x$SAGE64" = xyes ]; then echo "Building a 64-bit version of numpy" # HACK ALERT # HACK ALERT # HACK ALERT echo "HACK ALERT - HACK ALERT - HACK ALERT" echo "HACK ALERT - HACK ALERT - HACK ALERT" echo "HACK ALERT - HACK ALERT - HACK ALERT" echo "Creating a sort of fake gcc, which has the -m64 flag" echo "#!/usr/bin/env bash" > $SAGE_LOCAL/bin/gcc echo "`command -v gcc` -m64 \$@" >> $SAGE_LOCAL/bin/gcc chmod 755 $SAGE_LOCAL/bin/gcc fi This this file $SAGE_LOCAL/bin/gcc is used to build just numpy. After that, we can remove it if [ "x$SAGE64" = xyes ]; then echo "deleting fake gcc" rm $SAGE_LOCAL/bin/gcc fi and let the normal gcc program be used with an appropriate setting of CFLAGS to make other packages build 64-bit. I know this issue has been known in Sage for a long time (years) but I don't know if it was ever reported to the Numpy mailing list or added to any bug database Dave

3 6

numscons and Python 2.7 problems
by Christoph Gohlke 28 Jun '10

28 Jun '10

Dear NumPy developers, I was trying to build numpy 2.0.dev and scipy 0.9.dev using numscons 0.12.dev and Python 2.7b2 (32 bit) on Windows 7 64 bit and ran into two problems. First, Python 2.7's UserDict is now a new-style class (http://docs.python.org/dev/whatsnew/2.7.html). Apparently scons 1.2, which is part of numscons, is not compatible with this change and an AttributeError is thrown: AttributeError: NumpyEnvironment instance has no attribute 'SubstInFile': File "D:\numpy-trunk\numpy\core\SConstruct", line 2: GetInitEnvironment(ARGUMENTS).DistutilsSConscript('SConscript') File "C:\Python27\lib\site-packages\numscons\core\numpyenv.py", line 135: build_dir = '$build_dir', src_dir = '$src_dir') File "C:\Python27\lib\site-packages\numscons\scons-local\scons-local-1.2.0\SCons\Script\SConscript.py", line 553: return apply(_SConscript, [self.fs,] + files, subst_kw) File "C:\Python27\lib\site-packages\numscons\scons-local\scons-local-1.2.0\SCons\Script\SConscript.py", line 262: exec _file_ in call_stack[-1].globals File "D:\numpy-trunk\build\scons\numpy\core\SConscript", line 353: target = env.SubstInFile(pjoin(include_dir, '_numpyconfig.h'), Turns out that the env['BUILDERS'] dictionary is empty after the config.Finish() call in SConscript. A workaround is to copy the UserDict.py file from Python27\Lib to numscons\scons-local\scons-local-1.2.0\SCons and change UserDict back to an old-style class: @@ -1,6 +1,6 @@ """A more or less complete user-defined wrapper around dictionary objects.""" -class UserDict(object): +class UserDict: def __init__(self, dict=None, **kwargs): self.data = {} if dict is not None: @@ -80,10 +80,7 @@ def __iter__(self): return iter(self.data) -import _abcoll -_abcoll.MutableMapping.register(IterableUserDict) - class DictMixin: # Mixin defining all dictionary methods for classes that already have # a minimum dictionary interface including getitem, setitem, delitem, Second, when Python 2.7 is run on a 64 bit host, sys.platform.machine() is 'AMD64' even if Python is 32 bit (http://bugs.python.org/issue7860). Hence numscons always uses the win64 configuration on Python 2.7. This can be fixed by using "if '64 bit' in sys.version" or "platform.architecture()[0] == '64bit'" instead of "platform.machine() == 'AMD64'": diff --git a/numscons/checkers/config.py b/numscons/checkers/config.py index 4ca1079..198ccdb 100644 --- a/numscons/checkers/config.py +++ b/numscons/checkers/config.py @@ -18,7 +18,7 @@ def _get_win32_config_files(): import platform files = [pjoin(_CONFDIR, 'win32', 'perflib.cfg')] - if platform.machine() == 'AMD64': + if platform.architecture()[0] == '64bit': files.append(pjoin(_CONFDIR, 'win64', 'perflib.cfg')) return files diff --git a/numscons/core/compiler_config.py b/numscons/core/compiler_config.py index d6a79d9..606963e 100644 --- a/numscons/core/compiler_config.py +++ b/numscons/core/compiler_config.py @@ -29,7 +29,7 @@ def _get_win32_config_files(filename): import platform files = [pjoin(_CONFDIR, 'win32', filename)] - if platform.machine() == 'AMD64': + if platform.architecture()[0] == '64bit': files.append(pjoin(_CONFDIR, 'win64', filename)) return files diff --git a/numscons/core/misc.py b/numscons/core/misc.py index a546122..42836b9 100644 --- a/numscons/core/misc.py +++ b/numscons/core/misc.py @@ -59,7 +59,7 @@ def built_with_ifort(env): def is_python_win64(): # XXX: import it here because it takes some time import platform - return sys.platform == "win32" and platform.machine() == "AMD64" + return sys.platform == "win32" and platform.architecture()[0] == '64bit' def get_pythonlib_name(debug = 0): """Return the name of python library (necessary to link on NT with -- Christoph

5 5

Possible to use numexpr with user made ufuncs/scipy ufuncs?
by John Salvatier 28 Jun '10

28 Jun '10

Hello, Does anyone know whether it is possible to use numexpr with scipy ufuncs (such as those in scipy.special) or user made ufuncs? This functionality would be extremely useful. I don't see them in the list of supported functions (http://code.google.com/p/numexpr/wiki/Overview) and I get a 'TypeError: 'VariableNode' object is not callable' error when I try this. Best Regards, John Salvatier

3 8