Mailman 3 March 2013 - NumPy-Discussion

howto reduce along arbitrary axis
by Neal Becker March 26, 2013

March 26, 2013

In the following code, the function maxstar is applied along the last axis. Can anyone suggest how to modify this to apply reduction along a user-specified axis? def maxstar2 (a, b): return max (a, b) + log1p (exp (-abs (a - b))) def maxstar (u): s = u.shape[-1] if s == 1: return u[...,0] elif s == 2: return maxstar2 (u[...,0], u[...,1]) else: return maxstar2 ( maxstar (u[...,:s/2]), maxstar (u[...,s/2:]))

1 0

Unable to building numpy with openblas using bento or distutils
by Skipper Seabold March 25, 2013

March 25, 2013

Some help on this would be greatly appreciated. It's been recommended to use OpenBlas over ATLAS, so I've been trying to build numpy with openblas and have run into a few problems. 1) Build fails using bento master and waf 1.7.9, see below. 2) Distutils doesn't seem to be able to find lapack as part of atlas. I tried to skip a site.cfg and define environmental variables. No idea what I missed. I followed instructions found scattered over the internet and only understand vaguely the issues. Maybe someone can help. I'd be happy to update the wiki with any answers. To truly support OpenBlas, is it maybe necessary to make some additions to numpy/distutils/system_info.py? Thanks for having a look, Skipper Install OpenBlas ----------------------------- git clone git://github.com/xianyi/OpenBLAS cd OpenBlas Edit c_check to look for libpthreads in the right place (Kubuntu 12.10) |4 $ git diff c_check ``` diff --git a/c_check b/c_check index 4d82237..de0fd33 100644 --- a/c_check +++ b/c_check @@ -241,7 +241,7 @@ print CONFFILE "#define FUNDERSCORE\t$need_fu\n" if $need_fu if ($os eq "LINUX") { - @pthread = split(/\s+/, `nm /lib/libpthread.so* | grep _pthread_create`); + @pthread = split(/\s+/, `nm /lib/x86_64-linux-gnu/libpthread.so* | grep _pthread_create`); if ($pthread[2] ne "") { print CONFFILE "#define PTHREAD_CREATE_FUNC $pthread[2]\n"; ``` make fc=gfortran make PREFIX=~/.local install Everything looks ok, so far. Install NumPy --------------------------- Using numpy master I tried to use bento master and waf 1.7.9, following instructions from David's blog bentomaker configure --prefix=/home/skipper/.local --with-blas-lapack-libdir=/home/skipper/.local/lib --blas-lapack-type=openblas .. bentomaker build -j4 ``` <snip> [101/104] cshlib: build/numpy/core/src/umath/umath_tests.c.11.o -> build/numpy/core/umath_tests.so /usr/bin/ld: numpy/core/lib/libnpymath.a(halffloat.c.12.o): relocation R_X86_64_PC32 against symbol `npy_halfbits_to_floatbits' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value collect2: error: ld returned 1 exit status /usr/bin/ld: numpy/core/lib/libnpymath.a(halffloat.c.12.o): relocation R_X86_64_PC32 against symbol `npy_halfbits_to_floatbits' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value collect2: error: ld returned 1 exit status ``` No idea, so, let's try distutils export LAPACK=~/.local/lib/libopenblas.a export BLAS=~/.local/lib/libopenblas.a export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/.local/lib/ echo $LD_LIBRARY_PATH ``` :/usr/local/lib64/R/bin:/home/skipper/.local/lib/ ``` This step seems to be necessary? python setup.py config ``` Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 numpy/core/setup_common.py:88: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 8, with checksum f4362353e2d72f889fda0128aa015037, but recorded checksum for C API version 8 in codegen_dir/cversions.txt is 17321775fc884de0b1eda478cd61c74b. If functions were added in the C API, you have to update C_API_VERSION in numpy/core/setup_common.py. MismatchCAPIWarning) blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu'] NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu'] NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu'] NOT AVAILABLE /home/skipper/src/numpy-skipper/numpy/distutils/system_info.py:1501: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: Replacing _lib_names[0]=='blas' with 'openblas' Replacing _lib_names[0]=='openblas' with 'openblas' FOUND: libraries = ['openblas'] library_dirs = ['/home/skipper/.local/lib'] language = f77 FOUND: libraries = ['openblas'] library_dirs = ['/home/skipper/.local/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 non-existing path in 'numpy/lib': 'benchmarks' lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu'] NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib64 libraries lapack_atlas not found in /usr/local/lib64 libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib64 libraries lapack_atlas not found in /usr/lib64 libraries ptf77blas,ptcblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib/x86_64-linux-gnu libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /usr/local/lib64 libraries lapack_atlas not found in /usr/local/lib64 libraries f77blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib64 libraries lapack_atlas not found in /usr/lib64 libraries f77blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib libraries f77blas,cblas,atlas not found in /usr/lib/x86_64-linux-gnu libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu numpy.distutils.system_info.atlas_info NOT AVAILABLE /home/skipper/src/numpy-skipper/numpy/distutils/system_info.py:1415: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: Replacing _lib_names[0]=='lapack' with 'openblas' Replacing _lib_names[0]=='openblas' with 'openblas' FOUND: libraries = ['openblas'] library_dirs = ['/home/skipper/.local/lib'] language = f77 FOUND: libraries = ['openblas', 'openblas'] library_dirs = ['/home/skipper/.local/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running config ``` python setup.py build &> build.log Build log is here. Obviously it didn't go well, but I don't see anything to indicate problems. Sometimes I am able to get _dotblas.so built, though I don't know what causes it. This time I wasn't. https://gist.github.com/jseabold/7054ba9d85eae09eb402#file-numpy_build-log sudo python setup.py install &> install.log https://gist.github.com/jseabold/a0f5638b65d44aeff598#file-numpy_install-log >>> import numpy as np Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/numpy/__init__.py", line 138, in <module> import add_newdocs File "/usr/local/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in <module> from numpy.lib import add_newdoc File "/usr/local/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 15, in <module> from polynomial import * File "/usr/local/lib/python2.7/dist-packages/numpy/lib/polynomial.py", line 19, in <module> from numpy.linalg import eigvals, lstsq, inv File "/usr/local/lib/python2.7/dist-packages/numpy/linalg/__init__.py", line 50, in <module> from linalg import * File "/usr/local/lib/python2.7/dist-packages/numpy/linalg/linalg.py", line 25, in <module> from numpy.linalg import lapack_lite ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

4 7

Re: [Numpy-discussion] numpy array to C API
by Søren March 25, 2013

March 25, 2013

Thanks Valentin Your article fell in dry spot when a newbie in C/Python interfacing. Python-C-API fits perfectly with my current use-case. I got currious about the Ctypes approach as well as "Gaël Varoquaux’s blog post about avoiding data copies", but the link in the article didn't seem to work. (Under "Further Reading and References") cheers Søren On 21/03/2013 09:41, Valentin Haenel wrote: > Dear Søren, > > if you are new to interfacing python/numpy with C/C++, you may want to > check out: > > http://scipy-lectures.github.com/advanced/interfacing_with_c/interfacing_wi… > > Disclaimer: I am the author of this chapter, so this response is a bit > of a shameless plug :D > > Hope it helps. > > V- > > * Søren <sd(a)syntonetic.com> [2013-03-21]: >> Thanks Robert, for making that clear. >> >> I got a deprecated warning the second I added >> #include <numpy/arrayobject.h> >> and I got scared off too fast in my exploring phase. >> >> Cheers >> Søren >> >> On 20/03/2013 17:03, Robert Kern wrote: >>> On Wed, Mar 20, 2013 at 1:59 PM, Søren <sd(a)syntonetic.com> wrote: >>>> Greetings >>>> >>>> I'm extending our existing C/C++ software with Python/Numpy in order to do extra number crunching. >>>> It already works like a charm calling python with the C API <Python.h>. >>>> >>>> But what is the proper way of passing double arrays returned from Python/Numpy routines back to C? >>>> >>>> I came across PyArray but I can see in the compiler warnings, it is deprecated and I don't wanna start from scratch on legacy facilities. >>> What is this `PyArray` that you are referring to? There is nothing >>> named just `PyArray` to my knowledge. Do you mean direct access to the >>> `data` member of the PyArrayObject struct? Yes, that is deprecated. >>> Use the PyArray_DATA() macro to get a `void*` pointer to the start of >>> the data. >>> >>> http://docs.scipy.org/doc/numpy/reference/c-api.array.html#PyArray_DATA >>> >>> -- >>> Robert Kern >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion(a)scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion

3 2

picking elements with boolean masks
by Neal Becker March 25, 2013

March 25, 2013

starting with a NxM array, I want to select elements of the array using a set of boolean masks. The masks are simply where the indexes have a 0 or 1 in the corresponding bit position. For example, consider the case where M = 4. all_syms = np.arange (4) all_bits = np.arange (2) bit_mask = (all_syms[:,np.newaxis] >> all_bits) & 1 mask0 = bit_mask == 0 mask1 = bit_mask == 1 Maybe there's a more straightforward way to generate these masks. That's not my question. In [331]: mask1 Out[331]: array([[False, False], [ True, False], [False, True], [ True, True]], dtype=bool) OK, now I want to use this mask on D In [333]: D.shape Out[333]: (32400, 4) Just to simplify, let's just try the first row of D In [336]: D[0] Out[336]: array([ 0., 2., 2., 4.]) In [335]: D[0][mask1[...,0]] Out[335]: array([ 2., 4.]) that worked fine. But I want not just to apply one of the masks in the set (mask1 is [4,2], it has 2 masks), I want the results of applying all the masks (2 in this case) In [334]: D[0][mask1] --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-334-243c7a5e45a4> in <module>() ----> 1 D[0][mask1] ValueError: boolean index array should have 1 dimension Any ideas what's the best approach here?

1 1

howto apply-along-axis?
by Neal Becker March 24, 2013

March 24, 2013

I frequently find I have my 1d function that performs some reduction that I'd like to apply-along some axis of an n-d array. As a trivial example, def sum(u): return np.sum (u) In this case the function is probably C/C++ code, but that is irrelevant (I think). Is there a reasonably efficient way to do this within numpy?

3 2

Generalized inner?
by Jaime Fernández del Río March 24, 2013

March 24, 2013

The other day I found myself finding trailing edges in binary images doing something like this: arr = np.random.randint(2, size=1000).astype(np.int8) pattern = np.array([1, 1, 1, 1, 0, 0]) arr_match = 2*arr - 1 pat_match = 2*pattern - 1 from numpy.lib.stride_tricks import as_strided arr_win = as_strided(arr_match, shape=arr.shape[:-1] + (arr.shape[-1]-len(pattern)+1, len(pattern)), strides=arr.strides+arr.strides[-1:]) matches = np.einsum('...i, i', arr_win, pat_match) == len(pattern) While this works fine, this led me to thinking that all this functions (inner, dot, einsum, tensordot...) could be generalized to any other ufuncs apart from a pointwise np.multiply followed by an np.add reduction. It would be great if there was a np.gen_inner that allowed something like: np.gen_inner(arr_win, pattern, pointwise=np.equal, reduce=np.logical_and) I would like to think that such a generalization would be useful in other settings (although I can't think of any right now), and that it could find it's place in numpy, rather than in scipy.ndimage or the like. Does this make any sense? Is there any already existing way of doing this that I'm overlooking? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

2 1

Numpy 1.7.1
by Charles R Harris March 24, 2013

March 24, 2013

The Numpy 1.7.1 release process seems to have stalled. What do we need to finish up to get it going again? I think it would be nice to shoot for a release maybe the weekend after next. Chuck

4 4

C-API: Subclassing array for numpy 1.7 (with #define NPY_NO_DEPRECATED_API 0x00000007)
by Marc Gronle March 24, 2013

March 24, 2013

Hello together, we embedded Python 3 in a C++ environment. In this application I created a new class, that is a subclass from numpy array. Until now (numpy 1.6 or numpy 1.7 without the deprecation define (NPY_NO_DEPRECATED_API) the typedef for my class describing the object was something like typedef struct { PyArrayObject numpyArray; int myMember1; int myMember2; ... } PySubclassObject This always worked for me and a call to PyArray_NDIM(obj) returned the number of dimensions, where obj is of type PySubclassObject*. If I am now using numpy 1.7 this also works if the line #define NPY_NO_DEPRECATED_API 0x00000007 is not set. However, if I removed all deprecated content, there occurrs the first runtime error when creating an instance of PySubclassObject. This is due to the fact, that PyArrayObject now only is a tiny typedef of the following form typedef struct tagPyArrayObject { PyObject_HEAD } PyArrayObject; Previously this has been something like typedef struct tagPyArrayObject { PyObject_HEAD char *data; int nd; npy_intp *dimensions; ... } PyArrayObject; Usually, when creating only one np.array, there is extra space allocated depending on the size of PyArrayObject_fields. However, in my subclass I don't know how to add that extra space between the members numpyArray and myMember1. Finally, when calling PyArray_NDIM(obj) like above, the obj-pointer is casted in the macro to PyArrayObject_fields*. This yields an access conflict with myMember1, myMember2... and the members of PyArrayObject_fields. I hope this description was clear enough to understand my problem. Has anybody an idea what I need to change such that the subclassing also works for the new numpy structure? Thanks for any answer. Cheers Marc

1 0

Execution time difference between 2.7 and 3.2 using numpy
by Colin J. Williams March 23, 2013

March 23, 2013

8 19

Problem with numpy.records module
by Pierre Barbier de Reuille March 22, 2013

March 22, 2013

Hello, I am trying to use titles for record arrays. In the documentation, it is specified that any column can set to "None". However, using 'None' fails on numpy 1.6.2 because in np.core.records, on line 195, the "strip" method is called on the title object. First, I hope I understood the documentation correctly. If so, is it possible to replace the line 195 with: self._titles = [n.strip() if n is not None else None for n in titles[:self._nfields]] so 'None' elements are handled properly? Thanks, -- Barbier de Reuille Pierre

1 0

Videos of PyCon talks.
by Charles R Harris March 22, 2013

March 22, 2013

Here <http://pyvideo.org/category/33/pycon-us-2013>. Chuck

1 0

numpy array to C API
by Søren March 21, 2013

March 21, 2013

Greetings I'm extending our existing C/C++ software with Python/Numpy in order to do extra number crunching. It already works like a charm calling python with the C API <Python.h>. But what is the proper way of passing double arrays returned from Python/Numpy routines back to C? I came across PyArray but I can see in the compiler warnings, it is deprecated and I don't wanna start from scratch on legacy facilities. Going forward, what is the intended way of doing this with neat code on both sides and with a minimum of mem copy gymnastics overhead? thanks in advance Søren

5 8

how to efficiently select multiple slices from an array?
by Andreas Hilboll March 20, 2013

March 20, 2013

Cross-posting a question I asked on SO (http://stackoverflow.com/q/15527666/152439): Given an array d = np.random.randn(100) and an index array i = np.random.random_integers(low=3, high=d.size - 5, size=20) how can I efficiently create a 2d array r with r.shape = (20, 8) such that for all j=0..19, r[j] = d[i[j]-3:i[j]+5] In my case, the arrays are quite large (~200000 instead of 100 and 20), so something quick would be useful. Cheers, Andreas.

2 2

Add ability to disable the autogeneration of the function signature in a ufunc docstring.
by Warren Weckesser March 20, 2013

March 20, 2013

Hi all, In a recent scipy pull request (https://github.com/scipy/scipy/pull/459), I ran into the problem of ufuncs automatically generating a signature in the docstring using arguments such as 'x' or 'x1, x2'. scipy.special has a lot of ufuncs, and for most of them, there are much more descriptive or conventional argument names than 'x'. For now, we will include a nicer signature in the added docstring, and grudgingly put up with the one generated by the ufunc. In the long term, it would be nice to be able to disable the automatic generation of the signature. I submitted a pull request to numpy to allow that: https://github.com/numpy/numpy/pull/3149 Comments on the pull request would be appreciated. Thanks, Warren

3 5

Dot/inner products with broadcasting?
by Jaakko Luttinen March 20, 2013

March 20, 2013

Hi! How can I compute dot product (or similar multiply&sum operations) efficiently so that broadcasting is utilized? For multi-dimensional arrays, NumPy's inner and dot functions do not match the leading axes and use broadcasting, but instead the result has first the leading axes of the first input array and then the leading axes of the second input array. For instance, I would like to compute the following inner-product: np.sum(A*B, axis=-1) But numpy.inner gives: A = np.random.randn(2,3,4) B = np.random.randn(3,4) np.inner(A,B).shape # -> (2, 3, 3) instead of (2, 3) Similarly for dot product, I would like to compute for instance: np.sum(A[...,:,:,np.newaxis]*B[...,np.newaxis,:,:], axis=-2) But numpy.dot gives: In [12]: A = np.random.randn(2,3,4); B = np.random.randn(2,4,5) In [13]: np.dot(A,B).shape # -> (2, 3, 2, 5) instead of (2, 3, 5) I could use einsum for these operations, but I'm not sure whether that's as efficient as using some BLAS-supported(?) dot products. I couldn't find any function which could perform this kind of operations. NumPy's functions seem to either flatten the input arrays (vdot, outer) or just use the axes of the input arrays separately (dot, inner, tensordot). Any help? Best regards, Jaakko

2 5

fast numpy.fromfile skipping data chunks
by Andrea Cimatoribus March 20, 2013

March 20, 2013

Hi everybody, I hope this has not been discussed before, I couldn't find a solution elsewhere. I need to read some binary data, and I am using numpy.fromfile to do this. Since the files are huge, and would make me run out of memory, I need to read data skipping some records (I am reading data recorded at high frequency, so basically I want to read subsampling). At the moment, I came up with the code below, which is then compiled using cython. Despite the significant performance increase from the pure python version, the function is still much slower than numpy.fromfile, and only reads one kind of data (in this case uint32), otherwise I do not know how to define the array type in advance. I have basically no experience with cython nor c, so I am a bit stuck. How can I try to make this more efficient and possibly more generic? Thanks import numpy as np #For cython! cimport numpy as np from libc.stdint cimport uint32_t def cffskip32(fid, int count=1, int skip=0): cdef int k=0 cdef np.ndarray[uint32_t, ndim=1] data = np.zeros(count, dtype=np.uint32) if skip>=0: while k<count: try: data[k] = np.fromfile(fid, count=1, dtype=np.uint32) fid.seek(skip, 1) k +=1 except ValueError: data = data[:k] break return data

12 26

numpy reference array
by Matt U March 18, 2013

March 18, 2013

Is it possible to create a numpy array which points to the same data in a different numpy array (but in different order etc)? For example: Code: ------------------------------------------------------------------------------ import numpy as np a = np.arange(10) ids = np.array([0,0,5,5,9,9,1,1]) b = a[ids] a[0] = -1 b[0] #should be -1 if b[0] referenced the same data as a[0] 0 ------------------------------------------------------------------------------ ctypes almost does it for me, but the access is inconvenient. I would like to access b as a regular numpy array: Code: ------------------------------------------------------------------------------ import numpy as np import ctypes a = np.arange(10) ids = np.array([0,0,5,5,9,9,1,1]) b = [a[id:id+1].ctypes.data_as(ctypes.POINTER(ctypes.c_long)) for id in ids] a[0] = -1 b[0][0] #access is inconvenient -1 ------------------------------------------------------------------------------ Some more information: I've written a finite-element code, and I'm working on optimizing the python implementation. Profiling shows the slowest operation is the re-creation of an array that extracts edge degrees of freedom from the volume of the element (similar to b above). So, I'm trying to avoid copying the data every time, and just setting up 'b' once. The ctypes solution is sub-optimal since my code is mostly vectorized, that is, later I'd like to something like Code: ------------------------------------------------------------------------------ c[ids] = b[ids] + d[ids] ------------------------------------------------------------------------------ where c, and d are the same shape as b but contain different data. Any thoughts? If it's not possible that will save me time searching.

2 2

Memory issue with memory-mapped array assignment
by Thomas Robitaille March 18, 2013

March 18, 2013

Hi everyone, I've come across a memory issue when trying to assign data to slices of a Numpy memory-mapped array. The short story is that if I create a memory mapped array and try to add data to subsets of the array many times in a loop, the memory usage of my code grows over time, suggesting there is some kind of memory leak. More specifically, if I run the following script: import random import numpy as np image = np.memmap('image.np', mode='w+', dtype=np.float32, shape=(10000, 10000)) print("Before assignment") for i in range(1000): x = random.uniform(1000, 9000) y = random.uniform(1000, 9000) imin = int(x) - 128 imax = int(x) + 128 jmin = int(y) - 128 jmax = int(y) + 128 data = np.random.random((256,256)) image[imin:imax, jmin:jmax] = image[imin:imax, jmin:jmax] + data del x, y, imin, imax, jmin, jmax, data the memory usage goes up to ~300Mb after 1000 iterations (and proportionally more if I increase the number of iterations). I've written up a more detailed overview of the issue on stackoverflow (with memory profiling): http://stackoverflow.com/questions/15473377/memory-issue-with-numpy-memory-… Does anyone have any idea what is going on, and how I can avoid this issue? Thanks! Tom

1 0

indexing of arbitrary axis and arbitrary slice?
by Chao YUE March 18, 2013

March 18, 2013

Dear all, Is there some way to index the numpy array by specifying arbitrary axis and arbitrary slice, while not knowing the actual shape of the data? For example, I have a 3-dim data, data.shape = (3,4,5) Is there a way to retrieve data[:,0,:] by using something like np.retrieve_data(data,axis=2,slice=0), by this way you don't have to know the actual shape of the array. for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually be data[:,0,:,:] thanks in advance, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

2 2

Possible bug in numpy 1.6.1
by Ake Sandgren March 15, 2013

March 15, 2013

Hi! Found this thing that looks like a bug in core/src/multiarray/dtype_transfer.c diff -ru site/numpy/core/src/multiarray/dtype_transfer.c amd64_ubuntu1004-intel-acml/numpy/core/src/multiarray/dtype_transfer.c --- site/numpy/core/src/multiarray/dtype_transfer.c 2011-07-20 20:25:28.000000000 +0200 +++ amd64_ubuntu1004-intel-acml/numpy/core/src/multiarray/dtype_transfer.c 2013-03-14 22:09:38.000000000 +0100 @@ -1064,7 +1064,7 @@ _one_to_n_data *d = (_one_to_n_data *)data; PyArray_StridedTransferFn *subtransfer = d->stransfer, *stransfer_finish_src = d->stransfer_finish_src; - void *subdata = d->data, *data_finish_src = data_finish_src; + void *subdata = d->data, *data_finish_src = d->data_finish_src; npy_intp subN = d->N, dst_itemsize = d->dst_itemsize; while (N > 0) { -- Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden Internet: ake(a)hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90 7866126 Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se

2 3

Fox News
by Kevin Christman March 15, 2013

March 15, 2013

http://ezinsurance.org/bpln/iebpmnsmrme.htm

1 0

Any help from Numpy community?
by Birdada Simret March 14, 2013

March 14, 2013

* Any help from Numpy community [[ 0. 1.54 0. 0. 0. 1.08 1.08 1.08 ] [ 1.54 0. 1.08 1.08 1.08 0. 0. 0. ] [ 0. 1.08 0. 0. 0. 0. 0. 0. ] [ 0. 1.08 0. 0. 0. 0. 0. 0. ] [ 0. 1.08 0. 0. 0. 0. 0. 0. ] [ 1.08 0. 0. 0. 0. 0. 0. 0. ] [ 1.08 0. 0. 0. 0. 0. 0. 0. ] [ 1.08 0. 0. 0. 0. 0. 0. 0. ]] the above is the numpy array matrix. the numbers represents: C-C: 1.54 and C-H=1.08 So I want to write this form as C of index i is connected to C of index j C of index i is connected to H of index j (C(i),C(j)) # key C(i) and value C(j) (C(i),H(j)) # key C(i) and value H(j) ; the key C(i) can be repeated to fulfil as much as the values of H(j) To summarize, the out put may look like: C1 is connected to C2 C1 is connected to H1 C1 is connected to H3 C2 is connected to H2 etc.... Any guide is greatly appreciated, thanks birda *

2 5

Adopt Mersenne Twister 64bit?
by Siu Kwan Lam March 14, 2013

March 14, 2013

Hi all, I am redirecting a discussion on github issue tracker here. My original post (https://github.com/numpy/numpy/issues/3137): "The current implementation of the RNG seems to be MT19937-32. Since 64-bit machines are common nowadays, I am suggesting adding or upgrading to MT19937-64. Thoughts?" Let me start by answering to njsmith's comments on the issue tracker: > Would it be faster? Although I have not benchmarked the 64-bit implementation, it is likely that it will be faster on a 64-bit machine since the number of iteration (controlled by NN and MM in the reference implementation http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/VERSIONS/C-LANG/mt19937-64.c) is reduced by half. In addition, each generation in the 64-bit implementation produces a 64-bit random int which can be used to generate double precision random number. Unlike the 32-bit implementation which requires generating a pair of 32-bit random int. But, on a 32-bit machine, a 64-bit instruction is translated into 4 32-bit instructions; thus, it is likely to be slower. (1) > Use less memory? The amount of memory use will remain the same. The size of the RNG state is the same. > Provide higher quality randomness? My naive answer is that 32-bit and 64-bit implementation have the same 2^19937-1 period. Need to do some research and experiments. > Would it change the output of this program: > import numpy > numpy.random.seed(0) > print numpy.random.random() > ? Unfortunately, yes. The 64-bit implementation generates a different random number sequence with the same seed. (2) My suggestion to overcome (1) and (2) is to allow the user to select between the two implementations (and possibly different algorithms in the future). If user does not provide a choice, we use the MT19937-32 by default. numpy.random.set_state("MT19937_64", …) # choose the 64-bit implementation Thoughts? Best, Siu

5 15

can't run cython on mtrand.pyx
by Neal Becker March 14, 2013

March 14, 2013

Grabbed numpy-1.7.0 source. Cython is 0.18 cython mtrand.pyx produces lots of errors.

2 3

Yes, this one again "ImportError: No module named multiarray"
by Dinesh B Vadhia March 14, 2013

March 14, 2013

I've been using Numpy/Scipy for >5 years so know a little on how to get around them. Recently, I've needed to either freeze or create executables with tools such as PyInstaller, Cython, Py2exe and others on both Windows (XP 32-bit, 7 64-bit) and Ubuntu (12.04) Linux (64-bit). The test program (which runs perfectly with the Python interpreter) is very simple: import numpy def main(): print numpy.array([12, 23, 34, 45, 56, 67, 78, 89, 90]) return if __name__ == '__main__': main() The software versions are Python 2.7.3, Numpy 1.7.0, and Scipy 0.11. The "import numpy" causes an "ImportError: No module named multiarray". After endless Googling, I am none the wiser about what (really) causes the ImportError let alone what the solution is. The Traceback, similar to others found on the web, is: Traceback (most recent call last): File "test.py", ... File "C:\Python27\lib\site-packages\numpy\__init__.py", line 154, in <module> import add_newdocs File "C:\Python27\lib\site-packages\numpy\add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "C:\Python27\lib\site-packages\numpy\lib\__init__.py", line 4, in <module> from type_check import * File "C:\Python27\lib\site-packages\numpy\lib\type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "C:\Python27\lib\site-packages\numpy\core\__init__.py", line 5, in <module> import multiarray ImportError: No module named multiarray. Could someone shed some light on this - please? Thx.

4 5