Mailman 3 March 2013 - NumPy-Discussion

Re: [Numpy-discussion] Growing the contributor base of Numpy
by Andrea Cimatoribus March 27, 2013

March 27, 2013

Oh, I didn't even know it existed! > > Not sure if this is really relevant to the original message, but here is my opinion. I think that the numpy/scipy community would greatly benefit from a platform enabling easy sharing of code written by users. This would provide a database of solved problems, where people could dig without having to ask. I think that something like this exists for matlab, but I have no experience with it. If it exists for python, then it must be seriously under-advertised. The web provides many answers, but they are scattered in all sorts of places, and it is often impossible to contribute improvements to code found online. If such a database could enable some sort of collaborative development it would be a great added value for numpy, and would provide a natural source of new features or improvements for scipy and numpy. Supposedly that's what scipy-central is for, but it's somehow not yet reached critical mass and become a household name; I haven't looked hard enough to have any hypotheses about why not. Surya Kasturi is working on spiffing it up (see discussion on scipy-dev); I bet they could use some help if you want to scratch this itch.

1 0

Re: [Numpy-discussion] Growing the contributor base of Numpy
by Andrea Cimatoribus March 27, 2013

March 27, 2013

Not sure if this is really relevant to the original message, but here is my opinion. I think that the numpy/scipy community would greatly benefit from a platform enabling easy sharing of code written by users. This would provide a database of solved problems, where people could dig without having to ask. I think that something like this exists for matlab, but I have no experience with it. If it exists for python, then it must be seriously under-advertised. The web provides many answers, but they are scattered in all sorts of places, and it is often impossible to contribute improvements to code found online. If such a database could enable some sort of collaborative development it would be a great added value for numpy, and would provide a natural source of new features or improvements for scipy and numpy.

2 1

ANN: matplotlib 1.2.1 release
by Michael Droettboom March 27, 2013

March 27, 2013

I'm pleased to announce the release of matplotlib 1.2.1. This is a bug release and improves stability and quality over the 1.2.0 release from four months ago. All users on 1.2.0 are encouraged to upgrade. Since github no longer provides download hosting, our tarballs and binaries are back on SourceForge, and we have a master index of downloads here: http://matplotlib.org/downloads <http://matplotlib.org/downloads.html> Highlights include: - Usage of deprecated APIs in matplotlib are now displayed by default on all Python versions - Agg backend: Cleaner rendering of rectilinear lines when snapping to pixel boundaries, and fixes rendering bugs when using clip paths - Python 3: Fixes a number of missed Python 3 compatibility problems - Histograms and stacked histograms have a number of important bugfixes - Compatibility with more 3rd-party TrueType fonts - SVG backend: Image support in SVG output is consistent with other backends - Qt backend: Fixes leaking of window objects in Qt backend - hexbin with a log scale now works correctly - autoscaling works better on 3D plots - ...and numerous others. Enjoy! As always, there are number of good ways to get help with matplotlib listed on the homepage at http://matplotlib.org/ and I thank everyone for their continued support of this project. Mike Droettboom

1 0

Re: [Numpy-discussion] howto reduce along arbitrary axis
by Chao YUE March 26, 2013

March 26, 2013

Oh sorry, my fault... here is the answer by Nathaniel Smith: def retrieve_data(a, ax, idx): full_idx = [slice(None)] * a.ndim full_idx[ax] = idx return a[tuple(full_idx)] Or for the specific case where you do know the axis in advance, you just don't know how many trailing axes there are, use a[:, :, 0, ...] and the ... will expand to represent the appropriate number of :'s. probably you can sue something simlaer. Chao On Tue, Mar 26, 2013 at 3:33 PM, Neal Becker <ndbecker2(a)gmail.com> wrote: > Thank you, but do you also have an answer to this question? I only see > the question. > > > On Tue, Mar 26, 2013 at 10:23 AM, Chao YUE <chaoyuejoy(a)gmail.com> wrote: > >> Hi Neal, >> >> I forward you this mail which I think might be of help to your question. >> >> Chao >> >> ---------- Forwarded message ---------- >> From: Chao YUE <chaoyuejoy(a)gmail.com> >> Date: Sat, Mar 16, 2013 at 5:40 PM >> Subject: indexing of arbitrary axis and arbitrary slice? >> To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> >> >> >> Dear all, >> >> Is there some way to index the numpy array by specifying arbitrary axis >> and arbitrary slice, while >> not knowing the actual shape of the data? >> For example, I have a 3-dim data, data.shape = (3,4,5) >> Is there a way to retrieve data[:,0,:] by using something like >> np.retrieve_data(data,axis=2,slice=0), >> by this way you don't have to know the actual shape of the array. >> for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually >> be data[:,0,:,:] >> >> thanks in advance, >> >> Chao >> >> -- >> >> *********************************************************************************** >> Chao YUE >> Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) >> UMR 1572 CEA-CNRS-UVSQ >> Batiment 712 - Pe 119 >> 91191 GIF Sur YVETTE Cedex >> Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 >> >> ************************************************************************************ >> >> >> >> -- >> >> *********************************************************************************** >> Chao YUE >> Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) >> UMR 1572 CEA-CNRS-UVSQ >> Batiment 712 - Pe 119 >> 91191 GIF Sur YVETTE Cedex >> Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 >> >> ************************************************************************************ >> > > -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

1 0

howto reduce along arbitrary axis
by Chao YUE March 26, 2013

March 26, 2013

Hi Neal, I forward you this mail which I think might be of help to your question. Chao ---------- Forwarded message ---------- From: Chao YUE <chaoyuejoy(a)gmail.com> Date: Sat, Mar 16, 2013 at 5:40 PM Subject: indexing of arbitrary axis and arbitrary slice? To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> Dear all, Is there some way to index the numpy array by specifying arbitrary axis and arbitrary slice, while not knowing the actual shape of the data? For example, I have a 3-dim data, data.shape = (3,4,5) Is there a way to retrieve data[:,0,:] by using something like np.retrieve_data(data,axis=2,slice=0), by this way you don't have to know the actual shape of the array. for for 4-dim data, np.retrieve_data(data,axis=2,slice=0) will actually be data[:,0,:,:] thanks in advance, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************ -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

1 0

howto reduce along arbitrary axis
by Neal Becker March 26, 2013

March 26, 2013

In the following code, the function maxstar is applied along the last axis. Can anyone suggest how to modify this to apply reduction along a user-specified axis? def maxstar2 (a, b): return max (a, b) + log1p (exp (-abs (a - b))) def maxstar (u): s = u.shape[-1] if s == 1: return u[...,0] elif s == 2: return maxstar2 (u[...,0], u[...,1]) else: return maxstar2 ( maxstar (u[...,:s/2]), maxstar (u[...,s/2:]))

1 0

Unable to building numpy with openblas using bento or distutils
by Skipper Seabold March 25, 2013

March 25, 2013

Some help on this would be greatly appreciated. It's been recommended to use OpenBlas over ATLAS, so I've been trying to build numpy with openblas and have run into a few problems. 1) Build fails using bento master and waf 1.7.9, see below. 2) Distutils doesn't seem to be able to find lapack as part of atlas. I tried to skip a site.cfg and define environmental variables. No idea what I missed. I followed instructions found scattered over the internet and only understand vaguely the issues. Maybe someone can help. I'd be happy to update the wiki with any answers. To truly support OpenBlas, is it maybe necessary to make some additions to numpy/distutils/system_info.py? Thanks for having a look, Skipper Install OpenBlas ----------------------------- git clone git://github.com/xianyi/OpenBLAS cd OpenBlas Edit c_check to look for libpthreads in the right place (Kubuntu 12.10) |4 $ git diff c_check ``` diff --git a/c_check b/c_check index 4d82237..de0fd33 100644 --- a/c_check +++ b/c_check @@ -241,7 +241,7 @@ print CONFFILE "#define FUNDERSCORE\t$need_fu\n" if $need_fu if ($os eq "LINUX") { - @pthread = split(/\s+/, `nm /lib/libpthread.so* | grep _pthread_create`); + @pthread = split(/\s+/, `nm /lib/x86_64-linux-gnu/libpthread.so* | grep _pthread_create`); if ($pthread[2] ne "") { print CONFFILE "#define PTHREAD_CREATE_FUNC $pthread[2]\n"; ``` make fc=gfortran make PREFIX=~/.local install Everything looks ok, so far. Install NumPy --------------------------- Using numpy master I tried to use bento master and waf 1.7.9, following instructions from David's blog bentomaker configure --prefix=/home/skipper/.local --with-blas-lapack-libdir=/home/skipper/.local/lib --blas-lapack-type=openblas .. bentomaker build -j4 ``` <snip> [101/104] cshlib: build/numpy/core/src/umath/umath_tests.c.11.o -> build/numpy/core/umath_tests.so /usr/bin/ld: numpy/core/lib/libnpymath.a(halffloat.c.12.o): relocation R_X86_64_PC32 against symbol `npy_halfbits_to_floatbits' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value collect2: error: ld returned 1 exit status /usr/bin/ld: numpy/core/lib/libnpymath.a(halffloat.c.12.o): relocation R_X86_64_PC32 against symbol `npy_halfbits_to_floatbits' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value collect2: error: ld returned 1 exit status ``` No idea, so, let's try distutils export LAPACK=~/.local/lib/libopenblas.a export BLAS=~/.local/lib/libopenblas.a export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/.local/lib/ echo $LD_LIBRARY_PATH ``` :/usr/local/lib64/R/bin:/home/skipper/.local/lib/ ``` This step seems to be necessary? python setup.py config ``` Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 numpy/core/setup_common.py:88: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 8, with checksum f4362353e2d72f889fda0128aa015037, but recorded checksum for C API version 8 in codegen_dir/cversions.txt is 17321775fc884de0b1eda478cd61c74b. If functions were added in the C API, you have to update C_API_VERSION in numpy/core/setup_common.py. MismatchCAPIWarning) blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu'] NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu'] NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu'] NOT AVAILABLE /home/skipper/src/numpy-skipper/numpy/distutils/system_info.py:1501: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: Replacing _lib_names[0]=='blas' with 'openblas' Replacing _lib_names[0]=='openblas' with 'openblas' FOUND: libraries = ['openblas'] library_dirs = ['/home/skipper/.local/lib'] language = f77 FOUND: libraries = ['openblas'] library_dirs = ['/home/skipper/.local/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 non-existing path in 'numpy/lib': 'benchmarks' lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu'] NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib64 libraries lapack_atlas not found in /usr/local/lib64 libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib64 libraries lapack_atlas not found in /usr/lib64 libraries ptf77blas,ptcblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib/x86_64-linux-gnu libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /usr/local/lib64 libraries lapack_atlas not found in /usr/local/lib64 libraries f77blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib64 libraries lapack_atlas not found in /usr/lib64 libraries f77blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib libraries f77blas,cblas,atlas not found in /usr/lib/x86_64-linux-gnu libraries lapack_atlas not found in /usr/lib/x86_64-linux-gnu numpy.distutils.system_info.atlas_info NOT AVAILABLE /home/skipper/src/numpy-skipper/numpy/distutils/system_info.py:1415: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: Replacing _lib_names[0]=='lapack' with 'openblas' Replacing _lib_names[0]=='openblas' with 'openblas' FOUND: libraries = ['openblas'] library_dirs = ['/home/skipper/.local/lib'] language = f77 FOUND: libraries = ['openblas', 'openblas'] library_dirs = ['/home/skipper/.local/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running config ``` python setup.py build &> build.log Build log is here. Obviously it didn't go well, but I don't see anything to indicate problems. Sometimes I am able to get _dotblas.so built, though I don't know what causes it. This time I wasn't. https://gist.github.com/jseabold/7054ba9d85eae09eb402#file-numpy_build-log sudo python setup.py install &> install.log https://gist.github.com/jseabold/a0f5638b65d44aeff598#file-numpy_install-log >>> import numpy as np Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/numpy/__init__.py", line 138, in <module> import add_newdocs File "/usr/local/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in <module> from numpy.lib import add_newdoc File "/usr/local/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 15, in <module> from polynomial import * File "/usr/local/lib/python2.7/dist-packages/numpy/lib/polynomial.py", line 19, in <module> from numpy.linalg import eigvals, lstsq, inv File "/usr/local/lib/python2.7/dist-packages/numpy/linalg/__init__.py", line 50, in <module> from linalg import * File "/usr/local/lib/python2.7/dist-packages/numpy/linalg/linalg.py", line 25, in <module> from numpy.linalg import lapack_lite ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

4 7

Re: [Numpy-discussion] numpy array to C API
by Søren March 25, 2013

March 25, 2013

Thanks Valentin Your article fell in dry spot when a newbie in C/Python interfacing. Python-C-API fits perfectly with my current use-case. I got currious about the Ctypes approach as well as "Gaël Varoquaux’s blog post about avoiding data copies", but the link in the article didn't seem to work. (Under "Further Reading and References") cheers Søren On 21/03/2013 09:41, Valentin Haenel wrote: > Dear Søren, > > if you are new to interfacing python/numpy with C/C++, you may want to > check out: > > http://scipy-lectures.github.com/advanced/interfacing_with_c/interfacing_wi… > > Disclaimer: I am the author of this chapter, so this response is a bit > of a shameless plug :D > > Hope it helps. > > V- > > * Søren <sd(a)syntonetic.com> [2013-03-21]: >> Thanks Robert, for making that clear. >> >> I got a deprecated warning the second I added >> #include <numpy/arrayobject.h> >> and I got scared off too fast in my exploring phase. >> >> Cheers >> Søren >> >> On 20/03/2013 17:03, Robert Kern wrote: >>> On Wed, Mar 20, 2013 at 1:59 PM, Søren <sd(a)syntonetic.com> wrote: >>>> Greetings >>>> >>>> I'm extending our existing C/C++ software with Python/Numpy in order to do extra number crunching. >>>> It already works like a charm calling python with the C API <Python.h>. >>>> >>>> But what is the proper way of passing double arrays returned from Python/Numpy routines back to C? >>>> >>>> I came across PyArray but I can see in the compiler warnings, it is deprecated and I don't wanna start from scratch on legacy facilities. >>> What is this `PyArray` that you are referring to? There is nothing >>> named just `PyArray` to my knowledge. Do you mean direct access to the >>> `data` member of the PyArrayObject struct? Yes, that is deprecated. >>> Use the PyArray_DATA() macro to get a `void*` pointer to the start of >>> the data. >>> >>> http://docs.scipy.org/doc/numpy/reference/c-api.array.html#PyArray_DATA >>> >>> -- >>> Robert Kern >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion(a)scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion

3 2

picking elements with boolean masks
by Neal Becker March 25, 2013

March 25, 2013

starting with a NxM array, I want to select elements of the array using a set of boolean masks. The masks are simply where the indexes have a 0 or 1 in the corresponding bit position. For example, consider the case where M = 4. all_syms = np.arange (4) all_bits = np.arange (2) bit_mask = (all_syms[:,np.newaxis] >> all_bits) & 1 mask0 = bit_mask == 0 mask1 = bit_mask == 1 Maybe there's a more straightforward way to generate these masks. That's not my question. In [331]: mask1 Out[331]: array([[False, False], [ True, False], [False, True], [ True, True]], dtype=bool) OK, now I want to use this mask on D In [333]: D.shape Out[333]: (32400, 4) Just to simplify, let's just try the first row of D In [336]: D[0] Out[336]: array([ 0., 2., 2., 4.]) In [335]: D[0][mask1[...,0]] Out[335]: array([ 2., 4.]) that worked fine. But I want not just to apply one of the masks in the set (mask1 is [4,2], it has 2 masks), I want the results of applying all the masks (2 in this case) In [334]: D[0][mask1] --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-334-243c7a5e45a4> in <module>() ----> 1 D[0][mask1] ValueError: boolean index array should have 1 dimension Any ideas what's the best approach here?

1 1

howto apply-along-axis?
by Neal Becker March 24, 2013

March 24, 2013

I frequently find I have my 1d function that performs some reduction that I'd like to apply-along some axis of an n-d array. As a trivial example, def sum(u): return np.sum (u) In this case the function is probably C/C++ code, but that is irrelevant (I think). Is there a reasonably efficient way to do this within numpy?

3 2