Mailman 3 November 2012 - NumPy-Discussion

Numpy's policy for releasing memory
by Austin Bingham Nov. 13, 2012

Nov. 13, 2012

I'm trying to understand how numpy decides when to release memory and whether it's possible to exert any control over that. The situation is that I'm profiling memory usage on a system in which a great deal of the overall memory is tied up in ndarrays. Since numpy manages ndarray memory on its own (i.e. without the python gc, or so it seems), I'm finding that I can't do much to convince numpy to release memory when things get tight. For python object, for example, I can explicitly run gc.collect(). So, in an effort to at least understand the system better, can anyone tell me how/when numpy decides to release memory? And is there any way via either the Python or C-API to explicitly request release? Thanks. Austin

5 7

random.choice
by Alan G Isaac Nov. 13, 2012

Nov. 13, 2012

I just noticed that 1.7 is scheduled to add a random.choice function. I wonder if the best structure has been chosen. Specifically, it does not provide for array flattening, and it does not provide for subarray choice. Back in 2006 (?) Robert Kern suggested something like the below (forgive any mistranslation). I think the included functionality does belong in a choice function. I do like the provision for repeated sampling in the current proposal, however. Cheers, Alan Isaac def choice(x, axis=None): """Select an element or subarray uniformly randomly. If axis is None, then a single element is chosen from the entire array. Otherwise, a subarray is chosen from the given axis. """ x = np.asarray(x) if axis is None: length = np.multiply.reduce(x.shape) n = random.randint(length) return x.flat[n] else: n = random.randint(x.shape[axis]) idx = map(slice, x.shape) idx[axis] = n return x[tuple(idx)]

3 16

Scipy dot
by Nicolas SCHEFFER Nov. 13, 2012

Nov. 13, 2012

Hi, I've written a snippet of code that we could call scipy.dot, a drop-in replacement for numpy.dot. It's dead easy, and just answer the need of calling the right blas function depending on the type of arrays, C or F order (aka slowness of np.dot(A, A.T)) While this is not the scipy mailing list, I was wondering if this snippet would relevant and/or useful to others, numpy folks, scipy folks or could be integrated directly in numpy (so that we keep the nice A.dot(B) syntax) This bottleneck of temporary copies has been a problem for lots of users and it seems everybody has their own snippets. This code is probably not written as it should, I hope the community can help improving it! ;) First FIXME is to make it work for arrays of dimensions other than 2. Suggestions highly appreciated! Thanks! === Code (also on http://pastebin.com/QrRk0kEf) def dot(A, B, out=None): """ A drop in replaement for numpy.dot Computes A.B optimized using fblas call note: unlike in numpy the returned array is in F order""" import scipy.linalg as sp gemm = sp.get_blas_funcs('gemm', arrays=(A,B)) if out is None: lda, x, y, ldb = A.shape + B.shape if x != y: raise ValueError("matrices are not aligned") dtype = np.max([x.dtype for x in (A, B)]) out = np.empty((lda, ldb), dtype, order='F') if A.flags.c_contiguous and B.flags.c_contiguous: gemm(alpha=1., a=A.T, b=B.T, trans_a=True, trans_b=True, c=out, overwrite_c=True) if A.flags.c_contiguous and B.flags.f_contiguous: gemm(alpha=1., a=A.T, b=B, trans_a=True, c=out, overwrite_c=True) if A.flags.f_contiguous and B.flags.c_contiguous: gemm(alpha=1., a=A, b=B.T, trans_b=True, c=out, overwrite_c=True) if A.flags.f_contiguous and B.flags.f_contiguous: gemm(alpha=1., a=A, b=B, c=out, overwrite_c=True) return out == Timing (EPD, MKL): In [15]: A = np.array(np.random.randn(1000, 1000), 'f') In [16]: %timeit np.dot(A, A) 100 loops, best of 3: 7.19 ms per loop In [17]: %timeit np.dot(A.T, A.T) 10 loops, best of 3: 27.7 ms per loop In [18]: %timeit np.dot(A, A.T) 100 loops, best of 3: 18.3 ms per loop In [19]: %timeit np.dot(A.T, A) 100 loops, best of 3: 18.7 ms per loop In [20]: %timeit dot(A, A) 100 loops, best of 3: 7.16 ms per loop In [21]: %timeit dot(A.T, A.T) 100 loops, best of 3: 6.67 ms per loop In [22]: %timeit dot(A, A.T) 100 loops, best of 3: 6.79 ms per loop In [23]: %timeit dot(A.T, A) 100 loops, best of 3: 7.02 ms per loop

10 35

Compiling NumPy on Windows for Python 3.3
by Peter Cock Nov. 13, 2012

Nov. 13, 2012

Dear all, Since the NumPy 1.7.0b2 release didn't include a Windows (32 bit) installer for Python 3.3, I am considering compiling it myself for local testing. What compiler is recommended? Thanks, Peter

2 11

(no subject)
by Jeffrey Spencer Nov. 12, 2012

Nov. 12, 2012

http://www.a1bailbondsmd.com/wp-includes/ugoogle.html

1 0

Double-ended queues
by William Furnass Nov. 9, 2012

Nov. 9, 2012

Hi all, I want to be able to within a loop a) apply a mathematical operation to all elements in a vector (can be done atomically) then b) pop zero or more elements from one end of the vector and c) push zero or more elements on to the other end. So far I've used a collections.deque to store my vector as it should be more efficient than a numpy array for the appending and deletion of elements. However, I was wondering whether performance could be improved through the use of a homogeneously-typed double-ended queue i.e. a linked list equivalent of numpy.ndarray. Has anyone previously considered whether it would be worth including such a thing within the numpy package? Cheers, Will

6 6

Help compiling numpy with new gcc
by Filipe Pires Alvarenga Fernandes Nov. 9, 2012

Nov. 9, 2012

Hi I am trying to compile numpy with gcc 4.7.1 and I am having the following issue. "RuntimeError: Broken toolchain: cannot link a simple C program" I noticed that I need to pass the flag '-fno-use-linker-plugin' to be able to compile it. However, even though I did pass it by exporting the CFLAGS, it does not work. I guess that numpy do not use the CFLAGS for its internal extensions. How can I pass that option to it? Error below: [ 11s] compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.7 -c' [ 11s] gcc: _configtest.c [ 11s] gcc -pthread _configtest.o -o _configtest [ 11s] gcc: fatal error: -fuse-linker-plugin, but liblto_plugin.so not found [ 11s] compilation terminated. [ 11s] gcc: fatal error: -fuse-linker-plugin, but liblto_plugin.so not found [ 11s] compilation terminated. [ 11s] failure. [ 11s] removing: _configtest.c _configtest.o [ 11s] Traceback (most recent call last): [ 11s] File "setup.py", line 214, in <module> [ 11s] setup_package() [ 11s] File "setup.py", line 207, in setup_package [ 11s] configuration=configuration ) [ 11s] File "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/core.py", line 186, in setup [ 11s] return old_setup(**new_attr) [ 11s] File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup [ 11s] dist.run_commands() [ 11s] File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands [ 11s] self.run_command(cmd) [ 11s] File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command [ 11s] cmd_obj.run() [ 11s] File "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build.py", line 37, in run [ 11s] old_build.run(self) [ 11s] File "/usr/lib64/python2.7/distutils/command/build.py", line 127, in run [ 11s] self.run_command(cmd_name) [ 11s] File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command [ 11s] self.distribution.run_command(command) [ 11s] File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command [ 11s] cmd_obj.run() [ 11s] File "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py", line 152, in run [ 11s] self.build_sources() [ 11s] File "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py", line 163, in build_sources [ 11s] self.build_library_sources(*libname_info) [ 11s] File "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py", line 298, in build_library_sources [ 11s] sources = self.generate_sources(sources, (lib_name, build_info)) [ 11s] File "/home/abuild/rpmbuild/BUILD/numpy/numpy/distutils/command/build_src.py", line 385, in generate_sources [ 11s] source = func(extension, build_dir) [ 11s] File "numpy/core/setup.py", line 648, in get_mathlib_info [ 11s] raise RuntimeError("Broken toolchain: cannot link a simple C program") [ 11s] RuntimeError: Broken toolchain: cannot link a simple C program [ 11s] error: Bad exit status from /var/tmp/rpm-tmp.yO2SIE (%build) [ 11s] [ 11s] [ 11s] RPM build errors: [ 11s] Bad exit status from /var/tmp/rpm-tmp.yO2SIE (%build)

3 2

testing with amd libm/acml
by Neal Becker Nov. 8, 2012

Nov. 8, 2012

I'm trying to do a bit of benchmarking to see if amd libm/acml will help me. I got an idea that instead of building all of numpy/scipy and all of my custom modules against these libraries, I could simply use: LD_PRELOAD=/opt/amdlibm-3.0.2/lib/dynamic/libamdlibm.so:/opt/acml5.2.0/gfortran64/lib/libacml.so <my program here> I'm hoping that both numpy and my own dll's then will take advantage of these libraries. Do you think this will work?

5 18

numexpr question
by Neal Becker Nov. 8, 2012

Nov. 8, 2012

I'm interested in trying numexpr, but have a question (not sure where's the best forum to ask). The examples I see use ne.evaluate ("some string...") When used within a loop, I would expect the compilation from the string form to add significant overhead. I would have thought a pre-compiled form would be available, similar to a precompiled regexp. yes/no?

2 1

strange behavior of numpy.unique
by Phillip Feldman Nov. 7, 2012

Nov. 7, 2012

numpy.unique behaves as I would expect for small inputs like the following: In [12]: x= [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] In [13]: unique(x, return_index=True) Out[13]: (array([0, 1, 2, 3]), array([0, 2, 5, 9], dtype=int64)) But, when I give it something larger, the return index values do not always correspond to the first occurrences in the input. The documentation is silent on the question of how the return index values are chosen when a given element of x appears more than once. Either the documentation should be clarified, or better yet, the behavior should be changed.

4 4