Mailman 3 September 2013 - NumPy-Discussion

Join and stacking for structured arrays
by Aldcroft, Thomas 06 Sep '13

06 Sep '13

For the astropy Table class (which wraps numpy structured arrays), I wrote functions that perform table joins and concatenate tables along rows or columns. These are reasonably full-featured and handle most of the common needs for these operations. The join function here addresses some limitations in the lib.recfunctions.join_by() function. You can see the documentation for the Table interface to these functions at [1]. The reason I'm writing is that the implementation is split into a pure numpy-based module (no astropy dependence [5]) and the astropy bit. Essentially all of the real work is done in the numpy-based module, so I'm wondering if the numpy project is interested in this code for inclusion into numpy. If so, I'd be happy to put together a pull request. I would just need direction on: - Where to put it - What function names to use - How to setup for included Cython code The Python and Cython code and tests can be see at [2], [3], [4]. The tests are currently based on astropy Tables, so these would be modified to use plain ndarray or masked arrays. Cheers, Tom [1]: http://astropy.readthedocs.org/en/latest/table/operations.html [2]: https://github.com/astropy/astropy/blob/master/astropy/table/np_utils.py [3]: https://github.com/astropy/astropy/blob/master/astropy/table/_np_utils.pyx [4]: https://github.com/astropy/astropy/blob/master/astropy/table/tests/test_ope… [5]: There is a requirement for an OrderedDict which is currently provided by astropy for python 2.6.

1 0

Re: [Numpy-discussion] Funny business with 'is' operator?:
by Nathaniel Smith 06 Sep '13

06 Sep '13

The .data attribute is generated on the fly when accessed. So it returns an anonymous temporary that's deallocated as soon as it's no longer needed. a.data is b.data needs both objects, so both get allocated and then compared. In the second one though, each object gets allocated one at a time and then immediately released. So the a.data and b.data objects are different... but since they have non-overlapping lifetimes, they happen to be out at the same memory location. id() is only guaranteed to be unique while the object is alive. Notice also: a.data is a.data -> False -n On 6 Sep 2013 17:21, "James Bergstra" <bergstrj(a)iro.umontreal.ca> wrote: > Hi, could someone help me understand why this assertion fails? > > def test_is(self): > a = np.empty(1) > b = np.empty(1) > if a.data is not b.data: > assert id(a.data) != id(b.data) # <-- fail > > I'm trying to write an alternate may_share_memory function. > > Thanks, > - James > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >

1 0

Weird behavior of gufuncs
by Jaime Fernández del Río 06 Sep '13

06 Sep '13

Hi all, I am seeing some very weird behavior on a gufunc I coded. It has a pretty complicated signature: '(r,c,p),(i,j,k,n),(u,v),(d),(n,q)->(q,r,c)' And a single registered loop function, for types: uint8, uint16, uint16, uintp, uint8->uint8. In general it performs beautifully well, returning the expected results about x10 faster than a MATLAB version using a 12 core CPU (!) But today I found out that for certain sizes of the inputs, it doesn't even call the loop function and returns a zeroed output array immediately. I am using Python 2.7.5 and Numpy 1.7.1 under Windows 7, and have seen the behavior in both x64 and x86 versions. The thresholds vary, and I don't fully understand the logic behind it, but there is a clear threshold of sizes: keeping all other inputs the same, one byte more in the other argument and it returns all zeros, one byte less and everything works fine. The thresholds are different for 64 and 32 bit versions. I have a strong suspicion that this may be a NumPy 1.7 bug, because a colleague running MacOS and NumPy 1.8 compiled from the development branch isn't seeing any of this issues I have. It could of course be a Windows thing, but somehow that seems less likely, especially since there are other bugs in gufuncs that have been fixed in 1.8 but haven't been backported to 1.7. Tomorrow I am going to try and get access to a Linux box with Numpy 1.7 to try to reproduce it there. Does this sound familiar to anyone? Any known (and hopefully solved) issue that could trigger this behavior? A search for ufunc in github's issue tracker didn't give me any meaningful leads... Thanks! Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

2 1

ANN: Numpy 1.8.0 beta 1 release
by Charles R Harris 06 Sep '13

06 Sep '13

Hi all, I'm happy to announce the first beta release of Numpy 1.8.0. Please try this beta and report any issues on the numpy-dev mailing list. Source tarballs and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.8.0b1/. The Windows and OS X installers will follow when the infrastructure issues are dealt with. Chuck

8 25

A bug in numpy.random.shuffle?
by Fernando Perez 05 Sep '13

05 Sep '13

Hi all, I just ran into this rather weird behavior: http://nbviewer.ipython.org/6453869 In summary, as far as I can tell, shuffle is misbehaving when acting on arrays that have structured dtypes. I've seen the problem on 1.7.1 (official on ubuntu 13.04) as well as master as of a few minutes ago. Is this my misuse? It really looks like a bug to me... Cheers, f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail

7 16

Stick intersection path algorithm
by Josè Luis Mietta 05 Sep '13

05 Sep '13

Hi experts! I wanna study the intersection between line segments (sticks). I wrote a algorithm that generate a matrix, M, with N rows and N columns. The M-element Mij is 1 if stick number 'i' intersect stick number 'j' (obviously M is symmetric). Given two arbitrary sticks, i need a simple and effective algorithm that determinate if that two sticks are conected by a 'intersected-sticks' path. Any idea for that? Thanks a lot!

4 12

fast_any_all , a trivial but fast/useful helper function for numpy
by Graeme B. Bell 05 Sep '13

05 Sep '13

In my current GIS raster work I often have a situation where I generate code something like this: np.any([A>4, A==2, B==5, ...]) However, np.any() is quite slow. It's possible to use np.logical_or to solve the problem, but then you get nested logical_or's, since logical_or combines only two parameters. It's also possible to use integer maths e.g. (A>4)+(A==2)+(B==5)>0. The question is: which is best (syntactically, in terms of performance, etc)? I've written a little helper function to provide a faster version of any() and all(). It's embarrassingly simple - just a for loop. However, I think there's a syntactic advantage to using a helper function for this situation rather than writing it idiomatically each time; and it reduces the chance of a bug in idiomatic implementation. However, the code does not cover all the use cases currently addressed by np.any() and np.all(). I benchmarked to pick the fastest underlying implementation (logical_or rather than integer maths). The result is 14 to 17x faster than np.any() for this use case.* Code & benchmark here: https://github.com/gbb/numpy-fast-any-all Please feel welcome to use it or improve it :-) Graeme. * (Should this become an execution path in np.any()... ?)

5 8

dtype "metadata" attribute
by Andrew Collette 05 Sep '13

05 Sep '13

Hi, We have a pull request pending at the h5py project to use the dtype "metadata" attribute to implement special type support, as opposed to the current hacked-together solution based on field names: >>> dt = dtype('i', metadata={"key": 42}) >>> dt.metadata <dictproxy {'key': 42}> However, I can't find documentation on it anywhere; the "metadata" keyword doesn't even appear in the dtype docstring. Is this an officially supported feature in NumPy? I am mostly concerned about it going away in a future release. Thanks! Andrew Collette

1 0

Error occurance on Skimage 0.9.0 version
by Happyman 05 Sep '13

05 Sep '13

Hello, I downloaded skimage 0.9.0dev version and installed like following on Windows; cd C:\skimage\python setup.py install. One error occurred: Unable to find vcvarsall.bat . It is not the first time I have been taken such an error. What is the best way to avoid this problem??? Any answers will be appreciated

3 3

Minimal NumPy for distribution
by Frédéric Bastien 05 Sep '13

05 Sep '13

Hi, I have done some exploratory work with Theano to generate a shared library from a Theano function. This link with numpy c api. If we want to distribute this library and call it from C and/or python, what is the minimal installed part of NumPy needed? I suppose that only the c api is needed. Do someone already checked that? >From what I found on linux, numpy linked with a dynamic BLAS take 13MB. There is 2.7MB from .so, 5.3MB from .py and 3.8MB from .pyc. Can we just keep all .so and remove all .py and .pyc file? I suppose we need to modify the __init__.py file too. On Windows, with the unofficial 64 bit binary of NumPy linked with MKL, the numpy directory take 56MB. So I would also like to know witch shared library(dll) is needed. Here is the .so file found on linux: /lib/python2.7/site-packages/numpy/core/multiarray_tests.so /lib/python2.7/site-packages/numpy/core/multiarray.so /lib/python2.7/site-packages/numpy/core/_dotblas.so /lib/python2.7/site-packages/numpy/core/_sort.so /lib/python2.7/site-packages/numpy/core/scalarmath.so /lib/python2.7/site-packages/numpy/core/umath_tests.so /lib/python2.7/site-packages/numpy/core/umath.so /lib/python2.7/site-packages/numpy/numarray/_capi.so /lib/python2.7/site-packages/numpy/linalg/lapack_lite.so /lib/python2.7/site-packages/numpy/random/mtrand.so /lib/python2.7/site-packages/numpy/fft/fftpack_lite.so /lib/python2.7/site-packages/numpy/lib/_compiled_base.so Can I get rid of all shared lib outside of core? thanks Frédéric

2 2