NumPy-Discussion February 2015

numpy-discussion@python.org

53 participants
41 discussions

GSoC'15 - mentors & ideas
by Ralf Gommers 26 Feb '15

26 Feb '15

Hi all, On Fri, Feb 20, 2015 at 10:05 AM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: > Hi all, > > It's time to start preparing for this year's Google Summer of Code. There > is actually one urgent thing to be done (before 19.00 UTC today), which is > to get our ideas page in decent shape. It doesn't have to be final, but > there has to be enough on there for the organizers to judge it. This page > is here: https://github.com/scipy/scipy/wiki/GSoC-project-ideas. I'll be > reworking it and linking it from the PSF page today, but if you already > have new ideas please add them there. See > https://wiki.python.org/moin/SummerOfCode/OrgIdeasPageTemplate for this > year's template for adding a new idea. > The ideas page is now in pretty good shape. More ideas are very welcome though, especially easy or easy/intermediate ideas. Numpy right now has zero easy ones and Scipy only one and a half. What we also need is mentors. All ideas already have a potential mentor listed, however some ideas are from last year and I'm not sure that all those mentors really are available this year. And more than one potential mentor per idea is always good. So can everyone please add/remove his or her name on that page? I'm happy to take care of most of the organizational aspects this year, however I'll be offline for two weeks in July and from the end of August onwards, so I'll some help in those periods. Any volunteers? Thanks, Ralf

8 11

Objects exposing the array interface
by Jaime Fernández del Río 26 Feb '15

26 Feb '15

An issue was raised yesterday in github, regarding np.may_share_memory when run on a class exposing an array using the __array__ method. You can check the details here: https://github.com/numpy/numpy/issues/5604 Looking into it, I found out that NumPy doesn't really treat objects exposing __array__,, __array_interface__, or __array_struct__ as if they were proper arrays: 1. When converting these objects to arrays using PyArray_Converter, if the arrays returned by any of the array interfaces is not C contiguous, aligned, and writeable, a copy that is will be made. Proper arrays and subclasses are passed unchanged. This is the source of the error reported above. 2. When converting these objects using PyArray_OutputConverter, as well as in similar code in the ufucn machinery, anything other than a proper array or subclass raises an error. This means that, contrary to what the docs on subclassing say, see below, you cannot use an object exposing the array interface as an output parameter to a ufunc The following classes can be used to test this behavior: class Foo: def __init__(self, arr): self.arr = arr def __array__(self): return self.arr class Bar: def __init__(self, arr): self.arr = arr self.__array_interface__ = arr.__array_interface__ class Baz: def __init__(self, arr): self.arr = arr self.__array_struct__ = arr.__array_struct__ They all behave the same with these examples: >>> a = Foo(np.ones(5)) >>> np.add(a, a) array([ 2., 2., 2., 2., 2.]) >>> np.add.accumulate(a) array([ 1., 2., 3., 4., 5.]) >>> np.add(a, a, out=a) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: return arrays must be of ArrayType >>> np.add.accumulate(a, out=a) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: output must be an array I think this should be changed, and whatever gets handed by this methods/interfaces be treated as if it were an array or subclass of it. This is actually what the docs on subclassing say about __array__ here: http://docs.scipy.org/doc/numpy/reference/arrays.classes.html#numpy.class._… This also seems to contradict a rather cryptic comment in the code of PyArray_GetArrayParamsFromObject, which is part of the call sequence of this whole mess, see here: https://github.com/numpy/numpy/blob/maintenance/1.9.x/numpy/core/src/multia… /* * If op supplies the __array__ function. * The documentation says this should produce a copy, so * we skip this method if writeable is true, because the intent * of writeable is to modify the operand. * XXX: If the implementation is wrong, and/or if actual * usage requires this behave differently, * this should be changed! */ There has already been some discussion in the issue linked above, but I would appreciate any other thoughts on the idea of treating objects with some form of array interface as if they were arrays. Does it need a deprecation cycle? Is there some case I am not considering where this could go horribly wrong? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

2 3

PR, extended site.cfg capabilities
by Nick Papior Andersen 24 Feb '15

24 Feb '15

Dear all, I have initiated a PR-5597 <https://github.com/numpy/numpy/pull/5597>, which enables the reading of new flags from the site.cfg file. @rgommers requested that I posted some information on this site, possibly somebody could test it on their setup. So the PR basically enables reading these extra options in each section: runtime_library_dirs : Add runtime library directories to the shared libraries (overrides the dreaded LD_LIBRARY_PATH) extra_compile_args: Adds extra compile flags to the compilation extra_link_args: Adds extra flags when linking to libraries Note that this PR will "fix" a lot of issues down the line. Specifically all software which utilises numpy's distutils will benefit from this. As an example, I have successfully set runtime_library_dirs for site.cfg in numpy, where scipy, petsc4py, pygsl, slepc4py utilises these flags and this enables me to create environments without the need for LD_LIBRARY_PATH. The other options simply adds to the flexibility of the compilation to test different optimisations etc. For instance my OpenBLAS section looks like this: [openblas] library_dirs = /opt/openblas/0.2.13/gnu-4.9.2/lib include_dirs = /opt/openblas/0.2.13/gnu-4.9.2/include runtime_library_dirs = /opt/openblas/0.2.13/gnu-4.9.2/lib I hope this can be of use to somebody else than me :) Feel free to test it and provide feedback! -- Kind regards Nick

2 4

np.nonzero behavior with multidimensional arrays
by Jaime Fernández del Río 23 Feb '15

23 Feb '15

This was raised in SO today: http://stackoverflow.com/questions/28663142/why-is-np-wheres-result-read-on… np.nonzero (and np.where for boolean arrays) behave differently for 1-D and higher dimensional arrays: In the first case, a tuple with a single behaved base ndarray is returned: >>> a = np.ma.array(range(6)) >>> np.where(a > 3) (array([4, 5]),) >>> np.where(a > 3)[0].flags C_CONTIGUOUS : True F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In the second, a tuple with as many arrays as dimensions in the passed array is returned, but the arrays are not base ndarrays, but of the same subtype as was passed to the function. These arrays are also set as non-writeable: >>> np.where(a.reshape(2, 3) > 3) (masked_array(data = [1 1], mask = False, fill_value = 999999) , masked_array(data = [1 2], mask = False, fill_value = 999999) ) >>> np.where(a.reshape(2, 3) > 3)[0].flags C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : False WRITEABLE : False ALIGNED : True UPDATEIFCOPY : False I can't think of any reason that justifies this difference, and believe they should be made to return similar results. My feeling is that the proper behavior is the 1-D one, and that the behavior for multidimensional arrays should match it. Anyone can think of any reason that justifies the current behavior? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

3 3

One-byte string dtype: third time's the charm?
by Aldcroft, Thomas 23 Feb '15

23 Feb '15

The idea of a one-byte string dtype has been extensively discussed twice before, with a lot of good input and ideas, but no action [1, 2]. tl;dr: Perfect is the enemy of good. Can numpy just add a one-byte string dtype named 's' that uses latin-1 encoding as a bridge to enable Python 3 usage in the near term? A key consequence of not having a one-byte string dtype is that handling ASCII data stored in binary formats such as HDF or FITS is basically broken in Python 3. Packages like h5py, pytables, and astropy.io.fits all return text data arrays with the numpy 'S' type, and in fact have no direct support for the numpy wide unicode 'U' type. In Python 3, the 'S' type array cannot be compared with the Python str type, so that something like below fails: >>> mask = (names_array == "john") # FAIL Problems like this are now showing up in the wild [3]. Workarounds are also showing up, like a way to easily convert from 'S' to 'U' within astropy Tables [4], but this is really not a desirable way to go. Gigabyte-sized string data arrays are not uncommon, so converting to UCS-4 is a real memory and performance hit. For a good top-level summary of much of the previous thread discussion, see [5] from Chris Barker. Condensing this down to just a few points: - *Changing* the behavior of the existing 'S' type is going to break code and seems a bad idea. - *Adding* a new dtype 's' will work and allow highly performant conversion from 'S' to 's' via view(). - Using the latin-1 encoding will minimize code breakage vis-a-vis what works in Python 2 [6]. Using latin-1 is a pragmatic compromise that provides continuity to allow scientists to run their existing code in Python 3 and have things just work. It isn't perfect and it should not be the end of the story, but it would be good. This single issue is the *only* thing blocking me and my team from using Python 3 in operations. As a final point, I don't know the numpy internals at all, but it *seems* like this proposal is one of the easiest to implement amongst those that were discussed. Cheers, Tom [1]: http://mail.scipy.org/pipermail/numpy-discussion/2014-January/068622.html [2]: http://mail.scipy.org/pipermail/numpy-discussion/2014-July/070574.html [3]: https://github.com/astropy/astropy/issues/3311 [4]: http://astropy.readthedocs.org/en/latest/api/astropy.table.Table.html#astro… [5]: http://mail.scipy.org/pipermail/numpy-discussion/2014-July/070631.html [6]: It is not uncommon to store uint8 data in a bytestring

6 16

Google Summer of Code and NumFOCUS
by Raniere Silva 20 Feb '15

20 Feb '15

Hi, NumFOCUS has promotes and supports the ongoing research and development of open-source computing tools including NumPy. This year NumFOCUS want to try be a Google Summer of Code "umbrella" mentoring organization, Umbrella organizations are mentoring organizations accepted into the Google Summer of Code program that have other open source organizations working "under" them. Sometime organizations that work very closely or have very similar goals or communities may get put together under an "umbrella." Google stills expects all organizations under the umbrella, whether accepted into the program under their title or not, to adhere to all the rules and regulations of the program. From https://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2… To help promote and support NumPy. We encourage NumPy to apply to Google Summer of Code under your own title and will be very happy if you can also do with us. If you are interested, please check https://github.com/swcarpentry/gsoc2015 and https://github.com/swcarpentry/gsoc2015/blob/master/CONTRIBUTING.md. If you have any question, please email me directly. Thanks in advance, Raniere

2 2

GSoC'15 prep - ideas page
by Ralf Gommers 20 Feb '15

20 Feb '15

Hi all, It's time to start preparing for this year's Google Summer of Code. There is actually one urgent thing to be done (before 19.00 UTC today), which is to get our ideas page in decent shape. It doesn't have to be final, but there has to be enough on there for the organizers to judge it. This page is here: https://github.com/scipy/scipy/wiki/GSoC-project-ideas. I'll be reworking it and linking it from the PSF page today, but if you already have new ideas please add them there. See https://wiki.python.org/moin/SummerOfCode/OrgIdeasPageTemplate for this year's template for adding a new idea. Cheers, Ralf

1 0

PyCon?
by Nathaniel Smith 18 Feb '15

18 Feb '15

Hi all, It looks like I'll be at PyCon this year. Anyone else? Any interest in organizing a numpy sprint? -n -- Nathaniel J. Smith -- http://vorpus.org

2 1

ANN: ffnet-0.8.0 released
by Marek Wojciechowski 16 Feb '15

16 Feb '15

ffnet-0.8.0 has been released. ffnet is a fast and easy-to-use feed-forward neural network training solution for python This version supports python 3. Look at ffnet website: http://ffnet.sourceforge.net for installation instructions and documentation. Regards, -- Marek Wojciechowski --- Politechnika Łódzka Lodz University of Technology Treść tej wiadomości zawiera informacje przeznaczone tylko dla adresata. Jeżeli nie jesteście Państwo jej adresatem, bądź otrzymaliście ją przez pomyłkę prosimy o powiadomienie o tym nadawcy oraz trwałe jej usunięcie. This email contains information intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient or if you have received this message in error, please notify the sender and delete it from your system.

1 0

Matrix Class
by cjw 14 Feb '15

14 Feb '15

It seems to be agreed that there are weaknesses in the existing Numpy Matrix Class. Some problems are illustrated below. I'll try to put some suggestions over the coming weeks and would appreciate comments. Colin W. Test Script: if __name__ == '__main__': a= mat([4, 5, 6]) # Good print('a: ', a) b= mat([4, '5', 6]) # Not the expected result print('b: ', b) c= mat([[4, 5, 6], [7, 8]]) # Wrongly accepted as rectangular print('c: ', c) d= mat([[1, 2, 3]]) try: d[0, 1]= 'b' # Correctly flagged, not numeric except ValueError: print("d[0, 1]= 'b' # Correctly flagged, not numeric", ' ValueError') print('d: ', d) Result: *** Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)] on win32. *** >>> a: [[4 5 6]] b: [['4' '5' '6']] c: [[[4, 5, 6] [7, 8]]] d[0, 1]= 'b' # Correctly flagged, not numeric ValueError d: [[1 2 3]] >>> -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Matrix-Class-tp39719.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

11 20

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion February 2015