Mailman 3 July 2009 - NumPy-Discussion

How to detect ndarrays pointing to the same memory?
by Hans Meine 22 Jul '09

22 Jul '09

Hi, I have a simple question: How can I detect whether two arrays share the same data? >>> a = numpy.arange(10) >>> b = a.view(numpy.ndarray) >>> >>> a is not b # False, as expected True >>> a.data is b.data # I expected this to be True False >>> >>> a.data <read-write buffer for 0x8a6920, size 80, offset 0 at 0x7fad2536c4b0> >>> b.data # even the memory addresses differ?! <read-write buffer for 0x894810, size 80, offset 0 at 0x7fad2536c4f0> >>> >>> a.data[4] = "\x55" >>> b.data[4] == "\x55" # yes, the buffers point to the same data True Ciao, Hans

2 2

VIGRA, NumPy and Fortran-order (again)
by Hans Meine 22 Jul '09

22 Jul '09

Hi, as I mentioned in the past [1], we considered refactoring our VIGRA (an image analysis library [2]) python bindings to be based on NumPy [3]. However, we have the problem that VIGRA uses Fortran-order indexing (i.e. there's operator()(x, y) in C++), and this should of course be the same in Python. (It is more important to us to have the same indexing in VIGRA's python bindings and in VIGRA itself, than to have the same indexing as in e.g. MPL or PIL.) As discussing in-depth in [1], numpy does not support Fortran order very well. First, there are performance issues, but even more important: the order is not preserved when performing simple operations. :-( In [8]: numpy.isfortran(a) Out[8]: True In [9]: numpy.isfortran(a + 4) Out[9]: False In [10]: numpy.isfortran(a.copy()) Out[10]: False In the worst case, this means that VIGRA functions exported to python only for unstrided images cannot be called on the results of any numpy function call. Do you agree that this is a bug^H^H^Hmissing feature and how difficult would it be to implement that? The specs would be: preserve the *ordering* of the strides (i.e. we're using mixed-order for RGB images to be able to write image[x, y] = (r, g, b)), and in the case of multiple input arguments, use the same rules (i.e. array priority) as for the output type determination. If I understood Travis' comments in the above-mentioned thread [1] correctly, this would already fix some of the performance issues along the way (since it would suddenly allow the use of special, optimized code paths). Have a nice day, Hans [1] http://mail.scipy.org/pipermail/numpy-discussion/2007-November/029837.html [2] http://hci.iwr.uni-heidelberg.de/vigra/ [3] https://mailhost.informatik.uni-hamburg.de/pipermail/vigra/2009- May/000610.html

3 3

JIT ufuncs with PyPy
by Stéfan van der Walt 22 Jul '09

22 Jul '09

Hi all, Pieter Holtzhausen sent me this interesting link: http://morepypy.blogspot.com/2009/07/pypy-numeric-experiments.html """ I have been playing recently with the idea of NumPy and PyPy integration. My idea is to integrate PyPy's JIT with NumPy or at least a very basic subset of it. Time constraints make it impossible to hand write a JIT compiler that understands NumPy. But given PyPy's architecture we actually have a JIT generator, so we don't need to write one :-) Our JIT has shown that it can speed up small arithmetic examples significantly. What happens with something like NumPy? """ Regards Stéfan

1 0

Re: [Numpy-discussion] String manipulation
by David Goldsmith 21 Jul '09

21 Jul '09

--- On Tue, 7/21/09, Christopher Barker <Chris.Barker(a)noaa.gov> wrote: > but now when I try to split it up: > > In [38]: a = a.view(dtype='S13') > > I get: > > ValueError: new type not compatible with array. > > Shouldn't that work? > > -Chris Submit an enhancement ticket. ;-) DG

1 0

proper use of __new__
by Darren Dale 21 Jul '09

21 Jul '09

http://docs.scipy.org/doc/numpy/user/basics.subclassing.html#simple-example… includes an example where an instance attribute is set in __new__ . However, there is a warning at http://www.scipy.org/Subclasses: ----- Some words of caution The definition in the __new__ method of default values for the subclass attributes is strongly discouraged for several reasons: * It is not thread-safe. * An array of a given subclass can be created from an array of another subclass without any call to the __new__ method at all. This situation occurs with the use of the view method. In that case, we must still initialize the attributes of the new subclass with the __array_finalize__ method. ----- In my quantities subclasses, I have implementations for __new__ like: def __new__(cls, data, units='', dtype=None, copy=True): if isinstance(data, cls): if units: data = data.rescale(units) return np.array(data, dtype=dtype, copy=copy, subok=True) ret = np.array(data, dtype=dtype, copy=copy).view(cls) ret._dimensionality.update(validate_dimensionality(units)) return ret and another definition for __new__ that is used to implement global UnitQuantity instances representing immutable things like meter and foot, and that __new__ implementation sets several instance attributes. I thought this was acceptable practice, based on the discussion in basics.subclassing.html, but now I'm not sure. How would I safely set instance attributes based on args/kwargs passed to the constructor, if not in __new__? Thanks, Darren

3 7

String manipulation
by Nils Wagner 21 Jul '09

21 Jul '09

Hi all, Please consider two strings >>> line_a '12345678abcdefgh12345678' >>> line_b '12345678 abcdefgh 12345678' >>> line_b.split() ['12345678', 'abcdefgh', '12345678'] Is it possible to split line_a such that the output is ['12345678', 'abcdefgh', '12345678'] Nils

5 13

numpy/core/code_generators/../src/multiarray/iterators.c:1778:could not find function name
by Nils Wagner 21 Jul '09

21 Jul '09

Hi all, I cannot build numpy from svn. ... adding 'build/src.linux-x86_64-2.5/numpy/core/include/numpy/numpyconfig.h' to sources. executing numpy/core/code_generators/generate_numpy_api.py numpy/core/code_generators/../src/multiarray/iterators.c 1778 Traceback (most recent call last): File "setup.py", line 186, in <module> setup_package() File "setup.py", line 179, in setup_package configuration=configuration ) File "/data/home/nwagner/svn/numpy/numpy/distutils/core.py", line 184, in setup return old_setup(**new_attr) File "/data/home/nwagner/local/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/data/home/nwagner/local/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/data/home/nwagner/local/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/data/home/nwagner/svn/numpy/numpy/distutils/command/install.py", line 49, in run r = old_install.run(self) File "/data/home/nwagner/local/lib/python2.5/distutils/command/install.py", line 506, in run self.run_command('build') File "/data/home/nwagner/local/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/data/home/nwagner/local/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/data/home/nwagner/svn/numpy/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/data/home/nwagner/local/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/data/home/nwagner/local/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/data/home/nwagner/local/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/data/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 130, in run self.build_sources() File "/data/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 147, in build_sources self.build_extension_sources(ext) File "/data/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 250, in build_extension_sources sources = self.generate_sources(sources, ext) File "/data/home/nwagner/svn/numpy/numpy/distutils/command/build_src.py", line 307, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 484, in generate_api h_file, c_file, doc_file = m.generate_api(os.path.join(build_dir, header_dir)) File "numpy/core/code_generators/generate_numpy_api.py", line 185, in generate_api do_generate_api(targets, sources) File "numpy/core/code_generators/generate_numpy_api.py", line 194, in do_generate_api numpyapi_list = genapi.get_api_functions('NUMPY_API', sources[0]) File "numpy/core/code_generators/genapi.py", line 267, in get_api_functions functions.extend(find_functions(f, tagname)) File "numpy/core/code_generators/genapi.py", line 220, in find_functions 'could not find function name') genapi.ParseError: numpy/core/code_generators/../src/multiarray/iterators.c:1778:could not find function name

2 2

My identity
by Keith Goodman 20 Jul '09

20 Jul '09

Using a trick that Robert Kern recently posted to the list makes the identity function much faster. Current version: def identity(n, dtype=None): a = array([1]+n*[0],dtype=dtype) b = empty((n,n),dtype=dtype) b.flat = a return b Proposed version: def myidentity(n, dtype=None): a = zeros((n,n), dtype=dtype) a.flat[::n+1] = 1 return a >> timeit identity(1) 100000 loops, best of 3: 14.9 µs per loop >> timeit identity(10) 10000 loops, best of 3: 20 µs per loop >> timeit identity(100) 1000 loops, best of 3: 696 µs per loop >> timeit identity(1000) 10 loops, best of 3: 73.6 ms per loop >> timeit myidentity(1) 100000 loops, best of 3: 6.57 µs per loop >> timeit myidentity(10) 100000 loops, best of 3: 7.08 µs per loop >> timeit myidentity(100) 100000 loops, best of 3: 16.4 µs per loop >> timeit myidentity(1000) 100 loops, best of 3: 5.92 ms per loop It would also speed up the functions that use identity (for example np.linalg.inv). And it would make identity faster than eye. Are there any corner cases that the new version doesn't handle? Any other issues?

4 7

Re: [Numpy-discussion] My identity
by David Goldsmith 20 Jul '09

20 Jul '09

OK, thanks! (Forgive my being a stickler, but I think it's good to have it in the thread, for posterity.) DG --- On Mon, 7/20/09, Keith Goodman <kwgoodman(a)gmail.com> wrote: > From: Keith Goodman <kwgoodman(a)gmail.com> > Subject: Re: [Numpy-discussion] My identity > To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> > Date: Monday, July 20, 2009, 1:16 PM > On Mon, Jul 20, 2009 at 1:11 PM, > David Goldsmith<d_l_goldsmith(a)yahoo.com> > wrote: > > > > Just to be clear, in which namespace(s) are we talking > about making (or having made) the change: IIUC, the result > you're talking about would be inappropriate for > ufunc.identity. > > np.identity > np.matlib.identity > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

1 0

Re: [Numpy-discussion] My identity
by David Goldsmith 20 Jul '09

20 Jul '09

Just to be clear, in which namespace(s) are we talking about making (or having made) the change: IIUC, the result you're talking about would be inappropriate for ufunc.identity. DG --- On Mon, 7/20/09, Keith Goodman <kwgoodman(a)gmail.com> wrote: > From: Keith Goodman <kwgoodman(a)gmail.com> > Subject: Re: [Numpy-discussion] My identity > To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> > Date: Monday, July 20, 2009, 11:17 AM > On Mon, Jul 20, 2009 at 10:53 AM, > Pauli Virtanen<pav+sp(a)iki.fi> wrote: > > On 2009-07-20, Keith Goodman <kwgoodman(a)gmail.com> > wrote: > > [clip] > >> Oh, sorry, I misunderstood. Yes, a similar change > was made to eye but > >> not to identity. > > > > Nasty, duplicated code there it seems... > > So > > def myidentity2(n, dtype=None): > return eye(n, M=n, k=0, dtype=dtype) > > >> timeit myidentity(10) > 100000 loops, best of 3: 7.15 µs per loop > >> timeit eye(10) > 100000 loops, best of 3: 8.42 µs per loop > >> timeit myidentity2(10) > 100000 loops, best of 3: 9 µs per loop > > There also a second identity in np.matlib that has its own > code. That > could be changed to > > def myidentity2_matlib(n, dtype=None): > return asmatrix(eye(n, M=n, k=0, > dtype=dtype)) > > or > > def myidentity2_matlib(n, dtype=None): > return asmatrix(myidentity2(n, dtype)) > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

2 1