Mailman 3 September 2012 - NumPy-Discussion

Fwd: [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303)
by Nathaniel Smith 15 Oct '12

15 Oct '12

[Manual PR notification] ---------- Forwarded message ---------- From: timcera Date: Sat, Jun 9, 2012 at 10:13 PM Subject: [numpy] ENH: Initial implementation of a 'neighbor' calculation (#303) To: njsmith <njs(a)pobox.com> Each element is assigned the result of a function based on it's neighbors. Neighbors are selected based on a weight array. It uses the new pad routines to pad arrays if neighboring values are required that would be off the edge of the input array. Will be great to have the masked array settled because right now you can only sort of exclude from the neighborhood using a zero in the weight array. Zero or np.IGNORE don't affect np.sum, but functions like np.mean and np.std would give different answers. Because of this my early implementations of neighbor included an optional mask array along with the weight array, but I decided would be best to wait for the new masked arrays. This in some ways could be considered a generalization of a convolution, and comparison with existing numpy/scipy convolution results are included in the tests. The advantage to neighbor is that any function that accepts a 1-d array, and returns a single result, can be used instead of convolution only using summation. The convolution functions require the weight array to be flipped to get the same answer as neighbor. You can merge this Pull Request by running: git pull https://github.com/timcera/numpy neighbor Or you can view, comment on it, or merge it online at: https://github.com/numpy/numpy/pull/303 -- Commit Summary -- * ENH: Initial implementation of a 'neighbor' calculation where the each -- File Changes -- M numpy/lib/__init__.py (2) A numpy/lib/neighbor.py (305) A numpy/lib/tests/test_neighbor.py (278) -- Patch Links -- https://github.com/numpy/numpy/pull/303.patch https://github.com/numpy/numpy/pull/303.diff --- Reply to this email directly or view it on GitHub: https://github.com/numpy/numpy/pull/303

7 14

ANN: WinPython v2.7.3.0
by Pierre Raybaut 13 Oct '12

13 Oct '12

Hi all, I'm pleased to introduce my new contribution to the Python community: WinPython. WinPython v2.7.3.0 has been released and is available for 32-bit and 64-bit Windows platforms: http://code.google.com/p/winpython/ WinPython is a free open-source portable distribution of Python for Windows, designed for scientists. It is a full-featured (see http://code.google.com/p/winpython/wiki/PackageIndex) Python-based scientific environment: * Designed for scientists (thanks to the integrated libraries NumPy, SciPy, Matplotlib, guiqwt, etc.: * Regular *scientific users*: interactive data processing and visualization using Python with Spyder * *Advanced scientific users and software developers*: Python applications development with Spyder, version control with Mercurial and other development tools (like gettext) * *Portable*: preconfigured, it should run out of the box on any machine under Windows (without any installation requirements) and the folder containing WinPython can be moved to any location (local, network or removable drive) * *Flexible*: one can install (or should I write "use" as it's portable) as many WinPython versions as necessary (like isolated and self-consistent environments), even if those versions are running different versions of Python (2.7, 3.x in the near future) or different architectures (32bit or 64bit) on the same machine * *Customizable*: using the integrated package manager (wppm, as WinPython Package Manager), it's possible to install, uninstall or upgrade Python packages (see http://code.google.com/p/winpython/wiki/WPPM for more details on supported package formats). *WinPython is not an attempt to replace Python(x,y)*, this is just something different (see http://code.google.com/p/winpython/wiki/Roadmap): more flexible, easier to maintain, movable and less invasive for the OS, but certainly less user-friendly, with less packages/contents and without any integration to Windows explorer [*]. [*] Actually there is an optional integration into Windows explorer, providing the same features as the official Python installer regarding file associations and context menu entry (this option may be activated through the WinPython Control Panel). Enjoy! -Pierre

2 1

dtype '|S0' not understood
by David Warde-Farley 07 Oct '12

07 Oct '12

Howdy, It seems it's possible using e.g. In [25]: dtype([('foo', str)])Out[25]: dtype([('foo', '|S0')]) to get yourself a zero-length string. However dtype('|S0') results in a TypeError: data type not understood. I understand the stupidity of creating a 0-length string field but it's conceivable that it's accidental. For example, it could lead to a situation where you've created that field, are missing all the data you had meant to put in it, serialize with np.save, and upon np.load aren't able to get _any_ of your data back because the dtype descriptor is considered bogus (can you guess why I thought of this scenario?). It seems that either dtype(str) should do something more sensible than zero-length string, or it should be possible to create it with dtype('| S0'). Which should it be? David

3 3

specifying numpy as dependency in your project, install_requires
by Ralf Gommers 06 Oct '12

06 Oct '12

Hi, An issue I keep running into is that packages use: install_requires = ["numpy"] or install_requires = ['numpy >= 1.6'] in their setup.py. This simply doesn't work a lot of the time. I actually filed a bug against patsy for that (https://github.com/pydata/patsy/issues/5), but Nathaniel is right that it would be better to bring it up on this list. The problem is that if you use pip, it doesn't detect numpy (may work better if you had installed numpy with setuptools) and tries to automatically install or upgrade numpy. That won't work if users don't have the right compiler. Just as bad would be that it does work, and the user didn't want to upgrade for whatever reason. This isn't just my problem; at Wes' pandas tutorial at EuroScipy I saw other people have the exact same problem. My recommendation would be to not use install_requires for numpy, but simply do something like this in setup.py: try: import numpy except ImportError: raise ImportError("my_package requires numpy") or try: from numpy.version import short_version as npversion except ImportError: raise ImportError("my_package requires numpy") if npversion < '1.6': raise ImportError("Numpy version is %s; required is version >= 1.6" % npversion) Any objections, better ideas? Is there a good place to put it in the numpy docs somewhere? Ralf

7 10

memory-efficient loadtxt
by Paul Anton Letnes 03 Oct '12

03 Oct '12

Hello everyone, I've modified loadtxt to make it (potentially) more memory efficient. The idea is that if a user passes a seekable file, (s)he can also pass the 'seekable=True' kwarg. Then, loadtxt will count the number of lines (containing data) and allocate an array of exactly the right size to hold the loaded data. The downside is that the line counting more than doubles the runtime, as it loops over the file twice, and there's a sort-of unnecessary np.array function call in the loop. The branch is called faster-loadtxt, which is silly due to the runtime doubling, but I'm hoping that the false advertising is acceptable :) (I naively expected a speedup by removing some needless list manipulation.) I'm pretty sure that the function can be micro-optimized quite a bit here and there, and in particular, the main for loop is a bit duplicated right now. However, I got the impression that someone was working on a More Advanced (TM) C-based file reader, which will replace loadtxt; this patch is intended as a useful thing to have while we're waiting for that to appear. The patch passes all tests in the test suite, and documentation for the kwarg has been added. I've modified all tests to include the seekable kwarg, but that was mostly to check that all tests are passed also with this kwarg. I guess it's bit too late for 1.7.0 though? Should I make a pull request? I'm happy to take any and all suggestions before I do. Cheers Paul

3 7

Re: [Numpy-discussion] tests for casting table? (was: Numpy 1.7b1 API change cause big trouble)
by Matthew Brett 03 Oct '12

03 Oct '12

Hi, On Sun, Sep 9, 2012 at 6:12 PM, Frédéric Bastien <nouiz(a)nouiz.org> wrote: > The third is releated to change to the casting rules in numpy. Before > a scalar complex128 * vector float32 gived a vector of dtype > complex128. Now it give a vector of complex64. The reason is that now > the scalar of different category only change the category, not the > precision. I would consider a must that we warn clearly about this > interface change. Most people won't see it, but people that optimize > there code heavily could depend on such thing. It seems to me that it would be a very good idea to put the casting table results into the tests to make sure we are keeping track of this kind of thing. I'm happy to try to do it if no-one else more qualified has time. Best, Matthew

5 5

Behavior of .base
by Travis Oliphant 02 Oct '12

02 Oct '12

Hey all, In a github-discussion with Gael and Nathaniel, we came up with a proposal for .base that we should put before this list. Traditionally, .base has always pointed to None for arrays that owned their own memory and to the "most immediate" array object parent for arrays that did not own their own memory. There was a long-standing issue related to running out of stack space that this behavior created. Recently this behavior was altered so that .base always points to "the original" object holding the memory (something exposing the buffer interface). This created some problems for users who relied on the fact that most of the time .base pointed to an instance of an array object. The proposal here is to change the behavior of .base for arrays that don't own their own memory so that the .base attribute of an array points to "the most original object" that is still an instance of the type of the array. This would go into the 1.7.0 release so as to correct the issues reported. What are reactions to this proposal? -Travis

8 18

Making numpy sensible: backward compatibility please
by Gael Varoquaux 02 Oct '12

02 Oct '12

Hi numpy developers, First of all, thanks a lot for the hard work you put in numpy. I know very well that maintaining such a core library is a lot of effort and a service to the community. But "with great dedication, comes great responsibility" :). I find that Numpy is a bit of a wild horse, a moving target. I have just fixed a fairly nasty bug in scikit-learn [1] that was introduced by change of semantics in ordering when doing copies with numpy. I have been running working and developing the scikit-learn while tracking numpy's development tree and, as far as I can tell, I never saw warnings raised in our code that something was going to change, or had changed. In other settings, changes in array inheritance and 'base' propagation have made impossible some of our memmap-related usecase that used to work under previous numpy [2]. Other's have been hitting difficulties related to these changes in behavior [3]. Not to mention the new casting rules (default: 'same_kind') that break a lot of code, or the ABI change that, while not done an purpose, ended up causing us a lot of pain. My point here is that having code that works and gives correct results with new releases of numpy is more challenging that it should be. I cannot claim that I disagree with the changes that I mention above. They were all implemented for a good reason and can all be considered as overall improvements to numpy. However the situation is that given a complex codebase relying on numpy that works at a time t, the chances that it works flawlessly at time t + 1y are thin. I am not too proud that we managed to release scikit-learn 0.12 with a very ugly bug under numpy 1.7. That happened although we have 90% of test coverage, buildbots under different numpy versions, and a lot of people, including me, using our development tree on a day to day basis with bleeding edge numpy. Most code in research settings or RD industry does not benefit from such software engineering and I believe is much more likely to suffer from changes in numpy. I think that this is a cultural issue: priority is not given to stability and backward compatibility. I think that this culture is very much ingrained in the Python world, that likes iteratively cleaning its software design. For instance, I have the feeling that in the scikit-learn, we probably fall in the same trap. That said, such a behavior cannot fare well for a base scientific environment. People tell me that if they take old matlab code, the odds that it will still works is much higher than with Python code. As a geek, I tend to reply that we get a lot out of this mobility, because we accumulate less cruft. However, in research settings, for reproducibility reasons, ones need to be able to pick up an old codebase and trust its results without knowing its intricacies. >From a practical standpoint, I believe that people implementing large changes to the numpy codebase, or any other core scipy package, should think really hard about their impact. I do realise that the changes are discussed on the mailing lists, but there is a lot of activity to follow and I don't believe that it is possible for many of us to monitor the discussions. Also, putting more emphasis on backward compatibility is possible. For instance, the 'order' parameter added to np.copy could have defaulted to the old behavior, 'K', for a year, with a DeprecationWarning, same thing for the casting rules. Thank you for reading this long email. I don't mean it to be a complaint about the past, but more a suggestion on something to keep in mind when making changes to core projects. Cheers, Gaël ____ [1] https://github.com/scikit-learn/scikit-learn/commit/7842748cf777412c506a8c0… [2] http://mail.scipy.org/pipermail/numpy-discussion/2012-September/063985.html [3] http://mail.scipy.org/pipermail/numpy-discussion/2012-July/063126.html

9 17

Reductions with nditer working only with the last axis
by Sergio Pascual 02 Oct '12

02 Oct '12

Hello, I'm trying to understand how to work with nditer to do a reduction, in my case converting a 3d array into a 2d array. I followed the help here http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html and managed to create a function that applies reduction over the last axis of the input. With this function def nditer_sum(data, red_axes): it = numpy.nditer([data, None], flags=['reduce_ok', 'external_loop'], op_flags=[['readonly'], ['readwrite', 'allocate']], op_axes=[None, red_axes]) it.operands[1][...] = 0 for x, y in it: y[...] = x.sum() return it.operands[1] I can get something equivalent to data.sum(axis=2) >>> data = numpy.arange(2*3*4).reshape((2,3,4)) >>> nditer_sum(data, [0, 1, -1]) [[ 6 22 38] [54 70 86]] >>> data.sum(axis=2) [[ 6 22 38] [54 70 86]] So to get something equivalent to data.sum(axis=0) I though that it was enough to change the argument red_axes to [-1, 0,1] But the result is quite different. >>> data = numpy.arange(2*3*4).reshape((2,3,4)) >>> data.sum(axis=0) [[12 14 16 18] [20 22 24 26] [28 30 32 34]] >>> nditer_sum(data, [-1, 0, 1]) [[210 210 210 210] [210 210 210 210] [210 210 210 210]] In the for loop inside nditer_sum (for x,y in it:), the iterator is looping 2 times and giving an array of length 12 each time, instead of looping 12 times and giving an array of length 2 each time. I have read the numpy documentation several times and googled about this to no avail. Does anybody have an example of a reduction in the first axis of an array using nditer? Is this a bug? Regards, Sergio

3 2

silently ignored size mismatch (bug??)
by Neal Becker 01 Oct '12

01 Oct '12

In [19]: u = np.arange (10) In [20]: v = np.arange (10) In [21]: u[v] = u In [22]: u[v] = np.arange(11) silence...

3 3