Mailman 3 October 2012 - NumPy-Discussion

numpy distutils log error with easy_install
by Matthew Brett Oct. 6, 2012

Oct. 6, 2012

Hi, One of our kind users pointed out an error when using easy_install to install our package nipy. I've reproduced it now on a bare package using numpy distutils and having a trivial extension: https://github.com/matthew-brett/apkg To reproduce: git clone git://github.com/mathew-brett/apkg.git easy_install apkg You should get something like this: Processing apkg Running setup.py -q bdist_egg --dist-dir /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB Appending apkg configuration to Ignoring attempt to set 'name' (from '' to 'apkg') zip_safe flag not set; analyzing archive contents... Adding apkg 0.1 to easy-install.pth file Installed /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg Processing dependencies for apkg==0.1 Finished processing dependencies for apkg==0.1 /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252: RuntimeWarning: Parent module 'numpy.distutils' not found while handling absolute import from numpy.distutils import log Note the last error. All of nipy, scikits-learn and scikits-timeseries (that use numpy distutils) give this same error on easy_install. The error requires the use of an extension in the package, as in: https://github.com/matthew-brett/apkg/blob/master/apkg/setup.py#L4 It appears to be benign but it certainly confuses people doing easy_install. This is running current numpy master, but the same occurs with numpy 1.5.1 that I have to hand. Any ideas what could be causing this? Cheers, Matthew

2 8

ANN: WinPython v2.7.3.1
by Pierre Raybaut Oct. 5, 2012

Oct. 5, 2012

Hi all, WinPython v2.7.3.1 has been released and is available for 32-bit and 64-bit Windows platforms: http://code.google.com/p/winpython/ WinPython is a free open-source portable distribution of Python for Windows, designed for scientists. It is a full-featured (see http://code.google.com/p/winpython/wiki/PackageIndex) Python-based scientific environment: * Designed for scientists (thanks to the integrated libraries NumPy, SciPy, Matplotlib, guiqwt, etc.: * Regular *scientific users*: interactive data processing and visualization using Python with Spyder * *Advanced scientific users and software developers*: Python applications development with Spyder, version control with Mercurial and other development tools (like gettext) * *Portable*: preconfigured, it should run out of the box on any machine under Windows (without any installation requirements) and the folder containing WinPython can be moved to any location (local, network or removable drive) * *Flexible*: one can install (or should I write "use" as it's portable) as many WinPython versions as necessary (like isolated and self-consistent environments), even if those versions are running different versions of Python (2.7, 3.x in the near future) or different architectures (32bit or 64bit) on the same machine * *Customizable*: using the integrated package manager (wppm, as WinPython Package Manager), it's possible to install, uninstall or upgrade Python packages (see http://code.google.com/p/winpython/wiki/WPPM for more details on supported package formats). *WinPython is not an attempt to replace Python(x,y)*, this is just something different (see http://code.google.com/p/winpython/wiki/Roadmap): more flexible, easier to maintain, movable and less invasive for the OS, but certainly less user-friendly, with less packages/contents and without any integration to Windows explorer [*]. [*] Actually there is an optional integration into Windows explorer, providing the same features as the official Python installer regarding file associations and context menu entry (this option may be activated through the WinPython Control Panel). Enjoy!

1 0

ufuncs for structured arrays
by Jay Bourque Oct. 4, 2012

Oct. 4, 2012

All, I've submitted the following pull request for NumPy: https://github.com/numpy/numpy/pull/462 This change allows ufuncs to be registered for structured arrays by using a new API method PyUFunc_RegisterLoopForStructType. For example, a ufunc could be registered to take two arrays of type 'u8,u8,u8' and return an array of type 'u8,u8,u8'. I have a trivial example of this included in my pull request, along with further details of my changes. I suspect there might be a better way to do this, so any suggestions for improvements would be welcome. Thanks, -Jay continuum.io

2 3

ANN: HDF5 for Python (h5py) 2.1.0-final
by Andrew Collette Oct. 4, 2012

Oct. 4, 2012

Announcing HDF5 for Python (h5py) 2.1.0 ======================================= We are proud to announce the availability of HDF5 for Python (h5py) 2.1.0! This release has been a long time coming. Thanks to everyone who contributed code and filed bug reports! What's new in h5py 2.1 ----------------------- * The HDF5 Dimension Scales API is now available, along with high-level integration with Dataset objects. Thanks to D. Dale for implementing this. * Unicode scalar strings can now be stored in attributes. * Dataset objects now expose a .size property giving the total number of elements. * Many performance improvements and bug fixes About the project ----------------------- HDF5 for Python (h5py) is a general-purpose Python interface to the Hierarchical Data Format library, version 5. HDF5 is a mature scientific software library originally developed at NCSA, designed for the fast, flexible storage of enormous amounts of data. >From a Python programmer's perspective, HDF5 provides a robust way to store data, organized by name in a tree-like fashion. You can create datasets (arrays on disk) hundreds of gigabytes in size, and perform random-access I/O on desired sections. Datasets are organized in a filesystem-like hierarchy using containers called "groups", and accessed using the traditional POSIX /path/to/resource syntax. Downloads, FAQ and bug tracker are available at Google Code: * Google code site: http://h5py.googlecode.com Documentation is available at Alfven.org: * http://h5py.alfven.org

1 0

[ANN] MDP-3.3 released!
by Tiziano Zito Oct. 4, 2012

Oct. 4, 2012

We are glad to announce release 3.3 of the Modular toolkit for Data Processing (MDP). This a bug-fix release, all current users are invited to upgrade. MDP is a Python library of widely used data processing algorithms that can be combined according to a pipeline analogy to build more complex data processing software. The base of available algorithms includes signal processing methods (Principal Component Analysis, Independent Component Analysis, Slow Feature Analysis), manifold learning methods ([Hessian] Locally Linear Embedding), several classifiers, probabilistic methods (Factor Analysis, RBM), data pre-processing methods, and many others. What's new in version 3.3? -------------------------- - support sklearn versions up to 0.12 - cleanly support reload - fail gracefully if pp server does not start - several bug-fixes and improvements Resources --------- Download: http://sourceforge.net/projects/mdp-toolkit/files Homepage: http://mdp-toolkit.sourceforge.net Mailing list: http://lists.sourceforge.net/mailman/listinfo/mdp-toolkit-users Acknowledgments --------------- We thank the contributors to this release: Philip DeBoer, Yaroslav Halchenko. The MDP developers, Pietro Berkes Zbigniew Jędrzejewski-Szmek Rike-Benjamin Schuppner Niko Wilbert Tiziano Zito

1 0

memory-efficient loadtxt
by Paul Anton Letnes Oct. 3, 2012

Oct. 3, 2012

Hello everyone, I've modified loadtxt to make it (potentially) more memory efficient. The idea is that if a user passes a seekable file, (s)he can also pass the 'seekable=True' kwarg. Then, loadtxt will count the number of lines (containing data) and allocate an array of exactly the right size to hold the loaded data. The downside is that the line counting more than doubles the runtime, as it loops over the file twice, and there's a sort-of unnecessary np.array function call in the loop. The branch is called faster-loadtxt, which is silly due to the runtime doubling, but I'm hoping that the false advertising is acceptable :) (I naively expected a speedup by removing some needless list manipulation.) I'm pretty sure that the function can be micro-optimized quite a bit here and there, and in particular, the main for loop is a bit duplicated right now. However, I got the impression that someone was working on a More Advanced (TM) C-based file reader, which will replace loadtxt; this patch is intended as a useful thing to have while we're waiting for that to appear. The patch passes all tests in the test suite, and documentation for the kwarg has been added. I've modified all tests to include the seekable kwarg, but that was mostly to check that all tests are passed also with this kwarg. I guess it's bit too late for 1.7.0 though? Should I make a pull request? I'm happy to take any and all suggestions before I do. Cheers Paul

3 7

Re: [Numpy-discussion] tests for casting table? (was: Numpy 1.7b1 API change cause big trouble)
by Matthew Brett Oct. 3, 2012

Oct. 3, 2012

Hi, On Sun, Sep 9, 2012 at 6:12 PM, Frédéric Bastien <nouiz(a)nouiz.org> wrote: > The third is releated to change to the casting rules in numpy. Before > a scalar complex128 * vector float32 gived a vector of dtype > complex128. Now it give a vector of complex64. The reason is that now > the scalar of different category only change the category, not the > precision. I would consider a must that we warn clearly about this > interface change. Most people won't see it, but people that optimize > there code heavily could depend on such thing. It seems to me that it would be a very good idea to put the casting table results into the tests to make sure we are keeping track of this kind of thing. I'm happy to try to do it if no-one else more qualified has time. Best, Matthew

5 5

Behavior of .base
by Travis Oliphant Oct. 2, 2012

Oct. 2, 2012

Hey all, In a github-discussion with Gael and Nathaniel, we came up with a proposal for .base that we should put before this list. Traditionally, .base has always pointed to None for arrays that owned their own memory and to the "most immediate" array object parent for arrays that did not own their own memory. There was a long-standing issue related to running out of stack space that this behavior created. Recently this behavior was altered so that .base always points to "the original" object holding the memory (something exposing the buffer interface). This created some problems for users who relied on the fact that most of the time .base pointed to an instance of an array object. The proposal here is to change the behavior of .base for arrays that don't own their own memory so that the .base attribute of an array points to "the most original object" that is still an instance of the type of the array. This would go into the 1.7.0 release so as to correct the issues reported. What are reactions to this proposal? -Travis

8 18

Making numpy sensible: backward compatibility please
by Gael Varoquaux Oct. 2, 2012

Oct. 2, 2012

Hi numpy developers, First of all, thanks a lot for the hard work you put in numpy. I know very well that maintaining such a core library is a lot of effort and a service to the community. But "with great dedication, comes great responsibility" :). I find that Numpy is a bit of a wild horse, a moving target. I have just fixed a fairly nasty bug in scikit-learn [1] that was introduced by change of semantics in ordering when doing copies with numpy. I have been running working and developing the scikit-learn while tracking numpy's development tree and, as far as I can tell, I never saw warnings raised in our code that something was going to change, or had changed. In other settings, changes in array inheritance and 'base' propagation have made impossible some of our memmap-related usecase that used to work under previous numpy [2]. Other's have been hitting difficulties related to these changes in behavior [3]. Not to mention the new casting rules (default: 'same_kind') that break a lot of code, or the ABI change that, while not done an purpose, ended up causing us a lot of pain. My point here is that having code that works and gives correct results with new releases of numpy is more challenging that it should be. I cannot claim that I disagree with the changes that I mention above. They were all implemented for a good reason and can all be considered as overall improvements to numpy. However the situation is that given a complex codebase relying on numpy that works at a time t, the chances that it works flawlessly at time t + 1y are thin. I am not too proud that we managed to release scikit-learn 0.12 with a very ugly bug under numpy 1.7. That happened although we have 90% of test coverage, buildbots under different numpy versions, and a lot of people, including me, using our development tree on a day to day basis with bleeding edge numpy. Most code in research settings or RD industry does not benefit from such software engineering and I believe is much more likely to suffer from changes in numpy. I think that this is a cultural issue: priority is not given to stability and backward compatibility. I think that this culture is very much ingrained in the Python world, that likes iteratively cleaning its software design. For instance, I have the feeling that in the scikit-learn, we probably fall in the same trap. That said, such a behavior cannot fare well for a base scientific environment. People tell me that if they take old matlab code, the odds that it will still works is much higher than with Python code. As a geek, I tend to reply that we get a lot out of this mobility, because we accumulate less cruft. However, in research settings, for reproducibility reasons, ones need to be able to pick up an old codebase and trust its results without knowing its intricacies. >From a practical standpoint, I believe that people implementing large changes to the numpy codebase, or any other core scipy package, should think really hard about their impact. I do realise that the changes are discussed on the mailing lists, but there is a lot of activity to follow and I don't believe that it is possible for many of us to monitor the discussions. Also, putting more emphasis on backward compatibility is possible. For instance, the 'order' parameter added to np.copy could have defaulted to the old behavior, 'K', for a year, with a DeprecationWarning, same thing for the casting rules. Thank you for reading this long email. I don't mean it to be a complaint about the past, but more a suggestion on something to keep in mind when making changes to core projects. Cheers, Gaël ____ [1] https://github.com/scikit-learn/scikit-learn/commit/7842748cf777412c506a8c0… [2] http://mail.scipy.org/pipermail/numpy-discussion/2012-September/063985.html [3] http://mail.scipy.org/pipermail/numpy-discussion/2012-July/063126.html

9 17

Reductions with nditer working only with the last axis
by Sergio Pascual Oct. 1, 2012

Oct. 1, 2012

Hello, I'm trying to understand how to work with nditer to do a reduction, in my case converting a 3d array into a 2d array. I followed the help here http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html and managed to create a function that applies reduction over the last axis of the input. With this function def nditer_sum(data, red_axes): it = numpy.nditer([data, None], flags=['reduce_ok', 'external_loop'], op_flags=[['readonly'], ['readwrite', 'allocate']], op_axes=[None, red_axes]) it.operands[1][...] = 0 for x, y in it: y[...] = x.sum() return it.operands[1] I can get something equivalent to data.sum(axis=2) >>> data = numpy.arange(2*3*4).reshape((2,3,4)) >>> nditer_sum(data, [0, 1, -1]) [[ 6 22 38] [54 70 86]] >>> data.sum(axis=2) [[ 6 22 38] [54 70 86]] So to get something equivalent to data.sum(axis=0) I though that it was enough to change the argument red_axes to [-1, 0,1] But the result is quite different. >>> data = numpy.arange(2*3*4).reshape((2,3,4)) >>> data.sum(axis=0) [[12 14 16 18] [20 22 24 26] [28 30 32 34]] >>> nditer_sum(data, [-1, 0, 1]) [[210 210 210 210] [210 210 210 210] [210 210 210 210]] In the for loop inside nditer_sum (for x,y in it:), the iterator is looping 2 times and giving an array of length 12 each time, instead of looping 12 times and giving an array of length 2 each time. I have read the numpy documentation several times and googled about this to no avail. Does anybody have an example of a reduction in the first axis of an array using nditer? Is this a bug? Regards, Sergio

3 2