Mailman 3 September 2013 - NumPy-Discussion

find appropriate dtype based on a set of values
by Gregorio Bastardo Sept. 5, 2013

Sept. 5, 2013

Hi, I'd like to find the smallest possible representation of an array given a set of possible values. I've checked the function 'np.min_scalar_type', it works well for scalar input, but contrary to my assumption when array-like param is given: array's dtype is simply returned, instead of finding a more compact dtype for the array values. >>> np.version.version '1.7.0' >>> np.min_scalar_type(0) # ok dtype('uint8') >>> np.min_scalar_type(-1) # ok dtype('int8') >>> np.min_scalar_type( [0,-1] ) # int8 expected, returns platform-default int dtype('int32') >>> np.min_scalar_type( [0,256] ) # uint16 expected dtype('int32') >>> np.min_scalar_type([-1,256]) # int16 expected dtype('int32') Am I missing something? Anyone knows how to achieve the desired operation? Thanks a lot, Gregorio

3 6

Re: [Numpy-discussion] fast_any_all , a trivial but fast/useful helper function for numpy
by Graeme B. Bell Sept. 5, 2013

Sept. 5, 2013

Hi Robert, Thanks for proposing an alternative implementation approach. However, did you test your proposal before you made the assertion about its behaviour? >reduce(np.logical_or, inputs, False) >reduce(np.logical_and, inputs, True) This code consistently benchmarks 20% slower than the method I use (tested on two different machines several times). >Your fast_logic() is basically reduce(). No, it isn't. Updated benchmarks for your proposal and also for another alternative implemenation using boolean indexing at: https://github.com/gbb/numpy-fast-any-all/blob/master/BENCHMARK.md Three general points arising from this: 1 - idioms don't have test coverage Generally, by using idioms rather than functions, you risk mistyping or misusing the form of the idiom and thus introducing a bug. You also lose out on explicit testing and implicit 'real world testing' that tends to build up around library functions. 2 - idioms aren't maintained or updated (and they have a unknown shelf life) An idiom might be fast today (or not), it may be correct today, but tomorrow is unknown. A key problem is that the relative performance of the parts of a library like numpy will keep changing - sometimes substantially - and idiomatic approaches to overcome performance difficulties in the short term tend to become outdated and even harmful very quickly. As in this example, they can even be harmful from the moment they're written. Browsing a site like stackoverflow should show you both new and experienced users often taking inefficient approaches because of outdated idiomatic advice. 3 - idioms are OK, but functions are better, because implementation hiding and abstraction are good things. If you use a benchmarked/tested function which acknowledges a range of alternative implementations, you have a reasonable degree of confidence that you're getting the best performance and correct behaviour, because you can actually see the effects of the alternative implementations in benchmarks/test output. It's a lot more sensible to use a function from a publicly available library - any library - than to manually maintain a set of idioms and have to continually search your software for the idioms, benchmark them to see if they're still beneficial, and modify them when they're not. Graeme

1 0

ANN: HDF5 for Python (h5py) 2.2.0
by Andrew Collette Sept. 4, 2013

Sept. 4, 2013

Announcing HDF5 for Python (h5py) 2.2.0 ======================================= We are proud to announce that HDF5 for Python 2.2.0 is now available. Thanks to everyone who helped put together this release! The h5py package is a Pythonic interface to the HDF5 binary data format. It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want. H5py uses straightforward NumPy and Python metaphors, like dictionary and NumPy array syntax. For example, you can iterate over datasets in a file, or check out the .shape or .dtype attributes of datasets. You don't need to know anything special about HDF5 to get started. Documentation and download links are available at: http://www.h5py.org Parallel HDF5 ============= This version of h5py introduces support for MPI/Parallel HDF5, using the mpi4py package. Parallel HDF5 is the "native" method in HDF5 for sharing files and objects across multiple processes, in contrast to the "threading" package or "multiprocessing". There is a guide to using Parallel HDF5 at the h5py web site: http://www.h5py.org/docs/topics/mpi.html Other new features ================== * Support for Python 3.3 * Support for 16-bit "mini" floats * Access to the HDF5 scale-offset filter * Field names are now allowed when writing to a dataset * Region references now preserve the shape of their selections * File-resident "committed" types can be linked to datasets and attributes * Make object mtime tracking optional * A new "move" method on Group objects * Many new options for Group.copy * Access to HDF5's get_vfd_handle * Many bug fixes Acknowledgments =============== Special thanks to: * Thomas A Caswell * Konrad Hinsen * Darren Dale * Matt Zwier * Toon Verstraelen * Noel Dawe * Barry Wardel * Bradley M. Froehle * Dan Meliza * Johannes Meixner * John Benediktsson * Matthew Turk * syhw And everyone else who posted a bug report, contributed on the mailing list, or otherwise helped with this release.

1 0

Warning: 1.8.0b2 this weekend.
by Charles R Harris Sept. 4, 2013

Sept. 4, 2013

Hi All, I'd like to make a 1.8.0b2 this weekend to take care of the error cascade on 64 bit windows. If you have any fixes/bugs you would like to see in that beta now is the time to get the PRs in. Chuck

1 0

ANN: Scipy 0.13.0 beta 1 release
by Ralf Gommers Sept. 4, 2013

Sept. 4, 2013

Hi all, I'm happy to announce the availability of the first beta release of Scipy 0.13.0. Please try this beta and report any issues on the scipy-dev mailing list. Source tarballs and release notes can be found at https://sourceforge.net/projects/scipy/files/scipy/0.13.0b1/. Windows and OS X installers will follow later (we have a minor infrastructure issue to solve, and I'm at EuroScipy now). Cheers, Ralf

4 7

numpy.org down - any ETA on when it's back up?
by Ambrose LI Sept. 2, 2013

Sept. 2, 2013

Hi, numpy.org has been down for at least a few days and is currently accessible only through archive.org. So I’m just wondering if anyone with admin access to the site is working on bringing numpy.org back up. Apparently, github has recently changed their project subdomains from github.com to github.io, and (presumably as a result) numpy.github.com is no longer responding. I checked and saw that numpy.org’s cname has a ttl of just 60 seconds, so fixing it should fix the problem right away provided the fix isn’t going to cause a redirection loop. Thanks much -- cheers, -ambrose <http://gniw.ca>

2 1

Are NEPs still a thing?
by Juan Luis Cano Sept. 2, 2013

Sept. 2, 2013

The subject speaks for itself, I was diving into NumPy source code and saw some NEPs, and I wonder if there still function as a mechanism to propose changes to NumPy, and if it is documented somewhere.

2 1

Re: [Numpy-discussion] Deprecation of financial routines
by Joe Harrington Sept. 2, 2013

Sept. 2, 2013

> On 8/19/2013 2:37 AM, Juan Luis Cano wrote: >> https://github.com/numpy/numpy/issues/2880 >> >> it was suggested that we deprecate and eventually remove the financial >> functions in NumPy IDL has financial functions. Matlab has financial functions. Financial functions are something that a subset of potential customers of numerical packages look for. There is a strong tradition of people going from math/science/engineering into the finanical world, and people start on that route by playing with strategies and seeing what they can do relative to the market. To see how well they do, they need to calculate an internal rate of return, and so forth. The fin routines are tiny and don't require much maintenance once written. If we made an effort (putting up pages with examples of common financial calculations and collecting those under a topical web page, then linking to that page from various places and talking it up), I would think they could attract users looking for a free way to play with financial scenarios. Python is a perfect language for newbies to programming, which many who want to play with finances are. So, I would say we keep them. If ours are not the best, we should bring them up to snuff. I think we can endlessly debate namespace pollution. Developers hate it, but most normal users just don't mind a large number of routines in the top level, and many actively dislike the code litter of lots of long, dotted routine names. I've had a hard time getting my colleagues not to "from numpy import *". Some of the good ones even teach that in their seminars to convert IDL and IRAF users, and they chew me out when I suggest to them that it's a bad habit! A reorg that would bring us to a very heirarchical structure would be disruptive to existing code. Yet, there are maintenance and startup speed arguments in favor of a heirarchy. However we resolve it, I don't know that singling out the financial routines is the right short-term approach, and it wouldn't make much of a dent. So, I'd suggest we take it as two issues to solve separately 1) keeping or tossing the fin routines and 2) moving toward a heirarchy of routines or not. I'm -1 for deprecating the routines. I'll hold off on the second question until someone makes a clear proposal that resolves namespace pollution, and we discuss it. --jh--

2 1

Faster np.triu_indices
by Daniel Smith Sept. 2, 2013

Sept. 2, 2013

Hello all, I was noticing that `np.triu_indices` took quite awhile and discovered it creates an upper triu array and then uses `np.where`. This seems quite inefficient and I was curious if something like the following would be better: """ def fast_triu_indices(dim,k=0): tmp_range = np.arange(dim-k) rows = np.repeat(tmp_range,(tmp_range+1)[::-1]) cols = np.ones(rows.shape[0],dtype=np.int) inds = np.cumsum(tmp_range[1:][::-1]+1) np.put(cols,inds,np.arange(dim*-1+2+k,1)) cols[0] = k np.cumsum(cols,out=cols) return (rows,cols) """ This is just a first run at the function, and unfortunately does not work for k<0. However, it does return the correct results for k>=0 and is between 2-8 faster on my machine then `np.triu_indices`. Any thoughts on this? -Daniel

2 1