Mailman 3 August 2010 - NumPy-Discussion

Re: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior)
by Nils Becker 30 Aug '10

30 Aug '10

> I think (a corrected) density histogram is core functionality for > unequal bin lengths. > > The graph with raw count in the case of unequal bin sizes would be > quite misleading when plotted and interpreted on the real line and not > on discrete points (shaded areas instead of vertical lines). And as > the origin of this thread showed, it's not trivial to figure out what > the correct normalization is. > So, I think, if we drop the density normalization, we just need a new > function that does it. > > My 2c, > > Josef (Not a dev, but ) I agree with Josef. Throwing out normalization completely would in addition have to destroy backwards compatibility. It's another question whether the frequency ( not density! ) normalization is so simple that it is better left out, since everyone can be expected to be able to do a raw histogram and divide by the number of data points without a headache. Nils

1 0

Re: [Numpy-discussion] numpy histogram normed=True (bug / confusing behavior)
by Nils Becker 29 Aug '10

29 Aug '10

Hi again, first a correction: I posted > I believe np.histogram(data, bins, normed=True) effectively does : >>> np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]). >>> >>> However, it _should_ do >>> np.histogram(data, bins, normed=False) / bins_widths but there is a normalization missing; it should read I believe np.histogram(data, bins, normed=True) effectively does np.histogram(data, bins, normed=False) / (bins[-1] - bins[0]) / data.sum() However, it _should_ do np.histogram(data, bins, normed=False) / bins_widths / data.sum() Bruce Southey replied: > As I recall, there as issues with this aspect. > Please search the discussion regarding histogram especially David > Huard's reply in this thread: > http://thread.gmane.org/gmane.comp.python.numeric.general/22445 I think this discussion pertains to a switch in calling conventions which happened at the time. The last reply of D. Huard (to me) seems to say that they did not fix anything in the _old_ semantics, but that the new semantics is expected to work properly. I tried with an infinite bin: counts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf]) counts array([1,3]) ncounts, dmy = np.histogram([1,2,3,4], [0.5,1.5,np.inf], normed=1) ncounts array([0.,0.]) this also does not make a lot of sense to me. A better result would be array([0.25, 0.]), since 25% of the points fall in the first bin; 75% fall in the second but are spread out over an infinite interval, giving 0. This is what my second proposal would give. I cannot find anything wrong with it so far... Cheers, Nils

7 8

[ANN] Reminder: Autumn School "Advanced Scientific Programming in Python" in Trento, Italy
by Tiziano Zito 28 Aug '10

28 Aug '10

Reminder: Application deadline is August 31st, 2010! =================================================== Advanced Scientific Programming in Python ========================================= an Autumn School by the G-Node, the Center for Mind/Brain Sciences and the Fondazione Bruno Kessler Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists actually use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques with theoretical lectures and practical exercises tailored to the needs of a programming scientist. New skills will be tested in a real programming project: we will team up to develop an entertaining scientific computer game. We'll use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. Clean language design and easy extensibility are driving Python to become a standard tool for scientific computing. Some of the most useful open source libraries for scientific computing and visualization will be presented. This school is targeted at Post-docs and PhD students from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or Mathematica is absolutely required. A basic knowledge of the Python language is assumed. Participants without prior experience with Python should work through the proposed introductory materials. Date and Location ================= October 4th—8th, 2010. Trento, Italy. Preliminary Program =================== Day 0 (Mon Oct 4) — Software Carpentry & Advanced Python • Documenting code and using version control • Object-oriented programming, design patterns, and agile programming • Exception handling, lambdas, decorators, context managers, metaclasses Day 1 (Tue Oct 5) — Software Carpentry • Test-driven development, unit testing & Quality Assurance • Debugging, profiling and benchmarking techniques • Data serialization: from pickle to databases Day 2 (Wed Oct 6) — Scientific Tools for Python • Advanced NumPy • The Quest for Speed (intro): Interfacing to C • Programming project Day 3 (Thu Oct 7) — The Quest for Speed • Writing parallel applications in Python • When parallelization does not help: the starving CPUs problem • Programming project Day 4 (Fri Oct 8) — Practical Software Development • Efficient programming in teams • Programming project • The Pac-Man Tournament Every evening we will have the tutors' consultation hour: Tutors will answer your questions and give suggestions for your own projects Applications ============ You can apply on-line at http://www.g-node.org/python-autumnschool Applications must be submitted before August 31st, 2010. Notifications of acceptance will be sent by September 4th, 2010. No fee is charged but participants should take care of travel, living, and accommodation expenses. Candidates will be selected on the basis of their profile. Places are limited: acceptance rate in past editions was around 30%. Prerequisites ============= You are supposed to know the basics of Python to participate in the lectures! Look on the website for a list of introductory material. Faculty ======= • Francesc Alted, author of PyTables, Castelló de la Plana, Spain • Pietro Berkes, Volen Center for Complex Systems, Brandeis University, USA • Valentin Haenel, Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany • Zbigniew Jędrzejewski-Szmek, Faculty of Physics, University of Warsaw, Poland • Eilif Muller, The Blue Brain Project, Ecole Polytechnique Fédérale de Lausanne, Switzerland • Emanuele Olivetti, NeuroInformatics Laboratory, Fondazione Bruno Kessler and University of Trento, Italy • Rike-Benjamin Schuppner, Bernstein Center for Computational Neuroscience Berlin, Germany • Bartosz Teleńczuk, Institute for Theoretical Biology, Humboldt-Universität zu Berlin, Germany • Bastian Venthur, Berlin Institute of Technology and Bernstein Focus: Neurotechnology, Germany • Stéfan van der Walt, Applied Mathematics, University of Stellenbosch, South Africa • Tiziano Zito, Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany Organized by Paolo Avesani for the Center for Mind/Brain Sciences and the Fondazione Bruno Kessler, and by Zbigniew Jędrzejewski-Szmek and Tiziano Zito for the German Neuroinformatics Node of the INCF. Website: http://www.g-node.org/python-autumnschool Contact: python-info(a)g-node.org

1 0

bincount() optional length argument
by David Warde-Farley 27 Aug '10

27 Aug '10

I've been using bincount() in situations where I always want the count vector to be a certain fixed length, even if the upper bins are 0. e.g. I want the counts of 0 through 9, and if there are no 9's in the vector I'm counting, I'd still like it to be at least length 10. It's kind of a pain to have to check for this case and concatenate 0's, nevermind an extra array allocation. How would people feel about an optional argument to support this behaviour? I'd be happy with either a "minlength" or an "exactly this length" with values above this range being ignored, but it seems like the latter might be useful in more cases. What are other people's thoughts on this? bincount is in C so it'll be a slight pain to do, however I'd be willing to do it if the patch has a reasonable chance of being accepted. David

2 2

[ANN] carray 0.2: an in-memory compressed data container
by Francesc Alted 27 Aug '10

27 Aug '10

===================== Announcing carray 0.2 ===================== What it is ========== carray is a container for numerical data that can be compressed in-memory. The compresion process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. Having data compressed in-memory can reduce the stress of the memory subsystem. The net result is that carray operations can be faster than using a traditional ndarray object from NumPy. What's new ========== Two new `__iter__()` and `iter(start, stop, step)` iterators that allows to perform potentially complex operations much faster than using plain ndarrays. For example:: In [3]: a = np.arange(1e6) In [4]: time sum((v for v in a if v < 4)) CPU times: user 6.51 s, sys: 0.00 s, total: 6.51 s Wall time: 6.52 s Out[5]: 6.0 In [6]: b = ca.carray(a) In [7]: time sum((v for v in b if v < 4)) CPU times: user 0.73 s, sys: 0.04 s, total: 0.78 s Wall time: 0.75 s # 8.7x faster than ndarray Out[8]: 6.0 The `iter(start, stop, step)` iterator also allows to select slices specified by the `start`, `stop` and `step` parameters. Example:: In [9]: time sum((v for v in a[2::3] if v < 10)) CPU times: user 2.18 s, sys: 0.00 s, total: 2.18 s Wall time: 2.19 s Out[10]: 15.0 In [11]: time sum((v for v in b.iter(start=2, step=3) if v < 10)) CPU times: user 0.26 s, sys: 0.03 s, total: 0.30 s Wall time: 0.30 s # 7.3x faster than ndarray Out[12]: 15.0 The main advantage of these iterators is that you can use them in generators and hence, you don't need to waste memory for creating temporaries, which can be important when dealing with large arrays. For more info, see the new ``Using iterators`` section in USAGE.txt. Resources ========= Visit the main carray site repository at: http://github.com/FrancescAlted/carray You can download a source package from: http://github.com/FrancescAlted/carray/downloads Short tutorial: http://github.com/FrancescAlted/carray/blob/master/USAGE.txt Home of Blosc compressor: http://blosc.pytables.org Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- Enjoy! -- Francesc Alted

1 0

questions about a "complicated" user-defined dtype and the ufunc API
by Nathaniel Smith 27 Aug '10

27 Aug '10

I'm experimenting with a user-defined "enumeration" dtype -- where the underlying array holds a set of integers, but they (mostly) appear to the user as strings. (This would be potentially useful for representing categorical data, modeling hdf5 enumerations, etc.) So for each set of enumerated values, I have one Python-level object that stores the mapping between strings and integers (an instance of the 'Enum' class), and a few Python-level objects that represent each of the enumerated values (instances of the 'EnumValue' class). To map this into numpy, I've defined and registered a single custom dtype with 'EnumValue' as its typeobj, and then when I want to create an array of enumerations I make a copy of this registered dtype (with PyArray_DescrNewFromType) and stash a reference to the appropriate Enum object in its 'metadata' dictionary. Question 1: Is this overall approach -- of only calling PyArray_RegisterDataType once, and then sharing the resulting typenum among all my different dtype instances -- correct? It doesn't seem reasonable to register a new dtype for every different set of enumerations, because AFAICT this would create a memory leak (since you can't "unregister" a dtype). Anyway, that seems to be working okay, but then I wanted to teach "==" about my new dtype, so that I can compare to strings and such. So, I need to register my comparison function with the "np.equal" ufunc. Question 2: Am I missing something, or does the ufunc API make this impossible? The problem is that a "PyUFuncGenericFunction" doesn't have any way to find the dtypes of the arrays that it's working on. All of the PyArray_ArrFuncs functions take a pointer to the underlying ndarray as an argument, so that when working with a string or void array, you can find the actual dtype and figure out (e.g.) the size of the objects involved. But ufunc inner loops don't get that, so I guess it's just impossible to define a ufunc over variable-sized data, or data that you need to be able to see the dtype metadata to interpret? This seems easy enough to fix, and would probably allow the removal of a big pile of code in arrayobject.c that does special-case handling for "==" on strings and void arrays. (Another side-effect of the current special-case approach is that "==" and "np.equal" behave differently on arrays of strings.) But is there another option? Anyway, thanks for your help! I'll attach the code in case it helps (and because I'm not too confident I'm getting the details right!). Sample usage: >>> from npenum import *; import numpy as np >>> e = Enum(["low", "medium", "high"]) >>> a = np.array(["low", "low", "high", "medium"], dtype=e) >>> a array([low, low, high, medium], dtype=EnumValue) >>> a.view(Enum.get(a).inttype) array([0, 0, 2, 1], dtype=uint32) Thanks, -- Nathaniel

3 3

ANN: NumPy 1.5.0 release candidate 1
by Ralf Gommers 26 Aug '10

26 Aug '10

I am pleased to announce the availability of the first release candidate of NumPy 1.5.0. This will be the first NumPy release to include support for Python 3, as well as for Python 2.7. Please try this RC and report any problems on the NumPy mailing list. Especially with Python 3 testing will be very useful. On Linux and OS X building from source should be straightforward, for Windows a binary installer is provided. Binaries, sources and release notes can be found at https://sourceforge.net/projects/numpy/files/ Enjoy, Ralf

3 3

BUG: NumPy exposes the wrong include directory
by Stéfan van der Walt 26 Aug '10

26 Aug '10

Hi all, Under Ubuntu, the NumPy headers are dumped under /usr/include/python2.6. This is not the location that should be exposed when another copy of NumPy is installed. The attached patch lets from numpy.distutils.system_info import get_info print get_info['numpy'] return the correct path. However, it's a quick hack, and I'd like someone familiar with numpy.distutils to have a look and suggest a proper fix. Regards Stéfan

3 4

bug in dtype.__eq__ method ?
by Tiziano Zito 26 Aug '10

26 Aug '10

hi all, we just noticed the following weird thing: $ python Python 2.6.6rc2 (r266rc2:84114, Aug 18 2010, 07:33:44) [GCC 4.4.5 20100816 (prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.version.version '2.0.0.dev8469' >>> numpy.dtype('float64') == None True >>> numpy.dtype('float32') == None False is this a bug in the dtype.__eq__ method? ciao, tiziano

3 4

Polynomials
by Charles R Harris 26 Aug '10

26 Aug '10

Hi All, I've gone ahead and implemented the Laguerre and Hermite (H and He) polynomials, but at this point I'm loath to add them to numpy as the polynomial space is getting crowded. Scipy looks like a better spot to put these but there is already the orthogonal module there. OTOH, the orthogonal module uses poly1d for the representation and that is numerical poison. The versions I have use the direct series representations in the given basis for +, -, x, /, integration, and differentiation as well as doing direct conversions between domains and basis without going through ordinary power series, so that is a plus. They are also usable with mpmath for extended precision but at the price of sticking to python for the implementation, so they could be speeded up by restricting to doubles and complex. They don't supply the points and weights for Gauss integration, I tend to think those things belong under integration, but they could be included. Anyway, I'm looking to restart the polynomial discussion before doing something drastic so would appreciate any input. Chuck

3 4