Mailman 3 May 2013 - NumPy-Discussion

NumPy sprints at Scipy 2013, Austin: call for topics and hands to help
by David Cournapeau 26 May '13

26 May '13

Hi there, I agreed to help organising NumPy sprints during the scipy 2013 conference in Austin. As some of you may know, Stéfan and me will present a tutorial on NumPy C code, so if we do our job correctly, we should have a few new people ready to help out during the sprints. It would be good to: - have some focus topics for improvements - know who is going to be there at the sprint to work on things and/or help newcomers Things I'd like to work on myself is looking into splitting things from multiarray, think about a better internal API for dtype registration/hooks (with the goal to remove any date dtype hardcoding in both multiarray and ufunc machinery), but I am sure others have more interesting ideas :) thanks, David

4 6

ANN: New WinPython with Python 2.7.5 and 3.3.2 (32/64bit)
by Pierre Raybaut 25 May '13

25 May '13

Hi all, I am pleased to announce that four new versions of WinPython have been released yesterday with Python 2.7.5 and 3.3.2, 32 and 64 bits. Many packages have been added or upgraded. Special thanks to Christoph Gohlke for building most of the binary packages bundled in WinPython. WinPython is a free open-source portable distribution of Python for Windows, designed for scientists. It is a full-featured (see http://code.google.com/p/winpython/wiki/PackageIndex) Python-based scientific environment: * Designed for scientists (thanks to the integrated libraries NumPy, SciPy, Matplotlib, guiqwt, etc.: * Regular *scientific users*: interactive data processing and visualization using Python with Spyder * *Advanced scientific users and software developers*: Python applications development with Spyder, version control with Mercurial and other development tools (like gettext) * *Portable*: preconfigured, it should run out of the box on any machine under Windows (without any installation requirements) and the folder containing WinPython can be moved to any location (local, network or removable drive) * *Flexible*: one can install (or should I write "use" as it's portable) as many WinPython versions as necessary (like isolated and self-consistent environments), even if those versions are running different versions of Python (2.7, 3.x in the near future) or different architectures (32bit or 64bit) on the same machine * *Customizable*: using the integrated package manager (wppm, as WinPython Package Manager), it's possible to install, uninstall or upgrade Python packages (see http://code.google.com/p/winpython/wiki/WPPM for more details on supported package formats). *WinPython is not an attempt to replace Python(x,y)*, this is just something different (see http://code.google.com/p/winpython/wiki/Roadmap): more flexible, easier to maintain, movable and less invasive for the OS, but certainly less user-friendly, with less packages/contents and without any integration to Windows explorer [*]. [*] Actually there is an optional integration into Windows explorer, providing the same features as the official Python installer regarding file associations and context menu entry (this option may be activated through the WinPython Control Panel), and adding shortcuts to Windows Start menu. Enjoy! -Pierre

1 0

ANN: python-blosc 1.1 (final) released
by Francesc Alted 24 May '13

24 May '13

=========================== Announcing python-blosc 1.1 =========================== What is it? =========== python-blosc (http://blosc.pydata.org/) is a Python wrapper for the Blosc compression library. Blosc (http://blosc.org) is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Whether this is achieved or not depends of the data compressibility, the number of cores in the system, and other factors. See a series of benchmarks conducted for many different systems: http://blosc.org/trac/wiki/SyntheticBenchmarks. Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. There is also a handy command line for Blosc called Bloscpack (https://github.com/esc/bloscpack) that allows you to compress large binary datafiles on-disk. Although the format for Bloscpack has not stabilized yet, it allows you to effectively use Blosc from your favorite shell. What is new? ============ - Added new `compress_ptr` and `decompress_ptr` functions that allows to compress and decompress from/to a data pointer, avoiding an itermediate copy for maximum speed. Be careful, as these are low level calls, and user must make sure that the pointer data area is safe. - Since Blosc (the C library) already supports to be installed as an standalone library (via cmake), it is also possible to link python-blosc against a system Blosc library. - The Python calls to Blosc are now thread-safe (another consequence of recent Blosc library supporting this at C level). - Many checks on types and ranges of values have been added. Most of the calls will now complain when passed the wrong values. - Docstrings are much improved. Also, Sphinx-based docs are available now. Many thanks to Valentin Hänel for his impressive work for this release. For more info, you can see the release notes in: https://github.com/FrancescAlted/python-blosc/wiki/Release-notes More docs and examples are available in the documentation site: http://blosc.pydata.org Installing ========== python-blosc is in PyPI repository, so installing it is easy: $ pip install -U blosc # yes, you should omit the python- prefix Download sources ================ The sources are managed through github services at: http://github.com/FrancescAlted/python-blosc Documentation ============= There is Sphinx-based documentation site at: http://blosc.pydata.org/ Mailing list ============ There is an official mailing list for Blosc at: blosc(a)googlegroups.com http://groups.google.es/group/blosc Licenses ======== Both Blosc and its Python wrapper are distributed using the MIT license. See: https://github.com/FrancescAlted/python-blosc/blob/master/LICENSES for more details. Enjoy! -- Francesc Alted

1 0

stdint.h
by Charles R Harris 23 May '13

23 May '13

Hi all, Just seeking some info here. The file stdint.h was part of the C99 standard and has types for integers of specified width and thus could be used to simplify some of the numpy configuration. I'm curious as to which compilers might be a problem and what folks think of that possibility. Chuck

3 2

faster (selection based) median, 2013 edition
by Julian Taylor 23 May '13

23 May '13

hi, once again I want to bring up the median algorithm which is implemented in terms of sorting in numpy. median (and percentile and a couple more functions) can be more efficiently implemented in terms of a selection algorithm. The complexity can them be linear instead of linearithmic. I found numerous discussions of this in the list archives [1, 2, 3] but I did not find why those attempts failed, the threads all just seemed to stop. Did the previous attempts fail due to lack of time or was there a fundamental reason blocking this change? In the hope of the former, I went ahead and implemented a prototype of a partition function (similar to [3] but only one argument) and implemented median in terms of it. partition not like C++ partition, its equivalent to nth_element in C++, maybe its better to name it nth_element? The code is available here: https://github.com/juliantaylor/numpy/tree/select-median the partition interface is: ndarray.partition(kth, axis=-1) kth is an integer The array is transformed so the k-th element of the array is in its final sorted order, all below are smaller all above are greater, but the ordering is undefined Example: In [1]: d = np.arange(10); np.random.shuffle(d) In [2]: d Out[2]: array([1, 7, 0, 2, 5, 6, 8, 9, 3, 4]) In [3]: np.partition(d, 3) Out[3]: array([0, 1, 2, 3, 4, 6, 8, 9, 7, 5]) In [4]: _[3] == 3 Out[5]: True the performance of median improves as expected: old vs new, 5000, uniform shuffled, out of place: 100us vs 40us old vs new, 50000, uniform shuffled, out of place: 1.12ms vs 0.265ms old vs new, 500000, uniform shuffled, out of place: 14ms vs 2.81ms The implementation is very much still a prototype, apartition is not exposed (and only implemented as a quicksort) and there is only one algorithm (quickselect). One could still add median of medians for better worst case performance. If no blockers appear I want to fix this up and file a pull request to have this in numpy 1.8. Guidance on details of implementation in numpys C api is highly appreciated, its the first time I'm dealing with it. Cheers, Julian Taylor [1] http://thread.gmane.org/gmane.comp.python.numeric.general/50931/focus=50941 [2] http://thread.gmane.org/gmane.comp.python.numeric.general/32507/focus=41716 [3] http://thread.gmane.org/gmane.comp.python.numeric.general/32341/focus=32348

1 1

np.dot and 'out' bug
by Nicolas Rougier 23 May '13

23 May '13

Hi, From the dot documentation, I tried something simple: a = np.array([[1, 2], [3, 4]]) b = np.array([[1, 2], [3, 4]]) np.dot(a, b) -> array([[ 7, 10], [15, 22]]) And I got expected result but if I use either a or b as output, results are wrong (and nothing in the dot documentation prevents me from doing this): a = np.array([[1, 2], [3, 4]]) b = np.array([[1, 2], [3, 4]]) np.dot(a,b,out=a) -> array([[ 6, 20], [15, 46]]) a = np.array([[1, 2], [3, 4]]) b = np.array([[1, 2], [3, 4]]) np.dot(a,b,out=b) -> array([[ 6, 10], [30, 46]]) Can anyone confirm this behavior ? (tested using numpy 1.7.1) Nicolas

7 10

pyhdf packaging
by Andreas Hilboll 23 May '13

23 May '13

Hi, I would like to package pyhdf for Ubuntu and make the package publicly available. Since the license is not totally clear to me (I cannot find any information in the sources, and the cheeseshop says "public", which doesn't mean anything to me), I tried to contact the maintainer, Andre Gosselin, however the email bounces, so I guess he's gone. Can anyone point me to how to proceed from here? Cheers, Andreas.

2 2

Possible conversion bug with record array
by Nicolas Rougier 22 May '13

22 May '13

Hi all, I got a weird output from the following script: import numpy as np U = np.zeros(1, dtype=[('x', np.float32, (4,4))]) U[0] = np.eye(4) print U[0] # output: ([[0.0, 1.875, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 1.875], [0.0, 0.0, 0.0, 0.0]],) U[0] = np.eye(4, dtype=np.float32) print U[0] # output: ([[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0], [0.0, 0.0, 1.0, 0.0], [0.0, 0.0, 0.0, 1.0]],) The first output is obviously wrong. Can anyone confirm ? (using numpy 1.7.1 on osx 10.8.3) Nicolas

3 4

another indexing question
by Neal Becker 20 May '13

20 May '13

I have a system that transmits signals for an alphabet of M symbols over and additive Gaussian noise channel. The receiver has a 1-d array of complex received values. I'd like to find the means of the received values according to the symbol that was transmitted. So transmit symbol indexes might be: x = [0, 1, 2, 1, 3, ...] and receive output might be: y = [(1+1j), (1-1j), ...] Suppose the alphabet was M=4. Then I'd like to get an array of means m[0...3] that correspond to the values of y for each of the corresponding values of x. I can't think of a better way than manually using loops. Any tricks here?

3 2

Equvalent function for Ceil() and Floor()
by Bakhtiyor Zokhidov 20 May '13

20 May '13

Hello, I am using ceil() and floor() function to get upper and lower value of some numbers. Let's say: import math x1 = 0.35 y1 = 4.46 >>> math.ceil(x1) 1.0 >>> math.floor(y1) 4.0 The problem is that If I want to get upper and lower values for the certain step, for example, step = 0.25, ceil() function should give: new_ceil(x1, step) => 0.5 new_floor(y1, step) => 4.25 Because, the step is 0.25 Question: How I can I achieve those results by using ceil() and floor() function, or Is there any equvalent function for that? -- Bakhti

4 7