Mailman 3 July 2014 - NumPy-Discussion

Remove bento from numpy
by Charles R Harris July 9, 2014

July 9, 2014

Ralf likes the speed of bento, but it is not currently maintained and does not properly build numpy with all the optimizations added by Julian. I find the usual setup.py method fast enough and it has the advantage that all the numpy developers can deal with it. Thoughts? Chuck

9 23

ANN: python-blosc 1.2.7 released
by Francesc Alted July 7, 2014

July 7, 2014

============================= Announcing python-blosc 1.2.4 ============================= What is new? ============ This is a maintenance release, where included c-blosc sources have been updated to 1.4.0. This adds support for non-Intel architectures, most specially those not supporting unaligned access. For more info, you can have a look at the release notes in: https://github.com/Blosc/python-blosc/wiki/Release-notes More docs and examples are available in the documentation site: http://python-blosc.blosc.org What is it? =========== Blosc (http://www.blosc.org) is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc is the first compressor that is meant not only to reduce the size of large datasets on-disk or in-memory, but also to accelerate object manipulations that are memory-bound (http://www.blosc.org/docs/StarvingCPUs.pdf). See http://www.blosc.org/synthetic-benchmarks.html for some benchmarks on how much speed it can achieve in some datasets. Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. python-blosc (http://python-blosc.blosc.org/) is the Python wrapper for the Blosc compression library. There is also a handy command line and Python library for Blosc called Bloscpack (https://github.com/Blosc/bloscpack) that allows you to compress large binary datafiles on-disk. Installing ========== python-blosc is in PyPI repository, so installing it is easy: $ pip install -U blosc # yes, you should omit the python- prefix Download sources ================ The sources are managed through github services at: http://github.com/Blosc/python-blosc Documentation ============= There is Sphinx-based documentation site at: http://python-blosc.blosc.org/ Mailing list ============ There is an official mailing list for Blosc at: blosc(a)googlegroups.com http://groups.google.es/group/blosc Licenses ======== Both Blosc and its Python wrapper are distributed using the MIT license. See: https://github.com/Blosc/python-blosc/blob/master/LICENSES for more details. ---- **Enjoy data!** -- Francesc Alted

2 2

1.10-devel is open
by Charles R Harris July 7, 2014

July 7, 2014

Just so. The fixes for 1.9.0b1 are now in that branch ready for the next beta. Chuck

3 7

parsing dtype descriptors
by Ted Sandler July 7, 2014

July 7, 2014

Hi all, is there a spec or grammar for valid values of numpy dtype descriptor strings? I am writing code to parse ".npy" files from Java and want to be able to handle the range of ndarray descriptor strings. I came across this code: dtype = numpy.dtype(d['descr']) at line 267 in format.py: https://github.com/numpy/numpy/blob/master/numpy/lib/format.py However, I can't seem to find where it's implemented. Help is appreciated. Thanks! Ted

3 5

indexed assignment testcases
by Benjamin Root July 6, 2014

July 6, 2014

While trying to wrap my head around the issues with matplotlib's tri module and the new numpy indexing, I have made some test cases where I wonder if warnings should be issued. import numpy as np a = np.ones((10,)) all_false = np.zeros((10,), dtype=bool) a[all_false] = np.array([2.0]) # the shapes don't match here mask_in = np.array([False]*8 + [True, True]) a[mask_in] = np.array([]) # raises ValueError as expected a[mask_in] = np.array([[]]) # no exception because it is 2-D, for some reason (on master, but not release-0.9b1) a[mask_in] = np.array([2.0]) # This works and repeats 2.0 twice. I thought this wasn't supposed to happen anymore? Ben Root

4 6

Cython requirement?
by Benjamin Root July 6, 2014

July 6, 2014

When did Cython become a build requirement? I remember discussing the use of Cython a while back, and IIRC the agreement was that both the cython code and the generated C files would be included in version control so that cython wouldn't be a build requirement, only a developer requirement when modifying those files. I just did a git clean -fxd and rebase to current master, and I am getting a message indicating that I need Cython 0.19 to build numpy (I haven't updated cython in ages on this particular machine). ben@tigger:~/Programs/numpy$ python setup.py install --user Running from numpy source directory. Cythonizing sources Processing numpy/random/mtrand/mtrand.pyx Traceback (most recent call last): File "/home/ben/Programs/numpy/tools/cythonize.py", line 199, in <module> main() File "/home/ben/Programs/numpy/tools/cythonize.py", line 195, in main find_process_files(root_dir) File "/home/ben/Programs/numpy/tools/cythonize.py", line 187, in find_process_files process(cur_dir, fromfile, tofile, function, hash_db) File "/home/ben/Programs/numpy/tools/cythonize.py", line 161, in process processor_function(fromfile, tofile) File "/home/ben/Programs/numpy/tools/cythonize.py", line 59, in process_pyx raise Exception('Building %s requires Cython >= 0.19' % VENDOR) Exception: Building NumPy requires Cython >= 0.19 Traceback (most recent call last): File "setup.py", line 251, in <module> setup_package() File "setup.py", line 239, in setup_package generate_cython() File "setup.py", line 191, in generate_cython raise RuntimeError("Running cythonize failed!") RuntimeError: Running cythonize failed! Ben Root

3 3

Questions about fixes for 1.9.0rc2
by Charles R Harris July 6, 2014

July 6, 2014

Sebastian Seberg has fixed one class of test failures due to the indexing changes in numpy 1.9.0b1. There are some remaining errors, and in the case of the Matplotlib failures, they look to me to be Matplotlib bugs. The 2-d arrays that cause the error are returned by the overloaded _interpolate_single_key function in CubicTriInterpolator that is documented in the base class to return a 1-d array, whereas the actual dimensions are of the form (n, 1). The question is, what is the best work around here for these sorts errors? Can we afford to break Matplotlib and other packages on account of a bug that was previously accepted by Numpy? Perhaps a FutureWarning rather than an error would be more appropriate at this point, and that modification would be easy to make. Thoughts? Chuck

6 21

Fwd: [Python-ideas] PEP pre-draft: Support for indexing with keyword arguments
by Nathaniel Smith July 6, 2014

July 6, 2014

There's some discussion on python-ideas about making it possible for python indexing to accept kwargs, eg arr[1:2, foo=bar] Since numpy is a very heavy user of indexing which might benefit from this, I thought I should forward it here. If we have clear use cases for such a feature then that may strongly affect the discussion. I admit I can't actually think of any features this would enable for us though... -n ---------- Forwarded message ---------- From: "Stefano Borini" <stefano.borini(a)ferrara.linux.it> Date: 2 Jul 2014 00:17 Subject: [Python-ideas] PEP pre-draft: Support for indexing with keyword arguments To: "python-ideas(a)python.org" <python-ideas(a)python.org>, "Joseph Martinot-Lagarde" <joseph.martinot-lagarde(a)m4x.org> Cc: Dear all, after the first mailing list feedback, and further private discussion with Joseph Martinot-Lagarde, I drafted a first iteration of a PEP for keyword arguments in indexing. The document is available here. https://github.com/stefanoborini/pep-keyword/blob/master/PEP-XXX.txt The document is not in final form when it comes to specifications. In fact, it requires additional discussion about the best strategy to achieve the desired result. Particular attention has been devoted to present alternative implementation strategies, their pros and cons. I will examine all feedback tomorrow morning European time (in approx 10 hrs), and apply any pull requests or comments you may have. When the specification is finalized, or this community suggests that the PEP is in a form suitable for official submission despite potential open issues, I will submit it to the editor panel for further discussion, and deploy an actual implementation according to the agreed specification for a working test run. I apologize for potential mistakes in the PEP drafting and submission process, as this is my first PEP. Kind Regards, Stefano Borini _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

6 6

About the npz format
by onefire July 6, 2014

July 6, 2014

Hi all, I have been playing with the idea of using Numpy's binary format as a lightweight alternative to HDF5 (which I believe is the "right" way to do if one does not have a problem with the dependency). I am pretty happy with the npy format, but the npz format seems to be broken as far as performance is concerned (or I am missing obvious!). The following ipython session illustrates the issue: ln [1]: import numpy as np In [2]: x = np.linspace(1, 10, 50000000) In [3]: %time np.save("x.npy", x) CPU times: user 40 ms, sys: 230 ms, total: 270 ms Wall time: 488 ms In [4]: %time np.savez("x.npz", data = x) CPU times: user 657 ms, sys: 707 ms, total: 1.36 s Wall time: 7.7 s I can inspect the files to verify that they contain the same data, and I can change the example, but this seems to always hold (I am running Arch Linux, but I've done the test on other machines too): for bigger arrays, the npz format seems to add an unbelievable amount of overhead. Looking at Numpy's code, it looks like the real work is being done by Python's zipfile module, and I suspect that all the extra time is spent computing the crc32. Am I correct in my assumption (I am not familiar with zipfile's internals)? Or perhaps I am doing something really dumb and there is an easy way to speed things up? Assuming that I am correct, my next question is: why compute the crc32 at all? I mean, I know that it is part of what defines a "zip file", but is it really necessary for a npz file to be a (compliant) zip file? If, for example, I open the resulting npz file with a hex editor, and insert a bogus crc32, np.load will happily load the file anyway (Gnome's Archive Manager will do the same) To me this suggests that the fact that npz files are zip files is not that important. . Perhaps, people think that the ability to browse arrays and extract individual ones like they would do with a regular zip file is really important, but reading the little documentation that I found, I got the impression that npz files are zip files just because this was the easiest way to have multiple arrays in the same file. But my main point is: it should be fairly simple to make npz files much more efficient with simple changes like not computing checksums (or using a different algorithm like adler32). Let me know what you think about this. I've searched around the internet, and on places like Stackoverflow, it seems that the standard answer is: you are doing it wrong, forget Numpy's format and start using hdf5! Please do not give that answer. Like I said in the beginning, I am well aware of hdf5 and I use it on my "production code" (on C++). But I believe that there should be a lightweight alternative (right now, to use hdf5 I need to have installed the C library, the C++ wrappers, and the h5py library to play with the data using Python, that is a bit too heavy for my needs). I really like Numpy's format (if anything, it makes me feel better knowing that it is so easy to reverse engineer it, while the hdf5 format is very complicated), but the (apparent) poor performance of npz files if a deal breaker. Gilberto

7 22

Teaching Scipy BoF at SciPy
by Chris Barker July 5, 2014

July 5, 2014

HI Folks, I will be hosting a "Teaching the SciPy Stack" BoF at SciPy this year: https://conference.scipy.org/scipy2014/schedule/presentation/1762/ (Actually, I proposed it for the conference, but would be more than happy to have other folks join me in facilitating, hosting, etc.) I've put up a Wiki page to collect ideas for topics. Please take a look and add your $0.02: https://github.com/numpy/numpy/wiki/TeachingSciPy-BoF-at-Scipy-2014 See you there, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

3 2