Mailman 3 December 2015 - NumPy-Discussion

Re: [Numpy-discussion] FeatureRequest: support for array
by Peter Creasey 13 Dec '15

13 Dec '15

> > > > from itertools import chain > > def fromiter_awesome_edition(iterable): > > elem = next(iterable) > > dtype = whatever_numpy_does_to_infer_dtypes_from_lists(elem) > > return np.fromiter(chain([elem], iterable), dtype=dtype) > > > > I think this would be a huge win for usability. Always getting tripped up by > > the dtype requirement. I can submit a PR if people like this pattern. > > This isn't the semantics of np.array, though -- np.array will look at > the whole input and try to find a common dtype, so this can't be the > implementation for np.array(iter). E.g. try np.array([1, 1.0]) > > I can see an argument for making the dtype= argument to fromiter > optional, with a warning in the docs that it will guess based on the > first element and that you should specify it if you don't want that. > It seems potentially a bit error prone (in the sense that it might > make it easier to end up with code that works great when you test it > but then breaks later when something unexpected happens), but maybe > the usability outweighs that. I don't use fromiter myself so I don't > have a strong opinion. I’m -1 on this, from an occasional user of np.fromiter, also for the np.fromiter([1, 1.5, 2]) ambiguity reason. Pure python does a great job of preventing users from hurting themselves with limited precision arithmetic, however if their application makes them care enough about speed (to be using numpy) and memory (to be using np.fromiter), then it can almost always be assumed that the resulting dtype was important enough to be specified. P

1 0

Fast vectorized arithmetic with ~32 significant digits under Numpy
by Thomas Baruchel 13 Dec '15

13 Dec '15

>From time to time it is asked on forums how to extend precision of computation on Numpy array. The most common answer given to this question is: use the dtype=object with some arbitrary precision module like mpmath or gmpy. See http://stackoverflow.com/questions/6876377/numpy-arbitrary-precision-linear… or http://stackoverflow.com/questions/21165745/precision-loss-numpy-mpmath or http://stackoverflow.com/questions/15307589/numpy-array-with-mpz-mpfr-values While this is obviously the most relevant answer for many users because it will allow them to use Numpy arrays exactly as they would have used them with native types, the wrong thing is that from some point of view "true" vectorization will be lost. With years I got very familiar with the extended double-double type which has (for usual architectures) about 32 accurate digits with faster arithmetic than "arbitrary precision types". I even used it for research purpose in number theory and I got convinced that it is a very wonderful type as long as such precision is suitable. I often implemented it partially under Numpy, most of the time by trying to vectorize at a low-level the libqd library. But I recently thought that a very nice and portable way of implementing it under Numpy would be to use the existing layer of vectorization on floats for computing the arithmetic operations by "columns containing half of the numbers" rather than by "full numbers". As a proof of concept I wrote the following file: https://gist.github.com/baruchel/c86ed748939534d8910d I converted and vectorized the Algol 60 codes from http://szmoore.net/ipdf/documents/references/dekker1971afloating.pdf (Dekker, 1971). A test is provided at the end; for inverting 100,000 numbers, my type is about 3 or 4 times faster than GMPY and almost 50 times faster than MPmath. It should be even faster for some other operations since I had to create another np.ones array for testing this type because inversion isn't implemented here (which could of course be done). You can run this file by yourself (maybe you will have to discard mpmath or gmpy if you don't have it). I would like to discuss about the way to make available something related to that. a) Would it be relevant to include that in Numpy ? (I would think to some "contribution"-tool rather than including it in the core of Numpy because it would be painful to code all ufuncs; on the other hand I am pretty sure that many would be happy to perform several arithmetic operations by knowing that they can't use cos/sin/etc. on this type; in other words, I am not sure it would be a good idea to embed it as an every-day type but I think it would be nice to have it quickly available in some way). If you agree with that, in which way should I code it (the current link only is a "proof of concept"; I would be very happy to code it in some cleaner way)? b) Do you think such attempt should remain something external to Numpy itself and be released on my Github account without being integrated to Numpy? Best regards, -- Thomas Baruchel

11 12

Re: [Numpy-discussion] Memory mapping and NPZ files
by Sturla Molden 13 Dec '15

13 Dec '15

Mathieu Dubois <mathieu.dubois(a)icm-institute.org> wrote: > The point is precisely that, you can't do memory mapping with Npz files > (while it works with Npy files). The operating system can memory map any file. But as npz-files are compressed, you will need to uncompress the contents in your memory mapping to make sense of it. I would suggest you use PyTables instead of npz-files. It allows on the fly compression and uncompression (via blosc) and will probably do what you want. Sturla

2 1

Re: [Numpy-discussion] Memory mapping and NPZ files
by Erik Bray 12 Dec '15

12 Dec '15

On Wed, Dec 9, 2015 at 9:51 AM, Mathieu Dubois <mathieu.dubois(a)icm-institute.org> wrote: > Dear all, > > If I am correct, using mmap_mode with Npz files has no effect i.e.: > f = np.load("data.npz", mmap_mode="r") > X = f['X'] > will load all the data in memory. > > Can somebody confirm that? > > If I'm correct, the mmap_mode argument could be passed to the NpzFile class > which could in turn perform the correct operation. One way to handle that > would be to use the ZipFile.extract method to write the Npy file on disk and > then load it with numpy.load with the mmap_mode argument. Note that the user > will have to remove the file to reclaim disk space (I guess that's OK). > > One problem that could arise is that the extracted Npy file can be large > (it's the purpose of using memory mapping) and therefore it may be useful to > offer some control on where this file is extracted (for instance /tmp can be > too small to extract the file here). numpy.load could offer a new option for > that (passed to ZipFile.extract). I have struggled for a long time with a similar (albeit more obscure problem) with PyFITS / astropy.io.fits when it comes to supporting memory-mapping of compressed FITS files. For those unaware FITS is a file format used primarily in Astronomy. I have all kinds of wacky ideas for optimizing this, but at the moment when you load data from a compressed FITS file with memory-mapping enabled, obviously there's not much benefit because the contents of the file are uncompressed in memory (there is a *little* benefit in that the compressed data is mmap'd, but the compressed data is typically much smaller than the uncompressed data). Currently, in this case, I just issue a warning when the user explicitly requests mmap=True, but won't get much benefit from it. Maybe np.load could do the same, but I don't have a strong opinion about it. (I only added the warning in PyFITS because a user requested it and was kind enough to provide a patch--seemed reasonable). Erik

1 0

ANN: pyMIC v0.7 Released
by Klemm, Michael 12 Dec '15

12 Dec '15

Announcement: pyMIC v0.7 ========================= I'm happy to announce the release of pyMIC v0.7. pyMIC is a Python module to offload computation in a Python program to the Intel Xeon Phi coprocessor. It contains offloadable arrays and device management functions. It supports invocation of native kernels (C/C++, Fortran) and blends in with Numpy's array types for float, complex, and int data types. For more information and downloads please visit pyMIC's Github page: https://github.com/01org/pyMIC. You can find pyMIC's mailinglist at https://lists.01.org/mailman/listinfo/pymic. Full change log: ================= Version 0.7 ---------------------------- * Experimental support for Python 3. * 'None' arguments of kernels are converted to nullptr or NULL. * Switched to Python's distutils to build and install pyMIC. * Deprecated the build system based on Makefiles. Version 0.6 ---------------------------- * Experimental support for the Windows operating system. * Switched to Cython to generate the glue code for pyMIC. * Now using Markdown for README and CHANGELOG. * Introduced PYMIC_DEBUG=3 to trace argument passing for kernels. * Bugfix: added back the translate_device_pointer() function. * Bugfix: example SVD now respects order of the passed matrices when applying the `dgemm` routine. * Bugfix: fixed memory leak when invoking kernels. * Bugfix: fixed broken translation of fake pointers. * Refactoring: simplified bridge between pyMIC and LIBXSTREAM. Version 0.5 ---------------------------- * Introduced new kernel API that avoids insane pointer unpacking. * pyMIC now uses libxstreams as the offload back-end (https://github.com/hfp/libxstream). * Added smart pointers to make handling of fake pointers easier. Version 0.4 ---------------------------- * New low-level API to allocate, deallocate, and transfer data (see OffloadStream). * Support for in-place binary operators. * New internal design to handle offloads. Version 0.3 ---------------------------- * Improved handling of libraries and kernel invocation. * Trace collection (PYMIC_TRACE=1, PYMIC_TRACE_STACKS={none,compact,full}). * Replaced the device-centric API with a stream API. * Refactoring to better match PEP8 recommendations. * Added support for int(int64) and complex(complex128) data types. * Reworked the benchmarks and examples to fit the new API. * Bugfix: fixed syntax errors in OffloadArray. Version 0.2 ---------------------------- * Small improvements to the README files. * New example: Singular Value Decomposition. * Some documentation for the API functions. * Added a basic testsuite for unit testing (WIP). * Bugfix: benchmarks now use the latest interface. * Bugfix: numpy.ndarray does not offer an attribute 'order'. * Bugfix: number_of_devices was not visible after import. * Bugfix: member offload_array.device is now initialized. * Bugfix: use exception for errors w/ invoke_kernel & load_library. Version 0.1 ---------------------------- Initial release. Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Christian Lamprechter Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928

1 0

When to stop supporting Python 2.6?
by Charles R Harris 11 Dec '15

11 Dec '15

Hi All, Thought I would raise the topic apropos this post <http://www.curiousefficiency.org/posts/2015/04/stop-supporting-python26.html> . There is not a great advantage to dropping 2.6, OTOH, 2.7 has more features (memoryview) and we could clean up the code a bit. Along the same lines, dropping support for Python 3.2 would allow more cleanup. In fact, I'd like to get to 3.4 as soon as possible, but don't know what would be a reasonable schedule. The Python 3 series might be easier to move forward on, as I think that Python 3 is just now starting to become the dominant version in some areas. Chuck <http://www.curiousefficiency.org/posts/2015/04/stop-supporting-python26.html>

14 19

volunteers for 1.11.0 release manager?
by Charles R Harris 11 Dec '15

11 Dec '15

Hi All, Thought I'd bring this up. I think the 1.11.0 release should be fairly easy as releases go, so if someone wants to get some practice with making releases this is probably a good one to start with. We should branch 1.11.x sometime before the end of January. It is almost in branchable condition as is, IMHO, but there are some things to take care of: the pile of PRs, `__numpy_ufunc__`, and maybe a few more deprecations that should be changed to errors. Chuck

1 0

Should dtypes have an ndim attribute?
by Gerrit Holl 11 Dec '15

11 Dec '15

Hi, I have made a modest proposal in issue #6752 <https://github.com/numpy/numpy/issues/6752>. Basically, the proposal is to add an `ndim` attribute to dtypes. Currently, arrays have a shape and an ndim attribute, where ndim equals len(shape). dtype objects have a shape attribute, but no corresponding ndim. An ndim attribute would help in immediately determining whether a field in a structured dtype is multidimensional or not. Thoughts? Gerrit.

3 2

Re: [Numpy-discussion] Memory mapping and NPZ files
by Sebastian Berg 11 Dec '15

11 Dec '15

On Mi, 2015-12-09 at 15:51 +0100, Mathieu Dubois wrote: > Dear all, > > If I am correct, using mmap_mode with Npz files has no effect i.e.: > f = np.load("data.npz", mmap_mode="r") > X = f['X'] > will load all the data in memory. > My take on it is, that no, I do not want implicit extraction/copy of the file. However, npz files are not necessarily compressed, and I expect that in the non-compressed version, memory-mapping is possible on the uncompressed version. If that is possible, it would ideally work for uncompressed npz files and could raise an error which suggests to manually uncompress the file when mmap_mode is given. - Sebastian > Can somebody confirm that? > > If I'm correct, the mmap_mode argument could be passed to the NpzFile > class which could in turn perform the correct operation. One way to > handle that would be to use the ZipFile.extract method to write the > Npy file on disk and then load it with numpy.load with the mmap_mode > argument. Note that the user will have to remove the file to reclaim > disk space (I guess that's OK). > > One problem that could arise is that the extracted Npy file can be > large (it's the purpose of using memory mapping) and therefore it may > be useful to offer some control on where this file is extracted (for > instance /tmp can be too small to extract the file here). numpy.load > could offer a new option for that (passed to ZipFile.extract). > > Does it make sense? > > Thanks in advance, > Mathieu > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion

2 1

Re: [Numpy-discussion] Memory mapping and NPZ files
by Sturla Molden 11 Dec '15

11 Dec '15

Mathieu Dubois <mathieu.dubois(a)icm-institute.org> wrote: > Does it make sense? No. Memory mapping should just memory map, not do all sorts of crap. Sturla

2 1