Mailman 3 January 2014 - NumPy-Discussion

ANN: python-blosc 1.2.0 released
by Francesc Alted 26 Jan '14

26 Jan '14

============================= Announcing python-blosc 1.2.0 ============================= What is new? ============ This release adds support for the multiple compressors added in Blosc 1.3 series. The new compressors are: * lz4 (http://code.google.com/p/lz4/): A very fast compressor/decompressor. Could be thought as a replacement of the original BloscLZ, but it can behave better is some scenarios. * lz4hc (http://code.google.com/p/lz4/): This is a variation of LZ4 that achieves much better compression ratio at the cost of being much slower for compressing. Decompression speed is unaffected (and sometimes better than when using LZ4 itself!), so this is very good for read-only datasets. * snappy (http://code.google.com/p/snappy/): A very fast compressor/decompressor. Could be thought as a replacement of the original BloscLZ, but it can behave better is some scenarios. * zlib (http://www.zlib.net/): This is a classic. It achieves very good compression ratios, at the cost of speed. However, decompression speed is still pretty good, so it is a good candidate for read-only datasets. Selecting the compressor is just a matter of specifying the new `cname` parameter in compression functions. For example:: in = numpy.arange(N, dtype=numpy.int64) out = blosc.pack_array(in, cname="lz4") Just to have an overview of the differences between the different compressors in new Blosc, here it is the output of the included compress_ptr.py benchmark: https://github.com/ContinuumIO/python-blosc/blob/master/bench/compress_ptr.… that compresses/decompresses NumPy arrays with different data distributions:: Creating different NumPy arrays with 10**7 int64/float64 elements: *** np.copy() **** Time for memcpy(): 0.030 s *** the arange linear distribution *** *** blosclz *** Time for comp/decomp: 0.013/0.022 s. Compr ratio: 136.83 *** lz4 *** Time for comp/decomp: 0.009/0.031 s. Compr ratio: 137.19 *** lz4hc *** Time for comp/decomp: 0.103/0.021 s. Compr ratio: 165.12 *** snappy *** Time for comp/decomp: 0.012/0.045 s. Compr ratio: 20.38 *** zlib *** Time for comp/decomp: 0.243/0.056 s. Compr ratio: 407.60 *** the linspace linear distribution *** *** blosclz *** Time for comp/decomp: 0.031/0.036 s. Compr ratio: 14.27 *** lz4 *** Time for comp/decomp: 0.016/0.033 s. Compr ratio: 19.68 *** lz4hc *** Time for comp/decomp: 0.188/0.020 s. Compr ratio: 78.21 *** snappy *** Time for comp/decomp: 0.020/0.032 s. Compr ratio: 11.72 *** zlib *** Time for comp/decomp: 0.290/0.048 s. Compr ratio: 90.90 *** the random distribution *** *** blosclz *** Time for comp/decomp: 0.083/0.025 s. Compr ratio: 4.35 *** lz4 *** Time for comp/decomp: 0.022/0.034 s. Compr ratio: 4.65 *** lz4hc *** Time for comp/decomp: 1.803/0.039 s. Compr ratio: 5.61 *** snappy *** Time for comp/decomp: 0.028/0.023 s. Compr ratio: 4.48 *** zlib *** Time for comp/decomp: 3.146/0.073 s. Compr ratio: 6.17 That means that Blosc in combination with LZ4 can compress at speeds that can be up to 3x faster than a pure memcpy operation. Decompression is a bit slower (but still in the same order than memcpy()) probably because writing to memory is slower than reading. This was using an Intel Core i5-3380M CPU @ 2.90GHz, runnng Python 3.3 and Linux 3.7.10, but YMMV (and will vary!). For more info, you can have a look at the release notes in: https://github.com/ContinuumIO/python-blosc/wiki/Release-notes More docs and examples are available in the documentation site: http://blosc.pydata.org What is it? =========== python-blosc (http://blosc.pydata.org/) is a Python wrapper for the Blosc compression library. Blosc (http://blosc.org) is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Whether this is achieved or not depends of the data compressibility, the number of cores in the system, and other factors. See a series of benchmarks conducted for many different systems: http://blosc.org/trac/wiki/SyntheticBenchmarks. Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. There is also a handy command line for Blosc called Bloscpack (https://github.com/esc/bloscpack) that allows you to compress large binary datafiles on-disk. Although the format for Bloscpack has not stabilized yet, it allows you to effectively use Blosc from your favorite shell. Installing ========== python-blosc is in PyPI repository, so installing it is easy: $ pip install -U blosc # yes, you should omit the python- prefix Download sources ================ The sources are managed through github services at: http://github.com/ContinuumIO/python-blosc Documentation ============= There is Sphinx-based documentation site at: http://blosc.pydata.org/ Mailing list ============ There is an official mailing list for Blosc at: blosc(a)googlegroups.com http://groups.google.es/group/blosc Licenses ======== Both Blosc and its Python wrapper are distributed using the MIT license. See: https://github.com/ContinuumIO/python-blosc/blob/master/LICENSES for more details. -- Francesc Alted Continuum Analytics, Inc. -- Francesc Alted

1 0

Text array dtype for numpy
by Oscar Benjamin 25 Jan '14

25 Jan '14

There have been a few threads discussing the problems of how to do text with numpy arrays in Python 3. To make a slightly more concrete proposal, I've implemented a pure Python ndarray subclass that I believe can consistently handle text/bytes in Python 3. It is intended to be an illustration since I think that the real solution is a new dtype rather than an array subclass (so that it can be used in e.g. record arrays). The idea is that the array has an encoding. It stores strings as bytes. The bytes are encoded/decoded on insertion/access. Methods accessing the binary content of the array will see the encoded bytes. Methods accessing the elements of the array will see unicode strings. I believe it would not be as hard to implement as the proposals for variable length string arrays. The one caveat is that it will strip null characters from the end of any string. I'm not 100% that the byte stripping encoding function will always work but it will for all the encodings I know and it seems to work with all the encodings that Python has. The code is inline below and attached (in case there are encoding problems with this message!): Oscar #!/usr/bin/env python3 from numpy import ndarray, array class textarray(ndarray): '''ndarray for holding encoded text. This is for demonstration purposes only. The real proposal is to specify the encoding as a dtype rather than a subclass. Only works as a 1-d array. >>> a = textarray(['qwert', 'zxcvb'], encoding='ascii') >>> a textarray(['qwert', 'zxcvb'], dtype='|S5:ascii') >>> a[0] 'qwert' >>> a.tostring() b'qwertzxcvb' >>> a[0] = 'qwe' # shorter string >>> a[0] 'qwe' >>> a.tostring() b'qwe\\x00\\x00zxcvb' >>> a[0] = 'qwertyuiop' # longer string Traceback (most recent call last): ... ValueError: Encoded bytes don't fit >>> b = textarray(['Õscar', 'qwe'], encoding='utf-8') >>> b textarray(['Õscar', 'qwe'], dtype='|S6:utf-8') >>> b[0] 'Õscar' >>> b[0].encode('utf-8') b'\\xc3\\x95scar' >>> b.tostring() b'\\xc3\\x95scarqwe\\x00\\x00\\x00' >>> c = textarray(['qwe'], encoding='utf-32-le') >>> c textarray(['qwe'], dtype='|S12:utf-32-le') ''' def __new__(cls, strings, encoding='utf-8'): bytestrings = [s.encode(encoding) for s in strings] a = array(bytestrings, dtype='S').view(textarray) a.encoding = encoding return a def __repr__(self): slist = ', '.join(repr(self[n]) for n in range(len(self))) return "textarray([%s], \n dtype='|S%d:%s')"\ % (slist, self.itemsize, self.encoding) def __getitem__(self, index): bstring = ndarray.__getitem__(self, index) return self._decode(bstring) def __setitem__(self, index, string): bstring = string.encode(self.encoding) if len(bstring) > self.itemsize: raise ValueError("Encoded bytes don't fit") ndarray.__setitem__(self, index, bstring) def _decode(self, b): b = b + b'\0' * (4 - len(b) % 4) s = b.decode(self.encoding) for n, c in enumerate(reversed(s)): if c != '\0': return s[:len(s)-n] return s if __name__ == "__main__": import doctest doctest.testmod()

3 3

Comparison changes
by Sebastian Berg 25 Jan '14

25 Jan '14

Hi all, in https://github.com/numpy/numpy/pull/3514 I proposed some changes to the comparison operators. This includes: 1. Comparison with None will broadcast in the future, so that `arr == None` will actually compare all elements to None. (A FutureWarning for now) 2. I added that == and != will give FutureWarning when an error was raised. In the future they should not silence these errors anymore. (For example shape mismatches) 3. We used to use PyObject_RichCompareBool for equality which includes an identity check. I propose to not do that identity check since we have elementwise equality (returning an object array for objects would be nice in some ways, but I think that is only an option for a dedicated function). The reason is that for example >>> a = np.array([np.array([1, 2, 3]), 1]) >>> b = np.array([np.array([1, 2, 3]), 1]) >>> a == b will happen to work if it happens to be that `a[0] is b[0]`. This currently has no deprecation, since the logic is in the inner loop and I am not sure if it is easy to add well there. Are there objections/comments to these changes? Regards, Sebastian

3 3

Re: [Numpy-discussion] Numpy arrays vs typed memoryviews
by Sturla Molden 25 Jan '14

25 Jan '14

I think I have said this before, but its worth a repeat: Pickle (including cPickle) is a slow hog! That might not be the overhead you see, you just haven't noticed it yet. I saw this some years ago when I worked on shared memory arrays for Numpy (cf. my account on Github). Shared memory really did not help to speed up the IPC, because the entire overhead was dominated by pickle. (Shared memory is a fine way of saving RAM, though.) multiprocessing.Queue will use pickle for serialization, and is therefore not the right tool for numerical parallel computing with Cython or NumPy. In order to use multiprocessing efficiently with NumPy, we need a new Queue type that knows about NumPy arrays (and/or Cython memoryviews), and treat them as special cases. Getting rid of pickle altogether is the important part, not facilitating its use even further. It is easy to make a Queue type for Cython or NumPy arrays using a duplex pipe and couple of mutexes. Or you can use shared memory as ringbuffer and atomic compare-and-swap on the first bytes as spinlocks. It is not difficult to get the overhead of queuing arrays down to little more than a memcpy. I've been wanting to do this for a while, so maybe it is time to start a new toy project :) Sturla Neal Hughes <hughes.neal(a)gmail.com> wrote: > I like Cython a lot. My only complaint is that I have to keep switching > between the numpy array support and typed memory views. Both have there > advantages but neither can do every thing I need. > > Memoryviews have the clean syntax and seem to work better in cdef classes > and in inline functions. > > But Memoryviews can't be pickled and so can't be passed between > processes. Also there seems to be a high overhead on converting between > memory views and python numpy arrays. Where this overhead is a problem, > or where i need to use pythons multiprocessing module I tend to switch to numpy arrays. > > If memory views could be converted into python fast, and pickled I would > have no need for the old numpy array support. > > Wondering if these problems will ever be addressed, or if I am missing > something completely. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "cython-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to cython-users+unsubscribe(a)googlegroups.com. > For more options, visit <a > href="https://groups.google.com/groups/opt_out.">https://groups.google.com/groups/opt_out.</a> > > ------=_Part_1342_18667054.1390644115997 > Content-Type: text/html; charset=UTF-8 > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr">I like Cython a lot. My only complaint is that I have to k= > eep switching between the numpy array support and typed memory views. Both = > have there advantages but neither can do every thing I need.<div><br></div>= > <div>Memoryviews have the clean syntax and seem to work better in cdef clas= > ses and in inline functions.</div><div><br></div><div>But Memoryviews can't= > be pickled and so can't be passed between processes. Also there seems to b= > e a high overhead on converting between memory views and python numpy array= > s. Where this overhead is a problem, or where i need to use pythons multipr= > ocessing module I tend to switch to numpy arrays.</div><div><br></div><div>= > If memory views could be converted into python fast, and pickled I would ha= > ve no need for the old numpy array support.</div><div><br></div><div>Wonder= > ing if these problems will ever be addressed, or if I am missing something = > completely.<br></div><div><br></div><div><br></div></div> > > <p></p> > > -- <br /> > &nbsp;<br /> > --- <br /> > You received this message because you are subscribed to the Google Groups &= > quot;cython-users&quot; group.<br /> > To unsubscribe from this group and stop receiving emails from it, send an e= > mail to cython-users+unsubscribe(a)googlegroups.com.<br /> > For more options, visit <a href=3D"https://groups.google.com/groups/opt_out= > ">https://groups.google.com/groups/opt_out</a>.<br /> > > ------=_Part_1342_18667054.1390644115997--

1 0

using loadtxt to load a text file in to a numpy array
by Hedieh Ebrahimi 25 Jan '14

25 Jan '14

Hello, I am trying to use the following line of code : fileContent=loadtxt(filePath,dtype=str) in order to load a text file located at path= filePath in to a numpy array called fileContent. I´ve simplifed my file for the purpose of this question but the file looks something like this: file Content : C:\Users\Documents\Project\mytextfile1.txt C:\Users\Documents\Project\mytextfile2.txt C:\Users\Documents\Project\mytextfile3.txt I try to print my fileContent array after I read it and it looks like this : ["b'C:\\\\Users\\\\Documents\\\\Project\\\\mytextfile1.txt'" "b'C:\\\\Users\\\\Documents\\\\Project\\\\mytextfile2.txt'" "b'C:\\\\Users\\\\Documents\\\\Project\\\\mytextfile3.txt'"] Why is this happening and how can I prevent it ? Also if I have a line that starts like this in my file, python will crash on me. how can i fix this ? !--Timestep ( line in file starting with !-- ) I guess it has to have something to do with datatype. if I donot define the datatype it will be float by default which will give me an error an if I define the datatype as string as I did above, then I get to the problems that I mentioned above. I´d appreciate any help on how to fix this. Thanks

13 73

Catching out-of-memory error before it happens
by Dinesh Vadhia 25 Jan '14

25 Jan '14

I want to write a general exception handler to warn if too much data is being loaded for the ram size in a machine for a successful numpy array operation to take place. For example, the program multiplies two floating point arrays A and B which are populated with loadtext. While the data is being loaded, want to continuously check that the data volume doesn't pass a threshold that will cause on out-of-memory error during the A*B operation. The known variables are the amount of memory available, data type (floats in this case) and the numpy array operation to be performed. It seems this requires knowledge of the internal memory requirements of each numpy operation. For sake of simplicity, can ignore other memory needs of program. Is this possible?

6 10

np.array creation: unexpected behaviour
by Emanuele Olivetti 25 Jan '14

25 Jan '14

Hi, I just came across this unexpected behaviour when creating a np.array() from two other np.arrays of different shape. Have a look at this example: ---- import numpy as np a = np.zeros(3) b = np.zeros((2,3)) c = np.zeros((3,2)) ab = np.array([a, b]) print ab.shape, ab.dtype ac = np.array([a, c], dtype=np.object) print ac.shape, ac.dtype ac_no_dtype = np.array([a, c]) print ac_no_dtype.shape, ac_no_dtype.dtype ---- The output, with NumPy v1.6.1 (Ubuntu 12.04) is: ---- (2,) object (2, 3) object Traceback (most recent call last): File "/tmp/numpy_bug.py", line 9, in <module> ac_no_dtype = np.array([a, c]) ValueError: setting an array element with a sequence. ---- The result for 'ab' is what I expect. The one for 'ac' is a bit surprising. The one for ac_no_dtype even is more surprising. Is this an expected behaviour? Best, Emanuele

3 2

Re: [Numpy-discussion] (no subject)
by jennifer stone 24 Jan '14

24 Jan '14

Scipy doesn't have a function for the Laplace transform, it has only a > Laplace distribution in scipy.stats and a Laplace filter in scipy.ndimage. > An inverse Laplace transform would be very welcome I'd think - it has real > world applications, and there's no good implementation in any open source > library as far as I can tell. It's probably doable, but not the easiest > topic for a GSoC I think. From what I can find, the paper "Numerical > Transform Inversion Using Gaussian Quadrature" from den Iseger contains > what's considered the current state of the art algorithm. Browsing that > gives a reasonable idea of the difficulty of implementing `ilaplace`. A brief scanning through the paper "Numerical Transform Inversion Using Gaussian Quadrature" from den Iseger does indicate the complexity of the algorithm. But GSoC project or not, can't we work on it, step by step? As I would love to see a contender for Matlab's ilaplace on open source front!! > > You can have a look at https://github.com/scipy/scipy/pull/2908/files for > ideas. Most of the things that need improving or we really think we should > have in Scipy are listed there. Possible topics are not restricted to that > list though - it's more important that you pick something you're interested > in and have the required background and coding skills for. > Thanks a lot for the roadmap. Of the options provided, I found the 'Cython'ization of Cluster great. Would it be possible to do it as the Summer project if I spend the month learning Cython? Regards Janani > Cheers, > Ralf > > >

2 1

vstack and hstack performance penalty
by Dinesh Vadhia 24 Jan '14

24 Jan '14

When using vstack or hstack for large arrays, are there any performance penalties eg. takes longer time-wise or makes a copy of an array during operation ?

3 3

Re: [Numpy-discussion] (no subject)
by jennifer stone 23 Jan '14

23 Jan '14

Both scipy and numpy require GSOC > candidates to have a pull request accepted as part of the application > process. I'd suggest implementing a function not currently in scipy that > you think would be useful. That would also help in finding a mentor for the > summer. I'd also suggest getting familiar with cython. > > Chuck > Thanks a lot for the heads-up. I am yet to be familiarized with Cython and it indeed is playing a crucial role especially in the 'special' module > > I don't see you on github yet, are you there? If not, you should set up an > account to work in. See the developer guide > <http://docs.scipy.org/doc/numpy/dev/>for some pointers. > > Chuck > I am present on github but the profile at present is just a mark of humble mistakes of a beginner to open-sourcing, The id is https://github.com/jennystone. I hope to build upon my profile. Jennifer

1 0