Mailman 3 January 2014 - NumPy-Discussion

Text array dtype for numpy
by Oscar Benjamin 25 Jan '14

25 Jan '14

There have been a few threads discussing the problems of how to do text with numpy arrays in Python 3. To make a slightly more concrete proposal, I've implemented a pure Python ndarray subclass that I believe can consistently handle text/bytes in Python 3. It is intended to be an illustration since I think that the real solution is a new dtype rather than an array subclass (so that it can be used in e.g. record arrays). The idea is that the array has an encoding. It stores strings as bytes. The bytes are encoded/decoded on insertion/access. Methods accessing the binary content of the array will see the encoded bytes. Methods accessing the elements of the array will see unicode strings. I believe it would not be as hard to implement as the proposals for variable length string arrays. The one caveat is that it will strip null characters from the end of any string. I'm not 100% that the byte stripping encoding function will always work but it will for all the encodings I know and it seems to work with all the encodings that Python has. The code is inline below and attached (in case there are encoding problems with this message!): Oscar #!/usr/bin/env python3 from numpy import ndarray, array class textarray(ndarray): '''ndarray for holding encoded text. This is for demonstration purposes only. The real proposal is to specify the encoding as a dtype rather than a subclass. Only works as a 1-d array. >>> a = textarray(['qwert', 'zxcvb'], encoding='ascii') >>> a textarray(['qwert', 'zxcvb'], dtype='|S5:ascii') >>> a[0] 'qwert' >>> a.tostring() b'qwertzxcvb' >>> a[0] = 'qwe' # shorter string >>> a[0] 'qwe' >>> a.tostring() b'qwe\\x00\\x00zxcvb' >>> a[0] = 'qwertyuiop' # longer string Traceback (most recent call last): ... ValueError: Encoded bytes don't fit >>> b = textarray(['Õscar', 'qwe'], encoding='utf-8') >>> b textarray(['Õscar', 'qwe'], dtype='|S6:utf-8') >>> b[0] 'Õscar' >>> b[0].encode('utf-8') b'\\xc3\\x95scar' >>> b.tostring() b'\\xc3\\x95scarqwe\\x00\\x00\\x00' >>> c = textarray(['qwe'], encoding='utf-32-le') >>> c textarray(['qwe'], dtype='|S12:utf-32-le') ''' def __new__(cls, strings, encoding='utf-8'): bytestrings = [s.encode(encoding) for s in strings] a = array(bytestrings, dtype='S').view(textarray) a.encoding = encoding return a def __repr__(self): slist = ', '.join(repr(self[n]) for n in range(len(self))) return "textarray([%s], \n dtype='|S%d:%s')"\ % (slist, self.itemsize, self.encoding) def __getitem__(self, index): bstring = ndarray.__getitem__(self, index) return self._decode(bstring) def __setitem__(self, index, string): bstring = string.encode(self.encoding) if len(bstring) > self.itemsize: raise ValueError("Encoded bytes don't fit") ndarray.__setitem__(self, index, bstring) def _decode(self, b): b = b + b'\0' * (4 - len(b) % 4) s = b.decode(self.encoding) for n, c in enumerate(reversed(s)): if c != '\0': return s[:len(s)-n] return s if __name__ == "__main__": import doctest doctest.testmod()

3 3

Comparison changes
by Sebastian Berg 25 Jan '14

25 Jan '14

Hi all, in https://github.com/numpy/numpy/pull/3514 I proposed some changes to the comparison operators. This includes: 1. Comparison with None will broadcast in the future, so that `arr == None` will actually compare all elements to None. (A FutureWarning for now) 2. I added that == and != will give FutureWarning when an error was raised. In the future they should not silence these errors anymore. (For example shape mismatches) 3. We used to use PyObject_RichCompareBool for equality which includes an identity check. I propose to not do that identity check since we have elementwise equality (returning an object array for objects would be nice in some ways, but I think that is only an option for a dedicated function). The reason is that for example >>> a = np.array([np.array([1, 2, 3]), 1]) >>> b = np.array([np.array([1, 2, 3]), 1]) >>> a == b will happen to work if it happens to be that `a[0] is b[0]`. This currently has no deprecation, since the logic is in the inner loop and I am not sure if it is easy to add well there. Are there objections/comments to these changes? Regards, Sebastian

3 3

Re: [Numpy-discussion] Numpy arrays vs typed memoryviews
by Sturla Molden 25 Jan '14

25 Jan '14

I think I have said this before, but its worth a repeat: Pickle (including cPickle) is a slow hog! That might not be the overhead you see, you just haven't noticed it yet. I saw this some years ago when I worked on shared memory arrays for Numpy (cf. my account on Github). Shared memory really did not help to speed up the IPC, because the entire overhead was dominated by pickle. (Shared memory is a fine way of saving RAM, though.) multiprocessing.Queue will use pickle for serialization, and is therefore not the right tool for numerical parallel computing with Cython or NumPy. In order to use multiprocessing efficiently with NumPy, we need a new Queue type that knows about NumPy arrays (and/or Cython memoryviews), and treat them as special cases. Getting rid of pickle altogether is the important part, not facilitating its use even further. It is easy to make a Queue type for Cython or NumPy arrays using a duplex pipe and couple of mutexes. Or you can use shared memory as ringbuffer and atomic compare-and-swap on the first bytes as spinlocks. It is not difficult to get the overhead of queuing arrays down to little more than a memcpy. I've been wanting to do this for a while, so maybe it is time to start a new toy project :) Sturla Neal Hughes <hughes.neal(a)gmail.com> wrote: > I like Cython a lot. My only complaint is that I have to keep switching > between the numpy array support and typed memory views. Both have there > advantages but neither can do every thing I need. > > Memoryviews have the clean syntax and seem to work better in cdef classes > and in inline functions. > > But Memoryviews can't be pickled and so can't be passed between > processes. Also there seems to be a high overhead on converting between > memory views and python numpy arrays. Where this overhead is a problem, > or where i need to use pythons multiprocessing module I tend to switch to numpy arrays. > > If memory views could be converted into python fast, and pickled I would > have no need for the old numpy array support. > > Wondering if these problems will ever be addressed, or if I am missing > something completely. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "cython-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to cython-users+unsubscribe(a)googlegroups.com. > For more options, visit <a > href="https://groups.google.com/groups/opt_out.">https://groups.google.com/groups/opt_out.</a> > > ------=_Part_1342_18667054.1390644115997 > Content-Type: text/html; charset=UTF-8 > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"ltr">I like Cython a lot. My only complaint is that I have to k= > eep switching between the numpy array support and typed memory views. Both = > have there advantages but neither can do every thing I need.<div><br></div>= > <div>Memoryviews have the clean syntax and seem to work better in cdef clas= > ses and in inline functions.</div><div><br></div><div>But Memoryviews can't= > be pickled and so can't be passed between processes. Also there seems to b= > e a high overhead on converting between memory views and python numpy array= > s. Where this overhead is a problem, or where i need to use pythons multipr= > ocessing module I tend to switch to numpy arrays.</div><div><br></div><div>= > If memory views could be converted into python fast, and pickled I would ha= > ve no need for the old numpy array support.</div><div><br></div><div>Wonder= > ing if these problems will ever be addressed, or if I am missing something = > completely.<br></div><div><br></div><div><br></div></div> > > <p></p> > > -- <br /> > &nbsp;<br /> > --- <br /> > You received this message because you are subscribed to the Google Groups &= > quot;cython-users&quot; group.<br /> > To unsubscribe from this group and stop receiving emails from it, send an e= > mail to cython-users+unsubscribe(a)googlegroups.com.<br /> > For more options, visit <a href=3D"https://groups.google.com/groups/opt_out= > ">https://groups.google.com/groups/opt_out</a>.<br /> > > ------=_Part_1342_18667054.1390644115997--

1 0

using loadtxt to load a text file in to a numpy array
by Hedieh Ebrahimi 25 Jan '14

25 Jan '14

Hello, I am trying to use the following line of code : fileContent=loadtxt(filePath,dtype=str) in order to load a text file located at path= filePath in to a numpy array called fileContent. I´ve simplifed my file for the purpose of this question but the file looks something like this: file Content : C:\Users\Documents\Project\mytextfile1.txt C:\Users\Documents\Project\mytextfile2.txt C:\Users\Documents\Project\mytextfile3.txt I try to print my fileContent array after I read it and it looks like this : ["b'C:\\\\Users\\\\Documents\\\\Project\\\\mytextfile1.txt'" "b'C:\\\\Users\\\\Documents\\\\Project\\\\mytextfile2.txt'" "b'C:\\\\Users\\\\Documents\\\\Project\\\\mytextfile3.txt'"] Why is this happening and how can I prevent it ? Also if I have a line that starts like this in my file, python will crash on me. how can i fix this ? !--Timestep ( line in file starting with !-- ) I guess it has to have something to do with datatype. if I donot define the datatype it will be float by default which will give me an error an if I define the datatype as string as I did above, then I get to the problems that I mentioned above. I´d appreciate any help on how to fix this. Thanks

13 73

Catching out-of-memory error before it happens
by Dinesh Vadhia 25 Jan '14

25 Jan '14

I want to write a general exception handler to warn if too much data is being loaded for the ram size in a machine for a successful numpy array operation to take place. For example, the program multiplies two floating point arrays A and B which are populated with loadtext. While the data is being loaded, want to continuously check that the data volume doesn't pass a threshold that will cause on out-of-memory error during the A*B operation. The known variables are the amount of memory available, data type (floats in this case) and the numpy array operation to be performed. It seems this requires knowledge of the internal memory requirements of each numpy operation. For sake of simplicity, can ignore other memory needs of program. Is this possible?

6 10

np.array creation: unexpected behaviour
by Emanuele Olivetti 25 Jan '14

25 Jan '14

Hi, I just came across this unexpected behaviour when creating a np.array() from two other np.arrays of different shape. Have a look at this example: ---- import numpy as np a = np.zeros(3) b = np.zeros((2,3)) c = np.zeros((3,2)) ab = np.array([a, b]) print ab.shape, ab.dtype ac = np.array([a, c], dtype=np.object) print ac.shape, ac.dtype ac_no_dtype = np.array([a, c]) print ac_no_dtype.shape, ac_no_dtype.dtype ---- The output, with NumPy v1.6.1 (Ubuntu 12.04) is: ---- (2,) object (2, 3) object Traceback (most recent call last): File "/tmp/numpy_bug.py", line 9, in <module> ac_no_dtype = np.array([a, c]) ValueError: setting an array element with a sequence. ---- The result for 'ab' is what I expect. The one for 'ac' is a bit surprising. The one for ac_no_dtype even is more surprising. Is this an expected behaviour? Best, Emanuele

3 2

Re: [Numpy-discussion] (no subject)
by jennifer stone 24 Jan '14

24 Jan '14

Scipy doesn't have a function for the Laplace transform, it has only a > Laplace distribution in scipy.stats and a Laplace filter in scipy.ndimage. > An inverse Laplace transform would be very welcome I'd think - it has real > world applications, and there's no good implementation in any open source > library as far as I can tell. It's probably doable, but not the easiest > topic for a GSoC I think. From what I can find, the paper "Numerical > Transform Inversion Using Gaussian Quadrature" from den Iseger contains > what's considered the current state of the art algorithm. Browsing that > gives a reasonable idea of the difficulty of implementing `ilaplace`. A brief scanning through the paper "Numerical Transform Inversion Using Gaussian Quadrature" from den Iseger does indicate the complexity of the algorithm. But GSoC project or not, can't we work on it, step by step? As I would love to see a contender for Matlab's ilaplace on open source front!! > > You can have a look at https://github.com/scipy/scipy/pull/2908/files for > ideas. Most of the things that need improving or we really think we should > have in Scipy are listed there. Possible topics are not restricted to that > list though - it's more important that you pick something you're interested > in and have the required background and coding skills for. > Thanks a lot for the roadmap. Of the options provided, I found the 'Cython'ization of Cluster great. Would it be possible to do it as the Summer project if I spend the month learning Cython? Regards Janani > Cheers, > Ralf > > >

2 1

vstack and hstack performance penalty
by Dinesh Vadhia 24 Jan '14

24 Jan '14

When using vstack or hstack for large arrays, are there any performance penalties eg. takes longer time-wise or makes a copy of an array during operation ?

3 3

Re: [Numpy-discussion] (no subject)
by jennifer stone 23 Jan '14

23 Jan '14

Both scipy and numpy require GSOC > candidates to have a pull request accepted as part of the application > process. I'd suggest implementing a function not currently in scipy that > you think would be useful. That would also help in finding a mentor for the > summer. I'd also suggest getting familiar with cython. > > Chuck > Thanks a lot for the heads-up. I am yet to be familiarized with Cython and it indeed is playing a crucial role especially in the 'special' module > > I don't see you on github yet, are you there? If not, you should set up an > account to work in. See the developer guide > <http://docs.scipy.org/doc/numpy/dev/>for some pointers. > > Chuck > I am present on github but the profile at present is just a mark of humble mistakes of a beginner to open-sourcing, The id is https://github.com/jennystone. I hope to build upon my profile. Jennifer

1 0

cannot decode 'S'
by josef.pktd＠gmail.com 23 Jan '14

23 Jan '14

truncating null bytes in 'S' breaks decoding that needs them >>> a = np.array([si.encode('utf-16LE') for si in ['Õsc', 'zxc']], dtype='S') >>> a array([b'\xd5\x00s\x00c', b'z\x00x\x00c'], dtype='|S6') >>> [ai.decode('utf-16LE') for ai in a] Traceback (most recent call last): File "<pyshell#118>", line 1, in <module> [ai.decode('utf-16LE') for ai in a] File "<pyshell#118>", line 1, in <listcomp> [ai.decode('utf-16LE') for ai in a] File "C:\Programs\Python33\lib\encodings\utf_16_le.py", line 16, in decode return codecs.utf_16_le_decode(input, errors, True) UnicodeDecodeError: 'utf16' codec can't decode byte 0x63 in position 4: truncated data messy workaround (arrays in contrast to scalars are not truncated in `tostring`) >>> [a[i:i+1].tostring().decode('utf-16LE') for i in range(len(a))] ['Õsc', 'zxc'] Found while playing with examples in the other thread. Josef

2 1

NumPy-Discussion January 2014