Mailman 3 March 2011 - NumPy-Discussion

bug in genfromtxt for python 3.2
by josef.pktd＠gmail.com March 30, 2011

March 30, 2011

numpy/lib/test_io.py only uses StringIO in the test, no actual csv file If I give the filename than I get a TypeError: Can't convert 'bytes' object to str implicitly from the statsmodels mailing list example >>>> data = recfromtxt(open('./star98.csv', "U"), delimiter=",", skip_header=1, dtype=float) > Traceback (most recent call last): > File "<pyshell#30>", line 1, in <module> > data = recfromtxt(open('./star98.csv', "U"), delimiter=",", > skip_header=1, dtype=float) > File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", > line 1633, in recfromtxt > output = genfromtxt(fname, **kwargs) > File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", > line 1181, in genfromtxt > first_values = split_line(first_line) > File "C:\Programs\Python32\lib\site-packages\numpy\lib\_iotools.py", > line 206, in _delimited_splitter > line = line.split(self.comments)[0].strip(asbytes(" \r\n")) > TypeError: Can't convert 'bytes' object to str implicitly > > line 1184 in npyio (py32 sourcefile) > > if isinstance(fname, str): > fhd = np.lib._datasource.open(fname, 'U') > > seems to be the culprit for my case changing to binary solved this problem for me fhd = np.lib._datasource.open(fname, 'Ub') (I still have other errors but don't know yet where they are coming from.) Almost all problem with porting statsmodels to python 3.2 so far are input related, mainly reading csv files which are heavily used in the tests. All the "real" code seems to work fine with numpy and scipy (and matplotlib so far) for python 3.2 Josef

4 6

Warning: invalid value encountered in true_divide?
by Joon Ro March 30, 2011

March 30, 2011

Hi, After numpy upgrade, I started to get "Warning: invalid value encountered in true_divide," when I run a code which did now show any warning previously. What does it mean and where should I look to fix this? It does not stop my debugger so I could not identify where the message was from. Thank you, Joon

2 1

newbie question about boolean testing of array equality result
by Jonathan Hartley March 30, 2011

March 30, 2011

Hey people, I'm writing an application in which we evaluate user-supplied Python expressions. Sometimes, we want to do an equality test on the result of the evaluation, eg: result = eval(...) if result == float('nan'): ... If the result is a numpy array, then this raises a ValueError, since the array equality operator returns a new numpy array, and the coercion of this to a boolean for the if predicate then explicitly raises. Presumably this is well-known? For us, it is undesirable. Am I right to understand that any code which might ever encounter a numpy array therefore can never use an unguarded '/if x == y:/' construction? Is my workaround really to replace every instance of this with '/if not isinstance(x, numpy.array) and x==y:/' ? This pains me, because otherwise this code would have no dependency on numpy. (I can't just prepend 'isinstance(x, float)' because, unlike the example above, we don't always know the type of the equality RHS until runtime.) I can see that there's a pleasing symmetry to have all the array arithmetic operators and comparisons operate in an element-wise manner, but I think it's more important for __eq__ to follow it's usual semantics of returning a boolean. I'd way prefer it if the element-wise equality array generation was exposed as a different method. Have I misunderstood the situation greatly? Thanks for any pointers or suggestions. Jonathan -- Jonathan Hartley tartley(a)tartley.com http://tartley.com Made of meat. +44 7737 062 225 twitter/skype: tartley

6 6

histogram normed/density keyword (#1628)
by Ralf Gommers March 29, 2011

March 29, 2011

Hi all, For the 1.6 release #1628 needs to be resolved. A while ago there was a discussion about the normed keyword in histogram, which ATM has changed behavior compared to numpy 1.5.1. The preferred fix as I read the discussion (http://thread.gmane.org/gmane.comp.python.numeric.general/39746/focus=40089) was to leave normed alone and add a new keyword density. It would be helpful if someone can volunteer to fix this. Thanks, Ralf

1 1

Re: [Numpy-discussion] stable sort on a recarray ?
by Peter Butterworth March 29, 2011

March 29, 2011

np.lexsort does the job for both the single or multi-column stable sort cases, thanks. a = np.array([('a', 1, 1), ('a', 0, 1), ('a', 0, 0), ('b', 0, 2)], dtype=[('name', '|S10'), ('x', '<i4'), ('y', '<i4')]) sortind = np.lexsort([a['x'], a['name']]) >> sortind array([1, 2, 0, 3], dtype=int64) >> a[sortind] array([('a', 0, 1), ('a', 0, 0), ('a', 1, 1), ('b', 0, 2)], dtype=[('name', '|S10'), ('x', '<i4'), ('y', '<i4')]) The documentation could perhaps benefit from some clarification: >> http://docs.scipy.org/doc/numpy/reference/generated/numpy.lexsort.html#nump… . It is not mentioned on that page that lexsort is a stable sort. . no structured array / recarray example is given . it also states that "Structured arrays are sorted lexically by argsort", but fails to mention that the resulting sort is not stable. -- thanks, peter butterworth

1 0

Re: [Numpy-discussion] stable sort on a recarray ?
by butterw＠gmail.com March 29, 2011

March 29, 2011

>> sortind = np.argsort(x['name'], kind='mergesort'); x[sortind] The indirect sorting method that was suggested works for doing stable sort on recarrays / structured arrays based on a single-column. # It is necessary to specify kind='mergesort' because argsort is not stable: > np.argsort(np.ones(100)) array([ 0, 72, 71, 70, 69, 68, 67, 66, 65, 64, 63, 62, 61, 60, 59, 58, 57, 56, 55, 54, 53, 52, 73, 51, 74, 76, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 79, 78, 77, 75, 50, 49, 48, 21, 20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 22, 23, 24, 25, 47, 46, 45, 44, 43, 42, 41, 40, 39, 38, 98, 37, 35, 34, 33, 32, 31, 30, 29, 28, 27, 26, 36, 99]) Any suggestions on how to achieve stable sort based on multiple columns with numpy ? a = array([('a', 1, 1), ('a', 0, 1), ('a', 0, 0), ('b', 0, 2)], dtype=[('name', '|S10'), ('x', '<i4'), ('y', '<i4')]) name xy a 1 1 a 0 1 a 0 0 b 0 2 # perform sort on primary column 'name' first, then if required on secondary column 'x' >>> argsort(a, order=('name', 'x') array([2, 1, 0, 3]) # we get the expected result but the sort is not stable, desired result was: array([2, 0, 1, 3])

2 1

random number genration
by Alex Ter-Sarkissov March 29, 2011

March 29, 2011

If I want to generate a string of random bits with equal probability I run random.randint(0,2,size). What if I want a specific proportion of bits? In other words, each bit is 1 with probability p<1/2 and 0 with probability q=1-p? thanks

6 10

npz load on numpy 3
by josef.pktd＠gmail.com March 29, 2011

March 29, 2011

more python 3.2 fun a npz file saved with python 2.6 (I guess) that I try to read with python 3.2 I have no clue, since I never use .npz files >>> arr = np.load(r"..\scikits\statsmodels\tsa\vector_ar\tests\results\vars_results.npz") >>> arr <numpy.lib.npyio.NpzFile object at 0x0000000003874AC8> >>> dir(arr) ['__class__', '__contains__', '__del__', '__delattr__', '__dict__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_files', 'close', 'f', 'fid', 'files', 'items', 'iteritems', 'iterkeys', 'keys', 'zip'] >>> arr.keys() ['causality', 'orthirf', 'detomega', 'nirfs', 'loglike', 'stderr', 'crit', 'phis', 'nahead', 'totobs', 'type', 'obs', 'irf', 'coefs'] >>> arr['irf'] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Programs\Python32\lib\site-packages\numpy\lib\npyio.py", line 222, in __getitem__ return format.read_array(value) File "C:\Programs\Python32\lib\site-packages\numpy\lib\format.py", line 449, in read_array array = pickle.load(fp) UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 6: ordinal not in range(128) >>> Any ideas ? Josef

2 2

Array views
by Hugo Gagnon March 29, 2011

March 29, 2011

Hello, Say I have a few 1d arrays and one 2d array which columns I want to be the 1d arrays. I also want all the a's arrays to share the *same data* with the b array. If I call my 1d arrays a1, a2, etc. and my 2d array b, then b[:,0] = a1[:] b[:,1] = a2[:] ... won't work because apparently copying occurs. I tried it the other way around i.e. a1 = b[:,0] a2 = b[:,1] ... and it works but that doesn't help me for my problem. Is there a way to reformulate the first code snippet above but with shallow copying? Thanks, -- Hugo Gagnon -- Hugo Gagnon

7 22

stable sort on a recarray ?
by butterw＠gmail.com March 29, 2011

March 29, 2011

What is the recommended way to perform a stable sort on a recarray / structured array ? In the following example, I want to sort on name, while retaining the existing order in the case of equal values: values = [('a', 1), ('a', 0), ('b', 2)] dtype = [('name', 'S10'), ('val', 'i4')] >>> a = np.array(values, dtype=dtype) array([('a', 1), ('a', 0), ('b', 2)], dtype=[('name', '|S10'), ('val', '<i4')]) >>> np.sort(a, order='name') array([('a', 0), ('a', 1), ('b', 2)], dtype=[('name', '|S10'), ('val', '<i4')])

2 1