Mailman 3 August 2006 - NumPy-Discussion

For loop tips
by Keith Goodman Aug. 31, 2006

Aug. 31, 2006

I have a very long list that contains many repeated elements. The elements of the list can be either all numbers, or all strings, or all dates [datetime.date]. I want to convert the list into a matrix where each unique element of the list is assigned a consecutive integer starting from zero. I've done it by brute force below. Any tips for making it faster? (5x would make it useful; 10x would be a dream.) >> list2index.test() Numbers: 5.84955787659 seconds Characters: 24.3192870617 seconds Dates: 39.288228035 seconds import datetime, time from numpy import nan, asmatrix, ones def list2index(L): # Find unique elements in list uL = dict.fromkeys(L).keys() # Convert list to matrix L = asmatrix(L).T # Initialize return matrix idx = nan * ones((L.size, 1)) # Assign numbers to unique L values for i, uLi in enumerate(uL): idx[L == uLi,:] = i def test(): L = 5000*range(255) t1 = time.time() idx = list2index(L) t2 = time.time() print 'Numbers:', t2-t1, 'seconds' L = 5000*[chr(z) for z in range(255)] t1 = time.time() idx = list2index(L) t2 = time.time() print 'Characters:', t2-t1, 'seconds' d = datetime.date step = datetime.timedelta L = 5000*[d(2006,1,1)+step(z) for z in range(255)] t1 = time.time() idx = list2index(L) t2 = time.time() print 'Dates:', t2-t1, 'seconds'

4 8

BLAS not found in numpy 1.0b4
by Jonathan Taylor Aug. 31, 2006

Aug. 31, 2006

When trying to install 1.0b4 I had trouble getting it to detect my installed atlas. This was because the shipped site.cfg had; [atlas] library_dirs = /usr/lib/atlas/3dnow/ atlas_libs = lapack, blas but I had to change 3dnow to sse2 due to my current state of pentiumness. In any case it should probabally look in all the possible locations instead of just AMD's location. Cheers. Jon.

3 2

numpy and dtype
by humufr＠yahoo.fr Aug. 31, 2006

Aug. 31, 2006

Hi, sorry to bother you with probably plenty of stupid question but I would like to clarify my mind with dtype. I have a problem to view a recarray, I'm not sure but I suspect a bug or at least a problem I have an array with some data, the array is very big but I have no problem with numpy. In [44]: data_end Out[44]: array([[ 2.66000000e+02, 5.16300000e+04, 1.00000000e+00, ..., -1.04130435e+00, 1.47304565e+02, 4.27402449e+00], [ 2.66000000e+02, 5.16300000e+04, 2.00000000e+00, ..., -6.52190626e-01, 1.64214981e+02, 1.58334379e+01], [ 2.66000000e+02, 5.16300000e+04, 4.00000000e+00, ..., -7.65136838e-01, 1.33340195e+02, 9.84033298e+00], ..., [ 9.78000000e+02, 5.24310000e+04, 6.32000000e+02, ..., 3.06083832e+01, 6.71210251e+01, 1.18813887e+01], [ 9.78000000e+02, 5.24310000e+04, 6.36000000e+02, ..., 3.05993423e+01, 1.10403000e+02, 5.81539488e+00], [ 9.78000000e+02, 5.24310000e+04, 6.40000000e+02, ..., 3.05382938e+01, 1.26916304e+01, 3.25683937e+01]]) In [45]: data_end.shape Out[45]: (567486, 7) In [46]: data_end.dtype Out[46]: dtype('<f8') I want to have this array with certain dtype so I transform it in recarray (I have unable ot have different type for each columns in the array but I'm just beginning to play with dtype. So I did: In [47]: fields = ['PLATEID', 'MJD', 'FIBERID', 'RA', 'DEC','V_DISP','V_DISP_ERR'] In [48]: In [48]: type_descr = numpy.dtype({'names':fields,'formats': ['>i2','>i4','>i2','>f4','>f4','>f4','>f4']}) In [49]: b = numpy.rec.fromarrays(data_end.transpose(),type_descr) In [50]: b[:1] Out[50]: recarray([ (266, 51630, 1, 146.71420288085938, -1.041304349899292, 147.3045654296875, 4.274024486541748)], dtype=[('PLATEID', '>i2'), ('MJD', '>i4'), ('FIBERID', '>i2'), ('RA', '>f4'), ('DEC', '>f4'), ('V_DISP', '>f4'), ('V_DISP_ERR', '>f4')]) In [51]: b[1] Out[51]: (266, 51630, 2, 146.74412536621094, -0.65219062566757202, 164.21498107910156, 15.833437919616699) but I obtain an error when I want to print the recarray b (it's working for smallest array): In [54]: b[:10] Out[54]: recarray([ (266, 51630, 1, 146.71420288085938, -1.041304349899292, 147.3045654296875, 4.274024486541748), (266, 51630, 2, 146.74412536621094, -0.65219062566757202, 164.21498107910156, 15.833437919616699), (266, 51630, 4, 146.62857055664062, -0.76513683795928955, 133.34019470214844, 9.8403329849243164), (266, 51630, 6, 146.63166809082031, -0.98827779293060303, 146.91035461425781, 30.08709716796875), (266, 51630, 7, 146.91944885253906, -0.99049174785614014, 152.96893310546875, 12.429832458496094), (266, 51630, 9, 146.76339721679688, -0.81043314933776855, 347.72918701171875, 41.387767791748047), (266, 51630, 10, 146.62281799316406, -0.9513852596282959, 162.53567504882812, 24.676788330078125), (266, 51630, 11, 146.93409729003906, -0.67040395736694336, 266.56011962890625, 10.875675201416016), (266, 51630, 12, 146.96389770507812, -0.54500257968902588, 92.040328979492188, 18.999214172363281), (266, 51630, 13, 146.9635009765625, -0.75935173034667969, 72.828048706054688, 13.028598785400391)], dtype=[('PLATEID', '>i2'), ('MJD', '>i4'), ('FIBERID', '>i2'), ('RA', '>f4'), ('DEC', '>f4'), ('V_DISP', '>f4'), ('V_DISP_ERR', '>f4')]) So I would like to know if it's normal. And another question is it possile to do, in theory, something like: b = numpy.array(data_end,dtype=type_descr) or all individual array element must have the same dtype? To replace the context, I have a big fits table, I want to use only some columns from the table so I did: table = pyfits.getdata('gal_info_dr4_v5_1b.fit') #pyfits can't read, at least now the gzip file #the file is a fits table file so we look in the pyfits doc to read it! fields = ['PLATEID', 'MJD', 'FIBERID', 'RA', 'DEC','V_DISP','V_DISP_ERR'] type_descr = numpy.dtype({'names':fields,'formats': ['<i2','<i4','<i2','<f8','<f8','<f8','<f8']}) data_end = numpy.zeros((table.shape[0],len(fields))) for i in range(len(fields)): data_end[:,i] = table[:].field(fields[i]) but I want to keep the type from the fits file for each field perhaps there are a better way to do it. Thank you very much. N. The error when I try to print the big recarray: In [53]: b --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/gruel/Desktop/SDSS/DR4/<ipython console> /home/gruel/usr/lib/python2.4/site-packages/IPython/Prompts.py in __call__(self, arg) 514 515 # and now call a possibly user-defined print mechanism --> 516 manipulated_val = self.display(arg) 517 518 # user display hooks can change the variable to be stored in /home/gruel/usr/lib/python2.4/site-packages/IPython/Prompts.py in _display(self, arg) 538 """ 539 --> 540 return self.shell.hooks.result_display(arg) 541 542 # Assign the default display method: /home/gruel/usr/lib/python2.4/site-packages/IPython/hooks.py in __call__(self, *args, **kw) 132 #print "prio",prio,"cmd",cmd #dbg 133 try: --> 134 ret = cmd(*args, **kw) 135 return ret 136 except ipapi.TryNext, exc: /home/gruel/usr/lib/python2.4/site-packages/IPython/hooks.py in result_display(self, arg) 153 154 if self.rc.pprint: --> 155 out = pformat(arg) 156 if '\n' in out: 157 # So that multi-line strings line up with the left column of /usr/lib/python2.4/pprint.py in pformat(self, object) 108 def pformat(self, object): 109 sio = _StringIO() --> 110 self._format(object, sio, 0, 0, {}, 0) 111 return sio.getvalue() 112 /usr/lib/python2.4/pprint.py in _format(self, object, stream, indent, allowance, context, level) 126 self._readable = False 127 return --> 128 rep = self._repr(object, context, level - 1) 129 typ = _type(object) 130 sepLines = _len(rep) > (self._width - 1 - indent - allowance) /usr/lib/python2.4/pprint.py in _repr(self, object, context, level) 192 def _repr(self, object, context, level): 193 repr, readable, recursive = self.format(object, context.copy(), --> 194 self._depth, level) 195 if not readable: 196 self._readable = False /usr/lib/python2.4/pprint.py in format(self, object, context, maxlevels, level) 204 and whether the object represents a recursive construct. 205 """ --> 206 return _safe_repr(object, context, maxlevels, level) 207 208 /usr/lib/python2.4/pprint.py in _safe_repr(object, context, maxlevels, level) 289 return format % _commajoin(components), readable, recursive 290 --> 291 rep = repr(object) 292 return rep, (rep and not rep.startswith('<')), False 293 /home/gruel/usr/lib/python2.4/site-packages/numpy/core/numeric.py in array_repr(arr, max_line_width, precision, suppress_small) 389 if arr.size > 0 or arr.shape==(0,): 390 lst = array2string(arr, max_line_width, precision, suppress_small, --> 391 ', ', "array(") 392 else: # show zero-length shape unless it is (0,) 393 lst = "[], shape=%s" % (repr(arr.shape),) /home/gruel/usr/lib/python2.4/site-packages/numpy/core/arrayprint.py in array2string(a, max_line_width, precision, suppress_small, separator, prefix, style) 202 else: 203 lst = _array2string(a, max_line_width, precision, suppress_small, --> 204 separator, prefix) 205 return lst 206 /home/gruel/usr/lib/python2.4/site-packages/numpy/core/arrayprint.py in _array2string(a, max_line_width, precision, suppress_small, separator, prefix) 137 if a.size > _summaryThreshold: 138 summary_insert = "..., " --> 139 data = _leading_trailing(a) 140 else: 141 summary_insert = "" /home/gruel/usr/lib/python2.4/site-packages/numpy/core/arrayprint.py in _leading_trailing(a) 108 if a.ndim == 1: 109 if len(a) > 2*_summaryEdgeItems: --> 110 b = _gen.concatenate((a[:_summaryEdgeItems], 111 a[-_summaryEdgeItems:])) 112 else: TypeError: expected a readable buffer object Out[53]:

1 0

Unwanted upcast from uint64 to float64
by Torgil Svensson Aug. 31, 2006

Aug. 31, 2006

I'm using windows datetimes (100nano-seconds since 0001,1,1) as time in a numpy array and was hit by this behaviour. >>> numpy.__version__ '1.0b4' >>> a=numpy.array([632925394330000000L],numpy.uint64) >>> t=a[0] >>> t 632925394330000000L >>> type(t) <type 'numpy.uint64'> >>> t+1 6.3292539433e+017 >>> type(t+1) <type 'numpy.float64'> >>> t==(t+1) True I was trying to set t larger than any time in an array. Is there any reason for the scalar to upcast in this case? //Torgil

2 2

numpy/f2py module import segfault on HP-UX11.11
by Christophe-Blondeau Aug. 31, 2006

Aug. 31, 2006

1 0

Release of 1.0b5 this weekend
by Travis Oliphant Aug. 31, 2006

Aug. 31, 2006

Hi all, Classes start for me next Tuesday, and I'm teaching a class for which I will be using NumPy / SciPy extensively. I need to have a release of these two (and hopefully matplotlib) that work with each other. Therefore, I'm going to make a 1.0b5 release of NumPy over the weekend (probably Monday), and also get a release of SciPy out as well. At that point, I'll only be available for bug-fixes to 1.0. Therefore, the next release after 1.0b5 I would like to be 1.0rc1 (release-candidate 1). To facilitate that, after 1.0b5 there will be a feature-freeze (except for in the compatibility modules and the alter_code scripts which can still be modified to ease the transition burden). The 1.0rc1 release of NumPy will be mid September I suspect. Also, I recognize that the default-axis switch is a burden for those who have already transitioned code to use NumPy (for those just starting out it's not a big deal because of the compatibility layer). As a result, I've added a module called fix_default_axis whose converttree method will walk a hierarchy and change all .py files to fix the default axis problem in those files. This can be done in one of two ways (depending on the boolean argument import_change). If import_change is False a) Add and axis=<olddefault> keyword argument to any function whose default changed in 1.0b2 or 1.0b3, which does not already have the axis argument --- this method does not distinguish where the function came from and so can do the wrong thing with similarly named functions from other modules (.e.g. builtin sum and itertools.repeat). If import_change is True b) Change the location where the function is imported from numpy to numpy.oldnumeric where the default axis is the same as before. This approach looks for several flavors of the import statement and alters the import location for any function whose default axis argument changed --- this can get confused if you use from numpy import sum as mysum --- it will not replace that usage of sum. I used this script on the scipy tree in mode a) as a test (followed by a manual replacement of all? incorrect substitutions). I hope it helps. I know it's annoying to have such things change. But, it does make NumPy much more consistent with respect to the default axis argument. With a few exceptions (concatenate, diff, trapz, split, array_split), the rule is that you need to specify the axis if there is more than 1 dimension or it will ravel the input. -Travis

10 23

Use of numarray from numpy package
by LANDRIU David SAp Aug. 31, 2006

Aug. 31, 2006

Hello, I learned you answered me, but I did not get your message : can you send it to me again ? Thanks , David Landriu -------------------------------------------------------------------- David Landriu DAPNIA/SAp CEA SACLAY (France) Phone : (33|0)169088785 Fax : (33|0)169086577 ---------------------------------------------------------------------

2 1

fftfreq very slow; rfftfreq incorrect?
by Andrew Jaffe Aug. 31, 2006

Aug. 31, 2006

Hi all, the current implementation of fftfreq (which is meant to return the appropriate frequencies for an FFT) does the following: k = range(0,(n-1)/2+1)+range(-(n/2),0) return array(k,'d')/(n*d) I have tried this with very long (2**24) arrays, and it is ridiculously slow. Should this instead use arange (or linspace?) and concatenate rather than converting the above list? This seems to result in acceptable performance, but we could also perhaps even pre-allocate the space. The numpy.fft.rfftfreq seems just plain incorrect to me. It seems to produce lots of duplicated frequencies, contrary to the actual output of rfft: def rfftfreq(n,d=1.0): """ rfftfreq(n, d=1.0) -> f DFT sample frequencies (for usage with rfft,irfft). The returned float array contains the frequency bins in cycles/unit (with zero at the start) given a window length n and a sample spacing d: f = [0,1,1,2,2,...,n/2-1,n/2-1,n/2]/(d*n) if n is even f = [0,1,1,2,2,...,n/2-1,n/2-1,n/2,n/2]/(d*n) if n is odd **** None of these should be doubled, right? """ assert isinstance(n,int) return array(range(1,n+1),dtype=int)/2/float(n*d) Thanks, Andrew

4 6

Re: [Numpy-discussion] Use of numarray from numpy package [# INC NO 24609]
by LANDRIU David SAp Aug. 31, 2006

Aug. 31, 2006

Hello, I come back to my question : how to use numarray with the numpy installation ? After some update in the system there is another error message : >> AttributeError: 'module' object has no attribute 'NewAxis' It seems , from advice of the system manager, that an kind of alias failed to execute the right action. Thanks in advance for your answer, Cheers, David Landriu ------------- Begin Forwarded Message ------------- >Date: Wed, 30 Aug 2006 14:14:27 +0200 (MEST) >To: LANDRIU David SAp <landriu(a)discovery.saclay.cea.fr> >Subject: Re: Use of numarray from numpy package [# INC NO 24609] >From: User Support <user.support(a)cc.in2p3.fr> >Error-to: Jean-Rene Rouet <rouet(a)in2p3.fr> >X-CEA-Source: externe >X-CEA-DebugSpam: 7% >X-CEA-Spam-Report: No antispam rules were triggered by this message >X-CEA-Spam-Hits: __HAS_MSGID 0, __MIME_TEXT_ONLY 0, __SANE_MSGID 0, __STOCK_CRUFT 0 >MIME-Version: 1.0 >Content-Transfer-Encoding: 8bit >X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on discovery >X-Spam-Status: No, hits=0.1 required=4.0 tests=AWL autolearn=no version=2.63 >X-Spam-Level: > > >Réponse de User-Support à votre question : >------------------------------------------ > >Rebonjour >Essayez maintenat svp > >WW Voici ce que j'obtiens maintenant : {ccali22}~(0)>setenv PYTHONPATH /usr/local/lib/python2.3/site-packages/numpy {ccali22}~(0)> {ccali22}~(0)> {ccali22}~(0)>python Python 2.3.5 (#2, Oct 17 2005, 17:20:02) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-52)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from numarray import * Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: 'module' object has no attribute 'NewAxis' >>> ############################################## ############################################## Hello, is it necessary to install numarray separately to use numpy ? Indeed, after numpy installation, when I try to use it in the code, I get the same error as below : .../... Python 2.4.1 (#1, May 13 2005, 13:45:18) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from numarray import * Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/lib/python2.3/site-packages/numpy/numarray/__init__.py", line 1, in ? from util import * File "/usr/local/lib/python2.3/site-packages/numpy/numarray/util.py", line 2, in ? from numpy import geterr ImportError: No module named numpy >>> Thanks for your answer, Cheers, David Landriu -------------------------------------------------------------------- David Landriu DAPNIA/SAp CEA SACLAY (France) Phone : (33|0)169088785 Fax : (33|0)169086577 ---------------------------------------------------------------------

6 5

Changing Fatal error into ImportError?
by Fernando Perez Aug. 30, 2006

Aug. 30, 2006

Hi all, this was mentioned in the past, but I think it fell through the cracks: Python 2.3.4 (#1, Mar 10 2006, 06:12:09) [GCC 3.4.5 20051201 (Red Hat 3.4.5-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import mwadap Overwriting info=<function info at 0xb77198b4> from scipy.misc (was <function info at 0xb7704bc4> from numpy.lib.utils) RuntimeError: module compiled against version 90909 of C-API but this version of numpy is 1000002 Fatal Python error: numpy.core.multiarray failed to import... exiting. I really think that this should raise ImportError, but NOT kill the python interpreter. If this happens in the middle of a long-running interactive session, you'll lose all of your current state/work, where a simple ImportError would have been enough to tell you that this particular module needed recompilation. FatalError should be reserved for situations where the internal state of the Python VM itself can not realistically be expected to be sane (corruption, complete memory exhaustion for even internal allocations, etc.) But killing the user's session for a failed import is a bit much, IMHO. Cheers, f

2 2