From niklassaers at gmail.com Mon Jan 1 07:15:08 2007 From: niklassaers at gmail.com (Niklas Saers) Date: Mon, 1 Jan 2007 13:15:08 +0100 Subject: [Numpy-discussion] argmax() Message-ID: Hi, I'm quite new to numpy. My understanding of argmax() is that it gives the index of the highest number in the array, just like: from numpy import * a=array(range(4096)) a=a*15 argmax(a) gives 4095. But, when I use this within my program I have a little function: def localMaxima(set): maxPos = argmax(set) maxVal = set[maxPos] return maxPos, maxVal that I have had to extend a little bit to: def localMaxima(set): maxPos = argmax(set) try: maxVal = set[maxPos] print "ok" except: print "maxPos = " + str(maxPos) + ", while len(set) is " + str(len (set)) print set maxVal = 0.0 return maxPos, maxVal The problem is that set is an array of 2048 numbers. However, when I run this function I get: "maxPos = 7925760, while len(set) is 2048" I do not understand why I get such a high index. Do you have any suggestions? Cheers Nik From aisaac at american.edu Mon Jan 1 09:50:28 2007 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 1 Jan 2007 09:50:28 -0500 Subject: [Numpy-discussion] argmax() In-Reply-To: References: Message-ID: On Mon, 1 Jan 2007, Niklas Saers apparently wrote: > The problem is that set is an array of 2048 numbers. However, when I > run this function I get: > "maxPos = 7925760, while len(set) is 2048" > I do not understand why I get such a high index. Do you have any > suggestions? Can you post code that produces this problem? (Also, your numpy version.) Cheers, Alan Isaac From niklassaers at gmail.com Mon Jan 1 15:47:11 2007 From: niklassaers at gmail.com (Niklas Saers) Date: Mon, 1 Jan 2007 21:47:11 +0100 Subject: [Numpy-discussion] argmax() In-Reply-To: References: Message-ID: Hi, while answering your email (making a smaller version with data via pickle) I noticed that the variable "set" contained a very large matrix (I'm not quite sure but I believe I may have thrown 2048 dimensions at it with a multiplication that went wrong, giving a "set" object that pickled took 186547606 bytes. ;-) If it makes any sense I'd love to send you this variable that had len(2048) but was very much longer if stomped flat. :-) Cheers Nik On 1. jan. 2007, at 15.50, Alan G Isaac wrote: > On Mon, 1 Jan 2007, Niklas Saers apparently wrote: >> The problem is that set is an array of 2048 numbers. However, when I >> run this function I get: >> "maxPos = 7925760, while len(set) is 2048" >> I do not understand why I get such a high index. Do you have any >> suggestions? > > Can you post code that produces this problem? > (Also, your numpy version.) > > Cheers, > Alan Isaac > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From cjw at sympatico.ca Mon Jan 1 19:51:02 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon, 01 Jan 2007 19:51:02 -0500 Subject: [Numpy-discussion] Scalar values with a matrix Message-ID: <4599AC76.8000206@sympatico.ca> What is the best way to treat scalar values from a matrix? It seems that it's best to treat them as a simple Python values and not to leave them wrapped up in the ndarray structure. I would welcome advice. The problem is illustrated below. Colin W. # tMul.py import numpy.core as _n a= _n.arange(12).reshape((3, 4)) b= _n.arange(12).reshape((4, 3)) c= _n.dot(a, b) print `c` amat= _n.mat(a) bmat= _n.mat(b) cmat= amat*bmat print cmat print _n.dot(amat, _n.mat(5)) Output: >>> array([[ 42, 48, 54], [114, 136, 158], [186, 224, 262]]) [[ 42 48 54] [114 136 158] [186 224 262]] Traceback (most recent call last): File "", line 74, in run_nodebug File "C:\Documents and Settings\cjw\My Documents\PyDev\PyMatrix\tMiul.py", line 11, in print _n.dot(amat, _n.mat(5)) ValueError: objects are not aligned >>> From charlesr.harris at gmail.com Mon Jan 1 22:44:48 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 1 Jan 2007 20:44:48 -0700 Subject: [Numpy-discussion] Scalar values with a matrix In-Reply-To: <4599AC76.8000206@sympatico.ca> References: <4599AC76.8000206@sympatico.ca> Message-ID: On 1/1/07, Colin J. Williams wrote: > > What is the best way to treat scalar values from a matrix? > > It seems that it's best to treat them as a simple Python values and not > to leave them wrapped up in the ndarray structure. I would welcome > advice. > > The problem is illustrated below. > > Colin W. > > # tMul.py > import numpy.core as _n > a= _n.arange(12).reshape((3, 4)) > b= _n.arange(12).reshape((4, 3)) > c= _n.dot(a, b) > print `c` > amat= _n.mat(a) > bmat= _n.mat(b) > cmat= amat*bmat > print cmat > print _n.dot(amat, _n.mat(5)) mat(5) is a 1x1 matrix: In [5]: mat(5) Out[5]: matrix([[5]]) so the error is valid. If you want to do a scalar multiply try In [6]: amat*5 Out[6]: matrix([[ 0, 5, 10, 15], [20, 25, 30, 35], [40, 45, 50, 55]]) This is an example where using arrays, as opposed to matrices, is perhaps less confusing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkg at cs.nyu.edu Tue Jan 2 02:30:34 2007 From: mkg at cs.nyu.edu (Matthew Koichi Grimes) Date: Tue, 02 Jan 2007 02:30:34 -0500 Subject: [Numpy-discussion] subclassing recarray Message-ID: <459A0A1A.80203@cs.nyu.edu> Is it hard to subclass recarrays? I'm currently working on a subclass that doesn't add much; just some methods that sets all the values to zero and other similarly trivial things. It'd be nice to make it work, but if I have to use external functions on a plain recarray instead of this subclass, I won't be found weeping in the corner. The constructor of this subclass just calls recarray's __init__ method, but when it does this, I get an error. The class is currently just the following: class nnvalue(N.recarray): def __init__(self, shape): N.recarray.__init__(self, shape, dtype=N.dtype([('x', 'f8'), ('dx', 'f8'), ('delta', 'f8')])) def clearGradients(self): """Fills all members other than x with zeros.""" self.dx[:] = 0. self.deltas[:] = 0. The error message reads: Traceback (most recent call last): File "/home/mkg/projects/physlearn/src/pl/modules.py", line 287, in ? mod = DynModule(states.shape[1:], controls.shape[1:], states.shape[0]) File "/home/mkg/projects/physlearn/src/pl/modules.py", line 62, in __init__ self.params = nnvalue(pSize) File "/usr/lib/python2.4/site-packages/numpy/core/records.py", line 175, in __new__ descr = format_parser(formats, names, titles, aligned, byteorder)._descr File "/usr/lib/python2.4/site-packages/numpy/core/records.py", line 44, in __init__ self._parseFormats(formats, aligned) File "/usr/lib/python2.4/site-packages/numpy/core/records.py", line 52, in _parseFormats raise ValueError, "Need formats argument" ValueError: Need formats argument Any help would be much appreciated. -- Matt From stefan at sun.ac.za Tue Jan 2 03:22:19 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 2 Jan 2007 10:22:19 +0200 Subject: [Numpy-discussion] subclassing recarray In-Reply-To: <459A0A1A.80203@cs.nyu.edu> References: <459A0A1A.80203@cs.nyu.edu> Message-ID: <20070102082219.GE10637@mentat.za.net> Hi Matthew On Tue, Jan 02, 2007 at 02:30:34AM -0500, Matthew Koichi Grimes wrote: > Is it hard to subclass recarrays? I'm currently working on a subclass > that doesn't add much; just some methods that sets all the values to > zero and other similarly trivial things. It'd be nice to make it work, > but if I have to use external functions on a plain recarray instead of > this subclass, I won't be found weeping in the corner. Pierre wrote this useful wiki entry on subclassing: http://www.scipy.org/Subclasses It describes exactly what you want to do. Find attached some example code. Regards St?fan -------------- next part -------------- A non-text attachment was scrubbed... Name: subrec.py Type: text/x-python Size: 513 bytes Desc: not available URL: From svetosch at gmx.net Tue Jan 2 06:05:38 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Tue, 02 Jan 2007 12:05:38 +0100 Subject: [Numpy-discussion] Scalar values with a matrix In-Reply-To: References: <4599AC76.8000206@sympatico.ca> Message-ID: <459A3C82.9010506@gmx.net> Charles R Harris schrieb: > > mat(5) is a 1x1 matrix: > > In [5]: mat(5) > Out[5]: matrix([[5]]) > > so the error is valid. If you want to do a scalar multiply try > > In [6]: amat*5 > Out[6]: > matrix([[ 0, 5, 10, 15], > [20, 25, 30, 35], > [40, 45, 50, 55]]) Or you can do multiply(amat, mat(5)), even when everything is a matrix, and broadcasting will work. > This is an example where using arrays, as opposed to matrices, is > perhaps less confusing. > I beg to differ. This only shows that with matrices dot() will not automatically broadcast everything. IMHO that can be an advantage by preventing subtle bugs. If you want to rely on broadcasting with matrices, you could use multiply() (or kron(), depending on what you want). -sven (in his don-quijote-esque fight to keep the numpy list balanced between arrays and matrices... ;-) From aisaac at american.edu Tue Jan 2 09:04:03 2007 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 2 Jan 2007 09:04:03 -0500 Subject: [Numpy-discussion] argmax() In-Reply-To: References: Message-ID: On Mon, 1 Jan 2007, Niklas Saers apparently wrote: > If it makes any sense I'd love to send you this variable > that had len(2048) but was very much longer if stomped > flat. :-) It looks like you may already understand this, but just in case... import numpy r = numpy.random.random((100,5)) m1, m2 = numpy.argmax(r), r.argmax() print m1, m2 print r.flat[m1], r.flat[m2] try: print r[m1], r[m2] except: print "oops!" Cheers, Alan Isaac From pgmdevlist at gmail.com Tue Jan 2 09:31:33 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 2 Jan 2007 09:31:33 -0500 Subject: [Numpy-discussion] ANN: MaskedArray as a subclass of ndarray - followup Message-ID: <200701020931.33517.pgmdevlist@gmail.com> All, I've updated this famous reimplementation of maskedarray I keep ranting about. A new feature has been introduced : hard_mask. When a masked array is created with the flag hard_mask=True, the mask can only grow, not shrink. In other terms, masked values cannot be unmasked. The flag hard_mask is set to False by default. You can toggle the behavior with the `harden_mask` and `soften_mask` methods. >>> import maskedarray as MA >>> x=MA.array([1,2,3],mask=[1,0,0], hard_mask=True) >>> x[0]=999 >>> print x [-- 2 3] >>> x.soften_mask() >>> x[0]=999 >>> print x [999 2 3] I also put the file `timer_comparison.py`, that runs some unittests with each implementation (numpy.core.ma and maskedarray), and outputs the minimum times. On my machine, there doesn't seem to be a lot of differences, maskedarray being slightly faster. However, I'm not sure whether I can really trust the results. What would be the best way to compare the relative performances of numpy.core.ma and maskedarray ? Should I run tests on a function basis ? Thanks in advance for any idea/suggestion. P. From Oliver.Bock at barclaysglobal.com Tue Jan 2 18:48:19 2007 From: Oliver.Bock at barclaysglobal.com (Bock, Oliver BGI SYD) Date: Wed, 3 Jan 2007 10:48:19 +1100 Subject: [Numpy-discussion] The risks of empty() Message-ID: <7CEA131097088041B746B87258931E4A031AEFD7@sydnte2k030.insidelive.net> Some of my arrays are not fully populated. (I separately record which entries are valid.) I want to use numpy.empty() to speed up the creation of these arrays, but I'm worried about what will happen if I apply operations to the entire contents of these arrays. E.g. a + b I care about the results where valid entries align, but not otherwise. Given that numpy.empty() creates an ndarray using whatever junk it finds on the heap, it seems to me that there is the possibility that this could include bit patterns that are not valid floating point representations, which might raise floating point exceptions if used in operations like the one above (if they are "signalling" NaNs). Will this be a problem, or will the results of operations on invalid floating point numbers yield NaN? Or to put it another way: do I need to ensure that array data is initialised before using it? Oliver -- This message and any attachments are confidential, proprietary, and may be privileged. If this message was misdirected, Barclays Global Investors (BGI) does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of BGI, unless the author is authorized by BGI to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by BGI. Although BGI operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed. From robert.kern at gmail.com Tue Jan 2 19:04:13 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 02 Jan 2007 18:04:13 -0600 Subject: [Numpy-discussion] The risks of empty() In-Reply-To: <7CEA131097088041B746B87258931E4A031AEFD7@sydnte2k030.insidelive.net> References: <7CEA131097088041B746B87258931E4A031AEFD7@sydnte2k030.insidelive.net> Message-ID: <459AF2FD.2010104@gmail.com> Bock, Oliver BGI SYD wrote: > Some of my arrays are not fully populated. (I separately record which > entries are valid.) I want to use numpy.empty() to speed up the > creation of these arrays, but I'm worried about what will happen if I > apply operations to the entire contents of these arrays. E.g. > > a + b > > I care about the results where valid entries align, but not otherwise. > Given that numpy.empty() creates an ndarray using whatever junk it finds > on the heap, it seems to me that there is the possibility that this > could include bit patterns that are not valid floating point > representations, which might raise floating point exceptions if used in > operations like the one above (if they are "signalling" NaNs). Will > this be a problem, or will the results of operations on invalid floating > point numbers yield NaN? > > Or to put it another way: do I need to ensure that array data is > initialised before using it? You have essentially full control over floating point exceptions using seterr(), so you can silence even the signalling NaNs if you want. olderrstate = seterr(all='ignore') # Do stuff that might generate spurious warnings. seterr(**olderrstate) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.hochberg at ieee.org Tue Jan 2 19:08:05 2007 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 02 Jan 2007 17:08:05 -0700 Subject: [Numpy-discussion] The risks of empty() In-Reply-To: <7CEA131097088041B746B87258931E4A031AEFD7@sydnte2k030.insidelive.net> References: <7CEA131097088041B746B87258931E4A031AEFD7@sydnte2k030.insidelive.net> Message-ID: <459AF3E5.2040407@ieee.org> Bock, Oliver BGI SYD wrote: > Some of my arrays are not fully populated. (I separately record which > entries are valid.) I want to use numpy.empty() to speed up the > creation of these arrays, but I'm worried about what will happen if I > apply operations to the entire contents of these arrays. E.g. > > a + b > > I care about the results where valid entries align, but not otherwise. > Given that numpy.empty() creates an ndarray using whatever junk it finds > on the heap, it seems to me that there is the possibility that this > could include bit patterns that are not valid floating point > representations, which might raise floating point exceptions if used in > operations like the one above (if they are "signalling" NaNs). Will > this be a problem, or will the results of operations on invalid floating > point numbers yield NaN? > This depends on what the error state is set to. You can set it to ignore floating point errors, in which case this will almost certainly work. However, why take the chance. Why not just build your arrays on top of zeros instead of empty? Most of the ways that I can think of filling in a sparse array are slow enough to overwhelm the extra overhead of zeros versus empty. > Or to put it another way: do I need to ensure that array data is > initialised before using it? > I think that this should work if you set the err state correctly (for example (seterr(all="ignore"). However, I don't like shutting down the error checking unless absolutely necessary, and overall it just seems better to initialize the arrays. -tim From peridot.faceted at gmail.com Tue Jan 2 19:12:24 2007 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Tue, 2 Jan 2007 20:12:24 -0400 Subject: [Numpy-discussion] The risks of empty() In-Reply-To: <7CEA131097088041B746B87258931E4A031AEFD7@sydnte2k030.insidelive.net> References: <7CEA131097088041B746B87258931E4A031AEFD7@sydnte2k030.insidelive.net> Message-ID: On 02/01/07, Bock, Oliver BGI SYD wrote: > Some of my arrays are not fully populated. (I separately record which > entries are valid.) I want to use numpy.empty() to speed up the > creation of these arrays, but I'm worried about what will happen if I > apply operations to the entire contents of these arrays. E.g. > > a + b Have you looked at masked arrays? They are designed to do what you want. > I care about the results where valid entries align, but not otherwise. > Given that numpy.empty() creates an ndarray using whatever junk it finds > on the heap, it seems to me that there is the possibility that this > could include bit patterns that are not valid floating point > representations, which might raise floating point exceptions if used in > operations like the one above (if they are "signalling" NaNs). Will > this be a problem, or will the results of operations on invalid floating > point numbers yield NaN? There is indeed the possibility. Even with floating-point exceptions turned off, on some machines (e.g., Pentium Ms) NaNs are extremely slow to calculate with (because they are handled in software). I'm not sure that there *are* bit patterns that are not valid floating-point numbers, but in any case while using empty does not in practice seem to lead to trouble, you could have some surprising slowdowns if the array happens to be filled with NaNs. I recommend using masked arrays, which have the further advantage that values in invalid ("masked") entries are not computed at all. (If your invalid entries were few or arose naturally or you use (say) Opterons, I might recommend using NaNs to mark invalid entries.) > Or to put it another way: do I need to ensure that array data is > initialised before using it? It does not seem to be a problem in practice, but there are tools to help with what you want to do. A. M. Archibald From svetosch at gmx.net Wed Jan 3 04:00:10 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Wed, 03 Jan 2007 10:00:10 +0100 Subject: [Numpy-discussion] newbie: attempt at data frame In-Reply-To: References: Message-ID: <459B709A.2000607@gmx.net> Vincent Nijs schrieb: > If there is an easy way to read array data + variable names using the csv > module it would be great if that could be added to cookbook/InputOutput. I > couldn't figure out how to do it. > > Hi Vincent, of course it depends a little on how exactly your csv file looks like, but if you just have column headers and the actual data, you might try something like the following: import csv from numpy import mat # or array if you like file_to_read = file(filename, 'r') read_from = csv.reader(file_to_read, skipinitialspace = True) obslist = [] datalist = [] for line in read_from: obslist.append(line[0]) datalist.append(line[1:]) file_to_read.close() # (datalist should now be a nested list, first index rows, second # columns) # (still contains the headers) varnames = datalist.pop(0) # now the real data data = mat(datalist, dtype = float) -sven From eric at enthought.com Wed Jan 3 05:29:10 2007 From: eric at enthought.com (eric jones) Date: Wed, 03 Jan 2007 04:29:10 -0600 Subject: [Numpy-discussion] subclassing float64 (and friends) Message-ID: <459B8576.50005@enthought.com> Hey all, I am playing around with sub-classing the new-fangled float64 objects and friends. I really like the new ndarray subclassing features (__array_finalize__, etc.), and was exploring whether or not the scalars worked the same way. I've stubbed my toe right out of the blocks though. I can sub-class from standard python floats just fine, but when I try to do the same from float64, I get a traceback. (examples below) Anyone have ideas on how to do this correctly? thanks, eric class MyFloat(float): def __new__(cls, data, my_attr=None): obj = float.__new__(cls, data) obj.my_attr = my_attr return obj a = MyFloat(1.2,my_attr="hello") print a, a.my_attr output: 1.2 hello from numpy import float64 class MyFloat2(float64): def __new__(cls, data, my_attr=None): obj = float64.__new__(cls, data) obj.my_attr = my_attr return obj a = MyFloat2(1.2,my_attr="hello") print a, a.my_attr output: Traceback (most recent call last): File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", line 33, in ? a = MyFloat2(1.2,my_attr="hello") File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", line 30, in __new__ obj.my_attr = my_attr AttributeError: 'numpy.float64' object has no attribute 'my_attr' From svetosch at gmx.net Wed Jan 3 06:07:15 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Wed, 03 Jan 2007 12:07:15 +0100 Subject: [Numpy-discussion] newbie: attempt at data frame In-Reply-To: <459B709A.2000607@gmx.net> References: <459B709A.2000607@gmx.net> Message-ID: <459B8E63.9080401@gmx.net> Sven Schreiber schrieb: > Hi Vincent, of course it depends a little on how exactly your csv file > looks like, but if you just have column headers and the actual data, you > might try something like the following: > Ok sorry the previous thing doesn't work, I also stumbled over the strings. Here's the next attempt, also shorter. (this time even tested ;-) import csv from numpy import mat read_from = csv.reader(file(filename, 'r'), skipinitialspace = True) stringlist = [ line for line in read_from ] varnames = stringlist.pop(0)[1:] datalist = [ map(float, line[1:]) for line in stringlist ] # now the real data data = mat(datalist, dtype = float) I actually quite like it... python lists are very nice. This discards the observation labels, but it's not difficult to add that, of course. -sven From stefan at sun.ac.za Wed Jan 3 07:44:45 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 3 Jan 2007 14:44:45 +0200 Subject: [Numpy-discussion] subclassing float64 (and friends) In-Reply-To: <459B8576.50005@enthought.com> References: <459B8576.50005@enthought.com> Message-ID: <20070103124445.GD5779@mentat.za.net> On Wed, Jan 03, 2007 at 04:29:10AM -0600, eric jones wrote: > I am playing around with sub-classing the new-fangled float64 objects > and friends. I really like the new ndarray subclassing features > (__array_finalize__, etc.), and was exploring whether or not the scalars > worked the same way. I've stubbed my toe right out of the blocks > though. I can sub-class from standard python floats just fine, but when > I try to do the same from float64, I get a traceback. (examples below) > Anyone have ideas on how to do this correctly? > > from numpy import float64 > > class MyFloat2(float64): > > def __new__(cls, data, my_attr=None): > obj = float64.__new__(cls, data) > obj.my_attr = my_attr > return obj > > a = MyFloat2(1.2,my_attr="hello") > print a, a.my_attr > > > output: > Traceback (most recent call last): > File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", > line 33, in ? > a = MyFloat2(1.2,my_attr="hello") > File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", > line 30, in __new__ > obj.my_attr = my_attr > AttributeError: 'numpy.float64' object has no attribute 'my_attr' With classes defined in C I've often noticed that you can't add attributes, i.e. f = N.float64(1.2) f.x = 1 breaks with AttributeError: 'numpy.float64' object has no attribute 'x' The way to fix this for arrays is to first view the array as the new subclass, i.e. x = N.array([1]) class Ary(N.ndarray): pass x = x.view(Ary) x.y = 1 However, with floats I noticed that the following fails: import numpy as N f = N.float64(1.2) class Floaty(N.float64): pass f.view(Floaty) with TypeError: data type cannot be determined from type object Maybe this is part of the problem? Regards St?fan From mkg at cs.nyu.edu Wed Jan 3 15:39:51 2007 From: mkg at cs.nyu.edu (Matthew Koichi Grimes) Date: Wed, 03 Jan 2007 15:39:51 -0500 Subject: [Numpy-discussion] returning recarray records as plain arrays In-Reply-To: References: Message-ID: <459C1497.7000705@cs.nyu.edu> As per Stefan's help, I've made a subclass of recarray called nnvalue. It just fixes the dtype to [('x', 'f8'), ('dx', 'f8'), ('delta', 'f8)], and adds a few member functions. I basically treat nnvalue as a struct with three equal-shaped array fields: x, dx, and delta. I'd like it if, when I reference individual fields of a nnvalue, they were returned as plain arrays. Instead, they are returned as nnarrays with empty .x, .dx, and .deltas fields. These fields' dType is also still set to [('x', 'f8'), ('dx', 'f8'), ('delta', 'f8)], when I wish they'd just be 'f8'. The following illustrates the problem, followed by the awkward workaround I'm currently using: >>> nnv = nnvalue(3) # a 3-element nnvalue >>> nnv nnvalue([(0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0)], dtype=[('x', '>> nnv.x nnvalue([0.0, 0.0, 0.0], dtype=[('x', '>> nnv.x.view(dtype('f8')) array([ 0., 0., 0.]) How can I get my recarray subclass to return individual records as plain array views? -- Matt numpy-discussion-request at scipy.org wrote: > Send Numpy-discussion mailing list submissions to > numpy-discussion at scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://projects.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request at scipy.org > > You can reach the person managing the list at > numpy-discussion-owner at scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Numpy-discussion digest..." > > ------------------------------------------------------------------------ > > Today's Topics: > > 1. The risks of empty() (Bock, Oliver BGI SYD) > 2. Re: The risks of empty() (Robert Kern) > 3. Re: The risks of empty() (Tim Hochberg) > 4. Re: The risks of empty() (A. M. Archibald) > 5. Re: newbie: attempt at data frame (Sven Schreiber) > 6. subclassing float64 (and friends) (eric jones) > 7. Re: newbie: attempt at data frame (Sven Schreiber) > 8. Re: subclassing float64 (and friends) (Stefan van der Walt) > > > ------------------------------------------------------------------------ > > Subject: > [Numpy-discussion] The risks of empty() > From: > "Bock, Oliver BGI SYD" > Date: > Wed, 3 Jan 2007 10:48:19 +1100 > To: > > > To: > > > > Some of my arrays are not fully populated. (I separately record which > entries are valid.) I want to use numpy.empty() to speed up the > creation of these arrays, but I'm worried about what will happen if I > apply operations to the entire contents of these arrays. E.g. > > a + b > > I care about the results where valid entries align, but not otherwise. > Given that numpy.empty() creates an ndarray using whatever junk it finds > on the heap, it seems to me that there is the possibility that this > could include bit patterns that are not valid floating point > representations, which might raise floating point exceptions if used in > operations like the one above (if they are "signalling" NaNs). Will > this be a problem, or will the results of operations on invalid floating > point numbers yield NaN? > > Or to put it another way: do I need to ensure that array data is > initialised before using it? > > > Oliver > > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] The risks of empty() > From: > Robert Kern > Date: > Tue, 02 Jan 2007 18:04:13 -0600 > To: > Discussion of Numerical Python > > To: > Discussion of Numerical Python > > > Bock, Oliver BGI SYD wrote: > >> Some of my arrays are not fully populated. (I separately record which >> entries are valid.) I want to use numpy.empty() to speed up the >> creation of these arrays, but I'm worried about what will happen if I >> apply operations to the entire contents of these arrays. E.g. >> >> a + b >> >> I care about the results where valid entries align, but not otherwise. >> Given that numpy.empty() creates an ndarray using whatever junk it finds >> on the heap, it seems to me that there is the possibility that this >> could include bit patterns that are not valid floating point >> representations, which might raise floating point exceptions if used in >> operations like the one above (if they are "signalling" NaNs). Will >> this be a problem, or will the results of operations on invalid floating >> point numbers yield NaN? >> >> Or to put it another way: do I need to ensure that array data is >> initialised before using it? >> > > You have essentially full control over floating point exceptions using seterr(), > so you can silence even the signalling NaNs if you want. > > olderrstate = seterr(all='ignore') > # Do stuff that might generate spurious warnings. > seterr(**olderrstate) > > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] The risks of empty() > From: > Tim Hochberg > Date: > Tue, 02 Jan 2007 17:08:05 -0700 > To: > Discussion of Numerical Python > > To: > Discussion of Numerical Python > > > Bock, Oliver BGI SYD wrote: >> Some of my arrays are not fully populated. (I separately record which >> entries are valid.) I want to use numpy.empty() to speed up the >> creation of these arrays, but I'm worried about what will happen if I >> apply operations to the entire contents of these arrays. E.g. >> >> a + b >> >> I care about the results where valid entries align, but not otherwise. >> Given that numpy.empty() creates an ndarray using whatever junk it finds >> on the heap, it seems to me that there is the possibility that this >> could include bit patterns that are not valid floating point >> representations, which might raise floating point exceptions if used in >> operations like the one above (if they are "signalling" NaNs). Will >> this be a problem, or will the results of operations on invalid floating >> point numbers yield NaN? >> > This depends on what the error state is set to. You can set it to > ignore floating point errors, in which case this will almost certainly > work. > > However, why take the chance. Why not just build your arrays on top of > zeros instead of empty? Most of the ways that I can think of filling > in a sparse array are slow enough to overwhelm the extra overhead of > zeros versus empty. >> Or to put it another way: do I need to ensure that array data is >> initialised before using it? >> > I think that this should work if you set the err state correctly (for > example (seterr(all="ignore"). However, I don't like shutting down the > error checking unless absolutely necessary, and overall it just seems > better to initialize the arrays. > > -tim > > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] The risks of empty() > From: > "A. M. Archibald" > Date: > Tue, 2 Jan 2007 20:12:24 -0400 > To: > "Discussion of Numerical Python" > > To: > "Discussion of Numerical Python" > > > On 02/01/07, Bock, Oliver BGI SYD wrote: >> Some of my arrays are not fully populated. (I separately record which >> entries are valid.) I want to use numpy.empty() to speed up the >> creation of these arrays, but I'm worried about what will happen if I >> apply operations to the entire contents of these arrays. E.g. >> >> a + b > > Have you looked at masked arrays? They are designed to do what you want. > >> I care about the results where valid entries align, but not otherwise. >> Given that numpy.empty() creates an ndarray using whatever junk it finds >> on the heap, it seems to me that there is the possibility that this >> could include bit patterns that are not valid floating point >> representations, which might raise floating point exceptions if used in >> operations like the one above (if they are "signalling" NaNs). Will >> this be a problem, or will the results of operations on invalid floating >> point numbers yield NaN? > > There is indeed the possibility. Even with floating-point exceptions > turned off, on some machines (e.g., Pentium Ms) NaNs are extremely > slow to calculate with (because they are handled in software). I'm not > sure that there *are* bit patterns that are not valid floating-point > numbers, but in any case while using empty does not in practice seem > to lead to trouble, you could have some surprising slowdowns if the > array happens to be filled with NaNs. > > I recommend using masked arrays, which have the further advantage that > values in invalid ("masked") entries are not computed at all. (If your > invalid entries were few or arose naturally or you use (say) Opterons, > I might recommend using NaNs to mark invalid entries.) > >> Or to put it another way: do I need to ensure that array data is >> initialised before using it? > > It does not seem to be a problem in practice, but there are tools to > help with what you want to do. > > A. M. Archibald > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] newbie: attempt at data frame > From: > Sven Schreiber > Date: > Wed, 03 Jan 2007 10:00:10 +0100 > To: > Discussion of Numerical Python > > To: > Discussion of Numerical Python > > > Vincent Nijs schrieb: > > >> If there is an easy way to read array data + variable names using the csv >> module it would be great if that could be added to cookbook/InputOutput. I >> couldn't figure out how to do it. >> >> >> > > Hi Vincent, of course it depends a little on how exactly your csv file > looks like, but if you just have column headers and the actual data, you > might try something like the following: > > import csv > from numpy import mat # or array if you like > file_to_read = file(filename, 'r') > read_from = csv.reader(file_to_read, skipinitialspace = True) > obslist = [] > datalist = [] > for line in read_from: > obslist.append(line[0]) > datalist.append(line[1:]) > file_to_read.close() > # (datalist should now be a nested list, first index rows, second > # columns) > # (still contains the headers) > varnames = datalist.pop(0) > # now the real data > data = mat(datalist, dtype = float) > > -sven > > > > ------------------------------------------------------------------------ > > Subject: > [Numpy-discussion] subclassing float64 (and friends) > From: > eric jones > Date: > Wed, 03 Jan 2007 04:29:10 -0600 > To: > Discussion of Numerical Python > > To: > Discussion of Numerical Python > > > Hey all, > > I am playing around with sub-classing the new-fangled float64 objects > and friends. I really like the new ndarray subclassing features > (__array_finalize__, etc.), and was exploring whether or not the > scalars worked the same way. I've stubbed my toe right out of the > blocks though. I can sub-class from standard python floats just fine, > but when I try to do the same from float64, I get a traceback. > (examples below) Anyone have ideas on how to do this correctly? > > thanks, > eric > > class MyFloat(float): > > def __new__(cls, data, my_attr=None): > obj = float.__new__(cls, data) > obj.my_attr = my_attr > return obj > > a = MyFloat(1.2,my_attr="hello") > print a, a.my_attr > > output: > 1.2 hello > > > from numpy import float64 > > class MyFloat2(float64): > > def __new__(cls, data, my_attr=None): > obj = float64.__new__(cls, data) > obj.my_attr = my_attr > return obj > > a = MyFloat2(1.2,my_attr="hello") > print a, a.my_attr > > > output: > Traceback (most recent call last): > File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", > line 33, in ? > a = MyFloat2(1.2,my_attr="hello") > File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", > line 30, in __new__ > obj.my_attr = my_attr > AttributeError: 'numpy.float64' object has no attribute 'my_attr' > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] newbie: attempt at data frame > From: > Sven Schreiber > Date: > Wed, 03 Jan 2007 12:07:15 +0100 > To: > Discussion of Numerical Python > > To: > Discussion of Numerical Python > > > Sven Schreiber schrieb: > > >> Hi Vincent, of course it depends a little on how exactly your csv file >> looks like, but if you just have column headers and the actual data, you >> might try something like the following: >> >> > > Ok sorry the previous thing doesn't work, I also stumbled over the > strings. Here's the next attempt, also shorter. (this time even tested ;-) > > import csv > from numpy import mat > > read_from = csv.reader(file(filename, 'r'), skipinitialspace = True) > stringlist = [ line for line in read_from ] > varnames = stringlist.pop(0)[1:] > datalist = [ map(float, line[1:]) for line in stringlist ] > > # now the real data > data = mat(datalist, dtype = float) > > > I actually quite like it... python lists are very nice. This discards > the observation labels, but it's not difficult to add that, of course. > > -sven > > > > ------------------------------------------------------------------------ > > Subject: > Re: [Numpy-discussion] subclassing float64 (and friends) > From: > Stefan van der Walt > Date: > Wed, 3 Jan 2007 14:44:45 +0200 > To: > Discussion of Numerical Python > > To: > Discussion of Numerical Python > > > On Wed, Jan 03, 2007 at 04:29:10AM -0600, eric jones wrote: > >> I am playing around with sub-classing the new-fangled float64 objects >> and friends. I really like the new ndarray subclassing features >> (__array_finalize__, etc.), and was exploring whether or not the scalars >> worked the same way. I've stubbed my toe right out of the blocks >> though. I can sub-class from standard python floats just fine, but when >> I try to do the same from float64, I get a traceback. (examples below) >> Anyone have ideas on how to do this correctly? >> >> from numpy import float64 >> >> class MyFloat2(float64): >> >> def __new__(cls, data, my_attr=None): >> obj = float64.__new__(cls, data) >> obj.my_attr = my_attr >> return obj >> >> a = MyFloat2(1.2,my_attr="hello") >> print a, a.my_attr >> >> >> output: >> Traceback (most recent call last): >> File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", >> line 33, in ? >> a = MyFloat2(1.2,my_attr="hello") >> File "C:\wrk\eric\trunk\src\lib\geo\examples\scalar_subtype.py", >> line 30, in __new__ >> obj.my_attr = my_attr >> AttributeError: 'numpy.float64' object has no attribute 'my_attr' >> > > With classes defined in C I've often noticed that you can't add > attributes, i.e. > > f = N.float64(1.2) > f.x = 1 > > breaks with > > AttributeError: 'numpy.float64' object has no attribute 'x' > > The way to fix this for arrays is to first view the array as the new > subclass, i.e. > > x = N.array([1]) > class Ary(N.ndarray): > pass > x = x.view(Ary) > x.y = 1 > > However, with floats I noticed that the following fails: > > import numpy as N > f = N.float64(1.2) > class Floaty(N.float64): > pass > f.view(Floaty) > > with > > TypeError: data type cannot be determined from type object > > Maybe this is part of the problem? > > Regards > St?fan > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From pgmdevlist at gmail.com Wed Jan 3 16:05:27 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 3 Jan 2007 16:05:27 -0500 Subject: [Numpy-discussion] returning recarray records as plain arrays In-Reply-To: <459C1497.7000705@cs.nyu.edu> References: <459C1497.7000705@cs.nyu.edu> Message-ID: <200701031605.28262.pgmdevlist@gmail.com> On Wednesday 03 January 2007 15:39, Matthew Koichi Grimes wrote: > As per Stefan's help, I've made a subclass of recarray called nnvalue. > It just fixes the dtype to [('x', 'f8'), ('dx', 'f8'), ('delta', 'f8)], > and adds a few member functions. I basically treat nnvalue as a struct > with three equal-shaped array fields: x, dx, and delta. Matthew, Please try to post (pieces of) your code, so that we can have a better idea of how to help you. Thanks in advance P. From oliphant at ee.byu.edu Wed Jan 3 16:53:43 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 03 Jan 2007 14:53:43 -0700 Subject: [Numpy-discussion] subclassing float64 (and friends) In-Reply-To: <459B8576.50005@enthought.com> References: <459B8576.50005@enthought.com> Message-ID: <459C25E7.8040504@ee.byu.edu> eric jones wrote: >Hey all, > >I am playing around with sub-classing the new-fangled float64 objects >and friends. I really like the new ndarray subclassing features >(__array_finalize__, etc.), and was exploring whether or not the scalars >worked the same way. I've stubbed my toe right out of the blocks >though. I can sub-class from standard python floats just fine, but when >I try to do the same from float64, I get a traceback. (examples below) >Anyone have ideas on how to do this correctly? > > Unfortunately, the array scalars were not originally designed to be sub-classed. I believe I later set the flag saying they "could be" sub-classable. However, I did not alter the tp_new code to actually use the type information passed in as the first argument to actually create an instance of the sub-class --- effectively making the array scalars *not* sub-classable. Thus, the _arrtype_new code would need to be changed in scalartypes.inc.src to fix this. Notice that in Python the float_new command does a special check for when a sub-type is requested and calls float_subtype_new which creates a regular float and then copies the data over to the sub-type. This special-check is not done in the tp_new function (_arrtype_new) and so float64.__new__ will *always* return the array scalars are still not sub-classable (except for the void scalar). This could be changed, and needs to happen before sub-classing of array scalars will work. -Travis From oliphant at ee.byu.edu Wed Jan 3 17:30:30 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 03 Jan 2007 15:30:30 -0700 Subject: [Numpy-discussion] Scipy - Numpy incompatiblility when calling upon Old Numeric In-Reply-To: References: Message-ID: <459C2E86.8010207@ee.byu.edu> frbeaxs wrote: > I am using Python 2.4 with Numpy 0.9.8. Matpylib graphs function > under these two version except when old numeric must be called to > utilized the spline function in contouring the graphs leading to the > error: > The version of NumPy compatible with different versions of matplotlib is very important. Each release of Matplotlib should indicate which version of NumPy it works with. Typically, the latest release of matplotlib works with the latest release of NumPy (such should be the case now). -Travis From nwagner at iam.uni-stuttgart.de Wed Jan 3 17:31:20 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Wed, 03 Jan 2007 23:31:20 +0100 Subject: [Numpy-discussion] OpenSuse 10.2 and numpy Message-ID: Hi all, I have tried to build the latest numpy on a 64-bit machine using OpenSuSE10.2. I have build BLAS, LAPACK and ATLAS from scratch (g77). python setup.py build yields ... creating build/temp.linux-x86_64-2.5/numpy/linalg compile options: '-DATLAS_INFO="\"3.7.11\"" -Inumpy/core/include -Ibuild/src.linux-x86_64-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c' gcc: numpy/linalg/lapack_litemodule.c /usr/bin/g77 -g -Wall -shared build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o -L/usr/local/lib/atlas -L/usr/lib/python2.5/config -llapack -lf77blas -lcblas -latlas -lpython2.5 -lg2c -o build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: cannot find -lgcc_s /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: cannot find -lgcc_s error: Command "/usr/bin/g77 -g -Wall -shared build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o -L/usr/local/lib/atlas -L/usr/lib/python2.5/config -llapack -lf77blas -lcblas -latlas -lpython2.5 -lg2c -o build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so" failed with exit status 1 Any pointer how to fix this problem is appreciated. Nils g77 -v Reading specs from /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/specs Configured with: ../configure --enable-threads=posix --prefix=/usr --with-local-prefix=/usr/local --infodir=/usr/share/info --mandir=/usr/share/man --enable-languages=c,f77 --disable-checking --libdir=/usr/lib64 --disable-libgcj --with-slibdir=/lib64 --with-system-zlib --enable-shared --enable-__cxa_atexit x86_64-suse-linux Thread model: posix gcc version 3.3.5 20050117 (prerelease) (SUSE Linux) nwagner at noname:~/svn/numpy> gcc -v Using built-in specs. Target: x86_64-suse-linux Configured with: ../configure --enable-threads=posix --prefix=/usr --with-local-prefix=/usr/local --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.1.2 --enable-ssp --disable-libssp --disable-libgcj --with-slibdir=/lib64 --with-system-zlib --enable-shared --enable-__cxa_atexit --enable-libstdcxx-allocator=new --program-suffix=-4.1 --enable-version-specific-runtime-libs --without-system-libunwind --with-cpu=generic --host=x86_64-suse-linux Thread model: posix gcc version 4.1.2 20061115 (prerelease) (SUSE Linux) From eric at enthought.com Wed Jan 3 18:12:40 2007 From: eric at enthought.com (eric jones) Date: Wed, 03 Jan 2007 17:12:40 -0600 Subject: [Numpy-discussion] subclassing float64 (and friends) In-Reply-To: <459C25E7.8040504@ee.byu.edu> References: <459B8576.50005@enthought.com> <459C25E7.8040504@ee.byu.edu> Message-ID: <459C3868.40406@enthought.com> Thanks for the update. For now, I'll try doing what I need to by sub-classing float. But, I'm gonna miss __array_finalize__ :-). eric Travis Oliphant wrote: > eric jones wrote: > > >> Hey all, >> >> I am playing around with sub-classing the new-fangled float64 objects >> and friends. I really like the new ndarray subclassing features >> (__array_finalize__, etc.), and was exploring whether or not the scalars >> worked the same way. I've stubbed my toe right out of the blocks >> though. I can sub-class from standard python floats just fine, but when >> I try to do the same from float64, I get a traceback. (examples below) >> Anyone have ideas on how to do this correctly? >> >> >> > > Unfortunately, the array scalars were not originally designed to be > sub-classed. I believe I later set the flag saying they "could be" > sub-classable. However, I did not alter the tp_new code to actually use > the type information passed in as the first argument to actually create > an instance of the sub-class --- effectively making the array scalars > *not* sub-classable. > > Thus, the _arrtype_new code would need to be changed in > scalartypes.inc.src to fix this. Notice that in Python the float_new > command does a special check for when a sub-type is requested and calls > float_subtype_new which creates a regular float and then copies the data > over to the sub-type. > > This special-check is not done in the tp_new function > (_arrtype_new) and so float64.__new__ will *always* return the > array scalars are still not sub-classable (except for the void > scalar). This could be changed, and needs to happen before > sub-classing of array scalars will work. > > -Travis > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From mkg at cs.nyu.edu Wed Jan 3 23:57:43 2007 From: mkg at cs.nyu.edu (Matthew Koichi Grimes) Date: Wed, 03 Jan 2007 23:57:43 -0500 Subject: [Numpy-discussion] returning recarray records as plain arrays In-Reply-To: <200701031605.28262.pgmdevlist@gmail.com> References: <459C1497.7000705@cs.nyu.edu> <200701031605.28262.pgmdevlist@gmail.com> Message-ID: <459C8947.9060708@cs.nyu.edu> Pierre GM wrote: > On Wednesday 03 January 2007 15:39, Matthew Koichi Grimes wrote: > >> As per Stefan's help, I've made a subclass of recarray called nnvalue. >> It just fixes the dtype to [('x', 'f8'), ('dx', 'f8'), ('delta', 'f8)], >> and adds a few member functions. I basically treat nnvalue as a struct >> with three equal-shaped array fields: x, dx, and delta. >> > > Matthew, > Please try to post (pieces of) your code, so that we can have a better idea of > how to help you. > Thanks in advance > P. > It's essentially unchanged from Stefan's code: import numpy as N class nnvalue(N.recarray): dtype = N.dtype([('x','f8'),('dx','f8'),('deltas','f8')]) def __new__(subtype,shape): data = N.zeros(shape,subtype.dtype) return data.view(subtype) def __array_finalize__(self,obj): pass # for now def clearGradient(self): """Fills all members other than x with zeros.""" self['dx'][:] = 0. self['deltas'][:] = 0. From charlesr.harris at gmail.com Thu Jan 4 00:37:39 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 3 Jan 2007 22:37:39 -0700 Subject: [Numpy-discussion] OpenSuse 10.2 and numpy In-Reply-To: References: Message-ID: On 1/3/07, Nils Wagner wrote: > > Hi all, > > I have tried to build the latest numpy on a 64-bit machine > using OpenSuSE10.2. > I have build BLAS, LAPACK and ATLAS from scratch (g77). > > python setup.py build yields > > ... > creating build/temp.linux-x86_64-2.5/numpy/linalg > compile options: '-DATLAS_INFO="\"3.7.11\"" > -Inumpy/core/include > -Ibuild/src.linux-x86_64-2.5/numpy/core -Inumpy/core/src > -Inumpy/core/include -I/usr/include/python2.5 -c' > gcc: numpy/linalg/lapack_litemodule.c > /usr/bin/g77 -g -Wall -shared > build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o > -L/usr/local/lib/atlas -L/usr/lib/python2.5/config > -llapack -lf77blas -lcblas -latlas -lpython2.5 -lg2c -o > build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so > > /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: > cannot find -lgcc_s > > /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: > cannot find -lgcc_s > error: Command "/usr/bin/g77 -g -Wall -shared > build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o > -L/usr/local/lib/atlas -L/usr/lib/python2.5/config > -llapack -lf77blas -lcblas -latlas -lpython2.5 -lg2c -o > build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so" > failed with exit status 1 > > Any pointer how to fix this problem is appreciated. > > Nils > > g77 -v > Reading specs from > /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/specs > Configured with: ../configure --enable-threads=posix > --prefix=/usr --with-local-prefix=/usr/local > --infodir=/usr/share/info --mandir=/usr/share/man > --enable-languages=c,f77 --disable-checking > --libdir=/usr/lib64 --disable-libgcj --with-slibdir=/lib64 > --with-system-zlib --enable-shared --enable-__cxa_atexit > x86_64-suse-linux > Thread model: posix > gcc version 3.3.5 20050117 (prerelease) (SUSE Linux) > nwagner at noname:~/svn/numpy> gcc -v > Using built-in specs. > Target: x86_64-suse-linux > Configured with: ../configure --enable-threads=posix > --prefix=/usr --with-local-prefix=/usr/local > --infodir=/usr/share/info --mandir=/usr/share/man > --libdir=/usr/lib64 --libexecdir=/usr/lib64 > --enable-languages=c,c++,objc,fortran,obj-c++,java,ada > --enable-checking=release > --with-gxx-include-dir=/usr/include/c++/4.1.2 --enable-ssp > --disable-libssp --disable-libgcj --with-slibdir=/lib64 > --with-system-zlib --enable-shared --enable-__cxa_atexit > --enable-libstdcxx-allocator=new --program-suffix=-4.1 > --enable-version-specific-runtime-libs > --without-system-libunwind --with-cpu=generic > --host=x86_64-suse-linux > Thread model: posix > gcc version 4.1.2 20061115 (prerelease) (SUSE Linux) You might check the location of python. ISTR that SUSE puts it in /usr/local/lib, or maybe it is in one of the lib64 directories if you are running 64 bit linux. In any case, the directories -L/usr/local/lib/atlas -L/usr/lib/python2.5/config look suspicious to me. Apart from that I don't have any suggestions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Thu Jan 4 01:40:54 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 4 Jan 2007 01:40:54 -0500 Subject: [Numpy-discussion] returning recarray records as plain arrays In-Reply-To: <459C8947.9060708@cs.nyu.edu> References: <200701031605.28262.pgmdevlist@gmail.com> <459C8947.9060708@cs.nyu.edu> Message-ID: <200701040140.54440.pgmdevlist@gmail.com> On Wednesday 03 January 2007 23:57, Matthew Koichi Grimes wrote: > Pierre GM wrote: > > On Wednesday 03 January 2007 15:39, Matthew Koichi Grimes wrote: > >> As per Stefan's help, I've made a subclass of recarray called nnvalue. > >> It just fixes the dtype to [('x', 'f8'), ('dx', 'f8'), ('delta', 'f8)], > >> and adds a few member functions. I basically treat nnvalue as a struct > >> with three equal-shaped array fields: x, dx, and delta. Matthew, It seems that declaring your default dtype as a class argument and naming it dtype is the reason why things are messed up: you're basically overwriting the original ndarray.dtype You have 2 options: 1. Put the definition of your base dtype inside __new__: it's no longer a class argument, so it's not overwriting anything class nnvalue(N.recarray): ? ? def __new__(subtype,shape): _dtype = N.dtype([('x','f8'),('dx','f8'),('deltas','f8')]) data = N.zeros(shape,subtype.dtype) ? ? ? ? return data.view(subtype) 2. Call it something else: class nnvalue(N.recarray): __defaultdtype = N.dtype([('x','f8'),('dx','f8'),('deltas','f8')]) def __new__(subtype,shape): data = N.zeros(shape,subtype.__defaultdtype ) return data.view(subtype) From nwagner at iam.uni-stuttgart.de Thu Jan 4 03:45:51 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 04 Jan 2007 09:45:51 +0100 Subject: [Numpy-discussion] OpenSuse 10.2 and numpy In-Reply-To: References: Message-ID: On Wed, 3 Jan 2007 22:37:39 -0700 "Charles R Harris" wrote: > On 1/3/07, Nils Wagner >wrote: >> >> Hi all, >> >> I have tried to build the latest numpy on a 64-bit >>machine >> using OpenSuSE10.2. >> I have build BLAS, LAPACK and ATLAS from scratch (g77). >> >> python setup.py build yields >> >> ... >> creating build/temp.linux-x86_64-2.5/numpy/linalg >> compile options: '-DATLAS_INFO="\"3.7.11\"" >> -Inumpy/core/include >> -Ibuild/src.linux-x86_64-2.5/numpy/core -Inumpy/core/src >> -Inumpy/core/include -I/usr/include/python2.5 -c' >> gcc: numpy/linalg/lapack_litemodule.c >> /usr/bin/g77 -g -Wall -shared >> build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o >> -L/usr/local/lib/atlas -L/usr/lib/python2.5/config >> -llapack -lf77blas -lcblas -latlas -lpython2.5 -lg2c -o >> build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so >> >> /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: >> cannot find -lgcc_s >> >> /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: >> cannot find -lgcc_s >> error: Command "/usr/bin/g77 -g -Wall -shared >> build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o >> -L/usr/local/lib/atlas -L/usr/lib/python2.5/config >> -llapack -lf77blas -lcblas -latlas -lpython2.5 -lg2c -o >> build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so" >> failed with exit status 1 >> >> Any pointer how to fix this problem is appreciated. >> >> Nils >> >> g77 -v >> Reading specs from >> /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/specs >> Configured with: ../configure --enable-threads=posix >> --prefix=/usr --with-local-prefix=/usr/local >> --infodir=/usr/share/info --mandir=/usr/share/man >> --enable-languages=c,f77 --disable-checking >> --libdir=/usr/lib64 --disable-libgcj >>--with-slibdir=/lib64 >> --with-system-zlib --enable-shared --enable-__cxa_atexit >> x86_64-suse-linux >> Thread model: posix >> gcc version 3.3.5 20050117 (prerelease) (SUSE Linux) >> nwagner at noname:~/svn/numpy> gcc -v >> Using built-in specs. >> Target: x86_64-suse-linux >> Configured with: ../configure --enable-threads=posix >> --prefix=/usr --with-local-prefix=/usr/local >> --infodir=/usr/share/info --mandir=/usr/share/man >> --libdir=/usr/lib64 --libexecdir=/usr/lib64 >> --enable-languages=c,c++,objc,fortran,obj-c++,java,ada >> --enable-checking=release >> --with-gxx-include-dir=/usr/include/c++/4.1.2 >>--enable-ssp >> --disable-libssp --disable-libgcj --with-slibdir=/lib64 >> --with-system-zlib --enable-shared --enable-__cxa_atexit >> --enable-libstdcxx-allocator=new --program-suffix=-4.1 >> --enable-version-specific-runtime-libs >> --without-system-libunwind --with-cpu=generic >> --host=x86_64-suse-linux >> Thread model: posix >> gcc version 4.1.2 20061115 (prerelease) (SUSE Linux) > > > You might check the location of python. ISTR that SUSE >puts it in > /usr/local/lib, or maybe it is in one of the lib64 >directories if you are > running 64 bit linux. In any case, the directories >-L/usr/local/lib/atlas > -L/usr/lib/python2.5/config look suspicious to me. Apart >from that I don't > have any suggestions. > > Chuck I guess the problem is due to /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: cannot find -lgcc_s /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: cannot find -lgcc_s locate libgcc_s yields /lib64/libgcc_s.so.1 /lib/libgcc_s.so.1 /usr/lib64/gcc/x86_64-suse-linux/4.1.2/libgcc_s.so I am confused by the fact that lapack_lite.so is involved. I thought that it is obsolete if ATLAS is installed. Am I missing something ? I have build BLAS and LAPACK with g77. gfortran seems to be the new standard. Is that correct ? Should I build BLAS and LAPACK with gfortran ? Nils From meesters at uni-mainz.de Thu Jan 4 05:50:37 2007 From: meesters at uni-mainz.de (Christian Meesters) Date: Thu, 4 Jan 2007 11:50:37 +0100 Subject: [Numpy-discussion] OpenSuse 10.2 and numpy In-Reply-To: References: Message-ID: <200701041150.37790.meesters@uni-mainz.de> Hoi Nils, I'm still with 10.0 on my machine, but perhaps this helps: I followed exactely the instructions given here: http://pong.tamu.edu/tiki/tiki-view_blog_post.php?blogId=6&postId=97 and also sticked exactly to the recommended compiler flags given here: http://www.scipy.org/Installing_SciPy/BuildingGeneral I suggest you check your makefile(s). At least doing so worked for me (using g77 and gcc 4.0.2 on AMD 64 bit) ... HTH Christian From nwagner at iam.uni-stuttgart.de Thu Jan 4 06:12:06 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 04 Jan 2007 12:12:06 +0100 Subject: [Numpy-discussion] OpenSuse 10.2 and numpy In-Reply-To: <200701041150.37790.meesters@uni-mainz.de> References: <200701041150.37790.meesters@uni-mainz.de> Message-ID: On Thu, 4 Jan 2007 11:50:37 +0100 Christian Meesters wrote: > Hoi Nils, > > I'm still with 10.0 on my machine, but perhaps this >helps: > I followed exactely the instructions given here: > http://pong.tamu.edu/tiki/tiki-view_blog_post.php?blogId=6&postId=97 > and also sticked exactly to the recommended compiler >flags given here: > http://www.scipy.org/Installing_SciPy/BuildingGeneral > > I suggest you check your makefile(s). At least doing so >worked for me (using > g77 and gcc 4.0.2 on AMD 64 bit) ... > > HTH > Christian > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion I build the libraries in exact accordance with these instructions. What is the meaning of libgcc_s ? locate libgcc /lib64/libgcc_s.so.1 /lib/libgcc_s.so.1 /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/32/libgcc.a /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/32/libgcc_eh.a /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/libgcc.a /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/libgcc_eh.a /usr/lib64/gcc/x86_64-suse-linux/4.1.2/libgcc.a /usr/lib64/gcc/x86_64-suse-linux/4.1.2/libgcc_eh.a /usr/lib64/gcc/x86_64-suse-linux/4.1.2/libgcc_s.so /usr/lib/gcc/i586-suse-linux/2.95.3/libgcc.a /usr/lib/ooo-2.0/program/libgcc3_uno.so Nils From charlesr.harris at gmail.com Thu Jan 4 12:42:35 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 4 Jan 2007 10:42:35 -0700 Subject: [Numpy-discussion] OpenSuse 10.2 and numpy In-Reply-To: References: Message-ID: On 1/4/07, Nils Wagner wrote: > > On Wed, 3 Jan 2007 22:37:39 -0700 > "Charles R Harris" wrote: > > On 1/3/07, Nils Wagner > >wrote: > > I guess the problem is due to > > /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: > cannot find -lgcc_s > > /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: > cannot find -lgcc_s > > locate libgcc_s yields > > /lib64/libgcc_s.so.1 > /lib/libgcc_s.so.1 > /usr/lib64/gcc/x86_64-suse-linux/4.1.2/libgcc_s.so > > I am confused by the fact that lapack_lite.so is involved. > I thought that it is obsolete if ATLAS is installed. > Am I missing something ? I suspect that ATLAS wasn't found, you might need to edit site.cfg to give the build system some help. Also, a lot of the libgcc_s.so locations you found are probably symlinks. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwagner at iam.uni-stuttgart.de Thu Jan 4 13:57:28 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 04 Jan 2007 19:57:28 +0100 Subject: [Numpy-discussion] OpenSuse 10.2 cannot find -lgcc_s In-Reply-To: References: Message-ID: On Thu, 4 Jan 2007 10:42:35 -0700 "Charles R Harris" wrote: > On 1/4/07, Nils Wagner >wrote: >> >> On Wed, 3 Jan 2007 22:37:39 -0700 >> "Charles R Harris" wrote: >> > On 1/3/07, Nils Wagner >> >wrote: > > > > > >> >> I guess the problem is due to >> >> /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: >> cannot find -lgcc_s >> >> /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.5/../../../../x86_64-suse-linux/bin/ld: >> cannot find -lgcc_s >> >> locate libgcc_s yields >> >> /lib64/libgcc_s.so.1 >> /lib/libgcc_s.so.1 >> /usr/lib64/gcc/x86_64-suse-linux/4.1.2/libgcc_s.so >> >> I am confused by the fact that lapack_lite.so is >>involved. >> I thought that it is obsolete if ATLAS is installed. >> Am I missing something ? > > > I suspect that ATLAS wasn't found, you might need to >edit site.cfg to give > the build system some help. Also, a lot of the >libgcc_s.so locations you > found are probably symlinks. > > Chuck There exists a bug report https://bugzilla.novell.com/show_bug.cgi?id=218406 Nils From bthom at cs.hmc.edu Thu Jan 4 14:18:39 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Thu, 04 Jan 2007 11:18:39 -0800 Subject: [Numpy-discussion] ndarray newbie question Message-ID: Hello, I wrote a "display obj" method for viewing instance data: def dobj(obj) : """extended object viewer that displays arg""" print 'Class: ', obj.__class__.__name__, ' id:', id(obj) for k in obj.__dict__.keys() : print ' ', k, ': ', obj.__dict__[k], ' id:', id(obj.__dict__ [k]) and I'm wondering why it doesn't work on ndarrays: >>> m = numpy.zeros(3) >>> dobj(m) complains because: AttributeError: 'numpy.ndarray' object has no attribute '__dict__' I am also fairly new to python, but would have thought all objects had dictionaries, especially when they are comprised of other things, e.g. dir(m) produces a slew of components, e.g. tofile, ..., transpose, ..., view, etc. Thanks for any insight. --b From v-nijs at kellogg.northwestern.edu Thu Jan 4 14:40:08 2007 From: v-nijs at kellogg.northwestern.edu (Vincent Nijs) Date: Thu, 04 Jan 2007 13:40:08 -0600 Subject: [Numpy-discussion] ndarray newbie question In-Reply-To: Message-ID: --b, The only data type in python that has a keys() method is a dictionary. Unless it is a record array (http://www.scipy.org/RecordArrays) there is no information on variable names contained in the object. However, even record arrays don't have a keys() method. Vincent On 1/4/07 1:18 PM, "belinda thom" wrote: > Hello, > > I wrote a "display obj" method for viewing instance data: > > def dobj(obj) : > """extended object viewer that displays arg""" > print 'Class: ', obj.__class__.__name__, ' id:', id(obj) > for k in obj.__dict__.keys() : > print ' ', k, ': ', obj.__dict__[k], ' id:', id(obj.__dict__ > [k]) > > > and I'm wondering why it doesn't work on ndarrays: > >>>> m = numpy.zeros(3) >>>> dobj(m) > > complains because: > > AttributeError: 'numpy.ndarray' object has no attribute '__dict__' > > I am also fairly new to python, but would have thought all objects > had dictionaries, especially when they are comprised of other things, > e.g. > > dir(m) > > produces a slew of components, e.g. > > tofile, ..., transpose, ..., view, etc. > > Thanks for any insight. > > --b > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Thu Jan 4 14:47:35 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 04 Jan 2007 13:47:35 -0600 Subject: [Numpy-discussion] ndarray newbie question In-Reply-To: References: Message-ID: <459D59D7.9040207@gmail.com> belinda thom wrote: > Hello, > > I wrote a "display obj" method for viewing instance data: > > def dobj(obj) : > """extended object viewer that displays arg""" > print 'Class: ', obj.__class__.__name__, ' id:', id(obj) > for k in obj.__dict__.keys() : > print ' ', k, ': ', obj.__dict__[k], ' id:', id(obj.__dict__ > [k]) > > > and I'm wondering why it doesn't work on ndarrays: > > >>> m = numpy.zeros(3) > >>> dobj(m) > > complains because: > > AttributeError: 'numpy.ndarray' object has no attribute '__dict__' > > I am also fairly new to python, but would have thought all objects > had dictionaries, especially when they are comprised of other things, > e.g. > > dir(m) > > produces a slew of components, e.g. > > tofile, ..., transpose, ..., view, etc. Most types defined in C do not have a .__dict__ . In [1631]: s = 1 In [1632]: s.__dict__ --------------------------------------------------------------------------- Traceback (most recent call last) /Users/rkern/svn/mt-data/ in () : 'int' object has no attribute '__dict__' -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pgmdevlist at gmail.com Thu Jan 4 14:55:50 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 4 Jan 2007 14:55:50 -0500 Subject: [Numpy-discussion] ndarray newbie question In-Reply-To: References: Message-ID: <200701041455.51125.pgmdevlist@gmail.com> On Thursday 04 January 2007 14:18, belinda thom wrote: > Hello, > > I wrote a "display obj" method for viewing instance data: ... > and I'm wondering why it doesn't work on ndarrays: An (instance of) ndarray doesn't have a __dict__ attribute, as you've noticed. The class ndarray does have one. Is it because the class ndarray doesn't have an init method ? Dunno. Robert mentions it could be C related. Note that float(3), dict(a=3), list([3]) don't have __dict__ attributes either, when float, dict, and list do have one. You can still access the list of attributes an instance by dir, and a specific one by __getattribute__ for k in dir(obj): print k, obj.__getattr__(k) Or try to access the class of the instance to get the __dict__ for k in obj.__class__.__dict__.keys(): print k From bthom at cs.hmc.edu Thu Jan 4 15:16:28 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Thu, 04 Jan 2007 12:16:28 -0800 Subject: [Numpy-discussion] ndarray newbie question In-Reply-To: References: Message-ID: Vincent, Thanks again. On Jan 4, 2007, at 11:40 AM, Vincent Nijs wrote: > --b, > > The only data type in python that has a keys() method is a dictionary. Doh! > Unless it is a record array (http://www.scipy.org/RecordArrays) > there is no > information on variable names contained in the object. There's always dir()...but then you can't tell what is what (e.g. a whole slew of stuff is reported...vars, methods, ...). > However, even record > arrays don't have a keys() method. > > Vincent Some context: the type of introspection I'm often wishing I could do in a single, easy command usually has to do w/getting to know the software better. I've been frustrated w/Python at times in this regard because, for example, printing out all local variables requires knowing about deeper magic (e.g. locals()). I realize that open source is less likely to have the bells and whistles that corporate software can provide (e.g. w/VC++ its trivial to "watch" all local vars in the debugger), and I'm switching from matlab to pylab / numpy precisely because its open source. At the same time, its hard to imagine others don't have access to the kinds of tricks I'm starting to learn (e.g. dobj). If folks have other tricks of that type, I'm all ears and quite grateful. Regarding numpy ndarrays, if I were to write my own viewer, what internal variables make the most sense to show? Are .ndim, .size, and .shape the only ones I should care about? Danke, --b From pgmdevlist at gmail.com Thu Jan 4 15:33:39 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 4 Jan 2007 15:33:39 -0500 Subject: [Numpy-discussion] ndarray newbie question In-Reply-To: References: Message-ID: <200701041533.39677.pgmdevlist@gmail.com> > > However, even record > > arrays don't have a keys() method. You can access the attributes of a recarray by N.ndarray.__getattribute__(obj,'dtype').fields which is yet another dictproxy. > Some context: the type of introspection I'm often wishing I could do > in a single, easy command usually has to do w/getting to know the > software better. dir? obj.__doc__ ? > Regarding numpy ndarrays, if I were to write my own viewer, what > internal variables make the most sense to show? Are .ndim, .size, > and .shape the only ones I should care about? shape and dtype at minimum, I'd say. ndim is just the length of the shape tuple, size the product of the terms in the shape tuple, so you don't really need them for displaying results. That's basically how you define a ndarray: give it a shape and a dtype, and that's enough to allocate the corresponding memory. Your array might be empty, but you can always fill it later. From oliphant at ee.byu.edu Thu Jan 4 15:53:56 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 04 Jan 2007 13:53:56 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP Message-ID: <459D6964.30701@ee.byu.edu> I'm attaching my latest extended buffer-protocol PEP that is trying to get the array interface into Python. Basically, it is a translation of the numpy header files into something as simple as possible that can still be used to describe a complicated block of memory to another user. My purpose is to get feedback and criticisms from this community before display before the larger Python community. -Travis -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pep_buffer.txt URL: From oliphant at ee.byu.edu Thu Jan 4 16:16:58 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 04 Jan 2007 14:16:58 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP Message-ID: <459D6ECA.9010005@ee.byu.edu> I'm attaching my latest extended buffer-protocol PEP that is trying to get the array interface into Python. Basically, it is a translation of the numpy header files into something as simple as possible that can still be used to describe a complicated block of memory to another user. My purpose is to get feedback and criticisms from this community before display before the larger Python community. -Travis -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pep_buffer.txt URL: From Chris.Barker at noaa.gov Thu Jan 4 16:30:28 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 04 Jan 2007 13:30:28 -0800 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459D6964.30701@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> Message-ID: <459D71F4.7060507@noaa.gov> Travis, First, thanks for doing this -- Python really needs it! > While this approach > works, it requires attribute lookups which can be expensive when > sharing many small arrays. Ah, I do like reducing that overhead -- I know I use arrays a lot for small data sets too, so that overhead can be significant. I'm not well qualified to review the tech details, but to make sure I have this right: > static PyObject* bf_getarrayview (PyObject *obj) > > This function must return a new reference to a PyArrViewObject > which contains the details of the array information exposed by the > object. If failure occurs, then NULL is returned and an exception > set. So If I have some C code that wants to use any array passed in, I can just call: bf_getarrayview (obj) and if it doesn't return NULL, I have a valid array that I can query to see if it fits what I'm expecting. Have I got that right? If so, this would be great. By the way,, how compatible is this with the existing buffer protocol? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Thu Jan 4 16:49:32 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 04 Jan 2007 14:49:32 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459D71F4.7060507@noaa.gov> References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> Message-ID: <459D766C.40505@ee.byu.edu> Christopher Barker wrote: > Travis, > > First, thanks for doing this -- Python really needs it! > > While this approach > >> works, it requires attribute lookups which can be expensive when >> sharing many small arrays. >> > > Ah, I do like reducing that overhead -- I know I use arrays a lot for > small data sets too, so that overhead can be significant. > > I'm not well qualified to review the tech details, but to make sure I > have this right: > > >> static PyObject* bf_getarrayview (PyObject *obj) >> >> This function must return a new reference to a PyArrViewObject >> which contains the details of the array information exposed by the >> object. If failure occurs, then NULL is returned and an exception >> set. >> > > So If I have some C code that wants to use any array passed in, I can > just call: > > bf_getarrayview (obj) > Yes you could call this (but you would call it from the type object like this obj->ob_type->tp_as_buffer->bf_getarrayview(obj) Or more likely (and I should add this to the C-API) you would call. PyArrayView_FromObject(obj) which does this under the covers. > and if it doesn't return NULL, I have a valid array that I can query to > see if it fits what I'm expecting. > > Have I got that right? > > If so, this would be great. > > By the way,, how compatible is this with the existing buffer protocol? > It's basically orthogonal. In other-words, if you defined the array view protocol you would not need the buffer protocol at all. But you could easily define both. -Travis From oliphant.travis at ieee.org Thu Jan 4 17:57:34 2007 From: oliphant.travis at ieee.org (Travis E. Oliphant) Date: Thu, 04 Jan 2007 15:57:34 -0700 Subject: [Numpy-discussion] Update to Array-Interface PEP Message-ID: <459D865E.1080506@ieee.org> Here is a new update. I added the PY_POINTER, and PY_ARRAYOF specifications to simplify the extended specification and allow memory that is a pointer to another memory. This should be the last update today. -Travis -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pep_buffer.txt URL: From cjw at sympatico.ca Thu Jan 4 18:04:21 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu, 04 Jan 2007 18:04:21 -0500 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459D6964.30701@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> Message-ID: Travis Oliphant wrote: > > I'm attaching my latest extended buffer-protocol PEP that is trying to > get the array interface into Python. Basically, it is a translation of > the numpy header files into something as simple as possible that can > still be used to describe a complicated block of memory to another user. > > My purpose is to get feedback and criticisms from this community before > display before the larger Python community. > > -Travis > > It would help me to understand the proposal if it could be explained in terms of the methods of the existing buffer class/type: ['__add__', '__class__', '__cmp__', '__delattr__', '__delitem__', '__delslice__', '__doc__', '__getattribute__', '__getitem__', '__getslice__', '__hash__', '__init__', '__len__', '__mul__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmul__', '__setattr__', '__setitem__', '__setslice__', '__str__'] Numpy extends numarray's type/dtype object. This proposal appears to revert to the old letter codes. I have had very limited experience with C. Colin W. > > ------------------------------------------------------------------------ > > PEP: > Title: Extending the buffer protocol to include the array interface > Version: $Revision: $ > Last-Modified: $Date: $ > Author: Travis Oliphant > Status: Draft > Type: Standards Track > Created: 28-Aug-2006 > Python-Version: 2.6 > > Abstract > > This PEP proposes extending the tp_as_buffer structure to include > function pointers that incorporate information about the intended > shape and data-format of the provided buffer. In essence this will > place an array interface directly into Python. > > Rationale > > Several extensions to Python utilize the buffer protocol to share > the location of a data-buffer that is really an N-dimensional > array. However, there is no standard way to exchange the > additional N-dimensional array information so that the data-buffer > is interpreted correctly. The NumPy project introduced an array > interface (http://numpy.scipy.org/array_interface.shtml) through a > set of attributes on the object itself. While this approach > works, it requires attribute lookups which can be expensive when > sharing many small arrays. > > One of the key reasons that users often request to place something > like NumPy into the standard library is so that it can be used as > standard for other packages that deal with arrays. This PEP > provides a mechanism for extending the buffer protocol (which > already allows data sharing) to add the additional information > needed to understand the data. This should be of benefit to all > third-party modules that want to share memory through the buffer > protocol such as GUI toolkits, PIL, PyGame, CVXOPT, PyVoxel, > PyMedia, audio libraries, video libraries etc. > > > Proposal > > Add bf_getarrview and bf_relarrview function pointers to the > buffer protocol to allow objects to share a view on a memory > pointer including information about accessing it as an > N-dimensional array. Add the TP_HAS_ARRAY_BUFFER flag to types > that define this extended buffer protocol. > > Also a few additionsl C-API calls should perhaps be added to Python > to facilitate creating new PyArrViewObjects. > > Specification: > > static PyObject* bf_getarrayview (PyObject *obj) > > This function must return a new reference to a PyArrViewObject > which contains the details of the array information exposed by the > object. If failure occurs, then NULL is returned and an exception > set. > > static int bf_relarrayview(PyObject *obj) > > If not NULL then this will be called when the object returned by > bf_getarrview is destroyed so that the underlying object can be > aware when acquired "views" are released. > > The object that defines bf_getarrview should not re-allocate memory > (re-size itself) while views are extant. A 0 is returned on success > and a -1 and an error condition set on failure. > > The PyArrayViewObject has the structure > > typedef struct { > PyObject_HEAD > void *data; /* pointer to the beginning of data */ > int nd; /* the number of dimensions */ > Py_ssize_t *shape; /* c-array of size nd giving shape */ > Py_ssize_t *strides; /* SEE BELOW */ > PyObject *base; /* the object this is a "view" of */ > PyObject *format; /* SEE BELOW */ > int flags; /* SEE BELOW */ > } PyArrayViewObject; > > > strides -- a c-array of size nd providing the striding information > which is the number of bytes to skip to get to the next element > in that dimension. > > format -- a Python data-format object (PyDataFormatObject) which > contains information about how each item in the array > should be interpreted. > > flags -- an integer of flags. PYARR_WRITEABLE is the only flag > that must be set appropriately by types. > Other flags: PYARR_ALIGNED, PYARR_C_CONTIGUOUS, > PYARR_F_CONTIGUOUS, and PYARR_NOTSWAPPED can all be determined > from the rest of the PyArrayViewObject using the UpdateFlags C-API. > > The PyDataFormatObject has the structure > > typedef struct { > PyObject_HEAD > PySimpleformat primitive; /* basic primitive type */ > int flags; /* byte-order, isaligned */ > int itemsize; /* SEE BELOW */ > int alignment; /* SEE BELOW */ > PyObject *extended; /* SEE BELOW */ > } PyDataFormatObject; > > enum Pysimpleformat {PY_BIT='1', PY_BOOL='?', PY_BYTE='b', PY_SHORT='h', PY_INT='i', > PY_LONG='l', PY_LONGLONG='q', PY_UBYTE='B', PY_USHORT='H', PY_UINT='I', > PY_ULONG='L', PY_ULONGLONG='Q', PY_FLOAT='f', PY_DOUBLE='d', PY_LONGDOUBLE='g', > PY_CFLOAT='F', PY_CDOUBLE='D', PY_CLONGDOUBLE='G', PY_OBJECT='O', > PY_CHAR='c', PY_UCS2='u', PY_UCS4='w', PY_FUNCPTR='X', PY_VOIDPTR='V'}; > > Each of these simple formats has a special character code which can be used to > identify this primitive in a nested python list. > > > flags -- flags for the data-format object. Specified masks are > PY_NATIVEORDER > PY_BIGENDIAN > PY_LITTLEENDIAN > PY_IGNORE > > itemsize -- the total size represented by this data-format in bytes unless the > primitive is PY_BIT in which case it is the size in bits. > For data-formats that are simple 1-d arrays of the underlying primitive, > this total size can represent more than one primitive (with extended > still NULL). > > alignment -- For the primitive types this is offsetof(struct {char c; type v;},v) > > extended -- NULL if this is a primitive data-type or no additional information is > available. > > If primitive is PY_FUNCPTR, then this can be a tuple with >=1 element: > (args, {dim0, dim1, dim2, ...}). > > args -- A list (of at least length 2) of data-format objects > specifying the input argument formats with the last > argument specifying the output argument data-format > (use None for void inputs and/or outputs). > > For other primitives, this can be a tuple with >=2 elements: > (names, fields, {dim0, dim1, dim2, ...}) > Use None for both names and fields if they should be ignored. > > names -- An ordered list of string or unicode objects giving the names > of the fields for a structure data-format. > fields -- a Python dictionary with ordered-keys given by the list > in names. Each entry in the dictionary is > a 3-tuple containing (data-format-object, offset, > meta-information) where meta-information is Py_None if there > is no meta-information. Offset is given in bytes from the > start of the record or in bits if PY_BIT is the primitive. > > Any additional entries in the extended tuple (dim0, > dim1, etc.) are interpreted as integers which specify > that this data-format is an array of the given shape > of the fundamental data-format specified by the > remainder of the DataFormat Object. The dimensions > are specified so that the last-index is always assumed > to vary the fastest (C-order). > > > The constructor of a PyArrViewObject allocates the memory for shape and strides > and the destructor frees that memory. > > The constructor of a PyDataFormatObject allocates the objects it needs for fields, > names, and shape. > > C-API > > void PyArrayView_UpdateFlags(PyObject *view, int flags) > /* update the flags on the array view object provided */ > > PyDataFormatObject *Py_NewSimpleFormat(Pysimpleformat primitive) > /* return a new primitive data-format object */ > > PyDataFormatObject *Py_DataFormatFromCType(PyObject *ctype) > /* return a new data-format object from a ctype */ > > int Py_GetPrimitiveSize(Pysimpleformat primitive) > /* return the size (in bytes) of the provided primitive */ > > PyDataFormatObject *Py_AlignDataFormat(PyObject *format) > /* take a data-format object and construct an aligned data-format > object where all fields are aligned on appropriate boundaries > for the compiler */ > > > Discussion > > The information provided in the array view object is patterned > after the way a multi-dimensional array is defined in NumPy -- including > the data-format object which allows a variety of descriptions of memory > depending on the need. > > Reference Implementation > > Supplied when the PEP is accepted. > > Copyright > > This document is placed in the public domain. > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From Chris.Barker at noaa.gov Thu Jan 4 18:06:46 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 04 Jan 2007 15:06:46 -0800 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459D766C.40505@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> <459D766C.40505@ee.byu.edu> Message-ID: <459D8886.2000101@noaa.gov> Travis Oliphant wrote: >> bf_getarrayview (obj) >> > Yes you could call this (but you would call it from the type object like > this > > obj->ob_type->tp_as_buffer->bf_getarrayview(obj) > > Or more likely (and I should add this to the C-API) you would call. > > PyArrayView_FromObject(obj) yes, that's what I'm looking for -- please do add that to the C-API >> By the way,, how compatible is this with the existing buffer protocol? >> > It's basically orthogonal. In other-words, if you defined the array > view protocol you would not need the buffer protocol at all. But you > could easily define both. OK, so if one of these were passed into something expecting the buffer protocol, then it wouldn't work, but you could make an object conform to both protocols at once -- like numpy does now, I suppose -- very nice. Another question -- is this new approach in response to feedback from Guido and or other Python devs? This sure seems like a good way to go -- though it seems from the last discussion I followed at python-dev, most of the devs just didn't get how useful this would be! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From haase at msg.ucsf.edu Thu Jan 4 18:09:42 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 4 Jan 2007 15:09:42 -0800 Subject: [Numpy-discussion] How do I test if an array contains NaN ? Message-ID: Hi! Simple question: How do I test if an array contains NaN ? Or others like inf ...? Thanks, Sebastian Haase From oliphant at ee.byu.edu Thu Jan 4 18:15:30 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 04 Jan 2007 16:15:30 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> Message-ID: <459D8A92.8070908@ee.byu.edu> Colin J. Williams wrote: > Travis Oliphant wrote: > >> I'm attaching my latest extended buffer-protocol PEP that is trying to >> get the array interface into Python. Basically, it is a translation of >> the numpy header files into something as simple as possible that can >> still be used to describe a complicated block of memory to another user. >> >> My purpose is to get feedback and criticisms from this community before >> display before the larger Python community. >> >> -Travis >> >> >> > It would help me to understand the proposal if it could be explained in > terms of the methods of the existing buffer class/type: > ['__add__', '__class__', '__cmp__', '__delattr__', '__delitem__', > '__delslice__', '__doc__', '__getattribute__', '__getitem__', > '__getslice__', '__hash__', '__init__', '__len__', '__mul__', '__new__', > '__reduce__', '__reduce_ex__', '__repr__', '__rmul__', '__setattr__', > '__setitem__', '__setslice__', '__str__'] > > Numpy extends numarray's type/dtype object. This proposal appears to > revert to the old letter codes. > It extends what is done in the array and struct modules of Python. The old letter codes are useful on the c-level. They are 'hidden' behind an enumeration however, and so should not be a big deal. But, the letter codes are still useful in other contexts. > I have had very limited experience with C. > Then this proposal will not be meaningful for you. This is a proposal to extend something on the C-level. There is nothing on the Python level suggested by this proposal. -Travis From kwgoodman at gmail.com Thu Jan 4 18:14:43 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 4 Jan 2007 15:14:43 -0800 Subject: [Numpy-discussion] How do I test if an array contains NaN ? In-Reply-To: References: Message-ID: On 1/4/07, Sebastian Haase wrote: > How do I test if an array contains NaN ? > Or others like inf ...? isnan() ~isfinite() any() From oliphant at ee.byu.edu Thu Jan 4 18:19:14 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 04 Jan 2007 16:19:14 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459D8886.2000101@noaa.gov> References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> <459D766C.40505@ee.byu.edu> <459D8886.2000101@noaa.gov> Message-ID: <459D8B72.2040309@ee.byu.edu> >> PyArrayView_FromObject(obj) >> > > yes, that's what I'm looking for -- please do add that to the C-API > > >>> By the way,, how compatible is this with the existing buffer protocol? >>> >>> >> It's basically orthogonal. In other-words, if you defined the array >> view protocol you would not need the buffer protocol at all. But you >> could easily define both. >> > > OK, so if one of these were passed into something expecting the buffer > protocol, then it wouldn't work, but you could make an object conform to > both protocols at once -- like numpy does now, I suppose -- very nice. > > Another question -- is this new approach in response to feedback from > Guido and or other Python devs? This sure seems like a good way to go -- > though it seems from the last discussion I followed at python-dev, most > of the devs just didn't get how useful this would be! > The new approach is a response to the devs and to Sasha who had some relevant comments. Yes, I agree that many devs don't "get" how useful it would be because they have not written scientific or graphics-intensive applications. However, Guido is the one who encouraged me at SciPy 2006 to push this and so I think he is generally favorable to the idea. The Python devs will definitely push back. The strongest opposition seems to be from people that don't 'get' it and so don't want "dead interfaces" in Python. They would need to be convinced of how often such an interface would actually get used. I've tried to do that in the rationale, but many people actually posting to python-dev to support the basic idea (you don't have to support the specific implementation --- most are going to be uncomfortable knowing enough to make a stand). However, there is a great need for people to stand up and say: "We need something like this in Python..." -Travis From seb.haase at gmx.net Thu Jan 4 18:18:18 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Thu, 4 Jan 2007 15:18:18 -0800 Subject: [Numpy-discussion] How do I test if an array contains NaN ? In-Reply-To: References: Message-ID: On 1/4/07, Keith Goodman wrote: > On 1/4/07, Sebastian Haase wrote: > > How do I test if an array contains NaN ? > > Or others like inf ...? > > isnan() > ~isfinite() > any() Aah ! thanks, you mean I have to create an intermediate array that tells me for every element if is's a nan and then check any( ... ) on this !? That's OK for what I need -- seems excessive for large arrays though .... Thanks again, Sebastian From robert.kern at gmail.com Thu Jan 4 18:19:32 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 04 Jan 2007 17:19:32 -0600 Subject: [Numpy-discussion] How do I test if an array contains NaN ? In-Reply-To: References: Message-ID: <459D8B84.3090405@gmail.com> Sebastian Haase wrote: > Hi! > > Simple question: > How do I test if an array contains NaN ? > Or others like inf ...? In [1633]: isinf? Type: ufunc Base Class: Namespace: Interactive Docstring: y = isinf(x) returns True where x is +inf or -inf In [1634]: isnan? Type: ufunc Base Class: Namespace: Interactive Docstring: y = isnan(x) returns True where x is Not-A-Number In [1635]: isfinite? Type: ufunc Base Class: Namespace: Interactive Docstring: y = isfinite(x) returns True where x is finite -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kwgoodman at gmail.com Thu Jan 4 18:24:27 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 4 Jan 2007 15:24:27 -0800 Subject: [Numpy-discussion] How do I test if an array contains NaN ? In-Reply-To: References: Message-ID: On 1/4/07, Sebastian Haase wrote: > On 1/4/07, Keith Goodman wrote: > > On 1/4/07, Sebastian Haase wrote: > > > How do I test if an array contains NaN ? > > > Or others like inf ...? > > > > isnan() > > ~isfinite() > > any() > > Aah ! thanks, > you mean I have to create an intermediate array that tells me for > every element if is's a nan > and then check any( ... ) on this !? > > That's OK for what I need -- seems excessive for large arrays though .... Would this save memory? isnan(x.sum()) From fperez.net at gmail.com Thu Jan 4 18:25:09 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 4 Jan 2007 16:25:09 -0700 Subject: [Numpy-discussion] How do I test if an array contains NaN ? In-Reply-To: <459D8B84.3090405@gmail.com> References: <459D8B84.3090405@gmail.com> Message-ID: On 1/4/07, Robert Kern wrote: > Sebastian Haase wrote: > > Hi! > > > > Simple question: > > How do I test if an array contains NaN ? > > Or others like inf ...? > > In [1633]: isinf? > Type: ufunc > Base Class: > Namespace: Interactive > Docstring: > y = isinf(x) returns True where x is +inf or -inf [...] It's probably worth mentioning that IPython has (thanks to a user contributed implementation) search capabilities besides tab completion (which requires you to at least know the start of the string you want): In [3]: N.*nan*? N.isnan N.nan N.nan_to_num N.nanargmax N.nanargmin N.nanmax N.nanmin N.nansum In [4]: N.*inf*? N.finfo N.inf N.info N.infty N.isinf N.isneginf N.isposinf This can be very useful when querying a package for something you only vaguely know how it might be called. This just prints the list of names to stdout, you then need to actually type the one you want. Cheers, f From Chris.Barker at noaa.gov Thu Jan 4 18:51:40 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 04 Jan 2007 15:51:40 -0800 Subject: [Numpy-discussion] How do I test if an array contains NaN ? In-Reply-To: References: <459D8B84.3090405@gmail.com> Message-ID: <459D930C.8020602@noaa.gov> Fernando Perez wrote: > It's probably worth mentioning that IPython has (thanks to a user > contributed implementation) search capabilities besides tab completion very handy. And if you're an idiot like me that for some unknown reason still does not use ipython, you can always do: for i in dir(N): if "nan" in i: print i >>> import numpy as N >>> for i in dir(N): ... if "nan" in i: print i ... isnan nan nan_to_num nanargmax nanargmin nanmax nanmin nansum -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From kwgoodman at gmail.com Thu Jan 4 18:55:25 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 4 Jan 2007 15:55:25 -0800 Subject: [Numpy-discussion] How do I test if an array contains NaN ? In-Reply-To: <459D930C.8020602@noaa.gov> References: <459D8B84.3090405@gmail.com> <459D930C.8020602@noaa.gov> Message-ID: On 1/4/07, Christopher Barker wrote: > And if you're an idiot like me that for some unknown reason still does > not use ipython, you can always do: > > for i in dir(N): > if "nan" in i: print i I'm always impressed by those who take the stairs instead of the elevator. From Chris.Barker at noaa.gov Thu Jan 4 19:02:24 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 04 Jan 2007 16:02:24 -0800 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459D8B72.2040309@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> <459D766C.40505@ee.byu.edu> <459D8886.2000101@noaa.gov> <459D8B72.2040309@ee.byu.edu> Message-ID: <459D9590.80202@noaa.gov> Travis Oliphant wrote: > Guido is the one who > encouraged me at SciPy 2006 to push this and so I think he is generally > favorable to the idea. good sign. > The Python devs will definitely push back. The > strongest opposition seems to be from people that don't 'get' it and so > don't want "dead interfaces" in Python. well, none of us want dead interfaces in Python. > They would need to be convinced of how often such an interface would > actually get used. I've tried to do that in the rationale, but many > people actually posting to python-dev to support the basic idea > However, there is a > great need for people to stand up and say: "We need something like this > in Python..." let us know when there is a relevant thread to chime in on. However, what we really need is not people like me saying "I need this", but rather people that develop significant existing extension packages saying they'll actually use this in their package. People like: wxPython -- Robin Dunn PIL -- Fredrik Lundh PyOpenGL -- Who? PyObjC -- would it be useful there? (Ronald Oussoren) MatplotLib (but maybe it's already married to numpy...) PyGtk ? Who else? I know Robin Dunn is interested in using it in wxPython -- but probably only if someone contributes the code. I hope to do that some day, but I'm only barely qualified to do so. Fredrik accepted your submission of code to use the array interface in PIL, but he seemed skeptical of the idea. Perhaps lobbying (or even just surveying) some of these folks would be useful. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From haase at msg.ucsf.edu Thu Jan 4 19:15:09 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 4 Jan 2007 16:15:09 -0800 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray Message-ID: Hi! I'm continuing my code conversion to numpy. Trying to track down a segmentation fault I found 1) ndimage crashed because I was feeding (nan,nan) as shift value ..... 2) but where did I get the nan from ? I just found that there is code like this: (*paraphrased*) size = N.array( [255] ) size.resize( 2 ) ... trying this interactively I get an exception: >>> a = N.array([5]) >>> a [5] >>> a.resize(2) Traceback (most recent call last): File "", line 1, in ? ValueError: cannot resize an array that has been referenced or is referencing another array in this way. Use the resize function >>> N.__version__ '1.0.2.dev3487' in any case: inside the script it somehow generated a nan --- is there a bug in numpy !? I remember that there was some discussion about resize !? What should I add to the Scipy Wiki numarray page about this ? ( http://www.scipy.org/Converting_from_numarray ) Thanks, Sebastian Haase From seb.haase at gmx.net Thu Jan 4 19:29:32 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Thu, 4 Jan 2007 16:29:32 -0800 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray In-Reply-To: References: Message-ID: On 1/4/07, Sebastian Haase wrote: > >>> N.__version__ > '1.0.2.dev3487' > > in any case: inside the script it somehow generated a nan --- is > there a bug in numpy !? No bug here ! see below ! > I remember that there was some discussion about resize !? > What should I add to the Scipy Wiki numarray page about this ? > ( http://www.scipy.org/Converting_from_numarray ) > OK - the reference problem in my interactive shell came from the implicit '_' variable that always references the last result. But maybe even more worry some for the converting from numarray is this: >>> a = N.array([5]) >>> 999 # to kill '_' - reference 999 >>> a.resize(2) >>> a [5 0] in numarray you would get >>> a = na.array([5]) >>> a.resize(2) [5 5] >>> a [5 5] !! why is numpy filling with 0s and numarray repeats (cycles I think is more what it does !) the last element(s) ?? How did numeric do this ? - Sebastian From oliphant at ee.byu.edu Thu Jan 4 19:36:44 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 04 Jan 2007 17:36:44 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459D9590.80202@noaa.gov> References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> <459D766C.40505@ee.byu.edu> <459D8886.2000101@noaa.gov> <459D8B72.2040309@ee.byu.edu> <459D9590.80202@noaa.gov> Message-ID: <459D9D9C.30501@ee.byu.edu> Christopher Barker wrote: > eople like: > > wxPython -- Robin Dunn > PIL -- Fredrik Lundh > PyOpenGL -- Who? > PyObjC -- would it be useful there? (Ronald Oussoren) > MatplotLib (but maybe it's already married to numpy...) > PyGtk ? > It's a good start, but their is also PyMedia, PyVoxel, any video-library interface writers, any audo-library interface writers. Anybody who wants to wrap/write code that does some kind of manipulation on a chunk of data of a specific data-format. There are so many people who would use it that I don't feel qualified to speak for them all. -Travis From seb.haase at gmx.net Thu Jan 4 19:50:36 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Thu, 4 Jan 2007 16:50:36 -0800 Subject: [Numpy-discussion] compress numpy vs .numarray Message-ID: Hi! when calling compress I get this error message after moving to numpy: ValueError: 'condition must be 1-d array' Is the reason for this the change of the default axis from axis=0 to axis=None What does axis=None mean in this case !? Thanks, -Sebastian From seb.haase at gmx.net Thu Jan 4 19:55:07 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Thu, 4 Jan 2007 16:55:07 -0800 Subject: [Numpy-discussion] compress numpy vs .numarray In-Reply-To: References: Message-ID: On 1/4/07, Sebastian Haase wrote: > Hi! > when calling compress > I get this error message after moving to numpy: > > ValueError: 'condition must be 1-d array' > > Is the reason for this the change of the default axis from > axis=0 > to > axis=None > > What does axis=None mean in this case !? > Is N.extract() the equivalent of numarray.compress() ? -Sebastian From robert.kern at gmail.com Thu Jan 4 20:06:52 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 04 Jan 2007 19:06:52 -0600 Subject: [Numpy-discussion] compress numpy vs .numarray In-Reply-To: References: Message-ID: <459DA4AC.9050109@gmail.com> Sebastian Haase wrote: > Hi! > when calling compress > I get this error message after moving to numpy: > > ValueError: 'condition must be 1-d array' > > Is the reason for this the change of the default axis from > axis=0 > to > axis=None Not really, it's just that N-d arrays don't make much sense as condition masks. The only reasonable interpretation is that one takes the flattened form of both the condition mask and the target array; however, this is an implicit interpretation of what the user wants. As explicit is usually better than implicit, just flatten the condition mask. > What does axis=None mean in this case !? That the condition mask applies to the flattened array. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Jan 4 20:08:55 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 04 Jan 2007 19:08:55 -0600 Subject: [Numpy-discussion] compress numpy vs .numarray In-Reply-To: References: Message-ID: <459DA527.30800@gmail.com> Sebastian Haase wrote: > Is N.extract() the equivalent of numarray.compress() ? Only for the specific flattening behavior that you are talking about. However, numpy.extract() does not have the axis= argument that numarray.compress() does. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bthom at cs.hmc.edu Fri Jan 5 04:22:04 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Fri, 05 Jan 2007 01:22:04 -0800 Subject: [Numpy-discussion] ndarray newbie question Message-ID: <720258A3-43DE-46F0-AE7A-3E00B5E4AFBE@cs.hmc.edu> Pierre, Thanks for the reply. > Some context: the type of introspection I'm often wishing I could do > in a single, easy command usually has to do w/getting to know the > software better. dir? obj.__doc__ ? When I try the latter recommendation above w/numpy arrays, I get: In [1]: import numpy as N In [2]: a = N.array([1,2,3]) In [3]: a.__doc__? Type: NoneType Base Class: String Form: None Namespace: Interactive Docstring: Did you have something else in mind? Thx, --b From robert.kern at gmail.com Fri Jan 5 04:29:51 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 05 Jan 2007 03:29:51 -0600 Subject: [Numpy-discussion] ndarray newbie question In-Reply-To: <720258A3-43DE-46F0-AE7A-3E00B5E4AFBE@cs.hmc.edu> References: <720258A3-43DE-46F0-AE7A-3E00B5E4AFBE@cs.hmc.edu> Message-ID: <459E1A8F.5050906@gmail.com> belinda thom wrote: > Pierre, > > Thanks for the reply. > > > Some context: the type of introspection I'm often wishing I could do > > in a single, easy command usually has to do w/getting to know the > > software better. > > dir? > obj.__doc__ ? > > When I try the latter recommendation above w/numpy arrays, I get: > > In [1]: import numpy as N > > In [2]: a = N.array([1,2,3]) > > In [3]: a.__doc__? > Type: NoneType > Base Class: > String Form: None > Namespace: Interactive > Docstring: > > > > Did you have something else in mind? The ?s were not literal. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Fri Jan 5 04:56:21 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 5 Jan 2007 11:56:21 +0200 Subject: [Numpy-discussion] returning recarray records as plain arrays In-Reply-To: <200701040140.54440.pgmdevlist@gmail.com> References: <200701031605.28262.pgmdevlist@gmail.com> <459C8947.9060708@cs.nyu.edu> <200701040140.54440.pgmdevlist@gmail.com> Message-ID: <20070105095621.GA6020@mentat.za.net> On Thu, Jan 04, 2007 at 01:40:54AM -0500, Pierre GM wrote: > On Wednesday 03 January 2007 23:57, Matthew Koichi Grimes wrote: > > Pierre GM wrote: > > > On Wednesday 03 January 2007 15:39, Matthew Koichi Grimes wrote: > > >> As per Stefan's help, I've made a subclass of recarray called nnvalue. > > >> It just fixes the dtype to [('x', 'f8'), ('dx', 'f8'), ('delta', 'f8)], > > >> and adds a few member functions. I basically treat nnvalue as a struct > > >> with three equal-shaped array fields: x, dx, and delta. > > Matthew, > It seems that declaring your default dtype as a class argument and naming it > dtype is the reason why things are messed up: you're basically overwriting > the original ndarray.dtype Oops, unintended side-effect there! Cheers St?fan From pgmdevlist at gmail.com Fri Jan 5 04:56:56 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 5 Jan 2007 04:56:56 -0500 Subject: [Numpy-discussion] ndarray newbie question In-Reply-To: <720258A3-43DE-46F0-AE7A-3E00B5E4AFBE@cs.hmc.edu> References: <720258A3-43DE-46F0-AE7A-3E00B5E4AFBE@cs.hmc.edu> Message-ID: <200701050456.56344.pgmdevlist@gmail.com> belinda thom wrote: > ?> Some context: the type of introspection I'm often wishing I could do > ?> in a single, easy command usually has to do w/getting to know the > ?> software better. For generic introspection in Python, dir and .__doc__ are very useful. The ? trick works in ipython and prints the __doc__. > When I try the latter recommendation above w/numpy arrays, I get: ... > Type: NoneType > Base Class: > String Form: None > Namespace: Interactive > Docstring: > I agree that this one is not really informative. A >>> a.__class__.__doc__? provides a docstring interestingly truncated under ipython, but nothing useful either. However, you may want to consider the combination: >>> for k in dir(a): >>> try: >>> print "\n",k, getattr(a,k).__doc__ >>> except AttributeError: >>> pass But I'm a bit confused: what are you really trying to achieve by querying the methods of a numpy array ? The wiki page http://www.scipy.org/Numpy_Example_List_With_Doc is quite educational, and provides many examples of usage you won't find in the docstrings. From fperez.net at gmail.com Fri Jan 5 05:02:34 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 5 Jan 2007 03:02:34 -0700 Subject: [Numpy-discussion] ndarray newbie question In-Reply-To: <200701050456.56344.pgmdevlist@gmail.com> References: <720258A3-43DE-46F0-AE7A-3E00B5E4AFBE@cs.hmc.edu> <200701050456.56344.pgmdevlist@gmail.com> Message-ID: On 1/5/07, Pierre GM wrote: > belinda thom wrote: > > > Some context: the type of introspection I'm often wishing I could do > > > in a single, easy command usually has to do w/getting to know the > > > software better. > > For generic introspection in Python, dir and .__doc__ are very useful. > The ? trick works in ipython and prints the __doc__. > > > When I try the latter recommendation above w/numpy arrays, I get: > ... > > Type: NoneType > > Base Class: > > String Form: None > > Namespace: Interactive > > Docstring: > > > > I agree that this one is not really informative. A > >>> a.__class__.__doc__? > provides a docstring interestingly truncated under ipython, but nothing > useful either. You probably meant In [4]: a.__class__? Type: type Base Class: Namespace: Interactive File: /home/fperez/tmp/local/lib/python2.4/site-packages/numpy/__init__.py Docstring: An array object represents a multidimensional, homogeneous array of fixed-size items. An associated data-type-descriptor object details the data-type in an array (including byteorder and any fields). etc... Best, f From robert.kern at gmail.com Fri Jan 5 05:03:40 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 05 Jan 2007 04:03:40 -0600 Subject: [Numpy-discussion] ndarray newbie question In-Reply-To: <200701050456.56344.pgmdevlist@gmail.com> References: <720258A3-43DE-46F0-AE7A-3E00B5E4AFBE@cs.hmc.edu> <200701050456.56344.pgmdevlist@gmail.com> Message-ID: <459E227C.1000806@gmail.com> Pierre GM wrote: > I agree that this one is not really informative. A >>>> a.__class__.__doc__? > provides a docstring interestingly truncated under ipython, but nothing > useful either. "a.__class__?" is the command that you are after, not "a.__class__.__doc__?". a.__class__.__doc__ is a string, so interrogating it with ? will tell you information about strings rather than a.__class__ . -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From svetosch at gmx.net Fri Jan 5 05:14:37 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 05 Jan 2007 11:14:37 +0100 Subject: [Numpy-discussion] 1.0.1 release notes missing? Message-ID: <459E250D.70709@gmx.net> I found the release notes on the way to 1.0 very useful. However, they stop at 1.0, and I also didn't find any in the 1.0.1 source distribution. Do they exist somewhere? Or are there really only bugfixes in 1.0.1? That information would be useful, too. Thank you, Sven From stefan at sun.ac.za Fri Jan 5 06:22:58 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 5 Jan 2007 13:22:58 +0200 Subject: [Numpy-discussion] subclassing float64 (and friends) In-Reply-To: <459C3868.40406@enthought.com> References: <459B8576.50005@enthought.com> <459C25E7.8040504@ee.byu.edu> <459C3868.40406@enthought.com> Message-ID: <20070105112258.GD6020@mentat.za.net> On Wed, Jan 03, 2007 at 05:12:40PM -0600, eric jones wrote: > Thanks for the update. For now, I'll try doing what I need to by > sub-classing float. But, I'm gonna miss __array_finalize__ :-). Looks like r3493 is intended to fix this. The 'view' method works now, but I still see In [2]: import numpy as N In [3]: x = N.float64(12) In [4]: class Floaty(N.float64): ...: def __new__(subclass,data): ...: return data.view(subclass) In [6]: Floaty(x) Out[6]: 12.0 In [7]: type(Floaty(x)) Out[7]: versus In [8]: class Ary(N.ndarray): ...: def __new__(subclass,data): ...: return data.view(subclass) In [9]: Ary(N.array([1])) Out[9]: Ary([1]) In [10]: type(Ary(N.array([1]))) Out[10]: Cheers St?fan From faltet at carabos.com Fri Jan 5 06:58:02 2007 From: faltet at carabos.com (Francesc Altet) Date: Fri, 5 Jan 2007 12:58:02 +0100 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459D9D9C.30501@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> <459D9590.80202@noaa.gov> <459D9D9C.30501@ee.byu.edu> Message-ID: <200701051258.03232.faltet@carabos.com> A Divendres 05 Gener 2007 01:36, Travis Oliphant escrigu?: > Christopher Barker wrote: > > eople like: > > > > wxPython -- Robin Dunn > > PIL -- Fredrik Lundh > > PyOpenGL -- Who? > > PyObjC -- would it be useful there? (Ronald Oussoren) > > MatplotLib (but maybe it's already married to numpy...) > > PyGtk ? > > It's a good start, but their is also > > PyMedia, PyVoxel, any video-library interface writers, any audo-library > interface writers. > > Anybody who wants to wrap/write code that does some kind of manipulation > on a chunk of data of a specific data-format. Yeah. I think this is the case for PyTables. However, PyTables case should be similar to matplotlib: it needs so many features of NumPy that it is just conceivable it can live with just an implementation of the array interface. In any case, I think that if the PEP has success, it would suppose an extraordinary leap towards efficient data interchange in applications that doesn't need (or are reluctant to include) NumPy for their normal operation. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From ndbecker2 at gmail.com Fri Jan 5 07:20:29 2007 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 05 Jan 2007 07:20:29 -0500 Subject: [Numpy-discussion] Latest Array-Interface PEP References: <459D6964.30701@ee.byu.edu> Message-ID: Travis Oliphant wrote: > > I'm attaching my latest extended buffer-protocol PEP that is trying to > get the array interface into Python. Basically, it is a translation of > the numpy header files into something as simple as possible that can > still be used to describe a complicated block of memory to another user. > > My purpose is to get feedback and criticisms from this community before > display before the larger Python community. > > -Travis I'm wondering if having the buffer object specify the view is the right choice. I think the best choice is to separate the design into: buffer: provides an interface to memory array: provides a view of memory as an array of whatever dimensions 1. buffer may or may not map to contiguous memory. 2. multiple views of the same memory can be shared. These different views could represent different slicings. From oliphant at ee.byu.edu Fri Jan 5 09:23:35 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 05 Jan 2007 07:23:35 -0700 Subject: [Numpy-discussion] subclassing float64 (and friends) In-Reply-To: <20070105112258.GD6020@mentat.za.net> References: <459B8576.50005@enthought.com> <459C25E7.8040504@ee.byu.edu> <459C3868.40406@enthought.com> <20070105112258.GD6020@mentat.za.net> Message-ID: <459E5F67.8060803@ee.byu.edu> Stefan van der Walt wrote: > On Wed, Jan 03, 2007 at 05:12:40PM -0600, eric jones wrote: > >> Thanks for the update. For now, I'll try doing what I need to by >> sub-classing float. But, I'm gonna miss __array_finalize__ :-). >> > > Looks like r3493 is intended to fix this. The 'view' method works > now, but I still see > > In [2]: import numpy as N > > In [3]: x = N.float64(12) > > In [4]: class Floaty(N.float64): > ...: def __new__(subclass,data): > ...: return data.view(subclass) > > In [6]: Floaty(x) > Out[6]: 12.0 > > In [7]: type(Floaty(x)) > Out[7]: > > I did not change the view method. This will still always return base-class array scalars. Could look into it. There is no __array_finalize__, however, for array scalars. -Travis From oliphant at ee.byu.edu Fri Jan 5 09:25:59 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 05 Jan 2007 07:25:59 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> Message-ID: <459E5FF7.7080300@ee.byu.edu> Neal Becker wrote: > Travis Oliphant wrote: > > >> I'm attaching my latest extended buffer-protocol PEP that is trying to >> get the array interface into Python. Basically, it is a translation of >> the numpy header files into something as simple as possible that can >> still be used to describe a complicated block of memory to another user. >> >> My purpose is to get feedback and criticisms from this community before >> display before the larger Python community. >> >> -Travis >> > > I'm wondering if having the buffer object specify the view is the right > choice. I think the best choice is to separate the design into: > > buffer: provides an interface to memory > array: provides a view of memory as an array of whatever dimensions > > 1. buffer may or may not map to contiguous memory. > 2. multiple views of the same memory can be shared. These different views > could represent different slicings. > I don't understand your concerns, could you please help clarify? I'm not doing anything with the buffer object at all. I'm only using the buffer "protocol" (i.e. extending the set of function pointers pointed to by tp_as_buffer in the type-object). -Travis From matthew.brett at gmail.com Fri Jan 5 09:33:44 2007 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 5 Jan 2007 14:33:44 +0000 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459D9D9C.30501@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> <459D766C.40505@ee.byu.edu> <459D8886.2000101@noaa.gov> <459D8B72.2040309@ee.byu.edu> <459D9590.80202@noaa.gov> <459D9D9C.30501@ee.byu.edu> Message-ID: <1e2af89e0701050633x3bd45a7bm1734cb72492ecff@mail.gmail.com> Hi, > > wxPython -- Robin Dunn > > PIL -- Fredrik Lundh > > PyOpenGL -- Who? > > PyObjC -- would it be useful there? (Ronald Oussoren) > > MatplotLib (but maybe it's already married to numpy...) > > PyGtk ? > > > > It's a good start, but their is also > > PyMedia, PyVoxel, any video-library interface writers, any audo-library > interface writers. Is there already, or could there be, some sort of consortium of these that agree on the features in the PEP? Best, Matthew From ndbecker2 at gmail.com Fri Jan 5 09:38:49 2007 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 05 Jan 2007 09:38:49 -0500 Subject: [Numpy-discussion] Latest Array-Interface PEP References: <459D6964.30701@ee.byu.edu> <459E5FF7.7080300@ee.byu.edu> Message-ID: Travis Oliphant wrote: > Neal Becker wrote: >> Travis Oliphant wrote: >> >> >>> I'm attaching my latest extended buffer-protocol PEP that is trying to >>> get the array interface into Python. Basically, it is a translation of >>> the numpy header files into something as simple as possible that can >>> still be used to describe a complicated block of memory to another user. >>> >>> My purpose is to get feedback and criticisms from this community before >>> display before the larger Python community. >>> >>> -Travis >>> >> >> I'm wondering if having the buffer object specify the view is the right >> choice. I think the best choice is to separate the design into: >> >> buffer: provides an interface to memory >> array: provides a view of memory as an array of whatever dimensions >> >> 1. buffer may or may not map to contiguous memory. >> 2. multiple views of the same memory can be shared. These different >> views could represent different slicings. >> > > I don't understand your concerns, could you please help clarify? > > I'm not doing anything with the buffer object at all. I'm only using > the buffer "protocol" (i.e. extending the set of function pointers > pointed to by tp_as_buffer in the type-object). > > -Travis Several extensions to Python utilize the buffer protocol to share the location of a data-buffer that is really an N-dimensional array. However, there is no standard way to exchange the additional N-dimensional array information so that the data-buffer is interpreted correctly. I am questioning if this is the best concept. It says that the data-buffer will carry the information about it's interpretation as an N-dimensional array. I'm thinking that a buffer is just an interface to memory, and that the interpretation as an array of n-dimensions, for example, is best left to the application. I might want to at one time view the data as n-dimensional, but at another time as 1-dimensional, for example. From stefan at sun.ac.za Fri Jan 5 09:51:20 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 5 Jan 2007 16:51:20 +0200 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <459E5FF7.7080300@ee.byu.edu> Message-ID: <20070105145120.GE6020@mentat.za.net> On Fri, Jan 05, 2007 at 09:38:49AM -0500, Neal Becker wrote: > Several extensions to Python utilize the buffer protocol to share > the location of a data-buffer that is really an N-dimensional > array. However, there is no standard way to exchange the > additional N-dimensional array information so that the data-buffer > is interpreted correctly. > > I am questioning if this is the best concept. It says that the data-buffer > will carry the information about it's interpretation as an N-dimensional > array. > > I'm thinking that a buffer is just an interface to memory, and that the > interpretation as an array of n-dimensions, for example, is best left to > the application. I might want to at one time view the data as > n-dimensional, but at another time as 1-dimensional, for example. You can always choose to ignore that information if you don't need it. On the other hand, if you *do* need it, how would you otherwise interpret an N-dimensional array, given only a buffer? Cheers St?fan From charlesr.harris at gmail.com Fri Jan 5 10:49:08 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 5 Jan 2007 08:49:08 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <20070105145120.GE6020@mentat.za.net> References: <459D6964.30701@ee.byu.edu> <459E5FF7.7080300@ee.byu.edu> <20070105145120.GE6020@mentat.za.net> Message-ID: On 1/5/07, Stefan van der Walt wrote: > > On Fri, Jan 05, 2007 at 09:38:49AM -0500, Neal Becker wrote: > > Several extensions to Python utilize the buffer protocol to share > > the location of a data-buffer that is really an N-dimensional > > array. However, there is no standard way to exchange the > > additional N-dimensional array information so that the data-buffer > > is interpreted correctly. > > > > I am questioning if this is the best concept. It says that the > data-buffer > > will carry the information about it's interpretation as an N-dimensional > > array. > > > > I'm thinking that a buffer is just an interface to memory, and that the > > interpretation as an array of n-dimensions, for example, is best left to > > the application. I might want to at one time view the data as > > n-dimensional, but at another time as 1-dimensional, for example. > > You can always choose to ignore that information if you don't need it. > On the other hand, if you *do* need it, how would you otherwise > interpret an N-dimensional array, given only a buffer? I think Neal is suggesting some object that basically does nothing but hold a pointer(s) to memory. This memory can be used in various ways, one of which is to use it construct another type of object that provides a view with indices and such, i.e., an array. That way the memory isn't tied to arrays and could concievable be used in other ways. The idea is analagous to the data/model/view paradigm. It is a bit cleaner than just ignoring the array parts. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Fri Jan 5 10:55:00 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 05 Jan 2007 08:55:00 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <459E5FF7.7080300@ee.byu.edu> Message-ID: <459E74D4.2070100@ee.byu.edu> Neal Becker wrote: > Several extensions to Python utilize the buffer protocol to share > the location of a data-buffer that is really an N-dimensional > array. However, there is no standard way to exchange the > additional N-dimensional array information so that the data-buffer > is interpreted correctly. > >I am questioning if this is the best concept. It says that the data-buffer >will carry the information about it's interpretation as an N-dimensional >array. > >I'm thinking that a buffer is just an interface to memory, and that the >interpretation as an array of n-dimensions, for example, is best left to >the application. I might want to at one time view the data as >n-dimensional, but at another time as 1-dimensional, for example. > > > The simple data-buffer interpretation is still there. You can still use simple "chunk-of-memory" only interpretation of the buffer. All we are doing is adding a way for applications to ask if the object can be interpreted as a strided N-dimensional array of a particular data-format. So, this proposal does nothing to jeopardize the buffer-as-an-interface-to-memory only model. I'm only using a table of funciton pointers which is already there (tp_as_buffer) rather than request an additional table of function pointers on the type object (tp_as_array). I see the array view idea as fitting very nicely with the buffer protocol. -Travis From oliphant at ee.byu.edu Fri Jan 5 10:56:21 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 05 Jan 2007 08:56:21 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <459E5FF7.7080300@ee.byu.edu> <20070105145120.GE6020@mentat.za.net> Message-ID: <459E7525.6000100@ee.byu.edu> Charles R Harris wrote: > > > On 1/5/07, *Stefan van der Walt* > wrote: > > On Fri, Jan 05, 2007 at 09:38:49AM -0500, Neal Becker wrote: > > Several extensions to Python utilize the buffer protocol to > share > > the location of a data-buffer that is really an N-dimensional > > array. However, there is no standard way to exchange the > > additional N-dimensional array information so that the > data-buffer > > is interpreted correctly. > > > > I am questioning if this is the best concept. It says that the > data-buffer > > will carry the information about it's interpretation as an > N-dimensional > > array. > > > > I'm thinking that a buffer is just an interface to memory, and > that the > > interpretation as an array of n-dimensions, for example, is best > left to > > the application. I might want to at one time view the data as > > n-dimensional, but at another time as 1-dimensional, for example. > > You can always choose to ignore that information if you don't need it. > On the other hand, if you *do* need it, how would you otherwise > interpret an N-dimensional array, given only a buffer? > > > I think Neal is suggesting some object that basically does nothing but > hold a pointer(s) to memory. This memory can be used in various ways, > one of which is to use it construct another type of object that > provides a view with indices and such, i.e., an array. That way the > memory isn't tied to arrays and could concievable be used in other > ways. The idea is analagous to the data/model/view paradigm. It is a > bit cleaner than just ignoring the array parts. Such an object would be useful. I would submit that it is what the buffer object "should be" But, we are talking about a different concept here --- the buffer protocol. -Travis From charlesr.harris at gmail.com Fri Jan 5 11:27:45 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 5 Jan 2007 09:27:45 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459E7525.6000100@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> <459E5FF7.7080300@ee.byu.edu> <20070105145120.GE6020@mentat.za.net> <459E7525.6000100@ee.byu.edu> Message-ID: On 1/5/07, Travis Oliphant wrote: > > Charles R Harris wrote: > > > > > > > On 1/5/07, *Stefan van der Walt* > > wrote: > > > > On Fri, Jan 05, 2007 at 09:38:49AM -0500, Neal Becker wrote: > > > Several extensions to Python utilize the buffer protocol to > > share > > > the location of a data-buffer that is really an N-dimensional > > > array. However, there is no standard way to exchange the > > > additional N-dimensional array information so that the > > data-buffer > > > is interpreted correctly. > > > > > > I am questioning if this is the best concept. It says that the > > data-buffer > > > will carry the information about it's interpretation as an > > N-dimensional > > > array. > > > > > > I'm thinking that a buffer is just an interface to memory, and > > that the > > > interpretation as an array of n-dimensions, for example, is best > > left to > > > the application. I might want to at one time view the data as > > > n-dimensional, but at another time as 1-dimensional, for example. > > > > You can always choose to ignore that information if you don't need > it. > > On the other hand, if you *do* need it, how would you otherwise > > interpret an N-dimensional array, given only a buffer? > > > > > > I think Neal is suggesting some object that basically does nothing but > > hold a pointer(s) to memory. This memory can be used in various ways, > > one of which is to use it construct another type of object that > > provides a view with indices and such, i.e., an array. That way the > > memory isn't tied to arrays and could concievable be used in other > > ways. The idea is analagous to the data/model/view paradigm. It is a > > bit cleaner than just ignoring the array parts. > > Such an object would be useful. I would submit that it is what the > buffer object "should be" Yeah. The problem is that we have a buffer API, not a buffer object. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattknox_ca at hotmail.com Fri Jan 5 11:42:40 2007 From: mattknox_ca at hotmail.com (Matt Knox) Date: Fri, 5 Jan 2007 11:42:40 -0500 Subject: [Numpy-discussion] custom accumlators Message-ID: I made a post about this a while ago on the scipy-user mailing list, but I didn't receive much of a response so I'm just throwing it out there again (with more detail) in case it got overlooked. Basically, I'd like to be able to do accumulate operations with custom functions. numpy.vectorize does not seem to provide an accumulate method with the functions it returns. I'm hoping I don't have to write ufuncs in C to accomplish this, but I fear that may the case. Either way, it would be nice to know if it can, or cannot be done in an easy manner. I have lots of examples of where this kind of thing is useful, but I'll just outline two for now. Assume the parameter x in all the functions below is a 1-d array --------------------------------------------- Example 1 - exponential moving average: # naive brute force method... def expmave(x, k): result = numpy.array(x, copy=True) for i in range(1, result.size): result[i] = result[i-1] + k * (result[i] - result[i-1]) return result # slicker method (if it worked, which it doesn't)... def expmave(x, k): def expmave_sub(a, b): return a + k * (b - a) return numpy.vectorize(expmave_sub).accumulate(x) --------------------------------------------- Example 2 - forward fill a masked array: # naive brute force method... def forward_fill(x): result = ma.array(x, copy=True) for i in range(1, result.size): if result[i] is ma.masked: result[i] = result[i-1] return result # slicker method (if it worked, which it doesn't)... def forward_fill(x): def forward_fill_sub(a, b): if b is ma.masked: return a else: return b return numpy.vectorize(forward_fill_sub).accumulate(x) --------------------------------------------- Is their a good way to do these kinds of things without python looping? Or is that going to require writing a ufunc in C? Any help is greatly appreciated. Thanks, - Matt Knox From karol.langner at kn.pl Fri Jan 5 12:18:46 2007 From: karol.langner at kn.pl (Karol Langner) Date: Fri, 5 Jan 2007 18:18:46 +0100 Subject: [Numpy-discussion] custom accumlators In-Reply-To: References: Message-ID: <200701051818.47072.karol.langner@kn.pl> On Friday 05 of January 2007 17:42, Matt Knox wrote: > --------------------------------------------- > Example 1 - exponential moving average: > > # naive brute force method... > def expmave(x, k): > result = numpy.array(x, copy=True) > for i in range(1, result.size): > result[i] = result[i-1] + k * (result[i] - result[i-1]) > return result > > # slicker method (if it worked, which it doesn't)... > def expmave(x, k): > def expmave_sub(a, b): > return a + k * (b - a) > return numpy.vectorize(expmave_sub).accumulate(x) Why can't you simply use list comprehensions? Too slow? For example: def expmave(x, k): return [x[0]] + [x[i-1] + k*(x[i]-x[i-1]) for i in range(1,len(x))] Karol -- written by Karol Langner Fri Jan 5 17:58:27 CET 2007 From mattknox_ca at hotmail.com Fri Jan 5 13:29:19 2007 From: mattknox_ca at hotmail.com (Matt Knox) Date: Fri, 5 Jan 2007 18:29:19 +0000 (UTC) Subject: [Numpy-discussion] custom accumlators References: <200701051818.47072.karol.langner@kn.pl> Message-ID: > > On Friday 05 of January 2007 17:42, Matt Knox wrote: > > --------------------------------------------- > > Example 1 - exponential moving average: > > > > # naive brute force method... > > def expmave(x, k): > > result = numpy.array(x, copy=True) > > for i in range(1, result.size): > > result[i] = result[i-1] + k * (result[i] - result[i-1]) > > return result > > > > # slicker method (if it worked, which it doesn't)... > > def expmave(x, k): > > def expmave_sub(a, b): > > return a + k * (b - a) > > return numpy.vectorize(expmave_sub).accumulate(x) > > Why can't you simply use list comprehensions? Too slow? > > For example: > def expmave(x, k): > return [x[0]] + [x[i-1] + k*(x[i]-x[i-1]) for i in range(1,len(x))] > > Karol > That's not the same calculation. There is no cumulative effect in your function. Each iteration depends on the previous iteration, like a cumulative sum, etc. And as for the speed, I'm basically trying to determine what is the most efficient way to do it, outside of writing it in C. And I don't really want to create intermediate lists, ideally things would be kept as numpy arrays. I can't think of a loop free way to do it with numpy. Good try though. No harm in trying! That's the only way we learn things. - Matt From tim.hochberg at ieee.org Fri Jan 5 14:22:47 2007 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 05 Jan 2007 12:22:47 -0700 Subject: [Numpy-discussion] custom accumlators In-Reply-To: References: Message-ID: <459EA587.4080807@ieee.org> Matt Knox wrote: > I made a post about this a while ago on the scipy-user mailing list, but I didn't receive much of a response so I'm just throwing it out there again (with more detail) in case it got overlooked. > > Basically, I'd like to be able to do accumulate operations with custom functions. numpy.vectorize does not seem to provide an accumulate method with the functions it returns. Note that if you are looking for speed, numpy.vectorize is probably not what you are looking for even if it did work for this sort of stuff. > I'm hoping I don't have to write ufuncs in C to accomplish this, but I fear that may the case. Either way, it would be nice to know if it can, or cannot be done in an easy manner. > > I have lots of examples of where this kind of thing is useful, but I'll just outline two for now. > [SNIP] It is not obvious how to write these using numpy's primitives. You might want to look into one of psyco, pyrex or weave. Psyco's probably the easiest, but will get you the smallest (although still large) speed gain, if you use psyco there are some tricks I could suggest for speeding it up. Pyrex is probably the most polished, but I haven't used it with numpy so I can't comment on how natural that is. -tim From robert.kern at gmail.com Fri Jan 5 14:37:12 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 05 Jan 2007 13:37:12 -0600 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <459E5FF7.7080300@ee.byu.edu> Message-ID: <459EA8E8.3070900@gmail.com> Neal Becker wrote: > I'm thinking that a buffer is just an interface to memory, and that the > interpretation as an array of n-dimensions, for example, is best left to > the application. I might want to at one time view the data as > n-dimensional, but at another time as 1-dimensional, for example. Sure, but you need a standard way to communicate that extra information between different parts of your code and also between different third party libraries. That is what this PEP intends to provide. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Fri Jan 5 15:08:26 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 5 Jan 2007 13:08:26 -0700 Subject: [Numpy-discussion] custom accumlators In-Reply-To: <459EA587.4080807@ieee.org> References: <459EA587.4080807@ieee.org> Message-ID: On 1/5/07, Tim Hochberg wrote: > > Matt Knox wrote: > > I made a post about this a while ago on the scipy-user mailing list, but > I didn't receive much of a response so I'm just throwing it out there again > (with more detail) in case it got overlooked. > > > > Basically, I'd like to be able to do accumulate operations with custom > functions. numpy.vectorize does not seem to provide an accumulate method > with the functions it returns. > Note that if you are looking for speed, numpy.vectorize is probably not > what you are looking for even if it did work for this sort of stuff. > > > I'm hoping I don't have to write ufuncs in C to accomplish this, but I > fear that may the case. Either way, it would be nice to know if it can, or > cannot be done in an easy manner. > > > > I have lots of examples of where this kind of thing is useful, but I'll > just outline two for now. > > > > [SNIP] > > It is not obvious how to write these using numpy's primitives. You might > want to look into one of psyco, pyrex or weave. Psyco's probably the > easiest, but will get you the smallest (although still large) speed > gain, if you use psyco there are some tricks I could suggest for > speeding it up. Pyrex is probably the most polished, but I haven't used > it with numpy so I can't comment on how natural that is. > > -tim I think what he needs is something like a linear prediction code or a IIP filter. The place to look would be in scipy, either in signal processing or statistics (ARMA). I don't know that it is there, but it might (should) be. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Fri Jan 5 16:39:26 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 05 Jan 2007 13:39:26 -0800 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459D9D9C.30501@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> <459D766C.40505@ee.byu.edu> <459D8886.2000101@noaa.gov> <459D8B72.2040309@ee.byu.edu> <459D9590.80202@noaa.gov> <459D9D9C.30501@ee.byu.edu> Message-ID: <459EC58E.7000408@noaa.gov> Travis Oliphant wrote: > Christopher Barker wrote: >> wxPython -- Robin Dunn >> PIL -- Fredrik Lundh >> PyOpenGL -- Who? >> PyObjC -- would it be useful there? (Ronald Oussoren) >> MatplotLib (but maybe it's already married to numpy...) >> PyGtk ? > It's a good start, but their is also > > PyMedia, PyVoxel, any video-library interface writers, any audo-library > interface writers. right -- I didn't intend that to be a comprehensive list. > Anybody who wants to wrap/write code that does some kind of manipulation > on a chunk of data of a specific data-format. > > There are so many people who would use it that I don't feel qualified to > speak for them all. I think this is key -- we all know that there are a lot of people that *could* use it, and we might even say *should* use it. The question that I think the core python devs want answered is *will* they use it. That's why I suggest that rather than having a bunch of numpy users make comments to python-dev, we really need authors of packages like the above to make comments to python-dev, saying "I could use this, and I'll *will* use this if it's been put into the standard lib". I do think there is one issue that does need to be addressed. The current buffer protocol already allows modules to share data without copying -- but it doesn't provide any description of that data. This proposal would provide more description of that data, but still not describe it completely -- that's just not possible, so how helpful is the additional description -- I think a lot, but others are not convinced. Some examples, from my use. I use wxPython a fair bit, so I'll use that as an example. Example 1: One can now currently pass a buffer into a wxImage constructor to create an image from existing data. For instance, you can pass in an WxHx3 numpy array of unsigned bytes in to create a WxH RGB image. At the moment, all the wx.Image constructor checks is if you've passed in the correct amount of bytes. With this proposal, it could check to see if you passed in a WxHx3 array of bytes. How helpful would that be? it's still up to the programmer to make sure those bytes actually represent what you want. You may catch a few errors, like accidentally passing in a 3xWxH array. You also wouldn't have to pass in the size of the Image you wanted - which would be kind of nice. One could also write the Image constructor so that it could take a few different shapes of data, and do the right thing with each of them. How compelling are these advantages? Example 2: When you need to draw something like a whole lot of pixels or a long polyline, you can now pass into wxPython either: a list of (x,y) tuples, or any sequence of any (x,y) sequences. A Nx2 numpy array appears as the latter, but it ends up being slower than the list of tuples, because wxPython has some code to optimize accessing lists of tuples. Internally, wxWidgets has drawing methods that accept a Nx2 c-array of ints. With the proposed protocol, wxPython could recognize that such an array was passed in, and save a LOT of sequence unpacking, type checking and converting, etc. It could also take multiple data types -- floats, ints, etc, and do the right thing with each of those. This to me is more compelling than the Image example. By the way, Robin Dunn has said that he doesn't want to include numpy dependency in wxPython, but would probably accept code that did the above if it didn't add any dependencies. Francesc Altet wrote: > In any case, I think that if the PEP has success, it would suppose an > extraordinary leap towards efficient data interchange in applications that > doesn't need (or are reluctant to include) NumPy for their normal operation. That's the question -- is it an extraordinary leap over what you can now do with the existing buffer protocol? Matthew Brett wrote: > Is there already, or could there be, some sort of consortium of these > that agree on the features in the PEP? There isn't now, and that's my question -- what is the best way to involve the developers of some of the many packages that we envision using this. - random polling by the numpy devs and users? - more organized polling by numpy devs (or Travis) - just a note encouraging them to pipe in on the discussion here? - a note encouraging them to pipe in on the discussion at python-dev. I think the PEP has far more chances of success if it's seen as a request from a variety of package developers, not just the numpy crowd (which, after all, already has numpy) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jswhit at fastmail.fm Fri Jan 5 16:41:41 2007 From: jswhit at fastmail.fm (Jeff Whitaker) Date: Fri, 05 Jan 2007 14:41:41 -0700 Subject: [Numpy-discussion] custom accumlators In-Reply-To: References: Message-ID: <459EC615.3060208@fastmail.fm> Matt Knox wrote: > I made a post about this a while ago on the scipy-user mailing list, but I didn't receive much of a response so I'm just throwing it out there again (with more detail) in case it got overlooked. > > Basically, I'd like to be able to do accumulate operations with custom functions. numpy.vectorize does not seem to provide an accumulate method with the functions it returns. I'm hoping I don't have to write ufuncs in C to accomplish this, but I fear that may the case. Either way, it would be nice to know if it can, or cannot be done in an easy manner. > > I have lots of examples of where this kind of thing is useful, but I'll just outline two for now. > > Assume the parameter x in all the functions below is a 1-d array > > --------------------------------------------- > Example 1 - exponential moving average: > > # naive brute force method... > def expmave(x, k): > result = numpy.array(x, copy=True) > for i in range(1, result.size): > result[i] = result[i-1] + k * (result[i] - result[i-1]) > return result > > # slicker method (if it worked, which it doesn't)... > def expmave(x, k): > def expmave_sub(a, b): > return a + k * (b - a) > return numpy.vectorize(expmave_sub).accumulate(x) > > --------------------------------------------- > Example 2 - forward fill a masked array: > > # naive brute force method... > def forward_fill(x): > result = ma.array(x, copy=True) > for i in range(1, result.size): > if result[i] is ma.masked: result[i] = result[i-1] > return result > > # slicker method (if it worked, which it doesn't)... > def forward_fill(x): > def forward_fill_sub(a, b): > if b is ma.masked: return a > else: return b > return numpy.vectorize(forward_fill_sub).accumulate(x) > --------------------------------------------- > > Is their a good way to do these kinds of things without python looping? Or is that going to require writing a ufunc in C? Any help is greatly appreciated. > > Thanks, > > - Matt Knox > Matt: Here's a quick and dirty example of how to do this sort of thing in pyrex. I do it all the time, and it works quite well. # accumulator.pyx: _doublesize = sizeof(double) cdef extern from "Python.h": int PyObject_AsWriteBuffer(object, void **rbuf, Py_ssize_t *len) char *PyString_AsString(object) def accumulator(object x, double k): cdef Py_ssize_t buflen cdef int ndim, i cdef double *xdata cdef void *xbuff # make a copy by casting to an array of doubles. x = x.astype('f8') # if buffer api is supported, get pointer to data buffers. if PyObject_AsWriteBuffer(x, &xbuff, &buflen) <> 0: raise RuntimeError('object does not support buffer API') ndim = buflen/_doublesize xdata = xbuff for i from 1 <= i < ndim: xdata[i] = xdata[i-1] + k * (xdata[i] - xdata[i-1]) return x # test.py from accumulator import accumulator from numpy import linspace x = linspace(1.,10.,10) k = 0.1 print x x1 = accumulator(x,k) print x1 def expmave(x, k): result = x.copy() for i in range(1, result.size): result[i] = result[i-1] + k * (result[i] - result[i-1]) return result x2 = expmave(x,k) print x2 # should be the same as x1 # setup.py import os from distutils.core import setup, Extension from Pyrex.Distutils import build_ext setup(name = "accumulator", cmdclass = {'build_ext': build_ext}, keywords = ["python","map projections","GIS","mapping","maps"], ext_modules = [Extension("accumulator",["accumulator.pyx"])]) to build, just do python setup.py build_ext --inplace then run test.py. HTH, -Jeff -- Jeffrey S. Whitaker Phone : (303)497-6313 NOAA/OAR/CDC R/PSD1 FAX : (303)497-6449 325 Broadway Boulder, CO, USA 80305-3328 From haase at msg.ucsf.edu Fri Jan 5 16:50:14 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 5 Jan 2007 13:50:14 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 Message-ID: Hi! After I upgraded from numpy 1.0 to 1.0.1 I get an abort in a C-module: RuntimeError: module compiled against version 1000002 of C-API but this version of numpy is 1000009 Fatal Python error: numpy.core.multiarray failed to import... exiting. /opt/bin/priithonN: line 37: 1409 Aborted $PY $* I thought that within numpy 1.0 there was no recompile for external C-modules needed !? Please explain. Thanks, Sebastian Haase From rowen at cesmail.net Fri Jan 5 16:53:52 2007 From: rowen at cesmail.net (Russell E Owen) Date: Fri, 05 Jan 2007 13:53:52 -0800 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray References: Message-ID: In article , "Sebastian Haase" wrote: > On 1/4/07, Sebastian Haase wrote: > > > >>> N.__version__ > > '1.0.2.dev3487' > > > > in any case: inside the script it somehow generated a nan --- is > > there a bug in numpy !? > > No bug here ! see below ! > > > I remember that there was some discussion about resize !? > > What should I add to the Scipy Wiki numarray page about this ? > > ( http://www.scipy.org/Converting_from_numarray ) > > > > OK - the reference problem in my interactive shell came from the > implicit '_' variable that always references the last result. But > maybe even more worry some for the converting from numarray is this: > >>> a = N.array([5]) > >>> 999 # to kill '_' - reference > 999 > >>> a.resize(2) > >>> a > [5 0] > > in numarray you would get > >>> a = na.array([5]) > >>> a.resize(2) > [5 5] > >>> a > [5 5] > > !! why is numpy filling with 0s and numarray repeats (cycles I think > is more what it does !) the last element(s) ?? > > How did numeric do this ? Here's what I get for Numeric 24.2: >>> import Numeric as N >>> a = N.array([5]) >>> a array([5]) >>> a.resize(2) Traceback (most recent call last): File "", line 1, in ValueError: cannot resize an array that has been referenced or is referencing another array in this way. Use the resize function. >>> N.resize(a, 2) Traceback (most recent call last): File "", line 1, in File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-pac kages/Numeric/Numeric.py", line 422, in resize total_size = multiply.reduce(new_shape) ValueError: dimension not in array >>> N.resize(a, [2]) array([5, 5]) From robert.kern at gmail.com Fri Jan 5 16:59:06 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 05 Jan 2007 15:59:06 -0600 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: References: Message-ID: <459ECA2A.9010906@gmail.com> Sebastian Haase wrote: > Hi! > After I upgraded from numpy 1.0 to 1.0.1 > I get an abort in a C-module: > > RuntimeError: module compiled against version 1000002 of C-API but this > version of numpy is 1000009 > Fatal Python error: numpy.core.multiarray failed to import... exiting. > /opt/bin/priithonN: line 37: 1409 Aborted $PY $* > > I thought that within numpy 1.0 there was no recompile > for external C-modules needed !? Ummm, the number was bumped before the 1.0 release. Bump of the version number: http://projects.scipy.org/scipy/numpy/changeset/3361 Tagging the 1.0 release: http://projects.scipy.org/scipy/numpy/changeset/3396/tags/1.0 Are you sure that you weren't upgrading from a beta instead of the 1.0 release? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rowen at cesmail.net Fri Jan 5 16:57:50 2007 From: rowen at cesmail.net (Russell E Owen) Date: Fri, 05 Jan 2007 13:57:50 -0800 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray References: Message-ID: In article , "Sebastian Haase" wrote: >...But maybe even more worry some for the converting from numarray is this: > >>> a = N.array([5]) > >>> 999 # to kill '_' - reference > 999 > >>> a.resize(2) > >>> a > [5 0] > > in numarray you would get > >>> a = na.array([5]) > >>> a.resize(2) > [5 5] > >>> a > [5 5] I also checked the numpy 1.0.1 help and I confess I don't understand at all what it claims to do if the new size is larger. It first says it repeats a and then it says it zero-fills the output. >>> help(numpy.resize) Help on function resize in module numpy.core.fromnumeric: resize(a, new_shape) resize(a,new_shape) returns a new array with the specified shape. The original array's total size can be any size. It fills the new array with repeated copies of a. Note that a.resize(new_shape) will fill array with 0's beyond current definition of a. From seb.haase at gmx.net Fri Jan 5 17:00:50 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Fri, 5 Jan 2007 14:00:50 -0800 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray In-Reply-To: References: Message-ID: On 1/5/07, Russell E Owen wrote: > In article > , > "Sebastian Haase" wrote: > > > On 1/4/07, Sebastian Haase wrote: > > > > > >>> N.__version__ > > > '1.0.2.dev3487' > > > > > > in any case: inside the script it somehow generated a nan --- is > > > there a bug in numpy !? > > > > No bug here ! see below ! > > > > > I remember that there was some discussion about resize !? > > > What should I add to the Scipy Wiki numarray page about this ? > > > ( http://www.scipy.org/Converting_from_numarray ) > > > > > > > OK - the reference problem in my interactive shell came from the > > implicit '_' variable that always references the last result. But > > maybe even more worry some for the converting from numarray is this: > > >>> a = N.array([5]) > > >>> 999 # to kill '_' - reference > > 999 > > >>> a.resize(2) > > >>> a > > [5 0] > > > > in numarray you would get > > >>> a = na.array([5]) > > >>> a.resize(2) > > [5 5] > > >>> a > > [5 5] > > > > !! why is numpy filling with 0s and numarray repeats (cycles I think > > is more what it does !) the last element(s) ?? > > > > How did numeric do this ? > > Here's what I get for Numeric 24.2: > > >>> import Numeric as N > >>> a = N.array([5]) > >>> a > array([5]) > >>> a.resize(2) > Traceback (most recent call last): > File "", line 1, in > ValueError: cannot resize an array that has been referenced or is > referencing > another array in this way. Use the resize function. > >>> N.resize(a, 2) > Traceback (most recent call last): > File "", line 1, in > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-pac > kages/Numeric/Numeric.py", line 422, in resize > total_size = multiply.reduce(new_shape) > ValueError: dimension not in array > >>> N.resize(a, [2]) > array([5, 5]) > Thanks for testing -- to be complete I will append my tests for numarray and numpy. I would consider the different result of function vs. method a bug !! Please comment , Sebastian. >>> import numarray as na >>> import numpy as N >>> a = na.array([5]) >>> a.resize(2) [5 5] >>> a [5 5] >>> a = na.array([5]) >>> na.resize(a,2) [5 5] >>> a [5] >>> a = N.array([5]) >>> a.resize(2) >>> a [5 0] >>> a = N.array([5]) >>> N.resize(a, 2) [5 5] >>> a [5] ### Note: [5 5] vs. [5 0] !!! From peridot.faceted at gmail.com Fri Jan 5 17:01:26 2007 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Fri, 5 Jan 2007 17:01:26 -0500 Subject: [Numpy-discussion] custom accumlators In-Reply-To: References: <459EA587.4080807@ieee.org> Message-ID: On 05/01/07, Charles R Harris wrote: > On 1/5/07, Tim Hochberg wrote: > > Matt Knox wrote: > > > Basically, I'd like to be able to do accumulate operations with custom > functions. numpy.vectorize does not seem to provide an accumulate method > with the functions it returns. If I understand correctly, you want to be able to write a python function that combines two scalars and returns another, then apply it (via vectorize or some such) like a ufunc's accumulate, or like python's reduce (for lists; for some reason the analogous function for iterators doesn't seem to exist). Have you tried using reduce? > > Note that if you are looking for speed, numpy.vectorize is probably not > > what you are looking for even if it did work for this sort of stuff. In fact, anything that goes through python code for the "combine two scalars" will be slow. The slowness of looping in python is not because python's looping constructs are slow, it's because executing python code is slow. So vectorize is kind of a cheat - it doesn't actually run fast, but it is convenient. [text deleted] > I think what he needs is something like a linear prediction code or a IIP > filter. The place to look would be in scipy, either in signal processing or > statistics (ARMA). I don't know that it is there, but it might (should) be. I don't think so. As he said in his original post, he's looking for a general function to produce this from any function which combines two scalars to produce another, whether it's been implemented in signal processing or not. It would be fairly straightforward to add this ability to the functions returned from vectorize(); it's implemented in python, IIRC, so one would just turn it into a class (SoftUfunc?) (with a __call__) and add an accumulate() function. It won't avoid python looping, but neither does vectorize, and in any case it can't be fast (even if the provided function were implemented in C calling back and forth through python would probably be a slowdown). And *convenience* is actually one of the biggest advantages of ufuncs, fancy indexing, and all that numpy whatnot. That said, it might also be worth looking at whether numexpr can do this sort of thing - it's supposed to take a numpy expression and evaluate it efficiently in C (avoiding intermediate arrays and so on). Finally, if you're careful, you can in fact abuse slicing and augmented assignment to do this efficiently: In [9]: a = ones(10) In [10]: a[1:]+=a[:-1] In [11]: a Out[11]: array([ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10.]) Unfortunately, this cannot really be adapted to anything but the operators with augmented assignment and things with an output argument - which includes ufuncs, but not, alas, vectorize. Really it would be nice if what vectorize() returned were effectively a ufunc, supporting all the various operations we might want from a ufunc (albeit inefficiently). This should not be difficult, but I am not up to writing it this evening. A. M. Archibald From seb.haase at gmx.net Fri Jan 5 17:06:53 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Fri, 5 Jan 2007 14:06:53 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <459ECA2A.9010906@gmail.com> References: <459ECA2A.9010906@gmail.com> Message-ID: You are right again, of course ! Sorry for the noise - I should have just checked the date of my so file (which is August 15) At least I understood the "official numpy intention of version 1.0" right then - just checking ... Thanks, Sebastian. On 1/5/07, Robert Kern wrote: > Sebastian Haase wrote: > > Hi! > > After I upgraded from numpy 1.0 to 1.0.1 > > I get an abort in a C-module: > > > > RuntimeError: module compiled against version 1000002 of C-API but this > > version of numpy is 1000009 > > Fatal Python error: numpy.core.multiarray failed to import... exiting. > > /opt/bin/priithonN: line 37: 1409 Aborted $PY $* > > > > I thought that within numpy 1.0 there was no recompile > > for external C-modules needed !? > > Ummm, the number was bumped before the 1.0 release. > > Bump of the version number: > http://projects.scipy.org/scipy/numpy/changeset/3361 > > Tagging the 1.0 release: > http://projects.scipy.org/scipy/numpy/changeset/3396/tags/1.0 > > Are you sure that you weren't upgrading from a beta instead of the 1.0 release? > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From tim.hochberg at ieee.org Fri Jan 5 17:51:14 2007 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 05 Jan 2007 15:51:14 -0700 Subject: [Numpy-discussion] custom accumlators In-Reply-To: References: <459EA587.4080807@ieee.org> Message-ID: <459ED662.7040008@ieee.org> A. M. Archibald wrote: [SNIP] > > Really it would be nice if what vectorize() returned were effectively > a ufunc, supporting all the various operations we might want from a > ufunc (albeit inefficiently). This should not be difficult, but I am > not up to writing it this evening. > You might want to look at frompyfunc: def expmave2(x, k): def expmave_sub(a, b): return a + k * (b - a) return np.frompyfunc(expmave_sub, 2, 1).accumulate(x) It's amazing what you find when you dig around. -tim From tim.hochberg at ieee.org Fri Jan 5 17:57:31 2007 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 05 Jan 2007 15:57:31 -0700 Subject: [Numpy-discussion] custom accumlators In-Reply-To: <459ED662.7040008@ieee.org> References: <459EA587.4080807@ieee.org> <459ED662.7040008@ieee.org> Message-ID: <459ED7DB.1020103@ieee.org> Tim Hochberg wrote: > A. M. Archibald wrote: > > > [SNIP] > >> Really it would be nice if what vectorize() returned were effectively >> a ufunc, supporting all the various operations we might want from a >> ufunc (albeit inefficiently). This should not be difficult, but I am >> not up to writing it this evening. >> >> > You might want to look at frompyfunc: > > def expmave2(x, k): > def expmave_sub(a, b): > return a + k * (b - a) > return np.frompyfunc(expmave_sub, 2, 1).accumulate(x) > > > It's amazing what you find when you dig around. > I guess that this should have an astype since this comes back as an object array. It would be cool if there was a way to have it return certain types, just for efficiency reasons, but that would require actual work on someones part. -tim > From oliphant at ee.byu.edu Fri Jan 5 18:52:09 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 05 Jan 2007 16:52:09 -0700 Subject: [Numpy-discussion] custom accumlators In-Reply-To: <459ED662.7040008@ieee.org> References: <459EA587.4080807@ieee.org> <459ED662.7040008@ieee.org> Message-ID: <459EE4A9.3080703@ee.byu.edu> Tim Hochberg wrote: >A. M. Archibald wrote: > > >[SNIP] > > >>Really it would be nice if what vectorize() returned were effectively >>a ufunc, supporting all the various operations we might want from a >>ufunc (albeit inefficiently). This should not be difficult, but I am >>not up to writing it this evening. >> >> >> >You might want to look at frompyfunc: > > def expmave2(x, k): > def expmave_sub(a, b): > return a + k * (b - a) > return np.frompyfunc(expmave_sub, 2, 1).accumulate(x) > > > > Thanks, Tim. This is what I was going to mention. Vectorize is built on top of frompyfunc which lets you create "object-array" ufuncs from python functions. All ufuncs have an accumluate method. -Travis From v-nijs at kellogg.northwestern.edu Fri Jan 5 19:54:35 2007 From: v-nijs at kellogg.northwestern.edu (Vincent Nijs) Date: Fri, 05 Jan 2007 18:54:35 -0600 Subject: [Numpy-discussion] x[None] changes x.shape In-Reply-To: <459EE4A9.3080703@ee.byu.edu> Message-ID: Say I use a function that expects a boolean array called sel to be passed as an argument: def foo(x,sel = None): return x[sel] If x is a 1-d array and sel is a (1-d) boolean array, x.shape will give (n,) where n is len(x). However, if the default value None is used (i.e., when no boolean array is passed) x.shape will give (1,n). Is that expected behavior? If so is there an alternative default to I could use that would return the entire x array and have x.shape be (n,)? Vincent From robert.kern at gmail.com Fri Jan 5 20:00:27 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 05 Jan 2007 19:00:27 -0600 Subject: [Numpy-discussion] x[None] changes x.shape In-Reply-To: References: Message-ID: <459EF4AB.3050609@gmail.com> Vincent Nijs wrote: > Say I use a function that expects a boolean array called sel to be passed as > an argument: > > def foo(x,sel = None): > return x[sel] > > If x is a 1-d array and sel is a (1-d) boolean array, x.shape will give (n,) > where n is len(x). > > However, if the default value None is used (i.e., when no boolean array is > passed) x.shape will give (1,n). > > Is that expected behavior? Yes. numpy.newaxis is just an alias for None, so x[None] is the same as x[numpy.newaxis]. > If so is there an alternative default to I could use that would return the > entire x array and have x.shape be (n,)? def foo(x, sel=None): if sel is None: return sel else: return x[sel] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bthom at cs.hmc.edu Fri Jan 5 20:07:59 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Fri, 05 Jan 2007 17:07:59 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 Message-ID: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> Sebastian, I had the same problem awhile back; I'm curious---how'd you fix your mismatch (i.e. where'd you go to get code, did you run python setup?, ...). I realize these are very basic questions, but I've never installed anything from source (aside from using easy_install), so it would be nice to extend my very limited knowledge in this area. Thx, --b ============================ You are right again, of course ! Sorry for the noise - I should have just checked the date of my so file (which is August 15) At least I understood the "official numpy intention of version 1.0" right then - just checking ... Thanks, Sebastian. On 1/5/07, Robert Kern wrote: > Sebastian Haase wrote: > > Hi! > > After I upgraded from numpy 1.0 to 1.0.1 > > I get an abort in a C-module: > > > > RuntimeError: module compiled against version 1000002 of C-API but this > > version of numpy is 1000009 > > Fatal Python error: numpy.core.multiarray failed to import... exiting. > > /opt/bin/priithonN: line 37: 1409 Aborted $PY $* > > > > I thought that within numpy 1.0 there was no recompile > > for external C-modules needed !? > > Ummm, the number was bumped before the 1.0 release. > > Bump of the version number: > http://projects.scipy.org/scipy/numpy/changeset/3361 > > Tagging the 1.0 release: > http://projects.scipy.org/scipy/numpy/changeset/3396/tags/1.0 > > Are you sure that you weren't upgrading from a beta instead of the 1.0 release? > From seb.haase at gmx.net Fri Jan 5 20:32:13 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Fri, 5 Jan 2007 17:32:13 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> Message-ID: Hi, All I did is recompiling my (on source code file) C extension. I made sure that it was including the current numpy header files. I did not use anything related to distutils ( no "python setup.py ..." ). Does that answer your question ? -Sebastian On 1/5/07, belinda thom wrote: > Sebastian, > > I had the same problem awhile back; I'm curious---how'd you fix your > mismatch (i.e. where'd you go to get code, did you run python > setup?, ...). I realize these are very basic questions, but I've > never installed anything from source (aside from using easy_install), > so it would be nice to extend my very limited knowledge in this area. > > Thx, > > --b > > ============================ > > > You are right again, of course ! > Sorry for the noise - I should have just checked the date of my so > file (which is August 15) > > At least I understood the "official numpy intention of version 1.0" > right then - just checking ... > > Thanks, > Sebastian. > > On 1/5/07, Robert Kern wrote: > > Sebastian Haase wrote: > > > Hi! > > > After I upgraded from numpy 1.0 to 1.0.1 > > > I get an abort in a C-module: > > > > > > RuntimeError: module compiled against version 1000002 of C-API > but this > > > version of numpy is 1000009 > > > Fatal Python error: numpy.core.multiarray failed to import... > exiting. > > > /opt/bin/priithonN: line 37: 1409 Aborted $PY $* > > > > > > I thought that within numpy 1.0 there was no recompile > > > for external C-modules needed !? > > > > Ummm, the number was bumped before the 1.0 release. > > > > Bump of the version number: > > http://projects.scipy.org/scipy/numpy/changeset/3361 > > > > Tagging the 1.0 release: > > http://projects.scipy.org/scipy/numpy/changeset/3396/tags/1.0 > > > > Are you sure that you weren't upgrading from a beta instead of the > 1.0 release? > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From bthom at cs.hmc.edu Fri Jan 5 20:53:37 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Fri, 05 Jan 2007 17:53:37 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> Message-ID: <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> On Jan 5, 2007, at 5:32 PM, Sebastian Haase wrote: > Hi, > All I did is recompiling my (on source code file) C extension. I made > sure that it was including the current numpy header files. Where are these files located? What command did you use? (The gorey details would help me quite a bit, as I'm to compile-level installs, having usually relied on things like macports port to do the work for me)... > > I did not use anything related to distutils ( no "python > setup.py ..." ). > > Does that answer your question ? Almost... > > -Sebastian From robert.kern at gmail.com Fri Jan 5 21:01:50 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 05 Jan 2007 20:01:50 -0600 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> Message-ID: <459F030E.8060001@gmail.com> belinda thom wrote: > On Jan 5, 2007, at 5:32 PM, Sebastian Haase wrote: > >> Hi, >> All I did is recompiling my (on source code file) C extension. I made >> sure that it was including the current numpy header files. > > Where are these files located? What command did you use? (The gorey > details would help me quite a bit, as I'm to compile-level installs, > having usually relied on things like macports port to do the work for > me)... Actually, it's more likely to confuse the matter since Sebastian is using a quite nonstandard way of building Python extension modules. What packages do you want help with building from source? Which build instructions have you read? What issues still remain unclear for you after having read them? What platforms are you concerned with? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From seb.haase at gmx.net Fri Jan 5 21:20:44 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Fri, 5 Jan 2007 18:20:44 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> Message-ID: package header would usually be found in PREFIX/include/python2.x (with PREFIX being something like: /usr/lib/python24 or C:/python24 ) However for obscure reasons in numpy the header files are in PREFIX/lib/python2.x/site-packages/numpy/core You have to "somehow" add this as '-I ' to your compiler command line -- this is what setup.py would do for you. ( maybe you need to look in /usr/local/lib/... ) -Sebastian On 1/5/07, belinda thom wrote: > > On Jan 5, 2007, at 5:32 PM, Sebastian Haase wrote: > > > Hi, > > All I did is recompiling my (on source code file) C extension. I made > > sure that it was including the current numpy header files. > > Where are these files located? What command did you use? (The gorey > details would help me quite a bit, as I'm to compile-level installs, > having usually relied on things like macports port to do the work for > me)... > > > > > I did not use anything related to distutils ( no "python > > setup.py ..." ). > > > > Does that answer your question ? > > Almost... > > > > > -Sebastian > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From mattknox_ca at hotmail.com Fri Jan 5 22:01:44 2007 From: mattknox_ca at hotmail.com (Matt Knox) Date: Sat, 6 Jan 2007 03:01:44 +0000 (UTC) Subject: [Numpy-discussion] custom accumlators References: <459EA587.4080807@ieee.org> <459ED662.7040008@ieee.org> <459ED7DB.1020103@ieee.org> Message-ID: > >> > > You might want to look at frompyfunc: > > > > def expmave2(x, k): > > def expmave_sub(a, b): > > return a + k * (b - a) > > return np.frompyfunc(expmave_sub, 2, 1).accumulate(x) > > > > > > It's amazing what you find when you dig around. Thanks a lot everyone. This has been informative. For what it's worth, I did some performance comparisons... import numpy as np import profile def expmave1(x, k): def expmave_sub(a, b): return b + k * (a - b) return np.frompyfunc(expmave_sub, 2, 1).accumulate(x).astype(x.dtype) def expmave2(x, k): result = np.array(x, copy=True) for i in range(1, result.size): result[i] = result[i-1] + k * (result[i] - result[i-1]) return result testArray = np.cumprod(1 + np.random.normal(size=10000)/100) profile.run('expmave1(testArray, 0.2)') profile.run('expmave2(testArray, 0.2)') and the second function is faster, which I guess makes sense if frompyfunc is pure python, although the first one does have a nice elegance to it I think. And actually, the speed of expmave2 is quite sufficient for my use cases when working with standard arrays... it's faster than I thought because initially I was doing this with masked arrays directly and the __getitem__ and __setitem__ calls are very expensive for masked arrays (expensive to the point where they can't really be used in looping functions like this), but I can get around that by just filling the array first and then computing the appropriate resulting mask separately. Obviously it would be faster in pure C, but the effort isn't worth it for my needs. > That said, it might also be worth looking at whether numexpr can do > this sort of thing - it's supposed to take a numpy expression and > evaluate it efficiently in C (avoiding intermediate arrays and so on). numexpr does look interesting. If numexpr was adapted to be able to do accumulations with expressions passed to it, that would be super slick... but that is probably a few leaps beyond my current knowledge level to code that. Thanks again everybody, - Matt From tim.hochberg at ieee.org Fri Jan 5 22:23:57 2007 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 05 Jan 2007 20:23:57 -0700 Subject: [Numpy-discussion] custom accumlators In-Reply-To: References: <459EA587.4080807@ieee.org> <459ED662.7040008@ieee.org> <459ED7DB.1020103@ieee.org> Message-ID: <459F164D.4080904@ieee.org> Matt Knox wrote: >>>> >>>> >>> You might want to look at frompyfunc: >>> >>> def expmave2(x, k): >>> def expmave_sub(a, b): >>> return a + k * (b - a) >>> return np.frompyfunc(expmave_sub, 2, 1).accumulate(x) >>> >>> >>> It's amazing what you find when you dig around. >>> > > Thanks a lot everyone. This has been informative. For what it's worth, I did > some performance comparisons... > > import numpy as np > import profile > > def expmave1(x, k): > def expmave_sub(a, b): > return b + k * (a - b) > return np.frompyfunc(expmave_sub, 2, 1).accumulate(x).astype(x.dtype) > > > def expmave2(x, k): > result = np.array(x, copy=True) > for i in range(1, result.size): > result[i] = result[i-1] + k * (result[i] - result[i-1]) > return result > > testArray = np.cumprod(1 + np.random.normal(size=10000)/100) > > > profile.run('expmave1(testArray, 0.2)') > profile.run('expmave2(testArray, 0.2)') > > and the second function is faster, which I guess makes sense if frompyfunc is > pure python, although the first one does have a nice elegance to it I think. > Are you sure about this? I ran this case using timeit, and the first one was 5 times or so *faster* than the second case. I just dug around and frompyfunc is acutally implemented in C, although it has to call back into python to execute the function being vectorized. Can you try using timeit instead of profile and see what you get? For example: a = np.cumprod(1 + np.random.normal(size=10000)/10) if __name__ == '__main__': from timeit import Timer print Timer('expmave1(a, .5)', 'from scratch import np, expmave1, a').timeit(10) print Timer('expmave2(a, .5)', 'from scratch import np, expmave2, a').timeit(10) Anyway, I'm glad that all was helpful. -tim -tim From tim.hochberg at ieee.org Fri Jan 5 23:06:17 2007 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri, 05 Jan 2007 21:06:17 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459EC58E.7000408@noaa.gov> References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> <459D766C.40505@ee.byu.edu> <459D8886.2000101@noaa.gov> <459D8B72.2040309@ee.byu.edu> <459D9590.80202@noaa.gov> <459D9D9C.30501@ee.byu.edu> <459EC58E.7000408@noaa.gov> Message-ID: <459F2039.30407@ieee.org> Christopher Barker wrote: [SNIP] > I think the PEP has far more chances of success if it's seen as a > request from a variety of package developers, not just the numpy crowd > (which, after all, already has numpy This seems eminently sensible. Getting a few developers from other projects on board would help a lot; it might also reveal some deficiencies to the proposal that we don't see yet. I've only given the PEP a quick read through at this point, but here a couple of comments: 1. It seems very numpy-centric. That's not necessarily bad, but I think it would help to have some outsiders look it over -- perhaps they would see things that they need that it doesn't address. Conversely, there may universal opinion that some parts of it aren't needed, and we can strip the proposal down somewhat. 2. It seems pretty complicated. In particular, the PyDataFormatObject seems pretty complicated. This part in particular seems like it might be a hard sell, so I expect this is going to need considerable more motivation. For example: 1. Why do we need Py_ARRAYOF? Can't we get the same effect just using longer shape and strides arrays? 2. Is there any type besides Py_STRUCTURE that can have names and fields. If so, what and what do they mean. If not, you should just say that. 3. And on this topic, why a tuple of ([names,..], {field})? Why not simply a list of (name, dfobject, offset, meta) for example? And what's the meta information if it's not PyNone? Just a string? Anything at all? I'll try to give it a more thorough reading over the weekend. -tim From mattknox_ca at hotmail.com Sat Jan 6 01:21:44 2007 From: mattknox_ca at hotmail.com (Matt Knox) Date: Sat, 6 Jan 2007 06:21:44 +0000 (UTC) Subject: [Numpy-discussion] custom accumlators References: <459EA587.4080807@ieee.org> <459ED662.7040008@ieee.org> <459ED7DB.1020103@ieee.org> <459F164D.4080904@ieee.org> Message-ID: > Are you sure about this? I ran this case using timeit, and the first one > was 5 times or so *faster* than the second case. I just dug around and > frompyfunc is acutally implemented in C, although it has to call back > into python to execute the function being vectorized. Can you try using > timeit instead of profile and see what you get? For example: > > a = np.cumprod(1 + np.random.normal(size=10000)/10) > > if __name__ == '__main__': > from timeit import Timer > print Timer('expmave1(a, .5)', 'from scratch import np, > expmave1, a').timeit(10) > print Timer('expmave2(a, .5)', 'from scratch import np, > expmave2, a').timeit(10) > > Anyway, I'm glad that all was helpful. > > -tim > > -tim > wow, you're right. Good call. profile and timeit give conflicting results (and indeed, the timeit results are more accurate, I made a manual timer to test and compared). I'll have to be more careful using profile in the future. So that's pretty awesome then, a 5 times speed up using frompyfunc and accumulate vs the brute force way of doing it. And I think the code looks a little nicer too which is always nice. Definitely a technique I will be using in the future. - Matt From karol.langner at kn.pl Sat Jan 6 02:44:30 2007 From: karol.langner at kn.pl (Karol Langner) Date: Sat, 6 Jan 2007 08:44:30 +0100 Subject: [Numpy-discussion] custom accumlators In-Reply-To: References: <200701051818.47072.karol.langner@kn.pl> Message-ID: <200701060844.30298.karol.langner@kn.pl> On Friday 05 of January 2007 19:29, Matt Knox wrote: > > On Friday 05 of January 2007 17:42, Matt Knox wrote: > > > --------------------------------------------- > > > Example 1 - exponential moving average: > > > > > > # naive brute force method... > > > def expmave(x, k): > > > result = numpy.array(x, copy=True) > > > for i in range(1, result.size): > > > result[i] = result[i-1] + k * (result[i] - result[i-1]) > > > return result > > > > > > # slicker method (if it worked, which it doesn't)... > > > def expmave(x, k): > > > def expmave_sub(a, b): > > > return a + k * (b - a) > > > return numpy.vectorize(expmave_sub).accumulate(x) > > > > Why can't you simply use list comprehensions? Too slow? > > > > For example: > > def expmave(x, k): > > return [x[0]] + [x[i-1] + k*(x[i]-x[i-1]) for i in range(1,len(x))] > > > > Karol > > That's not the same calculation. There is no cumulative effect in your > function. > > Each iteration depends on the previous iteration, like a cumulative sum, > etc. > > And as for the speed, I'm basically trying to determine what is the most > efficient way to do it, outside of writing it in C. And I don't really want > to create intermediate lists, ideally things would be kept as numpy arrays. > I can't think of a loop free way to do it with numpy. > > Good try though. No harm in trying! That's the only way we learn things. You're right, I didn't think that throught. Sorry. -- written by Karol Langner Sat Jan 6 08:43:50 CET 2007 From seanl at literati.org Sat Jan 6 13:38:03 2007 From: seanl at literati.org (Sean R. Lynch) Date: Sat, 06 Jan 2007 10:38:03 -0800 Subject: [Numpy-discussion] Advanced selection, duplicate indices, and augmented assignment Message-ID: <459FEC8B.4010904@literati.org> I have an array of vertices and an array of indices that make a list of triangles. I'd like to calculate the normals for each vertex as a (pseudo-)average of the normals of each triangle that contains it, so I get smooth lighting in OpenGL. I'd also like to avoid looping in Python because the vertex positions need to be able to be updated interactively. I tried using the following code: def calculate_normals(self): i = self.indices faces = self.vertices[i].reshape((i.shape[0] / 3, 3, 3)) face_normals = cross(faces[:,1] - faces[:,0], faces[:,2] - faces[:,0]) normals = zeros(self.vertices.shape, float32) normals[i] += face_normals.repeat(3, axis=0) normals /= sqrt(sum(normals*normals, axis=1)).reshape(normals.shape[0], 1) self.normals = normals but this appears to only add the face normal of the last face that contains each vertex, so I don't get averaged normals but the normals of a single face each. A test in the interactive interpreter shows why: >>> x = zeros((3,)) >>> x[array([0, 1, 1])] += array([1, 1, 1]) >>> x array([ 1., 1., 0.]) If this worked the way I was hoping, the output would be [1 2 0] because it would add to element 1 twice due to its duplication in the advanced selection array. Is the current behavior intentional or is it an accident of implementation? I think the multiple operations on elements with duplicated indices is more intuitive and it makes it possible to "invert" the expansion due to duplicate indices, whereas dropping the duplicate indices throws away information. Are people relying on the current behavior with duplicate indices and augmented assignment? I'd be happy with another way to do this that doesn't require a loop in Python or lots of temporary arrays, as well. Does anybody know of one? -Sean P.S. Could somebody please update http://numpy.scipy.org/ to point to the new list? I wasted a lot of time subscribing and posting to the wrong list. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature URL: From v-nijs at kellogg.northwestern.edu Sat Jan 6 14:19:27 2007 From: v-nijs at kellogg.northwestern.edu (Vincent Nijs) Date: Sat, 06 Jan 2007 13:19:27 -0600 Subject: [Numpy-discussion] x[None] changes x.shape In-Reply-To: <459EF4AB.3050609@gmail.com> Message-ID: Ah ... I did not make the link between newaxis and None. While the proposed solution works I found an alternative that gave me exactly what I was looking for: def foo(x, sel=()) return x[sel] I.e., passing an empty tuple returns the entire array without changing x.shape or making a copy. Vincent On 1/5/07 7:00 PM, "Robert Kern" wrote: > Vincent Nijs wrote: >> Say I use a function that expects a boolean array called sel to be passed as >> an argument: >> >> def foo(x,sel = None): >> return x[sel] >> >> If x is a 1-d array and sel is a (1-d) boolean array, x.shape will give (n,) >> where n is len(x). >> >> However, if the default value None is used (i.e., when no boolean array is >> passed) x.shape will give (1,n). >> >> Is that expected behavior? > > Yes. numpy.newaxis is just an alias for None, so x[None] is the same as > x[numpy.newaxis]. > >> If so is there an alternative default to I could use that would return the >> entire x array and have x.shape be (n,)? > > def foo(x, sel=None): > if sel is None: > return sel > else: > return x[sel] From robert.kern at gmail.com Sat Jan 6 18:34:59 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 06 Jan 2007 17:34:59 -0600 Subject: [Numpy-discussion] Advanced selection, duplicate indices, and augmented assignment In-Reply-To: <459FEC8B.4010904@literati.org> References: <459FEC8B.4010904@literati.org> Message-ID: <45A03223.5050208@gmail.com> Sean R. Lynch wrote: >>>> x = zeros((3,)) >>>> x[array([0, 1, 1])] += array([1, 1, 1]) >>>> x > array([ 1., 1., 0.]) > > If this worked the way I was hoping, the output would be [1 2 0] because > it would add to element 1 twice due to its duplication in the advanced > selection array. > > Is the current behavior intentional or is it an accident of > implementation? It is an unavoidable consequence of the way Python interprets that code and the way numpy arrays are fundamentally implemented. See Travis Oliphant's, Tim Hochberg's and my posts in the thread "Histograms via indirect index arrays" for more details. http://projects.scipy.org/pipermail/numpy-discussion/2006-March/thread.html#6877 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tomar.sk.ac at gmail.com Sat Jan 6 18:55:51 2007 From: tomar.sk.ac at gmail.com (Satyendra Tomar) Date: Sun, 7 Jan 2007 00:55:51 +0100 Subject: [Numpy-discussion] Installing numpy from rpm Message-ID: <7deedc7e0701061555j785f5d6l1b2d58bde0753987@mail.gmail.com> Hi, I'm entirely new to Python. I just used "rpm -Uvh numpy-1.0.1-1.src.rpm" to install numpy from the rpm package, running under SuSE 10.1, 2.6.16.21, python 2.4.2, GCC 4.1.0 but python can't still find the numpy module. What I only find is /usr/src/packages/SPECS/numpy.spec /usr/src/packages/SOURCES/numpy-1.0.1.tar.gz What else I'm supposed to do? Thanks in advance. S. K. Tomar -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Sat Jan 6 22:03:43 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sat, 06 Jan 2007 20:03:43 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459F2039.30407@ieee.org> References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> <459D766C.40505@ee.byu.edu> <459D8886.2000101@noaa.gov> <459D8B72.2040309@ee.byu.edu> <459D9590.80202@noaa.gov> <459D9D9C.30501@ee.byu.edu> <459EC58E.7000408@noaa.gov> <459F2039.30407@ieee.org> Message-ID: <45A0630F.9050804@ee.byu.edu> Tim Hochberg wrote: > Christopher Barker wrote: > > [SNIP] > >> I think the PEP has far more chances of success if it's seen as a >> request from a variety of package developers, not just the numpy crowd >> (which, after all, already has numpy >> > This seems eminently sensible. Getting a few developers from other > projects on board would help a lot; it might also reveal some > deficiencies to the proposal that we don't see yet. > It would help quite a bit. Are there any suggestions of who to recruit to review the proposal? We should not forget that the NumPy world is quite diverse as well. > I've only given the PEP a quick read through at this point, but here a > couple of comments: > Thank you for taking the time to read through it. I know it takes precious effort to do all this, which is why it's been so slow in coming from my end. It is important to get a lot of discussion on something like this. A lot of what is in the PEP does stem from a lot of discussion that's happened in the past 10 years, but admittedly some of it doesn't (extended data-format descriptions for example.). > 1. It seems very numpy-centric. That's not necessarily bad, but I > think it would help to have some outsiders look it over -- perhaps > they would see things that they need that it doesn't address. > Conversely, there may universal opinion that some parts of it > aren't needed, and we can strip the proposal down somewhat. > Yes, this is true. I took the struct module, NumPy, and c-types as a guide for "what is needed" to be described in terms of memory. > 2. It seems pretty complicated. In particular, the PyDataFormatObject > seems pretty complicated. This part in particular seems like it > might be a hard sell, so I expect this is going to need > considerable more motivation. For example: > Yes, the PyDataFormatObject is complicated --- but I don't think un-necessarily so. I've stripped a lot of it away from what's in NumPy to reduce it already. The question really is how are you going to describe what an arbitrary chunk of memory represents. One could restrict it to primitive types and replace the PyDataFormatObject with the enumerated typed and just give up on describing more complicated structures. But, my contention is why? Numarray and NumPy and C-types have already laid a tremendous amount of groundwork in how we can represent complicated data-structures. They clearly exist so why shouldn't we have some mechansim to describe them. Once you decide to handle complicated types you need to replace the simple enumerated type with something that is "self-recursive" (i.e. so you can have fields of arbitrary data-types). This lends itself to some-kind of structure design like the PyDataFormatObject. The only difference in what I've proposed to the c-types approach is that c-types over-loads Python Type Objects. (In other-words the PyDataFormatObject equivalent in c-types is at it's core a PyTypeObject while here it is built on PyObject). > 1. Why do we need Py_ARRAYOF? Can't we get the same effect just > using longer shape and strides arrays? > Yes, this is true for a single data-format in isolation (and in fact exactly what you get when you instantiate in NumPy a data-type that is an array of another primitive data-type). However, how do you describe a structure whose second field is an array of a primitive type? This is where the ARRAYOF qualifier is needed. In NumPy, actually, it's not done this way, but a separate subarray field in the data-type object is used. After studying c-types, however, I think this approach is better. > 2. Is there any type besides Py_STRUCTURE that can have names > and fields. If so, what and what do they mean. If not, you > should just say that. > Yes, you can add fields to a multi-byte primitive if you want. This would be similar to thinking about the data-format as a C-like union. Perhaps the data-field has meaning as a 4-byte integer but the most-significant and least-significant bytes should also be addressable individually. > 3. And on this topic, why a tuple of ([names,..], {field})? Why > not simply a list of (name, dfobject, offset, meta) for > example? And what's the meta information if it's not PyNone? > Just a string? Anything at all? > The list of names is useful for having an ordered list so you can traverse the structure in field order. It is technically not necessary but it makes it a lot easier to parse a data-format object in offset order (it is used a bit in NumPy, for example). The meta information is a place holder for field tags and future growth (kind of like column headers in a spreadsheet). It started as a place to put a "longer" name or to pass along information about a field (like units) through. -Travis From tim.hochberg at ieee.org Sat Jan 6 22:25:08 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Sat, 6 Jan 2007 20:25:08 -0700 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> Message-ID: On 1/6/07, Robert Kern wrote: > > Sean R. Lynch wrote: > > >>>> x = zeros((3,)) > >>>> x[array([0, 1, 1])] += array([1, 1, 1]) > >>>> x > > array([ 1., 1., 0.]) > > > > If this worked the way I was hoping, the output would be [1 2 0] because > > > it would add to element 1 twice due to its duplication in the advanced > > selection array. > > > > Is the current behavior intentional or is it an accident of > > implementation? > > It is an unavoidable consequence of the way Python interprets that code > and the > way numpy arrays are fundamentally implemented. See Travis Oliphant's, Tim > Hochberg's and my posts in the thread "Histograms via indirect index > arrays" for > more details. > > > http://projects.scipy.org/pipermail/numpy-discussion/2006-March/thread.html#6877 Do we have to revisit that thread? I seem to recall it getting kind of cranky. To avoid reliving that, I will attempt dredge up the relevant issue: "a[indx]+=b" should be the same as "a[indx]=a[indx]+b". All else follow from that. If staring at that for a while doesn't enlighten you, then you will have to read that thread. [ I suspect that in theory we could make the += form behave as you expect, but that would break the identity above, which would confuse a bunch of people] -tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Sat Jan 6 22:35:16 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sat, 06 Jan 2007 20:35:16 -0700 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> Message-ID: <45A06A74.3090001@ee.byu.edu> Timothy Hochberg wrote: > > > > On 1/6/07, *Robert Kern* < robert.kern at gmail.com > > wrote: > > Sean R. Lynch wrote: > > >>>> x = zeros((3,)) > >>>> x[array([0, 1, 1])] += array([1, 1, 1]) > >>>> x > > array([ 1., 1., 0.]) > > > > If this worked the way I was hoping, the output would be [1 2 0] > because > > it would add to element 1 twice due to its duplication in the > advanced > > selection array. > > > > Is the current behavior intentional or is it an accident of > > implementation? > > It is an unavoidable consequence of the way Python interprets that > code and the > way numpy arrays are fundamentally implemented. See Travis > Oliphant's, Tim > Hochberg's and my posts in the thread "Histograms via indirect > index arrays" for > more details. > > http://projects.scipy.org/pipermail/numpy-discussion/2006-March/thread.html#6877 > > > Do we have to revisit that thread? I seem to recall it getting kind of > cranky. To avoid reliving that, I will attempt dredge up the relevant > issue: > > "a[indx]+=b" should be the same as "a[indx]=a[indx]+b". All else > follow from that. If staring at that for a while doesn't enlighten > you, then you will have to read that thread. > > [ I suspect that in theory we could make the += form behave as you > expect, but that would break the identity above, which would confuse a > bunch of people] > I don't think we could make it work as he expects (and not radically change NumPy itself so that other operations are very different) because of the limitations of Python. The point was that it is too complicated to do any differently (and is probably impossible). -Travis From tim.hochberg at ieee.org Sat Jan 6 22:53:47 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Sat, 6 Jan 2007 20:53:47 -0700 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: <45A06A74.3090001@ee.byu.edu> References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> <45A06A74.3090001@ee.byu.edu> Message-ID: On 1/6/07, Travis Oliphant wrote: > > Timothy Hochberg wrote: > > > > > > > > On 1/6/07, *Robert Kern* < robert.kern at gmail.com > > > wrote: > > > > Sean R. Lynch wrote: > > > > >>>> x = zeros((3,)) > > >>>> x[array([0, 1, 1])] += array([1, 1, 1]) > > >>>> x > > > array([ 1., 1., 0.]) > > > > > > If this worked the way I was hoping, the output would be [1 2 0] > > because > > > it would add to element 1 twice due to its duplication in the > > advanced > > > selection array. > > > > > > Is the current behavior intentional or is it an accident of > > > implementation? > > > > It is an unavoidable consequence of the way Python interprets that > > code and the > > way numpy arrays are fundamentally implemented. See Travis > > Oliphant's, Tim > > Hochberg's and my posts in the thread "Histograms via indirect > > index arrays" for > > more details. > > > > > http://projects.scipy.org/pipermail/numpy-discussion/2006-March/thread.html#6877 > > > > > > Do we have to revisit that thread? I seem to recall it getting kind of > > cranky. To avoid reliving that, I will attempt dredge up the relevant > > issue: > > > > "a[indx]+=b" should be the same as "a[indx]=a[indx]+b". All else > > follow from that. If staring at that for a while doesn't enlighten > > you, then you will have to read that thread. > > > > [ I suspect that in theory we could make the += form behave as you > > expect, but that would break the identity above, which would confuse a > > bunch of people] > > > > I don't think we could make it work as he expects (and not radically > change NumPy itself so that other operations are very different) because > of the limitations of Python. > > The point was that it is too complicated to do any differently (and is > probably impossible). Right you are -- I was suffering a thinko -- sorry. -tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Sun Jan 7 01:08:27 2007 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Sun, 7 Jan 2007 01:08:27 -0500 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> Message-ID: On 06/01/07, Timothy Hochberg wrote: > Do we have to revisit that thread? I seem to recall it getting kind of > cranky. To avoid reliving that, I will attempt dredge up the relevant issue: > > "a[indx]+=b" should be the same as "a[indx]=a[indx]+b". All else follow from > that. If staring at that for a while doesn't enlighten you, then you will > have to read that thread. I'm afraid I dredged this issue up previously, and I have no desire to do so again, but this is not an identity: In [3]: a = arange(5) In [4]: a[1:] = a[1:] + a[:-1] In [5]: a Out[5]: array([0, 1, 3, 5, 7]) In [6]: a = arange(5) In [7]: a[1:] += a[:-1] In [8]: a Out[8]: array([ 0, 1, 3, 6, 10]) In fact the python documentation simply states that the two should be "nearly identical". However, a subtler restriction, explained to me by Robert Kern (http://aspn.activestate.com/ASPN/Mail/Message/scipy-user/3257570), means that it is in basically infeasible and inappropriate to make this do what I would have wanted it to. Nothing, of course, prevents anyone from implementing this in a function; in fact I think some of the histogram functions are capable of efficiently doing the most common things one might want this for. A. M. Archibald A. M. Archibald From charlesr.harris at gmail.com Sun Jan 7 03:16:36 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 7 Jan 2007 01:16:36 -0700 Subject: [Numpy-discussion] Exported symbols and code reorganization. Message-ID: Hi all, I've started some experiments at breaking up the large multiarraymodule.cand arrayobject.c files. As part of this I also looked at what symbols are currently exported by the *.so libraries. Most are just the usual three python symbols, but multiarray, mtrand, and fftpack_lite have extras. ./core/multiarray.so 000530b4 T _fini 0004d450 T _flat_copyinto 00005f58 T _init 000107e0 T count_new_axes_0d 0002c160 T initmultiarray I'm guessing the extras are unintended and should be made static. ./fft/fftpack_lite.so 00008284 T _fini 000008a4 T _init 00003570 T cfftb 00003530 T cfftf 00003800 T cffti 00001070 T fftpack_cfftb 00001220 T fftpack_cfftf 00000df0 T fftpack_rfftb 000013d0 T fftpack_rfftf 00000c30 T initfftpack_lite 00005e20 T rfftb 000039b0 T rfftb1 00006f30 T rfftf 00003680 T rffti Is there a good reason to make all those global? Is the library intended to be used as more than a module? ./random/mtrand.so 000294d4 T _fini 00002da4 T _init 00026ef0 T init_by_array 00023090 T initmtrand 00027020 T loggam 00026db0 T rk_altfill 00028360 T rk_beta 00028fc0 T rk_binomial 00028760 T rk_binomial_btpe 000279f0 T rk_binomial_inversion 000281d0 T rk_chisquare 00026d10 T rk_devfill 00026b40 T rk_double 000273a0 T rk_exponential 00028210 T rk_f 00026ba0 T rk_fill 00028440 T rk_gamma 00026c20 T rk_gauss 00027c70 T rk_geometric 00027bf0 T rk_geometric_inversion 00027180 T rk_geometric_search 00027240 T rk_gumbel 00029430 T rk_hypergeometric 00029090 T rk_hypergeometric_hrua 00027560 T rk_hypergeometric_hyp 00026ae0 T rk_interval 000272a0 T rk_laplace 000271f0 T rk_logistic 00028510 T rk_lognormal 00027470 T rk_logseries 00026ab0 T rk_long 00028480 T rk_negative_binomial 00028270 T rk_noncentral_chisquare 000282f0 T rk_noncentral_f 000284d0 T rk_normal 000273d0 T rk_pareto 00027970 T rk_poisson 00027410 T rk_poisson_mult 00027770 T rk_poisson_ptrs 00027dc0 T rk_power 00026970 T rk_random 00026e10 T rk_randomseed 00027700 T rk_rayleigh 000268f0 T rk_seed 00027f00 T rk_standard_cauchy 00027360 T rk_standard_exponential 00027f40 T rk_standard_gamma 00028120 T rk_standard_t 00027640 T rk_triangular 00026a80 T rk_ulong 00027320 T rk_uniform 00028550 T rk_vonmises 00027e50 T rk_wald 00027e10 T rk_weibull 00027cd0 T rk_zipf I'm guessing at least some of these should be private. The script that generates these lists is for i in $(find . -name "*.so") ; do echo $i && nm $i | grep \ T ; done ; executed in the package directory. That brings up the main question I have about how to break up the C files. I note that most of the functions in multiarraymodule.c, for instance, are part of the C-API, and are tagged as belonging to either the MULTIARRAY_API or the OBJECT_API. Apparently the build system scans for these tags and extracts the files somewhere. So, what is this API, is it available somewhere or is the code just copied somewhere convenient. As to breaking up the files, the scan only covers the code in the two current files, included code from broken out parts is not seen. This strikes me as a bit of a kludge, but I am sure there is a reason for it. Anyway, I assume the build system can be fixed, so that brings up the question of how to break up the files. The maximal strategy is to make every API functions, with it's helper functions, a separate file. This adds a *lot* of files, but it is straight forward and modular. A less drastic approach is to start by breaking multiarraymodule into four files: the converters, the two apis, and the module functions. My own preference is for the bunch of files, but I suspect some will object. Any comments? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Jan 7 03:47:39 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 07 Jan 2007 02:47:39 -0600 Subject: [Numpy-discussion] Exported symbols and code reorganization. In-Reply-To: References: Message-ID: <45A0B3AB.3080500@gmail.com> Charles R Harris wrote: > Hi all, > > I've started some experiments at breaking up the large > multiarraymodule.c and arrayobject.c files. As part of this I also > looked at what symbols are currently exported by the *.so libraries. > Most are just the usual three python symbols, but multiarray, mtrand, > and fftpack_lite have extras. > > ./core/multiarray.so > 000530b4 T _fini > 0004d450 T _flat_copyinto > 00005f58 T _init > 000107e0 T count_new_axes_0d > 0002c160 T initmultiarray > > I'm guessing the extras are unintended and should be made static. > > ./fft/fftpack_lite.so > 00008284 T _fini > 000008a4 T _init > 00003570 T cfftb > 00003530 T cfftf > 00003800 T cffti > 00001070 T fftpack_cfftb > 00001220 T fftpack_cfftf > 00000df0 T fftpack_rfftb > 000013d0 T fftpack_rfftf > 00000c30 T initfftpack_lite > 00005e20 T rfftb > 000039b0 T rfftb1 > 00006f30 T rfftf > 00003680 T rffti > > Is there a good reason to make all those global? Is the library intended > to be used as more than a module? No, but as with mtrand, most of those arise from the fact that these functions are implemented in files other than the C file that contains the Python module code. In order for them to be called from the Python module code, they need to be exported, IIRMCC (If I Remember My C Correctly). This appears to be true of essentially every other extension module in my site-packages, so I don't think it's going to be much of a problem. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Sun Jan 7 12:49:13 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Sun, 07 Jan 2007 09:49:13 -0800 Subject: [Numpy-discussion] Installing numpy from rpm In-Reply-To: <7deedc7e0701061555j785f5d6l1b2d58bde0753987@mail.gmail.com> References: <7deedc7e0701061555j785f5d6l1b2d58bde0753987@mail.gmail.com> Message-ID: <45A13299.3030707@noaa.gov> Satyendra Tomar wrote: > I just used "rpm -Uvh numpy-1.0.1-1.src.rpm" to install numpy from the > rpm package, running under SuSE 10.1, 2.6.16.21 , > python 2.4.2 , GCC 4.1.0 but python can't still find the numpy module. You've installed a SOURCE rpm -- what you need to do is build that rpm, then install the binary it produces: rpmbuild --rebuild numpy-1.0.1-1.src.rpm then look near the end of the output for where is put the resulting binary rpm. It can then be installed as you did above. Alternatively, find a pre-built binary (on without .src. in the name)for your system. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception From charlesr.harris at gmail.com Sun Jan 7 13:07:15 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 7 Jan 2007 11:07:15 -0700 Subject: [Numpy-discussion] Exported symbols and code reorganization. In-Reply-To: <45A0B3AB.3080500@gmail.com> References: <45A0B3AB.3080500@gmail.com> Message-ID: On 1/7/07, Robert Kern wrote: > > Charles R Harris wrote: No, but as with mtrand, most of those arise from the fact that these > functions > are implemented in files other than the C file that contains the Python > module > code. In order for them to be called from the Python module code, they > need to > be exported, IIRMCC (If I Remember My C Correctly). This appears to be > true of > essentially every other extension module in my site-packages, so I don't > think > it's going to be much of a problem. I don't think unnecessary public symbols are much of a problem, either, but it bears on the question of breaking up the core numpy files. The parts could be treated in the same way, i.e., compiled separately and linked, or they could be treated as you suggested previously, declared static and the code included into one base file. I suspect the inclusion route is a bit easier from a portability standpoint, long link lists can be a problem and the making of libraries is platform dependent. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomar.sk.ac at gmail.com Sun Jan 7 14:42:23 2007 From: tomar.sk.ac at gmail.com (Satyendra Tomar) Date: Sun, 7 Jan 2007 20:42:23 +0100 Subject: [Numpy-discussion] Installing numpy from rpm In-Reply-To: <45A13299.3030707@noaa.gov> References: <7deedc7e0701061555j785f5d6l1b2d58bde0753987@mail.gmail.com> <45A13299.3030707@noaa.gov> Message-ID: <7deedc7e0701071142i42071b8v9847068eaeceefb8@mail.gmail.com> Thank you for your help. I was successful with your belowmentioned suggestions. Satyendra You've installed a SOURCE rpm -- what you need to do is build that rpm, > then install the binary it produces: > > rpmbuild --rebuild numpy-1.0.1-1.src.rpm > > then look near the end of the output for where is put the resulting > binary rpm. It can then be installed as you did above. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From seanl at literati.org Sun Jan 7 16:14:41 2007 From: seanl at literati.org (Sean R. Lynch) Date: Sun, 07 Jan 2007 13:14:41 -0800 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: <45A06A74.3090001@ee.byu.edu> References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> <45A06A74.3090001@ee.byu.edu> Message-ID: <45A162C1.6020706@literati.org> Travis Oliphant wrote: > I don't think we could make it work as he expects (and not radically > change NumPy itself so that other operations are very different) because > of the limitations of Python. > > The point was that it is too complicated to do any differently (and is > probably impossible). Ok, thanks for the explanation. In that case, is there a way to do what I want (accumulate values per index) without looping in Python? The histogram functions would only count the number of uses of a given index AFAICT. I want to sum all of the normals for a given index. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature URL: From charlesr.harris at gmail.com Sun Jan 7 20:02:16 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 7 Jan 2007 18:02:16 -0700 Subject: [Numpy-discussion] Missing Py_DECREF? Message-ID: In the following lines from the initmultiarray function in multiarraymodule.c there seems to be a missing Py_DECREF MultiArrayError = PyString_FromString ("multiarray.error"); PyDict_SetItemString (d, "error", MultiArrayError); Should, I think, be followed by Py_DECREF(MultiArrayError); Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.leslie at gmail.com Sun Jan 7 22:09:21 2007 From: tim.leslie at gmail.com (Tim Leslie) Date: Mon, 8 Jan 2007 14:09:21 +1100 Subject: [Numpy-discussion] Ticket #132 Message-ID: Hi all, I've been looking through some open tickets and found ticket #132 which suggests a number of ways to improve numpy's web presence. http://projects.scipy.org/scipy/numpy/ticket/132 The only outstanding item on this is point #2, "Pointing www.numpy.org to numeric.scipy.org instead of the SF page". This seems like a good idea, but hasn't been done yet. Are there plans to implement this at some point? Cheers, Tim From tim.leslie at gmail.com Mon Jan 8 00:25:40 2007 From: tim.leslie at gmail.com (Tim Leslie) Date: Mon, 8 Jan 2007 16:25:40 +1100 Subject: [Numpy-discussion] segfault in numpy.float64(z) for complex z Message-ID: Hi All, While debugging a scipy crash I came across the problem outlined in http://projects.scipy.org/scipy/numpy/ticket/412 Could someone shed some light on why this is happening, I got a bit lost in the numpy internals when I tried to track it down. Cheers, Tim From bthom at cs.hmc.edu Mon Jan 8 01:04:01 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Sun, 07 Jan 2007 22:04:01 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <459F030E.8060001@gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> Message-ID: On Jan 5, 2007, at 6:01 PM, Robert Kern wrote: > belinda thom wrote: >> On Jan 5, 2007, at 5:32 PM, Sebastian Haase wrote: >> >>> Hi, >>> All I did is recompiling my (on source code file) C extension. I >>> made >>> sure that it was including the current numpy header files. >> >> Where are these files located? What command did you use? (The gorey >> details would help me quite a bit, as I'm to compile-level installs, >> having usually relied on things like macports port to do the work for >> me)... > > Actually, it's more likely to confuse the matter since Sebastian is > using a > quite nonstandard way of building Python extension modules. > > What packages do you want help with building from source? Which build > instructions have you read? What issues still remain unclear for > you after > having read them? What platforms are you concerned with? Robert, I am running on Macs, Intels, G4s, and G5s using OS X 10.4.8. I am using MacPython 2.4.4 (whose universal installer successfully installs on both types of machines). I might consider using 2.5, but a core of some externally developed coursework doesn't run in 2.5. Currently I have kluged some disparate parts together to get a working system (by which I mean interactive matplotlib, numpy, and scipy are all working and pass their respective tests [matplotlib doesn't appear to come with a test suite, so I couldn't test that one as extensively]). I understand that Erin Sheldon has had no problems using Fink to set up this integrated configuration, but I tend to avoid Fink b/c I've heard it doesn't always play nice w/other Mac internals. Besides, I like macports. More info on my kluges can be found at: http://www.mail-archive.com/ numpy-discussion at scipy.org/msg00481.html, where Chris had mentioned the need to provide an integrated set of Mac OS X bundle(s?) to do this job. I am not in a position to do this because I don't know how to install these packages from source. Others have told me only expert users tend to foray into such an area. At the same time, if I wish to have my students set up their computers and use this development environment, I'd like it to be as easy as possible to do. There's other reasons --- mostly curiousity driven --- that motivate me to try a "rawer" (e.g. source, non-pkg based) install: 1) My current kludge only supports TkAgg, so I can't play w/the wx backend at all. Have no idea why 2) My current kludge only works w/numpy; under Numeric, matplotlib plotting dies. 3) If I used the numpy/scipy bundles from http://www.macpython.org/, I'd get the C API error Sebastian was originally talking about. I have no idea how to fix this. I have just completed installing ipython from source by running: python setup.py install in a directory from which I downloaded ipython via subversion. That was pretty painless (this is my first such install of a relatively big program, BTW). If scipy, numpy and matplotlib could be just as painlessly derived, I'd give it a try. Thanks for any comments, --b From lists.steve at arachnedesign.net Mon Jan 8 01:28:05 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Mon, 8 Jan 2007 01:28:05 -0500 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> Message-ID: <8F59FA49-F3E4-4B4D-9F48-A8EE77B16AC4@arachnedesign.net> Hi, > I have just completed installing ipython from source by running: > > python setup.py install > > in a directory from which I downloaded ipython via subversion. That > was pretty painless (this is my first such install of a relatively > big program, BTW). If scipy, numpy and matplotlib could be just as > painlessly derived, I'd give it a try. Although I'm using a MacPorts build of Python2.4, I really don't think I did anything all that special to get numpy to install from source. It's as easy as your ipython install ... you just need to make sure you have the Apple dev tools installed (XCode -- it's on your OS X install discs and also downloadable from apple.com/ developer (free registration required)). XCode will provide you with the gcc compiler you need to compile numpy. Installing scipy from source I think is also similarly easy, except that you need to make sure you d/l the fortran compiler from http:// hpc.sourceforge.net/ first. I routinely svn up and compile both of these packages w/ no special maneuvering on my part ... The install instructions on the wiki did the trick for me, though it's down at the moment: http://scipy.org/Installing_SciPy I have it compiled and working on both an Intel MBP, and a G4 tower (both 32bit). My GTK and wx widgets, etc. are also all installed from macports, and my matplotlib backend is set to GTKAgg. I admittedly haven't followed this thread, so I don't know exactly what the hang up is, but hope that helps some. -steve From robert.kern at gmail.com Mon Jan 8 01:49:30 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 08 Jan 2007 00:49:30 -0600 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> Message-ID: <45A1E97A.1040507@gmail.com> belinda thom wrote: > There's other reasons --- mostly curiousity driven --- that motivate > me to try a "rawer" (e.g. source, non-pkg based) install: > > 1) My current kludge only supports TkAgg, so I can't play w/the wx > backend at all. Have no idea why > 2) My current kludge only works w/numpy; under Numeric, matplotlib > plotting dies. > 3) If I used the numpy/scipy bundles from http://www.macpython.org/, > I'd get the C API error Sebastian was originally talking about. I > have no idea how to fix this. The way to "fix" it is to use or build binaries of scipy and matplotlib that were compiled against the same version of numpy that you have installed. * Make sure that you have the latest version of XCode available for your system. I believe this is 2.4.1 right now. http://developer.apple.com/tools/download/ * Install a FORTRAN compiler suitable for your system. For a Universal build of Python, you will need gfortran. Get binaries from here: http://hpc.sourceforge.net/ For PowerPC: http://prdownloads.sourceforge.net/hpc/gfortran-bin.tar.gz?download For Intel: http://prdownloads.sourceforge.net/hpc/gfortran-intel-bin.tar.gz?download Install them according to the instructions on that page. * Get the source code for numpy. The latest release is 1.0.1 and would be a good bet. http://downloads.sourceforge.net/numpy/numpy-1.0.1.tar.gz * Unpack it and cd to the source directory in the Terminal. * Run the setup script to build and install numpy. $ python setup.py build $ python setup.py install Depending on how you've set up permissions inside the Python framework and wherever you've configured scripts to be installed to, you might need to issue the last command with root permissions using sudo. * Get the source code for scipy. The latest release is 0.5.2 and would be a good bet. http://prdownloads.sourceforge.net/scipy/scipy-0.5.2.tar.gz * Unpack it and cd to the source directory in the Terminal. * Run the setup script to build and install scipy. Be sure to use the following command line arguments to ensure that your FORTRAN compiler is picked up correctly. $ python setup.py build_src build_clib --fcompiler=gnu95 build_ext --fcompiler=gnu95 build $ python setup.py install As with numpy, the latter command might need to be run with root permissions. matplotlib is trickier because it has more external dependencies. I suggest you ask on the matplotlib list for more information, but this is what I do: * I install the appropriate wxPython binary. I'm currently using the wxPython 2.6 series since 2.8 is so new; I'm not sure if matplotlib works with it. Depending on your Python version, use one of these: http://downloads.sourceforge.net/wxpython/wxPython2.6-osx-unicode-2.6.3.3-universal10.4-py2.4.dmg http://downloads.sourceforge.net/wxpython/wxPython2.6-osx-unicode-2.6.3.3-universal10.4-py2.5.dmg * I use MacPorts to install the other dependencies. $ sudo port install libjpeg $ sudo port install libpng $ sudo port install libfreetype * Now, get the matplotlib source. The latest release is 0.87.7 and is a good bet. http://downloads.sourceforge.net/matplotlib/matplotlib-0.87.7.tar.gz * Unpack it and cd to the source directory in the Terminal. * In order to tell matplotlib where to find the MacPorts libraries, you need to edit setupext.py: Index: setupext.py =================================================================== --- setupext.py (revision 2861) +++ setupext.py (working copy) @@ -48,7 +48,7 @@ 'linux2' : ['/usr/local', '/usr',], 'linux' : ['/usr/local', '/usr',], 'cygwin' : ['/usr/local', '/usr',], - 'darwin' : ['/sw/lib/freetype2', '/sw/lib/freetype219', '/usr/local', + 'darwin' : ['/opt/local', '/usr/local', '/usr', '/sw'], 'freebsd4' : ['/usr/local', '/usr'], 'freebsd5' : ['/usr/local', '/usr'], * Now just build and install like any other Python package. $ python setup.py build $ python setup.py install -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bthom at cs.hmc.edu Mon Jan 8 01:58:15 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Sun, 07 Jan 2007 22:58:15 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <45A1E97A.1040507@gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> Message-ID: Robert, You are incredible! Triple Thanks! Will begin trying some of this stuff as soon as I've dealt w/hotter fires. --b On Jan 7, 2007, at 10:49 PM, Robert Kern wrote: > belinda thom wrote: > >> There's other reasons --- mostly curiousity driven --- that motivate >> me to try a "rawer" (e.g. source, non-pkg based) install: >> >> 1) My current kludge only supports TkAgg, so I can't play w/the wx >> backend at all. Have no idea why >> 2) My current kludge only works w/numpy; under Numeric, matplotlib >> plotting dies. >> 3) If I used the numpy/scipy bundles from http://www.macpython.org/, >> I'd get the C API error Sebastian was originally talking about. I >> have no idea how to fix this. > > The way to "fix" it is to use or build binaries of scipy and > matplotlib that > were compiled against the same version of numpy that you have > installed. > > * Make sure that you have the latest version of XCode available for > your system. > I believe this is 2.4.1 right now. > > http://developer.apple.com/tools/download/ > > * Install a FORTRAN compiler suitable for your system. For a > Universal build of > Python, you will need gfortran. Get binaries from here: > > http://hpc.sourceforge.net/ > > For PowerPC: > > http://prdownloads.sourceforge.net/hpc/gfortran-bin.tar.gz?download > > For Intel: > > http://prdownloads.sourceforge.net/hpc/gfortran-intel-bin.tar.gz? > download > > Install them according to the instructions on that page. > > * Get the source code for numpy. The latest release is 1.0.1 and > would be a good > bet. > > http://downloads.sourceforge.net/numpy/numpy-1.0.1.tar.gz > > * Unpack it and cd to the source directory in the Terminal. > > * Run the setup script to build and install numpy. > > $ python setup.py build > $ python setup.py install > > Depending on how you've set up permissions inside the Python > framework and > wherever you've configured scripts to be installed to, you might > need to issue > the last command with root permissions using sudo. > > * Get the source code for scipy. The latest release is 0.5.2 and > would be a good > bet. > > http://prdownloads.sourceforge.net/scipy/scipy-0.5.2.tar.gz > > * Unpack it and cd to the source directory in the Terminal. > > * Run the setup script to build and install scipy. Be sure to use > the following > command line arguments to ensure that your FORTRAN compiler is > picked up correctly. > > $ python setup.py build_src build_clib --fcompiler=gnu95 build_ext > --fcompiler=gnu95 build > $ python setup.py install > > As with numpy, the latter command might need to be run with root > permissions. > > > matplotlib is trickier because it has more external dependencies. I > suggest you > ask on the matplotlib list for more information, but this is what I > do: > > * I install the appropriate wxPython binary. I'm currently using > the wxPython > 2.6 series since 2.8 is so new; I'm not sure if matplotlib works > with it. > Depending on your Python version, use one of these: > > http://downloads.sourceforge.net/wxpython/wxPython2.6-osx- > unicode-2.6.3.3-universal10.4-py2.4.dmg > http://downloads.sourceforge.net/wxpython/wxPython2.6-osx- > unicode-2.6.3.3-universal10.4-py2.5.dmg > > * I use MacPorts to install the other dependencies. > > $ sudo port install libjpeg > $ sudo port install libpng > $ sudo port install libfreetype > > * Now, get the matplotlib source. The latest release is 0.87.7 and > is a good bet. > > http://downloads.sourceforge.net/matplotlib/matplotlib-0.87.7.tar.gz > > * Unpack it and cd to the source directory in the Terminal. > > * In order to tell matplotlib where to find the MacPorts libraries, > you need to > edit setupext.py: > > Index: setupext.py > =================================================================== > --- setupext.py (revision 2861) > +++ setupext.py (working copy) > @@ -48,7 +48,7 @@ > 'linux2' : ['/usr/local', '/usr',], > 'linux' : ['/usr/local', '/usr',], > 'cygwin' : ['/usr/local', '/usr',], > - 'darwin' : ['/sw/lib/freetype2', '/sw/lib/freetype219', '/usr/ > local', > + 'darwin' : ['/opt/local', '/usr/local', > '/usr', '/sw'], > 'freebsd4' : ['/usr/local', '/usr'], > 'freebsd5' : ['/usr/local', '/usr'], > > > * Now just build and install like any other Python package. > > $ python setup.py build > $ python setup.py install > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a > harmless enigma > that is made terrible by our own mad attempt to interpret it as > though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From erin.sheldon at gmail.com Mon Jan 8 03:38:19 2007 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Mon, 8 Jan 2007 00:38:19 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <45A1E97A.1040507@gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> Message-ID: <331116dc0701080038wf58be61xcced1cc757198d65@mail.gmail.com> On 1/7/07, Robert Kern wrote: > belinda thom wrote: > > > There's other reasons --- mostly curiousity driven --- that motivate > > me to try a "rawer" (e.g. source, non-pkg based) install: > > > > 1) My current kludge only supports TkAgg, so I can't play w/the wx > > backend at all. Have no idea why > > 2) My current kludge only works w/numpy; under Numeric, matplotlib > > plotting dies. > > 3) If I used the numpy/scipy bundles from http://www.macpython.org/, > > I'd get the C API error Sebastian was originally talking about. I > > have no idea how to fix this. > > The way to "fix" it is to use or build binaries of scipy and matplotlib that > were compiled against the same version of numpy that you have installed. > > * Make sure that you have the latest version of XCode available for your system. > I believe this is 2.4.1 right now. > > http://developer.apple.com/tools/download/ > > * Install a FORTRAN compiler suitable for your system. For a Universal build of > Python, you will need gfortran. Get binaries from here: > > http://hpc.sourceforge.net/ > > For PowerPC: > > http://prdownloads.sourceforge.net/hpc/gfortran-bin.tar.gz?download > > For Intel: > > http://prdownloads.sourceforge.net/hpc/gfortran-intel-bin.tar.gz?download > > Install them according to the instructions on that page. > > * Get the source code for numpy. The latest release is 1.0.1 and would be a good > bet. > > http://downloads.sourceforge.net/numpy/numpy-1.0.1.tar.gz > > * Unpack it and cd to the source directory in the Terminal. > > * Run the setup script to build and install numpy. > > $ python setup.py build > $ python setup.py install > > Depending on how you've set up permissions inside the Python framework and > wherever you've configured scripts to be installed to, you might need to issue > the last command with root permissions using sudo. > > * Get the source code for scipy. The latest release is 0.5.2 and would be a good > bet. > > http://prdownloads.sourceforge.net/scipy/scipy-0.5.2.tar.gz > > * Unpack it and cd to the source directory in the Terminal. > > * Run the setup script to build and install scipy. Be sure to use the following > command line arguments to ensure that your FORTRAN compiler is picked up correctly. > > $ python setup.py build_src build_clib --fcompiler=gnu95 build_ext > --fcompiler=gnu95 build > $ python setup.py install > > As with numpy, the latter command might need to be run with root permissions. > > > matplotlib is trickier because it has more external dependencies. I suggest you > ask on the matplotlib list for more information, but this is what I do: > > * I install the appropriate wxPython binary. I'm currently using the wxPython > 2.6 series since 2.8 is so new; I'm not sure if matplotlib works with it. > Depending on your Python version, use one of these: > > http://downloads.sourceforge.net/wxpython/wxPython2.6-osx-unicode-2.6.3.3-universal10.4-py2.4.dmg > http://downloads.sourceforge.net/wxpython/wxPython2.6-osx-unicode-2.6.3.3-universal10.4-py2.5.dmg > > * I use MacPorts to install the other dependencies. > > $ sudo port install libjpeg > $ sudo port install libpng > $ sudo port install libfreetype > > * Now, get the matplotlib source. The latest release is 0.87.7 and is a good bet. > > http://downloads.sourceforge.net/matplotlib/matplotlib-0.87.7.tar.gz > > * Unpack it and cd to the source directory in the Terminal. > > * In order to tell matplotlib where to find the MacPorts libraries, you need to > edit setupext.py: > > Index: setupext.py > =================================================================== > --- setupext.py (revision 2861) > +++ setupext.py (working copy) > @@ -48,7 +48,7 @@ > 'linux2' : ['/usr/local', '/usr',], > 'linux' : ['/usr/local', '/usr',], > 'cygwin' : ['/usr/local', '/usr',], > - 'darwin' : ['/sw/lib/freetype2', '/sw/lib/freetype219', '/usr/local', > + 'darwin' : ['/opt/local', '/usr/local', > '/usr', '/sw'], > 'freebsd4' : ['/usr/local', '/usr'], > 'freebsd5' : ['/usr/local', '/usr'], > > > * Now just build and install like any other Python package. > > $ python setup.py build > $ python setup.py install > I just got a new MacBook Pro, core 2 duo. There is no numpy/scipy fink distro for the intels yet so I tried out Robert's suggestions above. I installed python 2.5 as the base. I can confirm that all is working well for numpy and scipy. Matplotlib compiled, but gave some odd warnings such as /usr/bin/ld: warning fat file: /usr/local/lib/libgcc_s.10.4.dylib does not contain an architecture that matches the specified -arch flag: ppc (file ignored) And when I run a "import pylab" with either wxAgg or TkAgg as my backend it just segfaults. Any ideas? The only way I differed from Robert's suggestions was I used fink to install jpeg,png,freetype. fink install libjpeg fink install libpng3 fink install freetype2 And I did not alter the setupext.py file. Erin > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Mon Jan 8 03:51:24 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 08 Jan 2007 02:51:24 -0600 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <331116dc0701080038wf58be61xcced1cc757198d65@mail.gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> <331116dc0701080038wf58be61xcced1cc757198d65@mail.gmail.com> Message-ID: <45A2060C.8010909@gmail.com> Erin Sheldon wrote: > I just got a new MacBook Pro, core 2 duo. There is no > numpy/scipy fink distro for the intels yet so I tried out > Robert's suggestions above. I installed python 2.5 as > the base. I can confirm that all is working well for > numpy and scipy. > > Matplotlib compiled, but gave some odd warnings such > as > > /usr/bin/ld: warning fat file: /usr/local/lib/libgcc_s.10.4.dylib does > not contain an architecture that matches the specified -arch flag: ppc > (file ignored) Hmm. That is a bit odd. That library comes from the gfortran install and is a standard gcc runtime library. It's probably getting picked up for the extension modules that use -L/usr/local/lib . Not great, but shouldn't be harmful unless if you really want Universal binaries that will run on PPC Macs, too. > And when I run a "import pylab" with either wxAgg or TkAgg as > my backend it just segfaults. Any ideas? The > only way I differed from Robert's suggestions was I used fink > to install jpeg,png,freetype. > fink install libjpeg > fink install libpng3 > fink install freetype2 > And I did not alter the setupext.py file. I don't know. Try running it under gdb to get a C stacktrace so we can tell where the segfault comes from. The lines starting with (gdb) are prompts where you should enter the lines I give below: [~]$ gdb pythonw GNU gdb 6.3.50-20050815 (Apple version gdb-573) (Fri Oct 20 15:50:43 GMT 2006) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-apple-darwin"...Reading symbols for shared libraries .. done (gdb) run Starting program: /usr/local/bin/pythonw Reading symbols for shared libraries . done Program received signal SIGTRAP, Trace/breakpoint trap. 0x8fe01010 in __dyld__dyld_start () (gdb) continue Continuing. Reading symbols for shared libraries . done Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. Reading symbols for shared libraries .. done >>> from matplotlib.pylab import * Reading symbols for shared libraries .... done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries ........ done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries ........................................................... done Reading symbols for shared libraries .. done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries ... done Reading symbols for shared libraries . done Reading symbols for shared libraries .. done Reading symbols for shared libraries . done Reading symbols for shared libraries .................. done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries . done >>> -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at icps.u-strasbg.fr Mon Jan 8 04:09:59 2007 From: david at icps.u-strasbg.fr (R. David) Date: Mon, 08 Jan 2007 10:09:59 +0100 Subject: [Numpy-discussion] linalg.qr versus scilab qr Message-ID: <45A20A67.1060004@icps.u-strasbg.fr> Hello, In scilab, the QR decomposition can produce "sorted" Q and R matrices, where diagonal elements of R are decreasing. Thus, a permutation vector E is returned, so that X*E = Q*R. Is there a way to compute such a permutation QR in numpy ? It seems that numpy.linalg.qr does not sort the diagonal R elements. Do I have to wrap around lapack's zgeqpf or zgeqp3 ? Romaric From stefan at sun.ac.za Mon Jan 8 04:42:36 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 8 Jan 2007 11:42:36 +0200 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray In-Reply-To: References: Message-ID: <20070108094236.GE8292@mentat.za.net> On Fri, Jan 05, 2007 at 01:57:50PM -0800, Russell E Owen wrote: > I also checked the numpy 1.0.1 help and I confess I don't understand at > all what it claims to do if the new size is larger. It first says it > repeats a and then it says it zero-fills the output. > > >>> help(numpy.resize) > Help on function resize in module numpy.core.fromnumeric: > > resize(a, new_shape) > resize(a,new_shape) returns a new array with the specified shape. > The original array's total size can be any size. It > fills the new array with repeated copies of a. > > Note that a.resize(new_shape) will fill array with 0's > beyond current definition of a. The docstring refers to the difference between N.resize(x,6) and x.resize(6) Cheers St?fan From robert.kern at gmail.com Mon Jan 8 05:52:13 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 8 Jan 2007 04:52:13 -0600 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <45A1E97A.1040507@gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> Message-ID: <3d375d730701080252m35568b63q1bf758c590655594@mail.gmail.com> On 1/8/07, Robert Kern wrote: > * I use MacPorts to install the other dependencies. > > $ sudo port install libjpeg > $ sudo port install libpng > $ sudo port install libfreetype I beg your pardon, this is actually $ sudo port install jpeg $ sudo port install libpng $ sudo port install freetype -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Mon Jan 8 10:11:03 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 08 Jan 2007 08:11:03 -0700 Subject: [Numpy-discussion] segfault in numpy.float64(z) for complex z In-Reply-To: References: Message-ID: <45A25F07.2000708@ee.byu.edu> Tim Leslie wrote: >Hi All, > >While debugging a scipy crash I came across the problem outlined in > >http://projects.scipy.org/scipy/numpy/ticket/412 > >Could someone shed some light on why this is happening, I got a bit >lost in the numpy internals when I tried to track it down. > > > Recent changes to arrtype_new no doubt. If somebody with a non-SVN version of numpy could verify that would be great. I suspect an error condition is not being trapped correctly. -Travis From stefan at sun.ac.za Mon Jan 8 10:20:44 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 8 Jan 2007 17:20:44 +0200 Subject: [Numpy-discussion] segfault in numpy.float64(z) for complex z In-Reply-To: <45A25F07.2000708@ee.byu.edu> References: <45A25F07.2000708@ee.byu.edu> Message-ID: <20070108152044.GP8292@mentat.za.net> On Mon, Jan 08, 2007 at 08:11:03AM -0700, Travis Oliphant wrote: > Tim Leslie wrote: > > >Hi All, > > > >While debugging a scipy crash I came across the problem outlined in > > > >http://projects.scipy.org/scipy/numpy/ticket/412 > > > >Could someone shed some light on why this is happening, I got a bit > >lost in the numpy internals when I tried to track it down. > > > > > > > > Recent changes to arrtype_new no doubt. If somebody with a non-SVN > version of numpy could verify that would be great. How recent? This was broken in r2679 already. Cheers St?fan From lbolla at gmail.com Mon Jan 8 10:23:59 2007 From: lbolla at gmail.com (lorenzo bolla) Date: Mon, 8 Jan 2007 16:23:59 +0100 Subject: [Numpy-discussion] reduction Message-ID: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> Hello all! I'm fairly new to Numpy and, while experimenting, I found a strange (i.e. not expected by me!) behaviour of arrays. I tried this (in comment what I get): x = arange(4) # x = array([0,1,2,3]) def myadd(x,y): # re-define the binary sum function return x + y reduce(myadd, x) # 6, as expected add.reduce(x) # 6, as expected def mysub(x,y): # re-define the binary diff function return x - y reduce(mysub, x) # -6, as expected subtract.reduce(x) # 2 ---> WHY? Can someone explain me this? Thank you in advance! lorenzo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jan 8 11:25:06 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 8 Jan 2007 09:25:06 -0700 Subject: [Numpy-discussion] reduction In-Reply-To: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> Message-ID: On 1/8/07, lorenzo bolla wrote: > > Hello all! > I'm fairly new to Numpy and, while experimenting, I found a strange (i.e. > not expected by me!) behaviour of arrays. > I tried this (in comment what I get): > > x = arange(4) # x = array([0,1,2,3]) > > def myadd(x,y): # re-define the binary sum function > return x + y > > reduce(myadd, x) # 6, as expected > add.reduce(x) # 6, as expected > > def mysub(x,y): # re-define the binary diff function > return x - y > > reduce(mysub, x) # -6, as expected > subtract.reduce(x) # 2 ---> WHY? > It might be a bug in the implementation. What is happening is that instead of subtracting the new number from the previous result, the previous result is being subtracted from the new number. So you start with 0, and the sequence of operations continues: 0 = 0 - 0 1 = 1 - 0 1 = 2 - 1 2 = 3 - 1 2 = 4 - 2 3 = 5 - 2 Definitely a bug. Want to file a ticket? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jan 8 11:31:34 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 8 Jan 2007 09:31:34 -0700 Subject: [Numpy-discussion] reduction In-Reply-To: References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> Message-ID: On 1/8/07, Charles R Harris wrote: > > > > On 1/8/07, lorenzo bolla wrote: > > > > Hello all! > > I'm fairly new to Numpy and, while experimenting, I found a strange (i.e. > > not expected by me!) behaviour of arrays. > > I tried this (in comment what I get): > > > > x = arange(4) # x = array([0,1,2,3]) > > > > def myadd(x,y): # re-define the binary sum function > > return x + y > > > > reduce(myadd, x) # 6, as expected > > add.reduce(x) # 6, as expected > > > > def mysub(x,y): # re-define the binary diff function > > return x - y > > > > reduce(mysub, x) # -6, as expected > > subtract.reduce(x) # 2 ---> WHY? > > > > It might be a bug in the implementation. What is happening is that instead > of subtracting the new number from the previous result, the previous result > is being subtracted from the new number. So you start with 0, and the > sequence of operations continues: > > 0 = 0 - 0 > 1 = 1 - 0 > 1 = 2 - 1 > 2 = 3 - 1 > 2 = 4 - 2 > 3 = 5 - 2 > > Definitely a bug. Want to file a ticket? > Or maybe not a bug. It depends on what reduce means for this operation. So either a bug or something that could use a bit of documentation. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbolla at gmail.com Mon Jan 8 11:38:42 2007 From: lbolla at gmail.com (lorenzo bolla) Date: Mon, 8 Jan 2007 17:38:42 +0100 Subject: [Numpy-discussion] reduction In-Reply-To: References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> Message-ID: <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> Well, I don't know if we should consider it a bug, but it definetely behaves not as expected by the standard "reduce", right? I'm very happy to help: just tell me how to "file a ticket"! (what does it mean, by the way?). thanks! lorenzo. On 1/8/07, Charles R Harris wrote: > > > > On 1/8/07, Charles R Harris wrote: > > > > > > > > On 1/8/07, lorenzo bolla < lbolla at gmail.com> wrote: > > > > > > Hello all! > > > I'm fairly new to Numpy and, while experimenting, I found a strange ( > > > i.e. not expected by me!) behaviour of arrays. > > > I tried this (in comment what I get): > > > > > > x = arange(4) # x = array([0,1,2,3]) > > > > > > def myadd(x,y): # re-define the binary sum function > > > return x + y > > > > > > reduce(myadd, x) # 6, as expected > > > add.reduce(x) # 6, as expected > > > > > > def mysub(x,y): # re-define the binary diff function > > > return x - y > > > > > > reduce(mysub, x) # -6, as expected > > > subtract.reduce(x) # 2 ---> WHY? > > > > > > > It might be a bug in the implementation. What is happening is that > > instead of subtracting the new number from the previous result, the previous > > result is being subtracted from the new number. So you start with 0, and the > > sequence of operations continues: > > > > 0 = 0 - 0 > > 1 = 1 - 0 > > 1 = 2 - 1 > > 2 = 3 - 1 > > 2 = 4 - 2 > > 3 = 5 - 2 > > > > Definitely a bug. Want to file a ticket? > > > > Or maybe not a bug. It depends on what reduce means for this operation. So > either a bug or something that could use a bit of documentation. > > Chuck > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbolla at gmail.com Mon Jan 8 11:40:18 2007 From: lbolla at gmail.com (lorenzo bolla) Date: Mon, 8 Jan 2007 17:40:18 +0100 Subject: [Numpy-discussion] reduction In-Reply-To: References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> Message-ID: <80c99e790701080840r719f6597vbe67b5f0551dda5@mail.gmail.com> oh, I forgot. It happens with "divide", too. lorenzo. On 1/8/07, Charles R Harris wrote: > > > > On 1/8/07, Charles R Harris wrote: > > > > > > > > On 1/8/07, lorenzo bolla < lbolla at gmail.com> wrote: > > > > > > Hello all! > > > I'm fairly new to Numpy and, while experimenting, I found a strange ( > > > i.e. not expected by me!) behaviour of arrays. > > > I tried this (in comment what I get): > > > > > > x = arange(4) # x = array([0,1,2,3]) > > > > > > def myadd(x,y): # re-define the binary sum function > > > return x + y > > > > > > reduce(myadd, x) # 6, as expected > > > add.reduce(x) # 6, as expected > > > > > > def mysub(x,y): # re-define the binary diff function > > > return x - y > > > > > > reduce(mysub, x) # -6, as expected > > > subtract.reduce(x) # 2 ---> WHY? > > > > > > > It might be a bug in the implementation. What is happening is that > > instead of subtracting the new number from the previous result, the previous > > result is being subtracted from the new number. So you start with 0, and the > > sequence of operations continues: > > > > 0 = 0 - 0 > > 1 = 1 - 0 > > 1 = 2 - 1 > > 2 = 3 - 1 > > 2 = 4 - 2 > > 3 = 5 - 2 > > > > Definitely a bug. Want to file a ticket? > > > > Or maybe not a bug. It depends on what reduce means for this operation. So > either a bug or something that could use a bit of documentation. > > Chuck > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Jan 8 11:53:11 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 08 Jan 2007 10:53:11 -0600 Subject: [Numpy-discussion] reduction In-Reply-To: References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> Message-ID: <45A276F7.5010001@gmail.com> Charles R Harris wrote: > Or maybe not a bug. It depends on what reduce means for this operation. > So either a bug or something that could use a bit of documentation. Numeric gives the expected answer, so this is a bug in numpy. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lou_boog2000 at yahoo.com Mon Jan 8 11:57:52 2007 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Mon, 8 Jan 2007 08:57:52 -0800 (PST) Subject: [Numpy-discussion] reduction In-Reply-To: <80c99e790701080840r719f6597vbe67b5f0551dda5@mail.gmail.com> Message-ID: <766239.90585.qm@web34407.mail.mud.yahoo.com> Sounds like a problem when the operation is a noncommuntative one. Maybe that would narrow down the bug problem. I.e. Reduce expects A op B = B op A, where op is any operation Just a thought. --- lorenzo bolla wrote: > oh, I forgot. It happens with "divide", too. > lorenzo. -- Lou Pecora My views are my own. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From charlesr.harris at gmail.com Mon Jan 8 12:12:40 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 8 Jan 2007 10:12:40 -0700 Subject: [Numpy-discussion] reduction In-Reply-To: <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> Message-ID: On 1/8/07, lorenzo bolla wrote: > > Well, I don't know if we should consider it a bug, but it definetely > behaves not as expected by the standard "reduce", right? > I'm very happy to help: just tell me how to "file a ticket"! (what does it > mean, by the way?). > thanks! > Go to http://projects.scipy.org/scipy/numpy, on the top right of the page there will be a menu. You need to be registered to log in and file a new ticket, so you might need to do that first. After registration, log in and a menu item for a new ticket will show up. Click on it and continue. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Jan 8 12:20:21 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 08 Jan 2007 11:20:21 -0600 Subject: [Numpy-discussion] reduction In-Reply-To: <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> Message-ID: <45A27D55.7080507@gmail.com> lorenzo bolla wrote: > I'm very happy to help: just tell me how to "file a ticket"! (what does > it mean, by the way?). Don't worry. I took care of it. http://projects.scipy.org/scipy/numpy/ticket/413 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Jan 8 12:20:45 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 8 Jan 2007 10:20:45 -0700 Subject: [Numpy-discussion] reduction In-Reply-To: References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> Message-ID: On 1/8/07, Charles R Harris wrote: > > > > On 1/8/07, lorenzo bolla wrote: > > > > Well, I don't know if we should consider it a bug, but it definetely > > behaves not as expected by the standard "reduce", right? > > I'm very happy to help: just tell me how to "file a ticket"! (what does > > it mean, by the way?). > > thanks! > > > > Go to http://projects.scipy.org/scipy/numpy, on the top right of the page > there will be a menu. You need to be registered to log in and file a new > ticket, so you might need to do that first. After registration, log in and a > menu item for a new ticket will show up. Click on it and continue. > Never mind, Robert already filed ticket # 413. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbolla at gmail.com Mon Jan 8 12:32:34 2007 From: lbolla at gmail.com (lorenzo bolla) Date: Mon, 8 Jan 2007 18:32:34 +0100 Subject: [Numpy-discussion] reduction In-Reply-To: References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> Message-ID: <80c99e790701080932i7b489bfblcc06964162449e8@mail.gmail.com> ops. I did it, too. should I delete it?? how?? thanks! On 1/8/07, Charles R Harris wrote: > > > > On 1/8/07, Charles R Harris wrote: > > > > > > > > On 1/8/07, lorenzo bolla < lbolla at gmail.com> wrote: > > > > > > Well, I don't know if we should consider it a bug, but it definetely > > > behaves not as expected by the standard "reduce", right? > > > I'm very happy to help: just tell me how to "file a ticket"! (what > > > does it mean, by the way?). > > > thanks! > > > > > > > Go to http://projects.scipy.org/scipy/numpy, on the top right of the > > page there will be a menu. You need to be registered to log in and file a > > new ticket, so you might need to do that first. After registration, log in > > and a menu item for a new ticket will show up. Click on it and continue. > > > > Never mind, Robert already filed ticket # 413. > > Chuck > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Jan 8 12:41:46 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 08 Jan 2007 11:41:46 -0600 Subject: [Numpy-discussion] reduction In-Reply-To: <80c99e790701080932i7b489bfblcc06964162449e8@mail.gmail.com> References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> <80c99e790701080932i7b489bfblcc06964162449e8@mail.gmail.com> Message-ID: <45A2825A.5090001@gmail.com> lorenzo bolla wrote: > ops. I did it, too. > should I delete it?? how?? I closed it as a duplicate. As a nondeveloper, you don't have the permissions to close tickets yourself. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.hochberg at ieee.org Mon Jan 8 12:49:46 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Mon, 8 Jan 2007 10:49:46 -0700 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: <45A06A74.3090001@ee.byu.edu> References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> <45A06A74.3090001@ee.byu.edu> Message-ID: On 1/6/07, Travis Oliphant wrote: > > Timothy Hochberg wrote: > > > > > > > > On 1/6/07, *Robert Kern* < robert.kern at gmail.com > > > wrote: > > > > Sean R. Lynch wrote: > > > > >>>> x = zeros((3,)) > > >>>> x[array([0, 1, 1])] += array([1, 1, 1]) > > >>>> x > > > array([ 1., 1., 0.]) > > > > > > If this worked the way I was hoping, the output would be [1 2 0] > > because > > > it would add to element 1 twice due to its duplication in the > > advanced > > > selection array. > > > > > > Is the current behavior intentional or is it an accident of > > > implementation? > > > > It is an unavoidable consequence of the way Python interprets that > > code and the > > way numpy arrays are fundamentally implemented. See Travis > > Oliphant's, Tim > > Hochberg's and my posts in the thread "Histograms via indirect > > index arrays" for > > more details. > > > > > http://projects.scipy.org/pipermail/numpy-discussion/2006-March/thread.html#6877 > > > > > > Do we have to revisit that thread? I seem to recall it getting kind of > > cranky. To avoid reliving that, I will attempt dredge up the relevant > > issue: > > > > "a[indx]+=b" should be the same as "a[indx]=a[indx]+b". All else > > follow from that. If staring at that for a while doesn't enlighten > > you, then you will have to read that thread. > > > > [ I suspect that in theory we could make the += form behave as you > > expect, but that would break the identity above, which would confuse a > > bunch of people] > > > > I don't think we could make it work as he expects (and not radically > change NumPy itself so that other operations are very different) because > of the limitations of Python. > > The point was that it is too complicated to do any differently (and is > probably impossible). Oh, you really shouldn't use words like "impossible" unless you want someone to figure out how to do it. Some people view that as a challenge ;-) It appears like it is possible, although it requires a certain amount dubious hackery. Basically, you need a hidden tag on arrays that keeps track of whether that array was the result of an inplace op and what kind. This gets combined with a little sys.getrefcount magic to determine if the object in question is a temporary. Below is a sketch of how it probably could be accomplished. Not that I necessarily think it's a good idea, particularly since the implementation is so dicey. Oh, it's quite possible there are some corner and not so corner cases that leak through too. import numpy as np import sys class BecauseImContrary(np.ndarray): """Super hacktastic way to get repeated values to work >>> a = np.arange(5).view(BecauseImContrary) >>> indices = [0, 1, 0, 2] # The basic test case, does repeated adds work. >>> a1 = a.copy() >>> a1[indices] += 1 >>> print a1 [2 2 3 3 4] # Now some tests to make sure it doesn't add things when it shouldn't >>> a1 = a.copy() >>> a1[indices] = a1[indices] + 1 >>> print a1 [1 2 3 3 4] >>> def nested(x): ... y = np.ones_like(x) ... y += 998 ... return y >>> hasattr(nested(a1), '_iop') False >>> a[indices] = a[indices] >>> print a [0 1 2 3 4] >>> a1 = a.copy() >>> a1 += 1 >>> print a1 [1 2 3 4 5] """ def __iadd__(self, other): result = np.ndarray.__iadd__(self, other) count = sys.getrefcount(self) assert count >= 6 if count == 6: result._iop = 'add' return result def __setitem__(self, key, value): count = sys.getrefcount(value) assert count >= 5 # Only works for 1D indices, cause I'm lazy. if count == 5 and hasattr(value, '_iop') and isinstance(key, ( np.ndarray, list)): key = np.asarray(key) for i in key: self[i] = 0 value = np.ones_like(key) * value for i, x in zip(key, value): self[i] += x else: np.ndarray.__setitem__(self, key, value) if __name__ == '__main__': import doctest, scratch doctest.testmod(scratch) -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Jan 8 13:04:22 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 08 Jan 2007 11:04:22 -0700 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> <45A06A74.3090001@ee.byu.edu> Message-ID: <45A287A6.2010503@ee.byu.edu> Timothy Hochberg wrote: > > > On 1/6/07, *Travis Oliphant* > wrote: > > Timothy Hochberg wrote: > > > > > > > > On 1/6/07, *Robert Kern* < robert.kern at gmail.com > > > >> > wrote: > > > > Sean R. Lynch wrote: > > > > >>>> x = zeros((3,)) > > >>>> x[array([0, 1, 1])] += array([1, 1, 1]) > > >>>> x > > > array([ 1., 1., 0.]) > > > > > > If this worked the way I was hoping, the output would be > [1 2 0] > > because > > > it would add to element 1 twice due to its duplication in the > > advanced > > > selection array. > > > > > > Is the current behavior intentional or is it an accident of > > > implementation? > > > > It is an unavoidable consequence of the way Python > interprets that > > code and the > > way numpy arrays are fundamentally implemented. See Travis > > Oliphant's, Tim > > Hochberg's and my posts in the thread "Histograms via indirect > > index arrays" for > > more details. > > > > > http://projects.scipy.org/pipermail/numpy-discussion/2006-March/thread.html#6877 > > > > > > > Do we have to revisit that thread? I seem to recall it getting > kind of > > cranky. To avoid reliving that, I will attempt dredge up the > relevant > > issue: > > > > "a[indx]+=b" should be the same as "a[indx]=a[indx]+b". All else > > follow from that. If staring at that for a while doesn't enlighten > > you, then you will have to read that thread. > > > > [ I suspect that in theory we could make the += form behave as you > > expect, but that would break the identity above, which would > confuse a > > bunch of people] > > > > I don't think we could make it work as he expects (and not radically > change NumPy itself so that other operations are very different) > because > of the limitations of Python. > > The point was that it is too complicated to do any differently (and is > probably impossible). > > > Oh, you really shouldn't use words like "impossible" unless you want > someone to figure out > how to do it. Some people view that as a challenge ;-) I know, that's why I put it in parenthesis and said "probably" ;-) -Travis From seb.haase at gmx.net Mon Jan 8 13:15:11 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Mon, 8 Jan 2007 10:15:11 -0800 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray In-Reply-To: <20070108094236.GE8292@mentat.za.net> References: <20070108094236.GE8292@mentat.za.net> Message-ID: On 1/8/07, Stefan van der Walt wrote: > On Fri, Jan 05, 2007 at 01:57:50PM -0800, Russell E Owen wrote: > > I also checked the numpy 1.0.1 help and I confess I don't understand at > > all what it claims to do if the new size is larger. It first says it > > repeats a and then it says it zero-fills the output. > > > > >>> help(numpy.resize) > > Help on function resize in module numpy.core.fromnumeric: > > > > resize(a, new_shape) > > resize(a,new_shape) returns a new array with the specified shape. > > The original array's total size can be any size. It > > fills the new array with repeated copies of a. > > > > Note that a.resize(new_shape) will fill array with 0's > > beyond current definition of a. > > The docstring refers to the difference between > > N.resize(x,6) > > and > > x.resize(6) > Hi St?fan, Why is there a needed for this very confusing dualty !? I would almost like to file a bug report on this ! (It definitily broke "backwards compatibility" for my code coming from numarray ) -Sebastian From oliphant at ee.byu.edu Mon Jan 8 13:37:03 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 08 Jan 2007 11:37:03 -0700 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray In-Reply-To: References: <20070108094236.GE8292@mentat.za.net> Message-ID: <45A28F4F.2070508@ee.byu.edu> Sebastian Haase wrote: >On 1/8/07, Stefan van der Walt wrote: > > >Hi St?fan, > >Why is there a needed for this very confusing dualty !? >I would almost like to file a bug report on this ! > >(It definitily broke "backwards compatibility" for my code coming from >numarray ) > > It's a remnant of trying to merge all the packages (done before the idea to have separate compatibility layers emerged). This one in particular is done because resize is used in this way at least once in the library code and so has a permanance it otherwise would not have enjoyed. I'm not sure we can change it now. At least not until 1.1 -Travis From seb.haase at gmx.net Mon Jan 8 13:52:58 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Mon, 8 Jan 2007 10:52:58 -0800 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray In-Reply-To: <45A28F4F.2070508@ee.byu.edu> References: <20070108094236.GE8292@mentat.za.net> <45A28F4F.2070508@ee.byu.edu> Message-ID: On 1/8/07, Travis Oliphant wrote: > Sebastian Haase wrote: > > >On 1/8/07, Stefan van der Walt wrote: > > > > > >Hi St?fan, > > > >Why is there a needed for this very confusing dualty !? > >I would almost like to file a bug report on this ! > > > >(It definitily broke "backwards compatibility" for my code coming from > >numarray ) > > > > > It's a remnant of trying to merge all the packages (done before the idea > to have separate compatibility layers emerged). > > This one in particular is done because resize is used in this way at > least once in the library code and so has a permanance it otherwise > would not have enjoyed. > > I'm not sure we can change it now. At least not until 1.1 > > -Travis I would suggest treating this as a real bug! Then it could be fixed immediately. I don't think that many people are relying on this behaviour, and if so: it is just so confusion that my guess is that they would be always using EITHER the function OR the method - and run into this "bug" as soon as they accidentally use the other one. Bug-fixes don't have to keep backwards compatibility. -Sebastian From kwgoodman at gmail.com Mon Jan 8 14:14:21 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 8 Jan 2007 11:14:21 -0800 Subject: [Numpy-discussion] Indexing with weave Message-ID: My first weave attempt weaves something to be desired. test() works, but test2() doesn't. It complains about the line "y = x(0,i);" In function 'PyObject* compiled_func(PyObject*, PyObject*)': blah.cpp:667: error: no match for call to '(py::object) (int, int&)' I am no match for that error message. What am I doing wrong? from scipy import weave from scipy.weave import converters import numpy.matlib as M def test(i): x = M.matrix([[1.0, 2.0, 3.0]]); code = """ return_val = x(0,i); """ y = weave.inline(code, ['x', 'i'], type_converters=converters.blitz) return y def test2(i): x = M.matrix([[1.0, 2.0, 3.0]]); code = """ double y; y = x(0,i); return_val = y; """ y = weave.inline(code, ['x', 'i'], type_converters=converters.blitz) return y From robert.kern at gmail.com Mon Jan 8 14:40:30 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 08 Jan 2007 13:40:30 -0600 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray In-Reply-To: References: <20070108094236.GE8292@mentat.za.net> <45A28F4F.2070508@ee.byu.edu> Message-ID: <45A29E2E.6030804@gmail.com> Sebastian Haase wrote: > I would suggest treating this as a real bug! > Then it could be fixed immediately. Deliberate design decisions don't turn into bugs just because you disagree with them. Neither do those where the original decider now disagrees with them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From seb.haase at gmx.net Mon Jan 8 15:24:01 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Mon, 8 Jan 2007 12:24:01 -0800 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray In-Reply-To: <45A29E2E.6030804@gmail.com> References: <20070108094236.GE8292@mentat.za.net> <45A28F4F.2070508@ee.byu.edu> <45A29E2E.6030804@gmail.com> Message-ID: On 1/8/07, Robert Kern wrote: > Sebastian Haase wrote: > > > I would suggest treating this as a real bug! > > Then it could be fixed immediately. > > Deliberate design decisions don't turn into bugs just because you disagree with > them. Neither do those where the original decider now disagrees with them. > Please explain again what the original decision was based on. I remember that there was an effort at some point to make methods as functions more consistent. -Sebastian From aisaac at american.edu Mon Jan 8 15:32:56 2007 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 8 Jan 2007 15:32:56 -0500 Subject: [Numpy-discussion] functions vs. methods In-Reply-To: References: <20070108094236.GE8292@mentat.za.net><45A28F4F.2070508@ee.byu.edu><45A29E2E.6030804@gmail.com> Message-ID: On Mon, 8 Jan 2007, Sebastian Haase apparently wrote: > Please explain again what the original decision was based > on. I think the real questions are: what do the numpy developers want in the future, and what is the right path from here to there? > I remember that there was an effort at some point to make > methods as functions more consistent. Yes, but my (user's) memory is that there were a couple exceptions for historical reasons. As a user now of essentially only numpy, I too would like to see those exceptions go away, both for personal convenience, and for the benefit of new users. Cheers, Alan Isaac From oliphant at ee.byu.edu Mon Jan 8 15:33:50 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 08 Jan 2007 13:33:50 -0700 Subject: [Numpy-discussion] functions vs. methods In-Reply-To: References: <20070108094236.GE8292@mentat.za.net><45A28F4F.2070508@ee.byu.edu><45A29E2E.6030804@gmail.com> Message-ID: <45A2AAAE.2090100@ee.byu.edu> Alan G Isaac wrote: >On Mon, 8 Jan 2007, Sebastian Haase apparently wrote: > > >>Please explain again what the original decision was based >>on. >> >> > >I think the real questions are: >what do the numpy developers want in the future, >and what is the right path from here to there? > > > >>I remember that there was an effort at some point to make >>methods as functions more consistent. >> >> > >Yes, but my (user's) memory is that there were a couple >exceptions for historical reasons. As a user now of >essentially only numpy, I too would like to see those >exceptions go away, both for personal convenience, and for >the benefit of new users. > > > I think we all want this to happen. I think we may have to wait until 1.1, however, to fix this issue because it is not technically a bug. It might be a good time to create the 1.0 branch and make the trunk 1.1 -Travis From robert.kern at gmail.com Mon Jan 8 15:37:02 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 08 Jan 2007 14:37:02 -0600 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray In-Reply-To: References: <20070108094236.GE8292@mentat.za.net> <45A28F4F.2070508@ee.byu.edu> <45A29E2E.6030804@gmail.com> Message-ID: <45A2AB6E.2090806@gmail.com> Sebastian Haase wrote: > On 1/8/07, Robert Kern wrote: >> Sebastian Haase wrote: >> >>> I would suggest treating this as a real bug! >>> Then it could be fixed immediately. >> Deliberate design decisions don't turn into bugs just because you disagree with >> them. Neither do those where the original decider now disagrees with them. >> > Please explain again what the original decision was based on. > I remember that there was an effort at some point to make methods as > functions more consistent. Travis just gave you an explanation. Yes there was quite a bit of effort to rationalize the methods and the functions, but that happened before 1.0 was released. We now have a commitment to maintain API compatibility for the 1.0 series. That commitment doesn't go away because it is inconvenient for us or because we missed an opportunity to rationalize the API more. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.hochberg at ieee.org Mon Jan 8 16:08:37 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Mon, 8 Jan 2007 14:08:37 -0700 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: <45A162C1.6020706@literati.org> References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> <45A06A74.3090001@ee.byu.edu> <45A162C1.6020706@literati.org> Message-ID: On 1/7/07, Sean R. Lynch wrote: > > Travis Oliphant wrote: > > > I don't think we could make it work as he expects (and not radically > > change NumPy itself so that other operations are very different) because > > of the limitations of Python. > > > > The point was that it is too complicated to do any differently (and is > > probably impossible). > > Ok, thanks for the explanation. > > In that case, is there a way to do what I want (accumulate values per > index) without looping in Python? The histogram functions would only > count the number of uses of a given index AFAICT. I want to sum all of > the normals for a given index. Sean, I'm pretty sure that at one point I had a way to do exactly what you are doing, however it's been a while and I don't know where that code wandered off to. I will think about it now that I'm done doing silly stuff and see if I can recall what the trick was. [Just don't want you to feel abandoned...] -tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Mon Jan 8 16:12:04 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Mon, 8 Jan 2007 23:12:04 +0200 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray In-Reply-To: References: <20070108094236.GE8292@mentat.za.net> <45A28F4F.2070508@ee.byu.edu> <45A29E2E.6030804@gmail.com> Message-ID: <20070108211203.GA5878@mentat.za.net> On Mon, Jan 08, 2007 at 12:24:01PM -0800, Sebastian Haase wrote: > On 1/8/07, Robert Kern wrote: > > Sebastian Haase wrote: > > > > > I would suggest treating this as a real bug! > > > Then it could be fixed immediately. > > > > Deliberate design decisions don't turn into bugs just because you disagree with > > them. Neither do those where the original decider now disagrees with them. > > > Please explain again what the original decision was based on. > I remember that there was an effort at some point to make methods as > functions more consistent. In this case, I tend to agree that the behaviour is unexpected. In some cases, like with sort, however, I think the difference in behaviour is useful: In [1]: import numpy as N In [2]: x = N.array([3,2,1,2]) In [3]: N.sort(x) # Don't tell Guido! Out[3]: array([1, 2, 2, 3]) In [4]: x Out[4]: array([3, 2, 1, 2]) In [5]: x.sort() In [6]: x Out[6]: array([1, 2, 2, 3]) Cheers St?fan From aisaac at american.edu Mon Jan 8 16:32:27 2007 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 8 Jan 2007 16:32:27 -0500 Subject: [Numpy-discussion] discussion about array.resize() -- compare to numarray In-Reply-To: <20070108211203.GA5878@mentat.za.net> References: <20070108094236.GE8292@mentat.za.net><45A28F4F.2070508@ee.byu.edu><45A29E2E.6030804@gmail.com> <20070108211203.GA5878@mentat.za.net> Message-ID: On Mon, 8 Jan 2007, Stefan van der Walt apparently wrote: > I tend to agree that the behaviour is unexpected. In some > cases, like with sort, however, I think the difference in > behaviour is useful: I'm going to speculate everyone agrees that the numpy 'sort' function is is useful. Perhaps the question is whether is should be renamed to 'sorted'. (No opinion.) Cheers, Alan Isaac From seb.haase at gmx.net Mon Jan 8 16:44:26 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Mon, 8 Jan 2007 13:44:26 -0800 Subject: [Numpy-discussion] functions vs. methods In-Reply-To: <45A2AAAE.2090100@ee.byu.edu> References: <20070108094236.GE8292@mentat.za.net> <45A28F4F.2070508@ee.byu.edu> <45A29E2E.6030804@gmail.com> <45A2AAAE.2090100@ee.byu.edu> Message-ID: On 1/8/07, Travis Oliphant wrote: > Alan G Isaac wrote: > > >On Mon, 8 Jan 2007, Sebastian Haase apparently wrote: > > > > > >>Please explain again what the original decision was based > >>on. > >> > >> > > > >I think the real questions are: > >what do the numpy developers want in the future, > >and what is the right path from here to there? > > > > > > > >>I remember that there was an effort at some point to make > >>methods as functions more consistent. > >> > >> > > > >Yes, but my (user's) memory is that there were a couple > >exceptions for historical reasons. As a user now of > >essentially only numpy, I too would like to see those > >exceptions go away, both for personal convenience, and for > >the benefit of new users. Could anyone provide a complete list of the "couple exceptions for historical reasons..." mentioned by Alan !? -Sebastian. From barrywark at gmail.com Mon Jan 8 16:58:26 2007 From: barrywark at gmail.com (Barry Wark) Date: Mon, 8 Jan 2007 13:58:26 -0800 Subject: [Numpy-discussion] Indexing with weave In-Reply-To: References: Message-ID: Keith, I think you want y = x[0,i] Remember that indexing in numpy/scipy is the python way (using []), not the matlab way (using () )... I've been bitten by the distinction many times. Barry On 1/8/07, Keith Goodman wrote: > My first weave attempt weaves something to be desired. > > test() works, but test2() doesn't. > > It complains about the line "y = x(0,i);" > > In function 'PyObject* compiled_func(PyObject*, PyObject*)': > blah.cpp:667: error: no match for call to '(py::object) (int, int&)' > > I am no match for that error message. What am I doing wrong? > > from scipy import weave > from scipy.weave import converters > import numpy.matlib as M > > def test(i): > x = M.matrix([[1.0, 2.0, 3.0]]); > code = """ > return_val = x(0,i); > """ > y = weave.inline(code, ['x', 'i'], type_converters=converters.blitz) > return y > > def test2(i): > x = M.matrix([[1.0, 2.0, 3.0]]); > code = """ > double y; > y = x(0,i); > return_val = y; > """ > y = weave.inline(code, ['x', 'i'], type_converters=converters.blitz) > return y > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From lou_boog2000 at yahoo.com Mon Jan 8 17:24:03 2007 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Mon, 8 Jan 2007 14:24:03 -0800 (PST) Subject: [Numpy-discussion] Where's append in NumPy?? In-Reply-To: <20070108211203.GA5878@mentat.za.net> Message-ID: <552033.74858.qm@web34412.mail.mud.yahoo.com> Where is append in NumPy? I see it in the numpy manual (I paid for and have the latest version), but when I invoke it, Python complains that it doesn't exist. In iPython a query like append? gives 'append" not found (after importing numpy). Other numpy functions are there (e.g. nansum on same page in the book). What am I missing? -- Lou Pecora My views are my own. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From kwgoodman at gmail.com Mon Jan 8 17:26:02 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 8 Jan 2007 14:26:02 -0800 Subject: [Numpy-discussion] Indexing with weave In-Reply-To: References: Message-ID: On 1/8/07, Barry Wark wrote: > I think you want > y = x[0,i] I followed the weave.inline example at http://www.scipy.org/PerformancePython which uses the C-like x(i,j) instead of the numpy x[i,j]. Changing test2 to y = x[0,i] gives "TypeError: cannot convert value to double" on the weave.inline line. > On 1/8/07, Keith Goodman wrote: > > My first weave attempt weaves something to be desired. > > > > test() works, but test2() doesn't. > > > > It complains about the line "y = x(0,i);" > > > > In function 'PyObject* compiled_func(PyObject*, PyObject*)': > > blah.cpp:667: error: no match for call to '(py::object) (int, int&)' > > > > I am no match for that error message. What am I doing wrong? > > > > from scipy import weave > > from scipy.weave import converters > > import numpy.matlib as M > > > > def test(i): > > x = M.matrix([[1.0, 2.0, 3.0]]); > > code = """ > > return_val = x(0,i); > > """ > > y = weave.inline(code, ['x', 'i'], type_converters=converters.blitz) > > return y > > > > def test2(i): > > x = M.matrix([[1.0, 2.0, 3.0]]); > > code = """ > > double y; > > y = x(0,i); > > return_val = y; > > """ > > y = weave.inline(code, ['x', 'i'], type_converters=converters.blitz) > > return y From fperez.net at gmail.com Mon Jan 8 17:27:06 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 8 Jan 2007 15:27:06 -0700 Subject: [Numpy-discussion] Where's append in NumPy?? In-Reply-To: <552033.74858.qm@web34412.mail.mud.yahoo.com> References: <20070108211203.GA5878@mentat.za.net> <552033.74858.qm@web34412.mail.mud.yahoo.com> Message-ID: On 1/8/07, Lou Pecora wrote: > Where is append in NumPy? I see it in the numpy > manual (I paid for and have the latest version), but > when I invoke it, Python complains that it doesn't > exist. In iPython a query like append? gives > 'append" not found (after importing numpy). Other > numpy functions are there (e.g. nansum on same page in > the book). What am I missing? In [1]: import numpy as N In [2]: N.*app*? N.append N.apply_along_axis N.apply_over_axes In [3]: N.append? Type: function Base Class: Namespace: Interactive File: /home/fperez/tmp/local/lib/python2.4/site-packages/numpy/lib/function_base.py Definition: N.append(arr, values, axis=None) Docstring: Append to the end of an array along axis (ravel first if None) Cheers, f From robert.kern at gmail.com Mon Jan 8 17:29:59 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 08 Jan 2007 16:29:59 -0600 Subject: [Numpy-discussion] Where's append in NumPy?? In-Reply-To: <552033.74858.qm@web34412.mail.mud.yahoo.com> References: <552033.74858.qm@web34412.mail.mud.yahoo.com> Message-ID: <45A2C5E7.8000909@gmail.com> Lou Pecora wrote: > Where is append in NumPy? I see it in the numpy > manual (I paid for and have the latest version), but > when I invoke it, Python complains that it doesn't > exist. In iPython a query like append? gives > 'append" not found (after importing numpy). Other > numpy functions are there (e.g. nansum on same page in > the book). What am I missing? What version of numpy are you using? They've been there since at least 1.0b4. In [1]: import numpy In [2]: numpy.app numpy.append numpy.apply_along_axis numpy.apply_over_axes In [2]: numpy.append? Type: function Base Class: Namespace: Interactive File: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy-1.0.2.dev3493-py2.5-macosx-10.4-i386.egg/numpy/lib/function_base.py Definition: numpy.append(arr, values, axis=None) Docstring: Append to the end of an array along axis (ravel first if None) In [3]: numpy.nansum? Type: function Base Class: Namespace: Interactive File: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy-1.0.2.dev3493-py2.5-macosx-10.4-i386.egg/numpy/lib/function_base.py Definition: numpy.nansum(a, axis=None) Docstring: Sum the array over the given axis, treating NaNs as 0. In [4]: print numpy.__version__ 1.0.2.dev3493 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.leslie at gmail.com Mon Jan 8 18:38:06 2007 From: tim.leslie at gmail.com (Tim Leslie) Date: Tue, 9 Jan 2007 10:38:06 +1100 Subject: [Numpy-discussion] segfault in numpy.float64(z) for complex z In-Reply-To: <20070108152044.GP8292@mentat.za.net> References: <45A25F07.2000708@ee.byu.edu> <20070108152044.GP8292@mentat.za.net> Message-ID: On 1/9/07, Stefan van der Walt wrote: > On Mon, Jan 08, 2007 at 08:11:03AM -0700, Travis Oliphant wrote: > > Tim Leslie wrote: > > > > >Hi All, > > > > > >While debugging a scipy crash I came across the problem outlined in > > > > > >http://projects.scipy.org/scipy/numpy/ticket/412 > > > > > >Could someone shed some light on why this is happening, I got a bit > > >lost in the numpy internals when I tried to track it down. > > > > > > > > > > > > > Recent changes to arrtype_new no doubt. If somebody with a non-SVN > > version of numpy could verify that would be great. > > How recent? This was broken in r2679 already. It looks like it's due to changes in r3493 http://projects.scipy.org/scipy/numpy/changeset/3493 The line at which it barfs is: if (robj->ob_type == type) return robj; which is a new piece of code from this changeset. Tim > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From stefan at sun.ac.za Mon Jan 8 18:50:24 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 9 Jan 2007 01:50:24 +0200 Subject: [Numpy-discussion] segfault in numpy.float64(z) for complex z In-Reply-To: References: <45A25F07.2000708@ee.byu.edu> <20070108152044.GP8292@mentat.za.net> Message-ID: <20070108235024.GA16625@mentat.za.net> On Tue, Jan 09, 2007 at 10:38:06AM +1100, Tim Leslie wrote: > On 1/9/07, Stefan van der Walt wrote: > > On Mon, Jan 08, 2007 at 08:11:03AM -0700, Travis Oliphant wrote: > > > Tim Leslie wrote: > > > > > > >Hi All, > > > > > > > >While debugging a scipy crash I came across the problem outlined in > > > > > > > >http://projects.scipy.org/scipy/numpy/ticket/412 > > > > > > > >Could someone shed some light on why this is happening, I got a bit > > > >lost in the numpy internals when I tried to track it down. > > > > > > > > > > > > > > > > > > Recent changes to arrtype_new no doubt. If somebody with a non-SVN > > > version of numpy could verify that would be great. > > > > How recent? This was broken in r2679 already. > > It looks like it's due to changes in r3493 > > http://projects.scipy.org/scipy/numpy/changeset/3493 > > The line at which it barfs is: > > if (robj->ob_type == type) return robj; > > which is a new piece of code from this changeset. You're right -- it wasn't broken in r2679 -- I had a newer version of multiarray lying around. You may proceed to kick me in the head now. Cheers St?fan From lou_boog2000 at yahoo.com Mon Jan 8 19:09:38 2007 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Mon, 8 Jan 2007 16:09:38 -0800 (PST) Subject: [Numpy-discussion] Where's append in NumPy?? In-Reply-To: <45A2C5E7.8000909@gmail.com> Message-ID: <408807.70331.qm@web34406.mail.mud.yahoo.com> After import numpy as N In [10]: print N.__version__ 1.1.2881 does that look right as a recent version? I still get In [2]: N.append? Object `N.append` not found. -- Lou Pecora My views are my own. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From robert.kern at gmail.com Mon Jan 8 19:12:57 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 08 Jan 2007 18:12:57 -0600 Subject: [Numpy-discussion] Where's append in NumPy?? In-Reply-To: <408807.70331.qm@web34406.mail.mud.yahoo.com> References: <408807.70331.qm@web34406.mail.mud.yahoo.com> Message-ID: <45A2DE09.7070807@gmail.com> Lou Pecora wrote: > After import numpy as N > > In [10]: print N.__version__ > 1.1.2881 > > does that look right as a recent version? No, that's very old. The version number had briefly gotten bumped to 1.1 in the repository, but we backed that out quickly. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From barrywark at gmail.com Mon Jan 8 20:09:11 2007 From: barrywark at gmail.com (Barry Wark) Date: Mon, 8 Jan 2007 17:09:11 -0800 Subject: [Numpy-discussion] Indexing with weave In-Reply-To: References: Message-ID: oops. and it works in test(). sorry. my eyes aren't what they used to be :) On 1/8/07, Keith Goodman wrote: > On 1/8/07, Barry Wark wrote: > > I think you want > > y = x[0,i] > > I followed the weave.inline example at > > http://www.scipy.org/PerformancePython > > which uses the C-like x(i,j) instead of the numpy x[i,j]. > > Changing test2 to y = x[0,i] gives "TypeError: cannot convert value to > double" on the weave.inline line. > > > On 1/8/07, Keith Goodman wrote: > > > My first weave attempt weaves something to be desired. > > > > > > test() works, but test2() doesn't. > > > > > > It complains about the line "y = x(0,i);" > > > > > > In function 'PyObject* compiled_func(PyObject*, PyObject*)': > > > blah.cpp:667: error: no match for call to '(py::object) (int, int&)' > > > > > > I am no match for that error message. What am I doing wrong? > > > > > > from scipy import weave > > > from scipy.weave import converters > > > import numpy.matlib as M > > > > > > def test(i): > > > x = M.matrix([[1.0, 2.0, 3.0]]); > > > code = """ > > > return_val = x(0,i); > > > """ > > > y = weave.inline(code, ['x', 'i'], type_converters=converters.blitz) > > > return y > > > > > > def test2(i): > > > x = M.matrix([[1.0, 2.0, 3.0]]); > > > code = """ > > > double y; > > > y = x(0,i); > > > return_val = y; > > > """ > > > y = weave.inline(code, ['x', 'i'], type_converters=converters.blitz) > > > return y > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From jdawe at eos.ubc.ca Mon Jan 8 20:12:04 2007 From: jdawe at eos.ubc.ca (Jordan Dawe) Date: Mon, 08 Jan 2007 17:12:04 -0800 Subject: [Numpy-discussion] Getting 3-D array values from a 2-D array of indexes Message-ID: <45A2EBE4.1070706@eos.ubc.ca> I have a 3D array A of shape (nz, ny, nx) and a 2D array B of shape (ny, nx) that contains integer indexes. I want to generate a 2D array C of shape (ny, nx) composed of the values of A at the z-indexes specified in B. Is there an easy way to do this? Jordan From tim.leslie at gmail.com Mon Jan 8 21:55:06 2007 From: tim.leslie at gmail.com (Tim Leslie) Date: Tue, 9 Jan 2007 13:55:06 +1100 Subject: [Numpy-discussion] asarray() behaviour. Message-ID: I get different behaviour from asarray depending on whether I pass in a list or an array, Additionally, neither of them give me the expected results, which is a TypeError complaining about being unable to use complex values as floats. Can someone shed some light on this. Cheers, Tim In [27]: N.asarray([1j,2,3], dtype=N.float64) --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/timl/src/scipy/Lib/linsolve/ /usr/lib/python2.4/site-packages/numpy/core/numeric.py in asarray(a, dtype, order) 130 are converted to base class ndarray. 131 """ --> 132 return array(a, dtype, copy=False, order=order) 133 134 def asanyarray(a, dtype=None, order=None): ValueError: setting an array element with a sequence. In [28]: N.asarray(N.array([1j,2,3]), dtype=N.float64) Out[28]: array([ 0., 2., 3.]) From charlesr.harris at gmail.com Mon Jan 8 22:17:19 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 8 Jan 2007 20:17:19 -0700 Subject: [Numpy-discussion] asarray() behaviour. In-Reply-To: References: Message-ID: On 1/8/07, Tim Leslie wrote: > > I get different behaviour from asarray depending on whether I pass in > a list or an array, Additionally, neither of them give me the expected > results, which is a TypeError complaining about being unable to use > complex values as floats. Can someone shed some light on this. As to the type, the numbers are automatically cast without warning. It is useful to be able to force this, and the easiest is by default without raising an error. It is also the Numarray way when using just the array constructor. The cast for complex numbers works the same way as the corresponding .astype() method. I suppose it might make sense to raise a warning when converting complex to real. Anyway, the following is expected. In [11]: a = arange(3).astype(double) In [12]: b = asarray(a, dtype=float32) In [13]: a = a + 1j In [14]: b = asarray(a, dtype=float32) Note that the cast to single precision from double also loses precision without warning. Given that, the fact that In [16]: b = array([1j, 2, 3], dtype=float32) raises an error should probably be considered a bug due to 1j not being recognized as a number. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.leslie at gmail.com Mon Jan 8 22:31:10 2007 From: tim.leslie at gmail.com (Tim Leslie) Date: Tue, 9 Jan 2007 14:31:10 +1100 Subject: [Numpy-discussion] More complex issues Message-ID: Hi all, Sorry for the bad pun, but I'm getting myself confused trying to understand the expected behaviour of how complex values are handled throughout numpy. My current confusion is to do with the difference between how and are handled. If someone could explain this difference that would be great. Cheers, Tim In [45]: z = N.complex128(1 + 2j) In [46]: N.float64(z) Out[46]: 1.0 In [47]: N.float64(1 + 2j) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/timl/src/scipy/Lib/linsolve/ TypeError: can't convert complex to float; use abs(z) From tim.hochberg at ieee.org Mon Jan 8 22:57:32 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Mon, 8 Jan 2007 20:57:32 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <45A0630F.9050804@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> <459D766C.40505@ee.byu.edu> <459D8886.2000101@noaa.gov> <459D8B72.2040309@ee.byu.edu> <459D9590.80202@noaa.gov> <459D9D9C.30501@ee.byu.edu> <459EC58E.7000408@noaa.gov> <459F2039.30407@ieee.org> <45A0630F.9050804@ee.byu.edu> Message-ID: On 1/6/07, Travis Oliphant wrote: > > Tim Hochberg wrote: > > Christopher Barker wrote: > > > > [SNIP] > > > >> I think the PEP has far more chances of success if it's seen as a > >> request from a variety of package developers, not just the numpy crowd > >> (which, after all, already has numpy > >> > > This seems eminently sensible. Getting a few developers from other > > projects on board would help a lot; it might also reveal some > > deficiencies to the proposal that we don't see yet. > > > It would help quite a bit. Are there any suggestions of who to recruit > to review the proposal? Before I can answer that, I need to ask you a question. How do you see this extension to the buffer protocol? Do you see it as an supplement to the earlier array protocol, or do you see it as a replacement? The reason that I ask is that the two projects that I use regularly are wxPython and PIL generally operate on relatively large data chunks and it's not clear that they would see much benefit over this mechanism versus the array protocol. I imagine that between us Chris Barker and I could hack together something for wxPython (not that I've asked him aout it). And code would probably go a long way to convincing people what a great idea this is. However, all else being equal, it'd be a lot easier to do this for the array protocol since there's no extra infrastructure involved. [SNIP] > 1. Why do we need Py_ARRAYOF? Can't we get the same effect just > > using longer shape and strides arrays? > > > Yes, this is true for a single data-format in isolation (and in fact > exactly what you get when you instantiate in NumPy a data-type that is > an array of another primitive data-type). However, how do you describe > a structure whose second field is an array of a primitive type? This is > where the ARRAYOF qualifier is needed. In NumPy, actually, it's not > done this way, but a separate subarray field in the data-type object is > used. After studying c-types, however, I think this approach is better. OK,. Needed for recursive data structures, check. > 2. Is there any type besides Py_STRUCTURE that can have names > > and fields. If so, what and what do they mean. If not, you > > should just say that. > > > Yes, you can add fields to a multi-byte primitive if you want. This > would be similar to thinking about the data-format as a C-like union. > Perhaps the data-field has meaning as a 4-byte integer but the > most-significant and least-significant bytes should also be addressable > individually. Hmm. I think I understand this somewhat better now, but I can't decide if it's cool or overkill. Is this a supporting a feature that ctypes has? > 3. And on this topic, why a tuple of ([names,..], {field})? Why > > not simply a list of (name, dfobject, offset, meta) for > > example? And what's the meta information if it's not PyNone? > > Just a string? Anything at all? > > > > The list of names is useful for having an ordered list so you can > traverse the structure in field order. It is technically not necessary > but it makes it a lot easier to parse a data-format object in offset > order (it is used a bit in NumPy, for example). Right, I got that. Between names and field you are simulating an ordered dict. What I still don't understand is why you chose to simulate this ordered dict using a list plus a dictionary rather than a list of tuples. This may well just be a matter of taste. However, for the small sizes I'd expect of these lists I would expect a list of of tuples would perform better than the dictionary solution. The meta information is a place holder for field tags and future growth > (kind of like column headers in a spreadsheet). It started as a place > to put a "longer" name or to pass along information about a field (like > units) through. > > OK. FWIW, the array protocol PEP seems more relevant to what I do since I'm not concerned so much with the overhead since I'm sending big chunks of data back and forth. -tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From bthom at cs.hmc.edu Tue Jan 9 00:34:08 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Mon, 08 Jan 2007 21:34:08 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 Message-ID: Thanks for the Intel update. I'm doing the same for my PPC (Powerbook, G4, OS X 10.4.8), and have some failures in scipy's tests. Did you try doing: >>> import numpy as N >>> N.test() and >>> import scipy as S >>> S.test()? For me, the latter one fails (both install and S.test() I/O appended). > I just got a new MacBook Pro, core 2 duo. There is no > numpy/scipy fink distro for the intels yet so I tried out > Robert's suggestions above. I installed python 2.5 as > the base. I can confirm that all is working well for > numpy and scipy. > > Matplotlib compiled, but gave some odd warnings such > as > Perhaps I should attempt to install matplot lib at this point---I don't know. Any idea if the FAILUREs I'm seeing are a problem? Thx, --b Below, I paste the output from python setup.py install (for scipy) as well as the command I ran inside of python to check scipy's functionality. bthom at tweedledee.local [5:51pm] ~/Download/scipy-0.5.2 4 % sudo python setup.py install Password: mkl_info: libraries mkl,vml,guide not found in /Library/Frameworks/ Python.framework/Versions/2.4/lib libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib libraries mkl,vml,guide not found in /opt/local/lib NOT AVAILABLE fftw3_info: libraries fftw3 not found in /Library/Frameworks/Python.framework/ Versions/2.4/lib libraries fftw3 not found in /usr/local/lib libraries fftw3 not found in /usr/lib libraries fftw3 not found in /opt/local/lib fftw3 not found NOT AVAILABLE fftw2_info: libraries rfftw,fftw not found in /Library/Frameworks/ Python.framework/Versions/2.4/lib libraries rfftw,fftw not found in /usr/local/lib libraries rfftw,fftw not found in /usr/lib libraries rfftw,fftw not found in /opt/local/lib fftw2 not found NOT AVAILABLE dfftw_info: libraries drfftw,dfftw not found in /Library/Frameworks/ Python.framework/Versions/2.4/lib libraries drfftw,dfftw not found in /usr/local/lib libraries drfftw,dfftw not found in /usr/lib libraries drfftw,dfftw not found in /opt/local/lib dfftw not found NOT AVAILABLE djbfft_info: NOT AVAILABLE blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/ vecLib.framework/Headers'] lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec'] non-existing path in 'Lib/linsolve': 'tests' umfpack_info: libraries umfpack not found in /Library/Frameworks/ Python.framework/Versions/2.4/lib libraries umfpack not found in /usr/local/lib libraries umfpack not found in /usr/lib libraries umfpack not found in /opt/local/lib /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/numpy/distutils/system_info.py:401: UserWarning: UMFPACK sparse solver (http://www.cise.ufl.edu/research/sparse/ umfpack/) not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [umfpack]) or by setting the UMFPACK environment variable. warnings.warn(self.notfounderror.__doc__) NOT AVAILABLE non-existing path in 'Lib/maxentropy': 'doc' Warning: Subpackage 'Lib' configuration returned as 'scipy' running install running build running config_fc running build_src building py_modules sources building library "dfftpack" sources building library "linpack_lite" sources building library "specfun" sources building library "statlib" sources building extension "scipy.cluster._vq" sources building extension "scipy.fftpack._fftpack" sources f2py options: [] adding 'build/src.macosx-10.3-fat-2.4/fortranobject.c' to sources. adding 'build/src.macosx-10.3-fat-2.4' to include_dirs. building extension "scipy.fftpack.convolve" sources f2py options: [] adding 'build/src.macosx-10.3-fat-2.4/fortranobject.c' to sources. adding 'build/src.macosx-10.3-fat-2.4' to include_dirs. building extension "scipy.integrate._quadpack" sources building extension "scipy.integrate._odepack" sources building extension "scipy.special._cephes" sources building extension "scipy.special.specfun" sources f2py options: ['--no-wrap-functions'] adding 'build/src.macosx-10.3-fat-2.4/fortranobject.c' to sources. adding 'build/src.macosx-10.3-fat-2.4' to include_dirs. building extension "scipy.stats.statlib" sources f2py options: ['--no-wrap-functions'] adding 'build/src.macosx-10.3-fat-2.4/fortranobject.c' to sources. adding 'build/src.macosx-10.3-fat-2.4' to include_dirs. building extension "scipy.stats.futil" sources f2py options: [] adding 'build/src.macosx-10.3-fat-2.4/fortranobject.c' to sources. adding 'build/src.macosx-10.3-fat-2.4' to include_dirs. building extension "scipy.stats.mvn" sources f2py options: [] adding 'build/src.macosx-10.3-fat-2.4/fortranobject.c' to sources. adding 'build/src.macosx-10.3-fat-2.4' to include_dirs. adding 'build/src.macosx-10.3-fat-2.4/Lib/stats/mvn- f2pywrappers.f' to sources. building extension "scipy.ndimage._nd_image" sources building data_files sources running build_py copying build/src.macosx-10.3-fat-2.4/scipy/__config__.py -> build/ lib.macosx-10.3-fat-2.4/scipy running build_clib customize UnixCCompiler customize UnixCCompiler using build_clib customize NAGFCompiler customize AbsoftFCompiler customize IbmFCompiler Could not locate executable g77 Could not locate executable f77 Could not locate executable f95 customize GnuFCompiler customize Gnu95FCompiler customize Gnu95FCompiler customize Gnu95FCompiler using build_clib running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext customize NAGFCompiler customize AbsoftFCompiler customize IbmFCompiler customize GnuFCompiler customize Gnu95FCompiler customize Gnu95FCompiler customize Gnu95FCompiler using build_ext running install_lib creating /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy creating /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/lib/blas copying build/lib.macosx-10.3-fat-2.4/scipy/lib/blas/__init__.py -> / Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/scipy/lib/blas copying build/lib.macosx-10.3-fat-2.4/scipy/lib/blas/cblas.so -> / Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/scipy/lib/blas /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/scipy/weave copying build/lib.macosx-10.3-fat-2.4/scipy/weave/wx_spec.py -> / Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/scipy/weave byte-compiling /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/__config__.py to __config__.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/__init__.py to __init__.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/__svn_version__.py to byte-compiling /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/weave/slice_handler.py to slice_handler.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/weave/standard_array_spec.py to standard_array_spec.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/weave/swig2_spec.py to swig2_spec.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/weave/swigptr.py to swigptr.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/weave/swigptr2.py to swigptr2.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/weave/vtk_spec.py to vtk_spec.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/weave/weave_version.py to weave_version.pyc byte-compiling /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/weave/wx_spec.py to wx_spec.pyc running install_data creating /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/sparse/tests copying Lib/sparse/tests/test_sparse.orig -> /Library/Frameworks/ Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy/ sparse/tests/ creating /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/maxentropy/examples copying Lib/maxentropy/examples/bergerexamplesimulated.py -> /Library/ Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/ scipy/maxentropy/examples/ copying Lib/maxentropy/examples/conditionalexample1.py -> /Library/ Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/ scipy/maxentropy/examples/ copying Lib/maxentropy/examples/conditionalexample2.py -> /Library/ Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/ scipy/maxentropy/examples/ copying Lib/maxentropy/examples/bergerexample.py -> /Library/ Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/ scipy/maxentropy/examples/ copying ./TOCHANGE.txt -> /Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/scipy copying ./DEVELOPERS.txt -> /Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/scipy copying ./THANKS.txt -> /Library/Frameworks/Python.framework/Versions/ 2.4/lib/python2.4/site-packages/scipy copying ./LICENSE.txt -> /Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/scipy copying ./LATEST.txt -> /Library/Frameworks/Python.framework/Versions/ 2.4/lib/python2.4/site-packages/scipy copying ./FORMAT_GUIDELINES.txt -> /Library/Frameworks/ Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy copying ./INSTALL.txt -> /Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/scipy creating /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/linalg/tests copying Lib/linalg/tests/test_blas.py -> /Library/Frameworks/ Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy/ linalg/tests/ site-packages/scipy/fftpack/tests/ creating /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/weave/doc copying Lib/weave/doc/tutorial.html -> /Library/Frameworks/ Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy/weave/ doc/ bthom at tweedledee.local [5:52pm] ~/Download/scipy-0.5.2 5 % python Python 2.4.4 (#1, Oct 18 2006, 10:34:39) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import scipy as S >>> S.test() Found 4 tests for scipy.io.array_import Found 1 tests for scipy.cluster.vq Found 128 tests for scipy.linalg.fblas Found 18 tests for scipy.fftpack.basic Found 4 tests for scipy.io.recaster Warning: FAILURE importing tests for /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/scipy/linsolve/umfpack/tests/test_umfpack.py:17: AttributeError: 'module' object has no attribute 'umfpack' (in ?) Found 4 tests for scipy.optimize.zeros Found 28 tests for scipy.io.mio Found 41 tests for scipy.linalg.basic Found 2 tests for scipy.maxentropy.maxentropy Found 358 tests for scipy.special.basic Found 128 tests for scipy.lib.blas.fblas Found 7 tests for scipy.linalg.matfuncs **************************************************************** WARNING: clapack module is empty ----------- See scipy/INSTALL.txt for troubleshooting. Notes: * If atlas library is not found by numpy/distutils/system_info.py, then scipy uses flapack instead of clapack. **************************************************************** Found 42 tests for scipy.lib.lapack Warning: FAILURE importing tests for /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/scipy/linsolve/umfpack/tests/test_umfpack.py:17: AttributeError: 'module' object has no attribute 'umfpack' (in ?) Found 1 tests for scipy.optimize.cobyla Found 16 tests for scipy.lib.blas Found 1 tests for scipy.integrate Found 14 tests for scipy.linalg.blas Found 4 tests for scipy.fftpack.helper Found 4 tests for scipy.signal.signaltools Found 0 tests for __main__ Don't worry about a warning regarding the number of bytes read. Warning: 1000000 bytes requested, 20 bytes read. .....Use minimum degree ordering on A'+A. ....................................../Library/Frameworks/ Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy/ interpolate/fitpack2.py:457: UserWarning: The coefficients of the spline returned have been computed as the minimal norm least-squares solution of a (numerically) rank deficient system (deficiency=7). If deficiency is large, the results may be inaccurate. Deficiency may strongly depend on the value of eps. warnings.warn(message) ........................................................................ ........................Ties preclude use of exact statistic. ..Ties preclude use of exact statistic. ........ **************************************************************** WARNING: clapack module is empty ----------- See scipy/INSTALL.txt for troubleshooting. Notes: * If atlas library is not found by numpy/distutils/system_info.py, then scipy uses flapack instead of clapack. **************************************************************** .......................F................................................ ............................................................ ...Result may be inaccurate, approximate err = 1.6582899877e-08 ...Result may be inaccurate, approximate err = 4.54747350886e-13 ......................................................F.......Residual: 1.05006926991e-07 . **************************************************************** WARNING: cblas module is empty ----------- See scipy/INSTALL.txt for troubleshooting. Notes: * If atlas library is not found by numpy/distutils/system_info.py, then scipy uses fblas instead of cblas. **************************************************************** .......F.............. ====================================================================== FAIL: test_smallest_same_kind (scipy.io.tests.test_recaster.test_recaster) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/io/tests/test_recaster.py", line 61, in test_smallest_same_kind assert C is not None, 'Got unexpected None from %s' % T AssertionError: Got unexpected None from ====================================================================== FAIL: check_dot (scipy.lib.tests.test_blas.test_fblas1_simple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/lib/blas/tests/test_blas.py", line 76, in check_dot assert_almost_equal(f([3j,-4,3-4j],[2,3,1]),-9+2j) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/testing/utils.py", line 156, in assert_almost_equal assert round(abs(desired - actual),decimal) == 0, msg AssertionError: Items are not equal: ACTUAL: (-1.9987130165100098+4.5784586609566519e-37j) DESIRED: (-9+2j) ====================================================================== FAIL: check_dot (scipy.linalg.tests.test_blas.test_fblas1_simple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/linalg/tests/test_blas.py", line 75, in check_dot assert_almost_equal(f([3j,-4,3-4j],[2,3,1]),-9+2j) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy/testing/utils.py", line 156, in assert_almost_equal assert round(abs(desired - actual),decimal) == 0, msg AssertionError: Items are not equal: ACTUAL: (-1.9987130165100098+4.5784443116603772e-37j) DESIRED: (-9+2j) ---------------------------------------------------------------------- Ran 1596 tests in 14.230s FAILED (failures=3) >>> From robert.kern at gmail.com Tue Jan 9 01:05:10 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 09 Jan 2007 00:05:10 -0600 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: References: Message-ID: <45A33096.8040507@gmail.com> belinda thom wrote: > Perhaps I should attempt to install matplot lib at this point---I > don't know. Any idea if the FAILUREs I'm seeing are a problem? No, they're not. One has been fixed in SVN, and the other two are known bugs probably related to gfortran. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Tue Jan 9 01:10:34 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 09 Jan 2007 00:10:34 -0600 Subject: [Numpy-discussion] Note to developers: Pyrex, Python 2.5, and exceptions Message-ID: <45A331DA.7080607@gmail.com> The official distribution of Pyrex has not been completely updated for Python 2.5. The change of the base Exception to new-style classes causes errors in some of the code that Pyrex generates to handle exceptions. I've been using the branch of Pyrex that the lxml project maintains which includes a fix for this: http://codespeak.net/svn/lxml/pyrex/ When regenerating mtrand.c from mtrand.pyx, please use this version instead of the official version. Thank you. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Tue Jan 9 02:00:31 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 09 Jan 2007 00:00:31 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> <459D766C.40505@ee.byu.edu> <459D8886.2000101@noaa.gov> <459D8B72.2040309@ee.byu.edu> <459D9590.80202@noaa.gov> <459D9D9C.30501@ee.byu.edu> <459EC58E.7000408@noaa.gov> <459F2039.30407@ieee.org> <45A0630F.9050804@ee.byu.edu> Message-ID: <45A33D8F.7000509@ee.byu.edu> Timothy Hochberg wrote: > > > On 1/6/07, *Travis Oliphant* > wrote: > > Tim Hochberg wrote: > > Christopher Barker wrote: > > > > [SNIP] > > > >> I think the PEP has far more chances of success if it's seen as a > >> request from a variety of package developers, not just the > numpy crowd > >> (which, after all, already has numpy > >> > > This seems eminently sensible. Getting a few developers from other > > projects on board would help a lot; it might also reveal some > > deficiencies to the proposal that we don't see yet. > > > It would help quite a bit. Are there any suggestions of who to > recruit > to review the proposal? > > > Before I can answer that, I need to ask you a question. How do you see > this extension to the buffer protocol? Do you see it as an supplement > to the earlier array protocol, or do you see it as a replacement? This is a replacement to the previously described array protocol PEP. This is how I'm trying to get the array protocol into Python. In that vein, it has two purposes: One is to make a better buffer protocol that includes a conception of an N-dimensional array in Python itself. If we can include this in Python then we get a lot of mileage out of all the people that write extension modules for Python that should really be making their memory available as an N-dimensional array (everytime I turn around there is a new wrapping to some library that is *not* using NumPy as the underlying extension). With the existence of ctypes it just starts to get worse as nobody thinks about exposing things as arrays anymore and so NumPy users don't get the ease of use we would get if the N-dimensional array concept were a part of Python itself. For example, I just found the FreeImage project which wraps a nice library using ctypes. But, it doesn't have a way to expose these images as numpy arrays. Now, it would probably take me only a few hours to make the connection between FreeImage and NumPy, but I'd like to see the day when it happens without me (or some other NumPy expert) having to do all the work. If ctypes objects exposed the extended buffer protocol for appropriate types, then I wouldn't have to do anything. Because the wrapped structures would be exposable as arrays and all of a sudden I say a = array(freeimobj) and I can do math on the array in Python. Or if I'm an extension module writer, I don't need to have NumPy (or rely on it) in order to do some computation on freeimobj in C itself. Sure, you can do it now (if the array protocol is followed --- but not many people have adopted it yet --- some have argued that it's "not in Python itself"). So, I guess, the big reason I'm pushing this is largely marketing. The buffer protcol is the "right" place to but the array protocol. The second reason is to ensure that the buffer protocol itself doesn't "disappear" in Python 3000. Not all the Python devs seem to really see the value of it. But, it can sometimes be unclear as to what the attitudes are. > > 2. Is there any type besides Py_STRUCTURE that can have > names > > and fields. If so, what and what do they mean. If > not, you > > should just say that. > > > Yes, you can add fields to a multi-byte primitive if you want. This > would be similar to thinking about the data-format as a C-like union. > Perhaps the data-field has meaning as a 4-byte integer but the > most-significant and least-significant bytes should also be > addressable > individually. > > > Hmm. I think I understand this somewhat better now, but I can't decide > if it's cool or overkill. Is this a supporting a feature that ctypes has? I don't know. It's basically a situation where it's easier to support it than to not and so it's there. > > > > 3. And on this topic, why a tuple of ([names,..], > {field})? Why > > not simply a list of (name, dfobject, offset, meta) for > > example? And what's the meta information if it's not > PyNone? > > Just a string? Anything at all? > > > > The list of names is useful for having an ordered list so you can > traverse the structure in field order. It is technically not > necessary > but it makes it a lot easier to parse a data-format object in offset > order (it is used a bit in NumPy, for example). > > > Right, I got that. Between names and field you are simulating an > ordered dict. What I still don't understand is why you chose to > simulate this ordered dict using a list plus a dictionary rather than > a list of tuples. This may well just be a matter of taste. However, > for the small sizes I'd expect of these lists I would expect a list of > of tuples would perform better than the dictionary solution. Ah. I misunderstood. You are right that if I had considered needing an ordered list of names up front, this kind of thing makes more sense. I think the reason for the choice of dictionary is that I was thinking of field access as attribute look-up which is just dictionary look-up. So, conceptually that was easier for me. But, tuples are probably less over-head (especially for small numbers of fields) with the expense of having to search for the field-name on field access. But, I'm trusting that dictionaries (especially small ones) are pretty optimized in Python (I haven't tested that assertion in this particular case, though). > > FWIW, the array protocol PEP seems more relevant to what I do since > I'm not concerned so much with the overhead since I'm sending big > chunks of data back and forth. This proposal is trying to get the array protocol *into* Python. So, this is the array protocol PEP. Anyone supportive of the array protocol should be interested in and thinking about this PEP. -Travis From Chris.Barker at noaa.gov Tue Jan 9 02:38:03 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 08 Jan 2007 23:38:03 -0800 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <459D71F4.7060507@noaa.gov> <459D766C.40505@ee.byu.edu> <459D8886.2000101@noaa.gov> <459D8B72.2040309@ee.byu.edu> <459D9590.80202@noaa.gov> <459D9D9C.30501@ee.byu.edu> <459EC58E.7000408@noaa.gov> <459F2039.30407@ieee.org> <45A0630F.9050804@ee.byu.edu> Message-ID: <45A3465B.7090302@noaa.gov> Timothy Hochberg wrote: > The reason that I ask is that the two projects that I use regularly are > wxPython and PIL generally operate on relatively large data chunks and > it's not clear that they would see much benefit over this mechanism > versus the array protocol. But is this mechanism any harder? It doesn't look like it to me. In fact, as I have written a tiny bit of Numeric extension code, this looks familiar and pretty easy to work with. > I imagine that between us Chris Barker and I could hack together > something for wxPython (not that I've asked him aout it). I'm not sure when I'll find the time, but I do want to do this. > And code would > probably go a long way to convincing people what a great idea this is. > However, all else being equal, it'd be a lot easier to do this for the > array protocol since there's no extra infrastructure involved. Is it that much infrastructure? It looks like this would, at the least, require an extra include file. If this flies , then that will be delivered with python 2.8? until then (and for older pythons) would various extension writers all need to add this extra file to their source? And might we get a mess with different versions floating around out there trying to interact? > FWIW, the array protocol PEP seems more relevant to what I do since I'm > not concerned so much with the overhead since I'm sending big chunks of > data back and forth. That's the biggest issue, but I think a lot of us use a lot of small arrays as well -- and while I don't know if it's a performance hit worth worrying about, it's always bugged me that is is faster to convert to a python list, then pass it in to wxPython than it is to just pass in the array directly. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception From stefan at sun.ac.za Tue Jan 9 02:50:54 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 9 Jan 2007 09:50:54 +0200 Subject: [Numpy-discussion] defmatrix.sum Message-ID: <20070109075054.GF16625@mentat.za.net> Hi all, I noticed that Tim Leslie picked up on the out=None in defmatrix.sum: def sum(self, axis=None, dtype=None, out=None): """Sum the matrix over the given axis. If the axis is None, sum over all dimensions. This preserves the orientation of the result as a row or column. """ return N.ndarray.sum(self, axis, dtype, out=None)._align(axis) Surely, that out=None should just be out? If out is specified, should sum() still return a value? Cheers St?fan From robert.kern at gmail.com Tue Jan 9 02:58:54 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 09 Jan 2007 01:58:54 -0600 Subject: [Numpy-discussion] defmatrix.sum In-Reply-To: <20070109075054.GF16625@mentat.za.net> References: <20070109075054.GF16625@mentat.za.net> Message-ID: <45A34B3E.9060303@gmail.com> Stefan van der Walt wrote: > Hi all, > > I noticed that Tim Leslie picked up on the out=None in defmatrix.sum: > > def sum(self, axis=None, dtype=None, out=None): > """Sum the matrix over the given axis. If the axis is None, sum > over all dimensions. This preserves the orientation of the > result as a row or column. > """ > return N.ndarray.sum(self, axis, dtype, out=None)._align(axis) > > Surely, that out=None should just be out? Yes. > If out is specified, should > sum() still return a value? Yes. Everything else with an out= parameter does. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bthom at cs.hmc.edu Tue Jan 9 03:07:05 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Tue, 09 Jan 2007 00:07:05 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <45A1E97A.1040507@gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> Message-ID: <7421A494-3033-43B9-A097-85DC1FC45861@cs.hmc.edu> Hi, I am hopefully almost done w/this exercise, but am uncertain how to interpret what the "---, +++, @@ -48,7" etc. I've looked at the setupext.py file on my machine but am unclear what changes I need to make. I tried google to find some interpretation, but was not successful. I'm somewhat familiar w/diff. Advice greatly appreciated. Thanks, --b On Jan 7, 2007, at 10:49 PM, Robert Kern wrote: > * Now, get the matplotlib source. The latest release is 0.87.7 and > is a good bet. > > http://downloads.sourceforge.net/matplotlib/matplotlib-0.87.7.tar.gz > > * Unpack it and cd to the source directory in the Terminal. > > * In order to tell matplotlib where to find the MacPorts libraries, > you need to > edit setupext.py: > > Index: setupext.py > =================================================================== > --- setupext.py (revision 2861) > +++ setupext.py (working copy) > @@ -48,7 +48,7 @@ > 'linux2' : ['/usr/local', '/usr',], > 'linux' : ['/usr/local', '/usr',], > 'cygwin' : ['/usr/local', '/usr',], > - 'darwin' : ['/sw/lib/freetype2', '/sw/lib/freetype219', '/usr/ > local', > + 'darwin' : ['/opt/local', '/usr/local', > '/usr', '/sw'], > 'freebsd4' : ['/usr/local', '/usr'], > 'freebsd5' : ['/usr/local', '/usr'], > > > * Now just build and install like any other Python package. > > $ python setup.py build > $ python setup.py install From robert.kern at gmail.com Tue Jan 9 03:11:47 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 09 Jan 2007 02:11:47 -0600 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <7421A494-3033-43B9-A097-85DC1FC45861@cs.hmc.edu> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> <7421A494-3033-43B9-A097-85DC1FC45861@cs.hmc.edu> Message-ID: <45A34E43.9040700@gmail.com> belinda thom wrote: > Hi, > > I am hopefully almost done w/this exercise, but am uncertain how to > interpret what the "---, +++, @@ -48,7" etc. I've looked at the > setupext.py file on my machine but am unclear what changes I need to > make. > > I tried google to find some interpretation, but was not successful. > I'm somewhat familiar w/diff. Advice greatly appreciated. It is a unified diff. I used it instead of describing it in words ("edit such-and-such line with so-and-so contents") because the latter is a pain in the butt and open for misinterpretation. Just edit that one line. Ignore the diff header if you don't understand it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Tue Jan 9 03:45:31 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 09 Jan 2007 17:45:31 +0900 Subject: [Numpy-discussion] Exported symbols and code reorganization. In-Reply-To: References: <45A0B3AB.3080500@gmail.com> Message-ID: <45A3562B.9070000@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On 1/7/07, *Robert Kern* > wrote: > > Charles R Harris wrote: > > > > > No, but as with mtrand, most of those arise from the fact that > these functions > are implemented in files other than the C file that contains the > Python module > code. In order for them to be called from the Python module code, > they need to > be exported, IIRMCC (If I Remember My C Correctly). This appears > to be true of > essentially every other extension module in my site-packages, so I > don't think > it's going to be much of a problem. > > > I don't think unnecessary public symbols are much of a problem, > either, but it bears on the question of breaking up the core numpy > files. The parts could be treated in the same way, i.e., compiled > separately and linked, or they could be treated as you suggested > previously, declared static and the code included into one base file. > I suspect the inclusion route is a bit easier from a portability > standpoint, long link lists can be a problem and the making of > libraries is platform dependent. > Bit late at joining the discussion... I started the same thing, splitting up those files. As Travis reminded me in a previous discussion, most symbols in numpy extensions are private (eg static functions) because that's the way python extension works: http://docs.python.org/ext/methodTable.html ("The method table must be passed to the interpreter in the module's initialization function. The initialization function must be named initname(), where name is the name of the module, and should be the only non-static item defined in the module file:") By not declaring them static, you are explicitly breaking this. Now, I don't see any other reason than avoiding polluting the global namespace with the potential to have clashing names (I don't think numpy is likely to be big enough so that the cost of loading the shared libraries would be prohibitive); but my understanding is that the whole point of having a list of pointer of functions when exporting a C Api from a python module is precisely for this same reason, eg using static, because that's the only portable way to avoid name clashes. The following page explains it fairly clearly: http://docs.python.org/ext/using-cobjects.html There are other, more elegant solutions to avoid name clashes, but they are platform and/or compiler dependent. I exposed the ones I found/heard about in a previous email (gcc specific, but I know at least MS compiler and Sun C compiler have a similar capability): " One elegant solution for this is non portable, unfortunately: recent gcc version has this functionality called new C++ visibility support, which also works for C source files. http://gcc.gnu.org/wiki/Visibility This file explains the different ways of limiting symbols in dso available http://people.redhat.com/drepper/dsohowto.pdf Having several include of C files is the easiest way, and I guess this would be the safest way to start splitting source files. A better way can always be used after anyway, I guess." cheers, David From pau.gargallo at gmail.com Tue Jan 9 04:37:52 2007 From: pau.gargallo at gmail.com (Pau Gargallo) Date: Tue, 9 Jan 2007 10:37:52 +0100 Subject: [Numpy-discussion] Getting 3-D array values from a 2-D array of indexes In-Reply-To: <45A2EBE4.1070706@eos.ubc.ca> References: <45A2EBE4.1070706@eos.ubc.ca> Message-ID: <6ef8f3380701090137u35dd867fudcefce202c398b93@mail.gmail.com> On 1/9/07, Jordan Dawe wrote: > I have a 3D array A of shape (nz, ny, nx) and a 2D array B of shape (ny, > nx) that contains integer indexes. I want to generate a 2D array C of > shape (ny, nx) composed of the values of A at the z-indexes specified in > B. Is there an easy way to do this? > > Jordan you can use integer indexing if you first generate a complete set of coordinates. In your case you have the z-indexes, so you can generate the y and x indices and then use integer indexing: >>> y,x = indices((ny,nx)) >>> C = A[B,y,x] hope it helps, pau From lou_boog2000 at yahoo.com Tue Jan 9 06:27:54 2007 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Tue, 9 Jan 2007 03:27:54 -0800 (PST) Subject: [Numpy-discussion] Where's append in NumPy?? In-Reply-To: <45A2DE09.7070807@gmail.com> Message-ID: <20070109112754.98033.qmail@web34403.mail.mud.yahoo.com> That may be it. I'll get a newer version. --- Robert Kern wrote: > Lou Pecora wrote: > > After import numpy as N > > > > In [10]: print N.__version__ > > 1.1.2881 > > > > does that look right as a recent version? > > No, that's very old. The version number had briefly > gotten bumped to 1.1 in the > repository, but we backed that out quickly. > > -- > Robert Kern -- Lou Pecora My views are my own. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From lou_boog2000 at yahoo.com Tue Jan 9 07:14:25 2007 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Tue, 9 Jan 2007 04:14:25 -0800 (PST) Subject: [Numpy-discussion] Where's append in NumPy?? Can't Installing NumPy. In-Reply-To: <20070109112754.98033.qmail@web34403.mail.mud.yahoo.com> Message-ID: <20070109121425.70804.qmail@web34414.mail.mud.yahoo.com> --- Lou Pecora wrote: > That may be it. I'll get a newer version. > No Luck. I downloaded numpy-1.0-py2.4-macosx10.4.dmg from the MacOSX package site, but the installer kept telling me there was nothing to install. I removed the previous NumPy and numpy.pth from the site packages and got the same message. How do I install numpy when another version is there (apparently erroneously marked as a later version)? -- Lou Pecora My views are my own. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From david at ar.media.kyoto-u.ac.jp Tue Jan 9 09:19:22 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 09 Jan 2007 23:19:22 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) Message-ID: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> Hi, I am finally implementing a C function to replace the current slow implementation of clip in python as promised a few weeks ago. The idea is to implement it as the following: def clip(input, min, max): a = input.copy() putmask(a, a <= min, min) putmask(a, a >= max, max) return a I don't have that much experience in writing general numpy functions, so I was wondering of other people could advise me on the following points. - should I care only about numeric types, eg something where PyArray_ISNUMBER returns true ? - If input is an array of type T1, min a scalar of type T2 and max a scalar of type T3, which type should be the reference, assuming they are all basic numeric types (float, int, etc...). Eg, if T1 is integer, T2 and T3 are float, what am I supposed to do ? Should I use the basic rule as told in numpy doc, and in that case, and do something like this to determine common datatype (pseudo code): int typenum; for each arg in arglist: typenum = PyArray_ObjectType(arg, typenum) - PyArray_PutMask does not assume anything on the datatype of the input array nor this alignement, and treat its binary content as a buffer to char*. Do I need to do the same, or can I just use "normal" pointer arithmetic on arrays which elements can be casted to basic C types ? cheers, David From charlesr.harris at gmail.com Tue Jan 9 10:08:11 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 9 Jan 2007 08:08:11 -0700 Subject: [Numpy-discussion] Exported symbols and code reorganization. In-Reply-To: <45A3562B.9070000@ar.media.kyoto-u.ac.jp> References: <45A0B3AB.3080500@gmail.com> <45A3562B.9070000@ar.media.kyoto-u.ac.jp> Message-ID: On 1/9/07, David Cournapeau wrote: > > Charles R Harris wrote: > > > > > > On 1/7/07, *Robert Kern* > > wrote: > > > > Charles R Harris wrote: > > > > > > > > > > No, but as with mtrand, most of those arise from the fact that > > these functions > > are implemented in files other than the C file that contains the > > Python module > > code. In order for them to be called from the Python module code, > > they need to > > be exported, IIRMCC (If I Remember My C Correctly). This appears > > to be true of > > essentially every other extension module in my site-packages, so I > > don't think > > it's going to be much of a problem. > > > > > > I don't think unnecessary public symbols are much of a problem, > > either, but it bears on the question of breaking up the core numpy > > files. The parts could be treated in the same way, i.e., compiled > > separately and linked, or they could be treated as you suggested > > previously, declared static and the code included into one base file. > > I suspect the inclusion route is a bit easier from a portability > > standpoint, long link lists can be a problem and the making of > > libraries is platform dependent. > > > Bit late at joining the discussion... I started the same thing, > splitting up those files. > > As Travis reminded me in a previous discussion, most symbols in numpy > extensions are private (eg static functions) because that's the way > python extension works: I think we can do what the multiarray module does, provide a table of function pointers in the _XXX_API module attribute. That way the random and fft functions would be available to c extension modules for those who wanted them while also being kept out of the module namespace. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Jan 9 10:19:18 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 9 Jan 2007 08:19:18 -0700 Subject: [Numpy-discussion] reduction In-Reply-To: <45A2825A.5090001@gmail.com> References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> <80c99e790701080932i7b489bfblcc06964162449e8@mail.gmail.com> <45A2825A.5090001@gmail.com> Message-ID: On 1/8/07, Robert Kern wrote: > > lorenzo bolla wrote: > > ops. I did it, too. > > should I delete it?? how?? > > I closed it as a duplicate. As a nondeveloper, you don't have the > permissions to > close tickets yourself. Now that this bug is closed, maybe we should make another point release so folks know they should upgrade. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Jan 9 12:05:02 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 09 Jan 2007 09:05:02 -0800 Subject: [Numpy-discussion] Where's append in NumPy?? Can't Installing NumPy. In-Reply-To: <20070109121425.70804.qmail@web34414.mail.mud.yahoo.com> References: <20070109121425.70804.qmail@web34414.mail.mud.yahoo.com> Message-ID: <45A3CB3E.2090905@noaa.gov> Lou, This is odd. I've always been able to do it. I sure wish Apple had provided a decent package management system! I have a dmg for 1.0.1 that seems to work well. I wonder where I got it? $ python2.4 Python 2.4.3 (#1, Apr 7 2006, 10:54:33) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.__version__ '1.0.1' >>> >>> help(numpy.append) Help on function append in module numpy.lib.function_base: append(arr, values, axis=None) Append to the end of an array along axis (ravel first if None) In any case, I'll send it straight to you in a separate email. Let me know how it works. -Chris Lou Pecora wrote: > --- Lou Pecora wrote: > >> That may be it. I'll get a newer version. >> > > > No Luck. > > I downloaded numpy-1.0-py2.4-macosx10.4.dmg from the > MacOSX package site, but the installer kept telling me > there was nothing to install. I removed the previous > NumPy and numpy.pth from the site packages and got the > same message. How do I install numpy when another > version is there (apparently erroneously marked as a > later version)? > > > > -- Lou Pecora > My views are my own. > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From tim.hochberg at ieee.org Tue Jan 9 12:35:12 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Tue, 9 Jan 2007 10:35:12 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <45A3465B.7090302@noaa.gov> References: <459D6964.30701@ee.byu.edu> <459D8886.2000101@noaa.gov> <459D8B72.2040309@ee.byu.edu> <459D9590.80202@noaa.gov> <459D9D9C.30501@ee.byu.edu> <459EC58E.7000408@noaa.gov> <459F2039.30407@ieee.org> <45A0630F.9050804@ee.byu.edu> <45A3465B.7090302@noaa.gov> Message-ID: On 1/9/07, Christopher Barker wrote: > > Timothy Hochberg wrote: > > The reason that I ask is that the two projects that I use regularly are > > wxPython and PIL generally operate on relatively large data chunks and > > it's not clear that they would see much benefit over this mechanism > > versus the array protocol. Let me preface my remarks by saying, that I was initially assuming that this was meant as a supplement to the earlier array protocol proposal, not a replacement as Travis subsequently explained. But is this mechanism any harder? It doesn't look like it to me. In > fact, as I have written a tiny bit of Numeric extension code, this looks > familiar and pretty easy to work with. I expect that the old proposal is easier to implement right now. We could implement the old array protocol in wxPython and have fairly seemless integration with numpy without any dependencies. To implement the new protocol, we'd need the C-API. Since that's not in Python at the moment, we'd have to include implementations of the various functions into wxPython. I suppose that wouldn't really be that bad assuming they are already implemented somewhere. It's a bit of a chicken and the egg problem though. > > I imagine that between us Chris Barker and I could hack together > > something for wxPython (not that I've asked him aout it). > > I'm not sure when I'll find the time, but I do want to do this. > > > And code would > > probably go a long way to convincing people what a great idea this is. > > However, all else being equal, it'd be a lot easier to do this for the > > array protocol since there's no extra infrastructure involved. > > Is it that much infrastructure? I'm not sure. It may not be that bad. I'd guess you'd need both an include file and a source file for the implementations of the functions. It looks like this would, at the least, require an extra include file. > If this flies , then that will be delivered with python 2.8? until then > (and for older pythons) would various extension writers all need to add > this extra file to their source? And might we get a mess with different > versions floating around out there trying to interact? Possibly. It shouldn't be a big deal if the API is frozen. But I expect the best thing to get this to work would be to implement this for as many projects as possible as trial patches before trying to get this into those projects officially. That way we can get some experience, tweak the API if necessary, then freeze it and release it officially. Like I said, I'll help with wxPython. I'm tempted to try with PIL as well, but I've never looked at the code there, not even tried to compile it, so I don't know how far I'd get. > FWIW, the array protocol PEP seems more relevant to what I do since I'm > > not concerned so much with the overhead since I'm sending big chunks of > > data back and forth. > > That's the biggest issue, but I think a lot of us use a lot of small > arrays as well -- and while I don't know if it's a performance hit worth > worrying about, it's always bugged me that is is faster to convert to a > python list, then pass it in to wxPython than it is to just pass in the > array directly. You should have seen it in the old days before I sped it up. Actually I think you probably did. Anyway, it seems like wxPython is low hanging fruit in the sense that we could probably get it done without too much trouble and it would be pretty easy. It's possible that Robin may not accept the patch until the relevant code goes into Python, but just having a patch available would be a useful template for other project and would show the performance gains this approach would lead to. At least I sure hope so. Travis: does the code implementing the C API exist already, or is that something that still needs to be written? -tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From perry at stsci.edu Tue Jan 9 13:37:56 2007 From: perry at stsci.edu (Perry Greenfield) Date: Tue, 9 Jan 2007 13:37:56 -0500 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <459D8886.2000101@noaa.gov> <459D8B72.2040309@ee.byu.edu> <459D9590.80202@noaa.gov> <459D9D9C.30501@ee.byu.edu> <459EC58E.7000408@noaa.gov> <459F2039.30407@ieee.org> <45A0630F.9050804@ee.byu.edu> <45A3465B.7090302@noaa.gov> Message-ID: <30A65018-3676-43FC-AD8D-5D4C1F95AF88@stsci.edu> A few comments regarding what I think justifies some sort of standard to be part of Python (understanding that there are various ways that it could be done, so I'm not commenting on the specifics here directly). I don't there is any harm in making the standard numpy-centric. In fact, I think the selling point is that this standard means that any extension can expose repetitive data to those that want to manipulate it *in Python* in a simple and standard way. While its possible that ultimately there are those that will pass these sorts of things from one extension into another, I don't see that as a common use for a while. What it does mean is that if you want a simple Python-only way of seeing and modifying such data, all you need to do is install numpy. You don't have write a C extension. If the extension writers use the descriptive field names, they can make the array they expose somewhat self-documenting so that turning it into an array and doing a little introspection may tell you all you need to know about the data. Rather than try to sell this as some neutral interface, I would make the numpy-dependence explicit (not that it excludes extension-to- extension direct use). It may be that the developers of various extensions are not the ones most interested in this capability (after all, they've build it to do what they want), but I wouldn't be surprised if many of the users of the extension would like it so they can do things the extension doesn't allow them. So one approach is to see what the respective user communities think about such capabilities. If they find out what this can do for them, they may pressure the developers for such support. People in the numpy community can also volunteer to implement the standard (but with this approach, it's a bit of a chicken and egg thing as someone has mentioned. You can't do it if it isn't in Python yet.) I do agree that the most persuasive approach would be to have at least some of the 3rd party extensions support this explicitly on the python-dev list. Perry From lou_boog2000 at yahoo.com Tue Jan 9 13:41:58 2007 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Tue, 9 Jan 2007 10:41:58 -0800 (PST) Subject: [Numpy-discussion] Where's append in NumPy?? Can't Installing NumPy. In-Reply-To: <45A3CB3E.2090905@noaa.gov> Message-ID: <372317.91697.qm@web34405.mail.mud.yahoo.com> Hi, Chris, Some answers below (sort of): --- Christopher Barker wrote: > Lou, > > This is odd. I've always been able to do it. I sure > wish Apple had > provided a decent package management system! Amen! > I have a dmg for 1.0.1 that seems to work well. I > wonder where I got it? I got my original NumPy that wouldn't install from the MacPython site. It was a package with the usual install dialog. Even wiping the existing NumPy from site-packages didn't work. Go figure. Finally got NumPy installed using the tarball from http://www.scipy.org/Download. Did the usual, sudo python setup install stuff and that worked. I now have NumPy 1.01 and it has 'append'. For others information: I had to update SciPy and Matplotlib, too. SciPy turned out *not* to be doable from its tarball on http://www.scipy.org/Download. I got the following error (after a whole LOT of output from compiling, etc.): /usr/local/bin/g77 -g -Wall -undefined dynamic_lookup...blah, blah.../scipy/fftpack/_fftpack.so /usr/bin/ld: can't locate file for: -lcc_dynamic collect2: ld returned 1 exit status /usr/bin/ld: can't locate file for: -lcc_dynamic collect2: ld returned 1 exit status error: Command "/usr/local/bin/g77 -g -Wall -undefined dynamic_lookup ...blah, blah.../scipy/fftpack/_fftpack.so" failed with exit status 1 Exit 1 Whatever collect2 is ld couldn;t find it. I finally got SciPy installed from the superpack at the http://www.scipy.org/Download site. That seems to work, giving me version 0.5.3dev for SciPy. Matplotlib installed fine from tarball and the setup stuff. I hope that's of use to someone out there. Each time I upgrade something it's an adventure. Thanks for the feedback. -- Lou Pecora My views are my own. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From robert.kern at gmail.com Tue Jan 9 14:24:40 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 09 Jan 2007 13:24:40 -0600 Subject: [Numpy-discussion] Where's append in NumPy?? Can't Installing NumPy. In-Reply-To: <372317.91697.qm@web34405.mail.mud.yahoo.com> References: <372317.91697.qm@web34405.mail.mud.yahoo.com> Message-ID: <45A3EBF8.8090109@gmail.com> Lou Pecora wrote: > /usr/local/bin/g77 -g -Wall -undefined > dynamic_lookup...blah, > blah.../scipy/fftpack/_fftpack.so > > /usr/bin/ld: can't locate file for: -lcc_dynamic Yup. You can't use g77 with gcc 4. See the instructions I give in the thread "recompiling needed for binary module after numpy 1.0". -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lou_boog2000 at yahoo.com Tue Jan 9 16:19:34 2007 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Tue, 9 Jan 2007 13:19:34 -0800 (PST) Subject: [Numpy-discussion] Where's append in NumPy?? Can't Installing NumPy. In-Reply-To: <45A3EBF8.8090109@gmail.com> Message-ID: <20070109211934.56579.qmail@web34414.mail.mud.yahoo.com> Ah, that does ring a bell. Sigh. I need to upgrade my memory banks. Sure is tough keeping these packages in sync. Thanks. I'll check it out. --- Robert Kern wrote: > Lou Pecora wrote: > > > /usr/local/bin/g77 -g -Wall -undefined > > dynamic_lookup...blah, > > blah.../scipy/fftpack/_fftpack.so > > > > /usr/bin/ld: can't locate file for: -lcc_dynamic > > Yup. You can't use g77 with gcc 4. See the > instructions I give in the thread > "recompiling needed for binary module after numpy > 1.0". > > -- > Robert Kern __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From oliphant at ee.byu.edu Tue Jan 9 16:32:50 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 09 Jan 2007 14:32:50 -0700 Subject: [Numpy-discussion] reduction In-Reply-To: References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> <80c99e790701080932i7b489bfblcc06964162449e8@mail.gmail.com> <45A2825A.5090001@gmail.com> Message-ID: <45A40A02.6050606@ee.byu.edu> Charles R Harris wrote: > > > On 1/8/07, *Robert Kern* > wrote: > > lorenzo bolla wrote: > > ops. I did it, too. > > should I delete it?? how?? > > I closed it as a duplicate. As a nondeveloper, you don't have the > permissions to > close tickets yourself. > > > Now that this bug is closed, maybe we should make another point > release so folks know they should upgrade. I agree. Let's shoot for a point release for next week. Other documentation improvements should be made by then. -Travis From oliphant at ee.byu.edu Tue Jan 9 16:33:58 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 09 Jan 2007 14:33:58 -0700 Subject: [Numpy-discussion] reduction In-Reply-To: References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> <80c99e790701080932i7b489bfblcc06964162449e8@mail.gmail.com> <45A2825A.5090001@gmail.com> Message-ID: <45A40A46.8040007@ee.byu.edu> Charles R Harris wrote: > > > On 1/8/07, *Robert Kern* > wrote: > > lorenzo bolla wrote: > > ops. I did it, too. > > should I delete it?? how?? > > I closed it as a duplicate. As a nondeveloper, you don't have the > permissions to > close tickets yourself. > > > Now that this bug is closed, maybe we should make another point > release so folks know they should upgrade. I agree. Let's shoot for a point release for next week. Other documentation improvements should be made by then. -Travis From oliphant at ee.byu.edu Tue Jan 9 16:35:05 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 09 Jan 2007 14:35:05 -0700 Subject: [Numpy-discussion] reduction In-Reply-To: References: <80c99e790701080723s7f962e2bkb9ff52edc5bd02d1@mail.gmail.com> <80c99e790701080838x5cf8e3a5jbe5265e4b1ca754b@mail.gmail.com> <80c99e790701080932i7b489bfblcc06964162449e8@mail.gmail.com> <45A2825A.5090001@gmail.com> Message-ID: <45A40A89.5020306@ee.byu.edu> Charles R Harris wrote: > > > On 1/8/07, *Robert Kern* > wrote: > > lorenzo bolla wrote: > > ops. I did it, too. > > should I delete it?? how?? > > I closed it as a duplicate. As a nondeveloper, you don't have the > permissions to > close tickets yourself. > > > Now that this bug is closed, maybe we should make another point > release so folks know they should upgrade. I agree. Let's shoot for a point release for next week. Other documentation improvements should be made by then. -Travis From erin.sheldon at gmail.com Wed Jan 10 00:15:47 2007 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Tue, 9 Jan 2007 21:15:47 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <45A2060C.8010909@gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> <331116dc0701080038wf58be61xcced1cc757198d65@mail.gmail.com> <45A2060C.8010909@gmail.com> Message-ID: <331116dc0701092115g2566fb3tc93f3ebcfbb59a01@mail.gmail.com> I'm finally getting to this, I'm on the road. Here is what gdb gives me --snip-- Reading symbols for shared libraries . done Reading symbols for shared libraries . done Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x5a6ae09a parse_fond (fond_data=0x5a6ae060
, have_sfnt=0x5a6ae09c, sfnt_id=0xbfff8f4e, lwfn_file_name=0xbfff8e4a "", face_index=0) at /usr/local/src/freetype-2.2.1/src/base/ftmac.c:628 628 /usr/local/src/freetype-2.2.1/src/base/ftmac.c: No such file or directory. in /usr/local/src/freetype-2.2.1/src/base/ftmac.c For some reason it is looking for something from freetype in /usr/local/src but this directory has never existed on my computer. This is a new computer and I used the migration assistant to copy things from my old computer, so perhaps some paths got messed up. I don't know how to address that type of issue on a Mac with no ld.so.conf type of file. Erin On 1/8/07, Robert Kern wrote: > Erin Sheldon wrote: > > I just got a new MacBook Pro, core 2 duo. There is no > > numpy/scipy fink distro for the intels yet so I tried out > > Robert's suggestions above. I installed python 2.5 as > > the base. I can confirm that all is working well for > > numpy and scipy. > > > > Matplotlib compiled, but gave some odd warnings such > > as > > > > /usr/bin/ld: warning fat file: /usr/local/lib/libgcc_s.10.4.dylib does > > not contain an architecture that matches the specified -arch flag: ppc > > (file ignored) > > Hmm. That is a bit odd. That library comes from the gfortran install and is a > standard gcc runtime library. It's probably getting picked up for the extension > modules that use -L/usr/local/lib . Not great, but shouldn't be harmful unless > if you really want Universal binaries that will run on PPC Macs, too. > > > And when I run a "import pylab" with either wxAgg or TkAgg as > > my backend it just segfaults. Any ideas? The > > only way I differed from Robert's suggestions was I used fink > > to install jpeg,png,freetype. > > fink install libjpeg > > fink install libpng3 > > fink install freetype2 > > And I did not alter the setupext.py file. > > I don't know. Try running it under gdb to get a C stacktrace so we can tell > where the segfault comes from. The lines starting with (gdb) are prompts where > you should enter the lines I give below: > > > [~]$ gdb pythonw > GNU gdb 6.3.50-20050815 (Apple version gdb-573) (Fri Oct 20 15:50:43 GMT 2006) > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-apple-darwin"...Reading symbols for shared > libraries .. done > > (gdb) run > Starting program: /usr/local/bin/pythonw > Reading symbols for shared libraries . done > > Program received signal SIGTRAP, Trace/breakpoint trap. > 0x8fe01010 in __dyld__dyld_start () > (gdb) continue > Continuing. > Reading symbols for shared libraries . done > Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) > [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > Reading symbols for shared libraries .. done > >>> from matplotlib.pylab import * > Reading symbols for shared libraries .... done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries ........ done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries > ........................................................... done > Reading symbols for shared libraries .. done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries ... done > Reading symbols for shared libraries . done > Reading symbols for shared libraries .. done > Reading symbols for shared libraries . done > Reading symbols for shared libraries .................. done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > >>> > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Wed Jan 10 00:41:30 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 09 Jan 2007 23:41:30 -0600 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <331116dc0701092115g2566fb3tc93f3ebcfbb59a01@mail.gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> <331116dc0701080038wf58be61xcced1cc757198d65@mail.gmail.com> <45A2060C.8010909@gmail.com> <331116dc0701092115g2566fb3tc93f3ebcfbb59a01@mail.gmail.com> Message-ID: <45A47C8A.1040300@gmail.com> Erin Sheldon wrote: > I'm finally getting to this, I'm on the road. Here is > what gdb gives me > > --snip-- > Reading symbols for shared libraries . done > Reading symbols for shared libraries . done > Reading symbols for shared libraries .. done > > Program received signal EXC_BAD_ACCESS, Could not access memory. > Reason: KERN_INVALID_ADDRESS at address: 0x5a6ae09a > parse_fond (fond_data=0x5a6ae060
, > have_sfnt=0x5a6ae09c, sfnt_id=0xbfff8f4e, lwfn_file_name=0xbfff8e4a > "", face_index=0) at > /usr/local/src/freetype-2.2.1/src/base/ftmac.c:628 > 628 /usr/local/src/freetype-2.2.1/src/base/ftmac.c: No such file > or directory. > in /usr/local/src/freetype-2.2.1/src/base/ftmac.c > > For some reason it is looking for something from freetype > in /usr/local/src but this directory has never existed > on my computer. This is a new computer and I used the migration > assistant to copy things from my old computer, so perhaps some > paths got messed up. I don't know how to address > that type of issue on a Mac with no ld.so.conf type of file. Your build of matplotlib is probably picking up a different freetype library than the one you intended, one which somebody else built. Possibly the one in /usr/X11R6/lib/. You can use otool(1) to figure out what shared libraries your extension modules are linking against: [matplotlib]$ otool -L ft2font.so ft2font.so: /opt/local/lib/libfreetype.6.dylib (compatibility version 10.0.0, current version 10.8.0) /opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3) /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.3) /usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Wed Jan 10 00:48:16 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 09 Jan 2007 23:48:16 -0600 Subject: [Numpy-discussion] Getting 3-D array values from a 2-D array of indexes In-Reply-To: <45A2EBE4.1070706@eos.ubc.ca> References: <45A2EBE4.1070706@eos.ubc.ca> Message-ID: <45A47E20.6030304@gmail.com> Jordan Dawe wrote: > I have a 3D array A of shape (nz, ny, nx) and a 2D array B of shape (ny, > nx) that contains integer indexes. I want to generate a 2D array C of > shape (ny, nx) composed of the values of A at the z-indexes specified in > B. Is there an easy way to do this? An alternative to Pau's suggestion is to use the .choose() method, which is tailor-made for this use-case. In [9]: nz, ny, nx = (3, 4, 5) In [10]: B = array([[0, 1, 2, 1, 0], [1, 2, 1, 0, 1], [2, 1, 0, 1, 2], [1, 0, 1, 2, 1]]) In [11]: B Out[11]: array([[0, 1, 2, 1, 0], [1, 2, 1, 0, 1], [2, 1, 0, 1, 2], [1, 0, 1, 2, 1]]) In [12]: A = arange(nz*ny*nx).reshape((nz,ny,nx)) In [13]: A Out[13]: array([[[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19]], [[20, 21, 22, 23, 24], [25, 26, 27, 28, 29], [30, 31, 32, 33, 34], [35, 36, 37, 38, 39]], [[40, 41, 42, 43, 44], [45, 46, 47, 48, 49], [50, 51, 52, 53, 54], [55, 56, 57, 58, 59]]]) In [14]: B.choose(A) Out[14]: array([[ 0, 21, 42, 23, 4], [25, 46, 27, 8, 29], [50, 31, 12, 33, 54], [35, 16, 37, 58, 39]]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Wed Jan 10 12:35:47 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 10 Jan 2007 10:35:47 -0700 Subject: [Numpy-discussion] Docstring standards for NumPy and SciPy Message-ID: <45A523F3.3070408@ee.byu.edu> There was a lively discussion on the SciPy List before Christmas regarding establishing a standard for documentation strings for NumPy / SciPy. I am very interested in establishing such a standard. A hearty thanks goes to William Stein for encouraging the discussion. I hope it is very clear that the developers of NumPy / SciPy are quite interested in good documentation strings but recognize that producing them can be fairly tedious and un-interesting work. This is the best explanation I can come up with for the relative paucity of documentation rather than some underlying agenda *not* to produce them. I suspect a standard has not been established largely because of all the discussions taking place within the documentation communities of epydoc, docutils, etc. and a relative unclarity on what to do about Math in docstrings. I'd like to get something done within the next few days (like placing the standard on a wiki and/or placing a HOWTO_DOCUMENT file in the distribution of NumPy). My preference is to use our own basic format for documentation with something that will translate the result into something that the epydoc package can process (like epytext or reStructuredText). The reason, I'd like to use our own simple syntax, is that I'm not fully happy with either epytext or reStructuredText. In general, I don't like a lot of line-noise and "formatting" extras. Unfortuntately both epytext and reStructuredText seem to have their fair share of such things. Robert wrote some great documentation for a few functions (apparently following a reStructuredText format). While I liked that he did this, it showed me that I don't very much like all the line-noise needed for structured text. I've looked through a large number of documentation strings that I've written over the years and believe that the following format suffices. I would like all documentation to follow this format. This format attempts to be a combination of epytext and restructured text with additions for latex-math. The purpose is to make a docstring readable but also allowing for some structured text directives. At some point we will have a sub-routine that will translate docstrings in this format to pure epytext or pure restructured text. """ one-line summary not using variable names or the function name A few sentences giving an extended description. Inputs: var1 -- Explanation variable2 -- Explanation Outputs: named, list, of, outputs named -- explanation list -- more detail of -- blah, blah. outputs -- even more blah Additional Inputs: kwdarg1 -- A little-used input not always needed. kwdarg2 -- Some keyword arguments can and should be given in Inputs Section. This is just for "little-used" inputs. Algorithm: Notes about the implemenation algorithm (if needed). This can have multiple paragraphs as can all sections. Notes: Additional notes if needed Authors: name (date): notes about what was done name (date): major documentation updates can be included here also. See also: * func1 -- any notes about the relationship * func2 -- * func3 -- (or this can be a comma separated list) func1, func2, func3 (For NumPy functions, these do not need to have numpy. namespace in front of them) (For SciPy they don't need the scipy. namespace in front of them). (Use np and sp for abbreviations to numpy and scipy if you need to reference the other package). Examples: examples in doctest format Comments: This section should include anything that should not be displayed in a help or other hard-copy output. Such things as substitution-directed directives should go here. """ Additional Information: For paragraphs, indentation is significant and indicates indentation in the output. New paragraphs are marked with blank line. Text-emphasis: Use *italics*, **bold**, and `courier` if needed in any explanations (but not for variable names and doctest code or multi-line code) Math: Use \[...\] or $...$ for math in latex format (remember to use the raw-format for your text string in such cases). Place it in a new-paragraph for displaystyle or in-line for inline style. References: Use L{reference-link} for any code links (except in the see-also section). The reference-link should contain the full path-name (unless the function is in the same name-space as this one is. Use http:// for any URL's Lists: * item1 - subitem + subsubitem * item2 * item3 or 1. item1 a. subitem i. subsubitem1 ii. subsubitem2 2. item2 3. item3 for lists. Definitions: descripition This is my description for any definitions needed. Addtional Code-blocks: {{{ for multi-line code-blocks that are not examples to be run but should be formatted as code. }}} Tables: Tables should be constructed as either: +------------------------+------------+----------+ | Header row, column 1 | Header 2 | Header 3 | +========================+============+==========+ | body row 1, column 1 | column 2 | column 3 | +------------------------+------------+----------+ | body row 2 | Cells may span | +------------------------+-----------------------+ or || Header row, column 1 || Header 2 || Header 3 || ------------------------------------------------------- || body row, column 1 || column 2 || column 3 || || body row 2 |||| Cells may span columns || Footnotes: [1] or [CITATION3] for Footnotes which are placed at the bottom of the docstring as [1] Footnote [CITATION3] Additional note. Substitution: Use |somename|{optional text} with (the next line is placed at the bottom of the docstring in the Comments: section) .. |somename| image::myfile.png or .. |somename| somedirective:: {optional text} for placing restructured text directives in the main text. Please address comments to this proposal, very soon. I'd like to finalize it within a few days. -Travis From lbolla at gmail.com Wed Jan 10 13:13:27 2007 From: lbolla at gmail.com (lorenzo bolla) Date: Wed, 10 Jan 2007 19:13:27 +0100 Subject: [Numpy-discussion] Python EM Project Message-ID: <80c99e790701101013p2f390b2v14a6fd9602e78abf@mail.gmail.com> Hi all, I'm not sure if I am slightly out of topic, but I can't find information anywhere else. Can someone tell me where the Python EM Project has gone? The website www.pythonemproject.com that used to host it is now a bunch of ads, property of DomainDrop. Also Robert Lytle, who seemed to be "responsible" for the project, cannot be contacted via e-mail. Thank you! Lorenzo -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Jan 10 14:17:36 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 10 Jan 2007 13:17:36 -0600 Subject: [Numpy-discussion] Python EM Project In-Reply-To: <80c99e790701101013p2f390b2v14a6fd9602e78abf@mail.gmail.com> References: <80c99e790701101013p2f390b2v14a6fd9602e78abf@mail.gmail.com> Message-ID: <45A53BD0.6090605@gmail.com> lorenzo bolla wrote: > Hi all, I'm not sure if I am slightly out of topic, but I can't find > information anywhere else. > Can someone tell me where the Python EM Project has gone? The website > > www.pythonemproject.com that used to host it is now a bunch of ads, property > of DomainDrop. Also Robert Lytle, who seemed to be "responsible" for the > project, cannot be contacted via e-mail. I don't think anyone else knows, either. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From faltet at carabos.com Wed Jan 10 14:28:14 2007 From: faltet at carabos.com (Francesc Altet) Date: Wed, 10 Jan 2007 20:28:14 +0100 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> Message-ID: <1168457294.3617.36.camel@localhost.localdomain> El dt 09 de 01 del 2007 a les 23:19 +0900, en/na David Cournapeau va escriure: > Hi, > > I am finally implementing a C function to replace the current slow > implementation of clip in python as promised a few weeks ago. The idea > is to implement it as the following: > > def clip(input, min, max): > a = input.copy() > putmask(a, a <= min, min) > putmask(a, a >= max, max) > return a > > I don't have that much experience in writing general numpy functions, so > I was wondering of other people could advise me on the following points. > Sorry, but not real experience writing extensions directly in C. However, you may want to experiment using numexpr for doing what you want. It's relatively easy to extend numexpr and adding a new opcode to its virtual machine. I'm attaching a patch for implementing such a clip routine (only for floating point types, but given the example, it should be straightforward to add support for integers as well). Also, you should note that using the fancy indexing of numpy seems to perform better than the putmask approach. Below are my figures for a small benchmark (also attached) for testing the performance of clip using several approaches: time (putmask)--> 1.38 time (where)--> 2.713 time (numexpr where)--> 1.291 time (fancy+assign)--> 0.967 time (numexpr clip)--> 0.596 It is interesting to see there how fancy-indexing + assignation is quite more efficient than putmask. Unfortunately, integrating this clip version officially in numexpr doesn't seem realistic as the main authors are trying hard to avoid cluttering the VM with too many opcodes (they don't want to overload the instruction cache of the CPU). Nevertheless, as the implementation of clip in numexpr should be rather optimal, its performance can still be used as a reference for other implementations. HTH and good luck with your own clip implementation, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth -------------- next part -------------- A non-text attachment was scrubbed... Name: clip-bench.py Type: text/x-python Size: 922 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: numexpr-clip.patch Type: text/x-patch Size: 2116 bytes Desc: not available URL: From bthom at cs.hmc.edu Wed Jan 10 14:52:57 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Wed, 10 Jan 2007 11:52:57 -0800 Subject: [Numpy-discussion] installing numpy, matplotlib, scipy from source on a Mac In-Reply-To: <3d375d730701080252m35568b63q1bf758c590655594@mail.gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> <3d375d730701080252m35568b63q1bf758c590655594@mail.gmail.com> Message-ID: I am posting this message to both numpy and matplotlib mailing lists because the thread relates to both. First, Robert Kern kindly provided step-by-step instructions for Macs (PPCs and Intels) regarding how to install FROM SOURCE the packages needed to allow Python to become a viable alternative for Matlab. Details regarding the installation of matplotlib (along with wx, which is supposed to allow the WxAgg backend to work), numpy, and scipy were provided. The reason for installing FROM SOURCE on a Mac is because various packages need to be compiled in such a way that they rely on the same underling shared libraries, etc. for things to work (I've described some problems w/installing from numpy, scipy, and matplotlib dmgs at http://www.mail-archive.com/ numpy-discussion at scipy.org/ msg00481.html). From these instructions, I've been able to acheive numpy.test() anad scipy.test() that run w/no failures -- hooray! However, after installing wx and matplotlib, various problems result: 1) warnings about fonts 2) wx fails to work I've appended the warnings below. These only occur the first time pylab is imported (does this make sense?). After that, I've appended the issues I've had when trying to use wx as a backend. One reason I'd like to be able to use wx is that it appears with TkAgg, running IDLE (w/the -n flag) can hang (see: http://www.mail-archive.com/matplotlib-users at lists.sourceforge.net/ msg02039.html). Point that originally confused me: although wx is needed for the WxAgg backend, you don't need to install the analogous Tk package to use TkAgg PROVIDED you're using MacPython, for that comes bundled w/ an Aqua-based Tk interface. I'm on a G4 PPC, w/OS X 10.4.8, and using MacPython 2.4. Any idea what wx doesn't work? Thanks, --b ONE-TIME WARNINGS: -------------------- Python 2.4.4 (#1, Oct 18 2006, 10:34:39) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. history mechanism set up >>> import pylab as P /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/matplotlib/font_manager.py:455: UserWarning: Could not open font file /Library/Fonts/NISC18030.ttf warnings.warn("Could not open font file %s"%fpath) /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/matplotlib/font_manager.py:455: UserWarning: Could not open font file /System/Library/Fonts/TimesLTMM warnings.warn("Could not open font file %s"%fpath) >>> WX / MATPLOTLIB FAILURE -------------------------- 4 % python Python 2.4.4 (#1, Oct 18 2006, 10:34:39) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. history mechanism set up >>> import matplotlib as M >>> import pylab as P >>> M.rcParams['interactive'] True >>> M.rcParams['backend'] 'WXAgg' >>> P.plot([1,2,3]) Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/matplotlib/backends/backend_wx.py", line 1048, in _onPaint self.draw(repaint=False) figManager.canvas.draw() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/matplotlib/backends/backend_wxagg.py", line 63, in draw self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None) MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create the wx.Bitmap >>> From cookedm at physics.mcmaster.ca Wed Jan 10 14:57:45 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed, 10 Jan 2007 11:57:45 -0800 Subject: [Numpy-discussion] Exported symbols and code reorganization. In-Reply-To: References: Message-ID: On Jan 7, 2007, at 00:16 , Charles R Harris wrote: > > That brings up the main question I have about how to break up the C > files. I note that most of the functions in multiarraymodule.c, for > instance, are part of the C-API, and are tagged as belonging to > either the MULTIARRAY_API or the OBJECT_API. Apparently the build > system scans for these tags and extracts the files somewhere. So, > what is this API, is it available somewhere or is the code just > copied somewhere convenient. As to breaking up the files, the scan > only covers the code in the two current files, included code from > broken out parts is not seen. This strikes me as a bit of a kludge, > but I am sure there is a reason for it. Anyway, I assume the build > system can be fixed, so that brings up the question of how to break > up the files. The maximal strategy is to make every API functions, > with it's helper functions, a separate file. This adds a *lot* of > files, but it is straight forward and modular. A less drastic > approach is to start by breaking multiarraymodule into four files: > the converters, the two apis, and the module functions. My own > preference is for the bunch of files, but I suspect some will object. The code for pulling out the ``MULTIARRAY_API`` and ``OBJECT_API`` (also ``UFUNC_API``) is in ``numpy/core/code_generators``. Taking ``MULTIARRAY_API`` as an example, the ``generate_array_api.py`` is run by the ``numpy/core/setup.py`` file to generate the multiarray (and object) API. The file ``numpy/core/code_generators/ array_api_order.txt`` is the order in which the API functions are added to the ``PyArray_API`` array; this is our guarantee that the binary API doesn't change when functions are added. The files scanned are listed ``in numpy/core/code_generators/genapi.py``, which is also the module that does the heavy lifting in extracting the tagged functions. The parameters and a leading comment for tagged functions are extracted from the API source files; code is generated that sets ``PyArray_API`` up correctly (this is put into ``__multiarray.c`` in the build ``src/`` directory), and a header file is created (``__multiarray.h`` in the build ``include/`` directory) that contains the ``#define's`` for calling the API functions as function pointers into ``PyArray_API``. Also, a file ``multiarray_api.txt`` is created that contains the leading comments for the tagged functions, along with the function signatures, in reStructuredText format. As to how to break up the files, I would prefer the four files approach. I find that a bunch of files (one for each function) to be difficult to work with, as I'm continually opening different files to find functions. It's also to me taking modularity to the extreme, where your first level of division into chunks (functions) are now the same size as your second level (files). Some of the functions, however, are quite large, and could do with breaking up into smaller functions. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From stefan at sun.ac.za Wed Jan 10 16:49:23 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 10 Jan 2007 23:49:23 +0200 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <1168457294.3617.36.camel@localhost.localdomain> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> Message-ID: <20070110214923.GP16625@mentat.za.net> On Wed, Jan 10, 2007 at 08:28:14PM +0100, Francesc Altet wrote: > El dt 09 de 01 del 2007 a les 23:19 +0900, en/na David Cournapeau va > escriure: > > Hi, > > > > I am finally implementing a C function to replace the current slow > > implementation of clip in python as promised a few weeks ago. The idea > > is to implement it as the following: > > > > def clip(input, min, max): > > a = input.copy() > > putmask(a, a <= min, min) > > putmask(a, a >= max, max) > > return a > > > > I don't have that much experience in writing general numpy functions, so > > I was wondering of other people could advise me on the following points. > > > > Sorry, but not real experience writing extensions directly in C. > However, you may want to experiment using numexpr for doing what you > want. It's relatively easy to extend numexpr and adding a new opcode to > its virtual machine. I'm attaching a patch for implementing such a clip > routine (only for floating point types, but given the example, it should > be straightforward to add support for integers as well). > > Also, you should note that using the fancy indexing of numpy seems to > perform better than the putmask approach. Below are my figures for a > small benchmark (also attached) for testing the performance of clip > using several approaches: > > time (putmask)--> 1.38 > time (where)--> 2.713 > time (numexpr where)--> 1.291 > time (fancy+assign)--> 0.967 > time (numexpr clip)--> 0.596 > > It is interesting to see there how fancy-indexing + assignation is quite > more efficient than putmask. Not on my machine: time (putmask)--> 0.181 time (where)--> 0.783 time (numexpr where)--> 0.26 time (fancy+assign)--> 0.202 Cheers St?fan From charlesr.harris at gmail.com Wed Jan 10 16:52:12 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 10 Jan 2007 14:52:12 -0700 Subject: [Numpy-discussion] Exported symbols and code reorganization. In-Reply-To: References: Message-ID: On 1/10/07, David M. Cooke wrote: > > On Jan 7, 2007, at 00:16 , Charles R Harris wrote: > > > > That brings up the main question I have about how to break up the C > > files. I note that most of the functions in multiarraymodule.c, for > > instance, are part of the C-API, and are tagged as belonging to > > either the MULTIARRAY_API or the OBJECT_API. Apparently the build > > system scans for these tags and extracts the files somewhere. So, > > what is this API, is it available somewhere or is the code just > > copied somewhere convenient. As to breaking up the files, the scan > > only covers the code in the two current files, included code from > > broken out parts is not seen. This strikes me as a bit of a kludge, > > but I am sure there is a reason for it. Anyway, I assume the build > > system can be fixed, so that brings up the question of how to break > > up the files. The maximal strategy is to make every API functions, > > with it's helper functions, a separate file. This adds a *lot* of > > files, but it is straight forward and modular. A less drastic > > approach is to start by breaking multiarraymodule into four files: > > the converters, the two apis, and the module functions. My own > > preference is for the bunch of files, but I suspect some will object. > > The code for pulling out the ``MULTIARRAY_API`` and ``OBJECT_API`` > (also ``UFUNC_API``) is in ``numpy/core/code_generators``. Taking > ``MULTIARRAY_API`` as an example, the ``generate_array_api.py`` is > run by the ``numpy/core/setup.py`` file to generate the multiarray > (and object) API. The file ``numpy/core/code_generators/ > array_api_order.txt`` is the order in which the API functions are > added to the ``PyArray_API`` array; this is our guarantee that the > binary API doesn't change when functions are added. The files scanned > are listed ``in numpy/core/code_generators/genapi.py``, which is also > the module that does the heavy lifting in extracting the tagged > functions. Looked to me like the order could change without causing problems. The include file was also written by the code generator and for extension modules was just a bunch of macros assigning the proper function pointer to the correct name. That brings up another bit, however. At some point I would like to break the include file into two parts, one for inclusion in the other numpy modules and another for inclusion in extension modules, the big #ifdef in the current file offends my sense of esthetics. It should also be possible to attach the function pointers to real function prototype like declarations, which would help extension modules check the code at compile time. The parameters and a leading comment for tagged functions are > extracted from the API source files; code is generated that sets > ``PyArray_API`` up correctly (this is put into ``__multiarray.c`` in > the build ``src/`` directory), and a header file is created > (``__multiarray.h`` in the build ``include/`` directory) that > contains the ``#define's`` for calling the API functions as function > pointers into ``PyArray_API``. Also, a file ``multiarray_api.txt`` is > created that contains the leading comments for the tagged functions, > along with the function signatures, in reStructuredText format. > > As to how to break up the files, I would prefer the four files > approach. I find that a bunch of files (one for each function) to be > difficult to work with, as I'm continually opening different files to > find functions. It's also to me taking modularity to the extreme, > where your first level of division into chunks (functions) are now > the same size as your second level (files). Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Wed Jan 10 17:39:23 2007 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Wed, 10 Jan 2007 17:39:23 -0500 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <20070110214923.GP16625@mentat.za.net> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> Message-ID: On 10/01/07, Stefan van der Walt wrote: > On Wed, Jan 10, 2007 at 08:28:14PM +0100, Francesc Altet wrote: > > El dt 09 de 01 del 2007 a les 23:19 +0900, en/na David Cournapeau va > > escriure: > > > Hi, > > > > > > I am finally implementing a C function to replace the current slow > > > implementation of clip in python as promised a few weeks ago. The idea > > > is to implement it as the following: > > > > > > def clip(input, min, max): > > > a = input.copy() > > > putmask(a, a <= min, min) > > > putmask(a, a >= max, max) > > > return a > > It is interesting to see there how fancy-indexing + assignation is quite > > more efficient than putmask. > > Not on my machine: It is almost certain that the relative efficiency of the different numpy methods depends on what fraction of the data actually needs clipping, so different test data (or real data) is likely to favour one over the other. Why not write the algorithm in C? None of the above is relevant for C, where the algorithm should be "traverse the array; for each element, clip it". If speed is a big concern, the inner loop could be customized for each data type, and max() and min() should probably be used. But in any case, it will almost certainly run faster than any of the above solutions. And distributing the C should pose no problems - just toss it in with the zillion other C functions in numpy. Perverse as it sounds, f2py might actually set things up rather tidily, leaving a FORTRAN function of a handful of lines; even the data type switching could be done in python. A. M. Archibald From cookedm at physics.mcmaster.ca Wed Jan 10 17:41:33 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed, 10 Jan 2007 14:41:33 -0800 Subject: [Numpy-discussion] Exported symbols and code reorganization. In-Reply-To: References: Message-ID: <0374ED3D-6DAE-419B-AF2B-AD2B965D2211@physics.mcmaster.ca> On Jan 10, 2007, at 13:52 , Charles R Harris wrote: > On 1/10/07, David M. Cooke wrote: On > Jan 7, 2007, at 00:16 , Charles R Harris wrote: > > > > That brings up the main question I have about how to break up the C > > files. I note that most of the functions in multiarraymodule.c, for > > instance, are part of the C-API, and are tagged as belonging to > > either the MULTIARRAY_API or the OBJECT_API. Apparently the build > > system scans for these tags and extracts the files somewhere. So, > > what is this API, is it available somewhere or is the code just > > copied somewhere convenient. As to breaking up the files, the scan > > only covers the code in the two current files, included code from > > broken out parts is not seen. This strikes me as a bit of a kludge, > > but I am sure there is a reason for it. Anyway, I assume the build > > system can be fixed, so that brings up the question of how to break > > up the files. The maximal strategy is to make every API functions, > > with it's helper functions, a separate file. This adds a *lot* of > > files, but it is straight forward and modular. A less drastic > > approach is to start by breaking multiarraymodule into four files: > > the converters, the two apis, and the module functions. My own > > preference is for the bunch of files, but I suspect some will > object. > > The code for pulling out the ``MULTIARRAY_API`` and ``OBJECT_API`` > (also ``UFUNC_API``) is in ``numpy/core/code_generators``. Taking > ``MULTIARRAY_API`` as an example, the ``generate_array_api.py`` is > run by the ``numpy/core/setup.py`` file to generate the multiarray > (and object) API. The file ``numpy/core/code_generators/ > array_api_order.txt`` is the order in which the API functions are > added to the ``PyArray_API`` array; this is our guarantee that the > binary API doesn't change when functions are added. The files scanned > are listed ``in numpy/core/code_generators/genapi.py``, which is also > the module that does the heavy lifting in extracting the tagged > functions. > > Looked to me like the order could change without causing problems. > The include file was also written by the code generator and for > extension modules was just a bunch of macros assigning the proper > function pointer to the correct name. That brings up another bit, > however. At some point I would like to break the include file into > two parts, one for inclusion in the other numpy modules and another > for inclusion in extension modules, the big #ifdef in the current > file offends my sense of esthetics. It should also be possible to > attach the function pointers to real function prototype like > declarations, which would help extension modules check the code at > compile time. No, the order is necessary for binary compatibility. If PyArray_API [3] points to function 'A', and PyArray_API[4] points to function 'B', then, if A and B are reversed in a newer version, any extension module compiled with the previous version will now call function 'B' instead of 'A', and vice versa. Adding functions to the end is ok, though. Instead of using an array, we could instead use a large struct, whose members are of right type as the function assigned to them, as in struct PyArray_API_t { PyObject *(*transpose)(PyArrayObject *, PyArray_Dims *); PyObject *(*take_from)(PyArrayObject *, PyObject *, int, PyArrayObject *, NPY_CLIPMODE); } struct PyArray_API_t PyArray_API = { PyArray_Transpose, PyArray_TakeFrom, } #define PyArray_Transpose (PyArray_API->transpose) This would give us better type-checking when compiling, and make it easier when running under gdb (when your extension crashes when calling into numpy, gdb would report the function as something like PyArray_API[31], because that's all the information it has). We would still have to guarantee the order for binary compability. One problem is that you'll have to make sure that the alignment of the fields doesn't change either (something that's not a problem for an array of pointers). Now, I was going to try to remove the order requirement, but never got around to it (you can see some of the initial work in numpy/core/ code_generators/genapi.py in the api_hash() routines). The idea is to have a unique identifier for each function (I use a hash of the name and the arguments, but for this example, let's just use the function name). An extension module, when compiled, would have a list of function names in the order it expects. In the import_array(), it would call numpy to give it the addresses corresponding to those names. As Python code, the above would look something like this: COMPILED_WITH_NUMPY_VERSION="1.2" API_names = ["PyArray_Transpose", "PyArray_TakeFrom"] def import_array(): global API API = numpy.get_c_api(API_names, COMPILED_WITH_NUMPY_VERSION) def a_routine(): API[3](an_array) (Of course, that'd have to be translated to C.) numpy.get_c_api would be responsible for putting things in the order that the module expects. One advantage of this method is that numpy.get_c_api can worry about how to be compatible with previous versions, if things change more than just adding functions. For instance, supplying different versions of functions becomes possible. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Chris.Barker at noaa.gov Wed Jan 10 18:19:07 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 10 Jan 2007 15:19:07 -0800 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> Message-ID: <45A5746B.4080002@noaa.gov> A. M. Archibald wrote: > Why not write the algorithm in C? I did just that a while back, for Numeric. I've enclosed the code for reference. Unfortunately, I never did figure out an efficient way to write this sort of thing for all types, so it only does doubles. Also, it does a bunch of special casing for discontiguous vs. contiguous arrays, and clipping to an array vs a scaler for the min and max arguments. Do what you will with the code... look for NumericExtras_fastclip in with the rest of the stuff in there -- much of which is now obsolete -- I haven't touched this is a few years. -Chris NOTE: is there a clean and efficient way to write custom functions of this sort for all types and both contiguous and discontiguous arrays? I guess I'm imaging some smart iterators and a micro version of C++ templates -- or maybe use C++ templates themselves. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: NumericExtras.c URL: From charlesr.harris at gmail.com Wed Jan 10 18:24:38 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 10 Jan 2007 16:24:38 -0700 Subject: [Numpy-discussion] Exported symbols and code reorganization. In-Reply-To: <0374ED3D-6DAE-419B-AF2B-AD2B965D2211@physics.mcmaster.ca> References: <0374ED3D-6DAE-419B-AF2B-AD2B965D2211@physics.mcmaster.ca> Message-ID: On 1/10/07, David M. Cooke wrote: > > On Jan 10, 2007, at 13:52 , Charles R Harris wrote: > > On 1/10/07, David M. Cooke wrote: On > > Jan 7, 2007, at 00:16 , Charles R Harris wrote: > > > > > > That brings up the main question I have about how to break up the C > > > files. I note that most of the functions in multiarraymodule.c, for > > > instance, are part of the C-API, and are tagged as belonging to > > > either the MULTIARRAY_API or the OBJECT_API. Apparently the build > > > system scans for these tags and extracts the files somewhere. So, > > > what is this API, is it available somewhere or is the code just > > > copied somewhere convenient. As to breaking up the files, the scan > > > only covers the code in the two current files, included code from > > > broken out parts is not seen. This strikes me as a bit of a kludge, > > > but I am sure there is a reason for it. Anyway, I assume the build > > > system can be fixed, so that brings up the question of how to break > > > up the files. The maximal strategy is to make every API functions, > > > with it's helper functions, a separate file. This adds a *lot* of > > > files, but it is straight forward and modular. A less drastic > > > approach is to start by breaking multiarraymodule into four files: > > > the converters, the two apis, and the module functions. My own > > > preference is for the bunch of files, but I suspect some will > > object. > > > > The code for pulling out the ``MULTIARRAY_API`` and ``OBJECT_API`` > > (also ``UFUNC_API``) is in ``numpy/core/code_generators``. Taking > > ``MULTIARRAY_API`` as an example, the ``generate_array_api.py`` is > > run by the ``numpy/core/setup.py`` file to generate the multiarray > > (and object) API. The file ``numpy/core/code_generators/ > > array_api_order.txt`` is the order in which the API functions are > > added to the ``PyArray_API`` array; this is our guarantee that the > > binary API doesn't change when functions are added. The files scanned > > are listed ``in numpy/core/code_generators/genapi.py``, which is also > > the module that does the heavy lifting in extracting the tagged > > functions. > > > > Looked to me like the order could change without causing problems. > > The include file was also written by the code generator and for > > extension modules was just a bunch of macros assigning the proper > > function pointer to the correct name. That brings up another bit, > > however. At some point I would like to break the include file into > > two parts, one for inclusion in the other numpy modules and another > > for inclusion in extension modules, the big #ifdef in the current > > file offends my sense of esthetics. It should also be possible to > > attach the function pointers to real function prototype like > > declarations, which would help extension modules check the code at > > compile time. > > No, the order is necessary for binary compatibility. If PyArray_API > [3] points to function 'A', and PyArray_API[4] points to function > 'B', then, if A and B are reversed in a newer version, any extension > module compiled with the previous version will now call function 'B' > instead of 'A', and vice versa. Adding functions to the end is ok, > though. Well, that is true if we don't want extension modules to need recompiling against new releases. However, I would like to replace the macros which are compiled to offsets in the API with actual functions, something like Py_Object* (*apifunc)(Py_Object *, blah, blah); Then the initialization function, instead of just making a copy of the API table pointer, could actually initialize the functions, i.e., apifunct = c_api[10]; and there would be something like a function prototype. I would also like to replace the generated include file for numpy modules with an actual permanent include so that the interface would be specified by function prototypes rather than generated from function definitions, but I suppose the current method isn't all that different. I have a question about the _import_array function _import_array(void) { PyObject *numpy = PyImport_ImportModule("numpy.core.multiarray"); PyObject *c_api = NULL; if (numpy == NULL) return -1; c_api = PyObject_GetAttrString(numpy, "_ARRAY_API"); if (c_api == NULL) {Py_DECREF(numpy); return -1;} if (PyCObject_Check(c_api)) { PyArray_API = (void **)PyCObject_AsVoidPtr(c_api); } Py_DECREF(c_api); Py_DECREF(numpy); Is there a reason that the pointer PyArray_API remains valid if the imported numpy module is garbage collected? Or is the imported module persistent even though the returned object goes away? I assume that python did a dlopen somewhere. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Jan 10 18:29:07 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 10 Jan 2007 17:29:07 -0600 Subject: [Numpy-discussion] Exported symbols and code reorganization. In-Reply-To: References: <0374ED3D-6DAE-419B-AF2B-AD2B965D2211@physics.mcmaster.ca> Message-ID: <45A576C3.5040601@gmail.com> Charles R Harris wrote: > Is there a reason that the pointer PyArray_API remains valid if the > imported numpy module is garbage collected? Or is the imported module > persistent even though the returned object goes away? I assume that > python did a dlopen somewhere. There is no provision in Python for safely unloading a C extension module, so we simply presume that sys.modules always holds a reference to the imported module and doesn't let it get garbage collected. People who futz with that get the segfaults they deserve. ;-) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Wed Jan 10 18:37:52 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 10 Jan 2007 16:37:52 -0700 Subject: [Numpy-discussion] Exported symbols and code reorganization. In-Reply-To: <45A576C3.5040601@gmail.com> References: <0374ED3D-6DAE-419B-AF2B-AD2B965D2211@physics.mcmaster.ca> <45A576C3.5040601@gmail.com> Message-ID: On 1/10/07, Robert Kern wrote: > > Charles R Harris wrote: > > > Is there a reason that the pointer PyArray_API remains valid if the > > imported numpy module is garbage collected? Or is the imported module > > persistent even though the returned object goes away? I assume that > > python did a dlopen somewhere. > > There is no provision in Python for safely unloading a C extension module, > so we > simply presume that sys.modules always holds a reference to the imported > module > and doesn't let it get garbage collected. People who futz with that get > the > segfaults they deserve. ;-) Ah, so that's why reload fails for C extension modules. Can't let the links change either. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at marquardt.sc Wed Jan 10 18:45:07 2007 From: christian at marquardt.sc (Christian Marquardt) Date: Thu, 11 Jan 2007 00:45:07 +0100 Subject: [Numpy-discussion] Docstring standards for NumPy and SciPy In-Reply-To: <45A523F3.3070408@ee.byu.edu> References: <45A523F3.3070408@ee.byu.edu> Message-ID: <1168472707.11383.14.camel@medea.marquardt.sc> Very nice! > """ > one-line summary not using variable names or the function name I would personally prefer that the first name contains the function (but nor arguments) - that way, when you are scrolling back in your terminal, or have a version printed out, you know what function/method the doc string belongs to without having to refer to examples lower down. > A few sentences giving an extended description. After the general description, but before giving the inputs and outputs, wouldn't it make sense to give the function signature as well? Something like named, list, of, outputs = my_function(var1, variable2 [,kwdarg1=...]) This would again reduce the need to have an extra look into the example section. Using the brackets, optional arguments and default settings could also be communicated easily. > Inputs: > var1 -- Explanation > variable2 -- Explanation [...skipped a bit...] > Notes: > Additional notes if needed > Authors: > name (date): notes about what was done > name (date): major documentation updates can be included here also. I'm all for mentioning the authors (certainly in the main docstrings of a module, say). *But* I would put that information at the very end of a docstring - when reading documentation, I'm primarily interested in the usage information, examples, the 'See also' and further notes or comments. Having to skip author names and even a (possibly long) list of modifications somewhere in the middle of the docstring, i.e. before getting to the that information I'm looking for each time seems like an annoyance (to me) that is not really necessary. Hope this helps / is useful, Christian. From oliphant at ee.byu.edu Wed Jan 10 18:50:41 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 10 Jan 2007 16:50:41 -0700 Subject: [Numpy-discussion] Docstring standards for NumPy and SciPy In-Reply-To: <1168472707.11383.14.camel@medea.marquardt.sc> References: <45A523F3.3070408@ee.byu.edu> <1168472707.11383.14.camel@medea.marquardt.sc> Message-ID: <45A57BD1.2080501@ee.byu.edu> Christian Marquardt wrote: >Very nice! > > > >>""" >>one-line summary not using variable names or the function name >> >> > >I would personally prefer that the first name contains the function (but >nor arguments) - that way, when you are scrolling back in your terminal, >or have a version printed out, you know what function/method the >doc string belongs to without having to refer to examples lower down. > > > >After the general description, but before giving the inputs and outputs, >wouldn't it make sense to give the function signature as well? Something >like > > named, list, of, outputs = my_function(var1, variable2 [,kwdarg1=...]) > >This would again reduce the need to have an extra look into the example >section. Using the brackets, optional arguments and default settings >could also be communicated easily. > > Both of these are normally not given because they are available from introspection, and most help facilities will print them out before the actual docstring. -Travis From robert.kern at gmail.com Wed Jan 10 18:53:31 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 10 Jan 2007 17:53:31 -0600 Subject: [Numpy-discussion] Docstring standards for NumPy and SciPy In-Reply-To: <1168472707.11383.14.camel@medea.marquardt.sc> References: <45A523F3.3070408@ee.byu.edu> <1168472707.11383.14.camel@medea.marquardt.sc> Message-ID: <45A57C7B.2020001@gmail.com> Christian Marquardt wrote: > Very nice! > >> """ >> one-line summary not using variable names or the function name > > I would personally prefer that the first name contains the function (but > nor arguments) - that way, when you are scrolling back in your terminal, > or have a version printed out, you know what function/method the > doc string belongs to without having to refer to examples lower down. How are you viewing the docstrings that wouldn't associate the docstring with the function? >> A few sentences giving an extended description. > > After the general description, but before giving the inputs and outputs, > wouldn't it make sense to give the function signature as well? Something > like > > named, list, of, outputs = my_function(var1, variable2 [,kwdarg1=...]) > > This would again reduce the need to have an extra look into the example > section. Using the brackets, optional arguments and default settings > could also be communicated easily. Except for functions defined in C extension modules, the function signature is introspectable. All of the good ways of viewing docstring information already extract this data (help(), pydoc, epydoc, IPython's ? operator). Again, how are you viewing docstrings that you don't see this information already? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From christian at marquardt.sc Wed Jan 10 19:13:54 2007 From: christian at marquardt.sc (Christian Marquardt) Date: Thu, 11 Jan 2007 01:13:54 +0100 Subject: [Numpy-discussion] Docstring standards for NumPy and SciPy In-Reply-To: <45A57C7B.2020001@gmail.com> References: <45A523F3.3070408@ee.byu.edu> <1168472707.11383.14.camel@medea.marquardt.sc> <45A57C7B.2020001@gmail.com> Message-ID: <1168474434.11383.19.camel@medea.marquardt.sc> On Wed, 2007-01-10 at 17:53 -0600, Robert Kern wrote: > How are you viewing the docstrings that wouldn't associate the docstring with > the function? print .__doc__ Like so: Python 2.4 (#1, Mar 22 2005, 21:42:42) [GCC 3.3.5 20050117 (prerelease) (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> def squared(x): ... """My docstring without function signature.""" ... return x*x ... >>> print squared.__doc__ My docstring without function signature. Sorry, I wasn't aware that help() prints the signature; I just don't use it very often... I guess you have a point there. Chris. From bthom at cs.hmc.edu Wed Jan 10 20:35:07 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Wed, 10 Jan 2007 17:35:07 -0800 Subject: [Numpy-discussion] [Matplotlib-users] installing numpy, matplotlib, scipy from source on a Mac In-Reply-To: <45A5716F.2060309@noaa.gov> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> <3d375d730701080252m35568b63q1bf758c590655594@mail.gmail.com> <45A5716F.2060309@noaa.gov> Message-ID: <0E8A3134-6997-4F90-8A2D-7ED5FA922EE0@cs.hmc.edu> Hi Chris, Robert, ...?, I'm glad you reminded me to make sure the path was right, b/c this time, I had forgotten to run the "Update Shell" command that came w/ MacPython2.4. This gave me: lrwxr-xr-x 1 root wheel 67 Jan 7 14:29 /usr/local/bin/pydoc@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/pydoc lrwxr-xr-x 1 root wheel 70 Jan 7 14:29 /usr/local/bin/pydoc2.4@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/ pydoc2.4 lrwxr-xr-x 1 root wheel 68 Jan 7 14:29 /usr/local/bin/python@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/python lrwxr-xr-x 1 root wheel 71 Jan 7 14:29 /usr/local/bin/python2.4@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/ python2.4 lrwxr-xr-x 1 root wheel 69 Jan 7 14:29 /usr/local/bin/pythonw@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/pythonw lrwxr-xr-x 1 root wheel 72 Jan 7 14:29 /usr/local/bin/pythonw2.4@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/ pythonw2.4 lrwxr-xr-x 1 root wheel 70 Jan 7 14:29 /usr/local/bin/smtpd.py@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/ smtpd.py lrwxr-xr-x 1 root wheel 73 Jan 7 14:29 /usr/local/bin/smtpd2.4.py@ - > ../../../Library/Frameworks/Python.framework/Versions/2.4/bin/ smtpd2.4.py Nevertheless, the python I was using in the prior email looks like the same as what I'm using w/these path updates, that is, assuming version strings are enough to know: From prior run: Python 2.4.4 (#1, Oct 18 2006, 10:34:39) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin New run: Python 2.4.4 (#1, Oct 18 2006, 10:34:39) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin As you'd suspected, it is indeed the case that: diff pythonw python reports no differences. So, I went back and retried the plotting w/wx as a backend and discovered that wx FAILS with PYTHONW and PYTHON (appended). If you saw how many notes I've collected on the different installs I've tried you'd laugh (except that its not funny). Its no wonder I can't keep all this stuff straight. (When I first embarked on this exercise several months ago, I tried macports; everything seemed to work fine, but the Wx backend failed---I remember something about a bitmap, so it was likely the same error I'm having now; and unfortunately w/the macports python, the tk/tcl stuff didn't come wrapped, so TkAgg was not an option). After an email w/Robert, I switched to MacPython. As a result, I now have TkAgg working (and its using Aqua instead of that awful, klunky looking Tk stuff). Fortunately (ha!), I have three versions of this stuff installed on 3 different Macs (a G4, a G5, and an Intel). So I can say with safety that, using TkAgg backend, here's what's still not working: 1) Using macpython matplotlib, numpy, and scipy packages, you can't run scipy.test(); complaint: RuntimeError: module compiled against version 1000002 of C-API but this version of numpy is 1000009 2) Using superpak, everything (except ipython) worked out of the box, PROVIDED I had already installed the necessary fortran pieces (I used g77v3.4-bin.tar.gz, found somewhere on the scipy site). 3) Using Robert's install-from-source method (this install used gfortran.bin.tar.gz, with the caveat that the expected WXAgg doesn't work) Machines 2 and 3 are enough to get me back to the real work I have to do. But it is obvious there's several real issues for Mac users who wish to use the scipy/numpy/matplotlib route. This coming summer I may have more time to devote to this stuff, and since it really irks me that there isn't a clean way to do this, I may end up figuring out how to make the needed binaries myself. In the meantime, I thought I'd never wish I was still using Windows... Thanks to everyone for the time spent on this. And for posterity's sake, when you use matplotlib with Numeric, it also crashes. I've verified that behavior on Machines 1 and 2 above. --b 54 % pythonw Python 2.4.4 (#1, Oct 18 2006, 10:34:39) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. history mechanism set up >>> import pylab as P >>> import matplotlib as M >>> M.rcParams['interactive'] True >>> M.rcParams['backend'] 'WXAgg' >>> P.plot([1,2,3]) Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/matplotlib/backends/backend_wx.py", line 1048, in _onPaint self.draw(repaint=False) draw_if_interactive() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/matplotlib/backends/backend_wx.py", line 1172, in draw_if_interactive figManager.canvas.draw() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/matplotlib/backends/backend_wxagg.py", line 63, in draw self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None) MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create the wx.Bitmap >> 55 % python Python 2.4.4 (#1, Oct 18 2006, 10:34:39) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. history mechanism set up >>> import pylab as P >>> import matplotlib as M >>> M.rcParams['interactive'] True >>> M.rcParams['backend'] 'WXAgg' >>> P.plot([1,2,3]) Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/matplotlib/backends/backend_wx.py", line 1048, in _onPaint self.draw(repaint=False) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/matplotlib/backends/backend_wxagg.py", line 63, in draw self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None) MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create the wx.Bitmap Traceback (most recent call last): File "", line 1, in ? File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/matplotlib/pylab.py", line 2022, in plot File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/matplotlib/backends/backend_wx.py", line 1172, in draw_if_interactive figManager.canvas.draw() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/matplotlib/backends/backend_wxagg.py", line 63, in draw self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), None) MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create the wx.Bitmap >>> On Jan 10, 2007, at 3:06 PM, Christopher Barker wrote: > > > Robert Kern wrote: > >> Try running with pythonw. > > That's probably not it -- as of MacPython 2.4, pythonw ands python > are the same. > > belinda thom wrote: >> And running w/pythonw does what it should :-). > > OK, now I'm confused: > > $ ls -l /Library/Frameworks/Python.framework/Versions/2.4/bin/ > python2.4 > -rwxrwxr-x 1 root admin 39936 Apr 7 2006 /Library/Frameworks/ > Python.framework/Versions/2.4/bin/python2.4 > > $ ls -l /Library/Frameworks/Python.framework/Versions/2.4/bin/ > pythonw2.4 > -rwxrwxr-x 1 root admin 39936 Apr 7 2006 /Library/Frameworks/ > Python.framework/Versions/2.4/bin/pythonw2.4 > > Those two look like the same binaries to me -- and diff tells me > they are. > > Are you sure you're running the same python with "python" and > "pythonw"? Try running them on the command line alone and see what > you get. > >> File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ >> python2.4/site-packages/matplotlib/backends/backend_wxagg.py", >> line 63, in draw >> self.bitmap = _convert_agg_to_wx_bitmap(self.get_renderer(), >> None) >> MemoryError: _wxagg.convert_agg_to_wx_bitmap(): could not create >> the wx.Bitmap > > This looks like the error we usually get when you've built the > wxAgg extension against a different version of wx than the one > you're running. That's easy to do, as Apple has provided an old wx > with it's Python, it is often found by default by the MPL build > process. > > Search this list for the way to fix that, or, if you really can't > find it, I'll dig it up. > > -Chris > > > > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov From tim.hochberg at ieee.org Wed Jan 10 22:18:57 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 10 Jan 2007 20:18:57 -0700 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> <45A06A74.3090001@ee.byu.edu> <45A162C1.6020706@literati.org> Message-ID: On 1/8/07, Timothy Hochberg wrote: > > > > On 1/7/07, Sean R. Lynch < seanl at literati.org> wrote: > > > > Travis Oliphant wrote: > > > > > I don't think we could make it work as he expects (and not radically > > > change NumPy itself so that other operations are very different) > > because > > > of the limitations of Python. > > > > > > The point was that it is too complicated to do any differently (and is > > > probably impossible). > > > > Ok, thanks for the explanation. > > > > In that case, is there a way to do what I want (accumulate values per > > index) without looping in Python? The histogram functions would only > > count the number of uses of a given index AFAICT. I want to sum all of > > the normals for a given index. > > > Sean, > > I'm pretty sure that at one point I had a way to do exactly what you are > doing, however it's been a while and I don't know where that code wandered > off to. I will think about it now that I'm done doing silly stuff and see if > I can recall what the trick was. > > [Just don't want you to feel abandoned...] > OK, *now* you can feel abandoned. I've thought about this for a couple of days and I can't see an efficient way to do it with the current set of primitives. I think that when I did normal blending before I might have been working on rectangular grid so every vertex had three neighbors or something like that. You're the second or third person to recently come up with a useful application for using inplace ops with repeated indices only to be stymied by that fact that they don't actually work that way. I've been wondering if we should try to come up with a way to support this usage. Although it might be feasible to make += work, the implementation is horrible, and there are all sorts of inefficiencies built into that method of doing it. The natural approach would seem to be to hang another method off the ufuncs. That way you get support for all of the various operations. I was thinking something like: unfunc.inplace (dest, indices, source) For example, for the add ufunc, this would be more or less equivalent to: for i, ind in enumerate(indices): dest[ind] += source[i] The main exception being that source would be broadcast appropriately so that add.inplace(dest, indices, 1), for example, will count how many times each index appears. I'm not completely convinced that this is necessary, nor that this is the way to approach it if it is. But I figured I'd throw it out onto the aether while I was thinking of it. -- //=||=\\ -------------- next part -------------- An HTML attachment was scrubbed... URL: From frbeaxs at earthlink.net Wed Jan 10 22:29:32 2007 From: frbeaxs at earthlink.net (frbeaxs) Date: Wed, 10 Jan 2007 19:29:32 -0800 Subject: [Numpy-discussion] Scipy - Numpy incompatiblility when calling upon Old Numeric Message-ID: Matplotlib ver 0.87.4. was the problem. An update to matplotlib-0.87.7 was all that was necessary. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jan 10 22:36:58 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 10 Jan 2007 20:36:58 -0700 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> <45A06A74.3090001@ee.byu.edu> <45A162C1.6020706@literati.org> Message-ID: On 1/10/07, Timothy Hochberg wrote: > > > > On 1/8/07, Timothy Hochberg wrote: > > > > > > > > On 1/7/07, Sean R. Lynch < seanl at literati.org> wrote: > > > > > > Travis Oliphant wrote: > > > > > > > I don't think we could make it work as he expects (and not radically > > > > change NumPy itself so that other operations are very different) > > > because > > > > of the limitations of Python. > > > > > > > > The point was that it is too complicated to do any differently (and > > > is > > > > probably impossible). > > > > > > Ok, thanks for the explanation. > > > > > > In that case, is there a way to do what I want (accumulate values per > > > index) without looping in Python? The histogram functions would only > > > count the number of uses of a given index AFAICT. I want to sum all of > > > the normals for a given index. > > > > > > Sean, > > > > I'm pretty sure that at one point I had a way to do exactly what you are > > doing, however it's been a while and I don't know where that code wandered > > off to. I will think about it now that I'm done doing silly stuff and see if > > I can recall what the trick was. > > > > [Just don't want you to feel abandoned...] > > > > > OK, *now* you can feel abandoned. I've thought about this for a couple of > days and I can't see an efficient way to do it with the current set of > primitives. I think that when I did normal blending before I might have been > working on rectangular grid so every vertex had three neighbors or something > like that. > > You're the second or third person to recently come up with a useful > application for using inplace ops with repeated indices only to be stymied > by that fact that they don't actually work that way. I've been wondering if > we should try to come up with a way to support this usage. Although it might > be feasible to make += work, the implementation is horrible, and there are > all sorts of inefficiencies built into that method of doing it. > > The natural approach would seem to be to hang another method off the > ufuncs. That way you get support for all of the various operations. I was > thinking something like: > > unfunc.inplace (dest, indices, source) > Yeah, something like that. A general indirect addressing primitive of some sort would be useful for these situations. The indices are sort of pointers into the destination, the problem is how to bind this to operators. It should almost be some sort lvalue analogous to **p = whatever, or in this case dest[index[j]] {assignment} src[j]. So maybe indirect(dest, indices, source, op). Then there is all the madness of fancy indexing (which I think I could have done without). Because the current applications are limited and could be met with the simplest 1D version, it would probably be best to start with the most basic functionality and then extend it if the need arose. Or maybe even some simple c extension that did the whole basic += thing for one dimension. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Wed Jan 10 23:18:10 2007 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Wed, 10 Jan 2007 23:18:10 -0500 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> <45A06A74.3090001@ee.byu.edu> <45A162C1.6020706@literati.org> Message-ID: On 10/01/07, Charles R Harris wrote: > > You're the second or third person to recently come up with a useful > application for using inplace ops with repeated indices only to be stymied > by that fact that they don't actually work that way. I've been wondering if > we should try to come up with a way to support this usage. Although it might > be feasible to make += work, the implementation is horrible, and there are > all sorts of inefficiencies built into that method of doing it. > > > > The natural approach would seem to be to hang another method off the > ufuncs. That way you get support for all of the various operations. I was > thinking something like: > > > > > > unfunc.inplace (dest, indices, source) > > Yeah, something like that. A general indirect addressing primitive of some > sort would be useful for these situations. The indices are sort of pointers > into the destination, the problem is how to bind this to operators. It > should almost be some sort lvalue analogous to **p = whatever, or in this > case dest[index[j]] {assignment} src[j]. So maybe indirect(dest, indices, > source, op). Then there is all the madness of fancy indexing (which I think > I could have done without). Because the current applications are limited and > could be met with the simplest 1D version, it would probably be best to > start with the most basic functionality and then extend it if the need > arose. Or maybe even some simple c extension that did the whole basic += > thing for one dimension. I disagree that it should be hung off ufuncs. That doesn't allow, for example, averaging of normal vectors (without some highly clever and hard-to-follow axis specification). I implemented one possible version in pure python, just to think about the UI (and to see if it could be made fast using sorting; I don't think it will actually help much, and I also note that whether it even has any hope depends on whether there are more objects to accumulate than destination slots, or fewer). The basic design is def accumulate(dest, indices, values, combine=N.add); indices should be a one-dimensional list of indices, though since dest can be an n-dimensional array of r-dimensional values, it's actually a two-dimensional array; values is a list of r-dimensional values, and combine should take two r-dimensional values and yield another. The reason for the r-dimensional values is to allow, for example, averaging of normals, or combination in some other way of colours, or summing of vectors (my original application). This implementation does not allow multidimensional arrays of indices, partly because it would be cumbersome to iterate over them, partly because a reshape (and possible copy) is not terribly offensive, and partly because it would make the calling convention (more) confusing. I am not seriously suggesting this implementation for use, but for consideration; if people find the primitive does what they want, one could reasonably simply implement it in C. A. M. Archibald -------------- next part -------------- A non-text attachment was scrubbed... Name: accumulate.py Type: text/x-python Size: 1397 bytes Desc: not available URL: From torgil.svensson at gmail.com Wed Jan 10 23:28:52 2007 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Thu, 11 Jan 2007 05:28:52 +0100 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <459D6964.30701@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> Message-ID: On 1/4/07, Travis Oliphant wrote: > > still be used to describe a complicated block of memory to another user. Thinking of the scope "seamless data exchange between modules" my concern with this PEP is that is might be too much focused on "block of memory" rather than "access to data". Data that can be interpreted as an n-dimensional array doesn't necessarily has to be represented directly as a block of memory. Example1: We have a very large amount of data with a compressed internal representation Example2: We might want to generate data "on the fly" as it's needed Example3: If module creators to deal with different byte alignments, contiguousness etc it'll lead to lots of code duplication and unnecessarily much work Is it possible to add a data access API to this PEP? Direct memory access could be available through this API with a function that return the memory address (or NULL if not available). We could have a default implementation for basic types with the option for module creators to override this. The problem with this, if we stick to the buffer protocol, is that it breaks the concept "buffer is memory" if that ever was a valid. This is of minor concern for me though. From david at ar.media.kyoto-u.ac.jp Thu Jan 11 00:03:15 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 11 Jan 2007 14:03:15 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <20070110214923.GP16625@mentat.za.net> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> Message-ID: <45A5C513.6070701@ar.media.kyoto-u.ac.jp> Stefan van der Walt wrote: > On Wed, Jan 10, 2007 at 08:28:14PM +0100, Francesc Altet wrote: >> El dt 09 de 01 del 2007 a les 23:19 +0900, en/na David Cournapeau va >> escriure: >>> Hi, >>> >>> I am finally implementing a C function to replace the current slow >>> implementation of clip in python as promised a few weeks ago. The idea >>> is to implement it as the following: >>> >>> def clip(input, min, max): >>> a = input.copy() >>> putmask(a, a <= min, min) >>> putmask(a, a >= max, max) >>> return a >>> >>> I don't have that much experience in writing general numpy functions, so >>> I was wondering of other people could advise me on the following points. >>> >> Sorry, but not real experience writing extensions directly in C. >> However, you may want to experiment using numexpr for doing what you >> want. It's relatively easy to extend numexpr and adding a new opcode to >> its virtual machine. I'm attaching a patch for implementing such a clip >> routine (only for floating point types, but given the example, it should >> be straightforward to add support for integers as well). >> >> Also, you should note that using the fancy indexing of numpy seems to >> perform better than the putmask approach. Below are my figures for a >> small benchmark (also attached) for testing the performance of clip >> using several approaches: >> >> time (putmask)--> 1.38 >> time (where)--> 2.713 >> time (numexpr where)--> 1.291 >> time (fancy+assign)--> 0.967 >> time (numexpr clip)--> 0.596 >> >> It is interesting to see there how fancy-indexing + assignation is quite >> more efficient than putmask. > > Not on my machine: > > time (putmask)--> 0.181 > time (where)--> 0.783 > time (numexpr where)--> 0.26 > time (fancy+assign)--> 0.202 When I started looking at those things, I did the indexing method, and someone else proposed putmask, function which I was not aware of at that time. Both are similar in speed, and vastly (almost an order of magnitude on moderately sized contiguous arrays) faster than the current numpy clip. My current C implementation does not use the equivalent of putmask. I try to determine which case are easy (basically, if the datatype is a numeric datatype and native endian), handle those directly, and for other cases (non native endian, objects, etc...), simply forwarding to the original function for now. The main difficulty is that I am not aware of all the datatypes that numpy functions are supposed to handle (for example, when I started, I didn't know that numpy could handle non native endian, which makes things a bit more complicated in C to support). cheers, David From lists.steve at arachnedesign.net Thu Jan 11 00:11:11 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Thu, 11 Jan 2007 00:11:11 -0500 Subject: [Numpy-discussion] Docstring standards for NumPy and SciPy In-Reply-To: <45A523F3.3070408@ee.byu.edu> References: <45A523F3.3070408@ee.byu.edu> Message-ID: <40545341-21EC-4EC9-934C-7EBD57956C78@arachnedesign.net> Hi, Overall I like your train of thought. If we're going use a modified take on the structured text: > Use *italics*, **bold**, and `courier` if needed in any explanations > (but not for variable names and doctest code or multi-line code) I've always been a fan of using /'s for italics, and single *'s for bold. Like, I /really/ liked it. Didn't you? ... well ... *didn't* you? -steve From david at ar.media.kyoto-u.ac.jp Thu Jan 11 00:41:55 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 11 Jan 2007 14:41:55 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A5746B.4080002@noaa.gov> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> Message-ID: <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> Christopher Barker wrote: > > > A. M. Archibald wrote: > >> Why not write the algorithm in C? > > I did just that a while back, for Numeric. I've enclosed the code for > reference. > > Unfortunately, I never did figure out an efficient way to write this > sort of thing for all types, so it only does doubles. Also, it does a > bunch of special casing for discontiguous vs. contiguous arrays, and > clipping to an array vs a scaler for the min and max arguments. To do the actual clipping if the datatypes are 'native' is trivial in C: a single loop, a comparison, that's it. I have become an irrational C++ hater with the time, so I don't use template (and I don't think C++ is welcomed in core numpy). For those easy case of template use, autogen works well enough for me; my impression is that numpy uses a similar system, eg for ufunc, etc... You can look at scipy/Lib/sandbox/cdavid/src/levinson1d.tpl for an example of autogen to generate function for any datatype you want; if you need more fancy template facilities like partial specialization and other crazy things mere mortals like me will never understand in C++, then I am not sure autogen can be used. I guess the method used in numpy is better to use for core functionalities, as it avoids the burden of installing one more tool for development. Now, I didn't know that clip was supposed to handle arrays as min/max values. At first, I didn't understand the need to care about contiguous/non contiguous; having non scalar for min/max makes it necessary to have special case for non contiguous. But again, it is important not to lose sight... The goal was to have faster clipping for matplotlib, and this cases are easy, because it is native type and scalar min/max, where contiguous or not does not matter as we traverse the input arrays element by element. If we pass non native endian, non contiguous arrays, there is actually a pretty good chance that the current implementation is already fast enough, and does not need to be changed anyway. Thanks for the suggestion and the precisions, David From faltet at carabos.com Thu Jan 11 08:09:56 2007 From: faltet at carabos.com (Francesc Altet) Date: Thu, 11 Jan 2007 14:09:56 +0100 Subject: [Numpy-discussion] =?iso-8859-1?q?Need_help_for_implementing_a_fa?= =?iso-8859-1?q?st_clip_in=09numpy_=28was_slow_clip=29?= In-Reply-To: <20070110214923.GP16625@mentat.za.net> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> Message-ID: <200701111409.58652.faltet@carabos.com> A Dimecres 10 Gener 2007 22:49, Stefan van der Walt escrigu?: > On Wed, Jan 10, 2007 at 08:28:14PM +0100, Francesc Altet wrote: > > El dt 09 de 01 del 2007 a les 23:19 +0900, en/na David Cournapeau va > > > > escriure: > > time (putmask)--> 1.38 > > time (where)--> 2.713 > > time (numexpr where)--> 1.291 > > time (fancy+assign)--> 0.967 > > time (numexpr clip)--> 0.596 > > > > It is interesting to see there how fancy-indexing + assignation is quite > > more efficient than putmask. > > Not on my machine: > > time (putmask)--> 0.181 > time (where)--> 0.783 > time (numexpr where)--> 0.26 > time (fancy+assign)--> 0.202 Yeah, a lot of difference indeed. Just for reference, my results above were done using a Duron (an Athlon but with only 128 KB of secondary cache) at 0.9 GHz. Now, using my laptop (Intel 4 @ 2 GHz, 512 KB of secondary cache), I get: time (putmask)--> 0.244 time (where)--> 2.111 time (numexpr where)--> 0.427 time (fancy+assign)--> 0.316 time (numexpr clip)--> 0.184 so, on my laptop fancy+assign is way slower than putmask. It should be noted also that the implementation of clip in numexpr (i.e. in pure C) is not that much faster than putmask (just a 30%); so perhaps it is not so necessary to come up with a pure C implementation for clip (or at least, on Intel P4 machines!). In any case, it is really shocking seeing how differently can perform the several CPU architectures on this apparently simple problem. BTW, I'm attaching a slightly enhanced version of the clip patch for numexpr that I used for the new benchmark showed here. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" -------------- next part -------------- A non-text attachment was scrubbed... Name: numexpr-clip2.patch Type: text/x-diff Size: 2126 bytes Desc: not available URL: From steve at shrogers.com Thu Jan 11 09:08:16 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Thu, 11 Jan 2007 07:08:16 -0700 Subject: [Numpy-discussion] Docstring standards for NumPy and SciPy In-Reply-To: <45A523F3.3070408@ee.byu.edu> References: <45A523F3.3070408@ee.byu.edu> Message-ID: <45A644D0.80905@shrogers.com> I'd prefer reStructuredText. I don't find the markup particularly noisy and it quickly fades into the background. Regards, Steve -- Steven H. Rogers, Ph.D., steve at shrogers.com Weblog: http://shrogers.com/weblog "He who refuses to do arithmetic is doomed to talk nonsense." -- John McCarthy From oliphant at ee.byu.edu Thu Jan 11 09:49:10 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 11 Jan 2007 07:49:10 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> Message-ID: <45A64E66.3080801@ee.byu.edu> Torgil Svensson wrote: > On 1/4/07, Travis Oliphant wrote: > >> still be used to describe a complicated block of memory to another user. >> > > Thinking of the scope "seamless data exchange between modules" my > concern with this PEP is that is might be too much focused on "block > of memory" rather than "access to data". Data that can be interpreted > as an n-dimensional array doesn't necessarily has to be represented > directly as a block of memory. > > Example1: We have a very large amount of data with a compressed > internal representation > > Example2: We might want to generate data "on the fly" as it's needed > > Example3: If module creators to deal with different byte alignments, > contiguousness etc it'll lead to lots of code duplication and > unnecessarily much work > > Is it possible to add a data access API to this PEP? Could you give an example of what you mean? I have no problem with such a concept. I'm mainly interested in getting the NumPy memory model into Python some-how. I know it's not the "only" way to think about memory, but it is a widely-used and useful way. -Travis From tim.hochberg at ieee.org Thu Jan 11 09:53:39 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Thu, 11 Jan 2007 07:53:39 -0700 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> <45A06A74.3090001@ee.byu.edu> <45A162C1.6020706@literati.org> Message-ID: On 1/10/07, A. M. Archibald wrote: > > On 10/01/07, Charles R Harris wrote: > > > > You're the second or third person to recently come up with a useful > > application for using inplace ops with repeated indices only to be > stymied > > by that fact that they don't actually work that way. I've been wondering > if > > we should try to come up with a way to support this usage. Although it > might > > be feasible to make += work, the implementation is horrible, and there > are > > all sorts of inefficiencies built into that method of doing it. > > > > > > The natural approach would seem to be to hang another method off the > > ufuncs. That way you get support for all of the various operations. I > was > > thinking something like: > > > > > > > > > unfunc.inplace (dest, indices, source) > > > > Yeah, something like that. A general indirect addressing primitive of > some > > sort would be useful for these situations. The indices are sort of > pointers > > into the destination, the problem is how to bind this to operators. It > > should almost be some sort lvalue analogous to **p = whatever, or in > this > > case dest[index[j]] {assignment} src[j]. So maybe indirect(dest, > indices, > > source, op). Then there is all the madness of fancy indexing (which I > think > > I could have done without). Because the current applications are limited > and > > could be met with the simplest 1D version, it would probably be best to > > start with the most basic functionality and then extend it if the need > > arose. Or maybe even some simple c extension that did the whole basic += > > thing for one dimension. > > I disagree that it should be hung off ufuncs. That doesn't allow, for > example, averaging of normal vectors (without some highly clever and > hard-to-follow axis specification). What makes you think this? To go back to the start of the thread, the OP wanted the following to work: def calculate_normals(indices, vertices): i = indices faces = vertices[i].reshape((i.shape[0] / 3, 3, 3)) # indices = 3 x N? face_normals = cross(faces[:,1] - faces[:,0], faces[:,2] -faces[:,0]) normals = zeros(vertices.shape, float32) normals[i] += face_normals.repeat(3, axis=0) normals /= sqrt(sum(normals*normals,axis=1)).reshape(normals.shape[0], 1) return normals Hanging it off of ufuncs, would just mean that the line: normals[i] += face_normals.repeat(3, axis=0) would get replaced by this line: add.inplace(normals, i, face_normals.repeat(3, axis=0)) I implemented one possible version in pure python, just to think about > the UI (and to see if it could be made fast using sorting; I don't > think it will actually help much, and I also note that whether it even > has any hope depends on whether there are more objects to accumulate > than destination slots, or fewer). The basic design is > > def accumulate(dest, indices, values, combine=N.add); Unless I'm mistaken, semantically, this is no different than add.inplace as defined above, you've just moved stuff around. In practice, I think that hanging something like this off of ufuncs is likely to be the way to go to get decent performance. Plus, that's the traditional way, in numpy, to deal with methods that are the same except for the operator at the core of them. I'm not in love with the name in 'inplace' necessarily, but I don't think accumulate will really fly either, accumulate already has a different meaning that is close enough to be confusing, but far enough apart that I don't see a way to shoehorn them together. [SNIP] -- //=||=\\ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jensj at fysik.dtu.dk Thu Jan 11 10:07:07 2007 From: jensj at fysik.dtu.dk (Jens =?ISO-8859-1?Q?J=F8rgen?= Mortensen) Date: Thu, 11 Jan 2007 16:07:07 +0100 Subject: [Numpy-discussion] Docstring standards for NumPy and SciPy In-Reply-To: <45A644D0.80905@shrogers.com> References: <45A523F3.3070408@ee.byu.edu> <45A644D0.80905@shrogers.com> Message-ID: <1168528027.12152.100.camel@doppler.fysik.dtu.dk> On Thu, 2007-01-11 at 07:08 -0700, Steven H. Rogers wrote: > I'd prefer reStructuredText. I don't find the markup particularly noisy and > it quickly fades into the background. I found some notes on the docutils page (about using reStructuredText in docstring): http://docutils.sourceforge.net/docs/dev/enthought-plan.html#docstring- density-whitespace-minimization Don't know how relevant it is ... Jens J?rgen From david at ar.media.kyoto-u.ac.jp Thu Jan 11 10:49:27 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 12 Jan 2007 00:49:27 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <200701111409.58652.faltet@carabos.com> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <200701111409.58652.faltet@carabos.com> Message-ID: <45A65C87.2040200@ar.media.kyoto-u.ac.jp> Francesc Altet wrote: > A Dimecres 10 Gener 2007 22:49, Stefan van der Walt escrigu?: >> On Wed, Jan 10, 2007 at 08:28:14PM +0100, Francesc Altet wrote: >>> El dt 09 de 01 del 2007 a les 23:19 +0900, en/na David Cournapeau va >>> >>> escriure: >>> time (putmask)--> 1.38 >>> time (where)--> 2.713 >>> time (numexpr where)--> 1.291 >>> time (fancy+assign)--> 0.967 >>> time (numexpr clip)--> 0.596 >>> >>> It is interesting to see there how fancy-indexing + assignation is quite >>> more efficient than putmask. >> Not on my machine: >> >> time (putmask)--> 0.181 >> time (where)--> 0.783 >> time (numexpr where)--> 0.26 >> time (fancy+assign)--> 0.202 > > Yeah, a lot of difference indeed. Just for reference, my results above were > done using a Duron (an Athlon but with only 128 KB of secondary cache) at 0.9 > GHz. Now, using my laptop (Intel 4 @ 2 GHz, 512 KB of secondary cache), I > get: > > time (putmask)--> 0.244 > time (where)--> 2.111 > time (numexpr where)--> 0.427 > time (fancy+assign)--> 0.316 > time (numexpr clip)--> 0.184 > > so, on my laptop fancy+assign is way slower than putmask. It should be noted > also that the implementation of clip in numexpr (i.e. in pure C) is not that > much faster than putmask (just a 30%); so perhaps it is not so necessary to > come up with a pure C implementation for clip (or at least, on Intel P4 > machines!). > > In any case, it is really shocking seeing how differently can perform the > several CPU architectures on this apparently simple problem. I am not sure it is such a simple problem: it involves massive branching. I have never taken a look a numexpr, but the idea seems really interesting, I will take a look at it when I will have some time. Anyway, I've just finished and tested a pure C implementation of the clip function. As it is, it should be able to replace PyArray_Clip calls by PyArray_FastClip in numpy/core/multiarray.c. The idea is that for 'easy' cases, it uses a trivial but fast implementation; for all other cases, it uses the old implementation for now. By easy cases, I mean scalar min and max, for non-complex number with native endianness (from npy_bool to npy_longdouble), which should cover most usages. There are still some things I am unsure: - the original clip is supposed to work with complex numbers, but I am not sure about the semantics in this case. - If you have a float32 input, but float64 min/max values, the original clip does not upcast the input. If you have integer input but floating point min/max, the original clip fails. Is this the wanted behaviour ? My implementation upcasts whenever possible; but then, I am not sure how to handle the cases where no copy is asked (which I am not handling myself for now). As for now, when PyArray_FastClip uses a fast implementation, it is roughly 5x faster for float32 and float64 input. I expect a double speed once the no copy option is implemented (again, for easy cases). I attached blop.c which implements the fast clip in a module, the clip_imp.c which implements the clipping for all native types (it is generated by autogen because I wanted to avoid depending on numpy.distutils for development), a Makefile and a test file which also profile the clip function with float32 inputs. Does it look Ok to other so that it can be commited to numpy (once the two above problems are solved, of course, to keep the same behaviour than PyArray_Clip) ? cheers, David -------------- next part -------------- A non-text attachment was scrubbed... Name: blop.c Type: text/x-csrc Size: 9544 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: clip_imp.c Type: text/x-csrc Size: 11219 bytes Desc: not available URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Makefile URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_take.py Type: text/x-python Size: 9896 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Thu Jan 11 10:58:28 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 12 Jan 2007 00:58:28 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A65C87.2040200@ar.media.kyoto-u.ac.jp> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <200701111409.58652.faltet@carabos.com> <45A65C87.2040200@ar.media.kyoto-u.ac.jp> Message-ID: <45A65EA4.9030600@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > Francesc Altet wrote: >> A Dimecres 10 Gener 2007 22:49, Stefan van der Walt escrigu?: >>> On Wed, Jan 10, 2007 at 08:28:14PM +0100, Francesc Altet wrote: >>>> El dt 09 de 01 del 2007 a les 23:19 +0900, en/na David Cournapeau va >>>> >>>> escriure: >>>> time (putmask)--> 1.38 >>>> time (where)--> 2.713 >>>> time (numexpr where)--> 1.291 >>>> time (fancy+assign)--> 0.967 >>>> time (numexpr clip)--> 0.596 >>>> >>>> It is interesting to see there how fancy-indexing + assignation is >>>> quite >>>> more efficient than putmask. >>> Not on my machine: >>> >>> time (putmask)--> 0.181 >>> time (where)--> 0.783 >>> time (numexpr where)--> 0.26 >>> time (fancy+assign)--> 0.202 >> >> Yeah, a lot of difference indeed. Just for reference, my results >> above were done using a Duron (an Athlon but with only 128 KB of >> secondary cache) at 0.9 GHz. Now, using my laptop (Intel 4 @ 2 GHz, >> 512 KB of secondary cache), I get: >> >> time (putmask)--> 0.244 >> time (where)--> 2.111 >> time (numexpr where)--> 0.427 >> time (fancy+assign)--> 0.316 >> time (numexpr clip)--> 0.184 >> >> so, on my laptop fancy+assign is way slower than putmask. It should >> be noted also that the implementation of clip in numexpr (i.e. in >> pure C) is not that much faster than putmask (just a 30%); so perhaps >> it is not so necessary to come up with a pure C implementation for >> clip (or at least, on Intel P4 machines!). >> >> In any case, it is really shocking seeing how differently can perform >> the several CPU architectures on this apparently simple problem. > I am not sure it is such a simple problem: it involves massive branching. To be more precise, you can do clipping without branching, but then the clipping is highly type and machine dependent (using bit mask and other tricks). It may worth the trouble for double, float and int, dunno. David From faltet at carabos.com Thu Jan 11 13:00:25 2007 From: faltet at carabos.com (Francesc Altet) Date: Thu, 11 Jan 2007 19:00:25 +0100 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A65EA4.9030600@ar.media.kyoto-u.ac.jp> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <200701111409.58652.faltet@carabos.com> <45A65C87.2040200@ar.media.kyoto-u.ac.jp> <45A65EA4.9030600@ar.media.kyoto-u.ac.jp> Message-ID: <1168538425.2595.15.camel@localhost.localdomain> El dv 12 de 01 del 2007 a les 00:58 +0900, en/na David Cournapeau va escriure: > David Cournapeau wrote: > > Francesc Altet wrote: > >> A Dimecres 10 Gener 2007 22:49, Stefan van der Walt escrigu?: > >>> On Wed, Jan 10, 2007 at 08:28:14PM +0100, Francesc Altet wrote: > >>>> El dt 09 de 01 del 2007 a les 23:19 +0900, en/na David Cournapeau va > >>>> > >>>> escriure: > >>>> time (putmask)--> 1.38 > >>>> time (where)--> 2.713 > >>>> time (numexpr where)--> 1.291 > >>>> time (fancy+assign)--> 0.967 > >>>> time (numexpr clip)--> 0.596 > >>>> > >>>> It is interesting to see there how fancy-indexing + assignation is > >>>> quite > >>>> more efficient than putmask. > >>> Not on my machine: > >>> > >>> time (putmask)--> 0.181 > >>> time (where)--> 0.783 > >>> time (numexpr where)--> 0.26 > >>> time (fancy+assign)--> 0.202 > >> > >> Yeah, a lot of difference indeed. Just for reference, my results > >> above were done using a Duron (an Athlon but with only 128 KB of > >> secondary cache) at 0.9 GHz. Now, using my laptop (Intel 4 @ 2 GHz, > >> 512 KB of secondary cache), I get: > >> > >> time (putmask)--> 0.244 > >> time (where)--> 2.111 > >> time (numexpr where)--> 0.427 > >> time (fancy+assign)--> 0.316 > >> time (numexpr clip)--> 0.184 > >> > >> so, on my laptop fancy+assign is way slower than putmask. It should > >> be noted also that the implementation of clip in numexpr (i.e. in > >> pure C) is not that much faster than putmask (just a 30%); so perhaps > >> it is not so necessary to come up with a pure C implementation for > >> clip (or at least, on Intel P4 machines!). > >> > >> In any case, it is really shocking seeing how differently can perform > >> the several CPU architectures on this apparently simple problem. > > I am not sure it is such a simple problem: it involves massive branching. > To be more precise, you can do clipping without branching, but then the > clipping is highly type and machine dependent (using bit mask and other > tricks). It may worth the trouble for double, float and int, dunno. Well, I don't know which this trick can be, but just for completeness I have run the benchmark on an AMD Opteron (2 GHZ, 1 MB of secondary cache) machine, and here are the timings: time (putmask)--> 0.5 time (where)--> 1.035 time (numexpr where)--> 0.263 time (fancy+assign)--> 0.311 time (numpy clip)--> 0.704 time (numexpr clip)--> 0.089 [Note that I've added the clip from numpy. See the new benchmark attached] Curiously enough, fancy+assign is again faster than putmask. Also interesting is the fact that AMD seems to have optimized the branching in Opterons quite a lot: the processor is only 2x faster in GHz than the Duron, and most of the benchs do run 3x or 4x faster, but the numexpr clip does perform more than 6x faster (!). -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth -------------- next part -------------- A non-text attachment was scrubbed... Name: clip-bench2.py Type: text/x-python Size: 1048 bytes Desc: not available URL: From torgil.svensson at gmail.com Thu Jan 11 14:58:27 2007 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Thu, 11 Jan 2007 20:58:27 +0100 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <45A64E66.3080801@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> Message-ID: On 1/11/07, Travis Oliphant wrote: > Torgil Svensson wrote: >> Example1: We have a very large amount of data with a compressed >> internal representation >> >> Example2: We might want to generate data "on the fly" as it's needed >> >> Example3: If module creators to deal with different byte alignments, >> contiguousness etc it'll lead to lots of code duplication and >> unnecessarily much work >> >> Is it possible to add a data access API to this PEP?> >> Could you give an example of what you mean? I have no problem with such >> a concept. I'm mainly interested in getting the NumPy memory model into >> Python some-how. I know it's not the "only" way to think about memory, >> but it is a widely-used and useful way. Sure. I'm not objecting the memory model, what I mean is that data access between modules has a wider scope than just a memory model. Maybe i'm completely out-of-scope here, I thought this was worth considering for the inter-module-data-sharing - scope. Say we want to access a huge array with 1 million text-strings from another module that has a compressed representation in memory. Here's a pseudo-code-example with most of the details completely made up: buffer = AnotherModule_GetBigArrayAsBuffer() aview=buffer->bf_getarrayview() indexes=NewList() for(i=0; ishape[0] ; ++i) for(j=0; jshape[1] ; ++j) { item=aview->get_from_index(i) /* item represents the data described by the PyDataFormatObject */ if (is_interesting_item(item)) ListAdd(indexes,NewList(i,j)) } indexarr=Numpy_ArrayFromLists(indexes) Here, we don't have to care about any data layout-issues; called module could even produce data on-the-fly. If I want direct memory access we could use a function that returns data, strides and flags. From charlesr.harris at gmail.com Thu Jan 11 15:14:11 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 11 Jan 2007 13:14:11 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> Message-ID: On 1/11/07, Torgil Svensson wrote: > > On 1/11/07, Travis Oliphant wrote: > > Torgil Svensson wrote: > >> Example1: We have a very large amount of data with a compressed > >> internal representation > >> > >> Example2: We might want to generate data "on the fly" as it's needed > >> > >> Example3: If module creators to deal with different byte alignments, > >> contiguousness etc it'll lead to lots of code duplication and > >> unnecessarily much work > >> > >> Is it possible to add a data access API to this PEP?> > >> Could you give an example of what you mean? I have no problem with > such > >> a concept. I'm mainly interested in getting the NumPy memory model > into > >> Python some-how. I know it's not the "only" way to think about memory, > >> but it is a widely-used and useful way. > > Sure. I'm not objecting the memory model, what I mean is that data > access between modules has a wider scope than just a memory model. > Maybe i'm completely out-of-scope here, I thought this was worth > considering for the inter-module-data-sharing - scope. This is where separating the memory block from the API starts to show advantages. OTOH, we should try to keep this all as simple and basic as possible. Trying to design for every potential use will lead to over design, it is a fine line to walk. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Thu Jan 11 15:36:09 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 11 Jan 2007 12:36:09 -0800 Subject: [Numpy-discussion] random permutation Message-ID: Why is the first element of the permutation always the same? Am I using random.permutation in the right way? >> x matrix([[0], [1], [2], [3]]) >> M.random.permutation(x) array([[0], [1], [1], [0]]) >> M.random.permutation(x) array([[0], [1], [2], [2]]) >> M.random.permutation(x) array([[0], [1], [0], [0]]) >> M.random.permutation(x) array([[0], [0], [2], [0]]) >> M.random.permutation(x) array([[0], [0], [1], [0]]) >> M.__version__ '1.0rc1' From robert.kern at gmail.com Thu Jan 11 15:41:30 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 11 Jan 2007 14:41:30 -0600 Subject: [Numpy-discussion] random permutation In-Reply-To: References: Message-ID: <45A6A0FA.2060504@gmail.com> Keith Goodman wrote: > Why is the first element of the permutation always the same? Am I > using random.permutation in the right way? >>> M.__version__ > '1.0rc1' This has been fixed in more recent versions. http://projects.scipy.org/scipy/numpy/ticket/374 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kwgoodman at gmail.com Thu Jan 11 16:33:15 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 11 Jan 2007 13:33:15 -0800 Subject: [Numpy-discussion] Warning messages after upgrading to 1.0.1 Message-ID: I get lots of warning messages after upgrading from 1.0rc1 to 1.0.1: Warning: divide by zero encountered in divide Warning: invalid value encountered in log Warning: invalid value encountered in subtract Warning: invalid value encountered in multiply What's a good way to turn off the warnings? From robert.kern at gmail.com Thu Jan 11 16:37:34 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 11 Jan 2007 15:37:34 -0600 Subject: [Numpy-discussion] Warning messages after upgrading to 1.0.1 In-Reply-To: References: Message-ID: <45A6AE1E.6060004@gmail.com> Keith Goodman wrote: > I get lots of warning messages after upgrading from 1.0rc1 to 1.0.1: > > Warning: divide by zero encountered in divide > Warning: invalid value encountered in log > Warning: invalid value encountered in subtract > Warning: invalid value encountered in multiply > > What's a good way to turn off the warnings? seterr(all='ignore') -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From seanl at literati.org Thu Jan 11 17:21:46 2007 From: seanl at literati.org (Sean Lynch) Date: Thu, 11 Jan 2007 14:21:46 -0800 Subject: [Numpy-discussion] Fwd: Advanced selection, duplicate indices, and augmented assignment In-Reply-To: References: <459FEC8B.4010904@literati.org> <45A03223.5050208@gmail.com> <45A06A74.3090001@ee.byu.edu> <45A162C1.6020706@literati.org> Message-ID: <45A6B87A.3080206@literati.org> Timothy Hochberg wrote: > > > Sean, > > I'm pretty sure that at one point I had a way to do exactly what you > are doing, however it's been a while and I don't know where that code > wandered off to. I will think about it now that I'm done doing silly > stuff and see if I can recall what the trick was. > > [Just don't want you to feel abandoned...] > Thanks, Tim. This isn't a showstopper for me yet, but when it is if you can't find the code I'll just write a little C function to do what I need. At some point I'll need to recalculate normals in something like real time, but it can slide for now. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 250 bytes Desc: OpenPGP digital signature URL: From kwgoodman at gmail.com Thu Jan 11 18:18:19 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 11 Jan 2007 15:18:19 -0800 Subject: [Numpy-discussion] No test file found Message-ID: I see a lot of 'No test file found' warnings when running numpy.test(). What does that mean? >> M.__version__ '1.0.1' >> M.test(10,10) Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/core/tests for module Found 13 tests for numpy.core.umath Found 9 tests for numpy.lib.arraysetops Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/distutils/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/fft/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/distutils/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/lib/tests for module Found 3 tests for numpy.fft.helper Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/distutils/tests for module Found 1 tests for numpy.lib.ufunclike Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/core/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/testing/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/testing/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/core/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/tests for module Found 2 tests for numpy.lib.polynomial Found 9 tests for numpy.core.records Found 26 tests for numpy.core.numeric Found 5 tests for numpy.distutils.misc_util Found 4 tests for numpy.lib.getlimits Found 31 tests for numpy.core.numerictypes Found 4 tests for numpy.core.scalarmath Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/linalg/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/random/tests for module Found 12 tests for numpy.lib.twodim_base Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/core/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/lib/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/tests for module Found 47 tests for numpy.lib.shape_base Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/fft/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/distutils/tests for module Found 4 tests for numpy.lib.index_tricks Found 32 tests for numpy.linalg.linalg Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/random/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/distutils/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/tests for module Found 42 tests for numpy.lib.type_check Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/lib/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/core/tests for module Found 1 tests for numpy.fft.fftpack Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/distutils/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/lib/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/core/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/lib/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/lib/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/random/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/testing/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/distutils/tests for module Found 188 tests for numpy.core.multiarray Found 36 tests for numpy.core.ma Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/core/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/linalg/tests for module Found 10 tests for numpy.core.defmatrix Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/fft/tests for module Found 41 tests for numpy.lib.function_base Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/distutils/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/core/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/testing/tests for module Warning: No test file found in /usr/lib/python2.4/site-packages/numpy/core/tests for module Found 0 tests for __main__ From stefan at sun.ac.za Thu Jan 11 18:25:53 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 11 Jan 2007 23:25:53 +0000 Subject: [Numpy-discussion] No test file found In-Reply-To: References: Message-ID: <20070111232553.GE1552@zaphod.lagged.za.net> On Thu, Jan 11, 2007 at 03:18:19PM -0800, Keith Goodman wrote: > I see a lot of 'No test file found' warnings when running > numpy.test(). What does that mean? It means your verbosity is set too high. You'll find that N.test(0,0) complains much less (although, realistically, you'd probably want to run N.test(1,0) or simply N.test()). Cheers St?fan From kwgoodman at gmail.com Thu Jan 11 18:32:21 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 11 Jan 2007 15:32:21 -0800 Subject: [Numpy-discussion] No test file found In-Reply-To: <20070111232553.GE1552@zaphod.lagged.za.net> References: <20070111232553.GE1552@zaphod.lagged.za.net> Message-ID: On 1/11/07, Stefan van der Walt wrote: > On Thu, Jan 11, 2007 at 03:18:19PM -0800, Keith Goodman wrote: > > I see a lot of 'No test file found' warnings when running > > numpy.test(). What does that mean? > > It means your verbosity is set too high. You'll find that N.test(0,0) > complains much less (although, realistically, you'd probably want to run > N.test(1,0) or simply N.test()). >> M.test(M.inf, -M.inf) ---------------------------------------------------------------------- Ran 672 tests in 0.765s From robert.kern at gmail.com Thu Jan 11 18:32:42 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 11 Jan 2007 17:32:42 -0600 Subject: [Numpy-discussion] No test file found In-Reply-To: References: Message-ID: <45A6C91A.7000203@gmail.com> Keith Goodman wrote: > I see a lot of 'No test file found' warnings when running > numpy.test(). What does that mean? The test collector tries to find a test module for every actual module. So numpy/dual.py would correspond to numpy/tests/test_dual.py . A lot of modules don't need to be tested because they don't contain real code (__init__.py's, info.py's, __version__.py's) or because their code is tested by other functions (most of the extension modules are tested by testing the Python modules where the functions are exposed). And of course, some modules which should have test coverage don't. Unfortunately, it's difficult to extract those from all of the false positives. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kwgoodman at gmail.com Thu Jan 11 18:38:39 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 11 Jan 2007 15:38:39 -0800 Subject: [Numpy-discussion] No test file found In-Reply-To: <45A6C91A.7000203@gmail.com> References: <45A6C91A.7000203@gmail.com> Message-ID: On 1/11/07, Robert Kern wrote: > Keith Goodman wrote: > > I see a lot of 'No test file found' warnings when running > > numpy.test(). What does that mean? > > The test collector tries to find a test module for every actual module. So > numpy/dual.py would correspond to numpy/tests/test_dual.py . A lot of modules > don't need to be tested because they don't contain real code (__init__.py's, > info.py's, __version__.py's) or because their code is tested by other functions > (most of the extension modules are tested by testing the Python modules where > the functions are exposed). > > And of course, some modules which should have test coverage don't. > Unfortunately, it's difficult to extract those from all of the false positives. I am learning a lot by upgrading. Thank you for the lessons. From Chris.Barker at noaa.gov Thu Jan 11 18:51:08 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 11 Jan 2007 15:51:08 -0800 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> Message-ID: <45A6CD6C.2030709@noaa.gov> David Cournapeau wrote: > To do the actual clipping if the datatypes are 'native' is trivial in C: > a single loop, a comparison, that's it. only if it's also Contiguous. From a quick look at your posted code, it looks like it only works for contiguous arrays. > I don't think C++ is welcomed in core numpy no , it's not. I avoid C and C++ whenever possible, but there looks to be some real promise in C++ templates, etc -- but it does get ugly! I"ve always wanted to give Blitz++ a try... > autogen works well enough for me; I didn't know about autogen -- that may be all we need. > Now, I didn't know that clip was supposed to handle arrays as min/max > values. one more nifty feature...And if you want to support broadcasting, even more so! > At first, I didn't understand the need to care about > contiguous/non contiguous; having non scalar for min/max makes it > necessary to have special case for non contiguous. I'm confused. This issue is that you can't just increment the pointer to get the next element if the array is non-contiguous.. you need to do all the strides, etc, math. > But again, it is > important not to lose sight... The goal was to have faster clipping for > matplotlib, and this cases are easy, because it is native type and > scalar min/max, where contiguous or not does not matter as we traverse > the input arrays element by element. You still have to traverse them properly, and I didn't find a way to do that equally efficiently for contiguous and non-contiguous arrays -- thus I special cased it. That doesn't mean there isn't a a way -- I really don't know much what I'm doing! What you've [posted doesn't look like it will work for non-contiguous arrays: > static int native_scalar_fbl_clip(npy_float* in, > npy_intp size, npy_float min, npy_float max) > { > //npy_float* kcoeff, npy_float* err) > npy_intp i; > > for (i = 0; i < size; ++i) { > if (in[i] <= min) { > in[i] = min; > } > if (in[i] >= max) { > in[i] = max; > } > } > return 0; > } This just loops through the whole C array from start to size -- only works for contiguous data. > If we pass non native endian, non > contiguous arrays, there is actually a pretty good chance that the > current implementation is already fast enough, and does not need to be > changed anyway. good point -- optimizing the important cases, while allowing all cases is a fine way to go. If anyone needs other cases optimized, that can be added later. I'd still like to know if anyone knows how to efficiently loop through all the elements of a non-contiguous array in C. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From bthom at cs.hmc.edu Thu Jan 11 22:07:13 2007 From: bthom at cs.hmc.edu (belinda thom) Date: Thu, 11 Jan 2007 19:07:13 -0800 Subject: [Numpy-discussion] How do I test if an array contains NaN ? In-Reply-To: References: <459D8B84.3090405@gmail.com> Message-ID: <115B1455-C3FC-4FE6-912F-1CFD3F8F02D4@cs.hmc.edu> > It's probably worth mentioning that IPython has (thanks to a user > contributed implementation) search capabilities besides tab completion > (which requires you to at least know the start of the string you > want): > > In [3]: N.*nan*? > N.isnan > N.nan > N.nan_to_num > N.nanargmax > N.nanargmin > N.nanmax > N.nanmin > N.nansum this thing can be used even more slickly: N.[a-z]*? prints only the "non private" stuff. i'm starting to find this greatly speeds up my development cycle. --b From torgil.svensson at gmail.com Thu Jan 11 22:28:00 2007 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Fri, 12 Jan 2007 04:28:00 +0100 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> Message-ID: On 1/11/07, Charles R Harris wrote: > On 1/11/07, Torgil Svensson wrote: > > Sure. I'm not objecting the memory model, what I mean is that data > > access between modules has a wider scope than just a memory model. > > Maybe i'm completely out-of-scope here, I thought this was worth > > considering for the inter-module-data-sharing - scope. > > This is where separating the memory block from the API starts to show > advantages. OTOH, we should try to keep this all as simple and basic as > possible. Trying to design for every potential use will lead to over design, > it is a fine line to walk. I Agree. I'm trying to look after a use case of my own here where I have a huge array (won't fit memory) with data that is very easy to compress (easily fit in memory). OTOH, I have yet no need to share this between modules but a simple data access API opens up a variety of options. I my mindset, I can slice and dice my huge array and the implementation behind the data access API will choose between having the views represented internally as intervals or lists of indexes. So i'm +1 for having all information concerning nd-array access on a logical level (shapes) in one API and let the memory-layout-details (strides, FORTRAN, C etc) live in another API and a module that wants to try to skip the api overhead (numpy) can always do something like: memory_interface=array_interface->get_memory_layout() if (memory_interface) { ... use memory_interface->strides ... etc } else { ... use array_interface->get_item_fom_index() ... etc } I'm guessing that most of the modules trying to access an array will choose to go through numpy for fast operations. Another use of an api is to do things like give an "RGB"-view of an image regardless of which weird image format lying below without having to convert the whole image in-memory and loose precision or memory. If we want the whole in-memory-RGB-copy we could just take the RGB-view, pass it to numpy and force numpy to do a copy. The module can then, in either case, operate on the image through numpy or return a numpy object to the user. (numpy is of course integrated in python by then) From tim.hochberg at ieee.org Thu Jan 11 23:24:15 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Thu, 11 Jan 2007 21:24:15 -0700 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A6CD6C.2030709@noaa.gov> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> Message-ID: On 1/11/07, Christopher Barker wrote: [CHOP] > I'd still like to know if anyone knows how to efficiently loop through > all the elements of a non-contiguous array in C. First, it's not that important if the array is contiguous for this sort of thing. What you really care about is whether it's simply-strided (or maybe single-strided would be a better term). Anyway, the last dimension of the array can be strided without making things more difficult. All you need to be able to do is to address the elements of the array as thedata[offset + stride*index]. That being said, the strategy that the ufuncs use, and possibly other functions as well, is to have the core functions operate only on simply-strided chunks of data. At a higher level, there is some magic that parcels up non-contiguous array into simply-strided chunks and feeds them to core functions. How efficient this is obviously depends on how large the chunks that the magic parceler manages to extract are, but typically it seems to work pretty well. I don't know whether the magic that does the parcelling is available for use by random functions or whether it's specific to the ufunc machinery, I never delved into that end of the ufuncs. -tim -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Fri Jan 12 00:08:29 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 12 Jan 2007 14:08:29 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A6CD6C.2030709@noaa.gov> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> Message-ID: <45A717CD.8080301@ar.media.kyoto-u.ac.jp> Christopher Barker wrote: > >> autogen works well enough for me; > > I didn't know about autogen -- that may be all we need. > numpy has code which already does something similar to autogen: you declare a function, and some template with a generic name, and the code generator replaces the generic name and type with some values. All the .src files in numpy/core follow this pattern. >> Now, I didn't know that clip was supposed to handle arrays as min/max >> values. > > one more nifty feature...And if you want to support broadcasting, even > more so! > >> At first, I didn't understand the need to care about >> contiguous/non contiguous; having non scalar for min/max makes it >> necessary to have special case for non contiguous. > > I'm confused. This issue is that you can't just increment the pointer to > get the next element if the array is non-contiguous.. you need to do all > the strides, etc, math. > Ok, so we don't mean the same thing by contiguous, and I should check that my definition is the actual one... For me, contiguous means that the array has C order, and a non contiguous array has a 'random' order, but still can go to the next element in the buffer by using standard C array addressing. In my mind, contiguous is about the relationship between the indexing of the array in C and the math indexing. According to the numpy ebook, the data buffer may: - not be aligned on word boundaries -> NPY_ALIGNED - not be native endianess -> NPY_ISNOTSWAPPED - not C contiguous (last index does not move first) -> NPY_CONTIGUOUS. I thought that as long as NPY_ALIGNED is true, you are sure that array->data[i] is the ith element of the buffer with the datatype of the array ? If the data are not aligned or not native endian, I just use the existing implementation; if you are not using the CPU endianness or alignment, you cannot expect to do things at a decent speed anyway. In my code, I differentiate alignment, endianness and scalar case. If any of this condition is not true, I just rely on the old implementation for now, which should make it easy to extend if necessary. cheers, David From david at ar.media.kyoto-u.ac.jp Fri Jan 12 00:12:31 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 12 Jan 2007 14:12:31 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> Message-ID: <45A718BF.8060202@ar.media.kyoto-u.ac.jp> Timothy Hochberg wrote: > > > On 1/11/07, *Christopher Barker* > wrote: > > [CHOP] > > > I'd still like to know if anyone knows how to efficiently loop through > all the elements of a non-contiguous array in C. > > > First, it's not that important if the array is contiguous for this > sort of thing. What you really care about is whether it's > simply-strided (or maybe single-strided would be a better term). > Anyway, the last dimension of the array can be strided without making > things more difficult. All you need to be able to do is to address the > elements of the array as thedata[offset + stride*index]. I don't understand why we need to do thedata[offset + stride * index] instead of thedata[index] when the data are aligned ? It looks like I seriously misunderstood the meaning of alignement... David > > That being said, the strategy that the ufuncs use, and possibly other > functions as well, is to have the core functions operate only on > simply-strided chunks of data. At a higher level, there is some magic > that parcels up non-contiguous array into simply-strided chunks and > feeds them to core functions. How efficient this is obviously depends > on how large the chunks that the magic parceler manages to extract > are, but typically it seems to work pretty well. > > I don't know whether the magic that does the parcelling is available > for use by random functions or whether it's specific to the ufunc > machinery, I never delved into that end of the ufuncs. > > -tim > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From oliphant at ee.byu.edu Fri Jan 12 00:36:09 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 11 Jan 2007 22:36:09 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> Message-ID: <45A71E49.1000701@ee.byu.edu> Torgil Svensson wrote: > On 1/11/07, Charles R Harris wrote: > >> On 1/11/07, Torgil Svensson wrote: >> >>> Sure. I'm not objecting the memory model, what I mean is that data >>> access between modules has a wider scope than just a memory model. >>> Maybe i'm completely out-of-scope here, I thought this was worth >>> considering for the inter-module-data-sharing - scope. >>> >> This is where separating the memory block from the API starts to show >> advantages. OTOH, we should try to keep this all as simple and basic as >> possible. Trying to design for every potential use will lead to over design, >> it is a fine line to walk. >> > > I Agree. I'm trying to look after a use case of my own here where I > have a huge array (won't fit memory) with data that is very easy to > compress (easily fit in memory). OTOH, I have yet no need to share > this between modules but a simple data access API opens up a variety > of options. > I think this is a good idea generally. I think the PIL would be much more open to this kind of API becauase the memory model of the PIL is different than ours. On the other hand, I think it would be a shame to not provide a basic N-d array memory model like NumPy has because it is used so often. > I my mindset, I can slice and dice my huge array and the > implementation behind the data access API will choose between having > the views represented internally as intervals or lists of indexes. > > So i'm +1 for having all information concerning nd-array access on a > logical level (shapes) in one API and let the memory-layout-details > (strides, FORTRAN, C etc) live in another API and a module that wants > to try to skip the api overhead (numpy) can always do something like: > I had originally thought to separate these out in to multiple calls anyway. Perhaps we could propose the same thing. Have a full struct interface as one option and a multiple-call interface like you propose as another. > memory_interface=array_interface->get_memory_layout() > if (memory_interface) > { > ... use memory_interface->strides ... etc > } > else > { > ... use array_interface->get_item_fom_index() ... etc > } > > I'm guessing that most of the modules trying to access an array will > choose to go through numpy for fast operations. > > Another use of an api is to do things like give an "RGB"-view of an > image regardless of which weird image format lying below without > having to convert the whole image in-memory and loose precision or > memory. This is true. So at what level do we propose the API. Single-item access for sure, but what about array_interface->get_block_from_slice() ? Such a thing would be very useful for all kinds of large data-sets, from images, and videos, to scientific data-sets. > If we want the whole in-memory-RGB-copy we could just take the > RGB-view, pass it to numpy and force numpy to do a copy. The module > can then, in either case, operate on the image through numpy or return > a numpy object to the user. (numpy is of course integrated in python > by then) > Getting this array_interface into Python goes a long way into making that happen, I think. -Travis From charlesr.harris at gmail.com Fri Jan 12 00:32:51 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 11 Jan 2007 22:32:51 -0700 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A718BF.8060202@ar.media.kyoto-u.ac.jp> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A718BF.8060202@ar.media.kyoto-u.ac.jp> Message-ID: On 1/11/07, David Cournapeau wrote: > > Timothy Hochberg wrote: > > > > > > On 1/11/07, *Christopher Barker* > > wrote: > > > > [CHOP] > > > > > > I'd still like to know if anyone knows how to efficiently loop > through > > all the elements of a non-contiguous array in C. > > > > > > First, it's not that important if the array is contiguous for this > > sort of thing. What you really care about is whether it's > > simply-strided (or maybe single-strided would be a better term). > > Anyway, the last dimension of the array can be strided without making > > things more difficult. All you need to be able to do is to address the > > elements of the array as thedata[offset + stride*index]. > I don't understand why we need to do thedata[offset + stride * index] > instead of thedata[index] when the data are aligned ? It looks like I > seriously misunderstood the meaning of alignement... I think you are confusing aligned for contiguous. Aligned normally means the data occurs on natural address boundaries for the architecture and item size, say multiples of 4 for 32 bit words on 32 bit machines. This contrasts with the case where a word is split across natural boundaries: some architectures support that with decreased performance, others don't. Then there are endianess issues, etc, etc. Anyway, the easiest data to deal with is contiguous data where the data items lie one after the other in (virtual) memory. The next easiest is where the spacing between data elements is fixed, an lastly there is the case of generally strided data. Chuck David > > > > That being said, the strategy that the ufuncs use, and possibly other > > functions as well, is to have the core functions operate only on > > simply-strided chunks of data. At a higher level, there is some magic > > that parcels up non-contiguous array into simply-strided chunks and > > feeds them to core functions. How efficient this is obviously depends > > on how large the chunks that the magic parceler manages to extract > > are, but typically it seems to work pretty well. > > > > I don't know whether the magic that does the parcelling is available > > for use by random functions or whether it's specific to the ufunc > > machinery, I never delved into that end of the ufuncs. > > > > -tim > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Fri Jan 12 00:50:40 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 11 Jan 2007 22:50:40 -0700 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> Message-ID: <45A721B0.8080508@ee.byu.edu> Timothy Hochberg wrote: > > > On 1/11/07, *Christopher Barker* > wrote: > > [CHOP] > > > I'd still like to know if anyone knows how to efficiently loop through > all the elements of a non-contiguous array in C. > > > First, it's not that important if the array is contiguous for this > sort of thing. What you really care about is whether it's > simply-strided (or maybe single-strided would be a better term). > Anyway, the last dimension of the array can be strided without making > things more difficult. All you need to be able to do is to address the > elements of the array as thedata[offset + stride*index]. This is right. Just remember that strides are "byte-strides" and not "element-strides" and so thedata had better be a pointer to a byte. > > That being said, the strategy that the ufuncs use, and possibly other > functions as well, is to have the core functions operate only on > simply-strided chunks of data. At a higher level, there is some magic > that parcels up non-contiguous array into simply-strided chunks and > feeds them to core functions. How efficient this is obviously depends > on how large the chunks that the magic parceler manages to extract > are, but typically it seems to work pretty well. This is one thing I've exposed (and made use of in more than one place) with NumPy. In Numeric, the magic was in a few lines of the ufuncobject file). Now, it is exposed in the concept of an array iterator. Anybody can take advantage of it as it there is a C-API call to get an array iterator from the array (it's actually the object returned by the .flat method). You can even get an iterator that iterates over one-less dimension than the array has (with the dimension using the smallest strides left "un-iterated" so that you can call an inner loop with it). You can see examples of this in several places. The basic template is: int axis=-1; /* -1 means let the code decide which axis, otherwise you can choose */ dit = (PyArrayIterObject *)PyArray_IterAllButAxis(dest, &axis); if (dit==NULL) /* error return*/ while (dit->index < dit->size) { /* do something dit->dataptr points to the first position of the first "line" PyArray_STRIDE(dest, axis) is the striding PyArray_DIM(dest, axis) is the size of the "line" */ PyArray_ITER_NEXT(dit); /* move to the next line */ } You see code like this all over in NumPy. The broadcasting of ufuncs is also exposed (so you can do this on multiple arrays). See the multi-iterator objects. -Travis -Travis From david at ar.media.kyoto-u.ac.jp Fri Jan 12 01:37:50 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 12 Jan 2007 15:37:50 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A718BF.8060202@ar.media.kyoto-u.ac.jp> Message-ID: <45A72CBE.7080609@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On 1/11/07, *David Cournapeau* > wrote: > > Timothy Hochberg wrote: > > > > > > On 1/11/07, *Christopher Barker* > > >> > wrote: > > > > [CHOP] > > > > > > I'd still like to know if anyone knows how to efficiently > loop through > > all the elements of a non-contiguous array in C. > > > > > > First, it's not that important if the array is contiguous for this > > sort of thing. What you really care about is whether it's > > simply-strided (or maybe single-strided would be a better term). > > Anyway, the last dimension of the array can be strided without > making > > things more difficult. All you need to be able to do is to > address the > > elements of the array as thedata[offset + stride*index]. > I don't understand why we need to do thedata[offset + stride * index] > instead of thedata[index] when the data are aligned ? It looks like I > seriously misunderstood the meaning of alignement... > > > I think you are confusing aligned for contiguous. Aligned normally > means the data occurs on natural address boundaries for the > architecture and item size, say multiples of 4 for 32 bit words on 32 > bit machines. I understand that definition of alignment, but I thought it also implied that the data buffer has an equal spacing between its elements, and this spacing is equal to the size of the type ? > This contrasts with the case where a word is split across natural > boundaries: some architectures support that with decreased > performance, others don't. Then there are endianess issues, etc, etc. > Anyway, the easiest data to deal with is contiguous data where the > data items lie one after the other in (virtual) memory. The next > easiest is where the spacing between data elements is fixed, So does that mean you can have a numpy array of let's say 8 bytes double, but that each item's address may be 16 bytes one from each other (if we suppose alignment) ? There are three concepts here: 1: address alignment: to be sure that data pointer is a multiple of the data type (a bit like buffer returned by posix_memalign). 2: data "packing": for an array with elements of size d bytes, for a given element, you get the next one by adding d bytes to the data pointer. 3: style of indexing, eg for a rank 4, what is the function (a, b, c, d) -> i such as array->data[i] = array[a, b, c, d]. So if I understand you correctly, contiguous is about points 2 and 3, and not 3 only as I first thought ? I am confused about the relation between (in bytes) strides, contiguous and alignment... Because when I read the pages 25 to 28 of the numpy ebook about contiguous memory layout, we talk only about items indexing and the relationship between multi-dimensional numpy indexing and one dimension indexing in the actual data buffer (point 3). I thought all numpy arrays were packed... David From david at ar.media.kyoto-u.ac.jp Fri Jan 12 01:52:41 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 12 Jan 2007 15:52:41 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A721B0.8080508@ee.byu.edu> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A721B0.8080508@ee.byu.edu> Message-ID: <45A73039.9070806@ar.media.kyoto-u.ac.jp> Travis Oliphant wrote: > > This is one thing I've exposed (and made use of in more than one place) > with NumPy. In Numeric, the magic was in a few lines of the ufuncobject > file). Now, it is exposed in the concept of an array iterator. Anybody > can take advantage of it as it there is a C-API call to get an array > iterator from the array (it's actually the object returned by the .flat > method). You can even get an iterator that iterates over one-less > dimension than the array has (with the dimension using the smallest > strides left "un-iterated" so that you can call an inner loop with it). The thing which confuses me is whether this is useful when you only one item of one array at a time. When I was implementing some functions for LPC, I took a look at your examples for array iterators and explanations in the numpy ebook, and it looked really helpful, indeed. For this kind of code, I needed to operate on several contiguous elements at a time. But here, for cliping with scalar min and max, I only need to access to one item at a time from the input array, and that's it; in particular, I don't care about the order of iteration. So the question really boils down to: "for a numpy array a of eg float32, am I guaranteed that a->data[sizeof(float32) * i] for 0 <= i < a.size gives me all the items of a, even for non contiguous arrays ?" cheers, David From gael.varoquaux at normalesup.org Fri Jan 12 02:16:30 2007 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 12 Jan 2007 08:16:30 +0100 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <45A64E66.3080801@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> Message-ID: <20070112071630.GA1822@clipper.ens.fr> Talking about the difference between the memory access model and the array API, maybe I am talking bullshit (I know next to nothing in these problems) but couldn' an efficient tree data structure be implemented on the memory buffer object ? I am pretty sure a simple tree read-only could, as for a tree that is edited, I am not so sure. Anyhow read-only tree are used a lot by some people. A lab next to mine uses them to describe results from there experiment. They store events in tree-like structures (I have been told that copied that from the CERN). They can then walk throught the tree in a very efficient way, and do statistical analysis on their collection of event. I am not sure if this can fit anywhere in the PEP, but it would sure enlarge its scope. Please enlight me. Ga?l From charlesr.harris at gmail.com Fri Jan 12 02:19:24 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 12 Jan 2007 00:19:24 -0700 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A72CBE.7080609@ar.media.kyoto-u.ac.jp> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A718BF.8060202@ar.media.kyoto-u.ac.jp> <45A72CBE.7080609@ar.media.kyoto-u.ac.jp> Message-ID: On 1/11/07, David Cournapeau wrote: > > Charles R Harris wrote: > > > > > > On 1/11/07, *David Cournapeau* > > wrote: > > > > Timothy Hochberg wrote: > > > > > > > > > On 1/11/07, *Christopher Barker* > > > > >> > > wrote: > > > > > > [CHOP] > > > > > > > > > I'd still like to know if anyone knows how to efficiently > > loop through > > > all the elements of a non-contiguous array in C. > > > > > > > > > First, it's not that important if the array is contiguous for this > > > sort of thing. What you really care about is whether it's > > > simply-strided (or maybe single-strided would be a better term). > > > Anyway, the last dimension of the array can be strided without > > making > > > things more difficult. All you need to be able to do is to > > address the > > > elements of the array as thedata[offset + stride*index]. > > I don't understand why we need to do thedata[offset + stride * > index] > > instead of thedata[index] when the data are aligned ? It looks like > I > > seriously misunderstood the meaning of alignement... > > > > > > I think you are confusing aligned for contiguous. Aligned normally > > means the data occurs on natural address boundaries for the > > architecture and item size, say multiples of 4 for 32 bit words on 32 > > bit machines. > I understand that definition of alignment, but I thought it also implied > that the data buffer has an equal spacing between its elements, and this > spacing is equal to the size of the type ? > > > > This contrasts with the case where a word is split across natural > > boundaries: some architectures support that with decreased > > performance, others don't. Then there are endianess issues, etc, etc. > > Anyway, the easiest data to deal with is contiguous data where the > > data items lie one after the other in (virtual) memory. The next > > easiest is where the spacing between data elements is fixed, > So does that mean you can have a numpy array of let's say 8 bytes > double, but that each item's address may be 16 bytes one from each other > (if we suppose alignment) ? Well, the common machines are 32 bit and 64 bit, so for instance extended precision (usually 80 bits) ends up as 96 bits (3*32) on the first and 128 (2*64) bits on the second, with the extra bits ignored. The items in c structures will often have empty spaces filling in between them unless specific compiler directives are invoked and the whole will be aligned on the appropriate boundary. For instance, on my machine struct bar { char a; int b; }; is 8 bytes long because the int forces the whole too be aligned on a 4 byte boundary. I have seen 16 byte alignments for altivec (128 bits) on PPC. So on and so forth. It is all very achitecture dependent. But to answer you guestion, alignment is something that describes the location of a *single* item in memory, usually various numbers, but sometimes structures also. Contiguous describes the spacing between items. There are three concepts here: > 1: address alignment: to be sure that data pointer is a multiple of > the data type (a bit like buffer returned by posix_memalign). > 2: data "packing": for an array with elements of size d bytes, for a > given element, you get the next one by adding d bytes to the data pointer. > 3: style of indexing, eg for a rank 4, what is the function (a, b, > c, d) -> i such as array->data[i] = array[a, b, c, d]. > > So if I understand you correctly, contiguous is about points 2 and > 3, and not 3 only as I first thought ? 2 and 3 are pretty much the same thing in c, but not in numpy. I am confused about the relation > between (in bytes) strides, contiguous and alignment... Because when I > read the pages 25 to 28 of the numpy ebook about contiguous memory > layout, we talk only about items indexing and the relationship between > multi-dimensional numpy indexing and one dimension indexing in the > actual data buffer (point 3). > I thought all numpy arrays were packed... The underlying data may be contiguous, but the view is not necessarily so. Transpose, for instance, just changes the view, not the storage, likewise, a[::2] returns a view that skips over every other item. Since x = a[::2] is a valid numpy array sharing memory with a, the items of x are not contiguous. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 12 02:21:42 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 12 Jan 2007 00:21:42 -0700 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A73039.9070806@ar.media.kyoto-u.ac.jp> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <1168457294.3617.36.camel@localhost.localdomain> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A721B0.8080508@ee.byu.edu> <45A73039.9070806@ar.media.kyoto-u.ac.jp> Message-ID: On 1/11/07, David Cournapeau wrote: > > Travis Oliphant wrote: > > > > This is one thing I've exposed (and made use of in more than one place) > > with NumPy. In Numeric, the magic was in a few lines of the ufuncobject > > file). Now, it is exposed in the concept of an array iterator. Anybody > > can take advantage of it as it there is a C-API call to get an array > > iterator from the array (it's actually the object returned by the .flat > > method). You can even get an iterator that iterates over one-less > > dimension than the array has (with the dimension using the smallest > > strides left "un-iterated" so that you can call an inner loop with it). > The thing which confuses me is whether this is useful when you only one > item of one array at a time. When I was implementing some functions for > LPC, I took a look at your examples for array iterators and explanations > in the numpy ebook, and it looked really helpful, indeed. For this kind > of code, I needed to operate on several contiguous elements at a time. > > But here, for cliping with scalar min and max, I only need to access to > one item at a time from the input array, and that's it; in particular, I > don't care about the order of iteration. So the question really boils > down to: > > "for a numpy array a of eg float32, am I guaranteed that > a->data[sizeof(float32) * i] for 0 <= i < a.size gives me all the items > of a, even for non contiguous arrays ?" No. That is what the array iterator is for. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 12 02:35:20 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 12 Jan 2007 00:35:20 -0700 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A721B0.8080508@ee.byu.edu> <45A73039.9070806@ar.media.kyoto-u.ac.jp> Message-ID: On 1/12/07, Charles R Harris wrote: > > > > On 1/11/07, David Cournapeau wrote: > > > > Travis Oliphant wrote: > > > > > > This is one thing I've exposed (and made use of in more than one > > place) > > > with NumPy. In Numeric, the magic was in a few lines of the > > ufuncobject > > > file). Now, it is exposed in the concept of an array > > iterator. Anybody > > > can take advantage of it as it there is a C-API call to get an array > > > iterator from the array (it's actually the object returned by the > > .flat > > > method). You can even get an iterator that iterates over one-less > > > dimension than the array has (with the dimension using the smallest > > > strides left "un-iterated" so that you can call an inner loop with > > it). > > The thing which confuses me is whether this is useful when you only one > > item of one array at a time. When I was implementing some functions for > > LPC, I took a look at your examples for array iterators and explanations > > in the numpy ebook, and it looked really helpful, indeed. For this kind > > of code, I needed to operate on several contiguous elements at a time. > > > > But here, for cliping with scalar min and max, I only need to access to > > one item at a time from the input array, and that's it; in particular, I > > > > don't care about the order of iteration. So the question really boils > > down to: > > > > "for a numpy array a of eg float32, am I guaranteed that > > a->data[sizeof(float32) * i] for 0 <= i < a.size gives me all the items > > of a, even for non contiguous arrays ?" > > > No. That is what the array iterator is for. > Although it is pretty common to make a copy of the array that *is* contiguous and pass that down. Doing so keeps life simple for the programmer and is pretty much required when interfacing to third party c and fortran routines. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 12 02:44:15 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 12 Jan 2007 00:44:15 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <20070112071630.GA1822@clipper.ens.fr> References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <20070112071630.GA1822@clipper.ens.fr> Message-ID: On 1/12/07, Gael Varoquaux wrote: > > Talking about the difference between the memory access model and the > array API, maybe I am talking bullshit (I know next to nothing in these > problems) but couldn' an efficient tree data structure be implemented on > the memory buffer object ? I am pretty sure a simple tree read-only > could, as for a tree that is edited, I am not so sure. Trees are nice, but they are not efficient for array type data. Traversing a tree usually requires some sort of stack (recursion), and a tree is not well structured for addressing data using indices. They just aren't appropriate for arrays, arrays are better represented by some sort of lattice. Anyhow read-only tree are used a lot by some people. A lab next to mine > uses them to describe results from there experiment. They store events in > tree-like structures (I have been told that copied that from the CERN). > They can then walk throught the tree in a very efficient way, and do > statistical analysis on their collection of event. Probably from ROOT? I am not sure if this can fit anywhere in the PEP, but it would sure > enlarge its scope. There is probably a tree module somewhere for python. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Fri Jan 12 02:54:04 2007 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 12 Jan 2007 08:54:04 +0100 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <20070112071630.GA1822@clipper.ens.fr> Message-ID: <20070112075403.GB1822@clipper.ens.fr> On Fri, Jan 12, 2007 at 12:44:15AM -0700, Charles R Harris wrote: > Trees are nice, but they are not efficient for array type data. Traversing > a tree usually requires some sort of stack (recursion), and a tree is not > well structured for addressing data using indices. They just aren't > appropriate for arrays, arrays are better represented by some sort of > lattice. Yes, indeed. I was just wondering if the PEP could be used for a performant implementation of trees. Basicaly that is mapping a tree to an array, which is possible. As far as performance, I think this is not performant at all when modifying the tree, but I do not know if it is possible to have an efficient traversing of the tree when it is mapped to an array. > Probably from ROOT? Yes. It seems a nice software for such things. The problem with it is that you have to learn C++, and experience shows that not everybody in an experimental lab is willing to do so. Ga?l From david at ar.media.kyoto-u.ac.jp Fri Jan 12 03:32:01 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 12 Jan 2007 17:32:01 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A718BF.8060202@ar.media.kyoto-u.ac.jp> <45A72CBE.7080609@ar.media.kyoto-u.ac.jp> Message-ID: <45A74781.7080505@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > Well, the common machines are 32 bit and 64 bit, so for instance > extended precision (usually 80 bits) ends up as 96 bits (3*32) on the > first and 128 (2*64) bits on the second, with the extra bits ignored. > The items in c structures will often have empty spaces filling in > between them unless specific compiler directives are invoked and the > whole will be aligned on the appropriate boundary. For instance, on my > machine > > struct bar { > char a; > int b; > }; So the data buffer in numpy is not really a C array, but more like a structure ? (I am talking about the actual binary data) In C, I think you always have sizeof(*a) bytes between a[0] and a[i]. That is sizeof(item) and addresses shift always match: we always have a[i] = (char*)a + sizeof(*a) * i. For example, in your example for extended precision, the computation may be done in 80 bits, but sizeof(long double) is 12 on my machine, not 10. There is a match between sizeof and spacing between items; otherwise, I don't see how pointer arithmetic would be possible with arrays in C otherwise ? But this goes quite further from my usual C knowledge, so I may be wrong here too. cheers, David From ndbecker2 at gmail.com Fri Jan 12 07:02:57 2007 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 12 Jan 2007 07:02:57 -0500 Subject: [Numpy-discussion] Latest Array-Interface PEP References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <45A71E49.1000701@ee.byu.edu> Message-ID: I believe we are converging, and this is pretty much the same design as I advocated. It is similar to boost::ublas. Storage is one concept. Interpretation of the storage is another concept. Numpy is a combination of a storage and interpretation. Storage could be dense or sparse. Allocated in various ways. Sparse can be implemented in different ways. Interpretation can be 1-d, 2-d. Zero-based, non-zero based. Also there is question of ownership (slices). From oliphant at ee.byu.edu Fri Jan 12 07:17:31 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 12 Jan 2007 05:17:31 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <45A71E49.1000701@ee.byu.edu> Message-ID: <45A77C5B.2030409@ee.byu.edu> Neal Becker wrote: > I believe we are converging, and this is pretty much the same design as I > advocated. It is similar to boost::ublas. > I'm grateful to hear that. It is nice when ideas come from several different corners. > Storage is one concept. > > Interpretation of the storage is another concept. > > Numpy is a combination of a storage and interpretation. > > Storage could be dense or sparse. Allocated in various ways. Sparse can be > implemented in different ways. > > Interpretation can be 1-d, 2-d. Zero-based, non-zero based. Also there is > question of ownership (slices). > How do we extend the buffer interface then? Do we have one API that allows sharing of storage and another that handles sharing of interpretation? How much detail should be in the interface regarding storage detail. Is there a possibility of having at least a few storage models "shareable" so that memory can be shared by others that view the data in the same way? -Travis From ndbecker2 at gmail.com Fri Jan 12 09:33:27 2007 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 12 Jan 2007 09:33:27 -0500 Subject: [Numpy-discussion] Latest Array-Interface PEP References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <45A71E49.1000701@ee.byu.edu> <45A77C5B.2030409@ee.byu.edu> Message-ID: Travis Oliphant wrote: > Neal Becker wrote: >> I believe we are converging, and this is pretty much the same design as I >> advocated. It is similar to boost::ublas. >> > I'm grateful to hear that. It is nice when ideas come from several > different corners. >> Storage is one concept. >> >> Interpretation of the storage is another concept. >> >> Numpy is a combination of a storage and interpretation. >> >> Storage could be dense or sparse. Allocated in various ways. Sparse can >> be implemented in different ways. >> >> Interpretation can be 1-d, 2-d. Zero-based, non-zero based. Also there >> is question of ownership (slices). >> > > How do we extend the buffer interface then? Do we have one API that > allows sharing of storage and another that handles sharing of > interpretation? > > How much detail should be in the interface regarding storage detail. > Is there a possibility of having at least a few storage models > "shareable" so that memory can be shared by others that view the data in > the same way? > How about: 1. A memory concept, of which buffer is an example. 2. A view concept. 3. A variety of common concrete types composing 1+2. So then, how do we use buffer in this scheme? I'm thinking that buffer isn't really the best thing to build on - but within this scheme buffer is a kind of memory (assuming it provides/could_be_made_to_provide the required interface). The view is not part of buffer, (as was proposed) but a separate piece. Still, I agree that we want a commonly used array object that includes both the memory and the view. I propose that we build it out of these more generic pieces, but also provide commonly used compositions of these pieces. I think this satisfies the desire for a self-describing array component, while allowing more flexibility and serving a wider usage. From bernhard.voigt at gmail.com Fri Jan 12 09:55:41 2007 From: bernhard.voigt at gmail.com (Bernhard Voigt) Date: Fri, 12 Jan 2007 15:55:41 +0100 Subject: [Numpy-discussion] Installation Message-ID: <21a270aa0701120655s71c7a625pbe049e932a9902e7@mail.gmail.com> Dear all, I've recently installed the last releases of numpy and scipy (tar archives from scipy.org). I encountered two problems: 1) During the configuration it seems the information from my site.cfg is not read. 2) Installation using atlas optimized lapack/blas libraries didn't work 1) My site.cfg is in numpy-1.0.1/ (same as the setup.py file) and has the following contents: [blas] library_dirs = ~/apps/usr/local/lib/ blas_libs = blas [lapack] library_dirs= ~/apps/usr/local/lib lapack_libs = lapack The output of the python setup.py config lists this: .... blas_info: FOUND: libraries = ['blas'] library_dirs = ['/afs/ifh.de/user/b/bvoigt/apps/lib'] language = f77 lapack_info: FOUND: libraries = ['lapack'] library_dirs = ['/afs/ifh.de/user/b/bvoigt/apps/lib'] language = f77 Which acutally aren't the libs from the path I quoted in site.cfg. I've put links to ~/apps/lib in order to get it run and this works. ~/apps/libs might be take from some python standard library path since this is where my python lives. 2) In addition to that I had another problem using ATLAS optimized blas and lapack libraries. Installation worked the same way that I had to put a link in ~/apps/lib/atlas. However, I got an unresolved symbold error that ATL_dgemm was not found, when I imported numpy.linalg. I searched for that type of failure when using altas and I found that the order of the libraries might be of importance when the numpy fortran wrappers are compiled: Taken from ( http://groups.google.com/group/sci.math.num-analysis/browse_frm/thread/8b62cc873a0a4721/ ): my problem was the order of the invication of the librarys ! if i use g77 dgt.f libf77blas.a libatlas.a i get no errormessage. but if i use g77 dgt.f libatlas.a libf77blas.a In this case quoted here, the compiler complained already about unresolved symbols, so I don't know if this is the same reason for the problem I observer. Anyway, I sorted the list of libraries in the numpy/distutils/system_info.py file in the method at the bottom. This had an effect, however another symbol couldn't be resolved :-( and I stopped playing around. Thanks for any suggestions/help solving those problems. Bernhard -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Jan 12 10:35:46 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 12 Jan 2007 08:35:46 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <45A71E49.1000701@ee.byu.edu> Message-ID: On 1/12/07, Neal Becker wrote: > > I believe we are converging, and this is pretty much the same design as I > advocated. It is similar to boost::ublas. Ah, *that* Neal Becker. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Fri Jan 12 11:21:28 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Fri, 12 Jan 2007 09:21:28 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <45A77C5B.2030409@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <45A71E49.1000701@ee.byu.edu> <45A77C5B.2030409@ee.byu.edu> Message-ID: On 1/12/07, Travis Oliphant wrote: > > Neal Becker wrote: > > I believe we are converging, and this is pretty much the same design as > I > > advocated. It is similar to boost::ublas. > > > I'm grateful to hear that. It is nice when ideas come from several > different corners. > > Storage is one concept. > > > > Interpretation of the storage is another concept. > > > > Numpy is a combination of a storage and interpretation. > > > > Storage could be dense or sparse. Allocated in various ways. Sparse can > be > > implemented in different ways. > > > > Interpretation can be 1-d, 2-d. Zero-based, non-zero based. Also there > is > > question of ownership (slices). > > > > How do we extend the buffer interface then? Do we have one API that > allows sharing of storage and another that handles sharing of > interpretation? > > How much detail should be in the interface regarding storage detail. > Is there a possibility of having at least a few storage models > "shareable" so that memory can be shared by others that view the data in > the same way? I'm concerned about the direction that this PEP seems to be going. The original proposal was borderline too complicated IMO, and now it seems headed in the direction of more complexity. Also, it seems that there are three different goals getting conflated here. None are bad, but they don't and probably shouldn't, all be addressed by the same PEP. 1. Allowing producers and consumers of blocks of data to share blocks efficiently. This is half of what the original PEP proposed. 2. Describing complex data types at the c-level. This is the other half of the PEP[1]. 3. Things that act like arrays, but have different storage methods. This details of this still seem pretty vague, but to the extent that I can figure them out, it doesn't seem useful or necessary to tie this into the rest of the array interface PEP. For example, "array_interface->get_block_from_slice()" has been mentioned. Why that instead of "PyObject_AsExtendedBuffer(PyObject_GetItem(index), ....)"[2]. I'll stop here, till I see some more details of what people have in mind, but at this point, I think that alternative memory models are a different problem that should be addressed separately. Sadly, I'm leaving town shortly and I'm running out of time, so I'll have to leave my objections in this somewhat vague state. Oh, the way that F. Lundh plans to expose PIL's data a chunk at a time is mentioned in this python-dev summary: http://www.python.org/dev/summary/2006-11-01_2006-11-15/ It doesn't seem necessary to have special support for this; all that is necessary is for the object returned by acquire_view to support the extended array protocol. [1] Remind me again why we can't simply use ctypes for this? It's already in the core. I'm sure it's less efficient, but you shouldn't need to parse the data structure information very often. I suspect that something that leveraged ctypes would meet less resistance. [2] Which reminds me. I never saw in the PEP what the actual call in the buffer protocol was supposed to look like. Is it something like: PyObject_AsExtendedBuffer(PyObject * obj, void **buffer, Py_ssize_t *buffer_len, funcptr *bf_getarrayview, funcptr *bf_relarrayview) ? -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam at thejenkins.org Fri Jan 12 14:09:29 2007 From: adam at thejenkins.org (Adam Jenkins) Date: Fri, 12 Jan 2007 14:09:29 -0500 Subject: [Numpy-discussion] Overload binary operators with custom array type Message-ID: <21ef68730701121109u3bb4b020gda2864d4a65da945@mail.gmail.com> I have written my own array class, let's call it "myarray", which behaves like a numpy.ndarray. I would like to support binary operators with mixed operand types, i.e.: lhs + rhs where one of the operands is a myarray object, and the other is a numpy.ndarray object. The problem is that I want the operation in this case to be handled by my add method, not by numpy's add method. If lhs is a myarray object, then this is no problem, since myarray.__add__(lhs, rhs) will be called. However if lhs is a numpy array and rhs is a myarray, then numpy.ndarray.__add__(lhs, rhs) will be called, whereas what I would like to happen would be for myarray.__radd__(rhs, lhs) to be called. Is there a way I can get numpy to do the latter? Python will only call the rhs.__radd__ method if lhs.__add__ returns NotImplemented, but as far as I can tell, the numpy.ndarray.__add__ method never returns NotImplemented. It seems that for "lhs + rhs", where lhs is a numpy array, numpy does one of two things: 1. if rhs has an __array__ method, than it calls rsh.__array__(), which is expected to return a numpy.ndarray object. Then it performs the binary operation with the result of __array__ for the rhs. 2. If rhs doesn't have an __array__ method, then it treats rhs as some kind of scalar, and tries to add each element of lhs to rhs. Under no circumstances does numpy just assume that it doesn't know what to do with rhs and return NotImplemented. My question is, is there a way I can get the effect of having myarray.__radd__(rhs, lhs) in this case, or in some way get numpy to invoke my code to handle this operation rather than just assuming rhs is a scalar? As for why I don't just implement myarray.__array__(), the reason is because it would be expensive. A myarray object doesn't actually contain the array data; rather it contains a handle referring to an array which resides in another process. The server process which actually contains the arrays can even be on another machine. So converting a myarray object to a numpy.ndarray involves transferring all the array data to the python process. For various reasons, for my application it's preferable that I handle binary operators with mixed numpy/myarray operand types myself, rather than converting the myarray object to a numpy.ndarray. Thanks in advance for any suggestions you can give. Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mi.Zhou at STJUDE.ORG Fri Jan 12 14:36:18 2007 From: Mi.Zhou at STJUDE.ORG (Zhou, Mi) Date: Fri, 12 Jan 2007 13:36:18 -0600 Subject: [Numpy-discussion] failed to build numpy and Numeric in Windows Message-ID: Hi, I am trying to build Numpy-1.01 in windows using "python setup.py build". My C compiler is MSVS2005. The compiler throws an error about umathmodule.c: "__umath_generated.c(72): error C2099: initializer is not a constant." Tried with Numeric-24.2, it gives the same error. The line of code it complains about is: static void * floor_data[] = { (void *)floor, (void *)floor, (void *)"floor", }; Does anybody know how to fix it? Thanks, Mi Zhou From tim.hochberg at ieee.org Fri Jan 12 14:37:48 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Fri, 12 Jan 2007 12:37:48 -0700 Subject: [Numpy-discussion] Overload binary operators with custom array type In-Reply-To: <21ef68730701121109u3bb4b020gda2864d4a65da945@mail.gmail.com> References: <21ef68730701121109u3bb4b020gda2864d4a65da945@mail.gmail.com> Message-ID: On 1/12/07, Adam Jenkins wrote: > > I have written my own array class, let's call it "myarray", which behaves > like a numpy.ndarray. I would like to support binary operators with mixed > operand types, i.e.: > > lhs + rhs > > where one of the operands is a myarray object, and the other is a > numpy.ndarray object. > > The problem is that I want the operation in this case to be handled by my > add method, not by numpy's add method. If lhs is a myarray object, then > this is no problem, since myarray.__add__(lhs, rhs) will be called. However > if lhs is a numpy array and rhs is a myarray, then numpy.ndarray.__add__(lhs, > rhs) will be called, whereas what I would like to happen would be for > myarray.__radd__(rhs, lhs) to be called. Is there a way I can get numpy to > do the latter? Python will only call the rhs.__radd__ method if lhs.__add__ > returns NotImplemented, but as far as I can tell, the numpy.ndarray.__add__ > method never returns NotImplemented. > > It seems that for "lhs + rhs", where lhs is a numpy array, numpy does one > of two things: > > 1. if rhs has an __array__ method, than it calls rsh.__array__(), which is > expected to return a numpy.ndarray object. Then it performs the binary > operation with the result of __array__ for the rhs. > > 2. If rhs doesn't have an __array__ method, then it treats rhs as some > kind of scalar, and tries to add each element of lhs to rhs. > > Under no circumstances does numpy just assume that it doesn't know what to > do with rhs and return NotImplemented. My question is, is there a way I can > get the effect of having myarray.__radd__(rhs, lhs) in this case, or in some > way get numpy to invoke my code to handle this operation rather than just > assuming rhs is a scalar? > > As for why I don't just implement myarray.__array__(), the reason is > because it would be expensive. A myarray object doesn't actually contain > the array data; rather it contains a handle referring to an array which > resides in another process. The server process which actually contains the > arrays can even be on another machine. So converting a myarray object to a > numpy.ndarray involves transferring all the array data to the python > process. For various reasons, for my application it's preferable that I > handle binary operators with mixed numpy/myarray operand types myself, > rather than converting the myarray object to a numpy.ndarray. You want __array_priority__; add an attribute __array_priority__ to your array-like object and set it's value to, say, 10 and ndarrays will defer to your object. -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From torgil.svensson at gmail.com Fri Jan 12 14:38:20 2007 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Fri, 12 Jan 2007 20:38:20 +0100 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <45A71E49.1000701@ee.byu.edu> References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <45A71E49.1000701@ee.byu.edu> Message-ID: On 1/12/07, Travis Oliphant wrote: > This is true. So at what level do we propose the API. Single-item > access for sure, but what about > > array_interface->get_block_from_slice() ? > > Such a thing would be very useful for all kinds of large data-sets, from > images, and videos, to scientific data-sets. That's definitely desirable. If the implementation can't handle that it can always throw. > > If we want the whole in-memory-RGB-copy we could just take the > > RGB-view, pass it to numpy and force numpy to do a copy. The module > > can then, in either case, operate on the image through numpy or return > > a numpy object to the user. (numpy is of course integrated in python > > by then) > > > Getting this array_interface into Python goes a long way into making > that happen, I think. Agree. We just have to be careful where to put the different pieces. From torgil.svensson at gmail.com Fri Jan 12 14:49:56 2007 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Fri, 12 Jan 2007 20:49:56 +0100 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <45A71E49.1000701@ee.byu.edu> <45A77C5B.2030409@ee.byu.edu> Message-ID: On 1/12/07, Neal Becker wrote: > How about: > > 1. A memory concept, of which buffer is an example. > 2. A view concept. > 3. A variety of common concrete types composing 1+2. Is there any difference between a view and an array that shares memory with another array? For access, are you suggesting 1. Single memory API 2. Single view API 3. Single array API where we can access View+Memory API ? From robert.kern at gmail.com Fri Jan 12 14:54:32 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 12 Jan 2007 13:54:32 -0600 Subject: [Numpy-discussion] failed to build numpy and Numeric in Windows In-Reply-To: References: Message-ID: <45A7E778.8040907@gmail.com> Zhou, Mi wrote: > Hi, > > I am trying to build Numpy-1.01 in windows using "python setup.py > build". My C compiler is MSVS2005. Hmm. I was pretty sure that MSVS2005 was not capable of building Python extension modules that work with the official Python interpreter builds. That doesn't explain or solve the problem that you state, but if the problem is restricted to MSVS2005, it may not be worth solving. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Mi.Zhou at STJUDE.ORG Fri Jan 12 15:07:23 2007 From: Mi.Zhou at STJUDE.ORG (Zhou, Mi) Date: Fri, 12 Jan 2007 14:07:23 -0600 Subject: [Numpy-discussion] failed to build numpy and Numeric in Windows Message-ID: Hi, Python on the same machine was built with MSVS2005. I think it is capable of building python modules. Interestingly, if I replace "floor" with "ceil" in that line of code. It will build just fine, and of course, the Numeric.floor() will work just like Numeric.ceil(): static void * floor_data[] = { (void *)ceil, (void *)ceil, (void *)"floor", }; Thanks, Mi -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern Sent: Friday, January 12, 2007 1:55 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] failed to build numpy and Numeric in Windows Zhou, Mi wrote: > Hi, > > I am trying to build Numpy-1.01 in windows using "python setup.py > build". My C compiler is MSVS2005. Hmm. I was pretty sure that MSVS2005 was not capable of building Python extension modules that work with the official Python interpreter builds. That doesn't explain or solve the problem that you state, but if the problem is restricted to MSVS2005, it may not be worth solving. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion From torgil.svensson at gmail.com Fri Jan 12 15:18:38 2007 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Fri, 12 Jan 2007 21:18:38 +0100 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <45A71E49.1000701@ee.byu.edu> <45A77C5B.2030409@ee.byu.edu> Message-ID: On 1/12/07, Timothy Hochberg wrote: > Things that act like arrays, but have different storage methods. This > details of this still seem pretty vague, but to the extent that I can figure > them out, it doesn't seem useful or necessary to tie this into the rest of > the array interface PEP. Looks like an array, act like an array, smells like an array = is an array > For example, > "array_interface->get_block_from_slice()" has been > mentioned. Why that instead of > "PyObject_AsExtendedBuffer(PyObject_GetItem(index), What is an "Extended buffer" ? Connecting that to array information doesn't feel intuitive. > ....)"[2]. I'll stop here, till I see some more details of what people have > in mind, but at this point, I think that alternative memory models are a > different problem that should be addressed separately. I agree > [1] Remind me again why we can't simply use ctypes for this? 1. ctypes is designed for "c types", not "array layout" 2. managing/creating complex formats in ctypes deviates from the clean, intuitive and simple (considerably compared to dtypes) => ugly code 3. Can ctypes handle anonymous lambda function pointers? > the core. I'm sure it's less efficient, but you shouldn't need to parse the > data structure information very often. I believe that'll be more common than you think; for example dynamically creating/combining/slicing recarrays with various data. //Torgil From tim.hochberg at ieee.org Fri Jan 12 15:47:10 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Fri, 12 Jan 2007 13:47:10 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <45A71E49.1000701@ee.byu.edu> <45A77C5B.2030409@ee.byu.edu> Message-ID: On 1/12/07, Torgil Svensson wrote: > > On 1/12/07, Timothy Hochberg wrote: > [CHOP] > > > the core. I'm sure it's less efficient, but you shouldn't need to parse > the > > data structure information very often. > > I believe that'll be more common than you think; for example > dynamically creating/combining/slicing recarrays with various data. I was unclear here. I didn't mean like that it would be infrequent in "once a month" sense. I meant that you would only need to look at the data structure information once per set of data that you are accessing and that you would typically extract many chunks of data from each set, so the amortized cost of parsing the data structure would be small. Trying to get out the door.... -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerard.vermeulen at grenoble.cnrs.fr Fri Jan 12 15:54:15 2007 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Fri, 12 Jan 2007 21:54:15 +0100 Subject: [Numpy-discussion] failed to build numpy and Numeric in Windows In-Reply-To: References: Message-ID: <20070112215415.09ed4102.gerard.vermeulen@grenoble.cnrs.fr> On Fri, 12 Jan 2007 14:07:23 -0600 "Zhou, Mi" wrote: > Hi, > > Python on the same machine was built with MSVS2005. I think it is > capable of building python modules. > > Interestingly, if I replace "floor" with "ceil" in that line of code. It > will build just fine, and of course, the Numeric.floor() will work just > like Numeric.ceil(): > > static void * floor_data[] = { (void *)ceil, (void *)ceil, (void > *)"floor", }; > > Thanks, > > Mi > I have found that I needed to patch the Python-2.4.3 distutils to build numpy and Numeric with MSVC-2005 (check the compiler flags): --- Python-2.4.3/Lib/distutils/msvccompiler.py.win64 2005-08-07 22:50:37.000000000 +0200 +++ Python-2.4.3/Lib/distutils/msvccompiler.py 2006-06-25 08:45:14.812500000 +0200 @@ -206,15 +206,21 @@ def __init__ (self, verbose=0, dry_run=0, force=0): CCompiler.__init__ (self, verbose, dry_run, force) self.__version = get_build_version() - if self.__version >= 7: + if 8.0 > self.__version >= 7: self.__root = r"Software\Microsoft\VisualStudio" self.__macros = MacroExpander(self.__version) else: + # XXX: don't know how to get the MSVS2005 info from the registry. + # So, use of distutils in a MSVS2005 32 or 64 bit command prompt + # to get the info from the environment. self.__root = r"Software\Microsoft\Devstudio" self.initialized = False def initialize(self): - self.__paths = self.get_msvc_paths("path") + if self.__version >= 8.0: + self.__paths = os.environ['PATH'].split(os.pathsep) + else: + self.__paths = self.get_msvc_paths("path") if len (self.__paths) == 0: raise DistutilsPlatformError, \ @@ -227,22 +233,38 @@ self.lib = self.find_exe("lib.exe") self.rc = self.find_exe("rc.exe") # resource compiler self.mc = self.find_exe("mc.exe") # message compiler - self.set_path_env_var('lib') - self.set_path_env_var('include') + if self.__version < 8.0: + self.set_path_env_var('lib') + self.set_path_env_var('include') # extend the MSVC path with the current path - try: - for p in string.split(os.environ['path'], ';'): - self.__paths.append(p) - except KeyError: - pass - os.environ['path'] = string.join(self.__paths, ';') + if self.__version < 8.0: + try: + for p in string.split(os.environ['path'], ';'): + self.__paths.append(p) + except KeyError: + pass + os.environ['path'] = string.join(self.__paths, ';') self.preprocess_options = None - self.compile_options = [ '/nologo', '/Ox', '/MD', '/W3', '/GX' , - '/DNDEBUG'] - self.compile_options_debug = ['/nologo', '/Od', '/MDd', '/W3', '/GX', - '/Z7', '/D_DEBUG'] + if self.__version < 8.0: + self.compile_options = [ + '/nologo', '/Ox', '/MD', '/W3', '/GX' , '/DNDEBUG'] + self.compile_options_debug = [ + '/nologo', '/Od', '/MDd', '/W3', '/GX', '/Z7', '/D_DEBUG'] + else: + # MSC-8.0 using /Ox (=/Ob2gity /Gs) chokes on DLLs containing: + # + # #include + # static void *p = (void *)ceil; + # + # because of the /Oi option (Numeric and NumPy use such code). + # Moreover, the compiler flags /Og as deprecated. + self.compile_options = [ + '/nologo', '/Ob2ty', '/MD', '/W3', '/EHsc' , '/DNDEBUG'] + self.compile_options_debug = [ + '/nologo', '/Od', '/MDd', '/W3', '/EHsc', '/Z7', '/D_DEBUG'] + self.ldflags_shared = ['/DLL', '/nologo', '/INCREMENTAL:NO'] if self.__version >= 7: This is clearly a compiler bug, that may go away with a bug fix. Gerard > -----Original Message----- > From: numpy-discussion-bounces at scipy.org > [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern > Sent: Friday, January 12, 2007 1:55 PM > To: Discussion of Numerical Python > Subject: Re: [Numpy-discussion] failed to build numpy and Numeric in > Windows > > Zhou, Mi wrote: > > Hi, > > > > I am trying to build Numpy-1.01 in windows using "python setup.py > > build". My C compiler is MSVS2005. > > Hmm. I was pretty sure that MSVS2005 was not capable of building Python > extension modules that work with the official Python interpreter builds. > That > doesn't explain or solve the problem that you state, but if the problem > is > restricted to MSVS2005, it may not be worth solving. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma > that is made terrible by our own mad attempt to interpret it as though > it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From adam at thejenkins.org Fri Jan 12 15:37:57 2007 From: adam at thejenkins.org (Adam Jenkins) Date: Fri, 12 Jan 2007 20:37:57 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Overload_binary_operators_with_custo?= =?utf-8?q?m_array=09type?= References: <21ef68730701121109u3bb4b020gda2864d4a65da945@mail.gmail.com> Message-ID: Timothy Hochberg ieee.org> writes: > You want __array_priority__; add an attribute __array_priority__ to your array-like object and set it's value to, say, 10 and ndarrays will defer to your object. Thank you very much. I had seen the __array_priority__ property mentioned in the Numpybook, but it's only documented there as being used to decide which operand's type the result of an operation should be converted to, so I didn't realize it would also be used to decide which operand to defer the operation to. I just tried it, and it does solve the problem for the mathematical operators like + and -, but doesn't seem to work for the comparison operators. That is, if I write lhs < rhs where lhs is a numpy.ndarray and rhs is a myarray, numpy still treats rhs as a scalar and tries to compare each element of lhs to rhs, even though I set myarray.__array_priority__. Is this intentional that the comparison operators ignore __array_priority__, or just a bug in ndarray? Thanks again. Adam From oliphant.travis at ieee.org Fri Jan 12 16:44:51 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 12 Jan 2007 14:44:51 -0700 Subject: [Numpy-discussion] Overload binary operators with custom array type In-Reply-To: References: <21ef68730701121109u3bb4b020gda2864d4a65da945@mail.gmail.com> Message-ID: Adam Jenkins wrote: > Timothy Hochberg ieee.org> writes: > >>You want __array_priority__; add an attribute __array_priority__ to your > > array-like object and set it's value to, say, 10 and ndarrays will defer to your > object. > > Thank you very much. I had seen the __array_priority__ property mentioned in > the Numpybook, but it's only documented there as being used to decide which > operand's type the result of an operation should be converted to, so I didn't > realize it would also be used to decide which operand to defer the operation to. > > I just tried it, and it does solve the problem for the mathematical operators > like + and -, but doesn't seem to work for the comparison operators. That is, > if I write > > lhs < rhs > > where lhs is a numpy.ndarray and rhs is a myarray, numpy still treats rhs as a > scalar and tries to compare each element of lhs to rhs, even though I set > myarray.__array_priority__. Is this intentional that the comparison operators > ignore __array_priority__, or just a bug in ndarray? Probably could be considered a bug. What does less(lhs, rhs) do? -Travis From adam at thejenkins.org Fri Jan 12 17:43:33 2007 From: adam at thejenkins.org (Adam Jenkins) Date: Fri, 12 Jan 2007 22:43:33 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Overload_binary_operators_with_custo?= =?utf-8?q?m_array=09type?= References: <21ef68730701121109u3bb4b020gda2864d4a65da945@mail.gmail.com> Message-ID: Travis Oliphant ieee.org> writes: > > Adam Jenkins wrote: > > I just tried it, and it does solve the problem for the > > mathematical operators like + and -, but doesn't seem to work for > > the comparison operators. That is, if I write > > > > lhs < rhs > > > > where lhs is a numpy.ndarray and rhs is a myarray, numpy still > > treats rhs as a scalar and tries to compare each element of lhs to > > rhs, even though I set myarray.__array_priority__. Is this > > intentional that the comparison operators ignore > > __array_priority__, or just a bug in ndarray? > > > Probably could be considered a bug. > > What does > > less(lhs, rhs) > > do? It does the same thing, calls rhs.__gt__ on each element of lhs. Hmm, I just tried numpy.add(lhs, rhs), and that returns NotImplemented, whereas "lhs+rhs" results in rhs.__radd__(lhs) being called. So "lhs References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <20070112071630.GA1822@clipper.ens.fr> <20070112075403.GB1822@clipper.ens.fr> Message-ID: <45A8136F.1090208@noaa.gov> Gael Varoquaux wrote: > Yes, indeed. I was just wondering if the PEP could be used for a > performant implementation of trees. That would be a whole new PEP, and one we're not the least bit ready for. > Basicaly that is mapping a tree to an > array, which is possible. Possible, but probably not very useful for dense data -- maybe for sparse arrays? The idea of an array API, rather than (actually in addition to) and array data structure is fabulous! It could be used for sparse arrays, for instance. I do think it's a topic for another PEP, and probably not even a PEP until we have at least some working code - maybe a sparse array and/or PIL image? > So at what level do we propose the API. Single-item > access for sure, but what about > > array_interface->get_block_from_slice() ? I think a slicing API is critical -- at least at the Python level, though at the C level is would sure be nice, and probably could allow for some good optimizations for getting a "block" of data out of some odd data structure. "Simple is better than complex." "Although practicality beats purity." "Now is better than never." Tells me that we should just focus on the array data structure for the PEP now. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Fri Jan 12 18:23:27 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 12 Jan 2007 16:23:27 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: <45A8136F.1090208@noaa.gov> References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <20070112071630.GA1822@clipper.ens.fr> <20070112075403.GB1822@clipper.ens.fr> <45A8136F.1090208@noaa.gov> Message-ID: <45A8186F.9010805@ee.byu.edu> Christopher Barker wrote: >Gael Varoquaux wrote: > > >>Yes, indeed. I was just wondering if the PEP could be used for a >>performant implementation of trees. >> >> > >That would be a whole new PEP, and one we're not the least bit ready for. > > > >>Basicaly that is mapping a tree to an >>array, which is possible. >> >> > >Possible, but probably not very useful for dense data -- maybe for >sparse arrays? > >The idea of an array API, rather than (actually in addition to) and >array data structure is fabulous! It could be used for sparse arrays, >for instance. > >I do think it's a topic for another PEP, and probably not even a PEP >until we have at least some working code - maybe a sparse array and/or >PIL image? > > > >>So at what level do we propose the API. Single-item >>access for sure, but what about >> >>array_interface->get_block_from_slice() ? >> >> > >I think a slicing API is critical -- at least at the Python level, >though at the C level is would sure be nice, and probably could allow >for some good optimizations for getting a "block" of data out of some >odd data structure. > >"Simple is better than complex." > >"Although practicality beats purity." > >"Now is better than never." > >Tells me that we should just focus on the array data structure for the >PEP now. > > > This is what I'm thinking of too, and I think what Tim is also concerned about. We have the array object and ctypes types that can expose this array data structure API (I'm calling it the extended buffer protocol) in Python now. -Travis From oliphant at ee.byu.edu Fri Jan 12 18:33:30 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 12 Jan 2007 16:33:30 -0700 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <45A64E66.3080801@ee.byu.edu> <45A71E49.1000701@ee.byu.edu> <45A77C5B.2030409@ee.byu.edu> Message-ID: <45A81ACA.6070305@ee.byu.edu> > > I'm concerned about the direction that this PEP seems to be going. The > original proposal was borderline too complicated IMO, and now it seems > headed in the direction of more complexity. Well at least people are talking about what they would like to see. But, I think we should reign in the discussion. > 1. Things that act like arrays, but have different storage methods. > This details of this still seem pretty vague, but to the extent > that I can figure them out, it doesn't seem useful or necessary > to tie this into the rest of the array interface PEP. For > example, "array_interface->get_block_from_slice()" has been > mentioned. Why that instead of > "PyObject_AsExtendedBuffer(PyObject_GetItem(index), ....)"[2]. > I'll stop here, till I see some more details of what people have > in mind, but at this point, I think that alternative memory > models are a different problem that should be addressed separately. > I'm leaning this way too. > Sadly, I'm leaving town shortly and I'm running out of time, so I'll > have to leave my objections in this somewhat vague state. > > Oh, the way that F. Lundh plans to expose PIL's data a chunk at a time > is mentioned in this python-dev summary: > http://www.python.org/dev/summary/2006-11-01_2006-11-15/ > It doesn't seem necessary to have special support for this; all that > is necessary is for the object returned by acquire_view to support the > extended array protocol. Yes, I agree. > > [1] Remind me again why we can't simply use ctypes for this? It's > already in the core. I'm sure it's less efficient, but you shouldn't > need to parse the data structure information very often. I suspect > that something that leveraged ctypes would meet less resistance. > Two reasons: 1) ctypes wasn't designed for this purpose specifically and leaves out certain things 2) ctypes uses many Python types instead of just a single python type (the PyDataFormatOjbect). > [2] Which reminds me. I never saw in the PEP what the actual call in > the buffer protocol was supposed to look like. Is it something like: > PyObject_AsExtendedBuffer(PyObject * obj, void **buffer, Py_ssize_t > *buffer_len, funcptr *bf_getarrayview, funcptr *bf_relarrayview) No, not like that. The bf_getarrayview function pointers hang-off of the as_buffer method table which is pointed to by the typeobject. You could always access the API using those function pointers, but it is more traditional to use an API call which adds some checking to make sure the function pointer is there and then calling the function pointer. I don't know if I go into a lot of detail there, but I should probably add more. PEP's are rather "expensive" for me in terms of how much immediate benefit the change to Python is to me personally versus that time spent writing them. The benefit here is much more long term in establishing a useful data-model that could be used by a lot of applications in Python to exchange data (and help ameliorate the proliferation of objects in Python that are essentially and should be NumPy arrays). -Travis From Chris.Barker at noaa.gov Fri Jan 12 19:09:47 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 12 Jan 2007 16:09:47 -0800 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A74781.7080505@ar.media.kyoto-u.ac.jp> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A718BF.8060202@ar.media.kyoto-u.ac.jp> <45A72CBE.7080609@ar.media.kyoto-u.ac.jp> <45A74781.7080505@ar.media.kyoto-u.ac.jp> Message-ID: <45A8234B.7000801@noaa.gov> I think it may have all been cleared up now, but just in case: >>> a array([[[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]], [[12, 13, 14, 15], [16, 17, 18, 19], [20, 21, 22, 23]]]) >>> a.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False So here is the most common case: Contiguous, aligned, etc. This means the data buffer is a "normal C array", and you can iterate through it in the normal C way. >>> b = a[:,2,:] >>> b array([[ 8, 9, 10, 11], [20, 21, 22, 23]]) So b is a slice pulled out of a, sharing the same data black, but not all of it! >>> b.flags C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False >>> So it is not longer a Contiguous array. This means that you are looking at the same block of memory as above, but only some of it is used for this array, and you need to use the strides to get at the elements that are part of this array. This is all aside from alignment issues, of which I know nothing. > So the data buffer in numpy is not really a C array, but more like a > structure ? (I am talking about the actual binary data) Is is a C array of bytes, and probably of the given type, but not all of it is used for this particular array object, or, in the case of Fortran ordering, it's used in a different order that a conventional C n-d array. > "for a numpy array a of eg float32, am I guaranteed that > a->data[sizeof(float32) * i] for 0 <= i < a.size gives me all the items > of a, even for non contiguous arrays ?" I think you got the answer to this, but in case you didn't: No, only if it's contiguous (I'm not sure about alignment) >> First, it's not that important if the array is contiguous for this >> > sort of thing. What you really care about is whether it's >> > simply-strided (or maybe single-strided would be a better term) but how do you get single strided? this is what always made it hard for me to know how to write this kind of code. >> > Anyway, the last dimension of the array can be strided without making >> > things more difficult. All you need to be able to do is to address the >> > elements of the array as thedata[offset + stride*index]. But how do you make it that simple for n-d arrays? that only works for 1-d. What I have come up with are some macros for the common cases -- 1-d, 2-d, 3-d: #define ARRAYVAL2(aType,a,i,j) ( *(aType *)(a->data + (i)*a->strides[0] + (j)*a->strides[1])) #define ARRAYVAL3(aType,a,i,j,k) ( *(aType *)(a->data + (i)*a->strides[0] + (j)*a->strides[1] + (k)*a->strides[2])) (this was for Numeric -- maybe it's slightly different no) This involves a fair bit of math for each index operation -- it could really slow down a simple function like clip. To do this for the general case, the only thing I could come up with was recursion, which seemed a bit heavy-weight, so I went with looping through all the indices to compute the offset. Ugly and slow. > Now, it is exposed in the concept of an array iterator. Anybody > can take advantage of it as it there is a C-API call to get an array > iterator from the array Is this iterator as efficient as incrementing the index for contiguous arrays? i.e. is there any point of special casing contiguous arrays if you use it? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Fri Jan 12 19:19:59 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 12 Jan 2007 17:19:59 -0700 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A8234B.7000801@noaa.gov> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <20070110214923.GP16625@mentat.za.net> <45A5746B.4080002@noaa.gov> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A718BF.8060202@ar.media.kyoto-u.ac.jp> <45A72CBE.7080609@ar.media.kyoto-u.ac.jp> <45A74781.7080505@ar.media.kyoto-u.ac.jp> <45A8234B.7000801@noaa.gov> Message-ID: <45A825AF.1070702@ee.byu.edu> Christopher Barker wrote: >>Now, it is exposed in the concept of an array iterator. Anybody >>can take advantage of it as it there is a C-API call to get an array >>iterator from the array >> >> > >Is this iterator as efficient as incrementing the index for contiguous >arrays? i.e. is there any point of special casing contiguous arrays if >you use it? > > Contiguous arrays are special-cased already (as are 1-d arrays), so not really. There is a little-bit of overhead in testing for the special-case that you can overcome by testing for the special-case outside the looping structure on your own. I'm not sure how much speed gain is accomplished doing that. -Travis From cournape at gmail.com Sat Jan 13 00:42:36 2007 From: cournape at gmail.com (David Cournapeau) Date: Sat, 13 Jan 2007 14:42:36 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <45A8234B.7000801@noaa.gov> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A718BF.8060202@ar.media.kyoto-u.ac.jp> <45A72CBE.7080609@ar.media.kyoto-u.ac.jp> <45A74781.7080505@ar.media.kyoto-u.ac.jp> <45A8234B.7000801@noaa.gov> Message-ID: <5b8d13220701122142s6e6c3f55rd28d6c21f4d36eba@mail.gmail.com> On 1/13/07, Christopher Barker wrote: > I think it may have all been cleared up now, but just in case: > > >>> a > array([[[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11]], > > [[12, 13, 14, 15], > [16, 17, 18, 19], > [20, 21, 22, 23]]]) > >>> a.flags > C_CONTIGUOUS : True > F_CONTIGUOUS : False > OWNDATA : True > WRITEABLE : True > ALIGNED : True > UPDATEIFCOPY : False > > So here is the most common case: Contiguous, aligned, etc. This means > the data buffer is a "normal C array", and you can iterate through it in > the normal C way. > > >>> b = a[:,2,:] > >>> b > array([[ 8, 9, 10, 11], > [20, 21, 22, 23]]) > > So b is a slice pulled out of a, sharing the same data black, but not > all of it! > > >>> b.flags > C_CONTIGUOUS : False > F_CONTIGUOUS : False > OWNDATA : False > WRITEABLE : True > ALIGNED : True > UPDATEIFCOPY : False > >>> > > So it is not longer a Contiguous array. This means that you are looking > at the same block of memory as above, but only some of it is used for > this array, and you need to use the strides to get at the elements that > are part of this array. > I finally understood that point, which prevents me from understanding the real issue. I totally forgot about views which are not copies in numpy (contrary to matlab, where my experience in C extension lies), which makes for all those case... I kepts thinking how come you could create a non aligned or a non packed array in numpy, but once you start thinking about views and everything, it all makes sense... So basically, in my case, I do not care about F_CONTIGUOUS vs C_CONTIGUOUS, but I care about contiguity. So my special case is really F_CONTIGUOUS or C_CONTIGUOUS. > This is all aside from alignment issues, of which I know nothing. > > > So the data buffer in numpy is not really a C array, but more like a > > structure ? (I am talking about the actual binary data) > > Is is a C array of bytes, and probably of the given type, but not all of > it is used for this particular array object, or, in the case of Fortran > ordering, it's used in a different order that a conventional C n-d array. > > > "for a numpy array a of eg float32, am I guaranteed that > > a->data[sizeof(float32) * i] for 0 <= i < a.size gives me all the items > > of a, even for non contiguous arrays ?" > > I think you got the answer to this, but in case you didn't: > > No, only if it's contiguous (I'm not sure about alignment) According to numpy ebook, you need alignement for deferencing pointers. But I guess non aligned arrays are not really common. > > >> First, it's not that important if the array is contiguous for this > >> > sort of thing. What you really care about is whether it's > >> > simply-strided (or maybe single-strided would be a better term) > > but how do you get single strided? this is what always made it hard for > me to know how to write this kind of code. I am also interested in that. You could imagine that some arrays are neither C or F contiguous, but still are "packed" (using some concatenatation, for example ?), and as such can be iterated easily. > > This involves a fair bit of math for each index operation -- it could > really slow down a simple function like clip. Indeed. But again, slower operations on those arrays is to be expected, and speeding them up is not all that important. At least, it is not important for the case I want to speed up. > > Is this iterator as efficient as incrementing the index for contiguous > arrays? i.e. is there any point of special casing contiguous arrays if > you use it? I think the problem is not that much incrementing the iterator, but more having contiguous memory access. For example, the iterator may not guarantee traversing the array "contiguously" even if the array is contiguous (I didn't look at the sources for the array iterator). If for example you access a C order array but iterating the F way, it will be much slower, because your computation becomes limited by main memory's speed. For some code related to linear prediction coding, I tried to take into account C_CONTIGUOUS vs other cases, and I realised that in most cases, it is not only easier but often faster to copy the data in a new contiguous data buffer and doing the computation on that. Basically, at least for those cases, it didn't worth the effort to handle complicated cases (the memory buffer being not that big). David From charlesr.harris at gmail.com Sat Jan 13 01:32:43 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 12 Jan 2007 23:32:43 -0700 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <5b8d13220701122142s6e6c3f55rd28d6c21f4d36eba@mail.gmail.com> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A718BF.8060202@ar.media.kyoto-u.ac.jp> <45A72CBE.7080609@ar.media.kyoto-u.ac.jp> <45A74781.7080505@ar.media.kyoto-u.ac.jp> <45A8234B.7000801@noaa.gov> <5b8d13220701122142s6e6c3f55rd28d6c21f4d36eba@mail.gmail.com> Message-ID: On 1/12/07, David Cournapeau wrote: > > On 1/13/07, Christopher Barker wrote: > > I think it may have all been cleared up now, but just in case: > > > > > but how do you get single strided? this is what always made it hard for > > me to know how to write this kind of code. Make a copy of the array. New copies are C_CONTIGUOUS by default. There is also the ascontiguousarray function: ascontiguousarray(a, dtype=None) Return 'a' as an array contiguous in memory (C order). Which makes a copy only when required. Since clip returns a new array anyway, this shouldn't be a problem except when clipping against another array, which you would also want to be contiguous. In practice, I don't think using ascontiguousarray for the clipping array would add much overhead as the array would likely be contiguous in the first place. You would probably want to match data types also. I think the needed ops are already implemented for the types at the c-level, __gt__ for instance, and in that case you can simply use the function pointer with some loss of speed. If you can determine the corresponding c-type, and I think you can, then it shouldn't be too much trouble to implement type specific clipping, but it might not be worth the effort. There are lots of code snippets in other functions similar to what you need that could be a good starting point. You just need to find them 8^) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Jan 13 02:04:47 2007 From: cournape at gmail.com (David Cournapeau) Date: Sat, 13 Jan 2007 16:04:47 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <45A718BF.8060202@ar.media.kyoto-u.ac.jp> <45A72CBE.7080609@ar.media.kyoto-u.ac.jp> <45A74781.7080505@ar.media.kyoto-u.ac.jp> <45A8234B.7000801@noaa.gov> <5b8d13220701122142s6e6c3f55rd28d6c21f4d36eba@mail.gmail.com> Message-ID: <5b8d13220701122304t2eaa1em6ae372c418637a96@mail.gmail.com> On 1/13/07, Charles R Harris wrote: > > > On 1/12/07, David Cournapeau wrote: > > On 1/13/07, Christopher Barker wrote: > > > I think it may have all been cleared up now, but just in case: > > > > > > > > but how do you get single strided? this is what always made it hard for > > > me to know how to write this kind of code. > > > > Make a copy of the array. New copies are C_CONTIGUOUS by default. There is > also the ascontiguousarray function: > > ascontiguousarray(a, dtype=None) > Return 'a' as an array contiguous in memory (C order). > > Which makes a copy only when required. Since clip returns a new array > anyway, This is not always true: clip as a array member functions accepts out as a keyword, even is out is the same than input. Something like a.clip(min, max, a) does work. I actually do not understand this out argument. Doing a.clip(min, max, a) for in place is a bit strange, isn't it ? Part of the problem is that I get the impression that clip has some behaviour which really depends on the current implementation (using addition and so on). For example, a.clip(0., 1.) does return a float32 array when a is a float32, and fails if a is an integer. There is no upcast, and I don't see any reason why; I thought that numpy was supposed to (try to ) upcast any necessary inputs when necessary ? Doing an equivalent of putmask(a, aM, M) is easier, but is not compatible with the current clip for many cases. this shouldn't be a problem except when clipping against another > array, which you would also want to be contiguous. In practice, I don't > think using ascontiguousarray for the clipping array would add much overhead > as the array would likely be contiguous in the first place. You would > probably want to match data types also. I think the needed ops are already > implemented for the types at the c-level, __gt__ for instance, and in that > case you can simply use the function pointer with some loss of speed. If you > can determine the corresponding c-type, and I think you can, then it > shouldn't be too much trouble to implement type specific clipping, but it > might not be worth the effort. Having type specific clip operation is really trivial compared to all other problems: a few lines of templated code, and generating them for all basic C types. The problem really is to know when you can use those functions, and the rules for conversion while keeping the same semantics than the original clip function which are quirky in some cases, David From torgil.svensson at gmail.com Sat Jan 13 02:44:35 2007 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Sat, 13 Jan 2007 08:44:35 +0100 Subject: [Numpy-discussion] Latest Array-Interface PEP In-Reply-To: References: <459D6964.30701@ee.byu.edu> <45A71E49.1000701@ee.byu.edu> <45A77C5B.2030409@ee.byu.edu> Message-ID: On 1/12/07, Timothy Hochberg wrote: > > > On 1/12/07, Torgil Svensson wrote: > > On 1/12/07, Timothy Hochberg wrote: > > [CHOP] > > > > > the core. I'm sure it's less efficient, but you shouldn't need to parse > the > > > data structure information very often. > > > > I believe that'll be more common than you think; for example > > dynamically creating/combining/slicing recarrays with various data. > > I was unclear here. I didn't mean like that it would be infrequent in "once > a month" sense. I meant that you would only need to look at the data > structure information once per set of data that you are accessing and that > you would typically extract many chunks of data from each set, so the > amortized cost of parsing the data structure would be small. That's correct, it's not a performance issue as I can see it. //Torgil From asafdav2 at gmail.com Sat Jan 13 05:59:51 2007 From: asafdav2 at gmail.com (asaf david) Date: Sat, 13 Jan 2007 12:59:51 +0200 Subject: [Numpy-discussion] About extending NumPy with c Message-ID: Hello i want to write a c function that receive and process a numpy array. to do so, i'm following this tutorial http://numpy.scipy.org/numpydoc/numpy-13.html as a start, i copy pasted the "A Simple Example" from the tutorial and compiled the module as usual with no errors/warnings. however, when trying to run the function on a legal numpy array, i'm getting an error message: "python.exe has encountered a problem and needs to close, send, dont send etc' i've narrowed it down and it happens when running this line if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &array)) any ideas how to solve it? thanks here's my complete c file: #include #include static PyObject *trace(PyObject *self, PyObject *args) { PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &array)) return NULL; if (array->nd != 2 || array->descr->type_num != PyArray_DOUBLE) { PyErr_SetString(PyExc_ValueError, "array must be two-dimensional and of type float"); return NULL; } n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); return PyFloat_FromDouble(sum); } static PyMethodDef foo_methods[] = { { "trace", trace, METH_VARARGS, NULL}, { NULL, NULL, 0, NULL } }; PyMODINIT_FUNC initfoo() { Py_InitModule3("foo", foo_methods, "bla bla"); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Sat Jan 13 09:20:15 2007 From: faltet at carabos.com (Francesc Altet) Date: Sat, 13 Jan 2007 15:20:15 +0100 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <5b8d13220701122142s6e6c3f55rd28d6c21f4d36eba@mail.gmail.com> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <45A5CE23.7040608@ar.media.kyoto-u.ac.jp> <45A6CD6C.2030709@noaa.gov> <45A718BF.8060202@ar.media.kyoto-u.ac.jp> <45A72CBE.7080609@ar.media.kyoto-u.ac.jp> <45A74781.7080505@ar.media.kyoto-u.ac.jp> <45A8234B.7000801@noaa.gov> <5b8d13220701122142s6e6c3f55rd28d6c21f4d36eba@mail.gmail.com> Message-ID: <1168698015.2584.31.camel@localhost.localdomain> El ds 13 de 01 del 2007 a les 14:42 +0900, en/na David Cournapeau va escriure: > > No, only if it's contiguous (I'm not sure about alignment) > According to numpy ebook, you need alignement for deferencing > pointers. But I guess non aligned arrays are not really common. Unaligned arrays can be quite common if you use recarrays often: In [18]:a=numpy.empty(3, dtype="S1,i4,f8") In [19]:a.dtype.fields.items() Out[19]: [('f0', (dtype('|S1'), 0)), ('f1', (dtype('int32'), 1)), ('f2', (dtype('float64'), 5))] In [20]:f1=a['f1'] In [21]:f1.flags Out[21]: C_CONTIGUOUS : False F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : False UPDATEIFCOPY : False And dereferencing unaligned pointers do cause a crash in general, but not in all architectures (i386 being one of them). Cheers, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From rdmoores at gmail.com Sat Jan 13 09:36:03 2007 From: rdmoores at gmail.com (Dick Moores) Date: Sat, 13 Jan 2007 06:36:03 -0800 Subject: [Numpy-discussion] Can numpy compute ln(640320**3 + 744)/163**.5 to 30 places? Message-ID: Can python compute ln(640320**3 + 744)/163**.5 to 30 places? I ran across this the other day. It's value is said to be the same as pi to about 30 places. I'd like to know if it is possible to do this in python with numpy. A friend pointed out that the Windows Calculator gives about 30 significant digits. I tried it and got 3.1415926535897932384626433832797. Pi to 32 places is 3.14159265358979323846264338327950, so the calculator is accurate to 31 significant digits (including the 3.). So, how about numpy? Thanks, Dick Moores Win XP Pro Python 2.5 yet to install numpy From cournape at gmail.com Sat Jan 13 11:07:33 2007 From: cournape at gmail.com (David Cournapeau) Date: Sun, 14 Jan 2007 01:07:33 +0900 Subject: [Numpy-discussion] Need help for implementing a fast clip in numpy (was slow clip) In-Reply-To: <1168698015.2584.31.camel@localhost.localdomain> References: <45A3A46A.4040208@ar.media.kyoto-u.ac.jp> <45A718BF.8060202@ar.media.kyoto-u.ac.jp> <45A72CBE.7080609@ar.media.kyoto-u.ac.jp> <45A74781.7080505@ar.media.kyoto-u.ac.jp> <45A8234B.7000801@noaa.gov> <5b8d13220701122142s6e6c3f55rd28d6c21f4d36eba@mail.gmail.com> <1168698015.2584.31.camel@localhost.localdomain> Message-ID: <5b8d13220701130807t205341a6la0b48417016d6003@mail.gmail.com> On 1/13/07, Francesc Altet wrote: > El ds 13 de 01 del 2007 a les 14:42 +0900, en/na David Cournapeau va > escriure: > > > No, only if it's contiguous (I'm not sure about alignment) > > According to numpy ebook, you need alignement for deferencing > > pointers. But I guess non aligned arrays are not really common. > > Unaligned arrays can be quite common if you use recarrays often: Well, this gives me one more unit test :) I didn't know any way of creating non aligned arrays... Thank you. cheers, David From cournape at gmail.com Sat Jan 13 11:14:02 2007 From: cournape at gmail.com (David Cournapeau) Date: Sun, 14 Jan 2007 01:14:02 +0900 Subject: [Numpy-discussion] About extending NumPy with c In-Reply-To: References: Message-ID: <5b8d13220701130814j58dfccb7o7bdd60f296e3eb33@mail.gmail.com> On 1/13/07, asaf david wrote: > Hello > i want to write a c function that receive and process a numpy array. to do > so, i'm following this tutorial > http://numpy.scipy.org/numpydoc/numpy-13.html > as a start, i copy pasted the "A Simple Example" from the tutorial and > compiled the module as usual with no errors/warnings. > however, when trying to run the function on a legal numpy array, i'm getting > an error message: > "python.exe has encountered a problem and needs to close, send, dont send > etc' > i've narrowed it down and it happens when running this line > > if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &array)) > > any ideas how to solve it? > thanks Well, the problem is pretty basic, but you would have a really hard time to find it.... The problem happens at that line, but not because of that line... It would happen with any call to the numpy C API (the first one for that matter). > PyMODINIT_FUNC initfoo() { > Py_InitModule3("foo", foo_methods, "bla bla"); > } > Here is your problem ! in your init function, you HAVE to call import_array(): PyMODINIT_FUNC initfoo() { Py_InitModule("foo", foo_methods, "bla bla"); import_array(); } This initializes the numpy C API. Without this call, all functions of the numpy C API are actually dangling pointers, and do not point to any valid function. David From kwgoodman at gmail.com Sat Jan 13 13:01:59 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 13 Jan 2007 10:01:59 -0800 Subject: [Numpy-discussion] random permutation In-Reply-To: <45A6A0FA.2060504@gmail.com> References: <45A6A0FA.2060504@gmail.com> Message-ID: On 1/11/07, Robert Kern wrote: > Keith Goodman wrote: > > Why is the first element of the permutation always the same? Am I > > using random.permutation in the right way? > > >>> M.__version__ > > '1.0rc1' > > This has been fixed in more recent versions. > > http://projects.scipy.org/scipy/numpy/ticket/374 I don't see any unit tests for numpy.random. I guess randomness is hard to test. Would it help to seed the random number generator and at least check that you get the same result you got before? No existing problems would be found. But new ones might be caught. I guess the alternative is a long running, occasionally failing, statistical tests. From seb.haase at gmx.net Sat Jan 13 13:16:17 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Sat, 13 Jan 2007 10:16:17 -0800 Subject: [Numpy-discussion] random permutation In-Reply-To: References: <45A6A0FA.2060504@gmail.com> Message-ID: On 1/13/07, Keith Goodman wrote: > On 1/11/07, Robert Kern wrote: > > Keith Goodman wrote: > > > Why is the first element of the permutation always the same? Am I > > > using random.permutation in the right way? > > > > >>> M.__version__ > > > '1.0rc1' > > > > This has been fixed in more recent versions. > > > > http://projects.scipy.org/scipy/numpy/ticket/374 > > I don't see any unit tests for numpy.random. I guess randomness is hard to test. > > Would it help to seed the random number generator and at least check > that you get the same result you got before? No existing problems > would be found. But new ones might be caught. > Hi, Is it guaranteed that a given seed produces the same sequence of rand-number between different platforms ? I thought this might only be guaranteed for "any given computer" to reproduce the same numbers. -Sebastian Haase From kwgoodman at gmail.com Sat Jan 13 13:26:45 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 13 Jan 2007 10:26:45 -0800 Subject: [Numpy-discussion] random permutation In-Reply-To: References: <45A6A0FA.2060504@gmail.com> Message-ID: On 1/13/07, Sebastian Haase wrote: > On 1/13/07, Keith Goodman wrote: > > On 1/11/07, Robert Kern wrote: > > > Keith Goodman wrote: > > > > Why is the first element of the permutation always the same? Am I > > > > using random.permutation in the right way? > > > > > > >>> M.__version__ > > > > '1.0rc1' > > > > > > This has been fixed in more recent versions. > > > > > > http://projects.scipy.org/scipy/numpy/ticket/374 > > > > I don't see any unit tests for numpy.random. I guess randomness is hard to test. > > > > Would it help to seed the random number generator and at least check > > that you get the same result you got before? No existing problems > > would be found. But new ones might be caught. > > > Hi, > Is it guaranteed that a given seed produces the same sequence of > rand-number between different platforms ? > I thought this might only be guaranteed for "any given computer" to > reproduce the same numbers. I hope, and expect, that it is system independent. Here's what I get: >> rs = numpy.random.RandomState([123, 901, 789]) >> rs.randn(4,1) array([[ 0.76072026], [ 1.27712191], [ 0.03497453], [ 0.09056668]]) >> rs.rand(4,1) array([[ 0.184306 ], [ 0.58967936], [ 0.52425903], [ 0.33389408]]) >> numpy.__version__ '1.0.1' Linux kel 2.6.18-3-686 #1 SMP Mon Dec 4 16:41:14 UTC 2006 i686 GNU/Linux From seb.haase at gmx.net Sat Jan 13 14:58:53 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Sat, 13 Jan 2007 11:58:53 -0800 Subject: [Numpy-discussion] random permutation In-Reply-To: References: <45A6A0FA.2060504@gmail.com> Message-ID: On 1/13/07, Keith Goodman wrote: > On 1/13/07, Sebastian Haase wrote: > > On 1/13/07, Keith Goodman wrote: > > > On 1/11/07, Robert Kern wrote: > > > > Keith Goodman wrote: > > > > > Why is the first element of the permutation always the same? Am I > > > > > using random.permutation in the right way? > > > > > > > > >>> M.__version__ > > > > > '1.0rc1' > > > > > > > > This has been fixed in more recent versions. > > > > > > > > http://projects.scipy.org/scipy/numpy/ticket/374 > > > > > > I don't see any unit tests for numpy.random. I guess randomness is hard to test. > > > > > > Would it help to seed the random number generator and at least check > > > that you get the same result you got before? No existing problems > > > would be found. But new ones might be caught. > > > > > Hi, > > Is it guaranteed that a given seed produces the same sequence of > > rand-number between different platforms ? > > I thought this might only be guaranteed for "any given computer" to > > reproduce the same numbers. > > I hope, and expect, that it is system independent. > > Here's what I get: > > >> rs = numpy.random.RandomState([123, 901, 789]) > >> rs.randn(4,1) > > array([[ 0.76072026], > [ 1.27712191], > [ 0.03497453], > [ 0.09056668]]) > >> rs.rand(4,1) > > array([[ 0.184306 ], > [ 0.58967936], > [ 0.52425903], > [ 0.33389408]]) > >> numpy.__version__ > '1.0.1' > > Linux kel 2.6.18-3-686 #1 SMP Mon Dec 4 16:41:14 UTC 2006 i686 GNU/Linux import numpy rs = numpy.random.RandomState([123, 901, 789]) rs.randn(4,1) [[ 0.76072026] [ 1.27712191] [ 0.03497453] [ 0.09056668]] rs.rand(4,1) [[ 0.184306 ] [ 0.58967936] [ 0.52425903] [ 0.33389408]] numpy.__version__ '1.0rc1' Windows XP- pentium4 - (non current numpy) - Looks promising - but how about PowerPC macs ... -Sebastian From kwgoodman at gmail.com Sat Jan 13 15:10:02 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 13 Jan 2007 12:10:02 -0800 Subject: [Numpy-discussion] random permutation In-Reply-To: References: <45A6A0FA.2060504@gmail.com> Message-ID: On 1/13/07, Sebastian Haase wrote: > On 1/13/07, Keith Goodman wrote: > > On 1/13/07, Sebastian Haase wrote: > > > On 1/13/07, Keith Goodman wrote: > > > > On 1/11/07, Robert Kern wrote: > > > > > Keith Goodman wrote: > > > > > > Why is the first element of the permutation always the same? Am I > > > > > > using random.permutation in the right way? > > > > > > > > > > >>> M.__version__ > > > > > > '1.0rc1' > > > > > > > > > > This has been fixed in more recent versions. > > > > > > > > > > http://projects.scipy.org/scipy/numpy/ticket/374 > > > > > > > > I don't see any unit tests for numpy.random. I guess randomness is hard to test. > > > > > > > > Would it help to seed the random number generator and at least check > > > > that you get the same result you got before? No existing problems > > > > would be found. But new ones might be caught. > > > > > > > Hi, > > > Is it guaranteed that a given seed produces the same sequence of > > > rand-number between different platforms ? > > > I thought this might only be guaranteed for "any given computer" to > > > reproduce the same numbers. > > > > I hope, and expect, that it is system independent. > > > > Here's what I get: > > > > >> rs = numpy.random.RandomState([123, 901, 789]) > > >> rs.randn(4,1) > > > > array([[ 0.76072026], > > [ 1.27712191], > > [ 0.03497453], > > [ 0.09056668]]) > > >> rs.rand(4,1) > > > > array([[ 0.184306 ], > > [ 0.58967936], > > [ 0.52425903], > > [ 0.33389408]]) > > >> numpy.__version__ > > '1.0.1' > > > > Linux kel 2.6.18-3-686 #1 SMP Mon Dec 4 16:41:14 UTC 2006 i686 GNU/Linux > import numpy > rs = numpy.random.RandomState([123, 901, 789]) > rs.randn(4,1) > [[ 0.76072026] > [ 1.27712191] > [ 0.03497453] > [ 0.09056668]] > rs.rand(4,1) > [[ 0.184306 ] > [ 0.58967936] > [ 0.52425903] > [ 0.33389408]] > numpy.__version__ > '1.0rc1' > > Windows XP- pentium4 - (non current numpy) - > > Looks promising - but how about PowerPC macs ... The random numbers are generated by a fixed algorithm. So they are not random at all. As long as everyone is using float64 I would think we'd all get the same (pseudo) random numbers. From seb.haase at gmx.net Sat Jan 13 15:28:14 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Sat, 13 Jan 2007 12:28:14 -0800 Subject: [Numpy-discussion] random permutation In-Reply-To: References: <45A6A0FA.2060504@gmail.com> Message-ID: On 1/13/07, Keith Goodman wrote: > On 1/13/07, Sebastian Haase wrote: > > On 1/13/07, Keith Goodman wrote: > > > On 1/13/07, Sebastian Haase wrote: > > > > On 1/13/07, Keith Goodman wrote: > > > > > On 1/11/07, Robert Kern wrote: > > > > > > Keith Goodman wrote: > > > > > > > Why is the first element of the permutation always the same? Am I > > > > > > > using random.permutation in the right way? > > > > > > > > > > > > >>> M.__version__ > > > > > > > '1.0rc1' > > > > > > > > > > > > This has been fixed in more recent versions. > > > > > > > > > > > > http://projects.scipy.org/scipy/numpy/ticket/374 > > > > > > > > > > I don't see any unit tests for numpy.random. I guess randomness is hard to test. > > > > > > > > > > Would it help to seed the random number generator and at least check > > > > > that you get the same result you got before? No existing problems > > > > > would be found. But new ones might be caught. > > > > > > > > > Hi, > > > > Is it guaranteed that a given seed produces the same sequence of > > > > rand-number between different platforms ? > > > > I thought this might only be guaranteed for "any given computer" to > > > > reproduce the same numbers. > > > > > > I hope, and expect, that it is system independent. > > > > > > Here's what I get: > > > > > > >> rs = numpy.random.RandomState([123, 901, 789]) > > > >> rs.randn(4,1) > > > > > > array([[ 0.76072026], > > > [ 1.27712191], > > > [ 0.03497453], > > > [ 0.09056668]]) > > > >> rs.rand(4,1) > > > > > > array([[ 0.184306 ], > > > [ 0.58967936], > > > [ 0.52425903], > > > [ 0.33389408]]) > > > >> numpy.__version__ > > > '1.0.1' > > > > > > Linux kel 2.6.18-3-686 #1 SMP Mon Dec 4 16:41:14 UTC 2006 i686 GNU/Linux > > import numpy > > rs = numpy.random.RandomState([123, 901, 789]) > > rs.randn(4,1) > > [[ 0.76072026] > > [ 1.27712191] > > [ 0.03497453] > > [ 0.09056668]] > > rs.rand(4,1) > > [[ 0.184306 ] > > [ 0.58967936] > > [ 0.52425903] > > [ 0.33389408]] > > numpy.__version__ > > '1.0rc1' > > > > Windows XP- pentium4 - (non current numpy) - > > > > Looks promising - but how about PowerPC macs ... > > The random numbers are generated by a fixed algorithm. So they are not > random at all. As long as everyone is using float64 I would think we'd > all get the same (pseudo) random numbers. I understand this - but I thought the algorithm might involve some rounding(-error) cases that would produce different results especially when changing the CPU (intel vs. PowerPC). I found this to be true even for non pseudo-random code that between CPU-types results were different within a given epsilon. -Sebastian From stefan at sun.ac.za Sat Jan 13 16:15:55 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sat, 13 Jan 2007 21:15:55 +0000 Subject: [Numpy-discussion] random permutation In-Reply-To: References: <45A6A0FA.2060504@gmail.com> Message-ID: <20070113211555.GA2727@zaphod.lagged.za.net> On Sat, Jan 13, 2007 at 10:01:59AM -0800, Keith Goodman wrote: > On 1/11/07, Robert Kern wrote: > > Keith Goodman wrote: > > > Why is the first element of the permutation always the same? Am I > > > using random.permutation in the right way? > > > > >>> M.__version__ > > > '1.0rc1' > > > > This has been fixed in more recent versions. > > > > http://projects.scipy.org/scipy/numpy/ticket/374 > > I don't see any unit tests for numpy.random. I guess randomness is hard to test. Every time we fix a bug, we add a corresponding test to make sure that it doesn't pop up again. In this case, take a look in numpy/core/tests/test_regression.py: def check_random_shuffle(self, level=rlevel): """Ticket #374""" a = N.arange(5).reshape((5,1)) b = a.copy() N.random.shuffle(b) assert_equal(sorted(b),a) Cheers St?fan From erin.sheldon at gmail.com Sat Jan 13 20:09:14 2007 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Sat, 13 Jan 2007 17:09:14 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <45A47C8A.1040300@gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> <331116dc0701080038wf58be61xcced1cc757198d65@mail.gmail.com> <45A2060C.8010909@gmail.com> <331116dc0701092115g2566fb3tc93f3ebcfbb59a01@mail.gmail.com> <45A47C8A.1040300@gmail.com> Message-ID: <331116dc0701131709r5b675921p4508edf33004c355@mail.gmail.com> On 1/9/07, Robert Kern wrote: > Erin Sheldon wrote: > > I'm finally getting to this, I'm on the road. Here is > > what gdb gives me > > > > --snip-- > > Reading symbols for shared libraries . done > > Reading symbols for shared libraries . done > > Reading symbols for shared libraries .. done > > > > Program received signal EXC_BAD_ACCESS, Could not access memory. > > Reason: KERN_INVALID_ADDRESS at address: 0x5a6ae09a > > parse_fond (fond_data=0x5a6ae060
, > > have_sfnt=0x5a6ae09c, sfnt_id=0xbfff8f4e, lwfn_file_name=0xbfff8e4a > > "", face_index=0) at > > /usr/local/src/freetype-2.2.1/src/base/ftmac.c:628 > > 628 /usr/local/src/freetype-2.2.1/src/base/ftmac.c: No such file > > or directory. > > in /usr/local/src/freetype-2.2.1/src/base/ftmac.c > > > > For some reason it is looking for something from freetype > > in /usr/local/src but this directory has never existed > > on my computer. This is a new computer and I used the migration > > assistant to copy things from my old computer, so perhaps some > > paths got messed up. I don't know how to address > > that type of issue on a Mac with no ld.so.conf type of file. > > Your build of matplotlib is probably picking up a different freetype library > than the one you intended, one which somebody else built. Possibly the one in > /usr/X11R6/lib/. You can use otool(1) to figure out what shared libraries your > extension modules are linking against: > > > [matplotlib]$ otool -L ft2font.so > ft2font.so: > /opt/local/lib/libfreetype.6.dylib (compatibility version 10.0.0, > current version 10.8.0) > /opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current > version 1.2.3) > /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version > 7.4.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version > 88.3.3) > /usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current > version 1.0.0) Here is what I get > otool -L /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/ft2font.so /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/ft2font.so: /usr/local/lib/libfreetype.6.dylib (compatibility version 10.0.0, current version 10.10.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3) /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.3) /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0) Is there something wrong with this? I still don't seen anything about /usr/local/src/freetype in here (btw I switched from fink to the versions installed by i-installer because the odd warnings disappeared when compiling matplotlib; the same segfault occurs with either). Erin From robert.kern at gmail.com Sat Jan 13 20:16:57 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 13 Jan 2007 19:16:57 -0600 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <331116dc0701131709r5b675921p4508edf33004c355@mail.gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <0259BACC-F404-4825-821F-ED27B41F7D6A@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> <331116dc0701080038wf58be61xcced1cc757198d65@mail.gmail.com> <45A2060C.8010909@gmail.com> <331116dc0701092115g2566fb3tc93f3ebcfbb59a01@mail.gmail.com> <45A47C8A.1040300@gmail.com> <331116dc0701131709r5b675921p4508edf33004c355@mail.gmail.com> Message-ID: <45A98489.3090309@gmail.com> Erin Sheldon wrote: > On 1/9/07, Robert Kern wrote: >> Erin Sheldon wrote: >>> I'm finally getting to this, I'm on the road. Here is >>> what gdb gives me >>> >>> --snip-- >>> Reading symbols for shared libraries . done >>> Reading symbols for shared libraries . done >>> Reading symbols for shared libraries .. done >>> >>> Program received signal EXC_BAD_ACCESS, Could not access memory. >>> Reason: KERN_INVALID_ADDRESS at address: 0x5a6ae09a >>> parse_fond (fond_data=0x5a6ae060
, >>> have_sfnt=0x5a6ae09c, sfnt_id=0xbfff8f4e, lwfn_file_name=0xbfff8e4a >>> "", face_index=0) at >>> /usr/local/src/freetype-2.2.1/src/base/ftmac.c:628 >>> 628 /usr/local/src/freetype-2.2.1/src/base/ftmac.c: No such file >>> or directory. >>> in /usr/local/src/freetype-2.2.1/src/base/ftmac.c >>> >>> For some reason it is looking for something from freetype >>> in /usr/local/src but this directory has never existed >>> on my computer. This is a new computer and I used the migration >>> assistant to copy things from my old computer, so perhaps some >>> paths got messed up. I don't know how to address >>> that type of issue on a Mac with no ld.so.conf type of file. >> Your build of matplotlib is probably picking up a different freetype library >> than the one you intended, one which somebody else built. Possibly the one in >> /usr/X11R6/lib/. You can use otool(1) to figure out what shared libraries your >> extension modules are linking against: >> >> >> [matplotlib]$ otool -L ft2font.so >> ft2font.so: >> /opt/local/lib/libfreetype.6.dylib (compatibility version 10.0.0, >> current version 10.8.0) >> /opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current >> version 1.2.3) >> /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version >> 7.4.0) >> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version >> 88.3.3) >> /usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current >> version 1.0.0) > > Here is what I get >> otool -L /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/ft2font.so > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/matplotlib/ft2font.so: > /usr/local/lib/libfreetype.6.dylib (compatibility version > 10.0.0, current version 10.10.0) > /usr/lib/libz.1.dylib (compatibility version 1.0.0, current > version 1.2.3) > /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, > current version 7.4.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, > current version 88.3.3) > /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, > current version 1.0.0) > > Is there something wrong with this? I still don't seen anything > about /usr/local/src/freetype in here (btw I switched from fink to the versions > installed by i-installer because the odd warnings disappeared when > compiling matplotlib; the same segfault occurs with either). Right. You won't see anything about /usr/local/src/freetype-2.2.1/ there. What you see is the location of the library when ft2font.so was built. That's the library that it is trying to link against at runtime. I presume that that is the one you installed with i-installer. Now, whoever built that library was building from sources that he unpacked in /usr/local/src/freetype-2.2.1/ . That information is recorded in the binary file for debugging purposes. Now, since the bug is actually in freetype, not matplotlib or numpy, I suggest finding a new build of freetype. I use MacPorts and have had absolutely no trouble with it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From erin.sheldon at gmail.com Sat Jan 13 21:01:37 2007 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Sat, 13 Jan 2007 18:01:37 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <45A98489.3090309@gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> <331116dc0701080038wf58be61xcced1cc757198d65@mail.gmail.com> <45A2060C.8010909@gmail.com> <331116dc0701092115g2566fb3tc93f3ebcfbb59a01@mail.gmail.com> <45A47C8A.1040300@gmail.com> <331116dc0701131709r5b675921p4508edf33004c355@mail.gmail.com> <45A98489.3090309@gmail.com> Message-ID: <331116dc0701131801i4a652105g43d9d6f330186e17@mail.gmail.com> On 1/13/07, Robert Kern wrote: --snip-- > > Now, since the bug is actually in freetype, not matplotlib or numpy, I suggest > finding a new build of freetype. I use MacPorts and have had absolutely no > trouble with it. I switched to using macports, and included your modifications to the setupext.py file. The cryptic warnings when compiling matplotlib have returned, e.g. /usr/bin/ld: for architecture ppc /usr/bin/ld: warning /opt/local/lib/libfreetype.dylib cputype (7, architecture i386) does not match cputype (18) for specified -arch flag: ppc (file not loaded) Note, this is an intel machine, so I don't know how ppc enters into it. Import of pylab still fails. The failure is now different. Using gdb I get --snip-- Reading symbols for shared libraries . done Reading symbols for shared libraries .. done 20:48:11: Debug: ../src/common/object.cpp(224): assert "sm_classTable->Get(m_className) == NULL" failed: class already in RTTI table - have you used IMPLEMENT_DYNAMIC_CLASS() twice (may be by linking some object module(s) twice)? ../src/common/object.cpp(224): assert "sm_classTable->Get(m_className) == NULL" failed: class already in RTTI table - have you used IMPLEMENT_DYNAMIC_CLASS() twice (may be by linking some object module(s) twice)? Program received signal SIGTRAP, Trace/breakpoint trap. which I cannot parse. Any ideas? Erin From robert.kern at gmail.com Sat Jan 13 21:10:09 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 13 Jan 2007 20:10:09 -0600 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <331116dc0701131801i4a652105g43d9d6f330186e17@mail.gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> <331116dc0701080038wf58be61xcced1cc757198d65@mail.gmail.com> <45A2060C.8010909@gmail.com> <331116dc0701092115g2566fb3tc93f3ebcfbb59a01@mail.gmail.com> <45A47C8A.1040300@gmail.com> <331116dc0701131709r5b675921p4508edf33004c355@mail.gmail.com> <45A98489.3090309@gmail.com> <331116dc0701131801i4a652105g43d9d6f330186e17@mail.gmail.com> Message-ID: <45A99101.2050203@gmail.com> Erin Sheldon wrote: > On 1/13/07, Robert Kern wrote: > --snip-- >> Now, since the bug is actually in freetype, not matplotlib or numpy, I suggest >> finding a new build of freetype. I use MacPorts and have had absolutely no >> trouble with it. > > > I switched to using macports, and included your modifications to > the setupext.py file. The cryptic warnings when compiling > matplotlib have returned, e.g. > > /usr/bin/ld: for architecture ppc > /usr/bin/ld: warning /opt/local/lib/libfreetype.dylib cputype (7, > architecture i386) does not match cputype (18) for specified -arch > flag: ppc (file not loaded) > > Note, this is an intel machine, so I don't know how ppc enters into it. You are using a Universal Python, so it tries to build ft2font.so as a Universal binary. However, the library it depends on wasn't built Universal. Consequently, the linker warns you that it can't make a Universal ft2font.so. > Import of pylab still fails. The failure is now different. Using gdb I get > --snip-- > Reading symbols for shared libraries . done > Reading symbols for shared libraries .. done > 20:48:11: Debug: ../src/common/object.cpp(224): assert > "sm_classTable->Get(m_className) == NULL" failed: class already in > RTTI table - have you used IMPLEMENT_DYNAMIC_CLASS() twice (may be by > linking some object module(s) twice)? > ../src/common/object.cpp(224): assert "sm_classTable->Get(m_className) > == NULL" failed: class already in RTTI table - have you used > IMPLEMENT_DYNAMIC_CLASS() twice (may be by linking some object > module(s) twice)? > > Program received signal SIGTRAP, Trace/breakpoint trap. > > which I cannot parse. Any ideas? Possibly you are linking against the wrong wx libraries. I missed that part in my instructions. http://sourceforge.net/mailarchive/message.php?msg_id=37895022 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sat Jan 13 23:54:59 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 13 Jan 2007 21:54:59 -0700 Subject: [Numpy-discussion] random permutation In-Reply-To: References: <45A6A0FA.2060504@gmail.com> Message-ID: On 1/13/07, Sebastian Haase wrote: I understand this - but I thought the algorithm might involve some > rounding(-error) cases that would produce different results especially > when changing the CPU (intel vs. PowerPC). I found this to be true > even for non pseudo-random code that between CPU-types results were > different within a given epsilon. Its all 32 bit words, or rather bits. IIRC, it is based on an irreducible polynomial over Z_2, so all the coefficients are 0 or 1. The original paper is worth a read if you like that sort of stuff. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Jan 14 00:22:36 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 13 Jan 2007 23:22:36 -0600 Subject: [Numpy-discussion] random permutation In-Reply-To: References: <45A6A0FA.2060504@gmail.com> Message-ID: <45A9BE1C.1020503@gmail.com> Charles R Harris wrote: > > On 1/13/07, *Sebastian Haase* > wrote: > > > > I understand this - but I thought the algorithm might involve some > rounding(-error) cases that would produce different results especially > when changing the CPU (intel vs. PowerPC). I found this to be true > even for non pseudo-random code that between CPU-types results were > different within a given epsilon. > > Its all 32 bit words, or rather bits. IIRC, it is based on an > irreducible polynomial over Z_2, so all the coefficients are 0 or 1. The > original paper is worth a read if you like that sort of stuff. Well, there are a couple of issues, here. One is the core Mersenne Twister PRNG. You are right that it should reliably produce the same bitstream on all supported platforms. In fact, for the same state vector (*not* the seed), it should produce the same results as the standard library's random.Random() implementation. Now, the various distributions are another matter. They do rely on floating point calculations, and I doubt that we can rely on them producing the exact same streams of output on different processors, particularly those that use extended-precision intermediates. Unfortunately, running distribution tests on all of the non-uniform distributions requires implementations of lots of special functions. I did run such tests using scipy.stats when I implemented the distributions, but I can't really turn those into unit tests for numpy. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Sun Jan 14 05:30:40 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 14 Jan 2007 19:30:40 +0900 Subject: [Numpy-discussion] Fast clip for native types, 2d version Message-ID: <45AA0650.7080208@ar.media.kyoto-u.ac.jp> Hi, First, I wanted to thank everybody who helped me to clarify many points concerning memory layout of numpy arrays; I think I have a much clearer idea of the way numpy arrays behave at the C level. I've used all those informations to correct my initial implementation of clip to improve the clip function for common cases: it speeds up things only for native endianness, and scalar min and max (both contiguous and non contiguous cases). I've attached the new version code (only for one type to avoid too big emails; you have to dl the archive to actually compile the implementation); the whole package with tests + profiling script is there: http://www.ar.media.kyoto-u.ac.jp/members/david/archives/fastclip.tgz If this looks Ok, I will prepare a patch against current numpy, with the C sources being generated by numpy.distutils instead of the tool I am using now (autogen) Now, to improve other cases (mainly implementing an in-place clip function + non scalar min/max), there are some clarifications needed, mainly related to broadcast rules the current clip implementation which seems to break numpy conventions: 1: the old implementation returns an array which has the same endianness than the input array. This is a bit odd, because when the input is byte swapped, the returned array is still byte swapped, which seems to be against numpy convention. Here is some code which seem odd to me (code assumes little endian machine) a = numpy.random.randn(3, 2) b = a.astype(a.dtype.newbyteorder('>')) c = b.copy() assert a.dtype.isnative assert not b.dtype.isnative assert not c.dtype.isnative # Endianness behaviour of basic operation with numpy arrays print (a + b).dtype.isnative #one arg is non native -> returns native print (b + c).dtype.isnative # both args not native -> returns native # Now, what's happening endian-wise with clip: print numpy.clip(a, -0.5, 0.5).dtype.isnative # everything native -> returns native print numpy.clip(b, -0.5, 0.5).dtype.isnative # input array non native -> returns non native print numpy.clip(b, a, 0.5).dtype.isnative # input array non native, native array min -> returns native The fact that the output's endianness depends on min/max arguments being arrays or not does not seem really coherent ? 2: the old implementation does not upcast the input array. If the input is int32, and min/max are float32, the function fails; if input is float32, and min/max float64, the output is still float32. Again, this seems against the expected numpy behaviour ? 3: the old implementation supports clipping with complex arrays. I don't see any obvious meaningful implementation of clipping in those cases (using the module to compare them ?) If breaking those oddities is allowed, this would make the improvements much simpler to code, cheers, David -------------- next part -------------- A non-text attachment was scrubbed... Name: fast_clip.c Type: text/x-csrc Size: 8695 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: clip_imp.c Type: text/x-csrc Size: 3509 bytes Desc: not available URL: From robert.kern at gmail.com Sun Jan 14 05:54:46 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 14 Jan 2007 04:54:46 -0600 Subject: [Numpy-discussion] Fast clip for native types, 2d version In-Reply-To: <45AA0650.7080208@ar.media.kyoto-u.ac.jp> References: <45AA0650.7080208@ar.media.kyoto-u.ac.jp> Message-ID: <45AA0BF6.7090307@gmail.com> David Cournapeau wrote: > 2: the old implementation does not upcast the input array. If the > input is int32, and min/max are float32, the function fails; if input is > float32, and min/max float64, the output is still float32. Again, this > seems against the expected numpy behaviour ? The latter is expected. As discussed previously here, the types of scalars are ignored. You do get an upcast when either min or max is a float64 array. In [1]: import numpy In [2]: a = numpy.linspace(0, 10, 101) In [3]: a = numpy.linspace(0, 10, 101).astype(numpy.float32) In [4]: a.clip(numpy.float64(5), numpy.float64(6)).dtype Out[4]: dtype('float32') In [5]: a.clip(numpy.ones(101, dtype=numpy.float64), numpy.float64(6)).dtype Out[5]: dtype('float64') The int32/float32 failure is odd, though. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Sun Jan 14 22:38:57 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 15 Jan 2007 12:38:57 +0900 Subject: [Numpy-discussion] Fast clip for native types, 2d version In-Reply-To: <45AA0BF6.7090307@gmail.com> References: <45AA0650.7080208@ar.media.kyoto-u.ac.jp> <45AA0BF6.7090307@gmail.com> Message-ID: <45AAF751.2010509@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > David Cournapeau wrote: > >> 2: the old implementation does not upcast the input array. If the >> input is int32, and min/max are float32, the function fails; if input is >> float32, and min/max float64, the output is still float32. Again, this >> seems against the expected numpy behaviour ? > > The latter is expected. As discussed previously here, Could you tell me where this was discussed, I think I missed it. > the types of scalars are > ignored. You do get an upcast when either min or max is a float64 array. > > > In [1]: import numpy > > In [2]: a = numpy.linspace(0, 10, 101) > > In [3]: a = numpy.linspace(0, 10, 101).astype(numpy.float32) > > In [4]: a.clip(numpy.float64(5), numpy.float64(6)).dtype > Out[4]: dtype('float32') > > In [5]: a.clip(numpy.ones(101, dtype=numpy.float64), numpy.float64(6)).dtype > Out[5]: dtype('float64') > > > The int32/float32 failure is odd, though. > Thanks for the precision. Is this the expected behaviour for endianness, too ? And why integer input works for float64 but not for float32 ? In [1]: import numpy In [2]: a = numpy.linspace(0, 10, 101).astype(numpy.int32) In [3]: a.clip(0, 1).dtype Out[3]: dtype('int32') In [4]: a.clip(numpy.float64(0), 1).dtype Out[4]: dtype('float64') In [5]: a.clip(numpy.float32(0), 1).dtype --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /usr/media/boulot/ TypeError: array cannot be safely cast to required type David From robert.kern at gmail.com Sun Jan 14 23:07:55 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 14 Jan 2007 22:07:55 -0600 Subject: [Numpy-discussion] Fast clip for native types, 2d version In-Reply-To: <45AAF751.2010509@ar.media.kyoto-u.ac.jp> References: <45AA0650.7080208@ar.media.kyoto-u.ac.jp> <45AA0BF6.7090307@gmail.com> <45AAF751.2010509@ar.media.kyoto-u.ac.jp> Message-ID: <45AAFE1B.2010203@gmail.com> David Cournapeau wrote: > Robert Kern wrote: >> David Cournapeau wrote: >> >>> 2: the old implementation does not upcast the input array. If the >>> input is int32, and min/max are float32, the function fails; if input is >>> float32, and min/max float64, the output is still float32. Again, this >>> seems against the expected numpy behaviour ? >> The latter is expected. As discussed previously here, > Could you tell me where this was discussed, I think I missed it. http://projects.scipy.org/pipermail/numpy-discussion/2006-November/024402.html > Thanks for the precision. Is this the expected behaviour for endianness, > too ? What endianness behaviour? > And why integer input works for float64 but not for float32 ? As I said, I don't know. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Sun Jan 14 23:16:34 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Mon, 15 Jan 2007 13:16:34 +0900 Subject: [Numpy-discussion] Fast clip for native types, 2d version In-Reply-To: <45AAFE1B.2010203@gmail.com> References: <45AA0650.7080208@ar.media.kyoto-u.ac.jp> <45AA0BF6.7090307@gmail.com> <45AAF751.2010509@ar.media.kyoto-u.ac.jp> <45AAFE1B.2010203@gmail.com> Message-ID: <45AB0022.8020202@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > David Cournapeau wrote: >> Robert Kern wrote: >>> David Cournapeau wrote: >>> >>>> 2: the old implementation does not upcast the input array. If the >>>> input is int32, and min/max are float32, the function fails; if input is >>>> float32, and min/max float64, the output is still float32. Again, this >>>> seems against the expected numpy behaviour ? >>> The latter is expected. As discussed previously here, >> Could you tell me where this was discussed, I think I missed it. > > http://projects.scipy.org/pipermail/numpy-discussion/2006-November/024402.html Ok, thank you. > >> Thanks for the precision. Is this the expected behaviour for endianness, >> too ? > > What endianness behaviour? This (assuming your machine is little endian): In [1]: import numpy In [2]: a = numpy.random.randn(3, 2) In [3]: b = a.astype(a.dtype.newbyteorder('>')) In [4]: a.clip(0., 1.).dtype.isnative Out[4]: True In [5]: b.clip(0., 1.).dtype.isnative Out[5]: False In [6]: b.clip(numpy.zeros(b.shape), 1.).dtype.isnative Out[6]: True If input is byteswapped, the output is also byteswapped for scalar min/max, but native if a native array is given as a min/max argument. David From robert.kern at gmail.com Sun Jan 14 23:47:41 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 14 Jan 2007 22:47:41 -0600 Subject: [Numpy-discussion] Fast clip for native types, 2d version In-Reply-To: <45AB0022.8020202@ar.media.kyoto-u.ac.jp> References: <45AA0650.7080208@ar.media.kyoto-u.ac.jp> <45AA0BF6.7090307@gmail.com> <45AAF751.2010509@ar.media.kyoto-u.ac.jp> <45AAFE1B.2010203@gmail.com> <45AB0022.8020202@ar.media.kyoto-u.ac.jp> Message-ID: <45AB076D.4080900@gmail.com> David Cournapeau wrote: > Robert Kern wrote: >> David Cournapeau wrote: >>> Robert Kern wrote: >>>> David Cournapeau wrote: >>>> >>>>> 2: the old implementation does not upcast the input array. If the >>>>> input is int32, and min/max are float32, the function fails; if input is >>>>> float32, and min/max float64, the output is still float32. Again, this >>>>> seems against the expected numpy behaviour ? >>>> The latter is expected. As discussed previously here, >>> Could you tell me where this was discussed, I think I missed it. >> http://projects.scipy.org/pipermail/numpy-discussion/2006-November/024402.html > Ok, thank you. >>> Thanks for the precision. Is this the expected behaviour for endianness, >>> too ? >> What endianness behaviour? > This (assuming your machine is little endian): > > In [1]: import numpy > > In [2]: a = numpy.random.randn(3, 2) > > In [3]: b = a.astype(a.dtype.newbyteorder('>')) > > In [4]: a.clip(0., 1.).dtype.isnative > Out[4]: True > > In [5]: b.clip(0., 1.).dtype.isnative > Out[5]: False > > In [6]: b.clip(numpy.zeros(b.shape), 1.).dtype.isnative > Out[6]: True > > If input is byteswapped, the output is also byteswapped for scalar > min/max, but native if a native array is given as a min/max argument. Yes, that seems to fit in with the rule. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From giorgio.luciano at chimica.unige.it Mon Jan 15 03:31:27 2007 From: giorgio.luciano at chimica.unige.it (Giorgio Luciano) Date: Mon, 15 Jan 2007 09:31:27 +0100 Subject: [Numpy-discussion] rounding all element of an array with more than 1d Message-ID: <45AB3BDF.4090507@chimica.unige.it> I've posted also to scipy group but with no luck. At first I thought it was an easy task but... I've to round the numbers inside of a 2d array but nothing seems to work. the only message I've found in the archive is an old function custom made for doing that.. wouldn't be useful to have it also included something that do this task in numpy ? here's and example of the error using round Traceback (most recent call last): File "", line 1, in round(vp) TypeError: only length-1 arrays can be converted to Python scalars Any feedback appreciated From nwagner at iam.uni-stuttgart.de Mon Jan 15 03:36:38 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 15 Jan 2007 09:36:38 +0100 Subject: [Numpy-discussion] rounding all element of an array with more than 1d In-Reply-To: <45AB3BDF.4090507@chimica.unige.it> References: <45AB3BDF.4090507@chimica.unige.it> Message-ID: <45AB3D16.3050501@iam.uni-stuttgart.de> Giorgio Luciano wrote: > I've posted also to scipy group but with no luck. > At first I thought it was an easy task but... > I've to round the numbers inside of a 2d array but nothing seems to > work. the only message I've found in the archive is an old function > custom made for doing that.. wouldn't be useful to have it also included > something that do this task in numpy ? > here's and example of the error using round > > > Traceback (most recent call last): > File "", line 1, in > round(vp) > TypeError: only length-1 arrays can be converted to Python scalars > > Any feedback appreciated > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > >>> a = random.rand(2,4) >>> a array([[ 0.88084812, 0.03112202, 0.72374646, 0.76379529], [ 0.20978501, 0.2647247 , 0.12690806, 0.78367502]]) >>> round_(a) array([[ 1., 0., 1., 1.], [ 0., 0., 0., 1.]]) HTH, Nils From giorgio.luciano at chimica.unige.it Mon Jan 15 03:47:05 2007 From: giorgio.luciano at chimica.unige.it (Giorgio Luciano) Date: Mon, 15 Jan 2007 09:47:05 +0100 Subject: [Numpy-discussion] rounding all element of an array with more than 1d In-Reply-To: <45AB3D16.3050501@iam.uni-stuttgart.de> References: <45AB3BDF.4090507@chimica.unige.it> <45AB3D16.3050501@iam.uni-stuttgart.de> Message-ID: <45AB3F89.2060508@chimica.unige.it> Yes it worked, sorry for bothering. the problem was theI used round with no underscore.. and so I made confusion with the two command Thank a lot !! > > From erin.sheldon at gmail.com Mon Jan 15 15:45:14 2007 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Mon, 15 Jan 2007 15:45:14 -0500 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <45A99101.2050203@gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <45A1E97A.1040507@gmail.com> <331116dc0701080038wf58be61xcced1cc757198d65@mail.gmail.com> <45A2060C.8010909@gmail.com> <331116dc0701092115g2566fb3tc93f3ebcfbb59a01@mail.gmail.com> <45A47C8A.1040300@gmail.com> <331116dc0701131709r5b675921p4508edf33004c355@mail.gmail.com> <45A98489.3090309@gmail.com> <331116dc0701131801i4a652105g43d9d6f330186e17@mail.gmail.com> <45A99101.2050203@gmail.com> Message-ID: <331116dc0701151245s407ed34bid4dfd9bfe08789a7@mail.gmail.com> On 1/13/07, Robert Kern wrote: > > Import of pylab still fails. The failure is now different. Using gdb I get > > --snip-- > > Reading symbols for shared libraries . done > > Reading symbols for shared libraries .. done > > 20:48:11: Debug: ../src/common/object.cpp(224): assert > > "sm_classTable->Get(m_className) == NULL" failed: class already in > > RTTI table - have you used IMPLEMENT_DYNAMIC_CLASS() twice (may be by > > linking some object module(s) twice)? > > ../src/common/object.cpp(224): assert "sm_classTable->Get(m_className) > > == NULL" failed: class already in RTTI table - have you used > > IMPLEMENT_DYNAMIC_CLASS() twice (may be by linking some object > > module(s) twice)? > > > > Program received signal SIGTRAP, Trace/breakpoint trap. > > > > which I cannot parse. Any ideas? > > Possibly you are linking against the wrong wx libraries. I missed that part in > my instructions. > > http://sourceforge.net/mailarchive/message.php?msg_id=37895022 I found that if I took the following steps, basically eveything works on my intel core 2 duo macbook. Install XCode Get gfortran http://hpc.sourceforge.net/ http://prdownloads.sourceforge.net/hpc/gfortran-intel-bin.tar.gz?download Go to this site: http://pythonmac.org/packages/py25-fat/index.html Get these packages and install them in this order python-2.5-macosx.dmg numpy-1.0.1-py2.5-macosx10.4-2006-12-12.dmg wxPython2.6-osx-unicode-2.6.3.3-universal10.4-py2.5.dmg matplotlib-0.87.7-py2.5-macosx10.4-2006-12-12.dmg Get Scipy. No binary on pythonmac for scipy yet, but the compile is straightforward with the right flags (thanks Robert). http://prdownloads.sourceforge.net/scipy/scipy-0.5.2.tar.gz To build and install: python setup.py build_src build_clib --fcompiler=gnu95 build_ext --fcompiler=gnu95 build sudo python setup.py install Erin From oliphant at ee.byu.edu Mon Jan 15 16:18:10 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 15 Jan 2007 14:18:10 -0700 Subject: [Numpy-discussion] Come to PyCon if you can Message-ID: <45ABEF92.3020303@ee.byu.edu> Greetings, This is a little plug for coming to PyCon Feb 22-25. I am going to be there from Thursday night (Feb. 22) until Saturday evening (Feb. 25), and would like to meet with anyone interested to discuss some of the issues surrounding NumPy/SciPy and the PEPs we are working on for Python. I would stay for the sprints but I can't this year. If others are interested in using the sprints for SciPy/NumPy development, then we can still coordinate efforts. I will be presenting a talk on Understanding NumPy which is targeted to the beginner/intermediate audience on Saturday morning. There are other talks as well that have a distinct scientific computing flavor that look very interesting. Hopefully we can organize a BOF session as well during the conference (on Friday evening preferrably). Here are more details: http://us.pycon.org/TX2007/ Early bird registration ends Jan 15 (today!) Best regards, -Travis O. From fperez.net at gmail.com Mon Jan 15 17:24:07 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 15 Jan 2007 15:24:07 -0700 Subject: [Numpy-discussion] Come to PyCon if you can In-Reply-To: <45ABEF92.3020303@ee.byu.edu> References: <45ABEF92.3020303@ee.byu.edu> Message-ID: On 1/15/07, Travis Oliphant wrote: > > Greetings, > > This is a little plug for coming to PyCon Feb 22-25. I am going to be > there from Thursday night (Feb. 22) until Saturday evening (Feb. 25), > and would like to meet with anyone interested to discuss some of the > issues surrounding NumPy/SciPy and the PEPs we are working on for > Python. I would stay for the sprints but I can't this year. If others > are interested in using the sprints for SciPy/NumPy development, then we > can still coordinate efforts. Another plug: there will be two ipython talks, one on the basics and one on the new distributed computing work. Unfortunately I won't be able to make it so Brian Granger will present both. And one more plug: if anyone is in the SF/Berkeley area, I'm co-organizing a workshop at the Math Sciences Research Institute there, on parallel/distributed computing for mathematical problems: http://sage.math.washington.edu/msri07/ The wiki has all the real details: http://sage.math.washington.edu:9001/msri07 This isn't python-specific, but you may find some of the talks interesting, and attendance is free if you don't need funding. The deadline for actual registration has passed, but if you are in the area, all talks are open to the public. Cheers, f From david at ar.media.kyoto-u.ac.jp Mon Jan 15 20:36:09 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Jan 2007 10:36:09 +0900 Subject: [Numpy-discussion] Should non ufunc numpy functions behave like ufunc regarding casting to output argument ? Message-ID: <45AC2C09.7030607@ar.media.kyoto-u.ac.jp> Hi, I am trying to add support for out argument to one C function using numpy API (still the clip function). I was wondering about the expected behaviour when out does not have the "expected" type. For example, using again the clip function (but the question is not specific to this function) In [1]: import numpy In [2]: a = numpy.linspace(0, 10, 101) In [3]: b = numpy.zeros(a.shape, dtype = numpy.int32) In [4]: print a.dtype float64 In [5]: a.clip(0.1, 0.5, b) Should this be equivalent to b = a.clip(0.1, 0.5); b = b.astype(numpy.int32) (ie, the casting is done at the end, similar to an ufunc) ? cheers, David From peridot.faceted at gmail.com Mon Jan 15 21:57:22 2007 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 15 Jan 2007 21:57:22 -0500 Subject: [Numpy-discussion] Should non ufunc numpy functions behave like ufunc regarding casting to output argument ? In-Reply-To: <45AC2C09.7030607@ar.media.kyoto-u.ac.jp> References: <45AC2C09.7030607@ar.media.kyoto-u.ac.jp> Message-ID: On 15/01/07, David Cournapeau wrote: > Hi, > > I am trying to add support for out argument to one C function using > numpy API (still the clip function). I was wondering about the expected > behaviour when out does not have the "expected" type. > For example, using again the clip function (but the question is not > specific to this function) > > In [1]: import numpy > > In [2]: a = numpy.linspace(0, 10, 101) > > In [3]: b = numpy.zeros(a.shape, dtype = numpy.int32) > > In [4]: print a.dtype > float64 > > In [5]: a.clip(0.1, 0.5, b) > > Should this be equivalent to b = a.clip(0.1, 0.5); b = > b.astype(numpy.int32) (ie, the casting is done at the end, similar to an > ufunc) ? Since the point of output arguments is to avoid allocating new storage, I'm not sure whether to say yes or no here... but if you're given an output array to store the answer in, you're more or less forced to convert it to that type (and layout, and whatnot) for storage - imagine, for example, it is every third element of a bigger array (and remember you do not have access to the name "b"). for example: a = numpy.arange(10) b = numpy.zeros(40).astype(numpy.uint8) a.clip(0.1,5.2,b[::4]) You can't do anything about the data type of b, so you don't really have any choice but to convert to uint8; one hopes that this will not require the allocation of an extra array of floats. A. M. Archibald From charlesr.harris at gmail.com Mon Jan 15 22:17:31 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 15 Jan 2007 20:17:31 -0700 Subject: [Numpy-discussion] Should non ufunc numpy functions behave like ufunc regarding casting to output argument ? In-Reply-To: References: <45AC2C09.7030607@ar.media.kyoto-u.ac.jp> Message-ID: On 1/15/07, A. M. Archibald wrote: > > On 15/01/07, David Cournapeau wrote: > > Hi, > > > > I am trying to add support for out argument to one C function using > > numpy API (still the clip function). I was wondering about the expected > > behaviour when out does not have the "expected" type. > > For example, using again the clip function (but the question is not > > specific to this function) > > > > In [1]: import numpy > > > > In [2]: a = numpy.linspace(0, 10, 101) > > > > In [3]: b = numpy.zeros(a.shape, dtype = numpy.int32) > > > > In [4]: print a.dtype > > float64 > > > > In [5]: a.clip(0.1, 0.5, b) > > > > Should this be equivalent to b = a.clip(0.1, 0.5); b = > > b.astype(numpy.int32) (ie, the casting is done at the end, similar to an > > ufunc) ? > > Since the point of output arguments is to avoid allocating new > storage, If we take that seriously, then an error should be raised on a shape, or type mismatch. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Mon Jan 15 22:27:59 2007 From: peridot.faceted at gmail.com (A. M. Archibald) Date: Mon, 15 Jan 2007 22:27:59 -0500 Subject: [Numpy-discussion] Should non ufunc numpy functions behave like ufunc regarding casting to output argument ? In-Reply-To: References: <45AC2C09.7030607@ar.media.kyoto-u.ac.jp> Message-ID: On 15/01/07, Charles R Harris wrote: > > Since the point of output arguments is to avoid allocating new > > storage, > > > If we take that seriously, then an error should be raised on a shape, or > type mismatch. In fact: In [10]: a = zeros(3) In [11]: b = zeros(4,dtype=uint8) In [12]: add(a,a,b) --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/peridot/ ValueError: invalid return array shape In [13]: add(a,a,b[:3]) Out[13]: array([0, 0, 0], dtype=uint8) In [14]: add(b,b,b.reshape((2,2))) --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/peridot/ ValueError: invalid return array shape We do raise an error on shape mismatch; type mismatches are used to forcibly cast the result - which is useful! So I'd say we take it seriously. It's one of the ways to make your code run faster. A. M. Archibald From david at ar.media.kyoto-u.ac.jp Mon Jan 15 22:41:28 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Jan 2007 12:41:28 +0900 Subject: [Numpy-discussion] Should non ufunc numpy functions behave like ufunc regarding casting to output argument ? In-Reply-To: References: <45AC2C09.7030607@ar.media.kyoto-u.ac.jp> Message-ID: <45AC4968.4010408@ar.media.kyoto-u.ac.jp> Charles R Harris wrote: > > > On 1/15/07, *A. M. Archibald* > wrote: > > On 15/01/07, David Cournapeau > wrote: > > Hi, > > > > I am trying to add support for out argument to one C > function using > > numpy API (still the clip function). I was wondering about the > expected > > behaviour when out does not have the "expected" type. > > For example, using again the clip function (but the question > is not > > specific to this function) > > > > In [1]: import numpy > > > > In [2]: a = numpy.linspace(0, 10, 101) > > > > In [3]: b = numpy.zeros(a.shape, dtype = numpy.int32) > > > > In [4]: print a.dtype > > float64 > > > > In [5]: a.clip(0.1, 0.5, b) > > > > Should this be equivalent to b = a.clip(0.1, 0.5); b = > > b.astype(numpy.int32) (ie, the casting is done at the end, > similar to an > > ufunc) ? > > Since the point of output arguments is to avoid allocating new > storage, > > > > If we take that seriously, then an error should be raised on a shape, > or type mismatch. For the shape, there is no problem, I guess. A different shape is an error (except if I want to support broadcasting...). The problem is really for out having same shape than in, but having different type than input. At first, I wanted to throw an error (for example, clipping an array which gives a float, and out is integer), but that would be incompatible with current clip behaviour. Concerning the point of avoiding allocating new storage, I am a bit suspicious: if the types do not match, and the casting is done at the end, then it means all internal computation will be done is whatever type is chosen by the function (I am using PyArray_CommonType for that), and the cast done at the end, meaning new storage. Actually, I find more logical to throw an error of the point is to avoid new storage, as giving a mismatched type out buffer would make the function create an internal buffer. David From david at ar.media.kyoto-u.ac.jp Mon Jan 15 22:44:48 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Jan 2007 12:44:48 +0900 Subject: [Numpy-discussion] Should non ufunc numpy functions behave like ufunc regarding casting to output argument ? In-Reply-To: <45AC4968.4010408@ar.media.kyoto-u.ac.jp> References: <45AC2C09.7030607@ar.media.kyoto-u.ac.jp> <45AC4968.4010408@ar.media.kyoto-u.ac.jp> Message-ID: <45AC4A30.2050806@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > Charles R Harris wrote: >> >> On 1/15/07, *A. M. Archibald* > > wrote: >> >> On 15/01/07, David Cournapeau > > wrote: >> > Hi, >> > >> > I am trying to add support for out argument to one C >> function using >> > numpy API (still the clip function). I was wondering about the >> expected >> > behaviour when out does not have the "expected" type. >> > For example, using again the clip function (but the question >> is not >> > specific to this function) >> > >> > In [1]: import numpy >> > >> > In [2]: a = numpy.linspace(0, 10, 101) >> > >> > In [3]: b = numpy.zeros(a.shape, dtype = numpy.int32) >> > >> > In [4]: print a.dtype >> > float64 >> > >> > In [5]: a.clip(0.1, 0.5, b) >> > >> > Should this be equivalent to b = a.clip(0.1, 0.5); b = >> > b.astype(numpy.int32) (ie, the casting is done at the end, >> similar to an >> > ufunc) ? >> >> Since the point of output arguments is to avoid allocating new >> storage, >> >> >> >> If we take that seriously, then an error should be raised on a shape, >> or type mismatch. > For the shape, there is no problem, I guess. A different shape is an > error (except if I want to support broadcasting...). The problem is > really for out having same shape than in, but having different type than > input. > > At first, I wanted to throw an error (for example, clipping an array > which gives a float, and out is integer), but that would be incompatible > with current clip behaviour. > > Concerning the point of avoiding allocating new storage, I am a bit > suspicious: if the types do not match, and the casting is done at the > end, then it means all internal computation will be done is whatever > type is chosen by the function (I am using PyArray_CommonType for that), > and the cast done at the end, meaning new storage. > > Actually, I find more logical to throw an error of the point is to avoid > new storage, as giving a mismatched type out buffer would make the > function create an internal buffer. Sorry, this last sentence is incomplete and does not really make sense: I meant Actually, I find more logical to throw an error of the point is to avoid new storage, as giving a mismatched type out buffer would make the function create an new array when using the numpy C Api function PyArray_ConvertToCommonType. David From david at ar.media.kyoto-u.ac.jp Tue Jan 16 00:34:36 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Jan 2007 14:34:36 +0900 Subject: [Numpy-discussion] Can PyArray_CastTo be used to "subcast" an array of float to int Message-ID: <45AC63EC.9060906@ar.media.kyoto-u.ac.jp> Hi, I would like to convert a double array to an *existing* array of int32 using the C numpy API. I tried using PyArray_CastTo, but this gives me an array full of zeros instead of converting the items to int. I am doing something like the following to convert a PyArrayObject in_o of double: ndescr = PyArray_DescrNewFromType(NPY_INT32); /* should be equivalent to astype */ ar = (PyArrayObject*)PyArray_FromArray((PyArrayObject*)in_o, ndescr, NPY_FORCECAST); if (ar == NULL) { goto clean_ndescr; } /* Using C Api for casting */ dims = PyArray_DIMS(in_o); Py_INCREF(ndescr); casted = (PyArrayObject*)PyArray_Zeros(((PyArrayObject*)in_o)->nd, dims, ndescr, 0); if (casted == NULL) { goto clean_ar; } st = PyArray_CastTo((PyArrayObject*)in_o, casted); if (st) { goto clean_casted; } When using this code in python with a = numpy.linspace(0, 10, 101) as input, I got array([0, 0, 1, 1, 2, 2, 3, 3, 4, 5] for ar, and array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) for array casted. Can't PyArray_CastTo be used to 'unsafely' cast an array, or is there something I am missing ? cheers, David From david at ar.media.kyoto-u.ac.jp Tue Jan 16 00:48:23 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Jan 2007 14:48:23 +0900 Subject: [Numpy-discussion] Can PyArray_CastTo be used to "subcast" an array of float to int In-Reply-To: <45AC63EC.9060906@ar.media.kyoto-u.ac.jp> References: <45AC63EC.9060906@ar.media.kyoto-u.ac.jp> Message-ID: <45AC6727.2090801@ar.media.kyoto-u.ac.jp> David Cournapeau wrote: > Hi, > > I would like to convert a double array to an *existing* array of > int32 using the C numpy API. I tried using PyArray_CastTo, but this > gives me an array full of zeros instead of converting the items to int. > I am doing something like the following to convert a PyArrayObject in_o > of double: > > ndescr = PyArray_DescrNewFromType(NPY_INT32); > > /* should be equivalent to astype */ > ar = (PyArrayObject*)PyArray_FromArray((PyArrayObject*)in_o, > ndescr, NPY_FORCECAST); > if (ar == NULL) { > goto clean_ndescr; > } > /* Using C Api for casting */ > dims = PyArray_DIMS(in_o); > Py_INCREF(ndescr); > casted = (PyArrayObject*)PyArray_Zeros(((PyArrayObject*)in_o)->nd, > dims, ndescr, 0); > if (casted == NULL) { > goto clean_ar; > } > st = PyArray_CastTo((PyArrayObject*)in_o, casted); > if (st) { > goto clean_casted; > } Just forget about it, I didn't notice that PyArray_CastTo takes its input as the 2d argument... Sorry for the noise, David From david at ar.media.kyoto-u.ac.jp Tue Jan 16 07:57:51 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 16 Jan 2007 21:57:51 +0900 Subject: [Numpy-discussion] Best way to compare two implementations of the same numpy function ? Message-ID: <45ACCBCF.9030901@ar.media.kyoto-u.ac.jp> Hi, I am finalizing my patch to SVN numpy to improve the clip function. I managed to modify the code so that the generated C sources are generated through the genapi system in numpy/core, etc.... and it compiles fine. The only problem I have left is "where" to expose the function to numpy ? I would like to have a new function to be able to test against the old version (I have a few tens of tests to compare results between current numpy clip and my function), but I am not sure how to do it so it can be commited as a patch to numpy SVN (if the patch is found acceptable by numpy developers, of cours), cheers, David From rlw at stsci.edu Tue Jan 16 08:00:42 2007 From: rlw at stsci.edu (Rick White) Date: Tue, 16 Jan 2007 08:00:42 -0500 Subject: [Numpy-discussion] Should non ufunc numpy functions behave like ufunc regarding casting to output argument ? In-Reply-To: <45AC4968.4010408@ar.media.kyoto-u.ac.jp> References: <45AC2C09.7030607@ar.media.kyoto-u.ac.jp> <45AC4968.4010408@ar.media.kyoto-u.ac.jp> Message-ID: <000C635D-34EE-4838-840F-F37E8EA30F21@stsci.edu> On Jan 15, 2007, at 10:41 PM, David Cournapeau wrote: > Concerning the point of avoiding allocating new storage, I am a bit > suspicious: if the types do not match, and the casting is done at the > end, then it means all internal computation will be done is whatever > type is chosen by the function (I am using PyArray_CommonType for > that), > and the cast done at the end, meaning new storage. Presumably you should do what ufuncs do: divide the computation up into blocks when the array is big. If a cast is required then you do the computation for each block, allocating new storage for that block. Then you do the cast for the block and copy it to the output array. That way you only have to allocate enough new storage for a single block, which is (potentially) much smaller than the whole array. Rick From cournape at gmail.com Tue Jan 16 10:09:31 2007 From: cournape at gmail.com (David Cournapeau) Date: Wed, 17 Jan 2007 00:09:31 +0900 Subject: [Numpy-discussion] Should non ufunc numpy functions behave like ufunc regarding casting to output argument ? In-Reply-To: <000C635D-34EE-4838-840F-F37E8EA30F21@stsci.edu> References: <45AC2C09.7030607@ar.media.kyoto-u.ac.jp> <45AC4968.4010408@ar.media.kyoto-u.ac.jp> <000C635D-34EE-4838-840F-F37E8EA30F21@stsci.edu> Message-ID: <5b8d13220701160709j74f939f4v56b3de060ced6975@mail.gmail.com> On 1/16/07, Rick White wrote: > On Jan 15, 2007, at 10:41 PM, David Cournapeau wrote: > > > Concerning the point of avoiding allocating new storage, I am a bit > > suspicious: if the types do not match, and the casting is done at the > > end, then it means all internal computation will be done is whatever > > type is chosen by the function (I am using PyArray_CommonType for > > that), > > and the cast done at the end, meaning new storage. > > Presumably you should do what ufuncs do: divide the computation up > into blocks when the array is big. If a cast is required then you do > the computation for each block, allocating new storage for that > block. Then you do the cast for the block and copy it to the output > array. > Well, at that point, why not making clip a ufunc, then... I was already wondering about that, but I don't know much about ufunc on C side. David From mkg at cs.nyu.edu Tue Jan 16 10:06:58 2007 From: mkg at cs.nyu.edu (Matthew Koichi Grimes) Date: Tue, 16 Jan 2007 10:06:58 -0500 Subject: [Numpy-discussion] weird numpy.any() behavior; bug, or 6-legged feature? Message-ID: <45ACEA12.7010406@cs.nyu.edu> Numpy's any() function gives unintuitive results when given a generator rather than a sequence: >>> import numpy as N >>> N.any( i < 0 for i in range(3) ) True If the generator is instead given as a list (using a list comprehension), then the expected answer is given: >>> N.any( [i < 0 for i in range(3)] ) False Python 2.5's built-in any() seems to deal fine with generators: >>> any( i < 0 for i in range(3) ) True My first guess at what's happening in numpy.any() is that the generator object itself is being interpreted as a "True", in the type-weak C sense of the word. Is this a bug? It seems like having to generate a whole list before calling any() defeats the whole appeal of using any() in the first place; namely that it'll exit on the first False returned by the generator, without going through the rest of the elements. -- Matt From robert.kern at gmail.com Tue Jan 16 14:14:27 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 16 Jan 2007 13:14:27 -0600 Subject: [Numpy-discussion] weird numpy.any() behavior; bug, or 6-legged feature? In-Reply-To: <45ACEA12.7010406@cs.nyu.edu> References: <45ACEA12.7010406@cs.nyu.edu> Message-ID: <45AD2413.5030709@gmail.com> Matthew Koichi Grimes wrote: > Is this a bug? It seems like having to generate a whole list before > calling any() defeats the whole appeal of using any() in the first > place; namely that it'll exit on the first False returned by the > generator, without going through the rest of the elements. No, it's not really a bug; it's a consequence of the fact that we can't reliably "just build an array" from any given generator without extra information. And even then, it's tricky. The generator object is being turned into a numpy.object_ scalar type; that's why you get the result that you do. Most numpy functions operate on arrays or things that can be turned into arrays. They don't operate in ideal ways on every kind of object that you might throw at them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From haase at msg.ucsf.edu Tue Jan 16 14:42:11 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 16 Jan 2007 11:42:11 -0800 Subject: [Numpy-discussion] how to recognize a numpy-scalar type at the C level and how to convert Message-ID: Hi! After converting to numpy my SWIG-wrapped code runs a bunch of error of this type: TypeError: argument number 9: a 'float' is expected, 'numpy.float32(-2.10786056519)' is received This comes of course from the fact that numpy introduced all these new scalar types (addressing the shortcoming of the (few) scalar type provided by standard python). What would be an easy way to "recognize" such a scalar type in C-API numpy code and can I extract the C-scalar value from it ? Thanks for any hints, Sebastian Haase From setthiry at hotmail.com Wed Jan 17 05:54:27 2007 From: setthiry at hotmail.com (josh kantor) Date: Wed, 17 Jan 2007 03:54:27 -0700 Subject: [Numpy-discussion] ownership of numpy arrays Message-ID: I am working on a project where I am doing some interfacing with numpy arrays at a C level. One thing that I want to do (using pyrex) is to call numpy functions, then directly access the underlying double * array of the resulting ndarray and operate on it in C/pyrex (I don't want to copy it). I want to ensure that the underlying double * wont be deallocated until I'm ready. I was going to take the resulting ndarray and set the ownership bit of the result to 0 and then I would be responsible for deallocating the memory when I was ready. Contrary to what I expected I noticed that it seems all numpy functions, for example numpy.linalg.pinv as well as numpy.linalg.eig return matrices that don't own their data. (If m is a numpy array) n=numpy.linalg.pinv(m) n.flags shows that n does not own its data. So I can't take their underlying data * with any certaintly that it won't be deallocated before I'm ready. If n doesn't own its data, who does. Is there a safe way to do what I want? Thanks, Josh _________________________________________________________________ The MSN Entertainment Guide to Golden Globes is here. Get all the scoop. http://tv.msn.com/tv/globes2007/?icid=nctagline2 From faltet at carabos.com Wed Jan 17 07:29:31 2007 From: faltet at carabos.com (Francesc Altet) Date: Wed, 17 Jan 2007 13:29:31 +0100 Subject: [Numpy-discussion] ownership of numpy arrays In-Reply-To: References: Message-ID: <1169036971.2518.16.camel@localhost.localdomain> El dc 17 de 01 del 2007 a les 03:54 -0700, en/na josh kantor va escriure: > I am working on a project where I am doing some interfacing with numpy > arrays at a C level. > > One thing that I want to do (using pyrex) is to call numpy functions, then > directly access the underlying double * array of the resulting ndarray and > operate on it in C/pyrex (I don't want to copy it). I want to ensure that > the underlying double * wont be deallocated until I'm ready. I was going to > take the resulting ndarray and set the ownership bit of the result to 0 and > then I would be responsible for deallocating the memory when I was ready. Well, one possibility is to always ensure that you have a reference to the array (using a Pyrex class variable, in case your Pyrex instance is going to live enough, or returning it to your Python function and bound it there to a variable until you need it). When you are done with the array, you only have to clear all references to it (clearing the Pyrex class variable, or doesn't returning it anymore) and Python will reclaim the memory. > Contrary to what I expected I noticed that it seems all numpy functions, for > example numpy.linalg.pinv as well as numpy.linalg.eig return matrices that > don't own their data. > > (If m is a numpy array) > > n=numpy.linalg.pinv(m) > > n.flags > > shows that n does not own its data. No idea of what's happening there. numpy.sin() does seem to preserve the ownership of the data: >>> numpy.sin(numpy.arange(3)).flags C_CONTIGUOUS : True F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False HTH, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From oliphant at ee.byu.edu Wed Jan 17 10:47:44 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 17 Jan 2007 08:47:44 -0700 Subject: [Numpy-discussion] ownership of numpy arrays In-Reply-To: References: Message-ID: <45AE4520.7040003@ee.byu.edu> josh kantor wrote: >I am working on a project where I am doing some interfacing with numpy >arrays at a C level. > >One thing that I want to do (using pyrex) is to call numpy functions, then >directly access the underlying double * array of the resulting ndarray and >operate on it in C/pyrex (I don't want to copy it). > A common need and easily done (though you need to take care to make sure you understand the general memory model for ndarray's or at least ensure you get an array following the memory model you are assuming). >I want to ensure that >the underlying double * wont be deallocated until I'm ready. I was going to >take the resulting ndarray and set the ownership bit of the result to 0 and >then I would be responsible for deallocating the memory when I was ready. > >Contrary to what I expected I noticed that it seems all numpy functions, for >example numpy.linalg.pinv as well as numpy.linalg.eig return matrices that >don't own their data. > >(If m is a numpy array) > >n=numpy.linalg.pinv(m) > >n.flags > >shows that n does not own its data. > >So I can't take their underlying data * with any certaintly that it won't be >deallocated before I'm ready. If n doesn't own its data, who does. Is there >a safe way to do what I want? > > If you are holding a reference to the underlying array then it won't be de-allocated (even if it doesn't own it's own memory). -Travis From haase at msg.ucsf.edu Wed Jan 17 13:02:28 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 17 Jan 2007 10:02:28 -0800 Subject: [Numpy-discussion] possible bug in recordarray assigning to a '4u4' field Message-ID: Hi, ProStr = N.rec.array(None, formats="u4,f4,u4,u4,4u4", names=('count','clock','InitDio', 'nDigital', 'nAnalog'), aligned=True, shape=1) ProStr[0] ['nAnalog'] = 1 I get this error message: ValueError: shape-mismatch on array construction ProStr ['nAnalog'] = 1 works fine. Surprisingly ProStr['nAnalog'] = [1,2,3,4] works. Could someone explain ? Thanks, Sebastian From Chris.Barker at noaa.gov Wed Jan 17 15:33:27 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 17 Jan 2007 12:33:27 -0800 Subject: [Numpy-discussion] recompiling needed for binary module after numpy 1.0 In-Reply-To: <45A99101.2050203@gmail.com> References: <647A37CC-8F5A-47DD-8062-D434688772B2@cs.hmc.edu> <459F030E.8060001@gmail.com> <45A1E97A.1040507@gmail.com> <331116dc0701080038wf58be61xcced1cc757198d65@mail.gmail.com> <45A2060C.8010909@gmail.com> <331116dc0701092115g2566fb3tc93f3ebcfbb59a01@mail.gmail.com> <45A47C8A.1040300@gmail.com> <331116dc0701131709r5b675921p4508edf33004c355@mail.gmail.com> <45A98489.3090309@gmail.com> <331116dc0701131801i4a652105g43d9d6f330186e17@mail.gmail.com> <45A99101.2050203@gmail.com> Message-ID: <45AE8817.9010304@noaa.gov> Robert Kern wrote: >> Program received signal SIGTRAP, Trace/breakpoint trap. >> >> which I cannot parse. Any ideas? > > Possibly you are linking against the wrong wx libraries. It's worth a try, but that doesn't look familiar. New MPL binaries have been put up on pythonmac.org/packages. I'm pretty sure they are linked against wxPython2.6.3.3 (which is also there). Maybe you should try those, and if you're still having problems, move this over to the MPL or pythonmac lists -- it's not a numpy issue. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From efiring at hawaii.edu Wed Jan 17 18:31:09 2007 From: efiring at hawaii.edu (Eric Firing) Date: Wed, 17 Jan 2007 13:31:09 -1000 Subject: [Numpy-discussion] ANN: MaskedArray as a subclass of ndarray - followup In-Reply-To: <200701020931.33517.pgmdevlist@gmail.com> References: <200701020931.33517.pgmdevlist@gmail.com> Message-ID: <45AEB1BD.4000003@hawaii.edu> Pierre GM wrote: > All, > I've updated this famous reimplementation of maskedarray I keep ranting about. [...] > I also put the file `timer_comparison.py`, that runs some unittests with each > implementation > (numpy.core.ma and maskedarray), and outputs the minimum times. > On my machine, there doesn't seem to be a lot of differences, maskedarray > being slightly faster. Same for mine: Thinkpad T41, Pentium M, ubuntu Edgy: efiring at manini:~/programs/py/tests$ python timer_comparison.py #1.................................................. numpy.core.ma: 0.492 - 0.493 maskedarray : 0.481 - 0.482 #2.................................................. numpy.core.ma: 1.440 - 1.440 maskedarray : 1.215 - 1.215 #3.................................................. numpy.core.ma: 2.272 - 2.274 maskedarray : 2.156 - 2.156 I admit that I have not studied the question, but my impression is that you have made some nice improvements. Numpy unified the Numeric/numarray split, but now we have a MaskedArray split. Any prospect for unification, say in numpy 1.1? Might it make sense for maskedarray to replace numpy.core.ma in 1.1? Eric From oliphant at ee.byu.edu Wed Jan 17 19:11:31 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 17 Jan 2007 17:11:31 -0700 Subject: [Numpy-discussion] ANN: MaskedArray as a subclass of ndarray - followup In-Reply-To: <45AEB1BD.4000003@hawaii.edu> References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> Message-ID: <45AEBB33.7050800@ee.byu.edu> Eric Firing wrote: >Pierre GM wrote: > > >>All, >>I've updated this famous reimplementation of maskedarray I keep ranting about. >> >> >[...] > > >>I also put the file `timer_comparison.py`, that runs some unittests with each >>implementation >>(numpy.core.ma and maskedarray), and outputs the minimum times. >>On my machine, there doesn't seem to be a lot of differences, maskedarray >>being slightly faster. >> >> > >Same for mine: Thinkpad T41, Pentium M, ubuntu Edgy: > >efiring at manini:~/programs/py/tests$ python timer_comparison.py >#1.................................................. >numpy.core.ma: 0.492 - 0.493 >maskedarray : 0.481 - 0.482 >#2.................................................. >numpy.core.ma: 1.440 - 1.440 >maskedarray : 1.215 - 1.215 >#3.................................................. >numpy.core.ma: 2.272 - 2.274 >maskedarray : 2.156 - 2.156 > > >I admit that I have not studied the question, but my impression is that >you have made some nice improvements. Numpy unified the >Numeric/numarray split, but now we have a MaskedArray split. Any >prospect for unification, say in numpy 1.1? Might it make sense for >maskedarray to replace numpy.core.ma in 1.1? > > This makes sense to me. I'm generally favorable to the new maskedarray (I actually like the idea of it being a sub-class). I'm just waiting for people that actually use the MaskedArray to comment. For 1.1 I would really like to move most of the often-used sub-classes of the ndarray to the C-level and merge in functionality from CVXOPT. -Travis From emanuele at relativita.com Thu Jan 18 12:09:54 2007 From: emanuele at relativita.com (Emanuele Olivetti) Date: Thu, 18 Jan 2007 18:09:54 +0100 Subject: [Numpy-discussion] numpy.random.permutation bug? Message-ID: <45AFA9E2.8050506@relativita.com> Look at this: ----------bug.py------- import numpy a=numpy.array([1,2]) b=a.sum() print type(b) c=numpy.random.permutation(b) ----------------------- If I run it (Python 2.5, numpy 1.0.1 on a Linux box) I get: ----------- #> python /tmp/bug.py Traceback (most recent call last): File "/tmp/bug.py", line 5, in c=numpy.random.permutation(b) File "mtrand.pyx", line 1227, in mtrand.RandomState.permutation File "mtrand.pyx", line 1211, in mtrand.RandomState.shuffle TypeError: len() of unsized object ----------- permutation() likes 'int' and dislikes 'numpy.int32' integers :( Seems a bug. HTH, Emanuele From kwgoodman at gmail.com Thu Jan 18 12:20:00 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 18 Jan 2007 09:20:00 -0800 Subject: [Numpy-discussion] numpy.random.permutation bug? In-Reply-To: <45AFA9E2.8050506@relativita.com> References: <45AFA9E2.8050506@relativita.com> Message-ID: On 1/18/07, Emanuele Olivetti wrote: > Look at this: > > ----------bug.py------- > import numpy > a=numpy.array([1,2]) > b=a.sum() > print type(b) > c=numpy.random.permutation(b) > ----------------------- > > If I run it (Python 2.5, numpy 1.0.1 on a Linux box) I get: > ----------- > #> python /tmp/bug.py > > Traceback (most recent call last): > File "/tmp/bug.py", line 5, in > c=numpy.random.permutation(b) > File "mtrand.pyx", line 1227, in mtrand.RandomState.permutation > File "mtrand.pyx", line 1211, in mtrand.RandomState.shuffle > TypeError: len() of unsized object > ----------- > > permutation() likes 'int' and dislikes 'numpy.int32' integers :( > Seems a bug. I don't think that bug is particular to int32. For example: >> numpy.random.permutation(1.2) TypeError: len() of unsized object From aisaac at american.edu Thu Jan 18 12:23:39 2007 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 18 Jan 2007 12:23:39 -0500 Subject: [Numpy-discussion] numpy.random.permutation bug? In-Reply-To: <45AFA9E2.8050506@relativita.com> References: <45AFA9E2.8050506@relativita.com> Message-ID: Confirmed for numpy 1.0. Cheers, Alan Isaac From robert.kern at gmail.com Thu Jan 18 12:31:39 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 18 Jan 2007 11:31:39 -0600 Subject: [Numpy-discussion] numpy.random.permutation bug? In-Reply-To: References: <45AFA9E2.8050506@relativita.com> Message-ID: <45AFAEFB.4000309@gmail.com> Keith Goodman wrote: > I don't think that bug is particular to int32. For example: > >>> numpy.random.permutation(1.2) > TypeError: len() of unsized object Well, that should raise an error, though maybe not that error. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Jan 18 12:31:50 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 18 Jan 2007 11:31:50 -0600 Subject: [Numpy-discussion] numpy.random.permutation bug? In-Reply-To: <45AFA9E2.8050506@relativita.com> References: <45AFA9E2.8050506@relativita.com> Message-ID: <45AFAF06.90205@gmail.com> Emanuele Olivetti wrote: > permutation() likes 'int' and dislikes 'numpy.int32' integers :( > Seems a bug. Yup. I should get around to fixing it later tonight. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From emanuele at relativita.com Thu Jan 18 12:34:04 2007 From: emanuele at relativita.com (Emanuele Olivetti) Date: Thu, 18 Jan 2007 18:34:04 +0100 Subject: [Numpy-discussion] numpy.random.permutation bug? In-Reply-To: <45AFAF06.90205@gmail.com> References: <45AFA9E2.8050506@relativita.com> <45AFAF06.90205@gmail.com> Message-ID: <45AFAF8C.6090401@relativita.com> Robert Kern wrote: > Emanuele Olivetti wrote: > > >> permutation() likes 'int' and dislikes 'numpy.int32' integers :( >> Seems a bug. >> > > Yup. I should get around to fixing it later tonight. > > Wow. Superfast! :) Emanuele From kwgoodman at gmail.com Thu Jan 18 12:37:42 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 18 Jan 2007 09:37:42 -0800 Subject: [Numpy-discussion] numpy.random.permutation bug? In-Reply-To: <45AFAEFB.4000309@gmail.com> References: <45AFA9E2.8050506@relativita.com> <45AFAEFB.4000309@gmail.com> Message-ID: On 1/18/07, Robert Kern wrote: > Keith Goodman wrote: > > I don't think that bug is particular to int32. For example: > > > >>> numpy.random.permutation(1.2) > > TypeError: len() of unsized object > > Well, that should raise an error, though maybe not that error. Is my example any different than the one given by Emanuele? He passed in a.sum(). >> numpy.random.permutation(1) array([0]) >> numpy.random.permutation(1.2) TypeError: len() of unsized object From robert.kern at gmail.com Thu Jan 18 12:51:27 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 18 Jan 2007 11:51:27 -0600 Subject: [Numpy-discussion] numpy.random.permutation bug? In-Reply-To: References: <45AFA9E2.8050506@relativita.com> <45AFAEFB.4000309@gmail.com> Message-ID: <45AFB39F.1030702@gmail.com> Keith Goodman wrote: > On 1/18/07, Robert Kern wrote: >> Keith Goodman wrote: >>> I don't think that bug is particular to int32. For example: >>> >>>>> numpy.random.permutation(1.2) >>> TypeError: len() of unsized object >> Well, that should raise an error, though maybe not that error. > > Is my example any different than the one given by Emanuele? He passed > in a.sum(). > >>> numpy.random.permutation(1) > array([0]) >>> numpy.random.permutation(1.2) > TypeError: len() of unsized object Yes. 1.2 is not an integer. In Emanuele's example, a.sum() was an integer, though not an int. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kwgoodman at gmail.com Thu Jan 18 12:57:49 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 18 Jan 2007 09:57:49 -0800 Subject: [Numpy-discussion] numpy.random.permutation bug? In-Reply-To: <45AFB39F.1030702@gmail.com> References: <45AFA9E2.8050506@relativita.com> <45AFAEFB.4000309@gmail.com> <45AFB39F.1030702@gmail.com> Message-ID: On 1/18/07, Robert Kern wrote: > Yes. 1.2 is not an integer. In Emanuele's example, a.sum() was an integer, > though not an int. Sorry, but I'm still confused. Does that mean that this is the expected behaviour? >> numpy.random.permutation(1) array([0]) whereas >> numpy.random.permutation(1.2) should raise an error? From kwgoodman at gmail.com Thu Jan 18 12:59:18 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 18 Jan 2007 09:59:18 -0800 Subject: [Numpy-discussion] numpy.random.permutation bug? In-Reply-To: References: <45AFA9E2.8050506@relativita.com> <45AFAEFB.4000309@gmail.com> <45AFB39F.1030702@gmail.com> Message-ID: On 1/18/07, Keith Goodman wrote: > On 1/18/07, Robert Kern wrote: > > Yes. 1.2 is not an integer. In Emanuele's example, a.sum() was an integer, > > though not an int. > > Sorry, but I'm still confused. Does that mean that this is the > expected behaviour? > > >> numpy.random.permutation(1) > array([0]) > > whereas > > >> numpy.random.permutation(1.2) > > should raise an error? Crap. I should have read the doc string: Type: builtin_function_or_method Base Class: String Form: Namespace: Interactive Docstring: Given an integer, return a shuffled sequence of integers >= 0 and < x; given a sequence, return a shuffled array copy. permutation(x) From mattknox_ca at hotmail.com Thu Jan 18 13:18:22 2007 From: mattknox_ca at hotmail.com (Matt Knox) Date: Thu, 18 Jan 2007 18:18:22 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?ANN=3A_MaskedArray_as_a_subclass_of_?= =?utf-8?q?ndarray_-=09followup?= References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> Message-ID: > This makes sense to me. I'm generally favorable to the new maskedarray > (I actually like the idea of it being a sub-class). I'm just waiting > for people that actually use the MaskedArray to comment. > > For 1.1 I would really like to move most of the often-used sub-classes > of the ndarray to the C-level and merge in functionality from CVXOPT. > > -Travis > I am definitely in favor of the new maskedarray implementation. I've been working with Pierre on a time series module which is a subclass of the new masked array implementation, and having it as a subclass of ndarray definitely has advantages (and no real disadvantages that I am aware of). Moving the implementation to the C-level would be awesome. In particular, __getitem__ and __setitem__ are incredibly slow with masked arrays compared to ndarrays, so using those inside python loops is basically a really bad idea currently. You always have to work with the _data and _mask attributes directly if you are concerned about performance. Also, there is a "bug" in Pierre's current implementation I spoke with him about, but currently have no solution for. numpy.add.accumulate doesn't work on arrays from the new maskedarray implementation, but does with the old one. The problem seems to arise when you over-ride __getitem__ in an ndarray sub-class. See the code below for a demonstration: import numpy import numpy.core.umath as umath from numpy.core.numeric import ndarray import numpy.core.numeric as numeric class Foo1(numeric.ndarray): def __new__(self, data=None): _data = numeric.array(data) return numeric.asanyarray(_data).view(self) def __array_finalize__(self, obj): if not hasattr(self, "_data"): if hasattr(obj,'_data'): self._data = obj._data else: self._data = obj def __array__ (self, t=None, context=None): return self._data def __array_wrap__(self, array, context=None): return Foo1(array) """ if you define this to return something other than what standard ndarray returns, accumulate doesn't work""" def __getitem__(self, index): return self._data[index] #return super(Foo1, self).__getitem__(index) class Foo2(object): def __init__(self, data=None): self._data = numeric.array(data) def __array__ (self, t=None, context=None): return self._data def __array_wrap__(self, array, context=None): return Foo2(array) def __getitem__(self, index): return self._data[index] def __str__(self): return str(self._data) def __add__(self, other): return umath.add(self._data, other._data) if __name__ == "__main__": from numpy import add ac = add.accumulate foo1 = Foo1([1,2,3,4]) foo2 = Foo2([1,2,3,4]) print ac(foo1), ac(foo2) From svetosch at gmx.net Fri Jan 19 04:56:16 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Fri, 19 Jan 2007 10:56:16 +0100 Subject: [Numpy-discussion] ANN: MaskedArray as a subclass of ndarray - followup In-Reply-To: References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> Message-ID: <45B095C0.3040602@gmx.net> Matt Knox schrieb: > > I am definitely in favor of the new maskedarray implementation. I've been > working with Pierre on a time series module which is a subclass of the new > masked array implementation, and having it as a subclass of ndarray definitely > has advantages (and no real disadvantages that I am aware of). > That time series module sounds very interesting! Is it available somewhere, or some documentation? Thank you, Sven From stefan at sun.ac.za Fri Jan 19 05:40:01 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 19 Jan 2007 12:40:01 +0200 Subject: [Numpy-discussion] ANN: MaskedArray as a subclass of ndarray -?followup In-Reply-To: References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> Message-ID: <20070119104001.GC8460@mentat.za.net> On Thu, Jan 18, 2007 at 06:18:22PM +0000, Matt Knox wrote: > > For 1.1 I would really like to move most of the often-used sub-classes > > of the ndarray to the C-level and merge in functionality from CVXOPT. > > Moving the implementation to the C-level would be awesome. In particular, > __getitem__ and __setitem__ are incredibly slow with masked arrays compared to > ndarrays, so using those inside python loops is basically a really bad idea > currently. You always have to work with the _data and _mask attributes directly > if you are concerned about performance. Moving the implementation to the C-level also has its downside. To me, at least, Python code is much more readable and hence easier to maintain. Is there a way that we can implement only the speed-critical methods in C? Cheers St?fan From stefan at sun.ac.za Fri Jan 19 05:52:17 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 19 Jan 2007 12:52:17 +0200 Subject: [Numpy-discussion] ANN: MaskedArray as a subclass of ndarray - followup In-Reply-To: <45B095C0.3040602@gmx.net> References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> <45B095C0.3040602@gmx.net> Message-ID: <20070119105217.GE8460@mentat.za.net> On Fri, Jan 19, 2007 at 10:56:16AM +0100, Sven Schreiber wrote: > Matt Knox schrieb: > > > > > I am definitely in favor of the new maskedarray implementation. I've been > > working with Pierre on a time series module which is a subclass of the new > > masked array implementation, and having it as a subclass of ndarray definitely > > has advantages (and no real disadvantages that I am aware of). > > > > That time series module sounds very interesting! Is it available > somewhere, or some documentation? It's in the scipy sandbox. Edit scipy/Lib/sandbox/enabled_packages.txt and add 'timeseries' on a line of its own, then recompile. Cheers St?fan From mattknox_ca at hotmail.com Fri Jan 19 08:18:18 2007 From: mattknox_ca at hotmail.com (Matt Knox) Date: Fri, 19 Jan 2007 13:18:18 +0000 (UTC) Subject: [Numpy-discussion] ANN: MaskedArray as a subclass of ndarray - followup References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> <45B095C0.3040602@gmx.net> Message-ID: > That time series module sounds very interesting! Is it available > somewhere, or some documentation? > > Thank you, > Sven > Not really any documentation yet, and the code is still in a state of flux, so expect frequent changes/additions at this point still. No concern is being given to "backwards compatability" right now either, so if you download the code now, expect stuff to break when you update to a newer version. Documentation will be available eventually, it's just a matter of time. You can look at the example script for some ideas, but the example script isn't fully updated to reflect the latest code either. If you do play around with it though, I would love to hear your thoughts on it and any criticisms/suggestions you may have. - Matt From mattknox_ca at hotmail.com Fri Jan 19 09:13:52 2007 From: mattknox_ca at hotmail.com (Matt Knox) Date: Fri, 19 Jan 2007 14:13:52 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?ANN=3A_MaskedArray_as_a_subclass_of_?= =?utf-8?q?ndarray=09-=3Ffollowup?= References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> <20070119104001.GC8460@mentat.za.net> Message-ID: > Moving the implementation to the C-level also has its downside. To > me, at least, Python code is much more readable and hence easier to > maintain. > > Is there a way that we can implement only the speed-critical methods > in C? > > Cheers > St?fan > Implementing the whole thing in C also has the side benefit of the possibility making a nice C level api available to these sub-classes. And I suspect the core numpy developers are comfortable enough with C that maintainability is *probably* not a huge concern here. But yeah, implementing even just the speed critical parts in C would still be a nice improvement. - Matt From stefan at sun.ac.za Fri Jan 19 11:08:37 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 19 Jan 2007 18:08:37 +0200 Subject: [Numpy-discussion] ANN: MaskedArray as a subclass of ndarray?-?followup In-Reply-To: References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> <20070119104001.GC8460@mentat.za.net> Message-ID: <20070119160837.GI8460@mentat.za.net> On Fri, Jan 19, 2007 at 02:13:52PM +0000, Matt Knox wrote: > > Moving the implementation to the C-level also has its downside. To > > me, at least, Python code is much more readable and hence easier to > > maintain. > > > > Is there a way that we can implement only the speed-critical methods > > in C? > > > > Cheers > > St?fan > > > > Implementing the whole thing in C also has the side benefit of the possibility > making a nice C level api available to these sub-classes. And I suspect the core > numpy developers are comfortable enough with C that maintainability is > *probably* not a huge concern here. A "nice C level api" sounds like the definition of oxymoron :) Why would we argue for more C than absolutely necessary in a Python-based library? It's not trivial to code or maintain C API based code properly. Isn't that, to some extent at least, why projects like PyPy receive so much attention? I wouldn't be surprised if such a translation of MA has many more bugs than the Python version, unless consistent and rigid unit testing is being done. Have a nice weekend! Cheers St?fan From faltet at carabos.com Fri Jan 19 11:44:21 2007 From: faltet at carabos.com (Francesc Altet) Date: Fri, 19 Jan 2007 17:44:21 +0100 Subject: [Numpy-discussion] ANN: MaskedArray as a subclass of ndarray?-?followup In-Reply-To: <20070119160837.GI8460@mentat.za.net> References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> <20070119104001.GC8460@mentat.za.net> <20070119160837.GI8460@mentat.za.net> Message-ID: <1169225061.2517.14.camel@localhost.localdomain> El dv 19 de 01 del 2007 a les 18:08 +0200, en/na Stefan van der Walt va escriure: > On Fri, Jan 19, 2007 at 02:13:52PM +0000, Matt Knox wrote: > > > Moving the implementation to the C-level also has its downside. To > > > me, at least, Python code is much more readable and hence easier to > > > maintain. > > > > > > Is there a way that we can implement only the speed-critical methods > > > in C? > > > > > > Cheers > > > St?fan > > > > > > > Implementing the whole thing in C also has the side benefit of the possibility > > making a nice C level api available to these sub-classes. And I suspect the core > > numpy developers are comfortable enough with C that maintainability is > > *probably* not a huge concern here. > > A "nice C level api" sounds like the definition of oxymoron :) Why > would we argue for more C than absolutely necessary in a Python-based > library? I agree. In that sense, it would be nice to have a look at Pyrex and implement the main classes with it. Its syntax is very similar to Python, so doing this kind of translation should be pretty easy (at least, much more than doing it in pure C). Pyrex has parts of its syntax that are more C-flavored, and meant to reach high-performance (similar to C code) in the parts of the code that really need it, or allows you to link with pure C code easily. Its only major disavantage from my point of view is that it doesn't have support for easy indexing multi-dimensional arrays (you have to simulate the indexes through unidimensional indexes, but this has never been a problem for me) but if the amount of multidimensional job that you have to do on your code is small (if any), then, it becomes a very poweful tool for doing extensions IMO (you already should know that the random sub-package in NumPy has been built using Pyrex). My 2 cents, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From robert.kern at gmail.com Fri Jan 19 14:00:44 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Jan 2007 13:00:44 -0600 Subject: [Numpy-discussion] ANN: MaskedArray as a subclass of ndarray?-?followup In-Reply-To: <20070119160837.GI8460@mentat.za.net> References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> <20070119104001.GC8460@mentat.za.net> <20070119160837.GI8460@mentat.za.net> Message-ID: <45B1155C.8010803@gmail.com> Stefan van der Walt wrote: > A "nice C level api" sounds like the definition of oxymoron :) Why > would we argue for more C than absolutely necessary in a Python-based > library? Well, it's more often absolutely necessary than you might think. Any common operation on the array should be written in C to avoid the Python function call overhead. We learned that with recarrays. The a.column_name notation (equivalent to a['column_name']) in implemented in a Python method __getattribute__. If you happen to use this in a tight loop, you'll find that __getattribute__ is a bottleneck. Unfortunately, the methods you might want to add or override on an ndarray subclass are going to be the ones that you would want to be efficient. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Fri Jan 19 16:30:43 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 19 Jan 2007 14:30:43 -0700 Subject: [Numpy-discussion] ANN: MaskedArray as a subclass of ndarray -?followup In-Reply-To: References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> <20070119104001.GC8460@mentat.za.net> Message-ID: <45B13883.7080103@ee.byu.edu> Matt Knox wrote: >>Moving the implementation to the C-level also has its downside. To >>me, at least, Python code is much more readable and hence easier to >>maintain. >> >>Is there a way that we can implement only the speed-critical methods >>in C? >> >>Cheers >>St?fan >> >> >> > >Implementing the whole thing in C also has the side benefit of the possibility >making a nice C level api available to these sub-classes. And I suspect the core >numpy developers are comfortable enough with C that maintainability is >*probably* not a huge concern here. > >But yeah, implementing even just the speed critical parts in C would still be a >nice improvement. > > Part of the trouble is that sometimes the speed critical parts are how slow Python functions (e.g. __getitem__) are. You want to avoid that function call and the only way to do that (that I know of) is to create the sub-class in C. -Travis From srinath at txcorp.com Fri Jan 19 16:47:06 2007 From: srinath at txcorp.com (Srinath Vadlamani) Date: Fri, 19 Jan 2007 14:47:06 -0700 Subject: [Numpy-discussion] svn numpy did not build on os x Message-ID: <45B13C5A.4030600@txcorp.com> I just did an svn up (1/19/07) and tried building numpy with no success. It built fine yesterday (1/18/07). This is on OS X tiger. Attached is the error output. I hope that attachments work with the mailing list. -- ========================================================== Srinath Vadlamani srinath at txcorp.com Tech-X Corp. o:303-996-2034 5621 Arapahoe Ave. Suite A f:303-448-7756 Boulder, CO 80303 ========================================================== -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: build.fail URL: From rowen at cesmail.net Fri Jan 19 17:24:57 2007 From: rowen at cesmail.net (Russell E. Owen) Date: Fri, 19 Jan 2007 14:24:57 -0800 Subject: [Numpy-discussion] rewriting in C (was ANN: Masked Array...) References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> <20070119104001.GC8460@mentat.za.net> <45B13883.7080103@ee.byu.edu> Message-ID: In article <45B13883.7080103 at ee.byu.edu>, Travis Oliphant wrote: > Matt Knox wrote: > > >>Moving the implementation to the C-level also has its downside. To > >>me, at least, Python code is much more readable and hence easier to > >>maintain. > >> > >>Is there a way that we can implement only the speed-critical methods > >>in C? > >> > >>Cheers > >>St??fan > >> > >> > >> > > > >Implementing the whole thing in C also has the side benefit of the > >possibility > >making a nice C level api available to these sub-classes. And I suspect the > >core > >numpy developers are comfortable enough with C that maintainability is > >*probably* not a huge concern here. > > > >But yeah, implementing even just the speed critical parts in C would still > >be a > >nice improvement. > > > > > Part of the trouble is that sometimes the speed critical parts are how > slow Python functions (e.g. __getitem__) are. You want to avoid that > function call and the only way to do that (that I know of) is to create > the sub-class in C. I'm curious why the low level stuff is in C instead of C++? I would have thought that C++ templates and possibly even the standard template library would be a huge win for coding array-type classes. -- Russell From pgmdevlist at gmail.com Fri Jan 19 17:28:51 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 19 Jan 2007 17:28:51 -0500 Subject: [Numpy-discussion] =?iso-8859-1?q?ANN=3A_MaskedArray_as_a_subclas?= =?iso-8859-1?q?s_of_ndarray_-=09followup?= In-Reply-To: References: <200701020931.33517.pgmdevlist@gmail.com> <45AEBB33.7050800@ee.byu.edu> Message-ID: <200701191728.51257.pgmdevlist@gmail.com> Eric, Travis, Thanks for the words of encouragements :) I'm all in favor of having maskedarray ported to C, but I won't be able to do it myself anytime soon. And I would have to learn C beforehands. Francesc's suggestion of using Pyrex sounds nice, I'll try and see what I can do with that > Moving the implementation to the C-level would be awesome. In particular, > __getitem__ and __setitem__ are incredibly slow with masked arrays compared > to ndarrays, so using those inside python loops is basically a really bad > idea currently. You always have to work with the _data and _mask attributes > directly if you are concerned about performance. Well, yeah, that's expected: __getitem__ tests whether the mask is defined (not nomask) before trying to access the item. If you're using it in a loop, you call the test each time, which is a bad idea. it's indeed far better to call the test beforehand, and process _data and _mask separately A fix would be to force the mask to an array of booleans all the time, but that would slow things down elsewhere,as a lot of functions are artificially accelerated with the nomask trick. A C implementation may render that trick obsolete... Another possibility would be to force the mask as an bool array, and keep an extra flag on top, like hasmask. Hasmask would be False by default, and set to True only if the mask is full of False. That'd require a mask.any() in __array_finalize__, which might still slow things down. > Also, there is a "bug" in Pierre's current implementation I spoke with him > about, but currently have no solution for. numpy.add.accumulate doesn't > work on arrays from the new maskedarray implementation, but does with the > old one. The fact that it works with 'old' masked arrays doesn't count: they're not real ndarrays. They use the __array__ method to communicate with the rest of numpy, that we shouldn't need. > The problem seems to arise when you over-ride __getitem__ in an > ndarray sub-class. See the code below for a demonstration: I'm not sure that's actually the source of the problem. ufuncs use the __array_wrap__ method to communicate with subclasses. ufuncs methods seem to bypass that. In the meantime, the method of the MA.ufuncs work as expected. Could somebody give me some simple explanation about the behaviour of ufuncs methods, on the Python side ? I'm obviously missing something here... From mjanikas at esri.com Fri Jan 19 17:40:32 2007 From: mjanikas at esri.com (Mark Janikas) Date: Fri, 19 Jan 2007 14:40:32 -0800 Subject: [Numpy-discussion] Fast vector multiplication Message-ID: <627102C921CD9745B070C3B10CB8199B010EBF7F@hardwire.esri.com> Hello all, I am trying to figure out the most efficient way to get the sum of the product of two vectors where id != id. E.g.: X = array([1,2,3]) Y = array([1,2,3]) Z = (1*2) + (1*3) + (2*1) + (2*3) + (3*1) + (3*2) = 22 I could obviously do this with loops, but I was wondering if there is a version of multiply that could be used... or a form of vectorize. Any ideas would be greatly appreciated. Mark Janikas Product Engineer ESRI, Geoprocessing 380 New York St. Redlands, CA 92373 909-793-2853 (2563) mjanikas at esri.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Fri Jan 19 17:51:51 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Fri, 19 Jan 2007 15:51:51 -0700 Subject: [Numpy-discussion] Fast vector multiplication In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBF7F@hardwire.esri.com> References: <627102C921CD9745B070C3B10CB8199B010EBF7F@hardwire.esri.com> Message-ID: On 1/19/07, Mark Janikas wrote: > > Hello all, > > > > I am trying to figure out the most efficient way to get the sum of the > product of two vectors where id != id. > > > > E.g.: > > > > X = array([1,2,3]) > > Y = array([1,2,3]) > > > > Z = (1*2) + (1*3) + (2*1) + (2*3) + (3*1) + (3*2) = 22 > > > > I could obviously do this with loops, but I was wondering if there is a > version of multiply that could be used? or a form of vectorize. Any ideas > would be greatly appreciated. > Isn't this just equivalent to: sum(X) * sum(Y) - dot(X, Y) ? If so, I would just use that. -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjanikas at esri.com Fri Jan 19 18:04:52 2007 From: mjanikas at esri.com (Mark Janikas) Date: Fri, 19 Jan 2007 15:04:52 -0800 Subject: [Numpy-discussion] Fast vector multiplication In-Reply-To: Message-ID: <627102C921CD9745B070C3B10CB8199B010EBF80@hardwire.esri.com> Doh!!! Nice call. Thanks for the input. MJ ________________________________ From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Timothy Hochberg Sent: Friday, January 19, 2007 2:52 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Fast vector multiplication On 1/19/07, Mark Janikas wrote: Hello all, I am trying to figure out the most efficient way to get the sum of the product of two vectors where id != id. E.g.: X = array([1,2,3]) Y = array([1,2,3]) Z = (1*2) + (1*3) + (2*1) + (2*3) + (3*1) + (3*2) = 22 I could obviously do this with loops, but I was wondering if there is a version of multiply that could be used... or a form of vectorize. Any ideas would be greatly appreciated. Isn't this just equivalent to: sum(X) * sum(Y) - dot(X, Y) ? If so, I would just use that. -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Jan 19 19:13:32 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Jan 2007 18:13:32 -0600 Subject: [Numpy-discussion] svn numpy did not build on os x In-Reply-To: <45B13C5A.4030600@txcorp.com> References: <45B13C5A.4030600@txcorp.com> Message-ID: <45B15EAC.2060503@gmail.com> Srinath Vadlamani wrote: > I just did an svn up (1/19/07) and tried building numpy with no > success. It built fine yesterday (1/18/07). This is on OS X tiger. > Attached is the error output. I hope that attachments work with the > mailing list. Well, it's working for me on Intel and OS X 10.4. Do you have a PPC or Intel Mac? fisttp seems to be an SSE3 instruction according to my Googling. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Jan 19 19:15:30 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Jan 2007 18:15:30 -0600 Subject: [Numpy-discussion] svn numpy did not build on os x In-Reply-To: <45B13C5A.4030600@txcorp.com> References: <45B13C5A.4030600@txcorp.com> Message-ID: <45B15F22.2040804@gmail.com> Srinath Vadlamani wrote: > I just did an svn up (1/19/07) and tried building numpy with no > success. It built fine yesterday (1/18/07). This is on OS X tiger. > Attached is the error output. I hope that attachments work with the > mailing list. Oh, and which version of gcc are you using? With a Universal build of Python (which you seem to be using), you will need to use gcc 4. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ellisonbg.net at gmail.com Fri Jan 19 19:29:53 2007 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 19 Jan 2007 17:29:53 -0700 Subject: [Numpy-discussion] svn numpy did not build on os x In-Reply-To: <45B13C5A.4030600@txcorp.com> References: <45B13C5A.4030600@txcorp.com> Message-ID: <6ce0ac130701191629y26524f93vae2146c20ecd500a@mail.gmail.com> I just did a clean checkout of numpy a few minutes ago (Intel mac book, Python 2.4.3 universal) and a simple python setup.py build worked fine. What Python are you using (where did you get it)? Is there something wierd going on with your gcc? Brian On 1/19/07, Srinath Vadlamani wrote: > I just did an svn up (1/19/07) and tried building numpy with no > success. It built fine yesterday (1/18/07). This is on OS X tiger. > Attached is the error output. I hope that attachments work with the > mailing list. > > -- > ========================================================== > Srinath Vadlamani srinath at txcorp.com > Tech-X Corp. o:303-996-2034 > 5621 Arapahoe Ave. Suite A f:303-448-7756 > Boulder, CO 80303 > ========================================================== > > > > Running from numpy source directory. > non-existing path in 'numpy/distutils': 'site.cfg' > No module named __svn_version__ > F2PY Version 2_3513 > blas_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers'] > > lapack_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec'] > > running build > running config_fc > running build_src > building py_modules sources > building extension "numpy.core.multiarray" sources > adding 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h' to sources. > executing numpy/core/code_generators/generate_array_api.py > adding 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h' to sources. > adding 'build/src.macosx-10.3-fat-2.4/numpy/core/src' to include_dirs. > numpy.core - nothing done with h_files= ['build/src.macosx-10.3-fat-2.4/numpy/core/src/scalartypes.inc', 'build/src.macosx-10.3-fat-2.4/numpy/core/src/arraytypes.inc', 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h', 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h'] > building extension "numpy.core.umath" sources > adding 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h' to sources. > executing numpy/core/code_generators/generate_ufunc_api.py > adding 'build/src.macosx-10.3-fat-2.4/numpy/core/__ufunc_api.h' to sources. > adding 'build/src.macosx-10.3-fat-2.4/numpy/core/src' to include_dirs. > numpy.core - nothing done with h_files= ['build/src.macosx-10.3-fat-2.4/numpy/core/src/scalartypes.inc', 'build/src.macosx-10.3-fat-2.4/numpy/core/src/arraytypes.inc', 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h', 'build/src.macosx-10.3-fat-2.4/numpy/core/__ufunc_api.h'] > building extension "numpy.core._sort" sources > adding 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h' to sources. > executing numpy/core/code_generators/generate_array_api.py > adding 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h' to sources. > numpy.core - nothing done with h_files= ['build/src.macosx-10.3-fat-2.4/numpy/core/config.h', 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h'] > building extension "numpy.core.scalarmath" sources > adding 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h' to sources. > executing numpy/core/code_generators/generate_array_api.py > adding 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h' to sources. > executing numpy/core/code_generators/generate_ufunc_api.py > adding 'build/src.macosx-10.3-fat-2.4/numpy/core/__ufunc_api.h' to sources. > numpy.core - nothing done with h_files= ['build/src.macosx-10.3-fat-2.4/numpy/core/config.h', 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h', 'build/src.macosx-10.3-fat-2.4/numpy/core/__ufunc_api.h'] > building extension "numpy.core._dotblas" sources > adding 'numpy/core/blasdot/_dotblas.c' to sources. > building extension "numpy.lib._compiled_base" sources > building extension "numpy.numarray._capi" sources > building extension "numpy.fft.fftpack_lite" sources > building extension "numpy.linalg.lapack_lite" sources > adding 'numpy/linalg/lapack_litemodule.c' to sources. > building extension "numpy.random.mtrand" sources > customize NAGFCompiler > customize AbsoftFCompiler > customize IbmFCompiler > Could not locate executable f77 > Could not locate executable f95 > customize GnuFCompiler > customize GnuFCompiler > customize GnuFCompiler using config > C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 > > compile options: '-Inumpy/core/src -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' > gcc: _configtest.c > _configtest.c:7:2: error: #error No _WIN32 > _configtest.c:7:2: error: #error No _WIN32 > lipo: can't figure out the architecture type of: /var/tmp//ccS3IyWE.out > _configtest.c:7:2: error: #error No _WIN32 > _configtest.c:7:2: error: #error No _WIN32 > lipo: can't figure out the architecture type of: /var/tmp//ccS3IyWE.out > failure. > removing: _configtest.c _configtest.o > building data_files sources > running build_py > copying build/src.macosx-10.3-fat-2.4/numpy/__config__.py -> build/lib.macosx-10.3-fat-2.4/numpy > copying build/src.macosx-10.3-fat-2.4/numpy/distutils/__config__.py -> build/lib.macosx-10.3-fat-2.4/numpy/distutils > copying numpy/f2py/__svn_version__.py -> build/lib.macosx-10.3-fat-2.4/numpy/f2py > copying numpy/core/__svn_version__.py -> build/lib.macosx-10.3-fat-2.4/numpy/core > running build_ext > customize UnixCCompiler > customize UnixCCompiler using build_ext > building 'numpy.core.multiarray' extension > compiling C sources > C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 > > compile options: '-Ibuild/src.macosx-10.3-fat-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.macosx-10.3-fat-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' > gcc: numpy/core/src/multiarraymodule.c > /var/tmp//ccDQNZ7Y.s:18606:no such 386 instruction: `fisttp' > /var/tmp//ccDQNZ7Y.s:18649:no such 386 instruction: `fisttp' > /var/tmp//ccDQNZ7Y.s:18690:no such 386 instruction: `fisttp' > /var/tmp//ccDQNZ7Y.s:20532:no such 386 instruction: `fisttp' > /var/tmp//ccDQNZ7Y.s:20575:no such 386 instruction: `fisttp' > /var/tmp//ccDQNZ7Y.s:20616:no such 386 instruction: `fisttp' > lipo: can't open input file: /var/tmp//cco1Zdpm.out (No such file or directory) > /var/tmp//ccDQNZ7Y.s:18606:no such 386 instruction: `fisttp' > /var/tmp//ccDQNZ7Y.s:18649:no such 386 instruction: `fisttp' > /var/tmp//ccDQNZ7Y.s:18690:no such 386 instruction: `fisttp' > /var/tmp//ccDQNZ7Y.s:20532:no such 386 instruction: `fisttp' > /var/tmp//ccDQNZ7Y.s:20575:no such 386 instruction: `fisttp' > /var/tmp//ccDQNZ7Y.s:20616:no such 386 instruction: `fisttp' > lipo: can't open input file: /var/tmp//cco1Zdpm.out (No such file or directory) > error: Command "gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Ibuild/src.macosx-10.3-fat-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.macosx-10.3-fat-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.macosx-10.3-fat-2.4/numpy/core/src/multiarraymodule.o" failed with exit status 1 > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > From seb.haase at gmx.net Fri Jan 19 20:22:11 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Fri, 19 Jan 2007 17:22:11 -0800 Subject: [Numpy-discussion] how to recognize a numpy-scalar type at the C level and how to convert In-Reply-To: References: Message-ID: Hi again, I finally bought the numpy-book. Browsing through the 370 pages I found this: PyArray IsScalar (op, cls) Evaluates true if op is an instance of PyArrType Type. Is this what I need to check for a given scalar type. Numpy is really much more extensive (comprehensive) than I thought. Thanks, Sebastian Haase On 1/16/07, Sebastian Haase wrote: > Hi! > After converting to numpy my SWIG-wrapped code runs a bunch of error > of this type: > > TypeError: argument number 9: a 'float' is expected, > 'numpy.float32(-2.10786056519)' is received > > This comes of course from the fact that numpy introduced all these new > scalar types (addressing the shortcoming of the (few) scalar type > provided by standard python). > > What would be an easy way to "recognize" such a scalar type in C-API > numpy code and can I extract the C-scalar value from it ? > > Thanks for any hints, > Sebastian Haase > From oliphant at ee.byu.edu Fri Jan 19 20:24:16 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 19 Jan 2007 18:24:16 -0700 Subject: [Numpy-discussion] how to recognize a numpy-scalar type at the C level and how to convert In-Reply-To: References: Message-ID: <45B16F40.9000604@ee.byu.edu> Sebastian Haase wrote: >Hi again, >I finally bought the numpy-book. >Browsing through the 370 pages I found this: >PyArray IsScalar (op, cls) >Evaluates true if op is an instance of PyArrType Type. > >Is this what I need to check for a given scalar type. > > Yes. Remember that cls can be a "Generic Class" as well to test for Integers, Floats, etc. -Travis From srinath at txcorp.com Fri Jan 19 20:41:05 2007 From: srinath at txcorp.com (Srinath Vadlamani) Date: Fri, 19 Jan 2007 18:41:05 -0700 Subject: [Numpy-discussion] svn numpy did not build on os x Message-ID: <45B17331.3030901@txcorp.com> ------------------------------ Message: 5 Date: Fri, 19 Jan 2007 17:29:53 -0700 From: "Brian Granger" Subject: Re: [Numpy-discussion] svn numpy did not build on os x To: "Discussion of Numerical Python" Message-ID: <6ce0ac130701191629y26524f93vae2146c20ecd500a at mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed I just did a clean checkout of numpy a few minutes ago (Intel mac book, Python 2.4.3 universal) and a simple python setup.py build worked fine. What Python are you using (where did you get it)? Is there something wierd going on with your gcc? Brian This is what I'm using: Python 2.4.4 (#1, Oct 18 2006, 10:34:39) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin I'll try a new check out and try it again. I don't know where I got python...where should I get it from? Srinath V. -- ========================================================== Srinath Vadlamani srinath at txcorp.com Tech-X Corp. o:303-996-2034 5621 Arapahoe Ave. Suite A f:303-448-7756 Boulder, CO 80303 ========================================================== From robert.kern at gmail.com Fri Jan 19 22:36:19 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Jan 2007 21:36:19 -0600 Subject: [Numpy-discussion] svn numpy did not build on os x In-Reply-To: <45B17331.3030901@txcorp.com> References: <45B17331.3030901@txcorp.com> Message-ID: <45B18E33.4070404@gmail.com> Srinath Vadlamani wrote: > This is what I'm using: > Python 2.4.4 (#1, Oct 18 2006, 10:34:39) > [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin > > I'll try a new check out and try it again. I don't know where I got > python...where should I get it from? www.python.org Now, what gcc are you using? For example, this is what I have: $ gcc --version i686-apple-darwin8-gcc-4.0.1 (GCC) 4.0.1 (Apple Computer, Inc. build 5367) Copyright (C) 2005 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From numpy at mspacek.mm.st Fri Jan 19 23:28:54 2007 From: numpy at mspacek.mm.st (Martin Spacek) Date: Fri, 19 Jan 2007 20:28:54 -0800 Subject: [Numpy-discussion] concatenate a tuple of lists 10x slower in numpy 1.0.1 Message-ID: <45B19A86.60609@mspacek.mm.st> Hello, I just upgraded from numpy 1.0b5 to 1.0.1, and I noticed that a part of my code that was using concatenate() was suddenly far slower. I downgraded to 1.0, and the slowdown disappeared. Here's the code and the profiler results for 1.0 and 1.0.1: >>> import numpy as np >>> np.version.version '1.0' >>> a = [] >>> for i in range(10): ... a.append([np.random.rand(100000)]) # make nested list of arrays ... >>> import profile >>> profile.run('np.concatenate(tuple(a))') # concatenate tuple of lists # gives a 10 x 100000 array 4 function calls in 0.046 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.045 0.045 0.045 0.045 :0(concatenate) 1 0.000 0.000 0.000 0.000 :0(setprofile) 1 0.001 0.001 0.046 0.046 :1(?) 1 0.000 0.000 0.046 0.046 profile:0(np.concatenate(tuple(a))) 0 0.000 0.000 profile:0(profiler) >>> import numpy as np >>> np.version.version '1.0.1' >>> a = [] >>> for i in range(10): ... a.append([np.random.rand(100000)]) ... >>> import profile >>> profile.run('np.concatenate(tuple(a))') 4 function calls in 0.532 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.531 0.531 0.531 0.531 :0(concatenate) 1 0.000 0.000 0.000 0.000 :0(setprofile) 1 0.001 0.001 0.532 0.532 :1(?) 1 0.000 0.000 0.532 0.532 profile:0(np.concatenate(tuple(a))) 0 0.000 0.000 profile:0(profiler) Going from numpy 1.0 to 1.0.1, there's a slowdown of over 10x. In retrospect, I'm doing this in a weird way. If I get rid of the tuple of lists, replace it with a flat list and use reshape instead, it's much faster and gives me the same 10 x 100000 resulting array: >>> b = [] >>> for i in range(10): ... b.append(np.random.rand(100000)) ... >>> profile.run('np.concatenate(b).reshape(10, 100000)') 5 function calls in 0.023 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.021 0.021 0.021 0.021 :0(concatenate) 1 0.000 0.000 0.000 0.000 :0(reshape) 1 0.000 0.000 0.000 0.000 :0(setprofile) 1 0.001 0.001 0.022 0.022 :1(?) 1 0.000 0.000 0.023 0.023 profile:0(np.concatenate(b).reshape(10, 100000)) 0 0.000 0.000 profile:0(profiler) The reshape method is equally fast for both 1.0 and 1.0.1. Still, I thought it prudent to bring up the slowdown with the tuple of lists method. Is this issue known? I ran these tests in Python 2.4.4 in a Windows console. I use the win32 py24 binaries. Cheers, Martin From oliphant at ee.byu.edu Sat Jan 20 00:27:18 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 19 Jan 2007 22:27:18 -0700 Subject: [Numpy-discussion] concatenate a tuple of lists 10x slower in numpy 1.0.1 In-Reply-To: <45B19A86.60609@mspacek.mm.st> References: <45B19A86.60609@mspacek.mm.st> Message-ID: <45B1A836.4020307@ee.byu.edu> Martin Spacek wrote: > Hello, > > I just upgraded from numpy 1.0b5 to 1.0.1, and I noticed that a part of > my code that was using concatenate() was suddenly far slower. I > downgraded to 1.0, and the slowdown disappeared. Here's the code > and the profiler results for 1.0 and 1.0.1: > I have not heard of this before. I cannot think what might have caused it. -Travis From vmas at carabos.com Sat Jan 20 04:13:36 2007 From: vmas at carabos.com (Vicent Mas (V+)) Date: Sat, 20 Jan 2007 10:13:36 +0100 Subject: [Numpy-discussion] Fast vector multiplication In-Reply-To: <627102C921CD9745B070C3B10CB8199B010EBF7F@hardwire.esri.com> References: <627102C921CD9745B070C3B10CB8199B010EBF7F@hardwire.esri.com> Message-ID: <200701201013.36447.vmas@carabos.com> El Friday, 19 de January de 2007 23:40 Mark Janikas escribi?: > Hello all, > > > > I am trying to figure out the most efficient way to get the sum of > the product of two vectors where id != id. > > > > E.g.: > > > > X = array([1,2,3]) > > Y = array([1,2,3]) > > > > Z = (1*2) + (1*3) + (2*1) + (2*3) + (3*1) + (3*2) = 22 > > > > I could obviously do this with loops, but I was wondering if there is > a version of multiply that could be used... or a form of vectorize. > Any ideas would be greatly appreciated. > > What about this? In [64]: a = numpy.array([1,2,3]) In [65]: b = numpy.array([4,5,6]).reshape(3,1) In [66]: numpy.multiply(a, b).sum() - numpy.dot(a, b).sum() Out[66]: 58 Hope it helps. -- :: \ / Vicent Mas http://www.carabos.com 0;0 / \ C?rabos Coop. Enjoy Data V V " " From david at ar.media.kyoto-u.ac.jp Sun Jan 21 01:04:30 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sun, 21 Jan 2007 15:04:30 +0900 Subject: [Numpy-discussion] rewriting in C (was ANN: Masked Array...) In-Reply-To: References: <200701020931.33517.pgmdevlist@gmail.com> <45AEB1BD.4000003@hawaii.edu> <45AEBB33.7050800@ee.byu.edu> <20070119104001.GC8460@mentat.za.net> <45B13883.7080103@ee.byu.edu> Message-ID: <45B3026E.2050300@ar.media.kyoto-u.ac.jp> Russell E. Owen wrote: > > I'm curious why the low level stuff is in C instead of C++? I would have > thought that C++ templates and possibly even the standard template > library would be a huge win for coding array-type classes. I don't know the specifics for numpy, but C++ has huge problems compared to C for code which is meant to be used by other languages: C++ has no ABI standard, loading C++ classes dynamically through dlopen mechanisms is difficult (it basically means rewriting a C api over C++, AFAIK). Also, template are extremely difficult to use in a sensible way, and many advanced tricks using templates are not well supported by often used compilers (looking at boost sources for compiler specific workaround would give you an idea); in numeric codes, template are useful to have one implementation for all the C type availble (float, double, int, etc...), but those are cases which can easily be generated by code generator tools (like autogen; numpy is using its own). There are cases where C++ is useful compared to C++; I don't think numpy is one of them, David From millman at berkeley.edu Sun Jan 21 08:09:37 2007 From: millman at berkeley.edu (Jarrod Millman) Date: Sun, 21 Jan 2007 05:09:37 -0800 Subject: [Numpy-discussion] testoob and numpy Message-ID: Hello, Is anyone using testoob with numpy (http://testoob.sourceforge.net/)? I am trying to figure out how to use it and would appreciate any pointers to get me started. My goal is to provide some documentation on the wiki about how to do this. It looks like Pearu had it working not too long ago: http://projects.scipy.org/scipy/numpy/changeset/2425 There is also a mention of testoob in the DISTUTILS.txt file. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ From rex at nosyntax.com Mon Jan 22 03:40:35 2007 From: rex at nosyntax.com (rex) Date: Mon, 22 Jan 2007 00:40:35 -0800 Subject: [Numpy-discussion] gcc vs Intel compiler Message-ID: <20070122084035.GL25947@x2.nosyntax.com> Has anyone done any benchmarks to compare Numpy's speed with these two compilers on a Core 2 Duo? Thanks, -rex From sturla at molden.no Mon Jan 22 06:18:49 2007 From: sturla at molden.no (Sturla Molden) Date: Mon, 22 Jan 2007 12:18:49 +0100 (CET) Subject: [Numpy-discussion] f2py Message-ID: <1734.129.240.194.142.1169464729.squirrel@webmail.uio.no> I have a question regarding f2py: Can it map NumPy arrays to Fortran 95 'dope vectors'? I can find many examples on interfacing Fortran 77 with NumPy, but not Fortran 95. This is of course very different as Fortran 77 are always contiguous, whereas Fortran 95 arrays are not. Thus a Fortran 95 compiler will pass a pointer to a struct referred to as a 'dope vector', which by the way is very similar to a NumPy ndarray. But the ABI for a 'dope vector' is implementation (compiler) dependent. So what does f2py do? Can it use Charm? Sturla Molden From pgmdevlist at gmail.com Mon Jan 22 10:07:22 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 22 Jan 2007 10:07:22 -0500 Subject: [Numpy-discussion] Misc Pyrex questions Message-ID: <200701221007.22776.pgmdevlist@gmail.com> Dear All, I started playing with Pyrex this week-end, to see how easy it would be to port some subclasses to C. A good thing is that it's not as bad as I was dreading. I do have a lot of question, however: - Can I create a subclass w/ Pyrex ? If yes, how ? I haven't been able to find any example in the numerous tutorials I browsed through, and my naive attempt to: cdef NewSubclass(ndarray) didn't work at all. - I want to create an output ndarray from a given an input sequence/ndarray. I'm using PyArray_FROM_OTF, which asks me for a dtype. How can I guess the dtype of the input sequence ? In Python, a numpy.asarray(obj) would do the trick, the dtype would be set to the maximum possible. - What'd be a more pyrex-esque way to code the resize and reshape functions ? - Is there a standard way to deal w/ __getitem__ ? especially when the item is not an integer (but a slice or a sequence) ? Thanks a lot in advance for your help. P. PS: Sorry for the cross-posting From faltet at carabos.com Mon Jan 22 11:18:24 2007 From: faltet at carabos.com (Francesc Altet) Date: Mon, 22 Jan 2007 17:18:24 +0100 Subject: [Numpy-discussion] Misc Pyrex questions In-Reply-To: <200701221007.22776.pgmdevlist@gmail.com> References: <200701221007.22776.pgmdevlist@gmail.com> Message-ID: <1169482704.2514.40.camel@localhost.localdomain> El dl 22 de 01 del 2007 a les 10:07 -0500, en/na Pierre GM va escriure: > Dear All, > I started playing with Pyrex this week-end, to see how easy it would be to > port some subclasses to C. A good thing is that it's not as bad as I was > dreading. I do have a lot of question, however: > > - Can I create a subclass w/ Pyrex ? If yes, how ? I haven't been able to find > any example in the numerous tutorials I browsed through, and my naive attempt > to: > cdef NewSubclass(ndarray) > didn't work at all. You should first inform to Pyrex about the definition of ndarray. For this, it's only necessary to declare this in a file (say definitions.pxd): cdef extern from "numpy/arrayobject.h": # Types ctypedef int npy_intp # Classes ctypedef extern class numpy.dtype [object PyArray_Descr]: cdef int type_num, elsize, alignment cdef char type, kind, byteorder, hasobject cdef object fields, typeobj ctypedef extern class numpy.ndarray [object PyArrayObject]: cdef char *data cdef int nd cdef npy_intp *dimensions cdef npy_intp *strides cdef object base cdef dtype descr cdef int flags # The NumPy initialization funtion void import_array() and then include this in your sources: from definitions cimport import_array, ndarray from now on, Pyrex knows about the ndarray object and you should be able to derive a class from it. Note that you should not forget to do a call to import_array() prior to access the numpy machinery. > - I want to create an output ndarray from a given an input sequence/ndarray. > I'm using PyArray_FROM_OTF, which asks me for a dtype. How can I guess the > dtype of the input sequence ? In Python, a numpy.asarray(obj) would do the > trick, the dtype would be set to the maximum possible. Just use the same syntax as in python: cdef ndarray myvar myvar = numpy.asarray(obj) and then you can access myvar as a regular ndarray object. For example: cdef char type mytype = myvar.descr.type That's one of the beauties of pyrex. > > - What'd be a more pyrex-esque way to code the resize and reshape functions ? Again, just use the python flavor: myvar = myvar.reshape((...)) > > - Is there a standard way to deal w/ __getitem__ ? especially when the item is > not an integer (but a slice or a sequence) ? def __getitem__(self, fieldName): if isinstance(key, int): ... elif isinstance(key, slice): ... elif isinstance(key, (list, tuple)): ... Remember, the pyrex manual is your friend. In particular, for object oriented programmimg with pyrex be sure to have a look at: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/extension_types.html Cheers, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From kwgoodman at gmail.com Mon Jan 22 12:23:46 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 22 Jan 2007 09:23:46 -0800 Subject: [Numpy-discussion] array is not broadcastable to correct shape Message-ID: What is the simplest way to make the sum of the positive elements in a Nx1 matrix equal 1? I end up converting the matrix to an array, doing the normalization, then converting the array back to a matrix. >> m = M.randn(10,1) >> m[m > 0,:] = m[m > 0,:] / m[m > 0,:].sum() --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) ValueError: array is not broadcastable to correct shape >> >> a = M.asarray(m) >> a[a > 0,:] = a[a > 0,:] / a[a > 0,:].sum() >> a[a > 0,:].sum() 1.0 >> m = M.asmatrix(a) From persed at princeton.edu Mon Jan 22 15:29:10 2007 From: persed at princeton.edu (Per B. Sederberg) Date: Mon, 22 Jan 2007 20:29:10 +0000 (UTC) Subject: [Numpy-discussion] setmember1d memory leak? Message-ID: Hi Everybody: I was having a massive memory leak problem in some simulations I coded that I traced down to the setmember1d method. After running for about an hour, my program, which should have been taking up .6% of the RAM was taking up 100% of my 4GB of RAM and would eventually get killed by the system. After replacing that method with the following inefficient code, the memory problems disappeared and I can run for days without my memory use increasing beyond .6%: def ismember(a,b): ainb = zeros(len(a),dtype=bool) for item in b: ainb = ainb | (a==item) return ainb Here's the information about my setup: % uname -a Linux 2.4.21-27.0.4.ELsmp #1 SMP Sat Apr 16 18:53:14 EDT 2005 x86_64 x86_64 x86_64 GNU/Linux % python Python 2.4.4 (#1, Jan 21 2007, 12:09:48) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-49)] on linux2 Type "help", "copyright", "credits" or "license" for more information. %>>> import numpy %>>> numpy.version.version '1.0.1' %>>> It is an AMD opteron machine running RedHat Enterprise with an older version of GCC. The same error occurred with an older version of python (the April 2006 release with version 2.4.3). Does anyone have any idea of what might be occurring in setmember1d in combination with this setup that would cause such a massive memory leak? Thanks, Per From robert.kern at gmail.com Mon Jan 22 15:38:54 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 22 Jan 2007 14:38:54 -0600 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: References: Message-ID: <45B520DE.1070602@gmail.com> Per B. Sederberg wrote: > Does anyone have any idea of what might be occurring in setmember1d in > combination with this setup that would cause such a massive memory leak? Can you check out numpy from SVN and see if you can reproduce the leak? I do not see a leak with a recent checkout on OS X with the following code: In [15]: from numpy import * In [16]: ar1 = arange(1000000) In [17]: ar2 = arange(3, 7) In [18]: import itertools In [19]: for i in itertools.count(1): ....: if not i % 1000: ....: print i ....: x = setmember1d(ar1, ar2) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From persed at princeton.edu Mon Jan 22 16:06:09 2007 From: persed at princeton.edu (Per B. Sederberg) Date: Mon, 22 Jan 2007 21:06:09 +0000 (UTC) Subject: [Numpy-discussion] setmember1d memory leak? References: <45B520DE.1070602@gmail.com> Message-ID: Robert Kern gmail.com> writes: > > Per B. Sederberg wrote: > > Does anyone have any idea of what might be occurring in setmember1d in > > combination with this setup that would cause such a massive memory leak? > > Can you check out numpy from SVN and see if you can reproduce the leak? > I do not > see a leak with a recent checkout on OS X with the following code: > > In [15]: from numpy import * > > In [16]: ar1 = arange(1000000) > > In [17]: ar2 = arange(3, 7) > > In [18]: import itertools > > In [19]: for i in itertools.count(1): > ....: if not i % 1000: > ....: print i > ....: x = setmember1d(ar1, ar2) > I tried it out and your test code does NOT break with the svn version of numpy, but memory use explodes with the current stable release. We're talking using up 2GB in around 20 seconds. So the next question is what changed to fix that bug? It looks like the only difference in the code is that the call to argsort now has (kind='mergesort'). Also, why does setmember1d require unique elements? It seems as though that really narrows its uses. Thanks, Per From robert.kern at gmail.com Mon Jan 22 16:21:33 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 22 Jan 2007 15:21:33 -0600 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: References: <45B520DE.1070602@gmail.com> Message-ID: <45B52ADD.2060704@gmail.com> Per B. Sederberg wrote: > So the next question is what changed to fix that bug? It looks like the only > difference in the code is that the call to argsort now has (kind='mergesort'). Possibly the default sort code had a bug that was fixed, too, or something deep in the array object. I've rerun the above test after removing the kind='mergesort', and I still see no leak. > Also, why does setmember1d require unique elements? It seems as though that > really narrows its uses. The algorithm it uses requires it. That's how it can be fast and not written with for loops. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From numpy at mspacek.mm.st Mon Jan 22 17:29:04 2007 From: numpy at mspacek.mm.st (Martin Spacek) Date: Mon, 22 Jan 2007 14:29:04 -0800 Subject: [Numpy-discussion] added 1.0.1 release notes to wiki Message-ID: <45B53AB0.4070103@mspacek.mm.st> Just a note that I've copied over the 1.0.1 release notes from SourceForge: http://sourceforge.net/project/shownotes.php?group_id=1369&release_id=468153 over to the wiki: http://scipy.org/ReleaseNotes/NumPy_1.0 Should 1.0.1 get its own page, as previous 0.9.x releases did? Martin From pgmdevlist at gmail.com Mon Jan 22 18:59:03 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 22 Jan 2007 18:59:03 -0500 Subject: [Numpy-discussion] Misc Pyrex questions In-Reply-To: <1169482704.2514.40.camel@localhost.localdomain> References: <200701221007.22776.pgmdevlist@gmail.com> <1169482704.2514.40.camel@localhost.localdomain> Message-ID: <200701221859.03447.pgmdevlist@gmail.com> On Monday 22 January 2007 11:18, Francesc Altet wrote: > You should first inform to Pyrex about the definition of ndarray. For > this, it's only necessary to declare this in a file (say > definitions.pxd): > from now on, Pyrex knows about the ndarray object and you should be able > to derive a class from it. Note that you should not forget to do a call > to import_array() prior to access the numpy machinery. OK, perfect. Turned out that it was quite silly: I had already two statements inlude 'Python.pxi' include 'numpy.pxi' taken from numpy.random.mtrand, and I was trying to subclass through `numpy.ndarray` when just `ndarray` was what I wanted... Wow, I'm impressed. [about guessing the dtype] > cdef ndarray myvar > myvar = numpy.asarray(obj) That easily ? What's the point of all the PyArray_FROM or _from or _ContiguousFrom implemented in mtrand.numpy.pxi ? I'm afraid I'm missing a point, here... Other example: what's more efficient ? myvar = numpy.empty(shape,dtype) or myvar = PyArray_EMPTY(dims, NPY_TYPE) > > - What'd be a more pyrex-esque way to code the resize and reshape > > functions ? > > Again, just use the python flavor: > > myvar = myvar.reshape((...)) I used that so far and it works nicely, but I wonder whether it was that efficient, hence my question for a more pyrex-esque way. Corollary below. > > - Is there a standard way to deal w/ __getitem__ ? especially when the > > item is not an integer (but a slice or a sequence) ? > > def __getitem__(self, fieldName): > if isinstance(key, int): > ... > elif isinstance(key, slice): > ... > elif isinstance(key, (list, tuple)): OK, I guess a specific example will help me explain myself: One of the issues with the new implementation of MaskedArray is that we overload __getitem__, which slow things down. I just copied the corresponding portion of the python code, namely _data = self._data.__getitem__(index) _mask = self._data.__getitem__(index) pasted it in the .pyx I'm writing, and lo! It works ! But elsewhere in the manual is given the example of a loop using `range`, when one should use some explicit interval, and my understanding was that using python expressions was not as efficient as having more proper C expressions. Is this the case here ? Do I have to reimplement __getitem__ on arrays, or could I just keep on using the current approach ? > Remember, the pyrex manual is your friend. In particular, for object > oriented programmimg with pyrex be sure to have a look at: > > http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/extens >ion_types.html Ahah ! Some more doc ! I couldn't find the link to it on the pyrex front page... (And obviously, I didn't look close enough in the overview page...) My bad, sorry. Francesc, thanks again. I'll probably contact you very soon off list (and will start thinking about a wiki page) From curiousjan at gmail.com Mon Jan 22 23:50:21 2007 From: curiousjan at gmail.com (Jan Strube) Date: Mon, 22 Jan 2007 20:50:21 -0800 Subject: [Numpy-discussion] setmember1d memory leak? Message-ID: I would also like to see a method that doesn't have the unique members requirement. If setmember1d cannot be modified to do that, is there another method that doesn't have these restrictions. Or could one be added? Thanks, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Jan 23 00:05:10 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 22 Jan 2007 23:05:10 -0600 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: References: Message-ID: <45B59786.5070508@gmail.com> Jan Strube wrote: > I would also like to see a method that doesn't have the unique members > requirement. > If setmember1d cannot be modified to do that, is there another method > that doesn't have these restrictions. Or could one be added? If you can write one that's reasonably efficient, yes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cimrman3 at ntc.zcu.cz Tue Jan 23 05:09:04 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 23 Jan 2007 11:09:04 +0100 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: <45B59786.5070508@gmail.com> References: <45B59786.5070508@gmail.com> Message-ID: <45B5DEC0.7070704@ntc.zcu.cz> Robert Kern wrote: > Jan Strube wrote: >> I would also like to see a method that doesn't have the unique members >> requirement. >> If setmember1d cannot be modified to do that, is there another method >> that doesn't have these restrictions. Or could one be added? > > If you can write one that's reasonably efficient, yes. Or you could just call unique1d prior to your call to setmember1d - it was meant to be used that way... you would not loose much speed that way, IMHO. r. From robert.kern at gmail.com Tue Jan 23 13:44:40 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 23 Jan 2007 12:44:40 -0600 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: <45B5DEC0.7070704@ntc.zcu.cz> References: <45B59786.5070508@gmail.com> <45B5DEC0.7070704@ntc.zcu.cz> Message-ID: <45B65798.8020407@gmail.com> Robert Cimrman wrote: > Robert Kern wrote: >> Jan Strube wrote: >>> I would also like to see a method that doesn't have the unique members >>> requirement. >>> If setmember1d cannot be modified to do that, is there another method >>> that doesn't have these restrictions. Or could one be added? >> If you can write one that's reasonably efficient, yes. > > Or you could just call unique1d prior to your call to setmember1d - it > was meant to be used that way... you would not loose much speed that > way, IMHO. But that doesn't do what they want. They want a function that gives the mask against their original array of the elements that are in the other array. The result of setmember1d(unique1d(ar1), unique1d(ar2)) is a mask against unique1d(ar1) not ar1 as they want. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From faltet at carabos.com Tue Jan 23 14:33:23 2007 From: faltet at carabos.com (Francesc Altet) Date: Tue, 23 Jan 2007 20:33:23 +0100 Subject: [Numpy-discussion] Misc Pyrex questions Message-ID: <1169580803.2521.34.camel@localhost.localdomain> El 2007-01-22 23:59:03 GMT, en/na Pierre GM va escriure: > Other example: what's more efficient ? > myvar = numpy.empty(shape,dtype) > or > myvar = PyArray_EMPTY(dims, NPY_TYPE) Ok. The second one indeed, but possibly less than you realize (just a 25%, see some benchmarks below). The main point is, as always, avoiding premature optimization. You should first port your code to Pyrex, and then concentrate on the hot points. If some python calls are critical for you, then, go ahead and choose the C call. > But elsewhere in the manual is given the example of a loop using `range`, when > one should use some explicit interval, and my understanding was that using > python expressions was not as efficient as having more proper C expressions. > Is this the case here ? Do I have to reimplement __getitem__ on arrays, or > could I just keep on using the current approach ? Special methods (like __getitem__) of Pyrex extensions performs exactly the same than a __getitem__ made in pure C extensions. So, you don't have be worried about that. A benchmark that I've made (I was curious too ;), and that I'm attaching, proves this. Here is the run on a pretty old machine: $ python run_bench.py ******************** NumPy times ********************************* time for __len__ (numpy)--> 0.203 sum (numpy)--> 49987.2991813 time for __getitem__ (numpy)--> 0.314 ******************** Pyrex times ********************************* time for __len__ (pyrex)--> 0.198 sum (pyrex)--> 49987.2991813 time for __getitem__ (pyrex)--> 0.172 ********* Comparing NumPy creation times (python and C) ********** time for creating an empty array (python)--> 3.305 time for creating an empty array (C)--> 2.664 In this case, the __getitem__ of Pyrex seems to perform better than the __getitem__ of the ndarray object written in C (almost a 2x, in fact). However, this is probably an ilusion, as the ndarray __getitem__ will do far more work than the Pyrex one. OTOH, the __len__ method is far more simple, and can be taken as the demonstration that the overhead of calling special methods in Pyrex from Python is similar to C counterparts. Finally, the difference of overhead in using a Python or a C call for creating an empty array is shown in the last part of the benchmark. All in all, a 25% of difference is not that much. Cheers, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From faltet at carabos.com Tue Jan 23 14:35:32 2007 From: faltet at carabos.com (Francesc Altet) Date: Tue, 23 Jan 2007 20:35:32 +0100 Subject: [Numpy-discussion] Misc Pyrex questions In-Reply-To: <1169580803.2521.34.camel@localhost.localdomain> References: <1169580803.2521.34.camel@localhost.localdomain> Message-ID: <1169580932.2521.36.camel@localhost.localdomain> Oops, the attachments! (always the same history) El dt 23 de 01 del 2007 a les 20:33 +0100, en/na Francesc Altet va escriure: > El 2007-01-22 23:59:03 GMT, en/na Pierre GM va escriure: > > Other example: what's more efficient ? > > myvar = numpy.empty(shape,dtype) > > or > > myvar = PyArray_EMPTY(dims, NPY_TYPE) > > Ok. The second one indeed, but possibly less than you realize (just a > 25%, see some benchmarks below). The main point is, as always, avoiding > premature optimization. You should first port your code to Pyrex, and > then concentrate on the hot points. If some python calls are critical > for you, then, go ahead and choose the C call. > > > But elsewhere in the manual is given the example of a loop using `range`, when > > one should use some explicit interval, and my understanding was that using > > python expressions was not as efficient as having more proper C expressions. > > Is this the case here ? Do I have to reimplement __getitem__ on arrays, or > > could I just keep on using the current approach ? > > Special methods (like __getitem__) of Pyrex extensions performs exactly > the same than a __getitem__ made in pure C extensions. So, you don't > have be worried about that. > > A benchmark that I've made (I was curious too ;), and that I'm > attaching, proves this. Here is the run on a pretty old machine: > > $ python run_bench.py > ******************** NumPy times ********************************* > time for __len__ (numpy)--> 0.203 > sum (numpy)--> 49987.2991813 > time for __getitem__ (numpy)--> 0.314 > ******************** Pyrex times ********************************* > time for __len__ (pyrex)--> 0.198 > sum (pyrex)--> 49987.2991813 > time for __getitem__ (pyrex)--> 0.172 > ********* Comparing NumPy creation times (python and C) ********** > time for creating an empty array (python)--> 3.305 > time for creating an empty array (C)--> 2.664 > > In this case, the __getitem__ of Pyrex seems to perform better than the > __getitem__ of the ndarray object written in C (almost a 2x, in fact). > However, this is probably an ilusion, as the ndarray __getitem__ will do > far more work than the Pyrex one. OTOH, the __len__ method is far more > simple, and can be taken as the demonstration that the overhead of > calling special methods in Pyrex from Python is similar to C > counterparts. Finally, the difference of overhead in using a Python or a > C call for creating an empty array is shown in the last part of the > benchmark. All in all, a 25% of difference is not that much. > > Cheers, > -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth -------------- next part -------------- """Here are some definitions for sharing between extensions. """ # Standard C functions. cdef extern from "stdlib.h": ctypedef long size_t void *malloc(size_t size) void free(void *ptr) # API for NumPy objects cdef extern from "numpy/arrayobject.h": # Types ctypedef int npy_intp # Functions object PyArray_GETITEM(object arr, void *itemptr) int PyArray_SETITEM(object arr, void *itemptr, object obj) object PyArray_EMPTY(int m, npy_intp *dims, int type, int fortran) # Classes ctypedef extern class numpy.dtype [object PyArray_Descr]: cdef int type_num, elsize, alignment cdef char type, kind, byteorder, hasobject cdef object fields, typeobj ctypedef extern class numpy.ndarray [object PyArrayObject]: cdef char *data cdef int nd cdef npy_intp *dimensions cdef npy_intp *strides cdef object base cdef dtype descr cdef int flags # The NumPy initialization funtion void import_array() -------------- next part -------------- A non-text attachment was scrubbed... Name: Makefile Type: text/x-makefile Size: 129 bytes Desc: not available URL: -------------- next part -------------- definitions.pxd pyrex_bench.pyx run_bench.py setup.py -------------- next part -------------- import numpy from definitions cimport import_array, \ malloc, free, npy_intp, \ PyArray_GETITEM, PyArray_EMPTY, \ ndarray, dtype # NumPy must be initialized import_array() ####################################################################### # Bench for calling special methods ####################################################################### cdef class myarray: """Class that implements some added features from a ndarray object. """ cdef long nslots cdef npy_intp stride cdef void *mydata cdef ndarray myarray def __init__(self, nparray): self.nslots = len(nparray) self.myarray = nparray self.mydata = self.myarray.data self.stride = self.myarray.strides[0] def __len__(self): return self.nslots def __getitem__(self, long key): cdef long offset offset = (key * self.stride) return PyArray_GETITEM(self.myarray, self.mydata + offset) ######################################################################## # Bench for creating NumPy objects ######################################################################## def empty1(shape, dtype_, niter): cdef int i cdef ndarray myarray for i from 0 <= i < niter: myarray = numpy.empty(shape, dtype_) def empty2(shape, dtype dtype_, niter): cdef int i, m, npy_type cdef npy_intp *dims cdef ndarray myarray npy_type = dtype_.type_num m = len(shape) dims = malloc(m * sizeof(npy_type)) for i from 0 <= i < m: dims[i] = shape[i] for i from 0 <= i < niter: myarray = PyArray_EMPTY(m, dims, npy_type, 0) free(dims) ## Local Variables: ## mode: python ## py-indent-offset: 2 ## tab-width: 2 ## fill-column: 78 ## End: -------------- next part -------------- A non-text attachment was scrubbed... Name: run_bench.py Type: text/x-python Size: 1164 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: setup.py Type: text/x-python Size: 565 bytes Desc: not available URL: From joseph.a.crider at boeing.com Tue Jan 23 14:47:51 2007 From: joseph.a.crider at boeing.com (Crider, Joseph A) Date: Tue, 23 Jan 2007 13:47:51 -0600 Subject: [Numpy-discussion] Unable to build numpy in cygwin Message-ID: I have been attempting to build recent versions of numpy and scipy in Cygwin on Windows XP for use in Cygwin, but I haven't succeeded with any version of numpy that I've tried newer than 0.9.8. It seems to be failing in building numarray. Here is the numarray portion of the output: creating build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray compile options: '-Inumpy/core/include -Ibuild/src.cygwin-1.5.23-i686-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: numpy/numarray/_capi.c gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.dll build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_checkFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_clearFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_checkAndReportFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' collect2: ld returned 1 exit status build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_checkFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_clearFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_checkAndReportFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' collect2: ld returned 1 exit status error: Command "gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.dll" failed with exit status 1 Any help would be appreciated. Thanks. J. Allen Crider 256-461-2699 From v-nijs at kellogg.northwestern.edu Tue Jan 23 20:38:19 2007 From: v-nijs at kellogg.northwestern.edu (Vincent Nijs) Date: Tue, 23 Jan 2007 19:38:19 -0600 Subject: [Numpy-discussion] numpy.test error on G5 PPC Message-ID: Mac OS X 10.4.8 G5 PPC Numpy svn: 2602 Target: powerpc-apple-darwin8 Configured with: /private/var/tmp/gcc/gcc-5367.obj~1/src/configure --disable-checking -enable-werror --prefix=/usr --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-]*$/s/$/-4.0/ --with-gxx-include-dir=/include/c++/4.0.0 --with-slibdir=/usr/lib --build=powerpc-apple-darwin8 --host=powerpc-apple-darwin8 --target=powerpc-apple-darwin8 Thread model: posix gcc version 4.0.1 (Apple Computer, Inc. build 5367) ====================================================================== FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packag es/numpy/core/tests/test_scalarmath.py", line 46, in check_large_types assert b == 6765201, "error with %r: got %r" % (t,b) AssertionError: error with : got 0.0 ====================================================================== FAIL: check_types (numpy.core.tests.test_scalarmath.test_types) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packag es/numpy/core/tests/test_scalarmath.py", line 19, in check_types assert a == 1, "error with %r: got %r" % (atype,a) AssertionError: error with : got 1.0 ---------------------------------------------------------------------- Ran 526 tests in 1.133s FAILED (failures=2) From robert.kern at gmail.com Tue Jan 23 20:50:18 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 23 Jan 2007 19:50:18 -0600 Subject: [Numpy-discussion] numpy.test error on G5 PPC In-Reply-To: References: Message-ID: <45B6BB5A.5080203@gmail.com> Vincent Nijs wrote: > ====================================================================== > FAIL: check_large_types (numpy.core.tests.test_scalarmath.test_power) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packag > es/numpy/core/tests/test_scalarmath.py", line 46, in check_large_types > assert b == 6765201, "error with %r: got %r" % (t,b) > AssertionError: error with : got 0.0 > > ====================================================================== > FAIL: check_types (numpy.core.tests.test_scalarmath.test_types) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packag > es/numpy/core/tests/test_scalarmath.py", line 19, in check_types > assert a == 1, "error with %r: got %r" % (atype,a) > AssertionError: error with : got 1.0 I've entered a ticket for these. http://projects.scipy.org/scipy/numpy/ticket/433 However, I don't see either of these behaviors on Intel OS X 10.4, and I do not have a G5 available to me to debug the problem. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rex at nosyntax.com Wed Jan 24 00:59:55 2007 From: rex at nosyntax.com (rex) Date: Tue, 23 Jan 2007 21:59:55 -0800 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL Message-ID: <20070124055955.GY12747@x2.nosyntax.com> I'm trying to build NumPy on an SUSE 10.2 Core 2 Duo system using the Intel compiler and Math Kernel Library, and Python 2.5 I want to see how much faster it is than the SUSE NumPy 1.0.1 rpm that recently became available (huge thanks to whomever it was that created it. Installing NumPy and SciPy has been a lot of work on SUSE systems in the past. Half the time I'd give up in frustration after hours of work with each new distribution, and wait until the next SUSE release and try again). I followed the instructions that are here: http://www.scipy.org/Installing_SciPy/Linux Any distribution with Intel C compiler and MKL I changed the line: cc_exe = 'icc -O2 -g -fomit-frame-pointer -mcpu=pentium4 -mtune=pentium4 -march=pentium4 -msse2 -axWN -Wall' to: cc_exe = 'icc -O2 -g -fomit-frame-pointer -mcpu=pentium4 -mtune=pentium4 -march=pentium4 -msse3 -xP -Wall' But the NumPy compile failed: c2d0:/usr/local/src/numpy-1.0.1 # python setup.py config --compiler=intel build_clib --compiler=intel build_ext --compiler=intel install Running from numpy source directory. Traceback (most recent call last): File "setup.py", line 89, in setup_package() File "setup.py", line 59, in setup_package from numpy.distutils.core import setup File "/usr/local/src/numpy-1.0.1/numpy/distutils/core.py", line 24, in from numpy.distutils.command import build_ext File "/usr/local/src/numpy-1.0.1/numpy/distutils/command/build_ext.py", line 16, in from numpy.distutils.system_info import combine_paths File "/usr/local/src/numpy-1.0.1/numpy/distutils/system_info.py", line 159, in so_ext = distutils.sysconfig.get_config_vars('SO')[0] or '' File "/usr/lib/python2.5/distutils/sysconfig.py", line 493, in get_config_vars func() File "/usr/lib/python2.5/distutils/sysconfig.py", line 352, in _init_posix raise DistutilsPlatformError(my_msg) distutils.errors.DistutilsPlatformError: invalid Python installation: unable to open /usr/lib/python2.5/config/Makefile (No such file or directory) There is no config directory in /usr/lib/python2.5/ These are the subdirectories: /bsddb /compiler /ctypes /disutils /email /encodings /hotshot /lib-dynload /plat-linux2 /site-packages /sqlite3 /wsgiref /xml The only Makefiles under Python2.5 are: ./site-packages/numpy/doc/swig/Makefile ./site-packages/numpy/doc/pyrex/Makefile Help anyone? I hope I don't have to build Python2.5 from source to build NumPy. Thanks, -rex From robert.kern at gmail.com Wed Jan 24 01:02:55 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 24 Jan 2007 00:02:55 -0600 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL In-Reply-To: <20070124055955.GY12747@x2.nosyntax.com> References: <20070124055955.GY12747@x2.nosyntax.com> Message-ID: <45B6F68F.6040805@gmail.com> rex wrote: > distutils.errors.DistutilsPlatformError: invalid Python installation: unable to open /usr/lib/python2.5/config/Makefile (No such file or directory) > > There is no config directory in /usr/lib/python2.5/ You need to install the development package for Python. Usually it's named something like python2.5-devel. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rex at nosyntax.com Wed Jan 24 02:20:18 2007 From: rex at nosyntax.com (rex) Date: Tue, 23 Jan 2007 23:20:18 -0800 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL In-Reply-To: <45B6F68F.6040805@gmail.com> References: <20070124055955.GY12747@x2.nosyntax.com> <45B6F68F.6040805@gmail.com> Message-ID: <20070124072018.GZ12747@x2.nosyntax.com> Robert Kern [2007-01-23 22:18]: > You need to install the development package for Python. Usually it's named > something like python2.5-devel. Thank you. Did that, and NumPy compiled with a Brazillion warnings, but no errors. Then I did: export LD_RUN_PATH=/opt/lib:/opt/intel/cc/9.1.042/lib:/opt/intel/mkl/8.1/lib/32 (because I used the Intel defaults, and those are the correct paths) But since the SUSE NumPy rpm is also installed, how do I determine which version is loaded when the command: from numpy import * is issued in python? Subjectively, it appears the new version is not being used. I expect a significant speed difference using the Intel compiler and MKL on a Core 2 Duo. Why is this so difficult? Thanks, -rex From david at ar.media.kyoto-u.ac.jp Wed Jan 24 02:28:23 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 24 Jan 2007 16:28:23 +0900 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL In-Reply-To: <20070124072018.GZ12747@x2.nosyntax.com> References: <20070124055955.GY12747@x2.nosyntax.com> <45B6F68F.6040805@gmail.com> <20070124072018.GZ12747@x2.nosyntax.com> Message-ID: <45B70A97.3090205@ar.media.kyoto-u.ac.jp> rex wrote: > Robert Kern [2007-01-23 22:18]: >> You need to install the development package for Python. Usually it's named >> something like python2.5-devel. > > Thank you. Did that, and NumPy compiled with a Brazillion warnings, but > no errors. > > Then I did: > export LD_RUN_PATH=/opt/lib:/opt/intel/cc/9.1.042/lib:/opt/intel/mkl/8.1/lib/32 > (because I used the Intel defaults, and those are the correct paths) > > But since the SUSE NumPy rpm is also installed, how do I determine > which version is loaded when the command: > from numpy import * > is issued in python? Subjectively, it appears the new version is not > being used. I expect a significant speed difference using the Intel > compiler and MKL on a Core 2 Duo. > > Why is this so difficult? > It is somewhat difficult to do something somewhat complicated :) In your case, one solution to set the dir where numpy is installed is to use the env variable PYTHONPATH. For example, I don't have root privileges on my workstation, and I installed everything in $HOME/local: python setup.py install --prefix=$HOME/local. Then, numpy is installed in $home/local/lib/python2.4/site-packages, and using this dir as a value for PYTHONPATH works (take care at the python version, replace python2.4 by the version you use) To check which numpy you use, you can simply do a import numpy; print numpy, which should print the full path, cheers, David From rex at nosyntax.com Wed Jan 24 03:33:48 2007 From: rex at nosyntax.com (rex) Date: Wed, 24 Jan 2007 00:33:48 -0800 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL In-Reply-To: <45B70A97.3090205@ar.media.kyoto-u.ac.jp> References: <20070124055955.GY12747@x2.nosyntax.com> <45B6F68F.6040805@gmail.com> <20070124072018.GZ12747@x2.nosyntax.com> <45B70A97.3090205@ar.media.kyoto-u.ac.jp> Message-ID: <20070124083348.GA12747@x2.nosyntax.com> David Cournapeau [2007-01-23 23:40]: > rex wrote: > > Robert Kern [2007-01-23 22:18]: > >> You need to install the development package for Python. Usually it's named > >> something like python2.5-devel. > > > > Thank you. Did that, and NumPy compiled with a Brazillion warnings, but > > no errors. > > > > Then I did: > > export LD_RUN_PATH=/opt/lib:/opt/intel/cc/9.1.042/lib:/opt/intel/mkl/8.1/lib/32 > > (because I used the Intel defaults, and those are the correct paths) > > > > But since the SUSE NumPy rpm is also installed, how do I determine > > which version is loaded when the command: > > from numpy import * > > is issued in python? Subjectively, it appears the new version is not > > being used. I expect a significant speed difference using the Intel > > compiler and MKL on a Core 2 Duo. > > > > Why is this so difficult? > > > It is somewhat difficult to do something somewhat complicated :) In your > case, one solution to set the dir where numpy is installed is to use the > env variable PYTHONPATH. But installing software is typically easy for Windows users. IMO, the difficulty of installing Linux applications is a huge barrier to wider adoption of Linux. I started trying Linux in 1994, and stopped using Windows entirely in 1999. I'm old (66), and becoming dumber as my brain shrinks, but I'm still reasonably sharp (or so I like to think). Over the years, NumPy and SciPy have been very difficult to install for me using SUSE (the SUSE developers have different ideas of what paths should be from most of the rest of the world. If I were King I'd lock 'em all in a room and tell them that if they could not agree on a directory structure for Linux in 48 hours, they would all be killed. Impending death tends to focus attention on the problem.) Two of my two closest friends have advanced degrees. One is a PhD in orbital mechanics from UCSD, and the other did everything for a PhD in computer science but complete his thesis. Both have tried Linux repeatedly, but found it to require more effort to install and maintain than they are willing to expend. If people of this caliber are repelled by Linux, I think developers need to wake up and smell the coffee. When people with PhDs in science are turned off by the difficulty the problem needs to be addressed. > To check which numpy you use, you can simply do a import numpy; print > numpy, which should print the full path, >>> import numpy >>> print numpy What am I to make of this? Is it the rpm numpy or is it the numpy I built using the Intel compiler and MKL? Thanks for the reply, but I'm still confused. -rex -- "When asked, as I frequently am, why I should concern myself so deeply with the conservation of animal life, I reply that I have been very lucky and that throughout my life the world has given me the most enormous pleasure. But the world is as delicate and as complicated as a spider's web. If you touch one thread you send shudders running through all the other threads. We are not just touching the web we are tearing great holes in it." Gerald Durrell 1925-1995. From nwagner at iam.uni-stuttgart.de Wed Jan 24 03:41:47 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Wed, 24 Jan 2007 09:41:47 +0100 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL In-Reply-To: <20070124083348.GA12747@x2.nosyntax.com> References: <20070124055955.GY12747@x2.nosyntax.com> <45B6F68F.6040805@gmail.com> <20070124072018.GZ12747@x2.nosyntax.com> <45B70A97.3090205@ar.media.kyoto-u.ac.jp> <20070124083348.GA12747@x2.nosyntax.com> Message-ID: <45B71BCB.4040302@iam.uni-stuttgart.de> rex wrote: > David Cournapeau [2007-01-23 23:40]: > >> rex wrote: >> >>> Robert Kern [2007-01-23 22:18]: >>> >>>> You need to install the development package for Python. Usually it's named >>>> something like python2.5-devel. >>>> >>> Thank you. Did that, and NumPy compiled with a Brazillion warnings, but >>> no errors. >>> >>> Then I did: >>> export LD_RUN_PATH=/opt/lib:/opt/intel/cc/9.1.042/lib:/opt/intel/mkl/8.1/lib/32 >>> (because I used the Intel defaults, and those are the correct paths) >>> >>> But since the SUSE NumPy rpm is also installed, how do I determine >>> which version is loaded when the command: >>> from numpy import * >>> is issued in python? Subjectively, it appears the new version is not >>> being used. I expect a significant speed difference using the Intel >>> compiler and MKL on a Core 2 Duo. >>> >>> Why is this so difficult? >>> >>> >> It is somewhat difficult to do something somewhat complicated :) In your >> case, one solution to set the dir where numpy is installed is to use the >> env variable PYTHONPATH. >> > > But installing software is typically easy for Windows users. IMO, the > difficulty of installing Linux applications is a huge barrier to wider > adoption of Linux. I started trying Linux in 1994, and stopped using > Windows entirely in 1999. I'm old (66), and becoming dumber as my brain > shrinks, but I'm still reasonably sharp (or so I like to think). Over > the years, NumPy and SciPy have been very difficult to install for me > using SUSE (the SUSE developers have different ideas of what paths > should be from most of the rest of the world. If I were King I'd lock > 'em all in a room and tell them that if they could not agree on a > directory structure for Linux in 48 hours, they would all be > killed. Impending death tends to focus attention on the problem.) > > Two of my two closest friends have advanced degrees. One is a PhD in > orbital mechanics from UCSD, and the other did everything for a PhD in > computer science but complete his thesis. Both have tried Linux > repeatedly, but found it to require more effort to install and maintain > than they are willing to expend. If people of this caliber are repelled > by Linux, I think developers need to wake up and smell the coffee. When > people with PhDs in science are turned off by the difficulty the problem > needs to be addressed. > > >> To check which numpy you use, you can simply do a import numpy; print >> numpy, which should print the full path, >> > > >>>> import numpy >>>> print numpy >>>> > > > What am I to make of this? Is it the rpm numpy or is it the numpy I > built using the Intel compiler and MKL? > > Thanks for the reply, but I'm still confused. > > -rex > Try import scipy print scipy.__version__ import numpy print numpy.__version__ Also very helpful is the output of numpy.show_config() scipy.show_config() Here I get >>> import scipy >>> print scipy.__version__ 0.5.3.dev2602 Nils From robert.kern at gmail.com Wed Jan 24 03:43:38 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 24 Jan 2007 02:43:38 -0600 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL In-Reply-To: <20070124083348.GA12747@x2.nosyntax.com> References: <20070124055955.GY12747@x2.nosyntax.com> <45B6F68F.6040805@gmail.com> <20070124072018.GZ12747@x2.nosyntax.com> <45B70A97.3090205@ar.media.kyoto-u.ac.jp> <20070124083348.GA12747@x2.nosyntax.com> Message-ID: <45B71C3A.8060908@gmail.com> rex wrote: >>>> import numpy >>>> print numpy > > > What am I to make of this? Is it the rpm numpy or is it the numpy I > built using the Intel compiler and MKL? Did you change the distutils installation location? See this page for the various ways to do that: http://docs.python.org/inst/alt-install-windows.html If not, then I strongly suspect that you overwrote the numpy that you installed by RPM. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Wed Jan 24 04:15:52 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 24 Jan 2007 18:15:52 +0900 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL In-Reply-To: <20070124083348.GA12747@x2.nosyntax.com> References: <20070124055955.GY12747@x2.nosyntax.com> <45B6F68F.6040805@gmail.com> <20070124072018.GZ12747@x2.nosyntax.com> <45B70A97.3090205@ar.media.kyoto-u.ac.jp> <20070124083348.GA12747@x2.nosyntax.com> Message-ID: <45B723C8.4030003@ar.media.kyoto-u.ac.jp> rex wrote: > David Cournapeau [2007-01-23 23:40]: > > But installing software is typically easy for Windows users. I am not sure installing numpy using MKL library is easier on windows :) > IMO, the > difficulty of installing Linux applications is a huge barrier to wider > adoption of Linux. I started trying Linux in 1994, and stopped using > Windows entirely in 1999. I'm old (66), and becoming dumber as my brain > shrinks, but I'm still reasonably sharp (or so I like to think). Over > the years, NumPy and SciPy have been very difficult to install for me > using SUSE (the SUSE developers have different ideas of what paths > should be from most of the rest of the world. If I were King I'd lock > 'em all in a room and tell them that if they could not agree on a > directory structure for Linux in 48 hours, they would all be > killed. Impending death tends to focus attention on the problem.) > > Two of my two closest friends have advanced degrees. One is a PhD in > orbital mechanics from UCSD, and the other did everything for a PhD in > computer science but complete his thesis. Both have tried Linux > repeatedly, but found it to require more effort to install and maintain > than they are willing to expend. If people of this caliber are repelled > by Linux, I think developers need to wake up and smell the coffee. When > people with PhDs in science are turned off by the difficulty the problem > needs to be addressed. My parents both have college degree in science, and my mother has a hard time using the mouse, and she is younger than you :) You are not trying to install a program, but compiling it with a specific set of libraries; I don't see how this is difficult than windows (I would say it is worse because of the lack of decent command line shell in windows). About numpy/scipy and linux: installing software on linux is not difficult; installing software which is not packaged for your distribution is. Installing from sources is easier on linux than on windows, in my experience, and building binaries which work on linux across distributions extremely difficult (and not a fun thing to do). The real problem of numpy/scipy now is that it is not packaged yet by all major distributions, but this should change soon. >>>> import numpy >>>> print numpy > > This means that the numpy you use in python is the system-wide path. Either this is the numpy installed by RPM, or the one you build yourself that you overwrote (not something you should do normally). Assuming a standard set of libraries, one way to build numpy and install it in your home directory (and as such avoiding messing up with the system, and you don't need root rights) - uncompress the numpy tarball and go to the numpy directory - run "python setup.py build" - run "python setup.py install --prefix=$HOME/local" to install everything in a local path. Now, when you run python, you need to tell him to look for python modules in your $HOME/local: PYTHONPATH=$HOME/local/lib/python2.4/site-packages/ python or PYTHONPATH=$HOME/local/lib/python2.5/site-packages/ python Depending on your python version (which you can check with python -V) This is not different on linux than on windows, by the way :) cheers, David > What am I to make of this? Is it the rpm numpy or is it the numpy I > built using the Intel compiler and MKL? > > Thanks for the reply, but I'm still confused. > > -rex From cimrman3 at ntc.zcu.cz Wed Jan 24 04:45:29 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 24 Jan 2007 10:45:29 +0100 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: <45B65798.8020407@gmail.com> References: <45B59786.5070508@gmail.com> <45B5DEC0.7070704@ntc.zcu.cz> <45B65798.8020407@gmail.com> Message-ID: <45B72AB9.6080106@ntc.zcu.cz> Robert Kern wrote: > Robert Cimrman wrote: >> Or you could just call unique1d prior to your call to setmember1d - it >> was meant to be used that way... you would not loose much speed that >> way, IMHO. > > But that doesn't do what they want. They want a function that gives the mask > against their original array of the elements that are in the other array. The > result of > > setmember1d(unique1d(ar1), unique1d(ar2)) > > is a mask against > > unique1d(ar1) > > not > > ar1 > > as they want. I see. I was thinking in terms of 'set member' - in set one assumes unique elements. A good name for this kind of function would be 'arraymember'. Naive pseudo-implementation: import numpy as nm def arraymember1d( ar1, ar2 ): ar = ar2.copy().sort() indx = findsorted( ar2, ar1 ) flag = nm.zeros( ar1.shape, dtype = nm.bool ) flag[indx] = True return flag Note that nm.searchsorted cannot be used here - a new function ('findsorted'?) would be needed. r. From robert.kern at gmail.com Wed Jan 24 05:04:03 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 24 Jan 2007 04:04:03 -0600 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: <45B72AB9.6080106@ntc.zcu.cz> References: <45B59786.5070508@gmail.com> <45B5DEC0.7070704@ntc.zcu.cz> <45B65798.8020407@gmail.com> <45B72AB9.6080106@ntc.zcu.cz> Message-ID: <45B72F13.6040202@gmail.com> Robert Cimrman wrote: > Robert Kern wrote: >> Robert Cimrman wrote: >>> Or you could just call unique1d prior to your call to setmember1d - it >>> was meant to be used that way... you would not loose much speed that >>> way, IMHO. >> But that doesn't do what they want. They want a function that gives the mask >> against their original array of the elements that are in the other array. The >> result of >> >> setmember1d(unique1d(ar1), unique1d(ar2)) >> >> is a mask against >> >> unique1d(ar1) >> >> not >> >> ar1 >> >> as they want. > > I see. I was thinking in terms of 'set member' - in set one assumes > unique elements. A good name for this kind of function would be > 'arraymember'. Oh yes, certainly. I think all of the functions in arraysetops should remain set operations. Other functionality will go elsewhere. I was simply pointing out that Jan and Per are specifically asking for other functionality. > Naive pseudo-implementation: > > import numpy as nm > def arraymember1d( ar1, ar2 ): > > ar = ar2.copy().sort() > indx = findsorted( ar2, ar1 ) > flag = nm.zeros( ar1.shape, dtype = nm.bool ) > flag[indx] = True > > return flag > > Note that nm.searchsorted cannot be used here - a new function > ('findsorted'?) would be needed. Yup. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rex at nosyntax.com Wed Jan 24 07:32:00 2007 From: rex at nosyntax.com (rex) Date: Wed, 24 Jan 2007 04:32:00 -0800 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL In-Reply-To: <45B71C3A.8060908@gmail.com> References: <20070124055955.GY12747@x2.nosyntax.com> <45B6F68F.6040805@gmail.com> <20070124072018.GZ12747@x2.nosyntax.com> <45B70A97.3090205@ar.media.kyoto-u.ac.jp> <20070124083348.GA12747@x2.nosyntax.com> <45B71C3A.8060908@gmail.com> Message-ID: <20070124123200.GB12747@x2.nosyntax.com> Robert Kern [2007-01-24 03:22]: > rex wrote: > >>>> import numpy > >>>> print numpy > > > > > > What am I to make of this? Is it the rpm numpy or is it the numpy I > > built using the Intel compiler and MKL? > > Did you change the distutils installation location? See this page for the > various ways to do that: > > http://docs.python.org/inst/alt-install-windows.html > > If not, then I strongly suspect that you overwrote the numpy that you installed > by RPM. Thanks for the URL. It would not have occurred to me to read "alt-install-windows" since I don't do Windows, and haven't since 1999. I did not change the distutils installation location. How do I determine what is loaded when "import numpy" is issued? That is, how does one distinguish between the RPM and the NumPy built using the Intel compiler and MKL? I suppose I can build NumPy under my user directory and reinstall the RPM, and then run some benchmarks. This is exceedingly frustrating; it's not new -- numpy/scipy are very important to me, but they have been very difficult to install under SUSE for years. I'm very grateful for the selfless contributions of those who have created numpy/scipy, but if I -- a highest honors grad -- have great difficulty making it work, what about others? -rex From martin.wiechert at gmx.de Wed Jan 24 10:39:54 2007 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Wed, 24 Jan 2007 16:39:54 +0100 Subject: [Numpy-discussion] C-API: datatype for arrays of indices Message-ID: <200701241639.54401.martin.wiechert@gmx.de> Hi list, does anybody know, what datatype arrays C-API functions like PyArray_SearchSorted PyArray_ArgSort return? Can I rely on them being npy_intp? Thanks, Martin From oliphant at ee.byu.edu Wed Jan 24 10:44:38 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 24 Jan 2007 08:44:38 -0700 Subject: [Numpy-discussion] C-API: datatype for arrays of indices In-Reply-To: <200701241639.54401.martin.wiechert@gmx.de> References: <200701241639.54401.martin.wiechert@gmx.de> Message-ID: <45B77EE6.6080903@ee.byu.edu> Martin Wiechert wrote: >Hi list, > >does anybody know, what datatype arrays C-API functions like > >PyArray_SearchSorted >PyArray_ArgSort > >return? Can I rely on them being npy_intp? > > > Yes. -Travis From martin.wiechert at gmx.de Wed Jan 24 10:48:32 2007 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Wed, 24 Jan 2007 16:48:32 +0100 Subject: [Numpy-discussion] C-API: datatype for arrays of indices In-Reply-To: <45B77EE6.6080903@ee.byu.edu> References: <200701241639.54401.martin.wiechert@gmx.de> <45B77EE6.6080903@ee.byu.edu> Message-ID: <200701241648.32463.martin.wiechert@gmx.de> On Wednesday 24 January 2007 16:44, Travis Oliphant wrote: > Martin Wiechert wrote: > >Hi list, > > > >does anybody know, what datatype arrays C-API functions like > > > >PyArray_SearchSorted > >PyArray_ArgSort > > > >return? Can I rely on them being npy_intp? > > Yes. > > -Travis > Thanks! > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Wed Jan 24 11:30:33 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 24 Jan 2007 09:30:33 -0700 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: <45B72AB9.6080106@ntc.zcu.cz> References: <45B59786.5070508@gmail.com> <45B5DEC0.7070704@ntc.zcu.cz> <45B65798.8020407@gmail.com> <45B72AB9.6080106@ntc.zcu.cz> Message-ID: On 1/24/07, Robert Cimrman wrote: > > Robert Kern wrote: > > Robert Cimrman wrote: > >> Or you could just call unique1d prior to your call to setmember1d - it > >> was meant to be used that way... you would not loose much speed that > >> way, IMHO. > > > > But that doesn't do what they want. They want a function that gives the > mask > > against their original array of the elements that are in the other > array. The > > result of > > > > setmember1d(unique1d(ar1), unique1d(ar2)) > > > > is a mask against > > > > unique1d(ar1) > > > > not > > > > ar1 > > > > as they want. > > I see. I was thinking in terms of 'set member' - in set one assumes > unique elements. A good name for this kind of function would be > 'arraymember'. > > Naive pseudo-implementation: > > import numpy as nm > def arraymember1d( ar1, ar2 ): > > ar = ar2.copy().sort() > indx = findsorted( ar2, ar1 ) > flag = nm.zeros( ar1.shape, dtype = nm.bool ) > flag[indx] = True > > return flag > > Note that nm.searchsorted cannot be used here - a new function > ('findsorted'?) would be needed. Maybe something like countelements(ar1, ar2) : a = sort(ar2) il = a.searchsorted(ar1, side='left') ir = a.searchsorted(ar1, side='right') return ir - il which returns an array containing the number of times the corresponding element of ar1 is contained in ar2. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at marquardt.sc Wed Jan 24 13:46:10 2007 From: christian at marquardt.sc (Christian Marquardt) Date: Wed, 24 Jan 2007 19:46:10 +0100 (CET) Subject: [Numpy-discussion] building NumPy with Intel CC & MKL In-Reply-To: <20070124123200.GB12747@x2.nosyntax.com> References: <20070124055955.GY12747@x2.nosyntax.com> <45B6F68F.6040805@gmail.com> <20070124072018.GZ12747@x2.nosyntax.com> <45B70A97.3090205@ar.media.kyoto-u.ac.jp> <20070124083348.GA12747@x2.nosyntax.com> <45B71C3A.8060908@gmail.com> <20070124123200.GB12747@x2.nosyntax.com> Message-ID: <20247.84.167.73.215.1169664370.squirrel@webmail.marquardt.sc> Dear rex, I'll try to explain... I hope it's not too basic. Python is searching for its modules along the PYTHONPATH, i.e. a list of directories where it expects to find whatever it needs. This is the same as the Unix shell (or the DOC command.com) is looking in the PATH in order to find programs or shell /batch scripts, or the dynamic loader is using LD_LIBRARY_PATH for finding shared libraries. >> >>>> import numpy >> >>>> print numpy >> > > '/usr/lib/python2.5/site-packages/numpy/__init__.pyc'> >> > >> > What am I to make of this? Is it the rpm numpy or is it the numpy I >> > built using the Intel compiler and MKL? This tells from which directory your Python installation actually loaded numpy from: It used the numpy installed in the directory /usr/lib/python2.5/site-packages/numpy By *convention* (as someone already pointed out before), the /usr/lib/python2.5/site-packages is the directory where the original system versions of python packages should be installed. In particular, the rpm version will very likely install it's stuff there. When installing additional python modules or packages via a command like python setup.py install the new packages will also be installed in that system directory. So if you have installed your Intel version of numpy with the above command, you might have overwritten the rpm stuff. There is a way to install in a different place; more on that below. You now probably want to find out if the numpy version in /usr/lib/... is the Intel one or the original rpm one. To do this, you can check if the MKL and Intel libraries are actually loaded by the shared libraries within the numpy installation. You can use the command ldd which shows which shared libraries are loaded by executables or other shared libraries. For example, in my installation, the command ldd /python2.5/site-packages/numpy/linalg/lapack_lite.so gives the following output: MEDEA /opt/apps/lib/python2.5/site-packages/numpy>ldd ./linalg/lapack_lite.so linux-gate.so.1 => (0xffffe000) libmkl_lapack32.so => /opt/intel/mkl/8.1/lib/32/libmkl_lapack32.so (0x40124000) libmkl_lapack64.so => /opt/intel/mkl/8.1/lib/32/libmkl_lapack64.so (0x403c8000) libmkl.so => /opt/intel/mkl/8.1/lib/32/libmkl.so (0x40692000) libvml.so => /opt/intel/mkl/8.1/lib/32/libvml.so (0x406f3000) libguide.so => /opt/intel/mkl/8.1/lib/32/libguide.so (0x4072c000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0x40785000) libimf.so => /opt/intel/fc/9.1/lib/libimf.so (0x40797000) libm.so.6 => /lib/tls/libm.so.6 (0x409d5000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x409f8000) libirc.so => /opt/intel/fc/9.1/lib/libirc.so (0x40a00000) libc.so.6 => /lib/tls/libc.so.6 (0x40a41000) libdl.so.2 => /lib/libdl.so.2 (0x40b5b000) /lib/ld-linux.so.2 (0x80000000) Note that the MKL libraries are referenced at the beginning - just look at the path names! If the output for your lapack_lite.so also contains references to the MKL libs, you've got the Intel version in /usr/lib/python2.5/.... (and have probably overwritten the rpm version). If you do not get any reference to the MKL stuff, it's still the rpm version which does not use the MKL. Now, let's assume that you have the rpm version in /usr/lib/python2.5/.... Maybe you'll want to reinstall the rpm to be sure that this is the case. You now want to a) install your Intel version in some well-defined place, and b) make sure that your Python picks that version up when importing numpy. To achieve a) one way is to reinstall numpy from the source as before, BUT with python setup.py --prefix= ^^^^^^^^^^^^^^^^^^^^^ is the path to some directory, e.g. python setup.py install --prefix=$HOME The latter would install numpy into the directory $HOME/lib/python2.5/site-packages/numpy Do an ls afterwards to check if numpy really arrived there. Instead of using the environment variable HOME, you can of course also any other directory you like. I'll stick to HOME in the following. For b), we have to tell python that modules are waiting for it to be picked up in $HOME/lib/python2.5/site-packages. You do that by setting the environment variable PYTHONPATH, as was also mentioned in this thread. In our example, you would do (for a bash or ksh) export PYTHONPATH=$HOME/lib/python2.5/site-packages As long as this variable is set and exported (i.e., visible in the environment of every program you start), the next instance of Python you'll start will now begin searching for modules in PYTHONPATH whenever you do an import, and only fall back to the ones in the system wide installation if it doesn't find the required module in PYTHONPATH. So, after having set PYTHONPATH in your environment, start up python and import numpy. Do the 'print numpy' within python again and look at the output. Does it point to the installation directory of your Intel version? Great; you're done. If not, this means that something went wrong. It might be that you had a typo in the export command or the directory name; it might mean that you didn't export the PYTHONPATH before running python; it might be that the installation had failed for some reason. You just have to play around a bit and see what's going on... but it's not difficult. Now that you have two versions of numpy, you can (kind of) switch between them by making use of the PYTHONPATH. If you unset it ('unset PYTHONPATH'), the next python session you are starting in the same shell/window will use the original system version. Setting PYTHONPATH again and having it point to your local site-packages directory activates the stuff you've installed in there. You cannot switch between the two numpy versions in the same session); if you want to try the other, you'll have to start a new python and make sure that the PYTHONPATH is set up appropriately for what you want. In the long run, and if you have decided which version to use, you can export PYTHONPATH in your $HOME/.profile and don't have to do that manually each time (which becomes quite cumbersome after a while, of course). Common practice is probably that you install your favourite versions or builds of python modules in one place (i.e. using $HOME as --prefix), and set PYTHONPATH accordingly. It's not a good idea to overwrite the system wide installations, but again - that's purely a convention, nothing more. Hope this helps a bit... Good luck! Christian. >> >> Did you change the distutils installation location? See this page for >> the >> various ways to do that: >> >> http://docs.python.org/inst/alt-install-windows.html >> >> If not, then I strongly suspect that you overwrote the numpy that you >> installed >> by RPM. > > Thanks for the URL. It would not have occurred to me to read > "alt-install-windows" since I don't do Windows, and haven't since 1999. > > I did not change the distutils installation location. > > How do I determine what is loaded when "import numpy" is issued? > That is, how does one distinguish between the RPM and the NumPy built > using the Intel compiler and MKL? I suppose I can build NumPy under my > user directory and reinstall the RPM, and then run some benchmarks. > > This is exceedingly frustrating; it's not new -- numpy/scipy are very > important to me, but they have been very difficult to install under SUSE > for years. > > I'm very grateful for the selfless contributions of those who have > created numpy/scipy, but if I -- a highest honors grad -- have great > difficulty making it work, what about others? > > -rex > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From jeremit0 at gmail.com Wed Jan 24 14:02:05 2007 From: jeremit0 at gmail.com (Jeremy Conlin) Date: Wed, 24 Jan 2007 14:02:05 -0500 Subject: [Numpy-discussion] svn numpy did not build on os x Message-ID: <3db594f70701241102t40e6395axd3c4a059933570ce@mail.gmail.com> I also have been unable to build numpy on OS X although my error(s) are slightly different from the original poster. I just installed python2.5 from the binary at python.org. Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. My gcc version is: $ gcc -v Using built-in specs. Target: i386-apple-darwin8.8.1 Configured with: ../gcc-4.3-20061230/configure --enable-languages=fortran,c,c++,java Thread model: posix gcc version 4.3.0 20061230 (experimental) I got gfortran from hpc.sourceforge.net. I just now noticed that my gcc is "experimental", I hope that isn't the issue. I also just downloaded numpy from svn. Can someone help me? I have copied the errors below. Thanks, Jeremy python setup.py build Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' No module named __svn_version__ F2PY Version 2_3512 blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers'] lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec'] running build running config_fc running build_src building py_modules sources building extension "numpy.core.multiarray" sources Generating build/src.macosx-10.3-fat-2.5/numpy/core/config.h customize NAGFCompiler customize AbsoftFCompiler customize IbmFCompiler Could not locate executable g77 Could not locate executable f77 Could not locate executable f95 customize GnuFCompiler customize Gnu95FCompiler customize Gnu95FCompiler customize Gnu95FCompiler using config C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 compile options: '-I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -Inumpy/core/src -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c' gcc: _configtest.c gcc: unrecognized option '-no-cpp-precomp' cc1: error: unrecognized command line option "-mno-fused-madd" cc1: error: unrecognized command line option "-arch" cc1: error: unrecognized command line option "-arch" cc1: error: unrecognized command line option "-Wno-long-double" gcc: unrecognized option '-no-cpp-precomp' cc1: error: unrecognized command line option "-mno-fused-madd" cc1: error: unrecognized command line option "-arch" cc1: error: unrecognized command line option "-arch" cc1: error: unrecognized command line option "-Wno-long-double" failure. removing: _configtest.c _configtest.o numpy/core/setup.py:48: DeprecationWarning: raising a string exception is deprecated raise "ERROR: Failed to test configuration" Traceback (most recent call last): File "setup.py", line 89, in setup_package() File "setup.py", line 82, in setup_package configuration=configuration ) File "/Users/jlconlin/bin/numpy/numpy/distutils/core.py", line 174, in setup return old_setup(**new_attr) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/jlconlin/bin/numpy/numpy/distutils/command/build_src.py", line 87, in run self.build_sources() File "/Users/jlconlin/bin/numpy/numpy/distutils/command/build_src.py", line 106, in build_sources self.build_extension_sources(ext) File "/Users/jlconlin/bin/numpy/numpy/distutils/command/build_src.py", line 212, in build_extension_sources sources = self.generate_sources(sources, ext) File "/Users/jlconlin/bin/numpy/numpy/distutils/command/build_src.py", line 270, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 48, in generate_config_h raise "ERROR: Failed to test configuration" ERROR: Failed to test configuration From robert.kern at gmail.com Wed Jan 24 14:20:05 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 24 Jan 2007 13:20:05 -0600 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL In-Reply-To: <20070124123200.GB12747@x2.nosyntax.com> References: <20070124055955.GY12747@x2.nosyntax.com> <45B6F68F.6040805@gmail.com> <20070124072018.GZ12747@x2.nosyntax.com> <45B70A97.3090205@ar.media.kyoto-u.ac.jp> <20070124083348.GA12747@x2.nosyntax.com> <45B71C3A.8060908@gmail.com> <20070124123200.GB12747@x2.nosyntax.com> Message-ID: <45B7B165.4050907@gmail.com> rex wrote: > Thanks for the URL. It would not have occurred to me to read > "alt-install-windows" since I don't do Windows, and haven't since 1999. The name of the file is a quirk of the documentation generation program. The table of contents, though, is quite clear about that chapter's generality. http://docs.python.org/inst/inst.html > This is exceedingly frustrating; it's not new -- numpy/scipy are very > important to me, but they have been very difficult to install under SUSE > for years. Yes. I would never inflict SuSE on anyone. I encourage you to try Ubuntu. It is mostly sane and especially so when it comes to Python. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Wed Jan 24 14:21:03 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 24 Jan 2007 13:21:03 -0600 Subject: [Numpy-discussion] svn numpy did not build on os x In-Reply-To: <3db594f70701241102t40e6395axd3c4a059933570ce@mail.gmail.com> References: <3db594f70701241102t40e6395axd3c4a059933570ce@mail.gmail.com> Message-ID: <45B7B19F.4070901@gmail.com> Jeremy Conlin wrote: > I also have been unable to build numpy on OS X although my error(s) > are slightly different from the original poster. I just installed > python2.5 from the binary at python.org. > > Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) > [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > > > My gcc version is: > > $ gcc -v > Using built-in specs. > Target: i386-apple-darwin8.8.1 > Configured with: ../gcc-4.3-20061230/configure > --enable-languages=fortran,c,c++,java > Thread model: posix > gcc version 4.3.0 20061230 (experimental) > > > I got gfortran from hpc.sourceforge.net. I just now noticed that my > gcc is "experimental", I hope that isn't the issue. It probably is the issue. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lists.steve at arachnedesign.net Wed Jan 24 14:23:48 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Wed, 24 Jan 2007 14:23:48 -0500 Subject: [Numpy-discussion] svn numpy did not build on os x In-Reply-To: <45B7B19F.4070901@gmail.com> References: <3db594f70701241102t40e6395axd3c4a059933570ce@mail.gmail.com> <45B7B19F.4070901@gmail.com> Message-ID: <6107B663-5BD2-4E9E-AE39-86B5905006DC@arachnedesign.net> Hi, >> I got gfortran from hpc.sourceforge.net. I just now noticed that my >> gcc is "experimental", I hope that isn't the issue. > > It probably is the issue. One thing to note is that (at least when I d/l'd it), the hpc fortran download also has inside it a version of gcc which it will dump into / usr/local/bin. You're normal/system-installed gcc is still in /usr/bin, however. Maybe you need to do some command-line gymnastics to make sure you pick up /usr/bin/gcc as opposed to the /usr/local/bin Just a thought, -steve From jeremit0 at gmail.com Wed Jan 24 14:30:38 2007 From: jeremit0 at gmail.com (Jeremy Conlin) Date: Wed, 24 Jan 2007 14:30:38 -0500 Subject: [Numpy-discussion] svn numpy did not build on os x In-Reply-To: <6107B663-5BD2-4E9E-AE39-86B5905006DC@arachnedesign.net> References: <3db594f70701241102t40e6395axd3c4a059933570ce@mail.gmail.com> <45B7B19F.4070901@gmail.com> <6107B663-5BD2-4E9E-AE39-86B5905006DC@arachnedesign.net> Message-ID: <3db594f70701241130i64854658tf77072557cac965c@mail.gmail.com> On 1/24/07, Steve Lianoglou wrote: > Hi, > > >> I got gfortran from hpc.sourceforge.net. I just now noticed that my > >> gcc is "experimental", I hope that isn't the issue. > > > > It probably is the issue. > > One thing to note is that (at least when I d/l'd it), the hpc fortran > download also has inside it a version of gcc which it will dump into / > usr/local/bin. > > You're normal/system-installed gcc is still in /usr/bin, however. > > Maybe you need to do some command-line gymnastics to make sure you > pick up /usr/bin/gcc as opposed to the /usr/local/bin > > Just a thought, > -steve That was the issue. I changed the order of the items in PATH to ensure /usr/bin comes before /usr/local/bin and now it works. Thanks for the help! Jeremy From robert.kern at gmail.com Wed Jan 24 14:34:31 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 24 Jan 2007 13:34:31 -0600 Subject: [Numpy-discussion] svn numpy did not build on os x In-Reply-To: <6107B663-5BD2-4E9E-AE39-86B5905006DC@arachnedesign.net> References: <3db594f70701241102t40e6395axd3c4a059933570ce@mail.gmail.com> <45B7B19F.4070901@gmail.com> <6107B663-5BD2-4E9E-AE39-86B5905006DC@arachnedesign.net> Message-ID: <45B7B4C7.2070000@gmail.com> Steve Lianoglou wrote: > Hi, > >>> I got gfortran from hpc.sourceforge.net. I just now noticed that my >>> gcc is "experimental", I hope that isn't the issue. >> It probably is the issue. > > One thing to note is that (at least when I d/l'd it), the hpc fortran > download also has inside it a version of gcc which it will dump into / > usr/local/bin. This is not currently true. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From joseph.a.crider at boeing.com Wed Jan 24 16:40:45 2007 From: joseph.a.crider at boeing.com (Crider, Joseph A) Date: Wed, 24 Jan 2007 15:40:45 -0600 Subject: [Numpy-discussion] Unable to build numpy in cygwin In-Reply-To: Message-ID: I am still clueless as to why numpy.numarray will not build for numpy 1.0 and later. As I am currently not using much from either numpy or scipy, but do want to be able to use some of the 1.0 features (especially the order keyword for sort on record arrays), I've decided to disable most of the subpackages of scipy that I'm not using right now and disable oldnumeric and numarray in numpy. That has made it possible to build in cygwin and may be sufficient for my needs for now, but I would still like to resolve this problem, as I hope to use more of numpy and scipy in the future. -----Original Message----- From: Crider, Joseph A Sent: Tuesday, January 23, 2007 1:48 PM To: numpy-discussion at scipy.org Subject: [Numpy-discussion] Unable to build numpy in cygwin I have been attempting to build recent versions of numpy and scipy in Cygwin on Windows XP for use in Cygwin, but I haven't succeeded with any version of numpy that I've tried newer than 0.9.8. It seems to be failing in building numarray. Here is the numarray portion of the output: creating build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray compile options: '-Inumpy/core/include -Ibuild/src.cygwin-1.5.23-i686-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: numpy/numarray/_capi.c gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.dll build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_checkFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_clearFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_checkAndReportFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' collect2: ld returned 1 exit status build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_checkFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_clearFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function `NA_checkAndReportFPErrors': /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: undefined reference to `_fetestexcept' /cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: undefined reference to `_feclearexcept' collect2: ld returned 1 exit status error: Command "gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.dll" failed with exit status 1 Any help would be appreciated. Thanks. J. Allen Crider 256-461-2699 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion From oliphant at ee.byu.edu Wed Jan 24 16:57:55 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 24 Jan 2007 14:57:55 -0700 Subject: [Numpy-discussion] Unable to build numpy in cygwin In-Reply-To: References: Message-ID: <45B7D663.1010105@ee.byu.edu> Crider, Joseph A wrote: >I am still clueless as to why numpy.numarray will not build for numpy >1.0 and later. As I am currently not using much from either numpy or >scipy, but do want to be able to use some of the 1.0 features >(especially the order keyword for sort on record arrays), I've decided >to disable most of the subpackages of scipy that I'm not using right now >and disable oldnumeric and numarray in numpy. That has made it possible >to build in cygwin and may be sufficient for my needs for now, but I >would still like to resolve this problem, as I hope to use more of numpy >and scipy in the future. > > It looks like the issue is in the platform-dependent code for checking the exception flag. The problem is that this code should be included in the distribution on CYGWIN. The numpy/core/include/numpy/fenv/ directory contains implementations of all of the "undefined reference" routines (without the preceeding underscore). Can you tell if this code is getting compiled during compilation on your platform? Some configuration issue is causing the code to not be available. In the short term, if you don't need numarray C-API compatibility, then just disable that package. It looks like the problematic functions are fetestexcept and feclearexcept. Are there any other errors given previously that would indicate a problem in finding the include files where the functions are defined? Wait. I see the problem now. I just copied over the stuff from numarray for _capi.c but we need to look for the headers in the fenv/fenv.c directory for CYGWIN (just like it's done in ufuncobject.h). I'll submit a fix to SVN shortly. -Travis >-----Original Message----- >From: Crider, Joseph A >Sent: Tuesday, January 23, 2007 1:48 PM >To: numpy-discussion at scipy.org >Subject: [Numpy-discussion] Unable to build numpy in cygwin > >I have been attempting to build recent versions of numpy and scipy in >Cygwin on Windows XP for use in Cygwin, but I haven't succeeded with any >version of numpy that I've tried newer than 0.9.8. It seems to be >failing in building numarray. Here is the numarray portion of the >output: > >creating build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray >compile options: '-Inumpy/core/include >-Ibuild/src.cygwin-1.5.23-i686-2.4/numpy/core -Inumpy/core/src >-Inumpy/core/include -I/usr/include/python2.4 -c' >gcc: numpy/numarray/_capi.c >gcc -shared -Wl,--enable-auto-image-base >build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o >-L/usr/lib/python2.4/config -lpython2.4 -o >build/lib.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.dll >build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function >`NA_checkFPErrors': >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: >undefined reference to `_fetestexcept' >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: >undefined reference to `_feclearexcept' >build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function >`NA_clearFPErrors': >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: >undefined reference to `_fetestexcept' >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: >undefined reference to `_feclearexcept' >build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function >`NA_checkAndReportFPErrors': >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: >undefined reference to `_fetestexcept' >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: >undefined reference to `_feclearexcept' >collect2: ld returned 1 exit status >build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function >`NA_checkFPErrors': >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: >undefined reference to `_fetestexcept' >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: >undefined reference to `_feclearexcept' >build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function >`NA_clearFPErrors': >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: >undefined reference to `_fetestexcept' >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: >undefined reference to `_feclearexcept' >build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o: In function >`NA_checkAndReportFPErrors': >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2947: >undefined reference to `_fetestexcept' >/cygdrive/c/Temp/Python/numpy-1.0.1/numpy/numarray/_capi.c:2954: >undefined reference to `_feclearexcept' >collect2: ld returned 1 exit status >error: Command "gcc -shared -Wl,--enable-auto-image-base >build/temp.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.o >-L/usr/lib/python2.4/config -lpython2.4 -o >build/lib.cygwin-1.5.23-i686-2.4/numpy/numarray/_capi.dll" failed with >exit status 1 > >Any help would be appreciated. Thanks. > >J. Allen Crider >256-461-2699 > >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at scipy.org >http://projects.scipy.org/mailman/listinfo/numpy-discussion >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at scipy.org >http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > From rex at nosyntax.com Wed Jan 24 19:25:33 2007 From: rex at nosyntax.com (rex) Date: Wed, 24 Jan 2007 16:25:33 -0800 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL (solved!) Message-ID: <20070125002533.GE12747@x2.nosyntax.com> Christian Marquardt [2007-01-24 11:09]: > > I'll try to explain... I hope it's not too basic. Christian, at this point you could explain that shoes are not interchangeable -- that they are built to be worn on the left foot or the right foot -- and I'd be grateful for the explanation. I've left much detail in what follows in the hope that the details may help someone who is also having trouble using the Intel MKL. > Python is searching for its modules along the PYTHONPATH, i.e. a list > of directories where it expects to find whatever it needs. This is the > same as the Unix shell (or the DOC command.com) is looking in the PATH in > order to find programs or shell /batch scripts, or the dynamic loader is using > LD_LIBRARY_PATH for finding shared libraries. > > >> >>>> import numpy > >> >>>> print numpy > >> > >> '/usr/lib/python2.5/site-packages/numpy/__init__.pyc'> > >> > > >> > What am I to make of this? Is it the rpm numpy or is it the numpy I > >> > built using the Intel compiler and MKL? > > This tells from which directory your Python installation actually loaded > numpy from: It used the numpy installed in the directory > > /usr/lib/python2.5/site-packages/numpy > > By *convention* (as someone already pointed out before), the > /usr/lib/python2.5/site-packages is the directory where the original > system versions of python packages should be installed. In particular, the > rpm version will very likely install it's stuff there. It did. > When installing additional python modules or packages via a command like > > python setup.py install > > the new packages will also be installed in that system directory. So if > you have installed your Intel version of numpy with the above command, you > might have overwritten the rpm stuff. There is a way to install in a > different place; more on that below. I'm 95% sure that command put numpy in /usr/local/lib/python25/site-packages It's possible I used --prefix= , but I don't recall doing so. > You now probably want to find out if the numpy version in /usr/lib/... is > the Intel one or the original rpm one. To do this, you can check if the > MKL and Intel libraries are actually loaded by the shared libraries within > the numpy installation. You can use the command ldd which shows which > shared libraries are loaded by executables or other shared libraries. For > example, in my installation, the command > > ldd /python2.5/site-packages/numpy/linalg/lapack_lite.so > > gives the following output: > > MEDEA /opt/apps/lib/python2.5/site-packages/numpy>ldd > ./linalg/lapack_lite.so > linux-gate.so.1 => (0xffffe000) > libmkl_lapack32.so => /opt/intel/mkl/8.1/lib/32/libmkl_lapack32.so > (0x40124000) > libmkl_lapack64.so => /opt/intel/mkl/8.1/lib/32/libmkl_lapack64.so > (0x403c8000) > libmkl.so => /opt/intel/mkl/8.1/lib/32/libmkl.so (0x40692000) > libvml.so => /opt/intel/mkl/8.1/lib/32/libvml.so (0x406f3000) > libguide.so => /opt/intel/mkl/8.1/lib/32/libguide.so (0x4072c000) > libpthread.so.0 => /lib/tls/libpthread.so.0 (0x40785000) > libimf.so => /opt/intel/fc/9.1/lib/libimf.so (0x40797000) > libm.so.6 => /lib/tls/libm.so.6 (0x409d5000) > libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x409f8000) > libirc.so => /opt/intel/fc/9.1/lib/libirc.so (0x40a00000) > libc.so.6 => /lib/tls/libc.so.6 (0x40a41000) > libdl.so.2 => /lib/libdl.so.2 (0x40b5b000) > /lib/ld-linux.so.2 (0x80000000) > > Note that the MKL libraries are referenced at the beginning - just look at > the path names! If the output for your lapack_lite.so also contains > references to the MKL libs, you've got the Intel version in > /usr/lib/python2.5/.... (and have probably overwritten the rpm version). > If you do not get any reference to the MKL stuff, it's still the rpm > version which does not use the MKL. > > Now, let's assume that you have the rpm version in /usr/lib/python2.5/.... > Maybe you'll want to reinstall the rpm to be sure that this is the case. > > You now want to a) install your Intel version in some well-defined place, > and b) make sure that your Python picks that version up when importing > numpy. > > To achieve a) one way is to reinstall numpy from the source as before, BUT > with > > python setup.py --prefix= > ^^^^^^^^^^^^^^^^^^^^^ > > is the path to some directory, e.g. > > python setup.py install --prefix=$HOME > > The latter would install numpy into the directory > > $HOME/lib/python2.5/site-packages/numpy > > Do an ls afterwards to check if numpy really arrived there. Instead of > using the environment variable HOME, you can of course also any other > directory you like. I'll stick to HOME in the following. > > For b), we have to tell python that modules are waiting for it to be > picked up in $HOME/lib/python2.5/site-packages. You do that by setting the > environment variable PYTHONPATH, as was also mentioned in this thread. In > our example, you would do (for a bash or ksh) > > export PYTHONPATH=$HOME/lib/python2.5/site-packages > > As long as this variable is set and exported (i.e., visible in the > environment of every program you start), the next instance of Python > you'll start will now begin searching for modules in PYTHONPATH whenever > you do an import, and only fall back to the ones in the system wide > installation if it doesn't find the required module in PYTHONPATH. > > So, after having set PYTHONPATH in your environment, start up python and > import numpy. Do the 'print numpy' within python again and look at the > output. Does it point to the installation directory of your Intel version? > Great; you're done. If not, this means that something went wrong. It might > be that you had a typo in the export command or the directory name; it > might mean that you didn't export the PYTHONPATH before running python; it > might be that the installation had failed for some reason. You just have > to play around a bit and see what's going on... but it's not difficult. It is when one cannot recall what one did yesterday. :( That's an overstatement, but my recall is becoming unreliable. > Now that you have two versions of numpy, you can (kind of) switch between > them by making use of the PYTHONPATH. If you unset it ('unset > PYTHONPATH'), the next python session you are starting in the same > shell/window will use the original system version. Setting PYTHONPATH > again and having it point to your local site-packages directory activates > the stuff you've installed in there. You cannot switch between the two > numpy versions in the same session); if you want to try the other, you'll > have to start a new python and make sure that the PYTHONPATH is set up > appropriately for what you want. > > In the long run, and if you have decided which version to use, you can > export PYTHONPATH in your $HOME/.profile and don't have to do that > manually each time (which becomes quite cumbersome after a while, of > course). > > Common practice is probably that you install your favourite versions or > builds of python modules in one place (i.e. using $HOME as --prefix), and > set PYTHONPATH accordingly. It's not a good idea to overwrite the system > wide installations, but again - that's purely a convention, nothing more. > > Hope this helps a bit... Good luck! Thank you for taking the time to write such a detailed explanation. If only the documentation were so detailed... I looked in /usr/lib/python2.5/site-packages/numpy and it was not obvious whether the rpm version is there or the version I compiled. So I did a 'find' from the /: find . -name "ctypeslib*" One of the results was: ./usr/local/lib/python2.5/site-packages/numpy/ctypeslib.py So the python setup.py command defaulted to /usr/local/... (a Good Thing, IMHO). I did: export PYTHONPATH=/usr/local/lib/python2.5/site-packages python Python 2.5 (r25:51908, Nov 27 2006, 19:14:46) [GCC 4.1.2 20061115 (prerelease) (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>import numpy Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.5/site-packages/numpy/__init__.py", line 36, in import core File "/usr/local/lib/python2.5/site-packages/numpy/core/__init__.py", line 5, in import multiarray ImportError: libsvml.so: cannot open shared object file: No such file or directory So I checked: ~> ldd /usr/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so linux-gate.so.1 => (0xffffe000) libpthread.so.0 => /lib/libpthread.so.0 (0xb7cc4000) libc.so.6 => /lib/libc.so.6 (0xb7b96000) /lib/ld-linux.so.2 (0x80000000) ~> ldd /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so linux-gate.so.1 => (0xffffe000) libmkl_lapack32.so => /opt/intel/mkl/8.1/lib/32/libmkl_lapack32.so (0xb7bd1000) libmkl_lapack64.so => /opt/intel/mkl/8.1/lib/32/libmkl_lapack64.so (0xb7907000) libmkl.so => /opt/intel/mkl/8.1/lib/32/libmkl.so (0xb78a6000) libvml.so => /opt/intel/mkl/8.1/lib/32/libvml.so (0xb786d000) libpthread.so.0 => /lib/libpthread.so.0 (0xb7830000) libsvml.so => not found libimf.so => not found libm.so.6 => /lib/libm.so.6 (0xb780a000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb77fe000) libirc.so => not found libc.so.6 => /lib/libc.so.6 (0xb76cf000) libdl.so.2 => /lib/libdl.so.2 (0xb76cb000) libguide.so => /opt/intel/mkl/8.1/lib/32/libguide.so (0xb7696000) /lib/ld-linux.so.2 (0x80000000) At this point my fading brain managed to recall that a 'source' command had to be issued to use the Intel compiler (icc). ~> source /opt/intel/cc/9.1.042/bin/iccvars.sh ~> python Python 2.5 (r25:51908, Nov 27 2006, 19:14:46) [GCC 4.1.2 20061115 (prerelease) (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> print numpy >>> Ah, FINALLY! So did a speed test with a little Monte Carlo program I wrote: ''' A program that uses Monte Carlo to estimate how often the number of rare events with a Poisson distribution will differ by a given amount. ''' import numpy as n from numpy.random import poisson from time import time lam = 4.0 # mu & var for Poisson distributed rands (they are equal in Poisson) N = 10 #number of times to run the program maxNumEvents = 20 #events larger than this are ignored numPois = 100000 #number of pairs of outcomes to generate freqA = 2 #number of times event A occurred freqB = 6 #number of times event B occurred print "#rands fraction [freqA,freqB] fraction [lam,lam] largest% total[mean,mean]" t0 = time() for g in range(1): for h in range(N): suma = n.zeros((maxNumEvents+1,maxNumEvents+1), int) #possible outcomes array count = poisson(lam, size =(numPois,2)) #generate array of pairs of Poissons for i in range(numPois): #if count[i,0] > maxNumEvents: continue #if count[i,1] > maxNumEvents: continue suma[count[i,0],count[i,1]] += 1 d = n.sum(suma) print d, float(suma[freqA,freqB])/d, float(suma[lam,lam])/d , suma.max(), suma[lam,lam] print 'time', time()-t0 Using the SUSE rpm: python relative_risk.py #rands fraction [2,6] fraction [lam,lam] largest% total[mean,mean] 100000 0.01539 0.03869 3869 3869 100000 0.01534 0.03766 3907 3766 100000 0.01553 0.03841 3859 3841 100000 0.01496 0.03943 3943 3943 100000 0.01513 0.03829 3856 3829 100000 0.01485 0.03825 3993 3825 100000 0.01545 0.03716 3859 3716 100000 0.01526 0.03909 3919 3909 100000 0.01491 0.03826 3913 3826 100000 0.01478 0.03771 3782 3771 time 2.38847184181 Using the MKL version: python relative_risk.py #rands fraction [2,6] fraction [lam,lam] largest% total[mean,mean] 100000 0.01502 0.03764 3895 3764 100000 0.01513 0.03841 3841 3841 100000 0.01511 0.03753 3810 3753 100000 0.01577 0.03766 3873 3766 100000 0.01541 0.0373 3963 3730 100000 0.01586 0.03862 3912 3862 100000 0.01552 0.03785 3870 3785 100000 0.01502 0.03854 3896 3854 100000 0.015 0.03803 3880 3803 100000 0.01515 0.03749 3855 3749 time 2.0455300808 So the rpm version only takes ~17% longer to run this program. I'm surprised that there isn't a larger difference. Perhaps there will be in a different type of program. BTW, the cpu is an Intel e6600 Core 2 Duo overclocked to 3.06 GHz (it will run reliably at 3.24 GHz). I've added these lines to .bashrc: source /opt/intel/cc/9.1.042/bin/iccvars.sh export PYTHONPATH=/usr/local/lib/python2.5/site-packages:/usr/lib/python2.5 export INCLUDE=/opt/intel/mkl/8.1/include:$INCLUDE export LD_LIBRARY_PATH=/usr/local/lib:/opt/intel/mkl/8.1/lib/32:$LD_LIBRARY_PATH I don't understand why the 'site-packages' must be included, but without it, numpy is loaded from /usr/lib/python/site-packages. Why does in look in the subdirectories in one case, but not in the other? Oh, well it works. Thanks much for the detailed explanation. It's greatly appreciated. :) Regards, -rex -- I know so little, but i once knew it fluently... From robert.kern at gmail.com Wed Jan 24 19:40:34 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 24 Jan 2007 18:40:34 -0600 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL (solved!) In-Reply-To: <20070125002533.GE12747@x2.nosyntax.com> References: <20070125002533.GE12747@x2.nosyntax.com> Message-ID: <45B7FC82.1050704@gmail.com> rex wrote: > I've added these lines to .bashrc: > source /opt/intel/cc/9.1.042/bin/iccvars.sh > export PYTHONPATH=/usr/local/lib/python2.5/site-packages:/usr/lib/python2.5 > export INCLUDE=/opt/intel/mkl/8.1/include:$INCLUDE > export LD_LIBRARY_PATH=/usr/local/lib:/opt/intel/mkl/8.1/lib/32:$LD_LIBRARY_PATH > > I don't understand why the 'site-packages' must be included, but without > it, numpy is loaded from /usr/lib/python/site-packages. Why does in look > in the subdirectories in one case, but not in the other? Oh, well it works. Because SuSE did not configure their Python installation to look in /usr/local/lib/python2.5/site-packages/. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From lists.steve at arachnedesign.net Wed Jan 24 21:15:16 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Wed, 24 Jan 2007 21:15:16 -0500 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL (solved!) In-Reply-To: <20070125002533.GE12747@x2.nosyntax.com> References: <20070125002533.GE12747@x2.nosyntax.com> Message-ID: Hi Rex, > Thank you for taking the time to write such a detailed explanation. If > only the documentation were so detailed... Now that you've gone through your odyssey trying to numpy/scipy w/ this particular combo (SuSE/MKL/IntelCC), now would be a great time to whip up wiki page ... you know .. for the documentation ;-) > So the rpm version only takes ~17% longer to run this program. I'm > surprised > that there isn't a larger difference. Perhaps there will be in a > different type of program. BTW, the cpu is an Intel e6600 Core 2 Duo > overclocked to 3.06 GHz (it will run reliably at 3.24 GHz). That's not so bad, though, is it? I'd also be interested in seeing some more benchmarks though .. I wonder if there is a standard benchmarking suite somewhere .. Congrats on completing the gauntlet, -steve From david at ar.media.kyoto-u.ac.jp Thu Jan 25 00:06:16 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 25 Jan 2007 14:06:16 +0900 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL (solved!) In-Reply-To: References: <20070125002533.GE12747@x2.nosyntax.com> Message-ID: <45B83AC8.6060005@ar.media.kyoto-u.ac.jp> Steve Lianoglou wrote: > Hi Rex, > >> Thank you for taking the time to write such a detailed explanation. If >> only the documentation were so detailed... > > Now that you've gone through your odyssey trying to numpy/scipy w/ > this particular combo (SuSE/MKL/IntelCC), now would be a great time > to whip up wiki page ... you know .. for the documentation ;-) > >> So the rpm version only takes ~17% longer to run this program. I'm >> surprised >> that there isn't a larger difference. Perhaps there will be in a >> different type of program. BTW, the cpu is an Intel e6600 Core 2 Duo >> overclocked to 3.06 GHz (it will run reliably at 3.24 GHz). > > That's not so bad, though, is it? I'd also be interested in seeing > some more benchmarks though .. I wonder if there is a standard > benchmarking suite somewhere .. > The code used for this benchmark uses only two few functions: poisson and sum, and I wouldn't be suprised that a lot of code is spent in python (vs in the core C functions), where the intel compiler doesn't make a big difference. Does this code uses the MKL at all ? The MKL gives an optimized fft and BLAS/LAPACK, right ? David From rex at nosyntax.com Thu Jan 25 01:13:32 2007 From: rex at nosyntax.com (rex) Date: Wed, 24 Jan 2007 22:13:32 -0800 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL (solved!) In-Reply-To: References: <20070125002533.GE12747@x2.nosyntax.com> Message-ID: <20070125061331.GF12747@x2.nosyntax.com> Steve Lianoglou [2007-01-24 20:06]: > > Now that you've gone through your odyssey trying to numpy/scipy w/ > this particular combo (SuSE/MKL/IntelCC), now would be a great time > to whip up wiki page ... you know .. for the documentation ;-) Yes, I should do that, but I want to optimize the compiler flags first, and try to get SciPy to build. > > So the rpm version only takes ~17% longer to run this program. I'm > > surprised that there isn't a larger difference. Perhaps there will be in a > > different type of program. BTW, the cpu is an Intel e6600 Core 2 Duo > > overclocked to 3.06 GHz (it will run reliably at 3.24 GHz). > > That's not so bad, though, is it? I'd also be interested in seeing > some more benchmarks though .. I wonder if there is a standard > benchmarking suite somewhere .. I think it should do much better. A few minutes ago I compiled a C math benchmark with : icc -o3 -parallel -xT and it ran 2.8x as fast as it did when compiled with gcc -o3. In fact, it ran at a little over a gigaflop, which is a higher speed than anyone has reported for this benchmark. > Congrats on completing the gauntlet, Thank. It's the 2nd time. I eventually succeed with an earlier version as well, thanks to Travis. -rex From david at ar.media.kyoto-u.ac.jp Thu Jan 25 01:20:45 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 25 Jan 2007 15:20:45 +0900 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL (solved!) In-Reply-To: <20070125061331.GF12747@x2.nosyntax.com> References: <20070125002533.GE12747@x2.nosyntax.com> <20070125061331.GF12747@x2.nosyntax.com> Message-ID: <45B84C3D.8070100@ar.media.kyoto-u.ac.jp> rex wrote: > > I think it should do much better. A few minutes ago I compiled a C > math benchmark with : > > icc -o3 -parallel -xT > > and it ran 2.8x as fast as it did when compiled with gcc -o3. In > fact, it ran at a little over a gigaflop, which is a higher speed than > anyone has reported for this benchmark. Without seeing the benchmark, it would be quite hard to know what's happening. Also, when you are using numpy, you are using python, and for some cases, it can be really easy to slow things down because you are doing something wrong (an example is using non contiguous arrays without knowing it; I got caught often when translating some matlab code to numpy); also the numeric code in numpy *may* be written in a way that icc cannot optimize as well as pure C code. All this is pure speculations, without seeing and running/profiling the actual code David From gnurser at googlemail.com Thu Jan 25 04:34:20 2007 From: gnurser at googlemail.com (George Nurser) Date: Thu, 25 Jan 2007 09:34:20 +0000 Subject: [Numpy-discussion] building NumPy with Intel CC & MKL (solved!) In-Reply-To: <45B84C3D.8070100@ar.media.kyoto-u.ac.jp> References: <20070125002533.GE12747@x2.nosyntax.com> <20070125061331.GF12747@x2.nosyntax.com> <45B84C3D.8070100@ar.media.kyoto-u.ac.jp> Message-ID: <1d1e6ea70701250134s66cc4bfav97277ca43efe88ac@mail.gmail.com> On 25/01/07, David Cournapeau wrote: > rex wrote: > > > > I think it should do much better. A few minutes ago I compiled a C > > math benchmark with : > > > > icc -o3 -parallel -xT > > > > and it ran 2.8x as fast as it did when compiled with gcc -o3. In > > fact, it ran at a little over a gigaflop, which is a higher speed than > > anyone has reported for this benchmark. > Without seeing the benchmark, it would be quite hard to know what's > happening. Also, when you are using numpy, you are using python, and for Perhaps compiling python itself with icc might give a useful speedup. Apparently somebody managed this for python 2.3 in 2003: http://mail.python.org/pipermail/c++-sig/2003-October/005824.html --George Nurser. From charlesr.harris at gmail.com Thu Jan 25 04:45:34 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 25 Jan 2007 02:45:34 -0700 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: References: <45B59786.5070508@gmail.com> <45B5DEC0.7070704@ntc.zcu.cz> <45B65798.8020407@gmail.com> <45B72AB9.6080106@ntc.zcu.cz> Message-ID: On 1/24/07, Charles R Harris wrote: > > > > On 1/24/07, Robert Cimrman wrote: > > > > Robert Kern wrote: > > > Robert Cimrman wrote: > > >> Or you could just call unique1d prior to your call to setmember1d - > > it > > >> was meant to be used that way... you would not loose much speed that > > >> way, IMHO. > > > > > > But that doesn't do what they want. They want a function that gives > > the mask > > > against their original array of the elements that are in the other > > array. The > > > result of > > > > > > setmember1d(unique1d(ar1), unique1d(ar2)) > > > > > For instance In [7]: def countmembers(a1, a2) : ...: a = sort(a2) ...: il = a.searchsorted(a1, side='l') ...: ir = a.searchsorted(a1, side='r') ...: return ir - il ...: In [8]: a2 = random.randint(0,10,(100,)) In [9]: a1 = arange(11) In [11]: a2 = random.randint(0,5,(100,)) In [12]: a1 = arange(10) In [13]: countmembers(a1,a2) Out[13]: array([16, 28, 16, 25, 15, 0, 0, 0, 0, 0]) The subtraction can be replaced by != to get a boolean mask. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Thu Jan 25 06:30:53 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Thu, 25 Jan 2007 12:30:53 +0100 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: References: <45B59786.5070508@gmail.com> <45B5DEC0.7070704@ntc.zcu.cz> <45B65798.8020407@gmail.com> <45B72AB9.6080106@ntc.zcu.cz> Message-ID: <45B894ED.6000700@ntc.zcu.cz> Charles R Harris wrote: > On 1/24/07, Charles R Harris wrote: >> >> >> >> On 1/24/07, Robert Cimrman wrote: >> > >> > Robert Kern wrote: >> > > Robert Cimrman wrote: >> > >> Or you could just call unique1d prior to your call to setmember1d - >> > it >> > >> was meant to be used that way... you would not loose much speed that >> > >> way, IMHO. >> > > >> > > But that doesn't do what they want. They want a function that gives >> > the mask >> > > against their original array of the elements that are in the other >> > array. The >> > > result of >> > > >> > > setmember1d(unique1d(ar1), unique1d(ar2)) >> > > >> >> > > > For instance > > In [7]: def countmembers(a1, a2) : > ...: a = sort(a2) > ...: il = a.searchsorted(a1, side='l') > ...: ir = a.searchsorted(a1, side='r') > ...: return ir - il > ...: > > In [8]: a2 = random.randint(0,10,(100,)) > > In [9]: a1 = arange(11) > > In [11]: a2 = random.randint(0,5,(100,)) > > In [12]: a1 = arange(10) > > In [13]: countmembers(a1,a2) > Out[13]: array([16, 28, 16, 25, 15, 0, 0, 0, 0, 0]) > > > The subtraction can be replaced by != to get a boolean mask. It looks good! Isn't it faster than setmember1d for unique input arrays? I do not like setmember1d much (it is long unlike other functions in arraysetops and looks clumsy to me now and I do not understand it anymore...), so feel free to replace it. BTW. setmember1d gives me the same mask as countmembers for several non-unique inputs I tried... r. From cimrman3 at ntc.zcu.cz Thu Jan 25 06:35:10 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Thu, 25 Jan 2007 12:35:10 +0100 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: <45B894ED.6000700@ntc.zcu.cz> References: <45B59786.5070508@gmail.com> <45B5DEC0.7070704@ntc.zcu.cz> <45B65798.8020407@gmail.com> <45B72AB9.6080106@ntc.zcu.cz> <45B894ED.6000700@ntc.zcu.cz> Message-ID: <45B895EE.3020704@ntc.zcu.cz> Robert Cimrman wrote: > Charles R Harris wrote: >> >> In [7]: def countmembers(a1, a2) : >> ...: a = sort(a2) >> ...: il = a.searchsorted(a1, side='l') >> ...: ir = a.searchsorted(a1, side='r') >> ...: return ir - il >> ...: >> The subtraction can be replaced by != to get a boolean mask. > > It looks good! Isn't it faster than setmember1d for unique input arrays? > I do not like setmember1d much (it is long unlike other functions in > arraysetops and looks clumsy to me now and I do not understand it > anymore...), so feel free to replace it. > > BTW. setmember1d gives me the same mask as countmembers for several > non-unique inputs I tried... But still a function like 'findsorted' returning a bool mask would be handy - one searchsorted-like call could be saved in setmember1d. cheers, r. From rex at nosyntax.com Thu Jan 25 06:50:24 2007 From: rex at nosyntax.com (rex) Date: Thu, 25 Jan 2007 03:50:24 -0800 Subject: [Numpy-discussion] Compiling Python with icc Message-ID: <20070125115024.GJ12747@x2.nosyntax.com> George Nurser [2007-01-25 02:05]: > Perhaps compiling python itself with icc might give a useful speedup. > Apparently somebody managed this for python 2.3 in 2003: > http://mail.python.org/pipermail/c++-sig/2003-October/005824.html Hello George, I saw that post yesterday, and just got around to trying it. It works. ./configure CC=icc --prefix=/usr/local In addition to commenting out #BASECFLAGS= -OPT:Olimit=0 I added -xT -parallel to the OPT= line for my Core 2 Duo CPU. The usual Make, Make install worked, and pybench now runs in 3.15 seconds vs 4.7 seconds with Python2.5 compiled with gcc. That's a 49% speed increase. http://svn.python.org/projects/external/pybench-2.0/ And, if psyco is used, pybench runs in 1.6 seconds for one iteration and then crashes. Psyco + icc results in a ~300% speed increase. Pybench needs to be updated for 1+ gigaflop systems. http://psyco.sourceforge.net/ -rex -- "I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone" --Bjorne Stroustrup (originator of C++ programming language) From svetosch at gmx.net Thu Jan 25 11:15:38 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Thu, 25 Jan 2007 17:15:38 +0100 Subject: [Numpy-discussion] kron bug with zero-length dimensions? Message-ID: <45B8D7AA.3040304@gmx.net> Hi, the kron(a,b) function seems to allow shapes such as (0,x) or (y,0) only for the second argument b, not for the first argument a. (See below for examples.) Maybe it's too harsh to call it a bug because the result is typically not defined mathematically. But then why differentiate between allowing those things for b (very useful), but not for a (thus requiring workarounds.) Thanks for your help, Sven >>> n.kron(n.ones((0,1)), n.ones((1,1))) Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.5/site-packages/numpy/lib/shape_base.py", line 572, in kron result = concatenate(result, axis=axis) ValueError: concatenation of zero-length sequences is impossible >>> n.kron(n.ones((1,0)), n.ones((1,1))) Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.5/site-packages/numpy/lib/shape_base.py", line 572, in kron result = concatenate(result, axis=axis) ValueError: concatenation of zero-length sequences is impossible >>> n.kron(n.ones((1,1)), n.ones((0,1))) array([], shape=(0, 1), dtype=float64) >>> n.kron(n.ones((1,1)), n.ones((1,0))) array([], shape=(1, 0), dtype=float64) >>> n.__version__ '1.0.1' From haase at msg.ucsf.edu Thu Jan 25 11:44:36 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 25 Jan 2007 08:44:36 -0800 Subject: [Numpy-discussion] Build numpy without support of "long double' on OS-X Message-ID: Hi! When I try running my code on panther (10.3) with a numpy that was built on tiger (10.4) it can't load numpy because of missing symbols in numpy/core/umath.so The symbols are _acoshl$LDBL128 _acosl$LDBL128 _asinhl$LDBL128 (see my post from 5 oct 2006: http://permalink.gmane.org/gmane.comp.python.numeric.general/8521 ) I traced the problem to the libmx system library. Since I really don't need "long double" (128 bit) operations - I was wondering if there is a flag to just turn them of? Will SciPy built with this ? (Is there an equivalent flag maybe ?) Thanks, Sebastian Haase From pjssilva at ime.usp.br Thu Jan 25 13:06:23 2007 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu, 25 Jan 2007 16:06:23 -0200 Subject: [Numpy-discussion] A bug in scipy.linalg.lu_factor? Message-ID: <1169748383.5132.4.camel@localhost.localdomain> Hello, I am trying to write some unit tests to my new "Automatic matrix" code and I think I bumped into a bug in scipy.linalg.lu_factor. If you give a matrix to it, it doesn't honor the overwrite_a option: In [1]:import numpy as num In [2]:M = num.mat(num.random.rand(2,2)) In [3]:print M [[ 0.33267781 0.2100424 ] [ 0.61852696 0.32244386]] In [4]:import scipy.linalg as la In [5]:LU, P = la.lu_factor(M, overwrite_a=0); print M [[ 0.33267781 0.63136882] [ 0.61852696 -0.06807478]] In [6]:LU, P = la.lu_factor(M, overwrite_a=0); print M [[ 0.63136882 0.52691517] [-0.06807478 0.6543966 ]] As you can see the matrix is changed by calling the function. can anyone confirm this? I am running numpy 1.0.1 and scipy 0.5.2. Best, Paulo From nwagner at iam.uni-stuttgart.de Thu Jan 25 13:37:01 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 25 Jan 2007 19:37:01 +0100 Subject: [Numpy-discussion] A bug in scipy.linalg.lu_factor? In-Reply-To: <1169748383.5132.4.camel@localhost.localdomain> References: <1169748383.5132.4.camel@localhost.localdomain> Message-ID: On Thu, 25 Jan 2007 16:06:23 -0200 "Paulo J. S. Silva" wrote: > Hello, > > I am trying to write some unit tests to my new >"Automatic matrix" code > and I think I bumped into a bug in >scipy.linalg.lu_factor. If you give a > matrix to it, it doesn't honor the overwrite_a option: > > In [1]:import numpy as num > > In [2]:M = num.mat(num.random.rand(2,2)) > > In [3]:print M > [[ 0.33267781 0.2100424 ] > [ 0.61852696 0.32244386]] > > In [4]:import scipy.linalg as la > > In [5]:LU, P = la.lu_factor(M, overwrite_a=0); print M > [[ 0.33267781 0.63136882] > [ 0.61852696 -0.06807478]] > > In [6]:LU, P = la.lu_factor(M, overwrite_a=0); print M > [[ 0.63136882 0.52691517] > [-0.06807478 0.6543966 ]] > > As you can see the matrix is changed by calling the >function. > > can anyone confirm this? I am running numpy 1.0.1 and >scipy 0.5.2. > > Best, > > Paulo > Hi Paulo, I can confirm this bug. 1.0.2.dev3511 0.5.3.dev2560 Nils > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From nwagner at iam.uni-stuttgart.de Thu Jan 25 13:46:03 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 25 Jan 2007 19:46:03 +0100 Subject: [Numpy-discussion] A bug in scipy.linalg.lu_factor? In-Reply-To: <1169748383.5132.4.camel@localhost.localdomain> References: <1169748383.5132.4.camel@localhost.localdomain> Message-ID: On Thu, 25 Jan 2007 16:06:23 -0200 "Paulo J. S. Silva" wrote: > Hello, > > I am trying to write some unit tests to my new >"Automatic matrix" code > and I think I bumped into a bug in >scipy.linalg.lu_factor. If you give a > matrix to it, it doesn't honor the overwrite_a option: > > In [1]:import numpy as num > > In [2]:M = num.mat( ) > > In [3]:print M > [[ 0.33267781 0.2100424 ] > [ 0.61852696 0.32244386]] > > In [4]:import scipy.linalg as la > > In [5]:LU, P = la.lu_factor(M, overwrite_a=0); print M > [[ 0.33267781 0.63136882] > [ 0.61852696 -0.06807478]] > > In [6]:LU, P = la.lu_factor(M, overwrite_a=0); print M > [[ 0.63136882 0.52691517] > [-0.06807478 0.6543966 ]] > > As you can see the matrix is changed by calling the >function. > > can anyone confirm this? I am running numpy 1.0.1 and >scipy 0.5.2. > > Best, > > Paulo > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion It works if you use M=num.random.rand(2,2) Nils I have filed a ticket. From pjssilva at ime.usp.br Thu Jan 25 13:56:54 2007 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu, 25 Jan 2007 16:56:54 -0200 Subject: [Numpy-discussion] A bug in scipy.linalg.lu_factor? In-Reply-To: References: <1169748383.5132.4.camel@localhost.localdomain> Message-ID: <1169751414.5132.5.camel@localhost.localdomain> Em Qui, 2007-01-25 ?s 19:46 +0100, Nils Wagner escreveu: > > It works if you use > M=num.random.rand(2,2) > > Nils > Yes, it works for arrays but not for matrices. I thought that scipy.linalg functions were supposed to work with matrices. Paulo From curiousjan at gmail.com Thu Jan 25 14:20:47 2007 From: curiousjan at gmail.com (Jan Strube) Date: Thu, 25 Jan 2007 11:20:47 -0800 Subject: [Numpy-discussion] setmember1d memory leak? Message-ID: > > > > For instance > > > > In [7]: def countmembers(a1, a2) : > > ...: a = sort(a2) > > ...: il = a.searchsorted(a1, side='l') > > ...: ir = a.searchsorted(a1, side='r') > > ...: return ir - il > > ...: > > > > In [8]: a2 = random.randint(0,10,(100,)) > > > > In [9]: a1 = arange(11) > > > > In [11]: a2 = random.randint(0,5,(100,)) > > > > In [12]: a1 = arange(10) > > > > In [13]: countmembers(a1,a2) > > Out[13]: array([16, 28, 16, 25, 15, 0, 0, 0, 0, 0]) > > > > > > The subtraction can be replaced by != to get a boolean mask. > > It looks good! Isn't it faster than setmember1d for unique input arrays? > I do not like setmember1d much (it is long unlike other functions in > arraysetops and looks clumsy to me now and I do not understand it > anymore...), so feel free to replace it. > > BTW. setmember1d gives me the same mask as countmembers for several > non-unique inputs I tried... > > r. Try this, then: countmembers(N.array([1,1]), N.array([2])) array([0, 0]) N.setmember1d(N.array([1,1]), N.array([2])) array([ True, False], dtype=bool) setmember1d really needs the first array to be unique. I thought about it quite a bit and tried to understand the code (which is no small feat and I don't claim I have succeeded). As far as I can tell, setmember1d gets it right for the duplicate element with the highest index, all other duplicates are found to be in the second array, independent of whether or not that's actually true. I found it easier to state my problem in terms of unique arrays rather than trying to figure out a general solution, but countmembers sure is nice. Jan ---------- Forwarded message ---------- > From: Robert Cimrman > To: Discussion of Numerical Python > Date: Thu, 25 Jan 2007 12:35:10 +0100 > Subject: Re: [Numpy-discussion] setmember1d memory leak? > Robert Cimrman wrote: > > Charles R Harris wrote: > >> > >> In [7]: def countmembers(a1, a2) : > >> ...: a = sort(a2) > >> ...: il = a.searchsorted(a1, side='l') > >> ...: ir = a.searchsorted(a1, side='r') > >> ...: return ir - il > >> ...: > >> The subtraction can be replaced by != to get a boolean mask. > > > > It looks good! Isn't it faster than setmember1d for unique input arrays? > > I do not like setmember1d much (it is long unlike other functions in > > arraysetops and looks clumsy to me now and I do not understand it > > anymore...), so feel free to replace it. > > > > BTW. setmember1d gives me the same mask as countmembers for several > > non-unique inputs I tried... > > But still a function like 'findsorted' returning a bool mask would be > handy - one searchsorted-like call could be saved in setmember1d. > > cheers, > r. > > > > > ---------- Forwarded message ---------- > From: rex > To: Discussion of Numerical Python > Date: Thu, 25 Jan 2007 03:50:24 -0800 > Subject: [Numpy-discussion] Compiling Python with icc > George Nurser [2007-01-25 02:05]: > > > Perhaps compiling python itself with icc might give a useful speedup. > > Apparently somebody managed this for python 2.3 in 2003: > > http://mail.python.org/pipermail/c++-sig/2003-October/005824.html > > Hello George, > > I saw that post yesterday, and just got around to trying it. It works. > > ./configure CC=icc --prefix=/usr/local > > In addition to commenting out > > #BASECFLAGS= -OPT:Olimit=0 > > I added > > -xT -parallel > > to the > > OPT= > > line for my Core 2 Duo CPU. The usual Make, Make install worked, and > pybench now runs in 3.15 seconds vs 4.7 seconds with Python2.5 compiled > with gcc. That's a 49% speed increase. > > http://svn.python.org/projects/external/pybench-2.0/ > > And, if psyco is used, pybench runs in 1.6 seconds for one iteration and > then crashes. Psyco + icc results in a ~300% speed increase. Pybench > needs to be updated for 1+ gigaflop systems. > > http://psyco.sourceforge.net/ > > -rex > -- > "I have always wished that my computer would be as easy to use as my > telephone. My wish has come true. I no longer know how to use my > telephone" > --Bjorne Stroustrup (originator of C++ programming language) > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Jan 25 15:29:28 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 25 Jan 2007 14:29:28 -0600 Subject: [Numpy-discussion] Build numpy without support of "long double' on OS-X In-Reply-To: References: Message-ID: <45B91328.5030601@gmail.com> Sebastian Haase wrote: > Hi! > When I try running my code on > panther (10.3) with a numpy that was built on tiger (10.4) > it can't load numpy because of missing symbols > in numpy/core/umath.so > The symbols are > _acoshl$LDBL128 > _acosl$LDBL128 > _asinhl$LDBL128 > > (see my post from 5 oct 2006: > http://permalink.gmane.org/gmane.comp.python.numeric.general/8521 ) > > I traced the problem to the libmx system library. > > Since I really don't need "long double" (128 bit) operations - I was > wondering if there is a flag to just turn them of? Generally speaking, you need to build binaries on the lowest-versioned OS X that you intend to run on. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nwagner at iam.uni-stuttgart.de Thu Jan 25 16:40:27 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Thu, 25 Jan 2007 22:40:27 +0100 Subject: [Numpy-discussion] A bug in scipy.linalg.lu_factor? In-Reply-To: <1169751414.5132.5.camel@localhost.localdomain> References: <1169748383.5132.4.camel@localhost.localdomain> <1169751414.5132.5.camel@localhost.localdomain> Message-ID: On Thu, 25 Jan 2007 16:56:54 -0200 "Paulo J. S. Silva" wrote: > Em Qui, 2007-01-25 ?s 19:46 +0100, Nils Wagner escreveu: > >> >> It works if you use >> M=num.random.rand(2,2) >> >> Nils >> > > Yes, it works for arrays but not for matrices. I thought >that > scipy.linalg functions were supposed to work with >matrices. > > Paulo > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion linalg.solve works with matrix input In [1]: import numpy as num In [2]: import scipy.linalg as la In [3]: M = num.mat(num.random.rand(2,2)) In [4]: b = num.random.rand(2) In [5]: x = la.solve(M,b) In [6]: print M*x [[ 0.29508067 0.17152755]] In [7]: b Out[7]: array([ 0.29508067, 0.17152755]) Nils From seb.haase at gmx.net Thu Jan 25 21:25:33 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Thu, 25 Jan 2007 18:25:33 -0800 Subject: [Numpy-discussion] Build numpy without support of "long double' on OS-X In-Reply-To: <45B91328.5030601@gmail.com> References: <45B91328.5030601@gmail.com> Message-ID: On 1/25/07, Robert Kern wrote: > Sebastian Haase wrote: > > Hi! > > When I try running my code on > > panther (10.3) with a numpy that was built on tiger (10.4) > > it can't load numpy because of missing symbols > > in numpy/core/umath.so > > The symbols are > > _acoshl$LDBL128 > > _acosl$LDBL128 > > _asinhl$LDBL128 > > > > (see my post from 5 oct 2006: > > http://permalink.gmane.org/gmane.comp.python.numeric.general/8521 ) > > > > I traced the problem to the libmx system library. > > > > Since I really don't need "long double" (128 bit) operations - I was > > wondering if there is a flag to just turn them of? > > Generally speaking, you need to build binaries on the lowest-versioned OS X that > you intend to run on. > The problem with building on 10.3 is that it generally comes only with gcc 3.3. I remember that some things require gcc4 - right ? I just found this http://developer.apple.com/documentation/DeveloperTools/Conceptual/CppRuntimeEnv/Articles/LibCPPDeployment.html which states: "Support for the 128-bit long double type was not introduced until Mac OS X 10.4." The easiest would be to be able to disable the long double functions. -Sebastian From lists.steve at arachnedesign.net Thu Jan 25 22:16:08 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Thu, 25 Jan 2007 22:16:08 -0500 Subject: [Numpy-discussion] Build numpy without support of "long double' on OS-X In-Reply-To: References: <45B91328.5030601@gmail.com> Message-ID: <96FD8240-79F5-4CF9-AF36-C9A6D62F08E4@arachnedesign.net> >> Generally speaking, you need to build binaries on the lowest- >> versioned OS X that >> you intend to run on. >> > The problem with building on 10.3 is that it generally comes only with > gcc 3.3. I remember that some things require gcc4 - right ? I think that might only bite you if you want to compile universal binaries, though I'm not sure if there are any other problems w/ gcc3.3, I'm pretty sure that's the big one. Of course .. that really shouldn't matter if you're just compiling it for yourself for just that cpu. -steve From ted.horst at earthlink.net Thu Jan 25 22:30:42 2007 From: ted.horst at earthlink.net (Ted Horst) Date: Thu, 25 Jan 2007 21:30:42 -0600 Subject: [Numpy-discussion] random permutation In-Reply-To: <20070113211555.GA2727@zaphod.lagged.za.net> References: <45A6A0FA.2060504@gmail.com> <20070113211555.GA2727@zaphod.lagged.za.net> Message-ID: <0F321064-0324-46A1-BAB2-AA4A0872F5C4@earthlink.net> BTW, This test doesn't work on python 2.3 because sorted does not exist there. Ted On Jan 13, 2007, at 15:15, Stefan van der Walt wrote: > On Sat, Jan 13, 2007 at 10:01:59AM -0800, Keith Goodman wrote: >> On 1/11/07, Robert Kern wrote: >>> Keith Goodman wrote: >>>> Why is the first element of the permutation always the same? Am I >>>> using random.permutation in the right way? >>> >>>>>> M.__version__ >>>> '1.0rc1' >>> >>> This has been fixed in more recent versions. >>> >>> http://projects.scipy.org/scipy/numpy/ticket/374 >> >> I don't see any unit tests for numpy.random. I guess randomness is >> hard to test. > > Every time we fix a bug, we add a corresponding test to make sure > that it > doesn't pop up again. In this case, take a look in > numpy/core/tests/test_regression.py: > > def check_random_shuffle(self, level=rlevel): > """Ticket #374""" > a = N.arange(5).reshape((5,1)) > b = a.copy() > N.random.shuffle(b) > assert_equal(sorted(b),a) > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Thu Jan 25 22:47:06 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 25 Jan 2007 21:47:06 -0600 Subject: [Numpy-discussion] random permutation In-Reply-To: <0F321064-0324-46A1-BAB2-AA4A0872F5C4@earthlink.net> References: <45A6A0FA.2060504@gmail.com> <20070113211555.GA2727@zaphod.lagged.za.net> <0F321064-0324-46A1-BAB2-AA4A0872F5C4@earthlink.net> Message-ID: <45B979BA.2040407@gmail.com> Ted Horst wrote: > BTW, This test doesn't work on python 2.3 because sorted does not > exist there. Fixed, thank you. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Jan 25 23:13:58 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 25 Jan 2007 22:13:58 -0600 Subject: [Numpy-discussion] Build numpy without support of "long double' on OS-X In-Reply-To: References: <45B91328.5030601@gmail.com> Message-ID: <45B98006.5030206@gmail.com> Sebastian Haase wrote: > The problem with building on 10.3 is that it generally comes only with > gcc 3.3. I remember that some things require gcc4 - right ? No, you're right. I thought this might have been available with 10.3.9 (the only version in the 10.3 series that can run Universal binaries anyways), but this is not true. > I just found this > http://developer.apple.com/documentation/DeveloperTools/Conceptual/CppRuntimeEnv/Articles/LibCPPDeployment.html > which states: > "Support for the 128-bit long double type was not introduced until Mac > OS X 10.4." > > The easiest would be to be able to disable the long double functions. I'll accept a patch if you can provide one. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From seb.haase at gmx.net Fri Jan 26 00:59:01 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Thu, 25 Jan 2007 21:59:01 -0800 Subject: [Numpy-discussion] Build numpy without support of "long double' on OS-X In-Reply-To: <96FD8240-79F5-4CF9-AF36-C9A6D62F08E4@arachnedesign.net> References: <45B91328.5030601@gmail.com> <96FD8240-79F5-4CF9-AF36-C9A6D62F08E4@arachnedesign.net> Message-ID: On 1/25/07, Steve Lianoglou wrote: > >> Generally speaking, you need to build binaries on the lowest- > >> versioned OS X that > >> you intend to run on. > >> > > The problem with building on 10.3 is that it generally comes only with > > gcc 3.3. I remember that some things require gcc4 - right ? > > I think that might only bite you if you want to compile universal > binaries, though I'm not sure if there are any other problems w/ > gcc3.3, I'm pretty sure that's the big one. > > Of course .. that really shouldn't matter if you're just compiling it > for yourself for just that cpu. > On the contrary ! I'm trying to provide a precompiled build of numpy together with a couple a handy functions and classes that I made myself, to establish Python as a development platform in multi-dimensional image analysis. I call it Priithon and it is build around wxPython, pyOpenG, SWIG and (of course) numpy [numarray until recently]. I have been working on this project for a couple years as part of my PhD, http://www.ucsf.edu/sedat/Priithon/PriithonHandbook.html And I'm trying to have all of Linux, Windows, OS-X (PPC) and OS-X (intel) supported. Now, I'm hoping to be able to include a version of 10.4's libmx.dyso .... that might be an acceptable workaround. Thanks, From lists.steve at arachnedesign.net Fri Jan 26 01:04:12 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Fri, 26 Jan 2007 01:04:12 -0500 Subject: [Numpy-discussion] Build numpy without support of "long double' on OS-X In-Reply-To: References: <45B91328.5030601@gmail.com> <96FD8240-79F5-4CF9-AF36-C9A6D62F08E4@arachnedesign.net> Message-ID: <455AB0D4-C556-4A83-B8B7-0C0897CA7A76@arachnedesign.net> >> Of course .. that really shouldn't matter if you're just compiling it >> for yourself for just that cpu. >> > On the contrary ! > I'm trying to provide a precompiled build of numpy together with a > couple a handy > functions and classes that I made myself, > to establish Python as a development platform in > multi-dimensional image analysis. > > I call it Priithon and it is build around wxPython, pyOpenG, SWIG and > (of course) numpy [numarray until recently]. > I have been working on this project for a couple years as part of > my PhD, > http://www.ucsf.edu/sedat/Priithon/PriithonHandbook.html > > And I'm trying to have all of > Linux, Windows, OS-X (PPC) and OS-X (intel) > supported. Wow ... cool! Kudos to you for that. It seems like I lead you astray with my advice/suggestion anyhow. Good luck with it. -steve From stefan at sun.ac.za Fri Jan 26 02:03:49 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 26 Jan 2007 09:03:49 +0200 Subject: [Numpy-discussion] inconsistent behaviour of mean, average and median Message-ID: <20070126070349.GC6989@mentat.za.net> Hi, I noticed the following behaviour for empty lists: In [4]: N.median([]) --------------------------------------------------------------------------- exceptions.IndexError Traceback (most recent call last) /home/stefan/ /home/stefan/lib/python2.4/site-packages/numpy/lib/function_base.py in median(m) 1081 return sorted[index] 1082 else: -> 1083 return (sorted[index-1]+sorted[index])/2.0 1084 1085 def trapz(y, x=None, dx=1.0, axis=-1): IndexError: index out of bounds In [5]: N.mean([]) Out[5]: nan In [6]: N.average([]) --------------------------------------------------------------------------- exceptions.ZeroDivisionError Traceback (most recent call last) /home/stefan/ /home/stefan/lib/python2.4/site-packages/numpy/lib/function_base.py in average(a, axis, weights, returned) 294 if not isinstance(d, ndarray): 295 if d == 0.0: --> 296 raise ZeroDivisionError, 'zero denominator in average()' 297 if returned: 298 return n/d, d ZeroDivisionError: zero denominator in average() Which is the ideal response -- NaN or an exception, and if an exception, of which kind? Cheers St?fan From oliphant at ee.byu.edu Fri Jan 26 04:00:48 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 26 Jan 2007 02:00:48 -0700 Subject: [Numpy-discussion] PyGame Port Message-ID: <45B9C340.3030706@ee.byu.edu> I just finished porting PyGame to use NumPy. It seemed to work fine. I ran only a few demos though, and so haven't fleshed out all the details. Please encourage library-writers to use NumPy when possible. -Travis -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pygame_983_numpy_patch.diff URL: From Mi.Zhou at STJUDE.ORG Fri Jan 12 12:29:09 2007 From: Mi.Zhou at STJUDE.ORG (Zhou, Mi) Date: Fri, 12 Jan 2007 11:29:09 -0600 Subject: [Numpy-discussion] failed to build numpy and Numeric in Windows Message-ID: Hi, I am trying to build Numpy-1.01 in windows using "python setup.py build". My C compiler is MSVS2005. The compiler throws an error about umathmodule.c: "__umath_generated.c(72): error C2099: initializer is not a constant." Tried with Numeric-24.2, it gives the same error. The line of code it complains about is: static void * floor_data[] = { (void *)floor, (void *)floor, (void *)"floor", }; Does anybody know how to fix it? Thanks, Mi Zhou From neuromathdan at gmail.com Tue Jan 23 20:29:47 2007 From: neuromathdan at gmail.com (Daniel Smith) Date: Tue, 23 Jan 2007 20:29:47 -0500 Subject: [Numpy-discussion] average() or mean() errors Message-ID: <45B6B68B.6050905@gmail.com> When calling the average() or mean() functions on a small array (3 numbers), I am seeing significant numerical errors (on the order of 1% with data to 8 significant digits). The code I am using is essentially: A = zeros(3) A[i] = X B = average(A) Is there something else I need to call to get better accuracy? Writing my own average function has sufficed, but I am curious if I am just calling the function wrong. From kwgoodman at gmail.com Fri Jan 26 12:27:31 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 26 Jan 2007 09:27:31 -0800 Subject: [Numpy-discussion] average() or mean() errors In-Reply-To: <45B6B68B.6050905@gmail.com> References: <45B6B68B.6050905@gmail.com> Message-ID: On 1/23/07, Daniel Smith wrote: > When calling the average() or mean() functions on a small array (3 > numbers), I am seeing significant numerical errors (on the order of 1% > with data to 8 significant digits). The code I am using is essentially: > > A = zeros(3) > > A[i] = X > > B = average(A) That sounds interesting. Could you show me an example with numbers? From stefan at sun.ac.za Fri Jan 26 12:27:36 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 26 Jan 2007 19:27:36 +0200 Subject: [Numpy-discussion] average() or mean() errors In-Reply-To: <45B6B68B.6050905@gmail.com> References: <45B6B68B.6050905@gmail.com> Message-ID: <20070126172736.GL6989@mentat.za.net> On Tue, Jan 23, 2007 at 08:29:47PM -0500, Daniel Smith wrote: > When calling the average() or mean() functions on a small array (3 > numbers), I am seeing significant numerical errors (on the order of 1% > with data to 8 significant digits). The code I am using is essentially: > > A = zeros(3) > A[i] = X > B = average(A) I'm not sure I understand: In [7]: A = N.zeros(3) In [8]: A[1] = 3. In [9]: N.average(A) Out[9]: 1.0 In [11]: A[0] = 2. In [12]: N.average(A) Out[12]: 1.66666666667 In [13]: (2+3+0)/3. Out[13]: 1.6666666666666667 In [14]: for i in range(1000): ....: A = N.random.rand(3) ....: assert N.average(A) == N.sum(A)/3. Maybe you can give a specific code snippet? Cheers St?fan From charlesr.harris at gmail.com Fri Jan 26 12:36:02 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 26 Jan 2007 10:36:02 -0700 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: <45B895EE.3020704@ntc.zcu.cz> References: <45B59786.5070508@gmail.com> <45B5DEC0.7070704@ntc.zcu.cz> <45B65798.8020407@gmail.com> <45B72AB9.6080106@ntc.zcu.cz> <45B894ED.6000700@ntc.zcu.cz> <45B895EE.3020704@ntc.zcu.cz> Message-ID: On 1/25/07, Robert Cimrman wrote: > > Robert Cimrman wrote: > > Charles R Harris wrote: > >> > >> In [7]: def countmembers(a1, a2) : > >> ...: a = sort(a2) > >> ...: il = a.searchsorted(a1, side='l') > >> ...: ir = a.searchsorted(a1, side='r') > >> ...: return ir - il > >> ...: > >> The subtraction can be replaced by != to get a boolean mask. > > > > It looks good! Isn't it faster than setmember1d for unique input arrays? > > I do not like setmember1d much (it is long unlike other functions in > > arraysetops and looks clumsy to me now and I do not understand it > > anymore...), so feel free to replace it. > > > > BTW. setmember1d gives me the same mask as countmembers for several > > non-unique inputs I tried... > > But still a function like 'findsorted' returning a bool mask would be > handy - one searchsorted-like call could be saved in setmember1d. This would be easy to add. I could put in an option side='mask' that would return 1 if the object is found, 0 otherwise. The type would be integer rather than boolean but I don't see that as a big problem. If I add that, I would like to change the keyword to mode instead of side and that brings up the question of how to change the interface. It is easy to use the same meaning for both mode and side for a while, but it would be nice to issue a deprecation warning for the latter and then remove it after some fixed period of time. This is a policy question and I think the numpy team needs a policy for such things. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From chanley at stsci.edu Fri Jan 26 12:38:50 2007 From: chanley at stsci.edu (Christopher Hanley) Date: Fri, 26 Jan 2007 12:38:50 -0500 Subject: [Numpy-discussion] average() or mean() errors In-Reply-To: <20070126172736.GL6989@mentat.za.net> References: <45B6B68B.6050905@gmail.com> <20070126172736.GL6989@mentat.za.net> Message-ID: <45BA3CAA.9070408@stsci.edu> I filed a similar bug report the other day. I believe that it has to do with the default size of the accumulator variable in the algorithms being used. Please see the following example, Python 2.4.3 (#2, Dec 7 2006, 11:01:45) [GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as n >>> a = n.array([132.00004,132.00006,132.00005],dtype=n.float32) >>> a.mean() 132.00006103515625 >>> a = n.array([132.00004,132.00006,132.00005],dtype=n.float64) >>> a.mean() 132.00004999999999 >>> In the first case, the calculation is done in single precision since that is the type of the input arrays. The second case the calculation is double precision. I think this is the effect that is being seen. The workaround would be to say numpy.average(obj,dtype=numpy.float64). Chris Stefan van der Walt wrote: > On Tue, Jan 23, 2007 at 08:29:47PM -0500, Daniel Smith wrote: > >> When calling the average() or mean() functions on a small array (3 >> numbers), I am seeing significant numerical errors (on the order of 1% >> with data to 8 significant digits). The code I am using is essentially: >> >> A = zeros(3) >> A[i] = X >> B = average(A) >> > > I'm not sure I understand: > > In [7]: A = N.zeros(3) > > In [8]: A[1] = 3. > > In [9]: N.average(A) > Out[9]: 1.0 > > In [11]: A[0] = 2. > > In [12]: N.average(A) > Out[12]: 1.66666666667 > > In [13]: (2+3+0)/3. > Out[13]: 1.6666666666666667 > > In [14]: for i in range(1000): > ....: A = N.random.rand(3) > ....: assert N.average(A) == N.sum(A)/3. > > Maybe you can give a specific code snippet? > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Fri Jan 26 13:27:32 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 26 Jan 2007 12:27:32 -0600 Subject: [Numpy-discussion] setmember1d memory leak? In-Reply-To: References: <45B59786.5070508@gmail.com> <45B5DEC0.7070704@ntc.zcu.cz> <45B65798.8020407@gmail.com> <45B72AB9.6080106@ntc.zcu.cz> <45B894ED.6000700@ntc.zcu.cz> <45B895EE.3020704@ntc.zcu.cz> Message-ID: <45BA4814.8050507@gmail.com> Charles R Harris wrote: > This would be easy to add. I could put in an option side='mask' that > would return 1 if the object is found, 0 otherwise. The type would be > integer rather than boolean but I don't see that as a big problem. If I > add that, I would like to change the keyword to mode instead of side and > that brings up the question of how to change the interface. It is easy > to use the same meaning for both mode and side for a while, but it would > be nice to issue a deprecation warning for the latter and then remove it > after some fixed period of time. This is a policy question and I think > the numpy team needs a policy for such things. If the semantics of the returned value is going to change so greatly, I would prefer that they be separate functions. I think integer 1s and 0s instead of booleans *is* a large problem. Such an array can't be used correctly as a mask in fancy indexing. Inevitably someone will try and wonder why they get an array with only the first and second elements of the source array. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Fri Jan 26 14:25:27 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 26 Jan 2007 12:25:27 -0700 Subject: [Numpy-discussion] Arrays of poly1d objects, is this a bug? Message-ID: Hi all, I'm puzzled by this behavior a colleague ran into: In [38]: p1=N.poly1d([1.]) In [39]: a=N.array([p1],dtype='O') In [40]: a Out[40]: array([], shape=(1, 0), dtype=object) In [42]: print a [] In [43]: N.__version__ Out[43]: '1.0.2.dev3512' He saw it running r3520 as well. This looks like a bug to me, since it seems impossible to make arrays of poly1d objects: In [44]: a=N.array([p1,p1],dtype='O') In [45]: a Out[45]: array([], shape=(2, 0), dtype=object) In [46]: print a [] Any hints? Thanks, f From martin.wiechert at gmx.de Fri Jan 26 14:50:16 2007 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 26 Jan 2007 20:50:16 +0100 Subject: [Numpy-discussion] C-API: deallocating a partially filled object array Message-ID: <200701262050.16983.martin.wiechert@gmx.de> Hi gurus, is it (in C) safe to deallocate an array of type NPY_OBJECT, which carries NULL pointers? TIA, Martin From robert.kern at gmail.com Fri Jan 26 15:03:13 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 26 Jan 2007 14:03:13 -0600 Subject: [Numpy-discussion] C-API: deallocating a partially filled object array In-Reply-To: <200701262050.16983.martin.wiechert@gmx.de> References: <200701262050.16983.martin.wiechert@gmx.de> Message-ID: <45BA5E81.8020906@gmail.com> Martin Wiechert wrote: > Hi gurus, > > is it (in C) safe to deallocate an array of type NPY_OBJECT, which carries > NULL pointers? Possibly, I'm not sure without doing some more code-diving. However, I strongly doubt that many other operations are safe. Depending on how you got such an array and how long it's expected to live, you might want to fix the NULL pointers. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Fri Jan 26 15:07:51 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 26 Jan 2007 22:07:51 +0200 Subject: [Numpy-discussion] Arrays of poly1d objects, is this a bug? In-Reply-To: References: Message-ID: <20070126200751.GO6989@mentat.za.net> On Fri, Jan 26, 2007 at 12:25:27PM -0700, Fernando Perez wrote: > Hi all, > > I'm puzzled by this behavior a colleague ran into: > > In [38]: p1=N.poly1d([1.]) > > In [39]: a=N.array([p1],dtype='O') > > In [40]: a > Out[40]: array([], shape=(1, 0), dtype=object) > > In [42]: print a > [] > > In [43]: N.__version__ > Out[43]: '1.0.2.dev3512' > > He saw it running r3520 as well. > > This looks like a bug to me, since it seems impossible to make arrays > of poly1d objects: > > In [44]: a=N.array([p1,p1],dtype='O') > > In [45]: a > Out[45]: array([], shape=(2, 0), dtype=object) > > In [46]: print a > [] I think the problem might be that an instance of poly1d can be viewed as an array (e.g. N.asarray(p1)). The following works: x = N.empty((3,),object) x[0] = p1 Cheers St?fan From robert.kern at gmail.com Fri Jan 26 15:12:47 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 26 Jan 2007 14:12:47 -0600 Subject: [Numpy-discussion] Build numpy without support of "long double' on OS-X In-Reply-To: References: <45B91328.5030601@gmail.com> Message-ID: <45BA60BF.8030000@gmail.com> Sebastian Haase wrote: > The easiest would be to be able to disable the long double functions. Actually, there are a number of other configuration items that are discovered by compiling small C programs and running them. There are a number of them that might give different answers on 10.3.9 if they could be compiled there. You would have to replace that configuration system with something that could load the configuration from a file or some other source. Since that's a big project, you might be content with modifying the code to hard-code the results for your own builds and documenting the differences from the result of a build from official sources. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From martin.wiechert at gmx.de Fri Jan 26 15:19:43 2007 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 26 Jan 2007 21:19:43 +0100 Subject: [Numpy-discussion] C-API: deallocating a partially filled object array In-Reply-To: <45BA5E81.8020906@gmail.com> References: <200701262050.16983.martin.wiechert@gmx.de> <45BA5E81.8020906@gmail.com> Message-ID: <200701262119.43450.martin.wiechert@gmx.de> On Friday 26 January 2007 21:03, Robert Kern wrote: > Martin Wiechert wrote: > > Hi gurus, > > > > is it (in C) safe to deallocate an array of type NPY_OBJECT, which > > carries NULL pointers? > > Possibly, I'm not sure without doing some more code-diving. However, I > strongly doubt that many other operations are safe. Depending on how you > got such an array and how long it's expected to live, you might want to fix > the NULL pointers. I'm asking because I'm using s.th. like the following idiom. Is the cleanup part clean? --- snip --- a = PyArray_SimpleNew (1, &n, NPY_OBJECT); for (i=0; i < n; i++) { new_obj = some_object_factory (...); if (new_obj == NULL) goto cleanup; ((PyObject **) PyArray_DATA (a)) [i] = new_obj; } return a; cleanup: Py_DECREF (a); return NULL; From seb.haase at gmx.net Fri Jan 26 15:21:15 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Fri, 26 Jan 2007 12:21:15 -0800 Subject: [Numpy-discussion] Build numpy without support of "long double' on OS-X In-Reply-To: <45BA60BF.8030000@gmail.com> References: <45B91328.5030601@gmail.com> <45BA60BF.8030000@gmail.com> Message-ID: On 1/26/07, Robert Kern wrote: > Sebastian Haase wrote: > > > The easiest would be to be able to disable the long double functions. > > Actually, there are a number of other configuration items that are discovered by > compiling small C programs and running them. There are a number of them that > might give different answers on 10.3.9 if they could be compiled there. You > would have to replace that configuration system with something that could load > the configuration from a file or some other source. > > Since that's a big project, you might be content with modifying the code to > hard-code the results for your own builds and documenting the differences from > the result of a build from official sources. > I have to admit that I don't know what "configuration items" are - but I can report that the simple approach of copying libmx.A.dyso from a 10.4 system to a directory which I added to DYLD_LIBRARY_PATH seems to work on the 10.3.9 PPC machine for the first tests I ran so far. (disclaimer: as I said I don't even use "long double" so I did not test this of course) -Sebastian From robert.kern at gmail.com Fri Jan 26 15:40:16 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 26 Jan 2007 14:40:16 -0600 Subject: [Numpy-discussion] Build numpy without support of "long double' on OS-X In-Reply-To: References: <45B91328.5030601@gmail.com> <45BA60BF.8030000@gmail.com> Message-ID: <45BA6730.2030206@gmail.com> Sebastian Haase wrote: > On 1/26/07, Robert Kern wrote: >> Sebastian Haase wrote: >> >>> The easiest would be to be able to disable the long double functions. >> Actually, there are a number of other configuration items that are discovered by >> compiling small C programs and running them. There are a number of them that >> might give different answers on 10.3.9 if they could be compiled there. You >> would have to replace that configuration system with something that could load >> the configuration from a file or some other source. >> >> Since that's a big project, you might be content with modifying the code to >> hard-code the results for your own builds and documenting the differences from >> the result of a build from official sources. > > I have to admit that I don't know what "configuration items" are - It's a general term, not a specific technical one. Look in the function generate_config_h() in numpy/core/setup.py (it's inside the function configuration()). All of the calls to config_cmd.try_run() and config_cmd.check_func() are constructing small C programs, building them, and trying to run them to see if they work. There may be others scattered around, but I don't think so. > but I can report that the simple approach of copying libmx.A.dyso > from a 10.4 system to a directory which I added to DYLD_LIBRARY_PATH > seems to work > on the 10.3.9 PPC machine for the first tests I ran so far. AFAIK, you can't redistribute that file legally. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Fri Jan 26 15:45:50 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 26 Jan 2007 13:45:50 -0700 Subject: [Numpy-discussion] C-API: deallocating a partially filled object array In-Reply-To: <200701262119.43450.martin.wiechert@gmx.de> References: <200701262050.16983.martin.wiechert@gmx.de> <45BA5E81.8020906@gmail.com> <200701262119.43450.martin.wiechert@gmx.de> Message-ID: <45BA687E.1050608@ee.byu.edu> Martin Wiechert wrote: >On Friday 26 January 2007 21:03, Robert Kern wrote: > > >>Martin Wiechert wrote: >> >> >>>Hi gurus, >>> >>>is it (in C) safe to deallocate an array of type NPY_OBJECT, which >>>carries NULL pointers? >>> >>> >>Possibly, I'm not sure without doing some more code-diving. However, I >>strongly doubt that many other operations are safe. Depending on how you >>got such an array and how long it's expected to live, you might want to fix >>the NULL pointers. >> >> > > > Object arrays with NULL pointers should be fine. Any DECREF (should be using) the XDECREF variant internally. So, yes, it should work fine. -Travis From martin.wiechert at gmx.de Fri Jan 26 15:47:37 2007 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Fri, 26 Jan 2007 21:47:37 +0100 Subject: [Numpy-discussion] =?iso-8859-1?q?C-API=3A_deallocating_a_partial?= =?iso-8859-1?q?ly_filled=09object_array?= In-Reply-To: <45BA687E.1050608@ee.byu.edu> References: <200701262050.16983.martin.wiechert@gmx.de> <200701262119.43450.martin.wiechert@gmx.de> <45BA687E.1050608@ee.byu.edu> Message-ID: <200701262147.37846.martin.wiechert@gmx.de> On Friday 26 January 2007 21:45, Travis Oliphant wrote: > Martin Wiechert wrote: > >On Friday 26 January 2007 21:03, Robert Kern wrote: > >>Martin Wiechert wrote: > >>>Hi gurus, > >>> > >>>is it (in C) safe to deallocate an array of type NPY_OBJECT, which > >>>carries NULL pointers? > >> > >>Possibly, I'm not sure without doing some more code-diving. However, I > >>strongly doubt that many other operations are safe. Depending on how you > >>got such an array and how long it's expected to live, you might want to > >> fix the NULL pointers. > > Object arrays with NULL pointers should be fine. Any DECREF (should be > using) the XDECREF variant internally. > > So, yes, it should work fine. > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion Thanks! From oliphant at ee.byu.edu Fri Jan 26 15:50:14 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 26 Jan 2007 13:50:14 -0700 Subject: [Numpy-discussion] Arrays of poly1d objects, is this a bug? In-Reply-To: References: Message-ID: <45BA6986.9070705@ee.byu.edu> Fernando Perez wrote: >Hi all, > >I'm puzzled by this behavior a colleague ran into: > >In [38]: p1=N.poly1d([1.]) > >In [39]: a=N.array([p1],dtype='O') > >In [40]: a >Out[40]: array([], shape=(1, 0), dtype=object) > >In [42]: print a >[] > > > Stefan correctly identified the problem. The dimension-discovery code is getting confused. This might actually be a fixable bug. But, in general getting the N.array() code to correctly guess where the "Object-line" should be drawn is difficult at best. The current code is a re-write of a previous code. But, still it cannot solve the poorly defined problem. So, I would suggest creating empty arrays and populating them... -Travis From fperez.net at gmail.com Fri Jan 26 16:00:34 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 26 Jan 2007 14:00:34 -0700 Subject: [Numpy-discussion] Arrays of poly1d objects, is this a bug? In-Reply-To: <45BA6986.9070705@ee.byu.edu> References: <45BA6986.9070705@ee.byu.edu> Message-ID: On 1/26/07, Travis Oliphant wrote: > Stefan correctly identified the problem. The dimension-discovery code > is getting confused. > This might actually be a fixable bug. But, in general getting the > N.array() code to correctly guess where the "Object-line" should be > drawn is difficult at best. The current code is a re-write of a > previous code. But, still it cannot solve the poorly defined problem. > > So, I would suggest creating empty arrays and populating them... That's fine, no need to try finagling with delicate code and possibly breaking who knows what other corner case. The empty-then-fill solution is just fine, even if it requires making a little helper routine (for the case where the size isn't known in advance). Many thanks to both for the help! Cheers, f From bbands at gmail.com Fri Jan 26 18:09:55 2007 From: bbands at gmail.com (BBands) Date: Fri, 26 Jan 2007 15:09:55 -0800 Subject: [Numpy-discussion] inplace operations Message-ID: <6e8360ad0701261509n242ec31ayf1da182a3e36c8ba@mail.gmail.com> If I have a NumPy array like so: [[1, 12], [2, 13], [3, 14], [4, 15], [5, 16], [6, 15], [7, 14]] How can I do an inplace diff, ending up with this? [[1, 0], [2, 1], [3, 1], [4, 1], [5, 1], [6, -1], [7, -1]] Also, can I covert to natural logs in place? [[1, 2.4849], [2, 2.5649], [3, 2.6391], [4, 2.7081], [5, 2.7726], [6, 2.7081], [7, 2.5649]] Thanks in advance, jab -- John Bollinger, CFA, CMT www.BollingerBands.com If you advance far enough, you arrive at the beginning. From oliphant at ee.byu.edu Fri Jan 26 18:34:07 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 26 Jan 2007 16:34:07 -0700 Subject: [Numpy-discussion] inplace operations In-Reply-To: <6e8360ad0701261509n242ec31ayf1da182a3e36c8ba@mail.gmail.com> References: <6e8360ad0701261509n242ec31ayf1da182a3e36c8ba@mail.gmail.com> Message-ID: <45BA8FEF.30507@ee.byu.edu> BBands wrote: >If I have a NumPy array like so: > >[[1, 12], > [2, 13], > [3, 14], > [4, 15], > [5, 16], > [6, 15], > [7, 14]] > >How can I do an inplace diff, ending up with this? > >[[1, 0], > [2, 1], > [3, 1], > [4, 1], > [5, 1], > [6, -1], > [7, -1]] > > Probably can be done (but it's a bit tricky because you have to be careful not to write over the result before you need it to calculate the difference). Let 'a' be the array you've given Try: b = a[:,1] b[:0:-1] -= b[5::-1] b[0] = 0 >Also, can I covert to natural logs in place? > >[[1, 2.4849], > [2, 2.5649], > [3, 2.6391], > [4, 2.7081], > [5, 2.7726], > [6, 2.7081], > [7, 2.5649]] > > Definitely *can't* be done because you are changing from integers to double-precision. If the original array is double precision than, numpy.log(a[:,1],a[:,1]) should do it (Makes use of the output argument that all ufuncs accept). -Travis From Chris.Barker at noaa.gov Fri Jan 26 18:35:08 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 26 Jan 2007 15:35:08 -0800 Subject: [Numpy-discussion] inplace operations In-Reply-To: <6e8360ad0701261509n242ec31ayf1da182a3e36c8ba@mail.gmail.com> References: <6e8360ad0701261509n242ec31ayf1da182a3e36c8ba@mail.gmail.com> Message-ID: <45BA902C.4060908@noaa.gov> BBands wrote: > If I have a NumPy array like so: > > [[1, 12], > [2, 13], > [3, 14], > [4, 15], > [5, 16], > [6, 15], > [7, 14]] > > How can I do an inplace diff, ending up with this? > > [[1, 0], > [2, 1], > [3, 1], > [4, 1], > [5, 1], > [6, -1], > [7, -1]] >>> import numpy as N >>> a = N.array([[1, 12], ... [2, 13], ... [3, 14], ... [4, 15], ... [5, 16], ... [6, 15], ... [7, 14]]) >>> >>> a array([[ 1, 12], [ 2, 13], [ 3, 14], [ 4, 15], [ 5, 16], [ 6, 15], [ 7, 14]]) >>> a[1:,1] = a[1:,1] - a[:-1,1] >>> a array([[ 1, 12], [ 2, 1], [ 3, 1], [ 4, 1], [ 5, 1], [ 6, -1], [ 7, -1]]) >>> a[0,1] = 0 > Also, can I covert to natural logs in place? >>> a array([[ 1., 12.], [ 2., 13.], [ 3., 14.], [ 4., 15.], [ 5., 16.], [ 6., 15.], [ 7., 14.]]) >>> N.log(a[:,1], a[:,1]) array([ 2.48490665, 2.56494936, 2.63905733, 2.7080502 , 2.77258872, 2.7080502 , 2.63905733]) >>> a array([[ 1. , 2.48490665], [ 2. , 2.56494936], [ 3. , 2.63905733], [ 4. , 2.7080502 ], [ 5. , 2.77258872], [ 6. , 2.7080502 ], [ 7. , 2.63905733]]) All ufuncs (like log() ) take an optional parameter that is the array you want the output to go to. If you pass in the same array, the operation happens in place. By the way, it looks like your first column is an index number. As The array structure keeps track of that for you, you might just as well store just the data in a single (N,) shaped array. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From kwgoodman at gmail.com Sat Jan 27 15:25:18 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 27 Jan 2007 12:25:18 -0800 Subject: [Numpy-discussion] Different results from repeated calculation Message-ID: I get slightly different results when I repeat a calculation. In a long simulation the differences snowball and swamp the effects I am trying to measure. In the attach script there are three tests. In test1, I construct matrices x and y and then repeatedly calculate z = calc(x,y). The result z is the same every time. So this test passes. In test2, I construct matrices x and y each time before calculating z = calc(x,y). Sometimes z is slightly different---of the order of 1e-21 to 1e-18. But the x's test to be equal and so do the y's. This test fails. (It doesn't fail on my friend's windows box; I'm running linux.) test3 is the same as test2 but I calculate z like this: z = calc(100*x ,y) / (100 * 100). This test passes. What is going on? Here is some sample output: >> import repeat >> repeat.all() 4 z different 8.47032947254e-22 5 z different 8.47032947254e-22 7 z different 8.47032947254e-22 9 z different 8.47032947254e-22 10 z different 8.47032947254e-22 16 z different 8.47032947254e-22 24 z different 8.47032947254e-22 25 z different 8.47032947254e-22 26 z different 8.47032947254e-22 27 z different 8.47032947254e-22 30 z different 8.47032947254e-22 32 z different 8.47032947254e-22 34 z different 8.47032947254e-22 35 z different 8.47032947254e-22 36 z different 8.47032947254e-22 39 z different 8.47032947254e-22 40 z different 8.47032947254e-22 41 z different 8.47032947254e-22 45 z different 8.47032947254e-22 46 z different 8.47032947254e-22 50 z different 8.47032947254e-22 52 z different 8.47032947254e-22 53 z different 8.47032947254e-22 55 z different 8.47032947254e-22 56 z different 8.47032947254e-22 58 z different 8.47032947254e-22 66 z different 8.47032947254e-22 67 z different 8.47032947254e-22 71 z different 8.47032947254e-22 73 z different 8.47032947254e-22 74 z different 8.47032947254e-22 83 z different 8.47032947254e-22 86 z different 8.47032947254e-22 87 z different 8.47032947254e-22 88 z different 8.47032947254e-22 89 z different 8.47032947254e-22 90 z different 8.47032947254e-22 92 z different 8.47032947254e-22 test1: 0 differences test2: 38 differences test3: 0 differences Repeated runs tend to give me the same number of differences in test2 for several runs. Then I get a new number of differences which last for severals runs... -------------- next part -------------- A non-text attachment was scrubbed... Name: repeat.py Type: text/x-python Size: 2031 bytes Desc: not available URL: From stefan at sun.ac.za Sat Jan 27 16:06:41 2007 From: stefan at sun.ac.za (Stefan) Date: Sat, 27 Jan 2007 21:06:41 +0000 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: Message-ID: <20070127210641.GA12685@zaphod.lagged.za.net> I can't reproduce this under Linux, r3510. Output is test1: 0 differences test2: 0 differences test3: 0 differences Does anyone else see this happening? Regards St?fan On Sat, Jan 27, 2007 at 12:25:18PM -0800, Keith Goodman wrote: > I get slightly different results when I repeat a calculation. In a > long simulation the differences snowball and swamp the effects I am > trying to measure. > > In the attach script there are three tests. > > In test1, I construct matrices x and y and then repeatedly calculate z > = calc(x,y). The result z is the same every time. So this test passes. > > In test2, I construct matrices x and y each time before calculating z > = calc(x,y). Sometimes z is slightly different---of the order of 1e-21 > to 1e-18. But the x's test to be equal and so do the y's. This test > fails. (It doesn't fail on my friend's windows box; I'm running > linux.) > > test3 is the same as test2 but I calculate z like this: z = calc(100*x > ,y) / (100 * 100). This test passes. > > What is going on? From mpmusu at cc.usu.edu Fri Jan 26 12:17:58 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Fri, 26 Jan 2007 10:17:58 -0700 Subject: [Numpy-discussion] python's random.random() faster than numpy.random.rand() ??? Message-ID: <45BA37C6.4010708@cc.usu.edu> Greetings: I've recently been working with numpy's random number generators and noticed that python's core random number generator is faster than numpy's for the uniform distribution. In other words, for a in range(1000000): b = random.random() #core python code is substantially faster than for a in range(1000000): b = numpy.random.rand() #numpy code For other distributions that I've worked with, numpy's random number generators are much faster than the core python generators. Any thoughts? Just wanted to mention it to you. Or should I be using some other approach to call numpy's uniform generator? -Mark -- Dr. Mark P. Miller Department of Biology 5305 Old Main Hill Utah State University Logan, UT 84322-5305 USA ><><><><><><><><><><><><><><><>< http://www.biology.usu.edu/people/facultyinfo.asp?username=mpmbio http://www.marksgeneticsoftware.net From simon at arrowtheory.com Sat Jan 27 17:00:18 2007 From: simon at arrowtheory.com (Simon Burton) Date: Sat, 27 Jan 2007 14:00:18 -0800 Subject: [Numpy-discussion] python's random.random() faster than numpy.random.rand() ??? In-Reply-To: <45BA37C6.4010708@cc.usu.edu> References: <45BA37C6.4010708@cc.usu.edu> Message-ID: <20070127140018.821d925f.simon@arrowtheory.com> On Fri, 26 Jan 2007 10:17:58 -0700 "Mark P. Miller" wrote: > > for a in range(1000000): > b = numpy.random.rand() #numpy code what about numpy.random.rand(1000000) ? Simon. From svetosch at gmx.net Sat Jan 27 17:04:34 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Sat, 27 Jan 2007 23:04:34 +0100 Subject: [Numpy-discussion] added 1.0.1 release notes to wiki In-Reply-To: <45B53AB0.4070103@mspacek.mm.st> References: <45B53AB0.4070103@mspacek.mm.st> Message-ID: <45BBCC72.5040805@gmx.net> Martin Spacek schrieb: > Just a note that I've copied over the 1.0.1 release notes from SourceForge: > > http://sourceforge.net/project/shownotes.php?group_id=1369&release_id=468153 > > over to the wiki: > > http://scipy.org/ReleaseNotes/NumPy_1.0 Thank you! > > Should 1.0.1 get its own page, as previous 0.9.x releases did? > > I doubt I'm the right person to answer this, but IMHO it belongs to the 1.0.x series. 1.1.x should get a different page maybe. -sven From robert.kern at gmail.com Sat Jan 27 17:08:27 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 27 Jan 2007 16:08:27 -0600 Subject: [Numpy-discussion] python's random.random() faster than numpy.random.rand() ??? In-Reply-To: <45BA37C6.4010708@cc.usu.edu> References: <45BA37C6.4010708@cc.usu.edu> Message-ID: <45BBCD5B.8080101@gmail.com> Mark P. Miller wrote: > Greetings: > > I've recently been working with numpy's random number generators and > noticed that python's core random number generator is faster than > numpy's for the uniform distribution. > > In other words, > > for a in range(1000000): > b = random.random() #core python code > > is substantially faster than > > for a in range(1000000): > b = numpy.random.rand() #numpy code > > > For other distributions that I've worked with, numpy's random number > generators are much faster than the core python generators. Yes, the standard library's non-uniform generators are implemented in Python. I'm not entirely sure where we are losing time in numpy, though. Perhaps the generated Pyrex code for the interface is suboptimal. Or perhaps we ought to compile with greater optimization. > Any thoughts? Just wanted to mention it to you. Or should I be using > some other approach to call numpy's uniform generator? If you need a lot of values, request a lot of values all at once. numpy.random.rand(1000000) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Sat Jan 27 17:11:58 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 27 Jan 2007 15:11:58 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: <20070127210641.GA12685@zaphod.lagged.za.net> References: <20070127210641.GA12685@zaphod.lagged.za.net> Message-ID: On 1/27/07, Stefan wrote: > > I can't reproduce this under Linux, r3510. Output is > > test1: 0 differences > test2: 0 differences > test3: 0 differences > > Does anyone else see this happening? Yes, test1: 0 differences test2: 51 differences test3: 0 differences Oddly, the relative error is always the same: 98 z different 2.0494565872e-16 99 z different 2.0494565872e-16 Which is nearly the same as the double precision 2.2204460492503131e-16, the difference being due to the fact that the precision is defined relative to 1, and the error in the computation are in a number relatively larger (more bits set, but not yet 2). So this looks like an error in the LSB of the floating number. Could be rounding, could be something not reset quite right. I'm thinking possibly hardware at this time, maybe compiler. Linux fedora 2.6.19-1.2895.fc6 #1 SMP Wed Jan 10 19:28:18 EST 2007 i686 athlon i386 GNU/Linux processor : 0 vendor_id : AuthenticAMD cpu family : 15 model : 12 model name : AMD Athlon(tm) 64 Processor 2800+ stepping : 0 cpu MHz : 1808.786 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow up ts fid vid ttp bogomips : 3618.83 Athlon 64 running 32 bit linux. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sat Jan 27 17:17:00 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 28 Jan 2007 00:17:00 +0200 Subject: [Numpy-discussion] python's random.random() faster than numpy.random.rand() ??? In-Reply-To: <45BA37C6.4010708@cc.usu.edu> References: <45BA37C6.4010708@cc.usu.edu> Message-ID: <20070127221700.GB5742@mentat.za.net> Hi Mark On Fri, Jan 26, 2007 at 10:17:58AM -0700, Mark P. Miller wrote: > I've recently been working with numpy's random number generators and > noticed that python's core random number generator is faster than > numpy's for the uniform distribution. > > In other words, > > for a in range(1000000): > b = random.random() #core python code > > is substantially faster than > > for a in range(1000000): > b = numpy.random.rand() #numpy code With numpy, you can get around the for-loop by doing N.random.random(1000000) which is much faster: In [7]: timeit for i in range(10000): random.random() 100 loops, best of 3: 3.92 ms per loop In [8]: timeit N.random.random(10000) 1000 loops, best of 3: 514 ?s per loop Cheers St?fan From stefan at sun.ac.za Sat Jan 27 17:37:56 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 28 Jan 2007 00:37:56 +0200 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> Message-ID: <20070127223756.GC5742@mentat.za.net> On Sat, Jan 27, 2007 at 03:11:58PM -0700, Charles R Harris wrote: > Does anyone else see this happening? > > > Yes, > > test1: 0 differences > test2: 51 differences > test3: 0 differences > > Oddly, the relative error is always the same: > > 98 z different 2.0494565872e-16 > 99 z different 2.0494565872e-16 > > Which is nearly the same as the double precision 2.2204460492503131e-16, the > difference being due to the fact that the precision is defined relative to 1, > and the error in the computation are in a number relatively larger (more bits > set, but not yet 2). > > So this looks like an error in the LSB of the floating number. Could be > rounding, could be something not reset quite right. I'm thinking possibly > hardware at this time, maybe compiler. Interesting! I don't see it on Linux alpha 2.6.17-10-386 #2 Fri Oct 13 18:41:40 UTC 2006 i686 GNU/Linux vendor_id : AuthenticAMD model name : AMD Athlon(tm) XP 2400+ flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow up ts but I do see it on Linux voyager 2.6.17-10-generic #2 SMP Fri Oct 13 18:45:35 UTC 2006 i686 GNU/Linux processor : 0 vendor_id : GenuineIntel model name : Genuine Intel(R) CPU T2300 @ 1.66GHz processor : 1 vendor_id : GenuineIntel model name : Genuine Intel(R) CPU T2300 @ 1.66GHz flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc pni monitor vmx est tm2 xtpr Both machines are running Ubuntu Edgy, exact same software versions. Cheers St?fan From svetosch at gmx.net Sat Jan 27 17:48:42 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Sat, 27 Jan 2007 23:48:42 +0100 Subject: [Numpy-discussion] kron bug with zero-length dimensions? In-Reply-To: <45B8D7AA.3040304@gmx.net> References: <45B8D7AA.3040304@gmx.net> Message-ID: <45BBD6CA.8070108@gmx.net> Sven Schreiber schrieb: > Hi, > > the kron(a,b) function seems to allow shapes such as (0,x) or (y,0) only > for the second argument b, not for the first argument a. (See below for > examples.) > > Maybe it's too harsh to call it a bug because the result is typically > not defined mathematically. But then why differentiate between allowing > those things for b (very useful), but not for a (thus requiring > workarounds.) > > Well in the meantime I learned several things; for my particular problem matlib.repmat is better suited than kron anyway (note that repmat isn't mentioned yet in the numpy book), and "zero-length dimensions" may not be the accurate term in numpy-speak (would zero-length axis be ok?), however I still haven't learned the reason for kron's partial limitation described above... any insights on this? thanks again, sven From charlesr.harris at gmail.com Sat Jan 27 18:00:33 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 27 Jan 2007 16:00:33 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: <20070127223756.GC5742@mentat.za.net> References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> Message-ID: On 1/27/07, Stefan van der Walt wrote: > > On Sat, Jan 27, 2007 at 03:11:58PM -0700, Charles R Harris wrote: > > Does anyone else see this happening? > > > > > > Yes, > > > > test1: 0 differences > > test2: 51 differences > > test3: 0 differences > > > > Oddly, the relative error is always the same: > > > > 98 z different 2.0494565872e-16 > > 99 z different 2.0494565872e-16 > > > > Which is nearly the same as the double precision 2.2204460492503131e-16, > the > > difference being due to the fact that the precision is defined relative > to 1, > > and the error in the computation are in a number relatively larger (more > bits > > set, but not yet 2). > > > > So this looks like an error in the LSB of the floating number. Could be > > rounding, could be something not reset quite right. I'm thinking > possibly > > hardware at this time, maybe compiler. > > Interesting! I don't see it on > > Linux alpha 2.6.17-10-386 #2 Fri Oct 13 18:41:40 UTC 2006 i686 GNU/Linux > vendor_id : AuthenticAMD > model name : AMD Athlon(tm) XP 2400+ > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow up ts > > > but I do see it on > > > Linux voyager 2.6.17-10-generic #2 SMP Fri Oct 13 18:45:35 UTC 2006 i686 > GNU/Linux > processor : 0 > vendor_id : GenuineIntel > model name : Genuine Intel(R) CPU T2300 @ 1.66GHz > processor : 1 > vendor_id : GenuineIntel > model name : Genuine Intel(R) CPU T2300 @ 1.66GHz > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc pni > monitor vmx est tm2 xtpr > > Both machines are running Ubuntu Edgy, exact same software versions. Hmmm, and your problem machine is running smp linux. As is mine; fedora uses smp even on single processor machines these days. I think we could use more data here comparing OSX Linux (single, smp) Window Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sat Jan 27 18:11:59 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 27 Jan 2007 16:11:59 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> Message-ID: On 1/27/07, Charles R Harris wrote: > Hmmm, and your problem machine is running smp linux. As is mine; fedora uses > smp even on single processor machines these days. I think we could use more > data here comparing > > OSX > Linux (single, smp) > Window OK, this is weird. I modified the repeat code a little to ease collecting of results, and all of a sudden the differences went away. If you look at the attached code, here's what happens for me: a) If I have line 77 like this (commented out): #print '-'*75 I get: [...] 94 z different 8.47032947254e-22 95 z different 8.47032947254e-22 96 z different 8.47032947254e-22 98 z different 8.47032947254e-22 99 z different 8.47032947254e-22 Numpy version: 1.0.2.dev3521 test1: 0 differences test2: 75 differences test3: 0 differences b) If I remove the comment char from that line, I get: tlon[~/Desktop]> python repeat.py --------------------------------------------------------------------------- Numpy version: 1.0.2.dev3521 test1: 0 differences test2: 0 differences test3: 0 differences That's it. One comment char removed, and something that's done /after/ the tests are actually executed. That kind of 'I add a printf() call and the bug disappears' is unpleasantly reminiscent of lurking pointer errors in C code... Cheers, f -------------- next part -------------- A non-text attachment was scrubbed... Name: repeat.py Type: text/x-python Size: 2129 bytes Desc: not available URL: From fperez.net at gmail.com Sat Jan 27 18:13:35 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 27 Jan 2007 16:13:35 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> Message-ID: On 1/27/07, Fernando Perez wrote: > On 1/27/07, Charles R Harris wrote: > > > Hmmm, and your problem machine is running smp linux. As is mine; fedora uses > > smp even on single processor machines these days. I think we could use more > > data here comparing > > > > OSX > > Linux (single, smp) Sorry, I forgot to add: tlon[~/Desktop]> uname -a Linux tlon 2.6.17-10-generic #2 SMP Tue Dec 5 22:28:26 UTC 2006 i686 GNU/Linux tlon[~/Desktop]> python -V Python 2.4.4c1 tlon[~/Desktop]> cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 15 model : 35 model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ This box runs up to date Ubuntu Edgy. Cheers, f From stefan at sun.ac.za Sat Jan 27 18:17:04 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 28 Jan 2007 01:17:04 +0200 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> Message-ID: <20070127231704.GF5742@mentat.za.net> On Sat, Jan 27, 2007 at 04:00:33PM -0700, Charles R Harris wrote: > Hmmm, and your problem machine is running smp linux. As is mine; fedora uses > smp even on single processor machines these days. I think we could use more > data here comparing It runs fine on this Ubuntu/Edgy machine, though: Linux genugtig 2.6.17-10-generic #2 SMP Tue Dec 5 21:16:35 UTC 2006 x86_64 GNU/Linux processor : 0 vendor_id : AuthenticAMD model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ processor : 1 vendor_id : AuthenticAMD model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow up pni lahf_lm cmp_legacy Cheers St?fan From charlesr.harris at gmail.com Sat Jan 27 18:19:03 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 27 Jan 2007 16:19:03 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> Message-ID: On 1/27/07, Fernando Perez wrote: > > On 1/27/07, Fernando Perez wrote: > > On 1/27/07, Charles R Harris wrote: > > > > > Hmmm, and your problem machine is running smp linux. As is mine; > fedora uses > > > smp even on single processor machines these days. I think we could use > more > > > data here comparing > > > > > > OSX > > > Linux (single, smp) > > Sorry, I forgot to add: > > tlon[~/Desktop]> uname -a > Linux tlon 2.6.17-10-generic #2 SMP Tue Dec 5 22:28:26 UTC 2006 i686 > GNU/Linux > tlon[~/Desktop]> python -V > Python 2.4.4c1 > tlon[~/Desktop]> cat /proc/cpuinfo > processor : 0 > vendor_id : AuthenticAMD > cpu family : 15 > model : 35 > model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4400+ > > This box runs up to date Ubuntu Edgy. Yep, I am thinking a race condition in the linux smp kernel. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Jan 27 18:23:10 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 27 Jan 2007 17:23:10 -0600 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> Message-ID: <45BBDEDE.7000501@gmail.com> Charles R Harris wrote: > Hmmm, and your problem machine is running smp linux. As is mine; fedora > uses smp even on single processor machines these days. I think we could > use more data here comparing > > OSX [today]$ python repeat.py --------------------------------------------------------------------------- Numpy version: 1.0.2.dev3520 test1: 0 differences test2: 0 differences test3: 0 differences [today]$ uname -a Darwin Sacrilege.local 8.8.2 Darwin Kernel Version 8.8.2: Thu Sep 28 20:43:26 PDT 2006; root:xnu-792.14.14.obj~1/RELEASE_I386 i386 i386 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Sat Jan 27 18:25:47 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 28 Jan 2007 01:25:47 +0200 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> Message-ID: <20070127232547.GG5742@mentat.za.net> On Sat, Jan 27, 2007 at 04:11:59PM -0700, Fernando Perez wrote: > OK, this is weird. I modified the repeat code a little to ease > collecting of results, and all of a sudden the differences went away. > If you look at the attached code, here's what happens for me: > > a) If I have line 77 like this (commented out): > > #print '-'*75 > > I get: > > [...] > 94 z different 8.47032947254e-22 > 95 z different 8.47032947254e-22 > 96 z different 8.47032947254e-22 > 98 z different 8.47032947254e-22 > 99 z different 8.47032947254e-22 > > Numpy version: 1.0.2.dev3521 > > test1: 0 differences > test2: 75 differences > test3: 0 differences > > > > b) If I remove the comment char from that line, I get: > > tlon[~/Desktop]> python repeat.py > > --------------------------------------------------------------------------- > Numpy version: 1.0.2.dev3521 > > test1: 0 differences > test2: 0 differences > test3: 0 differences > > > That's it. One comment char removed, and something that's done > /after/ the tests are actually executed. > > That kind of 'I add a printf() call and the bug disappears' is > unpleasantly reminiscent of lurking pointer errors in C code... I also ran the test under Valgrind, but no errors pop up. Very strange. Cheers St?fan From stefan at sun.ac.za Sat Jan 27 18:29:30 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 28 Jan 2007 01:29:30 +0200 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> Message-ID: <20070127232930.GH5742@mentat.za.net> On Sat, Jan 27, 2007 at 03:11:58PM -0700, Charles R Harris wrote: > So this looks like an error in the LSB of the floating number. Could be > rounding, could be something not reset quite right. I'm thinking possibly > hardware at this time, maybe compiler. > > Linux fedora 2.6.19-1.2895.fc6 #1 SMP Wed Jan 10 19:28:18 EST 2007 i686 athlon > i386 GNU/Linux > > processor : 0 > vendor_id : AuthenticAMD And just for the hell of it, with 4 CPUs :) Linux dirac 2.6.17-10-generic #2 SMP Tue Dec 5 21:16:35 UTC 2006 x86_64 GNU/Linux processor : 0 vendor_id : AuthenticAMD model name : Dual Core AMD Opteron(tm) Processor 275 processor : 1 vendor_id : AuthenticAMD model name : Dual Core AMD Opteron(tm) Processor 275 processor : 2 vendor_id : AuthenticAMD model name : Dual Core AMD Opteron(tm) Processor 275 processor : 3 vendor_id : AuthenticAMD model name : Dual Core AMD Opteron(tm) Processor 275 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow up pni lahf_lm cmp_legacy Works fine. Cheers St?fan From robert.kern at gmail.com Sat Jan 27 18:44:07 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 27 Jan 2007 17:44:07 -0600 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> Message-ID: <45BBE3C7.3000207@gmail.com> Fernando Perez wrote: > That's it. One comment char removed, and something that's done > /after/ the tests are actually executed. > > That kind of 'I add a printf() call and the bug disappears' is > unpleasantly reminiscent of lurking pointer errors in C code... Heh. Fantastic. It might be worthwhile porting this code to C to see what happens. If we can definitively point the finger at the kernel, that would be great (less work for me!). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Sat Jan 27 18:52:35 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 27 Jan 2007 16:52:35 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: <45BBE3C7.3000207@gmail.com> References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> <45BBE3C7.3000207@gmail.com> Message-ID: On 1/27/07, Robert Kern wrote: > Fernando Perez wrote: > > > That's it. One comment char removed, and something that's done > > /after/ the tests are actually executed. > > > > That kind of 'I add a printf() call and the bug disappears' is > > unpleasantly reminiscent of lurking pointer errors in C code... > > Heh. Fantastic. It might be worthwhile porting this code to C to see what > happens. If we can definitively point the finger at the kernel, that would be > great (less work for me!). It's definitely looking like something SMP related: on my laptop, with everything other than the hardware being identical (Linux distro, kernel, numpy build, etc), I can't make it fail no matter how I muck with it. I always get '0 differences'. The desktop is a dual-core AMD Athlon as indicated before, the laptop is an oldie Pentium III. They both run the same SMP-aware Ubuntu i686 kernel, since Ubuntu now ships a unified kernel, though obviously on the laptop the SMP code isn't active. Cheers, f From kwgoodman at gmail.com Sat Jan 27 19:23:34 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 27 Jan 2007 16:23:34 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> <45BBE3C7.3000207@gmail.com> Message-ID: On 1/27/07, Fernando Perez wrote: > It's definitely looking like something SMP related: on my laptop, with > everything other than the hardware being identical (Linux distro, > kernel, numpy build, etc), I can't make it fail no matter how I muck > with it. I always get '0 differences'. > > The desktop is a dual-core AMD Athlon as indicated before, the laptop > is an oldie Pentium III. They both run the same SMP-aware Ubuntu i686 > kernel, since Ubuntu now ships a unified kernel, though obviously on > the laptop the SMP code isn't active. After installing a kernel that is not smp aware, I still have the same problem. --------------------------------------------------------------------------- Numpy version: 1.0.1 test1: 0 differences test2: 55 differences test3: 0 differences $ uname -a Linux kel 2.6.18-3-486 #1 Mon Dec 4 15:59:52 UTC 2006 i686 GNU/Linux $ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Pentium(R) 4 CPU 2.80GHz stepping : 9 cpu MHz : 2793.143 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr bogomips : 5589.65 From wbaxter at gmail.com Sat Jan 27 20:52:24 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 27 Jan 2007 18:52:24 -0700 Subject: [Numpy-discussion] kron bug with zero-length dimensions? In-Reply-To: <45BBD6CA.8070108@gmx.net> References: <45B8D7AA.3040304@gmx.net> <45BBD6CA.8070108@gmx.net> Message-ID: Theres also numpy.tile which is like a repmat generalized to N-dim. --bb On 1/27/07, Sven Schreiber wrote: > Sven Schreiber schrieb: > > Hi, > > > > the kron(a,b) function seems to allow shapes such as (0,x) or (y,0) only > > for the second argument b, not for the first argument a. (See below for > > examples.) > > > > Maybe it's too harsh to call it a bug because the result is typically > > not defined mathematically. But then why differentiate between allowing > > those things for b (very useful), but not for a (thus requiring > > workarounds.) > > > > > > Well in the meantime I learned several things; > > for my particular problem matlib.repmat is better suited than kron > anyway (note that repmat isn't mentioned yet in the numpy book), > > and "zero-length dimensions" may not be the accurate term in numpy-speak > (would zero-length axis be ok?), > > however I still haven't learned the reason for kron's partial limitation > described above... any insights on this? > > thanks again, > sven > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From ctoczycki at yahoo.com Sat Jan 27 22:00:31 2007 From: ctoczycki at yahoo.com (Chris Toczycki) Date: Sat, 27 Jan 2007 21:00:31 -0600 Subject: [Numpy-discussion] Newbee question - looking to use Numpy to solve permutation problem Message-ID: <20070128030037.47F3C39C00A@new.scipy.org> I am looking to find a solution to a problem and believe that Numpy may help. I'd appreciate some advice and/or direction and I'll pursue from there. The problem is that I am looking for a most efficient solution where I have to select 25 solutions where the highest pair-covariance value is the lowest of all possible 25 solution combinations. There are possible 95 solutions to choose from and I have a 95 x 95 matrix of pair-covariance values. Thanks in advance. Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Jan 27 22:03:54 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 27 Jan 2007 21:03:54 -0600 Subject: [Numpy-discussion] ATTN: Developers - Pyrex 0.9.5 is out Message-ID: <45BC129A.7000807@gmail.com> I made a few minor adjustments to make sure that Pyrex 0.9.5 builds mtrand.pyx cleanly. It's a bit pickier than previous versions. However, it also generates C code that compiles without warnings! Consequently, I recommend that all numpy developers should upgrade to Pyrex 0.9.5 and cease using the generate_mtrand_c.py script which I shall remove shortly. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sat Jan 27 22:12:24 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 27 Jan 2007 21:12:24 -0600 Subject: [Numpy-discussion] Newbee question - looking to use Numpy to solve permutation problem In-Reply-To: <20070128030037.47F3C39C00A@new.scipy.org> References: <20070128030037.47F3C39C00A@new.scipy.org> Message-ID: <45BC1498.8040306@gmail.com> Chris Toczycki wrote: > I am looking to find a solution to a problem and believe that Numpy may > help. I'd appreciate some advice and/or direction and I'll pursue from > there. > > The problem is that I am looking for a most efficient solution where I > have to select 25 solutions where the highest pair-covariance value is > the lowest of all possible 25 solution combinations. There are possible > 95 solutions to choose from and I have a 95 x 95 matrix of > pair-covariance values. I'll give you a brief sketch: * Write a function that will extract the upper-triangle (including the diagonal) into a flattened array. a[triu(ones((len(a),len(a)))).astype(bool)] is the one-liner. * Write a function that takes an index from this flattened array to the pair of (i,j) indices that correspond to the same element in the original matrix. * Flatten your covariance matrix with the first function, use argsort() to find the indices into the flattened array in ascending order, extract the first 25, and then use the second function to get the (i,j) pair that you are looking for. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ctoczycki at yahoo.com Sun Jan 28 08:39:41 2007 From: ctoczycki at yahoo.com (Chris Toczycki) Date: Sun, 28 Jan 2007 07:39:41 -0600 Subject: [Numpy-discussion] Newbee question - looking to use Numpy to solve permutation problem In-Reply-To: <45BC1498.8040306@gmail.com> Message-ID: <20070128133943.825EC39C014@new.scipy.org> I believe that I didn't communicate my problem completely. Your suggestion to flatten the matrix should help develop a solution. Now as I progress down the sorted list I need to detect when I have 25 complete solution sets. My goal is to find a set of 25 where the highest collective covariance (within the 25 x 25 matrix) is the lowest. My concern is that this problem requires brut force permutation analysis. Chris -----Original Message----- From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Robert Kern Sent: Saturday, January 27, 2007 9:12 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Newbee question - looking to use Numpy to solve permutation problem Chris Toczycki wrote: > I am looking to find a solution to a problem and believe that Numpy may > help. I'd appreciate some advice and/or direction and I'll pursue from > there. > > The problem is that I am looking for a most efficient solution where I > have to select 25 solutions where the highest pair-covariance value is > the lowest of all possible 25 solution combinations. There are possible > 95 solutions to choose from and I have a 95 x 95 matrix of > pair-covariance values. I'll give you a brief sketch: * Write a function that will extract the upper-triangle (including the diagonal) into a flattened array. a[triu(ones((len(a),len(a)))).astype(bool)] is the one-liner. * Write a function that takes an index from this flattened array to the pair of (i,j) indices that correspond to the same element in the original matrix. * Flatten your covariance matrix with the first function, use argsort() to find the indices into the flattened array in ascending order, extract the first 25, and then use the second function to get the (i,j) pair that you are looking for. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion From kwgoodman at gmail.com Sun Jan 28 10:40:05 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 28 Jan 2007 07:40:05 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> <45BBE3C7.3000207@gmail.com> Message-ID: On 1/27/07, Keith Goodman wrote: > On 1/27/07, Fernando Perez wrote: > > It's definitely looking like something SMP related: on my laptop, with > > everything other than the hardware being identical (Linux distro, > > kernel, numpy build, etc), I can't make it fail no matter how I muck > > with it. I always get '0 differences'. > > > > The desktop is a dual-core AMD Athlon as indicated before, the laptop > > is an oldie Pentium III. They both run the same SMP-aware Ubuntu i686 > > kernel, since Ubuntu now ships a unified kernel, though obviously on > > the laptop the SMP code isn't active. > > After installing a kernel that is not smp aware, I still have the same problem. The problem goes away if I remove atlas (atlas3-sse2 for me). But that just introduces another problem: slowness. So is this something to report to Clint Whaley? Or does it have to do with how numpy uses atlas? From fperez.net at gmail.com Sun Jan 28 12:21:15 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 28 Jan 2007 10:21:15 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Keith Goodman wrote: > The problem goes away if I remove atlas (atlas3-sse2 for me). But that > just introduces another problem: slowness. > > So is this something to report to Clint Whaley? Or does it have to do > with how numpy uses atlas? I was wondering if atlas-sse2 might be the problem, since my desktop is an sse2 machine, but my laptop uses only sse (old PentiumIII). Why don't you try putting in just atlas-sse and seeing what happens? Cheers, f From charlesr.harris at gmail.com Sun Jan 28 12:24:22 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 28 Jan 2007 10:24:22 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> <20070127223756.GC5742@mentat.za.net> <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Keith Goodman wrote: > > On 1/27/07, Keith Goodman wrote: > > On 1/27/07, Fernando Perez wrote: > > > It's definitely looking like something SMP related: on my laptop, with > > > everything other than the hardware being identical (Linux distro, > > > kernel, numpy build, etc), I can't make it fail no matter how I muck > > > with it. I always get '0 differences'. > > > > > > The desktop is a dual-core AMD Athlon as indicated before, the laptop > > > is an oldie Pentium III. They both run the same SMP-aware Ubuntu i686 > > > kernel, since Ubuntu now ships a unified kernel, though obviously on > > > the laptop the SMP code isn't active. > > > > After installing a kernel that is not smp aware, I still have the same > problem. > > The problem goes away if I remove atlas (atlas3-sse2 for me). But that > just introduces another problem: slowness. > > So is this something to report to Clint Whaley? Or does it have to do > with how numpy uses atlas? Interesting, I wonder if ATLAS is resetting the FPU flags and changing the rounding mode? It is just the LSB of the mantissa that looks to be changing. Before reporting the problem it might be good to pin it down a bit more if possible. How come your computation is so sensitive to these small effects? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sun Jan 28 12:39:20 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 28 Jan 2007 10:39:20 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127223756.GC5742@mentat.za.net> <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Charles R Harris wrote: > > The problem goes away if I remove atlas (atlas3-sse2 for me). But that > > just introduces another problem: slowness. > > > > So is this something to report to Clint Whaley? Or does it have to do > > with how numpy uses atlas? > > > Interesting, I wonder if ATLAS is resetting the FPU flags and changing the > rounding mode? It is just the LSB of the mantissa that looks to be > changing. Before reporting the problem it might be good to pin it down a bit > more if possible. Well, the fact that I don't see the problem on a PentiumIII (with atlas-sse) but I see it on my desktop (atlas-sse2) should tell us something. The test code uses double arrays, and SSE2 has double precision support but it's purely 64-bit doubles. SSE is single-precision only, which means that for a double computation, ATLAS isn't used and the Intel FPU does the computation instead. Intel FPUs use 80 bits internally for intermediate operations (even though they only return a normal 64-bit double result), so it's fairly common to see this kind of thing. You can test things by writing a little program in C that does the same operations, and use this little trick: #include // Define DOUBLE to set the FPU in regular double-precision mode, disabling // the internal 80-bit mode which Intel FPUs have. //#define DOUBLE // ... later in the code's main(): // set FPU control word for double precision int cword = 4722; _FPU_SETCW(cword); This can show you if the problem is indeed caused by rounding differences between 64-bit and 80-bit mode. Cheers, f From kwgoodman at gmail.com Sun Jan 28 12:48:31 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 28 Jan 2007 09:48:31 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127223756.GC5742@mentat.za.net> <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Charles R Harris wrote: > How come your computation is so sensitive to these small effects? I don't know. The differences I am seeing are larger than in the test script---but still of the order of eps. Each time step of my simulation selects a maximum value and then uses that for the next time step. Depending on which item is chosen of several items that are very close in value, the simulation can head in a new direction. I guess, as always, I need to randomly perturb my parameters and look at the distribution of test results to see if the effect I am trying to measure is significant. I had no idea it was this sensitive. From kwgoodman at gmail.com Sun Jan 28 13:06:32 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 28 Jan 2007 10:06:32 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127223756.GC5742@mentat.za.net> <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Fernando Perez wrote: > On 1/28/07, Charles R Harris wrote: > > > > The problem goes away if I remove atlas (atlas3-sse2 for me). But that > > > just introduces another problem: slowness. > > > > > > So is this something to report to Clint Whaley? Or does it have to do > > > with how numpy uses atlas? > > > > > > Interesting, I wonder if ATLAS is resetting the FPU flags and changing the > > rounding mode? It is just the LSB of the mantissa that looks to be > > changing. Before reporting the problem it might be good to pin it down a bit > > more if possible. > > Well, the fact that I don't see the problem on a PentiumIII (with > atlas-sse) but I see it on my desktop (atlas-sse2) should tell us > something. The test code uses double arrays, and SSE2 has double > precision support but it's purely 64-bit doubles. SSE is > single-precision only, which means that for a double computation, > ATLAS isn't used and the Intel FPU does the computation instead. > Intel FPUs use 80 bits internally for intermediate operations (even > though they only return a normal 64-bit double result), so it's fairly > common to see this kind of thing. Removing ATLAS SSE2 does fix the problem. But why does test1 pass when ATLAS SSE2 is present? From charlesr.harris at gmail.com Sun Jan 28 13:07:02 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 28 Jan 2007 11:07:02 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127223756.GC5742@mentat.za.net> <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Fernando Perez wrote: > > On 1/28/07, Charles R Harris wrote: > > > > The problem goes away if I remove atlas (atlas3-sse2 for me). But that > > > just introduces another problem: slowness. > > > > > > So is this something to report to Clint Whaley? Or does it have to do > > > with how numpy uses atlas? > > > > > > Interesting, I wonder if ATLAS is resetting the FPU flags and changing > the > > rounding mode? It is just the LSB of the mantissa that looks to be > > changing. Before reporting the problem it might be good to pin it down a > bit > > more if possible. > > Well, the fact that I don't see the problem on a PentiumIII (with > atlas-sse) but I see it on my desktop (atlas-sse2) should tell us > something. The test code uses double arrays, and SSE2 has double > precision support but it's purely 64-bit doubles. SSE is > single-precision only, which means that for a double computation, > ATLAS isn't used and the Intel FPU does the computation instead. > Intel FPUs use 80 bits internally for intermediate operations (even > though they only return a normal 64-bit double result), so it's fairly > common to see this kind of thing. But how come it isn't consistent and seems to depend on timing? That is what makes me think there is a race somewhere in doing something, like setting flags . I googled yesterday for floating point errors and didn't find anything that looked relevant. Maybe I should try again with the combination of atlas and sse2. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Jan 28 13:10:55 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 28 Jan 2007 11:10:55 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Keith Goodman wrote: > > On 1/28/07, Fernando Perez wrote: > > On 1/28/07, Charles R Harris wrote: > > > > > > The problem goes away if I remove atlas (atlas3-sse2 for me). But > that > > > > just introduces another problem: slowness. > > > > > > > > So is this something to report to Clint Whaley? Or does it have to > do > > > > with how numpy uses atlas? > > > > > > > > > Interesting, I wonder if ATLAS is resetting the FPU flags and changing > the > > > rounding mode? It is just the LSB of the mantissa that looks to be > > > changing. Before reporting the problem it might be good to pin it down > a bit > > > more if possible. > > > > Well, the fact that I don't see the problem on a PentiumIII (with > > atlas-sse) but I see it on my desktop (atlas-sse2) should tell us > > something. The test code uses double arrays, and SSE2 has double > > precision support but it's purely 64-bit doubles. SSE is > > single-precision only, which means that for a double computation, > > ATLAS isn't used and the Intel FPU does the computation instead. > > Intel FPUs use 80 bits internally for intermediate operations (even > > though they only return a normal 64-bit double result), so it's fairly > > common to see this kind of thing. > > Removing ATLAS SSE2 does fix the problem. But why does test1 pass when > ATLAS SSE2 is present? It is strange, isn't it. I'm still thinking race condition, but where? I suppose even python could be involved someplace. BTW, your algorithm sounds like a great way to expose small descrepancies. There's a great test for floating point errors lurking in there. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Sun Jan 28 13:11:07 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 28 Jan 2007 10:11:07 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127223756.GC5742@mentat.za.net> <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Fernando Perez wrote: > [snip] The test code uses double arrays, and SSE2 has double > precision support but it's purely 64-bit doubles. SSE is > single-precision only, which means that for a double computation, > ATLAS isn't used and the Intel FPU does the computation instead. So since I use N.float64, ATLAS SSE won't help me? From kwgoodman at gmail.com Sun Jan 28 13:21:09 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 28 Jan 2007 10:21:09 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Charles R Harris wrote: > It is strange, isn't it. I'm still thinking race condition, but where? I > suppose even python could be involved someplace. > > BTW, your algorithm sounds like a great way to expose small descrepancies. > There's a great test for floating point errors lurking in there. Well, whoever can stop the butterfly from flapping its wings will be my hero. From fperez.net at gmail.com Sun Jan 28 13:22:03 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 28 Jan 2007 11:22:03 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Charles R Harris wrote: > But how come it isn't consistent and seems to depend on timing? That is what > makes me think there is a race somewhere in doing something, like setting > flags . I googled yesterday for floating point errors and didn't find > anything that looked relevant. Maybe I should try again with the combination > of atlas and sse2. There could be more than one thing at work here, I honestly don't know. I'm just trying to throw familiar-sounding data bits at the wall, perhaps somebody will see a pattern in the blobs. It worked for Pollock :) Cheers, f From fperez.net at gmail.com Sun Jan 28 13:23:42 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 28 Jan 2007 11:23:42 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Keith Goodman wrote: > On 1/28/07, Fernando Perez wrote: > > [snip] The test code uses double arrays, and SSE2 has double > > precision support but it's purely 64-bit doubles. SSE is > > single-precision only, which means that for a double computation, > > ATLAS isn't used and the Intel FPU does the computation instead. > > So since I use N.float64, ATLAS SSE won't help me? Well, the SSE part won't, but you're still better off with ATLAS than with the default reference BLAS implementation. I think even an ATLAS SSE has special code for double (not using any SSE-type engine) that's faster than the reference BLAS which is pure generic Fortran. Someone who knows the ATLAS internals please correct me if that's not the case. Cheers, f From kwgoodman at gmail.com Sun Jan 28 13:53:34 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 28 Jan 2007 10:53:34 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Fernando Perez wrote: > On 1/28/07, Keith Goodman wrote: > > On 1/28/07, Fernando Perez wrote: > > > [snip] The test code uses double arrays, and SSE2 has double > > > precision support but it's purely 64-bit doubles. SSE is > > > single-precision only, which means that for a double computation, > > > ATLAS isn't used and the Intel FPU does the computation instead. > > > > So since I use N.float64, ATLAS SSE won't help me? > > Well, the SSE part won't, but you're still better off with ATLAS than > with the default reference BLAS implementation. I think even an ATLAS > SSE has special code for double (not using any SSE-type engine) that's > faster than the reference BLAS which is pure generic Fortran. Someone > who knows the ATLAS internals please correct me if that's not the > case. That makes sense. Unfortunately my simulation gives different results with and without ATLAS SSE even though the test script I made doesn't detect the difference. From kwgoodman at gmail.com Sun Jan 28 14:02:19 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 28 Jan 2007 11:02:19 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <45BBE3C7.3000207@gmail.com> Message-ID: On 1/28/07, Keith Goodman wrote: > On 1/28/07, Fernando Perez wrote: > > On 1/28/07, Keith Goodman wrote: > > > On 1/28/07, Fernando Perez wrote: > > > > [snip] The test code uses double arrays, and SSE2 has double > > > > precision support but it's purely 64-bit doubles. SSE is > > > > single-precision only, which means that for a double computation, > > > > ATLAS isn't used and the Intel FPU does the computation instead. > > > > > > So since I use N.float64, ATLAS SSE won't help me? > > > > Well, the SSE part won't, but you're still better off with ATLAS than > > with the default reference BLAS implementation. I think even an ATLAS > > SSE has special code for double (not using any SSE-type engine) that's > > faster than the reference BLAS which is pure generic Fortran. Someone > > who knows the ATLAS internals please correct me if that's not the > > case. > > That makes sense. > > Unfortunately my simulation gives different results with and without > ATLAS SSE even though the test script I made doesn't detect the > difference. ATLAS BASE (no SSE or SSE2) also gives me different simulations results even though it passes the test script. From charlesr.harris at gmail.com Sun Jan 28 19:55:19 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 28 Jan 2007 17:55:19 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: Message-ID: On 1/28/07, Keith Goodman wrote: > > On 1/28/07, Keith Goodman wrote: > > On 1/28/07, Fernando Perez wrote: > > > On 1/28/07, Keith Goodman wrote: > > > > On 1/28/07, Fernando Perez wrote: > > > > > [snip] The test code uses double arrays, and SSE2 has double > > > > > precision support but it's purely 64-bit doubles. SSE is > > > > > single-precision only, which means that for a double computation, > > > > > ATLAS isn't used and the Intel FPU does the computation instead. > > > > > > > > So since I use N.float64, ATLAS SSE won't help me? > > > > > > Well, the SSE part won't, but you're still better off with ATLAS than > > > with the default reference BLAS implementation. I think even an ATLAS > > > SSE has special code for double (not using any SSE-type engine) that's > > > faster than the reference BLAS which is pure generic Fortran. Someone > > > who knows the ATLAS internals please correct me if that's not the > > > case. > > > > That makes sense. > > > > Unfortunately my simulation gives different results with and without > > ATLAS SSE even though the test script I made doesn't detect the > > difference. > > ATLAS BASE (no SSE or SSE2) also gives me different simulations > results even though it passes the test script. Hmmm, I wonder if stuff could be done in different orders. That could affect rounding. Even optimization settings could if someone wasn't careful to use parenthesis to force the order of evaluation. This is all very interesting. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpmusu at cc.usu.edu Mon Jan 29 10:58:35 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Mon, 29 Jan 2007 08:58:35 -0700 Subject: [Numpy-discussion] python's random.random() faster than numpy.random.rand() ??? Message-ID: <45BE19AB.2040808@cc.usu.edu> Thanks to all that responded. Yes...I need to look into pre-generating all of the random numbers that may be needed. It will not be trivial in the case of this particular application I am implementing, but is absolutely worth considering. Robert Kern's response was more along the lines of what I was trying to convey though. Does that particular bit of code require optimization? Thanks, -Mark -- Dr. Mark P. Miller Department of Biology 5305 Old Main Hill Utah State University Logan, UT 84322-5305 USA ><><><><><><><><><><><><><><><>< http://www.biology.usu.edu/people/facultyinfo.asp?username=mpmbio http://www.marksgeneticsoftware.net From Mark.Miller at usu.edu Mon Jan 29 11:34:19 2007 From: Mark.Miller at usu.edu (Mark P Miller) Date: Mon, 29 Jan 2007 16:34:19 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?python=27s_random=2Erandom=28=29_fas?= =?utf-8?b?dGVyIHRoYW4JbnVtcHkucmFuZG9tLnJhbmQoKSA/Pz8=?= References: <45BE19AB.2040808@cc.usu.edu> Message-ID: Sorry...this somehow ended up in the wrong place. Apologies...I'm still getting a feel for how this query system works! From kwgoodman at gmail.com Mon Jan 29 13:08:50 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 29 Jan 2007 10:08:50 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: Message-ID: On 1/27/07, Keith Goodman wrote: > I get slightly different results when I repeat a calculation. In a > long simulation the differences snowball and swamp the effects I am > trying to measure. Here's a unit test for the problem. I am distributing it in hopes of raising awareness of the problem. (What color should I make the Repeatability Wristbands?) I am sure others are having this problem without even knowing it. -------------- next part -------------- A non-text attachment was scrubbed... Name: repeat_test.py Type: text/x-python Size: 2682 bytes Desc: not available URL: From kwgoodman at gmail.com Mon Jan 29 13:20:43 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 29 Jan 2007 10:20:43 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> Message-ID: On 1/27/07, Charles R Harris wrote: > Oddly, the relative error is always the same: > > 98 z different 2.0494565872e-16 > 99 z different 2.0494565872e-16 > > Which is nearly the same as the double precision 2.2204460492503131e-16, the > difference being due to the fact that the precision is defined relative to > 1, and the error in the computation are in a number relatively larger (more > bits set, but not yet 2). Fernando (Ubuntu) and I (Debian) get a different result (8.47032947254e-22) than you (Fedora) do (2.0494565872e-16). From charlesr.harris at gmail.com Mon Jan 29 14:02:50 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 29 Jan 2007 12:02:50 -0700 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> Message-ID: On 1/29/07, Keith Goodman wrote: > > On 1/27/07, Charles R Harris wrote: > > > Oddly, the relative error is always the same: > > > > 98 z different 2.0494565872e-16 > > 99 z different 2.0494565872e-16 > > > > Which is nearly the same as the double precision 2.2204460492503131e-16, > the > > difference being due to the fact that the precision is defined relative > to > > 1, and the error in the computation are in a number relatively larger > (more > > bits set, but not yet 2). > > Fernando (Ubuntu) and I (Debian) get a different result > (8.47032947254e-22) than you (Fedora) do (2.0494565872e-16). That's odd, the LSB bit of the double precision mantissa is only about 2.2e-16, so you can't *get* differences as small as 8.4e-22 without about 70 bit mantissa's. Hmmm, and extended double precision only has 63 bit mantissa's. Are you sure you are computing the error correctly? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Mon Jan 29 14:12:19 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 29 Jan 2007 11:12:19 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> Message-ID: On 1/29/07, Charles R Harris wrote: > That's odd, the LSB bit of the double precision mantissa is only about > 2.2e-16, so you can't *get* differences as small as 8.4e-22 without about > 70 bit mantissa's. Hmmm, and extended double precision only has 63 bit > mantissa's. Are you sure you are computing the error correctly? That is odd. 8.4e-22 is just the output of the test script: abs(z - z0).max(). That abs is from python. From kwgoodman at gmail.com Mon Jan 29 14:33:13 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 29 Jan 2007 11:33:13 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> Message-ID: On 1/29/07, Keith Goodman wrote: > On 1/29/07, Charles R Harris wrote: > > > That's odd, the LSB bit of the double precision mantissa is only about > > 2.2e-16, so you can't *get* differences as small as 8.4e-22 without about > > 70 bit mantissa's. Hmmm, and extended double precision only has 63 bit > > mantissa's. Are you sure you are computing the error correctly? > > That is odd. > > 8.4e-22 is just the output of the test script: abs(z - z0).max(). That > abs is from python. By playing around with x and y I can get all sorts of values for abs(z - z0).max(). I can get down to the e-23 range and to 2.2e-16. I've also seen e-18 and e-22. From kwgoodman at gmail.com Mon Jan 29 14:52:00 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 29 Jan 2007 11:52:00 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> Message-ID: On 1/29/07, Keith Goodman wrote: > On 1/29/07, Keith Goodman wrote: > > On 1/29/07, Charles R Harris wrote: > > > > > That's odd, the LSB bit of the double precision mantissa is only about > > > 2.2e-16, so you can't *get* differences as small as 8.4e-22 without about > > > 70 bit mantissa's. Hmmm, and extended double precision only has 63 bit > > > mantissa's. Are you sure you are computing the error correctly? > > > > That is odd. > > > > 8.4e-22 is just the output of the test script: abs(z - z0).max(). That > > abs is from python. > > By playing around with x and y I can get all sorts of values for abs(z > - z0).max(). I can get down to the e-23 range and to 2.2e-16. I've > also seen e-18 and e-22. Here is a setting for x and y that gives me a difference (using the unit test in this thread) of 4.54747e-13! That is huge---and a serious problem. I am sure I can get bigger. # x data x = M.zeros((3,3)) x[0,0] = 9.0030140479499 x[0,1] = 9.0026474226671 x[0,2] = -9.0011270502873 x[1,0] = 9.0228605377994 x[1,1] = 9.0033715311274 x[1,2] = -9.0082367491299 x[2,0] = 9.0044783987583 x[2,1] = 0.0027488028057 x[2,2] = -9.0036113393360 # y data y = M.zeros((3,1)) y[0,0] =10.00088539878978 y[1,0] = 0.00667193234012 y[2,0] = 0.00032472712345 From kwgoodman at gmail.com Mon Jan 29 15:00:35 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 29 Jan 2007 12:00:35 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> Message-ID: On 1/29/07, Keith Goodman wrote: > On 1/29/07, Keith Goodman wrote: > > On 1/29/07, Keith Goodman wrote: > > > On 1/29/07, Charles R Harris wrote: > > > > > > > That's odd, the LSB bit of the double precision mantissa is only about > > > > 2.2e-16, so you can't *get* differences as small as 8.4e-22 without about > > > > 70 bit mantissa's. Hmmm, and extended double precision only has 63 bit > > > > mantissa's. Are you sure you are computing the error correctly? > > > > > > That is odd. > > > > > > 8.4e-22 is just the output of the test script: abs(z - z0).max(). That > > > abs is from python. > > > > By playing around with x and y I can get all sorts of values for abs(z > > - z0).max(). I can get down to the e-23 range and to 2.2e-16. I've > > also seen e-18 and e-22. > > Here is a setting for x and y that gives me a difference (using the > unit test in this thread) of 4.54747e-13! That is huge---and a serious > problem. I am sure I can get bigger. > > # x data > x = M.zeros((3,3)) > x[0,0] = 9.0030140479499 > x[0,1] = 9.0026474226671 > x[0,2] = -9.0011270502873 > x[1,0] = 9.0228605377994 > x[1,1] = 9.0033715311274 > x[1,2] = -9.0082367491299 > x[2,0] = 9.0044783987583 > x[2,1] = 0.0027488028057 > x[2,2] = -9.0036113393360 > > # y data > y = M.zeros((3,1)) > y[0,0] =10.00088539878978 > y[1,0] = 0.00667193234012 > y[2,0] = 0.00032472712345 OK. I guess I should be looking at the fractional difference instead of the absolute difference. The fractional difference is of order e-16. From robert.kern at gmail.com Mon Jan 29 15:03:08 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Jan 2007 14:03:08 -0600 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: <20070127210641.GA12685@zaphod.lagged.za.net> Message-ID: <45BE52FC.8040101@gmail.com> Keith Goodman wrote: > On 1/29/07, Keith Goodman wrote: >> On 1/29/07, Keith Goodman wrote: >>> On 1/29/07, Charles R Harris wrote: >>> >>>> That's odd, the LSB bit of the double precision mantissa is only about >>>> 2.2e-16, so you can't *get* differences as small as 8.4e-22 without about >>>> 70 bit mantissa's. Hmmm, and extended double precision only has 63 bit >>>> mantissa's. Are you sure you are computing the error correctly? >>> That is odd. >>> >>> 8.4e-22 is just the output of the test script: abs(z - z0).max(). That >>> abs is from python. >> By playing around with x and y I can get all sorts of values for abs(z >> - z0).max(). I can get down to the e-23 range and to 2.2e-16. I've >> also seen e-18 and e-22. > > Here is a setting for x and y that gives me a difference (using the > unit test in this thread) of 4.54747e-13! That is huge---and a serious > problem. I am sure I can get bigger. Check the size of z0. Only the relative difference abs((z-z0)/z0) is going to be about 1e-16. If you adjust the size of z0, the absolute difference will also change in size. In the original unittest that you wrote, z0 is about 1e-6, so 1e-22 corresponds to 1 ULP. With the data you give here, z0 is about 1e3, so 1e-13 also corresponds to 1 ULP. There is no (additional) problem here. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rowen at cesmail.net Mon Jan 29 15:57:54 2007 From: rowen at cesmail.net (Russell E. Owen) Date: Mon, 29 Jan 2007 12:57:54 -0800 Subject: [Numpy-discussion] Different results from repeated calculation References: Message-ID: On a PPC MacOS X box I don't see an error. If I append if __name__ == "__main__": run() to your test code and then run it I get: repeatability #1 ... ok repeatability #2 ... ok repeatability #3 ... ok ---------------------------------------------------------------------- Ran 3 tests in 0.156s OK From kwgoodman at gmail.com Mon Jan 29 16:10:20 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 29 Jan 2007 13:10:20 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: Message-ID: On 1/29/07, Russell E. Owen wrote: > On a PPC MacOS X box I don't see an error. If I append > if __name__ == "__main__": > run() > to your test code and then run it I get: > > repeatability #1 ... ok > repeatability #2 ... ok > repeatability #3 ... ok > > ---------------------------------------------------------------------- > Ran 3 tests in 0.156s > > OK So far no one has duplicated the problem on windows or mac. The problem has only been seen on linux with atlas3-sse2. (I get a similar problem with other versions of atlas.) Are you running atlas on your PPC mac? Perhaps atlas3-altivec? From robert.kern at gmail.com Mon Jan 29 16:47:27 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Jan 2007 15:47:27 -0600 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: Message-ID: <45BE6B6F.5050201@gmail.com> Keith Goodman wrote: > Here's a unit test for the problem. I am distributing it in hopes of > raising awareness of the problem. (What color should I make the > Repeatability Wristbands?) > > I am sure others are having this problem without even knowing it. Another datapoint using atlas3-base on Ubuntu AMD-64. Looking at the source package, I think it sets ISAEXT="sse2" for AMD-64 when building. rkern at rkernx2:~$ python repeat_test.py repeatability #1 ... ok repeatability #2 ... ok repeatability #3 ... ok ---------------------------------------------------------------------- Ran 3 tests in 0.043s OK rkern at rkernx2:~$ uname -a Linux rkernx2 2.6.17-10-generic #2 SMP Fri Oct 13 15:34:39 UTC 2006 x86_64 GNU/Linux rkern at rkernx2:~$ cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 15 model : 43 model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ stepping : 1 cpu MHz : 2211.346 cache size : 512 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow up pni lahf_lm cmp_legacy bogomips : 4426.03 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp processor : 1 vendor_id : AuthenticAMD cpu family : 15 model : 43 model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ stepping : 1 cpu MHz : 2211.346 cache size : 512 KB physical id : 0 siblings : 2 core id : 1 cpu cores : 2 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow up pni lahf_lm cmp_legacy bogomips : 4423.03 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jbednar at inf.ed.ac.uk Mon Jan 29 16:57:33 2007 From: jbednar at inf.ed.ac.uk (James A. Bednar) Date: Mon, 29 Jan 2007 21:57:33 +0000 Subject: [Numpy-discussion] Pickling ufuncs? Message-ID: <17854.28109.309939.896522@lodestar.inf.ed.ac.uk> Hi, Does anyone know whether it is possible to pickle and unpickle numpy ufuncs? I can't find anything about that on scipy.org or the mailing list archives. I have several important pieces of code that accept a numpy ufunc as an argument and later apply it to some data, while keeping a copy of the ufunc in an attribute. So far I have been able to pickle this code only by doing some very ugly hacks and workarounds. The ufuncs from Numeric 24 and earlier did not even deepcopy, which caused us lots of other problems, but deepcopying them works now in numpy 1.0.1. However, they still don't seem to pickle with the regular picklers. Is this deliberately disabled for some reason, or is there some workaround? Here's an example that illustrates the problem: I have a class Test defined in test.py: class Test(object): def __init__(self,a): self.a = a If I try this code at the commandline: import numpy from test import Test t=Test(numpy.multiply) import pickle s=pickle.dumps(t) I get "TypeError: can't pickle ufunc objects" Also, I can't reproduce it in a small example yet, but pickling completes without errors in our larger program that also stores ufuncs, yet I then get a different error on unpickling: File "/disk/home/lodestar1/jbednar/topographica/lib/python2.4/pickle.py", line 872, in load dispatch[key](self) File "/disk/home/lodestar1/jbednar/topographica/lib/python2.4/pickle.py", line 1097, in load_newobj obj = cls.__new__(cls, *args) TypeError: object.__new__(numpy.ufunc) is not safe, use numpy.ufunc.__new__() Any ideas? Thanks, Jim _______________________________________________________________________________ Unix> cat test.py class Test(object): def __init__(self,a): self.a = a Unix> bin/python Python 2.4.4 (#5, Nov 9 2006, 22:58:03) [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> from test import Test >>> t=Test(numpy.multiply) >>> import pickle >>> s=pickle.dumps(t) Traceback (most recent call last): File "", line 1, in ? File "/home/jb/lib/python2.4/pickle.py", line 1386, in dumps Pickler(file, protocol, bin).dump(obj) File "/home/jb/lib/python2.4/pickle.py", line 231, in dump self.save(obj) File "/home/jb/lib/python2.4/pickle.py", line 338, in save self.save_reduce(obj=obj, *rv) File "/home/jb/lib/python2.4/pickle.py", line 433, in save_reduce save(state) File "/home/jb/lib/python2.4/pickle.py", line 293, in save f(self, obj) # Call unbound method with explicit self File "/home/jb/lib/python2.4/pickle.py", line 663, in save_dict self._batch_setitems(obj.iteritems()) File "/home/jb/lib/python2.4/pickle.py", line 677, in _batch_setitems save(v) File "/home/jb/lib/python2.4/pickle.py", line 313, in save rv = reduce(self.proto) File "/home/jb/lib/python2.4/copy_reg.py", line 69, in _reduce_ex raise TypeError, "can't pickle %s objects" % base.__name__ TypeError: can't pickle ufunc objects >>> From kwgoodman at gmail.com Mon Jan 29 17:00:20 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 29 Jan 2007 14:00:20 -0800 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: <45BE6B6F.5050201@gmail.com> References: <45BE6B6F.5050201@gmail.com> Message-ID: On 1/29/07, Robert Kern wrote: > Keith Goodman wrote: > > Here's a unit test for the problem. I am distributing it in hopes of > > raising awareness of the problem. (What color should I make the > > Repeatability Wristbands?) > > > > I am sure others are having this problem without even knowing it. > > Another datapoint using atlas3-base on Ubuntu AMD-64. Looking at the source > package, I think it sets ISAEXT="sse2" for AMD-64 when building. > > > rkern at rkernx2:~$ python repeat_test.py > repeatability #1 ... ok > repeatability #2 ... ok > repeatability #3 ... ok > > ---------------------------------------------------------------------- > Ran 3 tests in 0.043s > > OK I ported the test to octave which like numpy uses Atlas. On my machine (debian etch atlas3-sse2) I get the problem in numpy but not in octave. Plus test1 always passes. So it is only when you reload x and y that the problem occurs. If you load x and y once (test1) and repeat the calculation, there is no problem. Do these two results point, however weakly, away from atlas? From sransom at nrao.edu Mon Jan 29 17:50:25 2007 From: sransom at nrao.edu (Scott Ransom) Date: Mon, 29 Jan 2007 17:50:25 -0500 Subject: [Numpy-discussion] Different results from repeated calculation In-Reply-To: References: Message-ID: <200701291750.25581.sransom@nrao.edu> On a 64-bit Intel Core2 Duo running Debian unstable with atlas3 (there is no specific atlas3-sse2 for AMD64 Debian, although I think that it is included) everything checks out fine: eiger:~$ uname -a Linux eiger 2.6.18-3-amd64 #1 SMP Sun Dec 10 19:57:44 CET 2006 x86_64 GNU/Linux eiger:~$ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Core(TM)2 CPU 6700 @ 2.66GHz stepping : 6 cpu MHz : 2660.009 cache size : 4096 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm bogomips : 5324.65 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: (same for 2nd CPU) Scott On Monday 29 January 2007 16:10, Keith Goodman wrote: > On 1/29/07, Russell E. Owen wrote: > > On a PPC MacOS X box I don't see an error. If I append > > if __name__ == "__main__": > > run() > > to your test code and then run it I get: > > > > repeatability #1 ... ok > > repeatability #2 ... ok > > repeatability #3 ... ok > > > > ------------------------------------------------------------------- > >--- Ran 3 tests in 0.156s > > > > OK > > So far no one has duplicated the problem on windows or mac. The > problem has only been seen on linux with atlas3-sse2. (I get a > similar problem with other versions of atlas.) > > Are you running atlas on your PPC mac? Perhaps atlas3-altivec? > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Scott M. Ransom Address: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: sransom at nrao.edu Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From oliphant at ee.byu.edu Mon Jan 29 17:55:06 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 29 Jan 2007 15:55:06 -0700 Subject: [Numpy-discussion] Pickling ufuncs? In-Reply-To: <17854.28109.309939.896522@lodestar.inf.ed.ac.uk> References: <17854.28109.309939.896522@lodestar.inf.ed.ac.uk> Message-ID: <45BE7B4A.906@ee.byu.edu> James A. Bednar wrote: >Hi, > >Does anyone know whether it is possible to pickle and unpickle numpy >ufuncs? > Not directly. Ufuncs are a built-in type and do not have the required __reduce__ method needed to be pickleable. It could be added, but hasn't been. > I can't find anything about that on scipy.org or the mailing >list archives. I have several important pieces of code that accept a >numpy ufunc as an argument and later apply it to some data, while >keeping a copy of the ufunc in an attribute. So far I have been able >to pickle this code only by doing some very ugly hacks and >workarounds. > > Is storing the name considered an ugly hack? >The ufuncs from Numeric 24 and earlier did not even deepcopy, which >caused us lots of other problems, but deepcopying them works now in >numpy 1.0.1. However, they still don't seem to pickle with the regular >picklers. Is this deliberately disabled for some reason, or is there >some workaround? > > No, nothing has been "disabled." The feature was never added. >Here's an example that illustrates the problem: > > >I have a class Test defined in test.py: > >class Test(object): > def __init__(self,a): > self.a = a > > Why don't you store the name of the ufunc instead: def __init__(self, a): self._a = a.__name__ Then, whenever you are going to use the ufunc you do import numpy func = getattr(numpy,self._a) Then, pickle should work. Alternatively you can write your own __reduce__ function for the Test class. Direct pickling of ufuncs is not a trivial issue as ufuncs contain code-pointers at their core which cannot really be pickled. It's not a simple problem to over-come in general. We could store the name of the ufunc, but for user-defined ufuncs these might come from a different package and which package the ufunc lives under is not stored as part of the ufunc. -Travis From oliphant at ee.byu.edu Mon Jan 29 18:08:07 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 29 Jan 2007 16:08:07 -0700 Subject: [Numpy-discussion] Cutting 1.0.2 release Message-ID: <45BE7E57.4000105@ee.byu.edu> I think it's time for the 1.0.2 release of NumPy. What outstanding issues need to be resolved before we do it? Hopefully, we can do it by the end of the week. -Travis From robert.kern at gmail.com Mon Jan 29 19:52:36 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Jan 2007 18:52:36 -0600 Subject: [Numpy-discussion] Pickling ufuncs? In-Reply-To: <45BE7B4A.906@ee.byu.edu> References: <17854.28109.309939.896522@lodestar.inf.ed.ac.uk> <45BE7B4A.906@ee.byu.edu> Message-ID: <45BE96D4.3040108@gmail.com> Travis Oliphant wrote: > Why don't you store the name of the ufunc instead: > > def __init__(self, a): > self._a = a.__name__ > > Then, whenever you are going to use the ufunc you do > > import numpy > func = getattr(numpy,self._a) > > Then, pickle should work. Or you can register pickler/unpickler functions for ufuncs: In [24]: import copy_reg In [25]: import numpy In [26]: def ufunc_pickler(ufunc): return ufunc.__name__ ....: In [28]: def ufunc_unpickler(name): import numpy return getattr(numpy, name) ....: In [31]: copy_reg.pickle(numpy.ufunc, ufunc_pickler, ufunc_unpickler) In [32]: import cPickle In [33]: cPickle.dumps(numpy.add) Out[33]: 'cnumpy.core.umath\nadd\np1\n.' In [34]: cPickle.loads(cPickle.dumps(numpy.add)) Out[34]: Note that this is a hack. It won't work for the ufuncs in scipy.special, for example. We should look into including __module__ information in ufuncs. This is how regular functions are pickled. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jbednar at inf.ed.ac.uk Mon Jan 29 20:29:22 2007 From: jbednar at inf.ed.ac.uk (James A. Bednar) Date: Tue, 30 Jan 2007 01:29:22 +0000 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 4, Issue 84 In-Reply-To: References: Message-ID: <17854.40818.675942.569683@lodestar.inf.ed.ac.uk> | Date: Mon, 29 Jan 2007 15:55:06 -0700 | From: Travis Oliphant | | James A. Bednar wrote: | | >Hi, | > | >Does anyone know whether it is possible to pickle and unpickle numpy | >ufuncs? | > | Not directly. Ufuncs are a built-in type and do not have the required | __reduce__ method needed to be pickleable. It could be added, but | hasn't been. Thanks for the quick reply! Please consider our request that it be added. Meanwhile, we'll do a workaround. We just wanted to make sure that we were not overlooking some obvious way we were supposed to be pickling them, such as through some special pickler hidden somewhere. :-) | > I can't find anything about that on scipy.org or the mailing | >list archives. I have several important pieces of code that accept a | >numpy ufunc as an argument and later apply it to some data, while | >keeping a copy of the ufunc in an attribute. So far I have been able | >to pickle this code only by doing some very ugly hacks and | >workarounds. | | Is storing the name considered an ugly hack? Yes -- see below. | >Here's an example that illustrates the problem: | > | > | >I have a class Test defined in test.py: | > | >class Test(object): | > def __init__(self,a): | > self.a = a | | Why don't you store the name of the ufunc instead: | | def __init__(self, a): | self._a = a.__name__ | | Then, whenever you are going to use the ufunc you do | | import numpy | func = getattr(numpy,self._a) | | Then, pickle should work. For numpy ufuncs alone this would work, but in general we want to support any ufunc-like operator, including ones that our API users define, not just those defined in the numpy namespace. Specifically, we have numerous classes that have a user-selectable "operator" attribute that is *typically* set to a numpy ufunc like add or max. But a user can pass anything he or she wants as an operator, such as return_first(), where that's defined as: class return_first(object): @staticmethod def reduce(x): return x[0] Anything that supports reduce() (as ufuncs do) will work for our purposes, but if we put in specific hacks for numpy, then our users will be limited to only the numpy ufuncs. Of course, we can add more hacks to limit the hacks to being applied only for ufuncs, but we've done something vaguely like this and it's pretty complicated. We just want to be able to treat ufuncs like any other function object in Python... | Alternatively you can write your own __reduce__ function for the Test | class. We can do that too, but we actually store such operators in numerous locations in different classes, so that too turns out to be complicated. | Direct pickling of ufuncs is not a trivial issue as ufuncs contain | code-pointers at their core which cannot really be pickled. It's | not a simple problem to over-come in general. We could store the | name of the ufunc, but for user-defined ufuncs these might come | from a different package and which package the ufunc lives under is | not stored as part of the ufunc. >From a technical perspective it's worked fine for us to simply pickle the name and then reconstruct the ufunc from the name when unpickling. It's just the practical side of how to do that relatively cleanly and without having to explain it all to our API users that's been a problem. And Robert's suggestion below looks like it will help clean that up. | From: Robert Kern | Subject: Re: [Numpy-discussion] Pickling ufuncs? | | Travis Oliphant wrote: | > Why don't you store the name of the ufunc instead: | | Or you can register pickler/unpickler functions for ufuncs: | | | In [24]: import copy_reg | | In [25]: import numpy | | In [26]: def ufunc_pickler(ufunc): | return ufunc.__name__ | ....: | | In [28]: def ufunc_unpickler(name): | import numpy | return getattr(numpy, name) | ....: | | In [31]: copy_reg.pickle(numpy.ufunc, ufunc_pickler, ufunc_unpickler) | | In [32]: import cPickle | | In [33]: cPickle.dumps(numpy.add) | Out[33]: 'cnumpy.core.umath\nadd\np1\n.' | | In [34]: cPickle.loads(cPickle.dumps(numpy.add)) | Out[34]: | | | Note that this is a hack. It won't work for the ufuncs in scipy.special, for | example. That sounds like a significantly cleaner hack than the ones we've been doing, so I think we'll try that out. Thanks! (I think we actually tried this years ago with Numeric, but from the above example it sounds worth trying again in numpy.) | We should look into including __module__ information in ufuncs. This is how | regular functions are pickled. I vote for that! Thanks, Jim From lists.steve at arachnedesign.net Mon Jan 29 23:29:24 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Mon, 29 Jan 2007 23:29:24 -0500 Subject: [Numpy-discussion] Selected and altering a submatrix Message-ID: <6639DBA0-8720-478B-AEF3-49F2D336670E@arachnedesign.net> Hi, I was just curious what the "correct" (fast) way to select and alter a submatrix. For example, say I have a 10x10 array and only want to add some number to the elements in the submatrix that consists of the [0,1,2] th rows, and [4,5,6]th colums. You can imagine that those rows/cols select a square in the top- middle of the 10x10 which I want to alter. The only way I can get this to work is if I iterate over the indices in one of the dimensions (say the rows) and use the column indices to slice out the relevant elements to add to .. is there a NumPy-thonic way to do this: === import numpy as N mat = N.zeros((10,10)) rows = [0,1,2] cols = [4,5,6] for row in rows: mat[row,cols] += 1 ==== I found something on the lists from a few years back that was in reference to numeric or numarray that suggested doing some gymnastics with take/put, but it still seemed as if there was no way to slice out this view of a matrix w/o making a copy. Thanks, -steve From seb.haase at gmx.net Mon Jan 29 23:44:40 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Mon, 29 Jan 2007 20:44:40 -0800 Subject: [Numpy-discussion] Selected and altering a submatrix In-Reply-To: <6639DBA0-8720-478B-AEF3-49F2D336670E@arachnedesign.net> References: <6639DBA0-8720-478B-AEF3-49F2D336670E@arachnedesign.net> Message-ID: How about mat[0:3, 4:7] += 1 -Sebastian On 1/29/07, Steve Lianoglou wrote: > Hi, > > I was just curious what the "correct" (fast) way to select and alter > a submatrix. > > For example, say I have a 10x10 array and only want to add some > number to the elements in the submatrix that consists of the [0,1,2] > th rows, and [4,5,6]th colums. > > You can imagine that those rows/cols select a square in the top- > middle of the 10x10 which I want to alter. > > The only way I can get this to work is if I iterate over the indices > in one of the dimensions (say the rows) and use the column indices to > slice out the relevant elements to add to .. is there a NumPy-thonic > way to do this: > > === > import numpy as N > mat = N.zeros((10,10)) > rows = [0,1,2] > cols = [4,5,6] > > for row in rows: > mat[row,cols] += 1 > > ==== > > I found something on the lists from a few years back that was in > reference to numeric or numarray that suggested doing some gymnastics > with take/put, but it still seemed as if there was no way to slice > out this view of a matrix w/o making a copy. > > Thanks, > -steve > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From haase at msg.ucsf.edu Mon Jan 29 23:45:20 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 29 Jan 2007 20:45:20 -0800 Subject: [Numpy-discussion] Selected and altering a submatrix In-Reply-To: References: <6639DBA0-8720-478B-AEF3-49F2D336670E@arachnedesign.net> Message-ID: How about mat[0:3, 4:7] += 1 -Sebastian On 1/29/07, Steve Lianoglou wrote: > > Hi, > > > > I was just curious what the "correct" (fast) way to select and alter > > a submatrix. > > > > For example, say I have a 10x10 array and only want to add some > > number to the elements in the submatrix that consists of the [0,1,2] > > th rows, and [4,5,6]th colums. > > > > You can imagine that those rows/cols select a square in the top- > > middle of the 10x10 which I want to alter. > > > > The only way I can get this to work is if I iterate over the indices > > in one of the dimensions (say the rows) and use the column indices to > > slice out the relevant elements to add to .. is there a NumPy-thonic > > way to do this: > > > > === > > import numpy as N > > mat = N.zeros((10,10)) > > rows = [0,1,2] > > cols = [4,5,6] > > > > for row in rows: > > mat[row,cols] += 1 > > > > ==== > > > > I found something on the lists from a few years back that was in > > reference to numeric or numarray that suggested doing some gymnastics > > with take/put, but it still seemed as if there was no way to slice > > out this view of a matrix w/o making a copy. > > > > Thanks, > > -steve > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > From kwgoodman at gmail.com Mon Jan 29 23:46:32 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 29 Jan 2007 20:46:32 -0800 Subject: [Numpy-discussion] Selected and altering a submatrix In-Reply-To: <6639DBA0-8720-478B-AEF3-49F2D336670E@arachnedesign.net> References: <6639DBA0-8720-478B-AEF3-49F2D336670E@arachnedesign.net> Message-ID: On 1/29/07, Steve Lianoglou wrote: > For example, say I have a 10x10 array and only want to add some > number to the elements in the submatrix that consists of the [0,1,2] > th rows, and [4,5,6]th colums. Here's one way to do it: i,j = N.mgrid[0:3,4:7] mat[i,j] += 1 From robert.kern at gmail.com Tue Jan 30 00:09:59 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Jan 2007 23:09:59 -0600 Subject: [Numpy-discussion] Selected and altering a submatrix In-Reply-To: <6639DBA0-8720-478B-AEF3-49F2D336670E@arachnedesign.net> References: <6639DBA0-8720-478B-AEF3-49F2D336670E@arachnedesign.net> Message-ID: <45BED327.2070601@gmail.com> Steve Lianoglou wrote: > === > import numpy as N > mat = N.zeros((10,10)) > rows = [0,1,2] > cols = [4,5,6] > > for row in rows: > mat[row,cols] += 1 > > ==== > > I found something on the lists from a few years back that was in > reference to numeric or numarray that suggested doing some gymnastics > with take/put, but it still seemed as if there was no way to slice > out this view of a matrix w/o making a copy. Actually, it's pretty easy these days to handle the general case (the other posts have sufficiently covered the case where your rows and columns are representable by slices). Just make sure that the index arrays row and cols are the right shape. Since you want mat[rows, cols] to be an array of shape (len(rows), len(cols)), each index should be of that shape *or* they need to broadcast to that shape. Thus, you could either have this: rows = [[0, 0, 0], [1, 1, 1], [2, 2, 2]] cols = [[4, 5, 6], [4, 5, 6], [4, 5, 6]] or you could have this: rows = [[0], [1], [2]] cols = [4, 5, 6] Here is a slightly more complicated example: In [25]: from numpy import * In [26]: A = arange(6*6).reshape((6,6)) In [27]: rows = array([0, 2, 3])[:,newaxis] In [28]: cols = array([5, 4, 1]) In [29]: A[rows, cols] Out[29]: array([[ 5, 4, 1], [17, 16, 13], [23, 22, 19]]) In [30]: A Out[30]: array([[ 0, 1, 2, 3, 4, 5], [ 6, 7, 8, 9, 10, 11], [12, 13, 14, 15, 16, 17], [18, 19, 20, 21, 22, 23], [24, 25, 26, 27, 28, 29], [30, 31, 32, 33, 34, 35]]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gruben at bigpond.net.au Tue Jan 30 06:20:27 2007 From: gruben at bigpond.net.au (Gary Ruben) Date: Tue, 30 Jan 2007 22:20:27 +1100 Subject: [Numpy-discussion] Cutting 1.0.2 release In-Reply-To: <45BE7E57.4000105@ee.byu.edu> References: <45BE7E57.4000105@ee.byu.edu> Message-ID: <45BF29FB.8070305@bigpond.net.au> One question, which may be an issue: Should ones, zeros and empty be generating arrays of floats by default now? Gary R. Travis Oliphant wrote: > I think it's time for the 1.0.2 release of NumPy. > > What outstanding issues need to be resolved before we do it? > > Hopefully, we can do it by the end of the week. > > -Travis From lists.steve at arachnedesign.net Tue Jan 30 09:44:14 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Tue, 30 Jan 2007 09:44:14 -0500 Subject: [Numpy-discussion] Selected and altering a submatrix In-Reply-To: <45BED327.2070601@gmail.com> References: <6639DBA0-8720-478B-AEF3-49F2D336670E@arachnedesign.net> <45BED327.2070601@gmail.com> Message-ID: <39A1E94A-B1D8-4A26-8FC2-BF38988F45D5@arachnedesign.net> Hi all, Thank you for your replies ... my question was initially unclear in that I knew that using normal slicing would do the trick, but it was rather a list of indices that couldn't be represented as a slice that was bugging me. Luckily, Robert's solution solves this problem for the general case. Thanks for the help, -steve On Jan 30, 2007, at 12:09 AM, Robert Kern wrote: > Steve Lianoglou wrote: >> === >> import numpy as N >> mat = N.zeros((10,10)) >> rows = [0,1,2] >> cols = [4,5,6] >> >> for row in rows: >> mat[row,cols] += 1 >> >> ==== >> >> I found something on the lists from a few years back that was in >> reference to numeric or numarray that suggested doing some gymnastics >> with take/put, but it still seemed as if there was no way to slice >> out this view of a matrix w/o making a copy. > > Actually, it's pretty easy these days to handle the general case > (the other > posts have sufficiently covered the case where your rows and > columns are > representable by slices). Just make sure that the index arrays row > and cols are > the right shape. Since you want mat[rows, cols] to be an array of > shape > (len(rows), len(cols)), each index should be of that shape *or* > they need to > broadcast to that shape. Thus, you could either have this: > > rows = [[0, 0, 0], > [1, 1, 1], > [2, 2, 2]] > cols = [[4, 5, 6], > [4, 5, 6], > [4, 5, 6]] > > or you could have this: > > rows = [[0], [1], [2]] > cols = [4, 5, 6] > > Here is a slightly more complicated example: > > > In [25]: from numpy import * > > In [26]: A = arange(6*6).reshape((6,6)) > > In [27]: rows = array([0, 2, 3])[:,newaxis] > > In [28]: cols = array([5, 4, 1]) > > In [29]: A[rows, cols] > Out[29]: > array([[ 5, 4, 1], > [17, 16, 13], > [23, 22, 19]]) > > In [30]: A > Out[30]: > array([[ 0, 1, 2, 3, 4, 5], > [ 6, 7, 8, 9, 10, 11], > [12, 13, 14, 15, 16, 17], > [18, 19, 20, 21, 22, 23], > [24, 25, 26, 27, 28, 29], > [30, 31, 32, 33, 34, 35]]) > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a > harmless enigma > that is made terrible by our own mad attempt to interpret it as > though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From aisaac at american.edu Tue Jan 30 11:04:44 2007 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 30 Jan 2007 11:04:44 -0500 Subject: [Numpy-discussion] Cutting 1.0.2 release In-Reply-To: <45BF29FB.8070305@bigpond.net.au> References: <45BE7E57.4000105@ee.byu.edu><45BF29FB.8070305@bigpond.net.au> Message-ID: On Tue, 30 Jan 2007, Gary Ruben apparently wrote: > One question, which may be an issue: Should ones, zeros > and empty be generating arrays of floats by default now? >>> import numpy as n >>> z = n.zeros((2,2)) >>> z.dtype dtype('float64') >>> o = n.ones((2,2)) >>> o.dtype dtype('float64') >>> e = n.empty((2,2)) >>> e.dtype dtype('float64') >>> n.__version__ '1.0' hth, Alan Isaac From Chris.Barker at noaa.gov Tue Jan 30 12:38:52 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 30 Jan 2007 09:38:52 -0800 Subject: [Numpy-discussion] Cutting 1.0.2 release In-Reply-To: <45BF29FB.8070305@bigpond.net.au> References: <45BE7E57.4000105@ee.byu.edu> <45BF29FB.8070305@bigpond.net.au> Message-ID: <45BF82AC.9070706@noaa.gov> Gary Ruben wrote: > One question, which may be an issue: Should ones, zeros and empty be > generating arrays of floats by default now? >>> import numpy as N >>> N.ones((3,)).dtype dtype('float64') >>> N.zeros((3,)).dtype dtype('float64') >>> N.__version__ '1.0.1' -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Tue Jan 30 16:51:16 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 30 Jan 2007 14:51:16 -0700 Subject: [Numpy-discussion] Pickling ufuncs? In-Reply-To: <45BE96D4.3040108@gmail.com> References: <17854.28109.309939.896522@lodestar.inf.ed.ac.uk> <45BE7B4A.906@ee.byu.edu> <45BE96D4.3040108@gmail.com> Message-ID: <45BFBDD4.4040604@ee.byu.edu> Robert Kern wrote: > Note that this is a hack. It won't work for the ufuncs in scipy.special, for > example. > > We should look into including __module__ information in ufuncs. This is how > regular functions are pickled. > > This sounds like a good idea. Would it be possible to add a "dict" object to the end of the ufunc structure without bumping up the Version number of the C-API? I think it would be possible, but I'm interested in other points of view. If we added the dict object then we could add the __module__ attribute to all ufuncs and use that in pickling. Then, we could add an API function that took a module name and found all ufuncs under that name and set their __module__ attribute correctly. Thoughts? -Travis From oliphant at ee.byu.edu Tue Jan 30 19:09:05 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 30 Jan 2007 17:09:05 -0700 Subject: [Numpy-discussion] Patch for RPy to use NumPy uploaded to www.scipy.org Message-ID: <45BFDE21.6020902@ee.byu.edu> I've added a patch to the Porting to NumPy page so that RPy uses NumPy instead of Numeric. It would be very helpful for unification if these package authors would accept these patches. NumPy is far enough along in development that I don't think there is any reason for new releases of packages to continue to support Numeric. For numarray-support in packages the transition may take a little longer. I know that some think that supporting multiple array packages may be a good idea. I think that is a recipe for long-term headaches. We are moving to get the array interface into Python. Until that happens (hopefully by Python 2.6), third-party code should write to NumPy. You can use the attribute-based array interface if you are willing to alter that at some point in the future when the array interface is moved into Python. -Travis From oliphant at ee.byu.edu Tue Jan 30 19:51:07 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 30 Jan 2007 17:51:07 -0700 Subject: [Numpy-discussion] Requests for NumPy Ports? Message-ID: <45BFE7FB.6080000@ee.byu.edu> I'm trying to help out the conversion to NumPy by offering patches to various third-party packages that have used Numeric in the past. Does anybody here have requests for which packages they would like to see converted to use NumPy? Ultimately, we are at the mercy of the package-owners to complete the conversion by applying the patches. But, I think it helps when there are patches already available for trying out. The packages I've worked on so far are here: http://www.scipy.org/Porting_to_NumPy Thanks, -Travis From tchur at optushome.com.au Tue Jan 30 19:56:07 2007 From: tchur at optushome.com.au (Tim Churches) Date: Wed, 31 Jan 2007 11:56:07 +1100 Subject: [Numpy-discussion] Patch for RPy to use NumPy uploaded to www.scipy.org In-Reply-To: <45BFDE21.6020902@ee.byu.edu> References: <45BFDE21.6020902@ee.byu.edu> Message-ID: <45BFE927.1050204@optushome.com.au> Travis Oliphant wrote: > I've added a patch to the Porting to NumPy page so that RPy uses NumPy > instead of Numeric. It would be very helpful for unification if these > package authors would accept these patches. NumPy is far enough along > in development that I don't think there is any reason for new releases > of packages to continue to support Numeric. > > For numarray-support in packages the transition may take a little > longer. I know that some think that supporting multiple array packages > may be a good idea. I think that is a recipe for long-term headaches. > We are moving to get the array interface into Python. Until that > happens (hopefully by Python 2.6), third-party code should write to > NumPy. You can use the attribute-based array interface if you are > willing to alter that at some point in the future when the array > interface is moved into Python. Yes, except that we have quite a lot of code in fielded applications which is written with Numeric, and which also uses RPy. We currently have no funds to do the Numeric to NumPy port in our code. If RPy dropped support for Numeric, we would be forced to use increasing old versions of RPy, which means we would be locked into increasing old versions of R. I would argue that it is premature for RPy to drop support for Numeric. Maybe by the time Python 2.6 is out and NumPy is part of Python. Tim C From robert.kern at gmail.com Tue Jan 30 20:04:25 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 30 Jan 2007 19:04:25 -0600 Subject: [Numpy-discussion] Patch for RPy to use NumPy uploaded to www.scipy.org In-Reply-To: <45BFE927.1050204@optushome.com.au> References: <45BFDE21.6020902@ee.byu.edu> <45BFE927.1050204@optushome.com.au> Message-ID: <45BFEB19.80404@gmail.com> Tim Churches wrote: > Travis Oliphant wrote: >> I've added a patch to the Porting to NumPy page so that RPy uses NumPy >> instead of Numeric. It would be very helpful for unification if these >> package authors would accept these patches. NumPy is far enough along >> in development that I don't think there is any reason for new releases >> of packages to continue to support Numeric. >> >> For numarray-support in packages the transition may take a little >> longer. I know that some think that supporting multiple array packages >> may be a good idea. I think that is a recipe for long-term headaches. >> We are moving to get the array interface into Python. Until that >> happens (hopefully by Python 2.6), third-party code should write to >> NumPy. You can use the attribute-based array interface if you are >> willing to alter that at some point in the future when the array >> interface is moved into Python. > > Yes, except that we have quite a lot of code in fielded applications > which is written with Numeric, and which also uses RPy. We currently > have no funds to do the Numeric to NumPy port in our code. If RPy > dropped support for Numeric, we would be forced to use increasing old > versions of RPy, which means we would be locked into increasing old > versions of R. I don't see why. RPy will use numpy internally to communicate with R. Your code will use Numeric internally to do whatever you do. As of Numeric 24.2, each can consume the other's arrays seamlessly. You might want to intersperse some Numeric.asarray() and numpy.asarray() calls at the boundaries, but that should be it. > I would argue that it is premature for RPy to drop support for Numeric. > Maybe by the time Python 2.6 is out and NumPy is part of Python. numpy itself will never be part of Python. We're aiming for a standardized array interface and an API for manipulating that interface to become part of Python. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tchur at optushome.com.au Tue Jan 30 20:43:02 2007 From: tchur at optushome.com.au (Tim Churches) Date: Wed, 31 Jan 2007 12:43:02 +1100 Subject: [Numpy-discussion] Patch for RPy to use NumPy uploaded to www.scipy.org In-Reply-To: <45BFEB19.80404@gmail.com> References: <45BFDE21.6020902@ee.byu.edu> <45BFE927.1050204@optushome.com.au> <45BFEB19.80404@gmail.com> Message-ID: <45BFF426.9020403@optushome.com.au> Robert Kern wrote: > I don't see why. RPy will use numpy internally to communicate with R. Your code > will use Numeric internally to do whatever you do. As of Numeric 24.2, each can > consume the other's arrays seamlessly. OK, if that is the case, then it might be OK for us if RPy dropped Numeric support. > You might want to intersperse some > Numeric.asarray() and numpy.asarray() calls at the boundaries, but that should > be it. Ah, so not quite seemless consumption of each other's arrays? We'll need to test. >> I would argue that it is premature for RPy to drop support for Numeric. >> Maybe by the time Python 2.6 is out and NumPy is part of Python. > > numpy itself will never be part of Python. We're aiming for a standardized array > interface and an API for manipulating that interface to become part of Python. Yes, OK. Tim C From robert.kern at gmail.com Tue Jan 30 21:38:30 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 30 Jan 2007 20:38:30 -0600 Subject: [Numpy-discussion] Patch for RPy to use NumPy uploaded to www.scipy.org In-Reply-To: <45BFF426.9020403@optushome.com.au> References: <45BFDE21.6020902@ee.byu.edu> <45BFE927.1050204@optushome.com.au> <45BFEB19.80404@gmail.com> <45BFF426.9020403@optushome.com.au> Message-ID: <45C00126.7020604@gmail.com> Tim Churches wrote: > Robert Kern wrote: >> I don't see why. RPy will use numpy internally to communicate with R. Your code >> will use Numeric internally to do whatever you do. As of Numeric 24.2, each can >> consume the other's arrays seamlessly. > > OK, if that is the case, then it might be OK for us if RPy dropped > Numeric support. > >> You might want to intersperse some >> Numeric.asarray() and numpy.asarray() calls at the boundaries, but that should >> be it. > > Ah, so not quite seemless consumption of each other's arrays? No, that's precisely the consumption that I was referring to. Generally, calling a function in Numeric on a numpy array will work just fine and vice versa. The only tricky bit would be code that doesn't "consume" arrays so much as perform operations on them. For example, def f(x): return N.sum(x, axis=0) def g(x): return x.typecode() f() would work fine with either N=Numeric or N=numpy on Numeric x arrays or numpy x arrays or indeed lists of numbers or anything else that could be converted to an array with asarray(). g() would not work with numpy arrays since numpy arrays don't have an .typecode() method. Of course, neither do lists of numbers and all of the other various things that a user might pass in hoping that the author programmed robustly and called asarray(). > We'll need > to test. Of course. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From haase at msg.ucsf.edu Wed Jan 31 01:11:18 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 30 Jan 2007 22:11:18 -0800 Subject: [Numpy-discussion] memmap close() and flush() Message-ID: Hi! Do numpy memmap have a way of explicitly flushing data to disk and/or closing the memmap. In numarray these were methods called memmappedArr.flush() and memmappedArr.close() Thanks, Sebastian From oliphant.travis at ieee.org Wed Jan 31 02:27:04 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed, 31 Jan 2007 00:27:04 -0700 Subject: [Numpy-discussion] memmap close() and flush() In-Reply-To: References: Message-ID: <45C044C8.1010505@ieee.org> Sebastian Haase wrote: > Hi! > Do numpy memmap have a way of explicitly > flushing data to disk > and/or > closing the memmap. > There is a sync method that performs the flush. To close the memmap, delete it. More detail: The memmap sub-class has a _mmap attribute that is the Python memory-map object. -Travis From pgmdevlist at gmail.com Wed Jan 31 07:15:05 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 31 Jan 2007 07:15:05 -0500 Subject: [Numpy-discussion] Subclassing ndarray in Pyrex Message-ID: <200701310715.06586.pgmdevlist@gmail.com> I'm still trying to subclass ndarray in Pyrex, without much success so far. I started to follow Francesc's suggestion (http://projects.scipy.org/pipermail/numpy-discussion/2007-January/025644.html), but that doesn't fit what I need: the myarray class Francesc introduced is just an object, not a subclass of ndarray... The closest I came to something vaguely running is the following (called subnd.pyx later on): #------------------- from definitions cimport import_array, \ malloc, free, npy_intp, \ PyArray_GETITEM, PyArray_EMPTY, \ ndarray, dtype # NumPy must be initialized import_array() import numpy as _N cdef class Sub_2(ndarray): cdef readonly object _info cdef ndarray __mask def __new__(self, object shape=None, object dtype=None, object buffer=None, object order=None, object infodict={}): self.__mask = _N.zeros(shape, _N.bool) self._info = infodict return property info: def __get__(self): return self._info def __set__(self, value): self._info = value property _mask: def __get__(self): return self.__mask def subarray(obj, info={}): _obj = _N.asarray(obj) print "_obj is: %s" % _obj _internal = Sub_2(shape=_obj.shape, dtype=_obj.dtype) _internal.flat[:] = _obj.flat[:] _internal.info = info return _internal #---------------------- However, I get a segfault when I try to play with it in Python: >>> import numpy as N >>> import subnd >>> L = N.array([1,2,3] >>> x = subarray(L) >>> x Sub_2([1, 2, 3]) >>> x+1 Sub_2([2, 3, 4]) >>> N.log(x) crash So obviously I'm missing something, but what ? Some kind of closure ? Moreover, I'm a bit disappointed with this method: if I understand correctly the subtleties of subclassing in pyrex, the ndarray.__new__ is called before Sub_2.__new__, with the same arguments. As ndarray.__new__ doesn't take optional arguments, the subclass can't either. That's a bit limiting, I have to call the constructor function all the time... I'm completely at loss, here. Any advice/help/suggestions would be more than welcome. From gruben at bigpond.net.au Wed Jan 31 08:48:05 2007 From: gruben at bigpond.net.au (Gary Ruben) Date: Thu, 01 Feb 2007 00:48:05 +1100 Subject: [Numpy-discussion] Cutting 1.0.2 release In-Reply-To: References: <45BE7E57.4000105@ee.byu.edu><45BF29FB.8070305@bigpond.net.au> Message-ID: <45C09E15.40202@bigpond.net.au> Thanks Alan & Chris, My apologies. I was trying ones(), zeros() and empty() in ipython 0.7.2 with the -pylab option and getting the wrong functions. On my system, ipython -pylab imports the namespace of the oldnumeric wrapper versions of ones(), zeros() and empty() and I had assumed these were the core numpy versions. Maybe this has been changed in ipython 0.7.3. Gary R. Alan G Isaac wrote: > On Tue, 30 Jan 2007, Gary Ruben apparently wrote: >> One question, which may be an issue: Should ones, zeros >> and empty be generating arrays of floats by default now? > >>>> import numpy as n >>>> z = n.zeros((2,2)) >>>> z.dtype > dtype('float64') >>>> o = n.ones((2,2)) >>>> o.dtype > dtype('float64') >>>> e = n.empty((2,2)) >>>> e.dtype > dtype('float64') >>>> n.__version__ > '1.0' > > hth, > Alan Isaac From lists.steve at arachnedesign.net Wed Jan 31 08:53:18 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Wed, 31 Jan 2007 08:53:18 -0500 Subject: [Numpy-discussion] Cutting 1.0.2 release In-Reply-To: <45C09E15.40202@bigpond.net.au> References: <45BE7E57.4000105@ee.byu.edu><45BF29FB.8070305@bigpond.net.au> <45C09E15.40202@bigpond.net.au> Message-ID: <2EF79895-7254-4C19-AEBF-ED9C082DBD95@arachnedesign.net> > Thanks Alan & Chris, > > My apologies. I was trying ones(), zeros() and empty() in ipython > 0.7.2 > with the -pylab option and getting the wrong functions. On my system, > ipython -pylab imports the namespace of the oldnumeric wrapper > versions > of ones(), zeros() and empty() and I had assumed these were the core > numpy versions. Maybe this has been changed in ipython 0.7.3. Hmm ... it doesn't appear to be so: In [2]: import IPython In [3]: IPython.__version__ Out[3]: '0.7.4.svn.r2010' In [4]: ones? Type: function Base Class: String Form: Namespace: Interactive File: /opt/local/lib/python2.4/site-packages/numpy/ oldnumeric/functions.py Definition: ones(shape, typecode='l', savespace=0, dtype=None) Docstring: ones(shape, dtype=int) returns an array of the given dimensions which is initialized to all ones. Is something that should change/be reported to the ipython ML maybe? -steve From gruben at bigpond.net.au Wed Jan 31 08:54:57 2007 From: gruben at bigpond.net.au (Gary Ruben) Date: Thu, 01 Feb 2007 00:54:57 +1100 Subject: [Numpy-discussion] Requests for NumPy Ports? In-Reply-To: <45BFE7FB.6080000@ee.byu.edu> References: <45BFE7FB.6080000@ee.byu.edu> Message-ID: <45C09FB1.5030508@bigpond.net.au> Hi Travis, The vpython project was attempting to migrate to numpy and may have achieved it, but I think there may have been some outstanding questions. I'm not sure of the status, but, since you ask, I'd like to see vpython converted. Gary R. Travis Oliphant wrote: > I'm trying to help out the conversion to NumPy by offering patches to > various third-party packages that have used Numeric in the past. > > Does anybody here have requests for which packages they would like to > see converted to use NumPy? Ultimately, we are at the mercy of the > package-owners to complete the conversion by applying the patches. But, > I think it helps when there are patches already available for trying out. > > The packages I've worked on so far are here: > http://www.scipy.org/Porting_to_NumPy > > > Thanks, > > -Travis From stefan at sun.ac.za Wed Jan 31 08:55:43 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 31 Jan 2007 15:55:43 +0200 Subject: [Numpy-discussion] pyrex c_numpy.pyx problem Message-ID: <20070131135543.GC7053@mentat.za.net> Hi, Pyrex 0.9.5.1 doesn't like the following snippet out of c_numpyx.pyx: ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: cdef int numiter cdef npy_intp size, index cdef int nd cdef npy_intp dimensions[NPY_MAXDIMS] cdef flatiter iters[NPY_MAXDIMS] which corresponds to typedef struct { PyObject_HEAD int numiter; /* number of iters */ npy_intp size; /* broadcasted size */ npy_intp index; /* current index */ int nd; /* number of dims */ npy_intp dimensions[NPY_MAXDIMS]; /* dimensions */ PyArrayIterObject *iters[NPY_MAXARGS]; /* iterators */ } PyArrayMultiIterObject; in ndarrayobject.h. I changed it to be ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: cdef int numiter cdef npy_intp size, index cdef int nd cdef npy_intp *dimensions cdef flatiter **iters after which it complains /tmp/pyrex/c_numpy.pxd:99:22: Pointer base type cannot be a Python object so I changed the last line to cdef flatiter iters That compiles, but it can't be right. Any advice/suggestions? Cheers St?fan From gruben at bigpond.net.au Wed Jan 31 08:58:45 2007 From: gruben at bigpond.net.au (Gary Ruben) Date: Thu, 01 Feb 2007 00:58:45 +1100 Subject: [Numpy-discussion] Cutting 1.0.2 release In-Reply-To: <2EF79895-7254-4C19-AEBF-ED9C082DBD95@arachnedesign.net> References: <45BE7E57.4000105@ee.byu.edu><45BF29FB.8070305@bigpond.net.au> <45C09E15.40202@bigpond.net.au> <2EF79895-7254-4C19-AEBF-ED9C082DBD95@arachnedesign.net> Message-ID: <45C0A095.9070502@bigpond.net.au> Actually, I just realised; it's not an ipython problem. I think it's a matplotlib problem. I'll report it there. Gary R. Steve Lianoglou wrote: >> Thanks Alan & Chris, >> >> My apologies. I was trying ones(), zeros() and empty() in ipython >> 0.7.2 >> with the -pylab option and getting the wrong functions. On my system, >> ipython -pylab imports the namespace of the oldnumeric wrapper >> versions >> of ones(), zeros() and empty() and I had assumed these were the core >> numpy versions. Maybe this has been changed in ipython 0.7.3. > > Hmm ... it doesn't appear to be so: From david at ar.media.kyoto-u.ac.jp Wed Jan 31 08:58:14 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 31 Jan 2007 22:58:14 +0900 Subject: [Numpy-discussion] Release 0.6.1 of pyaudio, renamed pyaudiolab Message-ID: <45C0A076.2090805@ar.media.kyoto-u.ac.jp> Hi There, A few months ago, I posted the first release of pyaudio, a python module to give numpy/scipy environment audio file IO capabilities (ala matlab wavread and co). I recently took time to update it significantly, and as several people showed interest in pyaudio recently, I thought this may interest some people here. The main improvements since last public annoucement: - renamed to pyaudiolab to avoid nameclash with other package pyaudio. - ability to seek into audio files and sync files for flushing IO buffers - matlab-like API: wavread and wavwrite - improvements of the API: more similar to numpy conventions, ability to retrieve most meta-data of a file. - some bug fixing (one bug was quite severe, leading to data corruption on ubuntu with python2.5, due to some weirdness in ubuntu python2.5 packaging) - a real documentation If some people manage to use it on something else than linux, I would happy to hear (particularly Mac OS X, which I cannot test myself). Cheers, David ====== pyaudiolab ====== * WHAT FOR ?: The Goal of pyaudiolab is to give to a numpy/scipy environment some basic audio IO facilities (ala sound, wavread, wavwrite of matlab). With pyaudiolab, you should be able to read and write most common audio files from and to numpy arrays. The underlying IO operations are done using libsndfile from Erik Castro Lopo (http://www.mega-nerd.com/libsndfile/), so he is the one who did the hard work. As libsndfile has support for a vast number of audio files (including wav, aiff, but also htk, ircam, and flac, an open source lossless codec), pyaudiolab enables the import from and export to a fairly large number of file formats. * WHERE TO GET ?: http://www.ar.media.kyoto-u.ac.jp/members/david/softwares/pyaudiolab/index.html#download bzr archive: http://www.ar.media.kyoto-u.ac.jp/members/david/archives/pyaudiolab/pyaudiolab.dev # HOW TO USE ?: http://www.ar.media.kyoto-u.ac.jp/members/david/softwares/pyaudiolab/index.html From bsouthey at gmail.com Wed Jan 31 09:09:16 2007 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 31 Jan 2007 08:09:16 -0600 Subject: [Numpy-discussion] Requests for NumPy Ports? In-Reply-To: <45BFE7FB.6080000@ee.byu.edu> References: <45BFE7FB.6080000@ee.byu.edu> Message-ID: Hi, BioPython (http://biopython.org) still uses Numeric: http://biopython.org/wiki/Download Last year there was some discussion on converting to NumPy. Regards Bruce On 1/30/07, Travis Oliphant wrote: > > I'm trying to help out the conversion to NumPy by offering patches to > various third-party packages that have used Numeric in the past. > > Does anybody here have requests for which packages they would like to > see converted to use NumPy? Ultimately, we are at the mercy of the > package-owners to complete the conversion by applying the patches. But, > I think it helps when there are patches already available for trying out. > > The packages I've worked on so far are here: > http://www.scipy.org/Porting_to_NumPy > > > Thanks, > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From jks at iki.fi Wed Jan 31 09:22:23 2007 From: jks at iki.fi (=?iso-8859-1?Q?Jouni_K=2E_Sepp=E4nen?=) Date: Wed, 31 Jan 2007 16:22:23 +0200 Subject: [Numpy-discussion] Requests for NumPy Ports? References: <45BFE7FB.6080000@ee.byu.edu> Message-ID: Travis Oliphant writes: > Does anybody here have requests for which packages they would like to > see converted to use NumPy? The mixture-modeling package PyMix uses numarray: http://algorithmics.molgen.mpg.de/pymix.html -- Jouni K. Sepp?nen http://www.iki.fi/jks From Chris.Barker at noaa.gov Wed Jan 31 11:56:50 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 31 Jan 2007 08:56:50 -0800 Subject: [Numpy-discussion] Requests for NumPy Ports? In-Reply-To: <45BFE7FB.6080000@ee.byu.edu> References: <45BFE7FB.6080000@ee.byu.edu> Message-ID: <45C0CA52.4050203@noaa.gov> Travis Oliphant wrote: > I'm trying to help out the conversion to NumPy by offering patches to > various third-party packages that have used Numeric in the past. Travis, you are amazing! Thanks for doing this. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From whughes at nb.sympatico.ca Tue Jan 30 19:59:47 2007 From: whughes at nb.sympatico.ca (William Hughes) Date: Tue, 30 Jan 2007 19:59:47 -0500 Subject: [Numpy-discussion] Conversion of Numeric arrays from/to numpy arrays Message-ID: <45BFEA03.5090806@nb.sympatico.ca> Hi: Is there a easy and efficient way to convert between Numeric and numpy arrays? - William Hughes From Chris.Barker at noaa.gov Wed Jan 31 12:08:48 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 31 Jan 2007 09:08:48 -0800 Subject: [Numpy-discussion] Patch for RPy to use NumPy uploaded to www.scipy.org In-Reply-To: <45BFE927.1050204@optushome.com.au> References: <45BFDE21.6020902@ee.byu.edu> <45BFE927.1050204@optushome.com.au> Message-ID: <45C0CD20.2000407@noaa.gov> Tim Churches wrote: > Yes, except that we have quite a lot of code in fielded applications > which is written with Numeric, and which also uses RPy. We currently > have no funds to do the Numeric to NumPy port in our code. If RPy > dropped support for Numeric, we would be forced to use increasing old > versions of RPy, which means we would be locked into increasing old > versions of R. Aside from Robert's point about passing Numeric arrays to numpy -- It really isn't that much work to port Numeric code to numpy, particularly if that code is all Python. Truly, numpy is not some brand new beast, it's the next generation Numeric -- think of it as Numeric2. Most of it is the same, and the differences are there because it was decided to sacrifice some backwards compatibility to get a truly BETTER api. Any code that is under active development (and ideally has some tests!) can be ported to numpy with very little extra work. If your code is not under active development, then you might as well stick with all the existing libs that already work. i.e. if you are upgrading RPy, you might as well upgrade to numpy as well. I haven't done it myself[1], but even if you have C code that works with Numeric arrays, porting that isn't too hard either -- do you think Travis would be able to do it for an assortment of other packages if it were? And he (and others on this list) will help! On the other hand, supporting multiple Num* packages really is a pain in the *&^@*, and the users are left with a limited subset of what they could have. -Chris [1] I haven't done it myself because I've found that most of the C code I wrote for Numeric is no longer necessary with a few additional features in numpy: record arrays, in=place byteswap, fromfile. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From kwgoodman at gmail.com Wed Jan 31 12:10:35 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 31 Jan 2007 09:10:35 -0800 Subject: [Numpy-discussion] Requests for NumPy Ports? In-Reply-To: <45BFE7FB.6080000@ee.byu.edu> References: <45BFE7FB.6080000@ee.byu.edu> Message-ID: On 1/30/07, Travis Oliphant wrote: > > I'm trying to help out the conversion to NumPy by offering patches to > various third-party packages that have used Numeric in the past. > > Does anybody here have requests for which packages they would like to > see converted to use NumPy? Ultimately, we are at the mercy of the > package-owners to complete the conversion by applying the patches. But, > I think it helps when there are patches already available for trying out. > > The packages I've worked on so far are here: > http://www.scipy.org/Porting_to_NumPy To remove numeric from my system I'd have to remove 34 packages. Some of the ones I use a lot: gedit meld nautilus matplotlib rhythmbox gnome-terminal What kind of matrix computations do these programs need? I'm surprised that so many packages use numeric. But that must mean that numpy has a bright future. From Chris.Barker at noaa.gov Wed Jan 31 12:11:15 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 31 Jan 2007 09:11:15 -0800 Subject: [Numpy-discussion] Conversion of Numeric arrays from/to numpy arrays In-Reply-To: <45BFEA03.5090806@nb.sympatico.ca> References: <45BFEA03.5090806@nb.sympatico.ca> Message-ID: <45C0CDB3.9040903@noaa.gov> William Hughes wrote: > Hi: > Is there a easy and efficient way to convert between > Numeric and numpy arrays? sure is: Numeric.asarray() numpy.asarray() Could it be any easier? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Wed Jan 31 12:18:50 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 31 Jan 2007 10:18:50 -0700 Subject: [Numpy-discussion] Requests for NumPy Ports? In-Reply-To: References: <45BFE7FB.6080000@ee.byu.edu> Message-ID: <45C0CF7A.6040807@ee.byu.edu> Keith Goodman wrote: >On 1/30/07, Travis Oliphant wrote: > > >>I'm trying to help out the conversion to NumPy by offering patches to >>various third-party packages that have used Numeric in the past. >> >>Does anybody here have requests for which packages they would like to >>see converted to use NumPy? Ultimately, we are at the mercy of the >>package-owners to complete the conversion by applying the patches. But, >>I think it helps when there are patches already available for trying out. >> >>The packages I've worked on so far are here: >>http://www.scipy.org/Porting_to_NumPy >> >> > >To remove numeric from my system I'd have to remove 34 packages. Some >of the ones I use a lot: > >gedit >meld >nautilus >matplotlib >rhythmbox >gnome-terminal > > Most of these are probably the gtk-python extension which can use Numeric (I submitted the original code for that). I will have to submit the patch to use NumPy. Thanks for the reminder. -Travis From fperez.net at gmail.com Wed Jan 31 12:31:48 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 31 Jan 2007 10:31:48 -0700 Subject: [Numpy-discussion] Cutting 1.0.2 release In-Reply-To: <45C0A095.9070502@bigpond.net.au> References: <45BE7E57.4000105@ee.byu.edu> <45BF29FB.8070305@bigpond.net.au> <45C09E15.40202@bigpond.net.au> <2EF79895-7254-4C19-AEBF-ED9C082DBD95@arachnedesign.net> <45C0A095.9070502@bigpond.net.au> Message-ID: On 1/31/07, Gary Ruben wrote: > Actually, I just realised; it's not an ipython problem. I think it's a > matplotlib problem. I'll report it there. Until mpl drops support for the compatibility layers, you may want to set up a simple pylab profile. In ~/.ipython make a file called 'ipythonrc-pylab' consisting of: ####### # Load default config include ipythonrc # Add single-line python statements here execute from numpy import * ######## Since pylab does a 'from .num?. import *' this will ensure that the top-level visible functions are the current numpy ones, not the compatibility layer ones. You then start things with: ipython -pylab -p pylab and you'll get: In [1]: zeros? Type: builtin_function_or_method Base Class: Namespace: Interactive Docstring: zeros((d1,...,dn),dtype=float,order='C') Return a new array of shape (d1,...,dn) and type typecode with all it's entries initialized to zero. Not ideal, but it's a little hack that will work in practice until we finish crossing the muddy river of backwards compatibility. Cheers, f From Chris.Barker at noaa.gov Wed Jan 31 12:37:35 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 31 Jan 2007 09:37:35 -0800 Subject: [Numpy-discussion] Requests for NumPy Ports? In-Reply-To: <45C0CF7A.6040807@ee.byu.edu> References: <45BFE7FB.6080000@ee.byu.edu> <45C0CF7A.6040807@ee.byu.edu> Message-ID: <45C0D3DF.1070606@noaa.gov> Travis Oliphant wrote: > Most of these are probably the gtk-python extension which can use > Numeric This strikes me as an excellent argument for including an n-d array in the Python standard lib. I can't imagine gtk-python is doing a lot of scientific number crunching -- it's probably using arrays as a handy and compact n-d data storage and exchange format. It might be interesting to see how it's really being used, and if the PEP would meet its needs. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From fperez.net at gmail.com Wed Jan 31 12:42:13 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 31 Jan 2007 10:42:13 -0700 Subject: [Numpy-discussion] memmap close() and flush() In-Reply-To: <45C044C8.1010505@ieee.org> References: <45C044C8.1010505@ieee.org> Message-ID: On 1/31/07, Travis Oliphant wrote: > Sebastian Haase wrote: > > Hi! > > Do numpy memmap have a way of explicitly > > flushing data to disk > > and/or > > closing the memmap. > > > There is a sync method that performs the flush. To close the memmap, > delete it. > > More detail: > The memmap sub-class has a _mmap attribute that is the Python > memory-map object. Quick question out of ignorance: shouldn't the API offer an explicit close method? The reason is that in python, doing 'del foo' doesn't ensure real deletion of the underlying object that will fire the close methods, since you may easily have other names pointing to the same object. Having an explicit, guaranteed way to call .close() sounds like a good thing to me, but I may well be missing something. Cheers, f From oliphant at ee.byu.edu Wed Jan 31 12:46:15 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 31 Jan 2007 10:46:15 -0700 Subject: [Numpy-discussion] Requests for NumPy Ports? In-Reply-To: <45C0D3DF.1070606@noaa.gov> References: <45BFE7FB.6080000@ee.byu.edu> <45C0CF7A.6040807@ee.byu.edu> <45C0D3DF.1070606@noaa.gov> Message-ID: <45C0D5E7.4@ee.byu.edu> Christopher Barker wrote: >Travis Oliphant wrote: > > >>Most of these are probably the gtk-python extension which can use >>Numeric >> >> > >This strikes me as an excellent argument for including an n-d array in >the Python standard lib. > > It is absolutely a good argument. If Python had the array interface, gtk would just use it and not have any dependency on Numeric or NumPy at all. Every windowing interface (and OpenGL) would have the same need. To me this is so obvious that I don't understand the resistance in the Python community to the concept. Perhaps I'm just mis-interpreting the feedback I've received, but it would be really helpful to move things forward if somebody with write priviledges would just say: "We need this array interface thing so let's hash out Travis's proposal." In other words we can argue the details but I need some more support for the general idea to continue to invest time in making the array interface work. I'm going to try and get the BDFL or somebody with similar influence to agree to basically say something like that at PyCon this year. If anybody going to PyCon want's to help me, let's get together. -Travis From oliphant at ee.byu.edu Wed Jan 31 12:47:40 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 31 Jan 2007 10:47:40 -0700 Subject: [Numpy-discussion] memmap close() and flush() In-Reply-To: References: <45C044C8.1010505@ieee.org> Message-ID: <45C0D63C.5080807@ee.byu.edu> Fernando Perez wrote: >On 1/31/07, Travis Oliphant wrote: > > >>Sebastian Haase wrote: >> >> >>>Hi! >>>Do numpy memmap have a way of explicitly >>>flushing data to disk >>>and/or >>>closing the memmap. >>> >>> >>> >>There is a sync method that performs the flush. To close the memmap, >>delete it. >> >>More detail: >> The memmap sub-class has a _mmap attribute that is the Python >>memory-map object. >> >> > >Quick question out of ignorance: shouldn't the API offer an explicit >close method? The reason is that in python, doing 'del foo' doesn't >ensure real deletion of the underlying object that will fire the >close methods, since you may easily have other names pointing to the >same object. Having an explicit, guaranteed way to call .close() >sounds like a good thing to me, but I may well be missing something. > > > I don't know. If you have other things pointing to it, should you really close it? At any rate you have access to the mmap file through the _mmap attribute. So, you can always do self._mmap.close() -Travis From fperez.net at gmail.com Wed Jan 31 12:53:48 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 31 Jan 2007 10:53:48 -0700 Subject: [Numpy-discussion] memmap close() and flush() In-Reply-To: <45C0D63C.5080807@ee.byu.edu> References: <45C044C8.1010505@ieee.org> <45C0D63C.5080807@ee.byu.edu> Message-ID: On 1/31/07, Travis Oliphant wrote: > Fernando Perez wrote: > I don't know. If you have other things pointing to it, should you > really close it? Well, it's like a file: you can close it because you've decided it's time to close it, and I think it's better that other references get an exception if they try to write to it when they shouldn't: In [2]: file1 = open('foo','w') In [3]: file1.write('Some text') In [4]: file2 = file1 In [5]: file1.close() In [6]: file2.write("I'm not the original owner but I'll try to write anyway") --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/fperez/ ValueError: I/O operation on closed file This seems like the right API to me. > At any rate you have access to the mmap file through the _mmap > attribute. So, you can always do > > self._mmap.close() I don't like an API that encourages access to internal semi-private members, and it seems to me that closing the object is a reasonably top-level operation whose impmlementation details (in this case the existence and name of the _mmap member) should be encapsulated out. Cheers, f From robert.kern at gmail.com Wed Jan 31 13:08:55 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 31 Jan 2007 12:08:55 -0600 Subject: [Numpy-discussion] pyrex c_numpy.pyx problem In-Reply-To: <20070131135543.GC7053@mentat.za.net> References: <20070131135543.GC7053@mentat.za.net> Message-ID: <45C0DB37.8090603@gmail.com> Stefan van der Walt wrote: > Any advice/suggestions? I've just committed a fix. I changed the [NPY_MAXDIMS] arrays to pointers (Pyrex doesn't care, really) and changed the flatiter *items to void **items. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From seb.haase at gmx.net Wed Jan 31 17:07:15 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Wed, 31 Jan 2007 14:07:15 -0800 Subject: [Numpy-discussion] memmap close() and flush() In-Reply-To: References: <45C044C8.1010505@ieee.org> <45C0D63C.5080807@ee.byu.edu> Message-ID: On 1/31/07, Fernando Perez wrote: > On 1/31/07, Travis Oliphant wrote: > > Fernando Perez wrote: > > > I don't know. If you have other things pointing to it, should you > > really close it? > > Well, it's like a file: you can close it because you've decided it's > time to close it, and I think it's better that other references get an > exception if they try to write to it when they shouldn't: > > In [2]: file1 = open('foo','w') > > In [3]: file1.write('Some text') > > In [4]: file2 = file1 > > In [5]: file1.close() > > In [6]: file2.write("I'm not the original owner but I'll try to write anyway") > --------------------------------------------------------------------------- > exceptions.ValueError Traceback (most > recent call last) > > /home/fperez/ > > ValueError: I/O operation on closed file > > > This seems like the right API to me. > > > At any rate you have access to the mmap file through the _mmap > > attribute. So, you can always do > > > > self._mmap.close() > > I don't like an API that encourages access to internal semi-private > members, and it seems to me that closing the object is a reasonably > top-level operation whose impmlementation details (in this case the > existence and name of the _mmap member) should be encapsulated out. > > Cheers, > > f After asking this question rather for acedemical reasons, I now realize that I indead must have a "somehow dangling" (i.e. not deleted) reference problem. I create my Medical-Image-File-class object on a 400MB file repeatedly (throwing the result away) and after 5 calles I get: File "/jws30/haase/PrLinN/Priithon/Mrc.py", line 55, in __init__ self.m = N.memmap(path, mode=mode) File "/home/haase/qqq/lib/python/numpy/core/memmap.py", line 67, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc) EnvironmentError: [Errno 12] Cannot allocate memory Calling gc.collect() seams to clean things up and I can create 4-5 times afterwards, before running out of memory space again. Note: My code is based on code that was tested and worked using numarray. Thanks, Sebastian From haase at msg.ucsf.edu Wed Jan 31 17:08:20 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 31 Jan 2007 14:08:20 -0800 Subject: [Numpy-discussion] memmap close() and flush() In-Reply-To: References: <45C044C8.1010505@ieee.org> <45C0D63C.5080807@ee.byu.edu> Message-ID: On 1/31/07, Fernando Perez wrote: > On 1/31/07, Travis Oliphant wrote: > > Fernando Perez wrote: > > > I don't know. If you have other things pointing to it, should you > > really close it? > > Well, it's like a file: you can close it because you've decided it's > time to close it, and I think it's better that other references get an > exception if they try to write to it when they shouldn't: > > In [2]: file1 = open('foo','w') > > In [3]: file1.write('Some text') > > In [4]: file2 = file1 > > In [5]: file1.close() > > In [6]: file2.write("I'm not the original owner but I'll try to write anyway") > --------------------------------------------------------------------------- > exceptions.ValueError Traceback (most > recent call last) > > /home/fperez/ > > ValueError: I/O operation on closed file > > > This seems like the right API to me. > > > At any rate you have access to the mmap file through the _mmap > > attribute. So, you can always do > > > > self._mmap.close() > > I don't like an API that encourages access to internal semi-private > members, and it seems to me that closing the object is a reasonably > top-level operation whose impmlementation details (in this case the > existence and name of the _mmap member) should be encapsulated out. > After asking this question rather for acedemical reasons, I now realize that I indead must have a "somehow dangling" (i.e. not deleted) reference problem. I create my Medical-Image-File-class object on a 400MB file repeatedly (throwing the result away) and after 5 calles I get: File "/jws30/haase/PrLinN/Priithon/Mrc.py", line 55, in __init__ self.m = N.memmap(path, mode=mode) File "/home/haase/qqq/lib/python/numpy/core/memmap.py", line 67, in __new__ mm = mmap.mmap(fid.fileno(), bytes, access=acc) EnvironmentError: [Errno 12] Cannot allocate memory Calling gc.collect() seams to clean things up and I can create 4-5 times afterwards, before running out of memory space again. Note: My code is based on code that was tested and worked using numarray. Thanks, Sebastian From haase at msg.ucsf.edu Wed Jan 31 17:13:37 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 31 Jan 2007 14:13:37 -0800 Subject: [Numpy-discussion] numpy.scipy.org refers to sourceforge mailing list Message-ID: Hi, On the numpy page: http://numpy.scipy.org/ the is this text: """ Questions? Ask them at the numpy-discussion at lists.sourceforge.net mailing list """ Is sourceforge.net still used for numpy ? Also I think it should point to the list sign-up-page since only members are allowed to posts to the list - is this correct ? - Sebastian. From gruben at bigpond.net.au Wed Jan 31 17:13:55 2007 From: gruben at bigpond.net.au (Gary Ruben) Date: Thu, 01 Feb 2007 09:13:55 +1100 Subject: [Numpy-discussion] Cutting 1.0.2 release In-Reply-To: References: <45BE7E57.4000105@ee.byu.edu> <45BF29FB.8070305@bigpond.net.au> <45C09E15.40202@bigpond.net.au> <2EF79895-7254-4C19-AEBF-ED9C082DBD95@arachnedesign.net> <45C0A095.9070502@bigpond.net.au> Message-ID: <45C114A3.80106@bigpond.net.au> Thanks Fernando, Good idea. I'll apply your suggestion. Gary Fernando Perez wrote: > On 1/31/07, Gary Ruben wrote: >> Actually, I just realised; it's not an ipython problem. I think it's a >> matplotlib problem. I'll report it there. > > Until mpl drops support for the compatibility layers, you may want to > set up a simple pylab profile. In ~/.ipython make a file called > 'ipythonrc-pylab' consisting of: > Not ideal, but it's a little hack that will work in practice until we > finish crossing the muddy river of backwards compatibility. > > Cheers, > > f From bbands at gmail.com Wed Jan 31 17:44:18 2007 From: bbands at gmail.com (BBands) Date: Wed, 31 Jan 2007 14:44:18 -0800 Subject: [Numpy-discussion] query to array Message-ID: <6e8360ad0701311444g62162cfs1d7ad6aa2c521a5a@mail.gmail.com> Good afternoon, The following works: import pyodbc import numpy as np connection = pyodbc.connect('DSN=DSNname') cursor = connection.cursor() symbol = 'ibm' request = "select to_days(Date), Close from price where symbol = '" + symbol + "' and date > '2006-01-01'" for row in cursor.execute(request): price.append([row[0], row[1]]) priceArray = np.array(price) Is there a better way to do this? Can fromiter be used? Could this query result be put into a wider array from the start such that columns 2... could be filled with calcs later? Thanks in advance, jab -- John Bollinger, CFA, CMT www.BollingerBands.com If you advance far enough, you arrive at the beginning. From kwgoodman at gmail.com Wed Jan 31 18:00:12 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 31 Jan 2007 15:00:12 -0800 Subject: [Numpy-discussion] query to array In-Reply-To: <6e8360ad0701311444g62162cfs1d7ad6aa2c521a5a@mail.gmail.com> References: <6e8360ad0701311444g62162cfs1d7ad6aa2c521a5a@mail.gmail.com> Message-ID: On 1/31/07, BBands wrote: > import pyodbc > import numpy as np > > connection = pyodbc.connect('DSN=DSNname') > cursor = connection.cursor() > symbol = 'ibm' > request = "select to_days(Date), Close from price where symbol = '" + > symbol + "' and date > '2006-01-01'" > for row in cursor.execute(request): > price.append([row[0], row[1]]) > priceArray = np.array(price) > > Is there a better way to do this? Can fromiter be used? Why not just pull it all at once with cursor.fetchall()? Then use np.asarray on the result. From tom.denniston at alum.dartmouth.org Wed Jan 31 18:04:02 2007 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Wed, 31 Jan 2007 17:04:02 -0600 Subject: [Numpy-discussion] query to array In-Reply-To: <6e8360ad0701311444g62162cfs1d7ad6aa2c521a5a@mail.gmail.com> References: <6e8360ad0701311444g62162cfs1d7ad6aa2c521a5a@mail.gmail.com> Message-ID: i would do something like the following. I don't have your odbc library so I mocked it up with a fake iterator called "it". This example would be for a two column result where the first is an int and the second a string. Note it creates a recarray which you can have match you database column names if you want to generalize the function. All you need to provide is the dtype equivalent to your query result layout. In [2]: import numpy In [3]: it = iter([(1,'String1'), (2, 'String2')]) In [4]: dtype = numpy.dtype([('intvalue', int), ('stringcolumn', '|S20')]) In [5]: numpy.fromiter(it, dtype) Out[5]: array([(1, 'String1'), (2, 'String2')], dtype=[('intvalue', ' wrote: > Good afternoon, > > The following works: > > import pyodbc > import numpy as np > > connection = pyodbc.connect('DSN=DSNname') > cursor = connection.cursor() > symbol = 'ibm' > request = "select to_days(Date), Close from price where symbol = '" + > symbol + "' and date > '2006-01-01'" > for row in cursor.execute(request): > price.append([row[0], row[1]]) > priceArray = np.array(price) > > Is there a better way to do this? Can fromiter be used? > > Could this query result be put into a wider array from the start such > that columns 2... could be filled with calcs later? > > Thanks in advance, > > jab > -- > John Bollinger, CFA, CMT > www.BollingerBands.com > > If you advance far enough, you arrive at the beginning. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From kwgoodman at gmail.com Wed Jan 31 18:19:14 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 31 Jan 2007 15:19:14 -0800 Subject: [Numpy-discussion] query to array In-Reply-To: References: <6e8360ad0701311444g62162cfs1d7ad6aa2c521a5a@mail.gmail.com> Message-ID: On 1/31/07, Tom Denniston wrote: > i would do something like the following. I don't have your odbc > library so I mocked it up with a fake iterator called "it". This > example would be for a two column result where the first is an int and > the second a string. Note it creates a recarray which you can have > match you database column names if you want to generalize the > function. All you need to provide is the dtype equivalent to your > query result layout. > > In [2]: import numpy > > In [3]: it = iter([(1,'String1'), (2, 'String2')]) > > In [4]: dtype = numpy.dtype([('intvalue', int), ('stringcolumn', '|S20')]) > > In [5]: numpy.fromiter(it, dtype) > Out[5]: > array([(1, 'String1'), (2, 'String2')], > dtype=[('intvalue', ' References: <6e8360ad0701311444g62162cfs1d7ad6aa2c521a5a@mail.gmail.com> Message-ID: Ok, I don't have the pyodbc so I can't test this example but it would look something like this assuming the date was a 10 char string (numpy doesn't support mx.DateTimes natively, much to my chagrin) and the Close was a float: > import pyodbc > import numpy as np > > connection = pyodbc.connect('DSN=DSNname') > cursor = connection.cursor() > symbol = 'ibm' > request = "select to_days(Date), Close from price where symbol = '" + > symbol + "' and date > '2006-01-01'" > dtype = numpy.dtype([('Date', '|S10'), ('Close', float)]) > resultIter = cursor.execute(request) > arr = numpy.fromiter(resultIter, dtype) As I said I cant test this because I don't have the lib, but it should work fine. Finally, the '|S10' is the string encoding for 10 character wide string. Others might know of a more elegant way to express this. --Tom On 1/31/07, Keith Goodman wrote: > On 1/31/07, Tom Denniston wrote: > > i would do something like the following. I don't have your odbc > > library so I mocked it up with a fake iterator called "it". This > > example would be for a two column result where the first is an int and > > the second a string. Note it creates a recarray which you can have > > match you database column names if you want to generalize the > > function. All you need to provide is the dtype equivalent to your > > query result layout. > > > > In [2]: import numpy > > > > In [3]: it = iter([(1,'String1'), (2, 'String2')]) > > > > In [4]: dtype = numpy.dtype([('intvalue', int), ('stringcolumn', '|S20')]) > > > > In [5]: numpy.fromiter(it, dtype) > > Out[5]: > > array([(1, 'String1'), (2, 'String2')], > > dtype=[('intvalue', ' > All this fromiter and recarray is confusing to me. But I'd like to learn. > > How would I hook all this up to a query? > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From haase at msg.ucsf.edu Wed Jan 31 22:39:04 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 31 Jan 2007 19:39:04 -0800 Subject: [Numpy-discussion] usage of __del__ in python classes Message-ID: Hi, Before I start I want to admit that I don't understand much about this. I just saw that the memmap class defines __del__ and that I had problems in the past when I added a 'def __del__' to a class of mine. So here is a quote, I would like to know if this is "standard knowledge" on this list or not. # I found the info originally here: http://arctrix.com/nas/python/gc/ # Circular references which are garbage are detected when the optional cycle detector is enabled (it's on by default), but can only be cleaned up if there are no Python-level __del__() methods involved. Refer to the documentation for the 'gc' module for more information about how __del__() methods are handled by the cycle detector, particularly the description of the garbage value. Notice: [warning] Due to the precarious circumstances under which __del__() methods are invoked, exceptions that occur during their execution are ignored, and a warning is printed to sys.stderr instead. Also, when __del__() is invoked in response to a module being deleted (e.g., when execution of the program is done), other globals referenced by the __del__() method may already have been deleted. For this reason, __del__() methods should do the absolute minimum needed to maintain external invariants. Cheers, Sebastian Haase From david at ar.media.kyoto-u.ac.jp Wed Jan 31 23:04:03 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 01 Feb 2007 13:04:03 +0900 Subject: [Numpy-discussion] usage of __del__ in python classes In-Reply-To: References: Message-ID: <45C166B3.5080000@ar.media.kyoto-u.ac.jp> Sebastian Haase wrote: > Hi, > Before I start I want to admit that I don't understand much about > this. I just saw that the memmap class defines __del__ and that I had > problems in the past when I added a 'def __del__' to a class of mine. > So here is a quote, I would like to know if this is "standard > knowledge" on this list or not. > > # I found the info originally here: http://arctrix.com/nas/python/gc/ > # Circular references which are garbage are detected when the > optional cycle detector is enabled (it's on by default), but can only > be cleaned up if there are no Python-level __del__() methods involved. > Refer to the documentation for the 'gc' module for more information > about how __del__() methods are handled by the cycle detector, > particularly the description of the garbage value. Notice: [warning] > Due to the precarious circumstances under which __del__() methods are > invoked, exceptions that occur during their execution are ignored, and > a warning is printed to sys.stderr instead. Also, when __del__() is > invoked in response to a module being deleted (e.g., when execution of > the program is done), other globals referenced by the __del__() method > may already have been deleted. For this reason, __del__() methods > should do the absolute minimum needed to maintain external invariants. > > This is particularly annoying when using ctypes, and you need to clean some ressources. I had this problem with pyaudiolab: Class A def __init__(self): # here call C library open def close(self): # here dispose of all handlers related to the file def __del__(self): # here dipose of all handlers (if not cleaned by close) As mentionned in the above paragraph, the problem is that sometimes (typically, when exiting the python interpreter) the C library reference is "out of scope" (not sure the expressionis appropriate for python) before calling __del__ on an instance of A. I am using ctypes for the interface with C in this case, so what I do is: def __del__(self, close_func = _snd.sf_close): # Stupid python needs the close_func, otherwise # it may clean ctypes before calling here if not(self.hdl == 0): close_func(self.hdl) self.hdl = 0 This is not pretty, and you better not call __del__ with your own close_func function, but that's the only solution I can think of. David