From faltet at carabos.com Tue Nov 1 01:37:26 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Nov 1 01:37:26 2005 Subject: [Numpy-discussion] byteswap questions In-Reply-To: References: Message-ID: <1130838789.2968.23.camel@localhost.localdomain> El dl 31 de 10 del 2005 a les 16:53 -0800, en/na Russell E. Owen va escriure: > I've got a module that can send array data to ds9. It screws up on > byteswapped data and I'm trying to fix it. I don't exactly know what a ds9 is, but I'm supposing here that it is kind of an exotic computer. > I need to know if the order is bigendian or littleendian, and I can't > find a documented way to get that, just an undocumented attribute > _byteorder. Yes, it's undocumented, but as far as I can tell, this works flawlessly in numarray. > 2) If the array is not in native byte order, make a native byte ordered > copy before sending it. > > The following seems to work: > if arr.isbyteswapped(): > arr = arr.copy() > > but is this the recommended way to get native-order copy? If you can afford doing a memory copy, yes, I think this is a good way to do it. > - Is "copy" guaranteed to return a copy in native byte order? The > documentation doesn't say. It does: In [23]:a._byteorder Out[23]:'big' In [24]:a.copy()._byteorder Out[24]:'little' In general, copy() will return you a well-behaved (native-byte ordered, non-strided and non-offsetted) array. > - I was tempted to use num.array(arr) to make the copy, but the > documentation doesn't say what "array" does if the input is a already a > numarray array. This also works: In [25]:array(a)._byteorder Out[25]:'little' and it is very similar to copy() in performance: In [34]:t1=Timer("b=a.copy()","import numarray; a=numarray.arange(10);a.byteswap();a._byteorder='big'") In [35]:t1.repeat(3,10000) Out[35]:[1.0613389015197754, 1.0313379764556885, 1.0303771495819092] In [36]:t2=Timer("b=numarray.array(a)","import numarray; a=numarray.arange(10);a.byteswap();a._byteorder='big'") In [37]:t2.repeat(3,10000) Out[37]:[1.2250277996063232, 1.2067301273345947, 1.2336349487304688] (perhaps copy is just a little bit faster) > As an aside, I tried to use the byteswapped method, but it has a totally > different effect: > > >>> d > array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], > ...type=Float32) > >>> d.isbyteswapped() > 1 > >>> dcopy = d.copy() > >>> d > array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], > ...type=Float32) > >>> dcopy.isbyteswapped() > 0 > >>> dswap = d.byteswapped() > >>> dswap > array([[ 1.91063654e-38, 1.91063654e-38, 1.91063654e-38, ..., > ...type=Float32) > > which I guess means I'd have to byteswap the byteswapped data. Aargh. This is the intended behaviour. From the docstrings: In [59]:a.byteswap? [snip...] Docstring: Byteswap data in place, leaving the _byteorder attribute untouched. So, you already have done a byteswapping, but you still need to tell the object that its byteorder has actually changed (if don't, numarray will convert the data to you local native order before printing it on the flight). Just issue a: dswap._byteorder = {"little":"big","big":"little"}[dswap._byteorder] and you are done. In fact, using byteswap() does the conversion in-place, and doesn't need a memory copy. This is actually faster than doing a copy() much of the times: In [60]:t3=Timer("b=a.byteswap();a._byteorder='little'","import numarray; a=numarray.arange(10);a.byteswap();a._byteorder='big'") In [61]:t3.repeat(3,10000) Out[61]:[0.23770284652709961, 0.23195695877075195, 0.23380589485168457] Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From faltet at carabos.com Tue Nov 1 03:38:57 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Nov 1 03:38:57 2005 Subject: [Numpy-discussion] Performance of the array protocol Message-ID: <1130846099.2968.39.camel@localhost.localdomain> Hi, I'm trying to start using the array protocol for conversion between Numeric <--> numarray (and newcore in the future), but I'm a bit disappointed because of its performance. For numarray --> Numeric we have: >>> t1=timeit.Timer("num=Numeric.array(na)", "import numarray; import Numeric; na=numarray.arange(10)") >>> t1.repeat(3,10000) [0.59375977516174316, 0.57908082008361816, 0.56574010848999023] >>>t2=timeit.Timer("num=Numeric.fromstring(na._data,typecode=na.typecode())", "import numarray; import Numeric; na=numarray.arange(10)") >>> t2.repeat(3,10000) [0.11653494834899902, 0.1140749454498291, 0.1141819953918457] i.e. the array protocol seems 5x slower than the fromstring() method. Conversely, for Numeric --> numarray: >>> t3=timeit.Timer("na=numarray.array(num)", "import numarray; import Numeric;num=Numeric.arange(10)") >>> t3.repeat(3,10000) [1.3475611209869385, 1.3277668952941895, 1.3417830467224121] >>>t4=timeit.Timer("na=numarray.array(buffer(num),type=num.typecode(),shape=num.shape)", "import numarray; import Numeric; num=Numeric.arange(10)") >>> t4.repeat(3,10000) [0.42027187347412109, 0.41690587997436523, 0.41626906394958496] in this case, the array protocol is 3x slower than using the buffer interface. I'm wondering whether this relatively poor performance in present implementation of the array protocol is surmountable or is an intrinsic limitation of it. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Tue Nov 1 08:13:06 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 08:13:06 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <1130846099.2968.39.camel@localhost.localdomain> References: <1130846099.2968.39.camel@localhost.localdomain> Message-ID: <436793CC.5030106@ee.byu.edu> Francesc Altet wrote: >Hi, > >I'm trying to start using the array protocol for conversion between >Numeric <--> numarray (and newcore in the future), but I'm a bit >disappointed because of its performance. For numarray --> Numeric we >have: > > > >>>>t1=timeit.Timer("num=Numeric.array(na)", "import numarray; import >>>> >>>> >Numeric; na=numarray.arange(10)") > > >>>>t1.repeat(3,10000) >>>> >>>> >[0.59375977516174316, 0.57908082008361816, 0.56574010848999023] > > >>>>t2=timeit.Timer("num=Numeric.fromstring(na._data,typecode=na.typecode())", "import numarray; import Numeric; na=numarray.arange(10)") >>>>t2.repeat(3,10000) >>>> >>>> >[0.11653494834899902, 0.1140749454498291, 0.1141819953918457] > >i.e. the array protocol seems 5x slower than the fromstring() method. > > > If you are going to be copying the data anyway, then there may be no advantage to the array protocol (in fact because it has to look up several attributes of the input object it can be slower). When you use Numeric.array(na) it makes a copy of the data by default. The idea is to be able to use the array protocol to not have to make copies of the data. Try using num = Numeric.array(na,copy=0) in your first timing runs and see what that provides. >Conversely, for Numeric --> numarray: > > > >>>>t3=timeit.Timer("na=numarray.array(num)", "import numarray; import >>>> >>>> >Numeric;num=Numeric.arange(10)") > > >>>>t3.repeat(3,10000) >>>> >>>> >[1.3475611209869385, 1.3277668952941895, 1.3417830467224121] > > >>>>t4=timeit.Timer("na=numarray.array(buffer(num),type=num.typecode(),shape=num.shape)", "import numarray; import Numeric; num=Numeric.arange(10)") >>>>t4.repeat(3,10000) >>>> >>>> >[0.42027187347412109, 0.41690587997436523, 0.41626906394958496] > >in this case, the array protocol is 3x slower than using the buffer >interface. > > Again, you are making copies of the data. I'm not sure how numarray handles the array protocol on consumption of the interface, so I can't comment further. -Travis From oliphant at ee.byu.edu Tue Nov 1 08:19:01 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 08:19:01 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <1130846099.2968.39.camel@localhost.localdomain> References: <1130846099.2968.39.camel@localhost.localdomain> Message-ID: <43679557.70508@ee.byu.edu> Francesc Altet wrote: >Hi, > >I'm trying to start using the array protocol for conversion between >Numeric <--> numarray (and newcore in the future), but I'm a bit >disappointed because of its performance. For numarray --> Numeric we >have: > > The other factor in this discussion is that you are creating and sharing relatively small arrays. The fromstring apporach is not a problem if all you need is a copy of the data for small arrays. The array protocol approach uses attribute lookups on an object. This is going to have overhead that will only be useful for large arrays (or arrays that you want to have share the same data region). So, I guess the answer to your question is that for small arrays it is an intrinsic limitation of the use of Python attributes in the array protocol. -Travis From faltet at carabos.com Tue Nov 1 09:20:16 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Nov 1 09:20:16 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <436793CC.5030106@ee.byu.edu> References: <1130846099.2968.39.camel@localhost.localdomain> <436793CC.5030106@ee.byu.edu> Message-ID: <1130866551.2968.69.camel@localhost.localdomain> El dt 01 de 11 del 2005 a les 09:11 -0700, en/na Travis Oliphant va escriure: > If you are going to be copying the data anyway, then there may be no > advantage to the array protocol (in fact because it has to look up > several attributes of the input object it can be slower). When you use > Numeric.array(na) it makes a copy of the data by default. > > The idea is to be able to use the array protocol to not have to make > copies of the data. Yes, I don't want to do a copy. And, in fact, I want to use moderately large array conversions (10**4 ~ 10**6 elements). > Try using num = Numeric.array(na,copy=0) in your first timing runs and > see what that provides. Good! Using copy=0 and larger arrays (10**5 elements) I'm getting now: >>> t1_2=timeit.Timer("num=Numeric.array(na,copy=0)", "import numarray; import Numeric; na=numarray.arange(100000)") >>> t1_2.repeat(3,1000) [0.064317941665649414, 0.060917854309082031, 0.07666015625] >>> t2_2=timeit.Timer("num=Numeric.fromstring(na._data,typecode=na.typecode())", "import numarray; import Numeric; na=numarray.arange(100000)") >>> t2_2.repeat(3,1000) [4.8582658767700195, 4.8404099941253662, 4.8652839660644531] So, the implementation of the array protocol in the numarray --> Numeric way is performing ashtonishingly well :-) For the records, using the array protocol without a copy gives: >>> t1=timeit.Timer("num=Numeric.array(na)", "import numarray; import Numeric; na=numarray.arange(100000)") >>> t1.repeat(3,1000) [5.014805793762207, 4.9959368705749512, 5.0420081615447998] i.e. almost as fast a the fromstring() method, which is very good as well! BTW, I'm wondering whether a False value for copy should be used as the default instead of True. IMO, many people would want to make use of the array protocol just to access easily the data, and making a copy() behind the scenes just for this might be potentially killer, specially for large objects. >> Conversely, for Numeric --> numarray: > Again, you are making copies of the data. I'm not sure how numarray > handles the array protocol on consumption of the interface, so I can't > comment further. Mmmm, I've tried disabling the copy, but unfortunately enough I can't get the same figures as above: >>> t3=timeit.Timer("na=numarray.array(num,copy=0)", "import numarray; import Numeric; num=Numeric.arange(100000)") >>> t3.repeat(3,10) [1.6356601715087891, 1.6529910564422607, 1.6299269199371338] >>>t4=timeit.Timer("na=numarray.array(buffer(num),type=num.typecode(),shape=num.shape)", "import numarray; import Numeric; num=Numeric.arange(100000)") >>> t4.repeat(3,1000) [0.045578956604003906, 0.043890953063964844, 0.043296098709106445] so, for the Numeric --> numarray way, the slowdown is more than three orders of magnitude than expected (note the fewer iterations for the first repeat loop). Maybe Todd can comment more on this. Thanks! -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Tue Nov 1 09:22:53 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 09:22:53 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <1130866551.2968.69.camel@localhost.localdomain> References: <1130846099.2968.39.camel@localhost.localdomain> <436793CC.5030106@ee.byu.edu> <1130866551.2968.69.camel@localhost.localdomain> Message-ID: <4367A44F.4040808@ee.byu.edu> Francesc Altet wrote: >BTW, I'm wondering whether a False value for copy should be used as the >default instead of True. IMO, many people would want to make use of the >array protocol just to access easily the data, and making a copy() >behind the scenes just for this might be potentially killer, specially >for large objects. > > Thats exactly what the asarray function is for. It is equivalent to array except its default for copy is False. -Travis From jmiller at stsci.edu Tue Nov 1 09:28:25 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue Nov 1 09:28:25 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <1130846099.2968.39.camel@localhost.localdomain> References: <1130846099.2968.39.camel@localhost.localdomain> Message-ID: <4367A56E.5040905@stsci.edu> Francesc Altet wrote: >Hi, > >I'm trying to start using the array protocol for conversion between >Numeric <--> numarray (and newcore in the future), but I'm a bit >disappointed because of its performance. For numarray --> Numeric we >have: > > > >>>>t1=timeit.Timer("num=Numeric.array(na)", "import numarray; import >>>> >>>> >Numeric; na=numarray.arange(10)") > > >>>>t1.repeat(3,10000) >>>> >>>> >[0.59375977516174316, 0.57908082008361816, 0.56574010848999023] > > >>>>t2=timeit.Timer("num=Numeric.fromstring(na._data,typecode=na.typecode())", "import numarray; import Numeric; na=numarray.arange(10)") >>>>t2.repeat(3,10000) >>>> >>>> >[0.11653494834899902, 0.1140749454498291, 0.1141819953918457] > >i.e. the array protocol seems 5x slower than the fromstring() method. > >Conversely, for Numeric --> numarray: > > > >>>>t3=timeit.Timer("na=numarray.array(num)", "import numarray; import >>>> >>>> >Numeric;num=Numeric.arange(10)") > > >>>>t3.repeat(3,10000) >>>> >>>> >[1.3475611209869385, 1.3277668952941895, 1.3417830467224121] > > >>>>t4=timeit.Timer("na=numarray.array(buffer(num),type=num.typecode(),shape=num.shape)", "import numarray; import Numeric; num=Numeric.arange(10)") >>>>t4.repeat(3,10000) >>>> >>>> >[0.42027187347412109, 0.41690587997436523, 0.41626906394958496] > >in this case, the array protocol is 3x slower than using the buffer >interface. > >I'm wondering whether this relatively poor performance in present >implementation of the array protocol is surmountable or is an intrinsic >limitation of it. > >Thanks, > > I was working on improving this for numarray yesterday so some improvements are in CVS now. I moved some of the Python array interface properties down to C attributes and the performance ratio for my debug Python is now <2x for numarray-->Numeric. numarray's array interface for Numeric-->numarray was degenerating to fromlist(); I added array interface "consumption" support for numerical arrays by beefing up numarray's array() function. I tweaked your benchmarks a little to support profiling as well as timing and attached the result. % python arrayif_bench.py numarray-->Numeric array_if: [0.35534501075744629, 0.36865997314453125, 0.36826896667480469] numarray-->Numeric fromstring: [0.36841487884521484, 0.21085405349731445, 0.20747494697570801] Numeric-->numarray array_if: [0.73384881019592285, 0.6396629810333252, 0.60234308242797852] Numeric-->numarray buffer_if: [0.36455512046813965, 0.24601507186889648, 0.23759102821350098] -------------- next part -------------- A non-text attachment was scrubbed... Name: arrayif_bench.py Type: application/x-python Size: 1103 bytes Desc: not available URL: From Chris.Barker at noaa.gov Tue Nov 1 10:07:59 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Nov 1 10:07:59 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <43679557.70508@ee.byu.edu> References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> Message-ID: <4367AEC8.4010707@noaa.gov> Travis Oliphant wrote: > So, I guess the answer to your question is that for small arrays it is > an intrinsic limitation of the use of Python attributes in the array > protocol. IIRC, in the early discussion of the array protocol, we had talked about defining a C struct, and a set of utilities to query that struct. Now, I guess it uses Python attributes. Do I recall incorrectly, or has there been a design change, or is this a prototype implementation? I guess I'd like to see an array protocol that is as fast as fromstring(), even for small arrays, though it's probably not a big deal. Also, when writing C code to work with an array, it might be easier, as well as faster, to not have to query Python attributes. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From faltet at carabos.com Tue Nov 1 10:16:10 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Nov 1 10:16:10 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367A56E.5040905@stsci.edu> References: <1130846099.2968.39.camel@localhost.localdomain> <4367A56E.5040905@stsci.edu> Message-ID: <1130869911.2968.77.camel@localhost.localdomain> El dt 01 de 11 del 2005 a les 12:27 -0500, en/na Todd Miller va escriure: > I was working on improving this for numarray yesterday so some > improvements are in CVS now. > > I moved some of the Python array interface properties down to C > attributes and the performance ratio for my debug Python is now <2x for > numarray-->Numeric. Good! so for numarray-->Numeric we have really good performance now. > numarray's array interface for Numeric-->numarray > was degenerating to fromlist(); I added array interface "consumption" > support for numerical arrays by beefing up numarray's array() function. I'm not sure what are you saying here. You mean that in CVS there is already code for Numeric-->numarray that does not degenerate to fromlist() anymore? If this is the case, it's great news, of course. I'll try it as soon as I can. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From jmiller at stsci.edu Tue Nov 1 10:24:21 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue Nov 1 10:24:21 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <1130869911.2968.77.camel@localhost.localdomain> References: <1130846099.2968.39.camel@localhost.localdomain> <4367A56E.5040905@stsci.edu> <1130869911.2968.77.camel@localhost.localdomain> Message-ID: <4367B28F.8020602@stsci.edu> Francesc Altet wrote: >El dt 01 de 11 del 2005 a les 12:27 -0500, en/na Todd Miller va >escriure: > > > >>numarray's array interface for Numeric-->numarray >>was degenerating to fromlist(); I added array interface "consumption" >>support for numerical arrays by beefing up numarray's array() function. >> >> > >I'm not sure what are you saying here. You mean that in CVS there is >already code for Numeric-->numarray that does not degenerate to >fromlist() anymore? > Yes. I added a check in array() for suppliers of the array-interface which are not NDArrays. It only works for numerical arrays. Todd From faltet at carabos.com Tue Nov 1 10:35:25 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Nov 1 10:35:25 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367B28F.8020602@stsci.edu> References: <1130846099.2968.39.camel@localhost.localdomain> <4367A56E.5040905@stsci.edu> <1130869911.2968.77.camel@localhost.localdomain> <4367B28F.8020602@stsci.edu> Message-ID: <1130871081.2968.80.camel@localhost.localdomain> El dt 01 de 11 del 2005 a les 13:23 -0500, en/na Todd Miller va escriure: > Yes. I added a check in array() for suppliers of the array-interface > which are not NDArrays. It only works for numerical arrays. Sounds good. I suppose that you will add the same check to asarray() in order to avoid the copy, isn't it? -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Tue Nov 1 10:53:54 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 10:53:54 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367AEC8.4010707@noaa.gov> References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> Message-ID: <4367B984.2060303@ee.byu.edu> Chris Barker wrote: > Travis Oliphant wrote: > >> So, I guess the answer to your question is that for small arrays it >> is an intrinsic limitation of the use of Python attributes in the >> array protocol. > > > IIRC, in the early discussion of the array protocol, we had talked > about defining a C struct, and a set of utilities to query that > struct. Now, I guess it uses Python attributes. Do I recall > incorrectly, or has there been a design change, or is this a prototype > implementation? I guess I'd like to see an array protocol that is as > fast as fromstring(), even for small arrays, though it's probably not > a big deal. Also, when writing C code to work with an array, it might > be easier, as well as faster, to not have to query Python attribute You are correct that it would be good to have a C-protocol that did not require attribute lookups (the source of any speed difference with fromstring). Not much progress has been made on a C-version of the protocol, though. I don't know how to do it without adding something to Python itself. At SciPy 2005, I outlined my vision for how we could proceed in that direction. There is a PEP in a subversion repository at http://svn.scipy.org/svn/PEP that explains my view. Basically, I think we should push for a simple (C-based) Nd array object that is nothing more than the current C-structure of the N-d array. Then, arrays would inherit from this base class but all of Python would be able to understand and query it's C-structure. If we could also get an array interface into the typeobject table, it would be a simple thing to populate this structure even with objects that didn't inherit from the base object. I am still interested in other ideas for how to implement the array interface in C, without adding something to the type-object table (we could push for that, but it might take more political effort). -Travis From cookedm at physics.mcmaster.ca Tue Nov 1 11:02:58 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Nov 1 11:02:58 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367AEC8.4010707@noaa.gov> (Chris Barker's message of "Tue, 01 Nov 2005 10:07:04 -0800") References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> Message-ID: "Chris Barker" writes: > Travis Oliphant wrote: >> So, I guess the answer to your question is that for small arrays it >> is an intrinsic limitation of the use of Python attributes in the >> array protocol. > > IIRC, in the early discussion of the array protocol, we had talked about > defining a C struct, and a set of utilities to query that struct. Now, I > guess it uses Python attributes. Do I recall incorrectly, or has there > been a design change, or is this a prototype implementation? I guess I'd > like to see an array protocol that is as fast as fromstring(), even for > small arrays, though it's probably not a big deal. Also, when writing C > code to work with an array, it might be easier, as well as faster, to > not have to query Python attributes. I had posted an idea before: http://thread.gmane.org/gmane.comp.python.numeric.general/2159 It would still be one attribute lookup, but then everything would be C-based from there on. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Tue Nov 1 12:39:41 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 12:39:41 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> Message-ID: <4367D279.1010700@ee.byu.edu> David M. Cooke wrote: >"C > >I had posted an idea before: > >http://thread.gmane.org/gmane.comp.python.numeric.general/2159 > >It would still be one attribute lookup, but then everything would be >C-based from there on. > > Thanks for reminding us of your idea again. This is a very good idea, that I think we could add. My only question is should Py_LONG_LONG be used or Py_intptr_t? Since the array has to be in memory anyway there does not seem any advantage to using Py_LONG_LONG. I also think a flags member of the structure is useful along with a typestr instead of typecode. I would say the C-struct should look like. We could push to get something like this in Python core, I think, so this Array_Interface header was available to everybody. typedef struct { int nd; char typekind; int itemsize; int flags; Py_intptr_t *shape; Py_intptr_t *strides; Py_intptr_t offset; void *data; } PyArray_Interface From cookedm at physics.mcmaster.ca Tue Nov 1 13:02:06 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Nov 1 13:02:06 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367D279.1010700@ee.byu.edu> (Travis Oliphant's message of "Tue, 01 Nov 2005 13:39:21 -0700") References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> <4367D279.1010700@ee.byu.edu> Message-ID: Travis Oliphant writes: > David M. Cooke wrote: > >>"C >> >>I had posted an idea before: >> >>http://thread.gmane.org/gmane.comp.python.numeric.general/2159 >> >>It would still be one attribute lookup, but then everything would be >>C-based from there on. >> >> > Thanks for reminding us of your idea again. This is a very good idea, > that I think we could add. My only question is should Py_LONG_LONG be > used or Py_intptr_t? Since the array has to be in memory anyway there > does not seem any advantage to using Py_LONG_LONG. Ok, I see how that works. I probably wasn't aware of the existence of Py_intptr_t at the time :-) > I also think a flags member of the structure is useful along with a > typestr instead of typecode. The point of a typecode instead of a typestr was to make it easy for code to determine the type. Endianness would be part of the flags, and the number of bytes for the type would be in itemsize. > I would say the C-struct should look > like. We could push to get something like this in Python core, I think, > so this Array_Interface header was available to everybody. > > typedef struct { > int nd; > char typekind; > int itemsize; > int flags; > Py_intptr_t *shape; > Py_intptr_t *strides; > Py_intptr_t offset; > void *data; > } PyArray_Interface If it's in the Python core, then this is fine. If we did it ourself as an informal protocol (like the array interface spec), I'd still add the version member as a sanity check. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Chris.Barker at noaa.gov Tue Nov 1 16:04:06 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Nov 1 16:04:06 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367D279.1010700@ee.byu.edu> References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> <4367D279.1010700@ee.byu.edu> Message-ID: <43680252.50009@noaa.gov> Travis Oliphant wrote: > like. We could push to get something like this in Python core, I think, > so this Array_Interface header was available to everybody. That would be great. Until then, it would still be a tiny header that others could easily include with their code. David version number would help keep things sorted out. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Tue Nov 1 16:49:32 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 16:49:32 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <43680252.50009@noaa.gov> References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> <4367D279.1010700@ee.byu.edu> <43680252.50009@noaa.gov> Message-ID: <43680CF6.4010000@ee.byu.edu> Chris Barker wrote: > Travis Oliphant wrote: > >> like. We could push to get something like this in Python core, I >> think, so this Array_Interface header was available to everybody. > > > That would be great. Until then, it would still be a tiny header that > others could easily include with their code. David version number > would help keep things sorted out. > I've placed an updated array interface description that includes this struct-based access on http://numeric.scipy.org Somebody will add support for this in scipy soon too. -Travis From faltet at carabos.com Wed Nov 2 00:37:41 2005 From: faltet at carabos.com (Francesc Altet) Date: Wed Nov 2 00:37:41 2005 Subject: [Numpy-discussion] More on array protocol Message-ID: <1130921630.3101.11.camel@localhost.localdomain> Hi, I'm having problems in using the array protocol in numarray-->Numeric in some situations. I'm using the PyBuffer_FromReadWriteMemory C call in order to create buffers that will be later used to create numarray objects (both NumArray and CharArray) in Python space. The problem is that the array protocol implementation is doing badly this conversion. I was able to create a small example in pure python that reproduces the problem: >>> import numarray >>> from numarray import memory >>> import Numeric >>> na=numarray.array([1,2]) >>> Numeric.array(na) array([1, 2]) but ... >>> wbuf = memory.writeable_buffer(na) >>> nawb = numarray.array(wbuf,type="Int32",shape=(2,)) >>> nawb array([1, 2]) >>> Numeric.array(nawb) array([ 2, 135510112]) I'm afraid that the problem should be related with the kind of buffer that is attached to the numarray object: >>> na._data >>> nawb._data Travis, Todd, are you going to support read-write buffers or should I try to move to using the memory module in order to cope with this? For a series of reasons I'd like to keep using regular read-write buffers, but anyway... I think I'll be able to do the change if absolutely necessary. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From faltet at carabos.com Wed Nov 2 05:08:06 2005 From: faltet at carabos.com (Francesc Altet) Date: Wed Nov 2 05:08:06 2005 Subject: [Numpy-discussion] Problem with array assignment Message-ID: <200511021406.56504.faltet@carabos.com> Hi, I've recently upgraded to Numeric 24.0 and numarray 1.4.1 and I'm having a problem that did not exist before: >>> import Numeric >>> import numarray >>> Numeric.__version__ '24.0' >>> numarray.__version__ '1.4.1' >>> na=numarray.zeros((3,),'f') >>> num=Numeric.zeros((3,),'f') >>> na[...] = num Traceback (most recent call last): File "", line 1, in ? ValueError: array sizes must be consistent. but... >>> na=numarray.zeros((3,),'i') >>> num=Numeric.zeros((3,),'i') >>> na[...] = num >>> na array([0, 0, 0]) ????? However, using Numeric 23.8 and numarray 1.3.3: >>> import Numeric >>> import numarray >>> Numeric.__version__ '23.8' >>> numarray.__version__ '1.3.3' >>> na=numarray.zeros((3,),'f') >>> num=Numeric.zeros((3,),'f') >>> na[...] = num >>> na array([ 0., 0., 0.], type=Float32) I don't know if the problem is coming from Numeric or numarray, but it only happens when one mix both: >>> Numeric.__version__ '24.0' >>> numarray.__version__ '1.4.1' >>> na=numarray.zeros((3,),'f') >>> na2=numarray.zeros((3,),'f') >>> na2[...] = na >>> na2 array([ 0., 0., 0.], type=Float32) >>> num=Numeric.zeros((3,),'f') >>> num2=Numeric.zeros((3,),'f') >>> num2[...] = num >>> num array([ 0., 0., 0.],'f') Perhaps the new array protocol is involved here? -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From fonnesbeck at gmail.com Wed Nov 2 05:55:58 2005 From: fonnesbeck at gmail.com (Chris Fonnesbeck) Date: Wed Nov 2 05:55:58 2005 Subject: [Numpy-discussion] 1.4.1 build failure under OSX Message-ID: <723eb6930511020555w5ec42098na5d1502b5608990d@mail.gmail.com> Hi, Compiling 1.4.1 under OSX with either gcc 3.3 or 4.0 fails (all previous versions of numarray compiled without error). Here is the full output: Using EXTRA_COMPILE_ARGS = ['-Ddarwin'] Using external BLAS and LAPACK running config Wrote config.h In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_convmodule.c:13: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_sortmodule.c:13: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_bytesmodule.c:13: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, ... several dozen of the above, then ... Packages/LinearAlgebra2/Src/lapack_litemodule.c:12: warning: function declaration isn't a prototype Packages/LinearAlgebra2/Src/lapack_litemodule.c:17: warning: function declaration isn't a prototype Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgeev': Packages/LinearAlgebra2/Src/lapack_litemodule.c:107: warning: implicit declaration of function 'dgeev_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dsyevd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:199: warning: implicit declaration of function 'dsyevd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zheevd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:296: warning: implicit declaration of function 'zheevd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgelsd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:340: warning: implicit declaration of function 'dgelsd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgesv': Packages/LinearAlgebra2/Src/lapack_litemodule.c:378: warning: implicit declaration of function 'dgesv_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgesdd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:418: warning: implicit declaration of function 'dgesdd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgetrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:449: warning: implicit declaration of function 'dgetrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zungqr': Packages/LinearAlgebra2/Src/lapack_litemodule.c:477: warning: implicit declaration of function 'zungqr_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dorgqr': Packages/LinearAlgebra2/Src/lapack_litemodule.c:508: warning: implicit declaration of function 'dorgqr_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgeqrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:539: warning: implicit declaration of function 'zgeqrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgeqrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:570: warning: implicit declaration of function 'dgeqrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dpotrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:594: warning: implicit declaration of function 'dpotrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgeev': Packages/LinearAlgebra2/Src/lapack_litemodule.c:632: warning: implicit declaration of function 'zgeev_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgelsd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:679: warning: implicit declaration of function 'zgelsd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgesv': Packages/LinearAlgebra2/Src/lapack_litemodule.c:714: warning: implicit declaration of function 'zgesv_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgesdd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:759: warning: implicit declaration of function 'zgesdd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgetrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:794: warning: implicit declaration of function 'zgetrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zpotrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:818: warning: implicit declaration of function 'zpotrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: At top level: Packages/LinearAlgebra2/Src/lapack_litemodule.c:851: warning: function declaration isn't a prototype Packages/LinearAlgebra2/Src/lapack_litemodule.c:848: warning: 'lapack_liteError' defined but not used powerpc-apple-darwin8-gcc-4.0.0: -framework: linker input file unused because linking not done powerpc-apple-darwin8-gcc-4.0.0: vecLib: linker input file unused because linking not done In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Packages/RandomArray2/Src/ranlib.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Packages/RandomArray2/Src/ranlibmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Packages/RandomArray2/Src/com.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_dotblas.c:12: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from Src/_dotblas.c:15: /System/Library/Frameworks/vecLib.framework/Headers/cblas.h:665: warning: function declaration isn't a prototype In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/high_levelmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/element_wisemodule.c:3: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/one_dimensionalmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/numpy_compatmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/numpy_compat2.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/testlite.c:26: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list Any ideas for a solution to this problem are welcome. -- Chris Fonnesbeck Atlanta, GA From Kalimeroc5 at wanadoo.fr Wed Nov 2 07:34:07 2005 From: Kalimeroc5 at wanadoo.fr (Philou) Date: Wed Nov 2 07:34:07 2005 Subject: [Numpy-discussion] enumerate indices and values of a multidimensionnal numarray.array Message-ID: <4368DC1C.4070905@wanadoo.fr> Hi list, I'm a new subsciber to this numpy list. I have a multidimensionnal array (>2). For example: x = numarray.array([[[1,2,3],[4,5,6]],[[1,2,3],[4,5,6]]]) print x [[[1 2 3] [4 5 6]] [[1 2 3] [4 5 6]]] I need to enumerate both indices and final values (1D) of the x array. So i need: (0,0,0) 1 (0,0,1) 2 (0,0,2) 2 (0,1,0) 4 (0,1,1) 5 (0,1,2) 6 (1,0,0) 1 (1,0,1) 2 How can i do without using the x.flat() method? Thanks a lot for the answer. regards, Philippe Collet From Chris.Barker at noaa.gov Wed Nov 2 09:44:07 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Nov 2 09:44:07 2005 Subject: [Numpy-discussion] enumerate indices and values of a multidimensionnal numarray.array In-Reply-To: <4368DC1C.4070905@wanadoo.fr> References: <4368DC1C.4070905@wanadoo.fr> Message-ID: <4368FAA8.4010503@noaa.gov> There was a discussion about this last month: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2850560 You should get some good ideas from that. -Chris Philou wrote: > Hi list, > I'm a new subsciber to this numpy list. > > I have a multidimensionnal array (>2). > For example: > x = numarray.array([[[1,2,3],[4,5,6]],[[1,2,3],[4,5,6]]]) > > print x > [[[1 2 3] > [4 5 6]] > > [[1 2 3] > [4 5 6]]] > > I need to enumerate both indices and final values (1D) of the x array. > So i need: > (0,0,0) 1 > (0,0,1) 2 > (0,0,2) 2 > (0,1,0) 4 > (0,1,1) 5 > (0,1,2) 6 > (1,0,0) 1 > (1,0,1) 2 > > How can i do without using the x.flat() method? > > Thanks a lot for the answer. > regards, > Philippe Collet > > > > > ------------------------------------------------------- > SF.Net email is sponsored by: > Tame your development challenges with Apache's Geronimo App Server. > Download > it for free - -and be entered to win a 42" plasma tv or your very own > Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmiller at stsci.edu Wed Nov 2 14:17:24 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed Nov 2 14:17:24 2005 Subject: [Numpy-discussion] Problem with array assignment In-Reply-To: <200511021406.56504.faltet@carabos.com> References: <200511021406.56504.faltet@carabos.com> Message-ID: <43693AB0.7000100@stsci.edu> I added support for the newcore "array interface" to the numarray C-API and this exception is now gone. The fix/enhancement is checked into CVS. Kick it around some... Todd Francesc Altet wrote: >Hi, > >I've recently upgraded to Numeric 24.0 and numarray 1.4.1 and I'm >having a problem that did not exist before: > > > >>>>import Numeric >>>>import numarray >>>>Numeric.__version__ >>>> >>>> >'24.0' > > >>>>numarray.__version__ >>>> >>>> >'1.4.1' > > >>>>na=numarray.zeros((3,),'f') >>>>num=Numeric.zeros((3,),'f') >>>>na[...] = num >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >ValueError: array sizes must be consistent. > >but... > > > >>>>na=numarray.zeros((3,),'i') >>>>num=Numeric.zeros((3,),'i') >>>>na[...] = num >>>>na >>>> >>>> >array([0, 0, 0]) > >????? > >However, using Numeric 23.8 and numarray 1.3.3: > > > >>>>import Numeric >>>>import numarray >>>>Numeric.__version__ >>>> >>>> >'23.8' > > >>>>numarray.__version__ >>>> >>>> >'1.3.3' > > >>>>na=numarray.zeros((3,),'f') >>>>num=Numeric.zeros((3,),'f') >>>>na[...] = num >>>>na >>>> >>>> >array([ 0., 0., 0.], type=Float32) > >I don't know if the problem is coming from Numeric or numarray, but it >only happens when one mix both: > > > >>>>Numeric.__version__ >>>> >>>> >'24.0' > > >>>>numarray.__version__ >>>> >>>> >'1.4.1' > > >>>>na=numarray.zeros((3,),'f') >>>>na2=numarray.zeros((3,),'f') >>>>na2[...] = na >>>>na2 >>>> >>>> >array([ 0., 0., 0.], type=Float32) > > >>>>num=Numeric.zeros((3,),'f') >>>>num2=Numeric.zeros((3,),'f') >>>>num2[...] = num >>>>num >>>> >>>> >array([ 0., 0., 0.],'f') > >Perhaps the new array protocol is involved here? > > > From rowen at cesmail.net Wed Nov 2 14:53:09 2005 From: rowen at cesmail.net (Russell E. Owen) Date: Wed Nov 2 14:53:09 2005 Subject: [Numpy-discussion] Re: byteswap questions References: <1130838789.2968.23.camel@localhost.localdomain> Message-ID: In article <1130838789.2968.23.camel at localhost.localdomain>, Francesc Altet wrote: > El dl 31 de 10 del 2005 a les 16:53 -0800, en/na Russell E. Owen va > escriure: > > I've got a module that can send array data to ds9. It screws up on > > byteswapped data and I'm trying to fix it. > > I don't exactly know what a ds9 is, but I'm supposing here that it is > kind of an exotic computer. It's an image display program. > > I need to know if the order is bigendian or littleendian, and I can't > > find a documented way to get that, just an undocumented attribute > > byteorder. > > Yes, it's undocumented, but as far as I can tell, this works flawlessly > in numarray. I'm sure it works now. I was hoping to know if it would continue working in future versions. It'd be nice to have a documented way to get at the info. >... > In general, copy() will return you a well-behaved (native-byte ordered, > non-strided and non-offsetted) array. The manual promises that a copy will be contiguous but I didn't see a promise of native byte order. I have submitted a documentation PR on sourceforge. ... Thanks for the explanation of what byteswapped was doing. That was very helpful. To use documented interfaces (i.e. not arra._byteorder) and to avoid byteswapping the input array, I think I'm going to be stuck doing something like: import sys if sys.byteorder == 'big': isBigendian = not arr.isbyteswapped() else: isBigendian = arr.isbyteswapped() I also submitted a request for a documented direct way to get the info on sourceforge. Regards, -- Russell From faltet at carabos.com Thu Nov 3 00:33:18 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Nov 3 00:33:18 2005 Subject: [Numpy-discussion] Problem with array assignment In-Reply-To: <43693AB0.7000100@stsci.edu> References: <200511021406.56504.faltet@carabos.com> <43693AB0.7000100@stsci.edu> Message-ID: <1131007743.2968.29.camel@localhost.localdomain> El dc 02 de 11 del 2005 a les 17:16 -0500, en/na Todd Miller va escriure: > I added support for the newcore "array interface" to the numarray C-API > and this exception is now gone. The fix/enhancement is checked into > CVS. Kick it around some... Yes. This issue has been solved: >>> numarray.__version__ '1.4.2' >>> Numeric.__version__ '24.0' >>> num=Numeric.zeros((3,),'f') >>> na=numarray.ones((3,),'f') >>> na[...] = num >>> na array([ 0., 0., 0.], type=Float32) Now, I've taken the opportunity and tested the Numeric<-->numarray conversion again: >>> t1_1=timeit.Timer("num=Numeric.array(na,copy=1)","import Numeric;import numarray; na=numarray.arange(10)") >>> t1_1.repeat(3,10000) [0.25493311882019043, 0.25352001190185547, 0.25211095809936523] and this figures compared with numarray 1.4.1: >>> t1.repeat(3,10000) [0.59375977516174316, 0.57908082008361816, 0.56574010848999023] shows that you have accelerated the conversion numarray-->Numeric more than 2x. Now, the Numeric-->numarray: >>> t3_1=timeit.Timer("na=numarray.array(num,copy=1)","import Numeric;import numarray; num=Numeric.arange(10)") >>> t3_1.repeat(3,10000) [0.40311408042907715, 0.40207314491271973, 0.39954400062561035] while the 1.4.1 performance was: >>> t3.repeat(3,10000) [1.3475611209869385, 1.3277668952941895, 1.3417830467224121] which is a factor 3x of improvement, and almost the same than using the buffer interface. Very nice job, Todd! Now, let's test the conversion for large objects, with data copy and without data copy. For numarray-->Numeric: >>> t2=timeit.Timer("num=Numeric.array(na,copy=0)","import Numeric;import numarray; na=numarray.arange(100000)") >>> t2.repeat(3,100) [0.0025169849395751953, 0.0025219917297363281, 0.0024950504302978516] >>> t2_copy=timeit.Timer("num=Numeric.array(na,copy=1)","import Numeric;import numarray; na=numarray.arange(100000)") >>> t2_copy.repeat(3,100) [0.50105500221252441, 0.49400091171264648, 0.49266600608825684] So it seems like if the data copy is taking place. For Numeric-->numarray: >>> t4=timeit.Timer("na=numarray.array(num,copy=0)","import Numeric;import numarray; num=Numeric.arange(100000)") >>> t4.repeat(3,100) [0.0054900646209716797, 0.0044760704040527344, 0.0048201084136962891] >>> t4_copy=timeit.Timer("na=numarray.array(num,copy=1)","import Numeric;importnumarray; num=Numeric.arange(100000)") >>> t4_copy.repeat(3,100) [0.0063569545745849609, 0.004302978515625, 0.0042738914489746094] Ooops! the times for conversion with copy and without copy are more or less the same. Perhaps the copy is not done? It seems so: >>> num=Numeric.arange(10) >>> na=numarray.array(num,copy=0) >>> na_copy=numarray.array(num,copy=1) >>> na.info() ... data pointer: 0x08312280 (DEBUG ONLY) >>> na_copy.info() ... data pointer: 0x08312280 (DEBUG ONLY) i.e. the same data pointer. So it seems that the Numeric-->numarray is not copying the data even in the case that we are asking to do that. Other minor things (maybe this is just because you are in the middle of refactoring): >>> na._data array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) while I would expect: >>> na._data Also: >>> na Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 930, in __repr__ return array_repr(self) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 1660, in array_repr lst = arrayprint.array2string( File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line 188, in array2string separator, prefix) File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line 140, in _array2string data = _gen.ravel(a) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 918, in copy c = _gen.NDArray.copy(self) File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in copy arr._itemsize) TypeError: NA_maybeLongsFromIntTuple: must be a sequence of integers. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From rlw at stsci.edu Thu Nov 3 06:01:59 2005 From: rlw at stsci.edu (Rick White) Date: Thu Nov 3 06:01:59 2005 Subject: [Numpy-discussion] Re: byteswap questions In-Reply-To: References: <1130838789.2968.23.camel@localhost.localdomain> Message-ID: On Wed, 2 Nov 2005, Russell E. Owen wrote: > To use documented interfaces (i.e. not arra._byteorder) and to avoid > byteswapping the input array, I think I'm going to be stuck doing > something like: > > import sys > if sys.byteorder == 'big': > isBigendian = not arr.isbyteswapped() > else: > isBigendian = arr.isbyteswapped() A simpler version: isBigendian = arr.isbyteswapped() != numarray.isBigEndian From jmiller at stsci.edu Thu Nov 3 10:10:05 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 3 10:10:05 2005 Subject: [Numpy-discussion] 1.4.1 build failure under OSX In-Reply-To: <723eb6930511020555w5ec42098na5d1502b5608990d@mail.gmail.com> References: <723eb6930511020555w5ec42098na5d1502b5608990d@mail.gmail.com> Message-ID: <436A522B.2080901@stsci.edu> Chris Fonnesbeck wrote: >Hi, > >Compiling 1.4.1 under OSX with either gcc 3.3 or 4.0 fails (all >previous versions of numarray compiled without error). Here is the >full output: > > > What version of OS-X are your running? Where is the actual error? (There are a lot of warnings now on OS-X, but 1.4.1 builds fine for me.) Todd >Using EXTRA_COMPILE_ARGS = ['-Ddarwin'] >Using external BLAS and LAPACK >running config >Wrote config.h >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Src/_convmodule.c:13: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Src/_sortmodule.c:13: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Src/_bytesmodule.c:13: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > >... several dozen of the above, then ... > >Packages/LinearAlgebra2/Src/lapack_litemodule.c:12: warning: function >declaration isn't a prototype >Packages/LinearAlgebra2/Src/lapack_litemodule.c:17: warning: function >declaration isn't a prototype >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgeev': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:107: warning: implicit >declaration of function 'dgeev_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dsyevd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:199: warning: implicit >declaration of function 'dsyevd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zheevd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:296: warning: implicit >declaration of function 'zheevd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgelsd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:340: warning: implicit >declaration of function 'dgelsd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgesv': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:378: warning: implicit >declaration of function 'dgesv_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgesdd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:418: warning: implicit >declaration of function 'dgesdd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgetrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:449: warning: implicit >declaration of function 'dgetrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zungqr': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:477: warning: implicit >declaration of function 'zungqr_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dorgqr': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:508: warning: implicit >declaration of function 'dorgqr_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgeqrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:539: warning: implicit >declaration of function 'zgeqrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgeqrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:570: warning: implicit >declaration of function 'dgeqrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dpotrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:594: warning: implicit >declaration of function 'dpotrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgeev': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:632: warning: implicit >declaration of function 'zgeev_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgelsd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:679: warning: implicit >declaration of function 'zgelsd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgesv': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:714: warning: implicit >declaration of function 'zgesv_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgesdd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:759: warning: implicit >declaration of function 'zgesdd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgetrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:794: warning: implicit >declaration of function 'zgetrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zpotrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:818: warning: implicit >declaration of function 'zpotrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: At top level: >Packages/LinearAlgebra2/Src/lapack_litemodule.c:851: warning: function >declaration isn't a prototype >Packages/LinearAlgebra2/Src/lapack_litemodule.c:848: warning: >'lapack_liteError' defined but not used >powerpc-apple-darwin8-gcc-4.0.0: -framework: linker input file unused >because linking not done >powerpc-apple-darwin8-gcc-4.0.0: vecLib: linker input file unused >because linking not done >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Packages/RandomArray2/Src/ranlib.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Packages/RandomArray2/Src/ranlibmodule.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Packages/RandomArray2/Src/com.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Src/_dotblas.c:12: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from Src/_dotblas.c:15: >/System/Library/Frameworks/vecLib.framework/Headers/cblas.h:665: >warning: function declaration isn't a prototype >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/high_levelmodule.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/element_wisemodule.c:3: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/one_dimensionalmodule.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/numpy_compatmodule.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/numpy_compat2.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/testlite.c:26: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list > > >Any ideas for a solution to this problem are welcome. > >-- >Chris Fonnesbeck >Atlanta, GA > > >------------------------------------------------------- >SF.Net email is sponsored by: >Tame your development challenges with Apache's Geronimo App Server. Download >it for free - -and be entered to win a 42" plasma tv or your very own >Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From rowen at cesmail.net Thu Nov 3 12:05:57 2005 From: rowen at cesmail.net (Russell E. Owen) Date: Thu Nov 3 12:05:57 2005 Subject: [Numpy-discussion] Re: byteswap questions References: <1130838789.2968.23.camel@localhost.localdomain> Message-ID: In article , Rick White wrote: > On Wed, 2 Nov 2005, Russell E. Owen wrote: > > > To use documented interfaces (i.e. not arra._byteorder) and to avoid > > byteswapping the input array, I think I'm going to be stuck doing > > something like: > > > > import sys > > if sys.byteorder == 'big': > > isBigendian = not arr.isbyteswapped() > > else: > > isBigendian = arr.isbyteswapped() > > A simpler version: > > isBigendian = arr.isbyteswapped() != numarray.isBigEndian Cute. isBigEndian doesn't seem to be present in Numeric 23.8. I wonder if it'll be in scipy.core. Based on stunningly fast responses by Jay T Miller to most of the PRs I submitted to sourceforge: - The byteswap and byteswapped methods do just affect the data and not the byteorder flag. The doc string for byteswapped said otherwise and that is fixed in the repository. I don't know if the manual has been clarified. - To toggle the byteorder flag, use the togglebyteorder method (duh, I wish I'd seen that one). - An official way to figure out if an array is byteswapped is to use the __array_typestr__ method and look at the first character of the returned string. "<" means little-endian, ">" means big-endian. This is apparently also part of the array interface for scipy.core. I sure hope there'll be a more obvious method to call than __array_typestr__ at some point. -- Russell From jmiller at stsci.edu Thu Nov 3 12:07:21 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 3 12:07:21 2005 Subject: [Numpy-discussion] Problem with array assignment In-Reply-To: <1131007743.2968.29.camel@localhost.localdomain> References: <200511021406.56504.faltet@carabos.com> <43693AB0.7000100@stsci.edu> <1131007743.2968.29.camel@localhost.localdomain> Message-ID: <436A6D98.8080708@stsci.edu> Francesc Altet wrote: >Now, let's test the conversion for large objects, with data copy and >without data copy. > > I didn't consider that case but was thinking the semantics of the array interface would be "buffer-like" and therefore non-copying. So at the moment numarray *never* copies and completely ignores the non-sequence array() parameters for the "array interface" case. >For numarray-->Numeric: > > > >>>>t2=timeit.Timer("num=Numeric.array(na,copy=0)","import >>>> >>>> >Numeric;import numarray; na=numarray.arange(100000)") > > >>>>t2.repeat(3,100) >>>> >>>> >[0.0025169849395751953, 0.0025219917297363281, 0.0024950504302978516] > > >>>>t2_copy=timeit.Timer("num=Numeric.array(na,copy=1)","import >>>> >>>> >Numeric;import numarray; na=numarray.arange(100000)") > > >>>>t2_copy.repeat(3,100) >>>> >>>> >[0.50105500221252441, 0.49400091171264648, 0.49266600608825684] > >So it seems like if the data copy is taking place. > > > I agree and was to be able to verify this by making buffer()s out of the resulting arrays to examine their data addresses. >For Numeric-->numarray: > > > >>>>t4=timeit.Timer("na=numarray.array(num,copy=0)","import >>>> >>>> >Numeric;import numarray; num=Numeric.arange(100000)") > > >>>>t4.repeat(3,100) >>>> >>>> >[0.0054900646209716797, 0.0044760704040527344, 0.0048201084136962891] > > >>>>t4_copy=timeit.Timer("na=numarray.array(num,copy=1)","import >>>> >>>> >Numeric;importnumarray; num=Numeric.arange(100000)") > > >>>>t4_copy.repeat(3,100) >>>> >>>> >[0.0063569545745849609, 0.004302978515625, 0.0042738914489746094] > >Ooops! the times for conversion with copy and without copy are more or >less the same. Perhaps the copy is not done? It seems so: > > > >>>>num=Numeric.arange(10) >>>>na=numarray.array(num,copy=0) >>>>na_copy=numarray.array(num,copy=1) >>>>na.info() >>>> >>>> >... >data pointer: 0x08312280 (DEBUG ONLY) > > >>>>na_copy.info() >>>> >>>> >... >data pointer: 0x08312280 (DEBUG ONLY) > >i.e. the same data pointer. So it seems that the Numeric-->numarray is >not copying the data even in the case that we are asking to do that. > > Yeah, the copy=X flag is totally ignored at the moment. >Other minor things (maybe this is just because you are in the middle of >refactoring): > > > >>>>na._data >>>> >>>> >array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > >while I would expect: > > > >>>>na._data >>>> >>>> >aliasing object 0x00000000> > >Also: > > This is actually correct I think. Since a Numeric array supports the buffer protocol, it can be used as a buffer. The "memory" output shown above is the output from one particular kind of buffer; i.e. it's the repr of numarray's memory object. > > >>>>na >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >930, in __repr__ > return array_repr(self) > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >1660, in array_repr > lst = arrayprint.array2string( > File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line >188, in array2string > separator, prefix) > File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line >140, in _array2string > data = _gen.ravel(a) > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >918, in copy > c = _gen.NDArray.copy(self) > File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, >in copy > arr._itemsize) >TypeError: NA_maybeLongsFromIntTuple: must be a sequence of integers. > > For some reason Numeric was returning None for __array_strides__ and hence this message. I fixed array() so that strides are not set when __array_strides__ returns None. With the added complexity of handling the shape, type, and copy parameters in array(), I'm now seeing performance like this from CVS: numarray-->Numeric array_if: [0.40103912353515625, 0.41119098663330078, 0.41148614883422852] numarray-->Numeric fromstring: [0.24139499664306641, 0.27034401893615723, 0.21657609939575195] Numeric-->numarray array_if: [0.81009912490844727, 0.79906082153320312, 0.74358081817626953] Numeric-->numarray buffer_if: [0.27320504188537598, 0.22041201591491699, 0.22667884826660156] Todd -------------- next part -------------- A non-text attachment was scrubbed... Name: arrayif_bench.py Type: application/x-python Size: 1292 bytes Desc: not available URL: From Chris.Barker at noaa.gov Thu Nov 3 12:08:55 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Nov 3 12:08:55 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric Message-ID: <436A6E0F.6080800@noaa.gov> Hi all, I suddenly seem to have the need for working with a bunch of different existing C and C++ code, so I'm looking for a way to make it easier. I love NumPy, so I really want to use NumPy arrays My needs fall into three categories: 1) Writing custom extension code from scratch: In the past, I've done this by just using the NumPy API, but it seems that I shouldn't have to do all that book keeping myself. 2) Wrapping existing libraries: At the moment, I'd like to wrap Richard Shewchuk's triangle: http://www.cs.cmu.edu/~quake/triangle.html Which is straight C, as well as the Proj4 map projections library, also C, and there may be others. 3) Wrapping in house code, C & C++, that is under development, but I want to be able to use it from Python and C++, and also perhaps to write tests for it in Python. The options I'm looking at are: Pyrex: This seems like perhaps the easiest way to write extension code, but it doesn't do any automatic wrapping. Boost::Python: This looks like a very nice way to write extension modules, and it even appears to already include support for NumPy arrays (which ones? is that support being maintained, and will it support SciPy.base?) SWIG: The has the major advantage of automatically wrapping code. I see this as a particular strength for wrapping our in house code that is under development. In theory, once I've written a bunch of type maps, i can just re-run it whenever the code base changes. For each of these, I'd love to hear what people's experiences have been, and it would be great if anyone can point me to samples that are small but non-trivial. Other options I should consider would be great too. One more question: Should I use NumPy arrays to pass data back and forth between Python and C, or should I use the new array object/protocol instead? If so, has anyone developed examples, SWIG type maps, etc for this? Thanks for any feedback, -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Thu Nov 3 12:34:10 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Nov 3 12:34:10 2005 Subject: [Numpy-discussion] Re: byteswap questions In-Reply-To: References: <1130838789.2968.23.camel@localhost.localdomain> Message-ID: <436A73FC.3070000@ee.byu.edu> Russell E. Owen wrote: >Cute. isBigEndian doesn't seem to be present in Numeric 23.8. I wonder >if it'll be in scipy.core. > > No, in Numeric the arrays are always assumed to be in machine byte order and so sys.byteorder tells all (Numeric.LittleEndian is a Boolean that is also defined). In scipy core, arr.flags.swapped tells you if the bytes are in swapped order. Then, sys.byteorder or scipy.LittleEndian indicates the machine order. Or you use the array_interface descriptor. is_big_endian = (arr.dtypestr[0] == '>') -Travis From jmiller at stsci.edu Thu Nov 3 13:26:35 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 3 13:26:35 2005 Subject: [Numpy-discussion] More on array protocol In-Reply-To: <1130921630.3101.11.camel@localhost.localdomain> References: <1130921630.3101.11.camel@localhost.localdomain> Message-ID: <436A804A.9010402@stsci.edu> Francesc Altet wrote: >Hi, > >I'm having problems in using the array protocol in numarray-->Numeric in >some situations. I'm using the PyBuffer_FromReadWriteMemory C call in >order to create buffers that will be later used to create numarray >objects (both NumArray and CharArray) in Python space. The problem is >that the array protocol implementation is doing badly this conversion. > >I was able to create a small example in pure python that reproduces the >problem: > > > >>>>import numarray >>>>from numarray import memory >>>>import Numeric >>>>na=numarray.array([1,2]) >>>>Numeric.array(na) >>>> >>>> >array([1, 2]) > >but ... > > > >>>>wbuf = memory.writeable_buffer(na) >>>>nawb = numarray.array(wbuf,type="Int32",shape=(2,)) >>>>nawb >>>> >>>> >array([1, 2]) > > >>>>Numeric.array(nawb) >>>> >>>> >array([ 2, 135510112]) > >I'm afraid that the problem should be related with the kind of buffer >that is attached to the numarray object: > > I can reproduce garbage here using 1.4.1 but not with CVS. >>>>na._data >>>> >>>> >aliasing object 0x00000000> > > >>>>nawb._data >>>> >>>> > > >Travis, Todd, are you going to support read-write buffers or should I >try to move to using the memory module in order to cope with this? For a >series of reasons I'd like to keep using regular read-write buffers, but >anyway... I think I'll be able to do the change if absolutely necessary. > > I pretty sure this problem has been solved... let me know if you're still experiencing problems here. Todd From bsouthey at gmail.com Thu Nov 3 14:33:26 2005 From: bsouthey at gmail.com (Bruce Southey) Date: Thu Nov 3 14:33:26 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: <436A6E0F.6080800@noaa.gov> References: <436A6E0F.6080800@noaa.gov> Message-ID: Hi, I found SIWG extremely to use but it only exposes a function to Python but not to numpy. Thus it is very slow for matrix functions. So if you want speed then you will have to deal with the APIs. Regards Bruce On 11/3/05, Chris Barker wrote: > > Hi all, > > I suddenly seem to have the need for working with a bunch of different > existing C and C++ code, so I'm looking for a way to make it easier. I > love NumPy, so I really want to use NumPy arrays My needs fall into > three categories: > > 1) Writing custom extension code from scratch: > > In the past, I've done this by just using the NumPy API, but it seems > that I shouldn't have to do all that book keeping myself. > > 2) Wrapping existing libraries: > > At the moment, I'd like to wrap Richard Shewchuk's triangle: > > http://www.cs.cmu.edu/~quake/triangle.html > > Which is straight C, as well as the Proj4 map projections library, also > C, and there may be others. > > 3) Wrapping in house code, C & C++, that is under development, but I > want to be able to use it from Python and C++, and also perhaps to write > tests for it in Python. > > The options I'm looking at are: > > Pyrex: > > This seems like perhaps the easiest way to write extension code, but it > doesn't do any automatic wrapping. > > Boost::Python: > > This looks like a very nice way to write extension modules, and it even > appears to already include support for NumPy arrays (which ones? is that > support being maintained, and will it support SciPy.base?) > > SWIG: > > The has the major advantage of automatically wrapping code. I see this > as a particular strength for wrapping our in house code that is under > development. In theory, once I've written a bunch of type maps, i can > just re-run it whenever the code base changes. > > > For each of these, I'd love to hear what people's experiences have been, > and it would be great if anyone can point me to samples that are small > but non-trivial. Other options I should consider would be great too. > > > One more question: Should I use NumPy arrays to pass data back and forth > between Python and C, or should I use the new array object/protocol > instead? If so, has anyone developed examples, SWIG type maps, etc for > this? > > Thanks for any feedback, > > -Chris > > > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > ------------------------------------------------------- > SF.Net email is sponsored by: > Tame your development challenges with Apache's Geronimo App Server. > Download > it for free - -and be entered to win a 42" plasma tv or your very own > Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Nov 3 15:43:51 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Nov 3 15:43:51 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: References: <436A6E0F.6080800@noaa.gov> Message-ID: <436AA07B.5000306@noaa.gov> Bruce Southey wrote: > Hi, > I found SWIG extremely to use but it only exposes a function to Python but > not to numpy. Thus it is very slow for matrix functions. So if you want > speed then you will have to deal with the APIs. Yes, but can't I deal with them in writing typemaps, and then let SWIG do the rest of the work? I think I've seen some examples like this somewhere, but it's been a while and I need to go digging more. Anyone got one? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rays at blue-cove.com Thu Nov 3 17:11:32 2005 From: rays at blue-cove.com (RayS) Date: Thu Nov 3 17:11:32 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric Message-ID: <6.2.3.4.2.20051103170446.066db3e0@pop-server.san.rr.com> Have you looked at ctypes code generator? http://starship.python.net/crew/theller/ctypes/codegen.html "The generator converts declarations in C header files into executable Python code: enums, structs, unions, function declarations, com interfaces, and preprocessor definitions." It has progressed well in the last several months, in my experience. I did some tests a while back that found it compared well with weave etc. http://sourceforge.net/mailarchive/forum.php?thread_id=7636431&forum_id=24606 Ray Schumacher From oliphant at ee.byu.edu Thu Nov 3 22:51:52 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Nov 3 22:51:52 2005 Subject: [Numpy-discussion] Final release of Numeric is actually 24.1 Message-ID: <436B04DF.9050005@ee.byu.edu> I made a final release of Numeric (24.1). This is "really" the last release of Numeric ;-) I did it because of David Cooke's excellent array_protocol enhancement of __array_struct__. I wanted to make sure the final Numeric fully supported the array protocol including the "faster version." I also tracked down a bug today in scipy core that was inherited from Numeric. A part of me wanted to not fix the bug in Numeric so that people who need a stable platform will move to scipy core :-) But the better part of me won, and I fixed the problem and made a new Numeric release. I do hope people are encouraged to move toward scipy core, however. It is stabilizing quite rapidly. All of scipy now builds with the new scipy core, and all of (full) scipy's 1300 tests pass for both 32-bit and 64-bit systems. I will be making a release of scipy_core (probably called version 0.6 this weekend). It is still missing finished records.py, ma.py, and chararray.py modules (being worked on). When these are available I will make release of scipy core version 1.0 Best regards, -Travis From stefan at sun.ac.za Thu Nov 3 23:38:21 2005 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu Nov 3 23:38:21 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: <436AA07B.5000306@noaa.gov> References: <436A6E0F.6080800@noaa.gov> <436AA07B.5000306@noaa.gov> Message-ID: <20051104073622.GA24072@sun.ac.za> Hi Chris On Thu, Nov 03, 2005 at 03:42:51PM -0800, Chris Barker wrote: > Bruce Southey wrote: > >Hi, > >I found SWIG extremely to use but it only exposes a function to Python but > >not to numpy. Thus it is very slow for matrix functions. So if you want > >speed then you will have to deal with the APIs. > > Yes, but can't I deal with them in writing typemaps, and then let SWIG > do the rest of the work? I think I've seen some examples like this > somewhere, but it's been a while and I need to go digging more. I use SWIG and Blitz++ this way, and it works well. I modified Fernando's typemap to work with templates. See attached (does this need to be modified for Numeric3?). It is easy to create a Blitz matrix from a Numeric Array without copying data. Unfortunately, Blitz jealously guards its data (restricted pointers), so that it is not so easy to do the conversion in the other direction. If anyone knows an answer to this problem, I'd be glad to hear it. St??fan -------------- next part -------------- // -*- C++ -*- %{ #include #include #include %} namespace blitz { template class Array { %typemap(in) Array & (Array M) { int T_size = sizeof(T); blitz::TinyVector shape(0); blitz::TinyVector strides(0); int *arr_dimensions = ((PyArrayObject*)$input)->dimensions; int *arr_strides = ((PyArrayObject*)$input)->strides; for (int i = 0; i < N; i++) { shape[i] = arr_dimensions[i]; strides[i] = arr_strides[i]/T_size; } M.reference(blitz::Array((T*) ((PyArrayObject*)$input)->data, shape, strides, blitz::neverDeleteData)); $1 = &M; } }; } %template(matrix1d) blitz::Array; %template(matrix2d) blitz::Array; %template(matrix3d) blitz::Array; %template(matrix1f) blitz::Array; %template(matrix2f) blitz::Array; %template(matrix3f) blitz::Array; From oliphant at ee.byu.edu Thu Nov 3 23:44:23 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Nov 3 23:44:23 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: <20051104073622.GA24072@sun.ac.za> References: <436A6E0F.6080800@noaa.gov> <436AA07B.5000306@noaa.gov> <20051104073622.GA24072@sun.ac.za> Message-ID: <436B1104.9040500@ee.byu.edu> Stefan van der Walt wrote: >Hi Chris > >On Thu, Nov 03, 2005 at 03:42:51PM -0800, Chris Barker wrote: > > >>Bruce Southey wrote: >> >> >>>Hi, >>>I found SWIG extremely to use but it only exposes a function to Python but >>>not to numpy. Thus it is very slow for matrix functions. So if you want >>>speed then you will have to deal with the APIs. >>> >>> >>Yes, but can't I deal with them in writing typemaps, and then let SWIG >>do the rest of the work? I think I've seen some examples like this >>somewhere, but it's been a while and I need to go digging more. >> >> > >I use SWIG and Blitz++ this way, and it works well. I modified >Fernando's typemap to work with templates. See attached (does >this need to be modified for Numeric3?). > > > Only a little bit. I'll mark the changes >St??fan > > >------------------------------------------------------------------------ > >// -*- C++ -*- > >%{ >#include >#include >#include > > #include "scipy/arrayobject.h" >%} > >namespace blitz { > > template class Array { > > %typemap(in) Array & (Array M) { > int T_size = sizeof(T); > > blitz::TinyVector shape(0); > blitz::TinyVector strides(0); > > int *arr_dimensions = ((PyArrayObject*)$input)->dimensions; > int *arr_strides = ((PyArrayObject*)$input)->strides; > > for (int i = 0; i < N; i++) { > shape[i] = arr_dimensions[i]; > strides[i] = arr_strides[i]/T_size; > } > > M.reference(blitz::Array((T*) ((PyArrayObject*)$input)->data, > shape, strides, blitz::neverDeleteData)); > > $1 = &M; > } > > }; >} > >%template(matrix1d) blitz::Array; >%template(matrix2d) blitz::Array; >%template(matrix3d) blitz::Array; > >%template(matrix1f) blitz::Array; >%template(matrix2f) blitz::Array; >%template(matrix3f) blitz::Array; > > On 32-bit platforms, this code would work fine. On 64-bit platforms, you would need to change at least the arr_dimensions and arr_strides arrays to be intp (typedef to an integer the size of a pointer on the platform). -Travis From chri at botan.su.se Fri Nov 4 03:43:59 2005 From: chri at botan.su.se (Chrissy Lanctot) Date: Fri Nov 4 03:43:59 2005 Subject: [Numpy-discussion] Re: Adams to you Message-ID: <000501c5e135$229a9700$579ea8c0@clerestory> Go overpay or your Meddicat isit our Pharm ss Sh od days, Quit ing f ions - v aExpre op. Get additional information - http://starterxlq.presetey.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Fri Nov 4 06:03:45 2005 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Fri Nov 4 06:03:45 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: <436AA07B.5000306@noaa.gov> References: <436A6E0F.6080800@noaa.gov> <436AA07B.5000306@noaa.gov> Message-ID: <1131096570.30402.6.camel@localhost.localdomain> Chris, I have written an wrapper to some COIN-OR libraries for Python using Swig. In this process I have developed some simple typemaps for numarrays array (floating point arrays). It worked flawlessly. I haven't changed the code for a while, but you may want to take a look at it, specially the numarray.i file where I define the numarray typemaps: http://www.ime.usp.br/~pjssilva/software.html Good luck, Paulo From faltet at carabos.com Fri Nov 4 08:40:23 2005 From: faltet at carabos.com (Francesc Altet) Date: Fri Nov 4 08:40:23 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray Message-ID: <200511041739.40801.faltet@carabos.com> Hi, I've detected another problem in CVS numarray when converting non-contiguous Numeric objects. With numarray 1.3.3 the next works: >>> Numeric.__version__ '23.8' >>> numarray.__version__ '1.3.3' >>> num=Numeric.array([1,2,3,4],'b') >>> num2=num[::2] >>> num2.iscontiguous() 0 >>> na=numarray.array(num2) >>> na array([1, 3]) but, with CVS version of numarray: >>> numarray.__version__ '1.4.2' >>> Numeric.__version__ '24.1' >>> na=numarray.array([1,2,3,4],'b') >>> num=Numeric.array([1,2,3,4],'b') >>> num2=num[::2] >>> num2.iscontiguous() 0 >>> na=numarray.array(num2) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 376, in array a = a.astype(type) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 863, in astype return self.copy() File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 923, in copy c = _gen.NDArray.copy(self) File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in copy arr._itemsize) numarray.libnumarray.error: copy1bytes: access beyond buffer. offset=2 buffersize=2 Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From paustin at eos.ubc.ca Fri Nov 4 09:21:23 2005 From: paustin at eos.ubc.ca (Philip Austin) Date: Fri Nov 4 09:21:23 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: <436AA07B.5000306@noaa.gov> References: <436A6E0F.6080800@noaa.gov> <436AA07B.5000306@noaa.gov> Message-ID: <17259.38980.663173.839733@owl.eos.ubc.ca> I'm planning to update my boost numeric wrappers (http://www.eos.ubc.ca/research/clouds/num_util.html) to scipy_core as soon as things settle down. Here's an example of some fortran program wrapped by boost that shows a couple of nice features: i) mirroring of python types in C++, with indexing, etc., ii) transparent memory management (all increfs and decrefs are handled by boost shared pointers) and iii) docstrings. I'd be happy to contribute to a page with some SWIG and boost examples for similar code fragments. #define PY_ARRAY_UNIQUE_SYMBOL PyArrayHandle #include #include #include "boost/scoped_array.hpp" namespace { const char* rcsid = ("@(#) $Id: thermwrap.C,v 1.2 2005/09/05 22:47:57 phil Exp $:"); }; using namespace std; namespace py = boost::python; namespace nbpl = num_util; typedef int wave_unit; extern "C" { void ptqv_(float& p,float& t, float& qv, float& psp, float& tsp, float& qsp, float& thsp, float& thesp); void ptpsp_(float& p,float& t, float& qv, float& psp, float& tsp, float& qsp, float& thpt,float& thes, float& thsp, float& thesp); void mccla_(const char* atmos, float* z, float* p, float* t,float* rvden, float* o3den, float* den, int& strnlength); void thinv_(float& p,float& t,float& thsp); void theinv_(float& p,float& t, float& qsp, float& thsp, float& thesp); } py::dict ptqv(float p, float t, float qv) { float psp,tsp,qsp,thsp,thesp; ptqv_(p,t,qv,psp,tsp,qsp,thsp,thesp); py::dict result; result["psp"]=psp; result["tsp"]=tsp; result["qsp"]=qsp; result["thsp"]=thsp; result["thesp"]=thesp; return result; } py::dict ptpsp(float p, float t, float psp) { float qv,tsp,qsp,thpt,thes,thsp,thesp; ptpsp_(p,t,qv,psp,tsp,qsp,thpt,thes,thsp,thesp); py::dict result; result["qv"]=qv; result["tsp"]=tsp; result["qsp"]=qsp; result["thpt"]=thpt; result["thes"]=thes; result["thsp"]=thsp; result["thesp"]=thesp; return result; } py::dict thinv(float p, float thsp) { float t; thinv_(p,t,thsp); py::dict result; result["t"]=t; return result; } py::dict theinv(float p, float t, float the) { //t input is first guess float qv,thsp; theinv_(p,t,qv,thsp,the); py::dict result; result["t"]=t; result["qsp"]=qv; result["thsp"]=thsp; return result; } py::numeric::array mcclatchey(string atmos) { int strlength=atmos.length(); int length=33; boost::scoped_array z(new float[length]); boost::scoped_array p(new float[length]); boost::scoped_array t(new float[length]); boost::scoped_array rvden(new float[length]); boost::scoped_array o3den(new float[length]); boost::scoped_array den(new float[length]); mccla_(atmos.c_str(), z.get(), p.get(), t.get(), rvden.get(), o3den.get(), den.get(),strlength ); std::vector dims; dims.push_back(6); dims.push_back(33); py::numeric::array standsound(nbpl::makeNum(dims, PyArray_DOUBLE)); double* sndPtr=(double*) nbpl::data(standsound); int index; for(int i=0;i<33;++i){ index=i; sndPtr[index]=z[i]; index=33 + i; sndPtr[index]=p[i]; index=2*33 + i; sndPtr[index]=t[i]; index=3*33 + i; sndPtr[index]=rvden[i]; index=4*33 + i; sndPtr[index]=o3den[i]; index=5*33 + i; sndPtr[index]=den[i]; } return standsound; } BOOST_PYTHON_MODULE(thermwrap) { using namespace boost::python; import_array(); numeric::array::set_module_and_type("Numeric", "ArrayType"); scope().attr("RCSID") = rcsid; scope().attr("__doc__") = "wrappers for BettsThermo.f routines: ptqv, ptpsp"; string docstring("input: pressure (hPa), temp (K), qv (kg/kg)\n"); docstring += "output psp (hPa), tsp (K), qsp (kg/kg), thsp (K), thesp (K)"; def("ptqv", ptqv,docstring.c_str()); docstring="input: pressure (hPa), temp (K), psp (hPa)\n"; docstring += "output: qv (kg/kg), tsp (K), qsp (kg/kg), thpt (theta(p,t)), thes (K), \n"; docstring +="thsp (theta(psl,tsl), thesp equiv. potential temp.(theta-e at sl)"; def("ptpsp",ptpsp,docstring.c_str()); def("thinv",thinv,"thinv"); def("theinv",theinv,"theinv"); def("mcclatchey",mcclatchey,"mcclatchey"); } From shulee at gmail.com Fri Nov 4 12:17:27 2005 From: shulee at gmail.com (Shu Li) Date: Fri Nov 4 12:17:27 2005 Subject: [Numpy-discussion] How to install SciPy Core Message-ID: <2fd52c3d0511041216w237b0d92v2e0e6cbeb488d1da@mail.gmail.com> Hi, I am a little confused about how to install SciPy Core. I am new to SciPy so I think starting with the new core is a good idea. When I installed the core package first and then scipy, in the process of installing scipy, it showed it also installed a lot of stuff to the core directory which made me worry that part of the new core is overwritten by old core. And when I installed scipy first, then some words on the web saying some "__init__.py" file will "break the scipy" certainly worried me because I don't know what the break here means and why people are not fixing it if it is avoidable. Also even when new core(0.4.2 beta) is installed the scipy still complained not seeing "Numeric", which added to the confusion. Could anybody offer some explanation? Thanks a lot! -- best, Sam (AKA Shu Li) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Fri Nov 4 12:36:58 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 4 12:36:58 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray In-Reply-To: <200511041739.40801.faltet@carabos.com> References: <200511041739.40801.faltet@carabos.com> Message-ID: <436BC62F.1010104@stsci.edu> This turned out to be a problem with the way numarray handles Numeric's multi-segment buffer protocol. I worked around this by implementing David Cooke's __array_struct__ array interface for numarray. I'm now seeing times like these: numarray-->Numeric array_if: [0.20523881912231445, 0.19724917411804199, 0.17538094520568848] numarray-->Numeric fromstring: [0.28353691101074219, 0.21145009994506836, 0.22541189193725586] Numeric-->numarray array_if: [0.40105700492858887, 0.34915614128112793, 0.39094209671020508] Numeric-->numarray buffer_if: [0.29201602935791016, 0.25055789947509766, 0.23227095603942871] As you can see, the numarray-->Numeric array_if is now faster than fromstring() and Numeric-->numarray array_if is now getting close to the time for constructing a numarray from a buffer. Todd Francesc Altet wrote: >Hi, > >I've detected another problem in CVS numarray when converting >non-contiguous Numeric objects. > >With numarray 1.3.3 the next works: > > > >>>>Numeric.__version__ >>>> >>>> >'23.8' > > >>>>numarray.__version__ >>>> >>>> >'1.3.3' > > >>>>num=Numeric.array([1,2,3,4],'b') >>>>num2=num[::2] >>>>num2.iscontiguous() >>>> >>>> >0 > > >>>>na=numarray.array(num2) >>>>na >>>> >>>> >array([1, 3]) > >but, with CVS version of numarray: > > > >>>>numarray.__version__ >>>> >>>> >'1.4.2' > > >>>>Numeric.__version__ >>>> >>>> >'24.1' > > >>>>na=numarray.array([1,2,3,4],'b') >>>>num=Numeric.array([1,2,3,4],'b') >>>>num2=num[::2] >>>>num2.iscontiguous() >>>> >>>> >0 > > >>>>na=numarray.array(num2) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 376, >in array > a = a.astype(type) > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 863, >in astype > return self.copy() > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 923, >in copy > c = _gen.NDArray.copy(self) > File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in >copy > arr._itemsize) >numarray.libnumarray.error: copy1bytes: access beyond buffer. offset=2 >buffersize=2 > >Cheers, > > > From dd55 at cornell.edu Sat Nov 5 13:47:55 2005 From: dd55 at cornell.edu (Darren Dale) Date: Sat Nov 5 13:47:55 2005 Subject: [Numpy-discussion] question about dotblas and ACML Message-ID: <200511051647.15381.dd55@cornell.edu> I just got a new Athlon 64 bit system. I'm in the process of building Numeric-24.1 against the AMD Core Math Library. Does anyone know if it is possible to enable the use_dotblas option in customize.py with ACML? Thanks, Darren From dd55 at cornell.edu Sat Nov 5 16:31:42 2005 From: dd55 at cornell.edu (Darren Dale) Date: Sat Nov 5 16:31:42 2005 Subject: [Numpy-discussion] numpy 24.x on 64-bit Athlon Message-ID: <200511051931.20596.dd55@cornell.edu> I just built Numeric 24.0 and 24.1 against ATLAS on a 64 bit athlon system. I get the following error for both packages. python test.py ........E.................................... ====================================================================== ERROR: Test the diagonal function. ---------------------------------------------------------------------- Traceback (most recent call last): File "test.py", line 584, in testDiagonal assert_eq(Numeric.diagonal(c,1), [[2,7,4], [2,7,4]]) File "test.py", line 28, in assert_eq assert eq(a,b) File "test.py", line 23, in eq raise ValueError("sequences have different shapes:\na%s=%r\nb%s=%r" % ValueError: sequences have different shapes: a(4, 2)=array([[5, 1], [6, 2], [7, 3], [8, 4]]) b(2, 3)=[[2, 7, 4], [2, 7, 4]] I also get the following error when I run newscipy's test(5,10): bench_random (scipy.fftpack.basic.test_basic.test_fft) Fast Fourier Transform ================================================= | real input | complex input ------------------------------------------------- size | scipy | Numeric | scipy | Numeric ------------------------------------------------- 100 | 0.10Segmentation fault Has anyone else had a problem building Numeric-24.x on a 64-bit AMD system? Thanks, Darren From dd55 at cornell.edu Sat Nov 5 16:41:33 2005 From: dd55 at cornell.edu (Darren Dale) Date: Sat Nov 5 16:41:33 2005 Subject: [Numpy-discussion] numpy 24.x on 64-bit Athlon In-Reply-To: <200511051931.20596.dd55@cornell.edu> References: <200511051931.20596.dd55@cornell.edu> Message-ID: <200511051941.04464.dd55@cornell.edu> On Saturday 05 November 2005 07:31 pm, Darren Dale wrote: > I just built Numeric 24.0 and 24.1 against ATLAS on a 64 bit athlon system. [...] > I also get the following error when I run newscipy's test(5,10): > > bench_random (scipy.fftpack.basic.test_basic.test_fft) > Fast Fourier Transform > ================================================= > > | real input | complex input > > ------------------------------------------------- > size | scipy | Numeric | scipy | Numeric > ------------------------------------------------- > 100 | 0.10Segmentation fault I should have noted that I only get the segfault from Scipy with Numeric 24.1. Darren From oliphant at ee.byu.edu Sun Nov 6 21:04:06 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sun Nov 6 21:04:06 2005 Subject: [Numpy-discussion] numpy 24.x on 64-bit Athlon In-Reply-To: <200511051931.20596.dd55@cornell.edu> References: <200511051931.20596.dd55@cornell.edu> Message-ID: <436EE012.5040206@ee.byu.edu> Darren Dale wrote: >I just built Numeric 24.0 and 24.1 against ATLAS on a 64 bit athlon system. I >get the following error for both packages. > >python test.py >........E.................................... >====================================================================== >ERROR: Test the diagonal function. >---------------------------------------------------------------------- >Traceback (most recent call last): > File "test.py", line 584, in testDiagonal > assert_eq(Numeric.diagonal(c,1), [[2,7,4], [2,7,4]]) > File "test.py", line 28, in assert_eq > assert eq(a,b) > File "test.py", line 23, in eq > raise ValueError("sequences have different shapes:\na%s=%r\nb%s=%r" % >ValueError: sequences have different shapes: >a(4, 2)=array([[5, 1], > [6, 2], > [7, 3], > [8, 4]]) >b(2, 3)=[[2, 7, 4], [2, 7, 4]] > > This error is a test problem not worth bothering with. The (>2d) diagonal function was always broken under Numeric. It was changed a while ago. The test is broken, but this doesn't mean anything is wrong with Numeric. >I also get the following error when I run newscipy's test(5,10): > >bench_random (scipy.fftpack.basic.test_basic.test_fft) > Fast Fourier Transform >================================================= > | real input | complex input >------------------------------------------------- > size | scipy | Numeric | scipy | Numeric >------------------------------------------------- > > > 100 | 0.10Segmentation fault > > Curious. Did you rebuild scipy completely? -Travis From dd55 at cornell.edu Mon Nov 7 05:12:52 2005 From: dd55 at cornell.edu (Darren Dale) Date: Mon Nov 7 05:12:52 2005 Subject: [Numpy-discussion] numpy 24.x on 64-bit Athlon In-Reply-To: <436EE012.5040206@ee.byu.edu> References: <200511051931.20596.dd55@cornell.edu> <436EE012.5040206@ee.byu.edu> Message-ID: <200511070811.08485.dd55@cornell.edu> On Monday 07 November 2005 12:03 am, Travis Oliphant wrote: > >I also get the following error when I run newscipy's test(5,10): > > > >bench_random (scipy.fftpack.basic.test_basic.test_fft) > > Fast Fourier Transform > >================================================= > > > > | real input | complex input > > > >------------------------------------------------- > > size | scipy | Numeric | scipy | Numeric > >------------------------------------------------- > > > > > > 100 | 0.10Segmentation fault > > Curious. Did you rebuild scipy completely? Yes, in fact it was the first build on a new computer. I just removed my build/, and site-packages/scipy and Numeric/ directories, rebuilt and reinstalled Numeric 24.1 and newcore/newscipy and still observe the segfault. I also tried installing Scipy-0.3.2, which did not have a problem with Numeric-24.1. Darren From khinsen at cea.fr Mon Nov 7 06:50:51 2005 From: khinsen at cea.fr (khinsen at cea.fr) Date: Mon Nov 7 06:50:51 2005 Subject: [Numpy-discussion] Final release of Numeric is actually 24.1 In-Reply-To: <436B04DF.9050005@ee.byu.edu> References: <436B04DF.9050005@ee.byu.edu> Message-ID: <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> On Nov 4, 2005, at 7:51, Travis Oliphant wrote: > I do hope people are encouraged to move toward scipy core, > however. It is stabilizing Has anyone actually done this for a non-trivial package? What stops me from even trying scipy.core is the large number of incompatible changes, even though they are mostly superficial. I cannot give up Numeric compatibility because all of my user base is using Numeric, and will likely stick to Numeric at least until scipy.core is out of beta, possibly longer. In the meantime, I would have to maintain two versions of all my code (nearly all of my Python modules are concerned by some of the changes), which is not something I am looking forward to. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From managan at llnl.gov Mon Nov 7 09:04:03 2005 From: managan at llnl.gov (Rob Managan) Date: Mon Nov 7 09:04:03 2005 Subject: [Numpy-discussion] bus error in check_kurtosis Message-ID: With todays svn versions (newcore 1440, newscipy 1423) scipy.test(level=1,verbosity=2) dies with a bus error on check_kurtosis check_basic (scipy.stats.stats.test_stats.test_mean) ... ok check_ravel (scipy.stats.stats.test_stats.test_mean) ... ok check_basic (scipy.stats.stats.test_stats.test_median) ... ok check_basic (scipy.stats.stats.test_stats.test_mode) ... ok check_kurtosis (scipy.stats.stats.test_stats.test_moments)Bus error This is on Mac OSX 10.3.9, python 2.4.1, gcc 3.3. Is this a known problem? -- *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- Rob Managan email managan at llnl.gov LLNL phone: 925-423-0903 P.O. Box 808, L-095 FAX: 925-422-3389 Livermore, CA 94551-0808 From oliphant at ee.byu.edu Mon Nov 7 09:10:51 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Nov 7 09:10:51 2005 Subject: [Numpy-discussion] Final release of Numeric is actually 24.1 In-Reply-To: <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> References: <436B04DF.9050005@ee.byu.edu> <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> Message-ID: <436F8A65.4070902@ee.byu.edu> khinsen at cea.fr wrote: > On Nov 4, 2005, at 7:51, Travis Oliphant wrote: > >> I do hope people are encouraged to move toward scipy core, however. >> It is stabilizing > > > Has anyone actually done this for a non-trivial package? What stops > me from even trying scipy.core is the large number of incompatible > changes, even though they are mostly superficial. There is a module called convertcode.py (in scipy/base) that will make nearly all of the required changes. We used it to convert all of scipy over and it was pretty effective. I can understand an installed base may find it more difficult to switch. That's why we are working hard to get scipy core out of beta as soon as possible. Immediately, we are really looking for people who are willing to run their code so we can track down remaining problems. -Travis From oliphant at ee.byu.edu Mon Nov 7 11:10:20 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Nov 7 11:10:20 2005 Subject: [Numpy-discussion] Numeric 24.1 tar ball and CVS were not consistent Message-ID: <436FA65C.3010202@ee.byu.edu> I just updated the CVS of Numeric with changes that went into the Numeric 24.1 release but were (unfortunately) not committed to CVS. These changes are likely responsible for the test failures that people have experienced when testing scipy with CVS versions of Numeric. Sorry about --- too many things to think about apparently... -Travis From strawman at astraw.com Mon Nov 7 12:10:47 2005 From: strawman at astraw.com (Andrew Straw) Date: Mon Nov 7 12:10:47 2005 Subject: [Numpy-discussion] Numeric 24.1 tar ball and CVS were not consistent In-Reply-To: <436FA65C.3010202@ee.byu.edu> References: <436FA65C.3010202@ee.byu.edu> Message-ID: <436FB48E.1030705@astraw.com> Hi Travis, The bug I reported[1] was originally detected with the Numeric 24.1 tar ball, but then I "upgraded" to CVS just to be sure that it hadn't been fixed. [1]: http://www.scipy.net/pipermail/scipy-dev/2005-November/003883.html So, I don't think the issue has been fixed. Let me know if you need more detailed logs, test output, whatever. Cheers! Andrew Travis Oliphant wrote: > > I just updated the CVS of Numeric with changes that went into the > Numeric 24.1 release but were (unfortunately) not committed to CVS. > These changes are likely responsible for the test failures that people > have experienced when testing scipy with CVS versions of Numeric. > > Sorry about --- too many things to think about apparently... > > -Travis > > > > > ------------------------------------------------------- > SF.Net email is sponsored by: > Tame your development challenges with Apache's Geronimo App Server. > Download > it for free - -and be entered to win a 42" plasma tv or your very own > Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From faltet at carabos.com Mon Nov 7 23:54:36 2005 From: faltet at carabos.com (Francesc Altet) Date: Mon Nov 7 23:54:36 2005 Subject: [Numpy-discussion] Final release of Numeric is actually 24.1 In-Reply-To: <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> References: <436B04DF.9050005@ee.byu.edu> <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> Message-ID: <1131437475.3010.13.camel@localhost.localdomain> El dl 07 de 11 del 2005 a les 15:50 +0100, en/na khinsen at cea.fr va escriure: > Has anyone actually done this for a non-trivial package? What stops > me from even trying scipy.core is the large number of incompatible > changes, even though they are mostly superficial. Well, I would like to migrate PyTables to scipy.core in the medium term. My plan to not bother the users is to start supporting scipy.base by using the array protocol. That means that, although numarray will still be at the core, users will be able to send and retrieve scipy.core objects into the library with almost native speed (thanks to the array protocol). When scipy.core eventually matures enough, I'll put it at the core of PyTables, while still supporting numarray and Numeric, and the users will (hopefully) hardly notice the change, becuase they will be able to use whatever numeric library they want, and the migration will be up to them. I really think that the array protocol (that already is supported in latest Numeric and numarray, bar some small issues) is a Good Thing (tm) to consider, specially for the developers. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From khinsen at cea.fr Tue Nov 8 00:12:57 2005 From: khinsen at cea.fr (khinsen at cea.fr) Date: Tue Nov 8 00:12:57 2005 Subject: [Numpy-discussion] Final release of Numeric is actually 24.1 In-Reply-To: <1131437475.3010.13.camel@localhost.localdomain> References: <436B04DF.9050005@ee.byu.edu> <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> <1131437475.3010.13.camel@localhost.localdomain> Message-ID: On 08.11.2005, at 09:11, Francesc Altet wrote: > Well, I would like to migrate PyTables to scipy.core in the medium > term. > My plan to not bother the users is to start supporting scipy.base by > using the array protocol. That means that, although numarray will > still ... > I really think that the array protocol (that already is supported in > latest Numeric and numarray, bar some small issues) is a Good Thing > (tm) > to consider, specially for the developers. True, but it doesn't solve my problem (I think). Users of my code don't see much of an array interface, it's hidden behind application- oriented abstraction layers. That also means that users don't care much if I use Numeric, numarray, or scipy.core, as long as there are no installation problems. However, my code is necessarily tied to one specific array package because it creates all of its arrays. It also does lots of array operations that go beyond the array protocol. So I have lots of dependencies on the chosen array package in many different modules. My problem is handling the transition period while I test scipy.core and while scipy.core stabilizes. I expect to make releases of my packages during that time, so I need to keep a code base that works with Numeric. Doing development on two code bases in parallel seems like asking for trouble. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr ------------------------------------------------------------------------ ------- From pontifor at yahoo.com Tue Nov 8 07:55:22 2005 From: pontifor at yahoo.com (Stefan Kuzminski) Date: Tue Nov 8 07:55:22 2005 Subject: [Numpy-discussion] initializing new ufuncs Message-ID: <20051108154423.23553.qmail@web50602.mail.yahoo.com> Hi, I'm writing some new ufuncs, and I'm wondering where I can insert some initialization and finalization code for each ufunc invocation. I can see where it would go in the generated code, but I don't know how to do it through the code generation API. thanks, Stefan Kuzminski __________________________________ Start your day with Yahoo! - Make it your home page! http://www.yahoo.com/r/hs From mithrandir42 at web.de Tue Nov 8 22:31:37 2005 From: mithrandir42 at web.de (N. Volbers) Date: Tue Nov 8 22:31:37 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file Message-ID: <437198A3.9020705@web.de> Hello everyone, I have been using Numeric and/or numarray for a while now and I now I have a question concerning the reading and writing data to a file. Until now I had used pycdf, which is an interface to the netcdf library, to save and load my numeric arrays. However, since my application already has too many dependencies, I would like to cut down on these two and replace the saving and loading by a method intrinisc to Numeric or numarray. Pickling is not an alternative to me, because I need to have a representation that can be read by other programs (and yet is faster than ASCII). NetCDF was a good choice for that, but I would really like to not depend on it. So my questions are: (1) Are the numarray functions 'fromfile' and 'tofile' 100% portable among platforms, i.e. do they automatically recognize endian-ness and such? (2) Does it make sense to still use numarray? I know Travis would say "use scipy_core". However, for me this would provide much unneeded functionality and I have not yet found an easy way to install scipy_core (it seems to require ATLAS and such, which are not so easy to install if you don't have it prepackaged). And after all, my goal is to cut down dependencies... (3) Let me restate question (2): Will numarray still be maintained? Or is it also deprecated? What would you advice someone who just needs the array interface? And of course, (4) What solutions do you use to save/load data to files? I would deeply appreciate any help on this subject, Niklas Volbers. From jmiller at stsci.edu Wed Nov 9 02:26:49 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed Nov 9 02:26:49 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <437198A3.9020705@web.de> References: <437198A3.9020705@web.de> Message-ID: <4371CE82.9080608@stsci.edu> N. Volbers wrote: > Hello everyone, > > I have been using Numeric and/or numarray for a while now and I now I > have a question concerning the reading and writing data to a file. > > Until now I had used pycdf, which is an interface to the netcdf > library, to save and load my numeric arrays. However, since my > application already has too many dependencies, I would like to cut > down on these two and replace the saving and loading by a method > intrinisc to Numeric or numarray. > Pickling is not an alternative to me, because I need to have a > representation that can be read by other programs (and yet is faster > than ASCII). NetCDF was a good choice for that, but I would really > like to not depend on it. > > So my questions are: > > (1) Are the numarray functions 'fromfile' and 'tofile' 100% portable > among platforms, i.e. do they automatically recognize endian-ness and > such? No. But depending on how general a solution you need here, this can be fairly easy (i.e. for numerical arrays only). > (2) Does it make sense to still use numarray? Absolutely... numarray works with records and memory mapping now. But you also need to keep a shrewd eye on scipy newcore and recognize that it will most probably replace numarray over the course of the next year or two as it becomes sufficiently complete and stable to do so. There are smart things to do now if you want to use numarray: a. Use the Numeric-compatible C-API as much as possible. b. Keep an eye on the introduction of newcore compatible typenames (int32 vs Int32), keywords (dtype vs. type), and attributes and use those as you write new code in numarray. c. Use the array protocol. > I know Travis would say "use scipy_core". However, for me this would > provide much unneeded functionality and I have not yet found an easy > way to install scipy_core (it seems to require ATLAS and such, which > are not so easy to install if you don't have it prepackaged). And > after all, my goal is to cut down dependencies... > > (3) Let me restate question (2): Will numarray still be maintained? numarray will be maintained at STScI until (a) newcore is ready to replace it or (b) our budget gets cut to the point that we cannot and no one else is interested. Neither of those is guaranteed to happen, but (a) looks likely to us. STScI has the same problems with installation and dependencies so they'll have to be solved before we use newcore either. > Or is it also deprecated? What would you advice someone who just > needs the array interface? Pay careful attention to the __array_struct__ attribute described here: http://numeric.scipy.org/array_interface.html It's the easiest and best performing method to interface from C. To interface from Python you have to use more of the protocol. > And of course, > > (4) What solutions do you use to save/load data to files? numarray was written to support astronomical data processing. The dominant data format in astronomy is called FITS. STScI has another package called PyFITS which is built on numarray and exposes the FITS format to Python. Regards, Todd From cjw at sympatico.ca Wed Nov 9 06:40:10 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Nov 9 06:40:10 2005 Subject: [Numpy-discussion] Numeric3.0 - Currently available under scipy/base as version 0.4.1 (Windows) Message-ID: <437209BF.8090607@sympatico.ca> I have a package based on subclassing numarray, which is working satisfactorily, and am looking at the possibility of transferring the package to work under the revised Numeric. My feeling is that the transfer is probably feasible but that it would be premature to work on it at this time. One of the problems is the cluttered namespace, through the use of "from X import *". This is a style which is deprecated, see page 401 of Alex Martelli's /Python in a Nutshell/. Another problem, at this stage, is that many doc strings are missing and that some which exist are a little cryptic. I would like to take another look when the next win32 binaries are available. Some further thoughts on the present state of Numeric3.0 are available here . Colin W, From strawman at astraw.com Wed Nov 9 10:33:40 2005 From: strawman at astraw.com (Andrew Straw) Date: Wed Nov 9 10:33:40 2005 Subject: [Numpy-discussion] Numeric CVS __array_struct__ interface is broken on 64 bit platforms Message-ID: <437240DD.9060203@astraw.com> Hi, A couple bugs have been reported (including mine last night) which indicate a problem with the following bit of code in Numerical/Src/arrayobject.c (near line 2200) on 64 bit platforms. I think it'll be a lot faster for someone else to fix it, so I'll leave it at this for now. if (strcmp(name, "__array_struct__") == 0) { PyArrayInterface *inter; inter = (PyArrayInterface *)malloc(sizeof(PyArrayInterface)); inter->version = 2; inter->nd = self->nd; if ((inter->nd == 0) || (sizeof(int) == sizeof(Py_intptr_t))) { inter->shape = (Py_intptr_t *)self->dimensions; inter->strides = (Py_intptr_t *)self->strides; } else { int i; for (i=0; ind; i++) { inter->shape[i] = self->dimensions[i]; inter->strides[i] = self->strides[i]; } } From oliphant at ee.byu.edu Wed Nov 9 11:22:27 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Nov 9 11:22:27 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <437198A3.9020705@web.de> References: <437198A3.9020705@web.de> Message-ID: <43724C52.7030704@ee.byu.edu> N. Volbers wrote: > (2) Does it make sense to still use numarray? I know Travis would say > "use scipy_core". However, for me this would provide much unneeded > functionality and I have not yet found an easy way to install > scipy_core (it seems to require ATLAS and such, which are not so easy > to install if you don't have it prepackaged). And after all, my goal > is to cut down dependencies... > What unneeded functionality is there. SciPy Core is just a Numeric replacement. It does not NEED Atlas, it just uses it if you have it --- exactly the same as Numeric and numarray. It is a misconception to say scipy core needs any thing else but Python installed. So, let's not let that rumor kill convergence to a single package. > > (4) What solutions do you use to save/load data to files? SciPy core arrays have tofile methods and a fromfile function. They are raw reading and writing --- nothing fancy. You need to use Pickling if you want to recognize endian-ness among platforms. What is your opposition to Pickling? I think you should take a look at PyTables for more elegant solutions. -Travis From oliphant at ee.byu.edu Wed Nov 9 12:35:45 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Nov 9 12:35:45 2005 Subject: [Numpy-discussion] Numeric3.0 - Currently available under scipy/base as version 0.4.1 (Windows) In-Reply-To: <437209BF.8090607@sympatico.ca> References: <437209BF.8090607@sympatico.ca> Message-ID: <43725D8A.9090307@ee.byu.edu> Colin J. Williams wrote: > > I have a package based on subclassing numarray, which is working > satisfactorily, and am looking at the possibility of transferring the > package to work under the revised Numeric. > > My feeling is that the transfer is probably feasible but that it would > be premature to work on it at this time. That's unfortunate. The more feedback we get early on about subclassing, the better. > > One of the problems is the cluttered namespace, through the use of > "from X import *". This is a style which is deprecated, see page 401 > of Alex Martelli's /Python in a Nutshell/. You will have to be more specific about what you think is wrong. What namespace is "cluttered" exactly. Just because use is made of from X import * in one module does not mean everything is "cluttered". SciPy Core makes use of the __all__ variables to control what gets imported and usually only specific functions are imported as necessary. > Another problem, at this stage, is that many doc strings are missing > and that some which exist are a little cryptic. I would submit there are more docstrings then Numeric had. Jump in and help. The water is fine. > > I would like to take another look when the next win32 binaries are > available. There has been much improvement since the last beta. I'm trying to track down some remaining memory leaks before releasing new windows binaries. The SVN code is always available for check out and it is quite easy to build. We could always use more build testers to make sure building is going as smoothly as we believe it is. > > Some further thoughts on the present state of Numeric3.0 are available > here . Most of your comments have more to do with differences between builtin types and Python classes than anything about scipy. The type-class chasm has shrunken of late, but there are still remaining issues. These are Python issues that I believe are of little concern. I will comment on your issues that are not related to the above comment: Use of package __init__.py to create namespace. If the epydoc and pydoc tools are not respecting the __init__.py code then I would say they are broken. Using the __init__.py this way frees the user from having to understand every little detail of the package structure (which could also change as better organization is obtained in the future). Use of the from X import Y style Please give more support here. Just because one Python user advocates against it is not sufficient argument. There is an argument to be made for avoiding attribute lookups by importing the names directly in this fashion. *Methods vs functions* I agree that methods and functions are somewhat redundant. However, the functions are still needed both to support lists and especially for back-wards compatibility. Your example using take is odd (perhaps it's a bug in an old release). I get consistent behavior. One problem is you define a 1-d array in one case and a 2-d array in another. Of course the answers will be different. One difference is that the default axis argument is different. The old functions have exactly the same default axis argument as they always did, while the methods have a default of None for the axis (which means treat the array as flat). Lack of basic information in the doc strings Your examples are all class constructors. First of all, there is no special __init__ method for the ndarray because __new__ takes care of it. Second of all, the new method does need better documentation. I'm not sure where to put it, though. The array_new function is placed in the TypeObject of the array type. The __new__ attribute is pasted on by PyTypeReady. I'm not sure how to give a docstring as well. I suspect the proper thing to do is place the information in the docstring for the Type Object. -Travis From cookedm at physics.mcmaster.ca Wed Nov 9 20:41:56 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Nov 9 20:41:56 2005 Subject: [Numpy-discussion] Python egg support in Numeric 24.2 in CVS Message-ID: <4800A2A2-7B62-4B0C-9596-37956F5FCCFD@physics.mcmaster.ca> I've added support to Numeric to allow a Python egg [1] to be made easily. Basically, run $ python setup.py bdist_egg and an .egg file should be laid in dist/, which you can install with easy_install. This will only work if you have setuptools installed. I've taken steps so that usual 'setup.py build; setup.py install' will still install Numeric the old way (with a .pth file). The header files for Numeric are now also installed in a package Numeric_headers, which contains a function get_numeric_include(). This returns the directory in which the headers are installed in, so you can use it in your setup.py. The docstring for the function explains how to use it. It's modelled after get_scipy_include() in scipy.base. This way, the header files are included in the egg, and can be used. I'd appreciate it if someone with eggperience (Robert?) could take a look at it before Travis collects Numeric for 24.2. [1] http://peak.telecommunity.com/DevCenter/PythonEggs -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From bahn at atomistix.com Thu Nov 10 01:53:15 2005 From: bahn at atomistix.com (Sune Rastad Bahn) Date: Thu Nov 10 01:53:15 2005 Subject: [Numpy-discussion] Bug or feature? Message-ID: <200511101052.05164.bahn@atomistix.com> Dear devs, Thanks a lot for a very nice and extremely useful python module. I recommend all my colleagues to try you package whenever they ask for a matlab like environment. This week one of these friends came by me with an oddity relating to the RandomArray module. try the following on your linux command prompt: while true; do python -c "from RandomArray import *;seed();print randint(100000)"; done and check the output. I get a steadily but slowly decreasing series of numbers. e.g: 4013 4013 4012 4012 4012 4012 4012 4012 4012 4012 4010 4010 4010 According to the documentation seed() should initiatilize from the system clock. If one checks the seed (using get_seed) it turns out to be the case indeed, but that it is only very slowly changing (once a second), and only on the second seed. e.g python -c "from RandomArray import *;seed();print get_seed()" (113161, 3079) (113161, 3079) (113161, 3080) (113161, 3080) (113161, 3080) (113161, 3080) My best guess is that this is the cause for the behavior seen above. This could be filed as a feature, but I believe it is not the behavior people would expect. At least the documentation should emphasize that two runs started within a short time ( upto 1s) will give the exact same result (experts on pseudo random numbers will know this, but the length of the time 1sec is surprisingly long). that the first random number is slowly varying (strongly correlated) with time (this is more surprising). One fix is to take one randomnumber and throw away, since the second number is not correlated with time (except from noted above). Best, Sune -- Software Application Manager Sune Rastad Bahn Phone: +45 35 320 638 Mobile: +45 23 455 997 Email: bahn at atomistix.com From bahn at atomistix.com Thu Nov 10 03:07:49 2005 From: bahn at atomistix.com (Sune Rastad Bahn) Date: Thu Nov 10 03:07:49 2005 Subject: [Numpy-discussion] BUG! Message-ID: <200511101206.38946.bahn@atomistix.com> Dear devs, Thanks a lot for a very nice and extremely useful python module. I recommend all my colleagues to try you package whenever they ask for a matlab like environment. This week one of these friends came by me and showed me a bug in RandomArray. try the following python -c "from RandomArray import *; seed(1874764637, 1152239787);print randint(0,1000)" and notice how the result is the upper limit (1000) even though the documentation clearly states that the number should be strictly below 1000. My guess is that this is an issue with the underlying ranlib library, but perhaps someone can clarify? best, Sune -- Software Application Manager Sune Rastad Bahn Phone: +45 35 320 638 Mobile: +45 23 455 997 Email: bahn at atomistix.com From joris at ster.kuleuven.be Thu Nov 10 03:58:00 2005 From: joris at ster.kuleuven.be (Joris De Ridder) Date: Thu Nov 10 03:58:00 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <43724C52.7030704@ee.byu.edu> References: <437198A3.9020705@web.de> <43724C52.7030704@ee.byu.edu> Message-ID: <200511101257.46704.joris@ster.kuleuven.be> On Wednesday 09 November 2005 20:21, Travis Oliphant wrote: > SciPy core arrays have tofile methods and a fromfile function. They are > raw reading and writing --- nothing fancy. You need to use Pickling if > you want to recognize endian-ness among platforms. What is your > opposition to Pickling? > > I think you should take a look at PyTables for more elegant solutions. The read_array and write_array of the old scipy.io are very handy. Any chance that they will be incorporated somehow in the new scipy? PyTables is powerful but perhaps a bit overkill if you just want to read or write a few columns in ascii format. Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From arnd.baecker at web.de Thu Nov 10 04:31:46 2005 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu Nov 10 04:31:46 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <200511101257.46704.joris@ster.kuleuven.be> References: <437198A3.9020705@web.de> <43724C52.7030704@ee.byu.edu> <200511101257.46704.joris@ster.kuleuven.be> Message-ID: On Thu, 10 Nov 2005, Joris De Ridder wrote: > On Wednesday 09 November 2005 20:21, Travis Oliphant wrote: > > > SciPy core arrays have tofile methods and a fromfile function. They are > > raw reading and writing --- nothing fancy. You need to use Pickling if > > you want to recognize endian-ness among platforms. What is your > > opposition to Pickling? > > > > I think you should take a look at PyTables for more elegant solutions. > > > The read_array and write_array of the old scipy.io are very handy. > Any chance that they will be incorporated somehow in the new scipy? > PyTables is powerful but perhaps a bit overkill if you just want to > read or write a few columns in ascii format. They are already there in newscipy (as most of the other routines from "old" scipy). Best, Arnd From Mithrandir42 at web.de Thu Nov 10 04:35:10 2005 From: Mithrandir42 at web.de (Niklas Volbers) Date: Thu Nov 10 04:35:10 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file Message-ID: <1124237199@web.de> First of all, thanks for your answers. Travis Oliphant schrieb am 09.11.05 20:23:16: > What unneeded functionality is there. SciPy Core is just a Numeric > replacement. It does not NEED Atlas, it just uses it if you have it --- > exactly the same as Numeric and numarray. > It is a misconception to say scipy core needs any thing else but Python > installed. So, let's not let that rumor kill convergence to a single > package. I am sorry for this faulty assumption of mine which you have now clarified. I had assumed that ATLAS was needed, just because scipy_core would not build on my system w/o a prior install of these libraries. With the knowledge, that it _should_ work, I have just retried to build scipy_core 0.4.2 on my rather fresh installation of Slackware Linux 10.2 (python 2.4.1) and still couldn't get it to build. I have attached the log files out.txt and err.txt to this e-mail, which I got by typing $ python setup.py build 1>out.txt 2>err.txt in the scipy_core main directory. The two files are compressed into a single tar.gz archive (5k), because I do not know if 60k attachments are o.k. on this list. Maybe you can help me solve my problem? > > (4) What solutions do you use to save/load data to files? > > SciPy core arrays have tofile methods and a fromfile function. They are > raw reading and writing --- nothing fancy. You need to use Pickling if > you want to recognize endian-ness among platforms. What is your > opposition to Pickling? >From how I understood pickling, it converts the python objects to a binary representation. What I want is a data representation that is independent of python. Anyway, I now have a few possibilities to explore besides netcdf (with the alternative file formats FITS and with pytables, i.e. HDF5, proposed here). > I think you should take a look at PyTables for more elegant solutions. I wasn't sure if it will stay compatible to scipy_core, because right now it uses numarray. But from what you said I am sure that eventually pytables might also switch to scipy_core. Best regards, and thanks for your efforts, Niklas. ______________________________________________________________________ XXL-Speicher, PC-Virenschutz, Spartarife & mehr: Nur im WEB.DE Club! Jetzt gratis testen! http://freemail.web.de/home/landingpad/?mc=021130 -------------- next part -------------- A non-text attachment was scrubbed... Name: messages.tar.gz Type: application/x-gzip Size: 5241 bytes Desc: not available URL: From arnd.baecker at web.de Thu Nov 10 04:51:02 2005 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu Nov 10 04:51:02 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <1124237199@web.de> References: <1124237199@web.de> Message-ID: On Thu, 10 Nov 2005, Niklas Volbers wrote: > > First of all, thanks for your answers. > > Travis Oliphant schrieb am 09.11.05 20:23:16: > > > What unneeded functionality is there. SciPy Core is just a Numeric > > replacement. It does not NEED Atlas, it just uses it if you have it --- > > exactly the same as Numeric and numarray. > > It is a misconception to say scipy core needs any thing else but Python > > installed. So, let's not let that rumor kill convergence to a single > > package. > > I am sorry for this faulty assumption of mine which you have now clarified. > > I had assumed that ATLAS was needed, just because scipy_core would > not build on my system w/o a prior install of these libraries. > > With the knowledge, that it _should_ work, I have just retried to build > scipy_core 0.4.2 on my rather fresh installation of Slackware Linux > 10.2 (python 2.4.1) and still couldn't get it to build. I have > attached the log files out.txt and err.txt to this e-mail, which I got > by typing > > $ python setup.py build 1>out.txt 2>err.txt > > in the scipy_core main directory. The two files are compressed into a > single tar.gz archive (5k), because I do not know if 60k attachments are > o.k. on this list. > > Maybe you can help me solve my problem? Did you try a recent check out? With Travis' speed of coding scipy_core version 0.4.2.1252 seems pretty old to me I tested In [95]: scipy.__core_version__ Out[95]: '0.4.3.1440' this morning without problems. Best, Arnd From faltet at carabos.com Thu Nov 10 05:04:02 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Nov 10 05:04:02 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <1124237199@web.de> References: <1124237199@web.de> Message-ID: <200511101403.41612.faltet@carabos.com> A Dijous 10 Novembre 2005 13:34, Niklas Volbers va escriure: > > I think you should take a look at PyTables for more elegant solutions. > > I wasn't sure if it will stay compatible to scipy_core, because right now > it uses numarray. But from what you said I am sure that eventually > pytables might also switch to scipy_core. Just to make things clear, PyTables uses numarray at its core, yes, but it does support Numeric also by doing conversions in the fastest way available (in fact, many people uses PyTables just for keeping Numeric arrays, with good results). Still, a copy in memory is needed for numarray-->Numeric conversions (not for the Numeric-->numarray way), but that will change very soon, after the array protocol implementation in numarray (and Numeric) stabilizes. The next step will be to offer support for scipy.core by using the array protocol as well. And finally, we plan (probably with the introduction of PyTables 2) to switch from numarray to scipy.core when the later would be stable enough, but always with support of the triad of numarray, Numeric and scipy.core, at least for a few years. Nevertheless, you must be aware that PyTables does require the HDF5 libraries [1], so, if what you want is to get rid of the maximum of dependencies, PyTables might be effectively overkill for you. [1] http://hdf.ncsa.uiuc.edu/HDF5/ Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From bsouthey at gmail.com Thu Nov 10 07:59:59 2005 From: bsouthey at gmail.com (Bruce Southey) Date: Thu Nov 10 07:59:59 2005 Subject: [Numpy-discussion] BUG! In-Reply-To: <200511101206.38946.bahn@atomistix.com> References: <200511101206.38946.bahn@atomistix.com> Message-ID: Hi, This appears to occur in numarray (1.3.3) but not Numeric (24.0b2). If you provide the shape of the array then this does not occur. Also, please note that your other email illustrates the well known problem of using computer clock as a seed - the computer clock does not change sufficiently fast enough so the seed remains the same. Regards Bruce On 11/10/05, Sune Rastad Bahn wrote: > > Dear devs, > > Thanks a lot for a very nice and extremely useful python module. > I recommend all my colleagues to try you package whenever they ask for a > matlab like environment. This week one of these friends came by me and > showed > me a bug in RandomArray. > > try the following > > python -c "from RandomArray import *; > seed(1874764637, 1152239787);print randint(0,1000)" > > and notice how the result is the upper limit (1000) > even though the documentation clearly states that the number should be > strictly below 1000. > > My guess is that this is an issue with the underlying ranlib library, but > perhaps someone can clarify? > > best, > Sune > > -- > Software Application Manager > Sune Rastad Bahn > Phone: +45 35 320 638 > Mobile: +45 23 455 997 > Email: bahn at atomistix.com > > > ------------------------------------------------------- > SF.Net email is sponsored by: > Tame your development challenges with Apache's Geronimo App Server. > Download > it for free - -and be entered to win a 42" plasma tv or your very own > Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Thu Nov 10 08:47:53 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 10 08:47:53 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray In-Reply-To: <1131193863.2849.8.camel@localhost.localdomain> References: <200511041739.40801.faltet@carabos.com> <436BC62F.1010104@stsci.edu> <1131193863.2849.8.camel@localhost.localdomain> Message-ID: <437379A2.5070203@stsci.edu> Francesc Altet wrote: >El dv 04 de 11 del 2005 a les 15:35 -0500, en/na Todd Miller va >escriure: > > >>This turned out to be a problem with the way numarray handles Numeric's >>multi-segment buffer protocol. I worked around this by implementing >>David Cooke's __array_struct__ array interface for numarray. >> >> > >Did you commit the changes in CVS? I'm getting the same problems with >the current CVS version: > > > >>>>Numeric.__version__ >>>> >>>> >'24.1' > > >>>>import numarray >>>>numarray.__version__ >>>> >>>> >'1.4.2' > > >>>>num=Numeric.array([1,2,3,4]) >>>>numarray.array(num) >>>> >>>> >array([1, 2, 3, 4]) > > >>>>num2=num[::2] >>>>numarray.array(num2) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >380, in array > a = a.astype(type) > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >867, in astype > return self.copy() > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >927, in copy > c = _gen.NDArray.copy(self) > File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, >in copy > arr._itemsize) >numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 >buffersize=8 > > This turned out to be a bug in numarray buffer size determination... striding wasn't accounted for so the buffer appeared to be too small. It's fixed now in CVS. There's still a backward compatibility problem for old Numerics which don't implement __array_struct__ and get multi-segment buffers through __array_data__. Regards, Todd From faltet at carabos.com Thu Nov 10 09:50:44 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Nov 10 09:50:44 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray In-Reply-To: <437379A2.5070203@stsci.edu> References: <200511041739.40801.faltet@carabos.com> <1131193863.2849.8.camel@localhost.localdomain> <437379A2.5070203@stsci.edu> Message-ID: <200511101850.26502.faltet@carabos.com> Ups, I've made a cvs update and not luck yet: >>> import numarray; numarray.__version__ '1.4.2' >>> import Numeric; Numeric.__version__ '24.1' >>> num=Numeric.array([1,2,3,4]) >>> numarray.array(num) array([1, 2, 3, 4]) >>> num2=num[::2] >>> numarray.array(num2) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 380, in array a = a.astype(type) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 867, in astype return self.copy() File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 927, in copy c = _gen.NDArray.copy(self) File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in copy arr._itemsize) numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 buffersize=8 I've made a rm -rf of old numarray extension, and rebuild everything from scratch. Regards, A Dijous 10 Novembre 2005 17:47, v?reu escriure: > Francesc Altet wrote: > >El dv 04 de 11 del 2005 a les 15:35 -0500, en/na Todd Miller va > > > >escriure: > >>This turned out to be a problem with the way numarray handles Numeric's > >>multi-segment buffer protocol. I worked around this by implementing > >>David Cooke's __array_struct__ array interface for numarray. > > > >Did you commit the changes in CVS? I'm getting the same problems with > > > >the current CVS version: > >>>>Numeric.__version__ > > > >'24.1' > > > >>>>import numarray > >>>>numarray.__version__ > > > >'1.4.2' > > > >>>>num=Numeric.array([1,2,3,4]) > >>>>numarray.array(num) > > > >array([1, 2, 3, 4]) > > > >>>>num2=num[::2] > >>>>numarray.array(num2) > > > >Traceback (most recent call last): > > File "", line 1, in ? > > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line > >380, in array > > a = a.astype(type) > > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line > >867, in astype > > return self.copy() > > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line > >927, in copy > > c = _gen.NDArray.copy(self) > > File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, > >in copy > > arr._itemsize) > >numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 > >buffersize=8 > > This turned out to be a bug in numarray buffer size determination... > striding wasn't accounted for so the buffer appeared to be too small. > It's fixed now in CVS. There's still a backward compatibility problem > for old Numerics which don't implement __array_struct__ and get > multi-segment buffers through __array_data__. > > Regards, > Todd -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jmiller at stsci.edu Thu Nov 10 09:54:36 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 10 09:54:36 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray In-Reply-To: <200511101850.26502.faltet@carabos.com> References: <200511041739.40801.faltet@carabos.com> <1131193863.2849.8.camel@localhost.localdomain> <437379A2.5070203@stsci.edu> <200511101850.26502.faltet@carabos.com> Message-ID: <4373894D.5010407@stsci.edu> Francesc Altet wrote: >Ups, I've made a cvs update and not luck yet: > > > >>>>import numarray; numarray.__version__ >>>> >>>> >'1.4.2' > > >>>>import Numeric; Numeric.__version__ >>>> >>>> >'24.1' > > >>>>num=Numeric.array([1,2,3,4]) >>>>numarray.array(num) >>>> >>>> >array([1, 2, 3, 4]) > > >>>>num2=num[::2] >>>>numarray.array(num2) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 380, >in array > a = a.astype(type) > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 867, >in astype > return self.copy() > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 927, >in copy > c = _gen.NDArray.copy(self) > File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in >copy > arr._itemsize) >numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 >buffersize=8 > >I've made a rm -rf of old numarray extension, and rebuild everything >from scratch. > >Regards, > >A Dijous 10 Novembre 2005 17:47, v?reu escriure: > > >>Francesc Altet wrote: >> >> >>>El dv 04 de 11 del 2005 a les 15:35 -0500, en/na Todd Miller va >>> >>>escriure: >>> >>> >>>>This turned out to be a problem with the way numarray handles Numeric's >>>>multi-segment buffer protocol. I worked around this by implementing >>>>David Cooke's __array_struct__ array interface for numarray. >>>> >>>> >>>Did you commit the changes in CVS? I'm getting the same problems with >>> >>>the current CVS version: >>> >>> >>>>>>Numeric.__version__ >>>>>> >>>>>> >>>'24.1' >>> >>> >>> >>>>>>import numarray >>>>>>numarray.__version__ >>>>>> >>>>>> >>>'1.4.2' >>> >>> >>> >>>>>>num=Numeric.array([1,2,3,4]) >>>>>>numarray.array(num) >>>>>> >>>>>> >>>array([1, 2, 3, 4]) >>> >>> >>> >>>>>>num2=num[::2] >>>>>>numarray.array(num2) >>>>>> >>>>>> >>>Traceback (most recent call last): >>> File "", line 1, in ? >>> File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >>>380, in array >>> a = a.astype(type) >>> File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >>>867, in astype >>> return self.copy() >>> File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >>>927, in copy >>> c = _gen.NDArray.copy(self) >>> File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, >>>in copy >>> arr._itemsize) >>>numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 >>>buffersize=8 >>> >>> So you're still seeing it? Does the attached work for you? Keep in mind, anonymous CVS lags on Source Forge. >>This turned out to be a bug in numarray buffer size determination... >>striding wasn't accounted for so the buffer appeared to be too small. >>It's fixed now in CVS. There's still a backward compatibility problem >>for old Numerics which don't implement __array_struct__ and get >>multi-segment buffers through __array_data__. >> >>Regards, >>Todd >> >> > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: striding.py Type: application/x-python Size: 164 bytes Desc: not available URL: From faltet at carabos.com Thu Nov 10 10:03:25 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Nov 10 10:03:25 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray In-Reply-To: <4373894D.5010407@stsci.edu> References: <200511041739.40801.faltet@carabos.com> <200511101850.26502.faltet@carabos.com> <4373894D.5010407@stsci.edu> Message-ID: <200511101903.08516.faltet@carabos.com> A Dijous 10 Novembre 2005 18:54, Todd Miller va escriure: > So you're still seeing it? Does the attached work for you? Nope. > Keep in mind, anonymous CVS lags on Source Forge. Oh yes. I forgot about this :-/. I'll try it again tomorrow... -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From tom.denniston at alum.dartmouth.org Thu Nov 10 10:49:42 2005 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Thu Nov 10 10:49:42 2005 Subject: [Numpy-discussion] BUG! In-Reply-To: <200511101206.38946.bahn@atomistix.com> References: <200511101206.38946.bahn@atomistix.com> Message-ID: I reported this bug a whle ago and it was fixed to the best of my knowledge On 11/10/05, Sune Rastad Bahn wrote: > Dear devs, > > Thanks a lot for a very nice and extremely useful python module. > I recommend all my colleagues to try you package whenever they ask for a > matlab like environment. This week one of these friends came by me and showed > me a bug in RandomArray. > > try the following > > python -c "from RandomArray import *; > seed(1874764637, 1152239787);print randint(0,1000)" > > and notice how the result is the upper limit (1000) > even though the documentation clearly states that the number should be > strictly below 1000. > > My guess is that this is an issue with the underlying ranlib library, but > perhaps someone can clarify? > > best, > Sune > > -- > Software Application Manager > Sune Rastad Bahn > Phone: +45 35 320 638 > Mobile: +45 23 455 997 > Email: bahn at atomistix.com > > > ------------------------------------------------------- > SF.Net email is sponsored by: > Tame your development challenges with Apache's Geronimo App Server. Download > it for free - -and be entered to win a 42" plasma tv or your very own > Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From mithrandir42 at web.de Thu Nov 10 11:09:17 2005 From: mithrandir42 at web.de (N. Volbers) Date: Thu Nov 10 11:09:17 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: References: <1124237199@web.de> Message-ID: <43739BDB.9080400@web.de> >>With the knowledge, that it _should_ work, I have just retried to build >>scipy_core 0.4.2 on my rather fresh installation of Slackware Linux >>10.2 (python 2.4.1) and still couldn't get it to build. I have >>attached the log files out.txt and err.txt to this e-mail, which I got >>by typing >> >>$ python setup.py build 1>out.txt 2>err.txt >> >>in the scipy_core main directory. The two files are compressed into a >>single tar.gz archive (5k), because I do not know if 60k attachments are >>o.k. on this list. >> >>Maybe you can help me solve my problem? > > > Did you try a recent check out? > With Travis' speed of coding scipy_core version 0.4.2.1252 > seems pretty old to me > > I tested > In [95]: scipy.__core_version__ > Out[95]: '0.4.3.1440' > this morning without problems. That was a wonderful idea! I checked out 0.4.3.1456 and it works just fine. Thank you very much. BTW, I checked out the source via $ svn co http://svn.scipy.org/svn/scipy_core/branches/newcore which took me a while to find (it was on the conference paper about scipy). Best regards, Niklas. From oliphant at ee.byu.edu Thu Nov 10 16:29:38 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Nov 10 16:29:38 2005 Subject: [Numpy-discussion] SciPy SVN altered so that newscipy and newcore are now the trunk. Message-ID: <4373E5D0.5050305@ee.byu.edu> I've finished altering the subversion repository of SciPy so that the new development is taking place on the trunk of both scipy and scipy_core. The old versions are under branches named oldcore and oldscipy. Get the new repositor(y,ies) using: *Core*: svn co http://svn.scipy.org/svn/scipy_core/trunk core *Full SciPy*: svn co http://svn.scipy.org/svn/scipy/trunk scipy Doing both will place two directories named core and scipy in your current directory containing the current state of both repositories. python setup.py install should work in each directory. The Freeze is now over. I want to track down the bug that Christopher Hanley noted and another f2py-related bug before making a release, which I expect to happen by the weekend. -Travis From cookedm at physics.mcmaster.ca Thu Nov 10 19:22:03 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Nov 10 19:22:03 2005 Subject: [Numpy-discussion] SciPy SVN altered so that newscipy and newcore are now the trunk. In-Reply-To: <4373E5D0.5050305@ee.byu.edu> (Travis Oliphant's message of "Thu, 10 Nov 2005 17:29:04 -0700") References: <4373E5D0.5050305@ee.byu.edu> Message-ID: Travis Oliphant writes: > I've finished altering the subversion repository of SciPy so that the > new development is taking place on the trunk of both scipy and scipy_core. > > The old versions are under branches named oldcore and oldscipy. > > Get the new repositor(y,ies) using: > > *Core*: > svn co http://svn.scipy.org/svn/scipy_core/trunk core > > *Full SciPy*: > svn co http://svn.scipy.org/svn/scipy/trunk scipy > > Doing both will place two directories named core and scipy in your > current directory containing the current state of both repositories. Alternatively, instead of doing a full checkout, you can switch your working copies: Within your (old) newcore directory: svn sw http://svn.scipy.org/svn/scipy_core/trunk and same for full scipy. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From faltet at carabos.com Fri Nov 11 00:50:23 2005 From: faltet at carabos.com (Francesc Altet) Date: Fri Nov 11 00:50:23 2005 Subject: [Numpy-discussion] A small glitch remains in CVS numarray Message-ID: <1131700085.2968.2.camel@localhost.localdomain> Hi, I've checked CVS numarray today and yes, the problem with strided arrays seems to be gone. It does remain some small (I think) issue: >>> import Numeric;Numeric.__version__ '24.1' >>> import numarray;numarray.__version__ '1.4.2' >>> num=Numeric.array([0.,1.]) >>> na=numarray.zeros(shape=2) >>> na[...] = num Traceback (most recent call last): File "", line 1, in ? TypeError: NA_DescrFromType: unknown type: -1 and also: >>> numarray.array(num) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 359, in array a = _numarray._array_from_array_struct(sequence.__array_struct__) TypeError: NA_DescrFromType: unknown type: -1 Regards, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From jmiller at stsci.edu Sat Nov 12 04:24:39 2005 From: jmiller at stsci.edu (Todd Miller) Date: Sat Nov 12 04:24:39 2005 Subject: [Numpy-discussion] A small glitch remains in CVS numarray In-Reply-To: <1131700085.2968.2.camel@localhost.localdomain> References: <1131700085.2968.2.camel@localhost.localdomain> Message-ID: <4375DEF7.4070800@stsci.edu> Francesc Altet wrote: >Hi, > >I've checked CVS numarray today and yes, the problem with strided arrays >seems to be gone. It does remain some small (I think) issue: > > > >>>>import Numeric;Numeric.__version__ >>>> >>>> >'24.1' > > >>>>import numarray;numarray.__version__ >>>> >>>> >'1.4.2' > > >>>>num=Numeric.array([0.,1.]) >>>>na=numarray.zeros(shape=2) >>>>na[...] = num >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >TypeError: NA_DescrFromType: unknown type: -1 > >and also: > > > >>>>numarray.array(num) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >359, in array > a = _numarray._array_from_array_struct(sequence.__array_struct__) >TypeError: NA_DescrFromType: unknown type: -1 > >Regards, > > > This was a cut-and-pasto in numarray's scipy type definition table. It's fixed in CVS. Thanks, Todd From mithrandir42 at web.de Sat Nov 12 15:24:48 2005 From: mithrandir42 at web.de (N. Volbers) Date: Sat Nov 12 15:24:48 2005 Subject: [Numpy-discussion] Numeric 24.2 build fails Message-ID: <43767ABF.1060204@web.de> Sorry to once again report a build error ;-) Numeric 24.1 builds just fine, while Numeric 24.2 produces the attaced messages (out.txt) and error messages (err.txt). HTH, Niklas. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: err.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: out.txt URL: From cjw at sympatico.ca Sun Nov 13 16:54:36 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Nov 13 16:54:36 2005 Subject: [Numpy-discussion] Numeric3.0 - Currently available under scipy/base as version 0.4.1 (Windows) In-Reply-To: <43725D8A.9090307@ee.byu.edu> References: <437209BF.8090607@sympatico.ca> <43725D8A.9090307@ee.byu.edu> Message-ID: <4377E058.7010404@sympatico.ca> An HTML attachment was scrubbed... URL: From rays at blue-cove.com Mon Nov 14 08:15:59 2005 From: rays at blue-cove.com (RayS) Date: Mon Nov 14 08:15:59 2005 Subject: [Numpy-discussion] FIFO/circular buffer class in Numeric? Message-ID: <6.2.3.4.2.20051114065042.02fc6a50@pop-server.san.rr.com> Has anyone implemented a fast, generic circular array buffer in Numpy? I didn't see one in the usual searches, but I don't want to re-invent one either. I did see a number of C implementations like http://www.codeproject.com/csharp/circularbuffer.asp (http://www.gois.ws/files/attach/200209/200209201850040.zip) which could be weave'd, I suppose. I have some previous implementations I've used, but what I had in mind was making a new FIFO class with attributes of address, state, pointer, size etc., and methods like flush (if I implement it as a memmap) and other things, like queue has. The important part of the idea is that multiple threads and processes could fully access it. Currently, I will have an ADC process fill it, and other processes use it. I'll post code as I go, in any case. Ray From jeff at taupro.com Tue Nov 15 22:44:14 2005 From: jeff at taupro.com (Jeff Rush) Date: Tue Nov 15 22:44:14 2005 Subject: [Numpy-discussion] Call for a Scientific/Engineering Tutorial or Two at PyCon 2006 Message-ID: <437AD50D.1070107@taupro.com> Greetings. I'm working with the Python Conference planning group, for a conference to be held in Dallas in 2006. As you probably know from postings to python-announce, this year, for the first time, we're having a full day (Feb 23, 2006) of trainer-compensated tutorials, structured as either half-day or full-day sessions. From the feedback we received at last year's PyCon, there was a strong interest in scientific/engineering uses of Python, which has a rich set of tools and frameworks. We hope this year to dedicate one room on Tutorial Day to such topics. While I was looking over the submissions for the tutorials tonight, I noticed there were none related to science/engineering. It would be cool to have a half-day class on one of the underlying packages used across disciplines. Note that these are to be professionally-run half or full day classes, suggested with handouts or notebooks. And the trainer will receive monetary compensation, depending upon the number of attendees that register for specific tutorials. If you or someone on your staff is interested, there is more information at: http://us.pycon.org/TX2006/CallForTutorials While the submission deadline expires tonight, we have some flexibility. Jeff Rush From jdhunter at ace.bsd.uchicago.edu Wed Nov 16 07:38:03 2005 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Wed Nov 16 07:38:03 2005 Subject: [Numpy-discussion] compiling 24.2 on solaris 8 Message-ID: <87ek5gd2xq.fsf@peds-pc311.bsd.uchicago.edu> To compile numpy 24.2 on solaris 8, I needed to make the following change to Packages/RNG/Src/ranf.c #if !defined(__sgi) //int gettimeofday(struct timeval *, struct timezone *); //old int gettimeofday(); //new #endif Not sure why __sgi is defined... JDH From cjw at sympatico.ca Wed Nov 16 15:09:03 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Nov 16 15:09:03 2005 Subject: [Numpy-discussion] The file DEVELOPERS.TXT Message-ID: <437BBBF7.4050202@sympatico.ca> scipy._import_tools.py has a comment referring to DEVELOPERS.TXT but I don't find it in the Windows download. From strawman at astraw.com Thu Nov 17 03:42:06 2005 From: strawman at astraw.com (Andrew Straw) Date: Thu Nov 17 03:42:06 2005 Subject: [Numpy-discussion] numarray CVS seems to have an __array_struct__ bug Message-ID: <437C6C81.1030503@astraw.com> Hi, I think there's a bug in the new __array_struct__ code in numarray CVS. Consider an object a which implements the __array_struct__ interface. If one constructs a numarray "view" of this array using aview = numarray.asarray(a), the numarray.ndarray instance does not keep a reference to a.__array_struct__. Thus, a segfault may occur because a.data can be freed although aview may attempt to access it. In particular code like this will crash: buf = numarray.asarray( array_struct_factory() ) # buffer is freed buf[1] = 2 # attempt is made to write to the non-existant buffer, causing a segfault I've "fixed" this issue with the following, which I'm sure is not the right way, but it's all I can do at this point. diff -u -r1.130 numarraycore.py --- Lib/numarraycore.py 4 Nov 2005 20:27:11 -0000 1.130 +++ Lib/numarraycore.py 17 Nov 2005 11:38:29 -0000 @@ -357,6 +357,7 @@ a = None if hasattr(sequence, "__array_struct__"): a = _numarray._array_from_array_struct(sequence.__array_struct__) + a._hack_to_hold_reference_ = sequence.__array_struct__ elif (hasattr(sequence, "__array_shape__") and hasattr(sequence, "__array_typestr__")): typestr = sequence.__array_typestr__ And while I'm at it, can the "from distutils.command.config import log" be turned off in setup.py? I like to see those errors, warnings, and status strings whiz past... From faltet at carabos.com Thu Nov 17 04:00:01 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Nov 17 04:00:01 2005 Subject: [Numpy-discussion] numarray CVS seems to have an __array_struct__ bug In-Reply-To: <437C6C81.1030503@astraw.com> References: <437C6C81.1030503@astraw.com> Message-ID: <200511171259.11108.faltet@carabos.com> A Dijous 17 Novembre 2005 12:41, Andrew Straw va escriure: > And while I'm at it, can the "from distutils.command.config import log" > be turned off in setup.py? I like to see those errors, warnings, and > status strings whiz past... +1 ;-) -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jmiller at stsci.edu Thu Nov 17 04:15:03 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 17 04:15:03 2005 Subject: [Numpy-discussion] numarray CVS seems to have an __array_struct__ bug In-Reply-To: <437C6C81.1030503@astraw.com> References: <437C6C81.1030503@astraw.com> Message-ID: <437C742C.8020305@stsci.edu> Nice catch Andrew, and very timely! Thanks, Todd Andrew Straw wrote: >Hi, > >I think there's a bug in the new __array_struct__ code in numarray CVS. > >Consider an object a which implements the __array_struct__ interface. If >one constructs a numarray "view" of this array using aview = >numarray.asarray(a), the numarray.ndarray instance does not keep a >reference to a.__array_struct__. Thus, a segfault may occur because >a.data can be freed although aview may attempt to access it. > >In particular code like this will crash: > >buf = numarray.asarray( array_struct_factory() ) # buffer is freed >buf[1] = 2 # attempt is made to write to the non-existant buffer, >causing a segfault > >I've "fixed" this issue with the following, which I'm sure is not the >right way, but it's all I can do at this point. > >diff -u -r1.130 numarraycore.py >--- Lib/numarraycore.py 4 Nov 2005 20:27:11 -0000 1.130 >+++ Lib/numarraycore.py 17 Nov 2005 11:38:29 -0000 >@@ -357,6 +357,7 @@ > a = None > if hasattr(sequence, "__array_struct__"): > a = >_numarray._array_from_array_struct(sequence.__array_struct__) >+ a._hack_to_hold_reference_ = sequence.__array_struct__ > elif (hasattr(sequence, "__array_shape__") and > hasattr(sequence, "__array_typestr__")): > typestr = sequence.__array_typestr__ > > >And while I'm at it, can the "from distutils.command.config import log" >be turned off in setup.py? I like to see those errors, warnings, and >status strings whiz past... > > >------------------------------------------------------- >This SF.Net email is sponsored by the JBoss Inc. Get Certified Today >Register for a JBoss Training Course. Free Certification Exam >for All Training Attendees Through End of 2005. For more info visit: >http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From curzio.basso at gmail.com Sat Nov 19 05:34:05 2005 From: curzio.basso at gmail.com (Curzio Basso) Date: Sat Nov 19 05:34:05 2005 Subject: [Numpy-discussion] compilation error on OSX 10.4 Message-ID: Hi all, I get the following error compiling an extension module using numarray 1.4.1 on Mac OSX 10.4: ----- building 'fmlio' extension gcc-3.3 -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DLINUX -I/home/basso/usr/include/python -I/sw/include/python2.4 -c src/fmlio.cpp -o build/temp.darwin-8.3.0-Power_Macintosh-2.4/src/fmlio.o In file included from /sw/include/python2.4/numarray/libnumarray.h:17, from /sw/include/python2.4/numarray/numarray.h:6, from src/fmlio.h:10, from src/fmlio.cpp:2: /sw/include/python2.4/numarray/nummacro.h:27: error: parse error before `;' token error: command 'gcc-3.3' failed with exit status 1 ----- the same happens with gcc-4. On Gentoo I can compile without problems. Did anyone else get a similar problem? Ah, and numarray.h is included right after Python.h. cheers curzio From curzio.basso at gmail.com Sat Nov 19 06:00:13 2005 From: curzio.basso at gmail.com (Curzio Basso) Date: Sat Nov 19 06:00:13 2005 Subject: [Numpy-discussion] Re: compilation error on OSX 10.4 In-Reply-To: References: Message-ID: > the same happens with gcc-4. On Gentoo I can compile without problems. I should correct myself: on Gentoo it compiles but I have version 1.3.1. curzio From curzio.basso at gmail.com Sat Nov 19 09:49:01 2005 From: curzio.basso at gmail.com (Curzio Basso) Date: Sat Nov 19 09:49:01 2005 Subject: [Numpy-discussion] Re: compilation error on OSX 10.4 In-Reply-To: References: Message-ID: > I should correct myself: on Gentoo it compiles but I have version 1.3.1. First of all, compilation fails also on Gentoo with version 1.4.1. Second, I can reproduce the result trying to compile the following code: -------------- #include #include "numarray/libnumarray.h" int main() { return 0; } ------------- Compiling the above code as C++, I get the error already reported, while compiling as C works fine. The reason is that at line 27 in nummacro.h the C++ keyword "operator" is used. Would it be possible to replace it with something else? like "operator_"? By the way, using extern "C" {...} does not help. cheers curzio From smaret at umich.edu Sat Nov 19 10:53:01 2005 From: smaret at umich.edu (Sebastien Maret) Date: Sat Nov 19 10:53:01 2005 Subject: [Numpy-discussion] Re: compilation error on OSX 10.4 References: Message-ID: Curzio Basso writes: > I get the following error compiling an extension module using numarray > 1.4.1 on Mac OSX 10.4: > ----- > building 'fmlio' extension > gcc-3.3 -fno-strict-aliasing -Wno-long-double -no-cpp-precomp > -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DLINUX > -I/home/basso/usr/include/python -I/sw/include/python2.4 -c > src/fmlio.cpp -o > build/temp.darwin-8.3.0-Power_Macintosh-2.4/src/fmlio.o > In file included from /sw/include/python2.4/numarray/libnumarray.h:17, > from /sw/include/python2.4/numarray/numarray.h:6, > from src/fmlio.h:10, > from src/fmlio.cpp:2: > /sw/include/python2.4/numarray/nummacro.h:27: error: parse error before `;' > token There is a bug report for this problem: http://sourceforge.net/tracker/?func=detail&atid=450446&aid=1350954&group_id=1369 The problem has been fixed in FreeBSD: http://www.freebsd.org/cgi/cvsweb.cgi/ports/math/py-numarray/files/patch-Include::numarray::nummacro.h Maybe this patch could be applied to numarray ? S?bastien From jdawe at u.washington.edu Sat Nov 19 11:31:02 2005 From: jdawe at u.washington.edu (Jordan Dawe) Date: Sat Nov 19 11:31:02 2005 Subject: [Numpy-discussion] scipy cygwin compilation error Message-ID: <437F7D48.9070406@u.washington.edu> I just tried to install scipy under cygwin using both 0.61 and the current svn. I get this: gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cygwin-1.5.18-i686-2.4/scipy/base/umath.dll build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UBYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2215: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `USHORT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2233: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UINT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2251: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `ULONG_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2269: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `BYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2305: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o:/usr/local/core/build/src/scipy/base/src/umathmodule.c:2325: more undefined references to `_feraiseexcept' follow build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_checkfperr': /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_clearfperr': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_GenericFunction': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `construct_reduce': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' collect2: ld returned 1 exit status build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UBYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2215: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `USHORT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2233: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UINT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2251: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `ULONG_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2269: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `BYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2305: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o:/usr/local/core/build/src/scipy/base/src/umathmodule.c:2325: more undefined references to `_feraiseexcept' follow build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_checkfperr': /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_clearfperr': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_GenericFunction': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `construct_reduce': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' collect2: ld returned 1 exit status error: Command "gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cygwin-1.5.18-i686-2.4/scipy/base/umath.dll" failed with exit status 1 Any suggestions? Jordan From oliphant at ee.byu.edu Sun Nov 20 01:04:01 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sun Nov 20 01:04:01 2005 Subject: [Numpy-discussion] Memory mapped files in scipy core Message-ID: <43803BC9.6050103@ee.byu.edu> I would appreciate understanding typically use cases for memory-mapped files. I am not sure I understand why certain choices were made for numarray's memmap and memmap slice classes. They seem like a lot of "extra" stuff and I'm not sure what all that stuff is for. Rather than just copy these over, I would like to understand what people typically want to do with memory-mapped files to see if scipy core doesn't already provide that. For example, write now I can open a file, use mmap to obtain a memory map object and then pass that object into frombuffer in scipy_core to get an ndarray whose memory maps a file on disk. Now, this ndarray can be sliced and indexed and manipulated all the while referring to the file on disk (well technically, I suppose, the memory-mapped object would need to be flushed to synchronize). Now, I could see wanting to make the process of opening the file, getting the mmap object and setting it's buffer to the array object a little easier. Thus, a simple memmap class would be a useful construct -- I could even see it inheriting from the ndarray directly and adding a few methods. I guess I just don't see why one would care about a memory-mapped slice object, when the mmaparray sub-class would be perfectly useful. On a related, but orthogonal note: My understanding is that using memory-mapped files for *very* large files will require modification to the mmap module in Python --- something I think we should push. One part of that process would be to add the C-struct array interface to the mmap module and the buffer object -- perhaps this is how we get the array interface into Python quickly. Then, if we could make a base-type mmap that did not use the buffer interface or the sequence interface (similar to the bigndarray in scipy_core) and therefore by-passed the problems with Python in those areas, then the current mmap object could inherit from the base class and provide current functionality while still exposing the array interface for access to >2GB files on 64-bit systems. Who would like to take up the ball for modifying mmap in Python in this fashion? -Travis From egsl1930 at yahoo.es Sun Nov 20 19:43:00 2005 From: egsl1930 at yahoo.es (Eduardo Stark) Date: Sun Nov 20 19:43:00 2005 Subject: [Numpy-discussion] Real life examples Message-ID: <20051121034233.90046.qmail@web26901.mail.ukl.yahoo.com> Dear naumarray and numpy community: Could you tell me where can I get real life examples to apply numarray. I am a Python and numarray begineer. Thanks Eduardo Stark --------------------------------- Correo Yahoo! Comprueba qu? es nuevo, aqu? http://correo.yahoo.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egsl1930 at yahoo.es Sun Nov 20 19:43:01 2005 From: egsl1930 at yahoo.es (Eduardo Stark) Date: Sun Nov 20 19:43:01 2005 Subject: [Numpy-discussion] Real life examples Message-ID: <20051121034234.55990.qmail@web26909.mail.ukl.yahoo.com> Dear naumarray and numpy community: Could you tell me where can I get real life examples to apply numarray. I am a Python and numarray begineer. Thanks Eduardo Stark --------------------------------- Correo Yahoo! Comprueba qu? es nuevo, aqu? http://correo.yahoo.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at enthought.com Mon Nov 21 06:46:01 2005 From: travis at enthought.com (Travis N. Vaught) Date: Mon Nov 21 06:46:01 2005 Subject: [Numpy-discussion] OT: Enthought IT Admin Job Posting Message-ID: <4381DD6D.9030202@enthought.com> All, I'm posting this here in the hopes that someone that is passionate about python/numeric/scipy will "answer the call": `Enthought, Inc. `__ (Austin, TX, USA) ==================================================================== **Job Description**: Position: IT Administrator --------------------------- Enthought is looking for an exceptional IT Administrator to manage the IT infrastructure for it's Austin, TX offices. This person will have a passion for supporting software development tools and environments. The target server platforms include Linux (RedHat and Fedora), Windows XP, and Solaris. Workstations are a mix of Windows and Linux. We're looking for an Admin that focuses on supporting software developers. Desired Skills and Capabilities: * B.S. in Computer Science of other related field (preferably not MIS) * 5+ Years Experience in Enterprise IT Administration Role * Ability to program or script in a programming language or shell - (Python, Perl and small C programs) * Ability to solve problems quickly and completely * Strong inter-personal and communication skills as well as a team player in a group of highly talented software developers * Willingness to pitch in wherever needed * Interested in making developer's lives easier Duties: * Perform installation, patching and other server maintenance to ensure security and stability of server infrastructure. * Maintain core infrastructure technologies such as Firewall, VPN, apache web server, mail server, NIS, NFS, DNS, DHCP, SSH, network-wide backups, RAID and Samba * Identify routine tasks and automate through shell/Python scripting * Perform on-call duties and out of hours maintenance as necessary * Configure Nortel phone system * Build RPMs for RH/FC Linux * Using development tools on Linux such as gcc/g++, make, autoconf, etc. * Working with Python and building & installing Python packages using distutils * Support developer tools such as SVN repositories, bug trackers, Software Project Management utilities Company: -------- Enthought is a scientific computing company located in downtown Austin, Texas. Founded in 2001, it has grown nearly 100% per year both in staff and revenue to become a stable and profitable technology company. This growth has been possible because of Enthought?s talented team and because of our commitment to developing quality software. We strive to combine advanced algorithm development with modern software practices, such as component based architectures, application scripting, and intuitive user interface design. We take a holistic approach to software development, in which architects and developers team with technical writers, human factors specialists, and project managers (always highly technical individuals) to develop a complete solution for our customers. Much of our work is based on the Python programming language, and we are actively engaged in open source development (www.scipy.org ). We?re lucky enough to work on interesting problems and are looking for talented people to join us. Some of our current efforts are in the areas of geophysics, electromagnetics, fluid dynamics, micro-rheology, CAD, 2-D and 3-D visualization, and others. All of these tools are developed as plug-ins into our Envisage "Scientific IDE" framework. * **Contact**: Travis N. Vaught, CEO * **E-mail contact**: jobs at enthought.com * **Web**: http://www.enthought.com/careers.htm -- ........................ Travis N. Vaught CEO Enthought, Inc. http://www.enthought.com ........................ From pearu at cens.ioc.ee Mon Nov 21 07:06:00 2005 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Mon Nov 21 07:06:00 2005 Subject: [Numpy-discussion] The file DEVELOPERS.TXT In-Reply-To: <437BBBF7.4050202@sympatico.ca> Message-ID: On Wed, 16 Nov 2005, Colin J. Williams wrote: > scipy._import_tools.py has a comment referring to DEVELOPERS.TXT but I > don't find it in the Windows download. This is fixed in scipy core SVN repositoy. Up-to-date DEVELOPERS.TXT (that used to reside in scipy source directory) can be now found as scipy/doc/DISTUTILS.txt file. Pearu From jmiller at stsci.edu Mon Nov 21 07:15:00 2005 From: jmiller at stsci.edu (Todd Miller) Date: Mon Nov 21 07:15:00 2005 Subject: [Numpy-discussion] Memory mapped files in scipy core In-Reply-To: <43803BC9.6050103@ee.byu.edu> References: <43803BC9.6050103@ee.byu.edu> Message-ID: <4381E3EA.9040500@stsci.edu> Travis Oliphant wrote: > > I would appreciate understanding typically use cases for memory-mapped > files. I am not sure I understand why certain choices were made for > numarray's memmap and memmap slice classes. They seem like a lot of > "extra" stuff and I'm not sure what all that stuff is for. > > Rather than just copy these over, I would like to understand what > people typically want to do with memory-mapped files to see if scipy > core doesn't already provide that. > > For example, write now I can open a file, use mmap to obtain a memory > map object and then pass that object into frombuffer in scipy_core to > get an ndarray whose memory maps a file on disk. > Now, this ndarray can be sliced and indexed and manipulated all the > while referring to the file on disk (well technically, I suppose, the > memory-mapped object would need to be flushed to synchronize). > Now, I could see wanting to make the process of opening the file, > getting the mmap object and setting it's buffer to the array object a > little easier. Thus, a simple memmap class would be a useful > construct -- I could even see it inheriting from the ndarray directly > and adding a few methods. I guess I just don't see why one would > care about a memory-mapped slice object, when the mmaparray sub-class > would be perfectly useful. > There are a few extra capabilities which are supported in numarray's memmap: 1. slice insertion 2. slice deletion 3. memmap based array resizing Each of these things implicitly changes the layout of the underlying file. Whether or not these features get used or justify the complexity is another matter. Because of 32-bit address space exhaustion and swap file issues, memory mapping was a disappointment at STSCI so I'm doubtful we used these features ourselves. Regards, Todd > On a related, but orthogonal note: > > My understanding is that using memory-mapped files for *very* large > files will require modification to the mmap module in Python --- > something I think we should push. One part of that process would be > to add the C-struct array interface to the mmap module and the buffer > object -- perhaps this is how we get the array interface into Python > quickly. Then, if we could make a base-type mmap that did not use > the buffer interface or the sequence interface (similar to the > bigndarray in scipy_core) and therefore by-passed the problems with > Python in those areas, then the current mmap object could inherit from > the base class and provide current functionality while still exposing > the array interface for access to >2GB files on 64-bit systems. > > Who would like to take up the ball for modifying mmap in Python in > this fashion? > > -Travis > > > > ------------------------------------------------------- > This SF.Net email is sponsored by the JBoss Inc. Get Certified Today > Register for a JBoss Training Course. Free Certification Exam > for All Training Attendees Through End of 2005. For more info visit: > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From a.h.jaffe at gmail.com Mon Nov 21 09:19:07 2005 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Mon Nov 21 09:19:07 2005 Subject: [Numpy-discussion] Problems with linalg.cholesky? Message-ID: [Originally posted to g.c.p.user...] Hi all, In the newest incarnation of scipy_core, I am having trouble with the cholesky(a) routine: /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy/linalg/basic_lite.py in cholesky_decomposition(a) 115 else: 116 lapack_routine = lapack_lite.dpotrf --> 117 results = lapack_routine('L', n, a, m, 0) 118 if results['info'] > 0: 119 raise LinAlgError, 'Matrix is not positive definite - Cholesky decomposition cannot be computed' LapackError: Parameter a is not contiguous in lapack_lite.dpotrf But this isn't true; I get this error even when I pass trivial and contiguous matrices such as the output of identity(). Other linalg routines (included complicated ones like singular_value_decomp) seem to work fine. Any ideas? Andrew From oliphant.travis at ieee.org Mon Nov 21 13:13:02 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Nov 21 13:13:02 2005 Subject: [Numpy-discussion] Re: Memory mapped files in scipy core In-Reply-To: <4381E3EA.9040500@stsci.edu> References: <43803BC9.6050103@ee.byu.edu> <4381E3EA.9040500@stsci.edu> Message-ID: > There are a few extra capabilities which are supported in numarray's > memmap: > > 1. slice insertion > 2. slice deletion > 3. memmap based array resizing Thanks for the extra explanation. I could see how these might require more stuff under the hood. -Travis From joostvanevert at gmail.com Tue Nov 22 05:58:10 2005 From: joostvanevert at gmail.com (Joost van Evert) Date: Tue Nov 22 05:58:10 2005 Subject: [Numpy-discussion] import problem In-Reply-To: References: Message-ID: <1132667852.32469.9.camel@inpc93.et.tudelft.nl> Dear list, does anyone understand importing modules? I want to import a new version of Numeric(24.2) that was installed in my home directory, but the old one in /usr/lib/python/site-packages gets imported. The command "python setup.py --install=~" was used to install. And ~/lib/python is the first item in my PYTHONPATH variable. So the contents of sys.path is ['','~/lib/python', ...] Does anyone know any means to get the new version imported? Except from deleting the old version of course;) Regards, Joost From aisaac at american.edu Tue Nov 22 06:05:07 2005 From: aisaac at american.edu (Alan G Isaac) Date: Tue Nov 22 06:05:07 2005 Subject: [Numpy-discussion] import problem In-Reply-To: <1132667852.32469.9.camel@inpc93.et.tudelft.nl> References: <1132667852.32469.9.camel@inpc93.et.tudelft.nl> Message-ID: On Tue, 22 Nov 2005, Joost van Evert apparently wrote: > does anyone understand importing modules? I want to import a new version > of Numeric(24.2) that was installed in my home directory, but the old > one in /usr/lib/python/site-packages gets imported. > The command "python setup.py --install=~" was used to install. And > ~/lib/python is the first item in my PYTHONPATH variable. So the > contents of sys.path is ['','~/lib/python', ...] > Does anyone know any means to get the new version imported? Except from > deleting the old version of course;) http://docs.python.org/tut/node8.html section 6.1.1 hth, Alan Isaac From faltet at carabos.com Wed Nov 23 03:40:06 2005 From: faltet at carabos.com (Francesc Altet) Date: Wed Nov 23 03:40:06 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 Message-ID: <200511231239.21489.faltet@carabos.com> PyTables is a library to deal with very large datasets. It leverages the excellent HDF5 and numarray libraries to allow doing that in a very efficient way using the Python language. More info in: http://pytables.sourceforge.net/ ========================= Announcing PyTables 1.2 ========================= The PyTables development team is happy to announce the availability of a new major version of PyTables package. This version sports a completely new in-memory tree implementation based around a *node cache system*. This system loads nodes only when needed and unloads them when they are rarely used. The new feature allows the opening and creation of HDF5 files with large hierarchies very quickly and with a low memory consumption (the object tree is no longer completely loaded in-memory), while retaining all the powerful browsing capabilities of the previous implementation of the object tree. You can read more about the dings and bells of the new cache system in: http://www.carabos.com/downloads/pytables/NewObjectTreeCache.pdf Also, Jeff Whitaker has kindly contributed a new module called tables.NetCDF. It is designed to be used as a drop-in replacement for Scientific.IO.NetCDF, with only minor actions to existing code. Also, if you have the Scientific.IO.NetCDF module installed, it allows to do conversions between HDF5 <--> NetCDF3 formats. Go to the PyTables web site for downloading the beast: http://pytables.sourceforge.net/ If you want more info about this release, please check out the more comprehensive announcement message available in: http://www.carabos.com/downloads/pytables/ANNOUNCE-1.2.html Acknowledgments =============== Thanks to the users who provided feature improvements, patches, bug reports, support and suggestions. See THANKS file in distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last but not least, a big thank you to THG (http://www.hdfgroup.org/) for sponsoring many of the new features recently introduced in PyTables. --- **Enjoy data!** -- The PyTables Team -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From Chris.Barker at noaa.gov Wed Nov 23 08:49:02 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Nov 23 08:49:02 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 In-Reply-To: <200511231239.21489.faltet@carabos.com> References: <200511231239.21489.faltet@carabos.com> Message-ID: <43849D3F.7050703@noaa.gov> Francesc Altet wrote: > Also, Jeff Whitaker has kindly contributed a new module called > tables.NetCDF. It is designed to be used as a drop-in replacement for > Scientific.IO.NetCDF, with only minor actions to existing code. What advantages does tables.NetCDF have over Scientific.IO.NetCDF? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jswhit at fastmail.fm Wed Nov 23 08:59:14 2005 From: jswhit at fastmail.fm (Jeff Whitaker) Date: Wed Nov 23 08:59:14 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 In-Reply-To: <43849D3F.7050703@noaa.gov> References: <200511231239.21489.faltet@carabos.com> <43849D3F.7050703@noaa.gov> Message-ID: <43849F38.3000208@fastmail.fm> Chris Barker wrote: > Francesc Altet wrote: > >> Also, Jeff Whitaker has kindly contributed a new module called >> tables.NetCDF. It is designed to be used as a drop-in replacement for >> Scientific.IO.NetCDF, with only minor actions to existing code. > > What advantages does tables.NetCDF have over Scientific.IO.NetCDF? > > -Chris > > > Chris: From my perspective, the transparent compression (using zlib and the hdf5 shuffle filter) is the biggest one. I get files that are a factor of 2-4 smaller. -Jeff -- Jeffrey S. Whitaker Phone : (303)497-6313 Meteorologist FAX : (303)497-6449 NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker at noaa.gov 325 Broadway Office : Skaggs Research Cntr 1D-124 Boulder, CO, USA 80303-3328 Web : http://tinyurl.com/5telg From faltet at carabos.com Wed Nov 23 09:01:06 2005 From: faltet at carabos.com (Francesc Altet) Date: Wed Nov 23 09:01:06 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 In-Reply-To: <43849D3F.7050703@noaa.gov> References: <200511231239.21489.faltet@carabos.com> <43849D3F.7050703@noaa.gov> Message-ID: <200511231759.50278.faltet@carabos.com> A Dimecres 23 Novembre 2005 17:47, Chris Barker va escriure: > Francesc Altet wrote: > > Also, Jeff Whitaker has kindly contributed a new module called > > tables.NetCDF. It is designed to be used as a drop-in replacement for > > Scientific.IO.NetCDF, with only minor actions to existing code. > > What advantages does tables.NetCDF have over Scientific.IO.NetCDF? Maybe Jeff would add something more, but the apparent ones are: - No 2 GB limitation - Support for compression (and other filters) Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jswhit at fastmail.fm Wed Nov 23 09:37:18 2005 From: jswhit at fastmail.fm (Jeff Whitaker) Date: Wed Nov 23 09:37:18 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 In-Reply-To: <200511231759.50278.faltet@carabos.com> References: <200511231239.21489.faltet@carabos.com> <43849D3F.7050703@noaa.gov> <200511231759.50278.faltet@carabos.com> Message-ID: <4384A7F4.3070309@fastmail.fm> Francesc Altet wrote: > A Dimecres 23 Novembre 2005 17:47, Chris Barker va escriure: > >> Francesc Altet wrote: >> >>> Also, Jeff Whitaker has kindly contributed a new module called >>> tables.NetCDF. It is designed to be used as a drop-in replacement for >>> Scientific.IO.NetCDF, with only minor actions to existing code. >>> >> What advantages does tables.NetCDF have over Scientific.IO.NetCDF? >> > > Maybe Jeff would add something more, but the apparent ones are: > > - No 2 GB limitation > - Support for compression (and other filters) > > Cheers, > > And any dimension can be 'unlimited' (not just the first). -Jeff -- Jeffrey S. Whitaker Phone : (303)497-6313 Meteorologist FAX : (303)497-6449 NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker at noaa.gov 325 Broadway Office : Skaggs Research Cntr 1D-124 Boulder, CO, USA 80303-3328 Web : http://tinyurl.com/5telg From faltet at carabos.com Wed Nov 23 10:12:04 2005 From: faltet at carabos.com (Francesc Altet) Date: Wed Nov 23 10:12:04 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 In-Reply-To: <4384A7F4.3070309@fastmail.fm> References: <200511231239.21489.faltet@carabos.com> <200511231759.50278.faltet@carabos.com> <4384A7F4.3070309@fastmail.fm> Message-ID: <200511231911.16720.faltet@carabos.com> A Dimecres 23 Novembre 2005 18:33, Jeff Whitaker va escriure: > > - No 2 GB limitation > > - Support for compression (and other filters) > And any dimension can be 'unlimited' (not just the first). Yes, but just one dimension at a time, at least for the time being. We would like to eliminate this limitation rather sooner than later, though. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From arajendr at ucsd.edu Wed Nov 23 15:48:01 2005 From: arajendr at ucsd.edu (Anoop) Date: Wed Nov 23 15:48:01 2005 Subject: [Numpy-discussion] Problem compiling scipy-core Message-ID: <1132789622.7570.4.camel@localhost.localdomain> Hi. I have problems installing scipy-core from source code on a Rocks Cluster distribution of Linux running on x86-64 hardware. The error messages tell me: /usr/bin/g77 -shared build/temp.linux-x86_64-2.3/scipy/corelib/blasdot/_dotblas.o -L/usr/lib -lblas -lg2c -o build/lib.linux-x86_64-2.3/scipy/lib/_dotblas.so /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.4/../../../libblas.so when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.4/../../../libblas.a when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas /usr/bin/ld: cannot find -lblas collect2: ld returned 1 exit status I really need to get this running. Help? Thanks, Anoop From dd55 at cornell.edu Wed Nov 23 16:08:01 2005 From: dd55 at cornell.edu (Darren Dale) Date: Wed Nov 23 16:08:01 2005 Subject: [Numpy-discussion] Problem compiling scipy-core In-Reply-To: <1132789622.7570.4.camel@localhost.localdomain> References: <1132789622.7570.4.camel@localhost.localdomain> Message-ID: <200511231907.25815.dd55@cornell.edu> On Wednesday 23 November 2005 6:47 pm, Anoop wrote: > Hi. I have problems installing scipy-core from source code on a Rocks > Cluster distribution of Linux running on x86-64 hardware. The error > messages tell me: > > /usr/bin/g77 -shared > build/temp.linux-x86_64-2.3/scipy/corelib/blasdot/_dotblas.o -L/usr/lib > -lblas -lg2c -o build/lib.linux-x86_64-2.3/scipy/lib/_dotblas.so > /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for > -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching > for -lblas /usr/bin/ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.4/../../../libblas.so when searching > for -lblas /usr/bin/ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.4/../../../libblas.a when searching > for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when > searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a > when searching for -lblas /usr/bin/ld: cannot find -lblas > collect2: ld returned 1 exit status I saw something similar to this just the other day. In my case, the problem was that scipy discovered broken libblas.* softlinks in /usr/lib and tried to build against them. It looks like something similar is happening here, check your atlas/blas/lapack installation. Darren -- Darren S. Dale, Ph.D. dd55 at cornell.edu From gyromagnetic at gmail.com Fri Nov 25 06:25:03 2005 From: gyromagnetic at gmail.com (gf) Date: Fri Nov 25 06:25:03 2005 Subject: [Numpy-discussion] help in improving data analysis code Message-ID: Hi, I am a newbie to Numeric/numarray programming and would appreciate your help in improving the code below (which I'm sure is quite ugly to an experienced numarray programmer). An analysis we are carrying out requires the following: 1. evaluate the mean of a set of data 2. eliminate the data point farthest from the mean 3. repeat steps 1 and 2 until a certain specified fraction of points has been eliminated. Since this analysis will have to be performed (probably repeatedly) on approximately ten thousand data sets, each of which contains 100-500 points, I would like the code to be as fast as possible. Thanks for your help. -g ==== from numarray import add, array, asarray, absolute, argsort, floor, take, size def mean(m,axis=0): m = asarray(m) return add.reduce(m,axis)/float(m.shape[axis]) def eliminate_outliers(dat,frac): num_to_eliminate = int(floor(size(dat,0)*frac)) for i in range(num_to_eliminate): ind = argsort(absolute(dat-mean(dat)),0) sdat = take(dat,ind,0)[:,0] dat = sdat[:-1] return dat #-------------------------------------------------------------------- if __name__ == "__main__": from MLab import rand sz = 100 nn = rand(sz,1) nn[:10] = 20*rand(10,1) nn[sz-10:] = -20*rand(10,1) print eliminate_outliers(nn,0.10) From faltet at carabos.com Fri Nov 25 07:28:04 2005 From: faltet at carabos.com (Francesc Altet) Date: Fri Nov 25 07:28:04 2005 Subject: [Numpy-discussion] help in improving data analysis code In-Reply-To: References: Message-ID: <200511251627.11520.faltet@carabos.com> A Divendres 25 Novembre 2005 15:24, gf va escriure: > > from numarray import add, array, asarray, absolute, argsort, floor, take, > size > > def mean(m,axis=0): > m = asarray(m) > return add.reduce(m,axis)/float(m.shape[axis]) > > def eliminate_outliers(dat,frac): > num_to_eliminate = int(floor(size(dat,0)*frac)) > for i in range(num_to_eliminate): > ind = argsort(absolute(dat-mean(dat)),0) > sdat = take(dat,ind,0)[:,0] > dat = sdat[:-1] > return dat > > #-------------------------------------------------------------------- > > if __name__ == "__main__": > from MLab import rand > sz = 100 > nn = rand(sz,1) > nn[:10] = 20*rand(10,1) > nn[sz-10:] = -20*rand(10,1) > print eliminate_outliers(nn,0.10) For sz=100, the next line of code is 10x faster on my machine (more if sz is bigger): print nn[argsort(abs(nn_c-nn_c.mean()),0)][:-int(sz*0.10),0] I haven't checked it very carefully, so you should double check it. BTW, you will need to use the numarray MLab interface: from numarray.mlab import rand Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Fri Nov 25 07:33:02 2005 From: faltet at carabos.com (Francesc Altet) Date: Fri Nov 25 07:33:02 2005 Subject: [Numpy-discussion] help in improving data analysis code In-Reply-To: <200511251627.11520.faltet@carabos.com> References: <200511251627.11520.faltet@carabos.com> Message-ID: <200511251632.04416.faltet@carabos.com> A Divendres 25 Novembre 2005 16:27, Francesc Altet va escriure: > print nn[argsort(abs(nn_c-nn_c.mean()),0)][:-int(sz*0.10),0] Ups. I have had a confusion. This should work better ;-) print nn[argsort(abs(nn-nn.mean()),0)][:-int(sz*0.10),0] -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From gyromagnetic at gmail.com Fri Nov 25 14:19:00 2005 From: gyromagnetic at gmail.com (gf) Date: Fri Nov 25 14:19:00 2005 Subject: [Numpy-discussion] Re: help in improving data analysis code In-Reply-To: References: Message-ID: From: Francesc Altet Re: help in improving data analysis code 2005-11-25 07:32 >>A Divendres 25 Novembre 2005 16:27, Francesc Altet va escriure: > > print nn[argsort(abs(nn_c-nn_c.mean()),0)][:-int(sz*0.10),0] >> >> Ups. I have had a confusion. This should work better ;-) >> >> print nn[argsort(abs(nn-nn.mean()),0)][:-int(sz*0.10),0] Hi Francesc, Thank you for the suggestions. Your code is performing a different task than mine was. In particular, I believe it does not 're-mean' the data after removing each point. However, based on the great ideas from your code, I now have the function below that looks to be more efficient (although I haven't measured it). Any additional suggestions are appreciated. -g ==== from numarray import argsort, floor, absolute def eliminate_outliers(data,frac): num_to_eliminate = int(floor(data.size())*frac) for i in range(num_to_eliminate): data = data[argsort(absolute(data-data.mean()),0)][:-1,0] return data if __name__ == "__main__": from numarray.mlab import rand sz = 100 nn = rand(sz,1) nn[:10] = 20*rand(10,1) nn[sz-10:] = -20*rand(10,1) print eliminate_outliers(nn,0.10) From service at paypal.com Tue Nov 29 05:40:25 2005 From: service at paypal.com (PayPal) Date: Tue Nov 29 05:40:25 2005 Subject: [Numpy-discussion] Your PayPal Account Could Be Suspended Message-ID: <20051129134036.F39C71FE227@gosecweb.gosecure.pt> An HTML attachment was scrubbed... URL: From faltet at carabos.com Tue Nov 1 01:37:26 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Nov 1 01:37:26 2005 Subject: [Numpy-discussion] byteswap questions In-Reply-To: References: Message-ID: <1130838789.2968.23.camel@localhost.localdomain> El dl 31 de 10 del 2005 a les 16:53 -0800, en/na Russell E. Owen va escriure: > I've got a module that can send array data to ds9. It screws up on > byteswapped data and I'm trying to fix it. I don't exactly know what a ds9 is, but I'm supposing here that it is kind of an exotic computer. > I need to know if the order is bigendian or littleendian, and I can't > find a documented way to get that, just an undocumented attribute > _byteorder. Yes, it's undocumented, but as far as I can tell, this works flawlessly in numarray. > 2) If the array is not in native byte order, make a native byte ordered > copy before sending it. > > The following seems to work: > if arr.isbyteswapped(): > arr = arr.copy() > > but is this the recommended way to get native-order copy? If you can afford doing a memory copy, yes, I think this is a good way to do it. > - Is "copy" guaranteed to return a copy in native byte order? The > documentation doesn't say. It does: In [23]:a._byteorder Out[23]:'big' In [24]:a.copy()._byteorder Out[24]:'little' In general, copy() will return you a well-behaved (native-byte ordered, non-strided and non-offsetted) array. > - I was tempted to use num.array(arr) to make the copy, but the > documentation doesn't say what "array" does if the input is a already a > numarray array. This also works: In [25]:array(a)._byteorder Out[25]:'little' and it is very similar to copy() in performance: In [34]:t1=Timer("b=a.copy()","import numarray; a=numarray.arange(10);a.byteswap();a._byteorder='big'") In [35]:t1.repeat(3,10000) Out[35]:[1.0613389015197754, 1.0313379764556885, 1.0303771495819092] In [36]:t2=Timer("b=numarray.array(a)","import numarray; a=numarray.arange(10);a.byteswap();a._byteorder='big'") In [37]:t2.repeat(3,10000) Out[37]:[1.2250277996063232, 1.2067301273345947, 1.2336349487304688] (perhaps copy is just a little bit faster) > As an aside, I tried to use the byteswapped method, but it has a totally > different effect: > > >>> d > array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], > ...type=Float32) > >>> d.isbyteswapped() > 1 > >>> dcopy = d.copy() > >>> d > array([[-9012., -9012., -9012., ..., -9012., -9012., -9012.], > ...type=Float32) > >>> dcopy.isbyteswapped() > 0 > >>> dswap = d.byteswapped() > >>> dswap > array([[ 1.91063654e-38, 1.91063654e-38, 1.91063654e-38, ..., > ...type=Float32) > > which I guess means I'd have to byteswap the byteswapped data. Aargh. This is the intended behaviour. From the docstrings: In [59]:a.byteswap? [snip...] Docstring: Byteswap data in place, leaving the _byteorder attribute untouched. So, you already have done a byteswapping, but you still need to tell the object that its byteorder has actually changed (if don't, numarray will convert the data to you local native order before printing it on the flight). Just issue a: dswap._byteorder = {"little":"big","big":"little"}[dswap._byteorder] and you are done. In fact, using byteswap() does the conversion in-place, and doesn't need a memory copy. This is actually faster than doing a copy() much of the times: In [60]:t3=Timer("b=a.byteswap();a._byteorder='little'","import numarray; a=numarray.arange(10);a.byteswap();a._byteorder='big'") In [61]:t3.repeat(3,10000) Out[61]:[0.23770284652709961, 0.23195695877075195, 0.23380589485168457] Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From faltet at carabos.com Tue Nov 1 03:38:57 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Nov 1 03:38:57 2005 Subject: [Numpy-discussion] Performance of the array protocol Message-ID: <1130846099.2968.39.camel@localhost.localdomain> Hi, I'm trying to start using the array protocol for conversion between Numeric <--> numarray (and newcore in the future), but I'm a bit disappointed because of its performance. For numarray --> Numeric we have: >>> t1=timeit.Timer("num=Numeric.array(na)", "import numarray; import Numeric; na=numarray.arange(10)") >>> t1.repeat(3,10000) [0.59375977516174316, 0.57908082008361816, 0.56574010848999023] >>>t2=timeit.Timer("num=Numeric.fromstring(na._data,typecode=na.typecode())", "import numarray; import Numeric; na=numarray.arange(10)") >>> t2.repeat(3,10000) [0.11653494834899902, 0.1140749454498291, 0.1141819953918457] i.e. the array protocol seems 5x slower than the fromstring() method. Conversely, for Numeric --> numarray: >>> t3=timeit.Timer("na=numarray.array(num)", "import numarray; import Numeric;num=Numeric.arange(10)") >>> t3.repeat(3,10000) [1.3475611209869385, 1.3277668952941895, 1.3417830467224121] >>>t4=timeit.Timer("na=numarray.array(buffer(num),type=num.typecode(),shape=num.shape)", "import numarray; import Numeric; num=Numeric.arange(10)") >>> t4.repeat(3,10000) [0.42027187347412109, 0.41690587997436523, 0.41626906394958496] in this case, the array protocol is 3x slower than using the buffer interface. I'm wondering whether this relatively poor performance in present implementation of the array protocol is surmountable or is an intrinsic limitation of it. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Tue Nov 1 08:13:06 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 08:13:06 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <1130846099.2968.39.camel@localhost.localdomain> References: <1130846099.2968.39.camel@localhost.localdomain> Message-ID: <436793CC.5030106@ee.byu.edu> Francesc Altet wrote: >Hi, > >I'm trying to start using the array protocol for conversion between >Numeric <--> numarray (and newcore in the future), but I'm a bit >disappointed because of its performance. For numarray --> Numeric we >have: > > > >>>>t1=timeit.Timer("num=Numeric.array(na)", "import numarray; import >>>> >>>> >Numeric; na=numarray.arange(10)") > > >>>>t1.repeat(3,10000) >>>> >>>> >[0.59375977516174316, 0.57908082008361816, 0.56574010848999023] > > >>>>t2=timeit.Timer("num=Numeric.fromstring(na._data,typecode=na.typecode())", "import numarray; import Numeric; na=numarray.arange(10)") >>>>t2.repeat(3,10000) >>>> >>>> >[0.11653494834899902, 0.1140749454498291, 0.1141819953918457] > >i.e. the array protocol seems 5x slower than the fromstring() method. > > > If you are going to be copying the data anyway, then there may be no advantage to the array protocol (in fact because it has to look up several attributes of the input object it can be slower). When you use Numeric.array(na) it makes a copy of the data by default. The idea is to be able to use the array protocol to not have to make copies of the data. Try using num = Numeric.array(na,copy=0) in your first timing runs and see what that provides. >Conversely, for Numeric --> numarray: > > > >>>>t3=timeit.Timer("na=numarray.array(num)", "import numarray; import >>>> >>>> >Numeric;num=Numeric.arange(10)") > > >>>>t3.repeat(3,10000) >>>> >>>> >[1.3475611209869385, 1.3277668952941895, 1.3417830467224121] > > >>>>t4=timeit.Timer("na=numarray.array(buffer(num),type=num.typecode(),shape=num.shape)", "import numarray; import Numeric; num=Numeric.arange(10)") >>>>t4.repeat(3,10000) >>>> >>>> >[0.42027187347412109, 0.41690587997436523, 0.41626906394958496] > >in this case, the array protocol is 3x slower than using the buffer >interface. > > Again, you are making copies of the data. I'm not sure how numarray handles the array protocol on consumption of the interface, so I can't comment further. -Travis From oliphant at ee.byu.edu Tue Nov 1 08:19:01 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 08:19:01 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <1130846099.2968.39.camel@localhost.localdomain> References: <1130846099.2968.39.camel@localhost.localdomain> Message-ID: <43679557.70508@ee.byu.edu> Francesc Altet wrote: >Hi, > >I'm trying to start using the array protocol for conversion between >Numeric <--> numarray (and newcore in the future), but I'm a bit >disappointed because of its performance. For numarray --> Numeric we >have: > > The other factor in this discussion is that you are creating and sharing relatively small arrays. The fromstring apporach is not a problem if all you need is a copy of the data for small arrays. The array protocol approach uses attribute lookups on an object. This is going to have overhead that will only be useful for large arrays (or arrays that you want to have share the same data region). So, I guess the answer to your question is that for small arrays it is an intrinsic limitation of the use of Python attributes in the array protocol. -Travis From faltet at carabos.com Tue Nov 1 09:20:16 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Nov 1 09:20:16 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <436793CC.5030106@ee.byu.edu> References: <1130846099.2968.39.camel@localhost.localdomain> <436793CC.5030106@ee.byu.edu> Message-ID: <1130866551.2968.69.camel@localhost.localdomain> El dt 01 de 11 del 2005 a les 09:11 -0700, en/na Travis Oliphant va escriure: > If you are going to be copying the data anyway, then there may be no > advantage to the array protocol (in fact because it has to look up > several attributes of the input object it can be slower). When you use > Numeric.array(na) it makes a copy of the data by default. > > The idea is to be able to use the array protocol to not have to make > copies of the data. Yes, I don't want to do a copy. And, in fact, I want to use moderately large array conversions (10**4 ~ 10**6 elements). > Try using num = Numeric.array(na,copy=0) in your first timing runs and > see what that provides. Good! Using copy=0 and larger arrays (10**5 elements) I'm getting now: >>> t1_2=timeit.Timer("num=Numeric.array(na,copy=0)", "import numarray; import Numeric; na=numarray.arange(100000)") >>> t1_2.repeat(3,1000) [0.064317941665649414, 0.060917854309082031, 0.07666015625] >>> t2_2=timeit.Timer("num=Numeric.fromstring(na._data,typecode=na.typecode())", "import numarray; import Numeric; na=numarray.arange(100000)") >>> t2_2.repeat(3,1000) [4.8582658767700195, 4.8404099941253662, 4.8652839660644531] So, the implementation of the array protocol in the numarray --> Numeric way is performing ashtonishingly well :-) For the records, using the array protocol without a copy gives: >>> t1=timeit.Timer("num=Numeric.array(na)", "import numarray; import Numeric; na=numarray.arange(100000)") >>> t1.repeat(3,1000) [5.014805793762207, 4.9959368705749512, 5.0420081615447998] i.e. almost as fast a the fromstring() method, which is very good as well! BTW, I'm wondering whether a False value for copy should be used as the default instead of True. IMO, many people would want to make use of the array protocol just to access easily the data, and making a copy() behind the scenes just for this might be potentially killer, specially for large objects. >> Conversely, for Numeric --> numarray: > Again, you are making copies of the data. I'm not sure how numarray > handles the array protocol on consumption of the interface, so I can't > comment further. Mmmm, I've tried disabling the copy, but unfortunately enough I can't get the same figures as above: >>> t3=timeit.Timer("na=numarray.array(num,copy=0)", "import numarray; import Numeric; num=Numeric.arange(100000)") >>> t3.repeat(3,10) [1.6356601715087891, 1.6529910564422607, 1.6299269199371338] >>>t4=timeit.Timer("na=numarray.array(buffer(num),type=num.typecode(),shape=num.shape)", "import numarray; import Numeric; num=Numeric.arange(100000)") >>> t4.repeat(3,1000) [0.045578956604003906, 0.043890953063964844, 0.043296098709106445] so, for the Numeric --> numarray way, the slowdown is more than three orders of magnitude than expected (note the fewer iterations for the first repeat loop). Maybe Todd can comment more on this. Thanks! -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Tue Nov 1 09:22:53 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 09:22:53 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <1130866551.2968.69.camel@localhost.localdomain> References: <1130846099.2968.39.camel@localhost.localdomain> <436793CC.5030106@ee.byu.edu> <1130866551.2968.69.camel@localhost.localdomain> Message-ID: <4367A44F.4040808@ee.byu.edu> Francesc Altet wrote: >BTW, I'm wondering whether a False value for copy should be used as the >default instead of True. IMO, many people would want to make use of the >array protocol just to access easily the data, and making a copy() >behind the scenes just for this might be potentially killer, specially >for large objects. > > Thats exactly what the asarray function is for. It is equivalent to array except its default for copy is False. -Travis From jmiller at stsci.edu Tue Nov 1 09:28:25 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue Nov 1 09:28:25 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <1130846099.2968.39.camel@localhost.localdomain> References: <1130846099.2968.39.camel@localhost.localdomain> Message-ID: <4367A56E.5040905@stsci.edu> Francesc Altet wrote: >Hi, > >I'm trying to start using the array protocol for conversion between >Numeric <--> numarray (and newcore in the future), but I'm a bit >disappointed because of its performance. For numarray --> Numeric we >have: > > > >>>>t1=timeit.Timer("num=Numeric.array(na)", "import numarray; import >>>> >>>> >Numeric; na=numarray.arange(10)") > > >>>>t1.repeat(3,10000) >>>> >>>> >[0.59375977516174316, 0.57908082008361816, 0.56574010848999023] > > >>>>t2=timeit.Timer("num=Numeric.fromstring(na._data,typecode=na.typecode())", "import numarray; import Numeric; na=numarray.arange(10)") >>>>t2.repeat(3,10000) >>>> >>>> >[0.11653494834899902, 0.1140749454498291, 0.1141819953918457] > >i.e. the array protocol seems 5x slower than the fromstring() method. > >Conversely, for Numeric --> numarray: > > > >>>>t3=timeit.Timer("na=numarray.array(num)", "import numarray; import >>>> >>>> >Numeric;num=Numeric.arange(10)") > > >>>>t3.repeat(3,10000) >>>> >>>> >[1.3475611209869385, 1.3277668952941895, 1.3417830467224121] > > >>>>t4=timeit.Timer("na=numarray.array(buffer(num),type=num.typecode(),shape=num.shape)", "import numarray; import Numeric; num=Numeric.arange(10)") >>>>t4.repeat(3,10000) >>>> >>>> >[0.42027187347412109, 0.41690587997436523, 0.41626906394958496] > >in this case, the array protocol is 3x slower than using the buffer >interface. > >I'm wondering whether this relatively poor performance in present >implementation of the array protocol is surmountable or is an intrinsic >limitation of it. > >Thanks, > > I was working on improving this for numarray yesterday so some improvements are in CVS now. I moved some of the Python array interface properties down to C attributes and the performance ratio for my debug Python is now <2x for numarray-->Numeric. numarray's array interface for Numeric-->numarray was degenerating to fromlist(); I added array interface "consumption" support for numerical arrays by beefing up numarray's array() function. I tweaked your benchmarks a little to support profiling as well as timing and attached the result. % python arrayif_bench.py numarray-->Numeric array_if: [0.35534501075744629, 0.36865997314453125, 0.36826896667480469] numarray-->Numeric fromstring: [0.36841487884521484, 0.21085405349731445, 0.20747494697570801] Numeric-->numarray array_if: [0.73384881019592285, 0.6396629810333252, 0.60234308242797852] Numeric-->numarray buffer_if: [0.36455512046813965, 0.24601507186889648, 0.23759102821350098] -------------- next part -------------- A non-text attachment was scrubbed... Name: arrayif_bench.py Type: application/x-python Size: 1103 bytes Desc: not available URL: From Chris.Barker at noaa.gov Tue Nov 1 10:07:59 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Nov 1 10:07:59 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <43679557.70508@ee.byu.edu> References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> Message-ID: <4367AEC8.4010707@noaa.gov> Travis Oliphant wrote: > So, I guess the answer to your question is that for small arrays it is > an intrinsic limitation of the use of Python attributes in the array > protocol. IIRC, in the early discussion of the array protocol, we had talked about defining a C struct, and a set of utilities to query that struct. Now, I guess it uses Python attributes. Do I recall incorrectly, or has there been a design change, or is this a prototype implementation? I guess I'd like to see an array protocol that is as fast as fromstring(), even for small arrays, though it's probably not a big deal. Also, when writing C code to work with an array, it might be easier, as well as faster, to not have to query Python attributes. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From faltet at carabos.com Tue Nov 1 10:16:10 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Nov 1 10:16:10 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367A56E.5040905@stsci.edu> References: <1130846099.2968.39.camel@localhost.localdomain> <4367A56E.5040905@stsci.edu> Message-ID: <1130869911.2968.77.camel@localhost.localdomain> El dt 01 de 11 del 2005 a les 12:27 -0500, en/na Todd Miller va escriure: > I was working on improving this for numarray yesterday so some > improvements are in CVS now. > > I moved some of the Python array interface properties down to C > attributes and the performance ratio for my debug Python is now <2x for > numarray-->Numeric. Good! so for numarray-->Numeric we have really good performance now. > numarray's array interface for Numeric-->numarray > was degenerating to fromlist(); I added array interface "consumption" > support for numerical arrays by beefing up numarray's array() function. I'm not sure what are you saying here. You mean that in CVS there is already code for Numeric-->numarray that does not degenerate to fromlist() anymore? If this is the case, it's great news, of course. I'll try it as soon as I can. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From jmiller at stsci.edu Tue Nov 1 10:24:21 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue Nov 1 10:24:21 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <1130869911.2968.77.camel@localhost.localdomain> References: <1130846099.2968.39.camel@localhost.localdomain> <4367A56E.5040905@stsci.edu> <1130869911.2968.77.camel@localhost.localdomain> Message-ID: <4367B28F.8020602@stsci.edu> Francesc Altet wrote: >El dt 01 de 11 del 2005 a les 12:27 -0500, en/na Todd Miller va >escriure: > > > >>numarray's array interface for Numeric-->numarray >>was degenerating to fromlist(); I added array interface "consumption" >>support for numerical arrays by beefing up numarray's array() function. >> >> > >I'm not sure what are you saying here. You mean that in CVS there is >already code for Numeric-->numarray that does not degenerate to >fromlist() anymore? > Yes. I added a check in array() for suppliers of the array-interface which are not NDArrays. It only works for numerical arrays. Todd From faltet at carabos.com Tue Nov 1 10:35:25 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Nov 1 10:35:25 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367B28F.8020602@stsci.edu> References: <1130846099.2968.39.camel@localhost.localdomain> <4367A56E.5040905@stsci.edu> <1130869911.2968.77.camel@localhost.localdomain> <4367B28F.8020602@stsci.edu> Message-ID: <1130871081.2968.80.camel@localhost.localdomain> El dt 01 de 11 del 2005 a les 13:23 -0500, en/na Todd Miller va escriure: > Yes. I added a check in array() for suppliers of the array-interface > which are not NDArrays. It only works for numerical arrays. Sounds good. I suppose that you will add the same check to asarray() in order to avoid the copy, isn't it? -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant at ee.byu.edu Tue Nov 1 10:53:54 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 10:53:54 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367AEC8.4010707@noaa.gov> References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> Message-ID: <4367B984.2060303@ee.byu.edu> Chris Barker wrote: > Travis Oliphant wrote: > >> So, I guess the answer to your question is that for small arrays it >> is an intrinsic limitation of the use of Python attributes in the >> array protocol. > > > IIRC, in the early discussion of the array protocol, we had talked > about defining a C struct, and a set of utilities to query that > struct. Now, I guess it uses Python attributes. Do I recall > incorrectly, or has there been a design change, or is this a prototype > implementation? I guess I'd like to see an array protocol that is as > fast as fromstring(), even for small arrays, though it's probably not > a big deal. Also, when writing C code to work with an array, it might > be easier, as well as faster, to not have to query Python attribute You are correct that it would be good to have a C-protocol that did not require attribute lookups (the source of any speed difference with fromstring). Not much progress has been made on a C-version of the protocol, though. I don't know how to do it without adding something to Python itself. At SciPy 2005, I outlined my vision for how we could proceed in that direction. There is a PEP in a subversion repository at http://svn.scipy.org/svn/PEP that explains my view. Basically, I think we should push for a simple (C-based) Nd array object that is nothing more than the current C-structure of the N-d array. Then, arrays would inherit from this base class but all of Python would be able to understand and query it's C-structure. If we could also get an array interface into the typeobject table, it would be a simple thing to populate this structure even with objects that didn't inherit from the base object. I am still interested in other ideas for how to implement the array interface in C, without adding something to the type-object table (we could push for that, but it might take more political effort). -Travis From cookedm at physics.mcmaster.ca Tue Nov 1 11:02:58 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Nov 1 11:02:58 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367AEC8.4010707@noaa.gov> (Chris Barker's message of "Tue, 01 Nov 2005 10:07:04 -0800") References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> Message-ID: "Chris Barker" writes: > Travis Oliphant wrote: >> So, I guess the answer to your question is that for small arrays it >> is an intrinsic limitation of the use of Python attributes in the >> array protocol. > > IIRC, in the early discussion of the array protocol, we had talked about > defining a C struct, and a set of utilities to query that struct. Now, I > guess it uses Python attributes. Do I recall incorrectly, or has there > been a design change, or is this a prototype implementation? I guess I'd > like to see an array protocol that is as fast as fromstring(), even for > small arrays, though it's probably not a big deal. Also, when writing C > code to work with an array, it might be easier, as well as faster, to > not have to query Python attributes. I had posted an idea before: http://thread.gmane.org/gmane.comp.python.numeric.general/2159 It would still be one attribute lookup, but then everything would be C-based from there on. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Tue Nov 1 12:39:41 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 12:39:41 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> Message-ID: <4367D279.1010700@ee.byu.edu> David M. Cooke wrote: >"C > >I had posted an idea before: > >http://thread.gmane.org/gmane.comp.python.numeric.general/2159 > >It would still be one attribute lookup, but then everything would be >C-based from there on. > > Thanks for reminding us of your idea again. This is a very good idea, that I think we could add. My only question is should Py_LONG_LONG be used or Py_intptr_t? Since the array has to be in memory anyway there does not seem any advantage to using Py_LONG_LONG. I also think a flags member of the structure is useful along with a typestr instead of typecode. I would say the C-struct should look like. We could push to get something like this in Python core, I think, so this Array_Interface header was available to everybody. typedef struct { int nd; char typekind; int itemsize; int flags; Py_intptr_t *shape; Py_intptr_t *strides; Py_intptr_t offset; void *data; } PyArray_Interface From cookedm at physics.mcmaster.ca Tue Nov 1 13:02:06 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Nov 1 13:02:06 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367D279.1010700@ee.byu.edu> (Travis Oliphant's message of "Tue, 01 Nov 2005 13:39:21 -0700") References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> <4367D279.1010700@ee.byu.edu> Message-ID: Travis Oliphant writes: > David M. Cooke wrote: > >>"C >> >>I had posted an idea before: >> >>http://thread.gmane.org/gmane.comp.python.numeric.general/2159 >> >>It would still be one attribute lookup, but then everything would be >>C-based from there on. >> >> > Thanks for reminding us of your idea again. This is a very good idea, > that I think we could add. My only question is should Py_LONG_LONG be > used or Py_intptr_t? Since the array has to be in memory anyway there > does not seem any advantage to using Py_LONG_LONG. Ok, I see how that works. I probably wasn't aware of the existence of Py_intptr_t at the time :-) > I also think a flags member of the structure is useful along with a > typestr instead of typecode. The point of a typecode instead of a typestr was to make it easy for code to determine the type. Endianness would be part of the flags, and the number of bytes for the type would be in itemsize. > I would say the C-struct should look > like. We could push to get something like this in Python core, I think, > so this Array_Interface header was available to everybody. > > typedef struct { > int nd; > char typekind; > int itemsize; > int flags; > Py_intptr_t *shape; > Py_intptr_t *strides; > Py_intptr_t offset; > void *data; > } PyArray_Interface If it's in the Python core, then this is fine. If we did it ourself as an informal protocol (like the array interface spec), I'd still add the version member as a sanity check. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Chris.Barker at noaa.gov Tue Nov 1 16:04:06 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Nov 1 16:04:06 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <4367D279.1010700@ee.byu.edu> References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> <4367D279.1010700@ee.byu.edu> Message-ID: <43680252.50009@noaa.gov> Travis Oliphant wrote: > like. We could push to get something like this in Python core, I think, > so this Array_Interface header was available to everybody. That would be great. Until then, it would still be a tiny header that others could easily include with their code. David version number would help keep things sorted out. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Tue Nov 1 16:49:32 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Nov 1 16:49:32 2005 Subject: [Numpy-discussion] Performance of the array protocol In-Reply-To: <43680252.50009@noaa.gov> References: <1130846099.2968.39.camel@localhost.localdomain> <43679557.70508@ee.byu.edu> <4367AEC8.4010707@noaa.gov> <4367D279.1010700@ee.byu.edu> <43680252.50009@noaa.gov> Message-ID: <43680CF6.4010000@ee.byu.edu> Chris Barker wrote: > Travis Oliphant wrote: > >> like. We could push to get something like this in Python core, I >> think, so this Array_Interface header was available to everybody. > > > That would be great. Until then, it would still be a tiny header that > others could easily include with their code. David version number > would help keep things sorted out. > I've placed an updated array interface description that includes this struct-based access on http://numeric.scipy.org Somebody will add support for this in scipy soon too. -Travis From faltet at carabos.com Wed Nov 2 00:37:41 2005 From: faltet at carabos.com (Francesc Altet) Date: Wed Nov 2 00:37:41 2005 Subject: [Numpy-discussion] More on array protocol Message-ID: <1130921630.3101.11.camel@localhost.localdomain> Hi, I'm having problems in using the array protocol in numarray-->Numeric in some situations. I'm using the PyBuffer_FromReadWriteMemory C call in order to create buffers that will be later used to create numarray objects (both NumArray and CharArray) in Python space. The problem is that the array protocol implementation is doing badly this conversion. I was able to create a small example in pure python that reproduces the problem: >>> import numarray >>> from numarray import memory >>> import Numeric >>> na=numarray.array([1,2]) >>> Numeric.array(na) array([1, 2]) but ... >>> wbuf = memory.writeable_buffer(na) >>> nawb = numarray.array(wbuf,type="Int32",shape=(2,)) >>> nawb array([1, 2]) >>> Numeric.array(nawb) array([ 2, 135510112]) I'm afraid that the problem should be related with the kind of buffer that is attached to the numarray object: >>> na._data >>> nawb._data Travis, Todd, are you going to support read-write buffers or should I try to move to using the memory module in order to cope with this? For a series of reasons I'd like to keep using regular read-write buffers, but anyway... I think I'll be able to do the change if absolutely necessary. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From faltet at carabos.com Wed Nov 2 05:08:06 2005 From: faltet at carabos.com (Francesc Altet) Date: Wed Nov 2 05:08:06 2005 Subject: [Numpy-discussion] Problem with array assignment Message-ID: <200511021406.56504.faltet@carabos.com> Hi, I've recently upgraded to Numeric 24.0 and numarray 1.4.1 and I'm having a problem that did not exist before: >>> import Numeric >>> import numarray >>> Numeric.__version__ '24.0' >>> numarray.__version__ '1.4.1' >>> na=numarray.zeros((3,),'f') >>> num=Numeric.zeros((3,),'f') >>> na[...] = num Traceback (most recent call last): File "", line 1, in ? ValueError: array sizes must be consistent. but... >>> na=numarray.zeros((3,),'i') >>> num=Numeric.zeros((3,),'i') >>> na[...] = num >>> na array([0, 0, 0]) ????? However, using Numeric 23.8 and numarray 1.3.3: >>> import Numeric >>> import numarray >>> Numeric.__version__ '23.8' >>> numarray.__version__ '1.3.3' >>> na=numarray.zeros((3,),'f') >>> num=Numeric.zeros((3,),'f') >>> na[...] = num >>> na array([ 0., 0., 0.], type=Float32) I don't know if the problem is coming from Numeric or numarray, but it only happens when one mix both: >>> Numeric.__version__ '24.0' >>> numarray.__version__ '1.4.1' >>> na=numarray.zeros((3,),'f') >>> na2=numarray.zeros((3,),'f') >>> na2[...] = na >>> na2 array([ 0., 0., 0.], type=Float32) >>> num=Numeric.zeros((3,),'f') >>> num2=Numeric.zeros((3,),'f') >>> num2[...] = num >>> num array([ 0., 0., 0.],'f') Perhaps the new array protocol is involved here? -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From fonnesbeck at gmail.com Wed Nov 2 05:55:58 2005 From: fonnesbeck at gmail.com (Chris Fonnesbeck) Date: Wed Nov 2 05:55:58 2005 Subject: [Numpy-discussion] 1.4.1 build failure under OSX Message-ID: <723eb6930511020555w5ec42098na5d1502b5608990d@mail.gmail.com> Hi, Compiling 1.4.1 under OSX with either gcc 3.3 or 4.0 fails (all previous versions of numarray compiled without error). Here is the full output: Using EXTRA_COMPILE_ARGS = ['-Ddarwin'] Using external BLAS and LAPACK running config Wrote config.h In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_convmodule.c:13: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_sortmodule.c:13: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_bytesmodule.c:13: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, ... several dozen of the above, then ... Packages/LinearAlgebra2/Src/lapack_litemodule.c:12: warning: function declaration isn't a prototype Packages/LinearAlgebra2/Src/lapack_litemodule.c:17: warning: function declaration isn't a prototype Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgeev': Packages/LinearAlgebra2/Src/lapack_litemodule.c:107: warning: implicit declaration of function 'dgeev_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dsyevd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:199: warning: implicit declaration of function 'dsyevd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zheevd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:296: warning: implicit declaration of function 'zheevd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgelsd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:340: warning: implicit declaration of function 'dgelsd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgesv': Packages/LinearAlgebra2/Src/lapack_litemodule.c:378: warning: implicit declaration of function 'dgesv_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgesdd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:418: warning: implicit declaration of function 'dgesdd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgetrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:449: warning: implicit declaration of function 'dgetrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zungqr': Packages/LinearAlgebra2/Src/lapack_litemodule.c:477: warning: implicit declaration of function 'zungqr_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dorgqr': Packages/LinearAlgebra2/Src/lapack_litemodule.c:508: warning: implicit declaration of function 'dorgqr_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgeqrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:539: warning: implicit declaration of function 'zgeqrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dgeqrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:570: warning: implicit declaration of function 'dgeqrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_dpotrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:594: warning: implicit declaration of function 'dpotrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgeev': Packages/LinearAlgebra2/Src/lapack_litemodule.c:632: warning: implicit declaration of function 'zgeev_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgelsd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:679: warning: implicit declaration of function 'zgelsd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgesv': Packages/LinearAlgebra2/Src/lapack_litemodule.c:714: warning: implicit declaration of function 'zgesv_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgesdd': Packages/LinearAlgebra2/Src/lapack_litemodule.c:759: warning: implicit declaration of function 'zgesdd_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zgetrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:794: warning: implicit declaration of function 'zgetrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function 'lapack_lite_zpotrf': Packages/LinearAlgebra2/Src/lapack_litemodule.c:818: warning: implicit declaration of function 'zpotrf_' Packages/LinearAlgebra2/Src/lapack_litemodule.c: At top level: Packages/LinearAlgebra2/Src/lapack_litemodule.c:851: warning: function declaration isn't a prototype Packages/LinearAlgebra2/Src/lapack_litemodule.c:848: warning: 'lapack_liteError' defined but not used powerpc-apple-darwin8-gcc-4.0.0: -framework: linker input file unused because linking not done powerpc-apple-darwin8-gcc-4.0.0: vecLib: linker input file unused because linking not done In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Packages/RandomArray2/Src/ranlib.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Packages/RandomArray2/Src/ranlibmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Packages/RandomArray2/Src/com.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Src/_dotblas.c:12: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from Src/_dotblas.c:15: /System/Library/Frameworks/vecLib.framework/Headers/cblas.h:665: warning: function declaration isn't a prototype In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/high_levelmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/element_wisemodule.c:3: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/one_dimensionalmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/numpy_compatmodule.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/numpy_compat2.c:1: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list In file included from /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, from Examples/convolve/testlite.c:26: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: warning: 'struct winsize' declared inside parameter list /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: warning: 'struct winsize' declared inside parameter list Any ideas for a solution to this problem are welcome. -- Chris Fonnesbeck Atlanta, GA From Kalimeroc5 at wanadoo.fr Wed Nov 2 07:34:07 2005 From: Kalimeroc5 at wanadoo.fr (Philou) Date: Wed Nov 2 07:34:07 2005 Subject: [Numpy-discussion] enumerate indices and values of a multidimensionnal numarray.array Message-ID: <4368DC1C.4070905@wanadoo.fr> Hi list, I'm a new subsciber to this numpy list. I have a multidimensionnal array (>2). For example: x = numarray.array([[[1,2,3],[4,5,6]],[[1,2,3],[4,5,6]]]) print x [[[1 2 3] [4 5 6]] [[1 2 3] [4 5 6]]] I need to enumerate both indices and final values (1D) of the x array. So i need: (0,0,0) 1 (0,0,1) 2 (0,0,2) 2 (0,1,0) 4 (0,1,1) 5 (0,1,2) 6 (1,0,0) 1 (1,0,1) 2 How can i do without using the x.flat() method? Thanks a lot for the answer. regards, Philippe Collet From Chris.Barker at noaa.gov Wed Nov 2 09:44:07 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Nov 2 09:44:07 2005 Subject: [Numpy-discussion] enumerate indices and values of a multidimensionnal numarray.array In-Reply-To: <4368DC1C.4070905@wanadoo.fr> References: <4368DC1C.4070905@wanadoo.fr> Message-ID: <4368FAA8.4010503@noaa.gov> There was a discussion about this last month: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2850560 You should get some good ideas from that. -Chris Philou wrote: > Hi list, > I'm a new subsciber to this numpy list. > > I have a multidimensionnal array (>2). > For example: > x = numarray.array([[[1,2,3],[4,5,6]],[[1,2,3],[4,5,6]]]) > > print x > [[[1 2 3] > [4 5 6]] > > [[1 2 3] > [4 5 6]]] > > I need to enumerate both indices and final values (1D) of the x array. > So i need: > (0,0,0) 1 > (0,0,1) 2 > (0,0,2) 2 > (0,1,0) 4 > (0,1,1) 5 > (0,1,2) 6 > (1,0,0) 1 > (1,0,1) 2 > > How can i do without using the x.flat() method? > > Thanks a lot for the answer. > regards, > Philippe Collet > > > > > ------------------------------------------------------- > SF.Net email is sponsored by: > Tame your development challenges with Apache's Geronimo App Server. > Download > it for free - -and be entered to win a 42" plasma tv or your very own > Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmiller at stsci.edu Wed Nov 2 14:17:24 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed Nov 2 14:17:24 2005 Subject: [Numpy-discussion] Problem with array assignment In-Reply-To: <200511021406.56504.faltet@carabos.com> References: <200511021406.56504.faltet@carabos.com> Message-ID: <43693AB0.7000100@stsci.edu> I added support for the newcore "array interface" to the numarray C-API and this exception is now gone. The fix/enhancement is checked into CVS. Kick it around some... Todd Francesc Altet wrote: >Hi, > >I've recently upgraded to Numeric 24.0 and numarray 1.4.1 and I'm >having a problem that did not exist before: > > > >>>>import Numeric >>>>import numarray >>>>Numeric.__version__ >>>> >>>> >'24.0' > > >>>>numarray.__version__ >>>> >>>> >'1.4.1' > > >>>>na=numarray.zeros((3,),'f') >>>>num=Numeric.zeros((3,),'f') >>>>na[...] = num >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >ValueError: array sizes must be consistent. > >but... > > > >>>>na=numarray.zeros((3,),'i') >>>>num=Numeric.zeros((3,),'i') >>>>na[...] = num >>>>na >>>> >>>> >array([0, 0, 0]) > >????? > >However, using Numeric 23.8 and numarray 1.3.3: > > > >>>>import Numeric >>>>import numarray >>>>Numeric.__version__ >>>> >>>> >'23.8' > > >>>>numarray.__version__ >>>> >>>> >'1.3.3' > > >>>>na=numarray.zeros((3,),'f') >>>>num=Numeric.zeros((3,),'f') >>>>na[...] = num >>>>na >>>> >>>> >array([ 0., 0., 0.], type=Float32) > >I don't know if the problem is coming from Numeric or numarray, but it >only happens when one mix both: > > > >>>>Numeric.__version__ >>>> >>>> >'24.0' > > >>>>numarray.__version__ >>>> >>>> >'1.4.1' > > >>>>na=numarray.zeros((3,),'f') >>>>na2=numarray.zeros((3,),'f') >>>>na2[...] = na >>>>na2 >>>> >>>> >array([ 0., 0., 0.], type=Float32) > > >>>>num=Numeric.zeros((3,),'f') >>>>num2=Numeric.zeros((3,),'f') >>>>num2[...] = num >>>>num >>>> >>>> >array([ 0., 0., 0.],'f') > >Perhaps the new array protocol is involved here? > > > From rowen at cesmail.net Wed Nov 2 14:53:09 2005 From: rowen at cesmail.net (Russell E. Owen) Date: Wed Nov 2 14:53:09 2005 Subject: [Numpy-discussion] Re: byteswap questions References: <1130838789.2968.23.camel@localhost.localdomain> Message-ID: In article <1130838789.2968.23.camel at localhost.localdomain>, Francesc Altet wrote: > El dl 31 de 10 del 2005 a les 16:53 -0800, en/na Russell E. Owen va > escriure: > > I've got a module that can send array data to ds9. It screws up on > > byteswapped data and I'm trying to fix it. > > I don't exactly know what a ds9 is, but I'm supposing here that it is > kind of an exotic computer. It's an image display program. > > I need to know if the order is bigendian or littleendian, and I can't > > find a documented way to get that, just an undocumented attribute > > byteorder. > > Yes, it's undocumented, but as far as I can tell, this works flawlessly > in numarray. I'm sure it works now. I was hoping to know if it would continue working in future versions. It'd be nice to have a documented way to get at the info. >... > In general, copy() will return you a well-behaved (native-byte ordered, > non-strided and non-offsetted) array. The manual promises that a copy will be contiguous but I didn't see a promise of native byte order. I have submitted a documentation PR on sourceforge. ... Thanks for the explanation of what byteswapped was doing. That was very helpful. To use documented interfaces (i.e. not arra._byteorder) and to avoid byteswapping the input array, I think I'm going to be stuck doing something like: import sys if sys.byteorder == 'big': isBigendian = not arr.isbyteswapped() else: isBigendian = arr.isbyteswapped() I also submitted a request for a documented direct way to get the info on sourceforge. Regards, -- Russell From faltet at carabos.com Thu Nov 3 00:33:18 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Nov 3 00:33:18 2005 Subject: [Numpy-discussion] Problem with array assignment In-Reply-To: <43693AB0.7000100@stsci.edu> References: <200511021406.56504.faltet@carabos.com> <43693AB0.7000100@stsci.edu> Message-ID: <1131007743.2968.29.camel@localhost.localdomain> El dc 02 de 11 del 2005 a les 17:16 -0500, en/na Todd Miller va escriure: > I added support for the newcore "array interface" to the numarray C-API > and this exception is now gone. The fix/enhancement is checked into > CVS. Kick it around some... Yes. This issue has been solved: >>> numarray.__version__ '1.4.2' >>> Numeric.__version__ '24.0' >>> num=Numeric.zeros((3,),'f') >>> na=numarray.ones((3,),'f') >>> na[...] = num >>> na array([ 0., 0., 0.], type=Float32) Now, I've taken the opportunity and tested the Numeric<-->numarray conversion again: >>> t1_1=timeit.Timer("num=Numeric.array(na,copy=1)","import Numeric;import numarray; na=numarray.arange(10)") >>> t1_1.repeat(3,10000) [0.25493311882019043, 0.25352001190185547, 0.25211095809936523] and this figures compared with numarray 1.4.1: >>> t1.repeat(3,10000) [0.59375977516174316, 0.57908082008361816, 0.56574010848999023] shows that you have accelerated the conversion numarray-->Numeric more than 2x. Now, the Numeric-->numarray: >>> t3_1=timeit.Timer("na=numarray.array(num,copy=1)","import Numeric;import numarray; num=Numeric.arange(10)") >>> t3_1.repeat(3,10000) [0.40311408042907715, 0.40207314491271973, 0.39954400062561035] while the 1.4.1 performance was: >>> t3.repeat(3,10000) [1.3475611209869385, 1.3277668952941895, 1.3417830467224121] which is a factor 3x of improvement, and almost the same than using the buffer interface. Very nice job, Todd! Now, let's test the conversion for large objects, with data copy and without data copy. For numarray-->Numeric: >>> t2=timeit.Timer("num=Numeric.array(na,copy=0)","import Numeric;import numarray; na=numarray.arange(100000)") >>> t2.repeat(3,100) [0.0025169849395751953, 0.0025219917297363281, 0.0024950504302978516] >>> t2_copy=timeit.Timer("num=Numeric.array(na,copy=1)","import Numeric;import numarray; na=numarray.arange(100000)") >>> t2_copy.repeat(3,100) [0.50105500221252441, 0.49400091171264648, 0.49266600608825684] So it seems like if the data copy is taking place. For Numeric-->numarray: >>> t4=timeit.Timer("na=numarray.array(num,copy=0)","import Numeric;import numarray; num=Numeric.arange(100000)") >>> t4.repeat(3,100) [0.0054900646209716797, 0.0044760704040527344, 0.0048201084136962891] >>> t4_copy=timeit.Timer("na=numarray.array(num,copy=1)","import Numeric;importnumarray; num=Numeric.arange(100000)") >>> t4_copy.repeat(3,100) [0.0063569545745849609, 0.004302978515625, 0.0042738914489746094] Ooops! the times for conversion with copy and without copy are more or less the same. Perhaps the copy is not done? It seems so: >>> num=Numeric.arange(10) >>> na=numarray.array(num,copy=0) >>> na_copy=numarray.array(num,copy=1) >>> na.info() ... data pointer: 0x08312280 (DEBUG ONLY) >>> na_copy.info() ... data pointer: 0x08312280 (DEBUG ONLY) i.e. the same data pointer. So it seems that the Numeric-->numarray is not copying the data even in the case that we are asking to do that. Other minor things (maybe this is just because you are in the middle of refactoring): >>> na._data array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) while I would expect: >>> na._data Also: >>> na Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 930, in __repr__ return array_repr(self) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 1660, in array_repr lst = arrayprint.array2string( File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line 188, in array2string separator, prefix) File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line 140, in _array2string data = _gen.ravel(a) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 918, in copy c = _gen.NDArray.copy(self) File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in copy arr._itemsize) TypeError: NA_maybeLongsFromIntTuple: must be a sequence of integers. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From rlw at stsci.edu Thu Nov 3 06:01:59 2005 From: rlw at stsci.edu (Rick White) Date: Thu Nov 3 06:01:59 2005 Subject: [Numpy-discussion] Re: byteswap questions In-Reply-To: References: <1130838789.2968.23.camel@localhost.localdomain> Message-ID: On Wed, 2 Nov 2005, Russell E. Owen wrote: > To use documented interfaces (i.e. not arra._byteorder) and to avoid > byteswapping the input array, I think I'm going to be stuck doing > something like: > > import sys > if sys.byteorder == 'big': > isBigendian = not arr.isbyteswapped() > else: > isBigendian = arr.isbyteswapped() A simpler version: isBigendian = arr.isbyteswapped() != numarray.isBigEndian From jmiller at stsci.edu Thu Nov 3 10:10:05 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 3 10:10:05 2005 Subject: [Numpy-discussion] 1.4.1 build failure under OSX In-Reply-To: <723eb6930511020555w5ec42098na5d1502b5608990d@mail.gmail.com> References: <723eb6930511020555w5ec42098na5d1502b5608990d@mail.gmail.com> Message-ID: <436A522B.2080901@stsci.edu> Chris Fonnesbeck wrote: >Hi, > >Compiling 1.4.1 under OSX with either gcc 3.3 or 4.0 fails (all >previous versions of numarray compiled without error). Here is the >full output: > > > What version of OS-X are your running? Where is the actual error? (There are a lot of warnings now on OS-X, but 1.4.1 builds fine for me.) Todd >Using EXTRA_COMPILE_ARGS = ['-Ddarwin'] >Using external BLAS and LAPACK >running config >Wrote config.h >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Src/_convmodule.c:13: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Src/_sortmodule.c:13: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Src/_bytesmodule.c:13: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > >... several dozen of the above, then ... > >Packages/LinearAlgebra2/Src/lapack_litemodule.c:12: warning: function >declaration isn't a prototype >Packages/LinearAlgebra2/Src/lapack_litemodule.c:17: warning: function >declaration isn't a prototype >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgeev': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:107: warning: implicit >declaration of function 'dgeev_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dsyevd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:199: warning: implicit >declaration of function 'dsyevd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zheevd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:296: warning: implicit >declaration of function 'zheevd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgelsd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:340: warning: implicit >declaration of function 'dgelsd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgesv': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:378: warning: implicit >declaration of function 'dgesv_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgesdd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:418: warning: implicit >declaration of function 'dgesdd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgetrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:449: warning: implicit >declaration of function 'dgetrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zungqr': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:477: warning: implicit >declaration of function 'zungqr_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dorgqr': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:508: warning: implicit >declaration of function 'dorgqr_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgeqrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:539: warning: implicit >declaration of function 'zgeqrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dgeqrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:570: warning: implicit >declaration of function 'dgeqrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_dpotrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:594: warning: implicit >declaration of function 'dpotrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgeev': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:632: warning: implicit >declaration of function 'zgeev_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgelsd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:679: warning: implicit >declaration of function 'zgelsd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgesv': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:714: warning: implicit >declaration of function 'zgesv_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgesdd': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:759: warning: implicit >declaration of function 'zgesdd_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zgetrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:794: warning: implicit >declaration of function 'zgetrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: In function >'lapack_lite_zpotrf': >Packages/LinearAlgebra2/Src/lapack_litemodule.c:818: warning: implicit >declaration of function 'zpotrf_' >Packages/LinearAlgebra2/Src/lapack_litemodule.c: At top level: >Packages/LinearAlgebra2/Src/lapack_litemodule.c:851: warning: function >declaration isn't a prototype >Packages/LinearAlgebra2/Src/lapack_litemodule.c:848: warning: >'lapack_liteError' defined but not used >powerpc-apple-darwin8-gcc-4.0.0: -framework: linker input file unused >because linking not done >powerpc-apple-darwin8-gcc-4.0.0: vecLib: linker input file unused >because linking not done >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Packages/RandomArray2/Src/ranlib.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Packages/RandomArray2/Src/ranlibmodule.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Packages/RandomArray2/Src/com.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Src/_dotblas.c:12: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from Src/_dotblas.c:15: >/System/Library/Frameworks/vecLib.framework/Headers/cblas.h:665: >warning: function declaration isn't a prototype >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/high_levelmodule.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/element_wisemodule.c:3: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/one_dimensionalmodule.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/numpy_compatmodule.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/numpy_compat2.c:1: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list >In file included from >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/Python.h:55, > from Examples/convolve/testlite.c:26: >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:396: >warning: 'struct winsize' declared inside parameter list >/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/pyport.h:397: >warning: 'struct winsize' declared inside parameter list > > >Any ideas for a solution to this problem are welcome. > >-- >Chris Fonnesbeck >Atlanta, GA > > >------------------------------------------------------- >SF.Net email is sponsored by: >Tame your development challenges with Apache's Geronimo App Server. Download >it for free - -and be entered to win a 42" plasma tv or your very own >Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From rowen at cesmail.net Thu Nov 3 12:05:57 2005 From: rowen at cesmail.net (Russell E. Owen) Date: Thu Nov 3 12:05:57 2005 Subject: [Numpy-discussion] Re: byteswap questions References: <1130838789.2968.23.camel@localhost.localdomain> Message-ID: In article , Rick White wrote: > On Wed, 2 Nov 2005, Russell E. Owen wrote: > > > To use documented interfaces (i.e. not arra._byteorder) and to avoid > > byteswapping the input array, I think I'm going to be stuck doing > > something like: > > > > import sys > > if sys.byteorder == 'big': > > isBigendian = not arr.isbyteswapped() > > else: > > isBigendian = arr.isbyteswapped() > > A simpler version: > > isBigendian = arr.isbyteswapped() != numarray.isBigEndian Cute. isBigEndian doesn't seem to be present in Numeric 23.8. I wonder if it'll be in scipy.core. Based on stunningly fast responses by Jay T Miller to most of the PRs I submitted to sourceforge: - The byteswap and byteswapped methods do just affect the data and not the byteorder flag. The doc string for byteswapped said otherwise and that is fixed in the repository. I don't know if the manual has been clarified. - To toggle the byteorder flag, use the togglebyteorder method (duh, I wish I'd seen that one). - An official way to figure out if an array is byteswapped is to use the __array_typestr__ method and look at the first character of the returned string. "<" means little-endian, ">" means big-endian. This is apparently also part of the array interface for scipy.core. I sure hope there'll be a more obvious method to call than __array_typestr__ at some point. -- Russell From jmiller at stsci.edu Thu Nov 3 12:07:21 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 3 12:07:21 2005 Subject: [Numpy-discussion] Problem with array assignment In-Reply-To: <1131007743.2968.29.camel@localhost.localdomain> References: <200511021406.56504.faltet@carabos.com> <43693AB0.7000100@stsci.edu> <1131007743.2968.29.camel@localhost.localdomain> Message-ID: <436A6D98.8080708@stsci.edu> Francesc Altet wrote: >Now, let's test the conversion for large objects, with data copy and >without data copy. > > I didn't consider that case but was thinking the semantics of the array interface would be "buffer-like" and therefore non-copying. So at the moment numarray *never* copies and completely ignores the non-sequence array() parameters for the "array interface" case. >For numarray-->Numeric: > > > >>>>t2=timeit.Timer("num=Numeric.array(na,copy=0)","import >>>> >>>> >Numeric;import numarray; na=numarray.arange(100000)") > > >>>>t2.repeat(3,100) >>>> >>>> >[0.0025169849395751953, 0.0025219917297363281, 0.0024950504302978516] > > >>>>t2_copy=timeit.Timer("num=Numeric.array(na,copy=1)","import >>>> >>>> >Numeric;import numarray; na=numarray.arange(100000)") > > >>>>t2_copy.repeat(3,100) >>>> >>>> >[0.50105500221252441, 0.49400091171264648, 0.49266600608825684] > >So it seems like if the data copy is taking place. > > > I agree and was to be able to verify this by making buffer()s out of the resulting arrays to examine their data addresses. >For Numeric-->numarray: > > > >>>>t4=timeit.Timer("na=numarray.array(num,copy=0)","import >>>> >>>> >Numeric;import numarray; num=Numeric.arange(100000)") > > >>>>t4.repeat(3,100) >>>> >>>> >[0.0054900646209716797, 0.0044760704040527344, 0.0048201084136962891] > > >>>>t4_copy=timeit.Timer("na=numarray.array(num,copy=1)","import >>>> >>>> >Numeric;importnumarray; num=Numeric.arange(100000)") > > >>>>t4_copy.repeat(3,100) >>>> >>>> >[0.0063569545745849609, 0.004302978515625, 0.0042738914489746094] > >Ooops! the times for conversion with copy and without copy are more or >less the same. Perhaps the copy is not done? It seems so: > > > >>>>num=Numeric.arange(10) >>>>na=numarray.array(num,copy=0) >>>>na_copy=numarray.array(num,copy=1) >>>>na.info() >>>> >>>> >... >data pointer: 0x08312280 (DEBUG ONLY) > > >>>>na_copy.info() >>>> >>>> >... >data pointer: 0x08312280 (DEBUG ONLY) > >i.e. the same data pointer. So it seems that the Numeric-->numarray is >not copying the data even in the case that we are asking to do that. > > Yeah, the copy=X flag is totally ignored at the moment. >Other minor things (maybe this is just because you are in the middle of >refactoring): > > > >>>>na._data >>>> >>>> >array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > >while I would expect: > > > >>>>na._data >>>> >>>> >aliasing object 0x00000000> > >Also: > > This is actually correct I think. Since a Numeric array supports the buffer protocol, it can be used as a buffer. The "memory" output shown above is the output from one particular kind of buffer; i.e. it's the repr of numarray's memory object. > > >>>>na >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >930, in __repr__ > return array_repr(self) > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >1660, in array_repr > lst = arrayprint.array2string( > File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line >188, in array2string > separator, prefix) > File "/usr/lib/python2.4/site-packages/numarray/arrayprint.py", line >140, in _array2string > data = _gen.ravel(a) > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >918, in copy > c = _gen.NDArray.copy(self) > File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, >in copy > arr._itemsize) >TypeError: NA_maybeLongsFromIntTuple: must be a sequence of integers. > > For some reason Numeric was returning None for __array_strides__ and hence this message. I fixed array() so that strides are not set when __array_strides__ returns None. With the added complexity of handling the shape, type, and copy parameters in array(), I'm now seeing performance like this from CVS: numarray-->Numeric array_if: [0.40103912353515625, 0.41119098663330078, 0.41148614883422852] numarray-->Numeric fromstring: [0.24139499664306641, 0.27034401893615723, 0.21657609939575195] Numeric-->numarray array_if: [0.81009912490844727, 0.79906082153320312, 0.74358081817626953] Numeric-->numarray buffer_if: [0.27320504188537598, 0.22041201591491699, 0.22667884826660156] Todd -------------- next part -------------- A non-text attachment was scrubbed... Name: arrayif_bench.py Type: application/x-python Size: 1292 bytes Desc: not available URL: From Chris.Barker at noaa.gov Thu Nov 3 12:08:55 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Nov 3 12:08:55 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric Message-ID: <436A6E0F.6080800@noaa.gov> Hi all, I suddenly seem to have the need for working with a bunch of different existing C and C++ code, so I'm looking for a way to make it easier. I love NumPy, so I really want to use NumPy arrays My needs fall into three categories: 1) Writing custom extension code from scratch: In the past, I've done this by just using the NumPy API, but it seems that I shouldn't have to do all that book keeping myself. 2) Wrapping existing libraries: At the moment, I'd like to wrap Richard Shewchuk's triangle: http://www.cs.cmu.edu/~quake/triangle.html Which is straight C, as well as the Proj4 map projections library, also C, and there may be others. 3) Wrapping in house code, C & C++, that is under development, but I want to be able to use it from Python and C++, and also perhaps to write tests for it in Python. The options I'm looking at are: Pyrex: This seems like perhaps the easiest way to write extension code, but it doesn't do any automatic wrapping. Boost::Python: This looks like a very nice way to write extension modules, and it even appears to already include support for NumPy arrays (which ones? is that support being maintained, and will it support SciPy.base?) SWIG: The has the major advantage of automatically wrapping code. I see this as a particular strength for wrapping our in house code that is under development. In theory, once I've written a bunch of type maps, i can just re-run it whenever the code base changes. For each of these, I'd love to hear what people's experiences have been, and it would be great if anyone can point me to samples that are small but non-trivial. Other options I should consider would be great too. One more question: Should I use NumPy arrays to pass data back and forth between Python and C, or should I use the new array object/protocol instead? If so, has anyone developed examples, SWIG type maps, etc for this? Thanks for any feedback, -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Thu Nov 3 12:34:10 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Nov 3 12:34:10 2005 Subject: [Numpy-discussion] Re: byteswap questions In-Reply-To: References: <1130838789.2968.23.camel@localhost.localdomain> Message-ID: <436A73FC.3070000@ee.byu.edu> Russell E. Owen wrote: >Cute. isBigEndian doesn't seem to be present in Numeric 23.8. I wonder >if it'll be in scipy.core. > > No, in Numeric the arrays are always assumed to be in machine byte order and so sys.byteorder tells all (Numeric.LittleEndian is a Boolean that is also defined). In scipy core, arr.flags.swapped tells you if the bytes are in swapped order. Then, sys.byteorder or scipy.LittleEndian indicates the machine order. Or you use the array_interface descriptor. is_big_endian = (arr.dtypestr[0] == '>') -Travis From jmiller at stsci.edu Thu Nov 3 13:26:35 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 3 13:26:35 2005 Subject: [Numpy-discussion] More on array protocol In-Reply-To: <1130921630.3101.11.camel@localhost.localdomain> References: <1130921630.3101.11.camel@localhost.localdomain> Message-ID: <436A804A.9010402@stsci.edu> Francesc Altet wrote: >Hi, > >I'm having problems in using the array protocol in numarray-->Numeric in >some situations. I'm using the PyBuffer_FromReadWriteMemory C call in >order to create buffers that will be later used to create numarray >objects (both NumArray and CharArray) in Python space. The problem is >that the array protocol implementation is doing badly this conversion. > >I was able to create a small example in pure python that reproduces the >problem: > > > >>>>import numarray >>>>from numarray import memory >>>>import Numeric >>>>na=numarray.array([1,2]) >>>>Numeric.array(na) >>>> >>>> >array([1, 2]) > >but ... > > > >>>>wbuf = memory.writeable_buffer(na) >>>>nawb = numarray.array(wbuf,type="Int32",shape=(2,)) >>>>nawb >>>> >>>> >array([1, 2]) > > >>>>Numeric.array(nawb) >>>> >>>> >array([ 2, 135510112]) > >I'm afraid that the problem should be related with the kind of buffer >that is attached to the numarray object: > > I can reproduce garbage here using 1.4.1 but not with CVS. >>>>na._data >>>> >>>> >aliasing object 0x00000000> > > >>>>nawb._data >>>> >>>> > > >Travis, Todd, are you going to support read-write buffers or should I >try to move to using the memory module in order to cope with this? For a >series of reasons I'd like to keep using regular read-write buffers, but >anyway... I think I'll be able to do the change if absolutely necessary. > > I pretty sure this problem has been solved... let me know if you're still experiencing problems here. Todd From bsouthey at gmail.com Thu Nov 3 14:33:26 2005 From: bsouthey at gmail.com (Bruce Southey) Date: Thu Nov 3 14:33:26 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: <436A6E0F.6080800@noaa.gov> References: <436A6E0F.6080800@noaa.gov> Message-ID: Hi, I found SIWG extremely to use but it only exposes a function to Python but not to numpy. Thus it is very slow for matrix functions. So if you want speed then you will have to deal with the APIs. Regards Bruce On 11/3/05, Chris Barker wrote: > > Hi all, > > I suddenly seem to have the need for working with a bunch of different > existing C and C++ code, so I'm looking for a way to make it easier. I > love NumPy, so I really want to use NumPy arrays My needs fall into > three categories: > > 1) Writing custom extension code from scratch: > > In the past, I've done this by just using the NumPy API, but it seems > that I shouldn't have to do all that book keeping myself. > > 2) Wrapping existing libraries: > > At the moment, I'd like to wrap Richard Shewchuk's triangle: > > http://www.cs.cmu.edu/~quake/triangle.html > > Which is straight C, as well as the Proj4 map projections library, also > C, and there may be others. > > 3) Wrapping in house code, C & C++, that is under development, but I > want to be able to use it from Python and C++, and also perhaps to write > tests for it in Python. > > The options I'm looking at are: > > Pyrex: > > This seems like perhaps the easiest way to write extension code, but it > doesn't do any automatic wrapping. > > Boost::Python: > > This looks like a very nice way to write extension modules, and it even > appears to already include support for NumPy arrays (which ones? is that > support being maintained, and will it support SciPy.base?) > > SWIG: > > The has the major advantage of automatically wrapping code. I see this > as a particular strength for wrapping our in house code that is under > development. In theory, once I've written a bunch of type maps, i can > just re-run it whenever the code base changes. > > > For each of these, I'd love to hear what people's experiences have been, > and it would be great if anyone can point me to samples that are small > but non-trivial. Other options I should consider would be great too. > > > One more question: Should I use NumPy arrays to pass data back and forth > between Python and C, or should I use the new array object/protocol > instead? If so, has anyone developed examples, SWIG type maps, etc for > this? > > Thanks for any feedback, > > -Chris > > > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > ------------------------------------------------------- > SF.Net email is sponsored by: > Tame your development challenges with Apache's Geronimo App Server. > Download > it for free - -and be entered to win a 42" plasma tv or your very own > Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Thu Nov 3 15:43:51 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Nov 3 15:43:51 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: References: <436A6E0F.6080800@noaa.gov> Message-ID: <436AA07B.5000306@noaa.gov> Bruce Southey wrote: > Hi, > I found SWIG extremely to use but it only exposes a function to Python but > not to numpy. Thus it is very slow for matrix functions. So if you want > speed then you will have to deal with the APIs. Yes, but can't I deal with them in writing typemaps, and then let SWIG do the rest of the work? I think I've seen some examples like this somewhere, but it's been a while and I need to go digging more. Anyone got one? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rays at blue-cove.com Thu Nov 3 17:11:32 2005 From: rays at blue-cove.com (RayS) Date: Thu Nov 3 17:11:32 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric Message-ID: <6.2.3.4.2.20051103170446.066db3e0@pop-server.san.rr.com> Have you looked at ctypes code generator? http://starship.python.net/crew/theller/ctypes/codegen.html "The generator converts declarations in C header files into executable Python code: enums, structs, unions, function declarations, com interfaces, and preprocessor definitions." It has progressed well in the last several months, in my experience. I did some tests a while back that found it compared well with weave etc. http://sourceforge.net/mailarchive/forum.php?thread_id=7636431&forum_id=24606 Ray Schumacher From oliphant at ee.byu.edu Thu Nov 3 22:51:52 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Nov 3 22:51:52 2005 Subject: [Numpy-discussion] Final release of Numeric is actually 24.1 Message-ID: <436B04DF.9050005@ee.byu.edu> I made a final release of Numeric (24.1). This is "really" the last release of Numeric ;-) I did it because of David Cooke's excellent array_protocol enhancement of __array_struct__. I wanted to make sure the final Numeric fully supported the array protocol including the "faster version." I also tracked down a bug today in scipy core that was inherited from Numeric. A part of me wanted to not fix the bug in Numeric so that people who need a stable platform will move to scipy core :-) But the better part of me won, and I fixed the problem and made a new Numeric release. I do hope people are encouraged to move toward scipy core, however. It is stabilizing quite rapidly. All of scipy now builds with the new scipy core, and all of (full) scipy's 1300 tests pass for both 32-bit and 64-bit systems. I will be making a release of scipy_core (probably called version 0.6 this weekend). It is still missing finished records.py, ma.py, and chararray.py modules (being worked on). When these are available I will make release of scipy core version 1.0 Best regards, -Travis From stefan at sun.ac.za Thu Nov 3 23:38:21 2005 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu Nov 3 23:38:21 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: <436AA07B.5000306@noaa.gov> References: <436A6E0F.6080800@noaa.gov> <436AA07B.5000306@noaa.gov> Message-ID: <20051104073622.GA24072@sun.ac.za> Hi Chris On Thu, Nov 03, 2005 at 03:42:51PM -0800, Chris Barker wrote: > Bruce Southey wrote: > >Hi, > >I found SWIG extremely to use but it only exposes a function to Python but > >not to numpy. Thus it is very slow for matrix functions. So if you want > >speed then you will have to deal with the APIs. > > Yes, but can't I deal with them in writing typemaps, and then let SWIG > do the rest of the work? I think I've seen some examples like this > somewhere, but it's been a while and I need to go digging more. I use SWIG and Blitz++ this way, and it works well. I modified Fernando's typemap to work with templates. See attached (does this need to be modified for Numeric3?). It is easy to create a Blitz matrix from a Numeric Array without copying data. Unfortunately, Blitz jealously guards its data (restricted pointers), so that it is not so easy to do the conversion in the other direction. If anyone knows an answer to this problem, I'd be glad to hear it. St??fan -------------- next part -------------- // -*- C++ -*- %{ #include #include #include %} namespace blitz { template class Array { %typemap(in) Array & (Array M) { int T_size = sizeof(T); blitz::TinyVector shape(0); blitz::TinyVector strides(0); int *arr_dimensions = ((PyArrayObject*)$input)->dimensions; int *arr_strides = ((PyArrayObject*)$input)->strides; for (int i = 0; i < N; i++) { shape[i] = arr_dimensions[i]; strides[i] = arr_strides[i]/T_size; } M.reference(blitz::Array((T*) ((PyArrayObject*)$input)->data, shape, strides, blitz::neverDeleteData)); $1 = &M; } }; } %template(matrix1d) blitz::Array; %template(matrix2d) blitz::Array; %template(matrix3d) blitz::Array; %template(matrix1f) blitz::Array; %template(matrix2f) blitz::Array; %template(matrix3f) blitz::Array; From oliphant at ee.byu.edu Thu Nov 3 23:44:23 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Nov 3 23:44:23 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: <20051104073622.GA24072@sun.ac.za> References: <436A6E0F.6080800@noaa.gov> <436AA07B.5000306@noaa.gov> <20051104073622.GA24072@sun.ac.za> Message-ID: <436B1104.9040500@ee.byu.edu> Stefan van der Walt wrote: >Hi Chris > >On Thu, Nov 03, 2005 at 03:42:51PM -0800, Chris Barker wrote: > > >>Bruce Southey wrote: >> >> >>>Hi, >>>I found SWIG extremely to use but it only exposes a function to Python but >>>not to numpy. Thus it is very slow for matrix functions. So if you want >>>speed then you will have to deal with the APIs. >>> >>> >>Yes, but can't I deal with them in writing typemaps, and then let SWIG >>do the rest of the work? I think I've seen some examples like this >>somewhere, but it's been a while and I need to go digging more. >> >> > >I use SWIG and Blitz++ this way, and it works well. I modified >Fernando's typemap to work with templates. See attached (does >this need to be modified for Numeric3?). > > > Only a little bit. I'll mark the changes >St??fan > > >------------------------------------------------------------------------ > >// -*- C++ -*- > >%{ >#include >#include >#include > > #include "scipy/arrayobject.h" >%} > >namespace blitz { > > template class Array { > > %typemap(in) Array & (Array M) { > int T_size = sizeof(T); > > blitz::TinyVector shape(0); > blitz::TinyVector strides(0); > > int *arr_dimensions = ((PyArrayObject*)$input)->dimensions; > int *arr_strides = ((PyArrayObject*)$input)->strides; > > for (int i = 0; i < N; i++) { > shape[i] = arr_dimensions[i]; > strides[i] = arr_strides[i]/T_size; > } > > M.reference(blitz::Array((T*) ((PyArrayObject*)$input)->data, > shape, strides, blitz::neverDeleteData)); > > $1 = &M; > } > > }; >} > >%template(matrix1d) blitz::Array; >%template(matrix2d) blitz::Array; >%template(matrix3d) blitz::Array; > >%template(matrix1f) blitz::Array; >%template(matrix2f) blitz::Array; >%template(matrix3f) blitz::Array; > > On 32-bit platforms, this code would work fine. On 64-bit platforms, you would need to change at least the arr_dimensions and arr_strides arrays to be intp (typedef to an integer the size of a pointer on the platform). -Travis From chri at botan.su.se Fri Nov 4 03:43:59 2005 From: chri at botan.su.se (Chrissy Lanctot) Date: Fri Nov 4 03:43:59 2005 Subject: [Numpy-discussion] Re: Adams to you Message-ID: <000501c5e135$229a9700$579ea8c0@clerestory> Go overpay or your Meddicat isit our Pharm ss Sh od days, Quit ing f ions - v aExpre op. Get additional information - http://starterxlq.presetey.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Fri Nov 4 06:03:45 2005 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Fri Nov 4 06:03:45 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: <436AA07B.5000306@noaa.gov> References: <436A6E0F.6080800@noaa.gov> <436AA07B.5000306@noaa.gov> Message-ID: <1131096570.30402.6.camel@localhost.localdomain> Chris, I have written an wrapper to some COIN-OR libraries for Python using Swig. In this process I have developed some simple typemaps for numarrays array (floating point arrays). It worked flawlessly. I haven't changed the code for a while, but you may want to take a look at it, specially the numarray.i file where I define the numarray typemaps: http://www.ime.usp.br/~pjssilva/software.html Good luck, Paulo From faltet at carabos.com Fri Nov 4 08:40:23 2005 From: faltet at carabos.com (Francesc Altet) Date: Fri Nov 4 08:40:23 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray Message-ID: <200511041739.40801.faltet@carabos.com> Hi, I've detected another problem in CVS numarray when converting non-contiguous Numeric objects. With numarray 1.3.3 the next works: >>> Numeric.__version__ '23.8' >>> numarray.__version__ '1.3.3' >>> num=Numeric.array([1,2,3,4],'b') >>> num2=num[::2] >>> num2.iscontiguous() 0 >>> na=numarray.array(num2) >>> na array([1, 3]) but, with CVS version of numarray: >>> numarray.__version__ '1.4.2' >>> Numeric.__version__ '24.1' >>> na=numarray.array([1,2,3,4],'b') >>> num=Numeric.array([1,2,3,4],'b') >>> num2=num[::2] >>> num2.iscontiguous() 0 >>> na=numarray.array(num2) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 376, in array a = a.astype(type) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 863, in astype return self.copy() File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 923, in copy c = _gen.NDArray.copy(self) File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in copy arr._itemsize) numarray.libnumarray.error: copy1bytes: access beyond buffer. offset=2 buffersize=2 Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From paustin at eos.ubc.ca Fri Nov 4 09:21:23 2005 From: paustin at eos.ubc.ca (Philip Austin) Date: Fri Nov 4 09:21:23 2005 Subject: [Numpy-discussion] Options for wrapping C and C++ code for use with Numeric In-Reply-To: <436AA07B.5000306@noaa.gov> References: <436A6E0F.6080800@noaa.gov> <436AA07B.5000306@noaa.gov> Message-ID: <17259.38980.663173.839733@owl.eos.ubc.ca> I'm planning to update my boost numeric wrappers (http://www.eos.ubc.ca/research/clouds/num_util.html) to scipy_core as soon as things settle down. Here's an example of some fortran program wrapped by boost that shows a couple of nice features: i) mirroring of python types in C++, with indexing, etc., ii) transparent memory management (all increfs and decrefs are handled by boost shared pointers) and iii) docstrings. I'd be happy to contribute to a page with some SWIG and boost examples for similar code fragments. #define PY_ARRAY_UNIQUE_SYMBOL PyArrayHandle #include #include #include "boost/scoped_array.hpp" namespace { const char* rcsid = ("@(#) $Id: thermwrap.C,v 1.2 2005/09/05 22:47:57 phil Exp $:"); }; using namespace std; namespace py = boost::python; namespace nbpl = num_util; typedef int wave_unit; extern "C" { void ptqv_(float& p,float& t, float& qv, float& psp, float& tsp, float& qsp, float& thsp, float& thesp); void ptpsp_(float& p,float& t, float& qv, float& psp, float& tsp, float& qsp, float& thpt,float& thes, float& thsp, float& thesp); void mccla_(const char* atmos, float* z, float* p, float* t,float* rvden, float* o3den, float* den, int& strnlength); void thinv_(float& p,float& t,float& thsp); void theinv_(float& p,float& t, float& qsp, float& thsp, float& thesp); } py::dict ptqv(float p, float t, float qv) { float psp,tsp,qsp,thsp,thesp; ptqv_(p,t,qv,psp,tsp,qsp,thsp,thesp); py::dict result; result["psp"]=psp; result["tsp"]=tsp; result["qsp"]=qsp; result["thsp"]=thsp; result["thesp"]=thesp; return result; } py::dict ptpsp(float p, float t, float psp) { float qv,tsp,qsp,thpt,thes,thsp,thesp; ptpsp_(p,t,qv,psp,tsp,qsp,thpt,thes,thsp,thesp); py::dict result; result["qv"]=qv; result["tsp"]=tsp; result["qsp"]=qsp; result["thpt"]=thpt; result["thes"]=thes; result["thsp"]=thsp; result["thesp"]=thesp; return result; } py::dict thinv(float p, float thsp) { float t; thinv_(p,t,thsp); py::dict result; result["t"]=t; return result; } py::dict theinv(float p, float t, float the) { //t input is first guess float qv,thsp; theinv_(p,t,qv,thsp,the); py::dict result; result["t"]=t; result["qsp"]=qv; result["thsp"]=thsp; return result; } py::numeric::array mcclatchey(string atmos) { int strlength=atmos.length(); int length=33; boost::scoped_array z(new float[length]); boost::scoped_array p(new float[length]); boost::scoped_array t(new float[length]); boost::scoped_array rvden(new float[length]); boost::scoped_array o3den(new float[length]); boost::scoped_array den(new float[length]); mccla_(atmos.c_str(), z.get(), p.get(), t.get(), rvden.get(), o3den.get(), den.get(),strlength ); std::vector dims; dims.push_back(6); dims.push_back(33); py::numeric::array standsound(nbpl::makeNum(dims, PyArray_DOUBLE)); double* sndPtr=(double*) nbpl::data(standsound); int index; for(int i=0;i<33;++i){ index=i; sndPtr[index]=z[i]; index=33 + i; sndPtr[index]=p[i]; index=2*33 + i; sndPtr[index]=t[i]; index=3*33 + i; sndPtr[index]=rvden[i]; index=4*33 + i; sndPtr[index]=o3den[i]; index=5*33 + i; sndPtr[index]=den[i]; } return standsound; } BOOST_PYTHON_MODULE(thermwrap) { using namespace boost::python; import_array(); numeric::array::set_module_and_type("Numeric", "ArrayType"); scope().attr("RCSID") = rcsid; scope().attr("__doc__") = "wrappers for BettsThermo.f routines: ptqv, ptpsp"; string docstring("input: pressure (hPa), temp (K), qv (kg/kg)\n"); docstring += "output psp (hPa), tsp (K), qsp (kg/kg), thsp (K), thesp (K)"; def("ptqv", ptqv,docstring.c_str()); docstring="input: pressure (hPa), temp (K), psp (hPa)\n"; docstring += "output: qv (kg/kg), tsp (K), qsp (kg/kg), thpt (theta(p,t)), thes (K), \n"; docstring +="thsp (theta(psl,tsl), thesp equiv. potential temp.(theta-e at sl)"; def("ptpsp",ptpsp,docstring.c_str()); def("thinv",thinv,"thinv"); def("theinv",theinv,"theinv"); def("mcclatchey",mcclatchey,"mcclatchey"); } From shulee at gmail.com Fri Nov 4 12:17:27 2005 From: shulee at gmail.com (Shu Li) Date: Fri Nov 4 12:17:27 2005 Subject: [Numpy-discussion] How to install SciPy Core Message-ID: <2fd52c3d0511041216w237b0d92v2e0e6cbeb488d1da@mail.gmail.com> Hi, I am a little confused about how to install SciPy Core. I am new to SciPy so I think starting with the new core is a good idea. When I installed the core package first and then scipy, in the process of installing scipy, it showed it also installed a lot of stuff to the core directory which made me worry that part of the new core is overwritten by old core. And when I installed scipy first, then some words on the web saying some "__init__.py" file will "break the scipy" certainly worried me because I don't know what the break here means and why people are not fixing it if it is avoidable. Also even when new core(0.4.2 beta) is installed the scipy still complained not seeing "Numeric", which added to the confusion. Could anybody offer some explanation? Thanks a lot! -- best, Sam (AKA Shu Li) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Fri Nov 4 12:36:58 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 4 12:36:58 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray In-Reply-To: <200511041739.40801.faltet@carabos.com> References: <200511041739.40801.faltet@carabos.com> Message-ID: <436BC62F.1010104@stsci.edu> This turned out to be a problem with the way numarray handles Numeric's multi-segment buffer protocol. I worked around this by implementing David Cooke's __array_struct__ array interface for numarray. I'm now seeing times like these: numarray-->Numeric array_if: [0.20523881912231445, 0.19724917411804199, 0.17538094520568848] numarray-->Numeric fromstring: [0.28353691101074219, 0.21145009994506836, 0.22541189193725586] Numeric-->numarray array_if: [0.40105700492858887, 0.34915614128112793, 0.39094209671020508] Numeric-->numarray buffer_if: [0.29201602935791016, 0.25055789947509766, 0.23227095603942871] As you can see, the numarray-->Numeric array_if is now faster than fromstring() and Numeric-->numarray array_if is now getting close to the time for constructing a numarray from a buffer. Todd Francesc Altet wrote: >Hi, > >I've detected another problem in CVS numarray when converting >non-contiguous Numeric objects. > >With numarray 1.3.3 the next works: > > > >>>>Numeric.__version__ >>>> >>>> >'23.8' > > >>>>numarray.__version__ >>>> >>>> >'1.3.3' > > >>>>num=Numeric.array([1,2,3,4],'b') >>>>num2=num[::2] >>>>num2.iscontiguous() >>>> >>>> >0 > > >>>>na=numarray.array(num2) >>>>na >>>> >>>> >array([1, 3]) > >but, with CVS version of numarray: > > > >>>>numarray.__version__ >>>> >>>> >'1.4.2' > > >>>>Numeric.__version__ >>>> >>>> >'24.1' > > >>>>na=numarray.array([1,2,3,4],'b') >>>>num=Numeric.array([1,2,3,4],'b') >>>>num2=num[::2] >>>>num2.iscontiguous() >>>> >>>> >0 > > >>>>na=numarray.array(num2) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 376, >in array > a = a.astype(type) > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 863, >in astype > return self.copy() > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 923, >in copy > c = _gen.NDArray.copy(self) > File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in >copy > arr._itemsize) >numarray.libnumarray.error: copy1bytes: access beyond buffer. offset=2 >buffersize=2 > >Cheers, > > > From dd55 at cornell.edu Sat Nov 5 13:47:55 2005 From: dd55 at cornell.edu (Darren Dale) Date: Sat Nov 5 13:47:55 2005 Subject: [Numpy-discussion] question about dotblas and ACML Message-ID: <200511051647.15381.dd55@cornell.edu> I just got a new Athlon 64 bit system. I'm in the process of building Numeric-24.1 against the AMD Core Math Library. Does anyone know if it is possible to enable the use_dotblas option in customize.py with ACML? Thanks, Darren From dd55 at cornell.edu Sat Nov 5 16:31:42 2005 From: dd55 at cornell.edu (Darren Dale) Date: Sat Nov 5 16:31:42 2005 Subject: [Numpy-discussion] numpy 24.x on 64-bit Athlon Message-ID: <200511051931.20596.dd55@cornell.edu> I just built Numeric 24.0 and 24.1 against ATLAS on a 64 bit athlon system. I get the following error for both packages. python test.py ........E.................................... ====================================================================== ERROR: Test the diagonal function. ---------------------------------------------------------------------- Traceback (most recent call last): File "test.py", line 584, in testDiagonal assert_eq(Numeric.diagonal(c,1), [[2,7,4], [2,7,4]]) File "test.py", line 28, in assert_eq assert eq(a,b) File "test.py", line 23, in eq raise ValueError("sequences have different shapes:\na%s=%r\nb%s=%r" % ValueError: sequences have different shapes: a(4, 2)=array([[5, 1], [6, 2], [7, 3], [8, 4]]) b(2, 3)=[[2, 7, 4], [2, 7, 4]] I also get the following error when I run newscipy's test(5,10): bench_random (scipy.fftpack.basic.test_basic.test_fft) Fast Fourier Transform ================================================= | real input | complex input ------------------------------------------------- size | scipy | Numeric | scipy | Numeric ------------------------------------------------- 100 | 0.10Segmentation fault Has anyone else had a problem building Numeric-24.x on a 64-bit AMD system? Thanks, Darren From dd55 at cornell.edu Sat Nov 5 16:41:33 2005 From: dd55 at cornell.edu (Darren Dale) Date: Sat Nov 5 16:41:33 2005 Subject: [Numpy-discussion] numpy 24.x on 64-bit Athlon In-Reply-To: <200511051931.20596.dd55@cornell.edu> References: <200511051931.20596.dd55@cornell.edu> Message-ID: <200511051941.04464.dd55@cornell.edu> On Saturday 05 November 2005 07:31 pm, Darren Dale wrote: > I just built Numeric 24.0 and 24.1 against ATLAS on a 64 bit athlon system. [...] > I also get the following error when I run newscipy's test(5,10): > > bench_random (scipy.fftpack.basic.test_basic.test_fft) > Fast Fourier Transform > ================================================= > > | real input | complex input > > ------------------------------------------------- > size | scipy | Numeric | scipy | Numeric > ------------------------------------------------- > 100 | 0.10Segmentation fault I should have noted that I only get the segfault from Scipy with Numeric 24.1. Darren From oliphant at ee.byu.edu Sun Nov 6 21:04:06 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sun Nov 6 21:04:06 2005 Subject: [Numpy-discussion] numpy 24.x on 64-bit Athlon In-Reply-To: <200511051931.20596.dd55@cornell.edu> References: <200511051931.20596.dd55@cornell.edu> Message-ID: <436EE012.5040206@ee.byu.edu> Darren Dale wrote: >I just built Numeric 24.0 and 24.1 against ATLAS on a 64 bit athlon system. I >get the following error for both packages. > >python test.py >........E.................................... >====================================================================== >ERROR: Test the diagonal function. >---------------------------------------------------------------------- >Traceback (most recent call last): > File "test.py", line 584, in testDiagonal > assert_eq(Numeric.diagonal(c,1), [[2,7,4], [2,7,4]]) > File "test.py", line 28, in assert_eq > assert eq(a,b) > File "test.py", line 23, in eq > raise ValueError("sequences have different shapes:\na%s=%r\nb%s=%r" % >ValueError: sequences have different shapes: >a(4, 2)=array([[5, 1], > [6, 2], > [7, 3], > [8, 4]]) >b(2, 3)=[[2, 7, 4], [2, 7, 4]] > > This error is a test problem not worth bothering with. The (>2d) diagonal function was always broken under Numeric. It was changed a while ago. The test is broken, but this doesn't mean anything is wrong with Numeric. >I also get the following error when I run newscipy's test(5,10): > >bench_random (scipy.fftpack.basic.test_basic.test_fft) > Fast Fourier Transform >================================================= > | real input | complex input >------------------------------------------------- > size | scipy | Numeric | scipy | Numeric >------------------------------------------------- > > > 100 | 0.10Segmentation fault > > Curious. Did you rebuild scipy completely? -Travis From dd55 at cornell.edu Mon Nov 7 05:12:52 2005 From: dd55 at cornell.edu (Darren Dale) Date: Mon Nov 7 05:12:52 2005 Subject: [Numpy-discussion] numpy 24.x on 64-bit Athlon In-Reply-To: <436EE012.5040206@ee.byu.edu> References: <200511051931.20596.dd55@cornell.edu> <436EE012.5040206@ee.byu.edu> Message-ID: <200511070811.08485.dd55@cornell.edu> On Monday 07 November 2005 12:03 am, Travis Oliphant wrote: > >I also get the following error when I run newscipy's test(5,10): > > > >bench_random (scipy.fftpack.basic.test_basic.test_fft) > > Fast Fourier Transform > >================================================= > > > > | real input | complex input > > > >------------------------------------------------- > > size | scipy | Numeric | scipy | Numeric > >------------------------------------------------- > > > > > > 100 | 0.10Segmentation fault > > Curious. Did you rebuild scipy completely? Yes, in fact it was the first build on a new computer. I just removed my build/, and site-packages/scipy and Numeric/ directories, rebuilt and reinstalled Numeric 24.1 and newcore/newscipy and still observe the segfault. I also tried installing Scipy-0.3.2, which did not have a problem with Numeric-24.1. Darren From khinsen at cea.fr Mon Nov 7 06:50:51 2005 From: khinsen at cea.fr (khinsen at cea.fr) Date: Mon Nov 7 06:50:51 2005 Subject: [Numpy-discussion] Final release of Numeric is actually 24.1 In-Reply-To: <436B04DF.9050005@ee.byu.edu> References: <436B04DF.9050005@ee.byu.edu> Message-ID: <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> On Nov 4, 2005, at 7:51, Travis Oliphant wrote: > I do hope people are encouraged to move toward scipy core, > however. It is stabilizing Has anyone actually done this for a non-trivial package? What stops me from even trying scipy.core is the large number of incompatible changes, even though they are mostly superficial. I cannot give up Numeric compatibility because all of my user base is using Numeric, and will likely stick to Numeric at least until scipy.core is out of beta, possibly longer. In the meantime, I would have to maintain two versions of all my code (nearly all of my Python modules are concerned by some of the changes), which is not something I am looking forward to. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From managan at llnl.gov Mon Nov 7 09:04:03 2005 From: managan at llnl.gov (Rob Managan) Date: Mon Nov 7 09:04:03 2005 Subject: [Numpy-discussion] bus error in check_kurtosis Message-ID: With todays svn versions (newcore 1440, newscipy 1423) scipy.test(level=1,verbosity=2) dies with a bus error on check_kurtosis check_basic (scipy.stats.stats.test_stats.test_mean) ... ok check_ravel (scipy.stats.stats.test_stats.test_mean) ... ok check_basic (scipy.stats.stats.test_stats.test_median) ... ok check_basic (scipy.stats.stats.test_stats.test_mode) ... ok check_kurtosis (scipy.stats.stats.test_stats.test_moments)Bus error This is on Mac OSX 10.3.9, python 2.4.1, gcc 3.3. Is this a known problem? -- *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- Rob Managan email managan at llnl.gov LLNL phone: 925-423-0903 P.O. Box 808, L-095 FAX: 925-422-3389 Livermore, CA 94551-0808 From oliphant at ee.byu.edu Mon Nov 7 09:10:51 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Nov 7 09:10:51 2005 Subject: [Numpy-discussion] Final release of Numeric is actually 24.1 In-Reply-To: <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> References: <436B04DF.9050005@ee.byu.edu> <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> Message-ID: <436F8A65.4070902@ee.byu.edu> khinsen at cea.fr wrote: > On Nov 4, 2005, at 7:51, Travis Oliphant wrote: > >> I do hope people are encouraged to move toward scipy core, however. >> It is stabilizing > > > Has anyone actually done this for a non-trivial package? What stops > me from even trying scipy.core is the large number of incompatible > changes, even though they are mostly superficial. There is a module called convertcode.py (in scipy/base) that will make nearly all of the required changes. We used it to convert all of scipy over and it was pretty effective. I can understand an installed base may find it more difficult to switch. That's why we are working hard to get scipy core out of beta as soon as possible. Immediately, we are really looking for people who are willing to run their code so we can track down remaining problems. -Travis From oliphant at ee.byu.edu Mon Nov 7 11:10:20 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Nov 7 11:10:20 2005 Subject: [Numpy-discussion] Numeric 24.1 tar ball and CVS were not consistent Message-ID: <436FA65C.3010202@ee.byu.edu> I just updated the CVS of Numeric with changes that went into the Numeric 24.1 release but were (unfortunately) not committed to CVS. These changes are likely responsible for the test failures that people have experienced when testing scipy with CVS versions of Numeric. Sorry about --- too many things to think about apparently... -Travis From strawman at astraw.com Mon Nov 7 12:10:47 2005 From: strawman at astraw.com (Andrew Straw) Date: Mon Nov 7 12:10:47 2005 Subject: [Numpy-discussion] Numeric 24.1 tar ball and CVS were not consistent In-Reply-To: <436FA65C.3010202@ee.byu.edu> References: <436FA65C.3010202@ee.byu.edu> Message-ID: <436FB48E.1030705@astraw.com> Hi Travis, The bug I reported[1] was originally detected with the Numeric 24.1 tar ball, but then I "upgraded" to CVS just to be sure that it hadn't been fixed. [1]: http://www.scipy.net/pipermail/scipy-dev/2005-November/003883.html So, I don't think the issue has been fixed. Let me know if you need more detailed logs, test output, whatever. Cheers! Andrew Travis Oliphant wrote: > > I just updated the CVS of Numeric with changes that went into the > Numeric 24.1 release but were (unfortunately) not committed to CVS. > These changes are likely responsible for the test failures that people > have experienced when testing scipy with CVS versions of Numeric. > > Sorry about --- too many things to think about apparently... > > -Travis > > > > > ------------------------------------------------------- > SF.Net email is sponsored by: > Tame your development challenges with Apache's Geronimo App Server. > Download > it for free - -and be entered to win a 42" plasma tv or your very own > Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From faltet at carabos.com Mon Nov 7 23:54:36 2005 From: faltet at carabos.com (Francesc Altet) Date: Mon Nov 7 23:54:36 2005 Subject: [Numpy-discussion] Final release of Numeric is actually 24.1 In-Reply-To: <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> References: <436B04DF.9050005@ee.byu.edu> <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> Message-ID: <1131437475.3010.13.camel@localhost.localdomain> El dl 07 de 11 del 2005 a les 15:50 +0100, en/na khinsen at cea.fr va escriure: > Has anyone actually done this for a non-trivial package? What stops > me from even trying scipy.core is the large number of incompatible > changes, even though they are mostly superficial. Well, I would like to migrate PyTables to scipy.core in the medium term. My plan to not bother the users is to start supporting scipy.base by using the array protocol. That means that, although numarray will still be at the core, users will be able to send and retrieve scipy.core objects into the library with almost native speed (thanks to the array protocol). When scipy.core eventually matures enough, I'll put it at the core of PyTables, while still supporting numarray and Numeric, and the users will (hopefully) hardly notice the change, becuase they will be able to use whatever numeric library they want, and the migration will be up to them. I really think that the array protocol (that already is supported in latest Numeric and numarray, bar some small issues) is a Good Thing (tm) to consider, specially for the developers. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From khinsen at cea.fr Tue Nov 8 00:12:57 2005 From: khinsen at cea.fr (khinsen at cea.fr) Date: Tue Nov 8 00:12:57 2005 Subject: [Numpy-discussion] Final release of Numeric is actually 24.1 In-Reply-To: <1131437475.3010.13.camel@localhost.localdomain> References: <436B04DF.9050005@ee.byu.edu> <0914ED30-AECF-4BF8-90B1-EF31318C4DC1@cea.fr> <1131437475.3010.13.camel@localhost.localdomain> Message-ID: On 08.11.2005, at 09:11, Francesc Altet wrote: > Well, I would like to migrate PyTables to scipy.core in the medium > term. > My plan to not bother the users is to start supporting scipy.base by > using the array protocol. That means that, although numarray will > still ... > I really think that the array protocol (that already is supported in > latest Numeric and numarray, bar some small issues) is a Good Thing > (tm) > to consider, specially for the developers. True, but it doesn't solve my problem (I think). Users of my code don't see much of an array interface, it's hidden behind application- oriented abstraction layers. That also means that users don't care much if I use Numeric, numarray, or scipy.core, as long as there are no installation problems. However, my code is necessarily tied to one specific array package because it creates all of its arrays. It also does lots of array operations that go beyond the array protocol. So I have lots of dependencies on the chosen array package in many different modules. My problem is handling the transition period while I test scipy.core and while scipy.core stabilizes. I expect to make releases of my packages during that time, so I need to keep a code base that works with Numeric. Doing development on two code bases in parallel seems like asking for trouble. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr ------------------------------------------------------------------------ ------- From pontifor at yahoo.com Tue Nov 8 07:55:22 2005 From: pontifor at yahoo.com (Stefan Kuzminski) Date: Tue Nov 8 07:55:22 2005 Subject: [Numpy-discussion] initializing new ufuncs Message-ID: <20051108154423.23553.qmail@web50602.mail.yahoo.com> Hi, I'm writing some new ufuncs, and I'm wondering where I can insert some initialization and finalization code for each ufunc invocation. I can see where it would go in the generated code, but I don't know how to do it through the code generation API. thanks, Stefan Kuzminski __________________________________ Start your day with Yahoo! - Make it your home page! http://www.yahoo.com/r/hs From mithrandir42 at web.de Tue Nov 8 22:31:37 2005 From: mithrandir42 at web.de (N. Volbers) Date: Tue Nov 8 22:31:37 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file Message-ID: <437198A3.9020705@web.de> Hello everyone, I have been using Numeric and/or numarray for a while now and I now I have a question concerning the reading and writing data to a file. Until now I had used pycdf, which is an interface to the netcdf library, to save and load my numeric arrays. However, since my application already has too many dependencies, I would like to cut down on these two and replace the saving and loading by a method intrinisc to Numeric or numarray. Pickling is not an alternative to me, because I need to have a representation that can be read by other programs (and yet is faster than ASCII). NetCDF was a good choice for that, but I would really like to not depend on it. So my questions are: (1) Are the numarray functions 'fromfile' and 'tofile' 100% portable among platforms, i.e. do they automatically recognize endian-ness and such? (2) Does it make sense to still use numarray? I know Travis would say "use scipy_core". However, for me this would provide much unneeded functionality and I have not yet found an easy way to install scipy_core (it seems to require ATLAS and such, which are not so easy to install if you don't have it prepackaged). And after all, my goal is to cut down dependencies... (3) Let me restate question (2): Will numarray still be maintained? Or is it also deprecated? What would you advice someone who just needs the array interface? And of course, (4) What solutions do you use to save/load data to files? I would deeply appreciate any help on this subject, Niklas Volbers. From jmiller at stsci.edu Wed Nov 9 02:26:49 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed Nov 9 02:26:49 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <437198A3.9020705@web.de> References: <437198A3.9020705@web.de> Message-ID: <4371CE82.9080608@stsci.edu> N. Volbers wrote: > Hello everyone, > > I have been using Numeric and/or numarray for a while now and I now I > have a question concerning the reading and writing data to a file. > > Until now I had used pycdf, which is an interface to the netcdf > library, to save and load my numeric arrays. However, since my > application already has too many dependencies, I would like to cut > down on these two and replace the saving and loading by a method > intrinisc to Numeric or numarray. > Pickling is not an alternative to me, because I need to have a > representation that can be read by other programs (and yet is faster > than ASCII). NetCDF was a good choice for that, but I would really > like to not depend on it. > > So my questions are: > > (1) Are the numarray functions 'fromfile' and 'tofile' 100% portable > among platforms, i.e. do they automatically recognize endian-ness and > such? No. But depending on how general a solution you need here, this can be fairly easy (i.e. for numerical arrays only). > (2) Does it make sense to still use numarray? Absolutely... numarray works with records and memory mapping now. But you also need to keep a shrewd eye on scipy newcore and recognize that it will most probably replace numarray over the course of the next year or two as it becomes sufficiently complete and stable to do so. There are smart things to do now if you want to use numarray: a. Use the Numeric-compatible C-API as much as possible. b. Keep an eye on the introduction of newcore compatible typenames (int32 vs Int32), keywords (dtype vs. type), and attributes and use those as you write new code in numarray. c. Use the array protocol. > I know Travis would say "use scipy_core". However, for me this would > provide much unneeded functionality and I have not yet found an easy > way to install scipy_core (it seems to require ATLAS and such, which > are not so easy to install if you don't have it prepackaged). And > after all, my goal is to cut down dependencies... > > (3) Let me restate question (2): Will numarray still be maintained? numarray will be maintained at STScI until (a) newcore is ready to replace it or (b) our budget gets cut to the point that we cannot and no one else is interested. Neither of those is guaranteed to happen, but (a) looks likely to us. STScI has the same problems with installation and dependencies so they'll have to be solved before we use newcore either. > Or is it also deprecated? What would you advice someone who just > needs the array interface? Pay careful attention to the __array_struct__ attribute described here: http://numeric.scipy.org/array_interface.html It's the easiest and best performing method to interface from C. To interface from Python you have to use more of the protocol. > And of course, > > (4) What solutions do you use to save/load data to files? numarray was written to support astronomical data processing. The dominant data format in astronomy is called FITS. STScI has another package called PyFITS which is built on numarray and exposes the FITS format to Python. Regards, Todd From cjw at sympatico.ca Wed Nov 9 06:40:10 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Nov 9 06:40:10 2005 Subject: [Numpy-discussion] Numeric3.0 - Currently available under scipy/base as version 0.4.1 (Windows) Message-ID: <437209BF.8090607@sympatico.ca> I have a package based on subclassing numarray, which is working satisfactorily, and am looking at the possibility of transferring the package to work under the revised Numeric. My feeling is that the transfer is probably feasible but that it would be premature to work on it at this time. One of the problems is the cluttered namespace, through the use of "from X import *". This is a style which is deprecated, see page 401 of Alex Martelli's /Python in a Nutshell/. Another problem, at this stage, is that many doc strings are missing and that some which exist are a little cryptic. I would like to take another look when the next win32 binaries are available. Some further thoughts on the present state of Numeric3.0 are available here . Colin W, From strawman at astraw.com Wed Nov 9 10:33:40 2005 From: strawman at astraw.com (Andrew Straw) Date: Wed Nov 9 10:33:40 2005 Subject: [Numpy-discussion] Numeric CVS __array_struct__ interface is broken on 64 bit platforms Message-ID: <437240DD.9060203@astraw.com> Hi, A couple bugs have been reported (including mine last night) which indicate a problem with the following bit of code in Numerical/Src/arrayobject.c (near line 2200) on 64 bit platforms. I think it'll be a lot faster for someone else to fix it, so I'll leave it at this for now. if (strcmp(name, "__array_struct__") == 0) { PyArrayInterface *inter; inter = (PyArrayInterface *)malloc(sizeof(PyArrayInterface)); inter->version = 2; inter->nd = self->nd; if ((inter->nd == 0) || (sizeof(int) == sizeof(Py_intptr_t))) { inter->shape = (Py_intptr_t *)self->dimensions; inter->strides = (Py_intptr_t *)self->strides; } else { int i; for (i=0; ind; i++) { inter->shape[i] = self->dimensions[i]; inter->strides[i] = self->strides[i]; } } From oliphant at ee.byu.edu Wed Nov 9 11:22:27 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Nov 9 11:22:27 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <437198A3.9020705@web.de> References: <437198A3.9020705@web.de> Message-ID: <43724C52.7030704@ee.byu.edu> N. Volbers wrote: > (2) Does it make sense to still use numarray? I know Travis would say > "use scipy_core". However, for me this would provide much unneeded > functionality and I have not yet found an easy way to install > scipy_core (it seems to require ATLAS and such, which are not so easy > to install if you don't have it prepackaged). And after all, my goal > is to cut down dependencies... > What unneeded functionality is there. SciPy Core is just a Numeric replacement. It does not NEED Atlas, it just uses it if you have it --- exactly the same as Numeric and numarray. It is a misconception to say scipy core needs any thing else but Python installed. So, let's not let that rumor kill convergence to a single package. > > (4) What solutions do you use to save/load data to files? SciPy core arrays have tofile methods and a fromfile function. They are raw reading and writing --- nothing fancy. You need to use Pickling if you want to recognize endian-ness among platforms. What is your opposition to Pickling? I think you should take a look at PyTables for more elegant solutions. -Travis From oliphant at ee.byu.edu Wed Nov 9 12:35:45 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Nov 9 12:35:45 2005 Subject: [Numpy-discussion] Numeric3.0 - Currently available under scipy/base as version 0.4.1 (Windows) In-Reply-To: <437209BF.8090607@sympatico.ca> References: <437209BF.8090607@sympatico.ca> Message-ID: <43725D8A.9090307@ee.byu.edu> Colin J. Williams wrote: > > I have a package based on subclassing numarray, which is working > satisfactorily, and am looking at the possibility of transferring the > package to work under the revised Numeric. > > My feeling is that the transfer is probably feasible but that it would > be premature to work on it at this time. That's unfortunate. The more feedback we get early on about subclassing, the better. > > One of the problems is the cluttered namespace, through the use of > "from X import *". This is a style which is deprecated, see page 401 > of Alex Martelli's /Python in a Nutshell/. You will have to be more specific about what you think is wrong. What namespace is "cluttered" exactly. Just because use is made of from X import * in one module does not mean everything is "cluttered". SciPy Core makes use of the __all__ variables to control what gets imported and usually only specific functions are imported as necessary. > Another problem, at this stage, is that many doc strings are missing > and that some which exist are a little cryptic. I would submit there are more docstrings then Numeric had. Jump in and help. The water is fine. > > I would like to take another look when the next win32 binaries are > available. There has been much improvement since the last beta. I'm trying to track down some remaining memory leaks before releasing new windows binaries. The SVN code is always available for check out and it is quite easy to build. We could always use more build testers to make sure building is going as smoothly as we believe it is. > > Some further thoughts on the present state of Numeric3.0 are available > here . Most of your comments have more to do with differences between builtin types and Python classes than anything about scipy. The type-class chasm has shrunken of late, but there are still remaining issues. These are Python issues that I believe are of little concern. I will comment on your issues that are not related to the above comment: Use of package __init__.py to create namespace. If the epydoc and pydoc tools are not respecting the __init__.py code then I would say they are broken. Using the __init__.py this way frees the user from having to understand every little detail of the package structure (which could also change as better organization is obtained in the future). Use of the from X import Y style Please give more support here. Just because one Python user advocates against it is not sufficient argument. There is an argument to be made for avoiding attribute lookups by importing the names directly in this fashion. *Methods vs functions* I agree that methods and functions are somewhat redundant. However, the functions are still needed both to support lists and especially for back-wards compatibility. Your example using take is odd (perhaps it's a bug in an old release). I get consistent behavior. One problem is you define a 1-d array in one case and a 2-d array in another. Of course the answers will be different. One difference is that the default axis argument is different. The old functions have exactly the same default axis argument as they always did, while the methods have a default of None for the axis (which means treat the array as flat). Lack of basic information in the doc strings Your examples are all class constructors. First of all, there is no special __init__ method for the ndarray because __new__ takes care of it. Second of all, the new method does need better documentation. I'm not sure where to put it, though. The array_new function is placed in the TypeObject of the array type. The __new__ attribute is pasted on by PyTypeReady. I'm not sure how to give a docstring as well. I suspect the proper thing to do is place the information in the docstring for the Type Object. -Travis From cookedm at physics.mcmaster.ca Wed Nov 9 20:41:56 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Nov 9 20:41:56 2005 Subject: [Numpy-discussion] Python egg support in Numeric 24.2 in CVS Message-ID: <4800A2A2-7B62-4B0C-9596-37956F5FCCFD@physics.mcmaster.ca> I've added support to Numeric to allow a Python egg [1] to be made easily. Basically, run $ python setup.py bdist_egg and an .egg file should be laid in dist/, which you can install with easy_install. This will only work if you have setuptools installed. I've taken steps so that usual 'setup.py build; setup.py install' will still install Numeric the old way (with a .pth file). The header files for Numeric are now also installed in a package Numeric_headers, which contains a function get_numeric_include(). This returns the directory in which the headers are installed in, so you can use it in your setup.py. The docstring for the function explains how to use it. It's modelled after get_scipy_include() in scipy.base. This way, the header files are included in the egg, and can be used. I'd appreciate it if someone with eggperience (Robert?) could take a look at it before Travis collects Numeric for 24.2. [1] http://peak.telecommunity.com/DevCenter/PythonEggs -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From bahn at atomistix.com Thu Nov 10 01:53:15 2005 From: bahn at atomistix.com (Sune Rastad Bahn) Date: Thu Nov 10 01:53:15 2005 Subject: [Numpy-discussion] Bug or feature? Message-ID: <200511101052.05164.bahn@atomistix.com> Dear devs, Thanks a lot for a very nice and extremely useful python module. I recommend all my colleagues to try you package whenever they ask for a matlab like environment. This week one of these friends came by me with an oddity relating to the RandomArray module. try the following on your linux command prompt: while true; do python -c "from RandomArray import *;seed();print randint(100000)"; done and check the output. I get a steadily but slowly decreasing series of numbers. e.g: 4013 4013 4012 4012 4012 4012 4012 4012 4012 4012 4010 4010 4010 According to the documentation seed() should initiatilize from the system clock. If one checks the seed (using get_seed) it turns out to be the case indeed, but that it is only very slowly changing (once a second), and only on the second seed. e.g python -c "from RandomArray import *;seed();print get_seed()" (113161, 3079) (113161, 3079) (113161, 3080) (113161, 3080) (113161, 3080) (113161, 3080) My best guess is that this is the cause for the behavior seen above. This could be filed as a feature, but I believe it is not the behavior people would expect. At least the documentation should emphasize that two runs started within a short time ( upto 1s) will give the exact same result (experts on pseudo random numbers will know this, but the length of the time 1sec is surprisingly long). that the first random number is slowly varying (strongly correlated) with time (this is more surprising). One fix is to take one randomnumber and throw away, since the second number is not correlated with time (except from noted above). Best, Sune -- Software Application Manager Sune Rastad Bahn Phone: +45 35 320 638 Mobile: +45 23 455 997 Email: bahn at atomistix.com From bahn at atomistix.com Thu Nov 10 03:07:49 2005 From: bahn at atomistix.com (Sune Rastad Bahn) Date: Thu Nov 10 03:07:49 2005 Subject: [Numpy-discussion] BUG! Message-ID: <200511101206.38946.bahn@atomistix.com> Dear devs, Thanks a lot for a very nice and extremely useful python module. I recommend all my colleagues to try you package whenever they ask for a matlab like environment. This week one of these friends came by me and showed me a bug in RandomArray. try the following python -c "from RandomArray import *; seed(1874764637, 1152239787);print randint(0,1000)" and notice how the result is the upper limit (1000) even though the documentation clearly states that the number should be strictly below 1000. My guess is that this is an issue with the underlying ranlib library, but perhaps someone can clarify? best, Sune -- Software Application Manager Sune Rastad Bahn Phone: +45 35 320 638 Mobile: +45 23 455 997 Email: bahn at atomistix.com From joris at ster.kuleuven.be Thu Nov 10 03:58:00 2005 From: joris at ster.kuleuven.be (Joris De Ridder) Date: Thu Nov 10 03:58:00 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <43724C52.7030704@ee.byu.edu> References: <437198A3.9020705@web.de> <43724C52.7030704@ee.byu.edu> Message-ID: <200511101257.46704.joris@ster.kuleuven.be> On Wednesday 09 November 2005 20:21, Travis Oliphant wrote: > SciPy core arrays have tofile methods and a fromfile function. They are > raw reading and writing --- nothing fancy. You need to use Pickling if > you want to recognize endian-ness among platforms. What is your > opposition to Pickling? > > I think you should take a look at PyTables for more elegant solutions. The read_array and write_array of the old scipy.io are very handy. Any chance that they will be incorporated somehow in the new scipy? PyTables is powerful but perhaps a bit overkill if you just want to read or write a few columns in ascii format. Joris Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From arnd.baecker at web.de Thu Nov 10 04:31:46 2005 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu Nov 10 04:31:46 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <200511101257.46704.joris@ster.kuleuven.be> References: <437198A3.9020705@web.de> <43724C52.7030704@ee.byu.edu> <200511101257.46704.joris@ster.kuleuven.be> Message-ID: On Thu, 10 Nov 2005, Joris De Ridder wrote: > On Wednesday 09 November 2005 20:21, Travis Oliphant wrote: > > > SciPy core arrays have tofile methods and a fromfile function. They are > > raw reading and writing --- nothing fancy. You need to use Pickling if > > you want to recognize endian-ness among platforms. What is your > > opposition to Pickling? > > > > I think you should take a look at PyTables for more elegant solutions. > > > The read_array and write_array of the old scipy.io are very handy. > Any chance that they will be incorporated somehow in the new scipy? > PyTables is powerful but perhaps a bit overkill if you just want to > read or write a few columns in ascii format. They are already there in newscipy (as most of the other routines from "old" scipy). Best, Arnd From Mithrandir42 at web.de Thu Nov 10 04:35:10 2005 From: Mithrandir42 at web.de (Niklas Volbers) Date: Thu Nov 10 04:35:10 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file Message-ID: <1124237199@web.de> First of all, thanks for your answers. Travis Oliphant schrieb am 09.11.05 20:23:16: > What unneeded functionality is there. SciPy Core is just a Numeric > replacement. It does not NEED Atlas, it just uses it if you have it --- > exactly the same as Numeric and numarray. > It is a misconception to say scipy core needs any thing else but Python > installed. So, let's not let that rumor kill convergence to a single > package. I am sorry for this faulty assumption of mine which you have now clarified. I had assumed that ATLAS was needed, just because scipy_core would not build on my system w/o a prior install of these libraries. With the knowledge, that it _should_ work, I have just retried to build scipy_core 0.4.2 on my rather fresh installation of Slackware Linux 10.2 (python 2.4.1) and still couldn't get it to build. I have attached the log files out.txt and err.txt to this e-mail, which I got by typing $ python setup.py build 1>out.txt 2>err.txt in the scipy_core main directory. The two files are compressed into a single tar.gz archive (5k), because I do not know if 60k attachments are o.k. on this list. Maybe you can help me solve my problem? > > (4) What solutions do you use to save/load data to files? > > SciPy core arrays have tofile methods and a fromfile function. They are > raw reading and writing --- nothing fancy. You need to use Pickling if > you want to recognize endian-ness among platforms. What is your > opposition to Pickling? >From how I understood pickling, it converts the python objects to a binary representation. What I want is a data representation that is independent of python. Anyway, I now have a few possibilities to explore besides netcdf (with the alternative file formats FITS and with pytables, i.e. HDF5, proposed here). > I think you should take a look at PyTables for more elegant solutions. I wasn't sure if it will stay compatible to scipy_core, because right now it uses numarray. But from what you said I am sure that eventually pytables might also switch to scipy_core. Best regards, and thanks for your efforts, Niklas. ______________________________________________________________________ XXL-Speicher, PC-Virenschutz, Spartarife & mehr: Nur im WEB.DE Club! Jetzt gratis testen! http://freemail.web.de/home/landingpad/?mc=021130 -------------- next part -------------- A non-text attachment was scrubbed... Name: messages.tar.gz Type: application/x-gzip Size: 5241 bytes Desc: not available URL: From arnd.baecker at web.de Thu Nov 10 04:51:02 2005 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu Nov 10 04:51:02 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <1124237199@web.de> References: <1124237199@web.de> Message-ID: On Thu, 10 Nov 2005, Niklas Volbers wrote: > > First of all, thanks for your answers. > > Travis Oliphant schrieb am 09.11.05 20:23:16: > > > What unneeded functionality is there. SciPy Core is just a Numeric > > replacement. It does not NEED Atlas, it just uses it if you have it --- > > exactly the same as Numeric and numarray. > > It is a misconception to say scipy core needs any thing else but Python > > installed. So, let's not let that rumor kill convergence to a single > > package. > > I am sorry for this faulty assumption of mine which you have now clarified. > > I had assumed that ATLAS was needed, just because scipy_core would > not build on my system w/o a prior install of these libraries. > > With the knowledge, that it _should_ work, I have just retried to build > scipy_core 0.4.2 on my rather fresh installation of Slackware Linux > 10.2 (python 2.4.1) and still couldn't get it to build. I have > attached the log files out.txt and err.txt to this e-mail, which I got > by typing > > $ python setup.py build 1>out.txt 2>err.txt > > in the scipy_core main directory. The two files are compressed into a > single tar.gz archive (5k), because I do not know if 60k attachments are > o.k. on this list. > > Maybe you can help me solve my problem? Did you try a recent check out? With Travis' speed of coding scipy_core version 0.4.2.1252 seems pretty old to me I tested In [95]: scipy.__core_version__ Out[95]: '0.4.3.1440' this morning without problems. Best, Arnd From faltet at carabos.com Thu Nov 10 05:04:02 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Nov 10 05:04:02 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: <1124237199@web.de> References: <1124237199@web.de> Message-ID: <200511101403.41612.faltet@carabos.com> A Dijous 10 Novembre 2005 13:34, Niklas Volbers va escriure: > > I think you should take a look at PyTables for more elegant solutions. > > I wasn't sure if it will stay compatible to scipy_core, because right now > it uses numarray. But from what you said I am sure that eventually > pytables might also switch to scipy_core. Just to make things clear, PyTables uses numarray at its core, yes, but it does support Numeric also by doing conversions in the fastest way available (in fact, many people uses PyTables just for keeping Numeric arrays, with good results). Still, a copy in memory is needed for numarray-->Numeric conversions (not for the Numeric-->numarray way), but that will change very soon, after the array protocol implementation in numarray (and Numeric) stabilizes. The next step will be to offer support for scipy.core by using the array protocol as well. And finally, we plan (probably with the introduction of PyTables 2) to switch from numarray to scipy.core when the later would be stable enough, but always with support of the triad of numarray, Numeric and scipy.core, at least for a few years. Nevertheless, you must be aware that PyTables does require the HDF5 libraries [1], so, if what you want is to get rid of the maximum of dependencies, PyTables might be effectively overkill for you. [1] http://hdf.ncsa.uiuc.edu/HDF5/ Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From bsouthey at gmail.com Thu Nov 10 07:59:59 2005 From: bsouthey at gmail.com (Bruce Southey) Date: Thu Nov 10 07:59:59 2005 Subject: [Numpy-discussion] BUG! In-Reply-To: <200511101206.38946.bahn@atomistix.com> References: <200511101206.38946.bahn@atomistix.com> Message-ID: Hi, This appears to occur in numarray (1.3.3) but not Numeric (24.0b2). If you provide the shape of the array then this does not occur. Also, please note that your other email illustrates the well known problem of using computer clock as a seed - the computer clock does not change sufficiently fast enough so the seed remains the same. Regards Bruce On 11/10/05, Sune Rastad Bahn wrote: > > Dear devs, > > Thanks a lot for a very nice and extremely useful python module. > I recommend all my colleagues to try you package whenever they ask for a > matlab like environment. This week one of these friends came by me and > showed > me a bug in RandomArray. > > try the following > > python -c "from RandomArray import *; > seed(1874764637, 1152239787);print randint(0,1000)" > > and notice how the result is the upper limit (1000) > even though the documentation clearly states that the number should be > strictly below 1000. > > My guess is that this is an issue with the underlying ranlib library, but > perhaps someone can clarify? > > best, > Sune > > -- > Software Application Manager > Sune Rastad Bahn > Phone: +45 35 320 638 > Mobile: +45 23 455 997 > Email: bahn at atomistix.com > > > ------------------------------------------------------- > SF.Net email is sponsored by: > Tame your development challenges with Apache's Geronimo App Server. > Download > it for free - -and be entered to win a 42" plasma tv or your very own > Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Thu Nov 10 08:47:53 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 10 08:47:53 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray In-Reply-To: <1131193863.2849.8.camel@localhost.localdomain> References: <200511041739.40801.faltet@carabos.com> <436BC62F.1010104@stsci.edu> <1131193863.2849.8.camel@localhost.localdomain> Message-ID: <437379A2.5070203@stsci.edu> Francesc Altet wrote: >El dv 04 de 11 del 2005 a les 15:35 -0500, en/na Todd Miller va >escriure: > > >>This turned out to be a problem with the way numarray handles Numeric's >>multi-segment buffer protocol. I worked around this by implementing >>David Cooke's __array_struct__ array interface for numarray. >> >> > >Did you commit the changes in CVS? I'm getting the same problems with >the current CVS version: > > > >>>>Numeric.__version__ >>>> >>>> >'24.1' > > >>>>import numarray >>>>numarray.__version__ >>>> >>>> >'1.4.2' > > >>>>num=Numeric.array([1,2,3,4]) >>>>numarray.array(num) >>>> >>>> >array([1, 2, 3, 4]) > > >>>>num2=num[::2] >>>>numarray.array(num2) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >380, in array > a = a.astype(type) > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >867, in astype > return self.copy() > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >927, in copy > c = _gen.NDArray.copy(self) > File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, >in copy > arr._itemsize) >numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 >buffersize=8 > > This turned out to be a bug in numarray buffer size determination... striding wasn't accounted for so the buffer appeared to be too small. It's fixed now in CVS. There's still a backward compatibility problem for old Numerics which don't implement __array_struct__ and get multi-segment buffers through __array_data__. Regards, Todd From faltet at carabos.com Thu Nov 10 09:50:44 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Nov 10 09:50:44 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray In-Reply-To: <437379A2.5070203@stsci.edu> References: <200511041739.40801.faltet@carabos.com> <1131193863.2849.8.camel@localhost.localdomain> <437379A2.5070203@stsci.edu> Message-ID: <200511101850.26502.faltet@carabos.com> Ups, I've made a cvs update and not luck yet: >>> import numarray; numarray.__version__ '1.4.2' >>> import Numeric; Numeric.__version__ '24.1' >>> num=Numeric.array([1,2,3,4]) >>> numarray.array(num) array([1, 2, 3, 4]) >>> num2=num[::2] >>> numarray.array(num2) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 380, in array a = a.astype(type) File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 867, in astype return self.copy() File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 927, in copy c = _gen.NDArray.copy(self) File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in copy arr._itemsize) numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 buffersize=8 I've made a rm -rf of old numarray extension, and rebuild everything from scratch. Regards, A Dijous 10 Novembre 2005 17:47, v?reu escriure: > Francesc Altet wrote: > >El dv 04 de 11 del 2005 a les 15:35 -0500, en/na Todd Miller va > > > >escriure: > >>This turned out to be a problem with the way numarray handles Numeric's > >>multi-segment buffer protocol. I worked around this by implementing > >>David Cooke's __array_struct__ array interface for numarray. > > > >Did you commit the changes in CVS? I'm getting the same problems with > > > >the current CVS version: > >>>>Numeric.__version__ > > > >'24.1' > > > >>>>import numarray > >>>>numarray.__version__ > > > >'1.4.2' > > > >>>>num=Numeric.array([1,2,3,4]) > >>>>numarray.array(num) > > > >array([1, 2, 3, 4]) > > > >>>>num2=num[::2] > >>>>numarray.array(num2) > > > >Traceback (most recent call last): > > File "", line 1, in ? > > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line > >380, in array > > a = a.astype(type) > > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line > >867, in astype > > return self.copy() > > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line > >927, in copy > > c = _gen.NDArray.copy(self) > > File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, > >in copy > > arr._itemsize) > >numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 > >buffersize=8 > > This turned out to be a bug in numarray buffer size determination... > striding wasn't accounted for so the buffer appeared to be too small. > It's fixed now in CVS. There's still a backward compatibility problem > for old Numerics which don't implement __array_struct__ and get > multi-segment buffers through __array_data__. > > Regards, > Todd -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jmiller at stsci.edu Thu Nov 10 09:54:36 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 10 09:54:36 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray In-Reply-To: <200511101850.26502.faltet@carabos.com> References: <200511041739.40801.faltet@carabos.com> <1131193863.2849.8.camel@localhost.localdomain> <437379A2.5070203@stsci.edu> <200511101850.26502.faltet@carabos.com> Message-ID: <4373894D.5010407@stsci.edu> Francesc Altet wrote: >Ups, I've made a cvs update and not luck yet: > > > >>>>import numarray; numarray.__version__ >>>> >>>> >'1.4.2' > > >>>>import Numeric; Numeric.__version__ >>>> >>>> >'24.1' > > >>>>num=Numeric.array([1,2,3,4]) >>>>numarray.array(num) >>>> >>>> >array([1, 2, 3, 4]) > > >>>>num2=num[::2] >>>>numarray.array(num2) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 380, >in array > a = a.astype(type) > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 867, >in astype > return self.copy() > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 927, >in copy > c = _gen.NDArray.copy(self) > File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, in >copy > arr._itemsize) >numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 >buffersize=8 > >I've made a rm -rf of old numarray extension, and rebuild everything >from scratch. > >Regards, > >A Dijous 10 Novembre 2005 17:47, v?reu escriure: > > >>Francesc Altet wrote: >> >> >>>El dv 04 de 11 del 2005 a les 15:35 -0500, en/na Todd Miller va >>> >>>escriure: >>> >>> >>>>This turned out to be a problem with the way numarray handles Numeric's >>>>multi-segment buffer protocol. I worked around this by implementing >>>>David Cooke's __array_struct__ array interface for numarray. >>>> >>>> >>>Did you commit the changes in CVS? I'm getting the same problems with >>> >>>the current CVS version: >>> >>> >>>>>>Numeric.__version__ >>>>>> >>>>>> >>>'24.1' >>> >>> >>> >>>>>>import numarray >>>>>>numarray.__version__ >>>>>> >>>>>> >>>'1.4.2' >>> >>> >>> >>>>>>num=Numeric.array([1,2,3,4]) >>>>>>numarray.array(num) >>>>>> >>>>>> >>>array([1, 2, 3, 4]) >>> >>> >>> >>>>>>num2=num[::2] >>>>>>numarray.array(num2) >>>>>> >>>>>> >>>Traceback (most recent call last): >>> File "", line 1, in ? >>> File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >>>380, in array >>> a = a.astype(type) >>> File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >>>867, in astype >>> return self.copy() >>> File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >>>927, in copy >>> c = _gen.NDArray.copy(self) >>> File "/usr/lib/python2.4/site-packages/numarray/generic.py", line 724, >>>in copy >>> arr._itemsize) >>>numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 >>>buffersize=8 >>> >>> So you're still seeing it? Does the attached work for you? Keep in mind, anonymous CVS lags on Source Forge. >>This turned out to be a bug in numarray buffer size determination... >>striding wasn't accounted for so the buffer appeared to be too small. >>It's fixed now in CVS. There's still a backward compatibility problem >>for old Numerics which don't implement __array_struct__ and get >>multi-segment buffers through __array_data__. >> >>Regards, >>Todd >> >> > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: striding.py Type: application/x-python Size: 164 bytes Desc: not available URL: From faltet at carabos.com Thu Nov 10 10:03:25 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Nov 10 10:03:25 2005 Subject: [Numpy-discussion] Yet another problem in CVS numarray In-Reply-To: <4373894D.5010407@stsci.edu> References: <200511041739.40801.faltet@carabos.com> <200511101850.26502.faltet@carabos.com> <4373894D.5010407@stsci.edu> Message-ID: <200511101903.08516.faltet@carabos.com> A Dijous 10 Novembre 2005 18:54, Todd Miller va escriure: > So you're still seeing it? Does the attached work for you? Nope. > Keep in mind, anonymous CVS lags on Source Forge. Oh yes. I forgot about this :-/. I'll try it again tomorrow... -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From tom.denniston at alum.dartmouth.org Thu Nov 10 10:49:42 2005 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Thu Nov 10 10:49:42 2005 Subject: [Numpy-discussion] BUG! In-Reply-To: <200511101206.38946.bahn@atomistix.com> References: <200511101206.38946.bahn@atomistix.com> Message-ID: I reported this bug a whle ago and it was fixed to the best of my knowledge On 11/10/05, Sune Rastad Bahn wrote: > Dear devs, > > Thanks a lot for a very nice and extremely useful python module. > I recommend all my colleagues to try you package whenever they ask for a > matlab like environment. This week one of these friends came by me and showed > me a bug in RandomArray. > > try the following > > python -c "from RandomArray import *; > seed(1874764637, 1152239787);print randint(0,1000)" > > and notice how the result is the upper limit (1000) > even though the documentation clearly states that the number should be > strictly below 1000. > > My guess is that this is an issue with the underlying ranlib library, but > perhaps someone can clarify? > > best, > Sune > > -- > Software Application Manager > Sune Rastad Bahn > Phone: +45 35 320 638 > Mobile: +45 23 455 997 > Email: bahn at atomistix.com > > > ------------------------------------------------------- > SF.Net email is sponsored by: > Tame your development challenges with Apache's Geronimo App Server. Download > it for free - -and be entered to win a 42" plasma tv or your very own > Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From mithrandir42 at web.de Thu Nov 10 11:09:17 2005 From: mithrandir42 at web.de (N. Volbers) Date: Thu Nov 10 11:09:17 2005 Subject: [Numpy-discussion] Writing/reading a numeric array to a file In-Reply-To: References: <1124237199@web.de> Message-ID: <43739BDB.9080400@web.de> >>With the knowledge, that it _should_ work, I have just retried to build >>scipy_core 0.4.2 on my rather fresh installation of Slackware Linux >>10.2 (python 2.4.1) and still couldn't get it to build. I have >>attached the log files out.txt and err.txt to this e-mail, which I got >>by typing >> >>$ python setup.py build 1>out.txt 2>err.txt >> >>in the scipy_core main directory. The two files are compressed into a >>single tar.gz archive (5k), because I do not know if 60k attachments are >>o.k. on this list. >> >>Maybe you can help me solve my problem? > > > Did you try a recent check out? > With Travis' speed of coding scipy_core version 0.4.2.1252 > seems pretty old to me > > I tested > In [95]: scipy.__core_version__ > Out[95]: '0.4.3.1440' > this morning without problems. That was a wonderful idea! I checked out 0.4.3.1456 and it works just fine. Thank you very much. BTW, I checked out the source via $ svn co http://svn.scipy.org/svn/scipy_core/branches/newcore which took me a while to find (it was on the conference paper about scipy). Best regards, Niklas. From oliphant at ee.byu.edu Thu Nov 10 16:29:38 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Nov 10 16:29:38 2005 Subject: [Numpy-discussion] SciPy SVN altered so that newscipy and newcore are now the trunk. Message-ID: <4373E5D0.5050305@ee.byu.edu> I've finished altering the subversion repository of SciPy so that the new development is taking place on the trunk of both scipy and scipy_core. The old versions are under branches named oldcore and oldscipy. Get the new repositor(y,ies) using: *Core*: svn co http://svn.scipy.org/svn/scipy_core/trunk core *Full SciPy*: svn co http://svn.scipy.org/svn/scipy/trunk scipy Doing both will place two directories named core and scipy in your current directory containing the current state of both repositories. python setup.py install should work in each directory. The Freeze is now over. I want to track down the bug that Christopher Hanley noted and another f2py-related bug before making a release, which I expect to happen by the weekend. -Travis From cookedm at physics.mcmaster.ca Thu Nov 10 19:22:03 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Nov 10 19:22:03 2005 Subject: [Numpy-discussion] SciPy SVN altered so that newscipy and newcore are now the trunk. In-Reply-To: <4373E5D0.5050305@ee.byu.edu> (Travis Oliphant's message of "Thu, 10 Nov 2005 17:29:04 -0700") References: <4373E5D0.5050305@ee.byu.edu> Message-ID: Travis Oliphant writes: > I've finished altering the subversion repository of SciPy so that the > new development is taking place on the trunk of both scipy and scipy_core. > > The old versions are under branches named oldcore and oldscipy. > > Get the new repositor(y,ies) using: > > *Core*: > svn co http://svn.scipy.org/svn/scipy_core/trunk core > > *Full SciPy*: > svn co http://svn.scipy.org/svn/scipy/trunk scipy > > Doing both will place two directories named core and scipy in your > current directory containing the current state of both repositories. Alternatively, instead of doing a full checkout, you can switch your working copies: Within your (old) newcore directory: svn sw http://svn.scipy.org/svn/scipy_core/trunk and same for full scipy. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From faltet at carabos.com Fri Nov 11 00:50:23 2005 From: faltet at carabos.com (Francesc Altet) Date: Fri Nov 11 00:50:23 2005 Subject: [Numpy-discussion] A small glitch remains in CVS numarray Message-ID: <1131700085.2968.2.camel@localhost.localdomain> Hi, I've checked CVS numarray today and yes, the problem with strided arrays seems to be gone. It does remain some small (I think) issue: >>> import Numeric;Numeric.__version__ '24.1' >>> import numarray;numarray.__version__ '1.4.2' >>> num=Numeric.array([0.,1.]) >>> na=numarray.zeros(shape=2) >>> na[...] = num Traceback (most recent call last): File "", line 1, in ? TypeError: NA_DescrFromType: unknown type: -1 and also: >>> numarray.array(num) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line 359, in array a = _numarray._array_from_array_struct(sequence.__array_struct__) TypeError: NA_DescrFromType: unknown type: -1 Regards, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From jmiller at stsci.edu Sat Nov 12 04:24:39 2005 From: jmiller at stsci.edu (Todd Miller) Date: Sat Nov 12 04:24:39 2005 Subject: [Numpy-discussion] A small glitch remains in CVS numarray In-Reply-To: <1131700085.2968.2.camel@localhost.localdomain> References: <1131700085.2968.2.camel@localhost.localdomain> Message-ID: <4375DEF7.4070800@stsci.edu> Francesc Altet wrote: >Hi, > >I've checked CVS numarray today and yes, the problem with strided arrays >seems to be gone. It does remain some small (I think) issue: > > > >>>>import Numeric;Numeric.__version__ >>>> >>>> >'24.1' > > >>>>import numarray;numarray.__version__ >>>> >>>> >'1.4.2' > > >>>>num=Numeric.array([0.,1.]) >>>>na=numarray.zeros(shape=2) >>>>na[...] = num >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >TypeError: NA_DescrFromType: unknown type: -1 > >and also: > > > >>>>numarray.array(num) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.4/site-packages/numarray/numarraycore.py", line >359, in array > a = _numarray._array_from_array_struct(sequence.__array_struct__) >TypeError: NA_DescrFromType: unknown type: -1 > >Regards, > > > This was a cut-and-pasto in numarray's scipy type definition table. It's fixed in CVS. Thanks, Todd From mithrandir42 at web.de Sat Nov 12 15:24:48 2005 From: mithrandir42 at web.de (N. Volbers) Date: Sat Nov 12 15:24:48 2005 Subject: [Numpy-discussion] Numeric 24.2 build fails Message-ID: <43767ABF.1060204@web.de> Sorry to once again report a build error ;-) Numeric 24.1 builds just fine, while Numeric 24.2 produces the attaced messages (out.txt) and error messages (err.txt). HTH, Niklas. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: err.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: out.txt URL: From cjw at sympatico.ca Sun Nov 13 16:54:36 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Nov 13 16:54:36 2005 Subject: [Numpy-discussion] Numeric3.0 - Currently available under scipy/base as version 0.4.1 (Windows) In-Reply-To: <43725D8A.9090307@ee.byu.edu> References: <437209BF.8090607@sympatico.ca> <43725D8A.9090307@ee.byu.edu> Message-ID: <4377E058.7010404@sympatico.ca> An HTML attachment was scrubbed... URL: From rays at blue-cove.com Mon Nov 14 08:15:59 2005 From: rays at blue-cove.com (RayS) Date: Mon Nov 14 08:15:59 2005 Subject: [Numpy-discussion] FIFO/circular buffer class in Numeric? Message-ID: <6.2.3.4.2.20051114065042.02fc6a50@pop-server.san.rr.com> Has anyone implemented a fast, generic circular array buffer in Numpy? I didn't see one in the usual searches, but I don't want to re-invent one either. I did see a number of C implementations like http://www.codeproject.com/csharp/circularbuffer.asp (http://www.gois.ws/files/attach/200209/200209201850040.zip) which could be weave'd, I suppose. I have some previous implementations I've used, but what I had in mind was making a new FIFO class with attributes of address, state, pointer, size etc., and methods like flush (if I implement it as a memmap) and other things, like queue has. The important part of the idea is that multiple threads and processes could fully access it. Currently, I will have an ADC process fill it, and other processes use it. I'll post code as I go, in any case. Ray From jeff at taupro.com Tue Nov 15 22:44:14 2005 From: jeff at taupro.com (Jeff Rush) Date: Tue Nov 15 22:44:14 2005 Subject: [Numpy-discussion] Call for a Scientific/Engineering Tutorial or Two at PyCon 2006 Message-ID: <437AD50D.1070107@taupro.com> Greetings. I'm working with the Python Conference planning group, for a conference to be held in Dallas in 2006. As you probably know from postings to python-announce, this year, for the first time, we're having a full day (Feb 23, 2006) of trainer-compensated tutorials, structured as either half-day or full-day sessions. From the feedback we received at last year's PyCon, there was a strong interest in scientific/engineering uses of Python, which has a rich set of tools and frameworks. We hope this year to dedicate one room on Tutorial Day to such topics. While I was looking over the submissions for the tutorials tonight, I noticed there were none related to science/engineering. It would be cool to have a half-day class on one of the underlying packages used across disciplines. Note that these are to be professionally-run half or full day classes, suggested with handouts or notebooks. And the trainer will receive monetary compensation, depending upon the number of attendees that register for specific tutorials. If you or someone on your staff is interested, there is more information at: http://us.pycon.org/TX2006/CallForTutorials While the submission deadline expires tonight, we have some flexibility. Jeff Rush From jdhunter at ace.bsd.uchicago.edu Wed Nov 16 07:38:03 2005 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Wed Nov 16 07:38:03 2005 Subject: [Numpy-discussion] compiling 24.2 on solaris 8 Message-ID: <87ek5gd2xq.fsf@peds-pc311.bsd.uchicago.edu> To compile numpy 24.2 on solaris 8, I needed to make the following change to Packages/RNG/Src/ranf.c #if !defined(__sgi) //int gettimeofday(struct timeval *, struct timezone *); //old int gettimeofday(); //new #endif Not sure why __sgi is defined... JDH From cjw at sympatico.ca Wed Nov 16 15:09:03 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Nov 16 15:09:03 2005 Subject: [Numpy-discussion] The file DEVELOPERS.TXT Message-ID: <437BBBF7.4050202@sympatico.ca> scipy._import_tools.py has a comment referring to DEVELOPERS.TXT but I don't find it in the Windows download. From strawman at astraw.com Thu Nov 17 03:42:06 2005 From: strawman at astraw.com (Andrew Straw) Date: Thu Nov 17 03:42:06 2005 Subject: [Numpy-discussion] numarray CVS seems to have an __array_struct__ bug Message-ID: <437C6C81.1030503@astraw.com> Hi, I think there's a bug in the new __array_struct__ code in numarray CVS. Consider an object a which implements the __array_struct__ interface. If one constructs a numarray "view" of this array using aview = numarray.asarray(a), the numarray.ndarray instance does not keep a reference to a.__array_struct__. Thus, a segfault may occur because a.data can be freed although aview may attempt to access it. In particular code like this will crash: buf = numarray.asarray( array_struct_factory() ) # buffer is freed buf[1] = 2 # attempt is made to write to the non-existant buffer, causing a segfault I've "fixed" this issue with the following, which I'm sure is not the right way, but it's all I can do at this point. diff -u -r1.130 numarraycore.py --- Lib/numarraycore.py 4 Nov 2005 20:27:11 -0000 1.130 +++ Lib/numarraycore.py 17 Nov 2005 11:38:29 -0000 @@ -357,6 +357,7 @@ a = None if hasattr(sequence, "__array_struct__"): a = _numarray._array_from_array_struct(sequence.__array_struct__) + a._hack_to_hold_reference_ = sequence.__array_struct__ elif (hasattr(sequence, "__array_shape__") and hasattr(sequence, "__array_typestr__")): typestr = sequence.__array_typestr__ And while I'm at it, can the "from distutils.command.config import log" be turned off in setup.py? I like to see those errors, warnings, and status strings whiz past... From faltet at carabos.com Thu Nov 17 04:00:01 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Nov 17 04:00:01 2005 Subject: [Numpy-discussion] numarray CVS seems to have an __array_struct__ bug In-Reply-To: <437C6C81.1030503@astraw.com> References: <437C6C81.1030503@astraw.com> Message-ID: <200511171259.11108.faltet@carabos.com> A Dijous 17 Novembre 2005 12:41, Andrew Straw va escriure: > And while I'm at it, can the "from distutils.command.config import log" > be turned off in setup.py? I like to see those errors, warnings, and > status strings whiz past... +1 ;-) -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jmiller at stsci.edu Thu Nov 17 04:15:03 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 17 04:15:03 2005 Subject: [Numpy-discussion] numarray CVS seems to have an __array_struct__ bug In-Reply-To: <437C6C81.1030503@astraw.com> References: <437C6C81.1030503@astraw.com> Message-ID: <437C742C.8020305@stsci.edu> Nice catch Andrew, and very timely! Thanks, Todd Andrew Straw wrote: >Hi, > >I think there's a bug in the new __array_struct__ code in numarray CVS. > >Consider an object a which implements the __array_struct__ interface. If >one constructs a numarray "view" of this array using aview = >numarray.asarray(a), the numarray.ndarray instance does not keep a >reference to a.__array_struct__. Thus, a segfault may occur because >a.data can be freed although aview may attempt to access it. > >In particular code like this will crash: > >buf = numarray.asarray( array_struct_factory() ) # buffer is freed >buf[1] = 2 # attempt is made to write to the non-existant buffer, >causing a segfault > >I've "fixed" this issue with the following, which I'm sure is not the >right way, but it's all I can do at this point. > >diff -u -r1.130 numarraycore.py >--- Lib/numarraycore.py 4 Nov 2005 20:27:11 -0000 1.130 >+++ Lib/numarraycore.py 17 Nov 2005 11:38:29 -0000 >@@ -357,6 +357,7 @@ > a = None > if hasattr(sequence, "__array_struct__"): > a = >_numarray._array_from_array_struct(sequence.__array_struct__) >+ a._hack_to_hold_reference_ = sequence.__array_struct__ > elif (hasattr(sequence, "__array_shape__") and > hasattr(sequence, "__array_typestr__")): > typestr = sequence.__array_typestr__ > > >And while I'm at it, can the "from distutils.command.config import log" >be turned off in setup.py? I like to see those errors, warnings, and >status strings whiz past... > > >------------------------------------------------------- >This SF.Net email is sponsored by the JBoss Inc. Get Certified Today >Register for a JBoss Training Course. Free Certification Exam >for All Training Attendees Through End of 2005. For more info visit: >http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From curzio.basso at gmail.com Sat Nov 19 05:34:05 2005 From: curzio.basso at gmail.com (Curzio Basso) Date: Sat Nov 19 05:34:05 2005 Subject: [Numpy-discussion] compilation error on OSX 10.4 Message-ID: Hi all, I get the following error compiling an extension module using numarray 1.4.1 on Mac OSX 10.4: ----- building 'fmlio' extension gcc-3.3 -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DLINUX -I/home/basso/usr/include/python -I/sw/include/python2.4 -c src/fmlio.cpp -o build/temp.darwin-8.3.0-Power_Macintosh-2.4/src/fmlio.o In file included from /sw/include/python2.4/numarray/libnumarray.h:17, from /sw/include/python2.4/numarray/numarray.h:6, from src/fmlio.h:10, from src/fmlio.cpp:2: /sw/include/python2.4/numarray/nummacro.h:27: error: parse error before `;' token error: command 'gcc-3.3' failed with exit status 1 ----- the same happens with gcc-4. On Gentoo I can compile without problems. Did anyone else get a similar problem? Ah, and numarray.h is included right after Python.h. cheers curzio From curzio.basso at gmail.com Sat Nov 19 06:00:13 2005 From: curzio.basso at gmail.com (Curzio Basso) Date: Sat Nov 19 06:00:13 2005 Subject: [Numpy-discussion] Re: compilation error on OSX 10.4 In-Reply-To: References: Message-ID: > the same happens with gcc-4. On Gentoo I can compile without problems. I should correct myself: on Gentoo it compiles but I have version 1.3.1. curzio From curzio.basso at gmail.com Sat Nov 19 09:49:01 2005 From: curzio.basso at gmail.com (Curzio Basso) Date: Sat Nov 19 09:49:01 2005 Subject: [Numpy-discussion] Re: compilation error on OSX 10.4 In-Reply-To: References: Message-ID: > I should correct myself: on Gentoo it compiles but I have version 1.3.1. First of all, compilation fails also on Gentoo with version 1.4.1. Second, I can reproduce the result trying to compile the following code: -------------- #include #include "numarray/libnumarray.h" int main() { return 0; } ------------- Compiling the above code as C++, I get the error already reported, while compiling as C works fine. The reason is that at line 27 in nummacro.h the C++ keyword "operator" is used. Would it be possible to replace it with something else? like "operator_"? By the way, using extern "C" {...} does not help. cheers curzio From smaret at umich.edu Sat Nov 19 10:53:01 2005 From: smaret at umich.edu (Sebastien Maret) Date: Sat Nov 19 10:53:01 2005 Subject: [Numpy-discussion] Re: compilation error on OSX 10.4 References: Message-ID: Curzio Basso writes: > I get the following error compiling an extension module using numarray > 1.4.1 on Mac OSX 10.4: > ----- > building 'fmlio' extension > gcc-3.3 -fno-strict-aliasing -Wno-long-double -no-cpp-precomp > -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DLINUX > -I/home/basso/usr/include/python -I/sw/include/python2.4 -c > src/fmlio.cpp -o > build/temp.darwin-8.3.0-Power_Macintosh-2.4/src/fmlio.o > In file included from /sw/include/python2.4/numarray/libnumarray.h:17, > from /sw/include/python2.4/numarray/numarray.h:6, > from src/fmlio.h:10, > from src/fmlio.cpp:2: > /sw/include/python2.4/numarray/nummacro.h:27: error: parse error before `;' > token There is a bug report for this problem: http://sourceforge.net/tracker/?func=detail&atid=450446&aid=1350954&group_id=1369 The problem has been fixed in FreeBSD: http://www.freebsd.org/cgi/cvsweb.cgi/ports/math/py-numarray/files/patch-Include::numarray::nummacro.h Maybe this patch could be applied to numarray ? S?bastien From jdawe at u.washington.edu Sat Nov 19 11:31:02 2005 From: jdawe at u.washington.edu (Jordan Dawe) Date: Sat Nov 19 11:31:02 2005 Subject: [Numpy-discussion] scipy cygwin compilation error Message-ID: <437F7D48.9070406@u.washington.edu> I just tried to install scipy under cygwin using both 0.61 and the current svn. I get this: gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cygwin-1.5.18-i686-2.4/scipy/base/umath.dll build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UBYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2215: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `USHORT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2233: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UINT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2251: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `ULONG_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2269: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `BYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2305: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o:/usr/local/core/build/src/scipy/base/src/umathmodule.c:2325: more undefined references to `_feraiseexcept' follow build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_checkfperr': /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_clearfperr': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_GenericFunction': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `construct_reduce': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' collect2: ld returned 1 exit status build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UBYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2215: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `USHORT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2233: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `UINT_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2251: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `ULONG_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2269: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `BYTE_multiply': /usr/local/core/build/src/scipy/base/src/umathmodule.c:2305: undefined reference to `_feraiseexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o:/usr/local/core/build/src/scipy/base/src/umathmodule.c:2325: more undefined references to `_feraiseexcept' follow build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_checkfperr': /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:475: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_clearfperr': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `PyUFunc_GenericFunction': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o: In function `construct_reduce': /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_fetestexcept' /usr/local/core/scipy/base/src/ufuncobject.c:506: undefined reference to `_feclearexcept' collect2: ld returned 1 exit status error: Command "gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.18-i686-2.4/build/src/scipy/base/src/umathmodule.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cygwin-1.5.18-i686-2.4/scipy/base/umath.dll" failed with exit status 1 Any suggestions? Jordan From oliphant at ee.byu.edu Sun Nov 20 01:04:01 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sun Nov 20 01:04:01 2005 Subject: [Numpy-discussion] Memory mapped files in scipy core Message-ID: <43803BC9.6050103@ee.byu.edu> I would appreciate understanding typically use cases for memory-mapped files. I am not sure I understand why certain choices were made for numarray's memmap and memmap slice classes. They seem like a lot of "extra" stuff and I'm not sure what all that stuff is for. Rather than just copy these over, I would like to understand what people typically want to do with memory-mapped files to see if scipy core doesn't already provide that. For example, write now I can open a file, use mmap to obtain a memory map object and then pass that object into frombuffer in scipy_core to get an ndarray whose memory maps a file on disk. Now, this ndarray can be sliced and indexed and manipulated all the while referring to the file on disk (well technically, I suppose, the memory-mapped object would need to be flushed to synchronize). Now, I could see wanting to make the process of opening the file, getting the mmap object and setting it's buffer to the array object a little easier. Thus, a simple memmap class would be a useful construct -- I could even see it inheriting from the ndarray directly and adding a few methods. I guess I just don't see why one would care about a memory-mapped slice object, when the mmaparray sub-class would be perfectly useful. On a related, but orthogonal note: My understanding is that using memory-mapped files for *very* large files will require modification to the mmap module in Python --- something I think we should push. One part of that process would be to add the C-struct array interface to the mmap module and the buffer object -- perhaps this is how we get the array interface into Python quickly. Then, if we could make a base-type mmap that did not use the buffer interface or the sequence interface (similar to the bigndarray in scipy_core) and therefore by-passed the problems with Python in those areas, then the current mmap object could inherit from the base class and provide current functionality while still exposing the array interface for access to >2GB files on 64-bit systems. Who would like to take up the ball for modifying mmap in Python in this fashion? -Travis From egsl1930 at yahoo.es Sun Nov 20 19:43:00 2005 From: egsl1930 at yahoo.es (Eduardo Stark) Date: Sun Nov 20 19:43:00 2005 Subject: [Numpy-discussion] Real life examples Message-ID: <20051121034233.90046.qmail@web26901.mail.ukl.yahoo.com> Dear naumarray and numpy community: Could you tell me where can I get real life examples to apply numarray. I am a Python and numarray begineer. Thanks Eduardo Stark --------------------------------- Correo Yahoo! Comprueba qu? es nuevo, aqu? http://correo.yahoo.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egsl1930 at yahoo.es Sun Nov 20 19:43:01 2005 From: egsl1930 at yahoo.es (Eduardo Stark) Date: Sun Nov 20 19:43:01 2005 Subject: [Numpy-discussion] Real life examples Message-ID: <20051121034234.55990.qmail@web26909.mail.ukl.yahoo.com> Dear naumarray and numpy community: Could you tell me where can I get real life examples to apply numarray. I am a Python and numarray begineer. Thanks Eduardo Stark --------------------------------- Correo Yahoo! Comprueba qu? es nuevo, aqu? http://correo.yahoo.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at enthought.com Mon Nov 21 06:46:01 2005 From: travis at enthought.com (Travis N. Vaught) Date: Mon Nov 21 06:46:01 2005 Subject: [Numpy-discussion] OT: Enthought IT Admin Job Posting Message-ID: <4381DD6D.9030202@enthought.com> All, I'm posting this here in the hopes that someone that is passionate about python/numeric/scipy will "answer the call": `Enthought, Inc. `__ (Austin, TX, USA) ==================================================================== **Job Description**: Position: IT Administrator --------------------------- Enthought is looking for an exceptional IT Administrator to manage the IT infrastructure for it's Austin, TX offices. This person will have a passion for supporting software development tools and environments. The target server platforms include Linux (RedHat and Fedora), Windows XP, and Solaris. Workstations are a mix of Windows and Linux. We're looking for an Admin that focuses on supporting software developers. Desired Skills and Capabilities: * B.S. in Computer Science of other related field (preferably not MIS) * 5+ Years Experience in Enterprise IT Administration Role * Ability to program or script in a programming language or shell - (Python, Perl and small C programs) * Ability to solve problems quickly and completely * Strong inter-personal and communication skills as well as a team player in a group of highly talented software developers * Willingness to pitch in wherever needed * Interested in making developer's lives easier Duties: * Perform installation, patching and other server maintenance to ensure security and stability of server infrastructure. * Maintain core infrastructure technologies such as Firewall, VPN, apache web server, mail server, NIS, NFS, DNS, DHCP, SSH, network-wide backups, RAID and Samba * Identify routine tasks and automate through shell/Python scripting * Perform on-call duties and out of hours maintenance as necessary * Configure Nortel phone system * Build RPMs for RH/FC Linux * Using development tools on Linux such as gcc/g++, make, autoconf, etc. * Working with Python and building & installing Python packages using distutils * Support developer tools such as SVN repositories, bug trackers, Software Project Management utilities Company: -------- Enthought is a scientific computing company located in downtown Austin, Texas. Founded in 2001, it has grown nearly 100% per year both in staff and revenue to become a stable and profitable technology company. This growth has been possible because of Enthought?s talented team and because of our commitment to developing quality software. We strive to combine advanced algorithm development with modern software practices, such as component based architectures, application scripting, and intuitive user interface design. We take a holistic approach to software development, in which architects and developers team with technical writers, human factors specialists, and project managers (always highly technical individuals) to develop a complete solution for our customers. Much of our work is based on the Python programming language, and we are actively engaged in open source development (www.scipy.org ). We?re lucky enough to work on interesting problems and are looking for talented people to join us. Some of our current efforts are in the areas of geophysics, electromagnetics, fluid dynamics, micro-rheology, CAD, 2-D and 3-D visualization, and others. All of these tools are developed as plug-ins into our Envisage "Scientific IDE" framework. * **Contact**: Travis N. Vaught, CEO * **E-mail contact**: jobs at enthought.com * **Web**: http://www.enthought.com/careers.htm -- ........................ Travis N. Vaught CEO Enthought, Inc. http://www.enthought.com ........................ From pearu at cens.ioc.ee Mon Nov 21 07:06:00 2005 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Mon Nov 21 07:06:00 2005 Subject: [Numpy-discussion] The file DEVELOPERS.TXT In-Reply-To: <437BBBF7.4050202@sympatico.ca> Message-ID: On Wed, 16 Nov 2005, Colin J. Williams wrote: > scipy._import_tools.py has a comment referring to DEVELOPERS.TXT but I > don't find it in the Windows download. This is fixed in scipy core SVN repositoy. Up-to-date DEVELOPERS.TXT (that used to reside in scipy source directory) can be now found as scipy/doc/DISTUTILS.txt file. Pearu From jmiller at stsci.edu Mon Nov 21 07:15:00 2005 From: jmiller at stsci.edu (Todd Miller) Date: Mon Nov 21 07:15:00 2005 Subject: [Numpy-discussion] Memory mapped files in scipy core In-Reply-To: <43803BC9.6050103@ee.byu.edu> References: <43803BC9.6050103@ee.byu.edu> Message-ID: <4381E3EA.9040500@stsci.edu> Travis Oliphant wrote: > > I would appreciate understanding typically use cases for memory-mapped > files. I am not sure I understand why certain choices were made for > numarray's memmap and memmap slice classes. They seem like a lot of > "extra" stuff and I'm not sure what all that stuff is for. > > Rather than just copy these over, I would like to understand what > people typically want to do with memory-mapped files to see if scipy > core doesn't already provide that. > > For example, write now I can open a file, use mmap to obtain a memory > map object and then pass that object into frombuffer in scipy_core to > get an ndarray whose memory maps a file on disk. > Now, this ndarray can be sliced and indexed and manipulated all the > while referring to the file on disk (well technically, I suppose, the > memory-mapped object would need to be flushed to synchronize). > Now, I could see wanting to make the process of opening the file, > getting the mmap object and setting it's buffer to the array object a > little easier. Thus, a simple memmap class would be a useful > construct -- I could even see it inheriting from the ndarray directly > and adding a few methods. I guess I just don't see why one would > care about a memory-mapped slice object, when the mmaparray sub-class > would be perfectly useful. > There are a few extra capabilities which are supported in numarray's memmap: 1. slice insertion 2. slice deletion 3. memmap based array resizing Each of these things implicitly changes the layout of the underlying file. Whether or not these features get used or justify the complexity is another matter. Because of 32-bit address space exhaustion and swap file issues, memory mapping was a disappointment at STSCI so I'm doubtful we used these features ourselves. Regards, Todd > On a related, but orthogonal note: > > My understanding is that using memory-mapped files for *very* large > files will require modification to the mmap module in Python --- > something I think we should push. One part of that process would be > to add the C-struct array interface to the mmap module and the buffer > object -- perhaps this is how we get the array interface into Python > quickly. Then, if we could make a base-type mmap that did not use > the buffer interface or the sequence interface (similar to the > bigndarray in scipy_core) and therefore by-passed the problems with > Python in those areas, then the current mmap object could inherit from > the base class and provide current functionality while still exposing > the array interface for access to >2GB files on 64-bit systems. > > Who would like to take up the ball for modifying mmap in Python in > this fashion? > > -Travis > > > > ------------------------------------------------------- > This SF.Net email is sponsored by the JBoss Inc. Get Certified Today > Register for a JBoss Training Course. Free Certification Exam > for All Training Attendees Through End of 2005. For more info visit: > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From a.h.jaffe at gmail.com Mon Nov 21 09:19:07 2005 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Mon Nov 21 09:19:07 2005 Subject: [Numpy-discussion] Problems with linalg.cholesky? Message-ID: [Originally posted to g.c.p.user...] Hi all, In the newest incarnation of scipy_core, I am having trouble with the cholesky(a) routine: /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy/linalg/basic_lite.py in cholesky_decomposition(a) 115 else: 116 lapack_routine = lapack_lite.dpotrf --> 117 results = lapack_routine('L', n, a, m, 0) 118 if results['info'] > 0: 119 raise LinAlgError, 'Matrix is not positive definite - Cholesky decomposition cannot be computed' LapackError: Parameter a is not contiguous in lapack_lite.dpotrf But this isn't true; I get this error even when I pass trivial and contiguous matrices such as the output of identity(). Other linalg routines (included complicated ones like singular_value_decomp) seem to work fine. Any ideas? Andrew From oliphant.travis at ieee.org Mon Nov 21 13:13:02 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Nov 21 13:13:02 2005 Subject: [Numpy-discussion] Re: Memory mapped files in scipy core In-Reply-To: <4381E3EA.9040500@stsci.edu> References: <43803BC9.6050103@ee.byu.edu> <4381E3EA.9040500@stsci.edu> Message-ID: > There are a few extra capabilities which are supported in numarray's > memmap: > > 1. slice insertion > 2. slice deletion > 3. memmap based array resizing Thanks for the extra explanation. I could see how these might require more stuff under the hood. -Travis From joostvanevert at gmail.com Tue Nov 22 05:58:10 2005 From: joostvanevert at gmail.com (Joost van Evert) Date: Tue Nov 22 05:58:10 2005 Subject: [Numpy-discussion] import problem In-Reply-To: References: Message-ID: <1132667852.32469.9.camel@inpc93.et.tudelft.nl> Dear list, does anyone understand importing modules? I want to import a new version of Numeric(24.2) that was installed in my home directory, but the old one in /usr/lib/python/site-packages gets imported. The command "python setup.py --install=~" was used to install. And ~/lib/python is the first item in my PYTHONPATH variable. So the contents of sys.path is ['','~/lib/python', ...] Does anyone know any means to get the new version imported? Except from deleting the old version of course;) Regards, Joost From aisaac at american.edu Tue Nov 22 06:05:07 2005 From: aisaac at american.edu (Alan G Isaac) Date: Tue Nov 22 06:05:07 2005 Subject: [Numpy-discussion] import problem In-Reply-To: <1132667852.32469.9.camel@inpc93.et.tudelft.nl> References: <1132667852.32469.9.camel@inpc93.et.tudelft.nl> Message-ID: On Tue, 22 Nov 2005, Joost van Evert apparently wrote: > does anyone understand importing modules? I want to import a new version > of Numeric(24.2) that was installed in my home directory, but the old > one in /usr/lib/python/site-packages gets imported. > The command "python setup.py --install=~" was used to install. And > ~/lib/python is the first item in my PYTHONPATH variable. So the > contents of sys.path is ['','~/lib/python', ...] > Does anyone know any means to get the new version imported? Except from > deleting the old version of course;) http://docs.python.org/tut/node8.html section 6.1.1 hth, Alan Isaac From faltet at carabos.com Wed Nov 23 03:40:06 2005 From: faltet at carabos.com (Francesc Altet) Date: Wed Nov 23 03:40:06 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 Message-ID: <200511231239.21489.faltet@carabos.com> PyTables is a library to deal with very large datasets. It leverages the excellent HDF5 and numarray libraries to allow doing that in a very efficient way using the Python language. More info in: http://pytables.sourceforge.net/ ========================= Announcing PyTables 1.2 ========================= The PyTables development team is happy to announce the availability of a new major version of PyTables package. This version sports a completely new in-memory tree implementation based around a *node cache system*. This system loads nodes only when needed and unloads them when they are rarely used. The new feature allows the opening and creation of HDF5 files with large hierarchies very quickly and with a low memory consumption (the object tree is no longer completely loaded in-memory), while retaining all the powerful browsing capabilities of the previous implementation of the object tree. You can read more about the dings and bells of the new cache system in: http://www.carabos.com/downloads/pytables/NewObjectTreeCache.pdf Also, Jeff Whitaker has kindly contributed a new module called tables.NetCDF. It is designed to be used as a drop-in replacement for Scientific.IO.NetCDF, with only minor actions to existing code. Also, if you have the Scientific.IO.NetCDF module installed, it allows to do conversions between HDF5 <--> NetCDF3 formats. Go to the PyTables web site for downloading the beast: http://pytables.sourceforge.net/ If you want more info about this release, please check out the more comprehensive announcement message available in: http://www.carabos.com/downloads/pytables/ANNOUNCE-1.2.html Acknowledgments =============== Thanks to the users who provided feature improvements, patches, bug reports, support and suggestions. See THANKS file in distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last but not least, a big thank you to THG (http://www.hdfgroup.org/) for sponsoring many of the new features recently introduced in PyTables. --- **Enjoy data!** -- The PyTables Team -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From Chris.Barker at noaa.gov Wed Nov 23 08:49:02 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Nov 23 08:49:02 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 In-Reply-To: <200511231239.21489.faltet@carabos.com> References: <200511231239.21489.faltet@carabos.com> Message-ID: <43849D3F.7050703@noaa.gov> Francesc Altet wrote: > Also, Jeff Whitaker has kindly contributed a new module called > tables.NetCDF. It is designed to be used as a drop-in replacement for > Scientific.IO.NetCDF, with only minor actions to existing code. What advantages does tables.NetCDF have over Scientific.IO.NetCDF? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jswhit at fastmail.fm Wed Nov 23 08:59:14 2005 From: jswhit at fastmail.fm (Jeff Whitaker) Date: Wed Nov 23 08:59:14 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 In-Reply-To: <43849D3F.7050703@noaa.gov> References: <200511231239.21489.faltet@carabos.com> <43849D3F.7050703@noaa.gov> Message-ID: <43849F38.3000208@fastmail.fm> Chris Barker wrote: > Francesc Altet wrote: > >> Also, Jeff Whitaker has kindly contributed a new module called >> tables.NetCDF. It is designed to be used as a drop-in replacement for >> Scientific.IO.NetCDF, with only minor actions to existing code. > > What advantages does tables.NetCDF have over Scientific.IO.NetCDF? > > -Chris > > > Chris: From my perspective, the transparent compression (using zlib and the hdf5 shuffle filter) is the biggest one. I get files that are a factor of 2-4 smaller. -Jeff -- Jeffrey S. Whitaker Phone : (303)497-6313 Meteorologist FAX : (303)497-6449 NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker at noaa.gov 325 Broadway Office : Skaggs Research Cntr 1D-124 Boulder, CO, USA 80303-3328 Web : http://tinyurl.com/5telg From faltet at carabos.com Wed Nov 23 09:01:06 2005 From: faltet at carabos.com (Francesc Altet) Date: Wed Nov 23 09:01:06 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 In-Reply-To: <43849D3F.7050703@noaa.gov> References: <200511231239.21489.faltet@carabos.com> <43849D3F.7050703@noaa.gov> Message-ID: <200511231759.50278.faltet@carabos.com> A Dimecres 23 Novembre 2005 17:47, Chris Barker va escriure: > Francesc Altet wrote: > > Also, Jeff Whitaker has kindly contributed a new module called > > tables.NetCDF. It is designed to be used as a drop-in replacement for > > Scientific.IO.NetCDF, with only minor actions to existing code. > > What advantages does tables.NetCDF have over Scientific.IO.NetCDF? Maybe Jeff would add something more, but the apparent ones are: - No 2 GB limitation - Support for compression (and other filters) Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jswhit at fastmail.fm Wed Nov 23 09:37:18 2005 From: jswhit at fastmail.fm (Jeff Whitaker) Date: Wed Nov 23 09:37:18 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 In-Reply-To: <200511231759.50278.faltet@carabos.com> References: <200511231239.21489.faltet@carabos.com> <43849D3F.7050703@noaa.gov> <200511231759.50278.faltet@carabos.com> Message-ID: <4384A7F4.3070309@fastmail.fm> Francesc Altet wrote: > A Dimecres 23 Novembre 2005 17:47, Chris Barker va escriure: > >> Francesc Altet wrote: >> >>> Also, Jeff Whitaker has kindly contributed a new module called >>> tables.NetCDF. It is designed to be used as a drop-in replacement for >>> Scientific.IO.NetCDF, with only minor actions to existing code. >>> >> What advantages does tables.NetCDF have over Scientific.IO.NetCDF? >> > > Maybe Jeff would add something more, but the apparent ones are: > > - No 2 GB limitation > - Support for compression (and other filters) > > Cheers, > > And any dimension can be 'unlimited' (not just the first). -Jeff -- Jeffrey S. Whitaker Phone : (303)497-6313 Meteorologist FAX : (303)497-6449 NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker at noaa.gov 325 Broadway Office : Skaggs Research Cntr 1D-124 Boulder, CO, USA 80303-3328 Web : http://tinyurl.com/5telg From faltet at carabos.com Wed Nov 23 10:12:04 2005 From: faltet at carabos.com (Francesc Altet) Date: Wed Nov 23 10:12:04 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2 In-Reply-To: <4384A7F4.3070309@fastmail.fm> References: <200511231239.21489.faltet@carabos.com> <200511231759.50278.faltet@carabos.com> <4384A7F4.3070309@fastmail.fm> Message-ID: <200511231911.16720.faltet@carabos.com> A Dimecres 23 Novembre 2005 18:33, Jeff Whitaker va escriure: > > - No 2 GB limitation > > - Support for compression (and other filters) > And any dimension can be 'unlimited' (not just the first). Yes, but just one dimension at a time, at least for the time being. We would like to eliminate this limitation rather sooner than later, though. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From arajendr at ucsd.edu Wed Nov 23 15:48:01 2005 From: arajendr at ucsd.edu (Anoop) Date: Wed Nov 23 15:48:01 2005 Subject: [Numpy-discussion] Problem compiling scipy-core Message-ID: <1132789622.7570.4.camel@localhost.localdomain> Hi. I have problems installing scipy-core from source code on a Rocks Cluster distribution of Linux running on x86-64 hardware. The error messages tell me: /usr/bin/g77 -shared build/temp.linux-x86_64-2.3/scipy/corelib/blasdot/_dotblas.o -L/usr/lib -lblas -lg2c -o build/lib.linux-x86_64-2.3/scipy/lib/_dotblas.so /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.4/../../../libblas.so when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/3.4.4/../../../libblas.a when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas /usr/bin/ld: cannot find -lblas collect2: ld returned 1 exit status I really need to get this running. Help? Thanks, Anoop From dd55 at cornell.edu Wed Nov 23 16:08:01 2005 From: dd55 at cornell.edu (Darren Dale) Date: Wed Nov 23 16:08:01 2005 Subject: [Numpy-discussion] Problem compiling scipy-core In-Reply-To: <1132789622.7570.4.camel@localhost.localdomain> References: <1132789622.7570.4.camel@localhost.localdomain> Message-ID: <200511231907.25815.dd55@cornell.edu> On Wednesday 23 November 2005 6:47 pm, Anoop wrote: > Hi. I have problems installing scipy-core from source code on a Rocks > Cluster distribution of Linux running on x86-64 hardware. The error > messages tell me: > > /usr/bin/g77 -shared > build/temp.linux-x86_64-2.3/scipy/corelib/blasdot/_dotblas.o -L/usr/lib > -lblas -lg2c -o build/lib.linux-x86_64-2.3/scipy/lib/_dotblas.so > /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for > -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching > for -lblas /usr/bin/ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.4/../../../libblas.so when searching > for -lblas /usr/bin/ld: skipping incompatible > /usr/lib/gcc/x86_64-redhat-linux/3.4.4/../../../libblas.a when searching > for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when > searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a > when searching for -lblas /usr/bin/ld: cannot find -lblas > collect2: ld returned 1 exit status I saw something similar to this just the other day. In my case, the problem was that scipy discovered broken libblas.* softlinks in /usr/lib and tried to build against them. It looks like something similar is happening here, check your atlas/blas/lapack installation. Darren -- Darren S. Dale, Ph.D. dd55 at cornell.edu From gyromagnetic at gmail.com Fri Nov 25 06:25:03 2005 From: gyromagnetic at gmail.com (gf) Date: Fri Nov 25 06:25:03 2005 Subject: [Numpy-discussion] help in improving data analysis code Message-ID: Hi, I am a newbie to Numeric/numarray programming and would appreciate your help in improving the code below (which I'm sure is quite ugly to an experienced numarray programmer). An analysis we are carrying out requires the following: 1. evaluate the mean of a set of data 2. eliminate the data point farthest from the mean 3. repeat steps 1 and 2 until a certain specified fraction of points has been eliminated. Since this analysis will have to be performed (probably repeatedly) on approximately ten thousand data sets, each of which contains 100-500 points, I would like the code to be as fast as possible. Thanks for your help. -g ==== from numarray import add, array, asarray, absolute, argsort, floor, take, size def mean(m,axis=0): m = asarray(m) return add.reduce(m,axis)/float(m.shape[axis]) def eliminate_outliers(dat,frac): num_to_eliminate = int(floor(size(dat,0)*frac)) for i in range(num_to_eliminate): ind = argsort(absolute(dat-mean(dat)),0) sdat = take(dat,ind,0)[:,0] dat = sdat[:-1] return dat #-------------------------------------------------------------------- if __name__ == "__main__": from MLab import rand sz = 100 nn = rand(sz,1) nn[:10] = 20*rand(10,1) nn[sz-10:] = -20*rand(10,1) print eliminate_outliers(nn,0.10) From faltet at carabos.com Fri Nov 25 07:28:04 2005 From: faltet at carabos.com (Francesc Altet) Date: Fri Nov 25 07:28:04 2005 Subject: [Numpy-discussion] help in improving data analysis code In-Reply-To: References: Message-ID: <200511251627.11520.faltet@carabos.com> A Divendres 25 Novembre 2005 15:24, gf va escriure: > > from numarray import add, array, asarray, absolute, argsort, floor, take, > size > > def mean(m,axis=0): > m = asarray(m) > return add.reduce(m,axis)/float(m.shape[axis]) > > def eliminate_outliers(dat,frac): > num_to_eliminate = int(floor(size(dat,0)*frac)) > for i in range(num_to_eliminate): > ind = argsort(absolute(dat-mean(dat)),0) > sdat = take(dat,ind,0)[:,0] > dat = sdat[:-1] > return dat > > #-------------------------------------------------------------------- > > if __name__ == "__main__": > from MLab import rand > sz = 100 > nn = rand(sz,1) > nn[:10] = 20*rand(10,1) > nn[sz-10:] = -20*rand(10,1) > print eliminate_outliers(nn,0.10) For sz=100, the next line of code is 10x faster on my machine (more if sz is bigger): print nn[argsort(abs(nn_c-nn_c.mean()),0)][:-int(sz*0.10),0] I haven't checked it very carefully, so you should double check it. BTW, you will need to use the numarray MLab interface: from numarray.mlab import rand Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Fri Nov 25 07:33:02 2005 From: faltet at carabos.com (Francesc Altet) Date: Fri Nov 25 07:33:02 2005 Subject: [Numpy-discussion] help in improving data analysis code In-Reply-To: <200511251627.11520.faltet@carabos.com> References: <200511251627.11520.faltet@carabos.com> Message-ID: <200511251632.04416.faltet@carabos.com> A Divendres 25 Novembre 2005 16:27, Francesc Altet va escriure: > print nn[argsort(abs(nn_c-nn_c.mean()),0)][:-int(sz*0.10),0] Ups. I have had a confusion. This should work better ;-) print nn[argsort(abs(nn-nn.mean()),0)][:-int(sz*0.10),0] -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From gyromagnetic at gmail.com Fri Nov 25 14:19:00 2005 From: gyromagnetic at gmail.com (gf) Date: Fri Nov 25 14:19:00 2005 Subject: [Numpy-discussion] Re: help in improving data analysis code In-Reply-To: References: Message-ID: From: Francesc Altet Re: help in improving data analysis code 2005-11-25 07:32 >>A Divendres 25 Novembre 2005 16:27, Francesc Altet va escriure: > > print nn[argsort(abs(nn_c-nn_c.mean()),0)][:-int(sz*0.10),0] >> >> Ups. I have had a confusion. This should work better ;-) >> >> print nn[argsort(abs(nn-nn.mean()),0)][:-int(sz*0.10),0] Hi Francesc, Thank you for the suggestions. Your code is performing a different task than mine was. In particular, I believe it does not 're-mean' the data after removing each point. However, based on the great ideas from your code, I now have the function below that looks to be more efficient (although I haven't measured it). Any additional suggestions are appreciated. -g ==== from numarray import argsort, floor, absolute def eliminate_outliers(data,frac): num_to_eliminate = int(floor(data.size())*frac) for i in range(num_to_eliminate): data = data[argsort(absolute(data-data.mean()),0)][:-1,0] return data if __name__ == "__main__": from numarray.mlab import rand sz = 100 nn = rand(sz,1) nn[:10] = 20*rand(10,1) nn[sz-10:] = -20*rand(10,1) print eliminate_outliers(nn,0.10) From service at paypal.com Tue Nov 29 05:40:25 2005 From: service at paypal.com (PayPal) Date: Tue Nov 29 05:40:25 2005 Subject: [Numpy-discussion] Your PayPal Account Could Be Suspended Message-ID: <20051129134036.F39C71FE227@gosecweb.gosecure.pt> An HTML attachment was scrubbed... URL: