From cbarker at jps.net Thu Nov 2 16:46:05 2000 From: cbarker at jps.net (Chris Barker) Date: Thu, 02 Nov 2000 13:46:05 -0800 Subject: [Numpy-discussion] "formstring()" in place? References: <004a01c037ee$964dd040$0200a8c0@home> <39EE3D2C.51A92A71@jps.net> <200010191725.TAA14895@chinon.cnrs-orleans.fr> Message-ID: <3A01E09D.14AFC501@jps.net> I have a narge array of type "1" (single bytes). I need to convert it to Int32, in the manner that fromstring() would. Right now, I am doing: Array = fromstring(Array.tostring(),'f') This works fine, but what concerns me is that I need to do this on potentially HUGE arrays, and if I understand this right, I am going to create a copy with tostring, and then another copy with fromstring, that then gets referenced to Array, at which point the first original copy gets de-referenced, and should be deleted, and the temporary one gets deleted at some point in this process. I don't know when stuff created in the middle of a statement gets deleted, so I could potentially have three copies of the data around at the same time, and at least two. Since it is exactly the same C array, I'd like to be able to do this without making any copies at all. Is it possible? It seems like it should be a simple matter of changing the typecode and shape, but is this possible? While I'm asking questions: can I byteswap in place as well? The greater problem: To give a little background, and to see if anyone has a better idea of how to do what I am doing, I thought I'd run through the task that I really need to do. I am reading a binary file full of a lot of data. I have some control over the form of the file, but it needs to be compact, so I can't just make everything the same large type. The file is essentially a whole bunch of records, each of which is a collection of a couple of different types, and which I would eventually like to get into a couple of NumPy arrays. My first cut at the problem was to read each record one at a time in a loop, and use the struct module to convert everything. This worked fine, but was pretty darn slow, so I am now doing it all with NumPy like this (one example, I have more complex ones): num_bytes = 9 # number of bytes in a record: two longs and a char # read all the data into a single byte array data = fromstring(file.read(num_bytes*num_timesteps*num_LEs),'1') # rearrange into 3-d array data.shape = (num_timesteps,num_LEs,num_bytes) # extract LE data: LEs = data[:,:,:8] # extract flag data flags = data[:,:,8] # convert LE data to longs LEs = fromstring(LEs.tostring(),Int32) if endian == 'L': # byteswap if required LEs = LEs.byteswapped() # convert to 3-d array LEs.shape = (num_timesteps,num_LEs,2) Anyone have any better ideas on how to do this? Thanks, -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ From jhauser at ifm.uni-kiel.de Thu Nov 2 19:05:47 2000 From: jhauser at ifm.uni-kiel.de (Janko Hauser) Date: Fri, 3 Nov 2000 01:05:47 +0100 (CET) Subject: [Numpy-discussion] "formstring()" in place? In-Reply-To: <3A01E09D.14AFC501@jps.net> References: <004a01c037ee$964dd040$0200a8c0@home> <39EE3D2C.51A92A71@jps.net> <200010191725.TAA14895@chinon.cnrs-orleans.fr> <3A01E09D.14AFC501@jps.net> Message-ID: <20001103000547.14011.qmail@lisboa.ifm.uni-kiel.de> Use the numpyio module from Travis. With this it should be possible to read the data directly and do any typecode conversion you want with. It has fread and fwrite functions, and it can be used with any NumPy type like Int0 in your case. It's part of the signaltools package. http://oliphant.netpedia.net/signaltools_0.5.2.tar.gz HTH, __Janko From stannous at cisco.com Thu Nov 2 21:45:24 2000 From: stannous at cisco.com (Sam Tannous) Date: Thu, 2 Nov 2000 21:45:24 -0500 Subject: [Numpy-discussion] Static linking with Python 2.0 (Setup.local) Message-ID: <20001102214524.A4240@cisco.com> I successfully installed the latest release with the distutils package. But now I need to statically link the Numeric package into my Python 2.0 interpreter (it's easier for me to ship "frozen" python binaries for some lab machines...one file instead of hundreds). Could anyone please email me (or post) the section of your Modules/Setup.local that can do this? Something like: _numpy _numpymodule.c -L/users/stannous/lib -L/usr/lib -lm -I/users/stannous/include just doesn't work... TIA, Sam From pauldubois at home.com Thu Nov 2 21:57:38 2000 From: pauldubois at home.com (Paul F. Dubois) Date: Thu, 2 Nov 2000 18:57:38 -0800 Subject: [Numpy-discussion] Static linking with Python 2.0 (Setup.local) In-Reply-To: <20001102214524.A4240@cisco.com> Message-ID: I believe you can deduce the modules being built and the sources / include / lib directories for them by examining setup.py. When we switched to distutils we chucked our Setup file. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Sam Tannous Sent: Thursday, November 02, 2000 6:45 PM To: numpy-discussion at sourceforge.net Subject: [Numpy-discussion] Static linking with Python 2.0 (Setup.local) I successfully installed the latest release with the distutils package. But now I need to statically link the Numeric package into my Python 2.0 interpreter (it's easier for me to ship "frozen" python binaries for some lab machines...one file instead of hundreds). Could anyone please email me (or post) the section of your Modules/Setup.local that can do this? Something like: _numpy _numpymodule.c -L/users/stannous/lib -L/usr/lib -lm -I/users/stannous/includ e just doesn't work... TIA, Sam _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net http://lists.sourceforge.net/mailman/listinfo/numpy-discussion From Bernd.Rinn at uni-konstanz.de Fri Nov 3 09:44:00 2000 From: Bernd.Rinn at uni-konstanz.de (Bernd Rinn) Date: Fri, 3 Nov 2000 15:44:00 +0100 Subject: [Numpy-discussion] performance of real_fft with fftpacklite.c Message-ID: <20001103154400.A17057@planck.physik.uni-konstanz.de> Hello, does anyone know why the performance of FFT.real with fftpacklite.c is so unballanced for n=2**i and different values of i? My example is: ======================================================================= from Numeric import array,Float from FFT import real_fft from time import time i=1 while i<20: n = 2**long(i) a=array(range(long(1),n),Float) anfang = time() b = real_fft(a) ende=time() print "i=", i, " time: ", ende-anfang i+=1 ======================================================================= and the result shows (on a Pentium-III-700 under Linux): ================================================================= i= 1 time: 0.000182032585144 i= 2 time: 0.000133991241455 i= 3 time: 0.00012195110321 i= 4 time: 0.000123977661133 i= 5 time: 0.000131964683533 i= 6 time: 0.000155925750732 i= 7 time: 0.000362992286682 i= 8 time: 0.000240921974182 i= 9 time: 0.000506043434143 i= 10 time: 0.00064492225647 i= 11 time: 0.00177395343781 i= 12 time: 0.0025269985199 i= 13 time: 0.886229038239 i= 14 time: 0.0219050645828 i= 15 time: 0.0808279514313 i= 16 time: 0.327404975891 i= 17 time: 482.979220986 i= 18 time: 0.803207993507 i= 19 time: 7782.23972797 ================================================================= when I am using an array a of length 2**19 and giving the command b=real_fft(a,n=2**long(20)) the time drops from over two hours CPU-time to about 1.5 seconds. I know that fftpacklite.c is not specially optimized, but isn't a FFT method with vectors lenghts that are powers of 2 be supposed to show a more predictible run-time behavior? Could perhaps anyone point me to a free FFTPACK FORTRAN package for Linux with g77 that performs better than the default package? Any hint would be greatly appreciated. Best regards, Bernd Rinn P.S.: Please CC to Bernd.Rinn at uni-konstanz.de since I am not a member of the list. -- Bernd Rinn Fakult?t f?r Physik Universit?t Konstanz Tel. 07531/88-3812, e-mail: Bernd.Rinn at uni-konstanz.de PGP-Fingerprint: 1F AC 31 64 FF EF A9 67 6E 0D 4C 26 0B E7 ED 5C From pauldubois at home.com Fri Nov 3 10:32:45 2000 From: pauldubois at home.com (Paul F. Dubois) Date: Fri, 3 Nov 2000 07:32:45 -0800 Subject: [Numpy-discussion] "formstring()" in place? In-Reply-To: <3A01E09D.14AFC501@jps.net> Message-ID: >>> y=array([1,2,3], '1') >>> y array([1, 2, 3],'1') >>> y.astype(Int32) array([1, 2, 3],'i') >>> -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Chris Barker Sent: Thursday, November 02, 2000 1:46 PM Cc: numpy-discussion at sourceforge.net Subject: [Numpy-discussion] "formstring()" in place? I have a narge array of type "1" (single bytes). I need to convert it to Int32, in the manner that fromstring() would. Right now, I am doing: Array = fromstring(Array.tostring(),'f') This works fine, but what concerns me is that I need to do this on potentially HUGE arrays, and if I understand this right, I am going to create a copy with tostring, and then another copy with fromstring, that then gets referenced to Array, at which point the first original copy gets de-referenced, and should be deleted, and the temporary one gets deleted at some point in this process. I don't know when stuff created in the middle of a statement gets deleted, so I could potentially have three copies of the data around at the same time, and at least two. Since it is exactly the same C array, I'd like to be able to do this without making any copies at all. Is it possible? It seems like it should be a simple matter of changing the typecode and shape, but is this possible? While I'm asking questions: can I byteswap in place as well? The greater problem: To give a little background, and to see if anyone has a better idea of how to do what I am doing, I thought I'd run through the task that I really need to do. I am reading a binary file full of a lot of data. I have some control over the form of the file, but it needs to be compact, so I can't just make everything the same large type. The file is essentially a whole bunch of records, each of which is a collection of a couple of different types, and which I would eventually like to get into a couple of NumPy arrays. My first cut at the problem was to read each record one at a time in a loop, and use the struct module to convert everything. This worked fine, but was pretty darn slow, so I am now doing it all with NumPy like this (one example, I have more complex ones): num_bytes = 9 # number of bytes in a record: two longs and a char # read all the data into a single byte array data = fromstring(file.read(num_bytes*num_timesteps*num_LEs),'1') # rearrange into 3-d array data.shape = (num_timesteps,num_LEs,num_bytes) # extract LE data: LEs = data[:,:,:8] # extract flag data flags = data[:,:,8] # convert LE data to longs LEs = fromstring(LEs.tostring(),Int32) if endian == 'L': # byteswap if required LEs = LEs.byteswapped() # convert to 3-d array LEs.shape = (num_timesteps,num_LEs,2) Anyone have any better ideas on how to do this? Thanks, -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net http://lists.sourceforge.net/mailman/listinfo/numpy-discussion From ransom at cfa.harvard.edu Fri Nov 3 10:43:36 2000 From: ransom at cfa.harvard.edu (Scott Ransom) Date: Fri, 03 Nov 2000 10:43:36 -0500 Subject: [Numpy-discussion] performance of real_fft with fftpacklite.c References: <20001103154400.A17057@planck.physik.uni-konstanz.de> Message-ID: <3A02DD28.F4CB9F8E@cfa.harvard.edu> Bernd Rinn wrote: > > Hello, > > does anyone know why the performance of FFT.real with fftpacklite.c is > so unballanced for n=2**i and different values of i? My example is: Hi, The problem is that your routine gives arrays of length 2**n-1 not 2**n! So for the arrays where the CPU time is huge, you are FFTing a length that is a prime number (or at least not easily factorable). FFTPACK has to do a brute force DFT instead of a FFT! (Ah, the beauty of n*log(n)...) You can see that this is the case by printing the lengths of the arrays: ------------------------------- from Numeric import array,Float from FFT import real_fft from time import time i=1 while i<20: n = 2**long(i) a=array(range(long(1),n),Float) print len(a) i=i+1 ------------------------------- What you should try instead is the following: ------------------------------- from Numeric import arange,Float from FFT import real_fft from time import time i=1 while i<20: n = 2**i a=arange(n, typecode=Float) anfang = time() b = real_fft(a) ende=time() print "i=", i, " time: ", ende-anfang i=i+1 ------------------------------- Which gives the following run-times on my Pentium 450 under Linux: i= 1 time: 0.000313997268677 i= 2 time: 0.000239014625549 i= 3 time: 0.000229954719543 i= 4 time: 0.000240087509155 i= 5 time: 0.000240087509155 i= 6 time: 0.000257968902588 i= 7 time: 0.000322103500366 i= 8 time: 0.000348091125488 i= 9 time: 0.000599980354309 i= 10 time: 0.000900983810425 i= 11 time: 0.0018150806427 i= 12 time: 0.00341892242432 i= 13 time: 0.00806891918182 i= 14 time: 0.0169370174408 i= 15 time: 0.038006067276 i= 16 time: 0.0883399248123 i= 17 time: 0.199723005295 i= 18 time: 0.661148071289 i= 19 time: 0.976199030876 Hope this helps, Scott -- Scott M. Ransom Address: Harvard-Smithsonian CfA Phone: (617) 495-4142 60 Garden St. MS 10 email: ransom at cfa.harvard.edu Cambridge, MA 02138 GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From cbarker at jps.net Fri Nov 3 14:40:45 2000 From: cbarker at jps.net (Chris Barker) Date: Fri, 03 Nov 2000 11:40:45 -0800 Subject: [Numpy-discussion] "formstring()" in place? References: <004a01c037ee$964dd040$0200a8c0@home> <39EE3D2C.51A92A71@jps.net> <200010191725.TAA14895@chinon.cnrs-orleans.fr> <3A01E09D.14AFC501@jps.net> <20001103000547.14011.qmail@lisboa.ifm.uni-kiel.de> Message-ID: <3A0314BD.9B0CEF95@jps.net> Janko Hauser wrote: > Use the numpyio module from Travis. With this it should be possible to > read the data directly and do any typecode conversion you want > with. It has fread and fwrite functions, and it can be used with any > NumPy type like Int0 in your case. It's part of the signaltools > package. > > http://oliphant.netpedia.net/signaltools_0.5.2.tar.gz I've downloaded it , and it looks pretty handy. It does include a byteswap-in-place, which I need. What is not clear to me from the minimal docs is whether I can read file set up like: long long char long long char .... and have it put the longs into one array, and the chars into another. Also, It wasn't clear whether I could put use it to read a file that has already been opened, starting at the file's current position. I am working with a file that has a text header, so I can't just suck in the whole thing until I've parsed out the header. I can figure out the answer to these questions with some reding of the source, but it wasn't obvious at first glance, so it would be great if someone knows the answer off the top of there head. Travis? By the way, there seem to be a few methods that produce a copy, rather than doing things in place, where it seems more intuitive to do it in place. byteswapped() and astype() come to mind. With byteswapped, I imagine it's rare that you would want to keep a copy around. With astype it would also be rare to keep a copy around, but since it changes the size of the array, I imagine it would be a lot harder to code as an in-place operation. Is there a reason these operations are not available in-place? or is it just that no one has seen enough of a need to write the code. -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ From cbarker at jps.net Fri Nov 3 14:28:55 2000 From: cbarker at jps.net (Chris Barker) Date: Fri, 03 Nov 2000 11:28:55 -0800 Subject: [Numpy-discussion] "formstring()" in place? References: Message-ID: <3A0311F7.9160400B@jps.net> "Paul F. Dubois" wrote: > >>> y=array([1,2,3], '1') > >>> y > array([1, 2, 3],'1') > >>> y.astype(Int32) > array([1, 2, 3],'i') Actually, this is exactly NOT what I want to do. In this case, each 1 byte interger was converted to a 4byte integer, of the same VALUE. What I want is to convert each SET of four bytes into a SINGLE 4 byte integer as it: >>> a = array([1,2,3,4],'1') >>> a = fromstring(a.tostring(),Int32) >>> a array([67305985],'i') The four one byte items in a are turned into one four byte item. What I want is to be able to do this in place, rather than have tostring() create a copy. I think fromstring may create a copy as well, having a possible total of three copies around at once. Does anyone know how many copies will be around at once with this line of code? -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ From jhauser at ifm.uni-kiel.de Fri Nov 3 16:06:00 2000 From: jhauser at ifm.uni-kiel.de (Janko Hauser) Date: Fri, 3 Nov 2000 22:06:00 +0100 (CET) Subject: [Numpy-discussion] "formstring()" in place? In-Reply-To: <3A0311F7.9160400B@jps.net> References: <3A0311F7.9160400B@jps.net> Message-ID: <20001103210600.15152.qmail@lisboa.ifm.uni-kiel.de> Chris Barker writes: > "Paul F. Dubois" wrote: > > >>> y=array([1,2,3], '1') > > >>> y > > array([1, 2, 3],'1') > > >>> y.astype(Int32) > > array([1, 2, 3],'i') > > Actually, this is exactly NOT what I want to do. In this case, each 1 > byte interger was converted to a 4byte integer, of the same VALUE. What > I want is to convert each SET of four bytes into a SINGLE 4 byte integer > as it: > > >>> a = array([1,2,3,4],'1') > >>> a = fromstring(a.tostring(),Int32) > >>> a > array([67305985],'i') > A brut force way would be to do the transformation yourself :-) >>> bits=array([1,256,256*256,256*256*256]) >>> sum(array([1,2,3,4])*bits) 67305985 So you need to reshape your array into (?,4) and multiply by bits. And regarding your numpyio question, you can also read characters, which are then put into an array by itself. It seems you have a very messy file format (but the data world is never easy) HTH, __Janko -- Institut fuer Meereskunde phone: 49-431-597 3989 Dept. Theoretical Oceanography fax : 49-431-565876 Duesternbrooker Weg 20 email: jhauser at ifm.uni-kiel.de 24105 Kiel, Germany From jhauser at ifm.uni-kiel.de Fri Nov 3 16:15:06 2000 From: jhauser at ifm.uni-kiel.de (Janko Hauser) Date: Fri, 3 Nov 2000 22:15:06 +0100 (CET) Subject: [Numpy-discussion] "formstring()" in place? In-Reply-To: <3A0311F7.9160400B@jps.net> References: <3A0311F7.9160400B@jps.net> Message-ID: <20001103211506.15164.qmail@lisboa.ifm.uni-kiel.de> Sorry I forgot to mention that these two operations can be done inplace, but the result can not be stored inplace, as the shape is changing. So you need to live with one copy, if your array a is of type 'i'. >>> a=array([1,2,3,4]) >>> multiply(a,bits,a) array([ 1, 512, 196608, 67108864]) >>> a array([ 1, 512, 196608, 67108864]) HTH, __Janko multiply(array([1,2,3,4]),bits,a) From pete at visionart.com Tue Nov 7 19:07:27 2000 From: pete at visionart.com (Pete Shinners) Date: Tue, 7 Nov 2000 16:07:27 -0800 Subject: [Numpy-discussion] bilinear/sampled scaling Message-ID: <012c01c04917$e3951e50$f73f93cd@visionart.com> i have image data in a 2D array that i'd like to change the size of. currently, i'm just doing a dirty sampled scaling that i came up with. this works, but can only do a 2x resize. dst[::2,::2] = src dst[1::2,::2] = src dst[:,1::2] = dst[:,::2] this isn't too bad, but it's definitely one of my bottlenecks, can the code to do this be sped through some ingenious use of numeric? i'm also trying to figure out a clean, fast way to do bilinear scaling on my data. it seems like there would be something in numeric to do linear resampling, but i haven't figured it out. i'd like to be able to resample stuff like audio data as well as just image data. thanks for any pointers into this. From jbaddor at physics.mcgill.ca Fri Nov 17 20:56:52 2000 From: jbaddor at physics.mcgill.ca (Jean-Bernard Addor) Date: Fri, 17 Nov 2000 20:56:52 -0500 (EST) Subject: [Numpy-discussion] Fatal Python error: Unprotected floating point exception Message-ID: Hi Numpy people! My nice numpy code generates very few Inf numbers, wich destroy the results of my longer simulations. I was dreaming to have the processor raising an interruption and python caching it to locate and understand quickly why and how it is happening and correct the code. I currently use Python 1.5.2 with Numeric 11 on debian linux 2 . I made some very desappointing test with the module fpectl The last result I got is: Fatal Python error: Unprotected floating point exception Abort Do I have to understand that my Numpy is not compatible with fpectl? Any idea if a more up to date Numpy would be compatible? I find no info on: http://sourceforge.net/projects/numpy Thanks for your help. Jean-Bernard From pauldubois at home.com Sat Nov 18 11:34:30 2000 From: pauldubois at home.com (Paul F. Dubois) Date: Sat, 18 Nov 2000 08:34:30 -0800 Subject: [Numpy-discussion] Fatal Python error: Unprotected floating point exception In-Reply-To: Message-ID: To make numpy work with fpectl someone needs to add some macro calls to its source. This has not been done. Please visit sourceforge.net/projects/numpy and get a new release of Numpy. Perhaps that will correct your problem. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Jean-Bernard Addor Sent: Friday, November 17, 2000 5:57 PM To: numpy-discussion at lists.sourceforge.net Subject: [Numpy-discussion] Fatal Python error: Unprotected floating point exception Hi Numpy people! My nice numpy code generates very few Inf numbers, wich destroy the results of my longer simulations. I was dreaming to have the processor raising an interruption and python caching it to locate and understand quickly why and how it is happening and correct the code. I currently use Python 1.5.2 with Numeric 11 on debian linux 2 . I made some very desappointing test with the module fpectl The last result I got is: Fatal Python error: Unprotected floating point exception Abort Do I have to understand that my Numpy is not compatible with fpectl? Any idea if a more up to date Numpy would be compatible? I find no info on: http://sourceforge.net/projects/numpy Thanks for your help. Jean-Bernard _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net http://lists.sourceforge.net/mailman/listinfo/numpy-discussion From Andreas.Reigber at dlr.de Mon Nov 20 12:09:21 2000 From: Andreas.Reigber at dlr.de (Anderl) Date: Mon, 20 Nov 2000 18:09:21 +0100 (CET) Subject: [Numpy-discussion] (no subject) Message-ID: Hello, I'm quite new to numpy, trying to migrate from IDL. I'm not able to find any 'shift' function in numpy, i.e. a function that shifts the content of an array by a certain number of elements. a = [1,2,3,4,5,6] shift(a,2) = [3,4,5,6,1,2] In IDL this works even on multidemiensional arrays, b=shift(a,4,7,5) shifts by 4 in the first, 7 in the second and 5 in the third component. Is there some similar module existing? Thanks & best regards, Andreas ------------------------------------------------------------------- Andreas Reigber Institut fuer Hochfrequenztechnik DLR - Oberpfaffenhofen Tel. : ++49-8153-282367 Postfach 1116 eMail: Andreas.Reigber at dlr.de D-82230 Wessling ------------------------------------------------------------------- From cbarker at jps.net Wed Nov 22 18:02:30 2000 From: cbarker at jps.net (Chris Barker) Date: Wed, 22 Nov 2000 15:02:30 -0800 Subject: [Numpy-discussion] Trouble with extension using NumPy on the Mac References: Message-ID: <3A1C5086.1679B31A@jps.net> Hi all, I'm cross posting this to the NumPy List and the MacPython list, because it involves NumPy on the Mac, so I'm not sure which group can be most helpfull. It took me a while to get this far, and I finally got everything to compile, but now I have a crashing problem. It seems to be a problem with PyArray_Type not being seen as a PyObject. I am pretty new to C, but I have consulted with a number of folks I work with that know a lot more than I do, and this seems to be a pretty esoteric C issue. It also seems to be compiler dependent, because I have all this working fine on Linux with gcc. I have chopped my problem down into a very small function that demonstrates the problem. The function takes a contiguous NumPy array of Floats (doubles) and multiplies every element by two (in place), and returns None. Here is the code as it works on Linux: #include "Python.h" #include "arrayobject.h" /* A function that doubles an array of Floats in place*/ static PyObject * minimal_doublearray(PyObject *self, PyObject *args) { PyArrayObject *array; int i, num_elements; double *data_array ; if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &array)) return NULL; if (array->descr->type_num != PyArray_DOUBLE) { PyErr_SetString(PyExc_ValueError, "array must be of type Float"); return NULL; } data_array = (double *) array->data; /*num_elements = PyArray_Size((PyObject *) array);*/ num_elements = PyArray_Size(array); printf("The size of the array is: %i elements\n",num_elements); for (i= 0; i < num_elements; i++) data_array[i] = 2 * data_array[i]; return Py_None; } static PyMethodDef minimalMethods[] = { {"doublearray", minimal_doublearray, METH_VARARGS}, {NULL, NULL} /* Sentinel */ }; void initminimal() { (void) Py_InitModule("minimal", minimalMethods); } Note that the call to: "PyArray_Size(array)" gives me a "./minimal.c:28: warning: passing arg 1 from incompatible pointer type " on gcc on linux. In CodeWarrior on the Mac, it is an fatal error. With the typcast (see previous commented out line) it gives no warnings, and compiles on both systems. Here is a small script to test it: #!/usr/bin/env python from Numeric import * import minimal print "\nTesting doublearray" a = arange(10.0) print a minimal.doublearray(a) print a print "the second version should be doubled" This works fine on Linux, and gives the appropriate errors if the wrong type object is passed in. On the Mac, it crashes. Trying various things, I found that it crashes on the if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &array)) return NULL; line. If I use: if (!PyArg_ParseTuple(args, "O", &array)) return NULL; It works. Then it crashes on the: num_elements = PyArray_Size((PyObject *) array); line. If I use another way to determine the number of elements, like: num_elements = 1; for (i=0,ind,i++) num_elements = num_elements * array->dimensions[1]; Then I can get it to work. What is going on? anyone have any suggestions? MacPython 1.5.2c The NumPy that came with MacPython 1.5.2c CodeWarrior pro 5. Thanks for any suggestions, -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ From sdhyok at email.unc.edu Thu Nov 23 17:18:51 2000 From: sdhyok at email.unc.edu (Daehyok Shin) Date: Thu, 23 Nov 2000 14:18:51 -0800 Subject: [Numpy-discussion] multiarray=a standard type in python? Message-ID: <000f01c0559b$5f87c8c0$bc111918@nc.rr.com> Would you tell me what's going on recently in making multiarray as a standard type of python? Daehyok Shin (Peter) From npirzkal at eso.org Mon Nov 27 09:08:28 2000 From: npirzkal at eso.org (Nor Pirzkal) Date: Mon, 27 Nov 2000 15:08:28 +0100 Subject: [Numpy-discussion] Freezing Numeric Python code problem under Linux Message-ID: <3A226ADC.DEA98107@eso.org> Hi, i am having troubles freezing some python code with call the Numeric 17.1.1 module. Here is an example of what happens under Linux Red Hat 6.2 and Python 2.0: $ cat t.py import Numeric a = Numeric.ones([10,10]) b = a * 10 print b $ python ~/Python-2.0/Tools/freeze/freeze.py -o tt t.py ... $ cd tt $ ./t Traceback (most recent call last): File "t.py", line 1, in ? File "/scisoft/python/lib/python2.0/site-packages/Numeric/Numeric.py", line 79, in ? import multiarray ImportError: /scisoft/python/lib/python2.0/site-packages/Numeric/multiarray.so: undefined symbol: _Py_NoneStruct The very same code, under Solaris 2.6 and Python 2.0 works just fine. Code which does not use the Numeric package freeze just fine under Linux, so I think that this point to some problem/incompatibility of Numeric with freeze.py. Does anybody have a suggestion, or a work-around?? Nor From pauldubois at home.com Mon Nov 27 18:04:34 2000 From: pauldubois at home.com (Paul F. Dubois) Date: Mon, 27 Nov 2000 15:04:34 -0800 Subject: [Numpy-discussion] quick optimization In-Reply-To: <006e01c02ca6$1ece3e40$0200a8c0@home> Message-ID: This optimization will be in the next release. Thanks! -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Pete Shinners Sent: Monday, October 02, 2000 10:58 AM To: Numpy Discussion Subject: [Numpy-discussion] quick optimization i've got a quick optimization for the arrayobject.c source. it speeds my usage of numpy up by about 100%. i've tested with other numpy apps and noticed a minimum of about 20% speed. anyways, in "do_sliced_copy", change out the following block: if (src_nd == 0 && dest_nd == 0) { for(j=0; j I have released 17.2. This is essentially a catch-up-to-CVS release. If you don't upgrade nothing astonishing will be missing from your life, unless you are using MA, which has been worked on some since 17.1. I installed a performance enhancement submitted by Pete Shinners, who said: "I've got a quick optimization for the arrayobject.c source. it speeds my usage of numpy up by about 100%. i've tested with other numpy apps and noticed a minimum of about 20% speed." From hoel at germanlloyd.org Tue Nov 28 06:38:31 2000 From: hoel at germanlloyd.org (Berthold =?iso-8859-1?q?H=F6llmann?=) Date: 28 Nov 2000 12:38:31 +0100 Subject: [Numpy-discussion] 17.2 source release In-Reply-To: "Paul F. Dubois"'s message of "Mon, 27 Nov 2000 16:45:01 -0800" References: Message-ID: "Paul F. Dubois" writes: > I have released 17.2. This is essentially a catch-up-to-CVS release. If you > don't upgrade nothing astonishing will be missing from your life, unless you > are using MA, which has been worked on some since 17.1. > > I installed a performance enhancement submitted by Pete Shinners, who said: > "I've got a quick optimization for the arrayobject.c source. > it speeds my usage of numpy up by about 100%. i've tested with > other numpy apps and noticed a minimum of about 20% speed." Hello, Is it 17.1.2 or is it missing on Sourceforge? O only see a 17.1.2 which has a corresponding file date, but no 10.2 Greetings Berthold -- email: hoel at GermanLloyd.org ) tel. : +49 (40) 3 61 49 - 73 74 ( C[_] These opinions might be mine, but never those of my employer. From pauldubois at home.com Tue Nov 28 09:51:47 2000 From: pauldubois at home.com (Paul F. Dubois) Date: Tue, 28 Nov 2000 06:51:47 -0800 Subject: [Numpy-discussion] 17.1.2 source release In-Reply-To: Message-ID: Thank you for pointing out my error; it is of course 17.1.2. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Berthold Hollmann Sent: Tuesday, November 28, 2000 3:39 AM To: dubois at users.sourceforge.net Cc: Numpy-Discussion at Lists. Sourceforge. Net Subject: Re: [Numpy-discussion] 17.2 source release "Paul F. Dubois" writes: > I have released 17.2. This is essentially a catch-up-to-CVS release. If you > don't upgrade nothing astonishing will be missing from your life, unless you > are using MA, which has been worked on some since 17.1. > > I installed a performance enhancement submitted by Pete Shinners, who said: > "I've got a quick optimization for the arrayobject.c source. > it speeds my usage of numpy up by about 100%. i've tested with > other numpy apps and noticed a minimum of about 20% speed." Hello, Is it 17.1.2 or is it missing on Sourceforge? O only see a 17.1.2 which has a corresponding file date, but no 10.2 Greetings Berthold -- email: hoel at GermanLloyd.org ) tel. : +49 (40) 3 61 49 - 73 74 ( C[_] These opinions might be mine, but never those of my employer. _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net http://lists.sourceforge.net/mailman/listinfo/numpy-discussion From sdhyok at email.unc.edu Wed Nov 29 15:28:57 2000 From: sdhyok at email.unc.edu (Daehyok Shin) Date: Wed, 29 Nov 2000 12:28:57 -0800 Subject: [Numpy-discussion] Initialization of array? References: <3A226ADC.DEA98107@eso.org> Message-ID: <002201c05a43$0007e540$1523a318@nc.rr.com> When I initialize an array, I use a = ones(shape)*initial_val But, I am wondering if Numpy has more efficient way. For example, a = array(initial_value, shape) Peter From cbarker at jps.net Wed Nov 29 14:13:39 2000 From: cbarker at jps.net (Chris Barker) Date: Wed, 29 Nov 2000 11:13:39 -0800 Subject: [Numpy-discussion] Initialization of array? References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> Message-ID: <3A255563.247AB2F0@jps.net> Daehyok Shin wrote: > When I initialize an array, I use > a = ones(shape)*initial_val > > But, I am wondering if Numpy has more efficient way. > For example, > a = array(initial_value, shape) I don't know if it's any more efficient (what you have is pretty fast already), but another option is to use resize: >>> shape = (3,4) >>> initial_val = 5.0 >>> resize(initial_val,shape) array([[ 5., 5., 5., 5.], [ 5., 5., 5., 5.], [ 5., 5., 5., 5.]]) >>> -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ From rob at hooft.net Wed Nov 29 16:00:58 2000 From: rob at hooft.net (Rob W. W. Hooft) Date: Wed, 29 Nov 2000 22:00:58 +0100 Subject: [Numpy-discussion] Initialization of array? In-Reply-To: <002201c05a43$0007e540$1523a318@nc.rr.com> References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> Message-ID: <14885.28298.768829.921606@temoleh.chem.uu.nl> >>>>> "DS" == Daehyok Shin writes: DS> When I initialize an array, I use a = ones(shape)*initial_val DS> But, I am wondering if Numpy has more efficient way. For example, DS> a = array(initial_value, shape) Looking at the definition of "ones": def ones(shape, typecode='l', savespace=0): """ones(shape, typecode=Int, savespace=0) returns an array of the given dimensions which is initialized to all ones. """ return zeros(shape, typecode, savespace)+array(1, typecode) It looks like you could try a=zeros(shape)+initial_val instead. Hm.. I might do some experimenting. Rob -- ===== rob at hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! ========= From sdhyok at email.unc.edu Wed Nov 29 19:29:50 2000 From: sdhyok at email.unc.edu (Daehyok Shin) Date: Wed, 29 Nov 2000 16:29:50 -0800 Subject: [Numpy-discussion] Initialization of array? References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> <3A255563.247AB2F0@jps.net> Message-ID: <00b001c05a64$a6f0c400$1523a318@nc.rr.com> Comparing the performance, resize() seems to be more effient than ones(). Daehyok Shin ----- Original Message ----- From: "Chris Barker" Cc: Sent: Wednesday, November 29, 2000 11:13 AM Subject: Re: [Numpy-discussion] Initialization of array? > Daehyok Shin wrote: > > When I initialize an array, I use > > a = ones(shape)*initial_val > > > > But, I am wondering if Numpy has more efficient way. > > For example, > > a = array(initial_value, shape) > > I don't know if it's any more efficient (what you have is pretty fast > already), but another option is to use resize: > > >>> shape = (3,4) > > >>> initial_val = 5.0 > > >>> resize(initial_val,shape) > > array([[ 5., 5., 5., 5.], > [ 5., 5., 5., 5.], > [ 5., 5., 5., 5.]]) > >>> > > -Chris > > > -- > Christopher Barker, > Ph.D. > cbarker at jps.net --- --- --- > http://www.jps.net/cbarker -----@@ -----@@ -----@@ > ------@@@ ------@@@ ------@@@ > Water Resources Engineering ------ @ ------ @ ------ @ > Coastal and Fluvial Hydrodynamics ------- --------- -------- > ------------------------------------------------------------------------ > ------------------------------------------------------------------------ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > http://lists.sourceforge.net/mailman/listinfo/numpy-discussion > From sdhyok at email.unc.edu Wed Nov 29 19:37:54 2000 From: sdhyok at email.unc.edu (Daehyok Shin) Date: Wed, 29 Nov 2000 16:37:54 -0800 Subject: [Numpy-discussion] Initialization of array? References: <3A226ADC.DEA98107@eso.org><002201c05a43$0007e540$1523a318@nc.rr.com> <14885.28298.768829.921606@temoleh.chem.uu.nl> Message-ID: <00b701c05a65$c754e400$1523a318@nc.rr.com> Initialization on huge arrays is frequent operations in scientific programming. It must be efficient as much as possible. So, I was surprisized to see the inner codes of ones() in Numpy. It maybe use calloc() rather than malloc() in C level, then for(..) for addition. Why not use malloc() and for(...) simultaneously in C level with a command such as: a = arrray(1,shape=(10000,10000)) Daehyok Shin ----- Original Message ----- From: "Rob W. W. Hooft" To: "Daehyok Shin" Cc: Sent: Wednesday, November 29, 2000 1:00 PM Subject: Re: [Numpy-discussion] Initialization of array? > >>>>> "DS" == Daehyok Shin writes: > > DS> When I initialize an array, I use a = ones(shape)*initial_val > > DS> But, I am wondering if Numpy has more efficient way. For example, > DS> a = array(initial_value, shape) > > Looking at the definition of "ones": > > def ones(shape, typecode='l', savespace=0): > """ones(shape, typecode=Int, savespace=0) returns an array of the given > dimensions which is initialized to all ones. > """ > return zeros(shape, typecode, savespace)+array(1, typecode) > > It looks like you could try a=zeros(shape)+initial_val instead. > > Hm.. I might do some experimenting. > > Rob > > -- > ===== rob at hooft.net http://www.hooft.net/people/rob/ ===== > ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== > ===== PGPid 0xFA19277D ========================== Use Linux! ========= > From rob at hooft.net Wed Nov 29 16:56:41 2000 From: rob at hooft.net (Rob W. W. Hooft) Date: Wed, 29 Nov 2000 22:56:41 +0100 Subject: [Numpy-discussion] Initialization of array? In-Reply-To: <002201c05a43$0007e540$1523a318@nc.rr.com> References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> Message-ID: <14885.31641.678148.786465@temoleh.chem.uu.nl> I had another look at the definition of "ones" and of another routine I frequently use: arange. It appears that even without rewriting them in C, some speedup can be achieved: - in ones(), the + 1 should be done "in place", saving about 15%, more if you run out of processor cache: amigo[167]~%3% /usr/local/bin/python test_ones.py Numeric.ones 10 -> 0.098ms Numeric.ones 100 -> 0.103ms Numeric.ones 1000 -> 0.147ms Numeric.ones 10000 -> 0.830ms Numeric.ones 100000 -> 11.900ms Numeric.zeros 10 -> 0.021ms Numeric.zeros 100 -> 0.022ms Numeric.zeros 1000 -> 0.026ms Numeric.zeros 10000 -> 0.290ms Numeric.zeros 100000 -> 4.000ms Add inplace 10 -> 0.091ms Add inplace 100 -> 0.094ms Add inplace 1000 -> 0.127ms Add inplace 10000 -> 0.690ms Add inplace 100000 -> 8.100ms Reshape 1 10 -> 0.320ms Reshape 1 100 -> 0.436ms Reshape 1 1000 -> 1.553ms Reshape 1 10000 -> 12.910ms Reshape 1 100000 -> 141.200ms Also notice that zeros() is 4-5 times faster than ones(), so it may pay to reimplement ones in C as well (it is used in indices() and arange()). The "resize 1" alternative is much slower. - in arange, additional 10% can be saved by adding brackets around (start+(stop-stop)) (in addition to the gain by the faster "ones"): amigo[168]~%3% /usr/local/bin/python test_arange.py Numeric.arange 10 -> 0.390ms Numeric.arange 100 -> 0.410ms Numeric.arange 1000 -> 0.670ms Numeric.arange 10000 -> 4.100ms Numeric.arange 100000 -> 59.000ms Optimized 10 -> 0.340ms Optimized 100 -> 0.360ms Optimized 1000 -> 0.580ms Optimized 10000 -> 3.500ms Optimized 100000 -> 48.000ms Regards, Rob Hooft -------------- next part -------------- A non-text attachment was scrubbed... Name: test_arange.py Type: application/octet-stream Size: 1308 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_ones.py Type: application/octet-stream Size: 812 bytes Desc: not available URL: -------------- next part -------------- -- ===== rob at hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! ========= From hinsen at cnrs-orleans.fr Thu Nov 30 09:25:36 2000 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Thu, 30 Nov 2000 15:25:36 +0100 Subject: [Numpy-discussion] Initialization of array? In-Reply-To: <14885.31641.678148.786465@temoleh.chem.uu.nl> (rob@hooft.net) References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> <14885.31641.678148.786465@temoleh.chem.uu.nl> Message-ID: <200011301425.PAA23184@chinon.cnrs-orleans.fr> > I had another look at the definition of "ones" and of another routine > I frequently use: arange. It appears that even without rewriting them > in C, some speedup can be achieved: > > - in ones(), the + 1 should be done "in place", saving about 15%, more > if you run out of processor cache: I'd also try assignment in place: def ones(shape, typecode='l', savespace=0): a = zeros(shape, typecode, savespace) a[len(shape)*[slice(0, None)]] = 1 return a Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From rob at hooft.net Thu Nov 30 10:33:46 2000 From: rob at hooft.net (Rob W. W. Hooft) Date: Thu, 30 Nov 2000 16:33:46 +0100 Subject: [Numpy-discussion] Initialization of array? In-Reply-To: <200011301425.PAA23184@chinon.cnrs-orleans.fr> References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> <14885.31641.678148.786465@temoleh.chem.uu.nl> <200011301425.PAA23184@chinon.cnrs-orleans.fr> Message-ID: <14886.29530.432385.896300@temoleh.chem.uu.nl> KH> I had another look at the definition of "ones" and of another routine KH> I frequently use: arange. It appears that even without rewriting them KH> in C, some speedup can be achieved: KH> KH> - in ones(), the + 1 should be done "in place", saving about 15%, more KH> if you run out of processor cache: KH> I'd also try assignment in place: KH> def ones(shape, typecode='l', savespace=0): KH> a = zeros(shape, typecode, savespace) KH> a[len(shape)*[slice(0, None)]] = 1 KH> return a This is even faster, but it is better to write "a[...] = 1", because your manual calculation of "..." gives a large overhead for small arrays. On another machine this time: Numeric.ones 10 -> 0.254ms Numeric.ones 100 -> 0.268ms Numeric.ones 1000 -> 0.340ms Numeric.ones 10000 -> 1.960ms Numeric.ones 100000 -> 29.300ms Numeric.zeros 10 -> 0.055ms Numeric.zeros 100 -> 0.059ms Numeric.zeros 1000 -> 0.068ms Numeric.zeros 10000 -> 0.430ms Numeric.zeros 100000 -> 9.800ms Add inplace 10 -> 0.246ms Add inplace 100 -> 0.255ms Add inplace 1000 -> 0.312ms Add inplace 10000 -> 1.270ms Add inplace 100000 -> 18.100ms Assign inplace 10 -> 0.192ms Assign inplace 100 -> 0.201ms Assign inplace 1000 -> 0.242ms Assign inplace 10000 -> 1.010ms Assign inplace 100000 -> 16.300ms Reshape 1 10 -> 0.842ms Reshape 1 100 -> 1.175ms Reshape 1 1000 -> 4.100ms Reshape 1 10000 -> 35.100ms Reshape 1 100000 -> 368.600ms Rob -- ===== rob at hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! ========= -------------- next part -------------- A non-text attachment was scrubbed... Name: test_ones.py Type: application/octet-stream Size: 958 bytes Desc: not available URL: From szport at runet.com Thu Nov 30 21:05:42 2000 From: szport at runet.com (szport at runet.com) Date: Thu, 30 November 2000 21:05:42 MSK Subject: [Numpy-discussion] (no subject) Message-ID: <200011301805.VAA04440@www.pob.ru> > I had another look at the definition of \"ones\" and of another routine > > I frequently use: arange. It appears that even without rewriting them > > in C, some speedup can be achieved: > > > > - in ones(), the + 1 should be done \"in place\", saving about 15%, more > > if you run out of processor cache: > > I\'d also try assignment in place: > > def ones(shape, typecode=\'l\', savespace=0): > a = zeros(shape, typecode, savespace) > a[len(shape)*[slice(0, None)]] = 1 > return a > > Konrad. Is the following definition faster or not? def ones(shape, typecode=\'l\', savespace=0): a = zeros( (product(shape),), typecode, savespace) a[:] = 1 a.shape = shape return a Zaur From cbarker at jps.net Thu Nov 2 16:46:05 2000 From: cbarker at jps.net (Chris Barker) Date: Thu, 02 Nov 2000 13:46:05 -0800 Subject: [Numpy-discussion] "formstring()" in place? References: <004a01c037ee$964dd040$0200a8c0@home> <39EE3D2C.51A92A71@jps.net> <200010191725.TAA14895@chinon.cnrs-orleans.fr> Message-ID: <3A01E09D.14AFC501@jps.net> I have a narge array of type "1" (single bytes). I need to convert it to Int32, in the manner that fromstring() would. Right now, I am doing: Array = fromstring(Array.tostring(),'f') This works fine, but what concerns me is that I need to do this on potentially HUGE arrays, and if I understand this right, I am going to create a copy with tostring, and then another copy with fromstring, that then gets referenced to Array, at which point the first original copy gets de-referenced, and should be deleted, and the temporary one gets deleted at some point in this process. I don't know when stuff created in the middle of a statement gets deleted, so I could potentially have three copies of the data around at the same time, and at least two. Since it is exactly the same C array, I'd like to be able to do this without making any copies at all. Is it possible? It seems like it should be a simple matter of changing the typecode and shape, but is this possible? While I'm asking questions: can I byteswap in place as well? The greater problem: To give a little background, and to see if anyone has a better idea of how to do what I am doing, I thought I'd run through the task that I really need to do. I am reading a binary file full of a lot of data. I have some control over the form of the file, but it needs to be compact, so I can't just make everything the same large type. The file is essentially a whole bunch of records, each of which is a collection of a couple of different types, and which I would eventually like to get into a couple of NumPy arrays. My first cut at the problem was to read each record one at a time in a loop, and use the struct module to convert everything. This worked fine, but was pretty darn slow, so I am now doing it all with NumPy like this (one example, I have more complex ones): num_bytes = 9 # number of bytes in a record: two longs and a char # read all the data into a single byte array data = fromstring(file.read(num_bytes*num_timesteps*num_LEs),'1') # rearrange into 3-d array data.shape = (num_timesteps,num_LEs,num_bytes) # extract LE data: LEs = data[:,:,:8] # extract flag data flags = data[:,:,8] # convert LE data to longs LEs = fromstring(LEs.tostring(),Int32) if endian == 'L': # byteswap if required LEs = LEs.byteswapped() # convert to 3-d array LEs.shape = (num_timesteps,num_LEs,2) Anyone have any better ideas on how to do this? Thanks, -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ From jhauser at ifm.uni-kiel.de Thu Nov 2 19:05:47 2000 From: jhauser at ifm.uni-kiel.de (Janko Hauser) Date: Fri, 3 Nov 2000 01:05:47 +0100 (CET) Subject: [Numpy-discussion] "formstring()" in place? In-Reply-To: <3A01E09D.14AFC501@jps.net> References: <004a01c037ee$964dd040$0200a8c0@home> <39EE3D2C.51A92A71@jps.net> <200010191725.TAA14895@chinon.cnrs-orleans.fr> <3A01E09D.14AFC501@jps.net> Message-ID: <20001103000547.14011.qmail@lisboa.ifm.uni-kiel.de> Use the numpyio module from Travis. With this it should be possible to read the data directly and do any typecode conversion you want with. It has fread and fwrite functions, and it can be used with any NumPy type like Int0 in your case. It's part of the signaltools package. http://oliphant.netpedia.net/signaltools_0.5.2.tar.gz HTH, __Janko From stannous at cisco.com Thu Nov 2 21:45:24 2000 From: stannous at cisco.com (Sam Tannous) Date: Thu, 2 Nov 2000 21:45:24 -0500 Subject: [Numpy-discussion] Static linking with Python 2.0 (Setup.local) Message-ID: <20001102214524.A4240@cisco.com> I successfully installed the latest release with the distutils package. But now I need to statically link the Numeric package into my Python 2.0 interpreter (it's easier for me to ship "frozen" python binaries for some lab machines...one file instead of hundreds). Could anyone please email me (or post) the section of your Modules/Setup.local that can do this? Something like: _numpy _numpymodule.c -L/users/stannous/lib -L/usr/lib -lm -I/users/stannous/include just doesn't work... TIA, Sam From pauldubois at home.com Thu Nov 2 21:57:38 2000 From: pauldubois at home.com (Paul F. Dubois) Date: Thu, 2 Nov 2000 18:57:38 -0800 Subject: [Numpy-discussion] Static linking with Python 2.0 (Setup.local) In-Reply-To: <20001102214524.A4240@cisco.com> Message-ID: I believe you can deduce the modules being built and the sources / include / lib directories for them by examining setup.py. When we switched to distutils we chucked our Setup file. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Sam Tannous Sent: Thursday, November 02, 2000 6:45 PM To: numpy-discussion at sourceforge.net Subject: [Numpy-discussion] Static linking with Python 2.0 (Setup.local) I successfully installed the latest release with the distutils package. But now I need to statically link the Numeric package into my Python 2.0 interpreter (it's easier for me to ship "frozen" python binaries for some lab machines...one file instead of hundreds). Could anyone please email me (or post) the section of your Modules/Setup.local that can do this? Something like: _numpy _numpymodule.c -L/users/stannous/lib -L/usr/lib -lm -I/users/stannous/includ e just doesn't work... TIA, Sam _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net http://lists.sourceforge.net/mailman/listinfo/numpy-discussion From Bernd.Rinn at uni-konstanz.de Fri Nov 3 09:44:00 2000 From: Bernd.Rinn at uni-konstanz.de (Bernd Rinn) Date: Fri, 3 Nov 2000 15:44:00 +0100 Subject: [Numpy-discussion] performance of real_fft with fftpacklite.c Message-ID: <20001103154400.A17057@planck.physik.uni-konstanz.de> Hello, does anyone know why the performance of FFT.real with fftpacklite.c is so unballanced for n=2**i and different values of i? My example is: ======================================================================= from Numeric import array,Float from FFT import real_fft from time import time i=1 while i<20: n = 2**long(i) a=array(range(long(1),n),Float) anfang = time() b = real_fft(a) ende=time() print "i=", i, " time: ", ende-anfang i+=1 ======================================================================= and the result shows (on a Pentium-III-700 under Linux): ================================================================= i= 1 time: 0.000182032585144 i= 2 time: 0.000133991241455 i= 3 time: 0.00012195110321 i= 4 time: 0.000123977661133 i= 5 time: 0.000131964683533 i= 6 time: 0.000155925750732 i= 7 time: 0.000362992286682 i= 8 time: 0.000240921974182 i= 9 time: 0.000506043434143 i= 10 time: 0.00064492225647 i= 11 time: 0.00177395343781 i= 12 time: 0.0025269985199 i= 13 time: 0.886229038239 i= 14 time: 0.0219050645828 i= 15 time: 0.0808279514313 i= 16 time: 0.327404975891 i= 17 time: 482.979220986 i= 18 time: 0.803207993507 i= 19 time: 7782.23972797 ================================================================= when I am using an array a of length 2**19 and giving the command b=real_fft(a,n=2**long(20)) the time drops from over two hours CPU-time to about 1.5 seconds. I know that fftpacklite.c is not specially optimized, but isn't a FFT method with vectors lenghts that are powers of 2 be supposed to show a more predictible run-time behavior? Could perhaps anyone point me to a free FFTPACK FORTRAN package for Linux with g77 that performs better than the default package? Any hint would be greatly appreciated. Best regards, Bernd Rinn P.S.: Please CC to Bernd.Rinn at uni-konstanz.de since I am not a member of the list. -- Bernd Rinn Fakult?t f?r Physik Universit?t Konstanz Tel. 07531/88-3812, e-mail: Bernd.Rinn at uni-konstanz.de PGP-Fingerprint: 1F AC 31 64 FF EF A9 67 6E 0D 4C 26 0B E7 ED 5C From pauldubois at home.com Fri Nov 3 10:32:45 2000 From: pauldubois at home.com (Paul F. Dubois) Date: Fri, 3 Nov 2000 07:32:45 -0800 Subject: [Numpy-discussion] "formstring()" in place? In-Reply-To: <3A01E09D.14AFC501@jps.net> Message-ID: >>> y=array([1,2,3], '1') >>> y array([1, 2, 3],'1') >>> y.astype(Int32) array([1, 2, 3],'i') >>> -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Chris Barker Sent: Thursday, November 02, 2000 1:46 PM Cc: numpy-discussion at sourceforge.net Subject: [Numpy-discussion] "formstring()" in place? I have a narge array of type "1" (single bytes). I need to convert it to Int32, in the manner that fromstring() would. Right now, I am doing: Array = fromstring(Array.tostring(),'f') This works fine, but what concerns me is that I need to do this on potentially HUGE arrays, and if I understand this right, I am going to create a copy with tostring, and then another copy with fromstring, that then gets referenced to Array, at which point the first original copy gets de-referenced, and should be deleted, and the temporary one gets deleted at some point in this process. I don't know when stuff created in the middle of a statement gets deleted, so I could potentially have three copies of the data around at the same time, and at least two. Since it is exactly the same C array, I'd like to be able to do this without making any copies at all. Is it possible? It seems like it should be a simple matter of changing the typecode and shape, but is this possible? While I'm asking questions: can I byteswap in place as well? The greater problem: To give a little background, and to see if anyone has a better idea of how to do what I am doing, I thought I'd run through the task that I really need to do. I am reading a binary file full of a lot of data. I have some control over the form of the file, but it needs to be compact, so I can't just make everything the same large type. The file is essentially a whole bunch of records, each of which is a collection of a couple of different types, and which I would eventually like to get into a couple of NumPy arrays. My first cut at the problem was to read each record one at a time in a loop, and use the struct module to convert everything. This worked fine, but was pretty darn slow, so I am now doing it all with NumPy like this (one example, I have more complex ones): num_bytes = 9 # number of bytes in a record: two longs and a char # read all the data into a single byte array data = fromstring(file.read(num_bytes*num_timesteps*num_LEs),'1') # rearrange into 3-d array data.shape = (num_timesteps,num_LEs,num_bytes) # extract LE data: LEs = data[:,:,:8] # extract flag data flags = data[:,:,8] # convert LE data to longs LEs = fromstring(LEs.tostring(),Int32) if endian == 'L': # byteswap if required LEs = LEs.byteswapped() # convert to 3-d array LEs.shape = (num_timesteps,num_LEs,2) Anyone have any better ideas on how to do this? Thanks, -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net http://lists.sourceforge.net/mailman/listinfo/numpy-discussion From ransom at cfa.harvard.edu Fri Nov 3 10:43:36 2000 From: ransom at cfa.harvard.edu (Scott Ransom) Date: Fri, 03 Nov 2000 10:43:36 -0500 Subject: [Numpy-discussion] performance of real_fft with fftpacklite.c References: <20001103154400.A17057@planck.physik.uni-konstanz.de> Message-ID: <3A02DD28.F4CB9F8E@cfa.harvard.edu> Bernd Rinn wrote: > > Hello, > > does anyone know why the performance of FFT.real with fftpacklite.c is > so unballanced for n=2**i and different values of i? My example is: Hi, The problem is that your routine gives arrays of length 2**n-1 not 2**n! So for the arrays where the CPU time is huge, you are FFTing a length that is a prime number (or at least not easily factorable). FFTPACK has to do a brute force DFT instead of a FFT! (Ah, the beauty of n*log(n)...) You can see that this is the case by printing the lengths of the arrays: ------------------------------- from Numeric import array,Float from FFT import real_fft from time import time i=1 while i<20: n = 2**long(i) a=array(range(long(1),n),Float) print len(a) i=i+1 ------------------------------- What you should try instead is the following: ------------------------------- from Numeric import arange,Float from FFT import real_fft from time import time i=1 while i<20: n = 2**i a=arange(n, typecode=Float) anfang = time() b = real_fft(a) ende=time() print "i=", i, " time: ", ende-anfang i=i+1 ------------------------------- Which gives the following run-times on my Pentium 450 under Linux: i= 1 time: 0.000313997268677 i= 2 time: 0.000239014625549 i= 3 time: 0.000229954719543 i= 4 time: 0.000240087509155 i= 5 time: 0.000240087509155 i= 6 time: 0.000257968902588 i= 7 time: 0.000322103500366 i= 8 time: 0.000348091125488 i= 9 time: 0.000599980354309 i= 10 time: 0.000900983810425 i= 11 time: 0.0018150806427 i= 12 time: 0.00341892242432 i= 13 time: 0.00806891918182 i= 14 time: 0.0169370174408 i= 15 time: 0.038006067276 i= 16 time: 0.0883399248123 i= 17 time: 0.199723005295 i= 18 time: 0.661148071289 i= 19 time: 0.976199030876 Hope this helps, Scott -- Scott M. Ransom Address: Harvard-Smithsonian CfA Phone: (617) 495-4142 60 Garden St. MS 10 email: ransom at cfa.harvard.edu Cambridge, MA 02138 GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 From cbarker at jps.net Fri Nov 3 14:40:45 2000 From: cbarker at jps.net (Chris Barker) Date: Fri, 03 Nov 2000 11:40:45 -0800 Subject: [Numpy-discussion] "formstring()" in place? References: <004a01c037ee$964dd040$0200a8c0@home> <39EE3D2C.51A92A71@jps.net> <200010191725.TAA14895@chinon.cnrs-orleans.fr> <3A01E09D.14AFC501@jps.net> <20001103000547.14011.qmail@lisboa.ifm.uni-kiel.de> Message-ID: <3A0314BD.9B0CEF95@jps.net> Janko Hauser wrote: > Use the numpyio module from Travis. With this it should be possible to > read the data directly and do any typecode conversion you want > with. It has fread and fwrite functions, and it can be used with any > NumPy type like Int0 in your case. It's part of the signaltools > package. > > http://oliphant.netpedia.net/signaltools_0.5.2.tar.gz I've downloaded it , and it looks pretty handy. It does include a byteswap-in-place, which I need. What is not clear to me from the minimal docs is whether I can read file set up like: long long char long long char .... and have it put the longs into one array, and the chars into another. Also, It wasn't clear whether I could put use it to read a file that has already been opened, starting at the file's current position. I am working with a file that has a text header, so I can't just suck in the whole thing until I've parsed out the header. I can figure out the answer to these questions with some reding of the source, but it wasn't obvious at first glance, so it would be great if someone knows the answer off the top of there head. Travis? By the way, there seem to be a few methods that produce a copy, rather than doing things in place, where it seems more intuitive to do it in place. byteswapped() and astype() come to mind. With byteswapped, I imagine it's rare that you would want to keep a copy around. With astype it would also be rare to keep a copy around, but since it changes the size of the array, I imagine it would be a lot harder to code as an in-place operation. Is there a reason these operations are not available in-place? or is it just that no one has seen enough of a need to write the code. -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ From cbarker at jps.net Fri Nov 3 14:28:55 2000 From: cbarker at jps.net (Chris Barker) Date: Fri, 03 Nov 2000 11:28:55 -0800 Subject: [Numpy-discussion] "formstring()" in place? References: Message-ID: <3A0311F7.9160400B@jps.net> "Paul F. Dubois" wrote: > >>> y=array([1,2,3], '1') > >>> y > array([1, 2, 3],'1') > >>> y.astype(Int32) > array([1, 2, 3],'i') Actually, this is exactly NOT what I want to do. In this case, each 1 byte interger was converted to a 4byte integer, of the same VALUE. What I want is to convert each SET of four bytes into a SINGLE 4 byte integer as it: >>> a = array([1,2,3,4],'1') >>> a = fromstring(a.tostring(),Int32) >>> a array([67305985],'i') The four one byte items in a are turned into one four byte item. What I want is to be able to do this in place, rather than have tostring() create a copy. I think fromstring may create a copy as well, having a possible total of three copies around at once. Does anyone know how many copies will be around at once with this line of code? -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ From jhauser at ifm.uni-kiel.de Fri Nov 3 16:06:00 2000 From: jhauser at ifm.uni-kiel.de (Janko Hauser) Date: Fri, 3 Nov 2000 22:06:00 +0100 (CET) Subject: [Numpy-discussion] "formstring()" in place? In-Reply-To: <3A0311F7.9160400B@jps.net> References: <3A0311F7.9160400B@jps.net> Message-ID: <20001103210600.15152.qmail@lisboa.ifm.uni-kiel.de> Chris Barker writes: > "Paul F. Dubois" wrote: > > >>> y=array([1,2,3], '1') > > >>> y > > array([1, 2, 3],'1') > > >>> y.astype(Int32) > > array([1, 2, 3],'i') > > Actually, this is exactly NOT what I want to do. In this case, each 1 > byte interger was converted to a 4byte integer, of the same VALUE. What > I want is to convert each SET of four bytes into a SINGLE 4 byte integer > as it: > > >>> a = array([1,2,3,4],'1') > >>> a = fromstring(a.tostring(),Int32) > >>> a > array([67305985],'i') > A brut force way would be to do the transformation yourself :-) >>> bits=array([1,256,256*256,256*256*256]) >>> sum(array([1,2,3,4])*bits) 67305985 So you need to reshape your array into (?,4) and multiply by bits. And regarding your numpyio question, you can also read characters, which are then put into an array by itself. It seems you have a very messy file format (but the data world is never easy) HTH, __Janko -- Institut fuer Meereskunde phone: 49-431-597 3989 Dept. Theoretical Oceanography fax : 49-431-565876 Duesternbrooker Weg 20 email: jhauser at ifm.uni-kiel.de 24105 Kiel, Germany From jhauser at ifm.uni-kiel.de Fri Nov 3 16:15:06 2000 From: jhauser at ifm.uni-kiel.de (Janko Hauser) Date: Fri, 3 Nov 2000 22:15:06 +0100 (CET) Subject: [Numpy-discussion] "formstring()" in place? In-Reply-To: <3A0311F7.9160400B@jps.net> References: <3A0311F7.9160400B@jps.net> Message-ID: <20001103211506.15164.qmail@lisboa.ifm.uni-kiel.de> Sorry I forgot to mention that these two operations can be done inplace, but the result can not be stored inplace, as the shape is changing. So you need to live with one copy, if your array a is of type 'i'. >>> a=array([1,2,3,4]) >>> multiply(a,bits,a) array([ 1, 512, 196608, 67108864]) >>> a array([ 1, 512, 196608, 67108864]) HTH, __Janko multiply(array([1,2,3,4]),bits,a) From pete at visionart.com Tue Nov 7 19:07:27 2000 From: pete at visionart.com (Pete Shinners) Date: Tue, 7 Nov 2000 16:07:27 -0800 Subject: [Numpy-discussion] bilinear/sampled scaling Message-ID: <012c01c04917$e3951e50$f73f93cd@visionart.com> i have image data in a 2D array that i'd like to change the size of. currently, i'm just doing a dirty sampled scaling that i came up with. this works, but can only do a 2x resize. dst[::2,::2] = src dst[1::2,::2] = src dst[:,1::2] = dst[:,::2] this isn't too bad, but it's definitely one of my bottlenecks, can the code to do this be sped through some ingenious use of numeric? i'm also trying to figure out a clean, fast way to do bilinear scaling on my data. it seems like there would be something in numeric to do linear resampling, but i haven't figured it out. i'd like to be able to resample stuff like audio data as well as just image data. thanks for any pointers into this. From jbaddor at physics.mcgill.ca Fri Nov 17 20:56:52 2000 From: jbaddor at physics.mcgill.ca (Jean-Bernard Addor) Date: Fri, 17 Nov 2000 20:56:52 -0500 (EST) Subject: [Numpy-discussion] Fatal Python error: Unprotected floating point exception Message-ID: Hi Numpy people! My nice numpy code generates very few Inf numbers, wich destroy the results of my longer simulations. I was dreaming to have the processor raising an interruption and python caching it to locate and understand quickly why and how it is happening and correct the code. I currently use Python 1.5.2 with Numeric 11 on debian linux 2 . I made some very desappointing test with the module fpectl The last result I got is: Fatal Python error: Unprotected floating point exception Abort Do I have to understand that my Numpy is not compatible with fpectl? Any idea if a more up to date Numpy would be compatible? I find no info on: http://sourceforge.net/projects/numpy Thanks for your help. Jean-Bernard From pauldubois at home.com Sat Nov 18 11:34:30 2000 From: pauldubois at home.com (Paul F. Dubois) Date: Sat, 18 Nov 2000 08:34:30 -0800 Subject: [Numpy-discussion] Fatal Python error: Unprotected floating point exception In-Reply-To: Message-ID: To make numpy work with fpectl someone needs to add some macro calls to its source. This has not been done. Please visit sourceforge.net/projects/numpy and get a new release of Numpy. Perhaps that will correct your problem. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Jean-Bernard Addor Sent: Friday, November 17, 2000 5:57 PM To: numpy-discussion at lists.sourceforge.net Subject: [Numpy-discussion] Fatal Python error: Unprotected floating point exception Hi Numpy people! My nice numpy code generates very few Inf numbers, wich destroy the results of my longer simulations. I was dreaming to have the processor raising an interruption and python caching it to locate and understand quickly why and how it is happening and correct the code. I currently use Python 1.5.2 with Numeric 11 on debian linux 2 . I made some very desappointing test with the module fpectl The last result I got is: Fatal Python error: Unprotected floating point exception Abort Do I have to understand that my Numpy is not compatible with fpectl? Any idea if a more up to date Numpy would be compatible? I find no info on: http://sourceforge.net/projects/numpy Thanks for your help. Jean-Bernard _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net http://lists.sourceforge.net/mailman/listinfo/numpy-discussion From Andreas.Reigber at dlr.de Mon Nov 20 12:09:21 2000 From: Andreas.Reigber at dlr.de (Anderl) Date: Mon, 20 Nov 2000 18:09:21 +0100 (CET) Subject: [Numpy-discussion] (no subject) Message-ID: Hello, I'm quite new to numpy, trying to migrate from IDL. I'm not able to find any 'shift' function in numpy, i.e. a function that shifts the content of an array by a certain number of elements. a = [1,2,3,4,5,6] shift(a,2) = [3,4,5,6,1,2] In IDL this works even on multidemiensional arrays, b=shift(a,4,7,5) shifts by 4 in the first, 7 in the second and 5 in the third component. Is there some similar module existing? Thanks & best regards, Andreas ------------------------------------------------------------------- Andreas Reigber Institut fuer Hochfrequenztechnik DLR - Oberpfaffenhofen Tel. : ++49-8153-282367 Postfach 1116 eMail: Andreas.Reigber at dlr.de D-82230 Wessling ------------------------------------------------------------------- From cbarker at jps.net Wed Nov 22 18:02:30 2000 From: cbarker at jps.net (Chris Barker) Date: Wed, 22 Nov 2000 15:02:30 -0800 Subject: [Numpy-discussion] Trouble with extension using NumPy on the Mac References: Message-ID: <3A1C5086.1679B31A@jps.net> Hi all, I'm cross posting this to the NumPy List and the MacPython list, because it involves NumPy on the Mac, so I'm not sure which group can be most helpfull. It took me a while to get this far, and I finally got everything to compile, but now I have a crashing problem. It seems to be a problem with PyArray_Type not being seen as a PyObject. I am pretty new to C, but I have consulted with a number of folks I work with that know a lot more than I do, and this seems to be a pretty esoteric C issue. It also seems to be compiler dependent, because I have all this working fine on Linux with gcc. I have chopped my problem down into a very small function that demonstrates the problem. The function takes a contiguous NumPy array of Floats (doubles) and multiplies every element by two (in place), and returns None. Here is the code as it works on Linux: #include "Python.h" #include "arrayobject.h" /* A function that doubles an array of Floats in place*/ static PyObject * minimal_doublearray(PyObject *self, PyObject *args) { PyArrayObject *array; int i, num_elements; double *data_array ; if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &array)) return NULL; if (array->descr->type_num != PyArray_DOUBLE) { PyErr_SetString(PyExc_ValueError, "array must be of type Float"); return NULL; } data_array = (double *) array->data; /*num_elements = PyArray_Size((PyObject *) array);*/ num_elements = PyArray_Size(array); printf("The size of the array is: %i elements\n",num_elements); for (i= 0; i < num_elements; i++) data_array[i] = 2 * data_array[i]; return Py_None; } static PyMethodDef minimalMethods[] = { {"doublearray", minimal_doublearray, METH_VARARGS}, {NULL, NULL} /* Sentinel */ }; void initminimal() { (void) Py_InitModule("minimal", minimalMethods); } Note that the call to: "PyArray_Size(array)" gives me a "./minimal.c:28: warning: passing arg 1 from incompatible pointer type " on gcc on linux. In CodeWarrior on the Mac, it is an fatal error. With the typcast (see previous commented out line) it gives no warnings, and compiles on both systems. Here is a small script to test it: #!/usr/bin/env python from Numeric import * import minimal print "\nTesting doublearray" a = arange(10.0) print a minimal.doublearray(a) print a print "the second version should be doubled" This works fine on Linux, and gives the appropriate errors if the wrong type object is passed in. On the Mac, it crashes. Trying various things, I found that it crashes on the if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &array)) return NULL; line. If I use: if (!PyArg_ParseTuple(args, "O", &array)) return NULL; It works. Then it crashes on the: num_elements = PyArray_Size((PyObject *) array); line. If I use another way to determine the number of elements, like: num_elements = 1; for (i=0,ind,i++) num_elements = num_elements * array->dimensions[1]; Then I can get it to work. What is going on? anyone have any suggestions? MacPython 1.5.2c The NumPy that came with MacPython 1.5.2c CodeWarrior pro 5. Thanks for any suggestions, -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ From sdhyok at email.unc.edu Thu Nov 23 17:18:51 2000 From: sdhyok at email.unc.edu (Daehyok Shin) Date: Thu, 23 Nov 2000 14:18:51 -0800 Subject: [Numpy-discussion] multiarray=a standard type in python? Message-ID: <000f01c0559b$5f87c8c0$bc111918@nc.rr.com> Would you tell me what's going on recently in making multiarray as a standard type of python? Daehyok Shin (Peter) From npirzkal at eso.org Mon Nov 27 09:08:28 2000 From: npirzkal at eso.org (Nor Pirzkal) Date: Mon, 27 Nov 2000 15:08:28 +0100 Subject: [Numpy-discussion] Freezing Numeric Python code problem under Linux Message-ID: <3A226ADC.DEA98107@eso.org> Hi, i am having troubles freezing some python code with call the Numeric 17.1.1 module. Here is an example of what happens under Linux Red Hat 6.2 and Python 2.0: $ cat t.py import Numeric a = Numeric.ones([10,10]) b = a * 10 print b $ python ~/Python-2.0/Tools/freeze/freeze.py -o tt t.py ... $ cd tt $ ./t Traceback (most recent call last): File "t.py", line 1, in ? File "/scisoft/python/lib/python2.0/site-packages/Numeric/Numeric.py", line 79, in ? import multiarray ImportError: /scisoft/python/lib/python2.0/site-packages/Numeric/multiarray.so: undefined symbol: _Py_NoneStruct The very same code, under Solaris 2.6 and Python 2.0 works just fine. Code which does not use the Numeric package freeze just fine under Linux, so I think that this point to some problem/incompatibility of Numeric with freeze.py. Does anybody have a suggestion, or a work-around?? Nor From pauldubois at home.com Mon Nov 27 18:04:34 2000 From: pauldubois at home.com (Paul F. Dubois) Date: Mon, 27 Nov 2000 15:04:34 -0800 Subject: [Numpy-discussion] quick optimization In-Reply-To: <006e01c02ca6$1ece3e40$0200a8c0@home> Message-ID: This optimization will be in the next release. Thanks! -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Pete Shinners Sent: Monday, October 02, 2000 10:58 AM To: Numpy Discussion Subject: [Numpy-discussion] quick optimization i've got a quick optimization for the arrayobject.c source. it speeds my usage of numpy up by about 100%. i've tested with other numpy apps and noticed a minimum of about 20% speed. anyways, in "do_sliced_copy", change out the following block: if (src_nd == 0 && dest_nd == 0) { for(j=0; j I have released 17.2. This is essentially a catch-up-to-CVS release. If you don't upgrade nothing astonishing will be missing from your life, unless you are using MA, which has been worked on some since 17.1. I installed a performance enhancement submitted by Pete Shinners, who said: "I've got a quick optimization for the arrayobject.c source. it speeds my usage of numpy up by about 100%. i've tested with other numpy apps and noticed a minimum of about 20% speed." From hoel at germanlloyd.org Tue Nov 28 06:38:31 2000 From: hoel at germanlloyd.org (Berthold =?iso-8859-1?q?H=F6llmann?=) Date: 28 Nov 2000 12:38:31 +0100 Subject: [Numpy-discussion] 17.2 source release In-Reply-To: "Paul F. Dubois"'s message of "Mon, 27 Nov 2000 16:45:01 -0800" References: Message-ID: "Paul F. Dubois" writes: > I have released 17.2. This is essentially a catch-up-to-CVS release. If you > don't upgrade nothing astonishing will be missing from your life, unless you > are using MA, which has been worked on some since 17.1. > > I installed a performance enhancement submitted by Pete Shinners, who said: > "I've got a quick optimization for the arrayobject.c source. > it speeds my usage of numpy up by about 100%. i've tested with > other numpy apps and noticed a minimum of about 20% speed." Hello, Is it 17.1.2 or is it missing on Sourceforge? O only see a 17.1.2 which has a corresponding file date, but no 10.2 Greetings Berthold -- email: hoel at GermanLloyd.org ) tel. : +49 (40) 3 61 49 - 73 74 ( C[_] These opinions might be mine, but never those of my employer. From pauldubois at home.com Tue Nov 28 09:51:47 2000 From: pauldubois at home.com (Paul F. Dubois) Date: Tue, 28 Nov 2000 06:51:47 -0800 Subject: [Numpy-discussion] 17.1.2 source release In-Reply-To: Message-ID: Thank you for pointing out my error; it is of course 17.1.2. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Berthold Hollmann Sent: Tuesday, November 28, 2000 3:39 AM To: dubois at users.sourceforge.net Cc: Numpy-Discussion at Lists. Sourceforge. Net Subject: Re: [Numpy-discussion] 17.2 source release "Paul F. Dubois" writes: > I have released 17.2. This is essentially a catch-up-to-CVS release. If you > don't upgrade nothing astonishing will be missing from your life, unless you > are using MA, which has been worked on some since 17.1. > > I installed a performance enhancement submitted by Pete Shinners, who said: > "I've got a quick optimization for the arrayobject.c source. > it speeds my usage of numpy up by about 100%. i've tested with > other numpy apps and noticed a minimum of about 20% speed." Hello, Is it 17.1.2 or is it missing on Sourceforge? O only see a 17.1.2 which has a corresponding file date, but no 10.2 Greetings Berthold -- email: hoel at GermanLloyd.org ) tel. : +49 (40) 3 61 49 - 73 74 ( C[_] These opinions might be mine, but never those of my employer. _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net http://lists.sourceforge.net/mailman/listinfo/numpy-discussion From sdhyok at email.unc.edu Wed Nov 29 15:28:57 2000 From: sdhyok at email.unc.edu (Daehyok Shin) Date: Wed, 29 Nov 2000 12:28:57 -0800 Subject: [Numpy-discussion] Initialization of array? References: <3A226ADC.DEA98107@eso.org> Message-ID: <002201c05a43$0007e540$1523a318@nc.rr.com> When I initialize an array, I use a = ones(shape)*initial_val But, I am wondering if Numpy has more efficient way. For example, a = array(initial_value, shape) Peter From cbarker at jps.net Wed Nov 29 14:13:39 2000 From: cbarker at jps.net (Chris Barker) Date: Wed, 29 Nov 2000 11:13:39 -0800 Subject: [Numpy-discussion] Initialization of array? References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> Message-ID: <3A255563.247AB2F0@jps.net> Daehyok Shin wrote: > When I initialize an array, I use > a = ones(shape)*initial_val > > But, I am wondering if Numpy has more efficient way. > For example, > a = array(initial_value, shape) I don't know if it's any more efficient (what you have is pretty fast already), but another option is to use resize: >>> shape = (3,4) >>> initial_val = 5.0 >>> resize(initial_val,shape) array([[ 5., 5., 5., 5.], [ 5., 5., 5., 5.], [ 5., 5., 5., 5.]]) >>> -Chris -- Christopher Barker, Ph.D. cbarker at jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------ From rob at hooft.net Wed Nov 29 16:00:58 2000 From: rob at hooft.net (Rob W. W. Hooft) Date: Wed, 29 Nov 2000 22:00:58 +0100 Subject: [Numpy-discussion] Initialization of array? In-Reply-To: <002201c05a43$0007e540$1523a318@nc.rr.com> References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> Message-ID: <14885.28298.768829.921606@temoleh.chem.uu.nl> >>>>> "DS" == Daehyok Shin writes: DS> When I initialize an array, I use a = ones(shape)*initial_val DS> But, I am wondering if Numpy has more efficient way. For example, DS> a = array(initial_value, shape) Looking at the definition of "ones": def ones(shape, typecode='l', savespace=0): """ones(shape, typecode=Int, savespace=0) returns an array of the given dimensions which is initialized to all ones. """ return zeros(shape, typecode, savespace)+array(1, typecode) It looks like you could try a=zeros(shape)+initial_val instead. Hm.. I might do some experimenting. Rob -- ===== rob at hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! ========= From sdhyok at email.unc.edu Wed Nov 29 19:29:50 2000 From: sdhyok at email.unc.edu (Daehyok Shin) Date: Wed, 29 Nov 2000 16:29:50 -0800 Subject: [Numpy-discussion] Initialization of array? References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> <3A255563.247AB2F0@jps.net> Message-ID: <00b001c05a64$a6f0c400$1523a318@nc.rr.com> Comparing the performance, resize() seems to be more effient than ones(). Daehyok Shin ----- Original Message ----- From: "Chris Barker" Cc: Sent: Wednesday, November 29, 2000 11:13 AM Subject: Re: [Numpy-discussion] Initialization of array? > Daehyok Shin wrote: > > When I initialize an array, I use > > a = ones(shape)*initial_val > > > > But, I am wondering if Numpy has more efficient way. > > For example, > > a = array(initial_value, shape) > > I don't know if it's any more efficient (what you have is pretty fast > already), but another option is to use resize: > > >>> shape = (3,4) > > >>> initial_val = 5.0 > > >>> resize(initial_val,shape) > > array([[ 5., 5., 5., 5.], > [ 5., 5., 5., 5.], > [ 5., 5., 5., 5.]]) > >>> > > -Chris > > > -- > Christopher Barker, > Ph.D. > cbarker at jps.net --- --- --- > http://www.jps.net/cbarker -----@@ -----@@ -----@@ > ------@@@ ------@@@ ------@@@ > Water Resources Engineering ------ @ ------ @ ------ @ > Coastal and Fluvial Hydrodynamics ------- --------- -------- > ------------------------------------------------------------------------ > ------------------------------------------------------------------------ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > http://lists.sourceforge.net/mailman/listinfo/numpy-discussion > From sdhyok at email.unc.edu Wed Nov 29 19:37:54 2000 From: sdhyok at email.unc.edu (Daehyok Shin) Date: Wed, 29 Nov 2000 16:37:54 -0800 Subject: [Numpy-discussion] Initialization of array? References: <3A226ADC.DEA98107@eso.org><002201c05a43$0007e540$1523a318@nc.rr.com> <14885.28298.768829.921606@temoleh.chem.uu.nl> Message-ID: <00b701c05a65$c754e400$1523a318@nc.rr.com> Initialization on huge arrays is frequent operations in scientific programming. It must be efficient as much as possible. So, I was surprisized to see the inner codes of ones() in Numpy. It maybe use calloc() rather than malloc() in C level, then for(..) for addition. Why not use malloc() and for(...) simultaneously in C level with a command such as: a = arrray(1,shape=(10000,10000)) Daehyok Shin ----- Original Message ----- From: "Rob W. W. Hooft" To: "Daehyok Shin" Cc: Sent: Wednesday, November 29, 2000 1:00 PM Subject: Re: [Numpy-discussion] Initialization of array? > >>>>> "DS" == Daehyok Shin writes: > > DS> When I initialize an array, I use a = ones(shape)*initial_val > > DS> But, I am wondering if Numpy has more efficient way. For example, > DS> a = array(initial_value, shape) > > Looking at the definition of "ones": > > def ones(shape, typecode='l', savespace=0): > """ones(shape, typecode=Int, savespace=0) returns an array of the given > dimensions which is initialized to all ones. > """ > return zeros(shape, typecode, savespace)+array(1, typecode) > > It looks like you could try a=zeros(shape)+initial_val instead. > > Hm.. I might do some experimenting. > > Rob > > -- > ===== rob at hooft.net http://www.hooft.net/people/rob/ ===== > ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== > ===== PGPid 0xFA19277D ========================== Use Linux! ========= > From rob at hooft.net Wed Nov 29 16:56:41 2000 From: rob at hooft.net (Rob W. W. Hooft) Date: Wed, 29 Nov 2000 22:56:41 +0100 Subject: [Numpy-discussion] Initialization of array? In-Reply-To: <002201c05a43$0007e540$1523a318@nc.rr.com> References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> Message-ID: <14885.31641.678148.786465@temoleh.chem.uu.nl> I had another look at the definition of "ones" and of another routine I frequently use: arange. It appears that even without rewriting them in C, some speedup can be achieved: - in ones(), the + 1 should be done "in place", saving about 15%, more if you run out of processor cache: amigo[167]~%3% /usr/local/bin/python test_ones.py Numeric.ones 10 -> 0.098ms Numeric.ones 100 -> 0.103ms Numeric.ones 1000 -> 0.147ms Numeric.ones 10000 -> 0.830ms Numeric.ones 100000 -> 11.900ms Numeric.zeros 10 -> 0.021ms Numeric.zeros 100 -> 0.022ms Numeric.zeros 1000 -> 0.026ms Numeric.zeros 10000 -> 0.290ms Numeric.zeros 100000 -> 4.000ms Add inplace 10 -> 0.091ms Add inplace 100 -> 0.094ms Add inplace 1000 -> 0.127ms Add inplace 10000 -> 0.690ms Add inplace 100000 -> 8.100ms Reshape 1 10 -> 0.320ms Reshape 1 100 -> 0.436ms Reshape 1 1000 -> 1.553ms Reshape 1 10000 -> 12.910ms Reshape 1 100000 -> 141.200ms Also notice that zeros() is 4-5 times faster than ones(), so it may pay to reimplement ones in C as well (it is used in indices() and arange()). The "resize 1" alternative is much slower. - in arange, additional 10% can be saved by adding brackets around (start+(stop-stop)) (in addition to the gain by the faster "ones"): amigo[168]~%3% /usr/local/bin/python test_arange.py Numeric.arange 10 -> 0.390ms Numeric.arange 100 -> 0.410ms Numeric.arange 1000 -> 0.670ms Numeric.arange 10000 -> 4.100ms Numeric.arange 100000 -> 59.000ms Optimized 10 -> 0.340ms Optimized 100 -> 0.360ms Optimized 1000 -> 0.580ms Optimized 10000 -> 3.500ms Optimized 100000 -> 48.000ms Regards, Rob Hooft -------------- next part -------------- A non-text attachment was scrubbed... Name: test_arange.py Type: application/octet-stream Size: 1308 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_ones.py Type: application/octet-stream Size: 812 bytes Desc: not available URL: -------------- next part -------------- -- ===== rob at hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! ========= From hinsen at cnrs-orleans.fr Thu Nov 30 09:25:36 2000 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Thu, 30 Nov 2000 15:25:36 +0100 Subject: [Numpy-discussion] Initialization of array? In-Reply-To: <14885.31641.678148.786465@temoleh.chem.uu.nl> (rob@hooft.net) References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> <14885.31641.678148.786465@temoleh.chem.uu.nl> Message-ID: <200011301425.PAA23184@chinon.cnrs-orleans.fr> > I had another look at the definition of "ones" and of another routine > I frequently use: arange. It appears that even without rewriting them > in C, some speedup can be achieved: > > - in ones(), the + 1 should be done "in place", saving about 15%, more > if you run out of processor cache: I'd also try assignment in place: def ones(shape, typecode='l', savespace=0): a = zeros(shape, typecode, savespace) a[len(shape)*[slice(0, None)]] = 1 return a Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From rob at hooft.net Thu Nov 30 10:33:46 2000 From: rob at hooft.net (Rob W. W. Hooft) Date: Thu, 30 Nov 2000 16:33:46 +0100 Subject: [Numpy-discussion] Initialization of array? In-Reply-To: <200011301425.PAA23184@chinon.cnrs-orleans.fr> References: <3A226ADC.DEA98107@eso.org> <002201c05a43$0007e540$1523a318@nc.rr.com> <14885.31641.678148.786465@temoleh.chem.uu.nl> <200011301425.PAA23184@chinon.cnrs-orleans.fr> Message-ID: <14886.29530.432385.896300@temoleh.chem.uu.nl> KH> I had another look at the definition of "ones" and of another routine KH> I frequently use: arange. It appears that even without rewriting them KH> in C, some speedup can be achieved: KH> KH> - in ones(), the + 1 should be done "in place", saving about 15%, more KH> if you run out of processor cache: KH> I'd also try assignment in place: KH> def ones(shape, typecode='l', savespace=0): KH> a = zeros(shape, typecode, savespace) KH> a[len(shape)*[slice(0, None)]] = 1 KH> return a This is even faster, but it is better to write "a[...] = 1", because your manual calculation of "..." gives a large overhead for small arrays. On another machine this time: Numeric.ones 10 -> 0.254ms Numeric.ones 100 -> 0.268ms Numeric.ones 1000 -> 0.340ms Numeric.ones 10000 -> 1.960ms Numeric.ones 100000 -> 29.300ms Numeric.zeros 10 -> 0.055ms Numeric.zeros 100 -> 0.059ms Numeric.zeros 1000 -> 0.068ms Numeric.zeros 10000 -> 0.430ms Numeric.zeros 100000 -> 9.800ms Add inplace 10 -> 0.246ms Add inplace 100 -> 0.255ms Add inplace 1000 -> 0.312ms Add inplace 10000 -> 1.270ms Add inplace 100000 -> 18.100ms Assign inplace 10 -> 0.192ms Assign inplace 100 -> 0.201ms Assign inplace 1000 -> 0.242ms Assign inplace 10000 -> 1.010ms Assign inplace 100000 -> 16.300ms Reshape 1 10 -> 0.842ms Reshape 1 100 -> 1.175ms Reshape 1 1000 -> 4.100ms Reshape 1 10000 -> 35.100ms Reshape 1 100000 -> 368.600ms Rob -- ===== rob at hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! ========= -------------- next part -------------- A non-text attachment was scrubbed... Name: test_ones.py Type: application/octet-stream Size: 958 bytes Desc: not available URL: From szport at runet.com Thu Nov 30 21:05:42 2000 From: szport at runet.com (szport at runet.com) Date: Thu, 30 November 2000 21:05:42 MSK Subject: [Numpy-discussion] (no subject) Message-ID: <200011301805.VAA04440@www.pob.ru> > I had another look at the definition of \"ones\" and of another routine > > I frequently use: arange. It appears that even without rewriting them > > in C, some speedup can be achieved: > > > > - in ones(), the + 1 should be done \"in place\", saving about 15%, more > > if you run out of processor cache: > > I\'d also try assignment in place: > > def ones(shape, typecode=\'l\', savespace=0): > a = zeros(shape, typecode, savespace) > a[len(shape)*[slice(0, None)]] = 1 > return a > > Konrad. Is the following definition faster or not? def ones(shape, typecode=\'l\', savespace=0): a = zeros( (product(shape),), typecode, savespace) a[:] = 1 a.shape = shape return a Zaur