Mailman 3 April 2008 - SciPy-User

Re: [SciPy-user] Sparse matrix: division by vector
by Dinesh B Vadhia April 29, 2008

April 29, 2008

Does this work for row sums? For example, assuming A is in CSR and rowSum[i] stores the i-th row sum can you do > A.data /= rowSum[A.indices] Dinesh -------------------------------------------------------------------------------- From: Nathan Bell <wnbell <at> gmail.com> Subject: Re: Sparse matrix: division by vector Newsgroups: gmane.comp.python.scientific.user Date: 2008-04-29 18:16:19 GMT (7 hours and 44 minutes ago) On Tue, Apr 29, 2008 at 12:16 PM, Dinesh B Vadhia <dineshbvadhia <at> hotmail.com> wrote: > > > Sparse matrix A is in csr_matrix format and I want to divide each column > element j of A by A's column j sum (where colSum is a numpy vector) ie. > > > A[:,j] = A[:,j]/colSum[j] > > What is the most efficient way to achieve this apart from brute force ie. > > > A[i,j] = A[i,j]/colSum[j] ? > > I can initially create A in a different c**_matrix method if that helps but > the final A has to be in csr_matrix form. > Assuing A is in CSR and colSum[j] stores the j-th column sum you can do A.data /= colSum[A.indices] -- Nathan Bell wnbell <at> gmail.com http://graphics.cs.uiuc.edu/~wnbell/

1 0

Sparse matrix: division by vector
by Dinesh B Vadhia April 29, 2008

April 29, 2008

Sparse matrix A is in csr_matrix format and I want to divide each column element j of A by A's column j sum (where colSum is a numpy vector) ie. > A[:,j] = A[:,j]/colSum[j] What is the most efficient way to achieve this apart from brute force ie. > A[i,j] = A[i,j]/colSum[j] ? I can initially create A in a different c**_matrix method if that helps but the final A has to be in csr_matrix form. Cheers! Dinesh

3 3

Linear Interpolation 2
by Andrea Gavana April 29, 2008

April 29, 2008

Hi All, as I didn't get any answer, I thought I might repost with some more analysis the trouble I am having with interp2d. The attached script produces 3 different results depending on the value of the parameter numColumns, i.e.: 1) numColumns = 20 Traceback (most recent call last): File "C:\MyProjects\LinearInterpolation.py", line 27, in <module> function = interp2d(xx, yy, z, kind="linear") File "C:\Python25\Lib\site-packages\scipy\interpolate\interpolate.py", line 113, in __init__ self.tck = fitpack.bisplrep(self.x, self.y, self.z, kx=kx, ky=ky, s=0.) File "C:\Python25\Lib\site-packages\scipy\interpolate\fitpack.py", line 702, in bisplrep tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) ValueError: Invalid inputs. 2) numColumns = 200 Traceback (most recent call last): File "C:\MyProjects\LinearInterpolation.py", line 27, in <module> function = interp2d(xx, yy, z, kind="linear") File "C:\Python25\Lib\site-packages\scipy\interpolate\interpolate.py", line 113, in __init__ self.tck = fitpack.bisplrep(self.x, self.y, self.z, kx=kx, ky=ky, s=0.) File "C:\Python25\Lib\site-packages\scipy\interpolate\fitpack.py", line 702, in bisplrep tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) MemoryError (MemoryError? With a 372x200 matrix???) 3) numColumns = 2000 Traceback (most recent call last): File "C:\MyProjects\LinearInterpolation.py", line 27, in <module> function = interp2d(xx, yy, z, kind="linear") File "C:\Python25\Lib\site-packages\scipy\interpolate\interpolate.py", line 113, in __init__ self.tck = fitpack.bisplrep(self.x, self.y, self.z, kx=kx, ky=ky, s=0.) File "C:\Python25\Lib\site-packages\scipy\interpolate\fitpack.py", line 702, in bisplrep tx,ty,nxest,nyest,wrk,lwrk1,lwrk2) OverflowError: long int too large to convert to int Is there anyone who could tell me, please, what I am doing wrong? This is on Windows XP 1GB RAM, 3.7 GHz, Python 2.5, scipy 0.6.0, numpy 1.0.4. Thank you for your suggestions. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/

1 0

code for weave / blitz function wrapper generation
by Hoyt Koepke April 29, 2008

April 29, 2008

Hello, I recently wrote a general purpose function for automatically creating wrappers to C++ functions using weave, and I thought others might find it useful as well. In particular, I think a good place for it would be as a method in the weave ext_module class if others agree. I'm also looking for thoughts, suggestions, and for more people to test it. All the heavy lifting is done by weave, so hopefully the subtle errors are minimal. I've attached the code and an example (with setup.py) for people to look at. To try it out, run ./setup.py build and then ./test_wrapper.py. The interesting function is in weave_wrap_functions.py. Here's a brief overview: To create a function module, you specify the python name, the cpp function's call name, a list of argument types as strings, and whether it returns a value. It supports argument types of (from the doc string): Scalars: "double" (python float) "float" (python float) "int" (python int) "long int" (python int) Blitz Arrays (converted from numpy arrays), 1 <= dim <= 11 "Array<double, dim>", (numpy dtype=float64) "Array<float, dim>", (numpy dtype=float32) "Array<long, dim>" (numpy dtype=int32) "Array<long int, dim>" (numpy dtype=int32) "Array<int, dim>" (numpy dtype=int32) "Array<short int, dim>" (numpy dtype=int16) "Array<short, dim>" (numpy dtype=int16) "Array<unsigned long int, dim>" (numpy dtype=uint32) "Array<unsigned int, dim>" (numpy dtype=uint32) "Array<unsigned, dim>" (numpy dtype=uint32) "Array<size_t, dim>" (numpy dtype=uint32) "Array<unsigned short int, dim>" (numpy dtype=uint16) C++ types: "string" (python string) Thus, for example, if you have the following c++ function definition: string printData(Array<float, 2>& array2d, Array<long int, 1>& idxarray, const string& msg, double a_double, int an_int); You could wrap it by calling add_function_wrapper(wrappermod, "printData", "example::printData", arglist = ["Array<float, 2>", "Array<long int, 1>", "string", "double", "int"], returns_value=True) This would add a function to the ext_module wrappermod called printData which simply converts everything and calls the function. Anyway, I hope this might be useful to someone! I'll welcome any feedback. --Hoyt -- +++++++++++++++++++++++++++++++++++ Hoyt Koepke UBC Department of Computer Science http://www.cs.ubc.ca/~hoytak/ hoytak(a)gmail.com +++++++++++++++++++++++++++++++++++

2 1

Re: [SciPy-user] [Sparse matrix library] csr_matrix and datatypes
by Dinesh B Vadhia April 29, 2008

April 29, 2008

For reference, the column index for each non-zero is a 4 byte integer. A int8 is an additional 1 byte (ie. total 4+1=5 bytes), an int16 is 2 bytes (4+2=6 bytes). And, how much is an int32 - another 2 bytes? -------------------------------------------------------------------------------- Date: Mon, 28 Apr 2008 11:30:42 -0500 From: "Nathan Bell" <wnbell(a)gmail.com> Subject: Re: [SciPy-user] [Sparse matrix library] csr_matrix and To: "SciPy Users List" <scipy-user(a)scipy.org> Message-ID: <d05265cb0804280930g64ade00cmf190ee60bf0ed447(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 On Mon, Apr 28, 2008 at 10:57 AM, Dinesh B Vadhia <dineshbvadhia(a)hotmail.com> wrote: > > > Okay, I see what is going on. Uhmmm! If I replace int8 with int16 or int32 > then this will increase the size of A in memory which will defeat the > purpose of the original exercise to minimize the size of A (as they are very > large). Unless there is a way to coerce the calculation A.sum(0) into a > int16 then looks like I'll have to evaluate A.sum(0) in brute force fashion > at the time of A's creation, save it and read it into the program at > execution. It's the difference between roughly 5 bytes per nonzero and 6 bytes per nonzero. The column index for each nonzero is a 4 byte integer, so there's little difference in the total memory cost (4+1 vs. 4+2). -- Nathan Bell wnbell(a)gmail.com http://graphics.cs.uiuc.edu/~wnbell/

2 1

constrained optimization
by John Hunter April 29, 2008

April 29, 2008

I need to do a N dimensional constrained optimization over a weight w vector with the constraints: * w[i] >=0 * w.sum() == 1.0 Scanning through the scipy.optimize docs, I see a number of examples where parameters can be bounded by a bracketing interval, but none where constraints can be placed on combinations of the parameters, eg the sum of them. One approach I am considering is doing a bracketed [0,1] constrained optimization over N-1 weights (assigning the last weight to be 1-sum others) and modifying my cost function to punish the optimizer when the N-1 input weights sum to more than one. Is there a better approach? Thanks, JDH

5 6

problem building svn scipy
by John Hunter April 28, 2008

April 28, 2008

With numpy r5083 and scipy r4176 I am getting the following build error building 'scipy.cluster._hierarchy_wrap' extension compiling C sources C compiler: /opt/app/g++lib6/gcc-3.4/bin/gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-I/home/titan/johnh/dev/lib/python2.4/site-packages/numpy/core/include -I/home/titan/johnh/dev/lib/python2.4/site-packages/numpy/core/include -I/opt/app/g++lib6/python-2.4/include/python2.4 -c' gcc: scipy/cluster/src/hierarchy_wrap.c scipy/cluster/src/hierarchy_wrap.c: In function `calculate_cluster_sizes_wrap': scipy/cluster/src/hierarchy_wrap.c:117: error: parse error before numeric constant scipy/cluster/src/hierarchy_wrap.c:120: error: invalid lvalue in unary `&' scipy/cluster/src/hierarchy_wrap.c:124: error: invalid type argument of `->' scipy/cluster/src/hierarchy_wrap.c: In function `cluster_maxclust_monocrit_wrap': scipy/cluster/src/hierarchy_wrap.c:251: warning: unused variable `cutoff' scipy/cluster/src/hierarchy_wrap.c: In function `pdist_euclidean_wrap': scipy/cluster/src/hierarchy_wrap.c:387: error: parse error before numeric constant johnh@flag:numpy> uname -a SunOS flag 5.10 Generic_118855-15 i86pc i386 i86pc johnh@flag:numpy> gcc --version gcc (GCC) 3.4.1 Any ideas? Thanks, JDH

3 4

Re: [SciPy-user] [Sparse matrix library] csr_matrix and column sum
by Dinesh B Vadhia April 28, 2008

April 28, 2008

I'm using the svn. There is no Traceback error but the results are incorrect based on previous results. Here is ouput from print statements that might help: > import numpy > import scipy > from scipy import sparse > colSum = scipy.asmatrix(scipy.zeros((1,J), dtype=numpy.float)) > colSum = A.sum(0) # A is a csr_matrix > for j in xrange(0, 25, 1): > print >> sys.stderr, colSum[0,j], 0 , -44 , 84 , -116 , -121 , -43 , -44 , -116 , -115 , -79 , 70 , -86 , 39 , -17 , -21 , -112 , 29 , -126 , -19 , 33 , 59 , -6 , 24 , 18 , 57 > # This is wrong because the matrix A contains positive integer 1's only. > colSum = A.todense().sum(0) > for j in xrange(0, 25, 1): > print >> sys.stderr, colSum[0,j], 768 , 724 , 1108 , 652 , 1927 , 2005 , 724 , 908 , 1421 , 1457 , 2118 , 1450 , 1575 , 3055 , 2283 , 656 , 1053 , 898 , 1517 , 1569 , 1339 , 762 , 3096 , 530 , 1081 ># This is correct and leads to the correct results but there is a large performance hit because of the .todense() Hope this helps to identify the problem. Dinesh -------------------------------------------------------------------------------- From: Nathan Bell <wnbell <at> gmail.com> Subject: Re: Sparse csr_matrix and column sum Newsgroups: gmane.comp.python.scientific.user Date: 2008-04-28 02:48:41 GMT (1 hour and 43 minutes ago) On Sun, Apr 27, 2008 at 9:32 PM, Dinesh B Vadhia <dineshbvadhia <at> hotmail.com> wrote: > > Sorry, I wasn't being imprecise as A.sum(0) didn't work and still doesn't - > I've just tried again. However, A.todense().sum(0) does work - but takes a > performance hit. My import statements are: You still haven't informed us what exactly "didn't work". Can you provide the error message you get when using A.sum(0) where A is a csr_matrix? Are you using SciPy from SVN or a previous version? -- Nathan Bell wnbell <at> gmail.com http://graphics.cs.uiuc.edu/~wnbell/

4 4

[Sparse matrix library] csr_matrix and
by Dinesh B Vadhia April 28, 2008

April 28, 2008

Okay, I see what is going on. Uhmmm! If I replace int8 with int16 or int32 then this will increase the size of A in memory which will defeat the purpose of the original exercise to minimize the size of A (as they are very large). Unless there is a way to coerce the calculation A.sum(0) into a int16 then looks like I'll have to evaluate A.sum(0) in brute force fashion at the time of A's creation, save it and read it into the program at execution. Dinesh -------------------------------------------------------------------------------- Message: 2 Date: Mon, 28 Apr 2008 00:29:29 -0500 From: "Nathan Bell" <wnbell(a)gmail.com> Subject: Re: [SciPy-user] [Sparse matrix library] csr_matrix and column sum To: "SciPy Users List" <scipy-user(a)scipy.org> Message-ID: <d05265cb0804272229q5ca927b4l7d70ff338122a230(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 On Sun, Apr 27, 2008 at 11:57 PM, Dinesh B Vadhia <dineshbvadhia(a)hotmail.com> wrote: > > 0 , -44 , 84 , -116 , -121 , -43 , -44 , -116 , -115 , -79 , 70 , -86 , 39 , > -17 , -21 , -112 , 29 , -126 , -19 , 33 , 59 , -6 , 24 , 18 , 57 > > > 768 , 724 , 1108 , 652 , 1927 , 2005 , 724 , 908 , 1421 , 1457 , 2118 , 1450 > , 1575 , 3055 , 2283 , 656 , 1053 , 898 , 1517 , 1569 , 1339 , 762 , 3096 , > 530 , 1081 > This is due to the fact that when integer arithmetic overflows (e.g. A + B is too large) the result "wraps around". The solution is to use a data type with a greater range of values (more bits). Replace your int8 data array with an int16 array and you will get the expected results (albeit using one more byte per nonzero) provided that the sums do not exceed 2^15 - 1. To be safe, you might use int32 and not worry about ranges as much. -- Nathan Bell wnbell(a)gmail.com http://graphics.cs.uiuc.edu/~wnbell/

2 1

Volterra system identification
by Georg Holzmann April 28, 2008

April 28, 2008

Hallo! Are there any libraries for volterra system identification somewhere in the scipy/numpy world ? I did not find something in the web ... Thanks for any hint, LG Georg

3 3