Sparse matrices and weave.inline, f2py or Pyrex
List; Has anyone ever attempted using sparse matrices in combination with either one of the following; weave.inline, f2py or Pyrex? What is the best option? Perhaps a case of personal preference? Reason I'm asking: I've gotten the solving part of my code to run satisfactorily, but building them with Python's for loops takes a heck of a long time, too long. Regards, William
On 8/11/06, William Hunter <willemjagter@gmail.com> wrote:
List;
Has anyone ever attempted using sparse matrices in combination with either one of the following; weave.inline, f2py or Pyrex?
What is the best option? Perhaps a case of personal preference?
Reason I'm asking: I've gotten the solving part of my code to run satisfactorily, but building them with Python's for loops takes a heck of a long time, too long.
Have you tried using vector indexing with numpy first? See if you can remove some of those python for loops. -- David Grant http://www.davidgrant.ca
Yes, well, my best attempt at it. You'll notice that the sparse matrices doesn't support fancy indexing, unfortunately. (I'm hoping someone can prove me wrong :-) Here's some code you can fiddle with (also attached - sparsefiddle.py), if you want. #!/usr/bin/env python from numpy import allclose, ix_, random from scipy import sparse Alil = sparse.lil_matrix((6,6)) A = Alil.todense() edof = [1,3,0,2] arb = random.rand(4,4) # Dense matrix for r in xrange(10): for c in xrange(10): A[ix_(edof,edof)] += r*c*arb # Sparse matrices for r in xrange(10): for c in xrange(10): # Alil[ix_(edof,edof)] += r*c*arb # Unfortunately doesn't work for idxR,valR in enumerate(edof): # This works, but slowly for idxC,valC in enumerate(edof): Alil[valR,valC] += r*c*arb[idxR,idxC] print allclose(A,Alil.todense()) # Should return a 'True' # EOF Regards, William On 11/08/06, David Grant <davidgrant@gmail.com> wrote:
On 8/11/06, William Hunter <willemjagter@gmail.com> wrote:
List;
Has anyone ever attempted using sparse matrices in combination with either one of the following; weave.inline, f2py or Pyrex?
What is the best option? Perhaps a case of personal preference?
Reason I'm asking: I've gotten the solving part of my code to run satisfactorily, but building them with Python's for loops takes a heck of a long time, too long.
Have you tried using vector indexing with numpy first? See if you can remove some of those python for loops.
-- David Grant http://www.davidgrant.ca _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
On 11/08/06, David Grant <davidgrant@gmail.com> wrote:
On 8/11/06, William Hunter <willemjagter@gmail.com> wrote:
List;
Has anyone ever attempted using sparse matrices in combination with either one of the following; weave.inline, f2py or Pyrex?
What is the best option? Perhaps a case of personal preference?
Reason I'm asking: I've gotten the solving part of my code to run satisfactorily, but building them with Python's for loops takes a heck of a long time, too long.
Have you tried using vector indexing with numpy first? See if you can remove some of those python for loops.
On 12/08/2006, at 1:50 PM, William Hunter wrote:
Yes, well, my best attempt at it. You'll notice that the sparse matrices doesn't support fancy indexing, unfortunately. (I'm hoping someone can prove me wrong :-)
Here's some code you can fiddle with (also attached - sparsefiddle.py), if you want.
[code snipped]
Yeah, sparse fancy indexing is still only partially supported. I don't think it'll ever be possible to make sparse support all of NumPy's indexing tricks. But I'd like to fill out the sparse indexing code so that more NumPy-like indexing operations are supported; this would at least improve usability. We could also definitely improve performance by a constant factor by taking the loops inside the methods such as __setitem__. You could try hacking the lil_matrix.__setitem__ method to accept index sequences of the form returned by ix_ to avoid all the extra method calls. (And I'd be happy to accept a patch!) It might be possible to speed up your code somewhat by doing the loops in a compiled language, but I expect this would be a complex endeavour and the gains would be small. To get speed gains of orders of magnitude you'd need to think about the optimal sparse data type and find an algorithm that's suitable for your matrix structure. I think constructing sparse matrices efficiently is inherently a tricky business ... -- Ed
OK, so that's it then. I'll give hacking lil_matrix.__setitem__ a shot, although it would be my first attempt at hacking. I suppose looking at the matrix.__setitem__ method is a good starting point, if there is such an <object.method>? Has there ever been an attempt at using dictionaries to work with sparse matrices? I'm exposing my obvious novice status here, but it seems possible if I look at the two intro texts on Python I have. One would have to make use of an iterative method like Jacobi or SOR. Robert Cimrman ones mentioned he'd like to see Tim Davies' CSparse wrapped. Me too. He mentioned using ctypes. Would that require less work than, say SWIG. WH On 13/08/06, Ed Schofield <schofield@ftw.at> wrote:
On 11/08/06, David Grant <davidgrant@gmail.com> wrote:
On 8/11/06, William Hunter <willemjagter@gmail.com> wrote:
List;
Has anyone ever attempted using sparse matrices in combination with either one of the following; weave.inline, f2py or Pyrex?
What is the best option? Perhaps a case of personal preference?
Reason I'm asking: I've gotten the solving part of my code to run satisfactorily, but building them with Python's for loops takes a heck of a long time, too long.
Have you tried using vector indexing with numpy first? See if you can remove some of those python for loops.
On 12/08/2006, at 1:50 PM, William Hunter wrote:
Yes, well, my best attempt at it. You'll notice that the sparse matrices doesn't support fancy indexing, unfortunately. (I'm hoping someone can prove me wrong :-)
Here's some code you can fiddle with (also attached - sparsefiddle.py), if you want.
[code snipped]
Yeah, sparse fancy indexing is still only partially supported. I don't think it'll ever be possible to make sparse support all of NumPy's indexing tricks. But I'd like to fill out the sparse indexing code so that more NumPy-like indexing operations are supported; this would at least improve usability. We could also definitely improve performance by a constant factor by taking the loops inside the methods such as __setitem__. You could try hacking the lil_matrix.__setitem__ method to accept index sequences of the form returned by ix_ to avoid all the extra method calls. (And I'd be happy to accept a patch!)
It might be possible to speed up your code somewhat by doing the loops in a compiled language, but I expect this would be a complex endeavour and the gains would be small. To get speed gains of orders of magnitude you'd need to think about the optimal sparse data type and find an algorithm that's suitable for your matrix structure. I think constructing sparse matrices efficiently is inherently a tricky business ...
-- Ed
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
William Hunter wrote:
Yes, well, my best attempt at it. You'll notice that the sparse matrices doesn't support fancy indexing, unfortunately. (I'm hoping someone can prove me wrong :-) Here's some code you can fiddle with (also attached - sparsefiddle.py), if you want.
Hi William, I assume you wish to assemble a large sparse matrix from small dense matrices (finite element method?) I use for this a function written in C, wrapped by SWIG. So if you are interested... r.
HI Robert; Your assumption is correct, that's exactly how I assemble the big sparse matrix, and yes, I am very definitely interested! Without a doubt. Thanks, William On 21/08/06, Robert Cimrman <cimrman3@ntc.zcu.cz> wrote:
William Hunter wrote:
Yes, well, my best attempt at it. You'll notice that the sparse matrices doesn't support fancy indexing, unfortunately. (I'm hoping someone can prove me wrong :-) Here's some code you can fiddle with (also attached - sparsefiddle.py), if you want.
Hi William,
I assume you wish to assemble a large sparse matrix from small dense matrices (finite element method?) I use for this a function written in C, wrapped by SWIG. So if you are interested...
r. _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
participants (5)
-
Andrew Straw -
David Grant -
Ed Schofield -
Robert Cimrman -
William Hunter