![](https://secure.gravatar.com/avatar/573ed3088eea742af8bcfeac90bc882d.jpg?s=120&d=mm&r=g)
Hi, indexing by slices is broken on a 64 bit machine. On a 32 bit one, all is ok. Is something missing in the INSTANTIATE_ALL macro in sparsetools.i? r. In [1]: import scipy In [2]: scipy.__version__ Out[2]: '0.7.0.dev4195' In [3]: import numpy In [4]: numpy.__version__ Out[4]: '1.1.0.dev5106' (Pdb) mtx <12160x12160 sparse matrix of type '<type 'numpy.float64'>' with 235578 stored elements in Compressed Sparse Row format> (Pdb) mtx.data array([ 0. , 0. , 0. , ..., -0.00540995, -0.00636418, 0.0202062 ]) (Pdb) mtx.data.dtype dtype('float64') (Pdb) mtx.indices array([ 0, 1, 36, ..., 12136, 12155, 12159], dtype=int32) (Pdb) mtx.indptr array([ 0, 15, 30, ..., 235543, 235560, 235578], dtype=int32) (Pdb) ir slice(8974, 12160, None) (Pdb) ic slice(8974, 12160, None) (Pdb) mtx[ir,ic] *** NotImplementedError: Wrong number of arguments for overloaded function 'get_csr_submatrix'. Possible C/C++ prototypes are: get_csr_submatrix< int,signed char >(int const,int const,int const [],int const [],signed char const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< signed char
*) get_csr_submatrix< int,unsigned char >(int const,int const,int const [],int const [],unsigned char const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< unsigned char > *) get_csr_submatrix< int,short >(int const,int const,int const [],int const [],short const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< short > *) get_csr_submatrix< int,unsigned short >(int const,int const,int const [],int const [],unsigned short const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< unsigned short > *) get_csr_submatrix< int,int >(int const,int const,int const [],int const [],int const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< int > *) get_csr_submatrix< int,unsigned int >(int const,int const,int const [],int const [],unsigned int const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< unsigned int > *) get_csr_submatrix< int,long long >(int const,int const,int const [],int const [],long long const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< long long > *) get_csr_submatrix< int,unsigned long long >(int const,int const,int const [],int const [],unsigned long long const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< unsigned long long > *) get_csr_submatrix< int,float >(int const,int const,int const [],int const [],float const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< float > *) get_csr_submatrix< int,double >(int const,int const,int const [],int const [],double const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< double > *) get_csr_submatrix< int,long double >(int const,int const,int const [],int const [],long double const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< long double *) get_csr_submatrix< int,npy_cfloat_wrapper >(int const,int const,int const [],int const [],npy_cfloat_wrapper const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< npy_cfloat_wrapper > *) get_csr_submatrix< int,npy_cdouble_wrapper >(int const,int const,int const [],int const [],npy_cdouble_wrapper const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< npy_cdouble_wrapper > *) get_csr_submatrix< int,npy_clongdouble_wrapper >(int const,int const,int const [],int const [],npy_clongdouble_wrapper const [],int const,int const,int const,int const,std::vector< int > *,std::vector< int > *,std::vector< npy_clongdouble_wrapper > *)
![](https://secure.gravatar.com/avatar/3e3b4dff04d6c945d0f990ea8c362abb.jpg?s=120&d=mm&r=g)
On Fri, May 9, 2008 at 4:46 AM, Robert Cimrman <cimrman3@ntc.zcu.cz> wrote:
Hi,
indexing by slices is broken on a 64 bit machine. On a 32 bit one, all is ok. Is something missing in the INSTANTIATE_ALL macro in sparsetools.i?
Robert, I can't reproduce your problem on my system (Ubuntu 8.04, Athlon 64): $uname -a Linux droog 2.6.24-16-generic #1 SMP Thu Apr 10 12:47:45 UTC 2008 x86_64 GNU/Linux Since your .indptr and .indices arrays are 32-bits, I doubt that the problem is with INSTANTIATE_ALL. Currently sparsetools does not instantiate functions using 64-bit indices because it doubles the compilation time and it's not likely that anyone will have sparse matrices with dimensions >= 2**31 anytime soon. OTOH I don't know what else it could be so please try the following: 1) do a completely fresh install of numpy/scipy (rm -rf build and rm -rf /site-packages/scipy ) See if the problem still exists. 2) instantiate additional "long long" versions of the sparsetools functions: First check that you have a recent SWIG: $ swig -version SWIG Version 1.3.34 Add the following after line 145 of sparsetools.i DECLARE_INDEX_TYPE( long long ) http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/sparsetools... Then, for each line that looks like: 174 %template(f_name) f_name<int,signed char>; 175 %template(f_name) f_name<int,unsigned char>; 176 %template(f_name) f_name<int,short>; ..... 187 %template(f_name) f_name<int,npy_clongdouble_wrapper>; add a long long version also, e.g. %template(f_name) f_name<long long,signed char>; http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/sparsetools... Lastly, regenerate the SWIG wrappers as described here: http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/sparsetools... This will rule out the possible backend problems. Also, can you tell me what your C compiler thinks sizeof(int) is? -- Nathan Bell wnbell@gmail.com http://graphics.cs.uiuc.edu/~wnbell/
![](https://secure.gravatar.com/avatar/573ed3088eea742af8bcfeac90bc882d.jpg?s=120&d=mm&r=g)
Hi Nathan, Nathan Bell wrote:
On Fri, May 9, 2008 at 4:46 AM, Robert Cimrman <cimrman3@ntc.zcu.cz> wrote:
Hi,
indexing by slices is broken on a 64 bit machine. On a 32 bit one, all is ok. Is something missing in the INSTANTIATE_ALL macro in sparsetools.i?
Robert, I can't reproduce your problem on my system (Ubuntu 8.04, Athlon 64):
$uname -a Linux droog 2.6.24-16-generic #1 SMP Thu Apr 10 12:47:45 UTC 2008 x86_64 GNU/Linux
$ uname -a Linux uk709n03-kme 2.6.22-gentoo-r8 #1 SMP Fri Sep 7 09:38:53 CEST 2007 x86_64 Intel(R) Xeon(R) CPU X5355 @ 2.66GHz GenuineIntel GNU/Linux
Since your .indptr and .indices arrays are 32-bits, I doubt that the problem is with INSTANTIATE_ALL. Currently sparsetools does not instantiate functions using 64-bit indices because it doubles the compilation time and it's not likely that anyone will have sparse matrices with dimensions >= 2**31 anytime soon. OTOH I don't know what else it could be so please try the following:
Yep. I do not need 64 bit indices. I have also installed swig 1.3.34 to match your version. However the problem persists and is really strange: - I have created a small test script and it works - in pdb, indexing with slices works too (Pdb) mtx[slice(2290, 2845, None),slice(2290, 2845, None)] <555x555 sparse matrix of type '<type 'numpy.float64'>' with 3313 stored elements in Compressed Sparse Row format> - but not with the 'ir', 'ic' (Pdb) ir slice(2290, 2845, None) (Pdb) mtx[ir,ic] *** NotImplementedError: Wrong number of arguments for overloaded function 'get_csr_submatrix'. Possible C/C++ prototypes are: ... So thanks for your suggestions, but I fear the bug might not be in scipy.sparse... r.
![](https://secure.gravatar.com/avatar/573ed3088eea742af8bcfeac90bc882d.jpg?s=120&d=mm&r=g)
Robert Cimrman wrote:
Hi Nathan,
Nathan Bell wrote:
On Fri, May 9, 2008 at 4:46 AM, Robert Cimrman <cimrman3@ntc.zcu.cz> wrote:
Hi,
indexing by slices is broken on a 64 bit machine. On a 32 bit one, all is ok. Is something missing in the INSTANTIATE_ALL macro in sparsetools.i?
Robert, I can't reproduce your problem on my system (Ubuntu 8.04, Athlon 64):
$uname -a Linux droog 2.6.24-16-generic #1 SMP Thu Apr 10 12:47:45 UTC 2008 x86_64 GNU/Linux
$ uname -a Linux uk709n03-kme 2.6.22-gentoo-r8 #1 SMP Fri Sep 7 09:38:53 CEST 2007 x86_64 Intel(R) Xeon(R) CPU X5355 @ 2.66GHz GenuineIntel GNU/Linux
Since your .indptr and .indices arrays are 32-bits, I doubt that the problem is with INSTANTIATE_ALL. Currently sparsetools does not instantiate functions using 64-bit indices because it doubles the compilation time and it's not likely that anyone will have sparse matrices with dimensions >= 2**31 anytime soon. OTOH I don't know what else it could be so please try the following:
Yep. I do not need 64 bit indices. I have also installed swig 1.3.34 to match your version.
However the problem persists and is really strange:
- I have created a small test script and it works
- in pdb, indexing with slices works too (Pdb) mtx[slice(2290, 2845, None),slice(2290, 2845, None)] <555x555 sparse matrix of type '<type 'numpy.float64'>' with 3313 stored elements in Compressed Sparse Row format>
- but not with the 'ir', 'ic' (Pdb) ir slice(2290, 2845, None) (Pdb) mtx[ir,ic] *** NotImplementedError: Wrong number of arguments for overloaded function 'get_csr_submatrix'. Possible C/C++ prototypes are: ...
So thanks for your suggestions, but I fear the bug might not be in scipy.sparse...
I tracked it in my code. Due to construction, the items of ir, ic slices (ir.start, ir.stop, ...) had 'numpy.int32' type. After retyping to 'int' everything works. Nathan, do you think that ensuring correct slice data deserves a ticket/fix? I cannot implement it now, but it would be IMHO fairly easy to do. cheers, r.
![](https://secure.gravatar.com/avatar/3e3b4dff04d6c945d0f990ea8c362abb.jpg?s=120&d=mm&r=g)
On Mon, May 12, 2008 at 4:34 AM, Robert Cimrman <cimrman3@ntc.zcu.cz> wrote:
I tracked it in my code. Due to construction, the items of ir, ic slices (ir.start, ir.stop, ...) had 'numpy.int32' type. After retyping to 'int' everything works.
Thanks for isolating the problem. Please try r4294: http://projects.scipy.org/scipy/scipy/changeset/4294
Nathan, do you think that ensuring correct slice data deserves a ticket/fix? I cannot implement it now, but it would be IMHO fairly easy to do.
If I had more time I'd test sparse indexing more exhaustively. The bug you found is probably not the only example of an integer-like object slipping through to sparsetools. The better approach may be to make SWIG convert integer-like singletons to C integers. -- Nathan Bell wnbell@gmail.com http://graphics.cs.uiuc.edu/~wnbell/
participants (2)
-
Nathan Bell
-
Robert Cimrman