amd64 problems, scipy 0.6.0 patch
Hi, To make sfepy work with the latest released scipy, use this patch: diff --git a/sfe/solvers/ls.py b/sfe/solvers/ls.py --- a/sfe/solvers/ls.py +++ b/sfe/solvers/ls.py @@ -1,7 +1,7 @@ from sfe.base.base import * from sfe.base.base import * from sfe.solvers.solvers import LinearSolver -import scipy.splinalg.dsolve.umfpack as um +import scipy.linsolve.umfpack as um um.configure( assumeSortedIndices = True ) ## However, today I discovered problems when using sfepy on amd64 (Debian): It compiles fine, but: $ ./schroedinger.py input/schroed.py warning: other missing: ['functions', 'modules', 'epbc', 'lcbc'] warning: left over: ['funV', 'options'] mesh.read = 0.04 mesh.split = 0.0 t = 0.0 Traceback (most recent call last): File "./schroedinger.py", line 110, in ? main() File "./schroedinger.py", line 107, in main evp = solveEigenProblem( conf, options ) File "./schroedinger.py", line 24, in solveEigenProblem pb = ProblemDefinition.fromConf( conf ) File "/home/ondra/sfepy/sfe/fem/problemDef.py", line 40, in fromConf domain.setupNeighbourLists() File "/home/ondra/sfepy/sfe/fem/domain.py", line 349, in setupNeighbourLists mu.sortRows( aux, nm.array( sortCols[mode], nm.int32 ) ) File "/home/ondra/sfepy/sfe/fem/extmods/meshutils.py", line 102, in sortRows return _meshutils.sortRows(*args) TypeError: array cannot be safely cast to required type This is imho related to the fact, that the 64bit system uses integers of a different default length. Ondrej
Ondrej Certik wrote:
However, today I discovered problems when using sfepy on amd64 (Debian):
It compiles fine, but:
$ ./schroedinger.py input/schroed.py warning: other missing: ['functions', 'modules', 'epbc', 'lcbc'] warning: left over: ['funV', 'options'] mesh.read = 0.04 mesh.split = 0.0 t = 0.0 Traceback (most recent call last): File "./schroedinger.py", line 110, in ? main() File "./schroedinger.py", line 107, in main evp = solveEigenProblem( conf, options ) File "./schroedinger.py", line 24, in solveEigenProblem pb = ProblemDefinition.fromConf( conf ) File "/home/ondra/sfepy/sfe/fem/problemDef.py", line 40, in fromConf domain.setupNeighbourLists() File "/home/ondra/sfepy/sfe/fem/domain.py", line 349, in setupNeighbourLists mu.sortRows( aux, nm.array( sortCols[mode], nm.int32 ) ) File "/home/ondra/sfepy/sfe/fem/extmods/meshutils.py", line 102, in sortRows return _meshutils.sortRows(*args) TypeError: array cannot be safely cast to required type
This is imho related to the fact, that the 64bit system uses integers of a different default length.
Yes, I have a 32-bit system only, so I did not pay much attention to this.
I will try to fix it, but having access to a 64-bit machine would be helpful - is it possible for you to make me some guest account?
thanks for trying sfepy! :) r.
This is imho related to the fact, that the 64bit system uses integers of a different default length.
Yes, I have a 32-bit system only, so I did not pay much attention to this.
I will try to fix it, but having access to a 64-bit machine would be helpful - is it possible for you to make me some guest account?
I created the account offlist. I am interested in getting sfepy working on all architectures, first I want to try my faster machine with 4 processors, we can also try some parallel solvers in there. And second, later I'd like to get into Debian, when we make it more robust and user friendly and documented. So just having the aim of passing quite strict Debian policies will improve the overall quality of the software.
thanks for trying sfepy! :)
Thanks for developing sfepy! :)
O.
I created the account offlist. I am interested in getting sfepy working on all architectures, first I want to try my faster machine with 4 processors, we can also try some parallel solvers in there. And second, later I'd like to get into Debian, when we make it more robust and user friendly and documented. So just having the aim of passing quite strict Debian policies will improve the overall quality of the software.
Yeah, it is good to set high aims.
I am now struggling with a weird thing, trying the code on your 64bit machine:
in C, I have the following declaration of function parameters (rawGraph() in fem.c): .., int32 nRow, int32 nCol, int32 nGr, .. where typedef int int32; is in sfe/fem/extmods/types.h
A strange thing is, that with shape = nm.array( shape, dtype = nm.int64 ) the call to the swigged C function works, but with nm.int32 does not! This is strange.
I have swig 1.3.31, the 64bit machine has 1.3.33, if it is of any relevance.
r.
On Feb 4, 2008 3:29 PM, Robert Cimrman cimr...@ntc.zcu.cz wrote:
I created the account offlist. I am interested in getting sfepy working on all architectures, first I want to try my faster machine with 4 processors, we can also try some parallel solvers in there. And second, later I'd like to get into Debian, when we make it more robust and user friendly and documented. So just having the aim of passing quite strict Debian policies will improve the overall quality of the software.
Yeah, it is good to set high aims.
I am now struggling with a weird thing, trying the code on your 64bit machine:
in C, I have the following declaration of function parameters (rawGraph() in fem.c): .., int32 nRow, int32 nCol, int32 nGr, .. where typedef int int32; is in sfe/fem/extmods/types.h
As I understand it, "int" is 64bit on Quad Core, so int32 is in fact 64bit here, if you use this declaration, isn't it?
A strange thing is, that with shape = nm.array( shape, dtype = nm.int64 ) the call to the swigged C function works, but with nm.int32 does not! This is strange.
Well, int32 above is 64bit as I understand it. So nm.int64 matches it, so it works. nm.int32 doesn't match it, so it doesn't work. No surprise for me.
Why not to use numpy types, instead of doing things on our own (=wrong:)?
Ondrej
On Feb 4, 2008 3:41 PM, Ondrej Certik ond...@certik.cz wrote:
On Feb 4, 2008 3:29 PM, Robert Cimrman cimr...@ntc.zcu.cz wrote:
I created the account offlist. I am interested in getting sfepy working on all architectures, first I want to try my faster machine with 4 processors, we can also try some parallel solvers in there. And second, later I'd like to get into Debian, when we make it more robust and user friendly and documented. So just having the aim of passing quite strict Debian policies will improve the overall quality of the software.
Yeah, it is good to set high aims.
I am now struggling with a weird thing, trying the code on your 64bit machine:
in C, I have the following declaration of function parameters (rawGraph() in fem.c): .., int32 nRow, int32 nCol, int32 nGr, .. where typedef int int32; is in sfe/fem/extmods/types.h
As I understand it, "int" is 64bit on Quad Core, so int32 is in fact 64bit here, if you use this declaration, isn't it?
A strange thing is, that with shape = nm.array( shape, dtype = nm.int64 ) the call to the swigged C function works, but with nm.int32 does not! This is strange.
Well, int32 above is 64bit as I understand it. So nm.int64 matches it, so it works. nm.int32 doesn't match it, so it doesn't work. No surprise for me.
Why not to use numpy types, instead of doing things on our own (=wrong:)?
Let's use the types from:
/usr/include/numpy/ndarrayobject.h
(the exact location of this file may vary on your system), search for the declaration of "npy_int32" and you will see how it depends on the "SIZEOF_LONG".
Ondrej
Ondrej Certik wrote:
As I understand it, "int" is 64bit on Quad Core, so int32 is in fact 64bit here, if you use this declaration, isn't it?
well, no. sizeof( int ) and sizeof( int32 ) print 4, sizeof( long ) prints 8... this is what is weird.
Why not to use numpy types, instead of doing things on our own (=wrong:)?
We just need a 4-byte integer, and 8-byte float.
r.
Ondrej Certik wrote:
(the exact location of this file may vary on your system), search for the declaration of "npy_int32" and you will see how it depends on the "SIZEOF_LONG".
Well, we can well uses this, but printf( "______%d\n", sizeof( npy_int32 ) ); prints ______4
so I still do not know why swig complains.
r.
On Feb 4, 2008 3:48 PM, Robert Cimrman cimr...@ntc.zcu.cz wrote:
Ondrej Certik wrote:
As I understand it, "int" is 64bit on Quad Core, so int32 is in fact 64bit here, if you use this declaration, isn't it?
well, no. sizeof( int ) and sizeof( int32 ) print 4, sizeof( long ) prints 8... this is what is weird.
Just for the record, this is the result on Quad Core:
$ cat sizeof.c /*
- 11.09.2000, c */ #include "stdio.h" #include "numpy/arrayobject.h"
typedef int int32;
int main( int argc, char **argv ) { printf( "%d %d %d %d %d\n", sizeof( short ), sizeof( int ), sizeof( long ), sizeof( float ), sizeof( double ) ); printf( "%d %d %d %d %d\n", sizeof( npy_int32 ), sizeof( int32 ), sizeof( long ), sizeof( float ), sizeof( double ) );
return( 0 ); } $ ./sizeof 2 4 8 4 8 2 4 8 4 8
Why not to use numpy types, instead of doing things on our own (=wrong:)?
We just need a 4-byte integer, and 8-byte float.
Well, we can well uses this, but printf( "______%d\n", sizeof( npy_int32 ) ); prints ______4
I thought the problem was in getting int32 to be 4 bytes. So the problem is somewhere else then. We need to dig into the swig wrappers.
Have you ever considered using Cython? It produces much nicer, shorter, faster and easier to debug wrappers.
Ondrej
Robert Cimrman wrote:
Ondrej Certik wrote:
(the exact location of this file may vary on your system), search for the declaration of "npy_int32" and you will see how it depends on the "SIZEOF_LONG".
Well, we can well uses this, but printf( "______%d\n", sizeof( npy_int32 ) ); prints ______4
so I still do not know why swig complains.
Now I am on some track:
out of 'int32 nRow, int32 nCol, int32 nGr', nRow, nCol are of type numpy.int32 (which causes problems), nGr is a regular python long (no problem, automatically converted to my C int32 (=int)). it seems that swig cannot convert numpy.int32 to the C int32 (which is the same), while it _can_ convert numpy.int64 to C int32...
It is easy to fix in my code, but I would like to understand this.
r.
On Feb 4, 2008 4:39 PM, Robert Cimrman
Robert Cimrman wrote:
Ondrej Certik wrote:
(the exact location of this file may vary on your system), search for the declaration of "npy_int32" and you will see how it depends on the "SIZEOF_LONG".
Well, we can well uses this, but printf( "______%d\n", sizeof( npy_int32 ) ); prints ______4
so I still do not know why swig complains.
Now I am on some track:
out of 'int32 nRow, int32 nCol, int32 nGr', nRow, nCol are of type numpy.int32 (which causes problems), nGr is a regular python long (no problem, automatically converted to my C int32 (=int)). it seems that swig cannot convert numpy.int32 to the C int32 (which is the same), while it _can_ convert numpy.int64 to C int32...
It is easy to fix in my code, but I would like to understand this.
I don't. But I would like to understand this: When running the code I posted above from your compiled sizeof binary, I got the results above. However, I decided to better check it myself, so I tried to compile the sizeof.c (see the code above) and it failed... I had to apply the following patch: $ diff -Naur /home/robert/software/sizeof.c sizeof.c --- /home/robert/software/sizeof.c 2008-02-04 16:04:42.143594058 +0100 +++ sizeof.c 2008-02-04 16:50:28.177622129 +0100 @@ -2,6 +2,7 @@ * 11.09.2000, c */ #include "stdio.h" +#include "Python.h" #include "numpy/arrayobject.h" typedef int int32; Now I can compile it with: $ cat Makefile all: gcc -I/usr/include/python2.4/ -I/usr/include/numpy -c -o sizeof.o sizeof.c gcc sizeof.o -o sizeof -lpython2.4 $ make And now I got a different result (!): $ ./sizeof 2 4 8 4 8 4 4 8 4 8 Maybe you could explain it? :) And here is the result on 32 bits: $ ./sizeof 2 4 4 4 8 4 4 4 4 8 Ondrej
Ondrej Certik wrote:
Just for the record, this is the result on Quad Core:
$ cat sizeof.c /*
- 11.09.2000, c */ #include "stdio.h" #include "numpy/arrayobject.h"
typedef int int32;
int main( int argc, char **argv ) { printf( "%d %d %d %d %d\n", sizeof( short ), sizeof( int ), sizeof( long ), sizeof( float ), sizeof( double ) ); printf( "%d %d %d %d %d\n", sizeof( npy_int32 ), sizeof( int32 ), sizeof( long ), sizeof( float ), sizeof( double ) );
return( 0 ); } $ ./sizeof 2 4 8 4 8 2 4 8 4 8
note that the second line is wrong, the first number (2) is still 'short' - I did not compile the sources, as there were some linking problems when replacing 'short' by 'npy_int32'. -> ignore the second line (it does not change the message...)
Why not to use numpy types, instead of doing things on our own (=wrong:)? We just need a 4-byte integer, and 8-byte float.
Well, we can well uses this, but printf( "______%d\n", sizeof( npy_int32 ) ); prints ______4
I thought the problem was in getting int32 to be 4 bytes. So the problem is somewhere else then. We need to dig into the swig wrappers.
Have you ever considered using Cython? It produces much nicer, shorter, faster and easier to debug wrappers.
I am mildly aware of Cython, but have not tried it yet. It may be a way to go, though. The 32/64 bit issues are well addressed there? What about the wrapper overhead (shorter means also faster?) Have you some examples? Will my C code work with it, or should I rewrite all into its language?
Anyway, things may change if we get to the point of generating parts of the 'C' code with symfe.
r.
Ondrej Certik wrote:
Now I can compile it with:
$ cat Makefile all: gcc -I/usr/include/python2.4/ -I/usr/include/numpy -c -o sizeof.o sizeof.c gcc sizeof.o -o sizeof -lpython2.4 $ make
And now I got a different result (!):
$ ./sizeof 2 4 8 4 8 4 4 8 4 8
Maybe you could explain it? :)
And here is the result on 32 bits:
$ ./sizeof 2 4 4 4 8 4 4 4 4 8
This are the correct answers. My previous post should clarify it.
r.
note that the second line is wrong, the first number (2) is still 'short' - I did not compile the sources, as there were some linking problems when replacing 'short' by 'npy_int32'.
You need to apply the patch, as I explained.
I am mildly aware of Cython, but have not tried it yet. It may be a way to go, though. The 32/64 bit issues are well addressed there? What about
The wrapper is so simple, that it shouldn't be a problem to investigate the problems.
the wrapper overhead (shorter means also faster?) Have you some
Yes. Swig uses a special python wrapper around it's autogenerated C wrappers.
examples?
Many. See e.g. the code of Sage for a real life usage. You can also browse the example I posted to scipy-dev. Cython is like Pyrex with some more userfriendly things, so you can also look at the pyrex example in the numpy tarball. cython.org also contains a lot of examples.
Will my C code work with it,
Yes.
or should I rewrite all into its language?
No. Only the swig *.i files will be rewritten in the Cython language.
Anyway, things may change if we get to the point of generating parts of the 'C' code with symfe.
I suggest to fix the problem in swig for now and be done with it. Cython is just an idea for a future work.
Ondrej
Ondrej Certik wrote:
You need to apply the patch, as I explained.
Thanks. You were faster :)
I am mildly aware of Cython, but have not tried it yet. It may be a way to go, though. The 32/64 bit issues are well addressed there? What about
The wrapper is so simple, that it shouldn't be a problem to investigate the problems.
the wrapper overhead (shorter means also faster?) Have you some
Yes. Swig uses a special python wrapper around it's autogenerated C wrappers.
Great!
Will my C code work with it,
Yes.
or should I rewrite all into its language?
No. Only the swig *.i files will be rewritten in the Cython language.
Hey, this sounds good!
Anyway, things may change if we get to the point of generating parts of the 'C' code with symfe.
I suggest to fix the problem in swig for now and be done with it. Cython is just an idea for a future work.
Sure. It is fixed on the test account - all tests pass except those relying on SVN scipy and pytables, which are not installed in the system.
robert@august:~/software/sfepy$ ./schroedinger.py input/schroed.py ... computing resonance frequencies... done in 1.10 s [ 56.58422132+0.j 120.86053155+0.j 120.94379536+0.j ..., 5449.54444587+0.j 5635.20027136+0.j 5856.39070200+0.j]
is the primary hg repository ready?
r.
Sure. It is fixed on the test account - all tests pass except those relying on SVN scipy and pytables, which are not installed in the system.
robert@august:~/software/sfepy$ ./schroedinger.py input/schroed.py ... computing resonance frequencies... done in 1.10 s [ 56.58422132+0.j 120.86053155+0.j 120.94379536+0.j ..., 5449.54444587+0.j 5635.20027136+0.j 5856.39070200+0.j]
is the primary hg repository ready?
Yes. The changes were committed here:
http://hg.sympy.org/sfepy/rev/a0d6d48b4229
Ondrej
participants (2)
-
Ondrej Certik
-
Robert Cimrman