Mailman 3 October 2011 - SciPy-Dev

scipy.sparse spdiags/dia_matrix additional calling mode
by Tony Stillfjord Oct. 7, 2011

Oct. 7, 2011

Dear all, Given that Python is not MATLAB, would it be possible to add an additional, in some cases more intuitive, way of calling the spdiags function? This function creates a sparse matrix given its non-zero diagonals. Currently, the first input has to be an array 'data' where every row corresponds to a diagonal - which one is specified in the next input, 'diags'. However, as the diagonals have different length, the actual values have to be zero-padded on either the left or right end to fit the array. This is not a big problem if all the elements in a diagonal are the same, for example spdiags([ones(4), -2*ones(4), ones(4)], (-1,0,1), 4, 4) works nicely as some values are just ignored. But when that is not the case, and all the values are already stored in some arrays, it does not make sense that the user should have to pad them with a number of zeros (remembering on which end to put them!) and then also construct one more temporary array to hold them. I suggest that spdiags accepts a list/tuple as data input, where every list element is supposed to be an array holding a diagonal of the correct size. That is, instead of doing something like spdiags(row_stack(r_[zeros(2), a], b, r_[c, zeros(3)]), (-2, 0, 3), n, n ) one would write spdiags([a,b,c], (-2, 0, 3), n, n) Does that make sense? What it boils down to is that you should not have to know how the matrix type is implemented to use it - right? The diagonals could just as well be stored in different ways than they are currently. When I had a look at the source code, it seems that all the work is done in the dia_matrix __init__ method, but the same argument applies there. Currently, you -can- send in a list [a,b,c] where a, b and c are arrays of different length, but the function atleast_2D() is applied to the list which turns it into an array of objects. This obviously causes problems. Kind regards, Tony Stillfjord

4 11

odepack.lib(vode.o) : error LNK2019
by Martin Dulovits Oct. 6, 2011

Oct. 6, 2011

I am trying to compile scipy for python 2.6 but with VC8 and Intel Fortran compiler for win64. I tryed Scipy-0.9.0 and the latest scipy-0.10.0b2 . I start compiling with this commands : scipy-0.10.0b2>python2.6.exe setup.py config --fcompiler=intelem config_fc scipy-0.10.0b2>python2.6.exe setup.py install I compiles some time but alway ends in the following linking errors : C:\Root\VS2005\VC\BIN\amd64\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:c:\Ro ot\Houdini\V11_1_035\python26\libs /LIBPATH:c:\Root\Houdini\V11_1_035\python26\l ibs /LIBPATH:c:\Root\Houdini\V11_1_035\python26\PC\VS8.0\win32release /LIBPATH:b uild\temp.win-amd64-2.6 /LIBPATH:c:\Root\Houdini\V11_1_035\python26\libs /LIBPAT H:c:\Root\Houdini\V11_1_035\python26\PC\VS8.0\win32release /LIBPATH:build\temp.w in-amd64-2.6 odepack.lib linpack_lite.lib mach.lib blas.lib /EXPORT:init_odepack build\temp.win-amd64-2.6\Release\scipy\integrate\_odepackmodule.obj /OUT:build\ lib.win-amd64-2.6\scipy\integrate\_odepack.pyd /IMPLIB:build\temp.win-amd64-2.6\ Release\scipy\integrate\_odepack.lib /MANIFESTFILE:build\temp.win-amd64-2.6\Rele ase\scipy\integrate\_odepack.pyd.manifest _odepackmodule.obj : warning LNK4197: export 'init_odepack' specified multiple t imes; using first specification Creating library build\temp.win-amd64-2.6\Release\scipy\integrate\_odepack.li b and object build\temp.win-amd64-2.6\Release\scipy\integrate\_odepack.exp odepack.lib(vode.o) : error LNK2019: unresolved external symbol dcopy_ reference d in function dvode_ odepack.lib(vode.o) : error LNK2019: unresolved external symbol dscal_ reference d in function dvode_ linpack_lite.lib(dgbfa.o) : error LNK2001: unresolved external symbol dscal_ linpack_lite.lib(dgefa.o) : error LNK2001: unresolved external symbol dscal_ linpack_lite.lib(dgesl.o) : error LNK2001: unresolved external symbol daxpy_ odepack.lib(vode.o) : error LNK2019: unresolved external symbol daxpy_ reference d in function dvstep_ linpack_lite.lib(dgbfa.o) : error LNK2001: unresolved external symbol daxpy_ linpack_lite.lib(dgefa.o) : error LNK2001: unresolved external symbol daxpy_ linpack_lite.lib(dgbsl.o) : error LNK2001: unresolved external symbol daxpy_ linpack_lite.lib(dgbfa.o) : error LNK2019: unresolved external symbol idamax_ re ferenced in function dgbfa_ linpack_lite.lib(dgefa.o) : error LNK2001: unresolved external symbol idamax_ linpack_lite.lib(dgbsl.o) : error LNK2019: unresolved external symbol ddot_ refe renced in function dgbsl_ linpack_lite.lib(dgesl.o) : error LNK2001: unresolved external symbol ddot_ build\lib.win-amd64-2.6\scipy\integrate\_odepack.pyd : fatal error LNK1120: 5 un resolved externals error: Command "C:\Root\VS2005\VC\BIN\amd64\link.exe /DLL /nologo /INCREMENTAL:N O /LIBPATH:c:\Root\Houdini\V11_1_035\python26\libs /LIBPATH:c:\Root\Houdini\V11_ 1_035\python26\libs /LIBPATH:c:\Root\Houdini\V11_1_035\python26\PC\VS8.0\win32re lease /LIBPATH:build\temp.win-amd64-2.6 /LIBPATH:c:\Root\Houdini\V11_1_035\pytho n26\libs /LIBPATH:c:\Root\Houdini\V11_1_035\python26\PC\VS8.0\win32release /LIBP ATH:build\temp.win-amd64-2.6 odepack.lib linpack_lite.lib mach.lib blas.lib /EXP ORT:init_odepack build\temp.win-amd64-2.6\Release\scipy\integrate\_odepackmodule .obj /OUT:build\lib.win-amd64-2.6\scipy\integrate\_odepack.pyd /IMPLIB:build\tem p.win-amd64-2.6\Release\scipy\integrate\_odepack.lib /MANIFESTFILE:build\temp.wi n-amd64-2.6\Release\scipy\integrate\_odepack.pyd.manifest" failed with exit stat us 1120 I hope you can help

1 0

Compilation problem on CentOS release 5.6
by Raphael Leplae Oct. 4, 2011

Oct. 4, 2011

Dear all, I am having troubles with the compilation of SciPy version 0.10.0b2 on CentOS 5.6. Here is the info: Kernel 2.6.18-238.9.1.el5 x86_64 Python 2.7.2 GCC 4.6.1 ATLAS 3.8.3 (rpm install with yum) NumPy 1.6.1 (compiled, tested and installed successfully) Defined in the shell: export BLAS=/usr/lib64/atlas/libcblas.so export ATLAS=/usr/lib64/atlas/libatlas.so export LAPACK=/usr/lib64/atlas/liblapack.so File site.cfg: [DEFAULT] libraries = gfortran, gfortranbegin library_dirs = /usr/local/lib:/usr/local/opt/gcc/4.6.1/lib When issuing the command: python setup.py build I get a big list of warnings and then error messages. The whole log is in attachment. After the line: gcc: build/src.linux-x86_64-2.7/scipy/fftpack/_fftpackmodule.c Error messages like this start to show up: /usr/lib/../lib64/crt1.o: In function `_start': (.text+0x20): undefined reference to `main' build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/fftpack/_fftpackmodule.o: In function `f2py_rout__fftpack_destroy_dct1_cache': /software/CC/local/src/python/Libraries/scipy-0.10.0b2/build/src.linux-x86_64-2.7/scipy/fftpack/_fftpackmodule.c:2532: undefined reference to `PyArg_ParseTupleAndK eywords' /software/CC/local/src/python/Libraries/scipy-0.10.0b2/build/src.linux-x86_64-2.7/scipy/fftpack/_fftpackmodule.c:2543: undefined reference to `PyErr_Occurred' /software/CC/local/src/python/Libraries/scipy-0.10.0b2/build/src.linux-x86_64-2.7/scipy/fftpack/_fftpackmodule.c:2553: undefined reference to `Py_BuildValue' build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/fftpack/_fftpackmodule.o: In function `f2py_rout__fftpack_destroy_dct2_cache': /software/CC/local/src/python/Libraries/scipy-0.10.0b2/build/src.linux-x86_64-2.7/scipy/fftpack/_fftpackmodule.c:2471: undefined reference to `PyArg_ParseTupleAndK eywords' Someone knows why those "undefined reference" errors are showing up? Thanks in advance for your help. Raphael -- ___________________________________________________________ Raphael Leplae, Ph.D. Operations manager Tel: +32 2 650 3727 Computing Center Fax: +32 2 650 3740 ULB/VUB Avenue A. Buyl, 91 - CP 197 1050 Brussels Belgium ___________________________________________________________

2 2

RSS feeds for pull requests
by Pauli Virtanen Oct. 3, 2011

Oct. 3, 2011

Hi, It seems that a part of the nitty-gritty development discussions has moved to the Github pull requests. Keeping tabs on these is not completely easy, since (i) AFAIK only the people listed as "repository owners" get email notifications of new PRs, and (ii) also they do not get notifications of changes unless they say something in the discussion. Here's one work-around to the situation: RSS feeds for everything that's going on in the pull requests. Github doesn't itself seem to offer this -- at least I didn't manage to find it -- but at least it was fairly simple [1] to patch it together. All chatter: http://new.scipy.org/numpy-chatter.xml http://new.scipy.org/scipy-chatter.xml Only new pull requests: http://new.scipy.org/numpy-pull.xml http://new.scipy.org/scipy-pull.xml They don't update in real time, and I'm not sure how useful these turn out to be in reality, but there it goes. [1] https://github.com/pv/github-pull-request-rss -- Pauli Virtanen

2 2

On the leastsq/curve_fit method
by Gianfranco Durin Oct. 3, 2011

Oct. 3, 2011

Dear all, I wanted to briefly show you the results of an analysis I did on the performances of the optimize.leastsq method for data fitting. I presented these results at the last Python in Physics workshop. You can download the pdf here: http://emma.inrim.it:8080/gdurin/talks. 1. The main concern is about the use of cov_x to estimate the error bar of the fitting parameters. In the docs, it is set that "this matrix must be multiplied by the residual standard deviation to get the covariance of the parameter estimates -- see curve_fits."" Unfortunately, this is not correct, or better it is only partially correct. It is correct if there are no error bars of the input data (the sigma of curve_fit is None). But if provided, they are used as "as weights in least-squares problem" (curve_fit doc), and cov_x gives directly the covariance of the parameter estimates (i.e. the diagonal terms are the errors in the parameters). See for instance here: http://www.gnu.org/s/gsl/manual/html_node/Computing-the-covariance-matrix-o…. This means that not only the doc needs fixing, but also the curve_fit code, those estimation of the parameters' error is INDEPENDENT of the values of the data errors in the case they are constant, which is clearly wrong. I have never provided a patch, but the fix should be quite simple, just please give me indication on how to do that. 2. The convergence of the fit in the most difficult cases (see page 15 of my presentation) can required up to about 3000 iterations, reduced to 800 when using analytical derivatives. For quite a long time I did not realized that the fit needed more iterations that the number set by maxfev, and thus I started to think that the leastsq was not good enough for 'hard' data. As a matter of fact, maxfev : int The maximum number of calls to the function. If zero, then 100*(N+1) is the maximum where N is the number of elements in x0. is pretty low, so I suggest to increase the prefactor 100 to 1000. A relatively huge number is not a problem, by the way, because if the system is sloppy. i.e. one parameter does not move too much, the routine stops and complains with "Both actual and predicted relative reductions in the sum of squares are at most 0.000000 and the relative error between two consecutive iterates is at most 0.000000" as in the case of boxBod at pg. 15. By the way, we should also advice that the in case of analytical derivative this number is half, even if I personally would keep the same number for both cases. This is all for the moment. Many thanks for your attention, and sorry for the long mail Gianfranco

5 11

Re: [SciPy-Dev] scipy.sparse vs. pysparse
by Pauli Virtanen Oct. 3, 2011

Oct. 3, 2011

Hi, Here is some optimization reducing the runtime overhead of scipy.sparse matrix-vector multiplication by a factor of 5. https://github.com/pv/scipy-work/compare/master...enh/sparse-speedup And a patch against Scipy 0.9.0 (@Tony: maybe you want to try it out?): https://github.com/pv/scipy-work/compare/v0.9.0...enh/sparse-speedup-0.9.pa… *** Quick benchmark: http://dl.dropbox.com/u/5453551/bench_sparse.py (Multiply vector with 1-D CSR Laplacian operator.) After: % N scipy.sparse pysparse 32 7.169e-06 1.048e-06 64 7.367e-06 1.787e-06 128 7.814e-06 3.284e-06 256 8.633e-06 6.336e-06 512 1.025e-05 1.241e-05 1024 1.435e-05 2.455e-05 2048 1.989e-05 4.882e-05 4096 3.384e-05 9.798e-05 8192 6.098e-05 0.0001959 Before: % N scipy.sparse pysparse 32 3.708e-05 1.032e-06 64 3.736e-05 1.803e-06 128 3.777e-05 3.368e-06 256 3.95e-05 6.324e-06 512 4.116e-05 1.267e-05 1024 4.661e-05 2.455e-05 2048 5.38e-05 4.873e-05 4096 6.946e-05 9.763e-05 8192 9.563e-05 0.0001959 The cross-over occurs around N ~ 300 instead of around N ~ 3000. The main reason for the overhead is that multiplication with a sparse Laplacian is a pretty lightweight operation, so the fact that some of scipy.sparse is written in pure Python starts to matter. -- Pauli Virtanen

2 1