Mailman 3 December 2004 - SciPy-Dev

Re: Fundamental scipy testing changes
by Todd Miller 11 Jan '05

11 Jan '05

The discussion below relates to porting scipy to numarray, starting with some small changes to scipy_test.testing. The discussion began as private e-mail but has crossed over to scipy-dev at Pearu's suggestion. On Tue, 2004-12-07 at 16:16, Pearu Peterson wrote: > Hi Todd, > > On 7 Dec 2004, Todd Miller wrote: > > > I took a closer look at how my scipy_base changes affect scipy.test() > > and found a couple things in my scipy_base changes that needed fixing. > > Sorry I didn't think of this sooner. These fixes are committed now on > > the numarray branch in CVS. I also honed in on what I think should be > > the starting point for discussion: the changes I made to > > scipy_test/testing.py. > > Note that assert_equal, assert_almost_equal, assert_approx_equal > were not meant to be used with array arguments (I didn't implement them > but its obvious from reading the code). Thanks for pointing this out... I noticed the array versions only peripherally and didn't understand the distinction. > For checking the equality of > array arguments assert_array_equal or assert_array_almost_equal should be > used. If some scipy test suite uses assert_equal, etc with array > arguments then I think this is a bug of this particular test suite, > not of testing.py. So, using scipy_base.all in assert_equal, etc is not > necessary (unless we want to drop assert_array_* functions). Understood. Are we agreed that it is appropriate to use all() in the assert_array_* functions? > > In order to do the numarray scipy_base changes and pass > > scipy_base.test(), I modified scipy_test/testing.py in ways which I > > assert are good. Mostly, I used all() in contexts where an array result > > was being used as a truth value. In those contexts, the array > > __nonzero__() function is executed. NumArray.__nonzero__() raises an > > exception causing many tests to fail that should succeed. Using all() > > reduces the "truth array" to a single scalar value, so > > NumArray.__nonzero__() is never called. > > > > Where I think early agreement needs to happen is that my testing.py > > changes are good even though they expose a handful of latent scipy bugs > > or unit test bugs because Numeric's __nonzero__() has the meaning of > > any() and not all(). So, for say an equality test, if any of the > > values were equal, the test would succeed even if most of the arrays' > > values were *not* equal. Switching to all() as I am advocating exposes > > hidden problems like these. > > At the moment I have no comments on this as I haven't tried the numarray > branch of scipy yet. I think we should square away the scipy_test.testing changes before anyone messes with the numarray branch. > Btw, your discussion about the changes to scipy > sounds reasonable. However, why do you say that using alltrue is an error? > I thought all(x) is equivalent to alltrue(ravel(x)) and in assert_array_* > functions the arguments to alltrue are already ravel'ed. I looked at my diffs more carefully and see that you're right, the alltrues are not bugs because they're only fed 1D arrays. I have not reexamined all my scipy_base changes for instances of alltrue. > > If my testing.py changes are agreed upon and the exposed bugs are either > > fixed or acknowledged as known, we have a better basis for examining > > the rest of the numarray changes to scipy_base. > > > > I attached 3 files for you to look at: > > > > 1. testing.diffs (changes to -r HEAD of scipy_test/testing.py) > > > > 2. results.HEAD (scipy.test() results against the HEAD of CVS using > > Numeric) > > I got 4 failures and can't see the nightly scoreboard yet so I don't > > know if these are expected. > > > > 3. results.testing.changes (scipy.test() results with testing.diffs > > applied) > > I got 11 failures (including the original 4) most of which I believe are > > either bugs or unit test bugs. > > The failures, where you see almost equal arrays, may occur when using > Fortran blas; My scipy.test() does complain about not finding clapack so maybe I am using a Fortran blas too. > these errors should be fixed by using proper decimal > argument in assert_array_almost_equal call of the corresponding tests (I > thought I have fixed these errors in scipy HEAD already). As you said I think, the (incorrect) changes to assert_equal exposed (incorrect) uses of assert_equal in array contexts. > Other failures need more attention. > > > If any of you would rather not be included in my future numarray > > mailings, or you have suggestions for a better venue for them, please > > let me know. > > I suggest using scipy-dev. > > Thanks, > Pearu I re-attached the attachments in case anyone on scipy-dev wants to look as well; there's nothing new there. Thanks for looking it over, Todd

5 7

RE: [SciPy-dev] CVS scipy.test() segfaulting on me on a Fedora2 box
by Charles Harris 10 Jan '05

10 Jan '05

Ok, I've got the reorganized version from cvs and still don't get a segfault using the distro BLAS and gcc 3.4.2. -----Original Message----- From: scipy-dev-bounces(a)scipy.net on behalf of Fernando Perez Sent: Wed 12/22/2004 12:07 AM To: SciPy Developers List Subject: Re: [SciPy-dev] CVS scipy.test() segfaulting on me on a Fedora2 box Charles Harris wrote: > Hi Fernando, > > Am I missing something? I don't get scipy.lib when I install from cvs, so I > am wondering if something has changed and I need to checkout scipy again? I did a fresh checkout, b/c I recalled hearing about a discussion on a directory reorg. Since CVS sucks so badly with layout operations, I played it safe and did a clean checkout. > Anyway, scipy.linalg.lapack.flapack.cheev(a) runs fine here on Fedora Core > 3, but I am using the blas libraries that came with the distro. Is Atlas to > be preferred to blas? ATLAS provides a tuned BLAS, optimized for your specific architecture. The functionality is the same as that in Fedora's generic BLAS, but the performance is higher. Cheers, f _______________________________________________ Scipy-dev mailing list Scipy-dev(a)scipy.net http://www.scipy.net/mailman/listinfo/scipy-dev

5 7

Patch: Tiny bugfixes
by Norbert Nemec 31 Dec '04

31 Dec '04

Hi there, see attached a patch with three miniscule bugfixes for the current CVS version The two fixes about real and imag are intended to make real(3.0) return 3.0 and not array([3.0]) Greetings, Norbert -- _________________________________________Norbert Nemec Bernhardstr. 2 ... D-93053 Regensburg Tel: 0941 - 2009638 ... Mobil: 0179 - 7475199 eMail: <Norbert(a)Nemec-online.de> -- _________________________________________Norbert Nemec Bernhardstr. 2 ... D-93053 Regensburg Tel: 0941 - 2009638 ... Mobil: 0179 - 7475199 eMail: <Norbert(a)Nemec-online.de>

1 0

CVS scipy.test() segfaulting on me on a Fedora2 box
by Fernando Perez 22 Dec '04

22 Dec '04

Hi all, I just updated to current CVS of f2py/scipy and Numeric 23.6, to try to root out some problems I was encountering with scipy.test(). Now I'm having a different problem, scipy is segfaulting on me :) I've narrowed it down to this: planck[pylab]> python -c 'import scipy;scipy.lib.lapack.test(verbosity=2)' !! No test file 'test_flapack.py' found for <module 'scipy.lib.lapack.flapack' from '...es/scipy/lib/lapack/flapack.so'> !! No test file 'test_clapack.py' found for <module 'scipy.lib.lapack.clapack' from '...es/scipy/lib/lapack/clapack.so'> Found 78 tests for scipy.lib.lapack !! No test file 'test_calc_lwork.py' found for <module 'scipy.lib.lapack.calc_lwork' from '...scipy/lib/lapack/calc_lwork.so'> Found 0 tests for __main__ check_gebal (scipy.lib.lapack.test_lapack.test_flapack_complex) ... ok check_heev (scipy.lib.lapack.test_lapack.test_flapack_complex)Segmentation fault If I go manually into test_lapack.py and comment out around line 93: #class test_flapack_complex(_test_lapack): # lapack = PrefixWrapper(flapack,'c') # decimal = 5 The whole suite runs fine: planck[pylab]> python -c 'import scipy;scipy.test(level=10,verbosity=2)' [...] check_basic (scipy.signal.signaltools.test_signaltools.test_wiener) ... ok ---------------------------------------------------------------------- Ran 1166 tests in 83.200s OK By digging a bit deeper into the testing code, I was able to track the segfault to this (a defined here is copied from the test code, the matrix so defined is correctly hermitian as required by cheev, so this shouldn't be a problem): In [1]: a = [[1,2,3],[2,2,3],[3,3,6]] In [2]: import scipy In [3]: scipy.lib.lapack.flapack.cheev(a) Segmentation fault The coredump is not terribly informative (unfortunately I don't have a debug build of python around to produce more details with): #0 0x55cd67c2 in ATL_cdotc_xp0yp0aXbX () from /usr/lib/python2.3/site-packages/scipy/lib/lapack/flapack.so #1 0xfee241b8 in ?? () #2 0x09bfb880 in ?? () #3 0xfee241f8 in ?? () #4 0x55cc4228 in cdotc_ () from /usr/lib/python2.3/site-packages/scipy/lib/lapack/flapack.so #5 0x4083e1dc in ?? () #6 0x560f1358 in ?? () from /usr/lib/python2.3/site-packages/scipy/lib/lapack/flapack.so #7 0x3ed413ce in ?? () #8 0x560f1358 in ?? () from /usr/lib/python2.3/site-packages/scipy/lib/lapack/flapack.so #9 0x3ed413ce in ?? () #10 0xfee24100 in ?? () #11 0x00000000 in ?? () Other calls to complex routines also appear to segfault, though I didn't test too many more. I wonder if anyone knows what may be going on here. This machine has in the past run successfully all scipy tests, and I did not change my ATLAS installation. For reference, here are all my version numbers (I just pulled scipy and f2py from CVS minutes ago, and grabbed Numpy from Sourceforge): In [9]: print sys.version 2.3.3 (#1, May 7 2004, 10:31:40) [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] In [10]: Numeric.__version__ Out[10]: '23.6' In [11]: f2py2e.__version__.version Out[11]: '2.44.240_1906' In [12]: scipy.__version__ Out[12]: '0.3.2_300.4521' I installed scipy with a straight 'setup.py install', without touching any of the config files (as I've done successfully in the past). Thanks for any help. I'll gladly provide more info if needed, f

3 5

RE: [SciPy-dev] CVS scipy.test() segfaulting on me on a Fedora2 box
by Charles Harris 22 Dec '04

22 Dec '04

Hi Fernando, Am I missing something? I don't get scipy.lib when I install from cvs, so I am wondering if something has changed and I need to checkout scipy again? Anyway, scipy.linalg.lapack.flapack.cheev(a) runs fine here on Fedora Core 3, but I am using the blas libraries that came with the distro. Is Atlas to be preferred to blas? As to optimization, I usually find -O2 runs faster than -O3 and that -mtune can make a big difference. -----Original Message----- From: scipy-dev-bounces(a)scipy.net on behalf of Fernando Perez Sent: Tue 12/21/2004 6:26 PM To: scipy Subject: [SciPy-dev] CVS scipy.test() segfaulting on me on a Fedora2 box Hi all, I just updated to current CVS of f2py/scipy and Numeric 23.6, to try to root out some problems I was encountering with scipy.test(). Now I'm having a different problem, scipy is segfaulting on me :) I've narrowed it down to this: planck[pylab]> python -c 'import scipy;scipy.lib.lapack.test(verbosity=2)' !! No test file 'test_flapack.py' found for <module 'scipy.lib.lapack.flapack' from '...es/scipy/lib/lapack/flapack.so'> !! No test file 'test_clapack.py' found for <module 'scipy.lib.lapack.clapack' from '...es/scipy/lib/lapack/clapack.so'> Found 78 tests for scipy.lib.lapack !! No test file 'test_calc_lwork.py' found for <module 'scipy.lib.lapack.calc_lwork' from '...scipy/lib/lapack/calc_lwork.so'> Found 0 tests for __main__ check_gebal (scipy.lib.lapack.test_lapack.test_flapack_complex) ... ok check_heev (scipy.lib.lapack.test_lapack.test_flapack_complex)Segmentation fault If I go manually into test_lapack.py and comment out around line 93: #class test_flapack_complex(_test_lapack): # lapack = PrefixWrapper(flapack,'c') # decimal = 5 The whole suite runs fine: planck[pylab]> python -c 'import scipy;scipy.test(level=10,verbosity=2)' [...] check_basic (scipy.signal.signaltools.test_signaltools.test_wiener) ... ok ---------------------------------------------------------------------- Ran 1166 tests in 83.200s OK By digging a bit deeper into the testing code, I was able to track the segfault to this (a defined here is copied from the test code, the matrix so defined is correctly hermitian as required by cheev, so this shouldn't be a problem): In [1]: a = [[1,2,3],[2,2,3],[3,3,6]] In [2]: import scipy In [3]: scipy.lib.lapack.flapack.cheev(a) Segmentation fault The coredump is not terribly informative (unfortunately I don't have a debug build of python around to produce more details with): #0 0x55cd67c2 in ATL_cdotc_xp0yp0aXbX () from /usr/lib/python2.3/site-packages/scipy/lib/lapack/flapack.so #1 0xfee241b8 in ?? () #2 0x09bfb880 in ?? () #3 0xfee241f8 in ?? () #4 0x55cc4228 in cdotc_ () from /usr/lib/python2.3/site-packages/scipy/lib/lapack/flapack.so #5 0x4083e1dc in ?? () #6 0x560f1358 in ?? () from /usr/lib/python2.3/site-packages/scipy/lib/lapack/flapack.so #7 0x3ed413ce in ?? () #8 0x560f1358 in ?? () from /usr/lib/python2.3/site-packages/scipy/lib/lapack/flapack.so #9 0x3ed413ce in ?? () #10 0xfee24100 in ?? () #11 0x00000000 in ?? () Other calls to complex routines also appear to segfault, though I didn't test too many more. I wonder if anyone knows what may be going on here. This machine has in the past run successfully all scipy tests, and I did not change my ATLAS installation. For reference, here are all my version numbers (I just pulled scipy and f2py from CVS minutes ago, and grabbed Numpy from Sourceforge): In [9]: print sys.version 2.3.3 (#1, May 7 2004, 10:31:40) [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] In [10]: Numeric.__version__ Out[10]: '23.6' In [11]: f2py2e.__version__.version Out[11]: '2.44.240_1906' In [12]: scipy.__version__ Out[12]: '0.3.2_300.4521' I installed scipy with a straight 'setup.py install', without touching any of the config files (as I've done successfully in the past). Thanks for any help. I'll gladly provide more info if needed, f _______________________________________________ Scipy-dev mailing list Scipy-dev(a)scipy.net http://www.scipy.net/mailman/listinfo/scipy-dev

3 2

[Fwd: [Numpy-discussion] started work on new version of LAPACK]
by Todd Miller 14 Dec '04

14 Dec '04

The LAPACK/ScaLAPACK team is soliciting input for a new version. This was originally posted to numpy-discussion but I thought I should pass it on to SciPy. Regards, Todd

1 0

Re: Two fortran/f2py questions
by Tom Loredo 13 Dec '04

13 Dec '04

Hi Pearu- Thanks a lot for the quick response! Regarding ignoring wrapping some utility subprograms via a directive: > But I think it would be a useful feature. So I'll add this request to > my todo list. Thanks! > Btw, if you would use scipy_distutils-style setup.py file > then you can add the "only: .. :" part to f2py_options I thought there must be something like this; thanks for spelling it out. > I am not sure if I follow you correctly here but when using f2py from > its CVS, all fortran objects have _cpointer attribute that is > a C pointer to the actual Fortran or C function. Well, I'm not sure I've followed you, either. 8-) Here's what I'm trying to do. I have fortran functions and subroutines that need to evaluate special functions as part of what they're doing. E.g., for gamma functions, right now I just define a function "gammaln" that returns the log of a gamma function, within the fortran source of my module. But Scipy already has such a function somewhere (in the Cephes library). Is there a way I can use that Cephes function? For example, could I just figure out the appropriate Cephes call (sorting out the appropriate underscores and pointers) and write a setup.py that will let my fortran routine use the Cephes library that Scipy has installed somewhere? For performance, I'd rather not have to send my fortran module a Python callback; after all, the raw Cephes routine is sitting *somewhere*. There are other examples; another module I have needs a Cholesky decomposition in the middle of a calculation. Right now it's just hard-coded in the C for the module. I should just call the appropriate lapack/atlas routine for this, and I'd love to know how to write the C code and accompanying setup.py file to be able to do this portably without the overhead of a Python callback. Thanks, Tom ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/

2 1

OpenCD
by Alan G Isaac 13 Dec '04

13 Dec '04

I could not help noticing that OpenCD has no scientific tools. It seems the Enthought enhance Python distribution meets the inclusion criteria: http://theopencd.sunsite.dk/criteria.php fwiw, Alan Isaac

4 7

Two fortran/f2py questions
by Tom Loredo 13 Dec '04

13 Dec '04

Hi folks- Is there a way to use f2py directives to indicate that a few subprograms in a fortran77 source file need not be wrapped/exposed? Right now I'm just using the "only:" keyword on the command line, but I would imagine there's a way to embed this info in the source. I just haven't been able to figure it out from the Users Guide. Is there documentation anywhere showing how one can have fortran source for a Python extension access a C or fortran function that is part of Scipy? The specific example driving this is a module I'm working on that uses some special functions. It seems there should be some way for me to call the cephes functions directly. But I'm not sure how to make code that will automatically link to the appropriate library, and do it portably (so the module can be distributed). If there's a good example of this within Scipy (as I suspect), just give me a pointer and I'll "go to the source." I thought there might be some docs on this within scipy_distutils, but I haven't found it if it's there. Thanks, Tom ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/

2 1

building lib.lapack without optimization
by Nils Wagner 03 Dec '04

03 Dec '04

Hi Pearu, It looks like an "optimization" problem With python setup_lapack.py build config_fc --noopt everything works fine. lib/lapack> python2.3 tests/test_lapack.py -v 10 Found 200 tests for __main__ check_gebal (__main__.test_clapack_complex) ... ok check_gehrd (__main__.test_clapack_complex) ... ok check_heev (__main__.test_clapack_complex) ... ok check_heev_complex (__main__.test_clapack_complex) ... ok check_heevd (__main__.test_clapack_complex) ... ok check_heevd_complex (__main__.test_clapack_complex) ... ok check_heevr (__main__.test_clapack_complex) ... ok check_heevr_complex (__main__.test_clapack_complex) ... ok check_heevr_irange (__main__.test_clapack_complex) ... ok check_heevr_irange_high (__main__.test_clapack_complex) ... ok check_heevr_irange_low (__main__.test_clapack_complex) ... ok check_heevr_vrange (__main__.test_clapack_complex) ... ok check_heevr_vrange_high (__main__.test_clapack_complex) ... ok check_heevr_vrange_low (__main__.test_clapack_complex) ... ok check_gebal (__main__.test_clapack_double) ... ok check_gehrd (__main__.test_clapack_double) ... ok check_syev (__main__.test_clapack_double) ... ok check_syevd (__main__.test_clapack_double) ... ok check_syevr (__main__.test_clapack_double) ... ok check_syevr_irange (__main__.test_clapack_double) ... ok check_syevr_irange_high (__main__.test_clapack_double) ... ok check_syevr_irange_low (__main__.test_clapack_double) ... ok check_syevr_irange_mid (__main__.test_clapack_double) ... ok check_syevr_vrange (__main__.test_clapack_double) ... ok check_syevr_vrange_high (__main__.test_clapack_double) ... ok check_syevr_vrange_low (__main__.test_clapack_double) ... ok check_syevr_vrange_mid (__main__.test_clapack_double) ... ok check_gebal (__main__.test_clapack_double_complex) ... ok check_gehrd (__main__.test_clapack_double_complex) ... ok check_heev (__main__.test_clapack_double_complex) ... ok check_heev_complex (__main__.test_clapack_double_complex) ... ok check_heevd (__main__.test_clapack_double_complex) ... ok check_heevd_complex (__main__.test_clapack_double_complex) ... ok check_heevr (__main__.test_clapack_double_complex) ... ok check_heevr_complex (__main__.test_clapack_double_complex) ... ok check_heevr_irange (__main__.test_clapack_double_complex) ... ok check_heevr_irange_high (__main__.test_clapack_double_complex) ... ok check_heevr_irange_low (__main__.test_clapack_double_complex) ... ok check_heevr_vrange (__main__.test_clapack_double_complex) ... ok check_heevr_vrange_high (__main__.test_clapack_double_complex) ... ok check_heevr_vrange_low (__main__.test_clapack_double_complex) ... ok check_gebal (__main__.test_clapack_float) ... ok check_gehrd (__main__.test_clapack_float) ... ok check_syev (__main__.test_clapack_float) ... ok check_syevd (__main__.test_clapack_float) ... ok check_syevr (__main__.test_clapack_float) ... ok check_syevr_irange (__main__.test_clapack_float) ... ok check_syevr_irange_high (__main__.test_clapack_float) ... ok check_syevr_irange_low (__main__.test_clapack_float) ... ok check_syevr_irange_mid (__main__.test_clapack_float) ... ok check_syevr_vrange (__main__.test_clapack_float) ... ok check_syevr_vrange_high (__main__.test_clapack_float) ... ok check_syevr_vrange_low (__main__.test_clapack_float) ... ok check_syevr_vrange_mid (__main__.test_clapack_float) ... ok check_gebal (__main__.test_flapack_complex) ... ok check_gehrd (__main__.test_flapack_complex) ... ok check_heev (__main__.test_flapack_complex) ... ok check_heev_complex (__main__.test_flapack_complex) ... ok check_heevd (__main__.test_flapack_complex) ... ok check_heevd_complex (__main__.test_flapack_complex) ... ok check_heevr (__main__.test_flapack_complex) ... ok check_heevr_complex (__main__.test_flapack_complex) ... ok check_heevr_irange (__main__.test_flapack_complex) ... ok check_heevr_irange_high (__main__.test_flapack_complex) ... ok check_heevr_irange_low (__main__.test_flapack_complex) ... ok check_heevr_vrange (__main__.test_flapack_complex) ... ok check_heevr_vrange_high (__main__.test_flapack_complex) ... ok check_heevr_vrange_low (__main__.test_flapack_complex) ... ok check_gebal (__main__.test_flapack_double) ... ok check_gehrd (__main__.test_flapack_double) ... ok check_syev (__main__.test_flapack_double) ... ok check_syevd (__main__.test_flapack_double) ... ok check_syevr (__main__.test_flapack_double) ... ok check_syevr_irange (__main__.test_flapack_double) ... ok check_syevr_irange_high (__main__.test_flapack_double) ... ok check_syevr_irange_low (__main__.test_flapack_double) ... ok check_syevr_irange_mid (__main__.test_flapack_double) ... ok check_syevr_vrange (__main__.test_flapack_double) ... ok check_syevr_vrange_high (__main__.test_flapack_double) ... ok check_syevr_vrange_low (__main__.test_flapack_double) ... ok check_syevr_vrange_mid (__main__.test_flapack_double) ... ok check_gebal (__main__.test_flapack_double_complex) ... ok check_gehrd (__main__.test_flapack_double_complex) ... ok check_heev (__main__.test_flapack_double_complex) ... ok check_heev_complex (__main__.test_flapack_double_complex) ... ok check_heevd (__main__.test_flapack_double_complex) ... ok check_heevd_complex (__main__.test_flapack_double_complex) ... ok check_heevr (__main__.test_flapack_double_complex) ... ok check_heevr_complex (__main__.test_flapack_double_complex) ... ok check_heevr_irange (__main__.test_flapack_double_complex) ... ok check_heevr_irange_high (__main__.test_flapack_double_complex) ... ok check_heevr_irange_low (__main__.test_flapack_double_complex) ... ok check_heevr_vrange (__main__.test_flapack_double_complex) ... ok check_heevr_vrange_high (__main__.test_flapack_double_complex) ... ok check_heevr_vrange_low (__main__.test_flapack_double_complex) ... ok check_gebal (__main__.test_flapack_float) ... ok check_gehrd (__main__.test_flapack_float) ... ok check_syev (__main__.test_flapack_float) ... ok check_syevd (__main__.test_flapack_float) ... ok check_syevr (__main__.test_flapack_float) ... ok check_syevr_irange (__main__.test_flapack_float) ... ok check_syevr_irange_high (__main__.test_flapack_float) ... ok check_syevr_irange_low (__main__.test_flapack_float) ... ok check_syevr_irange_mid (__main__.test_flapack_float) ... ok check_syevr_vrange (__main__.test_flapack_float) ... ok check_syevr_vrange_high (__main__.test_flapack_float) ... ok check_syevr_vrange_low (__main__.test_flapack_float) ... ok check_syevr_vrange_mid (__main__.test_flapack_float) ... ok ---------------------------------------------------------------------- Ran 108 tests in 0.245s OK lib/lapack>

2 12