Mailman 3 December 2008 - SciPy-Dev

help with xplot->matplotlib translation
by josef.pktd＠gmail.com Dec. 23, 2008

Dec. 23, 2008

stats.morestats contains several function creating plots with xplot if plot is not None: try: import scipy.xplt as xplt xplt.limits() except: pass plot.plot(svals, ppcc, 'x') plot.title('(%s) PPCC Plot' % dist) plot.xlabel('Prob Plot Corr. Coef.'),deltay=-0.01) plot.ylabel('Shape Values'),deltax=-0.01) try: plot.expand_limits(5) except: pass I would like to translate this to using plot=matplotlib.pylab, but I don't know anything about xplt and I am not very good with matplotlib. So, if someone familiar with these, can provide a translation for xplt.limits(), deltay=-0.01, deltax and expand_limits, then I could fix these bugs without hunting documentation for hours. If I delete deltay and deltax, then the function works, but I don't know if I get the indended graph. Thank you, Josef

2 1

build problem error: file 'ARPACK\FWRAPPERS\dummy.f' does not exist
by josef.pktd＠gmail.com Dec. 23, 2008

Dec. 23, 2008

I was trying to build the current scipy trunk, r5286, and got stuck with arpack this is the last part of the build log building 'arpack' library compiling Fortran sources Fortran f77 compiler: C:\Programs\MinGW\bin\g77.exe -g -Wall -fno-second-undersc ore -mno-cygwin -O3 -funroll-loops error: file 'ARPACK\FWRAPPERS\dummy.f' does not exist WindowsXP, mingw 3.4, svn export python25 setup.py bdist >>> np.version.version '1.3.0.dev6139' I don't see any ARPACK\FWRAPPERS directory anywhere. Josef

2 3

python2.6 and scipy
by Nils Wagner Dec. 23, 2008

Dec. 23, 2008

Hi all, I just installed scipy using python2.6 on openSuSE 11.1 Ran 3353 tests in 53.293s FAILED (KNOWNFAIL=2, SKIP=28, errors=124, failures=1) <nose.result.TextTestResult run=3353 errors=124 failures=1> Is it recommended to use python2.5 ? Nils

7 13

import scipy.xplt as xplt
by josef.pktd＠gmail.com Dec. 22, 2008

Dec. 22, 2008

in scipy.stats.probplot there is a `import scipy.xplt as xplt` Is this a historical artifact that can be removed? Josef

2 1

EPD Py2.5 v4.1.30101 Released
by Chris Casey Dec. 22, 2008

Dec. 22, 2008

Greetings, Enthought, Inc. is very pleased to announce the newest release of the Enthought Python Distribution (EPD) Py2.5 v4.1.30101: http://www.enthought.com/epd The size of the installer has be reduced by about half. Also, this is the first release to include a 3.1.0 version of the Enthought Tool Suite (http://code.enthought.com/), featuring Mayavi 3.1.0. This is also the first release to use Enthought's enhanced version of setuptools, Enstaller (http://code.enthought.com/projects/enstaller/). Windows installation enhancements, matplotlib and wx issues, and menu consistency accross platforms are among notable fixes. The full release notes for this release can be found here: https://svn.enthought.com/epd/wiki/Py25/4.1.30101/RelNotes Many thanks to the EPD team for putting this release together, and to the community of folks who have provided all of the valuable tools bundled here. Best Regards, Chris --------- About EPD --------- The Enthought Python Distribution (EPD) is a "kitchen-sink-included" distribution of the Python™ Programming Language, including over 80 additional tools and libraries. The EPD bundle includes NumPy, SciPy, IPython, 2D and 3D visualization, database adapters, and a lot of other tools right out of the box. http://www.enthought.com/products/epd.php It is currently available as an easy, single-click installer for Windows XP (x86), Mac OS X (a universal binary for Intel 10.4 and above) and RedHat EL3 (x86 and amd64). EPD is free for 30-day trial use and for use in degree-granting academic institutions. An annual Subscription and installation support are available for commercial use (http://www.enthought.com/products/epddownload.php ) including an Enterprise Subscription with support for particular deployment environments (http://www.enthought.com/products/enterprise.php ). _______________________________________________ Enthought-dev mailing list Enthought-dev(a)mail.enthought.com https://mail.enthought.com/mailman/listinfo/enthought-dev

1 0

ValueError: Lack of convergence in obrientransform.
by Nils Wagner Dec. 22, 2008

Dec. 22, 2008

>>> scipy.__version__ '0.7.0.dev5286' ====================================================================== ERROR: Tests Obrien transform ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/scipy/stats/tests/test_mstats_basic.py", line 469, in test_obrientransform assert_almost_equal(np.round(mstats.obrientransform(*args).T,4), File "/usr/local/lib64/python2.5/site-packages/scipy/stats/mstats_basic.py", line 1732, in obrientransform raise ValueError("Lack of convergence in obrientransform.") ValueError: Lack of convergence in obrientransform.

2 2

segmentation with scipy+icc during test
by Stijn De Weirdt Dec. 22, 2008

Dec. 22, 2008

hi all, i hope i can bother the dev list with this, as it's not a real user problem imho. i'm trying to compile python+numpy+scipy with intel toolkit (ictce 3.2.0.17, which has icc/ifc 11.0.069 and mkl 10.1.0.015 python 2.4.5 compiles and the make tests works fine numpy 1.2.1 compiles and the tests are also fine scipy 0.6.0 compiles, but one of the tests exits with a segmentation fault. (i also tried 0.7.0b1, but it also gives segfualt, but haven't verified if it's the same one) now i'm trying to figure out which test causes the crash and i have some questions related to that: a. how can i skip/reorder tests? (just to check if it's an individual issue or a broader one) b. the segfault itself: python -v -c 'import scipy; scipy.test(level=10,verbosity=100)' ... bench_random (scipy.fftpack.tests.test_basic.test_fft) Fast Fourier Transform ================================================= | real input | complex input ------------------------------------------------- size | scipy | numpy | scipy | numpy ------------------------------------------------- 100 | 0.07 | 0.06Segmentation fault looking at the code in scipy/fftpack/tests/test_basic.py, it's the complex part of bench_random that fails. the basic code that triggers it, is (at least that how i read it) x= random([size]).astype(cdouble)+random([size]).astype(cdou\ ble)*1j y = direct_dft(x) ## size < 500 z=fft(x) with fft loaded from scipy.fftpack (not from numpy) (i'm sure it segfaults during the fft(x) call, because i added some print statements to locate the exact code). when i add some pprint statements to dump the x values that were used during the segfault and write a small script doing just the above with those values, it doesn't segfault. so doing the exact steps (except from the self.measure) in a separate script, i'm unable to trigger the segfault, but the tests always fail. could someone shine some light on why this seems not reproducable outside the test environment? many thanks, stijn -- The system will shutdown in 5 minutes.

3 12

Non-deterministic test failures in scipy-0.7.0b1
by Roger Kramer Dec. 22, 2008

Dec. 22, 2008

I'm still looking into this, but in case anyone has ideas that could save me some time... The following occurs non-deterministically: ====================================================================== FAIL: test_x_stride (test_fblas.TestCgemv) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/rk/lib/python2.5/site-packages/scipy/lib/blas/tests/test_fblas.py", line 345, in test_x_stride assert_array_almost_equal(desired_y,y) File "/home/rk/lib/python2.5/site-packages/numpy/testing/utils.py", line 311, in assert_array_almost_equal header='Arrays are not almost equal') File "/home/rk/lib/python2.5/site-packages/numpy/testing/utils.py", line 296, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (mismatch 33.3333333333%) x: array([ -2.91183448 +2.91183448j, -14.95274162+16.95274162j, 3.63382101 +0.36617899j], dtype=complex64) y: array([ -2.91183448 +2.91183448j, -14.95274162+16.95274353j, 3.63382101 +0.36617899j], dtype=complex64) ------------------------------------------------------------------------ ...and test_y_stride occasionally fails, too. What's odder than the fact that the arrays don't really look 33% mismatched, is that on successive runs of scipy.test(), the failure may or may not occur, and I'm changing/touching -nothing- between the test runs. Normally, non-deterministic failures would scream race conditions, but since these are the only test failures it seems a stretch to implicate threading in general. In case it's relevant, I compiled (gcc 4.3.2) Netlib CBLAS and LAPACK 3.2 sources directly as shared libraries. Have had considerable trouble with LAPACK. Compiled code only passes the lapack test suite if the machine-dependent bits in INSTALL are compiled without optimization. Everything else (in LAPACK) is compiled with -O2 -march=native -mtune=native, and I'm on a Dell machine with an E2160 processor. Intel's "issues list" on this processor family is a fright.

2 1

changes to stats t-tests
by josef.pktd＠gmail.com Dec. 21, 2008

Dec. 21, 2008

I finally looked at the t-test in more detail and I would like to make three changes to the ttests 1) ttest_1samp: add axis argument, it is the only ttest and similar function without an axis argument. Currently the retrun for multi dimensional arrays is wrong, the array is flattened for the calculation of mean and variance, but the number of observations is taken along axis 0 (Test suite only check for 1-dim case). One problem is the default axis, the usual default axis in statistics, scipy.stats is zero, but it is not always consistently used ttest_rel(a,b,axis=None), ttest_ind(a, b, axis=0), sem(a, axis=0) 2) return for t statistic: return number instead of 0-dimensional array, no changes for higher dimensions Same for all three tests. current return (t-statistic, p-value) (array(0.81248591389165681), 0.41846234511362179) proposed return (0.81248591389165681, 0.41846234511362179) 3) clean up handling of zero division problem: current zero division problem doesn't make sense, it return a t-statistic arbitrarily set to one. This only applies to cases with zero variance proposed change, return either inf (if numerator different from zero) or zero (if numerator is also zero. Same for all three tests. current: >>> print st.ttest_rel( [0,0], [1,1],0) #mean is different (array(1.0), 0.49999999999999956) >>> print st.ttest_rel( [0,0], [0,0],0) #mean is the same (array(1.0), 0.49999999999999956) proposed: >>> print ttest_rel( [0,0], [1,1],0) (-1.#INF, 0.0) >>> print ttest_rel( [0,0], [0,0],0) (0.0, 1.0) Since, we are just before release, I don't know if I can make these changes now or after the release. Most of it fixes incorrect returns, and I'm writing the tests for all cases. Josef

3 3

scipy.optimize.nonlin rewrite
by Pauli Virtanen Dec. 20, 2008

Dec. 20, 2008

Hi all, I spent some time rewriting the scipy.optimize.nonlin module: http://github.com/pv/scipy/tree/ticket-791/scipy/optimize/nonlin.py The following things changed: - Support tolerance-based stopping conditions (cf. ticket #791) - Improved handling of input and return values from the functions, so that they are now easier to use. - Don't use np.matrix at all. - Jacobian approximations factored into classes; the iteration code is now in only one place, so trust-region handling or other improvements can be added to a single place later on. - There's now a line search to the direction inverting the Jacobian gave. (But there's no checking that it is an actual descent direction for some merit function.) - Rewrote docstrings. The routines should produce the same output as previously. The tests are still not very strong, however. But I have now some questions: * Getting this to 0.7.0; this is a complete rewrite, so is it too late, and is it better to wait for 0.8? * Some of the algorithms in there don't appear to work too well, and some appear to be redundant. I'd like to clean up this a bit, leaving only known-good stuff in. * I'd like to remove `broyden2` as the actual Jacobian approximation in this appears to be the same as in `broyden3`, and there does not appear to be much difference in the work involved in the two. Ondrej, since you wrote the original code, do you think there is a reason to keep both? * `broyden1_modified` appears to be, in the end if you work out the matrix algebra, updating the inverse Jacobian in a way that corresponds to J := J + (y - J s / 2) s^T / ||s||^2 for the Jacobian (with s = dx, y = df). Apart from the factor 1/2, it's Broyden's good method. [1] One can also verify that the updated inverse Jacobian does not satisfy the quasi-Newton condition s = J^{-1} y, and that `broyden1_modified` doesn't generate the same sequence as `broyden1`. Hence, I'd like to remove this routine, unless there's some literature that shows that the above works better than Broyden's method; Ondrej, do you agree? .. [1] http://en.wikipedia.org/wiki/Broyden%27s_method http://en.wikipedia.org/wiki/Sherman%E2%80%93Morrison_formula * Also, which articles were used as reference for the non-Quasi-Newton algorithms: - `broyden_modified`. This appears to be a bit exotic, and it uses several magic constants (`w0`, `wl`) whose meaning is not clear to me. A reference would be helpful here, also for the user who has to choose the parameters as appropriate for his/her specific problem. - `broyden_generalized`, `anderson`, `anderson2`. These appear to be variants of Anderson mixing, so probably we only want at most one of these. Also, broyden_generalized() == anderson(w0=0), am I correct? `anderson` and `anderson2` don't appear to function equivalently, and I suspect the update formula in the latter is wrong, since this algorithm can't solve any of the test problems. Do you have a reference for this? Is there a rule how `w0` should be chosen for some specific problem? - `excitingmixing`. A reference would be useful, to clarify the heuristics used. - `linearmixing`. I'm not sure that Scipy should advocate this method :) Linearmixing and excitingmixing also seem to require something from the objective function, possibly that the eigenvalues of its Jacobian have the opposite sign than `alpha`. For example, neither of them can find a solution for the equation ``I x = 0`` where ``I`` is the identity matrix (test function F2). So, I'm a bit tempted to remove also these routines, as it seems they probably are not too useful for general problems. * The code in there is still a bit naive implementation of the inexact (Quasi-)Newton method, and one could add eg. add trust-region handling or try to guarantee that the line search direction is always a decrease direction for a merit function. (I don't know if it's possible to do the latter unless one has a matrix representation of the Jacobian approximation.) So, I suspect there are problems for which eg. MINPACK code will find solutions, but for which the nonlin.py code fails. * One could add more algorithms suitable for large-scale problems; for example some limited-memory Broyden methods (eg. [1]) or Secant-Krylov methods [2]. .. [1] http://www.math.leidenuniv.nl/~verduyn/publications/reports/equadiff.ps .. [2] D. A. Knoll and D. E. Keyes. Jacobian free Newton-Krylov methods. Journal of Computational Physics, 20(2):357–397, 2004. I have implementations for both types of algorithms that could possibly go in after some polishing. -- Pauli Virtanen

13 34