Mailman 3 June 2012 - SciPy-Dev

branching 0.11.x
by Ralf Gommers June 12, 2012

June 12, 2012

Hi all, We're almost ready to branch 0.11.x and tag the first beta. Here's some things that are still TODO. Necessary to finish before the first beta: - Fix http://projects.scipy.org/scipy/ticket/1599 (work around segfaults due to single-precision shift-invert mode in Arpack) - https://github.com/scipy/scipy/pull/217 (Finishing optimize.minimize() interface changes) Can be fixed after the first beta: - http://projects.scipy.org/scipy/ticket/1669 (sparse.csgraph test failures) - https://github.com/scipy/scipy/pull/198 (fitting discrete distributions issue) Things that would be good to include, but will be postponed unless they're merged in time: - https://github.com/scipy/scipy/pull/236 (use memoization in Minpack routines) - https://github.com/scipy/scipy/pull/235 (t-test with unequal variances) - https://github.com/scipy/scipy/pull/173 (binned statistics) - any other PRs / patches that are ready If we can get the necessary items resolved this weekend, I plan to create the branch tomorrow evening (at or after 9pm GMT). Cheers, Ralf

1 3

scipy-central.org down?
by Warren Weckesser June 11, 2012

June 11, 2012

Is scipy-central.org working for anyone? I get a 403 error. Warren

4 5

Resolving PR 235: t-statistic = 0/0 case
by Junkshops June 11, 2012

June 11, 2012

Hi all, This mail references PR 235: https://github.com/scipy/scipy/pull/235 The PR adds a method for performing a t-test on 2 samples with unequal or unknown variances and changes the return value when the t-statistic = 0 / 0 to 0 from the previous code's return value, 1, for all t-test functions. The latter change is a point of contention for the pull. I take the position that t = 0/0 should return 0, while Josef, the primary scipy.stats maintainer, believes that t = 0/0 should return one. Normally I would try to resolve this directly with Josef, but unfortunately I haven't heard from him in over two days, and the beta release is scheduled for Saturday. As such, I'm writing the dev list to ask what to do next. Josef's position, to the best of my ability to understand it, is that: J1) Adding a small amount of noise to data sets that have otherwise equal means (e.g. [0,0,0,0], [0,0,0,1e-100]) results in a t-statistic of 1. Thus, as the mean difference approaches zero, t -> 1. J2) A data set with no mean difference and no variance is a zero probability event. As such, returning t = 1 is reasonable, as therefore p = 0.317-0.5 for a two tailed test depending on the degrees of freedom, and hence for standard values of alpha the null hypothesis will not be rejected, but the user gets some feedback that his data is suspect. I admit I don't completely understand the second argument. Hopefully when Josef resurfaces he can correct my representation of his argument if needed. My responses to these arguments are: J1) If you take the n-length vectors (x,...,x) and (x,...x,x+y) and solve for t, t = 1 for any value of x and y. This is simply due to the fact that the the mean difference is y/n and the pooled variance (e.g. the denominator of the t-statistic) is also y/n. Thus this is merely a special case and does not represent a limit as the mean difference approaches zero, since even for [0,0,0,0], [0,0,0,1e100] t = 1. J2) Strictly speaking, if we're pulling independent random samples from a normal distribution over the reals, say, any given set of samples has zero probability since there are an infinite number of possible samples. The sample [0,0,0] is just as probable as the sample [2.3, 7.4, 2.1]. In a discrete distribution, in fact, the sample [0,0,0] is *more* likely than any other sample. Also, we're now analyzing the legitimacy of the user's data, which is not the job of the t-test. The t-test is expected to take a data set(s) and provide a measure of the strength of the evidence that the means are different. Is it not expected to comment on the validity of the user's data. My arguments that 0 is the correct answer to return are: 1) if MD = mean difference and PV = pooled variance, then t = MD/PV. As MD -> 0 for any PV > 0, t -> 0. However, if we set t = 0/0 = 1 and MD = 0, as PV -> 0 we introduce a discontinuity at zero since t = 0 for any value of PV except 0, where t = 1. This implies that for MD = PV = 0, t = 1, but if the variance of the dataset is infinitesimally small but !=0, then t = 0. To me, this makes no sense. 2) The t-test's purpose is to measure the strength of the evidence against the null hypothesis that MD = 0. If MD = 0, as in the case we're discussing, by definition there is no evidence that MD != 0. Therefore p must = 1, and as a consequence t must = 0. I also consulted with a statistics professor - who agreed with my position - to make sure I wasn't talking out of turn. In summary, I think that if t = 0/0 we should return 0. However, both R and Matlab return NaN. To me this seems incorrect due to argument #2. Josef also mentioned there were users of scipy.stats.stats that didn't want NaN return values for some reason that I don't know offhand. How would the Scipy devs like to proceed? Cheers, Gavin

6 17

Bug fix pull-request for signal.bode()
by Bjørn Forsman June 10, 2012

June 10, 2012

Hi, I discovered a bug in signal.bode() that makes it unable to handle a systems with a zero-pole. signal.bode(([], [0], 1)) will provoke it. Please pull https://github.com/scipy/scipy/pull/253 to fix this bug. Best regards, Bjørn Forsman

1 0

[Probe] How amongst you are pro devs working for scipy ?
by David Kremer June 9, 2012

June 9, 2012

Hello everyone, First I would like to tell you that I am a somehow satisfied scipy user, even if I feel it lacks official recognition. First, my question is concerning the scipy developers. I would like to know, amongst you, who is working on scipy as paid-job, or only on its spare time. Basically it makes the developer community divides in two parts : the first is the community of hobbyists, the second the community of profesional developers. Knowing grossly the proportion of pro and hobbyist, I would have a second question, related to the first : who uses scipy in business related project (I mean, no academic, that is to say, any work that is not included into a public research project). I feel that scipy is a very big project, and must have a lot of maintainers, and I am wondering if this project is actually used by real people out there, apart from PhD student (including myself) and researchers. I know that scipy, in its field, may encounter two great concurrents (for the least), namely matlab and octave, even if the goal of scipy may be slightly different. Well, any developer or member of the community that could answer this question or give some hints on the related topic would be very nice. Thank you. David Kremer PhD student Angers, France

4 4

Procedure for new code submission (pyHHT)
by Stuart Mumford June 9, 2012

June 9, 2012

Hello, I am working on a EMD / HHT code for some analysis I am doing as part of my PhD. (It is currently here: https://github.com/Cadair/pyhht (kicked off from jaidevd's inital work)) I was wondering what the general procedure for getting something like this included in SciPy is. I know it requires a lot more work to even be considered but I would like to focus what I am doing with that end in mind. The Web here: http://scipy.github.com/dev-zone.html says that initially you should write a SciKit, is this still the way to go or is it fine just to create a repo with good quality documentation etc? Also is a HHT toolbox something that should be integrated into scipy.signal or (at risk of restarting an amusing debate) should it go into scikit.signal?? Stuart

6 14

Ubuntu PPA for NumPy / SciPy / ...
by Andreas Hilboll June 8, 2012

June 8, 2012

Hi, I just noticed that there's a PPA for NumPy/SciPy on Launchpad: https://launchpad.net/~scipy/+archive/ppa However, it's painfully outdated. Does anyone know of its status? Is it 'official'? Are there any plans in revitalizing it, possibly with adding other projects from the "scipy universe"? Is there help needed? Many questions, but possibly quite easy to answer ... Cheers, Andreas.

2 1

scipy 0.11.0 release schedule
by Ralf Gommers June 5, 2012

June 5, 2012

Hi all, It's time for a new release I think; 0.10.0 was released just over 6 months ago. There are a lot of PRs to merge, but no release blockers anymore as far as I know. Does anyone still have important fixes or other things that should go in? For the release schedule I propose: June 7: beta 1 June 17: rc 1 June 30: rc 2 July 7: final release Ralf

4 8

test failures with python 2.4
by Denis Laxalde June 4, 2012

June 4, 2012

Hi, Here is a list of test failures with python 2.4 (there are also failures with 2.7, see below). Interpolate: test_interpnd.TestCloughTocher2DInterpolator.test_dense ... Exception exceptions.TypeError: <exceptions.TypeError instance at 0x7fd4d67fecf8> in 'scipy.spatial.qhull._get_delaunay_info' ignored Segmentation fault Spatial: test_qhull.TestUtilities.test_convex_hull ... ERROR test_qhull.TestUtilities.test_degenerate_barycentric_transforms ... ERROR test_qhull.TestUtilities.test_find_simplex ... Exception exceptions.TypeError: <exceptions.TypeError instance at 0x7f9e857963b0> in 'scipy.spatial.qhull._get_delaunay_info' ignored FAIL test_qhull.TestUtilities.test_more_barycentric_transforms ... ERROR test_qhull.TestUtilities.test_plane_distance ... ok ====================================================================== ERROR: test_qhull.TestUtilities.test_convex_hull ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/denis/.local/lib/python2.4/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/denis/.local/lib/python2.4/site-packages/scipy/spatial/tests/test_qhull.py", line 75, in test_convex_hull assert_equal(tri.convex_hull, [[1, 2], [3, 2], [1, 0], [3, 0]]) File "qhull.pyx", line 1109, in scipy.spatial.qhull.Delaunay.convex_hull (scipy/spatial/qhull.c:8406) ValueError: cannot resize an array references or is referenced by another array in this way. Use the resize function ====================================================================== ERROR: test_qhull.TestUtilities.test_degenerate_barycentric_transforms ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/denis/.local/lib/python2.4/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/denis/.local/lib/python2.4/site-packages/scipy/spatial/tests/test_qhull.py", line 133, in test_degenerate_barycentric_transforms bad_count = np.isnan(tri.transform[:,0,0]).sum() File "qhull.pyx", line 1028, in scipy.spatial.qhull.Delaunay.transform (scipy/spatial/qhull.c:7449) File "qhull.pyx", line 398, in scipy.spatial.qhull._get_barycentric_transforms (scipy/spatial/qhull.c:4300) File "/home/denis/.local/lib/python2.4/site-packages/numpy/core/numeric.py", line 235, in asarray return array(a, dtype, copy=False, order=order) File "stringsource", line 366, in View.MemoryView.memoryview.__getitem__ (scipy/spatial/qhull.c:14704) File "stringsource", line 650, in View.MemoryView._unellipsify (scipy/spatial/qhull.c:17965) TypeError: Cannot index with type '<type 'int'>' ====================================================================== ERROR: test_qhull.TestUtilities.test_more_barycentric_transforms ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/denis/.local/lib/python2.4/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/denis/.local/lib/python2.4/site-packages/scipy/spatial/tests/test_qhull.py", line 157, in test_more_barycentric_transforms unit_cube=True) File "/home/denis/.local/lib/python2.4/site-packages/scipy/spatial/tests/test_qhull.py", line 96, in _check_barycentric_transforms c = barycentric_transform(tri.transform, centroids) File "qhull.pyx", line 1028, in scipy.spatial.qhull.Delaunay.transform (scipy/spatial/qhull.c:7449) File "qhull.pyx", line 398, in scipy.spatial.qhull._get_barycentric_transforms (scipy/spatial/qhull.c:4300) File "/home/denis/.local/lib/python2.4/site-packages/numpy/core/numeric.py", line 235, in asarray return array(a, dtype, copy=False, order=order) File "stringsource", line 366, in View.MemoryView.memoryview.__getitem__ (scipy/spatial/qhull.c:14704) File "stringsource", line 650, in View.MemoryView._unellipsify (scipy/spatial/qhull.c:17965) TypeError: Cannot index with type '<type 'int'>' ====================================================================== FAIL: test_qhull.TestUtilities.test_find_simplex ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/denis/.local/lib/python2.4/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/denis/.local/lib/python2.4/site-packages/scipy/spatial/tests/test_qhull.py", line 33, in test_find_simplex assert_equal(i, p[2], err_msg='%r' % (p,)) File "/home/denis/.local/lib/python2.4/site-packages/numpy/testing/utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/home/denis/.local/lib/python2.4/site-packages/numpy/testing/utils.py", line 707, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/home/denis/.local/lib/python2.4/site-packages/numpy/testing/utils.py", line 636, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (0.25, 0.25, 1) (mismatch 100.0%) x: array(-1, dtype=int32) y: array(1) ---------------------------------------------------------------------- Ran 291 tests in 13.335s FAILED (errors=3, failures=1) Weave: FAIL: test_set_complex (test_scxx_object.TestObjectSetItemOpKey) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/denis/.local/lib/python2.4/site-packages/scipy/weave/tests/test_scxx_object.py", line 943, in test_set_complex assert_equal(sys.getrefcount(key),4) # should be 3 File "/home/denis/.local/lib/python2.4/site-packages/numpy/testing/utils.py", line 313, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: 3 DESIRED: 4 Failures not specific to python 2.4. Sparse: ====================================================================== ERROR: adding a dense matrix to a sparse matrix ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", line 527, in test_add_dense sum1 = self.dat + self.datsp File "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line 133, in __getitem__ raise TypeError('index must be a pair of integers or slices') TypeError: index must be a pair of integers or slices ====================================================================== ERROR: test_matmat_sparse (test_base.TestDOK) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", line 425, in test_matmat_sparse assert_array_almost_equal( a2*bsp, a*b) File "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line 133, in __getitem__ raise TypeError('index must be a pair of integers or slices') TypeError: index must be a pair of integers or slices ====================================================================== ERROR: test_radd (test_base.TestDOK) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", line 279, in test_radd c = a + b File "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line 133, in __getitem__ raise TypeError('index must be a pair of integers or slices') TypeError: index must be a pair of integers or slices ====================================================================== ERROR: test_rsub (test_base.TestDOK) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", line 290, in test_rsub assert_array_equal((self.dat - self.datsp),[[0,0,0,0],[0,0,0,0],[0,0,0,0]]) File "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line 133, in __getitem__ raise TypeError('index must be a pair of integers or slices') TypeError: index must be a pair of integers or slices ====================================================================== ERROR: subtracting a dense matrix to/from a sparse matrix ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/tests/test_base.py", line 535, in test_sub_dense sum1 = 3*self.dat - self.datsp File "/home/denis/.local/lib/python2.4/site-packages/scipy/sparse/dok.py", line 133, in __getitem__ raise TypeError('index must be a pair of integers or slices') TypeError: index must be a pair of integers or slices ---------------------------------------------------------------------- Ran 1210 tests in 12.543s FAILED (KNOWNFAIL=4, errors=5) Special: ................../home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1645: RuntimeWarning: divide by zero encountered in iv c1 = special.iv(v, x) /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1645: RuntimeWarning: overflow encountered in iv c1 = special.iv(v, x) /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1646: RuntimeWarning: invalid value encountered in iv c2 = special.iv(v, x+0j) /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1652: RuntimeWarning: divide by zero encountered in divide dc = abs(c1/c2 - 1) /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1652: RuntimeWarning: invalid value encountered in divide dc = abs(c1/c2 - 1) /home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py:1659: RuntimeWarning: overflow encountered in iv assert_(dc[k] < 1e-9, (v[k], x[k], special.iv(v[k], x[k]), special.iv(v[k], x[k]+0j))) F...................................................K.K.............................................................................................................................................................................................................................................................................................................................................................................................K........K.................................................. ====================================================================== FAIL: test_iv_cephes_vs_amos_mass_test (test_basic.TestBessel) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/denis/.local/lib/python2.7/site-packages/scipy/special/tests/test_basic.py", line 1659, in test_iv_cephes_vs_amos_mass_test assert_(dc[k] < 1e-9, (v[k], x[k], special.iv(v[k], x[k]), special.iv(v[k], x[k]+0j))) File "/usr/lib/pymodules/python2.7/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: (189.2947429454936, 3.0238805556481037, 4.089165443940765e-317, 0j) ---------------------------------------------------------------------- Ran 514 tests in 6.908s FAILED (KNOWNFAIL=4, failures=1) Building/testing environment for python 2.4 (Debian wheezy/sid x86_64): NumPy version 1.6.2 NumPy is installed in /home/denis/.local/lib/python2.4/site-packages/numpy SciPy version 0.11.0.dev-127acfd SciPy is installed in /home/denis/.local/lib/python2.4/site-packages/scipy Python version 2.4.6 (#2, Sep 25 2009, 22:22:06) [GCC 4.3.4] nose version 1.1.2 -- Denis

2 5

contributing and maintainers docs
by Ralf Gommers June 4, 2012

June 4, 2012

Hi all, I've finally merged both the maintainers and contributing howto docs. If module maintainers could add some details on current status/issues and future plans and directions for development, that would be great. Cheers, Ralf

2 2