Mailman 3 September 2013 - SciPy-Dev

Re: [SciPy-Dev] [Numpy-discussion] Baffling error: ndarray += csc_matrix -> "ValueError: setting an array element with a sequence"
by Nathaniel Smith 27 Sep '13

27 Sep '13

On Fri, Sep 27, 2013 at 7:34 PM, Pauli Virtanen <pav(a)iki.fi> wrote: > 27.09.2013 19:33, Nathaniel Smith kirjoitti: > [clip] >> I really don't understand what arcane magic is used to make ndarray += >> csc_matrix work at all, but my question is, is it going to break when >> we complete the casting transition described above? It was just >> supposed to catch things like int += float. > > This maybe clarifies it: > >>>> import numpy >>>> import scipy.sparse >>>> x = numpy.ones((2,2)) >>>> y = scipy.sparse.csr_matrix(x) >>>> z = x >>>> z += y >>>> x > array([[ 1., 1.], > [ 1., 1.]]) >>>> z > matrix([[ 2., 2.], > [ 2., 2.]]) > > The execution flows like this: > > ndarray.__iadd__(arr, sparr) > np.add(arr, sparr, out=???) > return NotImplemented # wtf > return NotImplemented > Python does arr = sparr.__radd__(arr) > > Since Scipy master sparse matrices now have __numpy_ufunc__, but it > doesn't handle out= arguments, the second step currently raises a > TypeError (for Numpy master + Scipy master). > > And this is actually the correct thing to do, as having np.add return > NotImplemented is just broken. Only ndarray.__iadd__ has the authority > to return the NotImplemented. > > To make the in-place ops work again, it seems Numpy needs some > additional fixes in its binary op machinery, before __numpy_ufunc__ > business works fully as intended. Namely, the binary op routines will > need to catch TypeErrors and convert them to NotImplemented. > > The code paths where Numpy ufuncs currently return NotImplemented could > also be changed to raise TypeErrors, but I'm not sure if someone > somewhere relies on this behavior (I hope not). Okay, so I see three separate issues: 1) My original concern, that the upcoming casting change for in-place operations will cause some horrible interaction. Tentatively this seems like it might be okay since even after the "cast" succeeds, np.add is still just refusing to do the operation, so hopefully we can set it up so that it will continue to fail once the casting rule becomes more strict. 2) The issue that ufuncs return NotImplemented and it makes baby Guido cry. This is completely broken, agreed. Not sure when someone will get around to clearing this stuff up. 3) The issue of how to make an in-place like ndarray += sparse continue to work in the brave new __numpy_ufunc__ world. For this last issue, I think we disagree. It seems to me that the right answer is that csc_matrix.__numpy_ufunc__ needs to step up and start supporting out=! If I have a large dense ndarray and I try to += a sparse array to it, this operation should take no temporary memory and nnz time. Right now it sounds like it actually copies the large dense ndarray, which takes time and space proportional to its size. AFAICT the only way to avoid that is for scipy.sparse to implement out=. It shouldn't be that hard...? -n

1 0

Re: [SciPy-Dev] Baffling error: ndarray += csc_matrix -> "ValueError: setting an array element with a sequence"
by Nathaniel Smith 27 Sep '13

27 Sep '13

[CC'ing scipy-dev because see below] On Tue, Sep 24, 2013 at 6:45 PM, Nathaniel Smith <njs(a)pobox.com> wrote: > Hi all, > > I'm getting a very strange error, and unfortunately I can't seem to > reproduce it *except* when running on Travis-CI, so my debugging tools > are really limited. I'm wondering if anyone else has seen anything > like this? > > The offending line of code is: > > File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pyrerp/incremental_ls.py", > line 323, in append_bottom_half > self.xtx += xtx > ValueError: setting an array element with a sequence. > > And debug prints reveal that in the case that causes the error, > 'self.xtx' is an ndarray that prints as: > > [[ 0. 0.] > [ 0. 0.]] > > while 'xtx' is a scipy.sparse.csc.csc_matrix that prints as: > > (1, 0) 45.0 > (0, 0) 10.0 > (1, 1) 285.0 > (0, 1) 45.0 In accordance with the cosmic law governing such things, this turns out to be a bug in numpy that I introduced in 2397c9d4, specifically this line, which introduces an unchecked DEPRECATE: https://github.com/numpy/numpy/commit/2397c9d4#L5R528 (Plus some complicated nonsense involving virtualenvs that sometimes fall back on system libraries even though the virtualenv contains a perfectly good version of the same library etc. to make reproduction more confusing.) My test code was setting warnings to raise errors by default, and apparently ndarray += csc_matrix goes through some circuitous path that (AFAICT) involves creating an object ndarray containing the csc_matrix and calling the add ufunc, which somehow trips on the cast warning above. Then later on, at line 159 of arraytypes.c.src, the the @TYPE@_setitem function does: if (PyErr_Occurred()) { if (PySequence_Check(op) && !PyString_Check(op) && !PyUnicode_Check(op)) { PyErr_Clear(); PyErr_SetString(PyExc_ValueError, "setting an array element with a sequence."); } return -1; } and the PyErr_Occurred here catches the real error and replaces it with some nonsense. So three points: 1) Maybe this will be useful to someone googling later. 2) I guess I'll file some horrible patch against numpy that just throws away exceptions caused by that deprecation, because there is no way for PyArray_CanCastTypeTo to raise an error :-(. 3) This script raises a DeprecationWarning with numpy 1.7.1 and scipy 0.12.0: import warnings, numpy, scipy.sparse warnings.filterwarnings("always") a = numpy.zeros((2, 2)) b = scipy.sparse.csc_matrix([[0.0, 0.0], [0.0, 0.0]]) a += b test-script3.py:5: DeprecationWarning: Implicitly casting between incompatible kinds. In a future numpy release, this will raise an error. Use casting="unsafe" if this is intentional. I really don't understand what arcane magic is used to make ndarray += csc_matrix work at all, but my question is, is it going to break when we complete the casting transition described above? It was just supposed to catch things like int += float. -n

1 0

Scipy 1.0 roadmap
by Ralf Gommers 24 Sep '13

24 Sep '13

Hi all, At EuroScipy Pauli, David and I sat together and drafted a roadmap for Scipy 1.0. We then discussed this offline with some of the other currently most active core devs, to get it into a state that's ready for discussion on this list. So here it is: https://github.com/scipy/scipy/pull/2908 Our aim is for this roadmap to help guide us towards a 1.0 version, which will contain only code that we consider to be "of sufficient quality". Also, it will help to communicate to new and potential developers where their contributions are especially needed. In order to discuss/review this roadmap without generating a monster thread, I propose the following: - topics like "do we need a roadmap?" or "what does 1.0-ready really mean?" are discussed on this thread. - things in the General section (API changes, documentation/test/build guidelines, etc.), are discussed on this thread as well. - for discussion of module-specific content, start a new thread and name it "1.0 roadmap: <module_name>". - for minor things, comment on the PR. Cheers, Ralf

14 25

Comment on scipy.interpolate.UnivariateSpline error
by Padarn Wilson 23 Sep '13

23 Sep '13

Hi, I (well actually a friend) was recently trying to fit a UnivariateSpline to some data and then evaluate the derivative. The spline generated fine and could be evaluated (it looked reasonable) but the derivative would give the error: File "/usr/lib/pytho n2.7/dist-packages/scipy/interpolate/fitpack2.py", line 258, in derivatives raise ValueError("Error code returned by spalde: %s" % ier) ValueError: Error code returned by spalde: 10 It turned out that the x-array input was not in ascending order, as the documentation says is required. The exact data I was using lead to non-ascending knots. I recognise this error is not encountered if the documentation is followed, but the error is uninformative, and it seems like an easy mistake to make. Especially given the spline still looks fine otherwise. In case anyone wants to reproduce the example, the data are: 2.651,1.421 2.581,1.36 2.494,1.256 2.398,1.186 2.328,1.116 2.232,1.046 2.11,0.951 2.041,0.881 1.945,0.837 1.84,0.794 1.727,0.733 1.631,0.689 1.57,0.663 1.483,0.602 1.413,0.567 1.291,0.515 1.203,0.488 1.108,0.445 0.994,0.392 0.898,0.349 0.802,0.314 0.724,0.279 0.619,0.253 0.523,0.218 0.419,0.183 0.314,0.14 0.253,0.096 0.157,0.078 0.087,0.044 Thanks, Padarn

1 0

Voting functionality on scipy central
by Surya Kasturi 22 Sep '13

22 Sep '13

Hi there, At this time, we thought of giving a relook to Reputation system we built for SciPy Central. There are some of design decisions that have to be looked into 1. Should new revisions carry on reputation of previous revision? or 2. Should we attach reputation to "Submission" as a whole or each "Revision" individually? Thanks

3 6

how to turn internal names into API and freeze them forever
by josef.pktd＠gmail.com 22 Sep '13

22 Sep '13

The scipy.stats.distributions allow now keyword access to the shape parameters. Before, the names of shape parameters where just part of the signature, but couldn't be used in code, only positional argument where possible. Now, users can actually use them in their code. hypergeom uses `M,n,N` where I usually need to read the description for several minutes to remember what they are. most distributions use one letter parameters: Since I just checked numpy.random.triangular uses `mode` scipy.triang just uses `c` Josef

2 1

Re: [SciPy-Dev] Scipy 1.0 roadmap - stats
by josef.pktd＠gmail.com 22 Sep '13

22 Sep '13

On Sun, Sep 22, 2013 at 4:52 AM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: > > > > On Sun, Sep 22, 2013 at 7:52 AM, Christopher Jordan-Squire <cjordan1(a)uw.edu> > wrote: >> >> For scipy stats, is there anything on the table regarding somehow >> unifying the sampling in numpy.random and the distributions in >> scipy.stats? I'm specifically thinking of two issues: >> >> (1) There's a lot of duplication between numpy.random and scipy.stats >> but with different interfaces. This seems like something that ideally >> would be reduced. > > > numpy.random only provides sampling and only has about half the > distributions of scipy.stats. Sampling is really only a small part of what > scipy.stats provides (pdf, cdf, moments, fitting a distribution, etc.). So > I'm not bothered by that duplication. If we'd want to reduce it I think it > would have to be removed from numpy, which doesn't sound like a good idea. There is no code duplication, so it also never bothered me. scipy.stats distributions has quite a bit more overhead than numpy random if you call random number generation repeatedly instead of requesting one big array. There are some naming inconsistencies between scipy.stats and numpy.random, but I never looked systematically for that, and there is no open issue. The distributions in scipy.stats have restrictions on the choice of paramaterization because of the generic use of loc and scale. One issue that should be added to the roadmap is fixing the broadcasting of loc and scale in the scipy random numbers. I don't see a way to fix this in a backwards compatible way. --- Some functions like nanmean, nanstd and others can be removed from scipy.stats because they will be available in numpy. (when scipy requires a minimum version of numpy that contains those.) > >> >> (2) The interface for the distributions in scipy.stats seems to >> explicitly be for scalar random variables, so there's no multivariate >> normals, multinomials, dirichlet, wishart, etc.. Instead the sampling >> is in numpy.random, and pdf's aren't there. > > > Two days ago PR-2726 was merged, which adds a multivariate normal > distribution. Others can be added. IIRC there has been an enhancement ticket > for wishhart somewhere and there's a Python implementation floating around > somewhere. > > Cheers, > Ralf > >> >> Has this been discussed elsewhere? No, all the discussions for scipy.stats are inn github issues (including PRs) or on the mailing list. Josef >> >> On Sat, Sep 21, 2013 at 8:03 PM, Blake Griffith >> <blake.a.griffith(a)gmail.com> wrote: >> > >> >> sparse >> >> `````` >> >> >> >> Don't emulate np.matrix behavior, drop 2-D? >> > >> > >> > What is meant by this? Emulate np.array instead? >> > >> > _______________________________________________ >> > SciPy-Dev mailing list >> > SciPy-Dev(a)scipy.org >> > http://mail.scipy.org/mailman/listinfo/scipy-dev >> > >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev(a)scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev >

1 0

stats test coverage
by josef.pktd＠gmail.com 21 Sep '13

21 Sep '13

fresh from our CI and maintenance team an excerpt 85% build/testenv/lib/python2.7/site-packages/scipy/stats/_binned_statistic.py 393122104180.9 100% build/testenv/lib/python2.7/site-packages/scipy/stats/contingency.py 272373701.0 93% build/testenv/lib/python2.7/site-packages/scipy/stats/distributions.py 7960335631282280.9 100% build/testenv/lib/python2.7/site-packages/scipy/stats/__init__.py 346131301.0 79% build/testenv/lib/python2.7/site-packages/scipy/stats/kde.py 51312599260.8 61% build/testenv/lib/python2.7/site-packages/scipy/stats/morestats.py 16155803552250.6 67% build/testenv/lib/python2.7/site-packages/scipy/stats/mstats_basic.py 20508795912880.7 66% build/testenv/lib/python2.7/site-packages/scipy/stats/mstats_extras.py 466162107550.7 100% build/testenv/lib/python2.7/site-packages/scipy/stats/mstats.py 823301.0 94% build/testenv/lib/python2.7/site-packages/scipy/stats/_multivariate.py 49312511780.9 35% build/testenv/lib/python2.7/site-packages/scipy/stats/rv.py 72176110.4 83% build/testenv/lib/python2.7/site-packages/scipy/stats/stats.py 446310328541780.8 Josef

3 2

welcome Evgeni Burovski to the scipy team
by Ralf Gommers 19 Sep '13

19 Sep '13

Hi all, On behalf of the scipy developers I'd like to welcome Evgeni Burovski as a member of the core dev team. Evgeni has (among other things) spent a lot of effort over the last months on making the statistical distributions more user-friendly and accurate: http://www.ohloh.net/p/scipy/contributors/21026013018665. I'm sure more good stuff will follow:) Cheers, Ralf

6 5

ANN: SfePy 2013.3
by Robert Cimrman 18 Sep '13

18 Sep '13

I am pleased to announce release 2013.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method. The code is based on NumPy and SciPy packages. It is distributed under the new BSD license. Home page: http://sfepy.org Downloads, mailing list, wiki: http://code.google.com/p/sfepy/ Git (source) repository, issue tracker: http://github.com/sfepy Highlights of this release -------------------------- - implementation of Mesh topology data structures in C - implementation of regions based on C Mesh (*) - MultiProblem solver for conjugate solution of subproblems - new advanced examples (vibro-acoustics, Stokes flow with slip conditions) (*) Warning: region selection syntax has been changed in a principal way, see [1]. Besides the simple renaming, all regions meant for boundary conditions or boundary/surface integrals need to have their kind set explicitly to 'facet' (or 'edge' in 2D, 'face' in 3D). [1] http://sfepy.org/doc-devel/users_guide.html#regions For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Vladimír Lukeš

1 0