Mailman 3 June 2017 - SciPy-Dev

Are you using Bento to build scipy?
by CJ Carey 16 Jul '18

16 Jul '18

With the 1.0 release coming up, I figured this is a good time to take the pulse of the community regarding Bento. The project seems to have been abandoned before reaching stability, and it seems that the core problems it was trying to address have been at least partially fixed by the more official build systems. If you use Bento and would like to keep using it post 1.0, or if my previous sentence is wrong, please let me know!

10 15

Stochastic branch and bound optimization algorithm
by Benoit Rosa 26 Jul '17

26 Jul '17

Hi all, I recently implemented a stochastic branch and bound algorithm in python for one of my projects (namely, 3D registration without an initial guess, requiring global optimization over the SE3 group). After trying basinhopping without much success, I went for a modified stochastic branch and bound algorithm, described in those two papers (disclaimer: I'm an author in one of those) [1] C. Papazov and D. Burschka, Stochastic global optimization for robust point set registration, Comput. Vis. Image Understand. 115 (12) (2011) 1598-1609 [2] C. Gruijthuijsen, B. Rosa, P.T. Tran, J. Vander Sloten, E. Vander Poorten, D. Reynaerts, An automatic registration method for radiation-free catheter navigation guidance, J. Med. Robot Res. 1 (03) (2016), 1640009 Since I wanted to compare with other optimization algorithms, I implemented things by modifying the basinhopping file, and as such my implementation (git commit here: https://github.com/benoitrosa/scipy/commit/49a2c23b74b69dc4250e20e21db75bd0… ) is fully compatible with Scipy already. I have added a bit of documentation in the file too. If you want an idea, on my project (for which I can't share the code now), this algorithm finds the global optimum over the S03 space (i.e. rotations) in 4.5 seconds on average, where basinhopping takes more than 15 seconds, and doesn't necessarily converge to the correct solution. More about the algorithm itself: it's a common branch and bound algorithm (i.e. recursive traversal of a tree and bisecting of leaves to find an optimum), with two additions: 1 - The tree is traversed stochastically. This is governed by a temperature parameter, much like simulated annealing algorithms. At each node, there is a probability of going towards a "less promising" (understand: cost function higher) branch, this probability being governed by the temperature parameter. In the beginning, "bad" branches will be selected, ensuring an exploration of large portions of the space. As the number of iterations increases the temperature decreases, and the algorithms goes more towards the most promising branches/leaves, getting higher resolution in that part of the space. 2 - When creating a new leaf and evaluating the cost function there, the value is propagated down the tree, until reaching the root or a node with a lower cost. This ensures a smart traversal of the tree. Moreover, it also makes sure that at any time, the root node has the lowest cost, making it easy to query for the best solution when it is interrupted. Another advantage of this algorithm is that parameter tuning is pretty easy. It only needs the bounds of the search space (no initial guess), a sampling function (default is uniform) and a cost function (obviously). Additional parameters are the number of iterations, and the start and stop temperatures (start should be high to accept many "bad solutions" in the beginning, and stop should be tending to zero). If that algorithm is of interest to the SciPy community, I'd be happy to clean up the code and make a pull request ! Best, Benoit

3 3

Refactoring ndimage
by Jaime Fernández del Río 06 Jul '17

06 Jul '17

Hi all, I have started sending PRs for ndimage. It is my intention to keep at it through the summer, and would like to provide some context on what I am trying to achieve. A couple of years ago I mentored a GSoC to port ndimage to Cython that didn't do much progress. If nothing else, I think I did learn quite a bit about ndimage and what makes it hard to maintain. And I think one of the big ones is lack of encapsulation in the C code: ndimage defines four "classes" in ni_support.h <https://github.com/scipy/scipy/blob/master/scipy/ndimage/src/ni_support.h> that get used throughout, namely NI_Iterator, NI_LineBuffer, NI_FilterIterator and NI_CoordinateList. Unfortunately they are not very self contained, and e.g. to instantiate a NI_FilterIterator you typically have to: - declare a NI_FilterIterator variable, - declare two NI_Iterator variables, one for the input array, another for the output, - declare a npy_intp pointer of offsets and assign NULL to it, - call NI_InitFilterOffsets to initialize the offsets pointer, - call NI_InitFilterIterator to initialize the filter iterator, - call NI_InitPointIterator twice, once for the input, another for the output, - after each iteration call NI_FILTER_NEXT2 to advance all three iterators, - after iteration is finished, call free on the pointer of offsets. There is no good reason why we couldn't refactor this to being more like: - call NI_FilterIterator_New and assign its return to a NI_FilterIterator pointer, - after each iteration call NI_FilterIterator_Next to advance all involved pointers, - after iteration is finished, call NI_FilterIterator_Delete to release any memory. Proper encapsulation would have many benefits: - high level functions would not be cluttered with boilerplate, and would be easier to understand and follow, - chances for memory leaks and the like would be minimized, - we could wrap those "classes" in Python and test them thoroughly, - it would make the transition to Cython for the higher level functions, much simpler. As an example, the lowest hanging fruit in this would be #7527 <https://github.com/scipy/scipy/pull/7527>. So open source wise this is what I would like to spend my summer on. Any feedback is welcome, but I would especially like to hear about: - thoughts on the overall plan, - reviewer availability: I would like to make this as incremental as possible, but that means many smaller interdependent PRs, which require reviewer availability, - if anyone wants to join in the fun, I'm more than happy to mentor! Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

3 5

ANN: SciPy 0.19.1 release
by Ralf Gommers 27 Jun '17

27 Jun '17

On behalf of the Scipy development team I am pleased to announce the availability of Scipy 0.19.1. This is a bugfix-only release, no new features are included. This release requires Python 2.7 or 3.4-3.6 and NumPy 1.8.2 or greater. Source tarballs and release notes can be found at https://github.com/scipy/scipy/releases/tag/v0.19.1. OS X and Linux wheels are available from PyPI: https://pypi.python.org/pypi/scipy/0.19.1 Thanks to everyone who contributed! Cheers, Ralf -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 ========================== SciPy 0.19.1 Release Notes ========================== SciPy 0.19.1 is a bug-fix release with no new features compared to 0.19.0. The most important change is a fix for a severe memory leak in ``integrate.quad``. Authors ======= * Evgeni Burovski * Patrick Callier + * Yu Feng * Ralf Gommers * Ilhan Polat * Eric Quintero * Scott Sievert * Pauli Virtanen * Warren Weckesser A total of 9 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 0.19.1 - ------------------------ - - `#7214 <https://github.com/scipy/scipy/issues/7214>`__: Memory use in integrate.quad in scipy-0.19.0 - - `#7258 <https://github.com/scipy/scipy/issues/7258>`__: ``linalg.matrix_balance`` gives wrong transformation matrix - - `#7262 <https://github.com/scipy/scipy/issues/7262>`__: Segfault in daily testing - - `#7273 <https://github.com/scipy/scipy/issues/7273>`__: ``scipy.interpolate._bspl.evaluate_spline`` gets wrong type - - `#7335 <https://github.com/scipy/scipy/issues/7335>`__: scipy.signal.dlti(A,B,C,D).freqresp() fails Pull requests for 0.19.1 - ------------------------ - - `#7211 <https://github.com/scipy/scipy/pull/7211>`__: BUG: convolve may yield inconsistent dtypes with method changed - - `#7216 <https://github.com/scipy/scipy/pull/7216>`__: BUG: integrate: fix refcounting bug in quad() - - `#7229 <https://github.com/scipy/scipy/pull/7229>`__: MAINT: special: Rewrite a test of wrightomega - - `#7261 <https://github.com/scipy/scipy/pull/7261>`__: FIX: Corrected the transformation matrix permutation - - `#7265 <https://github.com/scipy/scipy/pull/7265>`__: BUG: Fix broken axis handling in spectral functions - - `#7266 <https://github.com/scipy/scipy/pull/7266>`__: FIX 7262: ckdtree crashes in query_knn. - - `#7279 <https://github.com/scipy/scipy/pull/7279>`__: Upcast half- and single-precision floats to doubles in BSpline... - - `#7336 <https://github.com/scipy/scipy/pull/7336>`__: BUG: Fix signal.dfreqresp for StateSpace systems - - `#7419 <https://github.com/scipy/scipy/pull/7419>`__: Fix several issues in ``sparse.load_npz``, ``save_npz`` - - `#7420 <https://github.com/scipy/scipy/pull/7420>`__: BUG: stats: allow integers as kappa4 shape parameters Checksums ========= MD5 ~~~ 72415e8da753eea97eb9820602931cb5 scipy-0.19.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl e0022540df2735eb0475071b266d5d71 scipy-0.19.1-cp27-cp27m-manylinux1_i686.whl f513eb4ea2086de169a502df7efb91c7 scipy-0.19.1-cp27-cp27m-manylinux1_x86_64.whl 906c3c59209d6249b5d8ce14cfa01382 scipy-0.19.1-cp27-cp27mu-manylinux1_i686.whl afbf8ffb4a4fe7c18e34cb8a313c18ee scipy-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl 5ba945b3404644244ab469883a1723f0 scipy-0.19.1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 9c02cdd79e4ffadddcce7b2212039816 scipy-0.19.1-cp34-cp34m-manylinux1_i686.whl 79c0ba3618466614744de9a2f5362bbc scipy-0.19.1-cp34-cp34m-manylinux1_x86_64.whl 602a741a54190e16698ff8b2fe9fd27c scipy-0.19.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl d6c2ecadd4df36eb61870227fae42d3a scipy-0.19.1-cp35-cp35m-manylinux1_i686.whl e7167c0a9cf270f89437e2fd09731636 scipy-0.19.1-cp35-cp35m-manylinux1_x86_64.whl fc2e4679e83590ff19c1a5c5b1aa4786 scipy-0.19.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 458615e9a56429a72038531dd5dcb3cb scipy-0.19.1-cp36-cp36m-manylinux1_i686.whl 65b1667ac56861da4cbc609960ed735b scipy-0.19.1-cp36-cp36m-manylinux1_x86_64.whl b704ebe9a28b8fe83d9f238d40031266 scipy-0.19.1.tar.gz cad6bac0638b176f72c00fe81ed54d19 scipy-0.19.1.tar.xz eb69261e5026ef2f3b9ae827caa7e5b8 scipy-0.19.1.zip SHA256 ~~~~~~ 1e8fedf602859b541ebae78667ccfc53158edef58d9ee19ee659309004565952 scipy-0.19.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 023ee29faa76c184a607e21076f097dc32f3abba7c71ece374588f95920aa993 scipy-0.19.1-cp27-cp27m-manylinux1_i686.whl 2a26d06a642e3c9107ca06df125f5dc5507abe2b87fd7865415d03ab654b0b43 scipy-0.19.1-cp27-cp27m-manylinux1_x86_64.whl b3e97be2cd9f052d984fc5ba2d441897971b744c64d658617944c47bc366f8ff scipy-0.19.1-cp27-cp27mu-manylinux1_i686.whl 78713101b32af384c564837fd7ae665a2a72cb6d49edbd8f32148d74724a65a8 scipy-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl 2c9e87d556b83a8e11de7a064856c3997bbaff7f9cf62bf63ff0623751549e4c scipy-0.19.1-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl b87a923ed390ba0aafcbbcc6f0e023d495f19d2bd22ae59bdef3b0aec6485b39 scipy-0.19.1-cp34-cp34m-manylinux1_i686.whl 2dc9bcfaa585d9d941fb1add0d160556fe8587c3800264a77643695565a2d279 scipy-0.19.1-cp34-cp34m-manylinux1_x86_64.whl 93825cc80c638d901099f657dfff852ad2421beb51cb7d1d3f91157741ebe287 scipy-0.19.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 4118647273c907ed984da52b71fa2bbb30229f1820fb79b1ed087c8bc9a20cca scipy-0.19.1-cp35-cp35m-manylinux1_i686.whl 4c26ef44e8bb2cd2aef11c60d163caa04670d6f42996789b209526677310ded2 scipy-0.19.1-cp35-cp35m-manylinux1_x86_64.whl 2a3b6ceadbb58d8b8d4a329f8219f9e6f17757ec6c85baf03987bbd2b728c263 scipy-0.19.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 909be30bdd1d8cd57609760ea72a1499543169ed6ea5c66fad50946582682cf8 scipy-0.19.1-cp36-cp36m-manylinux1_i686.whl 4de84e3870228e8afd55a6e63e762aa7c9d1e3bd9a9ef5ab716835a69c77d257 scipy-0.19.1-cp36-cp36m-manylinux1_x86_64.whl 9147f7b8b5ed4b80fefe82057a6d0d4bf3a50375a1d573407f84686d3b82c3bb scipy-0.19.1.tar.gz 0dca04c4860afdcb066cab4fd520fcffa8c85e9a7b5aa37a445308e899d728b3 scipy-0.19.1.tar.xz 2cc1baed85f7f702d958ad19e5d11c9dad917e981d3d0c299a7f0fd15027516a scipy-0.19.1.zip -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJZTTG2AAoJEO2+o3i/Gl69V8gH/3S199hh1Fcl0a05VOCgt7aZ zRKvdZhYkqGvWoWX+YwJBaZn0aXThJfIDE2lAGt4S9PyxObBKzsujI8OLIZF4zU5 PtAARrooOqDEH1CC39k2gscF9GBhxAlxwntM2Hd8h+4xsklAhFVxn4UT0M8igBoX txpaIqghzBJnenkMq6lqrj7vhupuulz0zrMnJU4LetpdSxX1lb5sGvWmZDTKd/nr xmthQ4TmHPbf9jAYCtLI4V6OUGFGLQ+d7IqiYvU+DVKNZFEgyPMnvN9QvSdu6diR JBEIBcijxM0BqjPwRoavCQjCHk37kR/G3UsgcEyHO2tr5zuOXogjjBo3Bei8jnI= =lFXD -----END PGP SIGNATURE-----

2 1

Possibly warnings creating errors in development test runs
by Sebastian Berg 27 Jun '17

27 Jun '17

Hi all, we just merged a PR [1], which (re)enables having warnings cause errors during testing (only in development mode of course). Since warnings (especially warnings such as about NaNs) can be system dependent, it is not unlikely that you may experience new errors during testing. These are easy to get rid of: For NaN warnings, you usually should add the numpy errstate context such as: ``` with np.errstate(invalid="ignore") ``` For most other warnings, add the now new `suppress_warnings` context manager (please do not use "ignore" warning filters, they can create issues), such as: ``` from scipy._lib._numpy_compat import suppress_warnings def test(): with suppress_warnings() as sup: sup.filter(UserWarning, "\nThe coefficients of the spline") ``` The `suppress_warnings` context also supports use as a decorator in case there are mass of similar suppressions needed in some file. You can also use `r = sup.record(...)` and check `len(r) == 1` or an `assert_warns` context if it would be good to test that the warning occurs. I hope this does not cause a lot of troubles, probably it is only a few occurrences, if any, but the new context may be good to know about in any case. Regards, Sebastian [1] https://github.com/scipy/scipy/pull/7525

2 1

Weird segfault in call to PyArray_SIZE
by Jaime Fernández del Río 27 Jun '17

27 Jun '17

While working in scipy.ndimage I tried to replace the following code, see here <https://github.com/scipy/scipy/blob/master/scipy/ndimage/src/ni_support.c#L…> for the source: npy_intp max_lines; int ii; for (ii = 0; ii < array->nd; ii++) max_lines *= array->dimensions[ii]; with the in theory equivalent: max_lines = PyArray_SIZE(array); Oddly enough, the result is a segfault as soon as PyArray_SIZE is called, e.g. by running the ndimage tests. Both numpy and scipy are the latest development versions, built by myself on OSX. Unfortunately I don't have Python and friends built with debug symbols, so the best I have been able to figure out, through printf debugging, is that the segfault happens before PyArray_MultiplyList is entered, which made me suspicious of it being related to import_array() not being called, but that doesn't seem to be the case. Any thoughts on what may be causing this? Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

4 5

PyRSB: Python interface to librsb sparse matrices library
by Michele Martone 26 Jun '17

26 Jun '17

Hi. I'm the author of the high performance multithreaded sparse matrix library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/ I'm *not* a user of SciPy/NumPy/Python, but using Cython I have written a proof-of-concept interface to librsb, named `PyRSB': https://github.com/michelemartone/pyrsb PyRSB is in a prototypal state; e.g. still lacks good error handling. Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. Advantages over csr_matrix are in fast multithreaded multiplication of huge sparse matrices. Intended application area is iterative solution of linear systems; particularly fast if with symmetric matrices and many rhs. With this email I am looking for prospective: - users/testers - developers (any interest to collaborate/adopt/include the project?) Looking forward for your feedback, Michele

4 6

moved 0.19.1 tag
by Ralf Gommers 22 Jun '17

22 Jun '17

Hi, Unfortunately I had to move the v0.19.1 tag by one commit, a 32-bit test error had snuck in (we don't have that in our regular TravisCI runs, to be fixed). If you have pulled from the repo in the last 24 hours or so and have a local v0.19.1 tag, do: git tag -d v0.19.1 git fetch upstream --tags to fix possible issues. Cheers, Ralf

4 8

preparing 0.19.1 release
by Ralf Gommers 18 Jun '17

18 Jun '17

Hi all, The 0.19.1 release is long overdue, so I'm aiming to get it out during the coming week. In particular the integrate.quad memory leak (gh-7214) is urgent. The PRs currently labelled with `backport-candidate` [1] are planned to go in. Backports for all but the two unmerged PRs are in https://github.com/scipy/scipy/pull/7502. If there's other issues/fixes that need to go in, please point them out. The one thing that needs a decision is the behavior for float32 input to interpolation routines: https://github.com/scipy/scipy/issues/7273. Opinions very welcome. Ralf [1] https://github.com/scipy/scipy/pulls?utf8=%E2%9C%93&q=is%3Apr%20label%3Abac…

1 0

How to preview changes to a doc file?
by Paul van Mulbregt 08 Jun '17

08 Jun '17

What is the recommended way to preview any documentation changes, such as after modifying restructured text (rst) files? After adding some additional math to an rst file, I’d like to see the result of the change, but haven’t found a good way to do so. [I’m using a Mac, with Python 3.6.] * Attempts to build the scipy doc have not succeeded for me ("setup.py build_sphinx" generates lots of dvipng errors; perhaps building the doc requires Py2.7???) * The Mac editor Atom has some rst preview packages, but I haven’t found one that can handle the \begin{eqnarray*} constructs in the scipy rst files. * Stand-alone rst-viewers which then opened in a browser didn’t like the math either. * One online viewer http://rst.ninjs.org <http://rst.ninjs.org/> did render the math in a fashion, after I had removed the ":nowrap:", and converted the backtick :math:`XXX` to stand-alone math. But it was a poor rendering when compared to the actual scipy html. How do you preview doc changes? Thanks, Paul

2 2