Mailman 3 August 2019 - SciPy-Dev

Sprints at EuroSciPy
by Juan Luis Cano 05 Sep '19

05 Sep '19

Hi all, There are some free slots for the EuroSciPy sprints and the organizers encouraged us to propose more: https://www.euroscipy.org/2019/program.html I wonder if there are other people attending EuroSciPy interested in sprinting, what topics could we choose (general bug triaging? high priority defects? a specific sub package needing some love? some work on numba-scipy?), and if people not attending (especially core developers) would be open to participate remotely, perhaps answering questions on IRC (or Matrix, see https://riot.im/app/#/room/#freenode_#scipy:matrix.org). I have a general interest in scipy.integrate, scipy.optimize and scipy.signal but I'm not an expert in anything. What I'd like is to see a SciPy sprint at EuroSciPy to, you know, honor the name of the conference :) If there's interest, I will propose it to the organizers. Best, -- Juan Luis Cano

2 2

NumPy 1.16.5 released
by Charles R Harris 28 Aug '19

28 Aug '19

Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.16.5 has been released. This release fixes bugs reported against the 1.16.4 release and backports several enhancements from master that seem appropriate for the LTS release series that is the last to support Python 2.7. Downstream developers should use Cython >= 0.29.2 and OpenBLAS >= 3.7 to avoid wrong results on the Skylake architecture. The NumPy wheels on PyPI are built from the OpenBLAS development branch in order to avoid those problems. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.16.5/>, source archives and release notes are available from Github <https://github.com/numpy/numpy/releases/tag/v1.16.5>. *Contributors* A total of 18 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Alexander Shadchin - Allan Haldane - Bruce Merry + - Charles Harris - Colin Snyder + - Dan Allan + - Emile + - Eric Wieser - Grey Baker + - Maksim Shabunin + - Marten van Kerkwijk - Matti Picus - Peter Andreas Entschev + - Ralf Gommers - Richard Harris + - Sebastian Berg - Sergei Lebedev + - Stephan Hoyer *Pull requests merged* A total of 23 pull requests were merged for this release. - gh-13742: ENH: Add project URLs to setup.py - gh-13823: TEST, ENH: fix tests and ctypes code for PyPy - gh-13845: BUG: use npy_intp instead of int for indexing array - gh-13867: TST: Ignore DeprecationWarning during nose imports - gh-13905: BUG: Fix use-after-free in boolean indexing - gh-13933: MAINT/BUG/DOC: Fix errors in _add_newdocs - gh-13984: BUG: fix byte order reversal for datetime64[ns] - gh-13994: MAINT,BUG: Use nbytes to also catch empty descr during allocation - gh-14042: BUG: np.array cleared errors occured in PyMemoryView_FromObject - gh-14043: BUG: Fixes for Undefined Behavior Sanitizer (UBSan) errors. - gh-14044: BUG: ensure that casting to/from structured is properly checked. - gh-14045: MAINT: fix histogram*d dispatchers - gh-14046: BUG: further fixup to histogram2d dispatcher. - gh-14052: BUG: Replace contextlib.suppress for Python 2.7 - gh-14056: BUG: fix compilation of 3rd party modules with Py_LIMITED_API... - gh-14057: BUG: Fix memory leak in dtype from dict contructor - gh-14058: DOC: Document array_function at a higher level. - gh-14084: BUG, DOC: add new recfunctions to `__all__` - gh-14162: BUG: Remove stray print that causes a SystemError on python 3.7 - gh-14297: TST: Pin pytest version to 5.0.1. - gh-14322: ENH: Enable huge pages in all Linux builds - gh-14346: BUG: fix behavior of structured_to_unstructured on non-trivial... - gh-14382: REL: Prepare for the NumPy 1.16.5 release. Cheers, Charles Harris

1 0

NumPy 1.17.1 released
by Charles R Harris 27 Aug '19

27 Aug '19

Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.17.1 has been released. This release contains a number of fixes for bugs reported against NumPy 1.17.0 along with a few documentation and build improvements. The Python versions supported are 3.5-3.7, note that Python 2.7 has been dropped. Python 3.8b3 should work with the released source packages, but there are no future guarantees. Downstream developers should use Cython >= 0.29.13 for Python 3.8 support and OpenBLAS >= 3.7 to avoid wrong results on the Skylake architecture. The NumPy wheels on PyPI are built from the OpenBLAS development branch in order to avoid those problems. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.17.1/>, source archives and release notes are available from Github <https://github.com/numpy/numpy/releases/tag/v1.17.1>. *Contributors* A total of 17 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Alexander Jung + - Allan Haldane - Charles Harris - Eric Wieser - Giuseppe Cuccu + - Hiroyuki V. Yamazaki - Jérémie du Boisberranger - Kmol Yuan + - Matti Picus - Max Bolingbroke + - Maxwell Aladago + - Oleksandr Pavlyk - Peter Andreas Entschev - Sergei Lebedev - Seth Troisi + - Vladimir Pershin + - Warren Weckesser *Pull requests merged* A total of 24 pull requests were merged for this release. - gh-14156: TST: Allow fuss in testing strided/non-strided exp/log loops - gh-14157: BUG: avx2_scalef_ps must be static - gh-14158: BUG: Remove stray print that causes a SystemError on python 3.7. - gh-14159: BUG: Fix DeprecationWarning in python 3.8. - gh-14160: BLD: Add missing gcd/lcm definitions to npy_math.h - gh-14161: DOC, BUILD: cleanups and fix (again) 'build dist' - gh-14166: TST: Add 3.8-dev to travisCI testing. - gh-14194: BUG: Remove the broken clip wrapper (Backport) - gh-14198: DOC: Fix hermitian argument docs in svd. - gh-14199: MAINT: Workaround for Intel compiler bug leading to failing test - gh-14200: TST: Clean up of test_pocketfft.py - gh-14201: BUG: Make advanced indexing result on read-only subclass writeable... - gh-14236: BUG: Fixed default BitGenerator name - gh-14237: ENH: add c-imported modules for freeze analysis in np.random - gh-14296: TST: Pin pytest version to 5.0.1 - gh-14301: BUG: Fix leak in the f2py-generated module init and `PyMem_Del`... - gh-14302: BUG: Fix formatting error in exception message - gh-14307: MAINT: random: Match type of SeedSequence.pool_size to DEFAULT_POOL_SIZE. - gh-14308: BUG: Fix numpy.random bug in platform detection - gh-14309: ENH: Enable huge pages in all Linux builds - gh-14330: BUG: Fix segfault in `random.permutation(x)` when x is a string. - gh-14338: BUG: don't fail when lexsorting some empty arrays (#14228) - gh-14339: BUG: Fix misuse of .names and .fields in various places (backport... - gh-14345: BUG: fix behavior of structured_to_unstructured on non-trivial... - gh-14350: REL: Prepare 1.17.1 release Cheers, Charles Harris

1 0

Rationalisation of finite differences for minimize
by Andrew Nelson 21 Aug '19

21 Aug '19

Dear devs, In order to fully address https://github.com/scipy/scipy/issues/6026 I propose the following behaviour for optimize.minimize. For minimize methods that use a jacobian: 1) if a callable jacobian is supplied simply use that. (no change) 2) if `jac is True`, then `fun` is assumed to return function and gradient. (no change) 3) if `jac is None` or `bool(jac)` evaluates to `False`, then use forward differences with an absolute step of epsilon. This behaviour ensures back compatibility is kept for those methods that already accept an `epsilon` or `eps` keyword. 4) have an additional `finite_diff_rel_step` keyword. For further information see `optimize._numdiff.approx_derivative`. This would be changed behaviour for some methods. trust-constr already has this keyword (which is why I suggested it). This keyword would only be added to those that already have an `epsilon` or `eps` keyword. 5) if `jac in ['2-point', '3-point', 'cs']`, in this case the finite differences would not use an absolute step size, but would use either forward differences/central differences/complex step for gradient calculation. The step size would be that specified by `finite_diff_rel_step`. This would only work for those methods that now possess the `finite_diff_rel_step` keyword. Gradient calculation would be done by `_differentiable_functions.ScalarFunction` which eventually calls `_numdiff.approx_derivative`. This approach is already used by trust-constr. `ScalarFunction` caches function evaluations. 6) there are some methods that only work if points 1 or 2 are satisfied (i.e. `callable(jac) or jac is True`). These methods are: trust-krylov, trust-ncg, trust-exact, dogleg, Newton-CG (I hope I have them all). They raise an exception if a callable gradient function is not provided by the user. It's unclear to me if these methods can use a gradient calculated by finite differences. If they can, then I propose to add the ability to use finite differences with points 1, 2, 4, 5 (not 3) above. I need feedback on this point from subject experts. 7) Possibly remove `approx_jacobian` from slsqp.py. This should be do-able by `approx_derivative` instead. 8) Fix up documentation for these options. Enhancements -------------------- a) the better tested `approx_derivative` is now used instead of `approx_fprime`. b) relative step sizes are much preferred to absolute step size as the truncation and round off errors are balanced. c) there is hopefully a more consistent interface to jac specification d) gradients calculated by central-differences are more accurate than 2-point calculation, it's nice to have the option to use that (at the expense of function evaluations). e) it should be possible to reduce the number of function evaluations by a small amount because ScalarFunction caches values. f) Bounds can be supplied to ScalarFunction/approx_derivative. If a parameter value is at a boundary, then either of those two functions should not take a step outside the boundary whilst evaluating the gradient. Drawbacks --------------- g) Technically this meets back compatibility according to documentation. However, some users may be using jac=`2-point`, which at the moment may be converted to an absolute step+forward difference calculation. The documentation doesn't say that an absolute step is being used internally. The proposed behaviour would move it to a relative step. Thus users would get a slightly changed implementation. Backwards compatibility is only assured by setting `jac=False` or `jac=None`. h) minimize is a complex beast with lots of interlocking behaviour, I don't think this will increase complexity by an appreciable amount, but there's definitely a possibility someone is upset because behaviour is changed slightly. I'd appreciate feedback on this, especially why gradient functions are required (point 6) for some methods. Hopefully it's not too controversial.

2 1

NaN insensitive circular statistics
by Burrell, Angeline 20 Aug '19

20 Aug '19

Hi Christoph, Thank you for your response. The circ methods (circmean, circstd, and circvar) do not currently use the nan_policy keyword, and I wasn't able to find any pull requests addressing this on the GitHub page. I propose to add the nan_policy keyword option to circmean, circstd, and circvar. This keyword would then be passed to _circfuncs_commons, which currently tests the input samples and returns the samples as an array of angles in radians so that the circular statistics can be calculated. This makes it a sensible place to identify or omit any NaN (if desired). This way, if nan_policy is 'propagate', the circ methods would behave as they currently do, 'raise' would case _circfuncs_commons to throw an error, and 'omit' would remove all the NaN values before testing the sample size. Cheers, Angeline

1 0

Pythran 0.9.3 - Hañv
by Serge Guelton 19 Aug '19

19 Aug '19

Hi folks, I just released version 0.9.3 of the Pythran package, Short reminder: Pythran is an ahead-of-time compiler for scientific Python, with a focus on high-level numerical kernels, parallelism and vectorisation. Here is a simple kernel example, with a pythran annotation. Note that that kernel is still Python-compatible (from https://stackoverflow.com/questions/57199248/) : import numpy as np #pythran export col_sum(int[:,:] or float[:,:], int[:]) def col_sum(data, idx): return data.T[idx].sum(0) The Pythran package is available on PyPI, Github and Conda https://pypi.org/project/pythran/ https://anaconda.org/conda-forge/pythran https://github.com/serge-sans-paille/pythran The interested reader can have a look to the changelog for details https://pythran.readthedocs.io/en/latest/Changelog.html Long story short: bug fixes and better 32bit arch support. Plus (Thanks to Miro Hrončok), pythran is now available on Fedora \o/ Huge thanks to all contributors and bug reporters: Jean Laroche Yann Diorcet DWesl Miro Hrončok Piotr Bartmann Jochen Schröder Sylwester Arabas Marti Bosch rorroiga Pierre Augier Anubhab Haldar nbecker

1 0

Season of Docs - welcome Anne, Maja, Brandon
by Ralf Gommers 19 Aug '19

19 Aug '19

Hi all, Google has announced the Season of Docs participants for this year [1]. We had a lot of excellent candidates and had to make some hard choices. We applied for extra slots, but unfortunately didn't win the lottery for those; we got one slot for NumPy and one for SciPy. We chose the projects of Anne for NumPy and Maja for SciPy: Anne Bonner, "Making "The Basics" a Little More Basic: Improving the Introductory NumPy Sections" [2] Maja Gwozdz, "User-oriented documentation and thorough restructuring" [3] That's not all though. There was some space left in the budget of the NumPy BIDS grant, and Stéfan has reserved that so we can accept more writers and provide them the same mentoring and funding as they would have gotten through GSoD. We could only start the conversations about that once Google made its decisions, so a further announcement will follow. However, we already have one extra project confirmed, from Brandon: Brandon David, "Improve the documentation of scipy.stats" (project details to be published). I will send out a poll to find a good time for everyone for a kickoff call. Our intent is to build a documentation team with multiple writers and mentors interacting and able to help each other out. And all of this will also interact with the numpy.org website redesign and the people putting energy into that:) I'm very happy to welcome Anne, Maja and Brandon! Cheers, Ralf [1] https://developers.google.com/season-of-docs/docs/participants/ [2] https://developers.google.com/season-of-docs/docs/participants/project-numpy [3] https://developers.google.com/season-of-docs/docs/participants/project-scipy

6 9

callable jacobians for CG, BFGS, Newton-CG, L-BFGS-B, TNC, SLSQP, dogleg, trust-ncg, trust-krylov, trust-exact and trust-constr.
by Andrew Nelson 19 Aug '19

19 Aug '19

Clarification question to close GH9042. Which of the minimizer methods (CG, BFGS, Newton-CG, L-BFGS-B, TNC, SLSQP, dogleg, trust-ncg, trust-krylov,trust-exact and trust-constr): - require a *callable* jacobian? - are satisfied with a jacobian estimated by finite differences? -- _____________________________________ Dr. Andrew Nelson _____________________________________

1 0

NaN insensitive circular statistics
by Christoph Baumgarten 18 Aug '19

18 Aug '19

Hi, thanks for your proposal. The Methods in stats often have a Keyword nan_policy that allows to omit NaNs or to raise an error. There are some inconsitencies though (you can find the issues on Github). What kind of treatment do you propose for circular mean / std? Christoph 1. NaN insensitive circular statistics (Burrell, Angeline) > 2. Re: triage team (Ralf Gommers) > 3. Re: Matrix identification methods (Joseph Fox-Rabinovitz) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 14 Aug 2019 17:49:49 +0000 > From: "Burrell, Angeline" <angeline.burrell(a)nrl.navy.mil> > To: "scipy-dev(a)python.org" <scipy-dev(a)python.org> > Subject: [SciPy-Dev] NaN insensitive circular statistics > Message-ID: <6DEB95FA-2CCA-4C32-9CD5-D77585D92375(a)contoso.com> > Content-Type: text/plain; charset="utf-8" > > Second attempt for a response for a new feature to scipy.stats, since I > was alerted that my first email was going into spam folders. I do hope to > hear back (positively or negatively) from the community. > > I have NaN insensitive versions of the SciPy circular mean and standard > deviation routines that I would like to contribute to the scipy.stats > subpackage. Since the hacking guidelines recommend discussing new > contributions on this mailing list, I'd like to get the communal go-ahead > before proceeding to integrate the routines and unit tests into scipy. For > consistency, I would also add a NaN insensitive version of the circular > variance routine. > > Cheers, > Angeline > > ----------------------------------------------------- > Dr. Angeline G. Burrell [she/her/hers] > Research Physicist, Bldg. 209 > Naval Research Laboratory (NRL) > 4555 Overlook Ave SW > Washington, DC 20375 > (P) 202-404-4065 > ----------------------------------------------------- > >

1 0

Cholesky decomposition with full pivot
by Joseph Gama 17 Aug '19

17 Aug '19

Hello, I recently added wrappers for Lapack pstrf and psft2 with the purpose of adding full pivot capabilities to the cholesky decomposition because it allows the use of positive semidefinite matrices. R has this feature and it is useful in many areas. All feedback is welcome! :)Tuxcell P.S.: my proposed changes ("is_hermitian" is currently in PR Matrix identification methods and test by tuxcell · Pull Request #10639 · scipy/scipy) | | | | | | | | | | | Matrix identification methods and test by tuxcell · Pull Request #10639 ... Adds several methods for checking matrices It implements these checks: is_hermitian, is_symmetric, is_skew_symm... | | | def _cholesky(a, lower=False, overwrite_a=False, clean=True, check_finite=True, full_pivot=False, pivot_tol=-1): """Common code for cholesky() and cho_factor().""" a1 = asarray_chkfinite(a) if check_finite else asarray(a) a1 = atleast_2d(a1) # Dimension check if a1.ndim != 2: raise ValueError('Input array needs to be 2 dimensional but received ' 'a {}d-array.'.format(a1.ndim)) # Squareness check if a1.shape[0] != a1.shape[1]: raise ValueError('Input array is expected to be square but has ' 'the shape: {}.'.format(a1.shape)) # Quick return for square empty array if a1.size == 0: return a1.copy(), lower if not is_hermitian(): raise LinAlgError("Expected symmetric or hermitian matrix") overwrite_a = overwrite_a or _datacopied(a1, a) # if the pivot flag is false, return the result if not full_pivot: potrf, = get_lapack_funcs(('potrf',), (a1,)) c, info = potrf(a1, lower=lower, overwrite_a=overwrite_a, clean=clean) if info > 0: raise LinAlgError("%d-th leading minor of the array is not positive " "definite" % info) if info < 0: raise ValueError('LAPACK reported an illegal value in {}-th argument' 'on entry to "POTRF".'.format(-info)) return c, lower else: # if the pivot flag is true, return the result plus rank and pivot pstrf, = get_lapack_funcs(('pstrf',), (a1,)) c, pivot, rank, info=pstrf(a1, lower=False, overwrite_a=False, clean=True, tol=pivot_tol) if info > 0: if rank == 0: raise LinAlgError("%d-th leading minor of the array is not positive " "semidefinite" % info) else: raise LinAlgError("The array is rank deficient with " "computed rank %d" % info) if info < 0: raise ValueError('LAPACK reported an illegal value in {}-th argument' 'on entry to "PSTRF".'.format(-info)) return c, lower, rank, pivot def cholesky(a, lower=False, overwrite_a=False, check_finite=True, full_pivot=False, pivot_tol=-1): """ Compute the Cholesky decomposition of a matrix. Returns the Cholesky decomposition, :math:`A = L L^*` or :math:`A = U^* U` of a Hermitian positive-definite matrix A. Parameters ---------- a : (M, M) array_like Matrix to be decomposed lower : bool, optional Whether to compute the upper or lower triangular Cholesky factorization. Default is upper-triangular. overwrite_a : bool, optional Whether to overwrite data in `a` (may improve performance). check_finite : bool, optional Whether to check that the input matrix contains only finite numbers. Disabling may give a performance gain, but may result in problems (crashes, non-termination) if the inputs do contain infinities or NaNs. full_pivot : bool, optional Whether to use full pivot or not pivot_tol : float, optional Tolerance for the pivot, if < 0 then tolerance = N*U*MAX( A(M,M) ) Returns ------- c : (M, M) ndarray Upper- or lower-triangular Cholesky factor of `a`. Raises ------ LinAlgError : if decomposition fails. Examples -------- >>> from scipy.linalg import cholesky >>> a = np.array([[1,-2j],[2j,5]]) >>> L = cholesky(a, lower=True) >>> L array([[ 1.+0.j, 0.+0.j], [ 0.+2.j, 1.+0.j]]) >>> L @ L.T.conj() array([[ 1.+0.j, 0.-2.j], [ 0.+2.j, 5.+0.j]]) """ if not full_pivot: c,lower = _cholesky(a, lower=lower, overwrite_a=overwrite_a, clean=True, check_finite=check_finite) return c else: c,lower, rank_bn, piv = _cholesky(a, lower=lower, overwrite_a=overwrite_a, clean=True, check_finite=check_finite, full_pivot=full_pivot, pivot_tol=pivot_tol) return c, rank_bn, piv

1 0