Mailman 3 August 2018 - SciPy-Dev

SciPy 1.0 paper writing proposal
by Ralf Gommers Jan. 29, 2019

Jan. 29, 2019

Hi all, TL;DR, let's write the long journal paper on SciPy that we've wanted for a while, let's form a small committee to coordinate, and get it out the door in 2-3 months. Motivation --------------- (credits for most of this text: Evgeni) Many scipy contributors' day jobs are in academia. Bibliometry -- papers in refereed journals and citations of papers by other papers -- is one of the main performance indicators in most academic establishments. Since we do not generate papers, scipy contributions are all but invisible for the purposes of a contributor's annual report. Of course, details vary wildly; in many cases a contributor manages to balance their time, or to argue common sense with their superiors, or get an approval for scipy work, or just ignores the issue altogether -- but sooner or later there is a form to be filled or boxes to be checked, and scipy contributions simply do not fit in. A peer-reviewed journal paper on scipy will help contributors get the academic credit they deserve. We can write *the* paper for SciPy 1.0, with overall project structure, goals, etc., and for specific features/modules a focus on say the last 3 years. History ---------- For SciPy 1.0 we had three targets on the publicity/credits front: an interesting release announcement, interesting blogs/stories (NumFOCUS blog, Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in the end, the rest was successful. [1] is a previous announcement on this list about writing (a) paper(s) on SciPy. We wanted both "short papers" to cover one or two releases (target journal JOSS) and a full paper as the authoritative reference for SciPy. We had an earlier attempt for a "short paper", it's mostly written but has stalled (see [2]). We ran out of steam on that one. To avoid that this time around, it would be good to have a clear public plan, target dates, and a small committee rather than one person to drive things forward. Proposal ------------ Here's a proposal for all aspects of this exercise that I can think about now. Some parts stolen from the AstroPy paper [3] (because their process worked quite well). Form a small coordination committee of 3-5 people that set up the paper structure, move things along when parts stall, propose/take decisions as needed, invite co-authors, and organise paper submission/rework. Paper writing to be done by whoever volunteers for a section, not just the coordination committee. First outline/structure to be done by committee, which then asks for review of structure and volunteers for section writing. Scope: a 6-10 page paper, covering history, package scope and structure, community/organisational aspects, key features and recent enhancements per module, and roadmap. Authorship: anyone who made a substantial contribution in the history of the project. Here "substantial" is interpreted as anything beyond a one-line doc fix. Rationale: better to be too inclusive than exclusive. Sign-up via a web form, we send the link to that form to all email addresses in the commit history till v1.0. Author order (details tbd by committee): 1. The SciPy Developers 2. Maintainers, paper writers, other key contributors - in order of contribution level 3. All other authors - alphabetically ordered Submission target: mid-April, to either PeerJ Computer Science or Journal of Open Research Software (tbd by committee). Comments? Volunteers for committee? References ---------------- [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html [2] https://github.com/scipy/scipy-articles/pull/4 [3] https://github.com/astropy/astropy-v2.0-paper Cheers, Ralf

12 19

Re: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions
by Nikolay Mayorov Sept. 17, 2018

Sept. 17, 2018

Hi, Aditya! Glad that you set up the blog and good job on setting up the documentation build as well. Curious, what is this blogging platform like? How do you create posts in it? As for your first post: while not strictly necessary I think it would be nice to see a more thorough introductory post with a brief overview, motivation and/or an example. Do you want to work on it? Best, Nikolay

4 15

General discussion on parallelisation
by Andrew Nelson Sept. 8, 2018

Sept. 8, 2018

As part of https://github.com/scipy/scipy/pull/8259 I'm proposing that a `workers` keyword is added to optimize.differential_evolution to parallelise some computation. The proposal is that: 1. the workers keyword accepts either an integer or an object with a map-like method. 2. If an integer is supplied then the parallelisation is taken care of by scipy (more on that later), with -1 signifying that all processors are to be used. 3. If an object with a map-like method is supplied, e.g. `multiprocessing.Pool.map`, `mpi4py.futures.MPIPoolExecutor.map`, etc, then the parallelisation is taken care of by that object. This allows the user to specify the parallelisation configuration for their problem. 4. If workers=1, then computation will be done by the builtin `map` function. Now we come to the under the hood part. I've written something called PoolWrapper ( https://github.com/andyfaff/scipy/blob/b14bb513c0ffb9807a67663d39b9ab399375…) which wraps `multiprocessing.Pool` to achieve the behaviour outlined above. It can be used as a context manager, or the user of the object can decide when to close the resources opened by PoolWrapper. I've looked at using joblib instead of PoolWrapper and it seems useful but it doesn't have a couple of bits of functionality that are needed for this specific problem: viz: 5. joblib.Parallel doesn't have a map method (desirable to allow 3) so a small wrapper would have to be created anyway. 6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the Parallel object is `__call__`ed. This leads to significant overhead. One can use the Parallel object with a context manager, which allows reuse of the Pool, but I don't think that's do-able in the context of using the DifferentialEvolutionSolver (DES) object as an iterator: >>> solver = DifferentialEvolutionSolver(func, bounds) >>> # use DES object as an iterator >>> for it in solver: ... res = next(solver) >>> print(res) >>> # use the DES.solve method >>> res = solver.solve() Whilst the DES object is not currently public (it's called by the differential_evolution function) it would be nice to expose it in the future, and people will want to use both approaches. Unfortunately with the first approach if we used joblib.Parallel we'd have to use Parallel.__call__ in DES.next() which has the overhead of creating/destroying Pools. For efficient use of resources the Pool should persist for the lifetime of the DES object. I also looked into `concurrent.futures.ProcessPoolExecutor`, but it's not available for Python 2.7. The purpose of this email is to elicit feedback for developing parallelisation strategy for scipy - what does the public interface look like, what does scipy do under the hood? Under the hood I think a mixture of PoolWrapper and joblib.Parallel could be used (with scipy vendoring joblib). A. -- _____________________________________ Dr. Andrew Nelson _____________________________________

5 18

1D parabolic PDE solver
by Saixoz Sept. 7, 2018

Sept. 7, 2018

Dear all, A recent task that I had to perform involved rewriting a large number of scripts and files from MATLAB to Python. In the process I noticed that Scipy lacks a PDE solver. When I tried to find alternatives, the options found were all unusable for various reasons. I wrote a solver which solves slab symmetric problems with a similar PDE form to that which MATLAB’s pdepe solves, namely c(x,u,t,du/dx)(du/dt) = (d/dx)(f(x,u,t,du/dx) + s(x,u,t,du/dx) using the method of lines approach, finite differences, and the BDF method provided by solve_ivp. I was wondering whether there would be some interest in adding the solver under scipy.integrate, to begin building a part of a collection. I feel that for very simple problems such as the ones that my solver can solve, users shouldn’t need to turn to overly complex packages. If there is interest, I would need to clean up the code and add tests and stability checks to it, help/advice would be appreciated. I believe BDF already has stability in the time grid, so I would only need to check stability in space grid? Can someone confirm? I’d also like to move a large chunk of the code to Cython since I wrote it in native Python and used Numba to speed it up, which still makes it rather slow. I’ve worked in C and Python before but never in Cython, could someone provide me with a link to get started? As far as legal issues go, there should be no concern – I explicitly avoided looking at and using the MATLAB source code as a reference since I wanted to leave publishing the solver under Scipy an option. I did use their solver to check some of the solutions that my solver produced though. The solver was written while working for the ETH Zurich, where my supervisor responded very enthusiastically to the possibility of providing the solver to the public under Scipy, so there should be no issues there either. All the best, Nicolas

2 1

Cython code generation with Jinja2 templates
by Mark Alexander Mikofski Aug. 31, 2018

Aug. 31, 2018

Hi All, I would like to use Jinja2 templates to generate the Cython pyx and pxd files necessary for the Cython Optimize Zeros API proposed in #8431, maybe in a follow up PR https://github.com/scipy/scipy/pull/8431 Does anyone have any experience with code generation in SciPy? Is it best to generate the files offline and then commit them to the repository, or is it better to add some scripting to either setup.py or to the CI yaml config files? I put something together in a branch in my scipy fork, and it recreates the existing code verbatim. See the templates folder and _generate_zeros_type.py script. https://github.com/mikofski/scipy/tree/cython_optimize_generate_code/scipy/… Any comments would be greatly appreciated. thanks! Mark -- Mark Mikofski, PhD (2005) *Fiat Lux*

5 8

Initializing error number in optimize zeros
by Mark Alexander Mikofski Aug. 30, 2018

Aug. 30, 2018

Hi, I have a question about the scalar function root finders in optimize/Zeros like brentq, etc, that are written in C, the error number is never explicitly initialized, so it could be previously set to -2 or -1, which could be misinterpreted as a failure. Any reason not to set the error number to zero explicitly at the start of the function? I've created an issue to track this : ENH: initialize params->error_num to zero in optimize/Zeros/*.c · Issue #9189 · scipy/scipy https://github.com/scipy/scipy/issues/9189 and implemented it in the cython_optimize_struct branch of my fork, eg bisect.c: https://github.com/mikofski/scipy/blob/cython_optimize_struct/scipy/optimiz… Please, any comment? Thanks! Mark

1 0

Plans for PyPy3 wheels?
by Juan Luis Cano Aug. 28, 2018

Aug. 28, 2018

Hi all, [I'm sending this to scipy-dev because I'm already subscribed, but I guess someone could bring this discussion to numpy-discussion as well] Lately PyPy3 (I'm only interested in Python 3) has matured a lot and it's becoming more and more interesting with time. The developers invested a lot of effort in improving the compatibility with CPython C extensions, and I'm happy to see that SciPy in particular passes all tests in PyPy3: https://circleci.com/gh/scipy/scipy/8318 There was an effort by Antonio Cuni, a PyPy developer, to produce wheels of popular scientific packages. However, it's still a non official thing, and I tried to link them to OpenBLAS but I'm not skilled (or patient) enough: https://github.com/antocuni/pypy-wheels/pull/6 Are there any plans to officially support PyPy3 (and perhaps PyPy) and upload official wheels to pypi.org? Lack of binary packages is holding me back for playing more with PyPy, and there are still no clear plans on what to do with conda packages: https://github.com/conda-forge/pypy2.7-feedstock/issues/1 Did other SciPy or NumPy developers play with PyPy3? Do you think this would be an interesting thing to have? What needs to be done to get there? Best, -- Juan Luis Cano

3 3

please review cython optimize zeros api
by Mark Alexander Mikofski Aug. 24, 2018

Aug. 24, 2018

Hi, PR 8431 is ready for review, IMHO. https://github.com/scipy/scipy/pull/8431 This PR adds a Cython API for root-finding of scalar functions in Optimize. It exposes brentq, brenth, ridder, bisect, newton, secant, and halley with 3 different callback signatures, two which are safe with Cython `nogil` which allows them to be called with Cython `prange`. this PR is a follow on to #8357, and allows efficient looping from C of large arrays with the same callback using any of the scalar-function root finders. I would appreciate your comments. Best, Mark -- Mark Mikofski, PhD (2005) *Fiat Lux*

1 0

Re: [SciPy-Dev] Proposal for some new fitting routines
by Robert Lucente Aug. 23, 2018

Aug. 23, 2018

2 1

Re: [SciPy-Dev] Proposal for some new fitting routines
by Christoph Baumgarten Aug. 23, 2018

Aug. 23, 2018

I have never used these methods, but they look useful and given the links to stackoverflow etc, there seems to be a certain "demand" for them. In stats, we have linregress as well as theilslopes (robust regression), i recently created PR #9101 to add another robust regression technique. so it could fit there. On Thu, Aug 23, 2018 at 6:02 PM <scipy-dev-request(a)python.org> wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request(a)python.org > > You can reach the person managing the list at > scipy-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Re: Proposal for some new fitting routines (Joseph Fox-Rabinovitz) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 22 Aug 2018 13:32:04 -0400 > From: Joseph Fox-Rabinovitz <jfoxrabinovitz(a)gmail.com> > To: SciPy Developers List <scipy-dev(a)python.org> > Subject: Re: [SciPy-Dev] Proposal for some new fitting routines > Message-ID: > < > CAAa1KPYoS__fKgb3o+4xt0s+Xn-1DivQSZsD0x-DqraXLFhj3Q(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Tue, Aug 21, 2018 at 11:02 PM <josef.pktd(a)gmail.com> wrote: > > > > > > > On Tue, Aug 21, 2018 at 10:12 PM, Joseph Fox-Rabinovitz < > > jfoxrabinovitz(a)gmail.com> wrote: > > > >> I had sent out an email about a PR proposing an exponential and a power > >> fit function, but I did not specify the PR number. The PR is #9158. > >> > >> Regards, > >> > >> - Joe > >> > >> > >> On Mon, Aug 20, 2018, 3:40 PM Joseph Fox-Rabinovitz < > >> jfoxrabinovitz(a)gmail.com> wrote: > >> > >>> I have added a function for fitting exponential and power curves. This > >>> seems to > >>> be a fairly common problem (e.g., see the list of selections from stack > >>> exchange > >>> below). The algorithm I have implemented here is objectively better > than > >>> the > >>> common solution of fitting to log(y) because it requires no up-front > >>> knowledge > >>> of the intercept, or any other estimations. > >>> > >>> Assuming that this PR is acceptable, I would like to get some criticism > >>> on where > >>> I chose to put it in the directory structure. My thought was to create > a > >>> separate > >>> module for such common non-iterative optimizations. > >>> > >>> I am also unsure of whether my citation of the original work is > formatted > >>> properly. I would like to make sure that the author gets the credit and > >>> publicity they > >>> deserve. The paper is mentioned (by the author) in answers to [1] and > >>> [2] below. > >>> > >>> List of relevant stack exchange questions: > >>> > >>> [1]: > >>> > https://stackoverflow.com/questions/3938042/fitting-exponential-decay-with-… > >>> [2]: > >>> > https://math.stackexchange.com/questions/1337601/fit-exponential-with-const… > >>> [3]: > >>> > https://math.stackexchange.com/questions/350754/fitting-exponential-curve-t… > >>> [4]: > >>> > https://math.stackexchange.com/questions/1999069/fit-curve-exponential-mayb… > >>> [5]: > >>> > https://mathematica.stackexchange.com/questions/87645/exponential-fit-to-da… > >>> [6]: > >>> > https://stackoverflow.com/questions/48099026/exponential-curve-fit-will-not… > >>> [7]: > >>> > https://stackoverflow.com/questions/50683768/exponent-curve-fitting-in-pyth… > >>> [8]: > >>> > https://stackoverflow.com/questions/49565152/curve-fit-an-exponential-decay… > >>> [9]: > >>> > https://stackoverflow.com/questions/21420792/exponential-curve-fitting-in-s… > >>> > >>> Regards, > >>> > >>> - Joe > >>> > >> > > > > The main question in my opinion is whether scipy should add functions in > > this area, this area essentially doesn't exist in scipy and is covered, > for > > example, by lmfit. > > > > Except for the scipy.stats.distribution and the simple linear model, > there > > are no fit or estimation methods for specific functions, AFAIK. ODR is > > related but also doesn't provide specific nonlinear functions > > scipy.optimize has the optimization algorithm but not specific nonlinear > > functions that are estimated. > > > > lmfit has built in functions for nonlinear estimation > > https://lmfit.github.io/lmfit-py/builtin_models.html > > and there some other less well known packages, AFAIR. > > > > (There was also a discussion about adding similar predefined functions to > > statsmodels, but they were never added and for those cases I usually > refer > > to lmfit.) > > > > lmfit definitely looks like a good choice to put the functions. On the > other hand, it offers much more advanced features than just a simple > regression. > > My guess is that, unless there is a clear plan to expand in this area in > > scipy, then pointing to lmfit would be the better approach. > > Nevertheless, there could also be demand for having something simple in > > scipy. E.g. scipy.stats is a bit "eclectic" and has some overlap with > > statsmodels, or the other way around. scipy.stats has simple linear > > regression but we never added a full OLS, > > > > On the other hand, the functions I propose are extremely simple, and would > do well alongside linregress if they were renamed expregress and > powregress. > > > > multiple linear regression to scipy.stats because there would be too much > > code duplication especially if there is feature creep and users want > > similar extended results as statsmodels provides. > > > > numpy has polynomial fitting as the only fitting function, AFAIK. > > > > Definitely out of scope for numpy and borderline for scipy. > > > > > > Josef > > > > > > > > > > > >> > >> _______________________________________________ > >> SciPy-Dev mailing list > >> SciPy-Dev(a)python.org > >> https://mail.python.org/mailman/listinfo/scipy-dev > >> > >> > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev(a)python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > >

1 0

Proposal for some new fitting routines
by Joseph Fox-Rabinovitz Aug. 22, 2018

Aug. 22, 2018

I have added a function for fitting exponential and power curves. This seems to be a fairly common problem (e.g., see the list of selections from stack exchange below). The algorithm I have implemented here is objectively better than the common solution of fitting to log(y) because it requires no up-front knowledge of the intercept, or any other estimations. Assuming that this PR is acceptable, I would like to get some criticism on where I chose to put it in the directory structure. My thought was to create a separate module for such common non-iterative optimizations. I am also unsure of whether my citation of the original work is formatted properly. I would like to make sure that the author gets the credit and publicity they deserve. The paper is mentioned (by the author) in answers to [1] and [2] below. List of relevant stack exchange questions: [1]: https://stackoverflow.com/questions/3938042/fitting-exponential-decay-with-… [2]: https://math.stackexchange.com/questions/1337601/fit-exponential-with-const… [3]: https://math.stackexchange.com/questions/350754/fitting-exponential-curve-t… [4]: https://math.stackexchange.com/questions/1999069/fit-curve-exponential-mayb… [5]: https://mathematica.stackexchange.com/questions/87645/exponential-fit-to-da… [6]: https://stackoverflow.com/questions/48099026/exponential-curve-fit-will-not… [7]: https://stackoverflow.com/questions/50683768/exponent-curve-fitting-in-pyth… [8]: https://stackoverflow.com/questions/49565152/curve-fit-an-exponential-decay… [9]: https://stackoverflow.com/questions/21420792/exponential-curve-fitting-in-s… Regards, - Joe

2 3

Re: [SciPy-Dev] SciPy-Dev Digest, Vol 178, Issue 18
by Matt Newville Aug. 22, 2018

Aug. 22, 2018

Hi All, On Tue, Aug 21, 2018 at 10:01 PM <scipy-dev-request(a)python.org> wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request(a)python.org > > You can reach the person managing the list at > scipy-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. NumPy 1.15.1 released. (Charles R Harris) > 2. Proposal for some new fitting routines (Joseph Fox-Rabinovitz) > 3. Re: Proposal for some new fitting routines (josef.pktd(a)gmail.com) > > > Message: 2 > Date: Tue, 21 Aug 2018 22:12:49 -0400 > From: Joseph Fox-Rabinovitz <jfoxrabinovitz(a)gmail.com> > To: SciPy Developers List <scipy-dev(a)python.org> > Subject: [SciPy-Dev] Proposal for some new fitting routines > Message-ID: > < > CAAa1KPZ76rECAtBNF_cNMgpQD2eE1YiZgo9notpUSpGA-_yO6Q(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I had sent out an email about a PR proposing an exponential and a power fit > function, but I did not specify the PR number. The PR is #9158. > > Regards, > > - Joe > > > On Mon, Aug 20, 2018, 3:40 PM Joseph Fox-Rabinovitz < > jfoxrabinovitz(a)gmail.com> wrote: > > > I have added a function for fitting exponential and power curves. This > > seems to > > be a fairly common problem (e.g., see the list of selections from stack > > exchange > > below). The algorithm I have implemented here is objectively better than > > the > > common solution of fitting to log(y) because it requires no up-front > > knowledge > > of the intercept, or any other estimations. > > > > Assuming that this PR is acceptable, I would like to get some criticism > on > > where > > I chose to put it in the directory structure. My thought was to create a > > separate > > module for such common non-iterative optimizations. > > > > I am also unsure of whether my citation of the original work is formatted > > properly. I would like to make sure that the author gets the credit and > > publicity they > > deserve. The paper is mentioned (by the author) in answers to [1] and [2] > > below. > > > My impression is that these are fine procedures to have readily available, but that they might not be a great fit for scipy.optimize since the basic algorithm is non-iterative. That is, the procedures seems closer to scipy.stats.linregress and so it might make more sense having them scipy.stats, perhaps calling the routines scipy.stats.expregress and .powregress. It would be very helpful to make the output a closer match to those of linregress, especially including the standard errors and correlation coefficients if possible. Since the algorithms are non-iterative, I'm not sure it makes a lot of sense to have these become the exponential and power-law fit methodsin lmfit (there are existing exponential and power-law models there already), but I would be inclined to use these algorithms (whether they go into scipy or not) to produce initial guesses for these models. Hope that helps (and thanks for asking!), --Matt Newville

1 0

NumPy 1.15.1 released.
by Charles R Harris Aug. 21, 2018

Aug. 21, 2018

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.15.1. This is a bugfix release for bugs and regressions reported following the 1.15.0 release. Noticeable fixes are - The annoying but harmless RuntimeWarning that "numpy.dtype size changed" has been suppressed. The long standing suppression was lost in the transition to pytest. - The update to Cython 0.28.3 exposed a problematic use of a gcc attribute used to prefer code size over speed in module initialization, possibly resulting in incorrect compiled code. This has been fixed in latest Cython but has been disabled here for safety. - Support for big-endian and ARMv8 architectures has been improved. The Python versions supported by this release are 2.7, 3.4-3.7. The wheels are linked with OpenBLAS v0.3.0, which should fix some of the linalg problems reported for NumPy 1.14. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.15.1/>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.15.1>. *Compatibility Note* The NumPy 1.15.x OS X wheels released on PyPI no longer contain 32-bit binaries. That will also be the case in future releases. See #11625 <https://github.com/numpy/numpy/issues/11625> for the related discussion. Those needing 32-bit support on the Mac should look elsewhere or build from source. *Contributors* A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Charles Harris * Chris Billington * Elliott Sales de Andrade + * Eric Wieser * Jeremy Manning + * Matti Picus * Ralf Gommers *Pull requests merged* A total of 24 pull requests were merged for this release. * `#11647 <https://github.com/numpy/numpy/pull/11647>`__: MAINT: Filter Cython warnings in ``__init__.py`` * `#11648 <https://github.com/numpy/numpy/pull/11648>`__: BUG: Fix doc source links to unwrap decorators * `#11657 <https://github.com/numpy/numpy/pull/11657>`__: BUG: Ensure singleton dimensions are not dropped when converting... * `#11661 <https://github.com/numpy/numpy/pull/11661>`__: BUG: Warn on Nan in minimum,maximum for scalars * `#11665 <https://github.com/numpy/numpy/pull/11665>`__: BUG: cython sometimes emits invalid gcc attribute * `#11682 <https://github.com/numpy/numpy/pull/11682>`__: BUG: Fix regression in void_getitem * `#11698 <https://github.com/numpy/numpy/pull/11698>`__: BUG: Make matrix_power again work for object arrays. * `#11700 <https://github.com/numpy/numpy/pull/11700>`__: BUG: Add missing PyErr_NoMemory after failing malloc * `#11719 <https://github.com/numpy/numpy/pull/11719>`__: BUG: Fix undefined functions on big-endian systems. * `#11720 <https://github.com/numpy/numpy/pull/11720>`__: MAINT: Make einsum optimize default to False. * `#11746 <https://github.com/numpy/numpy/pull/11746>`__: BUG: Fix regression in loadtxt for bz2 text files in Python 2. * `#11757 <https://github.com/numpy/numpy/pull/11757>`__: BUG: Revert use of `console_scripts`. * `#11758 <https://github.com/numpy/numpy/pull/11758>`__: BUG: Fix Fortran kind detection for aarch64 & s390x. * `#11759 <https://github.com/numpy/numpy/pull/11759>`__: BUG: Fix printing of longdouble on ppc64le. * `#11760 <https://github.com/numpy/numpy/pull/11760>`__: BUG: Fixes for unicode field names in Python 2 * `#11761 <https://github.com/numpy/numpy/pull/11761>`__: BUG: Increase required cython version on python 3.7 * `#11763 <https://github.com/numpy/numpy/pull/11763>`__: BUG: check return value of _buffer_format_string * `#11775 <https://github.com/numpy/numpy/pull/11775>`__: MAINT: Make assert_array_compare more generic. * `#11776 <https://github.com/numpy/numpy/pull/11776>`__: TST: Fix urlopen stubbing. * `#11777 <https://github.com/numpy/numpy/pull/11777>`__: BUG: Fix regression in intersect1d. * `#11779 <https://github.com/numpy/numpy/pull/11779>`__: BUG: Fix test sensitive to platform byte order. * `#11781 <https://github.com/numpy/numpy/pull/11781>`__: BUG: Avoid signed overflow in histogram * `#11785 <https://github.com/numpy/numpy/pull/11785>`__: BUG: Fix pickle and memoryview for datetime64, timedelta64 scalars * `#11786 <https://github.com/numpy/numpy/pull/11786>`__: BUG: Deprecation triggers segfault Cheers, Charles Harris

1 0

Improve Scipy's Stable PDF FFT Implementation
by Blair Azzopardi Aug. 17, 2018

Aug. 17, 2018

Hi I've written a short paper exploring optimising our Stable probability density FFT implementation. In my original PR (7374) I introduced a method by Mitnik. However, there is a better approach by Wang that significantly improves accuracy. Wang's method can be generalized using standard techniques improving accuracy even more but at the cost of CPU time. I felt I should write to the dev mailing list as I'm not sure if implementing what I feel is a generalized approach might not be established enough within the academic community for inclusion in Scipy. Although, that said, Wang's approach is cited by several other papers so shouldn't be a problem. The paper can be found in my github repository below. I'd like to submit to Arxiv but can't without an endorsement; perhaps someone here might be willing to help there. https://github.com/bsdz/docs/blob/master/papers/Stable%20Probability%20Dens… So, I plan to raise at least another PR to replace the FFT implementation of Mitnik with Wang's. A couple of questions that come to mind: 1) Is it worth me just introducing the general solution that defaults to Wang's? 2) Is it worth placing the general FFT characteristic function code into a separate module for approximating other integrals using FFT? If so, where would be a good place? Blair

2 2

Universal Method to Sort Complex Information Found | Quanta Magazine
by Neal Becker Aug. 15, 2018

Aug. 15, 2018

https://www.quantamagazine.org/universal-method-to-sort-complex-information…

1 0

Compiling scipy on macOS with conda environment
by Andrew Nelson Aug. 15, 2018

Aug. 15, 2018

Hi all, I've been having issues compiling scipy/master on macOS in a Python 3.6 conda based environment (numpy 1.15). I'm getting a whole load of undefined symbols, and the tail end of the build log is below. I've managed to get around the problem by setting LDFLAGS (based on https://groups.google.com/a/continuum.io/forum/#!topic/conda/cwqTbwQAyW0): export LDFLAGS="$LDFLAGS -undefined dynamic_lookup -bundle" I have no idea why this works. Not sure why the problem comes about, but I thought people might be interested in knowing what the solution could be. /usr/local/bin/gfortran -Wall -g -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs build/temp.macosx-10.9-x86_64-3.6/build/src.macosx-10.9-x86_64-3.6/scipy/fftpack/_fftpackmodule.o build/temp.macosx-10.9-x86_64-3.6/scipy/fftpack/src/zfft.o build/temp.macosx-10.9-x86_64-3.6/scipy/fftpack/src/drfft.o build/temp.macosx-10.9-x86_64-3.6/scipy/fftpack/src/zrfft.o build/temp.macosx-10.9-x86_64-3.6/scipy/fftpack/src/zfftnd.o build/temp.macosx-10.9-x86_64-3.6/build/src.macosx-10.9-x86_64-3.6/scipy/fftpack/src/dct.o build/temp.macosx-10.9-x86_64-3.6/build/src.macosx-10.9-x86_64-3.6/scipy/fftpack/src/dst.o build/temp.macosx-10.9-x86_64-3.6/build/src.macosx-10.9-x86_64-3.6/build/src.macosx-10.9-x86_64-3.6/scipy/fftpack/fortranobject.o -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin16/6.3.0 -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin16/6.3.0/../../.. -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin16/6.3.0/../../.. -Lbuild/temp.macosx-10.9-x86_64-3.6 -ldfftpack -lfftpack -lgfortran -o build/lib.macosx-10.9-x86_64-3.6/scipy/fftpack/_ fftpack.cpython-36m-darwin.so Undefined symbols for architecture x86_64: "_PyArg_ParseTupleAndKeywords", referenced from: _f2py_rout__fftpack_zfft in _fftpackmodule.o _f2py_rout__fftpack_drfft in _fftpackmodule.o _f2py_rout__fftpack_zrfft in _fftpackmodule.o _f2py_rout__fftpack_zfftnd in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfft_cache in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfftnd_cache in _fftpackmodule.o _f2py_rout__fftpack_destroy_drfft_cache in _fftpackmodule.o ... "_PyBytes_FromString", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyCapsule_GetPointer", referenced from: _PyInit__fftpack in _fftpackmodule.o _F2PyCapsule_AsVoidPtr in fortranobject.o "_PyCapsule_New", referenced from: _fortran_getattr in fortranobject.o _F2PyCapsule_FromVoidPtr in fortranobject.o "_PyCapsule_Type", referenced from: _PyInit__fftpack in _fftpackmodule.o _F2PyCapsule_Check in fortranobject.o "_PyComplex_Type", referenced from: _int_from_pyobj in _fftpackmodule.o "_PyDict_DelItemString", referenced from: _fortran_setattr in fortranobject.o "_PyDict_GetItemString", referenced from: _fortran_getattr in fortranobject.o "_PyDict_New", referenced from: _PyFortranObject_New in fortranobject.o _PyFortranObject_NewAsAttr in fortranobject.o _fortran_setattr in fortranobject.o "_PyDict_SetItemString", referenced from: _PyInit__fftpack in _fftpackmodule.o _F2PyDict_SetItemString in fortranobject.o _PyFortranObject_New in fortranobject.o _fortran_getattr in fortranobject.o _fortran_setattr in fortranobject.o "_PyErr_Clear", referenced from: _int_from_pyobj in _fftpackmodule.o _F2PyDict_SetItemString in fortranobject.o _fortran_getattr in fortranobject.o _fortran_repr in fortranobject.o _F2PyCapsule_FromVoidPtr in fortranobject.o _F2PyCapsule_AsVoidPtr in fortranobject.o "_PyErr_Format", referenced from: _PyInit__fftpack in _fftpackmodule.o _fortran_call in fortranobject.o _check_and_fix_dimensions in fortranobject.o "_PyErr_NewException", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyErr_NoMemory", referenced from: _fortran_getattr in fortranobject.o "_PyErr_Occurred", referenced from: _PyInit__fftpack in _fftpackmodule.o _f2py_rout__fftpack_zfft in _fftpackmodule.o _f2py_rout__fftpack_drfft in _fftpackmodule.o _f2py_rout__fftpack_zrfft in _fftpackmodule.o _f2py_rout__fftpack_zfftnd in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfft_cache in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfftnd_cache in _fftpackmodule.o ... "_PyErr_Print", referenced from: _PyInit__fftpack in _fftpackmodule.o _F2PyDict_SetItemString in fortranobject.o "_PyErr_SetString", referenced from: _PyInit__fftpack in _fftpackmodule.o _f2py_rout__fftpack_zfft in _fftpackmodule.o _f2py_rout__fftpack_drfft in _fftpackmodule.o _f2py_rout__fftpack_zrfft in _fftpackmodule.o _f2py_rout__fftpack_zfftnd in _fftpackmodule.o _f2py_rout__fftpack_cfft in _fftpackmodule.o _f2py_rout__fftpack_rfft in _fftpackmodule.o ... "_PyExc_AttributeError", referenced from: _PyInit__fftpack in _fftpackmodule.o _fortran_setattr in fortranobject.o "_PyExc_ImportError", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyExc_RuntimeError", referenced from: _PyInit__fftpack in _fftpackmodule.o _fortran_call in fortranobject.o "_PyExc_TypeError", referenced from: _fortran_call in fortranobject.o _array_from_pyobj in fortranobject.o "_PyExc_ValueError", referenced from: _array_from_pyobj in fortranobject.o _check_and_fix_dimensions in fortranobject.o "_PyImport_ImportModule", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyLong_AsLong", referenced from: _int_from_pyobj in _fftpackmodule.o "_PyMem_Free", referenced from: _fortran_dealloc in fortranobject.o _fortran_getattr in fortranobject.o "_PyMem_Malloc", referenced from: _fortran_getattr in fortranobject.o "_PyModule_Create2", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyModule_GetDict", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyNumber_Long", referenced from: _int_from_pyobj in _fftpackmodule.o "_PyOS_snprintf", referenced from: _fortran_getattr in fortranobject.o "_PyObject_GenericGetAttr", referenced from: _fortran_getattr in fortranobject.o "_PyObject_GetAttrString", referenced from: _PyInit__fftpack in _fftpackmodule.o _int_from_pyobj in _fftpackmodule.o _fortran_repr in fortranobject.o "_PySequence_Check", referenced from: _int_from_pyobj in _fftpackmodule.o "_PySequence_GetItem", referenced from: _int_from_pyobj in _fftpackmodule.o "_PyType_IsSubtype", referenced from: _int_from_pyobj in _fftpackmodule.o _array_from_pyobj in fortranobject.o "_PyType_Type", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyUnicode_Concat", referenced from: _fortran_getattr in fortranobject.o "_PyUnicode_FromFormat", referenced from: _fortran_repr in fortranobject.o "_PyUnicode_FromString", referenced from: _PyInit__fftpack in _fftpackmodule.o _fortran_getattr in fortranobject.o _fortran_repr in fortranobject.o "_PyUnicode_FromStringAndSize", referenced from: _fortran_getattr in fortranobject.o "_Py_BuildValue", referenced from: _f2py_rout__fftpack_zfft in _fftpackmodule.o _f2py_rout__fftpack_drfft in _fftpackmodule.o _f2py_rout__fftpack_zrfft in _fftpackmodule.o _f2py_rout__fftpack_zfftnd in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfft_cache in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfftnd_cache in _fftpackmodule.o _f2py_rout__fftpack_destroy_drfft_cache in _fftpackmodule.o ... "__PyObject_New", referenced from: _PyFortranObject_New in fortranobject.o _PyFortranObject_NewAsAttr in fortranobject.o "__Py_NoneStruct", referenced from: _f2py_rout__fftpack_zfft in _fftpackmodule.o _f2py_rout__fftpack_drfft in _fftpackmodule.o _f2py_rout__fftpack_zrfft in _fftpackmodule.o _f2py_rout__fftpack_zfftnd in _fftpackmodule.o _f2py_rout__fftpack_cfft in _fftpackmodule.o _f2py_rout__fftpack_rfft in _fftpackmodule.o _f2py_rout__fftpack_crfft in _fftpackmodule.o ... "_main", referenced from: implicit entry/start for main executable ld: symbol(s) not found for architecture x86_64 collect2: error: ld returned 1 exit status Undefined symbols for architecture x86_64: "_PyArg_ParseTupleAndKeywords", referenced from: _f2py_rout__fftpack_zfft in _fftpackmodule.o _f2py_rout__fftpack_drfft in _fftpackmodule.o _f2py_rout__fftpack_zrfft in _fftpackmodule.o _f2py_rout__fftpack_zfftnd in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfft_cache in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfftnd_cache in _fftpackmodule.o _f2py_rout__fftpack_destroy_drfft_cache in _fftpackmodule.o ... "_PyBytes_FromString", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyCapsule_GetPointer", referenced from: _PyInit__fftpack in _fftpackmodule.o _F2PyCapsule_AsVoidPtr in fortranobject.o "_PyCapsule_New", referenced from: _fortran_getattr in fortranobject.o _F2PyCapsule_FromVoidPtr in fortranobject.o "_PyCapsule_Type", referenced from: _PyInit__fftpack in _fftpackmodule.o _F2PyCapsule_Check in fortranobject.o "_PyComplex_Type", referenced from: _int_from_pyobj in _fftpackmodule.o "_PyDict_DelItemString", referenced from: _fortran_setattr in fortranobject.o "_PyDict_GetItemString", referenced from: _fortran_getattr in fortranobject.o "_PyDict_New", referenced from: _PyFortranObject_New in fortranobject.o _PyFortranObject_NewAsAttr in fortranobject.o _fortran_setattr in fortranobject.o "_PyDict_SetItemString", referenced from: _PyInit__fftpack in _fftpackmodule.o _F2PyDict_SetItemString in fortranobject.o _PyFortranObject_New in fortranobject.o _fortran_getattr in fortranobject.o _fortran_setattr in fortranobject.o "_PyErr_Clear", referenced from: _int_from_pyobj in _fftpackmodule.o _F2PyDict_SetItemString in fortranobject.o _fortran_getattr in fortranobject.o _fortran_repr in fortranobject.o _F2PyCapsule_FromVoidPtr in fortranobject.o _F2PyCapsule_AsVoidPtr in fortranobject.o "_PyErr_Format", referenced from: _PyInit__fftpack in _fftpackmodule.o _fortran_call in fortranobject.o _check_and_fix_dimensions in fortranobject.o "_PyErr_NewException", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyErr_NoMemory", referenced from: _fortran_getattr in fortranobject.o "_PyErr_Occurred", referenced from: _PyInit__fftpack in _fftpackmodule.o _f2py_rout__fftpack_zfft in _fftpackmodule.o _f2py_rout__fftpack_drfft in _fftpackmodule.o _f2py_rout__fftpack_zrfft in _fftpackmodule.o _f2py_rout__fftpack_zfftnd in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfft_cache in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfftnd_cache in _fftpackmodule.o ... "_PyErr_Print", referenced from: _PyInit__fftpack in _fftpackmodule.o _F2PyDict_SetItemString in fortranobject.o "_PyErr_SetString", referenced from: _PyInit__fftpack in _fftpackmodule.o _f2py_rout__fftpack_zfft in _fftpackmodule.o _f2py_rout__fftpack_drfft in _fftpackmodule.o _f2py_rout__fftpack_zrfft in _fftpackmodule.o _f2py_rout__fftpack_zfftnd in _fftpackmodule.o _f2py_rout__fftpack_cfft in _fftpackmodule.o _f2py_rout__fftpack_rfft in _fftpackmodule.o ... "_PyExc_AttributeError", referenced from: _PyInit__fftpack in _fftpackmodule.o _fortran_setattr in fortranobject.o "_PyExc_ImportError", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyExc_RuntimeError", referenced from: _PyInit__fftpack in _fftpackmodule.o _fortran_call in fortranobject.o "_PyExc_TypeError", referenced from: _fortran_call in fortranobject.o _array_from_pyobj in fortranobject.o "_PyExc_ValueError", referenced from: _array_from_pyobj in fortranobject.o _check_and_fix_dimensions in fortranobject.o "_PyImport_ImportModule", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyLong_AsLong", referenced from: _int_from_pyobj in _fftpackmodule.o "_PyMem_Free", referenced from: _fortran_dealloc in fortranobject.o _fortran_getattr in fortranobject.o "_PyMem_Malloc", referenced from: _fortran_getattr in fortranobject.o "_PyModule_Create2", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyModule_GetDict", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyNumber_Long", referenced from: _int_from_pyobj in _fftpackmodule.o "_PyOS_snprintf", referenced from: _fortran_getattr in fortranobject.o "_PyObject_GenericGetAttr", referenced from: _fortran_getattr in fortranobject.o "_PyObject_GetAttrString", referenced from: _PyInit__fftpack in _fftpackmodule.o _int_from_pyobj in _fftpackmodule.o _fortran_repr in fortranobject.o "_PySequence_Check", referenced from: _int_from_pyobj in _fftpackmodule.o "_PySequence_GetItem", referenced from: _int_from_pyobj in _fftpackmodule.o "_PyType_IsSubtype", referenced from: _int_from_pyobj in _fftpackmodule.o _array_from_pyobj in fortranobject.o "_PyType_Type", referenced from: _PyInit__fftpack in _fftpackmodule.o "_PyUnicode_Concat", referenced from: _fortran_getattr in fortranobject.o "_PyUnicode_FromFormat", referenced from: _fortran_repr in fortranobject.o "_PyUnicode_FromString", referenced from: _PyInit__fftpack in _fftpackmodule.o _fortran_getattr in fortranobject.o _fortran_repr in fortranobject.o "_PyUnicode_FromStringAndSize", referenced from: _fortran_getattr in fortranobject.o "_Py_BuildValue", referenced from: _f2py_rout__fftpack_zfft in _fftpackmodule.o _f2py_rout__fftpack_drfft in _fftpackmodule.o _f2py_rout__fftpack_zrfft in _fftpackmodule.o _f2py_rout__fftpack_zfftnd in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfft_cache in _fftpackmodule.o _f2py_rout__fftpack_destroy_zfftnd_cache in _fftpackmodule.o _f2py_rout__fftpack_destroy_drfft_cache in _fftpackmodule.o ... "__PyObject_New", referenced from: _PyFortranObject_New in fortranobject.o _PyFortranObject_NewAsAttr in fortranobject.o "__Py_NoneStruct", referenced from: _f2py_rout__fftpack_zfft in _fftpackmodule.o _f2py_rout__fftpack_drfft in _fftpackmodule.o _f2py_rout__fftpack_zrfft in _fftpackmodule.o _f2py_rout__fftpack_zfftnd in _fftpackmodule.o _f2py_rout__fftpack_cfft in _fftpackmodule.o _f2py_rout__fftpack_rfft in _fftpackmodule.o _f2py_rout__fftpack_crfft in _fftpackmodule.o ... "_main", referenced from: implicit entry/start for main executable ld: symbol(s) not found for architecture x86_64 collect2: error: ld returned 1 exit status error: Command "/usr/local/bin/gfortran -Wall -g -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs build/temp.macosx-10.9-x86_64-3.6/build/src.macosx-10.9-x86_64-3.6/scipy/fftpack/_fftpackmodule.o build/temp.macosx-10.9-x86_64-3.6/scipy/fftpack/src/zfft.o build/temp.macosx-10.9-x86_64-3.6/scipy/fftpack/src/drfft.o build/temp.macosx-10.9-x86_64-3.6/scipy/fftpack/src/zrfft.o build/temp.macosx-10.9-x86_64-3.6/scipy/fftpack/src/zfftnd.o build/temp.macosx-10.9-x86_64-3.6/build/src.macosx-10.9-x86_64-3.6/scipy/fftpack/src/dct.o build/temp.macosx-10.9-x86_64-3.6/build/src.macosx-10.9-x86_64-3.6/scipy/fftpack/src/dst.o build/temp.macosx-10.9-x86_64-3.6/build/src.macosx-10.9-x86_64-3.6/build/src.macosx-10.9-x86_64-3.6/scipy/fftpack/fortranobject.o -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin16/6.3.0 -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin16/6.3.0/../../.. -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin16/6.3.0/../../.. -Lbuild/temp.macosx-10.9-x86_64-3.6 -ldfftpack -lfftpack -lgfortran -o build/lib.macosx-10.9-x86_64-3.6/scipy/fftpack/_ fftpack.cpython-36m-darwin.so" failed with exit status 1

2 2

Cython optimize root finders
by Mark Alexander Mikofski Aug. 14, 2018

Aug. 14, 2018

Hi, After some previous discussion, I have implemented or wrapped some root finders from optimize in Cython for users who need performance improvements or low level control. See https://github.com/scipy/scipy/pull/8431#issuecomment-412699499 I would appreciate anyone's comments. Thanks, Mark

1 0

minor update to CoC
by Ralf Gommers Aug. 12, 2018

Aug. 12, 2018

Hi all, On the NumPy mailing list there was/is a long discussion about adopting a CoC. In particular, the same one as SciPy. People are in agreement on that but wanted one clarification - that the list of categories in the diversity statement was appended with a sentence saying that being in such a category does not excuse bad behavior. We would like to keep the SciPy and NumPy CoC's in sync, so that addition is being made at https://github.com/scipy/scipy/pull/9109 first. The exact addition is: ", to the extent that these do not conflict with this code of conduct." I plan to merge that PR soon, if there's any objection please comment. Cheers, Ralf

2 1

scikit prefix for microscopy file I/O library?
by Magnus Nord Aug. 9, 2018

Aug. 9, 2018

I'm one of the developers in the HyperSpy project, which is an open source python library for analysing multidimensional data, with a focus on electron microscopy. We're in the processes of splitting the project into smaller pieces, to i) make the code a bit more manageable, ii) make it easier for other projects to use specific functionalities without having to install a lot of extra dependencies. The current focus is moving the file reading/writing functionality to its own library, which includes readers for many types of proprietary electron microscopy file formats. The plan is to keep the GPLv3 license, and only depend on the more standard scientific python stack (numpy, ...). However, we've been having some issues in deciding on a name for the library, and someone suggested using a scikit- prefix (scikit-microscopyIO?, scikit-microscopy-io?). So I'm wondering if a scikit prefix would be appropriate for a library focused on file I/O of microscopy data? Discussion on this in the HyperSpy issue tracker: https://github.com/hyperspy/hyperspy/issues/1978 Magnus

3 6

Re: [SciPy-Dev] Review of PR 8293 - sampling of random variates
by Christoph Baumgarten Aug. 5, 2018

Aug. 5, 2018

A good source on the ratio of uniforms method is the book by Devroye: "non-uniform random variate generation" or the original paper "Computer Computer Generation of Random Variables using the ratio of uniform deviates" by kinderman / monahan. <scipy-dev-request(a)python.org> schrieb am Sa., 4. Aug. 2018, 18:03: > Send SciPy-Dev mailing list submissions to > scipy-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request(a)python.org > > You can reach the person managing the list at > scipy-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Re: Review of PR 8293 - sampling of random variates (Neal Becker) > 2. Re: Backwards Compatibility for low level LAPACK routines > (Ralf Gommers) > 3. Re: Backwards Compatibility for low level LAPACK routines > (Pauli Virtanen) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 4 Aug 2018 07:44:06 -0400 > From: Neal Becker <ndbecker2(a)gmail.com> > To: SciPy Developers List <scipy-dev(a)python.org> > Subject: Re: [SciPy-Dev] Review of PR 8293 - sampling of random > variates > Message-ID: > < > CAG3t+pF_5GMXGDJUJF-Z8n2iRHvTSpwG6_b4y1dMWSqwKS_29w(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Don't know about this particular code, but unuran is pretty comprehensive, > and there's a good book from the author. > > On Fri, Aug 3, 2018, 5:37 PM Phillip Feldman <phillip.m.feldman(a)gmail.com> > wrote: > > > The ability to sample random variates from a distribution where only the > > density is available seems quite useful. Is there a reference that > > describes the method? > > > > Phillip > > > > On Thu, Aug 2, 2018 at 11:43 PM, Christoph Baumgarten < > > christoph.baumgarten(a)gmail.com> wrote: > > > >> PR 8293 introduces a method to sample random variates for more complex > >> distributions to scipy.stats (such as hyperbolic distributions and the > >> generalized inverse gaussian (see PR 8681)). I think the PR is in good > >> shape, it would be great if it could be merged soon since i could then > >> continue to work on adding new distributions. The PR has been open for 7 > >> months now. If someone could continue the review, that would bei great. > >> Thanks! > >> > >> _______________________________________________ > >> SciPy-Dev mailing list > >> SciPy-Dev(a)python.org > >> https://mail.python.org/mailman/listinfo/scipy-dev > >> > >> > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev(a)python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > >

2 1

Review of PR 8293 - sampling of random variates
by Christoph Baumgarten Aug. 5, 2018

Aug. 5, 2018

PR 8293 introduces a method to sample random variates for more complex distributions to scipy.stats (such as hyperbolic distributions and the generalized inverse gaussian (see PR 8681)). I think the PR is in good shape, it would be great if it could be merged soon since i could then continue to work on adding new distributions. The PR has been open for 7 months now. If someone could continue the review, that would bei great. Thanks!

4 4

Backwards Compatibility for low level LAPACK routines
by Ilhan Polat Aug. 4, 2018

Aug. 4, 2018

Due their historical evolution, there are certain LAPACK wrappers that are not standardized. Some work with minimum lwork variables instead of their optimal values. Also these routines often return quite big arrays during the lwork queries, to demonstrate : import scipy.linalg as la la.lapack.dgeqrf(a=np.random.rand(400,400), lwork=-1) is a workspace size query (via lwork=-1). The current default size is "3*a.shape[0] + 1" hence 1201. However the optimal workspace size is 12800 on my machine. Therefore the mismatch is sometimes quite dramatic especially in some other routines. Notice also that to obtain this number the routine actually returns a 400-long array tau and requires the input matrix to be transferred back and forth. Moreover, they can't be handled via scipy.linalg.lapack._compute_lwork function. There are a few routines like this and I feel like they should be fixed and I'm willing to. However this means that their output signature is going to change which imply backwards compatibility breaks. I tried to see whether we could deprecate them with new wrappers with modified names, but to be honest, that would create too many duplicates. On the other hand I don't have a feeling of how much break this would mean out there in the wild. Is this break an acceptable one or not? (well, none is acceptable preferably, but in despair...) Any other alternatives, thoughts are most welcome. best, ilhan

4 7

Re: [SciPy-Dev] SciPy-Dev Digest, Vol 177, Issue 21
by Matt Newville Aug. 1, 2018

Aug. 1, 2018

Hi Andrew, Dickson, All, Sorry for barging in rather late to this discussion, and for responding to a digest, and if this appears to be too negative. > Date: Mon, 23 Jul 2018 23:36:37 -0400 > From: dickson41152 <dickson41152(a)protonmail.com> > To: Andrew Nelson <andyfaff(a)gmail.com> > Cc: scipy-dev <scipy-dev(a)python.org> > Subject: Re: [SciPy-Dev] Proposed Modifications to Nelder-Mead Simplex > Algorithm > Message-ID: > > <5xgSJzGhTqz3FgLpZlSQCqNBm-v5aytNC9kOBDasi_Mus8NNwoCKjPVEmrVqOpIn2B22oT1r0jOc8cNWJaBxRicbjjkAClPDRUwCRdPgAXs=@ > protonmail.com> > > Content-Type: text/plain; charset="utf-8" > > Thank you for your interest Andrew. > > > variable transformations to form unconstrained problems from bound > constrained problems > > I am considering writing a new transformation function to calculate the > equations on that web page that you linked to discussing variable > transformations. If it is as straight-forward as I think it will be to > write, then I may also do some comparisons between the "clipping" method > and the sin() / arcsin() / sqrt() method you highlighted. > > As an aside, I was told informally about the existence of that method you > referenced during a conversation I had last week, but I have been finding > it very difficult to find any written discussion (i.e. actual analytical > expressions) about that method via web searches before I made the > proposals. So thank you very much for linking that web page - it is very > useful information for my own personal work as well as the proposed Scipy > modifications! > > > user-specified objective function values for points in the > user-specified initial simplex > > I have indeed used a keyword in the manner you describe (and therefore > similar in style to the initial_simplex keyword) - it seemed to me to be a > "clean" way of implementing this modification and very straightforward to > implement since it is so similar to the initial_simplex handling. > > > evaluation budget count check > > I am thinking of writing checks which will effectively skip the execution > of the remainder of the solver algorithm if the budget has been met, before > finally sorting the current simplex of evaluations and returning the best. > My inclination is to avoid using StopIteration since it is an exception > i.e. it suggests some kind of problem has occurred, where as my proposed > idea is to handle evaluations up to the exact budget as a feature. > > Anyway I haven't yet made an attempt on this modification, so my thoughts > are tentative and may change - especially if it turns out that I am trying > to write spaghetti. > > dickson I think that adding bounds to Nelder-Mead Simplex is an interesting idea. Using the Minuit-style bounds - using smooth functions to transform infiniite "internal" values to finite "external" values - would definitely be an easy and stable way to do that. But, with all due respect, I also think it is the wrong approach ;). The main point of the lmfit package that Andrew linked to is that the fitting *algorithm* should not support bounds on variable values. I do understand that other scipy.optimize algorithms support bounds, and yes, I do think that is an unfortunate choice ;). Instead, the *parameters* should have bounds (and perhaps other properties), and the algorithm should work on these parameter objects. In this way one does not add bounds to Nelder-Mead, and then add bounds to Powell's method, and then add bounds to Levenberg-Marquardt, etc. Rather, one adds bounds to parameters, and has the methods use these parameters instead of scalar values, and thus one add bounds to *all* methods in a consistent and transferrable way. A few months ago Andrew suggested a class-based approach to implementing fitting algorithms. I think this idea has a lot of merit to it. I also believe that lmfit successfully demonstrates that having parameter objects as described briefly above does better encapsulate the desired properties (bounds, etc) *and* makes the code implementing the algorithms clearer and simpler. In fact, if I understand the original goals here to be: 1. Allowing bound constraints 2. Allowing the user to specify initial objective function values for any or all points in a user's initial simplex. 3. Checking the objective function evaluation count against the user-specified evaluation budget at each function evaluation . I would point out that lmfit has a Nelder-Mead solver that already does #1 and #3 and that (as with bounds), #3 may be best done not within the solver, but in outer code that calls the solver, so that this may be available in a consistent manner to multiple methods. If #2 means to memoize each step in the process to avoid repeated calculation, this does not currently exist within lmfit. Such a goal might really be best done in the solver method itself. It may be also be doable using a per-iteration callback (which lmfit does have) , though I don't know that anyone one has ever tried this. I apologize if this appears to be an outsider yelling "you're doing it all wrong". I rely heavily on scipy.optimize for my daily work, and would like to be able to make these tools better but I am over-committed and cannot contribute significantly to the scipy ecosystem myself other than try to support lmfit as best I can. Cheers, --Matt

2 1