Mailman 3 September 2018 - SciPy-Dev

Re: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions
by Nikolay Mayorov 17 Sep '18

17 Sep '18

Hi, Aditya! Glad that you set up the blog and good job on setting up the documentation build as well. Curious, what is this blogging platform like? How do you create posts in it? As for your first post: while not strictly necessary I think it would be nice to see a more thorough introductory post with a brief overview, motivation and/or an example. Do you want to work on it? Best, Nikolay

4 15

ANN: SfePy 2018.3
by Robert Cimrman 17 Sep '18

17 Sep '18

I am pleased to announce release 2018.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (limited support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: https://mail.python.org/mm3/mailman3/lists/sfepy.python.org/ Git (source) repository, issue tracker: https://github.com/sfepy/sfepy Highlights of this release -------------------------- - easier setting of values of variables - new script for outline edge extraction - new example: homogenization of a piezoelectric heterogeneous structure For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Cheers, Robert Cimrman --- Contributors to this release in alphabetical order: Robert Cimrman Vladimir Lukes

1 0

Pythran - 0.8.7
by Serge Guelton 16 Sep '18

16 Sep '18

Hi there, Pythran just released its 0.8.7 version. A quick reminder: Pythran is a compiler for scientific Python, it can be used to turn Python kernels as: #pythran export laplacien(float64[][][3]) import numpy as np def laplacien(image): out_image = np.abs(4*image[1:-1,1:-1] - image[0:-2,1:-1] - image[2:,1:-1] - image[1:-1,0:-2] - image[1:-1,2:]) valmax = np.max(out_image) valmax = max(1.,valmax)+1.E-9 out_image /= valmax return out_image In a native library that runs much faster. Note the ``#pythran export`` line that specifies that the native version only accepts 3D arrays with last dimension set to 3. It's available on pypi: https://pypi.org/project/pythran/ on conda-forge (linux and osx): https://anaconda.org/conda-forge/pythran and on github: https://github.com/serge-sans-paille/pythran Special Thanks for this release to: - Yann Diorcet for all his work on the Windows compatibility - Jean Laroche for the bug reports on OSX - Marteen for his « red dragon » gift - h-vetinari, vgroff, DerWeh, ucyo, Chronum94, paugier, gouarin and Dapid for their bug reports. - alexbw for setting up the conda-forge build This release closes the following issues: https://github.com/serge-sans-paille/pythran/issues?utf8=%E2%9C%93&q=is%3Ai… A more detailed changelog is available at https://pythran.readthedocs.io/en/latest/Changelog.html Enjoy, and if not, submit bug report :-) Serge

2 1

Re: [SciPy-Dev] Improving spatial.distance code and, performance
by Kane & Anna O'Donnell 16 Sep '18

16 Sep '18

Sorry if this doesn't work - I haven't used mailing lists before, and got the digest, which doesn't seem to allow me to reply to a specific thread. So I'm attempting to set the In-Reply-To header manually. Sorry if that's silly. Anyway: Yes, I believe there are good unit tests (Ralf actually ran them I think). From the basis of this, and the feedback in the issue/PR, it sounds like the preference would be for me to upgrade the distance metrics to cython bit-by-bit. However, I'm still unsure about scipy vs a new package, since scipy apparently won't support dependencies. For example, I could use Faiss or flann, but scipy wouldn't add them as a dependency. With that in mind, it seems the only option for offering users full-functionality (aside from rewriting those libraries from scratch etc.) is to offer a package. Unless scipy is willing to specify a run-time dependency (not sure of the correct terminology) - e.g. scipy doesn't depend on Faiss (when you install it), but if the user calls a distance metric with e.g. method='faiss', it will fail if faiss cannot be imported. Is this a possibility (and a good idea)? On 2018/09/12 4:00 am, scipy-dev-request(a)python.org wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request(a)python.org > > You can reach the person managing the list at > scipy-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Re: Improving spatial.distance code and performance > (Mark Alexander Mikofski) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 10 Sep 2018 10:05:17 -0700 > From: Mark Alexander Mikofski <mikofski(a)berkeley.edu> > To: SciPy Developers List <scipy-dev(a)python.org> > Subject: Re: [SciPy-Dev] Improving spatial.distance code and > performance > Message-ID: > <CAEqRcW3x6qZ6X_kpg2U54O7=E4+S6UtkoM3pFQRv8PzQQBCw+w(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I'm very interested to see how a successful cython/performance PR > progresses from a reviewers standpoint. > > On Sun, Sep 9, 2018, 5:10 PM Tyler Reddy <tyler.je.reddy(a)gmail.com> wrote: > >> Good to see some activity / interest in spatial. >> >> Definitely agree with Ralf's github comments re: using smaller / more >> tractable PRs -- it really is tough to sit down at night with 30 minutes of >> free time or whatever and look at a massive diff & not want to give up. >> >> I like the idea of making small / targeted / clearly demonstrated >> performance improvements without overhauling the entire infrastructure >> first, but maybe that's a controversial view if it just causes too much >> heterogeneity. >> >> Presumably the affected code is all thoroughly covered by unit tests? >> That's an important pre-requisite to have the confidence to really start >> making changes, esp. with older low-level code like that. >> >> On Thu, 6 Sep 2018 at 02:29, Kane & Anna O'Donnell < >> email.the.odonnells(a)gmail.com> wrote: >> >>> Hi all, >>> >>> TLDR; I'm wondering about a) porting the spatial.distance code to Cython, >>> and b) adding some performance optimizations, and I'd like the dev >>> community's input/feedback. >>> >>> For context (though I don't really need you to read these to give the >>> feedback I'm looking for), >>> >>> - original issue/proposal: https://github.com/scipy/scipy/issues/9205 >>> - PR: https://github.com/scipy/scipy/pull/9218 >>> >>> Before submitting the PR, I naively thought it was going to be nice >>> Cython or similar. Turns out it's some pretty old code, that I found pretty >>> hard to wrap my head around and understand. I eventually figured it out >>> after spending ages debugging a nastily hidden 'bug', and demonstrated the >>> performance optimizations, but it prompted the discussion about whether it >>> was best to port everything to Cython first. >>> >>> *Existing stuff to Cython* >>> >>> Doing so shouldn't be too hard, and it shouldn't change any >>> functionality, except to replace the distance functions with their Cython >>> ones (instead of the current approach, where the distance functions are >>> actually numpy things, and there's not supported access to the underlying C >>> stuff). A few possible 'bugs' (as above) should hopefully become non-issues >>> too. So, it'd be a win for performance (e.g. all the distance functions >>> will be much faster), and code quality, and future maintainability and >>> development. However, things to think about: >>> >>> - should I just replace like-for-like, or consider some nicer OOP stuff >>> like e.g. sklearn's approach (which is already Cython)? >>> https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/… >>> (I'm guessing the reason they rolled their own was because they found scipy >>> lacking, as above.) In fact, we could largely just copy that file over. Not >>> sure about the interplay between scipy and scikit learn though. >>> - what's the best way to structure a PR? One metric at a time, or the >>> whole caboodle? >>> >>> *Adding performance optimizations and future functionality* >>> >>> As per the links, this was initially about providing a nifty performance >>> hack. It should still be pretty easy to implement. Personally, I think it >>> makes sense to implement after the Cythonization - unless the community are >>> against that. >>> >>> However, there are other possibilities: >>> >>> - using 'indices' within calculations. E.g. when doing pdist, it might >>> pay to use a tree of some description. I also proposed another 'index' to >>> optimize the 'bail early' approach further (which would, I think, actually >>> work well with trees too). This would involve more API changes, possibly >>> significant. >>> - using approximate searches (e.g. Faiss). My understanding is that >>> depending on other libraries probably isn't really an option, so I'm not >>> sure what means. >>> - maybe other cool approaches like https://github.com/droyed/eucl_dist >>> - providing a way to 'tune' distance computations to be optimal to your >>> particular dataset and constraints (e.g. my 'bail early' optimization might >>> be a lot faster or a bit slower, depending on your data ... or you might be >>> OK with approximate matching with a 'low' error rate, etc.) >>> >>> I guess what I'd like to see is a single place where users can get access >>> to everything related to distance metrics and their uses, including all >>> sorts of optimizations etc. (possibly for different hardware, and possibly >>> distributed). To do that well is a pretty big undertaking, and I don't know >>> whether it's suited to scipy - e.g. maybe scipy doesn't really care about >>> distance stuff, or only wants to stick with 'simple' distance metric cases >>> (e.g. a few thousand vectors, etc.). So, maybe it'd be better to start a >>> completely new python package - which would probably be a lot easier to >>> develop as I'd have a lot more flexibility (e.g. to depend on other >>> packages, and not have to worry about breaking the scipy API etc.). On the >>> other hand (as discussed in the latest comment on the PR), that might not >>> be best - it might never get used/maintained etc. >>> >>> So, what does the community think is the best approach? I've got too >>> little context of what scipy is and what it's aiming for, and I don't want >>> to head off on the wrong tack. Comments on any of the other implied >>> questions would also be appreciated. >>> >>> Thanks, >>> >>> kodonnell >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev(a)python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev(a)python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >

2 1

CI issues
by Andrew Nelson 16 Sep '18

16 Sep '18

Hi all, there's currently some CI issues with various entries in the appveyor/travis test matrix failing. It's the same entries everytime, so I'm wondering if it's related to 4e743d57bd53f22ae5d30a567df3b565badac890 ? The error messages on appveyor and travis are similar, it looks like various imports aren't working properly. For an example see https://travis-ci.org/scipy/scipy/jobs/425548905.

4 6

Randomized linear algebra functionality
by Touqir Sajed 16 Sep '18

16 Sep '18

There seems to be some interest in implementing randomized linear algebra algorithms such as JL transformation, low-rank approximation, etc as written in https://github.com/scipy/scipy/wiki/GSoC-2018-project-ideas <https://ml-trckr.com/link/https%3A%2F%2Fgithub.com%2Fscipy%2Fscipy%2Fwiki%2…> . To the best of my knowledge, they are not implemented yet. If there still exists interests I would like to implement the Fast JL transformation and the randomized SVD for low-rank approximation. The pseudo code of Fast JL transformation is given here: https://pdfs.semanticscholar.org/ccf0/1e0375d11df5335b645161b4833e32380d89.… <https://ml-trckr.com/link/https%3A%2F%2Fpdfs.semanticscholar.org%2Fccf0%2F1…> . The randomized SVD is based on https://arxiv.org/pdf/0909.4061.pdf <https://ml-trckr.com/link/https%3A%2F%2Farxiv.org%2Fpdf%2F0909.4061.pdf/cBz…> . -- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 <https://ml-trckr.com/link/https%3A%2F%2Fca.linkedin.com%2Fin%2Ftouqir-sajed…>

5 7

On robust mean estimators
by Touqir Sajed 15 Sep '18

15 Sep '18

Talking about robust mean estimators, there is the trimmed mean estimator in scipy which unfortunately can be biased if the distribution is not symmetric. The median of the means estimator can be more robust and at the same time be unbiased. Should the median of the means estimator be added to scipy? The idea is to separate the whole sample set into small sized minibatches of size k and then outputting the median of their(k minibatches) mean values. -- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 <https://ml-trckr.com/link/https%3A%2F%2Fca.linkedin.com%2Fin%2Ftouqir-sajed…>

1 0

Large Scale Optimization algorithm implementation
by Touqir Sajed 15 Sep '18

15 Sep '18

Hi, So, the most efficient optimization algorithm implemented in Scipy is LBFGS-B in terms of memory consumption and runtime speed. It would be good to see algorithms suited for large scale optimization in scipy for the scenarios when Gradient computation is quite expensive and/or Number of variables is huge. SVRG and (SVRG+LBFGS) can both attain much better performance than LBFGS-B for this scenario. The links: (SVRG) : https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-… <https://ml-trckr.com/link/https%3A%2F%2Fpapers.nips.cc%2Fpaper%2F4937-accel…> (SVRG + LBFGS) : http://opt-ml.org/papers/OPT2015_paper_41.pdf <https://ml-trckr.com/link/http%3A%2F%2Fopt-ml.org%2Fpapers%2FOPT2015_paper_…> What do you think? Cheers, Touqir -- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 <https://ml-trckr.com/link/https%3A%2F%2Fca.linkedin.com%2Fin%2Ftouqir-sajed…>

2 1

ANN: PyData/Sparse 0.4.1
by Hameer Abbasi 12 Sep '18

12 Sep '18

============================= Announcing PyData/Sparse 0.4.1 ============================= (Apologies for the cross-posting) Hi everyone, This is a performance, bug-fix and feature release. The changelog can be seen at https://sparse.pydata.org/en/latest/changelog.html Highlights include: Faux In-place operations Mixed ndarray-sparse operations Fill-values other than zero Misc support for different functions What’s PyData/Sparse? ——————————— PyData/Sparse is a an N-dimensional sparse array library. It’s compatible with NumPy and follows the ndarray interface as closely as possible. It depends on NumPy, SciPy and Numba. Where can I find PyData/Sparse? ——————————————— The project is available on GitHub at https://github.com/pydata/sparse and is available on PyPI and conda-forge as “sparse”. Documentation is hosted as https://sparse.pydata.org. Best Regards, Hameer Abbasi

1 0

General discussion on parallelisation
by Andrew Nelson 09 Sep '18

09 Sep '18

As part of https://github.com/scipy/scipy/pull/8259 I'm proposing that a `workers` keyword is added to optimize.differential_evolution to parallelise some computation. The proposal is that: 1. the workers keyword accepts either an integer or an object with a map-like method. 2. If an integer is supplied then the parallelisation is taken care of by scipy (more on that later), with -1 signifying that all processors are to be used. 3. If an object with a map-like method is supplied, e.g. `multiprocessing.Pool.map`, `mpi4py.futures.MPIPoolExecutor.map`, etc, then the parallelisation is taken care of by that object. This allows the user to specify the parallelisation configuration for their problem. 4. If workers=1, then computation will be done by the builtin `map` function. Now we come to the under the hood part. I've written something called PoolWrapper ( https://github.com/andyfaff/scipy/blob/b14bb513c0ffb9807a67663d39b9ab399375…) which wraps `multiprocessing.Pool` to achieve the behaviour outlined above. It can be used as a context manager, or the user of the object can decide when to close the resources opened by PoolWrapper. I've looked at using joblib instead of PoolWrapper and it seems useful but it doesn't have a couple of bits of functionality that are needed for this specific problem: viz: 5. joblib.Parallel doesn't have a map method (desirable to allow 3) so a small wrapper would have to be created anyway. 6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the Parallel object is `__call__`ed. This leads to significant overhead. One can use the Parallel object with a context manager, which allows reuse of the Pool, but I don't think that's do-able in the context of using the DifferentialEvolutionSolver (DES) object as an iterator: >>> solver = DifferentialEvolutionSolver(func, bounds) >>> # use DES object as an iterator >>> for it in solver: ... res = next(solver) >>> print(res) >>> # use the DES.solve method >>> res = solver.solve() Whilst the DES object is not currently public (it's called by the differential_evolution function) it would be nice to expose it in the future, and people will want to use both approaches. Unfortunately with the first approach if we used joblib.Parallel we'd have to use Parallel.__call__ in DES.next() which has the overhead of creating/destroying Pools. For efficient use of resources the Pool should persist for the lifetime of the DES object. I also looked into `concurrent.futures.ProcessPoolExecutor`, but it's not available for Python 2.7. The purpose of this email is to elicit feedback for developing parallelisation strategy for scipy - what does the public interface look like, what does scipy do under the hood? Under the hood I think a mixture of PoolWrapper and joblib.Parallel could be used (with scipy vendoring joblib). A. -- _____________________________________ Dr. Andrew Nelson _____________________________________

5 18