Mailman 3 April 2018 - SciPy-Dev

SciPy 1.1.0 release schedule
by Pauli Virtanen 10 Apr '18

10 Apr '18

Hi, 2018-03-25 17:45 -0700, Ralf Gommers wrote: > It's 5 months after the 1.0 release, so it's time to start planning > for > 1.1. There's still quite a few open issues and PRs marked for 1.1.0 > (see > https://github.com/scipy/scipy/milestone/34), but not many that seem > blocking or really difficult to resolve. So I'd like to propose the > following schedule: > > April 11: branch 1.1.x > April 13: rc1 > April 27: rc2 (if needed) > May 4: final release Just a friendly heads up/reminder: staying roughly with the schedule proposed earlier, I'll branch 1.1.x on this Thursday, and work towards getting 1.1.0rc1 out during the weekend. If there are features almost ready to be merged, there's a few days before the branch point. After that, it's mainly bugfixes that'll get to 1.1.0. There are several feature PRs marked in the milestone and elsewhere that in principle might use help with reviewing/status triaging --- I've been working toward getting low-hanging ones in, but there are several which I'm not sure about. Nothing seems blocking, though. If there are blocking problems that need to be addressed currently in master before 1.1.0, please remind about them (I'm not aware of such issues though). Best, Pauli --

2 1

Re: [SciPy-Dev] [scipy/scipy] ENH: Add PDF, CDF and parameter estimation for Stable Distributions (#7374)
by Blair Azzopardi 10 Apr '18

10 Apr '18

On 10 April 2018 at 12:34, Andrea Karlova <andrea.karlova(a)gmail.com> wrote: > On 8 April 2018 at 18:05, Blair Azzopardi <blairuk(a)gmail.com> wrote: > >> > On 8 April 2018 at 12:49, Andrea Karlova <andrea.karlova(a)gmail.com> >> wrote: >> >> On 8 April 2018 at 05:21, Blair Azzopardi <blairuk(a)gmail.com> wrote: >> >>> On 8 April 2018 at 01:49, Andrea Karlova <andrea.karlova(a)gmail.com> >> wrote: >> >>>> On 28 November 2017 at 18:33, Blair Azzopardi <blairuk(a)gmail.com> >> wrote: >> >>>>> On Tue, 28 Nov 2017, 16:01 Andrea Karlova, < >> andrea.karlova(a)gmail.com> wrote: >> >>>>>> On 28 November 2017 at 13:17, Blair Azzopardi <blairuk(a)gmail.com> >> wrote: >> >>>>>>> ---------- Forwarded message --------- >> >>>>>>> From: An <notifications(a)github.com> >> >>>>>>> Date: Tue, 28 Nov 2017, 10:00 >> >>>>>>> Subject: Re: [scipy/scipy] ENH: Add PDF, CDF and parameter >> estimation for Stable Distributions (#7374) >> >>>>>>> To: scipy/scipy <scipy(a)noreply.github.com> >> >>>>>>> >> >>>>>>> Blair, is there a way to have a chat via email? >> >>>>>>> >> >>>>>>> On 24 November 2017 at 20:38, Blair Azzopardi < >> notifications(a)github.com> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>> Hi @an81 <https://github.com/an81>. Thank you for the 550+ page >> book. >> >>>>>>> Please can you be a bit more specific? Some sample code goes a >> long way >> >>>>>>> too. Also can you perhaps test the existing code and highlight >> where the >> >>>>>>> Gibbs effect might be more prominent? eg low alpha etc; perhaps >> this can be >> >>>>>>> just documented with a recommendation that users use quad in >> these cases >> >>>>>>> (already in code). This is until better implementation is >> available. >> >>>>>>> — >> >>>>>>> You are receiving this because you were mentioned. >> >>>>>>> Reply to this email directly, view it on GitHub >> >>>>>>> <https://github.com/scipy/scipy/pull/7374#issuecomment-346893569>, >> or mute >> >>>>>>> the thread >> >>>>>>> <https://github.com/notifications/unsubscribe-auth/AIfE8VNjH >> dqANvYfUG8Gg6feKb5np_kLks5s5ylhgaJpZM4NQBiP> >> >>>>>> >> >>>>>> Hi Andrea >> >>>>>> >> >>>>>> I hope you're well and yes no problem talking chatting via email. >> Actually makes sense. >> >>>>>> >> >>>>>> Also, I hope you don't mind me messaging you via an email address >> found in a previous comment. >> >>>>>> >> >>>>>> I took a look at that book you linked to but unfortunately I don't >> have enough time to process it currently. I will read it in time mind you. >> I've found a shorter paper that might offer similar suggestions to yours: >> >>>>>> >> >>>>>> http://prac.im.pwr.edu.pl/~hugo/publ/SFB2005-008_Borak_Haerd >> le_Weron.pdf >> >>>>>> >> >>>>>> Although I'm not 100% sure as it doesn't mention mejer g >> functions. What are your thoughts? >> >>>>>> >> >>>>>> Kind regards >> >>>>>> Blair >> >>>>> >> >>>>> Hi Blair, >> >>>>> >> >>>>> thats great. >> >>>>> Thx for your email. >> >>>>> >> >>>>> I ll give you access to my Dropbox folder where I have materials >> relevant to stable laws. >> >>>>> For working with stable laws I found really useful to understand >> >>>>> and actively switch between different parametrizations of the >> characteristic exponent. >> >>>>> Zolotarev and others would have polenty of parametrization and each >> of them is helpful >> >>>>> for different task. >> >>>>> >> >>>>> I guess I can share with you my python code on the github, >> >>>>> you can contribute to it if you would feel like so >> >>>>> and then we can just plug it into scipy lib. >> >>>>> >> >>>>> Is it ok to use this gmail account for sending you invitation to >> Dropbox? >> >>>>> >> >>>>> Thx, >> >>>>> Kind regards, >> >>>>> >> >>>>> Andrea >> >>>> >> >>>> Hi Andrea >> >>>> >> >>>> Yes, please do share with this email address. >> >>>> >> >>>> Is it possible you could commit your changes to the existing PR I've >> already set up? I believe this is possible by forking my repo (script fork) >> and committing to that. The PR includes parameter estimation and some >> general framework changes around this distribution too. Also I feel there >> are still use cases where it's useful keep fft method. I've added you a >> collaborator on my fork. >> >>>> >> >>>> https://github.com/bsdz/scipy >> >>>> >> >>>> Kind regards >> >>>> Blair >> >>> >> >>> Hi Blair, >> >>> >> >>> hope you are well. >> >>> I was looking into your code on stable laws in scipy. >> >>> I ll be running a group of 4 people at hackaton which is organized >> by AHL in 2 weeks time >> >>> and I am planning to revise and add more code into stable laws >> implementation. >> >>> >> >>> I was wondering if you can give me some quick update on what is done >> so far >> >>> and what are urgent issues at this point according to your opinion? >> >>> I have some ideas what I would like add, also the documentation page >> needs to be written, >> >>> but just a quick check on what methods you implemented for >> calculating of >> >>> pdfs, cdfs, and parameters estimates and how it went? >> >>> >> >>> Thanks a lot. >> >>> Kind regards, >> >>> Andrea >> >>> >> >> >> >> Hi Andrea >> >> >> >> Thanks for your email. I'm cc-ing this email along with out previous >> correspondence to the scipy-dev mailing list (as bottom post) . Please >> direct all your future messages here. >> >> >> >> It's good to hear you plan to improve my PR (coincidentally at my old >> employer Man, although at the time they weren't interested in Stable laws). >> >> >> >> I did implement your suggestions to use Zolotarev's formulations. All >> the code is under https://github.com/scipy/scipy/pull/7374 as you are >> aware. >> >> >> >> You could perhaps look at improving documentation, adding more / >> improving tests and any optimisations you can think of. >> >> >> >> Either I can re-add you as a collaborator to my repo (I removed you >> when I hadn't heard back from you previously) or you can email me a patch >> and I'll integrate it into PR 7374. >> >> >> >> Thanks >> >> Blair >> > >> > Hi Blair, >> > >> > yes, if you could add me to your repository, >> > so I can look into what is done and start to test your code, >> > so that I can look into some patches, that would be great. >> >> Hi Andrea >> >> Actually, just to be cautious. Could you fork my scipy repo (it has all >> the changes there) and then create a PR into that repo? I can then merge it >> and it should push upstream into my main PR. >> > > Sure, no worries. > > >> >> > How does your code performs in neighbourhood of alpha = 1? >> > How does it perform when the asymetry parameter beta gets close to its >> boundary values? >> >> Please run the code and check. My testing took samples from Nolan's >> public domain executables and tested against those values. Although I >> tested against a range of parameters I might not have tested those ranges >> specifically. It's all in the unit tests in the PR, see test class >> TestLevyStable. Extending the testing here would be helpful. >> > > Ok. > > >> >> > Do you have somewhere in Latex notes on your fft implementation, >> > with some theoretical investigation of the performance of the method? >> >> This is all in the code documentation along with references. >> > > Ok, so the answer to my question is no and there is only available info in > the comments in the code, right? > It's in the code, please look at function _pdf_from_cf_with_fft(), it has a docstring that refers to "[MS]". The full reference is then in the main docstring along with page numbers. https://github.com/scipy/scipy/pull/7374/files#diff-6e4287784ff3e7a5c6d89d4… > >> > I am planning to write a research paper about the implementation in the >> Scipy, which I ll put on my ArXiv, >> > and to which I ll refer in the documentation, so that if people are >> interested in the performance, they can look into paper. >> > So I ll be listing the co-authors of this piece of code there. >> > If there are other people who worked on the stable laws implementation >> in Scipy, it would be nice if I can have the list of them. >> > Also if you have some note on fft, I can join it into my research note >> and add you as a coauthor of the note, >> > depends upon you. >> >> So far, I'm the sole author of this PR. Again, all papers and references >> are in the PR. Please please do look at the code. >> > Ok, done. Thanks. > > >> >> > >> > Kind regards, >> > Andrea >> >> Just some additional points, you need to subscribe to the scipy-dev >> mailing list for your messages to get through. Also, the mailing list uses >> bottom posting for email messages; that means, all replies are bottom of >> original message (not the top.) >> >> > Sure. > Btw, it seems your messages aren't getting through to scip-dev mailing list. You can subscribe here: https://mail.python.org/mailman/listinfo/scipy-dev > > >> Look forward to your contribution - thanks! >> > > Me too :) > >> >> Blair >> >> >

1 0

Re: [SciPy-Dev] Dev suggestion: expand signal.slepian functions (Virginia Frey)
by 3ukip0s02＠sneakemail.com 10 Apr '18

10 Apr '18

On Sun, 8 Apr 2018 08:14:47 +1000, Virginia Frey wrote: > The current implementation of Slepians in scipy's signal module only > offers the zeroth order, so it is not possible to use for e.g. the > multitaper spectral analysis. In addition to that, because of the way > the eigensystem is solved, it breaks with a MemoryError when using high > numbers of points (i.e. a few thousand). The request for high numbers of > points is not unreasonable: people do take long time-traces and the FFT > makes it possible to compute Fourier transforms on those. > In SciPy 1.1.0, windows.slepian will be deprecated and replaced by windows.dpss : https://scipy.github.io/devdocs/generated/scipy.signal.windows.dpss.html https://github.com/scipy/scipy/pull/7802 https://github.com/scipy/scipy/issues/4354 Does that do what you want? I have working code for this, which is based on the implementation > suggested in [5] but is in pure Python. Aren't there copyright/license issues with using code from Numerical Recipes? > Once we have the full set of Slepians available, from there we (or I) > could also implement the multitaper spectral analysis technique > That would be useful! > So, what do you guys think of my proposal? If you like it, please advise > me on how to proceed as this is the first time I'm planning to > contribute. From what I took of the web page, I'll start by downloading > the repo - and then should I make a new branch? I'm also happy to send > some code through via email for your review first. Can you read through https://github.com/scipy/scipy/pull/7802 and comment if you see a problem with it, or it doesn't do all that you would want it to do? Also we were a little confused about how best to normalize the amplitude of the functions. In your work, do you use the L2 norm?

2 1

Re: [SciPy-Dev] [scipy/scipy] ENH: Add PDF, CDF and parameter estimation for Stable Distributions (#7374)
by Blair Azzopardi 08 Apr '18

08 Apr '18

> On 8 April 2018 at 12:49, Andrea Karlova <andrea.karlova(a)gmail.com> wrote: >> On 8 April 2018 at 05:21, Blair Azzopardi <blairuk(a)gmail.com> wrote: >>> On 8 April 2018 at 01:49, Andrea Karlova <andrea.karlova(a)gmail.com> wrote: >>>> On 28 November 2017 at 18:33, Blair Azzopardi <blairuk(a)gmail.com> wrote: >>>>> On Tue, 28 Nov 2017, 16:01 Andrea Karlova, <andrea.karlova(a)gmail.com> wrote: >>>>>> On 28 November 2017 at 13:17, Blair Azzopardi <blairuk(a)gmail.com> wrote: >>>>>>> ---------- Forwarded message --------- >>>>>>> From: An <notifications(a)github.com> >>>>>>> Date: Tue, 28 Nov 2017, 10:00 >>>>>>> Subject: Re: [scipy/scipy] ENH: Add PDF, CDF and parameter estimation for Stable Distributions (#7374) >>>>>>> To: scipy/scipy <scipy(a)noreply.github.com> >>>>>>> >>>>>>> Blair, is there a way to have a chat via email? >>>>>>> >>>>>>> On 24 November 2017 at 20:38, Blair Azzopardi < notifications(a)github.com> >>>>>>> wrote: >>>>>>> >>>>>>> Hi @an81 <https://github.com/an81>. Thank you for the 550+ page book. >>>>>>> Please can you be a bit more specific? Some sample code goes a long way >>>>>>> too. Also can you perhaps test the existing code and highlight where the >>>>>>> Gibbs effect might be more prominent? eg low alpha etc; perhaps this can be >>>>>>> just documented with a recommendation that users use quad in these cases >>>>>>> (already in code). This is until better implementation is available. >>>>>>> — >>>>>>> You are receiving this because you were mentioned. >>>>>>> Reply to this email directly, view it on GitHub >>>>>>> <https://github.com/scipy/scipy/pull/7374#issuecomment-346893569>, or mute >>>>>>> the thread >>>>>>> < https://github.com/notifications/unsubscribe-auth/AIfE8VNjHdqANvYfUG8Gg6feK… > >>>>>> >>>>>> Hi Andrea >>>>>> >>>>>> I hope you're well and yes no problem talking chatting via email. Actually makes sense. >>>>>> >>>>>> Also, I hope you don't mind me messaging you via an email address found in a previous comment. >>>>>> >>>>>> I took a look at that book you linked to but unfortunately I don't have enough time to process it currently. I will read it in time mind you. I've found a shorter paper that might offer similar suggestions to yours: >>>>>> >>>>>> http://prac.im.pwr.edu.pl/~hugo/publ/SFB2005-008_Borak_Haerdle_Weron.pdf >>>>>> >>>>>> Although I'm not 100% sure as it doesn't mention mejer g functions. What are your thoughts? >>>>>> >>>>>> Kind regards >>>>>> Blair >>>>> >>>>> Hi Blair, >>>>> >>>>> thats great. >>>>> Thx for your email. >>>>> >>>>> I ll give you access to my Dropbox folder where I have materials relevant to stable laws. >>>>> For working with stable laws I found really useful to understand >>>>> and actively switch between different parametrizations of the characteristic exponent. >>>>> Zolotarev and others would have polenty of parametrization and each of them is helpful >>>>> for different task. >>>>> >>>>> I guess I can share with you my python code on the github, >>>>> you can contribute to it if you would feel like so >>>>> and then we can just plug it into scipy lib. >>>>> >>>>> Is it ok to use this gmail account for sending you invitation to Dropbox? >>>>> >>>>> Thx, >>>>> Kind regards, >>>>> >>>>> Andrea >>>> >>>> Hi Andrea >>>> >>>> Yes, please do share with this email address. >>>> >>>> Is it possible you could commit your changes to the existing PR I've already set up? I believe this is possible by forking my repo (script fork) and committing to that. The PR includes parameter estimation and some general framework changes around this distribution too. Also I feel there are still use cases where it's useful keep fft method. I've added you a collaborator on my fork. >>>> >>>> https://github.com/bsdz/scipy >>>> >>>> Kind regards >>>> Blair >>> >>> Hi Blair, >>> >>> hope you are well. >>> I was looking into your code on stable laws in scipy. >>> I ll be running a group of 4 people at hackaton which is organized by AHL in 2 weeks time >>> and I am planning to revise and add more code into stable laws implementation. >>> >>> I was wondering if you can give me some quick update on what is done so far >>> and what are urgent issues at this point according to your opinion? >>> I have some ideas what I would like add, also the documentation page needs to be written, >>> but just a quick check on what methods you implemented for calculating of >>> pdfs, cdfs, and parameters estimates and how it went? >>> >>> Thanks a lot. >>> Kind regards, >>> Andrea >>> >> >> Hi Andrea >> >> Thanks for your email. I'm cc-ing this email along with out previous correspondence to the scipy-dev mailing list (as bottom post) . Please direct all your future messages here. >> >> It's good to hear you plan to improve my PR (coincidentally at my old employer Man, although at the time they weren't interested in Stable laws). >> >> I did implement your suggestions to use Zolotarev's formulations. All the code is under https://github.com/scipy/scipy/pull/7374 as you are aware. >> >> You could perhaps look at improving documentation, adding more / improving tests and any optimisations you can think of. >> >> Either I can re-add you as a collaborator to my repo (I removed you when I hadn't heard back from you previously) or you can email me a patch and I'll integrate it into PR 7374. >> >> Thanks >> Blair > > Hi Blair, > > yes, if you could add me to your repository, > so I can look into what is done and start to test your code, > so that I can look into some patches, that would be great. Hi Andrea Actually, just to be cautious. Could you fork my scipy repo (it has all the changes there) and then create a PR into that repo? I can then merge it and it should push upstream into my main PR. > How does your code performs in neighbourhood of alpha = 1? > How does it perform when the asymetry parameter beta gets close to its boundary values? Please run the code and check. My testing took samples from Nolan's public domain executables and tested against those values. Although I tested against a range of parameters I might not have tested those ranges specifically. It's all in the unit tests in the PR, see test class TestLevyStable. Extending the testing here would be helpful. > Do you have somewhere in Latex notes on your fft implementation, > with some theoretical investigation of the performance of the method? This is all in the code documentation along with references. > I am planning to write a research paper about the implementation in the Scipy, which I ll put on my ArXiv, > and to which I ll refer in the documentation, so that if people are interested in the performance, they can look into paper. > So I ll be listing the co-authors of this piece of code there. > If there are other people who worked on the stable laws implementation in Scipy, it would be nice if I can have the list of them. > Also if you have some note on fft, I can join it into my research note and add you as a coauthor of the note, > depends upon you. So far, I'm the sole author of this PR. Again, all papers and references are in the PR. Please please do look at the code. > > Kind regards, > Andrea Just some additional points, you need to subscribe to the scipy-dev mailing list for your messages to get through. Also, the mailing list uses bottom posting for email messages; that means, all replies are bottom of original message (not the top.) Look forward to your contribution - thanks! Blair

1 0

ENH: WIP: Cython optimize · Issue #8431 · scipy/scipy
by Mark Alexander Mikofski 08 Apr '18

08 Apr '18

I've started work on a cython_optimize.zeros package to address the need for fast looping of scalar zero methods over arrays, see issues 8354 and 7242. https://github.com/scipy/scipy/pull/8431 This PR is still a work in progress. I wanted to get feedback on the API(s) that would be most useful. I don't think there should be too many versions but it should be something widely acceptible. Sorry if my proposals are a bit premature or insufficiently developed, but I wanted to start the discussion. Here are some ideas to start with: Option 1: ctypedef double (*callback_type_tup)(double, tuple) - takes a double and a Python tuple with extra arguments, this is the only interface implemented so far, and only for bisect and a super-lite version of newton, it is modeled after the existing Python C-API methods in zeros.c for bisect and other solvers, see description in cython_optimize/__init__.py Option 2: ctypedef double (*callback_type_struct)(double, void*) - takes a double and a structure that contains a structure with the extra arguments, uses a wrapper to extract the extra arguments and call the function Option 3: ctypedef double (*callback_type)(double, void*) - takes a double and a structure that contains the extra arguments, this implementation does not have a wrapper function to extract the args and call the function Option 4: ctypedef double (*callback_type_array)(double, void*) - takes a double and a structure that contains an array with the extra arguments, this method uses a wrapper to extract the extra arguments and call the function I guess I'm leaning toward either option 2 or 3, but perhaps there are other API(s) that are more desirable? Also there are other questions that this PR raises like: is it okay to split up Newton, Halley, and secant? And is it okay to *not* return a `RootsResult` object from the Ridder, Brent, and bisect methods, or should the API(s) match their Python equivalents? Should there be a numpy cython API for super fast looping with native numpy arrays (or other array type like what?) Also note there is a parallel effort in PR 8357 which uses numpy but only with scipy.optimize.zeros.newton. IMHO both of these efforts are desirable and do not compete against each other. Thanks for your feedback!

1 0

Re: [SciPy-Dev] [scipy/scipy] ENH: Add PDF, CDF and parameter estimation for Stable Distributions (#7374)
by Blair Azzopardi 08 Apr '18

08 Apr '18

> On 8 April 2018 at 01:49, Andrea Karlova <andrea.karlova(a)gmail.com> wrote: > >> On 28 November 2017 at 18:33, Blair Azzopardi <blairuk(a)gmail.com> wrote: >> >>> On Tue, 28 Nov 2017, 16:01 Andrea Karlova, <andrea.karlova(a)gmail.com> wrote: >>> >>>> On 28 November 2017 at 13:17, Blair Azzopardi <blairuk(a)gmail.com> wrote: >>>> >>>>> ---------- Forwarded message --------- >>>>> From: An <notifications(a)github.com> >>>>> Date: Tue, 28 Nov 2017, 10:00 >>>>> Subject: Re: [scipy/scipy] ENH: Add PDF, CDF and parameter estimation for Stable Distributions (#7374) >>>>> To: scipy/scipy <scipy(a)noreply.github.com> >>>>> >>>>> >>>>> Blair, is there a way to have a chat via email? >>>>> >>>>> On 24 November 2017 at 20:38, Blair Azzopardi < notifications(a)github.com> >>>>> wrote: >>>>> >>>>> Hi @an81 <https://github.com/an81>. Thank you for the 550+ page book. >>>>> Please can you be a bit more specific? Some sample code goes a long way >>>>> too. Also can you perhaps test the existing code and highlight where the >>>>> Gibbs effect might be more prominent? eg low alpha etc; perhaps this can be >>>>> just documented with a recommendation that users use quad in these cases >>>>> (already in code). This is until better implementation is available. >>>>> >>>>> — >>>>> You are receiving this because you were mentioned. >>>>> Reply to this email directly, view it on GitHub >>>>> <https://github.com/scipy/scipy/pull/7374#issuecomment-346893569>, or mute >>>>> the thread >>>>> <https://github.com/notifications/unsubscribe-auth/ AIfE8VNjHdqANvYfUG8Gg6feKb5np_kLks5s5ylhgaJpZM4NQBiP> >>>> >>>> Hi Andrea >>>> >>>> I hope you're well and yes no problem talking chatting via email. Actually makes sense. >>>> >>>> Also, I hope you don't mind me messaging you via an email address found in a previous comment. >>>> >>>> I took a look at that book you linked to but unfortunately I don't have enough time to process it currently. I will read it in time mind you. I've found a shorter paper that might offer similar suggestions to yours: >>>> >>>> http://prac.im.pwr.edu.pl/~hugo/publ/SFB2005-008_Borak_ Haerdle_Weron.pdf >>>> >>>> Although I'm not 100% sure as it doesn't mention mejer g functions. What are your thoughts? >>>> >>>> Kind regards >>>> Blair >>> >>> Hi Blair, >>> >>> thats great. >>> Thx for your email. >>> >>> I ll give you access to my Dropbox folder where I have materials relevant to stable laws. >>> For working with stable laws I found really useful to understand >>> and actively switch between different parametrizations of the characteristic exponent. >>> Zolotarev and others would have polenty of parametrization and each of them is helpful >>> for different task. >>> >>> I guess I can share with you my python code on the github, >>> you can contribute to it if you would feel like so >>> and then we can just plug it into scipy lib. >>> >>> Is it ok to use this gmail account for sending you invitation to Dropbox? >>> >>> Thx, >>> Kind regards, >>> >>> Andrea >> >> Hi Andrea >> >> Yes, please do share with this email address. >> >> Is it possible you could commit your changes to the existing PR I've already set up? I believe this is possible by forking my repo (script fork) and committing to that. The PR includes parameter estimation and some general framework changes around this distribution too. Also I feel there are still use cases where it's useful keep fft method. I've added you a collaborator on my fork. >> >> https://github.com/bsdz/scipy >> >> Kind regards >> Blair > > Hi Blair, > > hope you are well. > I was looking into your code on stable laws in scipy. > I ll be running a group of 4 people at hackaton which is organized by AHL in 2 weeks time > and I am planning to revise and add more code into stable laws implementation. > > I was wondering if you can give me some quick update on what is done so far > and what are urgent issues at this point according to your opinion? > I have some ideas what I would like add, also the documentation page needs to be written, > but just a quick check on what methods you implemented for calculating of > pdfs, cdfs, and parameters estimates and how it went? > > Thanks a lot. > Kind regards, > Andrea > Hi Andrea Thanks for your email. I'm cc-ing this email along with out previous correspondence to the scipy-dev mailing list (as bottom post) . Please direct all your future messages here. It's good to hear you plan to improve my PR (coincidentally at my old employer Man, although at the time they weren't interested in Stable laws). I did implement your suggestions to use Zolotarev's formulations. All the code is under https://github.com/scipy/scipy/pull/7374 as you are aware. You could perhaps look at improving documentation, adding more / improving tests and any optimisations you can think of. Either I can re-add you as a collaborator to my repo (I removed you when I hadn't heard back from you previously) or you can email me a patch and I'll integrate it into PR 7374. Thanks Blair

1 0

Dev suggestion: expand signal.slepian functions
by Virginia Frey 08 Apr '18

08 Apr '18

Hi everyone, In my work I do a lot with Slepian functions, and I find scipy's current capability to make those functions very limited, so I propose to expand it. I'll start with a little motivation for why I think Slepians are great and why it's worth to expand scipy's Slepian capabilities, and then I propose the changes I would make and the new code I would add. *1. Motivation* Slepians (or, using the official term: "discrete, prolate, spheroidal sequences (DPSS)") are the provably optimal solution to the so-called spectral concentration problem [1], so they have a lot of application in frequency analysis [2] and recently my group and I started working on them in the context of ambient noise analysis for quantum computers [3, 4]. Slepians are the eigensolutions of the so-called Toelpitz matrix and there are therefor many of them. The different eigenvectors are called "orders" of the Slepians. In frequency analysis and in my work, it is very common to use multiple orders because (given the right bandwidth parameter) a set of Slepian orders forms a maximally spectrally concentrated "rectangle" in frequency space. Based on those Slepian orders, Thompson [2] developed the "multitaper" spectral reconstruction technique, in which multiple orders are applied as window functions and the optimal guess for the frequency spectrum is obtained by combining individual order estimates. *2. What's the current implementation* The current implementation of Slepians in scipy's signal module only offers the zeroth order, so it is not possible to use for e.g. the multitaper spectral analysis. In addition to that, because of the way the eigensystem is solved, it breaks with a MemoryError when using high numbers of points (i.e. a few thousand). The request for high numbers of points is not unreasonable: people do take long time-traces and the FFT makes it possible to compute Fourier transforms on those. *3. How could it be expanded?* I suggest to re-implement the existing function in order to: a) calculate a variable number of Slepian orders b) allow for higher point numbers by using a Newtonian solver for the eigenvalues I have working code for this, which is based on the implementation suggested in [5] but is in pure Python. It consists of three functions: the main Slepian function plus two little helper functions which would not be exposed to the user. Most of the code is linted and pep8'd but I haven't got unit tests yet. Once we have the full set of Slepians available, from there we (or I) could also implement the multitaper spectral analysis technique, but let's do one step at a time. On a side node, not sure if this is a good argument for expanding the library, but both a Slepian function and a multitaper function exist in Matlab as well. *4. What's next?* So, what do you guys think of my proposal? If you like it, please advise me on how to proceed as this is the first time I'm planning to contribute. From what I took of the web page, I'll start by downloading the repo - and then should I make a new branch? I'm also happy to send some code through via email for your review first. Thanks, and I'm looking forward to hearing back from you, Virginia (from sunny Sydney) ---- References: [1] https://en.wikipedia.org/wiki/Spectral_concentration_problem [2] Thompson, "Spectrum estimation and harmonic analysis". Proc. IEEE 70(9), 1982 DOI: 10.1109/PROC.1982.12433 <https://doi.org/10.1109/PROC.1982.12433> [3] V. Frey et al. "Application of optimal band-limited control protocols to quantum noise sensing", Nature Comms 8, 2189, 2017. DOI: 10.1038/s41467-017-02298-2 [4] L.M. Norris et al. "Optimally band-limited spectroscopy of control noise using a qubit sensor", arXiv preprint at https://arxiv.org/abs/1803.05538 [5] W.H. Press "Numerical Recipes in C++: The Art of Scientific Computing" Cambrige University Press, 3rd Edition

1 0

Re: [SciPy-Dev] New subpackage: scipy.data
by Matt Haberland 06 Apr '18

06 Apr '18

I wanted to extend Charles' comment that in addition to the test problems there are optimization *benchmarks*. For instance, in scipy/benchmarks/ benchmarks/linprog_benchmark_files there are ~90 benchmark problems (all NETLIB LP benchmarks as .npz files) totaling ~12MB. The current linprog benchmark only uses two of them by default. Sounds like these should be moved if space is such a concern. On Wed, Apr 4, 2018 at 7:54 AM, Charles R Harris <charlesr.harris(a)gmail.com> wrote: > > > On Mon, Apr 2, 2018 at 12:50 PM, Warren Weckesser < > warren.weckesser(a)gmail.com> wrote: > >> >> >> On Fri, Mar 30, 2018 at 8:17 PM, Ralf Gommers <ralf.gommers(a)gmail.com> >> wrote: >> >>> >>> >>> On Fri, Mar 30, 2018 at 12:03 PM, Eric Larson <larson.eric.d(a)gmail.com> >>> wrote: >>> >>>> Top-level module for them alone sounds overkill, and I'm not sure if >>>>> discoverability alone is enough. >>>>> >>>> >>>> Fine by me. And if we follow the idea that these should be added >>>> sparingly, we can maintain discoverability without it growing out of >>>> hand by populating the See Also sections of each function. >>>> >>> >>> I agree with this, the 2 images and 1 ECG signal (to be added) that we >>> have doesn't justify a top-level module. We don't want to grow more than >>> the absolute minimum of datasets. The package is already very large, which >>> is problematic in certain cases. E.g. numpy + scipy still fits in the AWS >>> Lambda limit of 50 MB, but there's not much margin. >>> >>> >> >> Note: this is a reply to the thread, and not specifically to Ralf's >> comments (but those are included). >> >> After reading all the replies, the first question that comes to mind is: >> should SciPy have *any* datasets? >> >> I think this question has already been answered: we have had functions >> that return images in scipy.misc for a long time, and I don't recall anyone >> ever suggesting that these be removed. (Well, there was lena(), but I >> don't think anyone had a problem with adding a replacement image.) And the >> pull request for the ECG dataset has been merged (added to scipy.misc), so >> there is current support among the developers for providing datasets. >> >> So the remaining questions are: >> (1) Where do the datasets reside? >> (2) What are the criteria for adding a new datasets? >> >> Here's my 2¢: >> >> (1) Where do the datasets reside? >> >> My preference is to keep all the datasets in the top-level module >> scipy.datasets. Robert preferred this module for discoverability, and I >> agree. By having all the datasets in one place, anyone can easily see what >> is available. Teachers and others developing educational material know >> where to find source material for examples. Developers, too, can easily >> look for examples to use in our docstrings or tutorials. (By the way, >> adding examples to the docstrings of all functions is an ongoing effort: >> https://github.com/scipy/scipy/issues/7168.) >> >> Also, there are many well-known datasets that could be used as examples >> for multiple scipy packages. For a concrete example, a dataset that I >> could see adding to scipy is the Hald cement dataset. SciPy should >> eventually have an implementation of the PCA decomposition, and it could >> conceivably live in scipy.linalg. It would be reasonable to use the Hald >> data in the docstrings of the new PCA function(s) (cf. >> https://www.mathworks.com/help/stats/pca.html). At the same time, the >> Hald data could enrich the docstrings of some functions in scipy.stats. >> >> Similarly, Fisher's iris dataset provides a well-known example that could >> be used in docstrings in both scipy.cluster and scipy.stats. >> >> >> (2) What are the criteria for adding a new datasets? >> >> So far, the only compelling reason I can see to even have datasets is to >> have interesting examples in the docstrings (or at least in our >> tutorials). For example, the docstring for scipy.ndimage.gaussian_filter >> and several other transformations in ndimage use the image returned by >> scipy.misc.ascent(): >> >> https://docs.scipy.org/doc/scipy/reference/generated/scipy. >> ndimage.gaussian_filter.html >> >> I could see the benefit of having well-known datasets such as Fisher's >> iris data, the Hald cement data, and some version of a sunspot activity >> time series, to be used in the docstrings in scipy.stats, scipy.signal, >> scipy.cluster, scipy.linalg, and elsewhere. >> >> Stéfan expressed regret about including datasets in sciki-image. The >> main issue seems to be "bloat". Scikit-image is an image processing >> library, so the datasets used there are likely all images, and there is a >> minimum size for a sample image to be useful as an example. For scipy, we >> already have two images, and I don't think we'll need more. The newly >> added ECG dataset is 116K (which is less than the existing image datasets: >> "ascent.dat" is 515K and "face.dat" is 1.5M). The potential datasets that >> I mentioned above (Hald, iris, sunspots) are all very small. If we are >> conservative about what we include, and focus on datasets chosen >> specifically to demonstrate scipy functionality, we should be able to avoid >> dataset bloat. >> >> This leads to my proposal for the criteria for adding a dataset: >> >> (a) Not too big. The size of a dataset should not exceed $MAX (but I >> don't have a good suggestion for what $MAX should be at the moment). >> (b) The dataset should be well-known, where "well-known" means that the >> dataset is one that is already widely used as an example and many people >> will know it by name (e.g. the iris dataset), or the dataset is a sample of >> a common signal type or format (e.g an ECG signal, or an image such as >> misc.ascent). >> (c) We actually *use* the dataset in one of *our* docstrings or >> tutorials. I don't think our datasets package should become a repository >> of interesting scientific data with no connection to the scipy code. Its >> purpose should be to enrich our documentation. (Note that by this >> criterion, the recently added ECG signal would not qualify!) >> >> To summarize: I'm in favor scipy.datasets, a conservatively curated >> subpackage containing well-known datasets. >> >> > There are also some standard functions used for testing optimization. I > wonder if it would be reasonable to make those public? > > Chuck > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev(a)python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > -- Matt Haberland Assistant Adjunct Professor in the Program in Computing Department of Mathematics 6617A Math Sciences Building, UCLA

2 1

TST: add benchmark for newton, secant, halley · Issue #8587 · scipy/scipy
by Mark Alexander Mikofski 05 Apr '18

05 Apr '18

Hi all, I've added some airspeed velocity benchmarks for the Newton methods in scipy.optimize.zeros. You can compare them to the c-language zeros methods like brentq. The 1st test function "f1", a parabola, is similar to the c-language "f2", also a parabola. This is the PR: https://github.com/scipy/scipy/pull/8587 Thank, Mark

1 0

bad formula for chi-squared probability density function
by Phillip Feldman 02 Apr '18

02 Apr '18

In the following page, the formula for the chi-squared density is incorrect: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chi2.html (See the Wikipedia page at https://en.wikipedia.org/wiki/Chi-squared_distribution#Probability_density_… for the correct expression). Phillip

3 3