From phillip.m.feldman at gmail.com Tue Oct 1 02:27:15 2019 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Mon, 30 Sep 2019 23:27:15 -0700 Subject: [SciPy-Dev] New Tutorial on Optimize Help In-Reply-To: References: Message-ID: Providing a function for the derivative is almost always better than finite-difference derivatives unless the cost of evaluating the derivative function is very high. On Mon, Sep 30, 2019 at 4:43 PM Christina Lee wrote: > Hi, > > I'm a SciPy technical writer and am currently rewriting the scipy.optimize > tutorial, focusing on `minimize` right now. While I've gotten a grasp of > the "how", I still want to explain "why". Why choose one option over > another? I could use information from those with more experience. > > A lot of methods are available. Most problems can have BFGS thrown at > them, but I want to explain something for those other cases. Other > situations could have features, like constraints or non-differentiability, > that lend themselves to a specific method. But the module still has a lot > of alternatives. Are they there for academic purposes? Are they the best > for some problems? How could someone find that out? > > For derivatives, users can choose to provide a function or three different > types of finite-difference schemes. > > When is providing a function better than finite-difference derivatives? > For Hessians, approximations are sometimes more efficient. How can we know > in advance if that's true? Is that ever true for gradients? > > How do we choose which finite-difference scheme? `3-point` and `cs` (if it > is the symmetric approximation I think) have higher-order accuracy, but > `cs` uses a point not yet computed. Is `3-point` ever not the way to go? > > Thanks for your expertise, > Christina > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrea.gavana at gmail.com Tue Oct 1 02:41:57 2019 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Tue, 1 Oct 2019 08:41:57 +0200 Subject: [SciPy-Dev] New Tutorial on Optimize Help In-Reply-To: References: Message-ID: Most real life optimization problems are characterized by objective functions that are slow to evaluate - depending on the nature of the problem, it can take from a few seconds to several hours for a single evaluation. For many of them, the gradient is not even available. So, the moment the number of parameters you?re optimizing on exceeds 4 or 5 then gradient-based methods (like BFGS, for example) become inefficient as you need numerical derivatives and so a lot of objective functions evaluations for a single gradient step. Scipy has a lot of powerful, diverse optimization algorithms so in this case you might want to try other approaches, although it would be nice to see implementations like MCS (branch and bound methods) that exist in matlab. I used them in the past and they are quite powerful when gradients are not available or too expensive to compute. Andrea. On Tue, 1 Oct 2019 at 08.27, Phillip Feldman wrote: > Providing a function for the derivative is almost always better than > finite-difference derivatives unless the cost of evaluating the derivative > function is very high. > > On Mon, Sep 30, 2019 at 4:43 PM Christina Lee > wrote: > >> Hi, >> >> I'm a SciPy technical writer and am currently rewriting the >> scipy.optimize tutorial, focusing on `minimize` right now. While I've >> gotten a grasp of the "how", I still want to explain "why". Why choose >> one option over another? I could use information from those with more >> experience. >> >> A lot of methods are available. Most problems can have BFGS thrown at >> them, but I want to explain something for those other cases. Other >> situations could have features, like constraints or non-differentiability, >> that lend themselves to a specific method. But the module still has a lot >> of alternatives. Are they there for academic purposes? Are they the best >> for some problems? How could someone find that out? >> >> For derivatives, users can choose to provide a function or three >> different types of finite-difference schemes. >> >> When is providing a function better than finite-difference derivatives? >> For Hessians, approximations are sometimes more efficient. How can we know >> in advance if that's true? Is that ever true for gradients? >> >> How do we choose which finite-difference scheme? `3-point` and `cs` (if >> it is the symmetric approximation I think) have higher-order accuracy, but >> `cs` uses a point not yet computed. Is `3-point` ever not the way to go? >> >> Thanks for your expertise, >> Christina >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From b.rosa at unistra.fr Tue Oct 1 07:13:36 2019 From: b.rosa at unistra.fr (Benoit Rosa) Date: Tue, 1 Oct 2019 13:13:36 +0200 Subject: [SciPy-Dev] New Tutorial on Optimize Help In-Reply-To: References: Message-ID: Hi Andrea, I did implement a branch-and-bound method in Scipy one or two years ago. I posted here, but nobody seemed to find much interest in incorporating it, so I didn't make a PR. What I implemented is a stochastic branch and bound method which is very efficient for non-convex problems (think of point cloud registration optimized over SE3 when you don't have an initial guess, for instance). References to scientific papers explaining the method in depth are provided in the commit : https://github.com/benoitrosa/scipy/commit/49a2c23b74b69dc4250e20e21db75bd071dfd92d I'd be happy to dig into it again and make a PR if there's interest. Beno?t On 01/10/2019 08:41, Andrea Gavana wrote: > Most real life optimization problems are characterized by objective > functions that are slow to evaluate - depending on the nature of the > problem, it can take from a few seconds to several hours for a single > evaluation. For many of them, the gradient is not even available. > > So, the moment the number of parameters you?re optimizing on exceeds 4 > or 5 then gradient-based methods (like BFGS, for example) become > inefficient as you need numerical derivatives and so a lot of > objective functions evaluations for a single gradient step. > > Scipy has a lot of powerful, diverse optimization algorithms so in > this case you might want to try other approaches, although it would be > nice to see implementations like MCS (branch and bound methods) that > exist in matlab. I used them in the past and they are quite powerful > when gradients are not available or too expensive to compute. > > Andrea. > > > On Tue, 1 Oct 2019 at 08.27, Phillip Feldman > > wrote: > > Providing a function for the derivative is almost always better > than finite-difference derivatives?unless the cost of evaluating > the derivative function is very high. > > On Mon, Sep 30, 2019 at 4:43 PM Christina Lee > > wrote: > > Hi, > > I'm a SciPy technical writer and am currently rewriting the > scipy.optimize tutorial, focusing on `minimize` right now.? > While I've gotten a grasp of the "how", I still want to > explain "why". Why choose one option over another? I could use > information from those with more experience. > > A lot of methods are available.?? Most problems canhave BFGS > thrown at them, but I want to explain something for those > other cases.? Other situations could have features, like > constraints or non-differentiability, that lend themselves to > a specific method. But the module still has a lot of > alternatives.? Are they there for academic purposes?? Are they > the best for some problems? How could someone find that out? > > For derivatives, users can choose to provide a function or > three different types of finite-difference schemes. > > When is providing a function better than finite-difference > derivatives?? For Hessians, approximations are sometimes more > efficient.? How can we know in advance if that's true? Is that > ever true for gradients? > > How do we choose which finite-difference scheme? `3-point` and > `cs` (if it is the symmetric approximation I think) have > higher-order accuracy, but `cs` uses a point not yet > computed.? Is `3-point` ever not the way to go? > > Thanks for your expertise, > Christina > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrea.gavana at gmail.com Tue Oct 1 10:35:50 2019 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Tue, 1 Oct 2019 16:35:50 +0200 Subject: [SciPy-Dev] New Tutorial on Optimize Help In-Reply-To: References: Message-ID: Hi Benoit, On Tue, 1 Oct 2019 at 13.23, Benoit Rosa wrote: > Hi Andrea, > > I did implement a branch-and-bound method in Scipy one or two years ago. I > posted here, but nobody seemed to find much interest in incorporating it, > so I didn't make a PR. > > What I implemented is a stochastic branch and bound method which is very > efficient for non-convex problems (think of point cloud registration > optimized over SE3 when you don't have an initial guess, for instance). > References to scientific papers explaining the method in depth are provided > in the commit : > https://github.com/benoitrosa/scipy/commit/49a2c23b74b69dc4250e20e21db75bd071dfd92d > > I'd be happy to dig into it again and make a PR if there's interest. > > Beno?t > Thanks, it looks very interesting. I hadn?t seen it before, I guess your message a year ago got lost in the maze of my emails... Out of curiosity, did you ever try to run your algorithm against the scipy benchmarks? I still carry (old) results from multiple optimization algorithms here: http://infinity77.net/global_optimization/ http://infinity77.net/global_optimization/multidimensional.html And it would be nice to see how your stochastic branch and bound compares to the rest... Andrea. > > > On 01/10/2019 08:41, Andrea Gavana wrote: > > Most real life optimization problems are characterized by objective > functions that are slow to evaluate - depending on the nature of the > problem, it can take from a few seconds to several hours for a single > evaluation. For many of them, the gradient is not even available. > > So, the moment the number of parameters you?re optimizing on exceeds 4 or > 5 then gradient-based methods (like BFGS, for example) become inefficient > as you need numerical derivatives and so a lot of objective functions > evaluations for a single gradient step. > > Scipy has a lot of powerful, diverse optimization algorithms so in this > case you might want to try other approaches, although it would be nice to > see implementations like MCS (branch and bound methods) that exist in > matlab. I used them in the past and they are quite powerful when gradients > are not available or too expensive to compute. > > Andrea. > > > On Tue, 1 Oct 2019 at 08.27, Phillip Feldman > wrote: > >> Providing a function for the derivative is almost always better than >> finite-difference derivatives unless the cost of evaluating the derivative >> function is very high. >> >> On Mon, Sep 30, 2019 at 4:43 PM Christina Lee >> wrote: >> >>> Hi, >>> >>> I'm a SciPy technical writer and am currently rewriting the >>> scipy.optimize tutorial, focusing on `minimize` right now. While I've >>> gotten a grasp of the "how", I still want to explain "why". Why choose >>> one option over another? I could use information from those with more >>> experience. >>> >>> A lot of methods are available. Most problems can have BFGS thrown at >>> them, but I want to explain something for those other cases. Other >>> situations could have features, like constraints or non-differentiability, >>> that lend themselves to a specific method. But the module still has a lot >>> of alternatives. Are they there for academic purposes? Are they the best >>> for some problems? How could someone find that out? >>> >>> For derivatives, users can choose to provide a function or three >>> different types of finite-difference schemes. >>> >>> When is providing a function better than finite-difference derivatives? >>> For Hessians, approximations are sometimes more efficient. How can we know >>> in advance if that's true? Is that ever true for gradients? >>> >>> How do we choose which finite-difference scheme? `3-point` and `cs` (if >>> it is the symmetric approximation I think) have higher-order accuracy, but >>> `cs` uses a point not yet computed. Is `3-point` ever not the way to go? >>> >>> Thanks for your expertise, >>> Christina >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > > _______________________________________________ > SciPy-Dev mailing listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Wed Oct 2 09:40:51 2019 From: toddrjen at gmail.com (Todd) Date: Wed, 2 Oct 2019 09:40:51 -0400 Subject: [SciPy-Dev] Directory for performance analysis Message-ID: Although scipy has a directory ("benchmarks") for automatically testing how the performance of individual functions has changed, there is no directory as far as I have found for analysis of how particular approaches to the same problem compare. I think it would be worthwhile to have one. For example the "scipy.signal.convolve" function has code to automatically choose whether to use the conventional convolution algorithm or the FFT-based algorithm, based on an analysis someone did. However, I cannot find the code that actually carried out that analysis. This is important because I want to extend that analysis to include another algorithm. There are a bunch of other situations where particular approaches are recommended as being faster in certain situations, or faster by certain amounts in certain situations. So I think that having a directory inside the scipy source tree that includes analyses like that would be worthwhile. I was thinking something like "benchmarks/analysis", although it could also be part of the documentation proper. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlucas7 at vt.edu Wed Oct 2 12:45:37 2019 From: rlucas7 at vt.edu (rlucas7 at vt.edu) Date: Wed, 2 Oct 2019 12:45:37 -0400 Subject: [SciPy-Dev] SciPy-Dev Digest, Vol 192, Issue 3 In-Reply-To: References: Message-ID: > On Oct 2, 2019, at 12:00 PM, scipy-dev-request at python.org wrote: > > Send SciPy-Dev mailing list submissions to > scipy-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request at python.org > > You can reach the person managing the list at > scipy-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Directory for performance analysis (Todd) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 2 Oct 2019 09:40:51 -0400 > From: Todd > To: SciPy Developers List > Subject: [SciPy-Dev] Directory for performance analysis > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Although scipy has a directory ("benchmarks") for automatically testing how > the performance of individual functions has changed, there is no directory > as far as I have found for analysis of how particular approaches to the > same problem compare. I think it would be worthwhile to have one. > > For example the "scipy.signal.convolve" function has code to automatically > choose whether to use the conventional convolution algorithm or the > FFT-based algorithm, based on an analysis someone did. However, I cannot > find the code that actually carried out that analysis. This is important > because I want to extend that analysis to include another algorithm. > > There are a bunch of other situations where particular approaches are > recommended as being faster in certain situations, or faster by certain > amounts in certain situations. > > So I think that having a directory inside the scipy source tree that > includes analyses like that would be worthwhile. I was thinking something > like "benchmarks/analysis", although it could also be part of the > documentation proper. Hi Todd, This sounds interesting. The place where this could be useful to the releases would be in determining defaults. Is this the use you have in mind or is there another use that I?m not thinking of at the moment? The downside to that could be if defaults are changed they could impact end users of SciPy. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > ------------------------------ > > End of SciPy-Dev Digest, Vol 192, Issue 3 > ***************************************** From toddrjen at gmail.com Thu Oct 3 09:04:24 2019 From: toddrjen at gmail.com (Todd) Date: Thu, 3 Oct 2019 09:04:24 -0400 Subject: [SciPy-Dev] Directory for performance analysis In-Reply-To: References: Message-ID: On Wed, Oct 2, 2019 at 1:59 PM wrote: > > From: Todd > > To: SciPy Developers List > > Subject: [SciPy-Dev] Directory for performance analysis > > Message-ID: > > > > Content-Type: text/plain; charset="utf-8" > > > > Although scipy has a directory ("benchmarks") for automatically testing > how > > the performance of individual functions has changed, there is no > directory > > as far as I have found for analysis of how particular approaches to the > > same problem compare. I think it would be worthwhile to have one. > > > > For example the "scipy.signal.convolve" function has code to > automatically > > choose whether to use the conventional convolution algorithm or the > > FFT-based algorithm, based on an analysis someone did. However, I cannot > > find the code that actually carried out that analysis. This is important > > because I want to extend that analysis to include another algorithm. > > > > There are a bunch of other situations where particular approaches are > > recommended as being faster in certain situations, or faster by certain > > amounts in certain situations. > > > > So I think that having a directory inside the scipy source tree that > > includes analyses like that would be worthwhile. I was thinking > something > > like "benchmarks/analysis", although it could also be part of the > > documentation proper. > > Hi Todd, > This sounds interesting. The place where this could be useful to the > releases would be in determining defaults. > > Is this the use you have in mind or is there another use that I?m not > thinking of at the moment? > > The downside to that could be if defaults are changed they could impact > end users of SciPy. > There were three situations I thought this could be useful. The first, as you suggested, is for setting defaults. Of course changing defaults needs to be carefully considered. But having the code for any decision readily available would make such decisions easier, I think. And it would allow future developers to tell how current defaults were decided on in a more reproducible way than mailing list threads. The second is for giving advice. In a number of places the documentation recommends certain approaches in certain situations, even if the code doesn't do it automatically. This could help with that. Third is for checking whether functions have the sort of performance characteristics they should have. In certain cases the relationship between the performance of one function relative to another should be relatively clear, even if the performance of an individual function is hard to determine do to differences in the processor, memory, underlying libraries, OS, etc. This could more reliably identify errors hurting performance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Oct 3 15:54:37 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 3 Oct 2019 21:54:37 +0200 Subject: [SciPy-Dev] request for a volunteer with devops/sysadmin skills, SciPy conda-forge Message-ID: Hi all, We have never had a working Windows build on conda-forge for SciPy. This is a bit painful, since it means that pretty much all SciPy/PyData users on Windows can't go "all conda-forge", and mixing the defaults and conda-forge channels is fairly fragile. The solution is now a lot closer, since rather than a Fortran compiler project it's now a devops project: Intel provided permission and a license to use the Intel Fortran compiler, which is the best way to build SciPy on Windows. So "all" that's needed is set that up on a VM somewhere and connect it to conda-forge. This is still nontrivial, it requires some good devops skills. Marius van Niekerk, one of the conda-forge core team members, provided details on status and next steps at https://github.com/conda-forge/scipy-feedstock/issues/80. He could really use a hand. If there's anyone here with the interest and skills to help out here, this would be a really high-impact contribution! Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Oct 3 17:54:19 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 03 Oct 2019 14:54:19 -0700 Subject: [SciPy-Dev] =?utf-8?q?Accepting_NEP_29_=E2=80=94_Recommend_Pytho?= =?utf-8?q?n_and_Numpy_version_support_as_a_community_policy_standard?= Message-ID: Hi all, we propose formally accepting the NumPy enhancement proposal 29: "Recommend Python and Numpy version support as a community policy standard" available at: https://numpy.org/neps/nep-0029-deprecation_policy.html If there are no objections within a week it may be accepted. This proposal is a recommendation to the larger ecosystem and thus should receive attention and acceptance from a wide audience. However, lets try to keep discussions on the NumPy mailing list. The most important points from the Abstract and Implementation sections are: "This NEP recommends that all projects across the Scientific Python ecosystem adopt a common ?time window-based? policy for support of Python and NumPy versions. Standardizing a recommendation for project support of minimum Python and NumPy versions will improve downstream project planning. ?" and: "We suggest that all projects adopt the following language into their development guidelines: This project supports: * All minor versions of Python released 42 months prior to the project, and at minimum the two latest minor versions. * All minor versions of numpy released in the 24 months prior to the project, and at minimum the last thee minor versions." For the full text, please refer to the link above. Cheers, Sebastian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From shekharrajak.1994 at gmail.com Fri Oct 4 01:17:08 2019 From: shekharrajak.1994 at gmail.com (Shekhar Rajak) Date: Fri, 4 Oct 2019 05:17:08 +0000 (UTC) Subject: [SciPy-Dev] request for a volunteer with devops/sysadmin skills, SciPy conda-forge In-Reply-To: References: Message-ID: <1412233987.2267216.1570166228512@mail.yahoo.com> Hi, I have some knowledge of DevOps technology. I can listen/watch the discussions and may help. Regards,Shekhar Prasad Rajak, Contact : +918142478937Blog?|?Github?|?TwitterSkype: shekhar.rajak1 On Friday, 4 October 2019, 01:25:23 am GMT+5:30, Ralf Gommers wrote: Hi all, We have never had a working Windows build on conda-forge for SciPy. This is a bit painful, since it means that pretty much all SciPy/PyData users on Windows can't go "all conda-forge", and mixing the defaults and conda-forge channels is fairly fragile. The solution is now a lot closer, since rather than a Fortran compiler project it's now a devops project: Intel provided permission and a license to use the Intel Fortran compiler, which is the best way to build SciPy on Windows. So "all" that's needed is set that up on a VM somewhere and connect it to conda-forge. This is still nontrivial, it requires some good devops skills. Marius van Niekerk, one of the conda-forge core team members, provided details on status and next steps at https://github.com/conda-forge/scipy-feedstock/issues/80. He could really use a hand. If there's anyone here with the interest and skills to help out here, this would be a really high-impact contribution! Cheers, Ralf _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From b.rosa at unistra.fr Fri Oct 4 03:13:45 2019 From: b.rosa at unistra.fr (Benoit Rosa) Date: Fri, 4 Oct 2019 09:13:45 +0200 Subject: [SciPy-Dev] New Tutorial on Optimize Help In-Reply-To: References: Message-ID: Hi Andrea, I started at some point looking at the benchmarks, but didn't get very far because at the time there were bugs in the benchmarking framework. From what I remember, the method is adapted for functions with a large, bounded search space, which is not always the case for the test functions. Some more work to detail for which specific class of problems it could be advantageous wrt other GO methods is still to be done. I'll have a look when I get a bit of time. Beno?t On 01/10/2019 16:35, Andrea Gavana wrote: > Hi Benoit, > > On Tue, 1 Oct 2019 at 13.23, Benoit Rosa > wrote: > > Hi Andrea, > > I did implement a branch-and-bound method in Scipy one or two > years ago. I posted here, but nobody seemed to find much interest > in incorporating it, so I didn't make a PR. > > What I implemented is a stochastic branch and bound method which > is very efficient for non-convex problems (think of point cloud > registration optimized over SE3 when you don't have an initial > guess, for instance). References to scientific papers explaining > the method in depth are provided in the commit : > https://github.com/benoitrosa/scipy/commit/49a2c23b74b69dc4250e20e21db75bd071dfd92d > > I'd be happy to dig into it again and make a PR if there's interest. > > Beno?t > > > > Thanks, it looks very interesting. I hadn?t seen it before, I guess > your message a year ago got lost in the maze of my emails... > > Out of curiosity, did you ever try to run your algorithm against the > scipy benchmarks? I still carry (old) results from multiple > optimization algorithms here: > > http://infinity77.net/global_optimization/ > > http://infinity77.net/global_optimization/multidimensional.html > > And it would be nice to see how your stochastic branch and bound > compares to the rest... > > Andrea. > > > > > > > On 01/10/2019 08:41, Andrea Gavana wrote: >> Most real life optimization problems are characterized by >> objective functions that are slow to evaluate - depending on the >> nature of the problem, it can take from a few seconds to several >> hours for a single evaluation. For many of them, the gradient is >> not even available. >> >> So, the moment the number of parameters you?re optimizing on >> exceeds 4 or 5 then gradient-based methods (like BFGS, for >> example) become inefficient as you need numerical derivatives and >> so a lot of objective functions evaluations for a single gradient >> step. >> >> Scipy has a lot of powerful, diverse optimization algorithms so >> in this case you might want to try other approaches, although it >> would be nice to see implementations like MCS (branch and bound >> methods) that exist in matlab. I used them in the past and they >> are quite powerful when gradients are not available or too >> expensive to compute. >> >> Andrea. >> >> >> On Tue, 1 Oct 2019 at 08.27, Phillip Feldman >> > > wrote: >> >> Providing a function for the derivative is almost always >> better than finite-difference derivatives?unless the cost of >> evaluating the derivative function is very high. >> >> On Mon, Sep 30, 2019 at 4:43 PM Christina Lee >> > wrote: >> >> Hi, >> >> I'm a SciPy technical writer and am currently rewriting >> the scipy.optimize tutorial, focusing on `minimize` right >> now.? While I've gotten a grasp of the "how", I still >> want to explain "why". Why choose one option over >> another? I could use information from those with more >> experience. >> >> A lot of methods are available.?? Most problems canhave >> BFGS thrown at them, but I want to explain something for >> those other cases. Other situations could have features, >> like constraints or non-differentiability, that lend >> themselves to a specific method. But the module still has >> a lot of alternatives.? Are they there for academic >> purposes?? Are they the best for some problems? How could >> someone find that out? >> >> For derivatives, users can choose to provide a function >> or three different types of finite-difference schemes. >> >> When is providing a function better than >> finite-difference derivatives?? For Hessians, >> approximations are sometimes more efficient.? How can we >> know in advance if that's true? Is that ever true for >> gradients? >> >> How do we choose which finite-difference scheme? >> `3-point` and `cs` (if it is the symmetric approximation >> I think) have higher-order accuracy, but `cs` uses a >> point not yet computed.? Is `3-point` ever not the way to go? >> >> Thanks for your expertise, >> Christina >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Oct 5 07:02:28 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 5 Oct 2019 13:02:28 +0200 Subject: [SciPy-Dev] New Tutorial on Optimize Help In-Reply-To: References: Message-ID: Hi Christina, On Tue, Oct 1, 2019 at 1:42 AM Christina Lee wrote: > Hi, > > I'm a SciPy technical writer and am currently rewriting the scipy.optimize > tutorial, focusing on `minimize` right now. While I've gotten a grasp of > the "how", I still want to explain "why". Why choose one option over > another? I could use information from those with more experience. > This is a great question. For all the methods offered by `minimize`, there's a nice overview in Table 1 of the preprint of the SciPy 1.0 paper ( https://arxiv.org/abs/1907.10121). It explains which methods support bounds, (in)equality constraints, 1st/2nd derivatives, etc. That table and the text in the caption have not found its way into the docs yet, including it in your tutorial would be awesome. LaTeX sources live in https://github.com/scipy/scipy-articles/tree/master/scipy-1.0 > A lot of methods are available. Most problems can have BFGS thrown at > them, but I want to explain something for those other cases. Other > situations could have features, like constraints or non-differentiability, > that lend themselves to a specific method. But the module still has a lot > of alternatives. Are they there for academic purposes? Are they the best > for some problems? How could someone find that out? > Table 1 was about `minimize` only, not about the global optimizers. The nicest overview of those is probably the benchmarks that Andrea pointed out (http://infinity77.net/global_optimization/multidimensional.html). The SciPy benchmarks in our repo (under benchmarks/benchmarks/go_benchmark_functions) started from Andrea's benchmarks. We've added to it after taking them over, and when we add new optimization routines we now require that they come with benchmarks and that those show that whatever we add is at least competitive, and preferably better, than what we already have. > > For derivatives, users can choose to provide a function or three different > types of finite-difference schemes. > > When is providing a function better than finite-difference derivatives? > For Hessians, approximations are sometimes more efficient. How can we know > in advance if that's true? Is that ever true for gradients? > This one has been answered already I think. When you have an analytical derivative, it's usually best to use it. But it's often a lot of work or not possible to obtain that analytical derivative. > > How do we choose which finite-difference scheme? `3-point` and `cs` (if it > is the symmetric approximation I think) have higher-order accuracy, but > `cs` uses a point not yet computed. Is `3-point` ever not the way to go? > This I don't know actually, hope an expert can weigh in. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From evgeny.burovskiy at gmail.com Sat Oct 5 07:52:42 2019 From: evgeny.burovskiy at gmail.com (Evgeni Burovski) Date: Sat, 5 Oct 2019 14:52:42 +0300 Subject: [SciPy-Dev] New Tutorial on Optimize Help In-Reply-To: References: Message-ID: >> How do we choose which finite-difference scheme? `3-point` and `cs` (if >> it is the symmetric approximation I think) >> > cs is actually "complex step" here. For first derivatives it does not suffer from roundoff errors. But it assumes that the cost function can be analytically continued into the complex plane. Otherwise, it can silently produce garbage. have higher-order accuracy, but `cs` uses a point not yet computed. Is >> `3-point` ever not the way to go? >> > cs is probably best if the cost function allows it. 3-point scheme is 2nd order, and automatically handles the boundaries (i.e.switches between symmetric and one-sided schemes at the boundaries). Choose the 2-point scheme (which us 1st order) only is the evaluations of the cost function are *extremely* costly, and you are really in a need to avoid that extra evaluation. But be prepared to lose accuracy and carefully check that the roundoff errors do not throw the minimization off. All that said, it's really better to avoid finite differences and provide derivatives explicitly (paper-and-pencil or automatic differentiation). Hope this helps, Evgeni > This I don't know actually, hope an expert can weigh in. > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From newville at cars.uchicago.edu Mon Oct 7 00:21:47 2019 From: newville at cars.uchicago.edu (Matt Newville) Date: Sun, 6 Oct 2019 23:21:47 -0500 Subject: [SciPy-Dev] New Tutorial on Optimize Help In-Reply-To: References: Message-ID: Hi Christina, On Mon, Sep 30, 2019 at 6:43 PM Christina Lee wrote: > Hi, > > I'm a SciPy technical writer and am currently rewriting the scipy.optimize > tutorial, focusing on `minimize` right now. While I've gotten a grasp of > the "how", I still want to explain "why". Why choose one option over > another? I could use information from those with more experience. > Thank you very much for taking this on -- this is heroic work. I would make no claims about what "most users" use these methods for, and doubt anyone else really knows this either. In my experience (maintainer of the lmfit library), it seems that many people using `scipy.optimize` are actually trying to solve curve-fitting problems. Focusing on the group of methods wrapped by `minimize()` is a fine starting point, but you may also want to consider explaining why the user would choose `minimize()` over other approaches. That is, describe what sort of problems `minimize()` is most suitable for, which it can be useful for but might not be the only approach, and for which it is not the most suitable. The methods of `minimize()` are all scalar minimizers: the user's objective function returns a single value ("cost") to minimize. If their objective function *really* calculates an array and then the user reduces that array to a scalar (say as sum-of-squares or log-likelihood) then they are almost certainly better of using `least_squares` or `leastsq` or a similar solver. It also seems to me that a fair number of people use `minimize()` when their problem actually is or can be made linear. That is, it might be good to clarify when iterative methods are needed, and when regression methods can be used. The methods in `minimize()` are inherently local solvers. Global optimization is a different category, but a tutorial would probably do well to describe the difference but also to be clear-eyed that many problems do not require a global solver as much as they need a local solver plus some prior knowledge and understanding of the problem to be solved. That is, many real problems are really solved with local solvers. Many (maybe most) problems actually do (and certainly should try to) start with a decent understanding of the scope of the problem. With a local solver, the task for the user is not a blind search, but refining variable values and trying to understand their correlations and uncertainties. That is all to say that you probably should not expect too much experience with these topics on the part of the reader of the tutorial. > A lot of methods are available. Most problems can have BFGS thrown at > them, but I want to explain something for those other cases. Other > situations could have features, like constraints or non-differentiability, > that lend themselves to a specific method. But the module still has a lot > of alternatives. Are they there for academic purposes? Are they the best > for some problems? How could someone find that out? > > Those are the right questions and some sort of answer themselves: The different methods represent the advance of history with the older and simpler ones really reflecting the aversion to storing results in memory and on performance (that is trying to find the solution with the least use of memory and the fewest evaluations of the objective function). Even now, and in this conversation, the emphasis is on performance (IMHO, that emphasis is to the detriment of "user-friendliness" and helping the user identify "correctness"). The fact that you asked about when the user would choose between different solvers and different methods for calculating derivatives means that this burden, which is really a mathematical detail (if an important one) of the method used, is put on the user of `minimize()`. > For derivatives, users can choose to provide a function or three different > types of finite-difference schemes. > > When is providing a function better than finite-difference derivatives? > For Hessians, approximations are sometimes more efficient. How can we know > in advance if that's true? Is that ever true for gradients? > > How do we choose which finite-difference scheme? `3-point` and `cs` (if it > is the symmetric approximation I think) have higher-order accuracy, but > `cs` uses a point not yet computed. Is `3-point` ever not the way to go? > Some of the other responses said that the user should provide functions for derivatives. I respectfully disagree that this should be emphasized in a tutorial. If (it can be a big "if") the user can write a function for the derivatives (of cost with respect to each variable), that usually means that the objective function is differentiable, and probably "sane". This means in turn that finite-difference methods really ought to work (in the sense of "not fail"). Finite-difference Jacobians will certainly mean more evaluations of the objective function. Assuming that the Jacobian function has a runtime that is about the same as the objective function, the total runtime will almost certainly be less when providing a Jacobian function. Does reduced runtime mean that analytic Jacobians should always be preferred? I think it does not. The user of `scipy.optimize.minimize()` should know that run-time is far less precious than write- and read-time. At the very least, a Jacobian function should be used only when the time spent *writing* the Jacobian function is less than the time saved running the program. That "less than" might even be "much less than". Changing runtime by a factor of 5 from 1 second to 200 milliseconds is certainly *not* worth writing a Jacobian (even if one types much faster and more accurately than average). Changing runtime to 100 hours to 20 hours might be worth spending some time on writing (and testing and maintaining!) the extra code of the Jacobian function. If you expect the program you are writing will be used 10,000 times, then sure, investigate a Jacobian function. Or figure out how to make the objective function faster. Or buy more compute power. I would hope a tutorial would emphasize other aspects of minimization over the technical detail and performance boost that might be found by providing an analytic Jacobian. These other aspects might include: how to compare solver methods and types of cost function (least-squares, log-probability, etc). how to investigate whether the solution is only local" or is stable and global. how starting values can affect the solution and run time. how one can evaluate the quality of the solution ("goodness of fit") and estimate the uncertainties in the values found for the variables. There is the open-ended but critical question of "is the objective function correct", or for curve-fitting "is this mathematical model correct"? Related to this is the question of whether the set of variables used is actually complete or if there are some latent variables that the objective function assumes but that should actually be tested? These are challenging topics, but also strike me as being of more importance and interest to the user of `scipy.optimize` than ultimate performance. That is, analytic Jacobians strike me as "advanced usage" and would be a fine next step after a tutorial is digested. Sorry that was so long, and feel free to use or ignore any of it as you see fit. Cheers, --Matt Newville -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian-achim.mueller at mpi-hd.mpg.de Tue Oct 8 07:32:55 2019 From: sebastian-achim.mueller at mpi-hd.mpg.de (Sebastian A. Mueller) Date: Tue, 08 Oct 2019 13:32:55 +0200 Subject: [SciPy-Dev] Contribution: Estimating the overlap of a ray with volume-cells (voxels) for e.g. tomography In-Reply-To: References: Message-ID: <10e1d25c0e49db5c306710ebc8ae2007@mpi-hd.mpg.de> Dear scipy, I wrote a package to estimate the distances a ray overlaps with volume-cells in three dimensions. This is needed to estimate system-matrices in tomography. The interface is only one function. It takes the ray's support and direction vectors, and the bin-edges for the space partitioning. It returns the overlaps of the ray and the voxels. The space-partitioning must be rectangular, but it does not need to be equally spaced. In combination with scipy.sparse, it can efficiently create system-matrices for iterative, tomographic reconstructions. The algorithm uses oct-tree-traversal for efficiency. On pypi: https://pypi.org/project/ray-voxel-overlap/0.0.2/ On github: https://github.com/cherenkov-plenoscope/ray_voxel_overlap I would like to contribute this to scipy. It addresses multiple fields in science such as medicine, biology, microscopy, physics, astronomy, geo-earth, image-processing. At its core, it is pure linear algebra. My package contains a pure python-implementation for development and testing, and a (much faster) C-implementation wrapped in cython. Do you think such a tool belongs into scipy? If not, where else should it go? This is my first attempt to contribute to scipy. Cheers Sebastian From ralf.gommers at gmail.com Tue Oct 8 09:03:39 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 8 Oct 2019 15:03:39 +0200 Subject: [SciPy-Dev] New Tutorial on Optimize Help In-Reply-To: References: Message-ID: On Mon, Oct 7, 2019 at 6:22 AM Matt Newville wrote: > Hi Christina, > > On Mon, Sep 30, 2019 at 6:43 PM Christina Lee > wrote: > >> Hi, >> >> I'm a SciPy technical writer and am currently rewriting the >> scipy.optimize tutorial, focusing on `minimize` right now. While I've >> gotten a grasp of the "how", I still want to explain "why". Why choose >> one option over another? I could use information from those with more >> experience. >> > > Thank you very much for taking this on -- this is heroic work. I would > make no claims about what "most users" use these methods for, and doubt > anyone else really knows this either. In my experience (maintainer of the > lmfit library), it seems that many people using `scipy.optimize` are > actually trying to solve curve-fitting problems. > > Focusing on the group of methods wrapped by `minimize()` is a fine > starting point, but you may also want to consider explaining why the user > would choose `minimize()` over other approaches. That is, describe what > sort of problems `minimize()` is most suitable for, which it can be useful > for but might not be the only approach, and for which it is not the most > suitable. > > The methods of `minimize()` are all scalar minimizers: the user's > objective function returns a single value ("cost") to minimize. If their > objective function *really* calculates an array and then the user reduces > that array to a scalar (say as sum-of-squares or log-likelihood) then they > are almost certainly better of using `least_squares` or `leastsq` or a > similar solver. It also seems to me that a fair number of people use > `minimize()` when their problem actually is or can be made linear. That > is, it might be good to clarify when iterative methods are needed, and when > regression methods can be used. > > The methods in `minimize()` are inherently local solvers. Global > optimization is a different category, but a tutorial would probably do well > to describe the difference but also to be clear-eyed that many problems do > not require a global solver as much as they need a local solver plus some > prior knowledge and understanding of the problem to be solved. That is, > many real problems are really solved with local solvers. Many (maybe most) > problems actually do (and certainly should try to) start with a decent > understanding of the scope of the problem. With a local solver, the task > for the user is not a blind search, but refining variable values and trying > to understand their correlations and uncertainties. > > That is all to say that you probably should not expect too much experience > with these topics on the part of the reader of the tutorial. > > >> A lot of methods are available. Most problems can have BFGS thrown at >> them, but I want to explain something for those other cases. Other >> situations could have features, like constraints or non-differentiability, >> that lend themselves to a specific method. But the module still has a lot >> of alternatives. Are they there for academic purposes? Are they the best >> for some problems? How could someone find that out? >> >> > Those are the right questions and some sort of answer themselves: The > different methods represent the advance of history with the older and > simpler ones really reflecting the aversion to storing results in memory > and on performance (that is trying to find the solution with the least use > of memory and the fewest evaluations of the objective function). Even now, > and in this conversation, the emphasis is on performance (IMHO, that > emphasis is to the detriment of "user-friendliness" and helping the user > identify "correctness"). > Thanks for this perspective Matt! Your whole email is spot on. Cheers, Ralf The fact that you asked about when the user would choose between different > solvers and different methods for calculating derivatives means that this > burden, which is really a mathematical detail (if an important one) of the > method used, is put on the user of `minimize()`. > > >> For derivatives, users can choose to provide a function or three >> different types of finite-difference schemes. >> >> When is providing a function better than finite-difference derivatives? >> For Hessians, approximations are sometimes more efficient. How can we know >> in advance if that's true? Is that ever true for gradients? >> >> How do we choose which finite-difference scheme? `3-point` and `cs` (if >> it is the symmetric approximation I think) have higher-order accuracy, but >> `cs` uses a point not yet computed. Is `3-point` ever not the way to go? >> > > Some of the other responses said that the user should provide functions > for derivatives. I respectfully disagree that this should be emphasized in > a tutorial. If (it can be a big "if") the user can write a function for > the derivatives (of cost with respect to each variable), that usually means > that the objective function is differentiable, and probably "sane". This > means in turn that finite-difference methods really ought to work (in the > sense of "not fail"). Finite-difference Jacobians will certainly mean more > evaluations of the objective function. Assuming that the Jacobian function > has a runtime that is about the same as the objective function, the total > runtime will almost certainly be less when providing a Jacobian function. > > Does reduced runtime mean that analytic Jacobians should always be > preferred? I think it does not. > > The user of `scipy.optimize.minimize()` should know that run-time is far > less precious than write- and read-time. At the very least, a Jacobian > function should be used only when the time spent *writing* the Jacobian > function is less than the time saved running the program. That "less than" > might even be "much less than". Changing runtime by a factor of 5 from 1 > second to 200 milliseconds is certainly *not* worth writing a Jacobian > (even if one types much faster and more accurately than average). > Changing runtime to 100 hours to 20 hours might be worth spending some > time on writing (and testing and maintaining!) the extra code of the > Jacobian function. If you expect the program you are writing will be used > 10,000 times, then sure, investigate a Jacobian function. Or figure out > how to make the objective function faster. Or buy more compute power. > > I would hope a tutorial would emphasize other aspects of minimization over > the technical detail and performance boost that might be found by providing > an analytic Jacobian. These other aspects might include: > how to compare solver methods and types of cost function > (least-squares, log-probability, etc). > how to investigate whether the solution is only local" or is stable and > global. > how starting values can affect the solution and run time. > how one can evaluate the quality of the solution ("goodness of fit") > and estimate the uncertainties in the values found for the variables. > > There is the open-ended but critical question of "is the objective > function correct", or for curve-fitting "is this mathematical model > correct"? Related to this is the question of whether the set of variables > used is actually complete or if there are some latent variables that the > objective function assumes but that should actually be tested? These are > challenging topics, but also strike me as being of more importance and > interest to the user of `scipy.optimize` than ultimate performance. That > is, analytic Jacobians strike me as "advanced usage" and would be a fine > next step after a tutorial is digested. > > Sorry that was so long, and feel free to use or ignore any of it as > you see fit. > Cheers, > > --Matt Newville > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Oct 8 09:13:27 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 8 Oct 2019 15:13:27 +0200 Subject: [SciPy-Dev] ways of storing notebook tutorials? Message-ID: Hi all, Christina is working on a nice scipy.optimize tutorial in notebook form. I like that in principle; Sphinx is not great for long-form tutorials. It does raise the question of how to integrate those notebooks. Notebooks with output in version control are quite problematic (size, unreadable diffs, breaking command line code search tools, etc.). I can think of a number of ways to start integrating notebooks, but I'd rather not reinvent this particular wheel. Are there recommendations for setups other projects have that work well? Pointers to discussions other projects have had on this topic would also be nice. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Oct 8 09:19:28 2019 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 8 Oct 2019 14:19:28 +0100 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: Hi, I use Jupytext all the time now, and I prefer the RMarkdown input format to Jupytext. Jupytext can convert .ipynb files to .Rmd files, and saves notebooks as both .ipynb and .Rmd. The workflow is pretty good - after installing Jupytext, you open your .Rmd file in the notebook interface, and run anything you want, in the standard interface. Then you save, if you need to, which saves to both .ipynb and .Rmd. You edit the .Rmd using your nice text editor. Then you can reload the notebook in Jupyter, with just a browser refresh, to see the changes, and to run any extra code you've added. I then store the .Rmd files only, in version control, keeping the .ipynb as outputs from the final build step. Cheers, Matthew On Tue, Oct 8, 2019 at 2:14 PM Ralf Gommers wrote: > > Hi all, > > Christina is working on a nice scipy.optimize tutorial in notebook form. I like that in principle; Sphinx is not great for long-form tutorials. It does raise the question of how to integrate those notebooks. Notebooks with output in version control are quite problematic (size, unreadable diffs, breaking command line code search tools, etc.). > > I can think of a number of ways to start integrating notebooks, but I'd rather not reinvent this particular wheel. Are there recommendations for setups other projects have that work well? Pointers to discussions other projects have had on this topic would also be nice. > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From larson.eric.d at gmail.com Tue Oct 8 09:21:57 2019 From: larson.eric.d at gmail.com (Eric Larson) Date: Tue, 8 Oct 2019 09:21:57 -0400 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: I've used sphinx-gallery in a number of projects (am now a maintainer). Basically you write Python code and it renders to html with links to downloadable notebooks in the page, automatic linking/cross-refs to function and class defs, nice index pages , etc. Eric On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett wrote: > Hi, > > I use Jupytext all the time now, and I prefer the RMarkdown input > format to Jupytext. > > Jupytext can convert .ipynb files to .Rmd files, and saves notebooks > as both .ipynb and .Rmd. > > The workflow is pretty good - after installing Jupytext, you open your > .Rmd file in the notebook interface, and run anything you want, in the > standard interface. > > Then you save, if you need to, which saves to both .ipynb and .Rmd. > > You edit the .Rmd using your nice text editor. > > Then you can reload the notebook in Jupyter, with just a browser > refresh, to see the changes, and to run any extra code you've added. > > I then store the .Rmd files only, in version control, keeping the > .ipynb as outputs from the final build step. > > Cheers, > > Matthew > > On Tue, Oct 8, 2019 at 2:14 PM Ralf Gommers > wrote: > > > > Hi all, > > > > Christina is working on a nice scipy.optimize tutorial in notebook form. > I like that in principle; Sphinx is not great for long-form tutorials. It > does raise the question of how to integrate those notebooks. Notebooks with > output in version control are quite problematic (size, unreadable diffs, > breaking command line code search tools, etc.). > > > > I can think of a number of ways to start integrating notebooks, but I'd > rather not reinvent this particular wheel. Are there recommendations for > setups other projects have that work well? Pointers to discussions other > projects have had on this topic would also be nice. > > > > Cheers, > > Ralf > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Tue Oct 8 09:18:19 2019 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 8 Oct 2019 09:18:19 -0400 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: <20191008131819.p7m6ur3qeromhgq3@phare.normalesup.org> On Tue, Oct 08, 2019 at 03:13:27PM +0200, Ralf Gommers wrote: > Christina is working on a nice scipy.optimize tutorial in notebook form. I like > that in principle; Sphinx is not great for long-form tutorials. It does raise > the question of how to integrate those notebooks. Have you considered sphinx-gallery? https://sphinx-gallery.github.io/ It can generate notebooks from pure-Python files and integrates nicely with sphinx. Ga?l From ralf.gommers at gmail.com Tue Oct 8 10:54:25 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 8 Oct 2019 16:54:25 +0200 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: On Tue, Oct 8, 2019 at 3:22 PM Eric Larson wrote: > I've used sphinx-gallery in a number > of projects (am now a maintainer). Basically you write Python code > > and it renders to html > > with links to downloadable notebooks in the page, automatic > linking/cross-refs to function and class defs, nice index pages > , etc. > That doesn't really answer my question I think. The starting point is: notebooks are nice to write a long-form tutorial in, ReST is not. And sphinx-gallery is nice for a set of independent small to medium sized examples, but not the best fit for a long tutorial on a whole module (e.g. http://scipy.github.io/devdocs/tutorial/optimize.html). > > Eric > > > On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett > wrote: > >> Hi, >> >> I use Jupytext all the time now, and I prefer the RMarkdown input >> format to Jupytext. >> >> Jupytext can convert .ipynb files to .Rmd files, and saves notebooks >> as both .ipynb and .Rmd. >> >> The workflow is pretty good - after installing Jupytext, you open your >> .Rmd file in the notebook interface, and run anything you want, in the >> standard interface. >> >> Then you save, if you need to, which saves to both .ipynb and .Rmd. >> >> You edit the .Rmd using your nice text editor. >> >> Then you can reload the notebook in Jupyter, with just a browser >> refresh, to see the changes, and to run any extra code you've added. >> >> I then store the .Rmd files only, in version control, keeping the >> .ipynb as outputs from the final build step. >> > Thanks, that sounds like a reasonable workflow. Seems like we can roundtrip notebook -> plain text format -> notebook. We'll still need to integrate that into the built html docs then, so the plain text format should be ReST. That doesn't seem supported yet: https://jupytext.readthedocs.io/en/latest/using-cli.html#testing-the-round-trip-conversion. Or am I missing something? Cheers, Ralf >> Cheers, >> >> Matthew >> >> On Tue, Oct 8, 2019 at 2:14 PM Ralf Gommers >> wrote: >> > >> > Hi all, >> > >> > Christina is working on a nice scipy.optimize tutorial in notebook >> form. I like that in principle; Sphinx is not great for long-form >> tutorials. It does raise the question of how to integrate those notebooks. >> Notebooks with output in version control are quite problematic (size, >> unreadable diffs, breaking command line code search tools, etc.). >> > >> > I can think of a number of ways to start integrating notebooks, but I'd >> rather not reinvent this particular wheel. Are there recommendations for >> setups other projects have that work well? Pointers to discussions other >> projects have had on this topic would also be nice. >> > >> > Cheers, >> > Ralf >> > >> > _______________________________________________ >> > SciPy-Dev mailing list >> > SciPy-Dev at python.org >> > https://mail.python.org/mailman/listinfo/scipy-dev >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Oct 8 11:04:42 2019 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 8 Oct 2019 16:04:42 +0100 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: Hi, On Tue, Oct 8, 2019 at 3:55 PM Ralf Gommers wrote: > > > > On Tue, Oct 8, 2019 at 3:22 PM Eric Larson wrote: >> >> I've used sphinx-gallery in a number of projects (am now a maintainer). Basically you write Python code and it renders to html with links to downloadable notebooks in the page, automatic linking/cross-refs to function and class defs, nice index pages, etc. > > > That doesn't really answer my question I think. The starting point is: notebooks are nice to write a long-form tutorial in, ReST is not. And sphinx-gallery is nice for a set of independent small to medium sized examples, but not the best fit for a long tutorial on a whole module (e.g. http://scipy.github.io/devdocs/tutorial/optimize.html). Right - I think sphinx-gallery is the right choice for these small examples and where you want to develop the tutorial in text. It solves the ReST output nicely. >> >> >> Eric >> >> >> On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett wrote: >>> >>> Hi, >>> >>> I use Jupytext all the time now, and I prefer the RMarkdown input >>> format to Jupytext. >>> >>> Jupytext can convert .ipynb files to .Rmd files, and saves notebooks >>> as both .ipynb and .Rmd. >>> >>> The workflow is pretty good - after installing Jupytext, you open your >>> .Rmd file in the notebook interface, and run anything you want, in the >>> standard interface. >>> >>> Then you save, if you need to, which saves to both .ipynb and .Rmd. >>> >>> You edit the .Rmd using your nice text editor. >>> >>> Then you can reload the notebook in Jupyter, with just a browser >>> refresh, to see the changes, and to run any extra code you've added. >>> >>> I then store the .Rmd files only, in version control, keeping the >>> .ipynb as outputs from the final build step. > > > Thanks, that sounds like a reasonable workflow. Seems like we can roundtrip notebook -> plain text format -> notebook. Right - exactly - and you can do that interactively, so you can save from the Jupyter interface, load in a text editor, save in the text editor and just reload the page in Jupyter to get the result. It will even keep the outputs as long as you haven't done very substantial edits. > We'll still need to integrate that into the built html docs then, so the plain text format should be ReST. That doesn't seem supported yet: https://jupytext.readthedocs.io/en/latest/using-cli.html#testing-the-round-trip-conversion. Or am I missing something? No, that is (currently) a problem that I haven't been working on - because I'm using Markdown output documents like jupyter-book and bookdown. One of my collaborators uses notebooks a lot and I was going to suggest to him that we have a separate site for the notebooks and the main docs, or at least, we build them separately, and perhaps combine them after building to HTML. Maybe jupyter-book would help here? Chris Holdgraf is the maintainer, and he's very responsive. Cheers, Matthew From andyfaff at gmail.com Tue Oct 8 11:27:27 2019 From: andyfaff at gmail.com (Andrew Nelson) Date: Tue, 8 Oct 2019 17:27:27 +0200 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: Another alternative is to use `jupyter-sphinx` and `nbsphinx` sphinx extensions. You can include a link to the unexecuted notebook in RST (e.g. https://github.com/refnx/refnx/blob/master/doc/index.rst), and when the doc build occurs it'll execute the notebook + output as part of the run. On Tue, 8 Oct 2019, 17:06 Matthew Brett, wrote: > Hi, > > On Tue, Oct 8, 2019 at 3:55 PM Ralf Gommers > wrote: > > > > > > > > On Tue, Oct 8, 2019 at 3:22 PM Eric Larson > wrote: > >> > >> I've used sphinx-gallery in a number of projects (am now a maintainer). > Basically you write Python code and it renders to html with links to > downloadable notebooks in the page, automatic linking/cross-refs to > function and class defs, nice index pages, etc. > > > > > > That doesn't really answer my question I think. The starting point is: > notebooks are nice to write a long-form tutorial in, ReST is not. And > sphinx-gallery is nice for a set of independent small to medium sized > examples, but not the best fit for a long tutorial on a whole module (e.g. > http://scipy.github.io/devdocs/tutorial/optimize.html). > > Right - I think sphinx-gallery is the right choice for these small > examples and where you want to develop the tutorial in text. It > solves the ReST output nicely. > > >> > >> > >> Eric > >> > >> > >> On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett > wrote: > >>> > >>> Hi, > >>> > >>> I use Jupytext all the time now, and I prefer the RMarkdown input > >>> format to Jupytext. > >>> > >>> Jupytext can convert .ipynb files to .Rmd files, and saves notebooks > >>> as both .ipynb and .Rmd. > >>> > >>> The workflow is pretty good - after installing Jupytext, you open your > >>> .Rmd file in the notebook interface, and run anything you want, in the > >>> standard interface. > >>> > >>> Then you save, if you need to, which saves to both .ipynb and .Rmd. > >>> > >>> You edit the .Rmd using your nice text editor. > >>> > >>> Then you can reload the notebook in Jupyter, with just a browser > >>> refresh, to see the changes, and to run any extra code you've added. > >>> > >>> I then store the .Rmd files only, in version control, keeping the > >>> .ipynb as outputs from the final build step. > > > > > > Thanks, that sounds like a reasonable workflow. Seems like we can > roundtrip notebook -> plain text format -> notebook. > > Right - exactly - and you can do that interactively, so you can save > from the Jupyter interface, load in a text editor, save in the text > editor and just reload the page in Jupyter to get the result. It will > even keep the outputs as long as you haven't done very substantial > edits. > > > We'll still need to integrate that into the built html docs then, so the > plain text format should be ReST. That doesn't seem supported yet: > https://jupytext.readthedocs.io/en/latest/using-cli.html#testing-the-round-trip-conversion. > Or am I missing something? > > No, that is (currently) a problem that I haven't been working on - > because I'm using Markdown output documents like jupyter-book and > bookdown. One of my collaborators uses notebooks a lot and I was > going to suggest to him that we have a separate site for the notebooks > and the main docs, or at least, we build them separately, and perhaps > combine them after building to HTML. Maybe jupyter-book would help > here? Chris Holdgraf is the maintainer, and he's very responsive. > > Cheers, > > Matthew > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shekharrajak.1994 at gmail.com Tue Oct 8 12:05:54 2019 From: shekharrajak.1994 at gmail.com (Shekhar Rajak) Date: Tue, 8 Oct 2019 16:05:54 +0000 (UTC) Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: <299532274.3454956.1570550754945@mail.yahoo.com> I agree that Jupyter notebook is better for writing long tutorials with output (images, graphs, different printing output). We can have a separate repository to write tutorials in IPython notebook for different topics and subtopics. It will be organized and new contributors can add their notebook tutorial about specific topic as well (maybe someone will come with solving really interesting real world problem). Once we have ipython notebook file then we can get the sharing link as well using :?https://nbviewer.jupyter.org/, by just using the github URL of the ipython notebook file. For example IPython file:?https://github.com/Shekharrajak/PyCon-SymPy-SymEngine/blob/master/SymPy_material/tutorial_exercises/01-Symbols-Derivatives-Functions.ipynb can be shared easily using nbviewer:?https://nbviewer.jupyter.org/github/Shekharrajak/PyCon-SymPy-SymEngine/blob/master/SymPy_material/tutorial_exercises/01-Symbols-Derivatives-Functions.ipynb We also have binder option (https://mybinder.org/v2/gh/Shekharrajak/PyCon-SymPy-SymEngine/master?filepath=SymPy_material/tutorial_exercises/01-Symbols-Derivatives-Functions.ipynb) So then can run the same file in browser as well without installing and setting up anything new. So while writing about the topic in docs (.rst file or .md file), the writer can refer to the notebook nbviewer URL. The notebook content will be automatically updated whenever ipynb file is updated. Regards,Shekhar Prasad Rajak, Contact : +918142478937Blog?|?Github?|?TwitterSkype: shekhar.rajak1 On Tuesday, 8 October 2019, 08:58:16 pm GMT+5:30, Andrew Nelson wrote: Another alternative is to use `jupyter-sphinx` and `nbsphinx` sphinx extensions. You can include a link to the unexecuted notebook in RST (e.g.?https://github.com/refnx/refnx/blob/master/doc/index.rst), and when the doc build occurs it'll execute the notebook?+ output as part of the run. On Tue, 8 Oct 2019, 17:06 Matthew Brett, wrote: Hi, On Tue, Oct 8, 2019 at 3:55 PM Ralf Gommers wrote: > > > > On Tue, Oct 8, 2019 at 3:22 PM Eric Larson wrote: >> >> I've used sphinx-gallery in a number of projects (am now a maintainer). Basically you write Python code and it renders to html with links to downloadable notebooks in the page, automatic linking/cross-refs to function and class defs, nice index pages, etc. > > > That doesn't really answer my question I think. The starting point is: notebooks are nice to write a long-form tutorial in, ReST is not. And sphinx-gallery is nice for a set of independent small to medium sized examples, but not the best fit for a long tutorial on a whole module (e.g. http://scipy.github.io/devdocs/tutorial/optimize.html). Right - I think sphinx-gallery is the right choice for these small examples and where you want to develop the tutorial in text.? It solves the ReST output nicely. >> >> >> Eric >> >> >> On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett wrote: >>> >>> Hi, >>> >>> I use Jupytext all the time now, and I prefer the RMarkdown input >>> format to Jupytext. >>> >>> Jupytext can convert .ipynb files to .Rmd files, and saves notebooks >>> as both .ipynb and .Rmd. >>> >>> The workflow is pretty good - after installing Jupytext, you open your >>> .Rmd file in the notebook interface, and run anything you want, in the >>> standard interface. >>> >>> Then you save, if you need to, which saves to both .ipynb and .Rmd. >>> >>> You edit the .Rmd using your nice text editor. >>> >>> Then you can reload the notebook in Jupyter, with just a browser >>> refresh, to see the changes, and to run any extra code you've added. >>> >>> I then store the .Rmd files only, in version control, keeping the >>> .ipynb as outputs from the final build step. > > > Thanks, that sounds like a reasonable workflow. Seems like we can roundtrip notebook -> plain text format -> notebook. Right - exactly - and you can do that interactively, so you can save from the Jupyter interface, load in a text editor, save in the text editor and just reload the page in Jupyter to get the result.? It will even keep the outputs as long as you haven't done very substantial edits. > We'll still need to integrate that into the built html docs then, so the plain text format should be ReST. That doesn't seem supported yet: https://jupytext.readthedocs.io/en/latest/using-cli.html#testing-the-round-trip-conversion. Or am I missing something? No, that is (currently) a problem that I haven't been working on - because I'm using Markdown output documents like jupyter-book and bookdown.? ?One of my collaborators uses notebooks a lot and I was going to suggest to him that we have a separate site for the notebooks and the main docs, or at least, we build them separately, and perhaps combine them after building to HTML.? ? Maybe jupyter-book would help here?? Chris Holdgraf is the maintainer, and he's very responsive. Cheers, Matthew _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev _______________________________________________ SciPy-Dev mailing list SciPy-Dev at python.org https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Tue Oct 8 14:26:14 2019 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Tue, 8 Oct 2019 20:26:14 +0200 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: <299532274.3454956.1570550754945@mail.yahoo.com> References: <299532274.3454956.1570550754945@mail.yahoo.com> Message-ID: In case you haven't seen it Bokeh just tweeted this https://twitter.com/BokehPlots/status/1181215112872509441 Maybe we can ask Bryan about the pros and cons. On Tue, Oct 8, 2019 at 6:07 PM Shekhar Rajak wrote: > I agree that Jupyter notebook is better for writing long tutorials with > output (images, graphs, different printing output). > > We can have a separate repository to write tutorials in IPython notebook > for different topics and subtopics. It will be organized and new > contributors can add their notebook tutorial about specific topic as well > (maybe someone will come with solving really interesting real world > problem). > > Once we have ipython notebook file then we can get the sharing link as > well using : https://nbviewer.jupyter.org/, by just using the github URL > of the ipython notebook file. > > For example IPython file: > https://github.com/Shekharrajak/PyCon-SymPy-SymEngine/blob/master/SymPy_material/tutorial_exercises/01-Symbols-Derivatives-Functions.ipynb > > can be shared easily using nbviewer: > https://nbviewer.jupyter.org/github/Shekharrajak/PyCon-SymPy-SymEngine/blob/master/SymPy_material/tutorial_exercises/01-Symbols-Derivatives-Functions.ipynb > > We also have binder option ( > https://mybinder.org/v2/gh/Shekharrajak/PyCon-SymPy-SymEngine/master?filepath=SymPy_material/tutorial_exercises/01-Symbols-Derivatives-Functions.ipynb) > So then can run the same file in browser as well without installing and > setting up anything new. > > So while writing about the topic in docs (.rst file or .md file), the > writer can refer to the notebook nbviewer URL. The notebook content will be > automatically updated whenever ipynb file is updated. > > > Regards, > Shekhar Prasad Rajak, > Contact : +918142478937 > Blog | Github > | Twitter > > Skype: shekhar.rajak1 > > > > On Tuesday, 8 October 2019, 08:58:16 pm GMT+5:30, Andrew Nelson < > andyfaff at gmail.com> wrote: > > > Another alternative is to use `jupyter-sphinx` and `nbsphinx` sphinx > extensions. You can include a link to the unexecuted notebook in RST (e.g. > https://github.com/refnx/refnx/blob/master/doc/index.rst), and when the > doc build occurs it'll execute the notebook + output as part of the run. > > On Tue, 8 Oct 2019, 17:06 Matthew Brett, wrote: > > Hi, > > On Tue, Oct 8, 2019 at 3:55 PM Ralf Gommers > wrote: > > > > > > > > On Tue, Oct 8, 2019 at 3:22 PM Eric Larson > wrote: > >> > >> I've used sphinx-gallery in a number of projects (am now a maintainer). > Basically you write Python code and it renders to html with links to > downloadable notebooks in the page, automatic linking/cross-refs to > function and class defs, nice index pages, etc. > > > > > > That doesn't really answer my question I think. The starting point is: > notebooks are nice to write a long-form tutorial in, ReST is not. And > sphinx-gallery is nice for a set of independent small to medium sized > examples, but not the best fit for a long tutorial on a whole module (e.g. > http://scipy.github.io/devdocs/tutorial/optimize.html). > > Right - I think sphinx-gallery is the right choice for these small > examples and where you want to develop the tutorial in text. It > solves the ReST output nicely. > > >> > >> > >> Eric > >> > >> > >> On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett > wrote: > >>> > >>> Hi, > >>> > >>> I use Jupytext all the time now, and I prefer the RMarkdown input > >>> format to Jupytext. > >>> > >>> Jupytext can convert .ipynb files to .Rmd files, and saves notebooks > >>> as both .ipynb and .Rmd. > >>> > >>> The workflow is pretty good - after installing Jupytext, you open your > >>> .Rmd file in the notebook interface, and run anything you want, in the > >>> standard interface. > >>> > >>> Then you save, if you need to, which saves to both .ipynb and .Rmd. > >>> > >>> You edit the .Rmd using your nice text editor. > >>> > >>> Then you can reload the notebook in Jupyter, with just a browser > >>> refresh, to see the changes, and to run any extra code you've added. > >>> > >>> I then store the .Rmd files only, in version control, keeping the > >>> .ipynb as outputs from the final build step. > > > > > > Thanks, that sounds like a reasonable workflow. Seems like we can > roundtrip notebook -> plain text format -> notebook. > > Right - exactly - and you can do that interactively, so you can save > from the Jupyter interface, load in a text editor, save in the text > editor and just reload the page in Jupyter to get the result. It will > even keep the outputs as long as you haven't done very substantial > edits. > > > We'll still need to integrate that into the built html docs then, so the > plain text format should be ReST. That doesn't seem supported yet: > https://jupytext.readthedocs.io/en/latest/using-cli.html#testing-the-round-trip-conversion. > Or am I missing something? > > No, that is (currently) a problem that I haven't been working on - > because I'm using Markdown output documents like jupyter-book and > bookdown. One of my collaborators uses notebooks a lot and I was > going to suggest to him that we have a separate site for the notebooks > and the main docs, or at least, we build them separately, and perhaps > combine them after building to HTML. Maybe jupyter-book would help > here? Chris Holdgraf is the maintainer, and he's very responsive. > > Cheers, > > Matthew > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Oct 9 06:56:18 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 9 Oct 2019 03:56:18 -0700 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: On Tue, Oct 8, 2019 at 8:27 AM Andrew Nelson wrote: > Another alternative is to use `jupyter-sphinx` and `nbsphinx` sphinx > extensions. You can include a link to the unexecuted notebook in RST (e.g. > https://github.com/refnx/refnx/blob/master/doc/index.rst), and when the > doc build occurs it'll execute the notebook + output as part of the run. > That looks like the closest fit so far. https://refnx.readthedocs.io/en/latest/getting_started.html looks nice. The parts your setup doesn't seem to do yet are (a) have a "download as notebook" link in the html docs, and (b) get rid of notebook output (e.g. https://raw.githubusercontent.com/refnx/refnx/master/doc/getting_started.ipynb contains the png images) in the repo. Both of those may not be too hard to add. > > On Tue, 8 Oct 2019, 17:06 Matthew Brett, wrote: > >> Hi, >> >> On Tue, Oct 8, 2019 at 3:55 PM Ralf Gommers >> wrote: >> > >> > >> > >> > On Tue, Oct 8, 2019 at 3:22 PM Eric Larson >> wrote: >> >> >> >> I've used sphinx-gallery in a number of projects (am now a >> maintainer). Basically you write Python code and it renders to html with >> links to downloadable notebooks in the page, automatic linking/cross-refs >> to function and class defs, nice index pages, etc. >> > >> > >> > That doesn't really answer my question I think. The starting point is: >> notebooks are nice to write a long-form tutorial in, ReST is not. And >> sphinx-gallery is nice for a set of independent small to medium sized >> examples, but not the best fit for a long tutorial on a whole module (e.g. >> http://scipy.github.io/devdocs/tutorial/optimize.html). >> >> Right - I think sphinx-gallery is the right choice for these small >> examples and where you want to develop the tutorial in text. It >> solves the ReST output nicely. >> >> >> >> >> >> >> Eric >> >> >> >> >> >> On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett >> wrote: >> >>> >> >>> Hi, >> >>> >> >>> I use Jupytext all the time now, and I prefer the RMarkdown input >> >>> format to Jupytext. >> >>> >> >>> Jupytext can convert .ipynb files to .Rmd files, and saves notebooks >> >>> as both .ipynb and .Rmd. >> >>> >> >>> The workflow is pretty good - after installing Jupytext, you open your >> >>> .Rmd file in the notebook interface, and run anything you want, in the >> >>> standard interface. >> >>> >> >>> Then you save, if you need to, which saves to both .ipynb and .Rmd. >> >>> >> >>> You edit the .Rmd using your nice text editor. >> >>> >> >>> Then you can reload the notebook in Jupyter, with just a browser >> >>> refresh, to see the changes, and to run any extra code you've added. >> >>> >> >>> I then store the .Rmd files only, in version control, keeping the >> >>> .ipynb as outputs from the final build step. >> > >> > >> > Thanks, that sounds like a reasonable workflow. Seems like we can >> roundtrip notebook -> plain text format -> notebook. >> >> Right - exactly - and you can do that interactively, so you can save >> from the Jupyter interface, load in a text editor, save in the text >> editor and just reload the page in Jupyter to get the result. It will >> even keep the outputs as long as you haven't done very substantial >> edits. >> >> > We'll still need to integrate that into the built html docs then, so >> the plain text format should be ReST. That doesn't seem supported yet: >> https://jupytext.readthedocs.io/en/latest/using-cli.html#testing-the-round-trip-conversion. >> Or am I missing something? >> >> No, that is (currently) a problem that I haven't been working on - >> because I'm using Markdown output documents like jupyter-book and >> bookdown. One of my collaborators uses notebooks a lot and I was >> going to suggest to him that we have a separate site for the notebooks >> and the main docs, or at least, we build them separately, and perhaps >> combine them after building to HTML. Maybe jupyter-book would help >> here? > > If we would start over, something like jupyter-book may be a better option. I'd prefer to avoid a major overhaul of all tutorials though, because there's a *lot* of content written in reST files already. Ideally we keep what we have in reST, and add the ability to write new ones in notebooks, which could be downloaded standalone but also integrate nicely with the existing Sphinx html build. Chris Holdgraf is the maintainer, and he's very responsive. >> > Thanks for the suggestion. Yes Chris is great, may ask him for advice. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Wed Oct 9 07:42:54 2019 From: andyfaff at gmail.com (Andrew Nelson) Date: Wed, 9 Oct 2019 13:42:54 +0200 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: On Wed, 9 Oct 2019, 12:57 Ralf Gommers, wrote: > > The parts your setup doesn't seem to do yet are (a) have a "download as >> notebook" link in the html docs, and (b) get rid of notebook output (e.g. >> https://raw.githubusercontent.com/refnx/refnx/master/doc/getting_started.ipynb >> contains the png images) in the repo. Both of those may not be too hard to >> add. >> > That's because I reached the limit of my knowledge, especially wrt sphinx and rst. I'm not claiming any further expertise, but execution of the notebook during docbuild seemed the best thing for me. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Oct 9 07:55:08 2019 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 9 Oct 2019 12:55:08 +0100 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: Oh - sorry - I should say that I have also used my own nb2plots package: http://matthew-brett.github.io/nb2plots/ Examples here: https://matthew-brett.github.io/teaching In particular - here's a page with the nbplots Sphinx extension: https://matthew-brett.github.io/teaching/glm_intro.html This is somewhere between Jupytext (where you can keep using the notebook as a format for editing) and sphinx-gallery (where you start and stay in text for editing). The workflow is: * Draft up a tutorial in the notebook * Convert to a RST page with the nb2plots command * Hand edit the RST a bit to get it into shape * Continue editing the RST to finish up the document * The extension builds the RST as a notebook for download, at the build stage, and adds a link to the notebook in the built page. See http://matthew-brett.github.io/nb2plots/worked_example.html for a short example. Cheers, Matthew On Wed, Oct 9, 2019 at 11:57 AM Ralf Gommers wrote: > > > > On Tue, Oct 8, 2019 at 8:27 AM Andrew Nelson wrote: >> >> Another alternative is to use `jupyter-sphinx` and `nbsphinx` sphinx extensions. You can include a link to the unexecuted notebook in RST (e.g. https://github.com/refnx/refnx/blob/master/doc/index.rst), and when the doc build occurs it'll execute the notebook + output as part of the run. > > > That looks like the closest fit so far. https://refnx.readthedocs.io/en/latest/getting_started.html looks nice. The parts your setup doesn't seem to do yet are (a) have a "download as notebook" link in the html docs, and (b) get rid of notebook output (e.g. https://raw.githubusercontent.com/refnx/refnx/master/doc/getting_started.ipynb contains the png images) in the repo. Both of those may not be too hard to add. > >> >> >> On Tue, 8 Oct 2019, 17:06 Matthew Brett, wrote: >>> >>> Hi, >>> >>> On Tue, Oct 8, 2019 at 3:55 PM Ralf Gommers wrote: >>> > >>> > >>> > >>> > On Tue, Oct 8, 2019 at 3:22 PM Eric Larson wrote: >>> >> >>> >> I've used sphinx-gallery in a number of projects (am now a maintainer). Basically you write Python code and it renders to html with links to downloadable notebooks in the page, automatic linking/cross-refs to function and class defs, nice index pages, etc. >>> > >>> > >>> > That doesn't really answer my question I think. The starting point is: notebooks are nice to write a long-form tutorial in, ReST is not. And sphinx-gallery is nice for a set of independent small to medium sized examples, but not the best fit for a long tutorial on a whole module (e.g. http://scipy.github.io/devdocs/tutorial/optimize.html). >>> >>> Right - I think sphinx-gallery is the right choice for these small >>> examples and where you want to develop the tutorial in text. It >>> solves the ReST output nicely. >>> >>> >> >>> >> >>> >> Eric >>> >> >>> >> >>> >> On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett wrote: >>> >>> >>> >>> Hi, >>> >>> >>> >>> I use Jupytext all the time now, and I prefer the RMarkdown input >>> >>> format to Jupytext. >>> >>> >>> >>> Jupytext can convert .ipynb files to .Rmd files, and saves notebooks >>> >>> as both .ipynb and .Rmd. >>> >>> >>> >>> The workflow is pretty good - after installing Jupytext, you open your >>> >>> .Rmd file in the notebook interface, and run anything you want, in the >>> >>> standard interface. >>> >>> >>> >>> Then you save, if you need to, which saves to both .ipynb and .Rmd. >>> >>> >>> >>> You edit the .Rmd using your nice text editor. >>> >>> >>> >>> Then you can reload the notebook in Jupyter, with just a browser >>> >>> refresh, to see the changes, and to run any extra code you've added. >>> >>> >>> >>> I then store the .Rmd files only, in version control, keeping the >>> >>> .ipynb as outputs from the final build step. >>> > >>> > >>> > Thanks, that sounds like a reasonable workflow. Seems like we can roundtrip notebook -> plain text format -> notebook. >>> >>> Right - exactly - and you can do that interactively, so you can save >>> from the Jupyter interface, load in a text editor, save in the text >>> editor and just reload the page in Jupyter to get the result. It will >>> even keep the outputs as long as you haven't done very substantial >>> edits. >>> >>> > We'll still need to integrate that into the built html docs then, so the plain text format should be ReST. That doesn't seem supported yet: https://jupytext.readthedocs.io/en/latest/using-cli.html#testing-the-round-trip-conversion. Or am I missing something? >>> >>> No, that is (currently) a problem that I haven't been working on - >>> because I'm using Markdown output documents like jupyter-book and >>> bookdown. One of my collaborators uses notebooks a lot and I was >>> going to suggest to him that we have a separate site for the notebooks >>> and the main docs, or at least, we build them separately, and perhaps >>> combine them after building to HTML. Maybe jupyter-book would help >>> here? > > > If we would start over, something like jupyter-book may be a better option. I'd prefer to avoid a major overhaul of all tutorials though, because there's a *lot* of content written in reST files already. Ideally we keep what we have in reST, and add the ability to write new ones in notebooks, which could be downloaded standalone but also integrate nicely with the existing Sphinx html build. > >>> Chris Holdgraf is the maintainer, and he's very responsive. > > > Thanks for the suggestion. Yes Chris is great, may ask him for advice. > > Cheers, > Ralf > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From andyfaff at gmail.com Wed Oct 9 08:24:54 2019 From: andyfaff at gmail.com (Andrew Nelson) Date: Wed, 9 Oct 2019 14:24:54 +0200 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: The overview for nbsphinx is at https://nbsphinx.readthedocs.io/en/0.4.3/ On Wed, 9 Oct 2019, 12:57 Ralf Gommers, wrote: > > > On Tue, Oct 8, 2019 at 8:27 AM Andrew Nelson wrote: > >> Another alternative is to use `jupyter-sphinx` and `nbsphinx` sphinx >> extensions. You can include a link to the unexecuted notebook in RST (e.g. >> https://github.com/refnx/refnx/blob/master/doc/index.rst), and when the >> doc build occurs it'll execute the notebook + output as part of the run. >> > > That looks like the closest fit so far. > https://refnx.readthedocs.io/en/latest/getting_started.html looks nice. > The parts your setup doesn't seem to do yet are (a) have a "download as > notebook" link in the html docs, and (b) get rid of notebook output (e.g. > https://raw.githubusercontent.com/refnx/refnx/master/doc/getting_started.ipynb > contains the png images) in the repo. Both of those may not be too hard to > add. > > >> >> On Tue, 8 Oct 2019, 17:06 Matthew Brett, wrote: >> >>> Hi, >>> >>> On Tue, Oct 8, 2019 at 3:55 PM Ralf Gommers >>> wrote: >>> > >>> > >>> > >>> > On Tue, Oct 8, 2019 at 3:22 PM Eric Larson >>> wrote: >>> >> >>> >> I've used sphinx-gallery in a number of projects (am now a >>> maintainer). Basically you write Python code and it renders to html with >>> links to downloadable notebooks in the page, automatic linking/cross-refs >>> to function and class defs, nice index pages, etc. >>> > >>> > >>> > That doesn't really answer my question I think. The starting point is: >>> notebooks are nice to write a long-form tutorial in, ReST is not. And >>> sphinx-gallery is nice for a set of independent small to medium sized >>> examples, but not the best fit for a long tutorial on a whole module (e.g. >>> http://scipy.github.io/devdocs/tutorial/optimize.html). >>> >>> Right - I think sphinx-gallery is the right choice for these small >>> examples and where you want to develop the tutorial in text. It >>> solves the ReST output nicely. >>> >>> >> >>> >> >>> >> Eric >>> >> >>> >> >>> >> On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett >>> wrote: >>> >>> >>> >>> Hi, >>> >>> >>> >>> I use Jupytext all the time now, and I prefer the RMarkdown input >>> >>> format to Jupytext. >>> >>> >>> >>> Jupytext can convert .ipynb files to .Rmd files, and saves notebooks >>> >>> as both .ipynb and .Rmd. >>> >>> >>> >>> The workflow is pretty good - after installing Jupytext, you open >>> your >>> >>> .Rmd file in the notebook interface, and run anything you want, in >>> the >>> >>> standard interface. >>> >>> >>> >>> Then you save, if you need to, which saves to both .ipynb and .Rmd. >>> >>> >>> >>> You edit the .Rmd using your nice text editor. >>> >>> >>> >>> Then you can reload the notebook in Jupyter, with just a browser >>> >>> refresh, to see the changes, and to run any extra code you've added. >>> >>> >>> >>> I then store the .Rmd files only, in version control, keeping the >>> >>> .ipynb as outputs from the final build step. >>> > >>> > >>> > Thanks, that sounds like a reasonable workflow. Seems like we can >>> roundtrip notebook -> plain text format -> notebook. >>> >>> Right - exactly - and you can do that interactively, so you can save >>> from the Jupyter interface, load in a text editor, save in the text >>> editor and just reload the page in Jupyter to get the result. It will >>> even keep the outputs as long as you haven't done very substantial >>> edits. >>> >>> > We'll still need to integrate that into the built html docs then, so >>> the plain text format should be ReST. That doesn't seem supported yet: >>> https://jupytext.readthedocs.io/en/latest/using-cli.html#testing-the-round-trip-conversion. >>> Or am I missing something? >>> >>> No, that is (currently) a problem that I haven't been working on - >>> because I'm using Markdown output documents like jupyter-book and >>> bookdown. One of my collaborators uses notebooks a lot and I was >>> going to suggest to him that we have a separate site for the notebooks >>> and the main docs, or at least, we build them separately, and perhaps >>> combine them after building to HTML. Maybe jupyter-book would help >>> here? >> >> > If we would start over, something like jupyter-book may be a better > option. I'd prefer to avoid a major overhaul of all tutorials though, > because there's a *lot* of content written in reST files already. Ideally > we keep what we have in reST, and add the ability to write new ones in > notebooks, which could be downloaded standalone but also integrate nicely > with the existing Sphinx html build. > > Chris Holdgraf is the maintainer, and he's very responsive. >>> >> > Thanks for the suggestion. Yes Chris is great, may ask him for advice. > > Cheers, > Ralf > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Thu Oct 10 02:34:06 2019 From: andyfaff at gmail.com (Andrew Nelson) Date: Thu, 10 Oct 2019 08:34:06 +0200 Subject: [SciPy-Dev] Github workflow Message-ID: devs, I'm involved in giving some feedback to Github at the moment, as related to how SciPy users experience Github. Does anyone have any feedback on: - Bugs / Feature requests - are there Github workflows you want to enable? What's the specific feature you want? This would be an ideal opportunity to tell GIthub of those features you'd really like to see fixed or implemented. A. -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Oct 10 12:11:17 2019 From: toddrjen at gmail.com (Todd) Date: Thu, 10 Oct 2019 12:11:17 -0400 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: On Thu, Oct 10, 2019 at 2:34 AM Andrew Nelson wrote: > devs, > I'm involved in giving some feedback to Github at the moment, as related > to how SciPy users experience Github. Does anyone have any feedback on: > > - Bugs / Feature requests - are there Github workflows you want to enable? > What's the specific feature you want? > > This would be an ideal opportunity to tell GIthub of those features you'd > really like to see fixed or implemented. > > A. > > The "quote reply" button in PR's doesn't add the name of the person you are talking to, nor does it link to the comment, and I can't find another way to make really threaded replies. At the very least having quotes automatically tag the person being quoted so they get a notification about it, but better yet having some sort of threaded reply system outside of comments on particular lines, would be useful in my opinion. Also, although there is a way to comment on particular lines of code in a PR, I can't figure out a way to link to particular lines of code. That would be useful. For example someone asks a question in a comment on one line, you can link to another line that has the answer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Thu Oct 10 12:34:18 2019 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Thu, 10 Oct 2019 18:34:18 +0200 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: On Thu, 10 Oct 2019 at 18:11, Todd wrote: > On Thu, Oct 10, 2019 at 2:34 AM Andrew Nelson wrote: > >> devs, >> I'm involved in giving some feedback to Github at the moment, as related >> to how SciPy users experience Github. Does anyone have any feedback on: >> >> - Bugs / Feature requests - are there Github workflows you want to >> enable? What's the specific feature you want? >> >> This would be an ideal opportunity to tell GIthub of those features you'd >> really like to see fixed or implemented. >> >> A. >> >> > The "quote reply" button in PR's doesn't add the name of the person you > are talking to, nor does it link to the comment, and I can't find another > way to make really threaded replies. At the very least having quotes > automatically tag the person being quoted so they get a notification about > it, but better yet having some sort of threaded reply system outside of > comments on particular lines, would be useful in my opinion. > > Also, although there is a way to comment on particular lines of code in a > PR, I can't figure out a way to link to particular lines of code. That > would be useful. For example someone asks a question in a comment on one > line, you can link to another line that has the answer. > That you can do: https://github.com/numpy/numpy/blob/master/numpy/core/machar.py#L128-L133 And even a permalink to the specific commit: https://github.com/numpy/numpy/blob/9451d6f26ab9ab1bd204bf1865ac743a5dff6ae0/numpy/core/machar.py#L128-L133 To do this, click on the line number. Shift-click to select a range of lines. _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Oct 10 13:19:51 2019 From: toddrjen at gmail.com (Todd) Date: Thu, 10 Oct 2019 13:19:51 -0400 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: On Thu, Oct 10, 2019, 12:34 Da?id wrote: > > > On Thu, 10 Oct 2019 at 18:11, Todd wrote: > >> On Thu, Oct 10, 2019 at 2:34 AM Andrew Nelson wrote: >> >>> devs, >>> I'm involved in giving some feedback to Github at the moment, as related >>> to how SciPy users experience Github. Does anyone have any feedback on: >>> >>> - Bugs / Feature requests - are there Github workflows you want to >>> enable? What's the specific feature you want? >>> >>> This would be an ideal opportunity to tell GIthub of those features >>> you'd really like to see fixed or implemented. >>> >>> A. >>> >>> >> The "quote reply" button in PR's doesn't add the name of the person you >> are talking to, nor does it link to the comment, and I can't find another >> way to make really threaded replies. At the very least having quotes >> automatically tag the person being quoted so they get a notification about >> it, but better yet having some sort of threaded reply system outside of >> comments on particular lines, would be useful in my opinion. >> >> Also, although there is a way to comment on particular lines of code in a >> PR, I can't figure out a way to link to particular lines of code. That >> would be useful. For example someone asks a question in a comment on one >> line, you can link to another line that has the answer. >> > > > That you can do: > https://github.com/numpy/numpy/blob/master/numpy/core/machar.py#L128-L133 > > And even a permalink to the specific commit: > > > https://github.com/numpy/numpy/blob/9451d6f26ab9ab1bd204bf1865ac743a5dff6ae0/numpy/core/machar.py#L128-L133 > > To do this, click on the line number. Shift-click to select a range of > lines. > Yes, you can link to lines in the repository or a commit, but not in the pull request itself as far as I can tell. I think it would be helpful when discussing the pull request to be able to link to a line inside the pull request interface. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Thu Oct 10 14:17:33 2019 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Thu, 10 Oct 2019 20:17:33 +0200 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: On Thu, 10 Oct 2019 at 19:21, Todd wrote: > > > On Thu, Oct 10, 2019, 12:34 Da?id wrote: > >> >> >> On Thu, 10 Oct 2019 at 18:11, Todd wrote: >> >>> On Thu, Oct 10, 2019 at 2:34 AM Andrew Nelson >>> wrote: >>> >>>> devs, >>>> I'm involved in giving some feedback to Github at the moment, as >>>> related to how SciPy users experience Github. Does anyone have any feedback >>>> on: >>>> >>>> - Bugs / Feature requests - are there Github workflows you want to >>>> enable? What's the specific feature you want? >>>> >>>> This would be an ideal opportunity to tell GIthub of those features >>>> you'd really like to see fixed or implemented. >>>> >>>> A. >>>> >>>> >>> The "quote reply" button in PR's doesn't add the name of the person you >>> are talking to, nor does it link to the comment, and I can't find another >>> way to make really threaded replies. At the very least having quotes >>> automatically tag the person being quoted so they get a notification about >>> it, but better yet having some sort of threaded reply system outside of >>> comments on particular lines, would be useful in my opinion. >>> >>> Also, although there is a way to comment on particular lines of code in >>> a PR, I can't figure out a way to link to particular lines of code. That >>> would be useful. For example someone asks a question in a comment on one >>> line, you can link to another line that has the answer. >>> >> >> >> That you can do: >> https://github.com/numpy/numpy/blob/master/numpy/core/machar.py#L128-L133 >> >> And even a permalink to the specific commit: >> >> >> https://github.com/numpy/numpy/blob/9451d6f26ab9ab1bd204bf1865ac743a5dff6ae0/numpy/core/machar.py#L128-L133 >> >> To do this, click on the line number. Shift-click to select a range of >> lines. >> > > Yes, you can link to lines in the repository or a commit, but not in the > pull request itself as far as I can tell. I think it would be helpful when > discussing the pull request to be able to link to a line inside the pull > request interface. > If I am understanding what you mean correctly, same trick still works, click on the line numbers: https://github.com/numpy/numpy/pull/14657/files#diff-758e4b79c5a0e4db54723d82a8436418R3617-R3619 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Oct 10 14:23:30 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 10 Oct 2019 20:23:30 +0200 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: On Thu, Oct 10, 2019 at 8:34 AM Andrew Nelson wrote: > devs, > I'm involved in giving some feedback to Github at the moment, as related > to how SciPy users experience Github. Does anyone have any feedback on: > > - Bugs / Feature requests - are there Github workflows you want to enable? > What's the specific feature you want? > > This would be an ideal opportunity to tell GIthub of those features you'd > really like to see fixed or implemented. > Getting rid of that super dangerous "delete my fork" button on merged PRs would be high on my list. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Thu Oct 10 14:32:45 2019 From: andyfaff at gmail.com (Andrew Nelson) Date: Thu, 10 Oct 2019 20:32:45 +0200 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: > > Getting rid of that super dangerous "delete my fork" button on merged PRs > would be high on my list. > I only ever see "delete my branch", not "delete my fork". Is there a danger to "deleting my (feature) branch"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Oct 10 14:34:29 2019 From: toddrjen at gmail.com (Todd) Date: Thu, 10 Oct 2019 14:34:29 -0400 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: On Thu, Oct 10, 2019 at 2:18 PM Da?id wrote: > > > On Thu, 10 Oct 2019 at 19:21, Todd wrote: > >> >> >> On Thu, Oct 10, 2019, 12:34 Da?id wrote: >> >>> >>> >>> On Thu, 10 Oct 2019 at 18:11, Todd wrote: >>> >>>> On Thu, Oct 10, 2019 at 2:34 AM Andrew Nelson >>>> wrote: >>>> >>>>> devs, >>>>> I'm involved in giving some feedback to Github at the moment, as >>>>> related to how SciPy users experience Github. Does anyone have any feedback >>>>> on: >>>>> >>>>> - Bugs / Feature requests - are there Github workflows you want to >>>>> enable? What's the specific feature you want? >>>>> >>>>> This would be an ideal opportunity to tell GIthub of those features >>>>> you'd really like to see fixed or implemented. >>>>> >>>>> A. >>>>> >>>>> >>>> The "quote reply" button in PR's doesn't add the name of the person you >>>> are talking to, nor does it link to the comment, and I can't find another >>>> way to make really threaded replies. At the very least having quotes >>>> automatically tag the person being quoted so they get a notification about >>>> it, but better yet having some sort of threaded reply system outside of >>>> comments on particular lines, would be useful in my opinion. >>>> >>>> Also, although there is a way to comment on particular lines of code in >>>> a PR, I can't figure out a way to link to particular lines of code. That >>>> would be useful. For example someone asks a question in a comment on one >>>> line, you can link to another line that has the answer. >>>> >>> >>> >>> That you can do: >>> https://github.com/numpy/numpy/blob/master/numpy/core/machar.py#L128-L133 >>> >>> And even a permalink to the specific commit: >>> >>> >>> https://github.com/numpy/numpy/blob/9451d6f26ab9ab1bd204bf1865ac743a5dff6ae0/numpy/core/machar.py#L128-L133 >>> >>> To do this, click on the line number. Shift-click to select a range of >>> lines. >>> >> >> Yes, you can link to lines in the repository or a commit, but not in the >> pull request itself as far as I can tell. I think it would be helpful when >> discussing the pull request to be able to link to a line inside the pull >> request interface. >> > > If I am understanding what you mean correctly, same trick still works, > click on the line numbers: > > > https://github.com/numpy/numpy/pull/14657/files#diff-758e4b79c5a0e4db54723d82a8436418R3617-R3619 > I see, thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From haberland at ucla.edu Thu Oct 10 14:38:48 2019 From: haberland at ucla.edu (Matt Haberland) Date: Thu, 10 Oct 2019 11:38:48 -0700 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: I second the request to have threaded comments. I would like to be able to "reply" to regular comments like I can "reply" to review comments. GitHub already has some nice features for comparing code diffs. For instance, sometimes it detects not only which line has changed but also what *part *of a line has changed. I would ask for several others, like being able to toggle the visibility of trivial diffs. For example, gh-10602 has a lot of documentation changes that involve extra indentation. It's difficult to tell whether the indentation is the *only* thing that has changed, or if other parts of the line have changed. It would be nice if we could add review comments to lines of a file that have not already been changed by a PR (and don't immediately precede/follow the changed lines). I like the "insert a suggestion" feature in code review. I'd appreciate a way to suggest changes to multiple lines at once. On Thu, Oct 10, 2019 at 9:12 AM Todd wrote: > On Thu, Oct 10, 2019 at 2:34 AM Andrew Nelson wrote: > >> devs, >> I'm involved in giving some feedback to Github at the moment, as related >> to how SciPy users experience Github. Does anyone have any feedback on: >> >> - Bugs / Feature requests - are there Github workflows you want to >> enable? What's the specific feature you want? >> >> This would be an ideal opportunity to tell GIthub of those features you'd >> really like to see fixed or implemented. >> >> A. >> >> > The "quote reply" button in PR's doesn't add the name of the person you > are talking to, nor does it link to the comment, and I can't find another > way to make really threaded replies. At the very least having quotes > automatically tag the person being quoted so they get a notification about > it, but better yet having some sort of threaded reply system outside of > comments on particular lines, would be useful in my opinion. > > Also, although there is a way to comment on particular lines of code in a > PR, I can't figure out a way to link to particular lines of code. That > would be useful. For example someone asks a question in a comment on one > line, you can link to another line that has the answer. > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Matt Haberland Assistant Adjunct Professor in the Program in Computing Department of Mathematics 6617A Math Sciences Building, UCLA -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Oct 10 15:40:11 2019 From: toddrjen at gmail.com (Todd) Date: Thu, 10 Oct 2019 15:40:11 -0400 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: On Thu, Oct 10, 2019 at 2:39 PM Matt Haberland wrote: > I second the request to have threaded comments. I would like to be able to > "reply" to regular comments like I can "reply" to review comments. > > GitHub already has some nice features for comparing code diffs. For > instance, sometimes it detects not only which line has changed but also > what *part *of a line has changed. I would ask for several others, like > being able to toggle the visibility of trivial diffs. For example, > gh-10602 has a lot of > documentation changes that involve extra indentation. It's difficult to > tell whether the indentation is the *only* thing that has changed, or if > other parts of the line have changed. > > Near the top left of the pull request there is a gear icon. If you click that that you can hide whitespace changes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Oct 10 16:10:00 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 10 Oct 2019 22:10:00 +0200 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: On Thu, Oct 10, 2019 at 8:33 PM Andrew Nelson wrote: > Getting rid of that super dangerous "delete my fork" button on merged PRs >> would be high on my list. >> > > I only ever see "delete my branch", not "delete my fork". Is there a > danger to "deleting my (feature) branch"? > That's always safe. But after you click it, you should be getting (in the exact same location) a "delete my fork". At least I am. So don't click twice! Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias.geier at gmail.com Fri Oct 11 13:35:19 2019 From: matthias.geier at gmail.com (Matthias Geier) Date: Fri, 11 Oct 2019 19:35:19 +0200 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: OK, nbsphinx has already been mentioned, so I don't have to (I'm its author, BTW). I just wanted to reinforce what has been said about not storing cell outputs in the repository. That's a good idea! Notebooks can be "cleaned" automatically, see: https://nbsphinx.readthedocs.io/en/0.4.3/usage.html#Using-Notebooks-with-Git It's also possible to have most of the repository without outputs, but only one branch containing executed notebooks. I've written about this recently: https://mg.readthedocs.io/git-jupyter.html Even without output, Jupyter notebooks are a bit annoying in version control, because all strings are quoted in the JSON representation and there are often plenty of escaped characters. To avoid this, something like jupytext is great, but it adds quite some of complexity for users. Ideally, the native Jupyter notebook format should be changed, but I don't know if that's likely to happen. FWIW, I've made a suggestion (https://jupyter-format.readthedocs.io/), but I guess that's a topic for the far future ... Regarding what Ralf said: > (a) have a "download as notebook" link in the html docs That's easy, just look at the nbsphinx docs, e.g. https://nbsphinx.readthedocs.io/en/0.4.3/code-cells.html. At the top of the page there is a download link to the notebook on Github and a link for launching it on Binder. Arbitrary such links can be created, see https://nbsphinx.readthedocs.io/en/0.4.3/prolog-and-epilog.html. cheers, Matthias On Wed, Oct 9, 2019 at 2:26 PM Andrew Nelson wrote: > > The overview for nbsphinx is at https://nbsphinx.readthedocs.io/en/0.4.3/ > > On Wed, 9 Oct 2019, 12:57 Ralf Gommers, wrote: >> >> >> >> On Tue, Oct 8, 2019 at 8:27 AM Andrew Nelson wrote: >>> >>> Another alternative is to use `jupyter-sphinx` and `nbsphinx` sphinx extensions. You can include a link to the unexecuted notebook in RST (e.g. https://github.com/refnx/refnx/blob/master/doc/index.rst), and when the doc build occurs it'll execute the notebook + output as part of the run. >> >> >> That looks like the closest fit so far. https://refnx.readthedocs.io/en/latest/getting_started.html looks nice. The parts your setup doesn't seem to do yet are (a) have a "download as notebook" link in the html docs, and (b) get rid of notebook output (e.g. https://raw.githubusercontent.com/refnx/refnx/master/doc/getting_started.ipynb contains the png images) in the repo. Both of those may not be too hard to add. >> >>> >>> >>> On Tue, 8 Oct 2019, 17:06 Matthew Brett, wrote: >>>> >>>> Hi, >>>> >>>> On Tue, Oct 8, 2019 at 3:55 PM Ralf Gommers wrote: >>>> > >>>> > >>>> > >>>> > On Tue, Oct 8, 2019 at 3:22 PM Eric Larson wrote: >>>> >> >>>> >> I've used sphinx-gallery in a number of projects (am now a maintainer). Basically you write Python code and it renders to html with links to downloadable notebooks in the page, automatic linking/cross-refs to function and class defs, nice index pages, etc. >>>> > >>>> > >>>> > That doesn't really answer my question I think. The starting point is: notebooks are nice to write a long-form tutorial in, ReST is not. And sphinx-gallery is nice for a set of independent small to medium sized examples, but not the best fit for a long tutorial on a whole module (e.g. http://scipy.github.io/devdocs/tutorial/optimize.html). >>>> >>>> Right - I think sphinx-gallery is the right choice for these small >>>> examples and where you want to develop the tutorial in text. It >>>> solves the ReST output nicely. >>>> >>>> >> >>>> >> >>>> >> Eric >>>> >> >>>> >> >>>> >> On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett wrote: >>>> >>> >>>> >>> Hi, >>>> >>> >>>> >>> I use Jupytext all the time now, and I prefer the RMarkdown input >>>> >>> format to Jupytext. >>>> >>> >>>> >>> Jupytext can convert .ipynb files to .Rmd files, and saves notebooks >>>> >>> as both .ipynb and .Rmd. >>>> >>> >>>> >>> The workflow is pretty good - after installing Jupytext, you open your >>>> >>> .Rmd file in the notebook interface, and run anything you want, in the >>>> >>> standard interface. >>>> >>> >>>> >>> Then you save, if you need to, which saves to both .ipynb and .Rmd. >>>> >>> >>>> >>> You edit the .Rmd using your nice text editor. >>>> >>> >>>> >>> Then you can reload the notebook in Jupyter, with just a browser >>>> >>> refresh, to see the changes, and to run any extra code you've added. >>>> >>> >>>> >>> I then store the .Rmd files only, in version control, keeping the >>>> >>> .ipynb as outputs from the final build step. >>>> > >>>> > >>>> > Thanks, that sounds like a reasonable workflow. Seems like we can roundtrip notebook -> plain text format -> notebook. >>>> >>>> Right - exactly - and you can do that interactively, so you can save >>>> from the Jupyter interface, load in a text editor, save in the text >>>> editor and just reload the page in Jupyter to get the result. It will >>>> even keep the outputs as long as you haven't done very substantial >>>> edits. >>>> >>>> > We'll still need to integrate that into the built html docs then, so the plain text format should be ReST. That doesn't seem supported yet: https://jupytext.readthedocs.io/en/latest/using-cli.html#testing-the-round-trip-conversion. Or am I missing something? >>>> >>>> No, that is (currently) a problem that I haven't been working on - >>>> because I'm using Markdown output documents like jupyter-book and >>>> bookdown. One of my collaborators uses notebooks a lot and I was >>>> going to suggest to him that we have a separate site for the notebooks >>>> and the main docs, or at least, we build them separately, and perhaps >>>> combine them after building to HTML. Maybe jupyter-book would help >>>> here? >> >> >> If we would start over, something like jupyter-book may be a better option. I'd prefer to avoid a major overhaul of all tutorials though, because there's a *lot* of content written in reST files already. Ideally we keep what we have in reST, and add the ability to write new ones in notebooks, which could be downloaded standalone but also integrate nicely with the existing Sphinx html build. >> >>>> Chris Holdgraf is the maintainer, and he's very responsive. >> >> >> Thanks for the suggestion. Yes Chris is great, may ask him for advice. >> >> Cheers, >> Ralf >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From matthias.geier at gmail.com Fri Oct 11 13:44:53 2019 From: matthias.geier at gmail.com (Matthias Geier) Date: Fri, 11 Oct 2019 19:44:53 +0200 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: On Thu, Oct 10, 2019 at 8:35 AM Andrew Nelson wrote: > > devs, > I'm involved in giving some feedback to Github at the moment, as related to how SciPy users experience Github. Does anyone have any feedback on: > > - Bugs / Feature requests - are there Github workflows you want to enable? What's the specific feature you want? > > This would be an ideal opportunity to tell GIthub of those features you'd really like to see fixed or implemented. I think this is a nice list by Aaron Meurer: https://www.asmeurer.com/blog/posts/github-cuts/ What annoys me most is that in long discussions some comments are hidden/collapsed and large diffs are also not shown by default. Typically, those are the most interesting and I can't use the "search" function of the browser to locate things. cheers, Matthias From pav at iki.fi Fri Oct 11 14:49:45 2019 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Oct 2019 21:49:45 +0300 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: <56358475aee7ef97f1a11bdaeee2bf9299741ac6.camel@iki.fi> to, 2019-10-10 kello 08:34 +0200, Andrew Nelson kirjoitti: > devs, > I'm involved in giving some feedback to Github at the moment, as > related to how SciPy users experience Github. Does anyone have any > feedback on: > > - Bugs / Feature requests - are there Github workflows you want to > enable? What's the specific feature you want? There are low-hanging fruits for improving the review workflow. Here's the most important: Add a button the PR submitter / maintainers can press, which adds a status marker for "requesting further review from *someone*". New PRs would have the status flag set, and the submitter could use this to mark a PR ready after making requested changes. This becomes important when you have a large number of pull requests at various stages of progress. The "Reviewers" box on the upper right on the PR page is something like this, but does not work in practice: (1) PR submitters generally don't use it, usually don't know who they should choose, and there's no way to set the default value. (2) Requesting specific developers to review something is not a good match for open-source projects with many developers. (3) In Github UI you can only filter for "Awaiting review from you" but not "Awaiting review from someone". (4) Most importantly, it's not possible to exclude PRs with the "Reviewed by someone, and currently not awaiting review" status out. Labels cannot be used for this as-is, because the PR submitter cannot modify them. The comment box is also not useful for this (no way to search/filter for that in Github UI, and the 100+ daily email volume from Github already prevents manual tracking). Pauli From pav at iki.fi Fri Oct 11 14:57:03 2019 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Oct 2019 21:57:03 +0300 Subject: [SciPy-Dev] Github workflow In-Reply-To: <56358475aee7ef97f1a11bdaeee2bf9299741ac6.camel@iki.fi> References: <56358475aee7ef97f1a11bdaeee2bf9299741ac6.camel@iki.fi> Message-ID: <082613d2dfd34d949aca3ff5e578cadf2f4c9c3e.camel@iki.fi> pe, 2019-10-11 kello 21:49 +0300, Pauli Virtanen kirjoitti: > to, 2019-10-10 kello 08:34 +0200, Andrew Nelson kirjoitti: > > devs, > > I'm involved in giving some feedback to Github at the moment, as > > related to how SciPy users experience Github. Does anyone have any > > feedback on: > > > > - Bugs / Feature requests - are there Github workflows you want to > > enable? What's the specific feature you want? > > There are low-hanging fruits for improving the review workflow. > Here's the most important: > > Add a button the PR submitter / maintainers can press, which adds a > status marker for "requesting further review from *someone*". New PRs > would have the status flag set, and the submitter could use this to > mark a PR ready after making requested changes. I'd also like to note that Aaron Meurer's blog post makes a similar point (although pushing commits instead of a button): https://www.asmeurer.com/blog/posts/github-reviews-gripes/ """ Next gripe, and this, I want to stress, makes the whole feature completely useless for my needs: reviews do not reset when new commits are pushed. [clip] But unlike your automated test suite, which reset and get a new go every time you push a change (because hey, who knows, maybe the change ACTUALLY FIXED THE ISSUE), reviews do not reset, unless the original reviewers explicitly change them. So my dream of being able to glance at the pull request list and see which PRs need changes has officially been piped. Even if the list actually showed what PRs have been reviewed, it would be a lie, because as soon as the PR author pushes a change, the review status becomes potentially outdated. """ From rmay31 at gmail.com Fri Oct 11 15:10:48 2019 From: rmay31 at gmail.com (Ryan May) Date: Fri, 11 Oct 2019 13:10:48 -0600 Subject: [SciPy-Dev] Github workflow In-Reply-To: <082613d2dfd34d949aca3ff5e578cadf2f4c9c3e.camel@iki.fi> References: <56358475aee7ef97f1a11bdaeee2bf9299741ac6.camel@iki.fi> <082613d2dfd34d949aca3ff5e578cadf2f4c9c3e.camel@iki.fi> Message-ID: > > > > > Add a button the PR submitter / maintainers can press, which adds a > > status marker for "requesting further review from *someone*". New PRs > > would have the status flag set, and the submitter could use this to > > mark a PR ready after making requested changes. > > I'd also like to note that Aaron Meurer's blog post makes a similar > point (although pushing commits instead of a button): > > https://www.asmeurer.com/blog/posts/github-reviews-gripes/ > > """ > Next gripe, and this, I want to stress, makes the whole feature > completely useless for my needs: reviews do not reset when new commits > are pushed. > I'm in favor of having the "request new review" or making it easier to query. I just want to clarify that under "Branch Protection" you can actually set that reviews are cleared on new pushes: [image: image.png] Ryan -- Ryan May -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 240406 bytes Desc: not available URL: From pav at iki.fi Fri Oct 11 15:16:59 2019 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 11 Oct 2019 22:16:59 +0300 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: <56358475aee7ef97f1a11bdaeee2bf9299741ac6.camel@iki.fi> <082613d2dfd34d949aca3ff5e578cadf2f4c9c3e.camel@iki.fi> Message-ID: pe, 2019-10-11 kello 13:10 -0600, Ryan May kirjoitti: [clip] > I'm in favor of having the "request new review" or making it easier > to query. I just want to clarify that under "Branch Protection" you > can actually set that reviews are cleared on new pushes: Interesting, I hadn't noticed that --- but it says it clears approvals. Does it also clear negative reviews (the case more important for me)? Pauli From rmay31 at gmail.com Fri Oct 11 15:26:15 2019 From: rmay31 at gmail.com (Ryan May) Date: Fri, 11 Oct 2019 13:26:15 -0600 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: <56358475aee7ef97f1a11bdaeee2bf9299741ac6.camel@iki.fi> <082613d2dfd34d949aca3ff5e578cadf2f4c9c3e.camel@iki.fi> Message-ID: On Fri, Oct 11, 2019 at 1:17 PM Pauli Virtanen wrote: > pe, 2019-10-11 kello 13:10 -0600, Ryan May kirjoitti: > [clip] > > I'm in favor of having the "request new review" or making it easier > > to query. I just want to clarify that under "Branch Protection" you > > can actually set that reviews are cleared on new pushes: > > Interesting, I hadn't noticed that --- but it says it clears approvals. > Does it also clear negative reviews (the case more important for me) Interesting, I just read through that before and didn't think about the distinction... I just checked a PR on my repo and, no, it does not clear "request changes" reviews. So there's a GitHub workflow improvement that would be nice: dismiss *all* reviews on push. Ryan -- Ryan May -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Oct 11 18:42:56 2019 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 11 Oct 2019 15:42:56 -0700 Subject: [SciPy-Dev] Github workflow In-Reply-To: References: Message-ID: <314001a01681464c3af33cf14ce5f370a54860ab.camel@sipsolutions.net> On Thu, 2019-10-10 at 08:34 +0200, Andrew Nelson wrote: > devs, > I'm involved in giving some feedback to Github at the moment, as > related to how SciPy users experience Github. Does anyone have any > feedback on: > > - Bugs / Feature requests - are there Github workflows you want to > enable? What's the specific feature you want? > Just a thought, but an "affects me" button on issues could be nice. Knowing which issues actually affect people (and should be prioritized) is sometimes difficult. And I know there were some ideas about sorting by "reactions", etc. but I am not sure that actually works. It also might be a nice way to make yourself nosy/watch an issue at the same time. - Sebastian > This would be an ideal opportunity to tell GIthub of those features > you'd really like to see fixed or implemented. > > A. > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From ralf.gommers at gmail.com Sat Oct 12 11:48:00 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 12 Oct 2019 17:48:00 +0200 Subject: [SciPy-Dev] SciPy user documentation survey - please participate Message-ID: Hi everyone, Maja, our technical writer for Season of Docs for SciPy, created a survey specifically about how users use the SciPy docs and what they would like to see improved. She sent out the link to scipy-dev before and it was shared on Twitter. We've received some really valuable responses, and would love to get more. If you're a SciPy user, this is a very easy way to help the project! https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc18-2PVEabTQg2_6g/viewform?usp=sf_link Apologies for the cross-post. We do a survey about once a decade and Maja is putting in a lot of work creating and analyzing the survey, so it's worth some more exposure and a couple of minutes of your time! Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sat Oct 12 11:53:34 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sat, 12 Oct 2019 17:53:34 +0200 Subject: [SciPy-Dev] Contribution: Estimating the overlap of a ray with volume-cells (voxels) for e.g. tomography In-Reply-To: <10e1d25c0e49db5c306710ebc8ae2007@mpi-hd.mpg.de> References: <10e1d25c0e49db5c306710ebc8ae2007@mpi-hd.mpg.de> Message-ID: Hi Sebastian, On Tue, Oct 8, 2019 at 1:33 PM Sebastian A. Mueller < sebastian-achim.mueller at mpi-hd.mpg.de> wrote: > Dear scipy, > > I wrote a package to estimate the distances a ray overlaps with > volume-cells in three dimensions. This is needed to estimate > system-matrices in tomography. The interface is only one function. It > takes the ray's support and direction vectors, and the bin-edges for the > space partitioning. It returns the overlaps of the ray and the voxels. > The space-partitioning must be rectangular, but it does not need to be > equally spaced. In combination with scipy.sparse, it can efficiently > create system-matrices for iterative, tomographic reconstructions. The > algorithm uses oct-tree-traversal for efficiency. > > On pypi: > https://pypi.org/project/ray-voxel-overlap/0.0.2/ > > On github: > https://github.com/cherenkov-plenoscope/ray_voxel_overlap > > I would like to contribute this to scipy. It addresses multiple fields > in science such as medicine, biology, microscopy, physics, astronomy, > geo-earth, image-processing. > At its core, it is pure linear algebra. My package contains a pure > python-implementation for development and testing, and a (much faster) > C-implementation wrapped in cython. > > Do you think such a tool belongs into scipy? If not, where else should > it go? > It could fit in the scope of SciPy. We don't have much other functionality like it though. My first thought was that it perhaps fits better in scikit-image, which already has a 2-D radon transform: https://scikit-image.org/docs/dev/auto_examples/transform/plot_radon_transform.html. Does that make sense? > This is my first attempt to contribute to scipy. > Welcome! Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Sun Oct 13 17:11:20 2019 From: toddrjen at gmail.com (Todd) Date: Sun, 13 Oct 2019 17:11:20 -0400 Subject: [SciPy-Dev] Github workflow In-Reply-To: <56358475aee7ef97f1a11bdaeee2bf9299741ac6.camel@iki.fi> References: <56358475aee7ef97f1a11bdaeee2bf9299741ac6.camel@iki.fi> Message-ID: On Fri, Oct 11, 2019, 14:52 Pauli Virtanen wrote: > to, 2019-10-10 kello 08:34 +0200, Andrew Nelson kirjoitti: > > devs, > > I'm involved in giving some feedback to Github at the moment, as > > related to how SciPy users experience Github. Does anyone have any > > feedback on: > > > > - Bugs / Feature requests - are there Github workflows you want to > > enable? What's the specific feature you want? > > There are low-hanging fruits for improving the review workflow. > Here's the most important: > > Add a button the PR submitter / maintainers can press, which adds a > status marker for "requesting further review from *someone*". New PRs > would have the status flag set, and the submitter could use this to > mark a PR ready after making requested changes. > > This becomes important when you have a large number of pull requests at > various stages of progress. > > The "Reviewers" box on the upper right on the PR page is something like > this, but does not work in practice: > > (1) PR submitters generally don't use it, usually don't know who they > should choose, and there's no way to set the default value. > (2) Requesting specific developers to review something is not a good > match for open-source projects with many developers. > (3) In Github UI you can only filter for "Awaiting review from you" but > not "Awaiting review from someone". > (4) Most importantly, it's not possible to exclude PRs with the > "Reviewed by someone, and currently not awaiting review" status out. > > Labels cannot be used for this as-is, because the PR submitter cannot > modify them. The comment box is also not useful for this (no way to > search/filter for that in Github UI, and the 100+ daily email volume > from Github already prevents manual tracking). > > Pauli > Along similar lines, I think a "rerun tests" button would be nice. The tests sometimes fail randomly, and if you forget to enable tests on your repository there is no easy way to make them run. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.pietro.lucente at gmail.com Mon Oct 14 05:25:56 2019 From: robert.pietro.lucente at gmail.com (Robert Lucente - Gmail) Date: Mon, 14 Oct 2019 05:25:56 -0400 Subject: [SciPy-Dev] Email rlucente@pipeline.com has stopped receiving notifications Message-ID: <011a01d58271$62a54010$27efc030$@gmail.com> This has happened once before. Perhaps there might be issues w/ other emails? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian-achim.mueller at mpi-hd.mpg.de Mon Oct 14 07:45:13 2019 From: sebastian-achim.mueller at mpi-hd.mpg.de (Sebastian A. Mueller) Date: Mon, 14 Oct 2019 13:45:13 +0200 Subject: [SciPy-Dev] Contribution: Estimating the overlap of a ray with volume-cells (voxels) for e.g. tomography In-Reply-To: References: <10e1d25c0e49db5c306710ebc8ae2007@mpi-hd.mpg.de> Message-ID: <2d0d5a22d2094e633857a5d4ef17c975@mpi-hd.mpg.de> On 2019-10-12 17:53, Ralf Gommers wrote: > Hi Sebastian, > > On Tue, Oct 8, 2019 at 1:33 PM Sebastian A. Mueller > wrote: > >> Dear scipy, >> >> I wrote a package to estimate the distances a ray overlaps with >> volume-cells in three dimensions. This is needed to estimate >> system-matrices in tomography. The interface is only one function. >> It >> takes the ray's support and direction vectors, and the bin-edges for >> the >> space partitioning. It returns the overlaps of the ray and the >> voxels. >> The space-partitioning must be rectangular, but it does not need to >> be >> equally spaced. In combination with scipy.sparse, it can efficiently >> >> create system-matrices for iterative, tomographic reconstructions. >> The >> algorithm uses oct-tree-traversal for efficiency. >> >> On pypi: >> https://pypi.org/project/ray-voxel-overlap/0.0.2/ >> >> On github: >> https://github.com/cherenkov-plenoscope/ray_voxel_overlap >> >> I would like to contribute this to scipy. It addresses multiple >> fields >> in science such as medicine, biology, microscopy, physics, >> astronomy, >> geo-earth, image-processing. >> At its core, it is pure linear algebra. My package contains a pure >> python-implementation for development and testing, and a (much >> faster) >> C-implementation wrapped in cython. >> >> Do you think such a tool belongs into scipy? If not, where else >> should >> it go? > > It could fit in the scope of SciPy. We don't have much other > functionality like it though. Maybe it fits into scipy.spatial, next to functions such as scipy.spatial.distance_matrix. This one has also a compact interface, and also solves a spatial problem. > My first thought was that it perhaps > fits better in scikit-image, which already has a 2-D radon transform: > https://scikit-image.org/docs/dev/auto_examples/transform/plot_radon_transform.html. > Does that make sense? Yes, the 2-D radon-transform in scikit-image is related to this. But the spatial overlap of a 3-D ray with volume-cells in a 3-D grid is more general than just tomography. The compact interface might allow users outside the tomographic-mindset to reuse it for non tomographic purpose. So what about adding estimate_overlap_of_ray_with_voxels() to scipy.spatial? >> This is my first attempt to contribute to scipy. > > Welcome! Thanks! > Cheers, > > Ralf Cheers, Sebastian From ralf.gommers at gmail.com Mon Oct 14 16:02:35 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 14 Oct 2019 22:02:35 +0200 Subject: [SciPy-Dev] changing scipy.org to just be about SciPy the library In-Reply-To: References: Message-ID: I just noticed my reply went only to Joe, so resending: On Wed, Sep 25, 2019 at 10:53 PM Ralf Gommers wrote: > > > On Tue, Sep 24, 2019 at 3:20 PM Joe Harrington wrote: > >> I'm not against it, but the stuff at the top of the page about related >> libraries, docs, getting started, etc. should not disappear, though it >> might move to another site, or be at the head of all the related sites. >> Since NumPy is the base package, and is now better known than SciPy, >> perhaps that material can move there. I think it would be a mistake to >> have one page per package and not to let new users know what the other >> packages are, how they fit together, how to get docs that describe using >> them together, how to connect to the community, etc. >> > > Agreed. I think the plan is to move that to pydata.org (seems preferred > by most), or otherwise a new site with an appropriate name. > > For a long time I've held off on proposing changes to scipy.org until we > had that new site up. As a result though, we've just had a confusing > scipy.org for way too long. So I've changed my mind on making a whole new > site a pre-requisite for updating the current one. SciPy the project > deserves a decent site that's less confusing than today. > > Also, there should be no confusion on a well written and well designed >> site. The text can concisely communicate that "SciPy" refers both to a >> specific package and to the community of packages, other resources, and >> people. >> > > +1 we definitely need to add that. > >> Whether it's "SciPy" or not, we need a term for this ecosystem. It isn't >> the worst thing in the world to change the term, especially as use has >> spread from science to pretty much everything numerical. We've used >> "Python Numerical Core" in a recent funding proposal and community white >> paper. "Numerical Python" is shorter, though the term has some history >> from the early days. I don't (yet!) have a strong opinion on what the term >> should be, as long as it is relatively short, descriptive, and obvious. >> >> It isn't even clear to me that the term itself needs its own web page, >> though of course that's a possibility. I think most people will land in >> our universe either by googling the term "numpy" or something like "python >> math library" (which might take them to the wrong place at first), or going >> to numpy.org, following someone's advice or a link. Wherever users are >> likely to land should have some intro material about the ecosystem at the >> top, and not just dive into the specifics of that module, as numpy.org >> does, currently. >> > The new numpy.org is on the way, I think you'll like it. If you or anyone > else wants a preview, building the `newsite` branch of > https://github.com/numpy/numpy.org with Hugo is straightforward:) > > Cheers, > Ralf > > > >> --jh-- >> On 9/24/19 5:59 AM, Ralf Gommers wrote: >> >> Hi all, >> >> The https://scipy.org/ website has always been about SciPy the ecosystem >> not SciPy the project/library, and it's always resulted in confusion. I >> propose we finally change that. From my review of a recent PR [1] to the >> scipy.org website repo: >> >> """ >> Perhaps the confusing thing here is that https://scipy.org is about >> "SciPy the ecosystem" rather than "SciPy the library". That's what we would >> really like to fix, then this becomes a lot more straightforward. So far I >> had in mind to have a replacement "ecosystem site" before changing >> scipy.org, however we could go without that. There really isn't that >> much content on the current site that we'd have to remove, and some of the >> things that are there now (like SciPy Stack) have become irrelevant. We >> could easily say that `scipy.org` will be about SciPy the library, while >> keeping a page like https://scipy.org/about.html that explains the >> ecosystem context. >> """ >> >> Besides some news about and doc hosting for NumPy releases, there's not >> much content about other projects, and those other projects don't seem to >> care. The "ecosystem" thing really has had its time. "SciPy ecosystem" >> isn't even a preferred term anymore I think, the "scientific" is seen as >> too narrow by a lot of people. PyData or NumPy would be better qualifiers >> (or some new umbrella term). >> >> We may not be able to do all the work straight away, but I want to get >> approval for the decision to make scipy.org about the SciPy project now. >> >> Thoughts? >> >> Cheers, >> Ralf >> >> [1] https://github.com/scipy/scipy.org/pull/309 >> >> _______________________________________________ >> SciPy-Dev mailing listSciPy-Dev at python.orghttps://mail.python.org/mailman/listinfo/scipy-dev >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Oct 15 09:48:55 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 15 Oct 2019 15:48:55 +0200 Subject: [SciPy-Dev] Contribution: Estimating the overlap of a ray with volume-cells (voxels) for e.g. tomography In-Reply-To: <2d0d5a22d2094e633857a5d4ef17c975@mpi-hd.mpg.de> References: <10e1d25c0e49db5c306710ebc8ae2007@mpi-hd.mpg.de> <2d0d5a22d2094e633857a5d4ef17c975@mpi-hd.mpg.de> Message-ID: On Mon, Oct 14, 2019 at 1:45 PM Sebastian A. Mueller < sebastian-achim.mueller at mpi-hd.mpg.de> wrote: > On 2019-10-12 17:53, Ralf Gommers wrote: > > Hi Sebastian, > > > > On Tue, Oct 8, 2019 at 1:33 PM Sebastian A. Mueller > > wrote: > > > >> Dear scipy, > >> > >> I wrote a package to estimate the distances a ray overlaps with > >> volume-cells in three dimensions. This is needed to estimate > >> system-matrices in tomography. The interface is only one function. > >> It > >> takes the ray's support and direction vectors, and the bin-edges for > >> the > >> space partitioning. It returns the overlaps of the ray and the > >> voxels. > >> The space-partitioning must be rectangular, but it does not need to > >> be > >> equally spaced. In combination with scipy.sparse, it can efficiently > >> > >> create system-matrices for iterative, tomographic reconstructions. > >> The > >> algorithm uses oct-tree-traversal for efficiency. > >> > >> On pypi: > >> https://pypi.org/project/ray-voxel-overlap/0.0.2/ > >> > >> On github: > >> https://github.com/cherenkov-plenoscope/ray_voxel_overlap > >> > >> I would like to contribute this to scipy. It addresses multiple > >> fields > >> in science such as medicine, biology, microscopy, physics, > >> astronomy, > >> geo-earth, image-processing. > >> At its core, it is pure linear algebra. My package contains a pure > >> python-implementation for development and testing, and a (much > >> faster) > >> C-implementation wrapped in cython. > >> > >> Do you think such a tool belongs into scipy? If not, where else > >> should > >> it go? > > > > It could fit in the scope of SciPy. We don't have much other > > functionality like it though. > > Maybe it fits into scipy.spatial, next to functions such as > scipy.spatial.distance_matrix. This one has also a compact interface, > and also solves a spatial problem. > > > My first thought was that it perhaps > > fits better in scikit-image, which already has a 2-D radon transform: > > > https://scikit-image.org/docs/dev/auto_examples/transform/plot_radon_transform.html > . > > Does that make sense? > > Yes, the 2-D radon-transform in scikit-image is related to this. But the > spatial overlap of a 3-D ray with volume-cells in a 3-D grid is more > general than just tomography. The compact interface might allow users > outside the tomographic-mindset to reuse it for non tomographic purpose. > So what about adding estimate_overlap_of_ray_with_voxels() to > scipy.spatial? > Looking at it again, I think a ray-voxel-overlap function could fit. Your code is pretty complex though for a single function. It seems to me that a lot of this complexity comes from the octree data structure. That's something we don't have in scipy.spatial, and has much wider applicability that just ray-voxel overlap. It would fit quite nicely next to cKDTree. The best looking octree implementation in Python I can find so quickly is https://github.com/mhogg/pyoctree. How does your implementation compare to that? If the latter is faster or more general, would you be able to build on top of it? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Tue Oct 15 10:21:40 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Tue, 15 Oct 2019 16:21:40 +0200 Subject: [SciPy-Dev] ways of storing notebook tutorials? In-Reply-To: References: Message-ID: On Fri, Oct 11, 2019 at 7:36 PM Matthias Geier wrote: > OK, nbsphinx has already been mentioned, so I don't have to (I'm its > author, BTW). > Cool. Thanks for sharing your expertise here Matthias. > I just wanted to reinforce what has been said about not storing cell > outputs in the repository. > That's a good idea! > > Notebooks can be "cleaned" automatically, see: > > https://nbsphinx.readthedocs.io/en/0.4.3/usage.html#Using-Notebooks-with-Git > > It's also possible to have most of the repository without outputs, but > only one branch containing executed notebooks. I've written about this > recently: https://mg.readthedocs.io/git-jupyter.html > > Even without output, Jupyter notebooks are a bit annoying in version > control, because all strings are quoted in the JSON representation and > there are often plenty of escaped characters. > To avoid this, something like jupytext is great, but it adds quite > some of complexity for users. > Ideally, the native Jupyter notebook format should be changed, but I > don't know if that's likely to happen. FWIW, I've made a suggestion > (https://jupyter-format.readthedocs.io/), but I guess that's a topic > for the far future ... > That does look interesting. Hopefully in the not-too-far-future .... > Regarding what Ralf said: > > > (a) have a "download as notebook" link in the html docs > > That's easy, just look at the nbsphinx docs, e.g. > https://nbsphinx.readthedocs.io/en/0.4.3/code-cells.html. > At the top of the page there is a download link to the notebook on > Github and a link for launching it on Binder. > Arbitrary such links can be created, see > https://nbsphinx.readthedocs.io/en/0.4.3/prolog-and-epilog.html. > This does look like the way to go. Cheers, Ralf > cheers, > Matthias > > On Wed, Oct 9, 2019 at 2:26 PM Andrew Nelson wrote: > > > > The overview for nbsphinx is at > https://nbsphinx.readthedocs.io/en/0.4.3/ > > > > On Wed, 9 Oct 2019, 12:57 Ralf Gommers, wrote: > >> > >> > >> > >> On Tue, Oct 8, 2019 at 8:27 AM Andrew Nelson > wrote: > >>> > >>> Another alternative is to use `jupyter-sphinx` and `nbsphinx` sphinx > extensions. You can include a link to the unexecuted notebook in RST (e.g. > https://github.com/refnx/refnx/blob/master/doc/index.rst), and when the > doc build occurs it'll execute the notebook + output as part of the run. > >> > >> > >> That looks like the closest fit so far. > https://refnx.readthedocs.io/en/latest/getting_started.html looks nice. > The parts your setup doesn't seem to do yet are (a) have a "download as > notebook" link in the html docs, and (b) get rid of notebook output (e.g. > https://raw.githubusercontent.com/refnx/refnx/master/doc/getting_started.ipynb > contains the png images) in the repo. Both of those may not be too hard to > add. > >> > >>> > >>> > >>> On Tue, 8 Oct 2019, 17:06 Matthew Brett, > wrote: > >>>> > >>>> Hi, > >>>> > >>>> On Tue, Oct 8, 2019 at 3:55 PM Ralf Gommers > wrote: > >>>> > > >>>> > > >>>> > > >>>> > On Tue, Oct 8, 2019 at 3:22 PM Eric Larson > wrote: > >>>> >> > >>>> >> I've used sphinx-gallery in a number of projects (am now a > maintainer). Basically you write Python code and it renders to html with > links to downloadable notebooks in the page, automatic linking/cross-refs > to function and class defs, nice index pages, etc. > >>>> > > >>>> > > >>>> > That doesn't really answer my question I think. The starting point > is: notebooks are nice to write a long-form tutorial in, ReST is not. And > sphinx-gallery is nice for a set of independent small to medium sized > examples, but not the best fit for a long tutorial on a whole module (e.g. > http://scipy.github.io/devdocs/tutorial/optimize.html). > >>>> > >>>> Right - I think sphinx-gallery is the right choice for these small > >>>> examples and where you want to develop the tutorial in text. It > >>>> solves the ReST output nicely. > >>>> > >>>> >> > >>>> >> > >>>> >> Eric > >>>> >> > >>>> >> > >>>> >> On Tue, Oct 8, 2019 at 9:20 AM Matthew Brett < > matthew.brett at gmail.com> wrote: > >>>> >>> > >>>> >>> Hi, > >>>> >>> > >>>> >>> I use Jupytext all the time now, and I prefer the RMarkdown input > >>>> >>> format to Jupytext. > >>>> >>> > >>>> >>> Jupytext can convert .ipynb files to .Rmd files, and saves > notebooks > >>>> >>> as both .ipynb and .Rmd. > >>>> >>> > >>>> >>> The workflow is pretty good - after installing Jupytext, you open > your > >>>> >>> .Rmd file in the notebook interface, and run anything you want, > in the > >>>> >>> standard interface. > >>>> >>> > >>>> >>> Then you save, if you need to, which saves to both .ipynb and > .Rmd. > >>>> >>> > >>>> >>> You edit the .Rmd using your nice text editor. > >>>> >>> > >>>> >>> Then you can reload the notebook in Jupyter, with just a browser > >>>> >>> refresh, to see the changes, and to run any extra code you've > added. > >>>> >>> > >>>> >>> I then store the .Rmd files only, in version control, keeping the > >>>> >>> .ipynb as outputs from the final build step. > >>>> > > >>>> > > >>>> > Thanks, that sounds like a reasonable workflow. Seems like we can > roundtrip notebook -> plain text format -> notebook. > >>>> > >>>> Right - exactly - and you can do that interactively, so you can save > >>>> from the Jupyter interface, load in a text editor, save in the text > >>>> editor and just reload the page in Jupyter to get the result. It will > >>>> even keep the outputs as long as you haven't done very substantial > >>>> edits. > >>>> > >>>> > We'll still need to integrate that into the built html docs then, > so the plain text format should be ReST. That doesn't seem supported yet: > https://jupytext.readthedocs.io/en/latest/using-cli.html#testing-the-round-trip-conversion. > Or am I missing something? > >>>> > >>>> No, that is (currently) a problem that I haven't been working on - > >>>> because I'm using Markdown output documents like jupyter-book and > >>>> bookdown. One of my collaborators uses notebooks a lot and I was > >>>> going to suggest to him that we have a separate site for the notebooks > >>>> and the main docs, or at least, we build them separately, and perhaps > >>>> combine them after building to HTML. Maybe jupyter-book would help > >>>> here? > >> > >> > >> If we would start over, something like jupyter-book may be a better > option. I'd prefer to avoid a major overhaul of all tutorials though, > because there's a *lot* of content written in reST files already. Ideally > we keep what we have in reST, and add the ability to write new ones in > notebooks, which could be downloaded standalone but also integrate nicely > with the existing Sphinx html build. > >> > >>>> Chris Holdgraf is the maintainer, and he's very responsive. > >> > >> > >> Thanks for the suggestion. Yes Chris is great, may ask him for advice. > >> > >> Cheers, > >> Ralf > >> > >> > >> _______________________________________________ > >> SciPy-Dev mailing list > >> SciPy-Dev at python.org > >> https://mail.python.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matteoravasi at gmail.com Wed Oct 16 02:50:50 2019 From: matteoravasi at gmail.com (Matteo Ravasi) Date: Wed, 16 Oct 2019 08:50:50 +0200 Subject: [SciPy-Dev] Contribution: Estimating the overlap of a ray with volume-cells (voxels) for e.g. tomography In-Reply-To: References: Message-ID: <43CF2825-58ED-4F32-9D53-17D50BDEA7C5@gmail.com> Hi Sebastian, I also looked into your package, ans I agree it would be a very nice addition to scipy. Just one question: in tomography it is common to know the start and end points of a ray and they may lie somewhere inside the grid extent. In your library I seem to be able to only have rays that cross straight through the grid as they can only be parametrized by only their support and angle. While this parametrization can be inferred from start-end point, being able to have rays within (but not all the way through the grid) would definitely make your routine more general and applicable to various problems - then more suited to scipy. Am I missing anything? Or do you think this feature could be easily added without changing the current code base too much? Thank you! iMR > On 15 Oct 2019, at 15:51, Ralf Gommers wrote: > > ? > > >> On Mon, Oct 14, 2019 at 1:45 PM Sebastian A. Mueller wrote: >> On 2019-10-12 17:53, Ralf Gommers wrote: >> > Hi Sebastian, >> > >> > On Tue, Oct 8, 2019 at 1:33 PM Sebastian A. Mueller >> > wrote: >> > >> >> Dear scipy, >> >> >> >> I wrote a package to estimate the distances a ray overlaps with >> >> volume-cells in three dimensions. This is needed to estimate >> >> system-matrices in tomography. The interface is only one function. >> >> It >> >> takes the ray's support and direction vectors, and the bin-edges for >> >> the >> >> space partitioning. It returns the overlaps of the ray and the >> >> voxels. >> >> The space-partitioning must be rectangular, but it does not need to >> >> be >> >> equally spaced. In combination with scipy.sparse, it can efficiently >> >> >> >> create system-matrices for iterative, tomographic reconstructions. >> >> The >> >> algorithm uses oct-tree-traversal for efficiency. >> >> >> >> On pypi: >> >> https://pypi.org/project/ray-voxel-overlap/0.0.2/ >> >> >> >> On github: >> >> https://github.com/cherenkov-plenoscope/ray_voxel_overlap >> >> >> >> I would like to contribute this to scipy. It addresses multiple >> >> fields >> >> in science such as medicine, biology, microscopy, physics, >> >> astronomy, >> >> geo-earth, image-processing. >> >> At its core, it is pure linear algebra. My package contains a pure >> >> python-implementation for development and testing, and a (much >> >> faster) >> >> C-implementation wrapped in cython. >> >> >> >> Do you think such a tool belongs into scipy? If not, where else >> >> should >> >> it go? >> > >> > It could fit in the scope of SciPy. We don't have much other >> > functionality like it though. >> >> Maybe it fits into scipy.spatial, next to functions such as >> scipy.spatial.distance_matrix. This one has also a compact interface, >> and also solves a spatial problem. >> >> > My first thought was that it perhaps >> > fits better in scikit-image, which already has a 2-D radon transform: >> > https://scikit-image.org/docs/dev/auto_examples/transform/plot_radon_transform.html. >> > Does that make sense? >> >> Yes, the 2-D radon-transform in scikit-image is related to this. But the >> spatial overlap of a 3-D ray with volume-cells in a 3-D grid is more >> general than just tomography. The compact interface might allow users >> outside the tomographic-mindset to reuse it for non tomographic purpose. >> So what about adding estimate_overlap_of_ray_with_voxels() to >> scipy.spatial? > > Looking at it again, I think a ray-voxel-overlap function could fit. Your code is pretty complex though for a single function. It seems to me that a lot of this complexity comes from the octree data structure. That's something we don't have in scipy.spatial, and has much wider applicability that just ray-voxel overlap. It would fit quite nicely next to cKDTree. > > The best looking octree implementation in Python I can find so quickly is https://github.com/mhogg/pyoctree. How does your implementation compare to that? If the latter is faster or more general, would you be able to build on top of it? > > Cheers, > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian-achim.mueller at mpi-hd.mpg.de Wed Oct 16 05:38:14 2019 From: sebastian-achim.mueller at mpi-hd.mpg.de (Sebastian A. Mueller) Date: Wed, 16 Oct 2019 11:38:14 +0200 Subject: [SciPy-Dev] Contribution: Estimating the overlap of a ray with volume-cells (voxels) for e.g. tomography In-Reply-To: References: <10e1d25c0e49db5c306710ebc8ae2007@mpi-hd.mpg.de> <2d0d5a22d2094e633857a5d4ef17c975@mpi-hd.mpg.de> Message-ID: <4ad9be1cf4e1ffa20ced626bc7df28d7@mpi-hd.mpg.de> On 2019-10-15 15:48, Ralf Gommers wrote: > On Mon, Oct 14, 2019 at 1:45 PM Sebastian A. Mueller > wrote: > >> On 2019-10-12 17:53, Ralf Gommers wrote: >>> Hi Sebastian, >>> >>> On Tue, Oct 8, 2019 at 1:33 PM Sebastian A. Mueller >>> wrote: >>> >>>> Dear scipy, >>>> >>>> I wrote a package to estimate the distances a ray overlaps with >>>> volume-cells in three dimensions. This is needed to estimate >>>> system-matrices in tomography. The interface is only one >> function. >>>> It >>>> takes the ray's support and direction vectors, and the bin-edges >> for >>>> the >>>> space partitioning. It returns the overlaps of the ray and the >>>> voxels. >>>> The space-partitioning must be rectangular, but it does not need >> to >>>> be >>>> equally spaced. In combination with scipy.sparse, it can >> efficiently >>>> >>>> create system-matrices for iterative, tomographic >> reconstructions. >>>> The >>>> algorithm uses oct-tree-traversal for efficiency. >>>> >>>> On pypi: >>>> https://pypi.org/project/ray-voxel-overlap/0.0.2/ >>>> >>>> On github: >>>> https://github.com/cherenkov-plenoscope/ray_voxel_overlap >>>> >>>> I would like to contribute this to scipy. It addresses multiple >>>> fields >>>> in science such as medicine, biology, microscopy, physics, >>>> astronomy, >>>> geo-earth, image-processing. >>>> At its core, it is pure linear algebra. My package contains a >> pure >>>> python-implementation for development and testing, and a (much >>>> faster) >>>> C-implementation wrapped in cython. >>>> >>>> Do you think such a tool belongs into scipy? If not, where else >>>> should >>>> it go? >>> >>> It could fit in the scope of SciPy. We don't have much other >>> functionality like it though. >> >> Maybe it fits into scipy.spatial, next to functions such as >> scipy.spatial.distance_matrix. This one has also a compact >> interface, >> and also solves a spatial problem. >> >>> My first thought was that it perhaps >>> fits better in scikit-image, which already has a 2-D radon >> transform: >>> >> > https://scikit-image.org/docs/dev/auto_examples/transform/plot_radon_transform.html. >>> Does that make sense? >> >> Yes, the 2-D radon-transform in scikit-image is related to this. But >> the >> spatial overlap of a 3-D ray with volume-cells in a 3-D grid is more >> >> general than just tomography. The compact interface might allow >> users >> outside the tomographic-mindset to reuse it for non tomographic >> purpose. >> So what about adding estimate_overlap_of_ray_with_voxels() to >> scipy.spatial? > > Looking at it again, I think a ray-voxel-overlap function could fit. > Your code is pretty complex though for a single function. The function: 245 lines, C, in _c_overlap_implementation.c 203 lines, cython, in _cython_overlap_implementation.pyx Tests: 509 lines, python Approx. 957 lines for the C and cython implementation. The function written in pure python: 207 lines, python, in _py_overlap.py Approx. 716 lines for the pure python implementation. > It seems to > me that a lot of this complexity comes from the octree data structure. > That's something we don't have in scipy.spatial, and has much wider > applicability that just ray-voxel overlap. Yes. Space-partitioning, such as octrees, can be used for more. They are good to avoid unnecessary computations in spatial structures. > It would fit quite nicely > next to cKDTree. Right. I often use spatial.cKDTree. It also uses a tree-structure to organize point-data. Here the tree (octo or binary does not matter much) is used as a book-keeping-tool to organize other information, such as point-data. > The best looking octree implementation in Python I can find so quickly > is https://github.com/mhogg/pyoctree. Also in pyoctree the octree is used to organize other information. In this case triangle-meshes. > How does your implementation > compare to that? The big difference is: Ray_voxel_overpal does not use an octree to organize other information. It has no data-structure representing a node in a tree. And therefore, there is no tree at all. But for traversing the volume-cells with a ray, it acts similar to octree-traversal. This is why I mentioned octrees. Octrees are often used synonym for the binary-search like traversing of spatial structures with rays. I am sorry when this buzz-word triggered more associations. In ray_voxel_overlap, the space-partitioning is represented by the bin-edges along each dimension. Assume we have 1e2 bin-egdes in each x, y, and z. In ray_voxel_overlap, this is represented using 3e2 floats. But creating all these nodes in a tree needs us to allocate 1e6 nodes just for the deepest branches in the tree. My ray_voxel_overlap on the other hand only does a binary-search along the bin-edges in each dimension to traverse the volume-cells. My ray_voxel_overlap does not implement an octree, it only implements the traversing of rectangular volume-cells with a ray based on binary-search. > If the latter is faster or more general, Pyoctree is focusing on the book-keeping of triangle-meshes. In computer-graphics this is important, in e.g. tomography it is not. My implementation supports non equally distant spacing of bin-edges which is important for some tomography but probably not for computer-graphics. So I think both are not general. cKDTree is more general, as it has less features. Unlike pyoctree and ray_voxel_overlap, cKDTree does not have the concept of a ray traversing its point-data. > would you be able to build on top of it? When dropping non equally distant spacing of bin-edges, one could build on top of Pyoctree. (Splits distances always in half: https://github.com/mhogg/pyoctree/blob/c457dd00fc4412ae096c4502b2863718f39d9cd1/pyoctree/cOctree.cpp#L455) Further, pyoctree would need to be added the calculation of the actual spatial overlap of a ray and a volume-cell: https://github.com/cherenkov-plenoscope/ray_voxel_overlap/blob/250ad53c6f05bd0aeb67a374383dc82d697a66d6/ray_voxel_overlap/_py_overlap.py#L141 vs. https://github.com/mhogg/pyoctree/blob/c457dd00fc4412ae096c4502b2863718f39d9cd1/pyoctree/cOctree.cpp#L286 If pyoctree's interface forces us to build the octree before making queries, I expect a noticeable overhead in memory-usage and run-time when allocating approx. 1e6 nodes. > Cheers, > > Ralf Cheers Sebastian > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev From sebastian-achim.mueller at mpi-hd.mpg.de Wed Oct 16 05:49:10 2019 From: sebastian-achim.mueller at mpi-hd.mpg.de (Sebastian A. Mueller) Date: Wed, 16 Oct 2019 11:49:10 +0200 Subject: [SciPy-Dev] Contribution: Estimating the overlap of a ray with volume-cells (voxels) for e.g. tomography In-Reply-To: <43CF2825-58ED-4F32-9D53-17D50BDEA7C5@gmail.com> References: <43CF2825-58ED-4F32-9D53-17D50BDEA7C5@gmail.com> Message-ID: On 2019-10-16 08:50, Matteo Ravasi wrote: > Hi Sebastian, > I also looked into your package, ans I agree it would be a very nice > addition to scipy. > > Just one question: in tomography it is common to know the start and > end points of a ray and they may lie somewhere inside the grid extent. > In your library I seem to be able to only have rays that cross > straight through the grid as they can only be parametrized by only > their support and angle. While this parametrization can be inferred > from start-end point, being able to have rays within (but not all the > way through the grid) would definitely make your routine more general > and applicable to various problems - then more suited to scipy. > > Am I missing anything? Or do you think this feature could be easily > added without changing the current code base too much? > > Thank you! > > iMR > Hi Matteo, you got it right. There is no start-point, and end-point in ray_voxel_overlap. I would have to write a unit-test to figure out if one has control over the start-point using the support-vector, but I did not implement the concept of an end-point. So this is a feature-request. I will put in on git-hub. Cheers, Sebastian From charlesr.harris at gmail.com Thu Oct 17 11:36:45 2019 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 17 Oct 2019 09:36:45 -0600 Subject: [SciPy-Dev] NumPy 1.17 3 released Message-ID: Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.17.3 has been released. This is a bugfix release. The Python versions supported in this release are 3.5-3.8. Downstream developers should use Cython >= 0.29.13 for Python 3.8 support and OpenBLAS >= 3.7 to avoid wrong results on the Skylake architecture. The NumP Wheels for this release can be downloaded from PyPI , source archives and release notes are available from Github . *Highlights* - Wheels for Python 3.8 - Boolean matmul fixed to use booleans instead of integers. *Compatibility notes* - The seldom used PyArray_DescrCheck macro has been changed/fixed. *Contributors* A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Allan Haldane - Charles Harris - Kevin Sheppard - Matti Picus - Ralf Gommers - Sebastian Berg - Warren Weckesser *Pull requests merged* A total of 12 pull requests were merged for this release. - gh-14456: MAINT: clean up pocketfft modules inside numpy.fft namespace. - gh-14463: BUG: random.hypergeometic assumes npy_long is npy_int64, hung... - gh-14502: BUG: random: Revert gh-14458 and refix gh-14557. - gh-14504: BUG: add a specialized loop for boolean matmul. - gh-14506: MAINT: Update pytest version for Python 3.8 - gh-14512: DOC: random: fix doc linking, was referencing private submodules. - gh-14513: BUG,MAINT: Some fixes and minor cleanup based on clang analysis - gh-14515: BUG: Fix randint when range is 2**32 - gh-14519: MAINT: remove the entropy c-extension module - gh-14563: DOC: remove note about Pocketfft license file (non-existing here). - gh-14578: BUG: random: Create a legacy implementation of random.binomial. - gh-14687: BUG: properly define PyArray_DescrCheck Cheers, Charles Harris -------------- next part -------------- An HTML attachment was scrubbed... URL: From grlee77 at gmail.com Fri Oct 18 15:29:05 2019 From: grlee77 at gmail.com (Gregory Lee) Date: Fri, 18 Oct 2019 15:29:05 -0400 Subject: [SciPy-Dev] ANN: PyWavelets 1.1.0 Message-ID: Hello all, We are very pleased to announce the release of PyWavelets 1.1. PyWavelets is Python package for wavelet analysis. This release includes enhanced functionality for both the stationary wavelet transforms (``swt``, ``swt2``, ``swtn``) as well as the continuous wavelet transform (``cwt``). In addition, there are a handful of bug fixes. This release has dropped Python 2.7 support and now requires Python >= 3.5. Wheels for Python 3.5-3.8 are currently being uploaded to PyPI. In addition to these changes to the software itself, a paper describing PyWavelets was recently published in The Journal of Open Source Software: https://joss.theoj.org/papers/10.21105/joss.01237 For full details see the release notes at: https://pywavelets.readthedocs.io/en/latest/release.1.1.0.html Cheers, Greg -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Fri Oct 18 18:24:09 2019 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Fri, 18 Oct 2019 16:24:09 -0600 Subject: [SciPy-Dev] Proposed SciPy 1.4.0 Release Schedule Message-ID: Hi, SciPy 1.3.0 was released May 17 (5 months ago), and I think we'd like to keep a roughly biannual release cadence. I'd like to propose the following schedule for 1.4.0: - November 14: branch 1.4.x - November 17: rc1 - December 1: rc2 (if needed) - December 10: final release The arrival of Python 3.8 and simultaneous responsibility to get at least one more 1.2.x LTS Python 2.7 release out the door means things will probably be busy on the release front until 2020. As always, it is a good idea to start tagging things that should be in 1.4.0 & please do help with reviewing PRs/issues that are tagged--current counts are: - PRs: 40 open with 1.4.0 milestone - issues: 17 open with 1.4.0 milestone While helping with that, also great if the release notes wiki is updated for appropriate changes: https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.4.0 Curating those final notes can be painful if the wiki isn't updated. Thoughts/objections for the schedule? Best wishes, Tyler -------------- next part -------------- An HTML attachment was scrubbed... URL: From prabhu at aero.iitb.ac.in Sun Oct 20 07:13:12 2019 From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran) Date: Sun, 20 Oct 2019 16:43:12 +0530 Subject: [SciPy-Dev] [ANN] SciPy India 2019 Message-ID: Hello, [Apologies for the cross-posting!] We are pleased to announce the SciPy India conference 2019. SciPy India is an annual conference on using Python for research and education. The conference is currently in its eleventh year and will be held at IIT Bombay on 29 & 30 November, 2019. The registration and call for papers are open. Please visit http://scipy.in to register and submit your proposals. Please spread the word! Call for Papers ============= We look forward to your submissions on the use of Python for scientific computing and education. This includes development of tools for pedagogy, exploration, modeling, and analysis. We welcome contributions from academia as well as industry. For details on the paper submission please see here: ??? http://scipy.in/ Important Dates ================ ? - Call for proposals end: 25th October 2019 ? - List of accepted proposals will be announced: 3rd November. We look forward to seeing you at SciPy India. Regards, Prabhu Ramachandran (For the SciPy organizing team) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian-achim.mueller at mpi-hd.mpg.de Mon Oct 21 05:25:38 2019 From: sebastian-achim.mueller at mpi-hd.mpg.de (Sebastian A. Mueller) Date: Mon, 21 Oct 2019 11:25:38 +0200 Subject: [SciPy-Dev] Contribution: Estimating the overlap of a ray with volume-cells (voxels) for e.g. tomography In-Reply-To: References: <10e1d25c0e49db5c306710ebc8ae2007@mpi-hd.mpg.de> <2d0d5a22d2094e633857a5d4ef17c975@mpi-hd.mpg.de> <4ad9be1cf4e1ffa20ced626bc7df28d7@mpi-hd.mpg.de> Message-ID: <113ee430f31c8d6c8a8276870f2052de@mpi-hd.mpg.de> On 2019-10-18 15:19, Ralf Gommers wrote: > Thanks for the explanation, makes sense now. Okay, let's forget about > my suggestion then. > > So we're back to: shall we include this in scipy.spatial more or less > in its current form? There does seem to be interest and it does fit. > Would be nice though if we can get someone else to step up to review. Review is good. I would be happy to receive comments. > Matteo had a good question that's still to be answered. Matteo's question is valid. I opened a [feature-request](https://github.com/cherenkov-plenoscope/ray_voxel_overlap/issues/4). Currently the rays have no limits. Adding control over start- and end-points sounds reasonable to address more applications. The effort to implement this feature might be worth the gain in possible users. > Perhaps if he > could review it to make sure it looks good and would be useful for his > use case, that would get it much closer to ready for inclusion? Sure. A review by a potential user is good. > > Cheers, > > Ralf Cheers, Sebastian From pierre.haessig at crans.org Mon Oct 21 05:42:47 2019 From: pierre.haessig at crans.org (Pierre Haessig) Date: Mon, 21 Oct 2019 11:42:47 +0200 Subject: [SciPy-Dev] ANN: PyWavelets 1.1.0 In-Reply-To: References: Message-ID: <4cc9905e-c74c-33e3-0cab-b35c474d0d28@crans.org> Le 18/10/2019 ? 21:29, Gregory Lee a ?crit?: > We are very pleased to announce the release of PyWavelets 1.1. > PyWavelets is Python package for wavelet analysis. Nice! I've only used PyWavelets 2-3 times over the recent years, and I've always found it simple and well functioning. Thanks for maintaining it! Best, Pierre From ralf.gommers at gmail.com Mon Oct 21 06:43:00 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 21 Oct 2019 12:43:00 +0200 Subject: [SciPy-Dev] not running CI on merge commits Message-ID: Hi all, I propose that we stop running CI on merges. There's a number of reasons for this, the current behavior: - significantly increases our CI resource usage, for little gain - delays CI runs on PRs regularly (many of us tend to do PR review and updates in batches) - causes Azure to send out spammy emails each time I merge a PR, I really don't need to know that a merge commit passed In the very rare case that CI on a PR is green and then fails after merge with a valid failure rather than a flake, we will find out soon enough anyway. So I really can't think of a good reason to keep triggering all CI on merging with master. Thoughts? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Mon Oct 21 07:02:05 2019 From: andyfaff at gmail.com (Andrew Nelson) Date: Mon, 21 Oct 2019 22:02:05 +1100 Subject: [SciPy-Dev] not running CI on merge commits In-Reply-To: References: Message-ID: I don't get azure emails from ci actions, so it's not everyone that gets them. We could disable one CI service for merges, leave the other active? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Oct 21 07:08:42 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 21 Oct 2019 13:08:42 +0200 Subject: [SciPy-Dev] not running CI on merge commits In-Reply-To: References: Message-ID: On Mon, Oct 21, 2019 at 1:02 PM Andrew Nelson wrote: > I don't get azure emails from ci actions, so it's not everyone that gets > them. > > We could disable one CI service for merges, leave the other active? > I honestly think we're better off disabling them all, but at least I'd like to disable TravisCI, since it can be delays for hours. Azure is a lot faster. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Mon Oct 21 11:37:57 2019 From: toddrjen at gmail.com (Todd) Date: Mon, 21 Oct 2019 11:37:57 -0400 Subject: [SciPy-Dev] Proposed SciPy 1.4.0 Release Schedule In-Reply-To: References: Message-ID: Would it be possible to get PR 10869 [1] tagged for 1.4.0? It implements the overlap-add convolution algorithm, which was originally requested 10 years ago. [1] https://github.com/scipy/scipy/pull/10869 On Fri, Oct 18, 2019 at 6:24 PM Tyler Reddy wrote: > Hi, > > SciPy 1.3.0 was released May 17 (5 months ago), and I think we'd like to > keep a roughly biannual release cadence. > > I'd like to propose the following schedule for 1.4.0: > - November 14: branch 1.4.x > - November 17: rc1 > - December 1: rc2 (if needed) > - December 10: final release > > The arrival of Python 3.8 and simultaneous responsibility to get at least > one more 1.2.x LTS Python 2.7 release out the door means things will > probably be busy on the release front until 2020. > > As always, it is a good idea to start tagging things that should be in > 1.4.0 & please do help with reviewing PRs/issues that are tagged--current > counts are: > > - PRs: 40 open with 1.4.0 milestone > - issues: 17 open with 1.4.0 milestone > > While helping with that, also great if the release notes wiki is updated > for appropriate changes: > https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.4.0 > > Curating those final notes can be painful if the wiki isn't updated. > > Thoughts/objections for the schedule? > > Best wishes, > Tyler > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Oct 21 11:46:14 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 21 Oct 2019 17:46:14 +0200 Subject: [SciPy-Dev] Proposed SciPy 1.4.0 Release Schedule In-Reply-To: References: Message-ID: On Sat, Oct 19, 2019 at 12:24 AM Tyler Reddy wrote: > Hi, > > SciPy 1.3.0 was released May 17 (5 months ago), and I think we'd like to > keep a roughly biannual release cadence. > > I'd like to propose the following schedule for 1.4.0: > - November 14: branch 1.4.x > - November 17: rc1 > - December 1: rc2 (if needed) > - December 10: final release > Sounds good to me. > > The arrival of Python 3.8 and simultaneous responsibility to get at least > one more 1.2.x LTS Python 2.7 release out the door means things will > probably be busy on the release front until 2020. > True. I think supporting 3.8 is fairly high-prio, there's a lot of demand for it. Doing a 1.3.2 release that supports it would make sense I guess. There don't seem to be any new commits on the 1.2.x branch nor urgent Python 2.7 fixes that need to go out. So do we need a 1.2.3 release? Cheers, Ralf > As always, it is a good idea to start tagging things that should be in > 1.4.0 & please do help with reviewing PRs/issues that are tagged--current > counts are: > > - PRs: 40 open with 1.4.0 milestone > - issues: 17 open with 1.4.0 milestone > > While helping with that, also great if the release notes wiki is updated > for appropriate changes: > https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.4.0 > > Curating those final notes can be painful if the wiki isn't updated. > > Thoughts/objections for the schedule? > > Best wishes, > Tyler > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juanlu001 at gmail.com Mon Oct 21 12:59:30 2019 From: juanlu001 at gmail.com (Juan Luis Cano) Date: Mon, 21 Oct 2019 18:59:30 +0200 Subject: [SciPy-Dev] Proposed SciPy 1.4.0 Release Schedule In-Reply-To: References: Message-ID: On Mon, Oct 21, 2019 at 5:46 PM Ralf Gommers wrote: > > There don't seem to be any new commits on the 1.2.x branch nor urgent > Python 2.7 fixes that need to go out. So do we need a 1.2.3 release? > It would be great to have SciPy wheels for Python 3.8 in all platforms early without having to wait for 1.4 (so packages depending on SciPy have an easier installation process). Regards, > > Cheers, > Ralf > > > >> As always, it is a good idea to start tagging things that should be in >> 1.4.0 & please do help with reviewing PRs/issues that are tagged--current >> counts are: >> >> - PRs: 40 open with 1.4.0 milestone >> - issues: 17 open with 1.4.0 milestone >> >> While helping with that, also great if the release notes wiki is updated >> for appropriate changes: >> https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.4.0 >> >> Curating those final notes can be painful if the wiki isn't updated. >> >> Thoughts/objections for the schedule? >> >> Best wishes, >> Tyler >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Juan Luis Cano -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillip.m.feldman at gmail.com Mon Oct 21 13:14:55 2019 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Mon, 21 Oct 2019 10:14:55 -0700 Subject: [SciPy-Dev] Proposed SciPy 1.4.0 Release Schedule In-Reply-To: References: Message-ID: It would be really nice to have support for overlap-add convolution. On Mon, Oct 21, 2019 at 8:38 AM Todd wrote: > Would it be possible to get PR 10869 [1] tagged for 1.4.0? It implements > the overlap-add convolution algorithm, which was originally requested 10 > years ago. > > [1] https://github.com/scipy/scipy/pull/10869 > > On Fri, Oct 18, 2019 at 6:24 PM Tyler Reddy > wrote: > >> Hi, >> >> SciPy 1.3.0 was released May 17 (5 months ago), and I think we'd like to >> keep a roughly biannual release cadence. >> >> I'd like to propose the following schedule for 1.4.0: >> - November 14: branch 1.4.x >> - November 17: rc1 >> - December 1: rc2 (if needed) >> - December 10: final release >> >> The arrival of Python 3.8 and simultaneous responsibility to get at least >> one more 1.2.x LTS Python 2.7 release out the door means things will >> probably be busy on the release front until 2020. >> >> As always, it is a good idea to start tagging things that should be in >> 1.4.0 & please do help with reviewing PRs/issues that are tagged--current >> counts are: >> >> - PRs: 40 open with 1.4.0 milestone >> - issues: 17 open with 1.4.0 milestone >> >> While helping with that, also great if the release notes wiki is updated >> for appropriate changes: >> https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.4.0 >> >> Curating those final notes can be painful if the wiki isn't updated. >> >> Thoughts/objections for the schedule? >> >> Best wishes, >> Tyler >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilhanpolat at gmail.com Mon Oct 21 14:51:23 2019 From: ilhanpolat at gmail.com (Ilhan Polat) Date: Mon, 21 Oct 2019 20:51:23 +0200 Subject: [SciPy-Dev] not running CI on merge commits In-Reply-To: References: Message-ID: +1 for disabling Travis (for me all of them would be better). We have to bisect back to find the offending one when something fails anyways. Also new PRs come almost every 12 hours so we are only a few more hours late at the worst. On Mon, Oct 21, 2019 at 1:09 PM Ralf Gommers wrote: > > > On Mon, Oct 21, 2019 at 1:02 PM Andrew Nelson wrote: > >> I don't get azure emails from ci actions, so it's not everyone that gets >> them. >> >> We could disable one CI service for merges, leave the other active? >> > > I honestly think we're better off disabling them all, but at least I'd > like to disable TravisCI, since it can be delays for hours. Azure is a lot > faster. > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.je.reddy at gmail.com Mon Oct 21 15:08:55 2019 From: tyler.je.reddy at gmail.com (Tyler Reddy) Date: Mon, 21 Oct 2019 13:08:55 -0600 Subject: [SciPy-Dev] Proposed SciPy 1.4.0 Release Schedule In-Reply-To: References: Message-ID: Alright, I'll move 1.3.2 with Python 3.8 support up on the priority list to start with. On Mon, 21 Oct 2019 at 11:15, Phillip Feldman wrote: > It would be really nice to have support for overlap-add convolution. > > On Mon, Oct 21, 2019 at 8:38 AM Todd wrote: > >> Would it be possible to get PR 10869 [1] tagged for 1.4.0? It implements >> the overlap-add convolution algorithm, which was originally requested 10 >> years ago. >> >> [1] https://github.com/scipy/scipy/pull/10869 >> >> On Fri, Oct 18, 2019 at 6:24 PM Tyler Reddy >> wrote: >> >>> Hi, >>> >>> SciPy 1.3.0 was released May 17 (5 months ago), and I think we'd like to >>> keep a roughly biannual release cadence. >>> >>> I'd like to propose the following schedule for 1.4.0: >>> - November 14: branch 1.4.x >>> - November 17: rc1 >>> - December 1: rc2 (if needed) >>> - December 10: final release >>> >>> The arrival of Python 3.8 and simultaneous responsibility to get at >>> least one more 1.2.x LTS Python 2.7 release out the door means things will >>> probably be busy on the release front until 2020. >>> >>> As always, it is a good idea to start tagging things that should be in >>> 1.4.0 & please do help with reviewing PRs/issues that are tagged--current >>> counts are: >>> >>> - PRs: 40 open with 1.4.0 milestone >>> - issues: 17 open with 1.4.0 milestone >>> >>> While helping with that, also great if the release notes wiki is updated >>> for appropriate changes: >>> https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.4.0 >>> >>> Curating those final notes can be painful if the wiki isn't updated. >>> >>> Thoughts/objections for the schedule? >>> >>> Best wishes, >>> Tyler >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From haberland at ucla.edu Mon Oct 21 15:38:58 2019 From: haberland at ucla.edu (Matt Haberland) Date: Mon, 21 Oct 2019 12:38:58 -0700 Subject: [SciPy-Dev] Proposed SciPy 1.4.0 Release Schedule In-Reply-To: References: Message-ID: Re: 1.2.3 release, I did get a request in gh-10915 for gh-10498 to be backported to 1.2. On Mon, Oct 21, 2019 at 8:47 AM Ralf Gommers wrote: > > > On Sat, Oct 19, 2019 at 12:24 AM Tyler Reddy > wrote: > >> Hi, >> >> SciPy 1.3.0 was released May 17 (5 months ago), and I think we'd like to >> keep a roughly biannual release cadence. >> >> I'd like to propose the following schedule for 1.4.0: >> - November 14: branch 1.4.x >> - November 17: rc1 >> - December 1: rc2 (if needed) >> - December 10: final release >> > > Sounds good to me. > >> >> The arrival of Python 3.8 and simultaneous responsibility to get at least >> one more 1.2.x LTS Python 2.7 release out the door means things will >> probably be busy on the release front until 2020. >> > > True. I think supporting 3.8 is fairly high-prio, there's a lot of demand > for it. Doing a 1.3.2 release that supports it would make sense I guess. > > There don't seem to be any new commits on the 1.2.x branch nor urgent > Python 2.7 fixes that need to go out. So do we need a 1.2.3 release? > > Cheers, > Ralf > > > >> As always, it is a good idea to start tagging things that should be in >> 1.4.0 & please do help with reviewing PRs/issues that are tagged--current >> counts are: >> >> - PRs: 40 open with 1.4.0 milestone >> - issues: 17 open with 1.4.0 milestone >> >> While helping with that, also great if the release notes wiki is updated >> for appropriate changes: >> https://github.com/scipy/scipy/wiki/Release-note-entries-for-SciPy-1.4.0 >> >> Curating those final notes can be painful if the wiki isn't updated. >> >> Thoughts/objections for the schedule? >> >> Best wishes, >> Tyler >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -- Matt Haberland Assistant Adjunct Professor in the Program in Computing Department of Mathematics 6617A Math Sciences Building, UCLA -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.alex.moore at gmail.com Sat Oct 26 23:09:38 2019 From: j.alex.moore at gmail.com (Jay Moore) Date: Sat, 26 Oct 2019 23:09:38 -0400 Subject: [SciPy-Dev] New to list Message-ID: Hi all, I am new to this mailing list and I would like to discuss a new feature I would like to add to scipy stats. I have never contributed code to open source projects before so any guidance would be greatly appreciated. I would like to add confidence intervals for fisher's exact testing. I've been working with the code for a couple of months now and would like to make this available to everyone using this package. Best, Jay Moore -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Oct 27 04:09:37 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 27 Oct 2019 09:09:37 +0100 Subject: [SciPy-Dev] New to list In-Reply-To: References: Message-ID: On Sun, Oct 27, 2019 at 4:09 AM Jay Moore wrote: > Hi all, > > I am new to this mailing list and I would like to discuss a new feature I > would like to add to scipy stats. I have never contributed code to open > source projects before so any guidance would be greatly appreciated. > Welcome Jay! You came to the right place, we use the mailing list to propose and discuss new features. I would like to add confidence intervals for fisher's exact testing. I've > been working with the code for a couple of months now and would like to > make this available to everyone using this package. > Sounds like a good idea to me. Do you have a proposed implementation already, or some ideas or questions about how to add this feature to the API? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.alex.moore at gmail.com Sun Oct 27 17:23:36 2019 From: j.alex.moore at gmail.com (Jay Moore) Date: Sun, 27 Oct 2019 14:23:36 -0700 Subject: [SciPy-Dev] Fishers exact confidence interval In-Reply-To: References: Message-ID: Hi Ralf, I have a proposed implementation. Should I collect unit tests and documentation and make a pull request? Best, Jay On Sun, Oct 27, 2019 at 09:00 wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request at python.org > > You can reach the person managing the list at > scipy-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. New to list (Jay Moore) > 2. Re: New to list (Ralf Gommers) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 26 Oct 2019 23:09:38 -0400 > From: Jay Moore > To: scipy-dev at python.org > Subject: [SciPy-Dev] New to list > Message-ID: > < > CAOz3X_5TnZOOCOsVxGqb4V+zGO+ndfL55crJf0er_pK6OsFD_Q at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi all, > > I am new to this mailing list and I would like to discuss a new feature I > would like to add to scipy stats. I have never contributed code to open > source projects before so any guidance would be greatly appreciated. > > I would like to add confidence intervals for fisher's exact testing. I've > been working with the code for a couple of months now and would like to > make this available to everyone using this package. > > Best, > Jay Moore > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/scipy-dev/attachments/20191026/2f486e35/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Sun, 27 Oct 2019 09:09:37 +0100 > From: Ralf Gommers > To: SciPy Developers List > Subject: Re: [SciPy-Dev] New to list > Message-ID: > < > CABL7CQiU6TxaCo0N9C2Myq0eKSaiac_HeTkCcrH_EJOhOPf5zg at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Sun, Oct 27, 2019 at 4:09 AM Jay Moore wrote: > > > Hi all, > > > > I am new to this mailing list and I would like to discuss a new feature I > > would like to add to scipy stats. I have never contributed code to open > > source projects before so any guidance would be greatly appreciated. > > > > Welcome Jay! You came to the right place, we use the mailing list to > propose and discuss new features. > > I would like to add confidence intervals for fisher's exact testing. I've > > been working with the code for a couple of months now and would like to > > make this available to everyone using this package. > > > > Sounds like a good idea to me. Do you have a proposed implementation > already, or some ideas or questions about how to add this feature to the > API? > > Cheers, > Ralf > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/scipy-dev/attachments/20191027/7330f989/attachment-0001.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > > > ------------------------------ > > End of SciPy-Dev Digest, Vol 192, Issue 31 > ****************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Oct 27 17:35:06 2019 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 27 Oct 2019 22:35:06 +0100 Subject: [SciPy-Dev] Fishers exact confidence interval In-Reply-To: References: Message-ID: On Sun, Oct 27, 2019 at 10:23 PM Jay Moore wrote: > Hi Ralf, > > I have a proposed implementation. Should I collect unit tests and > documentation and make a pull request? > Yes, please do. Cheers, Ralf > Best, > Jay > > > On Sun, Oct 27, 2019 at 09:00 wrote: > >> Send SciPy-Dev mailing list submissions to >> scipy-dev at python.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://mail.python.org/mailman/listinfo/scipy-dev >> or, via email, send a message with subject or body 'help' to >> scipy-dev-request at python.org >> >> You can reach the person managing the list at >> scipy-dev-owner at python.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of SciPy-Dev digest..." >> >> >> Today's Topics: >> >> 1. New to list (Jay Moore) >> 2. Re: New to list (Ralf Gommers) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Sat, 26 Oct 2019 23:09:38 -0400 >> From: Jay Moore >> To: scipy-dev at python.org >> Subject: [SciPy-Dev] New to list >> Message-ID: >> < >> CAOz3X_5TnZOOCOsVxGqb4V+zGO+ndfL55crJf0er_pK6OsFD_Q at mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> Hi all, >> >> I am new to this mailing list and I would like to discuss a new feature I >> would like to add to scipy stats. I have never contributed code to open >> source projects before so any guidance would be greatly appreciated. >> >> I would like to add confidence intervals for fisher's exact testing. I've >> been working with the code for a couple of months now and would like to >> make this available to everyone using this package. >> >> Best, >> Jay Moore >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: < >> http://mail.python.org/pipermail/scipy-dev/attachments/20191026/2f486e35/attachment-0001.html >> > >> >> ------------------------------ >> >> Message: 2 >> Date: Sun, 27 Oct 2019 09:09:37 +0100 >> From: Ralf Gommers >> To: SciPy Developers List >> Subject: Re: [SciPy-Dev] New to list >> Message-ID: >> < >> CABL7CQiU6TxaCo0N9C2Myq0eKSaiac_HeTkCcrH_EJOhOPf5zg at mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> On Sun, Oct 27, 2019 at 4:09 AM Jay Moore wrote: >> >> > Hi all, >> > >> > I am new to this mailing list and I would like to discuss a new feature >> I >> > would like to add to scipy stats. I have never contributed code to open >> > source projects before so any guidance would be greatly appreciated. >> > >> >> Welcome Jay! You came to the right place, we use the mailing list to >> propose and discuss new features. >> >> I would like to add confidence intervals for fisher's exact testing. I've >> > been working with the code for a couple of months now and would like to >> > make this available to everyone using this package. >> > >> >> Sounds like a good idea to me. Do you have a proposed implementation >> already, or some ideas or questions about how to add this feature to the >> API? >> >> Cheers, >> Ralf >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: < >> http://mail.python.org/pipermail/scipy-dev/attachments/20191027/7330f989/attachment-0001.html >> > >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> >> >> ------------------------------ >> >> End of SciPy-Dev Digest, Vol 192, Issue 31 >> ****************************************** >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From serge.guelton at telecom-bretagne.eu Thu Oct 31 02:32:25 2019 From: serge.guelton at telecom-bretagne.eu (Serge Guelton) Date: Thu, 31 Oct 2019 07:32:25 +0100 Subject: [SciPy-Dev] Pythran 0.9.4 - Hollsent Message-ID: <20191031063225.GA24775@sguelton.remote.csb> Hi folks, It's always a pleasure to announce a Pythran release, and here we go for a version bump! Short reminder: Pythran is an ahead-of-time compiler for scientific Python, with a focus on high-level numerical kernels, parallelism and vectorisation. Here is a simple kernel example, with a pythran annotation. Note that this kernel is still Python-compatible (from https://stackoverflow.com/questions/50658884): #pythran export euclidean_distance_square(float64[1,:], float64[:,:]) import numpy as np def euclidean_distance_square(x1, x2): return -2*np.dot(x1, x2.T) + np.sum(np.square(x1), axis=1)[:, np.newaxis] + np.sum(np.square(x2), axis=1) The Pythran package is available on PyPI, Github and Conda https://pypi.org/project/pythran/ https://anaconda.org/conda-forge/pythran https://github.com/serge-sans-paille/pythran Packages exist for archlinux and Fedora, they'll probably be updated soon :-) The interested reader can have a look to the changelog for details https://pythran.readthedocs.io/en/latest/Changelog.html Most notably, this release provides support for Python 3.7 and Python 3.8, as well as the ability to use clang-cl as a (better) backend compiler for Windows. Huge thanks to all contributors: paugier Anubhab Haldar Jean Laroche polo and bug reporters: paugier nbecker gdementen mgirardis MordicusEtCubitus Dapid piotrbartman m-romanov slayoo martibosch jeanlaroche From pierre.debuyl at kuleuven.be Thu Oct 31 08:01:09 2019 From: pierre.debuyl at kuleuven.be (Pierre de Buyl) Date: Thu, 31 Oct 2019 13:01:09 +0100 Subject: [SciPy-Dev] Documentation: user survey In-Reply-To: References: Message-ID: <20191031120109.GS2641@pi-x230> Hello, On Wed, Sep 18, 2019 at 10:57:30AM +0200, Maja Gwozdz wrote: > You can see the new version here: > https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc18-2PVEabTQg2_6g/viewform?usp=sf_link Is there any deadline for replying? Thank you, Pierre From maja.gwozdz.mkg33 at gmail.com Thu Oct 31 11:32:14 2019 From: maja.gwozdz.mkg33 at gmail.com (Maja Gwozdz) Date: Thu, 31 Oct 2019 16:32:14 +0100 Subject: [SciPy-Dev] Documentation: user survey In-Reply-To: <20191031120109.GS2641@pi-x230> References: <20191031120109.GS2641@pi-x230> Message-ID: Hi Pierre, I am planning to close the survey quite soon (Nov 5th). Best, Maja On Thu, 31 Oct 2019 at 13:07, Pierre de Buyl wrote: > Hello, > > On Wed, Sep 18, 2019 at 10:57:30AM +0200, Maja Gwozdz wrote: > > You can see the new version here: > > > https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc18-2PVEabTQg2_6g/viewform?usp=sf_link > > Is there any deadline for replying? > > Thank you, > > Pierre > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at python.org > https://mail.python.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre.debuyl at kuleuven.be Thu Oct 31 12:28:28 2019 From: pierre.debuyl at kuleuven.be (Pierre de Buyl) Date: Thu, 31 Oct 2019 17:28:28 +0100 Subject: [SciPy-Dev] Documentation: user survey In-Reply-To: References: <20191031120109.GS2641@pi-x230> Message-ID: <20191031162828.GT2641@pi-x230> Hi, On Thu, Oct 31, 2019 at 04:32:14PM +0100, Maja Gwozdz wrote: > I am planning to close the survey quite soon (Nov 5th). Thanks for the info! Pierre