Mailman 3 March 2020 - SciPy-Dev

Re: [SciPy-Dev] Discontinued Rackspace Open Compute Discount
by Robert Lucente 05 Oct '20

05 Oct '20

Several CI services are free for open source projects although there are throughput limits https://about.gitlab.com https://github.com/features/actions https://travis-ci.org https://www.appveyor.com -----Original Message----- >From: Olivier Grisel <olivier.grisel(a)ensta.org> >Sent: Feb 8, 2020 11:30 AM >To: SciPy Developers List <scipy-dev(a)python.org> >Subject: [SciPy-Dev] Discontinued Rackspace Open Compute Discount > >Hi all, > >I received an email end of January that I just discovered today >telling me that the Rackspace Open Compute Discount grant we use to >host http://wheels.scipy.org for nightly builds and release management >of our community has been discontinued on December 31st 2019. > >I would like to take down the Rackspace hosted cloudfiles as soon as >possible but we need to find a replacement as this will break all the >CIs that rely on nightly wheels and also all the wheel release >infrastructure for several project in the ecosystem. > >Apparently the fees for January and the beginning of February already >sum to $936.51. This is a large cost because there were a bunch of >unused left-over block volumes that I already deleted. > >To host wheels for nightly builds and release automation, a possible >alternatives include using the github release pages but they are not >really designed for this kind of workflow. > >Alternatively there is a PyPI repo hosting feature in Azure Pipelines >in public preview but I am not sure how it works, if it's free for >open source projects and if we could use it as a replacement for the >Rackspace cloud storage we currently use to host >http://wheels.scipy.org . > >https://docs.microsoft.com/en-us/azure/devops/pipelines/artifacts/pypi?view… > >Important: if you have uploaded any important files to one of the >cloud file storage / blob containers on Rackspace, please download a >copy as soon as possible. > >But please do not download things you do not really want to keep to >avoid adding any unnecessary bandwidth costs. If you have file write >permission on one of those containers, please delete the files you own >and do not need anymore. > >Feel free to join this room to discuss the matter interactively: >https://gitter.im/scikit-learn/nightly-build-hosting > >Feel free to reply to this email on the scipy-dev mailing list if you >have suggestions for setting up a free replacement. > >-- >Olivier >http://twitter.com/ogrisel - http://github.com/ogrisel >_______________________________________________ >SciPy-Dev mailing list >SciPy-Dev(a)python.org >https://mail.python.org/mailman/listinfo/scipy-dev

12 25

Tweedie distributions in scipy.stats
by Christian Lorentzen 17 Jul '20

17 Jul '20

Dear Scipy Developers and mailing list Readers I'd like to address the issue [1] to implement Tweedie distributions [2] in scipy.stats. Purpose The family of Tweedie distributions contains many known distributions like the Poisson and the Gamma distribution, but also distributions between them, aka compound poisson gamma distribution, see [3]. These are often appropriate for insurance claims and other fields, where one has a (Poisson) random count process of events and every event has a (Gamma) random size/amount. The distribution would enable simulations, maximum likelihood estimation of all parameters, choice and visualization of distributions, etc. Implementation I started PR [4] for Wrights generalized Bessel functions as a private function in scipy.special. Once this is ready, the pdf follows immediately. For the range of interest of Y ~ compound poisson gamma distribution, the distribution of Y has a point mass at zero and is otherwise continuous for Y>0. As already discussed in the issue [1], Tweedie might best fit as |rv_generic|. As such, it would be the first one, all others are either |rv_discrete| or |rv_continuous|. Without a template, I would need guidance how to implement a new rv_generic. References: [1] https://github.com/scipy/scipy/issues/11291 [2] https://en.wikipedia.org/wiki/Tweedie_distribution [3] https://en.wikipedia.org/wiki/Compound_Poisson_distribution [4] https://github.com/scipy/scipy/pull/11313 I'm looking forward to your feeback, thoughts and insights. Kind regards, Christian

4 10

SciPy Contribution: Gammatone Filters in scipy.signal
by Shashaank Narayanan 23 May '20

23 May '20

Hello SciPy Team, I am new to this mailing list, and I am interested in contributing to SciPy. I would like to suggest a new feature to be added to the scipy.signal module: gammatone filters. Gammatone filters are becoming increasingly popular in the fields of digital signal processing and music analysis as it effectively models the auditory filters of the human auditory system. Currently, there are very few implementations of gammatone filters available for Python, and these implementations are not generalized to basic finite impulse response (FIR) and infinite impulse response (IIR) filters like SciPy has. I have written my own gammatone FIR filter using NumPy based on Malcolm Slaney's 1993 paper on the topic ( https://engineering.purdue.edu/~malcolm/apple/tr35/PattersonsEar.pdf). This paper was used for Matlab's implementation of gammatone filters. I am in the process of writing a gammatone IIR filter with NumPy and SciPy. Please let me know if this feature will fit with the scipy.signal module. Appreciate your time and guidance. Thanks, Shashaank

5 6

Improving scaling factors in IIR SOS sections in scipy.signal
by Mojca Miklavec 31 Mar '20

31 Mar '20

Dear signal processing gurus, I'm using python for playing around, testing and quickly prototyping IIR filters for a piece of code which is otherwise written in C[++] & FPGA, but I miss one thing in zpk2sos, namely the complete scaling factor ends up in the first sos section, rather than being "evenly distributed" across all the sections. For floating point operations this doesn't make much of a difference, but not applying the scaling inside a sos section with integer arithmetics is potentially more problematic to work with (and I would like to have support for this to be able to compare calculations & results). Example: >>> signal.butter(8, 3E-4, output='sos') array([[ 2.42594124e-27, 4.85188247e-27, 2.42594124e-27, 1.00000000e+00, -1.99815208e+00, 9.98152971e-01], [ 1.00000000e+00, 2.00000000e+00, 1.00000000e+00, 1.00000000e+00, -1.99843306e+00, 9.98433944e-01], [ 1.00000000e+00, 2.00000000e+00, 1.00000000e+00, 1.00000000e+00, -1.99895244e+00, 9.98953323e-01], [ 1.00000000e+00, 2.00000000e+00, 1.00000000e+00, 1.00000000e+00, -1.99963144e+00, 9.99632331e-01]]) I would like to see the 2.4259e-27 part to be distributed among the four sos sections (rather than ending up with [1 2 1] everywhere else), which is also what Matlab returns. Since I heavily prefer using Python compared to Matlab (I don't even have the licence myself), I wanted to ask for your opinion before I start writing the code. Would such a change be acceptable upstream? And what would be the best way to handle "backward compatibility"? (I've never contributed to Python before, but I assume it's doable :) A bunch of unrelated questions: (1) I was playing with signal.sosfiltfilt(...) to apply an IIR filter to the input signal. I didn't check the source code, but the results look as if the filter was working partially "backwards". That is: it seems to be using some points from the "future" to determine the result, similar to how a FIR filter would take some points from the past + some points from the future to calculate the current point. I can send some screenshot, but I'm obviously missing something important here. Is there another approach to exactly simulate out[i] = b[0] * in[i] + b[1] * in[i-1] + b[2] * in[i-2] - a[1] * out[i-1] - a[2] * out[i-2] for the output? (2) I would like to simulate the filter with integer (fixed point) arithmetics. For a low-pass filter with either very low or very high cut-off frequency it's way too easy to either get into integer overflows, or be forced into cutting a lot of significant bits off. What's the least painful way to perform multiplications with a slightly higher bit count? Sadly something like dtype=np.int128 doesn't seem to exist :( I'm currently having some troubles with LP filters with cut-off frequency very close to the Nyquist frequency, and I'm thinking of perhaps improving the signal package first before investigating the problem any further, as that would help me simulate and compare where things go wrong. (3) Matlab seems to have some support for converting coefficients from sos sections into integers, and also tells you how many bits are needed to perform fixed-point arithmetics. It offers one specific way of rounding the coefficients to keep the calculations stable (I'm sorry, I don't remember what exactly it says, a friend showed it to me a while ago, but we no longer have access to that computer with Matlab installed). Does anyone know anything about this? I suspect it tries to round complex numbers close to the unit circle in a way that ensures the points remain inside the circle, but that's pure speculation. (It would be nice to be able to offer something similar on the python side.) (4) I have an ad-hoc simulation for filters with integer arithmetics. But it might be nice to add this support to scipy.signal as well, as this functionality might be generally useful. I can work on this, but I wouldn't mind a little bit of guidance on the subject if I start working in that direction. Anyway, I would start by fixing the coefficients first, and leave this one for the end. Thank you very much, Mojca PS: I'm relatively new to the subject of filtering digital signals, so please bear with me :)

2 2

Azure CI failures
by Warren Weckesser 30 Mar '20

30 Mar '20

Some Azure tests for a couple recent PRs are failing with the message ##[error]TF24668: The following team project collection is stopped: scipy-org. Start the collection and then try again. Administrator Reason: abuse The message says "Administrator Reason: abuse". Does anyone know what this is about? Warren

3 3

Alternative return value for io.loadmat
by Brunner, Clemens (clemens.brunner＠uni-graz.at) 29 Mar '20

29 Mar '20

Hello! Loading .mat files with scipy.io.loadmat returns a rather complex dict object with nested struct arrays. If the .mat file contains cell arrays, the resulting structure is particularly hard to parse for humans (see e.g. https://github.com/scipy/scipy/issues/7895). For this reason, people have come up with conversion routines that simplify the result of io.loadmat (e.g. http://blog.nephics.com/2019/08/28/better-loadmat-for-scipy). Simple example using the following MATLAB struct (download the .mat file here: https://github.com/scipy/scipy/files/1314287/matlab.zip): >> s = struct('mycell', {{'a', 'b', 'c'}}); Accessing the value 'a' with scipy.io.loadmat: >>> mat["s"]["mycell"][0, 0][0, 0][0] 'a' The solution from the blog post provides easier access to the same value because the nested struct arrays get flattened: >>> mat1["s"]["mycell"][0] 'a' In https://github.com/scipy/scipy/issues/7895 we propose to add such an alternative return value structure to scipy.io.loadmat. Do you think this is a useful addition that should be included? If so, what would be the best way to integrate it? Should scipy.io.loadmat get a new keyword argument that influences what is returned (i.e. the default behavior should return the current value, whereas setting the kwarg to a specific value would return the alternative value)? Or should there be a new function that gets the current value and converts it to the alternative structure? Thanks! Clemens

2 1

Re: [SciPy-Dev] Improving scaling factors in IIR SOS sections in scipy.signal
by 3ukip0s02＠sneakemail.com 25 Mar '20

25 Mar '20

There was some discussion about this at the time, but I don't think there has been any work on implementation. See https://github.com/scipy/scipy/pull/3717#issuecomment-45398832 and replies. (I don't think closed issues come up in Google searches as easily)

1 0

PR #11612
by Will Tirone 21 Mar '20

21 Mar '20

Hi everyone, First time contributor to Scipy here. I'm trying to fix a function, although for some reason it's failing the scipy.spatial distance testing ( https://github.com/scipy/scipy/pull/11612). I've read all the docs I can find and looked through the testing functions and I cannot seem to figure it out. I just followed an example of a different test for a similar function, although mine's still failing. I hate to keep asking my reviewer really basic questions, but is anyone available to take a look and see if 1) my fix isn't working or 2) I'm testing it incorrectly? And let me know if this isn't the usual format for sending stuff to the mailing list, first time doing this as well. Thanks in advance, really appreciate the community support! -Will

2 1

Shim for RandomState/Generator
by Andrew Nelson 18 Mar '20

18 Mar '20

Hi all, default random number generation is heading towards use of np.random.Generator, with RandomState being preserved for legacy usage. It would be nice to start using Generator's, but in order to do that in scipy we would need to be able to write code that worked with RandomState or Generator. Unfortunately there are a few methods that have changed their name, `randint `--> `integers` and `rand/random_sample` --> `random` spring to mind I was wondering if we could add a shim to go in `scipy._lib._util` that would permit code to be written as if it were using the Generator front end (at least most of it), but could be using either RandomState or Generator as a backend. An example for the shim code would be: ``` from scipy._lib._util import check_random_state from numpy.random import Generator, RandomState class RG(object): """ Shim object for working across RandomState and Generator. RandomState is legacy, Generator is the future. """ def __init__(self, seed): rng = check_random_state(seed) # methods = [f for f in dir(np.random.default_rng()) if not f.startswith('_')] # methods.remove('integers') # methods.remove('random') methods = ['beta', 'binomial', 'bytes', 'chisquare', 'choice', 'dirichlet', 'exponential', 'f', 'gamma', 'geometric', 'gumbel', 'hypergeometric', 'laplace', 'logistic', 'lognormal', 'logseries', 'multinomial', 'multivariate_normal', 'negative_binomial', 'noncentral_chisquare', 'noncentral_f', 'normal', 'pareto', 'permutation', 'poisson', 'power', 'rayleigh', 'shuffle', 'standard_cauchy', 'standard_exponential', 'standard_gamma', 'standard_normal', 'standard_t', 'triangular', 'uniform', 'vonmises', 'wald', 'weibull', 'zipf'] for method in methods: setattr(self, method, getattr(rng, method)) if isinstance(rng, RandomState): setattr(self, 'integers', rng.randint) setattr(self, 'random', rng.random_sample) elif isinstance(rng, Generator): setattr(self, 'integers', rng.integers) setattr(self, 'random', rng.random) setattr(self, 'multivariate_hypergeometric', rng.multivariate_hypergeometric) self.rng = rng @property def state(self): if isinstance(self.rng, RandomState): return self.rng.get_state() elif isinstance(self.rng, Generator): return self.rng.bit_generator.state @state.setter def state(self, state): if isinstance(self.rng, RandomState): self.rng.set_state(state) elif isinstance(self.rng, Generator): self.rng.bit_generator.state = state ``` -- _____________________________________ Dr. Andrew Nelson _____________________________________

2 1

NumPy 1.18.2 released.
by Charles R Harris 17 Mar '20

17 Mar '20

Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.18.2 has been released.This small release contains a fix for a performance regression in numpy/random and several bug/maintenance updates. The Python versions supported in this release are 3.5-3.8. Downstream developers should use Cython >= 0.29.15 for Python 3.8 support and OpenBLAS >= 3.7 to avoid errors on the Skylake architecture. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.18.2>, source archives and release notes are available from Github <https://github.com/numpy/numpy/releases/tag/v1.18.2>. *Contributors* A total of 5 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Ganesh Kathiresan + - Matti Picus - Sebastian Berg - przemb + *Pull requests merged* A total of 7 pull requests were merged for this release. - https://github.com/numpy/numpy/pull/15675: TST: move _no_tracing to testing._private - https://github.com/numpy/numpy/pull/15676: MAINT: Large overhead in some random functions - https://github.com/numpy/numpy/pull/15677: TST: Do not create gfortran link in azure Mac testing. - https://github.com/numpy/numpy/pull/15679: BUG: Added missing error check in ndarray.__contains__ - https://github.com/numpy/numpy/pull/15722: MAINT: use list-based APIs to call subprocesses - https://github.com/numpy/numpy/pull/15729: REL: Prepare for 1.18.2 release. - https://github.com/numpy/numpy/pull/15734: BUG: fix logic error when nm fails on 32-bit Cheers, Charles Harris

1 0