Mailman 3 October 2018 - SciPy-Dev

SciPy 1.2.0 release schedule
by Tyler Reddy 06 Feb '19

06 Feb '19

Hi all, It is almost 6 months after the 1.1.0 release on May 5, so probably time to plan the 1.2.0 release. It would be a good idea to look over the PRs with a 1.2.0 milestone <https://github.com/scipy/scipy/pulls?q=is%3Aopen+is%3Apr+milestone%3A1.2.0>, and tag anything else that should have this milestone appropriately. I'd like to propose the following schedule: Nov. 5: branch 1.2.x Nov. 8: rc1 Nov. 21: rc2 (if needed) Nov. 30: final release Thoughts? Best wishes, Tyler

5 25

SciPy 1.0 paper writing proposal
by Ralf Gommers 29 Jan '19

29 Jan '19

Hi all, TL;DR, let's write the long journal paper on SciPy that we've wanted for a while, let's form a small committee to coordinate, and get it out the door in 2-3 months. Motivation --------------- (credits for most of this text: Evgeni) Many scipy contributors' day jobs are in academia. Bibliometry -- papers in refereed journals and citations of papers by other papers -- is one of the main performance indicators in most academic establishments. Since we do not generate papers, scipy contributions are all but invisible for the purposes of a contributor's annual report. Of course, details vary wildly; in many cases a contributor manages to balance their time, or to argue common sense with their superiors, or get an approval for scipy work, or just ignores the issue altogether -- but sooner or later there is a form to be filled or boxes to be checked, and scipy contributions simply do not fit in. A peer-reviewed journal paper on scipy will help contributors get the academic credit they deserve. We can write *the* paper for SciPy 1.0, with overall project structure, goals, etc., and for specific features/modules a focus on say the last 3 years. History ---------- For SciPy 1.0 we had three targets on the publicity/credits front: an interesting release announcement, interesting blogs/stories (NumFOCUS blog, Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in the end, the rest was successful. [1] is a previous announcement on this list about writing (a) paper(s) on SciPy. We wanted both "short papers" to cover one or two releases (target journal JOSS) and a full paper as the authoritative reference for SciPy. We had an earlier attempt for a "short paper", it's mostly written but has stalled (see [2]). We ran out of steam on that one. To avoid that this time around, it would be good to have a clear public plan, target dates, and a small committee rather than one person to drive things forward. Proposal ------------ Here's a proposal for all aspects of this exercise that I can think about now. Some parts stolen from the AstroPy paper [3] (because their process worked quite well). Form a small coordination committee of 3-5 people that set up the paper structure, move things along when parts stall, propose/take decisions as needed, invite co-authors, and organise paper submission/rework. Paper writing to be done by whoever volunteers for a section, not just the coordination committee. First outline/structure to be done by committee, which then asks for review of structure and volunteers for section writing. Scope: a 6-10 page paper, covering history, package scope and structure, community/organisational aspects, key features and recent enhancements per module, and roadmap. Authorship: anyone who made a substantial contribution in the history of the project. Here "substantial" is interpreted as anything beyond a one-line doc fix. Rationale: better to be too inclusive than exclusive. Sign-up via a web form, we send the link to that form to all email addresses in the commit history till v1.0. Author order (details tbd by committee): 1. The SciPy Developers 2. Maintainers, paper writers, other key contributors - in order of contribution level 3. All other authors - alphabetically ordered Submission target: mid-April, to either PeerJ Computer Science or Journal of Open Research Software (tbd by committee). Comments? Volunteers for committee? References ---------------- [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html [2] https://github.com/scipy/scipy-articles/pull/4 [3] https://github.com/astropy/astropy-v2.0-paper Cheers, Ralf

12 19

building scipy for WebAssembly
by Roman Yurchak 02 Nov '18

02 Nov '18

Hello, I am currently working on building scipy for WebAssembly as part of pyodide project (https://github.com/iodide-project/pyodide) and I was hoping for some feedback on that process. There is a preliminary build in https://github.com/iodide-project/pyodide/pull/211 Currently, this build uses scipy 0.17.1 as from what I understood, that was one of the last versions that only included f77 without any f90 (https://github.com/scipy/scipy/issues/2829#issuecomment-223764054). In the WebAssembly environment there is currently no reliably working Fortran compiler (https://github.com/iodide-project/pyodide/issues/184), and f90 cannot be converted to C with f2c unlike f77. If one wanted to (experimentally) compile the latest version of scipy without a Fortran compiler what would be your suggestions? i.e. - are there any alternatives to f2c that you think might work for f90 - or any automatic converter f90 to f77 (so that f2c could be used) I did search but maybe I missed something. Alternatively, if that's really not realistic, could someone please comment on the rate of adoption for f90/f95 in the scipy code base? The decision to support f90 was taken before the 0.18 release in 2016 but I'm not sure what impact it had on the code base. In other words maybe there is a later version than 0.17.1 that might (mostly) work? Another point is linking BLAS/LAPACK. Currently reference BLAS and CLAPACK are linked statically as I haven't managed to do this dynamically yet. The issue is the package size. As LAPACK, which is quite large, gets repeatedly included in around ~10 different .so modules, resulting in a 170MB large package (after compression), as opposed to ~30MB compressed package without BLAS/LAPACK. That is quite problematic when one is expected to download the dependencies at each page load (excluding caching). I'm not sure if there are other distributions of scipy that use static linking of LAPACK? or other things worth trying to reduce the package size, short of trying to make dynamic linking work or try to detect and strip unused symbols? Also in scipy/linalg/setup.py I was wondering why/how the ATLAS_INFO macro defined the existence of `scipy.linalg._clapack`? For instance, when using CLAPACK (with libf2c), would the following be correct? lapack_opt = {'libraries': ['f2c', 'blas', 'lapack'], 'include_dirs': [], 'library_dirs': ['<path_to_blas_lapack_lib>'], 'language': 'f77', 'define_macros': [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]} which does not appear to build `scipy.linalg._clapack`. Finally, in later scipy versions, https://scipy.github.io/devdocs/building/linux.html suggests that with “export LAPACK=” one can build scipy without LAPACK, but then _flapack is still a mandatory import in scipy.linalg.__init___ (via .lapack) as far as I can tell. Is scipy.linalg (and anything that depends on it) expected to produce import errors in the latter case then? In general, any comments or suggestions would be appreciated, Thank you, Roman

2 2

Using Spark to scale SciPy?
by rlucente＠pipeline.com 31 Oct '18

31 Oct '18

I ran into a blog post titled Prediction at Scale with scikit-learn and PySpark Pandas UDFs by Michael Heilman https://medium.com/civis-analytics/prediction-at-scale-with-scikit-learn-and -pyspark-pandas-udfs-51d5ebfb2cd8 It seems to do a good job because it makes statements like One issue is that passing data between a) Java-based Spark execution processes, which send data between machines and can perform transformations super-efficiently, and b) a Python process (e.g., for predicting with scikit-learn) incurs some overhead due to serialization and inter-process communication. The article goes on to mention approaches to mitigate the above issues. For example: Having UDFs expect Pandas Series also saves converting between Python and NumPy floating point representations for scikit-learn There was actually a SciPy talk on this Performing Dimension Reduction at Scale with Applications to Public Sentiment Models | SciPy 2018 https://www.youtube.com/watch?v=31YeSfDklfc The other good thing is that it has code https://gist.github.com/mheilman/6ce261549b55bf4997ec102ad4e8d643#file-pyspa rk_pandas_udf_sklearn-ipynb I was wondering what approaches people have used to scale SciPy? The Bit Plumber <http://rlucente.blogspot.com/>

2 1

matrix_sqrt for singular symmetric square matrices
by josef.pktd＠gmail.com 29 Oct '18

29 Oct '18

I needed a more general version of a matrix square root for GAM in statsmodels. https://github.com/statsmodels/statsmodels/pull/5296 Is there already something like this in numpy/scipy land? Would there be interest in adding something like this? Improvements to my implementation? (threshold is not in terms of rcond, because I have more intuition about small eigen values.) (I still don't consider myself to be a linalg expert.) Josef def matrix_sqrt(mat, inverse=False, full=False, nullspace=False, threshold=1e-15): """matrix square root for symmetric matrices Usage is for decomposing a covariance function S into a square root R such that R' R = S if inverse is False, or R' R = pinv(S) if inverse is True Parameters ---------- mat : array_like, 2-d square symmetric square matrix for which square root or inverse square root is computed. There is no checking for whether the matrix is symmetric. A warning is issued if some singular values are negative, i.e. below the negative of the threshold. inverse : bool If False (default), then the matrix square root is returned. If inverse is True, then the matrix square root of the inverse matrix is returned. full : bool If full is False (default, then the square root has reduce number of rows if the matrix is singular, i.e. has singular values below the threshold. nullspace: bool If nullspace is true, then the matrix square root of the null space of the matrix is returned. threshold : float Singular values below the threshold are dropped. Returns ------- msqrt : ndarray matrix square root or square root of inverse matrix. """ # see also scipy.linalg null_space u, s, v = np.linalg.svd(mat) if np.any(s < -threshold): import warnings warnings.warn('some singular values are negative') if not nullspace: mask = s > threshold s[s < threshold] = 0 else: mask = s < threshold s[s > threshold] = 0 sqrt_s = np.sqrt(s[mask]) if inverse: sqrt_s = 1 / np.sqrt(s[mask]) if full: b = np.dot(u[:, mask], np.dot(np.diag(sqrt_s), v[mask])) else: b = np.dot(np.diag(sqrt_s), v[mask]) return b

5 12

Unable to build RPM package via setup.py bdist_rpm
by Yann Büchau 24 Oct '18

24 Oct '18

Hello everyone, After countless attempts and a good portion of frustration I am now asking for help in this mailinglist. I would like to build a numpy RPM package. There is the |setup.py bdist_rpm| command which does that. My final goal is to build numpy for SailfishOS (mobile operating system) myself. (There are numpy packages on OpenRepos.net, but they are outdated and I need an up-to-date numpy to package an up-to-date matplotlib…) Eventually, |python3 setup.py bdist_rpm| always fails with: |executing numpy/core/code_generators/generate_numpy_api.py Running from numpy source directory. /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'define_macros' warnings.warn(msg) error: [Errno 2] No such file or directory: 'numpy/core/code_generators/../src/multiarray/arraytypes.c.src' | I was able to boil the problem down to these files missing in the source tarball |build/bdist_rpm*/rpm/SOURCES/numpy-1.15.3.tar.gz|: |numpy/linalg/umath_linalg.c.src numpy/core/src/multiarray/arraytypes.c.src numpy/core/src/multiarray/scalartypes.c.src numpy/core/src/umath/_umath_tests.c.src numpy/core/src/umath/loops.h.src numpy/core/src/multiarray/nditer_templ.c.src numpy/core/src/umath/_operand_flag_tests.c.src numpy/core/src/umath/_struct_ufunc_tests.c.src numpy/core/src/umath/_rational_tests.c.src numpy/core/src/umath/scalarmath.c.src numpy/core/src/multiarray/_multiarray_tests.c.src numpy/core/src/multiarray/lowlevel_strided_loops.c.src numpy/core/src/umath/funcs.inc.src numpy/core/src/umath/loops.c.src numpy/core/src/multiarray/einsum.c.src | Interestingly, when I run |python3 setup.py sdist|, the created tarball under |dist/numpy-1.15.3.tar.gz| contains these files. So my question is: What is going wrong here? The |bdist_rpm| log shows that it is |running sdist|, but these |*.src|-files are not included in this run. What is different in the |bdist_rpm| run? I also was not able to just run |rpmbuild| directly due to strange errors like |error: Macro %__python has empty body| and the like. The |python3 setup.py install| mechanism works well both on my Ubuntu 18.04 and my Jolla Phone with SailfishOS. Just the |bdist_rpm| part does not work. I would be very delighted if someone could help me with this. Cheers, Yann

1 0

Finding signal upcrossings and declustered peaks
by Ali Cetin 22 Oct '18

22 Oct '18

Hi, this is my first attempt to write to this mailing list, so if I'm doing something wrong please bear with me. I've been working with extreme value analysis during my PhD and also in my current job. What we often do is to find signal upcrossings (such as mean and zero upcrossings) and find declustered peaks. That is, find the largest peak between two upcrossings, or the two largest peaks between two upcrossings etc... I'm proposing to add two new functions to the scipy.signal submodule: - find_upcross; takes in signal and returns index of upcrossings wrt user defined upcrossing level - find_peaks_dc; takes in signal and returns index of peaks (n largest) (find_peaks_dc is not necessarly easily incorporated into find_peaks, so it may be cleaner to have a separate function). Any thought on this? Cheers, Ali

3 8

NumPy 1.15.3 release
by Charles R Harris 22 Oct '18

22 Oct '18

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.15.3. This is a bugfix release for bugs and regressions reported following the 1.15.2 release. The most noticeable fix is probably for the memory leak <https://github.com/numpy/numpy/issues/12037> encountered when slicing classes derived from Numpy. The Python versions supported by this release are 2.7, 3.4-3.7. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.15.3/>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.15.3>. Compatibility Note ================== The NumPy 1.15.x OS X wheels released on PyPI no longer contain 32-bit binaries. That will also be the case in future releases. See `#11625 <https://github.com/numpy/numpy/issues/11625>`__ for the related discussion. Those needing 32-bit support should look elsewhere or build from source. Contributors ============ A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Allan Haldane * Charles Harris * Jeroen Demeyer * Kevin Sheppard * Matthew Bowden + * Matti Picus * Tyler Reddy Pull requests merged ==================== A total of 12 pull requests were merged for this release. * `#12080 <https://github.com/numpy/numpy/pull/12080>`__: MAINT: Blacklist some MSVC complex functions. * `#12083 <https://github.com/numpy/numpy/pull/12083>`__: TST: Add azure CI testing to 1.15.x branch. * `#12084 <https://github.com/numpy/numpy/pull/12084>`__: BUG: test_path() now uses Path.resolve() * `#12085 <https://github.com/numpy/numpy/pull/12085>`__: TST, MAINT: Fix some failing tests on azure-pipelines mac and... * `#12187 <https://github.com/numpy/numpy/pull/12187>`__: BUG: Fix memory leak in mapping.c * `#12188 <https://github.com/numpy/numpy/pull/12188>`__: BUG: Allow boolean subtract in histogram * `#12189 <https://github.com/numpy/numpy/pull/12189>`__: BUG: Fix in-place permutation * `#12190 <https://github.com/numpy/numpy/pull/12190>`__: BUG: limit default for get_num_build_jobs() to 8 * `#12191 <https://github.com/numpy/numpy/pull/12191>`__: BUG: OBJECT_to_* should check for errors * `#12192 <https://github.com/numpy/numpy/pull/12192>`__: DOC: Prepare for NumPy 1.15.3 release. * `#12237 <https://github.com/numpy/numpy/pull/12237>`__: BUG: Fix MaskedArray fill_value type conversion. * `#12238 <https://github.com/numpy/numpy/pull/12238>`__: TST: Backport azure-pipeline testing fixes for Mac Cheers, Charles Harris

1 0

Re: [SciPy-Dev] Strange code in scipy.signal.decimate
by Eric Quintero 18 Oct '18

18 Oct '18

Warren is correct; there were some changes to how the `dlti` objects were being internally passed around on PR 7835, and this extra instantiation flew under the radar. A PR to simplify the code would be very welcome. -Eric Q. > On Oct 18, 2018, at 7:07 PM, Warren Weckesser <warren.weckesser(a)gmail.com> wrote: > > On 10/18/18, Daniele Nicolodi <daniele(a)grinta.net> wrote: >> On 18-10-2018 15:28, Paul Hobson wrote: >>> Dan, >>> >>> I might be missing something, but dlti returns a subclass of >>> LinearTimeInvariant. Point is, it's not a tuple, but a class with >>> multiple attributes and method. Simple tuple unpacking won't likely work >>> unless the authors of LinearTimeInvariant really went out of their way >>> to make it so. >> >> The point is not to go through the `dlti` class at all. >> >> Please look at the replacement code I posted: it works. >> > > > There have been several incremental changes to that function over the > years. I suspect the last person to change it simply did not notice > that the code could be simplified. Your proposed change looks good. > > Warren > > >> Cheers, >> Dan >> >>> -Paul >>> >>> On Thu, Oct 18, 2018 at 1:58 PM Daniele Nicolodi <daniele(a)grinta.net >>> <mailto:daniele@grinta.net>> wrote: >>> >>> Hello, >>> >>> I was having a look at the scipy.signal.decimate() function and I >>> noticed some strange looking code: >>> >>> elif ftype == 'iir': >>> if n is None: >>> n = 8 >>> system = dlti(*cheby1(n, 0.05, 0.8 / q)) >>> b, a = system.num, system.den >>> >>> This is used to setup the anti aliasing low pass filter. What I don't >>> understand is the dance to obtain the IIR numerator and denominator >>> coefficients. Couldn't the above be simply as the code below? >>> >>> elif ftype == 'iir': >>> if n is None: >>> n = 8 >>> b, a = cheby1(n, 0.05, 0.8 / q) >>> >>> Am I missing something? >>> >>> Thanks! >>> >>> Cheers, >>> Dan >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev(a)python.org <mailto:SciPy-Dev@python.org> >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev(a)python.org >>> https://mail.python.org/mailman/listinfo/scipy-dev >>> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev(a)python.org >> https://mail.python.org/mailman/listinfo/scipy-dev >> > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev(a)python.org > https://mail.python.org/mailman/listinfo/scipy-dev

1 0

Strange code in scipy.signal.decimate
by Daniele Nicolodi 18 Oct '18

18 Oct '18

Hello, I was having a look at the scipy.signal.decimate() function and I noticed some strange looking code: elif ftype == 'iir': if n is None: n = 8 system = dlti(*cheby1(n, 0.05, 0.8 / q)) b, a = system.num, system.den This is used to setup the anti aliasing low pass filter. What I don't understand is the dance to obtain the IIR numerator and denominator coefficients. Couldn't the above be simply as the code below? elif ftype == 'iir': if n is None: n = 8 b, a = cheby1(n, 0.05, 0.8 / q) Am I missing something? Thanks! Cheers, Dan

3 4