Mailman 3 February 2020 - SciPy-Dev

Newbie question: submitting PRs, review
by Tom Roderick Feb. 16, 2020

Feb. 16, 2020

Hi folks This is my first time mailing the list, so forgive me if I come across naive -- it is because I am! I noticed that the metric in scipy.spatial.distance.sokalmichener is the wrong metric, as per many sources (e.g. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.352.6123&rep=rep1…) -- the rogerstanimoto dissimilarity is in its place. I'd like to correct it, but am not 100% sure I understand the process. So, wanted to reach out to the mailing list and hopefully get some mentoring. Is the Scipy process as simple as: fix code, add unit test, submit Github PR? Best, -Tom

3 5

adding Khatri-Rao product to linalg
by Ralf Gommers Feb. 16, 2020

Feb. 16, 2020

Hi all, I think this wasn't run by this list yet: https://github.com/scipy/scipy/pull/11456 is a PR to add the Khatri-Rao product to scipy.linalg. It seems like a good fit to me. Please comment here or on the PR soon if you disagree. Unless there's objections, we'll merge it in a few days. Cheers, Ralf

2 3

Discontinued Rackspace Open Compute Discount
by Olivier Grisel Feb. 8, 2020

Feb. 8, 2020

Hi all, I received an email end of January that I just discovered today telling me that the Rackspace Open Compute Discount grant we use to host http://wheels.scipy.org for nightly builds and release management of our community has been discontinued on December 31st 2019. I would like to take down the Rackspace hosted cloudfiles as soon as possible but we need to find a replacement as this will break all the CIs that rely on nightly wheels and also all the wheel release infrastructure for several project in the ecosystem. Apparently the fees for January and the beginning of February already sum to $936.51. This is a large cost because there were a bunch of unused left-over block volumes that I already deleted. To host wheels for nightly builds and release automation, a possible alternatives include using the github release pages but they are not really designed for this kind of workflow. Alternatively there is a PyPI repo hosting feature in Azure Pipelines in public preview but I am not sure how it works, if it's free for open source projects and if we could use it as a replacement for the Rackspace cloud storage we currently use to host http://wheels.scipy.org . https://docs.microsoft.com/en-us/azure/devops/pipelines/artifacts/pypi?view… Important: if you have uploaded any important files to one of the cloud file storage / blob containers on Rackspace, please download a copy as soon as possible. But please do not download things you do not really want to keep to avoid adding any unnecessary bandwidth costs. If you have file write permission on one of those containers, please delete the files you own and do not need anymore. Feel free to join this room to discuss the matter interactively: https://gitter.im/scikit-learn/nightly-build-hosting Feel free to reply to this email on the scipy-dev mailing list if you have suggestions for setting up a free replacement. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel

1 0

Extending scipy.stats.kstat
by Kevin Smith Feb. 7, 2020

Feb. 7, 2020

Hello, I am writing a function for multivariate k-statistics (minimum-variance unbiased estimators of multivariate cumulants), and am wondering if there would be any interest in adding this to code to scipy.stats. The current function, scipy.stats.kstat, is limited to univariate k-stats up to order 4, whereas my implementation extends to arbitrary order. Thanks, Kevin

2 2

Re: [SciPy-Dev] SciPy-Dev Digest, Vol 196, Issue 1
by Lucas Roberts Feb. 7, 2020

Feb. 7, 2020

On Sun, Feb 2, 2020 at 12:00 PM <scipy-dev-request(a)python.org> wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request(a)python.org > > You can reach the person managing the list at > scipy-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. language labels in github issues/PRs? (Lucas Roberts) > 2. Re: language labels in github issues/PRs? (Ralf Gommers) > 3. Re: language labels in github issues/PRs? (Kai Striega) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 1 Feb 2020 11:56:56 -0500 > From: Lucas Roberts <rlucas7(a)vt.edu> > To: scipy-dev(a)python.org > Subject: [SciPy-Dev] language labels in github issues/PRs? > Message-ID: > < > CADCr_xOm9iv4xKKBBAtnTCbt57bYmaG6c4QS9x5Y5oGaQwcNEQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi scipy-dev, > > I was looking at my stack of emails for scipy and noticed some comments on: > https://github.com/scipy/scipy/issues/3203# > and that the c code is involved. In the past I've marked these types of > emails for myself to flag for follow up (e.g. I'd like to work on them if > still open in the future). Problem is that the stack of these flags gets > too large in my inbox. I noticed there aren't tags for language in the > scipy github. Scipy has C, C++, Cython, Fortran, and Python (I think that > is all) used in various places. > > My question is would it be helpful to have language tags? > > We probably wouldnt' want a python tag but if an issue or a PR touches > C/C++/Cython/fortran adding a label would help discoverability and > exclusion. > > This might help searching of new contributors and for core team, e.g. you > could search for issues touching c code (in my case) or exclude issues > touching fortran (also my case, I don't know fortran). > > I didn't know how to exclude it Github and learned that it has some > searching functionality that allows to exclude issues/PRs with certain > tags: > > > https://help.github.com/en/github/searching-for-information-on-github/searc… > > -- > Sincerely, > -Lucas >

1 0

Implementation choices in scipy.sparse.linalg.LinearOperator
by matteo ravasi Feb. 4, 2020

Feb. 4, 2020

Dear Scipy Developers, some of you may be aware of the PyLops project (https://github.com/equinor/pylops), which heavily builds on the scipy.sparse.linalg.LinearOperator class of scipy and provides an extensive library of general purpose and domain specific linear operators. Recently a users asked for a functionality that would create the equivalent dense matrix representation of an operator (to be mostly used for teaching purposes). While implementing it, we came across two internal behaviours in the interface.py file that we would like to mention and discuss about: - The method _matmat and _rmatmat as implemented per today requires operators to work on (N, 1)/(M, 1) arrays. Whilst the first sentence in the Notes of LinearOperator states "The user-defined matvec() function must properly handle the case where v has shape (N,) as well as the (N,1) case", I tend to believe that the first option should be the encouraged one, as I do not really see any advantage with carrying over trailing dimensions in this case. Moreover, some of the linear solvers in scipy that can be used together with linear operator such as lsqr or lsmr, require the RHS b vector to have a single dimension (b : (m,) ndarray). Again, although a quick test showed that they work also when providing a b vector of shape (m, 1), this shows to me that the extra trailing dimension is just a nuisance and should be discouraged. Assuming that you agree with this argument, an alternative solution that works on (N, ) input is actually possible and currently implemented in PyLops for _matmat (see here https://github.com/equinor/pylops/blob/master/pylops/LinearOperator.py <https://github.com/equinor/pylops/blob/master/pylops/LinearOperator.py>). This avoids having implementations that work for both (N, ) and (N, 1), which sometimes requires adding additional checks and squeezes within the implementation of _matvec and _rmatvec in situations where trailing dimensions in the input vector are not really bringing any benefit. - The above mentioned new feature has been currently added as a method to the PyLops LinearOperator class (and named todense in a similar fashion to scipy.sparse.xxx_matrix.todense methods). Doing so, we realized that every time two operators are summed/subtracted/multiplied… (pretty much when one of the overloading of scipy.sparse.linalg.LinearOperator is invoked), the returned object is not of LinearOperator type anymore but of the type of the private class in scipy.sparse.linalg.interface used to perform the requested operator (e.g, _SumLinearOperator). I wonder if returning an object of a ‘private’ type is actually something recommended to do in scipy (or even if this happens anywhere else in scipy). Again, in PyLops we need to overload the __add__ method (and similar methods) and make it into a thin wrappers of the scipy equivalent, returning an object of our parent class LinearOperator. This is to ensure that the returned object would maintain the additional properties and methods of our Linearoperator class, but also to avoid ‘leaking’ any of the private classes to the end users. I wonder if also the scipy API should perhaps avoid returning to end users objects of some private classes. This email is not intended to criticise any of current behaviours in scipy.sparse.linalg.LinearOperator, rather to gain an understanding of some of the decisions made by the core developers of the scipy.sparse.linalg.interface submodule, which may all have their own right to be the way they are today :) Looking forward to hearing from you. Best wishes, MR

4 4

SciPy 1.0 Paper is online in Nature Methods
by Tyler Reddy Feb. 3, 2020

Feb. 3, 2020

Hi all, The SciPy 1.0 journal article is now published online in Nature Methods: https://www.nature.com/articles/s41592-019-0686-2 Well done team! Best wishes, Tyler

6 5

Proposed addition to scipy.sparse.csgraph.maximum_flow
by mhwende Feb. 2, 2020

Feb. 2, 2020

Hi, recently I have made use of the maximum flow algorithm for CSR matrices, i.e., the scipy.sparse.csgraph.maximum_flow function. My test case was a non-sparse, bipartite network with lots of connections between the two node sets. The value of the maximum flow was small in comparison to the total number of nodes and edges in the graph. I wanted to use the scipy.sparse.csgraph module, however with a depth first search (DFS) instead of the breadth-first search that you have implemented. Therefore, I have implemented a new option to use DFS instead of BFS with your implementation. When measuring the execution time for a random graph, there was a signnificant speedup of the DFS procedure compared to the BFS variant. During these tests, I also refactored the two other functions that you call within the main csgraph.maximum_flow function, i.e., _add_reverse_edges and _make_edge_pointers. Depending on the input graph, I found that the time spent in this functions can be larger than the execution time for the actual Edmond-Karps algorithm. Below, you can find a table showing the measured execution times in seconds. Within the columns of this table, the execution time is broken down into parts as follows: a) Adding reverse edges (function _add_reverse_edges). b) Making edge pointers (function _make_edge_pointers). c) Actual max flow computation (function _edmond_karps) d) Total execution time (function maximum_flow). Rows of the table refer to the changes which were made to the implementation, where each measurement was performed with either DFS or BFS for the construction of augmenting paths: 1) Original implementation. 2) After the reimplementation of _make_edge_pointers. 3) After the reimplementation of _add_reverse_edges. Here are the execution times: | a) | b) | c) | d) --------+-----------+-----------+-----------+----------- 1. BFS | 1.29 s | 3.81 s | 8.93 s | 14.08 s 1. DFS | 1.29 s | 3.82 s | 0.50 s | 5.66 s --------+-----------+-----------+-----------+----------- 2. BFS | 1.54 s | 0.22 s | 8.94 s | 10.76 s 2. DFS | 1.33 s | 0.22 s | 0.50 s | 2.10 s --------+-----------+-----------+-----------+----------- 3. BFS | 0.16 s | 0.27 s | 9.01 s | 9.50 s 3. DFS | 0.16 s | 0.23 s | 0.52 s | 0.96 s By comparing the total execution times for BFS in 1d) and for DFS in 3d), my code changes have led to a speedup of about 14, for the network example which I have mentioned. Please let me know if would like to incorporate some or all of my changes into your code base. My changes (a single commit) can be found here: https://github.com/mhwende/scipy/commit/e7be2dc29364fe95f6fae20a4c0775716cd… Finally, here is the script which I have used for testing. The above timings have been produced with the command: python scipymaxflowtest 2000 2000 --dfs=0 This will create a graph with 4000 nodes and 4,000,000 edges, with randomly chosen zero or unit capacities. # File scipymaxflowtest.py import time import argparse import numpy as np import scipy.sparse as sp def scipymaxflowtest(layer_sizes, dfs): layer_sizes = np.array(layer_sizes, dtype = np.int) num_layers = len(layer_sizes) print("Layer sizes:", *layer_sizes) print("Number of layers:", num_layers) num_nodes = 2 + np.sum(layer_sizes) print("Number of nodes:", num_nodes) num_edges = layer_sizes[0] + layer_sizes[-1] for i in range(layer_sizes.size - 1): num_edges += layer_sizes[i] * layer_sizes[i + 1] print("Number of edges:", num_edges) #data = np.ones(num_edges, dtype = np.int) data = np.random.randint(0, 2, num_edges) indices = np.zeros(num_edges, dtype = np.int) indptr = np.zeros(num_nodes + 1, dtype = np.int) ptr = 0 for j in range(layer_sizes[0]): indices[ptr] = 2 + j ptr += 1 indptr[1] = ptr indptr[2] = ptr layer_sum = 0 for k in range(num_layers - 1): next_layer_sum = layer_sum + layer_sizes[k] for i in range(layer_sizes[k]): for j in range(layer_sizes[k + 1]): indices[ptr] = 2 + next_layer_sum + j ptr += 1 indptr[2 + layer_sum + i + 1] = ptr layer_sum = next_layer_sum for i in range(layer_sizes[-1]): indices[ptr] = 1 ptr += 1 indptr[2 + layer_sum + i + 1] = ptr indptr[num_nodes] = ptr # Check that we set as many entries in the index array as there are edges. assert ptr == num_edges # Create CSR matrix from data and index arrays. a = sp.csr_matrix((data, indices, indptr), shape = (num_nodes, num_nodes)) if num_nodes <= 12: print("A =\n{}".format(a.toarray())) t0 = time.time() flow_result = sp.csgraph.maximum_flow(a, 0, 1, dfs) t1 = time.time() flow_value = flow_result.flow_value print("Maximum flow: {} (took {:.6f} s)".format(flow_value, t1 - t0)) return flow_value if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument( "layer_sizes", type = int, nargs = "+", help = "Number of nodes on individual graph layers.") parser.add_argument( "--dfs", type = int, default = 0, help = "Whether to user depth first search.") args = parser.parse_args() np.random.seed(4891243) scipymaxflowtest(args.layer_sizes, args.dfs)

2 1

language labels in github issues/PRs?
by Lucas Roberts Feb. 2, 2020

Feb. 2, 2020

Hi scipy-dev, I was looking at my stack of emails for scipy and noticed some comments on: https://github.com/scipy/scipy/issues/3203# and that the c code is involved. In the past I've marked these types of emails for myself to flag for follow up (e.g. I'd like to work on them if still open in the future). Problem is that the stack of these flags gets too large in my inbox. I noticed there aren't tags for language in the scipy github. Scipy has C, C++, Cython, Fortran, and Python (I think that is all) used in various places. My question is would it be helpful to have language tags? We probably wouldnt' want a python tag but if an issue or a PR touches C/C++/Cython/fortran adding a label would help discoverability and exclusion. This might help searching of new contributors and for core team, e.g. you could search for issues touching c code (in my case) or exclude issues touching fortran (also my case, I don't know fortran). I didn't know how to exclude it Github and learned that it has some searching functionality that allows to exclude issues/PRs with certain tags: https://help.github.com/en/github/searching-for-information-on-github/searc… -- Sincerely, -Lucas

3 2