Mailman 3 October 2019 - NumPy-Discussion

SciPy user documentation survey - please participate
by Ralf Gommers 12 Oct '19

12 Oct '19

Hi everyone, Maja, our technical writer for Season of Docs for SciPy, created a survey specifically about how users use the SciPy docs and what they would like to see improved. She sent out the link to scipy-dev before and it was shared on Twitter. We've received some really valuable responses, and would love to get more. If you're a SciPy user, this is a very easy way to help the project! https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc1… Apologies for the cross-post. We do a survey about once a decade and Maja is putting in a lot of work creating and analyzing the survey, so it's worth some more exposure and a couple of minutes of your time! Cheers, Ralf

1 0

np.genfromtxt StopIteration Error
by Stephen P. Molnar 11 Oct '19

11 Oct '19

I have been fighting with the genfromtxt function in numpy for a while now and am trying a slightly different approach. Here is the code: > > import os > import glob > import numpy as np > > fileList = [] > filesList = [] > > for files in glob.glob("*.log"): > fileName, fileExtension = os.path.splitext(files) > fileList.append(fileName) > filesList.append(files) > > print('fileList = ', fileList) > print('filesList = ', filesList) > > fname = filesList > print('fname = ', fname) > data = np.genfromtxt(fname, usecols=(1), skip_header=27, > skip_footer=1, encoding=None) > print(data) > > np.savetxt('fileList.dG', data, fmt='%12.9f', header='${d}') > print(data.dG) I am using the Spyder IDE which has a variable explorer which shows: filesList = ['C-VX3.log', '18-7.log', '14-7.log', '15-7.log'] fileList = ['C-VX3', '18-7', '14-7', '15-7'] so the lists that genfromtxt needs are being generated. Goggling 'numpy genfromtxt stopiteration error' does not seem to address this problem. At least, I didn't find plaything that I thought applied. I would greatly appreciate some assistance here. Thanks is advance. -- Stephen P. Molnar, Ph.D. www.molecular-modeling.net 614.312.7528 (c) Skype: smolnar1

3 4

Problem with np.savetxt
by Stephen P. Molnar 10 Oct '19

10 Oct '19

I am embarrassed to be asking this question, but I have exhausted Google at this point . I have a number of identically formatted text files from which I want to extract data, as an example (hopefully, putting these in as quotes will persevere the format): > ======================================================================= > PSOVina version 2.0 > Giotto H. K. Tai & Shirley W. I. Siu > > Computational Biology and Bioinformatics Lab > University of Macau > > Visit http://cbbio.cis.umac.mo for more information. > > PSOVina was developed based on the framework of AutoDock Vina. > > For more information about Vina, please visit http://vina.scripps.edu. > > ======================================================================= > > Output will be 13-7_out.pdbqt > Reading input ... done. > Setting up the scoring function ... done. > Analyzing the binding site ... done. > Using random seed: 1828390527 > Performing search ... done. > > Refining results ... done. > > mode | affinity | dist from best mode > | (kcal/mol) | rmsd l.b.| rmsd u.b. > -----+------------+----------+---------- > 1 -8.862004149 0.000 0.000 > 2 -8.403522829 2.992 6.553 > 3 -8.401384636 2.707 5.220 > 4 -7.886402037 4.907 6.862 > 5 -7.845519031 3.233 5.915 > 6 -7.837434227 3.954 5.641 > 7 -7.834584887 3.188 7.294 > 8 -7.694395765 3.746 7.553 > 9 -7.691211177 3.536 5.745 > 10 -7.670759445 3.698 7.587 > 11 -7.661882758 4.882 7.044 > 12 -7.636280303 2.347 3.284 > 13 -7.635788052 3.511 6.250 > 14 -7.611175249 2.427 3.449 > 15 -7.586368357 2.142 2.864 > 16 -7.531307666 2.976 4.980 > 17 -7.520501084 3.085 5.775 > 18 -7.512906514 4.220 7.672 > 19 -7.307403528 3.240 4.354 > 20 -7.256063348 3.694 7.252 > Writing output ... done. At this point, my python script consists of only the following: > #!/usr/bin/env python3 > # -*- coding: utf-8 -*- > """ > > Created on Tue Sep 24 07:51:11 2019 > > """ > import numpy as np > > data = [] > > data = np.genfromtxt("13-7.log", usecols=(1), dtype=None, > skip_header=27, skip_footer=1, encoding=None) > > print(data) > > np.savetxt('13-7', [data], fmt='%15.9f', header='13-7') The problem lies in tfe np.savetxt line, on execution I get: > runfile('/home/comp/Apps/Models/1-PhosphorusLigands/CombinedLigands/MOL/Docking/VINA20/SmallSet/DeltaGTable_V_s.py', > wdir='/home/comp/Apps/Models/1-PhosphorusLigands/CombinedLigands/MOL/Docking/VINA20/SmallSet', > current_namespace=True) > ['-8.839713733' '-8.743377250' '-8.151051167' '-8.090452911' > '-7.967494477' '-7.854890056' '-7.757417879' '-7.741557490' > '-7.643885488' '-7.611595767' '-7.507605524' '-7.413920814' > '-7.389408331' '-7.384446364' '-7.374206276' '-7.368808179' > '-7.346641418' '-7.325037898' '-7.309614787' '-7.113209147'] > Traceback (most recent call last): > > File > "/home/comp/Apps/Models/1-PhosphorusLigands/CombinedLigands/MOL/Docking/VINA20/SmallSet/DeltaGTable_V_s.py", > line 16, in <module> > np.savetxt('13-7', [data], fmt='%16.9f', header='13-7') > > File "<__array_function__ internals>", line 6, in savetxt > > File > "/home/comp/Apps/Miniconda3/lib/python3.7/site-packages/numpy/lib/npyio.py", > line 1438, in savetxt > % (str(X.dtype), format)) > > TypeError: Mismatch between array dtype ('<U12') and format specifier > ('%16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f > %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f %16.9f > %16.9f') The data is in the data file, but the only entry in '13-7', the saved file, is the label. Obviously, the error is in the format argument. Help will be much appreciated. Thanks in advance. -- Stephen P. Molnar, Ph.D. www.molecular-modeling.net 614.312.7528 (c) Skype: smolnar1

4 8

NEP 31 — Context-local and global overrides of the NumPy API
by Hameer Abbasi 10 Oct '19

10 Oct '19

Hello all, It was recently brought to my attention that my mails to NumPy-discussion were probably going into the spam folder for many people, so here I am trying from another email. Probably Google trying to force people onto their products as usual. 😉 Me, Ralf Gommers and Peter Bell (both cc’d) have come up with a proposal on how to solve the array creation and duck array problems. The solution is outlined in NEP-31, currently in the form of a PR, [1] Following the high level discussion in NEP-22. [2] It would be nice to get some feedback. Full-text of the NEP: ============================================================ NEP 31 — Context-local and global overrides of the NumPy API ============================================================ :Author: Hameer Abbasi <habbasi(a)quansight.com><mailto:habbasi@quansight.com> :Author: Ralf Gommers <rgommers(a)quansight.com><mailto:rgommers@quansight.com> :Author: Peter Bell <peterbell10(a)live.co.uk><mailto:peterbell10@live.co.uk> :Status: Draft :Type: Standards Track :Created: 2019-08-22 Abstract -------- This NEP proposes to make all of NumPy's public API overridable via an extensible backend mechanism, using a library called ``uarray`` `[1]`_ ``uarray`` provides global and context-local overrides, as well as a dispatch mechanism similar to NEP-18 `[2]`_. First experiences with ``__array_function__`` show that it is necessary to be able to override NumPy functions that *do not take an array-like argument*, and hence aren't overridable via ``__array_function__``. The most pressing need is array creation and coercion functions - see e.g. NEP-30 `[9]`_. This NEP proposes to allow, in an opt-in fashion, overriding any part of the NumPy API. It is intended as a comprehensive resolution to NEP-22 `[3]`_, and obviates the need to add an ever-growing list of new protocols for each new type of function or object that needs to become overridable. Motivation and Scope -------------------- The motivation behind ``uarray`` is manyfold: First, there have been several attempts to allow dispatch of parts of the NumPy API, including (most prominently), the ``__array_ufunc__`` protocol in NEP-13 `[4]`_, and the ``__array_function__`` protocol in NEP-18 `[2]`_, but this has shown the need for further protocols to be developed, including a protocol for coercion (see `[5]`_). The reasons these overrides are needed have been extensively discussed in the references, and this NEP will not attempt to go into the details of why these are needed. Another pain point requiring yet another protocol is the duck-array protocol (see `[9]`_). This NEP takes a more holistic approach: It assumes that there are parts of the API that need to be overridable, and that these will grow over time. It provides a general framework and a mechanism to avoid a design of a new protocol each time this is required. This NEP proposes the following: That ``unumpy`` `[8]`_ becomes the recommended override mechanism for the parts of the NumPy API not yet covered by ``__array_function__`` or ``__array_ufunc__``, and that ``uarray`` is vendored into a new namespace within NumPy to give users and downstream dependencies access to these overrides. This vendoring mechanism is similar to what SciPy decided to do for making ``scipy.fft`` overridable (see `[10]`_). Detailed description -------------------- **Note:** *This section will not attempt to explain the specifics or the mechanism of ``uarray``, that is explained in the ``uarray`` documentation.* `[1]`_ *However, the NumPy community will have input into the design of ``uarray``, and any backward-incompatible changes will be discussed on the mailing list.* The way we propose the overrides will be used by end users is:: import numpy.overridable as np with np.set_backend(backend): x = np.asarray(my_array, dtype=dtype) And a library that implements a NumPy-like API will use it in the following manner (as an example):: import numpy.overridable as np _ua_implementations = {} __ua_domain__ = "numpy" def __ua_function__(func, args, kwargs): fn = _ua_implementations.get(func, None) return fn(*args, **kwargs) if fn is not None else NotImplemented def implements(ua_func): def inner(func): _ua_implementations[ua_func] = func return func return inner @implements(np.asarray) def asarray(a, dtype=None, order=None): # Code here # Either this method or __ua_convert__ must # return NotImplemented for unsupported types, # Or they shouldn't be marked as dispatchable. # Provides a default implementation for ones and zeros. @implements(np.full) def full(shape, fill_value, dtype=None, order='C'): # Code here The only change this NEP proposes at its acceptance, is to make ``unumpy`` the officially recommended way to override NumPy. ``unumpy`` will remain a separate repository/package (which we propose to vendor to avoid a hard dependency, and use the separate ``unumpy`` package only if it is installed) rather than depend on for the time being), and will be developed primarily with the input of duck-array authors and secondarily, custom dtype authors, via the usual GitHub workflow. There are a few reasons for this: * Faster iteration in the case of bugs or issues. * Faster design changes, in the case of needed functionality. * ``unumpy`` will work with older versions of NumPy as well. * The user and library author opt-in to the override process, rather than breakages happening when it is least expected. In simple terms, bugs in ``unumpy`` mean that ``numpy`` remains unaffected. Advantanges of ``unumpy`` over other solutions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``unumpy`` offers a number of advantanges over the approach of defining a new protocol for every problem encountered: Whenever there is something requiring an override, ``unumpy`` will be able to offer a unified API with very minor changes. For example: * ``ufunc`` objects can be overridden via their ``__call__``, ``reduce`` and other methods. * ``dtype`` objects can be overridden via the dispatch/backend mechanism, going as far as to allow ``np.float32`` et. al. to be overridden by overriding ``__get__``. * Other functions can be overridden in a similar fashion. * ``np.asduckarray`` goes away, and becomes ``np.asarray`` with a backend set. * The same holds for array creation functions such as ``np.zeros``, ``np.empty`` and so on. This also holds for the future: Making something overridable would require only minor changes to ``unumpy``. Another promise ``unumpy`` holds is one of default implementations. Default implementations can be provided for any multimethod, in terms of others. This allows one to override a large part of the NumPy API by defining only a small part of it. This is to ease the creation of new duck-arrays, by providing default implementations of many functions that can be easily expressed in terms of others, as well as a repository of utility functions that help in the implementation of duck-arrays that most duck-arrays would require. The last benefit is a clear way to coerce to a given backend, and a protocol for coercing not only arrays, but also ``dtype`` objects and ``ufunc`` objects with similar ones from other libraries. This is due to the existence of actual, third party dtype packages, and their desire to blend into the NumPy ecosystem (see `[6]`_). This is a separate issue compared to the C-level dtype redesign proposed in `[7]`_, it's about allowing third-party dtype implementations to work with NumPy, much like third-party array implementations. Mixing NumPy and ``unumpy`` in the same file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Normally, one would only want to import only one of ``unumpy`` or ``numpy``, you would import it as ``np`` for familiarity. However, there may be situations where one wishes to mix NumPy and the overrides, and there are a few ways to do this, depending on the user's style:: import numpy.overridable as unumpy import numpy as np or:: import numpy as np # Use unumpy via np.overridable Related Work ------------ Previous override mechanisms ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * NEP-18, the ``__array_function__`` protocol. `[2]`_ * NEP-13, the ``__array_ufunc__`` protocol. `[3]`_ Existing NumPy-like array implementations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Dask: https://dask.org/ * CuPy: https://cupy.chainer.org/ * PyData/Sparse: https://sparse.pydata.org/ * Xnd: https://xnd.readthedocs.io/ * Astropy's Quantity: https://docs.astropy.org/en/stable/units/ Existing and potential consumers of alternative arrays ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Dask: https://dask.org/ * scikit-learn: https://scikit-learn.org/ * Xarray: https://xarray.pydata.org/ * TensorLy: http://tensorly.org/ Existing alternate dtype implementations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * ``ndtypes``: https://ndtypes.readthedocs.io/en/latest/ * Datashape: https://datashape.readthedocs.io * Plum: https://plum-py.readthedocs.io/ Implementation -------------- The implementation of this NEP will require the following steps: * Implementation of ``uarray`` multimethods corresponding to the NumPy API, including classes for overriding ``dtype``, ``ufunc`` and ``array`` objects, in the ``unumpy`` repository. * Moving backends from ``unumpy`` into the respective array libraries. Backward compatibility ---------------------- There are no backward incompatible changes proposed in this NEP. Alternatives ------------ The current alternative to this problem is NEP-30 plus adding more protocols (not yet specified) in addition to it. Even then, some parts of the NumPy API will remain non-overridable, so it's a partial alternative. The main alternative to vendoring ``unumpy`` is to simply move it into NumPy completely and not distribute it as a separate package. This would also achieve the proposed goals, however we prefer to keep it a separate package for now, for reasons already stated above. Discussion ---------- * ``uarray`` blogpost: https://labs.quansight.org/blog/2019/07/uarray-update-api-changes-overhead-… * The discussion section of NEP-18: https://numpy.org/neps/nep-0018-array-function-protocol.html#discussion * NEP-22: https://numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html * Dask issue #4462: https://github.com/dask/dask/issues/4462 * PR #13046: https://github.com/numpy/numpy/pull/13046 * Dask issue #4883: https://github.com/dask/dask/issues/4883 * Issue #13831: https://github.com/numpy/numpy/issues/13831 * Discussion PR 1: https://github.com/hameerabbasi/numpy/pull/3 * Discussion PR 2: https://github.com/hameerabbasi/numpy/pull/4 References and Footnotes ------------------------ .. _[1]: [1] uarray, A general dispatch mechanism for Python: https://uarray.readthedocs.io .. _[2]: [2] NEP 18 — A dispatch mechanism for NumPy’s high level array functions: https://numpy.org/neps/nep-0018-array-function-protocol.html .. _[3]: [3] NEP 22 — Duck typing for NumPy arrays – high level overview: https://numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html .. _[4]: [4] NEP 13 — A Mechanism for Overriding Ufuncs: https://numpy.org/neps/nep-0013-ufunc-overrides.html .. _[5]: [5] Reply to Adding to the non-dispatched implementation of NumPy methods: http://numpy-discussion.10968.n7.nabble.com/Adding-to-the-non-dispatched-im… .. _[6]: [6] Custom Dtype/Units discussion: http://numpy-discussion.10968.n7.nabble.com/Custom-Dtype-Units-discussion-t… .. _[7]: [7] The epic dtype cleanup plan: https://github.com/numpy/numpy/issues/2899 .. _[8]: [8] unumpy: NumPy, but implementation-independent: https://unumpy.readthedocs.io .. _[9]: [9] NEP 30 — Duck Typing for NumPy Arrays - Implementation: https://www.numpy.org/neps/nep-0030-duck-array-protocol.html .. _[10]: [10] http://scipy.github.io/devdocs/fft.html#backend-control Copyright --------- This document has been placed in the public domain. Best regards, Hameer Abbasi [1] https://github.com/numpy/numpy/pull/14389 [2] https://numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html

11 46

NumPy Community Meeting Wednesday, Oct. 9
by Sebastian Berg 08 Oct '19

08 Oct '19

Hi all, There will be a NumPy Community meeting Wednesday October 9 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

Python 3.8 numpy wheels
by Charles R Harris 08 Oct '19

08 Oct '19

Hi All, Thanks to the work of Matti Pincus and Matthew Brett, manylinux1 numpy wheels are now available for testing. Chuck

2 2

Np.genfromtxt Problem
by Stephen P. Molnar 05 Oct '19

05 Oct '19

I have a snippet of code #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ Created on Tue Sep 24 07:51:11 2019 """ import numpy as np files = [] data = np.genfromtxt(files, usecols=(3), dtype=None, skip_header=8, skip_footer=1, encoding=None) print(data) If file is a single file the code generates the data that I want. However I have a list of files that I want to process. According to numpy.genfromtxt fname can be a "File, filename, list, or generator to read." If I use [13-7a_apo-1acl.RMSD 13-7_apo-1acl.RMSD 14-7_apo-1acl.RMSD 15-7_apo-1acl.RMSD 17-7_apo-1acl.RMSD ] get the error: runfile('/home/comp/Apps/Models/1-PhosphorusLigands/CombinedLigands/MOL/Docking/Results/RMSDTable/Test/DeltaGTable_s.py', wdir='/home/comp/Apps/Models/1-PhosphorusLigands/CombinedLigands/MOL/Docking/Results/RMSDTable/Test', current_namespace=True) Traceback (most recent call last): File "/home/comp/Apps/Models/1-PhosphorusLigands/CombinedLigands/MOL/Docking/Results/RMSDTable/Test/DeltaGTable_s.py", line 12, in <module> data = np.genfromtxt(files, usecols=(3), dtype=None, skip_header=8, skip_footer=1, encoding=None) File "/home/comp/Apps/Miniconda3/lib/python3.7/site-packages/numpy/lib/npyio.py", line 1762, in genfromtxt next(fhd) StopIteration I have tried very combination of search terms that I can think of in order to find an example of how to make this work without success. How can I make this work? Thanks in advance. -- Stephen P. Molnar, Ph.D. www.molecular-modeling.net 614.312.7528 (c) Skype: smolnar1

4 4

Accepting NEP 29 — Recommend Python and Numpy version support as a community policy standard
by Sebastian Berg 03 Oct '19

03 Oct '19

Hi all, we propose formally accepting the NumPy enhancement proposal 29: "Recommend Python and Numpy version support as a community policy standard" available at: https://numpy.org/neps/nep-0029-deprecation_policy.html If there are no objections within a week it may be accepted. This proposal is a recommendation to the larger ecosystem and thus should receive attention and acceptance from a wide audience. However, lets try to keep discussions on the NumPy mailing list. The most important points from the Abstract and Implementation sections are: "This NEP recommends that all projects across the Scientific Python ecosystem adopt a common “time window-based” policy for support of Python and NumPy versions. Standardizing a recommendation for project support of minimum Python and NumPy versions will improve downstream project planning. …" and: "We suggest that all projects adopt the following language into their development guidelines: This project supports: * All minor versions of Python released 42 months prior to the project, and at minimum the two latest minor versions. * All minor versions of numpy released in the 24 months prior to the project, and at minimum the last thee minor versions." For the full text, please refer to the link above. Cheers, Sebastian

2 1

NumPy Community Meeting Wednesday, Oct. 2
by Sebastian Berg 01 Oct '19

01 Oct '19

Hi all, There will be a NumPy Community meeting Wednesday September 25 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

NumPy-Discussion October 2019