Mailman 3 February 2020 - NumPy-Discussion

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 161, Issue 14
by Inessa Pawson Feb. 14, 2020

Feb. 14, 2020

Documentation is imperative for project sustainability, yet often overlooked. Millions of NumPy stakeholders will benefit from this initiative. Melissa, Mars and Ralf, thank you for taking a lead on this! On Thu, Feb 13, 2020 at 3:05 AM <numpy-discussion-request(a)python.org> wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)python.org > > You can reach the person managing the list at > numpy-discussion-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > Today's Topics: > > 1. NEP 44 - Restructuring the NumPy Documentation (Melissa Mendon?a) > > > > ---------- Forwarded message ---------- > From: "Melissa Mendonça" <melissawm(a)gmail.com> > To: numpy-discussion(a)python.org > Cc: > Bcc: > Date: Wed, 12 Feb 2020 10:55:09 -0300 > Subject: [Numpy-discussion] NEP 44 - Restructuring the NumPy Documentation > Hi all, > > Please see the NEP below for a proposal to restructure the documentation > of NumPy. The main goal here is to make the documentation more visible and > organized, and also make contributions easier. > > Comments and feedback are welcome! > > > See https://github.com/numpy/numpy/pull/15554 for details. > > Best, > > Melissa > > ---- > > NEP 44 — Restructuring the NumPy Documentation > > Authors: Ralf Gommers, Melissa Mendonça, Mars Lee > Status: Draft > Type: Process > Created: 2020-02-11 > > Abstract > ====== > > This document proposes a restructuring of the NumPy Documentation, both in > form and content, with the goal of making it more organized and > discoverable for beginners and experienced users. > > Motivation and Scope > ================= > > See [here](numpy.org/devdocs) for the front page of the latest docs. The > organization is quite confusing and illogical (e.g. user and developer docs > are mixed). We propose the following: > > - Reorganizing the docs into the four categories mentioned in [1]; > - Creating dedicated sections for Tutorials and How-Tos, including > orientation on how to create new content; > - Adding an Explanations section for key concepts and techniques that > require deeper descriptions, some of which will be rearranged from the > Reference Guide. > > Usage and Impact > ============== > > The documentation is a fundamental part of any software project, > especially open source projects. In the case of NumPy, many beginners might > feel demotivated by the current structure of the documentation, since it is > difficult to discover what to learn (unless the user has a clear view of > what to look for in the Reference docs, which is not always the case). > > Looking at the results of a “NumPy Tutorial” search on any search engine > also gives an idea of the demand for this kind of content. Having official > high-level documentation written using up-to-date content and techniques > will certainly mean more users (and developers/contributors) are involved > in the NumPy community. > > Backward compatibility > ================== > > The restructuring will effectively demand a complete rewrite of links and > some of the current content. Input from the community will be useful for > identifying key links and pages that should not be broken. > > Detailed description > =============== > > As discussed in the article [1], there are four categories of doc content: > - Tutorials > - How-to guides > - Explanations > - Reference guide > > We propose to use those categories as the ones we use (for writing and > reviewing) whenever we add a new documentation section. > > The reasoning for this is that it is clearer both for > developers/documentation writers and to users where each information should > go, and the scope and tone of each document. For example, if explanations > are mixed with basic tutorials, beginners might be overwhelmed and > alienated. On the other hand, if the reference guide contains basic > how-tos, it might be difficult for experienced users to find the > information they need, quickly. > > Currently, there are many blogs and tutorials on the internet about NumPy > or using NumPy. One of the issues with this is that if users search for > this information and end up in an outdated (unofficial) tutorial before > they find the current official documentation, they end up creating content > that is confusing, especially for beginners. Having a better infrastructure > for the documentation also aims to solve this problem by giving users > high-level, up-to-date official documentation that can be easily updated. > > Status and ideas of each type of doc content > ------------------------------------------------------------ > > * Reference guide > > NumPy has a quite complete reference guide. All functions are documented, > most have examples, and most are cross-linked well with See Also sections. > Further improving the reference guide is incremental work that can be done > (and is being done) by many people. There are, however, many explanations > in the reference guide. These can be moved to a more dedicated Explanations > section on the docs. > > * How-to guides > > NumPy does not have many how-to’s. The subclassing and array ducktyping > section may be an example of a how-to. Others that could be added are: > - Parallelization (controlling BLAS multithreading with threadpoolctl, > using multiprocessing, random number generation, etc.) > - Storing and loading data (.npy/.npz format, text formats, Zarr, HDF5, > Bloscpack, etc.) > - Performance (memory layout, profiling, use with Numba, Cython, or > Pythran) > - Writing generic code that works with NumPy, Dask, CuPy, pydata/sparse, > etc. > > * Explanations > > There is a reasonable amount of content on fundamental NumPy concepts such > as indexing, vectorization, broadcasting, (g)ufuncs, and dtypes. This could > be organized better and clarified to ensure it’s really about explaining > the concepts and not mixed with tutorial or how-to like content. > > There are few explanations about anything other than those fundamental > NumPy concepts. > > Some examples of concepts that could be expanded: > - Copies vs. Views; > - BLAS and other linear algebra libraries; > - Fancy indexing. > > In addition, there are many explanations in the Reference Guide, which > should be moved to this new dedicated Explanations section. > > * Tutorials > > There’s a lot of scope for writing better tutorials. We have a new NumPy > for absolute beginners tutorial [3] (GSoD project of Anne Bonner). In > addition we need a number of tutorials addressing different levels of > experience with Python and NumPy. This could be done using engaging data > sets, ideas or stories. For example, curve fitting with polynomials and > functions in numpy.linalg could be done with the Keeling curve (decades > worth of CO2 concentration in air measurements) rather than with synthetic > random data. > > Ideas for tutorials (these capture the types of things that make sense, > they’re not necessarily the exact topics we propose to implement): > - Conway’s game of life with only NumPy (note: already in Nicolas > Rougier’s book) > - Using masked arrays to deal with missing data in time series measurements > - Using Fourier transforms to analyze the Keeling curve data, and > extrapolate it. > - Geospatial data (e.g. lat/lon/time to create maps for every year via a > stacked array, like gridMet data) > - Using text data and dtypes (e.g. use speeches from different people, > shape (n_speech, n_sentences, n_words)) > > The Preparing to Teach document [2] from the Software Carpentry Instructor > Training materials is a nice summary of how to write effective lesson plans > (and tutorials would be very similar). In addition to adding new tutorials, > we also propose a How to write a tutorial document, which would help users > contribute new high-quality content to the documentation. > > Data sets > ------------- > > Using interesting data in the NumPy docs requires giving all users access > to that data, either inside NumPy or in a separate package. The former is > not the best idea, since it’s hard to do without increasing the size of > NumPy significantly. Even for SciPy there has so far been no consensus on > this (see scipy PR 8707 on adding a new scipy.datasets subpackage). > > So we’ll aim for a new (pure Python) package, named numpy-datasets or > scipy-datasets or something similar. That package can take some lessons > from how, e.g., scikit-learn ships data sets. Small data sets can be > included in the repo, large data sets can be accessed via a downloader > class or function. > > Related Work > =========== > > Some examples of documentation organization in other projects: > - Documentation for Jupyter: https://jupyter.org/documentation > - Documentation for Python: https://docs.python.org/3/ > - Documentation for TensorFlow: https://www.tensorflow.org/learn > > These projects make the intended audience for each part of the > documentation more explicit, as well as previewing some of the content in > each section. > > Implementation > ============ > > Besides rewriting the current documentation to some extent, it would be > ideal to have a technical infrastructure that would allow more > contributions from the community. For example, if Jupyter Notebooks could > be submitted as-is as tutorials or How-Tos, this might create more > contributors and broaden the NumPy community. > > Similarly, if people could download some of the documentation in Notebook > format, this would certainly mean people would use less outdated material > for learning NumPy. > > It would also be interesting if the new structure for the documentation > makes translations easier. > > Currently, the documentation for NumPy can be confusing, especially for > beginners. Our proposal is to reorganize the docs in the following > structure: > > * For users: > - Absolute Beginners Tutorial > - main Tutorials section > - How To’s for common tasks with NumPy > - Reference Guide > - Explanations > - F2Py Guide > - Glossary > > * For developers/contributors: > - Contributor’s Guide > - Building and extending the documentation > - Benchmarking > - NumPy Enhancement Proposals > > * Meta information > - Reporting bugs > - Release Notes > - About NumPy > - License > > References and Footnotes > ==================== > > [1] What nobody tells you about documentation. > https://www.divio.com/blog/documentation/ > [2] Preparing to Teach (from the Software Carpentry Instructor Training > materials). > https://carpentries.github.io/instructor-training/15-lesson-study/index.html > [3] NumPy for absolute beginners Tutorial by Anne Bonner. > https://numpy.org/devdocs/user/absolute_beginners.html > > Copyright > ======== > > This document has been placed in the public domain. > > -- > Melissa Weber Mendonça > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > -- Every good wish, *Inessa Pawson* Executive Director Albus Code inessa(a)albuscode.org

1 0

Py-API: Deprecate `np.dtype(np.floating)` and similar dtype creation
by Sebastian Berg Feb. 14, 2020

Feb. 14, 2020

Hi all, In https://github.com/numpy/numpy/pull/15534 I would like to start deprecating creating dtypes from "abstract" scalar classes, such as: np.dtype(np.floating) is np.dtype(np.float64) While, at the same time, `isinstance(np.float32, np.floating)` is true. Right now `arr.astype(np.floating, copy=False)` and, more obviously, `arr.astype(np.dtype(np.floating), copy=False)` will cast a float32 array to float64. I think we should deprecate this, to consistently enable that in the future `dtype=np.floating` may choose to not cast a float32 array. Of course for the `astype` call the DeprecationWarning would be changed to a FutureWarning before we change the result value. A slight (but hopefully rare) annoyance is that `np.integer` might be used since it reads fairly well compared to `np.int_`. The large upstream packages such as SciPy or astropy seem to be clean in this regard, though (at least almost clean). Does anyone think this is a bad idea? To me these deprecations seem fairly straight forward, possibly flush out bugs/unintended behaviour, and necessary for consistent future behaviour. (More similar ones may have to follow). If there is some, but not much, hesitation, I can also add this to the NEP 41 draft. Although I currently feel it is the right thing to do even if we never had any new dtypes. - Sebastian

2 3

manylinux2010.
by Charles R Harris Feb. 14, 2020

Feb. 14, 2020

Hi All, Just a note that I've moved the nightly NumPy wheels builds to manylinux2010. Downstream projects testing against those wheels should check that they are using pip >= 19.0 in order to fetch those wheels. If there are any problems, please note them here. Chuck

2 2

NEP 38 - Universal SIMD intrinsics
by Matti Picus Feb. 13, 2020

Feb. 13, 2020

Together with Sayed Adel (cc) and Ralf, I am pleased to put the draft version of NEP 38 [0] up for discussion. As per NEP 0, this is the next step in the community accepting the approach layed out in the NEP. The NEP PR [1] has already garnered a fair amount of discussion about the viability of Universal SIMD Intrinsics, so I will try to capture some of that here as well. Abstract While compilers are getting better at using hardware-specific routines to optimize code, they sometimes do not produce optimal results. Also, we would like to be able to copy binary optimized C-extension modules from one machine to another with the same base architecture (x86, ARM, PowerPC) but with different capabilities without recompiling. We have a mechanism in the ufunc machinery to build alternative loops indexed by CPU feature name. At import (in InitOperators), the loop function that matches the run-time CPU info is chosen from the candidates.This NEP proposes a mechanism to build on that for many more features and architectures. The steps proposed are to: Establish a set of well-defined, architecture-agnostic, universal intrisics which capture features available across architectures. Capture these universal intrisics in a set of C macros and use the macros to build code paths for sets of features from the baseline up to the maximum set of features available on that architecture. Offer these as a limited number of compiled alternative code paths. At runtime, discover which CPU features are available, and choose from among the possible code paths accordingly. Motivation and Scope Traditionally NumPy has counted on the compilers to generate optimal code specifically for the target architecture. However few users today compile NumPy locally for their machines. Most use the binary packages which must provide run-time support for the lowest-common denominator CPU architecture. Thus NumPy cannot take advantage of more advanced features of their CPU processors, since they may not be available on all users’ systems. The ufunc machinery already has a loop-selection protocol based on dtypes, so it is easy to extend this to also select an optimal loop for specifically available CPU features at runtime. Traditionally, these features have been exposed through intrinsics which are compiler-specific instructions that map directly to assembly instructions. Recently there were discussions about the effectiveness of adding more intrinsics (e.g., `gh-11113`_ for AVX optimizations for floats). In the past, architecture-specific code was added to NumPy for fast avx512 routines in various ufuncs, using the mechanism described above to choose the best loop for the architecture. However the code is not generic and does not generalize to other architectures. Recently, OpenCV moved to using universal intrinsics in the Hardware Abstraction Layer (HAL) which provided a nice abstraction for common shared Single Instruction Multiple Data (SIMD) constructs. This NEP proposes a similar mechanism for NumPy. There are three stages to using the mechanism: - Infrastructure is provided in the code for abstract intrinsics. The ufunc machinery will be extended using sets of these abstract intrinsics, so that a single ufunc will be expressed as a set of loops, going from a minimal to a maximal set of possibly availabe intrinsics. - At compile time, compiler macros and CPU detection are used to turn the abstract intrinsics into concrete intrinsic calls. Any intrinsics not available on the platform, either because the CPU does not support them (and so cannot be tested) or because the abstract intrinsic does not have a parallel concrete intrinsic on the platform will not error, rather the corresponding loop will not be produced and added to the set of possibilities. - At runtime, the CPU detection code will further limit the set of loops available, and the optimal one will be chosen for the ufunc. The current NEP proposes only to use the runtime feature detection and optimal loop selection mechanism for ufuncs. Future NEPS may propose other uses for the proposed solution. Usage and Impact The end user will be able to get a list of intrinsics available for their platform and compiler. Optionally, the user may be able to specify which of the loops available at runtime will be used, perhaps via an environment variable to enable benchmarking the impact of the different loops. There should be no direct impact to naive end users, the results of all the loops should be identical to within a small number (1-3?) ULPs. On the other hand, users with more powerful machines should notice a significant performance boost. Binary releases - wheels on PyPI and conda packages The binaries released by this process will be larger since they include all possible loops for the architecture. Some packagers may prefer to limit the number of loops in order to limit the size of the binaries, we would hope they would still support a wide range of families of architectures. Note this problem already exists in the Intel MKL offering, where the binary package includes an extensive set of alternative shared objects (DLLs) for various CPU alternatives. Source builds See “Detailed Description” below. A source build where the packager knows details of the target machine could theoretically produce a smaller binary by choosing to compile only the loops needed by the target via command line arguments. How to run benchmarks to assess performance benefits Adding more code which use intrinsics will make the code harder to maintain. Therefore, such code should only be added if it yields a significant performance benefit. Assessing this performance benefit can be nontrivial. To aid with this, the implementation for this NEP will add a way to select which instruction sets can be used at runtime via environment variables. (name TBD). This ablility is critical for CI code verification. Diagnostics A new dictionary __cpu_features__ will be available to python. The keys are the available features, the value is a boolean whether the feature is available or not. Various new private C functions will be used internally to query available features. These might be exposed via specific c-extension modules for testing. Workflow for adding a new CPU architecture-specific optimization NumPy will always have a baseline C implementation for any code that may be a candidate for SIMD vectorization. If a contributor wants to add SIMD support for some architecture (typically the one of most interest to them), this is the proposed workflow: TODO (see https://github.com/numpy/numpy/pull/13516#issuecomment-558859638, needs to be worked out more) Reuse by other projects It would be nice if the universal intrinsics would be available to other libraries like SciPy or Astropy that also build ufuncs, but that is not an explicit goal of the first implementation of this NEP. ----------------------------------------------------------------------------------- My biased summary of select comments from the PR: (Raghuveer): A very similar SIMD library has been proposed for C++. Here is the link to the details: 1. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0214r8.pdf 2. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/n4808.pdf There is good discussion on the minimal/common set of instructions across architectures (which narrows down to loads, stores, arithmetic, compare, bitwise and shuffle instructions). Based on my developer experience so far, these instructions aren't by themselves enough to implement and optimize NumPy ufuncs. As i pointed out earlier, I think I would find it useful to learn the workflow of how to use instructions that don't fit in the Universal Intrinsic framework. (Raguveer) gave a well laid out table of currently proposed unversal intrinsics by use: load/store, reorder, operators, conversions, arithmatic and misc [2] which led to a long response from Sayed [3] with some sample code, demonstrating how more complex operations can be built up from the primitives. (catree) mentioned the Simd Library [4] and Halide [5] and asked about maintainability. (Ralf) responded [6] with concerns about competent developer bandwidth for code review. He also mentioned that our CI system currently supports all the architectures we are targeting (x86, aarch64, s390x, ppc64le) although some of these machines may not have the most advanced hardware to support the latest intrinsics. I apologize if my summary is not accurate, pleas correct any mistakes or misconceptions. ---------------------------------------------------------------------------------------- Barring complete rejection of the idea here, we will be pushing forward with PRs to implement this. Comments either on the mailing list or in those PRs are welcome. Matti [0] https://numpy.org/neps/nep-0038-SIMD-optimizations.html [1] https://github.com/numpy/numpy/pull/15228 [2] https://github.com/numpy/numpy/pull/15228#issuecomment-580479336 [3] https://github.com/numpy/numpy/pull/15228#issuecomment-580605718 [4] https://github.com/ermig1979/Simd [5] https://halide-lang.org [6] https://github.com/numpy/numpy/pull/15228#issuecomment-581029991

7 12

[ANN] 13ᵗʰ Advanced Scientific Programming in Python in Ghent, Belgium, 31 August—5 September, 2020
by Tiziano Zito Feb. 13, 2020

Feb. 13, 2020

13ᵗʰ Advanced Scientific Programming in Python ============================================== a Summer School by the ASPP faculty and the Ghent University https://aspp.school Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists have been trained to use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques and best practices which are standard in the industry, but especially tailored to the needs of a programming scientist. Lectures are devised to be interactive and to give the students enough time to acquire direct hands-on experience with the materials. Students will work in pairs throughout the school and will team up to practice the newly learned skills in a real programming project — an entertaining computer game. We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to become a standard tool for the programming scientist. This school is targeted at Master or PhD students and Post-docs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or R is absolutely required. Basic knowledge of Python and of a version control system such as git, subversion, mercurial, or bazaar is assumed. Participants without any prior experience with Python and/or git should work through the proposed introductory material before the course. We are striving hard to get a pool of students which is international and gender-balanced. Date & Location =============== 31 August–5 September, 2020. Ghent, Belgium. Application =========== You can apply online: https://aspp.school/wiki/applications Application deadline: 23:59 UTC, Sunday 24 May, 2020 There will be no deadline extension, so be sure to apply on time. Be sure to read the FAQ before applying: https://aspp.school/wiki/faq Participation is for free, i.e. no fee is charged! Accommodation in the student residence comes at no costs for participants. We are trying to arrange financial coverage for food expenses too, but this may not work. Participants however should take care of travel expenses by themselves. Program ======= • Version control with git and how to contribute to open source projects with GitHub • Best practices in data visualization • Testing and debugging scientific code • Advanced NumPy • Organizing, documenting, and distributing scientific code • Advanced scientific Python: context managers and generators • Writing parallel applications in Python • Profiling and speeding up scientific code with Cython and numba • Programming in teams Faculty ======= • Caterina Buizza, Personal Robotics Lab, Imperial College London UK • Lisa Schwetlick, Experimental and Biological Psychology, Universität Potsdam Germany • Nelle Varoquaux, CNRS, TIMC-IMAG, University Grenoble Alpes France • Nicolas P. Rougier, Inria Bordeaux Sud-Ouest, Institute of Neurodegenerative Disease, University of Bordeaux France • Pamela Hathway, Neural Reckoning, Imperial College London UK • Pietro Berkes, NAGRA Kudelski, Lausanne Switzerland • Rike-Benjamin Schuppner, Institute for Theoretical Biology, Humboldt-Universität zu Berlin Germany • Tiziano Zito, Department of Psychology, Humboldt-Universität zu Berlin Germany • Zbigniew Jędrzejewski-Szmek, Red Hat Inc., Warsaw Poland Organizers ========== Head of the organization for ASPP and responsible for the scientific program: • Tiziano Zito, Department of Psychology, Humboldt-Universität zu Berlin Germany Local team: • Nina Turk, Photonics Research Group, INTEC, Ghent University – imec Belgium • Freya Acar, Office for Data and Information, City of Ghent Belgium • Joan Juvert Institutional organizers: • Wim Bogaerts, Photonics Research Group, INTEC, Ghent University –imec Belgium • Sven Degroeve, VIB-UGent Center for Medical Biotechnology, Ghent Belgium • Jeroen Famaey, Department of Mathematics and Computer Science, University of Antwerp – iMinds Belgium • Bernard Manderick, Artificial Intelligence Lab, Vrije Universiteit Brussel Belgium Website: https://aspp.school Contact: info(a)aspp.school

1 0

[ANN] 13ᵗʰ Advanced Scientific Programming in Python in Ghent, Belgium, 31 August—5 September, 2020
by ASPP Feb. 13, 2020

Feb. 13, 2020

13ᵗʰ Advanced Scientific Programming in Python ============================================== a Summer School by the ASPP faculty and the Ghent University https://aspp.school Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists have been trained to use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques and best practices which are standard in the industry, but especially tailored to the needs of a programming scientist. Lectures are devised to be interactive and to give the students enough time to acquire direct hands-on experience with the materials. Students will work in pairs throughout the school and will team up to practice the newly learned skills in a real programming project — an entertaining computer game. We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to become a standard tool for the programming scientist. This school is targeted at Master or PhD students and Post-docs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or R is absolutely required. Basic knowledge of Python and of a version control system such as git, subversion, mercurial, or bazaar is assumed. Participants without any prior experience with Python and/or git should work through the proposed introductory material before the course. We are striving hard to get a pool of students which is international and gender-balanced. Date & Location =============== 31 August–5 September, 2020. Ghent, Belgium. Application =========== You can apply online: https://aspp.school/wiki/applications Application deadline: 23:59 UTC, Sunday 24 May, 2020 There will be no deadline extension, so be sure to apply on time. Be sure to read the FAQ before applying: https://aspp.school/wiki/faq Participation is for free, i.e. no fee is charged! Accommodation in the student residence comes at no costs for participants. We are trying to arrange financial coverage for food expenses too, but this may not work. Participants however should take care of travel expenses by themselves. Program ======= • Version control with git and how to contribute to open source projects with GitHub • Best practices in data visualization • Testing and debugging scientific code • Advanced NumPy • Organizing, documenting, and distributing scientific code • Advanced scientific Python: context managers and generators • Writing parallel applications in Python • Profiling and speeding up scientific code with Cython and numba • Programming in teams Faculty ======= • Caterina Buizza, Personal Robotics Lab, Imperial College London UK • Lisa Schwetlick, Experimental and Biological Psychology, Universität Potsdam Germany • Nelle Varoquaux, CNRS, TIMC-IMAG, University Grenoble Alpes France • Nicolas P. Rougier, Inria Bordeaux Sud-Ouest, Institute of Neurodegenerative Disease, University of Bordeaux France • Pamela Hathway, Neural Reckoning, Imperial College London UK • Pietro Berkes, NAGRA Kudelski, Lausanne Switzerland • Rike-Benjamin Schuppner, Institute for Theoretical Biology, Humboldt-Universität zu Berlin Germany • Tiziano Zito, Department of Psychology, Humboldt-Universität zu Berlin Germany • Zbigniew Jędrzejewski-Szmek, Red Hat Inc., Warsaw Poland Organizers ========== Head of the organization for ASPP and responsible for the scientific program: • Tiziano Zito, Department of Psychology, Humboldt-Universität zu Berlin Germany Local team: • Nina Turk, Photonics Research Group, INTEC, Ghent University – imec Belgium • Freya Acar, Office for Data and Information, City of Ghent Belgium • Joan Juvert Institutional organizers: • Wim Bogaerts, Photonics Research Group, INTEC, Ghent University –imec Belgium • Sven Degroeve, VIB-UGent Center for Medical Biotechnology, Ghent Belgium • Jeroen Famaey, Department of Mathematics and Computer Science, University of Antwerp – iMinds Belgium • Bernard Manderick, Artificial Intelligence Lab, Vrije Universiteit Brussel Belgium Website: https://aspp.school Contact: info(a)aspp.school

1 0

NEP 44 - Restructuring the NumPy Documentation
by Melissa Mendonça Feb. 12, 2020

Feb. 12, 2020

Hi all, Please see the NEP below for a proposal to restructure the documentation of NumPy. The main goal here is to make the documentation more visible and organized, and also make contributions easier. Comments and feedback are welcome! See https://github.com/numpy/numpy/pull/15554 for details. Best, Melissa ---- NEP 44 — Restructuring the NumPy Documentation Authors: Ralf Gommers, Melissa Mendonça, Mars Lee Status: Draft Type: Process Created: 2020-02-11 Abstract ====== This document proposes a restructuring of the NumPy Documentation, both in form and content, with the goal of making it more organized and discoverable for beginners and experienced users. Motivation and Scope ================= See [here](numpy.org/devdocs) for the front page of the latest docs. The organization is quite confusing and illogical (e.g. user and developer docs are mixed). We propose the following: - Reorganizing the docs into the four categories mentioned in [1]; - Creating dedicated sections for Tutorials and How-Tos, including orientation on how to create new content; - Adding an Explanations section for key concepts and techniques that require deeper descriptions, some of which will be rearranged from the Reference Guide. Usage and Impact ============== The documentation is a fundamental part of any software project, especially open source projects. In the case of NumPy, many beginners might feel demotivated by the current structure of the documentation, since it is difficult to discover what to learn (unless the user has a clear view of what to look for in the Reference docs, which is not always the case). Looking at the results of a “NumPy Tutorial” search on any search engine also gives an idea of the demand for this kind of content. Having official high-level documentation written using up-to-date content and techniques will certainly mean more users (and developers/contributors) are involved in the NumPy community. Backward compatibility ================== The restructuring will effectively demand a complete rewrite of links and some of the current content. Input from the community will be useful for identifying key links and pages that should not be broken. Detailed description =============== As discussed in the article [1], there are four categories of doc content: - Tutorials - How-to guides - Explanations - Reference guide We propose to use those categories as the ones we use (for writing and reviewing) whenever we add a new documentation section. The reasoning for this is that it is clearer both for developers/documentation writers and to users where each information should go, and the scope and tone of each document. For example, if explanations are mixed with basic tutorials, beginners might be overwhelmed and alienated. On the other hand, if the reference guide contains basic how-tos, it might be difficult for experienced users to find the information they need, quickly. Currently, there are many blogs and tutorials on the internet about NumPy or using NumPy. One of the issues with this is that if users search for this information and end up in an outdated (unofficial) tutorial before they find the current official documentation, they end up creating content that is confusing, especially for beginners. Having a better infrastructure for the documentation also aims to solve this problem by giving users high-level, up-to-date official documentation that can be easily updated. Status and ideas of each type of doc content ------------------------------------------------------------ * Reference guide NumPy has a quite complete reference guide. All functions are documented, most have examples, and most are cross-linked well with See Also sections. Further improving the reference guide is incremental work that can be done (and is being done) by many people. There are, however, many explanations in the reference guide. These can be moved to a more dedicated Explanations section on the docs. * How-to guides NumPy does not have many how-to’s. The subclassing and array ducktyping section may be an example of a how-to. Others that could be added are: - Parallelization (controlling BLAS multithreading with threadpoolctl, using multiprocessing, random number generation, etc.) - Storing and loading data (.npy/.npz format, text formats, Zarr, HDF5, Bloscpack, etc.) - Performance (memory layout, profiling, use with Numba, Cython, or Pythran) - Writing generic code that works with NumPy, Dask, CuPy, pydata/sparse, etc. * Explanations There is a reasonable amount of content on fundamental NumPy concepts such as indexing, vectorization, broadcasting, (g)ufuncs, and dtypes. This could be organized better and clarified to ensure it’s really about explaining the concepts and not mixed with tutorial or how-to like content. There are few explanations about anything other than those fundamental NumPy concepts. Some examples of concepts that could be expanded: - Copies vs. Views; - BLAS and other linear algebra libraries; - Fancy indexing. In addition, there are many explanations in the Reference Guide, which should be moved to this new dedicated Explanations section. * Tutorials There’s a lot of scope for writing better tutorials. We have a new NumPy for absolute beginners tutorial [3] (GSoD project of Anne Bonner). In addition we need a number of tutorials addressing different levels of experience with Python and NumPy. This could be done using engaging data sets, ideas or stories. For example, curve fitting with polynomials and functions in numpy.linalg could be done with the Keeling curve (decades worth of CO2 concentration in air measurements) rather than with synthetic random data. Ideas for tutorials (these capture the types of things that make sense, they’re not necessarily the exact topics we propose to implement): - Conway’s game of life with only NumPy (note: already in Nicolas Rougier’s book) - Using masked arrays to deal with missing data in time series measurements - Using Fourier transforms to analyze the Keeling curve data, and extrapolate it. - Geospatial data (e.g. lat/lon/time to create maps for every year via a stacked array, like gridMet data) - Using text data and dtypes (e.g. use speeches from different people, shape (n_speech, n_sentences, n_words)) The Preparing to Teach document [2] from the Software Carpentry Instructor Training materials is a nice summary of how to write effective lesson plans (and tutorials would be very similar). In addition to adding new tutorials, we also propose a How to write a tutorial document, which would help users contribute new high-quality content to the documentation. Data sets ------------- Using interesting data in the NumPy docs requires giving all users access to that data, either inside NumPy or in a separate package. The former is not the best idea, since it’s hard to do without increasing the size of NumPy significantly. Even for SciPy there has so far been no consensus on this (see scipy PR 8707 on adding a new scipy.datasets subpackage). So we’ll aim for a new (pure Python) package, named numpy-datasets or scipy-datasets or something similar. That package can take some lessons from how, e.g., scikit-learn ships data sets. Small data sets can be included in the repo, large data sets can be accessed via a downloader class or function. Related Work =========== Some examples of documentation organization in other projects: - Documentation for Jupyter: https://jupyter.org/documentation - Documentation for Python: https://docs.python.org/3/ - Documentation for TensorFlow: https://www.tensorflow.org/learn These projects make the intended audience for each part of the documentation more explicit, as well as previewing some of the content in each section. Implementation ============ Besides rewriting the current documentation to some extent, it would be ideal to have a technical infrastructure that would allow more contributions from the community. For example, if Jupyter Notebooks could be submitted as-is as tutorials or How-Tos, this might create more contributors and broaden the NumPy community. Similarly, if people could download some of the documentation in Notebook format, this would certainly mean people would use less outdated material for learning NumPy. It would also be interesting if the new structure for the documentation makes translations easier. Currently, the documentation for NumPy can be confusing, especially for beginners. Our proposal is to reorganize the docs in the following structure: * For users: - Absolute Beginners Tutorial - main Tutorials section - How To’s for common tasks with NumPy - Reference Guide - Explanations - F2Py Guide - Glossary * For developers/contributors: - Contributor’s Guide - Building and extending the documentation - Benchmarking - NumPy Enhancement Proposals * Meta information - Reporting bugs - Release Notes - About NumPy - License References and Footnotes ==================== [1] What nobody tells you about documentation. https://www.divio.com/blog/documentation/ [2] Preparing to Teach (from the Software Carpentry Instructor Training materials). https://carpentries.github.io/instructor-training/15-lesson-study/index.html [3] NumPy for absolute beginners Tutorial by Anne Bonner. https://numpy.org/devdocs/user/absolute_beginners.html Copyright ======== This document has been placed in the public domain. -- Melissa Weber Mendonça

1 0

Re: [Numpy-discussion] Proposal - extend histograms api to allow uneven bins
by Ross Barnowski Feb. 11, 2020

Feb. 11, 2020

Just a few thoughts re: the changes proposed in https://github.com/numpy/numpy/pull/14278 1. Though the PR is limited to the 'auto' kwarg, the issue of potential memory problems for the automated binning methods is a more general one (e.g. #15332 <https://github.com/numpy/numpy/issues/15332>). 2. The main concern that jumps out to me is downstream users who are relying on the implicit assumption of regular binning. This is of course bad practice and makes even less sense when using one of the bin estimators, so I'm not sure how big of a concern it is. However, there is likely downstream user code that relies on the regular binning assumption, especially since, as far as I know, NumPy has never implemented binning techniques that return irregular bins. 3. The astropy project have at least one estimator that returns irregular bins <https://docs.astropy.org/en/stable/visualization/histogram.html#>. I checked for issues <https://github.com/astropy/astropy/issues?utf8=%E2%9C%93&q=is%3Aissue+histo…> related to irregular binning: though they have many of the same problems with the automatic bin estimators (i.e. memory problems for inputs with outliers), I didn't see anything specifically related to irregular binning I just wanted to add my two cents. The binning-data-with-outliers problem is very common in high-resolution spectroscopy, and I have seen practitioners rely on the assumption of regular binning (e.g. divide the `range` by the number of bins) to specify bin centers even though this is not the right way to do things. Thanks for taking the time to write up your work! On Mon, Feb 10, 2020 at 10:53 PM <numpy-discussion-request(a)python.org> wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)python.org > > You can reach the person managing the list at > numpy-discussion-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Proposal - extend histograms api to allow uneven bins > (Alexander Reeves) > 2. Re: NEP 38 - Universal SIMD intrinsics (Ralf Gommers) > 3. Re: NEP 38 - Universal SIMD intrinsics (Matti Picus) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 10 Feb 2020 18:07:40 -0800 > From: Alexander Reeves <lxndr.rvs(a)gmail.com> > To: numpy-discussion(a)python.org > Subject: [Numpy-discussion] Proposal - extend histograms api to allow > uneven bins > Message-ID: > <CABeAeRyiVt4RJP2ew7= > C4c4itJO3WZptdD6yrhWV6NgUoqT_mQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Greetings, > > I have a PR that warrants discussion according to @seberg. See > https://github.com/numpy/numpy/pull/14278. > > It is an enhancement that fixes a bug. The original bug is that when using > the fd estimator on a dataset with small inter-quartile range and large > outliers, the current codebase produces more bins than memory allows. There > are several related bug reports (see #11879, #10297, #8203). > > In terms of scope, I restricted my changes to conditions where > np.histogram(bins='auto') defaults to the 'fd'. For the actual fix, I > actually enhanced the API. I used a suggestion from @eric-wieser to merge > empty histogram bins. In practice this solves the outsized bins issue. > > However @seberg is concerned that extending the API in this way may not be > the way to go. For example, if you use "auto" once, and then re-use the > bins, the uneven bins may not be what you want. > > Furthermore @eric-wieser is concerned that there may be a floating-point > devil in the details. He advocates using the hypothesis testing package to > increase our confidence that the current implementation adequately handles > corner cases. > > I would like to do my part in improving the code base. I don't have strong > opinions but I have to admit that I would like to eventually make a PR that > resolves these bugs. This has been a PR half a year in the making after > all. > > Thoughts? > > -areeves87 >

1 0

NumPy Development Meeting - Triage Focus
by Sebastian Berg Feb. 11, 2020

Feb. 11, 2020

Hi all, Our bi-weekly triage-focused NumPy development meeting is tomorrow (Wednesday, Februrary 12) at 11 am Pacific Time. Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg I encourage everyone to notify us of issues or PRs that you feel should be prioritized or simply discussed briefly. Just comment on it so we can label it, or add your PR/issue to this weeks topics for discussion. Best regards, Sebastian

1 0

Proposal - extend histograms api to allow uneven bins
by Alexander Reeves Feb. 11, 2020

Feb. 11, 2020

Greetings, I have a PR that warrants discussion according to @seberg. See https://github.com/numpy/numpy/pull/14278. It is an enhancement that fixes a bug. The original bug is that when using the fd estimator on a dataset with small inter-quartile range and large outliers, the current codebase produces more bins than memory allows. There are several related bug reports (see #11879, #10297, #8203). In terms of scope, I restricted my changes to conditions where np.histogram(bins='auto') defaults to the 'fd'. For the actual fix, I actually enhanced the API. I used a suggestion from @eric-wieser to merge empty histogram bins. In practice this solves the outsized bins issue. However @seberg is concerned that extending the API in this way may not be the way to go. For example, if you use "auto" once, and then re-use the bins, the uneven bins may not be what you want. Furthermore @eric-wieser is concerned that there may be a floating-point devil in the details. He advocates using the hypothesis testing package to increase our confidence that the current implementation adequately handles corner cases. I would like to do my part in improving the code base. I don't have strong opinions but I have to admit that I would like to eventually make a PR that resolves these bugs. This has been a PR half a year in the making after all. Thoughts? -areeves87

1 0