Mailman 3 April 2021 - NumPy-Discussion

GSoD'21 Ideas Disussion
by Mukulika Pahari April 19, 2021

April 19, 2021

Hello everyone! I am Mukulika Pahari, a Computer Engineering student from India. I wanted to implement the following ideas for Google Summer of Docs 2021 if given the opportunity. I have referred to both- the GSoD proposal <https://github.com/numpy/numpy/wiki/Google-Season-of-Docs-2021-Project-Idea…> and the NEP 44 <https://numpy.org/neps/nep-0044-restructuring-numpy-docs.html> proposal. Please give me feedback about the ideas! 1. Reorganising contents of the documentation into Reference Guide, How-Tos, Tutorials and Explanations as per structure proposed in NEP 44 <https://numpy.org/neps/nep-0044-restructuring-numpy-docs.html>. - Auditing existing documentation - Clearing misplaced content for example Explanations in Reference Guide, How-Tos in Explanations etc. - Establishing distinct Reference, How-Tos, Tutorials and Explanations sections with crosslinking where required 1. Reorganising landing page of NumPy docs <https://numpy.org/devdocs/>. - Moving the documentation structure to the left sidebar - Having NumPy Quickstart as the first thing people see when they land on Documentation 1. Writing new must-have tutorials (based on most-searched tutorials on Google). - “How to write a tutorial” guide - 3 Beginner Tutorials - 3 Intermediate Tutorials - 3 Advanced Tutorials 1. Writing How-Tos based on most-used functions, most asked doubts on StackOverflow, etc. 2. Revamping the User Guide. - Updating out-of-date references and refactoring content to the latest best practices - Adding non-textual images or graphics to enhance the textual explanations - Removing duplication to improve searchability I have proposed this work keeping in mind 30ish weeks of work including a few weeks for becoming familiar with the organisation and ironing out details like exactly which tutorials and how-tos to write. Please let me know if I can aim to achieve more in the timeframe. My experience with NumPy is currently limited to one data analysis project but I would love to learn more about its applications while restructuring and developing its docs! Thank you for your time.

2 1

timing of abs(complex)
by fiolj April 19, 2021

April 19, 2021

Hi all, I recently noted that absolute values of complex arrays are very slow (about a factor of five compared to some straightforward implementations). I would like to understand the origin, but could not trace the code. Please, consider these timings: In [67]: z = np.random.random(10000) + 1j*np.random.random(10000) In [68]: %timeit np.abs(z)**2 215 µs ± 2.09 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) In [69]: %timeit ((np.sqrt((z.real**2 + z.imag**2)))**2) 78.2 µs ± 193 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each) In [70]: %timeit (z.real**2 + z.imag**2) 40.1 µs ± 196 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each) In [71]: %timeit (z.conjugate()*z).real 43.7 µs ± 230 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each) Even considering the square root and/or additional square, does not account for the time spent In [72]: %timeit np.abs(z) 206 µs ± 970 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each) In [86]: %timeit np.sqrt(z.real**2 + z.imag**2) 70.1 µs ± 303 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each) In [73]: %timeit np.sqrt((z.conjugate()*z).real) 105 µs ± 2.23 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) I could not follow the code to understand what calculations are taking place for `abs()`. Any ideas? Regards, Juan

1 0

Inaccurate documentation of the random c-api
by zoj613 April 18, 2021

April 18, 2021

Hi All, https://numpy.org/devdocs/reference/random/c-api.html has an inaccurate description of the API that goes as follows: "/zig in the name are based on a ziggurat lookup algorithm is used instead of calculating the log, which is significantly faster. The non-ziggurat variants are used in corner cases and for legacy compatibility./". It appears this is no longer the case and instead there are functions that have `inv` in the name to signal the use of the inverse method for sampling. I submited a PR to reflect this and also added the missing function signatures at https://github.com/numpy/numpy/pull/18797/files Regards -- Sent from: http://numpy-discussion.10968.n7.nabble.com/

1 0

Improving performance of the `numpy.any` function.
by zoj613 April 15, 2021

April 15, 2021

Hi All, I was using numpy's `any` function earlier and realized that it might not be as performant as I assumed. See the code below: ``` In [1]: import numpy as np In [2]: a = np.zeros(1_000_000) In [3]: a[100] = 1 In [4]: b = np.zeros(2_000_000) In [5]: b[100] = 1 In [6]: %timeit np.any(a) 1.33 ms ± 15.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) In [7]: %timeit np.any(b) 2.65 ms ± 71.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [8] %timeit any(a) 13.4 µs ± 354 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) In [9]: %timeit any(b) 13.3 µs ± 219 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) ``` It looks like `np.any` does not terminate early as soon as it finds a non-zero element, compared to the built-in any function. So this prompted me to go look at the source code for `PyArray_Any`: https://github.com/numpy/numpy/blob/623bc1fae1d47df24e7f1e29321d0c0ba2771ce… . Taking a peek at the `PyArray_GenericReduceFunction` it looks like it just calls the `reduce` function from python on every single element of the array and there is no mechanism in place to short-circuit the call if an element has been found early. It seems worthwhile to have a specific function that allows early stopping of the reduction? So my question is: Is there a reason why maintainers did not implement such a feature? I would also like to know if everyone thinks its worthwhile to implement one? Im guessing there are many unsuspecting users like myself who call the `np.any` function on arrays assuming early-stopping is supported. I think existing code that calls the function would benefit a lot. Let me know what you think. Regards -- Sent from: http://numpy-discussion.10968.n7.nabble.com/

4 4

savetxt -> gzip: nondeterministic because of time stamp
by Joachim Wuttke April 14, 2021

April 14, 2021

If argument fname of savetxt(fname, X, ...) ends with ".gz" then array X is not only converted to text, but also compressed using gzip. The format gzip [1] has a timestamp. The Python module gzip.py [2] sets the timestamp according to an optional constructor argument "mtime". By default, the current time is used. This makes the file written by savetxt(*.gz, ...) non-deterministic. This is unexpected and confusing in a numerics context. I let different versions of a program generate *.gz files, and ran the "diff" util over pairs of output files to check whether any bit had changed. To my surprise, confusion, and desperation, output always had changed, and kept changing when I ran unchanged versions of my program over and again. So I learned the hard way that the *.gz files contain a timestamp. Regarding the module gzip.py, I submitted a pull request to improve description of the optional argument mtime, and hint at the possible choice mtime = 0 that makes outputs deterministic [3]. Regarding numpy, I'd propose a bolder measure: To let savetxt(fname, X, ...) store exactly the same information in compressed and uncompressed files, always invoke gzip with mtime = 0. I would like to follow up with a pull request, but I am unable to find out how numpy.savetxt is invoking gzip. Joachim [1] https://www.ietf.org/rfc/rfc1952.txt [2] https://docs.python.org/3/library/gzip.html [3] https://github.com/python/cpython/pull/25410

5 7

NumPy Community Meeting Wednesday
by Sebastian Berg April 14, 2021

April 14, 2021

Hi all, There will be a NumPy Community meeting Wednesday April 14th at 20:00 UTC. Everyone is invited and encouraged to join in and edit the work-in-progress meeting topics and notes at: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

Is there a defined way to "unpad" an array, and if not, should there be?
by Jeff Gostick April 13, 2021

April 13, 2021

I often find myself padding an array to do some processing on it (i.e. to avoid edge artifacts), then I need to remove the padding. I wish there was either a built in "unpad" function that accepted the same arguments as "pad", or that "pad" accepted negative numbers (e.g [-20, -4] would undo a padding of [20, 4]). This seems like a pretty obvious feature to me so maybe I've just missed something, but I have looked through all the open and closed issues on github and don't see anything related to this. Jeff G

7 10

files for 3D printing of NumPy logo
by Ralf Gommers April 11, 2021

April 11, 2021

Hi all, Andrej Krecker, who is a 3D printing engineer who likes NumPy, shared 3D printing files for the NumPy logo. It comes in two flavors, a really 3-D version, and a flatter (~0.5 cm thick) 2-D version. Check them out at https://www.thingiverse.com/thing:4821992. Cheers, Ralf

2 1

Type resolver related deprecations and changes
by Sebastian Berg April 8, 2021

April 8, 2021

Hi all, I have to do some changes to the type resolution, and I started these here: https://github.com/numpy/numpy/pull/18718 There are four changes: * Deprecate `signature="l"` and `signature=("l",)`, these are confusing since the signature should include all inputs and outputs. To only provide the output use `dtype="l"`. * Using `dtype=` for comparisons (e.g. `np.equal`) used to be weird: np.equal(1, 2, dtype=object) -> returns boolean np.equal(None, 2, dtype=object) -> returns object array The first one will now give a FutureWarning. Comparisons that provide a dtype other than object or bool give a DeprecationWarning (or fail). I hope the warning can be preserved when more refactoring happens. * NumPy *almost* always ignores any metadata, byte-order, time unit information from the `dtype` or `signature` arguments to ufuncs. Practically, the dtypes passed actually denote the DType types rather than the specific instance (which could be byte swapped). NumPy will now do this always and give a warning if byte-order or time unit is ignored! * It is THEORETICALLY possible to call `ufunc->type_resolver` in the C API (as opposed to providing it, which is somewhat OK). If someone does that they have to normalize the type tuple now, I don't really see a reason for keeping support, when NumPy will stop calling it itself almost always and anyone using it would probably be in trouble soon. To be clear: I have NOT found a single instance of such code in a code search. Even *providing* it – which is much more reasonable – is probably only done by astropy/pyerfa. ** Long example for the "time unit" dropping change ** For the third point, which is in theory the largest impact. Both pandas and astropy do not notice it (I also grepped scipy, its clean). These are the biggest changes: # The following will now warn on most systems (unchanged result): np.add(3, 5, dtype=">i32") # The biggest impact is for timedelta or datetimes: arr = np.arange(10, dtype="m8[s]") # The examples always ignored the time unit "ns" (using the # unit of `arr`. They now issue a warning: np.add(arr, arr, dtype="m8[ns]") np.maximum.reduce(arr, dtype="m8[ns]") # The following issue a warning but previously did return # a "ns" result. np.add(3, 5, dtype="m8[ns]") # Now return generic time units np.maximum(arr, arr, dtype="m8[ns]") # Now returns "s" (from `arr`) I doubt there is a good way to just keep the old behaviour. It is hopelessly inconsistent. (The result even depends on how you pass things, as the paths I deprecate align with `dtype=` but not with the `signature=(None, None, dtype)` equivalent call.) One thought I had just raising a hard error instead of a UserWarning right now. If you say that why not have `dtype=...` always be honored as the correct output dtype, I don't disagree. But it seems to me we probably would have wade through a FutureWarning first, so a warning is the "right direction". (Or just add an `output_dtypes` keyword argument.) Cheers, Sebastian

1 1

umPy Development Meeting Wednesday - Triage Focus
by Sebastian Berg April 6, 2021

April 6, 2021

Hi all, Our bi-weekly triage-focused NumPy development meeting is Wednesday, March 10th at 11 am Pacific Time (18:00 UTC). Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg I encourage everyone to notify us of issues or PRs that you feel should be prioritized, discussed, or reviewed. Best regards Sebastian

1 0