Mailman 3 July 2021 - NumPy-Discussion

Proposal for adding bit_count
by Ganesh Kathiresan Aug. 4, 2021

Aug. 4, 2021

Hi All, I am working <https://github.com/numpy/numpy/pull/19355> on a new UFunc, ` bit_count <https://en.wikipedia.org/wiki/Hamming_weight>` (popcount in other languages) that aims to count the number of 1-bits in the absolute value of an Integer. Implementation ---------------------------------- The primary reference for the implementation is CountBitsSetParallel <http://graphics.stanford.edu/~seander/bithacks.html#CountBitsSetParallel>. Here we take 12 operations to achieve the result which is the same as the lookup table method but does not suffer from memory issues or cache misses. The implementation is aimed at unsigned integers, absolute value of signed integers and objects that support the operation. Usage -------------- >>> np.bit_count(1023) 10 >>> a = np.array([2**i - 1 for i in range(16)]) >>> np.bit_count(a) array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]) >>> np.int32(1023).bit_count() 10 Notes ------------- 1. Python has included this method here <https://github.com/python/cpython/commit/8bd216dfede9cb2d5bedb67f20a30c9984…> (3.10+). Tracking issue <https://bugs.python.org/issue29882> 2. NumPy tracking issue <https://github.com/numpy/numpy/issues/16325> 3. Interesting read <https://archive.org/details/dr_dobbs_journal_vol_08/page/n185/mode/2up> on how we get the magic number. Needed a bit of digging :) Please let us know what you think about the implementation and where we can improve in terms of performance or interface. Regards, Ganesh

3 4

Adding POWER10 (VSX4) support to the SIMD framework
by Nicholai Tukanov July 31, 2021

July 31, 2021

I would like to understand how to go about extending the SIMD framework in order to add support for POWER10. Specifically, I would like to add the following instructions: `lxvp` and `stxvp` which loads/stores 256 bits into/from two vectors. I believe that this will be able to give a decent performance boost for those on POWER machines since it can halved the amount of loads/stores issued. Additionally, matrix engines (2-D SIMD instructions) are becoming quite popular due to their performance improvements for deep learning and scientific computing. Would it be beneficial to add these new advanced SIMD instructions into the framework or should these instructions be left to libraries such as OpenBLAS and MKL? Thank you, Nicholai Tukanov

2 1

NumPy Development Meeting Wednesday - Triage Focus
by Sebastian Berg July 27, 2021

July 27, 2021

Hi all, Our bi-weekly triage-focused NumPy development meeting is Wednesday, July 28th at 9 am Pacific Time (16:00 UTC). Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg I encourage everyone to notify us of issues or PRs that you feel should be prioritized, discussed, or reviewed. Best regards Sebastian

1 0

Newcomer's Meeting July 29: special accessibility mini-sprint!
by Melissa Mendonça July 26, 2021

July 26, 2021

Hello, folks! This week we have our planned Newcomer's meeting on July 29, at 8PM UTC. This time, we are teaming up with Tony Fast and Isabela Presedo-Floyd to propose a low-code mini-sprint focusing on accessibility. This is a great opportunity to make your first contribution to NumPy! We will be focusing on writing alt-text for images in our documentation. Alt-text provides a textual alternative to non-text content in web pages, and as cited in this WebAIM document [1]: "Alternative text serves several functions: - It is read by screen readers in place of images allowing the content and function of the image to be accessible to those with visual or certain cognitive disabilities. - It is displayed in place of the image in browsers if the image file is not loaded or when the user has chosen not to view images. - It provides a semantic meaning and description to images which can be read by search engines or be used to later determine the content of the image from page context alone." You can find some more information on how to add alt-text in [2]. As usual, all are welcome - feel free to join even if you just want to observe! To join on Zoom, use this link: https://zoom.us/j/6345425936 Hope to see you around! - Melissa [1] https://webaim.org/techniques/alttext/ [2] https://www.w3.org/WAI/tutorials/images/decision-tree/ ** You can click this link to get the correct time at your timezone: https://www.timeanddate.com/worldclock/fixedtime.html?msg=NumPy+Newcomers%2… <https://www.timeanddate.com/worldclock/fixedtime.html?msg=NumPy+Newcomer%27…> *** You can add the NumPy community calendar to your Google calendar by clicking this link: https://calendar.google.com/calendar /r?cid=YmVya2VsZXkuZWR1X2lla2dwaWdtMjMyamJobGRzZmIyYzJqODFjQGdyb3VwLmNhbGVuZGFyLmdvb2dsZS5jb20

1 0

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell July 22, 2021

July 22, 2021

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

Add count (and dtype) to packbits
by Neal Becker July 21, 2021

July 21, 2021

In my application I need to pack bits of a specified group size into integral values. Currently np.packbits only packs into full bytes. For example, I might have a string of bits encoded as a np.uint8 vector with each uint8 item specifying a single bit 1/0. I want to encode them 4 bits at a time into a np.uint32 vector. python code to implement this: --------------- def pack_bits (inp, bits_per_word, dir=1, dtype=np.int32): assert bits_per_word <= np.dtype(dtype).itemsize * 8 assert len(inp) % bits_per_word == 0 out = np.empty (len (inp)//bits_per_word, dtype=dtype) i = 0 o = 0 while i < len(inp): ret = 0 for b in range (bits_per_word): if dir > 0: ret |= inp[i] << b else: ret |= inp[i] << (bits_per_word - b - 1) i += 1 out[o] = ret o += 1 return out --------------- It looks like unpackbits has a "count" parameter but packbits does not. Also would be good to be able to specify an output dtype.

2 2

NumPy Community Meeting Wednesday
by Sebastian Berg July 20, 2021

July 20, 2021

Hi all, There will be a NumPy Community meeting Wednesday July 21st at 20:00 UTC. Everyone is invited and encouraged to join in and edit the work-in-progress meeting topics and notes at: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

NumPy Community Survey 2021 - Call for participation
by Stephanie Mendoza July 20, 2021

July 20, 2021

Hello NumPy Community, Last year’s inaugural community survey yielded participation from over 1,200 NumPy users from 75 different countries. The responses provided invaluable feedback from the NumPy community. You can view the results from the 2020 survey here: https://numpy.org/user-survey-2020/ It’s time for another survey, and we are counting on you once again. The 2021 survey is now LIVE, and you can participate using the following link: https://berkeley.qualtrics.com/jfe/form/SV_aaOONjgcBXDSl4q. Please feel free to share this link with others so that we can reach as much of the NumPy community as possible. Please direct any questions / concerns to stephaniemendoza2014(a)gmail.com. Sincerely, Stephanie Mendoza

1 0

Add smallest_normal and smallest_subnormal attributes to finfo
by Stephannie Jiménez Gacha July 19, 2021

July 19, 2021

Good afternoon, Given the discussions happened in the Data API consortium when looking into the attributes of `finfo` used in the wild, we found that `tiny` is used regularly but in a good amount of cases not for its intended purpose but rather as "just give me a small number". Following this we are proposing the addition of `smallest_normal` and `smallest_subnormal` attributes. Personally, I think that the `tiny` name is a little bit odd and misleading, so it will be great to leave that as an alias but have a clear name in this class. Right now the PR: https://github.com/numpy/numpy/pull/18536 has all the changes and all the values added were checked against IEEE-754 standard. One of the main concerns is the support of subnormal numbers in certain architectures, where the values can't be calculated accurately. Given the state of the discussion, we don't know if the best alternative is to not add the `smallest_subnormal` attribute and just add the `smallest_number` attribute as an alias to `tiny`. We open this to discussion to see what way we can go in order to get this PR merged. *Stephannie Jimenez Gacha*Software developer *Quansight* | Your Data Experts w: www.quansight.com e: sgacha(a)quansight.com <https://www.linkedin.com/company/quansight> <https://twitter.com/quansightai>

2 1

(no subject)
by Charles R Harris July 18, 2021

July 18, 2021

Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.21.1. The NumPy 1.21.1 is a maintenance release that fixes bugs discovered after the 1.21.0 release. OpenBLAS has also been updated to v0.3.17 to deal with arm64 problems. The Python versions supported for this release are 3.7-3.9. The 1.21.x series is compatible with development Python 3.10 and Python 3.10 will be officially supported after it is released. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.21.1/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.21.1>. Linux users will need pip >= 0.19.3 in order to install manylinux2010 and manylinux2014 wheels. *Contributors* A total of 11 people contributed to this release. People with a \"+\" by their names contributed a patch for the first time. - Bas van Beek - Charles Harris - Ganesh Kathiresan - Gregory R. Lee - Hugo Defois + - Kevin Sheppard - Matti Picus - Ralf Gommers - Sayed Adel - Sebastian Berg - Thomas J. Fan *Pull requests merged* A total of 26 pull requests were merged for this release. - #19311: REV,BUG: Replace `NotImplemented` with `typing.Any` - #19324: MAINT: Fixed the return-dtype of `ndarray.real` and `imag` - #19330: MAINT: Replace `"dtype[Any]"` with `dtype` in the definiton of\... - #19342: DOC: Fix some docstrings that crash pdf generation. - #19343: MAINT: bump scipy-mathjax - #19347: BUG: Fix arr.flat.index for large arrays and big-endian machines - #19348: ENH: add `numpy.f2py.get_include` function - #19349: BUG: Fix reference count leak in ufunc dtype handling - #19350: MAINT: Annotate missing attributes of `np.number` subclasses - #19351: BUG: Fix cast safety and comparisons for zero sized voids - #19352: BUG: Correct Cython declaration in random - #19353: BUG: protect against accessing base attribute of a NULL subarray - #19365: BUG, SIMD: Fix detecting AVX512 features on Darwin - #19366: MAINT: remove `print()`\'s in distutils template handling - #19390: ENH: SIMD architectures to show\_config - #19391: BUG: Do not raise deprecation warning for all nans in unique\... - #19392: BUG: Fix NULL special case in object-to-any cast code - #19430: MAINT: Use arm64-graviton2 for testing on travis - #19495: BUILD: update OpenBLAS to v0.3.17 - #19496: MAINT: Avoid unicode characters in division SIMD code comments - #19499: BUG, SIMD: Fix infinite loop during count non-zero on GCC-11 - #19500: BUG: fix a numpy.npiter leak in npyiter\_multi\_index\_set - #19501: TST: Fix a `GenericAlias` test failure for python 3.9.0 - #19502: MAINT: Start testing with Python 3.10.0b3. - #19503: MAINT: Add missing dtype overloads for object- and ctypes-based\... - #19510: REL: Prepare for NumPy 1.21.1 release. Cheers, Charles Harris

1 0