Mailman 3 August 2022 - NumPy-Discussion

next NumPy community meeting
by Inessa Pawson 22 Oct '23

22 Oct '23

The next NumPy community meeting will be held this Wednesday, August 17th at 18:00 (6 pm) UTC. Join us via Zoom: https://us06web.zoom.us/j/83278611437?pwd=ekhoLzlHRjdWc0NOY2FQM0NPemdkZz09 Everyone is welcome and encouraged to attend. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both For the notes from the previous meeting, visit: https://github.com/numpy/archive/blob/main/community_meetings/community-202… Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ Twitter: @inessapawson

1 30

reminder: put an upper bound on setuptools if you use numpy.distutils!
by Ralf Gommers 31 Aug '23

31 Aug '23

Hi all, After setuptools 65.0 was released a few days ago, all users of numpy.distutils had their builds broken. This is already fixed in setuptools 65.0.2 because the breakage was particularly bad. However, the next breakage may not be fixed anymore (and more breakages *are* expected). So this is a good time to remind you all that you should put an upper bound on the setuptools version you allow in the releases of your package - to the last version that is known to work with your package. Our official stance here is that setuptools versions >=60 are not supported - see the "deprecating numpy.distutils" thread: https://mail.python.org/archives/list/numpy-discussion@python.org/message/P…. Newer versions may work for you, depending on what features you use. They don't for NumPy and for SciPy; both projects pin to 59.X to avoid problems. For the recent issue with setuptools 65.0.0, see https://github.com/numpy/numpy/issues/22135. We have also made the warnings about this topic in our docs more explicit, see https://github.com/numpy/numpy/pull/22154. Cheers, Ralf

2 2

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Documentation Team meeting - Monday June 8th
by Melissa Mendonça 03 Dec '22

03 Dec '22

Hi all! A reminder that on Monday, June 8, we have another documentation team meeting at 3PM UTC**. If you wish to join on Zoom, you need to use this link https://zoom.us/j/420005230 Here's the permanent hackmd document with the meeting notes: https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg <https://www.google.com/url?q=https%3A%2F%2Fhackmd.io%2FoB_boakvRqKR-_2jRV-Q…> Hope to see you around (especially if you want to introduce yourself or discuss ideas for Google Season of Docs). ** You can click this link to get the correct time at your timezone: https://www.timeanddate.com/worldclock/fixedtime.html?msg=NumPy+Documentati… - Melissa

6 74

next NumPy triage meeting
by Inessa Pawson 27 Nov '22

27 Nov '22

The next NumPy triage meeting will be held this Wednesday, August 10th at 16:00 (4 pm) UTC. This is a meeting where we synchronously triage prioritized PRs and issues. Join us via Zoom: https://us06web.zoom.us/j/82096749952?pwd=MW9oUmtKQ1c3a2gydGk1RTdYUUVXZz09 Everyone is welcome to attend and contribute to a conversation. Please notify us of issues or PRs that you’d like to have reviewed by adding a GitHub link to them in the meeting agenda: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg. Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy email: inessa(a)albuscode.org

1 8

next NumPy Newcomers' Hour
by Inessa Pawson 02 Oct '22

02 Oct '22

Our next Newcomers' Hour will be held this Thursday, June 16th, at 4 pm UTC. Stop by to ask questions or just to say hi. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/3f3otyyuTte3FU9y3QzsLg?both Join the meeting via Zoom: https://us02web.zoom.us/j/87192457898 Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ Twitter: @inessapawson

1 3

f2py compile error with full path of signature file on Windows
by Johann Rohwer 04 Sep '22

04 Sep '22

2 1

Need help extending a NumPy array in C
by lpsmith＠uw.edu 03 Sep '22

03 Sep '22

Hello! I am maintaining a C++ codebase with extensive ties to Python bindings (via SWIG). One of the features of the code is that we define (in C) a subclass of a NumPy Array. Everything worked until we started getting this message with numpy 1.23: RuntimeError: Object of type <class 'NamedArray'> appears to be C subclassed NumPy array, void scalar, or allocated in a non-standard way.NumPy reserves the right to change the size of these structures. Projects are required to take this into account by either recompiling against a specific NumPy version or padding the struct and enforcing a maximum NumPy version. My problem is that I don't know how do to either of those things. I would have assumed that whatever I compiled against, it would always be compiled against a specific NumPy version, and I also assumed that 'enforcing a maximum NumPy version' would happen in requirements.txt for the Python package, but that also seems to not be the case. Any hints? Thank you! -Lucian Smith

3 6

Exporting numpy arrays to binary JSON (BJData) for better portability
by Qianqian Fang 01 Sep '22

01 Sep '22

To avoid derailing the other thread <https://mail.python.org/archives/list/numpy-discussion@python.org/thread/A4…> on extending .npy files, I am going to start a new thread on alternative array storage file formats using binary JSON - in case there is such a need and interest among numpy users specifically, i want to first follow up with Bill's question below regarding loading time On 8/25/22 11:02, Bill Ross wrote: > > |Can you give load times for these?| > as I mentioned in the earlier reply to Robert, the most memory-efficient (i.e. fast loading, disk-mmap-able) but not necessarily disk-efficient (i.e. may result in the largest data file sizes) BJData construct to store an ND array using BJData's ND-array container. I have to admit that both jdata and bjdata modules have not been extensively optimized for speed. with the current implementation, here are the loading time for a larger diagonal matrix (eye(10000)) a BJData file storing a single eye(10000) array using the ND-array container can be downloaded from here <http://neurojson.org/wiki/upload/eye1e4_bjd_raw_ndsyntax.jdb.zip>(file size: 1MB with zip, if decompressed, it is ~800MB, as the npy file) - this file was generated from a matlab encoder, but can be loaded using Python (see below Re Robert). |800000128 eye1e4.npy|| ||800000014 eye1e4_bjd_raw_ndsyntax.jdb|| || 813721 eye1e4_bjd_zlib.jdb|| || 113067 eye1e4_bjd_lzma.jdb| the loading time (from an nvme drive, Ubuntu 18.04, python 3.6.9, numpy 1.19.5) for each file is listed below: |0.179s eye1e4.npy (mmap_mode=None)|| ||0.001s eye1e4.npy (mmap_mode=r)|| ||0.718s eye1e4_bjd_raw_ndsyntax.jdb|| ||1.474s eye1e4_bjd_zlib.jdb|| ||0.635s eye1e4_bjd_lzma.jdb| clearly, mmapped loading is the fastest option without a surprise; it is true that the raw bjdata file is about 5x slower than npy loading, but given the main chunk of the data are stored identically (as contiguous buffer), I suppose with some optimization of the decoder, the gap between the two can be substantially shortened. The longer loading time of zlib/lzma (and similarly saving times) reflects a trade-off between smaller file sizes and time for compression/decompression/disk-IO. by no means I am saying the binary JSON format is readily available to deliver better speed with its current non-optimized implementation. I just want to bright the attention to this class of formats, and highlight that it's flexibility gives abundant mechanisms to create fast, disk-mapped IO, while allowing additional benefits such as compression, unlimited metadata for future extensions etc. > |> 8000128 eye5chunk.npy|| > ||> 5004297 eye5chunk_bjd_raw.jdb|| > ||> 10338 eye5chunk_bjd_zlib.jdb|| > ||> 2206 eye5chunk_bjd_lzma.jdb| > > For my case, I'd be curious about the time to add one 1T-entries file > to another. as I mentioned in the previous reply, bjdata is appendable <https://github.com/NeuroJSON/bjdata/blob/master/images/BJData_Diagram.pdf>, so you can simply append another array (or a slice) to the end of the file. > Thanks, > Bill also related, Re @Robert's question below > Are any of them supported by a Python BJData implementation? I didn't > see any option to get that done in the `bjdata` package you > recommended, for example. > https://github.com/NeuroJSON/pybj/blob/a46355a0b0df0bec1817b04368a5a5733586… the bjdata module currently only support nd-array in the decoder <https://github.com/NeuroJSON/pybj/blob/a46355a0b0df0bec1817b04368a5a5733586…> (i.e. map such buffer to a numpy.ndarray) - should be relatively trivial to add it to the encoder though. on the other side, the annotated format is currently supported. one can call jdata module (responsible for annotation-level encoding/decoding) as shown in my sample code, then call bjdata internally for data serialization. > Okay. Given your wording, it looked like you were claiming that the > binary JSON was supported by the whole ecosystem. Rather, it seems > like you can either get binary encoding OR the ecosystem support, but > not both at the same time. all in relative terms of course - json has ~100 listed parsers on it's website <https://www.json.org/json-en.html>, MessagePack - another flavor of binary JSON - listed <https://msgpack.org/index.html> ~50/60 parsers, and UBJSON listed <https://ubjson.org/libraries/> ~20 parsers. I am not familiar with npy parsers, but googling it returns only a few. also, most binary JSON implementations provided tools to convert to JSON and back, so, in that sense, whatever JSON has in its ecosystem can be "potentially" used for binary JSON files if one wants to. There are also recent publications comparing differences between various binary JSON formats in case anyone is interested https://github.com/ubjson/universal-binary-json/issues/115

5 10