Mailman 3 August 2022 - NumPy-Discussion

next NumPy community meeting
by Inessa Pawson Oct. 22, 2023

Oct. 22, 2023

The next NumPy community meeting will be held this Wednesday, August 17th at 18:00 (6 pm) UTC. Join us via Zoom: https://us06web.zoom.us/j/83278611437?pwd=ekhoLzlHRjdWc0NOY2FQM0NPemdkZz09 Everyone is welcome and encouraged to attend. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both For the notes from the previous meeting, visit: https://github.com/numpy/archive/blob/main/community_meetings/community-202… Cheers, … [View More]

1 30

reminder: put an upper bound on setuptools if you use numpy.distutils!
by Ralf Gommers Aug. 31, 2023

Aug. 31, 2023

Hi all, After setuptools 65.0 was released a few days ago, all users of numpy.distutils had their builds broken. This is already fixed in setuptools 65.0.2 because the breakage was particularly bad. However, the next breakage may not be fixed anymore (and more breakages *are* expected). So this is a good time to remind you all that you should put an upper bound on the setuptools version you allow in the releases of your package - to the last version that is known to work with your package. … [View More]

2 2

mixed mode arithmetic
by Neal Becker July 11, 2023

July 11, 2023

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem Feb. 18, 2023

Feb. 18, 2023

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How … [View More]

5 4

Documentation Team meeting - Monday June 8th
by Melissa Mendonça Dec. 3, 2022

Dec. 3, 2022

Hi all! A reminder that on Monday, June 8, we have another documentation team meeting at 3PM UTC**. If you wish to join on Zoom, you need to use this link https://zoom.us/j/420005230 Here's the permanent hackmd document with the meeting notes: https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg <https://www.google.com/url?q=https%3A%2F%2Fhackmd.io%2FoB_boakvRqKR-_2jRV-Q…> Hope to see you around (especially if you want to introduce yourself or discuss ideas for Google Season of Docs). ** You … [View More]

6 74

next NumPy triage meeting
by Inessa Pawson Nov. 27, 2022

Nov. 27, 2022

The next NumPy triage meeting will be held this Wednesday, August 10th at 16:00 (4 pm) UTC. This is a meeting where we synchronously triage prioritized PRs and issues. Join us via Zoom: https://us06web.zoom.us/j/82096749952?pwd=MW9oUmtKQ1c3a2gydGk1RTdYUUVXZz09 Everyone is welcome to attend and contribute to a conversation. Please notify us of issues or PRs that you’d like to have reviewed by adding a GitHub link to them in the meeting agenda: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg. Cheers, … [View More]

1 8

next NumPy Newcomers' Hour
by Inessa Pawson Oct. 2, 2022

Oct. 2, 2022

Our next Newcomers' Hour will be held this Thursday, June 16th, at 4 pm UTC. Stop by to ask questions or just to say hi. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/3f3otyyuTte3FU9y3QzsLg?both Join the meeting via Zoom: https://us02web.zoom.us/j/87192457898 Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ Twitter: @inessapawson

1 3

f2py compile error with full path of signature file on Windows
by Johann Rohwer Sept. 4, 2022

Sept. 4, 2022

2 1

Need help extending a NumPy array in C
by lpsmith＠uw.edu Sept. 3, 2022

Sept. 3, 2022

Hello! I am maintaining a C++ codebase with extensive ties to Python bindings (via SWIG). One of the features of the code is that we define (in C) a subclass of a NumPy Array. Everything worked until we started getting this message with numpy 1.23: RuntimeError: Object of type <class 'NamedArray'> appears to be C subclassed NumPy array, void scalar, or allocated in a non-standard way.NumPy reserves the right to change the size of these structures. Projects are required to take this … [View More]

3 6

Exporting numpy arrays to binary JSON (BJData) for better portability
by Qianqian Fang Sept. 1, 2022

Sept. 1, 2022

To avoid derailing the other thread <https://mail.python.org/archives/list/numpy-discussion@python.org/thread/A4…> on extending .npy files, I am going to start a new thread on alternative array storage file formats using binary JSON - in case there is such a need and interest among numpy users specifically, i want to first follow up with Bill's question below regarding loading time On 8/25/22 11:02, Bill Ross wrote: > > |Can you give load times for these?| > as I … [View More]mentioned in the earlier reply to Robert, the most memory-efficient (i.e. fast loading, disk-mmap-able) but not necessarily disk-efficient (i.e. may result in the largest data file sizes) BJData construct to store an ND array using BJData's ND-array container. I have to admit that both jdata and bjdata modules have not been extensively optimized for speed. with the current implementation, here are the loading time for a larger diagonal matrix (eye(10000)) a BJData file storing a single eye(10000) array using the ND-array container can be downloaded from here <http://neurojson.org/wiki/upload/eye1e4_bjd_raw_ndsyntax.jdb.zip>(file size: 1MB with zip, if decompressed, it is ~800MB, as the npy file) - this file was generated from a matlab encoder, but can be loaded using Python (see below Re Robert). |800000128 eye1e4.npy|| ||800000014 eye1e4_bjd_raw_ndsyntax.jdb|| || 813721 eye1e4_bjd_zlib.jdb|| || 113067 eye1e4_bjd_lzma.jdb| the loading time (from an nvme drive, Ubuntu 18.04, python 3.6.9, numpy 1.19.5) for each file is listed below: |0.179s eye1e4.npy (mmap_mode=None)|| ||0.001s eye1e4.npy (mmap_mode=r)|| ||0.718s eye1e4_bjd_raw_ndsyntax.jdb|| ||1.474s eye1e4_bjd_zlib.jdb|| ||0.635s eye1e4_bjd_lzma.jdb| clearly, mmapped loading is the fastest option without a surprise; it is true that the raw bjdata file is about 5x slower than npy loading, but given the main chunk of the data are stored identically (as contiguous buffer), I suppose with some optimization of the decoder, the gap between the two can be substantially shortened. The longer loading time of zlib/lzma (and similarly saving times) reflects a trade-off between smaller file sizes and time for compression/decompression/disk-IO. by no means I am saying the binary JSON format is readily available to deliver better speed with its current non-optimized implementation. I just want to bright the attention to this class of formats, and highlight that it's flexibility gives abundant mechanisms to create fast, disk-mapped IO, while allowing additional benefits such as compression, unlimited metadata for future extensions etc. > |> 8000128 eye5chunk.npy|| > ||> 5004297 eye5chunk_bjd_raw.jdb|| > ||> 10338 eye5chunk_bjd_zlib.jdb|| > ||> 2206 eye5chunk_bjd_lzma.jdb| > > For my case, I'd be curious about the time to add one 1T-entries file > to another. as I mentioned in the previous reply, bjdata is appendable <https://github.com/NeuroJSON/bjdata/blob/master/images/BJData_Diagram.pdf>, so you can simply append another array (or a slice) to the end of the file. > Thanks, > Bill also related, Re @Robert's question below > Are any of them supported by a Python BJData implementation? I didn't > see any option to get that done in the `bjdata` package you > recommended, for example. > https://github.com/NeuroJSON/pybj/blob/a46355a0b0df0bec1817b04368a5a5733586… the bjdata module currently only support nd-array in the decoder <https://github.com/NeuroJSON/pybj/blob/a46355a0b0df0bec1817b04368a5a5733586…> (i.e. map such buffer to a numpy.ndarray) - should be relatively trivial to add it to the encoder though. on the other side, the annotated format is currently supported. one can call jdata module (responsible for annotation-level encoding/decoding) as shown in my sample code, then call bjdata internally for data serialization. > Okay. Given your wording, it looked like you were claiming that the > binary JSON was supported by the whole ecosystem. Rather, it seems > like you can either get binary encoding OR the ecosystem support, but > not both at the same time. all in relative terms of course - json has ~100 listed parsers on it's website <https://www.json.org/json-en.html>, MessagePack - another flavor of binary JSON - listed <https://msgpack.org/index.html> ~50/60 parsers, and UBJSON listed <https://ubjson.org/libraries/> ~20 parsers. I am not familiar with npy parsers, but googling it returns only a few. also, most binary JSON implementations provided tools to convert to JSON and back, so, in that sense, whatever JSON has in its ecosystem can be "potentially" used for binary JSON files if one wants to. There are also recent publications comparing differences between various binary JSON formats in case anyone is interested https://github.com/ubjson/universal-binary-json/issues/115 [View Less]

5 10