Hi Matti, hi all,

@Matti: I don’t know what exactly you are referring to (Pull request or the Github project, links see below). Maybe some clarification is needed, which I hereby try to do ;)

A .npy file created by some appending process is a regular .npy file and does not need to be read in chunks. Processing arrays larger than the systems memory can already be done with memory mapping (numpy.load(… mmap_mode=...)), so no third-party support is needed to do so.

The idea is not necessarily to only write some known-but-fragmented content to a .npy file in chunks or to only handle files larger than the RAM.

It is more about the ability to append to a .npy file at any time and between program runs. For example, in our case, we have a large database-like file containing all (preprocessed) images of all videos used to train a neural network. When new video data arrives, it can simply be appended to the existing .npy file. When training the neural net, the data is simply memory mapped, which happens basically instantly and does not use extra space between multiple training processes. We have tried out various fancy, advanced data formats for this task, but most of them don’t provide the memory mapping feature which is very handy to keep the time required to test a code change comfortably low - rather, they have excessive parse/decompress times. Also other libraries can also be difficult to handle, see below.

One of my favorite features (besides memory mapping perhaps) is this one:

“… Be reverse engineered. Datasets often live longer than the programs that created them. A competent developer should be able to create a solution in his preferred programming language to read most NPY files that he has been given without much documentation. ..."

This is a big disadvantage with all the fancy formats out there: they require dedicated libraries. Some of these libraries don’t come with nice and free documentation (especially lacking easy-to-use/easy-to-understand code examples for the target language, e.g. C) and/or can be extremely complex, like HDF5. Yes, HDF5 has its users and is totally valid if one operates the world’s largest particle accelerator, but we have spend weeks finding some C/C++ library for it which does not expose bugs and is somehow documented. We actually failed and posted a bug which was fixed a year later or so. This can ruin entire projects - fortunately not ours, but it ate up a lot of time we could have spend more meaningful. On the other hand, I don’t see how e.g. zarr provides added value over .npy if one only needs the .npy features and maybe some append-data-along-one-axis feature. Yes, maybe there are some uses for two or three appendable axes, but I think having one axis to append to should cover a lot of use cases: this axis is typically time: video, audio, GPS, signal data in general, binary log data, "binary CSV" (lines in file): all of those only need one axis to append to.

The .npy format is so simple, it can be read e.g. in C in a few lines. Or accessed easily through Numpy and ctypes by pointers for high speed custom logic - not even requiring libraries besides Numpy.

Making .npy appendable is easy to implement. Yes, appending along one axis is limited as the .npy format itself. But I consider that rather to be a feature than a (actual) limitation, as it allows for fast and simple appends.

The question is if there is some support for an append-to-.npy-files-along-one-axis feature in the Numpy community and if so, about the details of the actual implementation. I made one suggestion in

and I offer to invest time to update/modify/finalize the PR. I’ve also created a library that can already append to .npy:

However, due to current limitations in the .npy format, the code is more complex than it could actually be (the library initializes and checks spare space in the header) and it needs to rewrite the header every time. Both could be made unnecessary with a very small addition to the .npy file format. Data would stay continuous (no fragmentation!), there should just be a way to indicate that the actual shape of the array should derived from the file size.

Best, Michael