On Fri, Apr 14, 2023 at 9:38 AM <alugowski@gmail.com> wrote:
Great!
Ralf Gommers wrote:
Hi, I'm the author of fast_matrix_market, a package that reads and writes Matrix Market files: https://github.com/alugowski/fast_matrix_market/tree/main/python The Python binding is API compatible with scipy.io.mmio, but about 25x faster. A few other handy improvements include 64-bit indices (only if necessary), direct csc/csr writes with no coo intermediary, longdouble. It passes the SciPy mmio test suite. Would there be any interest in somehow integrating this library into SciPy? The speed difference really does make a big difference for large files. The package is written in C++17 and pybind11. Threading is with a simple thread pool based on c++11 threads, but can be changed. A significant part of the speed increase is parsing with std::from_chars instead of strto*, which alone contributes a massive improvement. Thanks for this package and proposal Adam! I had a quick look at your
On Wed, Apr 12, 2023 at 9:57 AM alugowski@gmail.com wrote: package, and it looks good to me. This kind of upstreaming of code, when there's a clear performance benefit and the code is maintained, seems nice to me. Just to make sure: are you planning to continue maintaining this code? Either within SciPy only, or also as a separate package and keeping the two in sync?
Yes, I'm seeing increasing interest in the package so I'm going to keep maintaining it. I do have a lot of use of the C++ bindings, so it makes sense to keep it as a separate package in addition to within SciPy.
The C++17 should be fine as far as I can tell - our http://scipy.github.io/devdocs/dev/toolchain.html has an upgrade to C++17 marked for 2022 already, so as long as things build with what MSVC provides (the usual limiting factor), that should be good. Cheers, Ralf
I've gone through the MSVC pains already to build Windows wheels with cibuildwheel, so that shouldn't be an issue.
That all sounds good, thanks Adam. I think the next steps are then to proceed with your proposal integration and open a PR. Here is what I suggest: - add your code in a new private directory `scipy/io/_fast_matrix_market/src/` with a README explaining where the code comes from next to the `src` - integrate it in the SciPy build system in `scipy/io/meson.build - you have to (unfortunately) also still deal with the setup.py-based build in one way or the other. We're going to throw it out soon, so you can either ignore it and keep the slow `_mmio.py` implementation for that, or integrate the C++ code in `scipy/io/setup.py`. The latter is a little nicer, but in case it's tricky to get to work then keeping the slow code is also okay. The only reason we will still keep it around for the 1.11.0 release (probably) is conda-forge on Windows, so that's a limited number of users who'd get the slow code. Cheers, Ralf