Hi all,
I have been coaching a new contributor to SciPy
Thanks for doing that!
and we are being routinely bitten by weird problems on import when trying to follow the development installation instructions.
We have too many flavors of the instructions unfortunately, and not all of them are 100% up-to-date.
At this point, I'm not so concerned about getting things to work: I know that, with enough trial-and-error, we will manage at some point. Instead, I'd like to try to have a more solid mental model of why things fail (if that's even possible).
I am undecided and hesitant on how to proceed here since I feel I'm trying to solve too many problems at the same time. In no particular order:
1) I could try to get help on the specific errors we're getting. However, these are of the difficult-to-debug kind (undefined symbols and such) and I anticipate that we would end up in something like "you have a borked SciPy installation, please start from scratch" or "use conda instead".
Please note that NumPy 1.22.0/1.22.1 had a few serious issues that resulted in undefined symbol problems. Those should all have been solved in 1.22.2, so please ensure you use that. Or alternatively, start using the Meson build so you won't run into those issues in the first place - see
http://scipy.github.io/devdocs/dev/contributor/meson.html.
Using conda is indeed the best idea, because it's the only way to give uniform cross-platform instructions. If you want to install all build dependencies (Python, BLAS/LAPACK, compilers) otherwise, you'll likely have to figure out some of the details for yourself depending on which Linux distro you use (or macOS - but definitely do not try to develop on Windows except for with WSL).
2) I'd love to get a better understanding of what's the role of `conda develop .` in the development instructions, and how it's different from `pip install -e .`. I tried to look around and the only references I see about `conda develop` are the manual pages, and an old issue pointing to lack of documentation
https://github.com/conda/conda-build/issues/1992
`conda develop .` is useless and should be cleaned up in the docs if it's still present. Please also do not use `pip install . -e`; best not to use `pip` at all, but if you do then you should add `--no-build-isolation`. Pip is *not* a good developer CLI, using `python setup.py develop` instead.
3) I've been asking myself whether SciPy would consider moving to an src-layout. I did a quick search on the issue tracker and the mailing list archives, and don't see any references. I know this is a contentious topic though, so it would be a pity to start a flame war about it, and that's definitely not my intention. I'm just wondering if at least it would help us avoid a family of problems: the ones appearing when folks do `import scipy` from the root directory.
I'd say no, we won't consider that - way too much churn for little gain.
Part of the issue with the dev instructions is that `python runtests.py` and in-place builds are both used and are two incompatible options. With Meson we go to a single interface (currently `python dev.py`), inplace builds are not supported (at least for now). `runtests.py` is recommended by us in most places since it avoids things like issues with `import scipy` from the root of the repo.
4) We could continue the conversation we started in
https://github.com/scipy/scipy/issues/12633 and try to see if there's a "happy path" that folks can follow given some assumptions (modern Linux distro/WSL2, compilers coming from either apt or conda-forge, no dangling Python installations) to reduce the installation instructions to the minimum (which could be `pip install -e . && pytest`).
We're moving to Meson, and the instructions are:
1. Set up your environment with all dev dependencies and activate it.
2. `python dev.py`
We will keep `setup.py` around for one more release as a backup, but those instructions are all going to be updated.
Apologies if this rambling is not particularly helpful, hope I can get some help on these various points.
No worries, I totally understand the pain, we regularly have maintainers who cannot build SciPy either. It's difficult and fragile.
Cheers,
Ralf