Hi Ralf,
Thanks for the quick response. I answer inline:
Please note that NumPy 1.22.0/1.22.1 had a few serious issues that resulted in undefined symbol problems. Those should all have been solved in 1.22.2, so please ensure you use that.
True that, I'll keep it in mind.
Using conda is indeed the best idea, because it's the only way to give uniform cross-platform instructions.
For the environment creation I agree it's the best solution, absolutely.
`conda develop .` is useless and should be cleaned up in the docs if it's still present.
After inspecting it a bit more closely it was less scary than I thought (I should have done this before sending the email): it just creates a `conda.pth` with the source path, which is the equivalent of doing `export PYTHONPATH=$PYTHONPATH:$(pwd)`. So, not necessarily "useless" if used wisely? For example, with the new Meson workflow, one could do `conda develop $(pwd)/installdir/lib/pythonX.YY/site-packages/` rather than modifying `$PYTHONPATH`, although I agree that it's not a huge difference, and I see how `conda develop .` could be problematic.
I can't remember the details, but there was another problem with it where it wouldn't actually do anything. IIRC a conda maintainer said it was unmaintained or not recommended.
For Meson, I recommend sticking purely to `python dev.py`.
Please also do not use `pip install . -e`; best not to use `pip` at all, but if you do then you should add `--no-build-isolation`. Pip is *not* a good developer CLI, using `python setup.py develop` instead.
I recommend `python setup.py develop` until it is actually removed. It's not just that you need to use `--no-build-isolation` so easy to shoot yourself in the foot, something like a pip editable install is actually conceptually confused/wrong when you're developing on a project like SciPy. What you want is:
1. Create a dev environment (doesn't matter how - conda, virtualenv, standalone Python installer, Nix, Spack, ...) with all dependencies installed.
2. Activate the dev env
3. Build/install/test on the project.
The conceptual problem that `pip install . -e` has is that, when you're trying to do (3), it "reaches up" and messes with (1). You want to be using commands/tools that can do the build-install-test steps over and over without messing with the environment. python `runtests.py / dev.py / setup.py` all do this right, and `pip install` doesn't. Pip is primarily a package installer, and its use as a pure Python CLI (emphasis on pure Python, where you don't need to care about build isolation, not showing a build log, etc.) is secondary/accidental simply because there isn't a better dedicated tool. For packages with compiled code it's worse than for pure Python code. Unfortunately, most people working on packaging tools only work on pure Python packages, so they don't really see the problems nor the confused mental model.
Cheers,
Ralf
Part of the issue with the dev instructions is that `python runtests.py` and in-place builds are both used and are two incompatible options.
Good to keep in mind too.
We're moving to Meson, and the instructions are:
I cheered the effort when I saw it but hadn't actually tried it, now I just did and I'm thrilled to see it's so fast.
Thanks again for the responses, it is all clearer in my head now.
Best,
Juan Luis
On Fri, Feb 25, 2022 at 8:30 PM Juan Luis Cano Rodríguez <hello@juanlu.space> wrote:
Hi all,
I have been coaching a new contributor to SciPy
Thanks for doing that!
and we are being routinely bitten by weird problems on import when trying to follow the development installation instructions.
We have too many flavors of the instructions unfortunately, and not all of them are 100% up-to-date.
At this point, I'm not so concerned about getting things to work: I know that, with enough trial-and-error, we will manage at some point. Instead, I'd like to try to have a more solid mental model of why things fail (if that's even possible).
I am undecided and hesitant on how to proceed here since I feel I'm trying to solve too many problems at the same time. In no particular order:
1) I could try to get help on the specific errors we're getting. However, these are of the difficult-to-debug kind (undefined symbols and such) and I anticipate that we would end up in something like "you have a borked SciPy installation, please start from scratch" or "use conda instead".
Please note that NumPy 1.22.0/1.22.1 had a few serious issues that resulted in undefined symbol problems. Those should all have been solved in 1.22.2, so please ensure you use that. Or alternatively, start using the Meson build so you won't run into those issues in the first place - see http://scipy.github.io/devdocs/dev/contributor/meson.html.
Using conda is indeed the best idea, because it's the only way to give uniform cross-platform instructions. If you want to install all build dependencies (Python, BLAS/LAPACK, compilers) otherwise, you'll likely have to figure out some of the details for yourself depending on which Linux distro you use (or macOS - but definitely do not try to develop on Windows except for with WSL).
2) I'd love to get a better understanding of what's the role of `conda develop .` in the development instructions, and how it's different from `pip install -e .`. I tried to look around and the only references I see about `conda develop` are the manual pages, and an old issue pointing to lack of documentation
https://github.com/conda/conda-build/issues/1992
`conda develop .` is useless and should be cleaned up in the docs if it's still present. Please also do not use `pip install . -e`; best not to use `pip` at all, but if you do then you should add `--no-build-isolation`. Pip is *not* a good developer CLI, using `python setup.py develop` instead.
3) I've been asking myself whether SciPy would consider moving to an src-layout. I did a quick search on the issue tracker and the mailing list archives, and don't see any references. I know this is a contentious topic though, so it would be a pity to start a flame war about it, and that's definitely not my intention. I'm just wondering if at least it would help us avoid a family of problems: the ones appearing when folks do `import scipy` from the root directory.
I'd say no, we won't consider that - way too much churn for little gain.
Part of the issue with the dev instructions is that `python runtests.py` and in-place builds are both used and are two incompatible options. With Meson we go to a single interface (currently `python dev.py`), inplace builds are not supported (at least for now). `runtests.py` is recommended by us in most places since it avoids things like issues with `import scipy` from the root of the repo.
4) We could continue the conversation we started in
https://github.com/scipy/scipy/issues/12633 and try to see if there's a "happy path" that folks can follow given some assumptions (modern Linux distro/WSL2, compilers coming from either apt or conda-forge, no dangling Python installations) to reduce the installation instructions to the minimum (which could be `pip install -e . && pytest`).
We're moving to Meson, and the instructions are:1. Set up your environment with all dev dependencies and activate it.2. `python dev.py` We will keep `setup.py` around for one more release as a backup, but those instructions are all going to be updated.
Apologies if this rambling is not particularly helpful, hope I can get some help on these various points.
No worries, I totally understand the pain, we regularly have maintainers who cannot build SciPy either. It's difficult and fragile.
Cheers,Ralf
_______________________________________________
SciPy-Dev mailing list -- scipy-dev@python.org
To unsubscribe send an email to scipy-dev-leave@python.org
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: hello@juanlu.space
SciPy-Dev mailing list -- scipy-dev@python.org
To unsubscribe send an email to scipy-dev-leave@python.org
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: ralf.gommers@googlemail.com