Meson - C extension - Finding numpy includes in virtual env
I am converting a Numpy C extension Python project from distutils to meson. I've been following along the meson-python instructions ( https://meson-python.readthedocs.io/en/latest/tutorials/introduction.html) I've installed meson / ninja (and everything else...) into a virtualenv (python -m venv venv). Running Python 3.11 Unfortunately the following command fails: incdir_numpy = run_command(py, ['-c', 'import numpy; print(numpy.get_include())'], capture: true, check: false, ).stdout().strip() Capturing stderr, gives the error:
ModuleNotFoundError: No module named 'numpy' does not exist.
But If I hardcode incdir_numpy to my actual location, it works fine incdir_numpy = 'venv/lib/python3.11/site-packages/numpy/core/include' I've tried deactivating / activating the venv and rebuilding, no such luck. I feel like I'm missing something obvious about meson / numpy and virtual environments? My changes are here https://github.com/softwaredoug/np-sims/compare/meson?expand=1#diff-30d8f6be...
Hi Doug, On Sat, Nov 25, 2023, at 07:14, Doug Turnbull wrote:
Unfortunately the following command fails:
incdir_numpy = run_command(py, ['-c', 'import numpy; print(numpy.get_include())'], capture: true, check: false, ).stdout().strip()
In your repo it says stderr, but the version above (stdout) works for me. Perhaps you are using a different Python than the one in your virtual env, because meson was installed onto your path previously? Try `python -m pip install meson` and then invoking the meson binary directly from your virtualenv: venv/bin/meson. Stéfan
Thanks for taking the time Stefan 1. I had left in the stderr to print out the message (for whatever reason the log files weren't there) The two big things I'm noticing First, I'm attempting to build with "pip install ." or "python -m pip install ." I can confirm that the virtualenv python is used, but the sys.path is very messed up, and does not include the virtual environment. However, this does not occur if I simply run "meson build". With some jiggering of numpy's include path, it seems to build fine, with the correct sys.path So the fundamental issue is something happens with pip install . and sys.path, where the virtualenv isn't included. I don't think this is a numpy issue. Though I welcome any input / ideas / anything obvious I'm missing. It feels more like a meson-python issue, so I may follow up there. Thanks -Doug On Sat, Nov 25, 2023 at 5:30 PM Stefan van der Walt via NumPy-Discussion < numpy-discussion@python.org> wrote:
Hi Doug,
On Sat, Nov 25, 2023, at 07:14, Doug Turnbull wrote:
Unfortunately the following command fails:
incdir_numpy = run_command(py, ['-c', 'import numpy; print(numpy.get_include())'], capture: true, check: false, ).stdout().strip()
In your repo it says stderr, but the version above (stdout) works for me.
Perhaps you are using a different Python than the one in your virtual env, because meson was installed onto your path previously? Try `python -m pip install meson` and then invoking the meson binary directly from your virtualenv: venv/bin/meson.
Stéfan
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: softwaredoug@gmail.com
OK I'm an idiot, and learned a bit about packaging To debug, I ran `pip install . --no-build-isolation` it worked (using venv's numpy) I realized then 🤦 after visiting meson-python docs I needed to add "numpy" as a dependency in pyproject.toml. And that pip running with build isolation uses a different path / environment to gather dependencies (such as numpy). Not my virtualenv. Thanks for being a rubber ducky mailing list 🐥 - Doug On Sun, Nov 26, 2023 at 9:10 AM Doug Turnbull <softwaredoug@gmail.com> wrote:
Thanks for taking the time Stefan
1. I had left in the stderr to print out the message (for whatever reason the log files weren't there)
The two big things I'm noticing
First, I'm attempting to build with "pip install ." or "python -m pip install ." I can confirm that the virtualenv python is used, but the sys.path is very messed up, and does not include the virtual environment.
However, this does not occur if I simply run "meson build". With some jiggering of numpy's include path, it seems to build fine, with the correct sys.path
So the fundamental issue is something happens with pip install . and sys.path, where the virtualenv isn't included.
I don't think this is a numpy issue. Though I welcome any input / ideas / anything obvious I'm missing. It feels more like a meson-python issue, so I may follow up there.
Thanks -Doug
On Sat, Nov 25, 2023 at 5:30 PM Stefan van der Walt via NumPy-Discussion < numpy-discussion@python.org> wrote:
Hi Doug,
On Sat, Nov 25, 2023, at 07:14, Doug Turnbull wrote:
Unfortunately the following command fails:
incdir_numpy = run_command(py, ['-c', 'import numpy; print(numpy.get_include())'], capture: true, check: false, ).stdout().strip()
In your repo it says stderr, but the version above (stdout) works for me.
Perhaps you are using a different Python than the one in your virtual env, because meson was installed onto your path previously? Try `python -m pip install meson` and then invoking the meson binary directly from your virtualenv: venv/bin/meson.
Stéfan
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: softwaredoug@gmail.com
Hi Doug, On Sun, Nov 26, 2023, at 06:29, Doug Turnbull wrote:
To debug, I ran `pip install . --no-build-isolation` it worked (using venv's numpy)
When developing NumPy, we typically build in the existing environment. This is done either via `pip install -e .` (which installs hooks to trigger a re-compile upon import), or via the spin tool (https://github.com/scientific-python/spin), which have meson commands pre-bundled: pip install spin spin # lists commands available Best regards, Stéfan
I want to caution about using `pip install -e .` to get a development install of numpy. This will work fine working on numpy itself, but won’t be useful if you need to use the development version of numpy to build another library. This doesn’t work because in-place installs don’t install the numpy headers (arguably it was a bug that the old setuptools install did) into the git repo, so the include paths `np.get_include()` reports won’t be correct. See this meson-python issue: https://github.com/mesonbuild/meson-python/issues/429 For my work I tend to use a persistent build directory with build isolation disabled as discussed in the meson-python docs. This gives me fast rebuilds without using an in-place build. It does mean there’s a build and install step when you edit python code in numpy that would otherwise be unnecessary and sometimes the cache can go stale for reasons that aren’t totally obvious. In principle numpy could fix this by ensuring the headers get generated in the git repo in the place they’re supposed to be installed. I have no idea how hard it would be beyond that it would definitely require messing with the codegen scripts. On Sun, Nov 26, 2023 at 10:53 AM Stefan van der Walt via NumPy-Discussion < numpy-discussion@python.org> wrote:
Hi Doug,
On Sun, Nov 26, 2023, at 06:29, Doug Turnbull wrote:
To debug, I ran `pip install . --no-build-isolation` it worked (using venv's numpy)
When developing NumPy, we typically build in the existing environment. This is done either via `pip install -e .` (which installs hooks to trigger a re-compile upon import), or via the spin tool ( https://github.com/scientific-python/spin), which have meson commands pre-bundled:
pip install spin spin # lists commands available
Best regards, Stéfan
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: nathan12343@gmail.com
It looks like `spin build` does `meson build` and `meson install` and doesn't do `pip install`. I'd like numpy to be importable in a python environment of my choosing, so I tend to instead manually install numpy into that environment by invoking pip with something like `python -m pip install . -v --no-build-isolation -Cbuilddir=build -C'compile_args=-v' -C'setup_args=-Dbuildtype=debug'. I like seeing the compile command meson uses, so I pass in `-v` through meson's `compile_args` and I often need a debug build, so I set the build type manually as well. I could probably get the same effect by either manually activating the spin python environment (not sure how to do that) or using `spin run` somehow outside of the numpy tree, but what I have seems to work OK for me now so I haven't tried to mess with spin more. On Sun, Nov 26, 2023 at 1:57 PM Stefan van der Walt <stefanv@berkeley.edu> wrote:
On Sun, Nov 26, 2023, at 12:03, Nathan wrote:
For my work I tend to use a persistent build directory with build isolation disabled as discussed in the meson-python docs.
Out of curiosity, how is this different from, e.g., `spin build` which builds into `./build-install`?
Stéfan
Hi Nathan, On Tue, Nov 28, 2023, at 08:42, Nathan wrote:
It looks like `spin build` does `meson build` and `meson install` and doesn't do `pip install`. I'd like numpy to be importable in a python environment of my choosing, so I tend to instead manually install numpy into that environment by invoking pip with something like `python -m pip install . -v --no-build-isolation -Cbuilddir=build -C'compile_args=-v' -C'setup_args=-Dbuildtype=debug'. I like seeing the compile command meson uses, so I pass in `-v` through meson's `compile_args` and I often need a debug build, so I set the build type manually as well.
That makes sense. We recently added the `spin.pip.install` command for that purpose, but of course you don't *need* a command if you know the invocation :) Stéfan
On Sun, Nov 26, 2023 at 9:06 PM Nathan <nathan.goldbaum@gmail.com> wrote:
I want to caution about using `pip install -e .` to get a development install of numpy. This will work fine working on numpy itself, but won’t be useful if you need to use the development version of numpy to build another library. This doesn’t work because in-place installs don’t install the numpy headers (arguably it was a bug that the old setuptools install did) into the git repo, so the include paths `np.get_include()` reports won’t be correct.
This sounds about right - editable installs are only useful for working on `numpy` itself, and perhaps basic pure Python packages on top. But editable installs are fundamentally not complete installs, and should not be used for developing a set of packages together that depend on each other. I'll note that there's an active discussion (PEP 735 draft) which touched on a "workspaces" concept for this. Also, please do not ever use editable installs without `--no-build-isolation`, that may lead to weird issues. Cheers, Ralf
See this meson-python issue: https://github.com/mesonbuild/meson-python/issues/429
For my work I tend to use a persistent build directory with build isolation disabled as discussed in the meson-python docs. This gives me fast rebuilds without using an in-place build. It does mean there’s a build and install step when you edit python code in numpy that would otherwise be unnecessary and sometimes the cache can go stale for reasons that aren’t totally obvious.
In principle numpy could fix this by ensuring the headers get generated in the git repo in the place they’re supposed to be installed. I have no idea how hard it would be beyond that it would definitely require messing with the codegen scripts.
On Sun, Nov 26, 2023 at 10:53 AM Stefan van der Walt via NumPy-Discussion < numpy-discussion@python.org> wrote:
Hi Doug,
On Sun, Nov 26, 2023, at 06:29, Doug Turnbull wrote:
To debug, I ran `pip install . --no-build-isolation` it worked (using venv's numpy)
When developing NumPy, we typically build in the existing environment. This is done either via `pip install -e .` (which installs hooks to trigger a re-compile upon import), or via the spin tool ( https://github.com/scientific-python/spin), which have meson commands pre-bundled:
pip install spin spin # lists commands available
Best regards, Stéfan
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: nathan12343@gmail.com
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ralf.gommers@gmail.com
participants (4)
-
Doug Turnbull -
Nathan -
Ralf Gommers -
Stefan van der Walt