plan for moving to Meson
Hi all, With distutils now removed from the stdlib in the Python 3.12 release cycle, the clock is ticking a bit for dealing with our build system situation. With SciPy's move to Meson now basically complete - there are always loose ends & improvements, but the 1.9 releases have gone well - it's time to look at NumPy doing the same thing. Scikit-image has also merged support for Meson, and Pandas is about to. So all the generic & Python packaging related things we need should be in place. NumPy does have some hairy stuff of course that those other projects don't have (lots of config checks and platform-specific behavior, extensive SIMD support). It shouldn't be too difficult to get a baseline build - Linux/macOS with baseline SIMD flags - working, but there'll be a long tail of things that are hard to test or will need to be upstreamed to Meson. Here is a tracking issue where I wrote up a plan for how to approach the transition: https://github.com/numpy/numpy/issues/22546. Linked from there is a GitHub project board that we plan to use to divide up the work. And there is a `meson` branch in the repo with a start to the implementation. NumPy should default to building with Meson in the 1.25.0 release, which should also still build with `numpy.distutils`. And `numpy.distutils` will continue to be shipped for Python <3.12 for a while after that, until we determine that it's no longer needed. If anyone has ideas or concerns regarding the current plan, it'd be great to hear them - here or on the issue. Cheers, Ralf
On Fri, 2022-11-11 at 12:27 +0100, Ralf Gommers wrote:
Thanks for working on this! I fully trusting you, Stéfan, and everyone else involved to push this forward. It seems like most of the decisions were really made for us and the open points might be mainly about the how and maybe details about the what. To me it would also seem fine if you work on the main branch, but I guess that would just make things harder to iterate at this point. Since "NEP" was mentioned somewhere. There would be two goals of having a short one: 1. Settling on a specific approach for our build system 2. Informing users (about breakages/workarounds and how things work) I doubt you/we need it for 1. at this point or details for why meson, but if anyone disagrees then maybe we do ;). But 2. may very much be worthwhile. - Sebastian PS: I do like the idea of having short NEPs. My feeling is that the "painful" ones are the ones that are technically tricky. For those it is clear that they are necessary. The other issue is about document intent (sometimes a NEP may be more of a roadmap proposal then a concrete implementation one).
On Fri, Nov 11, 2022 at 1:52 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
I think there will be quite a few decisions on the way that different people may want to weigh in on. So we need kind of a "decision log" somewhere and a way to bubble up those decisions - which fits with the NEP idea (as long as it remains draft and can be extended). To give you a couple examples just from spending a few hours on build system changes today: (1) I noticed we still have a `oldnumeric.h` header with a comment that it can probably be removed. So I used the new GitHub code search, found the two actively developed projects that still use it, and asked them to get rid of their usages so that we can indeed safely remove it. And then added code + comment in the meson branch to not install it. (2) a more important one, the `.c.src` format. In SciPy we got rid of it, and we're not going to make Meson understand an ad-hoc templating method that only NumPy uses. So we have two choices: also get rid of it, or write a new custom preprocessing utility for NumPy's Meson build. I think we have too much code using it to remove it, so probably should go with the "new utility" option. But in case we did want to get rid of it, now would be a good time.
To me it would also seem fine if you work on the main branch, but I guess that would just make things harder to iterate at this point.
Yes, that'd be too disruptive at this point.
We should have regular html docs for everything, so the NEP on that point would be more an FYI with a pointer to those docs.
That does make sense to me. NEPs are not good as documentation - we should have more design & architecture docs that are maintained over time for that - but to document intent at the time a change was made they are quite nice. Cheers, Ralf
(2) a more important one, the `.c.src` format. In SciPy we got rid of it, and we're not going to make Meson understand an ad-hoc templating method that only NumPy uses. So we have two choices: also get rid of it, or write a new custom preprocessing utility for NumPy's Meson build. I think we have too much code using it to remove it, so probably should go with the "new utility" option. But in case we did want to get rid of it, now would be a good time.
As a comment from a peanut gallery, where the project board is not even visible (https://github.com/orgs/numpy/projects/7/views/7 404s for me --- it would be perfectly understandable if you prefer to keep it visible to select individuals!), and it has probably been discussed before: any thoughts to change it to e.g. tempita templating? Translating .c.src templates to .c.in is straightforward if tedious, as e.g. SciPy transition showed. This is of course quite a bit of work, but so is a new utility. Again, just throwing it out there :-). Cheers, Evgeni
On Fri, Nov 11, 2022 at 3:43 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Yes, definitely intended, thanks for fixing that. I didn't notice that it was set up as private, that's just my inexperience with the new GitHub Projects interface. Cheers, Ralf
The utility isn't from-scratch new, it's more the boring job of refactoring one of the numpy.distutils parts that should survive as a standalone thing. Cheers, Ralf
On Fri, 2022-11-11 at 12:27 +0100, Ralf Gommers wrote:
Thanks for working on this! I fully trusting you, Stéfan, and everyone else involved to push this forward. It seems like most of the decisions were really made for us and the open points might be mainly about the how and maybe details about the what. To me it would also seem fine if you work on the main branch, but I guess that would just make things harder to iterate at this point. Since "NEP" was mentioned somewhere. There would be two goals of having a short one: 1. Settling on a specific approach for our build system 2. Informing users (about breakages/workarounds and how things work) I doubt you/we need it for 1. at this point or details for why meson, but if anyone disagrees then maybe we do ;). But 2. may very much be worthwhile. - Sebastian PS: I do like the idea of having short NEPs. My feeling is that the "painful" ones are the ones that are technically tricky. For those it is clear that they are necessary. The other issue is about document intent (sometimes a NEP may be more of a roadmap proposal then a concrete implementation one).
On Fri, Nov 11, 2022 at 1:52 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
I think there will be quite a few decisions on the way that different people may want to weigh in on. So we need kind of a "decision log" somewhere and a way to bubble up those decisions - which fits with the NEP idea (as long as it remains draft and can be extended). To give you a couple examples just from spending a few hours on build system changes today: (1) I noticed we still have a `oldnumeric.h` header with a comment that it can probably be removed. So I used the new GitHub code search, found the two actively developed projects that still use it, and asked them to get rid of their usages so that we can indeed safely remove it. And then added code + comment in the meson branch to not install it. (2) a more important one, the `.c.src` format. In SciPy we got rid of it, and we're not going to make Meson understand an ad-hoc templating method that only NumPy uses. So we have two choices: also get rid of it, or write a new custom preprocessing utility for NumPy's Meson build. I think we have too much code using it to remove it, so probably should go with the "new utility" option. But in case we did want to get rid of it, now would be a good time.
To me it would also seem fine if you work on the main branch, but I guess that would just make things harder to iterate at this point.
Yes, that'd be too disruptive at this point.
We should have regular html docs for everything, so the NEP on that point would be more an FYI with a pointer to those docs.
That does make sense to me. NEPs are not good as documentation - we should have more design & architecture docs that are maintained over time for that - but to document intent at the time a change was made they are quite nice. Cheers, Ralf
(2) a more important one, the `.c.src` format. In SciPy we got rid of it, and we're not going to make Meson understand an ad-hoc templating method that only NumPy uses. So we have two choices: also get rid of it, or write a new custom preprocessing utility for NumPy's Meson build. I think we have too much code using it to remove it, so probably should go with the "new utility" option. But in case we did want to get rid of it, now would be a good time.
As a comment from a peanut gallery, where the project board is not even visible (https://github.com/orgs/numpy/projects/7/views/7 404s for me --- it would be perfectly understandable if you prefer to keep it visible to select individuals!), and it has probably been discussed before: any thoughts to change it to e.g. tempita templating? Translating .c.src templates to .c.in is straightforward if tedious, as e.g. SciPy transition showed. This is of course quite a bit of work, but so is a new utility. Again, just throwing it out there :-). Cheers, Evgeni
On Fri, Nov 11, 2022 at 3:43 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Yes, definitely intended, thanks for fixing that. I didn't notice that it was set up as private, that's just my inexperience with the new GitHub Projects interface. Cheers, Ralf
The utility isn't from-scratch new, it's more the boring job of refactoring one of the numpy.distutils parts that should survive as a standalone thing. Cheers, Ralf
participants (4)
-
Evgeni Burovski -
Ralf Gommers -
Sebastian Berg -
Stefan van der Walt