Endorsing SPECs 1, 6, 7, and 8
Hi all, TL;DR: NumPy should endorse some or all of the new SPECs if we like them. If you don't or do like them, please discuss, otherwise I suspect we will propose and endorsing them soon and do it if a few core maintainers agree. --- The Scientific Python project has the SPEC process to write up helpful tips/patterns [1] with the idea that projects "endorse" them to indicate that we think following the SPEC is a good idea. One example probably known best is SPEC 0, which is NEP 27 (the suggested minimum supported versions for dependencies, e.g. Python/library versions). There has been a lot of work recently to write some new SPECs, and the following ones are up for endorsement by NumPy: * 1 -- Lazy Loading of Submodules and Functions https://scientific-python.org/specs/spec-0001 * 6 -- Keys to the Castle https://scientific-python.org/specs/spec-0006/ * 7 -- Seeding Pseudo-Random Number Generation https://scientific-python.org/specs/spec-0007/ * 8 -- Securing the Release Process https://scientific-python.org/specs/spec-0008/ Endorsing them means we think they are good guidance for projects, not necessarily strictly following them [2]. Without input, I suspect we may endorse them soon (if a few maintainers give a thumbs up). But of course it would be great to discuss and maybe improve the SPECs in the progress! SPEC 7 is important, because it describes how we would like other libraries to use the new random number generation API in the future (but actually doesn't apply directly). Cheers, Sebastian [1] The SPECs are not necessarily fixed. If you think they are missing a note/solution, we can modify them. [2] With it's flat namespace, I am not sure e.g. lazy-loading is of much use to NumPy (just like we have longer compatibility than SPEC 0).
It seems to me that we should only endorse SPECs that we ourselves implement, otherwise it is kind of "do as I say, not as I do". For instance, it would be strange to endorse SPEC0 but stay with NEP 29. If we are to endorse SPEC0 without changing our version end-of-life timing, we should at least modify NEP 29 with some commentary about why we chose not to implementing SPEC0. If a SPEC is not relevant, then I don't think the NumPy project (as a project) can have an opinion on whether it is "good" for other projects. Individual contributors can of course endorse whatever they want, but as a project we should only weigh in when it is relevant to our community experience Matti On Mon, Oct 7, 2024 at 1:04 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
TL;DR: NumPy should endorse some or all of the new SPECs if we like them. If you don't or do like them, please discuss, otherwise I suspect we will propose and endorsing them soon and do it if a few core maintainers agree. ... Cheers,
Sebastian
On Mon, 2024-10-07 at 15:52 +0300, matti picus via NumPy-Discussion wrote:
It seems to me that we should only endorse SPECs that we ourselves implement, otherwise it is kind of "do as I say, not as I do". For instance, it would be strange to endorse SPEC0 but stay with NEP 29. If we are to endorse SPEC0 without changing our version end-of-life timing, we should at least modify NEP 29 with some commentary about why we chose not to implementing SPEC0. If a SPEC is not relevant,
I thought we always had in practice longer support than NEP 29 as well and it lived in NumPy. There was some change due the Python release cadence, IIRC, but wasn't that addressed in the SPEC 0 document? FWIW, even if we have slightly longer times and it thus isn't "adopted 100", I think it does inform us with respect to deprecation timelines? For SPEC 7, I think we certainly should endorse it, exactly because it is a recommendation for downstream. Even though we have no API where it applies. I don't have not formed a strong opinion about the other two yet. For those, I would agree that it may be a bit odd to "endorse" but not plan to adopt them at least in parts. - Sebastian
then I don't think the NumPy project (as a project) can have an opinion on whether it is "good" for other projects. Individual contributors can of course endorse whatever they want, but as a project we should only weigh in when it is relevant to our community experience Matti
On Mon, Oct 7, 2024 at 1:04 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
TL;DR: NumPy should endorse some or all of the new SPECs if we like them. If you don't or do like them, please discuss, otherwise I suspect we will propose and endorsing them soon and do it if a few core maintainers agree. ... Cheers,
Sebastian
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: sebastian@sipsolutions.net
I second Matti's comments about the validity of endorsing things we don't implement. Also, personally I really dislike the keys to castle spec, because I'm generally against having yearly check in reviews and such. --- Rohit ________________________________ From: Sebastian Berg <sebastian@sipsolutions.net> Sent: Monday, October 7, 2024 02:23 To: Discussion of Numerical Python Subject: [Numpy-discussion] Endorsing SPECs 1, 6, 7, and 8 Hi all, TL;DR: NumPy should endorse some or all of the new SPECs if we like them. If you don't or do like them, please discuss, otherwise I suspect we will propose and endorsing them soon and do it if a few core maintainers agree. --- The Scientific Python project has the SPEC process to write up helpful tips/patterns [1] with the idea that projects "endorse" them to indicate that we think following the SPEC is a good idea. One example probably known best is SPEC 0, which is NEP 27 (the suggested minimum supported versions for dependencies, e.g. Python/library versions). There has been a lot of work recently to write some new SPECs, and the following ones are up for endorsement by NumPy: * 1 -- Lazy Loading of Submodules and Functions https://scientific-python.org/specs/spec-0001 * 6 -- Keys to the Castle https://scientific-python.org/specs/spec-0006/ * 7 -- Seeding Pseudo-Random Number Generation https://scientific-python.org/specs/spec-0007/ * 8 -- Securing the Release Process https://scientific-python.org/specs/spec-0008/ Endorsing them means we think they are good guidance for projects, not necessarily strictly following them [2]. Without input, I suspect we may endorse them soon (if a few maintainers give a thumbs up). But of course it would be great to discuss and maybe improve the SPECs in the progress! SPEC 7 is important, because it describes how we would like other libraries to use the new random number generation API in the future (but actually doesn't apply directly). Cheers, Sebastian [1] The SPECs are not necessarily fixed. If you think they are missing a note/solution, we can modify them. [2] With it's flat namespace, I am not sure e.g. lazy-loading is of much use to NumPy (just like we have longer compatibility than SPEC 0). _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: rgoswami@quansight.com
On Mon, Oct 7, 2024, at 06:04, Rohit Goswami wrote:
I second Matti's comments about the validity of endorsing things we don't implement.
I don't think it is possible to make ecosystem-wide recommendation that will fit each project like a glove. At best, we can try to come together as a community, make sound recommendations, and accept that there will be exceptions depending on circumstances. And those exceptions may well apply to NumPy. E.g., being at the bottom of the stack, the NumPy project may recommend the drop schedules from SPEC0 for other projects, but may implement a different strategy to ensure wider compatibility.
Also, personally I really dislike the keys to castle spec, because I'm generally against having yearly check in reviews and such.
The SPECs are living documents, and are constructed based on input from the community. It would therefore be good to better understand your concern. Is it with the sentence "Review permissions regularly (say, every year) to maintain minimal permissions."? Having written that SPEC, to me that obviously feels like a fairly pragmatic, low-cost recommendation; but perhaps there are better ways to accomplish the same goal. An issue on https://github.com/scientific-python/specs or the thread at https://discuss.scientific-python.org/t/spec-6-keys-to-the-castle/777/2 could be good venues for further discussion. Best regards, Stéfan
Regarding thread safety - that's not a problem. At least for Python 3.13, the GIL is temporarily re-enabled during imports. That won't necessarily be true in the future, but separately CPython also uses per-module locks on import, so there shouldn't be any issues with threads simultaneously importing submodules. It looks like we already implement lazy-loading for e.g. linalg, fft, random, and other submodules. Does that lazy-loading mechanism conform to the SPEC? If not, should it? The keys to the castle SPEC makes sense to me, I'm fine with endorsing it. I believe that all of NumPy's online accounts are already spread out over multiple maintainers, so presumably we don't actually need to do much here to implement it? Since the legacy RNG interface cannot be deprecated and we encourage downstream to use it in tests according to the text of NEP 19, I'm not sure about the text in SPEC 7 that talks about deprecating using legacy RNGs. Or are you saying that we have now reached the point where we can update NEP 19 to encourage moving away from the legacy interface? From the text of NEP 19 regarding the legacy RNG interface:
This NEP does not propose that these requirements remain in perpetuity. After we have experience with the new PRNG subsystem, we can and should revisit these issues in future NEPs.
I don't have a problem with SPEC 8, although I suspect there might be a fair bit of work to get NumPy's CI to match the suggestions in the SPEC. On Tue, Oct 8, 2024 at 2:08 PM Joren Hammudoglu via NumPy-Discussion < numpy-discussion@python.org> wrote:
Is SPEC 1 thread-safe enough for py313+nogil? _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: nathan12343@gmail.com
On Tue, Oct 8, 2024 at 8:36 AM Nathan via NumPy-Discussion < numpy-discussion@python.org> wrote:
Since the legacy RNG interface cannot be deprecated and we encourage downstream to use it in tests according to the text of NEP 19, I'm not sure about the text in SPEC 7 that talks about deprecating using legacy RNGs. Or are you saying that we have now reached the point where we can update NEP 19 to encourage moving away from the legacy interface?
We have already always encouraged people to move away from the legacy interface in their APIs. SPEC 7 recommends a principled way for downstream projects to implement that move. NEP 19 acknowledged that sometimes one might still have a use case for creating a legacy RandomState object and calling it in their tests to generate test data (but not otherwise pass that RandomState object to the code under test), but that's not what SPEC 7 addresses. NEP 19 doesn't really actively recommend the use of RandomState for this purpose, just acknowledges that it's a valid use case that numpy will continue to support even while we push for the exclusive use of Generator inside of library/program code. NEP 19 doesn't need an update for us to endorse SPEC 7 (whether it needs one, separately, to clarify its intent is another question). -- Robert Kern
Thanks for clarifying! In that case I think endorsing SPEC 7 makes sense. On Tue, Oct 8, 2024 at 3:08 PM Robert Kern <robert.kern@gmail.com> wrote:
On Tue, Oct 8, 2024 at 8:36 AM Nathan via NumPy-Discussion < numpy-discussion@python.org> wrote:
Since the legacy RNG interface cannot be deprecated and we encourage downstream to use it in tests according to the text of NEP 19, I'm not sure about the text in SPEC 7 that talks about deprecating using legacy RNGs. Or are you saying that we have now reached the point where we can update NEP 19 to encourage moving away from the legacy interface?
We have already always encouraged people to move away from the legacy interface in their APIs. SPEC 7 recommends a principled way for downstream projects to implement that move.
NEP 19 acknowledged that sometimes one might still have a use case for creating a legacy RandomState object and calling it in their tests to generate test data (but not otherwise pass that RandomState object to the code under test), but that's not what SPEC 7 addresses. NEP 19 doesn't really actively recommend the use of RandomState for this purpose, just acknowledges that it's a valid use case that numpy will continue to support even while we push for the exclusive use of Generator inside of library/program code. NEP 19 doesn't need an update for us to endorse SPEC 7 (whether it needs one, separately, to clarify its intent is another question).
-- Robert Kern
Below is the input on this topic from Christian Lorentzen, who for some reason didn't get through the mailing list approval to post directly: Hi there IMHO, numpy should definitely endorse SPEC7, otherwise this SPEC should be removed or updated. About SPEC0, please note that according to the text in https://scientific-python.org/specs/spec-0000/ numpy already endorses it which I find strange (maybe a misunderstanding). In scikit-learn, we had a discussion about SPEC0 and decided to not endorse it because we want longer support for older versions. The extra cost so far is small enough. On Mon, Oct 7, 2024 at 12:05 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,
TL;DR: NumPy should endorse some or all of the new SPECs if we like them. If you don't or do like them, please discuss, otherwise I suspect we will propose and endorsing them soon and do it if a few core maintainers agree.
---
The Scientific Python project has the SPEC process to write up helpful tips/patterns [1] with the idea that projects "endorse" them to indicate that we think following the SPEC is a good idea.
One example probably known best is SPEC 0, which is NEP 27 (the suggested minimum supported versions for dependencies, e.g. Python/library versions).
There has been a lot of work recently to write some new SPECs, and the following ones are up for endorsement by NumPy:
* 1 -- Lazy Loading of Submodules and Functions https://scientific-python.org/specs/spec-0001 * 6 -- Keys to the Castle https://scientific-python.org/specs/spec-0006/ * 7 -- Seeding Pseudo-Random Number Generation https://scientific-python.org/specs/spec-0007/ * 8 -- Securing the Release Process https://scientific-python.org/specs/spec-0008/
Endorsing them means we think they are good guidance for projects, not necessarily strictly following them [2].
Without input, I suspect we may endorse them soon (if a few maintainers give a thumbs up). But of course it would be great to discuss and maybe improve the SPECs in the progress!
SPEC 7 is important, because it describes how we would like other libraries to use the new random number generation API in the future (but actually doesn't apply directly).
Cheers,
Sebastian
[1] The SPECs are not necessarily fixed. If you think they are missing a note/solution, we can modify them.
[2] With it's flat namespace, I am not sure e.g. lazy-loading is of much use to NumPy (just like we have longer compatibility than SPEC 0).
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ralf.gommers@gmail.com
participants (8)
-
Joren Hammudoglu -
matti picus -
Nathan -
Ralf Gommers -
Robert Kern -
Rohit Goswami -
Sebastian Berg -
Stefan van der Walt