NEP 29 and the faster Python release cadence
Was the faster CPython release cadence (PEP 602 https://peps.python.org/pep-0602/) ever discussed in relation to NEP 29 (https://numpy.org/neps/nep-0029-deprecation_policy.html)? NEP 29 currently says: "The current Python release cadence is 18 months so a 42 month window ensures that there will always be at least two minor versions of Python in the window." However, as of PEP 602, the release cadence of CPython is now 12 months, not 18 months. This schedule now means supporting 3-4 versions of Python at once, rather than 2-3 versions (for those who don't want to do the math, 48/18 is 2.3 and 48/12 is 3.5). PEP 602 started with Python 3.8, which was released in October, 2019. Python 3.8 is currently the oldest version supported in NEP 29, so we are about to enter the full cycle of the new cadence. In particular, when Python 3.11 is released in October of this year, the NEP 29 prescription will imply supporting 4 Python versions: 3.8, 3.9, 3.10, and 3.11, up until April, 2023 when 3.8 support would be dropped (and continuing like this: 3 versions between April and October; 4 versions between October and April). Furthermore, the NEP seems to imply that the exact CPython release schedule is not known: "The window is extended 6 months beyond the anticipated two-release interval for Python to provide resilience against small fluctuations / delays in its release schedule." I'm not sure how true this was before, but PEP 602 pretty clearly prescribes a Python release in October of every year. Delays presumably might still be possible, but I don't see why it's necessary to build that possibility into the default period in the NEP given that it should be an unusual situation. Personally I would prefer not ever supporting 4 simultaneous Python versions (I rather wish Python hadn't increased their cadence, but that ship has sailed). I don't know if it's possible to update the NEP to adjust for the new cadence. Even if it isn't, the text is clearly out of date and should be updated. I wasn't on this list back in 2019, so my apologies if this has already been discussed. Aaron Meurer
On Wed, Apr 6, 2022 at 3:06 PM Aaron Meurer <asmeurer@gmail.com> wrote:
Was the faster CPython release cadence (PEP 602 https://peps.python.org/pep-0602/) ever discussed in relation to NEP 29 (https://numpy.org/neps/nep-0029-deprecation_policy.html)?
NEP 29 currently says:
"The current Python release cadence is 18 months so a 42 month window ensures that there will always be at least two minor versions of Python in the window."
However, as of PEP 602, the release cadence of CPython is now 12 months, not 18 months. This schedule now means supporting 3-4 versions of Python at once, rather than 2-3 versions (for those who don't want to do the math, 48/18 is 2.3 and 48/12 is 3.5).
PEP 602 started with Python 3.8, which was released in October, 2019. Python 3.8 is currently the oldest version supported in NEP 29, so we are about to enter the full cycle of the new cadence. In particular, when Python 3.11 is released in October of this year, the NEP 29 prescription will imply supporting 4 Python versions: 3.8, 3.9, 3.10, and 3.11, up until April, 2023 when 3.8 support would be dropped (and continuing like this: 3 versions between April and October; 4 versions between October and April).
Furthermore, the NEP seems to imply that the exact CPython release schedule is not known: "The window is extended 6 months beyond the anticipated two-release interval for Python to provide resilience against small fluctuations / delays in its release schedule." I'm not sure how true this was before, but PEP 602 pretty clearly prescribes a Python release in October of every year. Delays presumably might still be possible, but I don't see why it's necessary to build that possibility into the default period in the NEP given that it should be an unusual situation.
Personally I would prefer not ever supporting 4 simultaneous Python versions (I rather wish Python hadn't increased their cadence, but that ship has sailed). I don't know if it's possible to update the NEP to adjust for the new cadence. Even if it isn't, the text is clearly out of date and should be updated.
My current plan is to support 4 Python versions in the 1.24 NumPy release cycle, then drop Python 3.8 in NumPy 1.25. The result going forward is that we will oscillate between supporting 3 and 4 Python versions, each version being supported for about 42 months.
I wasn't on this list back in 2019, so my apologies if this has already been discussed.
Chuck
On Thu, Apr 7, 2022 at 5:42 PM Charles R Harris <charlesr.harris@gmail.com> wrote:
On Wed, Apr 6, 2022 at 3:06 PM Aaron Meurer <asmeurer@gmail.com> wrote:
Was the faster CPython release cadence (PEP 602 https://peps.python.org/pep-0602/) ever discussed in relation to NEP 29 (https://numpy.org/neps/nep-0029-deprecation_policy.html)?
NEP 29 currently says:
"The current Python release cadence is 18 months so a 42 month window ensures that there will always be at least two minor versions of Python in the window."
However, as of PEP 602, the release cadence of CPython is now 12 months, not 18 months. This schedule now means supporting 3-4 versions of Python at once, rather than 2-3 versions (for those who don't want to do the math, 48/18 is 2.3 and 48/12 is 3.5).
PEP 602 started with Python 3.8, which was released in October, 2019. Python 3.8 is currently the oldest version supported in NEP 29, so we are about to enter the full cycle of the new cadence. In particular, when Python 3.11 is released in October of this year, the NEP 29 prescription will imply supporting 4 Python versions: 3.8, 3.9, 3.10, and 3.11, up until April, 2023 when 3.8 support would be dropped (and continuing like this: 3 versions between April and October; 4 versions between October and April).
Furthermore, the NEP seems to imply that the exact CPython release schedule is not known: "The window is extended 6 months beyond the anticipated two-release interval for Python to provide resilience against small fluctuations / delays in its release schedule." I'm not sure how true this was before, but PEP 602 pretty clearly prescribes a Python release in October of every year. Delays presumably might still be possible, but I don't see why it's necessary to build that possibility into the default period in the NEP given that it should be an unusual situation.
Personally I would prefer not ever supporting 4 simultaneous Python versions (I rather wish Python hadn't increased their cadence, but that ship has sailed). I don't know if it's possible to update the NEP to adjust for the new cadence. Even if it isn't, the text is clearly out of date and should be updated.
My current plan is to support 4 Python versions in the 1.24 NumPy release cycle, then drop Python 3.8 in NumPy 1.25. The result going forward is that we will oscillate between supporting 3 and 4 Python versions, each version being supported for about 42 months.
I wasn't on this list back in 2019, so my apologies if this has already been discussed.
To add to what Chuck said: this was indeed discussed during the NEP 29 discussion. The Python release cycle changing was a reason for having a time-based rather than a version-based approach. And Chuck's current plan is what was intended. Cheers, Ralf
Personally I would be in favor of updating NEP 29 to a support timespan in which at most 3 (minor) Python versions are supported. The development of Python is still at a high pace and NumPy is a high performance library which thrives when be able to adopt the latest Python features and having clean codebase without having "if sys.version_info[:2] < (3, n):" guards everywhere. Especially with the developments of the faster Faster CPython project and the continued improvements in type hinting support, I think shortening the support timespan a bit is reasonable, beneficial and proportional. More important, NEP 29 is adopted by many other projects, providing a signal to the ecosystem that's okay to focus on the latest Python versions. The overlap between users using a Python version more than 2.5 years old and wanting the latest features and performance is probably pretty small. Older NumPy and other project's releases will won't disappear and be usable when the Python requirements are lifted. In my opinion a 30 month support window would provide a nice balance. Each Python version is supported for over 2 minor releases, with updates on the maintenance branches extending that with another few months. You can comfortably stay a full minor Python version behind year-round and still use the latest NumPy. For even older Python versions the old NumPy releases will stay available, and it allow NumPy to move on to new Python features a full year earlier then with a 42 month support window. Which not only improves feature and performance adaptation, but also lowers maintenance, testing, backporting and CI effort. Ewout ter Hoeven (EwoutH)
On Tue, May 24, 2022 at 3:24 PM Ewout ter Hoeven < E.M.terHoeven@student.tudelft.nl> wrote:
Personally I would be in favor of updating NEP 29 to a support timespan in which at most 3 (minor) Python versions are supported. The development of Python is still at a high pace and NumPy is a high performance library which thrives when be able to adopt the latest Python features and having clean codebase without having "if sys.version_info[:2] < (3, n):" guards everywhere.
Especially with the developments of the faster Faster CPython project and the continued improvements in type hinting support, I think shortening the support timespan a bit is reasonable, beneficial and proportional.
More important, NEP 29 is adopted by many other projects, providing a signal to the ecosystem that's okay to focus on the latest Python versions. The overlap between users using a Python version more than 2.5 years old and wanting the latest features and performance is probably pretty small. Older NumPy and other project's releases will won't disappear and be usable when the Python requirements are lifted.
I don't think this is true. These graphs show that Python 3.7 is the version with the most downloaded wheels for, and our 1.21.x downloads are still higher than our 1.22.x downloads - very likely because we dropped 3.7 support in 1.22.x: https://pypistats.org/packages/numpy https://pepy.tech/project/numpy?versions=1.22.2&versions=1.22.3&versions=1.22.4&versions=1.21.6&versions=1.20.3 I have seen problems popping up already in a few places with latest numpy not supported what is still the most commonly used Python version (don't have links, sorry - but they were real packaging-related issues). So I don't think it makes sense to shorten the time window. I also don't think there's a need to drop one version that's urgent - it's some effort and CI time, but the balance is decent right now. Cheers, Ralf
In my opinion a 30 month support window would provide a nice balance. Each Python version is supported for over 2 minor releases, with updates on the maintenance branches extending that with another few months. You can comfortably stay a full minor Python version behind year-round and still use the latest NumPy. For even older Python versions the old NumPy releases will stay available, and it allow NumPy to move on to new Python features a full year earlier then with a 42 month support window. Which not only improves feature and performance adaptation, but also lowers maintenance, testing, backporting and CI effort.
Ewout ter Hoeven (EwoutH) _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ralf.gommers@gmail.com
To reiterate what Ralf said, the possibility of Python going to a faster cadence was one of the things on our mind when drafting NEP 29 (see https://numpy.org/neps/nep-0029-deprecation_policy.html#n-minor-versions-of-... for why did not go with a fixed number of versions) because the reality of the effort and timelines to upgrade deployments does not change with Python release frequency. The two things we are trying to balance here are the needs of the projects to use the cool new stuff in Python and the needs of our (institutional) users who are using our libraries for operations where updating the version of Python deployed is not trivial. There was a lot of discussion around how big the window should be with 42mo being about in the middle of what people advocated for. I am aware of institutions that are on an every-other Python upgrade pattern (py37 -> py39 -> (expected) py311) so always having at least 3 Pythons in the window is important. Based on what we have seen so far, I still think 42mo is a good choice, but do not think we have seen it in operations long enough to draw any conclusions (the downside of year-scale planning is you need to wait years to see if it worked out like you expected!) and hence do not think we should make any changes to the time window for another few years. That said, I am currently sympathetic to making the window longer and against making it shorter. It may be useful to have a discussion about what "support" means though given the combinatorial rise of (Python version) x (Python implementation) x (Operating system) x (architecture) that we are building wheels for. I think it would still be within the spirit of NEP 29 to run CI on CPython for all of the supported versions (maybe across the 3 big OS's) and then only publish wheels for the latest (2 version of Python) x (full matrix of the rest). Stealing some language/concepts from Microsoft (if I recall it correctly), we should sort out which entries in that support matrix are Level 1 (CI + wheels), Level 2 (CI), Level 3 (we test something that looks like this), and Level 4 (your on your own, but we will take your patches to un-break things as you send them). We would obviously need more thought out definitions of the levels as well. Tom On Wed, May 25, 2022 at 5:23 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Tue, May 24, 2022 at 3:24 PM Ewout ter Hoeven < E.M.terHoeven@student.tudelft.nl> wrote:
Personally I would be in favor of updating NEP 29 to a support timespan in which at most 3 (minor) Python versions are supported. The development of Python is still at a high pace and NumPy is a high performance library which thrives when be able to adopt the latest Python features and having clean codebase without having "if sys.version_info[:2] < (3, n):" guards everywhere.
Especially with the developments of the faster Faster CPython project and the continued improvements in type hinting support, I think shortening the support timespan a bit is reasonable, beneficial and proportional.
More important, NEP 29 is adopted by many other projects, providing a signal to the ecosystem that's okay to focus on the latest Python versions. The overlap between users using a Python version more than 2.5 years old and wanting the latest features and performance is probably pretty small. Older NumPy and other project's releases will won't disappear and be usable when the Python requirements are lifted.
I don't think this is true. These graphs show that Python 3.7 is the version with the most downloaded wheels for, and our 1.21.x downloads are still higher than our 1.22.x downloads - very likely because we dropped 3.7 support in 1.22.x: https://pypistats.org/packages/numpy
I have seen problems popping up already in a few places with latest numpy not supported what is still the most commonly used Python version (don't have links, sorry - but they were real packaging-related issues). So I don't think it makes sense to shorten the time window. I also don't think there's a need to drop one version that's urgent - it's some effort and CI time, but the balance is decent right now.
Cheers, Ralf
In my opinion a 30 month support window would provide a nice balance. Each Python version is supported for over 2 minor releases, with updates on the maintenance branches extending that with another few months. You can comfortably stay a full minor Python version behind year-round and still use the latest NumPy. For even older Python versions the old NumPy releases will stay available, and it allow NumPy to move on to new Python features a full year earlier then with a 42 month support window. Which not only improves feature and performance adaptation, but also lowers maintenance, testing, backporting and CI effort.
Ewout ter Hoeven (EwoutH) _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ralf.gommers@gmail.com
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: tcaswell@gmail.com
-- Thomas Caswell tcaswell@gmail.com
On Wed, May 25, 2022 at 4:56 PM Thomas Caswell <tcaswell@gmail.com> wrote:
To reiterate what Ralf said, the possibility of Python going to a faster cadence was one of the things on our mind when drafting NEP 29 (see https://numpy.org/neps/nep-0029-deprecation_policy.html#n-minor-versions-of-... for why did not go with a fixed number of versions) because the reality of the effort and timelines to upgrade deployments does not change with Python release frequency. The two things we are trying to balance here are the needs of the projects to use the cool new stuff in Python and the needs of our (institutional) users who are using our libraries for operations where updating the version of Python deployed is not trivial.
There was a lot of discussion around how big the window should be with 42mo being about in the middle of what people advocated for. I am aware of institutions that are on an every-other Python upgrade pattern (py37 -> py39 -> (expected) py311) so always having at least 3 Pythons in the window is important. Based on what we have seen so far, I still think 42mo is a good choice, but do not think we have seen it in operations long enough to draw any conclusions (the downside of year-scale planning is you need to wait years to see if it worked out like you expected!) and hence do not think we should make any changes to the time window for another few years. That said, I am currently sympathetic to making the window longer and against making it shorter.
It may be useful to have a discussion about what "support" means though given the combinatorial rise of (Python version) x (Python implementation) x (Operating system) x (architecture) that we are building wheels for. I think it would still be within the spirit of NEP 29 to run CI on CPython for all of the supported versions (maybe across the 3 big OS's) and then only publish wheels for the latest (2 version of Python) x (full matrix of the rest).
We cannot do that (yet, at least). Failing to publish wheels for a supported Python version on a major OS is far worse than dropping support completely. This will remain true until the time that Pip starts defaulting to wheels-only and never picks the sdist if there's an older release for that platform + Python combo. Stealing some language/concepts from Microsoft (if I recall it correctly),
we should sort out which entries in that support matrix are Level 1 (CI + wheels), Level 2 (CI), Level 3 (we test something that looks like this), and Level 4 (your on your own, but we will take your patches to un-break things as you send them). We would obviously need more thought out definitions of the levels as well.
Agreed, this would be useful to map out. Starting with what we currently do, and not mix in any changes in our CI/wheel coverage, in order to decouple the support model from other, more contentious proposals. Cheers, Ralf
Tom
On Wed, May 25, 2022 at 5:23 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Tue, May 24, 2022 at 3:24 PM Ewout ter Hoeven < E.M.terHoeven@student.tudelft.nl> wrote:
Personally I would be in favor of updating NEP 29 to a support timespan in which at most 3 (minor) Python versions are supported. The development of Python is still at a high pace and NumPy is a high performance library which thrives when be able to adopt the latest Python features and having clean codebase without having "if sys.version_info[:2] < (3, n):" guards everywhere.
Especially with the developments of the faster Faster CPython project and the continued improvements in type hinting support, I think shortening the support timespan a bit is reasonable, beneficial and proportional.
More important, NEP 29 is adopted by many other projects, providing a signal to the ecosystem that's okay to focus on the latest Python versions. The overlap between users using a Python version more than 2.5 years old and wanting the latest features and performance is probably pretty small. Older NumPy and other project's releases will won't disappear and be usable when the Python requirements are lifted.
I don't think this is true. These graphs show that Python 3.7 is the version with the most downloaded wheels for, and our 1.21.x downloads are still higher than our 1.22.x downloads - very likely because we dropped 3.7 support in 1.22.x: https://pypistats.org/packages/numpy
I have seen problems popping up already in a few places with latest numpy not supported what is still the most commonly used Python version (don't have links, sorry - but they were real packaging-related issues). So I don't think it makes sense to shorten the time window. I also don't think there's a need to drop one version that's urgent - it's some effort and CI time, but the balance is decent right now.
Cheers, Ralf
In my opinion a 30 month support window would provide a nice balance. Each Python version is supported for over 2 minor releases, with updates on the maintenance branches extending that with another few months. You can comfortably stay a full minor Python version behind year-round and still use the latest NumPy. For even older Python versions the old NumPy releases will stay available, and it allow NumPy to move on to new Python features a full year earlier then with a 42 month support window. Which not only improves feature and performance adaptation, but also lowers maintenance, testing, backporting and CI effort.
Ewout ter Hoeven (EwoutH)
On Wed, 25 May 2022, 4:54 pm Thomas Caswell, <tcaswell@gmail.com> wrote:
Stealing some language/concepts from Microsoft (if I recall it correctly), we should sort out which entries in that support matrix are Level 1 (CI + wheels), Level 2 (CI), Level 3 (we test something that looks like this), and Level 4 (your on your own, but we will take your patches to un-break things as you send them).
As someone who knows very little about packaging, what is the rationale behind the difference between levels 1 and 2? How much load does it add, given that the CI is already set up and building them? /David
I have seen problems popping up already in a few places with latest numpy not supported what is still the most commonly used Python version (don't have links, sorry - but they were real packaging-related issues). So I don't think it makes sense to shorten the time window. I also don't think there's a need to drop one version that's urgent - it's some effort and CI time, but the balance is decent right now.
It's hard to say the balance is decent right now if the faster cadence isn't even in full effect yet (as I noted in my original email). Generally I would say that dropping support only means that users of older versions would simply need to use an older version of the library. However this:
We cannot do that (yet, at least). Failing to publish wheels for a supported Python version on a major OS is far worse than dropping support completely. This will remain true until the time that Pip starts defaulting to wheels-only and never picks the sdist if there's an older release for that platform + Python combo.
sounds like if even the latest version doesn't support a given CPython version (and has CPython-versioned wheels), then pip installing it will fail. Is that correct?
There was a lot of discussion around how big the window should be with 42mo being about in the middle of what people advocated for. I am aware of institutions that are on an every-other Python upgrade pattern (py37 -> py39 -> (expected) py311) so always having at least 3 Pythons in the window is important. Based on what we have seen so far, I still think 42mo is a good choice, but do not think we have seen it in operations long enough to draw any conclusions (the downside of year-scale planning is you need to wait years to see if it worked out like you expected!) and hence do not think we should make any changes to the time window for another few years. That said, I am currently sympathetic to making the window longer and against making it shorter.
The reason I brought this up is because we were looking at whether it makes sense to use NEP 29 for SymPy. Obviously we aren't bound to using it, but given this apparent discrepancy in the text (which still exists), I thought I would mention it here. But I will say that from my point of view, supporting more Python versions is a burden. Having more builds on CI means longer wait times to merge PRs. And having to wait longer for new Python features is also annoying. Aaron Meurer
I created https://github.com/numpy/numpy/pull/21601 to update NEP29 to address PEP602. I'm not sure what the procedure for updating the NEP is. What I wrote may be too editorial, we could amend it to "PEP602 changed cadence, we are not reacting." with no explanation as well. Tom On Wed, May 25, 2022 at 10:41 PM Aaron Meurer <asmeurer@gmail.com> wrote:
I have seen problems popping up already in a few places with latest numpy not supported what is still the most commonly used Python version (don't have links, sorry - but they were real packaging-related issues). So I don't think it makes sense to shorten the time window. I also don't think there's a need to drop one version that's urgent - it's some effort and CI time, but the balance is decent right now.
It's hard to say the balance is decent right now if the faster cadence isn't even in full effect yet (as I noted in my original email).
Generally I would say that dropping support only means that users of older versions would simply need to use an older version of the library.
However this:
We cannot do that (yet, at least). Failing to publish wheels for a supported Python version on a major OS is far worse than dropping support completely. This will remain true until the time that Pip starts defaulting to wheels-only and never picks the sdist if there's an older release for that platform + Python combo.
sounds like if even the latest version doesn't support a given CPython version (and has CPython-versioned wheels), then pip installing it will fail. Is that correct?
There was a lot of discussion around how big the window should be with 42mo being about in the middle of what people advocated for. I am aware of institutions that are on an every-other Python upgrade pattern (py37 -> py39 -> (expected) py311) so always having at least 3 Pythons in the window is important. Based on what we have seen so far, I still think 42mo is a good choice, but do not think we have seen it in operations long enough to draw any conclusions (the downside of year-scale planning is you need to wait years to see if it worked out like you expected!) and hence do not think we should make any changes to the time window for another few years. That said, I am currently sympathetic to making the window longer and against making it shorter.
The reason I brought this up is because we were looking at whether it makes sense to use NEP 29 for SymPy. Obviously we aren't bound to using it, but given this apparent discrepancy in the text (which still exists), I thought I would mention it here. But I will say that from my point of view, supporting more Python versions is a burden. Having more builds on CI means longer wait times to merge PRs. And having to wait longer for new Python features is also annoying.
Aaron Meurer _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: tcaswell@gmail.com
-- Thomas Caswell tcaswell@gmail.com
On 26/5/22 05:40, Aaron Meurer wrote:
We cannot do that (yet, at least). Failing to publish wheels for a supported Python version on a major OS is far worse than dropping support completely. This will remain true until the time that Pip starts defaulting to wheels-only and never picks the sdist if there's an older release for that platform + Python combo. sounds like if even the latest version doesn't support a given CPython version (and has CPython-versioned wheels), then pip installing it will fail. Is that correct?
No. The problem is that if we do not provide binary wheels, pip installing Numpy will not fail. It will try to build from source. Even if this painfully and slowly succeeds, it means the user probably did not get the BLAS support they thought they were going to get. The situation for SciPy is similar but worse: it takes forever to build, and will probably likewise fail to build properly. Matti
On Thu, May 26, 2022 at 3:45 AM Matti Picus <matti.picus@gmail.com> wrote:
On 26/5/22 05:40, Aaron Meurer wrote:
We cannot do that (yet, at least). Failing to publish wheels for a supported Python version on a major OS is far worse than dropping support completely. This will remain true until the time that Pip starts defaulting to wheels-only and never picks the sdist if there's an older release for that platform + Python combo. sounds like if even the latest version doesn't support a given CPython version (and has CPython-versioned wheels), then pip installing it will fail. Is that correct?
No. The problem is that if we do not provide binary wheels, pip installing Numpy will not fail. It will try to build from source. Even if this painfully and slowly succeeds, it means the user probably did not get the BLAS support they thought they were going to get. The situation for SciPy is similar but worse: it takes forever to build, and will probably likewise fail to build properly.
If Python version 3.X isn't supported, and that non-support means it actually doesn't work, then it will fail. That is unless pip is smart enough to install the most recent version of NumPy that does support that version of Python (which would have wheels). Ralf made it sound like this is what would happen, but doesn't pip have python_requires to avoid this? Aaron Meurer
Matti
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: asmeurer@gmail.com
On Fri, May 27, 2022 at 6:00 AM Aaron Meurer <asmeurer@gmail.com> wrote:
On Thu, May 26, 2022 at 3:45 AM Matti Picus <matti.picus@gmail.com> wrote:
On 26/5/22 05:40, Aaron Meurer wrote:
We cannot do that (yet, at least). Failing to publish wheels for a supported Python version on a major OS is far worse than dropping support completely. This will remain true until the time that Pip starts defaulting to wheels-only and never picks the sdist if there's an older release for that platform + Python combo. sounds like if even the latest version doesn't support a given CPython version (and has CPython-versioned wheels), then pip installing it will fail. Is that correct?
No. The problem is that if we do not provide binary wheels, pip installing Numpy will not fail. It will try to build from source. Even if this painfully and slowly succeeds, it means the user probably did not get the BLAS support they thought they were going to get. The situation for SciPy is similar but worse: it takes forever to build, and will probably likewise fail to build properly.
If Python version 3.X isn't supported, and that non-support means it actually doesn't work, then it will fail. That is unless pip is smart enough to install the most recent version of NumPy that does support that version of Python (which would have wheels). Ralf made it sound like this is what would happen, but doesn't pip have python_requires to avoid this?
There's several issues, not sure which one you are asking about, so let me address all the options: 1. The release/sdist supports a Python release, but we have no wheels for it. Then, Pip's default behaviour is to build from source (sdist). This is highly likely to fail, because we have dependencies like compilers and a BLAS library that cannot even be expressed in metadata. We do not want `pip install numpy` to trigger building from source for the most commonly used platforms (Windows, macOS, Linux on x86-64/aarch64 at least), because that would lead to a flood of "build is broken" issues. 2. If there's a new Python release for which we don't yet have wheels or an sdist to support that, Pip will grab the last sdist that doesn't say python_requires='>=3.xx,<3.yy' (i.e., that doesn't have an upper bound). That will definitely fail. We can't really do anything about that other than raising a more informative error message at the start of the build. 3. Tools like Pip and Poetry do not handle python_requires correctly unfortunately. Long story, don't want to get into it, but let's just say it doesn't work as you'd expect from install_requires. Long story short: we do everything we can to get our metadata correct and have wheels for the most commonly used OSes and platforms, and *all* supported Python versions in a release. Any proposal that drops wheel coverage for Python version support is not realistic. Cheers, Ralf
On Thu, May 26, 2022 at 4:41 AM Aaron Meurer <asmeurer@gmail.com> wrote:
I have seen problems popping up already in a few places with latest numpy not supported what is still the most commonly used Python version (don't have links, sorry - but they were real packaging-related issues). So I don't think it makes sense to shorten the time window. I also don't think there's a need to drop one version that's urgent - it's some effort and CI time, but the balance is decent right now.
It's hard to say the balance is decent right now if the faster cadence isn't even in full effect yet (as I noted in my original email).
Generally I would say that dropping support only means that users of older versions would simply need to use an older version of the library.
That is not the full story unfortunately. The Python packaging tooling (Pip, Poetry et al) is imperfect, so if other packages depend on NumPy and then at install time a tool figures out that latest NumPy cannot be used, or somewhere there's an upper bound in the version constraints explicitly, some things tend to go wrong.
There was a lot of discussion around how big the window should be with 42mo being about in the middle of what people advocated for. I am aware of institutions that are on an every-other Python upgrade pattern (py37 -> py39 -> (expected) py311) so always having at least 3 Pythons in the window is important. Based on what we have seen so far, I still think 42mo is a good choice, but do not think we have seen it in operations long enough to draw any conclusions (the downside of year-scale planning is you need to wait years to see if it worked out like you expected!) and hence do not think we should make any changes to the time window for another few years. That said, I am currently sympathetic to making the window longer and against making it shorter.
The reason I brought this up is because we were looking at whether it makes sense to use NEP 29 for SymPy. Obviously we aren't bound to using it, but given this apparent discrepancy in the text (which still exists),
What discrepancy? Cheers, Ralf
https://github.com/numpy/numpy/pull/21601 has been merged, but we should make sure everyone is on board with the updated wording. The intent was to resolve the discrepancy I think Aaron is referring to (the text spoke of the 18mo release cycle in present tense) and to justify sticking with 42months despite the upstream change. I think it is good for the overall community if we have a (mostly) unified position on this, but if projects are going to disregard NEP 29 I am personally happy for them to run faster :) Tom On Thu, May 26, 2022 at 7:08 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, May 26, 2022 at 4:41 AM Aaron Meurer <asmeurer@gmail.com> wrote:
I have seen problems popping up already in a few places with latest numpy not supported what is still the most commonly used Python version (don't have links, sorry - but they were real packaging-related issues). So I don't think it makes sense to shorten the time window. I also don't think there's a need to drop one version that's urgent - it's some effort and CI time, but the balance is decent right now.
It's hard to say the balance is decent right now if the faster cadence isn't even in full effect yet (as I noted in my original email).
Generally I would say that dropping support only means that users of older versions would simply need to use an older version of the library.
That is not the full story unfortunately. The Python packaging tooling (Pip, Poetry et al) is imperfect, so if other packages depend on NumPy and then at install time a tool figures out that latest NumPy cannot be used, or somewhere there's an upper bound in the version constraints explicitly, some things tend to go wrong.
There was a lot of discussion around how big the window should be with 42mo being about in the middle of what people advocated for. I am aware of institutions that are on an every-other Python upgrade pattern (py37 -> py39 -> (expected) py311) so always having at least 3 Pythons in the window is important. Based on what we have seen so far, I still think 42mo is a good choice, but do not think we have seen it in operations long enough to draw any conclusions (the downside of year-scale planning is you need to wait years to see if it worked out like you expected!) and hence do not think we should make any changes to the time window for another few years. That said, I am currently sympathetic to making the window longer and against making it shorter.
The reason I brought this up is because we were looking at whether it makes sense to use NEP 29 for SymPy. Obviously we aren't bound to using it, but given this apparent discrepancy in the text (which still exists),
What discrepancy?
Cheers, Ralf
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: tcaswell@gmail.com
-- Thomas Caswell tcaswell@gmail.com
participants (7)
-
Aaron Meurer -
Charles R Harris -
David Menéndez Hurtado -
Ewout ter Hoeven -
Matti Picus -
Ralf Gommers -
Thomas Caswell