Bumping minimum Sphinx version to 3.2 for cpython 3.10?
During the development of cpython 3.10, Sphinx was bumped to 3.2.1. Problem is Sphinx 3 have some incompatibilities with Sphinx 2, some that we could work around, some are bit harder, so we may need to bump `needs_sphinx = '3.2'` (currently it is 1.8). I found two incompatibilities: - We're using :no-trim-doctest-flags:, introduced in Sphinx 3.2.0, if we build with Sphinx < 3.2.0 the blocks using it are dropped by Sphinx (we use it in library/doctest). - double backslashes in domain directives are no longer replaced by single backslashes. But a configuration value (strip_signature_backslash) can be used to have the same behavior on Sphinx 2 and 3. So yes, it's still possible to build the docs with Sphinx 1.8 using `make html SPHINXERRORHANDLING=`, it "works" but: - Doctests code blocks using no-trim-doctest-flags dissapear. - Some functions declarations are lacking a backslash, like print(*objects, sep=' ', end='n', ... Which is bad. For the first one we could still workaround with an ugly `sed -i /no-trim-doctest-flags/d`, or simply stop using it (and reopen bpo-36675). For the 2nd one we could use `strip_signature_backslash` in cpython 3.10 to have the same behavior. So the question is: Should we bump minimum version of Sphinx from 1.8 to 3.2.1 along with Python 3.10? For the moment this is where we go, but if you're having to maintain a release of Python along with Sphinx < 3, please make speak. Conversation has already started here [1] and here [2]. [1]: https://bugs.python.org/issue42843 [2]: https://github.com/python/cpython/pull/24142 -- [Julien Palard](https://mdk.fr)
On Tue, Jan 12, 2021 at 08:38:17PM +0000, Julien Palard via Python-Dev wrote:
- Some functions declarations are lacking a backslash, like print(*objects, sep=' ', end='n', ...
Which is bad.
Wouldn't this a bug with Sphinx? Why would that be special cased with a flag (strip_signature_backslash)?
For the moment this is where we go, but if
+1. Keeping the documentation dependency updated I noticed that you suggested not to backport this PR https://github.com/python/cpython/pull/24142 * Would that mean we have to careful not to use/merge sphinx features like `no-trim-doctest-flags` to older docs? * What would it take to keep all active versions of Cpython docs build at the same min version (possibly 3.2)? As Cpython developer, I see nothing bad, but platform maintainers seem to have some concerns, and it will be good to see points against this. Thank you, Senthil
Le 2021-01-13 à 00:09, Senthil Kumaran a écrit :
Wouldn't this a bug with Sphinx?
No, this is a documented breaking change between Sphinx 2 and Sphinx 3.
Why would that be special cased with a flag (strip_signature_backslash)?
The strip_signature_backslash has been introduced to allow Sphinx 3 to behave like Sphinx 2 in respect of backslash in signatures. It would allow us to write Sphinx 2 and Sphinx 3 compatible doc.
I noticed that you suggested not to backport this PR https://github.com/python/cpython/pull/24142
I'm against changing the major version of the Sphinx we use during the life of a release. It looks already hard for distrib maintainers to handle a Sphinx upgrade for the next release, let's not upgrade it on live releases. In other words: it may or may not be OK to bump Sphinx for a minor Python release, but it's clearly not for a patch release (only my point of view).
* Would that mean we have to careful not to use/merge sphinx features like `no-trim-doctest-flags` to older docs?
Exactly: backporting a Doc commit could not work, but it would be spotted by the tests and not mergeable, as we use `-W` flag of Sphinx build (treat warnings as errors).
* What would it take to keep all active versions of Cpython docs build at the same min version (possibly 3.2)?
As said previously, I don't think it's OK to bump Python 3.7 to use Sphinx 3 now.
As Cpython developer, I see nothing bad, but platform maintainers seem to have some concerns, and it will be good to see points against this.
That's exactly why I opened this thread, if it's OK for all platform maintainers, we'll do this, but if not, we can revert a few commits and postpone the Sphinx upgrade, no worries. -- [Julien Palard](https://mdk.fr)
Since documentation changes are backported to 3.8 and 3.9 stable branches, if we increase the minimum required Sphinx version in master, I would prefer to also increase it in 3.8 and 3.9 branches. I would prefer to not have to check manually if a doc backport PR is still compatible with Sphinx 2 or not. If we skip some doc backports, conflicts will become more likely with following doc backports. -- I looked at Sphinx and Python versions of Debian, Ubuntu and Fedora: https://bugs.python.org/issue42843#msg384963 In my list, there is only Debian Buster (stable) which doesn't have Sphinx 3 yet. It uses Python 3.7 and so would not be affected by Python 3.8 changes. -- The alternative is to keep Sphinx 2 support, use strip_signature_backslash and don't use :no-trim-doctest-flags: ? Victor On Tue, Jan 12, 2021 at 9:44 PM Julien Palard via Python-Dev <python-dev@python.org> wrote:
During the development of cpython 3.10, Sphinx was bumped to 3.2.1.
Problem is Sphinx 3 have some incompatibilities with Sphinx 2, some that we could work around, some are bit harder, so we may need to bump `needs_sphinx = '3.2'` (currently it is 1.8).
I found two incompatibilities:
- We're using :no-trim-doctest-flags:, introduced in Sphinx 3.2.0, if we build with Sphinx < 3.2.0 the blocks using it are dropped by Sphinx (we use it in library/doctest).
- double backslashes in domain directives are no longer replaced by single backslashes. But a configuration value (strip_signature_backslash) can be used to have the same behavior on Sphinx 2 and 3.
So yes, it's still possible to build the docs with Sphinx 1.8 using `make html SPHINXERRORHANDLING=`, it "works" but:
- Doctests code blocks using no-trim-doctest-flags dissapear. - Some functions declarations are lacking a backslash, like print(*objects, sep=' ', end='n', ...
Which is bad.
For the first one we could still workaround with an ugly `sed -i /no-trim-doctest-flags/d`, or simply stop using it (and reopen bpo-36675). For the 2nd one we could use `strip_signature_backslash` in cpython 3.10 to have the same behavior.
So the question is: Should we bump minimum version of Sphinx from 1.8 to 3.2.1 along with Python 3.10? For the moment this is where we go, but if you're having to maintain a release of Python along with Sphinx < 3, please make speak.
Conversation has already started here [1] and here [2].
[1]: https://bugs.python.org/issue42843 [2]: https://github.com/python/cpython/pull/24142
-- [Julien Palard](https://mdk.fr) _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/E64YE3DQ... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
On Wed, Jan 13, 2021 at 12:28:42AM +0100, Victor Stinner wrote:
The alternative is to keep Sphinx 2 support, use strip_signature_backslash and don't use :no-trim-doctest-flags: ?
+1. :no-trim-doctest-flags: was introduced to python docs only recently, so not using that is a reasonable suggestion. I support keeping same Sphinx version across all the supported python versions. -- Senthil
On 13.01.2021 2:47, Senthil Kumaran wrote:
On Wed, Jan 13, 2021 at 12:28:42AM +0100, Victor Stinner wrote:
The alternative is to keep Sphinx 2 support, use strip_signature_backslash and don't use :no-trim-doctest-flags: ? +1. :no-trim-doctest-flags: was introduced to python docs only recently, so not using that is a reasonable suggestion.
I support keeping same Sphinx version across all the supported python versions.
This is not a sustainable route since this way, there's no way to change the version at all.
-- Senthil _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HS2OYF7Q... Code of Conduct: http://python.org/psf/codeofconduct/
-- Regards, Ivan
On Wed, Jan 13, 2021 at 02:53:30AM +0300, Ivan Pozdeev via Python-Dev wrote:
I support keeping same Sphinx version across all the supported python versions.
This is not a sustainable route since this way, there's no way to change the version at all.
By supported, I mean the active versions that we backport patches to. Same as what Victor said. -- Senthil
On 13.01.2021 3:12, Senthil Kumaran wrote:
On Wed, Jan 13, 2021 at 02:53:30AM +0300, Ivan Pozdeev via Python-Dev wrote:
I support keeping same Sphinx version across all the supported python versions. This is not a sustainable route since this way, there's no way to change the version at all.
By supported, I mean the active versions that we backport patches to. Same as what Victor said.
That's what I meant, too. E.g. when 3.11 comes out and 3.8 is EOL, it will still have to stick with the old sphinx because 3.9 and 3.10 do -- and so on ad infinum.
-- Senthil
-- Regards, Ivan
On 13. 01. 21 0:28, Victor Stinner wrote:
I looked at Sphinx and Python versions of Debian, Ubuntu and Fedora: https://bugs.python.org/issue42843#msg384963
In my list, there is only Debian Buster (stable) which doesn't have Sphinx 3 yet. It uses Python 3.7 and so would not be affected by Python 3.8 changes.
Hello Victor. In Fedora 32, we have Sphinx 2.2.2 and Python 3.8. Fedora 32 goes EOL in May 2021. Until then, it would eb great if we could keep compatibility with Sphinx 2 in Python 3.8 (but if we don't, no big deal, we just won't update the docs there). -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok
On 1/13/21 12:28 AM, Victor Stinner wrote:
Since documentation changes are backported to 3.8 and 3.9 stable branches, if we increase the minimum required Sphinx version in master, I would prefer to also increase it in 3.8 and 3.9 branches.
I would prefer to not have to check manually if a doc backport PR is still compatible with Sphinx 2 or not. If we skip some doc backports, conflicts will become more likely with following doc backports.
--
I looked at Sphinx and Python versions of Debian, Ubuntu and Fedora: https://bugs.python.org/issue42843#msg384963
In my list, there is only Debian Buster (stable) which doesn't have Sphinx 3 yet. It uses Python 3.7 and so would not be affected by Python 3.8 changes.
That's not true. Ubuntu 20.04 LTS still sees updates to subminor Python 3.8 versions, having sphinx 1.8.5. Matthias
Hi Matthias, Le 2021-01-13 à 12:27, Matthias Klose a écrit :
That's not true. Ubuntu 20.04 LTS still sees updates to subminor Python 3.8 versions, having sphinx 1.8.5.
I do agree to *not* bump our Sphinx dependencies for already published Pythons. Would it be OK to bump Sphinx to 3.2 for Python 3.10? Or would it be better to keep Sphinx 2 compatibility for a few years (how many?) before requiring Sphinx 3? Bests, -- [Julien Palard](https://mdk.fr)
Le 2021-01-13 à 00:28, Victor Stinner a écrit :
Since documentation changes are backported to 3.8 and 3.9 stable branches, if we increase the minimum required Sphinx version in master, I would prefer to also increase it in 3.8 and 3.9 branches.
Bumping a dependency on the next release is already hard, I don't think we can bump dependencies on current releases like Python 3.8.
I would prefer to not have to check manually if a doc backport PR is still compatible with Sphinx 2 or not.
As we're using `-W` flag of sphinx-build, the tests would spot it and mark the backport PR as not mergeable. The doc would diverge from a branch to another, but it already does, so I do not worry a lot about conflicts.
The alternative is to keep Sphinx 2 support, use strip_signature_backslash and don't use :no-trim-doctest-flags: ?
Exact, totally doable for a few years to ease the migration, we could even keep Sphinx 1.8 support for a few years as it still works with those two "fixes" applied (it would imply reopening https://bugs.python.org/issue36675 but I'm not against it). -- [Julien Palard](https://mdk.fr)
13.01.21 01:28, Victor Stinner пише:
I looked at Sphinx and Python versions of Debian, Ubuntu and Fedora: https://bugs.python.org/issue42843#msg384963
In my list, there is only Debian Buster (stable) which doesn't have Sphinx 3 yet. It uses Python 3.7 and so would not be affected by Python 3.8 changes.
Also Ubuntu. Ubuntu 20.04 has only Sphinx 1.8.5, but Python 3.8 and 3.9.
12.01.21 22:38, Julien Palard via Python-Dev пише:
During the development of cpython 3.10, Sphinx was bumped to 3.2.1.
Problem is Sphinx 3 have some incompatibilities with Sphinx 2, some that we could work around, some are bit harder, so we may need to bump `needs_sphinx = '3.2'` (currently it is 1.8).
Sphinx version in the current Ubuntu LTS (20.04) is 1.8.5. Would not it cause problems with builting documentation on Ubuntu?
Hi Julien, I think that there are two items to consider: - `needs_sphinx` in `conf.py` - setting for Sphinx in `cpython/Doc/requirements.txt` I believe that the `needs_sphinx` field identifies the minimal version of Sphinx that can be used to build the docs. The actual version that is used to build the docs is in the `requirements.txt` file. On Wed, Jan 13, 2021 at 7:29 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
12.01.21 22:38, Julien Palard via Python-Dev пише:
During the development of cpython 3.10, Sphinx was bumped to 3.2.1.
Problem is Sphinx 3 have some incompatibilities with Sphinx 2, some that we could work around, some are bit harder, so we may need to bump `needs_sphinx = '3.2'` (currently it is 1.8).
Sphinx version in the current Ubuntu LTS (20.04) is 1.8.5. Would not it cause problems with builting documentation on Ubuntu? _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EJWTFPHZ... Code of Conduct: http://python.org/psf/codeofconduct/
My take on this is to keep it simple for CPython: Require the newer version of Sphinx on all branches we backport docs changes to. We, the CPython project, determine the exact version(s) of Sphinx a documentation build for a given version of CPython requires. If Sphinx has unfortunately made a breaking change (as appears to be the case?), we should update our docs for all versions we backport doc changes to to use the most modern version (along with the docs themselves). We should explicitly not care at all about people using external sphinx installs that are not fetched at the correct version via our Doc tree's Doc/requirements.txt. That some distros want to use their own incompatible-version packaged version of sphinx to build our docs is on them to handle on their own. Not on the CPython project. That was never a guaranteed thing that would work. Until now, they merely got lucky. This is a docs build-time dependency with no impact on anything at runtime so we should be free to change versions as we see fit. -gps On Wed, Jan 13, 2021 at 8:25 AM Carol Willing <willingc@gmail.com> wrote:
Hi Julien,
I think that there are two items to consider: - `needs_sphinx` in `conf.py` - setting for Sphinx in `cpython/Doc/requirements.txt`
I believe that the `needs_sphinx` field identifies the minimal version of Sphinx that can be used to build the docs. The actual version that is used to build the docs is in the `requirements.txt` file.
On Wed, Jan 13, 2021 at 7:29 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
During the development of cpython 3.10, Sphinx was bumped to 3.2.1.
Problem is Sphinx 3 have some incompatibilities with Sphinx 2, some
12.01.21 22:38, Julien Palard via Python-Dev пише: that
we could work around, some are bit harder, so we may need to bump `needs_sphinx = '3.2'` (currently it is 1.8).
Sphinx version in the current Ubuntu LTS (20.04) is 1.8.5. Would not it cause problems with builting documentation on Ubuntu? _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EJWTFPHZ... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S53K4P27... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Jan 13, 2021 at 7:25 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
12.01.21 22:38, Julien Palard via Python-Dev пише:
During the development of cpython 3.10, Sphinx was bumped to 3.2.1.
Problem is Sphinx 3 have some incompatibilities with Sphinx 2, some that we could work around, some are bit harder, so we may need to bump `needs_sphinx = '3.2'` (currently it is 1.8).
Sphinx version in the current Ubuntu LTS (20.04) is 1.8.5. Would not it cause problems with builting documentation on Ubuntu?
Why can't contributors install from PyPI? The venv created by the Docs/Makefile does a pip install, so I don't see why what version of Sphinx is packaged via APT is a critical blocker in upgrading Sphinx.
On 1/13/21 8:24 PM, Brett Cannon wrote:
On Wed, Jan 13, 2021 at 7:25 AM Serhiy Storchaka <storchaka@gmail.com <mailto:storchaka@gmail.com>> wrote:
12.01.21 22:38, Julien Palard via Python-Dev пише: > During the development of cpython 3.10, Sphinx was bumped to 3.2.1. > > Problem is Sphinx 3 have some incompatibilities with Sphinx 2, some that > we could work around, some are bit harder, so we may need to bump > `needs_sphinx = '3.2'` (currently it is 1.8).
Sphinx version in the current Ubuntu LTS (20.04) is 1.8.5. Would not it cause problems with builting documentation on Ubuntu?
Why can't contributors install from PyPI? The venv created by the Docs/Makefile does a pip install, so I don't see why what version of Sphinx is packaged via APT is a critical blocker in upgrading Sphinx.
I trust the CPython core devs, and I trust my distro's processes and packagers, but not necessarily the PyPI maintainers of Sphinx's requirements.
Thanks everyone for the feedback. I think the best way to handle this is to make the three next releases (3.10, 3.11, 3.12) Sphinx 2 and Sphinx 3 compatible, this would gather requiered benefits: - Ease of backporting: from any dev version we can backport documentation changes to the previous two releases. - Ease of packaging: We're not requiering Sphinx 3 today but in like 3 years, a point where this could be discussed again to see if everyone is ready. If this plan is OK for everyone, I'll try a PR soon™ to make this compatibility with Sphinx2/3 land in 3.10. -- [Julien Palard](https://mdk.fr)
On Fri, Jan 15, 2021 at 08:24:54AM +0000, Julien Palard via Python-Dev wrote:
I think the best way to handle this is to make the three next releases (3.10, 3.11, 3.12) Sphinx 2 and Sphinx 3 compatible, this would gather requiered benefits:
If this plan is OK for everyone, I'll try a PR soon™ to make this compatibility with Sphinx2/3 land in 3.10.
This sounds great. It makes one less thing to worry about from a CPython developer perspective. -- Senthil
participants (11)
-
Brett Cannon -
Carol Willing -
Gregory P. Smith -
Ivan Pozdeev -
Julien Palard -
Matthias Klose -
Miro Hrončok -
Petr Viktorin -
Senthil Kumaran -
Serhiy Storchaka -
Victor Stinner