Re: [Twisted-Python] Release blocker: Use latest pydoctor release ?
Thanks for the clarification. You have provided a lot of useful details. There are a lot of interconnected pieces, with multiple people working on different things. For now, I am going to proceed to finish the release as-is. The documentation may not be automatically deployed as part of the release, but I think that can be done as a follow-on task. -- Craig On Sat, Feb 27, 2021 at 4:53 PM Adi Roiban <adiroiban@gmail.com> wrote:
Hi,
Sorry for the delay.
We did the automation and Read The Docs hosting.
In parallel there was also the work required to host the pydoctor API docs on Read The Docs.
----------
Note that https://docs.twistedmatrix.com/en/twisted-20.3.0/api 404
That should host the files from https://twistedmatrix.com/documents/current/api/
---------
For `trunk` we have https://docs.twistedmatrix.com/en/latest/api/
That was the hard work done by Maarten to improve pydoctor and Twisted API docs.
-------------
There was a plan to do a new pydoctor release over the end of the year break... but that never happened.
The current PR that is "breaking" API links from narrative docs to API docs on Read The Docs version of Twisted docs is https://github.com/twisted/twisted/pull/1496
But even before that PR the API docs on Read the Docs were broken, as we had too many HTML pages and the build process on RTD was interrupted.
This is why I have approved the PR in the first place.
So https://github.com/twisted/twisted/pull/1466 should also be reverted
------------
So, if for this release the "official" Twisted documentation is to be hosted at https://twistedmatrix.com/documents/current/api/ then https://twistedmatrix.com/trac/ticket/10068 is not a blocker.
If the plan is to host the full and "official" Twisted documentation on Read The Docs, the https://twistedmatrix.com/trac/ticket/10068 is a blocker.
And the easiest solution is to do a new pydoctor release.
----------
I will try to follow up with Maarten over IRC to see if we can get a new pydoctor release.
In theory, the pydoctor release process is automated and any other Twisted developer from https://github.com/orgs/twisted/teams/twisted-contributors could do a release.
Maarten did the release work in the recent past but if he is busy, in theory anyone could step in and help with a new pydoctor release.
Regards
On Sat, 20 Feb 2021 at 17:47, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Hi,
I am confused. I thought that you and Thomas Graingert had completed all the automation and had changed the Twisted release scripts to host the documentation on readthedocs and that was all in place right now.
Is that not the case?
How much more work is necessary to completely host the next Twisted release's docs on readthedocs?
-- Craig
On Saturday, February 20, 2021, Adi Roiban <adiroiban@gmail.com> wrote:
Hi,
Sorry for the delay.
This was added as a blocker due to the changes from https://github.com/twisted/twisted/pull/1496 and the plan to use Read The Docs to host the official docs.
This is not a blocker if the official documentation for this release will continue to be hosted on the Twisted own server at https://twistedmatrix.com/documents/current/
It's a blocker only if the official documentation is to be hosted on Read The Docs at docs.twistedmatrix.com
I have updated the ticket to no longer block the release.
Cheers
On Sat, 20 Feb 2021 at 02:08, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Adi,
Is there anything to be done here, or is this still a release blocker?
-- Craig
On Sun, Feb 14, 2021 at 4:15 PM Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Hi,
This ticket: Use latest pydoctor release, https://twistedmatrix.com/trac/ticket/10068
is marked as a release blocker.
Has that been resolved or is there more work that needs to be done to fix that?
-- Craig
-- Adi Roiban
-- Adi Roiban
On Sunday, 28 February 2021 08:20:22 CET Craig Rodrigues wrote:
Thanks for the clarification.
You have provided a lot of useful details. There are a lot of interconnected pieces, with multiple people working on different things.
For now, I am going to proceed to finish the release as-is.
The documentation may not be automatically deployed as part of the release, but I think that can be done as a follow-on task.
I'll try to get a pydoctor release out today from the current master branch. While the release publishing is automated, writing the release notes is not, but that shouldn't take too long. There are some improvements in the works that aren't ready to merge yet but would improve Twisted's documentation, both in presentation (better navigation and presentation of types) and in accuracy (better handling of cyclic imports, which are becoming more common with the adoption of type annotations). However, I think that it would be preferable to get new docs out as soon as possible, rather than waiting for even better docs. Once those improvements are integrated, we could do another pydoctor release and update Twisted's API docs without there being a new Twisted release. Does this sound like a plan? Bye, Maarten
On Sunday, 28 February 2021 19:26:16 CET I wrote:
I'll try to get a pydoctor release out today from the current master branch. While the release publishing is automated, writing the release notes is not, but that shouldn't take too long.
pydoctor 21.2.0 is now available on PyPI: https://pypi.org/project/pydoctor Bye, Maarten
That sounds like a good plan. Thanks for working on this. Keep in mind the points regarding docs raised by Glyph here: https://twistedmatrix.com/pipermail/twisted-python/2021-February/065432.html -- Craig On Sunday, February 28, 2021, Maarten ter Huurne <maarten@treewalker.org> wrote:
However, I think that it would be preferable to get new docs out as soon as possible, rather than waiting for even better docs. Once those improvements are integrated, we could do another pydoctor release and update Twisted's API docs without there being a new Twisted release.
Does this sound like a plan?
On Sunday, 28 February 2021 22:02:48 CET Craig Rodrigues wrote:
That sounds like a good plan. Thanks for working on this.
Keep in mind the points regarding docs raised by Glyph here:
https://twistedmatrix.com/pipermail/twisted-python/2021-February/06543 2.html
Those are good points, but I meant API docs specifically and I don't think there should be any overlap between the wiki and the API docs. If I'm mistaken, please let me know. Bye, Maarten
-- Craig
On Sunday, February 28, 2021, Maarten ter Huurne <maarten@treewalker.org> wrote:
However, I think that it would be preferable to get new docs out as soon as possible, rather than waiting for even better docs. Once those improvements are integrated, we could do another pydoctor release and update Twisted's API docs without there being a new Twisted release.
Does this sound like a plan?
On Sun, Feb 28, 2021 at 1:50 PM Maarten ter Huurne <maarten@treewalker.org> wrote:
On Sunday, 28 February 2021 22:02:48 CET Craig Rodrigues wrote:
That sounds like a good plan. Thanks for working on this.
Keep in mind the points regarding docs raised by Glyph here:
https://twistedmatrix.com/pipermail/twisted-python/2021-February/06543 2.html
Those are good points, but I meant API docs specifically and I don't think there should be any overlap between the wiki and the API docs. If I'm mistaken, please let me know.
I think the direction that Glyph was mentioning is that if there are docs in the source tree, and duplicated docs on the wiki, we want to favor the docs in the source tree. Any duplicated docs on the wiki should either be deleted, or changed to put pointers to the docs in the source tree. With respect to API docs, I am not as familiar with the whole process, with how they are generated and what are doing with readthedocs vs. docs on twistedmatrix.com. API docs are generated and don't live in the source tree. For example: https://twistedmatrix.com/documents/current/api/twisted.html Is the long term direction to get rid of that, and point everything at readthedocs? Since you have done a lot of work in this area, can you shed some light on what you think the future direction of all this stuff should be with respect to the API docs? Thanks. -- Craig
On Sunday, 28 February 2021 23:04:49 CET Craig Rodrigues wrote:
With respect to API docs, I am not as familiar with the whole process, with how they are generated and what are doing with readthedocs vs. docs on twistedmatrix.com. API docs are generated and don't live in the source tree.
For example: https://twistedmatrix.com/documents/current/api/twisted.html
Is the long term direction to get rid of that, and point everything at readthedocs?
Yes, as far as I know the intention is to reduce the amount of infrastructure that has to be maintained by Twisted developers. Adi is doing the actual work for the migration; I only contribute indirectly by reviewing PRs that make pydoctor integrate better with Sphinx.
Since you have done a lot of work in this area, can you shed some light on what you think the future direction of all this stuff should be with respect to the API docs?
There are efforts to make the output of pydoctor more user friendly. This is mainly done by Tristan, but I occasionally work on it as well. In the next major release we should have a clearer presentation of parameter types, more navigation links and, if it's ready in time, a side bar containing a page outline. The main thing I'm working on is improving accuracy. Currently pydoctor can draw conclusions based on incomplete information if import cycles are present and with type annotations being added, there are more import cycles than before. To solve this, I want to separate the parsing and local analysis from the analysis that involves multiple documented objects, and run the latter only after all of the former is finished. On Twisted's side of improving accuracy, we currently have a lot of '@type' fields in the docstrings which aren't correct. In particular, there is still a lot of fallout from the Python 2 to 3 migration, where a documented type of 'str' can either mean 'bytes' or 'str'. I think that most '@type' fields should be replaced by type annotations, in which case mypy will verify that the documented type matches the type that the code is actually using. Something that might help here is to generate type stubs from the information found in docstrings and then automatically apply those stubs to Twisted's code. A possible implementation would be for pydoctor to export the result of docstring parsing as JSON and then a separate tool could generate type stubs from that data. Twisted's customizations to pydoctor are a bit of a pain, since changes in pydoctor break Twisted's API docs quite often. The customizations exist in two parts: code and templates. For the code customizations, these are done using subclassing, so the tight coupling makes it hard to change pydoctor's design without breaking the customizations. If possible, I'd prefer to remove the need for these customizations. If that isn't possible, we'd have to design a new plugin interface that is a lot more shielded from pydoctor's internals. For anyone interested, the details are discussed here: https://github.com/twisted/pydoctor/issues/315 For the template customizations, we're splitting up the templates into smaller chunks. This will eliminate or at least greatly reduce the amount of copy-pasted template content, which should allow Twisted to switch to new major pydoctor reeleases without having to sync the template content changes almost every time. The PR for the template rework is here: https://github.com/twisted/pydoctor/issues/299 Bye, Maarten
On Wed, Mar 3, 2021 at 7:18 AM Maarten ter Huurne <maarten@treewalker.org> wrote:
Yes, as far as I know the intention is to reduce the amount of infrastructure that has to be maintained by Twisted developers.
Adi is doing the actual work for the migration; I only contribute indirectly by reviewing PRs that make pydoctor integrate better with Sphinx.
There are efforts to make the output of pydoctor more user friendly. This is mainly done by Tristan, but I occasionally work on it as well. In the next major release we should have a clearer presentation of parameter types, more navigation links and, if it's ready in time, a side bar containing a page outline.
From what you have described, there are only a few more tasks to go before
Maarten, Thanks for the detailed explanation of what you, Adi, and Tristan are working towards. That is a lot of hard work! I think that using the type annotations in the code for documentation, rather than the Epydoc @type markup tag is WAY better than what we have now. this goal is fully completed, and the docs are published with the new pydoctor. I'm looking forward to the end result! -- Craig
participants (2)
-
Craig Rodrigues -
Maarten ter Huurne