Re: [Twisted-Python] Release blocker: Use latest pydoctor release ?
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
Thanks for the clarification. You have provided a lot of useful details. There are a lot of interconnected pieces, with multiple people working on different things. For now, I am going to proceed to finish the release as-is. The documentation may not be automatically deployed as part of the release, but I think that can be done as a follow-on task. -- Craig On Sat, Feb 27, 2021 at 4:53 PM Adi Roiban <adiroiban@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On Sunday, 28 February 2021 08:20:22 CET Craig Rodrigues wrote:
I'll try to get a pydoctor release out today from the current master branch. While the release publishing is automated, writing the release notes is not, but that shouldn't take too long. There are some improvements in the works that aren't ready to merge yet but would improve Twisted's documentation, both in presentation (better navigation and presentation of types) and in accuracy (better handling of cyclic imports, which are becoming more common with the adoption of type annotations). However, I think that it would be preferable to get new docs out as soon as possible, rather than waiting for even better docs. Once those improvements are integrated, we could do another pydoctor release and update Twisted's API docs without there being a new Twisted release. Does this sound like a plan? Bye, Maarten
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On Sunday, 28 February 2021 19:26:16 CET I wrote:
pydoctor 21.2.0 is now available on PyPI: https://pypi.org/project/pydoctor Bye, Maarten
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
That sounds like a good plan. Thanks for working on this. Keep in mind the points regarding docs raised by Glyph here: https://twistedmatrix.com/pipermail/twisted-python/2021-February/065432.html -- Craig On Sunday, February 28, 2021, Maarten ter Huurne <maarten@treewalker.org> wrote:
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On Sunday, 28 February 2021 22:02:48 CET Craig Rodrigues wrote:
Those are good points, but I meant API docs specifically and I don't think there should be any overlap between the wiki and the API docs. If I'm mistaken, please let me know. Bye, Maarten
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
On Sun, Feb 28, 2021 at 1:50 PM Maarten ter Huurne <maarten@treewalker.org> wrote:
I think the direction that Glyph was mentioning is that if there are docs in the source tree, and duplicated docs on the wiki, we want to favor the docs in the source tree. Any duplicated docs on the wiki should either be deleted, or changed to put pointers to the docs in the source tree. With respect to API docs, I am not as familiar with the whole process, with how they are generated and what are doing with readthedocs vs. docs on twistedmatrix.com. API docs are generated and don't live in the source tree. For example: https://twistedmatrix.com/documents/current/api/twisted.html Is the long term direction to get rid of that, and point everything at readthedocs? Since you have done a lot of work in this area, can you shed some light on what you think the future direction of all this stuff should be with respect to the API docs? Thanks. -- Craig
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On Sunday, 28 February 2021 23:04:49 CET Craig Rodrigues wrote:
Yes, as far as I know the intention is to reduce the amount of infrastructure that has to be maintained by Twisted developers. Adi is doing the actual work for the migration; I only contribute indirectly by reviewing PRs that make pydoctor integrate better with Sphinx.
There are efforts to make the output of pydoctor more user friendly. This is mainly done by Tristan, but I occasionally work on it as well. In the next major release we should have a clearer presentation of parameter types, more navigation links and, if it's ready in time, a side bar containing a page outline. The main thing I'm working on is improving accuracy. Currently pydoctor can draw conclusions based on incomplete information if import cycles are present and with type annotations being added, there are more import cycles than before. To solve this, I want to separate the parsing and local analysis from the analysis that involves multiple documented objects, and run the latter only after all of the former is finished. On Twisted's side of improving accuracy, we currently have a lot of '@type' fields in the docstrings which aren't correct. In particular, there is still a lot of fallout from the Python 2 to 3 migration, where a documented type of 'str' can either mean 'bytes' or 'str'. I think that most '@type' fields should be replaced by type annotations, in which case mypy will verify that the documented type matches the type that the code is actually using. Something that might help here is to generate type stubs from the information found in docstrings and then automatically apply those stubs to Twisted's code. A possible implementation would be for pydoctor to export the result of docstring parsing as JSON and then a separate tool could generate type stubs from that data. Twisted's customizations to pydoctor are a bit of a pain, since changes in pydoctor break Twisted's API docs quite often. The customizations exist in two parts: code and templates. For the code customizations, these are done using subclassing, so the tight coupling makes it hard to change pydoctor's design without breaking the customizations. If possible, I'd prefer to remove the need for these customizations. If that isn't possible, we'd have to design a new plugin interface that is a lot more shielded from pydoctor's internals. For anyone interested, the details are discussed here: https://github.com/twisted/pydoctor/issues/315 For the template customizations, we're splitting up the templates into smaller chunks. This will eliminate or at least greatly reduce the amount of copy-pasted template content, which should allow Twisted to switch to new major pydoctor reeleases without having to sync the template content changes almost every time. The PR for the template rework is here: https://github.com/twisted/pydoctor/issues/299 Bye, Maarten
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
On Wed, Mar 3, 2021 at 7:18 AM Maarten ter Huurne <maarten@treewalker.org> wrote:
From what you have described, there are only a few more tasks to go before
Maarten, Thanks for the detailed explanation of what you, Adi, and Tristan are working towards. That is a lot of hard work! I think that using the type annotations in the code for documentation, rather than the Epydoc @type markup tag is WAY better than what we have now. this goal is fully completed, and the docs are published with the new pydoctor. I'm looking forward to the end result! -- Craig
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On Sunday, 28 February 2021 08:20:22 CET Craig Rodrigues wrote:
I'll try to get a pydoctor release out today from the current master branch. While the release publishing is automated, writing the release notes is not, but that shouldn't take too long. There are some improvements in the works that aren't ready to merge yet but would improve Twisted's documentation, both in presentation (better navigation and presentation of types) and in accuracy (better handling of cyclic imports, which are becoming more common with the adoption of type annotations). However, I think that it would be preferable to get new docs out as soon as possible, rather than waiting for even better docs. Once those improvements are integrated, we could do another pydoctor release and update Twisted's API docs without there being a new Twisted release. Does this sound like a plan? Bye, Maarten
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On Sunday, 28 February 2021 19:26:16 CET I wrote:
pydoctor 21.2.0 is now available on PyPI: https://pypi.org/project/pydoctor Bye, Maarten
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
That sounds like a good plan. Thanks for working on this. Keep in mind the points regarding docs raised by Glyph here: https://twistedmatrix.com/pipermail/twisted-python/2021-February/065432.html -- Craig On Sunday, February 28, 2021, Maarten ter Huurne <maarten@treewalker.org> wrote:
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On Sunday, 28 February 2021 22:02:48 CET Craig Rodrigues wrote:
Those are good points, but I meant API docs specifically and I don't think there should be any overlap between the wiki and the API docs. If I'm mistaken, please let me know. Bye, Maarten
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
On Sun, Feb 28, 2021 at 1:50 PM Maarten ter Huurne <maarten@treewalker.org> wrote:
I think the direction that Glyph was mentioning is that if there are docs in the source tree, and duplicated docs on the wiki, we want to favor the docs in the source tree. Any duplicated docs on the wiki should either be deleted, or changed to put pointers to the docs in the source tree. With respect to API docs, I am not as familiar with the whole process, with how they are generated and what are doing with readthedocs vs. docs on twistedmatrix.com. API docs are generated and don't live in the source tree. For example: https://twistedmatrix.com/documents/current/api/twisted.html Is the long term direction to get rid of that, and point everything at readthedocs? Since you have done a lot of work in this area, can you shed some light on what you think the future direction of all this stuff should be with respect to the API docs? Thanks. -- Craig
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On Sunday, 28 February 2021 23:04:49 CET Craig Rodrigues wrote:
Yes, as far as I know the intention is to reduce the amount of infrastructure that has to be maintained by Twisted developers. Adi is doing the actual work for the migration; I only contribute indirectly by reviewing PRs that make pydoctor integrate better with Sphinx.
There are efforts to make the output of pydoctor more user friendly. This is mainly done by Tristan, but I occasionally work on it as well. In the next major release we should have a clearer presentation of parameter types, more navigation links and, if it's ready in time, a side bar containing a page outline. The main thing I'm working on is improving accuracy. Currently pydoctor can draw conclusions based on incomplete information if import cycles are present and with type annotations being added, there are more import cycles than before. To solve this, I want to separate the parsing and local analysis from the analysis that involves multiple documented objects, and run the latter only after all of the former is finished. On Twisted's side of improving accuracy, we currently have a lot of '@type' fields in the docstrings which aren't correct. In particular, there is still a lot of fallout from the Python 2 to 3 migration, where a documented type of 'str' can either mean 'bytes' or 'str'. I think that most '@type' fields should be replaced by type annotations, in which case mypy will verify that the documented type matches the type that the code is actually using. Something that might help here is to generate type stubs from the information found in docstrings and then automatically apply those stubs to Twisted's code. A possible implementation would be for pydoctor to export the result of docstring parsing as JSON and then a separate tool could generate type stubs from that data. Twisted's customizations to pydoctor are a bit of a pain, since changes in pydoctor break Twisted's API docs quite often. The customizations exist in two parts: code and templates. For the code customizations, these are done using subclassing, so the tight coupling makes it hard to change pydoctor's design without breaking the customizations. If possible, I'd prefer to remove the need for these customizations. If that isn't possible, we'd have to design a new plugin interface that is a lot more shielded from pydoctor's internals. For anyone interested, the details are discussed here: https://github.com/twisted/pydoctor/issues/315 For the template customizations, we're splitting up the templates into smaller chunks. This will eliminate or at least greatly reduce the amount of copy-pasted template content, which should allow Twisted to switch to new major pydoctor reeleases without having to sync the template content changes almost every time. The PR for the template rework is here: https://github.com/twisted/pydoctor/issues/299 Bye, Maarten
![](https://secure.gravatar.com/avatar/469df05f5dfd5b75fb3cb3a0868d36bf.jpg?s=120&d=mm&r=g)
On Wed, Mar 3, 2021 at 7:18 AM Maarten ter Huurne <maarten@treewalker.org> wrote:
From what you have described, there are only a few more tasks to go before
Maarten, Thanks for the detailed explanation of what you, Adi, and Tristan are working towards. That is a lot of hard work! I think that using the type annotations in the code for documentation, rather than the Epydoc @type markup tag is WAY better than what we have now. this goal is fully completed, and the docs are published with the new pydoctor. I'm looking forward to the end result! -- Craig
participants (2)
-
Craig Rodrigues
-
Maarten ter Huurne