A PEP to encourage people to mind docstrings in stub files
Hi! Sometimes you have to put docstrings in stub files e.g. when using code generation. In that case, stubs are the only place you can locate "objects" to add docstrings to. Thus it is frustrating to find that docstring-related software generally tends to ignore the presence of docstrings in stubs or to work incorrectly with those. I think we need a PEP to clarify the interaction between stubs and docstrings. This PEP might also encourage developers of docstring-related software to mind docstrings provided with stub files. -- Yours sincerely, Alexey Volobuev
Hi Alexey You wrote that docstring-related software generally fails to handle docstrings properly in stub files. I don't have experience of this, probably because I don't use such software. Please would you provide some examples. Ideal would be issued that have been raised (by you if need be) for such docstring-related software. This would help the people on this list get a better sense of the size and importance of this problem. best regards Jonathan
Some examples would be: * Pydoc does not generate documentation from stub files * Jupyter Notebooks don't show documentation from stub files (when using its shift+tab) There is a better way to grasp this problem. Consider you are developing some big library with stub files. .pyi files contain more function definitions than .py because some definitions are generated. So it is natural to write docstrings in .pyi files, just like we do in C++ .h files. The problem is there is no PEP or some other piece of Python documentation which states that is OK to write docstrings in .pyi files.
On Tue, Dec 22, 2020 at 08:07:25PM +0300, Alexey Volobuev wrote:
Hi!
Sometimes you have to put docstrings in stub files e.g. when using code generation. In that case, stubs are the only place you can locate "objects" to add docstrings to.
Thus it is frustrating to find that docstring-related software generally tends to ignore the presence of docstrings in stubs or to work incorrectly with those.
It is not clear to me whether you are talking about dynamic tools that operate on the objects themselves, like `help()` and `inspect.getdoc()`, or static analysis tools that are expecting to work with source code. Can you elaborate on the nature of the problem and your recommended solution? -- Steve
Actually, I am talking about both. It is not stated anywhere in PEPs or Python documentation that it is OK to put docstrings in .pyi files. So we have 2 problems: * Developers (of general purpose software) don't know should they put docstrings in .pyi files * Some other developers of docstring tools (like pydoc or help()) don't know if they should consider docstrings in .pyi files I think right now it is possible for any software dealing with docstrings to find those in corresponding .pyi files given .pyi matches corresponding .py.
Endorsing putting docstrings in stub files is an intriguing idea, but I can see some potential issues with the approach: From pep 484, the use-cases listed for stub files are: - Extension modules - Third-party modules whose authors have not yet added type hints - Standard library modules for which type hints have not yet been written - Modules that must be compatible with Python 2 and 3 - Modules that use annotations for other purposes Additionally, the pep mentions that stub files are "only for use by the type checker, not at runtime" -- Obviously these things could change, or be changed, but taking the current wording, I can see a few issues with trying to promote docstrings in stub files: 1. Arguably(?), The official way to get a docstring from an object is to call `inspect.getdoc()`, so if we want official support for docstrings in stubs, inspect.getdoc() would have to be able to load and query stub files for docstrings at "runtime". 2. At the very least we'd need a standard library way to extract docstrings from objects that includes doing so from stub files, ending up with many different implementations of stub loading/docstring extracting withing different projects would not be ideal. 3. The use-cases listed above (from pep 484) suggest that stubs are really a workaround for cases where type information can't be provided directly in the source code. It's not super obvious that these use cases apply to docstrings. Extension modules already can provide docstrings directly, and documenting third-party modules seems a little unlikely (happy to be wrong about this!) 4. Promoting stub files from technical 'workaround', to being recommended in non-workaround situations seems like a So, I think the above would need to be discussed/have some consensus as part of the debate. Personally, this isn't a change I'd directly support, but have no major opinions either way. Steve On Thu, Dec 24, 2020 at 10:57 AM Alexey Volobuev <alex.volobuev@gmail.com> wrote:
Actually, I am talking about both. It is not stated anywhere in PEPs or Python documentation that it is OK to put docstrings in .pyi files.
So we have 2 problems: * Developers (of general purpose software) don't know should they put docstrings in .pyi files * Some other developers of docstring tools (like pydoc or help()) don't know if they should consider docstrings in .pyi files
I think right now it is possible for any software dealing with docstrings to find those in corresponding .pyi files given .pyi matches corresponding .py. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/6NGRCJ... Code of Conduct: http://python.org/psf/codeofconduct/
Many Python developers rely heavily on docstrings for interactive feedback during coding. Docstrings are consumed by language servers like [Pylance](https://github.com/microsoft/pylance-release) and [Pyright](https://github.com/microsoft/pyright) for completion suggestions, signature help (when typing arguments for a function call), and hover text (when the mouse pointer hovers over a symbol). I can't speak for all Python language servers and smart editors, but Pylance and Pyright utilize docstrings found in stub files. If a stub file doesn't contain docstrings for a symbol, these tools attempt to locate the corresponding symbol in the Python source (".py") file and use any docstring found there. Finding the corresponding symbol in the source is sometimes difficult or impossible because stubs often have a different file layout than the corresponding source files, but these tools employ various heuristics that work in many cases. It's still somewhat hit or miss, so if we're going to talk about standardizing this, I'd want to encourage us to find a mechanism that is deterministic. If a symbol is implemented in another language (e.g. a Cython compiled binary), there is nowhere for a static analysis tool to find docstrings other than the type stub. We therefore encourage stub authors to include docstrings in their stubs if there is no corresponding source file. I'd assert that docstrings are way more important and useful for static analysis tools than they are for any dynamic (runtime) use cases, but maybe there are some runtime uses that I'm not considering. Language servers need to use static techniques because it is not safe (nor is it sufficiently performant) to load and execute Python source at edit time without the developer's knowledge just to extract docstrings. On a related topic, we should also discuss the practice of replacing default parameter values in stubs with "...". In cases where the type stub is the only source of truth for a language server, it would be valuable to preserve the default parameter values so they can be displayed in context as the developer is typing. Otherwise, they see "..." and don't know what default value to expect. -Eric --- Eric Traut Contributor to Pylance and Pyright Microsoft Corp.
Alexey Volobuev wrote:
Sometimes you have to put docstrings in stub files e.g. when using code generation. In that case, stubs are the only place you can locate "objects" to add docstrings to. Thus it is frustrating to find that docstring-related software generally tends to ignore the presence of docstrings in stubs or to work incorrectly with those. I think we need a PEP to clarify the interaction between stubs and docstrings. This PEP might also encourage developers of docstring-related software to mind docstrings provided with stub files.
Not having docstrings in type stubs is one of the small frustrations I have when working with Python. When working with combined Python/Typescript projects in PyCharm, I often find myself hitting Ctrl-B to jump to the definition of third-party JavaScript functions. This takes me to the appropriate *.d.ts file, which is JavaScript's equivalent of Python stubs files. But they also include JSDoc comments. I often wish I could do the same in Python. In my ideal world, every Python package (as well as the stdlib) would ship with stubs files that describe the package's public API, both for machine and human consumption. This could be automatically generated on build time from the sources, or it could be hand-crafted. This means, including types and docstrings. I also thing that providing the actual default, like Eric Traut suggested, would be a good idea.
How do we avoid having the docstrings in two places? Large communities (e.g. Jupyter notebook and IPython users) rely on docstrings available at runtime. There should be a tool to update the docstrings in stubs before I could recommend duplicating them there. Even for stubs corresponding to extension modules we would need such a tool, since those also typically expose docstrings (defined in the C code) at runtime. On Fri, Jan 1, 2021 at 7:11 AM <srittau@rittau.biz> wrote:
Alexey Volobuev wrote:
Sometimes you have to put docstrings in stub files e.g. when using code generation. In that case, stubs are the only place you can locate "objects" to add docstrings to. Thus it is frustrating to find that docstring-related software generally tends to ignore the presence of docstrings in stubs or to work incorrectly with those. I think we need a PEP to clarify the interaction between stubs and docstrings. This PEP might also encourage developers of docstring-related software to mind docstrings provided with stub files.
Not having docstrings in type stubs is one of the small frustrations I have when working with Python. When working with combined Python/Typescript projects in PyCharm, I often find myself hitting Ctrl-B to jump to the definition of third-party JavaScript functions. This takes me to the appropriate *.d.ts file, which is JavaScript's equivalent of Python stubs files. But they also include JSDoc comments. I often wish I could do the same in Python.
In my ideal world, every Python package (as well as the stdlib) would ship with stubs files that describe the package's public API, both for machine and human consumption. This could be automatically generated on build time from the sources, or it could be hand-crafted. This means, including types and docstrings. I also thing that providing the actual default, like Eric Traut suggested, would be a good idea. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PPXVKI... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Am 02.01.21 um 01:44 schrieb Guido van Rossum:
How do we avoid having the docstrings in two places? Large communities (e.g. Jupyter notebook and IPython users) rely on docstrings available at runtime. There should be a tool to update the docstrings in stubs before I could recommend duplicating them there.
Even for stubs corresponding to extension modules we would need such a tool, since those also typically expose docstrings (defined in the C code) at runtime.
Agreed. Having to maintain two (identical) docstrings will not work. I have set aside some time next week to work on typing-related things and will see whether I can cook up something. - Sebastian
participants (8)
-
Alexey Volobuev
-
Eric Traut
-
Guido van Rossum
-
Jonathan Fine
-
Sebastian Rittau
-
srittau@rittau.biz
-
Stestagg
-
Steven D'Aprano