Runtime-accessible attribute docstrings – take 2
A few weeks ago, I proposed on this mailing list to write docstrings for class attributes like this: @dataclass class A: x: int """Docstring for x.""" The main criticism, I think, was that it is weird to have the docstring *below* the attribute. To solve this problem, I propose to introduce a new kind of string: a d-string ('d' for 'docstring'; alternatively also 'v' because it looks a bit like a downward arrow, or 'a' for 'attribute docstring'). A d-string is a normal string, except that it is stored in __attrdoc__ when used inside a class. It is stored with the name of the variable *below* it as the key. Examples: @dataclass class InventoryItem: """Class for keeping track of an item in inventory.""" d"""Short name of the item.""" name: str d"""Price per unit in dollar.""" unit_price: float d"""Available quantity currently in the warehouse.""" quantity_on_hand: int = 0 InventoryItem.__attrdoc__ == { "name": "Short name of the item.", "unit_price": "Price per unit in dollar.", "quantity_on_hand": "Available quantity currently in the warehouse.", } ---- class HttpRequest(Enum): """Types of HTTP requests.""" d"""GET requests are used to retrieve data.""" GET = auto() d"""POST requests are used to insert/update remote data.""" POST = auto() HttpRequest.__attrdoc__ == { "GET": "GET requests are used to retrieve data.", "POST": "POST requests are used to insert/update remote data.", } d-strings can be combined with raw strings by using the prefix rd or dr. d-strings could also be used to document module-level constants: # in my_module.py: d"Number of days in a week." DAYS_PER_WEEK: Final = 7 my_module.__attrdoc__ == {"DAYS_PER_WEEK": "Number of days in a week."} -Thomas
I think it's an interesting idea. I made the same or at least similar suggestion in the previous thread, but it didn't receive any responses. I assume this is because people weren't very interested (but I also understand people are busy). Here's that message: https://mail.python.org/archives/list/python-ideas@python.org/message/SJEPCZ... --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler On Wed, Dec 8, 2021 at 6:56 AM <tmkehrenberg@gmail.com> wrote:
On Wed, Dec 08, 2021 at 11:50:55AM -0000, tmkehrenberg@gmail.com wrote:
Why would it be weird to put a docstring below the object that owns it? We already do that for class and function signatures. def func(): """Docstring is below the signature""" I would strongly prefer to have the docstring follow the attribute, like function and class docstrings follow the signature, not lead it.
To solve this problem, I propose to introduce a new kind of string: a d-string
Not another string prefix :-(
I'd like to suggest an alternative syntax that doesn't need a new string prefix, although it does require new syntax. I think it is fully backwards compatible. Rather than implicitly linking a bare string to the following (or previous?) attribute, this would explicitly link them by putting them on the same line (at least for short docstrings). Full declaration syntax becomes: name : Type = value : docstring_expression with the various parts being optional. Example: @dataclass class InventoryItem: """Class for keeping track of an item in inventory.""" name: str : "Short name of the item" unit_price: float : "Price per unit in dollar." quantity_on_hand: int = 0 : ( "Available quantity currently in the warehouse.") The docstring expression could evaluate to any value, not necessarily a string, but conventionally one would use a string, and I expect style guides would discourage or ban non-strings. (But see below for a possible objection.) Aside from the name, each part of the syntax is optional: # Type hint only. name: Type # Assignment only. name = value # Declare a type hint, and a docstring. name: Type : docstring # Assign a value, and a docstring, with no type hint. name = value : docstring # And all three: type hint, value and docstring. name: Type = value : docstring If this makes sense, one might even have an attribute with no type hint or value, only a docstring: name : : docstring which would assign the docstring to `__attrdoc__` but otherwise have no effect. Formatting long lines could be done either with parentheses as in the quantity_on_hand example above, or backslashes for those who don't hate them: quantity_on_hand: \ int = 0 : \ "Available quantity currently in the warehouse." Strings of course can use any prefix, raw strings, f-strings, triple- quoted strings, whatever you like. Since the docstring part is just an expression, it doesn't need to be a string literal. If people object to the "arbitrary expression" part as too YAGNI, we could limit the docstring part to be: * a string literal (using any prefix); * an f-string; * or one of the above, parenthesed. And maybe even drop the f-string and just allow an string literal, optionally with parentheses. -- Steve
On Wed, Dec 8, 2021 at 8:41 AM Steven D'Aprano <steve@pearwood.info> wrote:
IMO this is a much better idea than the OP's, or the similar d-string idea I had come up with. Here's an example of the kind of thing I imagine doing with this feature. If I were doing a set of engineering calculations, I might get started by writing pseudocode (but using the new docstrings) like this: #### cantilevered_beam.py ##### """Checking cantilevered beam designs.""" # material properties E_s : float : "elastic modulus of steel" F_y: float : "strength of steel" # beam properties I_beam : float : "moment of inertia of beam" S_beam : float : "section modulus of beam" L_beam : float : "cantilever length" phi_M : float : "moment limit of beam" # loads P_u : float : "applied ultimate force" M_u : float : "applied ultimate bending moment" # deflections delta_max : float : "maximum beam deflection" delta_limit : float : "beam deflection limit" check : bool : "Is the beam OK? - OK (true) or No Good (false)" ### end of module #### This code doesn't yet do anything, but there is a ton of information being shared here- information that can be used to write code later, quickly. It is a very nice module template that could be used over and over again (writing out the details of actual projects later). Of course you could write pseudocode containing all of this information today with regular strings. But the difference is that, if this were to be a language feature, the information is going to be available programmatically BEFORE actually implementing anything at all. If I were in a jupyter notebook, I could write something like this (you could so similar things a REPL): [CELL 1] import cantilevered_beam print("\n".join(f"{name}:\t{docstring}" for name, docstring in cantilevered_beam.__attrdoc__.items())) ...and it would nicely output all the documentation written in the template module above for my reference, and in the next cell I would interactively compose code like this (probably copying and pasting from above, and then typing over the type-hints and docstrings): [CELL 2] #### my_cantilevered_beam.py ##### """A check for a cantilevered beam design.""" # material properties E_s = 29_000 # ksi F_y = 50 # ksi # beam properties I_beam = 1_830 # in^4 for W 24x68 beam S_beam = 154 # in^3 for W 24x68 beam L_beam = 190.25 # inches phi_M = 0.9 * F_y * S_beam # kip-in P_u = 4.947 # kips M_u = P_u * L_beam # kip-in delta_max = P_u * L_beam**3 / (48 * E_s * I_beam) # inches delta_limit = 1 # inches check = True if M_u <= phi_M and delta_max <= delta_limit else False Could I accomplish something similar today using regular strings? Sure. But it seems to me that having an official way to document attributes/members has been an oft requested feature, and I really like imaging the kinds of things I would be inclined to do with it if it were supported in the language (rather than something I was off doing on my own as a low-experience programmer, and wondering if it was dumb or a mistake). --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Wed, Dec 08, 2021 at 11:50:55AM -0000, tmkehrenberg@gmail.com wrote:
I like the idea of standardizing a name like __attrdoc__. Broadly, I think it is a good idea to enable attribute docstrings. But I think bare strings are probably not a good idea, and I definitely think adding more string prefixes is not a good idea. Of the various ideas noted here and before, I think the one that resonates with me the most is to add a docstring parser (and possibly allow others to write their own if they write their docstrings in a different way). This would look like this. # add a docstring parser decorator @parse_attrdoc_from_docstring class A: """ Class docstring. x: Docstring for x y: Docstring for y """ x: int y: bool = False And functionally, that would be equivalent to doing the following directly. # or explicitly set __attrdoc__ class B: """Class docstring.""" __attrdoc__ = { "x": "Docstring for x", "y": "Docstring for y", } x: int y: bool = False If someone wanted to document an already written library and has a different (but consistent) convention in their class docstrings for class variables, then it shouldn't be too hard to modify the default parser. Maybe? - DLD
I propose there is already a viable option in typing.Annotated. Example: @dataclass class InventoryItem: """Class for keeping track of an item in inventory.""" name: Annotated[str, "Short name of the item."] unit_price: Annotated[float, "Price per unit in dollar."] ... I've been using this in production code for about a year (with code that generates OpenAPI document), with additional validation constraints, and it's proving to be quite usable. Paul On Wed, 2021-12-08 at 11:50 +0000, tmkehrenberg@gmail.com wrote:
On Wed, Dec 8, 2021 at 12:46 PM Paul Bryan <pbryan@anode.ca> wrote:
As I noted in the previous thread, a big downside IMO of using Annotated for docstrings is it really makes the help() output quite messy. The "docstrings" appear in at least 5 different places (as part of the Annotation objects). This could be fixed, I guess. But that would skip over a question that gets raised here, which is: are a docstring and an annotation really the same thing? Are there disadvantages to assuming they are? --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
I believe a Annotated[..., str] could become an attribute docstring if by consensus we deem it to be so. I can't see any disadvantages, perhaps save for the verbosity of `Annotated`. It certainly seems like an advantage to use an existing mechanism rather than define a new one that would appear to require changes to the interpreter. On Wed, 2021-12-08 at 13:04 -0500, Ricky Teachey wrote:
On Wed, Dec 8, 2021 at 1:20 PM Paul Bryan <pbryan@anode.ca> wrote:
I think this would be better than nothing. But it is a little verbose. And it requires you to supply the type hint. I use type hinting, but I don't want to use it every time I provide an Annotation/docstring. Maybe we could do both: establish the 2nd argument of Annotated as the docstring (and adjust the output of help() and other tools accordingly), but also provide a new "shortcut syntax" as Steven proposed: E_s : float : "the elastic modulus of steel" ... is sugar for: E_s : typing.Annotated[float, "the elastic modulus of steel"] ...and: x :: "spam" ...is sugar for: x : typing.Annotated[typing.Any, "spam"] --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
I'm using docstrings bellow the attributes (analogous to functions and classes), I think it works well. It helps with distinguishing the class docstring from the arguments. On 2021-12-08 13:25:55, Ricky Teachey wrote:
In this case, why not use the current behaviour of putting the docstring below the attribute? Shouldn't typing be encouraged on dataclasses, at least?
On Thu, Dec 9, 2021 at 11:08 AM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
Shouldn't typing be encouraged on dataclasses, at least?
Typing shouldn’t be encouraged for any particular class, nor the language itself. Typing policy is a matter for the project/organization to decide. But Annotations are required by dataclasses in any case. That’s how the decorator identifies fields. The contents of those annotations are up to the user. -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Dec 9, 2021 at 3:08 PM Christopher Barker <pythonchb@gmail.com> wrote:
Since not everyone uses dataclasses and to avoid confusion: what you mean here is that type HINTS are required by dataclasses, not Annotations. An Annotated is a type of type hint that contains an annotation, which we're loosely calling a member docstring in this thread (currently, a "member docstring" is not really a thing that exists, unless you count a comment or a string nearby the statement creating the member). On Thu, Dec 9, 2021 at 2:01 PM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
What is meant here by "the current behaviour"? Member docstrings are currently not a feature supported by the language. If you like putting a string under the member definition for reference, that sounds really nice. But baking it into the language so that the docstring has to come right after the member seems limiting to me. This location requirement makes a lot of sense for modules and classes. But for class members, I'd really like to be able to organize the docstring location in a more flexible way, like (using Steven's proposed syntax): class Steel: """A simple material model of steel.""" # parameters modulus :: "the linear elastic modulus" nu :: "the poisson's ratio" gamma: "the density" # values modulus: float = 29e6 # ksi nu: float = 0.3 gamma: float = 490 # lbs per cu ft --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Thu, Dec 9, 2021 at 3:23 PM Ricky Teachey <ricky@teachey.org> wrote:
A correction to my correction (Chris Barker very kindly replied to me off-list about it; what a mensch): type hints aren't required in a dataclass. All that is required is *something* in the "type hint slot" of the expression, and that something is often (officially?) called an annotation. So we have at least 3 meanings of annotation/annotate/annotated possibly at work in this thread (what fun): 1. an annotation (the slot where type hints usually go these days, but really, you can put anything there even when constructing a dataclass), 2. the Annotated type hint type, and 3. the annotation that is the second argument of the Annotated brackets operator. How confusing! --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Fri, Dec 10, 2021 at 11:46 AM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
I meant to ask about a (global) module member, not the module docstring itself. Like MY_GLOBAL below: """This is the module docs""" MY_GLOBAL = None """MY_GLOBAL docs""" class CLS: """CLS docs""" attr: int """CLS.attr doc""" Is this "global docstring" recognized by Sphinx as a docstring, too? --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Fri, Dec 10, 2021 at 9:57 PM Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote:
To my mind Sphinx is sufficiently widely used
And the system used for the official Python docs. that this settles the
"above or below" question.
So yes :-) However, what I haven’t seen in this thread is discussion of what I think is the key question: Where/how should class attribute doc strings be stored? Tacked on to the class __doc__ ? Another dict? __attr_doc__ Added to __annotaions__ ? Something else? If they are to be available at run time, they need to go somewhere… -CHB
--
Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sat, Dec 11, 2021, 1:19 PM Christopher Barker <pythonchb@gmail.com> wrote:
The __annotations__ already exists. Is that a point in favor? If the syntax could become sugar for creating an Annotated object in __annotations__, this would be a pretty convenient location to find them. On the other hand, not every type hinted variable will have a docstring. It might be inconvenient to have them there only. It seems like storing them in a so called __attr_doc__ might be the most straightforward thing to do. If that were done, does it matter that the contents of __attr_doc__ and __annotations__ are not coupled? The strings might need to be modified. Would be inconvenient to have to modify two places in such a situation.
On Sat, Dec 11, 2021 at 10:49 AM Ricky Teachey <ricky@teachey.org> wrote:
The __annotations__ already exists. Is that a point in favor?
Yes and no. Right now, any object can be stored in annotations — having a docstring tacked on would break who knows how much code. However, there is the new inspect.get_annotations function, which could be adapted to process doc strings. https://docs.python.org/3/howto/annotations.html If the syntax could become sugar for creating an Annotated object in
__annotations__, this would be a pretty convenient location to find them.
On the other hand, not every type hinted variable will have a docstring.
And not every attribute with a docstring would have an annotation. It seems like storing them in a so called __attr_doc__ might be the most
straightforward thing to do.
Yes, as doc strings really are a different thing. Though one might to also document other things like function parameters— which would be an argument for extending what __annotations__ is used for. If that were done, does it matter that the contents of __attr_doc__ and
__annotations__ are not coupled?
I don’t think so — they are quite different concepts. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
I think that because Sphinx could interpret strings below attributes for documentation, that perhaps we should go in that direction in Python proper. Personally, I find this more readable: class C: x: Annotated[str, "Doc string"] y: Annotated[int, "Doc string"] over: class C: x: str "Doc string" y: int "Doc string" On Sun, 2021-12-12 at 10:00 +1100, Steven D'Aprano wrote:
On Sat, Dec 11, 2021 at 3:03 PM Steven D'Aprano <steve@pearwood.info> wrote:
Well, no. In fact, you could always put anything you wanted into an annotation: In [10]: class C2: ...: x: "a docstring" = 123 ...: In [11]: inspect.get_annotations(C2) Out[11]: {'x': 'a docstring'} So the typing module contains a nifty thing they've called "Annotatated", and, I imagine type checkers (like MyPy) do something sensible with them, but there's nothing else going on here. Actually yes, there is, the typing,get_type_hints function strips out the annotation: In [13]: class C: ...: x: Annotated[int, "Doc string"] = 123 ...: In [14]: typing.get_type_hints(C) Out[14]: {'x': int} Anyway, take a look at the docs for Annotated: https://docs.python.org/3/library/typing.html#typing.Annotated It has a number of possible unspecified uses, but fundamentally, it's about the adding information to the type, not to the attribute -- e.g. not really intended for docstrings. Sure, typing.Annotated *could* be used that way, but I don't think we can say that the problem is solved. Heck you could just as easily provide a tuple for the annotation: In [17]: class C: ...: x: ("Doc string", int) = 123 ...: In [18]: inspect.get_annotations(C) Out[18]: {'x': ('Doc string', int)} The use cases for annotations are a bit up in the air at the moment -- see the ongoing discussion on python-dev. But I don't think there's any doubt that any future endorsed use of them will be compatible with static typing (if not restricted to it), so extending the use of annotations for docstrings would take a bit of work to define and accept that use. Perhaps a better way to do that than to use Annotated would be to introduce new "thing", maybe in teh typoing module, like "Documented": class C: x: Documented(doc_string="this is the doc string", type=int) = 123 and then inspect.get_annotations and typing,get_type_hints could be taught to extract the type from the Documented object, and we could add an inspect.get_docstrings() function, so that: typing.get_type_hints(C) {'x': int} inspect.get_docstrings(C) {'x': 'this is the doc string'} And, of course, future syntax could automatically create a Documented() object if there was a docstring in the appropriate place. Or just have a new dunder for it, e.g. __attr_doc__ Anyway, what I'm getting at is that it needs to be determined whare to put the docstrings before we can get very far with this idea. -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sat, 2021-12-11 at 17:02 -0800, Christopher Barker wrote:
Ah, good point. I've conflated the two because dataclass attributes are used to define the schema of the dataclass. For general purpose attribute docstrings (module, class), I agree Annotated is probably inappropriate.
I think this will suffer from the same problem as Annotated: it would document the type, not the attribute itself.
Anyway, what I'm getting at is that it needs to be determined whare to put the docstrings before we can get very far with this idea.
How is Sphinx is pulling the string below the attribute? Whatever mechanism is to be used, I propose it must allow for runtime introspection.
On Sat, Dec 11, 2021 at 05:02:39PM -0800, Christopher Barker wrote:
On Sat, Dec 11, 2021 at 3:03 PM Steven D'Aprano <steve@pearwood.info> wrote:
Didn't we decide there was an existing feature for this, no need for new syntax?
Well, no. In fact, you could always put anything you wanted into an annotation:
Right, and we want to put a documentation string in it! :-)
I don't think that the docs argue against this. Annotated is documented as the way to put arbitrary metadata into an annotation. What you do with it is up to the consumer of the metadata. The docs talk about this associating the metadata with the type, but that's because this is the *typing* module and *type* hints are still the primary use-case for annotations. And despite what the docs say, the metadata is actually, literally, associated with the variable (or its name at least). The annotation that gets stored in __annotations__ is **not** {T: "metadata"} as a naive reading of the docs might trick you into believing, but {"var": Annotated[T, "metadata"]} In any case, we can still use annotations for anything we like, and while typing is definitely and certainly the primary use for them, there will always be a way to opt-out. (Just don't run mypy!) Don't need type hints at all? Great, put your doc string as the annotation. μ: 'Number of steps from x0 to the start of the cycle.' λ: 'Length (period) of the cycle.' y: 'Value of f at the start of the cycle.' = None You might want to annotate the class with `no_type_check` if there is any chance the user might want to run mypy over your module, but that's entirely up to you. You do want type hints? Great, use Annotated. If there's an attribute that you don't want to give a type hint to, just use Any. (A small price to pay for using an existing feature, instead of waiting until Python 3.11 or 3.27 when it gets added to the core language.) μ: Annotated[int, 'Number of steps from x0 to the start of the cycle.'] λ: Annotated[int, 'Length (period) of the cycle.'] y: Annotated[Any, 'Value of f at the start of the cycle.' ] = None No new syntax required. No bare strings floating around. You can generate the doc strings dynamically, they don't have to be literals. lunch: Annotated[Meal, ' '.join(['spam']*SPAM_COUNT)]
Sure, typing.Annotated *could* be used that way, but I don't think we can say that the problem is solved.
I think that from a language standpoint, we absolutely can say that the problem is solved. We don't need new syntax. We *might* want to standardise on a decorator or other tool to extract those docstrings, possibly sticking them in a new __attrdoc__ dunder. Being a dunder, we should get the core dev's approval for this, even if it is a third-party tool that uses it. (I assume we don't already use __attrdoc__ for something?) If the first use of this is dataclasses, then that would be de facto core dev approval. So we have a tool that takes an annotation: var: Annotated[T, docstring, *args] extracts out the docstring to an __attrdoc__ dunder: # Class.__attrdoc__ {'var': docstring} and removes the annotation from the annotation: # Class.__annotation__ {'var': Annotated[T, *args]} # or just T if *args is empty? Dataclasses can automatically apply this tool (subject to backwards compatibility constraints), or we can apply it as a decorator to any other class we want. So we've gone from the *hard* problem of making a language change (new syntax, build the functionality into the interpreter) to the *easy* problem of writing a decorator.
Sure, annotations can be anything you like. But external static typing tools may not support this, and will be confused or break. Same for third-party runtime tools. Best practice is to stick as closely to the standard type-hinting behaviour as you can, or to completely opt-out with no_type_hints.
Annotated is explicitly documented as *arbitrary* metadata that is entirely up to the application to interpret. There's no reason to think that will change.
Now you're just arguing about the colour of the bike-shed :-) Fine, if we want to increase the complexity of *every* tool that operates on annotations, to specially recognise this new pseudo-type, we could make them recognise Documented["docstring", T] where T is optional and to be interpreted as Any if missing. But I suspect that the type checkers will say "Just use Annotated" but what do I know? Having introduced that special Documented type-hint, we still need to modify dataclasses to extract the docstring from it, and we still need to make it available to other classes via a decorator. So apart from the spelling, which I don't think is a win, we gain nothing. -- Steve
On Sat, Dec 11, 2021, 10:58 PM Steven D'Aprano <steve@pearwood.info> wrote:
But Steve, since the most utilized documentation tool in the python universe, sphinx, doesn't look at Annotated or in the lowercase-annotation part of an expression for this and instead looks at a bare string below the member definition, and since I presume it had been doing this for a long time (before Annotated was added not long ago, and probably before the annotation too?), doesn't this mean that right now there are at least three competing ways being used to provide attribute docstrings? It really seems like it would be good for the language to standardize this. Just like function or module docstrings were standardized long ago. As far as using the lowercase-annotation for the docstring: in a world of more and more type hinted python (to be clear: I don't always use type hints but I do like this world), if you want to make you don't create a problem later, using the annotation slot for your docstring: x: "spam" ... isn't really an option. So you have little choice but to use typing.Annotated right now (unless you want to take advantage of the sphinx way of doing it). But the problem with Annotated is Matt, imo: - it's a rather long spelling - you have to import it from typing - the help output isn't very useful when you use Annotated right now (this can be improved but probably only if Annotated is officially specified as the "right place") - without an official blessed agreement that the existing Annotated feature ought to be used for docstrings, there is little reason for other 3rd party help/documentation tools to look in that location for these docstrings I just think there's more to do here than add a decorator.
On Sun, Dec 12, 2021 at 12:38:06AM -0500, Ricky Teachey wrote:
But Steve, since the most utilized documentation tool in the python universe, sphinx, doesn't look at Annotated
Yet.
or in the lowercase-annotation part of an expression
Yet.
And before we had PEP 484 there were multiple competing and incompatible conventions for putting type hints in docstrings. https://www.python.org/dev/peps/pep-0484/ There are: - Epytext (Javadoc style) - ReST (Sphinx) - Googledoc - Numpydoc although some of them may no longer be widely used. https://stackoverflow.com/questions/3898572/what-are-the-most-common-python-... https://www.adrian.idv.hk/2018-02-18-docstring/ Static and runtime tools that wanted to get type hints had to guess the format and parse the docstring to get the types. Who still does that? If dataclasses, and other classes, move to using Annotated and __attrdoc__ for docstrings, the tools will follow. Runtime tools will look at the dunder, static tools will look at the annotation directly. **Which they will have to do anyway** if we add the convention that string literals following an assignment get recorded in __attrdoc__. Runtime tools will want to look in the dunder, not parse the source code. And static tools which aren't Sphinx will need to be adjusted to look at the string literals. So either way the tooling has to change. Sphinx is not the decider here. I dare say that they will want to continue supporting bare strings into the indefinite future, but new tools, and those with weaker backwards-compatibility constaints, will surely prefer to extract docstrings from __attrdoc__ than parsing the source code.
Of course it is. Just decorate your class with @no_type_hints.
All it takes is literally one stdlib class to start doing it officially, and people will pay attention.
I just think there's more to do here than add a decorator.
That depends on whether you mean, what do *we* need to do get the ball rolling, or what does the entire Python ecosystem need to do to transition to the day that every tool and library is actively using attribute docstrings both statically and at runtime? Of course the second part is a bigger job. But for the first, we can either (1): * bikeshed this for another six ~~months~~ weeks; * get at least one core dev to agree to sponsor a PEP; * write a PEP; * bikeshed some more; * wait for the Steering Council to hopefully accept the PEP; * modify the interpreter; * wait for the new version to be available; or we can (2) * implement and support the feature today; to get to the point that external tools can start using it. I'm not kidding, if somebody cared enough to do this, they could probably have a patch for dataclasses to support this within a couple of hours and a plugin for Sphinx within a day. (Spoken with the supreme confidence of somebody who knows that he will absolutely not have to do either.) As far as I am concerned, this is exactly the sort of use-case that Annotated was invented for. Docstrings are metadata. Annotated is for attaching metadata to an annotation that is associated with some variable. Done and done. -- Steve
On Sun, Dec 12, 2021, 1:57 AM Steven D'Aprano <steve@pearwood.info> wrote: On Sun, Dec 12, 2021 at 12:38:06AM -0500, Ricky Teachey wrote:
But Steve, since the most utilized documentation tool in the python universe, sphinx, doesn't look at Annotated
Yet.
or in the lowercase-annotation part of an expression
Yet. three
competing ways being used to provide attribute docstrings?
And before we had PEP 484 there were multiple competing and incompatible conventions for putting type hints in docstrings. https://www.python.org/dev/peps/pep-0484/ There are: - Epytext (Javadoc style) - ReST (Sphinx) - Googledoc - Numpydoc although some of them may no longer be widely used. https://stackoverflow.com/questions/3898572/what-are-the-most-common-python-... https://www.adrian.idv.hk/2018-02-18-docstring/ Static and runtime tools that wanted to get type hints had to guess the format and parse the docstring to get the types. Who still does that? If dataclasses, and other classes, move to using Annotated and __attrdoc__ for docstrings, the tools will follow. Runtime tools will look at the dunder, static tools will look at the annotation directly. **Which they will have to do anyway** if we add the convention that string literals following an assignment get recorded in __attrdoc__. Runtime tools will want to look in the dunder, not parse the source code. And static tools which aren't Sphinx will need to be adjusted to look at the string literals. So either way the tooling has to change. Sphinx is not the decider here. I dare say that they will want to continue supporting bare strings into the indefinite future, but new tools, and those with weaker backwards-compatibility constaints, will surely prefer to extract docstrings from __attrdoc__ than parsing the source code.
Of course it is. Just decorate your class with @no_type_hints. party
help/documentation tools to look in that location for these docstrings
All it takes is literally one stdlib class to start doing it officially, and people will pay attention.
I just think there's more to do here than add a decorator.
That depends on whether you mean, what do *we* need to do get the ball rolling, or what does the entire Python ecosystem need to do to transition to the day that every tool and library is actively using attribute docstrings both statically and at runtime? Of course the second part is a bigger job. But for the first, we can either (1): * bikeshed this for another six ~~months~~ weeks; * get at least one core dev to agree to sponsor a PEP; * write a PEP; * bikeshed some more; * wait for the Steering Council to hopefully accept the PEP; * modify the interpreter; * wait for the new version to be available; or we can (2) * implement and support the feature today; to get to the point that external tools can start using it. I'm not kidding, if somebody cared enough to do this, they could probably have a patch for dataclasses to support this within a couple of hours and a plugin for Sphinx within a day. (Spoken with the supreme confidence of somebody who knows that he will absolutely not have to do either.) As far as I am concerned, this is exactly the sort of use-case that Annotated was invented for. Docstrings are metadata. Annotated is for attaching metadata to an annotation that is associated with some variable. Done and done. -- Steve I agree. I think adding something like __attrdoc__ is a very good idea. Would like to see it in all classes, personally, and then have help() look there. But starting with dataclasses seems like a logical place and might get us started on that goal. And if we start this with dataclasses and it gets expanded, none of this would slam the door on the idea of using a bare string after the definition as sugar for populating __attr_doc__. In fact if there's any hope that would ever happen, first people would need to find themselves using Annotated far more often, and get tired of having to use the longish syntax with an import and agree on such a syntax change.
On Sat, Dec 11, 2021 at 10:55 PM Steven D'Aprano <steve@pearwood.info> wrote:
If dataclasses, and other classes, move to using Annotated and __attrdoc__ for docstrings,
OK -- THAT is the point I brought up earlier, and you responded with "this is settled" -- it's the __attrdoc__ part that needs to be standardized. As for Typing.Annotated: it has one advantage in that typing tools currently (presumably) already accommodate extracting the type information from it -- see what typing.get_type_hints() does.
the tools will follow. Runtime tools will look at the dunder, static tools will look at the annotation directly.
I hope not. *maybe* inspect.get_annotations() though.
but they do that anyway, for all other docstrings.
As far as using the lowercase-annotation for the docstring: in a world of
more and more type hinted python
yes, it's very clear that any standardized use of annotations needs to be compatible with type hints and type checkers, static and dynamic. Which i think is Steve's point -- typing.Annotated is indeed compatible with those uses.
and then your class attribute isn't type hinted. Of course you can do it, but it can't be the standard way to add documentation to class attributes — see above. All it takes is literally one stdlib class to start doing it officially,
and people will pay attention.
Sure -- but where do the docstrings extracted from a typing.Annoated object go? __attrdoc__ has been suggested -- but THAT requires some level of approval from the SC. As far as I am concerned, this is exactly the sort of use-case that
Annotated was invented for.
and yet the docs for typing.Annotated don't seem to say that. So what concerns me is that this may break other uses of Annotated, which are expecting types to be stored there. One problem with a docstrings is that they are "just a string". isinstance checks, as mentioned in the docs, are not so helpful. So we'd be establishing a new standard: “a bare string in an Annotated type is a docstring” -- adding a typing,Documented wouldn't be a much bigger lift ( I wonder if making it a subclass of Annotated would help). And I note that Annotated flattens nested Annotated types, so having both a docstring and other use of Annotated could be a bit tricky. I'm not a type hints user -- but I think this should be run by the MyPy and other folks to see if I’m right in my skepticism. Docstrings are metadata. Annotated is for
attaching metadata to an annotation that is associated with some variable. Done and done.
It’s not Done unless the community accepts it as standard practice. This is why we have all the Typing PEPs — annotations have been there a long time, but if various tools use them differently, they really aren’t that useful. As for using dataclasses as a way to introduce a convention: that could be a good route — but I doubt Eric would go for it[*] and in any case, it would in fact be introducing a new convention, so getting some consensus on what that convention should be would be helpful. -CHB [*] Eric, of course, can speak for himself, but so far, dataclasses deliberately don’t do anything with annotations other than pass them on, and there is active debate about whether they should do more, and if so, what?
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sun, 2021-12-12 at 12:23 -0800, Christopher Barker wrote:
Confirmed. I have tooling today using exactly that to dynamically generate API documentation.
But what's being annotated, the type or the attribute?
1. Docstrings today are attached to the things they're documenting by way of an included `__doc__` attrbute. `Annotated` is arguably "attached" to the type, to the extent that the type is embedded in the annotation itself. Illustration: SomeType = Annotated[str, "some type"] class A: x: SomeType y: SomeType Clearly, above we documented the type, not the attribute. We could solve this by allowing it to be re-annotated, which seems perfectly valid: SomeType = Annotated[str, "some type"] class A: x: Annotated[SomeType, "x docstring"] y: Annotated[SomeType, "y docstring"] The last bare string could "win" as the prevaling docstring:
Or, doc tools could even include them both to provide a more meaningful context. You could argue that since the type annotation of an attribute can be extended in this manner, `Annotated` is an acceptable way of documenting an attribute. 2. Adding a typing.Documented would definitely increase verbosity. I think we may already be pushing the envelope with `Annotated[...]`; `Annotated[..., Documented(...)] may be an annotation too far. Since there is not yet a convention for a bare string in `Annotated`, I suggest we reserve it the purpose of documenting the type or attribute.
And I note that Annotated flattens nested Annotated types, so having both a docstring and other use of Annotated could be a bit tricky.
I think as long as strings are considered documentation—regardless of where they appear in the annotation—we'd still be on solid ground.
I don't think dataclasses have to do anything with them. I'm literally embedding strings in Annotated today, for this exact purpose, with no impact on dataclasses. I think the work would be in documentation tools to interpret the annotations and produce something usable to the developer.
On Sun, Dec 12, 2021 at 12:48:36PM -0800, Paul Bryan wrote:
But what's being annotated, the type or the attribute?
That's easy to test:
Okay, so it's not the type. *wink*
A.__annotations__ {'attr': typing.Annotated[int, 'Doc string']}
It's the *class* that is annotated. But note that the mapping is between the attribute name to annotation, so in the sense that attributes are represented by their name, it is the attribute that is annotated. As it obviously has to be. After all, didn't we *literally* annotate the attribute? attr: annotation Don't be fooled by the use of the Annotated pseudo-type as the annotation. We're still annotating the attribute, just like the code shows, regardless of what type we annotate it as. If we had written: attr: Sequence[int] there would be no questions about what's being annotated, is it the type int or the attribute. It cannot be the type, because many different objects with many different annotations may share the same type. Unlike modules, classes and functions, the annotation cannot be on the attribute's *value*, because the attribute may not even have a value at this point. And even if it does, it might be immutable, or like the type, many different attributes, with different annotations, may be sharing the same value. This rules out putting the docstring on the attribute directly. The syntax shows us annotating the attribute, and that's exactly what got annotated. -- Steve
On Mon, 2021-12-13 at 09:57 +1100, Steven D'Aprano wrote:
OK, so it's not the type, except it kind of is. In my other example, I used `Annotated` to create what I guess is (and what you suggested) a pseudo-type? When it was annotated it had nothing to do with any attribute. To reiterate: SomeType = Annotated[str, "some type"] No attribute here (yet). Only when this is applied as a type hint to an attribute would it then apply. class A: attr: SomeType `Annotated` certainly appears to be intended to provide hints about the type. In PEP 593, `MaxLen` is an example of prescribing constraints on value. It would apply to an attribute if that attribute was annotated with it as a type hint. By the way, to see real code implementing this convention see: https://github.com/fondat/fondat-core/blob/main/fondat/validation.py#L41 Here's another place that defines annotations that can be used to enhance API documentation (e.g. OpenAPI): https://github.com/fondat/fondat-core/blob/main/fondat/annotation.py (Disclosure: this is code I wrote.) Being able to define a new type like this using Annotated has proven to be very powerful. You can apply constraints and
It's unfortunate that the name `Annotated` was selected, because it's definitely a point that can result in confusion. An attribute can have an annotation (i.e. has a type hint); however, is `Annotated` annotating the attribute? It's an object that contains multiple values, which can be applied to an attribute as an annotation (type hint).
Agree.
This rules out putting the docstring on the attribute directly.
Agree.
The syntax shows us annotating the attribute, and that's exactly what got annotated.
I'm still not in aligned on this point. I think I agree that if you annotate an attribute with `Annotated`, you're defining a pseudo type that can serve to document the attribute. However, as I've demonstrated, that can be inherited by defining that type in advance and using it to annotate multiple attributes. As I suggested, I think a reasonable approach could be to either take the last string in the annotation, or combine multiple strings if present.
On Sun, Dec 12, 2021 at 03:38:23PM -0800, Paul Bryan wrote:
OK, so it's not the type, except it kind of is.
Except it isn't annotating the type, it is annotating the attribute. We don't annotate *types*: int: Sequence[str] That would be a regular variable or attribute called "int" that just shadowed the builtin type int. We annotate *variables* (including function parameters and class attributes).
That's not an annotation. That's a type alias. Annotations follow a colon, not an equals sign.
Right. Now you have an annotation, and sure enough, we would find {'attr': typing.Annotated[str, "some type"]} in A.__annotations__. It is still annotating the attribute (to be precise, the attribute name).
And until some variable or attribute was actually annotated with it, it might as well not exist. Annotated allows us to create new types, by associating arbitrary metadata with an existing type. The meaning of that metadata is completely unspecified. So we can say: Vec = Annotated[List[Tuple[T, T]], MaxLen(10)] but that has no real meaning until you annotate a variable or attribute, in which case the Vec type is annotated onto the attribute. That use-case for Annotated is independent of the proposed use-case here. Sure you can use it to create new types, or type-aliases. But that's not the only thing we can use it for. The meaning of MaxLen(10) is unspecified. To a human reader, we can guess that it probably means that the list can have a length of no more than 10. Any tooling that comes across it is supposed to ignore it if it doesn't know how to interpret it. So for all we know, MaxLen(10) is not enforced by any tool, and it is purely there as documentation to the reader: "Please don't make your vector longer than ten items". We're not limited to only using Annotated to create new types. In an annotation, we can associate arbitrary metadata with the annotated attribute or variable. The interpretation of that metadata is still up to the consumer. Is it a type restriction? Is it a docstring? Is it something else? It's entirely up to the tooling. By the way, the proposal to just follow attributes with a string has already been considered and rejected: https://www.python.org/dev/peps/pep-0224/ -- Steve
On Mon, 2021-12-13 at 11:22 +1100, Steven D'Aprano wrote:
1. While I agree that assigning the `Annotated` value to `SomeType` makes `SomeType` a type alias, what do you call the specific instance of `Annotated[...]` itself? To date, I've been referring to it as a type, but that's also muddying the waters here. 2. I've been flyng fast and loose with the term "annotate". No help from PEP 593, which convolutes the situation with passages such as:
Annotated is parameterized with a type and an arbitrary list of Python values that represent the annotations.
To rephrase, if I add a string "some type" to an `Annotated` instance, it would not be documenting the (non-existent) attribute, it would arguably be documenting the type itself. Getting concrete: Coordinate = Annotated[int, "a graph coordinate", ValueRange(-100, 100)] ... @dataclass class Point: x: Annotated[Coordinate, "the horizontal coordinate"] y: Annotated[Coordinate, "the vertical coordinate"] I suggest that in these uses of strings in `Annotated`, the 1st case would serve to document the "coordinate" (pseudo-)type, while the 2nd and 3rd cases would serve to document the attributes themselves. In some cases, a developer may very well be satisfied with the documentation of the (pseudo-)type and not override it when annotating the attribute.
The use of `Annotated` is to decorate an existing type to provide additional metadata about it. The `Coordinate` type alias in the example above can be introspected at runtime, independently of its use in an annotation. If such a type alias were in a module, I would want its documentation string to be displayed to provide additional context about it, and help in determining if it's appropriate to use as an annotation of an variable, parameter, attribute.
So, I think we agree that an `Annotated` instance results in what's tantamount to a new type. And I hope by my examples above, we could agree that it would be reasonable to document such a type in such a way that its usage in an annotation could either inherit it, or override it.
I agree with the above.
On Sun, Dec 12, 2021 at 05:34:33PM -0800, Paul Bryan wrote:
Its a type. The specific type is a private implementation detail: >>> type(Annotated[int, 'something']) <class 'typing._AnnotatedAlias'> I just call it a type alias. If you want to be precise, it is an alias to int, augmented with some extra metadata.
To date, I've been referring to it as a type, but that's also muddying the waters here.
No no, it is absolutely a type! _AnnotatedAlias is an instance of type itself: >>> type(typing._AnnotatedAlias) <class 'type'> >>> isinstance(typing._AnnotatedAlias, type) True although Python plays some undocumented(?) shenanigans to make issubclass fail: >>> issubclass(Annotated[int, 'something'], type) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: issubclass() arg 1 must be a class
2. I've been flyng fast and loose with the term "annotate". No help from PEP 593, which convolutes the situation with passages such as:
I will have a lot more to say on that in another post, replying to Brendan. Later.
PEP 593 is absolutely clear on the fact that the semantics of the metadata are up to the consumer, and that Python does not require it to be a type or to be treated as a type. It describes the metadata as "an arbitrary list of Python values" so we're fine to shove docstrings in there. -- Steve
On 2021-12-12 16:22, Steven D'Aprano wrote:
You're still missing what people have mentioned a few times. It is true that this annotates an attribute: class Foo: x: int The question is what does this annotate: Annotated[int, "some text here"] In other words what does the use of Annotated in itself annotate. As Paul pointed out in his earlier message, you can create this Annotated thing without attaching it to any attribute. It seems that Annotated itself is annotated the type. In fact, there can be no debate about what Annotated annotates, since the documentation even says it explicitly (https://docs.python.org/3/library/typing.html#typing.Annotated):
Specifically, a type T can be annotated with metadata x via the typehint Annotated[T, x].
So yes, Annotated IS annotating the type. The text specified in `Annotated[T, text]` is creating a sort of augmented type, which is like "a type that is type T but additionally means such-and-such". It is true that that can type can itself later be used to annotate an attribute (or variable), but at that point it will be marking the variable as being of a type that incorporates the annotating text, not annotating the variable itself with that text. As such, I don't think Annotated is really a good choice for this. We really want the text of the annotation to be associated with the attribute itself, not with the attribute's type. Moreover, using Annotated requires the user to specify a type, but people should be able to specify annotations for documentation even if they're not using typing at all and not specifying any types. (They shouldn't have to resort to an awkward workaround like always throwing in a dummy Any type.) As an aside, seeing what the docs say about Annotated makes me think that "Annotated" is a very bad name for this thing. It confuses the idea of a type annotation (i.e., attached to a variable) with this type-incorporating-a-label, where neither the label nor the type is actually an annotation in the type-annotation sense (because they have not been attached to a variable to annotate it). It seems it would have been better to called Annotated "Tagged" or "Labeled" or some such thing to make it clear that when you using it you are defining a new special-purpose type for use in later annotations./ -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Sun, Dec 12, 2021 at 07:43:16PM -0800, Brendan Barnwell wrote:
The question is what does this annotate:
Annotated[int, "some text here"]
Nothing. There's no annotation there. That's an expression which returns a type alias. It is just an object. You can print it, put it in a list, pass it to a function, use it as a dict key, or garbage collect it. Or even use it to annotate a variable, but isolated as show, it isn't annotating anything. An annotation is "A label associated with a variable, a class attribute or a function parameter or return value" and that expression above is not a label associated with anything. https://docs.python.org/3/glossary.html#term-annotation To be an annotation, it must be part of a parameter annotation, or variable assignment annotation: def func(param:expression) -> expression: variable: expression All three of those expressions could legitimately be called an annotation, even if they're not types. Here's an analogy: you know how dicts work, right? (Of course you do.) We agree that in the dict {"Hello world": 42} the string "Hello world" is the key associated with value 42. So in the context of that dict, "Hello world" is a key. # helloworld.py print("Hello world") Is that string "Hello world" a key in a dict now? What's its value? How about in isolation, "Hello world", with no implied context and most especially no dicts anywhere in sight. Is it still a key if there is no dict for it to be a key of? I'm not asking whether it *could* be a key, if there was a dict. Obviously the answer is yes. I'm asking, what dict is "Hello world" the key of in the helloworld.py program above. If we can understand that being a key describes the *role* of a string, in a specific context, and is not a property of the string itself, we should be able to understand that being an annotation is likewise descibing the *role* of an object, in a specific context, and is not a property of the object itself. types: "Hello world" = (int, float, Annotated[int, "..."], str) Here, "Hello world" is an annotation, and the Annotated type alias is a element of a tuple. Earlier the same string was the output of a program, and before that it was a key. But it's the same string. Annotated[int, "some text here"] can likewise be an annotation, or the output of a program, or a key in a dict, or an element of a tuple, depending on context. With no context, it is just a type object. If its not part of an annotation statement or a function annotation, it's not an annotation. https://docs.python.org/3/glossary.html#term-function-annotation https://docs.python.org/3/reference/simple_stmts.html#annassign
In other words what does the use of Annotated in itself annotate.
In itself, nothing. There's no annotation syntax in that expression. It's not a function annotation or a variable annotation. It doesn't create an `__annotations__` dunder. It meets none of the definitions of "annotation".
You are right about the first part of your sentence: there can be no debate about this. The doc is simply wrong, and it has confused you and others into thinking that Annotated[T, ...] has annotated T. But if anyone wants to debate it, if you think the documentation is correct, then it is easy to prove me wrong, and I will happily admit that I have learned something new. All you need to do is show me the annotation syntax `target: expression` in either of the following: # A bare expression, with no context. Annotated[int, "some text"] # A type alias. MyAlias = Annotated[int, "some text"] Easy peasy. Point to the annotation syntax colon to prove that I am wrong, and the docs are right. Or you can show me the `__annotated__` mapping with int as the key that either of those two lines create. But if you can't do any of those things, then: - there is no annotation syntax; - there is no "label associated with a variable, a class attribute or a function parameter or return value"; - there is no `__annotations__` dunder to hold that association; and therefore **there is no annotation**. It's just an expression returning a type.
The text specified in `Annotated[T, text]` is creating a sort of augmented type
The docs and the implementation call them Type Aliases. "Augmented" is an excellent description, thank you for suggesting it. So we can agree that Annotated[T, *metadata] returns a type alias, based on T, but augmented with the metadata. That is an excellent way to describe it, that avoids the confusion with annotations. Yes, the primary use-case of such a type alias is to use it in annotations, but its just an object, you can use it for many things.
What you call "marking the variable" is what Python calls an annotation. And the meaning of annotations is **not specified by the language**. Typing is the primary use-case but it is not the only one. Any alternative use of annotations has to work with a widespread assumption that annotations will be types, but that is doable.
Which you can do with Any. That is hardly a major imposition for code that is already using typing, and for code that isn't using typing, you don't have to worry about it. Just use a string.
And they can do that to, just by *not using a type checker*. @no_type_checking class C: attr: "Doc string" Done. The doc string is recorded in the __annotations__ dunder, but you can post process it any way you like. If you want a *standard* mechanism that will be blessed by the language, then it has to interoperate with typing. We've already had the idea of following the variable with a bare string rejected: https://www.python.org/dev/peps/pep-0224/ and I see no reason to think that the situation has changed in any way since then. But if you want to propose re-opening PEP 224 and having another go at it, be my guest.
Right!
Indeed. -- Steve
On 2021-12-13 17:07, Steven D'Aprano wrote:
Then let's revisit this issue after we have fixed this problem by renaming typing.Annotated to something else (or creating a new name and deprecating the old name but keeping it for backward compatibility). -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Sun, Dec 12, 2021 at 3:00 PM Steven D'Aprano <steve@pearwood.info> wrote:
I know what a wink means, but I have no idea what your actual point is. It’s very hard not to get tripped up by the overloading of the word “annotation”, but typing.Annotated can exist outside of __annotations__ it is not an annotation itself, it has special kind of type object (type descriptor? What the heck do we call things that are used for type hints but are not regular Python instantiable types?) it’s a container for a type and other extra information that pertains to that type, not necessarily to the parameter or whatever it gets attached to. As far as I know, no one in this conversation has used typing.Annotated in a type checking system— we really should hear from those folks.
The attribute got annotated the typing.Annotated object, but what did the string ‘Doc string’ get attached to? As long as we are winking: class A: attr = “a string” A.attr.upper() exists, so obviously the class attribute has that method. Of course not. The attribute has a value that is a string, and that has the upper attribute. Similarly, in your example, attr has an annotation, and the value of that annotation has a string attached to it. I’m not saying that a convention couldn’t be established to store docstrings in annotations via the typing.Annotated type. But I’m not sure that it wouldn’t get in the way of other uses, and it’s really not that obvious or convenient either. -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sun, Dec 12, 2021 at 03:46:16PM -0800, Christopher Barker wrote:
I know what a wink means, but I have no idea what your actual point is.
My actual point is that the annotation: attr: int is annotating the attribute named "attr". That's pretty basic stuff: if you don't think that we've annotated the attribute attr, then what do you think we've done? Changing int to Annotated[int, metadata]: attr: Annotated[int, metadata] is still annotating attr, not T. What else could it be? The syntax for an annotation is (roughly): name colon object and that's exactly what we have. It's an annotation, and the thing being annotated is attr. I know the docs say: "a type T can be annotated with metadata x via the typehint Annotated[T, x]" but that is wrong and misleading. Better wording would be "a type T can be *associated* with metadata x via the typehint Annotated[T, x]" but that leaves the question, why is this feature called "Annotated"? To me the answer is that it clearly refers to the fact that using this feature, we can annotate attr with *both* T and some additional metadata x, without any restriction on the nature of x. A wrinkle in this is that we can also create new types, using type aliasing: PositiveIntegers = Annotated[int, MinValue(1)] (which, by the way, was one of the motivating use-cases for this feature, so it is important in the big picture of typing, not so important for this use-case though) and in that sense we might say that the new type, PositiveIntegers, is kinda-sorta made from annotating the original type, int, with the metadata. Except... there is no actual annotation there. Its an equals sign, not a colon, so it's a name binding, which the typing system interprets as a type alias. If you look at int, it has no annotations attached, so clearly we haven't annotated int. Which brings us back to my claim that the description is misleading and it should talk about *associating* the type T with the metadata, not annotating it. Semantics are important. Would we be having this argument now if the docs didn't make the (wrong!) claim that it is the type that is being annotated? I don't think so. It is attr being annotated, not the type. And the interpretation of the metadata is entirely up to the consumer of the annotation, which means we are completely entitled to use it for docstrings.
I don't think that there is any general term for types which are only intended for type hints and cannot be instantiated. If such a standard terminology exists, I don't know it. I agree that there should be.
Indirectly to the attribute's name "attr". The implementation uses a special type alias for that purpose. Using a tuple such as (int, 'Doc string') was considered by the PEP but rejected. Nothing in the PEP or the existing docs for Annotated precludes us from using the metadata as a docstring. The interpretation of the metadata has always been up to the consumer of the metadata. -- Steve
I'll go ahead and throw this out there: I was talking to a friend about this and he pointed out something simple but astute. Will this be expected to work? class C: x: Annotated [Any, "spam"] help(C.x) Obviously it shouldn't as things stand now because that's an AttributeError. But if I'm mucking about in the interpreter and I want to see the docstring of a runtime defined (but not import time) attribute, it would be kind of nice if it would work. And it seems reasonable to try and create a way for it to work.
On Sun, Dec 12, 2021 at 08:44:25PM -0500, Ricky Teachey wrote:
You can fix that by giving x a default value.
By default, for arbitrary classes, no, it *shouldn't* work, because for arbitrary classes we cannot treat type hints as doc strings. (Actually, to be clear, I say "we" but I mean the language and the stdlib, including help(). "We" as individuals can do anything we like.) At least not without buy in from the core devs, including people like Guido who care very much about typing. And that will probably need a PEP. That's the beauty of a decorator. We don't need a PEP or Steering Council approval to write a decorator (although we might need one to make it a built-in). Just do it! But to change the language means we have to go through a big cumbersome process that may take months or years. A *lightweight* change that doesn't actually change the language could be accepted without a PEP. If you submitted a PR to update help() so that it was smart enough to recognise strings as the second parameter of Annotated annotations and treat them as docstrings, I suspect that would be allowed as just a relatively small new feature. -- Steve
Steven D'Aprano writes:
On Sun, Dec 12, 2021 at 08:44:25PM -0500, Ricky Teachey wrote:
Why not? In the sense that we establish a standard protocol for attaching them, and if there isn't one there, then 'help' would ignore that attribute or provide an empty docstring or whatever default action seems appropriate (presumably it would at least tell us the type! even if the hint is not an Annotated).
It's true that changing help will require buy-in from the core devs, as it's a built-in. But:
But to change the language
Where's the language change? What am I missing? Or are you using this loosely to include builtins or even the whole stdlib? It needs one thing: a standard place to put the documentation. I don't think just stuffing a string in __metadata__ is a good idea; that will be an annoyance to existing users of the __metadata__ attribute. I think it probably should be an attribute on Annotated instances, to avoid tromping on existing uses of the __metadata__ attribute, while allowing existing users of Annotated to take advantage of the documentation function in a gradual way. While I agree this is going to need input from core devs that would likely delay things by months rather than weeks, I see no reason why it wouldn't be in 3.12. In fact, it seems likely that this change would be small and straightforward enough that it could be prototyped, polished, and wrapped in time for 3.11. I don't think it would be that hard to get buy-in to a plausible proposal, getting Guido's attention is probably harder. ;-) (If he kibozes this post I'll cry and then laugh.)
I don't think that will fly, as it would be rude to people who are already using Annotated for something else (especially if they've already transitioned from direct use of annotations!) I don't think Python the language or the stdlib should touch the __metadata__ attribute, it should be reserved to client code.
On Tue, Dec 14, 2021 at 12:38:55AM +0900, Stephen J. Turnbull wrote:
Anything is possible! And if this is for your own code you can do anything you like. But if you want this to be a language feature, that means getting Steering Council approval, and while I don't speak for them, I am confident, well, 90% confident, that they will agree with Guido that *annotations are primarily for type-hinting*. I am also confident (say, 75% confident) that they will *disagree* that annotations are **only** for type-hinting, so there is that. But of course if anyone thinks different, and believes that the SC will bless "string annotations are docstrings" as a language feature, write a PEP and go for it!
Yes, it is somewhat loose, meaning the language proper, the builtins and stdlib. Right now, we have: class C: attr: expression causes the annotation to be stored in the `__annotations__` dunder. If all you want is to write your own metaclass or decorator to read the __annotations__ and extract strings and stick them into C.mydocstrings, you can do that right now, no questions asked. But if you want to use a dunder like __attrdocs__, say, then dunders are reserved for use for the language and we're supposed to ask first before inventing new ones. So there's that. And if you want this to new behaviour -- extracting strings from annotations to use as docstrings -- to be in the language (including the stdlib and builtins) then again we need approval. At the very least we need at least one core dev to review, accept and merge the PR. And I expect that the core devs would push it back to the typing group and SC for a decision. (That's what I would do if I knew how to merge PRs :-)
There is no guarantee or language-wide meaning to the metadata, every consumer of Annotated type aliases *must* be prepared to skip metadata they don't know how to process. It's like consumers of email. The email headers can contain arbitrary metadata that you have no clue how to handle, you are required to leave it and skip over it. But the beauty of Annotated is that it already exists and we can start using it today. If anyone wishes to push for a syntax change, or a new convenience type in the typing module: attr: Docstring["this is a doc string"] then you can do so, but you will still have to get the change approved. And what are type checkers supposed to do with it? Now every type checker has to learn to try Docstring as an alias for Any. What if you want to declare a type as well?
I think it probably should be an attribute on Annotated instances, to avoid tromping on existing uses of the __metadata__ attribute,
Is there a syntax change? If there is, then you definitely need SC approval for syntax changed. If not, how does the Annotated type know which piece of metadata gets moved into the attribute? Annotated[Any, fe, fi, fo, fum, "docstring", eeny, meany, miny, mo] *We* can recognise that clearly item number 5 is the docstring, but how does Annotated know? We need to have a set of rules in mind. -- Steve
Just a short one, for everyone agreeing type.Annotated does the job, but thinks we need new syntax, because it is verbose: You can already do: from typing import Annotated as A And: attr: A[type, "docstring goes here"] I see no need for any new syntax. (and maybe adding typing.Docstring for the cases when one just wants the docs, but no type annotation) On Tue, 14 Dec 2021 at 06:19, Steven D'Aprano <steve@pearwood.info> wrote:
On Tue, Dec 14, 2021 at 10:23 AM Joao S. O. Bueno <jsbueno@python.org.br> wrote:
You're not wrong exactly. But there has been a desire to reduce the need for typing module imports in general, and I think eliminating the need to import Annotation for this specific use case might have merit, if is becomes blessed as The Way, and if we think it is a good idea to say to people "hey! providing docstrings for members is a really good practice, and so we're making is super duper easy and standardizing it!" (just as we encouraged people to provide docstrings in the past for modules and classes by making that syntax dead simple to use). --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
Hey all: There is discussion RIGHT NOW in the SC, and on python-dev about future "policy" around annotations, in response to the PEP 563 and 649 -- not clear where it's going to end up, but it is clear that the whole "are annotations only for typing" question will be made more clear. Anyway, I had the idea that one could create a subclass of typing.Annotated, and call it "Documented", and then have it do something a little special, while still looking like an Annotated to the Typing system, e.g. get_type_hints(). However: In [2]: class Documented(typing.Annotated): ...: pass ...: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-2-fb09faa96292> in <module> ----> 1 class Documented(typing.Annotated): 2 pass 3 ~/miniconda3/envs/py310/lib/python3.10/typing.py in __init_subclass__(cls, *args, **kwargs) 1676 1677 def __init_subclass__(cls, *args, **kwargs): -> 1678 raise TypeError( 1679 "Cannot subclass {}.Annotated".format(cls.__module__) 1680 ) TypeError: Cannot subclass __main__.Annotated So those type objects in typing are weird. I'm giving up :-( -CHB On Tue, Dec 14, 2021 at 8:36 AM Ricky Teachey <ricky@teachey.org> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Christopher Barker writes:
but it is clear that the whole "are annotations only for typing" question will be made more clear.
Can we please stop posting this? AFAICS, the basic principle is absolutely clear. For the foreseeable future: 1. Annotations are NOT "only for typing". 2. Uses of annotations that get in the way of typing will be NOT adopted or enabled in the Python language, builtins, or stdlib. 3. typing.Annotated means that there is no conflict between 1 and 2, although there may be inconvenience, ugliness, or lower performance implied for some non-typing applications. Guido and the typing PEP authors have stated 1 (repeatedly!) and 2 (more recently) below. However, the definition of "typing" may get clarified, if that's what you mean (runtime vs. static, are dataclasses a typing use case, etc).
I imagine this can be done by a Sufficiently Persistent Pythonista, but I suppose a factory function would be easier. I have PoC (too embarrassing to post) given the existence of an open slot in Annotated instances (see below). The main problems (as pointed out by multiple Steves) are - Any *syntax* (language change) for creating docstrings MUST be approved by the SC, and given the rejection of PEP 224 (exactly the original proposal for "docstrings below attributes and variables") and the bikeshedding that already has taken place in this iteration, this could take years. - Any change to help(), which is a builtin, will require SC approval, and great care will have to be taken with the design i.e. #2 above for sure, and #3 should be minimized. By #3 be minimized I have in mind that there are probably existing clients that moved to annotated and expect their strings as the second argument to Annotated[]. It would be rude to make them move again. So that kind of convention for the use of Annotated.__metadata__ should be avoided IMO. Although Annotated can't be subclassed, a minimal, noninvasive, change to Annotated that makes both #2 and #3 possible would be adding a new slot __attr_doc__ into which to stuff the docstring. That could be done with a class of a different name for experimentation. This would give people working out how to use this info in help() a single protocol for getting the docstring (even if in the end there's a different protocol used in typing.Annotated, it should be amenable to abstraction, making adaptation to a new protocol simple). Another possibility would be a specific new type DocstringType that could be placed anywhere in __metadata__ and the first instance would be considered the docstring for the identifier so Annotated. I really do think that if we can keep a lid on the bikeshedding this could be in 3.11. Note that the DocstringType approach doesn't get around the SC because we still want to change help() to recognize these docstrings.
On Sun, Dec 12, 2021 at 12:23:54PM -0800, Christopher Barker wrote:
Sorry, I don't understand your position here. Putting aside *how* we write attribute docstrings, isn't the point of them that they are available to runtime inspection? Say, by help(). So why do you hope that runtime tools don't inspect the dunder storing the attribute docstrings? And that static tools *don't* look at the annotation? (Or whatever mechanism we use.) They can't use the dunder because it doesn't exist yet, you haven't run the code. So I'm confused... we go to the trouble of deciding on a mechanism to write attribute docstrings, we write them in the code, and you hope that nobody uses them??? Have I misunderstood?
Okay, I'm sorry, I understand the statement to be referring to people who had no intention of using type-hints. For those people, no_type_hints is the right solution. But for those who want type hints as well as docstrings, it is not. Sorry for the confusion.
I don't think that needs a PEP or SC approval, I think that's a small enough change that any core developer who cares can just add that dunder to their class and start collecting docstrings in there. But at some point somebody will need to ask on the Python-Dev list and see if anyone objects and says yes, you need a PEP.
Have you read the PEP? Particularly this section: https://www.python.org/dev/peps/pep-0593/#consuming-annotations Quote: Ultimately, the responsibility of how to interpret the annotations (if at all) is the responsibility of the tool or library encountering the Annotated type. A tool or library encountering an Annotated type can scan through the annotations to determine if they are of interest (e.g., using isinstance()). Unknown annotations: When a tool or a library does not support annotations or encounters an unknown annotation it should just ignore it and treat annotated type as the underlying type. Similarly in the docs: https://docs.python.org/3/library/typing.html#typing.Annotated Until such time that the Steering Council announce that type-hints are no longer merely the preferred use for annotations, but the *only* use for annotations, and any other usage is banned, using annotations to capture variable docstrings is perfectly legitimate. The current SC has made it quite clear that they are happy with the status quo, that typing is the primary but not only use for annotations. Using docstrings in annotations would seem to be a good way to establish a strong non-typing use-case.
No, we can't be that gung-ho to say that bare strings have to be docstrings. Since typing is the primary use-case for annotations, including the metadata in Annotated, it would have to be opt-in. Any type-hint can be a string: either due to PEP 563, due to forward references, or just because the developer on a whim choses to stringify their type-hints. Opting-in would mean documenting your class' requirements and assumptions about the format of annotations, and then using something like a decorator or metaclass to extract the docstrings.
And I note that Annotated flattens nested Annotated types, so having both a docstring and other use of Annotated could be a bit tricky.
class MyClass: """Blah blah blah. Variable annotations of the form Annotated[T, 'string', *args] always interpret the string in the second position as a docstring. If you want to use Annotated but without a docstring, put None in the second position. """ spam: Annotated[int, 'Yummy meat-like product'] eggs: Annotated[float, 'Goes well with spam', Domain[0, 1]] cheese: Sequence[str] # No docstring. The metadata is *always* interpreted by the tool, this is no different.
I'm not a type hints user -- but I think this should be run by the MyPy and other folks
Naturally. I could be wrong. In theory. *wink*
I only mention dataclasses because that seemed to be the primary use-case mentioned from the start of this thread. I don't speak for Eric, and I haven't verified for myself that dataclasses would benefit from attribute docstrings. -- Steve
On Mon, Dec 13, 2021 at 7:14 AM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
What about having a docstring but no typing information? In this case that's impossible, no?
spam: Annotated[Any, 'Yummy meat-like product'] or spam: Annotated[None, 'Yummy meat-like product'] -CHB
On Mon, Dec 13, 2021 at 6:54 AM Steven D'Aprano <steve@pearwood.info> wrote:
The consideration I'm making here is that any standard use of annotations needs to be fully compatible with type hinting. That's even spelled out in PEP 563. "uses for annotations incompatible with the aforementioned PEPs should be considered deprecated."
OK -- but that brings us back to how I started this sub-thread -- where are the docstrings stored to be retrieved at runtime. If that's a new dunder, great, then future enhancements making it easier to add them would be compatible, and folks could hand-populate the dunder in any number of ways, maybe including the use of typing.Annotated. Quote:
yes, i read that. This trick is that that's just kicking the can down the road. - one level up from the original state of "annotations will be stored, and you can use them however you want". But then it was decided to standardize the use of annotations as type hints. ANd now we are changing how annotations work in order to better accommodate type hinting (PEP 563). In light of the lesson there, I don't think it's a good idea to semi-standardize one use for typing.Annotations without being more explicit in the intent. Until such time that the Steering Council announce that type-hints are
but not a good idea at this point -- at least not in a way that's incompatible with type hinting.
The current SC has made it quite clear that they are happy with the status quo, that typing is the primary but not only use for annotations.
Actually, that's not the least bit clear right now -- that exact point is being mulled over as we speak. Using docstrings in annotations would seem to be a good way to establish
a strong non-typing use-case.
Then someone needs to do it NOW. The SC has literally asked for use-cases for annotations for them to consider.
in an Annotated type is a docstring” -- adding a typing,Documented wouldn't
I think it might be doable -- I'm going to give that a try soon. No, we can't be that gung-ho to say that bare strings have to be
docstrings. Since typing is the primary use-case for annotations, including the metadata in Annotated, it would have to be opt-in.
exactly -- so someone develops a tool that looks for Annotated, and tries to extract docstrings from it. And someone else uses an Annotated with a string in it for some completely different reason. And then a third person runs the docstring tool on the code using Annotated for that other reason. That just seems like asking for a mess to me. Any type-hint can be a string: either due to PEP 563, due to forward
references, or just because the developer on a whim choses to stringify their type-hints.
yes, though that is getting cleaned up, and is already a bit cleaner if you use get_type_hints or get_annotations -- that's one reason they exist.
and dataclasses are the one use-case of annotations in the stdlib. (I'm pretty sure). However, on thinking about it, this is actually NOT a good use case. The dataclasse decorator uses analysis of the decorated class to set its Fields: If the class has a class attribute that has an annotation, it is a Field. However, those class attributes are NOT necessarily class attributes in the resulting class. I haven't checked the code carefully, but it looks like if they have an immutable default, they are kept as class attributes, and otherwise are not. In [48]: @dataclass ...: class Test: ...: x: int ...: y: float = 1.0 ...: l: list = field(default_factory=list) ...: In [49]: Test.x --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-49-e65584dd4a8c> in <module> ----> 1 Test.x AttributeError: type object 'Test' has no attribute 'x' In [50]: Test.y Out[50]: 1.0 In [51]: Test.l --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-51-2185bdd4fcc8> in <module> ----> 1 Test.l AttributeError: type object 'Test' has no attribute 'l' I don't know why the immutable default is kept as a class attribute -- it gets overridden by an instance attribute when initialized anyway. But the key point is that the class attributes used to make a dataclass are not class attributes of teh dataclass, they are turned in to instance attributes. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Thinking more about my example here: On Tue, Dec 14, 2021 at 01:50:45AM +1100, Steven D'Aprano wrote:
perhaps a better convention might be that the docstring is the *last* item in the Annotated metadata, rather than the second item (after the initial type). The will allow people to use arbitrarily long augmented types, and follow it with the docstring: spam: Annotated[int, Positive, GreaterThan[1], Union[Odd|Literal[2]], Prime, Wibble, Wobble, "Doc string goes here..." ]=2, That will also help ensure that the docstring stands out visually, as it will be last in the list, not stuck in the middle. -- Steve
If we proceed with using `Annotated`, I suggest it be the last string in the metadata. Using your example: spam: Annotated[ int, Positive, GreaterThan[1], "Doc string goes here..." Union[Odd|Literal[2]], Prime, Wibble, Wobble, ] = 2 In other words, strings would be reserved to specify documentation. On Tue, 2021-12-14 at 11:38 +1100, Steven D'Aprano wrote:
On Mon, Dec 13, 2021 at 05:10:52PM -0800, Paul Bryan wrote:
In other words, strings would be reserved to specify documentation.
We can't reserve strings to specify documentation. (1) Strings can be used for forward references. class Node: payload: Any # Arbitrary data. next: Node # Fails, because Node doesn't exist yet! So we use a string for forward references: class Node: payload: Any # Arbitrary data. next: "Node" Technically, we could ask folks to write that as next: ForwardRef["Node"] but I think most people will say "Not a chance". Besides, if PEP 563 becomes standard, then all annotations will be strings. (2) Annotated[...] metadata can be arbitrary objects, so we have to assume that somebody out there is already using strings as metadata. This is why I think that the best we can do is an opt-in system where individual classes can choose to use string annotations as docs, but not a language default. But you never know what the Steering Council will say unless you ask. -- Steve
Okay, here we go: ``` import typing def extract_docstrings(K): """Extract docstrings from the class K's annotations. Class attribute docstrings are extracted from the second item in Annotated[...] attributes. The original annotation is left unchanged. FIXME: Handling of PEP 563 string annotations may not be best practice. https://www.python.org/dev/peps/pep-0563/ """ d = {} for key, value in K.__annotations__.items(): if type(value) is str: if value.startswith('Annotated[') and value.endswith(']'): s = value[10: -1] items = s.split(',') if len(items) >= 2: doc = items[1].strip() d[key] = doc elif typing.get_origin(value) is typing.Annotated: d[key] = value.__metadata__[0] K.__attrdocs__ = d return K ``` And here's an example:
-- Steve
Christopher Barker writes:
-10. They need to be accessible via the attribute name. You could insert the name before tacking on, but surely whatever format you choose, there are projects where it would be the wrong style.
Another dict?
__attr_doc__
+0.4 for now.
Added to __annotaions__ ?
+0.6 for now. That's where mypy puts type annotations, I suspect you would have to use values of Annotated type, which might be annoying.
On Sat, Dec 11, 2021, 12:57 AM Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote: Simão Afonso writes:
To my mind Sphinx is sufficiently widely used that this settles the "above or below" question. Steve I agree. Even with this convention one could still accomplish separating the documentation section from the definition section, like: class Steel: """A simple material model of steel.""" # parameters E_s: float "the linear elastic modulus" nu: float "the poisson's ratio" gamma: float "the density" # values E_s = 29e6 # ksi nu = 0.3 gamma = 490 # lbs per cu ft As has already been observed, we already have something LIKE an attribute docstring, which is an Annotated type hint object. Should the bare docstring below the member become sugar for creating an Annotated type hint? x = 1 "spam" Would then be equivalent to: x: typing.Annotated[typing.Any, "spam"] This seems really nice to me.
On Thu, 2021-12-09 at 12:32 +1100, Steven D'Aprano wrote:
Indeed.
Should all classes do the same? How about top level module variables?
If we started with dataclass, it could be performed in the dataclass decorator and make_dataclass functions. I think this would be low- hanging fruit compared to classes and modules.
On Thu, Dec 9, 2021 at 12:12 AM Paul Bryan <pbryan@anode.ca> wrote:
Again: better than nothing. But I'd really like to see this for all classes. I suggest that if a proposal like this is accepted, the help() output for classes and modules should have a new section added containing the member docstring information; it essentially would insert the member information to the end of the main docstring WHEN DISPLAYED (not suggested attaching it permanently as part of the docstring). This new section would appear directly beneath the module/class docstring when using help. Additionally, other help()-ilke tools (such as the ipython ? operator) should be encouraged to concatenate the member docstring information directly below the class/module docstring when printing out help-like information. To illustrate how things currently stand, here's a current, sort of minimally complicated class definition (class B) with some annotations: In [1]: from typing import Annotated In [2]: class B: ...: """Docstring for the B class.""" ...: ...: a: Annotated[int, "here is what i'd call a medium length docstring"] ...: b: Annotated[str, "short docstring"] ...: ....and here is the help(B) output that i get (Windows command line): In [3]: help(B) Help on class B in module __main__: class B(builtins.object) | Docstring for the B class. | | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | __annotations__ = {'a': typing.Annotated[int, "here is what i'd call a... The member annotations aren't very discoverable this way... you can't even see all of the first one. If it is agreed that Annotated should be The Blessed Way to annotate members, this needs to be improved. Here is the same thing for a dataclass (class C); here you CAN at least see the whole of each docstring, but they are on one line together and it's not all that great, and bits of the Annotated objects appear in this help output at least 5 times by my count (maybe this can't be helped since it's part of the Annotated object): In [4]: from dataclasses import dataclass In [5]: @dataclass ...: class C: ...: """Docstring for the C class.""" ...: ...: a: Annotated[int, "here is what i'd call a medium length docstring"] ...: b: Annotated[str, "short docstring"] ...: In [6]: help(C) Help on class C in module __main__: class C(builtins.object) | C(a: typing.Annotated[int, "here is what i'd call a medium length docstring"], b: typing.Annotated[str, 'short docstring']) -> None | | Docstring for the C class. | | Methods defined here: | | __eq__(self, other) | | __init__(self, a: typing.Annotated[int, "here is what i'd call a medium length docstring"], b: typing.Annotated[str, 'short docstring']) -> None | | __repr__(self) | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | __annotations__ = {'a': typing.Annotated[int, "here is what i'd call a... | | __dataclass_fields__ = {'a': Field(name='a',type=typing.Annotated[int,... | | __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or... | | __hash__ = None Here is the ipython ? output for the dataclass, by the way-- it is FAR better! But this is only so nice because dataclasses adds the typing information to the __init__ signature. For regular classes (like class B above), we would not get this nice output when using ?. In [7]: C? Init signature: C( a: typing.Annotated[int, "here is what i'd call a medium length docstring"], b: typing.Annotated[str, 'short docstring'], ) -> None Docstring: Docstring for the C class. Type: type Subclasses: --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On 9/12/21 12:50 am, tmkehrenberg@gmail.com wrote:
The main criticism, I think, was that it is weird to have the docstring *below* the attribute.
I don't think that's a problem. The docstring of a class or function is also below the name of the class or function. I think it's actually more consistent that way. It seems excessive to invent a whole new type of string just for this purpose. The neat thing about the docstring convention is that it doesn't require any special syntax. It would also raise some other questions, such as what happens if you use a d-string in some other context? Does it just behave as an ordinary string? Is it disallowed?
Does this mean you could also use it on module-level functions? d"This function is really important." def f(): ... That would give you two different ways to document a function, and it seems you could use both of them at the same time, which could be confusing. If you're looking for documentation of a function, there are two places you would have to look -- in the function itself, and in the module where it's defined. Similarly for classes. -- Greg
On Wed, Dec 08, 2021 at 11:50:55AM -0000, tmkehrenberg@gmail.com wrote:
Perhaps I missed it, but I haven't seen anyone point out that this proposal has already been suggested and rejected more than twenty years ago: https://www.python.org/dev/peps/pep-0224/ -- Steve
Hmmm, well it seems that we already have support for class attribute docstrings, since Python 3.8. It's not documented, but soon will be. https://bugs.python.org/issue46076 class Card: """A card from a standard French deck""" __slots__ = { 'suit': 'Either "Spades", "Hearts", "Clubs" or "Diamonds"', 'rank': 'A positive integer in the range 2 <= x <= 14' } help() already knows to read the docstrings from the slot dict values. -- Steve
On Tue, Dec 14, 2021, 9:49 PM Paul Bryan <pbryan@anode.ca> wrote:
Also doesn't apply to module level members. To my mind these are significant downsides. And it also represents yet another way people are documenting attributes in the wild. Here's another I learned if which is supported by sphinx autodoc: comment tag colon docstring attribute statement Example: #: spam x = 1 Using this style, the documentation has to come before the attribute. Obviously- unless VERY big change is introduced- this "docstring" will never be runtime accessible since it's a comment. But it does represent yet another way people are doing documentation in the wild. Sigh. Rick.
On Tue, Dec 14, 2021 at 06:48:57PM -0800, Paul Bryan wrote:
Interesting. Some apparent downsides:
- doesn't apply to class attributes
There is that.
- objects with `__slots__` can't dynamically add attributes
Just give it a `__dict__` slot: __slots__ = {'__dict__': None} and now it can. -- Steve
I think it's an interesting idea. I made the same or at least similar suggestion in the previous thread, but it didn't receive any responses. I assume this is because people weren't very interested (but I also understand people are busy). Here's that message: https://mail.python.org/archives/list/python-ideas@python.org/message/SJEPCZ... --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler On Wed, Dec 8, 2021 at 6:56 AM <tmkehrenberg@gmail.com> wrote:
On Wed, Dec 08, 2021 at 11:50:55AM -0000, tmkehrenberg@gmail.com wrote:
Why would it be weird to put a docstring below the object that owns it? We already do that for class and function signatures. def func(): """Docstring is below the signature""" I would strongly prefer to have the docstring follow the attribute, like function and class docstrings follow the signature, not lead it.
To solve this problem, I propose to introduce a new kind of string: a d-string
Not another string prefix :-(
I'd like to suggest an alternative syntax that doesn't need a new string prefix, although it does require new syntax. I think it is fully backwards compatible. Rather than implicitly linking a bare string to the following (or previous?) attribute, this would explicitly link them by putting them on the same line (at least for short docstrings). Full declaration syntax becomes: name : Type = value : docstring_expression with the various parts being optional. Example: @dataclass class InventoryItem: """Class for keeping track of an item in inventory.""" name: str : "Short name of the item" unit_price: float : "Price per unit in dollar." quantity_on_hand: int = 0 : ( "Available quantity currently in the warehouse.") The docstring expression could evaluate to any value, not necessarily a string, but conventionally one would use a string, and I expect style guides would discourage or ban non-strings. (But see below for a possible objection.) Aside from the name, each part of the syntax is optional: # Type hint only. name: Type # Assignment only. name = value # Declare a type hint, and a docstring. name: Type : docstring # Assign a value, and a docstring, with no type hint. name = value : docstring # And all three: type hint, value and docstring. name: Type = value : docstring If this makes sense, one might even have an attribute with no type hint or value, only a docstring: name : : docstring which would assign the docstring to `__attrdoc__` but otherwise have no effect. Formatting long lines could be done either with parentheses as in the quantity_on_hand example above, or backslashes for those who don't hate them: quantity_on_hand: \ int = 0 : \ "Available quantity currently in the warehouse." Strings of course can use any prefix, raw strings, f-strings, triple- quoted strings, whatever you like. Since the docstring part is just an expression, it doesn't need to be a string literal. If people object to the "arbitrary expression" part as too YAGNI, we could limit the docstring part to be: * a string literal (using any prefix); * an f-string; * or one of the above, parenthesed. And maybe even drop the f-string and just allow an string literal, optionally with parentheses. -- Steve
On Wed, Dec 8, 2021 at 8:41 AM Steven D'Aprano <steve@pearwood.info> wrote:
IMO this is a much better idea than the OP's, or the similar d-string idea I had come up with. Here's an example of the kind of thing I imagine doing with this feature. If I were doing a set of engineering calculations, I might get started by writing pseudocode (but using the new docstrings) like this: #### cantilevered_beam.py ##### """Checking cantilevered beam designs.""" # material properties E_s : float : "elastic modulus of steel" F_y: float : "strength of steel" # beam properties I_beam : float : "moment of inertia of beam" S_beam : float : "section modulus of beam" L_beam : float : "cantilever length" phi_M : float : "moment limit of beam" # loads P_u : float : "applied ultimate force" M_u : float : "applied ultimate bending moment" # deflections delta_max : float : "maximum beam deflection" delta_limit : float : "beam deflection limit" check : bool : "Is the beam OK? - OK (true) or No Good (false)" ### end of module #### This code doesn't yet do anything, but there is a ton of information being shared here- information that can be used to write code later, quickly. It is a very nice module template that could be used over and over again (writing out the details of actual projects later). Of course you could write pseudocode containing all of this information today with regular strings. But the difference is that, if this were to be a language feature, the information is going to be available programmatically BEFORE actually implementing anything at all. If I were in a jupyter notebook, I could write something like this (you could so similar things a REPL): [CELL 1] import cantilevered_beam print("\n".join(f"{name}:\t{docstring}" for name, docstring in cantilevered_beam.__attrdoc__.items())) ...and it would nicely output all the documentation written in the template module above for my reference, and in the next cell I would interactively compose code like this (probably copying and pasting from above, and then typing over the type-hints and docstrings): [CELL 2] #### my_cantilevered_beam.py ##### """A check for a cantilevered beam design.""" # material properties E_s = 29_000 # ksi F_y = 50 # ksi # beam properties I_beam = 1_830 # in^4 for W 24x68 beam S_beam = 154 # in^3 for W 24x68 beam L_beam = 190.25 # inches phi_M = 0.9 * F_y * S_beam # kip-in P_u = 4.947 # kips M_u = P_u * L_beam # kip-in delta_max = P_u * L_beam**3 / (48 * E_s * I_beam) # inches delta_limit = 1 # inches check = True if M_u <= phi_M and delta_max <= delta_limit else False Could I accomplish something similar today using regular strings? Sure. But it seems to me that having an official way to document attributes/members has been an oft requested feature, and I really like imaging the kinds of things I would be inclined to do with it if it were supported in the language (rather than something I was off doing on my own as a low-experience programmer, and wondering if it was dumb or a mistake). --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Wed, Dec 08, 2021 at 11:50:55AM -0000, tmkehrenberg@gmail.com wrote:
I like the idea of standardizing a name like __attrdoc__. Broadly, I think it is a good idea to enable attribute docstrings. But I think bare strings are probably not a good idea, and I definitely think adding more string prefixes is not a good idea. Of the various ideas noted here and before, I think the one that resonates with me the most is to add a docstring parser (and possibly allow others to write their own if they write their docstrings in a different way). This would look like this. # add a docstring parser decorator @parse_attrdoc_from_docstring class A: """ Class docstring. x: Docstring for x y: Docstring for y """ x: int y: bool = False And functionally, that would be equivalent to doing the following directly. # or explicitly set __attrdoc__ class B: """Class docstring.""" __attrdoc__ = { "x": "Docstring for x", "y": "Docstring for y", } x: int y: bool = False If someone wanted to document an already written library and has a different (but consistent) convention in their class docstrings for class variables, then it shouldn't be too hard to modify the default parser. Maybe? - DLD
I propose there is already a viable option in typing.Annotated. Example: @dataclass class InventoryItem: """Class for keeping track of an item in inventory.""" name: Annotated[str, "Short name of the item."] unit_price: Annotated[float, "Price per unit in dollar."] ... I've been using this in production code for about a year (with code that generates OpenAPI document), with additional validation constraints, and it's proving to be quite usable. Paul On Wed, 2021-12-08 at 11:50 +0000, tmkehrenberg@gmail.com wrote:
On Wed, Dec 8, 2021 at 12:46 PM Paul Bryan <pbryan@anode.ca> wrote:
As I noted in the previous thread, a big downside IMO of using Annotated for docstrings is it really makes the help() output quite messy. The "docstrings" appear in at least 5 different places (as part of the Annotation objects). This could be fixed, I guess. But that would skip over a question that gets raised here, which is: are a docstring and an annotation really the same thing? Are there disadvantages to assuming they are? --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
I believe a Annotated[..., str] could become an attribute docstring if by consensus we deem it to be so. I can't see any disadvantages, perhaps save for the verbosity of `Annotated`. It certainly seems like an advantage to use an existing mechanism rather than define a new one that would appear to require changes to the interpreter. On Wed, 2021-12-08 at 13:04 -0500, Ricky Teachey wrote:
On Wed, Dec 8, 2021 at 1:20 PM Paul Bryan <pbryan@anode.ca> wrote:
I think this would be better than nothing. But it is a little verbose. And it requires you to supply the type hint. I use type hinting, but I don't want to use it every time I provide an Annotation/docstring. Maybe we could do both: establish the 2nd argument of Annotated as the docstring (and adjust the output of help() and other tools accordingly), but also provide a new "shortcut syntax" as Steven proposed: E_s : float : "the elastic modulus of steel" ... is sugar for: E_s : typing.Annotated[float, "the elastic modulus of steel"] ...and: x :: "spam" ...is sugar for: x : typing.Annotated[typing.Any, "spam"] --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
I'm using docstrings bellow the attributes (analogous to functions and classes), I think it works well. It helps with distinguishing the class docstring from the arguments. On 2021-12-08 13:25:55, Ricky Teachey wrote:
In this case, why not use the current behaviour of putting the docstring below the attribute? Shouldn't typing be encouraged on dataclasses, at least?
On Thu, Dec 9, 2021 at 11:08 AM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
Shouldn't typing be encouraged on dataclasses, at least?
Typing shouldn’t be encouraged for any particular class, nor the language itself. Typing policy is a matter for the project/organization to decide. But Annotations are required by dataclasses in any case. That’s how the decorator identifies fields. The contents of those annotations are up to the user. -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Dec 9, 2021 at 3:08 PM Christopher Barker <pythonchb@gmail.com> wrote:
Since not everyone uses dataclasses and to avoid confusion: what you mean here is that type HINTS are required by dataclasses, not Annotations. An Annotated is a type of type hint that contains an annotation, which we're loosely calling a member docstring in this thread (currently, a "member docstring" is not really a thing that exists, unless you count a comment or a string nearby the statement creating the member). On Thu, Dec 9, 2021 at 2:01 PM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
What is meant here by "the current behaviour"? Member docstrings are currently not a feature supported by the language. If you like putting a string under the member definition for reference, that sounds really nice. But baking it into the language so that the docstring has to come right after the member seems limiting to me. This location requirement makes a lot of sense for modules and classes. But for class members, I'd really like to be able to organize the docstring location in a more flexible way, like (using Steven's proposed syntax): class Steel: """A simple material model of steel.""" # parameters modulus :: "the linear elastic modulus" nu :: "the poisson's ratio" gamma: "the density" # values modulus: float = 29e6 # ksi nu: float = 0.3 gamma: float = 490 # lbs per cu ft --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Thu, Dec 9, 2021 at 3:23 PM Ricky Teachey <ricky@teachey.org> wrote:
A correction to my correction (Chris Barker very kindly replied to me off-list about it; what a mensch): type hints aren't required in a dataclass. All that is required is *something* in the "type hint slot" of the expression, and that something is often (officially?) called an annotation. So we have at least 3 meanings of annotation/annotate/annotated possibly at work in this thread (what fun): 1. an annotation (the slot where type hints usually go these days, but really, you can put anything there even when constructing a dataclass), 2. the Annotated type hint type, and 3. the annotation that is the second argument of the Annotated brackets operator. How confusing! --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Fri, Dec 10, 2021 at 11:46 AM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
I meant to ask about a (global) module member, not the module docstring itself. Like MY_GLOBAL below: """This is the module docs""" MY_GLOBAL = None """MY_GLOBAL docs""" class CLS: """CLS docs""" attr: int """CLS.attr doc""" Is this "global docstring" recognized by Sphinx as a docstring, too? --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Fri, Dec 10, 2021 at 9:57 PM Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote:
To my mind Sphinx is sufficiently widely used
And the system used for the official Python docs. that this settles the
"above or below" question.
So yes :-) However, what I haven’t seen in this thread is discussion of what I think is the key question: Where/how should class attribute doc strings be stored? Tacked on to the class __doc__ ? Another dict? __attr_doc__ Added to __annotaions__ ? Something else? If they are to be available at run time, they need to go somewhere… -CHB
--
Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sat, Dec 11, 2021, 1:19 PM Christopher Barker <pythonchb@gmail.com> wrote:
The __annotations__ already exists. Is that a point in favor? If the syntax could become sugar for creating an Annotated object in __annotations__, this would be a pretty convenient location to find them. On the other hand, not every type hinted variable will have a docstring. It might be inconvenient to have them there only. It seems like storing them in a so called __attr_doc__ might be the most straightforward thing to do. If that were done, does it matter that the contents of __attr_doc__ and __annotations__ are not coupled? The strings might need to be modified. Would be inconvenient to have to modify two places in such a situation.
On Sat, Dec 11, 2021 at 10:49 AM Ricky Teachey <ricky@teachey.org> wrote:
The __annotations__ already exists. Is that a point in favor?
Yes and no. Right now, any object can be stored in annotations — having a docstring tacked on would break who knows how much code. However, there is the new inspect.get_annotations function, which could be adapted to process doc strings. https://docs.python.org/3/howto/annotations.html If the syntax could become sugar for creating an Annotated object in
__annotations__, this would be a pretty convenient location to find them.
On the other hand, not every type hinted variable will have a docstring.
And not every attribute with a docstring would have an annotation. It seems like storing them in a so called __attr_doc__ might be the most
straightforward thing to do.
Yes, as doc strings really are a different thing. Though one might to also document other things like function parameters— which would be an argument for extending what __annotations__ is used for. If that were done, does it matter that the contents of __attr_doc__ and
__annotations__ are not coupled?
I don’t think so — they are quite different concepts. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
I think that because Sphinx could interpret strings below attributes for documentation, that perhaps we should go in that direction in Python proper. Personally, I find this more readable: class C: x: Annotated[str, "Doc string"] y: Annotated[int, "Doc string"] over: class C: x: str "Doc string" y: int "Doc string" On Sun, 2021-12-12 at 10:00 +1100, Steven D'Aprano wrote:
On Sat, Dec 11, 2021 at 3:03 PM Steven D'Aprano <steve@pearwood.info> wrote:
Well, no. In fact, you could always put anything you wanted into an annotation: In [10]: class C2: ...: x: "a docstring" = 123 ...: In [11]: inspect.get_annotations(C2) Out[11]: {'x': 'a docstring'} So the typing module contains a nifty thing they've called "Annotatated", and, I imagine type checkers (like MyPy) do something sensible with them, but there's nothing else going on here. Actually yes, there is, the typing,get_type_hints function strips out the annotation: In [13]: class C: ...: x: Annotated[int, "Doc string"] = 123 ...: In [14]: typing.get_type_hints(C) Out[14]: {'x': int} Anyway, take a look at the docs for Annotated: https://docs.python.org/3/library/typing.html#typing.Annotated It has a number of possible unspecified uses, but fundamentally, it's about the adding information to the type, not to the attribute -- e.g. not really intended for docstrings. Sure, typing.Annotated *could* be used that way, but I don't think we can say that the problem is solved. Heck you could just as easily provide a tuple for the annotation: In [17]: class C: ...: x: ("Doc string", int) = 123 ...: In [18]: inspect.get_annotations(C) Out[18]: {'x': ('Doc string', int)} The use cases for annotations are a bit up in the air at the moment -- see the ongoing discussion on python-dev. But I don't think there's any doubt that any future endorsed use of them will be compatible with static typing (if not restricted to it), so extending the use of annotations for docstrings would take a bit of work to define and accept that use. Perhaps a better way to do that than to use Annotated would be to introduce new "thing", maybe in teh typoing module, like "Documented": class C: x: Documented(doc_string="this is the doc string", type=int) = 123 and then inspect.get_annotations and typing,get_type_hints could be taught to extract the type from the Documented object, and we could add an inspect.get_docstrings() function, so that: typing.get_type_hints(C) {'x': int} inspect.get_docstrings(C) {'x': 'this is the doc string'} And, of course, future syntax could automatically create a Documented() object if there was a docstring in the appropriate place. Or just have a new dunder for it, e.g. __attr_doc__ Anyway, what I'm getting at is that it needs to be determined whare to put the docstrings before we can get very far with this idea. -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sat, 2021-12-11 at 17:02 -0800, Christopher Barker wrote:
Ah, good point. I've conflated the two because dataclass attributes are used to define the schema of the dataclass. For general purpose attribute docstrings (module, class), I agree Annotated is probably inappropriate.
I think this will suffer from the same problem as Annotated: it would document the type, not the attribute itself.
Anyway, what I'm getting at is that it needs to be determined whare to put the docstrings before we can get very far with this idea.
How is Sphinx is pulling the string below the attribute? Whatever mechanism is to be used, I propose it must allow for runtime introspection.
On Sat, Dec 11, 2021 at 05:02:39PM -0800, Christopher Barker wrote:
On Sat, Dec 11, 2021 at 3:03 PM Steven D'Aprano <steve@pearwood.info> wrote:
Didn't we decide there was an existing feature for this, no need for new syntax?
Well, no. In fact, you could always put anything you wanted into an annotation:
Right, and we want to put a documentation string in it! :-)
I don't think that the docs argue against this. Annotated is documented as the way to put arbitrary metadata into an annotation. What you do with it is up to the consumer of the metadata. The docs talk about this associating the metadata with the type, but that's because this is the *typing* module and *type* hints are still the primary use-case for annotations. And despite what the docs say, the metadata is actually, literally, associated with the variable (or its name at least). The annotation that gets stored in __annotations__ is **not** {T: "metadata"} as a naive reading of the docs might trick you into believing, but {"var": Annotated[T, "metadata"]} In any case, we can still use annotations for anything we like, and while typing is definitely and certainly the primary use for them, there will always be a way to opt-out. (Just don't run mypy!) Don't need type hints at all? Great, put your doc string as the annotation. μ: 'Number of steps from x0 to the start of the cycle.' λ: 'Length (period) of the cycle.' y: 'Value of f at the start of the cycle.' = None You might want to annotate the class with `no_type_check` if there is any chance the user might want to run mypy over your module, but that's entirely up to you. You do want type hints? Great, use Annotated. If there's an attribute that you don't want to give a type hint to, just use Any. (A small price to pay for using an existing feature, instead of waiting until Python 3.11 or 3.27 when it gets added to the core language.) μ: Annotated[int, 'Number of steps from x0 to the start of the cycle.'] λ: Annotated[int, 'Length (period) of the cycle.'] y: Annotated[Any, 'Value of f at the start of the cycle.' ] = None No new syntax required. No bare strings floating around. You can generate the doc strings dynamically, they don't have to be literals. lunch: Annotated[Meal, ' '.join(['spam']*SPAM_COUNT)]
Sure, typing.Annotated *could* be used that way, but I don't think we can say that the problem is solved.
I think that from a language standpoint, we absolutely can say that the problem is solved. We don't need new syntax. We *might* want to standardise on a decorator or other tool to extract those docstrings, possibly sticking them in a new __attrdoc__ dunder. Being a dunder, we should get the core dev's approval for this, even if it is a third-party tool that uses it. (I assume we don't already use __attrdoc__ for something?) If the first use of this is dataclasses, then that would be de facto core dev approval. So we have a tool that takes an annotation: var: Annotated[T, docstring, *args] extracts out the docstring to an __attrdoc__ dunder: # Class.__attrdoc__ {'var': docstring} and removes the annotation from the annotation: # Class.__annotation__ {'var': Annotated[T, *args]} # or just T if *args is empty? Dataclasses can automatically apply this tool (subject to backwards compatibility constraints), or we can apply it as a decorator to any other class we want. So we've gone from the *hard* problem of making a language change (new syntax, build the functionality into the interpreter) to the *easy* problem of writing a decorator.
Sure, annotations can be anything you like. But external static typing tools may not support this, and will be confused or break. Same for third-party runtime tools. Best practice is to stick as closely to the standard type-hinting behaviour as you can, or to completely opt-out with no_type_hints.
Annotated is explicitly documented as *arbitrary* metadata that is entirely up to the application to interpret. There's no reason to think that will change.
Now you're just arguing about the colour of the bike-shed :-) Fine, if we want to increase the complexity of *every* tool that operates on annotations, to specially recognise this new pseudo-type, we could make them recognise Documented["docstring", T] where T is optional and to be interpreted as Any if missing. But I suspect that the type checkers will say "Just use Annotated" but what do I know? Having introduced that special Documented type-hint, we still need to modify dataclasses to extract the docstring from it, and we still need to make it available to other classes via a decorator. So apart from the spelling, which I don't think is a win, we gain nothing. -- Steve
On Sat, Dec 11, 2021, 10:58 PM Steven D'Aprano <steve@pearwood.info> wrote:
But Steve, since the most utilized documentation tool in the python universe, sphinx, doesn't look at Annotated or in the lowercase-annotation part of an expression for this and instead looks at a bare string below the member definition, and since I presume it had been doing this for a long time (before Annotated was added not long ago, and probably before the annotation too?), doesn't this mean that right now there are at least three competing ways being used to provide attribute docstrings? It really seems like it would be good for the language to standardize this. Just like function or module docstrings were standardized long ago. As far as using the lowercase-annotation for the docstring: in a world of more and more type hinted python (to be clear: I don't always use type hints but I do like this world), if you want to make you don't create a problem later, using the annotation slot for your docstring: x: "spam" ... isn't really an option. So you have little choice but to use typing.Annotated right now (unless you want to take advantage of the sphinx way of doing it). But the problem with Annotated is Matt, imo: - it's a rather long spelling - you have to import it from typing - the help output isn't very useful when you use Annotated right now (this can be improved but probably only if Annotated is officially specified as the "right place") - without an official blessed agreement that the existing Annotated feature ought to be used for docstrings, there is little reason for other 3rd party help/documentation tools to look in that location for these docstrings I just think there's more to do here than add a decorator.
On Sun, Dec 12, 2021 at 12:38:06AM -0500, Ricky Teachey wrote:
But Steve, since the most utilized documentation tool in the python universe, sphinx, doesn't look at Annotated
Yet.
or in the lowercase-annotation part of an expression
Yet.
And before we had PEP 484 there were multiple competing and incompatible conventions for putting type hints in docstrings. https://www.python.org/dev/peps/pep-0484/ There are: - Epytext (Javadoc style) - ReST (Sphinx) - Googledoc - Numpydoc although some of them may no longer be widely used. https://stackoverflow.com/questions/3898572/what-are-the-most-common-python-... https://www.adrian.idv.hk/2018-02-18-docstring/ Static and runtime tools that wanted to get type hints had to guess the format and parse the docstring to get the types. Who still does that? If dataclasses, and other classes, move to using Annotated and __attrdoc__ for docstrings, the tools will follow. Runtime tools will look at the dunder, static tools will look at the annotation directly. **Which they will have to do anyway** if we add the convention that string literals following an assignment get recorded in __attrdoc__. Runtime tools will want to look in the dunder, not parse the source code. And static tools which aren't Sphinx will need to be adjusted to look at the string literals. So either way the tooling has to change. Sphinx is not the decider here. I dare say that they will want to continue supporting bare strings into the indefinite future, but new tools, and those with weaker backwards-compatibility constaints, will surely prefer to extract docstrings from __attrdoc__ than parsing the source code.
Of course it is. Just decorate your class with @no_type_hints.
All it takes is literally one stdlib class to start doing it officially, and people will pay attention.
I just think there's more to do here than add a decorator.
That depends on whether you mean, what do *we* need to do get the ball rolling, or what does the entire Python ecosystem need to do to transition to the day that every tool and library is actively using attribute docstrings both statically and at runtime? Of course the second part is a bigger job. But for the first, we can either (1): * bikeshed this for another six ~~months~~ weeks; * get at least one core dev to agree to sponsor a PEP; * write a PEP; * bikeshed some more; * wait for the Steering Council to hopefully accept the PEP; * modify the interpreter; * wait for the new version to be available; or we can (2) * implement and support the feature today; to get to the point that external tools can start using it. I'm not kidding, if somebody cared enough to do this, they could probably have a patch for dataclasses to support this within a couple of hours and a plugin for Sphinx within a day. (Spoken with the supreme confidence of somebody who knows that he will absolutely not have to do either.) As far as I am concerned, this is exactly the sort of use-case that Annotated was invented for. Docstrings are metadata. Annotated is for attaching metadata to an annotation that is associated with some variable. Done and done. -- Steve
On Sun, Dec 12, 2021, 1:57 AM Steven D'Aprano <steve@pearwood.info> wrote: On Sun, Dec 12, 2021 at 12:38:06AM -0500, Ricky Teachey wrote:
But Steve, since the most utilized documentation tool in the python universe, sphinx, doesn't look at Annotated
Yet.
or in the lowercase-annotation part of an expression
Yet. three
competing ways being used to provide attribute docstrings?
And before we had PEP 484 there were multiple competing and incompatible conventions for putting type hints in docstrings. https://www.python.org/dev/peps/pep-0484/ There are: - Epytext (Javadoc style) - ReST (Sphinx) - Googledoc - Numpydoc although some of them may no longer be widely used. https://stackoverflow.com/questions/3898572/what-are-the-most-common-python-... https://www.adrian.idv.hk/2018-02-18-docstring/ Static and runtime tools that wanted to get type hints had to guess the format and parse the docstring to get the types. Who still does that? If dataclasses, and other classes, move to using Annotated and __attrdoc__ for docstrings, the tools will follow. Runtime tools will look at the dunder, static tools will look at the annotation directly. **Which they will have to do anyway** if we add the convention that string literals following an assignment get recorded in __attrdoc__. Runtime tools will want to look in the dunder, not parse the source code. And static tools which aren't Sphinx will need to be adjusted to look at the string literals. So either way the tooling has to change. Sphinx is not the decider here. I dare say that they will want to continue supporting bare strings into the indefinite future, but new tools, and those with weaker backwards-compatibility constaints, will surely prefer to extract docstrings from __attrdoc__ than parsing the source code.
Of course it is. Just decorate your class with @no_type_hints. party
help/documentation tools to look in that location for these docstrings
All it takes is literally one stdlib class to start doing it officially, and people will pay attention.
I just think there's more to do here than add a decorator.
That depends on whether you mean, what do *we* need to do get the ball rolling, or what does the entire Python ecosystem need to do to transition to the day that every tool and library is actively using attribute docstrings both statically and at runtime? Of course the second part is a bigger job. But for the first, we can either (1): * bikeshed this for another six ~~months~~ weeks; * get at least one core dev to agree to sponsor a PEP; * write a PEP; * bikeshed some more; * wait for the Steering Council to hopefully accept the PEP; * modify the interpreter; * wait for the new version to be available; or we can (2) * implement and support the feature today; to get to the point that external tools can start using it. I'm not kidding, if somebody cared enough to do this, they could probably have a patch for dataclasses to support this within a couple of hours and a plugin for Sphinx within a day. (Spoken with the supreme confidence of somebody who knows that he will absolutely not have to do either.) As far as I am concerned, this is exactly the sort of use-case that Annotated was invented for. Docstrings are metadata. Annotated is for attaching metadata to an annotation that is associated with some variable. Done and done. -- Steve I agree. I think adding something like __attrdoc__ is a very good idea. Would like to see it in all classes, personally, and then have help() look there. But starting with dataclasses seems like a logical place and might get us started on that goal. And if we start this with dataclasses and it gets expanded, none of this would slam the door on the idea of using a bare string after the definition as sugar for populating __attr_doc__. In fact if there's any hope that would ever happen, first people would need to find themselves using Annotated far more often, and get tired of having to use the longish syntax with an import and agree on such a syntax change.
On Sat, Dec 11, 2021 at 10:55 PM Steven D'Aprano <steve@pearwood.info> wrote:
If dataclasses, and other classes, move to using Annotated and __attrdoc__ for docstrings,
OK -- THAT is the point I brought up earlier, and you responded with "this is settled" -- it's the __attrdoc__ part that needs to be standardized. As for Typing.Annotated: it has one advantage in that typing tools currently (presumably) already accommodate extracting the type information from it -- see what typing.get_type_hints() does.
the tools will follow. Runtime tools will look at the dunder, static tools will look at the annotation directly.
I hope not. *maybe* inspect.get_annotations() though.
but they do that anyway, for all other docstrings.
As far as using the lowercase-annotation for the docstring: in a world of
more and more type hinted python
yes, it's very clear that any standardized use of annotations needs to be compatible with type hints and type checkers, static and dynamic. Which i think is Steve's point -- typing.Annotated is indeed compatible with those uses.
and then your class attribute isn't type hinted. Of course you can do it, but it can't be the standard way to add documentation to class attributes — see above. All it takes is literally one stdlib class to start doing it officially,
and people will pay attention.
Sure -- but where do the docstrings extracted from a typing.Annoated object go? __attrdoc__ has been suggested -- but THAT requires some level of approval from the SC. As far as I am concerned, this is exactly the sort of use-case that
Annotated was invented for.
and yet the docs for typing.Annotated don't seem to say that. So what concerns me is that this may break other uses of Annotated, which are expecting types to be stored there. One problem with a docstrings is that they are "just a string". isinstance checks, as mentioned in the docs, are not so helpful. So we'd be establishing a new standard: “a bare string in an Annotated type is a docstring” -- adding a typing,Documented wouldn't be a much bigger lift ( I wonder if making it a subclass of Annotated would help). And I note that Annotated flattens nested Annotated types, so having both a docstring and other use of Annotated could be a bit tricky. I'm not a type hints user -- but I think this should be run by the MyPy and other folks to see if I’m right in my skepticism. Docstrings are metadata. Annotated is for
attaching metadata to an annotation that is associated with some variable. Done and done.
It’s not Done unless the community accepts it as standard practice. This is why we have all the Typing PEPs — annotations have been there a long time, but if various tools use them differently, they really aren’t that useful. As for using dataclasses as a way to introduce a convention: that could be a good route — but I doubt Eric would go for it[*] and in any case, it would in fact be introducing a new convention, so getting some consensus on what that convention should be would be helpful. -CHB [*] Eric, of course, can speak for himself, but so far, dataclasses deliberately don’t do anything with annotations other than pass them on, and there is active debate about whether they should do more, and if so, what?
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sun, 2021-12-12 at 12:23 -0800, Christopher Barker wrote:
Confirmed. I have tooling today using exactly that to dynamically generate API documentation.
But what's being annotated, the type or the attribute?
1. Docstrings today are attached to the things they're documenting by way of an included `__doc__` attrbute. `Annotated` is arguably "attached" to the type, to the extent that the type is embedded in the annotation itself. Illustration: SomeType = Annotated[str, "some type"] class A: x: SomeType y: SomeType Clearly, above we documented the type, not the attribute. We could solve this by allowing it to be re-annotated, which seems perfectly valid: SomeType = Annotated[str, "some type"] class A: x: Annotated[SomeType, "x docstring"] y: Annotated[SomeType, "y docstring"] The last bare string could "win" as the prevaling docstring:
Or, doc tools could even include them both to provide a more meaningful context. You could argue that since the type annotation of an attribute can be extended in this manner, `Annotated` is an acceptable way of documenting an attribute. 2. Adding a typing.Documented would definitely increase verbosity. I think we may already be pushing the envelope with `Annotated[...]`; `Annotated[..., Documented(...)] may be an annotation too far. Since there is not yet a convention for a bare string in `Annotated`, I suggest we reserve it the purpose of documenting the type or attribute.
And I note that Annotated flattens nested Annotated types, so having both a docstring and other use of Annotated could be a bit tricky.
I think as long as strings are considered documentation—regardless of where they appear in the annotation—we'd still be on solid ground.
I don't think dataclasses have to do anything with them. I'm literally embedding strings in Annotated today, for this exact purpose, with no impact on dataclasses. I think the work would be in documentation tools to interpret the annotations and produce something usable to the developer.
On Sun, Dec 12, 2021 at 12:48:36PM -0800, Paul Bryan wrote:
But what's being annotated, the type or the attribute?
That's easy to test:
Okay, so it's not the type. *wink*
A.__annotations__ {'attr': typing.Annotated[int, 'Doc string']}
It's the *class* that is annotated. But note that the mapping is between the attribute name to annotation, so in the sense that attributes are represented by their name, it is the attribute that is annotated. As it obviously has to be. After all, didn't we *literally* annotate the attribute? attr: annotation Don't be fooled by the use of the Annotated pseudo-type as the annotation. We're still annotating the attribute, just like the code shows, regardless of what type we annotate it as. If we had written: attr: Sequence[int] there would be no questions about what's being annotated, is it the type int or the attribute. It cannot be the type, because many different objects with many different annotations may share the same type. Unlike modules, classes and functions, the annotation cannot be on the attribute's *value*, because the attribute may not even have a value at this point. And even if it does, it might be immutable, or like the type, many different attributes, with different annotations, may be sharing the same value. This rules out putting the docstring on the attribute directly. The syntax shows us annotating the attribute, and that's exactly what got annotated. -- Steve
On Mon, 2021-12-13 at 09:57 +1100, Steven D'Aprano wrote:
OK, so it's not the type, except it kind of is. In my other example, I used `Annotated` to create what I guess is (and what you suggested) a pseudo-type? When it was annotated it had nothing to do with any attribute. To reiterate: SomeType = Annotated[str, "some type"] No attribute here (yet). Only when this is applied as a type hint to an attribute would it then apply. class A: attr: SomeType `Annotated` certainly appears to be intended to provide hints about the type. In PEP 593, `MaxLen` is an example of prescribing constraints on value. It would apply to an attribute if that attribute was annotated with it as a type hint. By the way, to see real code implementing this convention see: https://github.com/fondat/fondat-core/blob/main/fondat/validation.py#L41 Here's another place that defines annotations that can be used to enhance API documentation (e.g. OpenAPI): https://github.com/fondat/fondat-core/blob/main/fondat/annotation.py (Disclosure: this is code I wrote.) Being able to define a new type like this using Annotated has proven to be very powerful. You can apply constraints and
It's unfortunate that the name `Annotated` was selected, because it's definitely a point that can result in confusion. An attribute can have an annotation (i.e. has a type hint); however, is `Annotated` annotating the attribute? It's an object that contains multiple values, which can be applied to an attribute as an annotation (type hint).
Agree.
This rules out putting the docstring on the attribute directly.
Agree.
The syntax shows us annotating the attribute, and that's exactly what got annotated.
I'm still not in aligned on this point. I think I agree that if you annotate an attribute with `Annotated`, you're defining a pseudo type that can serve to document the attribute. However, as I've demonstrated, that can be inherited by defining that type in advance and using it to annotate multiple attributes. As I suggested, I think a reasonable approach could be to either take the last string in the annotation, or combine multiple strings if present.
On Sun, Dec 12, 2021 at 03:38:23PM -0800, Paul Bryan wrote:
OK, so it's not the type, except it kind of is.
Except it isn't annotating the type, it is annotating the attribute. We don't annotate *types*: int: Sequence[str] That would be a regular variable or attribute called "int" that just shadowed the builtin type int. We annotate *variables* (including function parameters and class attributes).
That's not an annotation. That's a type alias. Annotations follow a colon, not an equals sign.
Right. Now you have an annotation, and sure enough, we would find {'attr': typing.Annotated[str, "some type"]} in A.__annotations__. It is still annotating the attribute (to be precise, the attribute name).
And until some variable or attribute was actually annotated with it, it might as well not exist. Annotated allows us to create new types, by associating arbitrary metadata with an existing type. The meaning of that metadata is completely unspecified. So we can say: Vec = Annotated[List[Tuple[T, T]], MaxLen(10)] but that has no real meaning until you annotate a variable or attribute, in which case the Vec type is annotated onto the attribute. That use-case for Annotated is independent of the proposed use-case here. Sure you can use it to create new types, or type-aliases. But that's not the only thing we can use it for. The meaning of MaxLen(10) is unspecified. To a human reader, we can guess that it probably means that the list can have a length of no more than 10. Any tooling that comes across it is supposed to ignore it if it doesn't know how to interpret it. So for all we know, MaxLen(10) is not enforced by any tool, and it is purely there as documentation to the reader: "Please don't make your vector longer than ten items". We're not limited to only using Annotated to create new types. In an annotation, we can associate arbitrary metadata with the annotated attribute or variable. The interpretation of that metadata is still up to the consumer. Is it a type restriction? Is it a docstring? Is it something else? It's entirely up to the tooling. By the way, the proposal to just follow attributes with a string has already been considered and rejected: https://www.python.org/dev/peps/pep-0224/ -- Steve
On Mon, 2021-12-13 at 11:22 +1100, Steven D'Aprano wrote:
1. While I agree that assigning the `Annotated` value to `SomeType` makes `SomeType` a type alias, what do you call the specific instance of `Annotated[...]` itself? To date, I've been referring to it as a type, but that's also muddying the waters here. 2. I've been flyng fast and loose with the term "annotate". No help from PEP 593, which convolutes the situation with passages such as:
Annotated is parameterized with a type and an arbitrary list of Python values that represent the annotations.
To rephrase, if I add a string "some type" to an `Annotated` instance, it would not be documenting the (non-existent) attribute, it would arguably be documenting the type itself. Getting concrete: Coordinate = Annotated[int, "a graph coordinate", ValueRange(-100, 100)] ... @dataclass class Point: x: Annotated[Coordinate, "the horizontal coordinate"] y: Annotated[Coordinate, "the vertical coordinate"] I suggest that in these uses of strings in `Annotated`, the 1st case would serve to document the "coordinate" (pseudo-)type, while the 2nd and 3rd cases would serve to document the attributes themselves. In some cases, a developer may very well be satisfied with the documentation of the (pseudo-)type and not override it when annotating the attribute.
The use of `Annotated` is to decorate an existing type to provide additional metadata about it. The `Coordinate` type alias in the example above can be introspected at runtime, independently of its use in an annotation. If such a type alias were in a module, I would want its documentation string to be displayed to provide additional context about it, and help in determining if it's appropriate to use as an annotation of an variable, parameter, attribute.
So, I think we agree that an `Annotated` instance results in what's tantamount to a new type. And I hope by my examples above, we could agree that it would be reasonable to document such a type in such a way that its usage in an annotation could either inherit it, or override it.
I agree with the above.
On Sun, Dec 12, 2021 at 05:34:33PM -0800, Paul Bryan wrote:
Its a type. The specific type is a private implementation detail: >>> type(Annotated[int, 'something']) <class 'typing._AnnotatedAlias'> I just call it a type alias. If you want to be precise, it is an alias to int, augmented with some extra metadata.
To date, I've been referring to it as a type, but that's also muddying the waters here.
No no, it is absolutely a type! _AnnotatedAlias is an instance of type itself: >>> type(typing._AnnotatedAlias) <class 'type'> >>> isinstance(typing._AnnotatedAlias, type) True although Python plays some undocumented(?) shenanigans to make issubclass fail: >>> issubclass(Annotated[int, 'something'], type) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: issubclass() arg 1 must be a class
2. I've been flyng fast and loose with the term "annotate". No help from PEP 593, which convolutes the situation with passages such as:
I will have a lot more to say on that in another post, replying to Brendan. Later.
PEP 593 is absolutely clear on the fact that the semantics of the metadata are up to the consumer, and that Python does not require it to be a type or to be treated as a type. It describes the metadata as "an arbitrary list of Python values" so we're fine to shove docstrings in there. -- Steve
On 2021-12-12 16:22, Steven D'Aprano wrote:
You're still missing what people have mentioned a few times. It is true that this annotates an attribute: class Foo: x: int The question is what does this annotate: Annotated[int, "some text here"] In other words what does the use of Annotated in itself annotate. As Paul pointed out in his earlier message, you can create this Annotated thing without attaching it to any attribute. It seems that Annotated itself is annotated the type. In fact, there can be no debate about what Annotated annotates, since the documentation even says it explicitly (https://docs.python.org/3/library/typing.html#typing.Annotated):
Specifically, a type T can be annotated with metadata x via the typehint Annotated[T, x].
So yes, Annotated IS annotating the type. The text specified in `Annotated[T, text]` is creating a sort of augmented type, which is like "a type that is type T but additionally means such-and-such". It is true that that can type can itself later be used to annotate an attribute (or variable), but at that point it will be marking the variable as being of a type that incorporates the annotating text, not annotating the variable itself with that text. As such, I don't think Annotated is really a good choice for this. We really want the text of the annotation to be associated with the attribute itself, not with the attribute's type. Moreover, using Annotated requires the user to specify a type, but people should be able to specify annotations for documentation even if they're not using typing at all and not specifying any types. (They shouldn't have to resort to an awkward workaround like always throwing in a dummy Any type.) As an aside, seeing what the docs say about Annotated makes me think that "Annotated" is a very bad name for this thing. It confuses the idea of a type annotation (i.e., attached to a variable) with this type-incorporating-a-label, where neither the label nor the type is actually an annotation in the type-annotation sense (because they have not been attached to a variable to annotate it). It seems it would have been better to called Annotated "Tagged" or "Labeled" or some such thing to make it clear that when you using it you are defining a new special-purpose type for use in later annotations./ -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Sun, Dec 12, 2021 at 07:43:16PM -0800, Brendan Barnwell wrote:
The question is what does this annotate:
Annotated[int, "some text here"]
Nothing. There's no annotation there. That's an expression which returns a type alias. It is just an object. You can print it, put it in a list, pass it to a function, use it as a dict key, or garbage collect it. Or even use it to annotate a variable, but isolated as show, it isn't annotating anything. An annotation is "A label associated with a variable, a class attribute or a function parameter or return value" and that expression above is not a label associated with anything. https://docs.python.org/3/glossary.html#term-annotation To be an annotation, it must be part of a parameter annotation, or variable assignment annotation: def func(param:expression) -> expression: variable: expression All three of those expressions could legitimately be called an annotation, even if they're not types. Here's an analogy: you know how dicts work, right? (Of course you do.) We agree that in the dict {"Hello world": 42} the string "Hello world" is the key associated with value 42. So in the context of that dict, "Hello world" is a key. # helloworld.py print("Hello world") Is that string "Hello world" a key in a dict now? What's its value? How about in isolation, "Hello world", with no implied context and most especially no dicts anywhere in sight. Is it still a key if there is no dict for it to be a key of? I'm not asking whether it *could* be a key, if there was a dict. Obviously the answer is yes. I'm asking, what dict is "Hello world" the key of in the helloworld.py program above. If we can understand that being a key describes the *role* of a string, in a specific context, and is not a property of the string itself, we should be able to understand that being an annotation is likewise descibing the *role* of an object, in a specific context, and is not a property of the object itself. types: "Hello world" = (int, float, Annotated[int, "..."], str) Here, "Hello world" is an annotation, and the Annotated type alias is a element of a tuple. Earlier the same string was the output of a program, and before that it was a key. But it's the same string. Annotated[int, "some text here"] can likewise be an annotation, or the output of a program, or a key in a dict, or an element of a tuple, depending on context. With no context, it is just a type object. If its not part of an annotation statement or a function annotation, it's not an annotation. https://docs.python.org/3/glossary.html#term-function-annotation https://docs.python.org/3/reference/simple_stmts.html#annassign
In other words what does the use of Annotated in itself annotate.
In itself, nothing. There's no annotation syntax in that expression. It's not a function annotation or a variable annotation. It doesn't create an `__annotations__` dunder. It meets none of the definitions of "annotation".
You are right about the first part of your sentence: there can be no debate about this. The doc is simply wrong, and it has confused you and others into thinking that Annotated[T, ...] has annotated T. But if anyone wants to debate it, if you think the documentation is correct, then it is easy to prove me wrong, and I will happily admit that I have learned something new. All you need to do is show me the annotation syntax `target: expression` in either of the following: # A bare expression, with no context. Annotated[int, "some text"] # A type alias. MyAlias = Annotated[int, "some text"] Easy peasy. Point to the annotation syntax colon to prove that I am wrong, and the docs are right. Or you can show me the `__annotated__` mapping with int as the key that either of those two lines create. But if you can't do any of those things, then: - there is no annotation syntax; - there is no "label associated with a variable, a class attribute or a function parameter or return value"; - there is no `__annotations__` dunder to hold that association; and therefore **there is no annotation**. It's just an expression returning a type.
The text specified in `Annotated[T, text]` is creating a sort of augmented type
The docs and the implementation call them Type Aliases. "Augmented" is an excellent description, thank you for suggesting it. So we can agree that Annotated[T, *metadata] returns a type alias, based on T, but augmented with the metadata. That is an excellent way to describe it, that avoids the confusion with annotations. Yes, the primary use-case of such a type alias is to use it in annotations, but its just an object, you can use it for many things.
What you call "marking the variable" is what Python calls an annotation. And the meaning of annotations is **not specified by the language**. Typing is the primary use-case but it is not the only one. Any alternative use of annotations has to work with a widespread assumption that annotations will be types, but that is doable.
Which you can do with Any. That is hardly a major imposition for code that is already using typing, and for code that isn't using typing, you don't have to worry about it. Just use a string.
And they can do that to, just by *not using a type checker*. @no_type_checking class C: attr: "Doc string" Done. The doc string is recorded in the __annotations__ dunder, but you can post process it any way you like. If you want a *standard* mechanism that will be blessed by the language, then it has to interoperate with typing. We've already had the idea of following the variable with a bare string rejected: https://www.python.org/dev/peps/pep-0224/ and I see no reason to think that the situation has changed in any way since then. But if you want to propose re-opening PEP 224 and having another go at it, be my guest.
Right!
Indeed. -- Steve
On 2021-12-13 17:07, Steven D'Aprano wrote:
Then let's revisit this issue after we have fixed this problem by renaming typing.Annotated to something else (or creating a new name and deprecating the old name but keeping it for backward compatibility). -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Sun, Dec 12, 2021 at 3:00 PM Steven D'Aprano <steve@pearwood.info> wrote:
I know what a wink means, but I have no idea what your actual point is. It’s very hard not to get tripped up by the overloading of the word “annotation”, but typing.Annotated can exist outside of __annotations__ it is not an annotation itself, it has special kind of type object (type descriptor? What the heck do we call things that are used for type hints but are not regular Python instantiable types?) it’s a container for a type and other extra information that pertains to that type, not necessarily to the parameter or whatever it gets attached to. As far as I know, no one in this conversation has used typing.Annotated in a type checking system— we really should hear from those folks.
The attribute got annotated the typing.Annotated object, but what did the string ‘Doc string’ get attached to? As long as we are winking: class A: attr = “a string” A.attr.upper() exists, so obviously the class attribute has that method. Of course not. The attribute has a value that is a string, and that has the upper attribute. Similarly, in your example, attr has an annotation, and the value of that annotation has a string attached to it. I’m not saying that a convention couldn’t be established to store docstrings in annotations via the typing.Annotated type. But I’m not sure that it wouldn’t get in the way of other uses, and it’s really not that obvious or convenient either. -CHB
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sun, Dec 12, 2021 at 03:46:16PM -0800, Christopher Barker wrote:
I know what a wink means, but I have no idea what your actual point is.
My actual point is that the annotation: attr: int is annotating the attribute named "attr". That's pretty basic stuff: if you don't think that we've annotated the attribute attr, then what do you think we've done? Changing int to Annotated[int, metadata]: attr: Annotated[int, metadata] is still annotating attr, not T. What else could it be? The syntax for an annotation is (roughly): name colon object and that's exactly what we have. It's an annotation, and the thing being annotated is attr. I know the docs say: "a type T can be annotated with metadata x via the typehint Annotated[T, x]" but that is wrong and misleading. Better wording would be "a type T can be *associated* with metadata x via the typehint Annotated[T, x]" but that leaves the question, why is this feature called "Annotated"? To me the answer is that it clearly refers to the fact that using this feature, we can annotate attr with *both* T and some additional metadata x, without any restriction on the nature of x. A wrinkle in this is that we can also create new types, using type aliasing: PositiveIntegers = Annotated[int, MinValue(1)] (which, by the way, was one of the motivating use-cases for this feature, so it is important in the big picture of typing, not so important for this use-case though) and in that sense we might say that the new type, PositiveIntegers, is kinda-sorta made from annotating the original type, int, with the metadata. Except... there is no actual annotation there. Its an equals sign, not a colon, so it's a name binding, which the typing system interprets as a type alias. If you look at int, it has no annotations attached, so clearly we haven't annotated int. Which brings us back to my claim that the description is misleading and it should talk about *associating* the type T with the metadata, not annotating it. Semantics are important. Would we be having this argument now if the docs didn't make the (wrong!) claim that it is the type that is being annotated? I don't think so. It is attr being annotated, not the type. And the interpretation of the metadata is entirely up to the consumer of the annotation, which means we are completely entitled to use it for docstrings.
I don't think that there is any general term for types which are only intended for type hints and cannot be instantiated. If such a standard terminology exists, I don't know it. I agree that there should be.
Indirectly to the attribute's name "attr". The implementation uses a special type alias for that purpose. Using a tuple such as (int, 'Doc string') was considered by the PEP but rejected. Nothing in the PEP or the existing docs for Annotated precludes us from using the metadata as a docstring. The interpretation of the metadata has always been up to the consumer of the metadata. -- Steve
I'll go ahead and throw this out there: I was talking to a friend about this and he pointed out something simple but astute. Will this be expected to work? class C: x: Annotated [Any, "spam"] help(C.x) Obviously it shouldn't as things stand now because that's an AttributeError. But if I'm mucking about in the interpreter and I want to see the docstring of a runtime defined (but not import time) attribute, it would be kind of nice if it would work. And it seems reasonable to try and create a way for it to work.
On Sun, Dec 12, 2021 at 08:44:25PM -0500, Ricky Teachey wrote:
You can fix that by giving x a default value.
By default, for arbitrary classes, no, it *shouldn't* work, because for arbitrary classes we cannot treat type hints as doc strings. (Actually, to be clear, I say "we" but I mean the language and the stdlib, including help(). "We" as individuals can do anything we like.) At least not without buy in from the core devs, including people like Guido who care very much about typing. And that will probably need a PEP. That's the beauty of a decorator. We don't need a PEP or Steering Council approval to write a decorator (although we might need one to make it a built-in). Just do it! But to change the language means we have to go through a big cumbersome process that may take months or years. A *lightweight* change that doesn't actually change the language could be accepted without a PEP. If you submitted a PR to update help() so that it was smart enough to recognise strings as the second parameter of Annotated annotations and treat them as docstrings, I suspect that would be allowed as just a relatively small new feature. -- Steve
Steven D'Aprano writes:
On Sun, Dec 12, 2021 at 08:44:25PM -0500, Ricky Teachey wrote:
Why not? In the sense that we establish a standard protocol for attaching them, and if there isn't one there, then 'help' would ignore that attribute or provide an empty docstring or whatever default action seems appropriate (presumably it would at least tell us the type! even if the hint is not an Annotated).
It's true that changing help will require buy-in from the core devs, as it's a built-in. But:
But to change the language
Where's the language change? What am I missing? Or are you using this loosely to include builtins or even the whole stdlib? It needs one thing: a standard place to put the documentation. I don't think just stuffing a string in __metadata__ is a good idea; that will be an annoyance to existing users of the __metadata__ attribute. I think it probably should be an attribute on Annotated instances, to avoid tromping on existing uses of the __metadata__ attribute, while allowing existing users of Annotated to take advantage of the documentation function in a gradual way. While I agree this is going to need input from core devs that would likely delay things by months rather than weeks, I see no reason why it wouldn't be in 3.12. In fact, it seems likely that this change would be small and straightforward enough that it could be prototyped, polished, and wrapped in time for 3.11. I don't think it would be that hard to get buy-in to a plausible proposal, getting Guido's attention is probably harder. ;-) (If he kibozes this post I'll cry and then laugh.)
I don't think that will fly, as it would be rude to people who are already using Annotated for something else (especially if they've already transitioned from direct use of annotations!) I don't think Python the language or the stdlib should touch the __metadata__ attribute, it should be reserved to client code.
On Tue, Dec 14, 2021 at 12:38:55AM +0900, Stephen J. Turnbull wrote:
Anything is possible! And if this is for your own code you can do anything you like. But if you want this to be a language feature, that means getting Steering Council approval, and while I don't speak for them, I am confident, well, 90% confident, that they will agree with Guido that *annotations are primarily for type-hinting*. I am also confident (say, 75% confident) that they will *disagree* that annotations are **only** for type-hinting, so there is that. But of course if anyone thinks different, and believes that the SC will bless "string annotations are docstrings" as a language feature, write a PEP and go for it!
Yes, it is somewhat loose, meaning the language proper, the builtins and stdlib. Right now, we have: class C: attr: expression causes the annotation to be stored in the `__annotations__` dunder. If all you want is to write your own metaclass or decorator to read the __annotations__ and extract strings and stick them into C.mydocstrings, you can do that right now, no questions asked. But if you want to use a dunder like __attrdocs__, say, then dunders are reserved for use for the language and we're supposed to ask first before inventing new ones. So there's that. And if you want this to new behaviour -- extracting strings from annotations to use as docstrings -- to be in the language (including the stdlib and builtins) then again we need approval. At the very least we need at least one core dev to review, accept and merge the PR. And I expect that the core devs would push it back to the typing group and SC for a decision. (That's what I would do if I knew how to merge PRs :-)
There is no guarantee or language-wide meaning to the metadata, every consumer of Annotated type aliases *must* be prepared to skip metadata they don't know how to process. It's like consumers of email. The email headers can contain arbitrary metadata that you have no clue how to handle, you are required to leave it and skip over it. But the beauty of Annotated is that it already exists and we can start using it today. If anyone wishes to push for a syntax change, or a new convenience type in the typing module: attr: Docstring["this is a doc string"] then you can do so, but you will still have to get the change approved. And what are type checkers supposed to do with it? Now every type checker has to learn to try Docstring as an alias for Any. What if you want to declare a type as well?
I think it probably should be an attribute on Annotated instances, to avoid tromping on existing uses of the __metadata__ attribute,
Is there a syntax change? If there is, then you definitely need SC approval for syntax changed. If not, how does the Annotated type know which piece of metadata gets moved into the attribute? Annotated[Any, fe, fi, fo, fum, "docstring", eeny, meany, miny, mo] *We* can recognise that clearly item number 5 is the docstring, but how does Annotated know? We need to have a set of rules in mind. -- Steve
Just a short one, for everyone agreeing type.Annotated does the job, but thinks we need new syntax, because it is verbose: You can already do: from typing import Annotated as A And: attr: A[type, "docstring goes here"] I see no need for any new syntax. (and maybe adding typing.Docstring for the cases when one just wants the docs, but no type annotation) On Tue, 14 Dec 2021 at 06:19, Steven D'Aprano <steve@pearwood.info> wrote:
On Tue, Dec 14, 2021 at 10:23 AM Joao S. O. Bueno <jsbueno@python.org.br> wrote:
You're not wrong exactly. But there has been a desire to reduce the need for typing module imports in general, and I think eliminating the need to import Annotation for this specific use case might have merit, if is becomes blessed as The Way, and if we think it is a good idea to say to people "hey! providing docstrings for members is a really good practice, and so we're making is super duper easy and standardizing it!" (just as we encouraged people to provide docstrings in the past for modules and classes by making that syntax dead simple to use). --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
Hey all: There is discussion RIGHT NOW in the SC, and on python-dev about future "policy" around annotations, in response to the PEP 563 and 649 -- not clear where it's going to end up, but it is clear that the whole "are annotations only for typing" question will be made more clear. Anyway, I had the idea that one could create a subclass of typing.Annotated, and call it "Documented", and then have it do something a little special, while still looking like an Annotated to the Typing system, e.g. get_type_hints(). However: In [2]: class Documented(typing.Annotated): ...: pass ...: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-2-fb09faa96292> in <module> ----> 1 class Documented(typing.Annotated): 2 pass 3 ~/miniconda3/envs/py310/lib/python3.10/typing.py in __init_subclass__(cls, *args, **kwargs) 1676 1677 def __init_subclass__(cls, *args, **kwargs): -> 1678 raise TypeError( 1679 "Cannot subclass {}.Annotated".format(cls.__module__) 1680 ) TypeError: Cannot subclass __main__.Annotated So those type objects in typing are weird. I'm giving up :-( -CHB On Tue, Dec 14, 2021 at 8:36 AM Ricky Teachey <ricky@teachey.org> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Christopher Barker writes:
but it is clear that the whole "are annotations only for typing" question will be made more clear.
Can we please stop posting this? AFAICS, the basic principle is absolutely clear. For the foreseeable future: 1. Annotations are NOT "only for typing". 2. Uses of annotations that get in the way of typing will be NOT adopted or enabled in the Python language, builtins, or stdlib. 3. typing.Annotated means that there is no conflict between 1 and 2, although there may be inconvenience, ugliness, or lower performance implied for some non-typing applications. Guido and the typing PEP authors have stated 1 (repeatedly!) and 2 (more recently) below. However, the definition of "typing" may get clarified, if that's what you mean (runtime vs. static, are dataclasses a typing use case, etc).
I imagine this can be done by a Sufficiently Persistent Pythonista, but I suppose a factory function would be easier. I have PoC (too embarrassing to post) given the existence of an open slot in Annotated instances (see below). The main problems (as pointed out by multiple Steves) are - Any *syntax* (language change) for creating docstrings MUST be approved by the SC, and given the rejection of PEP 224 (exactly the original proposal for "docstrings below attributes and variables") and the bikeshedding that already has taken place in this iteration, this could take years. - Any change to help(), which is a builtin, will require SC approval, and great care will have to be taken with the design i.e. #2 above for sure, and #3 should be minimized. By #3 be minimized I have in mind that there are probably existing clients that moved to annotated and expect their strings as the second argument to Annotated[]. It would be rude to make them move again. So that kind of convention for the use of Annotated.__metadata__ should be avoided IMO. Although Annotated can't be subclassed, a minimal, noninvasive, change to Annotated that makes both #2 and #3 possible would be adding a new slot __attr_doc__ into which to stuff the docstring. That could be done with a class of a different name for experimentation. This would give people working out how to use this info in help() a single protocol for getting the docstring (even if in the end there's a different protocol used in typing.Annotated, it should be amenable to abstraction, making adaptation to a new protocol simple). Another possibility would be a specific new type DocstringType that could be placed anywhere in __metadata__ and the first instance would be considered the docstring for the identifier so Annotated. I really do think that if we can keep a lid on the bikeshedding this could be in 3.11. Note that the DocstringType approach doesn't get around the SC because we still want to change help() to recognize these docstrings.
On Sun, Dec 12, 2021 at 12:23:54PM -0800, Christopher Barker wrote:
Sorry, I don't understand your position here. Putting aside *how* we write attribute docstrings, isn't the point of them that they are available to runtime inspection? Say, by help(). So why do you hope that runtime tools don't inspect the dunder storing the attribute docstrings? And that static tools *don't* look at the annotation? (Or whatever mechanism we use.) They can't use the dunder because it doesn't exist yet, you haven't run the code. So I'm confused... we go to the trouble of deciding on a mechanism to write attribute docstrings, we write them in the code, and you hope that nobody uses them??? Have I misunderstood?
Okay, I'm sorry, I understand the statement to be referring to people who had no intention of using type-hints. For those people, no_type_hints is the right solution. But for those who want type hints as well as docstrings, it is not. Sorry for the confusion.
I don't think that needs a PEP or SC approval, I think that's a small enough change that any core developer who cares can just add that dunder to their class and start collecting docstrings in there. But at some point somebody will need to ask on the Python-Dev list and see if anyone objects and says yes, you need a PEP.
Have you read the PEP? Particularly this section: https://www.python.org/dev/peps/pep-0593/#consuming-annotations Quote: Ultimately, the responsibility of how to interpret the annotations (if at all) is the responsibility of the tool or library encountering the Annotated type. A tool or library encountering an Annotated type can scan through the annotations to determine if they are of interest (e.g., using isinstance()). Unknown annotations: When a tool or a library does not support annotations or encounters an unknown annotation it should just ignore it and treat annotated type as the underlying type. Similarly in the docs: https://docs.python.org/3/library/typing.html#typing.Annotated Until such time that the Steering Council announce that type-hints are no longer merely the preferred use for annotations, but the *only* use for annotations, and any other usage is banned, using annotations to capture variable docstrings is perfectly legitimate. The current SC has made it quite clear that they are happy with the status quo, that typing is the primary but not only use for annotations. Using docstrings in annotations would seem to be a good way to establish a strong non-typing use-case.
No, we can't be that gung-ho to say that bare strings have to be docstrings. Since typing is the primary use-case for annotations, including the metadata in Annotated, it would have to be opt-in. Any type-hint can be a string: either due to PEP 563, due to forward references, or just because the developer on a whim choses to stringify their type-hints. Opting-in would mean documenting your class' requirements and assumptions about the format of annotations, and then using something like a decorator or metaclass to extract the docstrings.
And I note that Annotated flattens nested Annotated types, so having both a docstring and other use of Annotated could be a bit tricky.
class MyClass: """Blah blah blah. Variable annotations of the form Annotated[T, 'string', *args] always interpret the string in the second position as a docstring. If you want to use Annotated but without a docstring, put None in the second position. """ spam: Annotated[int, 'Yummy meat-like product'] eggs: Annotated[float, 'Goes well with spam', Domain[0, 1]] cheese: Sequence[str] # No docstring. The metadata is *always* interpreted by the tool, this is no different.
I'm not a type hints user -- but I think this should be run by the MyPy and other folks
Naturally. I could be wrong. In theory. *wink*
I only mention dataclasses because that seemed to be the primary use-case mentioned from the start of this thread. I don't speak for Eric, and I haven't verified for myself that dataclasses would benefit from attribute docstrings. -- Steve
On Mon, Dec 13, 2021 at 7:14 AM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
What about having a docstring but no typing information? In this case that's impossible, no?
spam: Annotated[Any, 'Yummy meat-like product'] or spam: Annotated[None, 'Yummy meat-like product'] -CHB
On Mon, Dec 13, 2021 at 6:54 AM Steven D'Aprano <steve@pearwood.info> wrote:
The consideration I'm making here is that any standard use of annotations needs to be fully compatible with type hinting. That's even spelled out in PEP 563. "uses for annotations incompatible with the aforementioned PEPs should be considered deprecated."
OK -- but that brings us back to how I started this sub-thread -- where are the docstrings stored to be retrieved at runtime. If that's a new dunder, great, then future enhancements making it easier to add them would be compatible, and folks could hand-populate the dunder in any number of ways, maybe including the use of typing.Annotated. Quote:
yes, i read that. This trick is that that's just kicking the can down the road. - one level up from the original state of "annotations will be stored, and you can use them however you want". But then it was decided to standardize the use of annotations as type hints. ANd now we are changing how annotations work in order to better accommodate type hinting (PEP 563). In light of the lesson there, I don't think it's a good idea to semi-standardize one use for typing.Annotations without being more explicit in the intent. Until such time that the Steering Council announce that type-hints are
but not a good idea at this point -- at least not in a way that's incompatible with type hinting.
The current SC has made it quite clear that they are happy with the status quo, that typing is the primary but not only use for annotations.
Actually, that's not the least bit clear right now -- that exact point is being mulled over as we speak. Using docstrings in annotations would seem to be a good way to establish
a strong non-typing use-case.
Then someone needs to do it NOW. The SC has literally asked for use-cases for annotations for them to consider.
in an Annotated type is a docstring” -- adding a typing,Documented wouldn't
I think it might be doable -- I'm going to give that a try soon. No, we can't be that gung-ho to say that bare strings have to be
docstrings. Since typing is the primary use-case for annotations, including the metadata in Annotated, it would have to be opt-in.
exactly -- so someone develops a tool that looks for Annotated, and tries to extract docstrings from it. And someone else uses an Annotated with a string in it for some completely different reason. And then a third person runs the docstring tool on the code using Annotated for that other reason. That just seems like asking for a mess to me. Any type-hint can be a string: either due to PEP 563, due to forward
references, or just because the developer on a whim choses to stringify their type-hints.
yes, though that is getting cleaned up, and is already a bit cleaner if you use get_type_hints or get_annotations -- that's one reason they exist.
and dataclasses are the one use-case of annotations in the stdlib. (I'm pretty sure). However, on thinking about it, this is actually NOT a good use case. The dataclasse decorator uses analysis of the decorated class to set its Fields: If the class has a class attribute that has an annotation, it is a Field. However, those class attributes are NOT necessarily class attributes in the resulting class. I haven't checked the code carefully, but it looks like if they have an immutable default, they are kept as class attributes, and otherwise are not. In [48]: @dataclass ...: class Test: ...: x: int ...: y: float = 1.0 ...: l: list = field(default_factory=list) ...: In [49]: Test.x --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-49-e65584dd4a8c> in <module> ----> 1 Test.x AttributeError: type object 'Test' has no attribute 'x' In [50]: Test.y Out[50]: 1.0 In [51]: Test.l --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-51-2185bdd4fcc8> in <module> ----> 1 Test.l AttributeError: type object 'Test' has no attribute 'l' I don't know why the immutable default is kept as a class attribute -- it gets overridden by an instance attribute when initialized anyway. But the key point is that the class attributes used to make a dataclass are not class attributes of teh dataclass, they are turned in to instance attributes. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Thinking more about my example here: On Tue, Dec 14, 2021 at 01:50:45AM +1100, Steven D'Aprano wrote:
perhaps a better convention might be that the docstring is the *last* item in the Annotated metadata, rather than the second item (after the initial type). The will allow people to use arbitrarily long augmented types, and follow it with the docstring: spam: Annotated[int, Positive, GreaterThan[1], Union[Odd|Literal[2]], Prime, Wibble, Wobble, "Doc string goes here..." ]=2, That will also help ensure that the docstring stands out visually, as it will be last in the list, not stuck in the middle. -- Steve
If we proceed with using `Annotated`, I suggest it be the last string in the metadata. Using your example: spam: Annotated[ int, Positive, GreaterThan[1], "Doc string goes here..." Union[Odd|Literal[2]], Prime, Wibble, Wobble, ] = 2 In other words, strings would be reserved to specify documentation. On Tue, 2021-12-14 at 11:38 +1100, Steven D'Aprano wrote:
On Mon, Dec 13, 2021 at 05:10:52PM -0800, Paul Bryan wrote:
In other words, strings would be reserved to specify documentation.
We can't reserve strings to specify documentation. (1) Strings can be used for forward references. class Node: payload: Any # Arbitrary data. next: Node # Fails, because Node doesn't exist yet! So we use a string for forward references: class Node: payload: Any # Arbitrary data. next: "Node" Technically, we could ask folks to write that as next: ForwardRef["Node"] but I think most people will say "Not a chance". Besides, if PEP 563 becomes standard, then all annotations will be strings. (2) Annotated[...] metadata can be arbitrary objects, so we have to assume that somebody out there is already using strings as metadata. This is why I think that the best we can do is an opt-in system where individual classes can choose to use string annotations as docs, but not a language default. But you never know what the Steering Council will say unless you ask. -- Steve
Okay, here we go: ``` import typing def extract_docstrings(K): """Extract docstrings from the class K's annotations. Class attribute docstrings are extracted from the second item in Annotated[...] attributes. The original annotation is left unchanged. FIXME: Handling of PEP 563 string annotations may not be best practice. https://www.python.org/dev/peps/pep-0563/ """ d = {} for key, value in K.__annotations__.items(): if type(value) is str: if value.startswith('Annotated[') and value.endswith(']'): s = value[10: -1] items = s.split(',') if len(items) >= 2: doc = items[1].strip() d[key] = doc elif typing.get_origin(value) is typing.Annotated: d[key] = value.__metadata__[0] K.__attrdocs__ = d return K ``` And here's an example:
-- Steve
Christopher Barker writes:
-10. They need to be accessible via the attribute name. You could insert the name before tacking on, but surely whatever format you choose, there are projects where it would be the wrong style.
Another dict?
__attr_doc__
+0.4 for now.
Added to __annotaions__ ?
+0.6 for now. That's where mypy puts type annotations, I suspect you would have to use values of Annotated type, which might be annoying.
On Sat, Dec 11, 2021, 12:57 AM Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote: Simão Afonso writes:
To my mind Sphinx is sufficiently widely used that this settles the "above or below" question. Steve I agree. Even with this convention one could still accomplish separating the documentation section from the definition section, like: class Steel: """A simple material model of steel.""" # parameters E_s: float "the linear elastic modulus" nu: float "the poisson's ratio" gamma: float "the density" # values E_s = 29e6 # ksi nu = 0.3 gamma = 490 # lbs per cu ft As has already been observed, we already have something LIKE an attribute docstring, which is an Annotated type hint object. Should the bare docstring below the member become sugar for creating an Annotated type hint? x = 1 "spam" Would then be equivalent to: x: typing.Annotated[typing.Any, "spam"] This seems really nice to me.
On Thu, 2021-12-09 at 12:32 +1100, Steven D'Aprano wrote:
Indeed.
Should all classes do the same? How about top level module variables?
If we started with dataclass, it could be performed in the dataclass decorator and make_dataclass functions. I think this would be low- hanging fruit compared to classes and modules.
On Thu, Dec 9, 2021 at 12:12 AM Paul Bryan <pbryan@anode.ca> wrote:
Again: better than nothing. But I'd really like to see this for all classes. I suggest that if a proposal like this is accepted, the help() output for classes and modules should have a new section added containing the member docstring information; it essentially would insert the member information to the end of the main docstring WHEN DISPLAYED (not suggested attaching it permanently as part of the docstring). This new section would appear directly beneath the module/class docstring when using help. Additionally, other help()-ilke tools (such as the ipython ? operator) should be encouraged to concatenate the member docstring information directly below the class/module docstring when printing out help-like information. To illustrate how things currently stand, here's a current, sort of minimally complicated class definition (class B) with some annotations: In [1]: from typing import Annotated In [2]: class B: ...: """Docstring for the B class.""" ...: ...: a: Annotated[int, "here is what i'd call a medium length docstring"] ...: b: Annotated[str, "short docstring"] ...: ....and here is the help(B) output that i get (Windows command line): In [3]: help(B) Help on class B in module __main__: class B(builtins.object) | Docstring for the B class. | | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | __annotations__ = {'a': typing.Annotated[int, "here is what i'd call a... The member annotations aren't very discoverable this way... you can't even see all of the first one. If it is agreed that Annotated should be The Blessed Way to annotate members, this needs to be improved. Here is the same thing for a dataclass (class C); here you CAN at least see the whole of each docstring, but they are on one line together and it's not all that great, and bits of the Annotated objects appear in this help output at least 5 times by my count (maybe this can't be helped since it's part of the Annotated object): In [4]: from dataclasses import dataclass In [5]: @dataclass ...: class C: ...: """Docstring for the C class.""" ...: ...: a: Annotated[int, "here is what i'd call a medium length docstring"] ...: b: Annotated[str, "short docstring"] ...: In [6]: help(C) Help on class C in module __main__: class C(builtins.object) | C(a: typing.Annotated[int, "here is what i'd call a medium length docstring"], b: typing.Annotated[str, 'short docstring']) -> None | | Docstring for the C class. | | Methods defined here: | | __eq__(self, other) | | __init__(self, a: typing.Annotated[int, "here is what i'd call a medium length docstring"], b: typing.Annotated[str, 'short docstring']) -> None | | __repr__(self) | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | __annotations__ = {'a': typing.Annotated[int, "here is what i'd call a... | | __dataclass_fields__ = {'a': Field(name='a',type=typing.Annotated[int,... | | __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or... | | __hash__ = None Here is the ipython ? output for the dataclass, by the way-- it is FAR better! But this is only so nice because dataclasses adds the typing information to the __init__ signature. For regular classes (like class B above), we would not get this nice output when using ?. In [7]: C? Init signature: C( a: typing.Annotated[int, "here is what i'd call a medium length docstring"], b: typing.Annotated[str, 'short docstring'], ) -> None Docstring: Docstring for the C class. Type: type Subclasses: --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On 9/12/21 12:50 am, tmkehrenberg@gmail.com wrote:
The main criticism, I think, was that it is weird to have the docstring *below* the attribute.
I don't think that's a problem. The docstring of a class or function is also below the name of the class or function. I think it's actually more consistent that way. It seems excessive to invent a whole new type of string just for this purpose. The neat thing about the docstring convention is that it doesn't require any special syntax. It would also raise some other questions, such as what happens if you use a d-string in some other context? Does it just behave as an ordinary string? Is it disallowed?
Does this mean you could also use it on module-level functions? d"This function is really important." def f(): ... That would give you two different ways to document a function, and it seems you could use both of them at the same time, which could be confusing. If you're looking for documentation of a function, there are two places you would have to look -- in the function itself, and in the module where it's defined. Similarly for classes. -- Greg
On Wed, Dec 08, 2021 at 11:50:55AM -0000, tmkehrenberg@gmail.com wrote:
Perhaps I missed it, but I haven't seen anyone point out that this proposal has already been suggested and rejected more than twenty years ago: https://www.python.org/dev/peps/pep-0224/ -- Steve
Hmmm, well it seems that we already have support for class attribute docstrings, since Python 3.8. It's not documented, but soon will be. https://bugs.python.org/issue46076 class Card: """A card from a standard French deck""" __slots__ = { 'suit': 'Either "Spades", "Hearts", "Clubs" or "Diamonds"', 'rank': 'A positive integer in the range 2 <= x <= 14' } help() already knows to read the docstrings from the slot dict values. -- Steve
On Tue, Dec 14, 2021, 9:49 PM Paul Bryan <pbryan@anode.ca> wrote:
Also doesn't apply to module level members. To my mind these are significant downsides. And it also represents yet another way people are documenting attributes in the wild. Here's another I learned if which is supported by sphinx autodoc: comment tag colon docstring attribute statement Example: #: spam x = 1 Using this style, the documentation has to come before the attribute. Obviously- unless VERY big change is introduced- this "docstring" will never be runtime accessible since it's a comment. But it does represent yet another way people are doing documentation in the wild. Sigh. Rick.
On Tue, Dec 14, 2021 at 06:48:57PM -0800, Paul Bryan wrote:
Interesting. Some apparent downsides:
- doesn't apply to class attributes
There is that.
- objects with `__slots__` can't dynamically add attributes
Just give it a `__dict__` slot: __slots__ = {'__dict__': None} and now it can. -- Steve
participants (11)
-
Brendan Barnwell -
Christopher Barker -
David Lowry-Duda -
Greg Ewing -
Joao S. O. Bueno -
Paul Bryan -
Ricky Teachey -
Simão Afonso -
Stephen J. Turnbull -
Steven D'Aprano -
tmkehrenberg＠gmail.com