Re: [Python-ideas] Conventions for function annotations
On Sun, Dec 2, 2012 at 4:26 PM, Robert McGibbon <rmcgibbo@gmail.com> wrote:
You're missing the other key reason for requiring decorators that interpret function annotations: they're there for the benefit of *readers*, not just other software. Given your definition above, I don't know what the annotations are for, except by recognising the "tab_glob" call. However, that then breaks as soon as the expression is put into a named variable earlier in the file: def foo(filename : text_files): # What does this mean? pass But the reader can be told *explicitly* what the annotations are related to via a decorator: @tab_expansion def foo(filename : text_files): # Oh, it's just a tab expansion specifier pass Readers no longer have to guess from context, and if the tab_expansion decorator creates IPython-specific metadata, then the interpreter doesn't need to guess either. (Note that you *can* use ordinary mechanisms like class decorators, metaclasses, post-creation modification of classes and IDE snippet inclusion to avoid the need to type out the "this is what these annotations mean" decorator explicitly. However, that's just an application of Python's standard abstraction tools, rather than a further special case convention) Mixing annotations intended for different consumers is a fundamentally bad idea, as it encourages unreadable code and complex dances to avoid stepping on each other's toes. It's better to design a *separate* API that supports composition by passing the per-parameter details directly to a decorator factory (which then adds appropriate named attributes to the function), with annotations used just as syntactic sugar for simple cases where no composition is involved. The important first question to ask is "How would we solve this if annotations didn't exist?" and only *then* look at the shorthand case for function-annotations. For cases where function annotations make code more complex or less robust, *don't use them*. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick, Thanks! You make a very convincing argument. Especially if this represents the collective recommendation of the python core development team on the proper conventions surrounding the use of function annotations, I would encourage you guys to perhaps make it more widely known (blogs, etc). As python 3.x adoption continues to move forward, this type of thing could become an issue if shmucks like me start using the annotation feature more widely. -Robert On Dec 1, 2012, at 10:58 PM, Nick Coghlan wrote:
Mixing annotations intended for different consumers is a fundamentally bad idea, as it encourages unreadable code and complex dances to avoid stepping on each other's toes. It's better to design a *separate* API that supports composition by passing the per-parameter details directly to a decorator factory (which then adds appropriate named attributes to the function), with annotations used just as syntactic sugar for simple cases where no composition is involved.
On Sun, Dec 2, 2012 at 8:12 PM, Robert McGibbon <rmcgibbo@gmail.com> wrote:
Last time it came up, the collective opinion on python-dev was still to leave PEP 8 officially neutral on the topic so that people could experiment more freely with annotations and the community could help figure out what worked well and what didn't. Admittedly this was long enough ago that I don't remember the details, just the obvious consequence that PEP 8 remains largely silent on the matter, aside from declaring that function annotations are off-limits for standard library modules: "The Python standard library will not use function annotations as that would result in a premature commitment to a particular annotation style. Instead, the annotations are left for users to discover and experiment with useful annotation styles." Obviously, I'm personally rather less open-minded on the topic of *composition* in particular, as that's a feature I'm firmly convinced should be left in the hands of ordinary decorator usage. I believe trying to contort annotations to handle that cause is almost certain to result in something less readable than the already possible decorator equivalent. However, the flip-side of the argument is that if we assume my opinion is correct and document it as an official recommendation in PEP 8, then many people won't even *try* to come up with good approaches to composition for function annotations. Maybe there *is* an elegant, natural solution out there that's superior to using explicit calls to decorator factories for the cases that involve composition. If PEP 8 declares "just use decorator factories for cases involving composition, and always design your APIs with a non-annotation based fallback for such cases", would we be inadvertently shutting down at least some of the very experimentation we intended to allow? After all, while I don't think the composition proposal in this thread reached the bar of being more readable than just composing decorator factories to handle more complex cases, I *do* think it is quite a decent attempt. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2 December 2012 11:43, Nick Coghlan <ncoghlan@gmail.com> wrote:
My concern with this is that it's tricky to experiment with composition. If you want to simultaneously use annotations for, say, one framework that checks argument types, and one that documents individual arguments based on annotations, they need to be using the same mechanism to compose annotation values. Alternatively, the first one to access the annotations could decompose the values, leaving them in a form the second can understand - but that sounds brittle and opaque. Another proposed mechanism (Robert's idea) which I didn't mention above is to override __add__, so that multiple annotations can be composed like this: def my_io(filename, mode: tab('read','write') + typed(str) ='read'): ... As a possible workaround, here's a decorator for decorators that makes the following two definitions equivalent: https://gist.github.com/4189289 @check_argtypes def checked1(a:int, b:str): pass @check_argtypes(a=int, b=str) def checked2(a, b): pass With this, it's easy to use annotations where possible, and you benefit from the extra clarity, but it's equally simple to pass the values as arguments to the decorator, for instance if the annotations are already in use for something else. It should also work under Python 2, using the non-annotated version. Thomas
On 02/12/12 22:43, Nick Coghlan wrote:
I fear that this was a strategic mistake. The result, it seems to me, is that annotations have been badly neglected. I can't speak for others, but I heavily use the standard library as a guide to what counts as good practice in Python. I'm not a big user of third party libraries, and most of those are for 2.x, so with the lack of annotations in the std lib I've had no guidance as to what sort of things annotations could be used for apart from "type checking". I'm sure that I'm not the only one. -- Steven
Indeed. I've looked at annotations before, but I never understood the purpose. It seemed like a feature that was designed and implemented without some goal in mind, and where the community was supposed to discover the goal themselves. So, if I may ask, what was the original goal of annotations? The PEP gives some suggestions, but doesn't leave anything concrete. Was it designed to be an aid to IDEs, or static analysis tools that inspect source code? Something for applications themselves to munge through to provide special behaviors, like a command line parser, or runtime static checker? The local decorator influence might work, but that has the problem of only being able to be used once before we fall back to the old method. Would you rather: @tab_expand(filename=glob('*.txt')) @types def read_from_filename(filename:str, num_bytes:int) -> bytes: pass or @tab_expand(filename=glob('*.txt')) @types(filename=str, num_bytes=int, return_=bytes) def read_from_filename(filename, num_bytes): pass For consistency's sake, I'd prefer the latter. Note that we could take a convention, like Thomas suggests, and adopt both: @tab_expand @types def read_from_filename(filename:(str, glob('*.txt')), num_bytes:int) -> bytes: pass But that's a "worst of both worlds" approach: we lose the locality of which argument applies to which decorator (unless we make up rules about positioning in the tuple or something), and we gunk up the function signature, all to use a fancy new Python 3 feature. With a restricted and narrow focus, I could see them gaining adoption, but for now, it seems like extra syntax was introduced simply for the point of having extra syntax. On Sun, Dec 2, 2012 at 5:23 PM, Steven D'Aprano <steve@pearwood.info> wrote:
-- Jasper
On 4 December 2012 16:43, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
Using the decorator decorator I posted (https://gist.github.com/4189289 ), you could use these interchangeably, so the annotations are just a convenient alternative syntax for when you think they'd make the code more readable. Thomas
On 12/4/2012 11:43 AM, Jasper St. Pierre wrote:
A telling moment for me was during an early Py3k keynote at PyCon (perhaps it was in Dallas or Chicago?), Guido couldn't remember the word "annotation," and said, "you know, those things that aren't type declarations?" :-) --Ned.
On Tue, Dec 4, 2012 at 9:12 AM, Ned Batchelder <ned@nedbatchelder.com> wrote:
To the contrary. There were too many use cases that immediately looked important, and we couldn't figure out which ones would be the most important or how to combine them, so we decided to take a two-step approach: in step 1, we designed the syntax, whereas in step 2, we would design the semantics. The idea was very clear that once the syntax was settled people would be free to experiment with different semantics -- just not in the stdlib. The idea was also that eventually, from all those experiments, one would emerge that would be fit for the stdlib. The process was somewhat similar to the way decorators were introduced. In Python 2.3, we introduced things like staticmethod, classmethod and property. But we *didn't* introduce the @ syntax, because we couldn't agree about it at that point. Then, for 2.4, we sorted out the proper syntax, having by then conclusively discovered that the original way of using e.g. classmethod (an assignment after the end of the method definition) was hard on the human reader. (Of course, you may note that for decorators, we decided on semantics first, syntax second. But no two situations are quite the same, and in the case of annotations, without syntax it would be nearly impossible to experiment with semantics.)
Pretty much all of the above to some extent. But for me personally, the main goal was always to arrive at a notation to specify type constraints (and maybe other constraints) for arguments and return values. I've toyed at various times with specific ways of combining types. E.g. list[int] might mean a list of integers, and dict[str, tuple[float, float, float, bool]] might mean a dict mapping strings to tuples of three floats and a bool. But I felt it was much harder to get consensus about such a notation than about the syntax for argument annotations (think about how many objections you can bring in to these two examples :-) -- I've always had a strong desire to use "var: type = default" and to make the type a runtime expression to be evaluated at the same time as the default.
Heh. :-) -- --Guido van Rossum (python.org/~guido)
Check out http://www.artima.com/weblogs/viewpost.jsp?thread=89161 -eric On Tue, Dec 4, 2012 at 9:43 AM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
On Dec 2, 2012, at 3:43 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Admittedly this was long enough ago that I don't remember the details, just the obvious consequence that PEP 8 remains largely silent on the matter, aside from declaring that function annotations are off-limits for standard library modules:
PEP 8 is not "largely silent" on the subject: "'' The Python standard library will not use function annotations as that would result in a premature commitment to a particular annotation style. Instead, the annotations are left for users to discover and experiment with useful annotation styles. Early core developer attempts to use function annotations revealed inconsistent, ad-hoc annotation styles. For example: [str] was ambiguous as to whether it represented a list of strings or a value that could be either str or None. The notation open(file:(str,bytes)) was used for a value that could be either bytes or str rather than a 2-tuple containing a str value followed by a bytesvalue. The annotation seek(whence:int) exhibited an mix of over-specification and under-specification: int is too restrictive (anything with __index__ would be allowed) and it is not restrictive enough (only the values 0, 1, and 2 are allowed). Likewise, the annotation write(b: bytes) was also too restrictive (anything supporting the buffer protocol would be allowed). Annotations such as read1(n: int=None) were self-contradictory since None is not an int. Annotations such as source_path(self, fullname:str) -> objectwere confusing about what the return type should be. In addition to the above, annotations were inconsistent in the use of concrete types versus abstract types: int versus Integral and set/frozenset versus MutableSet/Set. Some annotations in the abstract base classes were incorrect specifications. For example, set-to-set operations require other to be another instance of Setrather than just an Iterable. A further issue was that annotations become part of the specification but weren't being tested. In most cases, the docstrings already included the type specifications and did so with greater clarity than the function annotations. In the remaining cases, the docstrings were improved once the annotations were removed. The observed function annotations were too ad-hoc and inconsistent to work with a coherent system of automatic type checking or argument validation. Leaving these annotations in the code would have made it more difficult to make changes later so that automated utilities could be supported. ''' Raymond
On Mon, Dec 3, 2012 at 6:09 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:
It's effectively silent on the matters at hand, which are: * the advisability of using annotations without an associated decorator that makes the interpretation currently in play explicit (while the examples given do illustrate why *not* doing this is a bad idea, it doesn't explicitly state that conclusion, merely "we're not going to use them in the standard library at this point") * the advisability of providing a pure annotations API, without any fallback to an explicit decorator factory * the advisability of handling composition within the annotations themselves, rather than by falling back to explicit decorator factories * the advisability of using the __annotations__ dictionary for long-term introspection, rather than using the decorator to move the information to a purpose-specific location in a separate function attribute I would be *quite delighted* if people are open to the idea of making a much stronger recommendation along the following lines explicit in PEP 8: ================== * If function annotations are used, it is recommended that: * the annotation details should be designed with a specific practical use case in mind * the annotations are used solely as a form of syntactic sugar for passing arguments to a decorator factory that would otherwise accept explicit per-parameter arguments * the decorator factory name should provide the reader of the code with a strong hint as to the intended meaning of the parameter annotations (or at least a convenient reference point to look up in the documentation) * in simple cases, using parameter and return type annotations will then allow the per-parameter details to be mapped easily by both the code author and later readers without requiring repetition of parameter names or careful alignment of factory arguments with parameter positions. * the explicit form remains available to handle more complex situations (such as applying multiple decorators to the same function) without requiring complicated conventions for composing independent annotations on a single function ================== In relation to the last point, I consider composing annotations to be analogous to composing function arguments. Writing: @g @f def annotated(arg1: (a, x), arg2: (b, y), arg3: (c, z)): ... instead of the much simpler: @g(x, y, z) @f(a, b, c) def annotated(arg1, arg2, arg3): ... is analagous to writing: args = [(a, x), (b, y), (c, z)] f(*(x[0] for x in args)) g(*(x[1] for x in args)) instead of the more obvious: f(a, b, c) g(x, y, z) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Dec 3, 2012 at 9:45 PM, Andrew Svetlov <andrew.svetlov@gmail.com>wrote:
I don't quite understand that comment - PEP 362 is purely an access mechanism. The underlying storage is still in __annotations__ (at least as far any annotations are concerned). However, using separate storage is a natural consequence of also providing an explicit decorator factory API, so I didn't bring it up. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Dec 03, 2012, at 09:08 PM, Nick Coghlan wrote:
I would be *quite delighted* if people are open to the idea of making a much stronger recommendation along the following lines explicit in PEP 8:
I am -1 for putting any of what followed in PEP 8, and in fact, I think the existing examples at the bottom of PEP 8 are inappropriate. PEP 8 should be prescriptive of explicit Python coding styles. Think "do this, not that". It should be as minimal as possible, and in general provide rules that can be easily referenced and perhaps automated (e.g. pep8.py). Some of the existing text in PEP 8 already doesn't fall under that rubric, but it's close enough (e.g. designing for inheritance). I don't think annotations reach the level of consensus or practical experience needed to be added to PEP 8. OTOH, I wouldn't oppose a new informational PEP labeled "Annotations Best Practices", where some of these principles can be laid out and explored. Cheers, -Barry
Hm. I agree PEP 8 seems an odd place for Nick's recommendation. Even if I were to agree with hos proposal I would think it belongs in a different PEP than PEP 8. But personally I haven't given up on using annotations to give type hints -- I think it can at some times be a useful augmentation to static analysis (whose use I see mostly as an aid to human readers and/or tools like linters, IDEs, and refactoring tools, not for guiding compiler optimizations). I know of several projects (both public and private) for improving the state of the art of Python static analysis with this goal in mind. With the advent of e.g. TypeScript and Dart in the JavaScript world, optional type annotations for dynamic languages appear to be becoming more fashionable, and maybe we can get some use out of them. FWIW, as far as e.g. 'int' being both overspecified and underspecified: I don't care about the underspecification so much, that's always going to happen; and for the overspecification, we can either use some abstract class instead, or simply state that the occurrence of certain concrete types must be taken as a shorthand for a specific abstract type. This could be part of the registration call of the concrete type, or something. Obviously this would require inventing and standardizing notations for things like "list of X", "tuple with items X, Y, Z", "either X or Y", and so on, as well as a standard way of combining annotations intended for different tools. *This* would be a useful discussion. What to do in the interim... I think the current language in PEP 8 is just fine until we have a better story. --Guido On Mon, Dec 3, 2012 at 7:34 AM, Barry Warsaw <barry@python.org> wrote:
-- --Guido van Rossum (python.org/~guido)
So long as any type hinting semantics are associated with a "@type_hints" decorator, none of those ideas conflict with my suggestions for good annotation usage practices. The explicit decorators effectively end up serving as dialect specifiers for the annotations, for the benefit of other software (by moving the metadata out to purpose specific attributes) and for readers (simply by being present). Anyway, the reactions here confirmed my recollection of a lack of consensus amongst the core team. I'll just put something up on my own site, instead. Cheers, Nick. -- Sent from my phone, thus the relative brevity :) On Dec 4, 2012 3:28 AM, "Guido van Rossum" <guido@python.org> wrote:
Just thought of a couple of usages which don't fit into the decorator model. The first is using the return annotation for early binding: def func(seq) -> dict(sorted=sorted): return func.__annotations__['return']['sorted'](seq) Stangely enough, this seems to run slightly faster than def func(seq, sorted=sorted): return sorted(seq) My test shows the first running in about 0.376s and the second in about 0.382s (python 3.3, 64bit). The second is passing information to base classes. This is a rather contrived example which could easily be solved (better) in plenty of other ways, but it does illustrate a pattern which someone else may be able to turn into a genuine use case. class NumberBase: def adjust(self, value): return self.adjust.__annotations__['return'](value) class NegativeInteger(NumberBase): def adjust(self, value) -> int: return super().adjust(-value)
Cheers David On Tue, Dec 4, 2012 at 1:02 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Dec 4, 2012 at 1:37 AM, David Townshend <aquavitae69@gmail.com> wrote:
You've got to be kidding...
Surely that's some kind of random variation. It's only a 2% difference.
This looks like a contrived way to use what is semantically equivalent to function attributes. The base class could write def adjust(self, value): return self.adjust.adjuster(value) and the subclass could write def adjust(self, value): return super().adjust(-value) adjust.adjuster = int Or invent a decorator to set the attribute: @set(adjuster=int) def adjust(self, value): return super().adjust(-value) But both of these feel quite awkward compared to just using a class attribute. class NumberBase: def adjust(self, value): return self.adjuster(value) class NegativeInteger(NumberBase): adjuster = int # No need to override adjust() IOW, this is not a line of thought to pursue. -- --Guido van Rossum (python.org/~guido)
On Wed, Dec 5, 2012 at 8:17 PM, Guido van Rossum <guido@python.org> wrote:
It's consistent. I ran several tests and came out with the same 2% difference every time.
IOW, this is not a line of thought to pursue.
I wasn't suggesting that this is a good idea, I was merely trying to point out that there are currently ways of using annotations beyond type declarations with decorators, and that there may be other use cases out there which will work well. Documenting recommendations that annotations only be used with decorators, or only be used for type declarations will limit the possibilities because nobody will bother to look further, and if they do, the ideas will no doubt be shut down as being bad style because they go against the recommended usage. I thought that limiting annotations like this was what you wanted to avoid? Having said that, I've never found a good use for annotations in my own code, so I'm not emotionally invested one way or the other. I do think that the best usage I've seen is exactly what is being discussed here and it would be great if there was some prescribed use for annotations. Perhaps people would actually use them then. David
On 2012-12-03, at 18:27 , Guido van Rossum wrote:
I've always felt that __getitem__ and __or__/__ror__ on type 1. looked rather good and 2. looked similar to informal type specs and type specs of other languages. Although that's the issue with annotations being Python syntax: it requires changing stuff fairly deep into Python to be able to experiment. The most bothersome part is that I "feel" "either X or Y" (aka `X | Y`) should be a set of type (and thus the same as {X, Y}[0]) but that doesn't work with `isinstance` or `issubclass`. Likewise, `(a, b, c)` in an annotation feels like it should mean the same as `tuple[a, b, c]` ("a tuple with 3 items of types resp. a, b and c") but that's at odds with the same type-checking functions. The first could be fixable by relaxing slightly the constraints of isinstance and issubclass, but not so for the second. [0] which works rather neatly for anonymous unions as `|` is the union of two sets, so the arithmetic would be `type | type -> typeset`, `type | typeset -> typeset` and `typeset | typeset -> typeset`, libraries could offer opaque types/typesets which would be composable without their users having to know whether they're type atoms or typesets
On Tue, Dec 4, 2012 at 10:12 AM, Masklinn <masklinn@masklinn.net> wrote:
So, instead of using def foo(a: int, b: str) -> float: <blah> you use from experimental_type_annotations import Int, Str, Float def foo(a: Int, b: Str) -> Float: <blah> And now we're ready for experimentation. [Warning: none of this is particularly new; I've had these things in my brain for years, as the referenced Artima blog post made clear.]
Note that in Python 3 you can override isinstance, by defining __instancecheck__ in the class: http://docs.python.org/3/reference/datamodel.html?highlight=__instancecheck_... So it shouldn't be a problem to make isinstance(42, Int) work. We can also make things like List[Int] and Dict[Str, Float] work, and even rig it so that isinstance([1, 2, 3], List[Int]) == True while isinstance([1, 2, 'booh'], List[Int]) == False Of course there are many bikeshedding topics like whether we should ever write List -- maybe we should write Iterable or Sequence instead, and maybe we have to be able to express mutability, and so on. The numeric tower (PEP 3141) is also good to keep in mind. I think that's all solvable once we start experimenting a bit. Some important issues to bikeshed over: - Tuples. Sometimes you want to say e.g. "a tuple of integers, don't mind the length"; other times you want to say e.g. "a tuple of fixed length containing an int and two strs". Perhaps the former should be expressed using ImmutableSequence[Int] and the second as Tuple[Int, Str, Str]. - Unions. We need a way to say "either X or Y". Given that we're defining our own objects we may actually be able to get away with writing e.g. "Int | Str" or "Str | List[Str]", and isinstance() would still work. It would also be useful to have a shorthand for "either T or None", written as Optional[T] or Optional(T). - Whether to design notations to express other constraints. E.g. "integer in range(10, 100)", or "one of the strings 'r', 'w' or 'a'", etc. You can go crazy on this. - Composability (Nick's pet peeve, in that he is against it). I propose that we reserve plain tuples for this. If an annotation has the form "x: (P, Q)" then that ought to mean that x must conform to both P and Q. Even though Nick doesn't like this, I don't think we should do everything with decorators. Surly, the decorators approach is good for certain use cases, and should take precedence if it is used. But e.g. IDEs that use annotations for suggestions and refactoring should not require everything to be decorated -- that would just make the code too busy. - Runtime enforcement. What should we use type annotations for? IDEs, static checkers (linters) and refactoring tools only need the annotations when they are parsing the code. While it is tempting to invent some kind of runtime checking that automatically checks the actual types against the annotations whenever a function is called, I think this is rarely useful, and often prohibitively slow. So I'd say don't focus on this. Instead, explicit type assertions like "assert isinstance(x, List[Int])" might be used, sparingly, for those cases where we'd otherwise write a manual assertion with the same meaning (which is also sparingly!). A decorator to do this might be useful (especially if there's a separate mechanism for turning actual checking on or off through some configuration mechanism).
I like this for declaring union types. I don't like it for composing constraints that are intended for different tools. -- --Guido van Rossum (python.org/~guido)
Nice, that seems very explicit. ImmutableSequence is long, but clear. In this specific case, should it be just Sequence, and a mutable one would be MutableSequence (to be consistent with collections.abc names?).
Definitely useful to have a notation for "either T or None", as it's a pretty heavily-used pattern. But what about using the same approach, something like "T | None" or "T | NoneType". Though if you use the real None rather than experimental_type_annotations.None, is that confusing? In any case, it seems unnecessary to have a special Optional(T) notation when you've already got the simple "T1 | T2" notation.
Yes, I think this is dangerous territory -- it could get crazy very fast. Statically typed languages don't have this. Then again, I guess type annotations have the potential to be *more* powerful in this regard. Still, it'd have to be an awfully nice and general notation for it to be useful. Even then, your "def" line complete with type/constraint annotations may get far too long to be readable... -Ben
On Wed, Dec 5, 2012 at 11:22 AM, Guido van Rossum <guido@python.org> wrote:
Optional is not the same as "or None" to me: Dict(a=Int, b=Int | None, c=Optional(Int)) suggests that b is required but might be None while c is not required, i.e., {'a': 3, b: None} is allowed while {'a': 3, c: None} is not. Ditto for Tuples: Tuple[Int, Str | None, Optional(Int)] where (3, None) matches as does (3, 'a', 4) but not (3, None, None). Optionals might be restricted to the end as matching in the middle would be complicated and possibly error-prone: Tuple[Int, Optional(Int | None), Int | Str, Int | None] --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
On Wed, Dec 5, 2012 at 3:01 PM, Guido van Rossum <guido@python.org> wrote:
Those are not the semantics I had in mind for Optional.
I know that. My point was that the standard meaning of the word optional is that something may or may not be given (or whatever the applicable verb is). That's quite different from saying it must be provided but may be None. Since you invited a bit of bikeshedding, I felt it was appropriate to point that out and then I got distracted by discussing the alternative that you weren't talking about. Sorry that was confusing. In C#, this is called Nullable and you can write Nullable<String> to indicate the type (String or null type). The shorthand for that is String?. If you want a shorthand to specify that None is allowed, I'd suggest ~Str. --- Bruce P.S. Optional[T] is not literally a shorthand for T | None as the former is 11 characters and the latter is 10 characters even if we include and count the spaces. :-) P.P.S. I don't think Str | None rather than Str | NoneType is confusing.
On 2012-12-05, at 20:22 , Guido van Rossum wrote:
My problem there was more about having e.g. Int | Float return a set, but isinstance not working with a set. But indeed it could return a TypeSet which would implement __instancecheck__.
Well if `|` is the "union operator", as Ben notes `T | None` works well, is clear and is sufficient. Though that's if and only if "Optional[T]" is equivalent to "T or None" which Bruce seems to disagree with. There's some history with this pattern: http://journal.stuffwithstuff.com/2010/08/23/void-null-maybe-and-nothing/ (bottom section, from "Or Some Other Solution")
Yes this is going in Oleg territory, a sound core is probably a good starting idea. Although basic enumerations ("one of the strings 'r', 'w' or 'a'") could be rather neat.
For IDEs, that's pretty much all the time though, either they're parsing the code or they're trying to perform static analysis on it, which uses the annotations.
Could be useful for debug or testing runs though, in the same way event-based profilers are prohibitively slow and can't be enabled all the time but are still useful. Plus it might be possible to enable/disable this mechanism with little to no source modification via sys.setprofile (I'm not sure what hooks it provides exactly and the documentation is rather sparse, so I'm not sure if the function object itself is available to the setprofile callback, looking at Lib/profiler.py it might only get the code object).
On Wed, Dec 5, 2012 at 12:34 PM, Masklinn <masklinn@masklinn.net> wrote:
Right, that's what I meant.
Actually, I find "T|None" somewhat impure, since None is not a type but a value. If you were allow this, what about "T|False"? And then what about "True|None"? (There's no way to make the latter work!) And I think "T|NoneType" is obscure; hence my proposal of Optional(T). (Not Optional[T], since Optional is not a type.)
Yeah, they're parsing it, but they're not executing it.
Hence my idea of using a decorator to enable this on specific functions. -- --Guido van Rossum (python.org/~guido)
On Thu, Dec 6, 2012, at 3:43, Masklinn wrote:
Why would Optional not be a type? It's coherent with Option or Maybe types in languages with such features, or C#'s Nullable.
C#'s Nullable doesn't really work outside a static typing system - when you assign a Nullable to an 'object' or a 'dynamic', you get either the original type (e.g. Int32) or a null reference (which has no type). It's a real type only as far as the static typing system goes: it can be the type of a field or a local variable, it _cannot_ be the type of an object on the heap. And since python doesn't have static typing...
On Thu, Dec 6, 2012 at 5:22 AM, Guido van Rossum <guido@python.org> wrote:
I'm not against using composition within a particular set of annotation semantics, I'm against developing a convention for arbitrary composition of annotations with *different* semantics. Instead, I'm advocating for the following guidelines to avoid treading on each others toes when experimenting with annotations and to leave scope for us to define standard annotation semantics at a future date: 1. Always use a decorator that expresses the annotation semantics in use (e.g. tab completion, type descriptions, parameter documentation) 2. Always *move* the annotations out to purpose-specific storage as part of the decorator (don't leave them in the annotations storage) 3. When analysing a function later, use only the purpose-specific attribute(s), not the raw annotations storage 4. To support composition with other sets of annotation semantics, always provide an alternate API that accepts the per-parameter details directly (e.g. by name or index) rather than relying solely on the annotations The reason for this is so that if, at some future point in the time, python-dev agrees to bless some particular set of semantics as *the* meaning of function annotations (such as the type hinting system being discussed), then that won't break anything. Otherwise, if people believe that it's OK for them to simply assume that the contents of the annotations mean whatever they mean for their particular project, then it *will* cause problems further down the road as annotations written for one set of semantics (e.g. tab completion, parameter documentation) get interpreted by a processor expecting different semantics (e.g. type hinting). Here's how your example experiment would look under such a scheme: from experimental_type_annotations import type_hints, Int, Str, Float # After type_hints runs, foo.__annotations__ would be empty, and the type # hinting data would instead be stored in (e.g.) a foo._type_hints attribute. @type_hints def foo(a: Int, b: Str) -> Float: <blah> This is then completely clear and unambigious: - readers can see clearly that these annotations are intended as type hints - the type hinting processor can see that there *is* type hinting information available, due to the presence of a _type_hints attribute - other automated processors see that there are no "default" annotations (which is good, since there is currently no such thing as "default" annotation semantics) Furthermore, (as noted elsewhere in the thread) an alternate API can then easily be provided that supports composition with other annotations: @type_hints(Int, Str, _return=Float) def foo(a, b): <blah> Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Nick, I understand your position completely (and I did before). I just disagree. :-) I think that requiring the experiment I am proposing to use a decorator on each function that uses it (rather than just an import at the top of the module) will cause too much friction, and the experiment won't get off the ground. That's why I am proposing a universal composition convention: When an annotation for a particular argument is a tuple, then any framework or decorator that tries to assign meanings to annotations must search the items of the tuple for one that it can understand. For the experimental type annotation system I am proposing this should be simple enough -- the type annotation system can require that the things it cares about must all be subclasses of a specific base class (let's call it TypeConstraint). If the annotation is not a tuple, it should be interpreted as a singleton tuple. Yes, it is possible that a mistake leaves an annotation unclaimed. But that's no worse than currently, where all annotations are ignored. And for TypeConstraint there is no runtime behavior anyway (unless you *do* add a decorator) -- its annotations are there for other tools to parse and interpret. It's like pylint directives -- if you accidentally misspell it 'pylnt' you get no error (but you may still notice that something's fishy, because when pylint runs it doesn't suppress the thing you tried to suppress :-). --Guido On Wed, Dec 5, 2012 at 9:27 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On Wed, Dec 5, 2012 at 9:22 PM, Guido van Rossum <guido@python.org> wrote:
Just to note: there are https://github.com/Deepwalker/trafaret library intended for checking on complex enough structures.
Nick, Thanks! You make a very convincing argument. Especially if this represents the collective recommendation of the python core development team on the proper conventions surrounding the use of function annotations, I would encourage you guys to perhaps make it more widely known (blogs, etc). As python 3.x adoption continues to move forward, this type of thing could become an issue if shmucks like me start using the annotation feature more widely. -Robert On Dec 1, 2012, at 10:58 PM, Nick Coghlan wrote:
Mixing annotations intended for different consumers is a fundamentally bad idea, as it encourages unreadable code and complex dances to avoid stepping on each other's toes. It's better to design a *separate* API that supports composition by passing the per-parameter details directly to a decorator factory (which then adds appropriate named attributes to the function), with annotations used just as syntactic sugar for simple cases where no composition is involved.
On Sun, Dec 2, 2012 at 8:12 PM, Robert McGibbon <rmcgibbo@gmail.com> wrote:
Last time it came up, the collective opinion on python-dev was still to leave PEP 8 officially neutral on the topic so that people could experiment more freely with annotations and the community could help figure out what worked well and what didn't. Admittedly this was long enough ago that I don't remember the details, just the obvious consequence that PEP 8 remains largely silent on the matter, aside from declaring that function annotations are off-limits for standard library modules: "The Python standard library will not use function annotations as that would result in a premature commitment to a particular annotation style. Instead, the annotations are left for users to discover and experiment with useful annotation styles." Obviously, I'm personally rather less open-minded on the topic of *composition* in particular, as that's a feature I'm firmly convinced should be left in the hands of ordinary decorator usage. I believe trying to contort annotations to handle that cause is almost certain to result in something less readable than the already possible decorator equivalent. However, the flip-side of the argument is that if we assume my opinion is correct and document it as an official recommendation in PEP 8, then many people won't even *try* to come up with good approaches to composition for function annotations. Maybe there *is* an elegant, natural solution out there that's superior to using explicit calls to decorator factories for the cases that involve composition. If PEP 8 declares "just use decorator factories for cases involving composition, and always design your APIs with a non-annotation based fallback for such cases", would we be inadvertently shutting down at least some of the very experimentation we intended to allow? After all, while I don't think the composition proposal in this thread reached the bar of being more readable than just composing decorator factories to handle more complex cases, I *do* think it is quite a decent attempt. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2 December 2012 11:43, Nick Coghlan <ncoghlan@gmail.com> wrote:
My concern with this is that it's tricky to experiment with composition. If you want to simultaneously use annotations for, say, one framework that checks argument types, and one that documents individual arguments based on annotations, they need to be using the same mechanism to compose annotation values. Alternatively, the first one to access the annotations could decompose the values, leaving them in a form the second can understand - but that sounds brittle and opaque. Another proposed mechanism (Robert's idea) which I didn't mention above is to override __add__, so that multiple annotations can be composed like this: def my_io(filename, mode: tab('read','write') + typed(str) ='read'): ... As a possible workaround, here's a decorator for decorators that makes the following two definitions equivalent: https://gist.github.com/4189289 @check_argtypes def checked1(a:int, b:str): pass @check_argtypes(a=int, b=str) def checked2(a, b): pass With this, it's easy to use annotations where possible, and you benefit from the extra clarity, but it's equally simple to pass the values as arguments to the decorator, for instance if the annotations are already in use for something else. It should also work under Python 2, using the non-annotated version. Thomas
On 02/12/12 22:43, Nick Coghlan wrote:
I fear that this was a strategic mistake. The result, it seems to me, is that annotations have been badly neglected. I can't speak for others, but I heavily use the standard library as a guide to what counts as good practice in Python. I'm not a big user of third party libraries, and most of those are for 2.x, so with the lack of annotations in the std lib I've had no guidance as to what sort of things annotations could be used for apart from "type checking". I'm sure that I'm not the only one. -- Steven
Indeed. I've looked at annotations before, but I never understood the purpose. It seemed like a feature that was designed and implemented without some goal in mind, and where the community was supposed to discover the goal themselves. So, if I may ask, what was the original goal of annotations? The PEP gives some suggestions, but doesn't leave anything concrete. Was it designed to be an aid to IDEs, or static analysis tools that inspect source code? Something for applications themselves to munge through to provide special behaviors, like a command line parser, or runtime static checker? The local decorator influence might work, but that has the problem of only being able to be used once before we fall back to the old method. Would you rather: @tab_expand(filename=glob('*.txt')) @types def read_from_filename(filename:str, num_bytes:int) -> bytes: pass or @tab_expand(filename=glob('*.txt')) @types(filename=str, num_bytes=int, return_=bytes) def read_from_filename(filename, num_bytes): pass For consistency's sake, I'd prefer the latter. Note that we could take a convention, like Thomas suggests, and adopt both: @tab_expand @types def read_from_filename(filename:(str, glob('*.txt')), num_bytes:int) -> bytes: pass But that's a "worst of both worlds" approach: we lose the locality of which argument applies to which decorator (unless we make up rules about positioning in the tuple or something), and we gunk up the function signature, all to use a fancy new Python 3 feature. With a restricted and narrow focus, I could see them gaining adoption, but for now, it seems like extra syntax was introduced simply for the point of having extra syntax. On Sun, Dec 2, 2012 at 5:23 PM, Steven D'Aprano <steve@pearwood.info> wrote:
-- Jasper
On 4 December 2012 16:43, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
Using the decorator decorator I posted (https://gist.github.com/4189289 ), you could use these interchangeably, so the annotations are just a convenient alternative syntax for when you think they'd make the code more readable. Thomas
On 12/4/2012 11:43 AM, Jasper St. Pierre wrote:
A telling moment for me was during an early Py3k keynote at PyCon (perhaps it was in Dallas or Chicago?), Guido couldn't remember the word "annotation," and said, "you know, those things that aren't type declarations?" :-) --Ned.
On Tue, Dec 4, 2012 at 9:12 AM, Ned Batchelder <ned@nedbatchelder.com> wrote:
To the contrary. There were too many use cases that immediately looked important, and we couldn't figure out which ones would be the most important or how to combine them, so we decided to take a two-step approach: in step 1, we designed the syntax, whereas in step 2, we would design the semantics. The idea was very clear that once the syntax was settled people would be free to experiment with different semantics -- just not in the stdlib. The idea was also that eventually, from all those experiments, one would emerge that would be fit for the stdlib. The process was somewhat similar to the way decorators were introduced. In Python 2.3, we introduced things like staticmethod, classmethod and property. But we *didn't* introduce the @ syntax, because we couldn't agree about it at that point. Then, for 2.4, we sorted out the proper syntax, having by then conclusively discovered that the original way of using e.g. classmethod (an assignment after the end of the method definition) was hard on the human reader. (Of course, you may note that for decorators, we decided on semantics first, syntax second. But no two situations are quite the same, and in the case of annotations, without syntax it would be nearly impossible to experiment with semantics.)
Pretty much all of the above to some extent. But for me personally, the main goal was always to arrive at a notation to specify type constraints (and maybe other constraints) for arguments and return values. I've toyed at various times with specific ways of combining types. E.g. list[int] might mean a list of integers, and dict[str, tuple[float, float, float, bool]] might mean a dict mapping strings to tuples of three floats and a bool. But I felt it was much harder to get consensus about such a notation than about the syntax for argument annotations (think about how many objections you can bring in to these two examples :-) -- I've always had a strong desire to use "var: type = default" and to make the type a runtime expression to be evaluated at the same time as the default.
Heh. :-) -- --Guido van Rossum (python.org/~guido)
Check out http://www.artima.com/weblogs/viewpost.jsp?thread=89161 -eric On Tue, Dec 4, 2012 at 9:43 AM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
On Dec 2, 2012, at 3:43 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Admittedly this was long enough ago that I don't remember the details, just the obvious consequence that PEP 8 remains largely silent on the matter, aside from declaring that function annotations are off-limits for standard library modules:
PEP 8 is not "largely silent" on the subject: "'' The Python standard library will not use function annotations as that would result in a premature commitment to a particular annotation style. Instead, the annotations are left for users to discover and experiment with useful annotation styles. Early core developer attempts to use function annotations revealed inconsistent, ad-hoc annotation styles. For example: [str] was ambiguous as to whether it represented a list of strings or a value that could be either str or None. The notation open(file:(str,bytes)) was used for a value that could be either bytes or str rather than a 2-tuple containing a str value followed by a bytesvalue. The annotation seek(whence:int) exhibited an mix of over-specification and under-specification: int is too restrictive (anything with __index__ would be allowed) and it is not restrictive enough (only the values 0, 1, and 2 are allowed). Likewise, the annotation write(b: bytes) was also too restrictive (anything supporting the buffer protocol would be allowed). Annotations such as read1(n: int=None) were self-contradictory since None is not an int. Annotations such as source_path(self, fullname:str) -> objectwere confusing about what the return type should be. In addition to the above, annotations were inconsistent in the use of concrete types versus abstract types: int versus Integral and set/frozenset versus MutableSet/Set. Some annotations in the abstract base classes were incorrect specifications. For example, set-to-set operations require other to be another instance of Setrather than just an Iterable. A further issue was that annotations become part of the specification but weren't being tested. In most cases, the docstrings already included the type specifications and did so with greater clarity than the function annotations. In the remaining cases, the docstrings were improved once the annotations were removed. The observed function annotations were too ad-hoc and inconsistent to work with a coherent system of automatic type checking or argument validation. Leaving these annotations in the code would have made it more difficult to make changes later so that automated utilities could be supported. ''' Raymond
On Mon, Dec 3, 2012 at 6:09 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:
It's effectively silent on the matters at hand, which are: * the advisability of using annotations without an associated decorator that makes the interpretation currently in play explicit (while the examples given do illustrate why *not* doing this is a bad idea, it doesn't explicitly state that conclusion, merely "we're not going to use them in the standard library at this point") * the advisability of providing a pure annotations API, without any fallback to an explicit decorator factory * the advisability of handling composition within the annotations themselves, rather than by falling back to explicit decorator factories * the advisability of using the __annotations__ dictionary for long-term introspection, rather than using the decorator to move the information to a purpose-specific location in a separate function attribute I would be *quite delighted* if people are open to the idea of making a much stronger recommendation along the following lines explicit in PEP 8: ================== * If function annotations are used, it is recommended that: * the annotation details should be designed with a specific practical use case in mind * the annotations are used solely as a form of syntactic sugar for passing arguments to a decorator factory that would otherwise accept explicit per-parameter arguments * the decorator factory name should provide the reader of the code with a strong hint as to the intended meaning of the parameter annotations (or at least a convenient reference point to look up in the documentation) * in simple cases, using parameter and return type annotations will then allow the per-parameter details to be mapped easily by both the code author and later readers without requiring repetition of parameter names or careful alignment of factory arguments with parameter positions. * the explicit form remains available to handle more complex situations (such as applying multiple decorators to the same function) without requiring complicated conventions for composing independent annotations on a single function ================== In relation to the last point, I consider composing annotations to be analogous to composing function arguments. Writing: @g @f def annotated(arg1: (a, x), arg2: (b, y), arg3: (c, z)): ... instead of the much simpler: @g(x, y, z) @f(a, b, c) def annotated(arg1, arg2, arg3): ... is analagous to writing: args = [(a, x), (b, y), (c, z)] f(*(x[0] for x in args)) g(*(x[1] for x in args)) instead of the more obvious: f(a, b, c) g(x, y, z) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Dec 3, 2012 at 9:45 PM, Andrew Svetlov <andrew.svetlov@gmail.com>wrote:
I don't quite understand that comment - PEP 362 is purely an access mechanism. The underlying storage is still in __annotations__ (at least as far any annotations are concerned). However, using separate storage is a natural consequence of also providing an explicit decorator factory API, so I didn't bring it up. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Dec 03, 2012, at 09:08 PM, Nick Coghlan wrote:
I would be *quite delighted* if people are open to the idea of making a much stronger recommendation along the following lines explicit in PEP 8:
I am -1 for putting any of what followed in PEP 8, and in fact, I think the existing examples at the bottom of PEP 8 are inappropriate. PEP 8 should be prescriptive of explicit Python coding styles. Think "do this, not that". It should be as minimal as possible, and in general provide rules that can be easily referenced and perhaps automated (e.g. pep8.py). Some of the existing text in PEP 8 already doesn't fall under that rubric, but it's close enough (e.g. designing for inheritance). I don't think annotations reach the level of consensus or practical experience needed to be added to PEP 8. OTOH, I wouldn't oppose a new informational PEP labeled "Annotations Best Practices", where some of these principles can be laid out and explored. Cheers, -Barry
Hm. I agree PEP 8 seems an odd place for Nick's recommendation. Even if I were to agree with hos proposal I would think it belongs in a different PEP than PEP 8. But personally I haven't given up on using annotations to give type hints -- I think it can at some times be a useful augmentation to static analysis (whose use I see mostly as an aid to human readers and/or tools like linters, IDEs, and refactoring tools, not for guiding compiler optimizations). I know of several projects (both public and private) for improving the state of the art of Python static analysis with this goal in mind. With the advent of e.g. TypeScript and Dart in the JavaScript world, optional type annotations for dynamic languages appear to be becoming more fashionable, and maybe we can get some use out of them. FWIW, as far as e.g. 'int' being both overspecified and underspecified: I don't care about the underspecification so much, that's always going to happen; and for the overspecification, we can either use some abstract class instead, or simply state that the occurrence of certain concrete types must be taken as a shorthand for a specific abstract type. This could be part of the registration call of the concrete type, or something. Obviously this would require inventing and standardizing notations for things like "list of X", "tuple with items X, Y, Z", "either X or Y", and so on, as well as a standard way of combining annotations intended for different tools. *This* would be a useful discussion. What to do in the interim... I think the current language in PEP 8 is just fine until we have a better story. --Guido On Mon, Dec 3, 2012 at 7:34 AM, Barry Warsaw <barry@python.org> wrote:
-- --Guido van Rossum (python.org/~guido)
So long as any type hinting semantics are associated with a "@type_hints" decorator, none of those ideas conflict with my suggestions for good annotation usage practices. The explicit decorators effectively end up serving as dialect specifiers for the annotations, for the benefit of other software (by moving the metadata out to purpose specific attributes) and for readers (simply by being present). Anyway, the reactions here confirmed my recollection of a lack of consensus amongst the core team. I'll just put something up on my own site, instead. Cheers, Nick. -- Sent from my phone, thus the relative brevity :) On Dec 4, 2012 3:28 AM, "Guido van Rossum" <guido@python.org> wrote:
Just thought of a couple of usages which don't fit into the decorator model. The first is using the return annotation for early binding: def func(seq) -> dict(sorted=sorted): return func.__annotations__['return']['sorted'](seq) Stangely enough, this seems to run slightly faster than def func(seq, sorted=sorted): return sorted(seq) My test shows the first running in about 0.376s and the second in about 0.382s (python 3.3, 64bit). The second is passing information to base classes. This is a rather contrived example which could easily be solved (better) in plenty of other ways, but it does illustrate a pattern which someone else may be able to turn into a genuine use case. class NumberBase: def adjust(self, value): return self.adjust.__annotations__['return'](value) class NegativeInteger(NumberBase): def adjust(self, value) -> int: return super().adjust(-value)
Cheers David On Tue, Dec 4, 2012 at 1:02 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Dec 4, 2012 at 1:37 AM, David Townshend <aquavitae69@gmail.com> wrote:
You've got to be kidding...
Surely that's some kind of random variation. It's only a 2% difference.
This looks like a contrived way to use what is semantically equivalent to function attributes. The base class could write def adjust(self, value): return self.adjust.adjuster(value) and the subclass could write def adjust(self, value): return super().adjust(-value) adjust.adjuster = int Or invent a decorator to set the attribute: @set(adjuster=int) def adjust(self, value): return super().adjust(-value) But both of these feel quite awkward compared to just using a class attribute. class NumberBase: def adjust(self, value): return self.adjuster(value) class NegativeInteger(NumberBase): adjuster = int # No need to override adjust() IOW, this is not a line of thought to pursue. -- --Guido van Rossum (python.org/~guido)
On Wed, Dec 5, 2012 at 8:17 PM, Guido van Rossum <guido@python.org> wrote:
It's consistent. I ran several tests and came out with the same 2% difference every time.
IOW, this is not a line of thought to pursue.
I wasn't suggesting that this is a good idea, I was merely trying to point out that there are currently ways of using annotations beyond type declarations with decorators, and that there may be other use cases out there which will work well. Documenting recommendations that annotations only be used with decorators, or only be used for type declarations will limit the possibilities because nobody will bother to look further, and if they do, the ideas will no doubt be shut down as being bad style because they go against the recommended usage. I thought that limiting annotations like this was what you wanted to avoid? Having said that, I've never found a good use for annotations in my own code, so I'm not emotionally invested one way or the other. I do think that the best usage I've seen is exactly what is being discussed here and it would be great if there was some prescribed use for annotations. Perhaps people would actually use them then. David
On 2012-12-03, at 18:27 , Guido van Rossum wrote:
I've always felt that __getitem__ and __or__/__ror__ on type 1. looked rather good and 2. looked similar to informal type specs and type specs of other languages. Although that's the issue with annotations being Python syntax: it requires changing stuff fairly deep into Python to be able to experiment. The most bothersome part is that I "feel" "either X or Y" (aka `X | Y`) should be a set of type (and thus the same as {X, Y}[0]) but that doesn't work with `isinstance` or `issubclass`. Likewise, `(a, b, c)` in an annotation feels like it should mean the same as `tuple[a, b, c]` ("a tuple with 3 items of types resp. a, b and c") but that's at odds with the same type-checking functions. The first could be fixable by relaxing slightly the constraints of isinstance and issubclass, but not so for the second. [0] which works rather neatly for anonymous unions as `|` is the union of two sets, so the arithmetic would be `type | type -> typeset`, `type | typeset -> typeset` and `typeset | typeset -> typeset`, libraries could offer opaque types/typesets which would be composable without their users having to know whether they're type atoms or typesets
On Tue, Dec 4, 2012 at 10:12 AM, Masklinn <masklinn@masklinn.net> wrote:
So, instead of using def foo(a: int, b: str) -> float: <blah> you use from experimental_type_annotations import Int, Str, Float def foo(a: Int, b: Str) -> Float: <blah> And now we're ready for experimentation. [Warning: none of this is particularly new; I've had these things in my brain for years, as the referenced Artima blog post made clear.]
Note that in Python 3 you can override isinstance, by defining __instancecheck__ in the class: http://docs.python.org/3/reference/datamodel.html?highlight=__instancecheck_... So it shouldn't be a problem to make isinstance(42, Int) work. We can also make things like List[Int] and Dict[Str, Float] work, and even rig it so that isinstance([1, 2, 3], List[Int]) == True while isinstance([1, 2, 'booh'], List[Int]) == False Of course there are many bikeshedding topics like whether we should ever write List -- maybe we should write Iterable or Sequence instead, and maybe we have to be able to express mutability, and so on. The numeric tower (PEP 3141) is also good to keep in mind. I think that's all solvable once we start experimenting a bit. Some important issues to bikeshed over: - Tuples. Sometimes you want to say e.g. "a tuple of integers, don't mind the length"; other times you want to say e.g. "a tuple of fixed length containing an int and two strs". Perhaps the former should be expressed using ImmutableSequence[Int] and the second as Tuple[Int, Str, Str]. - Unions. We need a way to say "either X or Y". Given that we're defining our own objects we may actually be able to get away with writing e.g. "Int | Str" or "Str | List[Str]", and isinstance() would still work. It would also be useful to have a shorthand for "either T or None", written as Optional[T] or Optional(T). - Whether to design notations to express other constraints. E.g. "integer in range(10, 100)", or "one of the strings 'r', 'w' or 'a'", etc. You can go crazy on this. - Composability (Nick's pet peeve, in that he is against it). I propose that we reserve plain tuples for this. If an annotation has the form "x: (P, Q)" then that ought to mean that x must conform to both P and Q. Even though Nick doesn't like this, I don't think we should do everything with decorators. Surly, the decorators approach is good for certain use cases, and should take precedence if it is used. But e.g. IDEs that use annotations for suggestions and refactoring should not require everything to be decorated -- that would just make the code too busy. - Runtime enforcement. What should we use type annotations for? IDEs, static checkers (linters) and refactoring tools only need the annotations when they are parsing the code. While it is tempting to invent some kind of runtime checking that automatically checks the actual types against the annotations whenever a function is called, I think this is rarely useful, and often prohibitively slow. So I'd say don't focus on this. Instead, explicit type assertions like "assert isinstance(x, List[Int])" might be used, sparingly, for those cases where we'd otherwise write a manual assertion with the same meaning (which is also sparingly!). A decorator to do this might be useful (especially if there's a separate mechanism for turning actual checking on or off through some configuration mechanism).
I like this for declaring union types. I don't like it for composing constraints that are intended for different tools. -- --Guido van Rossum (python.org/~guido)
Nice, that seems very explicit. ImmutableSequence is long, but clear. In this specific case, should it be just Sequence, and a mutable one would be MutableSequence (to be consistent with collections.abc names?).
Definitely useful to have a notation for "either T or None", as it's a pretty heavily-used pattern. But what about using the same approach, something like "T | None" or "T | NoneType". Though if you use the real None rather than experimental_type_annotations.None, is that confusing? In any case, it seems unnecessary to have a special Optional(T) notation when you've already got the simple "T1 | T2" notation.
Yes, I think this is dangerous territory -- it could get crazy very fast. Statically typed languages don't have this. Then again, I guess type annotations have the potential to be *more* powerful in this regard. Still, it'd have to be an awfully nice and general notation for it to be useful. Even then, your "def" line complete with type/constraint annotations may get far too long to be readable... -Ben
On Wed, Dec 5, 2012 at 11:22 AM, Guido van Rossum <guido@python.org> wrote:
Optional is not the same as "or None" to me: Dict(a=Int, b=Int | None, c=Optional(Int)) suggests that b is required but might be None while c is not required, i.e., {'a': 3, b: None} is allowed while {'a': 3, c: None} is not. Ditto for Tuples: Tuple[Int, Str | None, Optional(Int)] where (3, None) matches as does (3, 'a', 4) but not (3, None, None). Optionals might be restricted to the end as matching in the middle would be complicated and possibly error-prone: Tuple[Int, Optional(Int | None), Int | Str, Int | None] --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
On Wed, Dec 5, 2012 at 3:01 PM, Guido van Rossum <guido@python.org> wrote:
Those are not the semantics I had in mind for Optional.
I know that. My point was that the standard meaning of the word optional is that something may or may not be given (or whatever the applicable verb is). That's quite different from saying it must be provided but may be None. Since you invited a bit of bikeshedding, I felt it was appropriate to point that out and then I got distracted by discussing the alternative that you weren't talking about. Sorry that was confusing. In C#, this is called Nullable and you can write Nullable<String> to indicate the type (String or null type). The shorthand for that is String?. If you want a shorthand to specify that None is allowed, I'd suggest ~Str. --- Bruce P.S. Optional[T] is not literally a shorthand for T | None as the former is 11 characters and the latter is 10 characters even if we include and count the spaces. :-) P.P.S. I don't think Str | None rather than Str | NoneType is confusing.
On 2012-12-05, at 20:22 , Guido van Rossum wrote:
My problem there was more about having e.g. Int | Float return a set, but isinstance not working with a set. But indeed it could return a TypeSet which would implement __instancecheck__.
Well if `|` is the "union operator", as Ben notes `T | None` works well, is clear and is sufficient. Though that's if and only if "Optional[T]" is equivalent to "T or None" which Bruce seems to disagree with. There's some history with this pattern: http://journal.stuffwithstuff.com/2010/08/23/void-null-maybe-and-nothing/ (bottom section, from "Or Some Other Solution")
Yes this is going in Oleg territory, a sound core is probably a good starting idea. Although basic enumerations ("one of the strings 'r', 'w' or 'a'") could be rather neat.
For IDEs, that's pretty much all the time though, either they're parsing the code or they're trying to perform static analysis on it, which uses the annotations.
Could be useful for debug or testing runs though, in the same way event-based profilers are prohibitively slow and can't be enabled all the time but are still useful. Plus it might be possible to enable/disable this mechanism with little to no source modification via sys.setprofile (I'm not sure what hooks it provides exactly and the documentation is rather sparse, so I'm not sure if the function object itself is available to the setprofile callback, looking at Lib/profiler.py it might only get the code object).
On Wed, Dec 5, 2012 at 12:34 PM, Masklinn <masklinn@masklinn.net> wrote:
Right, that's what I meant.
Actually, I find "T|None" somewhat impure, since None is not a type but a value. If you were allow this, what about "T|False"? And then what about "True|None"? (There's no way to make the latter work!) And I think "T|NoneType" is obscure; hence my proposal of Optional(T). (Not Optional[T], since Optional is not a type.)
Yeah, they're parsing it, but they're not executing it.
Hence my idea of using a decorator to enable this on specific functions. -- --Guido van Rossum (python.org/~guido)
On Thu, Dec 6, 2012, at 3:43, Masklinn wrote:
Why would Optional not be a type? It's coherent with Option or Maybe types in languages with such features, or C#'s Nullable.
C#'s Nullable doesn't really work outside a static typing system - when you assign a Nullable to an 'object' or a 'dynamic', you get either the original type (e.g. Int32) or a null reference (which has no type). It's a real type only as far as the static typing system goes: it can be the type of a field or a local variable, it _cannot_ be the type of an object on the heap. And since python doesn't have static typing...
On Thu, Dec 6, 2012 at 5:22 AM, Guido van Rossum <guido@python.org> wrote:
I'm not against using composition within a particular set of annotation semantics, I'm against developing a convention for arbitrary composition of annotations with *different* semantics. Instead, I'm advocating for the following guidelines to avoid treading on each others toes when experimenting with annotations and to leave scope for us to define standard annotation semantics at a future date: 1. Always use a decorator that expresses the annotation semantics in use (e.g. tab completion, type descriptions, parameter documentation) 2. Always *move* the annotations out to purpose-specific storage as part of the decorator (don't leave them in the annotations storage) 3. When analysing a function later, use only the purpose-specific attribute(s), not the raw annotations storage 4. To support composition with other sets of annotation semantics, always provide an alternate API that accepts the per-parameter details directly (e.g. by name or index) rather than relying solely on the annotations The reason for this is so that if, at some future point in the time, python-dev agrees to bless some particular set of semantics as *the* meaning of function annotations (such as the type hinting system being discussed), then that won't break anything. Otherwise, if people believe that it's OK for them to simply assume that the contents of the annotations mean whatever they mean for their particular project, then it *will* cause problems further down the road as annotations written for one set of semantics (e.g. tab completion, parameter documentation) get interpreted by a processor expecting different semantics (e.g. type hinting). Here's how your example experiment would look under such a scheme: from experimental_type_annotations import type_hints, Int, Str, Float # After type_hints runs, foo.__annotations__ would be empty, and the type # hinting data would instead be stored in (e.g.) a foo._type_hints attribute. @type_hints def foo(a: Int, b: Str) -> Float: <blah> This is then completely clear and unambigious: - readers can see clearly that these annotations are intended as type hints - the type hinting processor can see that there *is* type hinting information available, due to the presence of a _type_hints attribute - other automated processors see that there are no "default" annotations (which is good, since there is currently no such thing as "default" annotation semantics) Furthermore, (as noted elsewhere in the thread) an alternate API can then easily be provided that supports composition with other annotations: @type_hints(Int, Str, _return=Float) def foo(a, b): <blah> Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Nick, I understand your position completely (and I did before). I just disagree. :-) I think that requiring the experiment I am proposing to use a decorator on each function that uses it (rather than just an import at the top of the module) will cause too much friction, and the experiment won't get off the ground. That's why I am proposing a universal composition convention: When an annotation for a particular argument is a tuple, then any framework or decorator that tries to assign meanings to annotations must search the items of the tuple for one that it can understand. For the experimental type annotation system I am proposing this should be simple enough -- the type annotation system can require that the things it cares about must all be subclasses of a specific base class (let's call it TypeConstraint). If the annotation is not a tuple, it should be interpreted as a singleton tuple. Yes, it is possible that a mistake leaves an annotation unclaimed. But that's no worse than currently, where all annotations are ignored. And for TypeConstraint there is no runtime behavior anyway (unless you *do* add a decorator) -- its annotations are there for other tools to parse and interpret. It's like pylint directives -- if you accidentally misspell it 'pylnt' you get no error (but you may still notice that something's fishy, because when pylint runs it doesn't suppress the thing you tried to suppress :-). --Guido On Wed, Dec 5, 2012 at 9:27 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On Wed, Dec 5, 2012 at 9:22 PM, Guido van Rossum <guido@python.org> wrote:
Just to note: there are https://github.com/Deepwalker/trafaret library intended for checking on complex enough structures.
participants (17)
-
Andrew Svetlov -
Antoine Pitrou -
Barry Warsaw -
Ben Hoyt -
Bruce Leban -
David Townshend -
Eric Snow -
Guido van Rossum -
Jasper St. Pierre -
Masklinn -
Ned Batchelder -
Nick Coghlan -
random832＠fastmail.us -
Raymond Hettinger -
Robert McGibbon -
Steven D'Aprano -
Thomas Kluyver