Function annotations (PEP 3107) are a very interesting new feature, but so far have gone largely unused. The only project I've seen using them is plac, a command-line option parser. One reason for this is that because function annotations can be used to mean anything, we're wary of doing anything in case we interfere with some other use case. A recent thread on ipython-dev touched on this [1], and we'd like to suggest some conventions to make annotations useful for everyone. 1. Code inspecting annotations should be prepared to ignore annotations it can't understand. 2. Code creating annotations should use wrapper classes to indicate what the annotation means. For instance, we are contemplating a way to specify options for a parameter, to be used in tab completion, so we would do something like this: from IPython.core.completer import options def my_io(filename, mode: options('read','write') ='read'): ... 3. There are a couple of important exceptions to 2: - Annotations that are simply a string can be used like a docstring, to be displayed to the user. Inspecting code should not expect to be able to parse any machine-readable information out of these strings. - Annotations that are a built-in type (int, str, etc.) indicate that the value should always be an instance of that type. Inspecting code may use these for type checking, introspection, optimisation, or other such purposes. Note that for now, I have limited this to built-in types, so other types can be used for other purposes, but this could be extended. For instance, the ABCs from collections (collections.Mapping et al.) could well be added to this category. 4. There should be a convention for attaching multiple annotations to one value. I propose that all code using annotations expects to handle tuples/lists of annotations. (We also considered dictionaries, but the result is long and ugly). So in this definition: def my_io(filename, mode: (options('read','write'), str, 'The mode in which to open the file') ='read'): ... the mode parameter has a set of options (ignored by frameworks that don't recognise it), should always be a string, and has a description. Any thoughts and suggestions are welcome. As an aside, we may also create a couple of decorators to fill in __annotations__ on Python 2, something like: @return_annotation('A file obect') @annotations(mode=(options('read','write'), str, 'The mode in which to open the file')) def my_io(filename, mode='read'): ... [1] http://mail.scipy.org/pipermail/ipython-dev/2012-November/010697.html Thanks, Thomas
I think code related to annotations is tightly coupled with annotated function usage context (decorator, metaclass, function caller). So annotation really can mean anything and it depends from context. I don't see use case when context need to ignore unexpected annotation. In my practice annotation is always expected if specified, absence of annotation for parameter is mark to do nothing with it (it can be allowed or disabled depending of context requirements). The same for multiple annotations. If your context allow it — that's up to you. Exact kind of composition to use depends from context — it can be tuple, dict, user-defined composition object. My point is: we dont need to restrict annotations in any way. If some libraries want to share annotations that means they are tightly enough coupled and can make rules for itself. All other code can go in the wild. On Sat, Dec 1, 2012 at 2:28 PM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
-- Thanks, Andrew Svetlov
I think annotations are potentially very useful for things like introspection and static analysis. For instance, your IDE could warn you if you pass a parameter that doesn't match the type specified in an annotation. In these cases, the code reading the annotations isn't coupled with the function definitions. I'm not aiming to restrict annotations, just to establish some conventions to make them useful. We have a convention, for instance, that attributes with a leading underscore are private. That's a useful basis that everyone understands, so when you do obj.<tab> in IPython, it doesn't show those attributes by default. I'd like to have some conventions of that nature around annotations. Thomas On 1 December 2012 14:59, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
On Sun, Dec 2, 2012 at 2:30 AM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
show those attributes by default. I'd like to have some conventions of that
nature around annotations.
Indeed, composability is a problem with annotations. I suspect the only way to resolve this systematically is to adopt a convention where annotations are used *strictly* for short-range communication with an associated decorator that transfers the annotation details to a *different* purpose-specific location for long-term introspection. Furthermore, if composability is going to be possible in general, annotations can really *only* be used as a convenience API, with an underlying API where the necessary details are supplied directly to the decorator. For example, here's an example using the main decorator API for a cffi callback declaration [1]: @cffi.callback("int (char *, int)"): def my_cb(arg1, arg2): ... The problem with this is that it can get complicated to map C-level types to parameter names as the function signature gets more complicated. So, what you may want to do is write a decorator that builds the CFFI signature from annotations on the individual parameters: @annotated_cffi_callback def my_cb(arg1: "char *", arg2: "int") -> "int": ... The decorator would turn that into an ordinary call to cffi.callback, so future introspection wouldn't look at the annotations mapping at all, it would look directly at the CFFI metadata. Annotations should probably only ever be introspected by their associated decorator, and if you really want to apply multiple decorators with annotation support to a single function, you're going to have to fall back to the non-annotation based API for at least some of them. Once you start trying to overload the annotation field with multiple annotations, the readability gain for closer association with the individual parameters is counterbalanced by the loss of association between the subannotations and their corresponding decorators. [1] http://cffi.readthedocs.org/en/latest/#callbacks Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
This proposal looks great. The only thing is that I don't understand the point of annotations in the first place, since Python has decorators. As the last part of your post describes, decorators can be used to do the same thing. With decorators, it is even possible to use annotation-like syntax: def defaults_as_parameter_metadata(f): names, args_name, kwargs_name, defaults = inspect.getargspec(f) assert len(names) == len(defaults) # To keep this example simple... f.parameter_metadata = {} for name, meta in zip(names, defaults): f.parameter_metadata[name] = meta f.__defaults__ = () # Again, for simplicity. return f @defaults_as_parameter_metadata def make_ice_cream(flavor=(options('vanilla', 'chocolate', ...), str, "What kind of delicious do you want?"), quantity=(positive, double, "How much (in pounds) do you want?")): ... I know this addresses a different issue, but I was directed to this thread from an answer that I got on StackOverflow<http://stackoverflow.com/questions/13784713/what-good-are-python-function-an...>, and this thread seems related enough. Sorry if I'm going off the rails here. On Saturday, December 1, 2012 4:28:50 AM UTC-8, Thomas Kluyver wrote:
I think code related to annotations is tightly coupled with annotated function usage context (decorator, metaclass, function caller). So annotation really can mean anything and it depends from context. I don't see use case when context need to ignore unexpected annotation. In my practice annotation is always expected if specified, absence of annotation for parameter is mark to do nothing with it (it can be allowed or disabled depending of context requirements). The same for multiple annotations. If your context allow it — that's up to you. Exact kind of composition to use depends from context — it can be tuple, dict, user-defined composition object. My point is: we dont need to restrict annotations in any way. If some libraries want to share annotations that means they are tightly enough coupled and can make rules for itself. All other code can go in the wild. On Sat, Dec 1, 2012 at 2:28 PM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
-- Thanks, Andrew Svetlov
I think annotations are potentially very useful for things like introspection and static analysis. For instance, your IDE could warn you if you pass a parameter that doesn't match the type specified in an annotation. In these cases, the code reading the annotations isn't coupled with the function definitions. I'm not aiming to restrict annotations, just to establish some conventions to make them useful. We have a convention, for instance, that attributes with a leading underscore are private. That's a useful basis that everyone understands, so when you do obj.<tab> in IPython, it doesn't show those attributes by default. I'd like to have some conventions of that nature around annotations. Thomas On 1 December 2012 14:59, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
On Sun, Dec 2, 2012 at 2:30 AM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
show those attributes by default. I'd like to have some conventions of that
nature around annotations.
Indeed, composability is a problem with annotations. I suspect the only way to resolve this systematically is to adopt a convention where annotations are used *strictly* for short-range communication with an associated decorator that transfers the annotation details to a *different* purpose-specific location for long-term introspection. Furthermore, if composability is going to be possible in general, annotations can really *only* be used as a convenience API, with an underlying API where the necessary details are supplied directly to the decorator. For example, here's an example using the main decorator API for a cffi callback declaration [1]: @cffi.callback("int (char *, int)"): def my_cb(arg1, arg2): ... The problem with this is that it can get complicated to map C-level types to parameter names as the function signature gets more complicated. So, what you may want to do is write a decorator that builds the CFFI signature from annotations on the individual parameters: @annotated_cffi_callback def my_cb(arg1: "char *", arg2: "int") -> "int": ... The decorator would turn that into an ordinary call to cffi.callback, so future introspection wouldn't look at the annotations mapping at all, it would look directly at the CFFI metadata. Annotations should probably only ever be introspected by their associated decorator, and if you really want to apply multiple decorators with annotation support to a single function, you're going to have to fall back to the non-annotation based API for at least some of them. Once you start trying to overload the annotation field with multiple annotations, the readability gain for closer association with the individual parameters is counterbalanced by the loss of association between the subannotations and their corresponding decorators. [1] http://cffi.readthedocs.org/en/latest/#callbacks Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
This proposal looks great. The only thing is that I don't understand the point of annotations in the first place, since Python has decorators. As the last part of your post describes, decorators can be used to do the same thing. With decorators, it is even possible to use annotation-like syntax: def defaults_as_parameter_metadata(f): names, args_name, kwargs_name, defaults = inspect.getargspec(f) assert len(names) == len(defaults) # To keep this example simple... f.parameter_metadata = {} for name, meta in zip(names, defaults): f.parameter_metadata[name] = meta f.__defaults__ = () # Again, for simplicity. return f @defaults_as_parameter_metadata def make_ice_cream(flavor=(options('vanilla', 'chocolate', ...), str, "What kind of delicious do you want?"), quantity=(positive, double, "How much (in pounds) do you want?")): ... I know this addresses a different issue, but I was directed to this thread from an answer that I got on StackOverflow<http://stackoverflow.com/questions/13784713/what-good-are-python-function-an...>, and this thread seems related enough. Sorry if I'm going off the rails here. On Saturday, December 1, 2012 4:28:50 AM UTC-8, Thomas Kluyver wrote:
participants (4)
-
Andrew Svetlov -
Daniel Wong -
Nick Coghlan -
Thomas Kluyver