On 20 August 2014 17:26, Ethan Furman <ethan@stoneleaf.us> wrote:

On 08/20/2014 09:04 AM, Guido van Rossum wrote:

...

I expect that we'll find a way for type and other annotations to coexist [...]

Another thing to keep in mind, and which my scription utility will probably switch to, is that we are not stuck only with __annotations__ -- we can also add other attributes, such as __scription__ or __plac__ or __my_custom_stuff__ or whatever.

It could easily be that coexisting means use a decorator instead of annotations, and store the info in a different attribute. As a bonus this method is even 2.x compatible.

The obvious thought is, if a decorator is a sufficiently good notation, is there any need for annotations at all (even for static typing)? But that argument does seem to imply that using annotations for anything *other* than static typing is pointless. (As in, has more disadvantages than advantages). I wonder if that's the real issue here? Paul

On Wed, Aug 20, 2014 at 12:43 PM, Ethan Furman <ethan@stoneleaf.us> wrote:

What can one do with an annotation that one cannot do with a decorator?

Spell "returns" in two characters.

On 20 August 2014 18:37, Andrea Censi <censi@mit.edu> wrote:

What can one do with a decorator that cannot be done with annotations?

Python 2 compatibility. Paul PS This discussion is leading me to feel that annotations aren't really that useful compared to a well-written decorator, so why *not* leave them specifically for static type information, as Guido suggested...

Let's add convenient namespacing to that list: @annotate('typing', one=str, two=int, three=int) @annotate('cli', two=int, three='t') def func(one, two, *, three=5): ... I might add something like this to sigtools.modifiers.annotate and .specifiers.signature to extract it.

On Wed, Aug 20, 2014 at 05:31:50PM +0100, Paul Moore wrote:

The obvious thought is, if a decorator is a sufficiently good notation, is there any need for annotations at all (even for static typing)?

Decorators are *not* sufficiently good notation. Decorators have many nice uses, but they are second class for this purpose since they require you to write the parameters twice: once in the decorator, and once in the actual function declaration. They also visually separate the parameter from its value. If you're thinking about editing code with one or two such functions, with simple parameter declarations, it's no big deal. They're not *very* far away, a line or two maybe, it's easy to cope. But as the number of parameters increases, or the complexity of the type declaration increases, you get code like this: @annotate(fe=Str[blah blah blah blah]) @annotate(fi=List[blah blah blah blah]) @annotate(fo=Dict[blah blah blah blah]) @annotate(fum=Tuple[blah blah blah blah]) @returns(int) def spam(fe=aaa, fi=bbb, fo=ccc, fum=ddd): ... and now you're looking at a solid wall of annotations, and the distance between the first decorator and the parameter list isn't "a line or two" any more. Or you get something like this, which may be a bit better: @annotate(fe=Str[blah blah blah blah], fi=List[blah blah blah blah], fo=Dict[blah blah blah blah], fum=Tuple[blah blah blah blah]) @returns(int) def spam(fe=aaa, fi=bbb, fo=ccc, fum=ddd): ... but still separates the type of fe from the declaration of fe by four lines. With annotations, everything[1] stays together: def spam(fe:Str[blah blah blah blah]=aaa, fi:List[blah blah blah blah]=bbb, fo:Dict[blah blah blah blah]=ccc, fum:Tuple[blah blah blah blah]=ddd, )->int: ... I'm going to take the liberty of quote Nick from an earlier message: [quote] I once had the "pleasure" of inheriting some code written in K&R style C, where the parameter type declarations were separate from the signature line: void foo(a, b, c) double a; char b; { ... } ANSI C, with inline typing, is far more readable :) [end quote] C programmers, hands up if you prefer to use K&R style declarations? I expect you will be in a very small minority. There is a reason why most languages with type declarations normally put them together with the parameter declarations. If anyone wishes to argue for decorator style *in preference* to annotations, a good place to start is with a list of programming languages which use similar syntax for their type declarations. [1] Not quite, you still have to write documentation separately, but since the documentation for a single parameter might be an entire paragraph of text, we don't want to put that in the parameter list. Some problems are just intractable. -- Steven

On Aug 24, 2014, at 7:39 AM, Ed Kellett <edk141@gmail.com> wrote:

I have a few questions:

- How often do multiple kinds of annotation end up on the same function?

In this thread, we've already talked about type checkers and documentation generators, both of which can use the __annotations__ dictionary legitimately. Now imagine that you are an end user that has installed a documentation generator and a static type analyzer. If both tools were to use the __annotations__ dictionary, then right now you could choose one or the other tool, but not both. However, if the standard I'm proposing was adopted, then each tool would choose its own UUID as its key, which would mean they could share entries in the annotations dictionary.

- Why UUIDs (rather than for e.g. PyPI packages, or some other namespace)?

Multiple reasons: - Using PyPI means that every programmer that is trying to follow the standard (including anyone who is just learning python) will create some name while practicing. That name will need to be pushed up to PyPI to ensure uniqueness. Since most of these names are only for learning, PyPI will immediately get flooded with a bunch of project names that are probably going to be abandoned almost immediately (I'm thinking of beginning programming classes especially). This would significantly degrade the utility of PyPI, which I want to avoid. All similar centrally-managed systems will suffer from the same problem. UUIDs don't have this problem; create and abandon them at will. - All other namespace systems will either suffer from the possibility of collisions, require a centrally managed repository of names, or will eventually reinvent UUIDs. We already have UUIDs, lets skip the first couple of headaches and just solve the problem. - Centrally managed systems have a much higher barrier to entry than simple UUIDs. Getting a new UUID to experiment with is trivial; "import uuid; uuid.uuid4()" is our complete program, requires no management on the part of PyPI (or any other third party), doesn't require internet access, etc. - UUIDs have no built-in human significance; it is VERY unlikely that multiple projects will accidentally choose the same UUID. E.g., it is likely that programmers developing different type checkers would choose 'type checker' as a key, and each project will have incompatible meanings/values for the 'type checker' key. This doesn't sound too bad until you start pulling in multiple frameworks from different sources, each of which uses a different, mutually incompatible type checker system. At that point, running any type checker will cause a crash as the type checker tries to read information from the frameworks you've just pulled in. - Google for a UUID. Any UUID. If you've just generated the UUID, you are unlikely to get one come up. Now google for 'UUID('f9bbc165-d904-4452-b858-fc5c9f104c87')'. I've just added it to the README for my annotizer project, and I expect googlebot to pick it up in the next few days. At that point, the only two places you should find mention of that UUID is on github, and in this thread. If you google for 'type checker', etc., how many hits do you get? How many of them relate to mypy, or even this thread? Once people are used to the standard, they'll know that to find out information about what project is associated with a given UUID they just need to google for it. This is a big win. Actually, do this as an experiment: don't look at the URL below, instead, wait a few days and google for the UUID above. See what comes up. - Making a standard for __annotations__ at this point isn't easy; we need a simple way of deciding if someone is complying the the standard. This is pretty easy if we adopt some UUID as a required key as I mentioned earlier.

- What is the point? A decorator could process the annotations and put some information in func._projectname__something instead of doing the UUID dance

Again, this isn't for consumption within a project, it is for users across projects. What if I want to use Sphinx (http://sphinx-doc.org/) and mypy (http://www.mypy-lang.org/) at the same time in my project? What happens in the following code? """ @sphinx_decorator(a, "Some documentation about a") @mypy_decorator(a, int)) def foo(a): pass """ Is it the same as: """ @mypy_decorator(a, int)) @sphinx_decorator(a, "Some documentation about a") def foo(a): pass """ Right now, as I understand it, the last applied decorator would win, which means 'func._projectname__something' would be set to either sphinx or mypy. That means that order matters for completely orthogonal concepts. This is bad. UUIDs solve this, and all the earlier problems.

- What would pydoc print for the function signature?

As I mentioned earlier, certain UUIDs might become de facto or de jure standards. In this case, projects that have common goals could settle on a common standard and publish a common UUID. Pydoc would know about these UUIDs (they would be published), and would know what to do for them. For UUIDs it doesn't understand, it could raise a warning, or simply ignore them. Before you take my comments above as proof the we don't need UUIDs, consider the fact that we are currently discussing type systems, and our thoughts may change in the future. I don't mean that there will be successive standards, I mean that there may be competing standards, at least until we really know what the best one will be. This is a case where creating and abandoning UUIDs will be trivial, but where using 'type checker' is going to lead to confusion.

On 24 August 2014 03:18, Cem Karan <cfkaran2@gmail.com> wrote:

...

On Aug 20, 2014, at 11:08 AM, Paul Moore <p.f.moore@gmail.com> wrote:

...

Sigh. I go away for a week and come back to a mega-thread I can never hope to catch up on :-)

TL; DR; - Although mypy looks interesting, I think it's too soon to close the door on all other uses of annotations. Let's find a solution that allows exploration of alternative uses for a while longer.

OK, can I just make some points regarding the static typing thread. First of all, I have no issue with the idea of static typing, and in fact I look forward to seeing what benefits it might have (if nothing else, the pointer to mypy, which I'd never heard of before, is appreciated). It won't be something I use soon (see below) but that's fine.

But Guido seems to be saying (on a number of occasions) that nobody is really using annotations, so he wants to focus on the static typing use case alone. I think this is a mistake. First of all, I see no reason why functions using typing annotations could not be introduced with a decorator[1]. So why insist that this is the *only* use of annotations, when it's pretty easy to allow others to co-exist?

Also, the "nobody is using annotations" argument? Personally, I know of a few other uses:

1. Argument parsers - at least 3 have been mentioned in the thread. 2. Structure unpacking - I think there is a library that uses annotations for this, although I may be wrong. 3. FFI bindings. I know I've seen this discussed, although I can't find a reference just now.

There are probably other ideas around as well (GUI bindings, documentation generation, ...) None are particularly mature, and most are just at the "ideas" stage, but typically the ideas I have seen are the sort of thing you'd write a library for, and Python 3 only libraries *really* aren't common yet.

The problem for people wanting to experiment with annotations, is that they need to be writing Python 3 only code. While Python 3 adoption is growing rapidly, I suspect that large applications are typically still focused on either going through, or tidying up after, a 2-3 migration. And new projects, while they may be developed from the ground up using Python 3, will typically be using programmers skilled in Python 2, to whom Python 3 features are not yet an "instinctive" part of the toolset. Apart from large standalone applications, there are smaller scripts (which are typically going to be too small to need programming-in-the-large features like annotations) and libraries (which really aren't yet in a position to drop Python 2.x totally, unless they have a fairly small user base).

So I don't see it as compelling that usage of annotations in the wild is not yet extensive.

Rather than close the door on alternative uses of annotations, can I suggest:

1. By all means bless mypy syntax as the standard static typing notation - this seems like a good thing. 2. Clarify that static typing annotations should be introduced with a decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy declaration in the stdlib, and have a registration protocols for tools to hook their own implementation into it.[2] 3. Leave the door open for other uses of decorators, at least until some of the more major libraries drop Python 2.x support completely (and hence can afford to have a dependency on a Python 3 only module that uses annotations). See at that stage if annotations take off. 4. If we get to a point where even libraries that *could* use annotations don't, then revisit the idea of restricting usage to just type information.

Paul

[1] Also, a decorator could allow a Python 2 compatible form by using decorator arguments as an alternative to annotations. [2] I've yet to see a clear explanation of how "a tool using type annotations" like an linter, editor, IDE or Python compiler would use them in such a way that precludes decoration as a means of signalling the annotation semantics.

A long while back I proposed a mechanism for sharing __annotations__ between multiple, non-cooperating projects. The basic idea is that each annotation becomes a dictionary. Each project (and 'project' is a very loosely defined concept here) chooses a UUID that it uses as key into the dictionary. The value is up to the project.

The advantage to this is manifold:

- Annotations can still have multiple uses by different groups without stepping on each other's toes. - If someone wants to make a standard, all they have to do is publish the UUID associated with their standard. For example, we might choose UUID('2cca6238-9fca-4053-aa3d-db9050e6b26b') as the official type information UUID. All projects that want to develop linters, documentation generators, etc., will use that UUID for all annotations, and the PEP will require it. - de facto standards can become de jure standards by blessing a particular UUID. - Guessing if this method is being used is relatively easy; if its a dictionary, and if every key is a UUID, it probably follows this standard. We can tighten it further by requiring some key-value pair be in every dictionary (i.e., {UUID('1ad60d50-8237-4b98-b2b1-69fd08ed575c'):"PEPXXXX"} is always in the dictionary). This makes it fairly simple to add without stomping on what people are already doing. - Finding the standard on the web should also be easy; while you might not find the PEP instantly, you'll probably zoom into it fairly fast.

Disadvantages:

- Typing out UUIDs is PAINFUL. I highly recommend using decorators instead. - Reading the __annotations__ dictionary will be difficult. pprint() should make this easier.

I have working proof-of-concept code at https://github.com/oranguman/annotizer that defines a decorator class that handles the UUID for you. It needs to be extended to parse out information, but it handles the 'other use cases' problem fairly well.

Thanks, Cem Karan

Thanks, Cem Karan

On Sun, Aug 24, 2014 at 09:32:38AM -0400, Cem Karan wrote:

On Aug 24, 2014, at 7:39 AM, Ed Kellett <edk141@gmail.com> wrote:

...

I have a few questions:

- How often do multiple kinds of annotation end up on the same function?

In this thread, we've already talked about type checkers and documentation generators, both of which can use the __annotations__ dictionary legitimately. Now imagine that you are an end user that has installed a documentation generator and a static type analyzer. If both tools were to use the __annotations__ dictionary, then right now you could choose one or the other tool, but not both.

Not necessarily. It depends on what the tools expect to find in the annotations. If type hints become the standard for annotations, which is Guido's proposal, then both tools will expect to find type hints, and both can use them. As for the cases where the tool or library wants to use annotations for something else, I'm quite happy to say that, in those cases, you can't use both. It's like docstrings. Docstrings are for documentation, including the interactive help system. If a library wants to use __doc__ for something else, it can (within the limits that it can only be a string, or None). But a user of that library cannot expect to *both* use the function __doc__ as a docstring *and* for this other purpose at the same time. They have to pick one.

However, if the standard I'm proposing was adopted, then each tool would choose its own UUID as its key, which would mean they could share entries in the annotations dictionary.

I don't think Guido wants to encourage multiple, incompatible uses for function annotations. It is going to be tricky enough to persuade him to merely allow other uses. My feeling is that this is a case of YAGNI. I don't think there are going to be so many competing uses for function annotations in common use that this will be necessary. -- Steven

On 24 August 2014 03:18, Cem Karan <cfkaran2@gmail.com> wrote:

...

Each project (and 'project' is a very loosely defined concept here) chooses a UUID that it uses as key into the dictionary.

Please, not UUIDs. As something meant to be used by humans, they're horrible. -- Greg

On Mon, Aug 25, 2014 at 9:17 AM, Cem Karan <cfkaran2@gmail.com> wrote:

@doc(a, "docs about a") @type(a, set(range(10))) def foo(a): pass

If this is your proposal, then how do UUIDs even help? You already effectively piggy-back off the standard module import system for uniqueness. Surely that's going to do everything that UUIDs would? ChrisA

On Mon, Aug 25, 2014 at 9:28 AM, Cem Karan <cfkaran2@gmail.com> wrote:

On Aug 24, 2014, at 7:22 PM, Chris Angelico <rosuav@gmail.com> wrote:

...

On Mon, Aug 25, 2014 at 9:17 AM, Cem Karan <cfkaran2@gmail.com> wrote:

...

@doc(a, "docs about a") @type(a, set(range(10))) def foo(a): pass

If this is your proposal, then how do UUIDs even help? You already effectively piggy-back off the standard module import system for uniqueness. Surely that's going to do everything that UUIDs would?

I'm not sure if I follow what you're saying. Do you mean 'if everyone is using my annotizer library, why bother with UUIDs, because each decorator will have a different meaning enforced by the import system'?

doc and type can't conflict, because they're separate names. (Though "type" would need to be renamed to avoid a conflict, unless you're suggesting that the type type should grow this functionality.) If they use their own selves as the keys (nobody ever said the keys have to be strings), there cannot be a conflict. The UUID scheme doesn't add anything that can't be done with existing objects with no hassle. ChrisA

On Aug 24, 2014, at 8:21 PM, Andrew Barnert <abarnert@yahoo.com> wrote:

On Aug 24, 2014, at 16:17, Cem Karan <cfkaran2@gmail.com> wrote:

...

On Aug 24, 2014, at 6:17 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

...

...

On 24 August 2014 03:18, Cem Karan <cfkaran2@gmail.com> wrote:

...

Each project (and 'project' is a very loosely defined concept here) chooses a UUID that it uses as key into the dictionary.

Please, not UUIDs. As something meant to be used by humans, they're horrible.

-- Greg

I understand your concerns about them, I really, really do. But can you think of a better way? All my prior points still hold. In my mind, UUIDs are the least bad alternative. We have to educate people as to what the annotations dictionary standard is, and then encourage libraries to use the standard and to hide the UUIDs from end users. My code does this already; if someone wants me to extend it in some way, put in a request, and I will. If the PSF wants it in the standard library, it's theirs! Please, download it and play with it, it's short to the point of almost being trivial (https://github.com/oranguman/annotizer).

Don't let the fact that the bare __annotations__ dict is going to look ugly; if we try to get a really powerful typing system in there, people will probably not want to write the annotations by hand anyways for the simple reason that it will be too hard to read. I mean, which would you rather read?

""" @doc(a, "docs about a") @type(a, set(range(10))) def foo(a): pass

def bar(a: {__docs_project_ID: "docs about a", __types_project_ID: set(range(10))}): pass """

Or:

@doc(a, "docs about a") def foo(a: set[10]): pass

Why does doc have to set an annotation in the first place? Annotations are just syntactic sugar for sticking stuff in a dict attached to the function as an attribute; when that syntactic sugar doesn't make your code more readable, there's no benefit at all to annotations. You could just make the @doc decorator store and find its stuff in a special __docinfo__ attribute instead, and it will look exactly the same to your users.

True, but I was worried about namespace conflicts. More than one documentation library might decide to use the __docinfo__ attribute, which could lead to problems. The __annotations__ dictionary seemed like a good place to store it since it didn't have any predefined meaning (except that now its being proposed that it should have a meaning).

The only question is: which data should be privileges to use the syntactic sugar? That's up to each project to decide; all Guido is suggesting is that we add a default presumption that it's the MyPy-style static types that get to use annotations.

As a side note, I'm not sure how your decorator syntax is going to work unless you've predefined a global named "a" with some kind of value that can be used to match parameters named "a". While there might be some way to do that dynamically (create a module subclass with a custom __dict__ that implements __missing__, and an import hook that applies that to your modules?), it seems pretty hacky, and doesn't buy you that much.

My code seems to work correctly (see https://github.com/oranguman/annotizer/blob/master/annotizer/annotizer.py) via some hackery involving inspect.signature(), but please feel free to take a look at it and tell me if I should do something different. Thanks, Cem Karan