Re: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
On Aug 13, 2014, at 12:44 PM, python-ideas-request@python.org wrote:
The goal is to make it possible to add type checking annotations to 3rd party modules (and even to the stdlib) while allowing unaltered execution of the program by the (unmodified) Python 3.5 interpreter.
Is the goal to "make it possible" or would it quickly become required (i.e. any time you write normal, readable Python, it would break someone's optimizer, refactorer, linter, etc.?) Are these annotations going to be pushed into every nook and cranny in the standard library and wend their way into the C code? Raymond
On Wed, Aug 13, 2014 at 1:18 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:
On Aug 13, 2014, at 12:44 PM, python-ideas-request@python.org wrote:
The goal is to make it possible to add type checking annotations to 3rd party modules (and even to the stdlib) while allowing unaltered execution of the program by the (unmodified) Python 3.5 interpreter.
Is the goal to "make it possible" or would it quickly become required (i.e. any time you write normal, readable Python, it would break someone's optimizer, refactorer, linter, etc.?)
Whoa, whoa. That's not at all the idea. Currently *nobody* uses type annotations because there's no standard notation. My goal is to enable their use by proposing a standard, nothing more.
Are these annotations going to be pushed into every nook and cranny in the standard library and wend their way into the C code?
There's no need for that. Mypy lets you create stub modules that shadow stdlib (and other) modules during typechecking, and it comes with a modest set of standard stubs for some of the most popular stdlib modules. In most cases it's actually much more effective to create a stub than to add the annotations to the stdlib source code -- the stdlib code itself often is difficult to type check due to various optimizations or backwards compatibility concerns, but writing stubs is relatively straightforward, and the stubs will give useful guidance to users of the stdlib who care to run mypy over their own code. -- --Guido van Rossum (python.org/~guido)
On Aug 13, 2014, at 2:09 PM, Guido van Rossum
The goal is to make it possible to add type checking annotations to 3rd party modules (and even to the stdlib) while allowing unaltered execution of the program by the (unmodified) Python 3.5 interpreter.
Is the goal to "make it possible" or would it quickly become required (i.e. any time you write normal, readable Python, it would break someone's optimizer, refactorer, linter, etc.?)
Whoa, whoa. That's not at all the idea. Currently *nobody* uses type annotations because there's no standard notation. My goal is to enable their use by proposing a standard, nothing more. ... Mypy lets you create stub modules that shadow stdlib (and other) modules during typechecking, and it comes with a modest set of standard stubs for some of the most popular stdlib modules. In most cases it's actually much more effective to create a stub than to add the annotations to the stdlib source code -- the stdlib code itself often is difficult to type check due to various optimizations or backwards compatibility concerns, but writing stubs is relatively straightforward, and the stubs will give useful guidance to users of the stdlib who care to run mypy over their own code.
+1 from me :-) In the PEP, please make it clear that you don't want any overly enthusiastic coredevs injecting the annotations into the standard library (have no doubt, some one would do it) and demanding changes to existing APIs where annotations didn't fit neatly (like Larry's proposal to change the long-standing APIs to use his nullable ints by adding None to optional int signatures). Raymond P.S. I would really like for the annotations to grow some way to communicate exceptions as well as return types (i.e. that list.index can raise a ValueError and list.pop can raise an IndexError). This would be only for the exceptions directly added by a function or method, not ones raised by the data (which is something the function can't control).
On 08/14/2014 12:59 PM, Raymond Hettinger wrote:
I would really like for the annotations to grow some way to communicate exceptions as well as return types
I'd hate for exception typing to be compulsory, though, since that turned out to be extremely annoying in Java. It would be even worse in Python, since there's no branch of the exception hierarchy corresponding to Java's RuntimeError for things that can be raised from anywhere. -- Greg
On Aug 13, 2014, at 6:33 PM, Greg Ewing
On 08/14/2014 12:59 PM, Raymond Hettinger wrote:
I would really like for the annotations to grow some way to communicate exceptions as well as return types
I'd hate for exception typing to be compulsory, though, since that turned out to be extremely annoying in Java.
It would be even worse in Python, since there's no branch of the exception hierarchy corresponding to Java's RuntimeError for things that can be raised from anywhere.
As I understand it, that would serve mostly documentational purpose. The only analysis we could do on that would be to check if an exception is ever caught by the caller somewhere up the stack. This could already be done by finding “raise” statements and going from there. But even with well annotated types that would be often hard to infer, limiting the usefulness of that analysis. Moreover, it wouldn’t surface problems when the code is using any type of framework that swallows exceptions to process them for the user. So that leaves documentation. Now this information already lands in docstrings, which lacks the “one-obvious-way-to-do-it” feel. However, thanks to the hackiness of that approach, users don’t expect the provided list of exceptions to be comprehensive. That would change with a first-class syntax for them. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
On 14.08.2014 02:59, Raymond Hettinger wrote:
P.S. I would really like for the annotations to grow some way to communicate exceptions as well as return types (i.e. that list.index can raise a ValueError and list.pop can raise an IndexError). This would be only for the exceptions directly added by a function or method, not ones raised by the data (which is something the function can't control).
+1 I made the same suggestion a couple of hours earlier. A standardized and introspectable way to add exception annotation is a common request. It's especially useful for C code because it's very hard to impossible to deduce exceptions from C code. For now there is no need for a new syntax. IMHO a decorator and standard location for exception annotations are sufficient. Perhaps somebody is able to come up with a syntax later. Such a decorator should also include the reason for an exception, too. For example: class list: @raises(IndexError, "when list is empty") def pop(self, index=None): pass @raises(ValueError) def index(self, value): pass Christian
On Thu, Aug 14, 2014 at 4:38 AM, Christian Heimes
I made the same suggestion a couple of hours earlier. A standardized and introspectable way to add exception annotation is a common request. It's especially useful for C code because it's very hard to impossible to deduce exceptions from C code.
For now there is no need for a new syntax. IMHO a decorator and standard location for exception annotations are sufficient. Perhaps somebody is able to come up with a syntax later.
The problem with declaring raisable exceptions is that it invites static verification of callers doing "the right thing" with them, as in Java, and that introduces noise in the software development process that is almost impossible to clean up. @raises(ValueError) def index(self, value): pass def found(self, value): return self.index(value) >= 0 Is found() correct? If not, I assure you the most common (and probably incorrect) solution will be: def found(self, value): try: return self.index(value) >= 0 except ValueError: return False Or worse: def found(self, value): try: return self.index(value) >= 0 except: return False The right thing to do with an exception is to let it through, unless you know exactly what you have to do about it. In this interview, Anders Hejlsberg talks about why checked exceptions were left out of the design of C#: http://www.artima.com/intv/handcuffs.html Cheers, -- Juancarlo *Añez*
On 14.08.2014 12:27, Juancarlo Añez wrote:
The problem with declaring raisable exceptions is that it invites static verification of callers doing "the right thing" with them, as in Java, and that introduces noise in the software development process that is almost impossible to clean up. [snip] The right thing to do with an exception is to let it through, unless you know exactly what you have to do about it.
In this interview, Anders Hejlsberg talks about why checked exceptions were left out of the design of C#:
Thanks for your input, Juancarlo. I don't think that anybody wants to add checked exceptions to Python at this point. It's one of my pain points with Java, too. As far as I can judge Raymond we both want to archive the same goal: improve introspection and self-documentation of code with a strong focus on bindings (e.g. C code). As I tried to explain earlier neither users nor tools are able to analyze code like builtin types and functions. Exception annotations could help IDEs to figure out what kind of exception could be raised. Does it sound better to you? The general idea is remotely related to mypy and should be addressed by a different PEP that accompanies the mypy PEP. Regards, Christian
On Thu, Aug 14, 2014 at 7:08 AM, Christian Heimes
As I tried to explain earlier neither users nor tools are able to analyze code like builtin types and functions. Exception annotations could help IDEs to figure out what kind of exception could be raised. Does it sound better to you?
It does, but it remains that annotations about exceptions will probably be inconsistent or incomplete: 1. If the function is changed, the annotation will remain valid even if the mentioned exceptions are not raised (this is the typical doc-comment problem). 2. The annotation cannot be validated unless the raisables of called functions (objects, actually) are analysed. Dealing with #2 means going the Java way and promoting incorrect exception handlers throughout. Not dealing with #2 means that the annotations are for documentation purposes only, so the same information can go in the doc-comment. This situation is unlike that for the proposed type signatures, because the intent for those is that they are strictly verified (with mypy or some other). In short, an exceptions signature would amount to: "This function may or may not raise the listed exceptions, and it may also raise other exceptions not in the list.". Yet, #2 can be dealt with by not being strict, and issuing warnings when things seem broken, taking note that the Python callables that can raise exceptions are not only functions and methods, but anything for which callable(o)==True. Cheers, -- Juancarlo *Añez*
On 14 August 2014 07:09, Guido van Rossum
On Wed, Aug 13, 2014 at 1:18 PM, Raymond Hettinger
wrote: On Aug 13, 2014, at 12:44 PM, python-ideas-request@python.org wrote:
The goal is to make it possible to add type checking annotations to 3rd party modules (and even to the stdlib) while allowing unaltered execution of the program by the (unmodified) Python 3.5 interpreter.
Is the goal to "make it possible" or would it quickly become required (i.e. any time you write normal, readable Python, it would break someone's optimizer, refactorer, linter, etc.?)
Whoa, whoa. That's not at all the idea. Currently *nobody* uses type annotations because there's no standard notation. My goal is to enable their use by proposing a standard, nothing more.
Given the general lack of familiarity with ABCs, I don't have any great fear of optional static typing becoming standard practice either. Just like ABCs, it's a useful way to formalise some checks when scaling up a code base to larger development teams. Most code bases aren't going to be that large, so it often isn't going to be worth the hassle. On the other hand, it's going to make it easier to detect certain kinds of structural errors more easily than even extensive unit testing, as well as helping out IDEs to provide better prompts to users, so I'm personally a fan of the idea. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (7)
-
Christian Heimes
-
Greg Ewing
-
Guido van Rossum
-
Juancarlo Añez
-
Nick Coghlan
-
Raymond Hettinger
-
Łukasz Langa