On 9 January 2016 at 12:08, Steven D'Aprano wrote:

On Fri, Jan 08, 2016 at 03:04:58PM -0800, Guido van Rossum wrote:

[...]

...

Since Python 2 doesn't support function annotations we've had to look for alternatives. We considered stub files, a magic codec, docstrings, and additional `# type:` comments. In the end we decided that `# type:` comments are the most robust approach. We've experimented a fair amount with this and we have a proposal for a standard. [...] - Stub files: this would complicate the analysis in mypy quite a bit, because it would have to parse both the .py file and the .pyi file and somehow combine the information gathered from both, and for each function it would have to use the types from the stub file to type-check the body of the function in the .py file. This would require a lot of additional plumbing. And if we were using Python 3 we would want to use in-line annotations anyway.

I don't understand this paragraph. Doesn't mypy (and any other type checker) have to support stub files? I thought that stub files are needed for extension files, among other things. So I would have expected that any Python 2 type checker would have to support stub files as well, regardless of whether inline #type comments are introduced or not.

Stub files are easy to use if you're using them *instead of* the original source file (e.g. annotating extension modules, or typeshed annotations for the standard library). Checking a stub file for consistency against the published API of the corresponding module also seems like it would be straightforward (while using both a stub file *and* inline annotations for the same API seems like it would be a bad idea, it's at least necessary to check that the *shape* of the API matches, even if there's no type information). However, if I'm understanding correctly, the problem Guido is talking about here is a different one: analysing a function *implementation* to ensure it is consistent with its own published API. That's relatively straightforward with inline annotations (whether function annotation based, comment based, or docstring based), but trickier if you have to pause the analysis, go look for the right stub file, load it, determine the expected public API, and then resume the analysis of the original function. The other downside of the stub file approach is the same reason it's not the preferred approach in Python 3: you can't see the annotations yourself when you're working on the function. Folks working mostly on solo and small team projects may not see the appeal of that, but when doing maintenance on large unfamiliar code bases, the improved local reasoning those kinds of inline notes help support can be very helpful.

Will Python 3 type checkers be expected to support #type comments as well as annotations and stub files?

#type comment support is already required for variables and attributes: https://www.python.org/dev/peps/pep-0484/#type-comments That requirement for type checkers to support comment based type hints would remain, even if we were to later add native syntactic support for variable and attribute typing. I read Guido's proposal here as offering something similar for function annotations, only going in the other direction: providing a variant spelling for function type hinting that can be used in single source Python 2/3 code bases that can't use function annotations. I don't have a strong opinion on the specifics, but am +1 on the general idea - I think the approach Dropbox are pursuing of adopting static type analysis first, and then migrating to Python 3 (or at least single source Python 2/3 support) second is going to prove to be a popular one, as it allows you to detect a lot of potential migration issues without necessarily having to be able to exercise those code paths in a test running under Python 3. The 3 kinds of annotation would then have 3 clear function level use cases: stub files: annotating third party libraries (e.g. for typeshed) #type comments: annotating single source Python 2/3 code function annotations: annotating Python 3 code Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Pavol Lisy, Saturday, January 09, 2016 01:54:

Could not something like this ->

def embezzle(self, account, funds=1000000, *fake_receipts): # def embezzle(self, account: str, funds: int = 1000000, *fake_receipts: str) -> None: """Embezzle funds from account using fake receipts.""" <code goes here>

make 1. transition from python2 to python3 more simple? 2. python3 checkers more easily changeable to understand new python2 standard? 3. simpler impact to documentation (means also simpler knowledbase to be learn) about annotations?

+1 on this, which is close to what I've been doing for a while now. 4. Educates people who have only seen Py2 prototypes to recognize what the Py3 annotations look like.

FWIW, we now fully support both Jedi and lib2to3 in Jython 2.7.1 master. With some other work this weekend, we should be releasing 2.7.1 beta 3 and then shortly a RC - we just fixed the last blocking bug. On Sat, Jan 9, 2016 at 11:52 AM, Jim Baker wrote:

+1, I would really like to try out type annotation support in Jython, given the potential for tying in with Java as a source of type annotations (basically the equivalent of stubs for free). I'm planning on sprinting on Jython 3 at PyCon, but let's face it, that's going to take a while to really finish.

re the two approaches, both are workable with Jython:

* lib2to3 is something we should support in Jython 2.7. There are a couple of data files that we don't support in the tests (too large of a method for Java bytecode in infinite_recursion.py, not terribly interesting), plus a few other tests that should work. Therefore lib2to3 should be in the next release (2.7.1).

* Jedi now works with the last commit to Jython 2.7 trunk, passing whatever it means to run random tests using its sith script against its source. (The sith test does not pass with either CPython or Jython's stdlib, starting with bad_coding.py.)

- Jim

Done: https://hg.python.org/peps/rev/06f8470390c2 (I'm happy to change or move this if there *is* a serious concern -- but I figured if there isn't I might as well get it over with. On Mon, Jan 11, 2016 at 9:27 AM, Guido van Rossum wrote:

Unless there's a huge outcry I'm going to add this as an informational section to PEP 484.

-- --Guido van Rossum (python.org/~guido)

-- --Guido van Rossum (python.org/~guido)

On Sat, 9 Jan 2016 at 11:31 Guido van Rossum wrote:

On Sat, Jan 9, 2016 at 1:54 AM, Pavol Lisy wrote:

...

Could not something like this ->

def embezzle(self, account, funds=1000000, *fake_receipts): # def embezzle(self, account: str, funds: int = 1000000, *fake_receipts: str) -> None: """Embezzle funds from account using fake receipts.""" <code goes here>

make 1. transition from python2 to python3 more simple? 2. python3 checkers more easily changeable to understand new python2 standard? 3. simpler impact to documentation (means also simpler knowledbase to be learn) about annotations?

There would still have to be some marker like "# type:" for the type checker to recognize -- I'm sure that plain comments with alternate 'def' statements are pretty common and we really don't want the type checker to be confused by those.

I don't like that the form you propose has so much repetition -- the design of Python 3 annotations intentionally is the least redundant possible, and my (really Jukka's) proposal tries to keep that property.

Modifying type checkers to support this syntax is easy (Jukka already did it for mypy).

Note that type checkers already have to parse the source code without the help of Python's ast module, because there are other things in comments: PEP 484 specifies variable annotations and a few forms of `# type: ignore` comments.

Regarding the idea of a decorator, this was discussed and rejected for the original PEP 484 proposal as well. The problem is similar to that with your 'def' proposal: too verbose. Also a decorator is more expensive (we're envisioning adding many thousands of decorators, and it would weigh down program startup). We don't envision needing to introspect __annotations__ at run time. (Also, we already use decorators quite heavily -- introducing a @typehint decorator would make the code less readable due to excessive stacking of decorators.)

Our needs at Dropbox are several: first, we want to add annotations to the code so that new engineers can learn their way around the code quicker and refactoring will be easier; second, we want to automatically check conformance to the annotations as part of our code review and continuous integration processes (this is where mypy comes in); third, once we have annotated enough of the code we want to start converting it to Python 3 with as much automation is feasible. The latter part is as yet unproven, but there's got to be a better way than manually checking the output of 2to3 (whose main weakness is that it does not know the types of variables). We see many benefits of annotations and automatically checking them using mypy -- but we don't want the them to affect the runtime at all.

To help answer the question about whether this could help with porting code to Python 3, the answer is "yes"; it's not essential but definitely would be helpful. Between Modernize, pylint, `python2.7 -3`, and `python3 -bb` you cover almost all of the issues that can arise in moving to Python 3. But notice that half of those tools are running your code under an interpreter with a certain flag flipped, which means run-time checks that require excellent test coverage. With type annotations you can do offline, static checking which is less reliant on your tests covering all corner cases. Depending on how the tools choose to handle representing str/unicode in Python 2/3 code (i.e., say that if you specify the type as 'str' it's an error and anything that is 'unicode' is considered the 'str' type in Python 3?), I don't see why mypy can't have a 2/3 compatibility mode that warns against uses of, e.g. the bytes type that don't directly translate between Python 2 and 3 like indexing. That kind of static warning would definitely be beneficial to anyone moving their code over as they wouldn't need to rely on e.g., `python3 -bb ` and their tests to catch that common issue with bytes and indexing. There is also the benefit of gradual porting with this kind of offline checking. Since you can slowly add more type information, you can slowly catch more issues in your code. Relying on `python3 -bb`, though, requires you have ported all of your code over first before running it under Python 3 to catch some issues.

On Sat, Jan 9, 2016 at 4:48 AM M.-A. Lemburg wrote:

On 09.01.2016 00:04, Guido van Rossum wrote:

...

Since Python 2 doesn't support function annotations we've had to look for alternatives. We considered stub files, a magic codec, docstrings, and additional `# type:` comments. In the end we decided that `# type:` comments are the most robust approach. We've experimented a fair amount with this and we have a proposal for a standard.

The proposal is very simple. Consider the following function with Python 3 annotations:

def embezzle(self, account: str, funds: int = 1000000, *fake_receipts: str) -> None: """Embezzle funds from account using fake receipts.""" <code goes here>

An equivalent way to write this in Python 2 is the following:

def embezzle(self, account, funds=1000000, *fake_receipts): # type: (str, int, *str) -> None """Embezzle funds from account using fake receipts.""" <code goes here>

By using comments, the annotations would not be available at runtime via an .__annotations__ attribute and every tool would have to implement a parser for extracting them.

Wouldn't it be better and more in line with standard Python syntax to use decorators to define them ?

@typehint("(str, int, *str) -> None") def embezzle(self, account, funds=1000000, *fake_receipts): """Embezzle funds from account using fake receipts.""" <code goes here>

This would work in Python 2 as well and could (optionally) add an .__annotations__ attribute to the function/method, automatically create a type annotations file upon import, etc.

The goal of the # type: comments as described is to have this information for offline analysis of code, not to make it available at run time. Yes, a decorator syntax could be adopted if anyone needs that. I don't expect anyone does. Decorators and attributes would add run time cpu and memory overhead whether the information was going to be used at runtime or not (likely not; nobody is likely to *deploy* code that looks at __annotations__). -gps

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Experts (#1, Jan 09 2016)

...

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/

---

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Jan 11, 2016 at 12:22 PM, Andrew Barnert wrote:

On Jan 11, 2016, at 10:42, Gregory P. Smith wrote:

The goal of the # type: comments as described is to have this information for offline analysis of code, not to make it available at run time. Yes, a decorator syntax could be adopted if anyone needs that. I don't expect anyone does. Decorators and attributes would add run time cpu and memory overhead whether the information was going to be used at runtime or not (likely not; nobody is likely to *deploy* code that looks at __annotations__).

These same arguments were made against PEP 484 in the first place, and (I think rightly) dismissed.

The way I recall it the argument was made against using decorators for PEP 484 and we rightly decided not to use decorators.

3.x code with annotations incurs a memory overhead, even though most runtime code is never going to use them. That was considered to be acceptable. So why isn't it acceptable for the same code before it's ported to 3.x? Or, conversely, if it isn't acceptable in 2.x, why isn't it a serious blocking regression that, once the port is completed and you're running under 3.x, you're now wasting memory for those useless annotations?

I'm not objecting to the memory overhead of using decorators, but to the execution time (the extra function call). And the scope for the proposal is much smaller -- while PEP 484 is the first step on a long road towards integrating gradual (i.e. OPTIONAL) typing into Python, the proposal on the table today is only meant for annotating Python 2.7 code so we can get rid of it more quickly.

Meanwhile, when _are_ annotations useful at runtime? Mostly during the kind of debugging that you'll be doing during something like a port from 2.x to 3.x. While you're still, by necessity, running under 2.x. If they're not useful there, it's hard to imagine why they'd be useful after the port is done, when you're deploying your 3.x code.

I'm not sure how to respond to this -- I disagree with your prediction but I don't think either of us really has any hard data from experience yet. I am however going to be building the kind of experience that might eventually be used to decide this, over the next few years. The first step is going to introduce annotations into Python 2.7 code, and I know my internal customers well enough to know that convincing them that we should use decorators for annotations would be a much bigger battle than putting annotations in comments. Since I have many other battles to fight I would like this one to be as short as possible. So it seems like using decorators (or backporting the syntax, as Google has

done) has better be acceptable for 2.7, or the PEP 484 design has a serious problem, and in a few months we're going to see Dropbox and Google and everyone else demanding a way to use type hinting without wasting memory on annotations are runtime in 3.x.

Again, I disagree with your assessment but it's difficult to prove anything without hard data. One possible argument may be that Python 3 offers a large package of combined run-time advantages, with some cost that's hard to separate. However, for Python 2.7 there's either a run-time cost or there's no run-time cost -- there's no run-time benefit. And I don't want to have to calculate how many extra machines we'll need to provision in order to make up for the run-time cost. -- --Guido van Rossum (python.org/~guido)

On 1/11/2016 5:38 PM, M.-A. Lemburg wrote:

To clarify: My suggestion to use a simple decorator with essentially the same syntax as proposed for the "# type: comments " was meant as *additional* allowed syntax, not necessarily as the only one to standardize.

Code with type comments will run on any standard 2.7 interpreter. Code with an @typehint decorator will have to either run on a nonstandard interpreter or import 'typehint' from somewhere other than the stdlib or define 'typehint' at the top of the file or have the decorators stripped out before public distribution. To me, these options come close to making the decorator inappropriate as a core dev recommendation. However, the new section of the PEP could have a short paragraph that mentions @typehint(typestring) as a possible alternative (with the options given above) and recommend that if a decorator is used, then the name should be 'typehint' (or something else agreed on) and that the typestring should be a quoted version of what would follow '# type: ' in a comment, 'as already defined above' (in the previous recommendation). In other words, Guido's current addition has two recommendations: 1. the syntax for a typestring 2. the use of a typestring (append it to a '# type: ' comment) If a decorator alternative uses the same syntax, a checker would need just one typestring parser. I think the conditional recommendation would be within the scope of what is appropriate for us to do.

I'm a bit worried that by standardizing on using comments for these annotations only, we'll end up having people not use the type annotations because they simply don't like the style of having function bodies begin with comments instead of doc-strings. I certainly wouldn't want to clutter up my code like that. Tools parsing Python 2 source code may also have a problem with this (e.g. not recognize the doc-string anymore).

I have to admit that I was not fully cognizant before than a comment could precede a docstring. -- Terry Jan Reedy

What about this? def embezzle(self, account: "PEP3107 annotation"): # type: (str) -> Any """Embezzle funds from account using fake receipts.""" <code goes here> --- And BTW in PEP484 text -> Functions with the @no_type_check decorator or with a # type: ignore comment should be treated as having no annotations. could be probably? -> Functions with the @no_type_check decorator or with a # type: ignore comment should be treated as having no type hints.

2016-01-11 22:52 GMT+01:00, Guido van Rossum :

On Mon, Jan 11, 2016 at 1:48 PM, Pavol Lisy wrote:

...

What about this?

def embezzle(self, account: "PEP3107 annotation"): # type: (str) -> Any """Embezzle funds from account using fake receipts.""" <code goes here>

I don't understand your proposal -- this is not valid Python 2.7 syntax so we cannot use it.

I had two things in my mind: 1. suggest some possible impact in the future. In time we are writing code compatible with python2 and python3 we will have type hints comments under python3 too. And because they are more compatible, there is risk(?) that they could be more popular then original PEP484 (for python3) proposal! 2. PEP484 describe possibility how to support other use of annotations and propose to use # type: ignore but similar method how to preserve other use of annotations could be (for example): # type: (str) -> Any and this could combine goodness of type-hints-tools and other types of annotations. At least in deprecation period (if there will be any) for other annotation types.

On Mon, Jan 11, 2016 at 5:39 PM, Steven D'Aprano wrote:

On Mon, Jan 11, 2016 at 12:22:55PM -0800, Andrew Barnert via Python-ideas wrote:

...

in a few months we're going to see Dropbox and Google and everyone else demanding a way to use type hinting without wasting memory on annotations are runtime in 3.x.

I would be happy to see a runtime switch similar to -O that drops annotations in 3.x, similar to how -OO drops docstrings.

Actually my experience with -OO (and even -O) suggest that that's not a great model (e.g. it can't work with libraries like PLY that inspect docstrings). A better model might be to let people select this on a per module basis. Though I could also see a future where __annotations__ is a more space-efficient data structure than dict. Have you already run into a situation where __annotations__ takes up too much space? -- --Guido van Rossum (python.org/~guido)

...

An equivalent way to write this in Python 2 is the following:

def embezzle(self, account, funds=1000000, *fake_receipts): # type: (str, int, *str) -> None """Embezzle funds from account using fake receipts.""" <code goes here>

By using comments, the annotations would not be available at runtime via an .__annotations__ attribute and every tool would have to implement a parser for extracting them.

Wouldn't it be better and more in line with standard Python syntax to use decorators to define them ?

@typehint("(str, int, *str) -> None") def embezzle(self, account, funds=1000000, *fake_receipts): """Embezzle funds from account using fake receipts.""" <code goes here>

I really like MAL's variation much better. Being able to see .__annotations__ at runtime feels like an important feature that we'd give up with the purely comment style. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.