Mailman 3 August 2014 - Python-ideas

Proposal: Use mypy syntax for function annotations
by Guido van Rossum Aug. 28, 2014

Aug. 28, 2014

[There is no TL;DR other than the subject line. Please read the whole thing before replying. I do have an appendix with some motivations for adding type annotations at the end.] Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can learn from Haskell (and other languages); yesterday he gave the same talk at Dropbox. The talk is online ( https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad strokes comes down to three suggestions: (a) Python should adopt mypy's syntax for function annotations (b) Python's use of mutabe containers by default is wrong (c) Python should adopt some kind of Abstract Data Types Proposals (b) and (c) don't feel particularly actionable (if you disagree please start a new thread, I'd be happy to discuss these further if there's interest) but proposal (a) feels right to me. So what is mypy? It is a static type checker for Python written by Jukka for his Ph.D. thesis. The basic idea is that you add type annotations to your program using some custom syntax, and when running your program using the mypy interpreter, type errors will be found during compilation (i.e., before the program starts running). The clever thing here is that the custom syntax is actually valid Python 3, using (mostly) function annotations: your annotated program will still run with the regular Python 3 interpreter. In the latter case there will be no type checking, and no runtime overhead, except to evaluate the function annotations (which are evaluated at function definition time but don't have any effect when the function is called). In fact, it is probably more useful to think of mypy as a heavy-duty linter than as a compiler or interpreter; leave the type checking to mypy, and the execution to Python. It is easy to integrate mypy into a continuous integration setup, for example. To read up on mypy's annotation syntax, please see the mypy-lang.org website. Here's just one complete example, to give a flavor: from typing import List, Dict def word_count(input: List[str]) -> Dict[str, int]: result = {} #type: Dict[str, int] for line in input: for word in line.split(): result[word] = result.get(word, 0) + 1 return result Note that the #type: comment is part of the mypy syntax; mypy uses comments to declare types in situations where no syntax is available -- although this particular line could also be written as follows: result = Dict[str, int]() Either way the entire function is syntactically valid Python 3, and a suitable implementation of typing.py (containing class definitions for List and Dict, for example) can be written to make the program run correctly. One is provided as part of the mypy project. I should add that many of mypy's syntactic choices aren't actually new. The basis of many of its ideas go back at least a decade: I blogged about this topic in 2004 (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- see also the two followup posts linked from the top there). I'll emphasize once more that mypy's type checking happens in a separate pass: no type checking happens at run time (other than what the interpreter already does, like raising TypeError on expressions like 1+"1"). There's a lot to this proposal, but I think it's possible to get a PEP written, accepted and implemented in time for Python 3.5, if people are supportive. I'll go briefly over some of the action items. *(1) A change of direction for function annotations* PEP 3107 <http://legacy.python.org/dev/peps/pep-3107/>, which introduced function annotations, is intentional non-committal about how function annotations should be used. It lists a number of use cases, including but not limited to type checking. It also mentions some rejected proposals that would have standardized either a syntax for indicating types and/or a way for multiple frameworks to attach different annotations to the same function. AFAIK in practice there is little use of function annotations in mainstream code, and I propose a conscious change of course here by stating that annotations should be used to indicate types and to propose a standard notation for them. (We may have to have some backwards compatibility provision to avoid breaking code that currently uses annotations for some other purpose. Fortunately the only issue, at least initially, will be that when running mypy to type check such code it will produce complaints about the annotations; it will not affect how such code is executed by the Python interpreter. Nevertheless, it would be good to deprecate such alternative uses of annotations.) *(2) A specification for what to add to Python 3.5* There needs to be at least a rough consensus on the syntax for annotations, and the syntax must cover a large enough set of use cases to be useful. Mypy is still under development, and some of its features are still evolving (e.g. unions were only added a few weeks ago). It would be possible to argue endlessly about details of the notation, e.g. whether to use 'list' or 'List', what either of those means (is a duck-typed list-like type acceptable?) or how to declare and use type variables, and what to do with functions that have no annotations at all (mypy currently skips those completely). I am proposing that we adopt whatever mypy uses here, keeping discussion of the details (mostly) out of the PEP. The goal is to make it possible to add type checking annotations to 3rd party modules (and even to the stdlib) while allowing unaltered execution of the program by the (unmodified) Python 3.5 interpreter. The actual type checker will not be integrated with the Python interpreter, and it will not be checked into the CPython repository. The only thing that needs to be added to the stdlib is a copy of mypy's typing.py module. This module defines several dozen new classes (and a few decorators and other helpers) that can be used in expressing argument types. If you want to type-check your code you have to download and install mypy and run it separately. The curious thing here is that while standardizing a syntax for type annotations, we technically still won't be adopting standard rules for type checking. This is intentional. First of all, fully specifying all the type checking rules would make for a really long and boring PEP (a much better specification would probably be the mypy source code). Second, I think it's fine if the type checking algorithm evolves over time, or if variations emerge. The worst that can happen is that you consider your code correct but mypy disagrees; your code will still run. That said, I don't want to *completely* leave out any specification. I want the contents of the typing.py module to be specified in the PEP, so that it can be used with confidence. But whether mypy will complain about your particular form of duck typing doesn't have to be specified by the PEP. Perhaps as mypy evolves it will take options to tell it how to handle certain edge cases. Forks of mypy (or entirely different implementations of type checking based on the same annotation syntax) are also a possibility. Maybe in the distant future a version of Python will take a different stance, once we have more experience with how this works out in practice, but for Python 3.5 I want to restrict the scope of the upheaval. *Appendix -- Why Add Type Annotations?* The argument between proponents of static typing and dynamic typing has been going on for many decades. Neither side is all wrong or all right. Python has traditionally fallen in the camp of extremely dynamic typing, and this has worked well for most users, but there are definitely some areas where adding type annotations would help. - Editors (IDEs) can benefit from type annotations; they can call out obvious mistakes (like misspelled method names or inapplicable operations) and suggest possible method names. Anyone who has used IntelliJ or Xcode will recognize how powerful these features are, and type annotations will make such features more useful when editing Python source code. - Linters are an important tool for teams developing software. A linter doesn't replace a unittest, but can find certain types of errors better or quicker. The kind of type checking offered by mypy works much like a linter, and has similar benefits; but it can find problems that are beyond the capabilities of most linters. - Type annotations are useful for the human reader as well! Take the above word_count() example. How long would it have taken you to figure out the types of the argument and return value without annotations? Currently most people put the types in their docstrings; developing a standard notation for type annotations will reduce the amount of documentation that needs to be written, and running the type checker might find bugs in the documentation, too. Once a standard type annotation syntax is introduced, it should be simple to add support for this notation to documentation generators like Sphinx. - Refactoring. Bob's talk has a convincing example of how type annotations help in (manually) refactoring code. I also expect that certain automatic refactorings will benefit from type annotations -- imagine a tool like 2to3 (but used for some other transformation) augmented by type annotations, so it will know whether e.g. x.keys() is referring to the keys of a dictionary or not. - Optimizers. I believe this is actually the least important application, certainly initially. Optimizers like PyPy or Pyston <https://github.com/dropbox/pyston> wouldn't be able to fully trust the type annotations, and they are better off using their current strategy of optimizing code based on the types actually observed at run time. But it's certainly feasible to imagine a future optimizer also taking type annotations into account. -- --Guido "I need a new hobby" van Rossum (python.org/~guido)

51 183

Add an introspection API to Executor
by Ram Rachum Aug. 27, 2014

Aug. 27, 2014

Sometimes I want to take a live executor, like a `ThreadPoolExecutor`, and check up on it. I want to know how many threads there are, how many are handling tasks and which tasks, how many are free, and which tasks are in the queue. I asked on Stack Overflow: http://stackoverflow.com/questions/25474204/checking-up-on-a-concurrent-fut… There's an answer there, but it uses private variables and it's not part of the API. I suggest it become a part of the API. There should be an API for checking on what the executor is currently doing and answering all the questions I raised above. Thanks, Ram.

8 25

Re: [Python-ideas] Proposal: Use mypy syntax for function annotations
by Terry Reedy Aug. 26, 2014

Aug. 26, 2014

On 8/25/2014 7:58 PM, Guido van Rossum wrote: > On Mon, Aug 25, 2014 at 4:36 PM, Terry Reedy <tjreedy(a)udel.edu > <mailto:tjreedy@udel.edu>> wrote: > I am referring to the current stdlib runtime use of .__annotations__ > directly by inspect.getfullargspec and now inspect.signature and > indirectly by Idle calltips. > > def f(n:[int, 'random non-type info'])->(int, 'more non-type info'): > pass > import inspect as ip > print(ip.formatargspec(*ip.__getfullargspec(f))) > print(str(ip.signature(f))) > > (n: [<class 'int'>, 'random non-type info']) -> (<class 'int'>, > 'more non-type info') > (n:[<class 'int'>, 'random non-type info']) -> (<class 'int'>, 'more > non-type info') > > (Idle currently displays the first, will change to the second. The > display difference is more pronounced (now, at least) for C > functions with .__text_signature__ from Argument Clinic.) > > To me, 'argument specification' and especially 'signature' imply > arguments names and types, but not other stuff. Certainly, name and > type is all that is wanted for a tip on how to write a call. Extra > stuff would typically be noise in this context -- especially for > beginners. > > > I see two possible concerns here -- can you clarify? > > (1) If a function is annotated with multiple *categories* of annotations > (e.g. type hints and html form specifiers) you want to change inspect to > only return the type hints. > > (2) If a function is annotated for a non-type-hinting category (e.g. > only html form specifiers) you want to change inspect to return nothing. Both are covered by "I would either like non-type-hinting annotations moved elsewhere by decorators (my older idea) *or* I would like inspect.signature to at least optionally ignore them (today's idea). The latter is more dependable since we control the inspect functions. I an not too worried that anyone would use the blessed type-hint syntax for something completely different. On the other hand, I can imagine that there might someday be more that one annotation system in general use. > I hadn't thought of either of these; I was only concerned about making > sure that a static type checker wouldn't be unduly confused by > non-type-hinting annotations. If, for instance, everything in typing or produced by the function is a subclass of BaseTyping or an instance of MetaTyping, then post-compile typing objects would be easy to distinguish. Source annotations are a harder problem. >I assume __annotations__ should always > return the full expression found in the syntactic position of the > annotation I think that should continue. > (though decorators can change this). Because .__annotations__ is a regular dict. annotation syntax in code --compiler>> initial .__annotation__ dict --decorators>> exposed .__annotation__ dict + any other attributes added. > I am worried about annotations other than type hints. Compact type > hints, especially for type-opaque parameter names, should improve > the usefulness of call tips. Extra noise detracts. > > If type hints were standardized so as to be easily recognizable, > .signature could be given an option to ignore anything else. > > So I think you are actually not excited about any mechanism to keep > other uses of annotations alive, I want there to be such a mechanism even though it is does not 'excite' me . Just as you are focused on type hints for static hints, I, *as an Idle maintainer* who just happens to have spent considerable time on the call tip feature, am focused on type hints for dynamic documentation. > and you would just as well only have them used for type hints? Too strong. > (That's fine with me, but there is the backward compatibility issue, I said in my first response that I don't think we should break code. This is me speaking a as core developer and community member. I do want multiple ramifications of annotation use considered, including run-time use. > and perhaps legitimate cool alternate uses of annotations. Most experiments do not amount to much, but a few, such as the metaclass hack, certainly have. Python should continue to allow experiments. > We still have the choice to allow only one category of > annotations per function.) This seems dubious to me. If strings, tuples, or dicts are allowed as annotations, it seems to me that they can be used to combine annotations. -- Terry Jan Reedy

1 0

Add nullifier argument to functools.reduce?
by Warren Weckesser Aug. 26, 2014

Aug. 26, 2014

I'd like to add an additional optional argument to functools.reduce. The argument is the "nullifier" of the reducing operation. It is a value such that function(nullifier, anything) returns nullifier. For example, if function(x, y) computes x*y, the nullifier is 0. If function(x, y) is the intersection of the sets x and y, the nullifier is the empty set. The argument would allow reduce to "short circuit" its calculation. When reduce encounters the nullifier, it can return immediately. This can provide a significant improvement in performance in some cases. The change is simple. Here, for example, is the "rough equivalent" for functools.reduce from the docs: def reduce(function, iterable, initializer=None): it = iter(iterable) if initializer is None: try: initializer = next(it) except StopIteration: raise TypeError('reduce() of empty sequence with no initial value') accum_value = initializer for x in it: accum_value = function(accum_value, x) return accum_value Here's how it looks with the optional nullifier argument; the only change is the new argument and an 'if' statement in the 'for' loop. def reduce(function, iterable, initializer=None, nullifier=None): it = iter(iterable) if initializer is None: try: initializer = next(it) except StopIteration: raise TypeError('reduce() of empty sequence with no initial value') accum_value = initializer for x in it: if nullifier is not None and accum_value == nullifier: break accum_value = function(accum_value, x) return accum_value (It might be better to use a distinct singleton for the default value of nullifier, to allow None to be a valid nullifier.) The actual implementation is in the extension module _functoolsmodule.c. It looks like the changes to the C code should be straightforward. Warren

13 27

Was: Annotations (and static typing), Now:Sharing __annotations__
by Cem Karan Aug. 25, 2014

Aug. 25, 2014

On Aug 24, 2014, at 9:22 PM, Chris Angelico <rosuav(a)gmail.com> wrote: > On Mon, Aug 25, 2014 at 11:15 AM, Cem Karan <cfkaran2(a)gmail.com> wrote: >> Because of this, using 'annotizer' as a key is meaningless; there could be many instances live at the same time. > > But that's exactly my point! You don't use "annotizer" as the key, you > use annotizer. You use the object itself, not the string. And now I feel dumb... You're right of course, by passing in the object itself, there are no namespace conflicts, inter-project coordination is nil, and on top of it all, you can perform introspection of the annotizer object itself to find out more information about it. UUIDs aren't required at all. Right-o, with your permission Chris, I'd like to amend my proposal so that instead of using UUIDs, the annotizer object itself is used as the key. Depending on what work is like this week, I'll modify my code to do this. Thanks, Cem Karan

2 1

Annotations (and static typing)
by Paul Moore Aug. 25, 2014

Aug. 25, 2014

Sigh. I go away for a week and come back to a mega-thread I can never hope to catch up on :-) TL; DR; - Although mypy looks interesting, I think it's too soon to close the door on all other uses of annotations. Let's find a solution that allows exploration of alternative uses for a while longer. OK, can I just make some points regarding the static typing thread. First of all, I have no issue with the idea of static typing, and in fact I look forward to seeing what benefits it might have (if nothing else, the pointer to mypy, which I'd never heard of before, is appreciated). It won't be something I use soon (see below) but that's fine. But Guido seems to be saying (on a number of occasions) that nobody is really using annotations, so he wants to focus on the static typing use case alone. I think this is a mistake. First of all, I see no reason why functions using typing annotations could not be introduced with a decorator[1]. So why insist that this is the *only* use of annotations, when it's pretty easy to allow others to co-exist? Also, the "nobody is using annotations" argument? Personally, I know of a few other uses: 1. Argument parsers - at least 3 have been mentioned in the thread. 2. Structure unpacking - I think there is a library that uses annotations for this, although I may be wrong. 3. FFI bindings. I know I've seen this discussed, although I can't find a reference just now. There are probably other ideas around as well (GUI bindings, documentation generation, ...) None are particularly mature, and most are just at the "ideas" stage, but typically the ideas I have seen are the sort of thing you'd write a library for, and Python 3 only libraries *really* aren't common yet. The problem for people wanting to experiment with annotations, is that they need to be writing Python 3 only code. While Python 3 adoption is growing rapidly, I suspect that large applications are typically still focused on either going through, or tidying up after, a 2-3 migration. And new projects, while they may be developed from the ground up using Python 3, will typically be using programmers skilled in Python 2, to whom Python 3 features are not yet an "instinctive" part of the toolset. Apart from large standalone applications, there are smaller scripts (which are typically going to be too small to need programming-in-the-large features like annotations) and libraries (which really aren't yet in a position to drop Python 2.x totally, unless they have a fairly small user base). So I don't see it as compelling that usage of annotations in the wild is not yet extensive. Rather than close the door on alternative uses of annotations, can I suggest: 1. By all means bless mypy syntax as the standard static typing notation - this seems like a good thing. 2. Clarify that static typing annotations should be introduced with a decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy declaration in the stdlib, and have a registration protocols for tools to hook their own implementation into it.[2] 3. Leave the door open for other uses of decorators, at least until some of the more major libraries drop Python 2.x support completely (and hence can afford to have a dependency on a Python 3 only module that uses annotations). See at that stage if annotations take off. 4. If we get to a point where even libraries that *could* use annotations don't, then revisit the idea of restricting usage to just type information. Paul [1] Also, a decorator could allow a Python 2 compatible form by using decorator arguments as an alternative to annotations. [2] I've yet to see a clear explanation of how "a tool using type annotations" like an linter, editor, IDE or Python compiler would use them in such a way that precludes decoration as a means of signalling the annotation semantics.

13 25

Optional Static Typing -- the Python Way
by Ethan Furman Aug. 24, 2014

Aug. 24, 2014

After reading through the OST threads, and learning a bit more about what static typing means, and the difference between nominal and structural subclassing, I think I have a better appreciation for the knee-jerk "no way!" reaction, and also how to make what seems like a foreign-to-Python concept pythonistic. The most-feared problem seems to be over-specification (it is certainly mine). I think the answer has been alluded to a couple times already, but just to hopefully bring it front and center: Instead of either 'nominal' or 'structural', use our own 'dynamical' subclassing scheme. In other words: def spamify(current: datetime.date, moved:int) -> bool: # ONE_DAY is a one day time delta, previously declared days = ONE_DAY * moved proposed_date = current + days return it_works(proposed_date) The type checker will not look to see if 'current' is a datetime.date, but rather will check that what datetime.date's __add__ works with (datetime.delta) and then check that what is passed in for current has an __add__ that also works with datetime.delta. I think this would be far more valuable than just nominal as it follows duck-typing, and hopefully less work than structural as it's only checking the methods and attributes actually used. On the minus side, there are times when we need *exactly* a datetime.date, so we would need a way to specify exact vs dynamic, but dynamic should be the default. -- ~Ethan~

10 24

Proposal : allowing grouping by relation
by Yotam Vaknin Aug. 23, 2014

Aug. 23, 2014

Hi, I am using groupby (from itertools) to group objects by a key. It would be very useful for me to be able to group objects by the relation of two consecutive objects or by an object relation to the first object in the current group. I think it should be done by adding a "relation" keyword to the function, that accept two argument functions that return true or false. It would also be useful to create a function that enable easily creating relation functions. (Like attrgetter does for keys) ls = "aaabcdddefgjklm" groupby(ls, relation=difference(3,key = ord)) #[['a', 'a', 'a', 'b', 'c', 'd', 'd', 'd', 'e', 'f', 'g'], ['j', 'k', 'l', 'm']] I think in this case the function won't return a key-group tuple, but just a group iterable. This is already very useful for me, to group event objects in a list if they are close enough in time. I wrote most of what I had in mind here: https://github.com/tomirendo/Grouper

3 2

Optional static typing -- late to the party
by Tennessee Leeuwenburg Aug. 22, 2014

Aug. 22, 2014

Hi all, I apologise in advance for being across only that proportion of the previous correspondence that I could read in about half an hour. Too much has been said to fully integrate it all. I have some responses to what was written, and one of those I feel has been largely missed. Please accept this as "just some stuff I think" and not any kind of strident criticism or positioning. The primary goal should be a syntax which enhances human readability. While I agree that having good static code analysis tools is very useful for the write --> test --> fix cycle, actually the more bugs you can catch during the human read/understand/write cycle, the better. The earlier you can communicate useful information about variable types, the better, and the earliest stage of that is code comprehension at the human level. I mostly agree with those who are preferring docstring embedding over signature decoration, particularly for complex cases, due to the desire to separate the complexity into structured sections. The signature is like the heading, and the annotation is like additional detail. I was led to a conclusion: what is wrong with allowing both? Indeed, clearly both are actually already supported by various tools. So perhaps there is actually a higher-level common concept -- what are the actual assertion which are going to be supported? Can I declare a variable only to be a list, or can it be a list-of-ints? Further, is it possible to standardise between some of the syntax/terminology, and some of the assertion types, such that they are consistent between function annotation syntaxes and docstring embedding syntaxes? Could I use function annotation for simple cases, but seamlessly move to a docstring-embedded approach for a more complex example? e.g. def frobnicate(x: int, y: int) -> float ''' Applies some kind of standard function ''' return x**2 / y is great. But def handle_complex_case(name, context, objective, datacube): ''' @param name: str, contains a "Firstname Lastname" string @param context: dict, contains a data record @param objective: int, contains a coded directive @param datacube: numpy.array, contains a field of continuous spatial data @return: instructionset: list(callable), a list of callable instructions ''' allows me to describe what's going on in that function in a way which supports both typing and static analysis. Apologies for inventing the docstring syntax rather than correctly using an existing syntax. However, my point is that in the first example, an annotation approach is *more* readable. In the second example, you really want to spend more time describing and annotating the function. If there could be some agreement about the details and consistency, I see no reason that a dual style could not be preferable, and no more complex to understand and use. Regards, -Tennessee Leeuwenburg -- -------------------------------------------------- Tennessee Leeuwenburg http://myownhat.blogspot.com/ "Don't believe everything you think"

10 16

Allow getting all the items out of a `threading.local`
by Ram Rachum Aug. 22, 2014

Aug. 22, 2014

I propose adding a method to `threading.local` that allows to check the values stored for any and all threads. I asked on Stack Overflow how to do that: http://stackoverflow.com/questions/25435908/python-getting-all-the-items-ou… But I got the answer that it's only possible on the pure Python implementation, not the C one. And it's quite ugly too. I propose a method be added to both the Python implementation and the C implementation of `threading.local` that allows checking the values stored for all threads. Thanks, Ram.

2 1