Mailman 3 August 2014 - Python-ideas

Decorators that execute once rather than every import (was: codec and decorator hacks)
by Gregory P. Smith 22 Aug '14

22 Aug '14

On Wed, Aug 20, 2014 at 7:25 AM, Donald Stufft <donald(a)stufft.io> wrote: > > mypy does have a codec that will ignore annotations on 2.x. > Note that all codec hacks and decorator hacks have a down side other than being hacks: They significantly slow down program import and start time. At least the results of the codec hack are cached when a .pyc is generated. Decorator executions are not. hmm.. could a flavor of decorator that is cacheable in .pyc's be created so the cost is only paid once rather than for each time the module is imported? -gps

3 2

Idea
by Scott Blair 22 Aug '14

22 Aug '14

Add continue to a try, except statement to continue where the code threw an error. Or a function of exceptions to get the line number at which it was thrown along with a way to go back to that line.

4 3

Mutating while iterating
by Aaron Brady 20 Aug '14

20 Aug '14

Hi, I asked about the inconsistency of the "RuntimeError" being raised when mutating a container while iterating over it here [1], "set and dict iteration" on Aug 16, 2012. [1] http://www.gossamer-threads.com/lists/python/python/1004659 Continuing new from the bugs issue page [2]: [2] http://bugs.python.org/issue22084 Other prior discussion [3] [4]: [3] http://bugs.python.org/issue19332 [4] http://bugs.python.org/issue6017 Thanks Mr. Storchaka for your comments. The new documentation didn't help. The current behavior is still a rare but inconsistent silent error. An implementation sketch in pseudocode might simplify the endeavor [5]: [5] http://home.comcast.net/~castironpi-misc/irc-0168%20mutating%20while%20iter… I gather we wouldn't want to pursue the "custom" data container, option "2e": we would still need both "malloc/free" and a reference count.

2 4

Re: [Python-ideas] Python-ideas Digest, Vol 93, Issue 118
by Jean-Charles Douet 20 Aug '14

20 Aug '14

Hi All, Just a few remarks : - Here is what is said about literate programming : http://www.literateprogramming.com/quotes_sd.html. Isn't that Pythonic ? - PyContracts, annotations, etc… look like « literate programming ». Doc is optional, type hint too, why not put the former with the latter ? - Now about PyContracts: I cannot prevent myself from thinking of it as a « CHECK CONSTRAINT » in PostgreSQL and co. - PyContracts aims at describing constraints, including them in docstrings. Thus documentation has to be more structured, more formatted and more informative about what's going on in the software being interpreted by Python automata. Perhaps structured doc becomes a solution to the Rice's theorem ? - Moreover, the « doctest » module could be very useful too. Whenever you write a docstring, you can have a test too; this module could test annotations and contracts too. Why not include it in standard library ? - Two systems of structured documentation : - CWEB : http://www-cs-faculty.stanford.edu/~uno/cweb.html - FunnelWeb - Are there collisions with these formats : ReSt, GRASS GIS' « boilerplating » in comments, #FIXME and #TODO for the IDEs, etc… Best regards, JCD.

1 0

type checking using PyContracts; was: Proposal: Use mypy syntax for function annotations
by Andrea Censi 20 Aug '14

20 Aug '14

Dear all, (Apologies for not replying to the original thread, I only signed up to the mailing list today) I wrote PyContracts[1], which implements type-checking functionality using decorators for Python 2.7+ and annotations in Python 3. [1] http://andreacensi.github.com/contracts/ > def word_count(input: List[str]) -> Dict[str, int]: Here's that example done using PyContracts and Python 3, and note the extra constraint '>=1' that PyContract can express: from contracts import contract @contract def word_count(input:'list(str)') -> 'dict(str:int,>=1): ... Here's that example done using PyContracts and Python 2.7, using a decorator instead of annotations: from contracts import contract @contract(input='list(str)', returns='dict(str:int,>=1)') def word_count(input): ... I would like to make a few points from my experience of writing PyContracts regarding the general issues that one encounters in adding type checking to Python. 1) Once one starts to think about it, "types" are very broad and if you design a system from scratch, make sure it is extensible. For example, you might want to have "positive integer" in addition to "integer" ('int', and 'int,>=0' in PyContracts). Also, let the users add their own types. 2) Dependent types are a must and they must have a simple syntax. For example, this is a PyContracts annotation that says the function takes a list of strings and returns a list of integers of the same size N: @contract(x='list[N](str)', returns='list[N](int)') def f(x): return list(map(len, x)) PyContract's convention is that and a,b,c,...,y,z bind to any variables and A,B,C,...,Z bind to integer variables. This makes numpy contracts very nice looking: @contract def gray2rgb(gray:'array[HxW](uint8)') -> 'array[HxWx3](uint8)': ... 3) There is an argument for using regular strings to describe annotations rather than Python constructs such as MyPy. One advantage is that the system is backwards compatible. It also makes the code much cleaner because you don't need to import the various symbols. Here's a simple example in MyPy: from typing typevar, Sequence T = typevar('T') # Declare type variable def first(seq: Sequence[T]) -> T: return seq[0] Here's the same example in PyContracts (using Python 3's annotations): from contracts import contract @contract def first(s: 'seq(type(t))') -> 'type(t)': return s[0] Same example in Python 2.7+ (using decorators): from contracts import contract @contract(s='seq(type(t))', returns='type(t)') def first(s): return s[0] (By the way, after a while, I actually like the decorator version better than Python 3's annotations, because you can see very easily the function's name and parameters, and only if you want you can then read the function annotations above.) cheers, A. -- Andrea Censi | http://censi.mit.edu | "Not all those who wander are lost." research scientist @ LIDS / Massachusetts Institute of Technology

2 1

Optional static typing -- the crossroads
by Guido van Rossum 20 Aug '14

20 Aug '14

I have read pretty much the entire thread up and down, and I don't think I can keep up with responding to every individual piece of feedback. (Also, a lot of responses cancel each other out. :-) I think there are three broad categories of questions to think about next. (A) Do we even need this? (B) What syntax to use? (C) Does/should it support <feature X>? Taking these in turn: (A) Do we even need a standard for optional static typing? Many people have shown either support for the idea, or pointed to some other system that addresses the same issue. On the other hand, several people have claimed that they don't need it, or that they worry it will make Python less useful for them. (However, many of the detractors seem to have their own alternative proposal. :-) In the end I don't think we can ever know for sure -- but my intuition tells me that as long as we keep it optional, there is a real demand. In any case, if we don't start building something we'll never know whether it'll be useful, so I am going to take a leap of faith and continue to promote this idea. I am going to make one additional assumption: the main use cases will be linting, IDEs, and doc generation. These all have one thing in common: it should be possible to run a program even though it fails to type check. Also, adding types to a program should not hinder its performance (nor will it help :-). (B) What syntax should a standard system for optional static typing use? There are many interesting questions here, but at the highest level there are a few choices that constrain the rest of the discussion, and I'd like to start with these. I see three or four "families" of approaches, and I think the first order is to pick a family. (1) The mypy family. (http://mypy-lang.org/) This is characterized by its use of PEP 3107 function annotations and the constraint that its syntax must be valid (current) Python syntax that can be evaluated without errors at function definition time. However, mypy also supports collecting annotations in separate "stub" files; this is how it handles annotations for the stdlib and C extensions. When mypy annotations occur inline (not in a stub file) they are used to type check the body of the annotated function as well as input for type checking its callers. (2) The pytypedecl family. (https://github.com/google/pytypedecl) This is a custom syntax that can only be used in separate stub files. Because it is not constrained by Python's current syntax, its syntax is slightly more elegant than mypy. (3) The PyCharm family. ( http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.…) This is a custom syntax that lives entirely in docstrings. There is also a way to use stub files with this. (In fact, every viable approach has to support some form of stub files, if only to describe signatures for C extensions.) (I suppose we could add a 4th family that puts everything in comments, but I don't think anyone is seriously working on such a thing, and I don't see any benefits.) There's also a variant of (1) that Łukasz Langa would like to see -- use the syntactic position of function annotations but using a custom syntax (e.g. one similar to the pytypedecl syntax) that isn't evaluated at function-definition time. This would have to use "from __future__ import <something>" for backward compatibility. I'm skeptical about this though; it is only slightly more elegant than mypy, and it would open the floodgates of unconstrained language design. So how to choose? I've read passionate attacks and defenses of each approach. I've got a feeling that the three projects aren't all that different in maturity (all are well beyond the toy stage, none are quite ready for prime time). In terms of specific type system features (e.g. forward references, generic types, duck typing) I expect they are all acceptable, and all probably need some work (and there's no reason to assume that work can't be done). All support stubs so you can specify signatures for code you can't edit (whether C extension, stdlib or just opaque 3rd party code). To me there is no doubt that (1) is the most Pythonic approach. When we discussed PEP 3107 (function annotations) it was always my goal that these would eventually be used for type annotations. There was no consensus at the time on what the rules for type checking should be, but their syntactic position was never in doubt. So we decided to introduce "annotations" in Python 3 in the hope that 3rd party experiments would eventually produce something satisfactory. Mypy is one such experiment. One of the important lessons I draw from mypy is that type annotations are most useful to linters, and should (normally) not be used to enforce types at run time. They are also not useful for code generation. None of that was obvious when we were discussing PEP 3107! I don't buy the argument that PEP 3107 promises that annotations are completely free of inherent semantics. It promises compatibility, and I take that very seriously, but I think it is reasonable to eventually deprecate other uses of annotations -- there aren't enough significant other uses for them to warrant crippling type annotations forever. In the meantime, we won't be breaking existing use of annotations -- but they may confuse a type checker, whether a stand-alone linter like mypy or built into an IDE like PyCharm, and that may serve as an encouragement to look for a different solution. Most of the thornier issues brought up against mypy wouldn't go away if we adopted another approach: whether to use concrete or abstract types, the use of type variables, how to define type equivalence, the relationship between a list of ints and a list of objects, how to spell "something that implements the buffer interface", what to do about JSON, binary vs. text I/O and the signature of open(), how to check code that uses isinstance(), how to shut up the type checker when you know better... The list goes on. There will be methods whose type signature can't be spelled (yet). There will be code distributed with too narrowly defined types. Some programmers will uglify their code to please the type checker. There are questions about what to do for older versions of Python. I find mypy's story here actually pretty good -- the mypy codec may be a hack, but so is any other approach. Only the __future__ approach really loses out here, because you can't add a new __future__ import to an old version. So there you have it. I am picking the mypy family and I hope we can start focusing on specific improvements to mypy. I also hope that somebody will write converters from pytypedecl and PyCharm stubs into mypy stubs, so that we can reuse the work already put into stub definitions for those two systems. And of course I hope that PyCharm and pytypedecl will adopt mypy's syntax (initially in addition to their native syntax, eventually as their sole syntax). PS. I realize I didn't discuss question (C) much. That's intentional -- we can now start discussing specific mypy features in separate threads (or in this one :-). -- --Guido van Rossum (python.org/~guido)

34 145

float comparison in doctest
by Kevin Davies 18 Aug '14

18 Aug '14

Erik Bray has created a nifty addition to doctest to compare floating point numbers using a `+FLOAT_CMP` flag. It is implemented within Astropy (http://www.astropy.org/) and the related issue is https://github.com/astropy/astropy/issues/2662. According to Erik, it may need some tweaks, but I think it would be a really useful feature to make available in the doctest package itself. Kevin

4 6

Re: [Python-ideas] Proposal: Use mypy syntax for function annotations
by willvarfar＠gmail.com 17 Aug '14

17 Aug '14

I fully support formalizing Python 3's annotations for type checking. I wrote - and use daily - my own type checker called obiwan https://pypi.python.org/pypi/obiwan Its a runtime type checker, and if enabled will check and enforce types on every call. I support the wider adoption and standardization of static type checkers, but runtime checkers are still wanted for very dynamic code and for checking external data e.g. I use obiwan for validating JSON. One small detail is that I feel the obiwan annotations are more pythonic than the mypy examples given. E.g. instead of: from typing import List, Dict def word_count(input: List[str]) -> Dict[str, int]: ... It would be: def word_count(input: [str]) -> {str, int}: ... Obiwan does not check types within functions; I was unwilling to try and overload comments! You can invoke obiwan to check things explicitly, but these are more as assertions. Anyway, when I look at the mypy in-function annotations (where comments are overloaded) I am cautious. It would be far nicer if we had annotations as part of the language instead, e.g. instead of: result = {} #type: Dict[str, int] It would be: result = {} -> {str, int} where we use the -> arrow again. I can see pros and cons for any implementation (as we'd want the annotation to be both to declare a type and to check a type, and want the annotation to be attached to the variable forever etc) so this would need a full PEP treatment and possibly be configurable as asserts are. But proper annotation support in the language rather than overloading comments would definitely be my preference. /Will /Will

3 3

RFC: Multiple Dispatch
by Matthew Rocklin 17 Aug '14

17 Aug '14

Hi all, I curate the multipledispatch library on PyPI. I believe that it is a natural continuation of singledispatch outlined in PEP 443 and included in functools 3.4. For those unaware, dispatching correctly and unambiguously on multiple inputs is a somewhat more complex problem than on a single input. I believe the approach taken in multipledispatch is fairly robust. I wrote a blogpost about MD a while ago: http://matthewrocklin.com/blog/work/2014/02/25/Multiple-Dispatch/ The docs pages live here: http://multiple-dispatch.readthedocs.org/en/latest/ And the github page lives here: https://github.com/mrocklin/multipledispatch Recommendations for improvement welcome. Thoughts on whether or not this is appropriate to include in the standard library also welcome. Best, -Matthew Rocklin

9 13

RFC: Multiple Dispatch
by Stéfane Fermigier 17 Aug '14

17 Aug '14

GvR wrote: > Are there good uses of singledispatch in the wild even? Martijn Faassen wrote a whole Web framework (Morepath = http://morepath.readthedocs.org/en/latest/) on top of it: http://blog.startifact.com/posts/reg-now-with-more-generic.html S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Founder, Nuxeo - Enterprise Content Management Platform - http://www.nuxeo.com/ Co-Founder and Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Founder & Vice-President, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Co-Founder & Co-organiser, Open World Forum - http://www.openworldforum.paris/fr/ --- "No problem is too small or too trivial if we can really do something about it." - Richard P. Feynman "Well done is better than well said." - Benjamin Franklin "There's no such thing as can't. You always have a choice." - Ken Gor

1 0