On 25 September 2016 at 11:55, אלעזר <elazarg@gmail.com> wrote:
I promised not to bother you, but I really can't. So here's what I felt I have to say. This email is quite long. Please do not feel obliged to read it. You might find some things you'll want to bash at the end though :)
Short-ish version:
1. Please consider disallowing the use of side effects of any kind in annotations, in that it is not promised when it will happen, if at all.
This may be part of the confusion, as Python is a language with a *reference implementation*, rather than relying solely on a documented language specification. Unless we specifically call something out in the language reference and/or the test suite as a CPython implementation detail, then "what CPython does" should be taken as the specification. While we're fairly permissive in allowing alternative implementations to deviate a bit and still call themselves Python, and sometimes alternate implementation authors point out quirky behaviours and we declare them to be bugs in CPython, "CPython correctly implements the Python language specification" is still the baseline assumption. So the order of evaluation for annotations with side effects has been defined since 3.0 came out:
def func(a:print("A1")=print("D1"), b:print("A2")=print("D2")) -> print("Areturn"): ... pass ... D1 D2 A1 A2 Areturn
That is, at function definition time: - default values are evaluated from left to right - annotations are evaluated from left to right
So that a change 3 years from now will be somewhat less likely to break things. Please consider doing this for version 3.6; it is feature-frozen, but this is not (yet) a feature, and I got the feeling it is hardly controversial.
I really have no interest in wasting the time of anybody here. If this request is not something you would ever consider, please ignore the rest of this email.
I don't think you're wasting anyone's time - this is a genuinely complex topic, and some of it relates to design instinct about what keeps a language relatively easy to learn. However, I do think we're talking past each other a bit. I suspect the above point regarding the differences between languages that are formally defined by a written specification and those like Python that let a particular implementation (in our case, CPython) fill in the details not otherwise written down may be a contributing factor to that Another may be that there are some things (like advanced metaprogramming techniques) where making them easy isn't actually a goal we necessarily pursue: we want to ensure they're *possible*, as in some situations they really are the best available answer, but we also want to guide folks towards simpler alternatives when those simpler alternatives are sufficient. PEP 487 is an interesting example of that, as that has the express goal of taking two broad categories of use cases that currently require a custom metaclass (implicitly affecting the definition of subclasses and letting descriptors know the attribute name they're bound to), and making them standard parts of the default class definition protocol. Ideally, this will lead to *fewer* custom metaclasses being defined in the future, with folks being able to instead rely on normal class definitions and those simpler extracted patterns.
2. A refined proposal for future versions of the language: the ASTs of the annotation-expressions will be bound to __raw_annotations__. * This is actually more in line to what PEP-3107 was about ("no assigned semantics"; except for a single sentence, it is only about expressions. Not objects).
PEP 3107 came with a reference implementation, it wasn't just the written PEP content: https://www.python.org/dev/peps/pep-3107/#implementation
* This is helpful even if the expression is evaluated at definition time, and can help in smoothing the transformation.
We talk about the idea of expression quoting and AST preservation fairly often, but it's not easy to extract from the archives unless you already know roughly what you're looking for - it tends to come up as a possible solution to *other* problems, and each time we either decide to leave the problem unsolved, or find a simpler alternative to letting the "syntactic support for AST metaprogramming" genie out of the bottle. Currently, the only supported interfaces for this are using the ast.parse() helper, or passing the ast.PyCF_ONLY_AST flag to the compile() builtin. This approach gives alternative implementations a fair bit of flexibility to *not* use that AST internally if it doesn't help their particular implementation. Once you start tying it in directly to language level features, though, it starts to remove a lot of that implementation flexibility.
3. The main benefit from my proposal is that contracts (examples, explanations, assertions, and types) are naturally expressible as (almost) arbitrary Python expressions, but not if they are evaluated or evaluatable, at definition time, by the interpreter. Why: because it is really written in a different language - *always*. This is the real reason behind the existence, and the current solutions, of the forward reference problem. In general it is much more flexible than current situation.
"More flexible" is only a virtue if you have concrete use cases in mind that can't otherwise be addressed today. Since you mention design-by-contract, you may want to take a look at https://www.python.org/dev/peps/pep-0316/ which is an old deferred proposal to support DBC by way of a particular formatting convention in docstrings, especially as special formatting in docstrings was one of the main ways folks did type annotations before PEP 3107 added dedicated syntax for them.
4. For compatibility, a new raw_annotations() function will be added, and a new annotations() function will be used to get the eval()ed version of them.
Nothing *new* can ever be added for compatibility reasons: by definition, preserving backwards compatibility means old code continuing to run *without modification*. New interfaces can be added to simplify migration of old code, but it's not the same thing as actually preserving backwards compatibility.
Similarly to dir(), locals() and globals(). * Accessing __annotations__ should work like calling annotations(), but frowned upon, as it might disappear in the far future. * Of course other `inspect` functions should give the same results as today. * Calling annotations()['a'] is like a eval(raw_annotations()['a']) which resembles eval(raw_input()).
I believe the last point has a very good reason, as explained later: it is an interpretation of a different language, foreign to the interpreter, although sometimes close enough to be useful. It is of course well formed, so the considerations are not really security-related.
Here you're getting into the question of expression quoting, and for a statement level version of that, you may want to explore the thread at https://mail.python.org/pipermail/python-ideas/2011-April/009765.html (I started that thread because I'd had an idea I needed to share so I could stop thinking about it, but I also think more syntactic sugar for metaprogramming isn't really something the vast majority of Python developers actually need) Mython, which was built as a variant of Python 2 with more metaprogramming features is also worth a look: http://mython.org/
I am willing to do any hard work that will make this proposal happen (evaluating existing libraries, implementing changes to CPython, etc) given a reasonable chance for acceptance.
I think the two basic road blocks you're running into are: - the order of evaluation for annotations with side effects is already well defined and has been since Python 3.0. It's just defined by the way CPython works as the reference implementation, rather than in English prose anywhere. - delayed evaluation already has two forms in Python (function scopes and quoted strings) and adding a third is a *really* controversial prospect, but if you don't add a third, you run into the fact that all function scopes inside a class scope are treated as methods by the compiler Stephen's post went into more detail on *why* that second point is so controversial: because it's a relatively major increase in the underlying complexity of the runtime execution model. The most recent run at it that I recall was my suggestion to extend f-strings (which are eagerly evaluated) to a more general purpose namespace capturing capability in https://www.python.org/dev/peps/pep-0501/ That's deferred pending more experience with f-strings between now and the 3.7 beta, but at this point I'll honestly be surprised if the simple expedient of "lambda: <f-string>" doesn't turn out to be sufficient to cover any delayed evaluation needs that arise in practice (folks tend not to put complex logic in their class bodies).
Stephen - I read your last email only after writing this one; I think I have partially addressed the lookup issue (with ASTs and scopes), and partially agree: if there's a problem implementing this feature, I should look deeper into it. But I want to know that it _might_ be considered seriously, _if_ it is implementable. I also think that Nick refuted the claim that evaluation time and lookup *today* are so simple to explain. I know I have hard time explaining them to people.
Most folks coming from pre-compiled languages like C++, C# & Java struggle with the fact that Python doesn't have separate compile time constructs (which deal with function, class and method declarations) and runtime constructs (which are your traditional control flow statements). Instead, Python just has runtime statements, and function and class definition are executed when encountered, just like any other statement. This fundamentally changes the relationship between compile time, definition time, and call time, most significantly by having "definition time" be something that happens during the operation of the program itself.
Nick, I have read your blog post about the high bar required for compatibility break, and I follow this mailing list for a while. So I agree with the reasoning (from my very, very little experience); I only want to understand where is this break of compatibility happen, because I can't see it.
This code works as a doctest today: >>> def func(a: "Expected output"): ... pass ... >>> print(func.__annotations__["a"]) Expected output Any change that breaks that currently valid doctest is necessarily a compatibility break for the way annotations are handled at runtime. It doesn't matter for that determination how small the change to fix the second command is, it only matters that it *would* have to change in some way. In particular, switching to delayed evaluation would break all the introspection tools that currently read annotations at runtime, both those in the standard library (like inspect.signature() and pydoc), and those in third party tools (like IDEs). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia