Before and after the colon in funciton defs.
(This idea may have been suggested before, because it seems too obvious to me.) How about if we remove the requirement that the colon be on the same line as the function name. And, what if the colon is then used to split the difference between the definition time, and call time code in function definitions. So that every thing before the colon is done at definition time. Everything after the colon is done at call time. def foo(...): """ doc string """ <function body> Then could become ... def foo(...) """ doc string """ # foo.__doc__ = """ doc string """ : <function body> I think this represents what is actually happening a bit better. One possibility for define time code is to have decorators listed that read in the order they are applied instead of bottom up. def foo(n) """ function to be decorated. """ @deco1 # foo = deco1(foo) The '@' notation still works. @deco2 # foo = deco2(foo) : <function body> Note, that putting the doc string after the decorators may be better as it would put the doc string on the decorated function instead of the original. I think there may be more things possible with this idea than the simple cases above. Cheers, Ron
On Sat, Sep 17, 2011 at 7:30 AM, Ron Adam <ron3200@gmail.com> wrote:
I don't see any benefit in offering a second way to spell docstrings or decorators (the existing spellings are easy enough to remember), but this suggestion did prompt an idea for potentially replacing uses of the default argument hack: allow a sequence of 'implicit locals' to be defined within square brackets after the parameter list. (This idea may have come up before, but if it has, I don't recall the arguments against it). Running through the 3 at least arguably legitimate use cases for the default argument hack: Micro-optimisation: # Current Python (internal to functools.lru_cache with default argument hack) def decorating_function(user_function, tuple=tuple, sorted=sorted, len=len, KeyError=KeyError): ... # 60 line function # Proposal - explicitly implicit locals to clarify real calling signature def decorating_function(user_function) [tuple=tuple, sorted=sorted, len=len, KeyError=KeyError)]: ... # 60 line function (Note: in practice, the lru_cache usage of the default argument hack doesn't really impact introspection, since the function is short-lived - it is the decorating function returned by the lru_cache decorator factory and hence only exists briefly as part of the function definition. The signatures of lru_cache itself and of the decorated function are not affected. I still like it as an example, since it isn't hypothetical - the standard library really uses the default argument feature this way). Early binding in a loop: # Current Python adders = [] for i in range(10): def f(x, _i=i): return x + _i adders.append(f) # Proposal adders = [] for i in range(10): def f(x) [i=i]: # Real calling signature is clear return x + i adders.append(f) Algorithmic shared state (without a class): # Current Python def f(*, _cache=[]): # Consenting adults, please don't override _cache when calling! # Proposal def f() [_cache=[]]: # Real calling signature is clear # _cache is an implicit local, not a keyword-only argument, so its safe To prevent grammar ambiguity, the list of implicit locals would most likely need to appear before the annotation for the function return type. The syntax within the list would be the same as those for arguments with default values (while I don't have a use case for allowing annotations, I don't have any reason to prevent them either, so reusing the same grammar fragment seems like reasonable idea) The implementation would also be basically the same as for arguments with default values (a list of names stored as an attribute on the code object and a dictionary on the function object to populate them) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, 2011-09-17 at 11:01 +1000, Nick Coghlan wrote:
I almost included an example similar to this. Using your adder example, it would have looked something like this. adders = [] for i in range(10): def f(x) # Real calling signature is clear f.initial_values(i=i) # done at def time : return x + i adders.append(f) Of course there is no way to preinitialize a functions name space. Your [i=i] notation would do something like that behind the scenes. I was thinking it would be good if the parts in front of the colon were valid python statements except in the case of the already special docstrings and decorators. It could be limited to only valid python commands. If it turns out that a particular command becomes very common, a special syntax could be considered. Having the colon on a separate line would be optional, so it wouldn't change existing code at all. I tried to find a way to use a decorator to do this and didn't find anything that worked nicely. It can be done nicely with class's, but the point of using a function is it would be more efficient. I think this also starts to get into the area of meta programming. In python library there are a number of functions of the form. def foo(*args, **kwds): return _foo(*args, **kwds) I presume there is a reason why they didn't just do foo = _foo. currently you can swap out the __code__ part of a function, but not the signature part. If you could, then this might be possible. def foo() # Using _foo's signature. foo.signature = _foo.signature foo.__code__ = _foo.__code__ : pass # using _foo.__code__ Cheers, Ron
On Sat, Sep 17, 2011 at 3:47 PM, Ron Adam <ron3200@gmail.com> wrote:
You did say that, but advanced no arguments in favour of breaking with the language aesthetic that way. Indented suites are introduced by a colon on the preceding header line and you'd need a really compelling argument to change that. "It might be nice" doesn't cut it. Specifically targeting existing uses of the default argument hack, on the other hand, comes with at least 3 already known use cases (i.e. name lookup micro-optimisation, early binding semantics and shared state for algorithms). People already tolerate messing with the function signature to obtain those behaviours, so it's reasonable to wonder if it is possible to provide a cleaner way to obtain the same effect. A simple set of definition time name bindings would likely suffice without greatly increasing the language complexity or the runtime overhead of function calls. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Sep 17, 2011 at 2:16 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Since the goal is to add some context, like a wrapper, it seems reasonable to use a specialization of @. @+ suggests adding something to the environment, and won't be ambiguous unless numbers become callable. @+tuple=tuple @+sorted=sorted @+len=len @+KeyError=KeyError def decorating_function(user_function): I would personally be fine with a restriction to @+name=<expr> but I suppose others might prefer a tuple, like @+(tuple, sorted, len, KeyError)=(tuple, sorted, len, KeyError) def decorating_function(user_function): -jJ
On Sun, 2011-09-18 at 15:32 -0400, Jim Jewett wrote:
If there was a method on a function object to set default values, then you could use a regular decorator. # Most likely to live in functools def preset(**kwds): def apply(f): f.set_default_values(kwds) return f return apply #--------------------------------- from functools import preset @preset(tuple=tuple, sorted=sorted, len=len, keyerror=keyerror) def foo(...): ... I was hoping there might be a more direct way to do it. My previous idea breaks the white space consistency, and Nicks idea adds special syntax. Decorators with long arguments make the function name stand out less. Is there a more recent patch for Pep 362? Pep 362 seems to be targeted at only introspection and really isn't a part of the function object. Is that correct? Cheers, Ron
I think something similar could be achieved with annotations, as in the following example. This is incomplete, and probably full of errors, but it gives the idea. def setlocals(func): def wrapper(*args, **kwargs): call_kwargs = [] for k, v in kwargs.items(): if k in func.__annotations__ and func.__annotations__[k] == 'local': raise AttributeError("Can't set argument") else: call_kwargs[k] = v return func(*args, **call_kwargs)
On Sep 18, 2011 10:59 PM, "Ron Adam" <ron3200@gmail.com> wrote:
On Mon, Sep 19, 2011 at 5:44 PM, David Townshend <aquavitae69@gmail.com> wrote:
Proposals that are uglier than the default argument hack itself (and I put both annotations+decorator and the '@+' idea in that category) just aren't worth the hassle. The *only* change even potentially worth making in this area would be one that: - looks good (i.e. is easy to read) - is easy to remember and write - doesn't hurt call time performance - addresses all the default argument hack use cases - doesn't have the downside of altering the function's externally visible signature My initialised locals idea *might* qualify (although I've yet to see anyone make a serious attempt at poking holes in it, so I'm not convinced of that), but the other suggestions in this thread haven't really offered anything to recommend them over the existing approach. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I agree, both are ugly (btw, mine wasn't a proposal, just an illustration that it is possible with the status quo). One question I have is would the locals be local to a specific call or the the function object? I don't know enough about the python language internals to know the correct terminology to explain what I mean, but from a users point of view, would the locals be reset at every call? The fact that they are locals implies that they don't exist outside a function call, but assigning them in the function definition implies that they exist within the function object in a similar way to the value of keyword arguments. David On Mon, Sep 19, 2011 at 11:35 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Nick Coghlan wrote:
[...] allow a sequence of 'implicit locals' to be defined within square brackets after the parameter list.
I'm not quite sure whether the benefits of this proposal are worth the further rococoization of the language. As far as I can see, the only benefit is keeping function signatures clean. Looking at the use cases one by one:
The proposed syntax isn't explicit enough for my taste to justify the change. What the programmer really wants to tell the compiler is: Bind the given names at compile time rather than at run time. I'd prefer if the syntax would reflect this intention, similar to the "global" and "nonlocal" declarations: def decorating_function(user_function): earlybind tuple, sorted, len, KeyError (Not being a native speaker of English, I don't even try to find a less dull name for this than "earlybind". And I'm quite aware that the bar for the introduction of a new keyword is quite high.)
I don't see too much benefit of the proposed syntax for this use case. If f() is a local throw-away function, I wouldn't worry about its signature. If f() is a longer-lived object and I do care about its signature, I'd uses a class: class Adder: def __init__(self, i): self.i = i def __call__(self, x): return x + self.i [...] adders.append(Adder(i)) I still think classes are the Python way to hide state, not closures. That said, this case would also be covered by the "earlybind" porposal above.
Again, I'd use a class to hide state in the first place. If someone really wants to avoid using a class for some reason, using function attributes would also be a viable way: def f(): try: cache = f._cache except AttributeError: cache = f._cache = [] Cheers, Sven
On Mon, Sep 19, 2011 at 11:22 PM, Sven Marnach <sven@marnach.net> wrote:
The thing is, real function objects genuinely *are* special. They have a privileged place in the interpreter, the inspect module and other introspection tools know more about how to deal with them, they have instance method descriptor behaviour built in, etc. Switching from a real function object to a custom class with a __call__ method is genuinely expensive in terms of the difficulty of writing the code in the first place, as well as in being able to read it later. Your 'Adder' example above, for instance, doesn't implement the descriptor protocol, so will behave like a staticmethod when placed in a class. That may be what you want, but it isn't easy to get instance method behaviour in the case where you would prefer that. $ python3 -m timeit -s "from call import Adder; f = Adder(1)" "f(5)" 1000000 loops, best of 3: 0.362 usec per loop $ python3 -m timeit -s "from call import Adder; f = Adder(1).__call__" "f(5)" 1000000 loops, best of 3: 0.222 usec per loop $ python3 -m timeit -s "from closure import adder; f = adder(1)" "f(5)" 10000000 loops, best of 3: 0.174 usec per loop $ python3 -m timeit -s "from default import adder; f = adder(1)" "f(5)" 10000000 loops, best of 3: 0.166 usec per loop When what you're trying to express is a single short algorithm, overriding __call__ isn't even in the contest - we aren't talking minor percentage differences in call overhead, we're talking more than double. You can gain a fair bit of that back by retrieving __call__ as a bound method in advance, but really, your algorithm needs to be complex enough for the difference in call overhead to be trivial before implementing __call__ becomes an attractive alternative to using a closure Now, if there are *multiple* algorithms operating on the same data, then obviously you want a class with multiple methods. But in that case, you're less likely to bless any one of them with privilege of occupying the '__call__' slot. Basically, classes make sense when the state is the most important thing, while functions focus on a specific algorithm. For the special case of "single algorithm with some associated state", a closure (or the default argument hack) will often be a better modelling tool than a class. (Obviously, anyone that disagrees with me on this point will consider this whole thread silly - however, the popularity of closures for the implementation of factory functions, including decorator factories, shows that there is plenty of code out there that happily follows this approach) With closures vs the default argument hack, the performance and introspection arguments don't apply - in both of these cases you have a real function, so the only trade-off is between forcing readers to understand how closures work and forcing them to ignore additional arguments that are there just to prepopulate the local namespace with some data. However, the closure approach has some genuine downsides from a readability point of view. The real function when using a closure is the inner one. The outer function definition, the trailing return statement and the invocation of the outer function are all boilerplate that obscures the actual purpose of the code. You can tidy some of that up with a decorator, but you can't avoid the need for the nested function in order to create the closure. And that's why people use the default argument hack today - they weigh those downsides up against the consequences of having a bit of noise in the function signature (as seen by introspection tools) and decide they're happy to accept that trade-off in return for a simple and straightforward expression of a single algorithm with some associated state. Various ideas for addressing this have been proposed in the past. PEP 3150's statement local namespaces are one, as are the assorted incarnations of the proposal for flagging arbitrary expressions within a function for evaluation at definition time rather than runtime (search the python-dev and python-ideas archives for phrase like 'once statement' and 'atdef' - alas, nobody has stepped forward to codify those ideas into a PEP). Where those proposals all come unstuck is that they try to do more than the default argument hack allows, *without compelling use cases to guide the additional semantics*. The pre-initialised locals concept deliberately avoids that problem by targeting exactly the use cases that are *already* supported via the default argument hack, just in a way that tries to avoid the negative effects on introspection and readability. Cheers, Nick. P.S. Code for the timeit runs: $ cat > call.py class Adder(object): def __init__(self, i): self.i = i def __call__(self, x): return x + self.i $ cat > closure.py def adder(i): def f(x): return x + i return f $ cat > default.py def adder(i): def f(x, _i=i): return x + _i return f -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, 2011-09-20 at 08:23 +1000, Nick Coghlan wrote:
Heh, On an abstract level, I do know there is something special about function objects. Putting my finger on it is a bit harder. Trying a few things to see what happens... At some point there does need to be a base object that can't be taken apart into it's smaller parts. That normally would be the object class. But with functions, we are referring to the base *executable* object. A function is is a union of other basic objects that can then be executed. (I think there can be some improvements in how a function object is constructed and organized.) I guess the real *special* magic is the byte code, CALL_FUNCTION, which knows how to use a function object. Along with the other Function related bytecodes.
Umm... "...default arguments hack allows,..."? Not well said, I think you meant, "...unstuck, by doing more than is needed...". Even so, I do understand your point you are trying to get across. Would your suggestion add any additional methods, or attributes to a function object? Would it need any additional byte codes? An alternative proposal, I came across the other day... don't remember exactly where, is to be able to put default arguments after the **kwds, parameter.
It's currently a syntax error. But that requires using an ** someplace. I don't really like this one, as I think function arguments are already hard enough to understand. For a language that is suppose to be easy to understand, that's not a good thing. Cheers, Ron
Ron Adam wrote:
I'd say it's more in the function object's __call__ method, which invokes the C code that knows how to interpret bytecodes. The CALL_FUNCTION bytecode just calls the __call__method of the object being called, whatever it happens to be. The function object's __get__ method also plays a part when a method call is involved, as does the __getattribute__ method of the type object. The magic tends to be spread around a fair bit in Python. -- Greg
Nick Coghlan schrieb am Di, 20. Sep 2011, um 08:23:14 +1000:
I actually *do* use the default argument hack for early binding myself. My main points were that the current status quo isn't too bad, there are alternatives if the function signature is really important, and that the cases which require both using closures *and* having a clean signature are too rare to justify yet another confusing-for-beginners syntax. (When I started learning Python in 1998, the language was praised for having a small language core. I didn't hear this very often in more recent times.) Assuming a new syntax *is* necessary -- there is no objective way to decide this after all -- I don't particularly like the proposed square bracket syntax because a) it puts the information at the wrong place. The first line of the function definition is for the outside appearance of the function, including signature and annotations. Separating the internal information on early bound variables from the signature is the primary goal of the proposal after all. b) it does not make the intention of early binding very clear. c) it consists of a list of the form [i=i] or [tuple=tuple, len=len] in most use cases, making it necessary to type every name that shall use early binding twice. That's why I would prefer a syntax that explicitly declares the names that should use early binding, similar to "global" or "nonlocal" statements, if such a syntax is deemed necessary at all.
How does PEP 3150 target the use cases of the default argument hack given in your original mail? Cheers, Sven
On 9/20/2011 10:00 AM, Sven Marnach wrote:
I have noticed the same. The big syntax additions are generators, comprehensions, decorators, and the with statement. The switch to unicode doesn't change syntax but can complicate text processing.
The problem with an earlybind statement in the body is that it is in the body, which should be runtime stuff. I would prefer something in the header. A semi-colon or other char would be sufficient syntactically: def f(a; len, alist=[]): alist.append(a); return len(alist) where a bare identifier like 'len' *means* 'len=len'. -- Terry Jan Reedy
On Tue, Sep 20, 2011 at 9:59 AM, Terry Reedy <tjreedy@udel.edu> wrote:
I would be much more interested in a tweak to the language semantics where the compiler is allowed to assume that "len means len" if there is no global assignment to it in a module. There are many past threads about this topic. It should make the feature proposed here unnecessary -- at least its use for manual micro-optimizations, which I mostly find an offense to the reader (even in the shortened form proposed here). -- --Guido van Rossum (python.org/~guido)
On 9/20/2011 1:09 PM, Guido van Rossum wrote:
On Tue, Sep 20, 2011 at 9:59 AM, Terry Reedy<tjreedy@udel.edu> wrote:
For those not party to the earlier threads, this alternate proposal would bread any code that monkeypatches an imported-module-only override of a builtin, something like: def mylen(o): return <whatever> import mod; mod.len = mylen I do not mind the semantic change itself, and it would benefit nearly all code, including module-level code would not be affected by def gimmicks. Some issues I see that perhaps were discussed, but I do not remember: 1) module.__setattr__ should then raise an exception at such attempts; 2) it makes the language more context sensitive than it currently is; 3) compile() should apply the rule even for single statements or even expressions. -- Terry Jan Reedy
On Wed, Sep 21, 2011 at 1:36 PM, Terry Reedy <tjreedy@udel.edu> wrote:
The two sticking points tend to be open() and print(). Overriding those externally can be *incredibly* useful for testing code that uses them, as well as in interacting with code that wasn't designed in a flexible way. Regardless, I'm going to stop mentioning the manual micro-optimisation use case for the default argument hack. Alex Gaynor pointed out that that aspect is completely irrelevant on PyPy, and as Guido notes, there are likely other ways to tackle name lookup optimisation even in CPython. That still leaves two arguably valid use cases: - early binding for name lookups - an algorithm with shared state (kind of like an inverted approach to creating a class with a custom __call__ method) Perhaps a non-syntax way to approach both of these would be to add a new decorator to 'functools': def closure(f): """Invokes the decorated function and returns the result after transcribing essential function metadata This can be used to easily share algorithm state and get early binding semantics for names. """ impl = f() impl.__name__ = f.__name__ doc = f.__doc__ if doc is not None: impl.__doc__ = doc impl.__dict__.update(f.__dict__) return impl This would be used as follows: @functools.closure def adder(i=i): # 'impl' defines call time signature "Increments 'x' by adder.value" def impl(x): impl.call_count += 1 return x + i impl.value = i impl.call_count = 0 return impl
Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 2011-09-21 at 15:19 +1000, Nick Coghlan wrote:
Simplifying things like this is one of the use cases of allowing define time statements. That's a lot of work to just avoid putting a keyword in the signature. And it's not easy to understand. Decorators could be a good way to do this, but the problem in these cases, is the function object doesn't have the needed support to make things like this easy. Probably the easiest and most direct way, would to be to add a new keyword 'static' as MRAB suggested, but have it be an expression instead of a command. value = (static <expression>) def adder(x): return x + (static i) # evaluate (static i) at compile time. The parentheses would be optional. The (static i) expression, could be spelled (i=i). I think that was what Guido was suggesting, but not as an expression. As an expression, you would then see things like.. i = (i=i). But that may only be poor style, because it's easy enough to just not do that. I think the expression form is better myself, it allows you to get both the compile time value, and the current value of an identifier. def value elapsed_frames(): """ Where f is the frame counter in the parent scope. """ return f - (static f) Cheers, Ron
When you say "compile time", do you literally mean "when the compiler creates the bytecode" or do you really intend this to be computed at function definition time (the same time when default values are evaluated)? The latter seems to make more sense. --Guido On Wed, Sep 21, 2011 at 11:12 AM, Ron Adam <ron3200@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On 21/09/2011 19:34, Guido van Rossum wrote:
It would be at function definition time.
def fib(n, known={}): if n in known: return known[n] if n < 2: f = n else: f = fib(n - 1) + fib(n - 2) known[n] = f return f to: def fib(n): known = static {} if n in known: return known[n] if n < 2: f = n else: f = fib(n - 1) + fib(n - 2) known[n] = f return f This: known = {} creates a dict and binds a name to it at execution time, whereas this: known = static {} creates a dict at definition time, storing a reference to it in the function object, and binds a name to it at execution time. I suppose it's like a class variable: class Foo: known = {} which is shared by its instances.
On Thu, Sep 22, 2011 at 8:38 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
And so we come full circle... it looks like Jan didn't have the time to write up a PEP following the last discussion of this topic [2], so perhaps you'd be interested in taking up the task? FWIW, I still quite like the explicit 'atdef' keyword I suggested last time [1] (and most of the arguments from that thread against the 'after-the-**' syntax are equally valid arguments against the '[] in the function header' syntax from this iteration). [1] http://mail.python.org/pipermail/python-ideas/2011-June/010565.html [2] http://mail.python.org/pipermail/python-ideas/2011-June/010552.html Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I hope I'm not bikeshedding here, but I much prefer decorators because they let you do more than just micro-optimization. They were mentioned before, in another thread, and I find them attractive. A decorator basically lets you inject locals into already-existing functions. So if you want to, e.g., test a function that uses a global "urlopen" function to download a file, by replacing urlopen. from urllib import urlopen from StringIO import StringIO def myfunction(): return urlopen('...').read() class TestMyFunction(unittest.TestCase): def setUp(self): self.func = inject(urlopen=lambda s: StringIO(s) def test_opens(self): self.assertEqual(self.func(), '...') There's also the benefits of not altering the language etc. -- it's just a function that returns a new function that's very similar to the old one. That simplicity appeals to me as well. And it still satisfies the use-case of the new "static" keyword. Devin On Wed, Sep 21, 2011 at 2:12 PM, Ron Adam <ron3200@gmail.com> wrote:
Devin Jeanpierre wrote:
I find this idea extremely exciting. But have I missed something? Does this inject function actually exist somewhere? If I recall correctly, I've seen a bytecode hack or two that seemed to work, but I didn't think that was officially supported. Comparing syntax: def func(a, b): static f, g, h f = ... g = ... h = ... return f(g(h(a+b))) @inject( f = ... g = ... h = ...) def func(a, b): return f(g(h(a+b))) or even: def func(a, b): return f(g(h(a+b))) another_func = @inject( f = ... g = ... h = ...)(func) As far as syntax goes, I think both variants look good, but a decorator has two major advantages: * you aren't limited to using it at function definition time, you can apply it after the event * it doesn't need to be a keyword, or even a built-in: it could go into functools -- Steven
On Wed, Sep 21, 2011 at 1:19 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
With the right tooling, even given these semantics, we could monkeypatch by getting in at module runtime. One system that uses an idea suitable for this is exocet, which was introduced with this blog post - http://washort.twistedmatrix.com/2011/01/introducing-exocet.html . I'm don't think exocet today is appropriate for use, but its basic ideas could make for much more controlled tests, and become necessary in the face of these optimizations. Mike
On 20/09/2011 17:59, Terry Reedy wrote:
To me that's like a 'static' declaration in C++, which in Python might be something like: def f(a): static len=len static alist=[] alist.append(a) return len(alist) What follows 'static' would be an assignment which is executed when the function is defined, much like a default argument, and would be shared between invocations in the same way.
I'm adding both new propositions: 1. the `def foo(...) [len=len]:` syntax, 2. the `len = (static len)` expression syntax -- to the PEP-draft I'm preparing -- which Nick suggested in June [http://mail.python.org/pipermail/python-ideas/2011-June/010569.html] (I'm sorry that that preparing lasts so much time, but my everyday- -activity-CPU has been overloaded a bit for a few months...). Ad 1: I think it's better than the `after-**` proposition from June, though still has some its drawbacks (Sven just mentioned some of them). Ad 2: I must admit that this one becomes my favorite syntax for early binding (though I don't like the abbreviated form '(i=i)'). IMHO it's not only clear (no all that questions about assignment semantics) but also elegant, explicit and consistent with some existing syntax constructs (especially with `yield EXPR`). Note that (as you can use any expression) it makes possible to use e.g. tuples of several names: spam, print, open = static (len, print, open) Big +1 from me. 3. Another variant could be with a colon (a bit similar to the lambda syntax): len = static: len spam, print, open = static: (spam, print, open) or even (reusing the existing keyword): len = def: len spam, print, open = def: (spam, print, open) (here "def" means: "at definition time") But I'd rather prefer the #2. Regards. *j
On Wed, Sep 21, 2011 at 7:01 PM, Jan Kaliszewski <zuo@chopin.edu.pl> wrote:
Agreed, though the the keyword should be on the LHS: static spam, print, open = len, print, open It's more consistent with other keywords that way and easier to spot when reading code. Also, "static" as a keyword is in use as an identifier in a number of places per google's codesearch (on the order of 100) including in Django. It's not like it's in super broad use, but it is in use. -eric
On Thu, Sep 22, 2011 at 12:46 PM, Chris Rebert <pyideas@rebertia.com> wrote:
+1 (FWIW) on this "assignment statement with leading keyword" variant. `static` is an excellent choice of keyword for this.
Care to elaborate? This has always struck me as one of the more nonsensical keyword choices in C, so I'd like to see some solid justification before perpetuating it in Python (particular when C/C++ intuitions about statics being process global will hinder rather than help in understanding the scope of this new namespace). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 2011-09-21 at 20:31 -0600, Eric Snow wrote:
The above could also be ... spam, print, open = (static len, print, open) As Jan pointed out, it would be very similar to how the 'yield' is used. It doesn't do anything to the identifiers on the left side. The only thing the static expression does is save a reference to what ever the expression part on the right evaluates to at the time the function is defined. Later, when the function is called, is when it's actually used. At define time... spam, print, open = static (len, print, open) Becomes this at call time... spam, print, open = _tuple #tuple created by static expression Cheers, Ron (Apologies for any double or delayed posts, I'm still trying to get some email glitches worked out.)
Well, now that we're getting down to picking bike shed colors, time to chime in: blah = static blah is worse than static blah = blah because it's easier to visually pick out something on the LHS and it's analogous to global/nonlocal. Anything having to do with parens is visually ugly and should be avoided unless absolutely needed for explicitness. One question I have is how this will work with control flow. This example is works with Python 2.7 and 3.2:
Presumably static would work the same way? Still, I find this sort of distasteful, since the behavior is a bit surprising to those not familiar with the in's and out's of the compilation model. Why not put the static declaration in the argument list, something like: def f(static l=[]): And maybe as a shortcut for static NAME=NAME, use just static NAME. fs = [] for i in range(10): def f(static i): return i fs.append(f) Looks OK to me.
On Thu, Sep 22, 2011 at 2:21 PM, Carl Matthew Johnson <cmjohnson.mailinglist@gmail.com> wrote:
Presumably static would work the same way? Still, I find this sort of distasteful, since the behavior is a bit surprising to those not familiar with the in's and out's of the compilation model.
There's a reason style guides say not to do that with global and nonlocal, even though the language technically allows it. Definition time variable declarations would work the same way.
Why not put the static declaration in the argument list, something like:
def f(static l=[]):
For all the same reasons people don't like the 'after **' and '[] in the function header' suggestions. Most notably, the fact that the evaluation of certain expressions at definition time is an implementation detail rather than part of the public API, so it doesn't really belong in the header. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 2011-09-21 at 18:21 -1000, Carl Matthew Johnson wrote:
Which way, will depend on what kind of behavior is decided on. Statement? or Expression? As a statement, it would be first and not have the assignment. static blah (if it does, then working out how that works will be tricky.) As an expression, the only thing that matters is that the names used in the expression, are findable at define time. No assignment is needed. The value takes the place of the static expression when the function it is in is called. Here the expression "(static y)" is replaced with the object y was bound to at the time the function was defined. y = ['red', 'blue', 'green'] ... def f() ... for x in (static y): ... The parentheses are optional if the expression only has single value. But I think it makes it easier to see what is being done. An expression you can be used in lambdas. A statement can't. Static statement: def get_adders(count): adders = [] for n in range(count): def add_it(x): static n return n + x adders.append(add_it) return adders Static expression: def get_adders(count): adders = [] for n in range(count): adders.append(lambda x: return (static n) + x) return adders
I can see why that is surprising. I don't think the static would have this issue in either form. Cheers, Ron
On Thu, Sep 22, 2011 at 3:47 AM, Ron Adam <ron3200@gmail.com> wrote:
As an expression, the only thing that matters is that the names used in the expression, are findable at define time.
Not just that the names are findable, but that they are already bound to something. def f(a): b=a static c=b should fail, and should probably be a syntax error to prevent it apparently working (but perhaps with the wrong value) when (and only when) a more global b was set earlier.
No assignment is needed.
Agreed; in particular, static imports should reduce the cost of importing from inside a function body. (But then will people effectively start spelling "import" as "static import" even at the top of a module?) -jJ
FWIW, I'm at best +0 on doing anything here; I'm -1 on the expression-style (static <expr>) form but could live with the statement (whether or not combined with assignment -- IIRC that was planned for nonlocal and I don't see why we couldn't upgrade nonlocal and global at the same time). The main argument for static would be that this is what all other relevant languages call it. If we really want to do something else, I recommend reviving Algol-60's "OWN" keyword. -- --Guido van Rossum (python.org/~guido)
On Wed, Sep 21, 2011 at 9:44 PM, Ron Adam <ron3200@gmail.com> wrote:
Yeah, "static" on the LHS does imply a different meaning. Perhaps "static" is the wrong keyword, for all the baggage it carries. And a real "static" statement may not even be the right solution[1]. We're looking at two approaches here: do it in the function "header" or in the body. Here're some pros and cons: In-header Pros: - implies definition time In-header Cons: - limited space so more info adds clutter - could be mistaken as part of function's signature In-body Pros: - less cluttered - more closely associated with function locals? In-body Cons: - easy to miss that it's a definition-time directive (particularly if on RHS of assignment) - evaluated expression must not reference any of the function's locals We definitely want the solution to be explicit that it's a definition-time behavior. If the new syntax is in the body then it'll have to be especially obvious. Either way, the new syntax should clearly isolate the expression and associate it with a name to which it will be bound in the locals. All things considered, I don't think the that RHS static expression fits the bill. The right solution seems to be pretty elusive here... As an aside, what would take precedence in the case of a name collision: default arguments or the new syntax? -eric [1] If a static statement had an assignment clause for the initial value, then it would work. However, it doesn't seem explicit enough that the value is evaluated at definition time. [2] Ideally we would just use attributes on the function object. However, this has a number of problems including performance and the ambiguity of to which object the function name points. (I have a patch in issue 12857 that addresses exactly that.)
Eric Snow wrote:
We're looking at two approaches here: do it in the function "header" or in the body. Here're some pros and cons:
There's a third approach, although it would probably take a bit more implementation effort: a decorator. # Demonstrate early-binding, micro-optimization and monkey-patching # in a single call: @inject(a=some_value(), len=len, sum=my_sum) def func(x): ... The downside is that the decorator gets called after the function is already created, therefore it would have to take an existing function and make a modified copy of it. This may be harder than doing the "static" bindings while the function is being assembled in the first place.
In-header Pros: - implies definition time
Likewise for the usual @ decorator syntax.
Neither apply to decorator syntax.
Decorator is also less cluttered, but not so much obvious what it does.
Neither of these is a problem for a decorator: using the @ syntax, it is clearly and obviously definition-time. And because the inject decorator is called outside the function, it's obvious that the bindings can't access the function locals. -- Steven
Eric Snow dixit (2011-09-21, 20:31):
But then you lose important advantages of the expression variant -- especially that assignment is not a part of the syntax construct so it not only more powerfull but also more explicit and clear for newcomers: any assignment is at run-time and normal assignment rules apply to it. A bit like in case of `yield EXPR`. Cheers. *j
On 22 September 2011 09:11, Jan Kaliszewski <zuo@chopin.edu.pl> wrote:
A concern I have with the expression variant is that I don't understand what it would mean except in the restricted contexts it's been discussed in. Can you describe the semantics of static EXPR in isolation, so that we're not just interpreting it in terms of its use in an assignment, and can understand the wider implications? Thanks, Paul.
On Thu, Sep 22, 2011 at 6:21 PM, Paul Moore <p.f.moore@gmail.com> wrote:
It isn't really that different from the statement version - a definition time expression would be calculated at definition time by the interpreter and the resulting value cached on the function object. At execution time, the cached value would be used in place of the original expression. (As an implementation detail, this would likely work by assigning such expressions an index in the function's closure list and populating them at definition time as cells) To use the classic "adder in a loop" example: # default argument hack adders = [] for i in range(1): def adder(x, i=i): return x + i adders.append(adder) # default argument hack (lambda) adders = [(lambda x, i=i: x + i) for i in range(10)] # Definition time statement (lambda variant not supported) adders = [] for i in range(1): def adder(x): atdef i=i # Could conceivably shorten this to just 'atdef i' return x + i adders.append(adder) # Definition time expression (using same parenthesis rules as yield expressions) adders = [] for i in range(1): def adder(x): return x + (atdef i) adders.append(adder) # Definition time expression (lambda) adders = [(lambda x: x + (atdef i)) for i in range(10)] I think there's a case to be made for the expression version - it's likely to require less boilerplate than the statement version in simple cases and is compatible with comprehension syntax (since it can be embedded inside a lambda expression). The analogy with yield expressions is a reasonable one - but instead of coming from send(), the result of the expression is retrieved from the cache on the function object. The other advantage of the expression version is that it avoids the issue of definition a new namespace where names can be looked up. Instead, it's just a mechanism for caching the values of certain expressions at definition time - at execution time, those values can then either be used directly or else assigned to an ordinary local variable. I wasn't initially a fan of the idea, but it's growing on me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 22 September 2011 14:58, Nick Coghlan <ncoghlan@gmail.com> wrote:
OK, I think I get it now. So the example a, b, c = atdef ([], {}, set()) would create and cache a tuple at definition-time, then retrieve and unpack it at runtime. Which means that it has the same effect as a = atdef [] b = atdef {} c = atdef set() with some slightly subtle differences in detail (there's only one cached "value" rather than 3, but it'd be hard to detect that in practice. I think that based on your definition, all my "but what about X" pathological examples have reasonably sane behaviours (you need a good understanding of what "definition time" actually means, but once you internalise the fact that def is an executable statement, that isn't too hard). I can see the attractions of the idea but: * I'm sorry but the keyword atdef is ugly. The only proposal I like is static, but I agree that it only makes sense to someone with a C-type background. * I remain unconvinced that the problem is severe enough to warrant a new keyword. * The semantics, although clear and easy to state, are probably one more step away from Python being a simple language. I can imagine beginners' heads exploding...[1] Paul. [1] Although anyone needing the semantics is arguably no longer a beginner :-)
On 22 September 2011 15:34, Paul Moore <p.f.moore@gmail.com> wrote:
I spotted one that might still be worth mentioning, though. a = 10 def foo(): a = 20 b = static a What should that mean? Clearly b isn't 20, as a wasn't set to 20 at function define time. But b = 10 (on the assumption that "static a" takes the value of a at define time) will confusing to users, I would suggest. Not allowed might be best, but I suspect it could be hard to implement. But as Guido's -1 on the expression form, this is probably moot. Paul.
Paul Moore wrote:
Also hard to specify without excluding one of the intended use cases: def foo(): len = static len or in the statement version, def foo(): static len = len I think we'd just have to accept that the RHS is evaluated outside the function scope, despite being written within it. That's how static initialisers in C work. -- Greg
On Fri, Sep 23, 2011 at 9:51 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Please read the previous thread from June (linked earlier in this thread). Decorator syntax cannot work without deep magic, because the compiler *doesn't know* that injected names need to be given special treatment. Python's scoping relies on the compiler being able to classify names at compile time into 3 kinds of reference: - local (direct references into the local variable namespace of the executing frame) - cells (indirect references via cells stored on the function object) - unknown (looked up by name at runtime, first in the module globals and then in the builtin namespace) These 3 reference types are baked into the immutable code objects by the compiler - you *cannot* change them later without hacking the bytecode and recreating the function object. Now, we have two 'magical' names ('super' and '__cell__') that cause the compiler to spontaneously do interesting things with namespaces to make Python 3's new simplified (and incredibly convenient) super() invocation work. However, aside from that special case, the rules are very simple: - names bound in the current function are locals (unless marked with 'nonlocal' or 'global') - names bound as locals in an outer function and referenced from the current function are looked up via cells - anything else is treated as an unknown name The 'nonlocal' and 'global' keywords override the 'local by default' behaviour for bound names (forcing the second or third interpretations respectively). The default argument hack effectively creates a 4th namespace option by using the default arguments as "pre-populated locals" - the argument passing machinery is set up so that any parameter not supplied as an argument is filled in on the current frame from its default argument value. By adding additional parameters that are *never* supplied as arguments, the author of a function can create arbitrary locals from expressions that are evaluated when the function is defined rather than when it is called. That means there are four very different ways of looking at potential replacements for this technique: 1. Leave the technique alone, but improve the introspection tools and conventions associated with it def f(x, _i=i): # pydoc would, by default, display the signature as 'f(x)' return x + _i Keyword-only arguments in Py3k already help with this approach to the question, especially when the 'hidden' keyword is prefixed with an underscore to indicate it isn't meant for public consumption. This approach is also highly amenable to monkey-patching, since the default arguments can be deliberately overridden at call time, just like any other parameter. It wouldn't be hard to adjust pydoc to leave out underscore-prefixed keyword only parameters by default, requiring an explicit request to include them. In other words, this approach just involves taking the existing default argument hack, tidying it up a bit, explaining it in the docs, and blessing it as the official way to do things and a technique that experienced Python programmers should know and understand. 2. Definition time parameters This approach keeps the pre-populated locals in the function header, but tweaks the spelling and storage so they're no longer part of the function signature. Two ideas have been put forward for this approach: def f(x, **, i=i): # extending the keyword-only syntax one step further return x + i def f(x) [i=i]: # adding a dedicated set of brackets return x + i The general consensus seems to be that these don't offer enough benefit over the status quo to be worth the hassle. 3. Definition time expressions With a wide variety of proposed spellings (e.g. once, static, atdef), this proposals aims to mark individual expressions for evaluation at function definition time and caching on the function object. At function call time, the value would be inserted in place of the expression. I explained this in my previous email, and Guido has already said '-1' to this approach, so I won't elaborate any further. 4. Function scoped variables This is the approach most analogous to C's static variables - named variables that are shared across all invocations of a function, rather than being local to the current invocation. In essence, each function becomes its own closure - just as a function can share state across invocations by using an outer function for storage, this technique would allow a function to use its *own* cell array for such storage. Framing the idea that way also suggests a fairly obvious spelling: def f(x): nonlocal i=i # Use 'f' as a closure over *itself* return x + i With this spelling, the above would be roughly equivalent to: def outer(): i = i def f(x): return x + i return f f = outer() The only visible difference would be that the cell referenced by 'i' would be stored directly on 'f' rather than on an outer function. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Sep 23, 2011 at 11:11 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Applying this 'functions as their own closure' concept to the classic counter problem: def counter(): x = 0 def increment(): nonlocal x x += 1 return x return increment would become: def counter(): def increment(): nonlocal x=0 x += 1 return x return increment Or, if you wanted a process-global counter: def global_counter(): nonlocal x=0 x += 1 return x Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
This is the last time you mention the decorator solution (aside from further explanation of the problem). Is it being discarded for that reason? I was under the impression that everyone who liked that idea was fully aware that it's deep magic. I would assume it actually creates an entirely new function object with new closure cell to bind the name, similar to your last solution involving the function closing over "itself", but more general and as part of a decorator above the function header. I'm not really seeing a problem other than maybe distaste because it seems "hacky". It can work, though. Devin On Thu, Sep 22, 2011 at 9:11 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Fri, Sep 23, 2011 at 11:39 AM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Yes, it's far too hard to explain what it does in terms of existing Python semantics (the Zen has something to say on that point). Bytecode hackery and other tricks would *work*, but deep magic should only be employed when there aren't any alternatives and the problem is sufficiently common. (PEP 3135, the new super semantics for 3.x, pushes the boundaries of what's reasonable, but it's so easy to *use* that it's worth the additional under the hood complexity). Magical behaviour is also far more likely to cause problems for other implementations - it tends to rely on assumptions that aren't explicitly guaranteed by the language specification. Use of default arguments for pre-initialised function locals is nowhere near common enough to justify deep magic as a solution, so the idea just isn't worth pursuing. That's the real benefit of the "nonlocal i=i" idea: it takes an *existing* concept (i.e. closures), and just tweaks it a bit by avoiding the need for a separate outer scope when all you really want to do is share some state between invocations. Anyone that already understands closures shouldn't have much trouble grasping the idea that the 'innermost containing scope' can now be the function itself rather than the next level up. Any implementation that already correctly handles closures should also be able to cope with the self reference without too much trouble. No new keywords, no new namespace semantics, just a slight tweak to the way the compiler handles def and nonlocal statements. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Sep 22, 2011 at 6:11 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yes, you're right that this can't be done with today's decorators. That doesn't mean it's impossible. Any of the other changes would also require compiler changes. Instead of @inject being a standard decorator, imagine a new kind of decorator that works as follows: a = 10 $inject(b=a) def foo() a = 20 return b+a where the effect of a $inject decorator is that it modifies the behavior of the function as follows: a = 10 _inject_ = inject_decorator(b=a) def foo(): locals().update(_inject_) a = 20 return b + a Yes, I know that update on locals() won't do the right thing but pretend it does. I think the semantics of this are pretty clear and more obvious than the suggested alternatives. Now what about these $-decorators? As defined here, it's pretty specialized to this one operation. Let's imagine that we define $-decorators to be a special class of decorators that are built into Python. We use $-decorators for cases we would like users to think of as decorators but that do some kind of magic that's beyond the capability of a regular decorator. To the user the $ is a signal to the user that this decorator is a bit more special than usual. Right now, there would only be one $-decorator but it's available next time it's desired to add something that just can't quite be a regular decorator. A good objection to adding new things to the language is that they increase the load on people trying to learn the language. If I see a syntax like: def f() [i=i]: pass I can't imagine how I would find out what that does. Search engines don't readily allow searching on special characters. On the other hand, when I see $inject, I go type "python $inject decorator" into my favorite search engine and whether it ignores the $ or not, I'm fairly likely to get good results. Whether the decorator-like syntax is spelled $inject, @@static or something else, I think it's going to be much easier to figure out and remember what's going on. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
Yes, you're right that this can't be done with today's decorators.
It can work with today's decorators and today's syntax, it just requires magic. (Magic doesn't mean "impossible things", it means "things that go below the expected level of abstraction". In this case, things like directly inspecting fields of the function object and creating a new one with modified fields -- that, or else mutating a supposedly-immutable object (also possible)) That doesn't blow down the whole idea of $ special syntactic decorators, though. And maybe even this case is good for them. Definitely there might be room for a more general-case thing for compile-time annotations than new extra-special syntax. Magical decorators that get understood by the compiler have been mentioned before, and are inappropriate for obvious reasons. But if they were prefixed by $ instead of @... I still like real decorators because they can be applied at run-time instead of just compile-time, which gives them additional use-cases. Devin On Thu, Sep 22, 2011 at 10:18 PM, Bruce Leban <bruce@leapyear.org> wrote:
On Thu, Sep 22, 2011 at 7:27 PM, Devin Jeanpierre <jeanpierreda@gmail.com>wrote:
I still like real decorators because they can be applied at run-time instead of just compile-time, which gives them additional use-cases.
The $inject special decorator would need to be recognized at compile time but would be applied at run time just as @decorators are. That is, I see no reason this couldn't work: def foo(a): $inject(b=a) def bar(c): return b+c return bar --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
Devin Jeanpierre wrote:
In this case, I believe that most of the work that needs to be done -- making a copy of the function and code object -- are not magic. They are fully supported in standard Python. The only "magic" is manipulating the bytecode of the code object to that it turns some globals into locals.
That doesn't blow down the whole idea of $ special syntactic decorators, though. And maybe even this case is good for them.
I don't see any reason to introduce extra syntax for a different sort of decorator. What benefit is there? I think this $ proposal actually undermines the argument I am trying to make. A big advantage of using a decorator is that it requires no new syntax and no magic symbols beyond the standard @ symbol. The argument "yes, I like decorators, but I want to use $ instead of @" doesn't really help.
I still like real decorators because they can be applied at run-time instead of just compile-time, which gives them additional use-cases.
I have negative interest in a "magic decorator" that can only work at compile time. If we're going to have that (unnecessary, in my opinion) limitation, then I prefer the static declaration proposal. -- Steven
On Fri, Sep 23, 2011 at 1:26 PM, Steven D'Aprano <steve@pearwood.info> wrote:
That may be the core of the confusion here. Bytecode hacking is NOT supported in standard Python. The language definition pretty much stops at "function objects have a __code__ attribute that refers to the immutable code object defining the algorithm they execute when called". Anything below that point is implementation dependent (hence the disclaimer on the output of the 'dis' module). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
I have read the previous thread, and I'm fully aware that this would be special. I'm pretty sure I even said it would be special in one of my posts :) Excluding the default "do nothing" position, and the minimalist "just add conventions to introspection tools" proposal, I believe that @inject is the least magical proposal made so far. The other proposals require a new keyword, new syntax, or both. They require the compiler to do extra work. @inject requires nothing from the compiler. It all happens when the decorator is called. Comparing it to the voodoo needed for super() is completely unfair. It also offers the most benefits: * all the benefits of the proposed "static" declaration (early-binding, micro-optimisation, monkey-patching) * the ability to patch pre-existing functions * no call-time cost (it's not a wrapper, its a new function) * fewer side-effects than conventional monkey-patching * familiar syntax * obvious semantics * no new keywords * no extra "line-noise" symbols What's not to like? Yes, inject() will be special. But I don't believe it will be magic, or at least not deep magic. We can get most of the way using supported Python functionality already: we can pull a function object apart, make a copy of the pieces as needed, and reassemble them into a new function. All that is supported by Python, none of it requires messing with private attributes: it's not much more magic than what functools.wraps() already does. We can even take the bytecode from the code object, and because it's just a string, we can perform transformations on it. The only "magic" is the knowledge of what transformations to perform. Take this simple example:
3 6 LOAD_FAST 0 (c) 9 LOAD_GLOBAL 0 (d) 12 BINARY_ADD 13 RETURN_VALUE @inject(d=2)(func) would probably look produce something like this: 2 6 LOAD_CONST 2 (2) 9 STORE_FAST 1 (d) 3 0 LOAD_CONST 1 (1) 3 STORE_FAST 0 (c) 4 12 LOAD_FAST 0 (c) 15 LOAD_FAST 1 (d) 18 BINARY_ADD 19 RETURN_VALUE If we had a Python assembler to match the disassembler, it would probably be easy. If we are really allergic to the idea of bytecode manipulation, perhaps there are other ways to solve this. Just tossing ideas around, maybe function objects should keep a copy of their AST around, so that instead of bytecode hacking, you manipulate the AST and recompile the function. That might be less "hacky" than manipulating the bytecode, but I don't know that carrying around the AST for every function is a cost that is worth bearing. But it's an idea. -- Steven
On Fri, Sep 23, 2011 at 1:17 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Bytecode hacks are inherently implementation dependent deep magic. Almost *nothing* about code objects is part of the language specification (IIRC, it expects them to exist, but that's about it). Jython and IronPython use JVM and CLR bytecode. PyPy uses a format at least similar to CPython bytecode (since it was convenient for them to do so), but there's absolutely no requirement for them to keep it the same as the two implementations evolve. The entire *point* of wpython is to use a word-oriented rather than byte-oriented format. Even CPython will happily change the bytecode format between releases - there's a reason a bytecode version marker is embedded in every .pyc file. Taking an existing function apart and putting it back together again in a different way is skating right along that border between language specification and implementation details. Proposals that play in that space are almost guaranteed to be a hack that pays no attention to the overall semantic concepts of the language design and they're simply *not* going to happen. Any reasonable piece of Python code *must* treat code objects as opaque blobs that the interpreter uses to define an algorithm. It can be fun, and sometimes even useful, to go further, but it would be thoroughly inappropriate for the interpreter core or standard library to indulge in that kind of thing. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sep 23, 2011, at 11:11 AM, Nick Coghlan wrote:
Interestingly, PEP 3104 proposes that syntax, as shorthand for nonlocal x; x = 3 though this was not adopted. I think your proposal is a very interesting, modest but useful extension of that original PEP. -Barry
On Fri, Sep 23, 2011 at 11:11 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Alex Gaynor pointed out the above would give an unbound local error. The actual rough equivalent would be more like: def outer(i=i): def f(x): return x + i return f f = outer() Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Instead of a $ decorator, you could let decorators have a __prepare__ method that returns a dict to use for locals, as metaclasses do today. I did a quick test in Python 3.2:
This shows that decorators are executed before the function is built. Since we're looking up the decorator before the function object has been created anyway, it wouldn't be impossible for the locals dictionary of the function to be "modified" before it's created, as we can do with metaclasses. If we did this, inject would look something like this: class inject: def __init__(self, **kwargs): self.kwargs = kwargs def __prepare__(self): return self.kwargs def __call__(self, f): return f And of course, there could be other applications for the __prepare__, such as using OrderedDict or whatever. (Not sure if we'd want to encourage that though…)
On Fri, Sep 23, 2011 at 2:34 PM, Carl Matthew Johnson <cmjohnson.mailinglist@gmail.com> wrote:
Instead of a $ decorator, you could let decorators have a __prepare__ method that returns a dict to use for locals, as metaclasses do today.
No, locals simply don't work like that. They're handled by frame objects and the eval loop, not function objects. There are a whole host of complicated interactions between the compiler's symbol table analysis and code generation, function, code and frame objects and the runtime evaluation loop going on when it comes to handling name lookups in Python. When you're looking at it purely from a Python user's point of view, there are plenty of ideas that seem potentially reasonable on the surface but simply don't fit with the underlying data model of the language. It's hard enough coming up with good proposals for semantic and syntactic tweaks when you *do* know how they all work together to achieve the current behaviour - the current setup is actually pretty good, and the number of ways we could make it worse vastly outnumbers the ways we could conceivably improve it. Try to keep this discussion in perspective: we're talking about handling a niche use case in a slightly more elegant fashion. The magnitude of language change that can be justified for this purpose is tiny. I've thrown plenty of ideas of my own at the task over the years (and seen even more ideas from others), and tweaking the syntax and semantics of nonlocal is the first of them that I've genuinely liked better than the status quo. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Sep 22, 2011 at 10:52 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yeah, that's what I found out when I tried to add in an implicit "__function__" in the locals. I mostly had it working, but it was a hack and a half (a lot worse than super's "@__class__" injection).
+1 -eric
On Sep 22, 2011, at 6:52 PM, Nick Coghlan wrote:
No, locals simply don't work like that. They're handled by frame objects and the eval loop, not function objects.
I feel like I have the same complaint that other people did earlier in the thread: I understand that locals don't work like that *now*. (And they understood that decorators can't do an inject *now*.) When I said "locals" I was using a shorthand, because I don't know all the correct terminology. What I'm proposing is a change to how the system works. I want the byte code for functions to be created differently after this change. Obviously, that can't be done by the function objects, but it will need to be done beforehand. There's probably much more to this sort of change than I realize--maybe even so much that it wouldn't be worth it to do it, since the benefit would be relatively small. If so, that's fine. But objecting that this isn't how it really works feels like it's missing the point to me. The more relevant objection is that it would be too huge of a project to make it work a different way. With respect, -- Carl
On Fri, Sep 23, 2011 at 3:44 PM, Carl Matthew Johnson <cmjohnson.mailinglist@gmail.com> wrote:
I feel like I have the same complaint that other people did earlier in the thread: I understand that locals don't work like that *now*. (And they understood that decorators can't do an inject *now*.) When I said "locals" I was using a shorthand, because I don't know all the correct terminology. What I'm proposing is a change to how the system works. I want the byte code for functions to be created differently after this change. Obviously, that can't be done by the function objects, but it will need to be done beforehand. There's probably much more to this sort of change than I realize--maybe even so much that it wouldn't be worth it to do it, since the benefit would be relatively small. If so, that's fine. But objecting that this isn't how it really works feels like it's missing the point to me. The more relevant objection is that it would be too huge of a project to make it work a different way.
That's exactly my point though - the default argument use cases to be addressed are *trivial*. A trivial problem requires a comparably trivial answer - there is *nothing* in the use cases being discussed that even comes close to justifying substantial changes to the way the language works. Changing the way locals works, allowing variable injection, those are huge, language changing ideas, that require powerful, compelling use cases to motivate them. As far as I am aware, those use cases don't exist (and if they do, they certainly haven't been raised in this thread). It would be really helpful if people could comment on ideas that have at least some chance of making it under that "simple enough to be worth doing" bar. Obviously, I consider adjusting 'nonlocal' so we can use it to store cell references on the function being defined to be one such idea, as it is essentially just blending PEP 3104 with the underlying implementation of PEP 3135's magic '__class__' reference in methods. As far as I can tell, the biggest potential practical problem with it is on the compiler side, where the AST nodes for the initialisation expressions would need to be lifted out for inclusion in the code generated in the surrounding scope. That's not impossible (we already do something similar in deciding whether a def statement is defining an ordinary function or a generator), but the state to be collected is a fair bit more complicated than anything the symbol analysis pass currently handles. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Is there a restriction that nonlocal can be used only inside nested functions and would that limitation have any impact. Also, would it make sense/be possible to execute nested class/function definitions at function definition time. Regards, -Krishnan
On 9/23/2011 7:26 AM, H. Krishnan wrote:
Is there a restriction that nonlocal can be used only inside nested functions.
Yes, althought the current 3.2.2 manual does not make that clear enough. I believe there is a tracker issue to clarify this.
Also, would it make sense/be possible to execute nested class/function definitions at function definition time.
No. The body of a function is executed when the function is called. However, it is already the case that code objects for nested functions *are* compiled just once. So just a nested function object has to be assembled with each call. Default args and cell vars are part of each new function object. -- Terry Jan Reedy
Ron Adam wrote:
Sorry, but that spelling is very far from obvious to me. According to the current meaning of 'nonlocal', it looks like a no-op. I don't understand the reasoning that leads from there to your proposed semantics. You'll also have to explain how that reasoning applies to the following variations: nonlocal i = i + i nonlocal i = i + j nonlocal i = j + k -- Greg
On Sat, 2011-09-24 at 11:12 +1200, Greg Ewing wrote:
Here's my view of what would happen, but you don't say weather or not those are defined together or if they are separate cases. nonlocal i Gives read write access to i in a parent frame. nonlocal i=i Creates a new 'i' in a functions own frame with the value of 'i' from a parent frame at the time the function is defined. (Note, it also makes sense to do it on the first time the function is called. I'm not sure there is any advantage to that.) Once the function is called, the access of 'i' would be in the local frame, and not effect the parent frames 'i' because the local 'i' is found first. As for the various cases, they would work the same except the initial value would be different. My 2cents for what it's worth. Cheers, Ron
On Fri, Sep 23, 2011 at 5:35 PM, ron adam <ron3200@gmail.com> wrote:
The only sensible meaning that could possibly assigned to "nonlocal <var> = <expr>" is that it should be strictly equivalent to nonlocal <var> <var> = <expr> Anything else is too confusing for our users. So your proposal is right out. -- --Guido van Rossum (python.org/~guido)
On 9/22/2011 9:11 PM, Nick Coghlan wrote:
def f(x, _i=i): # pydoc would, by default, display the signature as 'f(x)' return x + _i
I presume you meant def f(x, *, _i=i): return x+_i so that _i will be keyword only.
It certainly makes it impossible to 'accidentally' override with an extra positional parameter. -- Terry Jan Reedy
On Sep 22, 2011, at 3:58 AM, Nick Coghlan wrote:
This reminds me that there was a proposal a few weeks back to be able to refer to the function one is in absolutely. Right now, you can't just slap an attribute onto f in a loop and then get it back at exec time, since the variable f will refer to the last f defined in the loop. But suppose we had some sort of "THIS_FUNCTION" magic variable… Then this would work: fs = [] for i in range(10): def f(): return THIS_FUNC.value f.value = i [f() for f in fs] (Again, it has to be some sort of magical variable, since "return f.value" will refer to the 9 producing function at the end.) This could then be combined with a standard inject decorator. @inject(i=i) def f(): return THIS_FUNC.inject_values.i In cases where you're not in a loop or something, you could just write "f.values.i" instead of using the new magical variable.
On Thu, Sep 22, 2011 at 12:49 PM, Carl Matthew Johnson <cmjohnson.mailinglist@gmail.com> wrote:
This reminds me that there was a proposal a few weeks back to be able to refer to the function one is in absolutely. Right now, you can't just slap an attribute onto f in a loop and then get it back at exec time, since the variable f will refer to the last f defined in the loop. But suppose we had some sort of "THIS_FUNCTION" magic variable…
Yeah, a "THIS_FUNCTION" in the function locals didn't go very far. It would be too easy to abuse it and the implementation doesn't fit well into CPython (as far as I could see). I have a patch in for adding f_func to the frame object (you have to jump through the inspect module to get to it). However, that doesn't help so much for this.
Performance is another concern here. The default argument hack is pretty efficient compared to attribute lookups on the function object. -eric
On Sat, Sep 17, 2011 at 7:30 AM, Ron Adam <ron3200@gmail.com> wrote:
I don't see any benefit in offering a second way to spell docstrings or decorators (the existing spellings are easy enough to remember), but this suggestion did prompt an idea for potentially replacing uses of the default argument hack: allow a sequence of 'implicit locals' to be defined within square brackets after the parameter list. (This idea may have come up before, but if it has, I don't recall the arguments against it). Running through the 3 at least arguably legitimate use cases for the default argument hack: Micro-optimisation: # Current Python (internal to functools.lru_cache with default argument hack) def decorating_function(user_function, tuple=tuple, sorted=sorted, len=len, KeyError=KeyError): ... # 60 line function # Proposal - explicitly implicit locals to clarify real calling signature def decorating_function(user_function) [tuple=tuple, sorted=sorted, len=len, KeyError=KeyError)]: ... # 60 line function (Note: in practice, the lru_cache usage of the default argument hack doesn't really impact introspection, since the function is short-lived - it is the decorating function returned by the lru_cache decorator factory and hence only exists briefly as part of the function definition. The signatures of lru_cache itself and of the decorated function are not affected. I still like it as an example, since it isn't hypothetical - the standard library really uses the default argument feature this way). Early binding in a loop: # Current Python adders = [] for i in range(10): def f(x, _i=i): return x + _i adders.append(f) # Proposal adders = [] for i in range(10): def f(x) [i=i]: # Real calling signature is clear return x + i adders.append(f) Algorithmic shared state (without a class): # Current Python def f(*, _cache=[]): # Consenting adults, please don't override _cache when calling! # Proposal def f() [_cache=[]]: # Real calling signature is clear # _cache is an implicit local, not a keyword-only argument, so its safe To prevent grammar ambiguity, the list of implicit locals would most likely need to appear before the annotation for the function return type. The syntax within the list would be the same as those for arguments with default values (while I don't have a use case for allowing annotations, I don't have any reason to prevent them either, so reusing the same grammar fragment seems like reasonable idea) The implementation would also be basically the same as for arguments with default values (a list of names stored as an attribute on the code object and a dictionary on the function object to populate them) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, 2011-09-17 at 11:01 +1000, Nick Coghlan wrote:
I almost included an example similar to this. Using your adder example, it would have looked something like this. adders = [] for i in range(10): def f(x) # Real calling signature is clear f.initial_values(i=i) # done at def time : return x + i adders.append(f) Of course there is no way to preinitialize a functions name space. Your [i=i] notation would do something like that behind the scenes. I was thinking it would be good if the parts in front of the colon were valid python statements except in the case of the already special docstrings and decorators. It could be limited to only valid python commands. If it turns out that a particular command becomes very common, a special syntax could be considered. Having the colon on a separate line would be optional, so it wouldn't change existing code at all. I tried to find a way to use a decorator to do this and didn't find anything that worked nicely. It can be done nicely with class's, but the point of using a function is it would be more efficient. I think this also starts to get into the area of meta programming. In python library there are a number of functions of the form. def foo(*args, **kwds): return _foo(*args, **kwds) I presume there is a reason why they didn't just do foo = _foo. currently you can swap out the __code__ part of a function, but not the signature part. If you could, then this might be possible. def foo() # Using _foo's signature. foo.signature = _foo.signature foo.__code__ = _foo.__code__ : pass # using _foo.__code__ Cheers, Ron
On Sat, Sep 17, 2011 at 3:47 PM, Ron Adam <ron3200@gmail.com> wrote:
You did say that, but advanced no arguments in favour of breaking with the language aesthetic that way. Indented suites are introduced by a colon on the preceding header line and you'd need a really compelling argument to change that. "It might be nice" doesn't cut it. Specifically targeting existing uses of the default argument hack, on the other hand, comes with at least 3 already known use cases (i.e. name lookup micro-optimisation, early binding semantics and shared state for algorithms). People already tolerate messing with the function signature to obtain those behaviours, so it's reasonable to wonder if it is possible to provide a cleaner way to obtain the same effect. A simple set of definition time name bindings would likely suffice without greatly increasing the language complexity or the runtime overhead of function calls. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Sep 17, 2011 at 2:16 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Since the goal is to add some context, like a wrapper, it seems reasonable to use a specialization of @. @+ suggests adding something to the environment, and won't be ambiguous unless numbers become callable. @+tuple=tuple @+sorted=sorted @+len=len @+KeyError=KeyError def decorating_function(user_function): I would personally be fine with a restriction to @+name=<expr> but I suppose others might prefer a tuple, like @+(tuple, sorted, len, KeyError)=(tuple, sorted, len, KeyError) def decorating_function(user_function): -jJ
On Sun, 2011-09-18 at 15:32 -0400, Jim Jewett wrote:
If there was a method on a function object to set default values, then you could use a regular decorator. # Most likely to live in functools def preset(**kwds): def apply(f): f.set_default_values(kwds) return f return apply #--------------------------------- from functools import preset @preset(tuple=tuple, sorted=sorted, len=len, keyerror=keyerror) def foo(...): ... I was hoping there might be a more direct way to do it. My previous idea breaks the white space consistency, and Nicks idea adds special syntax. Decorators with long arguments make the function name stand out less. Is there a more recent patch for Pep 362? Pep 362 seems to be targeted at only introspection and really isn't a part of the function object. Is that correct? Cheers, Ron
I think something similar could be achieved with annotations, as in the following example. This is incomplete, and probably full of errors, but it gives the idea. def setlocals(func): def wrapper(*args, **kwargs): call_kwargs = [] for k, v in kwargs.items(): if k in func.__annotations__ and func.__annotations__[k] == 'local': raise AttributeError("Can't set argument") else: call_kwargs[k] = v return func(*args, **call_kwargs)
On Sep 18, 2011 10:59 PM, "Ron Adam" <ron3200@gmail.com> wrote:
On Mon, Sep 19, 2011 at 5:44 PM, David Townshend <aquavitae69@gmail.com> wrote:
Proposals that are uglier than the default argument hack itself (and I put both annotations+decorator and the '@+' idea in that category) just aren't worth the hassle. The *only* change even potentially worth making in this area would be one that: - looks good (i.e. is easy to read) - is easy to remember and write - doesn't hurt call time performance - addresses all the default argument hack use cases - doesn't have the downside of altering the function's externally visible signature My initialised locals idea *might* qualify (although I've yet to see anyone make a serious attempt at poking holes in it, so I'm not convinced of that), but the other suggestions in this thread haven't really offered anything to recommend them over the existing approach. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I agree, both are ugly (btw, mine wasn't a proposal, just an illustration that it is possible with the status quo). One question I have is would the locals be local to a specific call or the the function object? I don't know enough about the python language internals to know the correct terminology to explain what I mean, but from a users point of view, would the locals be reset at every call? The fact that they are locals implies that they don't exist outside a function call, but assigning them in the function definition implies that they exist within the function object in a similar way to the value of keyword arguments. David On Mon, Sep 19, 2011 at 11:35 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Nick Coghlan wrote:
[...] allow a sequence of 'implicit locals' to be defined within square brackets after the parameter list.
I'm not quite sure whether the benefits of this proposal are worth the further rococoization of the language. As far as I can see, the only benefit is keeping function signatures clean. Looking at the use cases one by one:
The proposed syntax isn't explicit enough for my taste to justify the change. What the programmer really wants to tell the compiler is: Bind the given names at compile time rather than at run time. I'd prefer if the syntax would reflect this intention, similar to the "global" and "nonlocal" declarations: def decorating_function(user_function): earlybind tuple, sorted, len, KeyError (Not being a native speaker of English, I don't even try to find a less dull name for this than "earlybind". And I'm quite aware that the bar for the introduction of a new keyword is quite high.)
I don't see too much benefit of the proposed syntax for this use case. If f() is a local throw-away function, I wouldn't worry about its signature. If f() is a longer-lived object and I do care about its signature, I'd uses a class: class Adder: def __init__(self, i): self.i = i def __call__(self, x): return x + self.i [...] adders.append(Adder(i)) I still think classes are the Python way to hide state, not closures. That said, this case would also be covered by the "earlybind" porposal above.
Again, I'd use a class to hide state in the first place. If someone really wants to avoid using a class for some reason, using function attributes would also be a viable way: def f(): try: cache = f._cache except AttributeError: cache = f._cache = [] Cheers, Sven
On Mon, Sep 19, 2011 at 11:22 PM, Sven Marnach <sven@marnach.net> wrote:
The thing is, real function objects genuinely *are* special. They have a privileged place in the interpreter, the inspect module and other introspection tools know more about how to deal with them, they have instance method descriptor behaviour built in, etc. Switching from a real function object to a custom class with a __call__ method is genuinely expensive in terms of the difficulty of writing the code in the first place, as well as in being able to read it later. Your 'Adder' example above, for instance, doesn't implement the descriptor protocol, so will behave like a staticmethod when placed in a class. That may be what you want, but it isn't easy to get instance method behaviour in the case where you would prefer that. $ python3 -m timeit -s "from call import Adder; f = Adder(1)" "f(5)" 1000000 loops, best of 3: 0.362 usec per loop $ python3 -m timeit -s "from call import Adder; f = Adder(1).__call__" "f(5)" 1000000 loops, best of 3: 0.222 usec per loop $ python3 -m timeit -s "from closure import adder; f = adder(1)" "f(5)" 10000000 loops, best of 3: 0.174 usec per loop $ python3 -m timeit -s "from default import adder; f = adder(1)" "f(5)" 10000000 loops, best of 3: 0.166 usec per loop When what you're trying to express is a single short algorithm, overriding __call__ isn't even in the contest - we aren't talking minor percentage differences in call overhead, we're talking more than double. You can gain a fair bit of that back by retrieving __call__ as a bound method in advance, but really, your algorithm needs to be complex enough for the difference in call overhead to be trivial before implementing __call__ becomes an attractive alternative to using a closure Now, if there are *multiple* algorithms operating on the same data, then obviously you want a class with multiple methods. But in that case, you're less likely to bless any one of them with privilege of occupying the '__call__' slot. Basically, classes make sense when the state is the most important thing, while functions focus on a specific algorithm. For the special case of "single algorithm with some associated state", a closure (or the default argument hack) will often be a better modelling tool than a class. (Obviously, anyone that disagrees with me on this point will consider this whole thread silly - however, the popularity of closures for the implementation of factory functions, including decorator factories, shows that there is plenty of code out there that happily follows this approach) With closures vs the default argument hack, the performance and introspection arguments don't apply - in both of these cases you have a real function, so the only trade-off is between forcing readers to understand how closures work and forcing them to ignore additional arguments that are there just to prepopulate the local namespace with some data. However, the closure approach has some genuine downsides from a readability point of view. The real function when using a closure is the inner one. The outer function definition, the trailing return statement and the invocation of the outer function are all boilerplate that obscures the actual purpose of the code. You can tidy some of that up with a decorator, but you can't avoid the need for the nested function in order to create the closure. And that's why people use the default argument hack today - they weigh those downsides up against the consequences of having a bit of noise in the function signature (as seen by introspection tools) and decide they're happy to accept that trade-off in return for a simple and straightforward expression of a single algorithm with some associated state. Various ideas for addressing this have been proposed in the past. PEP 3150's statement local namespaces are one, as are the assorted incarnations of the proposal for flagging arbitrary expressions within a function for evaluation at definition time rather than runtime (search the python-dev and python-ideas archives for phrase like 'once statement' and 'atdef' - alas, nobody has stepped forward to codify those ideas into a PEP). Where those proposals all come unstuck is that they try to do more than the default argument hack allows, *without compelling use cases to guide the additional semantics*. The pre-initialised locals concept deliberately avoids that problem by targeting exactly the use cases that are *already* supported via the default argument hack, just in a way that tries to avoid the negative effects on introspection and readability. Cheers, Nick. P.S. Code for the timeit runs: $ cat > call.py class Adder(object): def __init__(self, i): self.i = i def __call__(self, x): return x + self.i $ cat > closure.py def adder(i): def f(x): return x + i return f $ cat > default.py def adder(i): def f(x, _i=i): return x + _i return f -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, 2011-09-20 at 08:23 +1000, Nick Coghlan wrote:
Heh, On an abstract level, I do know there is something special about function objects. Putting my finger on it is a bit harder. Trying a few things to see what happens... At some point there does need to be a base object that can't be taken apart into it's smaller parts. That normally would be the object class. But with functions, we are referring to the base *executable* object. A function is is a union of other basic objects that can then be executed. (I think there can be some improvements in how a function object is constructed and organized.) I guess the real *special* magic is the byte code, CALL_FUNCTION, which knows how to use a function object. Along with the other Function related bytecodes.
Umm... "...default arguments hack allows,..."? Not well said, I think you meant, "...unstuck, by doing more than is needed...". Even so, I do understand your point you are trying to get across. Would your suggestion add any additional methods, or attributes to a function object? Would it need any additional byte codes? An alternative proposal, I came across the other day... don't remember exactly where, is to be able to put default arguments after the **kwds, parameter.
It's currently a syntax error. But that requires using an ** someplace. I don't really like this one, as I think function arguments are already hard enough to understand. For a language that is suppose to be easy to understand, that's not a good thing. Cheers, Ron
Ron Adam wrote:
I'd say it's more in the function object's __call__ method, which invokes the C code that knows how to interpret bytecodes. The CALL_FUNCTION bytecode just calls the __call__method of the object being called, whatever it happens to be. The function object's __get__ method also plays a part when a method call is involved, as does the __getattribute__ method of the type object. The magic tends to be spread around a fair bit in Python. -- Greg
Nick Coghlan schrieb am Di, 20. Sep 2011, um 08:23:14 +1000:
I actually *do* use the default argument hack for early binding myself. My main points were that the current status quo isn't too bad, there are alternatives if the function signature is really important, and that the cases which require both using closures *and* having a clean signature are too rare to justify yet another confusing-for-beginners syntax. (When I started learning Python in 1998, the language was praised for having a small language core. I didn't hear this very often in more recent times.) Assuming a new syntax *is* necessary -- there is no objective way to decide this after all -- I don't particularly like the proposed square bracket syntax because a) it puts the information at the wrong place. The first line of the function definition is for the outside appearance of the function, including signature and annotations. Separating the internal information on early bound variables from the signature is the primary goal of the proposal after all. b) it does not make the intention of early binding very clear. c) it consists of a list of the form [i=i] or [tuple=tuple, len=len] in most use cases, making it necessary to type every name that shall use early binding twice. That's why I would prefer a syntax that explicitly declares the names that should use early binding, similar to "global" or "nonlocal" statements, if such a syntax is deemed necessary at all.
How does PEP 3150 target the use cases of the default argument hack given in your original mail? Cheers, Sven
On 9/20/2011 10:00 AM, Sven Marnach wrote:
I have noticed the same. The big syntax additions are generators, comprehensions, decorators, and the with statement. The switch to unicode doesn't change syntax but can complicate text processing.
The problem with an earlybind statement in the body is that it is in the body, which should be runtime stuff. I would prefer something in the header. A semi-colon or other char would be sufficient syntactically: def f(a; len, alist=[]): alist.append(a); return len(alist) where a bare identifier like 'len' *means* 'len=len'. -- Terry Jan Reedy
On Tue, Sep 20, 2011 at 9:59 AM, Terry Reedy <tjreedy@udel.edu> wrote:
I would be much more interested in a tweak to the language semantics where the compiler is allowed to assume that "len means len" if there is no global assignment to it in a module. There are many past threads about this topic. It should make the feature proposed here unnecessary -- at least its use for manual micro-optimizations, which I mostly find an offense to the reader (even in the shortened form proposed here). -- --Guido van Rossum (python.org/~guido)
On 9/20/2011 1:09 PM, Guido van Rossum wrote:
On Tue, Sep 20, 2011 at 9:59 AM, Terry Reedy<tjreedy@udel.edu> wrote:
For those not party to the earlier threads, this alternate proposal would bread any code that monkeypatches an imported-module-only override of a builtin, something like: def mylen(o): return <whatever> import mod; mod.len = mylen I do not mind the semantic change itself, and it would benefit nearly all code, including module-level code would not be affected by def gimmicks. Some issues I see that perhaps were discussed, but I do not remember: 1) module.__setattr__ should then raise an exception at such attempts; 2) it makes the language more context sensitive than it currently is; 3) compile() should apply the rule even for single statements or even expressions. -- Terry Jan Reedy
On Wed, Sep 21, 2011 at 1:36 PM, Terry Reedy <tjreedy@udel.edu> wrote:
The two sticking points tend to be open() and print(). Overriding those externally can be *incredibly* useful for testing code that uses them, as well as in interacting with code that wasn't designed in a flexible way. Regardless, I'm going to stop mentioning the manual micro-optimisation use case for the default argument hack. Alex Gaynor pointed out that that aspect is completely irrelevant on PyPy, and as Guido notes, there are likely other ways to tackle name lookup optimisation even in CPython. That still leaves two arguably valid use cases: - early binding for name lookups - an algorithm with shared state (kind of like an inverted approach to creating a class with a custom __call__ method) Perhaps a non-syntax way to approach both of these would be to add a new decorator to 'functools': def closure(f): """Invokes the decorated function and returns the result after transcribing essential function metadata This can be used to easily share algorithm state and get early binding semantics for names. """ impl = f() impl.__name__ = f.__name__ doc = f.__doc__ if doc is not None: impl.__doc__ = doc impl.__dict__.update(f.__dict__) return impl This would be used as follows: @functools.closure def adder(i=i): # 'impl' defines call time signature "Increments 'x' by adder.value" def impl(x): impl.call_count += 1 return x + i impl.value = i impl.call_count = 0 return impl
Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 2011-09-21 at 15:19 +1000, Nick Coghlan wrote:
Simplifying things like this is one of the use cases of allowing define time statements. That's a lot of work to just avoid putting a keyword in the signature. And it's not easy to understand. Decorators could be a good way to do this, but the problem in these cases, is the function object doesn't have the needed support to make things like this easy. Probably the easiest and most direct way, would to be to add a new keyword 'static' as MRAB suggested, but have it be an expression instead of a command. value = (static <expression>) def adder(x): return x + (static i) # evaluate (static i) at compile time. The parentheses would be optional. The (static i) expression, could be spelled (i=i). I think that was what Guido was suggesting, but not as an expression. As an expression, you would then see things like.. i = (i=i). But that may only be poor style, because it's easy enough to just not do that. I think the expression form is better myself, it allows you to get both the compile time value, and the current value of an identifier. def value elapsed_frames(): """ Where f is the frame counter in the parent scope. """ return f - (static f) Cheers, Ron
When you say "compile time", do you literally mean "when the compiler creates the bytecode" or do you really intend this to be computed at function definition time (the same time when default values are evaluated)? The latter seems to make more sense. --Guido On Wed, Sep 21, 2011 at 11:12 AM, Ron Adam <ron3200@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On 21/09/2011 19:34, Guido van Rossum wrote:
It would be at function definition time.
def fib(n, known={}): if n in known: return known[n] if n < 2: f = n else: f = fib(n - 1) + fib(n - 2) known[n] = f return f to: def fib(n): known = static {} if n in known: return known[n] if n < 2: f = n else: f = fib(n - 1) + fib(n - 2) known[n] = f return f This: known = {} creates a dict and binds a name to it at execution time, whereas this: known = static {} creates a dict at definition time, storing a reference to it in the function object, and binds a name to it at execution time. I suppose it's like a class variable: class Foo: known = {} which is shared by its instances.
On Wed, Sep 21, 2011 at 3:59 PM, Guido van Rossum <guido@python.org> wrote:
Agreed, though just for assignment statements. Also, it may be confusing to allow more than one static assignment to the same name in the function body. Perhaps that should be a SyntaxError. -eric
On Thu, Sep 22, 2011 at 8:38 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
And so we come full circle... it looks like Jan didn't have the time to write up a PEP following the last discussion of this topic [2], so perhaps you'd be interested in taking up the task? FWIW, I still quite like the explicit 'atdef' keyword I suggested last time [1] (and most of the arguments from that thread against the 'after-the-**' syntax are equally valid arguments against the '[] in the function header' syntax from this iteration). [1] http://mail.python.org/pipermail/python-ideas/2011-June/010565.html [2] http://mail.python.org/pipermail/python-ideas/2011-June/010552.html Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I hope I'm not bikeshedding here, but I much prefer decorators because they let you do more than just micro-optimization. They were mentioned before, in another thread, and I find them attractive. A decorator basically lets you inject locals into already-existing functions. So if you want to, e.g., test a function that uses a global "urlopen" function to download a file, by replacing urlopen. from urllib import urlopen from StringIO import StringIO def myfunction(): return urlopen('...').read() class TestMyFunction(unittest.TestCase): def setUp(self): self.func = inject(urlopen=lambda s: StringIO(s) def test_opens(self): self.assertEqual(self.func(), '...') There's also the benefits of not altering the language etc. -- it's just a function that returns a new function that's very similar to the old one. That simplicity appeals to me as well. And it still satisfies the use-case of the new "static" keyword. Devin On Wed, Sep 21, 2011 at 2:12 PM, Ron Adam <ron3200@gmail.com> wrote:
Devin Jeanpierre wrote:
I find this idea extremely exciting. But have I missed something? Does this inject function actually exist somewhere? If I recall correctly, I've seen a bytecode hack or two that seemed to work, but I didn't think that was officially supported. Comparing syntax: def func(a, b): static f, g, h f = ... g = ... h = ... return f(g(h(a+b))) @inject( f = ... g = ... h = ...) def func(a, b): return f(g(h(a+b))) or even: def func(a, b): return f(g(h(a+b))) another_func = @inject( f = ... g = ... h = ...)(func) As far as syntax goes, I think both variants look good, but a decorator has two major advantages: * you aren't limited to using it at function definition time, you can apply it after the event * it doesn't need to be a keyword, or even a built-in: it could go into functools -- Steven
On Wed, Sep 21, 2011 at 1:19 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
With the right tooling, even given these semantics, we could monkeypatch by getting in at module runtime. One system that uses an idea suitable for this is exocet, which was introduced with this blog post - http://washort.twistedmatrix.com/2011/01/introducing-exocet.html . I'm don't think exocet today is appropriate for use, but its basic ideas could make for much more controlled tests, and become necessary in the face of these optimizations. Mike
On 20/09/2011 17:59, Terry Reedy wrote:
To me that's like a 'static' declaration in C++, which in Python might be something like: def f(a): static len=len static alist=[] alist.append(a) return len(alist) What follows 'static' would be an assignment which is executed when the function is defined, much like a default argument, and would be shared between invocations in the same way.
I'm adding both new propositions: 1. the `def foo(...) [len=len]:` syntax, 2. the `len = (static len)` expression syntax -- to the PEP-draft I'm preparing -- which Nick suggested in June [http://mail.python.org/pipermail/python-ideas/2011-June/010569.html] (I'm sorry that that preparing lasts so much time, but my everyday- -activity-CPU has been overloaded a bit for a few months...). Ad 1: I think it's better than the `after-**` proposition from June, though still has some its drawbacks (Sven just mentioned some of them). Ad 2: I must admit that this one becomes my favorite syntax for early binding (though I don't like the abbreviated form '(i=i)'). IMHO it's not only clear (no all that questions about assignment semantics) but also elegant, explicit and consistent with some existing syntax constructs (especially with `yield EXPR`). Note that (as you can use any expression) it makes possible to use e.g. tuples of several names: spam, print, open = static (len, print, open) Big +1 from me. 3. Another variant could be with a colon (a bit similar to the lambda syntax): len = static: len spam, print, open = static: (spam, print, open) or even (reusing the existing keyword): len = def: len spam, print, open = def: (spam, print, open) (here "def" means: "at definition time") But I'd rather prefer the #2. Regards. *j
On Wed, Sep 21, 2011 at 7:01 PM, Jan Kaliszewski <zuo@chopin.edu.pl> wrote:
Agreed, though the the keyword should be on the LHS: static spam, print, open = len, print, open It's more consistent with other keywords that way and easier to spot when reading code. Also, "static" as a keyword is in use as an identifier in a number of places per google's codesearch (on the order of 100) including in Django. It's not like it's in super broad use, but it is in use. -eric
On Thu, Sep 22, 2011 at 12:46 PM, Chris Rebert <pyideas@rebertia.com> wrote:
+1 (FWIW) on this "assignment statement with leading keyword" variant. `static` is an excellent choice of keyword for this.
Care to elaborate? This has always struck me as one of the more nonsensical keyword choices in C, so I'd like to see some solid justification before perpetuating it in Python (particular when C/C++ intuitions about statics being process global will hinder rather than help in understanding the scope of this new namespace). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 2011-09-21 at 20:31 -0600, Eric Snow wrote:
The above could also be ... spam, print, open = (static len, print, open) As Jan pointed out, it would be very similar to how the 'yield' is used. It doesn't do anything to the identifiers on the left side. The only thing the static expression does is save a reference to what ever the expression part on the right evaluates to at the time the function is defined. Later, when the function is called, is when it's actually used. At define time... spam, print, open = static (len, print, open) Becomes this at call time... spam, print, open = _tuple #tuple created by static expression Cheers, Ron (Apologies for any double or delayed posts, I'm still trying to get some email glitches worked out.)
Well, now that we're getting down to picking bike shed colors, time to chime in: blah = static blah is worse than static blah = blah because it's easier to visually pick out something on the LHS and it's analogous to global/nonlocal. Anything having to do with parens is visually ugly and should be avoided unless absolutely needed for explicitness. One question I have is how this will work with control flow. This example is works with Python 2.7 and 3.2:
Presumably static would work the same way? Still, I find this sort of distasteful, since the behavior is a bit surprising to those not familiar with the in's and out's of the compilation model. Why not put the static declaration in the argument list, something like: def f(static l=[]): And maybe as a shortcut for static NAME=NAME, use just static NAME. fs = [] for i in range(10): def f(static i): return i fs.append(f) Looks OK to me.
On Thu, Sep 22, 2011 at 2:21 PM, Carl Matthew Johnson <cmjohnson.mailinglist@gmail.com> wrote:
Presumably static would work the same way? Still, I find this sort of distasteful, since the behavior is a bit surprising to those not familiar with the in's and out's of the compilation model.
There's a reason style guides say not to do that with global and nonlocal, even though the language technically allows it. Definition time variable declarations would work the same way.
Why not put the static declaration in the argument list, something like:
def f(static l=[]):
For all the same reasons people don't like the 'after **' and '[] in the function header' suggestions. Most notably, the fact that the evaluation of certain expressions at definition time is an implementation detail rather than part of the public API, so it doesn't really belong in the header. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 2011-09-21 at 18:21 -1000, Carl Matthew Johnson wrote:
Which way, will depend on what kind of behavior is decided on. Statement? or Expression? As a statement, it would be first and not have the assignment. static blah (if it does, then working out how that works will be tricky.) As an expression, the only thing that matters is that the names used in the expression, are findable at define time. No assignment is needed. The value takes the place of the static expression when the function it is in is called. Here the expression "(static y)" is replaced with the object y was bound to at the time the function was defined. y = ['red', 'blue', 'green'] ... def f() ... for x in (static y): ... The parentheses are optional if the expression only has single value. But I think it makes it easier to see what is being done. An expression you can be used in lambdas. A statement can't. Static statement: def get_adders(count): adders = [] for n in range(count): def add_it(x): static n return n + x adders.append(add_it) return adders Static expression: def get_adders(count): adders = [] for n in range(count): adders.append(lambda x: return (static n) + x) return adders
I can see why that is surprising. I don't think the static would have this issue in either form. Cheers, Ron
On Thu, Sep 22, 2011 at 3:47 AM, Ron Adam <ron3200@gmail.com> wrote:
As an expression, the only thing that matters is that the names used in the expression, are findable at define time.
Not just that the names are findable, but that they are already bound to something. def f(a): b=a static c=b should fail, and should probably be a syntax error to prevent it apparently working (but perhaps with the wrong value) when (and only when) a more global b was set earlier.
No assignment is needed.
Agreed; in particular, static imports should reduce the cost of importing from inside a function body. (But then will people effectively start spelling "import" as "static import" even at the top of a module?) -jJ
FWIW, I'm at best +0 on doing anything here; I'm -1 on the expression-style (static <expr>) form but could live with the statement (whether or not combined with assignment -- IIRC that was planned for nonlocal and I don't see why we couldn't upgrade nonlocal and global at the same time). The main argument for static would be that this is what all other relevant languages call it. If we really want to do something else, I recommend reviving Algol-60's "OWN" keyword. -- --Guido van Rossum (python.org/~guido)
On Wed, Sep 21, 2011 at 9:44 PM, Ron Adam <ron3200@gmail.com> wrote:
Yeah, "static" on the LHS does imply a different meaning. Perhaps "static" is the wrong keyword, for all the baggage it carries. And a real "static" statement may not even be the right solution[1]. We're looking at two approaches here: do it in the function "header" or in the body. Here're some pros and cons: In-header Pros: - implies definition time In-header Cons: - limited space so more info adds clutter - could be mistaken as part of function's signature In-body Pros: - less cluttered - more closely associated with function locals? In-body Cons: - easy to miss that it's a definition-time directive (particularly if on RHS of assignment) - evaluated expression must not reference any of the function's locals We definitely want the solution to be explicit that it's a definition-time behavior. If the new syntax is in the body then it'll have to be especially obvious. Either way, the new syntax should clearly isolate the expression and associate it with a name to which it will be bound in the locals. All things considered, I don't think the that RHS static expression fits the bill. The right solution seems to be pretty elusive here... As an aside, what would take precedence in the case of a name collision: default arguments or the new syntax? -eric [1] If a static statement had an assignment clause for the initial value, then it would work. However, it doesn't seem explicit enough that the value is evaluated at definition time. [2] Ideally we would just use attributes on the function object. However, this has a number of problems including performance and the ambiguity of to which object the function name points. (I have a patch in issue 12857 that addresses exactly that.)
Eric Snow wrote:
We're looking at two approaches here: do it in the function "header" or in the body. Here're some pros and cons:
There's a third approach, although it would probably take a bit more implementation effort: a decorator. # Demonstrate early-binding, micro-optimization and monkey-patching # in a single call: @inject(a=some_value(), len=len, sum=my_sum) def func(x): ... The downside is that the decorator gets called after the function is already created, therefore it would have to take an existing function and make a modified copy of it. This may be harder than doing the "static" bindings while the function is being assembled in the first place.
In-header Pros: - implies definition time
Likewise for the usual @ decorator syntax.
Neither apply to decorator syntax.
Decorator is also less cluttered, but not so much obvious what it does.
Neither of these is a problem for a decorator: using the @ syntax, it is clearly and obviously definition-time. And because the inject decorator is called outside the function, it's obvious that the bindings can't access the function locals. -- Steven
Eric Snow dixit (2011-09-21, 20:31):
But then you lose important advantages of the expression variant -- especially that assignment is not a part of the syntax construct so it not only more powerfull but also more explicit and clear for newcomers: any assignment is at run-time and normal assignment rules apply to it. A bit like in case of `yield EXPR`. Cheers. *j
On 22 September 2011 09:11, Jan Kaliszewski <zuo@chopin.edu.pl> wrote:
A concern I have with the expression variant is that I don't understand what it would mean except in the restricted contexts it's been discussed in. Can you describe the semantics of static EXPR in isolation, so that we're not just interpreting it in terms of its use in an assignment, and can understand the wider implications? Thanks, Paul.
On Thu, Sep 22, 2011 at 6:21 PM, Paul Moore <p.f.moore@gmail.com> wrote:
It isn't really that different from the statement version - a definition time expression would be calculated at definition time by the interpreter and the resulting value cached on the function object. At execution time, the cached value would be used in place of the original expression. (As an implementation detail, this would likely work by assigning such expressions an index in the function's closure list and populating them at definition time as cells) To use the classic "adder in a loop" example: # default argument hack adders = [] for i in range(1): def adder(x, i=i): return x + i adders.append(adder) # default argument hack (lambda) adders = [(lambda x, i=i: x + i) for i in range(10)] # Definition time statement (lambda variant not supported) adders = [] for i in range(1): def adder(x): atdef i=i # Could conceivably shorten this to just 'atdef i' return x + i adders.append(adder) # Definition time expression (using same parenthesis rules as yield expressions) adders = [] for i in range(1): def adder(x): return x + (atdef i) adders.append(adder) # Definition time expression (lambda) adders = [(lambda x: x + (atdef i)) for i in range(10)] I think there's a case to be made for the expression version - it's likely to require less boilerplate than the statement version in simple cases and is compatible with comprehension syntax (since it can be embedded inside a lambda expression). The analogy with yield expressions is a reasonable one - but instead of coming from send(), the result of the expression is retrieved from the cache on the function object. The other advantage of the expression version is that it avoids the issue of definition a new namespace where names can be looked up. Instead, it's just a mechanism for caching the values of certain expressions at definition time - at execution time, those values can then either be used directly or else assigned to an ordinary local variable. I wasn't initially a fan of the idea, but it's growing on me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 22 September 2011 14:58, Nick Coghlan <ncoghlan@gmail.com> wrote:
OK, I think I get it now. So the example a, b, c = atdef ([], {}, set()) would create and cache a tuple at definition-time, then retrieve and unpack it at runtime. Which means that it has the same effect as a = atdef [] b = atdef {} c = atdef set() with some slightly subtle differences in detail (there's only one cached "value" rather than 3, but it'd be hard to detect that in practice. I think that based on your definition, all my "but what about X" pathological examples have reasonably sane behaviours (you need a good understanding of what "definition time" actually means, but once you internalise the fact that def is an executable statement, that isn't too hard). I can see the attractions of the idea but: * I'm sorry but the keyword atdef is ugly. The only proposal I like is static, but I agree that it only makes sense to someone with a C-type background. * I remain unconvinced that the problem is severe enough to warrant a new keyword. * The semantics, although clear and easy to state, are probably one more step away from Python being a simple language. I can imagine beginners' heads exploding...[1] Paul. [1] Although anyone needing the semantics is arguably no longer a beginner :-)
On 22 September 2011 15:34, Paul Moore <p.f.moore@gmail.com> wrote:
I spotted one that might still be worth mentioning, though. a = 10 def foo(): a = 20 b = static a What should that mean? Clearly b isn't 20, as a wasn't set to 20 at function define time. But b = 10 (on the assumption that "static a" takes the value of a at define time) will confusing to users, I would suggest. Not allowed might be best, but I suspect it could be hard to implement. But as Guido's -1 on the expression form, this is probably moot. Paul.
Paul Moore wrote:
Also hard to specify without excluding one of the intended use cases: def foo(): len = static len or in the statement version, def foo(): static len = len I think we'd just have to accept that the RHS is evaluated outside the function scope, despite being written within it. That's how static initialisers in C work. -- Greg
On Fri, Sep 23, 2011 at 9:51 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Please read the previous thread from June (linked earlier in this thread). Decorator syntax cannot work without deep magic, because the compiler *doesn't know* that injected names need to be given special treatment. Python's scoping relies on the compiler being able to classify names at compile time into 3 kinds of reference: - local (direct references into the local variable namespace of the executing frame) - cells (indirect references via cells stored on the function object) - unknown (looked up by name at runtime, first in the module globals and then in the builtin namespace) These 3 reference types are baked into the immutable code objects by the compiler - you *cannot* change them later without hacking the bytecode and recreating the function object. Now, we have two 'magical' names ('super' and '__cell__') that cause the compiler to spontaneously do interesting things with namespaces to make Python 3's new simplified (and incredibly convenient) super() invocation work. However, aside from that special case, the rules are very simple: - names bound in the current function are locals (unless marked with 'nonlocal' or 'global') - names bound as locals in an outer function and referenced from the current function are looked up via cells - anything else is treated as an unknown name The 'nonlocal' and 'global' keywords override the 'local by default' behaviour for bound names (forcing the second or third interpretations respectively). The default argument hack effectively creates a 4th namespace option by using the default arguments as "pre-populated locals" - the argument passing machinery is set up so that any parameter not supplied as an argument is filled in on the current frame from its default argument value. By adding additional parameters that are *never* supplied as arguments, the author of a function can create arbitrary locals from expressions that are evaluated when the function is defined rather than when it is called. That means there are four very different ways of looking at potential replacements for this technique: 1. Leave the technique alone, but improve the introspection tools and conventions associated with it def f(x, _i=i): # pydoc would, by default, display the signature as 'f(x)' return x + _i Keyword-only arguments in Py3k already help with this approach to the question, especially when the 'hidden' keyword is prefixed with an underscore to indicate it isn't meant for public consumption. This approach is also highly amenable to monkey-patching, since the default arguments can be deliberately overridden at call time, just like any other parameter. It wouldn't be hard to adjust pydoc to leave out underscore-prefixed keyword only parameters by default, requiring an explicit request to include them. In other words, this approach just involves taking the existing default argument hack, tidying it up a bit, explaining it in the docs, and blessing it as the official way to do things and a technique that experienced Python programmers should know and understand. 2. Definition time parameters This approach keeps the pre-populated locals in the function header, but tweaks the spelling and storage so they're no longer part of the function signature. Two ideas have been put forward for this approach: def f(x, **, i=i): # extending the keyword-only syntax one step further return x + i def f(x) [i=i]: # adding a dedicated set of brackets return x + i The general consensus seems to be that these don't offer enough benefit over the status quo to be worth the hassle. 3. Definition time expressions With a wide variety of proposed spellings (e.g. once, static, atdef), this proposals aims to mark individual expressions for evaluation at function definition time and caching on the function object. At function call time, the value would be inserted in place of the expression. I explained this in my previous email, and Guido has already said '-1' to this approach, so I won't elaborate any further. 4. Function scoped variables This is the approach most analogous to C's static variables - named variables that are shared across all invocations of a function, rather than being local to the current invocation. In essence, each function becomes its own closure - just as a function can share state across invocations by using an outer function for storage, this technique would allow a function to use its *own* cell array for such storage. Framing the idea that way also suggests a fairly obvious spelling: def f(x): nonlocal i=i # Use 'f' as a closure over *itself* return x + i With this spelling, the above would be roughly equivalent to: def outer(): i = i def f(x): return x + i return f f = outer() The only visible difference would be that the cell referenced by 'i' would be stored directly on 'f' rather than on an outer function. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Sep 23, 2011 at 11:11 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Applying this 'functions as their own closure' concept to the classic counter problem: def counter(): x = 0 def increment(): nonlocal x x += 1 return x return increment would become: def counter(): def increment(): nonlocal x=0 x += 1 return x return increment Or, if you wanted a process-global counter: def global_counter(): nonlocal x=0 x += 1 return x Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
This is the last time you mention the decorator solution (aside from further explanation of the problem). Is it being discarded for that reason? I was under the impression that everyone who liked that idea was fully aware that it's deep magic. I would assume it actually creates an entirely new function object with new closure cell to bind the name, similar to your last solution involving the function closing over "itself", but more general and as part of a decorator above the function header. I'm not really seeing a problem other than maybe distaste because it seems "hacky". It can work, though. Devin On Thu, Sep 22, 2011 at 9:11 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Fri, Sep 23, 2011 at 11:39 AM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Yes, it's far too hard to explain what it does in terms of existing Python semantics (the Zen has something to say on that point). Bytecode hackery and other tricks would *work*, but deep magic should only be employed when there aren't any alternatives and the problem is sufficiently common. (PEP 3135, the new super semantics for 3.x, pushes the boundaries of what's reasonable, but it's so easy to *use* that it's worth the additional under the hood complexity). Magical behaviour is also far more likely to cause problems for other implementations - it tends to rely on assumptions that aren't explicitly guaranteed by the language specification. Use of default arguments for pre-initialised function locals is nowhere near common enough to justify deep magic as a solution, so the idea just isn't worth pursuing. That's the real benefit of the "nonlocal i=i" idea: it takes an *existing* concept (i.e. closures), and just tweaks it a bit by avoiding the need for a separate outer scope when all you really want to do is share some state between invocations. Anyone that already understands closures shouldn't have much trouble grasping the idea that the 'innermost containing scope' can now be the function itself rather than the next level up. Any implementation that already correctly handles closures should also be able to cope with the self reference without too much trouble. No new keywords, no new namespace semantics, just a slight tweak to the way the compiler handles def and nonlocal statements. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Sep 22, 2011 at 6:11 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yes, you're right that this can't be done with today's decorators. That doesn't mean it's impossible. Any of the other changes would also require compiler changes. Instead of @inject being a standard decorator, imagine a new kind of decorator that works as follows: a = 10 $inject(b=a) def foo() a = 20 return b+a where the effect of a $inject decorator is that it modifies the behavior of the function as follows: a = 10 _inject_ = inject_decorator(b=a) def foo(): locals().update(_inject_) a = 20 return b + a Yes, I know that update on locals() won't do the right thing but pretend it does. I think the semantics of this are pretty clear and more obvious than the suggested alternatives. Now what about these $-decorators? As defined here, it's pretty specialized to this one operation. Let's imagine that we define $-decorators to be a special class of decorators that are built into Python. We use $-decorators for cases we would like users to think of as decorators but that do some kind of magic that's beyond the capability of a regular decorator. To the user the $ is a signal to the user that this decorator is a bit more special than usual. Right now, there would only be one $-decorator but it's available next time it's desired to add something that just can't quite be a regular decorator. A good objection to adding new things to the language is that they increase the load on people trying to learn the language. If I see a syntax like: def f() [i=i]: pass I can't imagine how I would find out what that does. Search engines don't readily allow searching on special characters. On the other hand, when I see $inject, I go type "python $inject decorator" into my favorite search engine and whether it ignores the $ or not, I'm fairly likely to get good results. Whether the decorator-like syntax is spelled $inject, @@static or something else, I think it's going to be much easier to figure out and remember what's going on. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
Yes, you're right that this can't be done with today's decorators.
It can work with today's decorators and today's syntax, it just requires magic. (Magic doesn't mean "impossible things", it means "things that go below the expected level of abstraction". In this case, things like directly inspecting fields of the function object and creating a new one with modified fields -- that, or else mutating a supposedly-immutable object (also possible)) That doesn't blow down the whole idea of $ special syntactic decorators, though. And maybe even this case is good for them. Definitely there might be room for a more general-case thing for compile-time annotations than new extra-special syntax. Magical decorators that get understood by the compiler have been mentioned before, and are inappropriate for obvious reasons. But if they were prefixed by $ instead of @... I still like real decorators because they can be applied at run-time instead of just compile-time, which gives them additional use-cases. Devin On Thu, Sep 22, 2011 at 10:18 PM, Bruce Leban <bruce@leapyear.org> wrote:
On Thu, Sep 22, 2011 at 7:27 PM, Devin Jeanpierre <jeanpierreda@gmail.com>wrote:
I still like real decorators because they can be applied at run-time instead of just compile-time, which gives them additional use-cases.
The $inject special decorator would need to be recognized at compile time but would be applied at run time just as @decorators are. That is, I see no reason this couldn't work: def foo(a): $inject(b=a) def bar(c): return b+c return bar --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
Devin Jeanpierre wrote:
In this case, I believe that most of the work that needs to be done -- making a copy of the function and code object -- are not magic. They are fully supported in standard Python. The only "magic" is manipulating the bytecode of the code object to that it turns some globals into locals.
That doesn't blow down the whole idea of $ special syntactic decorators, though. And maybe even this case is good for them.
I don't see any reason to introduce extra syntax for a different sort of decorator. What benefit is there? I think this $ proposal actually undermines the argument I am trying to make. A big advantage of using a decorator is that it requires no new syntax and no magic symbols beyond the standard @ symbol. The argument "yes, I like decorators, but I want to use $ instead of @" doesn't really help.
I still like real decorators because they can be applied at run-time instead of just compile-time, which gives them additional use-cases.
I have negative interest in a "magic decorator" that can only work at compile time. If we're going to have that (unnecessary, in my opinion) limitation, then I prefer the static declaration proposal. -- Steven
On Fri, Sep 23, 2011 at 1:26 PM, Steven D'Aprano <steve@pearwood.info> wrote:
That may be the core of the confusion here. Bytecode hacking is NOT supported in standard Python. The language definition pretty much stops at "function objects have a __code__ attribute that refers to the immutable code object defining the algorithm they execute when called". Anything below that point is implementation dependent (hence the disclaimer on the output of the 'dis' module). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
I have read the previous thread, and I'm fully aware that this would be special. I'm pretty sure I even said it would be special in one of my posts :) Excluding the default "do nothing" position, and the minimalist "just add conventions to introspection tools" proposal, I believe that @inject is the least magical proposal made so far. The other proposals require a new keyword, new syntax, or both. They require the compiler to do extra work. @inject requires nothing from the compiler. It all happens when the decorator is called. Comparing it to the voodoo needed for super() is completely unfair. It also offers the most benefits: * all the benefits of the proposed "static" declaration (early-binding, micro-optimisation, monkey-patching) * the ability to patch pre-existing functions * no call-time cost (it's not a wrapper, its a new function) * fewer side-effects than conventional monkey-patching * familiar syntax * obvious semantics * no new keywords * no extra "line-noise" symbols What's not to like? Yes, inject() will be special. But I don't believe it will be magic, or at least not deep magic. We can get most of the way using supported Python functionality already: we can pull a function object apart, make a copy of the pieces as needed, and reassemble them into a new function. All that is supported by Python, none of it requires messing with private attributes: it's not much more magic than what functools.wraps() already does. We can even take the bytecode from the code object, and because it's just a string, we can perform transformations on it. The only "magic" is the knowledge of what transformations to perform. Take this simple example:
3 6 LOAD_FAST 0 (c) 9 LOAD_GLOBAL 0 (d) 12 BINARY_ADD 13 RETURN_VALUE @inject(d=2)(func) would probably look produce something like this: 2 6 LOAD_CONST 2 (2) 9 STORE_FAST 1 (d) 3 0 LOAD_CONST 1 (1) 3 STORE_FAST 0 (c) 4 12 LOAD_FAST 0 (c) 15 LOAD_FAST 1 (d) 18 BINARY_ADD 19 RETURN_VALUE If we had a Python assembler to match the disassembler, it would probably be easy. If we are really allergic to the idea of bytecode manipulation, perhaps there are other ways to solve this. Just tossing ideas around, maybe function objects should keep a copy of their AST around, so that instead of bytecode hacking, you manipulate the AST and recompile the function. That might be less "hacky" than manipulating the bytecode, but I don't know that carrying around the AST for every function is a cost that is worth bearing. But it's an idea. -- Steven
On Fri, Sep 23, 2011 at 1:17 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Bytecode hacks are inherently implementation dependent deep magic. Almost *nothing* about code objects is part of the language specification (IIRC, it expects them to exist, but that's about it). Jython and IronPython use JVM and CLR bytecode. PyPy uses a format at least similar to CPython bytecode (since it was convenient for them to do so), but there's absolutely no requirement for them to keep it the same as the two implementations evolve. The entire *point* of wpython is to use a word-oriented rather than byte-oriented format. Even CPython will happily change the bytecode format between releases - there's a reason a bytecode version marker is embedded in every .pyc file. Taking an existing function apart and putting it back together again in a different way is skating right along that border between language specification and implementation details. Proposals that play in that space are almost guaranteed to be a hack that pays no attention to the overall semantic concepts of the language design and they're simply *not* going to happen. Any reasonable piece of Python code *must* treat code objects as opaque blobs that the interpreter uses to define an algorithm. It can be fun, and sometimes even useful, to go further, but it would be thoroughly inappropriate for the interpreter core or standard library to indulge in that kind of thing. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sep 23, 2011, at 11:11 AM, Nick Coghlan wrote:
Interestingly, PEP 3104 proposes that syntax, as shorthand for nonlocal x; x = 3 though this was not adopted. I think your proposal is a very interesting, modest but useful extension of that original PEP. -Barry
On Fri, Sep 23, 2011 at 11:11 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Alex Gaynor pointed out the above would give an unbound local error. The actual rough equivalent would be more like: def outer(i=i): def f(x): return x + i return f f = outer() Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Instead of a $ decorator, you could let decorators have a __prepare__ method that returns a dict to use for locals, as metaclasses do today. I did a quick test in Python 3.2:
This shows that decorators are executed before the function is built. Since we're looking up the decorator before the function object has been created anyway, it wouldn't be impossible for the locals dictionary of the function to be "modified" before it's created, as we can do with metaclasses. If we did this, inject would look something like this: class inject: def __init__(self, **kwargs): self.kwargs = kwargs def __prepare__(self): return self.kwargs def __call__(self, f): return f And of course, there could be other applications for the __prepare__, such as using OrderedDict or whatever. (Not sure if we'd want to encourage that though…)
On Fri, Sep 23, 2011 at 2:34 PM, Carl Matthew Johnson <cmjohnson.mailinglist@gmail.com> wrote:
Instead of a $ decorator, you could let decorators have a __prepare__ method that returns a dict to use for locals, as metaclasses do today.
No, locals simply don't work like that. They're handled by frame objects and the eval loop, not function objects. There are a whole host of complicated interactions between the compiler's symbol table analysis and code generation, function, code and frame objects and the runtime evaluation loop going on when it comes to handling name lookups in Python. When you're looking at it purely from a Python user's point of view, there are plenty of ideas that seem potentially reasonable on the surface but simply don't fit with the underlying data model of the language. It's hard enough coming up with good proposals for semantic and syntactic tweaks when you *do* know how they all work together to achieve the current behaviour - the current setup is actually pretty good, and the number of ways we could make it worse vastly outnumbers the ways we could conceivably improve it. Try to keep this discussion in perspective: we're talking about handling a niche use case in a slightly more elegant fashion. The magnitude of language change that can be justified for this purpose is tiny. I've thrown plenty of ideas of my own at the task over the years (and seen even more ideas from others), and tweaking the syntax and semantics of nonlocal is the first of them that I've genuinely liked better than the status quo. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Sep 22, 2011 at 10:52 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yeah, that's what I found out when I tried to add in an implicit "__function__" in the locals. I mostly had it working, but it was a hack and a half (a lot worse than super's "@__class__" injection).
+1 -eric
On Sep 22, 2011, at 6:52 PM, Nick Coghlan wrote:
No, locals simply don't work like that. They're handled by frame objects and the eval loop, not function objects.
I feel like I have the same complaint that other people did earlier in the thread: I understand that locals don't work like that *now*. (And they understood that decorators can't do an inject *now*.) When I said "locals" I was using a shorthand, because I don't know all the correct terminology. What I'm proposing is a change to how the system works. I want the byte code for functions to be created differently after this change. Obviously, that can't be done by the function objects, but it will need to be done beforehand. There's probably much more to this sort of change than I realize--maybe even so much that it wouldn't be worth it to do it, since the benefit would be relatively small. If so, that's fine. But objecting that this isn't how it really works feels like it's missing the point to me. The more relevant objection is that it would be too huge of a project to make it work a different way. With respect, -- Carl
On Fri, Sep 23, 2011 at 3:44 PM, Carl Matthew Johnson <cmjohnson.mailinglist@gmail.com> wrote:
I feel like I have the same complaint that other people did earlier in the thread: I understand that locals don't work like that *now*. (And they understood that decorators can't do an inject *now*.) When I said "locals" I was using a shorthand, because I don't know all the correct terminology. What I'm proposing is a change to how the system works. I want the byte code for functions to be created differently after this change. Obviously, that can't be done by the function objects, but it will need to be done beforehand. There's probably much more to this sort of change than I realize--maybe even so much that it wouldn't be worth it to do it, since the benefit would be relatively small. If so, that's fine. But objecting that this isn't how it really works feels like it's missing the point to me. The more relevant objection is that it would be too huge of a project to make it work a different way.
That's exactly my point though - the default argument use cases to be addressed are *trivial*. A trivial problem requires a comparably trivial answer - there is *nothing* in the use cases being discussed that even comes close to justifying substantial changes to the way the language works. Changing the way locals works, allowing variable injection, those are huge, language changing ideas, that require powerful, compelling use cases to motivate them. As far as I am aware, those use cases don't exist (and if they do, they certainly haven't been raised in this thread). It would be really helpful if people could comment on ideas that have at least some chance of making it under that "simple enough to be worth doing" bar. Obviously, I consider adjusting 'nonlocal' so we can use it to store cell references on the function being defined to be one such idea, as it is essentially just blending PEP 3104 with the underlying implementation of PEP 3135's magic '__class__' reference in methods. As far as I can tell, the biggest potential practical problem with it is on the compiler side, where the AST nodes for the initialisation expressions would need to be lifted out for inclusion in the code generated in the surrounding scope. That's not impossible (we already do something similar in deciding whether a def statement is defining an ordinary function or a generator), but the state to be collected is a fair bit more complicated than anything the symbol analysis pass currently handles. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Is there a restriction that nonlocal can be used only inside nested functions and would that limitation have any impact. Also, would it make sense/be possible to execute nested class/function definitions at function definition time. Regards, -Krishnan
On 9/23/2011 7:26 AM, H. Krishnan wrote:
Is there a restriction that nonlocal can be used only inside nested functions.
Yes, althought the current 3.2.2 manual does not make that clear enough. I believe there is a tracker issue to clarify this.
Also, would it make sense/be possible to execute nested class/function definitions at function definition time.
No. The body of a function is executed when the function is called. However, it is already the case that code objects for nested functions *are* compiled just once. So just a nested function object has to be assembled with each call. Default args and cell vars are part of each new function object. -- Terry Jan Reedy
Ron Adam wrote:
Sorry, but that spelling is very far from obvious to me. According to the current meaning of 'nonlocal', it looks like a no-op. I don't understand the reasoning that leads from there to your proposed semantics. You'll also have to explain how that reasoning applies to the following variations: nonlocal i = i + i nonlocal i = i + j nonlocal i = j + k -- Greg
On Sat, 2011-09-24 at 11:12 +1200, Greg Ewing wrote:
Here's my view of what would happen, but you don't say weather or not those are defined together or if they are separate cases. nonlocal i Gives read write access to i in a parent frame. nonlocal i=i Creates a new 'i' in a functions own frame with the value of 'i' from a parent frame at the time the function is defined. (Note, it also makes sense to do it on the first time the function is called. I'm not sure there is any advantage to that.) Once the function is called, the access of 'i' would be in the local frame, and not effect the parent frames 'i' because the local 'i' is found first. As for the various cases, they would work the same except the initial value would be different. My 2cents for what it's worth. Cheers, Ron
On Fri, Sep 23, 2011 at 5:35 PM, ron adam <ron3200@gmail.com> wrote:
The only sensible meaning that could possibly assigned to "nonlocal <var> = <expr>" is that it should be strictly equivalent to nonlocal <var> <var> = <expr> Anything else is too confusing for our users. So your proposal is right out. -- --Guido van Rossum (python.org/~guido)
On 9/22/2011 9:11 PM, Nick Coghlan wrote:
def f(x, _i=i): # pydoc would, by default, display the signature as 'f(x)' return x + _i
I presume you meant def f(x, *, _i=i): return x+_i so that _i will be keyword only.
It certainly makes it impossible to 'accidentally' override with an extra positional parameter. -- Terry Jan Reedy
On Sep 22, 2011, at 3:58 AM, Nick Coghlan wrote:
This reminds me that there was a proposal a few weeks back to be able to refer to the function one is in absolutely. Right now, you can't just slap an attribute onto f in a loop and then get it back at exec time, since the variable f will refer to the last f defined in the loop. But suppose we had some sort of "THIS_FUNCTION" magic variable… Then this would work: fs = [] for i in range(10): def f(): return THIS_FUNC.value f.value = i [f() for f in fs] (Again, it has to be some sort of magical variable, since "return f.value" will refer to the 9 producing function at the end.) This could then be combined with a standard inject decorator. @inject(i=i) def f(): return THIS_FUNC.inject_values.i In cases where you're not in a loop or something, you could just write "f.values.i" instead of using the new magical variable.
On Thu, Sep 22, 2011 at 12:49 PM, Carl Matthew Johnson <cmjohnson.mailinglist@gmail.com> wrote:
This reminds me that there was a proposal a few weeks back to be able to refer to the function one is in absolutely. Right now, you can't just slap an attribute onto f in a loop and then get it back at exec time, since the variable f will refer to the last f defined in the loop. But suppose we had some sort of "THIS_FUNCTION" magic variable…
Yeah, a "THIS_FUNCTION" in the function locals didn't go very far. It would be too easy to abuse it and the implementation doesn't fit well into CPython (as far as I could see). I have a patch in for adding f_func to the frame object (you have to jump through the inspect module to get to it). However, that doesn't help so much for this.
Performance is another concern here. The default argument hack is pretty efficient compared to attribute lookups on the function object. -eric
participants (24)
-
Adam Jorgensen -
Barry Warsaw -
Bruce Leban -
Carl Matthew Johnson -
Chris Rebert -
David Townshend -
Devin Jeanpierre -
Eric Snow -
Greg Ewing -
Guido van Rossum -
H. Krishnan -
Jakob Bowyer -
Jan Kaliszewski -
Jim Jewett -
Matt Joiner -
Mike Graham -
MRAB -
Nick Coghlan -
Paul Moore -
ron adam -
Ron Adam
-
Steven D'Aprano -
Sven Marnach -
Terry Reedy