[Python-ideas] combining two threads: switch statements and inline functions

[Andrew Barnert]

From: spir <denis.spir at gmail.com>

Sent: Wednesday, February 12, 2014 2:27 AM


> On 02/12/2014 11:07 AM, Andrew Barnert wrote:
>>  From: Steven D'Aprano <steve at pearwood.info>
>> 
>> 
>>  I think neither Alex Rodrigues's initial idea nor Bruce Leban's 
> side idea area any of these.
>> 
>>  [...]
>> 
>>  Meanwhile, Alex was proposing something very different. Really, what he 
> wants is runtime computational (i.e., bytecode) macros. Look at his motivating 
> example:
>> 
>>       @inline
>>       def element_wise(func):
>>           func = inline(func)
>>           for …:
>>               func()
>> 
>>       def is_symmetric(matrix):
>>           element_wise(lambda: if matrix[i][j] != matrix[j][i]: return 
> False)
>>           return True
>> 
>> 
>>  Forget about how the variables work for the moment (and the fact that, on 
> top of his main desire, he _also_ wants a statement inside his lambda…) and look 
> at that "return False". Clearly it has to return from not just the 
> lambda, but element_wise, and is_symmetric. That's the key here; that's 
> what makes it "inline" in Alex's terms. It's not a function on 
> the stack with its own frame, it's just code that gets compiled as a 
> function, but then executed inline directly in the calling function's frame. 
> This doesn't quite work for a few relatively minor reasons (functions get an 
> implicit "return None", variables are referred to by index and may not 
> have the right indices, etc.), but the idea isn't nonsense.
>> 
>>  Once you see that, dynamic vs. lexical scope is a red herring. Either way, 
> the stack frame in which matrix gets evaluated is the stack frame of 
> is_symmetric. The existing closure variable lookup does the right thing. (Note 
> that the definition of the lambda forces matrix to be looked up via *_DEREF in 
> is_symmetric, so the *_DEREF code in the lambda is correct when inlined.) And it 
> would _have_ to be right for the return-through to make any sense. Dynamic scope 
> doesn't seem to add any possibilities here that it wouldn't already add 
> in regular functions; I think it's orthogonal, and irrelevant, to the 
> proposal. But I could be wrong.
>> 
>>  Anyway, I don't think he really needs runtime macros; he only needs 
> "func = inline(func)" because element_wise itself is defined as a 
> function and converted to a macro with "@inline". If the latter were 
> compile-time, the former could be as well. (Imagine a "defmacro" 
> statement and a "lambdamacro" expression.) And that eliminates a whole 
> lot of complexity (dynamically generating dynamic code-fixup code, etc.). And if 
> you don't need runtime macros, you probably don't need computational 
> macros, which eliminates even more (e.g., in the AST, "matrix" is a 
> name, not an index). The key thing he wants is macros. But what he asked for is 
> runtime computational macros.
> 
> How does this relate to Ruby blocks, procs, and lambdas? One of them is to 
> solves the issue of which-is-the-actual-piece-of-code-to-return-from, IIRC 
> (dunno much of Ruby), like your motivating example of "inline" funcs 
> here.

I think this is what you're getting at:

Despite what Ruby programmers think, Ruby functions aren't at all like Python functions. They're not closures, they're not dynamically-created objects, they can't be passed around as first-class values, and so on. Ruby procs, on the other hand, _are_ all of that. However, it's functions that get stack frames; procs run parasitically within functions. This means that, in some ways, they're similar to runtime macros. Relevant here, a return statement in a proc actually returns from the calling function. (Let's ignore the weird quirks that make it hard to actually put a return statement in a proc…)

Blocks are just the syntax for defining proc bodies, and lambda is a special syntax that allows you to create procs with more function-like semantics (including "diminutive return", meaning return statements just exit the proc rather than its caller), which is the opposite of what we're looking for here. (Also, to make things more confusing, Ruby also has methods—which have nothing to do with classes; a method is what you get by calling the method function on the symbol for a function, and it's basically a proc-like thing that wraps the function and remembers its name.)

Python could adopt this kind of two-level callable scheme. At the Python level, we could have separate syntax for defining and calling "inline functions", and a separate type for them. We'd need some way to do "diminutive return" when you want it (usually) while allowing full return when you want that, whether Ruby's way (a return statement is a full return; falling off the end of the function diminutive-returns the last expression evaluated) or something different. Under the covers, I think all you'd need to do is add an inline function stack as a member of the normal function stack frame. This wouldn't be running them "inline" in the sense that others in the thread asked for, but I think it might give Alex everything he's asking for. I don't think this would be a change worth doing, but if someone disagrees, they can probably work it through from here.

Of course in Ruby, you can't create a proc from a function. (You can wrap a function in a proc by passing its name just by calling it in a block—or, just use a method instead—but either way, that doesn't do any good. The function's just going to return to the proc.) Just as I suggested that he doesn't actually need that feature with macros, I don't think he needs it with procs either. There's no reason to create element_wise and the lamdba as functions and then turn them into procs later; just define element_wise as a proc that takes a proc, and define the lambda as a proc, and you're done. But in Python, with the scheme I suggested above, it might be possible to do this, at least if the syntax for differentiating diminutive and full returns were available in normal functions (where they'd both do the same thing, but it could be useful to distinguish them for functions that you plan to turn into inline functions later).