On Thu, Jul 8, 2021 at 6:25 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, 6 Jul 2021, 7:56 am Jim Baker, <jim.baker@python.org> wrote:
On Mon, Jul 5, 2021, 2:40 PM Guido van Rossum <guido@python.org> wrote:
FWIW, we could make f-strings properly nest too, like you are proposing for backticks. It's just that we'd have to change the lexer. But it would not be any harder than would be for backticks (since it would be the same algorithm), nor would it be backward incompatible. So this is not an argument for backticks.
Good point. At some point, I was probably thinking of backticks without a tag, since JS supports this for their f-string like scenario. but if we always require a tag - so long as it's not a prefix already in use (b, f, r, fr, hopefully not forgetting as I type this email in a parking lot...) - then it can be disambiguated using standard quotes.
There's a deferred PEP proposing a resolution to the f-string nesting limitations: https://www.python.org/dev/peps/pep-0536/#motivation
Thanks for pointing that PEP out - this looks quite reasonable for both f-strings and for the tagged templates proposed here. I believe some limitations on nesting expressions in f-strings with quote changing was introduced in a relatively recent fix in 3.8 or 3.9, but I need to find the specifics.
Separately, should there be a way to *delay* evaluation of the templated expressions (like we explored in our private little prototype last year)?
I think so, but probably with an explicit marker on *each* deferred expression. I'm in favor of Julia's expression quote, which generally needs to be enclosed in parentheses, but possibly not needed in expression braces (at the risk of looking like a standalone format spec). https://docs.julialang.org/en/v1/manual/metaprogramming/
So this would like x = 42 d = deferred_tag"Some expr: {:(x*2)}"
All that is happening here is that this being wrapped in a lambda, which captures any scope lexically as usual. Then per that experiment you mentioned, it's possible to use that scope using fairly standard - or at least portable to other Python implementations - metaprogramming, including the deferred evaluation of the lambda. (No frame walking required!)
Other syntax could work for deferring.
It reminds me of the old simple implicit lambda proposal: https://www. python.org/dev/peps/pep-0312/
The old proposal has more ambiguities to avoid now due to type hinting syntax but a parentheses-required version could work: "(:call_result)"
PEP 312 - suitably modified - could be quite useful for both deferring evaluation in tagged templates and function calls (or other places where we might want to put in a lambda). Ideally we can use minimum parens as well, so it's tag"{:x+1}" f(:x+1, :y+2) (but I haven't looked at ambiguities here). A further idea that might make the approach in PEP 312 more powerful than simply being an implicit lambda is if this was like Julia's quoted expressions https://docs.julialang.org/en/v1/manual/metaprogramming/#Quoting and we recorded the *text body of the expression*. (Such recording would presumably be done in any template scheme, but let's make this explicit now.) So let's call this implicit lambda with recorded text body a *quoted expression*. What's nice about doing such quoted expressions is that the lambda also captures any lexical scope. So if I have an expression tag"{:x*y}", the variables x and y are referenced appropriately. This means it would be possible to do such things as rewrite complex expressions - eg an index query on a Pandas data frame - while avoiding the use of dynamic scope. I wrote a small piece of code to show how this can be exercised. The text of the expression is simply an attribute on the lambda named "body", but we might have a new type defined (eg types.QuotedExpression). ``` from functools import update_wrapper from textwrap import dedent from types import FunctionType def rewrite(f, new_body): # Create a temporary outer function with arguments for the free variables of # the original lambda wrapping an expression. This approach allows the new # inner function, which has a new body, to compile with its dependency on # free vars. # # When called, this new inner function code object can continue to access # these variables from the cell vars in the closure - even without this # outer function, which we discard. # # This rewriting is generalizable to arbitrary functions, although the # syntax we are exploring is only for expressions. # # A similar idea - for monkeypatching an inner function - is explored in # https://stackoverflow.com/a/27550237 by Martijn Pieters' detailed answer. # # Related ideas include https://en.wikipedia.org/wiki/Lambda_lifting and # lambda dropping where the free vars are lifted up/dropped from the # function parameters. This requires the (usual :) extra level of # indirection by another function. scoped = dedent(f""" def outer({", ".join(f.__code__.co_freevars)}): def inner(): return {new_body} """) capture = {} exec(scoped, f.__globals__, capture) # NOTE: outer is co_consts[0], inner is co_consts[1] - this may not be # guaranteed by the compiler, so iterate over if necessary. inner_code = capture["outer"].__code__.co_consts[1] new_f = FunctionType( inner_code, f.__globals__, closure=f.__closure__) update_wrapper(new_f, f) new_f.body = new_body return new_f def test_rewrite(): x = 2 y = 3 # x, y and free vars in scopeit, and its own nested lambdas, assigned # to f and g below. We will also introduce an additional free var z # in the parameter of scopeit. def scopeit(z): f = lambda: x * y + z f.body = "x * y + z" print(f"{f()=}") # We can do an arbitrary manipulation on the above expression, so # long as we use variables that are in the original scope (or symtab). # Note that adding new variables will look up globally (may or may # not exist, if not a NameError is raised). print(f.__code__.co_freevars) g = rewrite(f, "-(x * y + z)") print(f"{g()=}") scopeit(5) if __name__ == "__main__": test_rewrite() ``` Similar ideas were explored in https://github.com/jimbaker/fl-string-pep/issues - Jim On Thu, Jul 8, 2021 at 6:25 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, 6 Jul 2021, 7:56 am Jim Baker, <jim.baker@python.org> wrote:
On Mon, Jul 5, 2021, 2:40 PM Guido van Rossum <guido@python.org> wrote:
FWIW, we could make f-strings properly nest too, like you are proposing for backticks. It's just that we'd have to change the lexer. But it would not be any harder than would be for backticks (since it would be the same algorithm), nor would it be backward incompatible. So this is not an argument for backticks.
Good point. At some point, I was probably thinking of backticks without a tag, since JS supports this for their f-string like scenario. but if we always require a tag - so long as it's not a prefix already in use (b, f, r, fr, hopefully not forgetting as I type this email in a parking lot...) - then it can be disambiguated using standard quotes.
There's a deferred PEP proposing a resolution to the f-string nesting limitations: https://www.python.org/dev/peps/pep-0536/#motivation
Separately, should there be a way to *delay* evaluation of the templated expressions (like we explored in our private little prototype last year)?
I think so, but probably with an explicit marker on *each* deferred expression. I'm in favor of Julia's expression quote, which generally needs to be enclosed in parentheses, but possibly not needed in expression braces (at the risk of looking like a standalone format spec). https://docs.julialang.org/en/v1/manual/metaprogramming/
So this would like x = 42 d = deferred_tag"Some expr: {:(x*2)}"
All that is happening here is that this being wrapped in a lambda, which captures any scope lexically as usual. Then per that experiment you mentioned, it's possible to use that scope using fairly standard - or at least portable to other Python implementations - metaprogramming, including the deferred evaluation of the lambda. (No frame walking required!)
Other syntax could work for deferring.
It reminds me of the old simple implicit lambda proposal: https://www.python.org/dev/peps/pep-0312/
The old proposal has more ambiguities to avoid now due to type hinting syntax but a parentheses-required version could work: "(:call_result)"
Cheers, Nick.