[Python-ideas] A "local" pseudo-function

[Tim Peters]

[Tim]
>> Enormously harder to implement than binding expressions, and the
>> latter (to my eyes) capture many high-value use cases "good enough".

[Steven D'Aprano <steve at pearwood.info>]
> And yet you're suggesting an alternative which is harder and more
> confusing.

I am?  I said at the start that it was a "brain dump".  It was meant
to be a point of discussion for anyone interested.  I also said I was
more interested in real use cases from real code than in debating, and
I wasn't lying about that ;-)

Since no real new use cases (let alone compelling ones) have turned up
yet, I'm ready to drop it for now.


> What's the motivation here for re-introducing sublocal
> scopes, if they're hard to do and locals are "good enough"?

That's why I wanted to see if there were significant unaddressed use
cases.  That _my_ particular itches would be scratched "good enough"
if the PEP is accepted doesn't imply everyone's will be.  And my
particular itches will continue to annoy if the PEP is rejected.


> That's not a rhetorical question: why have you suggested this sublocal
> scoping idea?

Putting an idea out for discussion isn't suggesting it be adopted.
The list is named "python-ideas", not "python-advocacy-death-match"
;-)


> PEP 572 stopped talking about sublocals back in revision 2
> or so, and as far as I can see, *not a single objection* since has
> been that the variables weren't sublocal.

Meh.  Chris didn't seem all that thrilled about dropping them, and I
saw a number of messages more-than-less supporting the idea _before_
they were dropped.  When it became clear that the PEP didn't stand a
chance _unless_ they were dropped, nobody was willing to die for it,
because they weren't the PEP's _primary_ point.


> For what it is worth, if we ever did introduce a sublocal scope, I
> don't hate Nick's "given" block statement:
>
> https://www.python.org/dev/peps/pep-3150/

And Chris just tried introducing it again.   That's in reference to
the last sentence of your reply:

    I don't think that sublocal scopes is a recurring issue

How many times does it have to come up before "recurring" applies? ;-)
 I've seen it come up many times over ... well, literally decades by
now.


> ...
> While I started off with Python 1.5, I wasn't part of the discussions
> about nested scopes. But I'm astonished that you say that nested scopes
> were controversial. *Closures* I would completely believe, but mere
> lexical scoping? Astonishing.

But true.  Guido agonized over it for a long time.  Limiting to 3
scopes was a wholly deliberate design decision at the start, not just,
.e.g, due to lack of time to implement lexical scoping at the start.
And that shouldn't be surprising given Python's start as "somewhere
between a scripting language and C", and the many influences carried
over from Guido's time working on ABC's implementation team (ABC had
no lexical scoping either - nor, if I recall correctly, even textual
nesting of its flavor of functions).

I'm glad he tried it!  Everyone learned something from it.


> Even when I started, as a novice programmer who wouldn't have recognised
> the term "lexical scoping" if it fell on my head from a great height, I
> thought it was strange that inner functions couldn't see their
> surrounding function's variables. Nested scopes just seemed intuitively
> obvious: if a function sees the variables in the module surrounding it,
> then it should also see the variables in any function surrounding it.
>
> This behaviour in Python 1.5 made functions MUCH less useful:
>
>
> >>> def outer():
> ...     x = 1
> ...     def inner():
> ...             return x
> ...     return inner()
> ...
> >>> outer()
> Traceback (innermost last):
>   File "<stdin>", line 1, in ?
>   File "<stdin>", line 5, in outer
>   File "<stdin>", line 4, in inner
> NameError: x
>
>
> I think it is fair to say that inner functions in Python 1.5 were
> crippled to the point of uselessness.

I don't think that's fair to say.  A great many functions are in fact
... functions ;-)  That is, they compute a result from the arguments
passed to them.  They don't need more than that, although being able
to access globals and builtins and import whatever they want from the
standard library made them perfectly capable of doing a whole lot more
than just staring at their arguments.

To this day, _most_ of the nested functions I write would have worked
fine under the original scoping rules, because that's all they need.
Many of the rest are recursive, but would also work fine if I passed
their names into them and rewrote the bits of code to do recursive
calls via the passed-in name.  But, yes, I am relieved I don't need to
do the latter anymore ;-)

... [snip similar things about closures] ...


>> Since then, Python has gone down a pretty bizarre path, inventing
>> sublocal scopes on an ad hoc basis by "pure magic" when their absence
>> in some specific context seemed just too unbearable to live with
>> (e.g., in comprehensions).  So we already have sublocal scopes, but in
>> no explicit form that can be either exploited or explained.
>
> I'm not entirely sure that comprehensions (including generator
> expressions) alone counts as "a path" :-) but I agree with this. I'm not
> a fan of comprehensions being their own scope. As far as I am concerned,
> leakage of comprehension variables was never a problem that needed to be
> solved, and was (very occasionally) a useful feature. Mostly for
> introspection and debugging.
>
> But the decision was made for generator comprehensions to be in their
> own scope, and from there I guess it was inevitable that list
> comprehensions would have to match.

I didn't mind comprehensions "leaking" either.  But I expect the need
became acute when  generator expressions were introduced, because the
body of those can execute in an environment entirely unrelated to the
definition site:

    def process(g):
        i = 12
        for x in g:
            pass
        print(i) # was i clobbered?  nope!

    process(i for i in range(3))

... [snip more "head arguments" about "=" vs "=="] ...

>> But I could compromise ;-)
>>
>> - There must be at least one argument.
>> - The first argument must be a binding.
>> - All but the last argument must also be bindings.
>> - If there's more than one argument, the last argument must be an expression.

> That's not really a complete specification of the pseudo-function
> though, since sometimes the sublocals it introduces extend past the
> final parenthesis and into the subsequent block.

It was talking about compile-time-checkable syntactic requirements,
nothing about semantics.


> What will, for example, this function return?
>
> spam = eggs = "global"
> def func(arg=local(spam="sublocal", eggs="sublocal", 1)):
>     eggs = "local"
>     return (spam, eggs)

It would return (spam, "local"), for whatever value `spam` happened to
be bound to at the time of the call.  `local()` had magically extended
scope only in `if`/`elif` and `while` openers.  In this example, it
would the same if the `def` had been written

def func(arg=1):

Remember that default argument values are computed at the time `def`
is executed, and never again.  At that time, local(spam="sublocal",
eggs="sublocal", 1) created two local names that are never referenced,
threw the new scope away, and returned 1, which was saved away as the
default value for `arg`.

> Even if you think nobody will be tempted to write such "clever" (dumb?)
> code, the behaviour still has to be specified.

Of course, but that specific case wasn't even slightly subtle ;-)