really like what Paul Moore wrote here as it matches a *LOT* of what I have been feeling as I have been reading this whole discussion; specifically:

I've spend a lot of time thinking about this, and what the issues are.

I think they are multi-fold:

The full example, which took me a few hours to write is available here (its a very very reduced example from a real parser of the python language written in python):
Here is the result of running the code -- which reads & executes demo1.py (importing & executing demo2.py twice): [Not by executing, I mean the code is running its own parser to execute it & its own code to emulate an 'import' -- thus showing nested contexts):

It creates two input files for testing -- demo1.py:

print 1
print 8 - 2 * 3
import demo2
print 9 - sqrt(16)
print 10 / (8 - 2 * 3)
import demo2
print 2 * 2 * 2 + 3 - 4

And it also creates demo2.py:

print 3 * (2 - 1)
error
print 4

There are two syntax errors (on purpose) in the files, but since demo2.py is imported twice, this will show three syntax errors.

Running the code produces the following:

demo1.py#1: expression '1' evaluates to 1
demo1.py#2: expression '8 - 2 * 3' evaluates to 2
demo1.py#3: importing module demo2
demo2.py#1: expression '3 * (3 - 2)' evaluates to 3
demo2.py#2: UNKNOWN STATEMENT: 'error'
demo2.py#3: expression '4' evaluates to 4
demo1.py#4: UNKNOWN ATOM: ' sqrt(16)'
demo1.py#5: expression '10 / (8 - 2 * 3)' evaluates to 5
demo1.py#6: importing module demo2
demo2.py#1: expression '3 * (3 - 2)' evaluates to 3
demo2.py#2: UNKNOWN STATEMENT: 'error'
demo2.py#3: expression '4' evaluates to 4
demo1.py#7: expression '2 * 2 * 2 + 3 - 4' evaluates to 7

This code demonstrates all of the following:

So given all of the above, I'd first like to focus on the generator:

        def __iter__(self):
            while not self.finished:
                self.loop += 1

                yield self

I therefore suggest the following:

Here is the generator in my code sample:

    #
    #   Here is our generator to walk over a file.
    #
    #   This generator has three sections:
    #
    #       generator_start     - Always run when the generator is started.
    #                             This opens the file & reads it.
    #
    #       generator_next      - Run each time the generator needs to retrieve
    #                             The next value.
    #
    #       generator_stop      - Called when the generator is going to stop.
    #
    def iterate_lines(path):
        data_lines = None


        def generator_startup(path):
            nonlocal current_path, data_lines

            with open(path) as f:
                current_path = path
                data         = f.read()

            data_lines = tuple(data.splitlines())


        def generator_next():
            nonlocal current_line, line_number

            for current_line in data_lines:
                line_number += 1
                line_position = 0

                yield current_line

            generator_stop()


        def generator_stop():
            current_path  = None
            line_number   = 0
            line_position = 0


        generator_startup(path)

        return generator_next()

This generator demonstrates the following:

Here is (a first draft) proposal and how I would like to re-write the above generator, so it would have its own first class syntax:

    generator iterate_lines(path):
        local   data_lines = None
        context current_path, current_line, line_number, line_position

        start:
            with open(path) as f:
                current_path = path
                data         = f.read()

            data_lines = tuple(data.splitlines())

        next:
            for current_line in data_lines:
                line_number += 1
                line_position = 0

                yield current_line

        stop:
            current_path  = None
            line_number  = 0
            line_position  = 0

This:

  1. Adds a keyword 'generator' so its obvious this is a generator not a function.
  2. Declares it variables (data_lines)
  3. Declares which context variables it wants to use (current_path, currentline, line_number, & line_position)
  4. Has a start section that immediately gets executed.
  5. Has a next section that executes on each call to __next__ (and this is where the yield keyword must appear)
  6. Has a stop section that executes when the generator receives a StopIteration.
  7. The compiler could generate equally efficient code for generators as it does for current generators; while making the syntax clearer to the user.
  8. The syntax is chosen so the user can edit it & convert it to a class generator.

Given the above:

The reason for its own syntax is to allow us to think more clearly about the different parts of a generator  & then makes it easier for the user to choose which part of the generator interacts with contexts & which context.  In particular the user could interact with multiple contexts (one in the start section & a different one in the next section).

[Also for other generators I think the syntax needs to be extended, to something like:

    next(context):

       use context:

           ....

Allowing two new features --- requesting that the __next__ receive the context of the caller & secondly being able to use that context itself.

Next, moving on to contexts:

The sample code I have actually emulates contexts using non-local, so as to demonstrate the idea I am explaining.

Thanks,

Amit

P.S.: As I'm very new to python ideas, I'm not sure if I should start a separate thread to discuss this or use the current thread.  Also I'm not sure if I should attached the sample code here or not ... So I just provided the link above.

On Fri, Oct 13, 2017 at 4:29 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 13 October 2017 at 19:32, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
>>> It seems simpler to have one specially named and specially called function
>>> be special, rather than make the semantics
>>> more complicated for all functions.
>>
>
> It's not possible to special case __aenter__ and __aexit__ reliably
> (supporting wrappers, decorators, and possible side effects).
>
>> +1.  I think that would make it much more usable by those of us who are not
>> experts.
>
> I still don't understand what Steve means by "more usable", to be honest.

I'd consider myself a "non-expert" in async. Essentially, I ignore it
- I don't write the sort of applications that would benefit
significantly from it, and I don't see any way to just do "a little
bit" of async, so I never use it.

But I *do* see value in the context variable proposals here - if only
in terms of them being a way to write my code to respond to external
settings in an async-friendly way. I don't follow the underlying
justification (which is based in "we need this to let things work with
async/coroutines) at all, but I'm completely OK with the basic idea
(if I want to have a setting that behaves "naturally", like I'd expect
decimal contexts to do, it needs a certain amount of language support,
so the proposal is to add that). I'd expect to be able to write
context variables that my code could respond to using a relatively
simple pattern, and have things "just work". Much like I can write a
context manager using @contextmanager and yield, and not need to
understand all the intricacies of __enter__ and __exit__. (BTW,
apologies if I'm mangling the terminology here - write it off as part
of me being "not an expert" :-))

What I'm getting from this discussion is that even if I *do* have a
simple way of writing context variables, they'll still behave in ways
that seem mildly weird to me (as a non-async user). Specifically, my
head hurts when I try to understand what that decimal context example
"should do". My instincts say that the current behaviour is wrong -
but I'm not sure I can explain why. So on that example, I'd ask the
following of any proposal:

1. Users trying to write a context variable[1] shouldn't have to jump
through hoops to get "natural" behaviour. That means that suggestions
that the complexity be pushed onto decimal.context aren't OK unless
it's also accepted that the current behaviour is wrong, and the only
reason decimal.context needs to replicated is for backward
compatibility (and new code can ignore the problem).
2. The proposal should clearly establish what it views as "natural"
behaviour, and why. I'm not happy with "it's how decimal.context has
always behaved" as an explanation. Sure, people asking to break
backward compatibility should have a good justification, but equally,
people arguing to *preserve* an unintuitive current behaviour in new
code should be prepared to explain why it's not a bug. To put it
another way, context variables aren't required to be bug-compatible
with thread local storage.

[1] I'm assuming here that "settings that affect how a library behave"
is a common requirement, and the PEP is intended as the "one obvious
way" to implement them.

Nick's other async refactoring example is different. If the two forms
he showed don't behave identically in all contexts, then I'd consider
that to be a major problem. Saying that "coroutines are special" just
reads to me as "coroutines/async are sufficiently weird that I can't
expect my normal patterns of reasoning to work with them". (Apologies
if I'm conflating coroutines and async incorrectly - as a non-expert,
they are essentially indistinguishable to me). I sincerely hope that
isn't the message I should be getting - async is already more
inaccessible than I'd like for the average user.

The fact that Nick's async example immediately devolved into a
discussion that I can't follow at all is fine - to an extent. I don't
mind the experts debating implementation details that I don't need to
know about. But if you make writing context variables harder, just to
fix Nick's example, or if you make *using* async code like (either of)
Nick's forms harder, then I do object, because that's affecting the
end user experience.

In that context, I take Steve's comment as meaning "fiddling about
with how __aenter__ and __aexit__ work is fine, as that's internals
that non-experts like me don't care about - but making context
variables behave oddly because of this is *not* fine".

Apologies if the above is unhelpful. I've been lurking but not
commenting here, precisely because I *am* a non-expert, and I trust
the experts to build something that works. But when non-experts were
explicitly mentioned, I thought my input might be useful.

The following quote from the Zen seems particularly relevant here:

    If the implementation is hard to explain, it's a bad idea.

(although the one about needing to be Dutch to understand why
something is obvious might well trump it ;-))

Paul
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/