[Python-ideas] PEP draft: context variables

[Amit Green]

Generators are a wonderful feature of the python language, and one of its
best idea.

They are initially very intuitive to understand & easy to use.

However, once you get beyond that; they are actually quite confusing
because their behavior is not natural.

Thus they have a initial easy learning & acceptance curve; and then as you
go from initial use to more advanced use, there is a sudden "bump" in the
learning curve, which is not as smooth as it could be.

Specifically, the fact that when you call them, they do not actually call
your code  (but instead call a wrapper) is quite confusing.

Example:

    import __main__


    def read_and_prefix_each_line(path):
        with open(path) as f:
            data = f.read()

        for s in data.splitlines():
            yield '!' + s

    def print_prefixed_file(path):
        reader = read_and_prefix_each_line(path)                     #LINE
12

        print('Here is how %r looks prefixed' % path)

        for s in
reader:
#LINE 16
            print(s)

    print_prefixed_file(__main__.__file__)
    print_prefixed_file('nonexistent')

Will produce the following:

Traceback (most recent call last):
  File "x.py", line 20, in <module>
    print_prefixed_file('nonexistent')
  File "x.py", line 16, in print_prefixed_file
    for s in reader:
  File "x.py", line 5, in read_and_prefix_each_line
    with open(path) as f:
IOError: [Errno 2] No such file or directory: 'nonexistent'

This is quite confusing to a person who has been using generators for a
month, and thinks they understand them.

WHY is the traceback happening at line 16 instead of at line 12, when the
function is called?

It is much more intuitive, and natural [to a beginner], to expect the
failure to open the file "nonexistent" to happen at line 12, not line 16.

So, now, the user, while trying to figure out a bug, has to learn that:

   - NO, calling a generator (which looks like a function) does not
   actually call the body of the function (as the user defined it)
   - Instead it calls some generator wrapper.
   - And finally the first time the __next__ method of the wrapper is
   called, the body of the function (as the user defined it) gets called.

And this learning curve is quite steep.

It is made even harder by the following:

    >>> def f(): yield 1
    ...
    >>> f
    <function f at 0x7f748507c5f0>

So the user is now convinced that 'f' really is a function.

Further investigation makes it even more confusing:

    >>> f()
    <generator object f at 0x7f74850716e0>

At this point, the user starts to suspect that something is kind of unusual
about 'yield' keyword.

Eventually, after a while, the user starts to figure this out:

   >>> def f(): print('f started'); yield 1
    ...
    >>> f()
    <generator object f at 0x7f3a0baf5fc0>
    >>> f().__next__()
    f started
    1
    >>>

And eventually after reading
https://docs.python.org/3/reference/datamodel.html the following sentence:

"The following flag bits are defined for co_flags: bit 0x04 is set if the
function uses the *arguments syntax to accept an arbitrary number of
positional arguments; bit 0x08 is set if the function uses the **keywords
syntax to accept arbitrary keyword arguments; bit 0x20 is set if the
function is a generator."

Finally figures it out:

    >>> def f(): yield 1
    ...
    >>> f
    <function f at 0x7f73f38089d8>
    >>> f.__code__.co_flags & 0x20
    32

My point is ... this learning process is overly confusing to a new user &
not a smooth learning curve.

Here is, just a quick initial proposal on how to fix it:
    >>> def f(): yield 1
    >>> ...
    Syntax Error: the 'yield' keyword can only be used in a generator;
please be sure to use @generator before the definition of the generator

    >>> @generator
    ... def f(): yield 1
    ...
    >>> f
     <generator f at 0x7f73f38089d8>

Just the fact it says '<generator f at ...' instead of '<function f at ...'
would be a *BIG* help to starting users.

This would also really drive home, from the start, the idea to the user,
that:

   - A generator is special, is not a function
   - Calling the generator does not call the generator, but its wrapper.
   (the generator is not called until its __next_ method is called)


Next:


   1. Don't make this the default; but require: from __future__ import
   generator to activate this feature (for the next few releases of python)
   2. Regarding contexts, *REQUIRE* an argument to generator that tells it
   how to have the generator interact with contexts.  I.E.: Something like:

@generator(catpure_context_at_start = true)

def f(): yield 1

With three options: (a) capture context at start; (b) capture context on
first call to __next__; (c) don't capture context at all, but have it work
the natural way, you expressed two emails ago (i.e.: each time use the
context of the caller, not a special context for the generator).

Finally if a user attempts to user a generator with contexts but without
one of these three parameters, throw a syntax error & tell the user the
context usage must be specified.

The major point of all this, is make the learning curve easier for users,
so generators are:

   - Intuitive, easy to pick up & quick use (as they currently are)
   - Intuitive easy to pick up & quick to use, as you go from a beginner to
   a medium level user (making it easier to learn their specific in's & out's)
   - Intuitive, easy to pick up & quick to use, as you go from medium user
   to advanced user (and need to make them interact in different way with
   contexts, etc).








On Sun, Oct 15, 2017 at 9:33 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 15 October 2017 at 13:51, Amit Green <amit.mixie at gmail.com> wrote:
> > Once again, I think Paul Moore gets to the heart of the issue.
> >
> > Generators are simply confusing & async even more so.
> >
> > Per my earlier email, the fact that generators look like functions, but
> are
> > not functions, is at the root of the confusion.
>
> I don't agree. I don't find generators *at all* confusing. They are a
> very natural way of expressing things, as has been proven by how
> popular they are in the Python community.
>
> I don't *personally* understand async, but I'm currently willing to
> reserve judgement until I've been in a situation where it would be
> useful, and therefore needed to learn it.
>
> Paul
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20171015/9fdfb8d3/attachment.html>