[Python-ideas] Copy (and/or pickle) generators

[Joseph Jevnik]

This was already posted in the thread, but
https://github.com/llllllllll/cloudpickle-generators is just an
extension to the standard pickle machinery and is able to support
closures, nonlocals, and globals:
https://github.com/llllllllll/cloudpickle-generators/blob/master/cloudpickle_generators/tests/test_cloudpickle_generators.py.
It can even support the exotic case of a generator closing over
itself.

The state that needs to be serialized for a generator is:

1. the frame's locals
2. the frame's globals
3. the closure cells
4. the lasti of the frame
5. the frame's data stack
6. the frame's block stack
7. the frame's suspended exception*

The frame's suspended exception is the exception that is stored when
you have code like:

try:
    raise ValueError()
except Exception:
    yield value
    raise

The frame stores the (type, value, traceback) so that it can make the
raise statement work after the yield.

You need to be careful to check for recursion in the globals and
closure because the generator instance may get stored there. You also
need to check the locals because the generator instance could be sent
back into itself and stored in a local. You also need to check the
data stack for recursion because the instance could be sent into
itself and then left on the stack between yields, like if you use a
yield expression in the middle of a tuple creation like:

a = (the_generator_instance, (yield))

Extracting the lasti, data stack, block stack and held exception
require a little C, the rest can be pulled from pure Python.

On Wed, Jun 20, 2018 at 12:27 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 20 Jun 2018 12:15:18 -0400
> Yury Selivanov <yselivanov.ml at gmail.com>
> wrote:
>>
>> > Finally, another comment made the point that there wasn't a strong use case given for it. With the data science libraries that have sprung up around Python in the intervening years, I believe there now is one.
>>
>> As Guido has pointed out, pickling generators would require proper
>> pickling of the entire frame stack (otherwise generators that use
>> "global" or "nonlocal" won't unpickle correctly).
>
> Depends what level of automatic (magic?) correctness you're expecting.
> A generator is conceptually an iterator expressed in a different syntax.
> If you define an iterator object, it will probably get pickling for
> free, yet pickling it won't bother serializing the global variables that
> are accessed from its __next__() and send() methods.
>
> A generator needn't be different: you mainly have to be careful to
> serialize its module's __name__, so that you can lookup the frame's
> global dict by module name when the generator is recreated.
>
> By contrast, closure variables would be an issue.  But a first
> implementation could simply refuse to pickle generators that access an
> enclosing local state.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/