[Python-ideas] Copy (and/or pickle) generators

Yury Selivanov yselivanov.ml at gmail.com
Wed Jun 20 12:15:18 EDT 2018


On Wed, Jun 20, 2018 at 6:34 AM Micheál Keane <ffaristocrat at gmail.com> wrote:
[..]

> First, I couldn't find any real discussion about it after 2011 so I had no idea if the reasons it was ruled unfeasible with Python 2 still held nearly 10 years later with Python 3. I was mainly wondering if all the recent asynchronous work had changed things significantly. Apparently not?

No, as message passing works good enough.  The code is certainly more
readable when you don't have some "global state pickling" kind of
magic.

> Finally, another comment made the point that there wasn't a strong use case given for it. With the data science libraries that have sprung up around Python in the intervening years, I believe there now is one.

As Guido has pointed out, pickling generators would require proper
pickling of the entire frame stack (otherwise generators that use
"global" or "nonlocal" won't unpickle correctly). Ideally we should
also pickle thread locals and contextvars.  Even if Python supported
that, pickling and unpickling generators would be a slow operation, to
the point of being impracticable (and JIT-based Python implementations
would probably use the slowest path for any code that involves frame
pickling).

Instead you should try to encapsulate your state in a dedicated object
that is easy to pickle and unpickle.  Your generators can then work
with that state object instead of storing the state implicitly in
their local variables (this is similar to your workaround #1 but still
allows you to work with generators; just don't use local variables).
The state (or parts of it) can be an immutable collection/object,
which will make it easier to copy/pass it by reference at any point.
While this approach requires more work than just encapsulating the
state in generators, in the long run it should make your code simpler
and more scalable.

Yury


More information about the Python-ideas mailing list