[Python-ideas] Copy (and/or pickle) generators

[Micheál Keane]

Add a function to generator objects to copy the entire state of it:

Proposed example code:

game1 = complicated_game_type_thing()

# Progress the game to the first decision point
choices = game1.send(None)

# Choose something
response = get_a_response(choices)

# Copy the game generator
game2 = game1.copy()

# send the same response to each game
x = game1.send(response)
y = game2.send(response)

# verify the new set of choices is the same
assert x == y


History:

I found this stackoverflow Q&A
<https://stackoverflow.com/questions/7180212/why-cant-generators-be-pickled>
which
among other things linked to an in-depth explanation of why generators
could not be pickled
<http://peadrop.com/blog/2009/12/29/why-you-cannot-pickle-generators/> and
this enhancement request for 2.6 <https://bugs.python.org/issue1092962> on
the bugtracker. All the reasons given there are perfectly valid.... but
they were also given nearly 10 years ago. It may be time to revisit the
issue.

I couldn't turn up any previous threads here related to this so I'm
throwing it out for discussion.


Use case:

My work involves Monte Carlo Tree Searches of games, eventually in
combination with tensorflow. MCTS involves repeatedly copying the state of
a simulation to explore the potential outcomes of various choices in depth.

If you're doing a game like Chess or Go, a game state is dead simple to
summarize - you have a list of board positions with which pieces they have
and whose turn it is.

If you're doing complex games that don't have an easily summarized state at
any given moment, you start running into problems. Think something along
the lines of Magic the Gathering with complex turn sequences between
players and effect resolutions being done in certain orders that are
dependent on choices made by players, etc.

Generators are an ideal way to run these types of simulations but the
inability to copy the state of a generator makes it impossible to do this
in MCTS.

As Python is being increasingly used for data science, this use case will
be increasingly common. Being able to copy generators will save a lot of
work.

Keep in mind, I don't necessarily propose that generators should be fully
picklable; there are obviously a number of concerns and problems there.
Just being able to duplicate the generator's state within the interpreter
would be enough for my use case.


Workarounds:

The obvious choice is to refactor the simulation as an iterator that stores
each state as something that's easily copied/pickled. It's probably
possible but it'll require a lot of thought and code for each type of
simulation.

There's a Python2 package from 2009 called generator_tools
<https://pypi.org/project/generator_tools/> that purports to do this. I
haven't tried it yet to see if it still works in 2.x and it appears beyond
my skill level to port to 3.x.

PyPy & Stackless Python apparently support this within certain limits?


Thoughts?


Washington, DC  USA
ffaristocrat at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180619/8e11a069/attachment.html>