[Python-ideas] Copy (and/or pickle) generators

Guido van Rossum guido at python.org
Tue Jun 19 19:25:40 EDT 2018

The state of a generator is not much more that a single Python stack frame
plus an integer indicating where in the bytecode the resume point is. But
copying/pickling a stack frame is complicated -- it's not just all the
locals but also the try/except stack and the expression evaluation stack.
Have a look here:
https://github.com/python/cpython/blob/master/Include/frameobject.h. I'm
not sure that I want to sign up for making all that stuff copyable
(pickling will be an even harder challenge). But perhaps you (and/or
another fearless hacker) are interested in trying?

Or were you just trying to see if the core dev team has spare cycles to
implement this for you?


On Tue, Jun 19, 2018 at 3:56 PM Micheál Keane <ffaristocrat at gmail.com>

> Add a function to generator objects to copy the entire state of it:
> Proposed example code:
> game1 = complicated_game_type_thing()
> # Progress the game to the first decision point
> choices = game1.send(None)
> # Choose something
> response = get_a_response(choices)
> # Copy the game generator
> game2 = game1.copy()
> # send the same response to each game
> x = game1.send(response)
> y = game2.send(response)
> # verify the new set of choices is the same
> assert x == y
> History:
> I found this stackoverflow Q&A
> <https://stackoverflow.com/questions/7180212/why-cant-generators-be-pickled> which
> among other things linked to an in-depth explanation of why generators
> could not be pickled
> <http://peadrop.com/blog/2009/12/29/why-you-cannot-pickle-generators/> and
> this enhancement request for 2.6 <https://bugs.python.org/issue1092962> on
> the bugtracker. All the reasons given there are perfectly valid.... but
> they were also given nearly 10 years ago. It may be time to revisit the
> issue.
> I couldn't turn up any previous threads here related to this so I'm
> throwing it out for discussion.
> Use case:
> My work involves Monte Carlo Tree Searches of games, eventually in
> combination with tensorflow. MCTS involves repeatedly copying the state of
> a simulation to explore the potential outcomes of various choices in depth.
> If you're doing a game like Chess or Go, a game state is dead simple to
> summarize - you have a list of board positions with which pieces they have
> and whose turn it is.
> If you're doing complex games that don't have an easily summarized state
> at any given moment, you start running into problems. Think something
> along the lines of Magic the Gathering with complex turn sequences between
> players and effect resolutions being done in certain orders that are
> dependent on choices made by players, etc.
> Generators are an ideal way to run these types of simulations but the
> inability to copy the state of a generator makes it impossible to do this
> in MCTS.
> As Python is being increasingly used for data science, this use case will
> be increasingly common. Being able to copy generators will save a lot of
> work.
> Keep in mind, I don't necessarily propose that generators should be fully
> picklable; there are obviously a number of concerns and problems there.
> Just being able to duplicate the generator's state within the interpreter
> would be enough for my use case.
> Workarounds:
> The obvious choice is to refactor the simulation as an iterator that
> stores each state as something that's easily copied/pickled. It's probably
> possible but it'll require a lot of thought and code for each type of
> simulation.
> There's a Python2 package from 2009 called generator_tools
> <https://pypi.org/project/generator_tools/> that purports to do this. I
> haven't tried it yet to see if it still works in 2.x and it appears beyond
> my skill level to port to 3.x.
> PyPy & Stackless Python apparently support this within certain limits?
> Thoughts?
> Washington, DC  USA
> ffaristocrat at gmail.com
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180619/eeb850ac/attachment-0001.html>

More information about the Python-ideas mailing list