Combining Boost Serialization with Boost Python
I have a use case involving Boost Serialization and Boost Python. I have a (mostly) C++ library wrapped with Python. The C++ code uses Boost Serialization to periodically create checkpoints from which the user can resume later. Getting the serialization and python libraries to work together didn't present any special problems until I had a case in which the user had a derived Python class (from a C++ base). In searching the web, I have seen people asking about similar cases, but never an answer. See, e.g., http://stackoverflow.com/questions/7289897/boost-serialization-and-boost-pyt... <http://stackoverflow.com/questions/7289897/boost-serialization-and-boost-pyt...> . I have a partial solution to the problem using Boost Serialization and Python Pickle together. I would appreciate any advice on how to improve it. My solution is limited in that the state of the base C++ object is not restored. I also get an extra copy of the C++ object. Since my current use case has no state in the base object, I can live with these limitations. I would however, like to find a full solution. In the files I have attached, I can successfully run three variations on a simple test case. A class of type Base is passed to a class of type Caller. Caller calls the passed class's doit method twice, saves (checkpoints) its state, then calls the doit method again. The resume script restores the state of the Caller and Base classes, then calls the doit method again. If successful, the resumed doit call should be the same as the third doit call from the original script. In testcheckpoint1.py, the Base class is the original (C++) Base. It has no state, so the example is pretty trivial. Its doit method prints "Base::doit" -------------------------------------------------------------------------- % python testcheckpoint1.py Base::doit Base::doit checkpointed here Base::doit % python testresume.py resuming... Base::doit -------------------------------------------------------------------------- In testcheckpoint2.py I pass a derived Python class to Caller. The derived doit method displays "Derived:doit" and a counter: -------------------------------------------------------------------------- % python testcheckpoint2.py Derived::doit count = 1 Derived::doit count = 2 checkpointed here Derived::doit count = 3 % python testresume.py resuming... Derived::doit count = 3 -------------------------------------------------------------------------- In testcheckpoint3,py, I use a derived Python class with a non-trivial constructor, which is used to set the initial value of the counter: -------------------------------------------------------------------------- % python testcheckpoint3.py Derived2::doit count = 5 Derived2::doit count = 6 checkpointed here Derived2::doit count = 7 % python testresume.py resuming... Derived2::doit count = 7 -------------------------------------------------------------------------- As you can see, many things are possible with the code the way it currently exists. I would still like to figure out how to preserve the state of the base class. Any and all advice will be appreciated. --Jim Amundson <http://stackoverflow.com/questions/7289897/boost-serialization-and-boost-pyt...>
I have used this myself for some time. Here is an example: typedef boost::mt19937 rng_t; struct mt_pickle_suite : bp::pickle_suite { static bp::object getstate (const rng_t& rng) { std::ostringstream os; boost::archive::binary_oarchive oa(os); oa << rng; return bp::str (os.str()); } static void setstate(rng_t& rng, bp::object entries) { bp::str s = bp::extract<bp::str> (entries)(); std::string st = bp::extract<std::string> (s)(); std::istringstream is (st); boost::archive::binary_iarchive ia (is); ia >> rng; } };
On 03/14/2012 04:37 PM, James Amundson wrote:
I have a use case involving Boost Serialization and Boost Python. I have a (mostly) C++ library wrapped with Python. The C++ code uses Boost Serialization to periodically create checkpoints from which the user can resume later. Getting the serialization and python libraries to work together didn't present any special problems until I had a case in which the user had a derived Python class (from a C++ base). In searching the web, I have seen people asking about similar cases, but never an answer. See, e.g., http://stackoverflow.com/questions/7289897/boost-serialization-and-boost-pyt... <http://stackoverflow.com/questions/7289897/boost-serialization-and-boost-pyt...> .
It's been a long while since I last tried to pickle Boost.Python objects, but I do recall being a lot happier with the level of control I had when I just implemented my own __reduce__ methods rather than relying on the __getstate__ and __setstate__ defined by enable_pickling(). In many cases, it was most convenient to actually just write __reduce__ in pure-Python and add it to the wrapped classes in the __init__.py file. It would have to delegate to wrapped C++ methods to do the Boost.Serialization calls, of course. Using __reduce__ would allow you to provide a specific callable to reconstruct the Python derived class, which could then ensure it does exactly the right combination of regular unpickling and C++ deserialization. Sorry that's not the review of your "almost there" solution you were looking for, but I do think you might find the problem easier to solve with __reduce__. Jim
participants (3)
-
James Amundson -
Jim Bosch -
Neal Becker