[C++-sig] Serialization/pickle

[Neal Becker]

David Abrahams wrote:

> The following message is a courtesy copy of an article
> that has been posted to gmane.comp.lib.boost.user as well.
> 
> "Drumheller, Michael" <michael.drumheller at boeing.com> writes:
> 
>>>>  From: David Abrahams <dave <at> boost-consulting.com>
>>>>  Subject: Re: Boost.python serialization
>>>>  Newsgroups: gmane.comp.lib.boost.user
>>>>  Date: 2005-11-29 23:28:17 GMT (6 days, 17 hours and 9 minutes ago)
>>>>  
>>>>  "Drumheller, Michael" <michael.drumheller <at> boeing.com> writes:
>>>>  
>>>>  > Thank you for the help.  I have seen that link (the one you
>>>>  > supplied) before, but I posted my question because that link in
>>>>  > particular does not mention the words "Boost.Serialization" or
>>>>  > "Archive" at all.
>>>>  
>>>>  That's because the Boost Serialization library is unrelated.
>>
>> I understand that it is technically unrelated.  I am asking for
>> information about about patterns of actual usage.
> 
> I'm afraid I don't know that.
> 
>>>>  > I suppose what I was really getting at, and was probably not very
>>>>  > clear about (sorry :| ) was whether the specific approach (by
>>>>  > N. Becker) of using a stringstream wrapped with a
>>>>  > boost::archive::binary_oarchive is a standard idiom.
>>>>  
>>>>  "Standard?"
>>
>> OK: "common," "popular," "preferred," "regarded-as-best-practice,"
>> "effective."
> 
> Ditto.
> 
>>>>  > (Basically, I would have thought that "python pickle boost::archive"
>>>>  > would be a million-hit Google query, but it's only about a dozen. I
>>>>  > find that weird.  Do people just not serialize their C++ extensions
>>>>  > very often?)
>>>>  
>>>>  Yes, they do it very often.  There's usually no need to touch
>>>>  Boost.Serialization in order to do so, though.
>>
>> My extensions refer to one another, i.e., they form a significant object
>> hierarchy in and of themselves.  E.g., on the C++ side I might have an
>> instance x of class X, which contains a vector of shared_ptrs p1,...,pn
>> to instances y1,...,yn of another C++ class Y.  At pickle-time there may
>> be Python object z with a member u bound to to x and members w1,...,wm
>> bound to a subset of the y1,...,yn.  Is it even feasible to expect to be
>> able to simple pickle.dump z and have it all work?
> 
> Yes.  Of course you have to do some work in your wrapping code to say
> how X gets pickled.
> 
>> Please keep in mind:  I am just getting started wading into serializing
>> a pretty complicated set of strongly interdependent Python and C++
>> objects and I am just trying to get my bearings.  If this is the wrong
>> forum to be asking these questions, please tell me.
> 
> Well, the C++-sig _might_ be more appropriate:
> http://www.boost.org/more/mailing_lists.htm#cplussig (cross-posted there)
> 
> We (Boost Consulting) are actually planning to do something like this
> with one of our clients, but we haven't gotten started with it yet.  I
> think everything should "just work" as long as you take care not to
> try to serialize the same object both from the C++ side (using
> Boost.Serialization) and from the Python side (using pickle).  Because
> each system implements its own object tracking, you could end up
> representing the same object twice.
> 

I have 2 examples that might interest you.  I haven't looked at this code
for a while, so I don't exactly remember it.

The first is a wrapper for mersenne_twister.  First, I patched
mersenne_twisted.hpp, adding:

  friend class boost::serialization::access;

  template<class Archive>
  inline void save (Archive &ar, const unsigned int) const {
    for (int j = 0; j < state_size; ++j) {
      const UIntType x = compute (j);
      ar << boost::serialization::make_nvp("item", x);
    }
  }

  template<class Archive>
  inline void load (Archive &ar, const unsigned int) {
    for (int j = 0; j < state_size; ++j) {
      ar >> boost::serialization::make_nvp ("item", x[j]);
    }
    i = state_size;
  }

  template<class Archive>
  void serialize(Archive & ar, const unsigned int file_version) {
    boost::serialization::split_member(ar, *this, file_version);
  }

Then, I use this in my wrapper:
typedef boost::mt19937 rng_t;

struct mt_pickle_suite : python::pickle_suite {
    
  static python::object getstate (const rng_t& rng) {
    std::ostringstream os;
    boost::archive::binary_oarchive oa(os);
    oa << rng;
    return python::str (os.str());
  }

  static void
  setstate(rng_t& rng, python::object entries) {
    python::str s = python::extract<python::str> (entries)();
    std::string st = python::extract<std::string> (s)();
    std::istringstream is (st);
    
    boost::archive::binary_iarchive ia (is);
    ia >> rng;
  }
};

The second example is a wrapper for boost ublas vector:

namespace boost {
namespace serialization {

template<class T>
struct implementation_level<std::complex<T> >
{
  typedef mpl::integral_c_tag tag;
  //  typedef mpl::int_<primitive_type> type;
  typedef mpl::int_<object_serializable> type;  
  BOOST_STATIC_CONSTANT(
                        int,
                        value = implementation_level::type::value
                        );
};

template<class T>
struct tracking_level<std::complex<T> >
{
  typedef mpl::integral_c_tag tag;
  typedef mpl::int_<track_never> type;
  BOOST_STATIC_CONSTANT(
                        int,
                        value = tracking_level::type::value
                        );

};

}
}


namespace boost { namespace serialization {
template<class Archive, class T>
inline void serialize (Archive &ar, std::complex<T>& z, const
unsigned int file_version) {
  ar & boost::serialization::make_nvp ("real", real(z));
  ar & boost::serialization::make_nvp ("imag", imag(z));
  //    ar & real(z);
  //    ar & imag(z);
}
}
}

namespace boost {
namespace serialization {

template<class T>
struct implementation_level<ublas::vector<T> >
{
  typedef mpl::integral_c_tag tag;
  //  typedef mpl::int_<primitive_type> type;
  typedef mpl::int_<object_serializable> type;  
  BOOST_STATIC_CONSTANT(
                        int,
                        value = implementation_level::type::value
                        );
};

}
}

template<class Archive, class U>
inline void save (Archive &ar, const ublas::vector<U> &v, const unsigned
int) {
  unsigned int count = v.size();
  ar << BOOST_SERIALIZATION_NVP (count);
  typename ublas::vector<U>::const_iterator it = v.begin();
  while (count-- > 0) {
    ar << boost::serialization::make_nvp ("item", *it++);
  }
}

template<class Archive, class U>
inline void load (Archive &ar, ublas::vector<U> &v, const unsigned int) {
  unsigned int count;
  ar >> BOOST_SERIALIZATION_NVP (count);
  v.resize (count);
  typename ublas::vector<U>::iterator it = v.begin();
  while (count-- > 0) {
    ar >> boost::serialization::make_nvp ("item", *it++);
  }
}


namespace boost { namespace serialization {
template<class Archive, class U>
inline void serialize (Archive &ar, ublas::vector<U>& v, const unsigned int
file_version) {
  boost::serialization::split_free (ar, v, file_version);
}
}
}

namespace python = boost::python;
template <typename MatrixType>
struct vector_pickle_suite : pickle_suite
{
  static
  tuple
  getinitargs(const MatrixType &m)
  {
    return make_tuple(m.size());
  }

  static python::object getstate (const MatrixType& v) {
    std::ostringstream os;
    boost::archive::binary_oarchive oa(os);
    oa << v;
    return python::str (os.str());
  }

  static void
  setstate(MatrixType& v, python::object entries) {
    python::str s = python::extract<python::str> (entries)();
    std::string st = python::extract<std::string> (s)();
    std::istringstream is (st);
    
    boost::archive::binary_iarchive ia (is);
    ia >> v;
  }

};