[Python-Dev] pickling of large arrays

[Ralf W. Grosse-Kunstleve]

--- Tim Peters <tim.one@comcast.net> wrote:
> So it's unlikely you're going to get a change in what the proto 0 APPEND
> (which calls append()) and proto 1 APPENDS (which calls extend()) opcodes
> do.  Adding brand new opcodes is still possible, but I doubt it's possible
> for Guido or me to do the work.

Sounds like a chance for plan B: a thin wrapper around a Python string.
Called "pickle_string" in the revised prototype below. Ruling out the
possibility that users want to create C++ arrays of Python
pickle_strings this will work for all types. Creating a trivial type
like pickle_string is a snag with Boost.Python. But I am guessing that
the Numarray people must hate it, unless they have another simple
work-around.

I believe the brand new opcode is the better long-term solution for the
community. I'd be willing to invest a little time if others are also
interested and willing to help.

Ralf


import pickle

class pickle_string(object):

  def __init__(self, size):
    self.size = size # here we allocate the string
    self.string = "" # this will be a reference to the Python string object
    # the C++ implementation will write directly into the string contained here

  def __getstate__(self):
    return self.string # return a reference

  def __setstate__(self, state):
    self.string = state # copy a reference
    
class int_array(object):

  def __init__(self, elems):
    self.elems = list(elems)

  def __reduce__(self):
    return (int_array_factory,
            (len(self.elems),),
            None,
            int_array_iter(self.elems))

  def append(self, value):
    if (type(value) == type(pickle_string(0))):
      values = [int(x) for x in value.string.split(",")]
      self.elems.extend(values)
    else:
      self.elems.append(value)

class int_array_iter(object):

  def __init__(self, elems, buf_size=4):
    self.elems = elems
    self.buf_size = 4
    self.i = 0

  def __iter__(self):
    return self

  def next(self):
    if (self.i >= len(self.elems)): raise StopIteration
    result = pickle_string(123)
    for i in xrange(self.buf_size):
      result.string += str(self.elems[self.i]) + ","
      self.i += 1
      if (self.i == len(self.elems)): break
    result.string = result.string[:-1]
    return result

def int_array_factory(size):
  print "reserve:", size
  return int_array([])

f = int_array(range(11))
f.append(13)
print "f.elems:", f.elems
s = pickle.dumps(f)
print s
g = pickle.loads(s)
print "g.elems:", g.elems


__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - forms, calculators, tips, more
http://taxes.yahoo.com/