[IronPython] Portable use of pickle.dumps()

Robert Smallshire robert at smallshire.org.uk
Fri May 29 19:20:19 CEST 2009


Hi Michael,

[snip]

> > My opening paragraph may be ambiguously worded - by 
> 'interoperability' 
> > I didn't mean the ability to run the same code unchanged on CPython 
> > and IronPython (I have to change the code anyway to use a different 
> > database
> > adapter) - I meant interoperability between pickles persisted into a
> > database from both IronPython and CPython.
> >   
> 
> So are you telling the database that it is binary data or text?

Binary - the column is a BLOB, but SQLite is dynamically typed, and indeed
weakly typed, and so seems to store pretty much whatever you give it!

> 
> Is the question how do I go from a pickle string in 
> IronPython to a byte 
> array that I can pass to the database adaptor without going 
> through an 
> explicit encode (which will transform the data)?

Yes.

> 
> (One technique would be to explicitly use pickle protocol 0 which is 
> less efficient but only creates ascii characters - this is 
> actually the 
> default. Another alternative would be to use JSON or YAML instead of 
> pickle.)

I'm pickling data in the hundreds of megabytes to multi-gigabyte range, so
the binary pickling format is essential for performance and compactness
reasons. This much we know from our experience with the CPython end of
things.

> 
> Here is an example of getting a byte array from a binary pickle in 
> IronPython:
> 
>  >>> import pickle
>  >>> class A(object):
> ...  b = 'hello'
> ...  c = (None, 'fish', 7.2, 7j)
> ...  a = {1: 2}
> ...
>  >>> p = pickle.dumps(A(), protocol=2)
>  >>> p
> u'\x80\x02c__main__\nA\nq\x00)\x81q\x01}q\x02b.'
>  >>> from System import Array, Byte
>  >>> a = Array[Byte](tuple(Byte(ord(c)) for c in p))
>  >>> a
> Array[Byte]((<System.Byte object at 0x0000000000000033 [128]>, 
> <System.Byte obje...
> 
> I hope this is at least slightly helpful. :-)

Its definitely helpful. The Array[Byte] syntax is new to me, but now I've
looked it up, something along these lines looks to what I need. I'll also be
investigating how the 'bytes' type works in IronPython since it seems to be
a distict type from 'str' and close to what I need.  The database is
actually giving me Array[Byte] back in IronPython when I extract data that's
been inserted by CPython. In fact 'bytes' appears to be constructable from
IList[Byte] in IronPython.

Thanks again for you help,

Rob

P.S. I've just ordered IronPython in Action. :-)




More information about the Ironpython-users mailing list