[IronPython] Compatibility of cPython PickledObjects w/IronPython

Fri Sep 7 23:31:43 CEST 2007

This seems to be related to file I/O issues rather than Pickling - particularly the fact that the I/O is going through std-out / std-in.  For example if I do:

C:\Product\Released\IronPython-1.1 > cpy
Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import pickle
>>>
>>> l = [5, 12.45, True, 'testString']
>>>
>>> pickledList = pickle.dumps(l, 1)
>>> pickledList
']q\x00(K\x05G@(\xe6fffffI01\nU\ntestStringq\x01e.'
>>> ^Z

C:\Product\Released\IronPython-1.1 > .\ipy.exe
IronPython 1.1 (1.1) on .NET 2.0.50727.1408
Copyright (c) Microsoft Corporation. All rights reserved.
>>> import pickle
>>> pickle.loads(']q\x00(K\x05G@(\xe6fffffI01\nU\ntestStringq\x01e.')
[5, 12.45, True, 'testString']

I get the correct results.  But I see the same odd behavior you do w/ the full repro.

If I had a MessageBox.Show(repr(pList)) to child_pickle I get:

']q\x00(K\x05G@(\xb5fffffI01\nU\ntestStringq\x01e.'

Vs:

']q\x00(K\x05G@(\xe6fffffI01\nU\ntestStringq\x01e.'

As you can see the \xe6 byte has been replaced by the \xb5 byte - likely due to encoding issues (we open std-in w/ the default encoding of your console window).

Is there any reason you need to do this through std out?  I'm not actually entirely certain how we'd fix this - ultimately we need some encodings to be applied to the console input, maybe we could make a binary encoding.  But that might break other scenarios where users are expecting to be able to input localized data.  There does seem to be an interesting difference in the console behavior that deserves documentation though.

-----Original Message-----
From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of nico rauhut
Sent: Thursday, September 06, 2007 10:19 AM
To: 'Discussion of IronPython'
Subject: Re: [IronPython] Compatibility of cPython PickledObjects w/IronPython

I don't make any modifications to the list, so it should be the same:
I pickle the list in cPython and pass the picklestring through a pipe to
IronPython. In IronPython I unpickle the list, pickle it again and send it
through a pipe back to the cPython-Process. In cPython the list is unpickled
one more time. All states of the list are logged into logfiles.

Checking the logfiles I saw that the float-value 12.45 changed in IronPython
to 12.3542968750. Also the string looks a little bit strange after the list
was unpickled in cPython. I use Python 2.4.4 and IronPython 1.1 on WinXP.
Below the code I used:

cPython:

import os
import pickle

l = [5, 12.45, True, 'testString']
log = open("log_python.pkl", "wb")
cin, cout = os.popen2("D:\Dev\IronPython-1.1\ipy.exe child_pickle.py", "b")

pickledList = pickle.dumps(l, 1)
log.write(pickledList)
cin.write(pickledList)
cin.close()

pList = cout.read()
log.write("\n")
log.write(pList)

myList = pickle.loads(pList)
log.write("\n")
for item in myList:
        log.write(str(item) + "\n")
cout.close()
log.close()

IronPython:

import pickle
log = open("log_ipy.pkl", "wb")

pList = sys.stdin.read()
log.write(pList)

myList = pickle.loads(pList)
log.write("\n\n")
for item in myList:
        log.write(str(item) + "\n")

pResList = pickle.dumps(myList, 1)
log.write("\n\n")
log.write(pResList)
sys.stdout.write(pResList)
log.close()

logfile log_python (without picklestrings):

5
12.45
1

testStringq

logfile log_ipy (without picklestrings):

5
12.3542968750
True
testString

-----Ursprüngliche Nachricht-----
Von: users-bounces at lists.ironpython.com
[mailto:users-bounces at lists.ironpython.com] Im Auftrag von Dino Viehland
Gesendet: Mittwoch, 5. September 2007 22:32
An: Discussion of IronPython
Betreff: Re: [IronPython] Compatibility of cPython PickledObjects
w/IronPython

I don't believe we are aware of these issues - there are a couple of bugs
open against pickling but they seem to be around types we apparently don't
support (xrange, re) and features we don't support in cPickle (persistent
load support) and .NET interop (System.DateTime).

Is there anything other than floats w/ protocol 1?  Are the lists all the
same in each version?  W/ a little more detail we can open a bug and look at
getting the issues fixed.

-----Original Message-----
From: users-bounces at lists.ironpython.com
[mailto:users-bounces at lists.ironpython.com] On Behalf Of nico rauhut
Sent: Wednesday, September 05, 2007 10:18 AM
To: 'Discussion of IronPython'
Subject: Re: [IronPython] Compatibility of cPython Pickled Objects
w/IronPython

Hello,

I have run a few tests passing values from cPython to IronPython (and back)
using a pickled list (in order to integrate .NET-Components in Zope).
Unfortunately the results were not as good as expected.

The only protocol that seemed to be useful is version 0 of the
pickle-module. Using this protocol I was able to pass a list including some
floats, boolean and integer to IronPython, use these values in a .NET-class
and return the result-values to cPython. Only strings made some difficulties
(insecure string pickle).

Protocol-version 1 doesn't work properly. Especially float-values make some
trouble. When I unpickle my list in IronPython these values seemed to be
adulterated (...hmm don't know if this is the right word :)

Protocol-version 2 seemed to be entire useless as I wasn't able to unpickle
my list in cPython.

I tested the cPickle-module, too. But as expected the results were even
worse.

Are there any plans to improve the pickle-compatibility in future releases
of IronPython? Has somebody else made some other experiences with
Pickle/cPickle?

Kindly regards,
Nico Rauhut

-----Ursprüngliche Nachricht-----
Von: users-bounces at lists.ironpython.com
[mailto:users-bounces at lists.ironpython.com] Im Auftrag von Michael Foord
Gesendet: Sonntag, 15. Juli 2007 20:33
An: Discussion of IronPython
Betreff: Re: [IronPython] Compatibility of cPython Pickled Objects
w/IronPython

M. David Peterson wrote:
> As per the subject line, is there any reason why passing pickled
> objects between cPython and IronPython wouldn't just work? Of course I
> could just try and find out, but I hesitate to jump into unchartered
> territory as (and I assume I am not alone in this) I have a tendency
> to become infatuated with forcing things into submission when they
> don't "just work", eating away an entire dev day or two in the process.
>
> Anyone care to help a brother save a couple dev days and a few strands
> of hair? :D
>
> Thanks in advance!

I haven't tried it, but they *should* be compatible. Pickle uses its own
stack based language to write out objects - so for pure Python it should
work fine.

As pickle is a pure Python module which works (I'm pretty sure I tried
it back in the past !?!) with IronPython, the semantics should be unchanged.

Michael Foord
http://www.voidspace.org.uk/ironpython/index.shtml

>
> --
> /M:D
>
> M. David Peterson
> http://mdavid.name <http://mdavid.name> |
> http://www.oreillynet.com/pub/au/2354 | http://dev.aol.com/blog/3155
> ------------------------------------------------------------------------
>
> _______________________________________________
> users mailing list
> users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>

_______________________________________________
users mailing list
users at lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

_______________________________________________
Users mailing list
Users at lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
_______________________________________________
Users mailing list
Users at lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com

_______________________________________________
Users mailing list
Users at lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com