[Tutor] Cannot cPickle.load()

[Brian van den Broek]

Tim Peters said unto the world upon 2004-12-01 14:21:
> [Brian van den Broek]
> 
>>Tim Peters said unto the world upon 2004-11-30 20:54:
>>
>><SNIP>
>>
>>
>>>Change
>>>    "r"
>>>to
>>>    "rb"
>>>and try again.
>>>
>>>You're on Windows, pickles are binary files, and binary files must
>>>always be opened in binary mode on Windows.  When you wrote
>>>the data to the file to begin with, I hope you opened the file in
>>>binary mode then too.  Else the data in the file is corrupt now.
> 
> 
>>Tim's post confuses me. (I've cc'ed you Tim as somehow I doubt
>>you are a regular reader of Tutor -- apologies if that annoys.)
> 
> 
> No, I read Tutor regularly, I just don't have time to reply regularly <wink>.

Noted for future cases. It is nice (if a bit intimidating ;-) to know
that tutor is read by core developers, too.


>>I'm on Windows with Python 2.3.4 (waiting for extension updates
>>to use 2.4). When I run:
>>

<SNIPped code using pickle dump and load without binary mode flags>

>>
>>I get the output:

<SNIPped output showing said pickling preserved the data pickled>

>>
>>Which makes it look to me like pickle is working just fine without
>>using binary mode. I tried the same thing with cPickle and also
>>made sure to close IDLE and try reading the pickled data in a
>>command prompt Python.
>>
>>In all cases, I recovered the pickled data without difficulty.
>>
>>So, what have I misunderstood?
> 
> 
> It can work by accident, just as any binary file *may* not contain
> data that causes it to fail when opened in text mode on Windows.  In
> particular, because you didn't specify a pickle protocol in the above,
> it defaulted to protocol 0, which uses only printable characters. 
> Note that Hugo (the OP) said he was using protocol 2, which does not
> restrict itself to printable characters (and neither does protocol 1).
> 
> Even if your pickle happened to appear to work, there's no guarantee
> it will work if you try to load the same pickle on a non-Windows box: 
> Windows text mode transforms \n to \r\n on write, and does the reverse
> transformation on read.  Linux does neither, so you're not necessarily
> safe cross-platform even sticking to protocol 0 pickles.
> 
> The bottom line:  pickles are binary data, and if you want to avoid
> disaster, you must open pickle files in binary mode on Windows, for
> both writing and reading.  You *should* also do that on non-Windows
> platforms, although most non-Windows platforms don't make any
> distinction between binary and text modes.  If you do 'rb' and 'wb'
> anyway, your program will be portable across platforms.

OK, thanks. I just did my first pickling about a week ago, so I'm very
glad to have been straightened out before I had too much 'unhappy' code
to fix.

Best to all,

Brian vdB