[Tutor] Cannot cPickle.load()

[Tim Peters]

[Brian van den Broek]
> Tim Peters said unto the world upon 2004-11-30 20:54:
>
> <SNIP>
>
>
>> Change
>>     "r"
>> to
>>     "rb"
>> and try again.
>>
>> You're on Windows, pickles are binary files, and binary files must
>> always be opened in binary mode on Windows.  When you wrote
>> the data to the file to begin with, I hope you opened the file in
>> binary mode then too.  Else the data in the file is corrupt now.

> Tim's post confuses me. (I've cc'ed you Tim as somehow I doubt
> you are a regular reader of Tutor -- apologies if that annoys.)

No, I read Tutor regularly, I just don't have time to reply regularly <wink>.

> I'm on Windows with Python 2.3.4 (waiting for extension updates
> to use 2.4). When I run:
>
> <code>
> import pickle
>
> a_dict = {1:2, 3:4, 5:6}
> print a_dict
>
> pic_file = file('c:/testpickle.pic', 'w')
> pickle.dump(a_dict, pic_file)
> pic_file.close()
>
> a_dict = {}
> print a_dict
> 
> pic_file = file('c:/testpickle.pic', 'r')
> a_dict = pickle.load(pic_file)
> print a_dict
> </code>
> 
> I get the output:
> 
....
> {1: 2, 3: 4, 5: 6}
> {}
> {1: 2, 3: 4, 5: 6}
> 
> Which makes it look to me like pickle is working just fine without
> using binary mode. I tried the same thing with cPickle and also
> made sure to close IDLE and try reading the pickled data in a
> command prompt Python.
>
> In all cases, I recovered the pickled data without difficulty.
>
> So, what have I misunderstood?

It can work by accident, just as any binary file *may* not contain
data that causes it to fail when opened in text mode on Windows.  In
particular, because you didn't specify a pickle protocol in the above,
it defaulted to protocol 0, which uses only printable characters. 
Note that Hugo (the OP) said he was using protocol 2, which does not
restrict itself to printable characters (and neither does protocol 1).

Even if your pickle happened to appear to work, there's no guarantee
it will work if you try to load the same pickle on a non-Windows box: 
Windows text mode transforms \n to \r\n on write, and does the reverse
transformation on read.  Linux does neither, so you're not necessarily
safe cross-platform even sticking to protocol 0 pickles.

The bottom line:  pickles are binary data, and if you want to avoid
disaster, you must open pickle files in binary mode on Windows, for
both writing and reading.  You *should* also do that on non-Windows
platforms, although most non-Windows platforms don't make any
distinction between binary and text modes.  If you do 'rb' and 'wb'
anyway, your program will be portable across platforms.

BTW2, Hugo replied in private, saying that switching to binary mode
open() fixed his problem.  Of course this was obvious from what he
said in his first message <wink>.