[ python-Bugs-1102649 ] pickle files should be opened in binary mode

[SourceForge.net]

Bugs item #1102649, was opened at 2005-01-14 22:58
Message generated for change (Settings changed) made by birkenfeld
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1102649&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Documentation
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: John Machin (sjmachin)
>Assigned to: Reinhold Birkenfeld (birkenfeld)
Summary: pickle files should be opened in binary mode

Initial Comment:
pickle (and cPickle):

At _each_ mention of the pickle file, the docs should say 
that it should be opened with 'wb' or 'rb' mode as 
appropriate, so that a pickle written on one OS can be 
read reliably on another.

The example code at the end of the section should be 
updated to use the 'b' flag.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2005-02-24 21:08

Message:
Logged In: YES 
user_id=21627

sjmachin, it seems pretty clear what to do now. Would you
volunteer to come up with a patch yourself?

----------------------------------------------------------------------

Comment By: John Machin (sjmachin)
Date: 2005-01-19 14:51

Message:
Logged In: YES 
user_id=480138

Re Fred's question:
Refer to thread starting at 
http://mail.python.org/pipermail/python-dev/2003-
February/033362.html

Looks like the story is like this:

For pickle mode 1 or higher, always use binary mode for 
reading/writing.

For pickle mode 0, either (a) read/write in text mode and if 
moving to another OS, do so in text mode i.e. convert the line 
endings where necessary or (b) as for pickle mode 1+, stick 
with binary throughout.

Also should add a generalisation of Tim's comment re 
NotePad, e.g. something like """A file written with pickle mode 
0 and file mode 'wb' will contain lone linefeeds as line 
terminators. This will cause it to "look funny" when viewed on 
Windows or MacOS as a text file by editors like Notepad that 
do not understand this format."""

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2005-01-19 14:45

Message:
Logged In: YES 
user_id=31435

Yes, binary mode should always be used, regardless of 
protocol.  Else pickles aren't portable across boxes (in 
particular, Unix can't read a protocol 0 pickle produced on 
Windows if the latter was written to a text-mode file).  "text 
mode" was a horrible name for protocol 0.

----------------------------------------------------------------------

Comment By: Fred L. Drake, Jr. (fdrake)
Date: 2005-01-19 06:09

Message:
Logged In: YES 
user_id=3066

In response to irmin's comment:

freopen() is only an option for real file objects; pickles
are often stored or read from other sources.  These other
sources are usually binary to begin with, fortunately,
though this issue probably deserves some real coverage in
the documentation either way.


----------------------------------------------------------------------

Comment By: Fred L. Drake, Jr. (fdrake)
Date: 2005-01-19 06:06

Message:
Logged In: YES 
user_id=3066

Is this true in all cases?  Shouldn't files containing text
pickles (protocol 0) be opened in text mode?  (A problem,
given that all protocols should be readable without prior
knowledge of the protocol used to write the pickle.)

----------------------------------------------------------------------

Comment By: Irmen de Jong (irmen)
Date: 2005-01-16 16:07

Message:
Logged In: YES 
user_id=129426

Can't the pickle code just freopen() the file itself, using
binary mode?

Or is this against Python's rule "explicit is better than
implicit"

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1102649&group_id=5470