[Tutor] pickling, writing, reading individual lists from a file

Lie Ryan lie.1296 at gmail.com
Mon Nov 3 12:15:28 CET 2008

On Sun, 02 Nov 2008 23:20:47 -0800, Dinesh B Vadhia wrote:

> I want to pickle a bunch of lists and write each list separately to a
> fileand then read them back.  Here is my code with the EOF error:

> filename = 'lists.txt'
> fw = open(filename, 'w')
> for l in m:
>         n = pickle.dumps(l, 2) + "\n"
>         fw.write(n)
> fw.close()

That is because a pickle.dumps() produces a string of bytes, and may 
contain "\n". When you try to read the file, you split the file based on 
"\n", since the original pickled items contains "\n", you split the 
pickled data in places where it shouldn't be.

To solve your problem, you have several alternative possibilities:
1. just pickle the whole of them
2. put each pickled object in separate file (alternatively zipping them 
if you have a lot of them)
3. choose a separator that is guaranteed that pickle wouldn't emit
4. escape the separator you choose
5. put the pickled data in a list/dict, then pickle that list/dict

pickle.dump(m, 'lists.p', 2)

for i, l in enumerate(m):
    pickle.dump(l, 'list%s.p' % i, 2)

#from __future__ import with_statement
# uncomment the line above if python version < 2.6

separator = "something pickle wouldn't emit"
with open('lists.txt', 'w') as fw:
    for l in m:
        n = pickle.dumps(l, 2) + separator

separator = "|"
with open('lists.txt', 'w') as fw:
    for l in m:
        n = escape(pickle.dumps(l, 2)) + separator

# since the objects you're pickling is already in a list
# this solution is exactly the same as solution #1
# however, in cases where the object you intend to pickle
# is scattered, you should make a dict/list.

More information about the Tutor mailing list