[Tutor] pickling, writing, reading individual lists from a file

[Lie Ryan]

On Sun, 02 Nov 2008 23:20:47 -0800, Dinesh B Vadhia wrote:

> I want to pickle a bunch of lists and write each list separately to a
> fileand then read them back.  Here is my code with the EOF error:
> 

> filename = 'lists.txt'
> fw = open(filename, 'w')
> for l in m:
>         n = pickle.dumps(l, 2) + "\n"
>         fw.write(n)
> fw.close()

That is because a pickle.dumps() produces a string of bytes, and may 
contain "\n". When you try to read the file, you split the file based on 
"\n", since the original pickled items contains "\n", you split the 
pickled data in places where it shouldn't be.

To solve your problem, you have several alternative possibilities:
1. just pickle the whole of them
2. put each pickled object in separate file (alternatively zipping them 
if you have a lot of them)
3. choose a separator that is guaranteed that pickle wouldn't emit
4. escape the separator you choose
5. put the pickled data in a list/dict, then pickle that list/dict

1.
pickle.dump(m, 'lists.p', 2)

2.
for i, l in enumerate(m):
    pickle.dump(l, 'list%s.p' % i, 2)

3. 
#from __future__ import with_statement
# uncomment the line above if python version < 2.6

separator = "something pickle wouldn't emit"
with open('lists.txt', 'w') as fw:
    for l in m:
        n = pickle.dumps(l, 2) + separator
        fw.write(n)

4.
separator = "|"
with open('lists.txt', 'w') as fw:
    for l in m:
        n = escape(pickle.dumps(l, 2)) + separator
        fw.write(n)

5. 
# since the objects you're pickling is already in a list
# this solution is exactly the same as solution #1
# however, in cases where the object you intend to pickle
# is scattered, you should make a dict/list.