Load a list subset with pickle?
Robert Kern
robert.kern at gmail.com
Thu Oct 15 12:56:31 EDT 2009
On 2009-10-15 11:05 AM, Peng Yu wrote:
> On Tue, Oct 13, 2009 at 1:23 PM, Robert Kern<robert.kern at gmail.com> wrote:
>> On 2009-10-13 13:00 PM, Peng Yu wrote:
>>>
>>> I use pickle to dump a long list. But when I load it, I only want to
>>> load the first a few elements in the list. I am wondering if there is
>>> a easy way to do so? Thank you!
>>
>> Not by pickling the list. However, you can concatenate pickles, so you could
>> just pickle each item from the list in order to the same file and only
>> unpickle the first few.
>>
>> In [1]: import cPickle
>>
>> In [2]: from cStringIO import StringIO
>>
>> In [3]: very_long_list = range(10)
>>
>> In [4]: f = StringIO()
>>
>> In [5]: for item in very_long_list:
>> ...: cPickle.dump(item, f)
>> ...:
>> ...:
>>
>> In [6]: f.seek(0,0)
>>
>> In [7]: cPickle.load(f)
>> Out[7]: 0
>>
>> In [8]: cPickle.load(f)
>> Out[8]: 1
>>
>> In [9]: cPickle.load(f)
>> Out[9]: 2
>>
>> In [10]: cPickle.load(f)
>> Out[10]: 3
>
> How do I determine if I have loaded all the elements? I use the
> following code. I'm wondering if there is any better solution than
> this.
>
>
> ###############
> import pickle
>
> alist = [1, 2.0, 3, 4+6j]
>
> output=open('serialize_list.output/serialize_list.pkl', 'wb')
> for e in alist:
> pickle.dump(e, output)
> output.close()
>
> input=open('serialize_list.output/serialize_list.pkl', 'rb')
>
> try:
> while 1:
> e = pickle.load(input)
> print e
> except EOFError:
> pass
You could write out an integer with the number of expected elements at the very
beginning.
In [1]: import cPickle
In [2]: alist = [1, 2.0, 3, 4+6j]
In [3]: output = open('foo.pkl', 'wb')
In [4]: cPickle.dump(len(alist), output)
In [5]: for item in alist:
...: cPickle.dump(item, output)
...:
...:
In [6]: output.close()
In [7]: input = open('foo.pkl', 'rb')
In [8]: n = cPickle.load(input)
In [9]: n
Out[9]: 4
In [10]: for i in range(n):
....: print cPickle.load(input)
....:
....:
1
2.0
3
(4+6j)
In [11]: assert input.read(1) == ''
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
More information about the Python-list
mailing list