tarfile's tar.extractfile() file-like object incompatible with pickle.load()?

Matt Doucleff mattduke at ugcs.caltech.edu
Thu Aug 26 20:43:10 CEST 2004


Hi everyone!  I must be doing something wrong here :)  I have a
tarball that contains a single file whose contents are a pickled
object.  I would like to unpickle the object directly from the tarball
using the file-like object provided by extractfile().  Attempts to do
this result in EOFError.  However if I first extract to a temporary
file, then unpickle from there, it works.  The below code reproduces
the problem (on my machine at least).  I'm running Python 2.3.4,
manually installed on Debian Woody (original python removed).  Thanks!

This sample code creates (and then removes) files in the tmp directory
and in the current working directory.

# demonstrates extractfile/unpickle failure (bug?)

# pickle a dict to a temp file
# create tar file, add temp file to it, close tar file
# open tar file for reading
# obtain file-like object for pickled file using extractfile()
# attempt to unpickle dict from file-like object
# fails with EOFError exception

import tarfile
import pickle
import tempfile
import os

if __name__ == '__main__':
    try:
        hashtopickle = { 'a' : 1, 'b' : 2 }

        # pickle to temp file
        (fd, tmpfilename) = tempfile.mkstemp()
        tmpfile = os.fdopen(fd, 'w')
        pickle.dump(hashtopickle, tmpfile)
        tmpfile.close()
    
        # create tar; add temp file
        tar = tarfile.open('tarpickle.tar', 'w')
        tar.add(tmpfilename, 'pickledhash')
        tar.close()
    
        # remove temp file
        os.remove(tmpfilename)
    
        # open tarfile for reading, get filelike
        tar = tarfile.open('tarpickle.tar', 'r')
        filelike = tar.extractfile('pickledhash')
    
        # fails
        hashcopy = pickle.load(filelike)

    finally:
        # cleanup
        os.remove('tarpickle.tar')



More information about the Python-list mailing list