[python-win32] Sparse files and Pickle

[Tim Roberts]

On Sun, 3 Apr 2005 12:37:40 +0200, Graeme Glass <graemeglass at gmail.com> 
wrote:

>Could anyone tell me if it is possible to create a  sparse file on
>win32 platform using python, and if so how?
>  
>

Sparse files are only supported on Windows 2000 and more, on NTFS 
partitions.  You will not be able to use the Python file I/O operations 
to do this.  You have to send an ioctl (FSCTL_SET_SPARSE) to the file 
after it is opened, then you have to open a memory growable mapping of 
the file, and use that memory mapping to do your writes.

CreateFile, DeviceIoControl, and the memory mapping methods are all 
provided in the Win32 extensions, but it makes me wonder if it wouldn't 
be easier to write a C extension for this.

>And just out of interest, what is the practical limit for pickling an object?
>I pickled a 89MB binary read from a disk file, and although it worked,
>hogged my PC for about 8minutes making it unusable.
>  
>

I guess this depends on your threshhold of pain, and is likely to vary 
from PC to PC.  I view pickling as a way to encode small objects for 
later resurrection.  If I needed to refer to an 89MB disk file in an 
object, I would replace the data with the file name before pickling.  I 
thought pickle recognized a magic method name so the object could "help" 
put itself into a picklable state, but I don't see it now.

-- 
- Tim Roberts, timr at probo.com
  Providenza & Boekelheide, Inc.