File size limitations
thomas at xs4all.net
Fri Feb 4 12:19:37 CET 2000
On Thu, Feb 03, 2000 at 04:10:07PM -0700, Bjorn Pettersen wrote:
> gmc333 at my-deja.com wrote:
> > I'd like to use Python for some large-scale file processing on Windows
> > NT. There is a possibility that the size of the files I'll be working
> > with will exceed 2^32 bytes.
> > Can Python handle files that large? Random I/O is a specific concern,
> > since an implementation based on fseek/lseek system calls will fail
> > (these calls use 32-bit math, and bad things can happen with seeks that
> > go too far).
> I'm pretty sure Python uses the fseek/lseek etc. functions for file IO. It
> would be trivial to create an extension module with Swig
Actually, The Code does this:
ret = fseeko(f->f_fp, offset, whence);
ret = fseek64(f->f_fp, offset, whence);
ret = fseek(f->f_fp, offset, whence);
(see fileobject.c) which at least suggests python can handle 64bit
filesystems if they exist. Another hint was a posting a few weeks back by
someone who hit a bug because tell() returns a long object instead of an int
object, for very large files (on filesystems that support it.) Wether it
works on Windows NT is something else, but I'd hazard a 'yes' on that,
myself. It's easy to try, though, start the interpreter, open a file, seek()
to a very large number, and write some characters.
Thomas Wouters <thomas at xs4all.net>
Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
More information about the Python-list