Calling a C program from a Python Script
grante at visi.com
Thu Dec 9 20:26:49 CET 2004
On 2004-12-09, Brad Tilley <bradtilley at gmail.com> wrote:
> Steven Bethard wrote:
>>>> for root, files, dirs in os.walk(path)
>>>> for f in files:
>>>> x = file(f, 'rb')
>>>> data = x.read()
>> Remember that CPython is implemented in C, and so all the builtin types
>> (including file) basically execute C code directly. My experience with
>> Python file objects is that they are quite fast when you're doing simple
>> things like the example above.
> I'm dealing with a terabyte of files. Perhaps I should have mentioned that.
And you think you're going to read the entire file consisting
of terabytes of data into memory using either C or Python?
[That's the example you gave.]
Sounds like maybe you need to mmap() the files?
Or at least tell us what you're trying to do so we can make
more intelligent suggestions.
Grant Edwards grante Yow! I invented skydiving
at in 1989!
More information about the Python-list