Calling a C program from a Python Script

Grant Edwards grante at visi.com
Thu Dec 9 20:26:49 CET 2004


On 2004-12-09, Brad Tilley <bradtilley at gmail.com> wrote:
> Steven Bethard wrote:
>>>> for root, files, dirs in os.walk(path)
>>>>      for f in files:
>>>>          try:
>>>>              x = file(f, 'rb')
>>>>              data = x.read()
>>>>              x.close()
>> 
>> 
>> Remember that CPython is implemented in C, and so all the builtin types 
>> (including file) basically execute C code directly.  My experience with 
>> Python file objects is that they are quite fast when you're doing simple 
>> things like the example above.
>
> I'm dealing with a terabyte of files. Perhaps I should have mentioned that.

And you think you're going to read the entire file consisting
of terabytes of data into memory using either C or Python?
[That's the example you gave.]

Sounds like maybe you need to mmap() the files?

Or at least tell us what you're trying to do so we can make
more intelligent suggestions.

-- 
Grant Edwards                   grante             Yow!  I invented skydiving
                                  at               in 1989!
                               visi.com            



More information about the Python-list mailing list