building a large file
Steve Purcell
stephen_purcell at yahoo.com
Sat Apr 14 05:25:17 EDT 2001
Thomas Duterme wrote:
> Hi everyone,
>
> So I need to build a very large file. Essentially, I need
> to go through a directory and recursively append the
> contents of each file from that directory to a file.
>
> Here's what I'm doing right now:
>
> for x in os.listdir('.'):
> os.system('cat '+x+' >> mylargefile)
>
> Is there any smarter way to do this in python?
If you use the function os.path.walk() you can do everything in 100% Pure
Python (TM):
>>> import os
>>> output = open('/tmp/everything','wb')
>>> def append(out, dirname, names):
... for name in names:
... fullpath = os.path.join(dirname, name)
... if not os.path.isfile(fullpath): continue
... stream = open(fullpath,'rb')
... while 1:
... block = stream.read(512)
... if not block: break
... out.write(block)
... stream.close()
...
>>> os.path.walk('projects/python-snippets', append, output)
>>> output.close()
Notes:
* 'append' is called for the directory 'projects/python-snippets' and all
its subdirectories
* The 'names' argument of 'append' is a list containing the result of
'os.listdir' for 'dirname'
* 'append' loops through 'names', skipping names that are not files
* Files are copied blockwise to the 'out' stream: if files are smallish,
the 'while' loop could be replaced with "out.write(stream.read())"
See the documentation of os.path.walk for more details:
http://www.python.org/doc/lib/module-os.path.html
-Steve
--
Steve Purcell, Pythangelist
Get testing at http://pyunit.sourceforge.net/
Any opinions expressed herein are my own and not necessarily those of Yahoo
More information about the Python-list
mailing list