building a large file

[Steve Purcell]

Thomas Duterme wrote:
> Hi everyone,
> 
> So I need to build a very large file.  Essentially, I need
> to go through a directory and recursively append the
> contents of each file from that directory to a file.
> 
> Here's what I'm doing right now:
> 
> for x in os.listdir('.'):
> 	os.system('cat '+x+' >> mylargefile)
> 
> Is there any smarter way to do this in python?


If you use the function os.path.walk() you can do everything in 100% Pure
Python (TM):

    >>> import os
    >>> output = open('/tmp/everything','wb')
    >>> def append(out, dirname, names):
    ...     for name in names:
    ...         fullpath = os.path.join(dirname, name)
    ...         if not os.path.isfile(fullpath): continue
    ...         stream = open(fullpath,'rb')
    ...         while 1:
    ...             block = stream.read(512)
    ...             if not block: break  
    ...             out.write(block)
    ...         stream.close()
    ... 
    >>> os.path.walk('projects/python-snippets', append, output)
    >>> output.close()

Notes:
 * 'append' is called for the directory 'projects/python-snippets' and all
   its subdirectories
 * The 'names' argument of 'append' is a list containing the result of
   'os.listdir' for 'dirname'
 * 'append' loops through 'names', skipping names that are not files
 * Files are copied blockwise to the 'out' stream: if files are smallish,
   the 'while' loop could be replaced with "out.write(stream.read())"

See the documentation of os.path.walk for more details:

  http://www.python.org/doc/lib/module-os.path.html

-Steve

-- 
Steve Purcell, Pythangelist
Get testing at http://pyunit.sourceforge.net/
Any opinions expressed herein are my own and not necessarily those of Yahoo