[Tutor] why is os.path.walk so slow?

Garry Willgoose garry.willgoose at newcastle.edu.au
Wed Nov 4 13:16:17 CET 2009


I need to synchronize the files on my home and office machine and have  
been using someone else's code for this to date but have been  
frustrated by how slow it is in getting the information on files for  
the mounted drive from my office machine so I thought I'd experiment  
with a python facility for this. The code I've experimented with is as  
below

def visitfunc(arg,dirname,names):
   global filelist
   import os.path
   for name in names:
     if not os.path.isdir(dirname+'/'+name):
       fullname=dirname+'/'+name
       filelist.append([fullname,os.path.getmtime(fullname)])
   return()

def check(dir):
   global filelist
   import os.path
   filelist=[]
   os.path.walk(dir,visitfunc,'')
   print filelist
   return()

This is very fast for a directory on my local machine but  
significantly slower on the remote machine. Not surprising but I would  
have expected that the run time for the remote directory would be  
limited by my broadband speed but when I look at upload/download in  
real time it's less than 10% of maximum. Is this just par for the  
course or is there something I can do that better utilizes my  
broadband bandwidth?

====================================================================
Prof Garry Willgoose,
Australian Professorial Fellow in Environmental Engineering,
Director, Centre for Climate Impact Management (C2IM),
School of Engineering, The University of Newcastle,
Callaghan, 2308
Australia.

Centre webpage: www.c2im.org.au

Phone: (International) +61 2 4921 6050 (Tues-Fri AM); +61 2 6545 9574  
(Fri PM-Mon)
FAX: (International) +61 2 4921 6991 (Uni); +61 2 6545 9574 (personal  
and Telluric)
Env. Engg. Secretary: (International) +61 2 4921 6042

email:  garry.willgoose at newcastle.edu.au; g.willgoose at telluricresearch.com
email-for-life: garry.willgoose at alum.mit.edu
personal webpage: www.telluricresearch.com/garry
====================================================================
"Do not go where the path may lead, go instead where there is no path  
and leave a trail"
                           Ralph Waldo Emerson
====================================================================







More information about the Tutor mailing list