Fast way of extracting files from various folders

[Peter Otten]

subhabrata.banerji at gmail.com wrote:

> I have several millions of documents in several folders and subfolders in
> my machine. I tried to write a script as follows, to extract all the .doc
> files and to convert them in text, but it seems it is taking too much of
> time.
> 
> import os
> from fnmatch import fnmatch
> import win32com.client
> import zipfile, re
> def listallfiles2(n):
>     root = 'C:\Cand_Res'
>     pattern = "*.doc"
>     list1=[]
>     for path, subdirs, files in os.walk(root):
>         for name in files:
>             if fnmatch(name, pattern):
>                 file_name1=os.path.join(path, name)
>                 if ".doc" in file_name1:
>                     #EXTRACTING ONLY .DOC FILES
>                     if ".docx" not in file_name1:
>                         #print "It is A Doc file$$:",file_name1
>                         try:
>                             doc = win32com.client.GetObject(file_name1)
>                             text = doc.Range().Text
>                             text1=text.encode('ascii','ignore')
>                             text_word=text1.split()
>                             #print "Text for Document File Is:",text1
>                             list1.append(text_word)
>                             print "It is a Doc file"
>                         except:
>                             print "DOC ISSUE"
> 
> But it seems it is taking too much of time, to convert to text and to
> append to list. Is there any way I may do it fast? I am using Python2.7 on
> Windows 7 Professional Edition. Apology for any indentation error.
> 
> If any one may kindly suggest a solution.

It will not help the first time through your documents, but if you write the 
words for the word documents in one .txt file per .doc, and the original 
files rarely change you can read from the .txt files when you run your 
script a second time. Just make sure that the .txt is younger than the 
corresponding .doc by checking the file time.

In short: use a caching strategy.