Problems with file IO in a thread, and shutil

[MRAB]

On Jun 25, 4:11 pm, Roopesh <roopesh.... at gmail.com> wrote:
> Hi,
>
> I have a multithreaded application. There are two threads, T1 and T2.
> Suppose that there are two folders A, B.  Thread T1 fetches data from
> network and creates files in folder A and after creating each file, it
> moves the file to folder B, using shutil.move().
>
> Thread T2, takes files from folder B and processes it.
> Note: Only the thread T1 has access to the files in folder A.
>
> psuedo code
> =========
> class T1(thread):
>    def run():
>      self.download()
>
>   def download():
>      data = from_network
>      filename = os.path.join ( "A", "file1")
>
>      f = open ( filename, "w")
>      for d in data:
>         f.write ( d + "\n" )
>      f.flush()
>      f.close()
>
>     shutil.move(os.path.join ( "A", "file1"),  os.path.join("B",
> "file1"))
>
> class T2(thread):
>   run()
>       process(listdir(os.path.join("B")))
>
> All the files has similar contents. But in some cases(very rare), when
> thread T1 tries to move the newly created file from folder A to folder
> B, an exception occurs (on shutil.move()).
>
> The exception is WindowsError(32, 'The process cannot access the file
> because it is being used by another process'). I looked inside the
> shutil.move. What I found was due to this WindowsError, the usual
> os.rename() inside shutil.move() fails and the file is copied using
> copy2(src, dst). Finally os.unlink() also failed. (So though copying
> occurred, deleting the source file failed)
>
> I am not getting why this error comes. Has anyone faced similar
> situation? Please help me.
>
> Thanks and Regards
> Roopesh

Do you have any anti-virus/anti-spyware software installed? (If not,
why not? :-))

It might be that your anti-virus/anti-spyware software is seeing the
new file appear and scanning it. If that happens just when
shutil.move() attempts to move it then the move will fail. You could
have your code wait for a while and then try again.