File buffer flush issue in multi-process in Windows

[javen72 at gmail.com]

On 1月13日, 下午2时03分, jave... at gmail.com wrote:
> Hi,
>
> I encountered a very strange issue in file flush operation in Windows.
> Here's the scenario of my application:
>
>     1. The main thread of my application will create makefiles
> sequentially.
>     2. Once a makefile is generated, launch a new process calling
> nmake.exe to run it. The main thread then create another makefile
> until no more makefiles to create.
>     3. The number of new processes is limited by command line options.
>
> My application has been running for almost a year without any problem.
> But, after I made some changes recently to the content of makefile
> generated, "nmake.exe" in separate process sometimes complains that
> the makefile is not found. When I went into the directory, the
> makefile did exist there.
>
> Because I didn't change any thing related to file creation and the new
> makefiles are a little bit less than before, I guessed that the
> makefile just created hasn't been flushed to disk because of size
> change so the new process could not see it in a short time.
>
> So I decided add code to force flush file buffer after writing the
> file content (I didn't flush the file explicitly before). I did it
> like below:
>
>     Fd = open(File, "w")
>     Fd.write(Content)
>     Fd.flush()
>     os.fsync(Fd.fileno())
>     Fd.close()
>
> The strangest thing happened. The "makefile" missing happened more
> frequently than no flush operation. I searched the web but no answer
> there.
>
> Finally I tried to use Windows file API to create the file via pywin32
> extension. The problem's gone.
>
>         import win32file
>         Fd = win32file.CreateFile(
>                         File,
>                         win32file.GENERIC_WRITE,
>                         0,
>                         None,
>                         win32file.CREATE_ALWAYS,
>                         win32file.FILE_ATTRIBUTE_NORMAL,
>                         None
>                         )
>         win32file.WriteFile(Fd, str(Content), None)
>         win32file.FlushFileBuffers(Fd)
>         win32file.CloseHandle(Fd)
>
> I tried writing small python extension in C to make use Windows API to
> create file like above. It also works well, even I removed the
> FlushFileBuffers() calling.
>
> Did anyone encounter such problem before? Did I do something wrong in
> Python file operation? If not, I believe that there's a bug in Python
> file buffer mechanism. I'd appreciate any help or hints.
>
> Cheers,
> Javen

And the same application has no such problem in Linux.