[Twisted-Python] non-blocking interface for local file system operations
Hi, Is there an asynchronous wrapper for local file system operation (i.e. write a file to disk)? I was looking for something like what twisted.enterprise.adbapi do to database access. And I guess using deferToThread is not the right approach for I/O operations. Regards, Eduardo
Hi On 8/9/07, Eduardo Schettino <schettino72@gmail.com> wrote:
Is there an asynchronous wrapper for local file system operation (i.e. write a file to disk)?
I don't think there is anything that works out of the box. I think there is a python wrapper for posix aio somewhere, but AFAIK it is not integrated into twisted in any way. And I guess using deferToThread is not the right approach for I/O
operations.
Usually writes take _very_ little time. The data is only copied to kernel space and control is returned. The data is then written to disk later. Reading will probably be a bit more problematic, but it will still take little time compared to (common) network IO. I suggest starting with just doing writes and reads directly and if it becomes a problem, then start to solve it. You can create a small file wrapper which returns a defer (just use return defer.succeed() and then change the implementation if it becomes necessary. -- - Henrik
Hi Henrik, Thanks for the information. It was really helpful. I was thinking that my IO operations would be an issue. I will take your suggestion and do IO directly by now. Regards, Eduardo On 8/9/07, Henrik Thostrup Jensen <thostrup@gmail.com> wrote:
Hi
On 8/9/07, Eduardo Schettino <schettino72@gmail.com> wrote:
Is there an asynchronous wrapper for local file system operation (i.e. write a file to disk)?
I don't think there is anything that works out of the box. I think there is a python wrapper for posix aio somewhere, but AFAIK it is not integrated into twisted in any way.
And I guess using deferToThread is not the right approach for I/O operations.
Usually writes take _very_ little time. The data is only copied to kernel space and control is returned. The data is then written to disk later. Reading will probably be a bit more problematic, but it will still take little time compared to (common) network IO.
I suggest starting with just doing writes and reads directly and if it becomes a problem, then start to solve it. You can create a small file wrapper which returns a defer (just use return defer.succeed() and then change the implementation if it becomes necessary.
-- - Henrik _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Thanks for the information. It was really helpful. I was thinking that my IO operations would be an issue. I will take your suggestion and do IO directly by now.
One thing to consider if it really turns out to be a problem is to replace plain files with database. Comparing performance of file access with performance of database access is large and non-trivial topic, but there are usage schemes where databases win. And there are asynchronous drivers, like pgasync, available.
participants (3)
-
Eduardo Schettino -
Henrik Thostrup Jensen -
Marcin Kasperski