On 03:21 am, donal.mcmullan@gmail.com wrote:

I'm interested in doing some FUSE stuff with Twisted, mostly to prototype some ideas for profiling. Can anyone expand on Glyph's comment [1]:

8< - - - - - - - - - - - - - - - - - - - - - - - - - - - - The Python FUSE bindings obscure the issue because, unlike the C libfuse, they assume that your filesystem I/O is blocking, which severely limits the performance of python-based filesystems. �(You cannot receive more requests for I/O in your filesystem until the previous one has been completed with the pyfuse bindings, but you can in C.) 8< - - - - - - - - - - - - - - - - - - - - - - - - - - - -

I guess I'd like to know before I sink too much time into this, if there's an approach to Fuse/Twisted that would not have this limitation. This is a bit confused by the fact that there are three Fuse/Python bindings that I'm aware of: FusePython, FusePy, and S3QL [2].

http://sourceforge.net/apps/mediawiki/fuse/index.php?title=FusePython http://code.google.com/p/fusepy/ http://code.google.com/p/s3ql/

If you've already been down this road, it'd be good to hear from you.

So, sitting at the bottom of libfuse is a file descriptor. The kernel asks your process to service some filesystem operation by putting bytes into it that your process can read. Your process services the requests by putting bytes into it that the kernel can read. Each request/response pair is tagged, so there's no requirement that requests be responded to in the same order they're received. A good Twisted-based solution would use the reactor to read and write that file descriptor (alongside anything else going on in the process) and support Deferreds (or at least asynchronous responses) to allow out- of-order responses and concurrent request handling. I played around with fuse a little bit to implement something like sshfs a while ago. I did _not_ use the approach described above, because it was easier to use something from Armin Rigo's sandbox in a thread, dispatching request handling into Twisted, to achieve the necessary concurrency and so forth. You can find that code here: http://twistedmatrix.com/trac/browser/sandbox/exarkun/sshfs.py?rev=26941 It's just something I was messing around with. I'm sure you'll find lots of problems with it (I know of at least one). It might help with part of the general idea, though. To get rid of the use of threads (which is a good idea for a real tool), you just need to port Armin's parsing code into a Twisted Protocol class and hook it up to a FileDescriptor of some sort that reads and writes the fuse fd. Jean-Paul

On 03:21 am, donal.mcmullan@gmail.com wrote:

I'm interested in doing some FUSE stuff with Twisted, mostly to prototype some ideas for profiling. Can anyone expand on Glyph's comment [1]:

8< - - - - - - - - - - - - - - - - - - - - - - - - - - - - The Python FUSE bindings obscure the issue because, unlike the C libfuse, they assume that your filesystem I/O is blocking, which severely limits the performance of python-based filesystems. �(You cannot receive more requests for I/O in your filesystem until the previous one has been completed with the pyfuse bindings, but you can in C.) 8< - - - - - - - - - - - - - - - - - - - - - - - - - - - -

I guess I'd like to know before I sink too much time into this, if there's an approach to Fuse/Twisted that would not have this limitation. This is a bit confused by the fact that there are three Fuse/Python bindings that I'm aware of: FusePython, FusePy, and S3QL [2].

http://sourceforge.net/apps/mediawiki/fuse/index.php?title=FusePython http://code.google.com/p/fusepy/ http://code.google.com/p/s3ql/

If you've already been down this road, it'd be good to hear from you.

So, sitting at the bottom of libfuse is a file descriptor. The kernel asks your process to service some filesystem operation by putting bytes into it that your process can read. Your process services the requests by putting bytes into it that the kernel can read. Each request/response pair is tagged, so there's no requirement that requests be responded to in the same order they're received. A good Twisted-based solution would use the reactor to read and write that file descriptor (alongside anything else going on in the process) and support Deferreds (or at least asynchronous responses) to allow out- of-order responses and concurrent request handling. I played around with fuse a little bit to implement something like sshfs a while ago. I did _not_ use the approach described above, because it was easier to use something from Armin Rigo's sandbox in a thread, dispatching request handling into Twisted, to achieve the necessary concurrency and so forth. You can find that code here: http://twistedmatrix.com/trac/browser/sandbox/exarkun/sshfs.py?rev=26941 It's just something I was messing around with. I'm sure you'll find lots of problems with it (I know of at least one). It might help with part of the general idea, though. To get rid of the use of threads (which is a good idea for a real tool), you just need to port Armin's parsing code into a Twisted Protocol class and hook it up to a FileDescriptor of some sort that reads and writes the fuse fd. Jean-Paul