Well, I ought to add some words were I think twisted.ftp is heading too. The first things to do, is to add proper authentication, and filling in some holes (eg, resume and hash). A better path-handling, since os.path is silly under windows, and os.path.isabs('..'). Distributed is the next big spot. I'm not entirely safe with it yet, but I think it should be easy to nail. One pending 'TODO' is to make it deployable as a systemwide ftp (without requiring each and every user to setup their own ftp-server).
So, if I jump forward in time, and extend the view to storing eg. media-files generally, I think the need of a vfs of some kind arises. What follows is some random notes on it, with varying value, and passing the scope of twisted.ftp with lightyears. There's two properties it requires: it has to store metadata , and it has to support different 'filesystems'. The direct effect of the storing the metadata is that the differences between OSes and platforms can be compensated and metadata don't have to be lost due to the filesystems. Though, I think this can be a burden too (should it keep all the metadata in dicts and stored in fragile taps?). There's only one filesystem I know which can easily store metadata, and that is a database :) twisted.enterprise could turn handy here, though it seems to require an external SQL-server. Correct me if I'm wrong. Other filesystems? The OS's filesystem, FTP, Source-Code (hm), Some other requirements, is the ability to refer to a file independent of its physical location, pass the reference over PB, do basic operations, and of course, version-checking etc.
This issue is probably related to the "File Transfer"-layer which is mentioned in the TODO too:
008: File Transfer layer for PB. This would be especially nice for twisted.words; having standard a way to transfer "large" (100MB+) packets across or in tandem with a PB connection without breaking anything would be very good.
Benjamin Bruheim wrote:
So, if I jump forward in time, and extend the view to storing eg. media-files generally, I think the need of a vfs of some kind arises. What follows is some random notes on it, with varying value, and passing the scope of twisted.ftp with lightyears.
Take a look at Medusa (www.nightmare.com/medusa) VFS as an example of how *not* to do it. One thing I learned from it - string paths are EVIL, especially in FTP where paths such as /baz/.././//foo/../bar are legal. Paths should therefore be represented by lists, e.g. ['baz', 'foo', 'bar'].
The next problem is that it's usually hard to know if the last item is a file or a folder (if the path is '/foo/bar', is 'bar' a file or a folder?). In systems like Zope it's even harder since an object can function as both. I would suggest you get a WebDAV implementation going before you (or any other interested party) start working on a VFS layer. WebDAV is more generic (it support "collections", not "folders"), supports arbitary properties on objects (metadata), etc.. And supporting two different systems will amke sure you don't make any protocol specific design decisions (such as Medusa's VFS, where it passes the VFS the path string specified by the user, so the VFS must deal with '..', './../foo//bar' and all that crap.)
Awesome, Ben! And good feedback, Itamar! WebDAV would be a good choice -- supporting existing standards and all that.