[Twisted-Python] Implementing a custom FTP server
Hi, all,
I'm trying to implement a custom FTP server (eventually SFTP, but I want to get the basics down.) It won't operate on the file system; the back end will be a database, and the server will generate virtual directories and folders on the fly.
I have a basic Twisted-based FTP server running. I can log in, and I have successfully overridden ftp_LIST to display a custom "file" list.
I have gotten stuck on ftp_STOR and ftp_RETR. There is no implementation of ftp_STOR in the base FTP class, so I have no reference to work against, and I am also having a hard time figuring out how ftp_RETR works. How or where is the actual file data passed back and forth?
Any pointers would be greatly appreciated.
Thanks,
Jason
P.S. A lack of documentation aside, though, Twisted has been quite easy to work with so far. Even fun :)
P.P.S. is SFTP different enough from FTP that I should be thinking about implementing it from the start, rather than later, or can I just graft an SSL layer onto my FTP server?
On Mon, 2004-09-06 at 01:02, Jason Stitt wrote:
I'm trying to implement a custom FTP server (eventually SFTP, but I want to get the basics down.)
Are you aware that:
1. Twisted's FTP server has in theory support for generic backends? (though it is currently being rewritten, so you could help with that).
2. SFTP (SSH's transfer protocol) is very different protocol than FTP.
3. Twisted also has a SFTP implementation.
P.P.S. is SFTP different enough from FTP that I should be thinking about implementing it from the start, rather than later, or can I just graft an SSL layer onto my FTP server?
Oh, hm. Perhaps you should clarify. When you say "sftp" do you mean SSH2's sftp protocol (totally different than FTP) or FTP over TLS?
As far as details about how the protocol works, you should check out Dan Bernstein's FTP docs and of course the RFCs.
On Sep 6, 2004, at 2:15 PM, Itamar Shtull-Trauring wrote:
On Mon, 2004-09-06 at 01:02, Jason Stitt wrote:
I'm trying to implement a custom FTP server (eventually SFTP, but I want to get the basics down.)
Are you aware that:
- Twisted's FTP server has in theory support for generic backends?
(though it is currently being rewritten, so you could help with that).
I'm not quite sure what you mean. Right now, I am subclassing twisted.protocols.ftp.FTP and selectively overriding methods such as ftp_LIST, etc. Is there a better way?
And I'd be glad to help with the rewrite if I knew where to start, as it looks like I will be doing quite a bit of work with it.
SFTP (SSH's transfer protocol) is very different protocol than FTP.
Twisted also has a SFTP implementation.
I thought it might.
To answer your further questions, I'm not that experienced with programming the FTP or SFTP protocol (yet...) so I don't know a lot about the differences. I want to use SFTP eventually for security reasons but figured that FTP would be simpler to start out with. If they're that different, perhaps I should start looking into SFTP from the start.
Thanks,
Jason
On Mon, 2004-09-06 at 14:34 -0500, Jason Stitt wrote:
To answer your further questions, I'm not that experienced with programming the FTP or SFTP protocol (yet...) so I don't know a lot about the differences. I want to use SFTP eventually for security reasons but figured that FTP would be simpler to start out with. If they're that different, perhaps I should start looking into SFTP from the start.
It's easier to talk about the similarities between SFTP and FTP than the differences. The only real similarities between the two protocols are 1) the names 2) that they interface to a filesystem. Other than that, they're two entirely different protocols. The Interfaces that Conch's SFTP implementation uses are twisted.conch.interfaces.ISFTPServer and ISFTPFile.
-p
On Mon, 2004-09-06 at 14:34 -0500, Jason Stitt wrote: To answer your further questions, I'm not that experienced with programming the FTP or SFTP protocol (yet...) so I don't know a lot about the differences. I want to use SFTP eventually for security reasons but figured that FTP would be simpler to start out with. If they're that different, perhaps I should start looking into SFTP from the start.
Paul Swartz wrote:
It's easier to talk about the similarities between SFTP and FTP than the differences. The only real similarities between the two protocols are
- the names 2) that they interface to a filesystem. Other than that,
they're two entirely different protocols. The Interfaces that Conch's SFTP implementation uses are twisted.conch.interfaces.ISFTPServer and ISFTPFile.
G'dday Jason,
I'm chipping away in this area as well as I find time. I started with SFTP - but also want to support all filelike interfaces in the future.
I've opened up svn access to what I have so far:
svn list svn://twisted.cablelounge.com/welsh/trunk
Its very rough .. pretty embarrassing really :) .. hopefully its of some help though, and it'd be good to get feedback - particularly as you're working with FTP.
The general strategy is to represent your file store independent of any protocol. I've a simple sambaish like file store in file.py. Then for each protocol write an adaptor from your file store objects to the interface that protocols expects. fileadapt.py does this for SFTP. realm.py ties it all together.
You can play with it by creating a tap file:
mktap welsh -p 5822 -d /etc/ssh
or something similiar - it's pretty much conch ..
Then create a file 'welshpasswd' to define the accounts on your system and place it in the same directory as the tap file.
The file sets up a users variable:
users = { loginuserid : { nameOfVirtualDirectory : (uid, gid, realSystemDirectory), nameOfVirtualDirectory2 : (uid, gid, realSystemDirectory2), } }
example welshpasswd: --------------------- users = { "andy" : { "home" : ( 509, 509, "/home/andy" ), }, "az" : { "home" : ( 518, 518, "/home/az/incoming" ), "giiwa" : ( 108, 108, "/home/giiwa" ), "muddypigrecords" : ( 99, 99, "/home/muddypigrecords" ), }, } ----------------------
erg, apologies for the roughness :(
When az logins in, he'll see three directories, home, giiwa and muddypigrecords - while working in those directories he'll assume the uid and gid and be working in the physical directory as defined above.
Itamar Shtull-Trauring wrote:
- Twisted's FTP server has in theory support for generic backends?
(though it is currently being rewritten, so you could help with that).
Good luck! I'm yet to take a look, but wouldn't mind helping out if the FTP backend needs a bit of work to be made more accessible.
Andy.
G'dday Jason,
I'm chipping away in this area as well as I find time. I started with SFTP - but also want to support all filelike interfaces in the future.
This sounds like a good idea. Unfortunately, I cannot get twistd and mktap working properly on OS X. I've been just using reactor objects directly, as the simpler examples do. So I'll have to set it up on my Linux server sometime to try it out. I'll get you some feedback when I do.
Jason
Jason Stitt jason@pengale.com writes:
G'dday Jason,
I'm chipping away in this area as well as I find time. I started with SFTP - but also want to support all filelike interfaces in the future.
This sounds like a good idea. Unfortunately, I cannot get twistd and mktap working properly on OS X.
What have you been trying? Admittedly. I'm not using mktap, but I'm using twistd on OS X with approximately no issues at all (apart from the fact that it got installed in the back of beyond, under /Library/Frameworks/...).
Cheers, mwh
On Sep 7, 2004, at 5:21 AM, Michael Hudson wrote:
Jason Stitt jason@pengale.com writes:
This sounds like a good idea. Unfortunately, I cannot get twistd and mktap working properly on OS X.
What have you been trying? Admittedly. I'm not using mktap, but I'm using twistd on OS X with approximately no issues at all (apart from the fact that it got installed in the back of beyond, under /Library/Frameworks/...).
Originally, twistd contained a hard-coded check that told me I was using Windows (the nerve!) and told me to use twistw (which, of course, didn't work). I removed that. Now I'm still having some problems with twistd not respecting the working directory and, thus, not being able to find modules to import.
Jason
On Tue, 07 Sep 2004 13:23:50 -0500, Jason Stitt jason@pengale.com wrote:
On Sep 7, 2004, at 5:21 AM, Michael Hudson wrote:
Originally, twistd contained a hard-coded check that told me I was using Windows (the nerve!) and told me to use twistw (which, of course, didn't work).
If this happened to you, it's because you're not calling the right thing. You should be calling the twistd in Python2X\Scripts (IIRC), *not* Python2X\Lib\site-packages\twisted\scripts\twistd.py. If you use the twisted-environment command line that's placed into your Start menu, just typing "twistd" should work.
On Tue, 2004-09-07 at 14:23, Jason Stitt wrote:
didn't work). I removed that. Now I'm still having some problems with twistd not respecting the working directory and, thus, not being able to find modules to import.
This happens on all platforms when you run as root. It's a security feature. It will not be removed. So your best bet is to add the directory where your package is to PYTHONPATH, as with all Python programs.
Oh, and, why twistw? I'd expect twistd to use the unix version on Mac OS X.
On Tue, 07 Sep 2004 14:30:33 -0400, Itamar Shtull-Trauring itamar@itamarst.org wrote:
Oh, and, why twistw? I'd expect twistd to use the unix version on Mac OS X.
Oh, I didn't realize he was on OSX. I take it sys.platform != "posix" there? Guess that check should more explicitly check for win32.
On Sep 7, 2004, at 2:35 PM, Christopher Armstrong wrote:
On Tue, 07 Sep 2004 14:30:33 -0400, Itamar Shtull-Trauring itamar@itamarst.org wrote:
Oh, and, why twistw? I'd expect twistd to use the unix version on Mac OS X.
Oh, I didn't realize he was on OSX. I take it sys.platform != "posix" there? Guess that check should more explicitly check for win32.
uh, of course sys.platform == 'posix'
I dunno what the heck he is doing, but twistd has always worked fine for me on OS X.
-bob
On Tuesday 07 September 2004 02:47 pm, Bob Ippolito wrote:
uh, of course sys.platform == 'posix'
I dunno what the heck he is doing, but twistd has always worked fine for me on OS X.
Mac OS X 10.2.8 says this:
Python 2.2 (#1, 07/14/02, 23:25:09) [GCC Apple cpp-precomp 6.14] on darwin Type "help", "copyright", "credits" or "license" for more information.
import os, sys os.name
'posix'
sys.platform
'darwin'
On Windows, I expect os.name == "nt" and sys.platform[:3].lower() == "win".
-Fred
On Sep 7, 2004, at 1:35 PM, Christopher Armstrong wrote:
On Tue, 07 Sep 2004 14:30:33 -0400, Itamar Shtull-Trauring itamar@itamarst.org wrote:
Oh, and, why twistw? I'd expect twistd to use the unix version on Mac OS X.
Oh, I didn't realize he was on OSX. I take it sys.platform != "posix" there? Guess that check should more explicitly check for win32.
Right. sys.platform is 'darwin' on OS X.
-- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Sep 7, 2004, at 2:54 PM, Jason Stitt wrote:
On Sep 7, 2004, at 1:35 PM, Christopher Armstrong wrote:
On Tue, 07 Sep 2004 14:30:33 -0400, Itamar Shtull-Trauring itamar@itamarst.org wrote:
Oh, and, why twistw? I'd expect twistd to use the unix version on Mac OS X.
Oh, I didn't realize he was on OSX. I take it sys.platform != "posix" there? Guess that check should more explicitly check for win32.
Right. sys.platform is 'darwin' on OS X.
Presumably he meant os.name .. I don't think sys.platform == 'posix' anywhere.
-bob
All right. I think I had some version problems or something. I checked with my Linux install, and it uses twisted.python.runtime.platformType, which correctly identifies OS X as 'posix'.
So twistd is no longer telling me that I run Windows. Thank goodness.
Jason
On Tue, 2004-09-07 at 14:35, Christopher Armstrong wrote:
Oh, I didn't realize he was on OSX. I take it sys.platform != "posix" there? Guess that check should more explicitly check for win32.
The code does not check sys.platform. It checks twisted.python.runtime.platformType, which should be "posix" on OS X, regardless of what sys.platform is.
On Tue, 07 Sep 2004 15:54:45 -0400, Itamar Shtull-Trauring itamar@itamarst.org wrote:
On Tue, 2004-09-07 at 14:35, Christopher Armstrong wrote:
Oh, I didn't realize he was on OSX. I take it sys.platform != "posix" there? Guess that check should more explicitly check for win32.
The code does not check sys.platform. It checks twisted.python.runtime.platformType, which should be "posix" on OS X, regardless of what sys.platform is.
Sorry, I was accidentally looking at 1.3.0. :-)
On Sep 7, 2004, at 1:30 PM, Itamar Shtull-Trauring wrote:
On Tue, 2004-09-07 at 14:23, Jason Stitt wrote:
didn't work). I removed that. Now I'm still having some problems with twistd not respecting the working directory and, thus, not being able to find modules to import.
This happens on all platforms when you run as root. It's a security feature. It will not be removed. So your best bet is to add the directory where your package is to PYTHONPATH, as with all Python programs.
That makes sense. I just figured out I was getting a little confused -- not all my import problems were local. One of the examples I downloaded from the Twisted site (the SSH server, sshsimpleserver.py) tries to import realm from twisted.conch, which does not have a realm module (it's not in the API docs, either). Was this removed at some point, perhaps?
Jason
On Tue, 2004-09-07 at 14:11 -0500, Jason Stitt wrote:
One of the examples I downloaded from the Twisted site (the SSH server, sshsimpleserver.py) tries to import realm from twisted.conch, which does not have a realm module (it's not in the API docs, either). Was this removed at some point, perhaps?
Yes, the stuff in realm.py was refactored and moved into unix.py. SVN has a working version.
-p
Paul Swartz wrote:
On Tue, 2004-09-07 at 14:11 -0500, Jason Stitt wrote:
One of the examples I downloaded from the Twisted site (the SSH server, sshsimpleserver.py) tries to import realm from twisted.conch, which does not have a realm module (it's not in the API docs, either). Was this removed at some point, perhaps?
Yes, the stuff in realm.py was refactored and moved into unix.py. SVN has a working version.
Jason, just a quick note.
I've been putting off changing over to the new component architecture so I'm working against a pretty out of date version of trunk.
I'm currently running welsh against Twisted r10936 in pseudo production (if you call serving up a few mates sites production:)). It looks like Paul made the above refactored in r10415, so that's there.
This is a good spur to finally make the change over, so I should get to it in the next weekend of so.
Andy.
On Sep 7, 2004, at 7:40 PM, Andy Gayton wrote:
Paul Swartz wrote:
Jason, just a quick note.
I've been putting off changing over to the new component architecture so I'm working against a pretty out of date version of trunk.
It's not just you :) I downloaded SFTPGateway, a Twisted program that serves as a secure proxy to a regular FTP server, because I thought I could glean some insights into SFTP from it. But it doesn't use any of Twisted's actual SFTP implementation code, as far as I can tell, and I'm not sure I want to pick apart all the code they wrote from scratch. (Plus, it's for Zope, not standalone...)
In other news, I downloaded the latest SVN and am trying to pick apart the sshsimpleserver.py example again. The log output, below, claims the factory has been created and is opening the port, but I can't connect to it, and using nmap I can see that the port is not actually open. There's one warning I can't quite interpret. Can anyone help me figure this out?
The regular echoserver example works just fine.
Thanks,
Jason
--STARTUP-- 2004/09/07 22:01 EDT [-] Log opened. 2004/09/07 22:01 EDT [-] __main__.ExampleFactory starting on 5022 2004/09/07 22:01 EDT [-] Starting factory <__main__.ExampleFactory instance at 0x406f4d0c> 2004/09/07 22:01 EDT [-] disabling diffie-hellman-group-exchange because we cannot find moduli file
--SHUTDOWN-- 2004/09/07 22:02 EDT [-] Received SIGINT, shutting down. 2004/09/07 22:02 EDT [__main__.ExampleFactory] (Port 5022 Closed) 2004/09/07 22:02 EDT [__main__.ExampleFactory] Stopping factory <__main__.ExampleFactory instance at 0x406f4d0c> 2004/09/07 22:02 EDT [-] Main loop terminated.
On Tue, 2004-09-07 at 21:03 -0500, Jason Stitt wrote:
It's not just you :) I downloaded SFTPGateway, a Twisted program that serves as a secure proxy to a regular FTP server, because I thought I could glean some insights into SFTP from it. But it doesn't use any of Twisted's actual SFTP implementation code, as far as I can tell, and I'm not sure I want to pick apart all the code they wrote from scratch. (Plus, it's for Zope, not standalone...)
Yes it does, it inherits from filetransfer.FileTransferServer, which is Conch's SFTP implementation.
2004/09/07 22:01 EDT [-] disabling diffie-hellman-group-exchange because we cannot find moduli file
You can ignore that. There are two types of key exchanges that Conch supports: one using some fixed values, and other that uses a moduli (usually /etc/ssh/moduli) file to get lots of values. If it can't find the moduli file, it falls back to the other kind.
-p
On Tuesday 07 September 2004 10:03 pm, Jason Stitt wrote:
It's not just you :) I downloaded SFTPGateway, a Twisted program that serves as a secure proxy to a regular FTP server, because I thought I could glean some insights into SFTP from it. But it doesn't use any of Twisted's actual SFTP implementation code, as far as I can tell, and I'm not sure I want to pick apart all the code they wrote from scratch. (Plus, it's for Zope, not standalone...)
Hehe. As Paul noted, yes, the SFTP Gateway does use his SFTP implementation. There's a lot of code there, but it's really just the code that implements the FTP client that the SFTP server uses as the backend (in sftpgateway.connector). The FileTransferServer interface was finallized based substantially on what we did in SFTP Gateway.
I will note that it was written against Twisted 1.1.0, so the APIs on Twisted's side have changed a bit, but that mostly affects the code in sftpgateway.application and maybe sftpgateway.authentication. From what I recall from a discussion with Paul at PyCon'04, the changes in Twisted really are an improvement. I wish I had the time to bring that up to date.
-Fred
participants (8)
-
Andy Gayton -
Bob Ippolito -
Christopher Armstrong -
Fred L. Drake, Jr. -
Itamar Shtull-Trauring -
Jason Stitt -
Michael Hudson -
Paul Swartz