[Twisted-Python] Server/Client bi-directional data-transfer - need a solution
Hi, *The Goal:* - I'm trying to write a simple server/client app where the client can send the server either a small string like "hello" OR a small file (txt, pdf, ppt,.. let's say < 5MB ). - The server needs to figure out if the data it received is a simple string or a file and symmetrically may decide to send back to the client a simple string or a small file. - As a bonus, It's highly desirable that the transfer be secure - but it's ok for the first version if it isn't *Questions:* What's the cleanest way to accomplish this? I've been looking at the single Howto/Example that exists for *twisted.conch* but still can't understand several things. In particular 1. Should I create two separate channels between server and client - one for simple strings and one for files so that the client or server knows the type of data on that channel? 2. Should the files be mime-encoded at one end and mime-decoded at the other so that file-type is automatically handled? 3. I've read that there is a python package called paramiko that implements ssh2 transfer - how does this compare with twisted.conch? This is my first time playing around with twisted or any kind of networking stuff so any kind of help will be appreciated Thanks! Sajit
On Wednesday 12 December 2007, Sajit Rao wrote:
*The Goal:*
- I'm trying to write a simple server/client app where the client can send the server either a small string like "hello" OR a small file (txt, pdf, ppt,.. let's say < 5MB ). - The server needs to figure out if the data it received is a simple string or a file and symmetrically may decide to send back to the client a simple string or a small file. - As a bonus, It's highly desirable that the transfer be secure - but it's ok for the first version if it isn't
Would an HTTP POST suffice? You could easily test it using a web browser or a program like wget and adding a secure channel is done by switching from HTTP to HTTPS later. If your application works like an instant messenger, consider using Twisted Words, for example with the XMPP (Jabber) protocol. Does the client/server connection have to be able to pass a firewall or proxy? In that case HTTP or something that can be run over HTTP would be a big advantage.
*Questions:*
What's the cleanest way to accomplish this? I've been looking at the single Howto/Example that exists for *twisted.conch* but still can't understand several things. In particular
1. Should I create two separate channels between server and client - one for simple strings and one for files so that the client or server knows the type of data on that channel?
That depends on what protocol you want to use over the channel(s). If the protocol can send series of text strings and binaries over the same channel then you need only one. But if having separate channels helps in simplifying the protocol, then that might be better. One design decision to make is whether to keep the channel(s) open the whole time or open them when there is new data and then close them when the transfer is done. Or maybe keep the text string channel open all the time and open a file transfer channel on demand. What kind of performance characteristics do you need? Does latency on the text strings matter? Are you going to transfer 10 files a day or 10000?
2. Should the files be mime-encoded at one end and mime-decoded at the other so that file-type is automatically handled?
MIME encoding gives you a way to send the file name and type and it can be used over a text stream. However, the encoding of the data itself is not really needed if your channel is 8-bit clear and everything except e-mail is 8-bit clear nowadays (I think). For small files the overhead is neglible though.
3. I've read that there is a python package called paramiko that implements ssh2 transfer - how does this compare with twisted.conch?
I don't know. One thing you could look at is which main loop you want to use. If you want to use Twisted's reactor, conch would better integrate with that of course. But if you want to use a GUI toolkit not supported by Twisted, it would take some effort to merge the main loops, so then running paramiko in a thread might be easier. Another thing to look at is the license: paramiko is LGPL, which places more restrictions on its use than Twisted's MIT license. Depending on your situation that may or may not be important. Bye, Maarten
Thanks for your reply, Maarten.
I'm still chewing over it. In answer to your questions:
I will be sending short strings between client and server (bi-directionally)
most of the time (> 100 times a day) and ocassionaly files (each of size 5
MB or less and > 10 but <100 times a day).
I wouldn't mind sending strings or files along the same channel - but I
don't know how to do that
While I like the brevity of python and realize that Twisted deffereds are
the heart of the async model, I'm hampered by the scant documentation and
code fragments.
Could someone point me to how to send raw strings or binary files over the
same channel and have the other side interpret the data correctly?
Thanks,
Sajit
On Dec 12, 2007 5:20 AM, Maarten ter Huurne
On Wednesday 12 December 2007, Sajit Rao wrote:
*The Goal:*
- I'm trying to write a simple server/client app where the client can send the server either a small string like "hello" OR a small file (txt, pdf, ppt,.. let's say < 5MB ). - The server needs to figure out if the data it received is a simple string or a file and symmetrically may decide to send back to the client a simple string or a small file. - As a bonus, It's highly desirable that the transfer be secure - but it's ok for the first version if it isn't
Would an HTTP POST suffice? You could easily test it using a web browser or a program like wget and adding a secure channel is done by switching from HTTP to HTTPS later.
If your application works like an instant messenger, consider using Twisted Words, for example with the XMPP (Jabber) protocol.
Does the client/server connection have to be able to pass a firewall or proxy? In that case HTTP or something that can be run over HTTP would be a big advantage.
*Questions:*
What's the cleanest way to accomplish this? I've been looking at the single Howto/Example that exists for *twisted.conch* but still can't understand several things. In particular
1. Should I create two separate channels between server and client - one for simple strings and one for files so that the client or server knows the type of data on that channel?
That depends on what protocol you want to use over the channel(s). If the protocol can send series of text strings and binaries over the same channel then you need only one. But if having separate channels helps in simplifying the protocol, then that might be better.
One design decision to make is whether to keep the channel(s) open the whole time or open them when there is new data and then close them when the transfer is done. Or maybe keep the text string channel open all the time and open a file transfer channel on demand.
What kind of performance characteristics do you need? Does latency on the text strings matter? Are you going to transfer 10 files a day or 10000?
2. Should the files be mime-encoded at one end and mime-decoded at the other so that file-type is automatically handled?
MIME encoding gives you a way to send the file name and type and it can be used over a text stream. However, the encoding of the data itself is not really needed if your channel is 8-bit clear and everything except e-mail is 8-bit clear nowadays (I think). For small files the overhead is neglible though.
3. I've read that there is a python package called paramiko that implements ssh2 transfer - how does this compare with twisted.conch?
I don't know. One thing you could look at is which main loop you want to use. If you want to use Twisted's reactor, conch would better integrate with that of course. But if you want to use a GUI toolkit not supported by Twisted, it would take some effort to merge the main loops, so then running paramiko in a thread might be easier.
Another thing to look at is the license: paramiko is LGPL, which places more restrictions on its use than Twisted's MIT license. Depending on your situation that may or may not be important.
Bye, Maarten
Sajit Rao wrote:
Thanks for your reply, Maarten.
I'm still chewing over it. In answer to your questions:
I will be sending short strings between client and server (bi-directionally) most of the time (> 100 times a day) and ocassionaly files (each of size 5 MB or less and > 10 but <100 times a day).
I wouldn't mind sending strings or files along the same channel - but I don't know how to do that
While I like the brevity of python and realize that Twisted deffereds are the heart of the async model, I'm hampered by the scant documentation and code fragments.
Could someone point me to how to send raw strings or binary files over the same channel and have the other side interpret the data correctly?
This is really a protocol design question rather than Twisted-specific question. You might find Itamar's series of articles on protocol design to be helpful: http://webservices.xml.com/pub/a/ws/2003/11/25/protocols.html http://webservices.xml.com/pub/a/ws/2004/01/20/sessions.html http://webservices.xml.com/pub/a/ws/2004/02/25/protocols3.html http://webservices.xml.com/pub/a/ws/2004/04/21/protocols.html http://www.xml.com/pub/a/ws/2004/08/25/dprotocol.html (You can find these and other writings at http://www.itamarst.org/) -Andrew.
participants (3)
-
Andrew Bennetts -
Maarten ter Huurne -
Sajit Rao