Comms Protocol design dilemma

[Jeethu Rao]

Hi,
I've been working on a encrypted instant messenger server for
quite some time now. I've had to dump the code about 3 times
(I changed the protocols). And quite frankly I don't feel that
bad while throwing out code written in python, since its very
easy to rewrite the same functionality (Throwing away *hard written*
C code really breaks my heart.)

Now all the past three implementations are based on a packet
Oriented scheme <TCP/IP, Blowfish Encryption,16 byte packet hdr>.
My friends in here are suggesting me to use a line oriented protocol
(Like Microsoft Messenger), some have even suggested using XML
(Which I vehemently hate) like in Jabber. I'm of the opinion that
XML tends to waste a lot of bandwidth.

Since most of the data is binary, If I go in for line oriented protocol,
I'll have to base64 (or something similar) all the encrypted data,
Which will be an added overhead.

The primary dilemma is, What protocol is better suited, 'text based' or
'packet based' ?

The other problem is in the implementation.
The old implementations used a simple scheme, every incoming
Connection would be handled by a thread from a thread pool.
But now, the client is going to be connected to the server
All the time its online. Thus, a lot of clients means a lot
Of threads running in parallel, doing nothing. I'm not quite sure
If the Python threads waiting on sockets would yield as soon
as it gets its share of the VM (They're all serialized, right ?).
And since there's a lot of encryption/ decryption , hashing, random
number generation involved, I can pretty much rule out asynchronous
sockets. Is there any better strategy to tackle this problem?


Thanks,

Jeethu Rao