I'm working on a project (http://www.cakem.net/) that involves networking. I have to do a lot of decoding of network data into an internal representation, and encoding internal representations into a network format. The decoding part is going to present a problem, a problem that could easily be solved by continuations. I want to right the code so that it will work when there is no guarantee that I'll have an entire message before I start decoding it. This means the parser may have have to stop at any random point in time, and then be restarted again when more data is available. I see no way to do this without continuations, or threads, and I refuse to use threads for something like this. For now, I think I will do one of two things. Either build in the assumption that I will have the entire messages before I start parsing, or organize parsing as a series of steps that can be aborted in the middle and then restarted, and use exceptions to track whether the step was fully completed or not. Sort of a poor man's continuation. Anyway, I'm not a Python developer, just a user. I just wanted to add this to the things you consider when you decide what you intend to do next with Python. Continuations are important, and AFAIK, the stackless mod is the only way to get them. Thanks, -- "It does me no injury for my neighbor to say there are twenty gods or no God. It neither picks my pocket nor breaks my leg." --- Thomas Jefferson "Go to Heaven for the climate, Hell for the company." -- Mark Twain -- Eric Hopper (hopper@omnifarious.org http://www.omnifarious.org/~hopper) --
On Mon, Jan 19, 2004, Eric Mathew Hopper wrote:
I want to right the code so that it will work when there is no guarantee that I'll have an entire message before I start decoding it. This means the parser may have have to stop at any random point in time, and then be restarted again when more data is available. I see no way to do this without continuations, or threads, and I refuse to use threads for something like this.
Python 2.2 added generators that likely can handle what you're trying to do. However, python-dev is not the place to get help on this subject; please use comp.lang.python instead. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ A: No. Q: Is top-posting okay?
I'm working on a project (http://www.cakem.net/) that involves networking. I have to do a lot of decoding of network data into an internal representation, and encoding internal representations into a network format.
The decoding part is going to present a problem, a problem that could easily be solved by continuations.
I want to right the code so that it will work when there is no guarantee that I'll have an entire message before I start decoding it. This means the parser may have have to stop at any random point in time, and then be restarted again when more data is available. I see no way to do this without continuations, or threads, and I refuse to use threads for something like this.
For now, I think I will do one of two things. Either build in the assumption that I will have the entire messages before I start parsing, or organize parsing as a series of steps that can be aborted in the middle and then restarted, and use exceptions to track whether the step was fully completed or not. Sort of a poor man's continuation.
Anyway, I'm not a Python developer, just a user. I just wanted to add this to the things you consider when you decide what you intend to do next with Python. Continuations are important, and AFAIK, the stackless mod is the only way to get them.
Eric, I've taken a look at your protocol, and it looks to be simple enough to just decode the header (maybe partially) and determine (based on the length of the data received) if the entire message has been received yet. Certainly it does end up reparsing the header a few times, but as long as you only check to see if the message has been received on data read (which is guaranteed to be at least 1 byte for select/poll read events, and is commonly much larger), there shouldn't be too much reparsing. I've actually done the same thing with some nontrivial headers, and have had no problems maxing out a 100mbit connection with a PII-400 and a properly subclassed asyncore.dispatcher. Even though it would be computationally more efficient to use continuations or generators, they are not necessary. You will likely find more overhead with the cryptographic portions of your protocol than you will reparsing the header multiple times. - Josiah
On Jan 19, 2004, at 11:14 PM, Josiah Carlson wrote:
I'm working on a project (http://www.cakem.net/) that involves networking. I have to do a lot of decoding of network data into an internal representation, and encoding internal representations into a network format.
The decoding part is going to present a problem, a problem that could easily be solved by continuations.
You could write your implementation of the network protocol using asyncore (medusa), Twisted, and/or PEAK... all of which are good at asynchronous "stuff" to varying degrees. PEAK has a bunch of new things that make it easy to write asynchronous-acting but synchronous-looking code using generators, these may only be in CVS though.
Anyway, I'm not a Python developer, just a user. I just wanted to add this to the things you consider when you decide what you intend to do next with Python. Continuations are important, and AFAIK, the stackless mod is the only way to get them.
I agree, but good luck convincing everyone that it's important enough to merge the fork (it's a fork of Python, not a module). BTW, Christian says the 2.3 branch of Stackless is pretty close to ready. -bob
participants (4)
-
Aahz -
Bob Ippolito -
Eric Mathew Hopper -
Josiah Carlson