On 09:24 pm, mforbes@physics.ubc.ca wrote:

Why does writeSomeData not simply raise CONNECTION_LOST as an exception? Checking return values is quite un-pythonic. Is there a deep reason for this?

writeSomeData is an internal API that nobody should really use, so it's not factored for convenience; it's also deep down in the guts of the reactor's innermost loop, so it pays a lot of attention to efficiency. Raising exceptions is really, really slow, at least by the standards of that kind of code.

P.S. I came across this because I was trying to use twisted running in a thread to write data resulting from a long computation that I have not yet turned into a producer. The more conventional "write" method was failing if the socket backed up, and provided no simple way of determining if data was being dropped. My solution was to used writeSomeData, and then have the computation decide to throw out some of the data if it is being produced too rapidly, but I need to know how much has been sent so I can decide what to throw away...

You do know that Twisted APIs are not thread safe, right? You can't call write() from a thread?

On Oct 1, 2008, at 3:44 PM, Jean-Paul Calderone wrote: ...

...

Why does writeSomeData not simply raise CONNECTION_LOST as an exception? Checking return values is quite un-pythonic. Is there a deep reason for this?

The reason is probably that whoever wrote it thought that raising an exception would be enough slower than returning a special value that it would be worth doing something slightly unusual. Compared to returning a value, raising an exception is expensive in CPython, but I doubt there is any surviving benchmark which demonstrates that it makes a difference here.

However, it's a very low-level API and since it isn't intended to be used by application code, it doesn't seem like a very high priority to make it less confusing.

By the way, is there any convention by which "high level" API routines are distinguished from "low level" routines? It is really not that clear from the docs... (writeSomeData comes first in the listing etc.)

...

P.S. I came across this because I was trying to use twisted running in a thread to write data resulting from a long computation that I have not yet turned into a producer. The more conventional "write" method was failing if the socket backed up, and provided no simple way of determining if data was being dropped. My solution was to used writeSomeData, and then have the computation decide to throw out some of the data if it is being produced too rapidly, but I need to know how much has been sent so I can decide what to throw away...

This seems more interesting than the main point of your email. :) You should probably just do the work to turn the computation into a producer. This because:

* writeSomeData is low level and crufty, as you've discovered * Not all reactors have this API, so your program will break based on the reactor selected to run it * producers give you all the same information

Since you mentioned threads, I'll also point out that Twisted APIs are not safe to use from non-reactor threads without reactor.callFromThread. If you aren't using callFromThread, that would explain bad behavior from the write method.

I know that and am using callFromThread: The reason I am having bad behavior is because the socket is backing up. Once I put in the logic to throw out data, it works fine... but the lack of reactor support concerns me somewhat. I don't want the computation to stop, but think I could put an intermediate "push producer" that stops by discarding the data. I am threading because the computation is the main task, and the network stuff is simply a window into the state of the computation. I want the computation to run continually, and it needs to do so in a blocking sort of way, periodically checking and pushing the partial results (like once every 30 minutes or so). I want the calculation code to function on its own, with or without the twisted components and I don't see a simple way of doing this if twisted is in complete control without some type of threading. My current design is to write my calculation as if twisted did not exist, and just push the current results onto a Queue periodically. Then I run twisted in a separate thread with clients that periodically check the queue, pop stuff off and dispatch it to any clients who happen to connect and want the data. Is there a simpler way to do this all with twisted without using threads? (The main goal is having an extremely minimal set of hooks for the people writing the computational code, and having the computational code run as fast as possible.) Thanks for all of the suggestions and prompt comments! (And for one very "twisted" library;-) Michael.

On Oct 3, 2008, at 5:25 PM, glyph@divmod.com wrote:

...

I know that and am using callFromThread: The reason I am having bad behavior is because the socket is backing up. Once I put in the logic to throw out data, it works fine... but the lack of reactor support concerns me somewhat.

What do you mean by "backing up"? That doesn't really make sense. When a transport "backs up" in Twisted, it just allocates a larger and larger buffer to store the data that is queued to be sent. I thought that the socket buffer was backing up, but now see that write() does buffer. I incorrectly attributed some problems with data arriving out of order and/or getting lost with this... I will have to look deeper into what is causing these problems. I am using a push producer now and the server seems to be sending things fine. As far as I can tell, however, the receiver is sometimes getting the data out of order when data comes in too fast on the client side. I have to look into that and see what the cause is... maybe I am not being careful enough with the threads.

(On the client side, I want the user to have full access to a python interpreter with readline functionality etc. and the ability to plot things with matplotlib using Tkinter. As far as I can see, I can't get readline functionality with a manhole or similar interface controlled by twisted, so again I have to run twisted in a separate thread.)

And what is the "lack of reactor support" you're talking about? Oh, the "lack of support" for writeSomeData by some reactors bothered me and prompted me to use write;-)

...

I don't want the computation to stop, but think I could put an intermediate "push producer" that stops by discarding the data.

I don't *really* understand what you're trying to accomplish here; it soudns to me like you actually want a pull producer that just always sends the latest state in resumeProducing(), assuming it's always changing. Of course the "every 30 minutes" timeframe confuses that somewhat. However, using some kind of producer is definitely the way to go. The point here is that there are two forms of usage: one is when the computation is fast and the data is coming so fast that the client will have to drop some of it. I agree that this should be more of a pull producer, but the other form of usage is where the computation takes about 30 minutes before new data is ready. This should be implemented as a push producer.

...

Is there a simpler way to do this all with twisted without using threads? (The main goal is having an extremely minimal set of hooks for the people writing the computational code, and having the computational code run as fast as possible.)

You could use subprocesses. Valentino Volonghi is working on a convenient process-pool for use with Twisted: <https:// launchpad.net/ampoule/>. I don't think that would actually help you that much though, since you'd still need to have application- specific buffering logic (it's a slightly unusual requirement to throw away intermediary state depending on the buffer saturation).

Thanks for the suggestions and help. Michael.

On 09:24 pm, mforbes@physics.ubc.ca wrote:

Why does writeSomeData not simply raise CONNECTION_LOST as an exception? Checking return values is quite un-pythonic. Is there a deep reason for this?

writeSomeData is an internal API that nobody should really use, so it's not factored for convenience; it's also deep down in the guts of the reactor's innermost loop, so it pays a lot of attention to efficiency. Raising exceptions is really, really slow, at least by the standards of that kind of code.

P.S. I came across this because I was trying to use twisted running in a thread to write data resulting from a long computation that I have not yet turned into a producer. The more conventional "write" method was failing if the socket backed up, and provided no simple way of determining if data was being dropped. My solution was to used writeSomeData, and then have the computation decide to throw out some of the data if it is being produced too rapidly, but I need to know how much has been sent so I can decide what to throw away...

You do know that Twisted APIs are not thread safe, right? You can't call write() from a thread?

On Oct 1, 2008, at 3:44 PM, Jean-Paul Calderone wrote: ...

...

Why does writeSomeData not simply raise CONNECTION_LOST as an exception? Checking return values is quite un-pythonic. Is there a deep reason for this?

The reason is probably that whoever wrote it thought that raising an exception would be enough slower than returning a special value that it would be worth doing something slightly unusual. Compared to returning a value, raising an exception is expensive in CPython, but I doubt there is any surviving benchmark which demonstrates that it makes a difference here.

However, it's a very low-level API and since it isn't intended to be used by application code, it doesn't seem like a very high priority to make it less confusing.

By the way, is there any convention by which "high level" API routines are distinguished from "low level" routines? It is really not that clear from the docs... (writeSomeData comes first in the listing etc.)

...

P.S. I came across this because I was trying to use twisted running in a thread to write data resulting from a long computation that I have not yet turned into a producer. The more conventional "write" method was failing if the socket backed up, and provided no simple way of determining if data was being dropped. My solution was to used writeSomeData, and then have the computation decide to throw out some of the data if it is being produced too rapidly, but I need to know how much has been sent so I can decide what to throw away...

This seems more interesting than the main point of your email. :) You should probably just do the work to turn the computation into a producer. This because:

* writeSomeData is low level and crufty, as you've discovered * Not all reactors have this API, so your program will break based on the reactor selected to run it * producers give you all the same information

Since you mentioned threads, I'll also point out that Twisted APIs are not safe to use from non-reactor threads without reactor.callFromThread. If you aren't using callFromThread, that would explain bad behavior from the write method.

I know that and am using callFromThread: The reason I am having bad behavior is because the socket is backing up. Once I put in the logic to throw out data, it works fine... but the lack of reactor support concerns me somewhat. I don't want the computation to stop, but think I could put an intermediate "push producer" that stops by discarding the data. I am threading because the computation is the main task, and the network stuff is simply a window into the state of the computation. I want the computation to run continually, and it needs to do so in a blocking sort of way, periodically checking and pushing the partial results (like once every 30 minutes or so). I want the calculation code to function on its own, with or without the twisted components and I don't see a simple way of doing this if twisted is in complete control without some type of threading. My current design is to write my calculation as if twisted did not exist, and just push the current results onto a Queue periodically. Then I run twisted in a separate thread with clients that periodically check the queue, pop stuff off and dispatch it to any clients who happen to connect and want the data. Is there a simpler way to do this all with twisted without using threads? (The main goal is having an extremely minimal set of hooks for the people writing the computational code, and having the computational code run as fast as possible.) Thanks for all of the suggestions and prompt comments! (And for one very "twisted" library;-) Michael.

On Oct 3, 2008, at 5:25 PM, glyph@divmod.com wrote:

...

I know that and am using callFromThread: The reason I am having bad behavior is because the socket is backing up. Once I put in the logic to throw out data, it works fine... but the lack of reactor support concerns me somewhat.

What do you mean by "backing up"? That doesn't really make sense. When a transport "backs up" in Twisted, it just allocates a larger and larger buffer to store the data that is queued to be sent. I thought that the socket buffer was backing up, but now see that write() does buffer. I incorrectly attributed some problems with data arriving out of order and/or getting lost with this... I will have to look deeper into what is causing these problems. I am using a push producer now and the server seems to be sending things fine. As far as I can tell, however, the receiver is sometimes getting the data out of order when data comes in too fast on the client side. I have to look into that and see what the cause is... maybe I am not being careful enough with the threads.

(On the client side, I want the user to have full access to a python interpreter with readline functionality etc. and the ability to plot things with matplotlib using Tkinter. As far as I can see, I can't get readline functionality with a manhole or similar interface controlled by twisted, so again I have to run twisted in a separate thread.)

And what is the "lack of reactor support" you're talking about? Oh, the "lack of support" for writeSomeData by some reactors bothered me and prompted me to use write;-)

...

I don't want the computation to stop, but think I could put an intermediate "push producer" that stops by discarding the data.

I don't *really* understand what you're trying to accomplish here; it soudns to me like you actually want a pull producer that just always sends the latest state in resumeProducing(), assuming it's always changing. Of course the "every 30 minutes" timeframe confuses that somewhat. However, using some kind of producer is definitely the way to go. The point here is that there are two forms of usage: one is when the computation is fast and the data is coming so fast that the client will have to drop some of it. I agree that this should be more of a pull producer, but the other form of usage is where the computation takes about 30 minutes before new data is ready. This should be implemented as a push producer.

...

Is there a simpler way to do this all with twisted without using threads? (The main goal is having an extremely minimal set of hooks for the people writing the computational code, and having the computational code run as fast as possible.)

You could use subprocesses. Valentino Volonghi is working on a convenient process-pool for use with Twisted: <https:// launchpad.net/ampoule/>. I don't think that would actually help you that much though, since you'd still need to have application- specific buffering logic (it's a slightly unusual requirement to throw away intermediary state depending on the buffer saturation).

Thanks for the suggestions and help. Michael.