I've got an IPushProducer written with these 3 methods and it works fine - i.e. resumeProducing() is called, then eventually pauseProducing is called() when the producer produces data too fast; when the client disconnects early, the stopProducing method is called, etc..

  def pauseProducing(self):

  def resumeProducing(self):

  def stopProducing(self):

My problem is, the source of the data (a slow non-relational database) that I send back to the client from within resumeProducing generates data kind of slowly.  As in, it is quite rare that pauseProducing is ever called in my tests.  I feel that the data source generates data too slowly, and have figured out I will block the entire server during the resumeProducing operation (for example, during the first call to resumeProducing, I do not generate any data at all for the first 20 seconds during bootstrapping).

At this point, I am thinking about the following ideas of code changes to prevent blockage of the server:

1) forking off a thread so that I will not block other clients from talking to the server.  I will not have very many clients using this producer, so this would not result in any large numbers of threads which I know can be a problem.  I know how to spawn Python threads and use socket APIs, but it is not clear to me how I would get at the socket descriptor from within the twisted framework (I am using a LineReceiver subclass, which in this case is spawning the IPushProducer).  Are there any examples of spawning off a thread from a LineReceiver class and communicating using blocking calls within said thread?

2) returning from resumeProducing after a few seconds of production, even though I could produce more, so I do not block the server.  I have tested this, it works, and does let other clients get in, but I still feel it is suboptimal, since my data source is so slow, it still blocks the server during each of those few second intervals.

Any recommendations here?  Thank you.