I've got an IPushProducer written with these 3 methods and it works fine
- i.e. resumeProducing() is called, then eventually pauseProducing is
called() when the producer produces data too fast; when the client
disconnects early, the stopProducing method is called, etc..
My problem is, the source of the data (a slow non-relational database)
that I send back to the client from within resumeProducing generates
data kind of slowly. As in, it is quite rare that pauseProducing is
ever called in my tests. I feel that the data source generates data too
slowly, and have figured out I will block the entire server during the
resumeProducing operation (for example, during the first call to
resumeProducing, I do not generate any data at all for the first 20
seconds during bootstrapping).
At this point, I am thinking about the following ideas of code changes
to prevent blockage of the server:
1) forking off a thread so that I will not block other clients from
talking to the server. I will not have very many clients using this
producer, so this would not result in any large numbers of threads which
I know can be a problem. I know how to spawn Python threads and use
socket APIs, but it is not clear to me how I would get at the socket
descriptor from within the twisted framework (I am using a LineReceiver
subclass, which in this case is spawning the IPushProducer). Are there
any examples of spawning off a thread from a LineReceiver class and
communicating using blocking calls within said thread?
2) returning from resumeProducing after a few seconds of production,
even though I could produce more, so I do not block the server. I have
tested this, it works, and does let other clients get in, but I still
feel it is suboptimal, since my data source is so slow, it still blocks
the server during each of those few second intervals.
Any recommendations here? Thank you.