[Twisted-Python] Transmit queue - how would I go at it?
Hi,
I am working on an application where clients request files, and since each file is very large, the server ACKs the requests and then queues them to be served later, one at a time. I am relatively new to Twisted and was wondering what would be the best way to approach this.
Here is what I came up with so far:
1) Use Deferred callbacks, and string together callbacks to a "file sender" function - once a file is sent, the function is called back with a pointer to the next file in line. If I do so - is there some automatic garbage collection of the callbacks that have already been handled? (there may eventually be hundreds of files to be sent) In this approach, would there be a natural way to check if a new request is actually a repeat of a previous request already queued?
2) Use some type of queue - where each new file request is queued, and some single function handles them one at at time (I guess with asynchronous callbacks to itself every time the transport layer is free to send another file and the queue is not empty). Is there some queue structure like this in Twisted? I saw something called a DefferedQueue but could not find many examples or documentation about it and about its usage.
I mainly want to understand, from the "Twisted mindset" which would be the the right way to go? Also, say in the future I would want to implement priority queuing and handling of the requests rather than a FIFO - how would I do that in Twisted?
Any thoughts and/or code examples would be greatly appreciated Thanks! Nadav
On Fri, 2007-09-14 at 00:58 -0400, Nadav Aharony wrote:
Hi,
I am working on an application where clients request files, and since each file is very large, the server ACKs the requests and then queues them to be served later, one at a time. I am relatively new to Twisted and was wondering what would be the best way to approach this.
Hi Nadav,
I had a similar requirement with my application. You must offer an asynchronous interface while dealing with each request synchronously. I think that you should push your requests to a twisted.internet.defer.DeferredQueue and process one at a time with the aid of twisted.internet.defer.DeferredLock.
If we consider that each of your petitions has an associated deferred that will be called back when its finished: (I haven't run this)
from twisted.internet import defer
class OneAtATime(object): def __init__(self): super(OneAtATime, self).__init__() self.queue = defer.DeferredQueue() self.lock = defer.DeferredLock() self.check_queue()
def check_queue(self): self.queue.get().addCallback(self._process_one)
def _process_one(self, petition):
def acquire_cb(_): # do whatever with your petition # make sure that when you're done you call # self.lock.release() and self.check_queue() # on a state machine, this would be while transitioning # to the "idle" state pass
self.lock.acquire().addCallback(acquire_cb)
def queue_petition(self, petition): self.queue.put(petition) return petition.deferred
Regards, Pablo
Thanks Pablo! this looks pretty clear and neat, I'll try it out.
Just one newbie question - is the "_" in "acquire_cb(_)" just a placeholder for my own parameters, or does it have an actual meaning? I've never seen this done in python before, and couldn't find other examples for it online...
Cheers, Nadav
Pablo Martí wrote:
On Fri, 2007-09-14 at 00:58 -0400, Nadav Aharony wrote:
Hi,
I am working on an application where clients request files, and since each file is very large, the server ACKs the requests and then queues them to be served later, one at a time. I am relatively new to Twisted and was wondering what would be the best way to approach this.
Hi Nadav,
I had a similar requirement with my application. You must offer an asynchronous interface while dealing with each request synchronously. I think that you should push your requests to a twisted.internet.defer.DeferredQueue and process one at a time with the aid of twisted.internet.defer.DeferredLock.
If we consider that each of your petitions has an associated deferred that will be called back when its finished: (I haven't run this)
from twisted.internet import defer
class OneAtATime(object): def __init__(self): super(OneAtATime, self).__init__() self.queue = defer.DeferredQueue() self.lock = defer.DeferredLock() self.check_queue()
def check_queue(self): self.queue.get().addCallback(self._process_one) def _process_one(self, petition): def acquire_cb(_): # do whatever with your petition # make sure that when you're done you call # self.lock.release() and self.check_queue() # on a state machine, this would be while transitioning # to the "idle" state pass self.lock.acquire().addCallback(acquire_cb) def queue_petition(self, petition): self.queue.put(petition) return petition.deferredRegards, Pablo
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Sat, 2007-09-15 at 23:36 -0400, Nadav Aharony wrote:
Thanks Pablo! this looks pretty clear and neat, I'll try it out.
Just one newbie question - is the "_" in "acquire_cb(_)" just a placeholder for my own parameters, or does it have an actual meaning? I've never seen this done in python before, and couldn't find other examples for it online...
It is just a convention for cases where you're not interested in the argument ( DeferredLock.release() returns a reference of the lock )
Regards, Pablo
Pablo Martí wrote:
On Fri, 2007-09-14 at 00:58 -0400, Nadav Aharony wrote:
Hi,
I am working on an application where clients request files, and since each file is very large, the server ACKs the requests and then queues them to be served later, one at a time. I am relatively new to Twisted and was wondering what would be the best way to approach this.
Hi Nadav,
I had a similar requirement with my application. You must offer an asynchronous interface while dealing with each request synchronously. I think that you should push your requests to a twisted.internet.defer.DeferredQueue and process one at a time with the aid of twisted.internet.defer.DeferredLock.
If we consider that each of your petitions has an associated deferred that will be called back when its finished: (I haven't run this)
from twisted.internet import defer
class OneAtATime(object): def __init__(self): super(OneAtATime, self).__init__() self.queue = defer.DeferredQueue() self.lock = defer.DeferredLock() self.check_queue()
def check_queue(self): self.queue.get().addCallback(self._process_one) def _process_one(self, petition): def acquire_cb(_): # do whatever with your petition # make sure that when you're done you call # self.lock.release() and self.check_queue() # on a state machine, this would be while transitioning # to the "idle" state pass self.lock.acquire().addCallback(acquire_cb) def queue_petition(self, petition): self.queue.put(petition) return petition.deferredRegards, Pablo
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
http://foss.eepatents.com/AsynQueue
Nadav Aharony wrote:
Hi,
I am working on an application where clients request files, and since each file is very large, the server ACKs the requests and then queues them to be served later, one at a time. I am relatively new to Twisted and was wondering what would be the best way to approach this.
Thanks Ed!
This could be relevant to several places in the code I'm working on.
Regards, Nadav
Ed Suominen wrote:
http://foss.eepatents.com/AsynQueue
Nadav Aharony wrote:
Hi,
I am working on an application where clients request files, and since each file is very large, the server ACKs the requests and then queues them to be served later, one at a time. I am relatively new to Twisted and was wondering what would be the best way to approach this.
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (3)
-
Ed Suominen -
Nadav Aharony -
Pablo Martí