Hi !

I have a function which returns a deferred but will take many time to really process an item. This function will be feed with a lot of items. For example, the function could be called 1000 times per second and will take 30 minutes to really process each item. However, I don't know the delay. The function could also process the items in real time.

When the function takes 30 minutes to really process the item, the reactor will keep a lot of deferreds and huge portions of memory. Therefore, I would like to store myself those items (in a database for example) and wait the function to be ready to process all the items (because the function knows when it will be ready).

I could store the items directly in the database and asks the function an unique deferred whose callback will be handled when the function is ready to process my items. But since, the function could be a function that process items in real time (and more faster than a database), I would like to use this "fallback" method only if the reactor has already a lot of deferred for this function. Ideally, in this case, current deferred should be converted to database.

In short, I would like to be able to store/flush deferreds in database when I have a lot of them and keep them in memory if I have only a few of them.

For example, a naive implementation could be to implement subclass of Deferred and each Deferred instance can store itself to disk if a lot of instances of this subclass already exists. This is not exactly what I want since I need a mechanism that will also flush all instances to disk instead of keeping them in memory. So an instance should be able to replace a lot of instances and recreate them when the function becomes ready.

Does such a thing already exists? If it helps, I can guarantee that function will process items in FIFO order.

Thanks.