multiprocessing module in async db query

[Sheng]

Hi Philip,

multiprocessing.Queue is used to transfer data between processes, how
it could be helpful for solving my problem? Thanks!

Sheng

On Mar 8, 6:34 pm, Philip Semanchuk <phi... at semanchuk.com> wrote:
> On Mar 8, 2011, at 3:25 PM, Sheng wrote:
>
> > This looks like a tornado problem, but trust me, it is almost all
> > about the mechanism of multiprocessing module.
>
> [snip]
>
> > So the workflow is like this,
>
> > get() --> fork a subprocess to process the query request in
> > async_func() -> when async_func() returns, callback_func uses the
> > return result of async_func as the input argument, and send the query
> > result to the client.
>
> > So the problem is the the query result as the result of sql_command
> > might be too big to store them all in the memory, which in our case is
> > stored in the variable "data". Can I send return from the async method
> > early, say immediately after the query returns with the first result
> > set, then stream the results to the browser. In other words, can
> > async_func somehow notify callback_func to prepare receiving the data
> > before async_func actually returns?
>
> Hi Sheng,
> Have you looked at multiprocessing.Queue objects?
>
> HTH
> Philip