[Chicago] Popen/subprocess/Poll Question.

[Brian Ray]

On Thu, Jul 22, 2010 at 9:55 AM, Samir Faci <samir at esamir.com> wrote:

> I'm sure this is something fairly silly I'm overlooking, but I've been
> spending too much time on this... so I figured I should ask the powers
> that be.
>
>
This is complicated stuff. Do *not* feel bad.

I was working on something similar where I used a non-blocking Queue Server
like this one


http://www.google.com/codesearch/p?hl=en#EvwmCkrYNSE/src/queue_server.py&l=34&d=0

However, I modified the code so that the worker will periodically post back
some results to the server so the log can be displayed. You can run
Subprocess in the worker then post stuff back to the Queue.

The important elements here is how the Queue() waits by default.  Also an
important to know which *thing* your talking about when your posting
anything to the dispatching Queue server.  I changed the way the Queue and
the id handing works in my own code.

Also, I changed the way stopping stuff works. I wanted to be able to stop on
demand and not get a ghost server and make works end gracefully.  That was a
bit tricky because the code here suggests serve_forever.   I put a stop()
method on the workers and the server.  So in the Workers I sleep() and check
_stop periodically.  Likewise, when I am looking for Queue objects, put
those in loop and set the timeout Queue.get(True,10)  so that it does not
block.  Likewise, you will want to make one final call to the server to
release the socket (it may fail, which is fine). If things really get hung
on the worker or the server (or if I get impatient) I do a force stop with
psutil.

Oh, the other change I made is that I ended up using a WorkerPool Thread
that actually has a collection of Workers ready to do *something*.  It just
starts a bunch of workers that are watching polling waiting for the command
to start actually do something from the Server.  It has its own stop() which
just calls stop on all the worker threads.

Hope this helps,

-- Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/chicago/attachments/20100722/994584df/attachment.html>