Multiprocessing process termination

[Charles Hixson]

On 12/31/2014 01:18 PM, MRAB wrote:
> On 2014-12-31 19:33, Charles Hixson wrote:
>> In order to allow multiple processes to access a database (currently
>> SQLite) I want to run the process in a separate thread.  Because it will
>> be accessed from multiple processes I intent to use Queues for shifting
>> messages back and forth.  But none of the individuals processes will
>> know when all of the others are through.  It there a way to properly
>> close the database, or does it need to be done from the parent process?
>>
>> What I want to do is detect that a request to shutdown the Process has
>> been received, and then execute a close procedure that cleans things up
>> and shutdown.  Terminate is the obvious procedure to use, but that comes
>> with a warning not to use it if there is an queue attached (and without
>> defining attached).
>> OTOH, since the queue will never be read after the process has been
>> shutdown, perhaps rendering it unusable is the proper thing.
>>
>> Could someone comment on this step of the design?
>>
>> The basic idea is:
>>
>> class handledb(Process):
>>      def __init__(self, q, otherstuff):
>>          self.q  =  q
>>
>>     def run(self, msg):
>>        while (true):
>>            if  self.q.empty():
>>                sleep(someamountoftime)
>>            else:
>>                while (not self.q.empty()):
>>                      read and parse message
>>                      if message says to shutdown: self.shutdown()
>>
>>     def  shutdown(self):
>>           close things down
>>           ?? terminate ??                    <<-- should this be done
>> while q is live?
>>
> The parent process could tell the db process to shutdown (by putting a
> message in the queue) when the child processes have terminated.
> Alternatively, the parent process could tell the db process how many
> child processes there are and then shutdown when it has received that
> many notifications from child processes that they have finished.
>
> BTW, it's better to let the queue read block until a message arrives
> than to keep polling and sleeping if it's empty.
>
Thanks.  This design already depends on the parent process telling the 
db process when to shut down, but this depends on the queue (which is, I 
think, attached to the process) and so I am uncertain about calling 
terminate, as the docs say this can be expected to corrupt the queue. 
I'm hoping that this isn't a problem if the db process is the only 
reader of the queue, and this shouldn't happen until after all other 
writers to the queue have finished.  But the documentation has made me 
uneasy on this point.

A blocking read is better?  OK, that's an easy change, and seems quite 
reasonable.  But its the termination that I was worried about.  (This is 
only one use out of several similar cases, and I'd like to use the same 
pattern throughout, so I'd really rather get it right the first time. 
Particularly as MP programs won't necessarily throw the error predictably.)