Catching exceptions with multi-processing

Fabien fabien.maussion at gmail.com
Sat Jun 20 05:03:51 EDT 2015


On 06/20/2015 05:14 AM, Cameron Simpson wrote:
> I would keep your core logic Pythonic, raise exceptions. But I would
> wrap each task in something to catch any Exception subclass and report
> back to the queue. Untested example:
>
>   def subwrapper(q, callable, *args, **kwargs):
>     try:
>       q.put( ('COMPLETED', callable(*args, **kwargs)) )
>     except Exception as e:
>       q.put( ('FAILED', e, callable, args, kwargs) )
>
> then dispatch tasks like this:
>
>   pool.map(subwrapper, q, task1, dirs, chunksize=1)
>
> and have a thread (or main program) collect things from the queue for
> logging and other handling. Obviously you might return something more
> sophisticated that my simple tuple above, but I'm sure you get the idea.
>
> Cheers,
> Cameron Simpson

Perfect! Much more elegant and easier to implement on top of my existing 
workflow based on raising exceptions.

thanks to all responses,

Fabien






More information about the Python-list mailing list