threading : make stop the caller

[Terry Reedy]

On 6/19/2011 12:03 PM, Chris Angelico wrote:
> On Mon, Jun 20, 2011 at 1:39 AM, Laurent Claessens<moky.math at gmail.com>  wrote:
>> My problem is that when FileToCopyTask raises an error, the program does not
>> stop.
>> In fact when the error is Disk Full, I want to stop the whole program
>> because I know that the next task will fail too.
>
> If you're starting a thread for every file you're copying, you're
> starting a huge number of threads that probably will just end up
> fighting over the disk. To get a reasonably efficient early-abort, I'd
> recommend having a fixed pool of worker threads (say, ten of them),
> and have each thread (a) check if the early-abort flag is set, and
> then (b) start copying the next file in queue. Once the queue's empty
> or the early-abort flag is set, all ten threads will terminate when
> they finish their current transfers.
>
> (The ten threads figure is arbitrary. Optimum value for performance
> will come by adjusting this.)

I an not convinced that much of anything is gained by having multiple 
copying threads. It certainly adds complication -- hence this thread. I 
would just use shutil to copy files or directories of files.

Note that copytree has an option to pass a filter function:
"If ignore is given, it must be a callable that will receive as its 
arguments the directory being visited by copytree(), and a list of its 
contents, as returned by os.listdir(). Since copytree() is called 
recursively, the ignore callable will be called once for each directory 
that is copied. The callable must return a sequence of directory and 
file names relative to the current directory (i.e. a subset of the items 
in its second argument); these names will then be ignored in the copy 
process. ignore_patterns() can be used to create such a callable that 
ignores names based on glob-style patterns."

-- 
Terry Jan Reedy