21 Jan
2014
21 Jan
'14
8:27 p.m.
On Jan 21, 2014, at 4:20, Andrew Barnert
And this is very easy to solve: run the downloads on a thread pool, and as each one finishes, kick its post processing off to a process pool.
Wait, that's stupid. Even simpler: just use a flat process pool of 2N for everything (or whatever multiplier is appropriate for your load--although often a downloader doesn't want to do more than about 4-12 simultaneous downloads, which is already below 2N on most modern computers...).