On Jan 21, 2014, at 4:20, Andrew Barnert <abarnert@yahoo.com> wrote:

And this is very easy to solve: run the downloads on a thread pool, and as each one finishes, kick its post processing off to a process pool.

Wait, that's stupid. Even simpler: just use a flat process pool of 2N for everything (or whatever multiplier is appropriate for your load--although often a downloader doesn't want to do more than about 4-12 simultaneous downloads, which is already below 2N on most modern computers...).