Processing large CSV files - how to maximise throughput?
Steven D'Aprano
steve+comp.lang.python at pearwood.info
Thu Oct 24 23:35:18 EDT 2013
On Fri, 25 Oct 2013 02:10:07 +0000, Dave Angel wrote:
>> If I have multiple large CSV files to deal with, and I'm on a
>> multi-core machine, is there anything else I can do to boost
>> throughput?
>
> Start multiple processes. For what you're doing, there's probably no
> point in multithreading.
Since the bottleneck will probably be I/O, reading and writing data from
files, I expect threading actually may help.
--
Steven
More information about the Python-list
mailing list