Long running process - how to speed up?
Dan Stromberg
drsalists at gmail.com
Sat Feb 19 11:04:25 EST 2022
On Sat, Feb 19, 2022 at 3:29 AM Shaozhong SHI <shishaozhong at gmail.com>
wrote:
> I have a cvs file of 932956 row and have to have time.sleep in a Python
> script. It takes a long time to process.
>
> How can I speed up the processing? Can I do multi-processing?
>
How are you doing it right now?
Are you using the csv module?
You might be able to use the GNU "split" command as a prelude to using the
csv module in combination with multiprocessing. GNU split comes with
Linuxes, but I'm sure you can get it for Windows. MacOS comes with a
rather less powerful "split" command, but it still might work for you.
You also could try Pypy3.
HTH.
More information about the Python-list
mailing list