More CPUs doen't equal more speed
Bob van der Poel
bob at mellowood.ca
Sun May 26 14:56:00 EDT 2019
On Sun, May 26, 2019 at 11:05 AM Grant Edwards <grant.b.edwards at gmail.com>
> On 2019-05-23, Chris Angelico <rosuav at gmail.com> wrote:
> > On Fri, May 24, 2019 at 5:37 AM Bob van der Poel <bob at mellowood.ca>
> >> I've got a short script that loops though a number of files and
> >> processes them one at a time. I had a bit of time today and figured
> >> I'd rewrite the script to process the files 4 at a time by using 4
> >> different instances of python. My basic loop is:
> >> for i in range(0, len(filelist), CPU_COUNT):
> >> for z in range(i, i+CPU_COUNT):
> >> doit( filelist[z])
> >> With the function doit() calling up the program to do the
> >> lifting. Setting CPU_COUNT to 1 or 5 (I have 6 cores) makes no
> >> difference in total speed. I'm processing about 1200 files and my
> >> total duration is around 2 minutes. No matter how many cores I use
> >> the total is within a 5 second range.
> > Where's the part of the code that actually runs them across multiple
> > CPUs? Also, are you spending your time waiting on the disk, the CPU,
> > IPC, or something else?
> He said he's using N differenct Python instances, and he even provided
> the code that runs in each instance which is obviously processesing
> 1/Nth of the files.
> It's a pretty good bet that I/O is the limiting factor.
> I did say that ... I also retracted it. In my first eg. I was still
running one process at a time ... no multi-task at all. Parallel is up and
running and is very, very fast ... loads my CPUs to 100%
**** Listen to my FREE CD at http://www.mellowood.ca/music/cedars ****
Bob van der Poel ** Wynndel, British Columbia, CANADA **
EMAIL: bob at mellowood.ca
More information about the Python-list