[scikit-learn] Scikit Learn in a Cray computer

[Mauricio Reis]

My laptop has Intel I7 processor with 4 cores. When I run the program on 
Windows 10, the "joblib.cpu_count()" routine returns "4". In these 
cases, the same test I did on the Cray computer caused a 10% increase in 
the processing time of the DBScan routine when I used the "n_jobs = 4" 
parameter compared to the processing time of that routine without this 
parameter. Do you know what is the cause of the longer processing time 
when I use "n_jobs = 4" on my laptop?

---
Ats.,
Mauricio Reis

Em 28/06/2019 06:29, Brown J.B. via scikit-learn escreveu:
>> where you can see "ncpus = 1" (I still do not know why 4 lines were
>> printed -
>> 
>> (total of 40 nodes) and each node has 1 CPU and 1 GPU!
> 
>> #PBS -l select=1:ncpus=8:mpiprocs=8
>> aprun -n 4 p.sh ./ncpus.py
> 
> You can request 8 CPUs from a job scheduler, but if each node the
> script runs on contains only one virtual/physical core, then
> cpu_count() will return 1.
> If that CPU supports multi-threading, you would typically get 2.
> 
> For example, on my workstation:
> `--> egrep "processor|model name|core id" /proc/cpuinfo
> processor : 0
> model name : Intel(R) Core(TM) i3-4160 CPU @ 3.60GHz
> core id : 0
> processor : 1
> model name : Intel(R) Core(TM) i3-4160 CPU @ 3.60GHz
> core id : 1
> processor : 2
> model name : Intel(R) Core(TM) i3-4160 CPU @ 3.60GHz
> core id : 0
> processor : 3
> model name : Intel(R) Core(TM) i3-4160 CPU @ 3.60GHz
> core id : 1
> `--> python3 -c "from sklearn.externals import joblib;
> print(joblib.cpu_count())"
> 4
> 
> It seems that in this situation, if you're wanting to parallelize
> *independent* sklearn calculations (e.g., changing dataset or random
> seed), you'll ask for the MPI by PBS processes like you have, but
> you'll need to place the sklearn computations in a function and then
> take care of distributing that function call across the MPI processes.
> 
> Then again, if the runs are independent, it's a lot easier to write a
> for loop in a shell script that changes the dataset/seed and submits
> it to the job scheduler to let the job handler take care of the
> parallel distribution.
> (I do this when performing 10+ independent runs of sklearn modeling,
> where models use multiple threads during calculations; in my case,
> SLURM then takes care of finding the available nodes to distribute the
> work to.)
> 
> Hope this helps.
> J.B.
> _______________________________________________
> scikit-learn mailing list
> scikit-learn at python.org
> https://mail.python.org/mailman/listinfo/scikit-learn