On Tue, Sep 5, 2017 at 12:59 PM, Nathaniel Smith firstname.lastname@example.org wrote:
On Mon, Sep 4, 2017 at 8:59 PM, Giampaolo Rodola' email@example.com wrote:
Recently os.cpu_count() on Windows has been fixed in order to take
groups into account and return the number of all available CPUs: http://bugs.python.org/issue30581 This made me realize that os.cpu_count() does not return the number of *usable* CPUs, which could possibly represent a better default value for things like multiprocessing and process pools. It is currently possible to retrieve this info on UNIX with len(os.sched_getaffinity(0)) which takes CPU affinity and (I think) Linux cgroups into account, but it's not possible to do the same on Windows
provides this value via GetActiveProcessorCount() API. As such I am planning to implement this in psutil but would like to know
python-ideas feels about adding a new os.usable_cpu_count() function (or having os.cpu_count(usable=True)).
This was discussed in https://bugs.python.org/issue23530
It looks like the resolution at that time was:
- os.cpu_count() should *not* report the number of CPUs accessible to
the current process, but rather continue to report the number of CPUs that exist in the system (whatever that means in these days of virtualization... e.g. if you use KVM to set up a virtual machine with limited CPUs, then that changes os.cpu_count, but if you do it with docker then that doesn't).
- multiprocessing and similar should continue to treat os.cpu_count()
as if it returned the number of CPUs accessible to the current process.
Possibly some lines got crossed there...
-- Nathaniel J. Smith -- https://vorpus.org
I agree current os.cpu_count() behavior should remain unchanged. Point is multiprocessing & similar are currently not taking CPU affinity and Linux cgroups into account, spawning more processes than necessary.