On 18 Aug 2021, at 16:03, Chris Angelico firstname.lastname@example.org wrote:
On Thu, Aug 19, 2021 at 12:52 AM Marc-Andre Lemburg email@example.com wrote:
On 18.08.2021 15:58, Chris Angelico wrote: On Wed, Aug 18, 2021 at 10:37 PM Joao S. O. Bueno firstname.lastname@example.org wrote:
So, It is out of scope of Pythonmultiprocessing, and, as I perceive it, from the stdlib as a whole to be able to allocate specific cores for each subprocess - that is automatically done by the O.S. (and of course, the O.S. having an interface for it, one can write a specific Python library which would allow this granularity, and it could even check core capabilities).
Python does have a way to set processor affinity, so it's entirely possible that this would be possible. Might need external tools though.
There's os.sched_setaffinity(pid, mask) you could use from within a Python task scheduler, if this is managing child processes (you need the right permissions to set the affinity).
Right; I meant that it might require external tools to find out which processors you want to align with.
Or you could use the taskset command available on Linux to fire up a process on a specific CPU core. lscpu gives you more insight into the installed set of available cores.
Yes, those sorts of external tools.
It MAY be possible to learn about processors by reading /proc/cpuinfo, but that'd still be OS-specific (no idea which Unix-like operating systems have that, and certainly Windows doesn't).
And next you find out that you have to understand the NUMA details of your system because the memory attached to the CPUs is not the same speed.
All in all, far easier to just divide the job into far more pieces than you have processors, and then run a pool.
As other already stated using a worker pool solves this problem for you. All you have to do it break your big job into suitable small pieces.
ChrisA _______________________________________________ Python-ideas mailing list -- email@example.com To unsubscribe send an email to firstname.lastname@example.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://email@example.com/message/UQNSUS... Code of Conduct: http://python.org/psf/codeofconduct/