The worker pool approach is probably the way to go, but there is a fair bit of overhead to creating a multiprocessing job. So fewer, larger jobs are faster than many small jobs. 

So you do want to make the jobs as large as you can without wasting CPU time.

-CHB

On Wed, Aug 18, 2021 at 9:09 AM Barry <barry@barrys-emacs.org> wrote:


> On 18 Aug 2021, at 16:03, Chris Angelico <rosuav@gmail.com> wrote:
>
> On Thu, Aug 19, 2021 at 12:52 AM Marc-Andre Lemburg <mal@egenix.com> wrote:
>>
>>> On 18.08.2021 15:58, Chris Angelico wrote:
>>> On Wed, Aug 18, 2021 at 10:37 PM Joao S. O. Bueno <jsbueno@python.org.br> wrote:
>>>>
>>>> So,
>>>> It is out of scope of Pythonmultiprocessing, and, as I perceive it, from
>>>> the stdlib as a whole to be able to allocate specific cores for each subprocess -
>>>> that is automatically done by the O.S. (and of course, the O.S. having an interface
>>>> for it, one can write a specific Python library which would allow this granularity,
>>>> and it could even check core capabilities).
>>>
>>> Python does have a way to set processor affinity, so it's entirely
>>> possible that this would be possible. Might need external tools
>>> though.
>>
>> There's os.sched_setaffinity(pid, mask) you could use from within
>> a Python task scheduler, if this is managing child processes (you need
>> the right permissions to set the affinity).
>
> Right; I meant that it might require external tools to find out which
> processors you want to align with.
>
>> Or you could use the taskset command available on Linux to fire
>> up a process on a specific CPU core. lscpu gives you more insight
>> into the installed set of available cores.
>
> Yes, those sorts of external tools.
>
> It MAY be possible to learn about processors by reading /proc/cpuinfo,
> but that'd still be OS-specific (no idea which Unix-like operating
> systems have that, and certainly Windows doesn't).

And next you find out that you have to understand the NUMA details
of your system because the memory attached to the CPUs is not the same speed.

>
> All in all, far easier to just divide the job into far more pieces
> than you have processors, and then run a pool.

As other already stated using a worker pool solves this problem for you.
All you have to do it break your big job into suitable small pieces.

Barry
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-leave@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UQNSUSHUONT4AO6NJEPEUENQG2AINAMO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-leave@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/62AXMS62J2H7TBHANIXZTTS2RJPUZZ5Z/
Code of Conduct: http://python.org/psf/codeofconduct/
--
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython