Hello, before posting to python-dev I thought is is the best to discuss this here. And I assume that someone else had the same idea then me before. Maybe you can point me to the relevant discussion/ticket. I read about Intels hybrid CPUs. It means there are multiple cores e.g. 8 high-speed cores and 8 low-speed (but more energy efficient) cores combined in one CPU. In my use cases I do parallelize with Pythons multiprocessing package to work on millions of rows on pandas.DataFrame objects. This are task that are not vecotrizable. I simple cut the DataFrame horizontal in pieces (numbered by the available cores). But when the cores are different in there "speed" I need to know that. e.g. with a 16 core CPU where half of the cores low/slow and every core has 1 million rows to work on. The 8 high speed cores are finishing earlier and just waiting untill the slow cores are finished. It would be more efficient if the 8 high speed cores each would work on 1,3 million rows and the low speed cores each on 0,7 million rows. It is not perfect but better. I know that they will not finish all at the same timepoint. But their end time will be closer together. But to do this I need to know the type of the cores. Am I wrong? Are there any plans in the Python development taking this into account? Kind Christian
participants (8)
-
Barry
-
c.buhtz@posteo.jp
-
Chris Angelico
-
Christopher Barker
-
Joao S. O. Bueno
-
Marc-Andre Lemburg
-
Stephen J. Turnbull
-
Thomas Grainger