[Python-ideas] solving multi-core Python

[Wes Turner]

On Thu, Jun 25, 2015 at 10:25 AM, Sturla Molden <sturla.molden at gmail.com>
wrote:

> On 25/06/15 16:31, Nick Coghlan wrote:
>
>  3. The potential for collisions between objects means it isn't an
>> embarrassingly parallel problem where the different computational
>> threads can entirely ignore the existence of the other threads
>>
>
> Well, you can have a loop that updates all particles, e.g. by calling a
> coroutine associated with each particle, and then this loop is an
> embarrassingly parallel problem. You don't need to associate each particle
> with its own thread.
>
> It is bad to teach students to use one thread per particle anyway.
> Suddenly they write a system that have thousands of threads.


Understood that this is merely an example re: threading, but
BSP seems to be the higher-level algorithm for iterative graphs with
topology:

* https://en.wikipedia.org/wiki/Bulk_synchronous_parallel
*
http://googleresearch.blogspot.com/2009/06/large-scale-graph-computing-at-google.html
* https://giraph.apache.org/
* https://spark.apache.org/docs/latest/
  *
https://spark.apache.org/docs/latest/graphx-programming-guide.html#pregel-api
(BSP)
  *
https://spark.apache.org/news/spark-wins-daytona-gray-sort-100tb-benchmark.html
  * https://spark.apache.org/docs/latest/api/python/ (no graphx BSP yet,
unfortunately)
* https://github.com/xslogic/phoebus (Erlang, HDFS, Thrift)
* https://github.com/mnielsen/Pregel/blob/master/pregel.py (Python)

Intra-machine optimization could also be useful.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150625/74036ab1/attachment-0001.html>