PEP 255: Simple Generators

[Andrew Dalke]

Glyph Lefkowitz wrote:
>They do not fill my needs, or the needs of anyone working with simulation
>code, network dispatching, or massively parallel computing

It's been a while since I did massively parallel computing - back
when PVM and 128 connected machines was the hot thing.  But
I still remember a few things about it and I don't see how Python,
even with stackless, can handle many of the concerns we had:

  My code was distributed memory (network-of-workstation model).
  How would Python handle that?  Eg, how would it know which
  nodes to use and how to start up on those machines?

  Suppose I'm using a machine with special shared memory hardware
  to talk between different nodes.  How do I put Python objects
  into shared memory?

  Is there integration with MPI?

  Can multiple programs interact?  (Eg, our molecular dynamics code
  started up another PVM job to do fast electrostatics calculations.
  Both jobs were distributed on the same set of nodes and simply
  used PVM calls to talk to each other.)

  If they can interact, do they both need to be in Python?  If
  not, what's the serialization method?

Or are you limiting things to a single job on a single multiprocessor
node and where all memory is accessible to all the processors?


BTW, because I'm used to distributed memory machines, I write
my multiple processor code to work on threads through Queues.
I've needed something different only once and was considering
Stackless.  This Simple Generator PEP would have provided all
I needed.

(My code was something like

  def find_common_subsets(graph_A, graph_B, callback):
     loop:
       loop:
         common region identified:
            callback(subset)

which I wanted to convert into something like

  for subset in find_common_subsets2(graph_A, graph_B):
    print subset

The 'yield' statement works admirably for that, and for some
other tasks (like XML processing of records) where I want
to turn callbacks into iterators.
)

So I would like it (or something similar) even if it doesn't
solve all of my MP problems.

                    Andrew
                    dalke at acm.org