[Tutor] running multiple concurrent processes

[Oscar Benjamin]

On 3 November 2012 20:20, richard kappler <richkappler at gmail.com> wrote:
> To all, especially Dave, Oscar and Ramit, thanks for the discussion and
> help. Tino, as soon as I have something to put up, I will gladly put it up
> on Github. At the moment I only have a few snippets I've written in trying
> to learn the various bits everyone has been helping me with.
>
> Just for the record, I did use subprocess quite successfully in one of my
> preliminary programs earlier this summer as I started to learn python, used
> it to send out the text from the old chatbot to Festival TTS, so I am
> comfortable with the idea of running multiple processes from python to the
> command line. I'm not quite sure that's the way to go with the sensors and
> other bits though.

You may be right. The problem is that this is not a situation where
anyone can really tell you the definitive way of doing things. There
are many ways to do what you're doing and which is best depends on
details that are currently unknown (by you or by the rest of us).

>
> I'm wondering if I'm asking the wrong question. I have read and re-read your
> posts, and have read and re-read the documentation on multi-processing and
> threading. I have also read several tutorials on both. I feel that I
> understand the concepts a wee bit better, though not enough to come up with
> some workable processing or threading code. I have, however, still not
> figured out an answer to my question, which leads me to believe I have not
> asked the question properly. The alternative is that I'm just too damned
> dense to wrap my arms around this and I both think and hope that is not the
> case, so I will try to do a better job of articulating what I'm looking for.
> (Again, I suspect its in multiprocessing and threading, but I can't quite
> put my finger on a good approach).

The example below shows how to use multiprocessing to run two
different operations concurrently. Each operation is run in its own
process.

#!/usr/bin/env python

from multiprocessing import Process
from time import sleep

def update_brain():
    while True:
        sleep(1)
        print 'thinking...'

def chat_with_friends():
    while True:
        sleep(.3)
        print 'chatting...'

p1 = Process(target=update_brain)
p2 = Process(target=chat_with_friends)

p1.start()
p2.start()


>
> SO... while the bot program is running, I would like it to be continuously
> cognizant of the sensor data and any changes in that data. Let's distill it
> down to a single sensor to simplify the discussion. If I have a temperature
> sensor feeding data to the arduino, which feeds that data through serial to
> the Python program (let's call that pyBrain) that "is" the bot, (here's the
> meat of the question:) how do I make the pyBrain constantly aware of the
> temperature while doing other stuff?

It depends on how pyserial works. This sounds like the kind of
situation where a separate thread would be used. I say a thread rather
than a process because this is most likely an IO-bound operation. This
means that it consumes very little CPU and spends most of its time
waiting for the serial port to produce data.

I've never used pyserial but I imagine that it has a function called
something like get_data that returns data read from the device. I also
imagine that this function "blocks". That means that it causes
execution to pause until data is ready and then returns the data. It
is precisely this blocking that makes people want to use threads.
While one thread blocks execution of the others can resume so that
waiting for data in one operation does not cause every other operation
to pause.

The downside of using multiple threads or processes though is that it
makes it harder to share data between the different operations. Here's
an extension of the above that shows how to share an integer between
the two processes:

#!/usr/bin/env python

from multiprocessing import Process, Value
from time import sleep
from random import randint

def update_brain():
    global how_many_spams
    while True:
        sleep(1)
        n = randint(1, 5)
        print 'thinking...', n
        how_many_spams.value = n

def chat_with_friends():
    global how_many_spams
    while True:
        sleep(.3)
        print 'chatting:' + how_many_spams.value * ' spam'

how_many_spams = Value('i', 1)

p1 = Process(target=update_brain)
p2 = Process(target=chat_with_friends)

p1.start()
p2.start()

>
> Befuddled and bewildered while learning at an alarming rate, Richard
>

I suggest that you just write the application and divide pieces off
into threads/processes later if needed. Avoid sharing data between
different operations except when strictly necessary (then breaking out
one part into a separate process or thread won't be so painful).

I'll rephrase the rule of thumb from before:

Use separate threads to separate out operations that involve
blocking-IO calls that would otherwise cause the whole application to
suspend.

Use separate processes to separate out CPU-heavy operations that block
the other operations by consuming too much CPU time. On a multicore
machine the separate processes will hopefully run on different cores
so that you'll be making better use of the available CPU capacity.


Oscar