[Tutor] Multiprocessing

[Stephen M Smith]

I added your suggested line of code (in first test after the print statement
and then both before and after)  and get the same result. I also shut my
system down and restarted it and pycharm to make sure everything was as
clean as possible.Thanks.

 

Code:

 

import concurrent.futures

import sys

 

from multiprocessing import Process, freeze_support

 

import time

sys.stdout.flush()

print("starting up")

sys.stdout.flush()

def do_something(seconds):

    print(f'Sleeping {seconds} second(s)')

    time.sleep(1)

    return f"done sleeping...{seconds} seconds"

 

 

if __name__ == '__main__':

    print("about to go")

    freeze_support()

    # Process(target=do_something(1)).start

 

    with concurrent.futures.ProcessPoolExecutor() as executor:

        # f1 = executor.submit(do_something,1)

        secs = [10,9,8,7,6,5,4,3,2,1]

        results = [executor.submit(do_something, sec) for sec in secs]

        for f in concurrent.futures.as_completed(results):

            print(f.result())

 

Result:

 

starting up

about to go

starting up

starting up

starting up

starting up

Sleeping 10 second(s)

Sleeping 9 second(s)

starting up

Sleeping 8 second(s)

starting up

Sleeping 7 second(s)

starting up

starting up

Sleeping 6 second(s)

starting up

Sleeping 5 second(s)

starting up

Sleeping 4 second(s)

starting up

Sleeping 3 second(s)

Sleeping 2 second(s)

starting up

Sleeping 1 second(s)

starting up

starting up

starting up

starting up

done sleeping...10 seconds

done sleeping...9 seconds

done sleeping...8 seconds

done sleeping...6 seconds

done sleeping...7 seconds

done sleeping...1 seconds

done sleeping...4 seconds

done sleeping...2 seconds

done sleeping...3 seconds

done sleeping...5 seconds

 

Process finished with exit code 0

 

 

 

-----Original Message-----

From: Cameron Simpson <cs at cskk.id.au <mailto:cs at cskk.id.au> > 

Sent: Tuesday, September 22, 2020 12:03 AM

To: Stephen M Smith <stephen.m.smith at comcast.net
<mailto:stephen.m.smith at comcast.net> >

Cc: tutor at python.org <mailto:tutor at python.org> 

Subject: Re: [Tutor] Multiprocessing

 

On 21Sep2020 21:19, Stephen M Smith <stephen.m.smith at comcast.net
<mailto:stephen.m.smith at comcast.net> > wrote:

>I am trying to figure out how multiprocessing works for an application 

>I have.  My methodology is to vigorously read everything I can find on 

>the subject including examples. I have found an example and modified it a
bit.

>Here is the code and it works:

[...]

>print("starting up")

> 

>def do_something(seconds):

>    print(f'Sleeping {seconds} second(s)')

>    time.sleep(1)

>    return f"done sleeping...{seconds} seconds"

> 

> 

>if __name__ == '__main__':

>    print("about to go")

>    freeze_support()

>    # Process(target=do_something(1)).start

>    with concurrent.futures.ProcessPoolExecutor() as executor:

>        # f1 = executor.submit(do_something,1)

>        secs = [10,9,8,7,6,5,4,3,2,1]

>        results = [executor.submit(do_something, sec) for sec in secs]

>        for f in concurrent.futures.as_completed(results):

>            print(f.result())

> 

>What I don't get is the output, also pasted below. As you will see the 

>"starting up" message is displayed 17 times. I cannot begin to 

>understand how that statement is executed more than once and 17 times 

>makes no sense either. Thanks in advance for any insight you can provide.

 

Are you sure the above text is _exactly_ what you ran to get that output?
Let me explain why this question:

 

I suspect that the sys.stdout buffer has not been flushed. If
ProcessPoolExecutor works using the fork() OS call, each child is a clone of
the parent process, including its in-memory state.

 

Since the sys.stdout buffer contains "starting up\n", each child process has
such a buffer, which goes to the output when sys.stdout flushes its buffer.

 

My concern about the code is that I would also expect to see the "about to
go" output also replicated. And I do not.

 

I would also not expect this behaviour if the output is going to a tty
instead of a file, because sys.stdout is line buffered going to a tty, so
I'd expect _interactively_ this to not happen to you.

 

However, if stdout is going to a file, it will normally be block buffered,
which means the buffer is not actually written out until the buffer fills,
or until stdout is shutdown when the programme (or child

programme) finishes.

 

So when block buffered, each child will have a copy of the unflushed buffer
waiting to go out, so you get a copy from the main programme and once for
each child.

 

Try adding:

 

    sys.stdout.flush()

 

after the print statements before you start the ProcessPoolExecutor and see
if the output changes.

 

Cheers,

Cameron Simpson <cs at cskk.id.au <mailto:cs at cskk.id.au> >