[New-bugs-announce] [issue36399] multiprocessing: Queue does not work in virtualenv but works fine in main interpreter

Bernt Røskar Brenna report at bugs.python.org
Fri Mar 22 04:51:16 EDT 2019


New submission from Bernt Røskar Brenna <bernt.brenna at gmail.com>:

When executing mp_queue_example.py in the system interpreter:

D:\dev\eva_v_next>d:\python372\python.exe mp_queue_example.py
main, pid=892, executable=d:\python372\python.exe
process_worker, pid=12848, executable=d:\python372\python.exe
submitting...
submitting 0
submitting 1
submitting 2
Executing job 0
Executing job 1
Executing job 2

When executing mp_queue_example.py in a virtualenv:

D:\dev\eva_v_next>VENV\Scripts\python.exe mp_queue_example.py
main, pid=25888, executable=D:\dev\eva_v_next\VENV\Scripts\python.exe
process_worker, pid=28144, executable=D:\dev\eva_v_next\VENV\Scripts\python.exe
submitting...
submitting 0
submitting 1
submitting 2

## Here it hangs, Ctrl-C gives this:

Process Process-1:
Traceback (most recent call last):
  File "D:\dev\eva_v_next\mp_queue_example.py", line 13, in process_worker
    inp = input_q.get_nowait()
  File "D:\python372\lib\multiprocessing\queues.py", line 126, in get_nowait
    return self.get(False)
  File "D:\python372\lib\multiprocessing\queues.py", line 100, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\python372\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "D:\python372\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "D:\dev\eva_v_next\mp_queue_example.py", line 13, in process_worker
    inp = input_q.get_nowait()
KeyboardInterrupt
Traceback (most recent call last):
  File "mp_queue_example.py", line 43, in <module>
    main_process_experiment()
  File "mp_queue_example.py", line 39, in main_process_experiment
    p.join()
  File "D:\python372\lib\multiprocessing\process.py", line 140, in join
    res = self._popen.wait(timeout)
  File "D:\python372\lib\multiprocessing\popen_spawn_win32.py", line 80, in wait
    res = _winapi.WaitForSingleObject(int(self._handle), msecs)
KeyboardInterrupt


---

mp_queue_example.py:

import multiprocessing as mp
from queue import Empty
import os
import sys
import time


def process_worker(input_q: mp.Queue):
    print(f'process_worker, pid={os.getpid()}, executable={sys.executable}')
    while True:
        try:
            inp = input_q.get_nowait()
            if inp == 'STOP':
                break
            execute_job(inp)
        except Empty:
            pass


def execute_job(input_args):
    print(f'Executing job {input_args}')


def main_process_experiment():
    print(f"main, pid={os.getpid()}, executable={sys.executable}")
    input_q = mp.Queue()

    p = mp.Process(target=process_worker, args=(input_q, ))
    p.start()
    time.sleep(0.5)

    print('submitting...')
    for i in range(3):
        print(f'submitting {i}')
        input_q.put(i)
    input_q.put('STOP')

    p.join()


if __name__ == '__main__':
    main_process_experiment()

----------
components: Library (Lib)
files: mp_queue_example.py
messages: 338591
nosy: Bernt.Røskar.Brenna
priority: normal
severity: normal
status: open
title: multiprocessing: Queue does not work in virtualenv but works fine in main interpreter
type: behavior
versions: Python 3.7
Added file: https://bugs.python.org/file48228/mp_queue_example.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36399>
_______________________________________


More information about the New-bugs-announce mailing list