python notifying calling script that multiprocessing tasks have finished at lower level scripts
daiyueweng at gmail.com
Fri Apr 13 12:08:33 EDT 2018
Hi, I have a master script that executes two sequences (lists) of child
scripts, i.e. script_1 to script_3, and script_4 to_script_6 (the structure
is attached as a png file).
The execution is sequential, e.g. running script_1, then 2 then 3.
After executing the 1st sequence (script_1 to 3), master will execute the
2nd sequence (script_4 to 6).
Each child script will be calling a multiprocessing function to process a
Master script is like,
for seq in seqs_to_launch:
for script in seq:
Each child script is like,
def execute(self, data):
# some data processing
The multiprocessing function is like,
def multi_process_task(tasks, task_name):
cores_to_use = how_many_core()
handler = task_handling(task_name) # task handling class
task_blocks = slice_list(tasks, cores_to_use)
for block in task_blocks:
# spawn processes for each row block assigned to every cpu core
p = multiprocessing.Process(target=handler.execute, args=(block,))
I like to know how to notify the master when the 2nd sequence is finished
that all multiprocess tasks (seq 1 and 2) are completed, and then maybe
close the multiprocessing.
More information about the Python-list