1. 是否可以批量开启子进程
multiprocessing.Process 无法批量开启子进程
multiprocessing.Pool可以批量开启子进程
- p = Pool(2)
- p.map() /
- p.apply_sync()
- p.close()
- p.join()
- #或者用with方式
- with Pool( 2) as p:
- p.map() / p.apply_sync()
2. 进程间通信
multiprocessing.Process 可以直接用multiprocesssing.Queue等进行通信,可以参考下面的代码
- from multiprocessing import Process,Queue
- import os
- def run_proc(name,q):
- print( 'Run Child process %s (%s)' % (name, os.getpid()))
- q.put( 'value')
- def run_proce(name,q):
- print( 'Run Child process %s (%s)' % (name, os.getpid()))
- print(q.get())
- if __name__ == '__main__':
- print( 'Parent process %s.' % os.getpid())
- q = Queue()
- p = Process(target=run_proc,args=( 'test',q))
- p2 = Process(target=run_proce,args=( 'test2',q))
- print( 'Child Process will start')
- p.start()
- p.join()
- p2.start()
- p2.join()
- print( 'Child process end.')
multiprocessing.Pool不能直接用multiprocessing.Queue进行通信,只能通过共享内存,或者用multiprocessing.Manager()进行进程间通信。参考以下代码,代码一为使用multiprocessing.Queue进行通信,代码二是multiprocessing.Manager()进行进程间通信
代码一:
- from multiprocessing import Pool,Queue,Manager
- import os
- def run_proc(name,q):
- print( 'Run Child process %s (%s)' % (name, os.getpid()))
- q.put( 'value')
- if __name__ == '__main__':
- print( 'Parent process %s.' % os.getpid())
- q = Queue()
- p = Pool( 2)
- L = [ 1, 2]
- for l in L:
- p.apply_async(run_proc,args=(l,q))
- print( 'Child Process will start')
- p.close()
- p.join()
- print( 'Child process end.')
- print(q.get())
上面的代码执行后,控制台打印结果如下,并且程序一直未能结束。
- Parent process 4640.
- Child Process will start
- Child process end.
代码二:
- from multiprocessing import Pool,Queue,Manager
- import os
- def run_proc(name,q):
- print( 'Run Child process %s (%s)' % (name, os.getpid()))
- q.put( 'value')
- if __name__ == '__main__':
- print( 'Parent process %s.' % os.getpid())
- q = Manager().Queue()
- p = Pool( 2)
- L = [ 1, 2]
- for l in L:
- p.apply_async(run_proc,args=(l,q))
- print( 'Child Process will start')
- p.close()
- p.join()
- print( 'Child process end.')
- print(q.get())
代码二的执行结果如下:
- Parent process 4864.
- Child Process will start
- Run Child process 1 ( 6008)
- Run Child process 2 ( 5112)
- Child process end.
- value
Note:
1. multiprocess.Manager()只能在主进程中使用,因为multiprocess.Manager()初始化后会开启一个server process,而在子进程中因为默认的daemon为True,(即默认是守护进程,不允许创建子进程,并且父进程代码执行结束,子进程也会终止允许)。如果在子进程中使用multiprocess.Manager(),就会抛出异常:AssertionError:daemonic processes are not allowed to have children.
2. 因为在Linux或者Mac系统,multiprocessing.Pool默认开启子进程的方式是通过fork。也可以通过multiprocess.set_start_method()方法修改开启的方式为spawn/ forkserver。那么如果要调用该方法,那么 multiprocess.Manager()应该用在multiprocess.set_start_method()方法之后。否则,就会抛出异常:Runtimeerror: context has already been set
Ref:
廖雪峰-多进程
转自: Oops_newbie的博客
更多推荐
multiprocessing.Process,multiprocessing.Pool区别
发布评论