I have recently begun using multiprocessing for a variety of batch
jobs. It's a great library, and it's been quite useful. However, I have been bitten several times by situations where a worker process in a Pool will unexpectedly die, leaving multiprocessing hanging in a wait. A simple example of this is produced by the following:
"""
#!/usr/bin/env python
import multiprocessing, sys
def foo(x):
sys.exit(1)
multiprocessing.Pool(1).apply(foo, [1])
"""
The child will exit and the parent will hang forever. A similar occurrence happens if one pushes C-c while a child process is running (this special case is noted in http://bugs.python.org/issue8296) or killed by a signal.
Attached is a patch to handle unexpected terminations of children
processes and prevent the parent process from hanging. A test case is included. (Developed and tested on 64-bit Ubuntu.) Please let me know what you think. Thanks!