hi - I noticed some odd behaviour with subprocess when the output
(stdout) is large - I did some sleuthing and found a reproduceable case
- I tested this on solaris using python 2.4.2 (sparc) and using 2.5
(opteron) - I've attached a python script that details the problem - I
created 2 files ("small_file" and "large_file") that are 20480 and 20481
bytes, respectively ....
> ls -lrt ./tmp/*_file
-rw-r--r-- 1 wtorres staff 20481 Oct 10 12:32 ./tmp/large_file
-rw-r--r-- 1 wtorres staff 20480 Oct 10 12:32 ./tmp/small_file
the script cats each file, thus generating an appropriately sized stdout
- the workaround we came up with was to use our own file handle and use
the parameter stdout=my_file_handle.fileno() instead of stdout=PIPE -
you can see this in the attached example as well
> python ./tmp/brent.py
testing with a small file - these should both work
using my own file handle .... ok
using PIPE ... ok
testing with a large file - the second one here should fail
using my own file handle .... ok
using PIPE ...
Traceback (most recent call last):
File "./tmp/brent.py", line 43, in ?
test_popen_bug( large_file )
File "./tmp/brent.py", line 31, in test_popen_bug
rc = my_popen.wait()
File "/tools/python/2.4.2/solaris/lib/python2.4/subprocess.py", line
1007, in wait
pid, sts = os.waitpid(self.pid, 0)
KeyboardInterrupt
based on all this, it seems that the PIPE file handle has some buffer or
something that is filling up and not allowing us to continue... or
something. Thoughts?
-w
PIPE on POSIX are implemented using pipe system call so it will have all
the limitations of the system buffering that is used for implementing
pipes. The buffer size is system dependent. For example, on my Linux, I
hit the hang condition at 64K and not 20480 bytes (as is your case).
What basically happens is that the child fills up the pipe buffer and is
blocked until space is available. Since the parent in your test script
is not reading the bytes, space will never be freed up.
I suggest that the bug is closed as invalid.