subprocess research - max limit for piped output
I am trying to figure out what is maximum size for piped input in subprocess.check_output() I've got limitation of about 500Mb after which Python exits with MemoryError without any additional details. I have only 2.76Gb memory used out of 8Gb, so what limit do I hit? 1. subprocess output read buffer 2. Python limit on size of variable 3. some OS limit on output pipes Testcase attached. C:\discovery\interface\subprocess>py dead.py Testing size: 520Mb ..truncating to 545259520 .. Traceback (most recent call last): File "dead.py", line 66, in <module> backticks(r'type largefile') File "dead.py", line 36, in backticks output = subprocess.check_output(command, shell=True) File "C:\Python27\lib\subprocess.py", line 567, in check_output output, unused_err = process.communicate() File "C:\Python27\lib\subprocess.py", line 791, in communicate stdout = _eintr_retry_call(self.stdout.read) File "C:\Python27\lib\subprocess.py", line 476, in _eintr_retry_call return func(*args) MemoryError The process tried to write to a nonexistent pipe. -- anatoly t.
participants (1)
-
anatoly techtonik