OSError: [Errno 12] Cannot allocate memory
duncan smith
duncan at invalid.invalid
Wed Nov 30 19:54:50 EST 2016
On 30/11/16 17:53, Chris Kaynor wrote:
> On Wed, Nov 30, 2016 at 9:34 AM, duncan smith <duncan at invalid.invalid> wrote:
>> Hello,
>> I have had an issue with some code for a while now, and I have not
>> been able to solve it. I use the subprocess module to invoke dot
>> (Graphviz) to generate a file. But if I do this repeatedly I end up with
>> an error. The following traceback is from a larger application, but it
>> appears to be repeated calls to 'to_image' that is the issue.
>
> I don't see any glaring problems that would obviously cause this,
> however have you checked to see if the processes are actually exiting
> (it looks like you are on Linux, so the top command)?
>
>>
>>
>> Traceback (most recent call last):
>> File "<pyshell#80>", line 1, in <module>
>> z = link_exp.sim1((djt, tables), variables, 1000, 400, 600,
>> [0,1,2,3,4,5,6], [6,7,8,9,10], ind_gens=[link_exp.males_gen()],
>> ind_gens_names=['Forename'], seed='duncan')
>> File "link_exp.py", line 469, in sim1
>> RL_F2 = EM_setup(data)
>> File "link_exp.py", line 712, in full_EM
>> last_g = prop.djt.g
>> File "Nin.py", line 848, in draw_model
>> dot_g.to_image(filename, prog='dot', format=format)
>> File "dot.py", line 597, in to_image
>> to_image(str(self), filename, prog, format)
>> File "dot.py", line 921, in to_image
>> _execute('%s -T%s -o %s' % (prog, format, filename))
>> File "dot.py", line 887, in _execute
>> close_fds=True)
>> File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
>> errread, errwrite)
>> File "/usr/lib/python2.7/subprocess.py", line 1235, in _execute_child
>> self.pid = os.fork()
>> OSError: [Errno 12] Cannot allocate memory
>>
>>
>> The relevant (AFAICT) code is,
>>
>>
>> def to_image(text, filename, prog='dot', format='dot'):
>> # prog can be a series of commands
>> # like 'unflatten -l 3 | dot'
>> handle, temp_path = tempfile.mkstemp()
>> f = open(temp_path, 'w')
>> try:
>> f.write(text)
>> f.close()
>> progs = prog.split('|')
>> progs[0] = progs[0] + ' %s ' % temp_path
>> prog = '|'.join(progs)
>> _execute('%s -T%s -o %s' % (prog, format, filename))
>> finally:
>> f.close()
>> os.remove(temp_path)
>> os.close(handle)
>>
>> def _execute(command):
>> # shell=True security hazard?
>> p = subprocess.Popen(command, shell=True, stdin=subprocess.PIPE,
>> stdout=subprocess.PIPE,
>> stderr=subprocess.STDOUT,
>> close_fds=True)
>> output = p.stdout.read()
>> p.stdin.close()
>> p.stdout.close()
>> #p.communicate()
>> if output:
>> print output
>
> This code has a potential dead-lock. If you are calling it from
> multiple threads/processes, it could cause issues. This should be
> obvious, as your program will also not exit. The communicate call is
> safe, but commented out (you'd need to remove the three lines above it
> as well). Additionally, you could just set stdin=None rather than
> PIPE, which avoids the dead-lock, and you aren't using stdin anyways.
> This issues comes if the subprocess may ever wait for something to be
> written to stdin, it will block forever, but your call to read will
> also block until it closes stdout (or possibly other cases). Another
> option would be to close stdin before starting the read, however if
> you ever write to stdin, you'll reintroduce the same issue, depending
> on OS buffer sizes.
>
> My question above also comes from the fact that I am not 100% sure
> when stdout.read() will return. It is possible that a null or EOF
> could cause it to return before the process actually exits. The
> subprocess could also expliciting close its stdout, causing it to
> return while the process is still running. I'd recommend adding a
> p.wait() or just uncommenting the p.communicate() call to avoid these
> issues.
>
> Another, unrelated note, the security hazard depends on where the
> arguments to execute are coming from. If any of those are controlled
> from untrusted sources (namely, user input), you have a
> shell-injection attack. Imagine, for example, if the user requests the
> filename "a.jpg|wipehd" (note: I don't know the format command on
> Linux, so replace with your desired command). This will cause your
> code to wipe the HD by piping into the command. If all of the inputs
> are 100% sanitized or come from trusted sources, you're fine, however
> that can be extremely difficult to guarantee.
>
Thanks. So something like the following might do the job?
def _execute(command):
p = subprocess.Popen(command, shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
close_fds=True)
out_data, err_data = p.communicate()
if err_data:
print err_data
Duncan
More information about the Python-list
mailing list