GCC process not working as expected when called in Python (3.1.2) subprocess-shell, but OK otherwise

[Diez B. Roggisch]

Kingsley Turner <kingsley.turner at openfieldcommunications.com> writes:

>  Hi,
>
> I'm using GCC as a pre-processor for a C-like language (EDDL) to
> handle all the includes, macros, etc. producing a single source file
> for another compiler.  My python code massages the inputs (which
> arrive in a .zip file), then calls GCC.
>
> I have a problem where if I call GCC from my python script some of the
> #defines are not processed in the output.  However if I paste the
> exact same GCC command-line into a shell, I get a correct output.
>
> I'm calling GCC in this manner:
>
>     ### Execute GCC, keep stdout & stderr
>     err_out = open(error_filename,"wb")
>     process = subprocess.Popen(gcc_command, stderr=err_out,
> bufsize=81920, cwd=global_options['tmp'])
>     gcc_exit_code = process.wait()
>     log("GCC Exit Code %u" % (gcc_exit_code))
>     err_out.close()
>
> where gcc_command is:
>
>     "/usr/bin/gcc -I /tmp/dd-compile_1286930109.99 -I 
> /home/foo/eddl-includes -D__TOKVER__=600 -ansi -nostdinc -v -x c -E -o 
> /tmp/dd-compile_1286930109.99/11130201.ddl.OUT
> /tmp/dd-compile_1286930109.99/11130201.ddl"
>
> So when this code spawns GCC, the compiler does not really work 100%,
> but if I paste this exact command line, the output is perfect.
>
> I'm not really sure how to debug this.  I already checked the ulimits,
> and permissions shouldn't be a problem since it's all run by the same
> user, I also checked the environment - these were copied into the
> subshell.
>
> GCC produces no warnings, or errors.  The output is mostly OK, some
> other macros have been processed.
> If I diff the working output with the non-working one, the differences
> are only a bunch of skipped #defines.
>
> gcc version 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu5)
> Python 3.1.2 (release31-maint, Sep 17 2010, 20:27:33)
> Linux 2.6.35-22-generic #34-Ubuntu SMP Sun Oct 10 09:26:05 UTC 2010
> x86_64 GNU/Linux
>
> Any suggestions for helping me debug this would be much appreciated.

sounds nasty. Only thing I can imagine is that GCC wants specific
environment variables to exist. Maybe using "shell=True" helps? Or
capturing and passing an explicit environment?

Diez