Trying to chain processes together on a pipeline

[Peter Otten]

Andrew Berg wrote:

> I'm working on an audio/video converter script (moving from bash to
> Python for some extra functionality), and part of it is chaining the
> audio decoder (FFmpeg) either into SoX to change the volume and then to
> the Nero AAC encoder or directly into the Nero encoder. This is the
> chunk of code from my working bash script to give an idea of what I'm
> trying to accomplish (it's indented because it's nested inside a while
> loop and an if statement):
>>         if [ "$process_audio" = "true" ]
>>         then
>>             if [ $vol == 1.0 ]
>>             then
>>                 ffmpeg -i "${ifile_a}" -f wav - 2>$nul | neroaacenc
>> -ignorelength -q 0.4 -if - -of ${prefix}${zero}${ep}.m4a
>>             else
>>                 # the pipeline-as-file option of sox fails on Windows
>> 7, so I use the safe method since there's only one pipeline going into
>> sox
>>                 ffmpeg -i "${ifile_a}" -f sox - 2>$nul | sox -t sox -
>> -t wav - vol $vol 2>$nul | neroaacenc -ignorelength -q 0.4 -if - -of
>> ${prefix}${zero}${ep}.m4a
>>             fi
>>         else
>>             echo "Audio skipped."
>>         fi
> This is pretty easy and straightforward in bash, but not so in Python.
> This is what I have in Python (queue[position] points to an object I
> create earlier that holds a bunch of info on what needs to be encoded -
> input and output file names, command line options for the various
> encoders used, and so forth), but clearly it has some problems:
>>     try:
>>         ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
>> stdout=subprocess.PIPE, stderr=os.devnull)
>>     except WindowsError:
>>         error_info = str(sys.exc_info()[1])
>>         last_win_error_num = find_win_error_no(error_msg=error_info)
>>         if last_win_error_num == '2': # Error 2 = 'The system cannot
>> find the file specified'
>>             logger.critical('Could not execute ' +
>> queue[position].ffmpeg_exe + ': File not found.')
>>         elif last_win_error_num == '193': # Error 193 = '%1 is not a
>> valid Win32 application'
>>             logger.critical('Could not execute ' +
>> queue[position].ffmpeg_exe + ': It\'s not a valid Win32 application.')
>>         break
>>     if queue[position].vol != 1:
>>         try:
>>             sox_proc = subprocess.Popen(queue[position].sox_cmd,
>> stdin=ffmpeg_proc.stdout, stdout=subprocess.PIPE, stderr=os.devnull)
>>         except WindowsError:
>>             error_info = str(sys.exc_info()[1])
>>             last_win_error_num = find_win_error_no(error_msg=error_info)
>>             if last_win_error_num == '2': # Error 2 = 'The system
>> cannot find the file specified'
>>                 logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': File not found.')
>>             elif last_win_error_num == '193': # Error 193 = '%1 is not
>> a valid Win32 application'
>>                 logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': It\'s not a valid Win32 application.')
>>             break
>>         wav_pipe = sox_proc.stdout
>>     else:
>>         wav_pipe = ffmpeg_proc.stdout
>>     try:
>>         nero_aac_proc = subprocess.Popen(queue[position].nero_aac_cmd,
>> stdin=wav_pipe)
>>     except WindowsError:
>>         error_info = str(sys.exc_info()[1])
>>         last_win_error_num = find_win_error_no(error_msg=error_info)
>>         if last_win_error_num == '2': # Error 2 = 'The system cannot
>> find the file specified'
>>             logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': File not found.')
>>         elif last_win_error_num == '193': # Error 193 = '%1 is not a
>> valid Win32 application'
>>             logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': It\'s not a valid Win32 application.')
>>         break
>>     
>>     ffmpeg_proc.wait()
>>     if queue[position].vol != 1:
>>         sox_proc.wait()
>>     nero_aac_proc.wait()
>>     break
> Note: those break statements are there to break out of the while loop
> this is in.
> Firstly, that first assignment to ffmpeg_proc raises an exception:
>> Traceback (most recent call last):
>>   File "C:\Users\Bahamut\workspace\Disillusion\disillusion.py", line
>> 288, in <module>
>>     ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
>> stdout=subprocess.PIPE, stderr=os.devnull)
>>   File "C:\Python32\lib\subprocess.py", line 700, in __init__
>>     errread, errwrite) = self._get_handles(stdin, stdout, stderr)
>>   File "C:\Python32\lib\subprocess.py", line 861, in _get_handles
>>     errwrite = msvcrt.get_osfhandle(stderr.fileno())
>> AttributeError: 'str' object has no attribute 'fileno'
> I'm not really sure what it's complaining about since the exception
> propagates from the msvcrt module through the subprocess module into my
> program. I'm thinking it has to do my stderr assignment, but if that's
> not right, I don't know what is.

os.devnull is a string, but you need a writable file:

>>> subprocess.call(["ls"], stdout=os.devnull)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/subprocess.py", line 470, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.6/subprocess.py", line 614, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "/usr/lib/python2.6/subprocess.py", line 971, in _get_handles
    c2pwrite = stdout.fileno()
AttributeError: 'str' object has no attribute 'fileno'
>>> subprocess.call(["ls"], stdout=open(os.devnull, "w"))
0

> Secondly, there are no Popen.stdout.close() calls because I'm not sure
> where to put them.
> Thirdly, I have nearly identical except WindowsError: blocks repeated -
> I'm sure I can avoid this with decorators as suggested in a recent
> thread, but I haven't learned decorators yet. This code isn't very
> complete; I'm trying to get it to work in ideal conditions first, then
> worry about how to handle failure.

Start with factoring out common code into a good old function.

> Using Python 3 on Windows 7.