Trying to chain processes together on a pipeline

[Thomas Rachel]

Am 28.06.2011 07:57 schrieb Andrew Berg:
> I'm working on an audio/video converter script (moving from bash to
> Python for some extra functionality), and part of it is chaining the
> audio decoder (FFmpeg) either into SoX to change the volume and then to
> the Nero AAC encoder or directly into the Nero encoder. This is the
> chunk of code from my working bash script to give an idea of what I'm
> trying to accomplish (it's indented because it's nested inside a while
> loop and an if statement):
>>          if [ "$process_audio" = "true" ]
>>          then
>>              if [ $vol == 1.0 ]
>>              then
>>                  ffmpeg -i "${ifile_a}" -f wav - 2>$nul | neroaacenc
>> -ignorelength -q 0.4 -if - -of ${prefix}${zero}${ep}.m4a
>>              else
>>                  # the pipeline-as-file option of sox fails on Windows
>> 7, so I use the safe method since there's only one pipeline going into sox
>>                  ffmpeg -i "${ifile_a}" -f sox - 2>$nul | sox -t sox -
>> -t wav - vol $vol 2>$nul | neroaacenc -ignorelength -q 0.4 -if - -of
>> ${prefix}${zero}${ep}.m4a
>>              fi
>>          else
>>              echo "Audio skipped."
>>          fi
> This is pretty easy and straightforward in bash, but not so in Python.
> This is what I have in Python (queue[position] points to an object I
> create earlier that holds a bunch of info on what needs to be encoded -
> input and output file names, command line options for the various
> encoders used, and so forth), but clearly it has some problems:
>>      try:
>>          ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
>> stdout=subprocess.PIPE, stderr=os.devnull)
>>      except WindowsError:
>>          error_info = str(sys.exc_info()[1])
>>          last_win_error_num = find_win_error_no(error_msg=error_info)
>>          if last_win_error_num == '2': # Error 2 = 'The system cannot
>> find the file specified'
>>              logger.critical('Could not execute ' +
>> queue[position].ffmpeg_exe + ': File not found.')
>>          elif last_win_error_num == '193': # Error 193 = '%1 is not a
>> valid Win32 application'
>>              logger.critical('Could not execute ' +
>> queue[position].ffmpeg_exe + ': It\'s not a valid Win32 application.')
>>          break
>>      if queue[position].vol != 1:
>>          try:
>>              sox_proc = subprocess.Popen(queue[position].sox_cmd,
>> stdin=ffmpeg_proc.stdout, stdout=subprocess.PIPE, stderr=os.devnull)
>>          except WindowsError:
>>              error_info = str(sys.exc_info()[1])
>>              last_win_error_num = find_win_error_no(error_msg=error_info)
>>              if last_win_error_num == '2': # Error 2 = 'The system
>> cannot find the file specified'
>>                  logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': File not found.')
>>              elif last_win_error_num == '193': # Error 193 = '%1 is not
>> a valid Win32 application'
>>                  logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': It\'s not a valid Win32 application.')
>>              break
>>          wav_pipe = sox_proc.stdout
>>      else:
>>          wav_pipe = ffmpeg_proc.stdout
>>      try:
>>          nero_aac_proc = subprocess.Popen(queue[position].nero_aac_cmd,
>> stdin=wav_pipe)
>>      except WindowsError:
>>          error_info = str(sys.exc_info()[1])
>>          last_win_error_num = find_win_error_no(error_msg=error_info)
>>          if last_win_error_num == '2': # Error 2 = 'The system cannot
>> find the file specified'
>>              logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': File not found.')
>>          elif last_win_error_num == '193': # Error 193 = '%1 is not a
>> valid Win32 application'
>>              logger.critical('Could not execute ' +
>> queue[position].sox_exe + ': It\'s not a valid Win32 application.')
>>          break
>>
>>      ffmpeg_proc.wait()
>>      if queue[position].vol != 1:
>>          sox_proc.wait()
>>      nero_aac_proc.wait()
>>      break
> Note: those break statements are there to break out of the while loop
> this is in.
> Firstly, that first assignment to ffmpeg_proc raises an exception:
>> Traceback (most recent call last):
>>    File "C:\Users\Bahamut\workspace\Disillusion\disillusion.py", line
>> 288, in<module>
>>      ffmpeg_proc = subprocess.Popen(queue[position].ffmpeg_cmd,
>> stdout=subprocess.PIPE, stderr=os.devnull)
>>    File "C:\Python32\lib\subprocess.py", line 700, in __init__
>>      errread, errwrite) = self._get_handles(stdin, stdout, stderr)
>>    File "C:\Python32\lib\subprocess.py", line 861, in _get_handles
>>      errwrite = msvcrt.get_osfhandle(stderr.fileno())
>> AttributeError: 'str' object has no attribute 'fileno'
> I'm not really sure what it's complaining about since the exception
> propagates from the msvcrt module through the subprocess module into my
> program. I'm thinking it has to do my stderr assignment, but if that's
> not right, I don't know what is.
> Secondly, there are no Popen.stdout.close() calls because I'm not sure
> where to put them.

> Thirdly, I have nearly identical except WindowsError: blocks repeated -
> I'm sure I can avoid this with decorators as suggested in a recent
> thread, but I haven't learned decorators yet.

There is a variety of possibilities. As you are a starter, a function is 
prefecty fine, as Peter already pointed out. Then you have something like

def handle_winerr(exc):
     error_info = str(exc)
     last_win_error_num = find_win_error_no(error_msg=error_info)
     if last_win_error_num == '2': # Error 2 = 'The system cannot find 
the file specified'
         logger.critical('Could not execute ' +
             queue[position].sox_exe + ': File not found.')
     elif last_win_error_num == '193': # Error 193 = '%1 is not a valid 
Win32 application'
         logger.critical('Could not execute ' +
             queue[position].sox_exe + ': It\'s not a valid Win32 
application.')

try:
     <do_stuff>
except WindowsError:
     exc = sys.exc_info()[1]
     handle_winerr(exc)
     break


On the next step, you could write like this:

try:
     <do_stuff>
except WindowsError, exc:
     handle_winerr(exc)
     break


The way you work with the exception is not the very best - instead of 
parsing the stringified exception, you better would trigger on 
exc.winerror (it is an integer with the error number).

Or, even better, just pas the error information contained in the exception:

def handle_winerr(exc):
     logger.critical('Could not execute %s: %s' %
         (queue[position].sox_exe, exc.strerror))


Next step: If you always break out of the loop when there is an error, 
you could as well do the error handling outside the loop:


try:
     while <whatever>:
         <do_stuff>
except WindowsError, exc:
     handle_winerr(exc)



In the following, I'm pointing out some more or less advanced techniques 
which you also could use.

If you want to, you can create a context manager dealing with the stuff:

import contextlib

@contextlib.contextmanager
def winerr_handling():
     try:
         yield None
     except WindowsError, exc:
         handle_winerr(exc) # <-- the function from above

and then write

with winerr_handling():
     while <whatever>:
         <do_stuff>

And, if you absolutely want a decorator, you can do so:

def make_exc_ctx(hdlr):
     from functools import wraps
     @wraps(hdlr)
     def wrapper():
         try:
             yield None
         except WindowsError, exc:
             hdlr(exc) # <-- the function from above


and then

@make_exc_ctx
def winerr_handling(exc):
     logger.critical('Could not execute %s: %s' %
         (queue[position].sox_exe, exc.strerror))

with winerr_handling():
     while <whatever>:
         <do_stuff>


HTH,

Thomas