[Tutor] Interacting with stderr

[Cameron Simpson]

On 27Aug2014 22:42, Danny Yoo <dyoo at hashcollision.org> wrote:
>>> Rather than construct the pipeline through the shell, do it through
>>> Python if you can.  See:
>>>
>>> https://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline
>>
>> But his use case is not using a shell pipeline, so irrelevant. It just makes
>> things more complex for him.
>
>Did I misread the code?  Here's the snippet I thought was part of the question:
>
>>>       p = subprocess.call("avconv -v verbose -re -analyzeduration 1000 -ihttp://localhost:6498/ms2/1382097438004/0MediaPlayer+0+/octoshape+hVV+octolive.americaone.com+V+aone+V+live+V+ONECONNXT_DEMO1_HD_flv/aoneliveONECONNXTDEMO1HDflv -c:v rawvideo -c:a pcm_s16le -ar 48000 -f nut - | ./bmdplay -m 12 -f pipe:0",
>> shell=True)
>
>where there's a pipeline between the avconv process mbdplay process.

You didn't misread it. I did; I missed the pipe symbol.

Nonetheless, for the sake of getting his stuff working it should perhaps be 
chased later. In any case, he could easily be moving to putting that pipeline 
in a small shell script and just invoking the script from python. I've got any 
number of one or two line shell scripts like that.

I agree that using shell=True is usually worth trying to avoid.

>> Again, disagree. In this specific case, disagree strongly.
>>
>> Firstly, the OP is not feeding stuff to stdin nor collecting stdout.
>>
>> Secondly, the OP has made it clear that they're not in a position to wait
>> for the command to finish; they need to read stderr as it occurs because the
>> run time is very long and they need to act earlier than process completion.
>
>The OP's situation is somewhat unusual.  My apologies for not reading
>the thread thoroughly and closely.  I got distracted by the initial
>digest message.

I can imagine. I'm not a big fan of digests.

>> Thirdly, I generally consider advice to use .communicate bad advise.
>>
>> .communicate has many potential issues:
>>
>> Primarily, it hides all the mechanics. He will learn nothing.
>
>If it helps, let me qualify the statement that communicate() is
>usually an appropriate solution.  For simple, common cases where the
>interaction isn't that large or long-running, Popen.communicate() is
>probably the right tool.  That is, I would usually recommend
>Popen.communicate() _unless_ the details made it the wrong tool.  I
>misread the problem statement, so yeah, my bad.
>
>The OP's requirements are unusual: it's a long running process, it's
>generating a dynamic, large amount of stderr output, it's disregarding
>stdout, and there's a desire to process the stderr on the fly.  In
>that situation, your solution sounds reasonable.

Ta.

The other thing about communicate and (less so) call is that they do so much.  
When a particular part isn't working, figuring out what to fix is harder.

On a tutor list, what's our stance on exploring the operation at a lower level 
so the OP gets a better feel for what's going on? I'm aware we don't need to 
agree on that stance:-)

Cheers,
Cameron Simpson <cs at zip.com.au>

It is very easy to be blinded to the essential uselessness of them by
the sense of achievement you get from getting them to work at all. In
other words - their fundamental design flaws are completely hidden by
their superficial design flaws. - Douglas Adams on VCRs etc