[Tutor] "standard output: Broken pipe"

[Eric Brunson]

Martin Walsh wrote:
> Eric Brunson wrote:
>   
>> I'm coming in late to the discussion and thought that someone would 
>> explain it succinctly, but there have been so many correct statements 
>> which I feel fail to nail down the problem that I thought I'd chime in.
>>
>>     
>
> Hi Eric,
>
> Thank you for your considerate response. Your analysis is spot on --
> speaking for myself, of course -- I'm mostly just flailing here; not
> having a very strong grasp of how all the components work
> together (kernel, shell, python, and the subprocess module). But I'm
> really enjoying this thread and learning a lot in the 'process' ...
> ahem, sorry for the pun ... painful :)
>
>   
>> Here's the important concepts to understand.  The pipe is a construct of 
>> the shell, you are spawning a shell to run your entire command line.  
>> When the "make" finishes and stops accepting the input from "yes", the 
>> kernel sends a SIGPIPE to the parent process, i.e. the shell.  The 
>> shell's default behavior when receiving the SIGPIPE is to print an error 
>> message.
>>     
>
> You're right. Considering the shell is definitely important and I missed
> it. Yet given the following, slightly adapted from the subprocess module
> docs (your reference link below) ...
>
> from subprocess import Popen, PIPE
> p1 = Popen(["yes", "Hello"], stdout=PIPE)
> p2 = Popen(["head", "-n", "10"], stdin=p1.stdout, stdout=PIPE)
> output = p2.communicate()[0]
>
> ... still produces ...
>
> yes: standard output: Broken pipe
> yes: write error
>   

Looks like the "yes" command is getting the SIGPIPE and not the shell, 
as I originally thought.  I must have glossed over the "yes:" at the 
beginning.

> If run as a script, the message above shows up in the terminal after the
> python script terminates (or apparently so; shell prompt first, then
> error). And when using the interactive interpreter... nothing until
> exiting the interpreter, when the message again appears in the terminal.
> Perhaps this is some peculiarity of my environment, or more likely I am
> still missing something.
>
>   
>> Your problem, as I interpret it, is that you don't like seeing the error 
>> message.  So you have two choices:  1) find some way to filter it, 
>> because all it is is a message, or 2) get rid of the shell and handle 
>> the management of the subprocesses yourself.  It's been discussed how to 
>> suppress the output and it's been discussed how to intercept the signal, 
>> but I don't thing that anyone has pointed to the definitive python 
>> construct to handle it properly. 
>>
>> You've had several responses that correctly tell you how to handle a 
>> signal, but without any testing I'm pretty sure that none of that will 
>> do you any good with the subprocess.call() construct you're using.  The 
>> shell is your subprocess and it is already handling the signal, so you 
>> should never see it.  The trick is to spawn both processes via python 
>> *without* an intervening shell to interfere.  Here's the canonical 
>> example: http://docs.python.org/lib/node536.html.  You can spawn each 
>> subprocess, 'yes' and 'make' without the shell being required to pipe 
>> the output of one to the other, you can do it all completely in python.
>>     
>
> Based on my own tests, I don't think it makes a difference. I suspect
> the python override for sigpipe is inherited by all child processes,
> shell or otherwise, and the SIG_IGN is allowing 'yes' to see the broken
> pipe, report it and terminate -- where the default action would normally
> be to terminate only. This is, of course, not that far from wild
> speculation. Though, I am curious if others observe the same behavior.
>   

I'd have to break out some documentation to review signal mask 
inheritance, it's been so long it escapes me.

> Then again, the error is *not* displayed for any of tests where
> signal.signal(signal.SIGPIPE, signal.SIG_DFL) is added, in either the
> global scope or in the form of a preexec_fn argument passed to
> subprocess, with shell=True or without.
>
> IMHO, this is certainly a tripping point if true. Granted, probably a
> fringe case and relatively rare since it requires spawning a subprocess,
> which writes to a pipe that is broken before the subprocess ends, and is
> not implemented to handle the event gracefully. But I still find it
> weirdly intriguing.
>   

Since "yes" is designed to spit out its output indefinitely, it seems 
like a bad behavior to print a message on SIGPIPE.  It's always going to 
happen, it's the expected result.  The versions of "yes" on my linux box 
here and a solaris box at work don't seem to elicit the same error message.

Just another reason why I would look for a solution that would avoid the 
"yes" altogether.  :-)

>> Again, sorry to come in late, but while reading many absolutely factual 
>>     
>
> <off-topic-blather>
> As far as I'm concerned, no apologies are necessary -- not that you are
> apologizing to me necessarily :) But nonetheless, thank you for helping.
>
> In fact, a big thank you to all the tutors. I've been a member of the
> list for quite a few years now, reaping the benefits of your collective
> experience (like many others I would imagine), mostly by lurking. You
> provide a tremendous service to the python community, especially those
> new to python or programming in general, with near super-human patience
> and skill. Quite a feat! FWIW, thanks.
> </off-topic-blather>
>   

Even being one of the more vocal on the list, I'll add my vote to your 
commendations.  I learn just as much on the list as I feel I teach, it's 
a great forum with some great participants.

> Marty
>
>   
>> an correct responses, they all seemed to be either handling the output 
>> or discussing signal handling without, in my mind, pointing out that 
>> it's the shell that's giving you the problems.  I think it's Python's 
>> default behavior to ignore SIGPIPE, as Martin comments on in his latest 
>> reply.  It's just that the shell has already accepted and handled the 
>> signal.
>>
>> Interestingly, when you eliminate the shell and use something similar to 
>> the example code from above, you can pretty easily see how to get rid of 
>> the 'yes' and just feed the make subprocess input yourself, further 
>> simplifying the code.
>>
>> I'll admit, I've stated all this without actually running any test, so 
>> please, if anyone can show differently, I'm happy to have my 
>> understanding corrected.
>>
>> I hope that helps, not only with your specific problem, but from a "big 
>> picture" standpoint, also.  Nothing I hate more than writing code that I 
>> don't really understand.  :-)
>>
>> e.
>>
>> James wrote:
>>     
>>> Hi,
>>>
>>> I have a snippet of code in a Python script I'm whipping up that's  
>>> causing a not-so-pretty output.  Here's the code:
>>>
>>> subprocess.call( "yes '' | make oldconfig" , shell=True )
>>>
>>> When I run this code, Python loyally executes the command, and then I  
>>> see the following error on my console:
>>>
>>> -----
>>>
>>> yes: standard output: Broken pipe
>>> yes: write error
>>>
>>> -----
>>>
>>> I did some Googling and I believe found the reason for this error  
>>> ("yes" executes forever, and complains when Python kills the process  
>>> off).  However, I'd like to figure out a way to get rid of the error  
>>> (or hide it) so that it's not visible to the person running the  
>>> script (it's not the prettiest thing to see scroll by your screen :)).
>>>
>>> Thoughts / ideas?
>>>
>>> Thanks!
>>> .james
>>> _______________________________________________
>>> Tutor maillist  -  Tutor at python.org
>>> http://mail.python.org/mailman/listinfo/tutor
>>>   
>>>       
>> _______________________________________________
>> Tutor maillist  -  Tutor at python.org
>> http://mail.python.org/mailman/listinfo/tutor
>>     
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> http://mail.python.org/mailman/listinfo/tutor
>