os.popen hangs mutt

[Donn Cave]

Quoth "Jason R. Mastaler" <jason-dated-990201218.0d3ca1 at mastaler.com>:
| I need some help debugging a strange problem I'm having with the mutt
| mail client on certain BSD UNIX platforms.  I think it might be caused
| by a bug in os.popen.
|
| Basically, I use a custom Python "injection agent" which sits between
| mutt and the mail transport agent (qmail in my case).  It accepts the
| message from mutt as stdin, does some header manipulation, and hands
| it off to qmail-inject for delivery.
|
| mutt --> injection-agent --> qmail-inject --> recipient
|
| The problem is that on certain operating systems (FreeBSD, NetBSD,
| OpenBSD), this exchange hangs before reaching qmail-inject so the
| message is never delivered.  The injection-agent is left hanging
| around in the process table until manually killed.  For example,
|
| (jason at mail1):(~)$ echo | mutt jason
|
| (this command hangs, and the `ps' output looks like this)
| jason      737  0.0  1.1   900  1424 ??  I     10:44AM    0:00.03 python /home/jason/pyinject -- jason
|
| The strange thing is that if I implement the injection-agent using the
| popen2 module instead of os.popen, the problem goes away.  I've tried
| several different Python versions from 1.5.2 to 2.1 with the same
| results.  So the question is: why does os.popen hang while popen2
| doesn't?

My guess is that it will be blocking in wait(), called from pclose().
If its qmail-inject child hasn't exited, that would even be correct.
popen2() doesn't tie close() to wait().

I imagine something must have fallen out of your examples, but the
first omits what I think would be an important write('\n').

	Donn Cave, University Computing Services, University of Washington
	donn at u.washington.edu
--------
| Here is a simple prototype of the injection-agent that I use to
| reproduce the problem.  The program simply accepts the message on
| stdin, extracts the "To:" header, opens a pipe to qmail-inject, and
| writes the message to it.
|
| ============================================
|
| #!/usr/bin/env python
|
| import os
| import sys
| import rfc822
|
| message = sys.stdin
| message_headers = rfc822.Message(message)
|
| to = message_headers['to']
| message_body = message.read()
|
| inject = '/var/qmail/bin/qmail-inject ' + to
| pipeline = os.popen(inject, 'w')
| pipeline.write(str(message_headers))
| pipeline.write(message_body)
| pipeline.close()
|
| ============================================
|
| If I replace the last chunk of the above program with popen2, the
| problem goes away.  I.e,
|
| [...]
|
| import popen2
| pipeline = popen2.popen2(inject)[1]
| pipeline.write(str(message_headers))
| pipeline.write('\n')
| pipeline.write(message_body)
| pipeline.close()
|
| P.S - if anyone is brave enough, I have a 400k kernel process trace
| log of a hung mutt delivery attempt that might contain the answer.
|
| -- 
| (TMDA - http://tmda.sourceforge.net)
| (Python-based SPAM reduction system)
|