My motivation came from an instance when I was using subprocess.Popen for a Linux / Windows cross platform program. In part of the program, I was writing and reading to a cron like object. On Windows, it was a text file and on Linux it would be the crontab executable. Had I been able to substitute the "open()" function with my wrapper, it would have been the only change I had to make for cross platform compatibility; instead of having to change numerous lines because Linux would need Popen and Windows would need a regular file open(), I could simply make it so that if the platform was Linux, my wrapper is used in place of that. Just another example would be having an external program decrypt a file that can be in plain text or encrypted that might go something like this:

if encryptionEnabled:
    fileobj = subprocess.ProcessIOWrapper("gpg --decrypt supersecret.html.gpg")
else:
    fileobj = open("notsosecret.html")

From there, the functions would not have to be modified despite using a running process versus a file object.

On Tue, Jul 28, 2009 at 01:53, Gregory P. Smith <greg@krypto.org> wrote:
On Mon, Jul 27, 2009 at 5:32 PM, Glyph Lefkowitz<glyph@twistedmatrix.com> wrote:
> On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore <p.f.moore@gmail.com> wrote:
>>
>> I like MRAB's idea of using a (non-standard) "e" flag to include
>> stderr. So "r" reads from stdout, "re" reads from stdout+stderr.
>>
>> Anything more complicated probably should just use "raw" Popen
>> objects. Don't overcomplicate the interface.
>
> In my opinion, mangling stderr and stdout together is already an
> overcomplication.  It shouldn't be implemented.
>
> It seems like a good idea, until you realize that subtle changes to your OS,
> environment, or buffering behavior may result in arbitrary, unparseable
> output.

Agreed.  Leave stderr support out of this.  People who need stderr
should use the full subprocess.Popen interface.  Quick hack unixy
types will just run their process using a shell (which already seems
to be the default based on the "ls -l" example) and add 2>&1.  This
functionality is basically the equivalent of adding the | symbol on
either or both ends of a filename given to open() in perl.  (but
without being so gross).

I do wonder why you're trying to make it behave exactly like open()
including the mode= syntax.

Why not just define several names based on the behavior?

 ProcessReadWrapper()
 ProcessWriteWrapper()
 ProcessReadWriteWrapper()

-gps

>
> For example, let's say you've got a program whose output is a list of lines,
> each one containing a number.  Sometimes it tries to import gtk, and fails
> to open its display.
>
> That's fine, and you can still deal with it, as long as the interleaved
> output looks like this:
>
> 100
> 200
> Gtk-WARNING **: cannot open display:
> 300
> 400
>
> but of course the output might (although unlikely with such small chunks of
> output) end up looking like this, instead:
>
> 100
> 2Gtk-WAR0NING0 **:
> can30not 0open display:
>
> 400
>
> this is the sort of thing which is much more likely to happen once you start
> dealing with large volumes of data, where there are more page-boundaries for
> your buffers to get confused around, and you are playing with buffering
> options to improve performance.  In other words, it's something that fails
> only at scale or under load, and is therefore extremely difficult to debug.
>
> This option might be okay if it were allowed only on subprocesses opened in
> a text mode, and if the buffering logic involved forced stderr and stdout to
> be line-delimited, and interleave only lines, rather than arbitrary chunks
> of bytes.  Of course then if you use this flag with a program that outputs
> binary data with no newlines it will buffer forever and crash your program
> with a MemoryError, but at least that's easy to debug when it happens.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>