[Python-Dev] Implementing File Modes

[Eric Pruitt]

My motivation came from an instance when I was using subprocess.Popen for a
Linux / Windows cross platform program. In part of the program, I was
writing and reading to a cron like object. On Windows, it was a text file
and on Linux it would be the crontab executable. Had I been able to
substitute the "open()" function with my wrapper, it would have been the
only change I had to make for cross platform compatibility; instead of
having to change numerous lines because Linux would need Popen and Windows
would need a regular file open(), I could simply make it so that if the
platform was Linux, my wrapper is used in place of that. Just another
example would be having an external program decrypt a file that can be in
plain text or encrypted that might go something like this:

if encryptionEnabled:
    fileobj = subprocess.ProcessIOWrapper("gpg --decrypt
supersecret.html.gpg")
else:
    fileobj = open("notsosecret.html")

>From there, the functions would not have to be modified despite using a
running process versus a file object.

On Tue, Jul 28, 2009 at 01:53, Gregory P. Smith <greg at krypto.org> wrote:

> On Mon, Jul 27, 2009 at 5:32 PM, Glyph Lefkowitz<glyph at twistedmatrix.com>
> wrote:
> > On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> >>
> >> I like MRAB's idea of using a (non-standard) "e" flag to include
> >> stderr. So "r" reads from stdout, "re" reads from stdout+stderr.
> >>
> >> Anything more complicated probably should just use "raw" Popen
> >> objects. Don't overcomplicate the interface.
> >
> > In my opinion, mangling stderr and stdout together is already an
> > overcomplication.  It shouldn't be implemented.
> >
> > It seems like a good idea, until you realize that subtle changes to your
> OS,
> > environment, or buffering behavior may result in arbitrary, unparseable
> > output.
>
> Agreed.  Leave stderr support out of this.  People who need stderr
> should use the full subprocess.Popen interface.  Quick hack unixy
> types will just run their process using a shell (which already seems
> to be the default based on the "ls -l" example) and add 2>&1.  This
> functionality is basically the equivalent of adding the | symbol on
> either or both ends of a filename given to open() in perl.  (but
> without being so gross).
>
> I do wonder why you're trying to make it behave exactly like open()
> including the mode= syntax.
>
> Why not just define several names based on the behavior?
>
>  ProcessReadWrapper()
>  ProcessWriteWrapper()
>  ProcessReadWriteWrapper()
>
> -gps
>
> >
> > For example, let's say you've got a program whose output is a list of
> lines,
> > each one containing a number.  Sometimes it tries to import gtk, and
> fails
> > to open its display.
> >
> > That's fine, and you can still deal with it, as long as the interleaved
> > output looks like this:
> >
> > 100
> > 200
> > Gtk-WARNING **: cannot open display:
> > 300
> > 400
> >
> > but of course the output might (although unlikely with such small chunks
> of
> > output) end up looking like this, instead:
> >
> > 100
> > 2Gtk-WAR0NING0 **:
> > can30not 0open display:
> >
> > 400
> >
> > this is the sort of thing which is much more likely to happen once you
> start
> > dealing with large volumes of data, where there are more page-boundaries
> for
> > your buffers to get confused around, and you are playing with buffering
> > options to improve performance.  In other words, it's something that
> fails
> > only at scale or under load, and is therefore extremely difficult to
> debug.
> >
> > This option might be okay if it were allowed only on subprocesses opened
> in
> > a text mode, and if the buffering logic involved forced stderr and stdout
> to
> > be line-delimited, and interleave only lines, rather than arbitrary
> chunks
> > of bytes.  Of course then if you use this flag with a program that
> outputs
> > binary data with no newlines it will buffer forever and crash your
> program
> > with a MemoryError, but at least that's easy to debug when it happens.
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090729/8cd94d64/attachment-0001.htm>