[ python-Bugs-1546442 ] subprocess.Popen can't read file object as stdin after seek

[SourceForge.net]

Bugs item #1546442, was opened at 2006-08-25 14:52
Message generated for change (Settings changed) made by quiver
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1546442&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: GaryD (gazzadee)
>Assigned to: Peter Åstrand (astrand)
Summary: subprocess.Popen can't read file object as stdin after seek

Initial Comment:
When I use an existing file object as stdin for a call
to subprocess.Popen, then Popen cannot read the file if
I have called seek on it more than once.

eg. in the following python code:

>>> import subprocess
>>> rawfile = file('hello.txt', 'rb')
>>> rawfile.readline()
'line 1\n'
>>> rawfile.seek(0)
>>> rawfile.readline()
'line 1\n'
>>> rawfile.seek(0)
>>> process_object = subprocess.Popen(["cat"],
stdin=rawfile, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)

process_object.stdout now contains nothing, implying
that nothing was on process_object.stdin.

Note that this only applies for a non-trivial seek (ie.
where the file-pointer actually changes). Calling
seek(0) multiple times in a row does not change
anything (obviously).

I have not investigated whether this reveals a problem
with seek not changing the underlying file descriptor,
or a problem with Popen not handling the file
descriptor properly.

I have attached some complete python scripts that
demonstrate this problem. One shows cat working after
calling seek once, the other shows cat failing after
calling seek twice.

Python version being used:
Python 2.4.2 (#1, Nov  3 2005, 12:41:57)
[GCC 3.4.3-20050110 (Gentoo Linux 3.4.3.20050110,
ssp-3.4.3.20050110-0, pie-8.7 on linux2


----------------------------------------------------------------------

Comment By: lplatypus (ldeller)
Date: 2006-08-25 16:13

Message:
Logged In: YES 
user_id=1534394

I found the cause of this bug:

A libc FILE* (used by python file objects) may hold a
different file offset than the underlying OS file
descriptor.  The posix version of Popen._get_handles does
not take this into account, resulting in this bug.

The following patch against svn trunk fixes the problem.  I
don't have permission to attach files to this item, so I'll
have to paste the patch here:

Index: subprocess.py
===================================================================
--- subprocess.py       (revision 51581)
+++ subprocess.py       (working copy)
@@ -907,6 +907,12 @@
             else:
                 # Assuming file-like object
                 p2cread = stdin.fileno()
+                # OS file descriptor's file offset does not
necessarily match
+                # the file offset in the file-like object,
so do an lseek:
+                try:
+                    os.lseek(p2cread, stdin.tell(), 0)
+                except OSError:
+                    pass # file descriptor does not support
seek/tell

             if stdout is None:
                 pass
@@ -917,6 +923,12 @@
             else:
                 # Assuming file-like object
                 c2pwrite = stdout.fileno()
+                # OS file descriptor's file offset does not
necessarily match
+                # the file offset in the file-like object,
so do an lseek:
+                try:
+                    os.lseek(c2pwrite, stdout.tell(), 0)
+                except OSError:
+                    pass # file descriptor does not support
seek/tell

             if stderr is None:
                 pass
@@ -929,6 +941,12 @@
             else:
                 # Assuming file-like object
                 errwrite = stderr.fileno()
+                # OS file descriptor's file offset does not
necessarily match
+                # the file offset in the file-like object,
so do an lseek:
+                try:
+                    os.lseek(errwrite, stderr.tell(), 0)
+                except OSError:
+                    pass # file descriptor does not support
seek/tell

             return (p2cread, p2cwrite,
                     c2pread, c2pwrite,


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1546442&group_id=5470