[Python-bugs-list] [ python-Bugs-803610 ] os.close(3) raises
OSError: [Errno 9] Bad file descriptor
SourceForge.net
noreply at sourceforge.net
Thu Sep 11 18:55:45 EDT 2003
Bugs item #803610, was opened at 2003-09-10 06:18
Message generated for change (Comment added) made by syrah
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=803610&group_id=5470
Category: Python Library
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Syrah (syrah)
Assigned to: Nobody/Anonymous (nobody)
Summary: os.close(3) raises OSError: [Errno 9] Bad file descriptor
Initial Comment:
os.close(3) raises OSError: Bad file descriptor on
FreeBSD 5.1, with both Python 2.2.2(_2) and 2.3(_1).
Python 2.3 on Gentoo Linux 1.4 is not affected.
Interestingly, if you call os.fstat(3) first, a
subsequent call to os.close(3) succeeds.
And yes, you do need to set up fd #3.
Here is a demonstration (on FreeBSD):
[syrah at ripple filter]$ cat test.py
import os
os.close (3)
[syrah at ripple filter]$ python test.py 3> file
Traceback (most recent call last):
File "test.py", line 2, in ?
os.close (3)
OSError: [Errno 9] Bad file descriptor
[syrah at ripple filter]$ cat test2.py
import os
os.fstat (3)
os.close (3)
[syrah at ripple filter]$ python test2.py 3> file
[syrah at ripple filter]$ (success!)
----------------------------------------------------------------------
>Comment By: Syrah (syrah)
Date: 2003-09-12 00:55
Message:
Logged In: YES
user_id=827529
I think fd6 is created in the pipe call. But why is pipe
returning 4? It should return 0 or -1, shouldn't it?
Do I need to recomiple python (so that debug info is added
to the executable) to attach the debugger to it?
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2003-09-11 22:06
Message:
Logged In: YES
user_id=21627
I think you are misinterpreting the ktrace output. fd 6 is
the source code file, test.py, which can be seen by looking
at the read result for fd 6. Unfortunately, there is no
corresponding open call opening the file... (there is a
second open call for test.py, when Python opens the file to
print the backtrace, but there should be an earlier call).
If this ktrace output can be trusted, it appears that Python
does not call os.open(3) at all, but immediately decides on
an error - which could happen if errno was somehow
overwritten. However, I doubt that ktrace output, since it
fails to report the first open of test.py, which I know must
be there.
I recommend to run this under a debugger, and break at
posix_open.
----------------------------------------------------------------------
Comment By: Syrah (syrah)
Date: 2003-09-11 16:50
Message:
Logged In: YES
user_id=827529
Okay. I think I have attached a tgz file with 3 programs, 3
binary ktraces, and the ktraces converted to text files.
In all cases, the programs were run as follows:
[bash]$ program 3> file
Many thanks for your continued interest.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2003-09-11 06:01
Message:
Logged In: YES
user_id=21627
Was that the complete ktrace? I'm missing the point where it
outputs "Traceback (most recent call last):". If you have
stripped it down, can you please attach the entire thing
(please don't paste it)?
----------------------------------------------------------------------
Comment By: Syrah (syrah)
Date: 2003-09-10 22:45
Message:
Logged In: YES
user_id=827529
I agree that this is a very strange error. Python should be
directly calling the operating systems close function. It
is very puzzling that there is an error.
I did include in my initial post a parent that sets up fd 3.
I used bash. I was able to test successfully with bash. I
wrote a C program:
-------- test.c --------
#include <stdio.h>
int main () {
int i = close (3);
printf ("close: %i\n", i); }
--------
Using bash, I tested test.py, test2.py, and test.c (gcc
test.c -o test). test.c behaved properly. When I set up fd
3 with bash, test.c printed "close: 0". When I did not set
up fd 3, test.c printed "close: -1".
I therefore have bash working properly, C working properly
and test2.py working properly. The only thing that is not
working properly is test.py. To me, it seems that I have
isolated the problem.
However, this problem only manifests on FreeBSD. test.py
runs fine on Gentoo Linux. At present, those are the only
two OSes I have easy access to.
So... if you really want a test program that sets up fd3,
I'll write one - let me know. Would you prefer me to write
it in Python, C, PHP, Ruby, Perl? I'd prefer C as it is
most direct. But bash seems sufficient: "[bash]$ python
test.py 3> file" sets up file descriptor 3 so it writes into
a file named "file".
To look at it from another angle... is it possible that
Python, during its startup and initialization stuff messes
with file descriptor 3? Is it possible it does this in some
really non-standard way? Perhaps it does it indirectly by
calling some OS function that, as a side effect on FreeBSD
touches fd 3? Something is going on with file descriptions
0-4 _every_ time python runs. [Based on the ktrace's below,
it looks like
[syrah at ripple]$ python
Python 2.3 (#1, Sep 9 2003, 22:47:18)
[GCC 3.2.2 [FreeBSD] 20030205 (release)] on freebsd5
Type "help", "copyright", "credits" or "license" for more
information.
>>> import os
>>> os.fstat (0)
(8592, 113L, 67174144L, 1, 1001, 4, 0L, 1063206044,
1063206044, 1063206044)
>>> os.fstat (1)
(8592, 113L, 67174144L, 1, 1001, 4, 0L, 1063206046,
1063206046, 1063206046)
>>> os.fstat (2)
(8592, 113L, 67174144L, 1, 1001, 4, 0L, 1063206048,
1063206048, 1063206048)
>>> os.fstat (3)
(4096, 0L, 0L, 0, 1001, 1001, 0L, 1063206027, 1063206027,
1063206027)
>>> os.fstat (4)
(4096, 0L, 0L, 0, 1001, 1001, 0L, 1063206027, 1063206027,
1063206027)
>>> os.fstat (5)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
OSError: [Errno 9] Bad file descriptor
>>>
Also, it is very strange that after calling os.fstat(3),
os.close(3) works. This says to me that fd 3 is fine, but
Python thinks fd 3 is bogus, but
calling os.fstat (3) forces Python to check fd 3's real status.
I considered the possibility that os.fstat(x) will the next
os.close(x) to succeed, but that is not the case:
[bash]$ cat test3.py
import os
os.fstat (3)
os.close (3)
os.fstat (3)
os.close (3)
[bash]$ python test3.py 3> file
(an the error is raised on the second os.fstat(3) call, just
as you would expect)
Okay, here is your ktrace:
26520 python GIO fd 6 read 23 bytes
"import os
os.close (3)
"
26520 python RET read 23/0x17
26520 python CALL write(0x2,0x80e6708,0x4)
26520 python GIO fd 2 wrote 4 bytes
" "
26520 python RET write 4
26520 python CALL write(0x2,0xbfbfeb70,0xd)
26520 python GIO fd 2 wrote 13 bytes
"os.close (3)
"
26520 python RET write 13/0xd
26520 python CALL fstat(0x6,0xbfbfe670)
26520 python RET fstat 0
26520 python CALL fcntl(0x6,0x3,0)
26520 python RET fcntl 4
26520 python CALL fcntl(0x6,0x4,0)
26520 python RET fcntl 0
26520 python CALL close(0x6)
26520 python RET close 0
26520 python CALL write(0x2,0x813a3b4,0x7)
26520 python GIO fd 2 wrote 7 bytes
"OSError"
26520 python RET write 7
26520 python CALL write(0x2,0x80dbe5f,0x2)
26520 python GIO fd 2 wrote 2 bytes
": "
26520 python RET write 2
26520 python CALL write(0x2,0x81c5dfc,0x1d)
26520 python GIO fd 2 wrote 29 bytes
"[Errno 9] Bad file descriptor"
26520 python RET write 29/0x1d
26520 python CALL write(0x2,0x80e4d42,0x1)
26520 python GIO fd 2 wrote 1 byte
"
"
Note that when I call os.close (3), python calls close(0x6),
so it looks like python is "mapping" the file description
numbers. There are many other close calls the ktrace, but
_none_ of them close (0x3). This says to me that (possibly)
python is doing lots of fd manipulation, but fd 3 is already
open, so python never uses it.
Here is a ktrace of test2.py. It's very interesting. 6 is
fstated, closed, then 3 is fstated, fnctled, and closed
successfully.
26498 python GIO fd 6 read 36 bytes
"import os
os.fstat (3)
os.close (3)
"
26498 python RET read 36/0x24
26498 python CALL lseek(0x6,0,0,0,0x1)
26498 python RET lseek 36/0x24
26498 python CALL read(0x6,0x8203000,0x4000)
26498 python GIO fd 6 read 0 bytes
""
26498 python RET read 0
26498 python CALL fstat(0x6,0xbfbff450)
26498 python RET fstat 0
26498 python CALL fcntl(0x6,0x3,0)
26498 python RET fcntl 4
26498 python CALL fcntl(0x6,0x4,0)
26498 python RET fcntl 0
26498 python CALL close(0x6)
26498 python RET close 0
26498 python CALL fcntl(0x3,0x3,0)
26498 python RET fcntl 1
26498 python CALL fcntl(0x3,0x4,0x5)
26498 python RET fcntl 0
26498 python CALL fstat(0x3,0xbfbff290)
26498 python RET fstat 0
26498 python CALL fstat(0x3,0xbfbff260)
26498 python RET fstat 0
26498 python CALL fcntl(0x3,0x3,0)
26498 python RET fcntl 5
26498 python CALL fcntl(0x3,0x4,0x1)
26498 python RET fcntl 0
26498 python CALL close(0x3)
26498 python RET close 0
26498 python CALL sigaction(0x2,0xbfbff4b0,0)
26498 python RET sigaction 0
26498 python CALL setitimer(0x2,0xbfbff5f0,0)
26498 python RET setitimer 0
26498 python CALL close(0x4)
26498 python RET close 0
26498 python CALL close(0x5)
26498 python RET close 0
26498 python CALL fcntl(0,0x3,0)
26498 python RET fcntl 6
26498 python CALL fcntl(0,0x4,0x2)
26498 python RET fcntl 0
26498 python CALL fcntl(0x1,0x3,0)
26498 python RET fcntl 2
26498 python CALL fcntl(0x1,0x4,0x2)
26498 python RET fcntl 0
26498 python CALL fcntl(0x2,0x3,0)
26498 python RET fcntl 2
26498 python CALL fcntl(0x2,0x4,0x2)
26498 python RET fcntl 0
26498 python CALL exit(0)
Does fd 3 have some special significance on FreeBSD? Or on
other OSes? I'm aware of 0, 1 and 2 being stdin, out, err.
But I didn't know that 3 was anything special. And 3 is
what courier uses to communicate with its filters.
Okay... I just searched through the whole ktrace of test.py.
As far as I can tell, nothing touches fd 3 in the whole
ktrace. fcntl takes 0x3 as the second argument many times,
but that is a command, not a file descriptor.
So... this means that with test.py, Python decides (based on
an fstat (0x6)?) that os.close(3) will fail without ever
calling close(3) to see what will happen.
This could be a bug in FreeBSD. But then why did my C
program work as expected? Or perhaps there is a library
inbetween Python and the OS, a library that my c program
does not use.
Any suggestions for further diagnosis?
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2003-09-10 20:19
Message:
Logged In: YES
user_id=21627
os.close directly calls the operating system call close(2),
and reports an error if and only if the operating system
reports an error. So there is absolutely zero possibility
that the operatin system lets close(2) succeeed, yet python
returns OSError(9). As a consequence, if python raises
OSError(9), it was the operating system that has returned an
error before.
If you doubt this statement, please use ktrace/strace to
diagnose this further.
Also, if you *know* that a parent process has set up fd 3,
your test case is irrelevant. Your test case is only
meaningful in a context where a parent process has set up fd
3, however, you have not reported the code of that parent
process. Providing a minimal test case would be appreciated.
----------------------------------------------------------------------
Comment By: Syrah (syrah)
Date: 2003-09-10 17:33
Message:
Logged In: YES
user_id=827529
I'm writing a mail filter for the courier mail server.
Before courier forks the filter, courier sets up fd 3 to
communicate with the filter. When the filter has finished
initialization and is ready for work, the filter is supposed
to close fd 3. Courier detects the closure of fd 3 and
knows that the filter is ready to work.
In general, before a call to fork/exec, the parent process
can set up any number of file descriptors for the child to
inherit. If you are writing such a child in C, it is easy
to use these "extra" file descriptors. It would nice to be
able to write in Python instead.
Saying that "3 *is* a bad file descriptor" assumes that
every program will be run from the command line, and the the
command line will only set up fd's 0, 1 and 2. This is not
the case. In fact it is very easy to set up other file
descriptors on the command line using bash. See my example
above.
Another example of using more than 0, 1 and 2. The openssl
program can use arbitrary file descriptors (referenced by
number on the command line) to recieve passwords. This
allows you to cleanly pass in multiple passwords in a
reasonably secure manner. (Arguably more secure than
writing the passwords to a file, and definitely more secure
that specifying them on the command line.)
The problem manifest on FreeBSD. 3 is not always a bad fd
on FreeBSD. I've got a C program that has no problem
closing fd 3 (when fd 3 is properly set up).
Moreover, as I pointed out above, if I call os.fstat (3)
first, then I can os.close (3) without problem. If 3 was a
bad fd, the call to os.fstat (3) should generate an error.
It does not.
Please let me know if you have further questions.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2003-09-10 16:57
Message:
Logged In: YES
user_id=21627
Why is this a bug? 3 *is* a bad file descriptor, on the
systems which report it to be a bad file descriptor.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=803610&group_id=5470
More information about the Python-bugs-list
mailing list