[Python-bugs-list] [ python-Bugs-803610 ] os.close(3) raises OSError: [Errno 9] Bad file descriptor

SourceForge.net noreply at sourceforge.net
Thu Sep 11 19:13:11 EDT 2003


Bugs item #803610, was opened at 2003-09-10 06:18
Message generated for change (Comment added) made by syrah
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=803610&group_id=5470

Category: Python Library
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Syrah (syrah)
Assigned to: Nobody/Anonymous (nobody)
Summary: os.close(3) raises OSError: [Errno 9] Bad file descriptor

Initial Comment:
os.close(3) raises OSError: Bad file descriptor on

FreeBSD 5.1, with both Python 2.2.2(_2) and 2.3(_1). 

Python 2.3 on Gentoo Linux 1.4 is not affected.



Interestingly, if you call os.fstat(3) first, a

subsequent call to os.close(3) succeeds.



And yes, you do need to set up fd #3.



Here is a demonstration (on FreeBSD):



[syrah at ripple filter]$ cat test.py

import os

os.close (3)



[syrah at ripple filter]$ python test.py 3> file

Traceback (most recent call last):

  File "test.py", line 2, in ?

    os.close (3)

OSError: [Errno 9] Bad file descriptor



[syrah at ripple filter]$ cat test2.py

import os

os.fstat (3)

os.close (3)



[syrah at ripple filter]$ python test2.py 3> file



[syrah at ripple filter]$   (success!)



----------------------------------------------------------------------

>Comment By: Syrah (syrah)
Date: 2003-09-12 01:13

Message:
Logged In: YES 
user_id=827529

Well... I just got a FreeBSD 4.8 machine running.  (FreeBSD

5.1 is billed as experimental.)



The problem does NOT manifest on FreeBSD 4.8.  They are both

running Python 2.3.  (Although the 5.1 machine has

portrevision 1, whereas the 4.8 machine has portrevision 0.

 But they use the same sourcecode tarball.)



So... I'm willing to close out the bug report and assume

that the bug is in FreeBSD 5.1 somewhere.  If in the future

I determine otherwise, I'll reopen the bug then.



Let me know what you think.  Many thanks for all your help.

----------------------------------------------------------------------

Comment By: Syrah (syrah)
Date: 2003-09-12 00:55

Message:
Logged In: YES 
user_id=827529

I think fd6 is created in the pipe call.  But why is pipe

returning 4?  It should return 0 or -1, shouldn't it?



Do I need to recomiple python (so that debug info is added

to the executable) to attach the debugger to it?



----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-09-11 22:06

Message:
Logged In: YES 
user_id=21627

I think you are misinterpreting the ktrace output. fd 6 is

the source code file, test.py, which can be seen by looking

at the read result for fd 6. Unfortunately, there is no

corresponding open call opening the file... (there is a

second open call for test.py, when Python opens the file to

print the backtrace, but there should be an earlier call).



If this ktrace output can be trusted, it appears that Python

does not call os.open(3) at all, but immediately decides on

an error - which could happen if errno was somehow

overwritten. However, I doubt that ktrace output, since it

fails to report the first open of test.py, which I know must

be there.



I recommend to run this under a debugger, and break at

posix_open.

----------------------------------------------------------------------

Comment By: Syrah (syrah)
Date: 2003-09-11 16:50

Message:
Logged In: YES 
user_id=827529

Okay.  I think I have attached a tgz file with 3 programs, 3

binary ktraces, and the ktraces converted to text files.



In all cases, the programs were run as follows:



[bash]$ program 3> file



Many thanks for your continued interest.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-09-11 06:01

Message:
Logged In: YES 
user_id=21627

Was that the complete ktrace? I'm missing the point where it

outputs "Traceback (most recent call last):". If you have

stripped it down, can you please attach the entire thing

(please don't paste it)?

----------------------------------------------------------------------

Comment By: Syrah (syrah)
Date: 2003-09-10 22:45

Message:
Logged In: YES 
user_id=827529

I agree that this is a very strange error.  Python should be

directly calling the operating systems close function.  It

is very puzzling that there is an error.



I did include in my initial post a parent that sets up fd 3.

 I used bash.  I was able to test successfully with bash.  I

wrote a C program:



-------- test.c --------

#include <stdio.h>



int main () {

  int i = close (3);

  printf ("close: %i\n", i); }

--------



Using bash, I tested test.py, test2.py, and test.c (gcc

test.c -o test).  test.c behaved properly.  When I set up fd

3 with bash, test.c printed "close: 0".  When I did not set

up fd 3, test.c printed "close: -1".



I therefore have bash working properly, C working properly

and test2.py working properly.  The only thing that is not

working properly is test.py.  To me, it seems that I have

isolated the problem.



However, this problem only manifests on FreeBSD.  test.py

runs fine on Gentoo Linux.  At present, those are the only

two OSes I have easy access to.



So... if you really want a test program that sets up fd3,

I'll write one - let me know.  Would you prefer me to write

it in Python, C, PHP, Ruby, Perl?  I'd prefer C as it is

most direct.  But bash seems sufficient: "[bash]$ python

test.py 3> file" sets up file descriptor 3 so it writes into

a file named "file".



To look at it from another angle... is it possible that

Python, during its startup and initialization stuff messes

with file descriptor 3?  Is it possible it does this in some

really non-standard way?  Perhaps it does it indirectly by

calling some OS function that, as a side effect on FreeBSD

touches fd 3?  Something is going on with file descriptions

0-4 _every_ time python runs.  [Based on the ktrace's below,

it looks like



[syrah at ripple]$ python

Python 2.3 (#1, Sep  9 2003, 22:47:18) 

[GCC 3.2.2 [FreeBSD] 20030205 (release)] on freebsd5

Type "help", "copyright", "credits" or "license" for more

information.

>>> import os 

>>> os.fstat (0)

(8592, 113L, 67174144L, 1, 1001, 4, 0L, 1063206044,

1063206044, 1063206044)

>>> os.fstat (1)

(8592, 113L, 67174144L, 1, 1001, 4, 0L, 1063206046,

1063206046, 1063206046)

>>> os.fstat (2)

(8592, 113L, 67174144L, 1, 1001, 4, 0L, 1063206048,

1063206048, 1063206048)

>>> os.fstat (3)

(4096, 0L, 0L, 0, 1001, 1001, 0L, 1063206027, 1063206027,

1063206027)

>>> os.fstat (4)

(4096, 0L, 0L, 0, 1001, 1001, 0L, 1063206027, 1063206027,

1063206027)

>>> os.fstat (5)

Traceback (most recent call last):

  File "<stdin>", line 1, in ?

OSError: [Errno 9] Bad file descriptor

>>> 



Also, it is very strange that after calling os.fstat(3),

os.close(3) works.  This says to me that fd 3 is fine, but

Python thinks fd 3 is bogus, but

calling os.fstat (3) forces Python to check fd 3's real status.



I considered the possibility that os.fstat(x) will the next

os.close(x) to succeed, but that is not the case:



[bash]$ cat test3.py

import os

os.fstat (3)

os.close (3)

os.fstat (3)

os.close (3)



[bash]$ python test3.py 3> file

(an the error is raised on the second os.fstat(3) call, just

as you would expect)



Okay, here is your ktrace:



26520 python   GIO   fd 6 read 23 bytes

       "import os

        os.close (3)

       "

 26520 python   RET   read 23/0x17

 26520 python   CALL  write(0x2,0x80e6708,0x4)

 26520 python   GIO   fd 2 wrote 4 bytes

       "    "

 26520 python   RET   write 4

 26520 python   CALL  write(0x2,0xbfbfeb70,0xd)

 26520 python   GIO   fd 2 wrote 13 bytes

       "os.close (3)

       "

 26520 python   RET   write 13/0xd

 26520 python   CALL  fstat(0x6,0xbfbfe670)

 26520 python   RET   fstat 0

 26520 python   CALL  fcntl(0x6,0x3,0)

 26520 python   RET   fcntl 4

 26520 python   CALL  fcntl(0x6,0x4,0)

 26520 python   RET   fcntl 0

 26520 python   CALL  close(0x6)

 26520 python   RET   close 0

 26520 python   CALL  write(0x2,0x813a3b4,0x7)

 26520 python   GIO   fd 2 wrote 7 bytes

       "OSError"

 26520 python   RET   write 7

 26520 python   CALL  write(0x2,0x80dbe5f,0x2)

 26520 python   GIO   fd 2 wrote 2 bytes

       ": "

 26520 python   RET   write 2

 26520 python   CALL  write(0x2,0x81c5dfc,0x1d)

 26520 python   GIO   fd 2 wrote 29 bytes

       "[Errno 9] Bad file descriptor"

 26520 python   RET   write 29/0x1d

 26520 python   CALL  write(0x2,0x80e4d42,0x1)

 26520 python   GIO   fd 2 wrote 1 byte

       "

       "



Note that when I call os.close (3), python calls close(0x6),

so it looks like python is "mapping" the file description

numbers.  There are many other close calls the ktrace, but

_none_ of them close (0x3).  This says to me that (possibly)

python is doing lots of fd manipulation, but fd 3 is already

open, so python never uses it.



Here is a ktrace of test2.py.  It's very interesting.  6 is

fstated, closed, then 3 is fstated, fnctled, and closed

successfully.



 26498 python   GIO   fd 6 read 36 bytes

       "import os

        os.fstat (3)

        os.close (3)

       "

 26498 python   RET   read 36/0x24

 26498 python   CALL  lseek(0x6,0,0,0,0x1)

 26498 python   RET   lseek 36/0x24

 26498 python   CALL  read(0x6,0x8203000,0x4000)

 26498 python   GIO   fd 6 read 0 bytes

       ""

 26498 python   RET   read 0

 26498 python   CALL  fstat(0x6,0xbfbff450)

 26498 python   RET   fstat 0

 26498 python   CALL  fcntl(0x6,0x3,0)

 26498 python   RET   fcntl 4

 26498 python   CALL  fcntl(0x6,0x4,0)

 26498 python   RET   fcntl 0

 26498 python   CALL  close(0x6)

 26498 python   RET   close 0

 26498 python   CALL  fcntl(0x3,0x3,0)

 26498 python   RET   fcntl 1

 26498 python   CALL  fcntl(0x3,0x4,0x5)

 26498 python   RET   fcntl 0

 26498 python   CALL  fstat(0x3,0xbfbff290)

 26498 python   RET   fstat 0

 26498 python   CALL  fstat(0x3,0xbfbff260)

 26498 python   RET   fstat 0

 26498 python   CALL  fcntl(0x3,0x3,0)

 26498 python   RET   fcntl 5

 26498 python   CALL  fcntl(0x3,0x4,0x1)

 26498 python   RET   fcntl 0

 26498 python   CALL  close(0x3)

 26498 python   RET   close 0

 26498 python   CALL  sigaction(0x2,0xbfbff4b0,0)

 26498 python   RET   sigaction 0

 26498 python   CALL  setitimer(0x2,0xbfbff5f0,0)

 26498 python   RET   setitimer 0

 26498 python   CALL  close(0x4)

 26498 python   RET   close 0

 26498 python   CALL  close(0x5)

 26498 python   RET   close 0

 26498 python   CALL  fcntl(0,0x3,0)

 26498 python   RET   fcntl 6

 26498 python   CALL  fcntl(0,0x4,0x2)

 26498 python   RET   fcntl 0

 26498 python   CALL  fcntl(0x1,0x3,0)

 26498 python   RET   fcntl 2

 26498 python   CALL  fcntl(0x1,0x4,0x2)

 26498 python   RET   fcntl 0

 26498 python   CALL  fcntl(0x2,0x3,0)

 26498 python   RET   fcntl 2

 26498 python   CALL  fcntl(0x2,0x4,0x2)

 26498 python   RET   fcntl 0

 26498 python   CALL  exit(0)



Does fd 3 have some special significance on FreeBSD?  Or on

other OSes?  I'm aware of 0, 1 and 2 being stdin, out, err.

 But I didn't know that 3 was anything special.  And 3 is

what courier uses to communicate with its filters.



Okay... I just searched through the whole ktrace of test.py.

 As far as I can tell, nothing touches fd 3 in the whole

ktrace.  fcntl takes 0x3 as the second argument many times,

but that is a command, not a file descriptor.



So... this means that with test.py, Python decides (based on

an fstat (0x6)?) that os.close(3) will fail without ever

calling close(3) to see what will happen.



This could be a bug in FreeBSD.  But then why did my C

program work as expected?  Or perhaps there is a library

inbetween Python and the OS, a library that my c program

does not use.



Any suggestions for further diagnosis?

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-09-10 20:19

Message:
Logged In: YES 
user_id=21627

os.close directly calls the operating system call close(2),

and reports an error if and only if the operating system

reports an error. So there is absolutely zero possibility

that the operatin system lets close(2) succeeed, yet python

returns OSError(9). As a consequence, if python raises

OSError(9), it was the operating system that has returned an

error before.



If you doubt this statement, please use ktrace/strace to

diagnose this further.



Also, if you *know* that a parent process has set up fd 3,

your test case is irrelevant. Your test case is only

meaningful in a context where a parent process has set up fd

3, however, you have not reported the code of that parent

process. Providing a minimal test case would be appreciated.

----------------------------------------------------------------------

Comment By: Syrah (syrah)
Date: 2003-09-10 17:33

Message:
Logged In: YES 
user_id=827529

I'm writing a mail filter for the courier mail server.



Before courier forks the filter, courier sets up fd 3 to

communicate with the filter.  When the filter has finished

initialization and is ready for work, the filter is supposed

to close fd 3.  Courier detects the closure of fd 3 and

knows that the filter is ready to work.



In general, before a call to fork/exec, the parent process

can set up any number of file descriptors for the child to

inherit.  If you are writing such a child in C, it is easy

to use these "extra" file descriptors.  It would nice to be

able to write in Python instead.



Saying that "3 *is* a bad file descriptor" assumes that

every program will be run from the command line, and the the

command line will only set up fd's 0, 1 and 2.  This is not

the case.  In fact it is very easy to set up other file

descriptors on the command line using bash.  See my example

above.



Another example of using more than 0, 1 and 2.  The openssl

program can use arbitrary file descriptors (referenced by

number on the command line) to recieve passwords.  This

allows you to cleanly pass in multiple passwords in a

reasonably secure manner.  (Arguably more secure than

writing the passwords to a file, and definitely more secure

that specifying them on the command line.)



The problem manifest on FreeBSD.  3 is not always a bad fd

on FreeBSD.  I've got a C program that has no problem

closing fd 3 (when fd 3 is properly set up).



Moreover, as I pointed out above, if I call os.fstat (3)

first, then I can os.close (3) without problem.  If 3 was a

bad fd, the call to os.fstat (3) should generate an error. 

It does not.



Please let me know if you have further questions.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-09-10 16:57

Message:
Logged In: YES 
user_id=21627

Why is this a bug? 3 *is* a bad file descriptor, on the

systems which report it to be a bad file descriptor.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=803610&group_id=5470



More information about the Python-bugs-list mailing list