Integer overflow in fcntl on some Linux platforms. What's going on?

Noah noah at
Mon Sep 9 16:08:00 EDT 2002


Can anyone help me figure out what the problem is here?
On some Linux platforms when I try to execute the following code:

    s = struct.pack("HHHH", 24, 80, 0, 0)
    x = fcntl.ioctl(sys.stdout.fileno(), termios.TIOCSWINSZ, s)

I get this exception:

    'signed integer is greater than maximum'

It seems that Linux does not like the integer from sys.stdout.fileno(), but 
this has worked fine on all the BSD machines and one Linux 2.2.20 machine 
I tested it on. So far only RedHat 7.1 has this problem.
Or maybe it does not like the constant termios.TIOCSWINSZ (set win size), yet
it has no problem with the similar constant termios.TIOCGWINSZ (get win size).

The following is a test script that demonstrates this problem. 
It logs all steps to a file called LOG. 
It crashes after STEP 5 inside getwinsize() at STEP getwinsize() 2. 
This line is the ioctl() call. This is the results I get in my LOG file:
        STEP 1: pid: 274968
        STEP 2: stdout.fileno(): 1
        STEP 3: stdout.fileno() hex: 1
        STEP 4: termios.TIOCSWINSZ: 2148037735
        STEP getwinsize() 1.
        STEP getwinsize() 2.
        STEP getwinsize() 3.
        STEP 5: getwinsize() #1: (54, 80, 0, 0)
        STEP setwinsize() 1.
        STEP setwinsize() 2.
        EXCEPTION: signed integer is greater than maximum

Any help would be appreciated.


#!/usr/bin/env python
import os, sys, termios, fcntl
import struct

def setwinsize():
        # Assume ws_xpixel and ws_ypixel are zero.
        log ('STEP setwinsize() 1.')
        s = struct.pack("HHHH", 24, 80, 0, 0)
        log ('STEP setwinsize() 2.')
        x = fcntl.ioctl(sys.stdout.fileno(), termios.TIOCSWINSZ, s)
        log ('STEP setwinsize() 3.')

def getwinsize():
        log ('STEP getwinsize() 1.')
        s = struct.pack("HHHH", 0, 0, 0, 0)
        log ('STEP getwinsize() 2.')
        x = fcntl.ioctl(sys.stdout.fileno(), termios.TIOCGWINSZ, s)
        log ('STEP getwinsize() 3.')
        return struct.unpack("HHHH", x)

def log (s):
        fout = open ('LOG','a+')

pid = os.getpid()

log ('STEP 1: pid: ' + str (pid) )
log ('STEP 2: stdout.fileno(): ' + str(sys.stdout.fileno()) )
log ('STEP 3: stdout.fileno() hex: %x' % sys.stdout.fileno() )
log ('STEP 4: termios.TIOCSWINSZ: ' + str(termios.TIOCSWINSZ) )
        log ('STEP 5: getwinsize() #1: ' + str (getwinsize()))
        log ('STEP 6: getwinsize() #2: ' + str (getwinsize()))
except Exception, e:
        log ('EXCEPTION: ' + str(e))

print 'Log file written. It is called "LOG".'
print open('LOG').read()

More information about the Python-list mailing list