[ python-Bugs-1174606 ] Reading /dev/zero causes SystemError

[SourceForge.net]

Bugs item #1174606, was opened at 2005-04-01 04:48
Message generated for change (Comment added) made by jafo
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1174606&group_id=5470

Category: Python Interpreter Core
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Adam Olsen (rhamphoryncus)
Assigned to: Nobody/Anonymous (nobody)
Summary: Reading /dev/zero causes SystemError

Initial Comment:
$ python -c 'open("/dev/zero").read()'
Traceback (most recent call last):
  File "<string>", line 1, in ?
SystemError: ../Objects/stringobject.c:3316: bad
argument to internal function

Compare with this two variants:

$ python -c 'open("/dev/zero").read(2**31-1)'
Traceback (most recent call last):
  File "<string>", line 1, in ?
MemoryError

$ python -c 'open("/dev/zero").read(2**31)'
Traceback (most recent call last):
  File "<string>", line 1, in ?
OverflowError: long int too large to convert to int

The unsized read should produce either MemoryError or
OverflowError instead of SystemError.

Tested with Python 2.2, 2.3, and 2.4.

----------------------------------------------------------------------

>Comment By: Sean Reifschneider (jafo)
Date: 2005-04-06 06:52

Message:
Logged In: YES 
user_id=81797

Linux can do a very fast allocation if it has swap
available.  It reserves space, but does not actually assign
the memory until you try to use it.  In my case, I have 1GB
of RAM, around 700MB free, and another 2GB in swap.  So, I
have plenty unless I use it.  In C I can malloc 1GB and
unless I write every page in that block the system doesn't
really give the pages to the process.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2005-04-06 06:40

Message:
Logged In: YES 
user_id=21627

The problem is different. Instead, _PyString_Resize
complains that the new buffersize of the string is negative.
This in turn happens because the string manages to get
larger >2GB, which in turn happens because buffersize is
size_t, yet _PyString_Resize expects int.

I don't know how Linux manages to allocate such a large
string without thrashing.

There is a minor confusion with stat() as well:
new_buffersize tries to find out how much bytes are left to
the end of the file. In the case of /dev/zero, both fstat
and lseek are "lying" by returning 0. As lseek returns 0,
ftell is invoked and returns non-zero. Then, newbuffer does
not trust the values, and just adds BIGCHUNK.

----------------------------------------------------------------------

Comment By: Sean Reifschneider (jafo)
Date: 2005-04-06 03:39

Message:
Logged In: YES 
user_id=81797

I am able to reproduce this on a Fedora Core 3 Linux system:

>>> fp = open('/dev/zero', 'rb')
>>> d = fp.read()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
MemoryError
>>> print os.stat('/dev/zero').st_size
0

What about only trusting st_size if the file is a regular
file, not a directory or other type of special file?

Sean

----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2005-04-02 12:31

Message:
Logged In: YES 
user_id=4771

os.stat() doesn't always give consistent results on dev files.  On my machine for some reason os.stat('/dev/null') appears to be random (and extremely large).  I suspect that on the OP's machine os.stat('/dev/zero') is not 0 either, but a random number that turns out to be negative, hence a "bad argument" SystemError.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2005-04-01 21:42

Message:
Logged In: YES 
user_id=21627

I think it should trust the stat result, and then find that
it cannot allocate that much memory.

Actually, os.stat("/dev/zero").st_size is 0, so something
else must be going on.

----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2005-04-01 09:58

Message:
Logged In: YES 
user_id=4771

I think that file.read() with no argument needs to be more conservative.  Currently it asks and trusts the stat() to get the file size, but this can lead to just plain wrong results on special devices.  (I had the problem that open('/dev/null').read() would give a MemoryError!)

We can argue whether plain read() on special devices is a good idea or not, but I guess that not blindly trusting stat() if it returns huge values could be a good idea.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1174606&group_id=5470