[Python-bugs-list] [ python-Bugs-603724 ] setting file buffer size is unreliable

[SourceForge.net]

Bugs item #603724, was opened at 2002-09-03 00:37
Message generated for change (Comment added) made by loewis
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=603724&group_id=5470

Category: Python Interpreter Core
Group: Python 2.3
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Jeremy Yallop (yallop)
Assigned to: Nobody/Anonymous (nobody)
Summary: setting file buffer size is unreliable

Initial Comment:
The description of open() (ie file()) says:



   "If the buffering argument is given, 0 means

unbuffered, 1 means line buffered, and larger numbers

specify the buffer size."



PyFile_SetBufSize() passes the requested buffer size on

to setvbuf(), with NULL as setvbuf()'s second

parameter.  The C89 standard doesn't guarantee any

change to the buffer size when the second parameter is

NULL, and some stdio implementations (legitimately)

ignore the size parmater in such circumstances.  C99's

gives more guidelines, but nothing that can be relied upon:



   "If buf is not a null pointer, the array it points

to may be used instead of a buffer allocated by the

setvbuf function and the argument size     specifies

the size of the array; otherwise, size may determine

the size of  buffer allocated by the setvbuf function."

(7.19.5.6)



(What good is "may" to anyone?)



The result of all this is that



  fd = open('file', 'w', 8)



will not have the desired (and documented) effect

(flushing the output buffer every 8 characters) on some

platforms, so either the documentation of open() or the

code (PyFile_SetBufSize()) should be fixed.



The same problems exist with setbuf() as well.

----------------------------------------------------------------------

>Comment By: Martin v. Löwis (loewis)
Date: 2003-09-04 21:04

Message:
Logged In: YES 
user_id=21627

Fixed with #788249, in



fileobject.h 2.33

fileobject.c 2.181

fileobject.h 2.32.8.1

NEWS 1.831.4.33

fileobject.c 2.179.8.1



----------------------------------------------------------------------

Comment By: Andrew Gaul (gaul)
Date: 2003-08-22 10:48

Message:
Logged In: YES 
user_id=139865

See patch 788249 for a fix.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-09-12 10:12

Message:
Logged In: YES 
user_id=21627

I see, it appears indeed that the size argument is unused on

Linux.



Any volunteers to come up with a patch?

----------------------------------------------------------------------

Comment By: Jeremy Yallop (yallop)
Date: 2002-09-11 22:49

Message:
Logged In: YES 
user_id=479330

Debian on x86 with glibc-2.2.5 (which I believe uses Per

Bothner's libio) ignores the size parameter when the buffer

parameter is NULL.  I've attached a demonstration program

which calls setvbuf with a size of 2 and with either NULL or

a static buffer (depending on argc). Run it and kill (with

Ctrl-C or whatever) before it's finished.



Alternatively, the effect can be observed in python by

something like



  >>> fd = open('file', 'write', 2)

  >>> fd.write('Hello, world.\n')

  >>>

  Ctrl-Z

  [1]+  Stopped                 python

  $ killall -9 python

  $ cat file

  $



PS I'm not sure declaring Debian as unsupported would be

very popular :-).

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-09-11 18:56

Message:
Logged In: YES 
user_id=21627

Can you please report which systems ignore the setvbuf call?



Python makes loads of assumptions about systems which are

not backed by any standard. On systems which meet Python's

expectation, Python behaves as documented.



On systems which don't meet the expectations, we have two

options

a) declare the system as unsupported, or

b) fix the implementation for the specific system to meet

the documentation.



So in the abstract, I don't see a bug here - setting the

buffer size *is* reliable on Linux and Windows and many

other supported systems.

----------------------------------------------------------------------

Comment By: Jeremy Hylton (jhylton)
Date: 2002-09-04 00:35

Message:
Logged In: YES 
user_id=31392

The C99 rationale has a helpful comment on the subject of

setvbuf():



  C90 was not clear about what, if anything, the size

argument means

  when buf is a null pointer.  Existing practice is mixed: some

  implementations ignore it completely, other

implementations use it as

  guidance for determining the size of the buffer allocated

by setvbuf.

  C9X gives warning that size might be ignored in this case,

so portable

  programs must be sure to supply a reasonable value.



I'd be mostly inclined to change the documentation to say

that the buffer is under control of the C library and that

Python does whatever setvbuf() does.  Or however we write

the C99 weasel words in Python docstring-ease.



Thankfully, Linux and Windows both have sane setvbuf()

implementations.



----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=603724&group_id=5470