[Python-bugs-list] [ python-Bugs-633480 ] IOError w/ large string to CMD shell

[noreply@sourceforge.net]

Bugs item #633480, was opened at 2002-11-04 16:06
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=633480&group_id=5470

Category: Windows
Group: 3rd Party
>Status: Closed
>Resolution: Wont Fix
Priority: 4
Submitted By: Michael McCandless (mikemccand)
Assigned to: Tim Peters (tim_one)
Summary: IOError w/ large string to CMD shell

Initial Comment:

If I try to run this program with "python2.2.1 -u":

  # 61409 works!!
  print ' ' * 61410

(which tries to print a string of 61,410 space characters), 
I get this Traceback on Windows 2000 SP2:

  Traceback (most recent call last):
    File "write.py", line 4, in ?
      print ' ' * 61410
  IOError: [Errno 12] Not enough space

Without "-u", this runs fine, and if I change "print" 
to "import sys; sys.stdout.write" instead, it still fails.

I think this is happening in Objects/fileobject.c -- the 
calls to fwrite apparently don't write all bytes in win32 
when the number of bytes is large -- seems like it may 
be necessary to parcel the data up into smaller pieces 
or something?

NOTE: it's possible that the exact length of the string 
that fails may be larger on other OS's...

----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2002-11-07 12:10

Message:
Logged In: YES 
user_id=31435

I'm attaching a short C program showing that the same thing 
happens in straight C.  But there's a surprise:  turns out it 
has nothing to do with buffered versus unbuffered.  It's 
changing stdout to binary mode that does the damage.  This 
is also specific to writing to my Win2K console window 
("DOS box"); if cmdline redirection is used to redirect stdout 
to a file instead, it doesn't fail.

So, as threatened, closing this as WontFix -- if there's "a 
bug" here at all, it's in Microsoft's code.  They may well claim 
it's a design limitation, if it were possible to talk with a human 
being there <wink..

----------------------------------------------------------------------

Comment By: Michael McCandless (mikemccand)
Date: 2002-11-06 06:47

Message:
Logged In: YES 
user_id=323786


Tim,

Okay -- that sounds like the right approach here -- thanks 
again.  Don't think I'll be upgrading to Win98 too soon :)  And 
I don't have high hopes on getting a fix out of MSFT :)

Mike

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-11-05 23:55

Message:
Logged In: YES 
user_id=31435

The thing is, if you want to do this, you're going to *have* to 
worm around it.  I've got no magic to offer -- it's a vanilla C 
library call, and there's no sign that Python has anything to 
do with the failure.  Try doing a staight fwrite() to stdout, 
unbuffered, in C, and I expect you'll see the same thing (or 
if not, then *maybe* Python had something to do with it).

If it does fail in straight C, then filing a bug report with 
Microsoft is appropriate.  Or you could upgrade to Win98 
<wink>.

Assigning to me and marking 3rdParty; I expect to close it 
as WontFix unless evidence appears that not a Windows 
bug.

BTW, you may have better luck installing the Python Win32 
extensions and using the Win32 file API directly.  
Sometimes that works better than MS's std-C I/O 
implementation.

----------------------------------------------------------------------

Comment By: Michael McCandless (mikemccand)
Date: 2002-11-05 08:29

Message:
Logged In: YES 
user_id=323786

Thanks for looking into this Tim...

True, we can workaround it -- either don't run UNBUFFERED, 
or break up the strings before printing (override sys.stdout w/ 
our own file object that tries to break things up before 
printing), or don't run under "CMD" or other platform-specific 
shells that have this issue.

But it's rather unexpected and to a "Python newbie" would 
be "surprising"; though I suppose most would not run with "-
u".

Also, from the IOError that's raised I don't know how many 
bytes were actually successfully written (apparently 0 in this 
case).

NOTE: I'm able to reproduce on Windows XP Professional 
(no SP), running "cmd.exe", Python 2.2.2.  I've also tried 
changing the "screen buffer" sizes on the CMD and it does 
not seem to change things.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2002-11-04 16:34

Message:
Logged In: YES 
user_id=31435

Couldn't reproduce under Win98SE and 2.2.2 even 
boosting the size to a million.  It *may* have to do with 
console-size settings under cmd.exe.

Reduced the priority.  "So don't do that" comes to mind 
<wink>.  Seriously, if the platform C fwrite chokes, it's not 
Python's job to fix it -- it's Microsoft's.  The interpreter isn't 
dying here, it's raising a well-behaved exception, just 
passing on the error the platform fwrite() raised.  To me it's 
the same as the system malloc complaining that you don't 
have enough memory to satisfy a request:  you've 
exceeded a platform limitation.  If this is important to you, 
you can presumably fix it yourself by chunking up your 
writes in a loop.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=633480&group_id=5470