[ python-Bugs-1189525 ] file.write(x) where len(x) > 64*1024**2 is unreliable

[SourceForge.net]

Bugs item #1189525, was opened at 2005-04-25 10:51
Message generated for change (Comment added) made by tim_one
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1189525&group_id=5470

Category: Windows
Group: 3rd Party
Status: Closed
Resolution: Wont Fix
Priority: 5
Submitted By: Martin Gfeller (gfe)
Assigned to: Nobody/Anonymous (nobody)
Summary: file.write(x) where len(x) > 64*1024**2 is unreliable

Initial Comment:
Large file writes (over 64MB) are unreliable under 
Windows.

On my Windows 2000 SP4, it succeeds when writing to 
a file on a local driver, or on a network drive.

Howver, if I map the share of the local drive c$ as 
another drive, the same write fails with:

IOError: [Errno 22] Invalid argument


Filemon (www.sysinternals.com) shows that this is a 
STATUS_INSUFFICIENT_RESOURCES returned by 
IRP_MJ_WRITE.

Although this is probably a Windows bug, my 
suggestion is that Python should abstract from such 
problems, and break the writes into suitable chunks.

Best regards,
Martin 

----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2005-04-25 13:18

Message:
Logged In: YES 
user_id=31435

Python hides all the rough edges it can, where "can" means 
a combination of knowing in advance precisely where rough 
edges are, and of workarounds being practical.

In this case I'd _guess_ that it really has nothing to do with 
fwrite(), it has to do with limitations introduced by specific 
ways of mapping drives.  For all I know (and I don't know, one 
way or the other), Microsoft does document such limitations 
somewhere.  If so, they'll say it's a limitation, and not a bug.  
That's common enough.  It may even be that there's some 
non-obvious knob you could change on Windows to increase 
whatever resource it is that you're running out of when writing 
> 64MB to a map of a share of the local c$ drive.

But Python isn't the OS regardless, and there are hundreds 
of OS and platform C limitations Python can't sanely hide.  
This looks like one of 'em.

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2005-04-25 13:14

Message:
Logged In: YES 
user_id=849994

Well, when there's no bug documented on the Windows side,
Python cannot work around it.

----------------------------------------------------------------------

Comment By: Martin Gfeller (gfe)
Date: 2005-04-25 13:03

Message:
Logged In: YES 
user_id=884167

As you confirm that fwrite() has no documented size 
limitations, it must be a Windows bug. 

So I will need to find a more practical solution instead of 
assuming that Python will protect me from the rough edges of 
OS. 

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2005-04-25 12:48

Message:
Logged In: YES 
user_id=31435

Python's file.write() just calls the platform C's fwrite(), and 
MS's fwrite() docs don't admit to any size limitations.

If this is a documented bug on Windows, then your complaint 
is with Microsoft.

If it's not documented, how is anyone supposed to guess 
what "suitable" means in all cases?

IOW, this report is either misdirected, or impossible to fix 
sanely.  Sorry, but I'm closing as "3rd Party, WontFix" on 
these grounds.

If you care enough to research it in depth and create a patch, 
that may be a different story.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1189525&group_id=5470