[ python-Bugs-1478529 ] size limit exceeded for read() from network drive

[SourceForge.net]

Bugs item #1478529, was opened at 2006-04-28 12:46
Message generated for change (Comment added) made by tim_one
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1478529&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Windows
>Group: 3rd Party
>Status: Closed
>Resolution: Wont Fix
Priority: 5
Submitted By: Mark Sheppard (markshep)
Assigned to: Nobody/Anonymous (nobody)
Summary: size limit exceeded for read() from network drive

Initial Comment:
If you've got a network share mounted as a local drive
then Windows has a limit of 67,076,095 (0x03ff7fff)
bytes for a read from an open file on that drive.

Running the python read() method on an open file larger
than this size throws an "IOError: [Errno 22] Invalid
argument" exception.

A fix would be for python to internally use multiple
reads so as to not exceed this limit.


----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2006-05-02 21:45

Message:
Logged In: YES 
user_id=31435

Sorry, I'm closing this as "3rd Party, Won't Fix".  It's
certainly not Python's doing that Microsoft's stdio
implementation has peculiar undocumented warts (Python just
calls the platform C's fread() here), so at best this is a
request for enhancement rather than a Python bug.

If there is a bug here, it's Microsoft's bug, and then the
proper source for a fix is also Microsoft.  This is
especially true since the two people who have tried this
here don't see the same behavior -- we don't even know what
"the bug" is.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2006-05-02 15:00

Message:
Logged In: YES 
user_id=21627

I could reproduce the write problem on XPSP2; I get the
Win32 error ERROR_NO_SYSTEM_RESOURCES after fwrite returns
(from GetLastError).

I can't reproduce the fread problem, though: in Python,
f.read(90*2**20) just returns with a 90MiB string. So it
could be a limitation of your machine (e.g. it might not
have enough memory), or of the server machine. I'm hesitant
to add a work-around for that into Python if this isn't a
system limitation. Performing multiple reads is also bad:
what happens if the first read succeeds, and the second one
fails? It might be that the system *really* is out of resources.

----------------------------------------------------------------------

Comment By: Mark Sheppard (markshep)
Date: 2006-05-02 06:48

Message:
Logged In: YES 
user_id=1512331

I'm running Windows XP.  I've been unable to find any
documentation about this exact problem - only that fwrite
thing.  But my testing shows that it works if I do
file.read(67076095), but throws an exception with
file.read(67076096).

I'm not suggesting limiting all reads from Python.  All I'm
suggesting is that under the hood the Windows implementation
of Python's read() call actually uses multiple fread() (or
whatever) calls if more than 67076095 bytes need to be read.
 That's all.  No interface changes.


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-04-30 12:23

Message:
Logged In: YES 
user_id=31435

Martin, here's an MS article seemingly related to this:

http://support.microsoft.com/default.aspx?scid=kb;en-us;899149

However, it's about writing to a file on a network drive,
not reading from it.  It says that opening the file in 'w+b'
mode, instead of 'wb' mode, is a workaround.

I couldn't find anything documenting the same kind of
problem for reading.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2006-04-30 06:10

Message:
Logged In: YES 
user_id=21627

What version of Windows are you using? Do you know of any
documentation of this limit? (without actually testing, I
find it hard to believe that this limit exists in Windows)

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-04-29 09:23

Message:
Logged In: YES 
user_id=849994

How can it be determined whether exactly this restriction
caused the "invalid argument" error? If it can't, there's
nothing that can be done -- restricting all reads just
because of a Windows limitation doesn't seem right.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1478529&group_id=5470