[ python-Bugs-1478529 ] size limit exceeded for read() from network drive

[SourceForge.net]

Bugs item #1478529, was opened at 2006-04-28 17:46
Message generated for change (Comment added) made by markshep
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1478529&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Windows
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Mark Sheppard (markshep)
Assigned to: Nobody/Anonymous (nobody)
Summary: size limit exceeded for read() from network drive

Initial Comment:
If you've got a network share mounted as a local drive
then Windows has a limit of 67,076,095 (0x03ff7fff)
bytes for a read from an open file on that drive.

Running the python read() method on an open file larger
than this size throws an "IOError: [Errno 22] Invalid
argument" exception.

A fix would be for python to internally use multiple
reads so as to not exceed this limit.


----------------------------------------------------------------------

>Comment By: Mark Sheppard (markshep)
Date: 2006-05-02 11:48

Message:
Logged In: YES 
user_id=1512331

I'm running Windows XP.  I've been unable to find any
documentation about this exact problem - only that fwrite
thing.  But my testing shows that it works if I do
file.read(67076095), but throws an exception with
file.read(67076096).

I'm not suggesting limiting all reads from Python.  All I'm
suggesting is that under the hood the Windows implementation
of Python's read() call actually uses multiple fread() (or
whatever) calls if more than 67076095 bytes need to be read.
 That's all.  No interface changes.


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-04-30 17:23

Message:
Logged In: YES 
user_id=31435

Martin, here's an MS article seemingly related to this:

http://support.microsoft.com/default.aspx?scid=kb;en-us;899149

However, it's about writing to a file on a network drive,
not reading from it.  It says that opening the file in 'w+b'
mode, instead of 'wb' mode, is a workaround.

I couldn't find anything documenting the same kind of
problem for reading.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2006-04-30 11:10

Message:
Logged In: YES 
user_id=21627

What version of Windows are you using? Do you know of any
documentation of this limit? (without actually testing, I
find it hard to believe that this limit exists in Windows)

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-04-29 14:23

Message:
Logged In: YES 
user_id=849994

How can it be determined whether exactly this restriction
caused the "invalid argument" error? If it can't, there's
nothing that can be done -- restricting all reads just
because of a Windows limitation doesn't seem right.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1478529&group_id=5470