On Thursday, April 28, 2011 5:22pm, "Dan Halbert" <halbert@halwitz.org> said:
I'm having trouble loading a large remote .npy file on Windows XP. This is on numpy-1.3.0 on Windows XP SP3:
numpy.load(r'\\myserver\mydir\big.npy')
will fail with this sort of error being printed: "14328000 items requested but only 54 read"
I have tracked this down: it's not specific to numpy, or even Python, though there's a Python bug report about it: "size limit exceeded for read() from network drive" http://bugs.python.org/issue1478529 The problem stems from a buffer size limit of not quite 64MB for the Windows API ReadFile()function. ReadFile() is used (maybe indirectly) by the Microsoft fread() implementation, which is in turn used by Python file.read(). Apparently most of the time the underlying Windows code is cognizant of this limit, but under certain circumstances, a read from a large networked file is not handled correctly (I think when there is also an entry in "My Network Places" for the path). It's quite obscure, and there's nothing that can be fixed at the numpy level. Dan