[Numpy-discussion] numpy.load truncates read from network file on XP

[Nadav Horesh]

Several time I encountered problems in transfering large files between XP stations on a wireless network. Could be a result of the unsafe UDP prtocol used by microsoft network protocol (do not have a vista/7 machines to test it).

  Nadav

________________________________________
From: numpy-discussion-bounces at scipy.org [numpy-discussion-bounces at scipy.org] On Behalf Of Bruce Southey [bsouthey at gmail.com]
Sent: 29 April 2011 04:56
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] numpy.load truncates read from network file     on XP

On Thu, Apr 28, 2011 at 4:22 PM, Dan Halbert <halbert at halwitz.org> wrote:
> I'm having trouble loading a large remote .npy file on Windows XP. This is on numpy-1.3.0 on Windows XP SP3:
>
>    numpy.load(r'\\myserver\mydir\big.npy')
>
> will fail with this sort of error being printed:
>    "14328000 items requested but only 54 read"
> and then I get this with a backtrace:
>    "ValueError: total size of new array must be unchanged" (due to the truncated array)
>
> The file big.npy is a big 2d array, about 112MB.
>
> The same file when stored locally gives no error when read. I can also read it into an editor, or copy it, and I get the whole thing.
>
> More strangely, the same file when read from the same UNC path on Windows 7 64-bit (with the same 32-bit versions of all Python-related software) does not give an error either.
>
> The fileserver in question is a NetApp box. Any clues? My websearching hasn't netted any leads.
>
> Thanks a lot,
> Dan
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
Don't know what a 'NetApp box' is, what OS it runs or what are the
filesystem types being used. Really we need more information
especially what the file is and how it was created etc., as well as
the different OSes involved with the exact Python and numpy versions
on each system - numpy 1.3 is getting a little old now with 1.6 due
soon.

A previous error has been the different line endings between OS's when
transferring between Linux and windows. That should show up with a
smaller version of the file. So at least try finding the smallest file
that you gives an error.

Other other issue may be connection related in that Python is not
getting the complete file so you might want to read the file directly
from Python first or change some of XP's virtual memory settings.
There have been significant changes between XP and Win 7 in that
regard:
http://en.wikipedia.org/wiki/Comparison_of_Windows_Vista_and_Windows_XP
"File copy operations proved to be one area where Vista performs
better than XP. A 1.25 GB file was copied from a network share to each
desktop. For XP, it took 2 minutes and 54 seconds, for Vista with SP1
it took 2 minutes and 29 seconds. This test was done by CRN Test
Center, but it omitted the fact that a machine running Vista takes
circa one extra minute to boot, if compared to a similar one operating
XP. However, the Vista implementation of the file copy is arguably
more complete and correct as the file does not register as being
transferred until it has completely transferred. In Windows XP, the
file completed dialogue box is displayed prior to the file actually
finishing its copy or transfer, with the file completing after the
dialogue is displayed. This can cause an issue if the storage device
is ejected prior to the file being successfully transferred or copied
in windows XP due to the dialogue box's premature prompt."


Bruce
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion at scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion