writelines puzzle

[Peter Otten]

William R. Wing (Bill Wing) wrote:

> In the middle of a longer program that reads and plots data from a log
> file, I have added the following five lines (rtt_data is fully qualified
> file name):
> 
> wd = open(rtt_data, 'w')
> stat = wd.write(str(i))
> stat = wd.writelines(str(x_dates[:i]))
> stat = wd.writelines(str(y_rtt[:i]))
> wd.close()
> 
> The value of i is unknown before I have read through the input log file,
> but is typically in the neighborhood of 2500.  x_dates is a list of time
> stamps from the date2num method, that is values of the form
> 734716.72445602, day number plus decimal fraction of a day.  y_rtt is a
> list of three- or four-digit floating point numbers.  The x_dates and
> y_rtt lists are complete and plot correctly using matplotlib.  Reading and
> parsing the input log file and extracting the data I need is time
> consuming, so I decided to save the data for further analysis without the
> overhead of reading and parsing it every time.
> 
> Much to my surprise, when I looked at the output file, it only contained
> 160 characters.  Catting produces:
> 
> StraylightPro:Logs wrw$ cat RTT_monitor.dat
> 2354[ 734716.72185185  734716.72233796  734716.72445602 ..., 
> 734737.4440162
>   734737.45097222  734737.45766204][ 240.    28.5   73.3 ...,   28.4  
>   27.4   26.4]
> 
> Clearly I'm missing something fundamental about using the writelines
> method, and I'm sure it will be a DUH moment for me, but I'd sure
> appreciate someone telling me how to get that data all written out.  I
> certainly don't insist on writelines, but I would like the file to be
> human-readable.

When you apply str() to a numpy array big arrays are helpfully truncated, 
probably because the functionality is meant to be used in the interactive 
interpreter rather than to write to a file.
The default value is 1000 entries. One way to get the desired output is to 
increase the threshold:

>>> numpy.set_printoptions(threshold=4)
>>> print numpy.arange(10)
[0 1 2 ..., 7 8 9]
>>> numpy.set_printoptions(threshold=10)
>>> print numpy.arange(10)
[0 1 2 3 4 5 6 7 8 9]

Also, in

file.writelines(some_str)

writelines iterates over the characters of the some_string, so you should 
instead write the above as

file.write(some_str)

Your code will become 

assert numpy.get_printoptions()["threshold"] >= i
wd.write(str(x_dates[:i]))

If you intended to write one array entry at a time with writelines() here's 
how to do that:

wd.write("[")
wd.writelines("%s " % x for x in x_dates[:i])
wd.write("]\n")

numpy.savetxt() may also suit your needs.