struct.pack('d', ...) -> "frexp() result out of range" (was Re: PyExcelerator)

[John Machin]

On 3/06/2006 3:39 AM, tkpmep at hotmail.com wrote:
> I write data to Excel files using PyExcelerator 0.6.3.a and have done
> so successfully for small files (10-15 cells). I'm experiencing an
> error when writing a big chunk of data (10,000 cells) to Excel. By way
> of comparison, the same data writes perfectly well to a csv file using
> Python's built in csv module. I run the program in PyScripter, and the
> traceback shows the following sequence of calls:
> 
> main                          (my routine)
> writeData                    (my routine)
> save                           Line 563
> get_biff_data               Line 548
> get_biff_data               Line 1357
> __row_blocks_rec       Line 1276
> get_cells_biff_data      Line 200
> get_biff_data               Line 106

Grumble: When you need to post a traceback, please supply one that shows 
on each line which *file* the source line is in. In this case there are 
multiple files, and 9 classes spread over 4 files have a method called 
get_biff_data.

> 
> SystemError: frexp() result out of range
> 
> The line it stops at in get_biff_data is the line that starts with
> packed =
> 
> def get_biff_data(self):
>         rk_encoded = 0
> 
>         packed = struct.pack('<d', self.__number)
> 
> Any thoughts on what the problem could be?
> 

It is calling struct.pack (see structmodule.c in the Python source) to 
encode one of your cell values as an 8-byte little-endian IEEE 754 
floating-point number. This task is handed off to _PyFloat_Pack8() (see 
floatobject.c). Here's the relevant bit:

"""
	if (x < 0) {
		sign = 1;
		x = -x;
	}
	else
		sign = 0;

	f = frexp(x, &e);

	/* Normalize f to be in the range [1.0, 2.0) */
	if (0.5 <= f && f < 1.0) {
		f *= 2.0;
		e--;
	}
	else if (f == 0.0)
		e = 0;
	else {
		PyErr_SetString(PyExc_SystemError,
				"frexp() result out of range");
		return -1;
	}
"""
where x is your cell value as a C double.

frexp() is a standard C library routine. The above code is just checking 
that it has done its job correctly. AFAICT from reading K&R v2 p251, the 
code is correct.

Aside: isn't open source wonderful? Well it is for me. It could be 
wonderful for you too :-)

I'm not an IEEE 754 guru by any means, so I'm guessing your value could 
be an Inf (infinity) or a NaN (not a number) -- there's no explicit test 
for those in the code -- or you have stumbled on a value that's a corner 
case on your platform. Maybe it's -0.0 (zero with the sign bit set).

Grumble: When you have a problem like this, you should state what 
version of Python you are using, on what platform. The line you get when 
you run Python at the command line is a start e.g.
"""Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] 
on win32"""

Please tell us this, plus explicitly what OS and what hardware you are 
running on.

OK, now what's this dodgy number?? Unfortunately, the error is happening 
  right at the end of your process, in the workbook save method. The 
only info about which row/column is buried deep in pyExcelerator. We 
don't (yet) want to start poking about there ...

Try these steps:
1. Search your alternative CSV output file for the text "Inf" or "NaN" 
(case insensitive).
2. Put some code in your cell write routine to filter the values you are 
writing:

absval = abs(value)
SMALL = 0.000001 # depends on what is reasonable for your app
LARGE = 10.0 ** 10 # depends ...
if not (value == 0.0 or (SMALL <= absval <= LARGE)):
     print "strange value %r in row=%d col=%d" % (value, rownum, colnum)

Then come back and tell us what if anything you've found.

HTH,
John