savetxt() has fmt='%.18e' as default, but fmt='%.16e' is always sufficient
Double-precision numbers need at most 17 significant decimal digits to be serialised losslessly. Yet, savetxt() uses 19 by default, meaning that most files produced with savetxt() takes up about 9% more disk space than they need to, without any benefit. I have described the problem more detailed on Stackoverflow: https://stackoverflow.com/questions/77535380/minimum-number-of-digits-for-ex... Is there any reason behind the default choice of savetxt(..., fmt='%.18e')? If not, why not reduce it to savetxt(..., fmt='%.16e')?
On Sat, Nov 25, 2023 at 11:18 AM Jeppe Dakin <jeppe_dakin@hotmail.com> wrote:
Double-precision numbers need at most 17 significant decimal digits to be serialised losslessly. Yet, savetxt() uses 19 by default, meaning that most files produced with savetxt() takes up about 9% more disk space than they need to, without any benefit. I have described the problem more detailed on Stackoverflow:
https://stackoverflow.com/questions/77535380/minimum-number-of-digits-for-ex...
Is there any reason behind the default choice of savetxt(..., fmt='%.18e')? If not, why not reduce it to savetxt(..., fmt='%.16e')?
A long time ago when `savetxt()` was written, we did not use the reliable Dragon4 string representation algorithm that guarantees that floating point numbers are written out with the minimum number of decimal digits needed to reproduce the number. We may even have relied on the platform's floating point to string conversion routines, which were of variable quality. The extra digits accounted for that unreliability. It probably could be changed now, but I'd want more aggressive testing of the assertion of correctness (`random()`, as used in that StackOverflow demonstration, does *not* exercise a lot of the important edge cases in the floating point format). But if your true concern is that 9% of disk space, you probably don't want to be using `savetxt()` in any case. -- Robert Kern
Thanks for the explanation, Robert Kern. It seems then that indeed, these days the optimal default would be `fmt='%.16e'`. The test on StackOverflow was just a quick demonstration. See also the description for `DBL_DECIMAL_DIG` here: https://en.cppreference.com/w/cpp/header/cfloat
But if your true concern is that 9% of disk space, you probably don't want to be using `savetxt()` in any case. Well, I do in fact want to save as text, but I would rather not do so in an unnecessarily wasteful fashion. So i just set `fmt='%.16e'` myself, no biggie. The point, however, is that most users will not set this, and so `savetxt()` is a cause of wasted disk space generally, which I think should be fixed (if indeed we can be 100% certain that `fmt='%.16e'` is in fact always enough).
On Mon, 27 Nov 2023 at 17:19, Jeppe Dakin <jeppe_dakin@hotmail.com> wrote:
But if your true concern is that 9% of disk space, you probably don't want to be using `savetxt()` in any case. Well, I do in fact want to save as text, but I would rather not do so in an unnecessarily wasteful fashion.
`fmt='%s'` might be a good option if you want to save space: 1. it will output the nearest decimal value that round-trips 2. it handles the case of 32bit floats needing half as many digits 3. it does the right thing for integer values Sam
participants (3)
-
Jeppe Dakin -
Robert Kern -
Sam Mason