[Numpy-discussion] savetxt -> gzip: nondeterministic because of time stamp

Derek Homeier derek at astro.physik.uni-goettingen.de
Wed Apr 14 18:52:35 EDT 2021


On 15 Apr 2021, at 12:39 am, Robert Kern <robert.kern at gmail.com> wrote:
> 
> On Wed, Apr 14, 2021 at 6:16 PM Andrew Nelson <andyfaff at gmail.com> wrote:
> On Thu, 15 Apr 2021 at 07:15, Robert Kern <robert.kern at gmail.com> wrote:
> On Wed, Apr 14, 2021 at 4:37 PM Joachim Wuttke <j.wuttke at fz-juelich.de> wrote:
> Regarding numpy, I'd propose a bolder measure:
> To let savetxt(fname, X, ...) store exactly the same information in
> compressed and uncompressed files, always invoke gzip with mtime = 0.
> 
> I agree.
> 
> I might look into making a PR for this. To be clear what would the desired functionality be:
> 
> 1. Mandatory to have mtime = 0?
> 
> 2. Default mtime = 0, but `np.save*` has an extra `mtime` kwd that allows to set the mtime?
> 
> 3. Default mtime = time.time(), but `np.save*` has an extra `mtime` kwd that allows to set the mtime = 0?
> 
> 
> As Joachim says for testing/git-related purposes it is nice to have bit-wise unchanged files produced (such that the file-hash is unchanged), but I can also see that it might be nice to have a modification time when files contained in a zip file were last changed (e.g. write with numpy, check/inspect with a different module). Of course with the latter you could just look at the overall file-write date, they should be the same.
> 
> I suspect no one's actually looking at the timestamp inside the file (relevant XKCD comic[1] notwithstanding). I'd lean towards boldness here and just set mtime=0. If someone screams, then we can add in the option for them to get the old functionality. But moving closer to reliably reproducible file formats is a goal worth changing the default behavior.
> 
I was also a bit worried about that – a) people actively using the timestamp for information and
b) files with mtime=0 decompressing to files showing a creation date in 1970 or something.
But all the Unix timestamps at least seem to be determined differently, so maybe the benefits
do outweigh the risks.

> [1] https://xkcd.com/1172/  :D

Derek



More information about the NumPy-Discussion mailing list