[Numpy-discussion] savetxt -> gzip: nondeterministic because of time stamp

Robert Kern robert.kern at gmail.com
Wed Apr 14 18:29:46 EDT 2021

On Wed, Apr 14, 2021 at 6:20 PM Derek Homeier <
derek at astro.physik.uni-goettingen.de> wrote:

> On 14 Apr 2021, at 11:15 pm, Robert Kern <robert.kern at gmail.com> wrote:
> >
> > On Wed, Apr 14, 2021 at 4:37 PM Joachim Wuttke <j.wuttke at fz-juelich.de>
> wrote:
> > Regarding numpy, I'd propose a bolder measure:
> > To let savetxt(fname, X, ...) store exactly the same information in
> > compressed and uncompressed files, always invoke gzip with mtime = 0.
> >
> > I agree.
> >
> I would caution though that relying on the checksum or similar of the
> compressed data still
> does not seem a very robust check of the data itself - the compressed file
> would still definitely
> change with any change in compression level, and quite possibly with
> changes in the
> linked compression library (perhaps even a simple version update).

Sure, but in practice, that variability happens more rarely. In particular,
I'm not going to see that variability when I'm re-running the same workflow
with the same software versions inside of a system like DVC[1] that uses
file hashes to determine which parts of the workflow need to get rerun.
Whereas, I will see timestamp changes every time and thus invalidating the
cache when it's irrelevant.

[1] https://dvc.org/

> Shouldn’t you better verify the data buffers themselves? Outside of
> Python, the lzma utility
> xzcmp for example allows binary comparison of the content of compressed
> files independently
> from the timestamp or even compression method, and it handles gzip and
> bzip2 files as well.

We're often integrating with systems that only know how to compute hashes
of files and doesn't try to dig into the semantics of the files any deeper.
We're not looking to construct the most provably-correct cache, just make
some pragmatic choices that will reduce false cache misses with systems
that currently exist and enable *better*, if still imperfect, uses of those

> I would like to follow up with a pull request, but I am unable to
> > find out how numpy.savetxt is invoking gzip.
> >
> > `savetxt` uses the abstractions in this module to invoke gzip when the
> filename calls for it:
> >
> > https://github.com/numpy/numpy/blob/main/numpy/lib/_datasource.py#L115
> >
> One obstacle would be that this is setting gzip.open as _file_opener,
> which does not have the
> mtime option; getting this replaced with gzip.GzipFile to work in text
> mode would require to
> somehow replicate gzip.open’s io.TextIOWrapper wrapping of GzipFile.

That should be easily doable.

Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/numpy-discussion/attachments/20210414/50af7c12/attachment.html>

More information about the NumPy-Discussion mailing list