[borgbackup] Borg speed tuning on large files

[Alex Gorbachev]

Here is the latest round of benchmarks - lz4 is definitely a lot faster

Borg First Runlz4,19,23,20,409522569107527829648648/31/2015 13:51:009/1/2015
7:12:0017.435%2.8835Borg Next Runlz4,19,23,20,409522387572964369795849/1/2015
7:20:009/1/2015 21:58:0014.620%5.1242Borg First Runlz4,17,22,19,4095
22656182407912710409/2/2015 7:15:009/2/2015 23:37:0016.435%2.8638Borg Next
Runlz4,17,22,19,409522844942163960725769/3/2015 15:54:009/4/2015 6:58:0015.1
17%5.7741Borg Next Runlz4,17,22,19,409523200848803747252489/4/2015
13:20:009/5/2015
4:08:0014.816%6.1943Borg First Runlz4,19,23,21,409523581555688117031689/5/2015
10:21:009/6/2015 3:13:0016.934%2.9138Borg Next Runlz4,19,23,21,4095
23456232883870658569/6/2015 20:20:009/7/2015 11:52:0015.517%6.0641

On Tue, Sep 1, 2015 at 8:24 AM, Thomas Waldmann <tw at waldmann-edv.de> wrote:

> > Indeed, with lz4 the compression speed is the fastest it has been.  On
> > the first run, data is 35% of original size (same 2TB volume) and it
> > took 17 hours to compress vs. the previous 33 with LZMA.
>
> Ah, that's impressive.
>
> lz4 seems to like your data (usually it doesn't compress to 35%). :D
>
> A zlib,1 comparison value would have been nice here (as that is the
> fastest compression zlib can do [but usually slower than lz4]).
> lzma is known to be rather slow (but high compression).
>
> > Computed speed is 12 MB/s vs. the previous 5 MB/s, and we are not at
> > all disk bound (we do 100+ MB/s network transfers from it).
>
> If I divide 2TB original data by 17h backup time, I get 32MB/s.
> Your data rate is based on the compressed data.
>
>
> --
>
> GPG ID: FAF7B393
> GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20150911/64fdd661/attachment.html>