[CentralOH] Incrementally Downloading ISH Data

[Joe Shaw]

Hi,

On Thu, Mar 12, 2015 at 6:06 PM, <jep200404 at columbus.rr.com> wrote:

> FTP probably can not send a file beginning from an arbitrary point.
> wget can.
>

FTP can resume a download from an arbitrary point, and most servers
probably implement it by now.  It's not quite as flexible as an HTTP range
request, though.  (And FTP is a horrible protocol by modern standards.)
 See the REST command in http://www.faqs.org/rfcs/rfc959.html.


> I think rsync tends to resend whole file if any part of
>
it changes. I'm not sure about rsync.
>

No, rsync uses a block checksum algorithm and only transfers blocks that
are different. See https://rsync.samba.org/tech_report/

They mention that it's primarily optimized for smaller files, so I'm not
sure how efficient it is with large chunks of data.

Bittorrent protocol can pick up where it left off,
> and could retransmit only the 64 KByte chunks that
> have changed. So everyday we could easliy download
> just the changes. That would require a change on NWS server.
> The hard part would probably be getting them to change their
> server. Sales pitch could be that it would reduce the load
> on their servers, both my allowing incremental downloads,
> and by other peers sharing the load. We would need to test
> the ability for bittorrent to only download changes before
> pitching it.
>

If bittorrent is too much to swallow, here are other binary delta/diff
options available that don't require a specific transfer protocol:

   - xdelta - https://code.google.com/p/xdelta/
   - libxdiff (used by git) - http://www.xmailserver.org/xdiff-lib.html
   - bsdiff - http://www.daemonology.net/bsdiff/
   - courgette (developed by Google to push Chrome updates) -
   http://dev.chromium.org/developers/design-documents/software-updates-courgette

Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20150313/eb906a34/attachment.html>