[Boxbackup-dev] Feature Concept: Use local blocks to reduce restore bandwidth

Pete Jalajas pjalajas at tebuco.com
Fri Nov 8 05:10:49 GMT 2013


Hi again,

I've always had in the back of mind the concept that BoxBackup (and
others) can simplify the handling of very large files by breaking them
up into more manageable chunks, and just keep track of those chunks
needed to restore a file (an easy way to get around any 2GB limit or
whatever).  Well, these guys implemented that idea, but also put it on
steroids (as we say here in America):
  https://duplicati.googlecode.com/files/Block-basedstorageformat.pdf
Under "Reconstruction" on page 5, it says in effect I think:
---
When reconstructing files (restoring data), the [client-side] database
is queried to obtain [a listing of] all required blocks. ... Then the
database is [again] queried to find potential local files that may
hold the desired blocks. Should such [local!] blocks be found, they
are used to patch the target files [instead of retrieving those blocks
from the remote store].
---

I think lay folks generally tend to concentrate on how to improve
"backup" performance, but in reality, backup can generally be a slow
and leisurely, albeit careful, thing.  It's the "restore" that is
generally an emergency, urgent and stressful, and so anything to
improve that restore speed is important to consider.  Just guessing,
but it seems like their use-local-blocks concept could sometimes save
a few percent of the restore bandwidth.  A few percent here, a few
percent there, and pretty soon...

Anyway, FYI,
Pete



More information about the Boxbackup-dev mailing list