[Box Backup] Thoughts on reliability (Was: "Box Backup - corrupted store")
boxbackup at fluffy.co.uk
Mon Jun 13 10:50:56 BST 2005
On 11 Jun 2005, at 10:59, Gary wrote:
>> I suspect that in this case of corruption, a block got corrupted
>> somehow after it was committed.
>> Berkeley DB is only used on the client to track non-essential data.
> My bad, I have not looked at the server sources that deep, yet. At any
> rate, how does the server side guarantee consistency in case of an
> interrupted upload or interrupted server-side file write (especially
> for large files)?
I use three techniques, which I believe are all standard good
practice for software engineers.
1) Write to temp files, then move into place to commit. Everything
does this, as part of the lib/raidfile code. Since moving a file over
another should "always" work, this means that either all of a change
will happen, or nothing.
2) Careful ordering of operations, exception handlers and destructors
which clean up by default. If there's another failure while
processing a request, exception handlers and object destructors will
clear it up. By ordering the code carefully, everything can be undone
until the last minute. In addition, if my assumption that the "moves
into place" for turning the temp files into live files turns out to
be incorrect, the exception handlers will recover.
3) Fault tolerance -- expect things to go wrong. If something could
have gone wrong in the past, the code will be tolerant of the mess it
leaves behind and correct it as it goes along. The best example is
the "store info file", which contains details of the space used and
last ID allocated. This should be written to disc after every
request, but it would be vastly inefficient so it's written lazily
every few requests. If the server child process terminates
unexpectedly, it will be out of date. So the code which allocates a
new ID assumes that it may be out of date, and the housekeeping
routine will correct out of date space information.
>> It compares MD5 checksums of each block with the index. Using a hash
>> is good enough for digital signatures, so I hope it's good enough for
>> I suppose a MD5 of the encrypted data could be kept as well, and the
>> server could verify that the file is still OK?
> There seem to be two separate issues here: comparing server index
> content with client content (-q), and verifying that server index
> content is actually what is on a server hard drive. The first issue
> already been solved by comparing current client content blocks with
> last known uploaded blocks on the server (index), right?
> The second
> issue is currently not implemented, but, if I understand correctly, if
> a function is added to make sure that the server side index is
> what's currently on the server hard drive, then a complete end-to-end
> compare has been achieved.
> I guess we would need some kind of "association" of reported client
> blocks (index content) with factual server content.
You would have to trust the server to report correct results. Either
a checksum of the encrypted data could be added, but perhaps more
space efficiently a single checksum for the entire store file could
be added. There may be issues with calculating this when patches are
in use, so it's not as simple as it might seem.
> Did I get this right?
The principle I think is right, although the implementation may be
tricky. I would prefer a challenge-response system, where the client
challenges a server with some data, and it replies with some more
which could only be calculated if the server held a proper copy of
the file. This may not be practical, of course.
More information about the Boxbackup