[Box Backup-dev] Re: [Box Backup] bbackupd - read errors on database files

[Martin Ebourne]

On Tue, 2007-04-10 at 20:29 +0100, Chris Wilson wrote:
> Ben, I'd be happy to implement this (especially if Gary can provide his 
> quick and dirty code). What do you say, should we have it for the next 
> version (post 0.11)? Maybe as an optional feature, off by default? We 
> could store the MD5 checksum as an optional extended attribute, so it 
> would be backwards compatible, or else have a special magic value of the 
> MD5 field (maybe all zeroes) to indicate that no checksum has been stored.

FWIW I'm undecided on this. I don't know what data-set Gary has run it
on, but 15% overhead doesn't sound very realistic to me. Bearing in mind
it will have to read the full contents of every file on every scan
(unless you got funky and just checksumed a portion of the filesystem on
each scan, which has its own risks), rather than just the directories, I
think the overhead will be a lot higher. On a fairly full 50GB disk that
would be 50GB to read on every scan, rather than probably 500MB.

Sure we can make it optional and all these other things, but every
option has its cost, and especially in backup software you want to be
really conservative, an optional code path by definition will get a lot
less testing than a mandatory one and hence more risky.

Clearly file notification is the way to go, and any platform of any
interest supports it these days. Having said that we'll still need a
full scan recovery mode for cases where the file notification was not
running and checksumming the data would give ultimate peace of mind.
Given that it would only be fallback the cost wouldn't be so important
in this case. (I am envisaging eg. if bbackupd is started as part of the
bootup sequence and shutdown similarly there'd be no need for a scan,
rather like a clean mount/unmount, but if the machine or bbackupd
crashed a scan would be necessary.)

Maybe have a normal full scan and a checksum full scan and an option
that says every N scans is a checksum scan. This should encourage
everyone to run the code so it gets lots of testing. Paranoid people and
file notification people could set N=1 and others could set it higher to
avoid the overhead. Don't have a way of switching it off, but people
could always set N very high. There's no overhead on the initial backup
because the data all has to read and encrypted, etc anyway.

> If necessary, we can run the entire bbackupd test suite twice, once with 
> MD5 checksums enabled and once with them disabled, to make sure that 
> nothing gets broken either way. Otherwise, I can just write a test that 
> uses utimes().

If we have checksums they should be mandatory as I mentioned, to help
improve reliability. I think a specific test of the checksum should be
enough to check it works.

Cheers,

Martin.