[Box Backup] Bundled TDB fails to build

[Charles Lecklider]

On 20/10/2010 19:59, Chris Wilson wrote:
> Thanks for the great work you've been doing on this! It's really nice to
> have someone else working on Box Backup with me :) It's been a while
> since I had a co-developer :)

You might not say that once you see the next round of Windows-specific
changes ;-)

>> That branch is from 0.11rc8 so there's no QDBM vs. TDB issue.
> 
> I've reverted the switch to TDB, so the current trunk should use QDBM as
> your branch does. I hope that will make it easier to merge your
> improvements back into the trunk. Please let me know if you foresee any
> problems.

That's good news. With a couple of extremely minor changes (e.g. MSVC is
fussy about #include <> vs. #include "") it compiles and passes all
tests under both 32 and 64 bit.

> I'd like to know a bit more about the new directory format. I've been
> trying to parse the commits you've made, but so far I haven't found the
> difference between the old and new formats, e.g. a new type ID in the
> header. Please could you enlighten me?

The old format looked like this:

header
file
file
file
....
depinfo
depinfo
depinfo
...

Clearly you can't just append to that, so I changed it to look like this:

header
file
depinfo
file
depinfo
file
depinfo
....

By adding a new flag it was trivial to be able to read the old format as
well as the new, thus avoiding a messy conversion process.

The aim was to make 0.11 run fast enough to be usable with things like a
Maildir; it's been running here for a while now without problems, so
subject to review and other people testing I think it's ready for use
(but see below).

However, I only fixed half the problem - writing the directory back to
disc. The entire directory still gets read into memory, even when adding
a completely new file. That's daft.

I'm thinking about the implications of ditching the current format
completely and switching to a database of some sort, perhaps a QDBM
Villa, perhaps Kyoto Cabinet (http://fallabs.com/kyotocabinet/), perhaps
something else; my main concern is that stores can be moved between 32
and 64bit machines, which may be an issue with QDBM.
The only way to avoid pulling the entire directory into memory is to be
able to access each entry by ID directly, and without some pretty
significant changes that simply isn't possible.

As for the current set of changes working with RaidFile: I suspect they
don't. I'm with Ben on this - I don't see what RaidFile achieves now,
and it adds a large amount of complexity that makes doing significant
work prohibitively difficult.

Having said all that, my current focus is on the Windows client....

-C