[Box Backup-dev] Soft-RAID support

[David Sommerseth]

Chris Wilson wrote:
> Hi David,
> 
> On Sat, 6 Jun 2009, David Sommerseth wrote:
> 
>> Chris Wilson wrote:
>>> Support for it was never finished (no recovery procedure), it is pretty
>>> limited (only supports RAID 5 and three devices) and it was written at a
>>> time when OS/software and hardware RAID were not as ubiquitous or well
>>> supported as they are now.
>>
>> I would be willing, with some guidance to look into such a tool, if that
>> is the main criteria for dropping this support.
> 
> That would definitely be very helpful, thanks in advance. You can read 
> the encrypted objects (which are reconstructed successfully) and then 
> rewrite them, which will reestablish the redundant copies.

I'll grab the code soon after the holiday season is over, and poke into 
this.  I'd consider the program as a stand alone program somehow, which 
will do the recreation in a way which you suggest.  As a brief quick idea 
of how I could imagine it:

     root at host # bbackrecover --source-dir1 /path/to/origdata_part1 \
			     --source-dir2 /path/to/origdata_part2 \
  	  		     --recover-dir /path/to/recovered_part3

Only the missing part would then be recovered to the given directory in 
--recover-dir.  Not sure though, if it would be needed to write data to 
part1 and part2 directories in addition to the already mentioned part3. 
Does this approach seem sensible?

Any special parts in the code you'd recommend me to dig into before I begin 
to ask more questions?  And any of the Box Backup developers available on 
IRC channels?  Is the source code available via a public SCM URL?  (git, 
svn, cvs)

>> The soft-raid solution itself seems to work flawlessly and seems to 
>> only need this recovery tool.  Or are there any other issues which is 
>> not to well known with the soft-raid which should make me worried?  
>> Are there any critical bugs related to the current implementation?
> 
> No, I don't think so. All of our tests actually run in RAID mode, hence 
> the "more tested" aspect. However it does impose significant performance 
> limitations which may prevent me from making some optimisations to 
> reduce disk I/O in future, and the new refcount database will not be 
> mirrored, but it can be reconstructed by housekeeping in any case, so 
> it's more of a cache than a database.

That sounds good.  Of course I/O requests and performance are more complex 
when needing to keep control over three streams vs just one.  But this 
optimisation is also depending on how clever the OS is able to spread the 
tasks.  Of course, I do recognise that if all data is on the same device, 
an OS optimisation should probably be ignored.  It could also be that with 
some syscalls, it's possible to do, at least some of, this optimisation 
inside BoxBackup (usually done by sorting by inodes of the files being 
read/written to/from, afaik - considering the inodes for all of these three 
streams).  But in the case of using 3 different devices, the OS is the one 
which should do the optimisation.

>>> I can see your point about the usefulness of this for distributed
>>> encrypted backup. However I'm not convinced about the overall merits of
>>> storing the data in three separate locations.
>>
>> Regarding encryption, yes, that is one key element.  But if the
>> organisation looses one remote storage with the complete backup 
>> directory,
>> it got all the needed information needed to begin to crack the 
>> encryption.
>> If you need minimum 2 sets to be able to crack the encryption, you have
>> another layer of security.  And it was this combination which caught my
>> attention.  When you add locally encrypted disks, you have the third 
>> layer
>> of security.
> 
> That's a good point. I don't believe this idea has been proposed before, 
> and I guess Ben didn't have it in mind when he implemented BB RAID or 
> when he proposed to remove it.

And I who thought this feature was implemented for such a use-case ;-)

>>> However I think it may no longer be true. I suspect that few people 
>>> are using the userland RAID feature in production. If anyone except 
>>> David is, please speak up!
>>
>> I would also be interested in hearing others experiences as well!  If 
>> I'm the only one, I agree, it's not much point in continuing this 
>> support in BoxBackup.  Then I would need figure out another way how to 
>> solve this.  I will not continue on this path if soft-raid disappears 
>> for sure in BoxBackup.
> 
> As you have a good use case for it, I am not planning to remove it in 
> the near future. However I would be interested in thinking about better 
> ways to implement this, such as at the OS level. I do think it would be 
> more efficient, not less, to implement this at the block level in the OS 
> rather than in Box.

Thanks!  This sounds good.  Yeah, it would be possible to move it to 
kernel-space.  But I'm not sure this would gain too much interest, as you 
have dmraid and mdraid in kernel already (thinking Linux primarily).  To 
have a file-based soft-raid in addition, might be considered waste of time 
- and a more difficult case to optimise.  Anyway, I'll try to mention it 
for some kernel fs developers at work.

Another reason why not to depend on the OS here, is that this might not be 
possible or very difficult to implement such feature in all OS supported by 
Box Backup.

> I'm also planning to implement S3 client support in Box Backup fairly 
> soon, and I expect that most users will move to that as it frees them 
> from the need to ever buy more disks or take their systems offline for 
> disk upgrades. Unless we can find a good way to support userland RAID on 
> top of S3, I expect that these code paths will diverge significantly and 
> you may find that fewer users use libraidfile at all.

I haven't studied the S3 client in general much, and thus I have no idea 
how the Box Backup implementation for S3 will be.  But if the soft-raid 
code is kept inside Box Backup, it might be easier to setup three remote 
destinations as well, or one local and two remote.

Have you thought about supporting other remote protocols in addition?  Like 
ssh or webdav?  For me it sounds like you might plan such a remote layer is 
located in the bbstored, instead of assigning a local directory you assign 
a remote - or am I completely wrong?


Thanks anyway for a good response! :)


kind regards,

David Sommerseth