[Box Backup] RAID - software instead of hardware?

[Ben Summers]

On 7 Jul 2004, at 10:18, richard_eigenmann wrote:

> My thoughts:
>
> I'm not running Software or Hardware RAID. BoxBackup is my backup 
> tool. If it
> breaks I will simply back up again. Of course it would be highly 
> inconvenient
> if the backup breaks at the same time as the source also runs into 
> trouble.
> In my case I have my data replicated exactly between the main 
> workstation and
> my Laptop (using unison) giving me a first level of protection.

It's all a matter of choosing the right tools for the job. With Box 
Backup you have three choices: none, software (application level) and 
hardware (or OS level software). What you use depends on your 
situation.

For my own uses, application level software RAID makes the most sense.

>
> If we are thinking of the 5 year timeframe then the key element is 
> storage
> explosion. In 1999 I bought 10 and 25 GB disks and they were pretty 
> big then.
> Now you pick up 200GB for half the price.
>
> If / when your BoxBackup disk breaks then you will likely pick up a 
> new one
> which is magnitudes larger than what you had. This will not fit well 
> into the
> two other software or hardware raid partitions / devices. Chances are 
> you
> will then want to rebalance the size allocations and replace the other
> devices too.
>
> If run without software RAID BoxBackup simply uses a directory 
> structure.
> This you can move easily to a larger device as your storage 
> requirements
> grow. You have no hassle with repartitioning 3 potentially different 
> capacity
> devices. With LVM it should even be possible to use several devices 
> together
> to create one large filesystem if you have some old disks lying 
> around. But
> then my experience is that old disks come in sizes of 800MB or 4GB 
> which
> makes that a bit pointless.

Your arguments could also be used as benefits of software RAID -- ease 
of building filesystems on which to store stuff.

>
> To safeguard against a disk failure on the BoxBackup server I would 
> suggest
> getting a second backup server at a separate offsite location. Any 
> clapped
> out PC (that doesn't suffer the IDE 34GB BIOS limit) should be able to 
> run a
> BoxBackup server on a 200GB disk. 200GB are going round for +/- 130 
> USD.

Future versions will support lazy mirroring, where changes to the 
primary server will be replicated to the secondary server.

>
> I'm wondering if we could even help similar minded BoxBackup users to 
> offer
> each other backup storage capacity. Such as Alice and Bob who live on
> different continents and have a permanent connection (even if 
> relatively
> slow) each set up a clapped out PC with 200GB storage in the basement 
> and
> then back up to their local server and to the other's machine. This 
> implies
> network connectivity and the presence of a basement. It also implies a
> certain degree of trust between the parties as you are letting a 
> stranger
> onto your network. But then people are dating each other via the 
> Internet
> which seems to me to involve a higher degree of risk.

You can do a lot to minimize the risk. Since bbstored doesn't depend on 
anything else, you can very easily isolate it from the rest of your 
network by filtering packets, applying appropriate user privileges, and 
even chroot.

(If it used something else as the transport, say ssh, then this would 
be much more difficult, and lots more potential holes. But as it is, 
the independence and "no-dependencies" design gives big wins here.)

Maybe I should have a "disc space swap" thing on the web site. So 
people can swap space on their machines.

>
> Of course if you are running BoxBackup in a large corporate 
> infrastructure
> then you probably have Hardware Raid anyway. I have my doubts about how
> suitable BoxBackup is in that sort of environment though. How do you 
> comply
> with data retention requirements? How do you retrieve that fraudster's 
> .pst
> file from 3 years ago to get at that incriminating email? With tape 
> that is
> possible by keeping the yearly, monthly and possibly weekly tape sets 
> locked
> away in a vault for the retention period. BoxBackup would have to have 
> some
> sort of configurable minimum retention period definition for the 
> housekeeping
> process. Since this is likely to be 5 or 10 years it implies that we 
> need
> incremental updates in the backup vault if we want to have any chance 
> of
> surviving such a volume of data. Also we might have different retention
> requirements for different classes of data.

These things are all part of my future plan. Check the list archives. 
There's not all that much which needs to be done to implement them.

Ben