[Box Backup-dev] Re: Learning from ZFS (fwd)

Chris Wilson boxbackup-dev at fluffy.co.uk
Mon May 7 13:05:11 BST 2007


Hi all,

Ben and I have been discussing how we could possibly use Sun's Solaris ZFS 
to provide most of the functionality of Box Backup. We could get the rest 
with a couple of layers above and below ZFS, which would make Box much 
simpler and give us cool features like versioning and snapshots.

We think we should move this to the -dev list so that we can all discuss 
it.

We're discussing two possible approaches, not necessarily incompatible or 
either-or, just talking points:

Option 1. Any filesystem on client, sync to encrypted filesystem (like 
encfs) on top of zfs on top of iscsi block device (which is mounted from 
the server, over the net).

Advantages: supports any filesystem on client. Very simple server 
(effectively a network block device) which holds ZFS images which are 
managed entirely by the client. The server could in principle mount these 
images locally, but would see only encrypted filenames and data.

Disadvantages: more complex and less efficient than 2.

Option 2. Client runs encfs on top of local zfs, synchronises this to 
remote ZFS periodically, as Ben describes below.

Advantages: more efficient (we think), simpler Box code (encrypted 
filesystem only)

Disadvantages: requires kernel support, especially to run ZFS in the 
kernel in most cases. requires client to run zfs filesystems.

The message below is part of our discussion. If anyone is lost or needs 
more introduction to the issue, please ask. I'll reply to Ben's email 
below, shortly.

Cheers, Chris.
-- 
_ ___ __     _
  / __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |

---------- Forwarded message ----------
Date: Mon, 7 May 2007 12:29:46 +0100
From: Ben Summers <ben at fluffy.co.uk>
To: Chris Wilson <chris at qwirx.com>
Subject: Re: Learning from ZFS


On 7 May 2007, at 12:01, Chris Wilson wrote:

> Hi Ben,
> 
> On Mon, 7 May 2007, Ben Summers wrote:
> 
>> Well amusingly, I can just realised that you can almost implement a 
>> wonderful Box Backup system without any of the annoying bits using Solaris: 
>> a combination of zfs and iSCSI can do it quite nicely... but there are two 
>> missing bits, the encryption and the compression. The encryption should be 
>> easily solvable with a small bit of kernel code, but the compression is more 
>> tricky.
> 
> Let me see if I understand this: instead of a bbstored server, you run an 
> iscsi server? (which i guess you could do as a bit of software running on a 
> hosted *nix box, although I don't know whether it exists, whether it's secure 
> or deployable over the Internet).

You can always use a tunnel.

> 
> Then, as a client you'd use a mounted zfs filesystem?

Yes.

> 
> On Linux you could mount this on a cryptoloop filesystem, or alternatively 
> run an encrypted filesystem like encfs over the top of zfs. I'm not sure how 
> possible this is on BSD.

I'm sure this is a very solvable problem.


> 
>> So there could be a good model there, or perhaps just make a userland port 
>> of zfs.
> 
> A userland port of zfs sounds like a great idea. zfs is CDDL and apparently 
> mixes with BSD licenses without problems, it's just been ported to a BSD 
> kernel.
> 
> This could simplify the server down to block read/write operations,

I think an ideal server should just read/write blocks, but with snapshots so it 
can remove old stuff.


> or enable us to replace it with a commodity or custom iscsi implementation 
> (or Linux NBD).
> 
> Would it make sense to just encrypt filenames and contents, store them in ZFS 
> and use its API (including snapshots, versioning, deltas), building all of 
> that into the bbackupd client?
> 
> Does this layering make sense?
> 
> BackupDaemon
> ------------
> BackupClientContext
> ------------
> BackupClientDirectoryRecord
> ------------
> File name and data encryption
> ------------
> ZFS
> ------------
> iScsi client
> ------------
> Network
> ------------
> iScsi server

That's certainly one way of doing it, and one which is nice and cross platform 
with everything in userland. But if we're modifiying ZFS we might as well get 
it to use our block driver to avoid the dependency on iSCSI and all that 
entails.

The biggest costs of bbackupd are:

* scanning directories (various APIs can make this easier, but you'll still 
need to scan if the daemon is ever not running or misses anything)

* rsync

* tracking of objects for efficient renames, and not tracking everything to 
avoid huge costs of doing so.

So using ZFS on the client for it's backed up FS avoids the need for any of 
that, because you do a snapshot and ask for the differences between the 
snapshots. Here's how it would work:

Server exposes a zvol within a ZFS pool as an iSCSI target.

Client has all the data on ZFS filesystems. To do a backup, (everything runs on 
the client):

* Snapshot FS (atomic, quick and cheap)

* Mount iSCSI target

* zfs send | zfs receive to transfer the incremental changes to the server

* Dismount the iSCSI target

To protect against bugs on the client wiping data from the server, do snapshots 
of the zvol on the server to an appropriate schedule. Housekeeping on both ends 
is done by deleting redundant snapshots.

Now this doesn't give encryption, so the iSCSI target needs to have some crypto 
on top. Easy one to solve if your kernel can't do it anyway.

Then you have Box Backup! The only thing you don't get is the compression, 
which is more difficult to just patch on top. ZFS does do compression, but to 
take advantage of it you'd have to use it on the source FS, which I guess many 
won't want to do.

Since I'm probably going to move to Solaris properly for servers and things, 
I'm going to try this out. Then see what can be learned and how we can get this 
into Box Backup for systems which don't use ZFS as their filesystem.


We should move this to -dev, I think. You have my permission to post anything 
in this email. (how old-school netiquette I can be sometimes)

Ben








!DSPAM:463f0dae95882732111066!



More information about the Boxbackup-dev mailing list