From boxbackup at fluffy.co.uk Wed Sep 1 20:58:31 2004 From: boxbackup at fluffy.co.uk (Andreas Schrafl) Date: Wed, 01 Sep 2004 21:58:31 +0200 Subject: [Box Backup] trigger housekeeping Message-ID: <413629E7.3020107@jetnet.ch> I reduced the amount of my soft and hardlimit and hoped to get the olf files deleted. the next backup run said i'd have so add more storage but didn't delet any old files. when does housekeeping run? how can I trigger it? thanks for a great backup solution (I'm trying to make a windows client that is usable for "normal" windows users) andy From boxbackup at fluffy.co.uk Wed Sep 1 21:14:02 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 1 Sep 2004 21:14:02 +0100 Subject: [Box Backup] trigger housekeeping In-Reply-To: <413629E7.3020107@jetnet.ch> References: <413629E7.3020107@jetnet.ch> Message-ID: <77792E8A-FC53-11D8-B2DB-000A95E8BABA@fluffy.co.uk> On 1 Sep 2004, at 20:58, Andreas Schrafl wrote: > I reduced the amount of my soft and hardlimit and hoped to get the olf > files deleted. the next backup run said i'd have so add more storage > but didn't delet any old files. > > when does housekeeping run? Every x seconds, where x is specified in bbstored.conf as TimeBetweenHousekeeping. > how can I trigger it? Send a -HUP signal to bbstored. What do you think should have happened when you changed the limits. What would you like to have done? How would you like to have controlled this? I have server support for adding a flag to certain files and objects which means "delete as soon as possible", that is, when they become an old version or are marked as deleted by the client. I am intending to expose this functionality to the user via bbackupquery. > > thanks for a great backup solution > (I'm trying to make a windows client that is usable for "normal" > windows users) I'm glad you're finding it useful! Ben From boxbackup at fluffy.co.uk Wed Sep 1 21:46:14 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Wed, 1 Sep 2004 22:46:14 +0200 Subject: [Box Backup] multiple Path and/or ExcludeDir In-Reply-To: References: <17657408.1093965339870.JavaMail.www-data@www> <98A23F8E-FB69-11D8-A954-000A95AFF7F8@fluffy.co.uk> <20040831181953.GA1335@sarsation.hixsplit.hu> Message-ID: <20040901204614.GA1443@sarsation.hixsplit.hu> On Tue, Aug 31, 2004 at 07:31:09PM +0100, Ben Summers wrote: > I recommend you create separate entries for each directory within the > root directory that you want to back up. You really do not want to have > mount points within these directories, which you do at the moment. Hello Ben, Finally I changed configuration to this: BackupLocations { bin { Path = /bin } boot { Path = /boot } dev { Path = /dev } etc { Path = /etc ExcludeFile = /etc/box/bbackupd/xxxxx-FileEncKeys.raw } home { Path = /home } root { Path = /root } sbin { Path = /sbin } users { Path = /other } usr { Path = /usr } var { Path = /var } } Working perfectly. Thanks your advice! Have a nice day! Amily From boxbackup at fluffy.co.uk Wed Sep 1 22:13:09 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 1 Sep 2004 22:13:09 +0100 Subject: [Box Backup] multiple Path and/or ExcludeDir In-Reply-To: <20040901204614.GA1443@sarsation.hixsplit.hu> References: <17657408.1093965339870.JavaMail.www-data@www> <98A23F8E-FB69-11D8-A954-000A95AFF7F8@fluffy.co.uk> <20040831181953.GA1335@sarsation.hixsplit.hu> <20040901204614.GA1443@sarsation.hixsplit.hu> Message-ID: On 1 Sep 2004, at 21:46, dctester at freestart.hu wrote: > On Tue, Aug 31, 2004 at 07:31:09PM +0100, Ben Summers wrote: > >> I recommend you create separate entries for each directory within the >> root directory that you want to back up. You really do not want to >> have >> mount points within these directories, which you do at the moment. > > Hello Ben, > > Finally I changed configuration to this: > > BackupLocations > { > bin > { > Path = /bin > } > [snip] > usr > { > Path = /usr > } > var > { > Path = /var > } > } > > Working perfectly. Thanks your advice! That is my recommended configuration. I really must write the code which detects that your backup locations span mount points. And also stops /proc being backed up on Linux! I meant to reply to your previous post (I'm in the process of moving house at the moment, so things are difficult). I presume you changed the config file and then sent a -HUP signal to bbackupd. This was probably in the middle of backing up the contents of your computer's memory and processes from /proc, which would have taken a while, and it will have postponed the restart until it had finished a few files. So you wouldn't have noticed anything changing for quite a while, explaining why you didn't see things working. While I think the core backup engine of Box Backup works very nicely, there are a few bits around it which need a some work before version 1.00 can be reached. Thanks for all your patience. Ben From boxbackup at fluffy.co.uk Wed Sep 1 22:18:58 2004 From: boxbackup at fluffy.co.uk (Andreas Schrafl) Date: Wed, 01 Sep 2004 23:18:58 +0200 Subject: [Box Backup] trigger housekeeping In-Reply-To: <77792E8A-FC53-11D8-B2DB-000A95E8BABA@fluffy.co.uk> References: <413629E7.3020107@jetnet.ch> <77792E8A-FC53-11D8-B2DB-000A95E8BABA@fluffy.co.uk> Message-ID: <41363CC2.1040408@jetnet.ch> Ben Summers wrote: > > On 1 Sep 2004, at 20:58, Andreas Schrafl wrote: > >> I reduced the amount of my soft and hardlimit and hoped to get the olf >> files deleted. the next backup run said i'd have so add more storage >> but didn't delet any old files. >> >> when does housekeeping run? > > > Every x seconds, where x is specified in bbstored.conf as > TimeBetweenHousekeeping. > had it on the standard 900 so every 15 minutes. it's now more than a day that I reduced the limit and it is still like not deleted anything. > >> how can I trigger it? > > > Send a -HUP signal to bbstored. > tried this (kill -HUP processid) for bbstored and housekeeping but didn't change anything > What do you think should have happened when you changed the limits. What > would you like to have done? How would you like to have controlled this? > I'd just like to control this on the server. something like: bbstoredhousekeeping accountid or bbstoredhousekeeping would clean the files that can be deleted. perhaps I would be possible to specify the files to delete (old or deleted or both). I don't even need this on the client but havinig it there would be great to. I run 0.7 version and notyet 0.7PLUS > I have server support for adding a flag to certain files and objects > which means "delete as soon as possible", that is, when they become an > old version or are marked as deleted by the client. I am intending to > expose this functionality to the user via bbackupquery. > >> >> thanks for a great backup solution >> (I'm trying to make a windows client that is usable for "normal" >> windows users) > > > I'm glad you're finding it useful! > > Ben > > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > > From boxbackup at fluffy.co.uk Wed Sep 1 22:32:33 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 1 Sep 2004 22:32:33 +0100 Subject: [Box Backup] trigger housekeeping In-Reply-To: <41363CC2.1040408@jetnet.ch> References: <413629E7.3020107@jetnet.ch> <77792E8A-FC53-11D8-B2DB-000A95E8BABA@fluffy.co.uk> <41363CC2.1040408@jetnet.ch> Message-ID: <6FB57029-FC5E-11D8-B2DB-000A95E8BABA@fluffy.co.uk> On 1 Sep 2004, at 22:18, Andreas Schrafl wrote: > > > Ben Summers wrote: >> On 1 Sep 2004, at 20:58, Andreas Schrafl wrote: >>> I reduced the amount of my soft and hardlimit and hoped to get the >>> olf files deleted. the next backup run said i'd have so add more >>> storage but didn't delet any old files. >>> >>> when does housekeeping run? >> Every x seconds, where x is specified in bbstored.conf as >> TimeBetweenHousekeeping. > had it on the standard 900 so every 15 minutes. it's now more than a > day that I reduced the limit and it is still like not deleted > anything. Interesting. What is the output of bbstoreaccounts info x where x is the relevant account number? > >>> how can I trigger it? >> Send a -HUP signal to bbstored. > tried this (kill -HUP processid) for bbstored and housekeeping but > didn't change anything > >> What do you think should have happened when you changed the limits. >> What would you like to have done? How would you like to have >> controlled this? > I'd just like to control this on the server. something like: > bbstoredhousekeeping accountid > or > bbstoredhousekeeping > > would clean the files that can be deleted. perhaps I would be possible > to specify the files to delete (old or deleted or both). I don't even > need this on the client but havinig it there would be great to. It would have to be on the client, as the server has no idea what the files are called. > > I run 0.7 version and notyet 0.7PLUS Upgrading won't change this, but it would be worthwhile. Ben From boxbackup at fluffy.co.uk Thu Sep 2 23:15:26 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Fri, 3 Sep 2004 00:15:26 +0200 Subject: [Box Backup] multiple Path and/or ExcludeDir In-Reply-To: References: <17657408.1093965339870.JavaMail.www-data@www> <98A23F8E-FB69-11D8-A954-000A95AFF7F8@fluffy.co.uk> <20040831181953.GA1335@sarsation.hixsplit.hu> <20040901204614.GA1443@sarsation.hixsplit.hu> Message-ID: <20040902221526.GA8689@sarsation.hixsplit.hu> On Wed, Sep 01, 2004 at 10:13:09PM +0100, Ben Summers wrote: > I meant to reply to your previous post (I'm in the process of moving > house at the moment, so things are difficult). I presume you changed > the config file and then sent a -HUP signal to bbackupd. No, I totally stop the bbackupd daemon and recreated account on bbstored server (delete / create). > While I think the core backup engine of Box Backup works very nicely, > there are a few bits around it which need a some work before version > 1.00 can be reached. Thanks for all your patience. Lot of time is require to get a great job! :-) Have a nice day. Amily From boxbackup at fluffy.co.uk Thu Sep 2 07:42:26 2004 From: boxbackup at fluffy.co.uk (Joris) Date: Thu, 2 Sep 2004 08:42:26 +0200 (CEST) Subject: [Box Backup] "backing up" backups Message-ID: <1241.10.97.1.4.1094107346.squirrel@10.97.1.4> Hi, I'd like to copy my encrypted archives to another host, to protect against hard drive failure etc. (serveral sync tools are available, I was thinking about unison) I want to do this while the boxbackup-server is possibly accepting new data (don't want to stop the bbserver just to copy the archives). Is there a way to make sure the backups are in a complete state while the boxbackup server is possibly accepting fresh updates? How is this taken care of in the design? -- Greetings, Joris From boxbackup at fluffy.co.uk Fri Sep 3 17:02:56 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 3 Sep 2004 17:02:56 +0100 Subject: [Box Backup] "backing up" backups In-Reply-To: <1241.10.97.1.4.1094107346.squirrel@10.97.1.4> References: <1241.10.97.1.4.1094107346.squirrel@10.97.1.4> Message-ID: On 2 Sep 2004, at 07:42, Joris wrote: > Hi, > > > I'd like to copy my encrypted archives to another host, to protect > against > hard drive failure etc. If you have three hard discs, the server has built in software RAID. > (serveral sync tools are available, I was thinking > about unison) I want to do this while the boxbackup-server is possibly > accepting new data (don't want to stop the bbserver just to copy the > archives). > > Is there a way to make sure the backups are in a complete state while > the > boxbackup server is possibly accepting fresh updates? Not really. You could monitor the syslog output and detect when a client disconnects, I suppose. However, you could just not worry about it, and let the sync tool just do it's thing. If you needed to use the backup of the backup, then you can use the account fix tool to get everything to a state where you can get the files. The worst that could happen is that a file or directory is in the "lost+found" directory rather than where it should actually be. > > How is this taken care of in the design? Eventually you will be able to replicate an account to another server. When the client disconnects, changes will be transferred efficiently to the backup server. The existing code is written with this in mind. Ben From boxbackup at fluffy.co.uk Sat Sep 4 05:01:30 2004 From: boxbackup at fluffy.co.uk (brett) Date: Fri, 3 Sep 2004 21:01:30 -0700 Subject: [Box Backup] Re: boxbackup digest, Vol 1 #125 - 2 msgs In-Reply-To: <20040903110004.13971.16685.Mailman@love.warhead.org.uk> References: <20040903110004.13971.16685.Mailman@love.warhead.org.uk> Message-ID: <1A16AEF2-FE27-11D8-AF52-003065A1F884@librum.org> Maybe a better idea would be to add provisions to boxbackup for multihost backups. It's really the only thing i can think of that's missing ;) b On Sep 3, 2004, at 4:00 AM, boxbackup-request at fluffy.co.uk wrote: > Message: 2 > Date: Thu, 2 Sep 2004 08:42:26 +0200 (CEST) > From: "Joris" > To: boxbackup at fluffy.co.uk > Subject: [Box Backup] "backing up" backups > Reply-To: boxbackup at fluffy.co.uk > > Hi, > > > I'd like to copy my encrypted archives to another host, to protect > against > hard drive failure etc. (serveral sync tools are available, I was > thinking > about unison) I want to do this while the boxbackup-server is possibly > accepting new data (don't want to stop the bbserver just to copy the > archives). > > Is there a way to make sure the backups are in a complete state while > the > boxbackup server is possibly accepting fresh updates? > > How is this taken care of in the design? > > > -- > Greetings, > Joris > > > > --__--__-- > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > > > End of boxbackup Digest > > !DSPAM:41384f12322691907010421! > From boxbackup at fluffy.co.uk Sat Sep 4 08:09:59 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Sat, 4 Sep 2004 08:09:59 +0100 Subject: [Box Backup] Re: boxbackup digest, Vol 1 #125 - 2 msgs In-Reply-To: <1A16AEF2-FE27-11D8-AF52-003065A1F884@librum.org> References: <20040903110004.13971.16685.Mailman@love.warhead.org.uk> <1A16AEF2-FE27-11D8-AF52-003065A1F884@librum.org> Message-ID: <6EC0BA0D-FE41-11D8-B2DB-000A95E8BABA@fluffy.co.uk> I suppose you could run two copies of bbackupd... Ben On 4 Sep 2004, at 05:01, brett wrote: > > Maybe a better idea would be to add provisions to boxbackup for > multihost backups. It's really the only thing i can think of that's > missing ;) > > b > > On Sep 3, 2004, at 4:00 AM, boxbackup-request at fluffy.co.uk wrote: >> Message: 2 >> Date: Thu, 2 Sep 2004 08:42:26 +0200 (CEST) >> From: "Joris" >> To: boxbackup at fluffy.co.uk >> Subject: [Box Backup] "backing up" backups >> Reply-To: boxbackup at fluffy.co.uk >> >> Hi, >> >> >> I'd like to copy my encrypted archives to another host, to protect >> against >> hard drive failure etc. (serveral sync tools are available, I was >> thinking >> about unison) I want to do this while the boxbackup-server is possibly >> accepting new data (don't want to stop the bbserver just to copy the >> archives). >> >> Is there a way to make sure the backups are in a complete state while >> the >> boxbackup server is possibly accepting fresh updates? >> >> How is this taken care of in the design? >> >> >> -- >> Greetings, >> Joris >> >> >> >> --__--__-- From boxbackup at fluffy.co.uk Sat Sep 4 17:02:24 2004 From: boxbackup at fluffy.co.uk (brett) Date: Sat, 4 Sep 2004 09:02:24 -0700 Subject: [Box Backup] Re: boxbackup digest, Vol 1 #126 - 3 msgs In-Reply-To: <20040904110006.12178.14741.Mailman@love.warhead.org.uk> References: <20040904110006.12178.14741.Mailman@love.warhead.org.uk> Message-ID: That's an interesting point, especially since it wouldn't introduce any additional load on the backup server, but rather on the client. I think while it would be nice to have server-replication built in to bbstored, being able to have multiple copies of bbackupd running definitely alleviates the immediate need, and is probably better for most situations since it keeps the 'cost' paired with the 'need' b > From: Ben Summers > Subject: Re: [Box Backup] Re: boxbackup digest, Vol 1 #125 - 2 msgs > Date: Sat, 4 Sep 2004 08:09:59 +0100 > To: boxbackup at fluffy.co.uk > Reply-To: boxbackup at fluffy.co.uk > > > I suppose you could run two copies of bbackupd... > > Ben > > > On 4 Sep 2004, at 05:01, brett wrote: > >> >> Maybe a better idea would be to add provisions to boxbackup for >> multihost backups. It's really the only thing i can think of that's >> missing ;) >> >> b >> From boxbackup at fluffy.co.uk Wed Sep 8 00:34:32 2004 From: boxbackup at fluffy.co.uk (Andrew Farago) Date: Wed, 8 Sep 2004 11:34:32 +1200 Subject: [Box Backup] Client and Server on same machine Message-ID: <200409081134.32867.andrew@nbn.co.nz> Client and Server on same machine is possible? Because on my machine it doesn't works, the bbackupd give me the next message always: bbackupd[19245]: Exception caught (7/34), reset state and waiting to retry... Andrew Ps.:Of course just for test. From boxbackup at fluffy.co.uk Wed Sep 8 09:54:53 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 8 Sep 2004 09:54:53 +0100 Subject: [Box Backup] Client and Server on same machine In-Reply-To: <200409081134.32867.andrew@nbn.co.nz> References: <200409081134.32867.andrew@nbn.co.nz> Message-ID: On 8 Sep 2004, at 00:34, Andrew Farago wrote: > Client and Server on same machine is possible? Yes. The tests do it -- check out the configuration files in test/bbackupd/testfiles, everything you need will be there. > Because on my machine it doesn't works, the bbackupd give me the next > message > always: > bbackupd[19245]: Exception caught (7/34), reset state and waiting to > retry... > Andrew > Ps.:Of course just for test. Turn on ExtendedLogging, and see what the logs say on server and client. It would be nice to know exactly where this is going wrong -- without the context this error isn't amazingly helpful. You can look up the exception codes in the ExceptionCodes.txt file generated in the root of the distribution when you build it. Ben From boxbackup at fluffy.co.uk Thu Sep 9 03:53:33 2004 From: boxbackup at fluffy.co.uk (Andrew Farago) Date: Thu, 9 Sep 2004 14:53:33 +1200 Subject: [Box Backup] Client and Server on same machine In-Reply-To: References: <200409081134.32867.andrew@nbn.co.nz> Message-ID: <200409091453.33677.andrew.farago@nbn.co.nz> On Wednesday 08 September 2004 20:54, Ben Summers wrote: > > You can look up the exception codes in the ExceptionCodes.txt file > generated in the root of the distribution when you build it. > > Ben > How works the BoxBackup protocol? Maybe it is a networking problem in real. Can i keep a backup client behind a NAT firewall? Andrew From boxbackup at fluffy.co.uk Thu Sep 9 08:54:58 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 9 Sep 2004 08:54:58 +0100 Subject: [Box Backup] Client and Server on same machine In-Reply-To: <200409091453.33677.andrew.farago@nbn.co.nz> References: <200409081134.32867.andrew@nbn.co.nz> <200409091453.33677.andrew.farago@nbn.co.nz> Message-ID: <8BAC305C-0235-11D9-A4CA-000A95AFF7F8@fluffy.co.uk> On 9 Sep 2004, at 03:53, Andrew Farago wrote: > On Wednesday 08 September 2004 20:54, Ben Summers wrote: >> >> You can look up the exception codes in the ExceptionCodes.txt file >> generated in the root of the distribution when you build it. >> >> Ben >> > How works the BoxBackup protocol? Maybe it is a networking problem in > real. > Can i keep a backup client behind a NAT firewall? The backup protocol involves the client opening a simple TCP/IP connection to the server, which is encrypted using TLS (using OpenSSL). This is completely compatible with NAT firewalls, in fact, most of the clients I run are behind such things. Ben From boxbackup at fluffy.co.uk Sat Sep 11 23:04:59 2004 From: boxbackup at fluffy.co.uk (Mario S. Mommer) Date: Sat, 11 Sep 2004 15:04:59 -0700 (PDT) Subject: [Box Backup] openssl: no Message-ID: <20040911220459.75670.qmail@web41502.mail.yahoo.com> Hi, I am trying to compile boxbackup-0.07 on debian stable. I have installed openssl-0.9.7 from www.backports.org/debian, and libssl-dev. There are openssl header files in /usr/include/openssl, and in /usr/lib there is dementor:/usr/lib# ls -lF libssl* -rw-r--r-- 1 root root 255058 Jul 28 14:50 libssl.a lrwxrwxrwx 1 root root 15 Sep 11 21:21 libssl.so -> libssl.so.0.9.7 -rw-r--r-- 1 root root 182084 Mar 3 2004 libssl.so.0.9.6 -rw-r--r-- 1 root root 186372 Jul 28 14:50 libssl.so.0.9.7 -rw-r--r-- 1 root root 269186 Jul 28 14:50 libssl_pic.a dementor:/usr/lib# Well, whatever I do, if I run configure in boxbackup-0.07 it says Checking environment... OpenSSL 1: no OpenSSL 2: no curses: no ncurses: no USE_MALLOC: no Finished checking headers --------------------- WARNING: readline isn't installed --------------------- --------------------- WARNING: db is not installed -- will run in reduced efficiency mode without it. --------------------- ERROR: You need to install OpenSSL, preferably at least version 0.9.7. If you believe you have installed OpenSSL, check that the headers are installed as well ('dev' packages?) See documentation on web site if you need to add extra search paths. =================================end ./configure output=================== I don't really know what it wants, so I have a hard time figuring out why it is of the opinion that OpenSSL is not installed. I looked up the docs, but There is indeed a configuration file /etc/ssl/openssl.cnf which is left there by the debian package; so I don't believe there is aproblem with this. Any help would be greatly appreciated! Regards, Mario. __________________________________ Do you Yahoo!? New and Improved Yahoo! Mail - 100MB free storage! http://promotions.yahoo.com/new_mail From boxbackup at fluffy.co.uk Sun Sep 12 00:16:08 2004 From: boxbackup at fluffy.co.uk (Imran) Date: Sat, 11 Sep 2004 18:16:08 -0500 Subject: [Box Backup] openssl: no In-Reply-To: <20040911220459.75670.qmail@web41502.mail.yahoo.com> References: <20040911220459.75670.qmail@web41502.mail.yahoo.com> Message-ID: <20040911231132.M23754@naweb.com> Hmm, i've had issues with box backup finding the SSL on one of my servers. i think it was fedora core, or maybe redhat 8. anyway, it may be a similiar issue. I usually end up configuring boxbackup with this commandline: ./configure compile:-I/usr/local/ssl/include link:-L/usr/local/ssl/lib also check to make sure that (i don't know how it is on debian but on redhat...) the path for the ssl lib is in the /etc/ld.so.conf, i.e. /usr/local/ssl/lib, and that you run ldconfig. I always forget if that would matter for the configure script. anyway, try it and let us know... Imran Niazi --- http://www.niazi.net/ From boxbackup at fluffy.co.uk Sun Sep 12 09:54:39 2004 From: boxbackup at fluffy.co.uk (Mario S. Mommer) Date: Sun, 12 Sep 2004 01:54:39 -0700 (PDT) Subject: [Box Backup] openssl: no In-Reply-To: <20040911231132.M23754@naweb.com> Message-ID: <20040912085439.83264.qmail@web41511.mail.yahoo.com> --- Imran wrote: > Hmm, i've had issues with box backup finding the SSL on one of my > servers. i think it was fedora core, or maybe redhat 8. anyway, it may > be a similiar issue. I usually end up configuring boxbackup with this > commandline: > > ./configure compile:-I/usr/local/ssl/include link:-L/usr/local/ssl/lib Changed it to reflect my current situation but to no avail. I really wonder what is going on here, since in fact db is installed too. There seems to be a few mismatches between libc version and the backported openssl. Oh man. Has anyone in this list been successfull building boxbackup on debian vintage^Wstable? > also check to make sure that (i don't know how it is on debian but on > redhat...) the path for the ssl lib is in the /etc/ld.so.conf, i.e. > /usr/local/ssl/lib, and that you run ldconfig. I always forget if that > would > matter for the configure script. Well, /etc/ld.so.conf does not exists. Hm. I ran ldconfig but nothing changed. > anyway, try it and let us know... No good news, unfortunately :-| Regards, Mario. __________________________________ Do you Yahoo!? Yahoo! Mail Address AutoComplete - You start. We finish. http://promotions.yahoo.com/new_mail From boxbackup at fluffy.co.uk Sun Sep 12 10:11:25 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Sun, 12 Sep 2004 10:11:25 +0100 Subject: [Box Backup] openssl: no In-Reply-To: <20040912085439.83264.qmail@web41511.mail.yahoo.com> References: <20040912085439.83264.qmail@web41511.mail.yahoo.com> Message-ID: On 12 Sep 2004, at 09:54, Mario S. Mommer wrote: > > --- Imran wrote: >> Hmm, i've had issues with box backup finding the SSL on one of my >> servers. i think it was fedora core, or maybe redhat 8. anyway, it >> may >> be a similiar issue. I usually end up configuring boxbackup with this >> commandline: >> >> ./configure compile:-I/usr/local/ssl/include link:-L/usr/local/ssl/lib > > Changed it to reflect my current situation but to no avail. I really > wonder what is going on here, since in fact db is installed too. There > seems to be a few mismatches between libc version and the backported > openssl. Oh man. > > Has anyone in this list been successfull building boxbackup on debian > vintage^Wstable? That is actually my first Linux test platform. I wonder if you have some issues with your compiler. The latest version tests that you have a working compiler -- download this and try it instead. http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.07PLUS3.tgz > >> also check to make sure that (i don't know how it is on debian but on >> redhat...) the path for the ssl lib is in the /etc/ld.so.conf, i.e. >> /usr/local/ssl/lib, and that you run ldconfig. I always forget if >> that >> would >> matter for the configure script. > > Well, /etc/ld.so.conf does not exists. Hm. I ran ldconfig but nothing > changed. I am not a Linux expert, I'm afraid. But checking the compiler is the first thing -- you are also missing everything else it checks for, which sounds suspiciously like a non-working compiler. Ben From boxbackup at fluffy.co.uk Sun Sep 12 10:12:54 2004 From: boxbackup at fluffy.co.uk (Joris) Date: Sun, 12 Sep 2004 11:12:54 +0200 (CEST) Subject: [Box Backup] openssl: no In-Reply-To: <20040912085439.83264.qmail@web41511.mail.yahoo.com> References: <20040911231132.M23754@naweb.com> <20040912085439.83264.qmail@web41511.mail.yahoo.com> Message-ID: <1979.10.97.1.4.1094980374.squirrel@10.97.1.4> For what it's worth: the new stable release won't be long anymore. Optimists think next week, pessemists by the end of october. I've had no troubles building boxbackup on sarge. -- Greetings, Joris From boxbackup at fluffy.co.uk Sun Sep 12 11:55:33 2004 From: boxbackup at fluffy.co.uk (Mario S. Mommer) Date: Sun, 12 Sep 2004 03:55:33 -0700 (PDT) Subject: [Box Backup] openssl: no In-Reply-To: Message-ID: <20040912105533.70008.qmail@web41507.mail.yahoo.com> --- Ben Summers wrote: > > On 12 Sep 2004, at 09:54, Mario S. Mommer wrote: > > Has anyone in this list been successfull building boxbackup on debian > > vintage^Wstable? > > That is actually my first Linux test platform. > > I wonder if you have some issues with your compiler. The latest version > tests that you have a working compiler -- download this and try it > instead. > > http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.07PLUS3.tgz Yes, that was it. I had installed gcc but not g++. It configures and finds everything. Thanks! Regards, Mario. __________________________________ Do you Yahoo!? Yahoo! Mail - 50x more storage than other providers! http://promotions.yahoo.com/new_mail From boxbackup at fluffy.co.uk Sun Sep 12 13:21:33 2004 From: boxbackup at fluffy.co.uk (Mario S. Mommer) Date: Sun, 12 Sep 2004 05:21:33 -0700 (PDT) Subject: [Box Backup] openssl: no In-Reply-To: <20040912105533.70008.qmail@web41507.mail.yahoo.com> Message-ID: <20040912122133.14292.qmail@web41510.mail.yahoo.com> --- "Mario S. Mommer" wrote: > --- Ben Summers wrote: > > I wonder if you have some issues with your compiler. The latest > version > > tests that you have a working compiler -- download this and try it > > instead. > > > > http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.07PLUS3.tgz > > Yes, that was it. I had installed gcc but not g++. It configures and > finds > everything. ...and runs smoothly too :-) Many thanks! Regards, Mario. __________________________________ Do you Yahoo!? Yahoo! Mail - 50x more storage than other providers! http://promotions.yahoo.com/new_mail From boxbackup at fluffy.co.uk Mon Sep 13 12:23:44 2004 From: boxbackup at fluffy.co.uk (=?ISO-8859-1?Q?J=E9r=F4me_Schell?=) Date: Mon, 13 Sep 2004 13:23:44 +0200 Subject: [Box Backup] openssl: no In-Reply-To: <20040912085439.83264.qmail@web41511.mail.yahoo.com> References: <20040912085439.83264.qmail@web41511.mail.yahoo.com> Message-ID: <41458340.4070203@myreseau.org> Mario S. Mommer a ?crit : > > Has anyone in this list been successfull building boxbackup on debian > vintage^Wstable? > I didn't try the last revision yet but I have made Woody Debian packages for v0.07. You can add this to your sources.list if you want to give it a try : deb http://debian.taonix.org/ stable main and apt-get install boxbackup-client or apt-get install boxbackup-server Bye -- J?r?me From boxbackup at fluffy.co.uk Mon Sep 13 21:20:17 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Mon, 13 Sep 2004 21:20:17 +0100 Subject: [Box Backup] PATCH: Fixes for linux/amd64 Message-ID: <1095106817.21742.9.camel@avenin.ebourne.me.uk> --=-gBUE13NuUlagTewPSvYO Content-Type: text/plain Content-Transfer-Encoding: 7bit Hi, This patch fixes a bunch of warnings when compiling on Fedora Core 2, AMD64. Two of these warnings were definite bugs - the SplitString one caused an infinite loop and subsequent out of memory. Unfortunately a lot of these had to be fixed with casts. I've tested the patch on Fedora Core 2/i386 and they still seem ok. I didn't fix a whole load of warnings about mismatch between printf's %llx and the actual type, I guess that would probably result in warnings for 32 bit archs. Cheers, Martin. --=-gBUE13NuUlagTewPSvYO Content-Description: Content-Disposition: inline; filename=boxbackup-0.07-x86_64.patch Content-Type: text/x-patch; charset=UTF-8 Content-Transfer-Encoding: 7bit diff -ur boxbackup-0.07.orig/lib/backupclient/BackupStoreFileDiff.cpp boxbackup-0.07/lib/backupclient/BackupStoreFileDiff.cpp --- boxbackup-0.07.orig/lib/backupclient/BackupStoreFileDiff.cpp 2004-09-09 20:49:19.869539201 +0100 +++ boxbackup-0.07/lib/backupclient/BackupStoreFileDiff.cpp 2004-09-09 21:19:03.188201856 +0100 @@ -147,7 +147,7 @@ int64_t blockHeaderPosFromEnd = ((numBlocks * sizeof(file_BlockIndexEntry)) + sizeof(file_BlockIndexHeader)); // Sanity check - if(blockHeaderPosFromEnd > fileSize - sizeof(file_StreamFormat)) + if(blockHeaderPosFromEnd > static_cast(fileSize - sizeof(file_StreamFormat))) { THROW_EXCEPTION(BackupStoreException, BadBackupStoreFile) } @@ -627,7 +627,7 @@ break; } - if(rFoundBlocks.size() > (NumBlocks * BACKUP_FILE_DIFF_MAX_BLOCK_FIND_MULTIPLE) + if(static_cast(rFoundBlocks.size()) > (NumBlocks * BACKUP_FILE_DIFF_MAX_BLOCK_FIND_MULTIPLE) || sDiffTimedOut) { abortSearch = true; diff -ur boxbackup-0.07.orig/lib/backupclient/BackupStoreFileEncodeStream.cpp boxbackup-0.07/lib/backupclient/BackupStoreFileEncodeStream.cpp --- boxbackup-0.07.orig/lib/backupclient/BackupStoreFileEncodeStream.cpp 2004-09-09 20:49:19.871538971 +0100 +++ boxbackup-0.07/lib/backupclient/BackupStoreFileEncodeStream.cpp 2004-09-09 21:24:31.767368140 +0100 @@ -413,14 +413,14 @@ ++mInstructionNumber; // Skip instructions which don't contain any data - while(mInstructionNumber < mpRecipe->size() + while(mInstructionNumber < static_cast(mpRecipe->size()) && (*mpRecipe)[mInstructionNumber].mSpaceBefore == 0) { SkipPreviousBlocksInInstruction(); ++mInstructionNumber; } - if(mInstructionNumber >= mpRecipe->size()) + if(mInstructionNumber >= static_cast(mpRecipe->size())) { // End of blocks, go to next phase ++mStatus; diff -ur boxbackup-0.07.orig/lib/backupstore/BackupStoreInfo.cpp boxbackup-0.07/lib/backupstore/BackupStoreInfo.cpp --- boxbackup-0.07.orig/lib/backupstore/BackupStoreInfo.cpp 2004-09-09 20:49:19.945530451 +0100 +++ boxbackup-0.07/lib/backupstore/BackupStoreInfo.cpp 2004-09-09 21:33:06.883055976 +0100 @@ -258,7 +258,7 @@ } // Final check - if(info->mDeletedDirectories.size() != numDelObj) + if(static_cast(info->mDeletedDirectories.size()) != numDelObj) { THROW_EXCEPTION(BackupStoreException, BadStoreInfoOnLoad) } diff -ur boxbackup-0.07.orig/lib/common/ConversionString.cpp boxbackup-0.07/lib/common/ConversionString.cpp --- boxbackup-0.07.orig/lib/common/ConversionString.cpp 2004-09-09 20:49:19.908534711 +0100 +++ boxbackup-0.07/lib/common/ConversionString.cpp 2004-09-09 21:39:12.690935619 +0100 @@ -85,7 +85,7 @@ // Convert. char *numEnd = 0; - int32_t r = ::strtol(pString, &numEnd, 0); + long r = ::strtol(pString, &numEnd, 0); // Check that all the characters were used if(*numEnd != '\0') diff -ur boxbackup-0.07.orig/lib/common/Utils.cpp boxbackup-0.07/lib/common/Utils.cpp --- boxbackup-0.07.orig/lib/common/Utils.cpp 2004-09-09 20:49:19.910534481 +0100 +++ boxbackup-0.07/lib/common/Utils.cpp 2004-09-09 21:13:29.612610879 +0100 @@ -73,8 +73,8 @@ void SplitString(const std::string &String, char SplitOn, std::vector &rOutput) { // Split it up. - unsigned int b = 0; - unsigned int e = 0; + std::string::size_type b = 0; + std::string::size_type e = 0; while(e = String.find_first_of(SplitOn, b), e != String.npos) { // Get this string diff -ur boxbackup-0.07.orig/lib/raidfile/RaidFileRead.cpp boxbackup-0.07/lib/raidfile/RaidFileRead.cpp --- boxbackup-0.07.orig/lib/raidfile/RaidFileRead.cpp 2004-09-09 20:49:19.947530221 +0100 +++ boxbackup-0.07/lib/raidfile/RaidFileRead.cpp 2004-09-09 21:32:05.600112302 +0100 @@ -1252,7 +1252,7 @@ pos_type paritySize = st.st_size; FileSizeType parityLastData = 0; bool parityIntegralPlusOffT = ((paritySize % blockSize) == sizeof(FileSizeType)); - if(paritySize >= sizeof(parityLastData) && (parityIntegralPlusOffT || stripe1 != -1)) + if(paritySize >= static_cast(sizeof(parityLastData)) && (parityIntegralPlusOffT || stripe1 != -1)) { // Seek to near the end ASSERT(sizeof(FileSizeType) == 8); // compiler bug (I think) prevents from using 0 - sizeof(FileSizeType)... @@ -1367,7 +1367,7 @@ // No. Working out the size is easy. length = stripe2Size + (((stripe2Size / blockSize)+1) * blockSize); // Got last block size in there? - if((stripe2Size % blockSize) <= (blockSize - sizeof(pos_type))) + if((stripe2Size % blockSize) <= static_cast((blockSize - sizeof(pos_type)))) { // Yes... lastBlockHasSize = true; --=-gBUE13NuUlagTewPSvYO-- From boxbackup at fluffy.co.uk Mon Sep 13 22:14:33 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 13 Sep 2004 22:14:33 +0100 Subject: [Box Backup] PATCH: Fixes for linux/amd64 In-Reply-To: <1095106817.21742.9.camel@avenin.ebourne.me.uk> References: <1095106817.21742.9.camel@avenin.ebourne.me.uk> Message-ID: On 13 Sep 2004, at 21:20, Martin Ebourne wrote: > Hi, > > This patch fixes a bunch of warnings when compiling on Fedora Core 2, > AMD64. > > Two of these warnings were definite bugs - the SplitString one caused > an > infinite loop and subsequent out of memory. Unfortunately a lot of > these > had to be fixed with casts. > > I've tested the patch on Fedora Core 2/i386 and they still seem ok. Fantastic, thanks for that. I believe someone else could use this right now... I will get it applied to the next version. > > I didn't fix a whole load of warnings about mismatch between printf's > %llx and the actual type, I guess that would probably result in > warnings > for 32 bit archs. You're right, that would probably be an undesirable change. I may look at this later, but I don't really think it's worth the effort (unless shown otherwise). Thanks, much appreciated. Ben From boxbackup at fluffy.co.uk Mon Sep 13 23:03:49 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Mon, 13 Sep 2004 23:03:49 +0100 Subject: [Box Backup] Fedora Core 2 RPMs Message-ID: <1095113029.21742.22.camel@avenin.ebourne.me.uk> --=-pO8nwjUYQoDUU79I1u1l Content-Type: text/plain Content-Transfer-Encoding: 7bit Hi, Enclosed are files to make an RPM of box backup. These have been built for Fedora Core 2, but the RPM should probably compile and work on any RPM distribution. The compiled RPMs are available from here: http://www.ebourne.me.uk/rpms/boxbackup-0.07-3.src.rpm http://www.ebourne.me.uk/rpms/boxbackup-client-0.07-3.i386.rpm http://www.ebourne.me.uk/rpms/boxbackup-server-0.07-3.i386.rpm http://www.ebourne.me.uk/rpms/boxbackup-client-0.07-3.x86_64.rpm http://www.ebourne.me.uk/rpms/boxbackup-server-0.07-3.x86_64.rpm Please note this is not a permanent location, I would appreciate it if you could host these on the main website. The new files needed to build the RPM are attached. There are two start scripts and the spec file. These could probably be added to the tar distribution. The patch referenced by the spec file is the x86 patch I sent earlier - it can be removed when committed. Cheers, Martin. --=-pO8nwjUYQoDUU79I1u1l Content-Disposition: attachment; filename=boxbackup.spec Content-Type: text/plain; name=boxbackup.spec; charset=UTF-8 Content-Transfer-Encoding: 7bit %define bb_user_id 171 %define ident %{name}-%{version} Summary: An automatic on-line backup system for UNIX. Name: boxbackup Version: 0.07 Release: 3 License: BSD Group: Applications/Archiving URL: http://www.fluffy.co.uk/boxbackup/ Source0: %{ident}.tgz Source1: bbackupd Source2: bbstored Patch0: boxbackup-0.07-x86_64.patch Requires: openssl >= 0.9.7a BuildRoot: %{_tmppath}/%{ident}-%{release}-root BuildRequires: openssl-devel %description Box Backup is a completely automatic on-line backup system. Backed up files are stored encrypted on a filesystem on a remote server, which does not need to be trusted. The backup server runs as a daemon on the client copying only the changes within files, and old versions and deleted files are retained. It is designed to be easy and cheap to run a server and (optional) RAID is implemented in userland for ease of use. %package client Summary: An automatic on-line backup system for UNIX. Group: Applications/Archiving %description client Box Backup is a completely automatic on-line backup system. Backed up files are stored encrypted on a filesystem on a remote server, which does not need to be trusted. The backup server runs as a daemon on the client copying only the changes within files, and old versions and deleted files are retained. It is designed to be easy and cheap to run a server and (optional) RAID is implemented in userland for ease of use. This package contains the client. %package server Summary: An automatic on-line backup system for UNIX. Group: System Environment/Daemons %description server Box Backup is a completely automatic on-line backup system. Backed up files are stored encrypted on a filesystem on a remote server, which does not need to be trusted. The backup server runs as a daemon on the client copying only the changes within files, and old versions and deleted files are retained. It is designed to be easy and cheap to run a server and (optional) RAID is implemented in userland for ease of use. This package contains the server. %prep %setup -q %patch0 -p1 %build ./configure make %install rm -rf $RPM_BUILD_ROOT mkdir -p $RPM_BUILD_ROOT%{_docdir}/%{ident} mkdir -p $RPM_BUILD_ROOT%{_bindir} mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/rc.d/init.d mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/box/bbackupd mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/box/bbstored mkdir -p $RPM_BUILD_ROOT%{_var}/lib/box install -m 644 BUGS.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} install -m 644 LINUX.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} install -m 644 VERSION.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} install -m 644 CONTACT.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} install -m 644 DOCUMENTATION.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} install -m 644 ExceptionCodes.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} install -m 644 THANKS.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} install -m 644 LICENSE.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} install -m 644 TODO.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} # Client touch $RPM_BUILD_ROOT%{_sysconfdir}/box/bbackupd.conf install %{SOURCE1} $RPM_BUILD_ROOT%{_sysconfdir}/rc.d/init.d %define client_dir parcels/%{ident}-backup-client-Linux install %{client_dir}/bbackupd $RPM_BUILD_ROOT%{_bindir} install %{client_dir}/bbackupquery $RPM_BUILD_ROOT%{_bindir} install %{client_dir}/bbackupctl $RPM_BUILD_ROOT%{_bindir} install %{client_dir}/bbackupd-config $RPM_BUILD_ROOT%{_bindir} # Server touch $RPM_BUILD_ROOT%{_sysconfdir}/box/bbstored.conf touch $RPM_BUILD_ROOT%{_sysconfdir}/box/raidfile.conf install %{SOURCE2} $RPM_BUILD_ROOT%{_sysconfdir}/rc.d/init.d %define server_dir parcels/%{ident}-backup-server-Linux install %{server_dir}/bbstored $RPM_BUILD_ROOT%{_bindir} install %{server_dir}/bbstoreaccounts $RPM_BUILD_ROOT%{_bindir} install %{server_dir}/bbstored-certs $RPM_BUILD_ROOT%{_bindir} install %{server_dir}/bbstored-config $RPM_BUILD_ROOT%{_bindir} install %{server_dir}/raidfile-config $RPM_BUILD_ROOT%{_bindir} %pre server /usr/sbin/useradd -c "Box Backup" -u %{bb_user_id} \ -s /sbin/nologin -r -d %{backup_dir} box 2> /dev/null || : %post client /sbin/chkconfig --add bbackupd if [ ! -f %{_sysconfdir}/box/bbackupd.conf ]; then echo "You should run the following to configure the client:" echo "bbackupd-config /etc/box lazy /var/lib/box " echo "Then follow the instructions. Use this to start the client:" echo "service bbackupd start" fi %post server /sbin/chkconfig --add bbstored if [ ! -f %{_sysconfdir}/box/bbstored.conf ]; then echo "You should run the following to configure the server:" echo "raidfile-config /etc/box 2048 []" echo "bbstored-config /etc/box" `hostname` box echo "Then follow the instructions. Use this to start the server:" echo "service bbstored start" fi %preun client if [ $1 = 0 ]; then /sbin/service bbackupd stop > /dev/null 2>&1 /sbin/chkconfig --del bbackupd fi %preun server if [ $1 = 0 ]; then /sbin/service bbstored stop > /dev/null 2>&1 /sbin/chkconfig --del bbstored fi %clean rm -rf $RPM_BUILD_ROOT %files client %defattr(-,root,root,-) %dir %attr(700,root,root) %{_sysconfdir}/box/bbackupd %dir %attr(755,root,root) %{_var}/lib/box %doc %{_docdir}/%{ident}/*.txt %config %{_sysconfdir}/rc.d/init.d/bbackupd %config %ghost %{_sysconfdir}/box/bbackupd.conf %{_bindir}/bbackupd %{_bindir}/bbackupquery %{_bindir}/bbackupctl %{_bindir}/bbackupd-config %files server %defattr(-,root,root,-) %dir %attr(700,box,root) %{_sysconfdir}/box/bbstored %config %{_sysconfdir}/rc.d/init.d/bbstored %config %ghost %{_sysconfdir}/box/bbstored.conf %config %ghost %{_sysconfdir}/box/raidfile.conf %{_bindir}/bbstored %{_bindir}/bbstoreaccounts %{_bindir}/bbstored-certs %{_bindir}/bbstored-config %{_bindir}/raidfile-config %changelog * Thu Sep 9 2004 Martin Ebourne - 0.07-3 - Added x86_64 patch * Mon Sep 6 2004 Martin Ebourne - 0.07-1 - Initial build. --=-pO8nwjUYQoDUU79I1u1l Content-Disposition: attachment; filename=bbackupd Content-Type: application/x-shellscript; name=bbackupd Content-Transfer-Encoding: 7bit #! /bin/bash # # bbackupd Start/Stop the box backup daemon. # # chkconfig: 345 93 07 # description: bbackup is the client side deamon for Box Backup, a completely \ # automatic on-line backup system # processname: bbackupd # config: /etc/box # pidfile: /var/run/bbackupd.pid # Source function library. . /etc/init.d/functions RETVAL=0 # See how we were called. prog="bbackupd" # Check that configuration exists. [ -f /etc/box/$prog.conf ] || exit 0 start() { echo -n $"Starting $prog: " daemon $prog RETVAL=$? echo [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog return $RETVAL } stop() { echo -n $"Stopping $prog: " killproc $prog RETVAL=$? echo [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog return $RETVAL } rhstatus() { status $prog } restart() { stop start } reload() { echo -n $"Reloading $prog daemon configuration: " killproc $prog -HUP retval=$? echo return $RETVAL } case "$1" in start) start ;; stop) stop ;; restart) restart ;; status) rhstatus ;; condrestart) [ -f /var/lock/subsys/$prog ] && restart || : ;; *) echo $"Usage: $0 {start|stop|status|reload|restart|condrestart}" exit 1 esac exit $? --=-pO8nwjUYQoDUU79I1u1l Content-Disposition: attachment; filename=bbstored Content-Type: application/x-shellscript; name=bbstored Content-Transfer-Encoding: 7bit #! /bin/bash # # bbstored Start/Stop the box backup daemon. # # chkconfig: 345 93 07 # description: bbstore is the server side deamon for Box Backup, a completely \ # automatic on-line backup system # processname: bbstored # config: /etc/box # pidfile: /var/run/bbstored.pid # Source function library. . /etc/init.d/functions RETVAL=0 # See how we were called. prog="bbstored" # Check that configuration exists. [ -f /etc/box/$prog.conf ] || exit 0 start() { echo -n $"Starting $prog: " daemon $prog RETVAL=$? echo [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog return $RETVAL } stop() { echo -n $"Stopping $prog: " killproc $prog RETVAL=$? echo [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog return $RETVAL } rhstatus() { status $prog } restart() { stop start } reload() { echo -n $"Reloading $prog daemon configuration: " killproc $prog -HUP retval=$? echo return $RETVAL } case "$1" in start) start ;; stop) stop ;; restart) restart ;; status) rhstatus ;; condrestart) [ -f /var/lock/subsys/$prog ] && restart || : ;; *) echo $"Usage: $0 {start|stop|status|reload|restart|condrestart}" exit 1 esac exit $? --=-pO8nwjUYQoDUU79I1u1l-- From boxbackup at fluffy.co.uk Tue Sep 14 11:17:25 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 14 Sep 2004 11:17:25 +0100 Subject: [Box Backup] Working towards 0.08 Message-ID: <464372BA-0637-11D9-A58D-000A95AFF7F8@fluffy.co.uk> Hi It's been a while since I did a release, and there's the new storage system on the server which I should really get out there. I have been a bit preoccupied with buying a flat and then moving in to it to avoid impending homelessness, which wasn't a terribly fun episode, and has delayed things a bit. Some of you are running the 0.07PLUS3 version, and I haven't heard any complaints. We do need to check that it can restore files successfully though. So, could you 1) Try a compare command in bbackupquery to check that the latest versions are correct. (Nothing very much changed with regard to latest versions, so I would be very surprised if this didn't work -- but I would like to be reassured.) 2) Old versions can be stored as patches. Firstly, find a small file with multiple versions (use list with the -o option to list old versions) and see if a previous version can be fetched, and is correct. Then find a large one with multiple versions, which has had several small changes in the past. Use list with the -os options to display old versions, and the server storage requirements. If the sizes for old versions are much smaller than you'd expect when compared with the latest version, they're stored as patches. Try fetching a few, and checking their contents. I have been keeping an eye on my own installations, but they're all under OpenBSD and my preferred way of installing things, so I do need success reports from others. Is there anything else which should definitely go in? I'd prefer to do small changes, and then larger work after the release of 0.08. Thanks for your help and support. Ben From boxbackup at fluffy.co.uk Tue Sep 14 11:24:13 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 14 Sep 2004 11:24:13 +0100 Subject: [Box Backup] Cryptographic checking Message-ID: <39A57DF8-0638-11D9-A58D-000A95AFF7F8@fluffy.co.uk> I make certain claims for Box Backup with regard to it's use of cryptography. However, I am not an expert (as in, I do not do crypto every day for a living) and it has been historically shown that systems designed by non-professionals can have problems. That's not to say that there's dodgy crypto in Box Backup, quite the contrary. I have read a large number of books, and spoken to the professionals about things (and even recently designed a crypto system which has been checked and passed as fine by the professionals), so I do not think that confidence in the system is misplaced. But this is not good enough. Now that it is getting popular, I really think the crypto needs to be checked out by one of these professionals with a view to getting a statement from them for the web site. Fortunately, I know just the people to do this, but it's not particularly cheap to get this done. I would estimate between 1000 and 2000 UK pounds based on my previous experience of using them to check out things. I really can't afford this on my own. Is it worth me getting a quote from them and asking the user community for donations to get this funded? Thanks for any thoughts. Ben From boxbackup at fluffy.co.uk Tue Sep 14 12:48:27 2004 From: boxbackup at fluffy.co.uk (Peter Harrison) Date: Tue, 14 Sep 2004 23:48:27 +1200 Subject: [Box Backup] Cryptographic checking In-Reply-To: <39A57DF8-0638-11D9-A58D-000A95AFF7F8@fluffy.co.uk> References: <39A57DF8-0638-11D9-A58D-000A95AFF7F8@fluffy.co.uk> Message-ID: <200409142348.27517.peter.harrison@nothingbutnet.co.nz> On Tue, 14 Sep 2004 22:24, Ben Summers wrote: > But this is not good enough. Now that it is getting popular, I really > think the crypto needs to be checked out by one of these professionals > with a view to getting a statement from them for the web site. Peter Gutmann of the University of Auckland is an internationally renoun encryption expert - and has written crypto libraries. I don't think you could find anyone better to examine and endorse the application. I'm not sure what kind of money he would want though. I would have to talk with him about it. > Fortunately, I know just the people to do this, but it's not > particularly cheap to get this done. I would estimate between 1000 and > 2000 UK pounds based on my previous experience of using them to check > out things. I really can't afford this on my own. We would consider funding this effort. Right now I have coders working on some new functionality that we will be donating to the box backup project. This includes a web based registration system that sets up accounts via web interface, and a web (apache) based system for restoring files. > Is it worth me getting a quote from them and asking the user community > for donations to get this funded? I would like to know who would be doing the work personally - as they must have a pretty high profile. I think that it will probably cost more and take longer than you expect. We probably also need to do some prepatory work in documenting the protocols and procedures that the application uses. Trying to work out these things from code generally isn't where these guys start. Peter Gutmann is one of the foremost experts on Encryption. I also know Bill Raike personally. Bill Raike invented RPK public key encryption, and started the company called SecureMedia. Bill might be an option for certification of the software, although he is not as well known as Peter. There is also PGP inventor Phil Zimmerman, although I doubt he has much to do anymore with this kind of work. There are of course many other recognised experts - but I would strongly suggest we find someone who has some authority. Actually I think the first stage should be the documentation of the protocol used by Box Backup, and the submission of that protocol to a journal that concerns itself with Encryption. This means that experts in the field will have the opportunity to critique it without examination of the code. We should then seek to have commercial interests pay for a formal examination of both the protocol and the implementation - which must be considered separate things. We need to develop units tests to show that the box backup behaves correctly, and that it cannot be comprimised by buffer overruns and other subversion. I know that my company is willing to provide some financing to make this happen. We are not big though - I don't have hundreds of thousands of dollars to throw at this project, but its not nil either :) Once the protocol has been examined the protocol itself should be submitted to become a RFC, and perhaps to other standards bodies to become a formal standard in remote backup. In this way we are being open, and perhaps other companies will adopt our protocols, and thus we will have a larger effect than just writing the software. Regards, Peter From boxbackup at fluffy.co.uk Tue Sep 14 13:01:53 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 14 Sep 2004 13:01:53 +0100 Subject: [Box Backup] Cryptographic checking In-Reply-To: <200409142348.27517.peter.harrison@nothingbutnet.co.nz> References: <39A57DF8-0638-11D9-A58D-000A95AFF7F8@fluffy.co.uk> <200409142348.27517.peter.harrison@nothingbutnet.co.nz> Message-ID: On 14 Sep 2004, at 12:48, Peter Harrison wrote: > On Tue, 14 Sep 2004 22:24, Ben Summers wrote: > >> But this is not good enough. Now that it is getting popular, I really >> think the crypto needs to be checked out by one of these professionals >> with a view to getting a statement from them for the web site. > > Peter Gutmann of the University of Auckland is an internationally > renoun > encryption expert - and has written crypto libraries. I don't think > you could > find anyone better to examine and endorse the application. I'm not > sure what > kind of money he would want though. I would have to talk with him > about it. That would be appreciated. > >> Fortunately, I know just the people to do this, but it's not >> particularly cheap to get this done. I would estimate between 1000 and >> 2000 UK pounds based on my previous experience of using them to check >> out things. I really can't afford this on my own. > > We would consider funding this effort. Right now I have coders working > on some > new functionality that we will be donating to the box backup project. > This > includes a web based registration system that sets up accounts via web > interface, and a web (apache) based system for restoring files. I'm looking forward to seeing these! > >> Is it worth me getting a quote from them and asking the user community >> for donations to get this funded? > > I would like to know who would be doing the work personally - as they > must > have a pretty high profile. I have worked with Prof Fred Piper of the ISG at Royal Holloyway, University of London. He was also my head of department when I was studying there. Put his name into Google and see what happens! > I think that it will probably cost more and take > longer than you expect. We probably also need to do some prepatory > work in > documenting the protocols and procedures that the application uses. > Trying to > work out these things from code generally isn't where these guys start. If you look in the notes/ directory of the distribution, there is the beginnings of the required documentation. This would obviously be expanded a bit before commissioning the work. > > Peter Gutmann is one of the foremost experts on Encryption. I also > know Bill > Raike personally. Bill Raike invented RPK public key encryption, and > started > the company called SecureMedia. Bill might be an option for > certification of > the software, although he is not as well known as Peter. > > There is also PGP inventor Phil Zimmerman, although I doubt he has > much to do > anymore with this kind of work. There are of course many other > recognised > experts - but I would strongly suggest we find someone who has some > authority. > > Actually I think the first stage should be the documentation of the > protocol > used by Box Backup, and the submission of that protocol to a journal > that > concerns itself with Encryption. The protocol itself is merely TLS implemented using the standard OpenSSL library. What I'm more concerned about is the static storage of the data, something which couldn't be done using a standard method. > This means that experts in the field will > have the opportunity to critique it without examination of the code. > > We should then seek to have commercial interests pay for a formal > examination > of both the protocol and the implementation - which must be considered > separate things. We need to develop units tests to show that the box > backup > behaves correctly, and that it cannot be comprimised by buffer > overruns and > other subversion. > > I know that my company is willing to provide some financing to make > this > happen. We are not big though - I don't have hundreds of thousands of > dollars > to throw at this project, but its not nil either :) Thanks! > > Once the protocol has been examined the protocol itself should be > submitted to > become a RFC, and perhaps to other standards bodies to become a formal > standard in remote backup. In this way we are being open, and perhaps > other > companies will adopt our protocols, and thus we will have a larger > effect > than just writing the software. That would be a nice aim, although I'm not sure there is quite enough interest in the project yet. Perhaps if I get a couple more expressions of interest in this idea, I will have a word with Fred. Ben From boxbackup at fluffy.co.uk Tue Sep 14 13:24:44 2004 From: boxbackup at fluffy.co.uk (Joris) Date: Tue, 14 Sep 2004 14:24:44 +0200 (CEST) Subject: [Box Backup] Working towards 0.08 In-Reply-To: <464372BA-0637-11D9-A58D-000A95AFF7F8@fluffy.co.uk> References: <464372BA-0637-11D9-A58D-000A95AFF7F8@fluffy.co.uk> Message-ID: <2277.10.97.1.4.1095164684.squirrel@10.97.1.4> > Is there anything else which should definitely go in? I'd prefer to do > small changes, and then larger work after the release of 0.08. At some point, you should be able to freeze the storage method and not touch it again 'till the next major version. Same counts for the protocol. I still haven't looked into it very much, since I was occupied with very pressing matters, and I'm not a crypto specialist. So some of these suggestions will sound rather stupid. I touched them lightly in a previous mail also. 1) Make the disk storage atomic. A file/volume is fully changed or isn't changed at all. At any point in time I should be able to pull the power plug of the server and not have the risk of losing data by having half-changed data. Implementable in filesystem level by copying every edited file to a temp version, then issuing a move command once everything is complete. 2) Implement the backup pool system, where all backups made by clients are passed on to other servers. Preferably in a way only one version exists across the servers (see point one). 3) I don't know what your point of view is on independant development nor what your financial situation is alike, but a tip jar on the site meight fill rapidly... -- Greetings, Joris From boxbackup at fluffy.co.uk Tue Sep 14 13:41:32 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 14 Sep 2004 13:41:32 +0100 Subject: [Box Backup] Working towards 0.08 In-Reply-To: <2277.10.97.1.4.1095164684.squirrel@10.97.1.4> References: <464372BA-0637-11D9-A58D-000A95AFF7F8@fluffy.co.uk> <2277.10.97.1.4.1095164684.squirrel@10.97.1.4> Message-ID: <6825BCC2-064B-11D9-A58D-000A95AFF7F8@fluffy.co.uk> On 14 Sep 2004, at 13:24, Joris wrote: > >> Is there anything else which should definitely go in? I'd prefer to do >> small changes, and then larger work after the release of 0.08. > > At some point, you should be able to freeze the storage method and not > touch it again 'till the next major version. Same counts for the > protocol. This will happen when I'm happy enough to do a version 1.00. Right now there's not much that could change resulting in incompatibly. All formats are extensible in a backwards compatible manner to some degree. The only thing which might change things is supporting sparse files and multiple streams nicely -- these might need to change things to do it properly. (However, it can support two or more formats in parallel, so your old data stores don't get wiped out. It does this at the moment.) > I still haven't looked into it very much, since I was occupied with > very > pressing matters, and I'm not a crypto specialist. So some of these > suggestions will sound rather stupid. I touched them lightly in a > previous > mail also. > > > 1) Make the disk storage atomic. A file/volume is fully changed or > isn't > changed at all. At any point in time I should be able to pull the power > plug of the server and not have the risk of losing data by having > half-changed data. Implementable in filesystem level by copying every > edited file to a temp version, then issuing a move command once > everything > is complete. It already does this. Writing of files is atomic. Worse errors can be recovered automatically by the server. And if really bad things happen, the check and fix program can get all recoverable data back for you. > > 2) Implement the backup pool system, where all backups made by clients > are > passed on to other servers. Preferably in a way only one version exists > across the servers (see point one). Yes, this is something which is designed but not implemented. > > 3) I don't know what your point of view is on independant development > nor > what your financial situation is alike, but a tip jar on the site > meight > fill rapidly... Perhaps. I was just trying to get some indication of whether it would reach the required total -- I wouldn't want to ask if there was no chance of using tips for the stated purpose. Ben From boxbackup at fluffy.co.uk Tue Sep 14 16:08:10 2004 From: boxbackup at fluffy.co.uk (Pascal Lalonde) Date: Tue, 14 Sep 2004 11:08:10 -0400 Subject: [Box Backup] BoxBackup as a Windows Service (Cygwin) Message-ID: <1095174490.16504.61.camel@doublebarrel.doom.und> Did anyone have success at starting BoxBackup as a service under Windows (Cygwin)? I do a: cygrunsrv -I boxbackup -p /usr/local/bin/bbackup.exe Then: cygrunsrv -S boxbackup I get the error: cygrunsrv: Error starting a service: StartService: Win32 error 1058: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. I noticed that when installing sshd as a service via ssh-host-config, the script passes -D to sshd, which tells sshd not to fork in the background. Maybe boxbackup should have a similar option? Although I'm not sure if that is the problem. Ideas? Thanks, Pascal Lalonde From boxbackup at fluffy.co.uk Tue Sep 14 17:15:22 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 14 Sep 2004 17:15:22 +0100 Subject: [Box Backup] BoxBackup as a Windows Service (Cygwin) In-Reply-To: <1095174490.16504.61.camel@doublebarrel.doom.und> References: <1095174490.16504.61.camel@doublebarrel.doom.und> Message-ID: <4736B67A-0669-11D9-A58D-000A95AFF7F8@fluffy.co.uk> On 14 Sep 2004, at 16:08, Pascal Lalonde wrote: > Did anyone have success at starting BoxBackup as a service under > Windows > (Cygwin)? > > I do a: > cygrunsrv -I boxbackup -p /usr/local/bin/bbackup.exe > > Then: > cygrunsrv -S boxbackup > > I get the error: > cygrunsrv: Error starting a service: StartService: Win32 error 1058: > The service cannot be started, either because it is disabled or because > it has no enabled devices associated with it. > > I noticed that when installing sshd as a service via ssh-host-config, > the script passes -D to sshd, which tells sshd not to fork in the > background. Maybe boxbackup should have a similar option? Although I'm > not sure if that is the problem. > > Ideas? To avoid it forking, do bbackupd -c /path/to/config/file SINGLEPROCESS You must have an absolute reference to the config file, and everything be in that order. Hopefully someone who's managed this will give a better answer. I've never done this. Ben From boxbackup at fluffy.co.uk Tue Sep 14 22:50:20 2004 From: boxbackup at fluffy.co.uk (brett) Date: Tue, 14 Sep 2004 14:50:20 -0700 Subject: [Box Backup] Re: boxbackup digest, Vol 1 #132 - 6 msgs In-Reply-To: <20040914110006.13757.25759.Mailman@love.warhead.org.uk> References: <20040914110006.13757.25759.Mailman@love.warhead.org.uk> Message-ID: <128D018D-0698-11D9-AA66-000A95B9859A@librum.org> Ben, I've worked with a number of the labs in the US that have done FIPS 140-1 certification, and who are currently very active in the professional crypto world. I'll request that one of them go through it, and let you know the results. Can't beat free ;) b From boxbackup at fluffy.co.uk Wed Sep 15 07:46:25 2004 From: boxbackup at fluffy.co.uk (Per Thomsen) Date: Tue, 14 Sep 2004 23:46:25 -0700 Subject: [Box Backup] Checking that bbackupd's are backing up? Message-ID: <4147E541.7080503@reedtz.com> I would like to build in some centralized tracking of all my clients, to make sure that they are running, and backing things up. This would allow me to hunt down any clients that were shut down, or failed to restart after a reboot... Rather than distributing software to all clients, I would like to be able to do this on the bbstored server. I've been thinking along the lines of log-parsing, but then I looked at the output of 'bbstoreaccounts info'. The first 10 digits of the 'Client Store Marker' looks remarkably like a time(2) return value. Would it be safe to assume that the time stamp denotes the last time a file was uploaded from the client? If so, this would be a good (IMO) way to check that a client has backed *something* up in the last x seconds, and warning me if not. If people have other (and better) ways of doing this, I'd love to hear about them. Thanks, Per -- Per Reedtz Thomsen | The Reedtz Corporation | F: 209 883 4119 V: 209 883 4102 | pthomsen at reedtz.com | C: 415 425 4025 GPG ID: 1209784F | Yahoo! Chat: pthomsen | AIM: pthomsen From boxbackup at fluffy.co.uk Wed Sep 15 10:02:34 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 15 Sep 2004 10:02:34 +0100 Subject: [Box Backup] Checking that bbackupd's are backing up? In-Reply-To: <4147E541.7080503@reedtz.com> References: <4147E541.7080503@reedtz.com> Message-ID: On 15 Sep 2004, at 07:46, Per Thomsen wrote: > I would like to build in some centralized tracking of all my clients, > to make sure that they are running, and backing things up. This would > allow me to hunt down any clients that were shut down, or failed to > restart after a reboot... > > Rather than distributing software to all clients, I would like to be > able to do this on the bbstored server. I've been thinking along the > lines of log-parsing, but then I looked at the output of > 'bbstoreaccounts info'. The first 10 digits of the 'Client Store > Marker' looks remarkably like a time(2) return value. Would it be safe > to assume that the time stamp denotes the last time a file was > uploaded from the client? Unfortunately, no. This is quite apart from the fact that if bbackupd doesn't see anything to upload, it won't even contact the server. > > If so, this would be a good (IMO) way to check that a client has > backed *something* up in the last x seconds, and warning me if not. You're not the first who has asked about this feature! > > If people have other (and better) ways of doing this, I'd love to hear > about them. This software grew out of a system I was building. Each client would be running the backup software, as well as a generic error management daemon. This would report back errors to a central system for resolution. The servers were designed to just sit there and do their stuff, and be fairly passive about things since most of the difficult bits would be done by a generic system. This is not how it's actually being used in practice -- everyone wants it to be nice and self-contained, which is a perfectly reasonable thing to want. So I think that the server side management should be overhauled in the next version. Issues I can think of are * Use names instead of account numbers * Monitoring of client activity and error status * Additional reporting of space used on the server * Perhaps an overhaul of the account database, which was always a bit of a temporary measure until I got it reading the information from a proper database. * Feeding the status into another system, perhaps piping status into a script? I think the certificates are OK, unless anyone disagrees. I would like to get it right first time. Perhaps I could ask someone to start a specification document, which we can all revise until we're happy with it? I'd like it to be fairly detailed, specifying exactly how the user will interact with the system, and then I'll get it implemented in 0.09. Ben From boxbackup at fluffy.co.uk Wed Sep 15 09:43:07 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 15 Sep 2004 09:43:07 +0100 Subject: [Box Backup] Re: boxbackup digest, Vol 1 #132 - 6 msgs In-Reply-To: <128D018D-0698-11D9-AA66-000A95B9859A@librum.org> References: <20040914110006.13757.25759.Mailman@love.warhead.org.uk> <128D018D-0698-11D9-AA66-000A95B9859A@librum.org> Message-ID: <44319C88-06F3-11D9-BBE6-000A95AFF7F8@fluffy.co.uk> On 14 Sep 2004, at 22:50, brett wrote: > Ben, > > I've worked with a number of the labs in the US that have done FIPS > 140-1 certification, and who are currently very active in the > professional crypto world. I'll request that one of them go through > it, and let you know the results. > > Can't beat free ;) Absolutely! Thanks. Ben From boxbackup at fluffy.co.uk Wed Sep 15 16:40:35 2004 From: boxbackup at fluffy.co.uk (Per Thomsen) Date: Wed, 15 Sep 2004 08:40:35 -0700 Subject: [Box Backup] Checking that bbackupd's are backing up? In-Reply-To: References: <4147E541.7080503@reedtz.com> Message-ID: <41486273.6070508@reedtz.com> On 9/15/04 2:02 AM, Ben Summers wrote: > > On 15 Sep 2004, at 07:46, Per Thomsen wrote: > >> I would like to build in some centralized tracking of all my clients, >> to make sure that they are running, and backing things up. This would >> allow me to hunt down any clients that were shut down, or failed to >> restart after a reboot... >> >> Rather than distributing software to all clients, I would like to be >> able to do this on the bbstored server. I've been thinking along the >> lines of log-parsing, but then I looked at the output of >> 'bbstoreaccounts info'. The first 10 digits of the 'Client Store >> Marker' looks remarkably like a time(2) return value. Would it be >> safe to assume that the time stamp denotes the last time a file was >> uploaded from the client? > > > Unfortunately, no. > > This is quite apart from the fact that if bbackupd doesn't see > anything to upload, it won't even contact the server. Right. That's something that I'd have to deal with anyway. I am also considering writing a Nagios (www.nagios.org) service to check the 'aliveness' of bbackupd on the client, and bbstored on the server. This would let me know if the bbackupd daemon was not running. That might be a better way to go, long term. Of course, that only helps if you're already running Nagios (not a system I'd set up to check for just one or two things). >> If so, this would be a good (IMO) way to check that a client has >> backed *something* up in the last x seconds, and warning me if not. > > > You're not the first who has asked about this feature! > >> >> If people have other (and better) ways of doing this, I'd love to >> hear about them. > > > This software grew out of a system I was building. Each client would > be running the backup software, as well as a generic error management > daemon. This would report back errors to a central system for > resolution. The servers were designed to just sit there and do their > stuff, and be fairly passive about things since most of the difficult > bits would be done by a generic system. > > This is not how it's actually being used in practice -- everyone wants > it to be nice and self-contained, which is a perfectly reasonable > thing to want. > > So I think that the server side management should be overhauled in the > next version. Issues I can think of are > > * Use names instead of account numbers > * Monitoring of client activity and error status > * Additional reporting of space used on the server > * Perhaps an overhaul of the account database, which was always a bit > of a temporary measure until I got it reading the information from a > proper database. > * Feeding the status into another system, perhaps piping status into a > script? > > I think the certificates are OK, unless anyone disagrees. > > I would like to get it right first time. Perhaps I could ask someone > to start a specification document, which we can all revise until we're > happy with it? I'd like it to be fairly detailed, specifying exactly > how the user will interact with the system, and then I'll get it > implemented in 0.09. My time is limited, but I'd love to get the ball rolling. I should be able to pull something together by early next week. Any preferred formats? Thanks, Per -- Per Reedtz Thomsen | The Reedtz Corporation | F: 209 883 4119 V: 209 883 4102 | pthomsen at reedtz.com | C: 415 425 4025 GPG ID: 1209784F | Yahoo! Chat: pthomsen | AIM: pthomsen From boxbackup at fluffy.co.uk Wed Sep 15 16:48:45 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 15 Sep 2004 16:48:45 +0100 Subject: [Box Backup] Checking that bbackupd's are backing up? In-Reply-To: <41486273.6070508@reedtz.com> References: <4147E541.7080503@reedtz.com> <41486273.6070508@reedtz.com> Message-ID: On 15 Sep 2004, at 16:40, Per Thomsen wrote: > On 9/15/04 2:02 AM, Ben Summers wrote: >> >> This is quite apart from the fact that if bbackupd doesn't see >> anything to upload, it won't even contact the server. > > Right. That's something that I'd have to deal with anyway. I am also > considering writing a Nagios (www.nagios.org) service to check the > 'aliveness' of bbackupd on the client, and bbstored on the server. > This would let me know if the bbackupd daemon was not running. That > might be a better way to go, long term. Of course, that only helps if > you're already running Nagios (not a system I'd set up to check for > just one or two things). Nagios looks interesting. How would the plugin work? Attempt to make a connection to the server and check that the client has scanned the files recently (timestamp of /var/bbackupd/last_sync_finish), or just read the PID file and see if that process is active? [snip] >> >> I would like to get it right first time. Perhaps I could ask someone >> to start a specification document, which we can all revise until >> we're happy with it? I'd like it to be fairly detailed, specifying >> exactly how the user will interact with the system, and then I'll get >> it implemented in 0.09. > > My time is limited, but I'd love to get the ball rolling. I should be > able to pull something together by early next week. Any preferred > formats? That would be great. Plain text would be a good format, nice and easy to read and discuss on mailing lists. Thanks, Ben From boxbackup at fluffy.co.uk Wed Sep 15 17:24:19 2004 From: boxbackup at fluffy.co.uk (Per Thomsen) Date: Wed, 15 Sep 2004 09:24:19 -0700 Subject: [Box Backup] Checking that bbackupd's are backing up? In-Reply-To: References: <4147E541.7080503@reedtz.com> <41486273.6070508@reedtz.com> Message-ID: <41486CB3.1040203@reedtz.com> On 9/15/04 8:48 AM, Ben Summers wrote: [snip] > Nagios looks interesting. How would the plugin work? Attempt to make a > connection to the server and check that the client has scanned the > files recently (timestamp of /var/bbackupd/last_sync_finish), or just > read the PID file and see if that process is active? Yes, I was thinking that the plugin would do a bbackupquery and parse the output, to make sure that the process was functioning, and then check the timestamp as well, to make sure nothing is hung. > [snip] > >> >> My time is limited, but I'd love to get the ball rolling. I should be >> able to pull something together by early next week. Any preferred >> formats? > > > That would be great. Plain text would be a good format, nice and easy > to read and discuss on mailing lists. OK. I'll post a first draft on the list sometime next week. Thanks, Per -- Per Reedtz Thomsen | The Reedtz Corporation | F: 209 883 4119 V: 209 883 4102 | pthomsen at reedtz.com | C: 415 425 4025 GPG ID: 1209784F | Yahoo! Chat: pthomsen | AIM: pthomsen From boxbackup at fluffy.co.uk Wed Sep 15 17:33:41 2004 From: boxbackup at fluffy.co.uk (Per Thomsen) Date: Wed, 15 Sep 2004 09:33:41 -0700 Subject: [Box Backup] cygwin experts? Message-ID: <41486EE5.30807@reedtz.com> I'll be installing the cygwin build on a set of Windows boxes, and I wanted to see if anyone on the list knew how to ensure that bbackupd starts after a reboot. I'm no Windows (nor cygwin) expert, so any help would be appreciated. I have the build completed, and I can start bbackupd by hand, but I would obviously like to have it start, when Windows starts (not just when someone logs in). I tried the 'cygrunsrv' executable, and the service (apparently) installs correctly (using the '-I' option), but I get a Windows 1062 error, when I try to start it (the '-S' option). The error text for error 1062 is : The service has not been started. ERROR_SERVICE_NOT_ACTIVE Very helpful... ;-) Any pointers? Thanks, Per -- Per Reedtz Thomsen | The Reedtz Corporation | F: 209 883 4119 V: 209 883 4102 | pthomsen at reedtz.com | C: 415 425 4025 GPG ID: 1209784F | Yahoo! Chat: pthomsen | AIM: pthomsen From boxbackup at fluffy.co.uk Wed Sep 15 17:53:47 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 15 Sep 2004 17:53:47 +0100 Subject: [Box Backup] Checking that bbackupd's are backing up? In-Reply-To: <41486CB3.1040203@reedtz.com> References: <4147E541.7080503@reedtz.com> <41486273.6070508@reedtz.com> <41486CB3.1040203@reedtz.com> Message-ID: On 15 Sep 2004, at 17:24, Per Thomsen wrote: > On 9/15/04 8:48 AM, Ben Summers wrote: > > [snip] > >> Nagios looks interesting. How would the plugin work? Attempt to make >> a connection to the server and check that the client has scanned the >> files recently (timestamp of /var/bbackupd/last_sync_finish), or just >> read the PID file and see if that process is active? > > Yes, I was thinking that the plugin would do a bbackupquery and parse > the output, to make sure that the process was functioning, and then > check the timestamp as well, to make sure nothing is hung. bbackupquery tests the server, not the client, as it connects directly to the server. > >> [snip] >> >>> >>> My time is limited, but I'd love to get the ball rolling. I should >>> be able to pull something together by early next week. Any preferred >>> formats? >> >> >> That would be great. Plain text would be a good format, nice and easy >> to read and discuss on mailing lists. > > OK. I'll post a first draft on the list sometime next week. I look forward to it! Ben From boxbackup at fluffy.co.uk Wed Sep 15 18:01:15 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 15 Sep 2004 18:01:15 +0100 Subject: [Box Backup] cygwin experts? In-Reply-To: <41486EE5.30807@reedtz.com> References: <41486EE5.30807@reedtz.com> Message-ID: On 15 Sep 2004, at 17:33, Per Thomsen wrote: > I'll be installing the cygwin build on a set of Windows boxes, and I > wanted to see if anyone on the list knew how to ensure that bbackupd > starts after a reboot. I'm no Windows (nor cygwin) expert, so any help > would be appreciated. > > I have the build completed, and I can start bbackupd by hand, but I > would obviously like to have it start, when Windows starts (not just > when someone logs in). I tried the 'cygrunsrv' executable, and the > service (apparently) installs correctly (using the '-I' option), but I > get a Windows 1062 error, when I try to start it (the '-S' option). > The error text for error 1062 is : > > The service has not been started. ERROR_SERVICE_NOT_ACTIVE > > Very helpful... ;-) > > Any pointers? If you don't mind it being started when the user logs in, then you can put it in the "Run" part of the registry, I suppose. Usual disclaimer: I've never got it running. Ben From boxbackup at fluffy.co.uk Wed Sep 15 18:07:59 2004 From: boxbackup at fluffy.co.uk (Pascal Lalonde) Date: Wed, 15 Sep 2004 13:07:59 -0400 Subject: [Box Backup] BoxBackup as a Windows Service (Cygwin) In-Reply-To: <4736B67A-0669-11D9-A58D-000A95AFF7F8@fluffy.co.uk> References: <1095174490.16504.61.camel@doublebarrel.doom.und> <4736B67A-0669-11D9-A58D-000A95AFF7F8@fluffy.co.uk> Message-ID: <1095268079.21362.49.camel@doublebarrel.doom.und> On Tue, 2004-09-14 at 12:15, Ben Summers wrote: > On 14 Sep 2004, at 16:08, Pascal Lalonde wrote: > > > Did anyone have success at starting BoxBackup as a service under > > Windows > > (Cygwin)? > > > > I do a: > > cygrunsrv -I boxbackup -p /usr/local/bin/bbackup.exe > > > > Then: > > cygrunsrv -S boxbackup > > > > I get the error: > > cygrunsrv: Error starting a service: StartService: Win32 error 1058: > > The service cannot be started, either because it is disabled or because > > it has no enabled devices associated with it. > > > > I noticed that when installing sshd as a service via ssh-host-config, > > the script passes -D to sshd, which tells sshd not to fork in the > > background. Maybe boxbackup should have a similar option? Although I'm > > not sure if that is the problem. > > > > Ideas? > > To avoid it forking, do > > bbackupd -c /path/to/config/file SINGLEPROCESS > > You must have an absolute reference to the config file, and everything > be in that order. I don't know if I'm doing something wrong, but: /usr/local/bin/bbackupd -c /etc/box/bbackupd.conf SINGLEPROCESS starts a forked bbackupd, just as "bbackupd" would. I've tried this on OpenBSD 3.5, with Boxbackup 0.07PLUS3. Pascal From boxbackup at fluffy.co.uk Wed Sep 15 18:17:06 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 15 Sep 2004 18:17:06 +0100 Subject: [Box Backup] BoxBackup as a Windows Service (Cygwin) In-Reply-To: <1095268079.21362.49.camel@doublebarrel.doom.und> References: <1095174490.16504.61.camel@doublebarrel.doom.und> <4736B67A-0669-11D9-A58D-000A95AFF7F8@fluffy.co.uk> <1095268079.21362.49.camel@doublebarrel.doom.und> Message-ID: <11D58968-073B-11D9-BBE6-000A95AFF7F8@fluffy.co.uk> On 15 Sep 2004, at 18:07, Pascal Lalonde wrote: > On Tue, 2004-09-14 at 12:15, Ben Summers wrote: >> On 14 Sep 2004, at 16:08, Pascal Lalonde wrote: >> >>> Did anyone have success at starting BoxBackup as a service under >>> Windows >>> (Cygwin)? >>> >>> I do a: >>> cygrunsrv -I boxbackup -p /usr/local/bin/bbackup.exe >>> >>> Then: >>> cygrunsrv -S boxbackup >>> >>> I get the error: >>> cygrunsrv: Error starting a service: StartService: Win32 error 1058: >>> The service cannot be started, either because it is disabled or >>> because >>> it has no enabled devices associated with it. >>> >>> I noticed that when installing sshd as a service via ssh-host-config, >>> the script passes -D to sshd, which tells sshd not to fork in the >>> background. Maybe boxbackup should have a similar option? Although >>> I'm >>> not sure if that is the problem. >>> >>> Ideas? >> >> To avoid it forking, do >> >> bbackupd -c /path/to/config/file SINGLEPROCESS >> >> You must have an absolute reference to the config file, and everything >> be in that order. > I don't know if I'm doing something wrong, but: > /usr/local/bin/bbackupd -c /etc/box/bbackupd.conf SINGLEPROCESS > > starts a forked bbackupd, just as "bbackupd" would. > I've tried this on OpenBSD 3.5, with Boxbackup 0.07PLUS3. Ah, you're right. That should be /usr/local/bin/bbackupd /etc/box/bbackupd.conf SINGLEPROCESS This is only really a debug thing, don't consider this actual documentation for it or anything! Ben From boxbackup at fluffy.co.uk Wed Sep 15 20:39:38 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Wed, 15 Sep 2004 15:39:38 -0400 (EDT) Subject: [Box Backup] Checking that bbackupd's are backing up? In-Reply-To: <4147E541.7080503@reedtz.com> References: <4147E541.7080503@reedtz.com> Message-ID: I use a script to watch the logs, it's written in Perl, you are welcome to it. Rick On Tue, 14 Sep 2004, Per Thomsen wrote: > I would like to build in some centralized tracking of all my clients, to > make sure that they are running, and backing things up. This would allow > me to hunt down any clients that were shut down, or failed to restart > after a reboot... > > Rather than distributing software to all clients, I would like to be > able to do this on the bbstored server. I've been thinking along the > lines of log-parsing, but then I looked at the output of > 'bbstoreaccounts info'. The first 10 digits of the 'Client Store Marker' > looks remarkably like a time(2) return value. Would it be safe to assume > that the time stamp denotes the last time a file was uploaded from the > client? > > If so, this would be a good (IMO) way to check that a client has backed > *something* up in the last x seconds, and warning me if not. > > If people have other (and better) ways of doing this, I'd love to hear > about them. > > Thanks, > Per > > -- > Per Reedtz Thomsen | The Reedtz Corporation | F: 209 883 4119 > V: 209 883 4102 | pthomsen at reedtz.com | C: 415 425 4025 > GPG ID: 1209784F | Yahoo! Chat: pthomsen | AIM: pthomsen > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Wed Sep 15 20:44:19 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Wed, 15 Sep 2004 15:44:19 -0400 (EDT) Subject: [Box Backup] openssl: no In-Reply-To: <20040912085439.83264.qmail@web41511.mail.yahoo.com> References: <20040912085439.83264.qmail@web41511.mail.yahoo.com> Message-ID: I use Debian testing, but if you need help I'd be happy to try and give you a hand. Rick On Sun, 12 Sep 2004, Mario S. Mommer wrote: > > --- Imran wrote: > > Hmm, i've had issues with box backup finding the SSL on one of my > > servers. i think it was fedora core, or maybe redhat 8. anyway, it may > > be a similiar issue. I usually end up configuring boxbackup with this > > commandline: > > > > ./configure compile:-I/usr/local/ssl/include link:-L/usr/local/ssl/lib > > Changed it to reflect my current situation but to no avail. I really > wonder what is going on here, since in fact db is installed too. There > seems to be a few mismatches between libc version and the backported > openssl. Oh man. > > Has anyone in this list been successfull building boxbackup on debian > vintage^Wstable? > > > also check to make sure that (i don't know how it is on debian but on > > redhat...) the path for the ssl lib is in the /etc/ld.so.conf, i.e. > > /usr/local/ssl/lib, and that you run ldconfig. I always forget if that > > would > > matter for the configure script. > > Well, /etc/ld.so.conf does not exists. Hm. I ran ldconfig but nothing > changed. > > > anyway, try it and let us know... > > No good news, unfortunately :-| > > Regards, > Mario. > > > > __________________________________ > Do you Yahoo!? > Yahoo! Mail Address AutoComplete - You start. We finish. > http://promotions.yahoo.com/new_mail > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Wed Sep 15 23:43:29 2004 From: boxbackup at fluffy.co.uk (Adrian) Date: Wed, 15 Sep 2004 15:43:29 -0700 (MST) Subject: [Box Backup] Checking that bbackupd's are backing up? In-Reply-To: References: <4147E541.7080503@reedtz.com> Message-ID: <63432.199.64.0.252.1095288209.squirrel@199.64.0.252> Ben, Why not make bbackupd call home once every 24 hours? The actual time could be a config file option. If you specify 0, it turns this feature off. Actually, I already do this. I have cron touch a dummy file every day. That way bbackupd is forced to call home. (Since my clients are all used daily, I am not actually sure if this is working, but I set it up the last time that this question came up, and I have not had a miss except where there was an actual problem.) I also have "logwatch" set up to report the number of calls made by each client everyday. This has caught a lot of otherwise invisible problems (like the time my server's hard drive began to fail). Adrian > > On 15 Sep 2004, at 07:46, Per Thomsen wrote: > >> I would like to build in some centralized tracking of all my clients, to make sure that they are running, and backing things up. This would allow me to hunt down any clients that were shut down, or failed to restart after a reboot... >> Rather than distributing software to all clients, I would like to be able to do this on the bbstored server. I've been thinking along the lines of log-parsing, but then I looked at the output of >> 'bbstoreaccounts info'. The first 10 digits of the 'Client Store Marker' looks remarkably like a time(2) return value. Would it be safe to assume that the time stamp denotes the last time a file was >> uploaded from the client? > > Unfortunately, no. > > This is quite apart from the fact that if bbackupd doesn't see anything to upload, it won't even contact the server. > >> If so, this would be a good (IMO) way to check that a client has backed *something* up in the last x seconds, and warning me if not. > > You're not the first who has asked about this feature! > >> If people have other (and better) ways of doing this, I'd love to hear about them. > > This software grew out of a system I was building. Each client would be running the backup software, as well as a generic error management daemon. This would report back errors to a central system for > resolution. The servers were designed to just sit there and do their stuff, and be fairly passive about things since most of the difficult bits would be done by a generic system. > > This is not how it's actually being used in practice -- everyone wants it to be nice and self-contained, which is a perfectly reasonable thing to want. > > So I think that the server side management should be overhauled in the next version. Issues I can think of are > > * Use names instead of account numbers > * Monitoring of client activity and error status > * Additional reporting of space used on the server > * Perhaps an overhaul of the account database, which was always a bit of a temporary measure until I got it reading the information from a proper database. > * Feeding the status into another system, perhaps piping status into a script? > > I think the certificates are OK, unless anyone disagrees. > > I would like to get it right first time. Perhaps I could ask someone to start a specification document, which we can all revise until we're happy with it? I'd like it to be fairly detailed, specifying exactly how the user will interact with the system, and then I'll get it implemented in 0.09. > > Ben > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Thu Sep 16 00:14:36 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Thu, 16 Sep 2004 00:14:36 +0100 Subject: [Box Backup] Checking that bbackupd's are backing up? In-Reply-To: <63432.199.64.0.252.1095288209.squirrel@199.64.0.252> References: <4147E541.7080503@reedtz.com> <63432.199.64.0.252.1095288209.squirrel@199.64.0.252> Message-ID: <1095290076.3926.94.camel@avenin.ebourne.me.uk> On Wed, 2004-09-15 at 23:43, Adrian wrote: > I also have "logwatch" set up to report the number of calls made by each > client everyday. This has caught a lot of otherwise invisible problems > (like the time my server's hard drive began to fail). I'd be interested in the logwatch config file for this if you don't mind. Cheers, Martin. From boxbackup at fluffy.co.uk Thu Sep 16 04:36:59 2004 From: boxbackup at fluffy.co.uk (Adrian) Date: Wed, 15 Sep 2004 20:36:59 -0700 (MST) Subject: [Box Backup] Checking that bbackupd's are backing up? Message-ID: <2381.192.168.1.1.1095305819.squirrel@192.168.1.1> Martin, There are actually several files that have to be created/modified to set up logwatch, I suggest reading up on how to set up a service on your particular installation. If you need more guidance, let me know, and I will try to write up a little how-to. The file that does the actual work is a perl script something like below (I apologize in advance for the complete lack of comments). Adrian ################################################################# #!/usr/bin/perl ################################################################# while (defined($line = )) { if($line =~ /Login/){ @field=split(/ +/,$line); $field[8] =~ s/,//; $i=1; $flag=0; foreach $client (@clients) { if($client eq $field[8]) { $hits[$i]++; $flag=1; } $i++; } if ($flag==0) { push(@clients,$field[8]); $hits[$i]++; } } } $i=1; foreach $client (@clients) { print "Client ",$client," ", at hits[$i]," Times\n"; $i++; } exit(0); ################################################################# > On Wed, 2004-09-15 at 23:43, Adrian wrote: >> I also have "logwatch" set up to report the number of calls made by each client everyday. This has caught a lot of otherwise invisible problems (like the time my server's hard drive began to fail). > > I'd be interested in the logwatch config file for this if you don't mind. > > Cheers, > > Martin. > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Thu Sep 16 09:31:08 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 16 Sep 2004 09:31:08 +0100 Subject: [Box Backup] Checking that bbackupd's are backing up? In-Reply-To: <63432.199.64.0.252.1095288209.squirrel@199.64.0.252> References: <4147E541.7080503@reedtz.com> <63432.199.64.0.252.1095288209.squirrel@199.64.0.252> Message-ID: On 15 Sep 2004, at 23:43, Adrian wrote: > Ben, > > Why not make bbackupd call home once every 24 hours? The actual time > could be a config file option. If you specify 0, it turns this feature > off. Yes, that's the most likely solution to the problem. I was thinking more of an "always connect" option. But we'll see what people actually want when we get the spec done. > > Actually, I already do this. I have cron touch a dummy file every day. > That way bbackupd is forced to call home. (Since my clients are all > used > daily, I am not actually sure if this is working, but I set it up the > last > time that this question came up, and I have not had a miss except where > there was an actual problem.) Was the problem anything to do with Box Backup? > > I also have "logwatch" set up to report the number of calls made by > each > client everyday. This has caught a lot of otherwise invisible problems > (like the time my server's hard drive began to fail). That's certainly something you might want to know about. Ben From boxbackup at fluffy.co.uk Thu Sep 16 14:31:15 2004 From: boxbackup at fluffy.co.uk (Adrian) Date: Thu, 16 Sep 2004 06:31:15 -0700 (MST) Subject: [Box Backup] Checking that bbackupd's are backing up? In-Reply-To: References: <4147E541.7080503@reedtz.com> <63432.199.64.0.252.1095288209.squirrel@199.64.0.252> Message-ID: <4633.192.168.1.1.1095341475.squirrel@192.168.1.1> >> >> Actually, I already do this. I have cron touch a dummy file every day. >> That way bbackupd is forced to call home. (Since my clients are all >> used >> daily, I am not actually sure if this is working, but I set it up the >> last >> time that this question came up, and I have not had a miss except where >> there was an actual problem.) > > Was the problem anything to do with Box Backup? > Hard to say, I have not had any repeatable problems, but I have had bbackupd simply quit and I have had it go to 100% CPU usage and stay there? (This is mostly on Windows boxes running under cygwin.) Adrian From boxbackup at fluffy.co.uk Thu Sep 16 14:35:10 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 16 Sep 2004 14:35:10 +0100 Subject: [Box Backup] Checking that bbackupd's are backing up? In-Reply-To: <4633.192.168.1.1.1095341475.squirrel@192.168.1.1> References: <4147E541.7080503@reedtz.com> <63432.199.64.0.252.1095288209.squirrel@199.64.0.252> <4633.192.168.1.1.1095341475.squirrel@192.168.1.1> Message-ID: <3AE8008E-07E5-11D9-94E8-000A95AFF7F8@fluffy.co.uk> On 16 Sep 2004, at 14:31, Adrian wrote: > >>> >>> Actually, I already do this. I have cron touch a dummy file every >>> day. >>> That way bbackupd is forced to call home. (Since my clients are all >>> used >>> daily, I am not actually sure if this is working, but I set it up the >>> last >>> time that this question came up, and I have not had a miss except >>> where >>> there was an actual problem.) >> >> Was the problem anything to do with Box Backup? >> > > Hard to say, I have not had any repeatable problems, but I have had > bbackupd simply quit and I have had it go to 100% CPU usage and stay > there? (This is mostly on Windows boxes running under cygwin.) I'm guessing that's a cygwin issue. I've never seen that on a UNIX box. Let's hope we can get a Win32 client sometime soon! Ben From boxbackup at fluffy.co.uk Fri Sep 17 12:33:09 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 17 Sep 2004 12:33:09 +0100 Subject: [Box Backup] 0.07PLUS4 (almost 0.08) Message-ID: <59CF7A58-089D-11D9-9E39-000A95AFF7F8@fluffy.co.uk> New version for you all to try! This one will hopefully become 0.08 with only a few minor changes. http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.07PLUS4.tgz I am still hoping for some feedback from people using 0.07PLUS3 before I feel confident to release 0.08... http://lists.warhead.org.uk/pipermail/boxbackup/2004-September/ 000415.html Changes over 0.07PLUS3: * Minor bug fixes * Martin Ebourne's patches for 64 bit systems. * Martin Ebourne's RPM generation stuff for RedHat * Better detection of gcc on random German RedHat systems (why do Linux people make it so difficult to do the most basic things?) * Automatically clean up corrupt file tracking databases instead of refusing to do any backups. This has happened to a few people following a bad server crash. Please download this and test! Feedback very welcome. I shall be doing more building and running the tests on various platforms before release, but need more real world confirmations that it works. Thanks, Ben From boxbackup at fluffy.co.uk Mon Sep 20 14:39:10 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 20 Sep 2004 14:39:10 +0100 Subject: [Box Backup] Suggested change in behaviour Message-ID: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> Imagine this scenario. You have a single partition on a hard drive, mounted as /home. This drive fails, the server reboots, but fails to pass fsck checks as much is unreadable. You modify /etc/fstab so it isn't mounted, and reboot. The machine comes up. Unfortunately, you forgot to disable bbackupd running on boot, which you've got running in lazy mode. It runs, does it's initial scan, and marks everything in this partition as deleted. This is unlikely to help the situation much, and incidentally, has just happened to me. (Fortunately, I have "undelete" technology hidden in the server, and have just written a bit of code to allow access to it via bbackupquery, so it's not a calamity - even if it has also undeleted files which were deleted by the users before the partition vanished.) I propose a slight change in behaviour which may go some way towards resolving this problem, but I wonder if it will cause any other issues. * If bbackupd finds that the root of a Location (as specified in bbackupd.conf) contains no files or directories, it - will log a message "Backup location /x is empty, not changing store" at level LOG_ERR - will not modify the relevant location on the server Can anyone see any situation where this might cause problems? This will only be triggered if there's absolutely nothing to back up in a location. Thanks, Ben From boxbackup at fluffy.co.uk Mon Sep 20 15:02:15 2004 From: boxbackup at fluffy.co.uk (Richard Eigenmann) Date: Mon, 20 Sep 2004 16:02:15 +0200 Subject: [Box Backup] Suggested change in behaviour In-Reply-To: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> References: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> Message-ID: <1095688934.23708.21.camel@nurata.richinet.dyndns.org> Basically you want to stop the delete process when the directory at a mount point is not mounted and there was previously backed up data for that location. You recommend that each mount point should be backed up separately. However you might find situations where people have mount points further down the tree even if this is not such a good idea. I wonder if there might not be a better way to detect an unmounted directory? Use of fstab? But that doesn't help in your scenario. Perhaps have a list of mountpoints in the .conf file? You could also have boxbackup run in "safe mode" by default (as you describe) and use a --force-delete switch to ensure that wiped root directories get marked as deleted in the archive. The warning message could read something like "Directory not mounted or empty. Use command xxx to delete it's contents from the backup archive." Such a switch/command could for instance be useful for new users who are setting up boxbackup and back up a small directory first and then change the location of their root to back up everything. This would be something like forcing the garbage collection. But that should perhaps be called --force-gc and you have already made suggestions how this can be triggered. Regards, Richard On Mon, 2004-09-20 at 15:39, Ben Summers wrote: > > Imagine this scenario. You have a single partition on a hard drive, > mounted as /home. This drive fails, the server reboots, but fails to > pass fsck checks as much is unreadable. You modify /etc/fstab so it > isn't mounted, and reboot. The machine comes up. > > Unfortunately, you forgot to disable bbackupd running on boot, which > you've got running in lazy mode. It runs, does it's initial scan, and > marks everything in this partition as deleted. > > This is unlikely to help the situation much, and incidentally, has just > happened to me. (Fortunately, I have "undelete" technology hidden in > the server, and have just written a bit of code to allow access to it > via bbackupquery, so it's not a calamity - even if it has also > undeleted files which were deleted by the users before the partition > vanished.) > > I propose a slight change in behaviour which may go some way towards > resolving this problem, but I wonder if it will cause any other issues. > > * If bbackupd finds that the root of a Location (as specified in > bbackupd.conf) contains no files or directories, it > > - will log a message "Backup location /x is empty, not changing > store" at level LOG_ERR > > - will not modify the relevant location on the server > > Can anyone see any situation where this might cause problems? This will > only be triggered if there's absolutely nothing to back up in a > location. > > Thanks, > > Ben > > > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup -- Richard Eigenmann From boxbackup at fluffy.co.uk Mon Sep 20 15:09:58 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 20 Sep 2004 15:09:58 +0100 Subject: [Box Backup] Suggested change in behaviour In-Reply-To: <1095688934.23708.21.camel@nurata.richinet.dyndns.org> References: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> <1095688934.23708.21.camel@nurata.richinet.dyndns.org> Message-ID: On 20 Sep 2004, at 15:02, Richard Eigenmann wrote: > Basically you want to stop the delete process when the directory at a > mount point is not mounted and there was previously backed up data for > that location. Not precisely. I want to avoid marking files as deleted when there's absolutely nothing in that location -- which I believe will almost always be an indication that a Very Bad Thing has happened, and backups will be required ASAP. > > You recommend that each mount point should be backed up separately. I require this! > However you might find situations where people have mount points > further > down the tree even if this is not such a good idea. I must write the code which stops people from doing that. > > I wonder if there might not be a better way to detect an unmounted > directory? On BSDs, it should be trivially easy. > Use of fstab? But that doesn't help in your scenario. Perhaps > have a list of mountpoints in the .conf file? But I want to catch an obviously bad situation, not detect a specific bad situation. > > You could also have boxbackup run in "safe mode" by default (as you > describe) and use a --force-delete switch to ensure that wiped root > directories get marked as deleted in the archive. > > The warning message could read something like "Directory not mounted or > empty. Use command xxx to delete it's contents from the backup > archive." > > Such a switch/command could for instance be useful for new users who > are > setting up boxbackup and back up a small directory first and then > change > the location of their root to back up everything. This would be > something like forcing the garbage collection. But that should perhaps > be called --force-gc and you have already made suggestions how this can > be triggered. How about adding a flag EmptyLocationSafety = [yes|no] in the location records, which defaults to "yes"? (and mentioning this in the log message) Ben > > > On Mon, 2004-09-20 at 15:39, Ben Summers wrote: >> >> Imagine this scenario. You have a single partition on a hard drive, >> mounted as /home. This drive fails, the server reboots, but fails to >> pass fsck checks as much is unreadable. You modify /etc/fstab so it >> isn't mounted, and reboot. The machine comes up. >> >> Unfortunately, you forgot to disable bbackupd running on boot, which >> you've got running in lazy mode. It runs, does it's initial scan, and >> marks everything in this partition as deleted. >> >> This is unlikely to help the situation much, and incidentally, has >> just >> happened to me. (Fortunately, I have "undelete" technology hidden in >> the server, and have just written a bit of code to allow access to it >> via bbackupquery, so it's not a calamity - even if it has also >> undeleted files which were deleted by the users before the partition >> vanished.) >> >> I propose a slight change in behaviour which may go some way towards >> resolving this problem, but I wonder if it will cause any other >> issues. >> >> * If bbackupd finds that the root of a Location (as specified in >> bbackupd.conf) contains no files or directories, it >> >> - will log a message "Backup location /x is empty, not changing >> store" at level LOG_ERR >> >> - will not modify the relevant location on the server >> >> Can anyone see any situation where this might cause problems? This >> will >> only be triggered if there's absolutely nothing to back up in a >> location. >> >> Thanks, >> >> Ben >> >> >> >> _______________________________________________ >> boxbackup mailing list >> boxbackup at fluffy.co.uk >> http://lists.warhead.org.uk/mailman/listinfo/boxbackup > -- > Richard Eigenmann > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Mon Sep 20 15:18:42 2004 From: boxbackup at fluffy.co.uk (Pascal Lalonde) Date: Mon, 20 Sep 2004 10:18:42 -0400 Subject: [Box Backup] Suggested change in behaviour In-Reply-To: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> References: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> Message-ID: <1095689921.21630.33.camel@doublebarrel.doom.und> On Mon, 2004-09-20 at 09:39, Ben Summers wrote: > * If bbackupd finds that the root of a Location (as specified in > bbackupd.conf) contains no files or directories, it > > - will log a message "Backup location /x is empty, not changing > store" at level LOG_ERR > > - will not modify the relevant location on the server > > Can anyone see any situation where this might cause problems? This will > only be triggered if there's absolutely nothing to back up in a > location. I'm not sure the approach is good, because as soon as a file appears, everything will be flagged as deleted. For example, I usually use userquotas with /home. When rebooting, the quota.user file will be created in /home, so this feature wouldn't help in that case. To me this feels only as a dirty workaround for only one case of this particular problem. I think that the first thing to do when a drive fails is to *remember* that you're running BoxBackup :-) But I guess I could've made that mistake as well. I agree about the e-mail though. And I think that the tag feature could help this problem. Suppose that you setup the client so that it tags the store with the date (200X-XX-XX) at the end of each day. Right after you notice the death of your drive, you flag today's tag as read-only. Here the e-mail could remind you that something weird is happening. Or maybe everything should always be read-only (even if flagged as deleted). And the user can set a limit so that he gets a warning when that limit is reached, and then HE can order the server to do a cleanup of files which are flagged as deleted. The more I think about it, the more I believe that letting the server decide when to erase what is a bad idea. Or at least there should be a way to tell the server, through the client, never to erase anything without being told to. Pascal Lalonde From boxbackup at fluffy.co.uk Mon Sep 20 15:30:27 2004 From: boxbackup at fluffy.co.uk (Richard Eigenmann) Date: Mon, 20 Sep 2004 16:30:27 +0200 Subject: [Box Backup] Suggested change in behaviour In-Reply-To: References: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> <1095688934.23708.21.camel@nurata.richinet.dyndns.org> Message-ID: <1095690627.23708.34.camel@nurata.richinet.dyndns.org> Sounds ok. What about the situation where the user has a USB stick and would like it's contents to be backed up too whenever the stick happens to be plugged into the computer but sometimes it is not? Under SuSE I've seen such devices receiving hard to remember names and am not sure they get the same name every time you plug it in (can't really say since they were hard to remember). But this is going off topic. Regards, Richard From boxbackup at fluffy.co.uk Mon Sep 20 16:13:07 2004 From: boxbackup at fluffy.co.uk (Adrian) Date: Mon, 20 Sep 2004 08:13:07 -0700 (MST) Subject: [Box Backup] Suggested change in behaviour In-Reply-To: <1095689921.21630.33.camel@doublebarrel.doom.und> References: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> <1095689921.21630.33.camel@doublebarrel.doom.und> Message-ID: <50940.199.64.0.252.1095693187.squirrel@199.64.0.252> > I agree about the e-mail though. And I think that the tag feature could > help this problem. Suppose that you setup the client so that it tags the > store with the date (200X-XX-XX) at the end of each day. Right after you > notice the death of your drive, you flag today's tag as read-only. Herehousekeeping > the e-mail could remind you that something weird is happening. > I think that Pascal may be on to something here. Perhaps an added feature of bbackupquery could be to set the "restore date". For example: Today is Monday. I discover that something bad happened to my data on Saturday (file modified, accidentally deleted, drive crash, etc). I set the restore date in bbackupquery to Friday, and restore the affected file, directory, or mount point. Bbackupquery ignores changes to the data that occured after Friday, and restores things to the way they were on Friday. Problem solved. I think that as it stands, all of the data is there to do this. Between old versions and deleted versions, I should be able to go back to any date, until the houskeeping function starts really deleting files on the server. The server would have to keep track of how far back it could go (to report to bbackupquery). As it does the housekeeping, it would need to determine the earliest date for which it has a complete backup set (And perhaps adjust its deletion logic a little to optimize?). More server space would allow the client to go further back in time. Does this make sense? Adrian From boxbackup at fluffy.co.uk Mon Sep 20 21:01:45 2004 From: boxbackup at fluffy.co.uk (Richard Eigenmann) Date: Mon, 20 Sep 2004 22:01:45 +0200 Subject: [Box Backup] 0.07PLUS4 (almost 0.08) In-Reply-To: <59CF7A58-089D-11D9-9E39-000A95AFF7F8@fluffy.co.uk> References: <59CF7A58-089D-11D9-9E39-000A95AFF7F8@fluffy.co.uk> Message-ID: <1095710505.26690.1.camel@nurata.richinet.dyndns.org> Can confirm that this compiles ok under Suse 9.1 and on a funny German Fedora Core 2 system. Seems to back up ok too. Regards, Richard From boxbackup at fluffy.co.uk Mon Sep 20 20:36:25 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 20 Sep 2004 20:36:25 +0100 Subject: [Box Backup] Suggested change in behaviour In-Reply-To: <50940.199.64.0.252.1095693187.squirrel@199.64.0.252> References: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> <1095689921.21630.33.camel@doublebarrel.doom.und> <50940.199.64.0.252.1095693187.squirrel@199.64.0.252> Message-ID: <5C2D1F04-0B3C-11D9-8095-000A95AFF7F8@fluffy.co.uk> On 20 Sep 2004, at 16:13, Adrian wrote: > >> I agree about the e-mail though. And I think that the tag feature >> could >> help this problem. Suppose that you setup the client so that it tags >> the >> store with the date (200X-XX-XX) at the end of each day. Right after >> you >> notice the death of your drive, you flag today's tag as read-only. > Herehousekeeping >> the e-mail could remind you that something weird is happening. >> > > I think that Pascal may be on to something here. Perhaps an added > feature > of bbackupquery could be to set the "restore date". For example: > Today > is Monday. I discover that something bad happened to my data on > Saturday > (file modified, accidentally deleted, drive crash, etc). I set the > restore date in bbackupquery to Friday, and restore the affected file, > directory, or mount point. Bbackupquery ignores changes to the data > that > occured after Friday, and restores things to the way they were on > Friday. > Problem solved. This is exactly what I'm planning to do. > > I think that as it stands, all of the data is there to do this. > Between > old versions and deleted versions, I should be able to go back to any > date, until the houskeeping function starts really deleting files on > the > server. You're right, all the data is there. But it's the housekeeping which complicates this. Firstly, it could delete objects, which means you don't know whether or not what you're restoring is correct. Secondly, if you have bbackupd working in lazy mode, it could sub-optimally delete things. Also some of the flags (like "this item is deleted") need to be modified to be time based, rather than absolute. So there needs to be some tracking code to tag files, and to record when a tagged set becomes incomplete through housekeeping. This is not trivial to do well. > > The server would have to keep track of how far back it could go (to > report > to bbackupquery). As it does the housekeeping, it would need to > determine > the earliest date for which it has a complete backup set (And perhaps > adjust its deletion logic a little to optimize?). More server space > would > allow the client to go further back in time. > > Does this make sense? Absolutely. I just need the time to implement it. This is, of course, the correct solution to my problem this morning. But I'm just wondering if there's a simple tweek which would catch things most of the time. If a small bit of code would help people when something has failed, when they might be thinking more about getting the server back up than the backup system, then it would be worth it? The bit about the quota file does complicate matters, although if nothing was mounted there it wouldn't have been created. Some more thought required. But it does suggest that I should think through exactly how things will behave when things go wrong, and make sure nothing bad happens even if the sysadmin isn't paying full attention. Thanks for all the thoughts. Ben From boxbackup at fluffy.co.uk Mon Sep 20 20:24:47 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 20 Sep 2004 20:24:47 +0100 Subject: [Box Backup] USB sticks (was: Suggested change in behaviour) In-Reply-To: <1095690627.23708.34.camel@nurata.richinet.dyndns.org> References: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> <1095688934.23708.21.camel@nurata.richinet.dyndns.org> <1095690627.23708.34.camel@nurata.richinet.dyndns.org> Message-ID: On 20 Sep 2004, at 15:30, Richard Eigenmann wrote: > > What about the situation where the user has a USB stick and would like > it's contents to be backed up too whenever the stick happens to be > plugged into the computer but sometimes it is not? Under SuSE I've seen > such devices receiving hard to remember names and am not sure they get > the same name every time you plug it in (can't really say since they > were hard to remember). But this is going off topic. I suggest a little script which monitors when the stick is plugged in (seeing what the USB daemon is doing), then when it's mounted uses rsync to mirror all the data on the hard disc somewhere. Ben From boxbackup at fluffy.co.uk Mon Sep 20 20:29:06 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 20 Sep 2004 20:29:06 +0100 Subject: [Box Backup] Your help is requested! Message-ID: <56A25FDE-0B3B-11D9-8095-000A95AFF7F8@fluffy.co.uk> Hi, I know you've all got limited amounts of time available, but I would really appreciate some reports on the success or otherwise of the "store as patches on server" code I implemented for 0.07PLUS3. My instructions on how to test it are below. (I didn't get any responses when I posted it last week). Many thanks. I find the help from all my users invaluable in making this a good backup system. Ben > Some of you are running the 0.07PLUS3 version, and I haven't heard any > complaints. We do need to check that it can restore files successfully > though. So, could you > > 1) Try a compare command in bbackupquery to check that the latest > versions are correct. (Nothing very much changed with regard to latest > versions, so I would be very surprised if this didn't work -- but I > would like to be reassured.) > > 2) Old versions can be stored as patches. Firstly, find a small file > with multiple versions (use list with the -o option to list old > versions) and see if a previous version can be fetched, and is > correct. Then find a large one with multiple versions, which has had > several small changes in the past. Use list with the -os options to > display old versions, and the server storage requirements. If the > sizes for old versions are much smaller than you'd expect when > compared with the latest version, they're stored as patches. Try > fetching a few, and checking their contents. > > I have been keeping an eye on my own installations, but they're all > under OpenBSD and my preferred way of installing things, so I do need > success reports from others. > From boxbackup at fluffy.co.uk Mon Sep 20 21:31:39 2004 From: boxbackup at fluffy.co.uk (Nick Knight) Date: Mon, 20 Sep 2004 21:31:39 +0100 Subject: [Box Backup] Your help is requested! Message-ID: Hello, I am currently half way though building a gentoo server for this - it maybe another week until it is tested though - I will try to edge it on.. Nick -----Original Message----- From: boxbackup-admin at fluffy.co.uk [mailto:boxbackup-admin at fluffy.co.uk] On Behalf Of Ben Summers Sent: 20 September 2004 20:29 To: boxbackup at fluffy.co.uk Subject: [Box Backup] Your help is requested! Hi, I know you've all got limited amounts of time available, but I would=20 really appreciate some reports on the success or otherwise of the=20 "store as patches on server" code I implemented for 0.07PLUS3. My=20 instructions on how to test it are below. (I didn't get any responses=20 when I posted it last week). Many thanks. I find the help from all my users invaluable in making=20 this a good backup system. Ben > Some of you are running the 0.07PLUS3 version, and I haven't heard any > complaints. We do need to check that it can restore files successfully > though. So, could you > > 1) Try a compare command in bbackupquery to check that the latest=20 > versions are correct. (Nothing very much changed with regard to latest > versions, so I would be very surprised if this didn't work -- but I=20 > would like to be reassured.) > > 2) Old versions can be stored as patches. Firstly, find a small file=20 > with multiple versions (use list with the -o option to list old=20 > versions) and see if a previous version can be fetched, and is=20 > correct. Then find a large one with multiple versions, which has had=20 > several small changes in the past. Use list with the -os options to=20 > display old versions, and the server storage requirements. If the=20 > sizes for old versions are much smaller than you'd expect when=20 > compared with the latest version, they're stored as patches. Try=20 > fetching a few, and checking their contents. > > I have been keeping an eye on my own installations, but they're all=20 > under OpenBSD and my preferred way of installing things, so I do need=20 > success reports from others. > _______________________________________________ boxbackup mailing list boxbackup at fluffy.co.uk http://lists.warhead.org.uk/mailman/listinfo/boxbackup From boxbackup at fluffy.co.uk Mon Sep 20 21:49:50 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Mon, 20 Sep 2004 21:49:50 +0100 Subject: [Box Backup] Your help is requested! In-Reply-To: <56A25FDE-0B3B-11D9-8095-000A95AFF7F8@fluffy.co.uk> References: <56A25FDE-0B3B-11D9-8095-000A95AFF7F8@fluffy.co.uk> Message-ID: <1095713390.19284.44.camel@avenin.ebourne.me.uk> On Mon, 2004-09-20 at 20:29, Ben Summers wrote: > I know you've all got limited amounts of time available, but I would > really appreciate some reports on the success or otherwise of the > "store as patches on server" code I implemented for 0.07PLUS3. My > instructions on how to test it are below. (I didn't get any responses > when I posted it last week). i386 client, x86_64 server, both FC2. Seems to work ok here. Only been running the new version a few days but can confirm it has started storing patches automatically and seems to work on a couple of files I looked at. Latest versions compare looks ok too. Cheers, Martin. From boxbackup at fluffy.co.uk Mon Sep 20 21:53:38 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Mon, 20 Sep 2004 21:53:38 +0100 Subject: [Box Backup] Suggested change in behaviour In-Reply-To: References: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> <1095688934.23708.21.camel@nurata.richinet.dyndns.org> Message-ID: <1095713618.19284.50.camel@avenin.ebourne.me.uk> On Mon, 2004-09-20 at 15:09, Ben Summers wrote: > On 20 Sep 2004, at 15:02, Richard Eigenmann wrote: > > You recommend that each mount point should be backed up separately. > > I require this! > > > However you might find situations where people have mount points > > further > > down the tree even if this is not such a good idea. > > I must write the code which stops people from doing that. I have to ask why, because it's not clear to me. In fact on one of my systems I have directories I would like to backup but have mounts below them (which I don't necessarily want to backup and could exclude if that works). Cheers, Martin. PS. I wrote half an email on why I like'd Adrian's description of setting a 'restore time/date' when Ben's reply came through saying he's planning that anyhow. So I stopped. This is just a 'me too' for encouragement. ;) From boxbackup at fluffy.co.uk Mon Sep 20 22:03:46 2004 From: boxbackup at fluffy.co.uk (=?iso-8859-2?Q?Mitja_Mu=BEeni=E8?=) Date: Mon, 20 Sep 2004 23:03:46 +0200 Subject: [Box Backup] Your help is requested! In-Reply-To: <56A25FDE-0B3B-11D9-8095-000A95AFF7F8@fluffy.co.uk> Message-ID: <200409202101.i8KL1PVF025940@very.fluffy.co.uk> > I know you've all got limited amounts of time available, but I would > really appreciate some reports on the success or otherwise of the > "store as patches on server" code I implemented for 0.07PLUS3. My > instructions on how to test it are below. (I didn't get any responses > when I posted it last week). Ben, 0.007PLUS3 works fine on my OBSD server/clients and Cygwin clients. I've tested restoring a previous version of some random selected files (mostly when showing off box backup :) ) and 0 problems found so far. Haven't tried PLUS4 yet - lack of a decent internet connection from my new flat to my servers. Regards, Mitja From boxbackup at fluffy.co.uk Tue Sep 21 00:24:32 2004 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 21 Sep 2004 00:24:32 +0100 (BST) Subject: [Box Backup] Suggested change in behaviour In-Reply-To: <1095689921.21630.33.camel@doublebarrel.doom.und> Message-ID: Hi all, First post, please don't flame me too bad, yadda yadda :-) On Mon, 20 Sep 2004, Pascal Lalonde wrote: > The more I think about it, the more I believe that letting the server > decide when to erase what is a bad idea. > > Or at least there should be a way to tell the server, through the > client, never to erase anything without being told to. I agree with this. The client decides voluntarily not to exceed the soft limit of storage space. It could also decide, according to a policy set by its owner, when to delete expired copies of data. One such policy might be to retain copies of deleted files for 7 days, in which case you would be safe as long as: - you realised your mistake soon enough - the feature to restore to a previous date (sim-tag) was implemented - deletion-date was versioned as well, so that the client knows that the files were not deleted on the requested date, but later. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Tue Sep 21 08:36:15 2004 From: boxbackup at fluffy.co.uk (Eduardo Alvarenga) Date: Tue, 21 Sep 2004 04:36:15 -0300 (BRT) Subject: [Box Backup] USB sticks (was: Suggested change in behaviour) In-Reply-To: References: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> <1095688934.23708.21.camel@nurata.richinet.dyndns.org> <1095690627.23708.34.camel@nurata.richinet.dyndns.org> Message-ID: On Mon, 20 Sep 2004, Ben Summers wrote: >> What about the situation where the user has a USB stick and would >> like it's contents to be backed up too whenever the stick happens to >> be plugged into the computer but sometimes it is not? Under SuSE I've >> seen such devices receiving hard to remember names and am not sure >> they get the same name every time you plug it in (can't really say >> since they were hard to remember). But this is going off topic. > > I suggest a little script which monitors when the stick is plugged in > (seeing what the USB daemon is doing), then when it's mounted uses > rsync to mirror all the data on the hard disc somewhere. hotplugd(8) should do it on OpenBSD and Linux, with little differences. Regards, -- Eduardo A. Alvarenga Analista de Suporte ORC Angola - Luanda - Angola BR +55 (91) 8116-0036 AO +244 (91) 315-776 From boxbackup at fluffy.co.uk Tue Sep 21 09:16:39 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 21 Sep 2004 09:16:39 +0100 Subject: [Box Backup] Suggested change in behaviour In-Reply-To: References: Message-ID: <8FF71309-0BA6-11D9-9DA3-000A95AFF7F8@fluffy.co.uk> On 21 Sep 2004, at 00:24, Chris Wilson wrote: > Hi all, > > First post, please don't flame me too bad, yadda yadda :-) > > On Mon, 20 Sep 2004, Pascal Lalonde wrote: >> The more I think about it, the more I believe that letting the server >> decide when to erase what is a bad idea. >> >> Or at least there should be a way to tell the server, through the >> client, never to erase anything without being told to. > > I agree with this. The client decides voluntarily not to exceed the > soft > limit of storage space. It could also decide, according to a policy > set by > its owner, when to delete expired copies of data. While the client controls when data is uploaded, the server controls when it is removed. > One such policy might be > to retain copies of deleted files for 7 days, in which case you would > be > safe as long as: > > - you realised your mistake soon enough > - the feature to restore to a previous date (sim-tag) was implemented > - deletion-date was versioned as well, so that the client knows that > the > files were not deleted on the requested date, but later. I was planning a system of tagging which would efficiently allow sets to be marked. The server would then keep a record of sets. Sets could be marked for preservation in which case the server would not delete files from them. When files were deleted, sets affected would be marked as incomplete. From the client side, you would be able to list the sets available, and select any one as your current view. This is all possible, but there's quite a bit of code which needs to be written to do it well. Quite apart from writing the automated tests for all this, which is going to be a nightmare. However, this doesn't make a quick and simple safety check useless. I might put it in anyway, even though it is not the correct and final solution. Ben PS: Thanks for all the reports of success with 0.07PLUS3 and 0.07PLUS4 -- keep them coming! From boxbackup at fluffy.co.uk Tue Sep 21 09:10:33 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 21 Sep 2004 09:10:33 +0100 Subject: [Box Backup] Suggested change in behaviour In-Reply-To: <1095713618.19284.50.camel@avenin.ebourne.me.uk> References: <73FB2B18-0B0A-11D9-8095-000A95AFF7F8@fluffy.co.uk> <1095688934.23708.21.camel@nurata.richinet.dyndns.org> <1095713618.19284.50.camel@avenin.ebourne.me.uk> Message-ID: On 20 Sep 2004, at 21:53, Martin Ebourne wrote: > On Mon, 2004-09-20 at 15:09, Ben Summers wrote: >> On 20 Sep 2004, at 15:02, Richard Eigenmann wrote: >>> You recommend that each mount point should be backed up separately. >> >> I require this! >> >>> However you might find situations where people have mount points >>> further >>> down the tree even if this is not such a good idea. >> >> I must write the code which stops people from doing that. > > I have to ask why, because it's not clear to me. Consider what happens if you rename a file or directory. It is not desirable to simply treat it as a new file upload it again and delete the old name, for obvious reasons. So what I do is record the inode numbers of all directories and all files over the threshold set in the bbackupd.conf file. If an object appears which has never been seen before, the inode number is checked against this database and if it is present (and the object under the old name no longer exists) then a rename command is issued instead. The problem with mount points comes about because two filesystems may have the same inode numbers, rather obstructing this tracking. There are ways to solve this and allow mounts within locations, but they would be * Space inefficient on the client / server * Time inefficient on the client * Unnecessarily increase the complexity of the code when there are perfectly good simpler solutions I haven't got round to writing the code to enforce good behaviour because * Each platform has it's own amusing ways of reading mount points * If you do set things up wrongly, then you're in no danger of losing data, just being a bit inefficient with bandwidth and server resources. > > In fact on one of my systems I have directories I would like to backup > but have mounts below them (which I don't necessarily want to backup > and > could exclude if that works). You should exclude the mount points, then set up separate locations for the other mount points. > > PS. I wrote half an email on why I like'd Adrian's description of > setting a 'restore time/date' when Ben's reply came through saying he's > planning that anyhow. So I stopped. This is just a 'me too' for > encouragement. ;) :-) I really like the idea as well, but have time constraints. While I have a lot of freedom in my working life which allows me to develop the system during the day, I do not have the freedom from the requirement to actually earn a living. The life of a freelancer is an interesting life. Ben From boxbackup at fluffy.co.uk Tue Sep 21 10:27:05 2004 From: boxbackup at fluffy.co.uk (richard_eigenmann) Date: Tue, 21 Sep 2004 11:27:05 +0200 Subject: [Box Backup] Suggested change in behaviour Message-ID: <1095758825062.richard_eigenmann@compuserve.com> That's an interesting point that you use inodes. I had the mental image that you use hash codes of files or perhaps hash codes of blocks of files to determine what needs to be backed up (for no good reason of course). The use of inodes explains why I have seen whole directories of pictures being re-backed up after I copied them around and deleted the original location after I was happy that everything had been rearranged properly. The Netstore software I used previously must have had some sort of hash code over parts of files because it built up a rather large table of data. It wouldn't complain too bitterly about that table being wiped but would spend time and perhaps bandwith to re-create it. Regards, Richard From boxbackup at fluffy.co.uk Tue Sep 21 10:35:55 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 21 Sep 2004 10:35:55 +0100 Subject: [Box Backup] Suggested change in behaviour In-Reply-To: <1095758825062.richard_eigenmann@compuserve.com> References: <1095758825062.richard_eigenmann@compuserve.com> Message-ID: On 21 Sep 2004, at 10:27, richard_eigenmann wrote: > That's an interesting point that you use inodes. I had the mental > image that > you use hash codes of files or perhaps hash codes of blocks of files to > determine what needs to be backed up (for no good reason of course). > > The use of inodes explains why I have seen whole directories of > pictures > being re-backed up after I copied them around and deleted the original > location after I was happy that everything had been rearranged > properly. When a file is being updated, it does uses hashes of blocks of data within that file to determine what has changed. So if you have a multi-mb database, but only change one entry, only a small bit of data will be uploaded. However, this is only within single files. Hence the behaviour you saw. > > The Netstore software I used previously must have had some sort of > hash code > over parts of files because it built up a rather large table of data. > It > wouldn't complain too bitterly about that table being wiped but would > spend > time and perhaps bandwith to re-create it. I did consider a more global hash based approach, but it limited what I could do with the encryption, and quite frankly, was a little scary when it came to data integrity. Hashes are all very well and good, but probability being what it is you're going to get a collision at some point. Ben From boxbackup at fluffy.co.uk Tue Sep 21 21:59:49 2004 From: boxbackup at fluffy.co.uk (Tomasz Nowak) Date: Tue, 21 Sep 2004 22:59:49 +0200 Subject: [Box Backup] userland RAID 5 vs RAID 1 consideration Message-ID: <00ec01c4a01d$efa88130$0e00a8c0@NODUS> Hello, As far as I understand documentation, userspace RAID is done in kind of level 5 RAID model (3 disk, files are split into 2 stripes and one parity bit). I'd really like to see RAID level 1 (mirroring) implementation. And that's why I think it would be a good idea: 1. Storage performance is rather not the case on backup machine; RAID 1 would perfectly do the job. 2. Reliability of RAID 1 is just the same as RAID 5. 3. Each RAID 1 array is 33% cheaper then RAID 5 array. 4. Disk space.. well - 250 GB of Caviar diskspace is really enough for me :-) Just need mirroring of such pair to be sure my backup is disk-failure tolerant. WDYT? -- Tomasz Nowak From boxbackup at fluffy.co.uk Tue Sep 21 22:27:29 2004 From: boxbackup at fluffy.co.uk (Garry Glendown) Date: Tue, 21 Sep 2004 23:27:29 +0200 Subject: [Box Backup] userland RAID 5 vs RAID 1 consideration In-Reply-To: <00ec01c4a01d$efa88130$0e00a8c0@NODUS> References: <00ec01c4a01d$efa88130$0e00a8c0@NODUS> Message-ID: <41509CC1.8000904@regio.net> Tomasz Nowak wrote: > 1. Storage performance is rather not the case on backup > machine; RAID 1 would perfectly do the job. > 2. Reliability of RAID 1 is just the same as RAID 5. > 3. Each RAID 1 array is 33% cheaper then RAID 5 array. It is? 250G of RAID1 storage cost you 500G of raw storage, whereas 250G of RAID5 is only 375G of raw storage ... or other way around, two 250G HDs give you a total of 250G RAID1, three 250G HDs give you 500G of RAID5 ... for performance, I/O might be slightly better for RAID5, though CPU load is probably slightly higher ... -gg From boxbackup at fluffy.co.uk Wed Sep 22 01:51:48 2004 From: boxbackup at fluffy.co.uk (Pascal Lalonde) Date: Tue, 21 Sep 2004 20:51:48 -0400 Subject: [Box Backup] Your help is requested! In-Reply-To: <56A25FDE-0B3B-11D9-8095-000A95AFF7F8@fluffy.co.uk> References: <56A25FDE-0B3B-11D9-8095-000A95AFF7F8@fluffy.co.uk> Message-ID: <1095814308.23665.25.camel@doublebarrel.doom.und> Ok, I feel like I'm doing something *really* stupid here. I'll assume anything that comes at me after this question: How can I redirect bbackupquery's output (compare -a) to a file?? I know I already did it once, but it doesn't seem to work anymore. Here's what I do: bbackupquery "compare -a" quit > /tmp/compare.log This works: bbackupquery ls quit > /tmp/compare.log When I try to follow the file with tail -f, nothing shows up. Yesterday, after my full compare, I found my compare.log to be empty. Yet when I'm not redirecting to a file, it seems to work fine. Now, what am I doing wrong? Can someone try it and tell me how it goes? Same behavior on both OpenBSD 3.5 i386 and Gentoo i386. Pascal Lalonde From boxbackup at fluffy.co.uk Wed Sep 22 06:27:39 2004 From: boxbackup at fluffy.co.uk (Per Thomsen) Date: Tue, 21 Sep 2004 22:27:39 -0700 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft 0.01) Message-ID: <41510D4B.2030601@reedtz.com> All, Here are the things I have come up with from a user's perspective as requirements/specifications for server-side management. This is very rough and not necessarily thought through at all levels, but I thought I'd get it out there, so everyone could get a chance to see it, and shoot down / change / add to what's in here. 1. Symbolic Names for hosts. In addition to using a hex number to identify backup clients, a mapping must be put in place between the account number, and a name the user (backup admin) determines. This name can be up to 255 characters long (RFC 1034). For backwards compatibility, account numbers must still be accessible, and usable just as before. If needed command line switches can be applied to denote one or the other. Additionally, for management purposes it should be possible to explicitly set the account number during the creation of a new client (as well as the symbolic name). While the name can be arbitrary, the installation process should attempt to determine the full domain name of the host, and use that value as the default. 2. Client groups. To support the distinction of groups of boxes being backed up, the concept of groups of users should be implemented. Groups are collection of clients or other groups (with appropriate circular-dependency checking). The 'bbstoreaccounts' executable will add functionality to manage groups, including getting statistics from a group. The statistics will be the cumulative values of the same data as for a single client, with the addition of the following: - List of group members - ? 3. Client Monitoring. The client should be able to send 'heart beat' messages to the bbstored server. Configuration information for heart beat is kept on the bbstored server. It includes: For each client: - on/off switch. Clients can be monitored, but are not required to be. This is especially useful for mobile users, who are not connected to the internet all the time. - Heart beat interval. How often the client sends heart beat information. Given in seconds. Default is 900. Heart beat messages will not interfere with long-running syncs or restores (large files), but will insert itself as close to the interval as possible, to ensure that as few false error alerts as possible are sent to administrators. When bbackupd starts, it will register with the bbstored server, and request its configuration information. It will use this to send the messages at the appointed times. Also, bbstored will create a record of the now running bbackupd (in memory, mmap, or whatever works best), to hold the data for the statistics, as well as to ensure that only clients that have registered are being monitored. When a bbackupd daemon completes an orderly shutdown, it will 'de-register' itself from the bbstored service, to ensure that no false 'down alerts' go out. However, if bbackupd dies as a result of some failure, the record on the bbstored server will remain, and eventually cause alerts to go out to the backup administrator. The heart beat packet contains the following information: - Host identifier (name and/or account number) - uptime (ie. how long has bbackupd been running on this host) - timestamp of last sync (when was the last file uploaded) - Number of bytes synced since last heart beat message - Number of bytes restored since last heart beat message - any significant errors that have occured since last heart beat. - ? On the server side, a daemon (most likely bbstored) receives these heart beat messages, and keeps track of the status of all clients. It will keep a running counter of the byte-count statistics for the client, as well as a log of the significant errors. bbstore When a bbackup client daemon dies unexpectedly, the bbstored server will notice that there is no heart beat message from the client after approximately 2 x the heart beat interval. It will then notify backup administrators using the NotifySysAdmin.sh mechanism, or one very much like it. When a significant error occurs, and is logged with the server, a similar notification mechanism will be used to notify the backup administrator. 4. Space use reporting. Reporting of space consumption is needed at several levels: - The entire bbstored server (all RAID volumes being used for backups). - Each Volume. Ensuring that one single volume isn't bearing the brunt of the load, as well as for planning purposes. - By Group. This relates to item #2 in my list. It has very similar reporting requirements to the individual client, with the same additions as described in the Group section. - By Individual. This is already available in the current version. 5. Account Database. The ability to store the client account information in a database is crucial to the stability and scalability of the system. Change the use of text files to using a database. 6. Interaction with the rest of the world. Interfacing in an easy way to other systems for Monitoring and reporting purposes. Rather than nicely formatted output there should be an option in all commands to format the output for human and for script consumption. This data could then be used by products like Nagios (www.nagios.org) As I said this is a very rough draft. So, go crazy: edit, change, add, whatever. Thanks, Per -- Per Reedtz Thomsen | The Reedtz Corporation | F: 209 883 4119 V: 209 883 4102 | pthomsen at reedtz.com | C: 415 425 4025 GPG ID: 1209784F | Yahoo! Chat: pthomsen | AIM: pthomsen From boxbackup at fluffy.co.uk Wed Sep 22 09:05:20 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 22 Sep 2004 09:05:20 +0100 Subject: [Box Backup] userland RAID 5 vs RAID 1 consideration In-Reply-To: <41509CC1.8000904@regio.net> References: <00ec01c4a01d$efa88130$0e00a8c0@NODUS> <41509CC1.8000904@regio.net> Message-ID: <25C13A3A-0C6E-11D9-9734-000A95AFF7F8@fluffy.co.uk> On 21 Sep 2004, at 22:27, Garry Glendown wrote: > Tomasz Nowak wrote: >> 1. Storage performance is rather not the case on backup >> machine; RAID 1 would perfectly do the job. >> 2. Reliability of RAID 1 is just the same as RAID 5. >> 3. Each RAID 1 array is 33% cheaper then RAID 5 array. > > It is? 250G of RAID1 storage cost you 500G of raw storage, whereas > 250G of RAID5 is only 375G of raw storage ... or other way around, two > 250G HDs give you a total of 250G RAID1, three 250G HDs give you 500G > of RAID5 ... for performance, I/O might be slightly better for RAID5, > though CPU load is probably slightly higher ... Garry is correct. RAID5 gives you 66% of the hard discs as usable space, RAID1 only 50%. However, prices of hard discs may make RAID1 have a cheaper per-Gb cost than RAID5. With Box Backup's implementation of RAID, performance would be the same. (Turning files into their RAID representation is delayed until after all the data is written.) Of course, you can choose whatever RAID level you want by putting Box Backup into non-RAID mode and using the underlying OS to do it instead. Ben From boxbackup at fluffy.co.uk Wed Sep 22 09:15:33 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 22 Sep 2004 09:15:33 +0100 Subject: [Box Backup] Your help is requested! In-Reply-To: <1095814308.23665.25.camel@doublebarrel.doom.und> References: <56A25FDE-0B3B-11D9-8095-000A95AFF7F8@fluffy.co.uk> <1095814308.23665.25.camel@doublebarrel.doom.und> Message-ID: <933C80E0-0C6F-11D9-9734-000A95AFF7F8@fluffy.co.uk> On 22 Sep 2004, at 01:51, Pascal Lalonde wrote: > Ok, > > I feel like I'm doing something *really* stupid here. I'll assume > anything that comes at me after this question: > > How can I redirect bbackupquery's output (compare -a) to a file?? > > I know I already did it once, but it doesn't seem to work anymore. > > Here's what I do: > bbackupquery "compare -a" quit > /tmp/compare.log > > This works: > bbackupquery ls quit > /tmp/compare.log > > When I try to follow the file with tail -f, nothing shows up. > > Yesterday, after my full compare, I found my compare.log to be empty. > Yet when I'm not redirecting to a file, it seems to work fine. > > Now, what am I doing wrong? Can someone try it and tell me how it goes? > > Same behavior on both OpenBSD 3.5 i386 and Gentoo i386. That is very strange. Having the quoted command won't affect anything bbackupquery does. Perhaps it's a weird shell thing? What if you create a script #!/bin/sh bbackupquery "compare -a" quit and then redirect the output of the script instead? You aren't falling foul of any automated /tmp clearing cron jobs, are you? Ben From boxbackup at fluffy.co.uk Wed Sep 22 10:07:24 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 22 Sep 2004 10:07:24 +0100 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft 0.01) In-Reply-To: <41510D4B.2030601@reedtz.com> References: <41510D4B.2030601@reedtz.com> Message-ID: On 22 Sep 2004, at 06:27, Per Thomsen wrote: > All, > Here are the things I have come up with from a user's perspective as > requirements/specifications for server-side management. > > This is very rough and not necessarily thought through at all levels, > but I thought I'd get it out there, so everyone could get a chance to > see it, and shoot down / change / add to what's in here. All looks sensible. Some comments below, to start off the comments. > > 1. Symbolic Names for hosts. > In addition to using a hex number to identify backup clients, a mapping > must be put in place between the account number, and a name the user > (backup admin) determines. This name can be up to 255 characters long > (RFC 1034). > > For backwards compatibility, account numbers must still be accessible, > and usable just as before. If needed command line switches can be > applied to denote one or the other. Additionally, for management > purposes it should be possible to explicitly set the account number > during the creation of a new client (as well as the symbolic name). > > While the name can be arbitrary, the installation process should > attempt > to determine the full domain name of the host, and use that value as > the > default. Will these names replace the account numbers on the client side too? So instead of getting an account number from your administrator before configuring and generating the certificate, you would just use the symbolic name of your host? > > 2. Client groups. > To support the distinction of groups of boxes being backed up, the > concept of groups of users should be implemented. Groups are collection > of clients or other groups (with appropriate circular-dependency > checking). > > The 'bbstoreaccounts' executable will add functionality to manage > groups, including getting statistics from a group. The statistics will > be the cumulative values of the same data as for a single client, with > the addition of the following: > > - List of group members > - ? Is the requirement for a group to contain another group really essential? It complicates things somewhat. It would be much simpler if a client could be contained in one group, and one group only. > > 3. Client Monitoring. > The client should be able to send 'heart beat' messages to the bbstored > server. > > Configuration information for heart beat is kept on the bbstored > server. > It includes: > > For each client: > - on/off switch. Clients can be monitored, but are not required to be. > This is especially useful for mobile users, who are not connected to > the > internet all the time. > - Heart beat interval. How often the client sends heart beat > information. Given in seconds. Default is 900. > > Heart beat messages will not interfere with long-running syncs or > restores (large files), but will insert itself as close to the interval > as possible, to ensure that as few false error alerts as possible are > sent to administrators. > > When bbackupd starts, it will register with the bbstored server, and > request its configuration information. It will use this to send the > messages at the appointed times. > > Also, bbstored will create a record of the now running bbackupd (in > memory, mmap, or whatever works best), to hold the data for the > statistics, as well as to ensure that only clients that have registered > are being monitored. > > When a bbackupd daemon completes an orderly shutdown, it will > 'de-register' itself from the bbstored service, to ensure that no false > 'down alerts' go out. However, if bbackupd dies as a result of some > failure, the record on the bbstored server will remain, and eventually > cause alerts to go out to the backup administrator. > > The heart beat packet contains the following information: > > - Host identifier (name and/or account number) > - uptime (ie. how long has bbackupd been running on this host) > - timestamp of last sync (when was the last file uploaded) > - Number of bytes synced since last heart beat message > - Number of bytes restored since last heart beat message > - any significant errors that have occured since last heart beat. > - ? > > On the server side, a daemon (most likely bbstored) receives these > heart > beat messages, and keeps track of the status of all clients. It will > keep a running counter of the byte-count statistics for the client, as > well as a log of the significant errors. bbstore > > When a bbackup client daemon dies unexpectedly, the bbstored server > will > notice that there is no heart beat message from the client after > approximately 2 x the heart beat interval. It will then notify backup > administrators using the NotifySysAdmin.sh mechanism, or one very much > like it. > > When a significant error occurs, and is logged with the server, a > similar notification mechanism will be used to notify the backup > administrator. The current implementation will connect to the server every so many seconds (the UpdateStoreInterval) in lazy mode, or whenever the sync is started in snapshot mode. If there was an option to always connect to the server, instead of just when there was data to update, and sent in the additional information above, would this be a suitable mechanism? It seems slightly unnecessary to have a completely separate method for transferring this information. > > 4. Space use reporting. > Reporting of space consumption is needed at several levels: > > - The entire bbstored server (all RAID volumes being used for backups). > - Each Volume. Ensuring that one single volume isn't bearing the brunt > of the load, as well as for planning purposes. > - By Group. This relates to item #2 in my list. It has very similar > reporting requirements to the individual client, with the same > additions > as described in the Group section. > - By Individual. This is already available in the current version. > > 5. Account Database. > The ability to store the client account information in a database is > crucial to the stability and scalability of the system. Change the use > of text files to using a database. And of course, make the database so you can store information for many bbstored daemons in it. > > 6. Interaction with the rest of the world. > Interfacing in an easy way to other systems for Monitoring and > reporting > purposes. Rather than nicely formatted output there should be an option > in all commands to format the output for human and for script > consumption. This data could then be used by products like Nagios > (www.nagios.org) Is there a standard format which could be used. > > As I said this is a very rough draft. So, go crazy: edit, change, add, > whatever. I look forward to other comments too. Per -- thanks for your work on this. Ben From boxbackup at fluffy.co.uk Wed Sep 22 11:36:48 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Wed, 22 Sep 2004 11:36:48 +0100 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft 0.01) In-Reply-To: References: <41510D4B.2030601@reedtz.com> Message-ID: <20040922113648.sgk4gsck4w80w448@ebourne.me.uk> Ben Summers wrote: > Will these names replace the account numbers on the client side too? So > instead of getting an account number from your administrator before > configuring and generating the certificate, you would just use the > symbolic name of your host? I think the complete removal of account numbers in favour of names would be a definite usability enhancement. The default of using client hostname as the backup name makes sense and is what I'd do. Regarding upgradability/backwards compatability, rather than operating with names and numbers I'd simply relax restrictions on the number to make it a name. Thus upgraded systems would carry on functioning with their 'number' and admins would be free to change the number to a more suitable name at their leisure. >> 2. Client groups. One of the features of a group I think would be most useful, but I didn't see from the original post, would be the ability to set disk usage groupwide. ie. You could still set on an individual client basis, but you could also set a (presumably more restrictive) groupwide usage too, which would apply to all of those clients in aggregate. >> When a bbackup client daemon dies unexpectedly, the bbstored server will >> notice that there is no heart beat message from the client after >> approximately 2 x the heart beat interval. It will then notify backup >> administrators using the NotifySysAdmin.sh mechanism, or one very much >> like it. Whatever the mechanism is, it would be useful to have some way to send the email to a different address depending on the group. Also, I don't see here what happens when bbstored dies. If all the state for the clients is held in memory, that will be lost. I guess it would need to be persisted to disk. If a database was in use that would seem to be the right place. As to 'database', I always seem to end up running a SQL database on my machines, but I don't think that should be a requirement. If a 'real' database is decided upon, the way to go could be SQLite - or even better if you use libdbi that abstracts the SQL databases quite nicely and lets you use any of them, including SQLite (which can be built in). Alternatively there's always db4 at a push. > The current implementation will connect to the server every so many > seconds (the UpdateStoreInterval) in lazy mode, or whenever the sync is > started in snapshot mode. If there was an option to always connect to > the server, instead of just when there was data to update, and sent in > the additional information above, would this be a suitable mechanism? That would sound reasonable, and reduced complexity is always a good aim. > And of course, make the database so you can store information for many > bbstored daemons in it. Are there situations where it is beneficial to run multiple bbstored daemons? All I can think of is to store the backed up data at a different place in the filesystem. In which case this sounds like the sort of thing that should be settable on a groupwide basis, and then one daemon could suffice. One final point, I would like to be able to control this 'watchdog' functionality from the client end if possible, and the same for individual client diskspace usage within a group. You quite possibly don't have shell access to the server, but might need to change this stuff. On the other hand, the box owner is likely to want to set the limits on the group for obvious reasons. Cheers, Martin. From boxbackup at fluffy.co.uk Wed Sep 22 13:47:13 2004 From: boxbackup at fluffy.co.uk (richard_eigenmann) Date: Wed, 22 Sep 2004 14:47:13 +0200 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft0.01) Message-ID: <1095857233132.richard_eigenmann@compuserve.com> The following comes to mind when thinking about group backups: Consider the situation where we have a department with say 10 workstations that we would like to have a full backup of from the root of each machine's filesystem (assume a version of a widely used operating system). The directories will be filled for large parts with bloatware that can be found on most other machines albeit in different locations. Theoretically it would be sufficient to back up all this stuff just once but allow restore to each client that held the file at the time in question. If we implemented such a kind of "backup recognition" algorythm this could speed up backups of remote laptops as perhaps the documents the laptop user has been working on have already been backed up from users back at the base. I imagine this sort of feature could save somewhere between 0.5 and 5 GB per workstation that is doing a full backup. This could be significant to the scalability of the boxbackup. Of course this sort of thing probably would lead to massive redevelopment of code and should only be undertaken if there is a very strong demand. I for one don't need it. Regards, Richard From boxbackup at fluffy.co.uk Wed Sep 22 15:54:11 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 22 Sep 2004 15:54:11 +0100 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft 0.01) In-Reply-To: <20040922113648.sgk4gsck4w80w448@ebourne.me.uk> References: <41510D4B.2030601@reedtz.com> <20040922113648.sgk4gsck4w80w448@ebourne.me.uk> Message-ID: <4365AF74-0CA7-11D9-9734-000A95AFF7F8@fluffy.co.uk> On 22 Sep 2004, at 11:36, Martin Ebourne wrote: > Ben Summers wrote: >> Will these names replace the account numbers on the client side too? >> So >> instead of getting an account number from your administrator before >> configuring and generating the certificate, you would just use the >> symbolic name of your host? > > I think the complete removal of account numbers in favour of names > would be a > definite usability enhancement. The default of using client hostname > as the > backup name makes sense and is what I'd do. > > Regarding upgradability/backwards compatability, rather than operating > with > names and numbers I'd simply relax restrictions on the number to make > it a > name. Thus upgraded systems would carry on functioning with their > 'number' > and admins would be free to change the number to a more suitable name > at > their leisure. Account numbers are rather essential in the design, so will have to stay there. There needs to be a translation layer at some point. > >>> 2. Client groups. > > One of the features of a group I think would be most useful, but I > didn't see > from the original post, would be the ability to set disk usage > groupwide. > ie. You could still set on an individual client basis, but you could > also > set a (presumably more restrictive) groupwide usage too, which would > apply > to all of those clients in aggregate. That's a good point. How housekeeping will know which account to trim when the group goes over usage is an interesting question though. > >>> When a bbackup client daemon dies unexpectedly, the bbstored server >>> will >>> notice that there is no heart beat message from the client after >>> approximately 2 x the heart beat interval. It will then notify backup >>> administrators using the NotifySysAdmin.sh mechanism, or one very >>> much >>> like it. > > Whatever the mechanism is, it would be useful to have some way to send > the > email to a different address depending on the group. > > Also, I don't see here what happens when bbstored dies. If all the > state for > the clients is held in memory, that will be lost. I guess it would > need to > be persisted to disk. If a database was in use that would seem to be > the > right place. Yes. > > As to 'database', I always seem to end up running a SQL database on my > machines, but I don't think that should be a requirement. If a 'real' > database is decided upon, the way to go could be SQLite - or even > better if > you use libdbi that abstracts the SQL databases quite nicely and lets > you > use any of them, including SQLite (which can be built in). > Alternatively > there's always db4 at a push. Absolutely, a generic interface would be the way to go. SQLite for small installations, a database, perhaps on another machine, for larger ones. > >> The current implementation will connect to the server every so many >> seconds (the UpdateStoreInterval) in lazy mode, or whenever the sync >> is >> started in snapshot mode. If there was an option to always connect to >> the server, instead of just when there was data to update, and sent in >> the additional information above, would this be a suitable mechanism? > > That would sound reasonable, and reduced complexity is always a good > aim. I'm glad you agree. > >> And of course, make the database so you can store information for many >> bbstored daemons in it. > > Are there situations where it is beneficial to run multiple bbstored > daemons? If you have multiple independent backup servers, then running one daemon on each could be considered beneficial. > All I can think of is to store the backed up data at a different place > in > the filesystem. You can set multiple raidfile sets for that. > In which case this sounds like the sort of thing that should > be settable on a groupwide basis, and then one daemon could suffice. > > One final point, I would like to be able to control this 'watchdog' > functionality from the client end if possible, Whether or not it connects on every scan will be set in the client configuration file. > and the same for individual > client diskspace usage within a group. You quite possibly don't have > shell > access to the server, but might need to change this stuff. On the other > hand, the box owner is likely to want to set the limits on the group > for > obvious reasons. You could always have a little web interface to change the settings on the server. I suspect that splitting functionality between the client and server too much would just complicate things. Ben From boxbackup at fluffy.co.uk Wed Sep 22 14:45:13 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 22 Sep 2004 14:45:13 +0100 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft0.01) In-Reply-To: <1095857233132.richard_eigenmann@compuserve.com> References: <1095857233132.richard_eigenmann@compuserve.com> Message-ID: On 22 Sep 2004, at 13:47, richard_eigenmann wrote: > The following comes to mind when thinking about group backups: > > Consider the situation where we have a department with say 10 > workstations > that we would like to have a full backup of from the root of each > machine's > filesystem (assume a version of a widely used operating system). The > directories will be filled for large parts with bloatware that can be > found > on most other machines albeit in different locations. Theoretically it > would > be sufficient to back up all this stuff just once but allow restore to > each > client that held the file at the time in question. > > If we implemented such a kind of "backup recognition" algorythm this > could > speed up backups of remote laptops as perhaps the documents the laptop > user > has been working on have already been backed up from users back at the > base. > > I imagine this sort of feature could save somewhere between 0.5 and 5 > GB per > workstation that is doing a full backup. This could be significant to > the > scalability of the boxbackup. > > Of course this sort of thing probably would lead to massive > redevelopment of > code and should only be undertaken if there is a very strong demand. I > for > one don't need it. I would have thought that in such a department you would have a means of recreating the workstations very quickly, perhaps using Ghost for Windows or a custom installer for UNIX. In which case you'd only need to back up the data. Ben From boxbackup at fluffy.co.uk Wed Sep 22 16:51:16 2004 From: boxbackup at fluffy.co.uk (Per Thomsen) Date: Wed, 22 Sep 2004 08:51:16 -0700 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft 0.01) In-Reply-To: <20040922113648.sgk4gsck4w80w448@ebourne.me.uk> References: <41510D4B.2030601@reedtz.com> <20040922113648.sgk4gsck4w80w448@ebourne.me.uk> Message-ID: <41519F74.4010800@reedtz.com> On 9/22/04 3:36 AM, Martin Ebourne wrote: > Ben Summers wrote: > >> Will these names replace the account numbers on the client side too? So >> instead of getting an account number from your administrator before >> configuring and generating the certificate, you would just use the >> symbolic name of your host? > > > I think the complete removal of account numbers in favour of names > would be a > definite usability enhancement. The default of using client hostname > as the > backup name makes sense and is what I'd do. > > Regarding upgradability/backwards compatability, rather than operating > with > names and numbers I'd simply relax restrictions on the number to make > it a > name. Thus upgraded systems would carry on functioning with their > 'number' > and admins would be free to change the number to a more suitable name at > their leisure. In that regard, I think that when upgrading clients to use symbolic names, the number should stay, and be usable just as before. There doesn't seem to be any reason to remove it. If you've built up a system for managing your clients with the account numbers, I think you should still be able to use that. However, if you're new to BB and install a new server, you won't even know that the numbers are there, and will simply just use the symbolic names. > >>> 2. Client groups. >> > > One of the features of a group I think would be most useful, but I > didn't see > from the original post, would be the ability to set disk usage groupwide. > ie. You could still set on an individual client basis, but you could also > set a (presumably more restrictive) groupwide usage too, which would > apply > to all of those clients in aggregate. I completely agree. I was trying to convey that, but I guess it wasn't clear. To me, the aggregate group usage is the most important aspect of the group feature. It enables the administrator to get good trending information on a group basis, to predict disk usage, etc. > >>> When a bbackup client daemon dies unexpectedly, the bbstored server >>> will >>> notice that there is no heart beat message from the client after >>> approximately 2 x the heart beat interval. It will then notify backup >>> administrators using the NotifySysAdmin.sh mechanism, or one very much >>> like it. >> > > Whatever the mechanism is, it would be useful to have some way to send > the > email to a different address depending on the group. Good idea... > > Also, I don't see here what happens when bbstored dies. If all the > state for > the clients is held in memory, that will be lost. I guess it would > need to > be persisted to disk. If a database was in use that would seem to be the > right place. Yes, the data must be on disk. Preferably in a database. > > As to 'database', I always seem to end up running a SQL database on my > machines, but I don't think that should be a requirement. If a 'real' > database is decided upon, the way to go could be SQLite - or even > better if > you use libdbi that abstracts the SQL databases quite nicely and lets you > use any of them, including SQLite (which can be built in). Alternatively > there's always db4 at a push. Or pgSQL or mySQL. I like the idea of an external database, so I can create my own reports, etc. from the data there. > >> The current implementation will connect to the server every so many >> seconds (the UpdateStoreInterval) in lazy mode, or whenever the sync is >> started in snapshot mode. If there was an option to always connect to >> the server, instead of just when there was data to update, and sent in >> the additional information above, would this be a suitable mechanism? > > > That would sound reasonable, and reduced complexity is always a good aim. > Yes, I agree. No reason to create another way of connecting. Just simply always connect, if the client is configured to send heart beat messages. Thanks, Per -- Per Reedtz Thomsen | The Reedtz Corporation | F: 209 883 4119 V: 209 883 4102 | pthomsen at reedtz.com | C: 415 425 4025 GPG ID: 1209784F | Yahoo! Chat: pthomsen | AIM: pthomsen From boxbackup at fluffy.co.uk Wed Sep 22 17:19:05 2004 From: boxbackup at fluffy.co.uk (richard_eigenmann) Date: Wed, 22 Sep 2004 18:19:05 +0200 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft0.01) Message-ID: <1095869945607.richard_eigenmann@compuserve.com> > I would have thought that in such a department you would have a means > of recreating the workstations very quickly, perhaps using Ghost for > Windows or a custom installer for UNIX. In which case you'd only need > to back up the data. It depends on the sophistication of the administrator, the uniformity of the configuration, the size of the organisation and then the degree of protection you require. With some users you simply can't rely on them not storing their important data in applicaion directories, temporary locations, on the root of the disk or anywhere else that makes no sense. I always advocate creating a single directory that contains all the important data (in subfolders) so that it can be easily backed up and moved to the next machine. However, people tend to agree with me and then continue in their old habits until the inevitable happens. Of course there is also the issue of mounted lan drives that a user might want to have backed up with boxbackup. Ignoring the inonde / mount point issue for a moment we could find situations where the configuration would back up some Lan directories from multiple client machines as well as from the server backup too. I think setting up the client side backup is not something you want to leave to users! On the other hand I wonder where and why organisations would want to back up workstations at all. At my workplace I don't believe that my workstation is backed up. If it blows they clone a new one and I'm back on the lan. Local data will be gone. That is communicated and if you don't follow the guidelines then that's your problem. Laptops, however, can validly have data locally. Laptops can also be tricky with drivers and nonstandard configurations. If it's not a large organisation that can mandate the use of only one single standardised type of Laptop then backing up the OS with it's specific configuration probably makes a lot of sense. Regards, Richard From boxbackup at fluffy.co.uk Wed Sep 22 21:14:33 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Wed, 22 Sep 2004 21:14:33 +0100 Subject: [Box Backup] PATCH: Fix use of reload with start scripts Message-ID: <1095884072.7028.1.camel@avenin.ebourne.me.uk> --=-WNQRhE44uTwT0Mmrbiny Content-Type: text/plain Content-Transfer-Encoding: 7bit Fixes use of 'reload' with RPM start scripts. Cheers, Martin. --=-WNQRhE44uTwT0Mmrbiny Content-Disposition: attachment; filename=start-script.patch Content-Type: text/x-patch; name=start-script.patch; charset=UTF-8 Content-Transfer-Encoding: 7bit diff -ur boxbackup-0.07PLUS4.orig/contrib/rpm/bbackupd boxbackup-0.07PLUS4/contrib/rpm/bbackupd --- boxbackup-0.07PLUS4.orig/contrib/rpm/bbackupd 2004-09-17 11:53:42.000000000 +0100 +++ boxbackup-0.07PLUS4/contrib/rpm/bbackupd 2004-09-22 21:08:28.554763133 +0100 @@ -105,6 +105,9 @@ restart) restart ;; + reload) + reload + ;; status) rhstatus ;; diff -ur boxbackup-0.07PLUS4.orig/contrib/rpm/bbstored boxbackup-0.07PLUS4/contrib/rpm/bbstored --- boxbackup-0.07PLUS4.orig/contrib/rpm/bbstored 2004-09-17 11:53:42.000000000 +0100 +++ boxbackup-0.07PLUS4/contrib/rpm/bbstored 2004-09-22 21:08:42.185786549 +0100 @@ -105,6 +105,9 @@ restart) restart ;; + reload) + reload + ;; status) rhstatus ;; --=-WNQRhE44uTwT0Mmrbiny-- From boxbackup at fluffy.co.uk Thu Sep 23 06:51:54 2004 From: boxbackup at fluffy.co.uk (Garry Glendown) Date: Thu, 23 Sep 2004 07:51:54 +0200 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft0.01) In-Reply-To: <1095857233132.richard_eigenmann@compuserve.com> References: <1095857233132.richard_eigenmann@compuserve.com> Message-ID: <4152647A.1020909@regio.net> richard_eigenmann wrote: > If we implemented such a kind of "backup recognition" algorythm this could > speed up backups of remote laptops as perhaps the documents the laptop user > has been working on have already been backed up from users back at the base. > > I imagine this sort of feature could save somewhere between 0.5 and 5 GB per > workstation that is doing a full backup. This could be significant to the > scalability of the boxbackup. > > Of course this sort of thing probably would lead to massive redevelopment of > code and should only be undertaken if there is a very strong demand. I for > one don't need it. Your suggestion might have been derived from what some companies already sell as backup solutions ... e.g., InterXion, a large hoster, is selling exactly this feature set ... they say they have all mayor Windows version and many M$ apps "on file" and if the version found on the client machine matches the one on file, it is only stored as reference ... they even do this for other files, dynamically extending this to everything stored on their server ... IIRC, they still store 3 copies of each file even with matches ... Setting up such a feature will probably require a database of MD5 (or similar) checksums, filesize plus possibly file names (to reduce search time, at the cost of maybe not recognizing a match, but at the gain of reducing theoretical overlaps in MD5 checksum) ... probably want to define a minimum file size below which you wouldn't want to do all the searching/md5'ing due to little gain by matches ... plus if a file is used as a reference, the server must maintain it as long as there are at least 1 references to it ... if it is changed or deleted on the server it originated at, the old version must be kept intact for the other backups ... some work, but doable ... But then, at the moment a nice frontend for M$-users might be of a slightly higher importance ... -gg From boxbackup at fluffy.co.uk Thu Sep 23 07:22:24 2004 From: boxbackup at fluffy.co.uk (Per Thomsen) Date: Wed, 22 Sep 2004 23:22:24 -0700 Subject: [Box Backup] BoxBackup as a Windows Service (Cygwin) In-Reply-To: <11D58968-073B-11D9-BBE6-000A95AFF7F8@fluffy.co.uk> References: <1095174490.16504.61.camel@doublebarrel.doom.und> <4736B67A-0669-11D9-A58D-000A95AFF7F8@fluffy.co.uk> <1095268079.21362.49.camel@doublebarrel.doom.und> <11D58968-073B-11D9-BBE6-000A95AFF7F8@fluffy.co.uk> Message-ID: <41526BA0.7010001@reedtz.com> On 9/15/04 10:17 AM, Ben Summers wrote: > > On 15 Sep 2004, at 18:07, Pascal Lalonde wrote: > >> On Tue, 2004-09-14 at 12:15, Ben Summers wrote: >> >>> On 14 Sep 2004, at 16:08, Pascal Lalonde wrote: >>> >>> To avoid it forking, do >>> >>> bbackupd -c /path/to/config/file SINGLEPROCESS >>> >>> You must have an absolute reference to the config file, and everything >>> be in that order. >> >> I don't know if I'm doing something wrong, but: >> /usr/local/bin/bbackupd -c /etc/box/bbackupd.conf SINGLEPROCESS >> >> starts a forked bbackupd, just as "bbackupd" would. >> I've tried this on OpenBSD 3.5, with Boxbackup 0.07PLUS3. > > > Ah, you're right. That should be > > /usr/local/bin/bbackupd /etc/box/bbackupd.conf SINGLEPROCESS > > This is only really a debug thing, don't consider this actual > documentation for it or anything! I tried this, but was unable to get it to start a service, using the SINGLEPROCESS thingy. Has anyone been successful in getting bbackupd to run at boot time, as a service, or otherwise. Or even at login time. I tried adding it to the 'Run' part of the Windows registry as Ben suggested, but I couldn't get that to work either. I would be willing to donate some money to help get a production (or at least Beta) grade bbackupd running on win32. Anyone else in the same boat? My user base is largely on Windows machines. Thanks, Per -- Per Reedtz Thomsen | The Reedtz Corporation | F: 209 883 4119 V: 209 883 4102 | pthomsen at reedtz.com | C: 415 425 4025 GPG ID: 1209784F | Yahoo! Chat: pthomsen | AIM: pthomsen From boxbackup at fluffy.co.uk Thu Sep 23 09:29:58 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 23 Sep 2004 09:29:58 +0100 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft0.01) In-Reply-To: <1095869945607.richard_eigenmann@compuserve.com> References: <1095869945607.richard_eigenmann@compuserve.com> Message-ID: On 22 Sep 2004, at 17:19, richard_eigenmann wrote: > >> I would have thought that in such a department you would have a means >> of recreating the workstations very quickly, perhaps using Ghost for >> Windows or a custom installer for UNIX. In which case you'd only need >> to back up the data. > > It depends on the sophistication of the administrator, the uniformity > of the > configuration, the size of the organisation and then the degree of > protection > you require. With some users you simply can't rely on them not storing > their > important data in applicaion directories, temporary locations, on the > root of > the disk or anywhere else that makes no sense. > > I always advocate creating a single directory that contains all the > important > data (in subfolders) so that it can be easily backed up and moved to > the next > machine. However, people tend to agree with me and then continue in > their old > habits until the inevitable happens. In this case, the administrator should set up the machine so that they can only save data in places where they're supposed to be. > > Of course there is also the issue of mounted lan drives that a user > might > want to have backed up with boxbackup. Ignoring the inonde / mount > point > issue for a moment we could find situations where the configuration > would > back up some Lan directories from multiple client machines as well as > from > the server backup too. I think setting up the client side backup is not > something you want to leave to users! No. And it should run on a server, not clients. > > On the other hand I wonder where and why organisations would want to > back up > workstations at all. At my workplace I don't believe that my > workstation is > backed up. If it blows they clone a new one and I'm back on the lan. > Local > data will be gone. That is communicated and if you don't follow the > guidelines then that's your problem. That is a very sensible policy. > Laptops, however, can validly have data > locally. Laptops can also be tricky with drivers and nonstandard > configurations. If it's not a large organisation that can mandate the > use of > only one single standardised type of Laptop then backing up the OS > with it's > specific configuration probably makes a lot of sense. In which case you'd probably want to get the entire OS just to make sure. Ben From boxbackup at fluffy.co.uk Thu Sep 23 09:35:46 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 23 Sep 2004 09:35:46 +0100 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft0.01) In-Reply-To: <4152647A.1020909@regio.net> References: <1095857233132.richard_eigenmann@compuserve.com> <4152647A.1020909@regio.net> Message-ID: <90D910C5-0D3B-11D9-B2B3-000A95AFF7F8@fluffy.co.uk> On 23 Sep 2004, at 06:51, Garry Glendown wrote: > richard_eigenmann wrote: >> If we implemented such a kind of "backup recognition" algorythm this >> could >> speed up backups of remote laptops as perhaps the documents the >> laptop user >> has been working on have already been backed up from users back at >> the base. >> I imagine this sort of feature could save somewhere between 0.5 and 5 >> GB per >> workstation that is doing a full backup. This could be significant to >> the >> scalability of the boxbackup. >> Of course this sort of thing probably would lead to massive >> redevelopment of >> code and should only be undertaken if there is a very strong demand. >> I for >> one don't need it. > > Your suggestion might have been derived from what some companies > already sell as backup solutions ... e.g., InterXion, a large hoster, > is selling exactly this feature set ... they say they have all mayor > Windows version and many M$ apps "on file" and if the version found on > the client machine matches the one on file, it is only stored as > reference ... they even do this for other files, dynamically extending > this to everything stored on their server ... IIRC, they still store 3 > copies of each file even with matches ... > > Setting up such a feature will probably require a database of MD5 (or > similar) checksums, filesize plus possibly file names (to reduce > search time, at the cost of maybe not recognizing a match, but at the > gain of reducing theoretical overlaps in MD5 checksum) ... probably > want to define a minimum file size below which you wouldn't want to do > all the searching/md5'ing due to little gain by matches ... plus if a > file is used as a reference, the server must maintain it as long as > there are at least 1 references to it ... if it is changed or deleted > on the server it originated at, the old version must be kept intact > for the other backups ... some work, but doable ... I'll add it to my list of features for the future. > > But then, at the moment a nice frontend for M$-users might be of a > slightly higher importance ... Yes, there does seem to be a lot of demand for a Win32 version! Ben From boxbackup at fluffy.co.uk Thu Sep 23 11:35:06 2004 From: boxbackup at fluffy.co.uk (Sitkei Attila) Date: Thu, 23 Sep 2004 12:35:06 +0200 Subject: [Box Backup] BoxBackup as a Windows Service (Cygwin) In-Reply-To: <41526BA0.7010001@reedtz.com> References: <1095174490.16504.61.camel@doublebarrel.doom.und> <4736B67A-0669-11D9-A58D-000A95AFF7F8@fluffy.co.uk> <1095268079.21362.49.camel@doublebarrel.doom.und> <11D58968-073B-11D9-BBE6-000A95AFF7F8@fluffy.co.uk> <41526BA0.7010001@reedtz.com> Message-ID: <4152A6DA.9050703@pmihiv.hu> This is a multi-part message in MIME format. --------------090304080203000909060802 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Per Thomsen wrote: > I tried this, but was unable to get it to start a service, using the > SINGLEPROCESS thingy. > > Has anyone been successful in getting bbackupd to run at boot time, as > a service, or otherwise. Or even at login time. I tried adding it to > the 'Run' part of the Windows registry as Ben suggested, but I > couldn't get that to work either. > > I would be willing to donate some money to help get a production (or > at least Beta) grade bbackupd running on win32. Anyone else in the > same boat? My user base is largely on Windows machines. > Yeah, http://www.electrasoft.com/srvany/srvany.htm shows how to create a 'service' using srvany.exe utility. Done that and works without the SINGLEPROCESS option. For more details and custom usage see the attached .reg file. Yours --tef --------------090304080203000909060802 Content-Type: text/plain; name="bbackup.reg" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="bbackup.reg" REGEDIT4 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bbackup] "Type"=dword:00000010 "Start"=dword:00000002 "ErrorControl"=dword:00000001 "ImagePath"=hex(2):63,3a,5c,77,69,6e,6e,74,5c,73,79,73,74,65,6d,33,32,5c,73,72,\ 76,61,6e,79,2e,65,78,65,00 "DisplayName"="bbackup" "ObjectName"="PMIDOM\\teflon" "Description"="Ben Summers' backup and mirroring tool (client)" [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bbackup\Parameters] "Application"="c:\\bbackup\\bbackupd.exe" "AppDirectory"="c:\\bbackup" "AppParameters"="bbackupd.conf" [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bbackup\Security] "Security"=hex:01,00,14,80,a0,00,00,00,ac,00,00,00,14,00,00,00,30,00,00,00,02,\ 00,1c,00,01,00,00,00,02,80,14,00,ff,01,0f,00,01,01,00,00,00,00,00,01,00,00,\ 00,00,02,00,70,00,04,00,00,00,00,00,18,00,fd,01,02,00,01,01,00,00,00,00,00,\ 05,12,00,00,00,00,00,00,00,00,00,1c,00,ff,01,0f,00,01,02,00,00,00,00,00,05,\ 20,00,00,00,20,02,00,00,00,00,00,00,00,00,18,00,8d,01,02,00,01,01,00,00,00,\ 00,00,05,0b,00,00,00,20,02,00,00,00,00,1c,00,fd,01,02,00,01,02,00,00,00,00,\ 00,05,20,00,00,00,23,02,00,00,00,00,00,00,01,01,00,00,00,00,00,05,12,00,00,\ 00,01,01,00,00,00,00,00,05,12,00,00,00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\bbackup\Enum] "0"="Root\\LEGACY_BBACKUP\\0000" "Count"=dword:00000001 "NextInstance"=dword:00000001 --------------090304080203000909060802-- From boxbackup at fluffy.co.uk Thu Sep 23 12:22:33 2004 From: boxbackup at fluffy.co.uk (=?ISO-8859-15?Q?J=E9r=F4me_Schell?=) Date: Thu, 23 Sep 2004 13:22:33 +0200 Subject: [Box Backup] 0.07PLUS4 test Message-ID: <4152B1F9.4040704@myreseau.org> Hi all, Just put my hands in v0.07PLUS4 of boxbackup. Seems to work OK. As usual I made Debian package for it. If you're interrested just point you sources.list to : For Woody/stable : deb http://debian.taonix.org stable main For testing/Sarge : deb http://debian.taonix.org/ testing main Note that for woody you will need to install libssl 0.9.7 to be able to install boxbackup, see for example on : deb http://people.debian.org/~hmh/woody/ hmh/misc/ Then you can : apt-get install boxbackup-client or apt-get install boxbackup-server or apt-get install boxbackup-utils (just the certificate authority management script) I can just see a little problem with the generated bbackupd.conf file. In the comment for BackupLocations, it is said to put a regexp like that: ExcludeFilesRegex = *.(mp3|MP3)$ But this is not working on my Debian system (I get a (1/31) - Common BadRegularExpression error) I had to replace it by : ExcludeFilesRegex = \.(mp3|MP3)$ to make it work. Can anybody confirm this is Ok? Thanks Ben for your great work ;) -- J?r?me From boxbackup at fluffy.co.uk Thu Sep 23 12:37:41 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 23 Sep 2004 12:37:41 +0100 Subject: [Box Backup] 0.08 released Message-ID: I've just released version 0.08 (at last!). Grab it from http://www.fluffy.co.uk/boxbackup/ As usual, thanks for all your help in making this the best release yet! Ben From boxbackup at fluffy.co.uk Thu Sep 23 12:48:43 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 23 Sep 2004 12:48:43 +0100 Subject: [Box Backup] 0.07PLUS4 test In-Reply-To: <4152B1F9.4040704@myreseau.org> References: <4152B1F9.4040704@myreseau.org> Message-ID: <850AA47E-0D56-11D9-B2B3-000A95AFF7F8@fluffy.co.uk> On 23 Sep 2004, at 12:22, J=E9r=F4me Schell wrote: > Hi all, > > Just put my hands in v0.07PLUS4 of boxbackup. > Seems to work OK. > As usual I made Debian package for it. If you're interrested just=20 > point you sources.list to : > > For Woody/stable : > deb http://debian.taonix.org stable main > > For testing/Sarge : > deb http://debian.taonix.org/ testing main > > Note that for woody you will need to install libssl 0.9.7 to be able=20= > to install boxbackup, see for example on : > deb http://people.debian.org/~hmh/woody/ hmh/misc/ > > Then you can : > apt-get install boxbackup-client > or > apt-get install boxbackup-server > or > apt-get install boxbackup-utils (just the certificate authority=20 > management script) Thanks! But I'm afraid I've now disabled access to the 0.07PLUS4 release as=20 0.08 is out. Sorry about that. > > I can just see a little problem with the generated bbackupd.conf file.=20= > In the comment for BackupLocations, it is said to put a regexp like=20 > that: > ExcludeFilesRegex =3D *.(mp3|MP3)$ > > But this is not working on my Debian system (I get a (1/31) - Common=20= > BadRegularExpression error) > I had to replace it by : > ExcludeFilesRegex =3D \.(mp3|MP3)$ > to make it work. > Can anybody confirm this is Ok? The format of the regular expressions will depend on what is provided=20 by your platform, so yes, it could certainly be different on Debian. > > Thanks Ben for your great work ;) My pleasure. Ben From boxbackup at fluffy.co.uk Thu Sep 23 13:28:07 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Thu, 23 Sep 2004 13:28:07 +0100 Subject: [Box Backup] 0.08 released In-Reply-To: References: Message-ID: <1095942487.7028.14.camel@avenin.ebourne.me.uk> On Thu, 2004-09-23 at 12:37, Ben Summers wrote: > I've just released version 0.08 (at last!). Grab it from > > http://www.fluffy.co.uk/boxbackup/ > > As usual, thanks for all your help in making this the best release yet! > Fedora Core 2 RPMs built: http://www.ebourne.me.uk/rpms/boxbackup-0.08-1.src.rpm http://www.ebourne.me.uk/rpms/boxbackup-client-0.08-1.i386.rpm http://www.ebourne.me.uk/rpms/boxbackup-server-0.08-1.i386.rpm http://www.ebourne.me.uk/rpms/boxbackup-client-0.08-1.x86_64.rpm http://www.ebourne.me.uk/rpms/boxbackup-server-0.08-1.x86_64.rpm Both archs installed, working, and make test was clean. This is a temporary location - please upload them to SourceForge. Cheers, Martin. From boxbackup at fluffy.co.uk Thu Sep 23 13:27:30 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Thu, 23 Sep 2004 13:27:30 +0100 Subject: [Box Backup] 0.08 released In-Reply-To: References: Message-ID: <1095942449.7028.11.camel@avenin.ebourne.me.uk> On Thu, 2004-09-23 at 12:37, Ben Summers wrote: > I've just released version 0.08 (at last!). Grab it from > > http://www.fluffy.co.uk/boxbackup/ > > As usual, thanks for all your help in making this the best release yet! > Fedora Core 2 RPMs built: http://www.ebourne.me.uk/rpms/boxbackup-0.08-1.src.rpm http://www.ebourne.me.uk/rpms/boxbackup-client-0.08-1.i386.rpm http://www.ebourne.me.uk/rpms/boxbackup-server-0.08-1.i386.rpm http://www.ebourne.me.uk/rpms/boxbackup-client-0.08-1.x86_64.rpm http://www.ebourne.me.uk/rpms/boxbackup-server-0.08-1.x86_64.rpm Both archs installed, working, and make test was clean. This is a temporary location - please upload them to SourceForge. Cheers, Martin. From boxbackup at fluffy.co.uk Thu Sep 23 16:26:37 2004 From: boxbackup at fluffy.co.uk (=?ISO-8859-1?Q?J=E9r=F4me_Schell?=) Date: Thu, 23 Sep 2004 17:26:37 +0200 Subject: [Box Backup] 0.07PLUS4 test In-Reply-To: <850AA47E-0D56-11D9-B2B3-000A95AFF7F8@fluffy.co.uk> References: <4152B1F9.4040704@myreseau.org> <850AA47E-0D56-11D9-B2B3-000A95AFF7F8@fluffy.co.uk> Message-ID: <4152EB2D.9070005@myreseau.org> Ben Summers a ?crit : > Thanks! > > But I'm afraid I've now disabled access to the 0.07PLUS4 release as 0.08 > is out. Sorry about that. > Doh ! Bad luck for me :) Just giving a try at this new version. -- J?r?me From boxbackup at fluffy.co.uk Fri Sep 24 03:23:41 2004 From: boxbackup at fluffy.co.uk (Chris Smith) Date: Fri, 24 Sep 2004 14:23:41 +1200 Subject: [Box Backup] BoxBackup 0,08 SUSE Linux packages Message-ID: <4153852D.2050509@nothingbutnet.co.nz> Good Afternoon, I've spent this morning and a few hours yesterday altering Martin's .spec file and creating specific init scripts for SUSE Linux. These RPM's were built and compiled on SUSE Linux 9.0 but included is the spec file and custom init scripts which should work from SUSE 8.0 to SUSE 9.1, you just need to build them yourself! These packages are available from: http://nothingbutnet.co.nz/boxbackup-suse/9.0/boxbackup-client-0.08-1.i586.rpm http://nothingbutnet.co.nz/boxbackup-suse/9.0/boxbackup-server-0.08-1.i586.rpm The SUSE init scripts are available here: http://nothingbutnet.co.nz/boxbackup-suse/bbackupd http://nothingbutnet.co.nz/boxbackup-suse/bbstored The modified .spec is available here: http://nothingbutnet.co.nz/boxbackup-suse/boxbackup.spec The Source RPM including all of the above is available here: http://nothingbutnet.co.nz/boxbackup-suse/boxbackup-0.08-1.src.rpm KNOWN BUGS: The install process doesn't create the "box" user for the server daemon. Generates error but can be ignored as it runs as root instead. So it doesn't break things, just makes it possibly insecure. Martin: Inside your .spec you reference a variable "%{backup_dir}" which wasn't set causing the adduser command to fail (quietly, but user isn't added). What is that variable supposed to be? These RPM's have had a little bit of testing, but nothing extensive. Please report all SUSE packaging related bugs to me :) Kind regards, Chris Smith. From boxbackup at fluffy.co.uk Fri Sep 24 07:51:00 2004 From: boxbackup at fluffy.co.uk (=?ISO-8859-1?Q?J=E9r=F4me_Schell?=) Date: Fri, 24 Sep 2004 08:51:00 +0200 Subject: [Box Backup] 0.08 released In-Reply-To: References: Message-ID: <4153C3D4.60901@myreseau.org> Ben Summers a ?crit : > > I've just released version 0.08 (at last!). Grab it from > > http://www.fluffy.co.uk/boxbackup/ > > As usual, thanks for all your help in making this the best release yet! > Working ok. Debian packages are available here: http://debian.taonix.org/dists/ sources.list entries: For Woody/stable : deb http://debian.taonix.org stable main For testing/Sarge : deb http://debian.taonix.org/ testing main Bye -- J?r?me From boxbackup at fluffy.co.uk Fri Sep 24 00:07:07 2004 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 24 Sep 2004 00:07:07 +0100 (BST) Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft 0.01) In-Reply-To: Message-ID: Hi all, One more feature request on these lines: how about the ability to "port" clients from one server to another? For example, it would be useful to be able to copy the user's data between backup servers, and when the copy is complete, start sending "redirect" packets to the client whenever they contact their old server, so that they will connect to the new one instead. This would make load balancing and fault tolerance much easier, reducing the need for a customised load balancing proxy between client and server (which currently does not exist?) Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Fri Sep 24 00:16:59 2004 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 24 Sep 2004 00:16:59 +0100 (BST) Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft 0.01) In-Reply-To: <4365AF74-0CA7-11D9-9734-000A95AFF7F8@fluffy.co.uk> Message-ID: Hi all, > Account numbers are rather essential in the design, so will have to > stay there. There needs to be a translation layer at some point. That's a shame. I think that symbolic names would be much more client- and admin-friendly than account numbers. But what do I know? Might I enquire about the rationale for using account numbers in the first place? > That's a good point. How housekeeping will know which account to trim > when the group goes over usage is an interesting question though. Another reason for the client to handle deletion of files, rather than the server :-) I would say that the client knows best regarding its own data. > If you have multiple independent backup servers, then running one > daemon on each could be considered beneficial. I would definitely agree with this. I would even go so far as to suggest it for a "backup farm" configuration, as opposed to one big central server with lots of NFS mounts or NAS. Having a central database would also aid fault tolerance across the farm, by allowing the use of replication mechanisms to ensure that each backup server has its own copy of all the critical client details, making the system more scalable and robust. Would raidfile/RAID1 be a good way to distribute the backed-up data itself between backup servers? Say every server was part of a pair, and backed up its client data both to a local disk and to an NFS volume mounted from its pair. Then if one server goes down, you can instruct (or DNAT) clients into logging into the other server, at which point their files are still accessible through the local copy on that server? Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Fri Sep 24 00:33:35 2004 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 24 Sep 2004 00:33:35 +0100 (BST) Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft0.01) In-Reply-To: <90D910C5-0D3B-11D9-B2B3-000A95AFF7F8@fluffy.co.uk> Message-ID: Hi all, I second the call for duplicate file detection :-) > > But then, at the moment a nice frontend for M$-users might be of a > > slightly higher importance ... > > Yes, there does seem to be a lot of demand for a Win32 version! I'm about to start working on this. Feature requests please? My current list stands at: No server version (use Linux dammit :-) For the client bbackupd: * registry backups * open file backups * support for drive letters * backup Windows attributes and file permissions * native non-Cygwin compile * operation as a service (done already in 0.0.8?) * socket/pipe interface to GUI administrator New GUI admin tool: * start/stop/restart bbackupd * monitor status of bbackupd, using socket interface * overview status page (for dummies) * detailed status page: ** scanning/sending/waiting ** status of each file (last backup date, reason for not backing up if any) * GUI view of server (disk space used, files stored) * restore and compare GUI * configuration GUI (what to back up, lazy/standard, min and max age) I'm planning a GUI tool written in wxWindows, so it would run both on Windows and Linux, and hopefully MacOS X if/when that becomes a supported platform for boxbackup? I could definitely use some help on this project, if anyone is up for it :-) Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Fri Sep 24 08:10:56 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 24 Sep 2004 08:10:56 +0100 Subject: [Box Backup] Server redundancy and backup servers Message-ID: I have been discussing redundancy for servers off-list, and have come up with some plans and preliminary design notes. A copy is below for your comments. Ben Design objectives * Failure means the server cannot be contacted by the client. If a server can contacted by another server but not the client, then that server must still be considered down. * No central server. The objective above means server choice must be made by the client. * A misbehaving client should not cause the stores to lose syncronisation. * Assume that all servers have the same amount of disc space, and identical disc configuration. * Allow choice of primary and secondary on a per account basis. * Any connection can be dropped at any time, and the stores should be in a workable, if non-optimal, state. * As simple as possible. Avoid keeping large amounts of state about the accounts on another server. Server groups. * The client store marker is defined to change at the end of every sync (if and only if data changed) from the client. The client sync marker should increase each time the store is updated. This allows the server groups to determine easily if they are in sync, and which is the latest version. * Stores are grouped. Each server is a peer within the group. * On login, the server returns a list of all other servers in the group. The client records this list on disc. * When the client needs to obtain a connection to a store, it uses the following algorithm: Let S = last server successfully connected Let P = primary server Do { Attempt to connect to S If(S == P and S is not connected) { Pause; Try connecting to P again. } } While(S is not connected and not all servers have been tried) If(S is not connected) { Pause Start process again } Let CSM_S = client store marker from S If(S != P) { Attempt to connect to P again, but with a short timeout this time If(P is connected) { Let CSM_P = client store marker from P If(CSM == expected client store marker) { Disconnect S S = P } else { Disconnect P } } } This algorithm ensures that the client prefers to connect to the primary server, but will keep talking to the secondary server for as long as it's available and is at a later state than the primary store. (This gives time for the data to be transferred from the secondary to the primary and avoid repeat uploads of data.) * Servers within a group use the fast sync protocol to update accounts on a regular basis. Observations * The severs are simply peers. The primary server for an account is chosen merely by configuring the client. * If the servers simply use best efforts to keep each other up to date, the client will automatically choose the best server to contact. * Using the existing methods of handling unexpected changes to the client store marker, it doesn't matter whether a server is out of date or not. The existing code handles this occurance perfectly. * The servers do not need to check whether other servers are down. This fact is actually irrelevant, because it's the client's view of upness which is important. Accounts The accounts database must be identical on each machine. bbstoreaccounts will need to push changes to all servers. It will probably be necessary to change the account database, and store the limits within the database rather than in the stores of data. This is desirable anyway. Note: If another server is down, it won't be possible to update the account database. Alternatively, servers could update each other with changes to the accounts database on a lazy basis. This might cause issues with housekeeping unnecessarily deleting files which have to be replaced. Fast sync protocol. * Compare client store markers. End if they are the same. Otherwise, the server with the greater number becomes the source, and the lesser the target. * Zero client store marker on target * Send stream of deleted (by housekeeping) object IDs from source to target. Target deletes the objects immediately. * Send stream of object ID + hash of directories on source server to the target. * For each directory on the target server which doesn't exist, or doesn't have the right hash... - check objects exist, and transfer them - write directory, only if all the objects are correct - check for patches. Attempt to transfer by patch if new version exists * Each server records the client store marker it expects on the remote server. If that marker is not as expected, then the contents of the directories are checked as well, sending MD5 hashes across. This allows recovery from partial syncs. [This should probably be optimised if for when there's an empty store at one end.] * When an object is uploaded, the "last object ID used" value for that account should be kept within the acceptable range to allow recovery when syncing with the client. * Write new client store marker on target If a client connects during a fast sync, then that fast sync will be aborted to give the client the lock on the account. Optimised fast sync. It's undesirable for the fast sync to check every directory when it doesn't have to. During sync with a client a store * Keeps a list of changed directories by writing to disc (and flushing) every time a directory is saved back to disc. * Keep patches from previous versions to send to remote store * Connect after backup to remote stores, use fast sync to send changes over. This will allow short-cuts to be taken when syncing, and changes sent by patch. The cache of patches will need to be managed, deleting them when they are transferred to a peer or are too old. Housekeeping Deleted objects need to be kept in sync too. Housekeeping takes place indepedently on each server. Since housekeeping is a determinisitic process, this should not delete different files on different servers. A list of deleted objects is kept on each server during the housekeeping process. In the unlikely event that a server deletes an object that the source server doesn't, this object will be retrieved in the next fast sync. This is unlikely to happen because clients only add data. Typically, housekeeping on non-primary servers will never delete an object in that account. From boxbackup at fluffy.co.uk Fri Sep 24 08:54:26 2004 From: boxbackup at fluffy.co.uk (richard_eigenmann) Date: Fri, 24 Sep 2004 09:54:26 +0200 Subject: [Box Backup] Upgrading the client Message-ID: <1096012466829.richard_eigenmann@compuserve.com> I've thought up a scenario where the use of inodes to identify the file changes on the client is very problematic: moving your data and it's backup to a new computer or upgrading your disk. Examples: you bought a new laptop and move your data and boxbackup to the new machine. Or you run out of space and buy a new disk and move your data directories to that new drive. In this sort of situation boxbackup will detect a file change and will back up everything to the store. Clearly this is undesirable. It gets highly problematic in the situation where you have huge amounts of data say 30GB and your backup location is remote say via ADSL. If you had 200 kBit/s uplink that would translate to 416 hours of solid transfer or 17 days. With the current software I would recommend taking the computer to the backup server and connecting it directly to that Lan and making the big backup there. Regards, Richard From boxbackup at fluffy.co.uk Fri Sep 24 09:02:47 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 24 Sep 2004 09:02:47 +0100 Subject: [Box Backup] Upgrading the client In-Reply-To: <1096012466829.richard_eigenmann@compuserve.com> References: <1096012466829.richard_eigenmann@compuserve.com> Message-ID: <1F33F007-0E00-11D9-BF43-000A95AFF7F8@fluffy.co.uk> On 24 Sep 2004, at 08:54, richard_eigenmann wrote: > I've thought up a scenario where the use of inodes to identify the file > changes on the client is very problematic: moving your data and it's > backup > to a new computer or upgrading your disk. > > Examples: you bought a new laptop and move your data and boxbackup to > the new > machine. Or you run out of space and buy a new disk and move your data > directories to that new drive. > > In this sort of situation boxbackup will detect a file change and will > back > up everything to the store. Clearly this is undesirable. > > It gets highly problematic in the situation where you have huge > amounts of > data say 30GB and your backup location is remote say via ADSL. If you > had 200 > kBit/s uplink that would translate to 416 hours of solid transfer or > 17 days. > > With the current software I would recommend taking the computer to the > backup > server and connecting it directly to that Lan and making the big backup > there. Inode numbers are only used to work out file and directory renames. Timestamps are used for actual change detection. In fact, inode numbers simply wouldn't work for change detection, because they don't change when you modify the file. In your scenario, in the worst case new attributes will be uploaded for every file. This doesn't take very long. However, if you're transferring a filesystem, you'll probably use something which maintains inode numbers and file attributes properly, so it won't be a problem at all. And anyway, you wouldn't be renaming or moving files around, all it takes is one scan to get the inode database up to date on the new machine. (Yes, I have thought carefully about how Box Backup will be used and abused. I believe the design is robust and tolerant. It's certainly let me get away with things when bad stuff has happened.) Ben From boxbackup at fluffy.co.uk Fri Sep 24 09:15:49 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 24 Sep 2004 09:15:49 +0100 Subject: [Box Backup] Account migration (was: BoxBackup Server Side Management Specs (Draft 0.01)) In-Reply-To: References: Message-ID: On 24 Sep 2004, at 00:07, Chris Wilson wrote: > Hi all, > > One more feature request on these lines: how about the ability to > "port" > clients from one server to another? Yes, a rather handy thing to have. (I've been calling this "account migration".) At the moment, it can be accomplished by simply moving the data from one server to the other, and then adjusting the account databases and client configuration. So it's possible, just not neat. But nothing a script couldn't handle with ease. > > For example, it would be useful to be able to copy the user's data > between > backup servers, and when the copy is complete, start sending "redirect" > packets to the client whenever they contact their old server, so that > they > will connect to the new one instead. > > This would make load balancing and fault tolerance much easier, > reducing > the need for a customised load balancing proxy between client and > server > (which currently does not exist?) You've just preempted my posting of some design notes about precisely this. With only slight modifications, it could also do account migration. Ben From boxbackup at fluffy.co.uk Fri Sep 24 09:59:18 2004 From: boxbackup at fluffy.co.uk (richard_eigenmann) Date: Fri, 24 Sep 2004 10:59:18 +0200 Subject: [Box Backup] Upgrading the client Message-ID: <1096016358236.richard_eigenmann@compuserve.com> I'm not sure I follow. If I look at my own upgrade behaviour over the last few years: Typically I start off with a reasonably working machine which is ageing. In the mean time the Linux Distro has been upgraded. Hans Reiser has introduced a new version of the filesystem. The Crypto-thingy from SuSE insists on longer passwords, the disks are too small and data subdirectories are strewn around spare partitions. The new box arrives. I stick an up to date SuSE distro on it, partition the drive in a way that the fragmented subdirs all fit on one partition again, set up encryption for the data partition if it's a laptop. Then I copy all the data over to the new box and keep it in sync with Unison whilst I run the new box in parallel for a few weeks until it actually talks to the USB camera, CD Burner, Scanner, Printer and anything else that you never thought could not work (it has come a long way I must say). When everything is stable the old machine gets decomissioned and starts a new life with one of my parents. By copying the data instead of dd-ing the filesystem across I get whatever benefits the new filesystem has to offer. Without checksumming the files boxbackup has no way to recognize that the file on the new machine was already backed up. Is my upgrade method entirely uncommon? Would an administrator not do the same on an important data server in a network? What am I missing? Regards, Richard From boxbackup at fluffy.co.uk Fri Sep 24 10:31:37 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 24 Sep 2004 10:31:37 +0100 Subject: [Box Backup] Upgrading the client In-Reply-To: <1096016358236.richard_eigenmann@compuserve.com> References: <1096016358236.richard_eigenmann@compuserve.com> Message-ID: <88ADB771-0E0C-11D9-8477-000A95AFF7F8@fluffy.co.uk> On 24 Sep 2004, at 09:59, richard_eigenmann wrote: > I'm not sure I follow. > > If I look at my own upgrade behaviour over the last few years: > Typically I > start off with a reasonably working machine which is ageing. In the > mean time > the Linux Distro has been upgraded. Hans Reiser has introduced a new > version > of the filesystem. The Crypto-thingy from SuSE insists on longer > passwords, > the disks are too small and data subdirectories are strewn around spare > partitions. The new box arrives. I stick an up to date SuSE distro on > it, > partition the drive in a way that the fragmented subdirs all fit on one > partition again, set up encryption for the data partition if it's a > laptop. > Then I copy all the data over to the new box and keep it in sync with > Unison > whilst I run the new box in parallel for a few weeks until it actually > talks > to the USB camera, CD Burner, Scanner, Printer and anything else that > you > never thought could not work (it has come a long way I must say). When > everything is stable the old machine gets decomissioned and starts a > new life > with one of my parents. > > By copying the data instead of dd-ing the filesystem across I get > whatever > benefits the new filesystem has to offer. Without checksumming the > files > boxbackup has no way to recognize that the file on the new machine was > already backed up. > > Is my upgrade method entirely uncommon? Would an administrator not do > the > same on an important data server in a network? What am I missing? Maybe it's just the BSD world which has nice stable filesystems and things which aren't randomly tweeked all the time. Anyway, as long as you copy your files using a method which preserves the timestamps (most methods in fact), everything will be fine. And even if you don't manage to preserve them, if the files are bigger than the threshold set in the config file (64k by default) then the checksums will be checked anyway, avoiding an unnecessary upload. Ben From boxbackup at fluffy.co.uk Fri Sep 24 11:53:28 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 24 Sep 2004 11:53:28 +0100 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft 0.01) In-Reply-To: References: Message-ID: On 24 Sep 2004, at 00:16, Chris Wilson wrote: > Hi all, > >> Account numbers are rather essential in the design, so will have to >> stay there. There needs to be a translation layer at some point. > > That's a shame. I think that symbolic names would be much more client- > and > admin-friendly than account numbers. But what do I know? Might I > enquire > about the rationale for using account numbers in the first place? My intention was for this to be used within a larger system which would automatically provision backup accounts as part of a larger account. As such, numbers are much easier to allocate and use. The current code references accounts internally and in the protocol using 32 bit integers. Changing this would be rather annoying. > >> That's a good point. How housekeeping will know which account to trim >> when the group goes over usage is an interesting question though. > > Another reason for the client to handle deletion of files, rather than > the > server :-) I would say that the client knows best regarding its own > data. Yes, but the server knows more about how space is to be managed on the server. My approach to this will be to implement the backup set feature, which will allow the client to tell the server about it's policy. For the client itself to delete files seems a bit inefficient, as it would all have to be done over a network connection. > >> If you have multiple independent backup servers, then running one >> daemon on each could be considered beneficial. > > I would definitely agree with this. I would even go so far as to > suggest it for a "backup farm" configuration, as opposed to one big > central server with lots of NFS mounts or NAS. This was my intention, for there to be lots of servers with each effectively independent. This gives more fault tolerance, and distributes the processor load over many machines. > > Having a central database would also aid fault tolerance across the > farm, > by allowing the use of replication mechanisms to ensure that each > backup > server has its own copy of all the critical client details, making the > system more scalable and robust. Yes, there would be local copies of the database. > > Would raidfile/RAID1 be a good way to distribute the backed-up data > itself > between backup servers? Say every server was part of a pair, and > backed up > its client data both to a local disk and to an NFS volume mounted from > its > pair. Then if one server goes down, you can instruct (or DNAT) clients > into logging into the other server, at which point their files are > still > accessible through the local copy on that server? See other email on this subject. Ben From boxbackup at fluffy.co.uk Fri Sep 24 12:03:51 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 24 Sep 2004 12:03:51 +0100 Subject: [Box Backup] BoxBackup 0,08 SUSE Linux packages In-Reply-To: <4153852D.2050509@nothingbutnet.co.nz> References: <4153852D.2050509@nothingbutnet.co.nz> Message-ID: <6B2B1A1C-0E19-11D9-8477-000A95AFF7F8@fluffy.co.uk> On 24 Sep 2004, at 03:23, Chris Smith wrote: > I've spent this morning and a few hours yesterday altering Martin's > .spec file and creating specific init scripts for SUSE Linux. These > RPM's were built and compiled on SUSE Linux 9.0 but included is the > spec file and custom init scripts which should work from SUSE 8.0 to > SUSE 9.1, you just need to build them yourself! Not knowing much about Linux packaging, maybe someone here could enlighten me. What's the easiest way of including both RedHat and SuSE style RPM specifications in a single source distribution? Or maybe, what's the easiest way for me to support the various Linux variations without having to produce many different distribution files each time I do a release? Thanks, Ben From boxbackup at fluffy.co.uk Fri Sep 24 12:16:54 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 24 Sep 2004 12:16:54 +0100 Subject: [Box Backup] Win32 port (was: BoxBackup Server Side Management Specs (Draft0.01)) In-Reply-To: References: Message-ID: <3DDF53FA-0E1B-11D9-8477-000A95AFF7F8@fluffy.co.uk> On 24 Sep 2004, at 00:33, Chris Wilson wrote: > Hi all, > > I second the call for duplicate file detection :-) Unfortunately there's only one of me. > >>> But then, at the moment a nice frontend for M$-users might be of a >>> slightly higher importance ... >> >> Yes, there does seem to be a lot of demand for a Win32 version! > > I'm about to start working on this. Before you start changing code, it will be much easier if you work on a checkout of my CVS repository. The released code is run through a script which does various modifications like adding the license at the top of each file, so patches from released code are not as easy to integrate as they might be. I also have some extended notes on porting to Win32, if anyone wants a copy. (I have already sent one to Chris.) > Feature requests please? My current > list stands at: > > No server version (use Linux dammit :-) Surely you mean OpenBSD? Seriously, a Win32 server is unlikely as it depends on the behaviour of the UNIX filesystem. On Windows you can't delete or move another file over a file which is currently open. > > For the client bbackupd: > > * registry backups Dump the registry to a file and back that up? > * open file backups > * support for drive letters I would have thought that the existing mechanism of backup locations will handle this reasonably well. > * backup Windows attributes and file permissions > * native non-Cygwin compile > * operation as a service (done already in 0.0.8?) No. The client is essentially unchanged in 0.08. > * socket/pipe interface to GUI administrator > > New GUI admin tool: > > * start/stop/restart bbackupd > * monitor status of bbackupd, using socket interface > * overview status page (for dummies) > * detailed status page: > ** scanning/sending/waiting > ** status of each file (last backup date, reason for not backing up if > any) > * GUI view of server (disk space used, files stored) > * restore and compare GUI > * configuration GUI (what to back up, lazy/standard, min and max age) > > I'm planning a GUI tool written in wxWindows, so it would run both on > Windows and Linux, and hopefully MacOS X if/when that becomes a > supported > platform for boxbackup? It works on Mac OS X, it just doesn't back up the resource fork of the files. I will only really consider it supported when it does this, otherwise it might surprise users when files are restored. This requires multiple streams to be implemented, which currently there's no neat way of doing. Now that WinXP SP2 uses multiple streams to store file security information, it's probably required for Win32 as well. And I might as well support sparse files while I'm at it. Not a difficult task, but then, not exactly trivial either. But I should do it, mainly so I can backup my main work computer with my own software, instead of just all my servers. > > I could definitely use some help on this project, if anyone is up for > it > :-) A few other people have a bit of a start on this. I will try and put everyone in touch with each other. Ben From boxbackup at fluffy.co.uk Fri Sep 24 12:21:01 2004 From: boxbackup at fluffy.co.uk (Garry Glendown) Date: Fri, 24 Sep 2004 13:21:01 +0200 Subject: [Box Backup] Win32 port In-Reply-To: <3DDF53FA-0E1B-11D9-8477-000A95AFF7F8@fluffy.co.uk> References: <3DDF53FA-0E1B-11D9-8477-000A95AFF7F8@fluffy.co.uk> Message-ID: <4154031D.9030908@regio.net> Ben Summers wrote: > > On 24 Sep 2004, at 00:33, Chris Wilson wrote: > >> Hi all, >> >> I second the call for duplicate file detection :-) > > > Unfortunately there's only one of me. Also, there is one basic problem that just popped up in my mind ... BB uses encryption for storing the files - encryption done by the client IIRC ... therefore, the server has no way of telling which files are the same, and even if, a second client couldn't restore its file from the first client's storage, as he doesn't have the private key to extract the data ... so strike that feature off the list ... (dunno how the other companies handle it, but it would seem to me they do not handle security the same way BB does ...) -gg From boxbackup at fluffy.co.uk Fri Sep 24 12:34:58 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 24 Sep 2004 12:34:58 +0100 Subject: [Box Backup] Common file recognition (was: Win32 port) In-Reply-To: <4154031D.9030908@regio.net> References: <3DDF53FA-0E1B-11D9-8477-000A95AFF7F8@fluffy.co.uk> <4154031D.9030908@regio.net> Message-ID: On 24 Sep 2004, at 12:21, Garry Glendown wrote: > Ben Summers wrote: >> On 24 Sep 2004, at 00:33, Chris Wilson wrote: >>> Hi all, >>> >>> I second the call for duplicate file detection :-) >> Unfortunately there's only one of me. > > Also, there is one basic problem that just popped up in my mind ... BB > uses encryption for storing the files - encryption done by the client > IIRC ... therefore, the server has no way of telling which files are > the same, and even if, a second client couldn't restore its file from > the first client's storage, as he doesn't have the private key to > extract the data ... so strike that feature off the list ... (dunno > how the other companies handle it, but it would seem to me they do not > handle security the same way BB does ...) > I expect this would be done in a slightly different way to the usual backups. The client would query a list of these "common files" given filename, hash and length, and the server would tell it which it had. This would mean * You would have to upload all the common files to the server, unencrypted * The client would lose out on some confidentiality of data -- the server would know the client used a common file, and would know lots of filenames of files you were backing up. The latter issue could be minimized by either using hashes of filenames, or restricting the search for common files to specified directories (like C:\Windows or /usr). So it's possible to do, but it is a different mechanism. I don't think the commercial companies see encryption as something as important as I do. Ben From boxbackup at fluffy.co.uk Fri Sep 24 12:36:07 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Fri, 24 Sep 2004 12:36:07 +0100 Subject: [Box Backup] BoxBackup 0,08 SUSE Linux packages In-Reply-To: <6B2B1A1C-0E19-11D9-8477-000A95AFF7F8@fluffy.co.uk> References: <4153852D.2050509@nothingbutnet.co.nz> <6B2B1A1C-0E19-11D9-8477-000A95AFF7F8@fluffy.co.uk> Message-ID: <1096025767.7028.18.camel@avenin.ebourne.me.uk> On Fri, 2004-09-24 at 12:03, Ben Summers wrote: > Not knowing much about Linux packaging, maybe someone here could > enlighten me. What's the easiest way of including both RedHat and SuSE > style RPM specifications in a single source distribution? > > Or maybe, what's the easiest way for me to support the various Linux > variations without having to produce many different distribution files > each time I do a release? I'm looking at this now. I'll have one patch to apply to current 0.08 release which will fix it all up to work automatically on both. Cheers, Martin. From boxbackup at fluffy.co.uk Fri Sep 24 12:41:41 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Fri, 24 Sep 2004 12:41:41 +0100 Subject: [Box Backup] Win32 port (was: BoxBackup Server Side Management Specs (Draft0.01)) In-Reply-To: <3DDF53FA-0E1B-11D9-8477-000A95AFF7F8@fluffy.co.uk> References: <3DDF53FA-0E1B-11D9-8477-000A95AFF7F8@fluffy.co.uk> Message-ID: <1096026100.7028.22.camel@avenin.ebourne.me.uk> On Fri, 2004-09-24 at 12:16, Ben Summers wrote: > Before you start changing code, it will be much easier if you work on a > checkout of my CVS repository. Is this available? I didn't spot it on the website. I did mean to ask about this earlier. > > * registry backups > > Dump the registry to a file and back that up? Surely the registry is already in a file(s), somewhere in the windows directory. Cheers, Martin. From boxbackup at fluffy.co.uk Fri Sep 24 12:48:06 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 24 Sep 2004 12:48:06 +0100 Subject: [Box Backup] Win32 port (was: BoxBackup Server Side Management Specs (Draft0.01)) In-Reply-To: <1096026100.7028.22.camel@avenin.ebourne.me.uk> References: <3DDF53FA-0E1B-11D9-8477-000A95AFF7F8@fluffy.co.uk> <1096026100.7028.22.camel@avenin.ebourne.me.uk> Message-ID: <9999BAD2-0E1F-11D9-8477-000A95AFF7F8@fluffy.co.uk> On 24 Sep 2004, at 12:41, Martin Ebourne wrote: > On Fri, 2004-09-24 at 12:16, Ben Summers wrote: >> Before you start changing code, it will be much easier if you work on >> a >> checkout of my CVS repository. > > Is this available? I didn't spot it on the website. I did mean to ask > about this earlier. I will knock one up for you shortly. > >>> * registry backups >> >> Dump the registry to a file and back that up? > > Surely the registry is already in a file(s), somewhere in the windows > directory. Yes, but it's not terribly accessible, it's locked all over the place! Ben From boxbackup at fluffy.co.uk Fri Sep 24 13:09:45 2004 From: boxbackup at fluffy.co.uk (=?iso-8859-2?Q?Mitja_Mu=BEeni=E8?=) Date: Fri, 24 Sep 2004 14:09:45 +0200 Subject: [Box Backup] Win32 port (was: BoxBackup Server Side Management Specs (Draft0.01)) In-Reply-To: <3DDF53FA-0E1B-11D9-8477-000A95AFF7F8@fluffy.co.uk> Message-ID: <200409241207.i8OC7GVF003380@very.fluffy.co.uk> > I also have some extended notes on porting to Win32, if > anyone wants a > copy. (I have already sent one to Chris.) Can I trouble you for a copy? Best regards, Mitja From boxbackup at fluffy.co.uk Fri Sep 24 13:37:08 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Fri, 24 Sep 2004 13:37:08 +0100 Subject: [Box Backup] BoxBackup 0,08 SUSE Linux packages In-Reply-To: <4153852D.2050509@nothingbutnet.co.nz> References: <4153852D.2050509@nothingbutnet.co.nz> Message-ID: <1096029427.7028.39.camel@avenin.ebourne.me.uk> --=-cBEOfq3ONFiChLQID5i+ Content-Type: text/plain Content-Transfer-Encoding: 7bit On Fri, 2004-09-24 at 03:23, Chris Smith wrote: > I've spent this morning and a few hours yesterday altering Martin's > .spec file and creating specific init scripts for SUSE Linux. These > RPM's were built and compiled on SUSE Linux 9.0 but included is the spec > file and custom init scripts which should work from SUSE 8.0 to SUSE > 9.1, you just need to build them yourself! Cool. I've incorporated all of your changes back into the stuff I had and made the rpm spec automatically work out what distro it is building on and behave accordingly. The Grand Unified RPM Patch is included. ;) > Martin: Inside your .spec you reference a variable "%{backup_dir}" which > wasn't set causing the adduser command to fail (quietly, but user isn't > added). What is that variable supposed to be? Ah, yes, some late cleanup broke that. Doh! I've changed it to /var/empty. If SUSE doesn't have that can you pick something suitable. If you could apply this patch and check that the SUSE RPM all still works correctly then Ben can apply this as is. One question though, from my reading of http://www.suse.de/~mmj/Package-Conventions/ it would appear that there should be some calls to insserv, and that the rcbbackupd etc scripts need to be symlinked by the package. Is this correct or is that out of date? Ben, I've moved the bbstored and bbackupd startup scripts from contrib/rpm to contrib/redhat and added Chris's in contrib/suse. The patch is with -Nru so it will do that automatically or you can repository move if you prefer. (Can highly recommend Subversion to solve those sorts of problems. ;)) Updated RPMs for FC: http://www.ebourne.me.uk/rpms/boxbackup-0.08-2.src.rpm http://www.ebourne.me.uk/rpms/boxbackup-client-0.08-2.i386.rpm http://www.ebourne.me.uk/rpms/boxbackup-server-0.08-2.i386.rpm http://www.ebourne.me.uk/rpms/boxbackup-client-0.08-2.x86_64.rpm http://www.ebourne.me.uk/rpms/boxbackup-server-0.08-2.x86_64.rpm Cheers, Martin. --=-cBEOfq3ONFiChLQID5i+ Content-Disposition: attachment; filename=gurp.patch Content-Type: text/x-patch; name=gurp.patch; charset=UTF-8 Content-Transfer-Encoding: 7bit diff -urN boxbackup-0.08.orig/contrib/redhat/bbackupd boxbackup-0.08/contrib/redhat/bbackupd --- boxbackup-0.08.orig/contrib/redhat/bbackupd 1970-01-01 01:00:00.000000000 +0100 +++ boxbackup-0.08/contrib/redhat/bbackupd 2004-09-23 10:56:11.000000000 +0100 @@ -0,0 +1,83 @@ +#! /bin/bash +# +# bbackupd Start/Stop the box backup daemon. +# +# chkconfig: 345 93 07 +# description: bbackup is the client side deamon for Box Backup, a completely \ +# automatic on-line backup system +# processname: bbackupd +# config: /etc/box +# pidfile: /var/run/bbackupd.pid + +# Source function library. +. /etc/init.d/functions + +RETVAL=0 + +# See how we were called. + +prog="bbackupd" + +# Check that configuration exists. +[ -f /etc/box/$prog.conf ] || exit 0 + +start() { + echo -n $"Starting $prog: " + daemon $prog + RETVAL=$? + echo + [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog + return $RETVAL +} + +stop() { + echo -n $"Stopping $prog: " + killproc $prog + RETVAL=$? + echo + [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog + return $RETVAL +} + +rhstatus() { + status $prog +} + +restart() { + stop + start +} + +reload() { + echo -n $"Reloading $prog daemon configuration: " + killproc $prog -HUP + retval=$? + echo + return $RETVAL +} + +case "$1" in + start) + start + ;; + stop) + stop + ;; + restart) + restart + ;; + reload) + reload + ;; + status) + rhstatus + ;; + condrestart) + [ -f /var/lock/subsys/$prog ] && restart || : + ;; + *) + echo $"Usage: $0 {start|stop|status|reload|restart|condrestart}" + exit 1 +esac + +exit $? diff -urN boxbackup-0.08.orig/contrib/redhat/bbstored boxbackup-0.08/contrib/redhat/bbstored --- boxbackup-0.08.orig/contrib/redhat/bbstored 1970-01-01 01:00:00.000000000 +0100 +++ boxbackup-0.08/contrib/redhat/bbstored 2004-09-23 10:56:11.000000000 +0100 @@ -0,0 +1,83 @@ +#! /bin/bash +# +# bbstored Start/Stop the box backup daemon. +# +# chkconfig: 345 93 07 +# description: bbstore is the server side deamon for Box Backup, a completely \ +# automatic on-line backup system +# processname: bbstored +# config: /etc/box +# pidfile: /var/run/bbstored.pid + +# Source function library. +. /etc/init.d/functions + +RETVAL=0 + +# See how we were called. + +prog="bbstored" + +# Check that configuration exists. +[ -f /etc/box/$prog.conf ] || exit 0 + +start() { + echo -n $"Starting $prog: " + daemon $prog + RETVAL=$? + echo + [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog + return $RETVAL +} + +stop() { + echo -n $"Stopping $prog: " + killproc $prog + RETVAL=$? + echo + [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog + return $RETVAL +} + +rhstatus() { + status $prog +} + +restart() { + stop + start +} + +reload() { + echo -n $"Reloading $prog daemon configuration: " + killproc $prog -HUP + retval=$? + echo + return $RETVAL +} + +case "$1" in + start) + start + ;; + stop) + stop + ;; + restart) + restart + ;; + reload) + reload + ;; + status) + rhstatus + ;; + condrestart) + [ -f /var/lock/subsys/$prog ] && restart || : + ;; + *) + echo $"Usage: $0 {start|stop|status|reload|restart|condrestart}" + exit 1 +esac + +exit $? diff -urN boxbackup-0.08.orig/contrib/redhat/README.txt boxbackup-0.08/contrib/redhat/README.txt --- boxbackup-0.08.orig/contrib/redhat/README.txt 1970-01-01 01:00:00.000000000 +0100 +++ boxbackup-0.08/contrib/redhat/README.txt 2004-09-24 12:29:11.941983429 +0100 @@ -0,0 +1,7 @@ +These start scripts are for Fedora Core or RedHat Enterprise Linux. If +installed manually they should be placed in /etc/rc.d/init.d. + +They may also work for Mandrake. + +Martin Ebourne +martin at zepler.org diff -urN boxbackup-0.08.orig/contrib/rpm/bbackupd boxbackup-0.08/contrib/rpm/bbackupd --- boxbackup-0.08.orig/contrib/rpm/bbackupd 2004-09-23 10:56:11.000000000 +0100 +++ boxbackup-0.08/contrib/rpm/bbackupd 1970-01-01 01:00:00.000000000 +0100 @@ -1,83 +0,0 @@ -#! /bin/bash -# -# bbackupd Start/Stop the box backup daemon. -# -# chkconfig: 345 93 07 -# description: bbackup is the client side deamon for Box Backup, a completely \ -# automatic on-line backup system -# processname: bbackupd -# config: /etc/box -# pidfile: /var/run/bbackupd.pid - -# Source function library. -. /etc/init.d/functions - -RETVAL=0 - -# See how we were called. - -prog="bbackupd" - -# Check that configuration exists. -[ -f /etc/box/$prog.conf ] || exit 0 - -start() { - echo -n $"Starting $prog: " - daemon $prog - RETVAL=$? - echo - [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog - return $RETVAL -} - -stop() { - echo -n $"Stopping $prog: " - killproc $prog - RETVAL=$? - echo - [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog - return $RETVAL -} - -rhstatus() { - status $prog -} - -restart() { - stop - start -} - -reload() { - echo -n $"Reloading $prog daemon configuration: " - killproc $prog -HUP - retval=$? - echo - return $RETVAL -} - -case "$1" in - start) - start - ;; - stop) - stop - ;; - restart) - restart - ;; - reload) - reload - ;; - status) - rhstatus - ;; - condrestart) - [ -f /var/lock/subsys/$prog ] && restart || : - ;; - *) - echo $"Usage: $0 {start|stop|status|reload|restart|condrestart}" - exit 1 -esac - -exit $? diff -urN boxbackup-0.08.orig/contrib/rpm/bbstored boxbackup-0.08/contrib/rpm/bbstored --- boxbackup-0.08.orig/contrib/rpm/bbstored 2004-09-23 10:56:11.000000000 +0100 +++ boxbackup-0.08/contrib/rpm/bbstored 1970-01-01 01:00:00.000000000 +0100 @@ -1,83 +0,0 @@ -#! /bin/bash -# -# bbstored Start/Stop the box backup daemon. -# -# chkconfig: 345 93 07 -# description: bbstore is the server side deamon for Box Backup, a completely \ -# automatic on-line backup system -# processname: bbstored -# config: /etc/box -# pidfile: /var/run/bbstored.pid - -# Source function library. -. /etc/init.d/functions - -RETVAL=0 - -# See how we were called. - -prog="bbstored" - -# Check that configuration exists. -[ -f /etc/box/$prog.conf ] || exit 0 - -start() { - echo -n $"Starting $prog: " - daemon $prog - RETVAL=$? - echo - [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog - return $RETVAL -} - -stop() { - echo -n $"Stopping $prog: " - killproc $prog - RETVAL=$? - echo - [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog - return $RETVAL -} - -rhstatus() { - status $prog -} - -restart() { - stop - start -} - -reload() { - echo -n $"Reloading $prog daemon configuration: " - killproc $prog -HUP - retval=$? - echo - return $RETVAL -} - -case "$1" in - start) - start - ;; - stop) - stop - ;; - restart) - restart - ;; - reload) - reload - ;; - status) - rhstatus - ;; - condrestart) - [ -f /var/lock/subsys/$prog ] && restart || : - ;; - *) - echo $"Usage: $0 {start|stop|status|reload|restart|condrestart}" - exit 1 -esac - -exit $? diff -urN boxbackup-0.08.orig/contrib/rpm/boxbackup.spec boxbackup-0.08/contrib/rpm/boxbackup.spec --- boxbackup-0.08.orig/contrib/rpm/boxbackup.spec 2004-09-23 10:56:11.000000000 +0100 +++ boxbackup-0.08/contrib/rpm/boxbackup.spec 2004-09-24 13:09:14.194472302 +0100 @@ -1,12 +1,31 @@ %define bb_user_id 171 %define ident %{name}-%{version} +# Detect distribution. So far we only special-case SUSE. If you need to make +# any distro specific changes to get the package building on your system +# please email them to martin at zepler.org +#%define is_fc %(test -e %{_sysconfdir}/fedora-release && echo 1 || echo 0) +#%define is_mdk %(test -e %{_sysconfdir}/mandrake-release && echo 1 || echo 0) +#%define is_rh %(test -e %{_sysconfdir}/redhat-release && echo 1 || echo 0) +%define is_suse %(test -e %{_sysconfdir}/SuSE-release && echo 1 || echo 0) + +%if %{is_suse} +%define init_dir %{_sysconfdir}/init.d +%define dist suse +%define rc_start rc +%else +%define init_dir %{_sysconfdir}/rc.d/init.d +%define dist redhat +%define rc_start "service " +%endif + Summary: An automatic on-line backup system for UNIX. Name: boxbackup Version: 0.08 -Release: 1 +Release: 2 License: BSD Group: Applications/Archiving +Packager: Martin Ebourne URL: http://www.fluffy.co.uk/boxbackup/ Source0: %{ident}.tgz Requires: openssl >= 0.9.7a @@ -62,7 +81,7 @@ mkdir -p $RPM_BUILD_ROOT%{_docdir}/%{ident} mkdir -p $RPM_BUILD_ROOT%{_bindir} -mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/rc.d/init.d +mkdir -p $RPM_BUILD_ROOT%{init_dir} mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/box/bbackupd mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/box/bbstored mkdir -p $RPM_BUILD_ROOT%{_var}/lib/box @@ -79,7 +98,7 @@ # Client touch $RPM_BUILD_ROOT%{_sysconfdir}/box/bbackupd.conf -install contrib/rpm/bbackupd $RPM_BUILD_ROOT%{_sysconfdir}/rc.d/init.d +install -m 755 contrib/%{dist}/bbackupd $RPM_BUILD_ROOT%{init_dir} %define client_dir parcels/%{ident}-backup-client-Linux install %{client_dir}/bbackupd $RPM_BUILD_ROOT%{_bindir} install %{client_dir}/bbackupquery $RPM_BUILD_ROOT%{_bindir} @@ -89,7 +108,7 @@ # Server touch $RPM_BUILD_ROOT%{_sysconfdir}/box/bbstored.conf touch $RPM_BUILD_ROOT%{_sysconfdir}/box/raidfile.conf -install contrib/rpm/bbstored $RPM_BUILD_ROOT%{_sysconfdir}/rc.d/init.d +install -m 755 contrib/%{dist}/bbstored $RPM_BUILD_ROOT%{init_dir} %define server_dir parcels/%{ident}-backup-server-Linux install %{server_dir}/bbstored $RPM_BUILD_ROOT%{_bindir} install %{server_dir}/bbstoreaccounts $RPM_BUILD_ROOT%{_bindir} @@ -99,36 +118,37 @@ %pre server /usr/sbin/useradd -c "Box Backup" -u %{bb_user_id} \ - -s /sbin/nologin -r -d %{backup_dir} box 2> /dev/null || : + -s /sbin/nologin -r -d %{_var}/empty box 2> /dev/null || : %post client /sbin/chkconfig --add bbackupd if [ ! -f %{_sysconfdir}/box/bbackupd.conf ]; then echo "You should run the following to configure the client:" - echo "bbackupd-config /etc/box lazy /var/lib/box " + echo "bbackupd-config %{_sysconfdir}/box lazy " \ + "%{_var}/lib/box " echo "Then follow the instructions. Use this to start the client:" - echo "service bbackupd start" + echo "%{rc_start}bbackupd start" fi %post server /sbin/chkconfig --add bbstored if [ ! -f %{_sysconfdir}/box/bbstored.conf ]; then echo "You should run the following to configure the server:" - echo "raidfile-config /etc/box 2048 []" - echo "bbstored-config /etc/box" `hostname` box + echo "raidfile-config %{_sysconfdir}/box 2048 []" + echo "bbstored-config %{_sysconfdir}/box" `hostname` box echo "Then follow the instructions. Use this to start the server:" - echo "service bbstored start" + echo "%{rc_start}bbstored start" fi %preun client if [ $1 = 0 ]; then - /sbin/service bbackupd stop > /dev/null 2>&1 + %{init_dir}/bbackupd stop > /dev/null 2>&1 /sbin/chkconfig --del bbackupd fi %preun server if [ $1 = 0 ]; then - /sbin/service bbstored stop > /dev/null 2>&1 + %{init_dir}/bbstored stop > /dev/null 2>&1 /sbin/chkconfig --del bbstored fi @@ -141,7 +161,7 @@ %dir %attr(700,root,root) %{_sysconfdir}/box/bbackupd %dir %attr(755,root,root) %{_var}/lib/box %doc %{_docdir}/%{ident}/*.txt -%config %{_sysconfdir}/rc.d/init.d/bbackupd +%config %{init_dir}/bbackupd %config %ghost %{_sysconfdir}/box/bbackupd.conf %{_bindir}/bbackupd %{_bindir}/bbackupquery @@ -151,7 +171,7 @@ %files server %defattr(-,root,root,-) %dir %attr(700,box,root) %{_sysconfdir}/box/bbstored -%config %{_sysconfdir}/rc.d/init.d/bbstored +%config %{init_dir}/bbstored %config %ghost %{_sysconfdir}/box/bbstored.conf %config %ghost %{_sysconfdir}/box/raidfile.conf %{_bindir}/bbstored @@ -161,5 +181,9 @@ %{_bindir}/raidfile-config %changelog +* Fri Sep 24 2004 Martin Ebourne - 0.08-2 +- Added support for other distros +- Changes for SUSE provided by Chris Smith + * Mon Sep 16 2004 Martin Ebourne - 0.07-1 - Initial build diff -urN boxbackup-0.08.orig/contrib/rpm/README.txt boxbackup-0.08/contrib/rpm/README.txt --- boxbackup-0.08.orig/contrib/rpm/README.txt 2004-09-23 10:56:11.000000000 +0100 +++ boxbackup-0.08/contrib/rpm/README.txt 2004-09-24 12:33:18.187222972 +0100 @@ -6,11 +6,11 @@ where is the archive you downloaded of Box Backup. -This RPM should work on RedHat Enterprise, Fedora Core, Mandrake, and any -similar distributions. It has been developed and tested on Fedora Core. +This RPM should work on RedHat Enterprise, Fedora Core, Mandrake, SUSE, and +any similar distributions. It has been developed and tested on Fedora Core. -SUSE Linux will need edits to the init scripts (bbackupd and bbstored), and -may need changes to the spec files too. I don't run SUSE so can't be sure. +Changes for SUSE Linux were provided by Chris Smith +(chris.smith at nothingbutnet.co.nz). Martin Ebourne martin at zepler.org diff -urN boxbackup-0.08.orig/contrib/suse/bbackupd boxbackup-0.08/contrib/suse/bbackupd --- boxbackup-0.08.orig/contrib/suse/bbackupd 1970-01-01 01:00:00.000000000 +0100 +++ boxbackup-0.08/contrib/suse/bbackupd 2004-09-24 02:56:15.000000000 +0100 @@ -0,0 +1,100 @@ +#! /bin/sh +# +# Copyright (c)2004, Nothing But Net Limited +# +# +###################################################################### +# RELEASED AND PROVIDED TO YOU UNDER THE SAME LICENCE AS THE BOXBACKUP +# SUITE OF PROGRAMS. LICENCE MAY BE VIEWED HERE: +# +# http://www.fluffy.co.uk/boxbackup/license.html +###################################################################### +# +# /etc/init.d/bbackupd +# and its symbolic link +# /(usr/)sbin/rcbbackupd +# +### BEGIN INIT INFO +# Provides: bbackupd +# Required-Start: $named $network $local_fs $syslog +# X-UnitedLinux-Should-Start: $time ypbind sendmail +# Required-Stop: $named $network $localfs $syslog +# X-UnitedLinux-Should-Stop: $time ypbind sendmail +# Default-Start: 3 5 +# Default-Stop: 0 1 2 6 +# Short-Description: BoxBackup client side daemon +# Description: Client daemon for the BoxBackup software +# that allows you to communicate with a bbstored server. +### END INIT INFO +# + +# Check for missing binaries (stale symlinks should not happen) +BBACKUPD_BIN=/usr/bin/bbackupd +test -x $BBACKUPD_BIN || {echo "$BBACKUPD_BIN not installed"; exit 5} + +. /etc/rc.status + +# Reset status of this service +rc_reset + +case "$1" in + start) + echo -n "Starting bbackupd " + startproc $BBACKUPD_BIN + rc_status -v + ;; + + stop) + echo -n "Shutting down bbackupd " + killproc -TERM $BBACKUPD_BIN + rc_status -v + ;; + + try-restart|condrestart) + if test "$1" = "condrestart"; then + echo "${attn} Use try-restart ${done}(LSB)${attn} rather than condrestart ${warn}(RH)${norm}" + fi + $0 status + if test $? = 0; then + $0 restart + else + rc_reset # Not running is not a failure. + fi + rc_status + ;; + + restart) + $0 stop + $0 start + rc_status + ;; + + force-reload) + echo -n "Reload service bbackupd " + killproc -HUP $BBACKUPD_BIN + rc_status -v + ;; + + reload) + echo -n "Reload service bbackupd " + killproc -HUP $BBACKUPD_BIN + rc_status -v + ;; + + status) + echo -n "Checking for service bbackupd " + checkproc $BBACKUPD_BIN + rc_status -v + ;; + + probe) + test /etc/box/bbackupd.conf -nt /var/run/bbackupd.pid && echo reload + ;; + + *) + echo "Usage: $0 {start|stop|status|try-restart|restart|force-reload|reload|probe}" + exit 1 + ;; + +esac +rc_exit diff -urN boxbackup-0.08.orig/contrib/suse/bbstored boxbackup-0.08/contrib/suse/bbstored --- boxbackup-0.08.orig/contrib/suse/bbstored 1970-01-01 01:00:00.000000000 +0100 +++ boxbackup-0.08/contrib/suse/bbstored 2004-09-24 02:56:15.000000000 +0100 @@ -0,0 +1,100 @@ +#! /bin/sh +# +# Copyright (c)2004, Nothing But Net Limited +# +# +###################################################################### +# RELEASED AND PROVIDED TO YOU UNDER THE SAME LICENCE AS THE BOXBACKUP +# SUITE OF PROGRAMS. LICENCE MAY BE VIEWED HERE: +# +# http://www.fluffy.co.uk/boxbackup/license.html +###################################################################### +# +# /etc/init.d/bbackupd +# and its symbolic link +# /(usr/)sbin/rcbbackupd +# +### BEGIN INIT INFO +# Provides: bbackupd +# Required-Start: $named $network $local_fs $syslog +# X-UnitedLinux-Should-Start: $time ypbind sendmail +# Required-Stop: $named $network $localfs $syslog +# X-UnitedLinux-Should-Stop: $time ypbind sendmail +# Default-Start: 3 5 +# Default-Stop: 0 1 2 6 +# Short-Description: BoxBackup server side daemon +# Description: Client daemon for the BoxBackup software +# that allows you to communicate with a bbstored server. +### END INIT INFO +# + +# Check for missing binaries (stale symlinks should not happen) +BBACKUPD_BIN=/usr/bin/bbstored +test -x $BBACKUPD_BIN || {echo "$BBACKUPD_BIN not installed"; exit 5} + +. /etc/rc.status + +# Reset status of this service +rc_reset + +case "$1" in + start) + echo -n "Starting bbstored " + startproc $BBACKUPD_BIN + rc_status -v + ;; + + stop) + echo -n "Shutting down bstored " + killproc -TERM $BBACKUPD_BIN + rc_status -v + ;; + + try-restart|condrestart) + if test "$1" = "condrestart"; then + echo "${attn} Use try-restart ${done}(LSB)${attn} rather than condrestart ${warn}(RH)${norm}" + fi + $0 status + if test $? = 0; then + $0 restart + else + rc_reset # Not running is not a failure. + fi + rc_status + ;; + + restart) + $0 stop + $0 start + rc_status + ;; + + force-reload) + echo -n "Reload service bbstored " + killproc -HUP $BBACKUPD_BIN + rc_status -v + ;; + + reload) + echo -n "Reload service bbstored " + killproc -HUP $BBACKUPD_BIN + rc_status -v + ;; + + status) + echo -n "Checking for service bbstored " + checkproc $BBACKUPD_BIN + rc_status -v + ;; + + probe) + test /etc/box/bbstored.conf -nt /var/run/bbstored.pid && echo reload + ;; + + *) + echo "Usage: $0 {start|stop|status|try-restart|restart|force-reload|reload|probe}" + exit 1 + ;; + +esac +rc_exit diff -urN boxbackup-0.08.orig/contrib/suse/README.txt boxbackup-0.08/contrib/suse/README.txt --- boxbackup-0.08.orig/contrib/suse/README.txt 1970-01-01 01:00:00.000000000 +0100 +++ boxbackup-0.08/contrib/suse/README.txt 2004-09-24 12:31:08.965542115 +0100 @@ -0,0 +1,5 @@ +These start scripts are for SUSE Linux. If installed manually they should be +placed in /etc/init.d. + +Copyright (c)2004, Nothing But Net Limited + diff -urN boxbackup-0.08.orig/LINUX.txt boxbackup-0.08/LINUX.txt --- boxbackup-0.08.orig/LINUX.txt 2004-09-23 10:56:11.000000000 +0100 +++ boxbackup-0.08/LINUX.txt 2004-09-24 12:46:35.871776371 +0100 @@ -1,7 +1,6 @@ For instructions on building an RPM of Box Backup, see the contrib/rpm -directory. This is primarily for RedHat style systems, but notes are provided -on what needs to be modified for SUSE. +directory. Requirements: --=-cBEOfq3ONFiChLQID5i+-- From boxbackup at fluffy.co.uk Fri Sep 24 14:01:52 2004 From: boxbackup at fluffy.co.uk (richard_eigenmann) Date: Fri, 24 Sep 2004 15:01:52 +0200 Subject: [Box Backup] Common file recognition (was: Win32 port) Message-ID: <1096030912873.richard_eigenmann@compuserve.com> > I expect this would be done in a slightly different way to the usual > backups. The client would query a list of these "common files" given > filename, hash and length, and the server would tell it which it had. > > This would mean > > * You would have to upload all the common files to the server, > unencrypted > > * The client would lose out on some confidentiality of data -- the > server would know the client used a common file, and would know lots of > filenames of files you were backing up. > > The latter issue could be minimized by either using hashes of > filenames, or restricting the search for common files to specified > directories (like C:\Windows or /usr). > > So it's possible to do, but it is a different mechanism. I don't think > the commercial companies see encryption as something as important as I > do. We could have a pool of "common files" that can be set up as a baseline (perhaps backup a freshly set up Windows box). Boxbackup could perhaps even be configured to use some sort of regexp filtering to identify files to store in the common pool (*.exe, *.dll, *.cab) so that the pool doesn't get stale. Strong encryption is great and I think Ben has done a fantastic job. People's requirements may vary of course. Encrypting the traffic is certainly a best practice. There could be situations where you have secure network connections (VPN) where doing a second level of encryption is simply a waste of effort. Likewise having encrypted stores on the storage vault is a great feature and allows you to put your backups on machines that you would not normally let near your data. But there will be situations where the backup server is being administered by people who know what they are doing and where the encryption stands in the way of storage efficiency. Perhaps there are even legal issues that might prevent the use of encryption. Key escrow management issues would also not be an issue if the backup is unencrypted. I think we have more valuable enhancements on the to do list. But if we keep the idea of "shared backup files" in the back of our (Ben's) mind the development is less likely to go down a road where this becomes impossible to accomplish at a future date. Regards, Richard From boxbackup at fluffy.co.uk Fri Sep 24 14:08:44 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Fri, 24 Sep 2004 09:08:44 -0400 (EDT) Subject: [Box Backup] Server redundancy and backup servers In-Reply-To: References: Message-ID: I think it would be a good idea to have the client choose the initial, and failover server from the list in a random fashion, and then prefer the last server it talked to. So the algorithm would use the same code whenever it was unable to contact the last used server, and in the case of the first run, there would be no last used server. This way you don't have to configure the client to start with a particular server, it just works out. Rick On Fri, 24 Sep 2004, Ben Summers wrote: > > I have been discussing redundancy for servers off-list, and have come > up with some plans and preliminary design notes. A copy is below for > your comments. > > Ben > > > > Design objectives > > * Failure means the server cannot be contacted by the client. If a > server can contacted by another server but not the client, then that > server must still be considered down. > > * No central server. The objective above means server choice must be > made by the client. > > * A misbehaving client should not cause the stores to lose > syncronisation. > > * Assume that all servers have the same amount of disc space, and > identical disc configuration. > > * Allow choice of primary and secondary on a per account basis. > > * Any connection can be dropped at any time, and the stores should be > in a workable, if non-optimal, state. > > * As simple as possible. Avoid keeping large amounts of state about the > accounts on another server. > > > Server groups. > > * The client store marker is defined to change at the end of every sync > (if and only if data changed) from the client. The client sync marker > should increase each time the store is updated. This allows the server > groups to determine easily if they are in sync, and which is the latest > version. > > * Stores are grouped. Each server is a peer within the group. > > * On login, the server returns a list of all other servers in the > group. The client records this list on disc. > > * When the client needs to obtain a connection to a store, it uses the > following algorithm: > > Let S = last server successfully connected > Let P = primary server > Do > { > Attempt to connect to S > If(S == P and S is not connected) > { > Pause; > Try connecting to P again. > } > > } While(S is not connected and not all servers have been tried) > > If(S is not connected) > { > Pause > Start process again > } > > Let CSM_S = client store marker from S > > If(S != P) > { > Attempt to connect to P again, but with a short timeout this time > If(P is connected) > { > Let CSM_P = client store marker from P > If(CSM == expected client store marker) > { > Disconnect S > S = P > } > else > { > Disconnect P > } > } > } > > This algorithm ensures that the client prefers to connect to the > primary server, but will keep talking to the secondary server for as > long as it's available and is at a later state than the primary store. > (This gives time for the data to be transferred from the secondary to > the primary and avoid repeat uploads of data.) > > * Servers within a group use the fast sync protocol to update accounts > on a regular basis. > > > Observations > > * The severs are simply peers. The primary server for an account is > chosen merely by configuring the client. > > * If the servers simply use best efforts to keep each other up to date, > the client will automatically choose the best server to contact. > > * Using the existing methods of handling unexpected changes to the > client store marker, it doesn't matter whether a server is out of date > or not. The existing code handles this occurance perfectly. > > * The servers do not need to check whether other servers are down. This > fact is actually irrelevant, because it's the client's view of upness > which is important. > > > Accounts > > The accounts database must be identical on each machine. > bbstoreaccounts will need to push changes to all servers. It will > probably be necessary to change the account database, and store the > limits within the database rather than in the stores of data. This is > desirable anyway. > > Note: If another server is down, it won't be possible to update the > account database. > > Alternatively, servers could update each other with changes to the > accounts database on a lazy basis. This might cause issues with > housekeeping unnecessarily deleting files which have to be replaced. > > > Fast sync protocol. > > * Compare client store markers. End if they are the same. Otherwise, > the server with the greater number becomes the source, and the lesser > the target. > > * Zero client store marker on target > > * Send stream of deleted (by housekeeping) object IDs from source to > target. Target deletes the objects immediately. > > * Send stream of object ID + hash of directories on source server to > the target. > > * For each directory on the target server which doesn't exist, or > doesn't have the right hash... > - check objects exist, and transfer them > - write directory, only if all the objects are correct > - check for patches. Attempt to transfer by patch if new version exists > > * Each server records the client store marker it expects on the remote > server. If that marker is not as expected, then the contents of the > directories are checked as well, sending MD5 hashes across. This allows > recovery from partial syncs. [This should probably be optimised if for > when there's an empty store at one end.] > > * When an object is uploaded, the "last object ID used" value for that > account should be kept within the acceptable range to allow recovery > when syncing with the client. > > * Write new client store marker on target > > If a client connects during a fast sync, then that fast sync will be > aborted to give the client the lock on the account. > > > > > Optimised fast sync. > > It's undesirable for the fast sync to check every directory when it > doesn't have to. During sync with a client a store > > * Keeps a list of changed directories by writing to disc (and flushing) > every time a directory is saved back to disc. > > * Keep patches from previous versions to send to remote store > > * Connect after backup to remote stores, use fast sync to send changes > over. > > This will allow short-cuts to be taken when syncing, and changes sent > by patch. > > The cache of patches will need to be managed, deleting them when they > are transferred to a peer or are too old. > > > Housekeeping > > Deleted objects need to be kept in sync too. Housekeeping takes place > indepedently on each server. Since housekeeping is a determinisitic > process, this should not delete different files on different servers. > > A list of deleted objects is kept on each server during the > housekeeping process. > > In the unlikely event that a server deletes an object that the source > server doesn't, this object will be retrieved in the next fast sync. > This is unlikely to happen because clients only add data. > > Typically, housekeeping on non-primary servers will never delete an > object in that account. > > > > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Fri Sep 24 14:39:01 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 24 Sep 2004 14:39:01 +0100 Subject: [Box Backup] Server redundancy and backup servers In-Reply-To: References: Message-ID: <180D9AA6-0E2F-11D9-8477-000A95AFF7F8@fluffy.co.uk> On 24 Sep 2004, at 14:08, rprice at freeshell.org wrote: > I think it would be a good idea to have the client choose the initial, > and > failover server from the list in a random fashion, and then prefer the > last server it talked to. > > So the algorithm would use the same code whenever it was unable to > contact > the last used server, and in the case of the first run, there would be > no > last used server. > > This way you don't have to configure the client to start with a > particular > server, it just works out. What if you have your primary servers on a really fast connection, but with the backup ones at a less desirable location? You might want to keep control over where the client will connect to make sure it always talks to the primary location. The list of servers has to be got from somewhere, so you've got to configure it with something. So if you want to load-balance it as you suggest, just pick a random server when you configure it in the first place. The other problem is what happens in a failure situation. If there are four servers, and three go down, then the backup clients will all talk to the remaining one for ever more. Or have I missed the point somewhere? I've tried to devise a scheme where the administrator has control over what happens. Ben From boxbackup at fluffy.co.uk Fri Sep 24 18:14:38 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Fri, 24 Sep 2004 13:14:38 -0400 (EDT) Subject: [Box Backup] Server redundancy and backup servers In-Reply-To: <180D9AA6-0E2F-11D9-8477-000A95AFF7F8@fluffy.co.uk> References: <180D9AA6-0E2F-11D9-8477-000A95AFF7F8@fluffy.co.uk> Message-ID: No I don't think you missed my point. I was trying to make it work more automagically so that the administrator could let it take care of itself more. I suppose you could (make things more complicated) add an affinity value so that the faster servers would get used first. The only way to overcome the problem with all the servers except one failing would be to have the machines switch primary backups every X times. I suppose the client could query the server to see how busy it is, and then switch, but this all makes things more complicated... So perhaps leaving well enough alone is best. Rick On Fri, 24 Sep 2004, Ben Summers wrote: > > On 24 Sep 2004, at 14:08, rprice at freeshell.org wrote: > > > I think it would be a good idea to have the client choose the initial, > > and > > failover server from the list in a random fashion, and then prefer the > > last server it talked to. > > > > So the algorithm would use the same code whenever it was unable to > > contact > > the last used server, and in the case of the first run, there would be > > no > > last used server. > > > > This way you don't have to configure the client to start with a > > particular > > server, it just works out. > > What if you have your primary servers on a really fast connection, but > with the backup ones at a less desirable location? You might want to > keep control over where the client will connect to make sure it always > talks to the primary location. > > The list of servers has to be got from somewhere, so you've got to > configure it with something. So if you want to load-balance it as you > suggest, just pick a random server when you configure it in the first > place. > > The other problem is what happens in a failure situation. If there are > four servers, and three go down, then the backup clients will all talk > to the remaining one for ever more. > > Or have I missed the point somewhere? I've tried to devise a scheme > where the administrator has control over what happens. > > Ben > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Sat Sep 25 00:46:44 2004 From: boxbackup at fluffy.co.uk (Chris Smith) Date: Sat, 25 Sep 2004 11:46:44 +1200 Subject: [Box Backup] BoxBackup 0,08 SUSE Linux packages In-Reply-To: <1096029427.7028.39.camel@avenin.ebourne.me.uk> References: <4153852D.2050509@nothingbutnet.co.nz> <1096029427.7028.39.camel@avenin.ebourne.me.uk> Message-ID: <200409251146.44340.chris.smith@nothingbutnet.co.nz> On Sat, 25 Sep 2004 00:37, Martin Ebourne wrote: > Cool. I've incorporated all of your changes back into the stuff I had > and made the rpm spec automatically work out what distro it is building > on and behave accordingly. The Grand Unified RPM Patch is included. ;) Ah, thats awesome :) > Ah, yes, some late cleanup broke that. Doh! I've changed it to > /var/empty. If SUSE doesn't have that can you pick something suitable. Ok, a directory that is just empty? I'm sure I can handle that. > If you could apply this patch and check that the SUSE RPM all still > works correctly then Ben can apply this as is. Ok, I will. I'll need to sort out some remote access to my dev machine at work, as that is the only SUSE system I have access to at the moment. I'm not at work again until Thursday. > One question though, from > my reading of http://www.suse.de/~mmj/Package-Conventions/ it would > appear that there should be some calls to insserv, and that the > rcbbackupd etc scripts need to be symlinked by the package. Is this > correct or is that out of date? I read that document but not really at length. But you're right, it looks like I've done some things improperly (it works, but it may not upgrade properly or something). Would you be able to fix them up? > (Can highly recommend Subversion to solve > those sorts of problems. ;)) Most definitely! :) Regards, Chris. From boxbackup at fluffy.co.uk Sun Sep 26 06:55:00 2004 From: boxbackup at fluffy.co.uk (Per Thomsen) Date: Sat, 25 Sep 2004 22:55:00 -0700 Subject: [Box Backup] Upgrading from 0.07 -> 0.08 Message-ID: <415659B4.3000301@reedtz.com> I'm planning on upgrading to 0.08, but I'd like to understand the dependencies: I know that I won't be able to upgrade the server first, and then the clients, but can I go the other way (0.08/0.07 clients against 0.07 server)? Or do I have to stop everything, and then upgrade? Thanks in advance, Per -- Per Reedtz Thomsen | The Reedtz Corporation | F: 209 883 4119 V: 209 883 4102 | pthomsen at reedtz.com | C: 415 425 4025 GPG ID: 1209784F | Yahoo! Chat: pthomsen | AIM: pthomsen From boxbackup at fluffy.co.uk Sun Sep 26 06:58:59 2004 From: boxbackup at fluffy.co.uk (Per Thomsen) Date: Sat, 25 Sep 2004 22:58:59 -0700 Subject: [Box Backup] BoxBackup Server Side Management Specs (Draft 0.01) In-Reply-To: <41510D4B.2030601@reedtz.com> References: <41510D4B.2030601@reedtz.com> Message-ID: <41565AA3.8020404@reedtz.com> On 9/21/04 10:27 PM, Per Thomsen wrote: > All, > Here are the things I have come up with from a user's perspective as > requirements/specifications for server-side management. > [snip] > 3. Client Monitoring. > [snip again] > > The heart beat packet contains the following information: > > - Host identifier (name and/or account number) Thought of adding client version number here. Would be useful to know what version each of your clients are running. > - uptime (ie. how long has bbackupd been running on this host) > - timestamp of last sync (when was the last file uploaded) > - Number of bytes synced since last heart beat message > - Number of bytes restored since last heart beat message > - any significant errors that have occured since last heart beat. > - ? > [snip] > Thanks, Per -- Per Reedtz Thomsen | The Reedtz Corporation | F: 209 883 4119 V: 209 883 4102 | pthomsen at reedtz.com | C: 415 425 4025 GPG ID: 1209784F | Yahoo! Chat: pthomsen | AIM: pthomsen From boxbackup at fluffy.co.uk Sun Sep 26 08:44:45 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Sun, 26 Sep 2004 08:44:45 +0100 Subject: [Box Backup] Upgrading from 0.07 -> 0.08 In-Reply-To: <415659B4.3000301@reedtz.com> References: <415659B4.3000301@reedtz.com> Message-ID: On 26 Sep 2004, at 06:55, Per Thomsen wrote: > I'm planning on upgrading to 0.08, but I'd like to understand the > dependencies: > > I know that I won't be able to upgrade the server first, and then the > clients, but can I go the other way (0.08/0.07 clients against 0.07 > server)? > > Or do I have to stop everything, and then upgrade? Yes, I'm afraid so. A 0.07 client will just give an error connecting to a 0.08 server, so stop the server, upgrade, start it again. Then go round doing the same for every client -- nothing bad will happen if a 0.07 client connects to a 0.08 server, it's just that it won't manage to log in and do anything. Ben From boxbackup at fluffy.co.uk Mon Sep 27 01:19:58 2004 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Mon, 27 Sep 2004 01:19:58 +0100 (BST) Subject: [Box Backup] Account migration (was: BoxBackup Server Side Management Specs (Draft 0.01)) In-Reply-To: Message-ID: Hi Ben, > You've just preempted my posting of some design notes about precisely > this. With only slight modifications, it could also do account > migration. Thanks, I'm glad you've been thinking about it already. I don't actually need it yet, but I look forward to seeing it implemented, and if it isn't yet done when I need it, perhaps I can lend a hand in implementing it then? Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Mon Sep 27 11:56:37 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 27 Sep 2004 11:56:37 +0100 Subject: [Box Backup] Server redundancy and backup servers In-Reply-To: References: Message-ID: On 24 Sep 2004, at 08:10, Ben Summers wrote: > > I have been discussing redundancy for servers off-list, and have come > up with some plans and preliminary design notes. A copy is below for > your comments. > Thanks for the comments so far. It has just occurred to me that using the built in software RAID, a limited form of redundant servers could be created. Someone suggested this on the list a while back, and I've only just realised the implications. All you need are three identical servers. On each server, compose the RAID file sets from the local hard drives and the two hard drives from the other servers (mount the discs using NFS or something.) Run the bbstored daemon on each, and use round-robin DNS with a low TTL to send clients to different machines. It should then "just work". If any machine goes down, then the software RAID will kick in and no-one will notice, apart from the administrator who will notice the log messages. The changes required are: * Add communications between bbstored servers so that a client can log in even if another server is housekeeping that account. * Account database syncing between servers. * Raid file disc set restoration tools needs to be written (which is still currently lacking -- right now you have to move the existing files away in case they're needed, then blank every account and wait until the clients have uploaded everything again.) * Efficiency: write the raidfile daemon to offload RAID work, and write the temporary files to the local filesystem only. The advantage over the previous plan is that most of the work is already done -- none of the above is a particularly significant amount of effort. The disadvantage is that it limits clusters to three machines which are connected to each other with fast network connections. However, it is a rather neat and simple solution. Your thoughts, as always, are welcomed. Ben From boxbackup at fluffy.co.uk Tue Sep 28 20:57:49 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Tue, 28 Sep 2004 20:57:49 +0100 Subject: [Box Backup] BoxBackup 0,08 SUSE Linux packages In-Reply-To: <200409251146.44340.chris.smith@nothingbutnet.co.nz> References: <4153852D.2050509@nothingbutnet.co.nz> <1096029427.7028.39.camel@avenin.ebourne.me.uk> <200409251146.44340.chris.smith@nothingbutnet.co.nz> Message-ID: <1096401469.2223.4.camel@avenin.ebourne.me.uk> On Sat, 2004-09-25 at 00:46, Chris Smith wrote: > On Sat, 25 Sep 2004 00:37, Martin Ebourne wrote: > > One question though, from > > my reading of http://www.suse.de/~mmj/Package-Conventions/ it would > > appear that there should be some calls to insserv, and that the > > rcbbackupd etc scripts need to be symlinked by the package. Is this > > correct or is that out of date? > > I read that document but not really at length. But you're right, it looks like > I've done some things improperly (it works, but it may not upgrade properly > or something). Would you be able to fix them up? I don't have access to any SUSE machine, so although I could make the mods as per the document I'd have no way of testing them and more importantly, fixing it when it doesn't work. I suggest you download a source RPM for one of the base SUSE packages that does something similar and see how they did it. That's likely to be right then. Cheers, Martin. From boxbackup at fluffy.co.uk Wed Sep 29 06:47:47 2004 From: boxbackup at fluffy.co.uk (Per Thomsen) Date: Tue, 28 Sep 2004 22:47:47 -0700 Subject: [Box Backup] endian.h on Linux Message-ID: <415A4C83.7070405@reedtz.com> Hi, Just compiled 0.08 on Linux, and I get warnings (things seem to work), from the compiler that including /usr/include/asm/bytecode.h is not cool, and that endian.h should be used instead. Don't know if that's possible, but I just wanted to make sure that you knew about the warning. I'm running FC2 2.6.6 on x86. Thanks, Per -- Per Reedtz Thomsen | The Reedtz Corporation | F: 209 883 4119 V: 209 883 4102 | pthomsen at reedtz.com | C: 415 425 4025 GPG ID: 1209784F | Yahoo! Chat: pthomsen | AIM: pthomsen From boxbackup at fluffy.co.uk Wed Sep 29 08:38:33 2004 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Wed, 29 Sep 2004 08:38:33 +0100 Subject: [Box Backup] endian.h on Linux In-Reply-To: <415A4C83.7070405@reedtz.com> References: <415A4C83.7070405@reedtz.com> Message-ID: <1096443513.9352.3.camel@avenin.ebourne.me.uk> On Wed, 2004-09-29 at 06:47, Per Thomsen wrote: > Hi, > Just compiled 0.08 on Linux, and I get warnings (things seem to work), > from the compiler that including /usr/include/asm/bytecode.h is not > cool, and that endian.h should be used instead. Yes, that happens on x86 with the 2.6 kernel for me too. It does not happen with x86_64 even though it's the same setup. At a glance it looked like endian.h didn't contain the functions used, though I didn't actually try it. I was planning to get around to that sometime. As you say, it doesn't prevent it from actually working currently. Cheers, Martin. From boxbackup at fluffy.co.uk Wed Sep 29 08:45:59 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 29 Sep 2004 08:45:59 +0100 Subject: [Box Backup] endian.h on Linux In-Reply-To: <1096443513.9352.3.camel@avenin.ebourne.me.uk> References: <415A4C83.7070405@reedtz.com> <1096443513.9352.3.camel@avenin.ebourne.me.uk> Message-ID: <9AB6C4DC-11EB-11D9-A457-000A95AFF7F8@fluffy.co.uk> On 29 Sep 2004, at 08:38, Martin Ebourne wrote: > On Wed, 2004-09-29 at 06:47, Per Thomsen wrote: >> Hi, >> Just compiled 0.08 on Linux, and I get warnings (things seem to work), >> from the compiler that including /usr/include/asm/bytecode.h is not >> cool, and that endian.h should be used instead. > > Yes, that happens on x86 with the 2.6 kernel for me too. It does not > happen with x86_64 even though it's the same setup. > > At a glance it looked like endian.h didn't contain the functions used, > though I didn't actually try it. I was planning to get around to that > sometime. As you say, it doesn't prevent it from actually working > currently. You're right. If the lovely Linux developers would provide the 64 bit byte swapping function in a header I'm supposed to use, then I will gladly use it instead. Until then, I will be using the kernel header. Ben From boxbackup at fluffy.co.uk Thu Sep 30 20:13:16 2004 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Thu, 30 Sep 2004 20:13:16 +0100 (BST) Subject: [Box Backup] Win32 port (was: BoxBackup Server Side Management Specs (Draft0.01)) In-Reply-To: <3DDF53FA-0E1B-11D9-8477-000A95AFF7F8@fluffy.co.uk> Message-ID: Hi Ben, > > I second the call for duplicate file detection :-) > > Unfortunately there's only one of me. Sorry, I'd be happy to help out when I've made some progress on the GUI and Windows port and I've got a bit more time. > Before you start changing code, it will be much easier if you work on a > checkout of my CVS repository. The released code is run through a > script which does various modifications like adding the license at the > top of each file, so patches from released code are not as easy to > integrate as they might be. So far I have not needed to change any code, and I hope that I can maintain the Windows GUI semi-independently, only relying on some header files from Box. However, there does seem to be a fair amount of auto-generated and platform-dependent stuff that might require work on the build system to integrate properly. I will tackle that when I have something worth integrating :-) > I also have some extended notes on porting to Win32, if anyone wants a > copy. (I have already sent one to Chris.) Those notes are so useful that perhaps they should be in the notes directory? > > No server version (use Linux dammit :-) > > Surely you mean OpenBSD? Sorry, I should have said *nix (or !win32 :-) > Seriously, a Win32 server is unlikely as it depends on the behaviour of > the UNIX filesystem. On Windows you can't delete or move another file > over a file which is currently open. Well, I suspect I'll have to dive into device drivers to make open file support work, so maybe I can fix this at the same time, or make it easier for the next person. Personally I have no desire to run a server on Windows. > Dump the registry to a file and back that up? If properly integrated into the client itself, seems like a great way to go. I want to be really sure that it's written correctly, or that I get notification if it's not. > > * support for drive letters > > I would have thought that the existing mechanism of backup locations > will handle this reasonably well. What mechanism is that? There seems to be an implicitly assumed special object at the root of the filesystem with the name "/" (unless I misunderstand). That doesn't seem to map very nicely to Windows drive letters, especially when it comes to removable devices. Perhaps I could have a virtual root directory with "/c:/", "/d:/", etc. underneath it. > This requires multiple streams to be implemented, which currently > there's no neat way of doing. Now that WinXP SP2 uses multiple streams > to store file security information, it's probably required for Win32 as > well. And I might as well support sparse files while I'm at it. I don't plan to back up security information directly as a binary stream, because I suspect that it would be useless if restored onto a different machine or installation. Perhaps I could use such a stream for a textual, machine-independent representation of all the security information associated with a file, and it might be the best way to support that kind of detailed security data. Do you have concrete plans to implement this at some point? > A few other people have a bit of a start on this. I will try and put > everyone in touch with each other. Please do, I would be very interested to hear from anyone else who is working on Win32 support or a GUI, especially if they wish to collaborate, but at least to avoid duplication of effort. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Thu Sep 30 22:15:09 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 30 Sep 2004 22:15:09 +0100 Subject: [Box Backup] Win32 port (was: BoxBackup Server Side Management Specs (Draft0.01)) In-Reply-To: References: Message-ID: On 30 Sep 2004, at 20:13, Chris Wilson wrote: > Hi Ben, > >>> I second the call for duplicate file detection :-) >> >> Unfortunately there's only one of me. > > Sorry, I'd be happy to help out when I've made some progress on the GUI > and Windows port and I've got a bit more time. Great. > >> Before you start changing code, it will be much easier if you work on >> a >> checkout of my CVS repository. The released code is run through a >> script which does various modifications like adding the license at the >> top of each file, so patches from released code are not as easy to >> integrate as they might be. > > So far I have not needed to change any code, and I hope that I can > maintain the Windows GUI semi-independently, only relying on some > header > files from Box. Sounds good. Are you talking to the server directly, or just writing .conf files and using bbackupquery? > However, there does seem to be a fair amount of > auto-generated and platform-dependent stuff that might require work on > the > build system to integrate properly. I will tackle that when I have > something worth integrating :-) Yes, the build system might need a little work. > >> I also have some extended notes on porting to Win32, if anyone wants a >> copy. (I have already sent one to Chris.) > > Those notes are so useful that perhaps they should be in the notes > directory? I shall add them in. [snip] >> Seriously, a Win32 server is unlikely as it depends on the behaviour >> of >> the UNIX filesystem. On Windows you can't delete or move another file >> over a file which is currently open. > > Well, I suspect I'll have to dive into device drivers to make open file > support work, so maybe I can fix this at the same time, or make it > easier > for the next person. Personally I have no desire to run a server on > Windows. I can't see why you'd want to, given that Windows would give you absolutely no advantage and cost money. [snip] >>> * support for drive letters >> >> I would have thought that the existing mechanism of backup locations >> will handle this reasonably well. > > What mechanism is that? There seems to be an implicitly assumed special > object at the root of the filesystem with the name "/" (unless I > misunderstand). That doesn't seem to map very nicely to Windows drive > letters, especially when it comes to removable devices. Perhaps I could > have a virtual root directory with "/c:/", "/d:/", etc. underneath it. Absolutely no need for this. If you look in the .conf file, you might get a section like this: BackupLocations { home { Path = /home } usr-local { Path = /usr/local } } Each location is independent. The names don't even have to match the names of the path. In the root directory of the store, you get the location name, and then all the files below it. So under Windows you'd simply have something like BackupLocations { disc1 { Path = C:\ } system-disc { Path = D:\ } } Nothing complex or weird necessary. > >> This requires multiple streams to be implemented, which currently >> there's no neat way of doing. Now that WinXP SP2 uses multiple streams >> to store file security information, it's probably required for Win32 >> as >> well. And I might as well support sparse files while I'm at it. > > I don't plan to back up security information directly as a binary > stream, > because I suspect that it would be useless if restored onto a different > machine or installation. Perhaps I could use such a stream for a > textual, > machine-independent representation of all the security information > associated with a file, and it might be the best way to support that > kind > of detailed security data. There is a separate object which does permissions and security info. This just needs to be extended. The streams that I were talking about contain bits of information about the source of the file. So if it was downloaded from the internet, it can be marked as suspicious, and the user gets a warning when they execute it. > Do you have concrete plans to implement this at > some point? Yes. I'd like to get Mac OS X supported properly, for a start! (seeing as it's my primary platform and everything) > >> A few other people have a bit of a start on this. I will try and put >> everyone in touch with each other. > > Please do, I would be very interested to hear from anyone else who is > working on Win32 support or a GUI, especially if they wish to > collaborate, > but at least to avoid duplication of effort. This does need coordination. I could get another mailing list started if that would help? Ben