From maillist at diffingo.com Sun Jan 2 00:18:58 2011 From: maillist at diffingo.com (Stewart Adam) Date: Sat, 01 Jan 2011 19:18:58 -0500 Subject: [Box Backup] bsdmake/GNU make on OS X In-Reply-To: <20101231205614.GA21506@netinertia.co.uk> References: <4D1E2FDA.6080109@diffingo.com> <20101231205614.GA21506@netinertia.co.uk> Message-ID: <4D1FC472.4070308@diffingo.com> On 2010/12/31 3:56 PM, James O'Gorman wrote: > I'm assuming by "regular" Makefiles you are referring to GNU make. > > For what it's worth, I consider bsdmake to be superior to gmake in many > ways so if something's broken for OS X I'll see if I can fix it, > although I wasn't aware of any breakage. I'll fire up the buildbot again > as it's been offline for a while. > > But really if it's broken on OS X it should be broken on *BSD too and I > don't think it is. Nothing is currently broken with either style of Makefile, I was just being curious ;) Stewart From maillist at diffingo.com Sun Jan 2 00:26:09 2011 From: maillist at diffingo.com (Stewart Adam) Date: Sat, 01 Jan 2011 19:26:09 -0500 Subject: [Box Backup] bsdmake/GNU make on OS X In-Reply-To: References: <4D1E2FDA.6080109@diffingo.com> Message-ID: <4D1FC621.9030108@diffingo.com> On 2010/12/31 3:07 PM, Chris Wilson wrote: > 1. Ben, who wrote this code originally, prefers BSD to GNU in general; > 2. I think GNU make is not always installed on OS X machines. Is it part of > XCode? It gets installed as a part of XCode/"Developer Utilities" available from the factory-shipped OS X reinstallation discs. > 3. BSD make has a better dependency mechanism that results in faster builds > than GNU make on the same system. I didn't realize this... Good to know! Regards, Stewart From wouter.van.eekelen at serverffs.com Mon Jan 3 15:27:08 2011 From: wouter.van.eekelen at serverffs.com (Wouter van Eekelen) Date: Mon, 3 Jan 2011 16:27:08 +0100 Subject: [Box Backup] Error regarding SocketBindError / ListenAddresses Message-ID: I'm getting this error when starting the server: Jan 3 18:22:08 donkey Box Backup (bbstored)[1078]: NOTICE: Starting daemon, version 0.11rc2, config: /etc/boxbackup/bbstored.conf Jan 3 18:22:08 donkey Box Backup (bbstored)[1078]: WARNING: Exception thrown: ServerException(SocketBindError) at ../../lib/server/SocketListen.h(199) Jan 3 18:22:08 donkey Box Backup (bbstored)[1078]: FATAL: Terminating due to exception Server SocketBindError (Check the ListenAddresses directive in your config file -- must refer to local IP addresses only) (3/16) Jan 3 18:22:08 donkey bbstored/hk[1079]: NOTICE: Terminating daemon I've changed ListenAddresses to many options, such as: ListenAddresses = inet:localhost ListenAddresses = inet:0.0.0.0 ListenAddresses = inet:127.0.0.1 ListenAddresses = inet:217.18.70.2 ListenAddresses = localhost ListenAddresses = 0.0.0.0 ListenAddresses = 127.0.0.1 ListenAddresses = 217.18.70.2 What am I doing wrong? -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at netinertia.co.uk Mon Jan 3 15:31:07 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Mon, 3 Jan 2011 15:31:07 +0000 Subject: [Box Backup] Error regarding SocketBindError / ListenAddresses In-Reply-To: References: Message-ID: <20110103153107.GC21506@netinertia.co.uk> On Mon, Jan 03, 2011 at 04:27:08PM +0100, Wouter van Eekelen wrote: > I'm getting this error when starting the server: > Jan 3 18:22:08 donkey Box Backup (bbstored)[1078]: NOTICE: Starting daemon, > version 0.11rc2, config: /etc/boxbackup/bbstored.conf > Jan 3 18:22:08 donkey Box Backup (bbstored)[1078]: WARNING: Exception > thrown: ServerException(SocketBindError) at > ../../lib/server/SocketListen.h(199) > Jan 3 18:22:08 donkey Box Backup (bbstored)[1078]: FATAL: Terminating due > to exception Server SocketBindError (Check the ListenAddresses directive in > your config file -- must refer to local IP addresses only) (3/16) > Jan 3 18:22:08 donkey bbstored/hk[1079]: NOTICE: Terminating daemon Is there already something listening on TCP port 2201? Check with: netstat -an | grep 2201 It could be a stale daemon hanging around and holding the port open. This can happen after running the unit tests sometimes. James From wouter.van.eekelen at serverffs.com Mon Jan 3 15:33:40 2011 From: wouter.van.eekelen at serverffs.com (Wouter van Eekelen) Date: Mon, 3 Jan 2011 16:33:40 +0100 Subject: [Box Backup] Error regarding SocketBindError / ListenAddresses In-Reply-To: <20110103153107.GC21506@netinertia.co.uk> References: <20110103153107.GC21506@netinertia.co.uk> Message-ID: Not sure why I didn't think of that, problem resolved :) Thanks! Maybe such a check could be done by box backup itself, or simply add 'Maybe port 2201 is taken?' to remind users to check for that too. On Mon, Jan 3, 2011 at 4:31 PM, James O'Gorman wrote: > Is there already something listening on TCP port 2201? Check with: > > netstat -an | grep 2201 > > It could be a stale daemon hanging around and holding the port open. > This can happen after running the unit tests sometimes. > > James > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at netinertia.co.uk Mon Jan 3 15:36:17 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Mon, 3 Jan 2011 15:36:17 +0000 Subject: [Box Backup] Error regarding SocketBindError / ListenAddresses In-Reply-To: References: <20110103153107.GC21506@netinertia.co.uk> Message-ID: <20110103153617.GD21506@netinertia.co.uk> On Mon, Jan 03, 2011 at 04:33:40PM +0100, Wouter van Eekelen wrote: > Not sure why I didn't think of that, problem resolved :) Thanks! > Maybe such a check could be done by box backup itself, or simply add 'Maybe > port 2201 is taken?' to remind users to check for that too. I think Chris has amended the unit tests to attempt to kill off any daemons it left behind, but if the actual daemon hasn't died properly then I don't think the code is there to check. Arguably that's up to the sysadmin to sort out though :-) James From wouter.van.eekelen at serverffs.com Mon Jan 3 16:16:11 2011 From: wouter.van.eekelen at serverffs.com (Wouter van Eekelen) Date: Mon, 3 Jan 2011 17:16:11 +0100 Subject: [Box Backup] Error regarding SocketBindError / ListenAddresses In-Reply-To: <20110103153617.GD21506@netinertia.co.uk> References: <20110103153107.GC21506@netinertia.co.uk> <20110103153617.GD21506@netinertia.co.uk> Message-ID: Next problem, I'm getting a problem regarding SSL. I already re-generated the SSL certificates, updated openSSL, generated again, all not helping unfortunately. -- Client Jan 3 19:13:08 donkey Box Backup (bbackupd)[1954]: NOTICE: Beginning scan of local files Jan 3 19:13:08 donkey Box Backup (bbackupd)[1954]: ERROR: SSL error during Connect: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed Jan 3 19:13:08 donkey Box Backup (bbackupd)[1954]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(250) Jan 3 19:13:08 donkey Box Backup (bbackupd)[1954]: WARNING: Suppressing duplicate notification about backup-error Jan 3 19:13:08 donkey Box Backup (bbackupd)[1954]: ERROR: Exception caught (Connection TLSHandshakeFailed 7/30), reset state and waiting to retry... -- Server Jan 3 19:14:12 donkey Box Backup (bbstored)[1667]: WARNING: Message from child process 1692: Incoming connection from 217.18.XX.XX port 41161 Jan 3 19:14:12 donkey Box Backup (bbstored)[1692]: ERROR: SSL error during Accept: error:14094415:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate expired Jan 3 19:14:12 donkey Box Backup (bbstored)[1692]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(245) Jan 3 19:14:12 donkey Box Backup (bbstored)[1692]: ERROR: Error in child process, terminating connection: exception Connection TLSHandshakeFailed(7/30) I checked the certificates, they're all good. The server and client have the exact same checksums on the serverCA.pem and certificate itself. -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at netinertia.co.uk Mon Jan 3 16:24:24 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Mon, 3 Jan 2011 16:24:24 +0000 Subject: [Box Backup] Error regarding SocketBindError / ListenAddresses In-Reply-To: References: <20110103153107.GC21506@netinertia.co.uk> <20110103153617.GD21506@netinertia.co.uk> Message-ID: <20110103162424.GE21506@netinertia.co.uk> On Mon, Jan 03, 2011 at 05:16:11PM +0100, Wouter van Eekelen wrote: > Next problem, I'm getting a problem regarding SSL. I already re-generated > the SSL certificates, updated openSSL, generated again, all not helping > unfortunately. Is the time the same on both client and server? From your logs they're about a minute apart? > child process 1692: Incoming connection from 217.18.XX.XX port 41161 > Jan 3 19:14:12 donkey Box Backup (bbstored)[1692]: ERROR: SSL error during > Accept: error:14094415:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate ^^^^^^^^^^^^^^^^^^^^^^^ > expired ^^^^^^^ These seem to be a key here which makes me think it's a time sync issue. James From wouter.van.eekelen at serverffs.com Mon Jan 3 16:28:17 2011 From: wouter.van.eekelen at serverffs.com (Wouter van Eekelen) Date: Mon, 3 Jan 2011 17:28:17 +0100 Subject: [Box Backup] Error regarding SocketBindError / ListenAddresses In-Reply-To: <20110103162424.GE21506@netinertia.co.uk> References: <20110103153107.GC21506@netinertia.co.uk> <20110103153617.GD21506@netinertia.co.uk> <20110103162424.GE21506@netinertia.co.uk> Message-ID: The times are both synced, I must have copy pasted wrong. Here it is again: -- Server Jan 3 19:26:11 donkey Box Backup (bbstored)[1667]: WARNING: Message from child process 1750: Incoming connection from 217.18.XX.XX port 33007 Jan 3 19:26:11 donkey Box Backup (bbstored)[1750]: ERROR: SSL error during Accept: error:14094415:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate expired Jan 3 19:26:11 donkey Box Backup (bbstored)[1750]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(245) Jan 3 19:26:11 donkey Box Backup (bbstored)[1750]: ERROR: Error in child process, terminating connection: exception Connection TLSHandshakeFailed(7/30) -- Client Jan 3 19:26:11 donkey Box Backup (bbackupd)[1964]: NOTICE: Beginning scan of local files Jan 3 19:26:11 donkey Box Backup (bbackupd)[1964]: ERROR: SSL error during Connect: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed Jan 3 19:26:11 donkey Box Backup (bbackupd)[1964]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(250) Jan 3 19:26:11 donkey Box Backup (bbackupd)[1964]: WARNING: Suppressing duplicate notification about backup-error Jan 3 19:26:11 donkey Box Backup (bbackupd)[1964]: ERROR: Exception caught (Connection TLSHandshakeFailed 7/30), reset state and waiting to retry... On Mon, Jan 3, 2011 at 5:24 PM, James O'Gorman wrote: > Is the time the same on both client and server? From your logs they're > about a minute apart? > > These seem to be a key here which makes me think it's a time sync issue. > > James > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wouter.van.eekelen at serverffs.com Mon Jan 3 17:02:13 2011 From: wouter.van.eekelen at serverffs.com (Wouter van Eekelen) Date: Mon, 3 Jan 2011 18:02:13 +0100 Subject: [Box Backup] Error regarding SocketBindError / ListenAddresses In-Reply-To: References: <20110103153107.GC21506@netinertia.co.uk> <20110103153617.GD21506@netinertia.co.uk> <20110103162424.GE21506@netinertia.co.uk> Message-ID: Problem found! The bbstored-certs file has this on top: -- my $root_sign_period = '10000'; 10,000 days is 27.39 years, which is past 2038 (Y2K38), which made the certificate invalid: -- notBefore=Jan 3 15:59:18 2011 GMT notAfter=Apr 15 09:31:02 1902 GMT I've changed it to 9000, which is before 2038, so now it's all valid :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at netinertia.co.uk Mon Jan 3 17:04:31 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Mon, 3 Jan 2011 17:04:31 +0000 Subject: [Box Backup] Error regarding SocketBindError / ListenAddresses In-Reply-To: References: <20110103153107.GC21506@netinertia.co.uk> <20110103153617.GD21506@netinertia.co.uk> <20110103162424.GE21506@netinertia.co.uk> Message-ID: <20110103170430.GF21506@netinertia.co.uk> On Mon, Jan 03, 2011 at 06:02:13PM +0100, Wouter van Eekelen wrote: > Problem found! The bbstored-certs file has this on top: > -- > my $root_sign_period = '10000'; > > 10,000 days is 27.39 years, which is past 2038 (Y2K38), which made the > certificate invalid: > -- > notBefore=Jan 3 15:59:18 2011 GMT > notAfter=Apr 15 09:31:02 1902 GMT > > I've changed it to 9000, which is before 2038, so now it's all valid :) Aha, interesting! Perhaps we should change this default for 32-bit systems. Good spot, thanks. James From wouter.van.eekelen at serverffs.com Mon Jan 3 17:21:38 2011 From: wouter.van.eekelen at serverffs.com (Wouter van Eekelen) Date: Mon, 3 Jan 2011 18:21:38 +0100 Subject: [Box Backup] Pre-compiled clients Message-ID: Is there some overview page of pre-compiled clients? I've already found the latest Windows client, the Debian client is in their repositories - what about CentOs (preferably a rpm), Mac OS X and OpenBSD? -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at netinertia.co.uk Mon Jan 3 18:52:46 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Mon, 3 Jan 2011 18:52:46 +0000 Subject: [Box Backup] Pre-compiled clients In-Reply-To: References: Message-ID: <20110103185246.GG21506@netinertia.co.uk> On Mon, Jan 03, 2011 at 06:21:38PM +0100, Wouter van Eekelen wrote: > Is there some overview page of pre-compiled clients? > I've already found the latest Windows client, the Debian client is in > their repositories - what about CentOs (preferably a rpm), Mac OS X and > OpenBSD? I think it won't be in any distribution provided by Red Hat. The list archives will explain but I'm pretty sure Fedora rejected it for some weird reason. We do have an RPM spec file in the repository for anyone who wants to build RPMs though. Not sure about OS X, but there's nothing for OpenBSD yet. It's been on my todo list for a seriously long time (pkgsrc too), but I don't use OpenBSD as much as I used to so haven't got around to it. And for those that keep asking me, I will try and update the FreeBSD port soon! I'll probably ask for a repocopy of the current port and create a boxbackup-devel port. James From mitja at muzenic.net Mon Jan 3 20:10:16 2011 From: mitja at muzenic.net (=?iso-8859-2?Q?Mitja_Mu=BEeni=E8?=) Date: Mon, 3 Jan 2011 21:10:16 +0100 Subject: [Box Backup] Pre-compiled clients In-Reply-To: <20110103185246.GG21506@netinertia.co.uk> References: <20110103185246.GG21506@netinertia.co.uk> Message-ID: <010601cbab82$3d1bc780$b7535680$@net> > -----Original Message----- > From: boxbackup-bounces at boxbackup.org [mailto:boxbackup-bounces at boxbackup.org] > On Behalf Of James O'Gorman > Sent: Monday, January 03, 2011 7:53 PM > To: Box Backup > Subject: Re: [Box Backup] Pre-compiled clients > > On Mon, Jan 03, 2011 at 06:21:38PM +0100, Wouter van Eekelen wrote: > > Is there some overview page of pre-compiled clients? > > I've already found the latest Windows client, the Debian client is in > > their repositories - what about CentOs (preferably a rpm), Mac OS X and > > OpenBSD? > > I think it won't be in any distribution provided by Red Hat. The list > archives will explain but I'm pretty sure Fedora rejected it for some > weird reason. We do have an RPM spec file in the repository for anyone > who wants to build RPMs though. > > Not sure about OS X, but there's nothing for OpenBSD yet. It's been on > my todo list for a seriously long time (pkgsrc too), but I don't use > OpenBSD as much as I used to so haven't got around to it. There was an OpenBSD port done by a few individuals a year ago, but it never got commited into the OpenBSD ports tree or even went public because at the time boxbackup used to build only on one OpenBSD-supported arhitecture, i386 (32-bit Intel). In principle i386-only ports are acceptable only if the software is inherently i386-only (stuff like bios flasher tools or binaries that require linux emulation) and not if the software could be patched to make it to work. I couldn't fix the build on other arhitectures (among other the 64-bit intel and sparc) and couldn't motivate anybody else to do it. Since then the situation worsened and the recent boxbackup versions do not build on OpenBSD i386 anymore, unfortunately at the moment I can't remember if it was due to changes in boxbackup or to changes in OpenBSD. As far as I remember the binaries build, but the full parcel build fails due to manpage issues. If anybody can fix that, it would be much appreciated. Regards, Mitja > > And for those that keep asking me, I will try and update the FreeBSD > port soon! I'll probably ask for a repocopy of the current port and > create a boxbackup-devel port. > > James > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup From james at netinertia.co.uk Mon Jan 3 20:20:11 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Mon, 3 Jan 2011 20:20:11 +0000 Subject: [Box Backup] Pre-compiled clients In-Reply-To: <010601cbab82$3d1bc780$b7535680$@net> References: <20110103185246.GG21506@netinertia.co.uk> <010601cbab82$3d1bc780$b7535680$@net> Message-ID: <20110103202010.GH21506@netinertia.co.uk> On Mon, Jan 03, 2011 at 09:10:16PM +0100, Mitja Mu?eni? wrote: > There was an OpenBSD port done by a few individuals a year ago, but it never > got commited into the OpenBSD ports tree or even went public because at the > time boxbackup used to build only on one OpenBSD-supported arhitecture, i386 > (32-bit Intel). In principle i386-only ports are acceptable only if the > software is inherently i386-only (stuff like bios flasher tools or binaries > that require linux emulation) and not if the software could be patched to > make it to work. I couldn't fix the build on other arhitectures (among other > the 64-bit intel and sparc) and couldn't motivate anybody else to do it. Interesting - I wasn't aware it was broken on OpenBSD. Nobody had mentioned it before now. I'll see if I can find time to take a look. > Since then the situation worsened and the recent boxbackup versions do not > build on OpenBSD i386 anymore, unfortunately at the moment I can't remember > if it was due to changes in boxbackup or to changes in OpenBSD. As far as I > remember the binaries build, but the full parcel build fails due to manpage > issues. If anybody can fix that, it would be much appreciated. Ah, yes, there was a known problem with docbook not working properly if you didn't have the DTDs locally. I think Chris put a fix in for that. James From maillist at diffingo.com Tue Jan 4 20:52:46 2011 From: maillist at diffingo.com (Stewart Adam) Date: Tue, 04 Jan 2011 15:52:46 -0500 Subject: [Box Backup] Pre-compiled clients In-Reply-To: <20110103185246.GG21506@netinertia.co.uk> References: <20110103185246.GG21506@netinertia.co.uk> Message-ID: <4D23889E.5060705@diffingo.com> On 2011/01/03 1:52 PM, James O'Gorman wrote: > > I think it won't be in any distribution provided by Red Hat. The list > archives will explain but I'm pretty sure Fedora rejected it for some > weird reason. We do have an RPM spec file in the repository for anyone > who wants to build RPMs though. Now that the licensing has been cleared up the review can go forward, at the moment it's waiting on me (I have a few changes to make before it gets approved, I just haven't had the time to process that yet). In the mean time, if needed I will be happy to upload the binary & source RPMs or the spec file itself that I am currently using to prepare RPMs on CentOS 5.x as well as Fedora 12+. > Not sure about OS X, but there's nothing for OpenBSD yet. I don't have any packages ready, but I can provide the resulting parcels from my compile on OS X 10.6 (64-bit). Regards, Stewart From jp at jpsdomain.org Sat Jan 8 10:48:46 2011 From: jp at jpsdomain.org (JP Vossen) Date: Sat, 08 Jan 2011 05:48:46 -0500 Subject: [Box Backup] Bug report: serverCA.pem notAfter=Apr 20 02:52:14 1902 GMT Message-ID: <4D28410E.8010204@jpsdomain.org> I see the Box Backup Trac is locked down, so this is hereby my official bug report. :-) I've just updated the Debian bug at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=601882 too. Argh! Now that I know what to look for, I found this is already reported at http://lists.boxbackup.org/pipermail/boxbackup/2011-January/006126.html. I'm re-reporting it in the hopes that a) someone will make the trivial fix and b) to provide search engines more detail in the meantime (it'd be really nice if I'd found that about 3 hours ago)... I just installed Box Backup from the stock Lenny repos: boxbackup-client-0.11~rc2-5 boxbackup-server-0.11~rc2-5 I got the client connecting to the server, but then I got: SERVER: Jan 8 04:17:33 angstrom Box Backup (bbstored)[30573]: WARNING: Message from child process 31672: Incoming connection from 192.168.99.11 port 46789 Jan 8 04:17:33 angstrom Box Backup (bbstored)[31672]: ERROR: SSL error during Accept: error:14094415:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate expired Jan 8 04:17:33 angstrom Box Backup (bbstored)[31672]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(245) Jan 8 04:17:33 angstrom Box Backup (bbstored)[31672]: ERROR: Error in child process, terminating connection: exception Connection TLSHandshakeFailed(7/30) CLIENT: Jan 8 04:17:33 drake Box Backup (bbackupd)[3419]: NOTICE: Beginning scan of local files Jan 8 04:17:33 drake Box Backup (bbackupd)[3419]: ERROR: SSL error during Connect: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed Jan 8 04:17:33 drake Box Backup (bbackupd)[3419]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(250) Jan 8 04:17:33 drake Box Backup (bbackupd)[3419]: WARNING: Suppressing duplicate notification about backup-error Jan 8 04:17:33 drake Box Backup (bbackupd)[3419]: ERROR: Exception caught (Connection TLSHandshakeFailed 7/30), reset state and waiting to retry... Jan 8 04:17:43 drake Box Backup (bbackupd)[3419]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Since "sslv3 alert certificate expired" is kind of a clue, I started looking at the *.pem files. I'm guessing "notAfter=Apr 20 02:52:13 1902 GMT" is a Bad Thing... # for cert in bbstored/*.pem; do echo $cert; openssl x509 -in $cert -dates -noout; done bbstored/angstrom-cert.pem notBefore=Jan 8 09:21:57 2011 GMT notAfter=Sep 16 09:21:57 2024 GMT [... cruft removed] bbstored/clientCA.pem notBefore=Jan 8 09:20:29 2011 GMT notAfter=Apr 20 02:52:13 1902 GMT # for cert in ca/roots/*.pem; do echo $cert; openssl x509 -in $cert -dates -noout; done ca/roots/clientCA.pem notBefore=Jan 8 09:20:29 2011 GMT notAfter=Apr 20 02:52:13 1902 GMT ca/roots/serverCA.pem notBefore=Jan 8 09:20:30 2011 GMT notAfter=Apr 20 02:52:14 1902 GMT I tried an 'rm -rf ca' and 'bbstored-certs ca init' on two different machines and I got the same thing. The client and server are both using NTP and time is correct on both. Hmmmm, they are both 32-bit machines. And getting "1902" out of Perl seems shady given how it likes to fiddle the year by 1900. Are we maybe overflowing something? OK, yes, we are. If I change /usr/bin/bbstored-certs from '10000' to read "my $root_sign_period = '8888';" I now get: ca/roots/clientCA.pem notBefore=Jan 8 09:58:03 2011 GMT notAfter=May 10 09:58:03 2035 GMT ca/roots/serverCA.pem notBefore=Jan 8 09:58:04 2011 GMT notAfter=May 10 09:58:04 2035 GMT Various values in the '99\d\d' range failed, so I tried '8888' and it worked for me. Thanks for a great tool, JP ----------------------------|:::======|------------------------------- JP Vossen, CISSP |:::======| http://bashcookbook.com/ My Account, My Opinions |=========| http://www.jpsdomain.org/ ----------------------------|=========|------------------------------- "Microsoft Tax" = the additional hardware & yearly fees for the add-on software required to protect Windows from its own poorly designed and implemented self, while the overhead incidentally flattens Moore's Law. From chris at qwirx.com Sat Jan 8 20:24:32 2011 From: chris at qwirx.com (Chris Wilson) Date: Sat, 8 Jan 2011 20:24:32 +0000 (GMT) Subject: [Box Backup] boxbackup in debian/squeeze In-Reply-To: <8762v15h2n.fsf@faui44a.informatik.uni-erlangen.de> References: <87eibdv6gx.fsf@faui44a.informatik.uni-erlangen.de> <8762v15h2n.fsf@faui44a.informatik.uni-erlangen.de> Message-ID: Hi Reinhard, On Sat, 11 Dec 2010, Reinhard Tartler wrote: > rc8 packages built for lenny can be found here: > > http://backports.debian.org/debian-backports/pool/main/b/boxbackup/ > > These commands will enable the debian-backports repository: > > $ echo deb http://backports.debian.org/debian-backports lenny-backports main > /etc/apt/sources.list.d/debian-backports.list > $ apt-get update > > Build status (with logs) is here: > https://buildd.debian.org/status/package.php?p=boxbackup&suite=lenny-backports-sloppy > > Perhaps someone wants to add these notes to the boxbackup website? Done, thanks, and sorry for the delay. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Sat Jan 8 20:53:07 2011 From: chris at qwirx.com (Chris Wilson) Date: Sat, 8 Jan 2011 20:53:07 +0000 (GMT) Subject: [Box Backup] Bug report: serverCA.pem notAfter=Apr 20 02:52:14 1902 GMT In-Reply-To: <4D28410E.8010204@jpsdomain.org> References: <4D28410E.8010204@jpsdomain.org> Message-ID: Hi JP and all, On Sat, 8 Jan 2011, JP Vossen wrote: > I see the Box Backup Trac is locked down, Anyone can apply for an account :) Just send us an email. > Argh! Now that I know what to look for, I found this is already reported at > http://lists.boxbackup.org/pipermail/boxbackup/2011-January/006126.html. I'm > re-reporting it in the hopes that: > > a) someone will make the trivial fix I've just committed one, I hope it works for you :) > and b) to provide search engines more detail in the meantime (it'd be > really nice if I'd found that about 3 hours ago)... The post you found was made just a few days ago. I can find other posts from the archives using Google, but not that one, so I guess they may not have indexed it yet. I don't think we use a robots file to exclude any search engines. I'm not sure how we can provide more information? > I just installed Box Backup from the stock Lenny repos: > boxbackup-client-0.11~rc2-5 > boxbackup-server-0.11~rc2-5 Reinhard, I'm going to add this fix to 0.11 to make 0.11.1, would you like to update the Debian packages when it becomes available shortly? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From jp at jpsdomain.org Sun Jan 9 00:27:46 2011 From: jp at jpsdomain.org (JP Vossen) Date: Sat, 08 Jan 2011 19:27:46 -0500 Subject: [Box Backup] Bug report: serverCA.pem notAfter=Apr 20 02:52:14 1902 GMT In-Reply-To: References: <4D28410E.8010204@jpsdomain.org> Message-ID: <4D290102.6070308@jpsdomain.org> On 01/08/2011 03:53 PM, Chris Wilson wrote: > Hi JP and all, > > On Sat, 8 Jan 2011, JP Vossen wrote: > >> I see the Box Backup Trac is locked down, > > Anyone can apply for an account :) Just send us an email. OK, I will if I need one again. >> Argh! Now that I know what to look for, I found this is already >> reported at >> http://lists.boxbackup.org/pipermail/boxbackup/2011-January/006126.html. >> I'm re-reporting it in the hopes that: >> >> a) someone will make the trivial fix > > I've just committed one, I hope it works for you :) I'm up and working with my fix. >> and b) to provide search engines more detail in the meantime (it'd be >> really nice if I'd found that about 3 hours ago)... > > The post you found was made just a few days ago. I can find other posts > from the archives using Google, but not that one, so I guess they may > not have indexed it yet. I don't think we use a robots file to exclude > any search engines. I'm not sure how we can provide more information? I was originally searching on strings from my logs, and things like "bbstored 1902" and similar. My report should now generate hits on stuff that I'd search for, whenever it gets indexed. There is a way to ask Google to (re-)index a site or archive, couldn't hurt to hit it, but probably not a big deal. >> I just installed Box Backup from the stock Lenny repos: >> boxbackup-client-0.11~rc2-5 >> boxbackup-server-0.11~rc2-5 > > Reinhard, I'm going to add this fix to 0.11 to make 0.11.1, would you > like to update the Debian packages when it becomes available shortly? Cool. Thanks for the quick action on all of this! Later, JP ----------------------------|:::======|------------------------------- JP Vossen, CISSP |:::======| http://bashcookbook.com/ My Account, My Opinions |=========| http://www.jpsdomain.org/ ----------------------------|=========|------------------------------- "Microsoft Tax" = the additional hardware & yearly fees for the add-on software required to protect Windows from its own poorly designed and implemented self, while the overhead incidentally flattens Moore's Law. From chris at qwirx.com Sun Jan 9 01:50:50 2011 From: chris at qwirx.com (Chris Wilson) Date: Sun, 9 Jan 2011 01:50:50 +0000 (GMT) Subject: [Box Backup] Norton blocking boxbackup.org? In-Reply-To: <9E94BF29-5C0D-4802-AD53-95166EBED80C@netinertia.co.uk> References: <96da9c7754da17451aac43d2221d82f9@kaufmanfamily.net> <9E94BF29-5C0D-4802-AD53-95166EBED80C@netinertia.co.uk> Message-ID: Hi James, On Tue, 7 Dec 2010, James O'Gorman wrote: > On 7 Dec 2010, at 01:31, David H Kaufman wrote: > >> I proposed boxbackup to a colleague recently, but when he hit the web >> site for more info he was blocked by Norton SafeWeb. Apparently their >> scanner thinks it has found an old Windows XP key logger in one of the >> boxbackup tarballs - more details at this link: >> >> http://safeweb.norton.com/report/show?url=http:%2F%2Fwww.boxbackup.org%2Ftrac%2Fwiki >> >> There's also a place to click if you're the site owner, presumably to >> clear up the warning. Needless to say my colleague was a little put off >> by the warning. > > Thanks for letting us know. I'll register myself as the site owner and > see if I can get that re-evaluated. Did you get anywhere with this? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Sun Jan 9 02:10:20 2011 From: chris at qwirx.com (Chris Wilson) Date: Sun, 9 Jan 2011 02:10:20 +0000 (GMT) Subject: [Box Backup] Unable to restore old versions of files... In-Reply-To: <4D019F2C.7000302@mir.com> References: <4CFB09D3.2050508@mir.com> <4D019F2C.7000302@mir.com> Message-ID: Hi Matto, On Thu, 9 Dec 2010, Matto Marjanovic wrote: > On 12/04/10 22:41, Matto Marjanovic wrote: >> >> Today I discovered that I can't restore old versions of files from >> backup (i.e. versions marked with the 'o' flag). Attempting to do >> so causes the server's connection child process to throw an exception: > > Okey-doke, a bit of poking around with gdb and I figured out what the > problem is: it is a bug in the handler for "get" command, inadvertently > introduced 4 years ago in revision 891. > > The problem is in the construction of the filename for a temporary file; > the buggy code loses a chunk of the path (due to an std::ostringstream > constructor not being used correctly). Whether or not the bug manifests > itself depends on the directory permissions of where the mis-filed file > ends up. > > Here is a patch that fixes the bug (which is still present in HEAD): I've applied your patch, it will be in 0.11.1, many thanks! And sorry for the inconvenience. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From james at netinertia.co.uk Sun Jan 9 11:07:31 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Sun, 9 Jan 2011 11:07:31 +0000 Subject: [Box Backup] Norton blocking boxbackup.org? In-Reply-To: References: <96da9c7754da17451aac43d2221d82f9@kaufmanfamily.net> <9E94BF29-5C0D-4802-AD53-95166EBED80C@netinertia.co.uk> Message-ID: <20110109110730.GJ21506@netinertia.co.uk> On Sun, Jan 09, 2011 at 01:50:50AM +0000, Chris Wilson wrote: > Hi James, > > On Tue, 7 Dec 2010, James O'Gorman wrote: > > On 7 Dec 2010, at 01:31, David H Kaufman wrote: > > > >> I proposed boxbackup to a colleague recently, but when he hit the web > >> site for more info he was blocked by Norton SafeWeb. Apparently their > >> scanner thinks it has found an old Windows XP key logger in one of the > >> boxbackup tarballs - more details at this link: > >> > >> http://safeweb.norton.com/report/show?url=http:%2F%2Fwww.boxbackup.org%2Ftrac%2Fwiki > >> > >> There's also a place to click if you're the site owner, presumably to > >> clear up the warning. Needless to say my colleague was a little put off > >> by the warning. > > > > Thanks for letting us know. I'll register myself as the site owner and > > see if I can get that re-evaluated. > > Did you get anywhere with this? I did, and they removed the site report, but it seems they've reinstated it again. I suppose their heuristics are matching some string on that file that causes them to think there's something bad there. Might it be worth re-creating that tarball (or maybe even pulling it) to see if that sorts it out? In the meantime I'll log in and ask them to fix it again. James From siretart at tauware.de Sun Jan 9 15:56:35 2011 From: siretart at tauware.de (Reinhard Tartler) Date: Sun, 09 Jan 2011 16:56:35 +0100 Subject: [Box Backup] boxbackup in debian/squeeze In-Reply-To: (Chris Wilson's message of "Sat, 8 Jan 2011 20:24:32 +0000 (GMT)") References: <87eibdv6gx.fsf@faui44a.informatik.uni-erlangen.de> <8762v15h2n.fsf@faui44a.informatik.uni-erlangen.de> Message-ID: <874o9im5os.fsf@faui44a.informatik.uni-erlangen.de> On Sat, Jan 08, 2011 at 21:24:32 (CET), Chris Wilson wrote: > Hi Reinhard, > > On Sat, 11 Dec 2010, Reinhard Tartler wrote: > >> rc8 packages built for lenny can be found here: >> >> http://backports.debian.org/debian-backports/pool/main/b/boxbackup/ >> >> These commands will enable the debian-backports repository: >> >> $ echo deb http://backports.debian.org/debian-backports lenny-backports main > /etc/apt/sources.list.d/debian-backports.list >> $ apt-get update >> >> Build status (with logs) is here: >> https://buildd.debian.org/status/package.php?p=boxbackup&suite=lenny-backports-sloppy >> >> Perhaps someone wants to add these notes to the boxbackup website? > > Done, thanks, and sorry for the delay. I see that there is a type in both the description above and on the wiki. The wrong part reads: ,---- | $ echo deb http://backports.debian.org/debian-backports lenny-backports main >/etc/apt/sources.list.d/debian-backports.list | $ apt-get update `---- It should be replaced by this: ,---- | $ echo deb http://backports.debian.org/debian-backports lenny-backports-sloppy main >/etc/apt/sources.list.d/debian-backports.list | $ apt-get update `---- (the suite is 'lenny-backports-sloppy', not 'lenny-backports') Would you mind to create me a trac account and mail me the password? I could then update the instructions myself in future. Thanks! -- Gruesse/greetings, Reinhard Tartler, KeyID 945348A4 From siretart at tauware.de Sun Jan 9 15:31:11 2011 From: siretart at tauware.de (Reinhard Tartler) Date: Sun, 09 Jan 2011 16:31:11 +0100 Subject: [Box Backup] Bug#601882: boxbackup-server's bbstored-certs creates certs that expire in the past References: <4D28410E.8010204@jpsdomain.org> <4D28434A.5040208@jpsdomain.org> <87r5cno8q9.fsf@faui44a.informatik.uni-erlangen.de> Message-ID: <87aajam6v4.fsf@faui44a.informatik.uni-erlangen.de> Hi all, On Sat, Jan 08, 2011 at 21:53:07 (CET), Chris Wilson wrote: > Reinhard, I'm going to add this fix to 0.11 to make 0.11.1, would you > like to update the Debian packages when it becomes available shortly? Thanks for the offer, but I don't think a new upstream release will be necessary. Btw, AFAIUI 0.11 still isn't released, I see 'just' a 0.11rc8 package on http://boxbackup.org? So what would a potential 0.11.1 package contain then? Anyway, Debian 'lenny' (stable) ships with an 0.11rc2 package. Debian 'squeeze' will ship 0.11rc2 as well (rc8 missed the freeze), but has the proposed change already included. I'm going to include the fix in the package directly. The whole issue is tracked at http://bugs.debian.org/601506, and the Stable Release Manager (SRM, i.e., Adam) asked me to present a debdiff showing the changes for the update I'm proposing for debian *stable*. I'm doing this with this email, see below. > On Sat, 2011-01-08 at 13:55 +0100, Reinhard Tartler wrote: >> The fix for #601506 has been done with the -7 upload, but the change -6 >> would qualify for lenny IMO as well. SRM, would you be happy to include >> the changes from -6 and -7 in the next lenny point release? If yes, >> please advice what changes to include and what version number to use for >> an upload to stable-proposed-updates. On Sat, Jan 08, 2011 at 18:43:20 (CET), Adam D. Barratt wrote: > fwiw, most of the changes in -6 don't actually appear to be needed; the > problem only arises when the tests of the form "[ -z $VAR1 -o -z > $VAR2 ]" are used (and specifically only when $VAR1 is empty, as -z ends > up being applied to "-o"). > > Other than that, applying -6 and -7 to lenny would be okay. Please use > 0.11~rc2-5+lenny1 as the version number and send a debdiff to -release > for final pre-upload confirmation. I've taken these changes in anyways because I do know that they work in that way in squeeze. Here we go: --- boxbackup-0.11~rc2/debian/boxbackup-server.postinst +++ boxbackup-0.11~rc2/debian/boxbackup-server.postinst @@ -71,7 +71,7 @@ DIR3=`echo "$RET" | awk '{ print $3 }'` if [ -n $DIR1 ]; then - if [ -z $DIR2 -o -z $DIR3 ]; then + if [ -z "$DIR2" -o -z "$DIR3" ]; then DIR2=$DIR1 DIR3=$DIR1 fi diff -u boxbackup-0.11~rc2/debian/boxbackup-client.postinst boxbackup-0.11~rc2/debian/boxbackup-client.postinst --- boxbackup-0.11~rc2/debian/boxbackup-client.postinst +++ boxbackup-0.11~rc2/debian/boxbackup-client.postinst @@ -74,15 +74,15 @@ if [ "$RET" = "lazy" ]; then db_get boxbackup-client/UpdateStoreInterval UPDATE=$RET - [ -z $UPDATE ] && UPDATE="3600" + [ -z "$UPDATE" ] && UPDATE="3600" db_get boxbackup-client/MinimumFileAge FILEAGE=$RET - [ -z $FILEAGE ] && FILEAGE="21600" + [ -z "$FILEAGE" ] && FILEAGE="21600" db_get boxbackup-client/MaxUploadWait UPWAIT=$RET - [ -z $UPWAIT ] && UPWAIT="86400" + [ -z "$UPWAIT" ] && UPWAIT="86400" AUTO=yes else @@ -219,7 +219,7 @@ chmod 600 $BBKEY || true # SSL stuff - if [ ! -z $ACCOUNT ]; then + if [ ! -z "$ACCOUNT" ]; then if [ ! -e $BBPRIVKEY -a ! -e $BBCERT ]; then db_get boxbackup-client/generateCertificate diff -u boxbackup-0.11~rc2/debian/boxbackup-server.config boxbackup-0.11~rc2/debian/boxbackup-server.config --- boxbackup-0.11~rc2/debian/boxbackup-server.config +++ boxbackup-0.11~rc2/debian/boxbackup-server.config @@ -33,7 +33,7 @@ DIR3=`echo "$RET" | awk '{ print $3 }'` if [ -n $DIR1 ]; then - if [ -z $DIR2 -o -z $DIR3 ]; then + if [ -z "$DIR2" -o -z "$DIR3" ]; then DIR2=$DIR1 DIR3=$DIR1 fi diff -u boxbackup-0.11~rc2/debian/changelog boxbackup-0.11~rc2/debian/changelog --- boxbackup-0.11~rc2/debian/changelog +++ boxbackup-0.11~rc2/debian/changelog @@ -1,3 +1,13 @@ +boxbackup (0.11~rc2-5+lenny1) stable-proposed-updates; urgency=low + + * bin/bbstored/bbstored-certs: reduce root CA expiration date to avoid + Y2k38 overflow. Thanks to Clint Adams for + reporting it. Closes: #601506 + * Fix shell scripting in the debconf interaction code of the package's + postinst script. This should prevent problems like LP: #222999 + + -- Reinhard Tartler Sun, 09 Jan 2011 16:07:46 +0100 + boxbackup (0.11~rc2-5) unstable; urgency=low * Bugfix: "Please build-depend on docbook-xml". Thanks to Luca Falavigna only in patch2: unchanged: --- boxbackup-0.11~rc2.orig/bin/bbstored/bbstored-certs.in +++ boxbackup-0.11~rc2/bin/bbstored/bbstored-certs.in @@ -2,10 +2,10 @@ use strict; # validity period for root certificates -- default is a very long time -my $root_sign_period = '10000'; +my $root_sign_period = '5000'; # but less so for client certificates -my $sign_period = '5000'; +my $sign_period = '3000'; # check and get command line parameters if($#ARGV < 1) -- Gruesse/greetings, Reinhard Tartler, KeyID 945348A4 From adam at adam-barratt.org.uk Sun Jan 9 18:18:37 2011 From: adam at adam-barratt.org.uk (Adam D. Barratt) Date: Sun, 09 Jan 2011 18:18:37 +0000 Subject: [Box Backup] Bug#601882: boxbackup-server's bbstored-certs creates certs that expire in the past In-Reply-To: <87aajam6v4.fsf@faui44a.informatik.uni-erlangen.de> References: <4D28410E.8010204@jpsdomain.org> <4D28434A.5040208@jpsdomain.org> <87r5cno8q9.fsf@faui44a.informatik.uni-erlangen.de> <87aajam6v4.fsf@faui44a.informatik.uni-erlangen.de> Message-ID: <1294597117.2903.14846.camel@hathi.jungle.funky-badger.org> On Sun, 2011-01-09 at 16:31 +0100, Reinhard Tartler wrote: > On Sat, Jan 08, 2011 at 18:43:20 (CET), Adam D. Barratt wrote: > > fwiw, most of the changes in -6 don't actually appear to be needed; the > > problem only arises when the tests of the form "[ -z $VAR1 -o -z > > $VAR2 ]" are used (and specifically only when $VAR1 is empty, as -z ends > > up being applied to "-o"). > > > > Other than that, applying -6 and -7 to lenny would be okay. Please use > > 0.11~rc2-5+lenny1 as the version number and send a debdiff to -release > > for final pre-upload confirmation. > > I've taken these changes in anyways because I do know that they work in > that way in squeeze. Here we go: Thanks. Please go ahead. Regards, Adam From siretart at tauware.de Mon Jan 10 12:03:50 2011 From: siretart at tauware.de (Reinhard Tartler) Date: Mon, 10 Jan 2011 13:03:50 +0100 Subject: [Box Backup] Bug#601882: boxbackup-server's bbstored-certs creates certs that expire in the past References: <4D28410E.8010204@jpsdomain.org> <4D28434A.5040208@jpsdomain.org> <87r5cno8q9.fsf@faui44a.informatik.uni-erlangen.de> <87aajam6v4.fsf@faui44a.informatik.uni-erlangen.de> <1294597117.2903.14846.camel@hathi.jungle.funky-badger.org> Message-ID: <87wrmdj789.fsf@faui44a.informatik.uni-erlangen.de> On Sun, Jan 09, 2011 at 19:18:37 (CET), Adam D. Barratt wrote: > On Sun, 2011-01-09 at 16:31 +0100, Reinhard Tartler wrote: >> On Sat, Jan 08, 2011 at 18:43:20 (CET), Adam D. Barratt wrote: >> > fwiw, most of the changes in -6 don't actually appear to be needed; the >> > problem only arises when the tests of the form "[ -z $VAR1 -o -z >> > $VAR2 ]" are used (and specifically only when $VAR1 is empty, as -z ends >> > up being applied to "-o"). >> > >> > Other than that, applying -6 and -7 to lenny would be okay. Please use >> > 0.11~rc2-5+lenny1 as the version number and send a debdiff to -release >> > for final pre-upload confirmation. >> >> I've taken these changes in anyways because I do know that they work in >> that way in squeeze. Here we go: > > Thanks. Please go ahead. Uploaded. -- Gruesse/greetings, Reinhard Tartler, KeyID 945348A4 From achim+box at qustodium.net Mon Jan 10 23:19:58 2011 From: achim+box at qustodium.net (Achim) Date: Tue, 11 Jan 2011 00:19:58 +0100 Subject: [Box Backup] Trac account request In-Reply-To: <4D15B139.10109@diffingo.com> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> Message-ID: <4D2B941E.3000200@qustodium.net> Hello Stewart: On 25/12/2010 09:54, Stewart Adam wrote: > This is a feature I'm very interested in seeing in Box Backup - I > recently created a basic class that throttles read() operations from an > ifstream so I'm hoping that I can apply the code easily to the socket > streams in Box Backup and see how it performs. > > The algorithm tries to distribute the reads evenly over every second so > that you don't get burst reads which hog all available bandwidth at the > start of every second, but also so that you don't waste too much time > processing in tiny increments. Did you manage to move this further along? A per-client limitation of the upload bandwidth like in rsync would be really helpful to avoid a cluster of local bb clients from hogging the available DSL uplink. Modifying the DSL modem parameters for QoS seems a lot less practical (i.e. likely) than updating that "bandwidth" parameter in a box backup conf file. Thanks in advance, Achim From adam at adam-barratt.org.uk Mon Jan 10 13:43:13 2011 From: adam at adam-barratt.org.uk (Adam D. Barratt) Date: Mon, 10 Jan 2011 13:43:13 -0000 Subject: [Box Backup] Bug#601882: boxbackup-server's bbstored-certs creates certs that expire in the past In-Reply-To: <87wrmdj789.fsf@faui44a.informatik.uni-erlangen.de> References: <4D28410E.8010204@jpsdomain.org> <4D28434A.5040208@jpsdomain.org> <87r5cno8q9.fsf@faui44a.informatik.uni-erlangen.de> <87aajam6v4.fsf@faui44a.informatik.uni-erlangen.de> <1294597117.2903.14846.camel@hathi.jungle.funky-badger.org> <87wrmdj789.fsf@faui44a.informatik.uni-erlangen.de> Message-ID: <0c391f8147700fa3a2dd59649d745ae1.squirrel@adsl.funky-badger.org> On Mon, January 10, 2011 12:03, Reinhard Tartler wrote: > On Sun, Jan 09, 2011 at 19:18:37 (CET), Adam D. Barratt wrote: >> Thanks. Please go ahead. > > Uploaded. Marked for accept at the next dinstall; thanks. Regards, Adam From chris at qwirx.com Wed Jan 12 00:20:55 2011 From: chris at qwirx.com (Chris Wilson) Date: Wed, 12 Jan 2011 00:20:55 +0000 (GMT) Subject: [Box Backup] Trac account request In-Reply-To: <4D2B941E.3000200@qustodium.net> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> Message-ID: Hi Achim, On Tue, 11 Jan 2011, Achim wrote: > Did you manage to move this further along? A per-client limitation of > the upload bandwidth like in rsync would be really helpful to avoid a > cluster of local bb clients from hogging the available DSL uplink. > > Modifying the DSL modem parameters for QoS seems a lot less practical > (i.e. likely) than updating that "bandwidth" parameter in a box backup > conf file. I've implemented a rate limiting option in trunk. Simply set MaxUploadRate = 8000 (bytes per second) or similar on the client (in bbackupd.conf) to use it. Please have a look and see if it meets your needs. It's not as sophisticated as what Stewart was suggesting, distributing reads over a second. It does try to ensure at least one packet per second. I'm not sure how much spikes at one-second granularity matter. I think the method I've used is very CPU efficient but it does reduce bandwidth efficiency a little, by uploading smaller chunks of data to the server, and probably in smaller packets (<1500 bytes) at very low rates (<1500 bytes per second). It's also not 100% accurate because it doesn't take the OpenSSL encryption overhead into account, or any bandwidth use except streaming file and patch uploads. It could probably be adapted to be more intelligent if necessary. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From maddog at mir.com Wed Jan 12 03:22:16 2011 From: maddog at mir.com (Matto Marjanovic) Date: Tue, 11 Jan 2011 22:22:16 -0500 Subject: [Box Backup] Unable to restore old versions of files... In-Reply-To: References: <4CFB09D3.2050508@mir.com> <4D019F2C.7000302@mir.com> Message-ID: <4D2D1E68.6050700@mir.com> On 01/08/11 21:10, Chris Wilson wrote: > Hi Matto, > > On Thu, 9 Dec 2010, Matto Marjanovic wrote: >> On 12/04/10 22:41, Matto Marjanovic wrote: >>> >>> Today I discovered that I can't restore old versions of files from >>> backup (i.e. versions marked with the 'o' flag). Attempting to do >>> so causes the server's connection child process to throw an exception: >> >> Okey-doke, a bit of poking around with gdb and I figured out what the >> problem is: it is a bug in the handler for "get" command, inadvertently >> introduced 4 years ago in revision 891. >> >> The problem is in the construction of the filename for a temporary file; >> the buggy code loses a chunk of the path (due to an std::ostringstream >> constructor not being used correctly). Whether or not the bug manifests >> itself depends on the directory permissions of where the mis-filed file >> ends up. >> >> Here is a patch that fixes the bug (which is still present in HEAD): > > I've applied your patch, it will be in 0.11.1, many thanks! And sorry for the inconvenience. I'm honored, and you are welcome. There was no data loss (the best kind of bug), so "All Is Forgiven". -m From maillist at diffingo.com Wed Jan 12 07:08:41 2011 From: maillist at diffingo.com (Stewart Adam) Date: Wed, 12 Jan 2011 02:08:41 -0500 Subject: [Box Backup] Trac account request In-Reply-To: References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> Message-ID: <4D2D5379.5070700@diffingo.com> On 2011/01/11 7:20 PM, Chris Wilson wrote: > Hi Achim, > > On Tue, 11 Jan 2011, Achim wrote: > >> Did you manage to move this further along? A per-client limitation of the >> upload bandwidth like in rsync would be really helpful to avoid a cluster >> of local bb clients from hogging the available DSL uplink. >> >> Modifying the DSL modem parameters for QoS seems a lot less practical >> (i.e. likely) than updating that "bandwidth" parameter in a box backup >> conf file. > > I've implemented a rate limiting option in trunk. Simply set MaxUploadRate = > 8000 (bytes per second) or similar on the client (in bbackupd.conf) to use > it. Please have a look and see if it meets your needs. This is great, many thanks Chris! After some brief testing it seems to work very well and you are right - the CPU usage is very low (0.8% on my machine). I'm going to look into if we can make effectiveRateSoFar a time average of the past few seconds instead of the totals though, as that way if the link throughput changes (say another application finishes a transfer or laptop changes locations) then the calculation for the current rate will more accurately reflect the actual transfer rates. > It's not as sophisticated as what Stewart was suggesting, distributing reads > over a second. It does try to ensure at least one packet per second. I'm not > sure how much spikes at one-second granularity matter. My method was rather simple but just had the nice side effect of distributing writes over a second ;) I used what was essentially a method of finite differences - I divide a second into some "N" iterations (10 or 15 seemed to work well) and then allocate byteLimit/N bandwidth per iteration, sleeping in between if the allocated bandwidth gets used up early in an iteration. > I think the method I've used is very CPU efficient but it does reduce > bandwidth efficiency a little, by uploading smaller chunks of data to the > server, and probably in smaller packets (<1500 bytes) at very low rates > (<1500 bytes per second). It's also not 100% accurate because it doesn't > take the OpenSSL encryption overhead into account, or any bandwidth use > except streaming file and patch uploads. Both problems affected my implementation as well, I was thinking of forcing a minimum packet size to avoid the overhead problem. I don't think the SSL overhead is something to worry about... From testing it seems that there's roughly 2KB difference per 50KB (ie 50KB/s limit results in 52KB/s real world; 750KB/s limit results in ~775KB/s). Out of curiosity, do the trace messages accurately reflect when data is being sent out by bbackupd? Regards, Stewart From chris at qwirx.com Wed Jan 12 09:39:34 2011 From: chris at qwirx.com (Chris Wilson) Date: Wed, 12 Jan 2011 09:39:34 +0000 (GMT) Subject: [Box Backup] Upload rate limiting (was: Trac account request) In-Reply-To: <4D2D5379.5070700@diffingo.com> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> Message-ID: Hi Stewart, On Wed, 12 Jan 2011, Stewart Adam wrote: > On 2011/01/11 7:20 PM, Chris Wilson wrote: >> I've implemented a rate limiting option in trunk. Simply set >> MaxUploadRate = 8000 (bytes per second) or similar on the client (in >> bbackupd.conf) to use it. Please have a look and see if it meets your >> needs. > > This is great, many thanks Chris! After some brief testing it seems to > work very well and you are right - the CPU usage is very low (0.8% on my > machine). Great, I'm happy that you like it and it works for you. > I'm going to look into if we can make effectiveRateSoFar a time average > of the past few seconds instead of the totals though, as that way if the > link throughput changes (say another application finishes a transfer or > laptop changes locations) then the calculation for the current rate will > more accurately reflect the actual transfer rates. OK, I was thinking that it would be more stable and accurate across the whole lifetime of the connection to use the total bytes transferred, but you're right that if some other application uses enough bandwidth that bbackupd falls behind, then it will exceed the specified rate to "catch up", which is probably not desirable. > I don't think the SSL overhead is something to worry about... From > testing it seems that there's roughly 2KB difference per 50KB (ie 50KB/s > limit results in 52KB/s real world; 750KB/s limit results in ~775KB/s). OK, good. > Out of curiosity, do the trace messages accurately reflect when data is > being sent out by bbackupd? Not quite. The buffering is on the read side, but reading out the encrypted block stream that will be broken down into chunks for transmission over the wire by Protocol and SSL. You can put a breakpoint on RateLimitingStream::Read to see that it's called by Protocol::SendStream, which luckily tries to dispatch every block that it receives immediately over the wire (by calling SendStreamSendBlock), prefixed by a header byte. So more, smaller blocks means more header bytes, unfortunately. The protocol is wrapped in SSL, and SSL may write as few or as many bytes as it feels like when more data is pumped into it. But I think it tries to send as much as possible, i.e. a whole number of encrypted blocks, where the block size is determined by the encryption algorithm but probably 16 bytes if we're using AES. So I guess that less than 16 bytes would remain buffered in SSL after SendStreamSendBlock completes, and the rest has gone out to the OS at least, if not the wire. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From achim+box at qustodium.net Fri Jan 14 00:52:00 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Fri, 14 Jan 2011 01:52:00 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> Message-ID: <4D2F9E30.2090703@qustodium.net> Hello Chris: On 12/01/2011 10:39, Chris Wilson wrote: > On Wed, 12 Jan 2011, Stewart Adam wrote: >> On 2011/01/11 7:20 PM, Chris Wilson wrote: >>> I've implemented a rate limiting option in trunk. Simply set >>> MaxUploadRate = 8000 (bytes per second) or similar on the client (in >>> bbackupd.conf) to use it. Please have a look and see if it meets your >>> needs. Thank you so much for your work on this: I will build Windows and OS X clients and do some testing! This takes care of one of the few remaining issues with Box Backup. IMHO, Box Backup could become *the* FLOSS alternative to Mozy/Crashplan/SugarSync/..., there are only a few features missing before it becomes very very useful to "average" end-users: great job! My (personal, short) list of features that BB still needs for prime time: 1/ Bandwidth throttling: now done thanks to Chris, fantastic! 2/ Built-in VSS: I have collected everything I can find (including existing VSS code from Bacula) in the ticket's URL , but unfortunately I am not able to put the pieces together and implement this myself, although it appears to be rather straightforward. Charles, is there a chance that perhaps your Windows skills can help out? 3/ Restore folder dates: "Nice to have" 4/ Improved Boxi UI: we need a great way to show/select all existing versions of a file, and the GUI needs to be translated. Average users will restore via the GUI, so this has to be as simple as possible. I will also look into WX Widgets i18n and l10n and report back. If somebody has experience in that area, help or advise would be very welcome. Once the framework is in place, I am sure that we will receive several language files for inclusion. 5/ Resume syncs for big files: like throttling, this is needed for DSL, otherwise uploading a really big file might never finish when the DSL line is cut and the sync starts again at the beginning of BIGFILE. 6/ Fix 2GB limitation: Does Charles' changeset address this, as stated in his commit log? 7/ Snapshots ("restore important_contract.doc from 10 June 2008"): Seems impossible with the current algorithm: As suggested before, I would have no problems giving up backwards compatibility for that functionality. Default behaviour could be "as is", and sysadmins could switch on snapshots Looking at the active tickets [1], it appears that no real showstopper bugs remain. I was wondering whether it would be helpful to start-up a discussion (moderated by Chris assuming you are interested) that would get some user feedback on some of the remaining features and methods of implementation. For instance, would it be sensible to have a discussion about what feature set should become "Version 1.0" of Box Backup? Put another way, do you have any *rough* ETA about when the next version may be ready and stable for use, and what would be included? Best regards, Achim [1] -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From brendon at netcal.com Fri Jan 14 08:57:01 2011 From: brendon at netcal.com (Brendon Baumgartner) Date: Fri, 14 Jan 2011 08:57:01 +0000 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D2F9E30.2090703@qustodium.net> Message-ID: Achim- Can you add to the list a page on wikipedia? They have a few matrices comparing products and box backup isn't on any of them. It's free advertising for the project. I tried adding the page once, but it was deleted because they said I took content from boxbackup.org (which I did), but I don't think you guys would have minded. :) -Brendon On 1/13/11 4:52 PM, "Achim J. Latz" wrote: >Hello Chris: > >On 12/01/2011 10:39, Chris Wilson wrote: >> On Wed, 12 Jan 2011, Stewart Adam wrote: >>> On 2011/01/11 7:20 PM, Chris Wilson wrote: >>>> I've implemented a rate limiting option in trunk. Simply set >>>> MaxUploadRate = 8000 (bytes per second) or similar on the client (in >>>> bbackupd.conf) to use it. Please have a look and see if it meets your >>>> needs. > >Thank you so much for your work on this: I will build Windows and OS X >clients and do some testing! > >This takes care of one of the few remaining issues with Box Backup. >IMHO, Box Backup could become *the* FLOSS alternative to >Mozy/Crashplan/SugarSync/..., there are only a few features missing >before it becomes very very useful to "average" end-users: great job! > >My (personal, short) list of features that BB still needs for prime time: > >1/ Bandwidth throttling: now done thanks to Chris, fantastic! > >2/ Built-in VSS: I have collected everything I can find (including >existing VSS code from Bacula) in the ticket's URL >, but unfortunately I am not >able to put the pieces together and implement this myself, although it >appears to be rather straightforward. Charles, is there a chance that >perhaps your Windows skills can help out? > >3/ Restore folder dates: "Nice to have" > >4/ Improved Boxi UI: we need a great way to show/select all existing >versions of a file, and the GUI needs to be translated. Average users >will restore via the GUI, so this has to be as simple as possible. I >will also look into WX Widgets i18n and l10n and report back. If >somebody has experience in that area, help or advise would be very >welcome. Once the framework is in place, I am sure that we will receive >several language files for inclusion. > >5/ Resume syncs for big files: like throttling, this is needed for DSL, >otherwise uploading a really big file might never finish when the DSL >line is cut and the sync starts again at the beginning of BIGFILE. > >6/ Fix 2GB limitation: Does Charles' changeset address this, as stated >in his commit log? > >7/ Snapshots ("restore important_contract.doc from 10 June 2008"): Seems >impossible with the current algorithm: As suggested before, I would have >no problems giving up backwards compatibility for that functionality. >Default behaviour could be "as is", and sysadmins could switch on >snapshots > >Looking at the active tickets [1], it appears that no real showstopper >bugs remain. I was wondering whether it would be helpful to start-up a >discussion (moderated by Chris assuming you are interested) that would >get some user feedback on some of the remaining features and methods of >implementation. For instance, would it be sensible to have a discussion >about what feature set should become "Version 1.0" of Box Backup? > >Put another way, do you have any *rough* ETA about when the next version >may be ready and stable for use, and what would be included? > >Best regards, Achim W > From linus at brimstedt.se Fri Jan 14 10:30:54 2011 From: linus at brimstedt.se (Linus Brimstedt) Date: Fri, 14 Jan 2011 11:30:54 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D2F9E30.2090703@qustodium.net> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> Message-ID: Hello, Nice initiative! > > 4/ Improved Boxi UI: we need a great way to show/select all existing > versions of a file, and the GUI needs to be translated. Average users will > restore via the GUI, so this has to be as simple as possible. I will also > look into WX Widgets i18n and l10n and report back. If somebody has > experience in that area, help or advise would be very welcome. Once the > framework is in place, I am sure that we will receive several language files > for inclusion. > > Is there any GUI today that is under active development? Last time I checked, the projects seemed quite dead to me (perhaps they are just stable ;) I was toying myself with the idea of creating an UI using Java or making a web based one. However, I have not even started to look at this.. /Linus -------------- next part -------------- An HTML attachment was scrubbed... URL: From wouter.van.eekelen at serverffs.com Fri Jan 14 10:32:48 2011 From: wouter.van.eekelen at serverffs.com (Wouter van Eekelen) Date: Fri, 14 Jan 2011 11:32:48 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> Message-ID: I'm actually making a web based UI, but I'm afraid it won't be open sourced because I'm building it for a company. On Fri, Jan 14, 2011 at 11:30 AM, Linus Brimstedt wrote: > Hello, > > Nice initiative! > > > >> >> 4/ Improved Boxi UI: we need a great way to show/select all existing >> versions of a file, and the GUI needs to be translated. Average users will >> restore via the GUI, so this has to be as simple as possible. I will also >> look into WX Widgets i18n and l10n and report back. If somebody has >> experience in that area, help or advise would be very welcome. Once the >> framework is in place, I am sure that we will receive several language files >> for inclusion. >> >> > Is there any GUI today that is under active development? > Last time I checked, the projects seemed quite dead to me (perhaps they are > just stable ;) > > I was toying myself with the idea of creating an UI using Java or making a > web based one. However, I have not even started to look at this.. > > > > > /Linus > > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at invis.net Fri Jan 14 11:47:31 2011 From: lists at invis.net (Charles Lecklider) Date: Fri, 14 Jan 2011 11:47:31 +0000 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D2F9E30.2090703@qustodium.net> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> Message-ID: <4D3037D3.7080300@invis.net> On 14/01/2011 00:52, Achim J. Latz wrote: > 2/ Built-in VSS: I have collected everything I can find (including > existing VSS code from Bacula) in the ticket's URL > , but unfortunately I am not > able to put the pieces together and implement this myself, although it > appears to be rather straightforward. Charles, is there a chance that > perhaps your Windows skills can help out? There are a number of things that need to be done before I can add VSS, and a few things to consider while adding it. In no particular order: + I need to finish the 'new' emu code and check it in. The 'old' code was a good start but it'd become impossible to maintain. It was OK when there was just a 32bit mingw version, but I've now got an XP-specific 32 and 64-bit versions, and Vista/Win7 32 and 64-bit versions. Motivation is the issue here.... + XP does VSS in a different way than anything else. + I want to get my changes merged back into trunk first, otherwise my branch risks becoming a fork. + Right now I'm more interested in a GUI than VSS (see below). + I want the VSS code to be BSD, not GPL. > 4/ Improved Boxi UI: we need a great way to show/select all existing > versions of a file, and the GUI needs to be translated. Average users > will restore via the GUI, so this has to be as simple as possible. I > will also look into WX Widgets i18n and l10n and report back. If > somebody has experience in that area, help or advise would be very > welcome. Once the framework is in place, I am sure that we will receive > several language files for inclusion. I have a working MMC snap-in (not open source), but due to the license change it's on hold. I either need to find the motivation to write some IPC code to talk to bbackupquery, or lib/backupclient needs to be LGPL, or some other license modification to allow linking. -C From maillist at diffingo.com Fri Jan 14 17:30:51 2011 From: maillist at diffingo.com (Stewart Adam) Date: Fri, 14 Jan 2011 12:30:51 -0500 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> Message-ID: <4D30884B.5090509@diffingo.com> On 2011/01/14 5:30 AM, Linus Brimstedt wrote: > 4/ Improved Boxi UI: we need a great way to show/select all existing > versions of a file, and the GUI needs to be translated. Average users > will restore via the GUI, so this has to be as simple as possible. I > will also look into WX Widgets i18n and l10n and report back. If > somebody has experience in that area, help or advise would be very > welcome. Once the framework is in place, I am sure that we will receive > several language files for inclusion. > > > Is there any GUI today that is under active development? > Last time I checked, the projects seemed quite dead to me (perhaps they are > just stable ;) I am currently rewriting my backup application, fwbackups, into C++ and I am planning on using Box Backup as a backend. I would like my application to target mainly desktop users so my goal is to make the UI extremely simple and easy to use. Although I still have to create the restore UI, I've (mostly) completed the user and server administrator applications. The Qt UI files here: https://github.com/firewing1/fwbackups/tree/cplusplus I don't have any screenshots online yet, but I could post some if you don't have the Qt developer tools installed. Cheers, Stewart From orzel at freehackers.org Sat Jan 15 09:42:46 2011 From: orzel at freehackers.org (Thomas Capricelli) Date: Sat, 15 Jan 2011 10:42:46 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D2F9E30.2090703@qustodium.net> References: <4D0FD12F.1070901@gmx.net> <4D2F9E30.2090703@qustodium.net> Message-ID: <201101151042.46489.orzel@freehackers.org> Hi all, On Friday 14 January 2011 01:52:00 Achim J. Latz wrote: > My (personal, short) list of features that BB still needs for prime time: While we are at it, here is my (personal, short) list * Snapshotting, which is, i think what those tickets are about: http://www.boxbackup.org/trac/ticket/62 http://www.boxbackup.org/trac/ticket/52 Other stuff is (very much) nice to have, like bandwidth throttling, 2GB limitation or GUI, but weight really small compared to this feature. While performing my big comparison study of backup solutions, I've been very impressed by boxbackup and it was my final choice. Only after this did i notice that i could not request "restore those files/dirs as they were at given date/time". This seems so obvious a requirement that i did not check for this feature. Now, I understand that with boxbackup design this was not that 'obvious'. I think that this feature is, by far, the most important one missing. Is it possible to create an account on trac ? I'd like to subscribe to tickets to be notified when something happens, but i can't find how to create a login. For whoever is interested in creating a GUI, please dont do java or wxwidgets and use a proper multiplatform widgets set such as gtk, or, even better, Qt. best regards, Thomas -- Thomas Capricelli http://www.freehackers.org/thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From wouter.van.eekelen at serverffs.com Sat Jan 15 13:03:48 2011 From: wouter.van.eekelen at serverffs.com (Wouter van Eekelen) Date: Sat, 15 Jan 2011 14:03:48 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <201101151042.46489.orzel@freehackers.org> References: <4D0FD12F.1070901@gmx.net> <4D2F9E30.2090703@qustodium.net> <201101151042.46489.orzel@freehackers.org> Message-ID: I haven't finished my testing of box backup yet, but it isn't possible to restore files/dirs at a given time?! Like Thomas said, this seemed so obvious I didn't check for it either. On Sat, Jan 15, 2011 at 10:42 AM, Thomas Capricelli wrote: > Hi all, > > On Friday 14 January 2011 01:52:00 Achim J. Latz wrote: > > > My (personal, short) list of features that BB still needs for prime time: > > While we are at it, here is my (personal, short) list > > * Snapshotting, which is, i think what those tickets are about: > > http://www.boxbackup.org/trac/ticket/62 > > http://www.boxbackup.org/trac/ticket/52 > > Other stuff is (very much) nice to have, like bandwidth throttling, 2GB > limitation or GUI, but weight really small compared to this feature. > > While performing my big comparison study of backup solutions, I've been > very impressed by boxbackup and it was my final choice. > > Only after this did i notice that i could not request "restore those > files/dirs as they were at given date/time". This seems so obvious a > requirement that i did not check for this feature. Now, I understand that > with boxbackup design this was not that 'obvious'. I think that this feature > is, by far, the most important one missing. > > Is it possible to create an account on trac ? I'd like to subscribe to > tickets to be notified when something happens, but i can't find how to > create a login. > > For whoever is interested in creating a GUI, please dont do java or > wxwidgets and use a proper multiplatform widgets set such as gtk, or, even > better, Qt. > > best regards, > > Thomas > > -- > > Thomas Capricelli > > http://www.freehackers.org/thomas > > > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brendon at netcal.com Mon Jan 17 04:04:01 2011 From: brendon at netcal.com (Brendon Baumgartner) Date: Mon, 17 Jan 2011 04:04:01 +0000 Subject: [Box Backup] Server Running on Solaris? Message-ID: Does anyone run a server on Solaris? I've had some mixed results compiling for Solaris and want to make sure this is possible before committing to converting my server to Solaris and spending time doing the conversion. Any Solaris servers out there? Running with any problems? -Brendon -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at fluffy.co.uk Mon Jan 17 12:06:09 2011 From: ben at fluffy.co.uk (Ben Summers) Date: Mon, 17 Jan 2011 12:06:09 +0000 Subject: [Box Backup] Server Running on Solaris? In-Reply-To: References: Message-ID: <8ECEBB89-D35B-4958-8171-8C78D58A8B1D@fluffy.co.uk> On 17 Jan 2011, at 12:00, Brendon Baumgartner wrote: > > > Does anyone run a server on Solaris? I've had some mixed results compiling for Solaris and want to make sure this is possible before committing to converting my server to Solaris and spending time doing the conversion. > > Any Solaris servers out there? Running with any problems? I've been running Box Backup servers and clients on Solaris 10 for a few years. If you find it doesn't like the Solaris OpenSSL, get another one installed, and use a command line like ./configure --with-ssl-headers=/opt/csw/include/openssl --with-ssl-lib=/opt/csw/lib LDFLAGS="-R /opt/csw/lib" --prefix=/opt/boxbackup There should be an SMF manifest in SVN somewhere. Ben -- http://bens.me.uk From james at netinertia.co.uk Mon Jan 17 15:32:34 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Mon, 17 Jan 2011 15:32:34 +0000 Subject: [Box Backup] Server Running on Solaris? In-Reply-To: <8ECEBB89-D35B-4958-8171-8C78D58A8B1D@fluffy.co.uk> References: <8ECEBB89-D35B-4958-8171-8C78D58A8B1D@fluffy.co.uk> Message-ID: <20110117153234.GU21506@netinertia.co.uk> On Mon, Jan 17, 2011 at 12:06:09PM +0000, Ben Summers wrote: > > On 17 Jan 2011, at 12:00, Brendon Baumgartner > wrote: > > > > > > Does anyone run a server on Solaris? I've had some mixed results > > compiling for Solaris and want to make sure this is possible before > > committing to converting my server to Solaris and spending time > > doing the conversion. > > > > Any Solaris servers out there? Running with any problems? > > > I've been running Box Backup servers and clients on Solaris 10 for a > few years. > > If you find it doesn't like the Solaris OpenSSL, get another one > installed, and use a command line like I haven't tried on a very recent Solaris 10, but I suspect for stability reasons the sitation is the same; the OpenSSL that ships with Solaris 10 is a little broken, so it's definitely best to install another version. Personally I use Sunfreeware, but Blastwave/OpenCSW probably works too. You'll then need to follow as per Ben's instructions below. (This is also documented on the wiki.) > ./configure --with-ssl-headers=/opt/csw/include/openssl > --with-ssl-lib=/opt/csw/lib LDFLAGS="-R /opt/csw/lib" > --prefix=/opt/boxbackup > > There should be an SMF manifest in SVN somewhere. There is (written by you :-). I keep meaning to try it on OpenIndiana but haven't had time. James From achim+box at qustodium.net Mon Jan 17 22:26:19 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Mon, 17 Jan 2011 23:26:19 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: Message-ID: <4D34C20B.2060209@qustodium.net> Hello Brendon: On 14/01/2011 09:57, Brendon Baumgartner wrote: > Can you add to the list a page on wikipedia? They have a few matrices > comparing products and box backup isn't on any of them. About 2 years ago, I had the same experience and found out about BoxiBackup while reading a Usenix paper [0] that mainly talks about another remote backup system called Cumulus. > It's free > advertising for the project. I tried adding the page once, but it was > deleted because they said I took content from boxbackup.org (which I did), > but I don't think you guys would have minded. :) I agree that spreading the word about BoxiBackup is important, this might even attract additional developer talent. Perhaps you can pick this up directly with the current copyright holders for the web page ("Content ? Ben Summers and Chris Wilson, 2003-2008") so that you get a permission to use content under the license that Wikipedia requires? It would appear to be a waste of resources to redo a page that you already completed... Best regards, Achim [0] Project home page -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From achim+box at qustodium.net Mon Jan 17 22:36:45 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Mon, 17 Jan 2011 23:36:45 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> Message-ID: <4D34C47D.3080603@qustodium.net> Hello Linus: On 14/01/2011 11:30, Linus Brimstedt wrote: > Nice initiative! Thanks, I hope we can get some tangible results (i.e. code or documentation or ...) as a result of this discussion! > Is there any GUI today that is under active development? > Last time I checked, the projects seemed quite dead to me (perhaps they > are just stable ;) Boxi is a good step in the right direction. There have been some small developments as recent as yesterday [0], but IMHO Boxi is still: a) far more unstable than bbackupquery (crashes *frequently*, certain non-ASCII characters are only one of the triggers) b) using a UI that is still too complicated (too many tabs etc., see also my mock-ups for Boxi NG in this group) There are some other GUI options such as Adelin [1], but like you said, they all seem abandoned. I think it is very unfortunate that the power of BB is limited to power users at the moment. The lack of simple & intuitive GUI programs makes restore and changing existing configurations almost impossible for non-experts, even with the current Boxi versions. My limitation is that I have no idea about wxwidgets and advanced C++, so I basically cannot commit any code to Boxi to facilitate moving in that direction. Just today however, I did manage to get some rudimentary translation code working for Boxi (screen shots available, but this list has a limit on per-message size). I will post a separate message for that topic. Best regards, Achim [0] [1] -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From achim+box at qustodium.net Mon Jan 17 22:42:43 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Mon, 17 Jan 2011 23:42:43 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D30884B.5090509@diffingo.com> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D30884B.5090509@diffingo.com> Message-ID: <4D34C5E3.3020400@qustodium.net> Hello Stewart: On 14/01/2011 18:30, Stewart Adam wrote: > I am currently rewriting my backup application, fwbackups, into C++ and > I am planning on using Box Backup as a backend. I would like my > application to target mainly desktop users so my goal is to make the UI > extremely simple and easy to use. That would be amazing. I like the way Duplicati [0] works. > Although I still have to create the restore UI, I've (mostly) completed > the user and server administrator applications. The Qt UI files here: > https://github.com/firewing1/fwbackups/tree/cplusplus > > I don't have any screenshots online yet, but I could post some if you > don't have the Qt developer tools installed. Will this look similar to the existing fwbackups [1]? Otherwise, it would be great if you could post some screenshots on the web. Last question: Do you have a timeline for a perhaps not final, but at least working restore UI? Best regards, Achim [0] [1] -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From achim+box at qustodium.net Mon Jan 17 22:56:06 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Mon, 17 Jan 2011 23:56:06 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D3037D3.7080300@invis.net> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D3037D3.7080300@invis.net> Message-ID: <4D34C906.3060601@qustodium.net> Hello Charles: I reordered your message slightly to address the various issues at once: On 14/01/2011 12:47, Charles Lecklider wrote: > There are a number of things that need to be done before I can add VSS, > and a few things to consider while adding it. In no particular order: > [..] > > + XP does VSS in a different way than anything else. > [..] > + I want the VSS code to be BSD, not GPL. If you have a look at the VSS ticket , there is some documentation on how Bacula created a GPL interface that abstracts those differences away. The internal working has to be different of course, but their implementation (mostly reflected in the above ticket) seems pretty "pretty". Perhaps an inspiration? Is there a specific reason you prefer BSD, even if the cost is having to reimplement the VSS support classes? At the same time, this would of course limit the usefulness of the GPL-"branch" of Box Backup, if you cannot distribute it with the VSS component. I understand that Chris prefers the GPL to BSD, so this might be an impasse for that particular functionality. How can we resolve that? Also: since you seem to be very interested in Windows clients, how do you work around the current lack of VSS? > + I want to get my changes merged back into trunk first, otherwise my > branch risks becoming a fork. I guess that this is something that is Ben's/Chris' call? > + Right now I'm more interested in a GUI than VSS (see below). > > I have a working MMC snap-in (not open source), but due to the license > change it's on hold. I either need to find the motivation to write some > IPC code to talk to bbackupquery, or lib/backupclient needs to be LGPL, > or some other license modification to allow linking. I think I don't understand how the current licensing restricts you. Are you trying to say that you would like to share the MMC (Windows-specific, I presume) with the project if it could be BSD-licensed? What about dual-licensing it, since you are the copyright holder? Under the Box Backup project context, the MMC is GPL and can be linked with all components. For your clients, since you are the copyright holder, you can pick any license you like, using the "BSD"-licensed version of Box Backup for linking. Best regards, Achim -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From achim+box at qustodium.net Mon Jan 17 22:58:43 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Mon, 17 Jan 2011 23:58:43 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> Message-ID: <4D34C9A3.3000003@qustodium.net> Hello Wouter: On 14/01/2011 11:32, Wouter van Eekelen wrote: > I'm actually making a web based UI, but I'm afraid it won't be open > sourced because I'm building it for a company. Can you give us an idea of how you will manage around the need to have the (local, client-specific and secret) key to decrypt the data on the Box Backup server? Will your webinterface require the user to upload the key for temporary decryption on the server side? Best regards, Achim -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From james at netinertia.co.uk Mon Jan 17 23:09:01 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Mon, 17 Jan 2011 23:09:01 +0000 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D34C906.3060601@qustodium.net> References: <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D3037D3.7080300@invis.net> <4D34C906.3060601@qustodium.net> Message-ID: <20110117230900.GV21506@netinertia.co.uk> On Mon, Jan 17, 2011 at 11:56:06PM +0100, Achim J. Latz wrote: > On 14/01/2011 12:47, Charles Lecklider wrote: > > + I want the VSS code to be BSD, not GPL. > [...] > Is there a specific reason you prefer BSD, even if the cost is having to > reimplement the VSS support classes? At the same time, this would of > course limit the usefulness of the GPL-"branch" of Box Backup, if you > cannot distribute it with the VSS component. Personally I prefer BSDL too. Although I appreciate for the purposes of this project my contributions in code have been very minor so I didn't object too much to the GPL change. > > I have a working MMC snap-in (not open source), but due to the license > > change it's on hold. I either need to find the motivation to write some > > IPC code to talk to bbackupquery, or lib/backupclient needs to be LGPL, > > or some other license modification to allow linking. > > I think I don't understand how the current licensing restricts you. Are > you trying to say that you would like to share the MMC > (Windows-specific, I presume) with the project if it could be > BSD-licensed? What about dual-licensing it, since you are the copyright > holder? Under the Box Backup project context, the MMC is GPL and can be > linked with all components. For your clients, since you are the > copyright holder, you can pick any license you like, using the > "BSD"-licensed version of Box Backup for linking. The issue is that the MMC snap-in is closed source. This is incompatible with GPL but fine for BSDL. James From brendon at netcal.com Mon Jan 17 23:12:27 2011 From: brendon at netcal.com (Brendon Baumgartner) Date: Mon, 17 Jan 2011 23:12:27 +0000 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D34C20B.2060209@qustodium.net> References: <4D34C20B.2060209@qustodium.net> Message-ID: > -----Original Message----- > Sent: Monday, January 17, 2011 2:26 PM > To: Box Backup > Subject: Re: [Box Backup] Development feedback & discussion for Box > Backup 1.0? > > Hello Brendon: > > On 14/01/2011 09:57, Brendon Baumgartner wrote: > > Can you add to the list a page on wikipedia? They have a few matrices > > comparing products and box backup isn't on any of them. > > About 2 years ago, I had the same experience and found out about > BoxiBackup while reading a Usenix paper [0] that mainly talks about another > remote backup system called Cumulus. > > > It's free > > advertising for the project. I tried adding the page once, but it was > > deleted because they said I took content from boxbackup.org (which I > > did), but I don't think you guys would have minded. :) > > I agree that spreading the word about BoxiBackup is important, this might > even attract additional developer talent. > > Perhaps you can pick this up directly with the current copyright holders for > the web page ("Content (c) Ben Summers and Chris Wilson, 2003-2008") so > that you get a permission to use content under the license that Wikipedia > requires? It would appear to be a waste of resources to redo a page that you > already completed... > > Best regards, Achim > > [0] > ml/index.html> > Project home page Below is the _draft_ page I created that was deleted. Does anyone have experience with Wikipedia? Maybe Chris or James can take this on being their control of the content? That way if there is an issue, they can resolve it quickly. ! style="text-align: left; background: #ececec;" | [[Box Backup]] | [[BSD licenses|BSD]] | {{yes}} | {{yes}} | {{yes}} | {{no}} | {{no}} | {{no}} |- {{About|computer backup software|the penile bone found in some mammals|baculum}} {{Notability|date=January 2010}} {{ Infobox_Software | name = Box Backup | logo = | screenshot = | caption = | collapsible = | developer = Chris Wilson and team | latest release version = 0.11rc8 | latest release date = {{release date and age|2010|07|27}}https://www.boxbackup.org/svn/box/RELEASE/ | latest preview version = svn rev 2818https://www.boxbackup.org/svn/box/trunk/ | latest preview date = | operating_system = [[Cross-platform]] | programming_language = C++ | genre = [[Backup]] | license = [[BSD]] | website = http://www.boxbackup.org/ }} Box Backup is an open source, completely automatic, on-line backup system. It has the following key features: * All backed up data is stored on the server in files on a filesystem - no tape, archive or other special devices are required. * The server is trusted only to make files available when they are required - all data is encrypted and can be decoded only by the original client. This makes it ideal for backing up over an untrusted network (such as the Internet), or where the server is in an uncontrolled environment. * A backup daemon runs on systems to be backed up, and copies encrypted data to the server when it notices changes - so backups are continuous and up-to-date (although traditional snapshot backups are possible too). * Only changes within files are sent to the server, just like rsync, minimising the bandwidth used between clients and server. This makes it particularly suitable for backing up between distant locations, or over the Internet. * It behaves like tape - old file versions and deleted files are available. * Old versions of files on the server are stored as changes from the current version, minimising the storage space required on the server. Files are the server are also compressed to minimise their size. * Choice of backup behaviour - it can be optimised for document or server backup. * It is designed to be easy and cheap to run a server. It has a portable implementation, and optional RAID implemented in userland for reliability without complex server setup or expensive hardware. Box Backup is distributed under a BSD license. == Notes == ==External links== {{portal|Free software}} * [http://www.boxbackup.org/ Box Backup home page] * [http://www.boxbackup.org/trac Box Backup trac page] [[Category:Free software programmed in C++]] [[Category:Free backup software]] From james at netinertia.co.uk Mon Jan 17 23:15:43 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Mon, 17 Jan 2011 23:15:43 +0000 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: <4D34C20B.2060209@qustodium.net> Message-ID: <20110117231542.GW21506@netinertia.co.uk> On Mon, Jan 17, 2011 at 11:12:27PM +0000, Brendon Baumgartner wrote: > {{About|computer backup software|the penile bone found in some mammals|baculum}} Er, what? James From achim+box at qustodium.net Mon Jan 17 23:23:39 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Tue, 18 Jan 2011 00:23:39 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <20110117231542.GW21506@netinertia.co.uk> References: <4D34C20B.2060209@qustodium.net> <20110117231542.GW21506@netinertia.co.uk> Message-ID: <4D34CF7B.9050208@qustodium.net> On 18/01/2011 00:15, James O'Gorman wrote: > On Mon, Jan 17, 2011 at 11:12:27PM +0000, Brendon Baumgartner wrote: >> {{About|computer backup software|the penile bone found in some mammals|baculum}} > > Er, what? LOL :-) I guess this is owed to the Bacula article [1], which sounds very similar to the body part referenced above. Best regards, Achim [1] From chris at qwirx.com Mon Jan 17 23:30:28 2011 From: chris at qwirx.com (Chris Wilson) Date: Mon, 17 Jan 2011 23:30:28 +0000 (GMT) Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <201101151042.46489.orzel@freehackers.org> References: <4D0FD12F.1070901@gmx.net> <4D2F9E30.2090703@qustodium.net> <201101151042.46489.orzel@freehackers.org> Message-ID: Hi Thomas, On Sat, 15 Jan 2011, Thomas Capricelli wrote: > For whoever is interested in creating a GUI, please dont do java or > wxwidgets and use a proper multiplatform widgets set such as gtk, or, > even better, Qt. wxWidgets is cross-platform with native widgets on each platform. gtk is cross-platform but always looks like GTK (non-native in most cases). I don't like the non-native feel of GTK applications like Ethereal on Windows. What's wrong with wxWidgets? I might have chosen QT if it had (at the time) been fully open source for all platforms (as it is now) when I started writing Boxi in 2006. Now I'm afraid it's a bit late to change, and I don't (yet) see a good reason to throw away all the work I've done on Boxi and start again. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Mon Jan 17 23:32:40 2011 From: chris at qwirx.com (Chris Wilson) Date: Mon, 17 Jan 2011 23:32:40 +0000 (GMT) Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D34C20B.2060209@qustodium.net> References: <4D34C20B.2060209@qustodium.net> Message-ID: Hi Achim, On Mon, 17 Jan 2011, Achim J. Latz wrote: > On 14/01/2011 09:57, Brendon Baumgartner wrote: >> Can you add to the list a page on wikipedia? They have a few matrices >> comparing products and box backup isn't on any of them. > > About 2 years ago, I had the same experience and found out about BoxiBackup I think you meant Box Backup, with a space? Boxi is the Box Backup Interface, a separate project. > I agree that spreading the word about BoxiBackup is important, this > might even attract additional developer talent. Which I for one would definitely appreciate and find very motivating! > Perhaps you can pick this up directly with the current copyright holders > for the web page ("Content ? Ben Summers and Chris Wilson, 2003-2008") > so that you get a permission to use content under the license that > Wikipedia requires? It would appear to be a waste of resources to redo a > page that you already completed... I'm very happy to relicense the content on the Wiki if that helps, although I didn't write all of it so the other authors would have to be consulted. Is the license the only reason for not re-using the content on Wikipedia? I've had my work deleted by them for arbitrary reasons before, and stopped contributing as a result. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From lists at invis.net Mon Jan 17 23:37:55 2011 From: lists at invis.net (Charles Lecklider) Date: Mon, 17 Jan 2011 23:37:55 +0000 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <20110117230900.GV21506@netinertia.co.uk> References: <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D3037D3.7080300@invis.net> <4D34C906.3060601@qustodium.net> <20110117230900.GV21506@netinertia.co.uk> Message-ID: <4D34D2D3.6050202@invis.net> On 17/01/2011 23:09, James O'Gorman wrote: > Personally I prefer BSDL too. Although I appreciate for the purposes of > this project my contributions in code have been very minor so I didn't > object too much to the GPL change. Unfortunately I was busy with a project at the time so missed the entire debate.... Still, overall I'm OK with it, but I'd like to make one minor change. > The issue is that the MMC snap-in is closed source. This is incompatible > with GPL but fine for BSDL. Exactly. Having thought about this some more, I'd like to propose that lib/backupclient and lib/backupstore get put under the LGPL. That would let me do what I was doing before the change, yet still allow BB to be included in the various Linux distros that caused the change in the first place. -C From brendon at netcal.com Mon Jan 17 23:39:11 2011 From: brendon at netcal.com (Brendon Baumgartner) Date: Mon, 17 Jan 2011 23:39:11 +0000 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <20110117231542.GW21506@netinertia.co.uk> Message-ID: On 1/17/11 3:15 PM, "James O'Gorman" wrote: >On Mon, Jan 17, 2011 at 11:12:27PM +0000, Brendon Baumgartner wrote: >> {{About|computer backup software|the penile bone found in some >>mammals|baculum}} > >Er, what? > >James LOL. Like I said _draft_. :) I must have taken the page from the baculum page and went through and edited it. From lists at invis.net Mon Jan 17 23:45:42 2011 From: lists at invis.net (Charles Lecklider) Date: Mon, 17 Jan 2011 23:45:42 +0000 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D34C906.3060601@qustodium.net> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D3037D3.7080300@invis.net> <4D34C906.3060601@qustodium.net> Message-ID: <4D34D4A6.3050601@invis.net> On 17/01/2011 22:56, Achim J. Latz wrote: > If you have a look at the VSS ticket > , there is some documentation > on how Bacula created a GPL interface that abstracts those differences > away. The internal working has to be different of course, but their > implementation (mostly reflected in the above ticket) seems pretty > "pretty". Perhaps an inspiration? I intend to start with Win7 support and work backwards if time and motivation allow. There is already enough XP-specific code that I don't see a need to worry about API wrappers etc. > Is there a specific reason you prefer BSD, even if the cost is having to > reimplement the VSS support classes? At the same time, this would of > course limit the usefulness of the GPL-"branch" of Box Backup, if you > cannot distribute it with the VSS component. Not all my projects are open source. I don't want any hassle with other contributors having to assign copyright or grant rights - putting it under BSD just makes things simple. > Also: since you seem to be very interested in Windows clients, how do > you work around the current lack of VSS? By not using BB for things like Exchange or SQL server. Normal data is very rarely a problem. -C From achim+box at qustodium.net Tue Jan 18 00:06:28 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Tue, 18 Jan 2011 01:06:28 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <201101151042.46489.orzel@freehackers.org> References: <4D0FD12F.1070901@gmx.net> <4D2F9E30.2090703@qustodium.net> <201101151042.46489.orzel@freehackers.org> Message-ID: <4D34D984.8080101@qustodium.net> Hello Thomas: On 15/01/2011 10:42, Thomas Capricelli wrote: > While we are at it, here is my (personal, short) list > > * Snapshotting, which is, i think what those tickets are about: > > http://www.boxbackup.org/trac/ticket/62 I believe that this was addressed with a couple of fixes in September 2009, which eventually led to the possibility to turn off housekeeping by setting soft limit := hard limit. To be precise, I am talking about the threads "Soft/hard limits: "Soft limit must be less than the hard limit." on 16/9/2009 and "Mailbox backup is dangerous." in May/August/September 2009. If my understanding is correct and the behaviour is now corrected, perhaps the ticket should be updated and closed to reflect that? > http://www.boxbackup.org/trac/ticket/52 Controlling the size and maintenance of old vs. deleted files seems like a nice-to-have to me, especially given that you can avoid data loss with the setting described above. > While performing my big comparison study of backup solutions, I've been > very impressed by boxbackup and it was my final choice. > > Only after this did i notice that i could not request "restore those > files/dirs as they were at given date/time". This seems so obvious a > requirement that i did not check for this feature. Now, I understand > that with boxbackup design this was not that 'obvious'. I think that > this feature is, by far, the most important one missing. The thread "Mailbox backup is dangerous." in May/August/September 2009 contains some great ideas on how to implement snapshots. Unfortunately, as pointed out back then, I am unable to contribute any code... A thought experiment to eliminate my doubts about this feature: if bb is configured to run every 3600 seconds, does this not create a consistent snapshot at that point in time X (using LVM or VSS for added consistency)? Would it then not be a "simple" matter of adding a flag/timestamp to all files that have changed, perhaps even in a completely external DB? For a snapshot restore at point X, you restore all files that have been tagged with X. To restore the remaining files, use the version of each file which has a timestamp Y < X. Am I missing something? > Is it possible to create an account on trac ? I'd like to subscribe to > tickets to be notified when something happens, but i can't find how to > create a login. Chris can probably help you, perhaps just contact him directly. While you are at it, you could also update the web page to make the client software more visible, as you explained in December. > For whoever is interested in creating a GUI, please dont do java or > wxwidgets and use a proper multiplatform widgets set such as gtk, or, > even better, Qt. As per my other message about the current GUI status, the only currently sort-of-working implementation (Boxi) uses WX. Would you mind sharing why that is such a bad choice, especially compared to the non-existing alternatives that would have to be reimplemented from scratch? Best regards, Achim -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From achim+box at qustodium.net Tue Jan 18 00:11:47 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Tue, 18 Jan 2011 01:11:47 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: <4D34C20B.2060209@qustodium.net> Message-ID: <4D34DAC3.8070001@qustodium.net> Hello Chris: On 18/01/2011 00:32, Chris Wilson wrote: >> About 2 years ago, I had the same experience and found out about >> BoxiBackup > > I think you meant Box Backup, with a space? Boxi is the Box Backup > Interface, a separate project. I am of course aware of that. I was looking for a clever way to avoid writing Box Backup/Boxi as it seems awkward, and at least IMHO, both projects are parts of the same solution. I will refer to the projects by their full name from now on :-) >> I agree that spreading the word about BoxiBackup is important, this >> might even attract additional developer talent. > > Which I for one would definitely appreciate and find very motivating! (-: You are an army of one, Chris! Best regards, Achim -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From chris at qwirx.com Tue Jan 18 00:12:06 2011 From: chris at qwirx.com (Chris Wilson) Date: Tue, 18 Jan 2011 00:12:06 +0000 (GMT) Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: <4D0FD12F.1070901@gmx.net> <4D2F9E30.2090703@qustodium.net> <201101151042.46489.orzel@freehackers.org> Message-ID: Hi Thomas, On Mon, 17 Jan 2011, Chris Wilson wrote: > On Sat, 15 Jan 2011, Thomas Capricelli wrote: > >> For whoever is interested in creating a GUI, please dont do java or >> wxwidgets and use a proper multiplatform widgets set such as gtk, or, >> even better, Qt. > > wxWidgets is cross-platform with native widgets on each platform. gtk is > cross-platform but always looks like GTK (non-native in most cases). I > don't like the non-native feel of GTK applications like Ethereal on > Windows... > > I don't (yet) see a good reason to throw away all the work I've done on > Boxi and start again [with QT]. Having just seen the following post on the mingw-users mailing list: > Qt is nicely cross-platform, but has one major limitation: applications > feel like Qt apps regardless of platform, rather than feeling like > platform-native apps. [https://sourceforge.net/mailarchive/forum.php?thread_name=AANLkTinNsN5HkQpZUL%2BDbV0VkgFD7qMAiskAqFMTbtUy%40mail.gmail.com&forum_name=mingw-users] This makes me think that Qt has the same limitations as GTK in terms of native application feel, which makes me even less inclined to want to switch to it. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From achim+box at qustodium.net Tue Jan 18 00:20:03 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Tue, 18 Jan 2011 01:20:03 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D34D2D3.6050202@invis.net> References: <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D3037D3.7080300@invis.net> <4D34C906.3060601@qustodium.net> <20110117230900.GV21506@netinertia.co.uk> <4D34D2D3.6050202@invis.net> Message-ID: <4D34DCB3.3000701@qustodium.net> Hello Charles: On 18/01/2011 00:37, Charles Lecklider wrote: >> The issue is that the MMC snap-in is closed source. This is incompatible >> with GPL but fine for BSDL. > > Exactly. Having thought about this some more, I'd like to propose that > lib/backupclient and lib/backupstore get put under the LGPL. That would > let me do what I was doing before the change, yet still allow BB to be > included in the various Linux distros that caused the change in the > first place. Am I correct that you are advocating another licensing change in order to be able to distribute your proprietary binary? Since this sub-thread started about existing GUIs, I was under the impression that we are talking about options available to the project. I understand now that you are talking about "a GUI" that can be used with Box Backup, not necessarily one that is include in the project. Best regards, Achim -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From chris at qwirx.com Tue Jan 18 00:29:45 2011 From: chris at qwirx.com (Chris Wilson) Date: Tue, 18 Jan 2011 00:29:45 +0000 (GMT) Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D34C906.3060601@qustodium.net> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D3037D3.7080300@invis.net> <4D34C906.3060601@qustodium.net> Message-ID: Hi all, On Mon, 17 Jan 2011, Achim J. Latz wrote: > On 14/01/2011 12:47, Charles Lecklider wrote: >> I want the VSS code to be BSD, not GPL. > > Is there a specific reason you prefer BSD, even if the cost is having to > reimplement the VSS support classes? At the same time, this would of > course limit the usefulness of the GPL-"branch" of Box Backup, if you > cannot distribute it with the VSS component. > > I understand that Chris prefers the GPL to BSD, so this might be an > impasse for that particular functionality. How can we resolve that? I prefer GPL for code that I write myself, because it provides stronger protection of my rights as the author. I have no objection to using or including code from any other author under any compatible license, nor even changing the license of Boxi or Box Backup if the benefits outweigh the costs. There's no fundamental reason not to include code licensed under the BSD (without advertising clause) or dual-licensed (BSD and GPL) in Box Backup and Boxi, as the BSD license (without advertising clause) is compatible with the GPL. >> I want to get my changes merged back into trunk first, otherwise my >> branch risks becoming a fork. > > I guess that this is something that is Ben's/Chris' call? I would also really like to see Charles' work merged, sooner rather than later. Implementing snapshots and directory attributes will require changes to the directory format, and I don't want to make those changes until after Charles' changes are merged, because they would probably become incompatible forks. Charles has as much right (if not more!) as I do to merge his changes into the trunk, and has not asked me to review and merge them myself as far as I can recall, and I'm currently waiting for him to do one of those two things. >> I have a working MMC snap-in (not open source), but due to the license >> change it's on hold. I either need to find the motivation to write >> some IPC code to talk to bbackupquery, or lib/backupclient needs to be >> LGPL, or some other license modification to allow linking. I wrote only a tiny portion of the code in lib/backupclient, so I don't have much reason to object, except that adding a third license might further complicate the licensing situation. Does anyone have any objections to either: 1. Relicensing lib/backupclient under a dual LGPL-BSD license, or 2. Relicensing lib/backupclient under the BSD license (without advertising clause)? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From lists at invis.net Tue Jan 18 00:40:36 2011 From: lists at invis.net (Charles Lecklider) Date: Tue, 18 Jan 2011 00:40:36 +0000 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D3037D3.7080300@invis.net> <4D34C906.3060601@qustodium.net> Message-ID: <4D34E184.2020406@invis.net> On 18/01/2011 00:29, Chris Wilson wrote: > Does anyone have any objections to either: > > 1. Relicensing lib/backupclient under a dual LGPL-BSD license, or That's my choice, because... > 2. Relicensing lib/backupclient under the BSD license (without > advertising clause)? ... now that the change to GPL has been made I think it makes more sense to keep it than to go backwards. IIRC one of the arguments was that having lib/backupclient and store under BSD would allow someone to use the guts of BB without attribution or contribution. I think that's a very valid point, hence my suggestion for LGPL. -C From lists at invis.net Tue Jan 18 00:45:25 2011 From: lists at invis.net (Charles Lecklider) Date: Tue, 18 Jan 2011 00:45:25 +0000 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D34DCB3.3000701@qustodium.net> References: <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D3037D3.7080300@invis.net> <4D34C906.3060601@qustodium.net> <20110117230900.GV21506@netinertia.co.uk> <4D34D2D3.6050202@invis.net> <4D34DCB3.3000701@qustodium.net> Message-ID: <4D34E2A5.4030307@invis.net> On 18/01/2011 00:20, Achim J. Latz wrote: > Am I correct that you are advocating another licensing change in order > to be able to distribute your proprietary binary? No. Distribution isn't an issue. The issue is whether I waste time writing an IPC interface to bbackupquery to work around the _linking_ restrictions in the GPL, or whether a relatively minor license change can be made. > Since this sub-thread started about existing GUIs, I was under the > impression that we are talking about options available to the project. Ultimately I expect to open the source, but because it shares code with some of my other projects that's not something I can do right now. > I understand now that you are talking about "a GUI" that can be used > with Box Backup, not necessarily one that is include in the project. Correct. -C From maillist at diffingo.com Tue Jan 18 00:43:53 2011 From: maillist at diffingo.com (Stewart Adam) Date: Mon, 17 Jan 2011 19:43:53 -0500 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D34C5E3.3020400@qustodium.net> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D30884B.5090509@diffingo.com> <4D34C5E3.3020400@qustodium.net> Message-ID: <4D34E249.7010509@diffingo.com> On 2011/01/17 5:42 PM, Achim J. Latz wrote: > Hello Stewart: Hi Achim, Duplicati does have a nice way of implementing restores - the UI is very clean. I have some mock UIs finished, but they need a bit of improvement since I'd like to add widgets to select individual files from previous revisions (as opposed to always restoring the most recent version). >> Although I still have to create the restore UI, I've (mostly) completed >> the user and server administrator applications. The Qt UI files here: >> https://github.com/firewing1/fwbackups/tree/cplusplus >> >> I don't have any screenshots online yet, but I could post some if you >> don't have the Qt developer tools installed. > > Will this look similar to the existing fwbackups [1]? Otherwise, it would be > great if you could post some screenshots on the web. Similar to a certain extent... I'm re-using bits I like from the 1.43.x UI but making things even simpler when possible and redesigning the bits that no longer apply (for example, compression as this is done natively with BB). I've uploaded some screenshots of the latest compiles here: http://www.firewing1.com/downloads/fwbackups-cpp-screenshots/ I also included screenshots of the server administrator, which owners of a Box Backup server would use to administrate accounts & distribute keys to users. I still have a few things to do, for example a progress bar indicating the space that the backup will occupy in the "select files" page of the backup configuration but I like how it looks so far. > Last question: Do you have a timeline for a perhaps not final, but at least > working restore UI? I'm hoping within the next month I'll have the UI completely finished. Cheers, Stewart From achim+box at qustodium.net Tue Jan 18 00:45:58 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Tue, 18 Jan 2011 01:45:58 +0100 Subject: [Box Backup] Boxi: First translation available Message-ID: <4D34E2C6.7050309@qustodium.net> Hello list: I created a first, very basic form of internationalisation for Boxi and posted some screen shots for the German versions under Windows and OS X [1]. Remaining tasks (help is much appreciated): - Include in official SVN - Make sure that all strings are covered in translations - Add more languages - Add a language selector GUI and logic to switch language on the fly. Currently wxLANGUAGE_DEFAULT is used to determine the UI language at launch. - Ensure that this feature works in a cross-platform way (probably via autoconf/Makefile magic) - ... Best regards, Achim [1] -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From achim+box at qustodium.net Tue Jan 18 00:53:48 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Tue, 18 Jan 2011 01:53:48 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D34D4A6.3050601@invis.net> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D3037D3.7080300@invis.net> <4D34C906.3060601@qustodium.net> <4D34D4A6.3050601@invis.net> Message-ID: <4D34E49C.5040202@qustodium.net> Hello Charles: On 18/01/2011 00:45, Charles Lecklider wrote: >> Also: since you seem to be very interested in Windows clients, how do >> you work around the current lack of VSS? > > By not using BB for things like Exchange or SQL server. Normal data is > very rarely a problem. Just for the record and from my experience, locked files are actually quite common, even on a Windows workstation/PC: Outlook PST files come to mind, and Firefox/Thunderbird lock files, SQLite DBs used by running programs, and and and... Together, the read warnings those files generate create quite a lot of distraction. The user will quickly ignore those warnings (as there are so many of them), and this will eventually lead them to overlook an important warning. One might actually argue that not being able to back up PST files is a major shortcoming, given the install base of Outlook and the business criticality of email. Perhaps at some point, we will see VSS to solve all this ;-) Best regards, Achim -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From orzel at freehackers.org Tue Jan 18 08:53:17 2011 From: orzel at freehackers.org (Thomas Capricelli) Date: Tue, 18 Jan 2011 09:53:17 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: <4D0FD12F.1070901@gmx.net> <201101151042.46489.orzel@freehackers.org> Message-ID: <201101180953.17748.orzel@freehackers.org> On Tuesday 18 January 2011 00:30:28 Chris Wilson wrote: > wxWidgets is cross-platform with native widgets on each platform. gtk is > cross-platform but always looks like GTK (non-native in most cases). I > don't like the non-native feel of GTK applications like Ethereal on > Windows. What's wrong with wxWidgets? > > I might have chosen QT if it had (at the time) been fully open source for > all platforms (as it is now) when I started writing Boxi in 2006. Now I'm > afraid it's a bit late to change, and I don't (yet) see a good reason to > throw away all the work I've done on Boxi and start again. Hi Chris, Sorry for the wording, i did not mean to be rude or whatever. There are actually two points. First, i thought that boxi was dead. When searching for a GUI for boxbackup, i've found some mentionning of it, but only few and old ones. This and other facts (it's not packaged for gentoo while boxbackup is, for example) lead me to flag "boxi" as a dead project in my research. Now you mention it is somehow alive, i'll give it another try. I was not suggesting to switch to Qt from wx, rather that if some new project would be started, i highly recommend Qt. Switching is not a lighweight decision, and i would not throw this kind of idea in the wild without a serious argumenting. The second point is about wx versus Qt... and i'd like not to enter any flamewar or whatever. The applications i have here(linux) based on wx look ugly, very 80's, and most of the time just does not work (the last one was the 'launcher' of flightgear, i've been told the probleme was with NLS, but i did not really investigate further). Qt is also using the 'native' widgets, as wxwidgets does. The main differece being that wxwidget consider gtk 'native' on unix, while Qt has its own style on unix. But even independantly of the look, I just happen to think that Qt is a much much better platform to develop with. I'm confident it will stay very long and maintained, a lot more people would be able to help, and i'm sure the code would be cleaner, though i have to admit i've never much developed with wx, so i wont blame you if you don't buy this point. I, for example, would help on a Qt client, but i could not on wx. best regards, Thomas -- Thomas Capricelli http://www.freehackers.org/thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From orzel at freehackers.org Tue Jan 18 09:38:52 2011 From: orzel at freehackers.org (Thomas Capricelli) Date: Tue, 18 Jan 2011 10:38:52 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <201101180953.17748.orzel@freehackers.org> References: <4D0FD12F.1070901@gmx.net> <201101180953.17748.orzel@freehackers.org> Message-ID: <201101181038.52977.orzel@freehackers.org> On Tuesday 18 January 2011 09:53:17 Thomas Capricelli wrote: > Now you mention it is somehow alive, i'll give it another try We do speak about http://boxi.sourceforge.net/, right ? Here are a few notes from my tests: * I know the version name can be misleading and should not be relied upon... but still "0.1.0" looks _scary_. * this version was released on may 2005. Well, it's enough for most people to just stop here and label this as 'dead'. * I'm not able to checkout using the (usual sourceforge) line given: % svn co https://svn.sourceforge.net/svnroot/boxi/trunk/boxi svn: Il repository ? stato temporaneamente spostato in 'http://sourceforge.net/apps/trac/sourceforge/wiki/Subversion' (rough translation : the repository has been temporarily move) Hence, the following is using the 0.1.0 tarball. * it seems to me boxi includes a copy of boxbackup ?? Does this mean i can not use it with my current install of boxbackup ? * the ./configure passes with no problem, but compilation fails with things like DebugMemLeakFinder.cpp:117:12: error: ?::malloc? has not been declared BackupLocationsPanel.cc:413:63: error: no matching function for call to ?Location::SetName(wxCStrData)? ClientConnection.cc:425:39: error: ?class wxPathList? has no member named ?GetFirst? which seems not hard to fix, but would easily be interpreted by people as a lack of maintainance. DebugMemLeakFinder.cpp:260:4: error: cast from ?void*? to ?int? loses precision which makes me think that this has not been tested on 64 bits. Or has it ? Again, my purpose is not to complain, just to show what a quick look at boxi means today. I understand from this that gentoo packagers did not include boxi. There are really not the touchy kind of guys and i've seen software 'difficult to package' in gentoo before. And I also understand why i've labelled this as dead. Your mail on this list is actually the first hint i have that this is not completely dead. best wishes, Thomas -- Thomas Capricelli http://www.freehackers.org/thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at fluffy.co.uk Tue Jan 18 10:34:19 2011 From: ben at fluffy.co.uk (Ben Summers) Date: Tue, 18 Jan 2011 10:34:19 +0000 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: References: Message-ID: On 17 Jan 2011, at 23:36, Chris Wilson wrote: > On Mon, 17 Jan 2011, Achim J. Latz wrote: > >> Perhaps you can pick this up directly with the current copyright holders >> for the web page ("Content ? Ben Summers and Chris Wilson, 2003-2008") >> so that you get a permission to use content under the license that >> Wikipedia requires? It would appear to be a waste of resources to redo a >> page that you already completed... > > I'm very happy to relicense the content on the Wiki if that helps, > although I didn't write all of it so the other authors would have to be > consulted. I am also happy to relicense the content on the Wiki and any I wrote on the main web site, with a license of Chris Wilson's choosing. Ben -- http://bens.me.uk From achim+box at qustodium.net Tue Jan 18 10:47:52 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Tue, 18 Jan 2011 11:47:52 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <201101181038.52977.orzel@freehackers.org> References: <4D0FD12F.1070901@gmx.net> <201101180953.17748.orzel@freehackers.org> <201101181038.52977.orzel@freehackers.org> Message-ID: <4D356FD8.30407@qustodium.net> Hello Thomas: On 18/01/2011 10:38, Thomas Capricelli wrote: > * I know the version name can be misleading and should not be relied > upon... but still "0.1.0" looks _scary_. I agree, and the stability has still some way to go. However, the current SVN snapshot is a lot more advanced than 0.1.0. > * this version was released on may 2005. Well, it's enough for most > people to just stop here and label this as 'dead'. > > * I'm not able to checkout using the (usual sourceforge) line given: You can have a look at my step-by-step instructions [0, 1] to compile a fresh Boxi binary yourself. As pointed out in my recent message about translating Boxi, the process (for MingW certainly) works as of yesterday (-: Your feedback and bug reports are very welcome! Best regards, Achim [0] [1] -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From orzel at freehackers.org Tue Jan 18 14:16:55 2011 From: orzel at freehackers.org (Thomas Capricelli) Date: Tue, 18 Jan 2011 15:16:55 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D356FD8.30407@qustodium.net> References: <4D0FD12F.1070901@gmx.net> <201101181038.52977.orzel@freehackers.org> <4D356FD8.30407@qustodium.net> Message-ID: <201101181516.55408.orzel@freehackers.org> Well, i gave it a try.... i can checkout/autogen.sh/configure, but make fails with lot of error lines. I guess there's at least a compatibility problem with my wxwidgets (2.9.1). Thomas -- Thomas Capricelli http://www.freehackers.org/thomas On Tuesday 18 January 2011 11:47:52 Achim J. Latz wrote: > You can have a look at my step-by-step instructions [0, 1] to compile a > fresh Boxi binary yourself. As pointed out in my recent message about > translating Boxi, the process (for MingW certainly) works as of > yesterday (-: > > Your feedback and bug reports are very welcome! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jgoerzen at complete.org Wed Jan 19 18:49:39 2011 From: jgoerzen at complete.org (John Goerzen) Date: Wed, 19 Jan 2011 12:49:39 -0600 Subject: [Box Backup] Some boxbackup questions for 2011 Message-ID: <4D373243.3060501@complete.org> Hi folks, Thanks for your hard work on Boxbackup. I think it sounds like it has some excellent features, and I am (again) considering it for my use. Back in 2008, I was doing the same thing and asked some questions [1] on this mailing list. The main things I asked about included: 1. not storing of hard links 2. using mtime only, and not ctime, to identify file changes 3. not handling sparse files well 4. didn't back up block and character devices #1, #2, and #4 particularly made it unsuitable for my needs. I'm trying to research this again now, both for my personal needs and for my needs at work. I found bug #63 [2] indicating that hard links are still a problem. I heavily use hard links in my data files -- they are great for, for instance, preparing ISO images from a subset of my photos -- and also want to back up my entire system. The FAQ indicates that problem #2 still persists, though I wonder if the new inotify support may partially address that problem. The FeatureRequests page indicates #4 is still a problem as well. I'd like to ask about a few other things now: 1) The deduplication that Boxbackup does -- is it a block-level deduplication that spans the entire server repo? (That is, if 100MB out of a 1GB file on server A matches 100MB in the middle of a 2GB file on server B, that 100MB data will be stored only once) 2) I noticed the warning at https://www.boxbackup.org/trac/wiki/Installation and the corresponding ticket at https://www.boxbackup.org/trac/ticket/62 . That seems serious to me, but I wonder I could work around it pretty easily using bind mounts. Say I make a bind mount of /home to /backup/2011-01-19/home for today, and have /backup be backed up. Then I delete that bind mount, and tomorrow make one to /backup/2011-01-20/home, and so forth. In theory, if Boxbackup uses deduplication, this wouldn't cause any extra storage to be consumed. The question is: would it cause boxbackup to not consider a deleted file for deletion right away since it would still be in the backup as an old file at a different path? 3) I am interested in offsite backups over the Internet onto a service such as Amazon S3 or Rackspace Cloud Files. I have observed the warnings about this both in the wiki and the mailing lists. One question would be: could I achieve this using the RAID features, telling Boxbackup to use a local disk for all reads but to replicate writes, or would this still be inefficient? If the latter, would it be sensible to rsync the data store to a FUSE-mounted cloud fs? Thanks again for your hard work on this tool. I think it comes very close to being what I'm looking for. -- John [1] http://lists.boxbackup.org/pipermail/boxbackup/2008-April/004588.html [2] http://boxbackup.org/trac/ticket/63 From chris at qwirx.com Wed Jan 19 21:15:17 2011 From: chris at qwirx.com (Chris Wilson) Date: Wed, 19 Jan 2011 21:15:17 +0000 (GMT) Subject: [Box Backup] Some boxbackup questions for 2011 In-Reply-To: <4D373243.3060501@complete.org> References: <4D373243.3060501@complete.org> Message-ID: Hi John, On Wed, 19 Jan 2011, John Goerzen wrote: > Back in 2008, I was doing the same thing and asked some questions [1] on > this mailing list. The main things I asked about included: > > 1. not storing of hard links This would be difficult to do with the current code base. I'm slowly working on implementing reference counting, and when that's done, it should be much easier to implement full support for hard links without duplicating data, as well as implementing snapshots. However it's currently hard to do properly. Given your stated purpose: > I found bug #63 [2] indicating that hard links are still a problem. I > heavily use hard links in my data files -- they are great for, for > instance, preparing ISO images from a subset of my photos -- and also > want to back up my entire system. Would it be enough to exclude the directories where you create these ISO images from the backup? I'm afraid that Box Backup was never designed for full system backups, and it lacks some features that would be important for this, such as cross-filesystem backups and device files. The latter would not be hard to add, but until snapshots are implemented, it doesn't feel that useful to implement full-system backups. > 2. using mtime only, and not ctime, to identify file changes This is a great idea and should be easy to implement. The only issue would be if you run on a filesystem with atime enabled (the default on most systems), and you regularly run jobs that access the contents of all files, such as backups. In this case, Box Backup would end up endlessly comparing file contents, wasting a lot of CPU time, disk and bandwidth. I think it's easier or more sensible to fix the programs that mess with the mtime than to work around their brokenness in almost all cases (truecrpyt being the only exception that I can think of). > 3. not handling sparse files well This should not be hard to fix, on systems where an ioctl exists to identify the holes in sparse files, and if we implement logic to skip large blocks of zeroes whene restoring files. Even if not, I don't see it as a huge problem because we do use compression and sync compressed blocks, so > The FAQ indicates that problem #2 still persists, though I wonder if the > new inotify support may partially address that problem. inotify is not implemented yet, unfortunately. However it is another, good way to solve that problem. > 1) The deduplication that Boxbackup does -- is it a block-level > deduplication that spans the entire server repo? (That is, if 100MB > out of a 1GB file on server A matches 100MB in the middle of a 2GB > file on server B, that 100MB data will be stored only once) It's only within the current file at the moment. We could implement a global block cache without too much difficulty if it seems to be useful, at some small cost to security (an attacker who compromises the server could identify blocks shared between files). Also, it would not be between "server A" and "server B" in your example, as they probably have no idea about each other, but between "client A" and "client B" on the same server. > 2) I noticed the warning at > https://www.boxbackup.org/trac/wiki/Installation and the corresponding > ticket at https://www.boxbackup.org/trac/ticket/62 . That seems serious > to me, but I wonder I could work around it pretty easily using bind > mounts. You could also work around it by setting the soft limit equal to the hard limit, which causes the client to stop backing up files when it hits the limit, and then implementing a custom client that marks some older files for immediate deletion according to whatever policy you choose. Ultimately one major purpose of and reason for implementing snapshots is to give the client complete control over data retention on the server, by deleting whichever snapshots they no longer need access to, discarding all files and patches that are only contained in that snapshot in one go. However this is a challenging task. Most bug reports in rdiff-backup relate to corruption of the snapshot (reverse patch) repository, and they don't even implement encryption, as we do. > Say I make a bind mount of /home to /backup/2011-01-19/home for > today, and have /backup be backed up. Then I delete that bind mount, > and tomorrow make one to /backup/2011-01-20/home, and so forth. In > theory, if Boxbackup uses deduplication, this wouldn't cause any extra > storage to be consumed. Yes, it will just rename all the matching files under /backup/2011-01-19/home to be under /backup/2011-01-20/home instead. > The question is: would it cause boxbackup to not consider a deleted file > for deletion right away since it would still be in the backup as an old > file at a different path? No, once they are renamed they live permanently at the new path, because there is no reference counting or snapshots. This is another major reason for implementing snapshots. > 3) I am interested in offsite backups over the Internet onto a service > such as Amazon S3 or Rackspace Cloud Files. I have observed the > warnings about this both in the wiki and the mailing lists. There is no official support for it yet, and I gather that people who have tried to make it unofficially by backing up to a mounted fuse-s3 filesystem have run into problems, which doesn't surprise me. I think it's important for data consistency to implement a completely new store format that works well with Amazon S3, and to cope with the fact that the server is simply incapable of converting forward diffs to reverse diffs, so the original versions of files should be retained forever or discarded at great bandwidth cost. There are also missing consistency guarantees in using S3 as a target that make it difficult to implement such a filesystem and know that it will be robust in the face of temporary inconsistency. If you really need this any time soon, I would currently advise you to use an Amazon EBS store, an EC2 instance which mounts it and formats it with a conventional filesystem, and runs bbstored and thus performs the conversion of forward to reverse diffs automatically. Or use cumulus or duplicity instead. > One question would be: could I achieve this using the RAID features, > telling Boxbackup to use a local disk for all reads but to replicate > writes, or would this still be inefficient? I think you would end up rewriting entire files any time a chunk of them changed, unless you use a filesystem on top of S3 such as Jungle Disk, Elephant Drive or similar. > If the latter, would it be sensible to rsync the data store to a > FUSE-mounted cloud fs? I would not recommend it, but it is possible and might work well enough for you. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From achim+box at qustodium.net Wed Jan 19 22:37:37 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Wed, 19 Jan 2011 23:37:37 +0100 Subject: [Box Backup] Updates for OS X launchd file Message-ID: <4D3767B1.4080604@qustodium.net> Hello list: After looking at Apple's developer page for launchd daemons & services [0], I believe that we should add the following to contrib/mac_osx/org.boxbackup.bbackupd.plist.in * * * * org.boxbackup.bbackupd.plist.in * * * * [..] OnDemand RunAtLoad * * * * org.boxbackup.bbackupd.plist.in * * * * Best regards, Achim [0] -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From achim+box at qustodium.net Wed Jan 19 22:53:24 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Wed, 19 Jan 2011 23:53:24 +0100 Subject: [Box Backup] Updates for OS X launchd file In-Reply-To: <4D3767B1.4080604@qustodium.net> References: <4D3767B1.4080604@qustodium.net> Message-ID: <4D376B64.3000004@qustodium.net> Hello again: I just realised that RunAtLoad was already part of the file and I just did not see it, so the only addition is OnDemand: > * * * * org.boxbackup.bbackupd.plist.in * * * * > [..] > OnDemand > > > > * * * * org.boxbackup.bbackupd.plist.in * * * * Achim -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From james at netinertia.co.uk Wed Jan 19 23:02:34 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Wed, 19 Jan 2011 23:02:34 +0000 Subject: [Box Backup] Updates for OS X launchd file In-Reply-To: <4D376B64.3000004@qustodium.net> References: <4D3767B1.4080604@qustodium.net> <4D376B64.3000004@qustodium.net> Message-ID: <20110119230233.GX21506@netinertia.co.uk> Hi Achim, On Wed, Jan 19, 2011 at 11:53:24PM +0100, Achim J. Latz wrote: > Hello again: > > I just realised that RunAtLoad was already part of the file and I just > did not see it, so the only addition is OnDemand: > > > * * * * org.boxbackup.bbackupd.plist.in * * * * > > [..] > > OnDemand > > > > > > > > * * * * org.boxbackup.bbackupd.plist.in * * * * Thanks, I've commited this change. James From achim+box at qustodium.net Fri Jan 21 15:59:30 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Fri, 21 Jan 2011 16:59:30 +0100 Subject: [Box Backup] Fwd: Boxi always shows first entry with same names, even if that entry is deleted Message-ID: <4D39AD62.1000500@qustodium.net> Hello list: Still having problems with the issue outlined in the message below. Chris, is this something that is easily fixable in Boxi's logic? I also just run into a situation where Boxi tells me when clicking on a deleted folder: "Error listing directory on server: Object does not exist", but bbackupquery from the same build can enter the directory just fine with the "-d" command. One additional question on that: "cd -d" in bbackupquery allows to enter deleted versions of the directory. However, there seems to be no way to choose specifically which one, e.g. via the "-i" switch to identify one specific deleted version of a directory. Could this be added? Thanks, Achim -------- Original Message -------- Subject: Boxi always shows first entry with same names, even if that entry is deleted Date: Fri, 08 Oct 2010 17:28:22 +0200 From: Achim To: Boxbackup Good afternoon: Boxi at the moment always shows "Old Files" and "Deleted Files", ignoring the settings in the view menu. I have just realised the following: In bbackupquery, I see the following: query > list -d 00000002 -dX--- F_root 00005f40 -d---- F_root Two F_root, one "deleted" and the other not. In Boxi, I see this: -/(server root) +F_root It appears that Boxi will always show the first entry in its list, for instance "00000002 -dX--- F_root" instead of the more recent (and not deleted!) "00005f40 -d---- F_root". I say that because the files I get if I go into F_root in Boxi are the outdated ones in "00000002 -dX--- F_root". I am however interested in looking at "00005f40 -d---- F_root", but simply can't access that folder at the moment. Is there a (simple) way to have Boxi either show all folders (and not only "one per identical name"), or respect the setting of "View/Hide old/deleted files"? Otherwise, it appears that restoring in Boxi is pretty limited at the moment. Thanks, Achim From ell2 at live.se Sun Jan 23 20:39:28 2011 From: ell2 at live.se (Leif Linderstam) Date: Sun, 23 Jan 2011 21:39:28 +0100 Subject: [Box Backup] Error regarding SocketBindError / ListenAddresses In-Reply-To: <20110103170430.GF21506@netinertia.co.uk> References: , <20110103153107.GC21506@netinertia.co.uk>, , <20110103153617.GD21506@netinertia.co.uk>, , <20110103162424.GE21506@netinertia.co.uk>, , , <20110103170430.GF21506@netinertia.co.uk> Message-ID: A late addition to the discussion. The openssl command "ca" has options -startdate and -enddate in addition to -days. These explicit options do not have the 2038 limitation that -days has, if I remember correctly. The bbstored-certs script however uses the command "x509" which only has the -days option. If it is possible to switch to the "ca" command and set a far off end-date, then 32-bit systems would not have to be handled separately. /Leif Linderstam > Date: Mon, 3 Jan 2011 17:04:31 +0000 > From: james at netinertia.co.uk > To: boxbackup at boxbackup.org > Subject: Re: [Box Backup] Error regarding SocketBindError / ListenAddresses > > On Mon, Jan 03, 2011 at 06:02:13PM +0100, Wouter van Eekelen wrote: > > Problem found! The bbstored-certs file has this on top: > > -- > > my $root_sign_period = '10000'; > > > > 10,000 days is 27.39 years, which is past 2038 (Y2K38), which made the > > certificate invalid: > > -- > > notBefore=Jan 3 15:59:18 2011 GMT > > notAfter=Apr 15 09:31:02 1902 GMT > > > > I've changed it to 9000, which is before 2038, so now it's all valid :) > > Aha, interesting! Perhaps we should change this default for 32-bit > systems. > > Good spot, thanks. > > James > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup -------------- next part -------------- An HTML attachment was scrubbed... URL: From ell2 at live.se Mon Jan 24 20:37:06 2011 From: ell2 at live.se (Leif Linderstam) Date: Mon, 24 Jan 2011 21:37:06 +0100 Subject: [Box Backup] Another wish for 2011 Message-ID: Hi all Box Backup developers! As others have done, I would also like to thank you for your work on this system. Last fall I tried to compile this for a small NAS, the DNS-323. I got into trouble with the network communication between the Linux PC and the NAS, and the reason for this was that the compiler directives for packing C structs was ignored for the ARM architecture. I tried other packing directives, which the compiler did act on, but the result was even worse. The communication got out of sync at an even earlier stage than before. I also tried with manual padding of the failing C struct, which looked promising and got me further, but I got into more difficult problems later on. At this point I gave up and started looking at other options for backup software, but Box Backup does have the right set of features for me, especially if snapshots are added. So I will have another go at this. Now to my wish for Box Backup. Would it be possible to define the communication protocol explicitly on "paper"? The current approach in a way is ad hoc because it depends on what the compiler happens to generate. This makes it much harder to port Box Backup to a new platform. To a certain extent the same goes for the storage format, but that is not as vulnerable as the communication protocol. Also if I have understood the code correctly there is no support for multiple versions of the protocol, i.e. clients and servers from different versions of Box Backup will probably not be able to communicate with each other. This is troublesome when binaries are included in OS distributions, because an upgrade, or maybe even just an update, may install an incompatible version of Box Backup client or server. Versioning of the protocol (if the code supports more than one version), or a tagged protocol, or a combination of both would help. A tagged protocol can be made so that even if the receiver does not understand a particular tag it can still skip over it and continue with the rest of the message. This would make the system more robust to small changes in the protocol. I know that this would impact the current code quite heavily, but I believe the benefits would be big for the product's reliability. And as a bonus it would be easier to port. Cheers, Leif -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at qwirx.com Mon Jan 24 23:04:09 2011 From: chris at qwirx.com (Chris Wilson) Date: Mon, 24 Jan 2011 23:04:09 +0000 (GMT) Subject: [Box Backup] Box Backup protocol description, for the record Message-ID: Application for User Registered Port Number Name : Chris Wilson E-mail : chris+iana at qwirx.com Protocol Number : TCP Message Formats : Handshake message: char mIdent[32]; The ident used by both sides is currently "Box-Backup:v=C" Object header: u_int32_t mObjSize; u_int32_t mObjType; (followed by mObjSize bytes of data) Error message (mObjType = 0): int32 Type int32 SubType where Type is always 1000 and SubType is one of: CONSTANT ErrorType 1000 CONSTANT Err_WrongVersion 1 CONSTANT Err_NotInRightProtocolPhase 2 CONSTANT Err_BadLogin 3 CONSTANT Err_CannotLockStoreForWriting 4 CONSTANT Err_SessionReadOnly 5 CONSTANT Err_FileDoesNotVerify 6 CONSTANT Err_DoesNotExist 7 CONSTANT Err_DirectoryAlreadyExists 8 CONSTANT Err_CannotDeleteRoot 9 CONSTANT Err_TargetNameExists 10 CONSTANT Err_StorageLimitExceeded 11 CONSTANT Err_DiffFromFileDoesNotExist 12 CONSTANT Err_DoesNotExistInDirectory 13 CONSTANT Err_PatchConsistencyError 14 Version message (mObjType = 1): int32 Version Login Message mObjType = 2): int32 ClientID int32 Flags CONSTANT Flags_ReadOnly 1 LoginConfirmed message (mObjType = 3): int64 ClientStoreMarker int64 BlocksUsed int64 BlocksSoftLimit int64 BlocksHardLimit Finished message (mObjType = 4): no data fields Success message (mObjType = 5): int64 ObjectID SetClientStoreMarker message (mObjType = 6): int64 ClientStoreMarker GetObject (mObjType = 10) int64 ObjectID CONSTANT NoObject 0 # reply has stream following, if ObjectID != NoObject MoveObject (11): int64 ObjectID int64 MoveFromDirectory int64 MoveToDirectory int32 Flags Filename NewFilename CONSTANT Flags_MoveAllWithSameName 1 CONSTANT Flags_AllowMoveOverDeletedObject 2 GetObjectName (12) int64 ObjectID int64 ContainingDirectoryID CONSTANT ObjectID_DirectoryOnly 0 # set ObjectID to ObjectID_DirectoryOnly to only get info on the directory ObjectName (13) int32 NumNameElements int64 ModificationTime int64 AttributesHash int16 Flags # NumNameElements is zero if the object doesn't exist CONSTANT NumNameElements_ObjectDoesntExist 0 # a stream of Filename objects follows, if and only if NumNameElements > 0 CreateDirectory (20) int64 ContainingDirectoryID int64 AttributesModTime Filename DirectoryName # stream following containing attributes ListDirectory (21) int64 ObjectID int16 FlagsMustBeSet int16 FlagsNotToBeSet bool SendAttributes CONSTANT Flags_INCLUDE_EVERYTHING -1 CONSTANT Flags_EXCLUDE_NOTHING 0 CONSTANT Flags_EXCLUDE_EVERYTHING 15 CONSTANT Flags_File 1 CONSTANT Flags_Dir 2 CONSTANT Flags_Deleted 4 CONSTANT Flags_OldVersion 8 CONSTANT RootDirectory 1 ChangeDirAttributes (22) int64 ObjectID int64 AttributesModTime # stream following containing attributes DeleteDirectory (23) int64 ObjectID UndeleteDirectory (24) int64 ObjectID StoreFile (30) int64 DirectoryObjectID int64 ModificationTime int64 AttributesHash int64 DiffFromFileID # 0 if the file is not a diff Filename Filename # then send a stream containing the encoded file GetFile (31) int64 InDirectory int64 ObjectID # error returned if not a file, or does not exist # reply has stream following, containing an encoded file IN STREAM ORDER # (use GetObject to get it in file order) SetReplacementFileAttributes (32) int64 InDirectory int64 AttributesHash Filename Filename # stream follows containing attributes DeleteFile (33) int64 InDirectory Filename Filename # will return 0 if the object couldn't be found in the specified directory GetBlockIndexByID (34) int64 ObjectID GetBlockIndexByName (35) int64 InDirectory Filename Filename GetAccountUsage (40) # no data members AccountUsage (41) int64 BlocksUsed int64 BlocksInOldFiles int64 BlocksInDeletedFiles int64 BlocksInDirectories int64 BlocksSoftLimit int64 BlocksHardLimit int32 BlockSize GetIsAlive (42) # no data members IsAlive (43) # no data members Message Types : handshake, command, reply, error, stream. Since each command has its own sequence of parameters, given above, you could regard each command as a message type. Message opcodes : Operation codes are given in brackets after each message description above. Message Sequences : After handshake, the client sends Command messages to the server, and receives Reply messages (one of which is Error). Any command (except Handshake) may receive an Error reply. client message -> expected response from server Handshake -> Handshake Version -> Version Login -> LoginConfirmed Finished -> Finished SetClientStoreMarker -> Success GetObject -> Success (followed by stream) MoveObject -> Success GetObjectName -> ObjectName CreateDirectory -> Success ListDirectory -> Success + stream ChangeDirAttributes -> Success DeleteDirectory -> Success UndeleteDirectory -> Success StoreFile + stream -> Success GetFile -> Success + stream SetReplacementFileAttributes + stream -> Success DeleteFile -> Success GetBlockIndexByID -> Success + stream GetBlockIndexByName -> Success + stream GetAccountUsage -> AccountUsage GetIsAlive -> IsAlive Protocol functions : Management of encrypted files, directories and block indexes (checksums) on an untrusted remote backup server. Broadcast or Multicast used ? no How and what for Broadcast or Multicast is used (if used): Description : An open source, completely automatic on-line backup system for UNIX and Windows. All backed up data is stored on the server in files on a filesystem -- no tape or archive devices are used. The server is trusted only to make files available when they are required -- all data is encrypted. A backup daemon runs on systems to be backed up, and copies encrypted data to the server when it notices changes. We need a port number for the server to listen on for requests from backup daemons. Name of the port : Box Backup Store Service Short name of the port : boxbackupstore From chris at qwirx.com Mon Jan 24 23:10:46 2011 From: chris at qwirx.com (Chris Wilson) Date: Mon, 24 Jan 2011 23:10:46 +0000 (GMT) Subject: [Box Backup] Another wish for 2011 In-Reply-To: References: Message-ID: Hi Leif, On Mon, 24 Jan 2011, Leif Linderstam wrote: > As others have done, I would also like to thank you for your work on this system. You're most welcome :) > Last fall I tried to compile this for a small NAS, the DNS-323. I got > into trouble with the network communication between the Linux PC and the > NAS, and the reason for this was that the compiler directives for > packing C structs was ignored for the ARM architecture. I tried other > packing directives, which the compiler did act on, but the result was > even worse. The communication got out of sync at an even earlier stage > than before. I also tried with manual padding of the failing C struct, > which looked promising and got me further, but I got into more difficult > problems later on. I remember your email from last year. I'm sorry I have't had time to look into it further so far. Thanks for reminding me. > Now to my wish for Box Backup. Would it be possible to define the > communication protocol explicitly on "paper"? I believe I have already done so, as far as I understand it. I thought I had put it on the wiki, but apparently not, only in the email to IANA requesting a registered port number. I spent the last two hours expanding and annotating this description and converting it to wiki format, and then accidentally hit Ctrl+W, closing the current tab and losing my work. I may try again another day. > The current approach in a way is ad hoc It is intended to be thus, to some extent, to allow us to extend the protocol in future. Box Backup is not finished software and its protocol is not defined in any RFC yet, and may never be. > because it depends on what the compiler happens to generate. This however is clearly a bug. > Also if I have understood the code correctly there is no support for > multiple versions of the protocol, i.e. clients and servers from > different versions of Box Backup will probably not be able to > communicate with each other. This is troublesome when binaries are > included in OS distributions, because an upgrade, or maybe even just an > update, may install an incompatible version of Box Backup client or > server. Creating a protocol which is fully backwards and forwards compatible is an immense overhead to maintain, witness the Win32 API. There is limited support for a client to request a particular version and for the server to return the same version number, if it thinks it can safely talk to that client, even if the server actually supports later versions of the protocol as well. We have been reluctant to change the version number unless absolutely necessary because we don't (currently) have time to build in regression checks against previous versions of the code. Therefore you can connect to a 0.10 server with a 0.11 client, but if you use bbackupquery and try to undelete a file, it will fail with an exception because the 0.10 server does not support this command. > Versioning of the protocol (if the code supports more than one version), or a > tagged protocol, or a combination of both would help. The protocol is versioned and can be extended. It is unlikely to be tagged, especially at the expense of implementing important features. > A tagged protocol can be made so that even if the receiver does not > understand a particular tag it can still skip over it and continue with > the rest of the message. This would make the system more robust to small > changes in the protocol. That is currently possible, if the client uses a command that the server does not understand and correctly interprets and handles the server's error response. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From siretart at tauware.de Tue Jan 25 14:28:10 2011 From: siretart at tauware.de (Reinhard Tartler) Date: Tue, 25 Jan 2011 15:28:10 +0100 Subject: [Box Backup] daily builds of trunk for ubuntu Message-ID: <8762td12it.fsf@faui44a.informatik.uni-erlangen.de> Hi, I've just played a bit with launchpad's support for building ubuntu packages from bzr branches, since the boxbackup svn trunk is mirrored there: https://code.launchpad.net/~vcs-imports/boxbackup/trunk The resulting packages can be downloaded here: https://launchpad.net/~siretart/+archive/boxbackup-dailies Please let me know if you find this useful. If so, I'll add this piece of information to the wiki. -- Gruesse/greetings, Reinhard Tartler, KeyID 945348A4 From ell2 at live.se Tue Jan 25 20:35:39 2011 From: ell2 at live.se (Leif Linderstam) Date: Tue, 25 Jan 2011 21:35:39 +0100 Subject: [Box Backup] Another wish for 2011 In-Reply-To: References: , Message-ID: Hi Chris, Thanks for your quick response. > I remember your email from last year. I'm sorry I have't had time to look > into it further so far. Thanks for reminding me. I fully understand that. The features you seem to be working on, like snapshots, would have had a much higher priority for me as well if I had been working on this project. > I believe I have already done so, as far as I understand it. I thought I > had put it on the wiki, but apparently not, only in the email to IANA > requesting a registered port number. I spent the last two hours expanding > and annotating this description and converting it to wiki format, and then > accidentally hit Ctrl+W, closing the current tab and losing my work. I may > try again another day. I saw your other email with the protocol definition. That is enough for the time being, although a Wiki entry would be nice in the long run :) I am sorry to hear that you spent that much time on editing just to loose it. I have done the same myself a couple of times so now I am paranoid enough, when it comes to editing in a browser, that I copy the text over to a regular text editor from time to time, or just edit it "offline". > >> The current approach in a way is ad hoc > > It is intended to be thus, to some extent, to allow us to extend the > protocol in future. Box Backup is not finished software and its protocol > is not defined in any RFC yet, and may never be. My thinking with a tagged protocol was to make it easier to extend, but I understand that among all things to prioritize between this would not come high even if thought of as a good idea. >> because it depends on what the compiler happens to generate. > > This however is clearly a bug. Yes it is a bug in the compiler (which is what I think you meant), but when porting a piece of software one obstacle is bugs in the target environment and it is very seldom one can fix this by having the bugs sorted out. Most often one has to work around the bug, and different coding choices makes this harder or easier. The choices one makes might even make it more or less likely that bugs become a problem in the first place. > Creating a protocol which is fully backwards and forwards compatible is an > immense overhead to maintain, witness the Win32 API. There is limited Yes, but I still believe that a system like this at least has to handle two consecutive protocol versions in one direction unless it only is used in fairly small setting where all involved machines can reasonably be updated at the same time. Either the server or the client can handle two versions, but I guess that the best would be for the server to do so because it would then be able to spot any clients that has not yet been updated to the latest version. But of course all of this is just my opinion, just as with the rest of the comments :) >> A tagged protocol can be made so that even if the receiver does not >> understand a particular tag it can still skip over it and continue with >> the rest of the message. This would make the system more robust to small >> changes in the protocol. > > That is currently possible, if the client uses a command that the server > does not understand and correctly interprets and handles the server's > error response. I haven't looked into the code throughly enough to quite understand your comment. Can the client do something more intelligible than issue an error message that the communication failed? I was thinking more about being able to skip single fields than whole messages. For example, if a directory listing is requested the server might send answer messages with more information elements than the client knows about, but the client can still read the elements it does know about and skip the rest. E.g. if the client knows about file name and size, but not permissions, it can still print some information. Cheers, Leif ? ? ? ? ? ? ? ? ? From achim+box at qustodium.net Tue Jan 25 21:22:53 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Tue, 25 Jan 2011 22:22:53 +0100 Subject: [Box Backup] Development feedback & discussion for Box Backup 1.0? In-Reply-To: <4D34E249.7010509@diffingo.com> References: <4D0FD12F.1070901@gmx.net> <4D13CFE3.3060309@qustodium.net> <4D13E0FF.2010609@gmx.net> <4D15B139.10109@diffingo.com> <4D2B941E.3000200@qustodium.net> <4D2D5379.5070700@diffingo.com> <4D2F9E30.2090703@qustodium.net> <4D30884B.5090509@diffingo.com> <4D34C5E3.3020400@qustodium.net> <4D34E249.7010509@diffingo.com> Message-ID: <4D3F3F2D.6080703@qustodium.net> Hello Stewart: On 18/01/2011 01:43, Stewart Adam wrote: > I have some mock UIs finished, but they need a bit of improvement since > I'd like to add widgets to select individual files from previous > revisions (as opposed to always restoring the most recent version). > [..] > Similar to a certain extent... I'm re-using bits I like from the 1.43.x > UI but making things even simpler when possible and redesigning the bits > that no longer apply (for example, compression as this is done natively > with BB). > > I've uploaded some screenshots of the latest compiles here: > http://www.firewing1.com/downloads/fwbackups-cpp-screenshots/ Looks great so far. Would it be possible to enable the UI to be translatable? Your ideas for selecting individual versions would be interesting, I posted a couple of ideas under the topics "Next round: UI Ideas for Boxi" and "Suggestion for Boxi UI improvements". > I also included screenshots of the server administrator, which owners of > a Box Backup server would use to administrate accounts & distribute keys > to users. Will this be based on an API/web service that runs on a remote server, or is this UI used to administrate a locally running BB server? > I'm hoping within the next month I'll have the UI completely finished. Great stuff, please keep me updated on this and how I can help with the translations. Best regards, Achim -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From chris at qwirx.com Tue Jan 25 23:56:58 2011 From: chris at qwirx.com (Chris Wilson) Date: Tue, 25 Jan 2011 23:56:58 +0000 (GMT) Subject: [Box Backup] Another wish for 2011 In-Reply-To: References: , Message-ID: Hi Leif, On Tue, 25 Jan 2011, Leif Linderstam wrote: > Thanks for your quick response. No problem :) > I saw your other email with the protocol definition. That is enough for > the time being, although a Wiki entry would be nice in the long run :) I am working on it (offline now :) >>> The current approach in a way is ad hoc >> >> It is intended to be thus, to some extent, to allow us to extend the >> protocol in future. Box Backup is not finished software and its >> protocol is not defined in any RFC yet, and may never be. > > My thinking with a tagged protocol was to make it easier to extend, but > I understand that among all things to prioritize between this would not > come high even if thought of as a good idea. I just wanted to check what you meant by a "tagged" protocol? Every message in the Box Backup protocol (after the handshake) is "tagged" with a message type code and a length. If we really wanted to, we could define a new command, and either delete or deprecate (and maintain support for) the old command to maintain backwards compatibility. Is that what you mean by "tagged"? >>> because it depends on what the compiler happens to generate. >> >> This however is clearly a bug. > > Yes it is a bug in the compiler (which is what I think you meant) Actually I meant it's a bug in Box Backup. It should do whatever is necessary to maintain the wire protocol compatibility in the face of compiler differences. If it fails to do so then I consider that a bug in Box, not a compiler bug. >> Creating a protocol which is fully backwards and forwards compatible is an >> immense overhead to maintain, witness the Win32 API. There is limited > > Yes, but I still believe that a system like this at least has to handle > two consecutive protocol versions in one direction unless it only is > used in fairly small setting where all involved machines can reasonably > be updated at the same time. Either the server or the client can handle > two versions, but I guess that the best would be for the server to do so > because it would then be able to spot any clients that has not yet been > updated to the latest version. But of course all of this is just my > opinion, just as with the rest of the comments :) I agree in theory. It would be great if we had regression testing against the previous client version in new releases of Box Backup. We just don't have the infrastructure in place to do it at present. >>> A tagged protocol can be made so that even if the receiver does not >>> understand a particular tag it can still skip over it and continue >>> with the rest of the message. This would make the system more robust >>> to small changes in the protocol. >> >> That is currently possible, if the client uses a command that the >> server does not understand and correctly interprets and handles the >> server's error response. > > I haven't looked into the code throughly enough to quite understand your > comment. Can the client do something more intelligible than issue an > error message that the communication failed? Yes, the client can (in theory) detect an error response, and rather than just aborting the connection, it could retry the command with a different set of parameters, or try the older (deprecated) command instead. However, I'm not aware of a client that currently does so. > I was thinking more about being able to skip single fields than whole > messages. For example, if a directory listing is requested the server > might send answer messages with more information elements than the > client knows about, but the client can still read the elements it does > know about and skip the rest. E.g. if the client knows about file name > and size, but not permissions, it can still print some information. This is possible in some cases, again in theory, but I think the current clients would simply fail (throw an exception) if they received a structure with the wrong size, out of paranoia. This is a backup product after all, and we don't currently have the infrastructure to perform the kinds of regression testing that we would like to have before promising backwards compatibility of a newer server against an older client version. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From sune at molgaard.org Wed Jan 26 03:30:22 2011 From: sune at molgaard.org (=?ISO-8859-1?Q?Sune_M=F8lgaard?=) Date: Wed, 26 Jan 2011 04:30:22 +0100 Subject: [Box Backup] Frequent Connection refused (111) Message-ID: <4D3F954E.9010708@molgaard.org> Hi list, I seem to get Connection refused (111) exceedingly often but not always - the "not always" meaning that it shouldn't just be a simple question of me having forgotten to start the server :-P What output can I provide for troubleshooting? Server is an old box running Ubuntu 10.04LTS, client is reasonable and running Ubuntu 10.10. Network is 100Mbit/s and all other network-related stuff runs smoothly. Server also runs bbackupd for a few directories but never fails. Best regards, Sune M?lgaard -- "I really like the idea of C++, because of its reusable objects." he said, "I am really into the environment at the moment, and C++ appeals to me because you can recycle other peoples code. It is much more environmentally friendly." From chris at qwirx.com Wed Jan 26 09:39:28 2011 From: chris at qwirx.com (Chris Wilson) Date: Wed, 26 Jan 2011 09:39:28 +0000 (GMT) Subject: [Box Backup] Frequent Connection refused (111) In-Reply-To: <4D3F954E.9010708@molgaard.org> References: <4D3F954E.9010708@molgaard.org> Message-ID: Hi Sune, On Wed, 26 Jan 2011, Sune M?lgaard wrote: > I seem to get Connection refused (111) exceedingly often but not always > - the "not always" meaning that it shouldn't just be a simple question > of me having forgotten to start the server :-P > > What output can I provide for troubleshooting? Logs from the client would be a good start, especially if you see "connection refused" and then it works on the next attempt. You can make it log to a file with the -o command-line option, and this logs at the highest verbosity, which is useful. Also please tell us about the network between the client and server. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From sune at molgaard.org Wed Jan 26 10:39:01 2011 From: sune at molgaard.org (=?ISO-8859-1?Q?Sune_M=F8lgaard?=) Date: Wed, 26 Jan 2011 11:39:01 +0100 Subject: [Box Backup] Frequent Connection refused (111) In-Reply-To: References: <4D3F954E.9010708@molgaard.org> Message-ID: <4D3FF9C5.5000103@molgaard.org> Chris Wilson wrote: > Hi Sune, > > On Wed, 26 Jan 2011, Sune M?lgaard wrote: > >> I seem to get Connection refused (111) exceedingly often but not >> always - the "not always" meaning that it shouldn't just be a simple >> question of me having forgotten to start the server :-P >> >> What output can I provide for troubleshooting? > > Logs from the client would be a good start, especially if you see > "connection refused" and then it works on the next attempt. You can make > it log to a file with the -o command-line option, and this logs at the > highest verbosity, which is useful. > > Also please tell us about the network between the client and server. > > Cheers, Chris. > > > > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup Thanks for the reply, but it would seem that the ubuntu version (0.11-rc8) of bbackupd doesn't accept any -o option. I have, instead, modified the /etc/init.d file to not stipulate --quite, and instead added -V, and will monitor /var/log/daemon.log and report back. Network is 100Mbit/s wired with a TrendNet switch in-between, and I see no problems doing other network-related things between the machines. FW/NAT is by shorewall on the server. The network is also ipv6-enabled, but that shouldn't really matter, should it? Best regards, Sune -- A rose by any other name would smell as sweet; A chrysanthemum by any other name would be easier to spell. From sune at molgaard.org Wed Jan 26 10:54:51 2011 From: sune at molgaard.org (=?ISO-8859-1?Q?Sune_M=F8lgaard?=) Date: Wed, 26 Jan 2011 11:54:51 +0100 Subject: [Box Backup] Frequent Connection refused (111) In-Reply-To: References: <4D3F954E.9010708@molgaard.org> Message-ID: <4D3FFD7B.3000505@molgaard.org> Chris Wilson wrote: > Hi Sune, > > On Wed, 26 Jan 2011, Sune M?lgaard wrote: > >> I seem to get Connection refused (111) exceedingly often but not >> always - the "not always" meaning that it shouldn't just be a simple >> question of me having forgotten to start the server :-P >> >> What output can I provide for troubleshooting? > > Logs from the client would be a good start, especially if you see > "connection refused" and then it works on the next attempt. You can make > it log to a file with the -o command-line option, and this logs at the > highest verbosity, which is useful. > > Also please tell us about the network between the client and server. > > Cheers, Chris. > > > > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup Log from startup, which proceeded to fail to connect: Jan 26 11:51:57 jekaterina bbackupd[7209]: NOTICE: Box Backup Client v0.11rc8+2714, (c) Ben Summers and contributors 2003-2010 Jan 26 11:51:57 jekaterina bbackupd[7211]: NOTICE: Starting daemon, version: 0.11rc8+2714 Jan 26 11:51:57 jekaterina bbackupd[7211]: NOTICE: Using configuration file: /etc/boxbackup/bbackupd.conf Jan 26 11:51:57 jekaterina bbackupd[7211]: BackupDaemon::NotifySysadmin() called, event = backup-start Jan 26 11:51:57 jekaterina bbackupd[7211]: About to notify administrator about event backup-start, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-start "/etc/boxbackup/bbackupd.conf"' Jan 26 11:51:57 jekaterina bbackupd[7211]: NOTICE: Beginning scan of local files Jan 26 11:51:57 jekaterina bbackupd[7211]: Set maximum diffing time to 120 seconds Jan 26 11:51:57 jekaterina bbackupd[7211]: Set keep-alive time to 120 seconds Jan 26 11:51:57 jekaterina bbackupd[7211]: timer: next event: KeepAliveTime expires in 119.999992 seconds Jan 26 11:51:57 jekaterina bbackupd[7211]: timer: next event: KeepAliveTime expires in 119.999967 seconds Jan 26 11:51:57 jekaterina bbackupd[7211]: timer: next event: KeepAliveTime expires in 119.999949 seconds Jan 26 11:51:57 jekaterina bbackupd[7211]: timer: next event: KeepAliveTime expires in 119.999936 seconds Jan 26 11:51:57 jekaterina bbackupd[7211]: Opening connection to server 'jadis.molgaard.org'... Jan 26 11:51:57 jekaterina bbackupd[7211]: ERROR: Failed to connect to socket (type 1, name jadis.molgaard.org, port 2201): Connection refused (111) Jan 26 11:51:57 jekaterina bbackupd[7211]: Obtained 10 stack frames. Jan 26 11:51:57 jekaterina bbackupd[7211]: Stack frame 0: /usr/sbin/bbackupd(DumpStackBacktrace()+0x26) [0x4cb326] Jan 26 11:51:57 jekaterina bbackupd[7211]: Stack frame 1: /usr/sbin/bbackupd(SocketStream::Open(Socket::Type, std::string const&, int)+0x575) [0x496c85] Jan 26 11:51:57 jekaterina bbackupd[7211]: Stack frame 2: /usr/sbin/bbackupd(SocketStreamTLS::Open(TLSContext const&, Socket::Type, std::string const&, int)+0x21) [0x49cde1] Jan 26 11:51:57 jekaterina bbackupd[7211]: Stack frame 3: /usr/sbin/bbackupd(BackupClientContext::GetConnection()+0x33a) [0x427c3a] Jan 26 11:51:57 jekaterina bbackupd[7211]: Stack frame 4: /usr/sbin/bbackupd(BackupDaemon::SetupLocations(BackupClientContext&, Configuration const&)+0xae) [0x437dae] Jan 26 11:51:57 jekaterina bbackupd[7211]: Stack frame 5: /usr/sbin/bbackupd(BackupDaemon::RunSyncNow()+0xf16) [0x4415e6] Jan 26 11:51:57 jekaterina bbackupd[7211]: Stack frame 6: /usr/sbin/bbackupd(BackupDaemon::RunSyncNowWithExceptionHandling()+0x35) [0x442195] Jan 26 11:51:57 jekaterina bbackupd[7211]: Stack frame 7: /usr/sbin/bbackupd(BackupDaemon::Run2()+0x27b) [0x442b4b] Jan 26 11:51:57 jekaterina bbackupd[7211]: Stack frame 8: /usr/sbin/bbackupd(BackupDaemon::Run()+0x288) [0x443ae8] Jan 26 11:51:57 jekaterina bbackupd[7211]: Stack frame 9: /usr/sbin/bbackupd(Daemon::Main(std::string const&)+0x15b2) [0x485a42] Jan 26 11:51:57 jekaterina bbackupd[7211]: WARNING: Exception thrown: ConnectionException(Conn_SocketConnectError) at SocketStream.cpp(186) Jan 26 11:51:57 jekaterina bbackupd[7211]: timer: no more events, going to sleep. Jan 26 11:51:57 jekaterina bbackupd[7211]: BackupDaemon::NotifySysadmin() called, event = backup-error Jan 26 11:51:57 jekaterina bbackupd[7211]: About to notify administrator about event backup-error, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-error "/etc/boxbackup/bbackupd.conf"' Jan 26 11:51:57 jekaterina bbackupd[7211]: ERROR: Exception caught (Connection SocketConnectError (Probably a network issue between client and server, bad hostname, or server not running.) 7/15), reset state and waiting to retry... Jan 26 11:52:07 jekaterina bbackupd[7211]: NOTICE: Finished scan of local files Jan 26 11:52:07 jekaterina bbackupd[7211]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Jan 26 11:52:07 jekaterina bbackupd[7211]: BackupDaemon::NotifySysadmin() called, event = backup-finish Jan 26 11:52:07 jekaterina bbackupd[7211]: About to notify administrator about event backup-finish, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-finish "/etc/boxbackup/bbackupd.conf"' Jan 26 11:52:07 jekaterina bbackupd[7211]: Wait on command socket, delay = 135764630 -- Quitting vi is the most important command of that editor, and should be bound to something easy to type and available in all modes, for example the space bar. - Per Abrahamsen From ell2 at live.se Wed Jan 26 20:58:55 2011 From: ell2 at live.se (Leif Linderstam) Date: Wed, 26 Jan 2011 21:58:55 +0100 Subject: [Box Backup] Another wish for 2011 Message-ID: Hi Chris, > I just wanted to check what you meant by a "tagged" protocol? Every > message in the Box Backup protocol (after the handshake) is "tagged" with > a message type code and a length. If we really wanted to, we could define > a new command, and either delete or deprecate (and maintain support for) > the old command to maintain backwards compatibility. Is that what you mean > by "tagged"? Oups, the term "tagged" in this context is not as clear as I thought. I guess that happens when working with the same people for too long, you end up with your own specific meaning to some words within the group. What I meant was that each information element (or field) in a message has a name and size in addition to its value. The name would not be a string, but rather it would be a number preferably defined as constants in a central header file. The type of the element is not needed for the purpose of being able to skip an element, but might as well be included and is anyway closely related to the size of the element. So a field would consist of the following: ------------------------------ | Name | Type | Size | Value | ------------------------------ (In many cases the size is implied by the type and then not needed.) Some basic types like integers are needed, but also aggregate types. At least an array type is needed, and possibly also a struct-like thing. The name, or rather its number, would need to be unique within a message, but the easiest would probably be to make it globally unique. >> Yes it is a bug in the compiler (which is what I think you meant) > > Actually I meant it's a bug in Box Backup. It should do whatever is > necessary to maintain the wire protocol compatibility in the face of > compiler differences. If it fails to do so then I consider that a bug in > Box, not a compiler bug. Ah, I guessed totally wrong then. Nevertheless it is encouraging for my porting efforts that you also think that Box Backup needs a change here. I might be able to help with this if you want me to. If you can find the time to sort out what mechanism you want to be used, I could probably do the dirty work of implementing it. (I wouldn't want to go in and change based only on my own preferences, but I can gladly be part in a discussion about the mechanism if that would help.) But I cannot promise fast results, my days are currently filled with caring for our two small children :) Cheers, Leif -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at qwirx.com Wed Jan 26 23:25:43 2011 From: chris at qwirx.com (Chris Wilson) Date: Wed, 26 Jan 2011 23:25:43 +0000 (GMT) Subject: [Box Backup] Another wish for 2011 In-Reply-To: References: Message-ID: Hi Leif, On Wed, 26 Jan 2011, Leif Linderstam wrote: > > I just wanted to check what you meant by a "tagged" protocol? [...] > What I meant was that each information element (or field) in a message > has a name and size in addition to its value. The name would not be a > string, but rather it would be a number preferably defined as constants > in a central header file. The type of the element is not needed for the > purpose of being able to skip an element, but might as well be included > and is anyway closely related to the size of the element. > > So a field would consist of the following: > ------------------------------ > | Name | Type | Size | Value | > ------------------------------ > (In many cases the size is implied by the type and then not needed.) > Some basic types like integers are needed, but also aggregate types. At > least an array type is needed, and possibly also a struct-like thing. > The name, or rather its number, would need to be unique within a message, > but the easiest would probably be to make it globally unique. I think this would bloat the protocol quite a lot, so I'm not really in favour of adding it at the moment. What we could do is to handle message replies that are larger than expected, implying that extra fields have been added to the message structure, and the client could ignore the extra fields. This would allow us to add new fields without breaking backwards compatibility. We haven't yet needed to, but it might be worth implementing this in advance, to allow clients and servers to be forward compatible with future versions of the other side that supply additional data. > > Actually I meant it's a bug in Box Backup. It should do whatever is > > necessary to maintain the wire protocol compatibility in the face of > > compiler differences. If it fails to do so then I consider that a bug in > > Box, not a compiler bug. > > Ah, I guessed totally wrong then. Nevertheless it is encouraging for my > porting efforts that you also think that Box Backup needs a change here. I > might be able to help with this if you want me to. Yes please :) > If you can find the time to sort out what mechanism you want to be used, > I could probably do the dirty work of implementing it. The first task would be to identify the structures that need changing, and understand why the current mechanism is failing and whether we can fix it without breaking backwards compatibility. I don't have an armeb platform, but it might be possible for me to build and run an armeb server under qemu, and it might be efficient enough (on my 10-year-old desktop box) to be reasonable for testing. However I might instead have to send you blind attempts at fixes for you to test. > But I cannot promise fast results, my days are currently filled with > caring for our two small children :) I can't claim such a good excuse, but I also don't have as much time for working on Box or Boxi as I would like to have. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Wed Jan 26 23:30:39 2011 From: chris at qwirx.com (Chris Wilson) Date: Wed, 26 Jan 2011 23:30:39 +0000 (GMT) Subject: [Box Backup] Frequent Connection refused (111) In-Reply-To: <4D3FF9C5.5000103@molgaard.org> References: <4D3F954E.9010708@molgaard.org> <4D3FF9C5.5000103@molgaard.org> Message-ID: Hi Sune, On Wed, 26 Jan 2011, Sune M?lgaard wrote: >>> I seem to get Connection refused (111) exceedingly often but not >>> always - the "not always" meaning that it shouldn't just be a simple >>> question of me having forgotten to start the server :-P >>> >>> What output can I provide for troubleshooting? >> >> Logs from the client would be a good start, especially if you see >> "connection refused" and then it works on the next attempt... Also >> please tell us about the network between the client and server. > > Thanks for the reply, but it would seem that the ubuntu version > (0.11-rc8) of bbackupd doesn't accept any -o option. I have, instead, > modified the /etc/init.d file to not stipulate --quite, and instead > added -V, and will monitor /var/log/daemon.log and report back. Thanks, that seems to work well enough for now. > Network is 100Mbit/s wired with a TrendNet switch in-between, and I see > no problems doing other network-related things between the machines. > FW/NAT is by shorewall on the server. The network is also ipv6-enabled, > but that shouldn't really matter, should it? I notice the following lines from the logs that you sent: Jan 26 11:51:57 jekaterina bbackupd[7211]: Opening connection to server 'jadis.molgaard.org'... Jan 26 11:51:57 jekaterina bbackupd[7211]: ERROR: Failed to connect to socket (type 1, name jadis.molgaard.org, port 2201): Connection refused (111) I also note that jadis.molgaard.org appears to resolve to a public IP. Are the server and the client on the same NAT (private IP) network, and if so, how does the client manage to connect to the server using a public IP? Is that translation done by the NAT box? Also, are you running split horizon DNS to support IPv6 or NAT? What does jadis.molgaard.org resolve to on the client? (please send the output of the "host" command, against each nameserver IP in /etc/resolv.conf on the client, including IPv6 addresses). If it resolves to more than one IP, or different IPs depending on which nameserver is asked, this could result in non-deterministic (pseudo-random) behaviour. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From sune at molgaard.org Thu Jan 27 06:51:20 2011 From: sune at molgaard.org (=?ISO-8859-1?Q?Sune_M=F8lgaard?=) Date: Thu, 27 Jan 2011 07:51:20 +0100 Subject: [Box Backup] Frequent Connection refused (111) In-Reply-To: References: <4D3F954E.9010708@molgaard.org> <4D3FF9C5.5000103@molgaard.org> Message-ID: <4D4115E8.7010008@molgaard.org> Chris Wilson wrote: > I notice the following lines from the logs that you sent: > > Jan 26 11:51:57 jekaterina bbackupd[7211]: Opening connection to server > 'jadis.molgaard.org'... > Jan 26 11:51:57 jekaterina bbackupd[7211]: ERROR: Failed to connect to > socket (type 1, name jadis.molgaard.org, port 2201): Connection refused > (111) > > I also note that jadis.molgaard.org appears to resolve to a public IP. > Are the server and the client on the same NAT (private IP) network, and > if so, how does the client manage to connect to the server using a > public IP? Is that translation done by the NAT box? > > Also, are you running split horizon DNS to support IPv6 or NAT? What > does jadis.molgaard.org resolve to on the client? (please send the > output of the "host" command, against each nameserver IP in > /etc/resolv.conf on the client, including IPv6 addresses). If it > resolves to more than one IP, or different IPs depending on which > nameserver is asked, this could result in non-deterministic > (pseudo-random) behaviour. > > Cheers, Chris. I run my own DNS server with internal and external views, so on the client, jadis.molgaard.org resolves to 10.0.0.1. Incidentally, I also have my gf's external machine backup to the server, so I have connection attempts on the public IP forwarded to 10.0.0.1. The problem has existed way prior to the last setup, however. Incidentally, it also resolves to 192.168.1.1 on the internal view, as it serves as a wireless access point for that subnet, but I don't have any cards on my client on that subnet, so I didn't figure it would matter, but could this be it? Best regards, Sune M?lgaard -- First things first, but not necessarily in that order. - Doctor Who From chris at qwirx.com Thu Jan 27 09:39:15 2011 From: chris at qwirx.com (Chris Wilson) Date: Thu, 27 Jan 2011 09:39:15 +0000 (GMT) Subject: [Box Backup] Frequent Connection refused (111) In-Reply-To: <4D4115E8.7010008@molgaard.org> References: <4D3F954E.9010708@molgaard.org> <4D3FF9C5.5000103@molgaard.org> <4D4115E8.7010008@molgaard.org> Message-ID: Hi Sune, On Thu, 27 Jan 2011, Sune M?lgaard wrote: > I run my own DNS server with internal and external views, so on the > client, jadis.molgaard.org resolves to 10.0.0.1... Incidentally, it also > resolves to 192.168.1.1 on the internal view, as it serves as a wireless > access point for that subnet, but I don't have any cards on my client on > that subnet, so I didn't figure it would matter, but could this be it? Yes, it could well be. If the client picks this address (randomly from the list of A records returned by the DNS server), it will find a route to it and try to connect to it. If it can't find a route, it will fail, not try the next address. So only use multiple A records if they *all* work, not *any* of them. Do both "telnet 10.0.0.1 2201" and "telnet 192.168.1.1 2201" work on the client? Also, if you use bbackupquery on the client to connect to the server, do you get the same intermittent errors? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From sune at molgaard.org Thu Jan 27 11:40:45 2011 From: sune at molgaard.org (=?ISO-8859-1?Q?Sune_M=F8lgaard?=) Date: Thu, 27 Jan 2011 12:40:45 +0100 Subject: [Box Backup] Frequent Connection refused (111) In-Reply-To: References: <4D3F954E.9010708@molgaard.org> <4D3FF9C5.5000103@molgaard.org> <4D4115E8.7010008@molgaard.org> Message-ID: <4D4159BD.9070804@molgaard.org> Chris Wilson wrote: > Yes, it could well be. If the client picks this address (randomly from > the list of A records returned by the DNS server), it will find a route > to it and try to connect to it. If it can't find a route, it will fail, > not try the next address. So only use multiple A records if they *all* > work, not *any* of them. Ok, I thought it would pick one with a route to it > Do both "telnet 10.0.0.1 2201" and "telnet 192.168.1.1 2201" work on the > client? Naturaly not, since the client mahine is only on the 10.0.0.0/24 subnet > Also, if you use bbackupquery on the client to connect to the server, do > you get the same intermittent errors? I would imagine so > Cheers, Chris. Making a new, separate, view for wireless on my DNS server has, indeed, seemed to do the trick. Thank you very much :-) Best regards, Sune M?lgaard -- Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities. - Charles Bukowski From ell2 at live.se Thu Jan 27 13:53:04 2011 From: ell2 at live.se (Leif Linderstam) Date: Thu, 27 Jan 2011 14:53:04 +0100 Subject: [Box Backup] Another wish for 2011 In-Reply-To: References: , Message-ID: Hi Chris, > > If you can find the time to sort out what mechanism you want to be used, > > I could probably do the dirty work of implementing it. > > The first task would be to identify the structures that need changing, and > understand why the current mechanism is failing and whether we can fix it > without breaking backwards compatibility. > > I don't have an armeb platform, but it might be possible for me to build > and run an armeb server under qemu, and it might be efficient enough (on > my 10-year-old desktop box) to be reasonable for testing. However I might > instead have to send you blind attempts at fixes for you to test. It seems like a waste of your time to do that. I have compiled from the source before and can do so again. I will retrieve the source again and locate the first problematic struct. Then I can either propose a change in the code, or just point out the problem for you, whatever you feel is the best. Cheers, Leif From sune at molgaard.org Thu Jan 27 17:34:34 2011 From: sune at molgaard.org (=?ISO-8859-1?Q?Sune_M=F8lgaard?=) Date: Thu, 27 Jan 2011 18:34:34 +0100 Subject: [Box Backup] bbstoreaccounts check fix reults in crash on non-existent raid file Message-ID: <4D41ACAA.9060206@molgaard.org> This shouldn't be, right? sune at jadis:~$ sudo bbstoreaccounts check 75AB23C fix [sudo] password for sune: NOTICE: Will fix errors encountered during checking. INFO: Checking store account ID 0x075ab23c... INFO: Phase 1, check objects... TRACE: Max dir starting ID is 0x113400 WARNING: Spurious file backup/075ab23c/refcount.db found, deleting INFO: Phase 2, check directories... WARNING: File ID 0x2723f has different container ID, probably moved WARNING: File ID 0x272fb has different container ID, probably moved WARNING: File ID 0x28196 has different container ID, probably moved WARNING: File ID 0x285eb has different container ID, probably moved WARNING: File ID 0x28794 has different container ID, probably moved WARNING: File ID 0xa907 has different container ID, probably moved WARNING: File ID 0xf5b3 has different container ID, probably moved WARNING: File ID 0x113a4 has different container ID, probably moved WARNING: File ID 0x1ac69 has different container ID, probably moved WARNING: File ID 0x288e2 has different container ID, probably moved WARNING: File ID 0x28abc has different container ID, probably moved WARNING: File ID 0x2a5f5 has different container ID, probably moved WARNING: File ID 0x32708 has different container ID, probably moved WARNING: File ID 0x4f192 has different container ID, probably moved WARNING: File ID 0x4f165 has different container ID, probably moved WARNING: File ID 0x4f187 has different container ID, probably moved WARNING: File ID 0x4f198 has different container ID, probably moved WARNING: File ID 0x4f18f has different container ID, probably moved WARNING: File ID 0x4f185 has different container ID, probably moved WARNING: File ID 0x4f186 has different container ID, probably moved WARNING: File ID 0x4f16e has different container ID, probably moved WARNING: File ID 0x4f196 has different container ID, probably moved WARNING: File ID 0x4f188 has different container ID, probably moved WARNING: File ID 0xb439c has different container ID, probably moved WARNING: File ID 0xbad0f has different container ID, probably moved WARNING: File ID 0xbad08 has different container ID, probably moved WARNING: File ID 0xc0d1d has different container ID, probably moved WARNING: File ID 0xc0d13 has different container ID, probably moved WARNING: File ID 0xb438a has different container ID, probably moved WARNING: File ID 0xc3850 has different container ID, probably moved WARNING: File ID 0xc931f has different container ID, probably moved WARNING: File ID 0xc9c97 has different container ID, probably moved WARNING: File ID 0xc9c84 has different container ID, probably moved WARNING: File ID 0xc9c5f has different container ID, probably moved WARNING: File ID 0xdf30d has different container ID, probably moved WARNING: File ID 0xdf307 has different container ID, probably moved WARNING: File ID 0x2896a has different container ID, probably moved WARNING: File ID 0x28fd3 has different container ID, probably moved WARNING: File ID 0x2926f has different container ID, probably moved WARNING: File ID 0x94746 has different container ID, probably moved WARNING: File ID 0x965c4 has different container ID, probably moved WARNING: File ID 0xcf1ce has different container ID, probably moved WARNING: File ID 0xcf529 has different container ID, probably moved WARNING: File ID 0xd1371 has different container ID, probably moved WARNING: File ID 0x9714c has different container ID, probably moved WARNING: File ID 0x97322 has different container ID, probably moved WARNING: File ID 0x98075 has different container ID, probably moved WARNING: File ID 0x990a2 has different container ID, probably moved WARNING: File ID 0x99498 has different container ID, probably moved WARNING: File ID 0x9a3cb has different container ID, probably moved WARNING: File ID 0x9b322 has different container ID, probably moved WARNING: File ID 0x9c1ee has different container ID, probably moved WARNING: File ID 0x9c310 has different container ID, probably moved WARNING: File ID 0x9d095 has different container ID, probably moved WARNING: File ID 0x9dfb3 has different container ID, probably moved WARNING: File ID 0x9e1b3 has different container ID, probably moved WARNING: File ID 0x9f023 has different container ID, probably moved WARNING: File ID 0xa0529 has different container ID, probably moved WARNING: File ID 0xa0d3e has different container ID, probably moved WARNING: File ID 0xa142d has different container ID, probably moved WARNING: File ID 0xa2305 has different container ID, probably moved WARNING: File ID 0xa30ba has different container ID, probably moved WARNING: File ID 0xa4139 has different container ID, probably moved WARNING: File ID 0xa50c8 has different container ID, probably moved WARNING: File ID 0xa54b7 has different container ID, probably moved WARNING: File ID 0xa6468 has different container ID, probably moved WARNING: File ID 0xa6782 has different container ID, probably moved WARNING: File ID 0xa80e5 has different container ID, probably moved WARNING: File ID 0xa8be9 has different container ID, probably moved WARNING: File ID 0xa9457 has different container ID, probably moved WARNING: File ID 0xaa536 has different container ID, probably moved WARNING: File ID 0xab492 has different container ID, probably moved WARNING: File ID 0xabfc2 has different container ID, probably moved WARNING: File ID 0xad360 has different container ID, probably moved WARNING: File ID 0xae252 has different container ID, probably moved WARNING: File ID 0xaf18b has different container ID, probably moved WARNING: File ID 0xb0117 has different container ID, probably moved WARNING: File ID 0xb026a has different container ID, probably moved WARNING: File ID 0xb1293 has different container ID, probably moved WARNING: File ID 0xb1442 has different container ID, probably moved WARNING: File ID 0xb22a0 has different container ID, probably moved WARNING: File ID 0xb4185 has different container ID, probably moved WARNING: File ID 0xb43af has different container ID, probably moved WARNING: File ID 0xb51f5 has different container ID, probably moved WARNING: File ID 0xb60c8 has different container ID, probably moved WARNING: File ID 0xb6d38 has different container ID, probably moved WARNING: File ID 0xb715b has different container ID, probably moved WARNING: File ID 0xb80e5 has different container ID, probably moved WARNING: File ID 0xb9f3d has different container ID, probably moved WARNING: File ID 0xbae89 has different container ID, probably moved WARNING: File ID 0xbafe7 has different container ID, probably moved WARNING: File ID 0xbbf43 has different container ID, probably moved WARNING: File ID 0xbccc0 has different container ID, probably moved WARNING: File ID 0xbde02 has different container ID, probably moved WARNING: File ID 0xbe03b has different container ID, probably moved WARNING: File ID 0xbef78 has different container ID, probably moved WARNING: File ID 0xbff82 has different container ID, probably moved WARNING: File ID 0xc0db4 has different container ID, probably moved WARNING: File ID 0xc1ccd has different container ID, probably moved WARNING: File ID 0xc3885 has different container ID, probably moved WARNING: File ID 0xc4b72 has different container ID, probably moved WARNING: File ID 0xc4e9a has different container ID, probably moved WARNING: File ID 0xc5e8e has different container ID, probably moved WARNING: File ID 0xc62c2 has different container ID, probably moved WARNING: File ID 0xc648c has different container ID, probably moved WARNING: File ID 0xc756c has different container ID, probably moved WARNING: File ID 0xc84cf has different container ID, probably moved WARNING: File ID 0xc8627 has different container ID, probably moved WARNING: File ID 0xc95cc has different container ID, probably moved WARNING: File ID 0xca59b has different container ID, probably moved WARNING: File ID 0xcb7a3 has different container ID, probably moved WARNING: File ID 0xcbf42 has different container ID, probably moved WARNING: File ID 0xcc6e2 has different container ID, probably moved WARNING: File ID 0xcd1d3 has different container ID, probably moved WARNING: File ID 0xce09b has different container ID, probably moved WARNING: File ID 0xcf348 has different container ID, probably moved WARNING: File ID 0xd049d has different container ID, probably moved WARNING: File ID 0xd13ce has different container ID, probably moved WARNING: File ID 0x16a7e has different container ID, probably moved WARNING: File ID 0x16c1c has different container ID, probably moved WARNING: File ID 0x1837c has different container ID, probably moved WARNING: File ID 0x1883f has different container ID, probably moved WARNING: File ID 0x105f82 has different container ID, probably moved WARNING: File ID 0x107936 has different container ID, probably moved WARNING: File ID 0x107acf has different container ID, probably moved WARNING: File ID 0x108377 has different container ID, probably moved WARNING: File ID 0x1093eb has different container ID, probably moved WARNING: File ID 0x1096d8 has different container ID, probably moved WARNING: File ID 0x109b3b has different container ID, probably moved WARNING: File ID 0x109fc0 has different container ID, probably moved WARNING: File ID 0x10954e has different container ID, probably moved WARNING: File ID 0x109de2 has different container ID, probably moved WARNING: File ID 0x109557 has different container ID, probably moved WARNING: File ID 0x109ddc has different container ID, probably moved WARNING: File ID 0x1094fc has different container ID, probably moved WARNING: File ID 0x109dd9 has different container ID, probably moved WARNING: File ID 0x10f003 has different container ID, probably moved WARNING: File ID 0x110f6b has different container ID, probably moved WARNING: File ID 0x111c12 has different container ID, probably moved WARNING: File ID 0x113030 has different container ID, probably moved WARNING: File ID 0x113220 has different container ID, probably moved INFO: Phase 3, check root... INFO: Phase 4, fix unattached objects... WARNING: Object 0x90c1a is unattached. ERROR: Expected raidfile backup/075ab23c/0c/09/o1a does not exist TRACE: Obtained 8 stack frames. TRACE: Stack frame 0: bbstoreaccounts(DumpStackBacktrace()+0x22) [0x80dd832] TRACE: Stack frame 1: bbstoreaccounts(RaidFileRead::Open(int, std::string const&, long long*, int)+0xa3d) [0x8084a9d] TRACE: Stack frame 2: bbstoreaccounts(BackupStoreCheck::CheckUnattachedObjects()+0x536) [0x8070796] TRACE: Stack frame 3: bbstoreaccounts(BackupStoreCheck::Check()+0x47a) [0x806a76a] TRACE: Stack frame 4: bbstoreaccounts(CheckAccount(Configuration&, std::string const&, int, bool, bool)+0x2a0) [0x805dee0] TRACE: Stack frame 5: bbstoreaccounts(main+0x7db) [0x80605bb] TRACE: Stack frame 6: /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0x407bd6] TRACE: Stack frame 7: bbstoreaccounts() [0x805b421] WARNING: Exception thrown: RaidFileException(RaidFileDoesntExist) at RaidFileRead.cpp(1026) Exception: RaidFile RaidFileDoesntExist (Error when accessing a file on the store. Check the store with bbstoreaccounts check.) (2/11) sune at jadis:~$ How to fix? Best regards, Sune M?lgaard -- In answer to the question of why it happened, I offer the modest proposal that our Universe is simply one of those things which happen from time to time. - Edward P. Tryon From dave at bdisystems.co.uk Wed Jan 26 17:38:39 2011 From: dave at bdisystems.co.uk (David Bamford) Date: Wed, 26 Jan 2011 17:38:39 +0000 Subject: [Box Backup] Problem fixing corrupted store Message-ID: <1296063519.1282.11.camel@millhouse.backed-up.net> Hi I have one account on the server which seems to be corrupt and bbstoreaccounts check fix aborts after a couple of hours with a segmentation fault. Is there any way I can make it fix the account? Here are the last few lines output from bbstoreaccounts see attached screenshot. I couldn't cut and paste the text as I had to run it in virt-viewer because the seg fault kills the ssh connection. Thanks Dave -- Dave Bamford BDISystems.co.uk -------------- next part -------------- A non-text attachment was scrubbed... Name: screenshot.png Type: image/png Size: 141993 bytes Desc: not available URL: From chris at qwirx.com Thu Jan 27 21:46:10 2011 From: chris at qwirx.com (Chris Wilson) Date: Thu, 27 Jan 2011 21:46:10 +0000 (GMT) Subject: [Box Backup] Another wish for 2011 In-Reply-To: References: , Message-ID: Hi Leif, On Thu, 27 Jan 2011, Leif Linderstam wrote: >> The first task would be to identify the structures that need changing, >> and understand why the current mechanism is failing and whether we can >> fix it without breaking backwards compatibility. >> >> I don't have an armeb platform, but it might be possible for me to >> build and run an armeb server under qemu, and it might be efficient >> enough (on my 10-year-old desktop box) to be reasonable for testing. >> However I might instead have to send you blind attempts at fixes for >> you to test. > > It seems like a waste of your time to do that. I have compiled from the > source before and can do so again. I will retrieve the source again and > locate the first problematic struct. Then I can either propose a change > in the code, or just point out the problem for you, whatever you feel is > the best. Thanks, I will appreciate your help :) If you can identify the reason for the structure packing mismatch, and a workaround, then feel free to tell me what it is :) Otherwise let me know what you find out, and I'll do my best to help. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Thu Jan 27 21:53:56 2011 From: chris at qwirx.com (Chris Wilson) Date: Thu, 27 Jan 2011 21:53:56 +0000 (GMT) Subject: [Box Backup] Problem fixing corrupted store In-Reply-To: <1296063519.1282.11.camel@millhouse.backed-up.net> References: <1296063519.1282.11.camel@millhouse.backed-up.net> Message-ID: Hi David, On Wed, 26 Jan 2011, David Bamford wrote: > I have one account on the server which seems to be corrupt and > bbstoreaccounts check fix aborts after a couple of hours with a > segmentation fault. Is there any way I can make it fix the account? > > Here are the last few lines output from bbstoreaccounts see attached > screenshot. I couldn't cut and paste the text as I had to run it in > virt-viewer because the seg fault kills the ssh connection. It's very odd that it kills the SSH session. This makes me think that the (virtual?) server started killing processes for some reason, such as running out of memory (which could well be bbstoreaccounts' fault). Can you find any information in your system logs (on the virtual server) around the time that this happened? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Thu Jan 27 21:56:48 2011 From: chris at qwirx.com (Chris Wilson) Date: Thu, 27 Jan 2011 21:56:48 +0000 (GMT) Subject: [Box Backup] Problem fixing corrupted store In-Reply-To: References: <1296063519.1282.11.camel@millhouse.backed-up.net> Message-ID: Hi David, On Thu, 27 Jan 2011, Chris Wilson wrote: > On Wed, 26 Jan 2011, David Bamford wrote: > >> I have one account on the server which seems to be corrupt and >> bbstoreaccounts check fix aborts after a couple of hours with a >> segmentation fault. Is there any way I can make it fix the account? >> >> Here are the last few lines output from bbstoreaccounts see attached >> screenshot. I couldn't cut and paste the text as I had to run it in >> virt-viewer because the seg fault kills the ssh connection. > > It's very odd that it kills the SSH session. This makes me think that the > (virtual?) server started killing processes for some reason, such as running > out of memory (which could well be bbstoreaccounts' fault). Actually I think it is bbstoreaccounts' fault. I can see that it tried to do something with a ridiculously high entry ID, which might have caused ti to try to allocate terabytes of memory in recent (trunk) versions of bbstoreaccounts due to the reference counting model (which I'm going to change). My first question is how it got the idea that this ridiculously high object ID existed in the first place. I'm guessing you're using a 64-bit guest? There could be a loitering bug here, I'll investigate. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From dave at bdisystems.co.uk Fri Jan 28 17:06:46 2011 From: dave at bdisystems.co.uk (David Bamford) Date: Fri, 28 Jan 2011 17:06:46 +0000 Subject: [Box Backup] Problem fixing corrupted store In-Reply-To: References: <1296063519.1282.11.camel@millhouse.backed-up.net> Message-ID: <1296234406.1282.39.camel@millhouse.backed-up.net> On Thu, 2011-01-27 at 21:56 +0000, Chris Wilson wrote: > Hi David, > > On Thu, 27 Jan 2011, Chris Wilson wrote: > > On Wed, 26 Jan 2011, David Bamford wrote: > > > >> I have one account on the server which seems to be corrupt and > >> bbstoreaccounts check fix aborts after a couple of hours with a > >> segmentation fault. Is there any way I can make it fix the account? > >> > >> Here are the last few lines output from bbstoreaccounts see attached > >> screenshot. I couldn't cut and paste the text as I had to run it in > >> virt-viewer because the seg fault kills the ssh connection. > > > > It's very odd that it kills the SSH session. This makes me think that the > > (virtual?) server started killing processes for some reason, such as running > > out of memory (which could well be bbstoreaccounts' fault). > > Actually I think it is bbstoreaccounts' fault. I can see that it tried to > do something with a ridiculously high entry ID, which might have caused ti > to try to allocate terabytes of memory in recent (trunk) versions of > bbstoreaccounts due to the reference counting model (which I'm going to > change). > > My first question is how it got the idea that this ridiculously high > object ID existed in the first place. I'm guessing you're using a 64-bit > guest? There could be a loitering bug here, I'll investigate. > > Cheers, Chris. Thanks Chris I have since renamed the old store for this account and created a new one as I suspect store corruption, but you are right I think its a bstoreaccounts bug where it has picked up this high number. Yes everything is 64 bit (client and server and virtual machine) all running Debian Squeeze. I think the corruption may have been introduced as we had to switch NFS stores recently as one kept failing. We rsync the 2 stores daily on the local net, takes about 6 hours for 3Tb. If you want me to test it I still have the old data. Regards Dave -- Dave Bamford BDISystems.co.uk Log a Support Ticket Is your company data backed up? Read this Office: 020 8390 4588 Mobile: 07770 938 368 BDI Systems Ltd is registered in England and Wales no. 6958030 at New Malden Any prices quoted are subject to VAT at the prevailing rate. If this email is not intended for you please notify us. Thank you. From chris at qwirx.com Sat Jan 29 19:36:29 2011 From: chris at qwirx.com (Chris Wilson) Date: Sat, 29 Jan 2011 19:36:29 +0000 (GMT) Subject: [Box Backup] Problem fixing corrupted store In-Reply-To: <1296234406.1282.39.camel@millhouse.backed-up.net> References: <1296063519.1282.11.camel@millhouse.backed-up.net> <1296234406.1282.39.camel@millhouse.backed-up.net> Message-ID: Hi Dave, On Fri, 28 Jan 2011, David Bamford wrote: >>>> I have one account on the server which seems to be corrupt and >>>> bbstoreaccounts check fix aborts after a couple of hours with a >>>> segmentation fault. Is there any way I can make it fix the account? >>>> >>>> Here are the last few lines output from bbstoreaccounts see attached >>>> screenshot. I couldn't cut and paste the text as I had to run it in >>>> virt-viewer because the seg fault kills the ssh connection. >>> >>> It's very odd that it kills the SSH session. This makes me think that the >>> (virtual?) server started killing processes for some reason, such as running >>> out of memory (which could well be bbstoreaccounts' fault). Did you manage to find anything in the server's logs? >> Actually I think it is bbstoreaccounts' fault. I can see that it tried >> to do something with a ridiculously high entry ID, which might have >> caused ti to try to allocate terabytes of memory in recent (trunk) >> versions of bbstoreaccounts due to the reference counting model (which >> I'm going to change). I couldn't find an obvious cause for this in the bbstoreaccounts code, so I'm looking for other possible explanations. The strange entry appears to have object ID 0x7220363966653131. Do you actually have a file called something like 31/65/66/39/36/20/72/o31.rfw in this account? If so, is there any chance you could send it to me for testing? (I shouldn't be able to read the encrypted file data as I don't have the key). > I have since renamed the old store for this account and created a new > one as I suspect store corruption, but you are right I think its a > bstoreaccounts bug where it has picked up this high number. > Yes everything is 64 bit (client and server and virtual machine) all > running Debian Squeeze. What versions of Box Backup are you running on the client and server? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Sat Jan 29 19:37:35 2011 From: chris at qwirx.com (Chris Wilson) Date: Sat, 29 Jan 2011 19:37:35 +0000 (GMT) Subject: [Box Backup] bbstoreaccounts check fix reults in crash on non-existent raid file In-Reply-To: <4D41ACAA.9060206@molgaard.org> References: <4D41ACAA.9060206@molgaard.org> Message-ID: Hi Sune, On Thu, 27 Jan 2011, Sune M?lgaard wrote: > This shouldn't be, right? No, it seems that bbstoreaccounts somehow got the idea that this file should exist, even though it doesn't actually exist on disk. I'm investigating. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From dave at bdisystems.co.uk Sun Jan 30 18:08:40 2011 From: dave at bdisystems.co.uk (David Bamford) Date: Sun, 30 Jan 2011 18:08:40 +0000 Subject: [Box Backup] Problem fixing corrupted store In-Reply-To: References: <1296063519.1282.11.camel@millhouse.backed-up.net> <1296234406.1282.39.camel@millhouse.backed-up.net> Message-ID: <1296410920.1282.81.camel@millhouse.backed-up.net> On Sat, 2011-01-29 at 19:36 +0000, Chris Wilson wrote: > Hi Dave, > > On Fri, 28 Jan 2011, David Bamford wrote: > > >>>> I have one account on the server which seems to be corrupt and > >>>> bbstoreaccounts check fix aborts after a couple of hours with a > >>>> segmentation fault. Is there any way I can make it fix the account? > >>>> > >>>> Here are the last few lines output from bbstoreaccounts see attached > >>>> screenshot. I couldn't cut and paste the text as I had to run it in > >>>> virt-viewer because the seg fault kills the ssh connection. > >>> > >>> It's very odd that it kills the SSH session. This makes me think that the > >>> (virtual?) server started killing processes for some reason, such as running > >>> out of memory (which could well be bbstoreaccounts' fault). > > Did you manage to find anything in the server's logs? > > >> Actually I think it is bbstoreaccounts' fault. I can see that it tried > >> to do something with a ridiculously high entry ID, which might have > >> caused ti to try to allocate terabytes of memory in recent (trunk) > >> versions of bbstoreaccounts due to the reference counting model (which > >> I'm going to change). > > I couldn't find an obvious cause for this in the bbstoreaccounts code, so > I'm looking for other possible explanations. > > The strange entry appears to have object ID 0x7220363966653131. Do you > actually have a file called something like 31/65/66/39/36/20/72/o31.rfw in > this account? If so, is there any chance you could send it to me for > testing? (I shouldn't be able to read the encrypted file data as I don't > have the key). > > > I have since renamed the old store for this account and created a new > > one as I suspect store corruption, but you are right I think its a > > bstoreaccounts bug where it has picked up this high number. > > Yes everything is 64 bit (client and server and virtual machine) all > > running Debian Squeeze. > > What versions of Box Backup are you running on the client and server? > > Cheers, Chris. Hi Chris I can't find a file like you describe and the store directory structure doesn't go that deep. The versions I am running are 0.11 for client and server. I think its impractical to send the store as its about 50Gb for this account. In fact this is why I found the problem, there were about 25G of deleted files and they were not being removed to make room for new files, so I decreased the soft limit to 24G to try and get rid of them. This didn't work which is why I ran a check. I was also getting the following in the logs... > Jan 25 07:41:51 bart bbstored client=0x00001000[25429]: NOTICE: Login from Client ID 0x00001000 Read/Write > Jan 25 07:41:51 bart bbstored client=0x00001000[25429]: ERROR: Expected raidfile backup/00001000/9a/12/o6f does not exist > Jan 25 07:41:51 bart bbstored client=0x00001000[25429]: WARNING: Exception thrown: RaidFileException(RaidFileDoesntExist) at RaidFileRead.cpp(1086) > Jan 25 07:41:51 bart bbstored client=0x00001000[25429]: NOTICE: Connection statistics for BACKUP-00001000: IN=220 OUT=5537 TOTAL=5757 > Jan 25 07:41:51 bart bbstored[25429]: ERROR: Error in child process, terminating connection: RaidFile RaidFileDoesntExist (Error when accessing a file on the store. > Check the store with bbstoreaccounts check.) (2/11) I wouldn't break a leg trying to fix this as I have recreated the store for this account from scratch and its about 1/3 built already. There must have been corruption which when bbstoreaccounts tried to fix it caused some sort of overflow. Thanks dave -- Dave Bamford BDISystems.co.uk Log a Support Ticket Is your company data backed up? Read this Office: 020 8390 4588 Mobile: 07770 938 368 BDI Systems Ltd is registered in England and Wales no. 6958030 at New Malden Any prices quoted are subject to VAT at the prevailing rate. If this email is not intended for you please notify us. Thank you.