From boxbackup at fluffy.co.uk Thu Apr 1 11:48:08 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 1 Apr 2004 11:48:08 +0100 Subject: [Box Backup] redhat 7.3 In-Reply-To: <20040331130525.M15443@naweb.com> References: <20040331130525.M15443@naweb.com> Message-ID: <101E7FF5-83CA-11D8-A668-000A95AFF7F8@fluffy.co.uk> On 31 Mar 2004, at 14:36, ian at naweb.com wrote: > Hi all. > > Originally this message was going to be a question about a problem. > But I > just solved it so I'll just mention the problem/solution in case > someone would > run into it. > > I have a Redhat 7.3 box which has openssl 0.9.6 installed using rpm > and 0.9.7 > installed manually from the source in /usr/local/ssl (the normal > place). > > Problem was that the configure script would not see my 0.9.7 > installation and > would only see 0.9.6. What I did to fix it was that I reinstalled > openssl but > with the prefix set to redhats normal location. only problem is that > you > would be changing some of the libraries to newere versions. I am not > sure > exactly what consequences it will have but you'll probably have to, > atleast, > restart anything that uses ssl, and at worst recompile something... or > am I > wrong about that? anyway, the exact openssl config command was: > > ./config shared --prefix=/usr --openssldir=/usr/include/ssl > > I had no problem compiling on fedora core 1. So anyway, not sure if > something > can be done to fix the config script or maybe it was something wrong > with my > redhat 7.3 installation. I'll try a different 7.3 box in a couple of > days. You can manually add compile and link options to the configured script, see http://www.fluffy.co.uk/boxbackup/install.html You'd do something like ./configure compile:-I/usr/local/openssl/include link:-L/usr/local/openssl/lib perhaps... the paths are all dependent on your system. I think you subscribed a little to late to read this http://lists.warhead.org.uk/pipermail/boxbackup/2004-March/000140.html which is a version which has experimental support for old versions of OpenSSL. It's not pretty or terribly efficient, but seems to work. It does seem that most Linux systems, and a lot of other ones too, ship with 0.9.6. This is a bit of a pain if you just want to try Box Backup out, so I added this in to enable people to try it without too much hassle. Of course, for real live use, I do recommend upgrading to 0.9.7. Specifically, with the old version * The cipher has to be completely reinitialised every time it's used. This is less efficient than simply resetting the state, as can be done in 0.9.7. * There's no support for switching padding off (which is required to avoid using up huge amounts of extra space). I have to access the OpenSSL internals to emulate this, which is horrendously bad practise. Still, it doesn't have to be forward compatible, so I don't feel to bad about this. Thanks for posting your notes. Ben From boxbackup at fluffy.co.uk Thu Apr 1 23:04:08 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Thu, 1 Apr 2004 17:04:08 -0500 Subject: [Box Backup] redhat 7.3 + another issue In-Reply-To: <20040401110002.2099.81643.Mailman@love.warhead.org.uk> References: <20040401110002.2099.81643.Mailman@love.warhead.org.uk> Message-ID: <20040401214642.M48973@naweb.com> > You can manually add compile and link options to the configured > script, see > > http://www.fluffy.co.uk/boxbackup/install.html > > You'd do something like > > ./configure compile:-I/usr/local/openssl/include > link:-L/usr/local/openssl/lib Yeah,I didn't read that till afterwards. no big deal. It would have been much easier if I had done as you said. but it also got me to upgrade openssl 0.9.7 to the latest subversion > > perhaps... the paths are all dependent on your system. I think if installed manually, openssl always installs to the /usr/local/openssl dir. and normal Configure scripts usually assume /usr/local/openssl i think. > I think you subscribed a little to late to read this > > http://lists.warhead.org.uk/pipermail/boxbackup/2004-March/000140.html > > which is a version which has experimental support for old versions > of OpenSSL. It's not pretty or terribly efficient, but seems to work. thanks, but I didn't actually want to use 0.9.6. I had this problem. I tried on a redhat 8.0 box, (which had the same problem with openssl as above, but after fixing that...), compile had problem with readline. problem is on the redhat 8.0 mahcine I have mysql installed, and it comes with readline support, it seems. so the makebuildenv.pl script was picking up both readline.h in /usr/include/mysql/ and /usr/include/readline/ . I moved the mysql directory out of /usr/include, reran configure, and that got it to finish compiling. Imran From boxbackup at fluffy.co.uk Sat Apr 3 14:35:01 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Sat, 3 Apr 2004 08:35:01 -0500 Subject: [Box Backup] problems.... Message-ID: <20040403131224.M57404@naweb.com> Man, I wrote this super long email describing my problem and error logs. and my stupid webmail client session expired. anyway, basically i setup client & server. was having connection problem and turned on extended logging. found out that my hosts file was incorrectly setup. fixed it and the backup started going. forgot to turn off exteneded logging, so I killed both client & server. and turned off the logging, and restarted server & client. And then i saw that the server continued backing up, so I left the console and came back 15 minuts later and noticed that the backup was stopped at around 4gigs completed and some errors in the error logs. before I show you those errors, i was curious if these compile warnings are anything to worry about: g++ -g -Wall -I../../lib/common -DPLATFORM_LINUX -DPLATFORM_GCC3 -DBOX_VERSION="\"0.04PLUS1\"" -I/usr/local/openssl/include -c RaidFileUtil.cpp -o ../../debug/lib/raidfile/RaidFileUtil.o In file included from ../../lib/common/Box.h:159, from RaidFileUtil.cpp:47: /usr/include/asm/byteorder.h:6:2: warning: #warning using private kernel header; include instead! Server Log: Apr 3 01:31:03 backup bbstored[8063]: Starting daemon (config: /etc/box/bbstored.conf) (version 0.04PLUS1) Apr 3 01:31:03 backup bbstored/hk[8064]: Housekeeping process started Apr 3 01:31:03 backup bbstored/hk[8064]: Starting housekeeping Apr 3 01:31:03 backup bbstored/hk[8064]: On housekeeping, sizes in store do not match calculated sizes, correcting Apr 3 01:31:03 backup bbstored/hk[8064]: different (store,calc): acc 0x00040404, used (230722,230746), old (0,0), deleted (0,0), dirs (3933,3933) Apr 3 01:31:03 backup bbstored/hk[8064]: Finished housekeeping Apr 3 01:31:11 backup bbstored[8063]: Incoming connection from 209.126.207.150 port 54812 (handling in child 8065) Apr 3 01:31:11 backup bbstored[8065]: Certificate CN: BACKUP-040404 Apr 3 01:31:11 backup bbstored[8065]: Login: Client ID 00040404, Read/Write Apr 3 01:31:45 backup bbstored[8063]: Incoming connection from 209.126.207.150 port 54815 (handling in child 8067) Apr 3 01:31:45 backup bbstored[8067]: Certificate CN: BACKUP-040404 Apr 3 01:31:45 backup bbstored[8067]: Login: Client ID 00040404, Read-only Apr 3 01:46:03 backup bbstored/hk[8064]: Starting housekeeping Apr 3 01:46:07 backup bbstored/hk[8064]: Finished housekeeping Apr 3 01:46:57 backup bbstored[8067]: in server child, exception Connection Protocol_Timeout (Probably a network issue between client and server.) (7/41) -- terminating child Apr 3 01:55:48 backup bbstored[8063]: Incoming connection from 209.126.207.150 port 54908 (handling in child 8080) Apr 3 01:55:48 backup bbstored[8080]: Certificate CN: BACKUP-040404 Apr 3 01:55:48 backup bbstored[8080]: Login: Client ID 00040404, Read-only Apr 3 01:55:52 backup bbstored[8080]: in server child, exception Connection TLSReadFailed (Probably a network issue between client and server.) (7/34) -- terminating child Apr 3 01:56:05 backup bbstored[8063]: Incoming connection from 209.126.207.150 port 54909 (handling in child 8081) Apr 3 01:56:05 backup bbstored[8081]: Certificate CN: BACKUP-040404 Apr 3 01:56:10 backup bbstored[8081]: Failed to get write lock (for Client ID 00040404) Apr 3 01:56:10 backup bbstored[8081]: in server child, exception Connection TLSReadFailed (Probably a network issue between client and server.) (7/34) -- terminating child Apr 3 01:56:39 backup bbstored[8063]: Terminating daemon Apr 3 01:56:39 backup bbstored/hk[8064]: Terminating daemon Client Log: Apr 3 01:31:11 slag bbackupd[17014]: Starting daemon (config: /etc/box/bbackupd.conf) (version 0.04PLUS1) Apr 3 01:31:11 slag bbackupd[17014]: Beginning scan of local files Apr 3 01:31:11 slag bbackupd[17014]: Opening connection to server backup.naweb.com... Apr 3 01:31:11 slag bbackupd[17014]: Connection made, login successful Apr 3 01:55:45 slag bbackupd[17014]: Backup object failed, error when reading /home/ronald/Mail/xmycopy11 Apr 3 01:55:45 slag bbackupd[17014]: Error code when uploading was (7/41), Connection Protocol_Timeout (Probably a network issue between client and server.) Apr 3 01:56:05 slag bbackupd[17247]: Starting daemon (config: /etc/box/bbackupd.conf) (version 0.04PLUS1) Apr 3 01:56:05 slag bbackupd[17247]: Beginning scan of local files Apr 3 01:56:05 slag bbackupd[17247]: Opening connection to server backup.naweb.com... Apr 3 01:56:10 slag bbackupd[17247]: Exception caught (7/47), reset state and waiting to retry... Apr 3 01:56:39 slag bbackupd[17014]: Backup object failed, error when reading /home/ronald/Mail/xmycopy12 Apr 3 01:56:39 slag bbackupd[17014]: Error code when uploading was (7/33), Connection TLSWriteFailed (Probably a network issue between client and server.) Apr 3 01:56:39 slag bbackupd[17014]: SSL err during Write: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry Apr 3 01:56:39 slag bbackupd[17014]: Backup object failed, error when reading /home/ronald/Mail/xmycopy13 Apr 3 01:56:39 slag bbackupd[17014]: Error code when uploading was (7/33), Connection TLSWriteFailed (Probably a network issue between client and server.) Apr 3 01:56:39 slag bbackupd[17014]: SSL err during Write: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry Apr 3 01:56:39 slag bbackupd[17014]: SSL err during Write: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry Apr 3 01:56:39 slag bbackupd[17014]: Exception caught (7/33), reset state and waiting to retry... Apr 3 01:57:50 slag bbackupd[17247]: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Apr 3 01:57:50 slag bbackupd[17247]: Beginning scan of local files The current state is (maybe I'm just looking at it wrong and maybe its finished backing up data, but there are lots of errors. here is the server log with the debug version: Apr 3 04:27:11 backup bbstored[11788]: Starting daemon (config: /etc/box/bbstored.conf) (version 0.04PLUS1) Apr 3 04:27:11 backup bbstored/hk[11789]: Housekeeping process started Apr 3 04:27:11 backup bbstored/hk[11789]: Starting housekeeping Apr 3 04:27:11 backup bbstored/hk[11789]: TRACE: Exception thrown: RaidFileException(RaidFileDoesntExist) at RaidFileRead.cpp(1055) Apr 3 04:27:11 backup bbstored/hk[11789]: while housekeeping account 00040404, exception RaidFile RaidFileDoesntExist (2/11) -- aborting housekeeping run for this account Apr 3 04:27:11 backup bbstored/hk[11789]: Finished housekeeping Apr 3 04:28:24 backup bbstored[11788]: Incoming connection from 209.126.207.150 port 55588 (handling in child 11796) Apr 3 04:28:24 backup bbstored[11796]: Certificate CN: BACKUP-040404 Apr 3 04:28:24 backup bbstored[11796]: TRACE: Send block allocation size is 4 Apr 3 04:28:24 backup bbstored[11796]: Receive Version(0x1) Apr 3 04:28:24 backup bbstored[11796]: Send Version(0x1) Apr 3 04:28:24 backup bbstored[11796]: Receive Login(0x40404,0x0) Apr 3 04:28:24 backup bbstored[11796]: Login: Client ID 00040404, Read/Write Apr 3 04:28:24 backup bbstored[11796]: Send LoginConfirmed(0x3d727761647c0,0xf89ab,0x500000,0x5c0000) Apr 3 04:28:24 backup bbstored[11796]: Receive ListDirectory(0x1,0x2,0xc,false) Apr 3 04:28:24 backup bbstored[11796]: Send Success(0x1) Apr 3 04:28:24 backup bbstored[11796]: Sending stream, size 520 Apr 3 04:28:24 backup bbstored[11796]: Receive ListDirectory(0x2,0xffffffff,0xc,true) Apr 3 04:28:24 backup bbstored[11796]: Send Success(0x2) Apr 3 04:28:24 backup bbstored[11796]: Sending stream, size 4609 Apr 3 04:28:24 backup bbstored[11796]: Receive ListDirectory(0x3,0xffffffff,0xc,true) Apr 3 04:28:24 backup bbstored[11796]: Send Success(0x3) Apr 3 04:28:24 backup bbstored[11796]: Sending stream, size 881 Apr 3 04:28:24 backup bbstored[11796]: Receive ListDirectory(0x75,0xffffffff,0xc,true) Apr 3 04:28:24 backup bbstored[11796]: TRACE: Exception thrown: RaidFileException(RaidFileDoesntExist) at RaidFileRead.cpp(1055) Apr 3 04:28:24 backup bbstored[11796]: in server child, exception RaidFile RaidFileDoesntExist (2/11) -- terminating child Apr 3 04:30:04 backup bbstored[11788]: Incoming connection from 209.126.207.150 port 55596 (handling in child 11834) Apr 3 04:30:04 backup bbstored[11834]: Certificate CN: BACKUP-040404 Apr 3 04:30:04 backup bbstored[11834]: TRACE: Send block allocation size is 4 Apr 3 04:30:04 backup bbstored[11834]: Receive Version(0x1) Apr 3 04:30:04 backup bbstored[11834]: Send Version(0x1) Apr 3 04:30:04 backup bbstored[11834]: Receive Login(0x40404,0x0) Apr 3 04:30:04 backup bbstored[11834]: Login: Client ID 00040404, Read/Write Apr 3 04:30:04 backup bbstored[11834]: Send LoginConfirmed(0x3d727761647c0,0xf89ab,0x500000,0x5c0000) Apr 3 04:30:04 backup bbstored[11834]: Receive ListDirectory(0x1,0x2,0xc,false) Apr 3 04:30:04 backup bbstored[11834]: Send Success(0x1) Apr 3 04:30:04 backup bbstored[11834]: Sending stream, size 520 Apr 3 04:30:04 backup bbstored[11834]: Receive ListDirectory(0x2,0xffffffff,0xc,true) Apr 3 04:30:04 backup bbstored[11834]: Send Success(0x2) Apr 3 04:30:04 backup bbstored[11834]: Sending stream, size 4609 Apr 3 04:30:04 backup bbstored[11834]: Receive ListDirectory(0x3,0xffffffff,0xc,true) Apr 3 04:30:04 backup bbstored[11834]: Send Success(0x3) Apr 3 04:30:04 backup bbstored[11834]: Sending stream, size 881 Apr 3 04:30:04 backup bbstored[11834]: Receive ListDirectory(0x75,0xffffffff,0xc,true) Apr 3 04:30:04 backup bbstored[11834]: TRACE: Exception thrown: RaidFileException(RaidFileDoesntExist) at RaidFileRead.cpp(1055) Apr 3 04:30:04 backup bbstored[11834]: in server child, exception RaidFile RaidFileDoesntExist (2/11) -- terminating child Apr 3 04:30:46 backup bbstored/hk[11789]: TRACE: housekeeping received command 't' over interprocess comms Apr 3 04:30:46 backup bbstored[11788]: Terminating daemon Apr 3 04:30:46 backup bbstored/hk[11789]: Terminating daemon client log: Apr 3 04:28:24 slag bbackupd[19322]: Starting daemon (config: /etc/box/bbackupd.conf) (version 0.04PLUS1) Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Set maximum diffing time to 20 seconds Apr 3 04:28:24 slag bbackupd[19322]: Beginning scan of local files Apr 3 04:28:24 slag bbackupd[19322]: Opening connection to server backup.naweb.com... Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Send block allocation size is 4 Apr 3 04:28:24 slag bbackupd[19322]: Connection made, login successful Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Found mount point at / Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Found mount point at /proc Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Found mount point at /dev/pts Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Found mount point at /dev/shm Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Found mount point at /var Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Found mount point at /home Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Found mount point at /store Apr 3 04:28:24 slag bbackupd[19322]: TRACE: new location Apr 3 04:28:24 slag bbackupd[19322]: TRACE: 7 potential mount points Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /dev/pts Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /dev/shm Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /store Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /home Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /proc Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /var Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point / Apr 3 04:28:24 slag bbackupd[19322]: TRACE: mount point chosen for /bin is / Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Reallocating filename encoding/decoding buffer from 2 to 31 Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Block 80bd368 realloc(), but not in list. Error? Or allocated in startup static objects? Apr 3 04:28:24 slag bbackupd[19322]: TRACE: new location Apr 3 04:28:24 slag bbackupd[19322]: TRACE: 7 potential mount points Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /dev/pts Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /dev/shm Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /store Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /home Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /proc Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point /var Apr 3 04:28:24 slag bbackupd[19322]: TRACE: checking against mount point / Apr 3 04:28:24 slag bbackupd[19322]: TRACE: mount point chosen for /boot is / Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Reallocating filename encoding/decoding buffer from 31 to 32 Apr 3 04:28:24 slag bbackupd[19322]: TRACE: new location some more of the similiar trace errors and then: Apr 3 04:28:24 slag bbackupd[19322]: TRACE: mount point chosen for /var is /var Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Reallocating filename encoding/decoding buffer from 34 to 41 Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Reallocating filename encoding/decoding buffer from 41 to 50 Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Reallocating filename encoding/decoding buffer from 50 to 51 Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(334) Apr 3 04:28:24 slag bbackupd[19322]: TRACE: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(334) Apr 3 04:28:24 slag bbackupd[19322]: Exception caught (7/34), reset state and waiting to retry... Apr 3 04:30:04 slag bbackupd[19322]: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Apr 3 04:30:04 slag bbackupd[19322]: Beginning scan of local files Apr 3 04:30:04 slag bbackupd[19322]: Opening connection to server backup.naweb.com... Apr 3 04:30:04 slag bbackupd[19322]: TRACE: Send block allocation size is 4 Apr 3 04:30:04 slag bbackupd[19322]: Connection made, login successful Apr 3 04:30:04 slag bbackupd[19322]: TRACE: Found mount point at / .... Apr 3 04:30:04 slag bbackupd[19322]: TRACE: mount point chosen for /var is /var Apr 3 04:30:04 slag bbackupd[19322]: TRACE: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(334) Apr 3 04:30:04 slag bbackupd[19322]: TRACE: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(334) Apr 3 04:30:04 slag bbackupd[19322]: Exception caught (7/34), reset state and waiting to retry... lemme know what I did wrong. or maybe I'm just overlooking something obvious. Also hopefully i haven't overlooked somehting else to include in this email. let me know if you need any other info. Thanks! Imran From boxbackup at fluffy.co.uk Sat Apr 3 22:13:27 2004 From: boxbackup at fluffy.co.uk (Imran) Date: Sat, 3 Apr 2004 16:13:27 -0500 Subject: [Box Backup] Re: problems.... Message-ID: <20040403205736.M18983@naweb.com> Oh, one thing I forgot to mention is that its setup as raid, i.e. /raid/0.0, /raid/0.1 and /raid/0.2 In /raid/0.0/backup/ dirs, there are write.lock files in each of the IDs dirs (and they aren't there in /raid/0.1 or /raid/0.2). and o01.rf files in all of the /raid/0.0 dirs are of size zero (0). while the other two raid ones are of size 40 for notused accounts, and 520 for the account I was testing. o02.rf file on the other hadnd is 4096 on /raid/0.0 and /raid/0.2 but 513 bytes on /raid/0.1 (of course this file only exists for the account I was testing). also, if, lets say, /raid/0.0 harddrive crashed and I was to put in a new harddrive, wouldn't it rebuild the backup dir? i renamed the backup dir, and created a new backup dir with the correct permissions, but i just get lots of errors. maybe I have to give it a commandline paramater or maybe that isn't functional yet.. Imran From boxbackup at fluffy.co.uk Mon Apr 5 00:55:34 2004 From: boxbackup at fluffy.co.uk (Imran) Date: Sun, 4 Apr 2004 18:55:34 -0500 Subject: [Box Backup] Some mods Message-ID: <20040404233845.M30266@naweb.com> I don't know if its usefull or goes against your policy, but i made some changes to 0.4PLUS1 for better openssl support. Basically this only usefull for people that may have openssl installed from packages and then they manually compile openssl from openssl.org source tgz. Problem: Without this, I had to 'configure' using the compile: and link: paramater. and afterwards, I had to modify all the scripts that called the openssl binary and had to prepend /usr/local/ssl/bin/ to them. Files affected: Modified makebuildenv.pl Templatized the scripts/* files a little and put them into scripts.tmpl subdir Usage: ./configure openssl:/usr/local/ssl Changes: what this does is effectively does is 'configure compile:-I/usr/local/ssl/include link:-L/usr/local/ssl/lib'. but problem with that is manually installed openssl is will have openssl binary in /usr/local/ssl/bin directory. so I modified the scripts and put in a tempalte variable that gets replaced and new scripts files generated which have the openssl path variable. Issues: This makes the parcels thing not universal. i mean, ifyou compile on a machine with manually installed openssl and then try to run the parcel on a machine that has openssl installed with rpm etc. and not from source from openssl.org. But then again its probably not a problem, since you would want to make parcels for homogenous sets of systems. NOTE: my modifications of the scripts (that are now in scripts.tmpl directory) could be done a little better. (just realized it when i was writing this email). Instead of hard wiring the openssl to /usr/local/ssl/bin, i should have just added the /urs/local/ssl/bin to beginning of PATH. that way if openssl isn't there, it will atleast find it in /usr/bin etc. And this would make the parcels little less unique. Download: http://www.niazi.net/changes-0.4PLUS1.tgz It includes a diff file, as well as the new & old makebuildenv files. also there is a scripts.tmpl directory, as well as a CHANGES.txt which is a basic overview. Imran From boxbackup at fluffy.co.uk Mon Apr 5 07:19:02 2004 From: boxbackup at fluffy.co.uk (Imran) Date: Mon, 5 Apr 2004 01:19:02 -0500 Subject: [Box Backup] How to stop the server/client? Message-ID: <20040405002711.M57152@naweb.com> looks the initial bit of problems I had were with stopping the client/agent. What do I need to do to stop the bbackupd or bbstored? oh, after the problem email i sent couple of days ago, i decided to just blow awawy the /raid/*/backup/ directories and start fresh. afterwards the backup ran great... for about a day or so backing up. anyway i was testing the server out, see if I can kill it and restart it. and well you can't. now I have lots of RaidFile RaidFileDoesntExist (2/11) errors. and every client that tries to connect fails. (4 clients). How can I make it more resilient and/or recover from the raid problems? Atleaswt the errors need to have better outputs. if there is an error, print out what its looking for ,etc. the errors are too generic. btw ,I am trying to run this software on our production servers. I'll try to contribute code... but haven't coded in C/C++ in years. i do code perl however, so maybe help in that area. I will help in testing. I have some QA experience, professionally and personally for my own business. anyway, so far the program is excellent. besides the little things here and there, it works quite well. Pretty impressive set of functionality. I really like it. been looking for a similiar program for a little bit. Ok, gonna get back to work. gonna reinstall it and probe its limitations... Imran From boxbackup at fluffy.co.uk Mon Apr 5 09:25:58 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 5 Apr 2004 09:25:58 +0100 Subject: [Box Backup] problems.... In-Reply-To: <20040403131224.M57404@naweb.com> References: <20040403131224.M57404@naweb.com> Message-ID: On 3 Apr 2004, at 14:35, ian at naweb.com wrote: > Man, I wrote this super long email describing my problem and error > logs. and > my stupid webmail client session expired. Web developers never seem to think about these things. Expiry times on webmail is something you really need to get right! > > anyway, basically i setup client & server. was having connection > problem and > turned on extended logging. found out that my hosts file was > incorrectly > setup. fixed it and the backup started going. forgot to turn off > exteneded > logging, so I killed both client & server. and turned off the > logging, and > restarted server & client. And then i saw that the server continued > backing > up, so I left the console and came back 15 minuts later and noticed > that the > backup was stopped at around 4gigs completed and some errors in the > error > logs. before I show you those errors, i was curious if these compile > warnings > are anything to worry about: > > > > g++ -g -Wall -I../../lib/common -DPLATFORM_LINUX -DPLATFORM_GCC3 > -DBOX_VERSION="\"0.04PLUS1\"" -I/usr/local/openssl/include -c > RaidFileUtil.cpp -o ../../debug/lib/raidfile/RaidFileUtil.o > In file included from ../../lib/common/Box.h:159, > from RaidFileUtil.cpp:47: > /usr/include/asm/byteorder.h:6:2: warning: #warning using private > kernel > header; include instead! That's nothing to worry about, just a Linux thing. I needed a particular platform thing, and that's the only place I could find it. And then I found in later kernel versions, it's not allowed. [snip] > > lemme know what I did wrong. or maybe I'm just overlooking something > obvious. > Also hopefully i haven't overlooked somehting else to include in > this > email. let me know if you need any other info. I'm not entirely sure what's gone wrong, but from the errors about not being able to get the write lock, it looks like you're running multiple bbackupd clients against the same store account. There are a few errors which would be perfectly normal when a client disconnects, but will get cleared up eventually. So completely normal, I think, unless they get repeated lots of times. It's the repeated RaidFile error which worries me, but you mention in a later email that you deleted all the directories. There's a good chance it'll get confused if you do that while it's running. Ben From boxbackup at fluffy.co.uk Mon Apr 5 09:30:35 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 5 Apr 2004 09:30:35 +0100 Subject: [Box Backup] Re: problems.... In-Reply-To: <20040403205736.M18983@naweb.com> References: <20040403205736.M18983@naweb.com> Message-ID: <82583C1A-86DB-11D8-98E0-000A95E8BABA@fluffy.co.uk> On 3 Apr 2004, at 22:13, Imran wrote: > Oh, one thing I forgot to mention is that its setup as raid, i.e. > /raid/0.0, > /raid/0.1 and /raid/0.2 In /raid/0.0/backup/ dirs, there are > write.lock > files in each of the IDs dirs (and they aren't there in /raid/0.1 or > /raid/0.2). > > and o01.rf files in all of the /raid/0.0 dirs are of size zero (0). > while the > other two raid ones are of size 40 for notused accounts, and 520 for > the > account I was testing. > > > o02.rf file on the other hadnd is 4096 on /raid/0.0 and /raid/0.2 but > 513 > bytes on /raid/0.1 (of course this file only exists for the account > I was > testing). This is all perfectly normal. A fairly weird way of storing data is used in the raidfile system to make thing efficient on disc space and processor, and you're just seeing a few examples. > > also, if, lets say, /raid/0.0 harddrive crashed and I was to put in a > new > harddrive, wouldn't it rebuild the backup dir? i renamed the backup > dir, and > created a new backup dir with the correct permissions, but i just get > lots of > errors. But it should have still worked, I hope? > maybe I have to give it a commandline paramater or maybe that isn't > functional yet.. At the moment, there are two things I need to write: 1) raidfiled -- turns normal files into raidfiles in another process. This will make things vastly more efficient. Also it'll recover individual files as they're found. 2) raidfileutil -- does stuff like rebuilding arrays. They're not big things, it's just getting around to it. All the complex code is already written, it just needs to be packaged up into these utilities. The other functionality is more important to get tested, as it's more complex -- really it's just a matter of priorities. Ben From boxbackup at fluffy.co.uk Mon Apr 5 09:35:25 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 5 Apr 2004 09:35:25 +0100 Subject: [Box Backup] Some mods In-Reply-To: <20040404233845.M30266@naweb.com> References: <20040404233845.M30266@naweb.com> Message-ID: <2F5D1732-86DC-11D8-98E0-000A95E8BABA@fluffy.co.uk> On 5 Apr 2004, at 00:55, Imran wrote: > I don't know if its usefull or goes against your policy, but i made > some > changes to 0.4PLUS1 for better openssl support. > > Basically this only usefull for people that may have openssl installed > from > packages and then they manually compile openssl from openssl.org > source tgz. OpenSSL is the one thing which really causes problems when trying to compile Box Backup -- I'm very keen on any suggestions which make this easier. I'll check it out and incorporate the functionality. In general, I really appreciate patches. I do need a short note saying you're happy for them to go out under the current license, though. > > Issues: > This makes the parcels thing not universal. i mean, ifyou compile on a > machine with manually installed openssl and then try to run the parcel > on a > machine that has openssl installed with rpm etc. and not from source > from > openssl.org. > > But then again its probably not a problem, since you would want to make > parcels for homogenous sets of systems. I don't think this is a problem. Parcels are not meant to replace the ports or packages system for your platform. It's more an administrative convenience than anything else. Ben From boxbackup at fluffy.co.uk Mon Apr 5 09:45:25 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 5 Apr 2004 09:45:25 +0100 Subject: [Box Backup] How to stop the server/client? In-Reply-To: <20040405002711.M57152@naweb.com> References: <20040405002711.M57152@naweb.com> Message-ID: <9524638E-86DD-11D8-98E0-000A95E8BABA@fluffy.co.uk> On 5 Apr 2004, at 07:19, Imran wrote: > looks the initial bit of problems I had were with stopping the > client/agent. > > What do I need to do to stop the bbackupd or bbstored? Send a signal to the main server process. The PID is in /var/run/*.pid. xargs kill < /var/run/bbackupd.pid However... bbackupd will not stop until it's finished the current backup run, and the bbstored child processes handling the connections will not stop until that connection has finished. I should probably change this behaviour. :-) > > oh, after the problem email i sent couple of days ago, i decided to > just blow > awawy the /raid/*/backup/ directories and start fresh. afterwards the > backup > ran great... for about a day or so backing up. anyway i was testing > the > server out, see if I can kill it and restart it. and well you can't. > now I > have lots of RaidFile RaidFileDoesntExist (2/11) errors. and every > client > that tries to connect fails. (4 clients). I suspect you've killed the main server, then left the child processes in place. These would write just enough stuff to allow a new connection to succeed when they terminate, but then when it tried to access a file, it would fail. > > How can I make it more resilient and/or recover from the raid problems? Make sure everything is killed. Delete the raidfile directories. Make a zero length /etc/box/bbstored/accounts.txt file. Recreate the accounts with the bbstoreaccounts utility. > Atleaswt the errors need to have better outputs. if there is an > error, print > out what its looking for ,etc. the errors are too generic. I am aware that the errors are not terribly helpful. You seem to be running the debug versions, so you can find file and line numbers in the logs -- the comments in the source are more helpful. This is not a good solution, I know. It's something for me to get to soon. The problem is that I didn't originally intend to open source this software. If I had from the start, I would have made error messages more comprehensible to those who haven't written it. > > btw ,I am trying to run this software on our production servers. I'll > try to > contribute code... but haven't coded in C/C++ in years. i do code perl > however, so maybe help in that area. I will help in testing. I have > some QA > experience, professionally and personally for my own business. Thanks! Right now, I need lots of people to run it and let me know about their experiences. > > anyway, so far the program is excellent. besides the little things > here and > there, it works quite well. Pretty impressive set of functionality. > I really > like it. been looking for a similiar program for a little bit. > > Ok, gonna get back to work. gonna reinstall it and probe its > limitations... I'm glad it does what you need. I'll try and sort out any problems you find as quickly as I can, and of course, welcome suggestions for extra functionality. Ben From boxbackup at fluffy.co.uk Mon Apr 5 10:14:39 2004 From: boxbackup at fluffy.co.uk (Imran) Date: Mon, 5 Apr 2004 04:14:39 -0500 Subject: [Box Backup] Some mods Message-ID: <20040405090943.M24418@naweb.com> Oh forgot to include the license part. And I released my changes to 0.4PLUS1 into the BSD license of box backup. well I'm not sure the exact wording. would that suffice? Imran From boxbackup at fluffy.co.uk Mon Apr 5 10:45:44 2004 From: boxbackup at fluffy.co.uk (Imran) Date: Mon, 5 Apr 2004 04:45:44 -0500 Subject: [Box Backup] Re: problems.... Message-ID: <20040405091857.M30821@naweb.com> ok, i just turned off digest mode. not even sure why i set it to digest mode. A pain to reply to messages. >> >> also, if, lets say, /raid/0.0 harddrive crashed and I was to put in a >> new >> harddrive, wouldn't it rebuild the backup dir? i renamed the backup >> dir, and >> created a new backup dir with the correct permissions, but i just get >> lots of >> errors. > >But it should have still worked, I hope? Well the server was stuck. when a client would connect it would get the error that I pasted. and the server side would have the can't find file error. or something like that. I was thinking that maybe the /raid/0.0 setup was messed up and so i tried to erase that dir and hoped that maybe the system would automagically recover. but the problem either was in a different raid dir, or that it was some corruption of some files. This is the same state I was in today. backup was running well for about 18 or 20 hours. one thing I noticed is that if i am in the bbackupquery, and press ^D (ctrl-D) it seg faults. I forgot to test if it does the same when type in exit. anyway, i tried to do a kill on the server to stop it, and again same problem. on restart, if a client tries to connect, server spits a RaidFileDoesntExist (2/11) error. I restarted the server, and it gave me a "while housekeeping account xxx, exception RaidFile RaidFileDoesntExist (2/11) -- aborting housekeeping run for this account" line in the syslog for each of accounts I have installed (four). and when I start backupquery, i can do an list and it will ilst the root dirs. and then if I cd to any dir and do a list, it dies: query > ls 00000002 -d--- bin 00000003 -d--- boot 00000004 -d--- etc 00000005 -d--- home 00000006 -d--- initrd 00000007 -d--- lib 00000008 -d--- root 00000009 -d--- sbin 0000000a -d--- usr 0000000b -d--- var query > cd etc query > ls Exception: Connection TLSReadFailed (Probably a network issue between client and server.) (7/34) From boxbackup at fluffy.co.uk Tue Apr 6 01:43:53 2004 From: boxbackup at fluffy.co.uk (Pascal Lalonde) Date: Mon, 5 Apr 2004 20:43:53 -0400 Subject: [Box Backup] Works great since 4PLUS1 Message-ID: <20040406004353.GA6254@plasma.doom.und> Just to report success: We are running a server (0.04PLUS1) and 6 clients, and everything seems to work fine. Among those 6, there is also a Debian 3.0 stable. We've had to install OpenSSL-0.9.7 by hand, but with the configure options, everything went fine. The server is a P3 600, with a 120G IDE drive dedicated for the backups. So except for raid which I didn't test (only one HD), the rest is working fine. And bbstored maintains quite a small footprint. Pascal Lalonde From boxbackup at fluffy.co.uk Tue Apr 6 10:15:01 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 6 Apr 2004 10:15:01 +0100 Subject: [Box Backup] Re: problems.... In-Reply-To: <20040405091857.M30821@naweb.com> References: <20040405091857.M30821@naweb.com> Message-ID: On 5 Apr 2004, at 10:45, Imran wrote: > ok, i just turned off digest mode. not even sure why i set it to > digest mode. > A pain to reply to messages. > >>> >>> also, if, lets say, /raid/0.0 harddrive crashed and I was to put in a >>> new >>> harddrive, wouldn't it rebuild the backup dir? i renamed the backup >>> dir, and >>> created a new backup dir with the correct permissions, but i just get >>> lots of >>> errors. >> >> But it should have still worked, I hope? > > Well the server was stuck. when a client would connect it would get > the error > that I pasted. and the server side would have the can't find file > error. or > something like that. I was thinking that maybe the /raid/0.0 setup > was messed > up and so i tried to erase that dir and hoped that maybe the system > would > automagically recover. but the problem either was in a different raid > dir, or > that it was some corruption of some files. > > > This is the same state I was in today. backup was running well for > about 18 > or 20 hours. one thing I noticed is that if i am in the bbackupquery, > and > press ^D (ctrl-D) it seg faults. I've fixed that one! > I forgot to test if it does the same when > type in exit. anyway, i tried to do a kill on the server to stop it, > and > again same problem. on restart, if a client tries to connect, server > spits a > RaidFileDoesntExist (2/11) error. > > I restarted the server, and it gave me a "while housekeeping account > xxx, > exception RaidFile RaidFileDoesntExist (2/11) -- aborting housekeeping > run for > this account" line in the syslog for each of accounts I have installed > (four). I believe you deleted all the store files while the server was running. This is not something that the server is designed to cope with (although it will cope OK if you just delete one of the raid directories, but needs a new utility to be written to rebuild it). Can you replicate these problems after doing a clean re-install of the server (removing all data and configuration info)? It does sound like you've done something which has messed things up severely, and this wasn't done with any of the distributed software. Thanks, Ben From boxbackup at fluffy.co.uk Tue Apr 6 13:39:27 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 6 Apr 2004 13:39:27 +0100 Subject: [Box Backup] Works great since 4PLUS1 In-Reply-To: <20040406004353.GA6254@plasma.doom.und> References: <20040406004353.GA6254@plasma.doom.und> Message-ID: <716AFBD8-87C7-11D8-A485-000A95AFF7F8@fluffy.co.uk> On 6 Apr 2004, at 01:43, Pascal Lalonde wrote: > > Just to report success: > > We are running a server (0.04PLUS1) and 6 clients, and everything seems > to work fine. Among those 6, there is also a Debian 3.0 stable. We've > had to install OpenSSL-0.9.7 by hand, but with the configure options, > everything went fine. That's all good news! > > The server is a P3 600, with a 120G IDE drive dedicated for the > backups. So except for raid which I didn't test (only one HD), the rest > is working fine. And bbstored maintains quite a small footprint. The bbstored main process shouldn't grow at all. And since the child processes when something connects are just dumping files onto a disc, those shouldn't use much memory either. How much memory are the bbackupd processes using? I'm giving the raidfile stuff a good test, so it's good to have someone testing it in non-raidfile mode. Ben From boxbackup at fluffy.co.uk Tue Apr 6 14:33:56 2004 From: boxbackup at fluffy.co.uk (Pascal Lalonde) Date: Tue, 6 Apr 2004 09:33:56 -0400 Subject: [Box Backup] Works great since 4PLUS1 In-Reply-To: <716AFBD8-87C7-11D8-A485-000A95AFF7F8@fluffy.co.uk> References: <20040406004353.GA6254@plasma.doom.und> <716AFBD8-87C7-11D8-A485-000A95AFF7F8@fluffy.co.uk> Message-ID: <20040406133356.GA4718@plasma.doom.und> On Tue, Apr 06, 2004 at 01:39:27PM +0100, Ben Summers wrote: > > How much memory are the bbackupd processes using? Box #1: root 22909 0.0 1.4 2280 3540 ?? I 29Mar04 42:47.41 bbackupd: idle (bbackupd) Box #2: root 9959 0.0 3.5 1060 2304 ?? I 29Mar04 6:43.67 bbackupd: idle (bbackupd) Box #3: root 1899 0.0 0.9 1208 2424 ?? I 29Mar04 16:34.14 bbackupd: idle (bbackupd) Box #4: (This one runs Linux) root 18794 1.2 1.8 3600 2364 ? S Apr05 13:51 bbackupd And the server: _bbstored 23489 0.0 0.7 968 1340 ?? S 29Mar04 0:01.37 bbstored: server (bbstored) _bbstored 8024 0.0 0.7 1408 1384 ?? S 29Mar04 25:22.65 bbstored: housekeeping, idle (bbstored) Pascal Lalonde From boxbackup at fluffy.co.uk Tue Apr 6 14:37:32 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 6 Apr 2004 14:37:32 +0100 Subject: [Box Backup] Works great since 4PLUS1 In-Reply-To: <20040406133356.GA4718@plasma.doom.und> References: <20040406004353.GA6254@plasma.doom.und> <716AFBD8-87C7-11D8-A485-000A95AFF7F8@fluffy.co.uk> <20040406133356.GA4718@plasma.doom.und> Message-ID: <8E9A1415-87CF-11D8-A485-000A95AFF7F8@fluffy.co.uk> On 6 Apr 2004, at 14:33, Pascal Lalonde wrote: > On Tue, Apr 06, 2004 at 01:39:27PM +0100, Ben Summers wrote: >> >> How much memory are the bbackupd processes using? > > Box #1: > root 22909 0.0 1.4 2280 3540 ?? I 29Mar04 42:47.41 > bbackupd: idle (bbackupd) > > Box #2: > root 9959 0.0 3.5 1060 2304 ?? I 29Mar04 6:43.67 > bbackupd: idle (bbackupd) > > Box #3: > root 1899 0.0 0.9 1208 2424 ?? I 29Mar04 16:34.14 > bbackupd: idle (bbackupd) > > Box #4: (This one runs Linux) > root 18794 1.2 1.8 3600 2364 ? S Apr05 13:51 bbackupd > > And the server: > _bbstored 23489 0.0 0.7 968 1340 ?? S 29Mar04 0:01.37 > bbstored: server (bbstored) > _bbstored 8024 0.0 0.7 1408 1384 ?? S 29Mar04 25:22.65 > bbstored: housekeeping, idle (bbstored) That all looks about right for a few hundred Mb -- few Gb on each box (depending on how many directories you have). It's nice to see that my hard work persuading the STL libraries not to leak memory all over the place has paid off! Ben From boxbackup at fluffy.co.uk Tue Apr 6 17:19:58 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 6 Apr 2004 17:19:58 +0100 Subject: [Box Backup] 0.04PLUS2 Message-ID: <3FAF174B-87E6-11D8-A485-000A95AFF7F8@fluffy.co.uk> I've just uploaded another preview version. http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.04PLUS2.tgz This is basically a few bug fixes, but the significant addition is that the server is a bit more tolerant of being aborted unexpectedly. I think this might help with Imran's problems. Ben From boxbackup at fluffy.co.uk Wed Apr 7 01:10:52 2004 From: boxbackup at fluffy.co.uk (mailing lists) Date: Tue, 06 Apr 2004 21:10:52 -0300 Subject: [Box Backup] error Message-ID: <4073470C.5040708@coredump.com.ar> Hello I'm running an old version of boxbackup (0.03) on my home network. It is backing up a set of documents and graphics on from 2 o 3 directories on my ~ to a remote server. This was working fine when I set it up. I'd been a bit busy these last two months and found it broken last night. I'm seen this error on the client side: Apr 6 20:28:58 dante bbackupd[21512]: Exception caught (3/15), reset state and waiting to retry... Apr 6 20:47:42 dante bbackupd[21512]: Exception caught (3/15), reset state and waiting to retry... I checked RaidFileException.h and 15 is: DirectoryIncomplete = 15, but it doesn't really help and I'm not sure if that file is where I should check. I don't care about backedup documents as originals are fine, but I like the idea of having these things encrypted and backuped remotelly. Should I destroy the whole setup (directoryincomplete sounds scary) and restart with latest or is there any way to fix things up? Thanks Juan From boxbackup at fluffy.co.uk Wed Apr 7 09:20:45 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 7 Apr 2004 09:20:45 +0100 Subject: [Box Backup] error In-Reply-To: <4073470C.5040708@coredump.com.ar> References: <4073470C.5040708@coredump.com.ar> Message-ID: <780C826B-886C-11D8-B6A1-000A95AFF7F8@fluffy.co.uk> On 7 Apr 2004, at 01:10, mailing lists wrote: > Hello > > I'm running an old version of boxbackup (0.03) on my home network. > > It is backing up a set of documents and graphics on from 2 o > 3 directories on my ~ to a remote server. > > This was working fine when I set it up. I'd been a bit busy these > last two months and found it broken last night. > > I'm seen this error on the client side: > > Apr 6 20:28:58 dante bbackupd[21512]: Exception caught (3/15), reset > state and waiting to retry... > Apr 6 20:47:42 dante bbackupd[21512]: Exception caught (3/15), reset > state and waiting to retry... > > I checked RaidFileException.h and 15 is: DirectoryIncomplete = 15, > but it > doesn't really help and I'm not sure if that file is where I should > check. (3/15) is a Server SocketConnectError error, and means that the bbackupd daemon has failed to connect to the server. Check that server is still running, the hostname is correct, etc. I am aware that the messages are a bit unhelpful, especially in 0.03. It's a bit better in newer versions -- you would have seen an explanation in this case. > > I don't care about backedup documents as originals are fine, but > I like the idea of having these things encrypted and backuped > remotelly. Off-site backups are important! > > Should I destroy the whole setup (directoryincomplete sounds scary) > and > restart with latest or is there any way to fix things up? You could just make sure that the server is running OK. You don't say whether these are continual errors, or whether it manages to connect sometimes, or what the server is logging. (Check the documentation on the web site about enabling more logging -- you may only be seeing the error logs.) However, 0.03 has a few bugs which may affect the integrity of your backups. I would recommend moving to the latest version. It would be best to move the existing backups on the server to the side, get a complete new backup, and then delete the old backups. You might as well use the latest "preview" version, here http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.04PLUS2.tgz Let me know how you get on. Ben From boxbackup at fluffy.co.uk Wed Apr 7 12:01:24 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 7 Apr 2004 12:01:24 +0100 Subject: [Box Backup] Development plan Message-ID: Here's a plan for the immediate future of development of Box Backup. I'd welcome any suggestions and comments. * Release 0.05 early next week. This will be basically 0.04PLUS2 with a few minor extras in the config script. * Do a little more work, adding a few new features and complete some of the "TODO" items (see the TODO.txt file in the distribution). - Would a "maximum file size to backup" option be useful? - Any other requests? * Release 0.06, which should be basically feature complete. * Write the final raidfile tools (raidfiled, for out of process raidification, and raidfileutil for checking and rebuilding arrays) * More testing * Maybe some more work on the NetBSD, FreeBSD and Darwin ports? * Release 0.07 (or 1.00?) and advertise it as "stable", ready for production use. I trust people are keeping an eye on their backups, and running verifications occasionally... /usr/local/bin/bbackupquery "compare -a" quit Thanks for all your help with finding problems and creating a nice stable backup system. Ben From boxbackup at fluffy.co.uk Wed Apr 7 20:14:11 2004 From: boxbackup at fluffy.co.uk (Pascal Lalonde) Date: Wed, 7 Apr 2004 15:14:11 -0400 Subject: [Box Backup] Development plan In-Reply-To: References: Message-ID: <20040407191411.GA4965@plasma.doom.und> On Wed, Apr 07, 2004 at 12:01:24PM +0100, Ben Summers wrote: > > * Do a little more work, adding a few new features and complete some of > the "TODO" items (see the TODO.txt file in the distribution). > - Would a "maximum file size to backup" option be useful? Maybe, but I don't see any use for me. I'd be afraid to use it, just in case I have a very important big file which doesn't get backep up because I set a limit during the installation (and of course, I forgot about that limit). > * Maybe some more work on the NetBSD, FreeBSD and Darwin ports? I would've thought Linux is more of a problem than these... Lack of testers? > > * Release 0.07 (or 1.00?) and advertise it as "stable", ready for > production use. > There is one thing that I think would be really important before making it a 1.0 production release (at least, in my opinion), which is: tagging. For example, suppose I wish to keep a month behind, and I need to be able to recover the data exactly as it was 20 days ago (for example, we have a big database, which got corrupted and since this specific table is rarely used, we noticed 20 days later). Of course, BoxBackup doesn't can't distinguish corruption from normal changes, so the current version in the backup store is corrupted too. In order to be able to get back to a non-corrupted state, I need to ensure that: 1) the older files are still there 2) I can recover ALL of the database's related files for that specific date What I could do is tag the store each day, and then I can go back any day. But for this to work, tagged files must not be erased during housekeeping, so maybe there could be a flag to indicate bbstored to not erase files associated with a specific tag until the flag is removed (like once a month, after dumping the data to physical media). This could be a lifesaver in case someone accidentally erases a file, especially if the store is almost full (the file marked as deleted would soon get cleaned by bbstored's housekeeping). I don't know how the soft limit should deal with such tags. Maybe it would be appropriate to create more limits, like maximum history size, or dedicating a % of quota to history only. The other features I already spoke of in previous posts are mostly toys I would like to see (like not storing different versions of a file in full, just keep the differences), but I don't think they're critical for a 1.0 release. As you say, hard drive space is cheap nowadays. > > I trust people are keeping an eye on their backups, and running > verifications occasionally... > > /usr/local/bin/bbackupquery "compare -a" quit Everything looks good. Pascal Lalonde From boxbackup at fluffy.co.uk Thu Apr 8 10:30:15 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 8 Apr 2004 10:30:15 +0100 Subject: [Box Backup] Development plan In-Reply-To: <20040407191411.GA4965@plasma.doom.und> References: <20040407191411.GA4965@plasma.doom.und> Message-ID: <57E0ECD6-893F-11D8-A120-000A95AFF7F8@fluffy.co.uk> On 7 Apr 2004, at 20:14, Pascal Lalonde wrote: > On Wed, Apr 07, 2004 at 12:01:24PM +0100, Ben Summers wrote: >> >> * Do a little more work, adding a few new features and complete some >> of >> the "TODO" items (see the TODO.txt file in the distribution). >> - Would a "maximum file size to backup" option be useful? > Maybe, but I don't see any use for me. I'd be afraid to use it, just in > case I have a very important big file which doesn't get backep up > because I set a limit during the installation (and of course, I forgot > about that limit). That would be the worry. It might be better addressed by using regex excludes for known file types. But it could be useful, maybe? > >> * Maybe some more work on the NetBSD, FreeBSD and Darwin ports? > I would've thought Linux is more of a problem than these... Lack of > testers? Linux is more of a problem, definitely. With FreeBSD and NetBSD, there are just minor little issues which need looking at (which shouldn't affect normal use, but makes the tests fail). However, with Darwin there needs to be a lot of stuff done to cope with the funny Macintosh resource forks. I don't know of anyone running it on FreeBSD or NetBSD. > >> >> * Release 0.07 (or 1.00?) and advertise it as "stable", ready for >> production use. >> > There is one thing that I think would be really important before making > it a 1.0 production release (at least, in my opinion), which is: > tagging. Good point. I'm calling this "marking", and forgot all about it when writing this plan. > > For example, suppose I wish to keep a month behind, and I need to be > able to recover the data exactly as it was 20 days ago (for example, we > have a big database, which got corrupted and since this specific table > is rarely used, we noticed 20 days later). Of course, BoxBackup doesn't > can't distinguish corruption from normal changes, so the current > version > in the backup store is corrupted too. In order to be able to get back > to a > non-corrupted state, I need to ensure that: > 1) the older files are still there > 2) I can recover ALL of the database's related files for that specific > date > > What I could do is tag the store each day, and then I can go back any > day. But for this to work, tagged files must not be erased during > housekeeping, so maybe there could be a flag to indicate bbstored to > not > erase files associated with a specific tag until the flag is removed > (like once a month, after dumping the data to physical media). My plan is to define a set of marks. The current state of files can be marked, either automatically by bbackupd when it logs in at certain times, or manually by bbackupctl. In bbackupquery, you'll be able to list the current marks, and then switch to a "view" of that mark. When viewing a mark, you traverse (and restore from) the state of the store when that mark happened. > > This could be a lifesaver in case someone accidentally erases a file, > especially if the store is almost full (the file marked as deleted > would soon get cleaned by bbstored's housekeeping). Absolutely. It is required, especially as one of my aims is to emulate the behaviour of a sensible tape backup scheme. > > I don't know how the soft limit should deal with such tags. Maybe it > would be appropriate to create more limits, like maximum history size, > or dedicating a % of quota to history only. Here's a history of a file: 1 -- mark 1 2 -- mark 2 3 4 5 -- mark 3 6 7 -- mark 4 8 9 -- current version The files will be deleted in this order: 3, 4, 6, 8, 1, 2, 5, 7 That is, it will delete older versions first which aren't marked, then marked versions, oldest mark first. Within all files, it will delete the oldest files in these lists until it has cleared enough space. I suppose it will be worth having a "keep n marked versions" option, so you can keep the current file + n older versions. But then you couldn't treat weekly marks differently to daily marks. I'm sure I'll think of something nice and simple. But then, maybe you'd just let daily backups happen, and only mark the weekly states? I will add marking into 0.06... or maybe finish everything else off, and put it in 0.07? Depends on how complex it is, I suppose. > > The other features I already spoke of in previous posts are mostly toys > I would like to see (like not storing different versions of a file in > full, just keep the differences), but I don't think they're critical > for > a 1.0 release. As you say, hard drive space is cheap nowadays. I do think this is some way in the future. At the moment I'm more interested in correctness than efficiency. (But that's not to say that I have written it with efficiency in mind.) > >> >> I trust people are keeping an eye on their backups, and running >> verifications occasionally... >> >> /usr/local/bin/bbackupquery "compare -a" quit > Everything looks good. I'm glad to hear it! :-) Ben From boxbackup at fluffy.co.uk Fri Apr 9 09:33:32 2004 From: boxbackup at fluffy.co.uk (Bo Rasmussen) Date: Fri, 9 Apr 2004 10:33:32 +0200 Subject: [Box Backup] 0.04PLUS2 Message-ID: <002d01c41e0d$5569ed30$0a0aa8c0@CADesign.dk> Hey, Btw i'm running version 0.04plus2 on freebsd 4.8 and 4.9-stable, for 2 = days without any problems yet. - Bo -----Original Message----- From: "Ben Summers" Sent: 06-04-04 18:21:10 To: "boxbackup at fluffy.co.uk" Subject: [Box Backup] 0.04PLUS2 I've just uploaded another preview version. http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.04PLUS2.tgz This is basically a few bug fixes, but the significant addition is that=20 the server is a bit more tolerant of being aborted unexpectedly. I think this might help with Imran's problems. Ben _______________________________________________ boxbackup mailing list boxbackup at fluffy.co.uk http://lists.warhead.org.uk/mailman/listinfo/boxbackup From boxbackup at fluffy.co.uk Fri Apr 9 11:05:43 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 9 Apr 2004 11:05:43 +0100 Subject: [Box Backup] 0.04PLUS2 In-Reply-To: <002d01c41e0d$5569ed30$0a0aa8c0@CADesign.dk> References: <002d01c41e0d$5569ed30$0a0aa8c0@CADesign.dk> Message-ID: <766B8D26-8A0D-11D8-AA3E-000A95AFF7F8@fluffy.co.uk> On 9 Apr 2004, at 09:33, Bo Rasmussen wrote: > Hey, > > Btw i'm running version 0.04plus2 on freebsd 4.8 and 4.9-stable, for 2 > days without any problems yet. That's good news! I was wondering if anyone was using those ports. Have you verified the backups with /usr/local/bin/bbackupquery "compare -a" quit ? Thanks, Ben From boxbackup at fluffy.co.uk Fri Apr 9 17:06:11 2004 From: boxbackup at fluffy.co.uk (Bo Rasmussen) Date: Fri, 9 Apr 2004 18:06:11 +0200 Subject: [Box Backup] 0.04PLUS2 Message-ID: <003601c41e4c$910ecab0$0a0aa8c0@CADesign.dk> Hey, I was running 0.03 on the 4.8 server for about one month with no = problems, and just upgraded this one and installed it on a 4.9 server, = both as client against a 3.4-stable server without raid. I'm on holiday until monday, where i'll check the new setup. All this is running on i386 hardware, i'm gonna install the 0.04plus2 on = my hppa next weekend. But I noticed last time I ran the tests for 0.04plus2 on my 3.4 server, = that it failed two of the tests for bbstored, what could that be? Nothing else to report from denmark :) - Bo [The original message will be inserted by ters" Sent: 09-04-04 12:06:03 To: "boxbackup at fluffy.co.uk" Subject: Re: [Box Backup] 0.04PLUS2 On 9 Apr 2004, at 09:33, Bo Rasmussen wro On 9 Apr 2004, at 09:33, Bo Rasmussen wrote: > Hey, > > Btw i'm running version 0.04plus2 on freebsd 4.8 and 4.9-stable, for 2 = > days without any problems yet. That's good news! I was wondering if anyone was using those ports. Have you verified the backups with /usr/local/bin/bbackupquery "compare -a" quit ? Thanks, Ben _______________________________________________ boxbackup mailing list boxbackup at fluffy.co.uk http://lists.warhead.org.uk/mailman/listinfo/boxbackup From boxbackup at fluffy.co.uk Fri Apr 9 18:53:19 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 9 Apr 2004 18:53:19 +0100 Subject: [Box Backup] 0.04PLUS2 In-Reply-To: <003601c41e4c$910ecab0$0a0aa8c0@CADesign.dk> References: <003601c41e4c$910ecab0$0a0aa8c0@CADesign.dk> Message-ID: On 9 Apr 2004, at 17:06, Bo Rasmussen wrote: > Hey, > > I was running 0.03 on the 4.8 server for about one month with no > problems, I think the bugs were most noticeable when funny things happened, for example, file servers with clients with badly synchronized clocks, and random bad things happening. > and just upgraded this one and installed it on a 4.9 server, both as > client against a 3.4-stable server without raid. > > I'm on holiday until monday, where i'll check the new setup. > > All this is running on i386 hardware, i'm gonna install the 0.04plus2 > on my hppa next weekend. Cool! > > But I noticed last time I ran the tests for 0.04plus2 on my 3.4 > server, that it failed two of the tests for bbstored, what could that > be? Did you run the test as root? It tests to see what happens if a file is unreadable because of permissions. root is not troubled by permissions, so can read it anyway. And so the test fails. I should probably spit out a warning, and not do the test when root. > > Nothing else to report from denmark :) I'm glad to hear it! I'll probably do a new release tomorrow. Ben From boxbackup at fluffy.co.uk Sat Apr 10 12:31:43 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Sat, 10 Apr 2004 12:31:43 +0100 Subject: [Box Backup] 0.05 released Message-ID: Go get it! I've only added an extra config option over v0.04PLUS2, though, so not a huge amount of point in upgrading unless you're on 0.04PLUS1 or earlier. Ben From boxbackup at fluffy.co.uk Mon Apr 12 03:19:23 2004 From: boxbackup at fluffy.co.uk (Imran) Date: Sun, 11 Apr 2004 21:19:23 -0500 Subject: [Box Backup] exception BackupStore BadDirectoryFormat Message-ID: <20040412015423.M77998@naweb.com> Getting this in my server logs: Apr 10 22:13:43 backup bbstored[16937]: Certificate CN: BLAH-123456 Apr 10 22:13:43 backup bbstored[16937]: Login: Client ID 00123456, Read/Write Apr 10 22:15:01 backup bbstored/hk[16603]: Starting housekeeping Apr 10 22:15:19 backup bbstored[16937]: in server child, exception BackupStore BadDirectoryFormat (4/6) -- terminating child and corresponding client entry(looks like the client is 9 seconds behind): Apr 10 22:13:35 blah bbackupd[27329]: Connection made, login successful Apr 10 22:15:11 blah bbackupd[27329]: Exception caught (7/34), reset state and waiting to retry... Apr 10 22:16:53 blah bbackupd[27329]: File statistics: total file size uploaded 22948331, bytes already on server 0, encoded size 3837036 As in the log, looks like the client sent stuff to the server, so maybe its a certain directory thats having problem. Also I noticed that the bbackupd locked up or whateer. I was testing to see if i could still restore a randomo directory. so i fired up bbackupquery. tht worked. and exited d. and then i ran it again to look for a differnet direcctory. i got distracted and bbackupquery was at the prompt for about two hours and it timed out, of course. anyway I went to eh client and exited. noticed in the sys logs that there were no bbackupd messages for past hour. (usually there is the connection messages). also bbackupquery should keep track of timeouts. if its gonna timmeout, it should just timeout itself. or server should know if its a backupquery timeout, in that case print a more graceful message in the srever syslogs thanks for listening/reading... Imran From boxbackup at fluffy.co.uk Mon Apr 12 11:38:56 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 12 Apr 2004 11:38:56 +0100 Subject: [Box Backup] exception BackupStore BadDirectoryFormat In-Reply-To: <20040412015423.M77998@naweb.com> References: <20040412015423.M77998@naweb.com> Message-ID: <997AC228-8C6D-11D8-AE8C-000A95AFF7F8@fluffy.co.uk> On 12 Apr 2004, at 03:19, Imran wrote: > Getting this in my server logs: > > Apr 10 22:13:43 backup bbstored[16937]: Certificate CN: BLAH-123456 > Apr 10 22:13:43 backup bbstored[16937]: Login: Client ID 00123456, > Read/Write > Apr 10 22:15:01 backup bbstored/hk[16603]: Starting housekeeping > Apr 10 22:15:19 backup bbstored[16937]: in server child, exception > BackupStore > BadDirectoryFormat (4/6) -- terminating child > > > and corresponding client entry(looks like the client is 9 seconds > behind): > > Apr 10 22:13:35 blah bbackupd[27329]: Connection made, login successful > Apr 10 22:15:11 blah bbackupd[27329]: Exception caught (7/34), reset > state and > waiting to retry... > Apr 10 22:16:53 blah bbackupd[27329]: File statistics: total file size > uploaded 22948331, bytes already on server 0, encoded size 3837036 > > > As in the log, looks like the client sent stuff to the server, so > maybe its a > certain directory thats having problem. Sounds like it. I'm still not clear what happened in the run up to this. Am I right in thinking you... * Wait until a client is backing up to a server * Kill the server bbstored process (letting the sub-process which is handling the client remain) * Delete all the data from the server * Restart the bbstored process. If this is the case then I need to 1) Finish the server side recovery tools, and release them 2) Adjust the behaviour of the servers to signals, so sub-processes end too. and you need to wipe your entire server configuration (keeping a copy of all the *.pem certificate files) and then start again. Unless you start from an absolutely clean setup, I'm not sure there's much point in me looking into this too deeply. I can see exactly how this state could come about if I'm right in how you got here. > > Also I noticed that the bbackupd locked up or whateer. I was testing > to see > if i could still restore a randomo directory. so i fired up > bbackupquery. > tht worked. and exited d. and then i ran it again to look for a > differnet > direcctory. i got distracted and bbackupquery was at the prompt for > about two > hours and it timed out, of course. anyway I went to eh client and > exited. > noticed in the sys logs that there were no bbackupd messages for past > hour. > (usually there is the connection messages). In an error state, there will be connections every 100 seconds or so. In normal operation with the default configuration, you would expect one every hour, unless no files had changed, when it wouldn't connect at all. > > also bbackupquery should keep track of timeouts. if its gonna > timmeout, it > should just timeout itself. or server should know if its a backupquery > timeout, in that case print a more graceful message in the srever > syslogs Maybe. :-) There's more important stuff to do at the moment. Ben From boxbackup at fluffy.co.uk Mon Apr 12 12:53:07 2004 From: boxbackup at fluffy.co.uk (Imran) Date: Mon, 12 Apr 2004 06:53:07 -0500 Subject: [Box Backup] exception BackupStore BadDirectoryFormat In-Reply-To: <997AC228-8C6D-11D8-AE8C-000A95AFF7F8@fluffy.co.uk> References: <20040412015423.M77998@naweb.com> <997AC228-8C6D-11D8-AE8C-000A95AFF7F8@fluffy.co.uk> Message-ID: <20040412113713.M78304@naweb.com> On Mon, 12 Apr 2004 11:38:56 +0100, Ben Summers wrote > > As in the log, looks like the client sent stuff to the server, so > > maybe its a > > certain directory thats having problem. > > Sounds like it. I'm still not clear what happened in the run up to > this. Am I right in thinking you... > > * Wait until a client is backing up to a server > > * Kill the server bbstored process (letting the sub-process which is > handling the client remain) > > * Delete all the data from the server > > * Restart the bbstored process. > > If this is the case then I need to > > 1) Finish the server side recovery tools, and release them > > 2) Adjust the behaviour of the servers to signals, so sub-processes > end too. > > and you need to wipe your entire server configuration (keeping a > copy of all the *.pem certificate files) and then start again. > > Unless you start from an absolutely clean setup, I'm not sure > there's much point in me looking into this too deeply. I can see > exactly how this state could come about if I'm right in how you got here. No, I didn't clean anything. i had four clients running, going to one server. one of the client had the error I described. I didn't touch the processes. and then i noticed the process behaviour after the bbackupquery timeout. I think it was related to the timeout because that error had repeated several times, and only thing different was that I ran bbackupquery, and i had a timeout error and no other entry for an hour after that. And thats when i tried to kill the bbackupd and it wouldn't die. > > also bbackupquery should keep track of timeouts. if its gonna > > timmeout, it > > should just timeout itself. or server should know if its a backupquery > > timeout, in that case print a more graceful message in the srever > > syslogs > > Maybe. :-) There's more important stuff to do at the moment. I forgot to note severity. in this case, the graceful error thing was not that important. just wanted it on the table for eventually. From boxbackup at fluffy.co.uk Wed Apr 14 16:12:33 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 14 Apr 2004 16:12:33 +0100 Subject: [Box Backup] exception BackupStore BadDirectoryFormat In-Reply-To: <20040412113713.M78304@naweb.com> References: <20040412015423.M77998@naweb.com> <997AC228-8C6D-11D8-AE8C-000A95AFF7F8@fluffy.co.uk> <20040412113713.M78304@naweb.com> Message-ID: <278A3E5E-8E26-11D8-A099-000A95AFF7F8@fluffy.co.uk> On 12 Apr 2004, at 12:53, Imran wrote: > On Mon, 12 Apr 2004 11:38:56 +0100, Ben Summers wrote >>> As in the log, looks like the client sent stuff to the server, so >>> maybe its a >>> certain directory thats having problem. >> >> Sounds like it. I'm still not clear what happened in the run up to >> this. Am I right in thinking you... >> >> * Wait until a client is backing up to a server >> >> * Kill the server bbstored process (letting the sub-process which is >> handling the client remain) >> >> * Delete all the data from the server >> >> * Restart the bbstored process. >> >> If this is the case then I need to >> >> 1) Finish the server side recovery tools, and release them >> >> 2) Adjust the behaviour of the servers to signals, so sub-processes >> end too. >> >> and you need to wipe your entire server configuration (keeping a >> copy of all the *.pem certificate files) and then start again. >> >> Unless you start from an absolutely clean setup, I'm not sure >> there's much point in me looking into this too deeply. I can see >> exactly how this state could come about if I'm right in how you got >> here. > > No, I didn't clean anything. OK, then what you should do, at the very minimum, is * Edit the /etc/box/bbstored/accounts.txt file, and remove the reference to the dodgy account. * Delete the /backup/ directories, from all the raid directories. * Run bbstoreaccounts to recreate the account * Don't randomly delete files again. The backup client will be perfectly happy, and just upload everything again. You may see one error in the logs when it realises what's happened. Recovery tools will appear soon! > i had four clients running, going to one server. > one of the client had the error I described. I didn't touch the > processes. > and then i noticed the process behaviour after the bbackupquery > timeout. I > think it was related to the timeout because that error had repeated > several > times, and only thing different was that I ran bbackupquery, and i had > a > timeout error and no other entry for an hour after that. And thats > when i > tried to kill the bbackupd and it wouldn't die. It will only die when it has finished with the server. It will remember that it should, and die eventually. But it should die earlier though. Ben From boxbackup at fluffy.co.uk Thu Apr 15 00:57:22 2004 From: boxbackup at fluffy.co.uk (Imran) Date: Wed, 14 Apr 2004 18:57:22 -0500 Subject: [Box Backup] exception BackupStore BadDirectoryFormat In-Reply-To: <278A3E5E-8E26-11D8-A099-000A95AFF7F8@fluffy.co.uk> References: <20040412015423.M77998@naweb.com> <997AC228-8C6D-11D8-AE8C-000A95AFF7F8@fluffy.co.uk> <20040412113713.M78304@naweb.com> <278A3E5E-8E26-11D8-A099-000A95AFF7F8@fluffy.co.uk> Message-ID: <20040414234144.M93452@naweb.com> On Wed, 14 Apr 2004 16:12:33 +0100, Ben Summers wrote > > No, I didn't clean anything. > > OK, then what you should do, at the very minimum, is > > .... > > * Don't randomly delete files again. I think I may have been not clear. The directory error happened on its own. I mean, i installed the accounts, and started up 4 clients. checked on the server a day later, and everything looked like it had been backed up. That was when I noticed the errors (once per each time the backup cycle happens). Server and client were running. I was able to query and do ls of the stuff. even tried to restore a random directory and that worked. So it wasn't that I was doing anything weird to it. Anyway, after I did the tests, that was when I ran into a different problem. which was that I let the bbackupquery timeout, etc. But it turns out thats not why the client froze up. It was because I was doing a 'strace -p' on the bbackupd process, and that seemed to have frozen the process after I did a ctrl-c to get out of strace. It doesn't do that always. Only sometimes. I was doing an strace again to figure out what dir the bbackupd was trying to read when it got the error. but I accidentally triggered the hanging. to unhang it, i simply ran the strace again, which gave me some terminate type of error and kicked out of the strace. and then I noticed that the bbackupd started to write stuff to the logs. I ran strace again, ctrl-c, ran strace again, etc., for 3 or 4 more times. didn't notice that freezing again. I'm not an expert with strace so maybe strace allows you to pass signals or keyboard events (or atleast ctrl-c or pause or something) to the process its viewing? Anyway, maybe it was because i closed strace when the bbackupd gave the error and maybe was waiting for something. (because it was around 3 to 10 seconds after I received the BadDirectoryFormat, that I pressed ctrl-c in strace...both times that I produced that error. I'll try again, but for now I'm going to clean out that account and try the backup again. cheers, Imran From boxbackup at fluffy.co.uk Thu Apr 15 07:04:59 2004 From: boxbackup at fluffy.co.uk (Paul Arch) Date: Thu, 15 Apr 2004 14:04:59 +0800 Subject: [Box Backup] Cygwin build Message-ID: <006101c422af$95cabed0$0401050a@cel2000> Hi, today I have been attempting to build box backup under cygwin, with **some** success - getting the thing to build has been a little tricky, but I am getting close. So below are a few notes, and some questions :) Before I start, I must admit I have never programmed in c++ before, so I am in a little over my head when trying to debug some things ;) I have pretty much set the CYGWIN build environment/platform the same as if built under LINUX 1. Obviously there are no make options for CYGWIN environment, so I changed makebuildenv scripts the same as if the ENV was linux 2. Need to define the following : #define PLATFORM_LINUX #define PLATFORM_BERKLEY_DB_NOT_SUPPORTED #define PLATFORM_KQUEUE_NOT_SUPPORTED I dont know where to set this up globally, could someone please advise the right way ? 3. There were some 'missing' or extra files that needed to be included along the way: #include #include #include #include #include But this might of been something to do with me not using the 'makebuildenv' properly ? I dont know, I am no expert - I just figured out they needed to be included somewhere for something to work ;) 4. // This doesnt work : switch(::poll(&p, 1, (Timeout == IOStream::TimeOutInfinite)?INFTIM:Timeout)) Complained about INFTIM unknown. So I did this (dont know what I am doing here ;) ) switch(::poll(&p, 1, (Timeout == IOStream::TimeOutInfinite))) which allowed compile to continue 5. ./lib/common/BoxTime.cpp : BoxTime.cpp: In function `box_time_t GetCurrentBoxTime()': BoxTime.cpp:66: error: call of overloaded `SecondsToBoxTime(u_int32_t)' is ambiguous BoxTime.h:66: error: candidates are: box_time_t SecondsToBoxTime(long unsigned int) BoxTime.h:70: error: box_time_t SecondsToBoxTime(long long unsigned int) make[1]: *** [../../debug/lib/common/BoxTime.o] Error 1 make[1]: Leaving directory `/boxbackup-0.05/lib/common' make: *** [dep_modules] Error 2 changed SecondsToBoxTime to return(1) - dont know whats going on here ? which allowed compile to continue After all of the above (hopefully thats all I did ;) , I got from : /boxbackup-0.05/debug/bin/bbackupctl $ make [blah blah blah ] $ cd ../../debug/bin/bbackupctl /boxbackup-0.05/debug/bin/bbackupctl $ ls bbackupctl.exe bbackupctl.o /boxbackup-0.05/debug/bin/bbackupctl $ bbackupctl.exe Usage: bbackupctl [-q] [-c config_file] Commands are: sync -- start a syncronisation run now reload -- reload daemon configuration terminate -- terminate daemon now wait-for-sync -- wait until the next sync starts, then exit /boxbackup-0.05/debug/bin/bbackupctl $ Sooo.... I will try building everything else, and then I will have a play ! If people could offer advice where I can put these changes, and how to generate diffs, I would be more than happy to provide :) cheers Paul Arch Software Engineer S.D.M. Group Pty. Ltd. http://www.sdmgroup.com.au From boxbackup at fluffy.co.uk Thu Apr 15 09:47:17 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 15 Apr 2004 09:47:17 +0100 Subject: [Box Backup] Cygwin build In-Reply-To: <006101c422af$95cabed0$0401050a@cel2000> References: <006101c422af$95cabed0$0401050a@cel2000> Message-ID: <8016E54C-8EB9-11D8-830D-000A95AFF7F8@fluffy.co.uk> On 15 Apr 2004, at 07:04, Paul Arch wrote: > Hi, > > today I have been attempting to build box backup under cygwin, with > **some** > success - getting the thing to build has been a little tricky, but I am > getting close. So below are a few notes, and some questions :) > Before I start, I must admit I have never programmed in c++ before, so > I am > in a little over my head when trying to debug some things ;) Thanks for attempting this, it'd be nice to have it running on Cygwin. > > I have pretty much set the CYGWIN build environment/platform the same > as if > built under LINUX > > 1. Obviously there are no make options for CYGWIN environment, so I > changed > makebuildenv scripts the same as if the ENV was linux In BoxPlatform.pm, make sure everything is set as for Linux. Make sure $platform_define is set to PLATFORM_CYGWIN. > 2. Need to define the following : > #define PLATFORM_LINUX > #define PLATFORM_BERKLEY_DB_NOT_SUPPORTED > #define PLATFORM_KQUEUE_NOT_SUPPORTED > I dont know where to set this up globally, could someone please > advise the > right way ? In lib/common/BoxPlatform.h, add a section like this: #ifdef PLATFORM_CYGWIN #define PLATFORM_LINUX #define PLATFORM_BERKLEY_DB_NOT_SUPPORTED #define PLATFORM_KQUEUE_NOT_SUPPORTED #define INFTIM -1 #endif but put it *above* the PLATFORM_LINUX section (as it defines PLATFORM_LINUX, so needs to get the contents of that section too.) > 3. There were some 'missing' or extra files that needed to be included > along the way: > #include > #include > #include > #include > #include > > > But this might of been something to do with me not using the > 'makebuildenv' properly ? I dont know, I am no expert - I just > figured out > they needed to be included > somewhere for something to work ;) It's probably that Cygwin's include files include different include files than other platforms. It'll probably be easiest to put these #includes within the PLATFORM_CYGWIN BoxPlatform.h section. > 4. > // This doesnt work : > switch(::poll(&p, 1, (Timeout == > IOStream::TimeOutInfinite)?INFTIM:Timeout)) > Complained about INFTIM unknown. > > So I did this (dont know what I am doing here ;) ) > switch(::poll(&p, 1, (Timeout == IOStream::TimeOutInfinite))) > which allowed compile to continue That won't work, as I'm sure you'll have guessed. INFTIM needs to be -1, I added it to the BoxPlatform.h section above. > > 5. > ./lib/common/BoxTime.cpp : > BoxTime.cpp: In function `box_time_t GetCurrentBoxTime()': > BoxTime.cpp:66: error: call of overloaded > `SecondsToBoxTime(u_int32_t)' is > ambiguous > BoxTime.h:66: error: candidates are: box_time_t SecondsToBoxTime(long > unsigned > int) > BoxTime.h:70: error: box_time_t SecondsToBoxTime(long > long > unsigned int) > make[1]: *** [../../debug/lib/common/BoxTime.o] Error 1 > make[1]: Leaving directory `/boxbackup-0.05/lib/common' > make: *** [dep_modules] Error 2 > > changed SecondsToBoxTime to return(1) - dont know whats going on here ? > which allowed compile to continue That's possibly me making a tiny mistake. Try changing the function to read box_time_t GetCurrentBoxTime() { ASSERT(sizeof(uint32_t) == sizeof(time_t)); return SecondsToBoxTime((uint32_t)time(0)); } but it does worry me that the u_int32_t definition isn't quite right on Cygwin. But it could just be that the compiler is being totally paranoid for some reason. See what happens. > > > After all of the above (hopefully thats all I did ;) , I got from : > /boxbackup-0.05/debug/bin/bbackupctl > $ make > > [blah blah blah ] > > $ cd ../../debug/bin/bbackupctl > > /boxbackup-0.05/debug/bin/bbackupctl > $ ls > bbackupctl.exe bbackupctl.o > > /boxbackup-0.05/debug/bin/bbackupctl > $ bbackupctl.exe > Usage: bbackupctl [-q] [-c config_file] > Commands are: > sync -- start a syncronisation run now > reload -- reload daemon configuration > terminate -- terminate daemon now > wait-for-sync -- wait until the next sync starts, then exit > > /boxbackup-0.05/debug/bin/bbackupctl > $ Looks promising! > > Sooo.... I will try building everything else, and then I will have a > play ! > If people could offer advice where I can put these changes, and how to > generate diffs, I would be more than happy to provide :) Hopefully you'll just need minor changes to the BoxPlatform.* files, and you can just send me a new version. Unless Cygwin emulates UNIX filesystem behaviour, you may have problems with some of the tests. You may need to set up a server on a UNIX machine, and point bbackupd at it to test. This isn't ideal. Ben From boxbackup at fluffy.co.uk Thu Apr 15 10:09:50 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 15 Apr 2004 10:09:50 +0100 Subject: [Box Backup] exception BackupStore BadDirectoryFormat In-Reply-To: <20040414234144.M93452@naweb.com> References: <20040412015423.M77998@naweb.com> <997AC228-8C6D-11D8-AE8C-000A95AFF7F8@fluffy.co.uk> <20040412113713.M78304@naweb.com> <278A3E5E-8E26-11D8-A099-000A95AFF7F8@fluffy.co.uk> <20040414234144.M93452@naweb.com> Message-ID: On 15 Apr 2004, at 00:57, Imran wrote: > On Wed, 14 Apr 2004 16:12:33 +0100, Ben Summers wrote >>> No, I didn't clean anything. >> >> OK, then what you should do, at the very minimum, is >> >> .... >> >> * Don't randomly delete files again. > > I think I may have been not clear. Sorry. I must have got confused. > The directory error happened on its own. > I mean, i installed the accounts, and started up 4 clients. checked > on the > server a day later, and everything looked like it had been backed up. > That > was when I noticed the errors (once per each time the backup cycle > happens). > Server and client were running. I was able to query and do ls of the > stuff. > even tried to restore a random directory and that worked. I have a vague suspicion about what might have happened. Which platform are you running on, and which version of Box Backup were you using? Could you have been accidently running two bbackupds on one machine at the same time? > > So it wasn't that I was doing anything weird to it. Anyway, after I > did the > tests, that was when I ran into a different problem. which was that I > let the > bbackupquery timeout, etc. But it turns out thats not why the client > froze > up. It was because I was doing a 'strace -p' on the bbackupd process, > and > that seemed to have frozen the process after I did a ctrl-c to get out > of > strace. It doesn't do that always. Only sometimes. I was doing an > strace > again to figure out what dir the bbackupd was trying to read when it > got the > error. but I accidentally triggered the hanging. to unhang it, i > simply ran > the strace again, which gave me some terminate type of error and > kicked out of > the strace. and then I noticed that the bbackupd started to write > stuff to > the logs. I ran strace again, ctrl-c, ran strace again, etc., for 3 > or 4 more > times. didn't notice that freezing again. I'm not an expert with > strace so > maybe strace allows you to pass signals or keyboard events (or atleast > ctrl-c > or pause or something) to the process its viewing? Anyway, maybe it > was > because i closed strace when the bbackupd gave the error and maybe was > waiting > for something. (because it was around 3 to 10 seconds after I > received the > BadDirectoryFormat, that I pressed ctrl-c in strace...both times that I > produced that error. I'll try again, but for now I'm going to clean > out that > account and try the backup again. That shouldn't be related. Clients should be able to do just about anything, and the server should be unaffected. The code is written to be basically transactional, and to recover cleanly where it's not completely transactional. Ben From boxbackup at fluffy.co.uk Fri Apr 16 02:34:47 2004 From: boxbackup at fluffy.co.uk (Paul Arch) Date: Fri, 16 Apr 2004 09:34:47 +0800 Subject: [Box Backup] Cygwin build References: <006101c422af$95cabed0$0401050a@cel2000> <8016E54C-8EB9-11D8-830D-000A95AFF7F8@fluffy.co.uk> Message-ID: <00e201c42353$00e79050$0401050a@cel2000> > In lib/common/BoxPlatform.h, add a section like this: > > #ifdef PLATFORM_CYGWIN > #define PLATFORM_LINUX > #define PLATFORM_BERKLEY_DB_NOT_SUPPORTED > #define PLATFORM_KQUEUE_NOT_SUPPORTED > #define INFTIM -1 > #endif > Done. > It's probably that Cygwin's include files include different include > files than other platforms. It'll probably be easiest to put these > #includes within the PLATFORM_CYGWIN BoxPlatform.h section. > Done > > That won't work, as I'm sure you'll have guessed. INFTIM needs to be > -1, I added it to the BoxPlatform.h section above. > Done I really like how all of this is structured. This is the first time I have ever 'ported' something (well kinda ;) ) > > That's possibly me making a tiny mistake. Try changing the function to > read > > box_time_t GetCurrentBoxTime() > { > ASSERT(sizeof(uint32_t) == sizeof(time_t)); > return SecondsToBoxTime((uint32_t)time(0)); > } > Yep that worked. But I also had another problem, I added this to BoxTime.h to resolve ( please advise ? ) typedef unsigned int int32; inline box_time_t SecondsToBoxTime(int32 Seconds) { return ((box_time_t)Seconds * MICRO_SEC_IN_SEC_LL); } > > Looks promising! > With the above changes, and some more includes etc, I have compiled successfully : bbackupctl bbackupd I am currently tracking down issues in: bbackupquery bbstored bbstoreaccounts I seem to be having this problem (which relates to LinuxWorkaround somehow ? ) RaidFileRead.cpp: In static member function `static bool RaidFileRead::ReadDirectoryContents(int, const std::string&, int, std::vector >&)': RaidFileRead.cpp:1597: error: 'struct dirent' has no member named 'd_type' RaidFileRead.cpp:1597: error: `DT_REG' undeclared (first use this function) RaidFileRead.cpp:1597: error: (Each undeclared identifier is reported only once for each function it appears in.) RaidFileRead.cpp:1625: error: 'struct dirent' has no member named 'd_type' RaidFileRead.cpp:1625: error: `DT_DIR' undeclared (first use this function) make[1]: *** [../../debug/lib/raidfile/RaidFileRead.o] Error 1 make[1]: Leaving directory `/boxbackup-0.05/lib/raidfile' make: *** [dep_modules] Error 2 Urgh ! Something to do with Cygwin and it determining difference between DIR and FILEs. Anyway, I added a definition for "d_type" ( ** WARNING - I DONT KNOW WHAT I AM DOING ;) )for my dirent structure in /usr/include/sys/dirent.h, since it didnt seem to exist, and LinuxWorkaround_FinishDirentStruct was attempting to set it. I do believe this is correct, I have read elsewhere you need to use 'stat' within cygwin for DIR/FILE type. struct dirent { long d_version; ino_t d_ino; long d_fd; unsigned long __ino32; char d_name[256]; int d_type; }; So, it looks like **EVERYTHING** seems to compile **OK**. I do get these warnings : warning: ISO C requires whitespace after the macro name warning: extra tokens at end of #ifdef directive I will now setup server on linux box (actually upgrade to v0.05 from 0.03PLUS2), and try running the backup client. If this works, I have achieved what I have set out to do. I am reluctant to try running the server, especially because of the DIR/FILE issue. Shall keep you posted ;) cheers Paul Arch Software Engineer S.D.M. Group Pty. Ltd. http://www.sdmgroup.com.au From boxbackup at fluffy.co.uk Fri Apr 16 05:31:56 2004 From: boxbackup at fluffy.co.uk (Paul Arch) Date: Fri, 16 Apr 2004 12:31:56 +0800 Subject: [Box Backup] Cygwin build References: <006101c422af$95cabed0$0401050a@cel2000> <8016E54C-8EB9-11D8-830D-000A95AFF7F8@fluffy.co.uk> <00e201c42353$00e79050$0401050a@cel2000> Message-ID: <013801c4236b$c07f2f50$0401050a@cel2000> > I seem to be having this problem (which relates to LinuxWorkaround somehow > ? ) > RaidFileRead.cpp: In static member function `static bool > RaidFileRead::ReadDirectoryContents(int, const std::string&, int, > std::vector >&)': > RaidFileRead.cpp:1597: error: 'struct dirent' has no member named 'd_type' > RaidFileRead.cpp:1597: error: `DT_REG' undeclared (first use this function) > RaidFileRead.cpp:1597: error: (Each undeclared identifier is reported only > once > > for each function it appears in.) > RaidFileRead.cpp:1625: error: 'struct dirent' has no member named 'd_type' > RaidFileRead.cpp:1625: error: `DT_DIR' undeclared (first use this function) > make[1]: *** [../../debug/lib/raidfile/RaidFileRead.o] Error 1 > make[1]: Leaving directory `/boxbackup-0.05/lib/raidfile' > make: *** [dep_modules] Error 2 > > > Urgh ! Something to do with Cygwin and it determining difference between DIR > and FILEs. > Anyway, I added a definition for "d_type" ( ** WARNING - I DONT KNOW WHAT I > AM DOING ;) )for my dirent structure in /usr/include/sys/dirent.h, since it > didnt seem to exist, and LinuxWorkaround_FinishDirentStruct was attempting > to set it. I do believe this is correct, I have read elsewhere you need to > use 'stat' within cygwin for DIR/FILE type. > struct dirent > { > long d_version; > ino_t d_ino; > long d_fd; > unsigned long __ino32; > char d_name[256]; > int d_type; > }; > OK, this is getting me stuck. I am doing well, I have setup a server on a linux box, and run commands like 'bbackupd' and 'bbackupctl' from within cygwin, and it 'seems' to be talking. But I think I need to resolve the above issue before I can proceed, since I am getting from the server: Apr 16 12:55:49 sdmgbackup bbstored[4280]: Certificate CN: BACKUP-2 Apr 16 12:55:49 sdmgbackup bbstored[4280]: Receive Version(0x1) Apr 16 12:55:49 sdmgbackup bbstored[4280]: Send Version(0x1) Apr 16 12:55:49 sdmgbackup bbstored[4280]: Receive Login(0x2,0x1) Apr 16 12:55:49 sdmgbackup bbstored[4280]: Login: Client ID 00000002, Read-only Apr 16 12:55:49 sdmgbackup bbstored[4280]: Send LoginConfirmed(0x0,0x2,0x7d000,0xbb800) Apr 16 12:55:51 sdmgbackup bbstored[4280]: Receive ListDirectory(0x100000000000000,0xffffffff,0xc,true) Apr 16 12:55:51 sdmgbackup bbstored[4280]: in server child, exception RaidFile RaidFileDoesntExist (2/11) -- terminating child I can only guess that this is where my problem is at. I will have a look closer, but I am in over my head ;) cheers Paul From boxbackup at fluffy.co.uk Fri Apr 16 09:14:58 2004 From: boxbackup at fluffy.co.uk (Paul Arch) Date: Fri, 16 Apr 2004 16:14:58 +0800 Subject: [Box Backup] Cygwin build References: <006101c422af$95cabed0$0401050a@cel2000> <8016E54C-8EB9-11D8-830D-000A95AFF7F8@fluffy.co.uk> <00e201c42353$00e79050$0401050a@cel2000> <013801c4236b$c07f2f50$0401050a@cel2000> Message-ID: <016001c4238a$e8cf1140$0401050a@cel2000> > > > I seem to be having this problem (which relates to LinuxWorkaround > somehow > > ? ) > > RaidFileRead.cpp: In static member function `static bool > > RaidFileRead::ReadDirectoryContents(int, const std::string&, int, > > std::vector >&)': > > RaidFileRead.cpp:1597: error: 'struct dirent' has no member named 'd_type' > > RaidFileRead.cpp:1597: error: `DT_REG' undeclared (first use this > function) > Apr 16 12:55:49 sdmgbackup bbstored[4280]: Send > LoginConfirmed(0x0,0x2,0x7d000,0xbb800) > Apr 16 12:55:51 sdmgbackup bbstored[4280]: Receive > ListDirectory(0x100000000000000,0xffffffff,0xc,true) >From my last post, I am now thinking it is to do with perhaps byte order? Note the ListDirectory Command from above from cygwin. When running from linux, the numbers are like 0x1, 0x2 etc. Any ideas ? I tried #PLATFORM_SWAP64_REQUIRED I found somewhere, but no luck :) cheers From boxbackup at fluffy.co.uk Fri Apr 16 09:27:22 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 16 Apr 2004 09:27:22 +0100 Subject: [Box Backup] Cygwin build In-Reply-To: <00e201c42353$00e79050$0401050a@cel2000> References: <006101c422af$95cabed0$0401050a@cel2000> <8016E54C-8EB9-11D8-830D-000A95AFF7F8@fluffy.co.uk> <00e201c42353$00e79050$0401050a@cel2000> Message-ID: On 16 Apr 2004, at 02:34, Paul Arch wrote: >> >> That's possibly me making a tiny mistake. Try changing the function to >> read >> >> box_time_t GetCurrentBoxTime() >> { >> ASSERT(sizeof(uint32_t) == sizeof(time_t)); >> return SecondsToBoxTime((uint32_t)time(0)); >> } >> > Yep that worked. But I also had another problem, I added this to > BoxTime.h > to resolve ( please advise ? ) > typedef unsigned int int32; > inline box_time_t SecondsToBoxTime(int32 Seconds) > { > return ((box_time_t)Seconds * MICRO_SEC_IN_SEC_LL); > } What problems did this resolve? If it's a compile error in another file, it's best done by casting, for example if this doesn't work: SecondsToBoxTime(value) do SecondsToBoxTime((uint32_t)value) > >> >> Looks promising! >> > > With the above changes, and some more includes etc, I have compiled > successfully : > bbackupctl > bbackupd > > I am currently tracking down issues in: > bbackupquery > bbstored > bbstoreaccounts > I seem to be having this problem (which relates to LinuxWorkaround > somehow > ? ) > RaidFileRead.cpp: In static member function `static bool > RaidFileRead::ReadDirectoryContents(int, const std::string&, int, > std::vector >&)': > RaidFileRead.cpp:1597: error: 'struct dirent' has no member named > 'd_type' > RaidFileRead.cpp:1597: error: `DT_REG' undeclared (first use this > function) > RaidFileRead.cpp:1597: error: (Each undeclared identifier is reported > only > once > > for each function it appears in.) > RaidFileRead.cpp:1625: error: 'struct dirent' has no member named > 'd_type' > RaidFileRead.cpp:1625: error: `DT_DIR' undeclared (first use this > function) > make[1]: *** [../../debug/lib/raidfile/RaidFileRead.o] Error 1 > make[1]: Leaving directory `/boxbackup-0.05/lib/raidfile' > make: *** [dep_modules] Error 2 > > > Urgh ! Something to do with Cygwin and it determining difference > between DIR > and FILEs. Linux has a "bug" where it doesn't conform to standards, and omits to actually fill in the dirent structure. The Linux types implementing Cygwyn obviously took this a step further, and omitted the field entirely. The LinuxWorkaround_FinishDirentStruct finishes the structure by doing a stat(), and adjusting this field accordingly. This obviously won't work if the field is there. > Anyway, I added a definition for "d_type" ( ** WARNING - I DONT KNOW > WHAT I > AM DOING ;) )for my dirent structure in /usr/include/sys/dirent.h, > since it > didnt seem to exist, and LinuxWorkaround_FinishDirentStruct was > attempting > to set it. You must never alter system headers. Programs compiled with them won't work. > I do believe this is correct, I have read elsewhere you need to > use 'stat' within cygwin for DIR/FILE type. > struct dirent > { > long d_version; > ino_t d_ino; > long d_fd; > unsigned long __ino32; > char d_name[256]; > int d_type; > }; You should actually just alter the code in bbackupquery to use stat to work out which is which. As a #define with PLATFORM_CYGWYN, of course, wouldn't want to spoil it for the other platforms. > > > So, it looks like **EVERYTHING** seems to compile **OK**. > I do get these warnings : > warning: ISO C requires whitespace after the macro name > warning: extra tokens at end of #ifdef directive Which line and file? > > I will now setup server on linux box (actually upgrade to v0.05 from > 0.03PLUS2), and try running the backup client. If this works, I have > achieved what I have > set out to do. I am reluctant to try running the server, especially > because > of the DIR/FILE issue. Shall keep you posted ;) In the other message, you said it didn't, but as you've identified from the logs, you're having problems with 64 bit network byte order swapping -- in that it isn't swapping the order, so the server gets spurious values to work with. Other swaps seem to work though. You'll need to make sure that the hton64 and ntoh64 defines do actually swap stuff. It looks like the __cpu_to_be64 and __be64_to_cpu functions don't do anything on Cygwin. Ben From boxbackup at fluffy.co.uk Fri Apr 16 10:14:16 2004 From: boxbackup at fluffy.co.uk (Paul Arch) Date: Fri, 16 Apr 2004 17:14:16 +0800 Subject: [Box Backup] Cygwin build References: <006101c422af$95cabed0$0401050a@cel2000> <8016E54C-8EB9-11D8-830D-000A95AFF7F8@fluffy.co.uk> <00e201c42353$00e79050$0401050a@cel2000> Message-ID: <01a001c42393$31943e20$0401050a@cel2000> > > SecondsToBoxTime(value) BackupQueries.cpp:964: error: call of overloaded `SecondsToBoxTime(unsigned int)' is ambiguous > > do > SecondsToBoxTime((uint32_t)value) > Done, did the trick for the above (bbackupctl), but there seems to be other instances of this occuring when compiling bbackupd. Is it worth casting to fix, or can it be done somewhere else ? > You must never alter system headers. Programs compiled with them won't > work. np, I have a cygwin installation esp. for getting boxbackup going, but thanks for advice :) > You should actually just alter the code in bbackupquery to use stat to > work out which is which. As a #define with PLATFORM_CYGWYN, of course, > wouldn't want to spoil it for the other platforms. > OK, will have to give this a go. >You'll > need to make sure that the hton64 and ntoh64 defines do actually swap > stuff. It looks like the __cpu_to_be64 and __be64_to_cpu functions > don't do anything on Cygwin. > ;) looking better now from Cygwin Apr 16 17:32:30 sdmgbackup bbstored[2230]: Send LoginConfirmed(0x0,0x2,0x7d000,0xbb800) Apr 16 17:32:31 sdmgbackup bbstored[2230]: Receive ListDirectory(0x1,0xffffffff,0xc,true) Apr 16 17:32:31 sdmgbackup bbstored[2230]: Send Success(0x1) Apr 16 17:32:31 sdmgbackup bbstored[2230]: Sending stream, size 40 Testrun a backup gives ( syslogd for cygwin goes to Event Log in windows ): following information is part of the event: bbackupd : PID 1520 : File statistics: total file size uploaded 18560926, bytes already on server 0, encoded size 4855465. **yay** Backups seem to be happening to linux server. But, I cannot use "compare" for verify, probably has something to do with FILE/DIR stat stuff as noted previously. cheers From boxbackup at fluffy.co.uk Fri Apr 16 10:34:59 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 16 Apr 2004 10:34:59 +0100 Subject: [Box Backup] Cygwin build In-Reply-To: <01a001c42393$31943e20$0401050a@cel2000> References: <006101c422af$95cabed0$0401050a@cel2000> <8016E54C-8EB9-11D8-830D-000A95AFF7F8@fluffy.co.uk> <00e201c42353$00e79050$0401050a@cel2000> <01a001c42393$31943e20$0401050a@cel2000> Message-ID: <5432F62A-8F89-11D8-9BDB-000A95AFF7F8@fluffy.co.uk> On 16 Apr 2004, at 10:14, Paul Arch wrote: >> >> SecondsToBoxTime(value) > BackupQueries.cpp:964: error: call of overloaded > `SecondsToBoxTime(unsigned > int)' is ambiguous >> >> do >> SecondsToBoxTime((uint32_t)value) >> > Done, did the trick for the above (bbackupctl), but there seems to be > other > instances of this occuring when compiling bbackupd. Is it worth > casting to > fix, or can it be done somewhere else ? I'm not sure why it's doing that. Maybe some odd compiler thing? Maybe add a signed variant too? inline box_time_t SecondsToBoxTime(int32_t Seconds) { return ((box_time_t)Seconds * MICRO_SEC_IN_SEC_LL); } ? > >> You must never alter system headers. Programs compiled with them won't >> work. > > np, I have a cygwin installation esp. for getting boxbackup going, but > thanks for advice :) There's just no point in modifying the system headers, it won't alter the libraries which use them. And if you add stuff to the end, you'll end up overwriting important stuff. > >> You should actually just alter the code in bbackupquery to use stat to >> work out which is which. As a #define with PLATFORM_CYGWYN, of course, >> wouldn't want to spoil it for the other platforms. >> > OK, will have to give this a go. > >> You'll >> need to make sure that the hton64 and ntoh64 defines do actually swap >> stuff. It looks like the __cpu_to_be64 and __be64_to_cpu functions >> don't do anything on Cygwin. >> > ;) looking better now from Cygwin > Apr 16 17:32:30 sdmgbackup bbstored[2230]: Send > LoginConfirmed(0x0,0x2,0x7d000,0xbb800) > Apr 16 17:32:31 sdmgbackup bbstored[2230]: Receive > ListDirectory(0x1,0xffffffff,0xc,true) > Apr 16 17:32:31 sdmgbackup bbstored[2230]: Send Success(0x1) > Apr 16 17:32:31 sdmgbackup bbstored[2230]: Sending stream, size 40 > > Testrun a backup gives ( syslogd for cygwin goes to Event Log in > windows ): > > following information is part of the event: bbackupd : PID 1520 : File > statistics: total file size uploaded 18560926, bytes already on server > 0, > encoded size 4855465. > > **yay** > > Backups seem to be happening to linux server. Yes, I would agree that this appears to work. > But, I cannot use "compare" > for verify, probably has something to do with FILE/DIR stat stuff as > noted > previously. Here's a snippet of code for BackupQueries::Compare which should work on Cygwin: std::string fn(rLocalDir); fn += '/'; fn += localDirEn->d_name; struct stat st; if(::lstat(fn.c_str(), &st) != 0) { THROW_EXCEPTION(CommonException, OSFileError) } // Entry -- file or dir? if(S_ISREG(st.st_mode) || S_ISLNK(st.st_mode)) { // File or symbolic link localFiles.insert(std::string(localDirEn->d_name)); } else if(S_ISDIR(st.st_mode)) { // Directory localDirs.insert(std::string(localDirEn->d_name)); } It should be obvious where it goes, search for "Entry -- file or dir". Ben From boxbackup at fluffy.co.uk Fri Apr 16 16:53:42 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Fri, 16 Apr 2004 10:53:42 -0500 (CDT) Subject: [Box Backup] windows client Message-ID: I just found out about this project and just subscribed to the list. I'm curious if anyone has taken a look at OpenVPN, which also uses OpenSSL, and which has clients for W32 among other platforms. Just thought that might help in making this very useful tool available to more folks. Josh From boxbackup at fluffy.co.uk Sat Apr 17 07:13:01 2004 From: boxbackup at fluffy.co.uk (Paul Arch) Date: Sat, 17 Apr 2004 14:13:01 +0800 Subject: [Box Backup] Cygwin build References: <006101c422af$95cabed0$0401050a@cel2000> <8016E54C-8EB9-11D8-830D-000A95AFF7F8@fluffy.co.uk> <00e201c42353$00e79050$0401050a@cel2000> <01a001c42393$31943e20$0401050a@cel2000> <5432F62A-8F89-11D8-9BDB-000A95AFF7F8@fluffy.co.uk> Message-ID: <021b01c42443$09b78f50$0401050a@cel2000> > Here's a snippet of code for BackupQueries::Compare which should work > on Cygwin: Bingo, did the trick : >From command "compare -aq" : Local file '/cygdrive/c/Program Files/Intuit/QuickBooks/QBWIN.LOG' has different contents to store file '/accounts/QBWIN.LOG'. [blah blah] [ 1 (of 6) differences probably due to file modifications after the last upload] Differences: 6 (0 dirs excluded, 0 files excluded) Logging off... Session finished. It did take a while (maybe 10 minutes on 187mb dataset, I wasnt exactly counting). But I also run a compare -aq on my linux dataset (5.5gig local bbackupd and bbstored on XP2100 512mb ram), and it is still going so I am happy. I am close to say boxbackup works under CYGWIN_NT_5.0. I shall do some cleaning up and send you my modified files. Otherwise, do you think it is worthwhile attempted to run the testsuite ? cheers From boxbackup at fluffy.co.uk Sat Apr 17 07:16:43 2004 From: boxbackup at fluffy.co.uk (Paul Arch) Date: Sat, 17 Apr 2004 14:16:43 +0800 Subject: [Box Backup] windows client References: Message-ID: <022101c42443$8deb18a0$0401050a@cel2000> > I just found out about this project and just subscribed to the list. I'm curious if anyone has taken a look at OpenVPN, which also uses OpenSSL, and which has clients for W32 among other platforms. Just thought that might help in making this very useful tool available to more folks. > I have been working on getting boxbackup operating under a cygwin environment under windows. You "could" simply use the cygwin generated boxbackup 'exe' files with appropriate cygwin dll files if you wanted to. I have done this before with cygwins rsync.exe for some windows machines I have around, but of course it is somewhat slower. regards Paul Arch Software Engineer S.D.M. Group Pty. Ltd. http://www.sdmgroup.com.au From boxbackup at fluffy.co.uk Sat Apr 17 12:37:05 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Sat, 17 Apr 2004 12:37:05 +0100 Subject: [Box Backup] Cygwin build In-Reply-To: <021b01c42443$09b78f50$0401050a@cel2000> References: <006101c422af$95cabed0$0401050a@cel2000> <8016E54C-8EB9-11D8-830D-000A95AFF7F8@fluffy.co.uk> <00e201c42353$00e79050$0401050a@cel2000> <01a001c42393$31943e20$0401050a@cel2000> <5432F62A-8F89-11D8-9BDB-000A95AFF7F8@fluffy.co.uk> <021b01c42443$09b78f50$0401050a@cel2000> Message-ID: <8D569070-9063-11D8-8A18-000A95AFF7F8@fluffy.co.uk> On 17 Apr 2004, at 07:13, Paul Arch wrote: > > >> Here's a snippet of code for BackupQueries::Compare which should work >> on Cygwin: > > > > Bingo, did the trick : > >> From command "compare -aq" : > Local file '/cygdrive/c/Program Files/Intuit/QuickBooks/QBWIN.LOG' has > different contents to store file '/accounts/QBWIN.LOG'. > [blah blah] > [ 1 (of 6) differences probably due to file modifications after the > last > upload] > Differences: 6 (0 dirs excluded, 0 files excluded) > Logging off... > Session finished. That looks quite promising. > > It did take a while (maybe 10 minutes on 187mb dataset, I wasnt exactly > counting). But I also run a compare -aq on my linux dataset (5.5gig > local > bbackupd and bbstored on XP2100 512mb ram), and it is still going so I > am > happy. The protocol is not the most efficient for streaming data back and from the server, but then, it doesn't need to be. And in a way, this is also a good thing, because it means that it won't get in the way of other applications too much when they need to use the network. > > I am close to say boxbackup works under CYGWIN_NT_5.0. I shall do some > cleaning up and send you my modified files. Otherwise, do you think > it is > worthwhile attempted to run the testsuite ? I'm not sure. You'd have to get bbstored working to run it properly, and I doubt that's worth it. The filesystem behaviour of cygwin will probably mean that bbstored appears to work but is unreliable, so I'd prefer it wasn't ported at all -- I'll adjust the build scripts to omit it on Cygwin. It would be worth checking that the basics work, I'd like to know that test/backupdiff, common, compress and crypto all worked. I look forward to incorporating the changes. Ben From boxbackup at fluffy.co.uk Sun Apr 18 14:48:50 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Sun, 18 Apr 2004 09:48:50 -0400 (EDT) Subject: [Box Backup] Handling of changed backup configuration In-Reply-To: <7258D520-9121-11D8-8281-000A95AFF7F8@fluffy.co.uk> References: <7258D520-9121-11D8-8281-000A95AFF7F8@fluffy.co.uk> Message-ID: [ I initially botched my configuration because I didn't really understand how to exclude directories. I have my Oracle files under my /home directory in /home/oracle. I had initially thought you had to create a new file path to make it exclude things, so I created an Oracle file path and then excluded /home/oracle, but did not include anything. The backup software correctly started to backup /home/oracle. I fixed the config problem halfway, and restarted the backup daemon (I'm new and didn't know how to make it refresh). I asked Ben if the unwanted Oracle stuff would go away on it's own, or what would happen, since I didn't want it wasting space on his server. ] Okay, the oracle deletion worked flawlessly, it did exactly what you said.... Good Job! The stuff is deleted, but still there, so I assume it will go away on it's own over time. However the Oracle top level item that I created incorrectly is still there. Should it go away on it's own as well, it's completely gone from my configuration file now? Do you think people need a way to purposely delete backups? Say they backed up something they really, really, don't want a copy of? You seem to have thought about this carefully what's your opinion? (Something like the Sarbanes-Oxely stuff in the USA). I can see that something like that could really cause problems, but may also be useful if people are managing their own backups (for instance email retention laws where you probably want it to disappear on the crack of midnight the first day you can delete it). I also get the impression you need to be able to hold certain items back from deletion sort of indefinitely say if the court has ordered you to keep it for some reason. Do you think a config item for max retention time might be useful? Along with a config item to prevent something from getting deleted under any circumstances? Is there some way to list how much space you are allowed to have on the drive, along with how much you are using from the backup query client? Rick On Sun, 18 Apr 2004, Ben Summers wrote: > > Excluded items will be scheduled for deletion from the server. It won't > happen immediately, though, but they won't be restored unless you > specifically ask for it. Maybe you'd like to test this -- exclude those > files, wait for a sync to finish, then use bbackupquery to check that > the file has been marked as deleted. > > In general, I have tried to code the system to work as you'd expect. > > Ben From boxbackup at fluffy.co.uk Sun Apr 18 20:46:01 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Sun, 18 Apr 2004 20:46:01 +0100 Subject: [Box Backup] Handling of changed backup configuration In-Reply-To: References: <7258D520-9121-11D8-8281-000A95AFF7F8@fluffy.co.uk> Message-ID: <053BD603-9171-11D8-8281-000A95AFF7F8@fluffy.co.uk> On 18 Apr 2004, at 14:48, rprice at freeshell.org wrote: > [ > > I initially botched my configuration because I didn't really understand > how to exclude directories. > > I have my Oracle files under my /home directory in /home/oracle. I had > initially thought you had to create a new file path to make it exclude > things, so I created an Oracle file path and then excluded > /home/oracle, > but did not include anything. > > The backup software correctly started to backup /home/oracle. > > I fixed the config problem halfway, and restarted the backup daemon > (I'm > new and didn't know how to make it refresh). xargs kill -HUP < /var/run/bbackupd.pid -- but it may take a while to get round to reloading it, as it will wait until the sync is over. So you would have had to forcibly kill in in this case. Yes, I know, I'm going to change this. > > I asked Ben if the unwanted Oracle stuff would go away on it's own, or > what would happen, since I didn't want it wasting space on his server. > > ] > > Okay, the oracle deletion worked flawlessly, it did exactly what you > said.... Good Job! The stuff is deleted, but still there, so I assume > it > will go away on it's own over time. Yes. After you hit the soft limit on the account, the housekeeping process will start to remove it. > > However the Oracle top level item that I created incorrectly is still > there. Should it go away on it's own as well, it's completely gone from > my configuration file now? Now this, I must confess, is something I've yet to do. I need to remove removed locations from the store. I think I've been delaying because I feel I should wait a day or so to make sure that the adminstrator really wanted to do it, and that's a more complex thing to do. > > Do you think people need a way to purposely delete backups? Say they > backed up something they really, really, don't want a copy of? You > seem to > have thought about this carefully what's your opinion? (Something like > the > Sarbanes-Oxely stuff in the USA). > > I can see that something like that could really cause problems, but may > also be useful if people are managing their own backups (for instance > email retention laws where you probably want it to disappear on the > crack > of midnight the first day you can delete it). I also get the impression > you need to be able to hold certain items back from deletion sort of > indefinitely say if the court has ordered you to keep it for some > reason. I have been more interesting in preserving data than destroying it, so controlled deletion didn't really occur to me. But you're right, it is useful. There does need to be a way of retaining data more carefully. When I add the "mark" feature, you'll be able to specify how many marked files should exist, so you could mark weekly, and say you want to keep at least 6 weeks of marked files. > > Do you think a config item for max retention time might be useful? Sounds a sensible start. > Along > with a config item to prevent something from getting deleted under any > circumstances? I will try and come up with a sensible suggestion and post to the list for feedback. I think the suggestion above will be sufficient for keeping files -- only old and delete files get removed from the store, never files which correspond to files on the client disc. > > Is there some way to list how much space you are allowed to have on the > drive, along with how much you are using from the backup query client? I will add a command to bbackupquery -- I always use bbstoreaccounts on the server to find out such things, but this is not possible if you don't have that access to the server. Ben From boxbackup at fluffy.co.uk Mon Apr 19 19:47:04 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Mon, 19 Apr 2004 14:47:04 -0400 (EDT) Subject: [Box Backup] Re: boxbackup digest, Vol 1 #57 - 2 msgs In-Reply-To: <20040419110006.28498.18205.Mailman@love.warhead.org.uk> References: <20040419110006.28498.18205.Mailman@love.warhead.org.uk> Message-ID: > > However the Oracle top level item that I created incorrectly is still > > there. Should it go away on it's own as well, it's completely gone from > > my configuration file now? > > Now this, I must confess, is something I've yet to do. I need to remove > removed locations from the store. I think I've been delaying because I > feel I should wait a day or so to make sure that the adminstrator > really wanted to do it, and that's a more complex thing to do. Why don't you make it go a away when things hit the soft limit as with the other stuff? That way the admin has a while to discover their mistake... I suppose something in the client log the first time or two might be good as well. I run logcheck on my Debian machine, and so I see the various messages that BoxBackup sends out... > > > > > Do you think people need a way to purposely delete backups? Say they > > backed up something they really, really, don't want a copy of? You > > seem to > > have thought about this carefully what's your opinion? (Something like > > the > > Sarbanes-Oxely stuff in the USA). > > > > I can see that something like that could really cause problems, but may > > also be useful if people are managing their own backups (for instance > > email retention laws where you probably want it to disappear on the > > crack > > of midnight the first day you can delete it). I also get the impression > > you need to be able to hold certain items back from deletion sort of > > indefinitely say if the court has ordered you to keep it for some > > reason. > > I have been more interesting in preserving data than destroying it, so > controlled deletion didn't really occur to me. But you're right, it is > useful. > > There does need to be a way of retaining data more carefully. When I > add the "mark" feature, you'll be able to specify how many marked files > should exist, so you could mark weekly, and say you want to keep at > least 6 weeks of marked files. The Sarbanes-Oxely stuff though would require you to absolutely guarantee that something could not get deleted until it was untagged. > > > > > Do you think a config item for max retention time might be useful? > > Sounds a sensible start. And again for Sarbanes-Oxely it would have to get nuked immediately once it expired, it could not wait until you hit the soft limit. I suppose you have the wrinkle of what to do with the file if it still exists on the client, not backing it up could be counter-intuitive, but I guess you could only keep one copy or something in that case. Actually I think that's the wrong way of looking at it, see below: Maybe if the location had a min-retention-time and a max-retention-time, that way you would know the file (older copies of the current file on client disk as well) would be kept for at least the minimum time period, then all the older copies would for sure get nuked at the max-retention-time, the current backup of course would be retained for min-retention-time. Kind of like a conveyor belt with stuff falling off the end. The min-retention-time would supersede any garbage collection, you would know for sure that file would exist for that time period, if you run out of space, then you deal with it... Again the max-retention-time would nuke the backups immediately after they expired, it wouldn't wait for garbage collection. It might be interesting to have an API where you could programatically flag files to be unremovable, but then, maybe it makes more sense to have a regexp in the config file where you could specify files or directories as being undeletable. If someone needed to modify things, they could modify the config file from whatever program was managing the backup stuff. Finally, to comply with something like Sarbanes-Oxely, that might not be good enough, you may need the ability to mark a particular backup(s) of a file as being undeletable, but ensure the rest follow the current rules. After all, you want to nuke the stuff that is covered by that as quickly as possible so it can't be requested by your opponent. However that would probably complicate things, so going with the pure conveyor belt method would probably be better as a start. Finally, we would assume that the client has something on the client machine that would handle deleting files that had expired, so if the file needed to disappear, they would handle that. At that point the file's backups would follow the retention rules. Basically limit our scope to what we can reasonably control and worry about. The only other thing that comes to mind is perhaps some sort of optional logging when expired files and so on are deleted, that way someone/(some program) could be sure that the retention rules were being followed. > > Is there some way to list how much space you are allowed to have on the > > drive, along with how much you are using from the backup query client? > > I will add a command to bbackupquery -- I always use bbstoreaccounts on > the server to find out such things, but this is not possible if you > don't have that access to the server. I don't think you have really said this, but I see great potential in this software being used as a *utility* where someone rents out a server as a bit-bucket to others for a monthly fee, having proper retention rule hooks would allow it to be used as such in more situations. I don't personally have any real intention of using the software that way, but I can see the retention rules being useful to my friends who have businesses that I might back up onto my server. Rick From boxbackup at fluffy.co.uk Mon Apr 19 22:00:15 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 19 Apr 2004 22:00:15 +0100 Subject: [Box Backup] Re: boxbackup digest, Vol 1 #57 - 2 msgs In-Reply-To: References: <20040419110006.28498.18205.Mailman@love.warhead.org.uk> Message-ID: <8E88062E-9244-11D8-9814-000A95AFF7F8@fluffy.co.uk> On 19 Apr 2004, at 19:47, rprice at freeshell.org wrote: >>> However the Oracle top level item that I created incorrectly is still >>> there. Should it go away on it's own as well, it's completely gone >>> from >>> my configuration file now? >> >> Now this, I must confess, is something I've yet to do. I need to >> remove >> removed locations from the store. I think I've been delaying because I >> feel I should wait a day or so to make sure that the adminstrator >> really wanted to do it, and that's a more complex thing to do. > > Why don't you make it go a away when things hit the soft limit as with > the > other stuff? Marking it as deleted loses information, and would mean that if it reappeared, it'd have to be uploaded again. After the delay, it would be marked as deleted, and eventually be removed as the soft limit was exceeded. > That way the admin has a while to discover their mistake... I > suppose something in the client log the first time or two might be good > as well. I run logcheck on my Debian machine, and so I see the various > messages that BoxBackup sends out... Good. But it's also supposed to email you when things happen. > >> >>> >>> Do you think people need a way to purposely delete backups? Say they >>> backed up something they really, really, don't want a copy of? You >>> seem to >>> have thought about this carefully what's your opinion? (Something >>> like >>> the >>> Sarbanes-Oxely stuff in the USA). >>> >>> I can see that something like that could really cause problems, but >>> may >>> also be useful if people are managing their own backups (for instance >>> email retention laws where you probably want it to disappear on the >>> crack >>> of midnight the first day you can delete it). I also get the >>> impression >>> you need to be able to hold certain items back from deletion sort of >>> indefinitely say if the court has ordered you to keep it for some >>> reason. >> >> I have been more interesting in preserving data than destroying it, so >> controlled deletion didn't really occur to me. But you're right, it is >> useful. >> >> There does need to be a way of retaining data more carefully. When I >> add the "mark" feature, you'll be able to specify how many marked >> files >> should exist, so you could mark weekly, and say you want to keep at >> least 6 weeks of marked files. > > The Sarbanes-Oxely stuff though would require you to absolutely > guarantee > that something could not get deleted until it was untagged. Then don't delete it from the client. Things which match files on the client are never deleted from the store. > >> >>> >>> Do you think a config item for max retention time might be useful? >> >> Sounds a sensible start. > > And again for Sarbanes-Oxely it would have to get nuked immediately > once > it expired, it could not wait until you hit the soft limit. > > I suppose you have the wrinkle of what to do with the file if it still > exists on the client, not backing it up could be counter-intuitive, Surely the backup system merely reflects the state of the disc on the client, and retention is just about what to do on the server when files are deleted. If they exist on the client, they exist on the server. > but I > guess you could only keep one copy or something in that case. Actually > I > think that's the wrong way of looking at it, see below: > > Maybe if the location had a min-retention-time and a > max-retention-time, > that way you would know the file (older copies of the current file on > client disk as well) would be kept for at least the minimum time > period, > then all the older copies would for sure get nuked at the > max-retention-time, the current backup of course would be retained for > min-retention-time. Kind of like a conveyor belt with stuff falling off > the end. > > The min-retention-time would supersede any garbage collection, you > would > know for sure that file would exist for that time period, if you run > out > of space, then you deal with it... > > Again the max-retention-time would nuke the backups immediately after > they > expired, it wouldn't wait for garbage collection. > > It might be interesting to have an API where you could programatically > flag files to be unremovable, I was thinking of putting something in the bbackupd.conf file. > but then, maybe it makes more sense to have > a regexp in the config file where you could specify files or > directories > as being undeletable. If someone needed to modify things, they could > modify the config file from whatever program was managing the backup > stuff. > > Finally, to comply with something like Sarbanes-Oxely, that might not > be > good enough, you may need the ability to mark a particular backup(s) > of a > file as being undeletable, but ensure the rest follow the current > rules. > After all, you want to nuke the stuff that is covered by that as > quickly > as possible so it can't be requested by your opponent. However that > would > probably complicate things, so going with the pure conveyor belt method > would probably be better as a start. > > Finally, we would assume that the client has something on the client > machine that would handle deleting files that had expired, so if the > file > needed to disappear, they would handle that. At that point the file's > backups would follow the retention rules. Basically limit our scope to > what we can reasonably control and worry about. > > The only other thing that comes to mind is perhaps some sort of > optional > logging when expired files and so on are deleted, that way > someone/(some > program) could be sure that the retention rules were being followed. Or a command for bbackupquery to check? > >>> Is there some way to list how much space you are allowed to have on >>> the >>> drive, along with how much you are using from the backup query >>> client? >> >> I will add a command to bbackupquery -- I always use bbstoreaccounts >> on >> the server to find out such things, but this is not possible if you >> don't have that access to the server. I've done this now. It even draws a pretty little ASCII chart of usage. > > I don't think you have really said this, but I see great potential in > this > software being used as a *utility* where someone rents out a server as > a > bit-bucket to others for a monthly fee, having proper retention rule > hooks > would allow it to be used as such in more situations. Yes, I have considered this. :-) > > I don't personally have any real intention of using the software that > way, > but I can see the retention rules being useful to my friends who have > businesses that I might back up onto my server. It would all be configured client side, which makes this easier. Ben From boxbackup at fluffy.co.uk Tue Apr 20 09:38:28 2004 From: boxbackup at fluffy.co.uk (Paul Arch) Date: Tue, 20 Apr 2004 16:38:28 +0800 Subject: [Box Backup] Cygwin build References: <006101c422af$95cabed0$0401050a@cel2000> <8016E54C-8EB9-11D8-830D-000A95AFF7F8@fluffy.co.uk> <00e201c42353$00e79050$0401050a@cel2000> <01a001c42393$31943e20$0401050a@cel2000> <5432F62A-8F89-11D8-9BDB-000A95AFF7F8@fluffy.co.uk> <021b01c42443$09b78f50$0401050a@cel2000> <8D569070-9063-11D8-8A18-000A95AFF7F8@fluffy.co.uk> Message-ID: <054601c426b2$dafd3d00$0401050a@cel2000> > > It would be worth checking that the basics work, I'd like to know that > test/backupdiff, common, compress and crypto all worked. > Hi, I havent done much with this over the last couple of days (weekend + moving house ), but anyway I quickly ran make test and got : common: PASSED crypto: PASSED compress: PASSED basicserver: PASSED raidfile: make failed backupstore: make failed backupdiff: PASSED bbackupd: make failed Because of the 'inflexibility' of the cygwin/dos prompt environment I just ran that in, I cant tell you much more ! I havent applied any of the changes I did to get the main stuff to compile to the test suite. cheers Paul From boxbackup at fluffy.co.uk Tue Apr 20 09:56:40 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 20 Apr 2004 09:56:40 +0100 Subject: [Box Backup] Cygwin build In-Reply-To: <054601c426b2$dafd3d00$0401050a@cel2000> References: <006101c422af$95cabed0$0401050a@cel2000> <8016E54C-8EB9-11D8-830D-000A95AFF7F8@fluffy.co.uk> <00e201c42353$00e79050$0401050a@cel2000> <01a001c42393$31943e20$0401050a@cel2000> <5432F62A-8F89-11D8-9BDB-000A95AFF7F8@fluffy.co.uk> <021b01c42443$09b78f50$0401050a@cel2000> <8D569070-9063-11D8-8A18-000A95AFF7F8@fluffy.co.uk> <054601c426b2$dafd3d00$0401050a@cel2000> Message-ID: On 20 Apr 2004, at 09:38, Paul Arch wrote: > >> >> It would be worth checking that the basics work, I'd like to know that >> test/backupdiff, common, compress and crypto all worked. >> > > Hi, > I havent done much with this over the last couple of days (weekend + > moving > house ), but anyway I quickly ran make test and got : > > common: PASSED > crypto: PASSED > compress: PASSED > basicserver: PASSED > raidfile: make failed > backupstore: make failed > backupdiff: PASSED > bbackupd: make failed > > Because of the 'inflexibility' of the cygwin/dos prompt environment I > just > ran that in, I cant tell you much more ! I havent applied any of the > changes > I did to get the main stuff to compile to the test suite. That should be fine for a version of the client -- the ones which fail are ones which I wouldn't expect to work on Cygwin. Send me the changes, and I'll produce a distribution with them in for people to try! Thanks. Ben From boxbackup at fluffy.co.uk Tue Apr 20 15:33:37 2004 From: boxbackup at fluffy.co.uk (Flavio Curti) Date: Tue, 20 Apr 2004 16:33:37 +0200 Subject: [Box Backup] Removing account Message-ID: <20040420143337.GQ592@no-way.org> Hello First, I'd like to thank you for this fine software. It's exactly what I was looking for! (Development of duplicity which I used before seems to have stalled) I use it for 2 Servers right now but plan to extend it to all our servers (12). It would be great if there was a Windows version of the client as well! Anyway, on to my question: How to remove an account? As there is no ``bbstoreaccounts remove'' I thought about the following, but just wanted to be sure that I will not messup the database: - delete the backup dir (/backup/bbox/$HEXADDR/) - delete the account line from accounts.txt - delete the key (./clients/$HEX-cert.pm) Will this work as expected without messing up my other configs? Thank you & Greetz Flavio -- http://no-way.org/~fcu/ Mach mit bei der Community-Bibliothek - In Zuerich/CH http://zurich.communitybooks.org/ - Worldwide http://dlpdev.theps.net/ListOfExistingDlpNodes From boxbackup at fluffy.co.uk Tue Apr 20 16:59:23 2004 From: boxbackup at fluffy.co.uk (SITKEI Attila) Date: Tue, 20 Apr 2004 17:59:23 +0200 Subject: [Box Backup] packaging, set limits, thank you's Message-ID: <20040420155916.GA24825@uu.pmihiv.hu> Good evening, * Is there any plan out there for packaging this piece of software? I've thougth mainly on linux' distros because of their `have-all-in-packages' philosophy; as of primary goal I'd say debian, that I'm using on some systems. * What if I set limit for an account over the physically possible amount? And, additionally, it'd be great to be able to set limit regarding the bbstored's machine view: the amount of data that may be used by client (now the only possibility is the give this before compression). * I wish to thank the author for his work. The software is extremely useful and -- after some practices -- is easy to use. Bye --tef From boxbackup at fluffy.co.uk Tue Apr 20 18:40:43 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 20 Apr 2004 18:40:43 +0100 Subject: [Box Backup] Removing account In-Reply-To: <20040420143337.GQ592@no-way.org> References: <20040420143337.GQ592@no-way.org> Message-ID: On 20 Apr 2004, at 15:33, Flavio Curti wrote: > Hello > > First, I'd like to thank you for this fine software. It's exactly what > I > was looking for! (Development of duplicity which I used before seems to > have stalled) Box Backup will be finished! I spend quite a large proportion of my working day on it at the moment. > > I use it for 2 Servers right now but plan to extend it to all our > servers (12). It would be great if there was a Windows version of the > client as well! There should be a Cygwin client build soon. I plan to do a proper Windows port at some point. > > Anyway, on to my question: How to remove an account? As there is no > ``bbstoreaccounts remove'' I thought about the following, but just > wanted to be sure that I will not messup the database: > - delete the backup dir (/backup/bbox/$HEXADDR/) > - delete the account line from accounts.txt This is correct. > - delete the key (./clients/$HEX-cert.pm) Not necessary -- the server doesn't look at client certificates. > > Will this work as expected without messing up my other configs? Yes. I must write an account deletion tool sometime. I've been more interested in keeping data than deleting it so far. Thanks for trying it out, feedback on how it all works out would be appreciated. Ben From boxbackup at fluffy.co.uk Tue Apr 20 18:47:14 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 20 Apr 2004 18:47:14 +0100 Subject: [Box Backup] packaging, set limits, thank you's In-Reply-To: <20040420155916.GA24825@uu.pmihiv.hu> References: <20040420155916.GA24825@uu.pmihiv.hu> Message-ID: On 20 Apr 2004, at 16:59, SITKEI Attila wrote: > Good evening, > > * Is there any plan out there for packaging this piece of software? > I've thougth mainly on linux' distros because of their > `have-all-in-packages' philosophy; as of primary goal I'd say debian, > that I'm using on some systems. The 'parcels' system is a sort of packaging, aimed at making it relatively easy to deploy on multiple identical systems. With regard to ports and packages on various platforms, I am leaving it up to those who do that kind of thing -- it shouldn't be that hard for them to package it up nicely. But it's not something I intend to do. > > * What if I set limit for an account over the physically possible > amount? When the disc fills up, clients will start receiving errors from the server when they store files. The uploads will be retried. However, this is not a recommended thing to do. Accounts will, after time, tend to hover around the soft limit, so don't allocate more space than you have, and have a bit spare. > And, additionally, it'd be great to be able to set limit > regarding the bbstored's machine view: the amount of data that may be > used by client (now the only possibility is the give this before > compression). The storage limit is specified on the server, in disc space used on the server -- it *is* set in the bbstored's machine view of things. A little bit extra will be used for the file meta-data (the directory structure), but nothing too excessive. > > * I wish to thank the author for his work. The software is extremely > useful and -- after some practices -- is easy to use. You're welcome! Ben From boxbackup at fluffy.co.uk Wed Apr 21 08:07:30 2004 From: boxbackup at fluffy.co.uk (SITKEI Attila) Date: Wed, 21 Apr 2004 09:07:30 +0200 Subject: [Box Backup] packaging, set limits, thank you's In-Reply-To: References: <20040420155916.GA24825@uu.pmihiv.hu> Message-ID: <20040421070730.GB24825@uu.pmihiv.hu> On Tue, Apr 20, 2004 at 06:47:14PM +0100, Ben Summers wrote: > > And, additionally, it'd be great to be able to set limit > >regarding the bbstored's machine view: the amount of data that may be > >used by client (now the only possibility is the give this before > >compression). > > The storage limit is specified on the server, in disc space used on the > server -- it *is* set in the bbstored's machine view of things. A > little bit extra will be used for the file meta-data (the directory > structure), but nothing too excessive. in my example: $ /usr/local/bin/bbstoreaccounts info f22 Account ID: 00000f22 Last object ID: 27381 Blocks used: 4862516 (9497.10Mb) Blocks used by old files: 306779 (599.18Mb) Blocks used by deleted files: 27888 (54.47Mb) Blocks used by directories: 5976 (11.67Mb) Block soft limit: 6656000 (13000.00Mb) Block hard limit: 7168000 (14000.00Mb) Client store marker: 1082373546000000 $ df -h Filesystem Size Used Avail Capacity Mounted on /dev/wd0a 295M 200M 80.2M 71% / /dev/wd0d 8.9G 6.2G 2.3G 73% /data $ uname -a OpenBSD bdb 3.4 GENERIC#1 i386 $ the `Blocks used' value is true for the client's data's amount, but not for used storage on the server, why? --tef From boxbackup at fluffy.co.uk Wed Apr 21 09:40:10 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 21 Apr 2004 09:40:10 +0100 Subject: [Box Backup] packaging, set limits, thank you's In-Reply-To: <20040421070730.GB24825@uu.pmihiv.hu> References: <20040420155916.GA24825@uu.pmihiv.hu> <20040421070730.GB24825@uu.pmihiv.hu> Message-ID: <803B2D2C-936F-11D8-AF1A-000A95AFF7F8@fluffy.co.uk> On 21 Apr 2004, at 08:07, SITKEI Attila wrote: > On Tue, Apr 20, 2004 at 06:47:14PM +0100, Ben Summers wrote: > >>> And, additionally, it'd be great to be able to set limit >>> regarding the bbstored's machine view: the amount of data that may be >>> used by client (now the only possibility is the give this before >>> compression). >> >> The storage limit is specified on the server, in disc space used on >> the >> server -- it *is* set in the bbstored's machine view of things. A >> little bit extra will be used for the file meta-data (the directory >> structure), but nothing too excessive. > > in my example: > > $ /usr/local/bin/bbstoreaccounts info f22 > Account ID: 00000f22 > Last object ID: 27381 > Blocks used: 4862516 (9497.10Mb) > Blocks used by old files: 306779 (599.18Mb) > Blocks used by deleted files: 27888 (54.47Mb) > Blocks used by directories: 5976 (11.67Mb) > Block soft limit: 6656000 (13000.00Mb) > Block hard limit: 7168000 (14000.00Mb) > Client store marker: 1082373546000000 > $ df -h > Filesystem Size Used Avail Capacity Mounted on > /dev/wd0a 295M 200M 80.2M 71% / > /dev/wd0d 8.9G 6.2G 2.3G 73% /data > $ uname -a > OpenBSD bdb 3.4 GENERIC#1 i386 > $ > > the `Blocks used' value is true for the client's data's amount, but > not for used storage on the server, why? 1) The bbstoreaccounts info command only shows the space used by that single account. I assume you use your server for other things, or other accounts, so the usage listing will not match a df listing. 2) There are a number of possibilities why the space used listed on the server would match the size of the data on the client: a) Old and deleted files increase the total size (because the first entry includes all the other entries) b) You are using the raidfile system on the server, and the files on the client compress to about 66% of their original size (which is to be expected for the average file set). Then the raidfile system increases the size of the stored files by 50%, resulting in the space used on the server equaling the size of the files on the client. c) You are not using the raidfile system on the server, and the files on the client aren't very compressible. So each file stored is about the same size as that file on the client. Does this make sense? I can assure you that the sizes specified to bbstoreaccounts are set in server side usage. Ben From boxbackup at fluffy.co.uk Wed Apr 21 09:35:57 2004 From: boxbackup at fluffy.co.uk (Paul Arch) Date: Wed, 21 Apr 2004 16:35:57 +0800 Subject: [Box Backup] bbackupd suggestions Message-ID: <007a01c4277b$aad987e0$0401050a@cel2000> Hi, can bbackupd provide some more feedback (when a sync has been run) to the user ? I am thinking along the lines of say : - Total progress - Current file being synced - Current Speed - Estimated time left Actually, someone 'simular' to rsync's --progress would be very nice. I know boxbackup has been 'worked' around the fact for set and forget (lazy mode), so I dont know how hard it would be ? Also, is it possible to backup/sync individual 'BackupLocations' from /etc/box/bbackupd.conf ? cheers Paul Arch Software Engineer S.D.M. Group Pty. Ltd. http://www.sdmgroup.com.au From boxbackup at fluffy.co.uk Wed Apr 21 11:56:32 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 21 Apr 2004 11:56:32 +0100 Subject: [Box Backup] bbackupd suggestions In-Reply-To: <007a01c4277b$aad987e0$0401050a@cel2000> References: <007a01c4277b$aad987e0$0401050a@cel2000> Message-ID: <8CEF3F76-9382-11D8-AF1A-000A95AFF7F8@fluffy.co.uk> On 21 Apr 2004, at 09:35, Paul Arch wrote: > Hi, > > can bbackupd provide some more feedback (when a sync has been run) > to the > user ? > I am thinking along the lines of say : > - Total progress > - Current file being synced > - Current Speed > - Estimated time left > > Actually, someone 'simular' to rsync's --progress would be very nice. > I > know boxbackup has been 'worked' around the fact for set and forget > (lazy > mode), Lazy mode was in fact the original design, snapshot mode was added later. > so I dont know how hard it would be ? > > Also, is it possible to backup/sync individual 'BackupLocations' from > /etc/box/bbackupd.conf ? No. This isn't really what bbackupd was designed to do. It's designed to just sit there, looking after things. It's not a sync on demand tool. As for progress, well, it would be quite a bit of extra work adding the hooks into there. I didn't even think about it, as the backup system was designed to be something which would just sit there in the background doing backups when they were required. It did occur to me a few days ago that I should do a synchronisation tool, using the bbstored server. This would be what you want... the idea being to have a file hierarchy that you want to duplicate over multiple machines, and then be able to sync machines on demand. This would copy up changed files, and download changes with everything on the server encrypted (much like Apple's iDisk, but with encryption). bbackupd cannot do two-way file synchronisation. bbstored is really just a generic file server, with some special and interesting properties. Lots of extra utilities could be built with it, it's just a matter of time to write them. My focus at the moment is backup, though. Sorry. Ben From boxbackup at fluffy.co.uk Wed Apr 21 15:54:35 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Wed, 21 Apr 2004 10:54:35 -0400 (EDT) Subject: [Box Backup] Re: boxbackup digest, Vol 1 #59 - 8 msgs In-Reply-To: <20040421110002.7874.62111.Mailman@love.warhead.org.uk> References: <20040421110002.7874.62111.Mailman@love.warhead.org.uk> Message-ID: > Message: 2 > Date: Tue, 20 Apr 2004 17:59:23 +0200 > From: SITKEI Attila > To: boxbackup at fluffy.co.uk > Subject: [Box Backup] packaging, set limits, thank you's > Reply-To: boxbackup at fluffy.co.uk > > Good evening, > > * Is there any plan out there for packaging this piece of software? > I've thougth mainly on linux' distros because of their > `have-all-in-packages' philosophy; as of primary goal I'd say debian, > that I'm using on some systems. If you need any help with Debian, I'm up and running with it now... I've built packages before, but I'm not yet a official Debian developer. In the end, a package isn't absolutely required, and I can help you configure it as I have. Send an email to rprice at freeshell.org or if you think others would benefit, just send it to the list... Rick From boxbackup at fluffy.co.uk Sun Apr 25 00:22:28 2004 From: boxbackup at fluffy.co.uk (Flavio Curti) Date: Sun, 25 Apr 2004 01:22:28 +0200 Subject: [Box Backup] Removing account In-Reply-To: References: <20040420143337.GQ592@no-way.org> Message-ID: <20040424232228.GB28232@no-way.org> Hi Ben (an hi to the list) On Tue, Apr 20, 2004 at 06:40:43PM +0100, Ben Summers wrote: > Box Backup will be finished! I spend quite a large proportion of my > working day on it at the moment. > There should be a Cygwin client build soon. I plan to do a proper > Windows port at some point. Great to hear that! > >Anyway, on to my question: How to remove an account? As there is no > >``bbstoreaccounts remove'' I thought about the following, but just > >wanted to be sure that I will not messup the database: > > - delete the backup dir (/backup/bbox/$HEXADDR/) > > - delete the account line from accounts.txt > This is correct. > Thanks for trying it out, feedback on how it all works out would be > appreciated. Okay, after fixing the server (which just had to break before I could try :), it'd say it works. I deleted some accounts using this method, others were unaffected, I also re-added one account which is working fine right now... Hope it helps & thank you Flavio -- http://no-way.org/~fcu/ Mach mit bei der Community-Bibliothek - In Zuerich/CH http://zurich.communitybooks.org/ - Worldwide http://dlpdev.theps.net/ListOfExistingDlpNodes From boxbackup at fluffy.co.uk Mon Apr 26 00:02:07 2004 From: boxbackup at fluffy.co.uk (Flavio Curti) Date: Mon, 26 Apr 2004 01:02:07 +0200 Subject: [Box Backup] boxbackup on 64bit systems (= on alpha) Message-ID: <20040425230207.GD28232@no-way.org> Hi I tried to run bbackupd as a client on a alpha debian Linux. While it compiled fine and the daemon started, nothing happens. I suspect it's not 64bit safe, or little-endian / big endian problems... So, should it run? Thank you & Greetz Flavio some warnings from compiling are below, if you need more just say so ;) BackupDaemon.cpp: In member function `void BackupDaemon::Run2()': BackupDaemon.cpp:549: warning: long long int format, int64_t arg (arg 3) BackupDaemon.cpp:549: warning: long long int format, int64_t arg (arg 4) BackupDaemon.cpp:549: warning: long long int format, int64_t arg (arg 5) g++ -DNDEBUG -O2 -Wall -DPLATFORM_LINUX -DPLATFORM_GCC3 -DBOX_VERSION="\"0.05\"" -c ConversionString.cpp -o ../../release/lib/common/ConversionString.o ConversionString.cpp: In function `int32_t BoxConvert::_ConvertStringToInt(const char*, int)': ConversionString.cpp:101: warning: comparison is always false due to limited range of data type ConversionString.cpp:101: warning: comparison is always false due to limited range of data type g++ -DNDEBUG -O2 -Wall -DPLATFORM_LINUX -DPLATFORM_GCC3 -DBOX_VERSION="\"0.05\"" -c Utils.cpp -o ../../release/lib/common/Utils.o Utils.cpp: In function `void SplitString(const std::string&, char, std::vector >&)': Utils.cpp:76: warning: comparison is always true due to limited range of data type BackupStoreFileDiff.cpp: In static member function `static void BackupStoreFile::MoveStreamPositionToBlockIndex(IOStream&)': BackupStoreFileDiff.cpp:144: warning: comparison between signed and unsigned integer expressions BackupStoreFileDiff.cpp: In function `void SearchForMatchingBlocks(IOStream&, std::map, std::allocator > >&, BackupStoreFileCreation::BlocksAvailableEntry*, long int, int32_t*)': BackupStoreFileDiff.cpp:584: warning: comparison between signed and unsigned g++ -DNDEBUG -O2 -Wall -I../../lib/common -I../../lib/compress -I../../lib/crypto -I../../lib/server -DPLATFORM_LINUX -DPLATFORM_GCC3 -DBOX_VERSION="\"0.05\"" -c autogen_BackupProtocolClient.cpp -o ../../release/lib/backupclient/autogen_BackupProtocolClient.o autogen_BackupProtocolClient.cpp: In member function `virtual void BackupProtocolClientLoginConfirmed::LogSysLog(const char*) const': autogen_BackupProtocolClient.cpp:158: warning: long long unsigned int format, long int arg (arg 4) autogen_BackupProtocolClient.cpp:158: warning: long long unsigned int format, long int arg (arg 5) autogen_BackupProtocolClient.cpp:158: warning: long long unsigned int format, long int arg (arg 6) autogen_BackupProtocolClient.cpp:158: warning: long long unsigned int format, long int arg (arg 7) autogen_BackupProtocolClient.cpp: In member function `virtual void BackupProtocolClientSuccess::LogSysLog(const char*) const': autogen_BackupProtocolClient.cpp:218: warning: long long unsigned int format, long int arg (arg 4) autogen_BackupProtocolClient.cpp: In member function `virtual void BackupProtocolClientSetClientStoreMarker::LogSysLog(const char*) const': autogen_BackupProtocolClient.cpp:250: warning: long long unsigned int format, long int arg (arg 4) autogen_BackupProtocolClient.cpp: In member function `virtual void BackupProtocolClientGetObject::LogSysLog(const char*) const': autogen_BackupProtocolClient.cpp:282: warning: long long unsigned int format, long int arg (arg 4) autogen_BackupProtocolClient.cpp: In member function `virtual void .... -- http://no-way.org/~fcu/ Mach mit bei der Community-Bibliothek - In Zuerich/CH http://zurich.communitybooks.org/ - Worldwide http://dlpdev.theps.net/ListOfExistingDlpNodes From boxbackup at fluffy.co.uk Mon Apr 26 09:57:28 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 26 Apr 2004 09:57:28 +0100 Subject: [Box Backup] boxbackup on 64bit systems (= on alpha) In-Reply-To: <20040425230207.GD28232@no-way.org> References: <20040425230207.GD28232@no-way.org> Message-ID: On 26 Apr 2004, at 00:02, Flavio Curti wrote: > Hi > > I tried to run bbackupd as a client on a alpha debian Linux. While it > compiled fine and the daemon started, nothing happens. I suspect it's > not 64bit safe, or little-endian / big endian problems... > So, should it run? It should run. Is there anything in your logs? I've very carefully written it to be independent of endianness (all structures which go between machines have bytes swapped to network byte order) and all types which are important to be specific lengths are specified explicitly. There is one minor issue with endianness, where you can't restore on a platform with a different endianness to the one which backed it up, but I'm on the case and will fix that in the next release (in a backwards compatible way), and this won't affect you in this case. The errors you've listed all seem to be just warnings which are fairly harmless, although it'd be nice to fix them. Which compiler are you using? (output of gcc -v) I'm aware that it won't work on openbsd-spparc64 -- C++ exceptions (a basic language feature) just don't work on this particular platform, but I suspect that it'll work on Linux. Try running the tests, and see what happens perl ./runtest.pl ALL release It'll take a while, but should give a fair indication of what's likely to work and where it goes wrong. If I can have an account on that machine, then I can have a look sometime and see if I can fix things. It'd be useful to have a 64 bit machine to test on, as you can see, the code needs cleaning up a little. Thanks, Ben From boxbackup at fluffy.co.uk Wed Apr 28 13:17:45 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 28 Apr 2004 13:17:45 +0100 Subject: [Box Backup] 0.05PLUS1 Message-ID: <0E0F33D8-990E-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> I've just uploaded a new preview version to http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.05PLUS1.tgz This includes some slightly fundamental changes, but it's completely backwards compatible (and I spent quite a bit of time making sure it was.) So it's perfectly safe to install and use. However, 0.05PLUS1 clients need a 0.05PLUS1 server (although 0.05PLUS1 servers are happy with earlier clients). This also includes a THANKS.txt file, which is long overdue. Since I foolishly haven't been writing it as I've gone along, I would appreciate gentle reminders if you should be on the list, but I've missed you off. Right then, changes: * add usage command to bbackupquery - tells you how much space you're using on the server from the client side * bug fixes - various * add delete [yes] command to bbstoreaccounts - delete accounts as well as create them! Use the optional 'yes' argument to skip the confirmation question. * add check [fix] [quiet] command to bbstoreaccounts - more on this later * Use AES for file data - After long consideration, I think AES is a better choice of cipher for file data, mainly due to the larger block size. Blowfish is still used for meta data (as it's more space efficient with it's 64 bit block size), and Blowfish encoded files can still be restored. Account checking: If you do something like /usr/local/bin/bbstoreaccounts check 31 account 31 will be checked for errors. If you do /usr/local/bin/bbstoreaccounts check 31 fix these errors will be fixed. It may take a while to run, but will get all the readable files into a state where they can be restored, and repair all the problems which may cause the server to give errors. So if you have problems when files get corrupted (eg an unclean shutdown corrupts a file or two) then this should sort it out. I'd like to make sure that it's working correctly. If you could check some real-life accounts (without the fix option!) and report back, that would be appreciated. This doesn't fix errors in the underlying raid file system, I'm writing a separate utility to do that. Thanks for trying this all out! Ben From boxbackup at fluffy.co.uk Wed Apr 28 14:33:43 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 28 Apr 2004 14:33:43 +0100 Subject: [Box Backup] 0.05PLUS1 In-Reply-To: <0E0F33D8-990E-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> References: <0E0F33D8-990E-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> Message-ID: On 28 Apr 2004, at 13:17, Ben Summers wrote: > > I've just uploaded a new preview version to > > http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.05PLUS1.tgz > > This includes some slightly fundamental changes, but it's completely > backwards compatible (and I spent quite a bit of time making sure it > was.) So it's perfectly safe to install and use. However, 0.05PLUS1 > clients need a 0.05PLUS1 server (although 0.05PLUS1 servers are happy > with earlier clients). I forgot to mention that the way it does file attributes has changed slightly, so when you first run a new client against an old backup store, it will upload new file attributes for every file. This uses up a bit of bandwidth, and increases the size of the directories on the store by about 50 bytes a file. Not a huge problem. Ben From boxbackup at fluffy.co.uk Wed Apr 28 16:52:34 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 28 Apr 2004 16:52:34 +0100 Subject: [Box Backup] 0.05PLUS1B Message-ID: <10A02C2B-992C-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> Er, there was a slight problem with the 0.05PLUS1 archive. It wasn't quite backwards compatible enough. Anyway, a new version is now up http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.05PLUS1B.tgz Sorry about that. Ben From boxbackup at fluffy.co.uk Wed Apr 28 20:23:58 2004 From: boxbackup at fluffy.co.uk (Mailing Lists) Date: Wed, 28 Apr 2004 16:23:58 -0300 Subject: [Box Backup] 0.05PLUS1B In-Reply-To: <10A02C2B-992C-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> References: <10A02C2B-992C-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> Message-ID: <409004CE.1060108@coredump.com.ar> Ben Summers wrote: > > Er, there was a slight problem with the 0.05PLUS1 archive. It wasn't > quite backwards compatible enough. Anyway, a new version is now up > > http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.05PLUS1B.tgz > > Sorry about that. > > Ben > > > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup Hello It is not clear to me if upgrade is mandatory or just optional (theres a 'bug fixes' point in changes). I just upgraded to .04 lastnight :( thanks Juan From boxbackup at fluffy.co.uk Wed Apr 28 20:34:11 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Wed, 28 Apr 2004 20:34:11 +0100 Subject: [Box Backup] 0.05PLUS1B In-Reply-To: <409004CE.1060108@coredump.com.ar> References: <10A02C2B-992C-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> <409004CE.1060108@coredump.com.ar> Message-ID: <067717F2-994B-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> On 28 Apr 2004, at 20:23, Mailing Lists wrote: > Ben Summers wrote: >> Er, there was a slight problem with the 0.05PLUS1 archive. It wasn't >> quite backwards compatible enough. Anyway, a new version is now up >> http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.05PLUS1B.tgz >> Sorry about that. > > Hello > > It is not clear to me if upgrade is mandatory or just optional > (theres a 'bug fixes' point in changes). > > I just upgraded to .04 lastnight :( There is no compulsion to upgrade to these releases. The bugs fixed are unlikely to cause any data loss (although one of them did stop backups happening if files were renamed over files which had been deleted in the past...). I'd be grateful if you did, though. I release these "preview" versions to the list to make sure they're really ready for release to the wider world. I'm confident of their quality, but it's always good to get some feedback before a proper release. If you're running anything less than 0.05, then you do want to upgrade. If you're using an account on my server, then you must upgrade to this one -- as it turns out, it's not so backwards compatible with older clients, even if the data stores are fine in this respect. Oh, and another thing I forgot to mention was that if you're lucky, it might even build a backup client on Cygwin... Ben From boxbackup at fluffy.co.uk Thu Apr 29 01:20:35 2004 From: boxbackup at fluffy.co.uk (Paul Arch) Date: Thu, 29 Apr 2004 08:20:35 +0800 Subject: [Box Backup] 0.05PLUS1B References: <10A02C2B-992C-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> <409004CE.1060108@coredump.com.ar> <067717F2-994B-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> Message-ID: <09cf01c42d7f$caf496a0$0401050a@cel2000> > > Oh, and another thing I forgot to mention was that if you're lucky, it > might even build a backup client on Cygwin... > It compiles with no errors on cygwin :) :) Well done ! cheers Paul Arch Software Engineer S.D.M. Group Pty. Ltd. http://www.sdmgroup.com.au From boxbackup at fluffy.co.uk Thu Apr 29 11:33:00 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 29 Apr 2004 11:33:00 +0100 Subject: [Box Backup] 0.05PLUS1B In-Reply-To: <09cf01c42d7f$caf496a0$0401050a@cel2000> References: <10A02C2B-992C-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> <409004CE.1060108@coredump.com.ar> <067717F2-994B-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> <09cf01c42d7f$caf496a0$0401050a@cel2000> Message-ID: <963ACA50-99C8-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> On 29 Apr 2004, at 01:20, Paul Arch wrote: > >> >> Oh, and another thing I forgot to mention was that if you're lucky, it >> might even build a backup client on Cygwin... >> > > It compiles with no errors on cygwin :) :) > > Well done ! This of course, is mainly down to Paul's work, finding what works and doesn't work, then negotiating changes with me. I never managed to get my (dodgy, vendor modified) install of Cygwin to compile it. In other words, direct questions on Cygwin to Paul. Ben From boxbackup at fluffy.co.uk Thu Apr 29 11:37:38 2004 From: boxbackup at fluffy.co.uk (Flavio Curti) Date: Thu, 29 Apr 2004 12:37:38 +0200 Subject: [Box Backup] 0.05PLUS1B In-Reply-To: <09cf01c42d7f$caf496a0$0401050a@cel2000> References: <10A02C2B-992C-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> <409004CE.1060108@coredump.com.ar> <067717F2-994B-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> <09cf01c42d7f$caf496a0$0401050a@cel2000> Message-ID: <20040429103738.GG28232@no-way.org> Hello On Thu, Apr 29, 2004 at 08:20:35AM +0800, Paul Arch wrote: > > > > Oh, and another thing I forgot to mention was that if you're lucky, it > > might even build a backup client on Cygwin... > It compiles with no errors on cygwin :) :) Does it also work? Thank you & Greetz Flavio -- http://no-way.org/~fcu/ Mach mit bei der Community-Bibliothek - In Zuerich/CH http://zurich.communitybooks.org/ - Worldwide http://dlpdev.theps.net/ListOfExistingDlpNodes From boxbackup at fluffy.co.uk Thu Apr 29 11:43:21 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 29 Apr 2004 11:43:21 +0100 Subject: [Box Backup] 0.05PLUS1B In-Reply-To: <20040429103738.GG28232@no-way.org> References: <10A02C2B-992C-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> <409004CE.1060108@coredump.com.ar> <067717F2-994B-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> <09cf01c42d7f$caf496a0$0401050a@cel2000> <20040429103738.GG28232@no-way.org> Message-ID: <0874BF4E-99CA-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> On 29 Apr 2004, at 11:37, Flavio Curti wrote: > Hello > > On Thu, Apr 29, 2004 at 08:20:35AM +0800, Paul Arch wrote: >>> >>> Oh, and another thing I forgot to mention was that if you're lucky, >>> it >>> might even build a backup client on Cygwin... >> It compiles with no errors on cygwin :) :) > Does it also work? I believe it's been known to backup stuff to a Linux store server, and then bbackupquery reports it verifies OK. I think Paul is planning to do a web page about it, and maybe even provide a pre-compiled version. Reports of further success welcome. Ben From boxbackup at fluffy.co.uk Thu Apr 29 14:04:07 2004 From: boxbackup at fluffy.co.uk (Nick Suckling) Date: Thu, 29 Apr 2004 14:04:07 +0100 Subject: [Box Backup] 0.05PLUS1B In-Reply-To: <10A02C2B-992C-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> References: <10A02C2B-992C-11D8-8FC7-000A95AFF7F8@fluffy.co.uk> Message-ID: <1083243847.15479.132.camel@dolphin.cmedltd.com> Hi Ben, Just to say I have upgraded my OpenBSD server OpenBSD client and a Gentoo client all worked fine with 0.05PLUS1b. Cheers, Nick On Wed, 2004-04-28 at 16:52, Ben Summers wrote: > Er, there was a slight problem with the 0.05PLUS1 archive. It wasn't > quite backwards compatible enough. Anyway, a new version is now up > > http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.05PLUS1B.tgz > > Sorry about that. > > Ben > > > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > > _____________________________________________________________________ > This e-mail has been scanned for viruses by MCI's Internet Managed Scanning Services - powered by MessageLabs. For further information visit http://www.mci.com > From boxbackup at fluffy.co.uk Thu Apr 29 14:04:52 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Thu, 29 Apr 2004 09:04:52 -0400 (EDT) Subject: [Box Backup] New encryption format & Space usage In-Reply-To: <20040429110002.25129.65772.Mailman@love.warhead.org.uk> References: <20040429110002.25129.65772.Mailman@love.warhead.org.uk> Message-ID: > --__--__-- > > Message: 1 > To: boxbackup at fluffy.co.uk > From: Ben Summers > Date: Wed, 28 Apr 2004 13:17:45 +0100 > Subject: [Box Backup] 0.05PLUS1 > Reply-To: boxbackup at fluffy.co.uk > > > I've just uploaded a new preview version to > > http://www.fluffy.co.uk/boxbackup/bin/boxbackup-0.05PLUS1.tgz > > This includes some slightly fundamental changes, but it's completely > backwards compatible (and I spent quite a bit of time making sure it > was.) So it's perfectly safe to install and use. However, 0.05PLUS1 > clients need a 0.05PLUS1 server (although 0.05PLUS1 servers are happy > with earlier clients). > > This also includes a THANKS.txt file, which is long overdue. Since I > foolishly haven't been writing it as I've gone along, I would > appreciate gentle reminders if you should be on the list, but I've > missed you off. > > Right then, changes: > > * add usage command to bbackupquery > - tells you how much space you're using on the server from the client > side > > * bug fixes > - various > > * add delete [yes] command to bbstoreaccounts > - delete accounts as well as create them! Use the optional 'yes' > argument to skip the confirmation question. > > * add check [fix] [quiet] command to bbstoreaccounts > - more on this later > > * Use AES for file data > - After long consideration, I think AES is a better choice of cipher > for file data, mainly due to the larger block size. Blowfish is still > used for meta data (as it's more space efficient with it's 64 bit block > size), and Blowfish encoded files can still be restored. Hi Ben, How does it move the data forward? To try and explain: Does it convert the data when the client next connects, at server startup time, does it just let the old (encryption) format data expire? After re-reading your email it looks like it lets the old data expire, but I was just wondering. ---- One thing thats annoying me lately (and it's not your fault of course), is trying to find out what exactly I'm backing up when the daemon runs. I tried the option in the config file that should tell you what's going on, but what I was really hoping for was a list of files (and their sizes) that were being backed up. The problem behind this, is that I see some pretty huge numbers for amount of data backed up sometimes (and even for uploaded), and I feel that my machine should have been *quiescent* before the large amount of data was backed up. I also just hit the wall with the amount of data on your store, I think yesterday, and I don't think that I should be backing up enough stuff to cause that. I'm not saying I think your program is wrong, I'm just trying to figure out where the fat file(s) is/are so I can decide if I really want it/them backed up. (The script to notify me of the problem also had an error (Debian Testing), but I haven't had time to look at that yet. I started looking around last night, and I seemed to have some rather large files in my wife's email related folders, 'caught spam' (which only I check) where she gets a couple of hundred spam a day, and the files related to spamassassin. So I'm thinking they were the culprit, it may be that the file gets some big changes which spook the delta algorithm. I'm thinking of using *find* to try and hunt down the large files in the directories, but, I guess even some stats in the syslog would be helpful, IE number of files backed up, maybe even some size stats (smallest, largest, average - for file size and delta size), so I could try and decide if there were 1000 small files that were changed, or one huge file that somehow was changed. This sort of thing would be helpful for someone trying to manage their data on a third party data store (as you are for me). I'd be particularly interested in doing say trending analysis to find out what was happening, and to look for ominous changes. Or even to predict when I'd hit the wall, and need to *purchase* more space from the data store provider. Part of my interest is also related to bandwidth usage, I'm on a ADSL connection, but some of my friends who would be backing up to me would be on dialup, being able to manage the bandwidth usage would be helpful, knowing what sort of changes were happening would help that way (you could try and have multiple backup sets or know *why* a huge upload happened). Other possible options would be deferring large delta's until a set time, while letting small changes through. Although that could be risky and complicated. Rick From boxbackup at fluffy.co.uk Thu Apr 29 14:10:51 2004 From: boxbackup at fluffy.co.uk (Flavio Curti) Date: Thu, 29 Apr 2004 15:10:51 +0200 Subject: [Box Backup] New encryption format & Space usage In-Reply-To: References: <20040429110002.25129.65772.Mailman@love.warhead.org.uk> Message-ID: <20040429131051.GH28232@no-way.org> Hi, me again ;) On Thu, Apr 29, 2004 at 09:04:52AM -0400, rprice at freeshell.org wrote: > but what I was really hoping for was a list of files (and their sizes) > that were being backed up. I second that. I'd like to know: - when the last contact were - how many files/bytes were transferred - how many files were changed, new, deleted and so on Right now it's difficult to tell if the client really works... I was biten many times by backup-clients not really working, so it'd be nice if there was some way to gather some informations. ``bbackupctl stat'' comes to mind. Thank you very much & Greetz Flavio -- http://no-way.org/~fcu/ Mach mit bei der Community-Bibliothek - In Zuerich/CH http://zurich.communitybooks.org/ - Worldwide http://dlpdev.theps.net/ListOfExistingDlpNodes From boxbackup at fluffy.co.uk Thu Apr 29 14:21:09 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 29 Apr 2004 14:21:09 +0100 Subject: [Box Backup] New encryption format & Space usage In-Reply-To: <20040429131051.GH28232@no-way.org> References: <20040429110002.25129.65772.Mailman@love.warhead.org.uk> <20040429131051.GH28232@no-way.org> Message-ID: <14503281-99E0-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> On 29 Apr 2004, at 14:10, Flavio Curti wrote: > Hi, me again ;) > > On Thu, Apr 29, 2004 at 09:04:52AM -0400, rprice at freeshell.org wrote: >> but what I was really hoping for was a list of files (and their sizes) >> that were being backed up. > I second that. I'd like to know: > - when the last contact were > - how many files/bytes were transferred These two bits of info are available in the logs already -- make sure local6.info is logged somewhere with syslog. http://www.fluffy.co.uk/boxbackup/server.html > - how many files were changed, new, deleted and so on I could add that fairly easily. Expect it in the next version. > > Right now it's difficult to tell if the client really works... I was > biten many times by backup-clients not really working, so it'd be nice > if there was some way to gather some informations. ``bbackupctl stat'' > comes to mind. Would that be really better than looking in the logs? And to check the backup properly... /usr/local/bin/bbackupquery "compare -a" quit will give you a definitive answer regarding whether or not the backup has worked. It's worth doing occasionally, and I'd be grateful if you would, just to make sure this software is tested properly. (add the -q flag to check checksums only, not downloading the file data itself -- may not work with data uploaded with pre-0.05PLUS1B clients with 0.05PLUS1B bbackupquery, though) Ben From boxbackup at fluffy.co.uk Thu Apr 29 14:33:15 2004 From: boxbackup at fluffy.co.uk (Flavio Curti) Date: Thu, 29 Apr 2004 15:33:15 +0200 Subject: [Box Backup] New encryption format & Space usage In-Reply-To: <14503281-99E0-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> References: <20040429110002.25129.65772.Mailman@love.warhead.org.uk> <20040429131051.GH28232@no-way.org> <14503281-99E0-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> Message-ID: <20040429133315.GI28232@no-way.org> Hi Ben On Thu, Apr 29, 2004 at 02:21:09PM +0100, Ben Summers wrote: > These two bits of info are available in the logs already -- make sure > local6.info is logged somewhere with syslog. You are right... Sorry for not thinking this really through... > I could add that fairly easily. Expect it in the next version. Great! > >if there was some way to gather some informations. ``bbackupctl stat'' > >comes to mind. > Would that be really better than looking in the logs? Imho yes. But I can do a grep as easily, so don't worry ;) > And to check the backup properly... > [...] Thank you, I'll look into that one! One other question (just send me to the docs if it's there, even I did not find it): How do I find out how many/which versions of a file are stored on the Server? Ie I know I want file /etc/passwd and wanna find out what version are stored... Thank you & Greetz Flavio Curti -- http://no-way.org/~fcu/ Mach mit bei der Community-Bibliothek - In Zuerich/CH http://zurich.communitybooks.org/ - Worldwide http://dlpdev.theps.net/ListOfExistingDlpNodes From boxbackup at fluffy.co.uk Thu Apr 29 14:46:47 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 29 Apr 2004 14:46:47 +0100 Subject: [Box Backup] New encryption format & Space usage In-Reply-To: <20040429133315.GI28232@no-way.org> References: <20040429110002.25129.65772.Mailman@love.warhead.org.uk> <20040429131051.GH28232@no-way.org> <14503281-99E0-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> <20040429133315.GI28232@no-way.org> Message-ID: On 29 Apr 2004, at 14:33, Flavio Curti wrote: > Hi Ben > > On Thu, Apr 29, 2004 at 02:21:09PM +0100, Ben Summers wrote: >> These two bits of info are available in the logs already -- make sure >> local6.info is logged somewhere with syslog. > You are right... Sorry for not thinking this really through... >> I could add that fairly easily. Expect it in the next version. > Great! > >>> if there was some way to gather some informations. ``bbackupctl >>> stat'' >>> comes to mind. >> Would that be really better than looking in the logs? > Imho yes. But I can do a grep as easily, so don't worry ;) I'll consider adding it. :-) > >> And to check the backup properly... >> [...] > > Thank you, I'll look into that one! > > One other question (just send me to the docs if it's there, even I did > not find it): > > How do I find out how many/which versions of a file are stored on the > Server? Ie I know I want file /etc/passwd and wanna find out what > version are stored... http://www.fluffy.co.uk/boxbackup/retrieve.html Ben From boxbackup at fluffy.co.uk Thu Apr 29 15:00:32 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 29 Apr 2004 15:00:32 +0100 Subject: [Box Backup] New encryption format & Space usage In-Reply-To: References: <20040429110002.25129.65772.Mailman@love.warhead.org.uk> Message-ID: <9470308A-99E5-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> On 29 Apr 2004, at 14:04, rprice at freeshell.org wrote: >> >> * Use AES for file data >> - After long consideration, I think AES is a better choice of >> cipher >> for file data, mainly due to the larger block size. Blowfish is still >> used for meta data (as it's more space efficient with it's 64 bit >> block >> size), and Blowfish encoded files can still be restored. > > Hi Ben, > > How does it move the data forward? > > To try and explain: > > Does it convert the data when the client next connects, at server > startup > time, does it just let the old (encryption) format data expire? > > After re-reading your email it looks like it lets the old data expire, > but > I was just wondering. Files are made up of blocks, from about 4k -- 64k depending on file size. Each block is compressed if it is over 256 bytes long. Then it is encrypted. Each block has a 1 byte header saying whether it is compressed, and which encryption algorithm is used. (This was always the case.) For the AES thing, I've just added a new algorithm, and encode blocks with it by default (with a separate key). So new blocks (remember the diffs!) are AES, old ones are blowfish, and the store will gradually move over to them as time goes on. Files which have been patched may contain blocks with both encryption methods. There's another change regarding the block indices. There was a bug relating to endianness. To fix this, there's a new version of the file format, which is almost exactly the same. With backwards compatibility enabled, the old and new formats are parsed successfully. But you can't diff from old to new. Then the attributes: Changes are detected using a hash rather than the attribute mod time. The hash is stored in the same place as the attribute mod time was. So it won't match, so attributes will be uploaded for each file when it first sees a directory. Hope that explains it. > > > ---- > > One thing thats annoying me lately (and it's not your fault of > course), is > trying to find out what exactly I'm backing up when the daemon runs. I > tried the option in the config file that should tell you what's going > on, > but what I was really hoping for was a list of files (and their sizes) > that were being backed up. Extended logging will give you some hints. > > The problem behind this, is that I see some pretty huge numbers for > amount > of data backed up sometimes (and even for uploaded), and I feel that my > machine should have been *quiescent* before the large amount of data > was > backed up. In the default configs, changes are uploaded 6 hours ahead. So it would have to be quiescent for over 6 hours for the upload to make no changes. > > I also just hit the wall with the amount of data on your store, I think > yesterday, and I don't think that I should be backing up enough stuff > to > cause that. I'm not saying I think your program is wrong, I'm just > trying > to figure out where the fat file(s) is/are so I can decide if I really > want it/them backed up. (The script to notify me of the problem also > had > an error (Debian Testing), but I haven't had time to look at that yet. You are requested to test and possible change that by the config script -- but let me know if you have a patch for it. > > I started looking around last night, and I seemed to have some rather > large files in my wife's email related folders, 'caught spam' (which > only > I check) where she gets a couple of hundred spam a day, and the files > related to spamassassin. So I'm thinking they were the culprit, it may > be > that the file gets some big changes which spook the delta algorithm. Did you just upgrade the client to 0.05PLUS1B? It won't diff if the old version of the file was written before you upgraded. > > I'm thinking of using *find* to try and hunt down the large files in > the > directories, This is probably the best approach. > but, I guess even some stats in the syslog would be helpful, > IE number of files backed up, maybe even some size stats (smallest, > largest, average - for file size and delta size), so I could try and > decide if there were 1000 small files that were changed, or one huge > file > that somehow was changed. I'll put some more stats in. > > This sort of thing would be helpful for someone trying to manage their > data on a third party data store (as you are for me). I'd be > particularly > interested in doing say trending analysis to find out what was > happening, > and to look for ominous changes. Or even to predict when I'd hit the > wall, > and need to *purchase* more space from the data store provider. In bbackupquery, type "usage" for space usage on the server. (New in 0.05PLUS1B.) Or /usr/local/bin/bbackupquery usage quit as a simple one liner. (I've just increased your allocation a bit, BTW) > > Part of my interest is also related to bandwidth usage, I'm on a ADSL > connection, but some of my friends who would be backing up to me would > be > on dialup, being able to manage the bandwidth usage would be helpful, > knowing what sort of changes were happening would help that way (you > could > try and have multiple backup sets or know *why* a huge upload > happened). > Other possible options would be deferring large delta's until a set > time, > while letting small changes through. Although that could be risky and > complicated. I'm designing taking ADSL a minimum requirement for this system. That's not to say it couldn't be used with dial-up, it's just it's not designed for it. Ben From boxbackup at fluffy.co.uk Thu Apr 29 15:12:38 2004 From: boxbackup at fluffy.co.uk (Flavio Curti) Date: Thu, 29 Apr 2004 16:12:38 +0200 Subject: [Box Backup] Expiring files on the server Message-ID: <20040429141238.GJ28232@no-way.org> Hi, me again. I'm just checking my backupclient servers, and I noticed that one is no longer backing up, because it used all his space on the backup storage: sudo-root at gnu:/tmp# /usr/local/bin/bbstoreaccounts info 7 sudo-root at gnu:/tmp# /usr/local/bin/bbstoreaccounts info 7 Account ID: 00000007 Last object ID: 1408526 Blocks used: 6563819 (51279.84Mb) Blocks used by old files: 0 (0.00Mb) Blocks used by deleted files: 0 (0.00Mb) Blocks used by directories: 265033 (2070.57Mb) Block soft limit: 6553600 (51200.00Mb) Block hard limit: 7208960 (56320.00Mb) Client store marker: 1083125666000000 (notice how the hard limit is not reached?) On the client I get this: Apr 29 16:02:53 elefant bbackupd[23374]: Exceeded storage limits on server -- not uploading changes to files Apr 29 16:02:53 elefant bbackupd[23374]: Exceeded storage limits on server -- not uploading changes to files and this for the last 3 days, which is not good... How is this gonna work now? Shouldn't the store daemon regularly delete some files? For me it's actually more important to have the now back-upped instead of old versions... Thank you very much & Greets Flavio -- http://no-way.org/~fcu/ Mach mit bei der Community-Bibliothek - In Zuerich/CH http://zurich.communitybooks.org/ - Worldwide http://dlpdev.theps.net/ListOfExistingDlpNodes From boxbackup at fluffy.co.uk Thu Apr 29 15:16:38 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Thu, 29 Apr 2004 10:16:38 -0400 (EDT) Subject: [Box Backup] New encryption format & Space usage In-Reply-To: <20040429131051.GH28232@no-way.org> References: <20040429110002.25129.65772.Mailman@love.warhead.org.uk> <20040429131051.GH28232@no-way.org> Message-ID: Hi All, I actually get some of that info from my log files (I use logcheck so it emails me the output from the daemon). Some of what you want is syslogged. I was thinking though, that maybe it would be nice to have some sort of plugin facility so that others could write plugins so create that sort of report. The idea being that the plugin would be called on start of backup, for every file (with file stats, delta stats) and when the backup was finished. That way we could syslog or database-log however much data we wanted to keep. Ben wouldn't have to do all the work, and we'd all have more flexibility. Rick On Thu, 29 Apr 2004, Flavio Curti wrote: > Hi, me again ;) > > On Thu, Apr 29, 2004 at 09:04:52AM -0400, rprice at freeshell.org wrote: > > but what I was really hoping for was a list of files (and their sizes) > > that were being backed up. > I second that. I'd like to know: > - when the last contact were > - how many files/bytes were transferred > - how many files were changed, new, deleted and so on > > Right now it's difficult to tell if the client really works... I was > biten many times by backup-clients not really working, so it'd be nice > if there was some way to gather some informations. ``bbackupctl stat'' > comes to mind. > > Thank you very much & Greetz > > Flavio > > -- > http://no-way.org/~fcu/ > > Mach mit bei der Community-Bibliothek > - In Zuerich/CH http://zurich.communitybooks.org/ > - Worldwide http://dlpdev.theps.net/ListOfExistingDlpNodes > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Thu Apr 29 15:30:57 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Thu, 29 Apr 2004 10:30:57 -0400 (EDT) Subject: [Box Backup] New encryption format & Space usage In-Reply-To: <9470308A-99E5-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> References: <20040429110002.25129.65772.Mailman@love.warhead.org.uk> <9470308A-99E5-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> Message-ID: On Thu, 29 Apr 2004, Ben Summers wrote: > > One thing thats annoying me lately (and it's not your fault of > > course), is > > trying to find out what exactly I'm backing up when the daemon runs. I > > tried the option in the config file that should tell you what's going > > on, > > but what I was really hoping for was a list of files (and their sizes) > > that were being backed up. > > Extended logging will give you some hints. How do I turn that on? > > > > > The problem behind this, is that I see some pretty huge numbers for > > amount > > of data backed up sometimes (and even for uploaded), and I feel that my > > machine should have been *quiescent* before the large amount of data > > was > > backed up. > > In the default configs, changes are uploaded 6 hours ahead. So it would > have to be quiescent for over 6 hours for the upload to make no > changes. Yea, this was pretty frequent though, I just can't believe as much data changed on the disk as it was seemingly uploading/patching. As I said, I think I found the culprit in my wife's work email account which gets spammed pretty heavily. I'm not doubting what your program is saying, but I'm rather curious as to what is going on that's changing all that data. > > > > > I also just hit the wall with the amount of data on your store, I think > > yesterday, and I don't think that I should be backing up enough stuff > > to > > cause that. I'm not saying I think your program is wrong, I'm just > > trying > > to figure out where the fat file(s) is/are so I can decide if I really > > want it/them backed up. (The script to notify me of the problem also > > had > > an error (Debian Testing), but I haven't had time to look at that yet. > > You are requested to test and possible change that by the config script > -- but let me know if you have a patch for it. I will send you a patch without question, it just might take me a day or so. > > > > > I started looking around last night, and I seemed to have some rather > > large files in my wife's email related folders, 'caught spam' (which > > only > > I check) where she gets a couple of hundred spam a day, and the files > > related to spamassassin. So I'm thinking they were the culprit, it may > > be > > that the file gets some big changes which spook the delta algorithm. > > Did you just upgrade the client to 0.05PLUS1B? It won't diff if the old > version of the file was written before you upgraded. > > > > > I'm thinking of using *find* to try and hunt down the large files in > > the > > directories, > > This is probably the best approach. Yea, I understand that you don't want to clutter up the program, I wasn't asking you to put it huge query tools, I just want a bit more data to help me track stuff like this down. I'm thinking the plugin idea might be good (as in a previous email). > > > but, I guess even some stats in the syslog would be helpful, > > IE number of files backed up, maybe even some size stats (smallest, > > largest, average - for file size and delta size), so I could try and > > decide if there were 1000 small files that were changed, or one huge > > file > > that somehow was changed. > > I'll put some more stats in. Thanks :). I watch the syslogs like a hawk, for me that's the most useful place for stats right now. > > > > > This sort of thing would be helpful for someone trying to manage their > > data on a third party data store (as you are for me). I'd be > > particularly > > interested in doing say trending analysis to find out what was > > happening, > > and to look for ominous changes. Or even to predict when I'd hit the > > wall, > > and need to *purchase* more space from the data store provider. > > In bbackupquery, type "usage" for space usage on the server. (New in > 0.05PLUS1B.) Or > > /usr/local/bin/bbackupquery usage quit > > as a simple one liner. > > (I've just increased your allocation a bit, BTW) Thanks, one of the first things I tried out was the new usage command in the client. I appreciate your upping my limit, just so you know though, my concern wasn't that you hadn't given me enough space, but _why_ I _had_ run out of space with a perfectly reasonable upper limit. > > > > > Part of my interest is also related to bandwidth usage, I'm on a ADSL > > connection, but some of my friends who would be backing up to me would > > be > > on dialup, being able to manage the bandwidth usage would be helpful, > > knowing what sort of changes were happening would help that way (you > > could > > try and have multiple backup sets or know *why* a huge upload > > happened). > > Other possible options would be deferring large delta's until a set > > time, > > while letting small changes through. Although that could be risky and > > complicated. > > I'm designing taking ADSL a minimum requirement for this system. That's > not to say it couldn't be used with dial-up, it's just it's not > designed for it. So far it seems to work admirably with dialup just so you know... Rick From boxbackup at fluffy.co.uk Thu Apr 29 15:33:54 2004 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Thu, 29 Apr 2004 10:33:54 -0400 (EDT) Subject: [Box Backup] New encryption format & Space usage In-Reply-To: References: <20040429110002.25129.65772.Mailman@love.warhead.org.uk> <20040429131051.GH28232@no-way.org> Message-ID: Sorry, I forgot to add, that all we would hope for from Ben, would be a plugin with the basic stats more or less as he has now. If someone wanted more, they could write it, and (if there were nice) submit it for inclusion as a third party module. Rick On Thu, 29 Apr 2004 rprice at freeshell.org wrote: > Hi All, > > I actually get some of that info from my log files (I use logcheck so it > emails me the output from the daemon). > > Some of what you want is syslogged. > > I was thinking though, that maybe it would be nice to have some sort of > plugin facility so that others could write plugins so create that sort of > report. > > The idea being that the plugin would be called on start of backup, for > every file (with file stats, delta stats) and when the backup was > finished. > > That way we could syslog or database-log however much data we wanted to > keep. Ben wouldn't have to do all the work, and we'd all have more > flexibility. > > Rick > > > > On Thu, 29 Apr 2004, Flavio Curti wrote: > > > Hi, me again ;) > > > > On Thu, Apr 29, 2004 at 09:04:52AM -0400, rprice at freeshell.org wrote: > > > but what I was really hoping for was a list of files (and their sizes) > > > that were being backed up. > > I second that. I'd like to know: > > - when the last contact were > > - how many files/bytes were transferred > > - how many files were changed, new, deleted and so on > > > > Right now it's difficult to tell if the client really works... I was > > biten many times by backup-clients not really working, so it'd be nice > > if there was some way to gather some informations. ``bbackupctl stat'' > > comes to mind. > > > > Thank you very much & Greetz > > > > Flavio > > > > -- > > http://no-way.org/~fcu/ > > > > Mach mit bei der Community-Bibliothek > > - In Zuerich/CH http://zurich.communitybooks.org/ > > - Worldwide http://dlpdev.theps.net/ListOfExistingDlpNodes > > _______________________________________________ > > boxbackup mailing list > > boxbackup at fluffy.co.uk > > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Thu Apr 29 15:39:20 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 29 Apr 2004 15:39:20 +0100 Subject: [Box Backup] Expiring files on the server In-Reply-To: <20040429141238.GJ28232@no-way.org> References: <20040429141238.GJ28232@no-way.org> Message-ID: <000D0F54-99EB-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> On 29 Apr 2004, at 15:12, Flavio Curti wrote: > Hi, me again. > > I'm just checking my backupclient servers, and I noticed that one is no > longer backing up, because it used all his space on the backup storage: > sudo-root at gnu:/tmp# /usr/local/bin/bbstoreaccounts info 7 > > sudo-root at gnu:/tmp# /usr/local/bin/bbstoreaccounts info 7 > Account ID: 00000007 > Last object ID: 1408526 > Blocks used: 6563819 (51279.84Mb) > Blocks used by old files: 0 (0.00Mb) > Blocks used by deleted files: 0 (0.00Mb) > Blocks used by directories: 265033 (2070.57Mb) > Block soft limit: 6553600 (51200.00Mb) > Block hard limit: 7208960 (56320.00Mb) > Client store marker: 1083125666000000 > > (notice how the hard limit is not reached?) You're not supposed to reach the hard limit. Housekeeping attempts to keep you just below the soft limit, and the client won't upload files if the soft limit is exceeded when it connects. Errors will happen if it uploads a file which takes it over the hard limit. > > On the client I get this: > Apr 29 16:02:53 elefant bbackupd[23374]: Exceeded storage limits on > server -- not uploading changes to files > Apr 29 16:02:53 elefant bbackupd[23374]: Exceeded storage limits on > server -- not uploading changes to files > > and this for the last 3 days, which is not good... Didn't it email you? > How is this gonna work now? Shouldn't the store daemon regularly > delete > some files? For me it's actually more important to have the now > back-upped instead of old versions... There are no old versions to delete: > Blocks used by old files: 0 (0.00Mb) Therefore, the only thing that can happen is that an administrator changes the store allocation. Ben From boxbackup at fluffy.co.uk Thu Apr 29 15:46:46 2004 From: boxbackup at fluffy.co.uk (Flavio Curti) Date: Thu, 29 Apr 2004 16:46:46 +0200 Subject: [Box Backup] Expiring files on the server In-Reply-To: <000D0F54-99EB-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> References: <20040429141238.GJ28232@no-way.org> <000D0F54-99EB-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> Message-ID: <20040429144646.GK28232@no-way.org> Hi Ben On Thu, Apr 29, 2004 at 03:39:20PM +0100, Ben Summers wrote: > On 29 Apr 2004, at 15:12, Flavio Curti wrote: > >and this for the last 3 days, which is not good... > Didn't it email you? I'd have to check... > >Blocks used by old files: 0 (0.00Mb) > Therefore, the only thing that can happen is that an administrator > changes the store allocation. What is an old file? The source currently has 38GB, so I figured 50GB should be enough to keep one version and some diffs... Thank you very much & greetz Flavio -- http://no-way.org/~fcu/ Mach mit bei der Community-Bibliothek - In Zuerich/CH http://zurich.communitybooks.org/ - Worldwide http://dlpdev.theps.net/ListOfExistingDlpNodes From boxbackup at fluffy.co.uk Thu Apr 29 15:50:16 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 29 Apr 2004 15:50:16 +0100 Subject: [Box Backup] New encryption format & Space usage In-Reply-To: References: <20040429110002.25129.65772.Mailman@love.warhead.org.uk> <9470308A-99E5-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> Message-ID: <873DFE96-99EC-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> On 29 Apr 2004, at 15:30, rprice at freeshell.org wrote: > > > On Thu, 29 Apr 2004, Ben Summers wrote: > >>> One thing thats annoying me lately (and it's not your fault of >>> course), is >>> trying to find out what exactly I'm backing up when the daemon runs. >>> I >>> tried the option in the config file that should tell you what's going >>> on, >>> but what I was really hoping for was a list of files (and their >>> sizes) >>> that were being backed up. >> >> Extended logging will give you some hints. > > How do I turn that on? vi /etc/box/bbackupd.conf Uncomment the ExtendedLogging line... > >> >>> >>> The problem behind this, is that I see some pretty huge numbers for >>> amount >>> of data backed up sometimes (and even for uploaded), and I feel that >>> my >>> machine should have been *quiescent* before the large amount of data >>> was >>> backed up. >> >> In the default configs, changes are uploaded 6 hours ahead. So it >> would >> have to be quiescent for over 6 hours for the upload to make no >> changes. > > Yea, this was pretty frequent though, I just can't believe as much data > changed on the disk as it was seemingly uploading/patching. As I said, > I > think I found the culprit in my wife's work email account which gets > spammed pretty heavily. I'm not doubting what your program is saying, > but > I'm rather curious as to what is going on that's changing all that > data. It would be nice to know. [snip] >> I'm designing taking ADSL a minimum requirement for this system. >> That's >> not to say it couldn't be used with dial-up, it's just it's not >> designed for it. > > So far it seems to work admirably with dialup just so you know... It's just a matter of bandwidth really. And auto-dial. :-) On 29 Apr 2004, at 15:16, rprice at freeshell.org wrote: > Hi All, > > I actually get some of that info from my log files (I use logcheck so > it > emails me the output from the daemon). > > Some of what you want is syslogged. > > I was thinking though, that maybe it would be nice to have some sort of > plugin facility so that others could write plugins so create that sort > of > report. > > The idea being that the plugin would be called on start of backup, for > every file (with file stats, delta stats) and when the backup was > finished. > > That way we could syslog or database-log however much data we wanted to > keep. Ben wouldn't have to do all the work, and we'd all have more > flexibility. > Hmmm. So add a config line: PipeActionsToProcess = /path/to/utility When it starts to sync, it runs this utility. When it does something interesting, it writes a line to stdin which describes what it did. Maybe not a bad idea, and probably wouldn't take too long to implement. Ben From boxbackup at fluffy.co.uk Thu Apr 29 16:02:36 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Thu, 29 Apr 2004 16:02:36 +0100 Subject: [Box Backup] Expiring files on the server In-Reply-To: <20040429144646.GK28232@no-way.org> References: <20040429141238.GJ28232@no-way.org> <000D0F54-99EB-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> <20040429144646.GK28232@no-way.org> Message-ID: <4029FCF4-99EE-11D8-9FC1-000A95AFF7F8@fluffy.co.uk> On 29 Apr 2004, at 15:46, Flavio Curti wrote: > Hi Ben > > On Thu, Apr 29, 2004 at 03:39:20PM +0100, Ben Summers wrote: >> On 29 Apr 2004, at 15:12, Flavio Curti wrote: >>> and this for the last 3 days, which is not good... >> Didn't it email you? > I'd have to check... > >>> Blocks used by old files: 0 (0.00Mb) >> Therefore, the only thing that can happen is that an administrator >> changes the store allocation. > What is an old file? A version of a file which has been replaced by a newer version of the same file. These (and files which have been deleted) get removed by housekeeping until the size of the store is below the soft limit. > The source currently has 38GB, so I figured 50GB > should be enough to keep one version and some diffs... Is you're using raid on the server, and the files don't compress well, then you would get up to 50% expansion on the server. If you use bbackupquery and list with the '-s' option, it'll tell you how many blocks each file uses. And when you do get old versions, currently they aren't stored as diffs, but full files. I have come up with a nice plan to change this situation, but need to get around to implementing it. Ben From boxbackup at fluffy.co.uk Fri Apr 30 16:47:15 2004 From: boxbackup at fluffy.co.uk (Flavio Curti) Date: Fri, 30 Apr 2004 17:47:15 +0200 Subject: [Box Backup] boxbackup on 64bit systems (= on alpha) In-Reply-To: References: <20040425230207.GD28232@no-way.org> Message-ID: <20040430154715.GM28232@no-way.org> Hi Ben On Mon, Apr 26, 2004 at 09:57:28AM +0100, Ben Summers wrote: > It should run. Is there anything in your logs? Apr 30 17:31:29 droot bbackupd[12063]: Starting daemon (config: /etc/box/bbackupd.conf) (version 0.05PLUS1B) Apr 30 17:31:29 droot bbackupd[12063]: TRACE: Set maximum diffing time to 20 seconds Apr 30 17:31:29 droot bbackupd[12063]: TRACE: Exception thrown: CommonException(AssertFailed) at BoxTime.cpp(65) Apr 30 17:31:29 droot bbackupd[12063]: TRACE: Block 200c8020 freed, but not known. Error? Or allocated in startup static allocation? time_t is 64bit on this machine, I changed this in the source into uint_64, and it does start now... but it just hangs there and does nothing, in the Logs: Apr 30 17:32:48 droot bbackupd[12141]: Starting daemon (config: /etc/box/bbackupd.conf) (version 0.05PLUS1B) Apr 30 17:32:48 droot bbackupd[12141]: TRACE: Set maximum diffing time to 20 seconds Apr 30 17:32:48 droot bbackupd[12141]: Beginning scan of local files Apr 30 17:32:48 droot bbackupd[12141]: Opening connection to server gnu.backup.cyberlink.ch... Apr 30 17:32:48 droot bbackupd[12141]: TRACE: Send block allocation size is 4 Apr 30 17:32:48 droot bbackupd[12141]: Send Version(0x1) Apr 30 17:32:48 droot bbackupd[12141]: Receive Version(0x1) Apr 30 17:32:48 droot bbackupd[12141]: Send Login(0x10,0x0) Apr 30 17:32:48 droot bbackupd[12141]: Receive LoginConfirmed(0x0,0x6,0x80000,0xa0000) Apr 30 17:32:48 droot bbackupd[12141]: Connection made, login successful Apr 30 17:32:48 droot bbackupd[12141]: Send ListDirectory(0x1,0x2,0xc,false) Apr 30 17:32:48 droot bbackupd[12141]: Receive Success(0x1) Apr 30 17:32:48 droot bbackupd[12141]: Receiving stream, size 136 Apr 30 17:32:48 droot bbackupd[12141]: TRACE: Found mount point at / Apr 30 17:32:48 droot bbackupd[12141]: TRACE: Found mount point at /proc Apr 30 17:32:48 droot bbackupd[12141]: TRACE: Found mount point at /dev/pts Apr 30 17:32:48 droot bbackupd[12141]: TRACE: Found mount point at /dev/shm Apr 30 17:32:48 droot bbackupd[12141]: TRACE: new location nothing more after that... strace is not possible??! > The errors you've listed all seem to be just warnings which are fairly > harmless, although it'd be nice to fix them. Which compiler are you > using? (output of gcc -v) Reading specs from /usr/lib/gcc-lib/alpha-linux/3.3.3/specs Configured with: ../src/configure -v --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared --with-system-zlib --enable-nls --without-included-gettext --enable-__cxa_atexit --enable-clocale=gnu --enable-debug --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc alpha-linux Thread model: posix gcc version 3.3.3 (Debian 20040401) > perl ./runtest.pl ALL release I'm just running that one. If you are interested I'll send you the output... Thank you & Hope it helps Flavio -- http://no-way.org/~fcu/ Mach mit bei der Community-Bibliothek - In Zuerich/CH http://zurich.communitybooks.org/ - Worldwide http://dlpdev.theps.net/ListOfExistingDlpNodes From boxbackup at fluffy.co.uk Fri Apr 30 22:01:58 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 30 Apr 2004 22:01:58 +0100 Subject: [Box Backup] boxbackup on 64bit systems (= on alpha) In-Reply-To: <20040430154715.GM28232@no-way.org> References: <20040425230207.GD28232@no-way.org> <20040430154715.GM28232@no-way.org> Message-ID: <9E5BA7F4-9AE9-11D8-90F8-000A95AFF7F8@fluffy.co.uk> On 30 Apr 2004, at 16:47, Flavio Curti wrote: > Hi Ben > > On Mon, Apr 26, 2004 at 09:57:28AM +0100, Ben Summers wrote: >> It should run. Is there anything in your logs? > Apr 30 17:31:29 droot bbackupd[12063]: Starting daemon (config: > /etc/box/bbackupd.conf) (version 0.05PLUS1B) > Apr 30 17:31:29 droot bbackupd[12063]: TRACE: Set maximum diffing time > to 20 seconds > Apr 30 17:31:29 droot bbackupd[12063]: TRACE: Exception thrown: > CommonException(AssertFailed) at BoxTime.cpp(65) > Apr 30 17:31:29 droot bbackupd[12063]: TRACE: Block 200c8020 freed, but > not known. Error? Or allocated in startup static allocation? > > time_t is 64bit on this machine, I changed this in the source into > uint_64, and it does start now... Interesting difference, will get that sorted out. > but it just hangs there and does > nothing, in the Logs: > Apr 30 17:32:48 droot bbackupd[12141]: Starting daemon (config: > [snip] > Apr 30 17:32:48 droot bbackupd[12141]: TRACE: new location > > nothing more after that... strace is not possible??! It's weird that it should stop there. Is it using up any processor time, or just doing nothing? > >> The errors you've listed all seem to be just warnings which are fairly >> harmless, although it'd be nice to fix them. Which compiler are you >> using? (output of gcc -v) > Reading specs from /usr/lib/gcc-lib/alpha-linux/3.3.3/specs > Configured with: ../src/configure -v > --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang > --prefix=/usr > --mandir=/usr/share/man --infodir=/usr/share/info > --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared > --with-system-zlib --enable-nls --without-included-gettext > --enable-__cxa_atexit --enable-clocale=gnu --enable-debug > --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc > alpha-linux > Thread model: posix > gcc version 3.3.3 (Debian 20040401) > >> perl ./runtest.pl ALL release > I'm just running that one. If you are interested I'll send you the > output... That would be useful, thanks. It may be a little while before I can do anything with these results, though. Ben From boxbackup at fluffy.co.uk Fri Apr 30 22:06:43 2004 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Fri, 30 Apr 2004 22:06:43 +0100 Subject: [Box Backup] A success story Message-ID: <489AF048-9AEA-11D8-90F8-000A95AFF7F8@fluffy.co.uk> A couple of months ago, I made a backup server for a colleague. This was just an OpenBSD box running Samba, and Box Backup. This ran very happily, until last saturday when, with this particular person's usual bad luck with computer hardware, the hard disc failed. They don't make hard discs like they used to. This was a brand new Maxtor one. I'm not sure how surprised I am, given the price of the things. But old hard disc seem to go on for ever, and new ones fail. Anyway, I grabbed another hard disc, installed OpenBSD on it, and then left it restoring overnight. In the morning, it was as if nothing had happened. He's had it for a good few days now, so would have complained if anything was missing or corrupted. I hope none of you have had cause to restore from backups when you actually need them, but it's nice to know that when something did go wrong, the system worked. I was extremely glad it worked, I can tell you. Ben