From boxbackup at fluffy.co.uk Tue Jul 3 13:40:16 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Tue, 03 Jul 2007 14:40:16 +0200 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> Message-ID: <468A43B0.2010407@kontrapunkt.com> Hello Chris. These are my results: Code: Checked out revision 1719. ./bootstrap ./configure ...... ...... Regular expressions: yes Large files: yes Berkeley DB: yes Readline: yes Extended attributes: yes make ./runtest.pl ALL ...... ...... Skipping tests. *********** Server: ../../bin/bbstored/bbstored testfiles/bbstored.conf FAILURE: Couldn't start server at ../../lib/common/Test.h(224) FAILURE: Condition [bbstored_pid != -1 && bbstored_pid != 0] failed at testbbackupd.cpp(425) FAILED: 2 tests failed (first at ../../lib/common/Test.h:224) -------- common: PASSED crypto: PASSED compress: PASSED raidfile: PASSED basicserver: FAILED: 8 tests failed (first at ../../lib/common/Test.h:224) backupstore: FAILED: 2 tests failed (first at ../../lib/common/Test.h:224) backupstorefix: FAILED: 2 tests failed (first at ../../lib/common/Test.h:224) backupstorepatch: FAILED: 2 tests failed (first at ../../lib/common/Test.h:224) backupdiff: PASSED bbackupd: FAILED: 2 tests failed (first at ../../lib/common/Test.h:224) cd debug/test/basicserver (Im not shure which bbstored.conf file you wanted me to use) ../../bin/bbstored/bbstored -DVT /home/box/merge/debug/test/bbackupd/testfiles/bbstored.conf getopt: returned 'D' getopt: optind = 1 getopt: optopt = 0 getopt: optarg = getopt: argv[optind] = -DVT getopt: returned 'V' getopt: optind = 1 getopt: optopt = 0 getopt: optarg = getopt: argv[optind] = -DVT getopt: returned 'T' getopt: optind = 2 getopt: optopt = 0 getopt: optarg = getopt: argv[optind] = /home/box/merge/debug/test/bbackupd/testfiles/bbstored.conf 14:41:03 getopt: returned '?' 14:41:03 getopt: optind = 2 14:41:03 getopt: optopt = 0 14:41:03 getopt: optarg = 14:41:03 getopt: argv[optind] = /home/box/merge/debug/test/bbackupd/testfiles/bbstored.conf 14:41:03 Unknown error in getopt: returned '?' Thanks, Tobias Chris Wilson wrote: > Hi Tobias, > > On Fri, 29 Jun 2007, Tobias Balle-Petersen wrote: > >> 10. /root/merge/release/bin/bbstored/bbstored -DVT >> testfiles/bbstored.conf >> 09:50:04 Unknown error in getopt: returned '?' > > I've added some extra debugging to chris/merge. Please could you update > to the latest version and try this again, and send me the complete output? > > Cheers, Chris. From boxbackup at fluffy.co.uk Tue Jul 3 20:05:25 2007 From: boxbackup at fluffy.co.uk (Dave Bamford) Date: Tue, 03 Jul 2007 20:05:25 +0100 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: <468A43B0.2010407@kontrapunkt.com> References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> Message-ID: <468A9DF5.7010901@logical-progress.com> Hi Tobias Can you tell me what packages you installed to get yes for all the below. I can't seem to get Berkley DB support installed for Debian etch. Regards Dave Bamford Tobias Balle-Petersen wrote: > Hello Chris. > > > These are my results: > > Code: > Checked out revision 1719. > > ./bootstrap > > ./configure > ...... > ...... > Regular expressions: yes > Large files: yes > Berkeley DB: yes > Readline: yes > Extended attributes: yes > > > make > > ./runtest.pl ALL > ...... > ...... > Skipping tests. > *********** > Server: ../../bin/bbstored/bbstored testfiles/bbstored.conf > FAILURE: Couldn't start server at ../../lib/common/Test.h(224) > FAILURE: Condition [bbstored_pid != -1 && bbstored_pid != 0] failed at testbbackupd.cpp(425) > FAILED: 2 tests failed (first at ../../lib/common/Test.h:224) > -------- > common: PASSED > crypto: PASSED > compress: PASSED > raidfile: PASSED > basicserver: FAILED: 8 tests failed (first at ../../lib/common/Test.h:224) > backupstore: FAILED: 2 tests failed (first at ../../lib/common/Test.h:224) > backupstorefix: FAILED: 2 tests failed (first at ../../lib/common/Test.h:224) > backupstorepatch: FAILED: 2 tests failed (first at ../../lib/common/Test.h:224) > backupdiff: PASSED > bbackupd: FAILED: 2 tests failed (first at ../../lib/common/Test.h:224) > > > > cd debug/test/basicserver > > > (Im not shure which bbstored.conf file you wanted me to use) > ../../bin/bbstored/bbstored -DVT /home/box/merge/debug/test/bbackupd/testfiles/bbstored.conf > getopt: returned 'D' > getopt: optind = 1 > getopt: optopt = 0 > getopt: optarg = > getopt: argv[optind] = -DVT > getopt: returned 'V' > getopt: optind = 1 > getopt: optopt = 0 > getopt: optarg = > getopt: argv[optind] = -DVT > getopt: returned 'T' > getopt: optind = 2 > getopt: optopt = 0 > getopt: optarg = > getopt: argv[optind] = /home/box/merge/debug/test/bbackupd/testfiles/bbstored.conf > 14:41:03 getopt: returned '?' > 14:41:03 getopt: optind = 2 > 14:41:03 getopt: optopt = 0 > 14:41:03 getopt: optarg = > 14:41:03 getopt: argv[optind] = /home/box/merge/debug/test/bbackupd/testfiles/bbstored.conf > 14:41:03 Unknown error in getopt: returned '?' > > > > Thanks, > Tobias > > > > > > Chris Wilson wrote: > >> Hi Tobias, >> >> On Fri, 29 Jun 2007, Tobias Balle-Petersen wrote: >> >> >>> 10. /root/merge/release/bin/bbstored/bbstored -DVT >>> testfiles/bbstored.conf >>> 09:50:04 Unknown error in getopt: returned '?' >>> >> I've added some extra debugging to chris/merge. Please could you update >> to the latest version and try this again, and send me the complete output? >> >> Cheers, Chris. >> > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > > From boxbackup at fluffy.co.uk Tue Jul 3 22:16:04 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 3 Jul 2007 22:16:04 +0100 (BST) Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: <468A43B0.2010407@kontrapunkt.com> References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> Message-ID: Hi Tobias and all, On Tue, 3 Jul 2007, Tobias Balle-Petersen wrote: > 14:41:03 getopt: returned '?' > 14:41:03 getopt: optind = 2 > 14:41:03 getopt: optopt = 0 > 14:41:03 getopt: optarg = > 14:41:03 getopt: argv[optind] = /home/box/merge/debug/test/bbackupd/testfiles/bbstored.conf > 14:41:03 Unknown error in getopt: returned '?' When is a question mark not a question mark? Does anyone know why the case '?' could be ignored (skipped over) in this case, to hit the select statement's default clause? Tobias, please could you try the following: Don't delete anything or run any other commands :-) cd merge svn update cd bin/bbstored make cd ../../debug/test/bbackupd ../../bin/bbstored/bbstored -DVT testfiles/bbstored.conf and send me the output? This probably won't fix it, but the output should help to understand what on earth is going on here. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Tue Jul 3 23:18:46 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 3 Jul 2007 23:18:46 +0100 (BST) Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: <467FB654.2070107@kontrapunkt.com> References: <467FB654.2070107@kontrapunkt.com> Message-ID: Hi Tobias, On Mon, 25 Jun 2007, Tobias Balle-Petersen wrote: > I'm trying to compile the latest chris-merge of boxbackup on a Debian > PPC box but I cant make it work. Can you give me some more info about that debian PPC box? What kind of PPC (brand name and model) and what kind of Debian (Etch? Woody?) Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Sat Jul 7 15:05:25 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Sat, 07 Jul 2007 16:05:25 +0200 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> Message-ID: <468F9DA5.5020109@kontrapunkt.com> Hello Chris. Here are the results: # svn update U lib/server/Daemon.cpp Updated to revision 1725. # cd bin/bbstored # make # cd ../../debug/test/bbackupd # ../../bin/bbstored/bbstored -DVT testfiles/bbstored.conf 16:05:23 Starting daemon, version chris_merge_1719, config: testfiles/bbstored.conf So, it starts up doing this?? Tobias Chris Wilson wrote: > Hi Tobias and all, > > On Tue, 3 Jul 2007, Tobias Balle-Petersen wrote: > >> 14:41:03 getopt: returned '?' >> 14:41:03 getopt: optind = 2 >> 14:41:03 getopt: optopt = 0 >> 14:41:03 getopt: optarg = >> 14:41:03 getopt: argv[optind] = >> /home/box/merge/debug/test/bbackupd/testfiles/bbstored.conf >> 14:41:03 Unknown error in getopt: returned '?' > > When is a question mark not a question mark? Does anyone know why the > case '?' could be ignored (skipped over) in this case, to hit the select > statement's default clause? > > Tobias, please could you try the following: > > Don't delete anything or run any other commands :-) > > cd merge > svn update > cd bin/bbstored > make > cd ../../debug/test/bbackupd > ../../bin/bbstored/bbstored -DVT testfiles/bbstored.conf > > and send me the output? > > This probably won't fix it, but the output should help to understand > what on earth is going on here. > > Cheers, Chris. From boxbackup at fluffy.co.uk Sat Jul 7 15:06:24 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Sat, 07 Jul 2007 16:06:24 +0200 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: References: <467FB654.2070107@kontrapunkt.com> Message-ID: <468F9DE0.9060001@kontrapunkt.com> Hello Chris. It's an Apple G4 with Debian Etch. Tobias Chris Wilson wrote: > Hi Tobias, > > On Mon, 25 Jun 2007, Tobias Balle-Petersen wrote: > >> I'm trying to compile the latest chris-merge of boxbackup on a Debian >> PPC box but I cant make it work. > > Can you give me some more info about that debian PPC box? What kind of > PPC (brand name and model) and what kind of Debian (Etch? Woody?) > > Cheers, Chris. From boxbackup at fluffy.co.uk Sat Jul 7 15:48:05 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sat, 7 Jul 2007 15:48:05 +0100 (BST) Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: <468F9DA5.5020109@kontrapunkt.com> References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> Message-ID: Hi Tobias, > # ../../bin/bbstored/bbstored -DVT testfiles/bbstored.conf 16:05:23 > Starting daemon, version chris_merge_1719, config: > testfiles/bbstored.conf > > So, it starts up doing this?? Good, I think that means that I fixed it :-) Could you try running the tests again? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Sat Jul 7 16:49:46 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Sat, 07 Jul 2007 17:49:46 +0200 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> Message-ID: <468FB61A.5060805@kontrapunkt.com> Hello Chris. Do you want me to delete everything and do it from scratch? Tobias Chris Wilson wrote: > Hi Tobias, > >> # ../../bin/bbstored/bbstored -DVT testfiles/bbstored.conf 16:05:23 >> Starting daemon, version chris_merge_1719, config: >> testfiles/bbstored.conf >> >> So, it starts up doing this?? > > Good, I think that means that I fixed it :-) > > Could you try running the tests again? > > Cheers, Chris. From boxbackup at fluffy.co.uk Sat Jul 7 17:20:46 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sat, 7 Jul 2007 17:20:46 +0100 (BST) Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: <468FB61A.5060805@kontrapunkt.com> References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> Message-ID: Hi Tobias, > Do you want me to delete everything and do it from scratch? Yes please, if you don't mind. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Sat Jul 7 19:44:44 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Sat, 07 Jul 2007 20:44:44 +0200 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> Message-ID: <468FDF1C.4060709@kontrapunkt.com> Helo Chris. My results: As user root. Stop running bbstored 0.10 Delete /var/run/bbstored.pid # svn co http://bbdev.fluffy.co.uk/svn/box/chris/merge/ Checked out revision 1725. # ./bootstrap # ./configure Regular expressions: yes Large files: yes Berkeley DB: yes Readline: yes Extended attributes: yes # make # ./runtest ALL common: PASSED crypto: PASSED compress: PASSED raidfile: PASSED basicserver: PASSED backupstore: PASSED backupstorefix: PASSED backupstorepatch: PASSED backupdiff: FAILED: 1 tests failed (first at testbackupdiff.cpp:548) bbackupd: PASSED Trying to run the tests again returns: common: PASSED crypto: PASSED compress: PASSED raidfile: PASSED basicserver: PASSED backupstore: FAILED: 2 tests failed (first at ../../lib/common/Test.h:278) backupstorefix: PASSED backupstorepatch: PASSED backupdiff: FAILED: 1 tests failed (first at testbackupdiff.cpp:548) bbackupd: PASSED What could the reasons for the backupdiff failure be? I have a slow/older machine. # cat /proc/cpuinfo processor : 0 cpu : 7410, altivec supported temperature : 1-76 C (uncalibrated) clock : 533.333332MHz revision : 0.3 (pvr 800c 1103) bogomips : 66.30 timebase : 33290001 platform : PowerMac machine : PowerMac3,4 motherboard : PowerMac3,4 MacRISC2 MacRISC Power Macintosh detected as : 69 (PowerMac G4 Silver) pmac flags : 00000010 L2 cache : 1024K unified pmac-generation : NewWorld Thanks, Tobias Chris Wilson wrote: > Hi Tobias, > >> Do you want me to delete everything and do it from scratch? > > Yes please, if you don't mind. > > Cheers, Chris. From boxbackup at fluffy.co.uk Sat Jul 7 19:49:34 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sat, 7 Jul 2007 19:49:34 +0100 (BST) Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: <468FDF1C.4060709@kontrapunkt.com> References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> Message-ID: Hi Tobias, > What could the reasons for the backupdiff failure be? I have a > slow/older machine. Yes, that's almost certainly it. The line that failed was the time test. > backupstore: FAILED: 2 tests failed (first at ../../lib/common/Test.h:278) I'm interested in this, can you run the backupstore test alone, without changing anything, and send me the output (maybe not to the list if it's too long). ./runtest backupstore Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Sat Jul 7 20:02:23 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Sat, 07 Jul 2007 21:02:23 +0200 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> Message-ID: <468FE33F.1090400@kontrapunkt.com> Hello Chris. The result of tail ./runtest.pl backupstore: (cd ../../lib/common; make NODEPS=1) make[1]: Entering directory `/home/box/merge/lib/common' make[1]: `../../debug/lib/common/common.a' is up to date. make[1]: Leaving directory `/home/box/merge/lib/common' (cd ../../lib/compress; make NODEPS=1) make[1]: Entering directory `/home/box/merge/lib/compress' make[1]: `../../debug/lib/compress/compress.a' is up to date. make[1]: Leaving directory `/home/box/merge/lib/compress' (cd ../../lib/crypto; make NODEPS=1) make[1]: Entering directory `/home/box/merge/lib/crypto' make[1]: `../../debug/lib/crypto/crypto.a' is up to date. make[1]: Leaving directory `/home/box/merge/lib/crypto' (cd ../../lib/server; make NODEPS=1) make[1]: Entering directory `/home/box/merge/lib/server' make[1]: `../../debug/lib/server/server.a' is up to date. make[1]: Leaving directory `/home/box/merge/lib/server' (cd ../../lib/backupclient; make NODEPS=1) make[1]: Entering directory `/home/box/merge/lib/backupclient' make[1]: `../../debug/lib/backupclient/backupclient.a' is up to date. make[1]: Leaving directory `/home/box/merge/lib/backupclient' (cd ../../lib/raidfile; make NODEPS=1) make[1]: Entering directory `/home/box/merge/lib/raidfile' make[1]: `../../debug/lib/raidfile/raidfile.a' is up to date. make[1]: Leaving directory `/home/box/merge/lib/raidfile' (cd ../../lib/backupstore; make NODEPS=1) make[1]: Entering directory `/home/box/merge/lib/backupstore' make[1]: `../../debug/lib/backupstore/backupstore.a' is up to date. make[1]: Leaving directory `/home/box/merge/lib/backupstore' (cd ../../bin/bbstoreaccounts; make NODEPS=1) make[1]: Entering directory `/home/box/merge/bin/bbstoreaccounts' make[1]: Nothing to be done for `all'. make[1]: Leaving directory `/home/box/merge/bin/bbstoreaccounts' (cd ../../bin/bbstored; make NODEPS=1) make[1]: Entering directory `/home/box/merge/bin/bbstored' make[1]: Nothing to be done for `all'. make[1]: Leaving directory `/home/box/merge/bin/bbstored' TEST: test/backupstore Killing any running daemons... Removing old test files... Copying new test files... Running test backupstore in debug mode... Allocating filename encoding/decoding buffer with size 38 Reallocating filename encoding/decoding buffer from 38 to 40 Reallocating filename encoding/decoding buffer from 40 to 41 Reallocating filename encoding/decoding buffer from 41 to 46 Block 1010f858 realloc(), but not in list. Error? Or allocated in startup static objects? Reallocating filename encoding/decoding buffer from 46 to 51 Reallocating filename encoding/decoding buffer from 51 to 54 Reallocating filename encoding/decoding buffer from 54 to 428 Reallocating filename encoding/decoding buffer from 428 to 438 Reallocating EncodingBuffer from 53 to 256 Reallocating EncodingBuffer from 256 to 635 Reallocating EncodingBuffer from 367 to 2267 Read 4096 bytes at 4096, 6964 remain, eta 0s Reallocating EncodingBuffer from 1068 to 3244 Read 4096 bytes at 8192, 2868 remain, eta 0s Read 2868 bytes at 11060, 0 remain, eta 0s Read 4096 bytes at 4096, 4154 remain, eta 0s Reallocating EncodingBuffer from 1082 to 3258 Read 4154 bytes at 8250, 0 remain, eta 0s Trying to overwrite raidfile 0 test-info/info Obtained 8 stack frames. ./test [0x100a8bfc] ./test [0x1003b3a0] ./test [0x1001f3f4] ./test [0x10012d9c] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: RaidFileException(CannotOverwriteExistingFile) at RaidFileWrite.cpp(100) Obtained 7 stack frames. ./test [0x100a8bfc] ./test [0x1001e7d4] ./test [0x10013040] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: BackupStoreException(StoreInfoIsReadOnly) at BackupStoreInfo.cpp(291) Obtained 7 stack frames. ./test [0x100a8bfc] ./test [0x1001db28] ./test [0x100131dc] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: BackupStoreException(StoreInfoIsReadOnly) at BackupStoreInfo.cpp(366) Obtained 7 stack frames. ./test [0x100a8bfc] ./test [0x1001d728] ./test [0x10013358] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: BackupStoreException(StoreInfoIsReadOnly) at BackupStoreInfo.cpp(390) Obtained 7 stack frames. ./test [0x100a8bfc] ./test [0x1001d328] ./test [0x100134d4] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: BackupStoreException(StoreInfoIsReadOnly) at BackupStoreInfo.cpp(414) Obtained 7 stack frames. ./test [0x100a8bfc] ./test [0x1001df28] ./test [0x10013650] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: BackupStoreException(StoreInfoIsReadOnly) at BackupStoreInfo.cpp(510) Obtained 7 stack frames. ./test [0x100a8bfc] ./test [0x1001e3b0] ./test [0x100137cc] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: BackupStoreException(StoreInfoIsReadOnly) at BackupStoreInfo.cpp(490) Obtained 7 stack frames. ./test [0x100a8bfc] ./test [0x1001dcf8] ./test [0x10013b24] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: BackupStoreException(StoreInfoBlockDeltaMakesValueNegative) at BackupStoreInfo.cpp(370) Obtained 7 stack frames. ./test [0x100a8bfc] ./test [0x1001d8f8] ./test [0x10013ca0] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: BackupStoreException(StoreInfoBlockDeltaMakesValueNegative) at BackupStoreInfo.cpp(394) Obtained 7 stack frames. ./test [0x100a8bfc] ./test [0x1001d4f8] ./test [0x10013e1c] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: BackupSAccount 1234567 created toreException(StoreInfoBlockDeltaMakesValueNegative) at BackupStoreInfo.cpp(418) Obtained 7 stack frames. ./test [0x100a8bfc] ./test [0x1001e180] ./test [0x10014018] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: BackupStoreException(StoreInfoDirNotInList) at BackupStoreInfo.cpp(516) Starting server: ../../bin/bbstored/bbstored testfiles/bbstored.conf Waiting for server to start: done. Send block allocation size is 4 Protocol: Error received 1000/3 Obtained 8 stack frames. ./test [0x100a8bfc] ./test [0x100439cc] ./test [0x1001af28] ./test [0x10014b84] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1135) Send block allocation size is 4 Send block allocation size is 4 Protocol: Error received 1000/4 Obtained 9 stack frames. ./test [0x100a8bfc] ./test [0x100439cc] ./test [0x1001af28] ./test [0x1000b0ac] ./test [0x100150dc] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1135) Send block allocation size is 4 Read 4096 bytes at 4096, 6964 remain, eta 0s Reallocating EncodingBuffer from 1068 to 3244 Read 4096 bytes at 8192, 2868 remain, eta 0s Read 2868 bytes at 11060, 0 remain, eta 0s Protocol: Error received 1000/13 Obtained 10 stack frames. ./test [0x100a8bfc] ./test [0x10040db0] ./test [0x1001b948] ./test [0x1000a53c] ./test [0x1000b674] ./test [0x100150dc] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1516) testfiles/test0 Read 455 bytes at 455, 0 remain, eta 0s testfiles/test1 Read 2674 bytes at 2674, 0 remain, eta 0s Reallocating EncodingBuffer from 1068 to 3244 testfiles/test2 Read 4096 bytes at 4096, 129540 remain, eta 0s Reallocating EncodingBuffer from 1068 to 3244 Reallocating EncodingBuffer from 3244 to 5420 Read 4096 bytes at 8192, 125444 remain, eta 0s ...... ...... ...... Read 2564 bytes at 133636, 0 remain, eta 0s testfiles/test3 Read 4096 bytes at 4096, 2667 remain, eta 0s Reallocating EncodingBuffer from 1068 to 3244 Reallocating EncodingBuffer from 3244 to 5420 Read 2667 bytes at 6763, 0 remain, eta 0s testfiles/test4 Read 124 bytes at 124, 0 remain, eta 0s testfiles/test5 Read 1 bytes at 1, 0 remain, eta 0s testfiles/test6 testfiles/test7 Read 3246 bytes at 3246, 0 remain, eta 0s Reallocating EncodingBuffer from 1068 to 3244 Reallocating EncodingBuffer from 3244 to 4581 testfiles/test8 Read 3434 bytes at 3434, 0 remain, eta 0s Reallocating EncodingBuffer from 1068 to 3244 Reallocating EncodingBuffer from 3244 to 4769 testfiles/test9 Read 446 bytes at 446, 0 remain, eta 0s testfiles/testa Read 2466 bytes at 2466, 0 remain, eta 0s Reallocating EncodingBuffer from 1068 to 3244 testfiles/testb Read 4096 bytes at 4096, 466 remain, eta 0s Reallocating EncodingBuffer from 1068 to 3244 Reallocating EncodingBuffer from 3244 to 5420 Read 466 bytes at 4562, 0 remain, eta 0s testfiles/testc Read 234 bytes at 234, 0 remain, eta 0s testfiles/testd Read 435 bytes at 435, 0 remain, eta 0s 0 1 2 3 4 5 6 7 8 9 10 11 12 13 1183834633.450731: timer 0x7fb72480 initialised for 0 secs, will not fire 1183834633.501975: timer 0x7fb72480 destroyed Diff: 4109 new bytes found, 32 old blocks used Read 4109 bytes at 65549, 68100 remain, eta 0s Reallocating EncodingBuffer from 1071 to 3247 Reallocating EncodingBuffer from 3247 to 5423 Read 455 bytes at 455, 0 remain, eta 0s Protocol: Error received 1000/7 Obtained 9 stack frames. ./test [0x100a8bfc] ./test [0x10042b2c] ./test [0x1001bb68] ./test [0x1000ed0c] ./test [0x100150dc] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1259) Protocol: Error received 1000/10 Obtained 9 stack frames. ./test [0x100a8bfc] ./test [0x10042b2c] ./test [0x1001bb68] ./test [0x1000eee0] ./test [0x100150dc] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1259) Create subdirs, depth = 6, dirid = 16 Create subdirs, depth = 5, dirid = 17 Create subdirs, depth = 4, dirid = 18 Create subdirs, depth = 3, dirid = 19 Create subdirs, depth = 2, dirid = 1a Create subdirs, depth = 1, dirid = 1b Create subdirs, depth = 0, dirid = 1c Read 4096 bytes at 4096, 6964 remain, eta 0s Reallocating EncodingBuffer from 1068 to 3244 ...... ...... ...... Reallocating EncodingBuffer from 1068 to 3244 Read 4096 bytes at 8192, 2868 remain, eta 0s Read 2868 bytes at 11060, 0 remain, eta 0s Test for del: 16 Test for del: 17 ...... ...... ...... Test for del: 1f8 Test for del: 1fc Send block allocation size is 4 done. Limits on account 0x01234567 changed to 10 soft, 20000 hard Starting server: ../../bin/bbstored/bbstored testfiles/bbstored.conf Waiting for server to start: done. waiting for housekeeping: ..............................Limits on account 0x01234567 changed to 10 soft, 20 hard Send block allocation size is 4 Send block allocation size is 4 Read 4096 bytes at 4096, 2667 remain, eta 0s Reallocating EncodingBuffer from 1068 to 3244 Reallocating EncodingBuffer from 3244 to 5420 Read 2667 bytes at 6763, 0 remain, eta 0s Protocol: Error received 1000/11 Obtained 8 stack frames. ./test [0x100a8bfc] ./test [0x1004116c] ./test [0x1001b050] ./test [0x10015b74] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1485) Protocol: Error received 1000/11 Obtained 8 stack frames. ./test [0x100a8bfc] ./test [0x100423dc] ./test [0x1001b15c] ./test [0x10015de4] ./test [0x1001830c] ./test [0x10004d90] /lib/tls/libc.so.6 [0xf945994] /lib/tls/libc.so.6(__libc_start_main+0xb0) [0xf945ad0] Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1324) done. PASSED -------- backupstore: PASSED Thanks, Tobias Chris Wilson wrote: > I'm interested in this, can you run the backupstore test alone, without > changing anything, and send me the output (maybe not to the list if it's > too long). > > ./runtest backupstore > > Cheers, Chris. From boxbackup at fluffy.co.uk Sat Jul 7 20:16:42 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sat, 7 Jul 2007 20:16:42 +0100 (BST) Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: <468FE33F.1090400@kontrapunkt.com> References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> <468FE33F.1090400@kontrapunkt.com> Message-ID: Hi Tobias, Wee, that was a bit long :-) I'm not sure, but I have a sneaking feeling that I've seen this problem on Darwin (and therefore OSX) before, and that it relates to delayed deletion of files, which makes it a race condition and hard to pin down. I'd advise you not to run a mission-critical box backup server on this OSX box. However, I don't believe that this has changed since 0.10, so if you are already running servers on OSX then I don't see any particular reason not to upgrade them to this new version. Apart from that, all tests are fine in debug mode. Could you rerun in release mode please? ./runtest ALL release Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Sat Jul 7 20:57:14 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Sat, 07 Jul 2007 21:57:14 +0200 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> <468FE33F.1090400@kontrapunkt.com> Message-ID: <468FF01A.70804@kontrapunkt.com> Hello Chris. I'm running Debian on the machine in question. Not OS X. What was the problem you fixed that allows me to complete the tests now? I'll run a ./runtest.pl ALL release and post the results. Tobias Chris Wilson wrote: > Hi Tobias, > > Wee, that was a bit long :-) > > I'm not sure, but I have a sneaking feeling that I've seen this problem > on Darwin (and therefore OSX) before, and that it relates to delayed > deletion of files, which makes it a race condition and hard to pin down. > > I'd advise you not to run a mission-critical box backup server on this > OSX box. However, I don't believe that this has changed since 0.10, so > if you are already running servers on OSX then I don't see any > particular reason not to upgrade them to this new version. > > Apart from that, all tests are fine in debug mode. Could you rerun in > release mode please? > > ./runtest ALL release > > Cheers, Chris. From boxbackup at fluffy.co.uk Sat Jul 7 21:04:11 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sat, 7 Jul 2007 21:04:11 +0100 (BST) Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: <468FF01A.70804@kontrapunkt.com> References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> <468FE33F.1090400@kontrapunkt.com> <468FF01A.70804@kontrapunkt.com> Message-ID: Hi Tobias, > I'm running Debian on the machine in question. Not OS X. OK, sorry. That's interesting, if it's the same race condition then it must appear on Linux too. Will have to look out for it. > What was the problem you fixed that allows me to complete the tests now? I had getopt storing its result in a char, which should have been an int. Worked ok on most platforms, but not ARM or PowerPC. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Sat Jul 7 21:35:34 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Sat, 07 Jul 2007 22:35:34 +0200 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> <468FE33F.1090400@kontrapunkt.com> Message-ID: <468FF916.6050302@kontrapunkt.com> Hello Chris. Chris Wilson wrote: > Apart from that, all tests are fine in debug mode. Could you rerun in > release mode please? The results: WARNING: This test was run as root. Some tests have been omitted. done. PASSED -------- common: PASSED crypto: PASSED compress: PASSED raidfile: PASSED basicserver: PASSED backupstore: PASSED backupstorefix: PASSED backupstorepatch: PASSED backupdiff: PASSED bbackupd: PASSED backupdiff passes now?? Should I run as user box as well? Tobias From boxbackup at fluffy.co.uk Sat Jul 7 21:36:45 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Sat, 07 Jul 2007 22:36:45 +0200 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> <468FE33F.1090400@kontrapunkt.com> <468FF01A.70804@kontrapunkt.com> Message-ID: <468FF95D.3070101@kontrapunkt.com> Chris Wilson wrote: > I had getopt storing its result in a char, which should have been an > int. Worked ok on most platforms, but not ARM or PowerPC. So, this actually helped the project :) Thanks a LOT for all your help. Tobias From boxbackup at fluffy.co.uk Sat Jul 7 22:09:34 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sat, 7 Jul 2007 22:09:34 +0100 (BST) Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: <468FF95D.3070101@kontrapunkt.com> References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> <468FE33F.1090400@kontrapunkt.com> <468FF01A.70804@kontrapunkt.com> <468FF95D.3070101@kontrapunkt.com> Message-ID: Hi Tobias, > Chris Wilson wrote: >> I had getopt storing its result in a char, which should have been an >> int. Worked ok on most platforms, but not ARM or PowerPC. > > So, this actually helped the project :) Of course it did, otherwise I wouldn't have bothered :-) > Thanks a LOT for all your help. Thanks a lot for your patience and help :-) Cheers, Crhis. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Sat Jul 7 22:10:27 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sat, 7 Jul 2007 22:10:27 +0100 (BST) Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: <468FF916.6050302@kontrapunkt.com> References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> <468FE33F.1090400@kontrapunkt.com> <468FF916.6050302@kontrapunkt.com> Message-ID: Hi Tobias, > backupdiff passes now?? Maybe it takes about 40 seconds, so it doesn't always fail? maybe compiler optimisations make a difference? > Should I run as user box as well? Yes please, if you don't mind. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Sat Jul 7 22:35:17 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Sat, 07 Jul 2007 23:35:17 +0200 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> <468FE33F.1090400@kontrapunkt.com> <468FF916.6050302@kontrapunkt.com> Message-ID: <46900715.60604@kontrapunkt.com> Hello Chris. Changing ownership of files to box: chown -R box:box * Changing user to box: su box ./runtest.pl ALL release -------- common: output not found crypto: output not found compress: output not found raidfile: output not found basicserver: output not found backupstore: output not found backupstorefix: output not found backupstorepatch: output not found backupdiff: output not found bbackupd: output not found Hmm... There must be an error somewhere? I'm off to bed. Tobias Chris Wilson wrote: > Hi Tobias, > >> backupdiff passes now?? > > Maybe it takes about 40 seconds, so it doesn't always fail? maybe > compiler optimisations make a difference? > >> Should I run as user box as well? > > Yes please, if you don't mind. > > Cheers, Chris. From boxbackup at fluffy.co.uk Fri Jul 13 20:40:14 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 13 Jul 2007 20:40:14 +0100 (BST) Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: <468FDF1C.4060709@kontrapunkt.com> References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> Message-ID: Hi Tobias, On Sat, 7 Jul 2007, Tobias Balle-Petersen wrote: > Trying to run the tests again returns: > > common: PASSED > crypto: PASSED > compress: PASSED > raidfile: PASSED > basicserver: PASSED > backupstore: FAILED: 2 tests failed (first at ../../lib/common/Test.h:278) > backupstorefix: PASSED > backupstorepatch: PASSED > backupdiff: FAILED: 1 tests failed (first at testbackupdiff.cpp:548) > bbackupd: PASSED I've been running test/backupstore in a continuous loop for 5 days and I haven't found a single error. I don't understand what was wrong with this test on your machine as I can't reproduce it. I'd very much appreciate it if you use this version in production and let me know how it goes? Please don't forget to compare -a regularly (e.g. in an automatic scheduler such as cron) and report any unexpected compare failures. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Fri Jul 13 23:11:50 2007 From: boxbackup at fluffy.co.uk (James O'Gorman) Date: Fri, 13 Jul 2007 23:11:50 +0100 Subject: [Box Backup] OT: Server colocation Message-ID: <20070713221149.GC62388@netinertia.co.uk> Hi all, My colo provider have this evening announced they have been taken over by another company which will mean servers being moved from London Docklands to Maidenhead. This is far from ideal as 1) I don't drive, 2) it would take me at least three times as long to get to the DC. So, does anyone have any recomendations/experience with other colo providers in London? This will affect this project because if they do decide to move my server it could be offline for some time! James From boxbackup at fluffy.co.uk Fri Jul 13 23:16:16 2007 From: boxbackup at fluffy.co.uk (Stephen Nelson-Smith) Date: Fri, 13 Jul 2007 23:16:16 +0100 Subject: [Box Backup] OT: Server colocation In-Reply-To: <20070713221149.GC62388@netinertia.co.uk> References: <20070713221149.GC62388@netinertia.co.uk> Message-ID: On 7/13/07, James O'Gorman wrote: > Hi all, > > My colo provider have this evening announced they have been taken over > by another company which will mean servers being moved from London > Docklands to Maidenhead. Who's that then? > So, does anyone have any recomendations/experience with other colo > providers in London? Several - depends on what you want - I've used several in the past, am using one n particular at the moment, and have some other contacts too. How much are you paying? What's the bandwidth cost? How much power do you need? > James S. From boxbackup at fluffy.co.uk Fri Jul 13 23:27:38 2007 From: boxbackup at fluffy.co.uk (James O'Gorman) Date: Fri, 13 Jul 2007 23:27:38 +0100 Subject: [Box Backup] OT: Server colocation In-Reply-To: References: <20070713221149.GC62388@netinertia.co.uk> Message-ID: <20070713222737.GD62388@netinertia.co.uk> On Fri, Jul 13, 2007 at 11:16:16PM +0100, Stephen Nelson-Smith wrote: > On 7/13/07, James O'Gorman wrote: > > Hi all, > > > > My colo provider have this evening announced they have been taken over > > by another company which will mean servers being moved from London > > Docklands to Maidenhead. > > Who's that then? Black Cat Networks. I would post a link to the email but the archives are subscribers only. > > So, does anyone have any recomendations/experience with other colo > > providers in London? > > Several - depends on what you want - I've used several in the past, am > using one n particular at the moment, and have some other contacts > too. How much are you paying? What's the bandwidth cost? How much > power do you need? I was originally paying ?50/month with BCN for a 1U box, 100GB transfer (50p for each GB used over that), up to 0.5A, serial console, remote power cycle, as my IPv4 addresses as one can justify and an IPv6 /64. They bumped their prices up to ?75/month sometime last year due to RedBus doing so. Something similar would be nice :-) James From boxbackup at fluffy.co.uk Sat Jul 14 00:00:35 2007 From: boxbackup at fluffy.co.uk (Martin Ebourne) Date: Sat, 14 Jul 2007 00:00:35 +0100 Subject: [Box Backup] OT: Server colocation In-Reply-To: <20070713222737.GD62388@netinertia.co.uk> References: <20070713221149.GC62388@netinertia.co.uk> <20070713222737.GD62388@netinertia.co.uk> Message-ID: <1184367635.31710.4.camel@avenin.ebourne.me.uk> On Fri, 2007-07-13 at 23:27 +0100, James O'Gorman wrote: > I was originally paying =C2=A350/month with BCN for a 1U box, 100GB tra= nsfer > (50p for each GB used over that), up to 0.5A, serial console, remote > power cycle, as my IPv4 addresses as one can justify and an IPv6 /64. >=20 > They bumped their prices up to =C2=A375/month sometime last year due to > RedBus doing so. >=20 > Something similar would be nice :-) This is pretty similar: http://aaisp.net.uk/aa/aaisp/colocation.html Run by techies for techies, excellent customer service. Cheers, Martin. From boxbackup at fluffy.co.uk Sat Jul 14 09:45:45 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Sat, 14 Jul 2007 10:45:45 +0200 Subject: [Box Backup] Compiling boxbackup chris-merge on debian etch? In-Reply-To: References: <467FB654.2070107@kontrapunkt.com> <4683AECC.3010200@kontrapunkt.com> <4684B9D0.8050909@kontrapunkt.com> <468A43B0.2010407@kontrapunkt.com> <468F9DA5.5020109@kontrapunkt.com> <468FB61A.5060805@kontrapunkt.com> <468FDF1C.4060709@kontrapunkt.com> Message-ID: <46988D39.3080604@kontrapunkt.com> Hello Chris. I'm moving my data around and will soon start to use the server I compiled on debian etch PPC with your help. I'll report back. Tobias Chris Wilson wrote: > Hi Tobias, > > On Sat, 7 Jul 2007, Tobias Balle-Petersen wrote: > >> Trying to run the tests again returns: >> >> common: PASSED >> crypto: PASSED >> compress: PASSED >> raidfile: PASSED >> basicserver: PASSED >> backupstore: FAILED: 2 tests failed (first at >> ../../lib/common/Test.h:278) >> backupstorefix: PASSED >> backupstorepatch: PASSED >> backupdiff: FAILED: 1 tests failed (first at testbackupdiff.cpp:548) >> bbackupd: PASSED > > I've been running test/backupstore in a continuous loop for 5 days and I > haven't found a single error. I don't understand what was wrong with > this test on your machine as I can't reproduce it. > > I'd very much appreciate it if you use this version in production and > let me know how it goes? Please don't forget to compare -a regularly > (e.g. in an automatic scheduler such as cron) and report any unexpected > compare failures. > > Cheers, Chris. From boxbackup at fluffy.co.uk Mon Jul 16 09:59:50 2007 From: boxbackup at fluffy.co.uk (Richard Hurt) Date: Mon, 16 Jul 2007 04:59:50 -0400 Subject: [Box Backup] Wiki Link Farm Message-ID: <712ba87c0707160159u52a10fd1h9ed02794854adfe9@mail.gmail.com> It looks like the old Wiki page (http://boxbackup.hostworks.ca/index.php/Main_Page) has been corrupted and turned into a link farm. I suggest it be removed completely as it is presenting BoxBackup in a very bad light and we don't really need it anymore with the new Trac Wiki (http://bbdev.fluffy.co.uk/trac/wiki). The really bad thing is that the old page is the very first link in a Google search for "boxbackup wiki". :( Later... Richard From boxbackup at fluffy.co.uk Mon Jul 16 12:10:47 2007 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Mon, 16 Jul 2007 12:10:47 +0100 Subject: [Box Backup] Wiki Link Farm Message-ID: <54E9BD97-CEEA-4F97-B2F8-A50C56420075@fluffy.co.uk> Richard Hurt wrote: > > It looks like the old Wiki page > (http://boxbackup.hostworks.ca/index.php/Main_Page) has been corrupted > and turned into a link farm. I suggest it be removed completely as it > is presenting BoxBackup in a very bad light and we don't really need > it anymore with the new Trac Wiki > (http://bbdev.fluffy.co.uk/trac/wiki). > > The really bad thing is that the old page is the very first link in a > Google search for "boxbackup wiki". :( > I hate spammers. I have finally changed the link on the BB home page to the new wiki, and asked for the old one to be taken down. Ben From boxbackup at fluffy.co.uk Mon Jul 16 14:36:28 2007 From: boxbackup at fluffy.co.uk (Josh McCormack) Date: Mon, 16 Jul 2007 08:36:28 -0500 Subject: [Box Backup] feature question - sync Message-ID: Would it be possible to have live machines in various locations sync'ing up by pulling in changes from the server and updating themselves? In otherwords, say you had a desktop machine at the office, one at home, one at your in-laws house and a laptop. Have the all be the same "computer" with different locations and hardware, but all sync'ing up by backing up changes and updating themselves with any changes. -- Josh McCormack Owner, InteractiveQA Web testing & development http://www.interactiveqa.com 917.620.4902 From boxbackup at fluffy.co.uk Mon Jul 16 21:09:30 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Mon, 16 Jul 2007 21:09:30 +0100 (BST) Subject: [Box Backup] feature question - sync In-Reply-To: References: Message-ID: Hi Josh, On Mon, 16 Jul 2007, Josh McCormack wrote: > Would it be possible to have live machines in various locations > sync'ing up by pulling in changes from the server and updating > themselves? > > In otherwords, say you had a desktop machine at the office, one at > home, one at your in-laws house and a laptop. Have the all be the same > "computer" with different locations and hardware, but all sync'ing up > by backing up changes and updating themselves with any changes. Yes, it is possible, but Box Backup is probably not the best tool for this. For a start, you would have to put the same crypto keys on all machines. Then you would have to deal with the fact that bbackupd never downloads new data from the store, because it is expecting that it will be the only process writing to it. Instead of downloading new changes from other hosts, it will overwrite them. I don't think that we have enough information on the store to do a "proper" synchronisation. We would have to keep a local record of what we had during the last sync, and scan the store to see what had changed, if the client store marker was different to what we expected. I suppose it would be possible to use the running bbackupd state to do this, in theory, but only if bbackupd were never restarted. We could also use the StoreObjectInfoFile to persist that state, but I'm not sure that it's reliable enough and I'm worried that if it failed, it would propagate deletions of newly added files, and undo recent changes, to all machines in the group. Box Backup is not currently designed to do this and it's a lot of work, so not really a high priority for us at the moment, sorry. Unison or rsync might do what you want. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Tue Jul 17 16:48:18 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Tue, 17 Jul 2007 17:48:18 +0200 Subject: [Box Backup] Redundant locations not removed? Message-ID: <469CE4C2.90602@kontrapunkt.com> Hello... I'm running chris-merge on my debian bb store and my OS X bb client. I have removed 2 "sections" from the bbackupd.conf on the OS X client. In the log I get the following: 2 redundant locations in root directory found, will delete from store after 172800 seconds. 172800 second = 48 hours have now gone and the deleted "sections" are still visible on the store, and new files are not beeing backup because the store is full? How do I get rid of the deleted "sections" ? Thanks, Tobias From boxbackup at fluffy.co.uk Wed Jul 18 09:39:56 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 18 Jul 2007 09:39:56 +0100 (BST) Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <469CE4C2.90602@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> Message-ID: Hi Tobias, > I'm running chris-merge on my debian bb store and my OS X bb client. I > have removed 2 "sections" from the bbackupd.conf on the OS X client. > > In the log I get the following: > 2 redundant locations in root directory found, will delete from store > after 172800 seconds. > > 172800 second = 48 hours have now gone and the deleted "sections" are > still visible on the store, and new files are not beeing backup because > the store is full? That is very strange, thanks for reporting it. Please could you turn ExtendedLogging on on the client and let me know what it outputs during a backup run? Also, please include any log messages during that time. > How do I get rid of the deleted "sections" ? It would be possible for me to add a command to bbackupquery to delete them manually, but I'd prefer to fix the problem with autotamic deletion. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Wed Jul 18 19:15:08 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Wed, 18 Jul 2007 20:15:08 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: References: <469CE4C2.90602@kontrapunkt.com> Message-ID: <469E58AC.7000702@kontrapunkt.com> Hello Chris. This is what I get with extendedlogging on the client: Opening connection to server misora.local.kontrapunkt.dk... Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Send Version(0x1) Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Receive Version(0x1) Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Send Login(0x1,0x0) Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Receive LoginConfirmed(0x4357cf66abfa1,0x1411d082,0x13880000,0x14500000) Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Connection made, login successful Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Exceeded storage limits on server -- not uploading changes to files Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Send ListDirectory(0x1,0x2,0xc,false) Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Receive Success(0x1) Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Receiving stream, size 280 Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: 2 redundant locations in root directory found, will delete from store after 172800 seconds. Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Send ListDirectory(0x13c57f,0xffffffff,0xc,true) Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Receive Success(0x13c57f) Jul 18 19:52:47 yoiko Box Backup (bbackupd)[557]: Receiving stream, size 2121 .......... .......... .......... .......... .......... Jul 18 20:11:37 yoiko Box Backup (bbackupd)[557]: Receiving stream, size 929 Jul 18 20:11:37 yoiko Box Backup (bbackupd)[557]: Send ListDirectory(0x151c88,0xffffffff,0xc,true) Jul 18 20:11:37 yoiko Box Backup (bbackupd)[557]: Receive Success(0x151c88) Jul 18 20:11:37 yoiko Box Backup (bbackupd)[557]: Receiving stream, size 1049 Jul 18 20:11:37 yoiko Box Backup (bbackupd)[557]: Skipped file: server is full: /Volumes/raidD/clients/Cyncron/126339_Brochurer/126339_3 Progress/126339_3 AMM/FOTO/TIFF/Cyncron_6464.tif Jul 18 20:11:37 yoiko Box Backup (bbackupd)[557]: Skipped file: server is full: /Volumes/raidD/clients/Cyncron/126339_Brochurer/126339_3 Progress/126339_3 AMM/FOTO/TIFF/Cyncron_7197.tif .......... .......... .......... .......... .......... Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Skipped file: server is full: /Volumes/raidD/clients/Cyncron/126339_Brochurer/126339_3 Progress/126339_3 AMM/FOTO/TIFF/Cyncron_EKSTRA_7918.tif Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Skipped file: server is full: /Volumes/raidD/clients/Cyncron/126339_Brochurer/126339_3 Progress/126339_3 AMM/FOTO/TIFF/Cyncron_EKSTRA_7970.tif Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Skipped file: server is full: /Volumes/raidD/clients/Cyncron/126339_Brochurer/126339_3 Progress/126339_3 AMM/FOTO/TIFF/Cyncron_EKSTRA_7972.tif Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Skipped file: server is full: /Volumes/raidD/clients/Cyncron/126339_Brochurer/126339_3 Progress/126339_3 AMM/FOTO/TIFF/Cyncron_EKSTRA_8184.tif Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Send ListDirectory(0x153927,0xffffffff,0xc,true) Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Receive Success(0x153927) Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Receiving stream, size 89 Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Skipped file: server is full: /Volumes/raidB/PRE2/.DS_Store Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Send SetClientStoreMarker(0x4358da00f7d5e) Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Receive Success(0x4358da00f7d5e) Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Send Finished() Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Receive Finished() Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: Finished scan of local files Jul 18 20:11:38 yoiko Box Backup (bbackupd)[557]: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Thanks, Tobias Chris Wilson wrote: > Hi Tobias, > >> I'm running chris-merge on my debian bb store and my OS X bb client. I >> have removed 2 "sections" from the bbackupd.conf on the OS X client. >> >> In the log I get the following: >> 2 redundant locations in root directory found, will delete from store >> after 172800 seconds. >> >> 172800 second = 48 hours have now gone and the deleted "sections" are >> still visible on the store, and new files are not beeing backup >> because the store is full? > > That is very strange, thanks for reporting it. Please could you turn > ExtendedLogging on on the client and let me know what it outputs during > a backup run? Also, please include any log messages during that time. > >> How do I get rid of the deleted "sections" ? > > It would be possible for me to add a command to bbackupquery to delete > them manually, but I'd prefer to fix the problem with autotamic deletion. > > Cheers, Chris. From boxbackup at fluffy.co.uk Wed Jul 18 20:53:11 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 18 Jul 2007 20:53:11 +0100 (BST) Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <469E58AC.7000702@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> Message-ID: Hi Tobias, On Wed, 18 Jul 2007, Tobias Balle-Petersen wrote: > This is what I get with extendedlogging on the client: All looks normal except that the redundant locations are simply not deleted. Can you tell me how long bbackupd has been running on this machine? I guess you had to restart it to enable ExtendedLogging, so we might have to wait a couple of days until those redundant locations ought to be deleted. How long had it been running before you restarted it, if you know? (how long back to the last startup banner in your syslog)? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Wed Jul 18 21:40:33 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Wed, 18 Jul 2007 22:40:33 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> Message-ID: <469E7AC1.2090208@kontrapunkt.com> Hello Chris. > All looks normal except that the redundant locations are simply not > deleted. Can you tell me how long bbackupd has been running on this > machine? I guess you had to restart it to enable ExtendedLogging, so we > might have to wait a couple of days until those redundant locations > ought to be deleted. Yes, I'll let it run and get back to you. > How long had it been running before you restarted it, if you know? (how > long back to the last startup banner in your syslog)? It had been running since sunday... So about 2,5 days. I ran a check on the account on the store: Check store account ID 00000001 Phase 1, check objects... Phase 2, check directories... File ID be2 has different container ID, probably moved File ID 1890 has different container ID, probably moved File ID 8e16 has different container ID, probably moved File ID 8e17 has different container ID, probably moved ..... ..... ..... File ID 2433f has different container ID, probably moved File ID 24340 has different container ID, probably moved File ID 25c54 has different container ID, probably moved File ID 25c55 has different container ID, probably moved File ID 25c56 has different container ID, probably moved File ID 13baed has different container ID, probably moved Phase 3, check root... Phase 4, fix unattached objects... Phase 5, fix unrecovered inconsistencies... Phase 6, regenerate store info... Store account checked, no errors found. That looks normal right? I only have 19GB free on the partiotion that holds the BB-files on the store. Could that be of any importance? Any "unknown" overhead or something like that when deleting files? Thanks, Tobias From boxbackup at fluffy.co.uk Wed Jul 18 22:13:37 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 18 Jul 2007 22:13:37 +0100 (BST) Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <469E7AC1.2090208@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> Message-ID: Hi Tobias, On Wed, 18 Jul 2007, Tobias Balle-Petersen wrote: >> All looks normal except that the redundant locations are simply not >> deleted. Can you tell me how long bbackupd has been running on this >> machine? I guess you had to restart it to enable ExtendedLogging, so we >> might have to wait a couple of days until those redundant locations >> ought to be deleted. > > Yes, I'll let it run and get back to you. Thanks. Sounds like the best way to debug this right now. > It had been running since sunday... So about 2,5 days. So it should have started deleting the unused locations already. > I ran a check on the account on the store: ... > Store account checked, no errors found. > > That looks normal right? Yes, looks fine to me. > I only have 19GB free on the partiotion that holds the BB-files on the > store. Could that be of any importance? Any "unknown" overhead or > something like that when deleting files? You should have an exception in the logs on the client if it ran out of space on the server (I hope). Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Wed Jul 18 23:00:58 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 18 Jul 2007 23:00:58 +0100 (BST) Subject: [Box Backup] Redundant locations not removed? In-Reply-To: References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> Message-ID: Hi Tobias, > On Wed, 18 Jul 2007, Tobias Balle-Petersen wrote: > >> > All looks normal except that the redundant locations are simply not >> > deleted. Can you tell me how long bbackupd has been running on this >> > machine? I guess you had to restart it to enable ExtendedLogging, so we >> > might have to wait a couple of days until those redundant locations >> > ought to be deleted. >> >> Yes, I'll let it run and get back to you. > > Thanks. Sounds like the best way to debug this right now. Actually, I have a better idea :-) I think I just fixed a bug that would cause the deletion timer to be reset each run, so the unused locations would never be deleted. Could I ask you to update to the latest svn and build a new client on this machine, install it and run it for a few days, checking that the "will delete from store after" time keeps going down on each backup run? Thanks again for your help, and for testing this version for us! Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Thu Jul 19 13:46:41 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Thu, 19 Jul 2007 14:46:41 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> Message-ID: <469F5D31.90107@kontrapunkt.com> Chris Wilson wrote: > Actually, I have a better idea :-) I think I just fixed a bug that would > cause the deletion timer to be reset each run, so the unused locations > would never be deleted. > > Could I ask you to update to the latest svn and build a new client on > this machine, install it and run it for a few days, checking that the > "will delete from store after" time keeps going down on each backup run? I have reverted back to the 0.10 version of the client. It is absolutely crucial that I get the files backed up ASAP. When thats done, I'll build your latest client and we can continue testing. Thanks, Tobias From boxbackup at fluffy.co.uk Thu Jul 19 15:34:55 2007 From: boxbackup at fluffy.co.uk (Stephen Nelson-Smith) Date: Thu, 19 Jul 2007 15:34:55 +0100 Subject: [Box Backup] OT: Server colocation In-Reply-To: <20070713222737.GD62388@netinertia.co.uk> References: <20070713221149.GC62388@netinertia.co.uk> <20070713222737.GD62388@netinertia.co.uk> Message-ID: On 7/13/07, James O'Gorman wrote: > On Fri, Jul 13, 2007 at 11:16:16PM +0100, Stephen Nelson-Smith wrote: > > On 7/13/07, James O'Gorman wrote: > > > Hi all, > > > > > > My colo provider have this evening announced they have been taken ove= r > > > by another company which will mean servers being moved from London > > > Docklands to Maidenhead. > > > > Who's that then? > > Black Cat Networks. I would post a link to the email but the archives > are subscribers only. > > > > So, does anyone have any recomendations/experience with other colo > > > providers in London? > > > > Several - depends on what you want - I've used several in the past, am > > using one n particular at the moment, and have some other contacts > > too. How much are you paying? What's the bandwidth cost? How much > > power do you need? > > I was originally paying =A350/month with BCN for a 1U box, 100GB transfer > (50p for each GB used over that), up to 0.5A, serial console, remote > power cycle, as my IPv4 addresses as one can justify and an IPv6 /64. > > They bumped their prices up to =A375/month sometime last year due to > RedBus doing so. > > Something similar would be nice :-) One of my clients can match that, at =A370.00; Would be in HEX, so not far to go :). Interested, ping me off list. > James S. From boxbackup at fluffy.co.uk Thu Jul 19 17:13:30 2007 From: boxbackup at fluffy.co.uk (Wout Mertens) Date: Thu, 19 Jul 2007 18:13:30 +0200 Subject: [Box Backup] OT: Server colocation In-Reply-To: References: <20070713221149.GC62388@netinertia.co.uk> <20070713222737.GD62388@netinertia.co.uk> Message-ID: <20617503-11CF-4C1E-94F0-9043E5E63244@cisco.com> Have you considered Amazon's Web Services? http://www.amazon.com/AWS-home-page-Money/b/=20 ref=3Dsc_fe_l_1/105-3769240-3766816?=20 ie=3DUTF8&node=3D3435361&no=3D3435361&me=3DA36L942TSJ2AJA I recently took a look at them and they seem almost too good to be true. For example, storage (the S3 element) costs 0.15$/GB, and it's =20 enterprise-level and pay-as-you-go. You access it using HTTP (REST) =20 or SOAP. You pay per GB transfer, 0.10$/0.18$ per GB transferred =20 depending on direction. Pretty cheap if you ask me. You can use your own servers to access it, or you can let them host a =20= virtual image (the EC2 component) at very reasonable prices... I'm considering switching over to them, it seems to me I'll save =20 money over my current colo and gain reliability in the deal. --- Obligatory on-topic comment: Maybe boxbackup could be changed so all =20 you need is an S3 account and the boxbackup client instead of a server. I'm serious. What do you think? Can the boxbackup protocol be changed =20= so all you need is SOAP or HTTP? I'm thinking yes... Wout. On Jul 19, 2007, at 4:34 PM, Stephen Nelson-Smith wrote: > On 7/13/07, James O'Gorman wrote: >> On Fri, Jul 13, 2007 at 11:16:16PM +0100, Stephen Nelson-Smith wrote: >> > On 7/13/07, James O'Gorman wrote: >> > > Hi all, >> > > >> > > My colo provider have this evening announced they have been =20 >> taken over >> > > by another company which will mean servers being moved from =20 >> London >> > > Docklands to Maidenhead. >> > >> > Who's that then? >> >> Black Cat Networks. I would post a link to the email but the archives >> are subscribers only. >> >> > > So, does anyone have any recomendations/experience with other =20 >> colo >> > > providers in London? >> > >> > Several - depends on what you want - I've used several in the =20 >> past, am >> > using one n particular at the moment, and have some other contacts >> > too. How much are you paying? What's the bandwidth cost? How =20= >> much >> > power do you need? >> >> I was originally paying =A350/month with BCN for a 1U box, 100GB =20 >> transfer >> (50p for each GB used over that), up to 0.5A, serial console, remote >> power cycle, as my IPv4 addresses as one can justify and an IPv6 /64. >> >> They bumped their prices up to =A375/month sometime last year due to >> RedBus doing so. >> >> Something similar would be nice :-) > > One of my clients can match that, at =A370.00; Would be in HEX, so not > far to go :). > > Interested, ping me off list. > >> James > > S. > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup From boxbackup at fluffy.co.uk Thu Jul 19 20:35:38 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Thu, 19 Jul 2007 20:35:38 +0100 (BST) Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <469F5D31.90107@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> Message-ID: Hi Tobias, On Thu, 19 Jul 2007, Tobias Balle-Petersen wrote: >> Actually, I have a better idea :-) I think I just fixed a bug that >> would cause the deletion timer to be reset each run, so the unused >> locations would never be deleted. > > I have reverted back to the 0.10 version of the client. It is absolutely > crucial that I get the files backed up ASAP. When thats done, I'll build > your latest client and we can continue testing. I think this bug may have been in 0.10 as well (I certainly don't remember changing that portion of the code) but it would be very interesting to know if this fixes it for you :-) If it doesn't and you're in a hurry to get the files backed up, you could always increase the account limits temporarily, which should at least get the backups going again regardless of the unused locations, which we can sort out later. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Thu Jul 19 20:56:43 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Thu, 19 Jul 2007 20:56:43 +0100 (BST) Subject: [Box Backup] OT: Server colocation In-Reply-To: <20617503-11CF-4C1E-94F0-9043E5E63244@cisco.com> References: <20070713221149.GC62388@netinertia.co.uk> <20070713222737.GD62388@netinertia.co.uk> <20617503-11CF-4C1E-94F0-9043E5E63244@cisco.com> Message-ID: Hi Wout, On Thu, 19 Jul 2007, Wout Mertens wrote: > Obligatory on-topic comment: Maybe boxbackup could be changed so all you > need is an S3 account and the boxbackup client instead of a server. > > I'm serious. What do you think? Can the boxbackup protocol be changed so > all you need is SOAP or HTTP? I'm thinking yes... I have definitely considered this. The main problem is that at the moment, for efficiency, the Box client-server protocol has some special commands, for example the ability to get the server to send only parts of files, or to mark them as deleted for cleanup by housekeeping. We might want to add more in future, such as getting the server to compute checksums of encrypted blocks and return them together with the IVs. I assume that Amazon S3 has no such facilities, so it might be less efficient. But I would like to decouple the file-encryption part from the client-server part, for example to allow backing up to remote DAV/FTP/SFTP servers and local block devices. Too many ideas, too little time and skill (on my part) ;-) Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Thu Jul 19 21:29:22 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Thu, 19 Jul 2007 22:29:22 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> Message-ID: <469FC9A2.4060708@kontrapunkt.com> Chris Wilson wrote: > I think this bug may have been in 0.10 as well (I certainly don't > remember changing that portion of the code) but it would be very > interesting to know if this fixes it for you :-) Do you really mean that removing "locations" did not work in 0.10? I'm sure I have done it before. Tobias From boxbackup at fluffy.co.uk Tue Jul 24 12:16:18 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Tue, 24 Jul 2007 13:16:18 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> Message-ID: <46A5DF82.1060809@kontrapunkt.com> Hello Chris. I reverted back to 0.10 client from chris_merge 1696. The redundant locations were removed after 48 hours as they should be. My files are now being backed up. The store is still chris_merge. So it would seem there was an error in client v. 1696. Tobias Chris Wilson wrote: > Hi Tobias, > > On Thu, 19 Jul 2007, Tobias Balle-Petersen wrote: > >>> Actually, I have a better idea :-) I think I just fixed a bug that >>> would cause the deletion timer to be reset each run, so the unused >>> locations would never be deleted. >> >> I have reverted back to the 0.10 version of the client. It is >> absolutely crucial that I get the files backed up ASAP. When thats >> done, I'll build your latest client and we can continue testing. > > I think this bug may have been in 0.10 as well (I certainly don't > remember changing that portion of the code) but it would be very > interesting to know if this fixes it for you :-) > > If it doesn't and you're in a hurry to get the files backed up, you > could always increase the account limits temporarily, which should at > least get the backups going again regardless of the unused locations, > which we can sort out later. > > Cheers, Chris. From boxbackup at fluffy.co.uk Tue Jul 24 20:41:59 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 24 Jul 2007 20:41:59 +0100 (BST) Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <46A5DF82.1060809@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> Message-ID: Hi Tobias, On Tue, 24 Jul 2007, Tobias Balle-Petersen wrote: > I reverted back to 0.10 client from chris_merge 1696. The redundant > locations were removed after 48 hours as they should be. My files are > now being backed up. > > The store is still chris_merge. > > So it would seem there was an error in client v. 1696. Indeed, thanks for letting me know, will look into it. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Wed Jul 25 10:32:42 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Wed, 25 Jul 2007 11:32:42 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> Message-ID: <46A718BA.9070607@kontrapunkt.com> Hello Chris. My files are now starting to be backed up. The backups stops after 4 hours however. Looking at the stats for the store, data is beeing transfered until the connection ends. As far as I can tell, this would mean that this is not a SSL-timeout related problem. Client is 0.10 and store is chris_merge 1725. I'm running at session again now. If it stops after 4 hours, I'm expecting it will, I will revert to 0.10 on the store as well and run a sync again. Logs from the two sessions I have run below. ------------------------------------------- CLIENT 0.10: ------------------------------------------- Session 1: Jul 24 13:00:29 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) Jul 24 13:00:29 yoiko bbackupd[402]: Connection from command socket Jul 24 13:00:29 yoiko bbackupd[402]: Beginning scan of local files Jul 24 13:00:29 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... Jul 24 13:00:30 yoiko bbackupd[402]: Connection made, login successful Jul 24 17:01:39 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server probably reported an error.) 7/47), reset state and waiting to retry... Jul 24 17:01:49 yoiko bbackupd[402]: File statistics: total file size uploaded 93763175181, bytes already on server 0, encoded size 54271648859 Session 2: Jul 25 00:00:01 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) Jul 25 00:00:01 yoiko bbackupd[402]: Connection from command socket Jul 25 00:00:01 yoiko bbackupd[402]: Beginning scan of local files Jul 25 00:00:01 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... Jul 25 00:00:01 yoiko bbackupd[402]: Connection made, login successful Jul 25 03:45:21 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server probably reported an error.) 7/47), reset state and waiting to retry... Jul 25 03:45:31 yoiko bbackupd[402]: File statistics: total file size uploaded 81804900665, bytes already on server 0, encoded size 54427385858 ------------------------------------------- STORE chris_merge 1725: ------------------------------------------- Session 1: Jul 24 13:03:48 misora Box Backup (bbstored)[18666]: Incoming connection from 192.168.0.51 port 63456 (handling in child 32225) Jul 24 13:03:48 misora Box Backup (bbstored)[32225]: Certificate CN: BACKUP-1 Jul 24 13:03:48 misora Box Backup (bbstored)[32225]: Login: Client ID 00000001, Read/Write Jul 24 13:08:29 misora bbstored/hk[18667]: Starting housekeeping Jul 24 13:09:29 misora bbstored/hk[18667]: Finished housekeeping ....... ....... ....... Jul 24 16:53:30 misora bbstored/hk[18667]: Starting housekeeping Jul 24 16:54:30 misora bbstored/hk[18667]: Finished housekeeping Jul 24 17:04:58 misora Box Backup (bbstored)[32225]: Session finished Jul 24 17:04:58 misora Box Backup (bbstored)[32225]: Connection statistics for BACKUP-1: IN=54273151997 OUT=2917099 TOTAL=54276069096 Session 2: Jul 25 00:03:20 misora Box Backup (bbstored)[18666]: Incoming connection from 192.168.0.51 port 51921 (handling in child 10676) Jul 25 00:03:20 misora Box Backup (bbstored)[10676]: Certificate CN: BACKUP-1 Jul 25 00:03:20 misora Box Backup (bbstored)[10676]: Login: Client ID 00000001, Read/Write Jul 25 00:08:45 misora bbstored/hk[18667]: Starting housekeeping Jul 25 00:09:45 misora bbstored/hk[18667]: Finished housekeeping ....... ....... ....... Jul 25 03:38:46 misora bbstored/hk[18667]: Starting housekeeping Jul 25 03:39:46 misora bbstored/hk[18667]: Finished housekeeping Jul 25 03:48:41 misora Box Backup (bbstored)[10676]: Session finished Jul 25 03:48:41 misora Box Backup (bbstored)[10676]: Connection statistics for BACKUP-1: IN=54428917970 OUT=3562128 TOTAL=54432480098 Thanks, Tobias Chris Wilson wrote: > Hi Tobias, > > On Tue, 24 Jul 2007, Tobias Balle-Petersen wrote: > >> I reverted back to 0.10 client from chris_merge 1696. The redundant >> locations were removed after 48 hours as they should be. My files are >> now being backed up. >> >> The store is still chris_merge. >> >> So it would seem there was an error in client v. 1696. > > Indeed, thanks for letting me know, will look into it. > > Cheers, Chris. From boxbackup at fluffy.co.uk Wed Jul 25 14:25:04 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Wed, 25 Jul 2007 15:25:04 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <46A718BA.9070607@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> <46A718BA.9070607@kontrapunkt.com> Message-ID: <46A74F30.1030707@kontrapunkt.com> Hello Chris. As expected, the backup ended after approx. 4 hours: Jul 25 10:59:54 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) Jul 25 10:59:54 yoiko bbackupd[402]: Connection from command socket Jul 25 10:59:54 yoiko bbackupd[402]: Beginning scan of local files Jul 25 10:59:54 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... Jul 25 10:59:54 yoiko bbackupd[402]: Connection made, login successful Jul 25 15:06:14 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server probably reported an error.) 7/47), reset state and waiting to retry... Jul 25 15:06:24 yoiko bbackupd[402]: File statistics: total file size uploaded 93870658096, bytes already on server 0, encoded size 53966419139 I'm now running a sync with the chris_merge 1696 client and chris_merge 1725 store. If this fails, I'll revert to the 0.10 store as I wrote previously. Tobias Tobias Balle-Petersen wrote: > Hello Chris. > > My files are now starting to be backed up. The backups stops after 4 hours however. Looking at the > stats for the store, data is beeing transfered until the connection ends. As far as I can tell, this > would mean that this is not a SSL-timeout related problem. Client is 0.10 and store is chris_merge > 1725. > > I'm running at session again now. If it stops after 4 hours, I'm expecting it will, I will revert to > 0.10 on the store as well and run a sync again. > > > Logs from the two sessions I have run below. > > ------------------------------------------- > CLIENT 0.10: > ------------------------------------------- > > Session 1: > Jul 24 13:00:29 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) > Jul 24 13:00:29 yoiko bbackupd[402]: Connection from command socket > Jul 24 13:00:29 yoiko bbackupd[402]: Beginning scan of local files > Jul 24 13:00:29 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... > Jul 24 13:00:30 yoiko bbackupd[402]: Connection made, login successful > Jul 24 17:01:39 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server > probably reported an error.) 7/47), reset state and waiting to retry... > Jul 24 17:01:49 yoiko bbackupd[402]: File statistics: total file size uploaded 93763175181, bytes > already on server 0, encoded size 54271648859 > > Session 2: > Jul 25 00:00:01 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) > Jul 25 00:00:01 yoiko bbackupd[402]: Connection from command socket > Jul 25 00:00:01 yoiko bbackupd[402]: Beginning scan of local files > Jul 25 00:00:01 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... > Jul 25 00:00:01 yoiko bbackupd[402]: Connection made, login successful > Jul 25 03:45:21 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server > probably reported an error.) 7/47), reset state and waiting to retry... > Jul 25 03:45:31 yoiko bbackupd[402]: File statistics: total file size uploaded 81804900665, bytes > already on server 0, encoded size 54427385858 > > ------------------------------------------- > STORE chris_merge 1725: > ------------------------------------------- > > Session 1: > Jul 24 13:03:48 misora Box Backup (bbstored)[18666]: Incoming connection from 192.168.0.51 port > 63456 (handling in child 32225) > Jul 24 13:03:48 misora Box Backup (bbstored)[32225]: Certificate CN: BACKUP-1 > Jul 24 13:03:48 misora Box Backup (bbstored)[32225]: Login: Client ID 00000001, Read/Write > Jul 24 13:08:29 misora bbstored/hk[18667]: Starting housekeeping > Jul 24 13:09:29 misora bbstored/hk[18667]: Finished housekeeping > ....... > ....... > ....... > Jul 24 16:53:30 misora bbstored/hk[18667]: Starting housekeeping > Jul 24 16:54:30 misora bbstored/hk[18667]: Finished housekeeping > Jul 24 17:04:58 misora Box Backup (bbstored)[32225]: Session finished > Jul 24 17:04:58 misora Box Backup (bbstored)[32225]: Connection statistics for BACKUP-1: > IN=54273151997 OUT=2917099 TOTAL=54276069096 > > Session 2: > Jul 25 00:03:20 misora Box Backup (bbstored)[18666]: Incoming connection from 192.168.0.51 port > 51921 (handling in child 10676) > Jul 25 00:03:20 misora Box Backup (bbstored)[10676]: Certificate CN: BACKUP-1 > Jul 25 00:03:20 misora Box Backup (bbstored)[10676]: Login: Client ID 00000001, Read/Write > Jul 25 00:08:45 misora bbstored/hk[18667]: Starting housekeeping > Jul 25 00:09:45 misora bbstored/hk[18667]: Finished housekeeping > ....... > ....... > ....... > Jul 25 03:38:46 misora bbstored/hk[18667]: Starting housekeeping > Jul 25 03:39:46 misora bbstored/hk[18667]: Finished housekeeping > Jul 25 03:48:41 misora Box Backup (bbstored)[10676]: Session finished > Jul 25 03:48:41 misora Box Backup (bbstored)[10676]: Connection statistics for BACKUP-1: > IN=54428917970 OUT=3562128 TOTAL=54432480098 > > > Thanks, > Tobias > > > > Chris Wilson wrote: >> Hi Tobias, >> >> On Tue, 24 Jul 2007, Tobias Balle-Petersen wrote: >> >>> I reverted back to 0.10 client from chris_merge 1696. The redundant >>> locations were removed after 48 hours as they should be. My files are >>> now being backed up. >>> >>> The store is still chris_merge. >>> >>> So it would seem there was an error in client v. 1696. >> Indeed, thanks for letting me know, will look into it. >> >> Cheers, Chris. > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Wed Jul 25 14:42:58 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Wed, 25 Jul 2007 15:42:58 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <46A74F30.1030707@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> <46A718BA.9070607@kontrapunkt.com> <46A74F30.1030707@kontrapunkt.com> Message-ID: <46A75362.8090203@kontrapunkt.com> Hello Chris. The 1696 client did nothing when attempting a sync. I'm now back at 0.10 store and 0.10 client. Now files are not beeing backed up because the store is 99% full: Jul 25 15:32:23 yoiko bbackupd[24426]: Starting daemon (config: /etc/box_misora/bbackupd.conf) (version 0.10) Jul 25 15:34:28 yoiko bbackupd[24426]: Incoming connection from local (UNIX socket) Jul 25 15:34:28 yoiko bbackupd[24426]: Connection from command socket Jul 25 15:34:28 yoiko bbackupd[24426]: Beginning scan of local files Jul 25 15:34:28 yoiko bbackupd[24426]: Opening connection to server misora.local.kontrapunkt.dk... Jul 25 15:34:29 yoiko bbackupd[24426]: Connection made, login successful Jul 25 15:34:29 yoiko bbackupd[24426]: Exceeded storage limits on server -- not uploading changes to files Jul 25 15:34:29 yoiko bbackupd[24426]: About to notify administrator about event store-full, running script '/etc/box_misora/bbackupd/NotifySysadmin.sh store-full' Jul 25 15:34:29 yoiko bbackupd[24426]: Finished scan of local files Jul 25 15:34:29 yoiko bbackupd[24426]: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 The thing is, 73% of the files are deleted files. Shouldn't they be removed and the new files be backed up? Used 1331183.2Mb 99% *************************************** Old files 0.0Mb 0% Deleted files 972295.6Mb 73% ***************************** Directories 626.1Mb 0% Soft limit 1280000.0Mb 96% ************************************** Hard limit 1331200.0Mb 100% **************************************** Thanks, Tobias Tobias Balle-Petersen wrote: > Hello Chris. > > As expected, the backup ended after approx. 4 hours: > Jul 25 10:59:54 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) > Jul 25 10:59:54 yoiko bbackupd[402]: Connection from command socket > Jul 25 10:59:54 yoiko bbackupd[402]: Beginning scan of local files > Jul 25 10:59:54 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... > Jul 25 10:59:54 yoiko bbackupd[402]: Connection made, login successful > Jul 25 15:06:14 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server > probably reported an error.) 7/47), reset state and waiting to retry... > Jul 25 15:06:24 yoiko bbackupd[402]: File statistics: total file size uploaded 93870658096, bytes > already on server 0, encoded size 53966419139 > > I'm now running a sync with the chris_merge 1696 client and chris_merge 1725 store. > > If this fails, I'll revert to the 0.10 store as I wrote previously. > > Tobias > > > Tobias Balle-Petersen wrote: >> Hello Chris. >> >> My files are now starting to be backed up. The backups stops after 4 hours however. Looking at the >> stats for the store, data is beeing transfered until the connection ends. As far as I can tell, this >> would mean that this is not a SSL-timeout related problem. Client is 0.10 and store is chris_merge >> 1725. >> >> I'm running at session again now. If it stops after 4 hours, I'm expecting it will, I will revert to >> 0.10 on the store as well and run a sync again. >> >> >> Logs from the two sessions I have run below. >> >> ------------------------------------------- >> CLIENT 0.10: >> ------------------------------------------- >> >> Session 1: >> Jul 24 13:00:29 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) >> Jul 24 13:00:29 yoiko bbackupd[402]: Connection from command socket >> Jul 24 13:00:29 yoiko bbackupd[402]: Beginning scan of local files >> Jul 24 13:00:29 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... >> Jul 24 13:00:30 yoiko bbackupd[402]: Connection made, login successful >> Jul 24 17:01:39 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server >> probably reported an error.) 7/47), reset state and waiting to retry... >> Jul 24 17:01:49 yoiko bbackupd[402]: File statistics: total file size uploaded 93763175181, bytes >> already on server 0, encoded size 54271648859 >> >> Session 2: >> Jul 25 00:00:01 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) >> Jul 25 00:00:01 yoiko bbackupd[402]: Connection from command socket >> Jul 25 00:00:01 yoiko bbackupd[402]: Beginning scan of local files >> Jul 25 00:00:01 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... >> Jul 25 00:00:01 yoiko bbackupd[402]: Connection made, login successful >> Jul 25 03:45:21 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server >> probably reported an error.) 7/47), reset state and waiting to retry... >> Jul 25 03:45:31 yoiko bbackupd[402]: File statistics: total file size uploaded 81804900665, bytes >> already on server 0, encoded size 54427385858 >> >> ------------------------------------------- >> STORE chris_merge 1725: >> ------------------------------------------- >> >> Session 1: >> Jul 24 13:03:48 misora Box Backup (bbstored)[18666]: Incoming connection from 192.168.0.51 port >> 63456 (handling in child 32225) >> Jul 24 13:03:48 misora Box Backup (bbstored)[32225]: Certificate CN: BACKUP-1 >> Jul 24 13:03:48 misora Box Backup (bbstored)[32225]: Login: Client ID 00000001, Read/Write >> Jul 24 13:08:29 misora bbstored/hk[18667]: Starting housekeeping >> Jul 24 13:09:29 misora bbstored/hk[18667]: Finished housekeeping >> ....... >> ....... >> ....... >> Jul 24 16:53:30 misora bbstored/hk[18667]: Starting housekeeping >> Jul 24 16:54:30 misora bbstored/hk[18667]: Finished housekeeping >> Jul 24 17:04:58 misora Box Backup (bbstored)[32225]: Session finished >> Jul 24 17:04:58 misora Box Backup (bbstored)[32225]: Connection statistics for BACKUP-1: >> IN=54273151997 OUT=2917099 TOTAL=54276069096 >> >> Session 2: >> Jul 25 00:03:20 misora Box Backup (bbstored)[18666]: Incoming connection from 192.168.0.51 port >> 51921 (handling in child 10676) >> Jul 25 00:03:20 misora Box Backup (bbstored)[10676]: Certificate CN: BACKUP-1 >> Jul 25 00:03:20 misora Box Backup (bbstored)[10676]: Login: Client ID 00000001, Read/Write >> Jul 25 00:08:45 misora bbstored/hk[18667]: Starting housekeeping >> Jul 25 00:09:45 misora bbstored/hk[18667]: Finished housekeeping >> ....... >> ....... >> ....... >> Jul 25 03:38:46 misora bbstored/hk[18667]: Starting housekeeping >> Jul 25 03:39:46 misora bbstored/hk[18667]: Finished housekeeping >> Jul 25 03:48:41 misora Box Backup (bbstored)[10676]: Session finished >> Jul 25 03:48:41 misora Box Backup (bbstored)[10676]: Connection statistics for BACKUP-1: >> IN=54428917970 OUT=3562128 TOTAL=54432480098 >> >> >> Thanks, >> Tobias >> >> >> >> Chris Wilson wrote: >>> Hi Tobias, >>> >>> On Tue, 24 Jul 2007, Tobias Balle-Petersen wrote: >>> >>>> I reverted back to 0.10 client from chris_merge 1696. The redundant >>>> locations were removed after 48 hours as they should be. My files are >>>> now being backed up. >>>> >>>> The store is still chris_merge. >>>> >>>> So it would seem there was an error in client v. 1696. >>> Indeed, thanks for letting me know, will look into it. >>> >>> Cheers, Chris. >> _______________________________________________ >> boxbackup mailing list >> boxbackup at fluffy.co.uk >> http://lists.warhead.org.uk/mailman/listinfo/boxbackup >> > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Wed Jul 25 14:59:28 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Wed, 25 Jul 2007 15:59:28 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <46A75362.8090203@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> <46A718BA.9070607@kontrapunkt.com> <46A74F30.1030707@kontrapunkt.com> <46A75362.8090203@kontrapunkt.com> Message-ID: <46A75740.7010303@kontrapunkt.com> Hello Chris. I was a bit quick with my last mail. Housekeeping has now run on my store and deleted files. Thanks, Tobias Tobias Balle-Petersen wrote: > Hello Chris. > > The 1696 client did nothing when attempting a sync. I'm now back at 0.10 store and 0.10 client. > > Now files are not beeing backed up because the store is 99% full: > Jul 25 15:32:23 yoiko bbackupd[24426]: Starting daemon (config: /etc/box_misora/bbackupd.conf) > (version 0.10) > Jul 25 15:34:28 yoiko bbackupd[24426]: Incoming connection from local (UNIX socket) > Jul 25 15:34:28 yoiko bbackupd[24426]: Connection from command socket > Jul 25 15:34:28 yoiko bbackupd[24426]: Beginning scan of local files > Jul 25 15:34:28 yoiko bbackupd[24426]: Opening connection to server misora.local.kontrapunkt.dk... > Jul 25 15:34:29 yoiko bbackupd[24426]: Connection made, login successful > Jul 25 15:34:29 yoiko bbackupd[24426]: Exceeded storage limits on server -- not uploading changes to > files > Jul 25 15:34:29 yoiko bbackupd[24426]: About to notify administrator about event store-full, running > script '/etc/box_misora/bbackupd/NotifySysadmin.sh store-full' > Jul 25 15:34:29 yoiko bbackupd[24426]: Finished scan of local files > Jul 25 15:34:29 yoiko bbackupd[24426]: File statistics: total file size uploaded 0, bytes already on > server 0, encoded size 0 > > The thing is, 73% of the files are deleted files. Shouldn't they be removed and the new files be > backed up? > > Used 1331183.2Mb 99% *************************************** > Old files 0.0Mb 0% > Deleted files 972295.6Mb 73% ***************************** > Directories 626.1Mb 0% > Soft limit 1280000.0Mb 96% ************************************** > Hard limit 1331200.0Mb 100% **************************************** > > Thanks, > Tobias > > > > Tobias Balle-Petersen wrote: >> Hello Chris. >> >> As expected, the backup ended after approx. 4 hours: >> Jul 25 10:59:54 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) >> Jul 25 10:59:54 yoiko bbackupd[402]: Connection from command socket >> Jul 25 10:59:54 yoiko bbackupd[402]: Beginning scan of local files >> Jul 25 10:59:54 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... >> Jul 25 10:59:54 yoiko bbackupd[402]: Connection made, login successful >> Jul 25 15:06:14 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server >> probably reported an error.) 7/47), reset state and waiting to retry... >> Jul 25 15:06:24 yoiko bbackupd[402]: File statistics: total file size uploaded 93870658096, bytes >> already on server 0, encoded size 53966419139 >> >> I'm now running a sync with the chris_merge 1696 client and chris_merge 1725 store. >> >> If this fails, I'll revert to the 0.10 store as I wrote previously. >> >> Tobias >> >> >> Tobias Balle-Petersen wrote: >>> Hello Chris. >>> >>> My files are now starting to be backed up. The backups stops after 4 hours however. Looking at the >>> stats for the store, data is beeing transfered until the connection ends. As far as I can tell, this >>> would mean that this is not a SSL-timeout related problem. Client is 0.10 and store is chris_merge >>> 1725. >>> >>> I'm running at session again now. If it stops after 4 hours, I'm expecting it will, I will revert to >>> 0.10 on the store as well and run a sync again. >>> >>> >>> Logs from the two sessions I have run below. >>> >>> ------------------------------------------- >>> CLIENT 0.10: >>> ------------------------------------------- >>> >>> Session 1: >>> Jul 24 13:00:29 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) >>> Jul 24 13:00:29 yoiko bbackupd[402]: Connection from command socket >>> Jul 24 13:00:29 yoiko bbackupd[402]: Beginning scan of local files >>> Jul 24 13:00:29 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... >>> Jul 24 13:00:30 yoiko bbackupd[402]: Connection made, login successful >>> Jul 24 17:01:39 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server >>> probably reported an error.) 7/47), reset state and waiting to retry... >>> Jul 24 17:01:49 yoiko bbackupd[402]: File statistics: total file size uploaded 93763175181, bytes >>> already on server 0, encoded size 54271648859 >>> >>> Session 2: >>> Jul 25 00:00:01 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) >>> Jul 25 00:00:01 yoiko bbackupd[402]: Connection from command socket >>> Jul 25 00:00:01 yoiko bbackupd[402]: Beginning scan of local files >>> Jul 25 00:00:01 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... >>> Jul 25 00:00:01 yoiko bbackupd[402]: Connection made, login successful >>> Jul 25 03:45:21 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server >>> probably reported an error.) 7/47), reset state and waiting to retry... >>> Jul 25 03:45:31 yoiko bbackupd[402]: File statistics: total file size uploaded 81804900665, bytes >>> already on server 0, encoded size 54427385858 >>> >>> ------------------------------------------- >>> STORE chris_merge 1725: >>> ------------------------------------------- >>> >>> Session 1: >>> Jul 24 13:03:48 misora Box Backup (bbstored)[18666]: Incoming connection from 192.168.0.51 port >>> 63456 (handling in child 32225) >>> Jul 24 13:03:48 misora Box Backup (bbstored)[32225]: Certificate CN: BACKUP-1 >>> Jul 24 13:03:48 misora Box Backup (bbstored)[32225]: Login: Client ID 00000001, Read/Write >>> Jul 24 13:08:29 misora bbstored/hk[18667]: Starting housekeeping >>> Jul 24 13:09:29 misora bbstored/hk[18667]: Finished housekeeping >>> ....... >>> ....... >>> ....... >>> Jul 24 16:53:30 misora bbstored/hk[18667]: Starting housekeeping >>> Jul 24 16:54:30 misora bbstored/hk[18667]: Finished housekeeping >>> Jul 24 17:04:58 misora Box Backup (bbstored)[32225]: Session finished >>> Jul 24 17:04:58 misora Box Backup (bbstored)[32225]: Connection statistics for BACKUP-1: >>> IN=54273151997 OUT=2917099 TOTAL=54276069096 >>> >>> Session 2: >>> Jul 25 00:03:20 misora Box Backup (bbstored)[18666]: Incoming connection from 192.168.0.51 port >>> 51921 (handling in child 10676) >>> Jul 25 00:03:20 misora Box Backup (bbstored)[10676]: Certificate CN: BACKUP-1 >>> Jul 25 00:03:20 misora Box Backup (bbstored)[10676]: Login: Client ID 00000001, Read/Write >>> Jul 25 00:08:45 misora bbstored/hk[18667]: Starting housekeeping >>> Jul 25 00:09:45 misora bbstored/hk[18667]: Finished housekeeping >>> ....... >>> ....... >>> ....... >>> Jul 25 03:38:46 misora bbstored/hk[18667]: Starting housekeeping >>> Jul 25 03:39:46 misora bbstored/hk[18667]: Finished housekeeping >>> Jul 25 03:48:41 misora Box Backup (bbstored)[10676]: Session finished >>> Jul 25 03:48:41 misora Box Backup (bbstored)[10676]: Connection statistics for BACKUP-1: >>> IN=54428917970 OUT=3562128 TOTAL=54432480098 >>> >>> >>> Thanks, >>> Tobias >>> >>> >>> >>> Chris Wilson wrote: >>>> Hi Tobias, >>>> >>>> On Tue, 24 Jul 2007, Tobias Balle-Petersen wrote: >>>> >>>>> I reverted back to 0.10 client from chris_merge 1696. The redundant >>>>> locations were removed after 48 hours as they should be. My files are >>>>> now being backed up. >>>>> >>>>> The store is still chris_merge. >>>>> >>>>> So it would seem there was an error in client v. 1696. >>>> Indeed, thanks for letting me know, will look into it. >>>> >>>> Cheers, Chris. >>> _______________________________________________ >>> boxbackup mailing list >>> boxbackup at fluffy.co.uk >>> http://lists.warhead.org.uk/mailman/listinfo/boxbackup >>> >> _______________________________________________ >> boxbackup mailing list >> boxbackup at fluffy.co.uk >> http://lists.warhead.org.uk/mailman/listinfo/boxbackup >> > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Wed Jul 25 18:30:43 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 25 Jul 2007 18:30:43 +0100 (BST) Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <46A718BA.9070607@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> <46A718BA.9070607@kontrapunkt.com> Message-ID: Hi Tobias, > My files are now starting to be backed up. The backups stops after 4 > hours however. Looking at the stats for the store, data is beeing > transfered until the connection ends. As far as I can tell, this would > mean that this is not a SSL-timeout related problem. Client is 0.10 and > store is chris_merge 1725. > > I'm running at session again now. If it stops after 4 hours, I'm > expecting it will, I will revert to 0.10 on the store as well and run a > sync again. It hit a file with the 2GB bug and was unable to proceed further. This is why the new client is necessary. Reverting the store will not help. Now that your redundant locations are deleted, you can switch back to the new client. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Wed Jul 25 18:45:36 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 25 Jul 2007 18:45:36 +0100 (BST) Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <46A75362.8090203@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> <46A718BA.9070607@kontrapunkt.com> <46A74F30.1030707@kontrapunkt.com> <46A75362.8090203@kontrapunkt.com> Message-ID: Hi Tobias, > The 1696 client did nothing when attempting a sync. I'm now back at 0.10 > store and 0.10 client. Do you have logs of what was happening? I'm very surprised that it "did nothing". Could you run bbackupd from the command line with -DVT and see what it's doing, and tell me some info from the verbose debugging that will be output? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Wed Jul 25 21:44:46 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Wed, 25 Jul 2007 22:44:46 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> <46A718BA.9070607@kontrapunkt.com> Message-ID: <46A7B63E.9060405@kontrapunkt.com> Hello Chris. After approx. 4 hours the 0.10 store and 0.10 client combo fails with this in the log on the client: Jul 25 15:58:16 yoiko bbackupd[24426]: Beginning scan of local files Jul 25 15:58:16 yoiko bbackupd[24426]: Opening connection to server misora.local.kontrapunkt.dk... Jul 25 15:58:16 yoiko bbackupd[24426]: Connection made, login successful Jul 25 20:02:30 yoiko bbackupd[24426]: SSL err during Write: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry Jul 25 20:02:30 yoiko bbackupd[24426]: Exception caught (Connection TLSWriteFailed (Probably a network issue between client and server.) 7/33), reset state and waiting to retry... Jul 25 20:02:40 yoiko bbackupd[24426]: File statistics: total file size uploaded 86135671188, bytes already on server 0, encoded size 57007977101 This looks like the SSL timeout-problem to me. Chris Wilson wrote: > It hit a file with the 2GB bug and was unable to proceed further. But this always happens after 4 hours. Isn't that odd? > This > is why the new client is necessary. Reverting the store will not help. > Now that your redundant locations are deleted, you can switch back to > the new client. Back to the new client and 0.10 store running a sync. chris_merge client says nothing in the log when initiating a sync?? Looking at the store, I cen tell that the client has logged in. Odd ? Thanks, Tobias From boxbackup at fluffy.co.uk Wed Jul 25 21:50:46 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Wed, 25 Jul 2007 22:50:46 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> <46A718BA.9070607@kontrapunkt.com> <46A74F30.1030707@kontrapunkt.com> <46A75362.8090203@kontrapunkt.com> Message-ID: <46A7B7A6.4080501@kontrapunkt.com> Hello Chris. Chris Wilson wrote: > Do you have logs of what was happening? I'm very surprised that it "did > nothing". Could you run bbackupd from the command line with -DVT and see > what it's doing, and tell me some info from the verbose debugging that > will be output? Apparently chris_merge client says nothing in the log when initiating a sync while 0.10 does. This led me to believe the sync had not started. Tobias From boxbackup at fluffy.co.uk Wed Jul 25 22:06:52 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 25 Jul 2007 22:06:52 +0100 (BST) Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <46A7B63E.9060405@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> <46A718BA.9070607@kontrapunkt.com> <46A7B63E.9060405@kontrapunkt.com> Message-ID: Hi Tobias, On Wed, 25 Jul 2007, Tobias Balle-Petersen wrote: > After approx. 4 hours the 0.10 store and 0.10 client combo fails with > this in the log on the client: > > Jul 25 15:58:16 yoiko bbackupd[24426]: Beginning scan of local files > Jul 25 15:58:16 yoiko bbackupd[24426]: Opening connection to server misora.local.kontrapunkt.dk... > Jul 25 15:58:16 yoiko bbackupd[24426]: Connection made, login successful > Jul 25 20:02:30 yoiko bbackupd[24426]: SSL err during Write: error:1409F07F:SSL > routines:SSL3_WRITE_PENDING:bad write retry > Jul 25 20:02:30 yoiko bbackupd[24426]: Exception caught (Connection TLSWriteFailed (Probably a > network issue between client and server.) 7/33), reset state and waiting to retry... > Jul 25 20:02:40 yoiko bbackupd[24426]: File statistics: total file size uploaded 86135671188, bytes > already on server 0, encoded size 57007977101 > > This looks like the SSL timeout-problem to me. Yes, you're probably right, sorry. But it seems to think that it uploaded 57 GB of data? Is that possible in 4 hours? Do you have keepalives enabled? And could you try with ExtendedLogging turned on? > Chris Wilson wrote: >> It hit a file with the 2GB bug and was unable to proceed further. > > But this always happens after 4 hours. Isn't that odd? Yes, it is. But it might be hitting the same file at the same point in each scan. ExtendedLogging should help to debug this. Sorry for the problems. > Back to the new client and 0.10 store running a sync. chris_merge client > says nothing in the log when initiating a sync?? Looking at the store, I > cen tell that the client has logged in. Odd ? Yes, it is odd. Perhaps I haven't got the new logging code quite right? If you run bbackupd with the -V option, does it log any more info to the syslog during a sync? How do you start the sync? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Wed Jul 25 22:02:42 2007 From: boxbackup at fluffy.co.uk (E.W. Peter Jalajas) Date: Wed, 25 Jul 2007 14:02:42 -0700 (PDT) Subject: [Box Backup] Failed to setup location Message-ID: <875243.75032.qm@web60611.mail.yahoo.com> Hi all (and especially Chris?), We set up a Windows client with version chris_general_1692. The server is Box Backup Store Server vchris_merge_1725. Three other clients on that server are working fine. It seemed to work fine, got about 17GB stored (very quickly over client-side FIOS!), but a few days later I noticed that no new files were being added, and then the logs show we started getting the "Failed to setup location" error. Client and server logs below. I don't think it should be trying to setup that location, because it already exists with lots of files in it. That location is a mapped drive from another server on which we have full control rights. It may contain mapped locations below it (links to other servers in the network), which I know is a no-no, but I don't think that's our problem. We actually see the .lnk file in the bbackupquery backuplocation root, but we've deleted that file/link to try to fix the problem, to no avail. It could be that one or more bad files are being copied into that backup location via an automated script, but a look in the top directory showed all the files to be owned by the domain admin with full control rights. We tried to rename that BackupLocation by appending a "2" to its name, but that didn't help. So there is something wrong with that directory, but we don't know where to look for the problem. Any suggestions? Should I submit a bug that the logs give us no clue as to what is wrong with the location (and maybe that the error message is misleading)? Thanks, Pete The following event messages seem to be logged every 2 minutes (inverse chron order, newest at top): Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:52 AM User: N/A Computer: CLEANFILE04 Description: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Event Type: Error Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:42 AM User: N/A Computer: CLEANFILE04 Description: Exception caught (Common OSFileError (Error accessing a file. Check permissions.) 1/9), reset state and waiting to retry... Event Type: Error Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:42 AM User: N/A Computer: CLEANFILE04 Description: Not notifying administrator about event backup-error -- set NotifyScript to do this in future Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:42 AM User: N/A Computer: CLEANFILE04 Description: Receive Finished() Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:42 AM User: N/A Computer: CLEANFILE04 Description: Send Finished() Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:42 AM User: N/A Computer: CLEANFILE04 Description: Receive Success(0x4361899132bd3) Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:42 AM User: N/A Computer: CLEANFILE04 Description: Send SetClientStoreMarker(0x4361899132bd3) Event Type: Error Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:42 AM User: N/A Computer: CLEANFILE04 Description: Failed to setup location 'J04CommonFiles' path 'J:\04CommonFiles' Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:42 AM User: N/A Computer: CLEANFILE04 Description: Receiving stream, size 496 Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:41 AM User: N/A Computer: CLEANFILE04 Description: Receive Success(0x1) Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:41 AM User: N/A Computer: CLEANFILE04 Description: Send ListDirectory(0x1,0x2,0xc,false) Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:41 AM User: N/A Computer: CLEANFILE04 Description: Connection made, login successful Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:41 AM User: N/A Computer: CLEANFILE04 Description: Receive LoginConfirmed(0x436189074e355,0x1015977,0x1e00000,0x2300000) Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:41 AM User: N/A Computer: CLEANFILE04 Description: Send Login(0x10004018,0x0) Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:41 AM User: N/A Computer: CLEANFILE04 Description: Receive Version(0x1) Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:41 AM User: N/A Computer: CLEANFILE04 Description: Send Version(0x1) Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:41 AM User: N/A Computer: CLEANFILE04 Description: Opening connection to server ts4.tebucosafe.com... Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:59:41 AM User: N/A Computer: CLEANFILE04 Description: Beginning scan of local files Event Type: Information Event Source: Box Backup (bbackupd) Event Category: None Event ID: 1 Date: 7/25/2007 Time: 11:57:27 AM User: N/A Computer: CLEANFILE04 Description: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 petjal at ts4:~$ grep "\[3798\]" /var/log/messages Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Certificate CN: BACKUP-10004018 Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Receive Version(0x1) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Receive Version(0x1) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Send Version(0x1) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Send Version(0x1) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Receive Login(0x10004018,0x0) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Receive Login(0x10004018,0x0) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Login: Client ID 10004018, Read/Write Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Send LoginConfirmed(0x436196cadee84,0x1015977,0x1e00000,0x2300000) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Send LoginConfirmed(0x436196cadee84,0x1015977,0x1e00000,0x2300000) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Receive ListDirectory(0x1,0x2,0xc,false) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Receive ListDirectory(0x1,0x2,0xc,false) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Send Success(0x1) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Send Success(0x1) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Sending stream, size 496 Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Receive SetClientStoreMarker(0x436197263e2e8) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Receive SetClientStoreMarker(0x436197263e2e8) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Send Success(0x436197263e2e8) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Send Success(0x436197263e2e8) Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Receive Finished() Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Receive Finished() Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Session finished Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Send Finished() Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Send Finished() Jul 25 13:03:27 ts4 Box Backup (bbstored)[3798]: Connection statistics for BACKUP-10004018: IN=105 OUT=628 TOTAL=733 petjal at ts4:~$ From boxbackup at fluffy.co.uk Wed Jul 25 23:38:00 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 25 Jul 2007 23:38:00 +0100 (BST) Subject: [Box Backup] Failed to setup location In-Reply-To: <875243.75032.qm@web60611.mail.yahoo.com> References: <875243.75032.qm@web60611.mail.yahoo.com> Message-ID: Hi Pete, On Wed, 25 Jul 2007, E.W. Peter Jalajas wrote: > It seemed to work fine, got about 17GB stored (very quickly over > client-side FIOS!), but a few days later I noticed that no new files > were being added, and then the logs show we started getting the "Failed > to setup location" error. Client and server logs below. > > I don't think it should be trying to setup that location, because it > already exists with lots of files in it. > > That location is a mapped drive from another server on which we have > full control rights. It may contain mapped locations below it (links to > other servers in the network), which I know is a no-no, but I don't > think that's our problem. We actually see the .lnk file in the > bbackupquery backuplocation root, but we've deleted that file/link to > try to fix the problem, to no avail. It could be that one or more bad > files are being copied into that backup location via an automated > script, but a look in the top directory showed all the files to be owned > by the domain admin with full control rights. > > We tried to rename that BackupLocation by appending a "2" to its name, > but that didn't help. > > So there is something wrong with that directory, but we don't know where > to look for the problem. I think you're right. I'm surprised that no additional errors are logged. I'll have to test backing up mounted drives (I assume that J:\ is mapped from a network server?) and add some more debugging code when I have time. Sorry about this error. Does any older client work better for you? > Any suggestions? > > Should I submit a bug that the logs give us no clue as to what is wrong > with the location (and maybe that the error message is misleading)? I don't think the error is misleading, what makes you think that? I know that it doesn't give enough information in this case. Please do file a bug report about that. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Thu Jul 26 02:12:37 2007 From: boxbackup at fluffy.co.uk (E.W. Peter Jalajas) Date: Wed, 25 Jul 2007 18:12:37 -0700 (PDT) Subject: [Box Backup] Failed to setup location In-Reply-To: Message-ID: <108992.24686.qm@web60619.mail.yahoo.com> > > So there is something wrong with that directory, but we don't know where > > to look for the problem. > > I think you're right. I'm surprised that no additional errors are logged. > I'll have to test backing up mounted drives (I assume that J:\ is mapped > from a network server?) and add some more debugging code when I have time. > Sorry about this error. Does any older client work better for you? > Yes, J: is mapped to a share on another server in the Windows domain. The Box Backup service is running as the Domain Admin. OK, no problem, can you recommend an older client for us to try (and how I should get it [http link, svn command line, etc])? > > Should I submit a bug that the logs give us no clue as to what is wrong > > with the location (and maybe that the error message is misleading)? > > I don't think the error is misleading, what makes you think that? I know > that it doesn't give enough information in this case. Please do file a bug > report about that. I interpreted it to be misleading because the error said Failed to setup location 'J04CommonFiles' path 'J:\04CommonFiles' and I read that as that BB couldn't _create_ a _new_ BackupLocation, but in fact it shouldn't have been trying anyway and/or cuz that BackupLocation exists already. Based on my guess as to what is going on, I would think in this case it should report something about a bad file or link, and which one it was, or report something about why it couldn't create the BackupLocation. Thanks again, Chris, Pete From boxbackup at fluffy.co.uk Thu Jul 26 09:05:13 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Thu, 26 Jul 2007 09:05:13 +0100 (BST) Subject: [Box Backup] Failed to setup location In-Reply-To: <108992.24686.qm@web60619.mail.yahoo.com> References: <108992.24686.qm@web60619.mail.yahoo.com> Message-ID: Hi Pete, >> I think you're right. I'm surprised that no additional errors are >> logged. I'll have to test backing up mounted drives (I assume that J:\ >> is mapped from a network server?) and add some more debugging code when >> I have time. Sorry about this error. Does any older client work better >> for you? > > Yes, J: is mapped to a share on another server in the Windows domain. > The Box Backup service is running as the Domain Admin. > > OK, no problem, can you recommend an older client for us to try (and how > I should get it [http link, svn command line, etc])? Could you try 1569, and if that doesn't work 1280? You can get them from here: http://bbdev.fluffy.co.uk/trac/wiki/WindowsClientReleases Please download the clients as soon as possible as there is a scheduled downtime on our trac/svn server that might last all weekend. If one of those two versions does work for you, please could you try the intermediate ones to help narrow down where the problem appeared? > I interpreted it to be misleading because the error said > Failed to setup location 'J04CommonFiles' path 'J:\04CommonFiles' > and I read that as that BB couldn't _create_ a _new_ BackupLocation, but in fact it shouldn't have > been trying anyway and/or cuz that BackupLocation exists already. Yes, it's not supposed to mean that. How about "Failed to configure location" or "Failed to access location"? > Based on my guess as to what is going on, I would think in this case it > should report something about a bad file or link, and which one it was, > or report something about why it couldn't create the BackupLocation. Yes, it should have, in fact I couldn't yet find a code path where it doesn't report such an error, although I haven't searched exclusively. You do have write access to that directory, right? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Thu Jul 26 09:50:06 2007 From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen) Date: Thu, 26 Jul 2007 10:50:06 +0200 Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <46A718BA.9070607@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> <46A718BA.9070607@kontrapunkt.com> Message-ID: <46A8603E.20705@kontrapunkt.com> Hello Chris. I now see whats going on: When the client starts to upload to the store, the store is 96% full (soft limit). The upload continues until the store is full. Housekeeping on the store does not delete any "deleted" files while the client is logged in. The client gets disconnected when the store is full. The following housekeeping on the store will delete "deleted" files until the usage of the store is again 96% (soft limit). At this point I can initiate another sync. The 4 hours is the time it takes to move enough data to get from 96% to 99%(100%). Is this the way it is supposed to work? I guess I could lower the soft limit temporarely to get around this. Thanks, Tobias Tobias Balle-Petersen wrote: > Hello Chris. > > My files are now starting to be backed up. The backups stops after 4 hours however. Looking at the > stats for the store, data is beeing transfered until the connection ends. As far as I can tell, this > would mean that this is not a SSL-timeout related problem. Client is 0.10 and store is chris_merge > 1725. > > I'm running at session again now. If it stops after 4 hours, I'm expecting it will, I will revert to > 0.10 on the store as well and run a sync again. > > > Logs from the two sessions I have run below. > > ------------------------------------------- > CLIENT 0.10: > ------------------------------------------- > > Session 1: > Jul 24 13:00:29 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) > Jul 24 13:00:29 yoiko bbackupd[402]: Connection from command socket > Jul 24 13:00:29 yoiko bbackupd[402]: Beginning scan of local files > Jul 24 13:00:29 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... > Jul 24 13:00:30 yoiko bbackupd[402]: Connection made, login successful > Jul 24 17:01:39 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server > probably reported an error.) 7/47), reset state and waiting to retry... > Jul 24 17:01:49 yoiko bbackupd[402]: File statistics: total file size uploaded 93763175181, bytes > already on server 0, encoded size 54271648859 > > Session 2: > Jul 25 00:00:01 yoiko bbackupd[402]: Incoming connection from local (UNIX socket) > Jul 25 00:00:01 yoiko bbackupd[402]: Connection from command socket > Jul 25 00:00:01 yoiko bbackupd[402]: Beginning scan of local files > Jul 25 00:00:01 yoiko bbackupd[402]: Opening connection to server misora.local.kontrapunkt.dk... > Jul 25 00:00:01 yoiko bbackupd[402]: Connection made, login successful > Jul 25 03:45:21 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server > probably reported an error.) 7/47), reset state and waiting to retry... > Jul 25 03:45:31 yoiko bbackupd[402]: File statistics: total file size uploaded 81804900665, bytes > already on server 0, encoded size 54427385858 > > ------------------------------------------- > STORE chris_merge 1725: > ------------------------------------------- > > Session 1: > Jul 24 13:03:48 misora Box Backup (bbstored)[18666]: Incoming connection from 192.168.0.51 port > 63456 (handling in child 32225) > Jul 24 13:03:48 misora Box Backup (bbstored)[32225]: Certificate CN: BACKUP-1 > Jul 24 13:03:48 misora Box Backup (bbstored)[32225]: Login: Client ID 00000001, Read/Write > Jul 24 13:08:29 misora bbstored/hk[18667]: Starting housekeeping > Jul 24 13:09:29 misora bbstored/hk[18667]: Finished housekeeping > ....... > ....... > ....... > Jul 24 16:53:30 misora bbstored/hk[18667]: Starting housekeeping > Jul 24 16:54:30 misora bbstored/hk[18667]: Finished housekeeping > Jul 24 17:04:58 misora Box Backup (bbstored)[32225]: Session finished > Jul 24 17:04:58 misora Box Backup (bbstored)[32225]: Connection statistics for BACKUP-1: > IN=54273151997 OUT=2917099 TOTAL=54276069096 > > Session 2: > Jul 25 00:03:20 misora Box Backup (bbstored)[18666]: Incoming connection from 192.168.0.51 port > 51921 (handling in child 10676) > Jul 25 00:03:20 misora Box Backup (bbstored)[10676]: Certificate CN: BACKUP-1 > Jul 25 00:03:20 misora Box Backup (bbstored)[10676]: Login: Client ID 00000001, Read/Write > Jul 25 00:08:45 misora bbstored/hk[18667]: Starting housekeeping > Jul 25 00:09:45 misora bbstored/hk[18667]: Finished housekeeping > ....... > ....... > ....... > Jul 25 03:38:46 misora bbstored/hk[18667]: Starting housekeeping > Jul 25 03:39:46 misora bbstored/hk[18667]: Finished housekeeping > Jul 25 03:48:41 misora Box Backup (bbstored)[10676]: Session finished > Jul 25 03:48:41 misora Box Backup (bbstored)[10676]: Connection statistics for BACKUP-1: > IN=54428917970 OUT=3562128 TOTAL=54432480098 > > > Thanks, > Tobias > > > > Chris Wilson wrote: >> Hi Tobias, >> >> On Tue, 24 Jul 2007, Tobias Balle-Petersen wrote: >> >>> I reverted back to 0.10 client from chris_merge 1696. The redundant >>> locations were removed after 48 hours as they should be. My files are >>> now being backed up. >>> >>> The store is still chris_merge. >>> >>> So it would seem there was an error in client v. 1696. >> Indeed, thanks for letting me know, will look into it. >> >> Cheers, Chris. > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Thu Jul 26 13:26:56 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Thu, 26 Jul 2007 13:26:56 +0100 (BST) Subject: [Box Backup] Redundant locations not removed? In-Reply-To: <46A8603E.20705@kontrapunkt.com> References: <469CE4C2.90602@kontrapunkt.com> <469E58AC.7000702@kontrapunkt.com> <469E7AC1.2090208@kontrapunkt.com> <469F5D31.90107@kontrapunkt.com> <46A5DF82.1060809@kontrapunkt.com> <46A718BA.9070607@kontrapunkt.com> <46A8603E.20705@kontrapunkt.com> Message-ID: Hi Tobias, > When the client starts to upload to the store, the store is 96% full > (soft limit). The upload continues until the store is full. Housekeeping > on the store does not delete any "deleted" files while the client is > logged in. The client gets disconnected when the store is full. The > following housekeeping on the store will delete "deleted" files until > the usage of the store is again 96% (soft limit). At this point I can > initiate another sync. The 4 hours is the time it takes to move enough > data to get from 96% to 99%(100%). > > Is this the way it is supposed to work? > > I guess I could lower the soft limit temporarely to get around this. That's very interesting, the errors in the logs put me off suspecting this. Jul 25 20:02:30 yoiko bbackupd[24426]: SSL err during Write: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry Jul 25 20:02:30 yoiko bbackupd[24426]: Exception caught (Connection TLSWriteFailed (Probably a network issue between client and server.) 7/33), reset state and waiting to retry... Did you have anything in the server logs around this time, lime an exception? Also, we should probably catch this one and report the error properly: Jul 25 10:59:54 yoiko bbackupd[402]: Connection made, login successful Jul 25 15:06:14 yoiko bbackupd[402]: Exception caught (Connection Protocol_UnexpectedReply (Server probably reported an error.) 7/47), reset state and waiting to retry... Older clients (e.g. 0.10) should not back up above the soft limit, so they would never have got so far. Newer clients (like my branch) will ignore the soft limit and back up to the hard limit. So you should probably lower the soft limit permanently to ensure that there's always free space after housekeeping for however much new data needs to be uploaded. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Thu Jul 26 16:28:53 2007 From: boxbackup at fluffy.co.uk (E.W. Peter Jalajas) Date: Thu, 26 Jul 2007 08:28:53 -0700 (PDT) Subject: [Box Backup] Failed to setup location In-Reply-To: Message-ID: <763531.43506.qm@web60625.mail.yahoo.com> Thanks, Chris. > Please download the clients as soon as possible as there is a scheduled > downtime on our trac/svn server that might last all weekend. OK, I've downloaded all three, 1569, 1516, and 1280. (We tried 1662 last night to no avail; see the bug report.) > > I interpreted it to be misleading because the error said > > Failed to setup location 'J04CommonFiles' path 'J:\04CommonFiles' > > and I read that as that BB couldn't _create_ a _new_ BackupLocation, but in fact it shouldn't > have > > been trying anyway and/or cuz that BackupLocation exists already. > > Yes, it's not supposed to mean that. How about "Failed to configure > location" or "Failed to access location"? Oh, I think maybe I see now. How about just saying in English what the code is doing at that step (or just spitting out the code line if that is very easy, for now)? I still can't tell at the moment in your suggested phrasing whether the "configure" or "access" failed on the client box or the server, and/or whether it is the "BackupLocation" container-type thing or the OS "Path" or files that is/are having the problem. > > Based on my guess as to what is going on, I would think in this case it > > should report something about a bad file or link, and which one it was, > > or report something about why it couldn't create the BackupLocation. > > Yes, it should have, in fact I couldn't yet find a code path where it > doesn't report such an error, although I haven't searched exclusively. There's just a handful of directories and files at the root of the Path. I wish I had a tool of some sort to test each one for some appropriate property. We tried several different variants of things like (from my failing human memory): attrib (all in the top level had just the "A" flag) dir /q (all directories were BUILTIN\Administrators and files were DOMAIN\Administrators Can anyone think of any other tests we can run on these problematic files and directories to help show why they cause Box Backup to fail (as compared to those that do not fail)? > You do have write access to that directory, right? Yes, we can add and delete files in the top level of the BackupLocation Path on the client machine. I'll have to triple check that "we" are who we think we are..., but we've given the DOMAIN\Admin Full Control to those shares and we set up the Windows Service to run as that same DOMAIN\Admin. Um, I forget, why do we have to have write access again...? Thanks again, Chris. I appreciate all the help. Pete From boxbackup at fluffy.co.uk Thu Jul 26 21:23:05 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Thu, 26 Jul 2007 21:23:05 +0100 (BST) Subject: [Box Backup] Failed to setup location In-Reply-To: <763531.43506.qm@web60625.mail.yahoo.com> References: <763531.43506.qm@web60625.mail.yahoo.com> Message-ID: Hi Pete, > OK, I've downloaded all three, 1569, 1516, and 1280. (We tried 1662 > last night to no avail; see the bug report.) Thanks. I hope one of them works for you! >> Yes, it's not supposed to mean that. How about "Failed to configure >> location" or "Failed to access location"? > > Oh, I think maybe I see now. How about just saying in English what the > code is doing at that step (or just spitting out the code line if that > is very easy, for now)? I still can't tell at the moment in your > suggested phrasing whether the "configure" or "access" failed on the > client box or the server, and/or whether it is the "BackupLocation" > container-type thing or the OS "Path" or files that is/are having the > problem. That's a catch-all exception handler. There are not supposed to be any unhandled exceptions in that block which don't produce a more specific error message. The fact that there is one means that I have to double-check all code paths that might throw that error to see why one is being thrown and no error message produced. It might be an error on the client or the server, but the actual error in this case (Common OSFileError) seems to imply that it's an error on the client. > Can anyone think of any other tests we can run on these problematic > files and directories to help show why they cause Box Backup to fail (as > compared to those that do not fail)? No, sorry, I can't think of anything right now. I'll add some more debugging code which I hope will help, and test backups of mounted network drives. >> You do have write access to that directory, right? > > Yes, we can add and delete files in the top level of the BackupLocation > Path on the client machine. I'll have to triple check that "we" are who > we think we are..., but we've given the DOMAIN\Admin Full Control to > those shares and we set up the Windows Service to run as that same > DOMAIN\Admin. > > Um, I forget, why do we have to have write access again...? You shouldn't need write access, but it's one thing to eliminate, which I think you have done. Are you running Box Backup as a service? And does the service user (e.g. LocalSystem) have write access there? Does the location work if you run the backup from the command line as a normal user? > Thanks again, Chris. I appreciate all the help. Thank you for your patience and help! Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Thu Jul 26 21:45:22 2007 From: boxbackup at fluffy.co.uk (E.W. Peter Jalajas) Date: Thu, 26 Jul 2007 13:45:22 -0700 (PDT) Subject: [Box Backup] Failed to setup location In-Reply-To: Message-ID: <819133.80520.qm@web60618.mail.yahoo.com> Thanks again, Chris. >> OK, I've downloaded all three, 1569, 1516, and 1280. (We tried 1662 >> last night to no avail; see the bug report.) > Thanks. I hope one of them works for you! My process for jumping between these versions is: 1. Stop service. net stop boxbackup 2. Remove service. bbackupd -r 2b. Check to make sure bbackupd process is not still running. 3. Rename 3 bb*.exe files from current/old BB version to bb*_old.exe 4. Copy in 3 bb.exe files from new BB version. Not touching .conf, .dlls or others. 5. Install service. bbackupd -i <.conf file> (full path to .conf file, in quotes; necessity depends on version). 6. Start service. net start boxbackup > Are you running Box Backup as a service? And does the service user (e.g. > LocalSystem) have write access there? Yes, and yes. We had the service running as DOMAIN\Admin, though. > Does the location work if you run > the backup from the command line as a normal user? Good idea. We'll try that and report back. Thanks, Pete From boxbackup at fluffy.co.uk Sun Jul 29 23:23:41 2007 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 29 Jul 2007 23:23:41 +0100 (BST) Subject: [Box Backup] Failed to setup location In-Reply-To: <819133.80520.qm@web60618.mail.yahoo.com> References: <819133.80520.qm@web60618.mail.yahoo.com> Message-ID: Hi Pete, On Thu, 26 Jul 2007, E.W. Peter Jalajas wrote: > Thanks again, Chris. > >>> OK, I've downloaded all three, 1569, 1516, and 1280. (We tried 1662 >>> last night to no avail; see the bug report.) Please could you also try the latest release, 1781? It probably won't fix the problem but it does include much more debugging information in this case, which should help me to track down the problem. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Tue Jul 31 10:50:14 2007 From: boxbackup at fluffy.co.uk (Nuno Fernandes) Date: Tue, 31 Jul 2007 10:50:14 +0100 Subject: [Box Backup] Question about certificates Message-ID: <200707311050.15027.npf-mlists@eurotux.com> Hi, I have a question about managing certificates. We have a company CA and we've created a sub-ca just for boxbackup. How can we use it in bbstored-certs script? Aparently bbstored-certs /etc/box/bbstored/certs init creates 2 root CAs (one for clients and the other for servers). Why does it create 2 CAs? Is there any way i can use the Sub-CA created previously in bbstored-certs script? Thanks Nuno Fernandes From boxbackup at fluffy.co.uk Tue Jul 31 14:15:40 2007 From: boxbackup at fluffy.co.uk (Baltasar Cevc) Date: Tue, 31 Jul 2007 15:15:40 +0200 Subject: [Box Backup] Question about certificates In-Reply-To: <200707311050.15027.npf-mlists@eurotux.com> References: <200707311050.15027.npf-mlists@eurotux.com> Message-ID: <20070731151540.2c021ea2@pc-buna.former03.local> On Tue, 31 Jul 2007 10:50:14 +0100 Nuno Fernandes wrote: > Hi, > > I have a question about managing certificates. We have a company CA > and we've created a sub-ca just for boxbackup. > > How can we use it in bbstored-certs script? I assume that it should be enough to replace the files in the CA directory by your CAs (see below about the CAs), I'm not sure though. > Aparently > bbstored-certs /etc/box/bbstored/certs init > creates 2 root CAs (one for clients and the other for servers). Why > does it create 2 CAs? One is for validating servers, the other for validating clients. I think servers are just accepted as valid if they present a valid certificate signed by the server CA. For clients, the CN must match - I think it must BACKUP- (without zeros at the beginning), the certificate being signed by the client CA. Hope that helps, Baltasar