From boxbackup at fluffy.co.uk Thu Apr 3 09:24:57 2008 From: boxbackup at fluffy.co.uk (Tom Albers) Date: Thu, 03 Apr 2008 10:24:57 +0200 Subject: [Box Backup] Nice statistics. Message-ID: <1737485.bvOL2Q6ygN@kovoks.nl> --nextPart14207958.Wnf0qa4GBB Content-Type: text/plain Content-Transfer-Encoding: 7Bit Hi, Did anyone write some nice statistics about the daily result of the backup on the client side? It isn't that hard, because all information is now available in the logfiles. If anyone has any scripts, please make them available, maybe we can all improve it and have a great tool for this. Best, Tom Albers -- KovoKs B.V. KvK: 11033334 --nextPart14207958.Wnf0qa4GBB-- From boxbackup at fluffy.co.uk Thu Apr 3 10:22:18 2008 From: boxbackup at fluffy.co.uk (Kenny Millington) Date: Thu, 03 Apr 2008 10:22:18 +0100 Subject: [Box Backup] Nice statistics. In-Reply-To: <1737485.bvOL2Q6ygN@kovoks.nl> References: <1737485.bvOL2Q6ygN@kovoks.nl> Message-ID: <1207214538.2348.62.camel@helios> Hi, > Did anyone write some nice statistics about the daily result of the backup on the client side? > It isn't that hard, because all information is now available in the logfiles. If anyone has any scripts, > please make them available, maybe we can all improve it and have a great tool for this. I originally posted this back in October last year - hope it's useful:- I've written a (python) script based on the perl script Matt Brown posted to the list sometime ago for producing (e-mail) reports from BoxBackup logs. Alot of credit for this work goes to Matt for both the original idea and for providing a reasonable amount of testing of this script. The script (bbreporter.py) is being released under the GNU GPL v3 (or above) and is available from the following link:- http://www.3ait.co.uk/misc/bbreporter.py It's been tested on both Debian Sarge and Ubuntu Dapper. I don't believe it requires any python libraries not included by default. To use this script you need to make use of the BoxBackup config option "LogAllFileAccess = yes" and run the daemon with the -v command line switch. Running "bbreporter.py --help" should get you started (once you have log files to parse). The script is reasonably well commented where necessary so it shouldn't be too difficult to see how things work. (Infact I think I waffle on a bit too much about log rotation in the comments but I decided to leave it in. :-)) As far as log rotation goes... I didn't go for any of the options in the script comments, I elected to do the local6.* -/var/log/box syslog rule and add a local6.none to the /var/log/syslog line so that the boxbackup log messages don't clutter the syslog (especially now LogAllFileAccess is on). Right that's it, I hope it works for everyone and people find it useful. Patches are ofcourse welcome. If anyone has any questions I'll certainly do my best to answer them. -- Kenny Millington Systems Developer kenny.millington at 3ait.co.uk 3aIT Limited - Official Corporate Sponsor of the British Bobsleigh Team 4-10 Barttelot Rd Horsham West Sussex RH12 1DQ CoReg: 3866698 VATReg: 771388600 T: +44 (0)870 881 5097 F: +44 (0)870 116 0793 Visit www.3aIT.co.uk for Design, Systems, Support Disclaimer: The information contained within this email is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, any disclosure, copying or distribution of this email is prohibited and may be unlawful. The content of this email represents the views of the individual and not necessarily 3aIT Limited. 3aIT Limited reserves the right to monitor the content of all emails in accordance with lawful business practice. Whilst every effort is made to ensure that attachments are free from computer viruses before transmission, 3aIT Limited does not accept any liability in respect of any virus that is not detected. From boxbackup at fluffy.co.uk Thu Apr 3 13:47:48 2008 From: boxbackup at fluffy.co.uk (Tom Albers) Date: Thu, 03 Apr 2008 14:47:48 +0200 Subject: [Box Backup] Nice statistics. In-Reply-To: <1207214538.2348.62.camel@helios> Message-ID: <1361909.p7P3QKfTA7@kovoks.nl> --nextPart5142610.exSsSGF1Wt Content-Type: text/plain Content-Transfer-Encoding: 7Bit At Thursday 03 April 2008 11:22, you wrote: > I originally posted this back in October last year - hope it's useful:- Wow. I knew I saw something like it earlier, but could not find it in the archives. > It's been tested on both Debian Sarge and Ubuntu Dapper. I don't believe > it requires any python libraries not included by default. Installed on SUSE LINUX 10.1, runs out of the box too. > The script is reasonably well commented where necessary so it shouldn't > be too difficult to see how things work. (Infact I think I waffle on a > bit too much about log rotation in the comments but I decided to leave > it in. :-)) No You Don't. It's perfect as it is. Got me going in minutes. Keep it in. > As far as log rotation goes... I didn't go for any of the options in the > script comments, I elected to do the local6.* -/var/log/box syslog rule > and add a local6.none to the /var/log/syslog line so that the boxbackup > log messages don't clutter the syslog (especially now LogAllFileAccess > is on). Maybe you can add something for syslog-ng users? Something like: You can remoce the local6 from an existing filter if you wish adn add a new filter for the local6 facility: filter f_box { facility(local6); }; After that add a new destination and make sure the filter logs to that destionation: destination boxmessages { file("/var/log/box"); }; log { source(src); filter(f_box); destination(boxmessages); }; > Right that's it, I hope it works for everyone and people find it useful. > Patches are ofcourse welcome. Python is not my best language, but the report looks great, I'll let it run for a while ans see if the customers have feedback on it. If so, I'll try to patch it or let you know. Thanks for writing the script both! > If anyone has any questions I'll certainly do my best to answer them. Not completely related to the script, but can you tell me what '113182 file(s) were synchronised.' means? Something like 'checked and up to date' ? And I find '102400.0Mb' slightly hard to read (output of bbackupquery status i think). 100GB would be better. Also I'd love to make a dutch translation. Is it i18n aware? If so, do you have a pot file to translate? Best, Tom Albers -- KovoKs B.V. KvK: 11033334 --nextPart5142610.exSsSGF1Wt-- From boxbackup at fluffy.co.uk Thu Apr 3 14:09:54 2008 From: boxbackup at fluffy.co.uk (Kenny Millington) Date: Thu, 03 Apr 2008 14:09:54 +0100 Subject: [Box Backup] Nice statistics. In-Reply-To: <1361909.p7P3QKfTA7@kovoks.nl> References: <1361909.p7P3QKfTA7@kovoks.nl> Message-ID: <1207228194.2348.72.camel@helios> Hi, > Installed on SUSE LINUX 10.1, runs out of the box too. Good to know. > Maybe you can add something for syslog-ng users? Something like: Yeah, I probably should. Is syslog-ng the default for SuSE (or any distro) or does it have to be explicitly chosen by the sysadmin? > Not completely related to the script, but can you tell me what '113182 file(s) were synchronised.' means? > Something like 'checked and up to date' ? And I find '102400.0Mb' slightly hard to read (output of bbackupquery status > i think). 100GB would be better. Correct on both counts (I think - if not I assume someone will jump in and correct me quite quickly):- Synchronised files are ones which were checked and found to be up to date. The 102400.0Mb figure is pulled directly from bbackupquery so any changes to that outputting in GB should be directed at that application. *poke Chris* ;-) > Also I'd love to make a dutch translation. > Is it i18n aware? If so, do you have a pot file to translate? Um, no - I'm afraid it's not i18n aware... Although, Googling around it looks like it could be done without too much work - Python appears to have functions for it. -- Kenny Millington Systems Developer kenny.millington at 3ait.co.uk 3aIT Limited - Official Corporate Sponsor of the British Bobsleigh Team 4-10 Barttelot Rd Horsham West Sussex RH12 1DQ CoReg: 3866698 VATReg: 771388600 T: +44 (0)870 881 5097 F: +44 (0)870 116 0793 Visit www.3aIT.co.uk for Design, Systems, Support Disclaimer: The information contained within this email is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, any disclosure, copying or distribution of this email is prohibited and may be unlawful. The content of this email represents the views of the individual and not necessarily 3aIT Limited. 3aIT Limited reserves the right to monitor the content of all emails in accordance with lawful business practice. Whilst every effort is made to ensure that attachments are free from computer viruses before transmission, 3aIT Limited does not accept any liability in respect of any virus that is not detected. From boxbackup at fluffy.co.uk Thu Apr 3 23:32:59 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Thu, 3 Apr 2008 23:32:59 +0100 (BST) Subject: [Box Backup] Nice statistics. In-Reply-To: <1207228194.2348.72.camel@helios> References: <1361909.p7P3QKfTA7@kovoks.nl> <1207228194.2348.72.camel@helios> Message-ID: Hi Kenny, On Thu, 3 Apr 2008, Kenny Millington wrote: > > Not completely related to the script, but can you tell me what '113182 > > file(s) were synchronised.' means? Something like 'checked and up to > > date' ? And I find '102400.0Mb' slightly hard to read (output of > > bbackupquery status i think). 100GB would be better. > > Correct on both counts (I think - if not I assume someone will jump in > and correct me quite quickly):- > > Synchronised files are ones which were checked and found to be up to > date. Synchronised means that they are now up to date on the store. Either they were uploaded, or they had not changed since the last upload, but in either case the latest version is backed up and protected on the server. > The 102400.0Mb figure is pulled directly from bbackupquery so any > changes to that outputting in GB should be directed at that application. > > *poke Chris* ;-) Fixed in trunk :-) Will be in 0.11. New output looks like this: $ bbstoreaccounts -c testfiles/bbstored.conf info 1234567 Account ID: 0x01234567 Last object ID: 0x1d Used: 71 blocks, 142.00 kB, 3% | | Old files: 0 blocks, 0.00 B, 0% | | Deleted files: 0 blocks, 0.00 B, 0% | | Directories: 24 blocks, 48.00 kB, 1% | | Soft limit: 1000 blocks, 1.95 MB, 50% |******** | Hard limit: 2000 blocks, 3.91 MB, 100% |****************| Client store marker: 29 $ bbackupquery -c testfiles/bbackupd.conf -q usage quit Used: 71 blocks, 142.00 kB, 3% | | Old files: 0 blocks, 0.00 B, 0% | | Deleted files: 0 blocks, 0.00 B, 0% | | Directories: 24 blocks, 48.00 kB, 1% | | Soft limit: 1000 blocks, 1.95 MB, 50% |******** | Hard limit: 2000 blocks, 3.91 MB, 100% |****************| (I hope my mail client doesn't wrap the fine ascii art :-) Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Fri Apr 4 08:53:40 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Fri, 4 Apr 2008 08:53:40 +0100 Subject: [Box Backup] Nice statistics. In-Reply-To: <1207214538.2348.62.camel@helios> References: <1737485.bvOL2Q6ygN@kovoks.nl> <1207214538.2348.62.camel@helios> Message-ID: <7776814A-3E84-4057-8B61-C9D8D57640DE@mbrown.co.uk> Hi Kenny/Chris, > > I've written a (python) script based on the perl script Matt Brown > posted to the list sometime ago for producing (e-mail) reports from > BoxBackup logs. > Chris, does Box have a contrib ? Just thinking provided Kenny is agreeable could this be included to say the 0.12 (or 0.11) client package/src/parcel ? I personally have been using the script for months for Clients and they like the reassurance of seeing daily backup info, others I think would benefit also. Regards Matt From boxbackup at fluffy.co.uk Fri Apr 4 09:37:06 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 4 Apr 2008 09:37:06 +0100 (BST) Subject: [Box Backup] Nice statistics. In-Reply-To: <7776814A-3E84-4057-8B61-C9D8D57640DE@mbrown.co.uk> References: <1737485.bvOL2Q6ygN@kovoks.nl> <1207214538.2348.62.camel@helios> <7776814A-3E84-4057-8B61-C9D8D57640DE@mbrown.co.uk> Message-ID: Hi Matt, On Fri, 4 Apr 2008, Matt Brown wrote: > Chris, does Box have a contrib ? Just thinking provided Kenny is > agreeable could this be included to say the 0.12 (or 0.11) client > package/src/parcel ? > > I personally have been using the script for months for Clients and they > like the reassurance of seeing daily backup info, others I think would > benefit also. Yes, see /contrib. I'd be very happy to accept the script. What license is it under? GPLv3? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Fri Apr 4 09:44:03 2008 From: boxbackup at fluffy.co.uk (Kenny Millington) Date: Fri, 04 Apr 2008 09:44:03 +0100 Subject: [Box Backup] Nice statistics. In-Reply-To: References: <1737485.bvOL2Q6ygN@kovoks.nl> <1207214538.2348.62.camel@helios> <7776814A-3E84-4057-8B61-C9D8D57640DE@mbrown.co.uk> Message-ID: <1207298643.2348.90.camel@helios> Hi, > Yes, see /contrib. I'd be very happy to accept the script. What license is > it under? GPLv3? Yep - GPLv3. So feel free to include it in contrib/ :-) As an aside, is your source-control fine-grained enough to give me commit access to the script or should I just chuck patches at you as and when? (Not that I've changed it at all recently, mind...) -- Kenny Millington Systems Developer kenny.millington at 3ait.co.uk 3aIT Limited - Official Corporate Sponsor of the British Bobsleigh Team 4-10 Barttelot Rd Horsham West Sussex RH12 1DQ CoReg: 3866698 VATReg: 771388600 T: +44 (0)870 881 5097 F: +44 (0)870 116 0793 Visit www.3aIT.co.uk for Design, Systems, Support Disclaimer: The information contained within this email is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, any disclosure, copying or distribution of this email is prohibited and may be unlawful. The content of this email represents the views of the individual and not necessarily 3aIT Limited. 3aIT Limited reserves the right to monitor the content of all emails in accordance with lawful business practice. Whilst every effort is made to ensure that attachments are free from computer viruses before transmission, 3aIT Limited does not accept any liability in respect of any virus that is not detected. From boxbackup at fluffy.co.uk Fri Apr 4 11:25:31 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 4 Apr 2008 11:25:31 +0100 (BST) Subject: [Box Backup] Nice statistics. In-Reply-To: <1207298643.2348.90.camel@helios> References: <1737485.bvOL2Q6ygN@kovoks.nl> <1207214538.2348.62.camel@helios> <7776814A-3E84-4057-8B61-C9D8D57640DE@mbrown.co.uk> <1207298643.2348.90.camel@helios> Message-ID: Hi Kenny, On Fri, 4 Apr 2008, Kenny Millington wrote: > Yep - GPLv3. > > So feel free to include it in contrib/ :-) Thanks, will do unless anyone has any objections to the license (speak now, or forever hold your piece :-) > As an aside, is your source-control fine-grained enough to give me > commit access to the script Sorry, I'm not sure how to do that with svn. > or should I just chuck patches at you as and when? Please do. > (Not that I've changed it at all recently, mind...) You may have to now that the output format from bbackupquery has changed. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Fri Apr 4 12:15:13 2008 From: boxbackup at fluffy.co.uk (James O'Gorman) Date: Fri, 4 Apr 2008 12:15:13 +0100 Subject: [Box Backup] Nice statistics. In-Reply-To: References: <1737485.bvOL2Q6ygN@kovoks.nl> <1207214538.2348.62.camel@helios> <7776814A-3E84-4057-8B61-C9D8D57640DE@mbrown.co.uk> <1207298643.2348.90.camel@helios> Message-ID: <20080404111513.GR15350@netinertia.co.uk> On Fri, Apr 04, 2008 at 11:25:31AM +0100, Chris Wilson wrote: > > As an aside, is your source-control fine-grained enough to give me > > commit access to the script > > Sorry, I'm not sure how to do that with svn. It's pretty straightforward. I'll try and set this up later. James From boxbackup at fluffy.co.uk Fri Apr 4 19:39:55 2008 From: boxbackup at fluffy.co.uk (Peter Jalajas, GigaLock Backup Services) Date: Fri, 4 Apr 2008 14:39:55 -0400 Subject: [Box Backup] 0.11 precompiled windows client binary to test? Message-ID: <74d01c7a0804041139m62b0e56as50ea003cdf547b1a@mail.gmail.com> Hi Chris, Have you a 0.11rc Windows binary that I may test? Thanks, Pete From boxbackup at fluffy.co.uk Fri Apr 4 19:44:26 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 4 Apr 2008 19:44:26 +0100 (BST) Subject: [Box Backup] 0.11 precompiled windows client binary to test? In-Reply-To: <74d01c7a0804041139m62b0e56as50ea003cdf547b1a@mail.gmail.com> References: <74d01c7a0804041139m62b0e56as50ea003cdf547b1a@mail.gmail.com> Message-ID: Hi Pete, On Fri, 4 Apr 2008, Peter Jalajas, GigaLock Backup Services wrote: > Have you a 0.11rc Windows binary that I may test? Unfortunately not yet, I haven't finished merging my windows branch into the trunk, I hope to do this before releasing the next release candidate. I can supply a binary early if you like, but it may be missing some features with respect to the previous Windows builds, although all tests do pass. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Fri Apr 4 19:54:44 2008 From: boxbackup at fluffy.co.uk (Peter Jalajas, GigaLock Backup Services) Date: Fri, 4 Apr 2008 14:54:44 -0400 Subject: [Box Backup] 0.11 precompiled windows client binary to test? In-Reply-To: References: <74d01c7a0804041139m62b0e56as50ea003cdf547b1a@mail.gmail.com> Message-ID: <74d01c7a0804041154r6a8e81aeid653c31c7f43508b@mail.gmail.com> On Fri, Apr 4, 2008 at 2:44 PM, Chris Wilson wrote: > > Have you a 0.11rc Windows binary that I may test? > > Unfortunately not yet, I haven't finished merging my windows branch into > the trunk, I hope to do this before releasing the next release candidate. > I can supply a binary early if you like, but it may be missing some > features with respect to the previous Windows builds, although all tests > do pass. I'm flexible. Whatever helps you the most. From boxbackup at fluffy.co.uk Mon Apr 7 17:34:11 2008 From: boxbackup at fluffy.co.uk (Peter Jalajas, GigaLock Backup Services) Date: Mon, 7 Apr 2008 12:34:11 -0400 Subject: [Box Backup] Restoring old directory tree? Message-ID: <74d01c7a0804070934g1180506ek9053c3d300c783c6@mail.gmail.com> Hi all, I have a client that accidentally deleted a directory tree on Windows last Friday. Since then, newer versions of files have been created in that deleted directory tree and backed up to the store. The client wants to restore the versions of the files that were in the deleted directory tree at the time of deletion of the directory tree. We've run bbackupquery "restore -d -i 00004c0c restored" quit and that seemed to restore scores of files. The client is reviewing them now to see what they all had in common, to see if they meet his needs. Can someone explain what restoring a deleted directory tree does with respect to old, deleted and current files in that directory tree? I'm so fuzzy about how directories (old or deleted or even current) work in Box Backup, that I'm not even sure I'm asking the question properly. Thanks! Pete From boxbackup at fluffy.co.uk Mon Apr 7 21:53:02 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Mon, 7 Apr 2008 21:53:02 +0100 (BST) Subject: [Box Backup] Restoring old directory tree? In-Reply-To: <74d01c7a0804070934g1180506ek9053c3d300c783c6@mail.gmail.com> References: <74d01c7a0804070934g1180506ek9053c3d300c783c6@mail.gmail.com> Message-ID: Hi Pete, On Mon, 7 Apr 2008, Peter Jalajas, GigaLock Backup Services wrote: > I have a client that accidentally deleted a directory tree on Windows > last Friday. Since then, newer versions of files have been created in > that deleted directory tree and backed up to the store. The client > wants to restore the versions of the files that were in the deleted > directory tree at the time of deletion of the directory tree. > > We've run > bbackupquery "restore -d -i 00004c0c restored" quit > and that seemed to restore scores of files. The client is reviewing > them now to see what they all had in common, to see if they meet his > needs. > > Can someone explain what restoring a deleted directory tree does with > respect to old, deleted and current files in that directory tree? I'm > so fuzzy about how directories (old or deleted or even current) work in > Box Backup, that I'm not even sure I'm asking the question properly. I'm not 100% sure either, but I just read the restore code and here is my understanding: When bbackupd detects that a directory no longer exists, it will recurse down the directories on the server marking them and their contents as deleted. The latest version of each file that exists at that time is marked as deleted, rather than old. (I assume that bbackupd completed a sync before the deleted directory was recreated, otherwise the directory would not be marked as deleted on the store). Time passes. Housekeeping on the server may remove files which are marked as old and deleted if needed to keep the account below its soft limit. bbackupd detects that a directory has been created. It doesn't care that an old, deleted directory with the same name exists on the store. It uploads the directory and its contents without the deleted flag. The newly uploaded directory has a new object ID and its contents are kept separate from the old, deleted directory. The administrator runs the bbackupquery "restore" command with the -d flag and specifies the object ID of the deleted directory with the -i flag. bbackupquery reads that directory, and recurses down it, looking for files and directories which are flagged as "deleted" and not "old" (and by definition which have not already been removed by housekeeping) and restores those files. My understanding is that you will get a complete restore unless/except: The directory was recreated with new files before a bbackupd sync ran (or possibly during the sync). In this case, the top-level directory will not be marked as deleted, and if some of the files that it contained were replaced by files with the same name, then the versions that were deleted will now be marked as "old" instead of "deleted" (because bbackupd never had a chance to see that they were missing) and will be excluded from "restore -d", although the files may still on the store (unless removed by housekeeping) and can still be restored manually. Housekeeping has purged some of the deleted files. In this case I'm afraid you will not be able to restore them and you will not know that it has happened except that files are missing from the restore. If neither of these things has happened then I believe that you will be able to restore the directory as it was at the last sync before it was deleted, by doing as you have done, "restore -d" on the deleted directory ID. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Tue Apr 8 12:05:06 2008 From: boxbackup at fluffy.co.uk (Ben Summers) Date: Tue, 8 Apr 2008 12:05:06 +0100 Subject: [Box Backup] Restoring old directory tree? Message-ID: <0E064195-4EA3-468E-A041-F20463B43920@fluffy.co.uk> Chris wrote: > My understanding is that you will get a complete restore unless/ > except: > > The directory was recreated with new files before a bbackupd sync > ran (or > possibly during the sync). In this case, the top-level directory > will not > be marked as deleted, and if some of the files that it contained were > replaced by files with the same name, then the versions that were > deleted > will now be marked as "old" instead of "deleted" (because bbackupd > never > had a chance to see that they were missing) and will be excluded from > "restore -d", although the files may still on the store (unless > removed by > housekeeping) and can still be restored manually. > > Housekeeping has purged some of the deleted files. In this case I'm > afraid > you will not be able to restore them and you will not know that it has > happened except that files are missing from the restore. > > If neither of these things has happened then I believe that you will > be > able to restore the directory as it was at the last sync before it was > deleted, by doing as you have done, "restore -d" on the deleted > directory > ID. An edge case is: * Delete a file * Wait a few weeks * Delete the directory * Restore the deleted directory Now in this case, unless housekeeping got to it, you would also restore the file you deleted weeks ago, because there's no distinction between "deleted file" and "deleted file as part of a general directory tree deletion." Ben From boxbackup at fluffy.co.uk Tue Apr 8 19:57:58 2008 From: boxbackup at fluffy.co.uk (Imran Niazi) Date: Tue, 8 Apr 2008 11:57:58 -0700 Subject: [Box Backup] Restoring old directory tree? In-Reply-To: <0E064195-4EA3-468E-A041-F20463B43920@fluffy.co.uk> References: <0E064195-4EA3-468E-A041-F20463B43920@fluffy.co.uk> Message-ID: <20080408185101.M60648@naweb.com> Hello, I remember talking about similiar or same issue, where you want the state of a system at a certain point in time. I'm not sure what came of that, or if its planned. One way to implement this is, each record should have two dates associated to it, a 'start' date and a 'end' date. start date is when box backup saw the file in its current state. end date would be when the file disappeared or changed. In a case of a file deletion that would be it. But if a file changed, you have a new file record of the same name (i guess thats how its done now), and you would then have a start & end date. Start date would be the same as the end date of the previous revision of that file etc. I guess this is obvious and you might have already been implemented in a newer version. -Imran > An edge case is: > > * Delete a file > > * Wait a few weeks > > * Delete the directory > > * Restore the deleted directory > > Now in this case, unless housekeeping got to it, you would also > restore the file you deleted weeks ago, because there's no > distinction between "deleted file" and "deleted file as part of a > general directory tree deletion." From boxbackup at fluffy.co.uk Tue Apr 8 21:06:17 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 8 Apr 2008 21:06:17 +0100 (BST) Subject: [Box Backup] Restoring old directory tree? In-Reply-To: <20080408185101.M60648@naweb.com> References: <0E064195-4EA3-468E-A041-F20463B43920@fluffy.co.uk> <20080408185101.M60648@naweb.com> Message-ID: Hi Imran, On Tue, 8 Apr 2008, Imran Niazi wrote: > I remember talking about similiar or same issue, where you want the > state of a system at a certain point in time. I'm not sure what came of > that, or if its planned. I'm afraid the necessary support to do it properly has not yet been written as code. It may happen soon, I'm planning a journalling feature for backups onto S3 and for greater disk efficiency, and one side effect would be a snapshot of all files that existed at a particular point in time. > One way to implement this is, each record should have two dates > associated to it, a 'start' date and a 'end' date. start date is when > box backup saw the file in its current state. end date would be when > the file disappeared or changed. In a case of a file deletion that > would be it. But if a file changed, you have a new file record of the > same name (i guess thats how its done now), and you would then have a > start & end date. Start date would be the same as the end date of the > previous revision of that file etc. > > I guess this is obvious and you might have already been implemented in a > newer version. I don't think it even needs to be so complex. The "start date" for a file is the oldest revision of it that we have. All we'd need to implement this feature is the deletion date. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Tue Apr 8 21:07:26 2008 From: boxbackup at fluffy.co.uk (Pablo Fernandez) Date: Tue, 08 Apr 2008 17:07:26 -0300 Subject: [Box Backup] Restoring old directory tree? In-Reply-To: <20080408185101.M60648@naweb.com> References: <0E064195-4EA3-468E-A041-F20463B43920@fluffy.co.uk> <20080408185101.M60648@naweb.com> Message-ID: <1207685247.4084.12.camel@laptop> Hi Point in time recovery would be very useful. Are there plans of supporting it? Best regards, Pablo On Tue, 2008-04-08 at 11:57 -0700, Imran Niazi wrote: > Hello, > > I remember talking about similiar or same issue, where you want the state of a > system at a certain point in time. I'm not sure what came of that, or if its > planned. > > One way to implement this is, each record should have two dates associated to > it, a 'start' date and a 'end' date. start date is when box backup saw the > file in its current state. end date would be when the file disappeared or > changed. In a case of a file deletion that would be it. But if a file > changed, you have a new file record of the same name (i guess thats how its > done now), and you would then have a start & end date. Start date would be > the same as the end date of the previous revision of that file etc. > > I guess this is obvious and you might have already been implemented in a newer > version. > > -Imran > > > An edge case is: > > > > * Delete a file > > > > * Wait a few weeks > > > > * Delete the directory > > > > * Restore the deleted directory > > > > Now in this case, unless housekeeping got to it, you would also > > restore the file you deleted weeks ago, because there's no > > distinction between "deleted file" and "deleted file as part of a > > general directory tree deletion." > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Tue Apr 8 21:59:14 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 8 Apr 2008 21:59:14 +0100 (BST) Subject: [Box Backup] Restoring old directory tree? In-Reply-To: <1207685247.4084.12.camel@laptop> References: <0E064195-4EA3-468E-A041-F20463B43920@fluffy.co.uk> <20080408185101.M60648@naweb.com> <1207685247.4084.12.camel@laptop> Message-ID: Hi Pablo, On Tue, 8 Apr 2008, Pablo Fernandez wrote: > Point in time recovery would be very useful. Are there plans of > supporting it? I just answered that question :-) Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Tue Apr 8 23:30:26 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Tue, 8 Apr 2008 23:30:26 +0100 Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. Message-ID: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> Hi, I am seeing a rather strange issue, which I am sure is down to what I have done :-) I have been backing up a linux server to my box backup server (currently both running latest RC) for months without any problems - however I decided as the server and the backup server is in the same building it would be smarter to send the data to a remote server. So I created a new bbstored on the remote server (a 1 & 1 root server) and copied the same raw keys etc to it and the backup ran fine, however due to 1 & 1 randomly reseting connections to their servers (presumably to avoid the root servers being used in such a way) I was forced to point the data back to the original server so at least the data was being backed up - (the pipe would fail and box would fail to continue once the connection failed without restarting the sync, leaving me with incomplete backups). This is where the strange issue is, every day I get an email with the contents of the backup via the boxreporter.py script, and the contents are the same + changed files for that day. To make sure it was not a reporting issue, I ran the sync by hand and monitored the logs. Sure enough the same files are upload or patched - but the files in question have not changed for over a week now. It appears somehow, the files in question are always seen as changed and forces a sync or is cached (well thats the behavior) ? Any ideas how I can debug this to work out what is going on ? Could it be down to the maxdiffing time ? Matt From boxbackup at fluffy.co.uk Tue Apr 8 23:36:34 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 8 Apr 2008 23:36:34 +0100 (BST) Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. In-Reply-To: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> Message-ID: Hi Matt, On Tue, 8 Apr 2008, Matt Brown wrote: > I am seeing a rather strange issue, which I am sure is down to what I > have done :-) I doubt it, thanks for reporting it, I will definitely investigate. > So I created a new bbstored on the remote server (a 1 & 1 root server) > and copied the same raw keys etc to it and the backup ran fine, however > due to 1 & 1 randomly reseting connections to their servers (presumably > to avoid the root servers being used in such a way) Damn, that sucks. I had no idea 1&1 would do that. I haven't seen any evidence of it happening with a root server that I use there. What does it look like? Is it long-running or idle connections that get terminated, or any connection? Could it be a firewall state table timeout? > I was forced to point the data back to the original server so at least > the data was being backed up - (the pipe would fail and box would fail > to continue once the connection failed without restarting the sync, > leaving me with incomplete backups). I would definitely like to know more about this. Can you reproduce this issue? What version of Box were you using? What do you get in the logs after the aborted connection? > This is where the strange issue is, every day I get an email with the > contents of the backup via the boxreporter.py script, and the contents > are the same + changed files for that day. To make sure it was not a > reporting issue, I ran the sync by hand and monitored the logs. Sure > enough the same files are upload or patched - but the files in question > have not changed for over a week now. > > It appears somehow, the files in question are always seen as changed and > forces a sync or is cached (well thats the behavior) ? > > Any ideas how I can debug this to work out what is going on ? Could you try running bbackupd with LogAllFileAccess = yes and the -V option and see what it says about these files? Could it be that their modification date is in the future, or is being updated? > Could it be down to the maxdiffing time ? It could be, although I think that if Box thinks the file has changed then it will calculate and upload a patch, even if it's an empty patch (i.e. there were no actual changes), so it probably happened before that, when Box decided that it thought the file had been changed. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Tue Apr 8 23:51:04 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Tue, 8 Apr 2008 23:51:04 +0100 Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. In-Reply-To: References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> Message-ID: Hi Chris, >> So I created a new bbstored on the remote server (a 1 & 1 root >> server) >> and copied the same raw keys etc to it and the backup ran fine, >> however >> due to 1 & 1 randomly reseting connections to their servers >> (presumably >> to avoid the root servers being used in such a way) > > Damn, that sucks. I had no idea 1&1 would do that. I haven't seen any > evidence of it happening with a root server that I use there. What > does it > look like? Is it long-running or idle connections that get > terminated, or > any connection? Could it be a firewall state table timeout? Seems to be idle timeout, if I leave an ssh session open it gets closed. I do use their firewall option, so it could be a state thing. >> I was forced to point the data back to the original server so at >> least >> the data was being backed up - (the pipe would fail and box would >> fail >> to continue once the connection failed without restarting the sync, >> leaving me with incomplete backups). > > I would definitely like to know more about this. Can you reproduce > this > issue? What version of Box were you using? What do you get in the logs > after the aborted connection? Well currently the version I am using is v0.11rc2 but I noticed I am using the state db i.e StoreObjectInfoFile = /var/bbackupd/bbackupd.state It does say ... #This saves time and increases efficiency when # bbackupd is frequently stopped and started, since it removes the need # to rescan all directories on the remote server. However, it is new and # not yet heavily tested, so use with caution. So this could be the culprit .. tho it has behaved until now .... >> This is where the strange issue is, every day I get an email with the >> contents of the backup via the boxreporter.py script, and the >> contents >> are the same + changed files for that day. To make sure it was not a >> reporting issue, I ran the sync by hand and monitored the logs. Sure >> enough the same files are upload or patched - but the files in >> question >> have not changed for over a week now. >> >> It appears somehow, the files in question are always seen as >> changed and >> forces a sync or is cached (well thats the behavior) ? >> >> Any ideas how I can debug this to work out what is going on ? > > Could you try running bbackupd with LogAllFileAccess = yes and the -V > option and see what it says about these files? > Yep got all these enabled, as used for the reporting function .... > Could it be that their modification date is in the future, or is being > updated? > Timestamps look fine. >> Could it be down to the maxdiffing time ? > > It could be, although I think that if Box thinks the file has > changed then > it will calculate and upload a patch, even if it's an empty patch > (i.e. > there were no actual changes), so it probably happened before that, > when > Box decided that it thought the file had been changed. I could always blat the store for this account - which would clear the issue, however I am curious as to why it thinks the files are not sync'd I shall do some more digging to see what is occuring .. Matt From boxbackup at fluffy.co.uk Wed Apr 9 00:07:59 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Wed, 9 Apr 2008 00:07:59 +0100 Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. *SOLVED* In-Reply-To: References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> Message-ID: Chris, Ignore me, I am being an ass ! The report is sending the same info as it is looking at a log file which has not rotated for some weeks ( and is now 500mb +). I changed over to syslog-ng and by default cron (in Ubuntu) for logrotate picked up all the system logs and rotated them. The /var/log/box/box.log had not been picked up. The files I then compared (just noticed) were all small files, and looking at the conf more carefully... it does say: # The daemon does "changes only" uploads for files above this size (in bytes). # Files less than it are uploaded whole without this extra processing. DiffingUploadSizeThreshold = 8192 Hence the same files :-) Sorry for the false alarm, and wasting bandwidth .. I am going to get some sleep .. ;-) Matt From boxbackup at fluffy.co.uk Wed Apr 9 00:08:48 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 9 Apr 2008 00:08:48 +0100 (BST) Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. In-Reply-To: References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> Message-ID: Hi Matt, On Tue, 8 Apr 2008, Matt Brown wrote: > > >due to 1 & 1 randomly reseting connections to their servers > > >(presumably to avoid the root servers being used in such a way) > > > >Damn, that sucks. I had no idea 1&1 would do that. I haven't seen any > >evidence of it happening with a root server that I use there. What does > >it look like? Is it long-running or idle connections that get > >terminated, or any connection? Could it be a firewall state table > >timeout? > > Seems to be idle timeout, if I leave an ssh session open it gets closed. > I do use their firewall option, so it could be a state thing. Yeah, I don't use their firewall, and I've had commentions open for 24 hours. I'll open one now and see how long it lasts. > > >I was forced to point the data back to the original server so at > > >least the data was being backed up - (the pipe would fail and box > > >would fail to continue once the connection failed without restarting > > >the sync, leaving me with incomplete backups). > > > >I would definitely like to know more about this. Can you reproduce this > >issue? What version of Box were you using? What do you get in the logs > >after the aborted connection? > > Well currently the version I am using is v0.11rc2 but I noticed I am using > the state db > > i.e StoreObjectInfoFile = /var/bbackupd/bbackupd.state > > It does say ... > > #This saves time and increases efficiency when > # bbackupd is frequently stopped and started, since it removes the need > # to rescan all directories on the remote server. However, it is new and > # not yet heavily tested, so use with caution. > > So this could be the culprit .. tho it has behaved until now .... I'm afraid this option is still not well tested, but let's see what the logs say. > >Could you try running bbackupd with LogAllFileAccess = yes and the -V > >option and see what it says about these files? > > > Yep got all these enabled, as used for the reporting function .... > > >Could it be that their modification date is in the future, or is being > >updated? > > > Timestamps look fine. i.e. not changing? Please could you send me the logs for these files off-list? > > >Could it be down to the maxdiffing time ? > > > >It could be, although I think that if Box thinks the file has changed then > >it will calculate and upload a patch, even if it's an empty patch (i.e. > >there were no actual changes), so it probably happened before that, when > >Box decided that it thought the file had been changed. > > I could always blat the store for this account - which would clear the > issue, however I am curious as to why it thinks the files are not sync'd Me too, and worried, please don't blat the account as we may need it for evidence and to test any potential fixes. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Wed Apr 9 00:14:01 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Wed, 9 Apr 2008 00:14:01 +0100 Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. In-Reply-To: References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> Message-ID: Hi Chris, >>> Damn, that sucks. I had no idea 1&1 would do that. I haven't seen >>> any >>> evidence of it happening with a root server that I use there. What >>> does >>> it look like? Is it long-running or idle connections that get >>> terminated, or any connection? Could it be a firewall state table >>> timeout? >> >> Seems to be idle timeout, if I leave an ssh session open it gets >> closed. >> I do use their firewall option, so it could be a state thing. > > Yeah, I don't use their firewall, and I've had commentions open for 24 > hours. I'll open one now and see how long it lasts. > >>>> I was forced to point the data back to the original server so at >>>> least the data was being backed up - (the pipe would fail and box >>>> would fail to continue once the connection failed without >>>> restarting >>>> the sync, leaving me with incomplete backups). >>> >>> I would definitely like to know more about this. Can you reproduce >>> this >>> issue? What version of Box were you using? What do you get in the >>> logs >>> after the aborted connection? >> I can still help on this one if you wish, would be nice to backup offsite ;-) Matt From boxbackup at fluffy.co.uk Wed Apr 9 00:14:53 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 9 Apr 2008 00:14:53 +0100 (BST) Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. *SOLVED* In-Reply-To: References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> Message-ID: Hi Matt, On Wed, 9 Apr 2008, Matt Brown wrote: > The report is sending the same info as it is looking at a log file which > has not rotated for some weeks ( and is now 500mb +). I changed over to > syslog-ng and by default cron (in Ubuntu) for logrotate picked up all > the system logs and rotated them. The /var/log/box/box.log had not been > picked up. Ah, makes sense, I should have thought of that :-) > Sorry for the false alarm, and wasting bandwidth .. I am going to get > some sleep .. ;-) When you have a chance, could you help me investigate the other issue, with bbackupd not recovering after an interrupted connection? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Wed Apr 9 00:17:59 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 9 Apr 2008 00:17:59 +0100 (BST) Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. In-Reply-To: References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> Message-ID: Hi Matt, On Wed, 9 Apr 2008, Matt Brown wrote: > > > > >I was forced to point the data back to the original server so at > > > > >least the data was being backed up - (the pipe would fail and box > > > > >would fail to continue once the connection failed without > > > > >restarting the sync, leaving me with incomplete backups). > > > > > > > >I would definitely like to know more about this. Can you reproduce > > > >this issue? What version of Box were you using? What do you get in > > > >the logs after the aborted connection? > > > > > I can still help on this one if you wish, would be nice to backup offsite ;-) Yes please :-) Backup online over the Internet is the main purpose of Box Backup, and we can expect reset connections and all other kinds of badness out there on the "open tubes". And we must deal with it gracefully. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Wed Apr 9 00:19:50 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Wed, 9 Apr 2008 00:19:50 +0100 Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. *SOLVED* In-Reply-To: References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> Message-ID: Hi Chris, > When you have a chance, could you help me investigate the other issue, > with bbackupd not recovering after an interrupted connection? Sure, tis the least I can do .. Matt From boxbackup at fluffy.co.uk Wed Apr 9 08:34:30 2008 From: boxbackup at fluffy.co.uk (Kenny Millington) Date: Wed, 09 Apr 2008 08:34:30 +0100 Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. *SOLVED* In-Reply-To: References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> Message-ID: <1207726470.7485.47.camel@helios> Hi, > > The report is sending the same info as it is looking at a log file which > > has not rotated for some weeks ( and is now 500mb +). I changed over to > > syslog-ng and by default cron (in Ubuntu) for logrotate picked up all > > the system logs and rotated them. The /var/log/box/box.log had not been > > picked up. > > Ah, makes sense, I should have thought of that :-) Doh! This would be a bug for me then... bbreporter.py is *supposed* to discard scanned data (in snapshot mode) if it encounters a new start time in the logs so that it only reports on the most recent backup session. I'll see about a fix. PS Chris (or anyone), is bbreporter.py (in contrib) likely to make any of the official debs/rpms? Thanks, -- Kenny Millington Systems Developer kenny.millington at 3ait.co.uk 3aIT Limited - Official Corporate Sponsor of the British Bobsleigh Team 4-10 Barttelot Rd Horsham West Sussex RH12 1DQ CoReg: 3866698 VATReg: 771388600 T: +44 (0)870 881 5097 F: +44 (0)870 116 0793 Visit www.3aIT.co.uk for Design, Systems, Support Disclaimer: The information contained within this email is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, any disclosure, copying or distribution of this email is prohibited and may be unlawful. The content of this email represents the views of the individual and not necessarily 3aIT Limited. 3aIT Limited reserves the right to monitor the content of all emails in accordance with lawful business practice. Whilst every effort is made to ensure that attachments are free from computer viruses before transmission, 3aIT Limited does not accept any liability in respect of any virus that is not detected. From boxbackup at fluffy.co.uk Wed Apr 9 09:03:33 2008 From: boxbackup at fluffy.co.uk (Kenny Millington) Date: Wed, 09 Apr 2008 09:03:33 +0100 Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. *SOLVED* In-Reply-To: <1207726470.7485.47.camel@helios> References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> <1207726470.7485.47.camel@helios> Message-ID: <1207728213.7485.50.camel@helios> On Wed, 2008-04-09 at 08:34 +0100, Kenny Millington wrote: > Doh! This would be a bug for me then... bbreporter.py is *supposed* to > discard scanned data (in snapshot mode) if it encounters a new start > time in the logs so that it only reports on the most recent backup > session. > > I'll see about a fix. Fixed in 2137. HTH, -- Kenny Millington Systems Developer kenny.millington at 3ait.co.uk 3aIT Limited - Official Corporate Sponsor of the British Bobsleigh Team 4-10 Barttelot Rd Horsham West Sussex RH12 1DQ CoReg: 3866698 VATReg: 771388600 T: +44 (0)870 881 5097 F: +44 (0)870 116 0793 Visit www.3aIT.co.uk for Design, Systems, Support Disclaimer: The information contained within this email is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, any disclosure, copying or distribution of this email is prohibited and may be unlawful. The content of this email represents the views of the individual and not necessarily 3aIT Limited. 3aIT Limited reserves the right to monitor the content of all emails in accordance with lawful business practice. Whilst every effort is made to ensure that attachments are free from computer viruses before transmission, 3aIT Limited does not accept any liability in respect of any virus that is not detected. From boxbackup at fluffy.co.uk Wed Apr 9 17:38:55 2008 From: boxbackup at fluffy.co.uk (James O'Gorman) Date: Wed, 09 Apr 2008 17:38:55 +0100 Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. *SOLVED* In-Reply-To: <1207726470.7485.47.camel@helios> References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> <1207726470.7485.47.camel@helios> Message-ID: <47FCF11F.8040206@netinertia.co.uk> Kenny Millington wrote: > Doh! This would be a bug for me then... bbreporter.py is *supposed* to > discard scanned data (in snapshot mode) if it encounters a new start > time in the logs so that it only reports on the most recent backup > session. I think I forgot to mention: when I created your trac account I also added bbreporter.py as a component in the ticket system, with you as the owner, so tickets can be filed against it. Commits to svn can update the ticket using certain keywords (have a look at Chris's extensive commit history for some very good examples!). > PS Chris (or anyone), is bbreporter.py (in contrib) likely to make any > of the official debs/rpms? AFAIK there aren't any "official" packages, but the RPM spec file is in contrib/ so feel free to create a patch that puts it in $PREFIX/share/box/contrib or something. I might go ahead and do the same for my FreeBSD port. James From boxbackup at fluffy.co.uk Wed Apr 9 22:27:14 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 9 Apr 2008 22:27:14 +0100 (BST) Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. *SOLVED* In-Reply-To: <1207726470.7485.47.camel@helios> References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> <1207726470.7485.47.camel@helios> Message-ID: Hi Kenny, On Wed, 9 Apr 2008, Kenny Millington wrote: > PS Chris (or anyone), is bbreporter.py (in contrib) likely to make any > of the official debs/rpms? I've added it to the RPM spec file, I can't speak for the maintainers of other package formats. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Thu Apr 10 23:22:42 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Thu, 10 Apr 2008 23:22:42 +0100 Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. *SOLVED* In-Reply-To: <1207728213.7485.50.camel@helios> References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> <1207726470.7485.47.camel@helios> <1207728213.7485.50.camel@helios> Message-ID: <1D6007D4-41F8-4DAA-B085-EEBC4F49F0D9@mbrown.co.uk> Hi Kenny, >> I'll see about a fix. > > Fixed in 2137. Ta, I will try and get round to testing over the weekend, I have kept of copy of that rather large log :-) Does boxreporter.py work with the new bbackupquery style output ? Chris, hopefully I will have time this weekend to start on the reporting of the 1 & 1 issue, I am afraid I am kinda stacked at the mo with work. Regards Matt From boxbackup at fluffy.co.uk Thu Apr 10 23:30:50 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Thu, 10 Apr 2008 23:30:50 +0100 (BST) Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. *SOLVED* In-Reply-To: <1D6007D4-41F8-4DAA-B085-EEBC4F49F0D9@mbrown.co.uk> References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> <1207726470.7485.47.camel@helios> <1207728213.7485.50.camel@helios> <1D6007D4-41F8-4DAA-B085-EEBC4F49F0D9@mbrown.co.uk> Message-ID: Hi Matt, On Thu, 10 Apr 2008, Matt Brown wrote: > Does boxreporter.py work with the new bbackupquery style output ? I guess probably not yet, sorry. > Chris, hopefully I will have time this weekend to start on the reporting > of the 1 & 1 issue, I am afraid I am kinda stacked at the mo with work. That would be great, thanks. I'm around all weekend. By the way I left an SSH session idle for about 24 hours to our server at 1&1, and then for another 24 hours, and both times it was fine. So my guess is that the network-level problem is with the firewall that they've provided you. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sat Apr 12 15:09:15 2008 From: boxbackup at fluffy.co.uk (Tom Albers) Date: Sat, 12 Apr 2008 16:09:15 +0200 Subject: [Box Backup] Nice statistics. In-Reply-To: <1207214538.2348.62.camel@helios> Message-ID: <3653305.UraVoFQdeh@kovoks.nl> --nextPart1705151.ZoUOoRRbDe Content-Type: text/plain Content-Transfer-Encoding: 7Bit Op Thursday 03 April 2008 11:22 schreef u: > > Did anyone write some nice statistics about the daily result of the backup on > > the client side? > If anyone has any questions I'll certainly do my best to answer them. Hi, After running it for a few days, I've some remarks: * it does not seem to list the files that were deleted that day - I could be wrong though. I think it is crucial to see what is marked as deleted, so you easily check if someone deleted something by accident. * the warnings and errors should be listed at top instead of at the bottom. * I usually get an 'Unable to retrieve usage information.' for the backup usage info. Any idea why? Otherwise I'm happy with the script ;-) Best, Toma --nextPart1705151.ZoUOoRRbDe-- From boxbackup at fluffy.co.uk Sat Apr 12 16:02:17 2008 From: boxbackup at fluffy.co.uk (Kenny Millington) Date: Sat, 12 Apr 2008 16:02:17 +0100 Subject: [Box Backup] Nice statistics. In-Reply-To: <3653305.UraVoFQdeh@kovoks.nl> References: <3653305.UraVoFQdeh@kovoks.nl> Message-ID: <1208012537.7452.5.camel@helios> Hi, > After running it for a few days, I've some remarks: Cool - I always like feedback to get feedback. > * it does not seem to list the files that were deleted that day - I could be wrong though. > I think it is crucial to see what is marked as deleted, so you easily check if someone > deleted something by accident. It doesn't but then as far as I know this isn't put into the logs by the box client itself so it can't be scraped by the reporting script. > * the warnings and errors should be listed at top instead of at the bottom. I think this is a case of different target audiences, the way we use it is to report to us but also to our clients so they can see the backup has worked. They're not too fussed by errors and warnings. That being said, if the warnings/errors are listed under the list of files I probably should move them so their show above the file list. Is this what you meant? > * I usually get an 'Unable to retrieve usage information.' for the backup > usage info. Any idea why? Are you running the script as root? In any case, one simple check is to run "bbackupquery usage quit" as the user that you run bbreporter.py as, if that works then it should work from the script (unless Chris broke my regex for parsing the output... ;-)). Thanks! -- Kenny Millington Systems Developer kenny.millington at 3ait.co.uk 3aIT Limited - Official Corporate Sponsor of the British Bobsleigh Team 4-10 Barttelot Rd Horsham West Sussex RH12 1DQ CoReg: 3866698 VATReg: 771388600 T: +44 (0)870 881 5097 F: +44 (0)870 116 0793 Visit www.3aIT.co.uk for Design, Systems, Support Disclaimer: The information contained within this email is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, any disclosure, copying or distribution of this email is prohibited and may be unlawful. The content of this email represents the views of the individual and not necessarily 3aIT Limited. 3aIT Limited reserves the right to monitor the content of all emails in accordance with lawful business practice. Whilst every effort is made to ensure that attachments are free from computer viruses before transmission, 3aIT Limited does not accept any liability in respect of any virus that is not detected. From boxbackup at fluffy.co.uk Sat Apr 12 21:17:10 2008 From: boxbackup at fluffy.co.uk (Tom Albers) Date: Sat, 12 Apr 2008 22:17:10 +0200 Subject: [Box Backup] Nice statistics. In-Reply-To: <1208012537.7452.5.camel@helios> Message-ID: <1312539.zta4jIy4Vy@kovoks.nl> --nextPart2188235.O8a1xgbzHt Content-Type: text/plain Content-Transfer-Encoding: 7Bit Op Saturday 12 April 2008 17:02 schreef u: > > I think it is crucial to see what is marked as deleted, so you easily check > if someone > > deleted something by accident. > > It doesn't but then as far as I know this isn't put into the logs by the > box client itself so it can't be scraped by the reporting script. Ah, ok. Maybe Chris can fix that? > > * the warnings and errors should be listed at top instead of at the bottom. > That being said, if the warnings/errors are listed under the list of > files I probably should move them so their show above the file list. > Is this what you meant? Yes, I would suggest to move the 'backupped' file list to the bottom... > > * I usually get an 'Unable to retrieve usage information.' for the backup > > usage info. Any idea why? > > Are you running the script as root? Yes, as a cron job. > In any case, one simple check is to run "bbackupquery usage quit" as the > user that you run bbreporter.py as, if that works then it should work > from the script (unless Chris broke my regex for parsing the > output... ;-)). No, the first time it worked fine. I'll have a closer look next week. Thanks for the answers. Toma --nextPart2188235.O8a1xgbzHt-- From boxbackup at fluffy.co.uk Sat Apr 12 21:22:17 2008 From: boxbackup at fluffy.co.uk (Kenny Millington) Date: Sat, 12 Apr 2008 21:22:17 +0100 Subject: [Box Backup] Nice statistics. In-Reply-To: <1312539.zta4jIy4Vy@kovoks.nl> References: <1312539.zta4jIy4Vy@kovoks.nl> Message-ID: <1208031737.7452.8.camel@helios> Hi, > > In any case, one simple check is to run "bbackupquery usage quit" as the > > user that you run bbreporter.py as, if that works then it should work > > from the script (unless Chris broke my regex for parsing the > > output... ;-)). > No, the first time it worked fine. I'll have a closer look next week. Thanks for the answers. Just had another thought... I peeked at the cron.d/boxbackup script that we use and saw the following:- PATH=/bin:/sbin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin Can you check for something similar on yours and make sure the bbackupquery path is specified. The script needs this available in the PATH environment variable. HTH, -- Kenny Millington Systems Developer kenny.millington at 3ait.co.uk 3aIT Limited - Official Corporate Sponsor of the British Bobsleigh Team 4-10 Barttelot Rd Horsham West Sussex RH12 1DQ CoReg: 3866698 VATReg: 771388600 T: +44 (0)870 881 5097 F: +44 (0)870 116 0793 Visit www.3aIT.co.uk for Design, Systems, Support Disclaimer: The information contained within this email is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, any disclosure, copying or distribution of this email is prohibited and may be unlawful. The content of this email represents the views of the individual and not necessarily 3aIT Limited. 3aIT Limited reserves the right to monitor the content of all emails in accordance with lawful business practice. Whilst every effort is made to ensure that attachments are free from computer viruses before transmission, 3aIT Limited does not accept any liability in respect of any virus that is not detected. From boxbackup at fluffy.co.uk Sun Apr 13 02:05:30 2008 From: boxbackup at fluffy.co.uk (John Goerzen) Date: Sat, 12 Apr 2008 20:05:30 -0500 Subject: [Box Backup] Re: boxbackup -- confirmation of subscription -- request 724919 In-Reply-To: <20080413002810.10823.30558.Mailman@love.warhead.org.uk> References: <20080413002810.10823.30558.Mailman@love.warhead.org.uk> Message-ID: <200804122005.30365.jgoerzen@complete.org> On Saturday 12 April 2008 7:28:10 pm boxbackup-request at fluffy.co.uk wrote: > boxbackup -- confirmation of subscription -- request 724919 > > We have received a request from 63-245-179-205.kitusa.com for > subscription of your email address, , to the > boxbackup at fluffy.co.uk mailing list. To confirm the request, please > send a message to boxbackup-request at fluffy.co.uk, and either: > > - maintain the subject line as is (the reply's additional "Re:" is > ok), > > - or include the following line - and only the following line - in the > message body: > > confirm 724919 > > (Simply sending a 'reply' to this message should work from most email > interfaces, since that usually leaves the subject line in the right > form.) > > If you do not wish to subscribe to this list, please simply disregard > this message. Send questions to boxbackup-admin at fluffy.co.uk. From boxbackup at fluffy.co.uk Sun Apr 13 03:28:16 2008 From: boxbackup at fluffy.co.uk (John Goerzen) Date: Sat, 12 Apr 2008 21:28:16 -0500 Subject: [Box Backup] Hard Links and rdiff Message-ID: <20080413022816.GA23798@complete.org> Hi folks, I've been looking for a new backup program lately, and from the description on the web site and wiki, Box Backup looked like the perfect fit. Until I started doing some tests. I noticed that Box Backup has completely mangled all hard links. I can do a backup, and when I do a restore, it unpacks a separate copy of each file that was hard linked together. This is a showstopper for me. In addition to making Box Backup unsuitable for backing up the entire system (due to hardlinks in /bin, /sbin, /usr/bin, etc.), it also steps on the toes of people that use distributed version control systems like Git, Mercurial, or Darcs. Is there a configuration option somewhere to preserve hard links? Are there any other POSIX attributes that Box Backup may unexpectedly not restore? Does it preserve mtime, atime, ctime, symlinks, block and character devices, FIFO locations, EAs, and ACLs? Also, I tried in vain to find some details on its rdiff algorithm. Does it play rdiffs "backwards" like rdiff-backup, or forwards like duplicity? In the "backwards" scheme, the "full" data is always the most current, and you follow a chain of deltas backwards to older versions. This makes restores of recent versions easy, and also allows removal of the oldest data without breaking chains. duplicity goes the other way -- taking full backups, and storing binary deltas going forward. This removes the need to keep an accessible copy of the full recent data, at the expense of making restores of recent data slower and imposing some limits on the removal of older data. Is the rdiff algorithm applied on a per-file basis or on the entire backup file as a whole? If it's on a per-file basis, does it identify a file by name or by device/inode number? Thanks, -- John From boxbackup at fluffy.co.uk Sun Apr 13 03:44:29 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 13 Apr 2008 03:44:29 +0100 (BST) Subject: [Box Backup] Hard Links and rdiff In-Reply-To: <20080413022816.GA23798@complete.org> References: <20080413022816.GA23798@complete.org> Message-ID: Hi John, On Sat, 12 Apr 2008, John Goerzen wrote: > Until I started doing some tests. I noticed that Box Backup has > completely mangled all hard links. I can do a backup, and when I do a > restore, it unpacks a separate copy of each file that was hard linked > together. Umm yeah, I can imagine why it would do that :-) Sorry. > This is a showstopper for me. In addition to making Box Backup > unsuitable for backing up the entire system (due to hardlinks in /bin, > /sbin, /usr/bin, etc.) It is definitely not designed for backing up entire systems. Only user data which is not hardlinked. > it also steps on the toes of people that use distributed version control > systems like Git, Mercurial, or Darcs. Does mercurial really use hard links? That would be a bummer, that would never work on Windows or filesystems that don't support hard links. I can believe that Git would do that because, well, because of the name... :-) No idea about Darcs, I'll take your word for it. > Is there a configuration option somewhere to preserve hard links? Unfortunately not (yet). > Are there any other POSIX attributes that Box Backup may unexpectedly > not restore? Does it preserve mtime, atime It does preserve mtime and atime (although the value of preserving atime is approximately zero in my book, because you only have to sneeze to reset it). > ctime Preserving ctime is actually impossible without hacking the filesystem. Any change to the inode, including an attempt to change its ctime, would change its ctime. That's POSIX. > symlinks Are preserved. > block and character devices, FIFO locations Are not backed up at all. > EAs, and ACLs? POSIX EAs and ACLs are supported if and only if your Box Backup client (bbackupd) was compiled with support for them. Unfortunately there's no way to tell if that's the case with a binary package right now, although I can see that it would be a good idea to add one. > Also, I tried in vain to find some details on its rdiff algorithm. docs/backup/encrypt_rsync.txt > Does it play rdiffs "backwards" like rdiff-backup, or forwards like > duplicity? In the "backwards" scheme, the "full" data is always the > most current, and you follow a chain of deltas backwards to older > versions. This makes restores of recent versions easy, and also > allows removal of the oldest data without breaking chains. It is backwards, one of the biggest advantages over duplicity in my view (otherwise I'd quite possibly be using and working on duplicity instead). > duplicity goes the other way -- taking full backups, and storing binary > deltas going forward. This removes the need to keep an accessible copy > of the full recent data, at the expense of making restores of recent > data slower and imposing some limits on the removal of older data. Restore time of much-changed files tends to infinity. Not good. And how exactly do you remove old versions? Don't you have to rewrite huge archive files? Does that work well over the Internet? (I doubt it). > Is the rdiff algorithm applied on a per-file basis or on the entire > backup file as a whole? Per file. > If it's on a per-file basis, does it identify a file by name or by > device/inode number? It depends, it uses name normally but it also tracks inode number for use in case there's no match on the store with the same name. It helps reduce backup bandwidth usage when files are renamed, but doesn't help if they are copied. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sun Apr 13 03:49:15 2008 From: boxbackup at fluffy.co.uk (John Goerzen) Date: Sat, 12 Apr 2008 21:49:15 -0500 Subject: [Box Backup] Re: Hard Links and rdiff Message-ID: <20080413024915.GA24296@complete.org> On 2008-04-13, John Goerzen wrote: > links? Are there any other POSIX attributes that Box Backup may > unexpectedly not restore? Does it preserve mtime, atime, ctime, > symlinks, block and character devices, FIFO locations, EAs, and > ACLs? Two other questions along these lines. One is relating to sparse files. The other relates to this item in the FAQ: "If a file is modified or replaced by another file with the same modification time, Box Backup will ignore the fact that the file has changed." That would imply to me that boxbackup is looking only at mtime when determining what needs to be backed up. In particular, it is likely ignoring ctime, inode, and device -- at least one of which should have changed in the scenario described. -- John From boxbackup at fluffy.co.uk Sun Apr 13 03:54:19 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 13 Apr 2008 03:54:19 +0100 (BST) Subject: [Box Backup] Re: Hard Links and rdiff In-Reply-To: <20080413024915.GA24296@complete.org> References: <20080413024915.GA24296@complete.org> Message-ID: Hi John, On Sat, 12 Apr 2008, John Goerzen wrote: > Two other questions along these lines. One is relating to sparse > files. I don't believe we treat them as sparse files during backup or restore. You've certainly managed to find a lot of problems or limitations Box Backup very quickly. Well done :-) > The other relates to this item in the FAQ: > > "If a file is modified or replaced by another file with the same > modification time, Box Backup will ignore the fact that the file has > changed." > > That would imply to me that boxbackup is looking only at mtime when > determining what needs to be backed up. In particular, it is likely > ignoring ctime, inode, and device -- at least one of which should have > changed in the scenario described. Yes, it is only using mtime to determine whether files have changed. I've been proposing changing this to ctime for a while, mainly confounded by the fact that Windows doesn't have ctime at all and therefore this would lead to more inconsistency between platforms. We've also been talking about using an inotify-type interface where supported by the OS to detect file changes in real time and avoid the need to scan the entire tree, but so far nobody has implemented it. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sun Apr 13 04:20:04 2008 From: boxbackup at fluffy.co.uk (Kjell Christian Bruheim) Date: Sun, 13 Apr 2008 05:20:04 +0200 Subject: [Box Backup] Re: boxbackup -- confirmation of subscription -- request 724919 In-Reply-To: <200804122005.30365.jgoerzen@complete.org> References: <20080413002810.10823.30558.Mailman@love.warhead.org.uk> <200804122005.30365.jgoerzen@complete.org> Message-ID: <48017BE4.7050902@supportingit.com> John Goerzen wrote: > On Saturday 12 April 2008 7:28:10 pm boxbackup-request at fluffy.co.uk wrote: > >> boxbackup -- confirmation of subscription -- request 724919 >> >> We have received a request from 63-245-179-205.kitusa.com for >> subscription of your email address, , to the >> boxbackup at fluffy.co.uk mailing list. To confirm the request, please >> send a message to boxbackup-request at fluffy.co.uk, and either: >> >> - maintain the subject line as is (the reply's additional "Re:" is >> ok), >> >> - or include the following line - and only the following line - in the >> message body: >> >> confirm 724919 >> >> (Simply sending a 'reply' to this message should work from most email >> interfaces, since that usually leaves the subject line in the right >> form.) >> >> If you do not wish to subscribe to this list, please simply disregard >> this message. Send questions to boxbackup-admin at fluffy.co.uk. >> > > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Sun Apr 13 16:07:40 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Sun, 13 Apr 2008 16:07:40 +0100 Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. *SOLVED* In-Reply-To: References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> <1207726470.7485.47.camel@helios> <1207728213.7485.50.camel@helios> <1D6007D4-41F8-4DAA-B085-EEBC4F49F0D9@mbrown.co.uk> Message-ID: Hi Chris, >> Chris, hopefully I will have time this weekend to start on the >> reporting >> of the 1 & 1 issue, I am afraid I am kinda stacked at the mo with >> work. > > That would be great, thanks. I'm around all weekend. > > By the way I left an SSH session idle for about 24 hours to our > server at > 1&1, and then for another 24 hours, and both times it was fine. So my > guess is that the network-level problem is with the firewall that > they've > provided you. Do you want me to send the results/outputs on or off list ? Regards Matt From boxbackup at fluffy.co.uk Sun Apr 13 16:09:22 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 13 Apr 2008 16:09:22 +0100 (BST) Subject: [Box Backup] Files are uploaded/patched daily even though they have not changed in weeks. *SOLVED* In-Reply-To: References: <28D140EB-9D3E-4CE1-8214-C130D12E354D@mbrown.co.uk> <1207726470.7485.47.camel@helios> <1207728213.7485.50.camel@helios> <1D6007D4-41F8-4DAA-B085-EEBC4F49F0D9@mbrown.co.uk> Message-ID: Hi Matt, On Sun, 13 Apr 2008, Matt Brown wrote: > Do you want me to send the results/outputs on or off list ? If it's over about 20K, offlist please. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sun Apr 13 16:13:36 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Sun, 13 Apr 2008 16:13:36 +0100 Subject: [Box Backup] Connection fails during backup to remote server Message-ID: Hi Chirs, As promised, here is the start of testing with the two servers. The remote server is a 1 & 1 server using the firewall they supply, running Ubuntu x64 Dapper 6.06 LTS the client is Ubuntu x86 Dapper 6.06 LTS. I have recreated the account on the remote server and here is the logs from the remote server (Hercules): Apr 13 15:22:36 hercules Box Backup (bbstored)[27413]: NOTICE: Starting daemon, version 0.11rc2, config: /etc/box/bbstored.confApr 13 15:22:46 hercules Box Backup (bbstored)[27413]: WARNING: Message from child process 27422: Incoming connection from x.x.x.x port 41260 Apr 13 15:22:47 hercules Box Backup (bbstored)[27422]: NOTICE: Login from Client ID 0x00000001 Read/Write Apr 13 15:22:47 hercules Box Backup (bbstored)[27422]: Sending stream, size 40 Apr 13 15:22:47 hercules Box Backup (bbstored)[27422]: Receiving stream, size 49 Apr 13 15:22:47 hercules last message repeated 3 times Apr 13 15:22:48 hercules Box Backup (bbstored)[27422]: Sending stream, size 89 Apr 13 15:22:48 hercules Box Backup (bbstored)[27422]: Receiving stream, size 49 Apr 13 15:22:48 hercules Box Backup (bbstored)[27422]: Receiving stream, size uncertain Apr 13 15:22:49 hercules last message repeated 9 times Apr 13 15:24:33 hercules last message repeated 4 times Apr 13 15:57:32 hercules Box Backup (bbstored)[27422]: WARNING: Exception thrown: BackupStoreException(ReadFileFromStreamTimedOut) at BackupContext.cpp(479) Apr 13 15:57:32 hercules Box Backup (bbstored)[27422]: ERROR: Error in child process, terminating connection: BackupStore ReadFileFromStreamTimedOut (4/21) Here are the logs from the client (Orion): Apr 13 15:22:40 orion Backup (bbackupd)[10407]: Incoming connection from local (UNIX socket) Apr 13 15:22:40 orion Backup (bbackupd)[10407]: Connection from command socket Apr 13 15:22:40 orion Backup (bbackupd)[10407]: NOTICE: Beginning scan of local files Apr 13 15:22:40 orion Backup (bbackupd)[10407]: Opening connection to server 'hercules.xxxxxxxxxxx... Apr 13 15:22:41 orion Backup (bbackupd)[10407]: Connection made, login successful Apr 13 15:22:41 orion Backup (bbackupd)[10407]: Not deleting unused entries - none in list Apr 13 15:22:41 orion Backup (bbackupd)[10407]: NOTICE: About to notify administrator about event backup-start, running script '/etc/ box/bbackupd/NotifySysadmin.sh backup-start' list of files being synced. Apr 13 15:24:27 orion Backup (bbackupd)[10407]: Uploading complete file: /home/matt/3aitweb.tar.gz Apr 13 15:59:10 orion Backup (bbackupd)[10407]: WARNING: Exception thrown: ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(442) Apr 13 15:59:10 orion Backup (bbackupd)[10407]: ERROR: Failed to upload file: /home/matt/3aitweb.tar.gz: caught exception: Connection TLSWriteFailed (Probably a network issue between client and server.) (7/33) Apr 13 15:59:10 orion Backup (bbackupd)[10407]: ERROR: SSL error during Write: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry Apr 13 15:59:10 orion Backup (bbackupd)[10407]: WARNING: Exception thrown: ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(442) Apr 13 15:59:10 orion Backup (bbackupd)[10407]: WARNING: Suppressing duplicate notification about backup-error Apr 13 15:59:10 orion Backup (bbackupd)[10407]: ERROR: Exception caught (Connection TLSWriteFailed (Probably a network issue between client and server.) 7/33), reset state and waiting to retry... Apr 13 15:59:20 orion Backup (bbackupd)[10407]: NOTICE: File statistics: total file size uploaded 127939748, bytes already on server 0, encoded size 70796099 A total of 68MB was transferred before the connection died on this occasion. Regards Matt From boxbackup at fluffy.co.uk Sun Apr 13 16:17:25 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 13 Apr 2008 16:17:25 +0100 (BST) Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: Message-ID: Hi Matt, On Sun, 13 Apr 2008, Matt Brown wrote: > As promised, here is the start of testing with the two servers. The > remote server is a 1 & 1 server using the firewall they supply, running > Ubuntu x64 Dapper 6.06 LTS the client is Ubuntu x86 Dapper 6.06 LTS. > > I have recreated the account on the remote server and here is the logs > from the remote server (Hercules): > > Apr 13 15:22:48 hercules Box Backup (bbstored)[27422]: Receiving stream, size > uncertain > Apr 13 15:22:49 hercules last message repeated 9 times > Apr 13 15:24:33 hercules last message repeated 4 times > Apr 13 15:57:32 hercules Box Backup (bbstored)[27422]: WARNING: Exception > thrown: BackupStoreException(ReadFileFromStreamTimedOut) at > BackupContext.cpp(479) ... > Apr 13 15:24:27 orion Backup (bbackupd)[10407]: Uploading complete file: > /home/matt/3aitweb.tar.gz > Apr 13 15:59:10 orion Backup (bbackupd)[10407]: WARNING: Exception thrown: > ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(442) First of all, this looks like a classic SSL timeout. Did you enable the SslKeepAliveTime option in your client config? If not, don't do it yet, it would be helpful to debug the other issue first: > Apr 13 15:59:10 orion Backup (bbackupd)[10407]: ERROR: Exception caught > (Connection TLSWriteFailed (Probably a network issue between client and > server.) 7/33), reset state and waiting to retry... > Apr 13 15:59:20 orion Backup (bbackupd)[10407]: NOTICE: File statistics: total > file size uploaded 127939748, bytes already on server 0, encoded size 70796099 > > A total of 68MB was transferred before the connection died on this occasion. Did the client retry after this (about 100 seconds later) or not? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sun Apr 13 16:24:34 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Sun, 13 Apr 2008 16:24:34 +0100 Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: Message-ID: Hi Chris, >> Apr 13 15:24:27 orion Backup (bbackupd)[10407]: Uploading complete >> file: >> /home/matt/3aitweb.tar.gz >> Apr 13 15:59:10 orion Backup (bbackupd)[10407]: WARNING: Exception >> thrown: >> ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(442) > > First of all, this looks like a classic SSL timeout. Did you enable > the > SslKeepAliveTime option in your client config? If not, don't do it > yet, it > would be helpful to debug the other issue first: > I dont think so, is this an option at compile time ? or a setting in the bbackupd.conf ? >> Apr 13 15:59:10 orion Backup (bbackupd)[10407]: ERROR: Exception >> caught >> (Connection TLSWriteFailed (Probably a network issue between client >> and >> server.) 7/33), reset state and waiting to retry... >> Apr 13 15:59:20 orion Backup (bbackupd)[10407]: NOTICE: File >> statistics: total >> file size uploaded 127939748, bytes already on server 0, encoded >> size 70796099 >> >> A total of 68MB was transferred before the connection died on this >> occasion. > > Did the client retry after this (about 100 seconds later) or not? It didn't appear to (where would I have seen this happen ?), I was watching the bandwidth in/out of the network and saw the connection go idle, not further traffic appeared to occur and after a long idle out the logs reported the failure as above. Matt From boxbackup at fluffy.co.uk Sun Apr 13 16:32:17 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 13 Apr 2008 16:32:17 +0100 (BST) Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: Message-ID: Hi Matt, On Sun, 13 Apr 2008, Matt Brown wrote: > >First of all, this looks like a classic SSL timeout. Did you enable the > >SslKeepAliveTime option in your client config? If not, don't do it yet, > >it would be helpful to debug the other issue first: > > I dont think so, is this an option at compile time ? or a setting in the > bbackupd.conf ? bbackupd.conf. > > >(Connection TLSWriteFailed (Probably a network issue between client and > > >server.) 7/33), reset state and waiting to retry... > > >Apr 13 15:59:20 orion Backup (bbackupd)[10407]: NOTICE: File statistics: > > >total > > >file size uploaded 127939748, bytes already on server 0, encoded size > > >70796099 > > > >Did the client retry after this (about 100 seconds later) or not? > > It didn't appear to (where would I have seen this happen ?) In the client logs, about 100 seconds after "reset state and waiting to retry", you should at least see another message from bbackupd procesds 10407. If not, is that process still running? Could you attach strace to it to see what it's doing? Could you also try running bbackupd from the command line with -DVT and see what it outputs after it prints "reset state and waiting to retry"? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sun Apr 13 16:41:29 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Sun, 13 Apr 2008 16:41:29 +0100 Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: Message-ID: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Hi Chris, >> I dont think so, is this an option at compile time ? or a setting >> in the >> bbackupd.conf ? > > bbackupd.conf. > Ah ok .. >> >> It didn't appear to (where would I have seen this happen ?) > > In the client logs, about 100 seconds after "reset state and waiting > to > retry", you should at least see another message from bbackupd procesds > 10407. If not, is that process still running? Could you attach > strace to > it to see what it's doing? > > Could you also try running bbackupd from the command line with -DVT > and > see what it outputs after it prints "reset state and waiting to > retry"? > Do you want me to enable sslkeepalive or leave this off for now .. Matt From boxbackup at fluffy.co.uk Sun Apr 13 16:43:13 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 13 Apr 2008 16:43:13 +0100 (BST) Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: Hi Matt, On Sun, 13 Apr 2008, Matt Brown wrote: > Do you want me to enable sslkeepalive or leave this off for now .. Please leave it off for now. I hope we can find out why the client daemon isn't retrying. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sun Apr 13 16:49:58 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Sun, 13 Apr 2008 16:49:58 +0100 Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: <34666750-82FB-4666-A3CE-42DEA2082641@mbrown.co.uk> Hi Chris, >> Do you want me to enable sslkeepalive or leave this off for now .. > > Please leave it off for now. I hope we can find out why the client > daemon > isn't retrying. Sure, I did notice this tho ... 16:41:43 NOTICE: Starting daemon, version 0.11rc2, config: /etc/box/ bbackupd.conf 16:41:43 WARNING: Store object info file out of date. Will re-cache from store. (/var/bbackupd/bbackupd.state) 16:41:43 Wait on command socket, delay = 1024000000 16:42:59 Incoming connection from local (UNIX socket) 16:42:59 Connection from command socket 16:42:59 Receiving command 'sync' over command socket 16:42:59 Receiving command 'quit' over command socket 16:42:59 Closing command connection 16:42:59 NOTICE: Beginning scan of local files 16:42:59 Set maximum diffing time to 50 seconds -> 16:42:59 Set keep-alive time to 60 seconds 16:42:59 timer 0xbfc9d0e8 has not expired, triggering in 59.999979 seconds 16:42:59 timer 0xbfc9d0e8 has not expired, triggering in 59.999711 seconds 16:42:59 timer 0xbfc9d0e8 has not expired, triggering in 59.999536 seconds 16:42:59 timer 0xbfc9d2c8 has not expired, triggering in 59.999536 seconds 16:42:59 timer 0xbfc9d2c8 has not expired, triggering in 59.999235 seconds 16:42:59 Opening connection to server 'hercules.xxxx... 16:42:59 Send block allocation size is 1024 16:42:59 Connection made, login successful Matt From boxbackup at fluffy.co.uk Sun Apr 13 18:06:29 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Sun, 13 Apr 2008 18:06:29 +0100 Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: Hi Chris, 17:47:29 Read 4096 bytes at 282624, 15770 remain, eta 0s 17:47:29 Read 4096 bytes at 286720, 11674 remain, eta 0s 17:47:29 Read 4096 bytes at 290816, 7578 remain, eta 0s 17:47:29 Read 4096 bytes at 294912, 3482 remain, eta 0s 17:47:29 Read 3482 bytes at 298394, 0 remain, eta 0s 18:02:29 WARNING: Exception thrown: ConnectionException(Conn_Protocol_Timeout) at Protocol.cpp(229) 18:02:29 ERROR: Failed to upload file: /home/matt/Maildir/.Archive/cur/ 1185382629 .M525281P8575V0000000000000301I00784FA6_3448.orion,S=298394:2,S: caught exception: Connection Protocol_Timeout (Probably a network issue between client and server.) (7/41) 18:02:31 WARNING: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(377) 18:02:31 sizeof(sEventNames) == 24 18:02:31 sizeof(*sEventNames) == 4 18:02:31 NotifyEvent__MAX == 5 18:02:31 BackupDaemon::NotifySysadmin() called, event = backup-error 18:02:31 NOTICE: About to notify administrator about event backup- error, running script '/etc/box/bbackupd/NotifySysadmin.sh backup-error' 18:02:31 ERROR: Exception caught (Connection Protocol_Timeout (Probably a network issue between client and server.) 7/41), reset state and waiting to retry... 18:02:41 NOTICE: File statistics: total file size uploaded 293402698, bytes already on server 0, encoded size 205668173 18:02:41 Wait on command socket, delay = 1024000000 Matt From boxbackup at fluffy.co.uk Sun Apr 13 18:34:10 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Sun, 13 Apr 2008 18:34:10 +0100 Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: <883AB704-D4F1-4F57-9E12-B088CC1A5431@mbrown.co.uk> Hi Chris, > 17:47:29 Read 4096 bytes at 282624, 15770 remain, eta 0s > 17:47:29 Read 4096 bytes at 286720, 11674 remain, eta 0s > 17:47:29 Read 4096 bytes at 290816, 7578 remain, eta 0s > 17:47:29 Read 4096 bytes at 294912, 3482 remain, eta 0s > 17:47:29 Read 3482 bytes at 298394, 0 remain, eta 0s > 18:02:29 WARNING: Exception thrown: > ConnectionException(Conn_Protocol_Timeout) at Protocol.cpp(229) > 18:02:29 ERROR: Failed to upload file: /home/matt/Maildir/.Archive/ > cur/ > 1185382629 > .M525281P8575V0000000000000301I00784FA6_3448.orion,S=298394:2,S: > caught exception: Connection Protocol_Timeout (Probably a network > issue between client and server.) (7/41) > 18:02:31 WARNING: Exception thrown: > ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(377) > 18:02:31 sizeof(sEventNames) == 24 > 18:02:31 sizeof(*sEventNames) == 4 > 18:02:31 NotifyEvent__MAX == 5 > 18:02:31 BackupDaemon::NotifySysadmin() called, event = backup-error > 18:02:31 NOTICE: About to notify administrator about event backup- > error, running script '/etc/box/bbackupd/NotifySysadmin.sh backup- > error' > 18:02:31 ERROR: Exception caught (Connection Protocol_Timeout > (Probably a network issue between client and server.) 7/41), reset > state and waiting to retry... > 18:02:41 NOTICE: File statistics: total file size uploaded > 293402698, bytes already on server 0, encoded size 205668173 > 18:02:41 Wait on command socket, delay = 1024000000 18:19:45 Wait on command socket, delay = 1024000000 Matt From boxbackup at fluffy.co.uk Sun Apr 13 20:22:53 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 13 Apr 2008 20:22:53 +0100 (BST) Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: Hi Matt, On Sun, 13 Apr 2008, Matt Brown wrote: > 18:02:31 ERROR: Exception caught (Connection Protocol_Timeout (Probably a > network issue between client and server.) 7/41), reset state and waiting to > retry... > 18:02:41 NOTICE: File statistics: total file size uploaded 293402698, bytes > already on server 0, encoded size 205668173 > 18:02:41 Wait on command socket, delay = 1024000000 Are you running in snapshot or lazy mode? Because if it's snapshot mode, I can see why it wouldn't retry automatically. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sun Apr 13 21:33:39 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Sun, 13 Apr 2008 21:33:39 +0100 Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: Hi Chris, > Are you running in snapshot or lazy mode? Because if it's snapshot > mode, I > can see why it wouldn't retry automatically. Yes, snapshot mode. Regards Matt From boxbackup at fluffy.co.uk Sun Apr 13 21:36:06 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 13 Apr 2008 21:36:06 +0100 (BST) Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: Hi Matt, On Sun, 13 Apr 2008, Matt Brown wrote: > >Are you running in snapshot or lazy mode? Because if it's snapshot > >mode, I can see why it wouldn't retry automatically. > > Yes, snapshot mode. OK, so would you expect bbackupd to retry automatically in case of a network error during an (explicitly-requested) sync? I guess so, but I could be wrong. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sun Apr 13 23:55:52 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Sun, 13 Apr 2008 23:55:52 +0100 Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: Hi Chris, >> Yes, snapshot mode. > > OK, so would you expect bbackupd to retry automatically in case of a > network error during an (explicitly-requested) sync? I guess so, but I > could be wrong. Well, I would envisage that once a snapshot has been started that it would sync until its completion. All the clients we backup use the snapshot method out of hours mainly to reduce network traffic during the day. In this instance the connection/socket is reset presumably by the 1 & 1 firewall states which in turn cause the connection to be severed/ terminated ... obviously a couple of things can be re-arranged to make this work - i.e iptables or alternative location. Would it be better to run all backups in lazy mode ? Well I could, but the reason for running backups in snapshot mode is to set a time slot when the bandwidth is off peak and to reduce traffic during the day - with my current DSL provider I get around 350GB of free bandwidth at night between 8pm and 8am - so it make sense to backup at these times, I am also guessing that once box incorporates mark states this will require a snapshot type sync ? Matt From boxbackup at fluffy.co.uk Mon Apr 14 00:04:21 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Mon, 14 Apr 2008 00:04:21 +0100 (BST) Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: Hi Matt, On Sun, 13 Apr 2008, Matt Brown wrote: > >OK, so would you expect bbackupd to retry automatically in case of a > >network error during an (explicitly-requested) sync? I guess so, but I > >could be wrong. > > Well, I would envisage that once a snapshot has been started that it would > sync until its completion. Yeah, seems reasonable to me too. I can send you a patch to test if you're interested. > All the clients we backup use the snapshot method out of hours mainly to > reduce network traffic during the day. > > In this instance the connection/socket is reset presumably by the 1 & 1 > firewall states which in turn cause the connection to be > severed/terminated ... obviously a couple of things can be re-arranged > to make this work - i.e iptables or alternative location. Seems like a good idea, but I agree that we should recover and try again if the sync fails, even in snapshot mode (currently we don't). > Would it be better to run all backups in lazy mode ? Well I could, but > the reason for running backups in snapshot mode is to set a time slot > when the bandwidth is off peak and to reduce traffic during the day - > with my current DSL provider I get around 350GB of free bandwidth at > night between 8pm and 8am - so it make sense to backup at these times, Yeah, makes sense to me too. > I am also guessing that once box incorporates mark states this will > require a snapshot type sync ? Sorry, what are "mark states"? Do you mean like a point-in-time snapshot of all files on the filesystem at that time, to fix the deletion time problem? If so, then I think we could just do snapshots at irregular intervals, which is more or less exactly what we do now. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Mon Apr 14 00:17:15 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Mon, 14 Apr 2008 00:17:15 +0100 Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: Hi Chris, >> Well, I would envisage that once a snapshot has been started that >> it would >> sync until its completion. > > Yeah, seems reasonable to me too. I can send you a patch to test if > you're > interested. Sure, I am happy to help testing as always :-) >> In this instance the connection/socket is reset presumably by the 1 >> & 1 >> firewall states which in turn cause the connection to be >> severed/terminated ... obviously a couple of things can be re- >> arranged >> to make this work - i.e iptables or alternative location. > > Seems like a good idea, but I agree that we should recover and try > again > if the sync fails, even in snapshot mode (currently we don't). Well on reflection, maybe people use the lazy mode more than snapshot ? maybe some feedback from other box users could confirm ? as this does provide a better recovery solution to files which are always changing, and presumably the connection would be automatically re- tried on the next lazy run ? in addition to any client retries currently coded for.. >> I am also guessing that once box incorporates mark states this will >> require a snapshot type sync ? > > Sorry, what are "mark states"? Do you mean like a point-in-time > snapshot > of all files on the filesystem at that time, to fix the deletion time > problem? If so, then I think we could just do snapshots at irregular > intervals, which is more or less exactly what we do now. Sorry, yes my bad I was using the wording as per the Box Backup web page: Panned future work * Mark state (marks the current state of the filesystem, to emulate a tape change and restoring from that tape later) So far we have only had to restore the odd files here and there that people have deleted/corrupted.. however I am not sure how you would go about recovering so a whole directory from 2 days ago, is that possible currently ? Please don't think I am knocking the project, its works very well and has been 100% successful in recovering data ! I wasnt sure if I had hit a bug, but would appear to be more of a feature request :-) Matt From boxbackup at fluffy.co.uk Mon Apr 14 00:24:54 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Mon, 14 Apr 2008 00:24:54 +0100 (BST) Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---128931150-1617812520-1208129094=:25757 Content-Type: TEXT/PLAIN; charset=US-ASCII Hi Matt, On Mon, 14 Apr 2008, Matt Brown wrote: > > >Well, I would envisage that once a snapshot has been started that it > > >would sync until its completion. > > > >Yeah, seems reasonable to me too. I can send you a patch to test if > >you're interested. > > Sure, I am happy to help testing as always :-) Great, thanks! Patch attached. > Well on reflection, maybe people use the lazy mode more than snapshot? I think so, I think that's why this wasn't discovered before. > maybe some feedback from other box users could confirm ? as this does > provide a better recovery solution to files which are always changing, > and presumably the connection would be automatically re-tried on the > next lazy run ? in addition to any client retries currently coded for.. Yes, the retry was entirely dependent on lazy mode being enabled. > Sorry, yes my bad I was using the wording as per the Box Backup web page: > > Panned future work > > * Mark state (marks the current state of the filesystem, to emulate a tape > change and restoring from that tape later) OK, we should probably tidy that up and firm it up into a proposal. Although when anyone will have time to implement it is anyone's guess right now. > So far we have only had to restore the odd files here and there that > people have deleted/corrupted.. however I am not sure how you would go > about recovering so a whole directory from 2 days ago, is that possible > currently ? Yes it is, did you see my email responding to Pete J about this a couple of days ago? The main issue is, as Ben pointed out, that files deleted more than two days ago and still on the store will also be restored, because we don't track deletion dates. > Please don't think I am knocking the project, its works very well and > has been 100% successful in recovering data ! I wasnt sure if I had hit > a bug, but would appear to be more of a feature request :-) I think this is a bug, and I really appreciate you taking the time yet again to help track it down and verify the fix. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | ---128931150-1617812520-1208129094=:25757 Content-Type: TEXT/PLAIN; charset=US-ASCII; name=backupdaemon-snapshot-error-recovery.patch Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename=backupdaemon-snapshot-error-recovery.patch SW5kZXg6IGJpbi9iYmFja3VwZC9CYWNrdXBEYWVtb24uY3BwDQo9PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09DQotLS0gYmluL2JiYWNrdXBkL0JhY2t1cERhZW1v bi5jcHAJKHJldmlzaW9uIDIxNDgpDQorKysgYmluL2JiYWNrdXBkL0JhY2t1 cERhZW1vbi5jcHAJKHdvcmtpbmcgY29weSkNCkBAIC03NTYsNiArNzU2LDgg QEANCiAJLy8gU2V0IHN0YXRlDQogCVNldFN0YXRlKFN0YXRlX0lkbGUpOw0K IA0KKwlib29sIGRvU3luY0ZvcmNlZEJ5UHJldmlvdXNTeW5jRXJyb3IgPSBm YWxzZTsNCisNCiAJLy8gTG9vcCBhcm91bmQgZG9pbmcgYmFja3Vwcw0KIAlk bw0KIAl7DQpAQCAtNzg2LDcgKzc4OCw5IEBADQogCQkJCS8vIElmIHRoZXJl IGlzbid0IGF1dG9tYXRpYyBiYWNrdXAgaGFwcGVuaW5nLCANCiAJCQkJLy8g c2V0IGEgbG9uZyBkZWxheS4gQW5kIGxpbWl0IGRlbGF5cyBhdCB0aGUgDQog CQkJCS8vIHNhbWUgdGltZS4NCi0JCQkJaWYoIWF1dG9tYXRpY0JhY2t1cCB8 fCByZXF1aXJlZERlbGF5ID4gDQorCQkJCWlmKCEoYXV0b21hdGljQmFja3Vw IHx8DQorCQkJCQlkb1N5bmNGb3JjZWRCeVByZXZpb3VzU3luY0Vycm9yKSB8 fA0KKwkJCQkJcmVxdWlyZWREZWxheSA+IA0KIAkJCQkJU2Vjb25kc1RvQm94 VGltZShNQVhfU0xFRVBfVElNRSkpDQogCQkJCXsNCiAJCQkJCXJlcXVpcmVk RGVsYXkgPSBTZWNvbmRzVG9Cb3hUaW1lKA0KQEAgLTgyMiwxMyArODI2LDE0 IEBADQogCQkJCQl9DQogCQkJCX0NCiAJCQkJDQotCQkJfSB3aGlsZSgoIWF1 dG9tYXRpY0JhY2t1cCB8fCAoY3VycmVudFRpbWUgPCBuZXh0U3luY1RpbWUp KSAmJiAhZG9TeW5jICYmICFTdG9wUnVuKCkpOw0KKwkJCX0gd2hpbGUoKCFh dXRvbWF0aWNCYWNrdXAgfHwgKGN1cnJlbnRUaW1lIDwgbmV4dFN5bmNUaW1l KSkgJiYgIWRvU3luYyAmJiAhZG9TeW5jRm9yY2VkQnlQcmV2aW91c1N5bmNF cnJvciAmJiAhU3RvcFJ1bigpKTsNCiAJCX0NCiANCiAJCS8vIFRpbWUgb2Yg c3luYyBzdGFydCwgYW5kIGlmIGl0J3MgdGltZSBmb3IgYW5vdGhlciBzeW5j IA0KIAkJLy8gKGFuZCB3ZSdyZSBkb2luZyBhdXRvbWF0aWMgc3luY3MpLCBz ZXQgdGhlIGZsYWcNCiAJCWJveF90aW1lX3QgY3VycmVudFN5bmNTdGFydFRp bWUgPSBHZXRDdXJyZW50Qm94VGltZSgpOw0KLQkJaWYoYXV0b21hdGljQmFj a3VwICYmIGN1cnJlbnRTeW5jU3RhcnRUaW1lID49IG5leHRTeW5jVGltZSkN CisJCWlmKChhdXRvbWF0aWNCYWNrdXAgfHwgZG9TeW5jRm9yY2VkQnlQcmV2 aW91c1N5bmNFcnJvcikgJiYNCisJCQljdXJyZW50U3luY1N0YXJ0VGltZSA+ PSBuZXh0U3luY1RpbWUpDQogCQl7DQogCQkJZG9TeW5jID0gdHJ1ZTsNCiAJ CX0NCkBAIC0xMTEzLDYgKzExMTgsMTAgQEANCiAJCQkJCQlsYXN0U3luY1Rp bWUsIG5leHRTeW5jVGltZSk7DQogDQogCQkJCS8vIC0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQorDQorCQkJCS8v IElmIHdlIHdlcmUgcmV0cnlpbmcgYWZ0ZXIgYW4gZXJyb3IsDQorCQkJCS8v IG5vdyB3b3VsZCBiZSBhIGdvb2QgdGltZSB0byBzdG9wIDotKQ0KKwkJCQlk b1N5bmNGb3JjZWRCeVByZXZpb3VzU3luY0Vycm9yID0gZmFsc2U7DQogCQkJ fQ0KIAkJCWNhdGNoKEJveEV4Y2VwdGlvbiAmZSkNCiAJCQl7DQpAQCAtMTE5 Miw2ICsxMjAxLDcgQEANCiAJCQkJCQlSYW5kb206OlJhbmRvbUludCgNCiAJ CQkJCQkJdXBkYXRlU3RvcmVJbnRlcnZhbCA+PiANCiAJCQkJCQkJU1lOQ19Q RVJJT0RfUkFORE9NX0VYVFJBX1RJTUVfU0hJRlRfQlkpOw0KKwkJCQkJZG9T eW5jRm9yY2VkQnlQcmV2aW91c1N5bmNFcnJvciA9IHRydWU7DQogCQkJCX0N CiAJCQl9DQogDQo= ---128931150-1617812520-1208129094=:25757-- From boxbackup at fluffy.co.uk Wed Apr 16 14:31:32 2008 From: boxbackup at fluffy.co.uk (Tom Albers) Date: Wed, 16 Apr 2008 15:31:32 +0200 Subject: [Box Backup] Output for deleted files. Message-ID: <1300888.LlY6XQFQjK@kovoks.nl> --nextPart2944403.nfCgM8ggzE Content-Type: text/plain Content-Transfer-Encoding: 7Bit Chris, Can you output something for files that are marked as deleted. So this output can be used for the statistics? We see the uploaded files, etc, but don't see any output regarding files that are deleted from the disk. Which I think is pretty crucial for the statistics. The backup-maintainer can check if someone within the company has deleted something from the disk that should not be deleted. Best, Tom Albers -- KovoKs B.V. KvK:11033334 --nextPart2944403.nfCgM8ggzE-- From boxbackup at fluffy.co.uk Thu Apr 17 01:14:18 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Thu, 17 Apr 2008 01:14:18 +0100 (BST) Subject: [Box Backup] Output for deleted files. In-Reply-To: <1300888.LlY6XQFQjK@kovoks.nl> References: <1300888.LlY6XQFQjK@kovoks.nl> Message-ID: Hi Tom, On Wed, 16 Apr 2008, Tom Albers wrote: > Can you output something for files that are marked as deleted. So this > output can be used for the statistics? We see the uploaded files, etc, > but don't see any output regarding files that are deleted from the disk. > Which I think is pretty crucial for the statistics. The > backup-maintainer can check if someone within the company has deleted > something from the disk that should not be deleted. Yes, I'm working on it, but it's not trivial because we don't currently save enough information in the delete list to identify the files which are to be deleted to a human (only directory ID and filename). So it's quite an invasive patch and I'm not sure it should be in 0.11. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sat Apr 19 13:51:15 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Sat, 19 Apr 2008 13:51:15 +0100 Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: Hi Chris, >> Sure, I am happy to help testing as always :-) > > Great, thanks! Patch attached. Just to give you some feedback on the patch, the connections do appear to restore and continue ... Apr 19 05:40:19 hercules Box Backup (bbstored)[4399]: WARNING: Exception thrown: BackupStoreException(AddedFileExceedsStorageLimit) at BackupContext.cpp(569) Apr 19 05:40:20 hercules Box Backup (bbstored)[4399]: NOTICE: Session finished for Client ID 0x00000001 Apr 19 05:40:30 hercules Box Backup (bbstored)[32470]: WARNING: Message from child process 6965: Incoming connection from x.x.x.x port 57108 Apr 19 05:40:31 hercules Box Backup (bbstored)[6965]: NOTICE: Login from Client ID 0x00000001 Read/Write Apr 19 05:40:31 hercules Box Backup (bbstored)[6965]: Sending stream, size 232 Apr 19 05:40:32 hercules Box Backup (bbstored)[6965]: Sending stream, size 577 Apr 19 05:40:32 hercules Box Backup (bbstored)[6965]: Sending stream, size 1993 Apr 19 05:40:32 hercules Box Backup (bbstored)[6965]: Sending stream, size 201 Apr 19 05:40:32 hercules Box Backup (bbstored)[6965]: Sending stream, size 4537 Apr 19 05:40:32 hercules Box Backup (bbstored)[6965]: Sending stream, size 89 Here I set the limit for this account to 5G and once the store had hit this limit the connection stopped, but then re-initiated the backup... Regards Matt From boxbackup at fluffy.co.uk Sun Apr 20 16:56:30 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 20 Apr 2008 16:56:30 +0100 (BST) Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: Hi Matt, On Sat, 19 Apr 2008, Matt Brown wrote: >> Great, thanks! Patch attached. > > Just to give you some feedback on the patch, the connections do appear > to restore and continue ... [...] > Here I set the limit for this account to 5G and once the store had hit > this limit the connection stopped, but then re-initiated the backup... Looks fine to me. Is this the correct behaviour for you? Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at fluffy.co.uk Mon Apr 21 09:45:02 2008 From: boxbackup at fluffy.co.uk (Timothy Wilson) Date: Mon, 21 Apr 2008 18:45:02 +1000 Subject: [Box Backup] Restoring files to the bbstored machine Message-ID: Hello guys, I was wondering if it is possible to restore some files from a backup to some location on my server? For example, I might have an account '000001'; can I restore some files using the bbackupquery tool to a location on the server, rather than the client? The only way I can think of doing this is by running the backup client on the server with account 000001's config and certs. However, this is going to be problematic if I need to do this for more than a handful of clients.. Kind regards, Timothy. From boxbackup at fluffy.co.uk Mon Apr 21 18:54:51 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Mon, 21 Apr 2008 18:54:51 +0100 Subject: [Box Backup] Connection fails during backup to remote server In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> Message-ID: <057998E1-1C1B-4297-83ED-090E9A69B448@mbrown.co.uk> Hi Chris, > Looks fine to me. Is this the correct behaviour for you? I think so, now I have increased the limits I will let it run for a few days and watch the behavior ... I will keep you posted. Regards Matt From boxbackup at fluffy.co.uk Mon Apr 21 13:21:54 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Mon, 21 Apr 2008 13:21:54 +0100 (BST) Subject: [Box Backup] Restoring files to the bbstored machine In-Reply-To: References: Message-ID: Hi Timothy, On Mon, 21 Apr 2008, Timothy Wilson wrote: > I was wondering if it is possible to restore some files from a backup to > some location on my server? For example, I might have an account > '000001'; can I restore some files using the bbackupquery tool to a > location on the server, rather than the client? The only way I can think > of doing this is by running the backup client on the server with account > 000001's config and certs. However, this is going to be problematic if I > need to do this for more than a handful of clients.. Not unless you have the certificates and encryption keys. This is by design. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Mon Apr 21 18:23:41 2008 From: boxbackup at fluffy.co.uk (Dave Bamford) Date: Mon, 21 Apr 2008 18:23:41 +0100 Subject: [Box Backup] Restoring files to the bbstored machine In-Reply-To: References: Message-ID: <480CCD9D.4070706@logical-progress.com> I have written a web page with php and MySQL to do this for multiple clients. You can specify a config file with the bbackupquery command so I use the format bbackupd.xxxxxx where xxxxx is the account number then you just need to point at the keys from the config file and restore to a directory of your choice. Its safer to use a web server on the internal network to do this. If you want the php code the I can dig it out for you. Best regards Dave Bamford. Timothy Wilson wrote: > Hello guys, > > I was wondering if it is possible to restore some files from a backup > to some location on my server? For example, I might have an account > '000001'; can I restore some files using the bbackupquery tool to a > location on the server, rather than the client? The only way I can > think of doing this is by running the backup client on the server with > account 000001's config and certs. However, this is going to be > problematic if I need to do this for more than a handful of clients.. > > Kind regards, > Timothy. > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > > From boxbackup at fluffy.co.uk Mon Apr 21 21:32:13 2008 From: boxbackup at fluffy.co.uk (Jason) Date: Mon, 21 Apr 2008 16:32:13 -0400 Subject: [Box Backup] exception BackupStore InvalidBackupStoreFilename (4/3) Message-ID: Hello, I've been using boxbackup for a while now, but unfortunately I haven't been paying as much attention to my logs as I should have. It appears I've been getting this message in the logs for quite some time now, so I'm not sure what happened in order to cause it to start. Apr 21 10:13:33 [bbstored/hk] while housekeeping account 00000003, exception BackupStore InvalidBackupStoreFilename (4/3) -- aborting housekeeping run for this account I looked for more information on this exception in trac, the exceptions list on the website, and in the mailing list archives but didn't find any information on what map be happening. I even took a brief look at the code which throws this exception, but it wasn't immediately obvious to me what was going on. So, can anyone tell me what could be causing this and how I might fix it? This is on the bbstored server, and I've got a few other clients that connect to this server and they have identical boxbackup configuration files, I believe. So perhaps it has something to do with some of the files that are actually being backed up? Well, any help is appreciated. Thank you very much. Jason From boxbackup at fluffy.co.uk Mon Apr 21 22:27:42 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Mon, 21 Apr 2008 22:27:42 +0100 (BST) Subject: [Box Backup] exception BackupStore InvalidBackupStoreFilename (4/3) In-Reply-To: References: Message-ID: Hi Jason, > I've been using boxbackup for a while now, but unfortunately I haven't > been paying as much attention to my logs as I should have. It appears > I've been getting this message in the logs for quite some time now, so > I'm not sure what happened in order to cause it to start. > > Apr 21 10:13:33 [bbstored/hk] while housekeeping account 00000003, > exception BackupStore InvalidBackupStoreFilename (4/3) -- aborting > housekeeping run for this account > > I looked for more information on this exception in trac, the > exceptions list on the website, and in the mailing list archives but > didn't find any information on what map be happening. I even took a > brief look at the code which throws this exception, but it wasn't > immediately obvious to me what was going on. > > So, can anyone tell me what could be causing this Probably corruption of the filesystem that the store is on. > and how I might fix > it? bbstoreaccounts check fix > This is on the bbstored server, and I've got a few other clients that > connect to this server and they have identical boxbackup configuration > files, I believe. So perhaps it has something to do with some of the > files that are actually being backed up? I hope not, but if it reappears let us know. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Mon Apr 21 22:50:00 2008 From: boxbackup at fluffy.co.uk (Victor Meirans) Date: Tue, 22 Apr 2008 00:50:00 +0300 Subject: [Box Backup] Restoring files to the bbstored machine In-Reply-To: <480CCD9D.4070706@logical-progress.com> References: <480CCD9D.4070706@logical-progress.com> Message-ID: <480D0C08.8010403@virtualis.lv> Hello Dave, apparently I am looking for this solution too, so if you can, please send me too that code, It would be highly appreciated... Best regards, -- ViC on 2008.04.21 20:23 Dave Bamford said the following: > I have written a web page with php and MySQL to do this for multiple > clients. You can specify a config file with the bbackupquery command > so I use the format bbackupd.xxxxxx where xxxxx is the account number > then you just need to point at the keys from the config file and restore > to a directory of your choice. > Its safer to use a web server on the internal network to do this. > > If you want the php code the I can dig it out for you. > > Best regards > > Dave Bamford. > > Timothy Wilson wrote: >> Hello guys, >> >> I was wondering if it is possible to restore some files from a backup >> to some location on my server? For example, I might have an account >> '000001'; can I restore some files using the bbackupquery tool to a >> location on the server, rather than the client? The only way I can >> think of doing this is by running the backup client on the server with >> account 000001's config and certs. However, this is going to be >> problematic if I need to do this for more than a handful of clients.. >> >> Kind regards, >> Timothy. >> _______________________________________________ >> boxbackup mailing list >> boxbackup at fluffy.co.uk >> http://lists.warhead.org.uk/mailman/listinfo/boxbackup >> >> > > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Tue Apr 22 00:00:48 2008 From: boxbackup at fluffy.co.uk (Timothy Wilson) Date: Tue, 22 Apr 2008 09:00:48 +1000 Subject: [Box Backup] Restoring files to the bbstored machine In-Reply-To: <480CCD9D.4070706@logical-progress.com> References: <480CCD9D.4070706@logical-progress.com> Message-ID: Hello Dave, That code would benefit me too. If you could find it, I'd be most appreciative! @Chris: yes, I have they keys for each machine :) Kind regards, Timothy. On 22/04/2008, Dave Bamford wrote: > I have written a web page with php and MySQL to do this for multiple > clients. You can specify a config file with the bbackupquery command > so I use the format bbackupd.xxxxxx where xxxxx is the account number > then you just need to point at the keys from the config file and restore > to a directory of your choice. > Its safer to use a web server on the internal network to do this. > > If you want the php code the I can dig it out for you. > > Best regards > > Dave Bamford. > > Timothy Wilson wrote: > > > > > Hello guys, > > > > I was wondering if it is possible to restore some files from a backup > > to some location on my server? For example, I might have an account > > '000001'; can I restore some files using the bbackupquery tool to a > > location on the server, rather than the client? The only way I can > > think of doing this is by running the backup client on the server with > > account 000001's config and certs. However, this is going to be > > problematic if I need to do this for more than a handful of clients.. > > > > Kind regards, > > Timothy. > > _______________________________________________ > > boxbackup mailing list > > boxbackup at fluffy.co.uk > > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > > > > > > > > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > From boxbackup at fluffy.co.uk Wed Apr 23 20:48:13 2008 From: boxbackup at fluffy.co.uk (Dave Bamford) Date: Wed, 23 Apr 2008 20:48:13 +0100 Subject: [Box Backup] Restoring files to the bbstored machine In-Reply-To: References: <480CCD9D.4070706@logical-progress.com> Message-ID: <480F927D.7000700@logical-progress.com> Hi here is the code you can probably infer the sql table format from it it also updates the usage in the table. It needs some refinement but basically works. You have to ensure that the folder used to recover files is writeable by Apache. regards Dave '.$file.''; } } // now do the files echo '
ID
Flags
Date
Time
Size
Name
'; foreach($retval as $entry) { // split id flags date file $id = substr($entry,0,8); $flags = substr($entry,9,6); $date = substr($entry,16,10); $time = substr($entry,27,8); $size = substr($entry,36,5); $file = substr($entry,42); if (substr($flags,0,1) == 'f'){ echo '
'.$id.'
'.$flags.'
'.$date.'
'.$time.'
'.$size.'
'; } } } $chown = 'sudo /usr/sbin/chown _www'; $tmpdir = '/tmp'; ?> box backup query account=".$_POST['account']; $_SESSION['account'] = $_POST['account']; } ?>
show usage'; if (isset($_GET['wd'])){ if (!strcmp($_GET['wd'],'showusage')) { exec($bquery.' usage exit',$retval); list($r0,$r1,$used) = split("[\t ]+",$retval[0]); list($r0,$r1,$r2,$old) = split("[\t ]+",$retval[1]); list($r0,$r1,$r2,$deleted) = split("[\t ]+",$retval[2]); list($r0,$r1,$directories) = split("[\t ]+",$retval[3]); list($r0,$r1,$r2,$softlim) = split("[\t ]+",$retval[4]); list($r0,$r1,$r2,$hardlim) = split("[\t ]+",$retval[5]); $used = trim($used,'Mb'); $old = trim($old,'Mb'); $deleted = trim($deleted,'Mb'); $directories = trim($directories,'Mb'); $softlim = trim($softlim,'Mb'); $hardlim = trim($hardlim,'Mb'); $sql = 'UPDATE account SET `usage` ='.$used.', `old` = '.$old.', `deleted` = '.$deleted.', `directories` = '.$directories.', `softlim` = '.$softlim.', `hardlim` = '.$hardlim.' WHERE num = "'.$_SESSION['account'].'"'; //echo "sql=".$sql; if ($ret = mysql_query ($sql)){ echo "
used=".$used."
"; echo "old=".$old."
"; echo "deleted=".$deleted."
"; echo "directories=".$directories."
"; echo "soft limit=".$softlim."
"; echo "hard limit=".$hardlim."
"; echo "
Database updated"; } else { echo "
Failed to update database"; } } else { exec($bquery.' "list -osdt \"'.$_GET['wd'].'\"" exit', $retval); $parent = substr($_GET['wd'],0,strrpos($_GET['wd'],"/")); echo '
back to '.$parent.''; echo '
current folder = '.$_GET['wd']; displayfiles($retval); } } else { exec($bquery.' "list -osdt" exit', $retval); displayfiles($retval); } if (isset($_GET['id'])){ echo "
getting file"; echo($bquery.'"cd \"'.$_GET['wd'].'\"" "get -i '.$_GET['id'].' \"/home/dave/'.$_GET['file'].'\"" exit'); exec($bquery.' "cd \"'.$_GET['wd'].'\"" "get -i '.$_GET['id'].' \"/home/dave/'.$_GET['file'].'\"" exit',$retval); } ?> From boxbackup at fluffy.co.uk Wed Apr 23 19:32:40 2008 From: boxbackup at fluffy.co.uk (Stuart D. Gathman) Date: Wed, 23 Apr 2008 14:32:40 -0400 (EDT) Subject: [Box Backup] Alternate server Message-ID: Looking at the overview, the server daemon just keeps files in a tree - and not the real filenames either. So services like S3 could be the server with some tweaking to use HTTP on the client end. It is hard to find remote backup software for linux that does both encryption and incremental - so this could be a popular option. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From boxbackup at fluffy.co.uk Wed Apr 23 21:05:27 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 23 Apr 2008 21:05:27 +0100 (BST) Subject: [Box Backup] Alternate server In-Reply-To: References: Message-ID: Hi Stuart, On Wed, 23 Apr 2008, Stuart D. Gathman wrote: > Looking at the overview, the server daemon just keeps files in a tree - > and not the real filenames either. So services like S3 could be the > server with some tweaking to use HTTP on the client end. It is hard to > find remote backup software for linux that does both encryption and > incremental - so this could be a popular option. Yes, you are not the first to suggest this and I am planning to implement it. The main problems are: * S3 does not support files over 2GB, so they have to be split up * S3 is unreliable (frequent failure to write a file, with error 500, that subsequently succeeds if retried) * S3 does not support the client-server bandwidth efficiency stuff that bbackupd does when talking to bbstored, so you have to run bbstored on an EC2 server * Testing all this would get expensive for me (monthly charges for an EC2 server and S3 account and storage of test data) * I need to write an S3 emulator for the unit tests Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Wed Apr 23 22:13:24 2008 From: boxbackup at fluffy.co.uk (Stuart D. Gathman) Date: Wed, 23 Apr 2008 17:13:24 -0400 (EDT) Subject: [Box Backup] Alternate server In-Reply-To: Message-ID: On Wed, 23 Apr 2008, Chris Wilson wrote: > Yes, you are not the first to suggest this and I am planning to implement > it. The main problems are: > > * S3 does not support files over 2GB, so they have to be split up Sounds nasty. > * S3 is unreliable (frequent failure to write a file, with error 500, that > subsequently succeeds if retried) Ditto. > * S3 does not support the client-server bandwidth efficiency stuff that > bbackupd does when talking to bbstored, so you have to run bbstored on > an EC2 server So client is not trivially adapted for S3. It is easier to specialize the server to use S3 for backend. Especially when it needs to handle nastiness like the above. > * Testing all this would get expensive for me (monthly charges for an EC2 > server and S3 account and storage of test data) > * I need to write an S3 emulator for the unit tests For this, just use a local web server with PUT/DELETE enabled. Put warnings in, and rely on early adopters to test with actual S3. Should just be a change of URL and authentication. Thanks for the info. Looks like I'll be using "across-town" remote backup for the time being with boxbackup if I can get it set up easily (otherwise I'll use rsync). It is a great feature that the server runs as a regular user! "Across-town" handles fire and tornadoes, just not nuclear attack. And frankly, in that case, I won't care! -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From boxbackup at fluffy.co.uk Wed Apr 23 22:19:43 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 23 Apr 2008 22:19:43 +0100 (BST) Subject: [Box Backup] Alternate server In-Reply-To: References: Message-ID: Hi Stuart, On Wed, 23 Apr 2008, Stuart D. Gathman wrote: > > * S3 does not support the client-server bandwidth efficiency stuff > > that bbackupd does when talking to bbstored, so you have to run > > bbstored on an EC2 server > > So client is not trivially adapted for S3. It is easier to specialize > the server to use S3 for backend. Especially when it needs to handle > nastiness like the above. Yes. > > * Testing all this would get expensive for me (monthly charges for an EC2 > > server and S3 account and storage of test data) ... > For this, just use a local web server with PUT/DELETE enabled. Put > warnings in, and rely on early adopters to test with actual S3. Should > just be a change of URL and authentication. How will I know that it behaves the same as S3? Especially with regard to some features that might be useful, such as partial file retrieval with Byte-Range. Also, what webserver supports this? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Thu Apr 24 01:23:22 2008 From: boxbackup at fluffy.co.uk (Pablo Fernandez) Date: Wed, 23 Apr 2008 21:23:22 -0300 Subject: [Box Backup] Alternate server In-Reply-To: References: Message-ID: <1208996602.6915.3.camel@laptop> Hi For this I'm sure we can all put together enough funding for this. GIven the low rates they have I'm sure something like even $50 would go a long way on testing... That would buy about 50 GB. Best regards, Pablo On Wed, 2008-04-23 at 21:05 +0100, Chris Wilson wrote: > Hi Stuart, > > On Wed, 23 Apr 2008, Stuart D. Gathman wrote: > > > Looking at the overview, the server daemon just keeps files in a tree - > > and not the real filenames either. So services like S3 could be the > > server with some tweaking to use HTTP on the client end. It is hard to > > find remote backup software for linux that does both encryption and > > incremental - so this could be a popular option. > > Yes, you are not the first to suggest this and I am planning to implement > it. The main problems are: > > * S3 does not support files over 2GB, so they have to be split up > * S3 is unreliable (frequent failure to write a file, with error 500, that > subsequently succeeds if retried) > * S3 does not support the client-server bandwidth efficiency stuff that > bbackupd does when talking to bbstored, so you have to run bbstored on > an EC2 server > * Testing all this would get expensive for me (monthly charges for an EC2 > server and S3 account and storage of test data) > * I need to write an S3 emulator for the unit tests > > Cheers, Chris. From boxbackup at fluffy.co.uk Thu Apr 24 01:46:01 2008 From: boxbackup at fluffy.co.uk (Stuart D. Gathman) Date: Wed, 23 Apr 2008 20:46:01 -0400 (EDT) Subject: [Box Backup] Alternate server In-Reply-To: Message-ID: On Wed, 23 Apr 2008, Chris Wilson wrote: > > For this, just use a local web server with PUT/DELETE enabled. Put > > warnings in, and rely on early adopters to test with actual S3. Should > > just be a change of URL and authentication. > > How will I know that it behaves the same as S3? Especially with regard to > some features that might be useful, such as partial file retrieval with > Byte-Range. Also, what webserver supports this? Apache supports put/delete and byte-range. However, I am looking at the S3 docs. Amazon says max object is now 5G. Still will need splitting, just harder to test :-) S3 does not use basic authentication over https, it uses http (you need to store encrypted files for privacy) and their own custom AWS authentication. You could still do useful testing with Basic authentication (uses same header) on a local server. S3 using the REST interface will store arbitrary metadata using custom HTTP headers, and supports md5 based if-modified. That, and the XML responses for errors and retrieving buckets (directories) would require servlet emulation. (Apache returns HTML intended to display as a directory when you GET a directory.) So, while I think you *could* write a storage engine that worked with both S3 and apache, it would use only a subset of functionality. I would agree that the emulator is the way to go. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From boxbackup at fluffy.co.uk Fri Apr 25 10:13:13 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Fri, 25 Apr 2008 10:13:13 +0100 Subject: [Box Backup] Connection fails during backup to remote server *Resolved* In-Reply-To: <057998E1-1C1B-4297-83ED-090E9A69B448@mbrown.co.uk> References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> <057998E1-1C1B-4297-83ED-090E9A69B448@mbrown.co.uk> Message-ID: Hi Chris, > I will keep you posted. The patch is working great for me ... The server and store have now fully sync'd but here was an example from earlier this week which caused a few retry events... Apr 21 23:29:00 orion Backup (bbackupd)[27749]: Synchronised file: / data/Documents_unfiled/Leon 2.dvdproj/Contents/Resources/ MPEG /._V_ED4C146C5942D440FD5F228236F9FDCE6D6F7669_6000000_P_S1447.960_E2229.880 Apr 21 23:29:00 orion Backup (bbackupd)[27749]: Uploading complete file: /data/Documents_unfiled/Leon 2.dvdproj/Contents/Resources/MPEG/ V_ED4C146C5942D440FD5F228236F9FDCE6D6F7669_6000000_P_S2229.880_E2894.840 Apr 22 01:45:05 orion Backup (bbackupd)[27749]: WARNING: Exception thrown: ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(442) Apr 22 01:45:05 orion Backup (bbackupd)[27749]: ERROR: Failed to upload file: /data/Documents_unfiled/Leon 2.dvdproj/Contents/Resources/ MPEG/ V_ED4C146C5942D440FD5F228236F9FDCE6D6F7669_6000000_P_S2229.880_E2894.840 : caught exception: Connection TLSWriteFailed (Probably a network issue between client and server.) (7/33) Apr 22 01:45:05 orion Backup (bbackupd)[27749]: ERROR: SSL error during Write: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry Apr 22 01:45:05 orion Backup (bbackupd)[27749]: WARNING: Exception thrown: ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(442) Apr 22 01:45:05 orion Backup (bbackupd)[27749]: NOTICE: About to notify administrator about event backup-error, running script '/etc/ box/bbackupd/NotifySysadmin.sh backup-error' Apr 22 01:45:05 orion Backup (bbackupd)[27749]: ERROR: Exception caught (Connection TLSWriteFailed (Probably a network issue between client and server.) 7/33), reset state and waiting to retry... Apr 22 01:45:15 orion Backup (bbackupd)[27749]: NOTICE: File statistics: total file size uploaded 753134760, bytes already on server 95948772, encoded size 475659264 Apr 22 01:45:15 orion Backup (bbackupd)[27749]: NOTICE: Beginning scan of local files Apr 22 01:45:15 orion Backup (bbackupd)[27749]: Opening connection to server 'hercules.xxxxxxxxxx'... Apr 22 01:45:20 orion Backup (bbackupd)[27749]: WARNING: Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1135) Apr 22 01:45:20 orion Backup (bbackupd)[27749]: WARNING: Suppressing duplicate notification about backup-error Apr 22 01:45:20 orion Backup (bbackupd)[27749]: ERROR: Exception caught (Connection Protocol_UnexpectedReply (Server probably reported an error.) 7/47), reset state and waiting to retry... Apr 22 01:45:30 orion Backup (bbackupd)[27749]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Apr 22 01:46:45 orion Backup (bbackupd)[27749]: NOTICE: Beginning scan of local files Apr 22 01:46:45 orion Backup (bbackupd)[27749]: Opening connection to server 'hercules.xxxxxxx'... Apr 22 01:46:50 orion Backup (bbackupd)[27749]: WARNING: Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1135) Apr 22 01:46:50 orion Backup (bbackupd)[27749]: WARNING: Suppressing duplicate notification about backup-error Apr 22 01:46:50 orion Backup (bbackupd)[27749]: ERROR: Exception caught (Connection Protocol_UnexpectedReply (Server probably reported an error.) 7/47), reset state and waiting to retry... Apr 22 01:47:00 orion Backup (bbackupd)[27749]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Apr 22 01:48:15 orion Backup (bbackupd)[27749]: NOTICE: Beginning scan of local files Apr 22 01:48:15 orion Backup (bbackupd)[27749]: Opening connection to server 'hercules.xxxxxxx'... Apr 22 01:48:20 orion Backup (bbackupd)[27749]: WARNING: Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1135) Apr 22 01:48:20 orion Backup (bbackupd)[27749]: WARNING: Suppressing duplicate notification about backup-error Apr 22 01:48:20 orion Backup (bbackupd)[27749]: ERROR: Exception caught (Connection Protocol_UnexpectedReply (Server probably reported an error.) 7/47), reset state and waiting to retry... Apr 22 01:48:30 orion Backup (bbackupd)[27749]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Apr 22 01:49:45 orion Backup (bbackupd)[27749]: NOTICE: Beginning scan of local files Apr 22 01:49:45 orion Backup (bbackupd)[27749]: Opening connection to server 'hercules.xxxxxxx'... Apr 22 01:49:50 orion Backup (bbackupd)[27749]: WARNING: Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1135) Apr 22 01:49:50 orion Backup (bbackupd)[27749]: WARNING: Suppressing duplicate notification about backup-error Apr 22 01:49:50 orion Backup (bbackupd)[27749]: ERROR: Exception caught (Connection Protocol_UnexpectedReply (Server probably reported an error.) 7/47), reset state and waiting to retry... Apr 22 01:50:00 orion Backup (bbackupd)[27749]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Apr 22 01:51:15 orion Backup (bbackupd)[27749]: NOTICE: Beginning scan of local files Apr 22 01:51:15 orion Backup (bbackupd)[27749]: Opening connection to server 'hercules.xxxxxxx'... Apr 22 01:51:20 orion Backup (bbackupd)[27749]: WARNING: Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1135) Apr 22 01:51:20 orion Backup (bbackupd)[27749]: WARNING: Suppressing duplicate notification about backup-error Apr 22 01:51:20 orion Backup (bbackupd)[27749]: ERROR: Exception caught (Connection Protocol_UnexpectedReply (Server probably reported an error.) 7/47), reset state and waiting to retry... Apr 22 01:51:30 orion Backup (bbackupd)[27749]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Apr 22 01:52:45 orion Backup (bbackupd)[27749]: NOTICE: Beginning scan of local files Apr 22 01:52:45 orion Backup (bbackupd)[27749]: Opening connection to server 'hercules.xxxxxxx'... Apr 22 01:52:50 orion Backup (bbackupd)[27749]: WARNING: Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1135) Apr 22 01:52:50 orion Backup (bbackupd)[27749]: WARNING: Suppressing duplicate notification about backup-error Apr 22 01:52:50 orion Backup (bbackupd)[27749]: ERROR: Exception caught (Connection Protocol_UnexpectedReply (Server probably reported an error.) 7/47), reset state and waiting to retry... Apr 22 01:53:00 orion Backup (bbackupd)[27749]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Apr 22 01:54:15 orion Backup (bbackupd)[27749]: NOTICE: Beginning scan of local files Apr 22 01:54:15 orion Backup (bbackupd)[27749]: Opening connection to server 'hercules.xxxxxxx'... Apr 22 01:54:20 orion Backup (bbackupd)[27749]: WARNING: Exception thrown: ConnectionException(Conn_Protocol_UnexpectedReply) at autogen_BackupProtocolClient.cpp(1135) Apr 22 01:54:20 orion Backup (bbackupd)[27749]: WARNING: Suppressing duplicate notification about backup-error Apr 22 01:54:20 orion Backup (bbackupd)[27749]: ERROR: Exception caught (Connection Protocol_UnexpectedReply (Server probably reported an error.) 7/47), reset state and waiting to retry... Apr 22 01:54:30 orion Backup (bbackupd)[27749]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Apr 22 01:55:45 orion Backup (bbackupd)[27749]: NOTICE: Beginning scan of local files Apr 22 01:55:45 orion Backup (bbackupd)[27749]: Opening connection to server 'hercules.xxxxxxx'... Apr 22 01:55:46 orion Backup (bbackupd)[27749]: Connection made, login successful Apr 22 01:55:46 orion Backup (bbackupd)[27749]: Not deleting unused entries - none in list Apr 22 01:55:46 orion Backup (bbackupd)[27749]: NOTICE: About to notify administrator about event backup-start, running script '/etc/ box/bbackupd/NotifySysadmin.sh backup-start' Apr 22 01:55:46 orion Backup (bbackupd)[27749]: Scanning directory: / home Apr 22 01:55:46 orion Backup (bbackupd)[27749]: Skipping excluded directory: /home/ftp Regards Matt From boxbackup at fluffy.co.uk Fri Apr 25 20:00:17 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 25 Apr 2008 20:00:17 +0100 (BST) Subject: [Box Backup] Connection fails during backup to remote server *Resolved* In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> <057998E1-1C1B-4297-83ED-090E9A69B448@mbrown.co.uk> Message-ID: Hi Matt, On Fri, 25 Apr 2008, Matt Brown wrote: > The patch is working great for me ... Great, many thanks for testing! > The server and store have now fully sync'd but here was an example from > earlier this week which caused a few retry events... Thanks, this showed me that there was a small bug in the patcg: after an error, it retried after exactly ten seconds, instead of approximately 100 seconds as it should have. I think I've fixed that now. I assume that the multiple Conn_Protocol_UnexpectedReply errors after the disconnection are because the store thinks that the client is still logged in, and therefore the account is write-locked and the client can't log in again. There's not enough debugging code to verify that assumption, so I'm adding more now. I'm still surprised that you're being disconnected when you have KeepAliveTime enabled. Would it be possible for you to run the client or server with ExtendedLogging enabled for a bit, so that we can check whether KeepAlive messages are actually being sent, and at the right intervals? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Fri Apr 25 22:59:25 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Fri, 25 Apr 2008 14:59:25 -0700 (PDT) Subject: [Box Backup] constant scan of local files on server Message-ID: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> Before I get into my question I wanted to say hello to you all. This software looks great and I'm looking forward to working with everyone. I'm running Debian Etch x86 on vmware-server 1.03, and I've installed the RC2 candidate for testing for my own use and possibly for clients later down the line. This is the first time I've worked with this software, but after reading some of the list archives some of my questions have been answered. So far as I can tell I've successfully installed the server and correctly set up the certs. When I run "bbstored SINGLEPROCESS -c /etc/box/bbstored.conf" I get this on the command line, and stays there until canceled: NOTICE: Starting daemon, version 0.11rc2, config: /etc/box/bbstored.conf but I also get these lines in my /var/log/messages file (I have not yet gotten the box log file separated), where it seems to scan local files every few seconds (see messages below). Is this normal, or something that needs to be fixed? Apr 25 14:49:18 debian Box Backup (bbackupd)[6109]: NOTICE: Beginning scan of local files Apr 25 14:49:18 debian Box Backup (bbackupd)[6109]: NOTICE: About to notify administrator about event backup-start, running script '/usr/bin/perl testfiles/notifyscript.pl backup-start' Apr 25 14:49:18 debian Box Backup (bbackupd)[6109]: NOTICE: Finished scan of local files Apr 25 14:49:18 debian Box Backup (bbackupd)[6109]: NOTICE: About to notify administrator about event backup-finish, running script '/usr/bin/perl testfiles/notifyscript.pl backup-finish' Apr 25 14:49:18 debian Box Backup (bbackupd)[6109]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Apr 25 14:49:21 debian Box Backup (bbackupd)[6109]: NOTICE: Beginning scan of local files Apr 25 14:49:21 debian Box Backup (bbackupd)[6109]: NOTICE: About to notify administrator about event backup-start, running script '/usr/bin/perl testfiles/notifyscript.pl backup-start' Apr 25 14:49:21 debian Box Backup (bbackupd)[6109]: NOTICE: Finished scan of local files Apr 25 14:49:21 debian Box Backup (bbackupd)[6109]: NOTICE: About to notify administrator about event backup-finish, running script '/usr/bin/perl testfiles/notifyscript.pl backup-finish' Apr 25 14:49:21 debian Box Backup (bbackupd)[6109]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Thank you! -- Sean From boxbackup at fluffy.co.uk Fri Apr 25 23:07:34 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 25 Apr 2008 23:07:34 +0100 (BST) Subject: [Box Backup] constant scan of local files on server In-Reply-To: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> Message-ID: Hi Admin, On Fri, 25 Apr 2008, admin at vancocomputing.net wrote: > I'm running Debian Etch x86 on vmware-server 1.03, and I've installed > the RC2 candidate for testing for my own use and possibly for clients > later down the line. This is the first time I've worked with this > software, but after reading some of the list archives some of my > questions have been answered. > > So far as I can tell I've successfully installed the server and > correctly set up the certs. When I run "bbstored SINGLEPROCESS -c > /etc/box/bbstored.conf" Where did you see the reference to SINGLEPROCESS? This command is outdated, I'm surprised that it works at all. If you want to run a Box backup daemon on the console, using the -DV options is a better way to do it. I get this on the command line, and stays there > until canceled: > > NOTICE: Starting daemon, version 0.11rc2, config: /etc/box/bbstored.conf > > but I also get these lines in my /var/log/messages file (I have not yet > gotten the box log file separated), where it seems to scan local files > every few seconds (see messages below). Is this normal, or something that > needs to be fixed? Did you configure the client's UpdateStoreInterval to 3 seconds? If so, then this is expected behaviour. If not, please post your configuration file. Also, it appears that your bbackupd is not configured to back up any files, is that correct? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Fri Apr 25 23:12:57 2008 From: boxbackup at fluffy.co.uk (Matt Brown) Date: Fri, 25 Apr 2008 23:12:57 +0100 Subject: [Box Backup] Connection fails during backup to remote server *Resolved* In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> <057998E1-1C1B-4297-83ED-090E9A69B448@mbrown.co.uk> Message-ID: Hi Chris, >> The patch is working great for me ... > > Great, many thanks for testing! No probs. >> The server and store have now fully sync'd but here was an example >> from >> earlier this week which caused a few retry events... > > Thanks, this showed me that there was a small bug in the patcg: > after an > error, it retried after exactly ten seconds, instead of > approximately 100 > seconds as it should have. I think I've fixed that now. Happy to try the new code if you like. > I assume that the multiple Conn_Protocol_UnexpectedReply errors > after the > disconnection are because the store thinks that the client is still > logged > in, and therefore the account is write-locked and the client can't > log in > again. There's not enough debugging code to verify that assumption, > so I'm > adding more now. > I'm still surprised that you're being disconnected when you have > KeepAliveTime enabled. Would it be possible for you to run the > client or > server with ExtendedLogging enabled for a bit, so that we can check > whether KeepAlive messages are actually being sent, and at the right > intervals? I don't think I enabled it, as I thought you wished me to continue with it off to see the behavior ? I could be wrong tho :-) I can switch it on if that helps ? Regards Matt From boxbackup at fluffy.co.uk Fri Apr 25 23:15:20 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 25 Apr 2008 23:15:20 +0100 (BST) Subject: [Box Backup] Connection fails during backup to remote server *Resolved* In-Reply-To: References: <081A821F-F1D2-40F4-A8A8-996900BEF1B0@mbrown.co.uk> <057998E1-1C1B-4297-83ED-090E9A69B448@mbrown.co.uk> Message-ID: Hi Matt, On Fri, 25 Apr 2008, Matt Brown wrote: > Happy to try the new code if you like. If you could test RC3 for me when it comes out (hopefully in a few days) that would be fantastic, and a much better use of your time :-) > >I'm still surprised that you're being disconnected when you have > >KeepAliveTime enabled. Would it be possible for you to run the client > >or server with ExtendedLogging enabled for a bit, so that we can check > >whether KeepAlive messages are actually being sent, and at the right > >intervals? > > I don't think I enabled it, as I thought you wished me to continue with > it off to see the behavior ? I could be wrong tho :-) Sorry, I forgot about that. Please could you check whether it's enabled for me? > I can switch it on if that helps ? Yes please, enable it and keep an eye out for repeats of the problem if you don't mind. Your assistance has been fantastic, thanks again! Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Fri Apr 25 23:15:55 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Fri, 25 Apr 2008 15:15:55 -0700 (PDT) Subject: [Box Backup] constant scan of local files on server In-Reply-To: References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> Message-ID: <5137.64.161.120.82.1209161755.squirrel@www.vancocomputing.net> Thank you for the quick response. Please call me Sean ;) > Where did you see the reference to SINGLEPROCESS? This command is > outdated, I'm surprised that it works at all. If you want to run a Box > backup daemon on the console, using the -DV options is a better way to do > it. I found that command in the list archives, back in March or April. I ran the command with the -DV options as you suggested and here is what I got: NOTICE: Starting daemon, version 0.11rc2, config: /etc/box/bbstored.conf Housekeeping process started Starting housekeeping Finished housekeeping I gave it about 10 minutes to change, but it did not after "Finished housekeeping". > Did you configure the client's UpdateStoreInterval to 3 seconds? If so, > then this is expected behaviour. If not, please post your configuration > file. I had not yet delved that deeply into the configs, so I guess shame on me. I'll look at it and make any necessary changes and get back to you on the change in behavior. After a quick google, isn't that setting for clients? Does the client get installed on the server automatically? I was under the impression that it did not, but I may have inadvertently installed the client at 1 am the other day. > Also, it appears that your bbackupd is not configured to back up any > files, is that correct? Yes, that is correct. For the last few days, as I've had time, I've been concentrating on just getting the server itself running correctly. -- Sean Van Couwenberghe Vanco Computing www.vancocomputing.net (707) 578-5802 From boxbackup at fluffy.co.uk Fri Apr 25 23:24:22 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 25 Apr 2008 23:24:22 +0100 (BST) Subject: [Box Backup] constant scan of local files on server In-Reply-To: <5137.64.161.120.82.1209161755.squirrel@www.vancocomputing.net> References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> <5137.64.161.120.82.1209161755.squirrel@www.vancocomputing.net> Message-ID: Hi Sean, On Fri, 25 Apr 2008, admin at vancocomputing.net wrote: > Thank you for the quick response. Please call me Sean ;) OK :-) > > Where did you see the reference to SINGLEPROCESS? This command is > > outdated, I'm surprised that it works at all. If you want to run a Box > > backup daemon on the console, using the -DV options is a better way to > > do it. > > I found that command in the list archives, back in March or April. I ran > the command with the -DV options as you suggested and here is what I > got: > > NOTICE: Starting daemon, version 0.11rc2, config: /etc/box/bbstored.conf > Housekeeping process started > Starting housekeeping > Finished housekeeping > > I gave it about 10 minutes to change, but it did not after "Finished > housekeeping". It will not do anything until the next housekeeping time (TimeBetweenHousekeeping in bbstored.conf) or it's contacted by the client (bbackupd). > > Did you configure the client's UpdateStoreInterval to 3 seconds? If so, > > then this is expected behaviour. If not, please post your configuration > > file. > > I had not yet delved that deeply into the configs, so I guess shame on me. > I'll look at it and make any necessary changes and get back to you on the > change in behavior. After a quick google, isn't that setting for clients? Yes. > Does the client get installed on the server automatically? I was under > the impression that it did not, but I may have inadvertently installed > the client at 1 am the other day. It depends how you install it. If you installed from source, you get two parcels, client and server. This log message: > Apr 25 14:49:18 debian Box Backup (bbackupd)[6109]: NOTICE: Beginning > scan of local files indicates that the client (bbackupd) is running. > > Also, it appears that your bbackupd is not configured to back up any > > files, is that correct? > > Yes, that is correct. For the last few days, as I've had time, I've been > concentrating on just getting the server itself running correctly. I'm sorry that it took so long, please let us know where and how we can improve the process or the documentation. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Fri Apr 25 23:41:01 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Fri, 25 Apr 2008 15:41:01 -0700 (PDT) Subject: [Box Backup] constant scan of local files on server In-Reply-To: References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> <5137.64.161.120.82.1209161755.squirrel@www.vancocomputing.net> Message-ID: <6568.64.161.120.82.1209163261.squirrel@www.vancocomputing.net> > It depends how you install it. If you installed from source, you get two parcels, client and server. This log message: > >> Apr 25 14:49:18 debian Box Backup (bbackupd)[6109]: NOTICE: Beginning scan of local files > > indicates that the client (bbackupd) is running. Now I must admit to be somewhat confused, and I apologize for having to ask some basic questions. I have not yet run the configuration script for the client, so I'm not sure where to go for the client options. Would this be the "../../bin/bbstored/bbstored testfiles/bbstored.conf" (shown in ps)? If so, that's a bit odd, as that path doesn't exist on my system. It also seems I have the backup client running twice, and I'm not sure how that happened. How can I separate the two, or is it just a matter or executing the right binaries and options? I'm not sure that I should have the client running on my BoxBackup server. > I'm sorry that it took so long, please let us know where and how we can improve the process or the documentation. This did not take me long at all, other than an hour here and there wasted from working way past my bedtime. I've been quite busy working on my clients and can only find time to work on this scattered throughout the week. The only part that was a bit unclear was how to manage the certs, but a howto mentioned in a past e-mail helped me be sure that I ended up doing it correctly (although not quite as suggested... I created and signed the certs on the same server). Just so you know when I'm finished I will write up the requirements for installing this on Debian for you, and others on the net, to use. There were quite a few installation pre-install requirements for compiling from source on Debian that weren't listed on your site. -- Sean Van Couwenberghe Vanco Computing www.vancocomputing.net (707) 578-5802 From boxbackup at fluffy.co.uk Fri Apr 25 23:52:25 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Fri, 25 Apr 2008 23:52:25 +0100 (BST) Subject: [Box Backup] constant scan of local files on server In-Reply-To: <6568.64.161.120.82.1209163261.squirrel@www.vancocomputing.net> References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> <5137.64.161.120.82.1209161755.squirrel@www.vancocomputing.net> <6568.64.161.120.82.1209163261.squirrel@www.vancocomputing.net> Message-ID: Hi Sean, On Fri, 25 Apr 2008, admin at vancocomputing.net wrote: > > It depends how you install it. If you installed from source, you get two > parcels, client and server. This log message: > > > >> Apr 25 14:49:18 debian Box Backup (bbackupd)[6109]: NOTICE: Beginning > scan of local files > > > > indicates that the client (bbackupd) is running. > > Now I must admit to be somewhat confused, and I apologize for having to > ask some basic questions. I have not yet run the configuration script for > the client, so I'm not sure where to go for the client options. Would this > be the "../../bin/bbstored/bbstored testfiles/bbstored.conf" (shown in > ps)? bbstored is the server, so no. But if you have a bbackupd running, the config file could well be on the command line of it. If not, the default is probably /etc/box/bbackupd.conf. > If so, that's a bit odd, as that path doesn't exist on my system. It's probably relative to one of the test directories. This probably just means that you ran the tests, and for some reason bbstored wasn't killed after the test finished. > It also seems I have the backup client running twice, and I'm not sure > how that happened. Yes, that is strange. Could you post the output from ps? > How can I separate the two, or is it just a matter or executing the > right binaries and options? I'm not sure that I should have the client > running on my BoxBackup server. It is, as you say, just a matter of executing the right binaries. If you installed the client parcel, you could just remove the files listed in parcels/scripts/install-backup-client: /usr/local/sbin/bbackupd /usr/local/sbin/bbackupquery /usr/local/sbin/bbackupctl /usr/local/sbin/bbackupd-config /usr/local/sbin/LICENSE.txt > The only part that was a bit unclear was how to manage the certs, Yes, I'm sorry that is a bit complicated. PKI is always quite a pain. > but a howto mentioned in a past e-mail helped me be sure that I ended up > doing it correctly (although not quite as suggested... I created and > signed the certs on the same server). Which HOWTO did you use? We should probably rewrite the installation instructions. > Just so you know when I'm finished I will write up the requirements for > installing this on Debian for you, and others on the net, to use. There > were quite a few installation pre-install requirements for compiling > from source on Debian that weren't listed on your site. Please do, thanks in advance for your help. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Fri Apr 25 23:56:53 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Fri, 25 Apr 2008 15:56:53 -0700 (PDT) Subject: [Box Backup] constant scan of local files on server In-Reply-To: References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> <5137.64.161.120.82.1209161755.squirrel@www.vancocomputing.net> <6568.64.161.120.82.1209163261.squirrel@www.vancocomputing.net> Message-ID: <6799.64.161.120.82.1209164213.squirrel@www.vancocomputing.net> First, I wanted to thank you for your assistance. I'd like to say I rarely need help, but I can't ;) >> It also seems I have the backup client running twice, and I'm not sure >> how that happened. > > Yes, that is strange. Could you post the output from ps? As you mentioned earlier, running a test may not have properly killed it (I now realize I ran the test for the client, not the server), but here's the ps output. I won't kill anything until I hear back from you. backup:/etc/init.d# ps aux | grep bb root 5996 0.0 0.3 6764 1968 ? S 12:33 0:00 ../../bin/bbstored/bbstored testfiles/bbstored.conf root 5997 0.0 0.3 6720 1760 ? S 12:33 0:09 ../../bin/bbstored/bbstored testfiles/bbstored.conf root 6109 0.1 0.6 6800 3264 ? S 12:34 0:18 ../../bin/bbackupd/bbackupd testfiles/bbackupd.conf root 2628 0.0 0.1 2848 696 pts/0 S+ 15:56 0:00 grep bb > >> How can I separate the two, or is it just a matter or executing the >> right binaries and options? I'm not sure that I should have the client >> running on my BoxBackup server. > > It is, as you say, just a matter of executing the right binaries. If you > installed the client parcel, you could just remove the files listed in > parcels/scripts/install-backup-client: > > /usr/local/sbin/bbackupd > /usr/local/sbin/bbackupquery > /usr/local/sbin/bbackupctl > /usr/local/sbin/bbackupd-config > /usr/local/sbin/LICENSE.txt > Thanks! > > Which HOWTO did you use? We should probably rewrite the installation > instructions. In my opinion, it's a little awkwardly written/arranged and I would have done it quite differently, but it did help: http://xkr47.outerspace.dyndns.org/howtos/boxbackup/ I'm setting up the init.d entries for bbstored now. -- Sean Van Couwenberghe Vanco Computing www.vancocomputing.net (707) 578-5802 From boxbackup at fluffy.co.uk Sat Apr 26 00:04:26 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sat, 26 Apr 2008 00:04:26 +0100 (BST) Subject: [Box Backup] constant scan of local files on server In-Reply-To: <6799.64.161.120.82.1209164213.squirrel@www.vancocomputing.net> References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> <5137.64.161.120.82.1209161755.squirrel@www.vancocomputing.net> <6568.64.161.120.82.1209163261.squirrel@www.vancocomputing.net> <6799.64.161.120.82.1209164213.squirrel@www.vancocomputing.net> Message-ID: Hi Sean, On Fri, 25 Apr 2008, admin at vancocomputing.net wrote: > >> It also seems I have the backup client running twice, and I'm not > >> sure how that happened. > > > > Yes, that is strange. Could you post the output from ps? > > As you mentioned earlier, running a test may not have properly killed it > (I now realize I ran the test for the client, not the server), but > here's the ps output. I won't kill anything until I hear back from you. > > backup:/etc/init.d# ps aux | grep bb > root 5996 0.0 0.3 6764 1968 ? S 12:33 0:00 > ../../bin/bbstored/bbstored testfiles/bbstored.conf > root 5997 0.0 0.3 6720 1760 ? S 12:33 0:09 > ../../bin/bbstored/bbstored testfiles/bbstored.conf > root 6109 0.1 0.6 6800 3264 ? S 12:34 0:18 > ../../bin/bbackupd/bbackupd testfiles/bbackupd.conf > root 2628 0.0 0.1 2848 696 pts/0 S+ 15:56 0:00 grep bb You have two server (bbstored) processes running, and just one client (bbackupd), since 12:34 today. bbstored normally starts two processes (listener and housekeeping) so this means that it was started just once, and split in two as expected. You can safely kill all of these. They are only used for unit tests. > > Which HOWTO did you use? We should probably rewrite the installation > > instructions. > > In my opinion, it's a little awkwardly written/arranged and I would have > done it quite differently, but it did help: > > http://xkr47.outerspace.dyndns.org/howtos/boxbackup/ Could you tell me what you took from here that you found particularly useful, or that wasn't covered by the standard documentation (and therefore probably should be)? > I'm setting up the init.d entries for bbstored now. You may find the examples in the contrib/ directory useful for this. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sat Apr 26 00:13:05 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Fri, 25 Apr 2008 16:13:05 -0700 (PDT) Subject: [Box Backup] constant scan of local files on server In-Reply-To: References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> <5137.64.161.120.82.1209161755.squirrel@www.vancocomputing.net> <6568.64.161.120.82.1209163261.squirrel@www.vancocomputing.net> <6799.64.161.120.82.1209164213.squirrel@www.vancocomputing.net> Message-ID: <7453.64.161.120.82.1209165185.squirrel@www.vancocomputing.net> Chris, > Could you tell me what you took from here that you found particularly > useful, or that wasn't covered by the standard documentation (and > therefore probably should be)? I have not read the full HowTo yet, but I'd say that the "Setting up the server" and "Setting up the client" were the most helpful. Those sections said how to create the certs, which your site does as well, but also states what certs go to the server and which to the client, and what to do with them. If these instructions are on your site, I have yet to find them (or at least to recognize them for what they are). Other than this, it'd say that your instructions are more comprehensive. > You have two server (bbstored) processes running, and just one client > (bbackupd), since 12:34 today. > > bbstored normally starts two processes (listener and housekeeping) so this > means that it was started just once, and split in two as expected. > > You can safely kill all of these. They are only used for unit tests. Thank you for the clarification. I'll be installing clients elsewhere to test tomorrow, hopefully. I'll be keeping an eye out for the .11 Windows client release. -- Sean Van Couwenberghe Vanco Computing www.vancocomputing.net (707) 578-5802 From boxbackup at fluffy.co.uk Sat Apr 26 00:27:12 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Fri, 25 Apr 2008 16:27:12 -0700 (PDT) Subject: SOLVED - Re: [Box Backup] constant scan of local files on server In-Reply-To: References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> <5137.64.161.120.82.1209161755.squirrel@www.vancocomputing.net> <6568.64.161.120.82.1209163261.squirrel@www.vancocomputing.net> Message-ID: <7691.64.161.120.82.1209166032.squirrel@www.vancocomputing.net> Chris, FYI: > It is, as you say, just a matter of executing the right binaries. If you > installed the client parcel, you could just remove the files listed in > parcels/scripts/install-backup-client: > > /usr/local/sbin/bbackupd > /usr/local/sbin/bbackupquery > /usr/local/sbin/bbackupctl > /usr/local/sbin/bbackupd-config > /usr/local/sbin/LICENSE.txt > It turns out I did not have the client installed (and the executables are in /usr/local/bin, not sbin, on Debian), so I think the logs were from the leftover client running from when I ran some tests (the same one you said it was OK to kill). The log entries no longer show up when I manually run the server now that I've killed the bbackupd process. Thanks for your help! I think I'm able to resume on my own now. -- Sean Van Couwenberghe Vanco Computing www.vancocomputing.net (707) 578-5802 From boxbackup at fluffy.co.uk Sat Apr 26 14:27:59 2008 From: boxbackup at fluffy.co.uk (James O'Gorman) Date: Sat, 26 Apr 2008 14:27:59 +0100 Subject: SOLVED - Re: [Box Backup] constant scan of local files on server In-Reply-To: <7691.64.161.120.82.1209166032.squirrel@www.vancocomputing.net> References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> <5137.64.161.120.82.1209161755.squirrel@www.vancocomputing.net> <6568.64.161.120.82.1209163261.squirrel@www.vancocomputing.net> <7691.64.161.120.82.1209166032.squirrel@www.vancocomputing.net> Message-ID: <48132DDF.10909@netinertia.co.uk> admin at vancocomputing.net wrote: > Chris, FYI: > >> It is, as you say, just a matter of executing the right binaries. If you >> installed the client parcel, you could just remove the files listed in >> parcels/scripts/install-backup-client: >> >> /usr/local/sbin/bbackupd >> /usr/local/sbin/bbackupquery >> /usr/local/sbin/bbackupctl >> /usr/local/sbin/bbackupd-config >> /usr/local/sbin/LICENSE.txt Hm, I should fix that. LICENSE.txt should definitely not go in sbin! > It turns out I did not have the client installed (and the executables are > in /usr/local/bin, not sbin, on Debian) When RC3 is released the binaries will move from bin/ to sbin/ - this is a change which was made after the RC2 release. You can blame me for that :-) James From boxbackup at fluffy.co.uk Sat Apr 26 18:02:10 2008 From: boxbackup at fluffy.co.uk (Magnus Homann) Date: Sat, 26 Apr 2008 19:02:10 +0200 Subject: [Box Backup] Restoring a corrupted archive? Message-ID: <48136012.4010608@homann.se> Hi! My name is Magnus, and my disk crashed. :-) At least, fsck -f -y is spending a lot of time working on it, not finished yet. It's a ext3 on Ubuntu. Luckily, i have run boxbackup on it. :-) Unfortunately, the bbstored is writing to the same disk. :-( Luckily, I rsync every night the files written by bbstored to a remote server. :-) Unfortunately, the crash was probably while rsyncing. :-( So, I have a local disk with a messed up file system, backed up to the same file system - yes, I know... - and then rsynced off-site. What can bbstored do with an archive that is partly rsynched from a newer version? (As I wrote, the fsck was finished, lots of files gone). Can I check if the bbstored archive is consistent on my local disk? Can I recreate the remote archive that may/may not be consistent? What non-.intrusive checks can I make, can I start bbstored in a 'read-only' mode? In short, I need help. :-) Thanks for any pointers, Magnus From boxbackup at fluffy.co.uk Sat Apr 26 18:14:44 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sat, 26 Apr 2008 18:14:44 +0100 (BST) Subject: [Box Backup] Restoring a corrupted archive? In-Reply-To: <48136012.4010608@homann.se> References: <48136012.4010608@homann.se> Message-ID: Hi Magnus, On Sat, 26 Apr 2008, Magnus Homann wrote: > My name is Magnus, and my disk crashed. :-) Sorry to hear that. How long have you been a disk crasher for? > Luckily, i have run boxbackup on it. :-) Unfortunately, the bbstored is > writing to the same disk. :-( Luckily, I rsync every night the files > written by bbstored to a remote server. :-) Unfortunately, the crash was > probably while rsyncing. :-( I think you're probably in luck. > So, I have a local disk with a messed up file system, backed up to the > same file system - yes, I know... - and then rsynced off-site. > > What can bbstored do with an archive that is partly rsynched from a > newer version? It should just work after you run "bbstoreaccounts check fix" to correct any problems with the account. I hope you have a copy of the encryption key :-) > Can I check if the bbstored archive is consistent on my local disk? Can I > recreate the remote archive that may/may not be consistent? Just run "bbstoreaccounts check fix" on it as well. > What non-.intrusive checks can I make, can I start bbstored in a > 'read-only' mode? Unfortunately there is no such mode at the moment. I can see how it might be useful, particularly to disable housekeeping. You will need read/write access to check and fix the account, but you can probably remount the filesystem read-only after that and run bbstored on it. If not, let me know how it fails and I'll try to come up with a quick hacky patch to fix it. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sat Apr 26 19:00:09 2008 From: boxbackup at fluffy.co.uk (Magnus Homann) Date: Sat, 26 Apr 2008 20:00:09 +0200 Subject: [Box Backup] Restoring a corrupted archive? In-Reply-To: References: <48136012.4010608@homann.se> Message-ID: <48136DA9.1000502@homann.se> Chris Wilson wrote: > On Sat, 26 Apr 2008, Magnus Homann wrote: >> So, I have a local disk with a messed up file system, backed up to the >> same file system - yes, I know... - and then rsynced off-site. >> >> What can bbstored do with an archive that is partly rsynched from a >> newer version? > > It should just work after you run "bbstoreaccounts check fix" to correct > any problems with the account. I hope you have a copy of the encryption > key :-) Yes, it's in the backup! Just kidding, they are on another remote server (and not on the same disk anyway). It appears that my local archive is corrupted, the directory called 'e2' is actually a big file right now, and according to 'du -sm' I'm 20 gig short on my local side. :-) I guess my best bet is to rsync the 20 gig back locally (no boxbackup SW on remote machine), and take it from there. Oh god, that will only take me 24 hour or so... Magnus From boxbackup at fluffy.co.uk Sun Apr 27 19:37:52 2008 From: boxbackup at fluffy.co.uk (Johann Glaser) Date: Sun, 27 Apr 2008 20:37:52 +0200 Subject: [Box Backup] Mount points invented Message-ID: <1209321472.4354.23.camel@hansi> Hi! Today I've used BoxBackup 0.11~rc2-3 (Debian Package) for the first time after it was updated from 0.10. There are now several of these error messages in /var/log/daemon.log: Ignored directory: /home/mon: is a mount point; create a new location if you want to back it up I didn't modify any config file or alter the disk layout for some time, so this error message is surely the effect of the new version of BoxBackup. I have several locations, where one is home { Path = /home # some ExcludeDirs } It is important to note that on my PC /home is a symlink to /small/home (because I changed the partition and directory layout quite some time ago and the old root partition is now mounted at /small (you guess the reason I changed the partitions :-) )). Contrary to the above mentioned error message there are _no_ mount points below /home, /small or /small/home. There are only 4 subdirectories, each belonging to one user at the machine. All these 4 subdirs have been refused by bbackupd. Could you please have a look whats the reason for this error message? How can I get my home directory backed up again as before? Thanks Hansi From boxbackup at fluffy.co.uk Sun Apr 27 20:27:44 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 27 Apr 2008 20:27:44 +0100 (BST) Subject: [Box Backup] Mount points invented In-Reply-To: <1209321472.4354.23.camel@hansi> References: <1209321472.4354.23.camel@hansi> Message-ID: Hi Johann, On Sun, 27 Apr 2008, Johann Glaser wrote: > Today I've used BoxBackup 0.11~rc2-3 (Debian Package) for the first time > after it was updated from 0.10. There are now several of these error > messages in /var/log/daemon.log: > Ignored directory: /home/mon: is a mount point; create a new location > if you want to back it up > > I didn't modify any config file or alter the disk layout for some time, > so this error message is surely the effect of the new version of > BoxBackup. Indeed it is. > I have several locations, where one is > home { > Path = /home > # some ExcludeDirs > } > It is important to note that on my PC /home is a symlink to /small/home > (because I changed the partition and directory layout quite some time > ago and the old root partition is now mounted at /small (you guess the > reason I changed the partitions :-) )). > > Contrary to the above mentioned error message there are _no_ mount > points below /home, /small or /small/home. There are only 4 > subdirectories, each belonging to one user at the machine. All these 4 > subdirs have been refused by bbackupd. > > Could you please have a look whats the reason for this error message? I know what the problem is. The filesystem of /small/* is different to the one containing the symlink /home. We need to use lstat rather than stat to identify the parent directory's filesystem when it's a symbolic link. > How can I get my home directory backed up again as before? You could change your backup locations to use /small/home instead of /home. I'll look into fixing the test for the next rc. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Sun Apr 27 20:39:32 2008 From: boxbackup at fluffy.co.uk (Johann Glaser) Date: Sun, 27 Apr 2008 21:39:32 +0200 Subject: [Box Backup] Mount points invented In-Reply-To: References: <1209321472.4354.23.camel@hansi> Message-ID: <1209325172.4354.31.camel@hansi> Hi Chris! Thanks for the fast response! > I know what the problem is. The filesystem of /small/* is different to the > one containing the symlink /home. We need to use lstat rather than stat to > identify the parent directory's filesystem when it's a symbolic link. I see. > > How can I get my home directory backed up again as before? > > You could change your backup locations to use /small/home instead of > /home. I'll look into fixing the test for the next rc. But that would require a new full-backup of my /home because the location was changed completely, isn't it? Bye Hansi From boxbackup at fluffy.co.uk Sun Apr 27 20:42:41 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Sun, 27 Apr 2008 20:42:41 +0100 (BST) Subject: [Box Backup] Mount points invented In-Reply-To: <1209325172.4354.31.camel@hansi> References: <1209321472.4354.23.camel@hansi> <1209325172.4354.31.camel@hansi> Message-ID: Hi Johann, On Sun, 27 Apr 2008, Johann Glaser wrote: > > > How can I get my home directory backed up again as before? > > > > You could change your backup locations to use /small/home instead of > > /home. I'll look into fixing the test for the next rc. > > But that would require a new full-backup of my /home because the > location was changed completely, isn't it? No, it should just resume backing up based on whatever files are still on the store (I hope they're not marked as deleted on the store). Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Mon Apr 28 03:52:50 2008 From: boxbackup at fluffy.co.uk (Sean Van Couwenberghe) Date: Sun, 27 Apr 2008 19:52:50 -0700 Subject: [Box Backup] Debian Etch tarball install HowTo complete In-Reply-To: References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> <6568.64.161.120.82.1209163261.squirrel@www.vancocomputing.net> Message-ID: <200804271952.51304.admin@vancocomputing.net> I've finished my server and client installs and configs, and my first backup is set to run at 1 am. As promised, here is the HowTo I wrote up. I may tweak it a little more to fill in some additional details (i.e. additional explanations of steps), but the meat is there: http://www.vancocomputing.com/blogs/linux/2008/04/25/boxbackup-installation-and-configuration/ Please let me know if you see any errors or missing info that should be included, but I did my best to make it accurate and easily understandable for any skill level. Thanks again for you help! -- Sean Van Couwenberghe Vanco Computing (707) 578-5802 (888) 908-2626 www.vancocomputing.net > > Just so you know when I'm finished I will write up the requirements for > > installing this on Debian for you, and others on the net, to use. There > > were quite a few installation pre-install requirements for compiling > > from source on Debian that weren't listed on your site. > > Please do, thanks in advance for your help. > > Cheers, Chris. From boxbackup at fluffy.co.uk Mon Apr 28 12:33:00 2008 From: boxbackup at fluffy.co.uk (Dave Bamford) Date: Mon, 28 Apr 2008 12:33:00 +0100 Subject: [Box Backup] Debian Etch tarball install HowTo complete In-Reply-To: <200804271952.51304.admin@vancocomputing.net> References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> <6568.64.161.120.82.1209163261.squirrel@www.vancocomputing.net> <200804271952.51304.admin@vancocomputing.net> Message-ID: <4815B5EC.8030807@logical-progress.com> Hi Sean This will be really useful I have set up a server on 3 Debian Etch systems now and have a small howto of my own but have never been able to get readline to work "enable-gnu-readline" was probably where I went wrong. I usually set up Raid 1 on the OS first using mdadm and format as jfs so Box does not use its own raid. Not sure about the pros and cons of this. I install the OS on one drive and use 2 drives in Raid 1 for the store, now using 750G SATA as best price per gig. I got stuck trying to install the os and he store with Raid, there are a few howtos on the web but not easy to follow. Thanks Dave Bamford Sean Van Couwenberghe wrote: > I've finished my server and client installs and configs, and my first backup > is set to run at 1 am. > > As promised, here is the HowTo I wrote up. I may tweak it a little more to > fill in some additional details (i.e. additional explanations of steps), but > the meat is there: > > http://www.vancocomputing.com/blogs/linux/2008/04/25/boxbackup-installation-and-configuration/ > > Please let me know if you see any errors or missing info that should be > included, but I did my best to make it accurate and easily understandable for > any skill level. > > Thanks again for you help! > > From boxbackup at fluffy.co.uk Mon Apr 28 15:54:41 2008 From: boxbackup at fluffy.co.uk (Sean Van Couwenberghe) Date: Mon, 28 Apr 2008 07:54:41 -0700 Subject: [Box Backup] Debian Etch tarball install HowTo complete In-Reply-To: <4815B5EC.8030807@logical-progress.com> References: <4490.64.161.120.82.1209160765.squirrel@www.vancocomputing.net> <200804271952.51304.admin@vancocomputing.net> <4815B5EC.8030807@logical-progress.com> Message-ID: <200804280754.41888.admin@vancocomputing.net> Dave, I'm glad this will be useful to you. To be honest, I had to ask a friend for help figuring out the --enable-gnu-readline. I see that I've excluded a few optional setttings and methods, so I'll be adding them over the next week or two, after I get my setup working. Cheers! -- Sean Van Couwenberghe Vanco Computing (707) 578-5802 (888) 908-2626 www.vancocomputing.net On Monday 28 April 2008 04:33, Dave Bamford wrote: > Hi Sean > > This will be really useful I have set up a server on 3 Debian Etch systems > now and have a small howto of my own but have never been able to get > readline to work "enable-gnu-readline" was probably where I went wrong. > > I usually set up Raid 1 on the OS first using mdadm and format as jfs so > Box does not use its own raid. Not sure about the pros and cons of this. > > I install the OS on one drive and use 2 drives in Raid 1 for the store, now > using 750G SATA as best price per gig. > > I got stuck trying to install the os and he store with Raid, there are a > few howtos on the web but not easy to follow. > > Thanks > > Dave Bamford > > Sean Van Couwenberghe wrote: > > I've finished my server and client installs and configs, and my first > > backup is set to run at 1 am. > > > > As promised, here is the HowTo I wrote up. I may tweak it a little more > > to fill in some additional details (i.e. additional explanations of > > steps), but the meat is there: > > > > http://www.vancocomputing.com/blogs/linux/2008/04/25/boxbackup-installati > >on-and-configuration/ > > > > Please let me know if you see any errors or missing info that should be > > included, but I did my best to make it accurate and easily understandable > > for any skill level. > > > > Thanks again for you help! > > _______________________________________________ > boxbackup mailing list > boxbackup at fluffy.co.uk > http://lists.warhead.org.uk/mailman/listinfo/boxbackup From boxbackup at fluffy.co.uk Mon Apr 28 17:06:38 2008 From: boxbackup at fluffy.co.uk (Sean Van Couwenberghe) Date: Mon, 28 Apr 2008 09:06:38 -0700 Subject: [Box Backup] client connection error Message-ID: <200804280906.38741.admin@vancocomputing.net> I'm getting some errors when the linux client kicks off (both client and server are the same version, 0.11 rc2): Apr 28 08:00:41 ns2 Box Backup (bbackupd)[5131]: NOTICE: Beginning scan of local files Apr 28 08:00:41 ns2 Box Backup (bbackupd)[5131]: WARNING: Exception thrown: ConnectionException(Conn_SocketConnectError) at SocketStream.cpp(223) Apr 28 08:00:41 ns2 Box Backup (bbackupd)[5131]: WARNING: Suppressing duplicate notification about backup-error Apr 28 08:00:51 ns2 Box Backup (bbackupd)[5131]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 I am able to connect to the backup server by running "telnet [backup_server] 2201", but when I run the backup itself it fails with the above messages. On the backup server logs, I get these lines: Apr 28 08:57:13 debian Box Backup (bbstored)[5050]: WARNING: Message from child process 5055: Incoming connection from [ip_address] port 44518 Apr 28 08:57:14 debian Box Backup (bbstored)[5055]: ERROR: SSL error during Accept: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number Apr 28 08:57:14 debian Box Backup (bbstored)[5055]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(245) Apr 28 08:57:14 debian Box Backup (bbstored)[5055]: ERROR: Error in child process, terminating connection: exception Connection TLSHandshakeFailed(7/30) Do you have any suggestions for me? Thank you! -- Sean Van Couwenberghe Vanco Computing (707) 578-5802 (888) 908-2626 www.vancocomputing.net From boxbackup at fluffy.co.uk Mon Apr 28 17:46:33 2008 From: boxbackup at fluffy.co.uk (Stuart D. Gathman) Date: Mon, 28 Apr 2008 12:46:33 -0400 (EDT) Subject: [Box Backup] Restoring a corrupted archive? In-Reply-To: <48136DA9.1000502@homann.se> Message-ID: On Sat, 26 Apr 2008, Magnus Homann wrote: > I guess my best bet is to rsync the 20 gig back locally (no boxbackup SW > on remote machine), and take it from there. Oh god, that will only take > me 24 hour or so... Can't you restore to USB disk and courier/FedEx it? -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From boxbackup at fluffy.co.uk Mon Apr 28 17:35:42 2008 From: boxbackup at fluffy.co.uk (Sean Van Couwenberghe) Date: Mon, 28 Apr 2008 09:35:42 -0700 Subject: [Box Backup] Re: client connection error In-Reply-To: <200804280906.38741.admin@vancocomputing.net> References: <200804280906.38741.admin@vancocomputing.net> Message-ID: <200804280935.43093.admin@vancocomputing.net> I'm sorry, what I said in the first e-mail was incorrect. I am getting those logs, but they are caused by telnetting to bbstored on its port. When I fire off "/usr/local/bin/bbackupctl -q sync", the system seems to try to connect to the server (at least from the client side), but the server logs no activity (which suggests to me that the client isn't initiating a connection request). I turned on the ExtendedLogging option on the server, but still no logged activity on the server side. Could I have missed something in the client config? This is the command the configuration script told me to run, so I'm pretty sure I got everything set up. Is port 2201 the correct port? Sean On Monday 28 April 2008 09:06, you wrote: > I'm getting some errors when the linux client kicks off (both client and > server are the same version, 0.11 rc2): > > Apr 28 08:00:41 ns2 Box Backup (bbackupd)[5131]: NOTICE: Beginning scan of > local files > Apr 28 08:00:41 ns2 Box Backup (bbackupd)[5131]: WARNING: Exception thrown: > ConnectionException(Conn_SocketConnectError) at SocketStream.cpp(223) > Apr 28 08:00:41 ns2 Box Backup (bbackupd)[5131]: WARNING: Suppressing > duplicate notification about backup-error > Apr 28 08:00:51 ns2 Box Backup (bbackupd)[5131]: NOTICE: File statistics: > total file size uploaded 0, bytes already on server 0, encoded size 0 > > I am able to connect to the backup server by running "telnet > [backup_server] 2201", but when I run the backup itself it fails with the > above messages. > > On the backup server logs, I get these lines: > > Apr 28 08:57:13 debian Box Backup (bbstored)[5050]: WARNING: Message from > child process 5055: Incoming connection from [ip_address] port 44518 > Apr 28 08:57:14 debian Box Backup (bbstored)[5055]: ERROR: SSL error during > Accept: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number > Apr 28 08:57:14 debian Box Backup (bbstored)[5055]: WARNING: Exception > thrown: ConnectionException(Conn_TLSHandshakeFailed) at > SocketStreamTLS.cpp(245) Apr 28 08:57:14 debian Box Backup > (bbstored)[5055]: ERROR: Error in child process, terminating connection: > exception Connection > TLSHandshakeFailed(7/30) > > Do you have any suggestions for me? > > Thank you! From boxbackup at fluffy.co.uk Mon Apr 28 22:08:15 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Mon, 28 Apr 2008 22:08:15 +0100 (BST) Subject: [Box Backup] Re: client connection error In-Reply-To: <200804280935.43093.admin@vancocomputing.net> References: <200804280906.38741.admin@vancocomputing.net> <200804280935.43093.admin@vancocomputing.net> Message-ID: Hi Sean, On Mon, 28 Apr 2008, Sean Van Couwenberghe wrote: > I'm sorry, what I said in the first e-mail was incorrect. I am getting those > logs, but they are caused by telnetting to bbstored on its port. > > When I fire off "/usr/local/bin/bbackupctl -q sync", the system seems to try > to connect to the server (at least from the client side), but the server logs > no activity (which suggests to me that the client isn't initiating a > connection request). I turned on the ExtendedLogging option on the server, > but still no logged activity on the server side. > > Could I have missed something in the client config? This is the command the > configuration script told me to run, so I'm pretty sure I got everything set > up. Is port 2201 the correct port? Port 2201 is correct. Please could you run bbackupd with the -DVT options and tell us what it prints out? Could it be that the StoreHostname in bbackupd.conf is incorrect, or that your firewall is blocking connections from the user that bbackupd runs as? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Mon Apr 28 23:28:49 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Mon, 28 Apr 2008 15:28:49 -0700 (PDT) Subject: [Box Backup] Re: client connection error In-Reply-To: References: <200804280906.38741.admin@vancocomputing.net> <200804280935.43093.admin@vancocomputing.net> Message-ID: <49719.64.161.120.82.1209421729.squirrel@www.vancocomputing.net> Chris, > Could it be that the StoreHostname in bbackupd.conf is incorrect, or that > your firewall is blocking connections from the user that bbackupd runs as? The top 10 lines of my bbackupd.conf file is listed below. Just for clarification, there is no software on the backup server (so there's nothing that can block by user), and the port forwarding is taking place "upstream" of the computer at a router. The Linksys router is in front of all computers but one (the one that I installed the client on). The client is open to the world and has a firewall installed, but allows outgoing traffic. I know there's no blockage from the either the firewall or the router because I can connect to port 2201 via other protocols (telnet and ssh). Having said that, I currently have the bbackupd StoreHostname set to the IP of the router, which is forwarding that port to the backup server (Yes I'm using NAT... this is just my temporary/testing environment. Eventually I'll bypass the router). If the NAT might be causing trouble, let me know and I'll try to get it onto a public IP. That could be difficult with my setup. > Please could you run bbackupd with the -DVT options and tell us what it > prints out? The "/usr/local/bin/bbackupd -DVT" command returned this (please note that I'm configured for snapshot mode): 15:07:18 NOTICE: Starting daemon, version 0.11rc2, config: /etc/box/bbackupd.conf 15:07:18 Wait on command socket, delay = 1024000000 bbackupd.conf: ns2:/etc/box# cat bbackupd.conf StoreHostname = [router IP] AccountNumber = 0x1 KeysFile = /etc/box/bbackupd/1-FileEncKeys.raw CertificateFile = /etc/box/bbackupd/1-cert.pem PrivateKeyFile = /etc/box/bbackupd/1-key.pem TrustedCAsFile = /etc/box/bbackupd/serverCA.pem DataDirectory = /var/bbackupd (what does this control? I can't find documentation on this line) Sean From boxbackup at fluffy.co.uk Mon Apr 28 23:41:36 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Mon, 28 Apr 2008 23:41:36 +0100 (BST) Subject: [Box Backup] Re: client connection error In-Reply-To: <49719.64.161.120.82.1209421729.squirrel@www.vancocomputing.net> References: <200804280906.38741.admin@vancocomputing.net> <200804280935.43093.admin@vancocomputing.net> <49719.64.161.120.82.1209421729.squirrel@www.vancocomputing.net> Message-ID: Hi Sean, On Mon, 28 Apr 2008, admin at vancocomputing.net wrote: > > Could it be that the StoreHostname in bbackupd.conf is incorrect, or > > that your firewall is blocking connections from the user that bbackupd > > runs as? > > The top 10 lines of my bbackupd.conf file is listed below. > > Just for clarification, there is no software on the backup server (so > there's nothing that can block by user), and the port forwarding is > taking place "upstream" of the computer at a router. The Linksys router > is in front of all computers but one (the one that I installed the > client on). The client is open to the world and has a firewall > installed, but allows outgoing traffic. > > I know there's no blockage from the either the firewall or the router > because I can connect to port 2201 via other protocols (telnet and ssh). > Having said that, I currently have the bbackupd StoreHostname set to the > IP of the router, which is forwarding that port to the backup server > (Yes I'm using NAT... this is just my temporary/testing environment. > Eventually I'll bypass the router). If the NAT might be causing trouble, > let me know and I'll try to get it onto a public IP. That could be > difficult with my setup. Just to check that I understood your setup correctly, the machines are arranged something like this? [bbackupd client] -- [internet] -- [NAT router] -- [bbstored server] with the NAT router forwarding port 2201 to the bbstored server? That should work. And you can telnet, on the client, to port 2201 on the NAT router, and you get connected to the bbstored server? (which tries to speak SSL to you) > > Please could you run bbackupd with the -DVT options and tell us what it > > prints out? > > The "/usr/local/bin/bbackupd -DVT" command returned this (please note that > I'm configured for snapshot mode): > > 15:07:18 NOTICE: Starting daemon, version 0.11rc2, config: > /etc/box/bbackupd.conf > 15:07:18 Wait on command socket, delay = 1024000000 Please could you try requesting a sync with "bbackupctl sync" while running bbackupd with -DVT and let me know what bbackupd says? > DataDirectory = /var/bbackupd (what does this control? I can't find > documentation on this line) bbackupd stores temporary state files in there: the last_sync_start and last_sync_finish marker files (which are not necessary for Box Backup operation, but may be useful to admins) and the inode map, which is used to track renamed files. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Tue Apr 29 00:40:26 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Mon, 28 Apr 2008 16:40:26 -0700 (PDT) Subject: [Box Backup] Re: client connection error In-Reply-To: References: <200804280906.38741.admin@vancocomputing.net> <200804280935.43093.admin@vancocomputing.net> <49719.64.161.120.82.1209421729.squirrel@www.vancocomputing.net> Message-ID: <53442.64.161.120.82.1209426026.squirrel@www.vancocomputing.net> Chris, > Just to check that I understood your setup correctly, the machines are > arranged something like this? > > [bbackupd client] -- [internet] -- [NAT router] -- [bbstored server] > > with the NAT router forwarding port 2201 to the bbstored server? That > should work. Yes, you are correct in your understanding. I was afraid that I was as clear as mud. Just FYI here's a more accurate logical rendition (physical layout is a mess thanks to vmware ;) : internet -- [DSL modem] -- [switch] -- [NAT Router] -- [bbstored server] \ --[bbackupd client] > Please could you try requesting a sync with "bbackupctl sync" while > running bbackupd with -DVT and let me know what bbackupd says? Wow, I got a eyefull of files processed on the client (THANK YOU!! - after finishing this e-mail I've seen at least a half gig transfered), it's taking some time... I got to see activity in real time as I ran it with the -DVT in one SSH window and the sync in another. Basically every file was listed as "not on server". Here's what's showing up in /var/log/box on the bbstored server (repeats many times, but not all say "101 times"): Apr 28 16:31:15 debian last message repeated 101 times Apr 28 16:31:15 debian Box Backup (bbstored)[5233]: Receiving stream, size 49 Apr 28 16:31:15 debian Box Backup (bbstored)[5233]: Receiving stream, size uncertain I presume then that this is working? On the bbstored server in /backup_files/backup/00000001/ I'm seeing quite a few .rfw files. I presume those are the compressed and encrypted files from my client? I thought the ssh connection to that system was a bit slow to respond, which tells me alot of data is being transfered. Should I modify the init.d script to allow -DVT, or something else? Oh, BTW, "/etc/init.d/bbstored restart" does not start the daemon back up, even though the text says "restarted". Is there any way to run a test to see how much data will be transfered after compression/encryption? I only have 100GB allocated to bbstored ATM, and I don't want to accidentally crash anything. It would also be nice to have an option (I know, you're still working on it, and this is just a low-priority "convenience" request) to see the items transfered in KB, MB or GB instead of bytes. BTW, thank you for the clarification on the /var/bbackupd. That was pretty much was I had guessed shortly after sending the last e-mail. Sean From boxbackup at fluffy.co.uk Tue Apr 29 00:53:14 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 29 Apr 2008 00:53:14 +0100 (BST) Subject: [Box Backup] Re: client connection error In-Reply-To: <53442.64.161.120.82.1209426026.squirrel@www.vancocomputing.net> References: <200804280906.38741.admin@vancocomputing.net> <200804280935.43093.admin@vancocomputing.net> <49719.64.161.120.82.1209421729.squirrel@www.vancocomputing.net> <53442.64.161.120.82.1209426026.squirrel@www.vancocomputing.net> Message-ID: Hi Sean, On Mon, 28 Apr 2008, admin at vancocomputing.net wrote: > > Please could you try requesting a sync with "bbackupctl sync" while > > running bbackupd with -DVT and let me know what bbackupd says? > > Wow, I got a eyefull of files processed on the client (THANK YOU!! - > after finishing this e-mail I've seen at least a half gig transfered), > it's taking some time... I got to see activity in real time as I ran it > with the -DVT in one SSH window and the sync in another. Basically every > file was listed as "not on server". That's to be expected if nothing was backed up so far. > I presume then that this is working? On the bbstored server in > /backup_files/backup/00000001/ I'm seeing quite a few .rfw files. I > presume those are the compressed and encrypted files from my client? Yes. > I thought the ssh connection to that system was a bit slow to respond, > which tells me alot of data is being transfered. Should I modify the > init.d script to allow -DVT, or something else? You should not need to run bbackupd with -DVT normally, and if you do that in the init script it will probably never terminate. Could you try running bbackupd with -kVT instead? It should go into the background but continue writing to your console. This may mess up the console a bit, but it will help to figure out why it's not working when you just run bbackupd normally in the background. > Oh, BTW, "/etc/init.d/bbstored restart" does not start the daemon back > up, even though the text says "restarted". Can you have a look at the initscript to see what's wrong? I'm afraid I didn't write it and don't have a debian box offhand to test on. Logs in /var/log/messages (or /var/log/box) may help to understand the problem. > Is there any way to run a test to see how much data will be transfered > after compression/encryption? I'm afraid not, but you could guess a 2:1 compression ratio unless you have a lot of uncompressable binary files, in which case it's likely to be more like 1:1. > I only have 100GB allocated to bbstored ATM, and I don't want to > accidentally crash anything. Nothing will crash, the client will stop uploading when it hits the limit. > It would also be nice to have an option (I know, you're still working on > it, and this is just a low-priority "convenience" request) to see the > items transfered in KB, MB or GB instead of bytes. I'll think about it, but probably the best option is to run a log parser like bbreporter.py and have that process the numbers into something more human-readable for you. Which specific log messages are you referring to? > BTW, thank you for the clarification on the /var/bbackupd. That was > pretty much was I had guessed shortly after sending the last e-mail. No problem, I've written it up on the wiki now as well. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Tue Apr 29 01:29:47 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Mon, 28 Apr 2008 17:29:47 -0700 (PDT) Subject: [Box Backup] Re: client connection error In-Reply-To: References: <200804280906.38741.admin@vancocomputing.net> <200804280935.43093.admin@vancocomputing.net> <49719.64.161.120.82.1209421729.squirrel@www.vancocomputing.net> <53442.64.161.120.82.1209426026.squirrel@www.vancocomputing.net> Message-ID: <55434.64.161.120.82.1209428987.squirrel@www.vancocomputing.net> Chris, > You should not need to run bbackupd with -DVT normally, and if you do that > in the init script it will probably never terminate. > > Could you try running bbackupd with -kVT instead? It should go into the > background but continue writing to your console. This may mess up the > console a bit, but it will help to figure out why it's not working when > you just run bbackupd normally in the background. I ran the daemon using -kVT as you requested. I saw no difference from running that or with -DVT, at least not on the screen output or the logs. Does this help any? If not, I'll do whatever you need. If needed to be done long-term, writing to the console isn't a huge problem, but could I just send the output to /dev/null? >> Oh, BTW, "/etc/init.d/bbstored restart" does not start the daemon back >> up, even though the text says "restarted". > > Can you have a look at the initscript to see what's wrong? I'm afraid I > didn't write it and don't have a debian box offhand to test on. Logs in > /var/log/messages (or /var/log/box) may help to understand the problem. I'm not very familiar with init.d scripts, although the last few minutes of reading has been very informative. I think I've narrowed it down to the following: bbstored - stops and starts fine, but will not come up on a restart. Running restart right afterwards brings the daemon back up, so it might just need a pause inserted between the stop/start commands. bbackupd - won't stop either on a stop or restart, and has to be killed. Afterwards it will come up on a start or restart command. If you need access to a Debian server, I'll offer mine up. It's a VM and I don't have any sensitive data on it. I honestly have no idea why they behave differently, and I've compared the two init.d scripts. They are the same but for the bin file they execute. >> Is there any way to run a test to see how much data will be transfered >> after compression/encryption? > > I'm afraid not, but you could guess a 2:1 compression ratio unless you > have a lot of uncompressable binary files, in which case it's likely to be > more like 1:1. No problem, it's just good to know how it handles encryption. Are you able to tell me what it uses to compress the data? >> It would also be nice to have an option (I know, you're still working on >> it, and this is just a low-priority "convenience" request) to see the >> items transfered in KB, MB or GB instead of bytes. > > I'll think about it, but probably the best option is to run a log parser > like bbreporter.py and have that process the numbers into something more > human-readable for you. > > Which specific log messages are you referring to? *Blush* well... I was looking at the console output, and it turns out the logs don't list the data size. My apologies. I suppose that would be interesting, but largely unnecessary. I also noticed that, oddly enough, I've only gotten one e-mail notice for success/fail of the job, despite firing off a sync multiple times. Does it only e-mail when there's a problem? If so, I should have gotten several, as I only now got a successful backup. I didn't change that part of the config. I'm sorry for asking so many questions. I very much appreciate your help. After I work all this out I'll be modifying my HowTo that I linked to you yesterday. Sean From boxbackup at fluffy.co.uk Tue Apr 29 01:40:14 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 29 Apr 2008 01:40:14 +0100 (BST) Subject: [Box Backup] Re: client connection error In-Reply-To: <55434.64.161.120.82.1209428987.squirrel@www.vancocomputing.net> References: <200804280906.38741.admin@vancocomputing.net> <200804280935.43093.admin@vancocomputing.net> <49719.64.161.120.82.1209421729.squirrel@www.vancocomputing.net> <53442.64.161.120.82.1209426026.squirrel@www.vancocomputing.net> <55434.64.161.120.82.1209428987.squirrel@www.vancocomputing.net> Message-ID: Hi Sean, On Mon, 28 Apr 2008, admin at vancocomputing.net wrote: > I ran the daemon using -kVT as you requested. I saw no difference from > running that or with -DVT, at least not on the screen output or the > logs. Does this help any? Does it appear to be working, i.e. backing up files? What does it do when you run it with no options and then request a sync? What do you get in the client logs? > If needed to be done long-term, writing to the console isn't a huge > problem, but could I just send the output to /dev/null? I'd rather fix it so that it works without any options as it should. > I'm not very familiar with init.d scripts, although the last few minutes > of reading has been very informative. I think I've narrowed it down to the > following: > > bbstored - stops and starts fine, but will not come up on a restart. > Running restart right afterwards brings the daemon back up, so it might > just need a pause inserted between the stop/start commands. > > bbackupd - won't stop either on a stop or restart, and has to be killed. > Afterwards it will come up on a start or restart command. > > If you need access to a Debian server, I'll offer mine up. It's a VM and I > don't have any sensitive data on it. That could be very useful, thanks. > No problem, it's just good to know how it handles encryption. Are you > able to tell me what it uses to compress the data? Sorry, I'm afraid I don't know offhand. docs/backup/backup_encryption.txt mentions PKCS padding. I'd guess it's done by an OpenSSL block cipher. > I also noticed that, oddly enough, I've only gotten one e-mail notice > for success/fail of the job, despite firing off a sync multiple times. > Does it only e-mail when there's a problem? Yes, and only at the end of the backup run. Does that explain it? > If so, I should have gotten several, as I only now got a successful > backup. I didn't change that part of the config. You should have log messages about "Notifying admin" for each email that ought to have been sent. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Tue Apr 29 02:02:17 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Mon, 28 Apr 2008 18:02:17 -0700 (PDT) Subject: [Box Backup] Re: client connection error In-Reply-To: References: <200804280906.38741.admin@vancocomputing.net> <200804280935.43093.admin@vancocomputing.net> <49719.64.161.120.82.1209421729.squirrel@www.vancocomputing.net> <53442.64.161.120.82.1209426026.squirrel@www.vancocomputing.net> <55434.64.161.120.82.1209428987.squirrel@www.vancocomputing.net> Message-ID: <55526.64.161.120.82.1209430937.squirrel@www.vancocomputing.net> Chris, I'll do my best to answer your questions. > Hi Sean, > > On Mon, 28 Apr 2008, admin at vancocomputing.net wrote: > > Does it appear to be working, i.e. backing up files? What does it do when > you run it with no options and then request a sync? What do you get in the > client logs? Yes, it seemed to be properly functioning. It processed the files, but only uploaded a small amount due to the first backup being on a few minutes before. Below are the client logs from the last successful backup, which was with the -kVT options. Apr 28 17:02:20 ns2 Box Backup (bbackupd)[11167]: Incoming connection from local (UNIX socket) Apr 28 17:02:20 ns2 Box Backup (bbackupd)[11167]: Connection from command socket Apr 28 17:02:20 ns2 Box Backup (bbackupd)[11167]: NOTICE: Beginning scan o f local files Apr 28 17:02:20 ns2 Box Backup (bbackupd)[11167]: Opening connection to se rver '64.142.81.150'... Apr 28 17:02:20 ns2 Box Backup (bbackupd)[11167]: Connection made, login s uccessful Apr 28 17:02:20 ns2 Box Backup (bbackupd)[11167]: Not deleting unused entr ies - none in list Apr 28 17:02:20 ns2 Box Backup (bbackupd)[11167]: NOTICE: About to notify administrator about event backup-start, running script '/etc/box/bbackupd/ NotifySysadmin.sh backup-start' Apr 28 17:04:16 ns2 Box Backup (bbackupd)[11167]: NOTICE: Finished scan of local files Apr 28 17:04:16 ns2 Box Backup (bbackupd)[11167]: NOTICE: About to notify administrator about event backup-finish, running script '/etc/box/bbackupd /NotifySysadmin.sh backup-finish' Apr 28 17:04:17 ns2 Box Backup (bbackupd)[11167]: NOTICE: File statistics: total file size uploaded 846, bytes already on server 0, encoded size 1144 Here are the logs from the latest attempt at running a sync with the daemon running with no options and the daemon started by "/etc/init.d/bbackupd start" (I added some data to /home to trigger a new data upload): Apr 28 17:56:13 ns2 Box Backup (bbackupd)[11775]: NOTICE: Beginning scan of local files Apr 28 17:56:13 ns2 Box Backup (bbackupd)[11775]: NOTICE: About to notify administrator about event backup-start, running script '/etc/box/bbackupd/NotifySysadmin.sh backup-start' Apr 28 17:58:49 ns2 Box Backup (bbackupd)[11775]: NOTICE: Finished scan of local files Apr 28 17:58:49 ns2 Box Backup (bbackupd)[11775]: NOTICE: About to notify administrator about event backup-finish, running script '/etc/box/bbackupd/NotifySysadmin.sh backup-finish' Apr 28 17:58:49 ns2 Box Backup (bbackupd)[11775]: NOTICE: File statistics: total file size uploaded 56001420, bytes already on server 0, encoded size 50316420 It looks like it uploaded just fine. I really don't understand why the backup didn't start the first several times until you had me run the daemon with the -DVT options. It seems to be working fine (I've confirmed new data on the bbstored server). >> If you need access to a Debian server, I'll offer mine up. It's a VM and >> I >> don't have any sensitive data on it. > > That could be very useful, thanks. No problem. After all, whatever helps you helps me and everyone else. >> If so, I should have gotten several, as I only now got a successful backup. I didn't change that part of the config. > > You should have log messages about "Notifying admin" for each email that ought to have been sent. Yes, I've gotten several lines with "Notifying admin" for start, finish and failures, but I've only ever received one e-mail. That time was this morning when I first attempted a backup. Thank you! Sean From boxbackup at fluffy.co.uk Tue Apr 29 02:10:25 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 29 Apr 2008 02:10:25 +0100 (BST) Subject: [Box Backup] Re: client connection error In-Reply-To: <55526.64.161.120.82.1209430937.squirrel@www.vancocomputing.net> References: <200804280906.38741.admin@vancocomputing.net> <200804280935.43093.admin@vancocomputing.net> <49719.64.161.120.82.1209421729.squirrel@www.vancocomputing.net> <53442.64.161.120.82.1209426026.squirrel@www.vancocomputing.net> <55434.64.161.120.82.1209428987.squirrel@www.vancocomputing.net> <55526.64.161.120.82.1209430937.squirrel@www.vancocomputing.net> Message-ID: Hi Sean, On Mon, 28 Apr 2008, admin at vancocomputing.net wrote: > It looks like it uploaded just fine. I really don't understand why the > backup didn't start the first several times until you had me run the > daemon with the -DVT options. It seems to be working fine (I've confirmed > new data on the bbstored server). I don't know either. It's a shame that we can't reproduce it. Please keep an eye on your backups and run a compare command regularly, and let me know if it appears to stop backing up for any reason. Do you have logs from the early runs which should have backed things up but didn't? > Yes, I've gotten several lines with "Notifying admin" for start, finish > and failures, but I've only ever received one e-mail. That time was this > morning when I first attempted a backup. You don't get emails for backup-start and backup-finish by default, only for errors. If you run the command shown in those log lines (for a failure), do you get an email as expected? If not, what output do you get and what's in your mail server logs? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Tue Apr 29 02:27:47 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Mon, 28 Apr 2008 18:27:47 -0700 (PDT) Subject: [Box Backup] Re: client connection error In-Reply-To: References: <200804280906.38741.admin@vancocomputing.net> <200804280935.43093.admin@vancocomputing.net> <49719.64.161.120.82.1209421729.squirrel@www.vancocomputing.net> <53442.64.161.120.82.1209426026.squirrel@www.vancocomputing.net> <55434.64.161.120.82.1209428987.squirrel@www.vancocomputing.net> <55526.64.161.120.82.1209430937.squirrel@www.vancocomputing.net> Message-ID: <55643.64.161.120.82.1209432467.squirrel@www.vancocomputing.net> Hey Chris, > I don't know either. It's a shame that we can't reproduce it. Please keep > an eye on your backups and run a compare command regularly, and let me > know if it appears to stop backing up for any reason. > > Do you have logs from the early runs which should have backed things up > but didn't? > I sure do have those client logs. They will be at the end of the e-mail and a quick note. > You don't get emails for backup-start and backup-finish by default, only > for errors. > > If you run the command shown in those log lines (for a failure), do you > get an email as expected? If not, what output do you get and what's in > your mail server logs? If I run the command manually, I do get an error e-mail. I see in the local server's logs (backup mail server, normally dormant) that it sent me two error messages regarding the backup errors. One from this morning and one from a few minutes ago when I ran the script manually. Thanks! Sean --------------------------- Apr 28 07:26:27 ns2 Box Backup (bbackupd)[5131]: NOTICE: Starting daemon, version 0.11rc2, config: /etc/box/bbackupd.conf Apr 28 07:32:34 ns2 Box Backup (bbackupd)[5131]: NOTICE: Beginning scan of local files Apr 28 07:32:34 ns2 Box Backup (bbackupd)[5131]: WARNING: Exception thrown: ConnectionException(Conn_SocketConnectError) at S ocketStream.cpp(223) Apr 28 07:32:34 ns2 Box Backup (bbackupd)[5131]: NOTICE: About to notify administrator about event backup-error, running scri pt '/etc/box/bbackupd/NotifySysadmin.sh backup-error' Apr 28 07:32:45 ns2 Box Backup (bbackupd)[5131]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Apr 28 07:51:28 ns2 Box Backup (bbackupd)[5131]: NOTICE: Beginning scan of local files Apr 28 07:51:28 ns2 Box Backup (bbackupd)[5131]: WARNING: Exception thrown: ConnectionException(Conn_SocketConnectError) at S ocketStream.cpp(223) Apr 28 07:51:28 ns2 Box Backup (bbackupd)[5131]: WARNING: Suppressing duplicate notification about backup-error Apr 28 07:51:38 ns2 Box Backup (bbackupd)[5131]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 [IT DID NOT AUTOMATICALLY RUN THE NOTIFYSYSADMIN.SH SCRIPT FROM THIS POINT ON] - sorry for the caps, wanted to make a delineation from logs and the note. Repeated about 10 times, for 10 or so failed backup attempts. No associated logs on the bbstored server, as if the client didn't even make a connection attempt. From boxbackup at fluffy.co.uk Tue Apr 29 14:11:20 2008 From: boxbackup at fluffy.co.uk (Brent Saner) Date: Tue, 29 Apr 2008 09:11:20 -0400 Subject: [Box Backup] spaces (and other ugliness) in Exclude paths Message-ID: <48171E78.5060907@acetechgroup.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 in one of my exclude paths, i have a space. this is a GNU/Linux client, but it's backing up a Windows share (in other words, stored locally and shared via cifs to Windows clients. the directory itself is on an ext3 fs, obviously). normal BASH-magic does not work: i've tried single quotes, double quotes, and even tried escaping the path with a \ but all to no avail. it just doesn't seem to want to parse that whitespace. and, unfortunately, i don't see any examples or references on how to do this in list archives, community documentation, etc. (and it's DEFINITELY not one of the examples given in the conf file). anyone have any idea how to do this? using v.0.10 if this is a bug and has been fixed in .11, let me know. - -- I wish ZORK had co-op mode. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) iD8DBQFIFx548u2Zh4MtlQoRA2UTAJ9l4S0a3uSiLyY3uZnbv8lLrJqBNQCeO0ni m2Gkl2ZZY9gqX78k/uCUVnU= =rert -----END PGP SIGNATURE----- From boxbackup at fluffy.co.uk Tue Apr 29 15:25:51 2008 From: boxbackup at fluffy.co.uk (James O'Gorman) Date: Tue, 29 Apr 2008 15:25:51 +0100 Subject: [Box Backup] spaces (and other ugliness) in Exclude paths In-Reply-To: <48171E78.5060907@acetechgroup.com> References: <48171E78.5060907@acetechgroup.com> Message-ID: <48172FEF.1060701@netinertia.co.uk> Brent Saner wrote: > in one of my exclude paths, i have a space. > > this is a GNU/Linux client, but it's backing up a Windows share (in > other words, stored locally and shared via cifs to Windows clients. the > directory itself is on an ext3 fs, obviously). > > normal BASH-magic does not work: i've tried single quotes, double > quotes, and even tried escaping the path with a \ but all to no avail. > it just doesn't seem to want to parse that whitespace. > > and, unfortunately, i don't see any examples or references on how to do > this in list archives, community documentation, etc. (and it's > DEFINITELY not one of the examples given in the conf file). Can you post what you've got in the config file now, and what you've tried? You might need to use ExcludeDirsRegex instead of ExcludeDir. e.g. ExcludeDirsRegex /some/dir\swith\sspaces James From boxbackup at fluffy.co.uk Tue Apr 29 15:46:59 2008 From: boxbackup at fluffy.co.uk (Brent Saner) Date: Tue, 29 Apr 2008 10:46:59 -0400 Subject: [Box Backup] spaces (and other ugliness) in Exclude paths In-Reply-To: <48172FEF.1060701@netinertia.co.uk> References: <48171E78.5060907@acetechgroup.com> <48172FEF.1060701@netinertia.co.uk> Message-ID: <481734E3.4060909@acetechgroup.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 James O'Gorman wrote: | Can you post what you've got in the config file now, and what you've tried? what's in now: 1 foo 2 { 3 Path = /foo/bar/foobar 4 ExcludeDir = /foo/bar/foobar/fobz/"TWO words" 5 } what i have tried before: ... 4 ExcludeDir = /foo/bar/foobar/fobz/'TWO words' ... as well as: ... 4 ExcludeDir = /foo/bar/foobar/fobz/TWO\ words ... and variations thereof, like: ... 4 ExcludeDir = /foo/bar/foobar/fobz/"TWO\ words" ... etc. | | You might need to use ExcludeDirsRegex instead of ExcludeDir. | | e.g. ExcludeDirsRegex /some/dir\swith\sspaces thanks! i'll give that a shot and come running back with my tail between my legs if it doesn't work. :D - -- I wish ZORK had co-op mode. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) iD8DBQFIFzTj8u2Zh4MtlQoRA1zfAKCACWFOujaD3qrd7nmTHQ17HUudfQCeMP3w Xbyjwzg+nloVO595TbrGBRQ= =UJ0H -----END PGP SIGNATURE----- From boxbackup at fluffy.co.uk Tue Apr 29 16:43:22 2008 From: boxbackup at fluffy.co.uk (James O'Gorman) Date: Tue, 29 Apr 2008 16:43:22 +0100 Subject: [Box Backup] spaces (and other ugliness) in Exclude paths In-Reply-To: <481734E3.4060909@acetechgroup.com> References: <48171E78.5060907@acetechgroup.com> <48172FEF.1060701@netinertia.co.uk> <481734E3.4060909@acetechgroup.com> Message-ID: <4817421A.4080909@netinertia.co.uk> Brent Saner wrote: > James O'Gorman wrote: > | Can you post what you've got in the config file now, and what you've > tried? > > what's in now: > > 1 foo > 2 { > 3 Path = /foo/bar/foobar > 4 ExcludeDir = /foo/bar/foobar/fobz/"TWO words" > 5 } Had you tried it just as: ExcludeDir = /foo/bar/foobar/fobz/TWO words I'm not sure if that would work or not. > | You might need to use ExcludeDirsRegex instead of ExcludeDir. > | > | e.g. ExcludeDirsRegex /some/dir\swith\sspaces > > thanks! i'll give that a shot and come running back with my tail between > my legs if it doesn't work. :D Let us know either way, it'll be nice to have in the archives :-) James From boxbackup at fluffy.co.uk Tue Apr 29 21:11:22 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 29 Apr 2008 21:11:22 +0100 (BST) Subject: [Box Backup] spaces (and other ugliness) in Exclude paths In-Reply-To: <4817421A.4080909@netinertia.co.uk> References: <48171E78.5060907@acetechgroup.com> <48172FEF.1060701@netinertia.co.uk> <481734E3.4060909@acetechgroup.com> <4817421A.4080909@netinertia.co.uk> Message-ID: Hi Brent and James, On Tue, 29 Apr 2008, James O'Gorman wrote: > Brent Saner wrote: > > > 1 foo > > 2 { > > 3 Path = /foo/bar/foobar > > 4 ExcludeDir = /foo/bar/foobar/fobz/"TWO words" > > 5 } > > Had you tried it just as: > ExcludeDir = /foo/bar/foobar/fobz/TWO words > > I'm not sure if that would work or not. My understanding is that it should work. Box Backup treats everything from the equals sign to the end of the line, with white space at the beginning and end trimmed off, as the configuration value. No escaping is supported. In practice, that should not be a problem unless you have a newline in your file name :-) Brent, please let us know if that doesn't work and I'll investigate. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Tue Apr 29 22:55:03 2008 From: boxbackup at fluffy.co.uk (boxbackup at fluffy.co.uk) Date: Tue, 29 Apr 2008 14:55:03 -0700 (PDT) Subject: [Box Backup] Re: client connection error SOLVED Message-ID: <17401.64.161.120.82.1209506103.squirrel@www.vancocomputing.net> Chris, just wanted to let you know that this morning's snapshot backup (initiated via cron) ran successfully. Please let me know if there's anything else you need from me, otherwise it looks like the situation has been solved. I'll let you know if I encounter similar problems on my other Debian clients. Feel free to contact me when you're ready to test on a Debian system and I'll set you up. Again, thank you for all of your help. Sean > Hey Chris, > >> I don't know either. It's a shame that we can't reproduce it. Please keep >> an eye on your backups and run a compare command regularly, and let me know if it appears to stop backing up for any reason. >> >> Do you have logs from the early runs which should have backed things up but didn't? >> > > I sure do have those client logs. They will be at the end of the e-mail and a quick note. > >> You don't get emails for backup-start and backup-finish by default, only for errors. >> >> If you run the command shown in those log lines (for a failure), do you get an email as expected? If not, what output do you get and what's in your mail server logs? > > If I run the command manually, I do get an error e-mail. I see in the local server's logs (backup mail server, normally dormant) that it sent me two error messages regarding the backup errors. One from this morning and one from a few minutes ago when I ran the script manually. > > > Thanks! > > Sean > > --------------------------- > > Apr 28 07:26:27 ns2 Box Backup (bbackupd)[5131]: NOTICE: Starting daemon, version 0.11rc2, config: /etc/box/bbackupd.conf > Apr 28 07:32:34 ns2 Box Backup (bbackupd)[5131]: NOTICE: Beginning scan of local files > Apr 28 07:32:34 ns2 Box Backup (bbackupd)[5131]: WARNING: Exception thrown: ConnectionException(Conn_SocketConnectError) at S > ocketStream.cpp(223) > Apr 28 07:32:34 ns2 Box Backup (bbackupd)[5131]: NOTICE: About to notify administrator about event backup-error, running scri > pt '/etc/box/bbackupd/NotifySysadmin.sh backup-error' > Apr 28 07:32:45 ns2 Box Backup (bbackupd)[5131]: NOTICE: File statistics: total file size uploaded 0, bytes already on server > 0, encoded size 0 > Apr 28 07:51:28 ns2 Box Backup (bbackupd)[5131]: NOTICE: Beginning scan of local files > Apr 28 07:51:28 ns2 Box Backup (bbackupd)[5131]: WARNING: Exception thrown: ConnectionException(Conn_SocketConnectError) at S > ocketStream.cpp(223) > Apr 28 07:51:28 ns2 Box Backup (bbackupd)[5131]: WARNING: Suppressing duplicate notification about backup-error > Apr 28 07:51:38 ns2 Box Backup (bbackupd)[5131]: NOTICE: File statistics: total file size uploaded 0, bytes already on server > 0, encoded size 0 > [IT DID NOT AUTOMATICALLY RUN THE NOTIFYSYSADMIN.SH SCRIPT FROM THIS POINT ON] - sorry for the caps, wanted to make a delineation from logs and the note. > > Repeated about 10 times, for 10 or so failed backup attempts. No associated logs on the bbstored server, as if the client didn't even make a connection attempt. -- Sean Van Couwenberghe Vanco Computing www.vancocomputing.net (707) 578-5802 From boxbackup at fluffy.co.uk Tue Apr 29 23:06:24 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Tue, 29 Apr 2008 23:06:24 +0100 (BST) Subject: [Box Backup] Re: client connection error SOLVED In-Reply-To: <17401.64.161.120.82.1209506103.squirrel@www.vancocomputing.net> References: <17401.64.161.120.82.1209506103.squirrel@www.vancocomputing.net> Message-ID: Hi Sean, On Tue, 29 Apr 2008, admin at vancocomputing.net wrote: > Chris, just wanted to let you know that this morning's snapshot backup > (initiated via cron) ran successfully. Great, thanks for letting me know. Also please don't forget to set up regular compares to verify your backups. > Please let me know if there's anything else you need from me, Not at the moment, thanks, but please get back to us if you have problems. > Feel free to contact me when you're ready to test on a Debian system and > I'll set you up. If you could email me the address, login and password of an account with sudo access off-list that would be great, thanks. Feel free to grab my pgp key and encrypt the mail if you prefer. The key ID and instructions to get it are on the website. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Wed Apr 30 08:20:58 2008 From: boxbackup at fluffy.co.uk (Magnus Homann) Date: Wed, 30 Apr 2008 09:20:58 +0200 Subject: [Box Backup] Restoring a corrupted archive? In-Reply-To: References: <48136012.4010608@homann.se> Message-ID: <48181DDA.2050004@homann.se> Hi all, I have now rsycned the data back (33 GB, about four days including such exciting events as ISP taking down the line for maintenace, and the remote machine had a complete kernel breakdown...) I did bbackupstore check fix, and got quite a few directories in lost+found0 (about 200). What to do with them? Is there a way to restore just 'missing' files as reported by "compare -aq", or do I have to delete the entire set of files I have still left, and do a complete restore? The documentation.txt is a bit unclear what "restore -d" does. -- Magnus Homann From boxbackup at fluffy.co.uk Wed Apr 30 10:00:19 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 30 Apr 2008 10:00:19 +0100 (BST) Subject: [Box Backup] Restoring a corrupted archive? In-Reply-To: <48181DDA.2050004@homann.se> References: <48136012.4010608@homann.se> <48181DDA.2050004@homann.se> Message-ID: Hi Magnus, On Wed, 30 Apr 2008, Magnus Homann wrote: > I have now rsycned the data back (33 GB, about four days including such > exciting events as ISP taking down the line for maintenace, and the > remote machine had a complete kernel breakdown...) > > I did bbackupstore check fix, and got quite a few directories in > lost+found0 (about 200). What to do with them? Download them with bbackupquery and check their contents to see if they're useful, otherwise let the store delete them. > Is there a way to restore just 'missing' files as reported by "compare > -aq", or do I have to delete the entire set of files I have still left, > and do a complete restore? You could generate a list of bbackupquery commands from the compare output with sed. > The documentation.txt is a bit unclear what "restore -d" does. Yes, sorry, it was discussed here on the mailing list this month, I think, please check the archives. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at fluffy.co.uk Wed Apr 30 10:58:05 2008 From: boxbackup at fluffy.co.uk (Magnus Homann) Date: Wed, 30 Apr 2008 11:58:05 +0200 Subject: [Box Backup] Restoring a corrupted archive? In-Reply-To: References: <48136012.4010608@homann.se> <48181DDA.2050004@homann.se> Message-ID: <481842AD.4080101@homann.se> Chris Wilson wrote: > On Wed, 30 Apr 2008, Magnus Homann wrote: >> Is there a way to restore just 'missing' files as reported by "compare >> -aq", or do I have to delete the entire set of files I have still left, >> and do a complete restore? > > You could generate a list of bbackupquery commands from the compare output > with sed. I go for the (re)move of the remaining files. >> The documentation.txt is a bit unclear what "restore -d" does. > > Yes, sorry, it was discussed here on the mailing list this month, I > think, please check the archives. Read it, and did not understand. :-) Got a TransformFailed(6/4) when restoring. I guess that can due to a file that is partly rsynced from a crashing local machine to a remote storage? I read that I should find the file in the archive and remove it, is that the right solution? -- Magnus Homann From boxbackup at fluffy.co.uk Wed Apr 30 16:11:28 2008 From: boxbackup at fluffy.co.uk (Magnus Homann) Date: Wed, 30 Apr 2008 17:11:28 +0200 Subject: [Box Backup] Restoring a corrupted archive? In-Reply-To: <481842AD.4080101@homann.se> References: <48136012.4010608@homann.se> <48181DDA.2050004@homann.se> <481842AD.4080101@homann.se> Message-ID: <48188C20.6050302@homann.se> After resolving the compression errors by finding the offending file in the store, remove it with normal file system rm, and run bbstoreaccounts check fix, I can pass the compression problems. Is it possibel to remove file with bbackupquery -i instead? The I don?t have to run bbstoraccounts check fix, which takes a long time. Ok, next issue: CannotDecodeDiffedFilesWithoutCombining (4/56) ??? What is the reason? How can I fix it? Magnus From boxbackup at fluffy.co.uk Wed Apr 30 22:19:33 2008 From: boxbackup at fluffy.co.uk (Magnus Homann) Date: Wed, 30 Apr 2008 23:19:33 +0200 Subject: [Box Backup] Restoring a corrupted archive? In-Reply-To: <48188C20.6050302@homann.se> References: <48136012.4010608@homann.se> <48181DDA.2050004@homann.se> <481842AD.4080101@homann.se> <48188C20.6050302@homann.se> Message-ID: <4818E265.2020107@homann.se> The archive seems quite corrupted, or the erropr handling of the restore function is not so robust. After fixing the CannotDecodeDiffedFilesWithoutCombining error in previous post by removing the object and run bbackupstore check fix, I have now encountered the following issue. BackupStore AttribuesNotLoaded (4/26) The error appears to occur on a directory, and if I remove the object '0xbfd9', it is rebuilt with subsequent error of the same kind. It seems the retore process is not very friendly to a corrupted archive. I run Ubuntu 0.10-1 version of boxbackup. Is there any use in downloading .011rc3 and use that server and client? Will that REALYY mess my system up? I'm starting to get a bit desperate here... Magnus From boxbackup at fluffy.co.uk Wed Apr 30 22:29:23 2008 From: boxbackup at fluffy.co.uk (Chris Wilson) Date: Wed, 30 Apr 2008 22:29:23 +0100 (BST) Subject: [Box Backup] Restoring a corrupted archive? In-Reply-To: <4818E265.2020107@homann.se> References: <48136012.4010608@homann.se> <48181DDA.2050004@homann.se> <481842AD.4080101@homann.se> <48188C20.6050302@homann.se> <4818E265.2020107@homann.se> Message-ID: Hi Magnus, On Wed, 30 Apr 2008, Magnus Homann wrote: > The archive seems quite corrupted, or the erropr handling of the restore > function is not so robust. > > After fixing the CannotDecodeDiffedFilesWithoutCombining error in previous > post by removing the object and run bbackupstore check fix, I have now > encountered the following issue. > > BackupStore AttribuesNotLoaded (4/26) > > The error appears to occur on a directory, and if I remove the object > '0xbfd9', it is rebuilt with subsequent error of the same kind. > > It seems the retore process is not very friendly to a corrupted archive. I run > Ubuntu 0.10-1 version of boxbackup. Is there any use in downloading .011rc3 > and use that server and client? Will that REALYY mess my system up? > > I'm starting to get a bit desperate here... This really should not be happening. Is this all data that you copied from the remote system, i.e. no remnants of your old corrupt local backup? Is it possible that the local copy got corrupted while it was being rsynced? Also, were you running compare regularly on your files? When did you last run it? It's possible that these errors were in your store for a while but never got detected because compare wasn't run. How exactly did your local filesystem get corrupted? Was it user error or a hardware failure? Could bad hardware have been slowly corrupting the local copy? Do you have compressed encrypted files over 2GB in the remote repository? When you say 0xbfd9 is recreated, do you know by what? If it's by bbstoreaccounts check fix, could you tell me what it outputs when it does this? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software |