From brendon at netcal.com Fri Apr 11 20:23:17 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Fri, 11 Apr 2014 19:23:17 +0000 Subject: [Box Backup] compiling from svn Message-ID: I'm trying to compile from svn and I'm getting what you see below. Is it a code issue or my issue? If its my issue, any idea what it might be? I'm using debian 7. $ make (cd bin/bbackupd; make RELEASE=1) make[1]: Entering directory `/tmp/w/trunk/bin/bbackupd' make[2]: Entering directory `/tmp/w/trunk/lib/server' [CXX] lib/server/SocketStream.cpp [CXX] lib/server/TLSContext.cpp [CXX] lib/server/TcpNice.cpp [CXX] lib/server/WinNamedPipeStream.cpp [CXX] lib/server/autogen_ConnectionException.cpp SocketStream.cpp: In member function 'virtual bool SocketStream::GetPeerCredentials(uid_t&, gid_t&)': SocketStream.cpp:466:5: error: '::getpeereid' has not been declared make[2]: *** [../../release/lib/server/SocketStream.o] Error 1 make[2]: *** Waiting for unfinished jobs.... make[2]: Leaving directory `/tmp/w/trunk/lib/server' make[1]: *** [dep_modules] Error 2 make[1]: Leaving directory `/tmp/w/trunk/bin/bbackupd' make: *** [release/bin/bbackupd/bbackupd] Error 2 $ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at qwirx.com Fri Apr 11 23:02:33 2014 From: chris at qwirx.com (Chris Wilson) Date: Fri, 11 Apr 2014 23:02:33 +0100 (BST) Subject: [Box Backup] compiling from svn Message-ID: Hi Brendon, On Fri, 11 Apr 2014, Brendon Baumgartner wrote: > I?m trying to compile from svn and I?m getting what you see below. Is it a > code issue or my issue? If its my issue, any idea what it might be? Sorry about that. It's probably a fault in my code. > SocketStream.cpp: In member function ?virtual bool > SocketStream::GetPeerCredentials(uid_t&, gid_t&)?: > SocketStream.cpp:466:5: error: ?::getpeereid? has not been declared This code has not changed in a long time (since 2005), and I haven't seen this error before, so it might be something new in Debian 7. getpeereid() is only supposed to be used if it was detected by configure. Please could you send me your lib/common/BoxConfig.h and config.log files so I can see if it was detected on your system, and hopefully figure out why we can't actually use it? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Sat Apr 12 20:58:23 2014 From: chris at qwirx.com (Chris Wilson) Date: Sat, 12 Apr 2014 20:58:23 +0100 (BST) Subject: [Box Backup] Competition, development, future direction? In-Reply-To: References: <5018668E.1080102@qustodium.net> <50376FA7.3080303@invis.net> Message-ID: Hi all, On Sat, 8 Feb 2014, Chris Wilson wrote: > On Fri, 24 Aug 2012, Charles Lecklider wrote: > >> Once upon a time I had a working (albeit extremely basic) MMC snap-in >> for BB. Unfortunately, while I wasn't paying attention the license >> changed to GPL, so I can no longer link lib/backupclient (there used to >> be a dependency on lib/backupstore too, but I think recent restructuring >> has removed that - I've not checked in detail yet); yes, obviously I can >> start parsing the output from bbackupquery and so forth, but there are >> too many problems with that approach for me to be comfortable with it. >> >> On the 17th January 2011 I asked about getting those two libs put under >> the LGPL. On the 18th Chris said: >> >>> Does anyone have any objections to either: >>> >>> 1. Relicensing lib/backupclient under a dual LGPL-BSD license, or >>> >>> 2. Relicensing lib/backupclient under the BSD license (without advertising >>> clause)? >> >> But I think that's as far as things went. >> >> If you read the LICENSE file, lib/backupclient and lib/backupstore are >> only GPL. I don't think they need to be BSD at all, so it's just a >> matter of changing two libs from GPL to LGPL, nothing more. >> >> I know having an MMC snap-in won't directly help the *nix side of >> things, but my feeling is that allowing the UI to link to BB would allow >> more flexibility which in turn may produce better results. >> >> In any case, I can't see how the license change would hurt. > > I am still slightly against changing the license again. The reason I wanted > the GPL license in the first place (apart from compatibility with readline, > which we may have lost by using a modified GPL, and by requiring openssl) was > to ensure that any improvements made to Box Backup, by anyone, came back into > the public domain. > > Are you so against the GPL that you cannot have your code published under it > at all? Or are you restricted by the license on some other code that it links > with? > > I would consider dual-licensing the library parts under both the original > license (Ben's license, BSD with attribution) and the GPL, so you could > distribute the MMC snapin using that license. But it's already complex and I > don't really want to make it more complex if I can avoid it. I have not heard any objections to such a relicensing. I am still not keen on the 2-clause BSD license, because it allows people to make commercial products using our work without either giving credit or contributing back to the community. So what I propose is to END the current mixed license state, and relicense the ENTIRE code base under the same license, a dual license which allows you to use it under: * EITHER the original Box Backup BSD license, the 4-clause BSD license with advertising clause: https://www.boxbackup.org/browser/box/trunk/LICENSE.txt?rev=2290 * OR the Box Backup GPL, which is GPLv2 with exceptions to allow linking against OpenSSL and VSS libraries. The reason is that: * This requires all redistribution of the code or resulting binaries to EITHER give us credit OR include the full source code under the GPL, satisfying my requirements. * Hopefully it will allow Charles' derived works to be released and distributed. * The Box Backup GPL version satisfies Fedora's license requirements, because it can be used under the GPL without complying with the obnoxious advertising clause. On 2010-01-23 we relicensed the core libraries under BSD 2-clause and the Box Backup GPL. The BSD 2-clause license is less restrictive than the original 4-clause version, and thus we can restrict the license on this code further (relicense as above) without additional permission from coyright holders. The Box Backup libraried (lib/bbackupd and lib/bbstored), daemons and tools were licensed under the original Box Backup license until r2596, and the Box Backup GPL since then. So to dual license them would require permission only from the contributors whose code has been added since r2596 to allow their code to be dual-licensed under the original Box Backup BSD license as well. Apart from me, the only other direct committer was James O'Gorman: r2680 | jamesog | 2010-04-11 17:18:00 +0100 (Sun, 11 Apr 2010) | 2 lines r2854 | jamesog | 2011-01-19 23:01:54 +0000 (Wed, 19 Jan 2011) | 5 lines r2860 | jamesog | 2011-02-19 17:25:33 +0000 (Sat, 19 Feb 2011) | 2 lines r2948 | jamesog | 2011-04-28 21:55:26 +0100 (Thu, 28 Apr 2011) | 2 lines r2951 | jamesog | 2011-05-01 17:34:15 +0100 (Sun, 01 May 2011) | 2 lines r2952 | jamesog | 2011-05-01 22:14:59 +0100 (Sun, 01 May 2011) | 2 lines r2953 | jamesog | 2011-05-02 12:21:33 +0100 (Mon, 02 May 2011) | 6 lines r2954 | jamesog | 2011-05-02 12:25:49 +0100 (Mon, 02 May 2011) | 5 lines r2955 | jamesog | 2011-05-02 14:17:45 +0100 (Mon, 02 May 2011) | 2 lines r2956 | jamesog | 2011-05-02 14:36:11 +0100 (Mon, 02 May 2011) | 2 lines r2979 | jamesog | 2011-08-21 11:20:08 +0100 (Sun, 21 Aug 2011) | 2 lines r3110 | jamesog | 2012-05-24 21:01:14 +0100 (Thu, 24 May 2012) | 4 lines and the patches by other people that I merged were: ------------------------------------------------------------------------ r2835 | chris | 2011-01-09 01:56:36 +0000 (Sun, 09 Jan 2011) | 5 lines Fix incorrect generation of temporary file name for applying patches while restoring old versions of files, many thanks to Matto Marjanovic for debugging this issue and submitting a patch! ------------------------------------------------------------------------ r3027 | chris | 2011-10-18 00:19:48 +0100 (Tue, 18 Oct 2011) | 3 lines Patch to support building on MinGW, thanks to Paolo Tosco (http://open3dalign.org, http://open3dqsar.org) ------------------------------------------------------------------------ r3034 | chris | 2011-10-31 21:09:45 +0000 (Mon, 31 Oct 2011) | 2 lines Improve error message on bbackupd command socket bind failure, thanks to Christophe. ------------------------------------------------------------------------ r3111 | chris | 2012-05-26 19:13:01 +0100 (Sat, 26 May 2012) | 10 lines bbackupquery readline improvements, thanks to Paolo Tosco: ------------------------------------------------------------------------ r3181 | chris | 2013-09-02 23:10:21 +0100 (Mon, 02 Sep 2013) | 2 lines Fix missing include, thanks Arnaud. So if I can get explicit permission from these authors: James O'Gorman, Matto Marjanovic, Paolo Tosca, Christophe and Arnaud, then we can change the license as stated above. If we don't get permission from a particular contributor then we could remove their contributed code. I will contact these contributors individually asking for permission. I am setting a target date of 01/05/2014 for the relicensing. Please let me know before then if you have any objections. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Sat Apr 12 20:59:30 2014 From: chris at qwirx.com (Chris Wilson) Date: Sat, 12 Apr 2014 20:59:30 +0100 (BST) Subject: [Box Backup] Proposal: Switch to Git version control Message-ID: Hi all, James O'Gorman and I have been discussing the merits of switching to using Git instead of Subversion for version control, and moving the code to Github. Advantages that I'm aware of: * I already use Git for many projects, and I'm now very familiar and comfortable with it. I started using git-svn on Box Backup earlier this year, to get the advantages of Git such as working offline and working with branches easily. I would get more benefit if our main repository was in Git, not in Subversion. * We'd get credit on Github for our contributions to Box Backup. * We'd have the Github issue tracker which is much better than Trac's and solves part of the spam problem that forced us to disable registrations on Trac, which prevents people from submitting tickets. The main disadvantage that I can think of is that checking out and updating the source code would require knowledge of Git instead of Subversion, which is newer and therefore less common. I am NOT currently proposing to move the website or the mailing list. I know that hosting the mailing list on Mailman is painful for James, and would be happy to investigate alternatives that make it easier, but we can do that independently. I see no particular reason to move the website off Trac unless we can find a better Wiki solution. Github Pages might be it, or it might not. It would be nice to be able to edit Pages in Markdown, clone a copy of the entire wiki and convert it to HTML as offline documentation. James has already done a lot of work in preparing a Git-SVN configuration that can be used to migrate our code with full history to Github. I think he's using this configuration that he sent me by email a month ago: >>> My migrate had converted most of the Subversion usernames to "Full Name >>> " as is the Git style, and I think excluded the git-svn stuff in the >>> commit messages, as well as creating each person's branches. >> >> How did you do it? > > Not easily! It took a lot of fiddling and tweaking and spending hours > re-running the import. Here's my .git/config: [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true ignorecase = true precomposeunicode = false [svn-remote "svn"] noMetadata = 1 ignore-paths = ^(bitten|boxbackup-web|james|packages) url = https://www.boxbackup.org/svn/box fetch = trunk:refs/remotes/trunk branches = ./*:refs/remotes/* branches = ben/*:refs/remotes/ben/* branches = chris/{acl,boxi,general,retry-debug}:refs/remotes/chris/* branches = chris/win32/*:refs/remotes/chris/win32/* branches = chromi/*:refs/remotes/chromi/* branches = gary/{0.09-orig,boxbackup-0.09,boxbackup-0.09-mod}:refs/remotes/gary/* branches = features/*:refs/remotes/features/* branches = invisnet/{perfhack1,vs2010}:refs/remotes/invisnet/* branches = martin/*:refs/remotes/martin/* branches = nick/*:refs/remotes/nick/* tags = RELEASE/*:refs/remotes/tags/* tags = snapshots/*:refs/remotes/tags/* [svn] authorsfile = /Users/jamesog/git/users.txt > And the authorsfile looks like: ben = Ben Summers cbkm = Kenny Millington chris = Chris Wilson chromi = Jonathan Morton daveb = Dave Bamford gary = Gary invisnet = Charles Lecklider jamesog = James O'Gorman martin = Martin Ebourne nick = Nick Knight per = Per Reedtz Thomsen petjal = Pete Jalajas stenor = Stefan Norlin > I couldn't find the email addresses for Gary or Nick. If you could confirm your Github email addresses then we can make sure you get full credit on Github for your commits if the migration goes ahead. So please could you let us know if you agree or disagree with this change? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Sat Apr 12 23:08:32 2014 From: chris at qwirx.com (Chris Wilson) Date: Sat, 12 Apr 2014 23:08:32 +0100 (BST) Subject: [Box Backup] compiling from svn Message-ID: Hi Brendon, On Fri, 11 Apr 2014, Chris Wilson wrote: > On Fri, 11 Apr 2014, Brendon Baumgartner wrote: > >> SocketStream.cpp: In member function ?virtual bool >> SocketStream::GetPeerCredentials(uid_t&, gid_t&)?: >> SocketStream.cpp:466:5: error: ?::getpeereid? has not been declared > > This code has not changed in a long time (since 2005), and I haven't seen > this > error before, so it might be something new in Debian 7. > > getpeereid() is only supposed to be used if it was detected by configure. > Please could you send me your lib/common/BoxConfig.h and config.log files so > I > can see if it was detected on your system, and hopefully figure out why we > can't actually use it? Thanks for the report. I think I've found and fixed the problem. Please could you update from svn and try to compile again? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Sat Apr 12 23:09:42 2014 From: chris at qwirx.com (Chris Wilson) Date: Sat, 12 Apr 2014 23:09:42 +0100 (BST) Subject: [Box Backup] Competition, development, future direction? Message-ID: Hi Paolo, On Sat, 12 Apr 2014, Paolo Tosco wrote: > I definitely agree with the re-licensing. By the way, I have just released > WinEditLine 2.101 (formerly MinGWEditLine); compared to the latest > MinGWEditLine, WinEditLine builds also with MSVC compilers and has a few > bugfixes. Thanks! I normally compile on Windows with MinGW myself, but I am passing your message on to the list, so that those who use MSVC builds will be able to benefit from this. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From cam.lafit at azerttyu.net Sun Apr 13 12:49:48 2014 From: cam.lafit at azerttyu.net (cam.lafit at azerttyu.net) Date: Sun, 13 Apr 2014 13:49:48 +0200 Subject: [Box Backup] Proposal: Switch to Git version control In-Reply-To: References: Message-ID: Hi Git-svn is not very great :) At least because is not bijective. I use subgit a hook to syncrhonize git and svn via some hooks (http://subgit.com/). I use it on other opensource projects (alternc and spip). In this case we can conserve svn users and add git user or provide two access. And commit via git or svn are equivalents. We have more time to do a change users behaviour. We can add also a github mirror. If you want I can help to do this. Km From siretart at gmail.com Sun Apr 13 14:19:11 2014 From: siretart at gmail.com (Reinhard Tartler) Date: Sun, 13 Apr 2014 09:19:11 -0400 Subject: [Box Backup] Proposal: Switch to Git version control In-Reply-To: References: Message-ID: On Sat, Apr 12, 2014 at 3:59 PM, Chris Wilson wrote: > So please could you let us know if you agree or disagree with this change? With my Debian packaging hat on, I'd also very much welcome this step! -- regards, Reinhard From brendon at netcal.com Mon Apr 14 18:35:37 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Mon, 14 Apr 2014 17:35:37 +0000 Subject: [Box Backup] compiling from svn In-Reply-To: References: Message-ID: Thanks. That fixed it! -----Original Message----- From: Chris Wilson [mailto:chris+pine at qwirx.com] Sent: Saturday, April 12, 2014 15:04 To: Brendon Baumgartner Cc: Box Backup Subject: RE: [Box Backup] compiling from svn Hi Brendon, On Fri, 11 Apr 2014, Chris Wilson wrote: > On Fri, 11 Apr 2014, Brendon Baumgartner wrote: > >> SocketStream.cpp: In member function ?virtual bool >> SocketStream::GetPeerCredentials(uid_t&, gid_t&)?: >> SocketStream.cpp:466:5: error: ?::getpeereid? has not been declared > > This code has not changed in a long time (since 2005), and I haven't seen this > error before, so it might be something new in Debian 7. > > getpeereid() is only supposed to be used if it was detected by configure. > Please could you send me your lib/common/BoxConfig.h and config.log files so I > can see if it was detected on your system, and hopefully figure out why we > can't actually use it? Thanks for the report. I think I've found and fixed the problem. Please could you update from svn and try to compile again? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Mon Apr 14 19:22:19 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Mon, 14 Apr 2014 18:22:19 +0000 Subject: [Box Backup] issue with refcount/housekeeping Message-ID: Chris- Now that I'm running the svn version of box backup I have some new issues. Previously I was running the very old version in debian : 0.11.1~r2837-1. I'm seeing these in the log and they aren't going away after the 900 seconds when housekeeping is supposed to run. Apr 14 10:58:43 modena bbstored[7223]: NOTICE: Handling incoming connection from x.x.229.2 port 21515 Apr 14 10:58:43 modena bbstored client=0x00000021[7223]: WARNING: Failed to open file: /u2/boxbackup/backup/00000021/refcount.rdb.rfw: No such file or directory (2) Apr 14 10:58:43 modena bbstored client=0x00000021[7223]: WARNING: Exception thrown: CommonException(OSFileOpenError) (/u2/boxbackup/backup/00000021/refcount.rdb.rfw) at FileStream.cpp:86 Apr 14 10:58:43 modena bbstored client=0x00000021[7223]: WARNING: Exception thrown: BackupStoreException(CorruptReferenceCountDatabase) (Reference count database is missing or corrupted, cannot safely open account. Housekeeping will fix this automatically when it next runs.) at BackupStoreContext.cpp:232 So I manually tried running it. # bbstoreaccounts housekeep 21 TRACE: Starting housekeeping on account 0x00000021 WARNING: Failed to open file: /u2/boxbackup/backup/00000021/refcount.rdb.rfw: No such file or directory (2) TRACE: Obtained 9 stack frames. TRACE: Stack frame 0: DumpStackBacktrace()+0x23 TRACE: Stack frame 1: FileStream::AfterOpen()+0xbec TRACE: Stack frame 2: FileStream::FileStream(std::string const&, int, int)+0x5f TRACE: Stack frame 3: BackupStoreRefCountDatabase::Load(BackupStoreAccountDatabase::Entry const&, bool)+0x5a TRACE: Stack frame 4: HousekeepStoreAccount::DoHousekeeping(bool)+0xddf TRACE: Stack frame 5: BackupStoreAccountsControl::HousekeepAccountNow(int)+0x3f3 TRACE: Stack frame 6: main+0xa27 TRACE: Stack frame 7: __libc_start_main+0xfd TRACE: Stack frame 8: unknown address 0x432b39 WARNING: Exception thrown: CommonException(OSFileOpenError) (/u2/boxbackup/backup/00000021/refcount.rdb.rfw) at FileStream.cpp:86 WARNING: Reference count database was missing or corrupted during housekeeping, cannot check it for errors. TRACE: Finished housekeeping on account 0x00000021 TRACE: Finished housekeeping on account 0x00000021 # I tried it a second time and it appears to work without errors: # bbstoreaccounts housekeep 21 TRACE: Starting housekeeping on account 0x00000021 TRACE: Finished housekeeping on account 0x00000021 TRACE: Finished housekeeping on account 0x00000021 # But this is still showing in the logs: Apr 14 11:16:23 modena bbstored[7891]: NOTICE: Handling incoming connection from x.x.229.2 port 2828 Apr 14 11:16:23 modena bbstored client=0x00000021[7891]: WARNING: Failed to open file: /u2/boxbackup/backup/00000021/refcount.rdb.rfw: Permission denied (13) Apr 14 11:16:23 modena bbstored client=0x00000021[7891]: WARNING: Exception thrown: CommonException(AccessDenied) (/u2/boxbackup/backup/00000021/refcount.rdb.rfw) at FileStream.cpp:81 Apr 14 11:16:23 modena bbstored client=0x00000021[7891]: WARNING: Exception thrown: BackupStoreException(CorruptReferenceCountDatabase) (Reference count database is missing or corrupted, cannot safely open account. Housekeeping will fix this automatically when it next runs.) at BackupStoreContext.cpp:232 Apr 14 11:16:23 modena bbstored client=0x00000021[7891]: NOTICE: Connection statistics for 0x00000021 (name=): IN=60 OUT=44 NET_IN=16 TOTAL=104 Maybe the issue is because the client is running an older version? Thanks for any help. -Brendon From brendon at netcal.com Tue Apr 15 03:38:24 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Tue, 15 Apr 2014 02:38:24 +0000 Subject: [Box Backup] issue with refcount/housekeeping Message-ID: Strange. After a bit it seems these errors have vanished. I'll keep an eye on it. -----Original Message----- From: Brendon Baumgartner Sent: Monday, April 14, 2014 11:22 To: Box Backup Subject: issue with refcount/housekeeping Chris- Now that I'm running the svn version of box backup I have some new issues. Previously I was running the very old version in debian : 0.11.1~r2837-1. I'm seeing these in the log and they aren't going away after the 900 seconds when housekeeping is supposed to run. Apr 14 10:58:43 modena bbstored[7223]: NOTICE: Handling incoming connection from x.x.229.2 port 21515 Apr 14 10:58:43 modena bbstored client=0x00000021[7223]: WARNING: Failed to open file: /u2/boxbackup/backup/00000021/refcount.rdb.rfw: No such file or directory (2) Apr 14 10:58:43 modena bbstored client=0x00000021[7223]: WARNING: Exception thrown: CommonException(OSFileOpenError) (/u2/boxbackup/backup/00000021/refcount.rdb.rfw) at FileStream.cpp:86 Apr 14 10:58:43 modena bbstored client=0x00000021[7223]: WARNING: Exception thrown: BackupStoreException(CorruptReferenceCountDatabase) (Reference count database is missing or corrupted, cannot safely open account. Housekeeping will fix this automatically when it next runs.) at BackupStoreContext.cpp:232 So I manually tried running it. # bbstoreaccounts housekeep 21 TRACE: Starting housekeeping on account 0x00000021 WARNING: Failed to open file: /u2/boxbackup/backup/00000021/refcount.rdb.rfw: No such file or directory (2) TRACE: Obtained 9 stack frames. TRACE: Stack frame 0: DumpStackBacktrace()+0x23 TRACE: Stack frame 1: FileStream::AfterOpen()+0xbec TRACE: Stack frame 2: FileStream::FileStream(std::string const&, int, int)+0x5f TRACE: Stack frame 3: BackupStoreRefCountDatabase::Load(BackupStoreAccountDatabase::Entry const&, bool)+0x5a TRACE: Stack frame 4: HousekeepStoreAccount::DoHousekeeping(bool)+0xddf TRACE: Stack frame 5: BackupStoreAccountsControl::HousekeepAccountNow(int)+0x3f3 TRACE: Stack frame 6: main+0xa27 TRACE: Stack frame 7: __libc_start_main+0xfd TRACE: Stack frame 8: unknown address 0x432b39 WARNING: Exception thrown: CommonException(OSFileOpenError) (/u2/boxbackup/backup/00000021/refcount.rdb.rfw) at FileStream.cpp:86 WARNING: Reference count database was missing or corrupted during housekeeping, cannot check it for errors. TRACE: Finished housekeeping on account 0x00000021 TRACE: Finished housekeeping on account 0x00000021 # I tried it a second time and it appears to work without errors: # bbstoreaccounts housekeep 21 TRACE: Starting housekeeping on account 0x00000021 TRACE: Finished housekeeping on account 0x00000021 TRACE: Finished housekeeping on account 0x00000021 # But this is still showing in the logs: Apr 14 11:16:23 modena bbstored[7891]: NOTICE: Handling incoming connection from x.x.229.2 port 2828 Apr 14 11:16:23 modena bbstored client=0x00000021[7891]: WARNING: Failed to open file: /u2/boxbackup/backup/00000021/refcount.rdb.rfw: Permission denied (13) Apr 14 11:16:23 modena bbstored client=0x00000021[7891]: WARNING: Exception thrown: CommonException(AccessDenied) (/u2/boxbackup/backup/00000021/refcount.rdb.rfw) at FileStream.cpp:81 Apr 14 11:16:23 modena bbstored client=0x00000021[7891]: WARNING: Exception thrown: BackupStoreException(CorruptReferenceCountDatabase) (Reference count database is missing or corrupted, cannot safely open account. Housekeeping will fix this automatically when it next runs.) at BackupStoreContext.cpp:232 Apr 14 11:16:23 modena bbstored client=0x00000021[7891]: NOTICE: Connection statistics for 0x00000021 (name=): IN=60 OUT=44 NET_IN=16 TOTAL=104 Maybe the issue is because the client is running an older version? Thanks for any help. -Brendon From chris at qwirx.com Tue Apr 15 10:21:56 2014 From: chris at qwirx.com (Chris Wilson) Date: Tue, 15 Apr 2014 10:21:56 +0100 (BST) Subject: [Box Backup] issue with refcount/housekeeping In-Reply-To: References: Message-ID: Hi Brendon, On Tue, 15 Apr 2014, Brendon Baumgartner wrote: > Apr 14 11:16:23 modena bbstored client=0x00000021[7891]: WARNING: Failed > to open file: /u2/boxbackup/backup/00000021/refcount.rdb.rfw: Permission > denied (13) Note that this is a *different* error than the first one: Permission denied. Because you ran housekeeping as root, some files are now owned as root, so boxbackup will not be able to update them. Perhaps it was able to delete the refcount db itself at the next housekeeping, and recover from that error by itself, but other files still owned by root will create problems for you in future. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Tue Apr 15 22:12:12 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Tue, 15 Apr 2014 21:12:12 +0000 Subject: [Box Backup] issue with refcount/housekeeping In-Reply-To: References: Message-ID: > -----Original Message----- > From: boxbackup-bounces at boxbackup.org [mailto:boxbackup- > bounces at boxbackup.org] On Behalf Of Chris Wilson > Sent: Tuesday, April 15, 2014 02:22 > To: Box Backup > Subject: Re: [Box Backup] issue with refcount/housekeeping > > Hi Brendon, > > On Tue, 15 Apr 2014, Brendon Baumgartner wrote: > > > Apr 14 11:16:23 modena bbstored client=0x00000021[7891]: WARNING: > Failed > > to open file: /u2/boxbackup/backup/00000021/refcount.rdb.rfw: > Permission > > denied (13) > > Note that this is a *different* error than the first one: Permission > denied. > > Because you ran housekeeping as root, some files are now owned as root, > so > boxbackup will not be able to update them. > > Perhaps it was able to delete the refcount db itself at the next > housekeeping, and recover from that error by itself, but other files > still owned by root will create problems for you in future. > > Cheers, Chris. Yea. That makes sense. For some reason I thought it was like bbackupctl where it triggered something -- in this case the server to run housekeeping. I'll check my permissions. -BB From brendon at netcal.com Wed Apr 16 19:20:13 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Wed, 16 Apr 2014 18:20:13 +0000 Subject: [Box Backup] how to properly use snapshot backups? Message-ID: It appears snapshot mode doesn't work or I don't know how to read the log or I don't know how to configure it :) I had this server running in lazy mode and decided I wanted backups to only run at a set schedule. I changed bbackupd.conf and set "AutomaticBackup = no". That didn't seem to work because the server still ran backups continuously. I then re-created the bbackupd.conf file using the config script and it gave me a few different results in the config file so I figured that was the problem. I let the new config run and now I have everything as correct as I can think of but backups still run continuously. See below for the current setup. Notice the backup starts again right after one finishes at "Apr 16 10:46:55". (log below). _BB # grep bba cron Apr 13 22:03:01 hosting crond[25816]: (root) CMD (/usr/sbin/bbackupctl sync) Apr 15 22:03:01 hosting crond[30211]: (root) CMD (/usr/sbin/bbackupctl sync) # # grep bba messages |grep NOTICE ... Apr 15 15:22:15 hosting bbackupd[31477]: NOTICE: Starting daemon, version: b3328 Apr 15 15:22:15 hosting bbackupd[31477]: NOTICE: Using configuration file: /etc/boxbackup/bbackupd.conf Apr 15 22:03:03 hosting bbackupd[31477]: NOTICE: Beginning scan of local files Apr 15 22:03:04 hosting bbackupd[31477]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_boot.n Apr 15 22:03:04 hosting bbackupd[31477]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_home.n Apr 15 22:03:05 hosting bbackupd[31477]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_.n Apr 16 10:46:55 hosting bbackupd[31477]: NOTICE: Finished scan of local files Apr 16 10:46:55 hosting bbackupd[31477]: NOTICE: File statistics: total file size uploaded 11876833036, bytes already on server 3552099899, encoded size 1872330161, 29681 files uploaded, 4192 dirs created Apr 16 10:46:55 hosting bbackupd[31477]: NOTICE: Beginning scan of local files Apr 16 10:46:55 hosting bbackupd[31477]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_boot.n Apr 16 10:46:55 hosting bbackupd[31477]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_home.n Apr 16 10:46:55 hosting bbackupd[31477]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_.n # Pertinent lines from bbackupd.conf : AutomaticBackup = no UpdateStoreInterval = 0 MinimumFileAge = 0 MaxUploadWait = 0 From chris at qwirx.com Fri Apr 18 11:03:17 2014 From: chris at qwirx.com (Chris Wilson) Date: Fri, 18 Apr 2014 11:03:17 +0100 (BST) Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hi Brendon, On Wed, 16 Apr 2014, Brendon Baumgartner wrote: > It appears snapshot mode doesn't work or I don't know how to read the > log or I don't know how to configure it :) > > I had this server running in lazy mode and decided I wanted backups to > only run at a set schedule. I changed bbackupd.conf and set > "AutomaticBackup = no". That didn't seem to work because the server > still ran backups continuously. That should be all that you have to change. Did you restart bbackupd after making that change? The command "bbackupctl -V status" will tell you whether the backup daemon thinks that automatic backups are enabled or not If your bbackupctl command doesn't recognise the -V option, try without it. I did some manual testing and was not able to reproduce the bug. If it's not a mistake in the configuration file, I'm not sure why it could be doing this, so I added some more debugging code to check. Please could you update your subversion checkout, recompile and run bbackupd with the -V option? The start of each backup should look something like this: INFO: About to notify administrator about event backup-start, running script '/usr/bin/perl testfiles/notifyscript.pl backup-start "testfiles/bbackupd.conf"' TRACE: timer CommandSocketPollTimer (0x1a15920) initialised for 1000 ms, to fire at 10:38:35.774607 TRACE: timer CommandSocketPollTimer (0x1a15920) added to global queue, rescheduling TRACE: timer: next event: CommandSocketPollTimer at 10:38:35.774607 NOTICE: Beginning scan of local files Just before that, you should see something like this: INFO: No automatic backups, waiting for bbackupctl snapshot command TRACE: Wait on command socket, delay = 1024000000 Or this: INFO: Automatic backups are enabled, next one starting at 10:35:56 INFO: Automatic backups are enabled, next one starting now Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Fri Apr 18 18:32:43 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Fri, 18 Apr 2014 17:32:43 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Thanks! I downloaded 3341 and installed it. Started it with -V and ran a sync. We'll see what happens. So far this is what I got... looks correct. I'll see what happens when it finishes. (See #1 below) One strange thing though is that I was seeing this socket error messages. I checked the socket file and everything is running as root. I stopped and started bbackupd and was getting the same error. I then deleted it and restarted bbackupd and the error stopped occurring. I checked the file before and after and it was the same. Strange but maybe somehow this is related to my problem? Hmm... then again maybe not. I just looked through the old logs and don't see this error. It just started this morning. #2 -- socket error output: Apr 18 10:01:28 hosting bbackupd[12635]: NOTICE: Box Backup Client vtrunk_3341, (c) Ben Summers and contributors 2003-2014 Apr 18 10:01:28 hosting bbackupd[12637]: NOTICE: Starting daemon, version: trunk_3341 Apr 18 10:01:28 hosting bbackupd[12637]: NOTICE: Using configuration file: /etc/boxbackup/bbackupd.conf Apr 18 10:01:41 hosting bbackupctl[12658]: Using configuration file /etc/boxbackup/bbackupd.conf Apr 18 10:01:41 hosting bbackupctl[12658]: NOTICE: state 0 Idle Apr 18 10:01:41 hosting bbackupd[12637]: ERROR: Failed to write to socket: Broken pipe (32) Apr 18 10:01:41 hosting bbackupd[12637]: WARNING: Exception thrown: ConnectionException(Conn_SocketWriteError) at SocketStream.cpp(313) Apr 18 10:01:41 hosting bbackupd[12637]: NOTICE: Failed to write to command socket: Connection SocketWriteError (Probably a network issue between client and server.) #1 -- Looking good output: Apr 18 10:20:28 hosting bbackupd[18988]: NOTICE: Box Backup Client vtrunk_3341, (c) Ben Summers and contributors 2003-2014 Apr 18 10:20:28 hosting bbackupd[18990]: NOTICE: Starting daemon, version: trunk_3341 Apr 18 10:20:28 hosting bbackupd[18990]: NOTICE: Using configuration file: /etc/boxbackup/bbackupd.conf Apr 18 10:20:28 hosting bbackupd[18990]: No automatic backups, waiting for bbackupctl snapshot command Apr 18 10:20:51 hosting bbackupctl[19100]: Using configuration file /etc/boxbackup/bbackupd.conf Apr 18 10:20:51 hosting bbackupd[18990]: Incoming connection from local (UNIX socket) Apr 18 10:20:51 hosting bbackupd[18990]: Connection from command socket Apr 18 10:20:51 hosting bbackupd[18990]: Starting a backup immediately due to bbackupctl sync command Apr 18 10:20:51 hosting bbackupd[18990]: About to notify administrator about event backup-start, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-start "/etc/boxbackup/bbackupd.conf"' Apr 18 10:20:51 hosting bbackupd[18990]: NOTICE: Beginning scan of local files Apr 18 10:20:51 hosting bbackupd[18990]: Opening connection to server 'bbox.netcal.com'... Apr 18 10:20:52 hosting bbackupd[18990]: Connection made, login successful Apr 18 10:20:52 hosting bbackupd[18990]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_boot.n Apr 18 10:20:52 hosting bbackupd[18990]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_home.n Apr 18 10:20:52 hosting bbackupd[18990]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_.n Apr 18 10:20:52 hosting bbackupd[18990]: Not deleting unused entries - none in list Apr 18 10:20:52 hosting bbackupd[18990]: Ignoring apparent hard link count on directory: /boot/grub, nlink=2 Apr 18 10:20:52 hosting bbackupd[18990]: Ignoring apparent hard link count on directory: /boot/lost+found, nlink=2 Apr 18 10:20:52 hosting bbackupd[18990]: Ignoring apparent hard link count on directory: /home/r1llc, nlink=15 From chris at qwirx.com Fri Apr 18 21:41:51 2014 From: chris at qwirx.com (Chris Wilson) Date: Fri, 18 Apr 2014 21:41:51 +0100 (BST) Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hi Brendon, On Fri, 18 Apr 2014, Brendon Baumgartner wrote: > Apr 18 10:01:28 hosting bbackupd[12635]: NOTICE: Box Backup Client vtrunk_3341, (c) Ben Summers and contributors 2003-2014 > Apr 18 10:01:28 hosting bbackupd[12637]: NOTICE: Starting daemon, version: trunk_3341 > Apr 18 10:01:28 hosting bbackupd[12637]: NOTICE: Using configuration file: /etc/boxbackup/bbackupd.conf > Apr 18 10:01:41 hosting bbackupctl[12658]: Using configuration file /etc/boxbackup/bbackupd.conf > Apr 18 10:01:41 hosting bbackupctl[12658]: NOTICE: state 0 Idle > Apr 18 10:01:41 hosting bbackupd[12637]: ERROR: Failed to write to socket: Broken pipe (32) > Apr 18 10:01:41 hosting bbackupd[12637]: WARNING: Exception thrown: ConnectionException(Conn_SocketWriteError) at SocketStream.cpp(313) > Apr 18 10:01:41 hosting bbackupd[12637]: NOTICE: Failed to write to command socket: Connection SocketWriteError (Probably a network issue between client and server.) Did this only happen once? It looks a bit like bbackupd tried to write something to the connected bbackupctl client, for example notifying it about a sync start or finish, just at the same time that the client was disconnecting because it already had what it wanted. This could happen very occasionally, randomly, and should not be a cause for concern, because bbackupd should ignore the error. If it happened more than once this morning, OR there is no obvious cause (e.g. a backup didn't start immediately afterwards), OR the daemon died instead of ignoring the SocketWriteError, please let me know. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Fri Apr 18 22:05:59 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Fri, 18 Apr 2014 21:05:59 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Chris- What do you think? It appears to be looping because the backup failed? I left it running for the moment. Just grabbed the latest pertinent logs. I removed some things about file exceptions where the log shows paths. They started with: "WARNING: Exception thrown: CommonException(OSFileError) (Failed to stat file:" . I didn't want the paths shown so I left it as "..." below. # grep bba messages|grep -v 'hard link' ... Apr 18 10:20:28 hosting bbackupd[18988]: NOTICE: Box Backup Client vtrunk_3341, (c) Ben Summers and contributors 2003-2014 Apr 18 10:20:28 hosting bbackupd[18990]: NOTICE: Starting daemon, version: trunk_3341 Apr 18 10:20:28 hosting bbackupd[18990]: NOTICE: Using configuration file: /etc/boxbackup/bbackupd.conf Apr 18 10:20:28 hosting bbackupd[18990]: No automatic backups, waiting for bbackupctl snapshot command Apr 18 10:20:51 hosting bbackupctl[19100]: Using configuration file /etc/boxbackup/bbackupd.conf Apr 18 10:20:51 hosting bbackupd[18990]: Incoming connection from local (UNIX socket) Apr 18 10:20:51 hosting bbackupd[18990]: Connection from command socket Apr 18 10:20:51 hosting bbackupd[18990]: Starting a backup immediately due to bbackupctl sync command Apr 18 10:20:51 hosting bbackupd[18990]: About to notify administrator about event backup-start, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-start "/etc/boxbackup/bbackupd.conf"' Apr 18 10:20:51 hosting bbackupd[18990]: NOTICE: Beginning scan of local files Apr 18 10:20:51 hosting bbackupd[18990]: Opening connection to server 'x.x.x'... Apr 18 10:20:52 hosting bbackupd[18990]: Connection made, login successful Apr 18 10:20:52 hosting bbackupd[18990]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_boot.n Apr 18 10:20:52 hosting bbackupd[18990]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_home.n Apr 18 10:20:52 hosting bbackupd[18990]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_.n Apr 18 10:20:52 hosting bbackupd[18990]: Not deleting unused entries - none in list ... Apr 18 11:14:09 hosting bbackupd[18990]: WARNING: Exception thrown: ServerException(Protocol_BadUsage) (Sending a stream with a definite size of zero is not allowed in the protocol) at Protocol.cpp:703 Apr 18 11:14:09 hosting bbackupd[18990]: WARNING: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(340) Apr 18 11:14:09 hosting bbackupd[18990]: About to notify administrator about event backup-error, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-error "/etc/boxbackup/bbackupd.conf"' Apr 18 11:14:10 hosting bbackupd[18990]: ERROR: Exception caught (Server Protocol_BadUsage 3/43), reset state and waiting to retry... Apr 18 11:14:20 hosting bbackupd[18990]: NOTICE: Finished scan of local files Apr 18 11:14:20 hosting bbackupd[18990]: NOTICE: File statistics: total file size uploaded 117131076, bytes already on server 76394380, encoded size 26155641, 7498 files uploaded, 4819 dirs created Apr 18 11:14:20 hosting bbackupd[18990]: About to notify administrator about event backup-finish, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-finish "/etc/boxbackup/bbackupd.conf"' Apr 18 11:14:20 hosting bbackupd[18990]: Last backup was not successful, next one starting at 10:22:31 Apr 18 11:14:20 hosting bbackupd[18990]: Last backup was not successful, next one starting now Apr 18 11:14:20 hosting bbackupd[18990]: About to notify administrator about event backup-start, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-start "/etc/boxbackup/bbackupd.conf"' Apr 18 11:14:20 hosting bbackupd[18990]: NOTICE: Beginning scan of local files Apr 18 11:14:20 hosting bbackupd[18990]: Opening connection to server 'x.x.x'... Apr 18 11:14:20 hosting bbackupd[18990]: Connection made, login successful Apr 18 11:14:20 hosting bbackupd[18990]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_boot.n Apr 18 11:14:20 hosting bbackupd[18990]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_home.n Apr 18 11:14:20 hosting bbackupd[18990]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_.n Apr 18 11:14:20 hosting bbackupd[18990]: Not deleting unused entries - none in list Apr 18 13:23:02 hosting bbackupd[18990]: MaximumDiffingTime reached - suspending file diff # From chris at qwirx.com Fri Apr 18 23:41:53 2014 From: chris at qwirx.com (Chris Wilson) Date: Fri, 18 Apr 2014 23:41:53 +0100 (BST) Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hi Brendon, On Fri, 18 Apr 2014, Brendon Baumgartner wrote: > What do you think? It appears to be looping because the backup failed? Yes, it appears so. > Apr 18 11:14:09 hosting bbackupd[18990]: WARNING: Exception thrown: > ServerException(Protocol_BadUsage) (Sending a stream with a definite > size of zero is not allowed in the protocol) at Protocol.cpp:703 This appears to be the problem. It looks like a bug in bbackupd. I'm not sure exactly what's causing it, but I've added some debugging code that will hopefully help us to get closer to the cause. If you're willing to help, please could you run these commands: cd boxbackup-trunk svn update ./config.status ./infrastructure/makebuildenv.pl make killall bbackupd release/bin/bbackupd -DV Look out for the error above in the logs. Just before it should be a new line: "Failed to send stream after command:" Could you tell me what it says? It should include a filename, which of course you can censor if you like, but it would be useful to know if there's anything special about that file, for example if it's a zero byte file or a socket or something. Cheers, chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Mon Apr 21 18:06:20 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Mon, 21 Apr 2014 17:06:20 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Chris- Thanks for the help so far on this. Here is what you asked for. The file is a directory. Seems okay to me. /home is mounted with these options : ext3 (rw,noatime,usrquota) root at hosting [/var/log]# ls -lR /home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000000/c51 /home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000000/c51: total 80 drwxr-xr-x 3 peter peter 4096 Apr 21 07:58 ./ drwxr-xr-x 3713 peter peter 69632 Apr 21 09:45 ../ drwxr-xr-x 2 peter peter 4096 Apr 21 07:58 1d7/ /home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000000/c51/1d7: total 12 drwxr-xr-x 2 peter peter 4096 Apr 21 07:58 ./ drwxr-xr-x 3 peter peter 4096 Apr 21 07:58 ../ -rw-r--r-- 1 peter peter 3388 Apr 21 07:58 c511d75c0797a06be2d8361b1282a262.php root at hosting [/var/log]# file /home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000000/c51 /home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000000/c51: directory root at hosting [/var/log]# Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Exception thrown: CommonException(OSFileError) (Failed to stat file: /home/peter/public_html/betterbeerblog. com/wp-content/cache/object/000000/c51: No such file or directory (2)) at BackupClientFileAttributes.cpp:379 Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Failed to read attributes of directory, cannot check for rename, assuming new: '/home/peter/public_html/bett erbeerblog.com/wp-content/cache/object/000000/c51' Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Exception thrown: ServerException(Protocol_BadUsage) (Sending a stream with a definite size of zero is not a llowed in the protocol) at Protocol.cpp:703 Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Failed to send stream after command: CreateDirectory(0xd88723,0x0,"c51"): Server Protocol_BadUsage Apr 20 22:26:21 hosting bbackupd[21890]: WARNING: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(340) Apr 20 22:26:21 hosting bbackupd[21890]: WARNING: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(340) Apr 20 22:26:21 hosting bbackupd[21890]: WARNING: Suppressing duplicate notification about backup-error Apr 20 22:26:21 hosting bbackupd[21890]: ERROR: Exception caught (Connection TLSReadFailed (Probably a network issue between client and server, or a problem with the server.) 7/34), reset state and waiting to retry... From chris at qwirx.com Mon Apr 21 18:44:38 2014 From: chris at qwirx.com (Chris Wilson) Date: Mon, 21 Apr 2014 18:44:38 +0100 (BST) Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hi Brendon, On Mon, 21 Apr 2014, Brendon Baumgartner wrote: > Thanks for the help so far on this. Here is what you asked for. The file > is a directory. Seems okay to me. Thank you for your help! I can see some parts of the problem, but not quite why it could happen, yet. First of all, this is a cache directory, and therefore there may not be any point in backing it up. You might want to exclude it to save space on the server, bandwidth and backup time. (But if you could wait until we've finished debugging this problem, that would be helpful to me :) Secondly, the error happened last night: > Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Exception thrown: CommonException(OSFileError) (Failed to stat file: /home/peter/public_html/betterbeerblog. > com/wp-content/cache/object/000000/c51: No such file or directory (2)) at BackupClientFileAttributes.cpp:379 But the directory appears to have been modified this morning, as do its contents: > root at hosting [/var/log]# ls -lR /home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000000/c51 > /home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000000/c51: > total 80 > drwxr-xr-x 3 peter peter 4096 Apr 21 07:58 ./ Is it possible that it was created this morning at 07:58? (I think you can use the stat command to find out). If so, then possibly it existed previous and was deleted during the backup run. Of course we should be able to cope with that, but it's one of those annoying edge cases which is both hard to test, and unlikely to happen in real life, which is probably why it survived ~10 years without being detected. It will also be hard for you to reproduce. Does it happen on every backup run? Does it affect different files each time? (Please search through your logs for the 'Failed to stat file' error, which has been there for a while.) > Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Failed to read > attributes of directory, cannot check for rename, assuming new: > '/home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000000/c51' This is the third part of the problem: I'm assuming that I can create a directory with no attributes, because I can't read them. I don't think that's valid, and I don't know what the client will do if it finds a directory with no attributes on the server. I might have to skip the directory completely if we can't read it's attributes. > Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Exception thrown: > ServerException(Protocol_BadUsage) (Sending a stream with a definite > size of zero is not allowed in the protocol) at Protocol.cpp:703 This is the fourth part of the problem. Because there are no attributes, we try to upload an empty stream. This is disallowed by the protocol since the beginning. I don't know exactly why, and I could remove the assertion that prevents it from happening, but I don't know whether anything else would break as a result. I've never seen this before apart from bugs in my code, so I think that for now, I should assume that this is a side effect of the bug above, and not change it. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Mon Apr 21 19:44:41 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Mon, 21 Apr 2014 18:44:41 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Chris! > Hi Brendon, > > On Mon, 21 Apr 2014, Brendon Baumgartner wrote: > > > Thanks for the help so far on this. Here is what you asked for. The > > file is a directory. Seems okay to me. > > Thank you for your help! > > I can see some parts of the problem, but not quite why it could happen, > yet. > > First of all, this is a cache directory, and therefore there may not be > any point in backing it up. You might want to exclude it to save space on > the server, bandwidth and backup time. (But if you could wait until we've > finished debugging this problem, that would be helpful to me :) Yea, I already created a task for the future to disable backing this up, but we'll leave it for now if it helps cause the problem :) > Secondly, the error happened last night: > > > Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Exception thrown: > CommonException(OSFileError) (Failed to stat file: > /home/peter/public_html/betterbeerblog. > > com/wp-content/cache/object/000000/c51: No such file or directory (2)) > > at BackupClientFileAttributes.cpp:379 > > But the directory appears to have been modified this morning, as do its > contents: > > > root at hosting [/var/log]# ls -lR > > /home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000 > > 000/c51 > > /home/peter/public_html/betterbeerblog.com/wp- > content/cache/object/000000/c51: > > total 80 > > drwxr-xr-x 3 peter peter 4096 Apr 21 07:58 ./ > > Is it possible that it was created this morning at 07:58? (I think you can > use the stat command to find out). If so, then possibly it existed > previous and was deleted during the backup run. Of course we should be > able to cope with that, but it's one of those annoying edge cases which is > both hard to test, and unlikely to happen in real life, which is probably > why it survived ~10 years without being detected. Yea, its possible it was deleted and re-created. I'm not really sure how these wordpress things/plugins work. # stat /home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000000/c51 File: `/home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000000/c51' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: fd02h/64770d Inode: 3032931 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 612/ peter) Gid: ( 609/ peter) Access: 2014-04-21 09:51:02.000000000 -0700 Modify: 2014-04-21 07:58:10.000000000 -0700 Change: 2014-04-21 07:58:10.000000000 -0700 # > > It will also be hard for you to reproduce. Does it happen on every backup > run? Does it affect different files each time? (Please search through your > logs for the 'Failed to stat file' error, which has been there for a > while.) I searched through the logs, and it only showed up once since Apr 19 11:17. There have been multiple backup cycles but the error has only appeared once. Here is a greped list of backups since that date/time: http://pastebin.com/P4jAEF5i Raw logs : http://66.160.177.101/~brendon/bb/ (files log1 and log2) Also, none of the backups complete successfully. They all end with something like this: Apr 19 19:24:57 hosting bbackupd[21890]: WARNING: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(340) Apr 19 19:24:57 hosting bbackupd[21890]: WARNING: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(340) Apr 19 19:24:57 hosting bbackupd[21890]: About to notify administrator about event backup-error, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-error "/etc/boxbackup/bbackupd.conf"' Apr 19 19:24:57 hosting bbackupd[21890]: ERROR: Exception caught (Connection TLSReadFailed (Probably a network issue between client and server, or a problem with the server.) 7/34), reset state and waiting to retry... > > > Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Failed to read > > attributes of directory, cannot check for rename, assuming new: > > '/home/peter/public_html/betterbeerblog.com/wp- > content/cache/object/000000/c51' > > This is the third part of the problem: I'm assuming that I can create a > directory with no attributes, because I can't read them. I don't think > that's valid, and I don't know what the client will do if it finds a > directory with no attributes on the server. I might have to skip the > directory completely if we can't read it's attributes. > > > Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Exception thrown: > > ServerException(Protocol_BadUsage) (Sending a stream with a definite > > size of zero is not allowed in the protocol) at Protocol.cpp:703 > > This is the fourth part of the problem. Because there are no attributes, > we try to upload an empty stream. This is disallowed by the protocol since > the beginning. I don't know exactly why, and I could remove the assertion > that prevents it from happening, but I don't know whether anything else > would break as a result. I've never seen this before apart from bugs in my > code, so I think that for now, I should assume that this is a side effect > of the bug above, and not change it. > > Cheers, Chris. > -- Let me know what's next. -BB From chris at qwirx.com Thu Apr 24 21:22:03 2014 From: chris at qwirx.com (Chris Wilson) Date: Thu, 24 Apr 2014 21:22:03 +0100 (BST) Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hi Brendon, On Mon, 21 Apr 2014, Brendon Baumgartner wrote: > Also, none of the backups complete successfully. They all end with something like this: > Apr 19 19:24:57 hosting bbackupd[21890]: WARNING: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(340) > Apr 19 19:24:57 hosting bbackupd[21890]: WARNING: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(340) > Apr 19 19:24:57 hosting bbackupd[21890]: About to notify administrator about event backup-error, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-error "/etc/boxbackup/bbackupd.conf"' > Apr 19 19:24:57 hosting bbackupd[21890]: ERROR: Exception caught (Connection TLSReadFailed (Probably a network issue between client and server, or a problem > with the server.) 7/34), reset state and waiting to retry... This looks like a server error. Is there anything in the logs on the server that might indicate what went wrong at the end of each backup? Or anything suspicious on the client side just before Conn_TLSReadFailed? >> This is the third part of the problem: I'm assuming that I can create a >> directory with no attributes, because I can't read them. I don't think >> that's valid, and I don't know what the client will do if it finds a >> directory with no attributes on the server. I might have to skip the >> directory completely if we can't read it's attributes. >> >>> Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Exception thrown: >>> ServerException(Protocol_BadUsage) (Sending a stream with a definite >>> size of zero is not allowed in the protocol) at Protocol.cpp:703 >> >> This is the fourth part of the problem. Because there are no attributes, >> we try to upload an empty stream. This is disallowed by the protocol since >> the beginning. I don't know exactly why, and I could remove the assertion >> that prevents it from happening, but I don't know whether anything else >> would break as a result. I've never seen this before apart from bugs in my >> code, so I think that for now, I should assume that this is a side effect >> of the bug above, and not change it. > > Let me know what's next. I will try and work up tests and patches for both of these bugs this weekend. Hope that's soon enough. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Thu Apr 24 23:59:02 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Thu, 24 Apr 2014 22:59:02 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: > > Hi Brendon, > > On Mon, 21 Apr 2014, Brendon Baumgartner wrote: > > > Also, none of the backups complete successfully. They all end with > something like this: > > Apr 19 19:24:57 hosting bbackupd[21890]: WARNING: Exception thrown: > ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(340) > > Apr 19 19:24:57 hosting bbackupd[21890]: WARNING: Exception thrown: > ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(340) > > Apr 19 19:24:57 hosting bbackupd[21890]: About to notify administrator > about event backup-error, running script > '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-error > "/etc/boxbackup/bbackupd.conf"' > > Apr 19 19:24:57 hosting bbackupd[21890]: ERROR: Exception caught > (Connection TLSReadFailed (Probably a network issue between client and > server, or a problem > > with the server.) 7/34), reset state and waiting to retry... > > This looks like a server error. Is there anything in the logs on the > server that might indicate what went wrong at the end of each backup? Or > anything suspicious on the client side just before Conn_TLSReadFailed? This is what I see on the server... Apr 19 19:24:57 modena bbstored client=0x00000013[22281]: WARNING: Exception thrown: BackupStoreException(StoreInfoBlockDeltaMakesValueNegative) (Failed to reduce mBlocksInCurrentFiles from 30068 by -59124) at BackupStoreInfo.cpp:468 Apr 19 19:24:57 modena bbstored client=0x00000013[22281]: NOTICE: Connection statistics for 0x00000013 (name=maxdistro-hosting): IN=1287058406 OUT=472926927 NET_IN=814131479 TOTAL=1759985333 > >> This is the third part of the problem: I'm assuming that I can create > a > >> directory with no attributes, because I can't read them. I don't > think > >> that's valid, and I don't know what the client will do if it finds a > >> directory with no attributes on the server. I might have to skip the > >> directory completely if we can't read it's attributes. > >> > >>> Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Exception thrown: > >>> ServerException(Protocol_BadUsage) (Sending a stream with a definite > >>> size of zero is not allowed in the protocol) at Protocol.cpp:703 > >> > >> This is the fourth part of the problem. Because there are no > attributes, > >> we try to upload an empty stream. This is disallowed by the protocol > since > >> the beginning. I don't know exactly why, and I could remove the > assertion > >> that prevents it from happening, but I don't know whether anything > else > >> would break as a result. I've never seen this before apart from bugs > in my > >> code, so I think that for now, I should assume that this is a side > effect > >> of the bug above, and not change it. > > > > Let me know what's next. > > I will try and work up tests and patches for both of these bugs this > weekend. Hope that's soon enough. Okay. Sounds good. bbackupd causes heavy load on one server which is why I switched to snapshot backups which is looping because of these bugs. I had to disable bbackupd for now so I basically have one computer that is vulnerable right now. -Brendon From chris at qwirx.com Sat Apr 26 21:06:54 2014 From: chris at qwirx.com (Chris Wilson) Date: Sat, 26 Apr 2014 21:06:54 +0100 (BST) Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hi Brendon, On Mon, 21 Apr 2014, Chris Wilson wrote: >> Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Failed to read attributes >> of directory, cannot check for rename, assuming new: >> '/home/peter/public_html/betterbeerblog.com/wp-content/cache/object/000000/c51' > > This is the third part of the problem: I'm assuming that I can create a > directory with no attributes, because I can't read them. I don't think that's > valid, and I don't know what the client will do if it finds a directory with > no attributes on the server. I might have to skip the directory completely if > we can't read it's attributes. I think this is the real problem. I've modified bbackupd to hopefully prevent it from happening. Please could you update this client and see if the Protocol_BadUsage error happens again? >> Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Exception thrown: >> ServerException(Protocol_BadUsage) (Sending a stream with a definite >> size of zero is not allowed in the protocol) at Protocol.cpp:703 > > This is the fourth part of the problem. Because there are no attributes, > we try to upload an empty stream. This is disallowed by the protocol > since the beginning. I don't know exactly why, and I could remove the > assertion that prevents it from happening, but I don't know whether > anything else would break as a result. I've never seen this before apart > from bugs in my code, so I think that for now, I should assume that this > is a side effect of the bug above, and not change it. I'm erring on the side of caution and not modifying this assertion for the time being. I'm hoping that the change to BackupClientFileAttributes will prevent the only known trigger of this assertion from happening. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Mon Apr 28 19:46:41 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Mon, 28 Apr 2014 18:46:41 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hey Chris- > Hi Brendon, > > On Mon, 21 Apr 2014, Chris Wilson wrote: > > >> Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Failed to read > attributes > >> of directory, cannot check for rename, assuming new: > >> '/home/peter/public_html/betterbeerblog.com/wp- > content/cache/object/000000/c51' > > > > This is the third part of the problem: I'm assuming that I can create > a > > directory with no attributes, because I can't read them. I don't think > that's > > valid, and I don't know what the client will do if it finds a > directory with > > no attributes on the server. I might have to skip the directory > completely if > > we can't read it's attributes. > > I think this is the real problem. I've modified bbackupd to hopefully > prevent it from happening. Please could you update this client and see > if > the Protocol_BadUsage error happens again? Okay, I have installed b3345 on the client #13 and let it run. > >> Apr 20 22:11:21 hosting bbackupd[21890]: WARNING: Exception thrown: > >> ServerException(Protocol_BadUsage) (Sending a stream with a definite > >> size of zero is not allowed in the protocol) at Protocol.cpp:703 > > > > This is the fourth part of the problem. Because there are no > attributes, > > we try to upload an empty stream. This is disallowed by the protocol > > since the beginning. I don't know exactly why, and I could remove the > > assertion that prevents it from happening, but I don't know whether > > anything else would break as a result. I've never seen this before > apart > > from bugs in my code, so I think that for now, I should assume that > this > > is a side effect of the bug above, and not change it. > > I'm erring on the side of caution and not modifying this assertion for > the > time being. I'm hoping that the change to BackupClientFileAttributes > will > prevent the only known trigger of this assertion from happening. Okay. Fingers crossed. I'll check it in a few days and see what's in the logs. -BB From brendon at netcal.com Mon Apr 28 20:23:32 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Mon, 28 Apr 2014 19:23:32 +0000 Subject: [Box Backup] how to properly use snapshot backups? References: Message-ID: I've got an update on this faster than I expected. It ran for a few minutes and died. I've stopped the bbackupd process on the client for now. SERVER LOG: Apr 28 11:52:43 kernel: [3629443.882942] bbstored[32076]: segfault at 20 ip 000000000046d363 sp 00007ffff63ac490 error 6 in bbstored[400000+1c5000] CLIENT LOG: Apr 28 11:40:40 hosting bbackupd[26384]: NOTICE: Box Backup Client vb3345, (c) Ben Summers and contributors 2003-2014 Apr 28 11:40:40 hosting bbackupd[26387]: NOTICE: Starting daemon, version: b3345 Apr 28 11:40:40 hosting bbackupd[26387]: NOTICE: Using configuration file: /etc/boxbackup/bbackupd.conf Apr 28 11:41:19 hosting bbackupctl[26556]: Using configuration file /etc/boxbackup/bbackupd.conf Apr 28 11:41:19 hosting bbackupd[26387]: NOTICE: Beginning scan of local files Apr 28 11:41:19 hosting bbackupd[26387]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_boot.n Apr 28 11:41:19 hosting bbackupd[26387]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_home.n Apr 28 11:41:19 hosting bbackupd[26387]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_.n Apr 28 11:52:43 hosting bbackupd[26387]: WARNING: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(340) Apr 28 11:52:43 hosting bbackupd[26387]: ERROR: Failed to upload file: : caught exception: Connection TLSReadFailed (Probably a network issue between client and server, or a problem with the server.) (7/34) Apr 28 11:52:43 hosting bbackupd[26387]: WARNING: Exception thrown: ConnectionException(Conn_TLSReadFailed) at SocketStreamTLS.cpp(340) Apr 28 11:52:44 hosting bbackupd[26387]: ERROR: Exception caught (Connection TLSReadFailed (Probably a network issue between client and server, or a problem with the server.) 7/34), reset state and waiting to retry... Apr 28 11:52:54 hosting bbackupd[26387]: NOTICE: Finished scan of local files Apr 28 11:52:54 hosting bbackupd[26387]: NOTICE: File statistics: total file size uploaded 14856531, bytes already on server 12955320, encoded size 1342770, 377 files uploaded, 0 dirs created Apr 28 11:52:54 hosting bbackupd[26387]: NOTICE: Beginning scan of local files Apr 28 11:52:54 hosting bbackupd[26387]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_boot.n Apr 28 11:52:54 hosting bbackupd[26387]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_home.n Apr 28 11:52:54 hosting bbackupd[26387]: NOTICE: Found an incomplete ID map database, deleting it to start afresh: /var/lib/box/mnt_.n From chris at qwirx.com Mon Apr 28 22:44:35 2014 From: chris at qwirx.com (Chris Wilson) Date: Mon, 28 Apr 2014 22:44:35 +0100 (BST) Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hi Brendon, On Mon, 28 Apr 2014, Brendon Baumgartner wrote: > I've got an update on this faster than I expected. It ran for a few > minutes and died. I've stopped the bbackupd process on the client for > now. > > SERVER LOG: Apr 28 11:52:43 kernel: [3629443.882942] bbstored[32076]: > segfault at 20 ip 000000000046d363 sp 00007ffff63ac490 error 6 in > bbstored[400000+1c5000] Thanks, I think I've identified something that could cause this error. However it relies on the server having a corrupt directory object, which bbstoreaccounts check fix should have repaired. Did you run bbstoreaccounts check fix on this account? If not, please DO NOT do so yet, as I'd like to check that the error is caught and fixed more gracefully in real life. However I'd like to know whether you've run it or not, and whether check (without fix) reports any errors on this account still. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Tue Apr 29 20:15:43 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Tue, 29 Apr 2014 19:15:43 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hi Chris. > Hi Brendon, > > On Mon, 28 Apr 2014, Brendon Baumgartner wrote: > > > I've got an update on this faster than I expected. It ran for a few > > minutes and died. I've stopped the bbackupd process on the client for > > now. > > > > SERVER LOG: Apr 28 11:52:43 kernel: [3629443.882942] bbstored[32076]: > > segfault at 20 ip 000000000046d363 sp 00007ffff63ac490 error 6 in > > bbstored[400000+1c5000] > > Thanks, I think I've identified something that could cause this error. > However it relies on the server having a corrupt directory object, which > bbstoreaccounts check fix should have repaired. Did you run > bbstoreaccounts check fix on this account? I ran it over the weekend (check/fix) and it fixed a LOT of errors. I just ran another check (no fix) and it found 7 errors (a longer log at the end of the msg) > If not, please DO NOT do so yet, as I'd like to check that the error is > caught and fixed more gracefully in real life. However I'd like to know > whether you've run it or not, and whether check (without fix) reports > any > errors on this account still. A snip of the check ... INFO: File ID 0xf8b837 has different container ID, probably moved INFO: File ID 0xf94801 has different container ID, probably moved INFO: File ID 0xf465bc has different container ID, probably moved INFO: Phase 3, check root... INFO: Phase 4, fix unattached objects... INFO: Phase 5, fix unrecovered inconsistencies... INFO: Phase 6, regenerate store info... INFO: Current files: 4595286, old files: 279163, deleted files: 1269960, directories: 434814 WARNING: Soft limit for account changed to ensure housekeeping doesn't delete files on next run. WARNING: Hard limit for account changed to ensure housekeeping doesn't delete files on next run. ERROR: BlocksUsed changed from 87572016 to 87572038 ERROR: BlocksInCurrentFiles changed from 55281655 to 55281659 ERROR: BlocksInOldFiles changed from 6536954 to 6536972 ERROR: BlocksInDirectories changed from 539373 to 539374 ERROR: NumOldFiles changed from 279161 to 279163 WARNING: Finished checking store account ID 0x00000013: 7 errors found WARNING: No changes to the store account have been made. WARNING: Run again with fix option to fix these errors WARNING: Block 0x7ff10dcd5010 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x7ff10af41010 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x7ff10adb8010 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x7ff10ac2f010 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x14d4b10 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x7ff10aaa6010 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x7ff10a91d010 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x7ff10a794010 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x165cca0 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x17e4cb0 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x196ccc0 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x1af4cd0 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x1c7cce0 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x1e04cf0 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x1f8cd00 freed, but not known. Error? Or allocated in startup static allocation? WARNING: Block 0x2114ed0 freed, but not known. Error? Or allocated in startup static allocation? From chris at qwirx.com Tue Apr 29 20:25:02 2014 From: chris at qwirx.com (Chris Wilson) Date: Tue, 29 Apr 2014 20:25:02 +0100 (BST) Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hi Brendon, On Tue, 29 Apr 2014, Brendon Baumgartner wrote: >>> SERVER LOG: Apr 28 11:52:43 kernel: [3629443.882942] bbstored[32076]: >>> segfault at 20 ip 000000000046d363 sp 00007ffff63ac490 error 6 in >>> bbstored[400000+1c5000] >> >> Thanks, I think I've identified something that could cause this error. I think I may have fixed that bug. Please could you update and build a new bbstored on the server and see if it fixes the problem? > I ran it over the weekend (check/fix) and it fixed a LOT of errors. I > just ran another check (no fix) and it found 7 errors (a longer log at > the end of the msg) OK thanks. This snip only lists 5 of the 7 errors: > ERROR: BlocksUsed changed from 87572016 to 87572038 > ERROR: BlocksInCurrentFiles changed from 55281655 to 55281659 > ERROR: BlocksInOldFiles changed from 6536954 to 6536972 > ERROR: BlocksInDirectories changed from 539373 to 539374 > ERROR: NumOldFiles changed from 279161 to 279163 These are just minor differences in block accounting. It looks like I haven't found all the edge cases yet. But are there some ERROR log entries before this that might give us information about the two missing errors? I haven't needed to log into your server so far, thanks for sending your bbstored :) Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Tue Apr 29 20:28:39 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Tue, 29 Apr 2014 19:28:39 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Chris- I found one more error! The rest were probably lost in my scrollback buffer not going back far enough. INFO: File ID 0xdf0f8f has different container ID, probably moved INFO: File ID 0xdf9494 has different container ID, probably moved ERROR: Directory 0xcb70ee entry for 0xff5568 has wrong size 1, should be 17 INFO: Directory ID 0xcb70ee was OK after fixing INFO: File ID 0xdb4e39 has different container ID, probably moved INFO: File ID 0xe87d1c has different container ID, probably moved I'll install the new version now. From brendon at netcal.com Tue Apr 29 20:37:38 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Tue, 29 Apr 2014 19:37:38 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Chris- > >>> SERVER LOG: Apr 28 11:52:43 kernel: [3629443.882942] > bbstored[32076]: > >>> segfault at 20 ip 000000000046d363 sp 00007ffff63ac490 error 6 in > >>> bbstored[400000+1c5000] > >> > >> Thanks, I think I've identified something that could cause this > error. > > I think I may have fixed that bug. Please could you update and build a > new > bbstored on the server and see if it fixes the problem? Okay. I installed the new (3349) version on the server. I'll start client #13 up again and see what happens. I should have something by tomorrow. -BB From james at netinertia.co.uk Wed Apr 30 20:14:54 2014 From: james at netinertia.co.uk (James O'Gorman) Date: Wed, 30 Apr 2014 20:14:54 +0100 Subject: [Box Backup] Competition, development, future direction? In-Reply-To: References: <5018668E.1080102@qustodium.net> <50376FA7.3080303@invis.net> Message-ID: <20140430191454.GV1755@netinertia.co.uk> Hi Chris, On Sat, Apr 12, 2014 at 08:58:23PM +0100, Chris Wilson wrote: > I have not heard any objections to such a relicensing. > > I am still not keen on the 2-clause BSD license, because it allows people > to make commercial products using our work without either giving credit or > contributing back to the community. > > So what I propose is to END the current mixed license state, and relicense > the ENTIRE code base under the same license, a dual license which allows > you to use it under: > > * EITHER the original Box Backup BSD license, the 4-clause BSD license > with advertising clause: > https://www.boxbackup.org/browser/box/trunk/LICENSE.txt?rev=2290 > > * OR the Box Backup GPL, which is GPLv2 with exceptions to allow linking > against OpenSSL and VSS libraries. [snip] > So if I can get explicit permission from these authors: James O'Gorman, > Matto Marjanovic, Paolo Tosca, Christophe and Arnaud, then we can change > the license as stated above. Apologies for the delay in responding. I'm fine with this, so you have my permission. Cheers. James From chris at qwirx.com Wed Apr 30 20:33:58 2014 From: chris at qwirx.com (Chris Wilson) Date: Wed, 30 Apr 2014 20:33:58 +0100 (BST) Subject: [Box Backup] Competition, development, future direction? In-Reply-To: <20140430191454.GV1755@netinertia.co.uk> References: <5018668E.1080102@qustodium.net> <50376FA7.3080303@invis.net> <20140430191454.GV1755@netinertia.co.uk> Message-ID: Hi all, On Wed, 30 Apr 2014, James O'Gorman wrote: >> So if I can get explicit permission from these authors: James O'Gorman, >> Matto Marjanovic, Paolo Tosca, Christophe and Arnaud, then we can change >> the license as stated above. > > Apologies for the delay in responding. I'm fine with this, so you have > my permission. Thanks James. Paolo has also agreed, and I've sent reminders to Matto and Arnaud (as far as I can tell, I wrote the patch for Christophe's issue, so he doesn't need to give permission). Charles, can you confirm that this license change satisfies you, and that you could contribute your code in this case? (There's not much point making the effort of changing the license again if not.) Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Wed Apr 30 22:46:33 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Wed, 30 Apr 2014 21:46:33 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Update. Client # 13 finished a backup successfully. So snapshots are working. Should we close out this thread? I looked through the bbstored logs and don't see anything obvious wrong going on with any clients. -BB > Chris- > > > >>> SERVER LOG: Apr 28 11:52:43 kernel: [3629443.882942] > > bbstored[32076]: > > >>> segfault at 20 ip 000000000046d363 sp 00007ffff63ac490 error 6 in > > >>> bbstored[400000+1c5000] > > >> > > >> Thanks, I think I've identified something that could cause this > > error. > > > > I think I may have fixed that bug. Please could you update and build a > > new bbstored on the server and see if it fixes the problem? > > > Okay. I installed the new (3349) version on the server. I'll start client > #13 up again and see what happens. I should have something by tomorrow. > > -BB From chris at qwirx.com Wed Apr 30 22:58:16 2014 From: chris at qwirx.com (Chris Wilson) Date: Wed, 30 Apr 2014 22:58:16 +0100 (BST) Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hi Brendon, On Wed, 30 Apr 2014, Brendon Baumgartner wrote: > Update. Client # 13 finished a backup successfully. So snapshots are > working. Should we close out this thread? I looked through the bbstored > logs and don't see anything obvious wrong going on with any clients. Just to check, could you have a look in your server error logs for any messages about "Missing entry for directory"? This would have caused bbstored to crash before. So if there are no errors of this type, then something else was causing the server to crash, and we haven't gotten to the bottom of it yet. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Wed Apr 30 23:08:08 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Wed, 30 Apr 2014 22:08:08 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Chris- > Hi Brendon, > > On Wed, 30 Apr 2014, Brendon Baumgartner wrote: > > > Update. Client # 13 finished a backup successfully. So snapshots are > > working. Should we close out this thread? I looked through the > > bbstored logs and don't see anything obvious wrong going on with any > clients. > > Just to check, could you have a look in your server error logs for any > messages about "Missing entry for directory"? This would have caused > bbstored to crash before. Yea, I can't find anything like that. This is all I could find like that and they are old. http://pastebin.com/6NbdYsT3 > So if there are no errors of this type, then something else was causing > the server to crash, and we haven't gotten to the bottom of it yet. > > Cheers, Chris. -Brendon From chris at qwirx.com Wed Apr 30 23:26:44 2014 From: chris at qwirx.com (Chris Wilson) Date: Wed, 30 Apr 2014 23:26:44 +0100 (BST) Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Hi Brendon, On Wed, 30 Apr 2014, Brendon Baumgartner wrote: >> On Wed, 30 Apr 2014, Brendon Baumgartner wrote: >> >>> Update. Client # 13 finished a backup successfully. So snapshots are >>> working. Should we close out this thread? I looked through the >>> bbstored logs and don't see anything obvious wrong going on with any >> clients. >> >> Just to check, could you have a look in your server error logs for any >> messages about "Missing entry for directory"? This would have caused >> bbstored to crash before. > > Yea, I can't find anything like that. This is all I could find like that and they are old. > http://pastebin.com/6NbdYsT3 OK, that's not entirely surprising because I wasn't sure how the problem that I fixed could be happening after you ran a check fix on the account. I don't think we can close this thread yet. Please can you keep an eye on your clients and look out for errors and warnings in the server logs? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From brendon at netcal.com Wed Apr 30 23:29:36 2014 From: brendon at netcal.com (Brendon Baumgartner) Date: Wed, 30 Apr 2014 22:29:36 +0000 Subject: [Box Backup] how to properly use snapshot backups? In-Reply-To: References: Message-ID: Chris- > Hi Brendon, > > On Wed, 30 Apr 2014, Brendon Baumgartner wrote: > >> On Wed, 30 Apr 2014, Brendon Baumgartner wrote: > >> > >>> Update. Client # 13 finished a backup successfully. So snapshots are > >>> working. Should we close out this thread? I looked through the > >>> bbstored logs and don't see anything obvious wrong going on with any > >> clients. > >> > >> Just to check, could you have a look in your server error logs for > any > >> messages about "Missing entry for directory"? This would have caused > >> bbstored to crash before. > > > > Yea, I can't find anything like that. This is all I could find like > that and they are old. > > http://pastebin.com/6NbdYsT3 > > OK, that's not entirely surprising because I wasn't sure how the problem > that I fixed could be happening after you ran a check fix on the > account. > > I don't think we can close this thread yet. Please can you keep an eye > on > your clients and look out for errors and warnings in the server logs? Got it. Will do. -BB