From siretart at tauware.de Mon Apr 4 07:17:47 2011 From: siretart at tauware.de (Reinhard Tartler) Date: Mon, 04 Apr 2011 08:17:47 +0200 Subject: [Box Backup] Reboot server.... "unknown error (backup-error)" In-Reply-To: <4D93AD40.4030705@shic.co.uk> (Steve's message of "Wed, 30 Mar 2011 23:22:56 +0100") References: <4D80E2B8.8080908@shic.co.uk> <4D81360F.8070807@shic.co.uk> <4D833EB3.4020002@shic.co.uk> <4D934DC5.2020406@shic.co.uk> <4D9376C7.5040901@eastlink.ca> <4D93AD40.4030705@shic.co.uk> Message-ID: <87mxk6r14k.fsf@faui44a.informatik.uni-erlangen.de> On Thu, Mar 31, 2011 at 00:22:56 (CEST), Steve wrote: > On 30/03/11 19:30, Jack Warkentin wrote: >> Before fixing, the two-digit numbers after the "K" and "S" were the >> same. This meant that the client was started before the server, since >> S20boxbackup-client alphanumerically precedes S20boxbackup-server. By >> changing the digits as I have done, the server gets started first, as >> it now precedes the client alphanumerically. Also, the client gets >> killed before the server. > > Hi Jack, > > Yes, I can confirm that the two-digit numbers after K and S are all '20' > on my system. > > This fix looks like something that would be best rolled into the package > - so that all users get the advantage, and we existing users don't have > to worry about patches when we upgrade... > > Do we know who maintains the package? That would be me. I'm installing the init scripts provided by the upstream boxbackup package, which doesn't take these interdependencies into account. -- Gruesse/greetings, Reinhard Tartler, KeyID 945348A4 From achim+box at qustodium.net Tue Apr 12 11:17:54 2011 From: achim+box at qustodium.net (Achim J. Latz) Date: Tue, 12 Apr 2011 12:17:54 +0200 Subject: [Box Backup] 2GB limitation Message-ID: <4DA426D2.4040900@qustodium.net> Hello Charles (and list): In your commit log, it sounds like you fixed the 2GB limitation: Did you (or anybody else) have a chance to verify/test this, or is this a "theoretical" fix? Thanks, Achim -- Achim J. Latz, Qustodium Internet Security achim.latz at qustodium.net ? http://www.qustodium.net Data Encryption ? Backup Automatisation ? E-Mail Protection From lists at invis.net Wed Apr 13 14:12:10 2011 From: lists at invis.net (Charles Lecklider) Date: Wed, 13 Apr 2011 14:12:10 +0100 Subject: [Box Backup] 2GB limitation In-Reply-To: <4DA426D2.4040900@qustodium.net> References: <4DA426D2.4040900@qustodium.net> Message-ID: <4DA5A12A.8020104@invis.net> On 12/04/2011 11:17, Achim J. Latz wrote: > Did you (or anybody else) have a chance to verify/test this, or is this > a "theoretical" fix? I've not tested it properly, but I did manage to backup a 4GB ISO and run a full verify without problems; so, "works for me", YMMV. Having said that, that's the only place in the code where Box deals with file sizes in Windows, so if this isn't a proper fix there's something more serious going on. -C From yanghatespam at gmail.com Sun Apr 17 07:27:40 2011 From: yanghatespam at gmail.com (Yang Zhang) Date: Sat, 16 Apr 2011 23:27:40 -0700 Subject: [Box Backup] ERROR: Failed to get extended attributes for '...': Argument list too long Message-ID: I got an error from boxbackup. syslog says: Apr 16 23:06:15 zs bbackupd[16807]: ERROR: Failed to get extended attributes for '/.../testnet/': Argument list too long (7) Apr 16 23:06:15 zs bbackupd[16807]: WARNING: Exception thrown: CommonException(OSFileError) at BackupClientFileAttributes.cpp(561) Apr 16 23:06:15 zs bbackupd[16807]: WARNING: Failed to read attributes of directory, cannot check for rename, assuming new: '/.../testnet' Apr 16 23:06:15 zs bbackupd[16807]: ERROR: Failed to get extended attributes for '/.../testnet': Argument list too long (7) Apr 16 23:06:15 zs bbackupd[16807]: WARNING: Exception thrown: CommonException(OSFileError) at BackupClientFileAttributes.cpp(561) Apr 16 23:06:15 zs bbackupd[16807]: ERROR: Exception caught (Common OSFileError (Error accessing a file. Check permissions.) 1/9), reset state and waiting to retry... Indeed, getfattr on that directory gave me the "Argument list too long" error. I don't yet understand what it means, but I'm wondering if there's any way to have boxbackup ignore the bad extended attribute and continue backing up the rest of my system. Thanks. -- Yang Zhang http://yz.mit.edu/ From chris at qwirx.com Sun Apr 17 08:02:00 2011 From: chris at qwirx.com (Chris Wilson) Date: Sun, 17 Apr 2011 09:02:00 +0200 (CAT) Subject: [Box Backup] ERROR: Failed to get extended attributes for '...': Argument list too long In-Reply-To: References: Message-ID: Hi Yang Zhang, On Sat, 16 Apr 2011, Yang Zhang wrote: > I got an error from boxbackup. syslog says: > > Apr 16 23:06:15 zs bbackupd[16807]: ERROR: Failed to get extended > attributes for '/.../testnet/': Argument list too long (7) > Apr 16 23:06:15 zs bbackupd[16807]: WARNING: Exception thrown: > CommonException(OSFileError) at BackupClientFileAttributes.cpp(561) > Apr 16 23:06:15 zs bbackupd[16807]: WARNING: Failed to read attributes > of directory, cannot check for rename, assuming new: '/.../testnet' > Apr 16 23:06:15 zs bbackupd[16807]: ERROR: Failed to get extended > attributes for '/.../testnet': Argument list too long (7) > Apr 16 23:06:15 zs bbackupd[16807]: WARNING: Exception thrown: > CommonException(OSFileError) at BackupClientFileAttributes.cpp(561) > Apr 16 23:06:15 zs bbackupd[16807]: ERROR: Exception caught (Common > OSFileError (Error accessing a file. Check permissions.) 1/9), reset > state and waiting to retry... > > Indeed, getfattr on that directory gave me the "Argument list too > long" error. I don't yet understand what it means, but I'm wondering > if there's any way to have boxbackup ignore the bad extended attribute > and continue backing up the rest of my system. Thanks. Which version of Box Backup are you running, and on what operating system? I think the "Argument list too long" error is the long version of E2BIG, which simply means that the extended attributes won't fit into the buffer provided. But we did try to measure their size first to make the buffer big enough, so this is weird. Fixing it properly will probably require me adding some extra debugging code to the trunk, and you downloading and compiling it. Would that be OK for you? Box Backup is also supposed to recover from this error rather than aborting. I'd be interested to get some backtraces for this, which you might be able to get by adding the -V option to bbackupd (perhaps combined with a logfile, by adding "LogFile = /tmp/bbackupd.log" to your bbackupd.conf file). You might be able to move past the error by excluding that directory from the backup. Cheers, Chris. From yanghatespam at gmail.com Mon Apr 18 21:42:56 2011 From: yanghatespam at gmail.com (Yang Zhang) Date: Mon, 18 Apr 2011 13:42:56 -0700 Subject: [Box Backup] ERROR: Failed to get extended attributes for '...': Argument list too long In-Reply-To: References: Message-ID: On Sun, Apr 17, 2011 at 12:02 AM, Chris Wilson wrote: > Hi Yang Zhang, > > On Sat, 16 Apr 2011, Yang Zhang wrote: > >> I got an error from boxbackup. ?syslog says: >> >> Apr 16 23:06:15 zs bbackupd[16807]: ERROR: Failed to get extended >> attributes for '/.../testnet/': Argument list too long (7) >> Apr 16 23:06:15 zs bbackupd[16807]: WARNING: Exception thrown: >> CommonException(OSFileError) at BackupClientFileAttributes.cpp(561) >> Apr 16 23:06:15 zs bbackupd[16807]: WARNING: Failed to read attributes >> of directory, cannot check for rename, assuming new: '/.../testnet' >> Apr 16 23:06:15 zs bbackupd[16807]: ERROR: Failed to get extended >> attributes for '/.../testnet': Argument list too long (7) >> Apr 16 23:06:15 zs bbackupd[16807]: WARNING: Exception thrown: >> CommonException(OSFileError) at BackupClientFileAttributes.cpp(561) >> Apr 16 23:06:15 zs bbackupd[16807]: ERROR: Exception caught (Common >> OSFileError (Error accessing a file. Check permissions.) 1/9), reset >> state and waiting to retry... >> >> Indeed, getfattr on that directory gave me the "Argument list too >> long" error. ?I don't yet understand what it means, but I'm wondering >> if there's any way to have boxbackup ignore the bad extended attribute >> and continue backing up the rest of my system. ?Thanks. > > Which version of Box Backup are you running, and on what operating system? > > I think the "Argument list too long" error is the long version of E2BIG, > which simply means that the extended attributes won't fit into the buffer > provided. But we did try to measure their size first to make the buffer big > enough, so this is weird. Fixing it properly will probably require me adding > some extra debugging code to the trunk, and you downloading and compiling > it. Would that be OK for you? > > Box Backup is also supposed to recover from this error rather than aborting. > I'd be interested to get some backtraces for this, which you might be able > to get by adding the -V option to bbackupd (perhaps combined with a logfile, > by adding "LogFile = /tmp/bbackupd.log" to your bbackupd.conf file). > > You might be able to move past the error by excluding that directory from > the backup. > > Cheers, Chris. > > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup > I'm using the one in Ubuntu 10.04 LTS, which is 0.11~rc3~r2502-2. Unfortunately I probably won't be able to repro this - I ultimately just removed that attribute with setfattr -x, after guessing that it probably doesn't contain anything important.... -- Yang Zhang http://yz.mit.edu/ From yanghatespam at gmail.com Tue Apr 19 02:49:01 2011 From: yanghatespam at gmail.com (Yang Zhang) Date: Mon, 18 Apr 2011 18:49:01 -0700 Subject: [Box Backup] Clearing out server space and TLS ConnectionExceptions Message-ID: I overallocated for boxbackup on my server: $ sudo bbstoreaccounts info 1 Account ID: 0x00000001 Last object ID: 0x97754 Used: 113358650 blocks, 432.43 GB, 94% |*************** | Old files: 6044239 blocks, 23.06 GB, 5% | | Deleted files: 5894 blocks, 23.02 MB, 0% | | Directories: 55819 blocks, 218.04 MB, 0% | | Soft limit: 115343360 blocks, 440.00 GB, 96% |*************** | Hard limit: 120061952 blocks, 458.00 GB, 100% |****************| Client store marker: 620372 $ df -h /media/pod Filesystem Size Used Avail Use% Mounted on /dev/sdb1 459G 435G 875M 100% /media/pod On the client, I've been getting the following errors for a while now. These repeat almost exactly, even the file that is being failed on. I suspect they're related to the space issue, but not sure - are they? Apr 18 17:57:30 zs bbackupd[16807]: WARNING: Exception thrown: ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(404) Apr 18 17:57:30 zs bbackupd[16807]: ERROR: Failed to upload file: /media/yang_/.../foo.zip: caught exception: Connection TLSWriteFailed (Probably a network issue between client and server.) (7/33) Apr 18 17:57:30 zs bbackupd[16807]: ERROR: SSL error while writing: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry Apr 18 17:57:30 zs bbackupd[16807]: WARNING: Exception thrown: ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(404) Apr 18 17:57:30 zs bbackupd[16807]: WARNING: Suppressing duplicate notification about backup-error Apr 18 17:57:30 zs bbackupd[16807]: ERROR: Exception caught (Connection TLSWriteFailed (Probably a network issue between client and server.) 7/33), reset state and waiting to retry... Apr 18 17:57:40 zs bbackupd[16807]: NOTICE: Finished scan of local files Apr 18 17:57:40 zs bbackupd[16807]: NOTICE: File statistics: total file size uploaded 1940512251, bytes already on server 66702, encoded size 981351356 Apr 18 17:57:40 zs bbackupd[16807]: NOTICE: Beginning scan of local files I've tried removing some items from my client's /etc/boxbackup/bbackupd.conf's BackupLocations, and I also tried removing a lot of data from some of the backup locations, but bbackupquery shows they're still on the server. Should I just use bbackupquery's "delete" command on these? (I tried looking up what this does exactly, but didn't find anything.) Also, is there any other way to have the server garbage-collect the files that it no longer needs to mirror, rather than needing to rummage through the backed-up files and manually delete them? And as for resizing the server account, is it safe to just lower the account soft/hard limits to something under the current usage? Is the current state of affairs due mainly to boxbackup's limits exceeding the actual disk space? Thanks in advance for answers to any questions. -- Yang Zhang http://yz.mit.edu/ From yanghatespam at gmail.com Sat Apr 23 06:10:00 2011 From: yanghatespam at gmail.com (Yang Zhang) Date: Fri, 22 Apr 2011 22:10:00 -0700 Subject: [Box Backup] Clearing out server space and TLS ConnectionExceptions In-Reply-To: References: Message-ID: *bump* Also, a small update: I tried just going ahead and running the following, but got an error: $ sudo bbstoreaccounts setlimit 1 430M 450M ERROR: Failed to lock the account, did not change limits. Try again later. I'm pretty stuck. On Mon, Apr 18, 2011 at 6:49 PM, Yang Zhang wrote: > I overallocated for boxbackup on my server: > > $ sudo bbstoreaccounts info 1 > ? ? ? ? ?Account ID: 0x00000001 > ? ? ?Last object ID: 0x97754 > ? ? ? ? ? ? ? ?Used: ?113358650 blocks, ?432.43 GB, ?94% |*************** | > ? ? ? ? ? Old files: ? ?6044239 blocks, ? 23.06 GB, ? 5% | ? ? ? ? ? ? ? ?| > ? ? ? Deleted files: ? ? ? 5894 blocks, ? 23.02 MB, ? 0% | ? ? ? ? ? ? ? ?| > ? ? ? ? Directories: ? ? ?55819 blocks, ?218.04 MB, ? 0% | ? ? ? ? ? ? ? ?| > ? ? ? ? ?Soft limit: ?115343360 blocks, ?440.00 GB, ?96% |*************** | > ? ? ? ? ?Hard limit: ?120061952 blocks, ?458.00 GB, 100% |****************| > ?Client store marker: 620372 > > $ df -h /media/pod > Filesystem ? ? ? ? ? ?Size ?Used Avail Use% Mounted on > /dev/sdb1 ? ? ? ? ? ? 459G ?435G ?875M 100% /media/pod > > On the client, I've been getting the following errors for a while now. > ?These repeat almost exactly, even the file that is being failed on. > I suspect they're related to the space issue, but not sure - are they? > > Apr 18 17:57:30 zs bbackupd[16807]: WARNING: Exception thrown: > ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(404) > Apr 18 17:57:30 zs bbackupd[16807]: ERROR: Failed to upload file: > /media/yang_/.../foo.zip: caught exception: Connection TLSWriteFailed > (Probably a network issue between client and server.) (7/33) > Apr 18 17:57:30 zs bbackupd[16807]: ERROR: SSL error while writing: > error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry > Apr 18 17:57:30 zs bbackupd[16807]: WARNING: Exception thrown: > ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(404) > Apr 18 17:57:30 zs bbackupd[16807]: WARNING: Suppressing duplicate > notification about backup-error > Apr 18 17:57:30 zs bbackupd[16807]: ERROR: Exception caught > (Connection TLSWriteFailed (Probably a network issue between client > and server.) 7/33), reset state and waiting to retry... > Apr 18 17:57:40 zs bbackupd[16807]: NOTICE: Finished scan of local files > Apr 18 17:57:40 zs bbackupd[16807]: NOTICE: File statistics: total > file size uploaded 1940512251, bytes already on server 66702, encoded > size 981351356 > Apr 18 17:57:40 zs bbackupd[16807]: NOTICE: Beginning scan of local files > > I've tried removing some items from my client's > /etc/boxbackup/bbackupd.conf's BackupLocations, and I also tried > removing a lot of data from some of the backup locations, but > bbackupquery shows they're still on the server. ?Should I just use > bbackupquery's "delete" command on these? ?(I tried looking up what > this does exactly, but didn't find anything.) ?Also, is there any > other way to have the server garbage-collect the files that it no > longer needs to mirror, rather than needing to rummage through the > backed-up files and manually delete them? > > And as for resizing the server account, is it safe to just lower the > account soft/hard limits to something under the current usage? ?Is the > current state of affairs due mainly to boxbackup's limits exceeding > the actual disk space? > > Thanks in advance for answers to any questions. > > -- > Yang Zhang > http://yz.mit.edu/ > -- Yang Zhang http://yz.mit.edu/ From chris at qwirx.com Sat Apr 23 13:15:15 2011 From: chris at qwirx.com (Chris Wilson) Date: Sat, 23 Apr 2011 14:15:15 +0200 (CAT) Subject: [Box Backup] Clearing out server space and TLS ConnectionExceptions In-Reply-To: References: Message-ID: Hi Yang, On Fri, 22 Apr 2011, Yang Zhang wrote: > *bump* Sorry, I thought I had sent a reply, but it didn't make it to the list. I'm suffering from terrible Internet connections in Zambia at the moment. > Also, a small update: I tried just going ahead and running the > following, but got an error: > > $ sudo bbstoreaccounts setlimit 1 430M 450M > ERROR: Failed to lock the account, did not change limits. Try again later. That probably means that a client is currently connected. If you kill the server process that's servicing the client, the lock should be removed or time out quite quickly. Failing that, stop the client. >> I overallocated for boxbackup on my server: >> >> $ sudo bbstoreaccounts info 1 >> ? ? ? ? ?Account ID: 0x00000001 >> ? ? ?Last object ID: 0x97754 >> ? ? ? ? ? ? ? ?Used: ?113358650 blocks, ?432.43 GB, ?94% |*************** | >> ? ? ? ? ? Old files: ? ?6044239 blocks, ? 23.06 GB, ? 5% | ? ? ? ? ? ? ? ?| >> ? ? ? Deleted files: ? ? ? 5894 blocks, ? 23.02 MB, ? 0% | ? ? ? ? ? ? ? ?| >> ? ? ? ? Directories: ? ? ?55819 blocks, ?218.04 MB, ? 0% | ? ? ? ? ? ? ? ?| >> ? ? ? ? ?Soft limit: ?115343360 blocks, ?440.00 GB, ?96% |*************** | >> ? ? ? ? ?Hard limit: ?120061952 blocks, ?458.00 GB, 100% |****************| >> ?Client store marker: 620372 >> >> On the client, I've been getting the following errors for a while now. >> ?These repeat almost exactly, even the file that is being failed on. >> I suspect they're related to the space issue, but not sure - are they? >> >> Apr 18 17:57:30 zs bbackupd[16807]: WARNING: Exception thrown: >> ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(404) >> Apr 18 17:57:30 zs bbackupd[16807]: ERROR: Failed to upload file: >> /media/yang_/.../foo.zip: caught exception: Connection TLSWriteFailed >> (Probably a network issue between client and server.) (7/33) It could be, but without logs from the server it's hard to tell. This is a generic disconnection, probably due to an error on the server side, which was not caught and handled gracefully. If you can get server logs, that would help to identify the error and handle it more gracefully in future. But you're probably right that it's an error due to running out of disk space. It seems that the client was trying to upload foo.zip when it failed, and had already written 1.9 GB of data to the server, even though there's only 875 MB free on that disk. I can imagine that that would cause an ENOSPC error from the OS. It could also be a network issue, as the message says. >> I've tried removing some items from my client's >> /etc/boxbackup/bbackupd.conf's BackupLocations, and I also tried >> removing a lot of data from some of the backup locations, but >> bbackupquery shows they're still on the server. Are they there, but just marked as deleted? If they're not deleted yet, it's probably because the client still thinks that it has space left to fill (because it hasn't reached the soft limit yet), so it keeps uploading. I think you need to reduce the soft and hard limits. >>?Should I just use bbackupquery's "delete" command on these? ?(I tried >> looking up what this does exactly, but didn't find anything.) That would also mark the files as deleted. but the client will upload them again when it can, so it's probably not the best solution. >>?Also, is there any >> other way to have the server garbage-collect the files that it no >> longer needs to mirror, rather than needing to rummage through the >> backed-up files and manually delete them? The server doesn't really know anything about the files except the flags set by the client. If the client doesn't need a file any more then it should mark it as deleted. The housekeeping process will remove and delete files until the account is below the soft limit, which it currently is already, so unless you reduce the soft limit it won't do much. There is an undocumented command from client to server to force delete files right now, but I don't think it's possible to use it without changing the code. >> And as for resizing the server account, is it safe to just lower the >> account soft/hard limits to something under the current usage? ?Is the >> current state of affairs due mainly to boxbackup's limits exceeding the >> actual disk space? I think so, yes. Cheers, Chris. From yanghatespam at gmail.com Mon Apr 25 19:49:09 2011 From: yanghatespam at gmail.com (Yang Zhang) Date: Mon, 25 Apr 2011 11:49:09 -0700 Subject: [Box Backup] Clearing out server space and TLS ConnectionExceptions In-Reply-To: References: Message-ID: OK, I lowered the soft and hard limits, and after letting things run for a day, I now see an erroneous percentage being reported by bbstoreaccounts: $ sudo bbstoreaccounts info 1 Account ID: 0x00000001 Last object ID: 0x984fe Used: 84864499 blocks, 323.73 GB, 73667% |****************| Old files: 0 blocks, 0.00 B, 0% | | Deleted files: 0 blocks, 0.00 B, 0% | | Directories: 49952 blocks, 195.12 MB, 43% |****** | Soft limit: 110080 blocks, 430.00 MB, 95% |*************** | Hard limit: 115200 blocks, 450.00 MB, 100% |****************| Client store marker: 623870 There actually *is* plenty of free space left on the volume that I'm using (exclusively/dedicated) to store backups on: $ df -h /media/pod Filesystem Size Used Avail Use% Mounted on /dev/sdb1 459G 324G 112G 75% /media/pod Despite the apparent free space, on the client I see the following in /var/log/syslog: Apr 25 11:33:12 zs bbackupd[14805]: NOTICE: Beginning scan of local files Apr 25 11:33:13 zs bbackupd[14805]: WARNING: Exceeded storage hard-limit on server, not uploading changes to files Apr 25 11:33:13 zs bbackupd[14805]: WARNING: Skipped file: server is full: /home/yang/.git-achievements-action.log Apr 25 11:33:13 zs bbackupd[14805]: WARNING: Skipped file: server is full: /home/yang/.viminfo ... Any ideas? Thanks. On Sat, Apr 23, 2011 at 5:15 AM, Chris Wilson wrote: > Hi Yang, > > On Fri, 22 Apr 2011, Yang Zhang wrote: > >> *bump* > > Sorry, I thought I had sent a reply, but it didn't make it to the list. I'm > suffering from terrible Internet connections in Zambia at the moment. > >> Also, a small update: ?I tried just going ahead and running the >> following, but got an error: >> >> $ sudo bbstoreaccounts setlimit 1 430M 450M >> ERROR: ? Failed to lock the account, did not change limits. Try again >> later. > > That probably means that a client is currently connected. If you kill the > server process that's servicing the client, the lock should be removed or > time out quite quickly. Failing that, stop the client. > >>> I overallocated for boxbackup on my server: >>> >>> $ sudo bbstoreaccounts info 1 >>> ? ? ? ? ?Account ID: 0x00000001 >>> ? ? ?Last object ID: 0x97754 >>> ? ? ? ? ? ? ? ?Used: ?113358650 blocks, ?432.43 GB, ?94% |*************** >>> | >>> ? ? ? ? ? Old files: ? ?6044239 blocks, ? 23.06 GB, ? 5% | >>> ?| >>> ? ? ? Deleted files: ? ? ? 5894 blocks, ? 23.02 MB, ? 0% | >>> ?| >>> ? ? ? ? Directories: ? ? ?55819 blocks, ?218.04 MB, ? 0% | >>> ?| >>> ? ? ? ? ?Soft limit: ?115343360 blocks, ?440.00 GB, ?96% |*************** >>> | >>> ? ? ? ? ?Hard limit: ?120061952 blocks, ?458.00 GB, 100% >>> |****************| >>> ?Client store marker: 620372 >>> >>> On the client, I've been getting the following errors for a while now. >>> ?These repeat almost exactly, even the file that is being failed on. >>> I suspect they're related to the space issue, but not sure - are they? >>> >>> Apr 18 17:57:30 zs bbackupd[16807]: WARNING: Exception thrown: >>> ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(404) >>> Apr 18 17:57:30 zs bbackupd[16807]: ERROR: Failed to upload file: >>> /media/yang_/.../foo.zip: caught exception: Connection TLSWriteFailed >>> (Probably a network issue between client and server.) (7/33) > > It could be, but without logs from the server it's hard to tell. This is a > generic disconnection, probably due to an error on the server side, which > was not caught and handled gracefully. If you can get server logs, that > would help to identify the error and handle it more gracefully in future. > > But you're probably right that it's an error due to running out of disk > space. It seems that the client was trying to upload foo.zip when it failed, > and had already written 1.9 GB of data to the server, even though there's > only 875 MB free on that disk. I can imagine that that would cause an ENOSPC > error from the OS. > > It could also be a network issue, as the message says. > >>> I've tried removing some items from my client's >>> /etc/boxbackup/bbackupd.conf's BackupLocations, and I also tried >>> removing a lot of data from some of the backup locations, but >>> bbackupquery shows they're still on the server. > > Are they there, but just marked as deleted? > > If they're not deleted yet, it's probably because the client still thinks > that it has space left to fill (because it hasn't reached the soft limit > yet), so it keeps uploading. > > I think you need to reduce the soft and hard limits. > >>> ?Should I just use bbackupquery's "delete" command on these? ?(I tried >>> looking up what this does exactly, but didn't find anything.) > > That would also mark the files as deleted. but the client will upload them > again when it can, so it's probably not the best solution. > >>> ?Also, is there any >>> other way to have the server garbage-collect the files that it no >>> longer needs to mirror, rather than needing to rummage through the >>> backed-up files and manually delete them? > > The server doesn't really know anything about the files except the flags set > by the client. If the client doesn't need a file any more then it should > mark it as deleted. The housekeeping process will remove and delete files > until the account is below the soft limit, which it currently is already, so > unless you reduce the soft limit it won't do much. > > There is an undocumented command from client to server to force delete files > right now, but I don't think it's possible to use it without changing the > code. > >>> And as for resizing the server account, is it safe to just lower the >>> account soft/hard limits to something under the current usage? ?Is the >>> current state of affairs due mainly to boxbackup's limits exceeding the >>> actual disk space? > > I think so, yes. > > Cheers, Chris. -- Yang Zhang http://yz.mit.edu/ From boxbackup at hostingbrothers.nl Tue Apr 26 10:12:08 2011 From: boxbackup at hostingbrothers.nl (Roy) Date: Tue, 26 Apr 2011 11:12:08 +0200 Subject: [Box Backup] Clearing out server space and TLS ConnectionExceptions In-Reply-To: References: Message-ID: <4DB68C68.7030407@hostingbrothers.nl> Hi Yang Zhang, Op 25-4-2011 20:49, Yang Zhang schreef: > OK, I lowered the soft and hard limits, and after letting things run > for a day, I now see an erroneous percentage being reported by > bbstoreaccounts: > > $ sudo bbstoreaccounts info 1 > Account ID: 0x00000001 > Last object ID: 0x984fe > Used: 84864499 blocks, 323.73 GB, 73667% |****************| > Old files: 0 blocks, 0.00 B, 0% | | > Deleted files: 0 blocks, 0.00 B, 0% | | > Directories: 49952 blocks, 195.12 MB, 43% |****** | > Soft limit: 110080 blocks, 430.00 MB, 95% |*************** | > Hard limit: 115200 blocks, 450.00 MB, 100% |****************| > Client store marker: 623870 You have used more than the hard and soft limit, so that's why the percentage is that high. It is not about the disk it is stored on. So the percentage is correct. > There actually *is* plenty of free space left on the volume that I'm > using (exclusively/dedicated) to store backups on: > > $ df -h /media/pod > Filesystem Size Used Avail Use% Mounted on > /dev/sdb1 459G 324G 112G 75% /media/pod > > Despite the apparent free space, on the client I see the following in > /var/log/syslog: > > Apr 25 11:33:12 zs bbackupd[14805]: NOTICE: Beginning scan of local files > Apr 25 11:33:13 zs bbackupd[14805]: WARNING: Exceeded storage > hard-limit on server, not uploading changes to files > Apr 25 11:33:13 zs bbackupd[14805]: WARNING: Skipped file: server is > full: /home/yang/.git-achievements-action.log > Apr 25 11:33:13 zs bbackupd[14805]: WARNING: Skipped file: server is > full: /home/yang/.viminfo > ... > > Any ideas? Thanks. Here it says you are over your hard-limit. Thus you cannot store anymore. You have to enlarge the hard-limit. Regards, Roy From yanghatespam at gmail.com Wed Apr 27 04:13:54 2011 From: yanghatespam at gmail.com (Yang Zhang) Date: Tue, 26 Apr 2011 20:13:54 -0700 Subject: [Box Backup] "WARNING: Found conflicting parent ID for file ID" Message-ID: I'm seeing the following on my (Ubuntu 10.04) client using the Ubuntu package of boxbackup-client: Apr 26 19:00:35 zs bbackupd[14805]: NOTICE: Beginning scan of local files Apr 26 19:00:35 zs bbackupd[14805]: NOTICE: Deleting unused locations from store root... Apr 26 19:00:36 zs bbackupd[14805]: WARNING: Failed to access file: /home/yang/.gvfs: Permission denied Apr 26 19:02:21 zs bbackupd[14805]: WARNING: Found conflicting parent ID for file ID 5508939 (/home/yang/.local/share/Trash/files/pod2/mockup/.git/objects/pack/pack-75d4956d5386619cc4fe7dc1160e500cad5486af.pack): expected 226184 but found 229226 (same directory used in two different locations?) Apr 26 19:04:39 zs bbackupd[14805]: WARNING: Suppressing duplicate notification about read-error Apr 26 19:04:39 zs bbackupd[14805]: NOTICE: Finished scan of local files Apr 26 19:04:39 zs bbackupd[14805]: NOTICE: File statistics: total file size uploaded 1232707, bytes already on server 0, encoded size 1050451 ... Apr 26 20:00:37 zs bbackupd[14805]: NOTICE: Beginning scan of local files Apr 26 20:00:38 zs bbackupd[14805]: WARNING: Failed to access file: /home/yang/.gvfs: Permission denied Apr 26 20:00:42 zs bbackupd[14805]: WARNING: Found conflicting parent ID for file ID 5776366 (/home/yang/.cache/google-chrome/Default/Cache/f_0104c7): expected 3896 but found 218341 (same directory used in two different locations?) Apr 26 20:00:42 zs bbackupd[14805]: WARNING: Found conflicting parent ID for file ID 5776280 (/home/yang/.cache/google-chrome/Default/Cache/f_0104c6): expected 3896 but found 218341 (same directory used in two different locations?) Apr 26 20:02:44 zs bbackupd[14805]: WARNING: Found conflicting parent ID for file ID 5508939 (/home/yang/.local/share/Trash/files/pod2/mockup/.git/objects/pack/pack-75d4956d5386619cc4fe7dc1160e500cad5486af.pack): expected 226184 but found 229226 (same directory used in two different locations?) Apr 26 20:05:13 zs bbackupd[14805]: WARNING: Suppressing duplicate notification about read-error Apr 26 20:05:13 zs bbackupd[14805]: NOTICE: Finished scan of local files Apr 26 20:05:13 zs bbackupd[14805]: NOTICE: File statistics: total file size uploaded 7109008, bytes already on server 0, encoded size 6249330 I'm wondering if I need to be concerned about this. I found this thread: http://lists.boxbackup.org/pipermail/boxbackup/2009-February/004972.html but it never quite got to the point of explaining the above (unless I'm missing something). The set of files that I get a warning on changes from run to run. Here's what `stat` says about the above files: $ stat /home/yang/.cache/google-chrome/Default/Cache/f_0104c7 File: `/home/yang/.cache/google-chrome/Default/Cache/f_0104c7' Size: 77281 Blocks: 152 IO Block: 4096 regular file Device: 801h/2049d Inode: 5776366 Links: 1 Access: (0600/-rw-------) Uid: ( 1000/ yang) Gid: ( 1000/ yang) Access: 2011-04-26 19:47:19.282771854 -0700 Modify: 2011-04-26 19:46:41.952771378 -0700 Change: 2011-04-26 19:46:41.952771378 -0700 $ stat /home/yang/.cache/google-chrome/Default/Cache/f_0104c6 File: `/home/yang/.cache/google-chrome/Default/Cache/f_0104c6' Size: 66168 Blocks: 136 IO Block: 4096 regular file Device: 801h/2049d Inode: 5776280 Links: 1 Access: (0600/-rw-------) Uid: ( 1000/ yang) Gid: ( 1000/ yang) Access: 2011-04-26 19:47:19.252771798 -0700 Modify: 2011-04-26 19:46:41.452147412 -0700 Change: 2011-04-26 19:46:41.452147412 -0700 $ stat /home/yang/.local/share/Trash/files/pod2/mockup/.git/objects/pack/pack-75d4956d5386619cc4fe7dc1160e500cad5486af.pack File: `/home/yang/.local/share/Trash/files/pod2/mockup/.git/objects/pack/pack-75d4956d5386619cc4fe7dc1160e500cad5486af.pack' Size: 690831 Blocks: 1352 IO Block: 4096 regular file Device: 801h/2049d Inode: 5508939 Links: 2 Access: (0664/-rw-rw-r--) Uid: ( 1000/ yang) Gid: ( 1003/ pod) Access: 2011-04-17 22:57:26.284246154 -0700 Modify: 2010-05-14 22:51:22.042823571 -0700 Change: 2010-05-18 15:46:22.311198496 -0700 Thanks in advance for any answers. From yanghatespam at gmail.com Wed Apr 27 04:15:42 2011 From: yanghatespam at gmail.com (Yang Zhang) Date: Tue, 26 Apr 2011 20:15:42 -0700 Subject: [Box Backup] Clearing out server space and TLS ConnectionExceptions In-Reply-To: <4DB68C68.7030407@hostingbrothers.nl> References: <4DB68C68.7030407@hostingbrothers.nl> Message-ID: On Tue, Apr 26, 2011 at 2:12 AM, Roy wrote: > Hi Yang Zhang, > > Op 25-4-2011 20:49, Yang Zhang schreef: >> >> OK, I lowered the soft and hard limits, and after letting things run >> for a day, I now see an erroneous percentage being reported by >> bbstoreaccounts: >> >> $ sudo bbstoreaccounts info 1 >> ? ? ? ? ? Account ID: 0x00000001 >> ? ? ? Last object ID: 0x984fe >> ? ? ? ? ? ? ? ? Used: ? 84864499 blocks, ?323.73 GB, 73667% >> |****************| >> ? ? ? ? ? ?Old files: ? ? ? ? ?0 blocks, ? ?0.00 ?B, ? 0% | >> ?| >> ? ? ? ?Deleted files: ? ? ? ? ?0 blocks, ? ?0.00 ?B, ? 0% | >> ?| >> ? ? ? ? ?Directories: ? ? ?49952 blocks, ?195.12 MB, ?43% |****** >> ?| >> ? ? ? ? ? Soft limit: ? ? 110080 blocks, ?430.00 MB, ?95% |*************** >> | >> ? ? ? ? ? Hard limit: ? ? 115200 blocks, ?450.00 MB, 100% >> |****************| >> ?Client store marker: 623870 > > You have used more than the hard and soft limit, so that's why the > percentage is that high. It is not about the disk it is stored on. So the > percentage is correct. Thanks, I mistakenly entered M instead of G (and never noticed). Things seem to be running smoothly now. > >> There actually *is* plenty of free space left on the volume that I'm >> using (exclusively/dedicated) to store backups on: >> >> $ df -h /media/pod >> Filesystem ? ? ? ? ? ?Size ?Used Avail Use% Mounted on >> /dev/sdb1 ? ? ? ? ? ? 459G ?324G ?112G ?75% /media/pod >> >> Despite the apparent free space, on the client I see the following in >> /var/log/syslog: >> >> Apr 25 11:33:12 zs bbackupd[14805]: NOTICE: Beginning scan of local files >> Apr 25 11:33:13 zs bbackupd[14805]: WARNING: Exceeded storage >> hard-limit on server, not uploading changes to files >> Apr 25 11:33:13 zs bbackupd[14805]: WARNING: Skipped file: server is >> full: /home/yang/.git-achievements-action.log >> Apr 25 11:33:13 zs bbackupd[14805]: WARNING: Skipped file: server is >> full: /home/yang/.viminfo >> ... >> >> Any ideas? ?Thanks. > > Here it says you are over your hard-limit. Thus you cannot store anymore. > You have to enlarge the hard-limit. > > Regards, > > Roy > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup > -- Yang Zhang http://yz.mit.edu/ From chris at qwirx.com Wed Apr 27 09:13:33 2011 From: chris at qwirx.com (Chris Wilson) Date: Wed, 27 Apr 2011 10:13:33 +0200 (CAT) Subject: [Box Backup] "WARNING: Found conflicting parent ID for file ID" In-Reply-To: References: Message-ID: Hi Yang, On Tue, 26 Apr 2011, Yang Zhang wrote: > I'm seeing the following on my (Ubuntu 10.04) client using the Ubuntu > package of boxbackup-client: > > Apr 26 19:02:21 zs bbackupd[14805]: WARNING: Found conflicting parent ID > for file ID 5508939 > (/home/yang/.local/share/Trash/files/pod2/mockup/.git/objects/pack/pack-75d4956d5386619cc4fe7dc1160e500cad5486af.pack): > expected 226184 but found 229226 (same directory used in two different > locations?) [...] > I'm wondering if I need to be concerned about this. Yes, because it's likely that Box Backup will "detect" a rename that never happened, and start moving files around in your backup. The data will still be there, but it might be difficult for you to find it when it comes to restore. To find out if this has affected you already, please run a test restore. It's a good idea to run these regularly. > I found this thread: > > http://lists.boxbackup.org/pipermail/boxbackup/2009-February/004972.html > > but it never quite got to the point of explaining the above (unless I'm > missing something). The most common cause is that you are backing up the same directory in more than one backup location. This can happen even if it's not obvious, for example if you use bind mounts (mount --bind) or mount the same filesystem twice on Linux (which does the same thing, transparently). The second most common cause is probably that there's something weird about the filesystem that you're backing up, and it has duplicate or changing inode numbers. This can happen with smbfs and various virtual/encrypting filesystems for example. Do either of these apply in your case? Can we see your configuration file? I notice that some of the files you're backing up have hard links (e.g. the git repositories). This is currently unsupported with Box Backup and will produce this problem. I'm not sure whether to deliberately exclude all files with multiple hard links with a big loud warning message, or allow people to do it in the knowledge that the results may well be incorrect. I think I'll just add a warning for now. Cheers, Chris.