[Box Backup] Clearing out server space and TLS ConnectionExceptions

[Chris Wilson]

Hi Yang,

On Fri, 22 Apr 2011, Yang Zhang wrote:

> *bump*

Sorry, I thought I had sent a reply, but it didn't make it to the list. 
I'm suffering from terrible Internet connections in Zambia at the moment.

> Also, a small update:  I tried just going ahead and running the
> following, but got an error:
>
> $ sudo bbstoreaccounts setlimit 1 430M 450M
> ERROR:   Failed to lock the account, did not change limits. Try again later.

That probably means that a client is currently connected. If you kill the 
server process that's servicing the client, the lock should be removed or 
time out quite quickly. Failing that, stop the client.

>> I overallocated for boxbackup on my server:
>>
>> $ sudo bbstoreaccounts info 1
>>          Account ID: 0x00000001
>>      Last object ID: 0x97754
>>                Used:  113358650 blocks,  432.43 GB,  94% |*************** |
>>           Old files:    6044239 blocks,   23.06 GB,   5% |                |
>>       Deleted files:       5894 blocks,   23.02 MB,   0% |                |
>>         Directories:      55819 blocks,  218.04 MB,   0% |                |
>>          Soft limit:  115343360 blocks,  440.00 GB,  96% |*************** |
>>          Hard limit:  120061952 blocks,  458.00 GB, 100% |****************|
>>  Client store marker: 620372
>>
>> On the client, I've been getting the following errors for a while now.
>>  These repeat almost exactly, even the file that is being failed on.
>> I suspect they're related to the space issue, but not sure - are they?
>>
>> Apr 18 17:57:30 zs bbackupd[16807]: WARNING: Exception thrown:
>> ConnectionException(Conn_TLSWriteFailed) at SocketStreamTLS.cpp(404)
>> Apr 18 17:57:30 zs bbackupd[16807]: ERROR: Failed to upload file:
>> /media/yang_/.../foo.zip: caught exception: Connection TLSWriteFailed
>> (Probably a network issue between client and server.) (7/33)

It could be, but without logs from the server it's hard to tell. This is a 
generic disconnection, probably due to an error on the server side, which 
was not caught and handled gracefully. If you can get server logs, that 
would help to identify the error and handle it more gracefully in future.

But you're probably right that it's an error due to running out of disk 
space. It seems that the client was trying to upload foo.zip when it 
failed, and had already written 1.9 GB of data to the server, even though 
there's only 875 MB free on that disk. I can imagine that that would cause 
an ENOSPC error from the OS.

It could also be a network issue, as the message says.

>> I've tried removing some items from my client's
>> /etc/boxbackup/bbackupd.conf's BackupLocations, and I also tried
>> removing a lot of data from some of the backup locations, but
>> bbackupquery shows they're still on the server.

Are they there, but just marked as deleted?

If they're not deleted yet, it's probably because the client still thinks 
that it has space left to fill (because it hasn't reached the soft limit 
yet), so it keeps uploading.

I think you need to reduce the soft and hard limits.

>> Should I just use bbackupquery's "delete" command on these?  (I tried
>> looking up what this does exactly, but didn't find anything.)

That would also mark the files as deleted. but the client will upload them 
again when it can, so it's probably not the best solution.

>> Also, is there any
>> other way to have the server garbage-collect the files that it no
>> longer needs to mirror, rather than needing to rummage through the
>> backed-up files and manually delete them?

The server doesn't really know anything about the files except the flags 
set by the client. If the client doesn't need a file any more then it 
should mark it as deleted. The housekeeping process will remove and delete 
files until the account is below the soft limit, which it currently is 
already, so unless you reduce the soft limit it won't do much.

There is an undocumented command from client to server to force delete 
files right now, but I don't think it's possible to use it without 
changing the code.

>> And as for resizing the server account, is it safe to just lower the 
>> account soft/hard limits to something under the current usage?  Is the 
>> current state of affairs due mainly to boxbackup's limits exceeding the 
>> actual disk space?

I think so, yes.

Cheers, Chris.