[Box Backup] Feature request - for lousy connections

[Florian Eyben]

Hi Chris,

thanks for your quick replay. I get your point there. So changing the 
TLS layer is tabu, I understand.
Well, let me provide more details on the problem:

Let's say, I have three directories I want to sync:
/dir1   (1000 files)
/dir2   (2000 files)
/dir3   (100 files)
These are listed in this order in the bbackupd.conf file. (There are 
more files actually, but this is only a quantitative example)

I change a file in dir2 and add a file to dir3, and start the sync. 
Then, first all files in dir1 are scanned, no changes there. This scan, 
however takes a while, because server communication is required, I 
guess. I cannot give you exact numbers right now for my setting, but 
it's approx. 5 hours scan time, if there are no modifications, i.e. no 
data to transmit. I see an entry, transmitted, already on the server, 
etc. in the syslog at the end of the sync. Then I can also see how long 
the sync took. If that information is useful I can extract it from the 
next backup sync and post it.
Let's say, dir2 gets synced, my changed file transferred, and then - be 
it the isp (all isps in germany drop the connection after 24h, except 
the expensive bussiness lines), or some other reason - the connection 
drops. "boxbackupctl sync-and-wait" will now exit with "sync finished"  
(maybe this should be changed to "sync incomplete...?". I know it's 
possible to use the notify-admin.sh here, so this issue is minor). When 
I restart the sync, the sync will start at /dir1 again. Ok, it doesn't 
transmit the changed file in /dir2 again, because it detects no change 
there - fair point. However, the scanning takes a while, and if the 
connection drops again before reaching /dir3, dir3 will never get synced!
This is something I observed in my actual setup. The sync ran from 2-9am 
every day. stopping at 9 (not the isp..., but some connection failure). 
It did sync some files, because I added some files to (in my example) 
dir2. However, for the last 5 days it seemed to try to upload everything 
in dir2 without syncing my files in dir3.
Now - in principle this is ok, because all data is equally valueable, 
however if indeed the scanning of dir1 and dir2 take a few hours, then 
this time is wasted, if I know the last sync left off somewhere in the 
middle of dir2, then I could start resuming there - ignoring changes in 
/dir1 of course, because the changed files in /dir3 are older and thus 
should be added to the backup with a higher priority.

I am running the debian lenny 0.11rc2 package on both the server and the 
client.

Thanks again for the help!
Cheers,
Florian


Chris Wilson schrieb:
> Hi Florian,
>
> On Tue, 30 Mar 2010, Florian Eyben wrote:
>
>> I have noticed that a resume function exists for the restore command, 
>> but not for the compare or sync commands. I am on a DSL line which 
>> disconnects every 24 hours (or sometimes more often). Thus, I 
>> (re)start the sync after a disconnect, however it starts to sync from 
>> the beginning, which causes a few hours of overhead in my case (250 
>> GB on a slow CPU to save power). The same is true for comparing 
>> (quick compare).
>
> Backup should automatically resume more or less from where it left 
> off. It will need to download remote directory listings, but files 
> already completely uploaded should not be uploaded again.
>
>> During DSL disconnect the connection is down for approx. 1-2 minutes.
>> So my solution would be: Instead of detecting a communication error 
>> TLSWriteError, or similar, and terminating the operation, wouldn't it 
>> be easy to wait for a connection resume for a configurable timeout 
>> and then resuming the connection? This might be transparently 
>> implemented on the TLS connection layer, thus the higher levels won't 
>> even notice it... Do you think this is possible, or are there some 
>> issues why it might not work as proposed?
>
> I don't think OpenSSL supports this at the TLS layer (perhaps using 
> DTLS it does?), especially as the IP address might change. It would 
> also open you to compromise if an attacker (e.g. someone in the same 
> office) could kick your computer offline and then connect to the 
> server and resume the session with your login and keys. So this would 
> have to use a session key to log into the server again, which opens up 
> more vulnerabilities.
>
> The application also does not know, without checking on the server, 
> how much data was successfully received and saved there, because it 
> did not receive confirmation. So it has to check the server again anyway.
>
> I think this would be difficult to implement, and would add security 
> vulnerabilities and increase risk of bugs with consequent risk of data 
> loss through faulty backups. There is also a workaround in that you 
> could switch to an ISP that doesn't suck. So I'm reluctant to 
> implement it as proposed.
>
> If there is a shortcut, such as caching directories locally with 
> checksums, or if there is a bug that causes more data to be uploaded 
> than needs to be, then we could look at that as a change with less 
> potential impact.
>
> I'd also like to see the actual bug demonstrated before tackling it, 
> with bandwidth numbers and estimated savings.
>
> It would be reasonable for a compare operation to support resuming, I 
> agree.
>
> Cheers, Chris.