[Box Backup] Feature request - for lousy connections

[Chris Wilson]

Hi Florian,

On Tue, 30 Mar 2010, Florian Eyben wrote:

> thanks for your quick replay. I get your point there. So changing the TLS 
> layer is tabu, I understand.

Not entirely off limits, but certainly something that would require a 
massive amount of thought and care to achieve.

> Well, let me provide more details on the problem:
>
> Let's say, I have three directories I want to sync:
> /dir1   (1000 files)
> /dir2   (2000 files)
> /dir3   (100 files)
> These are listed in this order in the bbackupd.conf file. (There are more 
> files actually, but this is only a quantitative example)
>
> I change a file in dir2 and add a file to dir3, and start the sync. Then, 
> first all files in dir1 are scanned, no changes there. This scan, however 
> takes a while, because server communication is required, I guess. I cannot 
> give you exact numbers right now for my setting, but it's approx. 5 hours 
> scan time, if there are no modifications, i.e. no data to transmit. I see an 
> entry, transmitted, already on the server, etc. in the syslog at the end of 
> the sync. Then I can also see how long the sync took. If that information is 
> useful I can extract it from the next backup sync and post it.
> Let's say, dir2 gets synced, my changed file transferred, and then - be it 
> the isp (all isps in germany drop the connection after 24h, except the 
> expensive bussiness lines), or some other reason - the connection drops. 
> "boxbackupctl sync-and-wait" will now exit with "sync finished"  (maybe this 
> should be changed to "sync incomplete...?". I know it's possible to use the 
> notify-admin.sh here, so this issue is minor). When I restart the sync, the 
> sync will start at /dir1 again. Ok, it doesn't transmit the changed file in 
> /dir2 again, because it detects no change there - fair point. However, the 
> scanning takes a while, and if the connection drops again before reaching 
> /dir3, dir3 will never get synced!

Then the time taken to scan is the problem, not the downloading and 
updating of file or directory data.

Scanning a few thousand files should only take a few seconds. Even a 
hundred thousand should take tens of minutes. I'm curious why it takes 
five hours.

Could you run some software to graph the bandwidth use on your internet 
connection, e.g. mrtg or cacti or vnstat or ntop? It would also be 
interesting to see detailed logs of what bbackupd is doing when it takes 
five hours to sync. Perhaps you could run "bbackupd -kVT" from the command 
line (or set LogFile=/tmp/log.txt and run the daemon with -V) and send me 
the logs of a single sync run by private email?

Cheers, Chris.
-- 
_ ___ __     _
  / __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |