[Box Backup] Feature request - for lousy connections

[Florian Eyben]

Hi Chris,

Chris Wilson schrieb:
> Hi Florian,
>
> On Tue, 30 Mar 2010, Florian Eyben wrote:
>
>> thanks for your quick replay. I get your point there. So changing the 
>> TLS layer is tabu, I understand.
>
> Not entirely off limits, but certainly something that would require a 
> massive amount of thought and care to achieve.
>
>> Well, let me provide more details on the problem:
>>
>> Let's say, I have three directories I want to sync:
>> /dir1   (1000 files)
>> /dir2   (2000 files)
>> /dir3   (100 files)
>> These are listed in this order in the bbackupd.conf file. (There are 
>> more files actually, but this is only a quantitative example)
>>
>> I change a file in dir2 and add a file to dir3, and start the sync. 
>> Then, first all files in dir1 are scanned, no changes there. This 
>> scan, however takes a while, because server communication is 
>> required, I guess. I cannot give you exact numbers right now for my 
>> setting, but it's approx. 5 hours scan time, if there are no 
>> modifications, i.e. no data to transmit. I see an entry, transmitted, 
>> already on the server, etc. in the syslog at the end of the sync. 
>> Then I can also see how long the sync took. If that information is 
>> useful I can extract it from the next backup sync and post it.
>> Let's say, dir2 gets synced, my changed file transferred, and then - 
>> be it the isp (all isps in germany drop the connection after 24h, 
>> except the expensive bussiness lines), or some other reason - the 
>> connection drops. "boxbackupctl sync-and-wait" will now exit with 
>> "sync finished"  (maybe this should be changed to "sync 
>> incomplete...?". I know it's possible to use the notify-admin.sh 
>> here, so this issue is minor). When I restart the sync, the sync will 
>> start at /dir1 again. Ok, it doesn't transmit the changed file in 
>> /dir2 again, because it detects no change there - fair point. 
>> However, the scanning takes a while, and if the connection drops 
>> again before reaching /dir3, dir3 will never get synced!
>
> Then the time taken to scan is the problem, not the downloading and 
> updating of file or directory data.
>
Yes. Uploading / Downloading times we cannot change.
> Scanning a few thousand files should only take a few seconds. Even a 
> hundred thousand should take tens of minutes. I'm curious why it takes 
> five hours.
>
I am not sure how bb identifies changed files. For my understanding it 
has to retrieve some file lists from the server. But I would assume they 
are quite small, so downloading times should not be of concern.
One explanation might be that my CPU is a 500Mhz VIA cpu, my internet 
bandwidth is 6 Mbit/s downstream and 0.6 Mbit/s upstream.
The inital off-line sync of my 250 GB disc via USB too more than 3 days.
> Could you run some software to graph the bandwidth use on your 
> internet connection, e.g. mrtg or cacti or vnstat or ntop? It would 
> also be interesting to see detailed logs of what bbackupd is doing 
> when it takes five hours to sync. Perhaps you could run "bbackupd 
> -kVT" from the command line (or set LogFile=/tmp/log.txt and run the 
> daemon with -V) and send me the logs of a single sync run by private 
> email?
>
I will try to get some logs as soon as possible (might take a few days 
though). Thanks for the recommandation of bandwidth usage monitoring 
tools, I'll try to work with them.

Florian.
> Cheers, Chris.