[Box Backup] Trac account request

[Stewart Adam]

On 2011/01/11 7:20 PM, Chris Wilson wrote:
> Hi Achim,
>
> On Tue, 11 Jan 2011, Achim wrote:
>
>> Did you manage to move this further along? A per-client limitation of the
>> upload bandwidth like in rsync would be really helpful to avoid a cluster
>> of local bb clients from hogging the available DSL uplink.
>>
>> Modifying the DSL modem parameters for QoS seems a lot less practical
>> (i.e. likely) than updating that "bandwidth" parameter in a box backup
>> conf file.
>
> I've implemented a rate limiting option in trunk. Simply set MaxUploadRate =
> 8000 (bytes per second) or similar on the client (in bbackupd.conf) to use
> it. Please have a look and see if it meets your needs.
This is great, many thanks Chris! After some brief testing it seems to work 
very well and you are right - the CPU usage is very low (0.8% on my machine).

I'm going to look into if we can make effectiveRateSoFar a time average of 
the past few seconds instead of the totals though, as that way if the link 
throughput changes (say another application finishes a transfer or laptop 
changes locations) then the calculation for the current rate will more 
accurately reflect the actual transfer rates.

> It's not as sophisticated as what Stewart was suggesting, distributing reads
> over a second. It does try to ensure at least one packet per second. I'm not
> sure how much spikes at one-second granularity matter.
My method was rather simple but just had the nice side effect of 
distributing writes over a second ;)

I used what was essentially a method of finite differences - I divide a 
second into some "N" iterations (10 or 15 seemed to work well) and then 
allocate byteLimit/N bandwidth per iteration, sleeping in between if the 
allocated bandwidth gets used up early in an iteration.

> I think the method I've used is very CPU efficient but it does reduce
> bandwidth efficiency a little, by uploading smaller chunks of data to the
> server, and probably in smaller packets (<1500 bytes) at very low rates
> (<1500 bytes per second). It's also not 100% accurate because it doesn't
> take the OpenSSL encryption overhead into account, or any bandwidth use
> except streaming file and patch uploads.
Both problems affected my implementation as well, I was thinking of forcing 
a minimum packet size to avoid the overhead problem.

I don't think the SSL overhead is something to worry about... From testing 
it seems that there's roughly 2KB difference per 50KB (ie 50KB/s limit 
results in 52KB/s real world; 750KB/s limit results in ~775KB/s).

Out of curiosity, do the trace messages accurately reflect when data is 
being sent out by bbackupd?

Regards,
Stewart