[Box Backup] Memory usage house keeping through the roof

[Jan Haastrecht]

Chris Wilson schreef:
> Hi Jan,
>
> On Mon, 11 Feb 2013, Jan Haastrecht wrote:
>
>> The house keeping process uses over 100MB of memory here for a 280GB store of 
>> maximum capacity 800GB with a lot of free space left. How can the memory 
>> usage be predicted given some variables (like number of files deleted, size 
>> of store, and others)? This should have been a core design consideration and 
>> it surprises me that I cannot find a formula for it anywhere.  Currently, the 
>> housekeeping process (why is it even running when there is a lot of space 
>> still free?) is stopping me from using boxbackup. Can you do anything about 
>> the memory usage?
>>
>> The way I would like to see it working would be a concurrently running 
>> housekeeping process (not in a separate address space) where the maximum 
>> memory usage would configurable at build time or at program start. For 
>> example the maximum would be around 10MB and anything else that would be 
>> needed would we written in 10MB batches to external storage (as configured in 
>> some configuration variable ExternalStorage=/mnt/boxbackup-var/run).
>
> I think it uses more than it used to since I added reference counting, which 
> is experimental. I am planning to memory-map this database from disk, to 
> reduce in-core memory usage, but haven't implemented that yet. In the mean 
> time, swap should work just as well.
>
> It uses 4 bytes per object, so 4 MB if you have 1 million objects (files and 
> directories) in your account. How many do you have?
I have about 5 million (more than I expected). It's using around 140MB (of which 
a part is swap). I can't see whether it has completed housekeeping.
>
> Cheers, Chris.