[Box Backup] Memory usage house keeping through the roof

[Chris Wilson]

Hi Jan,

On Mon, 11 Feb 2013, Jan Haastrecht wrote:

> The house keeping process uses over 100MB of memory here for a 280GB 
> store of maximum capacity 800GB with a lot of free space left. How can 
> the memory usage be predicted given some variables (like number of files 
> deleted, size of store, and others)? This should have been a core design 
> consideration and it surprises me that I cannot find a formula for it 
> anywhere.  Currently, the housekeeping process (why is it even running 
> when there is a lot of space still free?) is stopping me from using 
> boxbackup. Can you do anything about the memory usage?
>
> The way I would like to see it working would be a concurrently running 
> housekeeping process (not in a separate address space) where the maximum 
> memory usage would configurable at build time or at program start. For 
> example the maximum would be around 10MB and anything else that would be 
> needed would we written in 10MB batches to external storage (as 
> configured in some configuration variable 
> ExternalStorage=/mnt/boxbackup-var/run).

I think it uses more than it used to since I added reference counting, 
which is experimental. I am planning to memory-map this database from 
disk, to reduce in-core memory usage, but haven't implemented that yet. 
In the mean time, swap should work just as well.

It uses 4 bytes per object, so 4 MB if you have 1 million objects (files 
and directories) in your account. How many do you have?

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <chris+sig at qwirx.com> Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer |
\__/_/_/_//_/___/ | We are GNU : free your mind & your software |