[Box Backup] Current files greater than used files

[Dave Bamford]

Hi Peter

the figures for current files look correct to me. Used = current + old +
deleted + directories.
My figures were correct after checking and fixing the account.

My blocksize is 2048, it seems to work OK but I store all the backups on
a NFS attached machine. Some time ago Chris changed the code so it
worked much faster and I use an async connection.

I agree about the housekeeping, it can take a long time on big accounts
and therefore take quite a bit of time to get around the whole store. I
have about 4Tb of backups currently.

Regards

Dave

On Tue, 2013-10-29 at 09:58 -0400, Peter Jalajas, GigaLock Backup
Services wrote:
> Hi Chris and Dave,
> 
> Here's another example of the "Current files" problem from my server
> this morning:
>   $ sudo bbstoreaccounts info ${macct}
> ...
>       Last object ID: 0x2b3f90
>                 Used:   31955740 blocks,  121.90 GB,  60% |*********       |
>        Current files:    2509999 blocks,    9.57 GB,   4% |                |
>            Old files:        185 blocks,  740.00 kB,   0% |                |
>        Deleted files:   29390948 blocks,  112.12 GB,  56% |********        |
>          Directories:      54608 blocks,  213.31 MB,   0% |                |
>           Soft limit:   35096852 blocks,  133.88 GB,  66% |**********      |
>           Hard limit:   52428800 blocks,  200.00 GB, 100% |****************|
>  Client store marker: 2834320
>           Live Files: 273743
>            Old Files: 28
>        Deleted Files: 212264
>          Directories: 52429
>              Enabled: yes
> 
> (In recent days, this client was choking on the limits, so I greatly
> expanded the Exclusion list on this client, hence the large Deleted
> files.)
> 
> Note that my server is in a bit of a state of shock, because I, most
> likely completely ignorantly, changed the Block Size in
>   /etc/boxbackup/raidfile.conf
> from
>   BlockSize = 512
> to
>   BlockSize = 4096
> I had used 512 in complete ignorance when I set up the server.  I have
> since learned to use:
>   $ sudo tune2fs -l ${mdevice} | grep -i "block size"
>       Block size:               4096
> (I think that is correct--let me know if otherwise.)
> 
> This client was throwing some errors, so I killed the bbstored service
> and did a:
>   $ sudo bbstoreaccounts housekeep ${macct}
>   $ sudo bbstoreaccounts check ${macct} fix
>   $ sudo bbstoreaccounts info ${macct}
> and got the results above.
> 
> Hope my ignorance helps in your diagnosis.
> 
> A couple of other random but perhaps related thoughts:
> 
>   1.  I always try to remember to put a huge gap (many GB; many times
> larger than than largest possible client file to be backed up) between
> the soft and hard limits, to leave room for the service to do its
> work.
> 
>   2.  I'm still trying to figure out the best balance between
> UpdateStoreInterval on the client and TimeBetweenHousekeeping on the
> server.  I currently use, perhaps in ignorance,
> TimeBetweenHousekeeping = 86400.  I was finding that the clients were
> so very large and taking so many hours to backup, and reconnecting too
> often, and so housekeeping wasn't being run, I think.  Thinking out
> loud about related Feature Requests:
>         2a:  A client setting to replace UpdateStoreInterval with
> something like UpdateStoreIntervalAfterLastSuccessfulBackup.  Thus,
> the client would backup not every so many wall-clock hours, but so
> many wall-clock hours _after_ the last backup completion.
>         2b:  A server setting that runs housekeeping immediately after
> the client disconnects after a successful backup, perhaps with a
> configurable logic, like, MinimumIntervalBetweenHK = 86400 (at least
> once a day, but not more than that).  (And make sure HK completes
> before allowing the client to reconnect, which I think, but am not
> sure, is the current situation.)
> 
> Thanks!
> Pete
> 
> 
> On Tue, Oct 22, 2013 at 4:18 PM, Chris Wilson wrote:
> > Hi Dave,
> >
> >
> > On Wed, 9 Oct 2013, Dave Bamford wrote:
> >
> >> How is it possible for the Current files to be much greater than the used
> >> files. See stats
> >
> >
> > The "blocks in current files" is a relatively recently added measure. As
> > such, I haven't written a lot of tests for it, and perhaps I got the
> > calculation of the adjustments due to creating files and directories wrong
> > somewhere.
> >
> > It would help me to work out how, if you could run bbstoreaccounts check
> > against the store and see if it adjusts the reference counts, and if so
> > which ones and by how much. Could you do that for me and let me know the
> > results?
> >
> >
> >> As you can see from the stats, if the deleted files were removed we would
> >> be under quota by about 5%
> >
> >
> > The Used Blocks is the one that counts for quota purposes. I added "current
> > files" in the hope of helping people to understand how their current usage
> > was split between current, old and deleted files, although clearly that
> > doesn't help if it's not accurate, so I'd like to fix it.
> >
> > In this account, the Used Blocks is very close to the hard limit, and hence
> > uploading a large file (e.g. 1.4 GB or more) would fail because it would
> > take the account over quota. Housekeeping should still be trying to remove
> > old and deleted files to bring the account back under the soft limit. Do you
> > know if it succeeded? Does it if you run housekeeping manually, with
> > "bbstoreaccounts housekeep 1001"?
> >
> > Cheers, Chris.
> _______________________________________________
> Boxbackup mailing list
> Boxbackup at boxbackup.org
> http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup