[Box Backup] No housekeeping?

[dave bamford]

Housekeeping has crashed again but this time I had the -V switch on

Here are the results...
Hope they mean something.

Regards

Dave


> Jun 10 12:38:24 bart bbstored/hk[11295]: ERROR: Failed to rename file: /backups/box/backup/00001000/28/03/o4a.rfwX to /backups/box/backup/00001000/28/03/o4a.rfw: No such file or directory (2)
> Jun 10 12:38:24 bart bbstored/hk[11295]: Obtained 10 stack frames.
> Jun 10 12:38:24 bart bbstored/hk[11295]: Stack frame 0: bbstored(DumpStackBacktrace()+0x26) [0x4dbff6]
> Jun 10 12:38:24 bart bbstored/hk[11295]: Stack frame 1: bbstored(RaidFileWrite::Commit(bool)+0x503) [0x46f4d3]
> Jun 10 12:38:24 bart bbstored/hk[11295]: Stack frame 2: bbstored(HousekeepStoreAccount::DeleteFile(long, long, BackupStoreDirectory&, std::string const&, long)+0x106) [0x43c9a6]
> Jun 10 12:38:24 bart bbstored/hk[11295]: Stack frame 3: bbstored(HousekeepStoreAccount::DeleteFiles()+0x114) [0x43e5a4]
> Jun 10 12:38:24 bart bbstored/hk[11295]: Stack frame 4: bbstored(HousekeepStoreAccount::DoHousekeeping(bool)+0x632) [0x43f122]
> Jun 10 12:38:24 bart bbstored/hk[11295]: Stack frame 5: bbstored(BackupStoreDaemon::RunHousekeepingIfNeeded()+0x446) [0x424f56]
> Jun 10 12:38:24 bart bbstored/hk[11295]: Stack frame 6: bbstored(BackupStoreDaemon::HousekeepingProcess()+0xaf) [0x425daf]
> Jun 10 12:38:24 bart bbstored/hk[11295]: Stack frame 7: bbstored(BackupStoreDaemon::Run()+0x278) [0x435d18]
> Jun 10 12:38:24 bart bbstored/hk[11295]: Stack frame 8: bbstored(Daemon::Main(std::string const&)+0x1582) [0x499c22]
> Jun 10 12:38:24 bart bbstored/hk[11295]: Stack frame 9: bbstored(Daemon::Main(char const*, int, char const**)+0x24c) [0x4969cc]
> Jun 10 12:38:24 bart bbstored/hk[11295]: WARNING: Exception thrown: RaidFileException(OSError) at RaidFileWrite.cpp(325)
> Jun 10 12:38:25 bart bbstored/hk[11295]: ERROR: Failed to delete file: /backups/box/backup/00001000/28/03/o4a.rfwX: No such file or directory (2)
> Jun 10 12:38:25 bart bbstored/hk[11295]: Obtained 10 stack frames.
> Jun 10 12:38:25 bart bbstored/hk[11295]: Stack frame 0: bbstored(DumpStackBacktrace()+0x26) [0x4dbff6]
> Jun 10 12:38:25 bart bbstored/hk[11295]: Stack frame 1: bbstored(RaidFileWrite::Discard()+0x338) [0x46af78]
> Jun 10 12:38:25 bart bbstored/hk[11295]: Stack frame 2: bbstored(RaidFileWrite::~RaidFileWrite()+0x1a) [0x46b8da]
> Jun 10 12:38:25 bart bbstored/hk[11295]: Stack frame 3: bbstored(HousekeepStoreAccount::DeleteFile(long, long, BackupStoreDirectory&, std::string const&, long)+0xd75) [0x43d615]
> Jun 10 12:38:25 bart bbstored/hk[11295]: Stack frame 4: bbstored(HousekeepStoreAccount::DeleteFiles()+0x114) [0x43e5a4]
> Jun 10 12:38:25 bart bbstored/hk[11295]: Stack frame 5: bbstored(HousekeepStoreAccount::DoHousekeeping(bool)+0x632) [0x43f122]
> Jun 10 12:38:25 bart bbstored/hk[11295]: Stack frame 6: bbstored(BackupStoreDaemon::RunHousekeepingIfNeeded()+0x446) [0x424f56]
> Jun 10 12:38:25 bart bbstored/hk[11295]: Stack frame 7: bbstored(BackupStoreDaemon::HousekeepingProcess()+0xaf) [0x425daf]
> Jun 10 12:38:25 bart bbstored/hk[11295]: Stack frame 8: bbstored(BackupStoreDaemon::Run()+0x278) [0x435d18]
> Jun 10 12:38:25 bart bbstored/hk[11295]: Stack frame 9: bbstored(Daemon::Main(std::string const&)+0x1582) [0x499c22]
> Jun 10 12:38:25 bart bbstored/hk[11295]: WARNING: Exception thrown: RaidFileException(OSError) at RaidFileWrite.cpp(380)


On Mon, 2012-01-23 at 17:26 +0000, dave bamford wrote:
> Hi Peter
> 
> I had a similar problem and it turned out Housekeeping was aborting on a
> corrupt account and never restarting until I restarted bbstored.
> Try starting bbstored -v and check the logs
> 
> Regards
> 
> Dave Bamford
> 
> On Mon, 2012-01-23 at 17:43 +0100, Peter Hall wrote:
> > Hi fellow boxbackup users,
> > 
> > 
> > My store is full, and it seems housekeeping isn't running!
> > 
> > 
> > # bbstoreaccounts info 1
> >           Account ID: 0x00000001
> >       Last object ID: 0x1106496
> >                 Used:  243199923 blocks,  927.73 GB,  99% |
> > *************** |
> >            Old files:    3849882 blocks,   14.69 GB,   1% |
> >      |
> >        Deleted files:  127698533 blocks,  487.13 GB,  52% |********
> >      |
> >          Directories:     451091 blocks,    1.72 GB,   0% |
> >      |
> >           Soft limit:  230400000 blocks,  878.91 GB,  94% |
> > *************** |
> >           Hard limit:  243200000 blocks,  927.73 GB, 100% |
> > ****************|
> >  Client store marker: 17851542
> > 
> > 
> > Plenty of deleted files to remove and make space for new ones, but
> > it's been like this for days now.
> > 
> > 
> > I've tried restarting the server, and drastically lowering the
> > housekeeping limit:
> > # grep -i keep /etc/boxbackup/bbstored.conf
> > TimeBetweenHousekeeping = 120
> > 
> > 
> > But still no cleaning of the old files.
> > 
> > 
> > All the references to housekeeping I can find in the logs are:
> > client=0x00000001[29150]: WARNING: Reference count database is missing
> > or corrupted, creating a new one, expect housekeeping to find and fix
> > problems with reference counts later.
> > 
> > 
> > I have run a fix of the account with "bbstoreaccounts check 1 fix". It
> > found and fixed three errors on the first run, subsequent runs find no
> > errors.
> > 
> > 
> > bbstored seems to be doing something though, it takes around 45%
> > memory and 50% cpu if I consult 'top'.
> > 
> > 
> > Is it normal that it can take several days before I see any
> > housekeeping progress on a store of this size?
> > 
> > 
> > Thanks in advance,
> > Peter
> > _______________________________________________
> > Boxbackup mailing list
> > Boxbackup at boxbackup.org
> > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup
> 
> 
> _______________________________________________
> Boxbackup mailing list
> Boxbackup at boxbackup.org
> http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup