[Box Backup] Full server store not deleting old and deleted files (boxwin0.09)

Wed Mar 1 22:36:54 GMT 2006

Thanks, Ben.

Yes, no output at all; I couldn't see "housekeeping" or "hk" anywhere
in the log.  

So, I changed the hard limit on problematic account 00000014
temporarily.  I then ran a bbackupctl sync and it seemed to be doing
something, backedup a couple of little files, but hung for too many
minutes on an 8MB QuickBooks file.  It's an old file, so I zipped it
and deleted the original.  Restarting the Windows client service, I
could see in the Event Viewer that it was still trying to backup the
unzipped file.  So I stopped the windows service again, and stopped the
server daemons with a sudo killall bbstored.  I restarted
/etc/init.d/bbstored and it ran housekeeping at 95% CPU for over 20
minutes (350MHz PII with 512MB RAM).  But it finally stopped. 

petjal at ns1b:~$ grep -i hk /var/log/box Mar  1 16:43:37 localhost
bbstored/hk[8995]: Housekeeping process started
Mar  1 16:43:37 localhost bbstored/hk[8995]: Starting housekeeping
Mar  1 16:43:37 localhost bbstored/hk[8995]: Finished housekeeping
Mar  1 16:43:37 localhost bbstored/hk[8995]: Terminating daemon
Mar  1 16:44:17 localhost bbstored/hk[9036]: Housekeeping process
started
Mar  1 16:44:17 localhost bbstored/hk[9036]: Starting housekeeping
Mar  1 16:44:20 localhost bbstored/hk[9036]: On housekeeping, sizes in
store do not match calculated sizes, correcting
Mar  1 16:44:21 localhost bbstored/hk[9036]: different (store,calc):
acc 0x00000011, used (159014,159666), old (8227,8257), deleted
(8263,8263), dirs (1233,1233)
Mar  1 16:44:25 localhost bbstored/hk[9036]: On housekeeping, sizes in
store do not match calculated sizes, correcting
Mar  1 16:44:25 localhost bbstored/hk[9036]: different (store,calc):
acc 0x00000014, used (663024,694959), old (16068,16560), deleted
(96893,96893), dirs (1965,2030)
Mar  1 16:45:49 localhost bbstored/hk[9036]: Housekeeping giving way to
connection for account 0x00000014
Mar  1 16:45:50 localhost bbstored/hk[9036]: Account 0x00000014,
removed 12372 blocks (1669 files, 0 dirs) was interrupted
Mar  1 16:45:52 localhost bbstored/hk[9036]: Account 0x01abc23d,
removed 93 blocks (0 files, 93 dirs)
Mar  1 16:45:52 localhost bbstored/hk[9036]: Account 0x123abc4d,
removed 1 blocks (0 files, 1 dirs)
Mar  1 16:45:52 localhost bbstored/hk[9036]: Finished housekeeping
Mar  1 16:54:59 localhost bbstored/hk[9036]: Terminating daemon
Mar  1 16:55:19 localhost bbstored/hk[9614]: Housekeeping process
started
Mar  1 16:55:20 localhost bbstored/hk[9614]: Starting housekeeping
Mar  1 17:18:36 localhost bbstored/hk[9614]: Account 0x00000014,
removed 44398 blocks (3816 files, 19 dirs)
Mar  1 17:18:36 localhost bbstored/hk[9614]: Finished housekeeping
Mar  1 17:19:36 localhost bbstored/hk[9614]: Starting housekeeping
Mar  1 17:19:39 localhost bbstored/hk[9614]: Finished housekeeping

I restarted the service on the windows client and it is now stuck
trying to backup the old file that doesn't exist anymore; from the
windows Event Viewer:
  Send ListDirectory(0x5647,0xffffffff,0xc,true).
  Receive Success(0x5647).
  Receiving stream, size 489.
  Send GetBlockIndexByName(0x5647,"Hallmos 101503 2.QBB").
  Receive Success(0x5648).
  Receiving stream, size 89148.
  Sending stream, size uncertain.
  Send  
StoreFile(0x5647,0x40dca63288f40,0x263be2696554d151,0x5648,"Hallmos
101503 2.QBB").
  Sending stream, size uncertain.

>From /var/log/box on the server:
Mar  1 17:26:37 localhost bbstored[11200]: Send Success(0x5647)
Mar  1 17:26:37 localhost bbstored[11200]: Sending stream, size 489
Mar  1 17:26:37 localhost bbstored[11200]: Receive
GetBlockIndexByName(0x5647,OPAQUE)
Mar  1 17:26:37 localhost bbstored[11200]: Receive
GetBlockIndexByName(0x5647,OPAQUE)
Mar  1 17:26:37 localhost bbstored[11200]: Send Success(0x5648)
Mar  1 17:26:37 localhost bbstored[11200]: Send Success(0x5648)
Mar  1 17:26:37 localhost bbstored[11200]: Sending stream, size 89148
Mar  1 17:26:54 localhost bbstored[11200]: Receive
StoreFile(0x5647,0x40dca63288f40,0x263be2696554d151,0x5648,OPAQUE)
Mar  1 17:26:54 localhost bbstored[11200]: Receive
StoreFile(0x5647,0x40dca63288f40,0x263be2696554d151,0x5648,OPAQUE)
Mar  1 17:26:54 localhost bbstored[11200]: Receiving stream, size
uncertain
Mar  1 17:34:37 localhost bbstored/hk[9614]: Starting housekeeping
Mar  1 17:34:38 localhost bbstored/hk[9614]: Finished housekeeping

The problem again is the the .QBB file doesn't exist anymore; it has
been replaced by the .ZIP file of the same name.  And the bb system is
hanging on the non-existent .QBB file.  What should I do?  (I haven't
checked exactly how much storage on the client is represented by the
BackupLocations, as compared to the soft limit.)

Thanks again,
Pete

--- Ben Summers <ben at fluffy.co.uk> wrote:

> 
> On 1 Mar 2006, at 14:04, E.W. Peter Jalajas wrote:
> 
> >>>
> >>> So, why haven't the old and deleted files been deleted?  And what
> >>> should I do to fix it?
> >>
> >> Check your server logs to see if there's anything about
> interrupted
> >> housekeeping.
> >
> > These all showed nothing:
> >   530  grep -i house /var/log/box.0
> >   531  grep -i house /var/log/box
> >   532  grep -i hk /var/log/box
> >   533  grep -i hk /var/log/box.0
> 
> No output at all? Or no messages about interruptions? If the former, 
> 
> this suggests that there was no housekeeping activity in that  
> timeframe, so you need to search further.
> 
> >
> >> If the housekeeping takes too long and the client
> >> connects too frequently, then housekeeping never deletes anything.
> >
> > Here's the timings from their bbackupd.conf:
> >
> > UpdateStoreInterval = 3600
> > MinimumFileAge = 21600
> > MaxUploadWait = 86400
> > FileTrackingSizeThreshold = 65535
> > DiffingUploadSizeThreshold = 8192
> > MaximumDiffingTime = 20
> >
> >>
> >> This is something to be aware of, and the reason I want to move to
> >> ref-counted stores in the 0.20 version.
> >>
> >
> > So, for now, should I just increase their quota?  Temporarily? 
> Should
> > I put the known large-file BackupLocations (those with the big
> > QuickBooks file) at the end of the bbackupd.conf listing?
> 
> Increase the hard limit temporarily. Check the logs for the message  
> where it says it deleted stuff. If you don't find that in the near  
> future, then you need to check that the client does not have more  
> data than your soft limit.
> 
> But the first task is to find out exactly what the housekeeping is  
> doing.
> 
> Ben
> 
> 
> 
> _______________________________________________
> boxbackup mailing list
> boxbackup at fluffy.co.uk
> http://lists.warhead.org.uk/mailman/listinfo/boxbackup
> 