[Box Backup-commit] [boxbackup/boxbackup] 09603c: Refactor BackupStoreCheck to use BackupFileSystem, ...

Fri Sep 29 21:50:39 BST 2017

  Branch: refs/heads/s3_support_merge
  Home:   https://github.com/boxbackup/boxbackup
  Commit: 09603cc8db9ac1f5f908f41d4adc5d3eb0639406
      https://github.com/boxbackup/boxbackup/commit/09603cc8db9ac1f5f908f41d4adc5d3eb0639406
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.cpp
    M lib/backupstore/BackupFileSystem.h
    M lib/backupstore/BackupStoreCheck.cpp
    M lib/backupstore/BackupStoreCheck2.cpp
    M lib/backupstore/BackupStoreException.txt

  Log Message:
  -----------
  Refactor BackupStoreCheck to use BackupFileSystem, part 7

Move most low-level filesystem structure check code into BackupFileSystem, and
add specialised implementation for S3.

  Commit: 9a346bfa3369f911552aee38ce970e62adb07720
      https://github.com/boxbackup/boxbackup/commit/9a346bfa3369f911552aee38ce970e62adb07720
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupAccountControl.cpp
    M lib/backupstore/BackupStoreCheck.cpp
    M lib/backupstore/BackupStoreCheck.h
    M lib/backupstore/BackupStoreCheck2.cpp

  Log Message:
  -----------
  Refactor BackupStoreCheck to use BackupFileSystem, part 8

Remove last vestiges of RaidFile from BackupStoreCheck.

  Commit: e07d16017aa923a796d7f88dbe51069ce7048370
      https://github.com/boxbackup/boxbackup/commit/e07d16017aa923a796d7f88dbe51069ce7048370
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupStoreCheck2.cpp

  Log Message:
  -----------
  Rename some local variables in BackupStoreCheck

  Commit: e23da3a8d2f567d85be83a51c32728d36bf37614
      https://github.com/boxbackup/boxbackup/commit/e23da3a8d2f567d85be83a51c32728d36bf37614
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupclient/BackupDaemonConfigVerify.cpp
    M lib/backupstore/BackupAccountControl.cpp
    M lib/backupstore/BackupFileSystem.cpp
    M lib/backupstore/BackupFileSystem.h
    M lib/backupstore/BackupStoreException.txt
    M test/s3store/testfiles/bbackupd.conf

  Log Message:
  -----------
  Add configuration for S3 local cache directory

  Commit: 11f011b26c72b7ce8034973365a977de3706f33b
      https://github.com/boxbackup/boxbackup/commit/11f011b26c72b7ce8034973365a977de3706f33b
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.cpp

  Log Message:
  -----------
  Fix repeated calls to TryGetLock() when already holding lock

  Commit: d7a4ec74e29eae55895dbec47a002971db4becf6
      https://github.com/boxbackup/boxbackup/commit/d7a4ec74e29eae55895dbec47a002971db4becf6
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.cpp
    M lib/backupstore/BackupFileSystem.h

  Log Message:
  -----------
  Implement RefCountDatabase methods in S3BackupFileSystem

  Commit: a72fa9640ffd8e2ec7fb9c74c67e0ebbe1420ee3
      https://github.com/boxbackup/boxbackup/commit/a72fa9640ffd8e2ec7fb9c74c67e0ebbe1420ee3
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.h

  Log Message:
  -----------
  Fix writeback of refcount databases on BackupFileSystem destruction

Call ReleaseLock in the child destructor (not the parent) to allow the refcount
database to clean itself up (normally involving calling
BackupFileSystem::SaveRefCountDatabase) before the BackupFileSystem is too
destroyed to be able to fulfil the request.

  Commit: ad908f2a0284ebe8d805aa869064a2f488a3974f
      https://github.com/boxbackup/boxbackup/commit/ad908f2a0284ebe8d805aa869064a2f488a3974f
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/bbstored/BBStoreDHousekeeping.cpp
    M lib/bbstored/BackupStoreDaemon.cpp

  Log Message:
  -----------
  Wait for housekeeping to finish before bbstored exits

Should fix random test failures due to housekeeping process realising that the
master has asked it to terminate, and logging that it has, after the master has
already died and the test finished, confusing the test runner with its extra
output. Should also help make the occasionally-reported housekeeping crashes
more obvious and easier to debug.

  Commit: c76dcb82a35daf7f49732853900211c3717db6c8
      https://github.com/boxbackup/boxbackup/commit/c76dcb82a35daf7f49732853900211c3717db6c8
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.cpp
    M lib/backupstore/BackupFileSystem.h

  Log Message:
  -----------
  Remove useless Get/Put/HeadObject methods from S3BackupFileSystem

These should not have been part of the API, and are not needed since mrClient
has the same methods.

  Commit: 7d9805e8d594a34f0f1ac7b2638ff632d1eab9bf
      https://github.com/boxbackup/boxbackup/commit/7d9805e8d594a34f0f1ac7b2638ff632d1eab9bf
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupAccountControl.cpp
    M lib/backupstore/BackupAccountControl.h
    M lib/backupstore/BackupFileSystem.cpp
    M lib/backupstore/BackupFileSystem.h
    M lib/backupstore/BackupStoreInfo.cpp
    M lib/backupstore/BackupStoreInfo.h
    R test/s3store/testfiles/store/subdir/dirs/create-me.txt
    M test/s3store/tests3store.cpp

  Log Message:
  -----------
  Refactor BackupAccountControl to use BackupFileSystem

  Commit: 9f42efabf6ea6e8126684ed3956ab66cba591388
      https://github.com/boxbackup/boxbackup/commit/9f42efabf6ea6e8126684ed3956ab66cba591388
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.cpp
    M lib/backupstore/BackupStoreFile.cpp
    M lib/backupstore/BackupStoreFile.h
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Refactor BackupStoreFile::VerifyStream from write to read side

This allows us to wrap an existing stream (not necessarily a file, e.g. a
streaming HTTPRequest) in a VerifyStream and pass that stream to a
S3PutFileCompleteTransaction, which isn't just going to CopyStreamTo()
somewhere, but do something more complex to it.

  Commit: 989caf643448717262b246c3e4c1b2dc7570d16c
      https://github.com/boxbackup/boxbackup/commit/989caf643448717262b246c3e4c1b2dc7570d16c
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/StoreTestUtils.cpp
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Fix a long-standing bug in the set_refcount() test helper function

Previously it would truncate the expected refcount list at the first zero entry,
instead of just removing zero entries from the end.

Also move some test code to test_server_commands which should have been there,
since they were causing issues with the check for the refcount database in
test_server_housekeeping. It's hard to disentangle these two changes.

  Commit: 5b60e321499b733d41233ad177b5f293292c36db
      https://github.com/boxbackup/boxbackup/commit/5b60e321499b733d41233ad177b5f293292c36db
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Add extra refcount checks to test_server_housekeeping

  Commit: b9418a119d9433ada9dd4316936446ce932e7c8b
      https://github.com/boxbackup/boxbackup/commit/b9418a119d9433ada9dd4316936446ce932e7c8b
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupCommands.cpp
    M lib/backupstore/BackupFileSystem.cpp
    M lib/backupstore/BackupFileSystem.h
    M lib/backupstore/BackupStoreContext.cpp
    M lib/backupstore/BackupStoreContext.h
    M lib/backupstore/BackupStoreDirectory.h
    M lib/backupstore/BackupStoreException.txt
    M lib/backupstore/StoreTestUtils.cpp
    A lib/common/ByteCountingStream.h
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Refactor BackupStoreContext to use BackupFileSystem

  Commit: 5bdc48ee08ee4e24c8b5f787436eaeb68c133163
      https://github.com/boxbackup/boxbackup/commit/5bdc48ee08ee4e24c8b5f787436eaeb68c133163
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M infrastructure/makebuildenv.pl.in

  Log Message:
  -----------
  Enable and fix warnings in infrastructure/makebuildenv.pl

  Commit: c9cbada78760b6ba054c5734f001c2e53f662d6a
      https://github.com/boxbackup/boxbackup/commit/c9cbada78760b6ba054c5734f001c2e53f662d6a
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M infrastructure/makebuildenv.pl.in

  Log Message:
  -----------
  Fix Makefile dependency tracking of configured files

This allows protocol files to be regenerated automatically if
makeprotocol.pl.in changes, and similarly exception files and
makeexception.pl.in.

  Commit: 78eb7385b98bd626b9f86b51c63456277ffa0f5b
      https://github.com/boxbackup/boxbackup/commit/78eb7385b98bd626b9f86b51c63456277ffa0f5b
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M bin/bbackupctl/bbackupctl.cpp
    M bin/bbackupquery/bbackupquery.cpp
    M lib/bbackupd/BackupDaemon.cpp
    M lib/bbstored/BBStoreDHousekeeping.cpp
    M lib/common/CommonException.txt
    M lib/common/FdGetLine.cpp
    M lib/common/FdGetLine.h
    R lib/common/GetLine.cpp
    R lib/common/GetLine.h
    M lib/common/IOStream.cpp
    M lib/common/IOStreamGetLine.cpp
    M lib/common/IOStreamGetLine.h
    A lib/common/LineBuffer.cpp
    A lib/common/LineBuffer.h
    M lib/httpserver/HTTPException.txt
    M lib/httpserver/HTTPHeaders.cpp
    M lib/httpserver/HTTPRequest.cpp
    M lib/httpserver/HTTPResponse.cpp
    M lib/httpserver/S3Simulator.cpp
    M lib/httpserver/S3Simulator.h
    M lib/server/ServerControl.cpp
    M lib/server/SocketStream.cpp
    A test/backupstore/testfiles/bbackupd.conf
    A test/backupstore/testfiles/s3simulator.conf
    M test/backupstorefix/testbackupstorefix.cpp
    M test/basicserver/testbasicserver.cpp
    M test/bbackupd/testbbackupd.cpp
    M test/common/testcommon.cpp

  Log Message:
  -----------
  Refactor GetLine

The class name was a problem, because the method that we want to call on it is
also called GetLine, which looks too much like a constructor in C++, so I
renamed it to LineBuffer.  I haven't yet renamed the child classes.

Also, it was impossible to distinguish between the different causes of
GetLine() returning false (timeout and signal) and EOF could only be checked by
calling a different function. Now all of these causes throw different
exceptions.

This really fixes the problem with attaching a debugger to a process waiting in
GetLine, which would previously cause the read() to return immediately with
EINTR, which we never retried because it looked like a timeout.

  Commit: a4083f39f9cc5794e369373d78091ea4ca024ae9
      https://github.com/boxbackup/boxbackup/commit/a4083f39f9cc5794e369373d78091ea4ca024ae9
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/common/DebugMemLeakFinder.cpp
    M lib/common/MemLeakFinder.h

  Log Message:
  -----------
  Implement placement operator delete

"If the object is being created as part of a new expression, and an exception
is thrown, the object’s memory is deallocated by calling the appropriate
deallocation function. If the object is being created with a placement new
operator, the corresponding placement delete operator is called—that is, the
delete function that takes the same additional parameters as the placement new
operator. If no matching placement delete is found, no deallocation takes
place."

So to avoid memory leaks, we need to implement placement delete operators that
correspond to our placement new, which we use for leak detection (ironically)
in debug builds.

  Commit: 54d2d9b125b08053a4afdd28f4ee2d451cd539c1
      https://github.com/boxbackup/boxbackup/commit/54d2d9b125b08053a4afdd28f4ee2d451cd539c1
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupclient/BackupDaemonConfigVerify.cpp
    M lib/backupstore/BackupAccountControl.cpp
    M lib/httpserver/S3Client.h

  Log Message:
  -----------
  Enable support for S3 virtual hosts in BackupAccountControl

This enables the S3 bucket name to be included in the virtual hostname, for
example bucketname.s3.amazonaws.com, when connecting to a different hostname
such as localhost. This makes bucket names usable in tests with
BackupAccountControl.

  Commit: 1c49e464c19d8e9465b825fe188d0aa61493e2e1
      https://github.com/boxbackup/boxbackup/commit/1c49e464c19d8e9465b825fe188d0aa61493e2e1
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.cpp
    M lib/backupstore/BackupFileSystem.h
    M lib/backupstore/BackupStoreException.txt

  Log Message:
  -----------
  Properly implement S3BackupFileSystem::GetPermanentRefCountDatabase

  Commit: 504eaadbd5e39c149c5e2ee2154f7ee5f6df1955
      https://github.com/boxbackup/boxbackup/commit/504eaadbd5e39c149c5e2ee2154f7ee5f6df1955
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.cpp

  Log Message:
  -----------
  Remove always-false assertion in S3BackupFileSystem::GetDirectory

This was designed to stop untested code from running. Now we are ready to use
this code, with tests soon to come.

  Commit: d842757bff00631bbbca04cb68bec091d46e400e
      https://github.com/boxbackup/boxbackup/commit/d842757bff00631bbbca04cb68bec091d46e400e
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.h
    M lib/backupstore/StoreTestUtils.cpp
    M lib/backupstore/StoreTestUtils.h
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Add store test helpers for specialised tests

  Commit: 2279450b287a3d5b553932262348be7a599be2bb
      https://github.com/boxbackup/boxbackup/commit/2279450b287a3d5b553932262348be7a599be2bb
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-09-29 (Fri, 29 Sep 2017)

  Changed paths:
    M lib/backupstore/BackupStoreContext.h
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Make BackupStoreContext::ClearDirectoryCache() public

Needed to clear the cache in backupstore tests specialised for S3.

Compare: https://github.com/boxbackup/boxbackup/compare/707b67501528...2279450b287a