[Box Backup-commit] [boxbackup/boxbackup] dbb4d0: Improve exception handling in single-process serve...

Sat Nov 11 19:41:34 GMT 2017

  Branch: refs/heads/s3_support_merge
  Home:   https://github.com/boxbackup/boxbackup
  Commit: dbb4d0ee445cff5f7539e05bb9f059bf134ab5c9
      https://github.com/boxbackup/boxbackup/commit/dbb4d0ee445cff5f7539e05bb9f059bf134ab5c9
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/server/ServerStream.h

  Log Message:
  -----------
  Improve exception handling in single-process servers

Don't allow an uncaught exception to kill the whole server. This makes it
consistent with multi-process servers, where death of a child worker does
not stop the main server process, so it continues to serve requests.

This makes the HTTPServer robust to exceptions (such as the child
disconnecting while the server is streaming a response) on Windows.

(cherry picked from commit 4a515b9ce8e616b357dc07bee27aa7b3692c3602)

  Commit: 8d6de20b05a4f2f5d00d3a2df0af133cef43648b
      https://github.com/boxbackup/boxbackup/commit/8d6de20b05a4f2f5d00d3a2df0af133cef43648b
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/backupstore/StoreTestUtils.cpp
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Fix a long-standing bug in the set_refcount() test helper function

Previously it would truncate the expected refcount list at the first zero entry,
instead of just removing zero entries from the end.

Also move some test code to test_server_commands which should have been there,
since they were causing issues with the check for the refcount database in
test_server_housekeeping. It's hard to disentangle these two changes.

  Commit: 807758bf2847e77374a8194d3b5f96f413582a04
      https://github.com/boxbackup/boxbackup/commit/807758bf2847e77374a8194d3b5f96f413582a04
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Add extra refcount checks to test_server_housekeeping

  Commit: 0db44389ede4dcb413decd9378e84bfbb9ba92d0
      https://github.com/boxbackup/boxbackup/commit/0db44389ede4dcb413decd9378e84bfbb9ba92d0
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/backupstore/BackupCommands.cpp
    M lib/backupstore/BackupFileSystem.cpp
    M lib/backupstore/BackupFileSystem.h
    M lib/backupstore/BackupStoreContext.cpp
    M lib/backupstore/BackupStoreContext.h
    M lib/backupstore/BackupStoreDirectory.h
    M lib/backupstore/BackupStoreException.txt
    M lib/backupstore/StoreTestUtils.cpp
    A lib/common/ByteCountingStream.h
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Refactor BackupStoreContext to use BackupFileSystem

  Commit: 3e5cfc5a5c12d043c9fd4ab04e7235fd8d9d6cc7
      https://github.com/boxbackup/boxbackup/commit/3e5cfc5a5c12d043c9fd4ab04e7235fd8d9d6cc7
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M infrastructure/makebuildenv.pl.in

  Log Message:
  -----------
  Enable and fix warnings in infrastructure/makebuildenv.pl

  Commit: 41e8e5025b47c2b24c5fef18a8ccf9a133430770
      https://github.com/boxbackup/boxbackup/commit/41e8e5025b47c2b24c5fef18a8ccf9a133430770
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M infrastructure/makebuildenv.pl.in

  Log Message:
  -----------
  Fix Makefile dependency tracking of configured files

This allows protocol files to be regenerated automatically if
makeprotocol.pl.in changes, and similarly exception files and
makeexception.pl.in.

  Commit: 3dd9d73deaee2f4e5ca0df938276bd8b07e9b187
      https://github.com/boxbackup/boxbackup/commit/3dd9d73deaee2f4e5ca0df938276bd8b07e9b187
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M bin/bbackupctl/bbackupctl.cpp
    M bin/bbackupquery/bbackupquery.cpp
    M lib/bbackupd/BackupDaemon.cpp
    M lib/bbstored/BBStoreDHousekeeping.cpp
    M lib/common/CommonException.txt
    M lib/common/FdGetLine.cpp
    M lib/common/FdGetLine.h
    R lib/common/GetLine.cpp
    R lib/common/GetLine.h
    M lib/common/IOStream.cpp
    M lib/common/IOStreamGetLine.cpp
    M lib/common/IOStreamGetLine.h
    A lib/common/LineBuffer.cpp
    A lib/common/LineBuffer.h
    M lib/httpserver/HTTPException.txt
    M lib/httpserver/HTTPHeaders.cpp
    M lib/httpserver/HTTPRequest.cpp
    M lib/httpserver/HTTPResponse.cpp
    M lib/httpserver/S3Simulator.cpp
    M lib/httpserver/S3Simulator.h
    M lib/server/ServerControl.cpp
    M lib/server/SocketStream.cpp
    A test/backupstore/testfiles/bbackupd.conf
    A test/backupstore/testfiles/s3simulator.conf
    M test/backupstorefix/testbackupstorefix.cpp
    M test/basicserver/testbasicserver.cpp
    M test/bbackupd/testbbackupd.cpp
    M test/common/testcommon.cpp

  Log Message:
  -----------
  Refactor GetLine

The class name was a problem, because the method that we want to call on it is
also called GetLine, which looks too much like a constructor in C++, so I
renamed it to LineBuffer.  I haven't yet renamed the child classes.

Also, it was impossible to distinguish between the different causes of
GetLine() returning false (timeout and signal) and EOF could only be checked by
calling a different function. Now all of these causes throw different
exceptions.

This really fixes the problem with attaching a debugger to a process waiting in
GetLine, which would previously cause the read() to return immediately with
EINTR, which we never retried because it looked like a timeout.

  Commit: f96f28242ce075cadbe7b0c014406ee1d5204617
      https://github.com/boxbackup/boxbackup/commit/f96f28242ce075cadbe7b0c014406ee1d5204617
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/common/DebugMemLeakFinder.cpp
    M lib/common/MemLeakFinder.h

  Log Message:
  -----------
  Implement placement operator delete

"If the object is being created as part of a new expression, and an exception
is thrown, the object’s memory is deallocated by calling the appropriate
deallocation function. If the object is being created with a placement new
operator, the corresponding placement delete operator is called—that is, the
delete function that takes the same additional parameters as the placement new
operator. If no matching placement delete is found, no deallocation takes
place."

So to avoid memory leaks, we need to implement placement delete operators that
correspond to our placement new, which we use for leak detection (ironically)
in debug builds.

  Commit: f54cabc13a8bea9fa79eab0b7f55cf0813d93525
      https://github.com/boxbackup/boxbackup/commit/f54cabc13a8bea9fa79eab0b7f55cf0813d93525
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/backupclient/BackupDaemonConfigVerify.cpp
    M lib/backupstore/BackupAccountControl.cpp
    M lib/httpserver/S3Client.h

  Log Message:
  -----------
  Enable support for S3 virtual hosts in BackupAccountControl

This enables the S3 bucket name to be included in the virtual hostname, for
example bucketname.s3.amazonaws.com, when connecting to a different hostname
such as localhost. This makes bucket names usable in tests with
BackupAccountControl.

  Commit: d88d8898f1b2fd6d80abf7b942f5710a357b8096
      https://github.com/boxbackup/boxbackup/commit/d88d8898f1b2fd6d80abf7b942f5710a357b8096
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.cpp
    M lib/backupstore/BackupFileSystem.h
    M lib/backupstore/BackupStoreException.txt

  Log Message:
  -----------
  Properly implement S3BackupFileSystem::GetPermanentRefCountDatabase

  Commit: 92b38baa4e1aadffa1448803307a767a8f270094
      https://github.com/boxbackup/boxbackup/commit/92b38baa4e1aadffa1448803307a767a8f270094
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.cpp

  Log Message:
  -----------
  Remove always-false assertion in S3BackupFileSystem::GetDirectory

This was designed to stop untested code from running. Now we are ready to use
this code, with tests soon to come.

  Commit: 82661c5bfbb8fa7b5dcb01c55e01b9adb2bbab53
      https://github.com/boxbackup/boxbackup/commit/82661c5bfbb8fa7b5dcb01c55e01b9adb2bbab53
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.h
    M lib/backupstore/StoreTestUtils.cpp
    M lib/backupstore/StoreTestUtils.h
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Add store test helpers for specialised tests

  Commit: a519bd959feead68c4c146275de55d09cd2d6ff4
      https://github.com/boxbackup/boxbackup/commit/a519bd959feead68c4c146275de55d09cd2d6ff4
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/backupstore/BackupStoreContext.h
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Make BackupStoreContext::ClearDirectoryCache() public

Needed to clear the cache in backupstore tests specialised for S3.

  Commit: 4331cc2f8e4a8096b9ccc86206a9b5431c096f9c
      https://github.com/boxbackup/boxbackup/commit/4331cc2f8e4a8096b9ccc86206a9b5431c096f9c
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M test/httpserver/testhttpserver.cpp

  Log Message:
  -----------
  Test that 100-continue headers are sent and received correctly

This tests that HTTPRequest::SendHeaders includes the "Expect: 100-continue"
header only when expected. A separate test calls HTTPRequest::SendWithStream
which sends these headers, and therefore checks that both sides correctly
handle such exchanges.

  Commit: f0240f49e6bce61fe1390dcee5c8be7ad7b1fa87
      https://github.com/boxbackup/boxbackup/commit/f0240f49e6bce61fe1390dcee5c8be7ad7b1fa87
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/backupstore/BackupAccountControl.h
    M lib/backupstore/BackupFileSystem.cpp
    M lib/backupstore/BackupFileSystem.h
    M lib/backupstore/BackupProtocol.h
    M lib/common/Test.cpp
    M lib/common/Test.h
    M lib/httpserver/S3Client.cpp
    M lib/server/makeprotocol.pl.in
    M modules.txt
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  Test that the S3 backend properly implements basic commands

This extends test_server_commands() to the S3 backend implementation, using
test specialisation.

  Commit: 6823cb30f4d035dcc9638d0ae57eeff0e0972fcc
      https://github.com/boxbackup/boxbackup/commit/6823cb30f4d035dcc9638d0ae57eeff0e0972fcc
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M infrastructure/buildenv-testmain-template.cpp

  Log Message:
  -----------
  Completely disable syslog logging in tests

(cherry picked from commit 3a8ca0c97e7346912889df72a1a87b80b430fe7a)

  Commit: 64b9db11b893a86fc57f50982ef66e4e32336730
      https://github.com/boxbackup/boxbackup/commit/64b9db11b893a86fc57f50982ef66e4e32336730
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    R infrastructure/cmake/build/bin_bbackupd.vcxproj.user
    R infrastructure/cmake/build/bin_bbstored.vcxproj.user
    R infrastructure/cmake/build/test_backupstore.vcxproj.user
    R infrastructure/cmake/build/test_backupstorefix.vcxproj.user
    R infrastructure/cmake/build/test_bbackupd.vcxproj.user
    R infrastructure/cmake/build/test_common.vcxproj.user
    R infrastructure/cmake/build/test_httpserver.vcxproj.user
    R infrastructure/cmake/build/test_raidfile.vcxproj.user

  Log Message:
  -----------
  Remove *.vcxproj.user from disposable CMake build directory

(cherry picked from commit 9df71c89a0fc935ee9965ddd8c4bcc2e1ad769a4)

  Commit: e5b4cb88370647f66e7b0b080d045626956ff0fb
      https://github.com/boxbackup/boxbackup/commit/e5b4cb88370647f66e7b0b080d045626956ff0fb
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M infrastructure/cmake/CMakeLists.txt
    M infrastructure/m4/boxbackup_tests.m4
    M lib/bbackupd/BackupClientContext.cpp
    M lib/bbackupd/BackupClientContext.h
    M lib/server/TcpNice.cpp
    M lib/server/TcpNice.h

  Log Message:
  -----------
  Add support for TcpNice on OSX, remove on Windows

I'm not convinced that it does anything on Windows without an RTT estimate,
which we can't get without administrator access:
https://msdn.microsoft.com/en-us/library/windows/desktop/bb485738(v=vs.85).aspx

(cherry picked from commit b646d22f415bbc8f5b5b065ed7462166591404f2)

  Commit: 2b19b5dc5c5dd785a0eefbc6bd5fc26387143ee0
      https://github.com/boxbackup/boxbackup/commit/2b19b5dc5c5dd785a0eefbc6bd5fc26387143ee0
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/backupstore/BackupAccountControl.cpp

  Log Message:
  -----------
  Move functions around to reduce diffs to s3_support

  Commit: b2b4ee40c44f233b99d18aaa005f160bf7208028
      https://github.com/boxbackup/boxbackup/commit/b2b4ee40c44f233b99d18aaa005f160bf7208028
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/backupstore/BackupAccountControl.cpp
    M lib/backupstore/BackupAccountControl.h
    M lib/win32/getopt_long.cpp

  Log Message:
  -----------
  Fix compile error in BSD getopt_long.cpp

error: '&&' within '||' [-Werror,-Wlogical-op-parentheses]

  Commit: da48c7198adce3f9c4ae407071c67a725c6362d2
      https://github.com/boxbackup/boxbackup/commit/da48c7198adce3f9c4ae407071c67a725c6362d2
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/httpserver/SimpleDBClient.h

  Log Message:
  -----------
  Fix SimpleDB timeouts: should be in milliseconds, not seconds

(cherry picked from commit 24bc49f30b9ce54d5024777f9c34d70a476d9663)

  Commit: 4d1ecde58c3e70fcebeaa1bf2cc8c600f9eeba7b
      https://github.com/boxbackup/boxbackup/commit/4d1ecde58c3e70fcebeaa1bf2cc8c600f9eeba7b
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/server/ServerStream.h
    M test/backupstore/testbackupstore.cpp

  Log Message:
  -----------
  ServerStream and test/backupstore: improve comments slightly

(cherry picked from commit c50aac67ee6ce6af9d63d5b96290585f2c5b7c21)

  Commit: 373f995aef15c6c8d97809149617efb8de1ce933
      https://github.com/boxbackup/boxbackup/commit/373f995aef15c6c8d97809149617efb8de1ce933
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/httpserver/S3Simulator.cpp

  Log Message:
  -----------
  S3Simulator: add some TRACE logging to help debug 100-continue responses

(cherry picked from commit ca39b88d4a73b64de68400dfa29270f203fd0f14)

  Commit: a3ab8bef278f0d7082033d98b8279dbecb570b30
      https://github.com/boxbackup/boxbackup/commit/a3ab8bef278f0d7082033d98b8279dbecb570b30
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/httpserver/S3Client.h

  Log Message:
  -----------
  Increase timeouts in S3Client to help with debugging

(cherry picked from commit cc14b3b4b9ec509233000c7d0a995580afd97f15)

  Commit: a391e7ca8a0e44d098058ea271e5f1e2d8046ca8
      https://github.com/boxbackup/boxbackup/commit/a391e7ca8a0e44d098058ea271e5f1e2d8046ca8
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/server/SocketStream.cpp

  Log Message:
  -----------
  SocketStream: use THROW_SOCKET_ERROR for better messages on Windows

(cherry picked from commit feddbef8052208508a2e542a1a70c23138f476b8)

  Commit: 1dbed47e9201e8702ada20fc44f4ab45b5fda6a2
      https://github.com/boxbackup/boxbackup/commit/1dbed47e9201e8702ada20fc44f4ab45b5fda6a2
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/common/Logging.cpp
    M lib/common/Logging.h

  Log Message:
  -----------
  Enable overriding log level by category as well as filename

This extends the -L command-line option to allow overriding by
filename/category=level as well as the existing support for overriding by
filename=level.

(cherry picked from commit 4421c53ab59adb494308f9c9f881a6a57bfd571c)

  Commit: 66b84e76001efba985462e6d9d37f7fb66719fc2
      https://github.com/boxbackup/boxbackup/commit/66b84e76001efba985462e6d9d37f7fb66719fc2
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/win32/emu.cpp

  Log Message:
  -----------
  Windows syslog() emulation: improve error messages

If syslog can't log to the Windows event log for some reason, instead of
just printing the log message to the console, include some explanatory
text about what went wrong.

(cherry picked from commit c492c90062d2e88db1016f13bd912629477a8a46)

  Commit: 0784dbe4a1ee33db00b11239b6eb08093fba586f
      https://github.com/boxbackup/boxbackup/commit/0784dbe4a1ee33db00b11239b6eb08093fba586f
  Author: Chris Wilson <chris+github at qwirx.com>
  Date:   2017-11-11 (Sat, 11 Nov 2017)

  Changed paths:
    M lib/backupstore/BackupFileSystem.cpp

  Log Message:
  -----------
  Fix rare crash while tearing down S3BackupFileSystem

ReleaseLock() could fail if it can't contact the S3/SimpleDB server, but
destructors aren't allowed to throw exceptions, so catch this case and log
a warning instead (because we failed to release the lock that we were
holding).

(cherry picked from commit 6226c014dd2988a6a29cd21f6cc29fa0e7816607)

Compare: https://github.com/boxbackup/boxbackup/compare/1eb2d9671456...0784dbe4a1ee