From boxbackup-dev at boxbackup.org Sun Mar 15 01:01:13 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sun, 15 Mar 2009 01:01:13 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2449 - box/trunk/test/httpserver Message-ID: <20090315010113.8B3E3326027@www.boxbackup.org> Author: chris Date: 2009-03-15 01:01:12 +0000 (Sun, 15 Mar 2009) New Revision: 2449 Modified: box/trunk/test/httpserver/testhttpserver.cpp Log: Fix compile error on FreeBSD due to missing #include of signal.h Modified: box/trunk/test/httpserver/testhttpserver.cpp =================================================================== --- box/trunk/test/httpserver/testhttpserver.cpp 2009-01-24 23:23:27 UTC (rev 2448) +++ box/trunk/test/httpserver/testhttpserver.cpp 2009-03-15 01:01:12 UTC (rev 2449) @@ -13,6 +13,10 @@ #include #include +#ifdef HAVE_SIGNAL_H + #include +#endif + #include #include "autogen_HTTPException.h" From boxbackup-dev at boxbackup.org Sun Mar 15 10:19:53 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sun, 15 Mar 2009 10:19:53 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2450 - box/boxbackup-web Message-ID: <20090315101953.E3A38326025@www.boxbackup.org> Author: chris Date: 2009-03-15 10:19:53 +0000 (Sun, 15 Mar 2009) New Revision: 2450 Modified: box/boxbackup-web/index.html Log: Link to Box Backup Explorer by Thimo Kraemer from website. Modified: box/boxbackup-web/index.html =================================================================== --- box/boxbackup-web/index.html 2009-03-15 01:01:12 UTC (rev 2449) +++ box/boxbackup-web/index.html 2009-03-15 10:19:53 UTC (rev 2450) @@ -164,9 +164,12 @@

There are a number of separate projects which are related to Box Backup. These include:

-

Boxi -- a cross platform GUI front end -for Box Backup.

+

Boxi -- a cross platform native GUI front end +for Box Backup, by Chris Wilson.

+

Box Backup Explorer -- +graphical web frontend for Box Backup, by Thimo Kraemer.

+

Download

The latest stable release is 0.10. We also have a From boxbackup-dev at boxbackup.org Sun Mar 15 20:20:37 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sun, 15 Mar 2009 20:20:37 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2451 - box/trunk/lib/server Message-ID: <20090315202037.D6DA0326025@www.boxbackup.org> Author: chris Date: 2009-03-15 20:20:37 +0000 (Sun, 15 Mar 2009) New Revision: 2451 Modified: box/trunk/lib/server/SocketStream.cpp Log: Fix crash when freebsd (maybe *bsd) returns ECONNRESET on close(), which presumably means that the socket is already closed, causing bbackupd test to fail. Thanks to Victor Meirans for reporting the problem. See http://wiki.powerdns.com/cgi-bin/trac.fcgi/changeset/1280 and http://wiki.powerdns.com/cgi-bin/trac.fcgi/ticket/192 for example. Modified: box/trunk/lib/server/SocketStream.cpp =================================================================== --- box/trunk/lib/server/SocketStream.cpp 2009-03-15 10:19:53 UTC (rev 2450) +++ box/trunk/lib/server/SocketStream.cpp 2009-03-15 20:20:37 UTC (rev 2451) @@ -368,7 +368,8 @@ #endif { BOX_LOG_SYS_ERROR("Failed to close socket"); - THROW_EXCEPTION(ServerException, SocketCloseError) + // don't throw an exception here, assume that the socket was + // already closed or closing. } mSocketHandle = INVALID_SOCKET_VALUE; } From boxbackup-dev at boxbackup.org Sun Mar 15 22:45:46 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sun, 15 Mar 2009 22:45:46 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2452 - box/trunk/test/bbackupd Message-ID: <20090315224546.258AC326025@www.boxbackup.org> Author: chris Date: 2009-03-15 22:45:45 +0000 (Sun, 15 Mar 2009) New Revision: 2452 Modified: box/trunk/test/bbackupd/testbbackupd.cpp Log: Fix race condition where bbackupd would run just before test finished waiting for its 90 second error timeout on slow machines, e.g. FreeBSD VM, causing bbackupd test to fail. Modified: box/trunk/test/bbackupd/testbbackupd.cpp =================================================================== --- box/trunk/test/bbackupd/testbbackupd.cpp 2009-03-15 20:20:37 UTC (rev 2451) +++ box/trunk/test/bbackupd/testbbackupd.cpp 2009-03-15 22:45:45 UTC (rev 2452) @@ -2618,6 +2618,8 @@ TEST_THAT(::rename("testfiles/0_2/backup/01234567/info.rf.bak", "testfiles/0_2/backup/01234567/info.rf") == 0); + int store_fixed_time = time(NULL); + // Check that we DO get errors on compare (cannot do this // until after we fix the store, which creates a race) compareReturnValue = ::system(BBACKUPQUERY " " @@ -2632,7 +2634,7 @@ TEST_THAT(!TestFileExists("testfiles/" "notifyran.backup-start.wait-snapshot.1")); - // Set a tag for the notify script to distinguist from + // Set a tag for the notify script to distinguish from // previous runs. { int fd1 = open("testfiles/notifyscript.tag", @@ -2642,8 +2644,9 @@ TEST_THAT(close(fd1) == 0); } - // bbackupd should pause for about 90 seconds - wait_for_backup_operation(85); + // bbackupd should pause for about 90 seconds from store_fixed_time, + // so check that it hasn't run after 85 seconds from store_fixed_time + wait_for_backup_operation(85 - time(NULL) + store_fixed_time); TEST_THAT(!TestFileExists("testfiles/" "notifyran.backup-start.wait-snapshot.1")); @@ -2713,6 +2716,8 @@ TEST_THAT(::rename("testfiles/0_2/backup/01234567/info.rf.bak", "testfiles/0_2/backup/01234567/info.rf") == 0); + store_fixed_time = time(NULL); + // Check that we DO get errors on compare (cannot do this // until after we fix the store, which creates a race) compareReturnValue = ::system(BBACKUPQUERY " " @@ -2737,8 +2742,9 @@ TEST_THAT(close(fd1) == 0); } - // bbackupd should pause for at least 90 seconds - wait_for_backup_operation(85); + // bbackupd should pause for about 90 seconds from store_fixed_time, + // so check that it hasn't run after 85 seconds from store_fixed_time + wait_for_backup_operation(85 - time(NULL) + store_fixed_time); TEST_THAT(!TestFileExists("testfiles/" "notifyran.backup-start.wait-automatic.1")); From boxbackup-dev at boxbackup.org Mon Mar 16 21:46:04 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Mon, 16 Mar 2009 21:46:04 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2453 - box/trunk/lib/httpserver Message-ID: <20090316214604.916A1326025@www.boxbackup.org> Author: chris Date: 2009-03-16 21:46:03 +0000 (Mon, 16 Mar 2009) New Revision: 2453 Modified: box/trunk/lib/httpserver/HTTPRequest.cpp Log: Fix type of string position variable to avoid failure (and warning) on platforms where size_t is signed, e.g. FreeBSD. Modified: box/trunk/lib/httpserver/HTTPRequest.cpp =================================================================== --- box/trunk/lib/httpserver/HTTPRequest.cpp 2009-03-15 22:45:45 UTC (rev 2452) +++ box/trunk/lib/httpserver/HTTPRequest.cpp 2009-03-16 21:46:03 UTC (rev 2453) @@ -121,10 +121,10 @@ // Didn't get the request line, probably end of connection which had been kept alive return false; } -// TRACE1("Request line: %s\n", requestLine.c_str()); + BOX_TRACE("Request line: " << requestLine); // Check the method - unsigned int p = 0; // current position in string + size_t p = 0; // current position in string p = requestLine.find(' '); // end of first word if (p == std::string::npos) From boxbackup-dev at boxbackup.org Mon Mar 16 21:46:22 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Mon, 16 Mar 2009 21:46:22 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2454 - box/trunk/test/httpserver Message-ID: <20090316214622.1FBE3326025@www.boxbackup.org> Author: chris Date: 2009-03-16 21:46:21 +0000 (Mon, 16 Mar 2009) New Revision: 2454 Modified: box/trunk/test/httpserver/testhttpserver.cpp Log: Add comment about running test as root. Modified: box/trunk/test/httpserver/testhttpserver.cpp =================================================================== --- box/trunk/test/httpserver/testhttpserver.cpp 2009-03-16 21:46:03 UTC (rev 2453) +++ box/trunk/test/httpserver/testhttpserver.cpp 2009-03-16 21:46:21 UTC (rev 2454) @@ -580,6 +580,8 @@ TEST_EQUAL(404, response.GetResponseCode()); } + // Make file inaccessible, should cause server to return a 403 error, + // unless of course the test is run as root :) { TEST_THAT(chmod("testfiles/testrequests.pl", 0) == 0); HTTPRequest request(HTTPRequest::Method_GET, From boxbackup-dev at boxbackup.org Wed Mar 18 21:02:33 2009 From: boxbackup-dev at boxbackup.org (Box Backup) Date: Wed, 18 Mar 2009 21:02:33 -0000 Subject: [Box Backup-commit] #55: Should store and preserve directory timestamps Message-ID: <042.d3558eb4997af452f53a3ec236d827e4@boxbackup.org> #55: Should store and preserve directory timestamps ---------------------------------------------+------------------------------ Reporter: chris | Owner: chris Type: defect | Status: new Priority: normal | Milestone: 0.20 Component: bbackupd | Version: 0.11rc2 Keywords: directory timestamp mtime ctime | ---------------------------------------------+------------------------------ See [http://lists.warhead.org.uk/pipermail/boxbackup/2008-February/004236.html original email] and [http://lists.warhead.org.uk/pipermail/boxbackup/2009-March/005034.html specific request] for directory timestamps to be saved. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at boxbackup.org Thu Mar 19 23:28:55 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Thu, 19 Mar 2009 23:28:55 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2455 - box/trunk/bin/bbackupd Message-ID: <20090319232855.D9E19326025@www.boxbackup.org> Author: chris Date: 2009-03-19 23:28:54 +0000 (Thu, 19 Mar 2009) New Revision: 2455 Modified: box/trunk/bin/bbackupd/BackupDaemon.cpp Log: Wrap long lines for readability. Modified: box/trunk/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/trunk/bin/bbackupd/BackupDaemon.cpp 2009-03-16 21:46:21 UTC (rev 2454) +++ box/trunk/bin/bbackupd/BackupDaemon.cpp 2009-03-19 23:28:54 UTC (rev 2455) @@ -697,9 +697,8 @@ if(mDeleteStoreObjectInfoFile && !DeleteStoreObjectInfo()) { - BOX_ERROR("Failed to delete the " - "StoreObjectInfoFile, backup cannot " - "continue safely."); + BOX_ERROR("Failed to delete the StoreObjectInfoFile, " + "backup cannot continue safely."); THROW_EXCEPTION(ClientException, FailedToDeleteStoreObjectInfoFile); } @@ -998,8 +997,9 @@ // // Function // Name: BackupDaemon::UseScriptToSeeIfSyncAllowed() -// Purpose: Private. Use a script to see if the sync should be allowed (if configured) -// Returns -1 if it's allowed, time in seconds to wait otherwise. +// Purpose: Private. Use a script to see if the sync should be +// allowed now (if configured). Returns -1 if it's +// allowed, time in seconds to wait otherwise. // Created: 21/6/04 // // -------------------------------------------------------------------------- @@ -1021,7 +1021,8 @@ pid_t pid = 0; try { - std::auto_ptr pscript(LocalProcessStream(conf.GetKeyValue("SyncAllowScript").c_str(), pid)); + std::auto_ptr pscript(LocalProcessStream( + conf.GetKeyValue("SyncAllowScript").c_str(), pid)); // Read in the result IOStreamGetLine getLine(*pscript); From boxbackup-dev at boxbackup.org Fri Mar 20 00:06:27 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Fri, 20 Mar 2009 00:06:27 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2456 - box/chris/win32/support Message-ID: <20090320000627.A67F6326025@www.boxbackup.org> Author: chris Date: 2009-03-20 00:06:26 +0000 (Fri, 20 Mar 2009) New Revision: 2456 Modified: box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch Log: Add fix for invalid static declarations for gcc 4.x compilation, thanks to Achim. Modified: box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch =================================================================== --- box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch 2009-03-19 23:28:54 UTC (rev 2455) +++ box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch 2009-03-20 00:06:26 UTC (rev 2456) @@ -322,3 +322,15 @@ +rm cert.tmp + +echo passed all tests +diff -ru openssl-0.9.8b/include/openssl/e_os2.h openssl-0.9.8b-mingw/include/openssl/e_os2.h +--- openssl-0.9.8b/include/openssl/e_os2.h 2009-03-20 00:02:19.000000000 +0000 ++++ openssl-0.9.8b-mingw/include/openssl/e_os2.h 2009-03-20 00:02:34.000000000 +0000 +@@ -264,7 +264,7 @@ + # define OPENSSL_IMPLEMENT_GLOBAL(type,name) \ + extern type _hide_##name; \ + type *_shadow_##name(void) { return &_hide_##name; } \ +- static type _hide_##name ++ type _hide_##name + # define OPENSSL_DECLARE_GLOBAL(type,name) type *_shadow_##name(void) + # define OPENSSL_GLOBAL_REF(name) (*(_shadow_##name())) + #else From boxbackup-dev at boxbackup.org Fri Mar 20 00:42:54 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Fri, 20 Mar 2009 00:42:54 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2457 - box/trunk Message-ID: <20090320004254.76C88326025@www.boxbackup.org> Author: chris Date: 2009-03-20 00:42:54 +0000 (Fri, 20 Mar 2009) New Revision: 2457 Modified: box/trunk/parcels.txt Log: We no longer require mgwz.dll as we build zlib as a static library. Modified: box/trunk/parcels.txt =================================================================== --- box/trunk/parcels.txt 2009-03-20 00:06:26 UTC (rev 2456) +++ box/trunk/parcels.txt 2009-03-20 00:42:54 UTC (rev 2457) @@ -22,7 +22,6 @@ END-ONLY ONLY:mingw32 - script /bin/mgwz.dll script /bin/mingwm10.dll END-ONLY From boxbackup-dev at boxbackup.org Fri Mar 20 00:52:04 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Fri, 20 Mar 2009 00:52:04 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2458 - box/trunk/docs/api-docs/backup Message-ID: <20090320005204.ACFCB326044@www.boxbackup.org> Author: chris Date: 2009-03-20 00:52:03 +0000 (Fri, 20 Mar 2009) New Revision: 2458 Modified: box/trunk/docs/api-docs/backup/win32_build_on_linux_using_mingw.txt Log: Note that Wine and binfmts are required to cross-compile successfully. Note that you have to find mingwm10.dll yourself, and where to find it on Debian/Ubuntu, and to modify parcels.txt to match. Modified: box/trunk/docs/api-docs/backup/win32_build_on_linux_using_mingw.txt =================================================================== --- box/trunk/docs/api-docs/backup/win32_build_on_linux_using_mingw.txt 2009-03-20 00:42:54 UTC (rev 2457) +++ box/trunk/docs/api-docs/backup/win32_build_on_linux_using_mingw.txt 2009-03-20 00:52:03 UTC (rev 2458) @@ -3,7 +3,7 @@ Install the MinGW cross-compiler for Windows: -- Debian users can "apt-get install mingw32" +- Debian and Ubuntu users can "apt-get install mingw32" - Fedora and SuSE users can download RPM packages from [http://mirzam.it.vu.nl/mingw/] @@ -12,8 +12,17 @@ cross-compiler package will start with this prefix. The documentation below assumes that it is "i386-mingw32-". Adjust to taste. -Download Zlib from [http://www.zlib.net/], unpack and enter source directory: +You will also need to install Wine and the Linux kernel "binary formats" +(binfmt) support, so that you can run Windows executables on Linux, +otherwise the configure scripts will not work properly with a cross-compiler. +On Ubuntu, run: + apt-get install wine binfmt-support + /etc/init.d/binfmt-support start + +Start by downloading Zlib from [http://www.zlib.net/], unpack and enter +source directory: + export CC=i386-mingw32-gcc export AR="i386-mingw32-ar rc" export RANLIB="i386-mingw32-ranlib" @@ -48,11 +57,36 @@ export AR=i386-mingw32-ar ./configure --host=i386-mingw32 --prefix=/usr/local/i386-mingw32/ make winshared + +If you get this error: + + ./dftables.exe pcre_chartables.c + /bin/bash: ./dftables.exe: cannot execute binary file + make: *** [pcre_chartables.c] Error 126 + +then run: + + wine ./dftables.exe pcre_chartables.c + make winshared + +to complete the build. Finally: + cp .libs/libpcreposix.a /usr/local/i386-pc-mingw32/lib cp pcreposix.h /usr/local/i386-pc-mingw32/include -Configure Box with: +You will need to find a copy of mingwm10.dll that matches your cross-compiler. +Most MinGW distributions should come with it. On Debian and Ubuntu, for some +bizarre reason, you'll find it compressed as +/usr/share/doc/mingw32-runtime/mingwm10.dll.gz, in which case you'll +have to un-gzip it with "gzip -d". Copy it to a known location, e.g. +/usr/local/i386-mingw32/bin. +Download and extract Box Backup, and change into the base directory, +e.g. boxbackup-0.11rc2. Change the path to mingwm10.dll in parcels.txt to +match where you found or installed it. + +Now configure Box with: + ./configure --host=i386-mingw32 \ CXXFLAGS="-mthreads -I/usr/local/i386-mingw32/include" \ LDFLAGS=" -mthreads -L/usr/local/i386-mingw32/lib" \ From boxbackup-dev at boxbackup.org Fri Mar 20 18:33:17 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Fri, 20 Mar 2009 18:33:17 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2459 - box/RELEASE Message-ID: <20090320183317.1A362326025@www.boxbackup.org> Author: chris Date: 2009-03-20 18:33:16 +0000 (Fri, 20 Mar 2009) New Revision: 2459 Added: box/RELEASE/0.11rc3/ Log: Create a new branch for 0.11rc3 From boxbackup-dev at boxbackup.org Sat Mar 21 18:48:20 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sat, 21 Mar 2009 18:48:20 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2460 - in box/trunk: bin/bbackupd bin/bbackupquery lib/backupclient lib/backupstore lib/common lib/raidfile lib/server lib/win32 test/bbackupd Message-ID: <20090321184820.4EB94325FF8@www.boxbackup.org> Author: chris Date: 2009-03-21 18:48:19 +0000 (Sat, 21 Mar 2009) New Revision: 2460 Modified: box/trunk/bin/bbackupd/BackupClientDirectoryRecord.cpp box/trunk/bin/bbackupd/Win32ServiceFunctions.cpp box/trunk/bin/bbackupquery/BackupQueries.cpp box/trunk/lib/backupclient/BackupClientFileAttributes.cpp box/trunk/lib/backupclient/BackupClientFileAttributes.h box/trunk/lib/backupclient/BackupStoreFile.cpp box/trunk/lib/backupclient/BackupStoreFileDiff.cpp box/trunk/lib/backupstore/BackupStoreAccountDatabase.cpp box/trunk/lib/common/BoxPlatform.h box/trunk/lib/common/FileModificationTime.h box/trunk/lib/common/FileStream.cpp box/trunk/lib/common/Test.cpp box/trunk/lib/common/Utils.cpp box/trunk/lib/raidfile/RaidFileUtil.cpp box/trunk/lib/server/Daemon.cpp box/trunk/lib/win32/emu.cpp box/trunk/lib/win32/emu.h box/trunk/test/bbackupd/testbbackupd.cpp Log: Fix tests (hopefully) on Win32 for struct stat ino_t change from 16 to 64 bits. Modified: box/trunk/bin/bbackupd/BackupClientDirectoryRecord.cpp =================================================================== --- box/trunk/bin/bbackupd/BackupClientDirectoryRecord.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/bin/bbackupd/BackupClientDirectoryRecord.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -149,12 +149,12 @@ // so byte order isn't considered. MD5Digest currentStateChecksum; - struct stat dest_st; + EMU_STRUCT_STAT dest_st; // Stat the directory, to get attribute info // If it's a symbolic link, we want the link target here // (as we're about to back up the contents of the directory) { - if(::stat(rLocalPath.c_str(), &dest_st) != 0) + if(EMU_STAT(rLocalPath.c_str(), &dest_st) != 0) { // The directory has probably been deleted, so // just ignore this error. In a future scan, this @@ -199,8 +199,8 @@ std::vector files; bool downloadDirectoryRecordBecauseOfFutureFiles = false; - struct stat link_st; - if(::lstat(rLocalPath.c_str(), &link_st) != 0) + EMU_STRUCT_STAT link_st; + if(EMU_LSTAT(rLocalPath.c_str(), &link_st) != 0) { // Report the error (logs and // eventual email to administrator) @@ -258,7 +258,7 @@ ::memset(&checksum_info, 0, sizeof(checksum_info)); struct dirent *en = 0; - struct stat file_st; + EMU_STRUCT_STAT file_st; std::string filename; while((en = ::readdir(dirHandle)) != 0) { @@ -292,7 +292,7 @@ // prefer S_IFREG, S_IFDIR... int type = en->d_type; #else - if(::lstat(filename.c_str(), &file_st) != 0) + if(EMU_LSTAT(filename.c_str(), &file_st) != 0) { // Report the error (logs and // eventual email to administrator) @@ -387,7 +387,7 @@ #ifdef WIN32 // We didn't stat the file before, // but now we need the information. - if(::lstat(filename.c_str(), &file_st) != 0) + if(emu_stat(filename.c_str(), &file_st) != 0) { rNotifier.NotifyFileStatFailed(this, filename, @@ -687,8 +687,8 @@ // BLOCK { // Stat the file - struct stat st; - if(::lstat(filename.c_str(), &st) != 0) + EMU_STRUCT_STAT st; + if(EMU_LSTAT(filename.c_str(), &st) != 0) { rNotifier.NotifyFileStatFailed(this, filename, strerror(errno)); @@ -757,8 +757,8 @@ if(!isDir && isCurrentVersion) { // Check that the object we found in the ID map doesn't exist on disc - struct stat st; - if(::stat(localPotentialOldName.c_str(), &st) != 0 && errno == ENOENT) + EMU_STRUCT_STAT st; + if(EMU_STAT(localPotentialOldName.c_str(), &st) != 0 && errno == ENOENT) { // Doesn't exist locally, but does exist on the server. // Therefore we can safely rename it to this new file. @@ -1278,8 +1278,8 @@ if(isDir && isCurrentVersion) { // Check that the object doesn't exist already - struct stat st; - if(::stat(localPotentialOldName.c_str(), &st) != 0 && errno == ENOENT) + EMU_STRUCT_STAT st; + if(EMU_STAT(localPotentialOldName.c_str(), &st) != 0 && errno == ENOENT) { // Doesn't exist locally, but does exist on the server. // Therefore we can safely rename it. Modified: box/trunk/bin/bbackupd/Win32ServiceFunctions.cpp =================================================================== --- box/trunk/bin/bbackupd/Win32ServiceFunctions.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/bin/bbackupd/Win32ServiceFunctions.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -203,7 +203,7 @@ { if (pConfigFileName != NULL) { - struct stat st; + EMU_STRUCT_STAT st; if (emu_stat(pConfigFileName, &st) != 0) { Modified: box/trunk/bin/bbackupquery/BackupQueries.cpp =================================================================== --- box/trunk/bin/bbackupquery/BackupQueries.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/bin/bbackupquery/BackupQueries.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -914,8 +914,8 @@ } // Does file exist? - struct stat st; - if(::stat(args[1].c_str(), &st) == 0 || errno != ENOENT) + EMU_STRUCT_STAT st; + if(EMU_STAT(args[1].c_str(), &st) == 0 || errno != ENOENT) { BOX_ERROR("The local file '" << args[1] << " already exists."); return; @@ -1146,8 +1146,8 @@ } // Does local file already exist? (don't want to overwrite) - struct stat st; - if(::stat(localName.c_str(), &st) == 0 || errno != ENOENT) + EMU_STRUCT_STAT st; + if(EMU_STAT(localName.c_str(), &st) == 0 || errno != ENOENT) { BOX_ERROR("The local file " << localName << " already exists, " "will not overwrite it."); @@ -1227,8 +1227,8 @@ std::string syncTimeFilename(mrConfiguration.GetKeyValue("DataDirectory") + DIRECTORY_SEPARATOR_ASCHAR); syncTimeFilename += "last_sync_start"; // Stat it to get file time - struct stat st; - if(::stat(syncTimeFilename.c_str(), &st) == 0) + EMU_STRUCT_STAT st; + if(EMU_STAT(syncTimeFilename.c_str(), &st) == 0) { // Files modified after this time shouldn't be on the server, so report errors slightly differently LatestFileUploadTime = FileModificationTime(st) - @@ -1404,8 +1404,8 @@ { bool modifiedAfterLastSync = false; - struct stat st; - if(::stat(rLocalDir.c_str(), &st) == 0) + EMU_STRUCT_STAT st; + if(EMU_STAT(rLocalDir.c_str(), &st) == 0) { if(FileAttrModificationTime(st) > rParams.LatestFileUploadTime()) @@ -1439,8 +1439,8 @@ rParams.NotifyDirComparing(rLocalDir, rStoreDir); // Get info on the local directory - struct stat st; - if(::lstat(rLocalDir.c_str(), &st) != 0) + EMU_STRUCT_STAT st; + if(EMU_LSTAT(rLocalDir.c_str(), &st) != 0) { // What kind of error? if(errno == ENOTDIR || errno == ENOENT) @@ -1494,8 +1494,8 @@ { bool modifiedAfterLastSync = false; - struct stat st; - if(::stat(rLocalDir.c_str(), &st) == 0) + EMU_STRUCT_STAT st; + if(EMU_STAT(rLocalDir.c_str(), &st) == 0) { if(FileAttrModificationTime(st) > rParams.LatestFileUploadTime()) @@ -1547,8 +1547,8 @@ #ifndef HAVE_VALID_DIRENT_D_TYPE std::string fn(MakeFullPath (rLocalDir, localDirEn->d_name)); - struct stat st; - if(::lstat(fn.c_str(), &st) != 0) + EMU_STRUCT_STAT st; + if(EMU_LSTAT(fn.c_str(), &st) != 0) { THROW_EXCEPTION(CommonException, OSFileError) } @@ -1638,8 +1638,8 @@ { int64_t fileSize = 0; - struct stat st; - if(::stat(localPath.c_str(), &st) == 0) + EMU_STRUCT_STAT st; + if(EMU_STAT(localPath.c_str(), &st) == 0) { fileSize = st.st_size; } @@ -1771,8 +1771,8 @@ { bool modifiedAfterLastSync = false; - struct stat st; - if(::stat(localPath.c_str(), &st) == 0) + EMU_STRUCT_STAT st; + if(EMU_STAT(localPath.c_str(), &st) == 0) { if(FileModificationTime(st) > rParams.LatestFileUploadTime()) @@ -1845,8 +1845,8 @@ bool modifiedAfterLastSync = false; // Check the dir modification time - struct stat st; - if(::stat(localPath.c_str(), &st) == 0 && + EMU_STRUCT_STAT st; + if(EMU_STAT(localPath.c_str(), &st) == 0 && FileModificationTime(st) > rParams.LatestFileUploadTime()) { Modified: box/trunk/lib/backupclient/BackupClientFileAttributes.cpp =================================================================== --- box/trunk/lib/backupclient/BackupClientFileAttributes.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/backupclient/BackupClientFileAttributes.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -302,8 +302,8 @@ StreamableMemBlock *pnewAttr = 0; try { - struct stat st; - if(::lstat(Filename, &st) != 0) + EMU_STRUCT_STAT st; + if(EMU_LSTAT(Filename, &st) != 0) { BOX_LOG_SYS_ERROR("Failed to stat file: '" << Filename << "'"); @@ -395,7 +395,7 @@ // Created: 2003/10/07 // // -------------------------------------------------------------------------- -void BackupClientFileAttributes::FillAttributes(StreamableMemBlock &outputBlock, const char *Filename, struct stat &st, bool ZeroModificationTimes) +void BackupClientFileAttributes::FillAttributes(StreamableMemBlock &outputBlock, const char *Filename, EMU_STRUCT_STAT &st, bool ZeroModificationTimes) { outputBlock.ResizeBlock(sizeof(attr_StreamFormat)); attr_StreamFormat *pattr = (attr_StreamFormat*)outputBlock.GetBuffer(); @@ -1037,7 +1037,7 @@ // Created: 25/4/04 // // -------------------------------------------------------------------------- -uint64_t BackupClientFileAttributes::GenerateAttributeHash(struct stat &st, const std::string &filename, const std::string &leafname) +uint64_t BackupClientFileAttributes::GenerateAttributeHash(EMU_STRUCT_STAT &st, const std::string &filename, const std::string &leafname) { if(sAttributeHashSecretLength == 0) { Modified: box/trunk/lib/backupclient/BackupClientFileAttributes.h =================================================================== --- box/trunk/lib/backupclient/BackupClientFileAttributes.h 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/backupclient/BackupClientFileAttributes.h 2009-03-21 18:48:19 UTC (rev 2460) @@ -15,7 +15,7 @@ #include "StreamableMemBlock.h" #include "BoxTime.h" -struct stat; +EMU_STRUCT_STAT; // declaration // -------------------------------------------------------------------------- // @@ -53,11 +53,13 @@ static void SetBlowfishKey(const void *pKey, int KeyLength); static void SetAttributeHashSecret(const void *pSecret, int SecretLength); - static uint64_t GenerateAttributeHash(struct stat &st, const std::string &filename, const std::string &leafname); + static uint64_t GenerateAttributeHash(EMU_STRUCT_STAT &st, const std::string &filename, const std::string &leafname); static void FillExtendedAttr(StreamableMemBlock &outputBlock, const char *Filename); private: - static void FillAttributes(StreamableMemBlock &outputBlock, const char *Filename, struct stat &st, bool ZeroModificationTimes); + static void FillAttributes(StreamableMemBlock &outputBlock, + const char *Filename, EMU_STRUCT_STAT &st, + bool ZeroModificationTimes); static void FillAttributesLink(StreamableMemBlock &outputBlock, const char *Filename, struct stat &st); void WriteExtendedAttr(const char *Filename, int xattrOffset) const; Modified: box/trunk/lib/backupclient/BackupStoreFile.cpp =================================================================== --- box/trunk/lib/backupclient/BackupStoreFile.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/backupclient/BackupStoreFile.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -272,8 +272,8 @@ void BackupStoreFile::DecodeFile(IOStream &rEncodedFile, const char *DecodedFilename, int Timeout, const BackupClientFileAttributes *pAlterativeAttr) { // Does file exist? - struct stat st; - if(::stat(DecodedFilename, &st) == 0) + EMU_STRUCT_STAT st; + if(EMU_STAT(DecodedFilename, &st) == 0) { THROW_EXCEPTION(BackupStoreException, OutputFileAlreadyExists) } @@ -1260,8 +1260,8 @@ // is it a symlink? bool sourceIsSymlink = false; { - struct stat st; - if(::lstat(Filename, &st) == -1) + EMU_STRUCT_STAT st; + if(EMU_LSTAT(Filename, &st) == -1) { THROW_EXCEPTION(CommonException, OSFileError) } Modified: box/trunk/lib/backupclient/BackupStoreFileDiff.cpp =================================================================== --- box/trunk/lib/backupclient/BackupStoreFileDiff.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/backupclient/BackupStoreFileDiff.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -130,8 +130,8 @@ { // Is it a symlink? { - struct stat st; - if(::lstat(Filename, &st) != 0) + EMU_STRUCT_STAT st; + if(EMU_LSTAT(Filename, &st) != 0) { THROW_EXCEPTION(CommonException, OSFileError) } Modified: box/trunk/lib/backupstore/BackupStoreAccountDatabase.cpp =================================================================== --- box/trunk/lib/backupstore/BackupStoreAccountDatabase.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/backupstore/BackupStoreAccountDatabase.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -208,8 +208,8 @@ // -------------------------------------------------------------------------- box_time_t BackupStoreAccountDatabase::GetDBFileModificationTime() const { - struct stat st; - if(::stat(pImpl->mFilename.c_str(), &st) == -1) + EMU_STRUCT_STAT st; + if(EMU_STAT(pImpl->mFilename.c_str(), &st) == -1) { THROW_EXCEPTION(CommonException, OSFileError) } Modified: box/trunk/lib/common/BoxPlatform.h =================================================================== --- box/trunk/lib/common/BoxPlatform.h 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/common/BoxPlatform.h 2009-03-21 18:48:19 UTC (rev 2460) @@ -38,10 +38,6 @@ // must define this before importing #define __MSVCRT_VERSION__ 0x0601 #endif - - // stop sys/types.h from defining its own ino_t as short, - // because we want a bigger one :) - #define _INO_T_ #endif #ifdef HAVE_SYS_TYPES_H Modified: box/trunk/lib/common/FileModificationTime.h =================================================================== --- box/trunk/lib/common/FileModificationTime.h 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/common/FileModificationTime.h 2009-03-21 18:48:19 UTC (rev 2460) @@ -14,7 +14,7 @@ #include "BoxTime.h" -inline box_time_t FileModificationTime(struct stat &st) +inline box_time_t FileModificationTime(EMU_STRUCT_STAT &st) { #ifndef HAVE_STRUCT_STAT_ST_MTIMESPEC box_time_t datamodified = ((int64_t)st.st_mtime) * (MICRO_SEC_IN_SEC_LL); @@ -26,7 +26,7 @@ return datamodified; } -inline box_time_t FileAttrModificationTime(struct stat &st) +inline box_time_t FileAttrModificationTime(EMU_STRUCT_STAT &st) { box_time_t statusmodified = #ifdef HAVE_STRUCT_STAT_ST_MTIMESPEC @@ -45,7 +45,7 @@ return statusmodified; } -inline box_time_t FileModificationTimeMaxModAndAttr(struct stat &st) +inline box_time_t FileModificationTimeMaxModAndAttr(EMU_STRUCT_STAT &st) { #ifndef HAVE_STRUCT_STAT_ST_MTIMESPEC box_time_t datamodified = ((int64_t)st.st_mtime) * (MICRO_SEC_IN_SEC_LL); Modified: box/trunk/lib/common/FileStream.cpp =================================================================== --- box/trunk/lib/common/FileStream.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/common/FileStream.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -224,8 +224,8 @@ // -------------------------------------------------------------------------- IOStream::pos_type FileStream::BytesLeftToRead() { - struct stat st; - if(::fstat(mOSFileHandle, &st) != 0) + EMU_STRUCT_STAT st; + if(EMU_FSTAT(mOSFileHandle, &st) != 0) { THROW_EXCEPTION(CommonException, OSFileError) } Modified: box/trunk/lib/common/Test.cpp =================================================================== --- box/trunk/lib/common/Test.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/common/Test.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -25,28 +25,28 @@ bool TestFileExists(const char *Filename) { - struct stat st; - return ::stat(Filename, &st) == 0 && (st.st_mode & S_IFDIR) == 0; + EMU_STRUCT_STAT st; + return EMU_STAT(Filename, &st) == 0 && (st.st_mode & S_IFDIR) == 0; } bool TestFileNotEmpty(const char *Filename) { - struct stat st; - return ::stat(Filename, &st) == 0 && (st.st_mode & S_IFDIR) == 0 && + EMU_STRUCT_STAT st; + return EMU_STAT(Filename, &st) == 0 && (st.st_mode & S_IFDIR) == 0 && st.st_size > 0; } bool TestDirExists(const char *Filename) { - struct stat st; - return ::stat(Filename, &st) == 0 && (st.st_mode & S_IFDIR) == S_IFDIR; + EMU_STRUCT_STAT st; + return EMU_STAT(Filename, &st) == 0 && (st.st_mode & S_IFDIR) == S_IFDIR; } // -1 if doesn't exist int TestGetFileSize(const char *Filename) { - struct stat st; - if(::stat(Filename, &st) == 0) + EMU_STRUCT_STAT st; + if(EMU_STAT(Filename, &st) == 0) { return st.st_size; } Modified: box/trunk/lib/common/Utils.cpp =================================================================== --- box/trunk/lib/common/Utils.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/common/Utils.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -163,8 +163,8 @@ // -------------------------------------------------------------------------- bool FileExists(const char *Filename, int64_t *pFileSize, bool TreatLinksAsNotExisting) { - struct stat st; - if(::lstat(Filename, &st) != 0) + EMU_STRUCT_STAT st; + if(EMU_LSTAT(Filename, &st) != 0) { if(errno == ENOENT) { @@ -208,8 +208,8 @@ // -------------------------------------------------------------------------- int ObjectExists(const std::string& rFilename) { - struct stat st; - if(::stat(rFilename.c_str(), &st) != 0) + EMU_STRUCT_STAT st; + if(EMU_STAT(rFilename.c_str(), &st) != 0) { if(errno == ENOENT) { Modified: box/trunk/lib/raidfile/RaidFileUtil.cpp =================================================================== --- box/trunk/lib/raidfile/RaidFileUtil.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/raidfile/RaidFileUtil.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -57,7 +57,12 @@ // Get unique ID if(pRevisionID != 0) { - (*pRevisionID) = FileModificationTime(st); + #ifdef WIN32 + *pRevisionID = st.st_mtime; + #else + *pRevisionID = FileModificationTime(st); + #endif + #ifdef BOX_RELEASE_BUILD // The resolution of timestamps may be very // low, e.g. 1 second. So add the size to it @@ -98,7 +103,12 @@ // Revision ID if(pRevisionID != 0) { - int64_t rid = FileModificationTime(st); + #ifdef WIN32 + int64_t rid = st.st_mtime; + #else + int64_t rid = FileModificationTime(st); + #endif + if(rid > revisionID) revisionID = rid; revisionIDplus += st.st_size; } Modified: box/trunk/lib/server/Daemon.cpp =================================================================== --- box/trunk/lib/server/Daemon.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/server/Daemon.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -962,9 +962,9 @@ box_time_t Daemon::GetConfigFileModifiedTime() const { - struct stat st; + EMU_STRUCT_STAT st; - if(::stat(GetConfigFileName().c_str(), &st) != 0) + if(EMU_STAT(GetConfigFileName().c_str(), &st) != 0) { if (errno == ENOENT) { Modified: box/trunk/lib/win32/emu.cpp =================================================================== --- box/trunk/lib/win32/emu.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/win32/emu.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -21,149 +21,9 @@ #include // message resource definitions for syslog() - #include "messages.h" -// our implementation for a timer, based on a -// simple thread which sleeps for a period of time - -static bool gTimerInitialised = false; -static bool gFinishTimer; -static CRITICAL_SECTION gLock; - DWORD winerrno; - -typedef struct -{ - int countDown; - int interval; -} -Timer_t; - -std::list gTimerList; -static void (__cdecl *gTimerFunc) (int) = NULL; - -int setitimer(int type, struct itimerval *timeout, void *arg) -{ - assert(gTimerInitialised); - - if (ITIMER_REAL != type) - { - errno = ENOSYS; - return -1; - } - - EnterCriticalSection(&gLock); - - // we only need seconds for the mo! - if (timeout->it_value.tv_sec == 0 && - timeout->it_value.tv_usec == 0) - { - gTimerList.clear(); - } - else - { - Timer_t ourTimer; - ourTimer.countDown = timeout->it_value.tv_sec; - ourTimer.interval = timeout->it_interval.tv_sec; - gTimerList.push_back(ourTimer); - } - - LeaveCriticalSection(&gLock); - - // indicate success - return 0; -} - -static unsigned int WINAPI RunTimer(LPVOID lpParameter) -{ - gFinishTimer = false; - - while (!gFinishTimer) - { - std::list::iterator it; - EnterCriticalSection(&gLock); - - for (it = gTimerList.begin(); it != gTimerList.end(); it++) - { - Timer_t& rTimer(*it); - - rTimer.countDown --; - if (rTimer.countDown == 0) - { - if (gTimerFunc != NULL) - { - gTimerFunc(0); - } - if (rTimer.interval) - { - rTimer.countDown = rTimer.interval; - } - else - { - // mark for deletion - rTimer.countDown = -1; - } - } - } - - for (it = gTimerList.begin(); it != gTimerList.end(); it++) - { - Timer_t& rTimer(*it); - - if (rTimer.countDown == -1) - { - gTimerList.erase(it); - - // the iterator is now invalid, so restart search - it = gTimerList.begin(); - - // if the list is now empty, don't try to increment - // the iterator again - if (it == gTimerList.end()) break; - } - } - - LeaveCriticalSection(&gLock); - // we only need to have a 1 second resolution - Sleep(1000); - } - - return 0; -} - -int SetTimerHandler(void (__cdecl *func ) (int)) -{ - gTimerFunc = func; - return 0; -} - -void InitTimer(void) -{ - assert(!gTimerInitialised); - - InitializeCriticalSection(&gLock); - - // create our thread - HANDLE ourThread = (HANDLE)_beginthreadex(NULL, 0, RunTimer, 0, - CREATE_SUSPENDED, NULL); - SetThreadPriority(ourThread, THREAD_PRIORITY_LOWEST); - ResumeThread(ourThread); - - gTimerInitialised = true; -} - -void FiniTimer(void) -{ - assert(gTimerInitialised); - gFinishTimer = true; - EnterCriticalSection(&gLock); - DeleteCriticalSection(&gLock); - gTimerInitialised = false; -} - -//Our constants we need to keep track of -//globals struct passwd gTempPasswd; bool EnableBackupRights() @@ -715,11 +575,13 @@ // // Function // Name: emu_fstat -// Purpose: replacement for fstat supply a windows handle +// Purpose: replacement for fstat. Supply a windows handle. +// Returns a struct emu_stat to have room for 64-bit +// file identifier in st_ino (mingw allows only 16!) // Created: 25th October 2004 // // -------------------------------------------------------------------------- -int emu_fstat(HANDLE hdir, struct stat * st) +int emu_fstat(HANDLE hdir, struct emu_stat * st) { if (hdir == INVALID_HANDLE_VALUE) { @@ -751,7 +613,7 @@ ULARGE_INTEGER conv; conv.HighPart = fi.nFileIndexHigh; conv.LowPart = fi.nFileIndexLow; - st->st_ino = (_ino_t)conv.QuadPart; + st->st_ino = conv.QuadPart; // get the time information st->st_ctime = ConvertFileTimeToTime_t(&fi.ftCreationTime); @@ -898,12 +760,14 @@ // // Function // Name: emu_stat -// Purpose: replacement for the lstat and stat functions, -// works with unicode filenames supplied in utf8 format +// Purpose: replacement for the lstat and stat functions. +// Works with unicode filenames supplied in utf8. +// Returns a struct emu_stat to have room for 64-bit +// file identifier in st_ino (mingw allows only 16!) // Created: 25th October 2004 // // -------------------------------------------------------------------------- -int emu_stat(const char * pName, struct stat * st) +int emu_stat(const char * pName, struct emu_stat * st) { HANDLE handle = OpenFileByNameUtf8(pName, FILE_READ_ATTRIBUTES | FILE_READ_EA); Modified: box/trunk/lib/win32/emu.h =================================================================== --- box/trunk/lib/win32/emu.h 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/lib/win32/emu.h 2009-03-21 18:48:19 UTC (rev 2460) @@ -1,5 +1,17 @@ // emulates unix syscalls to win32 functions +#ifdef WIN32 + #define EMU_STRUCT_STAT struct emu_stat + #define EMU_STAT emu_stat + #define EMU_FSTAT emu_fstat + #define EMU_LSTAT emu_stat +#else + #define EMU_STRUCT_STAT struct stat + #define EMU_STAT ::stat + #define EMU_FSTAT ::fstat + #define EMU_LSTAT ::lstat +#endif + #if ! defined EMU_INCLUDE && defined WIN32 #define EMU_INCLUDE @@ -25,17 +37,9 @@ #ifdef __MINGW32__ typedef uint32_t u_int32_t; - typedef uint64_t _ino_t; - typedef _ino_t ino_t; - #define _INO_T_ #else typedef unsigned int mode_t; typedef unsigned int pid_t; - - // must define _INO_T_DEFINED before including - // to replace it with our own. - typedef u_int64_t _ino_t; - #define _INO_T_DEFINED #endif // set up to include the necessary parts of Windows headers @@ -81,11 +85,6 @@ #define fileno(struct_file) _fileno(struct_file) #endif -int SetTimerHandler(void (__cdecl *func ) (int)); -int setitimer(int type, struct itimerval *timeout, void *arg); -void InitTimer(void); -void FiniTimer(void); - struct passwd { char *pw_name; char *pw_passwd; @@ -187,13 +186,6 @@ #define timespec timeval -//not available in win32 -struct itimerval -{ - timeval it_interval; - timeval it_value; -}; - //win32 deals in usec not nsec - so need to ensure this follows through #define tv_nsec tv_usec @@ -313,27 +305,19 @@ TCHAR f_mntonname[MAX_PATH]; }; -#if 0 -// I think this should get us going -// Although there is a warning about -// mount points in win32 can now exists - which means inode number can be -// duplicated, so potential of a problem - perhaps this needs to be -// implemented with a little more thought... TODO - -struct stat { - //_dev_t st_dev; - u_int64_t st_ino; +struct emu_stat { + int st_dev; + uint64_t st_ino; DWORD st_mode; short st_nlink; short st_uid; short st_gid; //_dev_t st_rdev; - u_int64_t st_size; + uint64_t st_size; time_t st_atime; time_t st_mtime; time_t st_ctime; }; -#endif // 0 // need this for conversions time_t ConvertFileTimeToTime_t(FILETIME *fileTime); @@ -342,8 +326,8 @@ int emu_chdir (const char* pDirName); int emu_mkdir (const char* pPathName); int emu_unlink (const char* pFileName); -int emu_fstat (HANDLE file, struct stat* st); -int emu_stat (const char* pName, struct stat* st); +int emu_fstat (HANDLE file, struct emu_stat* st); +int emu_stat (const char* pName, struct emu_stat* st); int emu_utimes (const char* pName, const struct timeval[]); int emu_chmod (const char* pName, mode_t mode); char* emu_getcwd (char* pBuffer, int BufSize); @@ -352,14 +336,22 @@ #define chdir(directory) emu_chdir (directory) #define mkdir(path, mode) emu_mkdir (path) #define unlink(file) emu_unlink (file) -#define stat(filename, struct) emu_stat (filename, struct) -#define lstat(filename, struct) emu_stat (filename, struct) -#define fstat(handle, struct) emu_fstat (handle, struct) #define utimes(buffer, times) emu_utimes (buffer, times) #define chmod(file, mode) emu_chmod (file, mode) #define getcwd(buffer, size) emu_getcwd (buffer, size) #define rename(oldname, newname) emu_rename (oldname, newname) +// Not safe to replace stat/fstat/lstat on mingw at least, as struct stat +// has a 16-bit st_ino and we need a 64-bit one. +// +// #define stat(filename, struct) emu_stat (filename, struct) +// #define lstat(filename, struct) emu_stat (filename, struct) +// #define fstat(handle, struct) emu_fstat (handle, struct) +// +// But lstat doesn't exist on Windows, so we have to provide something: + +#define lstat(filename, struct) stat(filename, struct) + int statfs(const char * name, struct statfs * s); int poll(struct pollfd *ufds, unsigned long nfds, int timeout); Modified: box/trunk/test/bbackupd/testbbackupd.cpp =================================================================== --- box/trunk/test/bbackupd/testbbackupd.cpp 2009-03-20 18:33:16 UTC (rev 2459) +++ box/trunk/test/bbackupd/testbbackupd.cpp 2009-03-21 18:48:19 UTC (rev 2460) @@ -258,9 +258,9 @@ bool attrmatch(const char *f1, const char *f2) { - struct stat s1, s2; - TEST_THAT(::lstat(f1, &s1) == 0); - TEST_THAT(::lstat(f2, &s2) == 0); + EMU_STRUCT_STAT s1, s2; + TEST_THAT(EMU_LSTAT(f1, &s1) == 0); + TEST_THAT(EMU_LSTAT(f2, &s2) == 0); #ifdef HAVE_SYS_XATTR_H { @@ -2027,6 +2027,8 @@ printf("\n==== Check that read-only directories and " "their contents can be restored.\n"); + int compareReturnValue; + { #ifdef WIN32 TEST_THAT(::system("chmod 0555 testfiles/" @@ -2039,7 +2041,7 @@ wait_for_sync_end(); // too new wait_for_sync_end(); // should be backed up now - int compareReturnValue = ::system(BBACKUPQUERY " " + compareReturnValue = ::system(BBACKUPQUERY " " "-Wwarning " "-c testfiles/bbackupd.conf " "\"compare -cEQ Test1 testfiles/TestDir1\" " @@ -2083,8 +2085,6 @@ } - int compareReturnValue; - #ifdef WIN32 printf("\n==== Check that filenames in UTF-8 " "can be backed up\n"); From boxbackup-dev at boxbackup.org Sat Mar 21 20:48:38 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sat, 21 Mar 2009 20:48:38 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2461 - box/trunk/lib/common Message-ID: <20090321204838.1A0A1325FF8@www.boxbackup.org> Author: chris Date: 2009-03-21 20:48:37 +0000 (Sat, 21 Mar 2009) New Revision: 2461 Modified: box/trunk/lib/common/BoxPlatform.h Log: Include emu.h on all platforms to support EMU_*_STAT. Modified: box/trunk/lib/common/BoxPlatform.h =================================================================== --- box/trunk/lib/common/BoxPlatform.h 2009-03-21 18:48:19 UTC (rev 2460) +++ box/trunk/lib/common/BoxPlatform.h 2009-03-21 20:48:37 UTC (rev 2461) @@ -172,9 +172,10 @@ #ifdef WIN32 #define WIN32_LEAN_AND_MEAN - #include "emu.h" #endif +#include "emu.h" + // Solaris has no dirfd(x) macro or function, and we need one for // intercept tests. We cannot define macros with arguments directly // using AC_DEFINE, so do it here instead of in configure.ac. From boxbackup-dev at boxbackup.org Sun Mar 22 10:09:00 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sun, 22 Mar 2009 10:09:00 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2462 - box/chris/win32/support Message-ID: <20090322100900.4ADA9325FF8@www.boxbackup.org> Author: chris Date: 2009-03-22 10:08:59 +0000 (Sun, 22 Mar 2009) New Revision: 2462 Modified: box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch Log: Patch the *other* copy of e_os2.h as well. Modified: box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch =================================================================== --- box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch 2009-03-21 20:48:37 UTC (rev 2461) +++ box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch 2009-03-22 10:08:59 UTC (rev 2462) @@ -334,3 +334,15 @@ # define OPENSSL_DECLARE_GLOBAL(type,name) type *_shadow_##name(void) # define OPENSSL_GLOBAL_REF(name) (*(_shadow_##name())) #else +diff -ru openssl-0.9.8b/e_os2.h openssl-0.9.8b-mingw/e_os2.h +--- openssl-0.9.8b/e_os2.h 2009-03-20 00:02:19.000000000 +0000 ++++ openssl-0.9.8b-mingw/e_os2.h 2009-03-20 00:02:34.000000000 +0000 +@@ -264,7 +264,7 @@ + # define OPENSSL_IMPLEMENT_GLOBAL(type,name) \ + extern type _hide_##name; \ + type *_shadow_##name(void) { return &_hide_##name; } \ +- static type _hide_##name ++ type _hide_##name + # define OPENSSL_DECLARE_GLOBAL(type,name) type *_shadow_##name(void) + # define OPENSSL_GLOBAL_REF(name) (*(_shadow_##name())) + #else From boxbackup-dev at boxbackup.org Sun Mar 22 10:33:20 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sun, 22 Mar 2009 10:33:20 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2463 - box/trunk/docs/api-docs/backup Message-ID: <20090322103320.94783325FF8@www.boxbackup.org> Author: chris Date: 2009-03-22 10:33:20 +0000 (Sun, 22 Mar 2009) New Revision: 2463 Modified: box/trunk/docs/api-docs/backup/win32_build_on_linux_using_mingw.txt Log: Need to copy libpcre.a from PCRE as well as libpcreposix.a. Modified: box/trunk/docs/api-docs/backup/win32_build_on_linux_using_mingw.txt =================================================================== --- box/trunk/docs/api-docs/backup/win32_build_on_linux_using_mingw.txt 2009-03-22 10:08:59 UTC (rev 2462) +++ box/trunk/docs/api-docs/backup/win32_build_on_linux_using_mingw.txt 2009-03-22 10:33:20 UTC (rev 2463) @@ -71,6 +71,7 @@ to complete the build. Finally: + cp .libs/libpcre.a /usr/local/i386-pc-mingw32/lib cp .libs/libpcreposix.a /usr/local/i386-pc-mingw32/lib cp pcreposix.h /usr/local/i386-pc-mingw32/include From boxbackup-dev at boxbackup.org Tue Mar 24 15:20:25 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 24 Mar 2009 15:20:25 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2464 - box/trunk/test/httpserver/testfiles Message-ID: <20090324152025.9FD0C325FF8@www.boxbackup.org> Author: chris Date: 2009-03-24 15:20:24 +0000 (Tue, 24 Mar 2009) New Revision: 2464 Modified: box/trunk/test/httpserver/testfiles/testrequests.pl Log: Debug logging of reason for request failure. Modified: box/trunk/test/httpserver/testfiles/testrequests.pl =================================================================== --- box/trunk/test/httpserver/testfiles/testrequests.pl 2009-03-22 10:33:20 UTC (rev 2463) +++ box/trunk/test/httpserver/testfiles/testrequests.pl 2009-03-24 15:20:24 UTC (rev 2464) @@ -9,7 +9,7 @@ print "GET request...\n"; my $response1 = $ua->get("$url_base/test-one/34/341s/234?p1=vOne&p2=vTwo"); -exit 1 unless $response1->is_success(); +die $response1->content unless $response1->is_success(); my $content = $response1->content(); From boxbackup-dev at boxbackup.org Tue Mar 24 15:22:54 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 24 Mar 2009 15:22:54 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2465 - in box/trunk: bin/bbackupctl bin/bbackupquery bin/bbstoreaccounts lib/backupclient lib/common lib/raidfile Message-ID: <20090324152254.430C0325FF8@www.boxbackup.org> Author: chris Date: 2009-03-24 15:22:54 +0000 (Tue, 24 Mar 2009) New Revision: 2465 Modified: box/trunk/bin/bbackupctl/bbackupctl.cpp box/trunk/bin/bbackupquery/BackupQueries.cpp box/trunk/bin/bbackupquery/bbackupquery.cpp box/trunk/bin/bbstoreaccounts/bbstoreaccounts.cpp box/trunk/lib/backupclient/BackupStoreFile.h box/trunk/lib/common/Logging.cpp box/trunk/lib/common/Timer.cpp box/trunk/lib/common/WaitForEvent.h box/trunk/lib/raidfile/RaidFileRead.cpp Log: Switch to C++ include file names. Modified: box/trunk/bin/bbackupctl/bbackupctl.cpp =================================================================== --- box/trunk/bin/bbackupctl/bbackupctl.cpp 2009-03-24 15:20:24 UTC (rev 2464) +++ box/trunk/bin/bbackupctl/bbackupctl.cpp 2009-03-24 15:22:54 UTC (rev 2465) @@ -10,12 +10,13 @@ #include "Box.h" #include -#include #ifdef HAVE_UNISTD_H #include #endif +#include + #include "MainHelper.h" #include "BoxPortsAndFiles.h" #include "BackupDaemonConfigVerify.h" Modified: box/trunk/bin/bbackupquery/BackupQueries.cpp =================================================================== --- box/trunk/bin/bbackupquery/BackupQueries.cpp 2009-03-24 15:20:24 UTC (rev 2464) +++ box/trunk/bin/bbackupquery/BackupQueries.cpp 2009-03-24 15:22:54 UTC (rev 2465) @@ -13,7 +13,6 @@ #include #endif -#include #include #include #include @@ -25,10 +24,11 @@ #include #endif -#include +#include #include #include #include +#include #include "BackupClientFileAttributes.h" #include "BackupClientMakeExcludeList.h" Modified: box/trunk/bin/bbackupquery/bbackupquery.cpp =================================================================== --- box/trunk/bin/bbackupquery/bbackupquery.cpp 2009-03-24 15:20:24 UTC (rev 2464) +++ box/trunk/bin/bbackupquery/bbackupquery.cpp 2009-03-24 15:22:54 UTC (rev 2465) @@ -15,7 +15,6 @@ #include #include -#include #ifdef HAVE_SYS_TYPES_H #include @@ -38,6 +37,8 @@ #endif #endif +#include + #include "MainHelper.h" #include "BoxPortsAndFiles.h" #include "BackupDaemonConfigVerify.h" Modified: box/trunk/bin/bbstoreaccounts/bbstoreaccounts.cpp =================================================================== --- box/trunk/bin/bbstoreaccounts/bbstoreaccounts.cpp 2009-03-24 15:20:24 UTC (rev 2464) +++ box/trunk/bin/bbstoreaccounts/bbstoreaccounts.cpp 2009-03-24 15:22:54 UTC (rev 2465) @@ -11,12 +11,12 @@ #include #include -#include #include #include #include +#include #include #include #include Modified: box/trunk/lib/backupclient/BackupStoreFile.h =================================================================== --- box/trunk/lib/backupclient/BackupStoreFile.h 2009-03-24 15:20:24 UTC (rev 2464) +++ box/trunk/lib/backupclient/BackupStoreFile.h 2009-03-24 15:22:54 UTC (rev 2465) @@ -10,8 +10,7 @@ #ifndef BACKUPSTOREFILE__H #define BACKUPSTOREFILE__H -#include - +#include #include #include "BackupClientFileAttributes.h" Modified: box/trunk/lib/common/Logging.cpp =================================================================== --- box/trunk/lib/common/Logging.cpp 2009-03-24 15:20:24 UTC (rev 2464) +++ box/trunk/lib/common/Logging.cpp 2009-03-24 15:22:54 UTC (rev 2465) @@ -10,7 +10,6 @@ #include "Box.h" #include -#include #include #ifdef HAVE_SYSLOG_H @@ -20,6 +19,7 @@ #include #endif +#include #include #include "BoxTime.h" Modified: box/trunk/lib/common/Timer.cpp =================================================================== --- box/trunk/lib/common/Timer.cpp 2009-03-24 15:20:24 UTC (rev 2464) +++ box/trunk/lib/common/Timer.cpp 2009-03-24 15:22:54 UTC (rev 2465) @@ -15,6 +15,7 @@ #include "Box.h" #include +#include #include "Timer.h" #include "Logging.h" Modified: box/trunk/lib/common/WaitForEvent.h =================================================================== --- box/trunk/lib/common/WaitForEvent.h 2009-03-24 15:20:24 UTC (rev 2464) +++ box/trunk/lib/common/WaitForEvent.h 2009-03-24 15:22:54 UTC (rev 2465) @@ -10,7 +10,7 @@ #ifndef WAITFOREVENT__H #define WAITFOREVENT__H -#include +#include #ifdef HAVE_KQUEUE #include Modified: box/trunk/lib/raidfile/RaidFileRead.cpp =================================================================== --- box/trunk/lib/raidfile/RaidFileRead.cpp 2009-03-24 15:20:24 UTC (rev 2464) +++ box/trunk/lib/raidfile/RaidFileRead.cpp 2009-03-24 15:22:54 UTC (rev 2465) @@ -12,7 +12,6 @@ #include #include #include -#include #include #include @@ -26,10 +25,11 @@ #include #endif -#include -#include +#include +#include +#include +#include #include -#include #include "RaidFileRead.h" #include "RaidFileException.h" From boxbackup-dev at boxbackup.org Tue Mar 24 16:55:46 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 24 Mar 2009 16:55:46 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2466 - in box/trunk/lib: common server Message-ID: <20090324165546.D1B63325FF8@www.boxbackup.org> Author: chris Date: 2009-03-24 16:55:46 +0000 (Tue, 24 Mar 2009) New Revision: 2466 Modified: box/trunk/lib/common/Logging.cpp box/trunk/lib/common/Logging.h box/trunk/lib/server/Daemon.cpp box/trunk/lib/server/Daemon.h Log: Add ability to specify a named log facility for syslog loggins, requested by Kenny Millington. Modified: box/trunk/lib/common/Logging.cpp =================================================================== --- box/trunk/lib/common/Logging.cpp 2009-03-24 15:22:54 UTC (rev 2465) +++ box/trunk/lib/common/Logging.cpp 2009-03-24 16:55:46 UTC (rev 2466) @@ -216,6 +216,11 @@ } } +void Logging::SetFacility(int facility) +{ + spSyslog->SetFacility(facility); +} + Logger::Logger() : mCurrentLevel(Log::EVERYTHING) { @@ -390,9 +395,9 @@ return true; } -Syslog::Syslog() +Syslog::Syslog() : mFacility(LOG_LOCAL6) { - ::openlog("Box Backup", LOG_PID, LOG_LOCAL6); + ::openlog("Box Backup", LOG_PID, mFacility); } Syslog::~Syslog() @@ -404,9 +409,34 @@ { mName = rProgramName; ::closelog(); - ::openlog(mName.c_str(), LOG_PID, LOG_LOCAL6); + ::openlog(mName.c_str(), LOG_PID, mFacility); } +void Syslog::SetFacility(int facility) +{ + mFacility = facility; + ::closelog(); + ::openlog(mName.c_str(), LOG_PID, mFacility); +} + +int Syslog::GetNamedFacility(const std::string& rFacility) +{ + #define CASE_RETURN(x) if (rFacility == #x) { return LOG_ ## x; } + CASE_RETURN(LOCAL0) + CASE_RETURN(LOCAL1) + CASE_RETURN(LOCAL2) + CASE_RETURN(LOCAL3) + CASE_RETURN(LOCAL4) + CASE_RETURN(LOCAL5) + CASE_RETURN(LOCAL6) + CASE_RETURN(DAEMON) + #undef CASE_RETURN + + BOX_ERROR("Unknown log facility '" << rFacility << "', " + "using default LOCAL6"); + return LOG_LOCAL6; +} + bool FileLogger::Log(Log::Level Level, const std::string& rFile, int line, std::string& rMessage) { Modified: box/trunk/lib/common/Logging.h =================================================================== --- box/trunk/lib/common/Logging.h 2009-03-24 15:22:54 UTC (rev 2465) +++ box/trunk/lib/common/Logging.h 2009-03-24 16:55:46 UTC (rev 2466) @@ -202,6 +202,7 @@ { private: std::string mName; + int mFacility; public: Syslog(); @@ -211,6 +212,8 @@ int line, std::string& rMessage); virtual const char* GetType() { return "Syslog"; } virtual void SetProgramName(const std::string& rProgramName); + virtual void SetFacility(int facility); + static int GetNamedFacility(const std::string& rFacility); }; // -------------------------------------------------------------------------- @@ -260,6 +263,7 @@ } static void SetProgramName(const std::string& rProgramName); static std::string GetProgramName() { return sProgramName; } + static void SetFacility(int facility); class Guard { Modified: box/trunk/lib/server/Daemon.cpp =================================================================== --- box/trunk/lib/server/Daemon.cpp 2009-03-24 15:22:54 UTC (rev 2465) +++ box/trunk/lib/server/Daemon.cpp 2009-03-24 16:55:46 UTC (rev 2466) @@ -480,6 +480,13 @@ const Configuration &serverConfig( mapConfiguration->GetSubConfiguration("Server")); + if(serverConfig.KeyExists("LogFacility")) + { + std::string facility = + serverConfig.GetKeyValue("LogFacility"); + Logging::SetFacility(Syslog::GetNamedFacility(facility)); + } + // Open PID file for writing pidFileName = serverConfig.GetKeyValue("PidFile"); FileHandleGuard<(O_WRONLY | O_CREAT | O_TRUNC), (S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH)> pidFile(pidFileName.c_str()); Modified: box/trunk/lib/server/Daemon.h =================================================================== --- box/trunk/lib/server/Daemon.h 2009-03-24 15:22:54 UTC (rev 2465) +++ box/trunk/lib/server/Daemon.h 2009-03-24 16:55:46 UTC (rev 2466) @@ -105,6 +105,7 @@ #define DAEMON_VERIFY_SERVER_KEYS \ ConfigurationVerifyKey("PidFile", ConfigTest_Exists), \ + ConfigurationVerifyKey("LogFacility", 0), \ ConfigurationVerifyKey("User", ConfigTest_LastEntry) #endif // DAEMON__H From boxbackup-dev at boxbackup.org Tue Mar 24 23:29:11 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 24 Mar 2009 23:29:11 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2467 - box/trunk/bin/bbackupquery Message-ID: <20090324232911.09D0F325FF8@www.boxbackup.org> Author: chris Date: 2009-03-24 23:29:10 +0000 (Tue, 24 Mar 2009) New Revision: 2467 Modified: box/trunk/bin/bbackupquery/BoxBackupCompareParams.h Log: Include for std::auto_ptr Modified: box/trunk/bin/bbackupquery/BoxBackupCompareParams.h =================================================================== --- box/trunk/bin/bbackupquery/BoxBackupCompareParams.h 2009-03-24 16:55:46 UTC (rev 2466) +++ box/trunk/bin/bbackupquery/BoxBackupCompareParams.h 2009-03-24 23:29:10 UTC (rev 2467) @@ -10,6 +10,7 @@ #ifndef BOXBACKUPCOMPAREPARAMS__H #define BOXBACKUPCOMPAREPARAMS__H +#include #include #include "BoxTime.h" From boxbackup-dev at boxbackup.org Fri Mar 27 23:42:22 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Fri, 27 Mar 2009 23:42:22 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2468 - box/trunk/distribution/boxbackup Message-ID: <20090327234222.1263C326899@www.boxbackup.org> Author: chris Date: 2009-03-27 23:42:20 +0000 (Fri, 27 Mar 2009) New Revision: 2468 Modified: box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt Log: Fix patching of ExceptionCodes.txt to actually work on Linux, where sed -i can't have an empty argument. Modified: box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt =================================================================== --- box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt 2009-03-24 23:29:10 UTC (rev 2467) +++ box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt 2009-03-27 23:42:20 UTC (rev 2468) @@ -29,7 +29,7 @@ docs/box-html/man-html docs/htmlguide/manpages docs/man MKDIR docs/docbook -RUN cd docs; sed -i "" -e '/^ExceptionCodes/,/^$/s/\(.*\)/# &/' Makefile +RUN cd docs; sed -i -e '/^ExceptionCodes/,/^$/s/\(.*\)/# &/' Makefile docs/Makefile docs/docbook/Makefile docs/ExceptionCodes.xml docs/docbook/ExceptionCodes.xml docs/adminguide.xml docs/docbook/adminguide.xml From boxbackup-dev at boxbackup.org Fri Mar 27 23:44:17 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Fri, 27 Mar 2009 23:44:17 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2469 - box/trunk/infrastructure Message-ID: <20090327234417.3FD6F326899@www.boxbackup.org> Author: chris Date: 2009-03-27 23:44:17 +0000 (Fri, 27 Mar 2009) New Revision: 2469 Modified: box/trunk/infrastructure/makeparcels.pl.in Log: Don't skip building docs if we can't find a SVN version, as it breaks build on exported copies. It doesn't do any harm in distributions either, as long as timestamps on the pre-built man pages are correct, and allows users to modify the docbook source and rebuild as long as they have xsltproc. Modified: box/trunk/infrastructure/makeparcels.pl.in =================================================================== --- box/trunk/infrastructure/makeparcels.pl.in 2009-03-27 23:42:20 UTC (rev 2468) +++ box/trunk/infrastructure/makeparcels.pl.in 2009-03-27 23:44:17 UTC (rev 2469) @@ -211,7 +211,7 @@ EOF # Only build the docs if we're building from trunk. # Releases have the docs pre-made. - print MAKE < Author: chris Date: 2009-03-27 23:50:30 +0000 (Fri, 27 Mar 2009) New Revision: 2470 Modified: box/trunk/infrastructure/makedistribution.pl.in Log: Allow substitutions and removing private sections for distribution in .in files. Simplify code a little and improve comments. Modified: box/trunk/infrastructure/makedistribution.pl.in =================================================================== --- box/trunk/infrastructure/makedistribution.pl.in 2009-03-27 23:44:17 UTC (rev 2469) +++ box/trunk/infrastructure/makedistribution.pl.in 2009-03-27 23:50:30 UTC (rev 2470) @@ -7,7 +7,9 @@ my %comment_chars = ('cpp' => '// ', 'h' => '// ', 'pl' => '# ', 'pm' => '# ', '' => '# '); # other extensions which need text copying, just to remove the private stuff -my %text_files = ('txt' => 1, 'spec' => 1); +# .in is included here, as these could be any kind of source, but clearly +# they have text substitutions run on them by autoconf, so we can too :) +my %text_files = ('txt' => 1, 'spec' => 1, 'in' => 1); # files which don't get the license added my %no_license = (); # 'filename' => 1 @@ -189,9 +191,9 @@ # licensed or not? if(exists $comment_chars{$ext} && !exists $no_license{$fn} && $license_in_dir) { - my $b = $comment_chars{$ext}; + # copy as text, inserting license + # print "source copy $fn to $base_name/$dst_fn\n"; - # copy as text, inserting license my $in = gensym; open $in,$fn; open OUT,">$base_name/$dst_fn"; @@ -204,6 +206,7 @@ } # write license + my $b = $comment_chars{$ext}; for(@license) { print OUT $b,$_,"\n" @@ -225,32 +228,32 @@ close OUT; close $in; } - else + elsif(exists $text_files{$ext}) { - if(exists $text_files{$ext}) + # copy this as text, to remove private stuff + # print "text copy $fn to $base_name/$dst_fn\n"; + + my $in = gensym; + open $in,$fn; + open OUT,">$base_name/$dst_fn"; + + while(<$in>) { - # copy this as text, to remove private stuff - my $in = gensym; - open $in,$fn; - open OUT,">$base_name/$dst_fn"; - - while(<$in>) + unless(skip_non_applicable_section($_, $in, $fn)) { - unless(skip_non_applicable_section($_, $in, $fn)) - { - print OUT - } - } + print OUT + } + } - close OUT; - close $in; - } - else - { - # copy as binary - system 'cp',$fn,"$base_name/$dst_fn" - } + close OUT; + close $in; } + else + { + # copy as binary + # print "binary copy $fn to $base_name/$dst_fn\n"; + system 'cp',$fn,"$base_name/$dst_fn" + } # copy executable bit from src if(-x $fn) From boxbackup-dev at boxbackup.org Fri Mar 27 23:50:58 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Fri, 27 Mar 2009 23:50:58 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2471 - box/trunk/distribution Message-ID: <20090327235058.9E545326899@www.boxbackup.org> Author: chris Date: 2009-03-27 23:50:58 +0000 (Fri, 27 Mar 2009) New Revision: 2471 Modified: box/trunk/distribution/COMMON-MANIFEST.txt Log: No need to include makedistribution.pl in a distribution. Modified: box/trunk/distribution/COMMON-MANIFEST.txt =================================================================== --- box/trunk/distribution/COMMON-MANIFEST.txt 2009-03-27 23:50:30 UTC (rev 2470) +++ box/trunk/distribution/COMMON-MANIFEST.txt 2009-03-27 23:50:58 UTC (rev 2471) @@ -19,7 +19,6 @@ MKDIR infrastructure infrastructure/buildenv-testmain-template.cpp infrastructure/makebuildenv.pl.in -infrastructure/makedistribution.pl.in infrastructure/makeparcels.pl.in infrastructure/parcelpath.pl infrastructure/printversion.pl From boxbackup-dev at boxbackup.org Fri Mar 27 23:52:02 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Fri, 27 Mar 2009 23:52:02 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2472 - box/trunk/distribution Message-ID: <20090327235202.E541A326899@www.boxbackup.org> Author: chris Date: 2009-03-27 23:52:02 +0000 (Fri, 27 Mar 2009) New Revision: 2472 Modified: box/trunk/distribution/COMMON-MANIFEST.txt Log: Undo that, configure won't work without it. Modified: box/trunk/distribution/COMMON-MANIFEST.txt =================================================================== --- box/trunk/distribution/COMMON-MANIFEST.txt 2009-03-27 23:50:58 UTC (rev 2471) +++ box/trunk/distribution/COMMON-MANIFEST.txt 2009-03-27 23:52:02 UTC (rev 2472) @@ -19,6 +19,7 @@ MKDIR infrastructure infrastructure/buildenv-testmain-template.cpp infrastructure/makebuildenv.pl.in +infrastructure/makedistribution.pl.in infrastructure/makeparcels.pl.in infrastructure/parcelpath.pl infrastructure/printversion.pl From boxbackup-dev at boxbackup.org Sat Mar 28 00:21:14 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sat, 28 Mar 2009 00:21:14 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2473 - box/trunk/distribution/boxbackup Message-ID: <20090328002114.E109332602D@www.boxbackup.org> Author: chris Date: 2009-03-28 00:21:14 +0000 (Sat, 28 Mar 2009) New Revision: 2473 Modified: box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt Log: Put the sed -i parameter back, but without the space, to make both linux and bsd happy. Modified: box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt =================================================================== --- box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt 2009-03-27 23:52:02 UTC (rev 2472) +++ box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt 2009-03-28 00:21:14 UTC (rev 2473) @@ -29,7 +29,7 @@ docs/box-html/man-html docs/htmlguide/manpages docs/man MKDIR docs/docbook -RUN cd docs; sed -i -e '/^ExceptionCodes/,/^$/s/\(.*\)/# &/' Makefile +RUN cd docs; sed -i"" -e '/^ExceptionCodes/,/^$/s/\(.*\)/# &/' Makefile docs/Makefile docs/docbook/Makefile docs/ExceptionCodes.xml docs/docbook/ExceptionCodes.xml docs/adminguide.xml docs/docbook/adminguide.xml From boxbackup-dev at boxbackup.org Sat Mar 28 12:25:06 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sat, 28 Mar 2009 12:25:06 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2474 - in box/trunk: distribution/boxbackup docs docs/docbook docs/docbook/html docs/images docs/tools Message-ID: <20090328122506.1F640326025@www.boxbackup.org> Author: chris Date: 2009-03-28 12:25:05 +0000 (Sat, 28 Mar 2009) New Revision: 2474 Added: box/trunk/docs/api-notes/ box/trunk/docs/docbook/ box/trunk/docs/docbook/adminguide.xml box/trunk/docs/docbook/bb-book.xsl box/trunk/docs/docbook/bb-man.xsl.tmpl box/trunk/docs/docbook/bb-nochunk-book.xsl box/trunk/docs/docbook/bbackupctl.xml box/trunk/docs/docbook/bbackupd-config.xml box/trunk/docs/docbook/bbackupd.conf.xml box/trunk/docs/docbook/bbackupd.xml box/trunk/docs/docbook/bbackupquery.xml box/trunk/docs/docbook/bbstoreaccounts.xml box/trunk/docs/docbook/bbstored-certs.xml box/trunk/docs/docbook/bbstored-config.xml box/trunk/docs/docbook/bbstored.conf.xml box/trunk/docs/docbook/bbstored.xml box/trunk/docs/docbook/html/ box/trunk/docs/docbook/html/favicon.ico box/trunk/docs/docbook/instguide.xml box/trunk/docs/docbook/raidfile-config.xml box/trunk/docs/docbook/raidfile.conf.xml box/trunk/docs/images/ box/trunk/docs/images/bblogo-alpha.xcf box/trunk/docs/tools/ box/trunk/docs/tools/generate_except_xml.pl Removed: box/trunk/docs/adminguide.xml box/trunk/docs/api-docs/ box/trunk/docs/bb-book.xsl box/trunk/docs/bb-man.xsl.tmpl box/trunk/docs/bb-nochunk-book.xsl box/trunk/docs/bbackupctl.xml box/trunk/docs/bbackupd-config.xml box/trunk/docs/bbackupd.conf.xml box/trunk/docs/bbackupd.xml box/trunk/docs/bbackupquery.xml box/trunk/docs/bblogo-alpha.xcf box/trunk/docs/bbstoreaccounts.xml box/trunk/docs/bbstored-certs.xml box/trunk/docs/bbstored-config.xml box/trunk/docs/bbstored.conf.xml box/trunk/docs/bbstored.xml box/trunk/docs/favicon.ico box/trunk/docs/generate_except_xml.pl box/trunk/docs/html/ box/trunk/docs/instguide.xml box/trunk/docs/raidfile-config.xml box/trunk/docs/raidfile.conf.xml Modified: box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt box/trunk/docs/Makefile Log: Reorganise docs in trunk to match distribution layout, which is cleaner, and makes Makefile work on distributions and trunk equally. Modified: box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt =================================================================== --- box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt 2009-03-28 12:25:05 UTC (rev 2474) @@ -19,39 +19,39 @@ test/bbackupd test/bbackupd/testfiles test/backupdiff -docs/api-docs/raidfile docs/api-notes -docs/api-docs/raidfile/lib_raidfile docs/api-notes/lib_raidfile -docs/api-docs/backup docs/api-notes -docs/box-html docs/htmlguide -docs/box-html/adminguide docs/htmlguide/adminguide -docs/box-html/images docs/htmlguide/images -docs/box-html/instguide docs/htmlguide/instguide -docs/box-html/man-html docs/htmlguide/manpages +docs/Makefile +docs/tools +docs/api-notes +docs/api-notes/lib_raidfile +docs/htmlguide +docs/htmlguide/adminguide +docs/htmlguide/images +docs/htmlguide/instguide +docs/htmlguide/manpages docs/man -MKDIR docs/docbook -RUN cd docs; sed -i"" -e '/^ExceptionCodes/,/^$/s/\(.*\)/# &/' Makefile -docs/Makefile docs/docbook/Makefile -docs/ExceptionCodes.xml docs/docbook/ExceptionCodes.xml -docs/adminguide.xml docs/docbook/adminguide.xml -docs/bb-book.xsl docs/docbook/bb-book.xsl -docs/bb-man.xsl.tmpl docs/docbook/bb-man.xsl.tmpl -docs/bb-nochunk-book.xsl docs/docbook/bb-nochunk-book.xsl -docs/bbackupctl.xml docs/docbook/bbackupctl.xml -docs/bbackupd-config.xml docs/docbook/bbackupd-config.xml -docs/bbackupd.conf.xml docs/docbook/bbackupd.conf.xml -docs/bbackupd.xml docs/docbook/bbackupd.xml -docs/bbackupquery.xml docs/docbook/bbackupquery.xml -docs/bbstoreaccounts.xml docs/docbook/bbstoreaccounts.xml -docs/bbstored-certs.xml docs/docbook/bbstored-certs.xml -docs/bbstored-config.xml docs/docbook/bbstored-config.xml -docs/bbstored.conf.xml docs/docbook/bbstored.conf.xml -docs/bbstored.xml docs/docbook/bbstored.xml -docs/instguide.xml docs/docbook/instguide.xml -docs/raidfile-config.xml docs/docbook/raidfile-config.xml -docs/raidfile.conf.xml docs/docbook/raidfile.conf.xml -docs/html docs/docbook/html -docs/html/images docs/docbook/html/images -RUN svn revert docs/Makefile +# RUN cd docs; sed -i"" -e '/^ExceptionCodes/,/^$/s/\(.*\)/# &/' Makefile +docs/docbook/Makefile +docs/docbook/ExceptionCodes.xml +docs/docbook/adminguide.xml +docs/docbook/bb-book.xsl +docs/docbook/bb-man.xsl.tmpl +docs/docbook/bb-nochunk-book.xsl +docs/docbook/bbackupctl.xml +docs/docbook/bbackupd-config.xml +docs/docbook/bbackupd.conf.xml +docs/docbook/bbackupd.xml +docs/docbook/bbackupquery.xml +docs/docbook/bbstoreaccounts.xml +docs/docbook/bbstored-certs.xml +docs/docbook/bbstored-config.xml +docs/docbook/bbstored.conf.xml +docs/docbook/bbstored.xml +docs/docbook/instguide.xml +docs/docbook/raidfile-config.xml +docs/docbook/raidfile.conf.xml +docs/docbook/html +docs/docbook/html/images +# RUN svn revert docs/Makefile TODO.txt BUGS.txt contrib Modified: box/trunk/docs/Makefile =================================================================== --- box/trunk/docs/Makefile 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/Makefile 2009-03-28 12:25:05 UTC (rev 2474) @@ -3,33 +3,39 @@ # Process DocBook to HTML DBPROC=xsltproc -BOOKXSL=bb-book.xsl -NOCHUNKBOOKXSL=bb-nochunk-book.xsl -MANXSL=bb-man.xsl -HTMLPREFIX=box-html -VPATH= adminguide -.SUFFIXES: .html .xml .1 .5 .8 +DOCBOOK_DIR = docbook +HTML_DIR = htmlguide +MAN_DIR = man + +BOOKXSL = $(DOCBOOK_DIR)/bb-book.xsl +NOCHUNKBOOKXSL = $(DOCBOOK_DIR)/bb-nochunk-book.xsl +MANXSL = $(DOCBOOK_DIR)/bb-man.xsl + +# VPATH= adminguide +# .SUFFIXES: .html .xml .1 .5 .8 + all: docs docs: instguide adminguide manpages - @mkdir -p $(HTMLPREFIX)/images - @cp html/images/*.png $(HTMLPREFIX)/images/. - @cp html/*.css $(HTMLPREFIX)/. + @mkdir -p $(HTML_DIR)/images + @cp $(DOCBOOK_DIR)/html/images/*.png $(HTML_DIR)/images/. + @cp $(DOCBOOK_DIR)/html/*.css $(HTML_DIR)/. + @cp $(DOCBOOK_DIR)/html/*.ico $(HTML_DIR)/. -adminguide: $(HTMLPREFIX)/adminguide/index.html +adminguide: $(HTML_DIR)/adminguide/index.html -$(HTMLPREFIX)/adminguide/index.html: adminguide.xml ExceptionCodes.xml $(BOOKXSL) +$(HTML_DIR)/adminguide/index.html: $(DOCBOOK_DIR)/adminguide.xml $(DOCBOOK_DIR)/ExceptionCodes.xml $(BOOKXSL) # docname=`echo $@ | sed -e 's/\/index.html//'` - $(DBPROC) -o $(HTMLPREFIX)/adminguide/ $(BOOKXSL) adminguide.xml + $(DBPROC) -o $(HTML_DIR)/adminguide/ $(BOOKXSL) $< -instguide: $(HTMLPREFIX)/instguide/index.html +instguide: $(HTML_DIR)/instguide/index.html -$(HTMLPREFIX)/instguide/index.html: instguide.xml $(BOOKXSL) - $(DBPROC) -o $(HTMLPREFIX)/instguide/ $(BOOKXSL) instguide.xml +$(HTML_DIR)/instguide/index.html: $(DOCBOOK_DIR)/instguide.xml $(BOOKXSL) + $(DBPROC) -o $(HTML_DIR)/instguide/ $(BOOKXSL) $< -ExceptionCodes.xml: ../ExceptionCodes.txt - perl ./generate_except_xml.pl +$(DOCBOOK_DIR)/ExceptionCodes.xml: ../ExceptionCodes.txt + perl tools/generate_except_xml.pl $< $@ manpages: $(MANXSL) man-dirs man-nroff man-html @@ -45,46 +51,49 @@ fi; \ sed -e "s,%%DOCBOOK%%,$${DOCBOOK}," $(MANXSL).tmpl > $(MANXSL) -man-dirs: man/.there $(HTMLPREFIX)/man-html/.there +man-dirs: man/.there $(HTML_DIR)/man-html/.there -$(HTMLPREFIX)/man-html/.there: - if [ ! -d $(HTMLPREFIX)/man-html ]; then mkdir -p $(HTMLPREFIX)/man-html; touch $(HTMLPREFIX)/man-html/.there; fi +$(HTML_DIR)/man-html/.there: + mkdir -p $(HTML_DIR)/man-html + touch $(HTML_DIR)/man-html/.there man/.there: - if [ ! -d man ]; then mkdir man; touch man/.there; fi + then mkdir -p man + touch man/.there NROFF_PAGES = bbackupd.8 bbackupd-config.8 bbackupctl.8 bbackupquery.8 \ bbstored.8 bbstored-config.8 bbstoreaccounts.8 bbstored-certs.8 \ raidfile-config.8 \ bbackupd.conf.5 bbstored.conf.5 raidfile.conf.5 -man-nroff: $(NROFF_PAGES) +NROFF_FILES = $(NROFF_PAGES:%=$(MAN_DIR)/%.gz) -HTML_PAGES = bbackupd.html bbackupd-config.html bbackupctl.html \ - bbackupquery.html bbstored.html bbstored-config.html \ - bbstoreaccounts.html bbstored-certs.html raidfile-config.html \ - bbackupd.conf.html bbstored.conf.html raidfile.conf.html +man-nroff: $(NROFF_FILES) -man-html: $(HTML_PAGES) +HTML_FILES_1 = $(NROFF_PAGES:%.5=%.html) +HTML_FILES_2 = $(HTML_FILES_1:%.8=%.html) +HTML_FILES = $(HTML_FILES_2:%=$(HTML_DIR)/man-html/%) -.xml.html: - @$(DBPROC) -o $@ $(NOCHUNKBOOKXSL) $< - @cp $@ $(HTMLPREFIX)/man-html/. +man-html: $(HTML_FILES) -.xml.8 .xml.5: $(MANXSL) - @$(DBPROC) -o $@ $(MANXSL) $< - @cp $@ man/ - @rm -f man/$@.gz - @gzip -f -9 man/$@ +$(HTML_DIR)/man-html/%.html: $(DOCBOOK_DIR)/%.xml $(NOCHUNKBOOKXSL) + $(DBPROC) -o $@ $(NOCHUNKBOOKXSL) $< +$(MAN_DIR)/%.8.gz: $(DOCBOOK_DIR)/%.xml $(MANXSL) + $(DBPROC) -o $(@:.gz=) $(MANXSL) $< + gzip $(@:.gz=) + +$(MAN_DIR)/%.5.gz: $(DOCBOOK_DIR)/%.xml $(MANXSL) + $(DBPROC) -o $(@:.gz=) $(MANXSL) $< + gzip $(@:.gz=) + dockit: clean docs - tar zcf documentation-kit-0.10.tar.gz $(HTMLPREFIX)/ + tar zcf documentation-kit-0.10.tar.gz $(HTML_DIR)/ clean: - if [ -d ./$(HTMLPREFIX) ]; then rm -rf $(HTMLPREFIX) ; fi - if [ -d ./man ]; then rm -rf ./man/; fi - if [ -f ExceptionCodes.xml ]; then rm ExceptionCodes.xml; fi - rm -f $(NROFF_PAGES) $(HTML_PAGES) - if [ -f documentation-kit-0.10.tar.gz ]; then rm documentation-kit-0.10.tar.gz; fi + rm -f $(HTML_FILES) + rm -f $(NROFF_FILES) + rm -f $(DOCBOOK_DIR)/ExceptionCodes.xml + rm -f documentation-kit-0.10.tar.gz Deleted: box/trunk/docs/adminguide.xml =================================================================== --- box/trunk/docs/adminguide.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/adminguide.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,1981 +0,0 @@ - - -]> - - Box Backup administrator's guide - - - License - - Copyright ?? 2003 - 2007, Ben Summers and contributors. All rights - reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are - met: - - - - Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - - - Redistributions in binary form must reproduce the above - copyright notice, this list of conditions and the following disclaimer - in the documentation and/or other materials provided with the - distribution. - - - - All use of this software and associated advertising materials - must display the following acknowledgement: This product includes - software developed by Ben Summers and contributors. - - - - The names of the Authors may not be used to endorse or promote - products derived from this software without specific prior written - permission. - - - - [Where legally impermissible the Authors do not disclaim liability - for direct physical injury or death caused solely by defects in the - software unless it is modified by a third party.] - - THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR - IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES - OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN - NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED - TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR - PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF - LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING - NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS - SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - - - Configuration - -

- System configuration - -
- Server - - After you've downloaded and compiled the programs you need to - install the programs on your server. As root do the following: - - make install-backup-server - - This assumes that you are installing on the same server that you - compiled the software on. If not, copy the - boxbackup-x.xx-backup-server-OSNAME.tgz file to the server you want to - run on, and install there. For example (on Mac OS X): - - tar zxvf boxbackup-0.10-server-darwin8.5.0.tgz -cd boxbackup-0.10-server-darwin8.5.0 -./install-backup-server - - Then create the user for the backup daemon on the server: - - useradd _bbstored - - Box Backup has a built-in software RAID facility (redundant - array of inexpensive disks) for the backup store. This allows you to - spread the store data over three disks, and recover from the loss of - any one disk without losing data. However, this is now deprecated, and - you are recommended to use the software or hardware RAID facilities of - your operating system instead. Use the following command if you want - to create a simple server without Box Backup RAID: - - mkdir /tmp/boxbackupRepository # Create the directory -chown _bbstored /tmp/boxbackupRepository/ # Change the owner to the new boxbackup daemon user - -/usr/local/sbin/raidfile-config /etc/box/ 1024 /tmp/boxbackupRepository - -#substitute 1024 with the desired blocksize -#substitute /tmp/boxbackupRepository with a directory that exists where you want the backup store located -#/usr/local/sbin/raidfile-config --help shows you the options - - Then create the configuration file /etc/box/bbstored.conf The - hostname is tricky as it is used for two things: The name of the - server in the certificate and the address the server is listening on. - Since you might be using NAT, might move the server around or the - domain name might change, choose a name that describes the server. - When the network address of the server changes, you need to update the - ListenAddresses directive in the - /etc/box/bbstored.conf file. - - /usr/local/sbin/bbstored-config /etc/box hostname _bbstored - - This last step outputs 5 instructions that you must execute to - the letter. A lot of questions are raised on the mailing list because - these steps have not been followed properly. - - TODO: Expand on this. Explain the 5 steps in detail. - - If you want to run the server as a non-root user, look here. -
- -
- Certificate Management - - There are two steps involved to create an account. You need to - create the account on the server, and sign a certificate to give the - client permission to connect to the server. - - Running a Certification Authority for TLS (SSL) connections is - not trivial. However, a script to does most of the work in a way which - should be good enough for most deployments. - - - The certificate authority directory is intended to be stored - on another server. It should not be kept on the backup server, in - order to limit the impact of a server compromise. The instructions - and the script assume that it will be kept elsewhere, so will ask - you to copy files to and from the CA. - - - - SSL certificates contain validity dates, including a "valid - from" time. If the clock on the machine which signs the certificates - is not syncronised to the clocks of the machines using these - certificates, you will probably get strange errors until the start - time is reached on all machines. If you get strange errors when - attempting to use new certificates, check the clocks on all machines - (client, store and CA). You will probably just need to wait a while - until the certificates become valid, rather than having to - regenerate them. - - -
- Set up a Certificate Authority - - It is recommended that you keep your Certificate Authority on - a separate machine than either the client or the server, preferably - without direct network access. The contents of this directory - control who can access your backup store server. - - To setup the basic key structure, do the following: - - /usr/local/sbin/bbstored-certs ca init - - (See OpenSSL notes if you - get an OpenSSL error) - - This creates the directory called ca in - the current directory, and initialises it with basic keys. -
- -
- Sign a server certificate - - When you use the bbstored-config script to - set up a config file for a server, it will generate a certificate - request (CSR) for you. Transfer it to the machine with your CA, then - do: - - /usr/local/sbin/bbstored-certs ca sign-server hostname-csr.pem - - This signs the certificate for the server. Follow the - instructions in the output on which files to install on the server. - The CSR file is now no longer needed. Make sure you run this command - from the directory above the directory 'ca'. - - TODO: Explain instructions in output. -
- -
- Set up an account - - Choose an account number for the user. This must be unique on - the server, and is presented as a 31 bit number in hex greater than - 0, for example, 1 or 75AB23C. Then on the backup store server, - create the account with: - - /usr/local/sbin/bbstoreaccounts create 75AB23C 0 4096M 4505M - - This looks complicated. The numbers are, in order: - - - - The account number allocated (hex) - - - - The RAID disc set (0 if you use raidfile-config and don't - add a new set) - - - - Soft limit (size) - - - - Hard limit (size) - - - - The sizes are are specified in Mb, Gb, or blocks, depending on - the suffix. 1M specifies 1 Mb, 1G specifies 1 Gb, and 1B specifies 1 - block, the size of which depends on how you have configured the - raidfile system with raidfile-config. - - In this example, I have allocated 4Gb (assuming you use 2048 - byte blocks as per my example) as the soft limit, and 4Gb + 10% as - the hard limit. - - NOTE The sizes specified here are pre-RAID. So if you are - using userland RAID, you are actually allocating two-thirds of this - amount. This means that, when you take compression into account, - that if you allocate 2Gb on the server, it'll probably hold about - 2Gb of backed up files (depending on the compressability of those - files). - - The backup client will (voluntarily) try not to upload more - data than is allowed by the soft limit. The store server will refuse - to accept a file if it would take it over the hard limit, and when - doing housekeeping for this account, try and delete old versions and - deleted files to reduce the space taken to below the soft - limit. - - This command will create some files on disc in the raid file - directories (if you run as root, the utility will change to the user - specified in the bbstored.conf file to write them) and update the - accounts file. A server restart is not required. - - NOTE If you get a message saying 'Exception: RaidFile (2/8)', - the directories you specified in the raidfile.conf are not writable - by the _bbstored user -- fix it, and try again. - - Finally, tell the user their account number, and the hostname - of your server. They will use this to set up the backup client, and - send you a CSR. This has the account number embedded in it, and you - should be sure that it has the right account number in it. - - Sign this CSR with this command: - - /usr/local/sbin/bbstored-certs ca sign 75AB23C-csr.pem - - Don't forget to check that the embedded account number is - correct! Then send the two files back to the user, as instructed by - the script. - - Please read the Troubleshooting page if you have - problems. - - TODO: Link to troubleshooting... -
-
- -
- Log Files - - You may wish to see what's going on with the server. Edit - /etc/syslog.conf, and add: - - local6.info /var/log/box -local5.info /var/log/raidfile - - Note: Separators must be tabs, - otherwise these entries will be ignored. - - touch /var/log/box -touch /var/log/raidfile - - Set up log rotation for these new log files. For example, if you - have /etc/newsyslog.conf, add the following lines - to it: - - /var/log/box 644 7 2000 * Z -/var/log/raidfile 644 7 2000 * Z - - If you have /etc/logrotate.d, create a new - file in there (for example - /etc/logrotate.d/boxbackup) containing the - following: - - /var/log/box /var/log/raidfile { - weekly - create - compress - rotate 52 -} - - Then restart syslogd, for example: - - /etc/init.d/syslogd restart -
- -
- Configuring a client - - Before you can do any configuration, you need to know the - hostname of the server you will be using, and your account number on - that server. - - Later in the process, you will need to send a certificate - request to the administrator of that server for it to be - signed. - - Installation is covered in the compiling and installing section. - You only need the backup-client parcel. - - It is important that you read all the output of the config - scripts. See the end of this page for an example. - - The backup client has to be run as root, because it needs to - read all your files to back them up, although it is possible to back - up a single user's files by running it as that user. (Tip: specify a - directory other than /etc/box, and then give the - alternate config file as the first argument to - bbackupd). However, it will fall over if you don't - give yourself read access to one of your files. - -
- Basic configuration - - Run the bbackupd-config script to generate - the configuration files and generate a private key and certificate - request. - - /usr/local/sbin/bbackupd-config /etc/box lazy 999 hostname /var/bbackupd /home - - (See OpenSSL notes if you - get an OpenSSL error) - - The items in bold need to be changed. In order, they are the - account number, the hostname of the server you're using, and - finally, the directories you want backed up. You can include as many - you want here. - - However, the directories you specify must not contain other - mounted file systems within them at any depth. Specify them - separately, one per mount point. No checks are currently made to - catch bad configuration of this nature! - - You may also want to consider changing the mode from lazy to - snapshot, depending on what your system is used for: - - - - Lazy Mode - - - This mode regularly scans the files, with only a rough - schedule. It uploads files as and when they are changed, if - the latest version is more than a set age. This is good for - backing up user's documents stored on a server, and spreads - the load out over the day. - - - - - Snapshot Mode - - - This mode emulates the traditional backup behaviour of - taking a snapshot of the filesystem. The backup daemon does - absolutely nothing until it is instructed to make a backup - using the bbackupctl utility (probably as a cron job), at - which point it uploads all files which have been changed since - the last time it uploaded. - - - - - When you run the config script, it will tell you what you need - to do next. Don't forget to read all the output. An example is shown - at the end of this page, but the instructions for your installation - may be different. -
- -
- Certificates - - After you have sent your certificate request off to the server - administrator and received your certificate and CA root back, - install them where instructed by the bbackupd-config script during - basic bbackupd configuration. - - You can then run the daemon (as root) by running - /usr/local/sbin/bbackupd, and of course, adding it - to your system's startup scripts. The first time it's run it will - upload everything. Interrupting it and restarting it will only - upload files which were not uploaded before - it's very - tolerant. - - If you run in snapshot mode, you will need to add a cron job - to schedule backups. The config script will tell you the exact - command to use for your system. - - Please read the Troubleshooting page if you have - problems. - - Remember to make a traditional backup of the keys file, as - instructed. You cannot restore files without it. - - It is recommended that you backup up all of /etc/box as it - will make things easier if you need to restore files. But only the - keys are absolutely essential. - - If you want to see what it's doing in more detail (probably a - good idea), follow the instructions in the server setup to create - new log files with syslog. -
- -
- Adding and removing backed up locations - - By editing the /etc/box/bbackupd.conf file, you can add and - remove directories to back up - see comments in this file for help. - Send bbackupd a HUP signal after you modify it. - - When you remove a location, it will not be marked as deleted - immediately. Instead, bbackupd waits about two days before doing so, - just in case you change your mind. After this, it will be eventually - removed from the store by the housekeeping process. Run as - root. - - The backup client is designed to be run as root. It is - possible to run without root, but this is not recommended. Clock - synchronisation for file servers. - - If you are using the backup client to backup a filesystem - served from a fileserver, you should ideally ensure that the - fileserver clocks are synchronised with the fileserver. - - bbackupd will cope perfectly well if the clocks are not - synchronised. Errors up to about half an hour cause no problems. - Larger discrepancies cause a loss of efficiency and the potential to - back up a file during a write process. - - There is a configuration parameter MaxFileTimeInFuture, which - specifies how far in the future a file must be for it to be uploaded - as soon as it is seen. You should not need to adjust this (default - is 2 days). Instead, get those clocks synchronised. Excluding files - and directories from the backup. - - Within the bbackupd.conf file, there is a section named - BackupLocations which specifies which locations on disc should be - backed up. It has subsections, each of which is in the - format: - - name - { - Path = /path/of/directory - (optional exclude directives) - } - - name is derived from the Path - by the config script, but should merely be unique. - - The exclude directives are of the form: - - [Exclude|AlwaysInclude][File|Dir][|sRegex] = regex or full pathname - - (The regex suffix is shown as 'sRegex' to make File or Dir - plural) - - For example: - - ExcludeDir = /home/guest-user - ExcludeFilesRegex = *.(mp3|MP3)\$ - AlwaysIncludeFile = /home/username/veryimportant.mp3 - - This excludes the directory /home/guest-user from the backup - along with all mp3 files, except one MP3 file in particular. - - In general, Exclude excludes a file or directory, unless the - directory is explicitly mentioned in a AlwaysInclude - directive. - - If a directive ends in Regex, then it is a regular expression - rather than a explicit full pathname. See - - man 7 re_format - - for the regex syntax on your platform. -
- -
- Example configuration output - - This is an example of output from the bbstored-config - script. - - - Follow the instructions output by your script, not the ones - here -- they may be different for your system. - - - /usr/local/sbin/bbackupd-config /etc/box lazy 51 server.example.com /var/bbackupd /home /etc/samba - -Setup bbackupd config utility. - -Configuration: - Writing configuration file: /etc/box/bbackupd.conf - Account: 51 - Server hostname: server.example.com - Directories to back up: - /home - /etc/samba - -Note: If other file systems are mounted inside these directories, then problems may occur -with files on the store server being renamed incorrectly. This will cause efficiency -problems, but not affect the integrity of the backups. - -WARNING: Directories not checked against mountpoints. Check mounted filesystems manually. - -Creating /etc/box... -Creating /etc/box/bbackupd -Generating private key... - [OpenSSL output omitted] - -Generating keys for file backup -Writing notify script /etc/box/bbackupd/NotifyStoreFull.sh -Writing configuration file /etc/box/bbackupd.conf - -=================================================================== - -bbackupd basic configuration complete. - -What you need to do now... - -1) Make a backup of /etc/box/bbackupd/51-FileEncKeys.raw - This should be a secure offsite backup. - Without it, you cannot restore backups. Everything else can - be replaced. But this cannot. - KEEP IT IN A SAFE PLACE, OTHERWISE YOUR BACKUPS ARE USELESS. - -2) Send /etc/box/bbackupd/51-csr.pem - to the administrator of the backup server, and ask for it to - be signed. - -3) The administrator will send you two files. Install them as - /etc/box/bbackupd/51-cert.pem - /etc/box/bbackupd/serverCA.pem - after checking their authenticity. - -4) You may wish to read the configuration file - /etc/box/bbackupd.conf - and adjust as appropraite. - - There are some notes in it on excluding files you do not - wish to be backed up. - -5) Review the script - /etc/box/bbackupd/NotifyStoreFull.sh - and check that it will email the right person when the store - becomes full. This is important -- when the store is full, no - more files will be backed up. You want to know about this. - -6) Start the backup daemon with the command - /usr/local/sbin/bbackupd - in /etc/rc.local, or your local equivalent. - Note that bbackupd must run as root. - -=================================================================== - - Remember to make a secure, offsite backup of your backup keys, - as described in Basic - configuration above. If you do not, and that key is lost, you - have no backups. -
-
- -
- Configuration Options - - Box Backup has many options in its configuration file. We will - try to list them all here. - - First of all, here is an example configuration file, for - reference: - - - Example Configuration File - - StoreHostname = localhost -AccountNumber = 0x2 - -KeysFile = /etc/box/2-FileEncKeys.raw -CertificateFile = /etc/box/2-cert.pem -PrivateKeyFile = /etc/box/2-key.pem -TrustedCAsFile = /etc/box/serverCA.pem -DataDirectory = /var/run/boxbackup -NotifyScript = /etc/box/NotifySysadmin.sh -CommandSocket = /var/run/box/bbackupd.sock - -UpdateStoreInterval = 86400 -MinimumFileAge = 3600 -MaxUploadWait = 7200 -FileTrackingSizeThreshold = 65536 -DiffingUploadSizeThreshold = 65536 -MaximumDiffingTime = 20 -ExtendedLogging = no -LogAllFileAccess = yes - -Server -{ - PidFile = /var/run/bbackupd.pid -} -BackupLocations -{ - etc - { - Path = /etc - } - home - { - Path = /home - ExcludeDir = /home/shared - ExcludeDir = /home/chris/.ccache - ExcludeDir = /home/chris/.mozilla/firefox/vvvkq3vp.default/Cache - } -} - - - As you can see from the example above, the configuration file - has a number of subsections, enclosed in curly braces {}. Some options - appear outside of any subsection, and we will refer to these as root options. The available options in - each section are described below. - - Every option has the form name = value. Names are - not case-sensitive, but values are. Depending on the option, the value - may be: - - - - a path (to a file or directory); - - - - a number (usually in seconds or bytes); - - - - a boolean (the word Yes or No); - - - - a hostname (or IP address). - - - - Paths are specified in native format, i.e. a full Windows path - with drive letter on Windows clients, or a full Unix path on Unix - clients. - - - Example: - - StoreObjectInfoFile = - /var/state/boxbackup/bbackupd.dat - - StoreObjectInfoFile = C:\Program Files\Box - Backup\data\bbackupd.dat - The use of relative paths (which do not start with a - forward slash on Unix, or a drive specification on Windows) is - possible but not recommended, since they are interpreted relative to - the current working directory when bbackupd was started, which is - liable to change unexpectedly over time. - - Numbers which start with "0x" are interpreted as hexadecimal. - Numbers which do not start with "0x" are interpreted as - decimal. - -
- Root Options - - These options appear outside of any subsection. By convention - they are at the beginning of the configuration file. - - Some options are required, and some are optional. - - - - StoreHostname (required) - - - The Internet host name (DNS name) or IP address of the - server. This is only used to connect to the server. - - - - - AccountNumber (required) - - - The number of the client's account on the server. This - must be provided by the server operator, and must match the - account number in the client's certificate, otherwise the - client will not be able to log into the server. - - The account number may be specified in hexadecimal - (starting with 0x, as in the example above) or in decimal, but - since the server operator works in hexadecimal, that format is - highly recommended and is the default. - - - - - KeysFile (required) - - - The path to the file containing the encryption key used - for data encryption of client file data and filenames. This is - the most important file to keep safe, since without it your - backups cannot be decrypted and are useless. Likewise, if an - attacker gets access to this key and to your encrypted - backups, he can decrypt them and read all your data. - - Do not change the encryption key without deleting all - files from the account on the server first. None of your old - files on the store will be readable if you do so, and if you - change it back, none of the files uploaded with the new key - will be readable. - - - - - CertificateFile (required) - - - The path to the OpenSSL client certificate in PEM - format. This is supplied by the server operator in response to - the certificate request which you send to them. Together with - the PrivateKeyFile, this provides access to the store server - and the encrypted data stored there. - - It is not critical to protect this file or to back it up - safely, since it can be regenerated by creating a new - certificate request, and asking the server operator to sign - it. You may wish to back it up, together with the - PrivateKeyFile, to avoid this inconvenience if you lose all - your data and need quick access to your backups. - - If you do back them up, you should keep them in a - separate location to the KeysFile, since any person holding - the KeysFile and the PrivateKeyFile can gain access to your - encrypted data and decrypt it. - - - - - PrivateKeyFile (required) - - - The path to the OpenSSL private key in PEM format. This - is generated at the same time as the certificate request, but - there is no need to send it to the server operator, and you - should not do so, in case the communication is intercepted by - an attacker. Together with the CertificateFile, this provides - access to the store server and the encrypted data stored - there. - - See the notes under CertificateFile for information - about backing up this file. - - - - - TrustedCAsFile (required) - - - The path to the OpenSSL certificate of the Client - Certificate Authority (CCA), in PEM format. This is supplied - by the server operator along with your account details, or - along with your signed client certificate. This is used to - verify that the server which you are connecting to is - authorised by the person who signed your certificate. It - protects you against DNS and ARP poisoning and IP spoofing - attacks. - - - - - DataDirectory (required) - - - The path to a directory where bbackupd will keep local - state information. This consists of timestamp files which - identify the last backup start and end times, used by - bbackupquery to determine whether files - have changed, and optionally a database of inode numbers, - which are used to check for files being renamed. The database - is only saved if Box Backup is built with Berkeley Database - (BDB) support. - - - - - NotifyScript (optional) - - - The path to the script or command to run when the Box - Backup client detects an error during the backup process. This - is normally used to notify the client system administrator by - e-mail when a backup fails for any reason. - - The script or command is called with one of the - following additional arguments to identify the cause of the - problem: - - - - store-full - - - The backup store is full. No new files are being - uploaded. If some files are marked as deleted, they - should be removed in due course by the server's - housekeeping process. Otherwise, you need to remove some - files from your backup set, or ask the store operator - for more space. - - - - - read-error - - - One or more files which were supposed to be backed - up could not be read. This could be due to: - - running the server as a non-root user; - - - - backing up a mounted filesystem such as - NFS; - - - - access control lists being applied to some - files; - - - - SELinux being enabled; - - - - trying to back up open files under - Windows; - - - - strange directory permissions such as 0000 or - 0400. - - Check the client logs, e.g. - /var/log/bbackupd on Unix, or the Windows Event Viewer - in Control Panel > Administrative Tools, for more - information about which files are not being backed up - and why. - - - - - backup-error - - - There was a communications error with the server, - or an unexpected exception was encountered during a - backup run. Check the client logs, e.g. - /var/log/box on Unix, or the - Windows Event Viewer in Control Panel > - Administrative Tools, for more information about the - problem. - - You may wish to check your Internet access to the - server, check that the server is running, and ask your - server operator to check your account on the - server. - - - - - - - - CommandSocket (optional) - - - The path to the Unix socket which - bbackupd creates when running, and which - bbackupctl uses to communicate with it, for - example to force a sync or a configuration reload. If this - option is omitted, no socket will be created, and - bbackupctl will not function. - - Unix sockets appear within the filesystem on Unix, as a - special type of file, and must be created in a directory which - exists and to which bbackupd has write access, and bbackupctl - has read access. - - On Windows, the path is ignored, and a named - pipe is created instead. This does not currently - have any security attached, so it can be accessed by any user. - Unlike a Unix socket it can also be accessed remotely. Please - use this option with extreme caution on Windows, and only on - fully trusted networks. - - - - - AutomaticBackup (optional) - - - Enable or disable the client from connecting - automatically to the store every - UpdateStoreInterval seconds. When - enabled (set to Yes), the client is in - Lazy Mode. When disabled (set to - No), it is in Snapshot - Mode. This setting is optional, and the default - value is Yes. - - - - - UpdateStoreInterval (required) - - - The approximate time between successive connections to - the server, in seconds, when the client is in Lazy - Mode. The actual time is randomised slightly to - prevent "rush hour" traffic jams on the server, where many - clients try to connect at the same time. - - This value is ignored if the client is in - Snapshot Mode. However, it is still - required. It can be set to zero in this case. - - You will probably need to experiment with the value of - this option. A good value to start with is probably 86400 - seconds, which is one day. - - - - - MinimumFileAge (required) - - - The number of seconds since a file was last modified - before it will be backed up. The reason for this is to avoid - repeatedly backing up files which are repeatedly changing. A - good value is about 3600 seconds (one hour). If set to zero, - files which have changed will always be backed up on the next - backup run. - - The MaxUploadWait option - overrides this option in some circumstances. - - - - - MaxUploadWait (required) - - - The number of seconds since a file was last uploaded - before it will be uploaded again, even if it keeps changing. - The reason for this is to ensure that files which are - continuously modified are eventually uploaded anyway. This - should be no less than the value of - MinimumFileAge. A good value is about - 14400 seconds (4 hours). - - - - - MaxFileTimeInFuture (optional) - - - The maximum time that a file's timestamp can be in the - future, before it will be backed up anyway. Due to clock - synchronisation problems, it is inevitable that you will - occasionally see files timestamped in the future. Normally, - for files which are dated only slightly in the future, you - will want to wait until after the file's date before backing - it up. However, for files whose dates are very wrong (more - than a few hours) you will normally prefer to back them up - immediately. - - A good value is about 7200 seconds (2 hours) to cope - with potential problems when moving in and out of daylight - saving time, if applicable in your timezone. The default - value, if this setting is not provided, is 172800 seconds (2 - days). - - - - - FileTrackingSizeThreshold (required) - - - The minimum size of files which will be tracked by inode - number to detect renames. It is not worth detecting renames of - small files, since they are quick to upload again in full, and - keeping their inode numbers in memory increases the client's - memory usage and slows down searches. Larger files should be - tracked to avoid wasting space on the store and long - uploads. - - A good value is about 65536 bytes (64 kilobytes). - - - - - DiffingUploadSizeThreshold (required) - - - The minimum size of files which will be compared to the - old file on the server, and for which only changes will be - uploaded. It is not worth comparing small files, since they - are quick to upload again in full, and sending the entire file - reduces the risk of data loss if the store is accidentally - corrupted. Larger files should have only their differences - uploaded to avoid wasting space on the store and long - uploads. - - A good value is about 65536 bytes (64 kilobytes). - - - - - MaximumDiffingTime (optional) - - - The maximum time for which the client will attempt to - find differences between the current version and the old - version in the store, before giving up and uploading the - entire file again. Very large files (several gigabytes) may - take a very long time to scan for changes, but would also take - a very long time to upload again and use a lot of space on the - store, so it is normally worth omitting this value. - - Use this option only if, for some bizarre reason, you - prefer to upload really large files in full rather than spend - a long time scanning them for changes. - - - - - KeepAliveTime (optional) - - - The interval (in seconds) between sending Keep-Alive - messages to the server while performing long operations such - as finding differences in large files, or scanning large - directories. - - These messages ensure that the SSL connection is not - closed by the server, or an intervening firewall, due to lack - of activity. - - The server will normally wait up to 15 minutes (900 - seconds) before disconnecting the client, so the value should - be given and should be less than 900. Some firewalls may time - out inactive connections after 10 or 5 minutes. - - A good value is 300 seconds (5 minutes). You may need to - reduce this if you frequently see TLSReadFailed or - TLSWriteFailed errors on the client. - - - - - StoreObjectInfoFile (optional) - - - Enables the use of a state file, which stores the - client's internal state when the client is not running. This - is useful on clients machines which are frequently shut down, - for example desktop and laptop computers, because it removes - the need for the client to recontact the store and rescan all - directories on the first backup run, which may take some time. - This feature is somewhat experimental and not well tested. - - - This is option is disabled by default, in which case the - state is stored in memory only. The value is the path to the - state file. - - - - - ExtendedLogging (optional) - - - Enables the connection debugging mode of the client, - which writes all commands sent to or received from the server - to the system logs. This generates a lot - of output, so it should only be used when instructed, or when - you suspect a connection problem or client-server protocol - error (and you know how to interpret the output). - - This is a boolean value, which may be set to - Yes or No. The default is of - course No. - - - - - ExtendedLogFile (optional, new in 0.11) - - - Enables the same debugging output as - ExtendedLogging, but written to a file - instead of the system logs. This is useful if you need - extended logging, but do not have access to the system logs, - for example if you are not the administrator of the - computer. - - The value is the path to the file where these logs will - be written. If omitted, extended logs will not be written to a - file. This is entirely independent of the - ExtendedLogging option. It does not - make much sense to use both at the same time. - - - - - LogAllFileAccess (optional, new in 0.11) - - - Enables logging of all local file and directory access, - file uploads (full and differential), and excluded files. This - may be useful if the client is failing to upload a particular - file, or crashing while trying to upload it. The logs will be - sent to the system log or Windows Event Viewer. - - This generates a lot - of output, so it should only be used when instructed, or when - you suspect that bbackupd is skipping some files and want to - know why. Because it is verbose, the messages are hidden by - default even if the option is enabled. To see them, you must - run bbackupd with at least one -v option. - - This is a boolean value, which may be set to - Yes or No. The default is of - course No. - - - - - SyncAllowScript (optional) - - - The path to the script or command to run when the client - is about to start an automatic backup run, and wishes to know - whether it is safe to do so. This is useful for clients which - do not always have access to the server, for example laptops - and computers on dial-up Internet connections. - - The script should either output the word - now if the backup should proceed, or else a - number, in seconds, which indicates how long the client should - wait before trying to connect again. Any other output will - result in an error on the client, and the backup will not - run. - - This value is optional, and by default no such script is - used. - - - -
- -
- Server Section - - These options appear within the Server subsection, which is at - the root level. - - - - PidFile - - - This option enables the client to write its processs - identifier (PID) to the specified file after starting. The - file will be deleted when the client daemon exits for any - reason. This is disabled by default, but is recommended - whenever you run the client daemon as a daemon (in the - background), which is usually the case. This file can be used - by scripts to determine whether the daemon is still running, - and to send it messages to reload its configuration or to - terminate. - - - Example Server Section - - Server -{ - PidFile = /var/state/boxbackup/bbackupd.pid -} - - - - -
- -
- Backup Locations Section - - This section serves only as a container for all defined backup - locations. - - - Example Backup Locations Section - - BackupLocations -{ - etc - { - Path = /etc - } - home - { - Path = /home - ExcludeDir = /home/shared - ExcludeDir = /home/chris/.ccache - ExcludeDir = /home/chris/.mozilla/firefox/vvvkq3vp.default/Cache - } -} - - - Each subsection is a backup location. The name of the - subsection is the name that will be used on the server. The root - directory of the account on the server contains one subdirectory per - location. The name should be simple, not containing any spaces or - special characters. - - If you do not define any locations, the client will not back - up any files! - - It is currently not recommended to back up the root directory - of the filesystem on Unix. Box Backup is designed to back up - important data and configuration files, not full systems. - Nevertheless, nothing prevents you from doing so if you - desire. - - On Windows, it is currently not possible to back up files - which are open (currently in use), such as open documents in - Microsoft Office, and system files such as the registry and the - paging file. You will get an error for each open file which the - client attempts to back up. Once the file has been closed, it will - be backed up normally. System files will always be open, and should - be excluded from your backups. -
-
-
- - - - Administration - - This chapter deals with the dauily running and management of the Box - Backup system. It explains most day-to-day tasks. - -
- Regular Maintenance - - The steps involved in maintaining and keeping the backup sets - healthy are outlined in this section. - -
- Controlling a backup client - - The bbackupctl program sends control commands to the bbackupd - daemon. It must be run as the same user as the daemon, and there is no - exception for root. - - The command line syntax is: - - /usr/local/sbin/bbackupctl [-q] [-c config-file] command - - The -q option reduces the amount of output the program emits, - and -c allows an alternative configuration file to be - specified. - - Valid commands are: - - - - terminate - - Stop the bbackupd daemon now (equivalent to kill) - - - - reload - - Reload the configuration file (equivalent to kill - -HUP) - - - - sync - - Connect to the server and synchronise files now - - - - bbackupctl communicates with - the server via a UNIX domain socket, specified in bbackupd.conf with - the CommandSocket directive. This does not need to be specified, and - bbackupd will run without the command - socket, but in this case bbackupctl will not be able to communicate - with the daemon. - - Some platforms cannot check the user id of the connecting - process, so this command socket becomes a denial of service security - risk. bbackupd will warn you when it - starts up if this is the case on your platform, and you should - consider removing the CommandSocket directive on these - platforms. -
- -
- Using bbackupctl to perform snapshots - - bbackupctl's main purpose is to - implement snapshot based backups, emulating the behaviour of - traditional backup software. - - Use bbackupd-config to write a configuration file in snapshot - mode, and then run the following command as a cron job. - - /usr/local/sbin/bbackupctl -q sync - - This will cause the backup daemon to upload all changed files - immediately. bbackupctl will exit - almost immediately, and will not output anything unless there is an - error. -
- -
- Checking storage space used on the server - -
- From the client machine - - bbackupquery can tell you how much space is used on the server - for this account. Either use the usage command in interactive mode, - or type: - - /usr/local/sbin/bbackupquery -q usage quit - - to show the space used as a single command. -
- -
- On the server - - bbstoreaccounts allows you to query the space used, and change - the limits. To display the space used on the server for an account, - use: - - /usr/local/sbin/bbstoreaccounts info 75AB23C - - To adjust the soft and hard limits on an account, use: - - /usr/local/sbin/bbstoreaccounts setlimit 75AB23C new-soft-limit new-hard-limit - - You do not need to restart the server. -
-
- -
- Verify and restore files - - Backups are no use unless you can restore them. The bbackupquery - utility does this and more. - - You don't provide any login information to it, as it just picks - up the data it needs from /etc/box/bbackupd.conf. You should run it as - root so it can find everything it needs. - - Full documentation can be found in the bbackupquery manual page. It follows - the model of a command line sftp client quite closely. - - TODO: Link to bbackupquery man-page here. - - On systems where GNU readline is available (by default) it uses - that for command line history and editing. Otherwise it falls back to - very basic UNIX text entry. - - TODO: Did the readline dependency change to editline? - -
- Using bbackupquery - - bbackupquery is the tool you use to verify, restore and - investigate your backup files with. When invoked, it simply logs - into the server using the certificates you have listed in - bbackupd.conf. - - After you run bbackupquery, you will see a prompt, allowing - you to execute commands. The list (or ls) command lets you view - files in the store. It works much like unix ls, but with different - options. An example: - - [pthomsen at host bbackupquery]$ bbackupquery -Box Backup Query Tool v0.10, (c) Ben Summers and contributors 2003-2006 -Using configuration file /etc/box/bbackupd.conf -Connecting to store... -Handshake with store... -Login to store... -Login complete. - -Type "help" for a list of commands. - -query > ls -00000002 -d---- mp3 -00000003 -d---- video -00000004 -d---- home-pthomsen -00000005 -d---- root -query > - - The ls commands shows the directories that are backed up. Now - we'll take a closer look at the home-pthomsen directory: - - query > cd home-pthomsen -query > ls -00002809 f----- sample.tiff -0000280a f----- s3.tiff -0000280b f----- s4.tiff -0000280d f----- s2.tiff -0000280e f----- foo.pdf -0000286c f----- core.28720 -0000339a -d---- .emacs.d -0000339d -d---- bbackup-contrib -00003437 f----- calnut.compare.txt -0000345d f----- DSCN1783.jpg -0000345e f----- DSCN1782.jpg -query > - - The ls command takes the following options; - - - - -r -- recursively list - all files - - - - -d -- list deleted - files/directories - - - - -o -- list old versions - of files/directories - - - - -I -- don't display - object ID - - - - -F -- don't display - flags - - - - -t -- show file - modification time (and attr mod time if has the object has - attributes, ~ separated) - - - - -s -- show file size in - blocks used on server (only very approximate indication of size - locally) - - - - The flags displayed from the ls command are as follows: - - - f = file - - d = directory - - X = deleted - - o = old version - - R = remove from server as soon as marked deleted or - old - - a = has attributes stored in directory record which - override attributes in backup file - -
- -
- Verify backups - - As with any backup system, you should frequently check that - your backups are working properly by comparing them. Box Backup - makes this very easy and completely automatic. All you have to do is - schedule the bbackupquery compare command to run - regularly, and check its output. You can run the command manually as - follows: - - /usr/local/sbin/bbackupquery "compare -a" quit - - This command will report all the differences found between the - store and the files on disc. It will download everything, so may - take a while. You should expect to see some differences on a typical - compare, because files which have recently changed are unlikely to - have been uploaded yet. It will also tell you how many files have - been modified since the last backup run, since these will normally - have changed, and such failures are expected. - - You are strongly recommended to add this command as a - cron job, at least once a month, and to check the - output for anything suspicious, particularly a large number of - compare failures, failures on files that have not been modified, or - any error (anything except a compare mismatch) that occurs during - the compare operation. - - Consider keeping a record of these messages and comparing them - with a future verification. - - If you would like to do a "quick" check which just downloads - file checksums and compares against that, then run: - - /usr/local/sbin/bbackupquery "compare -aq" quit - - However, this does not check that the file attributes are - correct, and since the checksums are generated on the client they - may not reflect the data on the server if there is a problem -- the - server cannot check the encrypted contents. View this as a quick - indication, rather than a definite check that your backup verifies - correctly. -
- -
- Restore backups - - You will need the keys file created when you configured the - server. Without it, you cannot restore the files; this is the - downside of encrypted backups. However, by keeping the small keys - file safe, you indirectly keep your entire backup safe. - - The first step is to recreate the configuration of the backup - client. It's probably best to have stored the /etc/box directory - with your keys. But if you're recreating it, all you really need is - to have got the login infomation correct (ie the certs and - keys). - - Don't run bbackupd yet! It will mark all your files as deleted - if you do, which is not hugely bad in terms of losing data, just a - major inconvenience. (This assumes that you are working from a blank - slate. If you want to restore some files to a different location, - it's fine to restore while bbackupd is running, just do it outside a - backed up directory to make sure it doesn't start uploading the - restored files.) - - Type: - - /usr/local/sbin/bbackupquery - - to run it in interactive mode. - - Type: - - list - - to see a list of the locations stored on the server. - - For each location you want to restore, type: - - restore name-on-server local-dir-name - - The directory specified by local-dir-name must not exist yet. - If the restore is interrupted for any reason, repeat the above - steps, but add the -r flag to the - restore command to tell it to resume. -
- -
- Retrieving deleted and old files - - Box Backup makes old versions of files and files you have - deleted available, subject to there being enough disc space on the - server to hold them. - - This is how to retrieve them using bbackupquery. Future - versions will make this far more user-friendly. - - Firstly, run bbackupquery in interactive mode. It behaves in a - similar manner to a command line sftp client. - - /usr/local/sbin/bbackupquery - - Then navigate to the directory containing the file you want, - using list, cd and pwd. - - query > cd home/profiles/USERNAME - - List the directory, using the "o" option to list the files - available without filtering out everything apart from the current - version. (if you want to see deleted files as well, use list - -odt) - - query > list -ot -00000078 f--o- 2004-01-21T20:17:48 NTUSER.DAT -00000079 f--o- 2004-01-21T20:17:48 ntuser.dat.LOG -0000007a f--o- 2004-01-21T17:55:12 ntuser.ini -0000007b f---- 2004-01-12T15:32:00 ntuser.pol -0000007c -d--- 1970-01-01T00:00:00 Templates -00000089 -d--- 1970-01-01T00:00:00 Start Menu -000000a0 -d--- 1970-01-01T00:00:00 SendTo -000000a6 -d--- 1970-01-01T00:00:00 Recent -00000151 -d--- 1970-01-01T00:00:00 PrintHood -00000152 -d--- 1970-01-01T00:00:00 NetHood -00000156 -d--- 1970-01-01T00:00:00 My Documents -0000018d -d--- 1970-01-01T00:00:00 Favorites -00000215 -d--- 1970-01-01T00:00:00 Desktop -00000219 -d--- 1970-01-01T00:00:00 Cookies -0000048b -d--- 1970-01-01T00:00:00 Application Data -000005da -d--- 1970-01-01T00:00:00 UserData -0000437e f--o- 2004-01-24T02:45:43 NTUSER.DAT -0000437f f--o- 2004-01-24T02:45:43 ntuser.dat.LOG -00004380 f--o- 2004-01-23T17:01:29 ntuser.ini -00004446 f--o- 2004-01-24T02:45:43 NTUSER.DAT -00004447 f--o- 2004-01-24T02:45:43 ntuser.dat.LOG -000045f4 f---- 2004-01-26T15:54:16 NTUSER.DAT -000045f5 f---- 2004-01-26T15:54:16 ntuser.dat.LOG -000045f6 f---- 2004-01-26T16:54:31 ntuser.ini - - (this is a listing from a server which is used as a Samba - server for a network of Windows clients.) You now need to fetch the - file using it's ID, rather than it's name. The ID is the hex number - in the first column. Fetch it like this: - - query > get -i 0000437e NTUSER.DAT -Object ID 0000437e fetched successfully. - - The object is now available on your local machine. You can use - lcd to move around, and sh ls to list directories on your local - machine. -
-
-
- -
- Fixing corruptions of store data - - This section gives help on what to do if your server has suffered - corruption, for example, after an unclean shutdown or other operating - system or hardware problem. - - In general, as updates to the store are made in an atomic manner, - the most likely result is wasted disc space. However, if really bad - things happen, or you believe that there is a lot of wasted space, then - these instructions will help to restore your data. - - You know you will need to do something if you get strange errors, - and bbackupd attempts to contact the server every 100 seconds or so. Or - if one of the discs in your RAID disc set has failed. - - After following these instructions, the end result will be that - bbackupquery will be able to see all the files which were stored on your - server, and retrieve them. Some of them may be in lost+found directories - in the root of the store (or in their original position if they have - been moved) but they will all be able to be retrieved. - - After you have retrieved the files you want, bbackupd will upload - new versions where necessary, and after about two days, mark any - lost+found directories as deleted. Finally, those directories will be - removed by the housekeeping process on the server. - - These instructions assume you're working on account 1234. Replace - this with the account number that you actually want to check (the one - that is experiencing errors). These steps will need to be repeated for - all affected accounts. - -
- Stop bbackupd - - First, make sure that bbackupd is not running on the client - machine for the account you are going to recover. Use - bbackupctl terminate to stop it. This step is not - strictly necessary, but is recommended. During any checks on the - account, bbackupd will be unable to log in, and after they are - complete, the account is marked as changed on the server so bbackupd - will perform a complete scan. -
- -
- Are you using RAID on the server? - - The raidfile recovery tools have not been written, and probably - will not be, since Box Backup RAID is deprecated. However, when two - out of three files are available, the server will successfully allow - access to your data, even if it complains a lot in the logs. The best - thing to do is to fix the accounts, if necessary, and retrieve any - files you need. Then move the old store directories aside (in case you - need them) and start afresh with new accounts, and let the clients - upload all their data again. -
- -
- Check and fix the account - - First, run the check utility, and see what errors it - reports. - - /usr/local/sbin/bbstoreaccounts check 1234 - - This will take some time, and use a fair bit of memory (about 16 - bytes per file and directory). If the output looks plausible and - reports errors which need fixing, run it again but with the fix - flag: - - /usr/local/sbin/bbstoreaccounts check 1234 fix - - This will fix any errors, and remove unrecoverable files. - Directories will be recreated if necessary. - - NOTE: The utility may adjust - the soft and hard limits on the account to make sure that housekeeping - will not remove anything -- check these afterwards. -
- -
- Grab any files you need with bbackupquery - - At this point, you will have a working store. Every file which - was on the server, and wasn't corrupt, will be available. - - On the client, use bbackupquery to log in and examine the store. - (type help at the prompt for instructions). Retrieve any files you - need, paying attention to any lost+found directories in the root - directory of the store. - - You can skip this step if you are sure that the client machine - is fine -- in this case, bbackupd will bring the store up to - date. -
- -
- Restart bbackupd - - Restart bbackupd on the client machine. The store account will - be brought up to date, and files in the wrong place will be marked for - eventual deletion. -
-
- -
- Troubleshooting - - If you are trying to fix a store after your disc has been - corrupted, see Fixing corruptions of - store data. - - Unfortunately, the error messages are not particularly helpful at - the moment. This page lists some of the common errors, and the most - likely causes of them. - - When an error occurs, you will see a message like 'Exception: - RaidFile/OSFileError (2/8)' either on the screen or in your log files. - (it is recommended you set up another log file as recommended in the - server setup instructions.) - - This error may not be particularly helpful, although some do have - extra information about probable causes. To get further information, - check the ExceptionCodes.txt file in the root of the distribution. This - file is generated by the ./configure script, so you will need to have - run that first. - - Some common causes of exceptions are listed below. - - Please email me with any other codes you get, and I will let you - know what they mean, and add notes here. - -
- RaidFile (2/8) - - This is found either when running bbstoreaccounts or in the - bbstored logs. - - Problem: The directories you - specified in the raidfile.conf are not writable by the _bbstored - user. - - Resolution: Change permissions - appropriately. -
- -
- Common (1/2) - - This usually occurs when the configuration files can't be - opened. - - Problem: You created your - configurations in non-standard locations, and the programs cannot find - them. - - Resolution: Explicitly specify - configuration file locations to daemons and programs. For - example - - /usr/local/sbin/bbstored /some/other/dir/bbstored.config /usr/local/sbin/bbackupquery -c /some/other/dir/bbackupd.config - - (daemons specify the name as the first argument, utility - programs with the -c option). - - Problem: bbstored can't find - the raidfile.conf file specified in bbstored.conf. - - Resolution: Edit bbstored.conf - to point to the correct location of this additional configuration - file. -
- -
- Server (3/16) - - The server can't listen for connections on the IP address - specified when you configured it. - - Problem: This probably means - you've specified the wrong hostname to bbstored-config -- maybe your - server is behind a NAT firewall? - - Resolution: Edit bbstored.conf - and correct the ListenAddresses line. You should replace the server - address with the IP address of your machine. -
- -
- Connection (7/x) - - These errors all relate to connections failing -- you may see - them during operation if there are network failures or other problems - between the client and server. The backup system will recover from - them automatically. - -
- Connection (7/30) - SSL problems - - Log snippet from client side: - - bbackupd[1904]: Opening connection to server xxxx.xxx... -bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 -bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed -bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:asn1 encoding routines:ASN1_verify:EVP lib -bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -bbackupd[1904]: TRACE: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(237) -bbackupd[1904]: Exception caught (7/30), reset state and waiting to retry... - - And from the server: - - bbstored[19291]: Incoming connection from xx.xxx.xx.xxx port xxxxx (handling in child xxxxx) -bbstored[21588]: SSL err during Accept: error:xxxxxxxx:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error -bbstored[21588]: in server child, exception Connection TLSHandshakeFailed (7/30) -- terminating child - - Solution: Create a new CA on - the server side and re-generate the client certificate. Re-creating - the client certificate request is not necessary. -
-
- -
- Advanced troubleshooting - - If this really doesn't help, then using the DEBUG builds of the - system will give you much more information -- a more descriptive - exception message and the file and line number where the error - occurred. - - For example, if you are having problems with bbstoreaccounts, - build the debug version with: - - cd boxbackup-0.0 -cd bin/bbstoreaccounts -make - - Within the module directories, make defaults to building the - debug version. At the top level, it defaults to release. - - This will build an executable in debug/bin/bbstoreaccounts which - you can then use instead of the release version. It will give far more - useful error messages. - - When you get an error message, use the file and line number to - locate where the error occurs in the code. There will be comments - around that line to explain why the exception happened. - - If you are using a debug version of a daemon, these extended - messages are found in the log files. -
-
-
- - &__ExceptionCodes__elfjz3fu; - - - Running without root - - It is possible to run both the server and client without root - privileges. - -
- Server - - The server, by default, runs as a non-root user. However, it - expects to be run as root and changes user to a specified user as soon - as it can, simply for administrative convenience. The server uses a port - greater than 1024, so it doesn't need root to start. - - To run it entirely as a non-root user, edit the bbstored.conf - file, and remove the User directive from the Server section. Then simply - run the server as your desired user. -
- -
- Client - - The client requires root for normal operation, since it must be - able to access all files to back them up. However, it is possible to run - the client as a non-root user, with certain limitations. - - Follow the installation instructions, but install the executable - files manually to somewhere in your home directory. Then use - bbackupd-config to configure the daemon, but use a directory other than - /etc/box, probably somewhere in your home directory. - - All directories you specify to be backed up must be readable, and - all files must be owned by the user and readable to that user. - - Important: If any file or directory is not readable by this user, - the backup process will skip that file or directory. Keep an eye on the - logs for reports of this failure. - - Non-root operation of the backup client is recommended only for - testing, and should not be relied on in a production environment. -
-
- Deleted: box/trunk/docs/bb-book.xsl =================================================================== --- box/trunk/docs/bb-book.xsl 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bb-book.xsl 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,17 +0,0 @@ - - - - - - - - - - - - - Deleted: box/trunk/docs/bb-man.xsl.tmpl =================================================================== --- box/trunk/docs/bb-man.xsl.tmpl 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bb-man.xsl.tmpl 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,9 +0,0 @@ - - - - - - - - Deleted: box/trunk/docs/bb-nochunk-book.xsl =================================================================== --- box/trunk/docs/bb-nochunk-book.xsl 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bb-nochunk-book.xsl 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,17 +0,0 @@ - - - - - - - - - - - - - Deleted: box/trunk/docs/bbackupctl.xml =================================================================== --- box/trunk/docs/bbackupctl.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bbackupctl.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,205 +0,0 @@ - - - - bbackupctl - - 8 - - Box Backup - - Box Backup - - 0.11 - - - - bbackupctl - - Control the Box Backup client daemon - - - - - bbackupctl - - -q - - -c config-file - - command - - - - - Description - - bbackupctl sends commands to a running - bbackupd daemon on a client machine. It can be used to - force an immediate backup, tell the daemon to reload its configuration - files or stop the daemon. If bbackupd is configured in - snapshot mode, it will not back up automatically, and the - bbackupctl must be used to tell it when to start a - backup. - - Communication with the bbackupd daemon takes place over a local - socket (not over the network). Some platforms (notably Windows) can't - determine if the user connecting on this socket has the correct - credentials to execute the commands. On these platforms, ANY local user - can interfere with bbackupd. To avoid this, remove the CommandSocket - option from bbackupd.conf, which will also disable bbackupctl. See the - Client Configuration page for more information. - - bbackupctl needs to read the - bbackupd configuration file to find out the name of the - CommandSocket. If you have to tell bbackupd where to - find the configuration file, you will have to tell - bbackupctl as well. The default on Unix systems is - usually /etc/box/bbackupd.conf. On Windows systems, - it is bbackupd.conf in the same directory where - bbackupd.exe is located. If - bbackupctl cannot find or read the configuration file, - it will log an error message and exit. - - bbackupctl usually writes error messages to the - console and the system logs. If it is not doing what you expect, please - check these outputs first of all. - - - - - - - Run in quiet mode. - - - - - config-file - - - Specify configuration file. - - - - - - Commands - - The following commands are available in bbackupctl: - - - - terminate - - - This command cleanly shuts down bbackupd. - This is better than killing or terminating it any other - way. - - - - - reload - - - Causes the bbackupd daemon to re-read all - its configuration files. Equivalent to kill - -HUP. - - - - - sync - - - Initiates a backup. If no files need to be backed up, no - connection will be made to the server. - - - - - force-sync - - - Initiates a backup, even if the - SyncAllowScript says that no backup should run - now. - - - - - wait-for-sync - - - Passively waits until the next backup starts of its own - accord, and then terminates. - - - - - wait-for-end - - - Passively waits until the next backup starts of its own - accord and finishes, and then terminates. - - - - - sync-and-wait - - - Initiates a backup, waits for it to finish, and then - terminates. - - - - - - - - Files - - /etc/box/bbackupd.conf - - - - See Also - - - bbackupd.conf - - 5 - , - bbackupd-config - - 8 - , - bbackupctl - - 8 - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Deleted: box/trunk/docs/bbackupd-config.xml =================================================================== --- box/trunk/docs/bbackupd-config.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bbackupd-config.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,153 +0,0 @@ - - - - bbackupd-config - - 8 - - Box Backup - - Box Backup - - 0.11 - - - - bbackupd-config - - Box Backup client daemon configuration file - generator - - - - - bbackupd-config - - config-dir - - backup-mode - - account-num - - server-hostname - - working-dir - - backup-dir - - backup-dir ... - - - - - Description - - The bbackupd-config script creates configuration files and client - certificates. It takes at least six parameters: - - - - config-dir - - - Configuration directory. Usually - /etc/box. - - - - - backup-mode - - - Either lazy or snapshot. - - - - - account-num - - - The client account number. This is set by the bbstored - administrator. - - - - - server-hostname - - - The hostname or IP address of the bbstored server. - - - - - working-dir - - - A directory to keep temporary state files. This is usually - something like /var/bbackupd. This can be - changed in bbackupd.conf later on if - required. - - - - - backup-dir - - - A space-separated list of directories to be backed up. Note - that this does not traverse mount - points. - - - - - - - Files - - /etc/box/bbackupd.conf - - /etc/box/bbackupd/NotifySysAdmin.sh - - - - See Also - - - bbackupd.conf - - 5 - , - bbackupd - - 8 - , - bbackupctl - - 8 - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Deleted: box/trunk/docs/bbackupd.conf.xml =================================================================== --- box/trunk/docs/bbackupd.conf.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bbackupd.conf.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,479 +0,0 @@ - - - - bbackupd.conf - - 5 - - Box Backup - - Box Backup - - 0.11 - - - - bbackupd.conf - - Box Backup client daemon configuration file - - - - - /etc/box/bbackupd.conf - - - - - Description - - - - AccountNumber - - - The account number of this client. This is set by the admin of - the store server. - - - - - UpdateStoreInterval - - - Specifies the interval between scanning of the local discs. To - avoid cycles of load on the server, this time is randomly adjusted - by a small percentage as the daemon runs. Defaults to 1 hour. - - - - - MinimumFileAge - - - Specifies how long since a file was last modified before it - will be uploaded. Defaults to 6 hours. - - - - - MaxUploadWait - - - If a file is repeatedly modified it won't be uploaded - immediately in case it's modified again. However it should be - uploaded eventually. This is how long we should wait after first - noticing a change. Defaults to 1 day. - - - - - MaxFileTimeInFuture - - - - - - - - AutomaticBackup - - - - - - - - SyncAllowScript - - - Use this to temporarily stop bbackupd from syncronising or - connecting to the store. This specifies a program or script script - which is run just before each sync, and ideally the full path to the - interpreter. It will be run as the same user bbackupd is running as, - usually root. - - The script prints either "now" or a number to STDOUT (and a - terminating newline, no quotes). If the result was "now", then the - sync will happen. If it's a number, then the script will be asked - again in that number of seconds. - - For example, you could use this on a laptop to only backup - when on a specific network. - - - - - MaximumDiffingTime - - - How much time should be spent on diffing files. - - - - - DeleteRedundantLocationsAfter - - - - - - - - FileTrackingSizeThreshold - - - - - - - - DiffingUploadSizeThreshold - - - - - - - - StoreHostname - - - The hostname or IP address of the - bbstored - - 8 - server. - - - - - StorePort - - - The port used by the server. Defaults to 2201. - - - - - ExtendedLogging - - - Logs everything that happens between the client and server. - The - bbackupd - - 8 - client must also be started with - . - - - - - ExtendedLogFile - - - - - - - - LogAllFileAccess - - - - - - - - LogFile - - - - - - - - LogFileLevel - - - - - - - - CommandSocket - - - Where the command socket is created in the filesystem. - - - - - KeepAliveTime - - - - - - - - StoreObjectInfoFile - - - - - - - - NotifyScript - - - The location of the script which runs at certain events. This - script is generated by - bbackupd-config - - 8 - . Defaults to - /etc/box/bbackupd/NotifySysAdmin.sh. - - - - - NotifyAlways - - - - - - - - CertificateFile - - - The path to the client's public certificate. - - - - - PrivateKeyFile - - - The path to the client's private key. This should only be - readable by root. - - - - - TrustedCAsFile - - - The Certificate Authority created by - bbstored-certs - - 8 - . - - - - - KeysFile - - - The data encryption key. This must be kept safe at all costs, your data is - useless without it! - - - - - DataDirectory - - - A directory to keep temporary state files. This is usually - something like /var/bbackupd. - - - - - Server - - - This section relates to the running daemon. - - - - PidFile - - - The location of the process ID file. Defaults to - /var/run/bbackupd.pid. - - - - - - - - BackupLocations - - - This section defines each directory to be backed up. Each - entry must have at least a Path entry and, optionally, include and - exclude directives. - - Multiple include and exclude directives may appear. - - - - Path - - - The path to back up. - - - - - ExcludeFile - - - Exclude a single file. - - - - - ExcludeFilesRegex - - - Exclude multiple files based on a regular expression. - See - re_format - - 7 - . - - - - - ExcludeDir - - - Exclude a single directory. - - - - - ExcludeDirsRegex - - - Exclude multiple directories based on a regular - expression. See - re_format - - 7 - . - - - - - AlwaysIncludeFile - - - Include a single file from a directory which has been - excluded. - - - - - AlwaysIncludeFilesRegex - - - Include multiple files from an excluded directory, - based on a regular expression. - - - - - AlwaysIncludeDir - - - Include a single directory from a directory which has - been excluded. - - - - - AlwaysIncludeDirsRegex - - - Include multiple directories from an excluded - directory, based on a regular expression. - - - - - - - - - - Examples - - The following is an example of a backup location: - - home -{ - Path = /home - ExcludeDir = /home/guest - ExcludeDir = /home/[^/]+/tmp - ExcludeFilesRegex = .*\.(mp3|MP3)$ - AlwaysIncludeFile = /home/someuser/importantspeech.mp3 -} - - - - Files - - /etc/box/bbackupd.conf - - - - See Also - - - bbackupd - - 8 - , - bbackupd-config - - 8 - , - bbackupctl - - 8 - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Deleted: box/trunk/docs/bbackupd.xml =================================================================== --- box/trunk/docs/bbackupd.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bbackupd.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,209 +0,0 @@ - - - - bbackupd - - 8 - - Box Backup - - Box Backup - - 0.11 - - - - bbackupd - - Box Backup client daemon - - - - - bbackupd - - -DFkqvVT - - -c config-file - - -t tag - - - - - Description - - bbackupd runs on client computers in the - background, finding new files to back up. When it is time for a backup, - bbackupd will connect to the server - (bbstored) to upload the files. - - A running bbackupd daemon can be controlled with - the bbackupctl command, to make it shut down, reload - its configuration, or start an immediate backup. - - bbackupd needs to be configured to tell it which - files to back up, how often, and to which server (running - bbstored). See the Client Configuration page for more - information. For this, you must write a configuration file. You must - either place it in the default location, or tell - bbackupd where to find it. - - You can check the default location with the - option. The default on Unix systems is usually - /etc/box/bbackupd.conf. On Windows systems, it is - bbackupd.conf in the same directory where - bbackupd.exe is located. If bbackupd cannot find or - read the configuration file, it will log an error message and exit. - - bbackupd usually writes log messages to the - system logs, using the facility local5, which you can - use to filter them to send them to a separate file. It can also write them - to the console, see options below. If bbackupd is not - doing what you expect, please check the logs first of all. - - - Options - - - - config-file - - - Use the specified configuration file. If - is omitted, the last argument is the configuration file. If none - is specified, the default is used (see above). - - - - - - - - Debugging mode. Do not fork into the background (do not run - as a daemon). Not available on Windows. - - - - - - - - No-fork mode. Same as for bbackupd. Not - available on Windows. - - - - - - - - Keep console open after fork, keep writing log messages to - it. Not available on Windows. - - - - - - - - Run more quietly. Reduce verbosity level by one. Available - levels are NOTHING, FATAL, - ERROR, WARNING, - NOTICE, INFO, - TRACE, EVERYTHING. Default - level is NOTICE in non-debugging builds. Use - once to drop to WARNING level, twice for - ERROR level, four times for no logging at - all. - - - - - -v - - - Run more verbosely. Increase verbosity level by one. Use - once to raise to INFO level, twice for - TRACE level, three times for - EVERYTHING (currently the same as - TRACE). - - - - - - - - Run at maximum verbosity (EVERYTHING - level). - - - - - tag - - - Tag each console message with specified marker. Mainly - useful in testing when running multiple daemons on the same - console. - - - - - - - - Timestamp each line of console output. - - - - - - - - Files - - /etc/box/bbackupd.conf - - - - See Also - - - bbackupd.conf - - 5 - , - bbackupd-config - - 8 - , - bbackupctl - - 8 - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Deleted: box/trunk/docs/bbackupquery.xml =================================================================== --- box/trunk/docs/bbackupquery.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bbackupquery.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,506 +0,0 @@ - - - - bbackupquery - - 8 - - Box Backup - - Box Backup - - 0.11 - - - - bbackupquery - - Box Backup store query and file retrieval - - - - - bbackupquery - - -q - - -c configfile - - command ... - - - - - Description - - bbackupquery is the main way of interacting with - the backup store from a Box Backup client machine. It supports both - interactive and batch modes of operation. - - It can be used to reviewing the status of a client machine's backup - store, getting status from the store server. The main use is to retrieve - files and directories when needed. - - bbackupquery supports interactive and batch modes - of operation. Interactive mode allows for interaction with the server much - like an interactive FTP client. - - Batch mode is invoked by putting commands into the invocation of - bbackupquery. Example: - - bbackupquery "list home-dirs" quit - - Note that commands that contain spaces are enclosed in double - quotes. If the quit command is omitted, after the - preceding commands are completed, bbackupquery will - enter interactive mode. - - - - Options - - - - - - - Quiet. Suppresses status output while running. - - - - - - - - Use configfile instead of the default bbackupd.conf file. - Can be a relative or full path. - - - - - - - Commands - - The commands that can be used in bbackupquery are listed - below. - - - - help - - - Displays the basic help message, which gives information about - the commands available in bbackupquery. Use the - form help command to get help on a specific - command. - - - - - quit - - - End the session with the store server, and quit - bbackupquery. - - - - - cd options - directory-name - - - Change directory. Options: - - - - - consider deleted directories for traversal - - - - - - - - consider old versions of directories for traversal. - This option should never be useful in a correctly formed - store. - - - - - - - - lcd - local-directory-name - - - Change directory on the client machine. To list the contents - of the local directory, type sh ls (on Unix-like - machines). - - - - - list options - directory-name - - - The list (or its synonym ls) command lists - the content of the current, or specified, directory. The options are - as follows: - - - - - - - recursively list all files - - - - - - - - list deleted files and directories - - - - - - - - list old versions of files and directories - - - - - - - - don't display object IDs - - - - - - - - don't display flags - - - - - - - - show file modification time (and attr mod time, if the - object has attributes). - - - - - - - - show file size in blocks used on server. Note that - this is only a very approximate indication of local file - size. - - - - - - - - ls options - directory-name - - - Synonym for list. - - - - - pwd - - - Print current directory, always relative to the backup store - root. - - - - - sh shell-command - - - Everything after the sh is passed to a shell and run. All - output from the command is displayed in the client. - - Example: to list the contents of the current directory on the - client machine type sh ls. - - - - - compare -a - - - - - - - - compare -l - location-name - - - - - - - - compare store-dir-name - local-dir-name - - - Compare the current data in the store with the data on the - disc. Please note that all the data will be downloaded from the - store, so this can be a very lengthy process depending on the size - of the store, and the size of the part you are comparing. - - Options: - - - - - - - compare all locations. - - - - - - - - compare one backup location as specified in the - configuration file. This compares one of the top level store - directories. - - - - - - - - set return code. The return code is set to the - following values, if quit is the next command. So, if - another command is run after the compare, the return code - will not refer to the compare. This option is very useful - for automating compares. Return code values: - - -- no differences were - found - - - - -- differences were - found - - - - -- an error occured - - - - - - - - - - get object-filename - local-filename - - - - - - - - get -i object-id - local-filename - - - Gets a file from the store. Object is specified as the - filename within the current directory. Local filename is optional. - Ignores old and deleted files when searching the directory for the - file to retrieve. - - To get an old or deleted file, use the - option and select the object as a hex object ID (first column in - listing). The local filename must be specified. - - - - - getobject object-id - local-filename - - - Gets the object specified by the object id (in hex) and stores - the raw contents in the local file specified. Note: This is only - useful for debugging as it does not decode files from the stored - format, which is encrypted and compressed. - - - - - restore -d - directory-name - local-directory-name - - - - - - - - restore -r - - - Restores a directory to the local disc. The local directory - specified must not exist (unless a previous restore is being - restarted). The root cannot be restored -- restore locations - individually. - - Options: - - - - - - - restore a deleted directory - - - - - - - - resume an interrupted restore - - - If a restore operation is interrupted for any - reason, it can be restarted using the switch. - Restore progress information is saved in a file at regular intervals - during the restore operation to allow restarts. - - - - - usage -m - - - Show space used on the server for this account. Display - fields: - - Used: Total amount of space used on - the server - - - - Old files: Space used by old - files - - - - Deleted files: Space used by - deleted files - - - - Directories: Space used by the - directory structure - - - - When Used exceeds the soft limit, the - server will start to remove old and deleted files until the usage - drops below the soft limit. After a while, you should expect to see - the usage stay at just below the soft limit. You only need more - space if the space used by old and deleted files is near - zero. - - The option displays output in - machine-readable form. - - - - - - - Bugs - - If you find a bug in Box Backup and you want to let us know about - it, join the mailing - list and send us a description of the problem there. - - To report a bug, give us at least the following information: - - - - The version of Box Backup you are running - - - - The platform you are running on (hardware and OS), for both - client and server. - - - - If possible attach your config files (bbstored.conf, - bbackupd.conf) to the bug report. - - - - Also attach any log file output that helps shed light on the - problem you are seeing. - - - - And last but certainly not least, a description of what you are - seeing, in as much detail as possible. - - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Deleted: box/trunk/docs/bblogo-alpha.xcf =================================================================== (Binary files differ) Deleted: box/trunk/docs/bbstoreaccounts.xml =================================================================== --- box/trunk/docs/bbstoreaccounts.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bbstoreaccounts.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,386 +0,0 @@ - - - - bbstoreaccounts - - 8 - - Box Backup - - Box Backup - - 0.11 - - - - bbstoreaccounts - - Box Backup store accounts manager - - - - - bbstoreaccounts - - -c config-file - - command - - account-id - - command-specific arguments - - - - - Description - - bbstoreaccounts is the tool for managing accounts - on the store server. It can be used to view information related to - accounts, as well as create, change and delete accounts on the store - server. - - bbstoreaccounts always takes at least 2 - parameters: the command name and the account ID. Some commands require - additional parameters, and some commands have optional parameters. - - - Options - - - - - - - The configfile to use for connecting to the store. Default - is /etc/box/bbstored.conf. - - - - - - - Commands - - The commands tells bbstoreaccounts what action to perform. - - - - check account-id - fix - - - The check command verifies the - integrity of the store account given, and optionally fixes any - corruptions. Note: It is - recommended to run the 'simple' check command (without - fix) before using the fix - option. This gives an overview of the extent of any problems, - before attempting to fix them. - - - - - create account-id - disc-set soft-limit - hard-limit - - - Creates a new store account with the parameters given. The - parameters are as follows: - - - - - - - The ID of the new account to be created. A 32-bit - hexadecimal number. Cannot already exist on the - server. - - - - - - - - The disc set from - raidfile.conf - - 5 - where the backups for this client will - be stored. A number. Each RAID-file set has a number in - raidfile.conf. This number is what's used. - - - - - - - - The soft limit is the amount of storage that the - server will guarantee to be available for - storage. - - - - - - - - The amount of storage that the the server will - allow, before rejecting uploads, and starting to - eliminate old and deleted files to get back down to - soft-limit. - - - - - - - - delete account-id - yes - - - Deletes the account from the store server completely. - Removes all backups and deletes all references to the account in - the config files. - - delete will ask for confirmation from - the user, when called. Using the flag, - eliminates that need. This is useful when deleting accounts from - within a script or some other automated means. 0 - - - - - info account-id - - - Display information about the given account. - Example:[root]# bbstoreaccounts info 1 - Account ID: 00000001 - Last object ID: 58757 - Blocks used: 9864063 (38531.50Mb) - Blocks used by old files: 62058 (242.41Mb) -Blocks used by deleted files: 34025 (132.91Mb) - Blocks used by directories: 6679 (26.09Mb) - Block soft limit: 11796480 (46080.00Mb) - Block hard limit: 13107200 (51200.00Mb) - Client store marker: 1139559852000000 - - Explanation: - - - - Account ID - - - The account ID being displayed. - - - - - Last Object ID - - - A counter that keeps track of the objects that - have been backed up. This number refers to the last file - that was written to the store. The ID is displayed as a - decimal number, and the object ID can be converted to a - path name to a file as follows: convert the number to - hex (e.g.: 58757 => 0xE585); The last backed up file - will be (relative from the client's store root): - e5/o85.rfw. Longer numbers infer - more directories in the structure, so as an example - 3952697264 as the last object ID gives 0xEB995FB0, which - translates to a backup pathname of - eb/99/5f/ob0.rfw. - - - - - Blocks used - - - The number of blocks used by the store. The size - in Mb depends on the number of blocks, as well as the - block size for the disc set given in - raidfile.conf - - 5 - . In this case the block size is - 4096. - - - - - Blocks used by old files - - - The number of blocks occupied by files that have - newer versions in the store. This data is at risk for - being removed during housekeeping. - - - - - Blocks used by deleted files - - - The number of blocks used by files that have been - deleted on the client. This data is at risk for being - removed during housekeeping. - - - - - Blocks used by directories - - - The number of blocks used by directories in the - store. - - - - - Block soft limit - - - The soft limit in blocks. The soft limit is the - maximum guaranteed storage space available to the - account. When housekeeping starts, and the old and - deleted files are removed, they are removed in - chronological order (oldest first), until the data used - is less than the soft limit. - - - - - Block hard limit - - - The hard limit in blocks. The hard limit is the - most amount of storage the server will allow in an - account. Any data above this amount will be rejected. - Housekeeping will reduce the storage use, so more data - can be uploaded. - - - - - Client store marker - - - - bbstored - - 8 - uses this number to determine if it - needs to rescan the entire store. If this number is - different from the last time it checked, a rescan will - take place. - - - - - - - - setlimit account-id - soft-limit hard-limit - - - Changes the storage space allocation for the given - account. No server restart is needed. - - Parameters: - - - - - - - The ID of the account to be modified. - - - - - - - - The soft limit is the amount of storage that the - server will guarantee to be available for - storage. - - - - - - - - The amount of storage that the the server will - allow before rejecting uploads and starting to eliminate - old and deleted files to get back down to - . - - - - - - - - - - - Examples - - Create an account with ID 3af on disc set 0, with a 20GB soft-limit - and a 22GB hard-limit:bbstoreaccounts create 3af 0 20G 22GAlter - existing account ID 20 to have a 50GB soft-limit and a 55GB - hard-limit:bbstoreaccounts setlimit 20 50G 55G - - - - Files - - /etc/box/bbstored/accounts.txt - - - - See Also - - - bbstored - - 8 - , - bbstored-config - - 8 - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Deleted: box/trunk/docs/bbstored-certs.xml =================================================================== --- box/trunk/docs/bbstored-certs.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bbstored-certs.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,180 +0,0 @@ - - - - bbstored-certs - - 8 - - Box Backup - - Box Backup - - 0.11 - - - - bbstored-certs - - Manage certificates for the Box Backup system - - - - - bbstored-certs - - certs-dir - - command - - arguments - - - - - Description - - bbstored-certs creates and signs certificates for - use in Box Backup. It allows the user to create and sign the server keys, - as well as signing client keys. - - All commands must be followed by the certs-dir, - which is the directory in which the certificates are stored. - - - Commands - - There are 3 commands: - - - - init - - - Create the certs-dir, and generate the - server keys for bbstored. certs-dir cannot - exist before running the command. - - - - - sign-server - servercsrfile - - - Sign the server certificate. The - servercsrfile is the file generated by the - init command. - - - - - sign - clientcsrfile - - - Sign a client certificate. The - clientcsrfile is generated during client setup. - See - bbackupd-config - - 8 - . Send the signed certificate back to the client, - and install according to the instructions given by - bbackupd-config. - - - - - - - - Files - - - raidfile-config - - 8 - generates the - raidfile.conf - - 5 - file. - - - - Bugs - - If you find a bug in Box Backup, and you want to let us know about - it, join the mailing - list, and send a description of the problem there. - - To report a bug, give us at least the following information: - - - - The version of Box Backup you are running - - - - The platform you are running on (hardware and OS), for both - client and server. - - - - If possible attach your config files (bbstored.conf, - bbackupd.conf) to the bug report. - - - - Also attach any log file output that helps shed light on the - problem you are seeing. - - - - And last but certainly not least, a description of what you are - seeing, in as much detail as possible. - - - - - - See Also - - - bbstored-config - - 8 - , - bbstored.conf - - 5 - , - bbstoreaccounts - - 8 - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Deleted: box/trunk/docs/bbstored-config.xml =================================================================== --- box/trunk/docs/bbstored-config.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bbstored-config.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,148 +0,0 @@ - - - - bbstored-config - - 8 - - Box Backup - - Box Backup - - 0.11 - - - - bbstored-config - - Box Backup store daemon configuration file - generator - - - - - bbstored-config - - configdir - - servername - - username - - - - - Description - - The bbstored-config script creates configuration files and server - certificates for a bbstored instance. It takes three parameters: - - - - configdir - - - The directory where config files will reside. A - bbstored subdirectory will be created where - several config files will reside. The - bbstored.conf file will be created in - configdir. - - - - - servername - - - The name of the server that is being configured. Usually the - fully qualified domain name of the machine in question. - - - - - username - - - The name of the user that should be running the - bbstored process. Recommended name: - _bbstored. - - - - - A valid - raidfile.conf - - 5 - must be found in configdir. Several steps are taken - during the run of bbstored-config: - - - - Server certificates are created. This requires interaction from - the operator. - - - - The RAID volumes are checked to ensure that the configuration is - consistent and will work. - - - - Instructions for next steps to take are shown. These steps may - be different for different OS platforms, so pay close attention to - these instructions. - - - - - - Files - - /etc/box/bbstored.conf - - - - See Also - - - bbstored.conf - - 5 - , - bbstored - - 8 - , - bbstored-certs - - 8 - , - raidfile-config - - 8 - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Deleted: box/trunk/docs/bbstored.conf.xml =================================================================== --- box/trunk/docs/bbstored.conf.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bbstored.conf.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,211 +0,0 @@ - - - - bbstored.conf - - 5 - - Box Backup - - Box Backup - - 0.11 - - - - bbstored.conf - - Box Backup store daemon configuration file - - - - - /etc/box/bbstored.conf - - - - - Description - - The following configuration options are valid: - - - - RaidFileConf - - - Specifies the path to the - raidfile.conf - - 5 - . This is normally - /etc/box/raidfile.conf. - - - - - AccountDatabase - - - Specifies the path to the account database created by - - bbstoreaccounts - - 8 - . This is usually - /etc/box/bbstored/accounts.txt. - - - - - ExtendedLogging - - - Specifies whether extended logging should be enabled to show - what commands are being received from clients. - - - - - TimeBetweenHousekeeping - - - How long between scanning for files which need - deleting. - - - - - Server - - - These options relate to the actual daemon. - - PidFile - - - The location of the pidfile, where the daemon's - process ID is kept. - - - - - User - - - The user to run as. - - - - - ListenAddresses - - - The interface addresses to listen on. Hostnames may be - used instead of IP addresses. The format is: - or - . - - - - - CertificateFile - - - The path to the server's public certificate. - - - - - PrivateKeyFile - - - The path to the server's private key. This should only - be readable by root and/or the . - - - - - TrustedCAsFile - - - The Certificate Authority created by - bbstored-certs - - 8 - . - - - - - - - - - - Examples - - The following is an example bbstored.conf: - - RaidFileConf = /etc/box/raidfile.conf -AccountDatabase = /etc/box/bbstored/accounts.txt - -TimeBetweenHousekeeping = 900 - -Server -{ - PidFile = /var/run/bbstored.pid - User = _bbstored - ListenAddresses = inet:server.example.com - CertificateFile = /etc/box/bbstored/server.example.com-cert.pem - PrivateKeyFile = /etc/box/bbstored/server.example.com-key.pem - TrustedCAsFile = /etc/box/bbstored/clientCA.pem -} - - - - Files - - /etc/box/bbstored.conf - - - - See Also - - - bbstored - - 8 - , - bbstored-config - - 8 - , - raidfile-config - - 8 - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Deleted: box/trunk/docs/bbstored.xml =================================================================== --- box/trunk/docs/bbstored.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/bbstored.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,90 +0,0 @@ - - - - bbstored - - 8 - - Box Backup - - Box Backup - - 0.11 - - - - bbstored - - Box Backup store daemon - - - - - bbstored - - config-file - - - - - Description - - bbstored runs on a central server. Clients - running bbackupd connect to the server and upload - files. - - The only argument is optional and specifies a non-default - configuration file. By default it will look for the configuration file as - /etc/box/bbstored.conf. - - - - Files - - /etc/box/bbstored.conf - - - - See Also - - - bbstored.conf - - 5 - , - bbstored-config - - 8 - , - raidfile-config - - 8 - , - raidfile.conf - - 5 - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Copied: box/trunk/docs/docbook/adminguide.xml (from rev 2463, box/trunk/docs/adminguide.xml) =================================================================== --- box/trunk/docs/docbook/adminguide.xml (rev 0) +++ box/trunk/docs/docbook/adminguide.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,1981 @@ + + +]> + + Box Backup administrator's guide + + + License + + Copyright ?? 2003 - 2007, Ben Summers and contributors. All rights + reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + + + Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + + + Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following disclaimer + in the documentation and/or other materials provided with the + distribution. + + + + All use of this software and associated advertising materials + must display the following acknowledgement: This product includes + software developed by Ben Summers and contributors. + + + + The names of the Authors may not be used to endorse or promote + products derived from this software without specific prior written + permission. + + + + [Where legally impermissible the Authors do not disclaim liability + for direct physical injury or death caused solely by defects in the + software unless it is modified by a third party.] + + THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS OR + IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN + NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED + TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR + PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING + NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + + + Configuration + +
+ System configuration + +
+ Server + + After you've downloaded and compiled the programs you need to + install the programs on your server. As root do the following: + + make install-backup-server + + This assumes that you are installing on the same server that you + compiled the software on. If not, copy the + boxbackup-x.xx-backup-server-OSNAME.tgz file to the server you want to + run on, and install there. For example (on Mac OS X): + + tar zxvf boxbackup-0.10-server-darwin8.5.0.tgz +cd boxbackup-0.10-server-darwin8.5.0 +./install-backup-server + + Then create the user for the backup daemon on the server: + + useradd _bbstored + + Box Backup has a built-in software RAID facility (redundant + array of inexpensive disks) for the backup store. This allows you to + spread the store data over three disks, and recover from the loss of + any one disk without losing data. However, this is now deprecated, and + you are recommended to use the software or hardware RAID facilities of + your operating system instead. Use the following command if you want + to create a simple server without Box Backup RAID: + + mkdir /tmp/boxbackupRepository # Create the directory +chown _bbstored /tmp/boxbackupRepository/ # Change the owner to the new boxbackup daemon user + +/usr/local/sbin/raidfile-config /etc/box/ 1024 /tmp/boxbackupRepository + +#substitute 1024 with the desired blocksize +#substitute /tmp/boxbackupRepository with a directory that exists where you want the backup store located +#/usr/local/sbin/raidfile-config --help shows you the options + + Then create the configuration file /etc/box/bbstored.conf The + hostname is tricky as it is used for two things: The name of the + server in the certificate and the address the server is listening on. + Since you might be using NAT, might move the server around or the + domain name might change, choose a name that describes the server. + When the network address of the server changes, you need to update the + ListenAddresses directive in the + /etc/box/bbstored.conf file. + + /usr/local/sbin/bbstored-config /etc/box hostname _bbstored + + This last step outputs 5 instructions that you must execute to + the letter. A lot of questions are raised on the mailing list because + these steps have not been followed properly. + + TODO: Expand on this. Explain the 5 steps in detail. + + If you want to run the server as a non-root user, look here. +
+ +
+ Certificate Management + + There are two steps involved to create an account. You need to + create the account on the server, and sign a certificate to give the + client permission to connect to the server. + + Running a Certification Authority for TLS (SSL) connections is + not trivial. However, a script to does most of the work in a way which + should be good enough for most deployments. + + + The certificate authority directory is intended to be stored + on another server. It should not be kept on the backup server, in + order to limit the impact of a server compromise. The instructions + and the script assume that it will be kept elsewhere, so will ask + you to copy files to and from the CA. + + + + SSL certificates contain validity dates, including a "valid + from" time. If the clock on the machine which signs the certificates + is not syncronised to the clocks of the machines using these + certificates, you will probably get strange errors until the start + time is reached on all machines. If you get strange errors when + attempting to use new certificates, check the clocks on all machines + (client, store and CA). You will probably just need to wait a while + until the certificates become valid, rather than having to + regenerate them. + + +
+ Set up a Certificate Authority + + It is recommended that you keep your Certificate Authority on + a separate machine than either the client or the server, preferably + without direct network access. The contents of this directory + control who can access your backup store server. + + To setup the basic key structure, do the following: + + /usr/local/sbin/bbstored-certs ca init + + (See OpenSSL notes if you + get an OpenSSL error) + + This creates the directory called ca in + the current directory, and initialises it with basic keys. +
+ +
+ Sign a server certificate + + When you use the bbstored-config script to + set up a config file for a server, it will generate a certificate + request (CSR) for you. Transfer it to the machine with your CA, then + do: + + /usr/local/sbin/bbstored-certs ca sign-server hostname-csr.pem + + This signs the certificate for the server. Follow the + instructions in the output on which files to install on the server. + The CSR file is now no longer needed. Make sure you run this command + from the directory above the directory 'ca'. + + TODO: Explain instructions in output. +
+ +
+ Set up an account + + Choose an account number for the user. This must be unique on + the server, and is presented as a 31 bit number in hex greater than + 0, for example, 1 or 75AB23C. Then on the backup store server, + create the account with: + + /usr/local/sbin/bbstoreaccounts create 75AB23C 0 4096M 4505M + + This looks complicated. The numbers are, in order: + + + + The account number allocated (hex) + + + + The RAID disc set (0 if you use raidfile-config and don't + add a new set) + + + + Soft limit (size) + + + + Hard limit (size) + + + + The sizes are are specified in Mb, Gb, or blocks, depending on + the suffix. 1M specifies 1 Mb, 1G specifies 1 Gb, and 1B specifies 1 + block, the size of which depends on how you have configured the + raidfile system with raidfile-config. + + In this example, I have allocated 4Gb (assuming you use 2048 + byte blocks as per my example) as the soft limit, and 4Gb + 10% as + the hard limit. + + NOTE The sizes specified here are pre-RAID. So if you are + using userland RAID, you are actually allocating two-thirds of this + amount. This means that, when you take compression into account, + that if you allocate 2Gb on the server, it'll probably hold about + 2Gb of backed up files (depending on the compressability of those + files). + + The backup client will (voluntarily) try not to upload more + data than is allowed by the soft limit. The store server will refuse + to accept a file if it would take it over the hard limit, and when + doing housekeeping for this account, try and delete old versions and + deleted files to reduce the space taken to below the soft + limit. + + This command will create some files on disc in the raid file + directories (if you run as root, the utility will change to the user + specified in the bbstored.conf file to write them) and update the + accounts file. A server restart is not required. + + NOTE If you get a message saying 'Exception: RaidFile (2/8)', + the directories you specified in the raidfile.conf are not writable + by the _bbstored user -- fix it, and try again. + + Finally, tell the user their account number, and the hostname + of your server. They will use this to set up the backup client, and + send you a CSR. This has the account number embedded in it, and you + should be sure that it has the right account number in it. + + Sign this CSR with this command: + + /usr/local/sbin/bbstored-certs ca sign 75AB23C-csr.pem + + Don't forget to check that the embedded account number is + correct! Then send the two files back to the user, as instructed by + the script. + + Please read the Troubleshooting page if you have + problems. + + TODO: Link to troubleshooting... +
+
+ +
+ Log Files + + You may wish to see what's going on with the server. Edit + /etc/syslog.conf, and add: + + local6.info /var/log/box +local5.info /var/log/raidfile + + Note: Separators must be tabs, + otherwise these entries will be ignored. + + touch /var/log/box +touch /var/log/raidfile + + Set up log rotation for these new log files. For example, if you + have /etc/newsyslog.conf, add the following lines + to it: + + /var/log/box 644 7 2000 * Z +/var/log/raidfile 644 7 2000 * Z + + If you have /etc/logrotate.d, create a new + file in there (for example + /etc/logrotate.d/boxbackup) containing the + following: + + /var/log/box /var/log/raidfile { + weekly + create + compress + rotate 52 +} + + Then restart syslogd, for example: + + /etc/init.d/syslogd restart +
+ +
+ Configuring a client + + Before you can do any configuration, you need to know the + hostname of the server you will be using, and your account number on + that server. + + Later in the process, you will need to send a certificate + request to the administrator of that server for it to be + signed. + + Installation is covered in the compiling and installing section. + You only need the backup-client parcel. + + It is important that you read all the output of the config + scripts. See the end of this page for an example. + + The backup client has to be run as root, because it needs to + read all your files to back them up, although it is possible to back + up a single user's files by running it as that user. (Tip: specify a + directory other than /etc/box, and then give the + alternate config file as the first argument to + bbackupd). However, it will fall over if you don't + give yourself read access to one of your files. + +
+ Basic configuration + + Run the bbackupd-config script to generate + the configuration files and generate a private key and certificate + request. + + /usr/local/sbin/bbackupd-config /etc/box lazy 999 hostname /var/bbackupd /home + + (See OpenSSL notes if you + get an OpenSSL error) + + The items in bold need to be changed. In order, they are the + account number, the hostname of the server you're using, and + finally, the directories you want backed up. You can include as many + you want here. + + However, the directories you specify must not contain other + mounted file systems within them at any depth. Specify them + separately, one per mount point. No checks are currently made to + catch bad configuration of this nature! + + You may also want to consider changing the mode from lazy to + snapshot, depending on what your system is used for: + + + + Lazy Mode + + + This mode regularly scans the files, with only a rough + schedule. It uploads files as and when they are changed, if + the latest version is more than a set age. This is good for + backing up user's documents stored on a server, and spreads + the load out over the day. + + + + + Snapshot Mode + + + This mode emulates the traditional backup behaviour of + taking a snapshot of the filesystem. The backup daemon does + absolutely nothing until it is instructed to make a backup + using the bbackupctl utility (probably as a cron job), at + which point it uploads all files which have been changed since + the last time it uploaded. + + + + + When you run the config script, it will tell you what you need + to do next. Don't forget to read all the output. An example is shown + at the end of this page, but the instructions for your installation + may be different. +
+ +
+ Certificates + + After you have sent your certificate request off to the server + administrator and received your certificate and CA root back, + install them where instructed by the bbackupd-config script during + basic bbackupd configuration. + + You can then run the daemon (as root) by running + /usr/local/sbin/bbackupd, and of course, adding it + to your system's startup scripts. The first time it's run it will + upload everything. Interrupting it and restarting it will only + upload files which were not uploaded before - it's very + tolerant. + + If you run in snapshot mode, you will need to add a cron job + to schedule backups. The config script will tell you the exact + command to use for your system. + + Please read the Troubleshooting page if you have + problems. + + Remember to make a traditional backup of the keys file, as + instructed. You cannot restore files without it. + + It is recommended that you backup up all of /etc/box as it + will make things easier if you need to restore files. But only the + keys are absolutely essential. + + If you want to see what it's doing in more detail (probably a + good idea), follow the instructions in the server setup to create + new log files with syslog. +
+ +
+ Adding and removing backed up locations + + By editing the /etc/box/bbackupd.conf file, you can add and + remove directories to back up - see comments in this file for help. + Send bbackupd a HUP signal after you modify it. + + When you remove a location, it will not be marked as deleted + immediately. Instead, bbackupd waits about two days before doing so, + just in case you change your mind. After this, it will be eventually + removed from the store by the housekeeping process. Run as + root. + + The backup client is designed to be run as root. It is + possible to run without root, but this is not recommended. Clock + synchronisation for file servers. + + If you are using the backup client to backup a filesystem + served from a fileserver, you should ideally ensure that the + fileserver clocks are synchronised with the fileserver. + + bbackupd will cope perfectly well if the clocks are not + synchronised. Errors up to about half an hour cause no problems. + Larger discrepancies cause a loss of efficiency and the potential to + back up a file during a write process. + + There is a configuration parameter MaxFileTimeInFuture, which + specifies how far in the future a file must be for it to be uploaded + as soon as it is seen. You should not need to adjust this (default + is 2 days). Instead, get those clocks synchronised. Excluding files + and directories from the backup. + + Within the bbackupd.conf file, there is a section named + BackupLocations which specifies which locations on disc should be + backed up. It has subsections, each of which is in the + format: + + name + { + Path = /path/of/directory + (optional exclude directives) + } + + name is derived from the Path + by the config script, but should merely be unique. + + The exclude directives are of the form: + + [Exclude|AlwaysInclude][File|Dir][|sRegex] = regex or full pathname + + (The regex suffix is shown as 'sRegex' to make File or Dir + plural) + + For example: + + ExcludeDir = /home/guest-user + ExcludeFilesRegex = *.(mp3|MP3)\$ + AlwaysIncludeFile = /home/username/veryimportant.mp3 + + This excludes the directory /home/guest-user from the backup + along with all mp3 files, except one MP3 file in particular. + + In general, Exclude excludes a file or directory, unless the + directory is explicitly mentioned in a AlwaysInclude + directive. + + If a directive ends in Regex, then it is a regular expression + rather than a explicit full pathname. See + + man 7 re_format + + for the regex syntax on your platform. +
+ +
+ Example configuration output + + This is an example of output from the bbstored-config + script. + + + Follow the instructions output by your script, not the ones + here -- they may be different for your system. + + + /usr/local/sbin/bbackupd-config /etc/box lazy 51 server.example.com /var/bbackupd /home /etc/samba + +Setup bbackupd config utility. + +Configuration: + Writing configuration file: /etc/box/bbackupd.conf + Account: 51 + Server hostname: server.example.com + Directories to back up: + /home + /etc/samba + +Note: If other file systems are mounted inside these directories, then problems may occur +with files on the store server being renamed incorrectly. This will cause efficiency +problems, but not affect the integrity of the backups. + +WARNING: Directories not checked against mountpoints. Check mounted filesystems manually. + +Creating /etc/box... +Creating /etc/box/bbackupd +Generating private key... + [OpenSSL output omitted] + +Generating keys for file backup +Writing notify script /etc/box/bbackupd/NotifyStoreFull.sh +Writing configuration file /etc/box/bbackupd.conf + +=================================================================== + +bbackupd basic configuration complete. + +What you need to do now... + +1) Make a backup of /etc/box/bbackupd/51-FileEncKeys.raw + This should be a secure offsite backup. + Without it, you cannot restore backups. Everything else can + be replaced. But this cannot. + KEEP IT IN A SAFE PLACE, OTHERWISE YOUR BACKUPS ARE USELESS. + +2) Send /etc/box/bbackupd/51-csr.pem + to the administrator of the backup server, and ask for it to + be signed. + +3) The administrator will send you two files. Install them as + /etc/box/bbackupd/51-cert.pem + /etc/box/bbackupd/serverCA.pem + after checking their authenticity. + +4) You may wish to read the configuration file + /etc/box/bbackupd.conf + and adjust as appropraite. + + There are some notes in it on excluding files you do not + wish to be backed up. + +5) Review the script + /etc/box/bbackupd/NotifyStoreFull.sh + and check that it will email the right person when the store + becomes full. This is important -- when the store is full, no + more files will be backed up. You want to know about this. + +6) Start the backup daemon with the command + /usr/local/sbin/bbackupd + in /etc/rc.local, or your local equivalent. + Note that bbackupd must run as root. + +=================================================================== + + Remember to make a secure, offsite backup of your backup keys, + as described in Basic + configuration above. If you do not, and that key is lost, you + have no backups. +
+
+ +
+ Configuration Options + + Box Backup has many options in its configuration file. We will + try to list them all here. + + First of all, here is an example configuration file, for + reference: + + + Example Configuration File + + StoreHostname = localhost +AccountNumber = 0x2 + +KeysFile = /etc/box/2-FileEncKeys.raw +CertificateFile = /etc/box/2-cert.pem +PrivateKeyFile = /etc/box/2-key.pem +TrustedCAsFile = /etc/box/serverCA.pem +DataDirectory = /var/run/boxbackup +NotifyScript = /etc/box/NotifySysadmin.sh +CommandSocket = /var/run/box/bbackupd.sock + +UpdateStoreInterval = 86400 +MinimumFileAge = 3600 +MaxUploadWait = 7200 +FileTrackingSizeThreshold = 65536 +DiffingUploadSizeThreshold = 65536 +MaximumDiffingTime = 20 +ExtendedLogging = no +LogAllFileAccess = yes + +Server +{ + PidFile = /var/run/bbackupd.pid +} +BackupLocations +{ + etc + { + Path = /etc + } + home + { + Path = /home + ExcludeDir = /home/shared + ExcludeDir = /home/chris/.ccache + ExcludeDir = /home/chris/.mozilla/firefox/vvvkq3vp.default/Cache + } +} + + + As you can see from the example above, the configuration file + has a number of subsections, enclosed in curly braces {}. Some options + appear outside of any subsection, and we will refer to these as root options. The available options in + each section are described below. + + Every option has the form name = value. Names are + not case-sensitive, but values are. Depending on the option, the value + may be: + + + + a path (to a file or directory); + + + + a number (usually in seconds or bytes); + + + + a boolean (the word Yes or No); + + + + a hostname (or IP address). + + + + Paths are specified in native format, i.e. a full Windows path + with drive letter on Windows clients, or a full Unix path on Unix + clients. + + + Example: + + StoreObjectInfoFile = + /var/state/boxbackup/bbackupd.dat + + StoreObjectInfoFile = C:\Program Files\Box + Backup\data\bbackupd.dat + The use of relative paths (which do not start with a + forward slash on Unix, or a drive specification on Windows) is + possible but not recommended, since they are interpreted relative to + the current working directory when bbackupd was started, which is + liable to change unexpectedly over time. + + Numbers which start with "0x" are interpreted as hexadecimal. + Numbers which do not start with "0x" are interpreted as + decimal. + +
+ Root Options + + These options appear outside of any subsection. By convention + they are at the beginning of the configuration file. + + Some options are required, and some are optional. + + + + StoreHostname (required) + + + The Internet host name (DNS name) or IP address of the + server. This is only used to connect to the server. + + + + + AccountNumber (required) + + + The number of the client's account on the server. This + must be provided by the server operator, and must match the + account number in the client's certificate, otherwise the + client will not be able to log into the server. + + The account number may be specified in hexadecimal + (starting with 0x, as in the example above) or in decimal, but + since the server operator works in hexadecimal, that format is + highly recommended and is the default. + + + + + KeysFile (required) + + + The path to the file containing the encryption key used + for data encryption of client file data and filenames. This is + the most important file to keep safe, since without it your + backups cannot be decrypted and are useless. Likewise, if an + attacker gets access to this key and to your encrypted + backups, he can decrypt them and read all your data. + + Do not change the encryption key without deleting all + files from the account on the server first. None of your old + files on the store will be readable if you do so, and if you + change it back, none of the files uploaded with the new key + will be readable. + + + + + CertificateFile (required) + + + The path to the OpenSSL client certificate in PEM + format. This is supplied by the server operator in response to + the certificate request which you send to them. Together with + the PrivateKeyFile, this provides access to the store server + and the encrypted data stored there. + + It is not critical to protect this file or to back it up + safely, since it can be regenerated by creating a new + certificate request, and asking the server operator to sign + it. You may wish to back it up, together with the + PrivateKeyFile, to avoid this inconvenience if you lose all + your data and need quick access to your backups. + + If you do back them up, you should keep them in a + separate location to the KeysFile, since any person holding + the KeysFile and the PrivateKeyFile can gain access to your + encrypted data and decrypt it. + + + + + PrivateKeyFile (required) + + + The path to the OpenSSL private key in PEM format. This + is generated at the same time as the certificate request, but + there is no need to send it to the server operator, and you + should not do so, in case the communication is intercepted by + an attacker. Together with the CertificateFile, this provides + access to the store server and the encrypted data stored + there. + + See the notes under CertificateFile for information + about backing up this file. + + + + + TrustedCAsFile (required) + + + The path to the OpenSSL certificate of the Client + Certificate Authority (CCA), in PEM format. This is supplied + by the server operator along with your account details, or + along with your signed client certificate. This is used to + verify that the server which you are connecting to is + authorised by the person who signed your certificate. It + protects you against DNS and ARP poisoning and IP spoofing + attacks. + + + + + DataDirectory (required) + + + The path to a directory where bbackupd will keep local + state information. This consists of timestamp files which + identify the last backup start and end times, used by + bbackupquery to determine whether files + have changed, and optionally a database of inode numbers, + which are used to check for files being renamed. The database + is only saved if Box Backup is built with Berkeley Database + (BDB) support. + + + + + NotifyScript (optional) + + + The path to the script or command to run when the Box + Backup client detects an error during the backup process. This + is normally used to notify the client system administrator by + e-mail when a backup fails for any reason. + + The script or command is called with one of the + following additional arguments to identify the cause of the + problem: + + + + store-full + + + The backup store is full. No new files are being + uploaded. If some files are marked as deleted, they + should be removed in due course by the server's + housekeeping process. Otherwise, you need to remove some + files from your backup set, or ask the store operator + for more space. + + + + + read-error + + + One or more files which were supposed to be backed + up could not be read. This could be due to: + + running the server as a non-root user; + + + + backing up a mounted filesystem such as + NFS; + + + + access control lists being applied to some + files; + + + + SELinux being enabled; + + + + trying to back up open files under + Windows; + + + + strange directory permissions such as 0000 or + 0400. + + Check the client logs, e.g. + /var/log/bbackupd on Unix, or the Windows Event Viewer + in Control Panel > Administrative Tools, for more + information about which files are not being backed up + and why. + + + + + backup-error + + + There was a communications error with the server, + or an unexpected exception was encountered during a + backup run. Check the client logs, e.g. + /var/log/box on Unix, or the + Windows Event Viewer in Control Panel > + Administrative Tools, for more information about the + problem. + + You may wish to check your Internet access to the + server, check that the server is running, and ask your + server operator to check your account on the + server. + + + + + + + + CommandSocket (optional) + + + The path to the Unix socket which + bbackupd creates when running, and which + bbackupctl uses to communicate with it, for + example to force a sync or a configuration reload. If this + option is omitted, no socket will be created, and + bbackupctl will not function. + + Unix sockets appear within the filesystem on Unix, as a + special type of file, and must be created in a directory which + exists and to which bbackupd has write access, and bbackupctl + has read access. + + On Windows, the path is ignored, and a named + pipe is created instead. This does not currently + have any security attached, so it can be accessed by any user. + Unlike a Unix socket it can also be accessed remotely. Please + use this option with extreme caution on Windows, and only on + fully trusted networks. + + + + + AutomaticBackup (optional) + + + Enable or disable the client from connecting + automatically to the store every + UpdateStoreInterval seconds. When + enabled (set to Yes), the client is in + Lazy Mode. When disabled (set to + No), it is in Snapshot + Mode. This setting is optional, and the default + value is Yes. + + + + + UpdateStoreInterval (required) + + + The approximate time between successive connections to + the server, in seconds, when the client is in Lazy + Mode. The actual time is randomised slightly to + prevent "rush hour" traffic jams on the server, where many + clients try to connect at the same time. + + This value is ignored if the client is in + Snapshot Mode. However, it is still + required. It can be set to zero in this case. + + You will probably need to experiment with the value of + this option. A good value to start with is probably 86400 + seconds, which is one day. + + + + + MinimumFileAge (required) + + + The number of seconds since a file was last modified + before it will be backed up. The reason for this is to avoid + repeatedly backing up files which are repeatedly changing. A + good value is about 3600 seconds (one hour). If set to zero, + files which have changed will always be backed up on the next + backup run. + + The MaxUploadWait option + overrides this option in some circumstances. + + + + + MaxUploadWait (required) + + + The number of seconds since a file was last uploaded + before it will be uploaded again, even if it keeps changing. + The reason for this is to ensure that files which are + continuously modified are eventually uploaded anyway. This + should be no less than the value of + MinimumFileAge. A good value is about + 14400 seconds (4 hours). + + + + + MaxFileTimeInFuture (optional) + + + The maximum time that a file's timestamp can be in the + future, before it will be backed up anyway. Due to clock + synchronisation problems, it is inevitable that you will + occasionally see files timestamped in the future. Normally, + for files which are dated only slightly in the future, you + will want to wait until after the file's date before backing + it up. However, for files whose dates are very wrong (more + than a few hours) you will normally prefer to back them up + immediately. + + A good value is about 7200 seconds (2 hours) to cope + with potential problems when moving in and out of daylight + saving time, if applicable in your timezone. The default + value, if this setting is not provided, is 172800 seconds (2 + days). + + + + + FileTrackingSizeThreshold (required) + + + The minimum size of files which will be tracked by inode + number to detect renames. It is not worth detecting renames of + small files, since they are quick to upload again in full, and + keeping their inode numbers in memory increases the client's + memory usage and slows down searches. Larger files should be + tracked to avoid wasting space on the store and long + uploads. + + A good value is about 65536 bytes (64 kilobytes). + + + + + DiffingUploadSizeThreshold (required) + + + The minimum size of files which will be compared to the + old file on the server, and for which only changes will be + uploaded. It is not worth comparing small files, since they + are quick to upload again in full, and sending the entire file + reduces the risk of data loss if the store is accidentally + corrupted. Larger files should have only their differences + uploaded to avoid wasting space on the store and long + uploads. + + A good value is about 65536 bytes (64 kilobytes). + + + + + MaximumDiffingTime (optional) + + + The maximum time for which the client will attempt to + find differences between the current version and the old + version in the store, before giving up and uploading the + entire file again. Very large files (several gigabytes) may + take a very long time to scan for changes, but would also take + a very long time to upload again and use a lot of space on the + store, so it is normally worth omitting this value. + + Use this option only if, for some bizarre reason, you + prefer to upload really large files in full rather than spend + a long time scanning them for changes. + + + + + KeepAliveTime (optional) + + + The interval (in seconds) between sending Keep-Alive + messages to the server while performing long operations such + as finding differences in large files, or scanning large + directories. + + These messages ensure that the SSL connection is not + closed by the server, or an intervening firewall, due to lack + of activity. + + The server will normally wait up to 15 minutes (900 + seconds) before disconnecting the client, so the value should + be given and should be less than 900. Some firewalls may time + out inactive connections after 10 or 5 minutes. + + A good value is 300 seconds (5 minutes). You may need to + reduce this if you frequently see TLSReadFailed or + TLSWriteFailed errors on the client. + + + + + StoreObjectInfoFile (optional) + + + Enables the use of a state file, which stores the + client's internal state when the client is not running. This + is useful on clients machines which are frequently shut down, + for example desktop and laptop computers, because it removes + the need for the client to recontact the store and rescan all + directories on the first backup run, which may take some time. + This feature is somewhat experimental and not well tested. + + + This is option is disabled by default, in which case the + state is stored in memory only. The value is the path to the + state file. + + + + + ExtendedLogging (optional) + + + Enables the connection debugging mode of the client, + which writes all commands sent to or received from the server + to the system logs. This generates a lot + of output, so it should only be used when instructed, or when + you suspect a connection problem or client-server protocol + error (and you know how to interpret the output). + + This is a boolean value, which may be set to + Yes or No. The default is of + course No. + + + + + ExtendedLogFile (optional, new in 0.11) + + + Enables the same debugging output as + ExtendedLogging, but written to a file + instead of the system logs. This is useful if you need + extended logging, but do not have access to the system logs, + for example if you are not the administrator of the + computer. + + The value is the path to the file where these logs will + be written. If omitted, extended logs will not be written to a + file. This is entirely independent of the + ExtendedLogging option. It does not + make much sense to use both at the same time. + + + + + LogAllFileAccess (optional, new in 0.11) + + + Enables logging of all local file and directory access, + file uploads (full and differential), and excluded files. This + may be useful if the client is failing to upload a particular + file, or crashing while trying to upload it. The logs will be + sent to the system log or Windows Event Viewer. + + This generates a lot + of output, so it should only be used when instructed, or when + you suspect that bbackupd is skipping some files and want to + know why. Because it is verbose, the messages are hidden by + default even if the option is enabled. To see them, you must + run bbackupd with at least one -v option. + + This is a boolean value, which may be set to + Yes or No. The default is of + course No. + + + + + SyncAllowScript (optional) + + + The path to the script or command to run when the client + is about to start an automatic backup run, and wishes to know + whether it is safe to do so. This is useful for clients which + do not always have access to the server, for example laptops + and computers on dial-up Internet connections. + + The script should either output the word + now if the backup should proceed, or else a + number, in seconds, which indicates how long the client should + wait before trying to connect again. Any other output will + result in an error on the client, and the backup will not + run. + + This value is optional, and by default no such script is + used. + + + +
+ +
+ Server Section + + These options appear within the Server subsection, which is at + the root level. + + + + PidFile + + + This option enables the client to write its processs + identifier (PID) to the specified file after starting. The + file will be deleted when the client daemon exits for any + reason. This is disabled by default, but is recommended + whenever you run the client daemon as a daemon (in the + background), which is usually the case. This file can be used + by scripts to determine whether the daemon is still running, + and to send it messages to reload its configuration or to + terminate. + + + Example Server Section + + Server +{ + PidFile = /var/state/boxbackup/bbackupd.pid +} + + + + +
+ +
+ Backup Locations Section + + This section serves only as a container for all defined backup + locations. + + + Example Backup Locations Section + + BackupLocations +{ + etc + { + Path = /etc + } + home + { + Path = /home + ExcludeDir = /home/shared + ExcludeDir = /home/chris/.ccache + ExcludeDir = /home/chris/.mozilla/firefox/vvvkq3vp.default/Cache + } +} + + + Each subsection is a backup location. The name of the + subsection is the name that will be used on the server. The root + directory of the account on the server contains one subdirectory per + location. The name should be simple, not containing any spaces or + special characters. + + If you do not define any locations, the client will not back + up any files! + + It is currently not recommended to back up the root directory + of the filesystem on Unix. Box Backup is designed to back up + important data and configuration files, not full systems. + Nevertheless, nothing prevents you from doing so if you + desire. + + On Windows, it is currently not possible to back up files + which are open (currently in use), such as open documents in + Microsoft Office, and system files such as the registry and the + paging file. You will get an error for each open file which the + client attempts to back up. Once the file has been closed, it will + be backed up normally. System files will always be open, and should + be excluded from your backups. +
+
+
+
+ + + Administration + + This chapter deals with the dauily running and management of the Box + Backup system. It explains most day-to-day tasks. + +
+ Regular Maintenance + + The steps involved in maintaining and keeping the backup sets + healthy are outlined in this section. + +
+ Controlling a backup client + + The bbackupctl program sends control commands to the bbackupd + daemon. It must be run as the same user as the daemon, and there is no + exception for root. + + The command line syntax is: + + /usr/local/sbin/bbackupctl [-q] [-c config-file] command + + The -q option reduces the amount of output the program emits, + and -c allows an alternative configuration file to be + specified. + + Valid commands are: + + + + terminate + + Stop the bbackupd daemon now (equivalent to kill) + + + + reload + + Reload the configuration file (equivalent to kill + -HUP) + + + + sync + + Connect to the server and synchronise files now + + + + bbackupctl communicates with + the server via a UNIX domain socket, specified in bbackupd.conf with + the CommandSocket directive. This does not need to be specified, and + bbackupd will run without the command + socket, but in this case bbackupctl will not be able to communicate + with the daemon. + + Some platforms cannot check the user id of the connecting + process, so this command socket becomes a denial of service security + risk. bbackupd will warn you when it + starts up if this is the case on your platform, and you should + consider removing the CommandSocket directive on these + platforms. +
+ +
+ Using bbackupctl to perform snapshots + + bbackupctl's main purpose is to + implement snapshot based backups, emulating the behaviour of + traditional backup software. + + Use bbackupd-config to write a configuration file in snapshot + mode, and then run the following command as a cron job. + + /usr/local/sbin/bbackupctl -q sync + + This will cause the backup daemon to upload all changed files + immediately. bbackupctl will exit + almost immediately, and will not output anything unless there is an + error. +
+ +
+ Checking storage space used on the server + +
+ From the client machine + + bbackupquery can tell you how much space is used on the server + for this account. Either use the usage command in interactive mode, + or type: + + /usr/local/sbin/bbackupquery -q usage quit + + to show the space used as a single command. +
+ +
+ On the server + + bbstoreaccounts allows you to query the space used, and change + the limits. To display the space used on the server for an account, + use: + + /usr/local/sbin/bbstoreaccounts info 75AB23C + + To adjust the soft and hard limits on an account, use: + + /usr/local/sbin/bbstoreaccounts setlimit 75AB23C new-soft-limit new-hard-limit + + You do not need to restart the server. +
+
+ +
+ Verify and restore files + + Backups are no use unless you can restore them. The bbackupquery + utility does this and more. + + You don't provide any login information to it, as it just picks + up the data it needs from /etc/box/bbackupd.conf. You should run it as + root so it can find everything it needs. + + Full documentation can be found in the bbackupquery manual page. It follows + the model of a command line sftp client quite closely. + + TODO: Link to bbackupquery man-page here. + + On systems where GNU readline is available (by default) it uses + that for command line history and editing. Otherwise it falls back to + very basic UNIX text entry. + + TODO: Did the readline dependency change to editline? + +
+ Using bbackupquery + + bbackupquery is the tool you use to verify, restore and + investigate your backup files with. When invoked, it simply logs + into the server using the certificates you have listed in + bbackupd.conf. + + After you run bbackupquery, you will see a prompt, allowing + you to execute commands. The list (or ls) command lets you view + files in the store. It works much like unix ls, but with different + options. An example: + + [pthomsen at host bbackupquery]$ bbackupquery +Box Backup Query Tool v0.10, (c) Ben Summers and contributors 2003-2006 +Using configuration file /etc/box/bbackupd.conf +Connecting to store... +Handshake with store... +Login to store... +Login complete. + +Type "help" for a list of commands. + +query > ls +00000002 -d---- mp3 +00000003 -d---- video +00000004 -d---- home-pthomsen +00000005 -d---- root +query > + + The ls commands shows the directories that are backed up. Now + we'll take a closer look at the home-pthomsen directory: + + query > cd home-pthomsen +query > ls +00002809 f----- sample.tiff +0000280a f----- s3.tiff +0000280b f----- s4.tiff +0000280d f----- s2.tiff +0000280e f----- foo.pdf +0000286c f----- core.28720 +0000339a -d---- .emacs.d +0000339d -d---- bbackup-contrib +00003437 f----- calnut.compare.txt +0000345d f----- DSCN1783.jpg +0000345e f----- DSCN1782.jpg +query > + + The ls command takes the following options; + + + + -r -- recursively list + all files + + + + -d -- list deleted + files/directories + + + + -o -- list old versions + of files/directories + + + + -I -- don't display + object ID + + + + -F -- don't display + flags + + + + -t -- show file + modification time (and attr mod time if has the object has + attributes, ~ separated) + + + + -s -- show file size in + blocks used on server (only very approximate indication of size + locally) + + + + The flags displayed from the ls command are as follows: + + + f = file + + d = directory + + X = deleted + + o = old version + + R = remove from server as soon as marked deleted or + old + + a = has attributes stored in directory record which + override attributes in backup file + +
+ +
+ Verify backups + + As with any backup system, you should frequently check that + your backups are working properly by comparing them. Box Backup + makes this very easy and completely automatic. All you have to do is + schedule the bbackupquery compare command to run + regularly, and check its output. You can run the command manually as + follows: + + /usr/local/sbin/bbackupquery "compare -a" quit + + This command will report all the differences found between the + store and the files on disc. It will download everything, so may + take a while. You should expect to see some differences on a typical + compare, because files which have recently changed are unlikely to + have been uploaded yet. It will also tell you how many files have + been modified since the last backup run, since these will normally + have changed, and such failures are expected. + + You are strongly recommended to add this command as a + cron job, at least once a month, and to check the + output for anything suspicious, particularly a large number of + compare failures, failures on files that have not been modified, or + any error (anything except a compare mismatch) that occurs during + the compare operation. + + Consider keeping a record of these messages and comparing them + with a future verification. + + If you would like to do a "quick" check which just downloads + file checksums and compares against that, then run: + + /usr/local/sbin/bbackupquery "compare -aq" quit + + However, this does not check that the file attributes are + correct, and since the checksums are generated on the client they + may not reflect the data on the server if there is a problem -- the + server cannot check the encrypted contents. View this as a quick + indication, rather than a definite check that your backup verifies + correctly. +
+ +
+ Restore backups + + You will need the keys file created when you configured the + server. Without it, you cannot restore the files; this is the + downside of encrypted backups. However, by keeping the small keys + file safe, you indirectly keep your entire backup safe. + + The first step is to recreate the configuration of the backup + client. It's probably best to have stored the /etc/box directory + with your keys. But if you're recreating it, all you really need is + to have got the login infomation correct (ie the certs and + keys). + + Don't run bbackupd yet! It will mark all your files as deleted + if you do, which is not hugely bad in terms of losing data, just a + major inconvenience. (This assumes that you are working from a blank + slate. If you want to restore some files to a different location, + it's fine to restore while bbackupd is running, just do it outside a + backed up directory to make sure it doesn't start uploading the + restored files.) + + Type: + + /usr/local/sbin/bbackupquery + + to run it in interactive mode. + + Type: + + list + + to see a list of the locations stored on the server. + + For each location you want to restore, type: + + restore name-on-server local-dir-name + + The directory specified by local-dir-name must not exist yet. + If the restore is interrupted for any reason, repeat the above + steps, but add the -r flag to the + restore command to tell it to resume. +
+ +
+ Retrieving deleted and old files + + Box Backup makes old versions of files and files you have + deleted available, subject to there being enough disc space on the + server to hold them. + + This is how to retrieve them using bbackupquery. Future + versions will make this far more user-friendly. + + Firstly, run bbackupquery in interactive mode. It behaves in a + similar manner to a command line sftp client. + + /usr/local/sbin/bbackupquery + + Then navigate to the directory containing the file you want, + using list, cd and pwd. + + query > cd home/profiles/USERNAME + + List the directory, using the "o" option to list the files + available without filtering out everything apart from the current + version. (if you want to see deleted files as well, use list + -odt) + + query > list -ot +00000078 f--o- 2004-01-21T20:17:48 NTUSER.DAT +00000079 f--o- 2004-01-21T20:17:48 ntuser.dat.LOG +0000007a f--o- 2004-01-21T17:55:12 ntuser.ini +0000007b f---- 2004-01-12T15:32:00 ntuser.pol +0000007c -d--- 1970-01-01T00:00:00 Templates +00000089 -d--- 1970-01-01T00:00:00 Start Menu +000000a0 -d--- 1970-01-01T00:00:00 SendTo +000000a6 -d--- 1970-01-01T00:00:00 Recent +00000151 -d--- 1970-01-01T00:00:00 PrintHood +00000152 -d--- 1970-01-01T00:00:00 NetHood +00000156 -d--- 1970-01-01T00:00:00 My Documents +0000018d -d--- 1970-01-01T00:00:00 Favorites +00000215 -d--- 1970-01-01T00:00:00 Desktop +00000219 -d--- 1970-01-01T00:00:00 Cookies +0000048b -d--- 1970-01-01T00:00:00 Application Data +000005da -d--- 1970-01-01T00:00:00 UserData +0000437e f--o- 2004-01-24T02:45:43 NTUSER.DAT +0000437f f--o- 2004-01-24T02:45:43 ntuser.dat.LOG +00004380 f--o- 2004-01-23T17:01:29 ntuser.ini +00004446 f--o- 2004-01-24T02:45:43 NTUSER.DAT +00004447 f--o- 2004-01-24T02:45:43 ntuser.dat.LOG +000045f4 f---- 2004-01-26T15:54:16 NTUSER.DAT +000045f5 f---- 2004-01-26T15:54:16 ntuser.dat.LOG +000045f6 f---- 2004-01-26T16:54:31 ntuser.ini + + (this is a listing from a server which is used as a Samba + server for a network of Windows clients.) You now need to fetch the + file using it's ID, rather than it's name. The ID is the hex number + in the first column. Fetch it like this: + + query > get -i 0000437e NTUSER.DAT +Object ID 0000437e fetched successfully. + + The object is now available on your local machine. You can use + lcd to move around, and sh ls to list directories on your local + machine. +
+
+
+ +
+ Fixing corruptions of store data + + This section gives help on what to do if your server has suffered + corruption, for example, after an unclean shutdown or other operating + system or hardware problem. + + In general, as updates to the store are made in an atomic manner, + the most likely result is wasted disc space. However, if really bad + things happen, or you believe that there is a lot of wasted space, then + these instructions will help to restore your data. + + You know you will need to do something if you get strange errors, + and bbackupd attempts to contact the server every 100 seconds or so. Or + if one of the discs in your RAID disc set has failed. + + After following these instructions, the end result will be that + bbackupquery will be able to see all the files which were stored on your + server, and retrieve them. Some of them may be in lost+found directories + in the root of the store (or in their original position if they have + been moved) but they will all be able to be retrieved. + + After you have retrieved the files you want, bbackupd will upload + new versions where necessary, and after about two days, mark any + lost+found directories as deleted. Finally, those directories will be + removed by the housekeeping process on the server. + + These instructions assume you're working on account 1234. Replace + this with the account number that you actually want to check (the one + that is experiencing errors). These steps will need to be repeated for + all affected accounts. + +
+ Stop bbackupd + + First, make sure that bbackupd is not running on the client + machine for the account you are going to recover. Use + bbackupctl terminate to stop it. This step is not + strictly necessary, but is recommended. During any checks on the + account, bbackupd will be unable to log in, and after they are + complete, the account is marked as changed on the server so bbackupd + will perform a complete scan. +
+ +
+ Are you using RAID on the server? + + The raidfile recovery tools have not been written, and probably + will not be, since Box Backup RAID is deprecated. However, when two + out of three files are available, the server will successfully allow + access to your data, even if it complains a lot in the logs. The best + thing to do is to fix the accounts, if necessary, and retrieve any + files you need. Then move the old store directories aside (in case you + need them) and start afresh with new accounts, and let the clients + upload all their data again. +
+ +
+ Check and fix the account + + First, run the check utility, and see what errors it + reports. + + /usr/local/sbin/bbstoreaccounts check 1234 + + This will take some time, and use a fair bit of memory (about 16 + bytes per file and directory). If the output looks plausible and + reports errors which need fixing, run it again but with the fix + flag: + + /usr/local/sbin/bbstoreaccounts check 1234 fix + + This will fix any errors, and remove unrecoverable files. + Directories will be recreated if necessary. + + NOTE: The utility may adjust + the soft and hard limits on the account to make sure that housekeeping + will not remove anything -- check these afterwards. +
+ +
+ Grab any files you need with bbackupquery + + At this point, you will have a working store. Every file which + was on the server, and wasn't corrupt, will be available. + + On the client, use bbackupquery to log in and examine the store. + (type help at the prompt for instructions). Retrieve any files you + need, paying attention to any lost+found directories in the root + directory of the store. + + You can skip this step if you are sure that the client machine + is fine -- in this case, bbackupd will bring the store up to + date. +
+ +
+ Restart bbackupd + + Restart bbackupd on the client machine. The store account will + be brought up to date, and files in the wrong place will be marked for + eventual deletion. +
+
+ +
+ Troubleshooting + + If you are trying to fix a store after your disc has been + corrupted, see Fixing corruptions of + store data. + + Unfortunately, the error messages are not particularly helpful at + the moment. This page lists some of the common errors, and the most + likely causes of them. + + When an error occurs, you will see a message like 'Exception: + RaidFile/OSFileError (2/8)' either on the screen or in your log files. + (it is recommended you set up another log file as recommended in the + server setup instructions.) + + This error may not be particularly helpful, although some do have + extra information about probable causes. To get further information, + check the ExceptionCodes.txt file in the root of the distribution. This + file is generated by the ./configure script, so you will need to have + run that first. + + Some common causes of exceptions are listed below. + + Please email me with any other codes you get, and I will let you + know what they mean, and add notes here. + +
+ RaidFile (2/8) + + This is found either when running bbstoreaccounts or in the + bbstored logs. + + Problem: The directories you + specified in the raidfile.conf are not writable by the _bbstored + user. + + Resolution: Change permissions + appropriately. +
+ +
+ Common (1/2) + + This usually occurs when the configuration files can't be + opened. + + Problem: You created your + configurations in non-standard locations, and the programs cannot find + them. + + Resolution: Explicitly specify + configuration file locations to daemons and programs. For + example + + /usr/local/sbin/bbstored /some/other/dir/bbstored.config /usr/local/sbin/bbackupquery -c /some/other/dir/bbackupd.config + + (daemons specify the name as the first argument, utility + programs with the -c option). + + Problem: bbstored can't find + the raidfile.conf file specified in bbstored.conf. + + Resolution: Edit bbstored.conf + to point to the correct location of this additional configuration + file. +
+ +
+ Server (3/16) + + The server can't listen for connections on the IP address + specified when you configured it. + + Problem: This probably means + you've specified the wrong hostname to bbstored-config -- maybe your + server is behind a NAT firewall? + + Resolution: Edit bbstored.conf + and correct the ListenAddresses line. You should replace the server + address with the IP address of your machine. +
+ +
+ Connection (7/x) + + These errors all relate to connections failing -- you may see + them during operation if there are network failures or other problems + between the client and server. The backup system will recover from + them automatically. + +
+ Connection (7/30) - SSL problems + + Log snippet from client side: + + bbackupd[1904]: Opening connection to server xxxx.xxx... +bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 +bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed +bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:asn1 encoding routines:ASN1_verify:EVP lib +bbackupd[1904]: SSL err during Connect: error:xxxxxxxx:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed +bbackupd[1904]: TRACE: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(237) +bbackupd[1904]: Exception caught (7/30), reset state and waiting to retry... + + And from the server: + + bbstored[19291]: Incoming connection from xx.xxx.xx.xxx port xxxxx (handling in child xxxxx) +bbstored[21588]: SSL err during Accept: error:xxxxxxxx:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error +bbstored[21588]: in server child, exception Connection TLSHandshakeFailed (7/30) -- terminating child + + Solution: Create a new CA on + the server side and re-generate the client certificate. Re-creating + the client certificate request is not necessary. +
+
+ +
+ Advanced troubleshooting + + If this really doesn't help, then using the DEBUG builds of the + system will give you much more information -- a more descriptive + exception message and the file and line number where the error + occurred. + + For example, if you are having problems with bbstoreaccounts, + build the debug version with: + + cd boxbackup-0.0 +cd bin/bbstoreaccounts +make + + Within the module directories, make defaults to building the + debug version. At the top level, it defaults to release. + + This will build an executable in debug/bin/bbstoreaccounts which + you can then use instead of the release version. It will give far more + useful error messages. + + When you get an error message, use the file and line number to + locate where the error occurs in the code. There will be comments + around that line to explain why the exception happened. + + If you are using a debug version of a daemon, these extended + messages are found in the log files. +
+
+
+ + &__ExceptionCodes__elfjz3fu; + + + Running without root + + It is possible to run both the server and client without root + privileges. + +
+ Server + + The server, by default, runs as a non-root user. However, it + expects to be run as root and changes user to a specified user as soon + as it can, simply for administrative convenience. The server uses a port + greater than 1024, so it doesn't need root to start. + + To run it entirely as a non-root user, edit the bbstored.conf + file, and remove the User directive from the Server section. Then simply + run the server as your desired user. +
+ +
+ Client + + The client requires root for normal operation, since it must be + able to access all files to back them up. However, it is possible to run + the client as a non-root user, with certain limitations. + + Follow the installation instructions, but install the executable + files manually to somewhere in your home directory. Then use + bbackupd-config to configure the daemon, but use a directory other than + /etc/box, probably somewhere in your home directory. + + All directories you specify to be backed up must be readable, and + all files must be owned by the user and readable to that user. + + Important: If any file or directory is not readable by this user, + the backup process will skip that file or directory. Keep an eye on the + logs for reports of this failure. + + Non-root operation of the backup client is recommended only for + testing, and should not be relied on in a production environment. +
+
+
Copied: box/trunk/docs/docbook/bb-book.xsl (from rev 2463, box/trunk/docs/bb-book.xsl) =================================================================== --- box/trunk/docs/docbook/bb-book.xsl (rev 0) +++ box/trunk/docs/docbook/bb-book.xsl 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,17 @@ + + + + + + + + + + + + + Copied: box/trunk/docs/docbook/bb-man.xsl.tmpl (from rev 2463, box/trunk/docs/bb-man.xsl.tmpl) =================================================================== --- box/trunk/docs/docbook/bb-man.xsl.tmpl (rev 0) +++ box/trunk/docs/docbook/bb-man.xsl.tmpl 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,9 @@ + + + + + + + + Copied: box/trunk/docs/docbook/bb-nochunk-book.xsl (from rev 2463, box/trunk/docs/bb-nochunk-book.xsl) =================================================================== --- box/trunk/docs/docbook/bb-nochunk-book.xsl (rev 0) +++ box/trunk/docs/docbook/bb-nochunk-book.xsl 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,17 @@ + + + + + + + + + + + + + Copied: box/trunk/docs/docbook/bbackupctl.xml (from rev 2463, box/trunk/docs/bbackupctl.xml) =================================================================== --- box/trunk/docs/docbook/bbackupctl.xml (rev 0) +++ box/trunk/docs/docbook/bbackupctl.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,205 @@ + + + + bbackupctl + + 8 + + Box Backup + + Box Backup + + 0.11 + + + + bbackupctl + + Control the Box Backup client daemon + + + + + bbackupctl + + -q + + -c config-file + + command + + + + + Description + + bbackupctl sends commands to a running + bbackupd daemon on a client machine. It can be used to + force an immediate backup, tell the daemon to reload its configuration + files or stop the daemon. If bbackupd is configured in + snapshot mode, it will not back up automatically, and the + bbackupctl must be used to tell it when to start a + backup. + + Communication with the bbackupd daemon takes place over a local + socket (not over the network). Some platforms (notably Windows) can't + determine if the user connecting on this socket has the correct + credentials to execute the commands. On these platforms, ANY local user + can interfere with bbackupd. To avoid this, remove the CommandSocket + option from bbackupd.conf, which will also disable bbackupctl. See the + Client Configuration page for more information. + + bbackupctl needs to read the + bbackupd configuration file to find out the name of the + CommandSocket. If you have to tell bbackupd where to + find the configuration file, you will have to tell + bbackupctl as well. The default on Unix systems is + usually /etc/box/bbackupd.conf. On Windows systems, + it is bbackupd.conf in the same directory where + bbackupd.exe is located. If + bbackupctl cannot find or read the configuration file, + it will log an error message and exit. + + bbackupctl usually writes error messages to the + console and the system logs. If it is not doing what you expect, please + check these outputs first of all. + + + + + + + Run in quiet mode. + + + + + config-file + + + Specify configuration file. + + + + + + Commands + + The following commands are available in bbackupctl: + + + + terminate + + + This command cleanly shuts down bbackupd. + This is better than killing or terminating it any other + way. + + + + + reload + + + Causes the bbackupd daemon to re-read all + its configuration files. Equivalent to kill + -HUP. + + + + + sync + + + Initiates a backup. If no files need to be backed up, no + connection will be made to the server. + + + + + force-sync + + + Initiates a backup, even if the + SyncAllowScript says that no backup should run + now. + + + + + wait-for-sync + + + Passively waits until the next backup starts of its own + accord, and then terminates. + + + + + wait-for-end + + + Passively waits until the next backup starts of its own + accord and finishes, and then terminates. + + + + + sync-and-wait + + + Initiates a backup, waits for it to finish, and then + terminates. + + + + + + + + Files + + /etc/box/bbackupd.conf + + + + See Also + + + bbackupd.conf + + 5 + , + bbackupd-config + + 8 + , + bbackupctl + + 8 + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Copied: box/trunk/docs/docbook/bbackupd-config.xml (from rev 2463, box/trunk/docs/bbackupd-config.xml) =================================================================== --- box/trunk/docs/docbook/bbackupd-config.xml (rev 0) +++ box/trunk/docs/docbook/bbackupd-config.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,153 @@ + + + + bbackupd-config + + 8 + + Box Backup + + Box Backup + + 0.11 + + + + bbackupd-config + + Box Backup client daemon configuration file + generator + + + + + bbackupd-config + + config-dir + + backup-mode + + account-num + + server-hostname + + working-dir + + backup-dir + + backup-dir ... + + + + + Description + + The bbackupd-config script creates configuration files and client + certificates. It takes at least six parameters: + + + + config-dir + + + Configuration directory. Usually + /etc/box. + + + + + backup-mode + + + Either lazy or snapshot. + + + + + account-num + + + The client account number. This is set by the bbstored + administrator. + + + + + server-hostname + + + The hostname or IP address of the bbstored server. + + + + + working-dir + + + A directory to keep temporary state files. This is usually + something like /var/bbackupd. This can be + changed in bbackupd.conf later on if + required. + + + + + backup-dir + + + A space-separated list of directories to be backed up. Note + that this does not traverse mount + points. + + + + + + + Files + + /etc/box/bbackupd.conf + + /etc/box/bbackupd/NotifySysAdmin.sh + + + + See Also + + + bbackupd.conf + + 5 + , + bbackupd + + 8 + , + bbackupctl + + 8 + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Copied: box/trunk/docs/docbook/bbackupd.conf.xml (from rev 2463, box/trunk/docs/bbackupd.conf.xml) =================================================================== --- box/trunk/docs/docbook/bbackupd.conf.xml (rev 0) +++ box/trunk/docs/docbook/bbackupd.conf.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,479 @@ + + + + bbackupd.conf + + 5 + + Box Backup + + Box Backup + + 0.11 + + + + bbackupd.conf + + Box Backup client daemon configuration file + + + + + /etc/box/bbackupd.conf + + + + + Description + + + + AccountNumber + + + The account number of this client. This is set by the admin of + the store server. + + + + + UpdateStoreInterval + + + Specifies the interval between scanning of the local discs. To + avoid cycles of load on the server, this time is randomly adjusted + by a small percentage as the daemon runs. Defaults to 1 hour. + + + + + MinimumFileAge + + + Specifies how long since a file was last modified before it + will be uploaded. Defaults to 6 hours. + + + + + MaxUploadWait + + + If a file is repeatedly modified it won't be uploaded + immediately in case it's modified again. However it should be + uploaded eventually. This is how long we should wait after first + noticing a change. Defaults to 1 day. + + + + + MaxFileTimeInFuture + + + + + + + + AutomaticBackup + + + + + + + + SyncAllowScript + + + Use this to temporarily stop bbackupd from syncronising or + connecting to the store. This specifies a program or script script + which is run just before each sync, and ideally the full path to the + interpreter. It will be run as the same user bbackupd is running as, + usually root. + + The script prints either "now" or a number to STDOUT (and a + terminating newline, no quotes). If the result was "now", then the + sync will happen. If it's a number, then the script will be asked + again in that number of seconds. + + For example, you could use this on a laptop to only backup + when on a specific network. + + + + + MaximumDiffingTime + + + How much time should be spent on diffing files. + + + + + DeleteRedundantLocationsAfter + + + + + + + + FileTrackingSizeThreshold + + + + + + + + DiffingUploadSizeThreshold + + + + + + + + StoreHostname + + + The hostname or IP address of the + bbstored + + 8 + server. + + + + + StorePort + + + The port used by the server. Defaults to 2201. + + + + + ExtendedLogging + + + Logs everything that happens between the client and server. + The + bbackupd + + 8 + client must also be started with + . + + + + + ExtendedLogFile + + + + + + + + LogAllFileAccess + + + + + + + + LogFile + + + + + + + + LogFileLevel + + + + + + + + CommandSocket + + + Where the command socket is created in the filesystem. + + + + + KeepAliveTime + + + + + + + + StoreObjectInfoFile + + + + + + + + NotifyScript + + + The location of the script which runs at certain events. This + script is generated by + bbackupd-config + + 8 + . Defaults to + /etc/box/bbackupd/NotifySysAdmin.sh. + + + + + NotifyAlways + + + + + + + + CertificateFile + + + The path to the client's public certificate. + + + + + PrivateKeyFile + + + The path to the client's private key. This should only be + readable by root. + + + + + TrustedCAsFile + + + The Certificate Authority created by + bbstored-certs + + 8 + . + + + + + KeysFile + + + The data encryption key. This must be kept safe at all costs, your data is + useless without it! + + + + + DataDirectory + + + A directory to keep temporary state files. This is usually + something like /var/bbackupd. + + + + + Server + + + This section relates to the running daemon. + + + + PidFile + + + The location of the process ID file. Defaults to + /var/run/bbackupd.pid. + + + + + + + + BackupLocations + + + This section defines each directory to be backed up. Each + entry must have at least a Path entry and, optionally, include and + exclude directives. + + Multiple include and exclude directives may appear. + + + + Path + + + The path to back up. + + + + + ExcludeFile + + + Exclude a single file. + + + + + ExcludeFilesRegex + + + Exclude multiple files based on a regular expression. + See + re_format + + 7 + . + + + + + ExcludeDir + + + Exclude a single directory. + + + + + ExcludeDirsRegex + + + Exclude multiple directories based on a regular + expression. See + re_format + + 7 + . + + + + + AlwaysIncludeFile + + + Include a single file from a directory which has been + excluded. + + + + + AlwaysIncludeFilesRegex + + + Include multiple files from an excluded directory, + based on a regular expression. + + + + + AlwaysIncludeDir + + + Include a single directory from a directory which has + been excluded. + + + + + AlwaysIncludeDirsRegex + + + Include multiple directories from an excluded + directory, based on a regular expression. + + + + + + + + + + Examples + + The following is an example of a backup location: + + home +{ + Path = /home + ExcludeDir = /home/guest + ExcludeDir = /home/[^/]+/tmp + ExcludeFilesRegex = .*\.(mp3|MP3)$ + AlwaysIncludeFile = /home/someuser/importantspeech.mp3 +} + + + + Files + + /etc/box/bbackupd.conf + + + + See Also + + + bbackupd + + 8 + , + bbackupd-config + + 8 + , + bbackupctl + + 8 + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Copied: box/trunk/docs/docbook/bbackupd.xml (from rev 2463, box/trunk/docs/bbackupd.xml) =================================================================== --- box/trunk/docs/docbook/bbackupd.xml (rev 0) +++ box/trunk/docs/docbook/bbackupd.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,209 @@ + + + + bbackupd + + 8 + + Box Backup + + Box Backup + + 0.11 + + + + bbackupd + + Box Backup client daemon + + + + + bbackupd + + -DFkqvVT + + -c config-file + + -t tag + + + + + Description + + bbackupd runs on client computers in the + background, finding new files to back up. When it is time for a backup, + bbackupd will connect to the server + (bbstored) to upload the files. + + A running bbackupd daemon can be controlled with + the bbackupctl command, to make it shut down, reload + its configuration, or start an immediate backup. + + bbackupd needs to be configured to tell it which + files to back up, how often, and to which server (running + bbstored). See the Client Configuration page for more + information. For this, you must write a configuration file. You must + either place it in the default location, or tell + bbackupd where to find it. + + You can check the default location with the + option. The default on Unix systems is usually + /etc/box/bbackupd.conf. On Windows systems, it is + bbackupd.conf in the same directory where + bbackupd.exe is located. If bbackupd cannot find or + read the configuration file, it will log an error message and exit. + + bbackupd usually writes log messages to the + system logs, using the facility local5, which you can + use to filter them to send them to a separate file. It can also write them + to the console, see options below. If bbackupd is not + doing what you expect, please check the logs first of all. + + + Options + + + + config-file + + + Use the specified configuration file. If + is omitted, the last argument is the configuration file. If none + is specified, the default is used (see above). + + + + + + + + Debugging mode. Do not fork into the background (do not run + as a daemon). Not available on Windows. + + + + + + + + No-fork mode. Same as for bbackupd. Not + available on Windows. + + + + + + + + Keep console open after fork, keep writing log messages to + it. Not available on Windows. + + + + + + + + Run more quietly. Reduce verbosity level by one. Available + levels are NOTHING, FATAL, + ERROR, WARNING, + NOTICE, INFO, + TRACE, EVERYTHING. Default + level is NOTICE in non-debugging builds. Use + once to drop to WARNING level, twice for + ERROR level, four times for no logging at + all. + + + + + -v + + + Run more verbosely. Increase verbosity level by one. Use + once to raise to INFO level, twice for + TRACE level, three times for + EVERYTHING (currently the same as + TRACE). + + + + + + + + Run at maximum verbosity (EVERYTHING + level). + + + + + tag + + + Tag each console message with specified marker. Mainly + useful in testing when running multiple daemons on the same + console. + + + + + + + + Timestamp each line of console output. + + + + + + + + Files + + /etc/box/bbackupd.conf + + + + See Also + + + bbackupd.conf + + 5 + , + bbackupd-config + + 8 + , + bbackupctl + + 8 + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Copied: box/trunk/docs/docbook/bbackupquery.xml (from rev 2463, box/trunk/docs/bbackupquery.xml) =================================================================== --- box/trunk/docs/docbook/bbackupquery.xml (rev 0) +++ box/trunk/docs/docbook/bbackupquery.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,506 @@ + + + + bbackupquery + + 8 + + Box Backup + + Box Backup + + 0.11 + + + + bbackupquery + + Box Backup store query and file retrieval + + + + + bbackupquery + + -q + + -c configfile + + command ... + + + + + Description + + bbackupquery is the main way of interacting with + the backup store from a Box Backup client machine. It supports both + interactive and batch modes of operation. + + It can be used to reviewing the status of a client machine's backup + store, getting status from the store server. The main use is to retrieve + files and directories when needed. + + bbackupquery supports interactive and batch modes + of operation. Interactive mode allows for interaction with the server much + like an interactive FTP client. + + Batch mode is invoked by putting commands into the invocation of + bbackupquery. Example: + + bbackupquery "list home-dirs" quit + + Note that commands that contain spaces are enclosed in double + quotes. If the quit command is omitted, after the + preceding commands are completed, bbackupquery will + enter interactive mode. + + + + Options + + + + + + + Quiet. Suppresses status output while running. + + + + + + + + Use configfile instead of the default bbackupd.conf file. + Can be a relative or full path. + + + + + + + Commands + + The commands that can be used in bbackupquery are listed + below. + + + + help + + + Displays the basic help message, which gives information about + the commands available in bbackupquery. Use the + form help command to get help on a specific + command. + + + + + quit + + + End the session with the store server, and quit + bbackupquery. + + + + + cd options + directory-name + + + Change directory. Options: + + + + + consider deleted directories for traversal + + + + + + + + consider old versions of directories for traversal. + This option should never be useful in a correctly formed + store. + + + + + + + + lcd + local-directory-name + + + Change directory on the client machine. To list the contents + of the local directory, type sh ls (on Unix-like + machines). + + + + + list options + directory-name + + + The list (or its synonym ls) command lists + the content of the current, or specified, directory. The options are + as follows: + + + + + + + recursively list all files + + + + + + + + list deleted files and directories + + + + + + + + list old versions of files and directories + + + + + + + + don't display object IDs + + + + + + + + don't display flags + + + + + + + + show file modification time (and attr mod time, if the + object has attributes). + + + + + + + + show file size in blocks used on server. Note that + this is only a very approximate indication of local file + size. + + + + + + + + ls options + directory-name + + + Synonym for list. + + + + + pwd + + + Print current directory, always relative to the backup store + root. + + + + + sh shell-command + + + Everything after the sh is passed to a shell and run. All + output from the command is displayed in the client. + + Example: to list the contents of the current directory on the + client machine type sh ls. + + + + + compare -a + + + + + + + + compare -l + location-name + + + + + + + + compare store-dir-name + local-dir-name + + + Compare the current data in the store with the data on the + disc. Please note that all the data will be downloaded from the + store, so this can be a very lengthy process depending on the size + of the store, and the size of the part you are comparing. + + Options: + + + + + + + compare all locations. + + + + + + + + compare one backup location as specified in the + configuration file. This compares one of the top level store + directories. + + + + + + + + set return code. The return code is set to the + following values, if quit is the next command. So, if + another command is run after the compare, the return code + will not refer to the compare. This option is very useful + for automating compares. Return code values: + + -- no differences were + found + + + + -- differences were + found + + + + -- an error occured + + + + + + + + + + get object-filename + local-filename + + + + + + + + get -i object-id + local-filename + + + Gets a file from the store. Object is specified as the + filename within the current directory. Local filename is optional. + Ignores old and deleted files when searching the directory for the + file to retrieve. + + To get an old or deleted file, use the + option and select the object as a hex object ID (first column in + listing). The local filename must be specified. + + + + + getobject object-id + local-filename + + + Gets the object specified by the object id (in hex) and stores + the raw contents in the local file specified. Note: This is only + useful for debugging as it does not decode files from the stored + format, which is encrypted and compressed. + + + + + restore -d + directory-name + local-directory-name + + + + + + + + restore -r + + + Restores a directory to the local disc. The local directory + specified must not exist (unless a previous restore is being + restarted). The root cannot be restored -- restore locations + individually. + + Options: + + + + + + + restore a deleted directory + + + + + + + + resume an interrupted restore + + + If a restore operation is interrupted for any + reason, it can be restarted using the switch. + Restore progress information is saved in a file at regular intervals + during the restore operation to allow restarts. + + + + + usage -m + + + Show space used on the server for this account. Display + fields: + + Used: Total amount of space used on + the server + + + + Old files: Space used by old + files + + + + Deleted files: Space used by + deleted files + + + + Directories: Space used by the + directory structure + + + + When Used exceeds the soft limit, the + server will start to remove old and deleted files until the usage + drops below the soft limit. After a while, you should expect to see + the usage stay at just below the soft limit. You only need more + space if the space used by old and deleted files is near + zero. + + The option displays output in + machine-readable form. + + + + + + + Bugs + + If you find a bug in Box Backup and you want to let us know about + it, join the mailing + list and send us a description of the problem there. + + To report a bug, give us at least the following information: + + + + The version of Box Backup you are running + + + + The platform you are running on (hardware and OS), for both + client and server. + + + + If possible attach your config files (bbstored.conf, + bbackupd.conf) to the bug report. + + + + Also attach any log file output that helps shed light on the + problem you are seeing. + + + + And last but certainly not least, a description of what you are + seeing, in as much detail as possible. + + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Copied: box/trunk/docs/docbook/bbstoreaccounts.xml (from rev 2463, box/trunk/docs/bbstoreaccounts.xml) =================================================================== --- box/trunk/docs/docbook/bbstoreaccounts.xml (rev 0) +++ box/trunk/docs/docbook/bbstoreaccounts.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,386 @@ + + + + bbstoreaccounts + + 8 + + Box Backup + + Box Backup + + 0.11 + + + + bbstoreaccounts + + Box Backup store accounts manager + + + + + bbstoreaccounts + + -c config-file + + command + + account-id + + command-specific arguments + + + + + Description + + bbstoreaccounts is the tool for managing accounts + on the store server. It can be used to view information related to + accounts, as well as create, change and delete accounts on the store + server. + + bbstoreaccounts always takes at least 2 + parameters: the command name and the account ID. Some commands require + additional parameters, and some commands have optional parameters. + + + Options + + + + + + + The configfile to use for connecting to the store. Default + is /etc/box/bbstored.conf. + + + + + + + Commands + + The commands tells bbstoreaccounts what action to perform. + + + + check account-id + fix + + + The check command verifies the + integrity of the store account given, and optionally fixes any + corruptions. Note: It is + recommended to run the 'simple' check command (without + fix) before using the fix + option. This gives an overview of the extent of any problems, + before attempting to fix them. + + + + + create account-id + disc-set soft-limit + hard-limit + + + Creates a new store account with the parameters given. The + parameters are as follows: + + + + + + + The ID of the new account to be created. A 32-bit + hexadecimal number. Cannot already exist on the + server. + + + + + + + + The disc set from + raidfile.conf + + 5 + where the backups for this client will + be stored. A number. Each RAID-file set has a number in + raidfile.conf. This number is what's used. + + + + + + + + The soft limit is the amount of storage that the + server will guarantee to be available for + storage. + + + + + + + + The amount of storage that the the server will + allow, before rejecting uploads, and starting to + eliminate old and deleted files to get back down to + soft-limit. + + + + + + + + delete account-id + yes + + + Deletes the account from the store server completely. + Removes all backups and deletes all references to the account in + the config files. + + delete will ask for confirmation from + the user, when called. Using the flag, + eliminates that need. This is useful when deleting accounts from + within a script or some other automated means. 0 + + + + + info account-id + + + Display information about the given account. + Example:[root]# bbstoreaccounts info 1 + Account ID: 00000001 + Last object ID: 58757 + Blocks used: 9864063 (38531.50Mb) + Blocks used by old files: 62058 (242.41Mb) +Blocks used by deleted files: 34025 (132.91Mb) + Blocks used by directories: 6679 (26.09Mb) + Block soft limit: 11796480 (46080.00Mb) + Block hard limit: 13107200 (51200.00Mb) + Client store marker: 1139559852000000 + + Explanation: + + + + Account ID + + + The account ID being displayed. + + + + + Last Object ID + + + A counter that keeps track of the objects that + have been backed up. This number refers to the last file + that was written to the store. The ID is displayed as a + decimal number, and the object ID can be converted to a + path name to a file as follows: convert the number to + hex (e.g.: 58757 => 0xE585); The last backed up file + will be (relative from the client's store root): + e5/o85.rfw. Longer numbers infer + more directories in the structure, so as an example + 3952697264 as the last object ID gives 0xEB995FB0, which + translates to a backup pathname of + eb/99/5f/ob0.rfw. + + + + + Blocks used + + + The number of blocks used by the store. The size + in Mb depends on the number of blocks, as well as the + block size for the disc set given in + raidfile.conf + + 5 + . In this case the block size is + 4096. + + + + + Blocks used by old files + + + The number of blocks occupied by files that have + newer versions in the store. This data is at risk for + being removed during housekeeping. + + + + + Blocks used by deleted files + + + The number of blocks used by files that have been + deleted on the client. This data is at risk for being + removed during housekeeping. + + + + + Blocks used by directories + + + The number of blocks used by directories in the + store. + + + + + Block soft limit + + + The soft limit in blocks. The soft limit is the + maximum guaranteed storage space available to the + account. When housekeeping starts, and the old and + deleted files are removed, they are removed in + chronological order (oldest first), until the data used + is less than the soft limit. + + + + + Block hard limit + + + The hard limit in blocks. The hard limit is the + most amount of storage the server will allow in an + account. Any data above this amount will be rejected. + Housekeeping will reduce the storage use, so more data + can be uploaded. + + + + + Client store marker + + + + bbstored + + 8 + uses this number to determine if it + needs to rescan the entire store. If this number is + different from the last time it checked, a rescan will + take place. + + + + + + + + setlimit account-id + soft-limit hard-limit + + + Changes the storage space allocation for the given + account. No server restart is needed. + + Parameters: + + + + + + + The ID of the account to be modified. + + + + + + + + The soft limit is the amount of storage that the + server will guarantee to be available for + storage. + + + + + + + + The amount of storage that the the server will + allow before rejecting uploads and starting to eliminate + old and deleted files to get back down to + . + + + + + + + + + + + Examples + + Create an account with ID 3af on disc set 0, with a 20GB soft-limit + and a 22GB hard-limit:bbstoreaccounts create 3af 0 20G 22GAlter + existing account ID 20 to have a 50GB soft-limit and a 55GB + hard-limit:bbstoreaccounts setlimit 20 50G 55G + + + + Files + + /etc/box/bbstored/accounts.txt + + + + See Also + + + bbstored + + 8 + , + bbstored-config + + 8 + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Copied: box/trunk/docs/docbook/bbstored-certs.xml (from rev 2463, box/trunk/docs/bbstored-certs.xml) =================================================================== --- box/trunk/docs/docbook/bbstored-certs.xml (rev 0) +++ box/trunk/docs/docbook/bbstored-certs.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,180 @@ + + + + bbstored-certs + + 8 + + Box Backup + + Box Backup + + 0.11 + + + + bbstored-certs + + Manage certificates for the Box Backup system + + + + + bbstored-certs + + certs-dir + + command + + arguments + + + + + Description + + bbstored-certs creates and signs certificates for + use in Box Backup. It allows the user to create and sign the server keys, + as well as signing client keys. + + All commands must be followed by the certs-dir, + which is the directory in which the certificates are stored. + + + Commands + + There are 3 commands: + + + + init + + + Create the certs-dir, and generate the + server keys for bbstored. certs-dir cannot + exist before running the command. + + + + + sign-server + servercsrfile + + + Sign the server certificate. The + servercsrfile is the file generated by the + init command. + + + + + sign + clientcsrfile + + + Sign a client certificate. The + clientcsrfile is generated during client setup. + See + bbackupd-config + + 8 + . Send the signed certificate back to the client, + and install according to the instructions given by + bbackupd-config. + + + + + + + + Files + + + raidfile-config + + 8 + generates the + raidfile.conf + + 5 + file. + + + + Bugs + + If you find a bug in Box Backup, and you want to let us know about + it, join the mailing + list, and send a description of the problem there. + + To report a bug, give us at least the following information: + + + + The version of Box Backup you are running + + + + The platform you are running on (hardware and OS), for both + client and server. + + + + If possible attach your config files (bbstored.conf, + bbackupd.conf) to the bug report. + + + + Also attach any log file output that helps shed light on the + problem you are seeing. + + + + And last but certainly not least, a description of what you are + seeing, in as much detail as possible. + + + + + + See Also + + + bbstored-config + + 8 + , + bbstored.conf + + 5 + , + bbstoreaccounts + + 8 + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Copied: box/trunk/docs/docbook/bbstored-config.xml (from rev 2463, box/trunk/docs/bbstored-config.xml) =================================================================== --- box/trunk/docs/docbook/bbstored-config.xml (rev 0) +++ box/trunk/docs/docbook/bbstored-config.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,148 @@ + + + + bbstored-config + + 8 + + Box Backup + + Box Backup + + 0.11 + + + + bbstored-config + + Box Backup store daemon configuration file + generator + + + + + bbstored-config + + configdir + + servername + + username + + + + + Description + + The bbstored-config script creates configuration files and server + certificates for a bbstored instance. It takes three parameters: + + + + configdir + + + The directory where config files will reside. A + bbstored subdirectory will be created where + several config files will reside. The + bbstored.conf file will be created in + configdir. + + + + + servername + + + The name of the server that is being configured. Usually the + fully qualified domain name of the machine in question. + + + + + username + + + The name of the user that should be running the + bbstored process. Recommended name: + _bbstored. + + + + + A valid + raidfile.conf + + 5 + must be found in configdir. Several steps are taken + during the run of bbstored-config: + + + + Server certificates are created. This requires interaction from + the operator. + + + + The RAID volumes are checked to ensure that the configuration is + consistent and will work. + + + + Instructions for next steps to take are shown. These steps may + be different for different OS platforms, so pay close attention to + these instructions. + + + + + + Files + + /etc/box/bbstored.conf + + + + See Also + + + bbstored.conf + + 5 + , + bbstored + + 8 + , + bbstored-certs + + 8 + , + raidfile-config + + 8 + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Copied: box/trunk/docs/docbook/bbstored.conf.xml (from rev 2463, box/trunk/docs/bbstored.conf.xml) =================================================================== --- box/trunk/docs/docbook/bbstored.conf.xml (rev 0) +++ box/trunk/docs/docbook/bbstored.conf.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,211 @@ + + + + bbstored.conf + + 5 + + Box Backup + + Box Backup + + 0.11 + + + + bbstored.conf + + Box Backup store daemon configuration file + + + + + /etc/box/bbstored.conf + + + + + Description + + The following configuration options are valid: + + + + RaidFileConf + + + Specifies the path to the + raidfile.conf + + 5 + . This is normally + /etc/box/raidfile.conf. + + + + + AccountDatabase + + + Specifies the path to the account database created by + + bbstoreaccounts + + 8 + . This is usually + /etc/box/bbstored/accounts.txt. + + + + + ExtendedLogging + + + Specifies whether extended logging should be enabled to show + what commands are being received from clients. + + + + + TimeBetweenHousekeeping + + + How long between scanning for files which need + deleting. + + + + + Server + + + These options relate to the actual daemon. + + PidFile + + + The location of the pidfile, where the daemon's + process ID is kept. + + + + + User + + + The user to run as. + + + + + ListenAddresses + + + The interface addresses to listen on. Hostnames may be + used instead of IP addresses. The format is: + or + . + + + + + CertificateFile + + + The path to the server's public certificate. + + + + + PrivateKeyFile + + + The path to the server's private key. This should only + be readable by root and/or the . + + + + + TrustedCAsFile + + + The Certificate Authority created by + bbstored-certs + + 8 + . + + + + + + + + + + Examples + + The following is an example bbstored.conf: + + RaidFileConf = /etc/box/raidfile.conf +AccountDatabase = /etc/box/bbstored/accounts.txt + +TimeBetweenHousekeeping = 900 + +Server +{ + PidFile = /var/run/bbstored.pid + User = _bbstored + ListenAddresses = inet:server.example.com + CertificateFile = /etc/box/bbstored/server.example.com-cert.pem + PrivateKeyFile = /etc/box/bbstored/server.example.com-key.pem + TrustedCAsFile = /etc/box/bbstored/clientCA.pem +} + + + + Files + + /etc/box/bbstored.conf + + + + See Also + + + bbstored + + 8 + , + bbstored-config + + 8 + , + raidfile-config + + 8 + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Copied: box/trunk/docs/docbook/bbstored.xml (from rev 2463, box/trunk/docs/bbstored.xml) =================================================================== --- box/trunk/docs/docbook/bbstored.xml (rev 0) +++ box/trunk/docs/docbook/bbstored.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,90 @@ + + + + bbstored + + 8 + + Box Backup + + Box Backup + + 0.11 + + + + bbstored + + Box Backup store daemon + + + + + bbstored + + config-file + + + + + Description + + bbstored runs on a central server. Clients + running bbackupd connect to the server and upload + files. + + The only argument is optional and specifies a non-default + configuration file. By default it will look for the configuration file as + /etc/box/bbstored.conf. + + + + Files + + /etc/box/bbstored.conf + + + + See Also + + + bbstored.conf + + 5 + , + bbstored-config + + 8 + , + raidfile-config + + 8 + , + raidfile.conf + + 5 + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Copied: box/trunk/docs/docbook/html/favicon.ico (from rev 2463, box/trunk/docs/favicon.ico) =================================================================== (Binary files differ) Copied: box/trunk/docs/docbook/instguide.xml (from rev 2463, box/trunk/docs/instguide.xml) =================================================================== --- box/trunk/docs/docbook/instguide.xml (rev 0) +++ box/trunk/docs/docbook/instguide.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,766 @@ + + + + Box Backup Build and Installation Guide + + + License + + Copyright © 2003 - 2007, Ben Summers and contributors. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + + + + Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + + + Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following disclaimer + in the documentation and/or other materials provided with the + distribution. + + + + All use of this software and associated advertising materials + must display the following acknowledgement: This product includes + software developed by Ben Summers. + + + + The names of the Authors may not be used to endorse or promote + products derived from this software without specific prior written + permission. + + + + + [Where legally impermissible the Authors do not disclaim liability + for direct physical injury or death caused solely by defects in the + software unless it is modified by a third party.] + + THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS + OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR + ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING + IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + POSSIBILITY OF SUCH DAMAGE. + + + + Introduction + + The backup daemon, bbackupd, runs on all machines to be backed up. + The store server daemon, bbstored runs on a central server. Data is sent + to the store server, which stores all data on local filesystems, that is, + only on local hard drives. Tape or other archive media is not used. + + The system is designed to be easy to set up and run, and cheap to + use. Once set up, there should be no need for user or administrative + intervention, apart from usual system maintenance. + +
+ Client daemon + + bbackupd is configured with a list of directories to back up. It + has a lazy approach to backing up data. Every so often, the directories + are scanned, and new data is uploaded to the server. This new data must + be over a set age before it is uploaded. This prevents rapid revisions + of a file resulting in many uploads of the same file in a short period + of time. + + It can also operate in a snapshot mode, which behaves like + traditional backup software. When instructed by an external bbackupctl + program, it will upload all changed files to the server. + + The daemon is always running, although sleeping most of the time. + In lazy mode, it is completely self contained -- scripts running under + cron jobs are not used. The objective is to keep files backed up, not to + make snapshots of the filesystem at particular points in time + available. + + If an old version of the file is present on the server, a modified + version of the rsync algorithm is used to upload only the changed + portions of the file. + + After a new version is uploaded, the old version is still + available (subject to disc space on the server). Similarly, a deleted + file is still available. The only limit to their availability is space + allocated to this account on the server + + Future versions will add the ability to mark the current state of + files on the server, and restore from this mark. This will emulate the + changing of tapes in a tape backup system. + +
+ Restoration + + Restoring files is performed using a query tool, bbackupquery. + This can be used to restore entire directories, or as an 'FTP-like' + tool to list and retrieve individual files. Old versions and deleted + files can be retrieved using this tool for as long as they are kept on + the server. +
+ +
+ Client Resource Usage + + bbackupd uses only a minimal amount of disc space to store + records on uploaded files -- less than 32 bytes per directory and file + over a set size threshold. However, it minimises the amount of queries + it must make to the server by storing, in memory, a data structure + which allows it to determine what data is new. It does not need to + store a record of all files, essentially just the directory names and + last modification times. This is not a huge amount of memory. + + If there are no changes to the directories, then the client will + not even connect to the server. +
+
+ +
+ Security + + Box Backup is designed to be secure in several ways. The data + stored on the backup store server is encrypted using secret-key + cryptography. Additionally, the transport layer is encrypted using TLS, + to ensure that the communications can't be snooped. + +
+ Encryption + + The files, directories, filenames and file attributes are all + encrypted. By examining the stored files on the server, it is only + possible to determine the approximate sizes of a files and the tree + structure of the disc (not names, just number of files and + subdirectories in a directory). By monitoring the actions performed by + a client, it is possible to determine the frequency and approximate + scope of changes to files and directories. + + The connections between the server and client are encrypted + using TLS (latest version of SSL). Traffic analysis is possible to + some degree, but limited in usefulness. + + An attacker will not be able to recover the backed up data + without the encryption keys. Of course, you won't be able to recover + your files without the keys either, so you must make a conventional, + secure, backup of these keys. +
+ +
+ Authentication + + SSL certificates are used to authenticate clients. UNIX user + accounts are not used to minimise the dependence on the configuration + of the operating system hosting the server. + + A script is provided to run the necessary certification + authority with minimal effort. +
+
+ +
+ Server daemon + + The server daemon is designed to be simple to deploy, and run on + the cheapest hardware possible. To avoid the necessity to use expensive + hardware RAID or software RAID with complex setup, it (optionally) + stores files using RAID techniques. + + It does not need to run as a privileged user. + + Each account has a set amount of disc space allocated, with a soft + and a hard limit. If the account exceeds the soft limit, a housekeeping + process will start deleting old versions and deleted files to reduce the + space used to below the soft limit. If the backup client attempts to + upload a file which causes the store to exceed the hard limit, the + upload will be refused. +
+
+ + + Building and installing + +
+ Before you start + + Firstly, check that all the clocks on your clients, servers and + signing machines are accurate and in sync. A disagreement in time + between a client and a server is the biggest cause of installation + difficulties, as the times in the generated certificates will cause + login failures if the start date is in the future. +
+ +
+ Box Backup compile + + In the following instructions, replace 0.00 with the actual + version number of the archive you have downloaded. + + For help building on Windows, see the Windows + Compile Appendix. And if you want to build a Linux RPM, look here. + + You need the latest version of OpenSSL, as some of the extra APIs + it provides are required. You should have this anyway, as earlier + versions have security flaws. (If you have an earlier version installed, + the configuration script will give you instructions on enabling + experimental support for older versions.) + + See OpenSSL notes for more information + on OpenSSL issues. + + There are some notes in the archive about compiling on various + platforms within the boxbackup-0.00 directory -- read them first. For + example, if you are compiling under Linux, look for LINUX.txt as + boxbackup-0.00/LINUX.txt after untaring the archive. + + Download the archive, then in that directory type + + tar xvzf boxbackup-0.00.tgz +cd boxbackup-0.00 +./configure +make + + The server and client will be built and packaged up for + installation on this machine, or ready to be transferred as tar files to + another machine for installation. + + This builds two parcels of binaries and scripts, 'backup-client' + and 'backup-server'. The generated installation scripts assumes you want + everything installed in /usr/local/bin + + Optionally, type make test to run + all the tests. +
+ +
+ Local installation + + Type make install-backup-client + to install the backup client. + + Type make install-backup-server + to install the backup server. +
+ +
+ Remote installation + + In the parcels directory, there are tar files for each parcel. The + name reflects the version and platform you have built it for. + + Transfer this tar file to the remote server, and unpack it, then + run the install script. For example: + + tar xvzf boxbackup-0.00-backup-server-OpenBSD.tgz +cd boxbackup-0.00-backup-server-OpenBSD +./install-backup-server +
+ +
+ Configure options + + You can use arguments to the configure script to adjust the + compile and link lines in the generated Makefiles, should this be + necessary for your platform. The configure script takes the usual GNU + autoconf arguments, a full list of which can be obtained with --help. Additional options for Box Backup + include: + + + + + + --enable-gnu-readline + + Use GNU readline if present. Linking Box Backup against + GNU readline may create licence implications if you then + distribute the binaries. libeditline is also supported as a safe + alternative, and is used by default if available. + + + + --disable-largefile + + Omit support for large files + + + + --with-bdb-lib=DIR + + Specify Berkeley DB library location + + + + --with-bdb-headers=DIR + + Specify Berkeley DB headers location + + + + --with-random=FILE + + Use FILE as random number seed (normally + auto-detected) + + + + --with-tmp-dir=DIR + + Directory for temporary files (normally /tmp) + + + + + + See OpenSSL notes for the OpenSSL + specific options. +
+ +
+ Tests + + There are a number of unit tests provided. To compile and run one + type: + + ./runtest.pl bbackupd release +./runtest.pl common debug +./runtest.pl ALL + + The runtest.pl script will compile and run the test. The first + argument is the test name, and the second the type of build. Use ALL as + a test name to run all the tests. + + The output from the tests is slightly muddled using this script. + If you're developing, porting or trying out new things, it might be + better to use the following scheme: + + cd test/bbackupd +make +cd ../../debug/test/bbackupd +./t + + or in release mode... + + cd test/bbackupd +make -D RELEASE +cd ../../release/test/bbackupd +./t + + (use RELEASE=1 with GNU make) + + I tend to use two windows, one for compilation, and one for + running tests. +
+
+ + + Box Backup and SSL + +
+ General notes + + Ideally, you need to use version 0.9.7 or later of OpenSSL. If + this is installed on your system by default (and it is on most recent + releases of UNIX like OSes) then everything should just work. + + However, if it isn't, you have a few options. + +
+ Upgrade your installation + + The best option is to upgrade your installation to use 0.9.7. + Hopefully your package manager will make this easy for you. This may + require reinstallation of lots of software which depends on OpenSSL, + so may not be ideal. + + (But as there have been a few security flaws in OpenSSL + recently, you probably want to upgrade it anyway.) +
+ +
+ Install another OpenSSL + + The second best option is to install another copy. If you + download and install from source, it will probably install into + /usr/local/ssl. You can then configure Box Backup to use it + using: + + ./configure --with-ssl-headers=/usr/local/ssl/include --with-ssl-lib=/usr/local/ssl/lib + + which will set up the various includes and libraries for + you. + + The configuration scripts may be a problem, depending on your + installation. See below for more information. +
+ +
+ Use the old version of OpenSSL + + If you have an old version installed, the configuration script + will give you instructions on how to enable support for older + versions. Read the warnings, and please, whatever you do, don't + release binary packages or ports which enable this option. + + You may have issues with the configuration scripts, see + below. +
+
+ +
+ If you have problems with the config scripts + + If you get OpenSSL related errors with the configuration scripts, + there are two things to check: + + + + The bin directory within your OpenSSL directory is in the path + (if you have installed another version) + + + + You have an openssl.cnf file which works and can be + found. + + + +
+ OpenSSL config file + + You need to have an openssl.cnf file. The default will generally + work well (see example at end). Make sure the openssl utility can find + it, either set the OPENSSL_CONF environment variable, or install it + into the location that is mentioned in the error messages. + + Example OpenSSL config file: + + # +# OpenSSL example configuration file. +# This is mostly being used for generation of certificate requests. +# + +RANDFILE = /dev/arandom + +#################################################################### +[ req ] +default_bits = 1024 +default_keyfile = privkey.pem +distinguished_name = req_distinguished_name +attributes = req_attributes + +[ req_distinguished_name ] +countryName = Country Name (2 letter code) +#countryName_default = AU +countryName_min = 2 +countryName_max = 2 + +stateOrProvinceName = State or Province Name (full name) +#stateOrProvinceName_default = Some-State + +localityName = Locality Name (eg, city) + +0.organizationName = Organization Name (eg, company) +#0.organizationName_default = Internet Widgits Pty Ltd + +# we can do this but it is not needed normally :-) +#1.organizationName = Second Organization Name (eg, company) +#1.organizationName_default = CryptSoft Pty Ltd + +organizationalUnitName = Organizational Unit Name (eg, section) +#organizationalUnitName_default = + +commonName = Common Name (eg, fully qualified host name) +commonName_max = 64 + +emailAddress = Email Address +emailAddress_max = 64 + +[ req_attributes ] +challengePassword = A challenge password +challengePassword_min = 4 +challengePassword_max = 20 + +unstructuredName = An optional company name + +[ x509v3_extensions ] + +nsCaRevocationUrl = http://www.cryptsoft.com/ca-crl.pem +nsComment = "This is a comment" + +# under ASN.1, the 0 bit would be encoded as 80 +nsCertType = 0x40 +
+
+
+ + + Compiling bbackupd on Windows using Visual C++ + + This Appendix explains how to build the bbackupd daemon for Windows + using the Visual C++ compiler. + + If you have any problems following these instructions, please sign + up to the mailing + list and report them to us, or send an email to Chris Wilson. Thanks! + + Note: bbstored will not be built + with this process. bbstored is not currently supported on Windows. There + are no plans for bbstored support on Windows. + +
+ Tools + + You will need quite a bit of software to make this work. All of it + is available for free on the Internet, although Visual C++ Express has + license restrictions and a time limit. + +
+ Visual C++ + + Microsoft's Visual C++ compiler and development environment are + part of their commercial product Visual Studio. Visual + Studio 2005 is supported, and 2003 should work as well. + + You can also download + a free copy of Visual C++ 2005 Express. This copy must be registered + (activated) within 30 days, and is free for one year. + + You will need the Platform + SDK to allow you to compile Windows applications. +
+ +
+ Perl + + Download and install ActivePerl for + Windows, which you can probably find here. +
+ +
+ Libraries + + You will need to download and install several libraries. They + must all be built in the same directory, to be able to link + properly. + + Choose a directory where you will unpack and compile OpenSSL, + Zlib and Box Backup. We will call this the base directory. An example + might be: + + C:\Documents and Settings\Your Username\Desktop\Box + + Make sure you know the full path to this directory. + +
+ OpenSSL + + You will need to compile OpenSSL using Visual C++. The latest + release at this time, OpenSSL 0.9.8a, does not compile with Visual + C++ 2005 out of the box, so you need a + patched version. The original + source and patch + are also available. + + To compile OpenSSL: + + + + Use a Windows unzipper such as WinZip (free trial) to + extract the openssl-0.9.8a-vc2005.tar.gz archive, + which you just downloaded, into the base directory. + + + + Rename the folder from openssl-0.9.8a-vc2005 to openssl + + + + Open a command shell (run cmd.exe) and cd to the openssl directory + + + + Run the following commands: + + perl Configure VC-WIN32 +ms\do_ms +"c:\program files\Microsoft Visual Studio 8\Common7\Tools\vsvars32.bat" +nmake -f ms\ntdll.mak + + +
+ +
+ Zlib + + You will need to download the Zlib compiled DLL. + Create a directory called zlib in + the base directory, and unzip the file you just downloaded into that + directory. You don't need to compile anything. +
+
+ +
+ Download Box Backup + + The first version of Box Backup that's known to compile and with + Visual C++ 2005 is available on the Subversion + server. However, this version has not been extensively tested + and may be out of date. + + The changes are expected to be merged into the Subversion trunk + at some point, and this page should then be updated. If in doubt, + please sign up to the mailing + list and ask us. + + To get the source code out of Subversion you will need a Subversion + client for Windows. After installing it, open a new command + prompt, go to the base directory, and type: + + svn co http://www.boxbackup.org/svn/box/chris/win32/vc2005-compile-fixes/ boxbackup + + This should create a directory called boxbackup inside the base directory. +
+ +
+ Configure Box Backup + + Open a command prompt, change to the base directory then + boxbackup, and run win32.bat to configure the sources. Otherwise, + Visual C++ will complain about missing files whose names start with + autogen, and missing config.h. +
+ +
+ Compile Box Backup + + Open Visual C++. Choose "File/Open/Project", navigate to the + base directory, then to boxbackup\infrastructure\msvc\2005 (or + 2003 if using Visual Studio 2003), + and open any project or solution file in that directory. + + Press F7 to compile Box Backup. If the compilation is + successful, boxbackup\Debug\bbackupd.exe will be + created. +
+ +
+ Install Box Backup + + Create the destination directory, C:\Program Files\Box Backup\bbackupd. + + Write a configuration file, keys and certificate on a Unix + machine, and copy them into the Box + Backup directory, together with the following files from + the base directory: + + + + boxbackup\Debug\bbackupd.exe + + + + openssl\out32dll\libeay32.dll + + + + openssl\out32dll\ssleay32.dll + + + + zlib\zlib1.dll + + + + Ensure that the user running Box Backup can read from the + Box Backup directory, and write to + the bbackupd directory inside + it. + + Run Box Backup by double-clicking on it, and check that it + connects to the server. If the window opens and closes immediately, + it's probably due to a problem with the configuration file - check the + Windows Event Viewer for details. +
+ +
+ Windows Service + + Box Backup can also run as a Windows service, in which case it + will be automatically started at boot time in the background. To + install this, open a command prompt, and run: + + cd "C:\Program Files\Box Backup" +bbackupd.exe -i + + This should output Box Backup service installed. +
+
+
+ + + Compilation and installation by building an RPM on + Linux + + It is very easy to build an RPM of Box Backup on Linux platforms. + This should work on all Red Hat distributions (including Fedora), SuSE, + and probably others too. + + Given that you have the correct development packages installed + simply execute this command (replacing the version number): + + rpmbuild -ta boxbackup-0.00.tgz + + rpmbuild will report where the packages have been written to, and + these can be installed in the normal manner. + + If you have never built an RPM before you should set up a convenient + build area as described in the RPM + book. + + If you wish to customise the package you can find the spec file in + the contrib/rpm directory. + +
Copied: box/trunk/docs/docbook/raidfile-config.xml (from rev 2463, box/trunk/docs/raidfile-config.xml) =================================================================== --- box/trunk/docs/docbook/raidfile-config.xml (rev 0) +++ box/trunk/docs/docbook/raidfile-config.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,198 @@ + + + + raidfile-config + + 8 + + Box Backup + + Box Backup + + 0.11 + + + + raidfile-config + + Configure Box Backup's RAID files + + + + + raidfile-config + + config-dir + + blocksize + + dir1 dir2 dir3 + + + + + Description + + raidfile-config creates a raidfile.conf file for Box Backup. This + file holds information about the directories used to store backups in. Box + Backup supports userland RAID, in a restricted RAID5 configuration, where + 3 and only 3 'drives' are supported. You can read more about RAID5 (and + other RAID-levels) here. + + + Parameters + + The parameters are as follows: + + + + config-dir + + + The directory path where configuration files are located. + Usually this is /etc/box. + raidfile.conf will be written in this + directory. + + + + + blocksize + + + The block size used for file storage in the system, in + bytes. Using a multiple of the file system block size is a good + strategy. Depending on the size of the files you will be backing + up, this multiple varies. Of course it also depends on the native + block size of your file system. + + + + + dir1 + + + The first directory in the built-in RAID array. + + + + + dir2 + + + The second directory in the built-in RAID array. If you are + not using the built-in RAID functionality, this field should be + ignored. You should not use the built-in RAID if you have a + hardware RAID solution or if you're using another type of software + RAID (like md on Linux). + + + + + dir3 + + + The third directory in the built-in RAID array. The same + notes that apply to dir2 also apply to + dir3. + + + + + Note that there are currently no way to add multiple disk sets to + the raidfile.conf file using command line tools, etc. See + raidfile.conf + + 5 + for details on adding more disks. + + + + + Bugs + + If you find a bug in Box Backup, and you want to let us know about + it, join the mailing + list, and send a description of the problem there. + + To report a bug, give us at least the following information: + + + + The version of Box Backup you are running + + + + The platform you are running on (hardware and OS), for both + client and server. + + + + If possible attach your config files (bbstored.conf, + bbackupd.conf) to the bug report. + + + + Also attach any log file output that helps shed light on the + problem you are seeing. + + + + And last but certainly not least, a description of what you are + seeing, in as much detail as possible. + + + + + + Files + + raidfile-config generates the + raidfile.conf + + 5 + file. + + + + See Also + + + bbstored-config + + 8 + , + bbstored.conf + + 5 + , + raidfile.conf + + 5 + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Copied: box/trunk/docs/docbook/raidfile.conf.xml (from rev 2463, box/trunk/docs/raidfile.conf.xml) =================================================================== --- box/trunk/docs/docbook/raidfile.conf.xml (rev 0) +++ box/trunk/docs/docbook/raidfile.conf.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,143 @@ + + + + raidfile.conf + + 5 + + Box Backup + + Box Backup + + 0.11 + + + + raidfile.conf + + Userland RAID for Box Backup + + + + + /etc/box/raidfile.conf + + + + + Description + + The raidfile.conf is usually generated by + raidfile-config + + 8 + but may be manually edited if the store locations move + or if more than one disc set is required. + + + + discX + + + Specifies a set of discs. + + + + SetNumber + + + The set number of the RAID disc, referenced by each + account. + + + + + BlockSize + + + The block size of the file system (usually 2048). + Under BSD with FFS, set this to your file system's + fragment size (most likely an 8th of the block + size). + + + + + Dir0 + + + The first directory in the RAID array. + + + + + Dir1 + + + The second directory in the RAID array. If you do + not wish to use the built-in RAID functionality, this + field should be set to the same as + Dir0. You should not use the built-in + RAID if you have a hardware RAID solution or if you're + using another type of software RAID (like md on + Linux). + + + + + Dir2 + + + The third directory in the RAID array. The same + notes that apply to Dir2 also apply to + Dir3. + + + + + + + + + + Files + + /etc/box/raidfile.conf + + + + See Also + + + raidfile-config + + 8 + , + bbstored.conf + + 5 + + + + + Authors + + + Ben Summers + + + + Per Thomsen + + + + James O'Gorman + + + Deleted: box/trunk/docs/favicon.ico =================================================================== (Binary files differ) Deleted: box/trunk/docs/generate_except_xml.pl =================================================================== --- box/trunk/docs/generate_except_xml.pl 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/generate_except_xml.pl 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,74 +0,0 @@ -#!/usr/bin/perl -w -use strict; - -open (EXCEPT, "<../ExceptionCodes.txt") or die "Can't open ../ExceptionCodes.txt: $!\n"; -open (DOCBOOK, ">ExceptionCodes.xml") or die "Can't open Exceptioncodes.xml for writing: $!\n"; - -print DOCBOOK < - - - Exception codes - -EOD -my $sectionName; -my $sectionNum; -my $sectionDesc; -my $exceptionCode; -my $exceptionShortDesc; -my $exceptionLongDesc; -while() -{ - next if(m/^#/); - chomp; - if(m/^EXCEPTION TYPE (\w+) (\d+)/) - { - $sectionName = ucfirst(lc($1)); - $sectionNum = $2; - if($sectionName ne "Common") - { - $sectionDesc = "the " . $sectionName; - } - else - { - $sectionDesc = "any"; - } - print DOCBOOK < - $sectionName Exceptions ($sectionNum) - - These are exceptions that can occur in $sectionDesc module - of the system. - - -EOD - } - - # The END TYPE line - if(m/^END TYPE$/) - { - print DOCBOOK " \n \n"; - } - - # The actual exceptions - if(m/(\(\d+\/\d+\)) - (\w+ \w+)(?: - )?(.*)$/) - { - $exceptionCode = $1; - $exceptionShortDesc = $2; - $exceptionLongDesc = $3; - - print DOCBOOK " \n "; - print DOCBOOK $exceptionCode . ": " . $exceptionShortDesc . ""; - if($exceptionLongDesc ne "") - { - print DOCBOOK " -- " . $exceptionLongDesc; - } - print DOCBOOK "\n \n"; - } -} - -print DOCBOOK "\n"; - -close EXCEPT; -close DOCBOOK; - Copied: box/trunk/docs/images/bblogo-alpha.xcf (from rev 2463, box/trunk/docs/bblogo-alpha.xcf) =================================================================== (Binary files differ) Deleted: box/trunk/docs/instguide.xml =================================================================== --- box/trunk/docs/instguide.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/instguide.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,766 +0,0 @@ - - - - Box Backup Build and Installation Guide - - - License - - Copyright © 2003 - 2007, Ben Summers and contributors. - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are - met: - - - - Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - - - Redistributions in binary form must reproduce the above - copyright notice, this list of conditions and the following disclaimer - in the documentation and/or other materials provided with the - distribution. - - - - All use of this software and associated advertising materials - must display the following acknowledgement: This product includes - software developed by Ben Summers. - - - - The names of the Authors may not be used to endorse or promote - products derived from this software without specific prior written - permission. - - - - - [Where legally impermissible the Authors do not disclaim liability - for direct physical injury or death caused solely by defects in the - software unless it is modified by a third party.] - - THIS SOFTWARE IS PROVIDED BY THE AUTHORS "AS IS" AND ANY EXPRESS - OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED - WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR - PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR - ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS - OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) - HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, - STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING - IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - POSSIBILITY OF SUCH DAMAGE. - - - - Introduction - - The backup daemon, bbackupd, runs on all machines to be backed up. - The store server daemon, bbstored runs on a central server. Data is sent - to the store server, which stores all data on local filesystems, that is, - only on local hard drives. Tape or other archive media is not used. - - The system is designed to be easy to set up and run, and cheap to - use. Once set up, there should be no need for user or administrative - intervention, apart from usual system maintenance. - -
- Client daemon - - bbackupd is configured with a list of directories to back up. It - has a lazy approach to backing up data. Every so often, the directories - are scanned, and new data is uploaded to the server. This new data must - be over a set age before it is uploaded. This prevents rapid revisions - of a file resulting in many uploads of the same file in a short period - of time. - - It can also operate in a snapshot mode, which behaves like - traditional backup software. When instructed by an external bbackupctl - program, it will upload all changed files to the server. - - The daemon is always running, although sleeping most of the time. - In lazy mode, it is completely self contained -- scripts running under - cron jobs are not used. The objective is to keep files backed up, not to - make snapshots of the filesystem at particular points in time - available. - - If an old version of the file is present on the server, a modified - version of the rsync algorithm is used to upload only the changed - portions of the file. - - After a new version is uploaded, the old version is still - available (subject to disc space on the server). Similarly, a deleted - file is still available. The only limit to their availability is space - allocated to this account on the server - - Future versions will add the ability to mark the current state of - files on the server, and restore from this mark. This will emulate the - changing of tapes in a tape backup system. - -
- Restoration - - Restoring files is performed using a query tool, bbackupquery. - This can be used to restore entire directories, or as an 'FTP-like' - tool to list and retrieve individual files. Old versions and deleted - files can be retrieved using this tool for as long as they are kept on - the server. -
- -
- Client Resource Usage - - bbackupd uses only a minimal amount of disc space to store - records on uploaded files -- less than 32 bytes per directory and file - over a set size threshold. However, it minimises the amount of queries - it must make to the server by storing, in memory, a data structure - which allows it to determine what data is new. It does not need to - store a record of all files, essentially just the directory names and - last modification times. This is not a huge amount of memory. - - If there are no changes to the directories, then the client will - not even connect to the server. -
-
- -
- Security - - Box Backup is designed to be secure in several ways. The data - stored on the backup store server is encrypted using secret-key - cryptography. Additionally, the transport layer is encrypted using TLS, - to ensure that the communications can't be snooped. - -
- Encryption - - The files, directories, filenames and file attributes are all - encrypted. By examining the stored files on the server, it is only - possible to determine the approximate sizes of a files and the tree - structure of the disc (not names, just number of files and - subdirectories in a directory). By monitoring the actions performed by - a client, it is possible to determine the frequency and approximate - scope of changes to files and directories. - - The connections between the server and client are encrypted - using TLS (latest version of SSL). Traffic analysis is possible to - some degree, but limited in usefulness. - - An attacker will not be able to recover the backed up data - without the encryption keys. Of course, you won't be able to recover - your files without the keys either, so you must make a conventional, - secure, backup of these keys. -
- -
- Authentication - - SSL certificates are used to authenticate clients. UNIX user - accounts are not used to minimise the dependence on the configuration - of the operating system hosting the server. - - A script is provided to run the necessary certification - authority with minimal effort. -
-
- -
- Server daemon - - The server daemon is designed to be simple to deploy, and run on - the cheapest hardware possible. To avoid the necessity to use expensive - hardware RAID or software RAID with complex setup, it (optionally) - stores files using RAID techniques. - - It does not need to run as a privileged user. - - Each account has a set amount of disc space allocated, with a soft - and a hard limit. If the account exceeds the soft limit, a housekeeping - process will start deleting old versions and deleted files to reduce the - space used to below the soft limit. If the backup client attempts to - upload a file which causes the store to exceed the hard limit, the - upload will be refused. -
-
- - - Building and installing - -
- Before you start - - Firstly, check that all the clocks on your clients, servers and - signing machines are accurate and in sync. A disagreement in time - between a client and a server is the biggest cause of installation - difficulties, as the times in the generated certificates will cause - login failures if the start date is in the future. -
- -
- Box Backup compile - - In the following instructions, replace 0.00 with the actual - version number of the archive you have downloaded. - - For help building on Windows, see the Windows - Compile Appendix. And if you want to build a Linux RPM, look here. - - You need the latest version of OpenSSL, as some of the extra APIs - it provides are required. You should have this anyway, as earlier - versions have security flaws. (If you have an earlier version installed, - the configuration script will give you instructions on enabling - experimental support for older versions.) - - See OpenSSL notes for more information - on OpenSSL issues. - - There are some notes in the archive about compiling on various - platforms within the boxbackup-0.00 directory -- read them first. For - example, if you are compiling under Linux, look for LINUX.txt as - boxbackup-0.00/LINUX.txt after untaring the archive. - - Download the archive, then in that directory type - - tar xvzf boxbackup-0.00.tgz -cd boxbackup-0.00 -./configure -make - - The server and client will be built and packaged up for - installation on this machine, or ready to be transferred as tar files to - another machine for installation. - - This builds two parcels of binaries and scripts, 'backup-client' - and 'backup-server'. The generated installation scripts assumes you want - everything installed in /usr/local/bin - - Optionally, type make test to run - all the tests. -
- -
- Local installation - - Type make install-backup-client - to install the backup client. - - Type make install-backup-server - to install the backup server. -
- -
- Remote installation - - In the parcels directory, there are tar files for each parcel. The - name reflects the version and platform you have built it for. - - Transfer this tar file to the remote server, and unpack it, then - run the install script. For example: - - tar xvzf boxbackup-0.00-backup-server-OpenBSD.tgz -cd boxbackup-0.00-backup-server-OpenBSD -./install-backup-server -
- -
- Configure options - - You can use arguments to the configure script to adjust the - compile and link lines in the generated Makefiles, should this be - necessary for your platform. The configure script takes the usual GNU - autoconf arguments, a full list of which can be obtained with --help. Additional options for Box Backup - include: - - - - - - --enable-gnu-readline - - Use GNU readline if present. Linking Box Backup against - GNU readline may create licence implications if you then - distribute the binaries. libeditline is also supported as a safe - alternative, and is used by default if available. - - - - --disable-largefile - - Omit support for large files - - - - --with-bdb-lib=DIR - - Specify Berkeley DB library location - - - - --with-bdb-headers=DIR - - Specify Berkeley DB headers location - - - - --with-random=FILE - - Use FILE as random number seed (normally - auto-detected) - - - - --with-tmp-dir=DIR - - Directory for temporary files (normally /tmp) - - - - - - See OpenSSL notes for the OpenSSL - specific options. -
- -
- Tests - - There are a number of unit tests provided. To compile and run one - type: - - ./runtest.pl bbackupd release -./runtest.pl common debug -./runtest.pl ALL - - The runtest.pl script will compile and run the test. The first - argument is the test name, and the second the type of build. Use ALL as - a test name to run all the tests. - - The output from the tests is slightly muddled using this script. - If you're developing, porting or trying out new things, it might be - better to use the following scheme: - - cd test/bbackupd -make -cd ../../debug/test/bbackupd -./t - - or in release mode... - - cd test/bbackupd -make -D RELEASE -cd ../../release/test/bbackupd -./t - - (use RELEASE=1 with GNU make) - - I tend to use two windows, one for compilation, and one for - running tests. -
-
- - - Box Backup and SSL - -
- General notes - - Ideally, you need to use version 0.9.7 or later of OpenSSL. If - this is installed on your system by default (and it is on most recent - releases of UNIX like OSes) then everything should just work. - - However, if it isn't, you have a few options. - -
- Upgrade your installation - - The best option is to upgrade your installation to use 0.9.7. - Hopefully your package manager will make this easy for you. This may - require reinstallation of lots of software which depends on OpenSSL, - so may not be ideal. - - (But as there have been a few security flaws in OpenSSL - recently, you probably want to upgrade it anyway.) -
- -
- Install another OpenSSL - - The second best option is to install another copy. If you - download and install from source, it will probably install into - /usr/local/ssl. You can then configure Box Backup to use it - using: - - ./configure --with-ssl-headers=/usr/local/ssl/include --with-ssl-lib=/usr/local/ssl/lib - - which will set up the various includes and libraries for - you. - - The configuration scripts may be a problem, depending on your - installation. See below for more information. -
- -
- Use the old version of OpenSSL - - If you have an old version installed, the configuration script - will give you instructions on how to enable support for older - versions. Read the warnings, and please, whatever you do, don't - release binary packages or ports which enable this option. - - You may have issues with the configuration scripts, see - below. -
-
- -
- If you have problems with the config scripts - - If you get OpenSSL related errors with the configuration scripts, - there are two things to check: - - - - The bin directory within your OpenSSL directory is in the path - (if you have installed another version) - - - - You have an openssl.cnf file which works and can be - found. - - - -
- OpenSSL config file - - You need to have an openssl.cnf file. The default will generally - work well (see example at end). Make sure the openssl utility can find - it, either set the OPENSSL_CONF environment variable, or install it - into the location that is mentioned in the error messages. - - Example OpenSSL config file: - - # -# OpenSSL example configuration file. -# This is mostly being used for generation of certificate requests. -# - -RANDFILE = /dev/arandom - -#################################################################### -[ req ] -default_bits = 1024 -default_keyfile = privkey.pem -distinguished_name = req_distinguished_name -attributes = req_attributes - -[ req_distinguished_name ] -countryName = Country Name (2 letter code) -#countryName_default = AU -countryName_min = 2 -countryName_max = 2 - -stateOrProvinceName = State or Province Name (full name) -#stateOrProvinceName_default = Some-State - -localityName = Locality Name (eg, city) - -0.organizationName = Organization Name (eg, company) -#0.organizationName_default = Internet Widgits Pty Ltd - -# we can do this but it is not needed normally :-) -#1.organizationName = Second Organization Name (eg, company) -#1.organizationName_default = CryptSoft Pty Ltd - -organizationalUnitName = Organizational Unit Name (eg, section) -#organizationalUnitName_default = - -commonName = Common Name (eg, fully qualified host name) -commonName_max = 64 - -emailAddress = Email Address -emailAddress_max = 64 - -[ req_attributes ] -challengePassword = A challenge password -challengePassword_min = 4 -challengePassword_max = 20 - -unstructuredName = An optional company name - -[ x509v3_extensions ] - -nsCaRevocationUrl = http://www.cryptsoft.com/ca-crl.pem -nsComment = "This is a comment" - -# under ASN.1, the 0 bit would be encoded as 80 -nsCertType = 0x40 -
-
-
- - - Compiling bbackupd on Windows using Visual C++ - - This Appendix explains how to build the bbackupd daemon for Windows - using the Visual C++ compiler. - - If you have any problems following these instructions, please sign - up to the mailing - list and report them to us, or send an email to Chris Wilson. Thanks! - - Note: bbstored will not be built - with this process. bbstored is not currently supported on Windows. There - are no plans for bbstored support on Windows. - -
- Tools - - You will need quite a bit of software to make this work. All of it - is available for free on the Internet, although Visual C++ Express has - license restrictions and a time limit. - -
- Visual C++ - - Microsoft's Visual C++ compiler and development environment are - part of their commercial product Visual Studio. Visual - Studio 2005 is supported, and 2003 should work as well. - - You can also download - a free copy of Visual C++ 2005 Express. This copy must be registered - (activated) within 30 days, and is free for one year. - - You will need the Platform - SDK to allow you to compile Windows applications. -
- -
- Perl - - Download and install ActivePerl for - Windows, which you can probably find here. -
- -
- Libraries - - You will need to download and install several libraries. They - must all be built in the same directory, to be able to link - properly. - - Choose a directory where you will unpack and compile OpenSSL, - Zlib and Box Backup. We will call this the base directory. An example - might be: - - C:\Documents and Settings\Your Username\Desktop\Box - - Make sure you know the full path to this directory. - -
- OpenSSL - - You will need to compile OpenSSL using Visual C++. The latest - release at this time, OpenSSL 0.9.8a, does not compile with Visual - C++ 2005 out of the box, so you need a - patched version. The original - source and patch - are also available. - - To compile OpenSSL: - - - - Use a Windows unzipper such as WinZip (free trial) to - extract the openssl-0.9.8a-vc2005.tar.gz archive, - which you just downloaded, into the base directory. - - - - Rename the folder from openssl-0.9.8a-vc2005 to openssl - - - - Open a command shell (run cmd.exe) and cd to the openssl directory - - - - Run the following commands: - - perl Configure VC-WIN32 -ms\do_ms -"c:\program files\Microsoft Visual Studio 8\Common7\Tools\vsvars32.bat" -nmake -f ms\ntdll.mak - - -
- -
- Zlib - - You will need to download the Zlib compiled DLL. - Create a directory called zlib in - the base directory, and unzip the file you just downloaded into that - directory. You don't need to compile anything. -
-
- -
- Download Box Backup - - The first version of Box Backup that's known to compile and with - Visual C++ 2005 is available on the Subversion - server. However, this version has not been extensively tested - and may be out of date. - - The changes are expected to be merged into the Subversion trunk - at some point, and this page should then be updated. If in doubt, - please sign up to the mailing - list and ask us. - - To get the source code out of Subversion you will need a Subversion - client for Windows. After installing it, open a new command - prompt, go to the base directory, and type: - - svn co http://www.boxbackup.org/svn/box/chris/win32/vc2005-compile-fixes/ boxbackup - - This should create a directory called boxbackup inside the base directory. -
- -
- Configure Box Backup - - Open a command prompt, change to the base directory then - boxbackup, and run win32.bat to configure the sources. Otherwise, - Visual C++ will complain about missing files whose names start with - autogen, and missing config.h. -
- -
- Compile Box Backup - - Open Visual C++. Choose "File/Open/Project", navigate to the - base directory, then to boxbackup\infrastructure\msvc\2005 (or - 2003 if using Visual Studio 2003), - and open any project or solution file in that directory. - - Press F7 to compile Box Backup. If the compilation is - successful, boxbackup\Debug\bbackupd.exe will be - created. -
- -
- Install Box Backup - - Create the destination directory, C:\Program Files\Box Backup\bbackupd. - - Write a configuration file, keys and certificate on a Unix - machine, and copy them into the Box - Backup directory, together with the following files from - the base directory: - - - - boxbackup\Debug\bbackupd.exe - - - - openssl\out32dll\libeay32.dll - - - - openssl\out32dll\ssleay32.dll - - - - zlib\zlib1.dll - - - - Ensure that the user running Box Backup can read from the - Box Backup directory, and write to - the bbackupd directory inside - it. - - Run Box Backup by double-clicking on it, and check that it - connects to the server. If the window opens and closes immediately, - it's probably due to a problem with the configuration file - check the - Windows Event Viewer for details. -
- -
- Windows Service - - Box Backup can also run as a Windows service, in which case it - will be automatically started at boot time in the background. To - install this, open a command prompt, and run: - - cd "C:\Program Files\Box Backup" -bbackupd.exe -i - - This should output Box Backup service installed. -
-
-
- - - Compilation and installation by building an RPM on - Linux - - It is very easy to build an RPM of Box Backup on Linux platforms. - This should work on all Red Hat distributions (including Fedora), SuSE, - and probably others too. - - Given that you have the correct development packages installed - simply execute this command (replacing the version number): - - rpmbuild -ta boxbackup-0.00.tgz - - rpmbuild will report where the packages have been written to, and - these can be installed in the normal manner. - - If you have never built an RPM before you should set up a convenient - build area as described in the RPM - book. - - If you wish to customise the package you can find the spec file in - the contrib/rpm directory. - -
Deleted: box/trunk/docs/raidfile-config.xml =================================================================== --- box/trunk/docs/raidfile-config.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/raidfile-config.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,198 +0,0 @@ - - - - raidfile-config - - 8 - - Box Backup - - Box Backup - - 0.11 - - - - raidfile-config - - Configure Box Backup's RAID files - - - - - raidfile-config - - config-dir - - blocksize - - dir1 dir2 dir3 - - - - - Description - - raidfile-config creates a raidfile.conf file for Box Backup. This - file holds information about the directories used to store backups in. Box - Backup supports userland RAID, in a restricted RAID5 configuration, where - 3 and only 3 'drives' are supported. You can read more about RAID5 (and - other RAID-levels) here. - - - Parameters - - The parameters are as follows: - - - - config-dir - - - The directory path where configuration files are located. - Usually this is /etc/box. - raidfile.conf will be written in this - directory. - - - - - blocksize - - - The block size used for file storage in the system, in - bytes. Using a multiple of the file system block size is a good - strategy. Depending on the size of the files you will be backing - up, this multiple varies. Of course it also depends on the native - block size of your file system. - - - - - dir1 - - - The first directory in the built-in RAID array. - - - - - dir2 - - - The second directory in the built-in RAID array. If you are - not using the built-in RAID functionality, this field should be - ignored. You should not use the built-in RAID if you have a - hardware RAID solution or if you're using another type of software - RAID (like md on Linux). - - - - - dir3 - - - The third directory in the built-in RAID array. The same - notes that apply to dir2 also apply to - dir3. - - - - - Note that there are currently no way to add multiple disk sets to - the raidfile.conf file using command line tools, etc. See - raidfile.conf - - 5 - for details on adding more disks. - - - - - Bugs - - If you find a bug in Box Backup, and you want to let us know about - it, join the mailing - list, and send a description of the problem there. - - To report a bug, give us at least the following information: - - - - The version of Box Backup you are running - - - - The platform you are running on (hardware and OS), for both - client and server. - - - - If possible attach your config files (bbstored.conf, - bbackupd.conf) to the bug report. - - - - Also attach any log file output that helps shed light on the - problem you are seeing. - - - - And last but certainly not least, a description of what you are - seeing, in as much detail as possible. - - - - - - Files - - raidfile-config generates the - raidfile.conf - - 5 - file. - - - - See Also - - - bbstored-config - - 8 - , - bbstored.conf - - 5 - , - raidfile.conf - - 5 - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Deleted: box/trunk/docs/raidfile.conf.xml =================================================================== --- box/trunk/docs/raidfile.conf.xml 2009-03-28 00:21:14 UTC (rev 2473) +++ box/trunk/docs/raidfile.conf.xml 2009-03-28 12:25:05 UTC (rev 2474) @@ -1,143 +0,0 @@ - - - - raidfile.conf - - 5 - - Box Backup - - Box Backup - - 0.11 - - - - raidfile.conf - - Userland RAID for Box Backup - - - - - /etc/box/raidfile.conf - - - - - Description - - The raidfile.conf is usually generated by - raidfile-config - - 8 - but may be manually edited if the store locations move - or if more than one disc set is required. - - - - discX - - - Specifies a set of discs. - - - - SetNumber - - - The set number of the RAID disc, referenced by each - account. - - - - - BlockSize - - - The block size of the file system (usually 2048). - Under BSD with FFS, set this to your file system's - fragment size (most likely an 8th of the block - size). - - - - - Dir0 - - - The first directory in the RAID array. - - - - - Dir1 - - - The second directory in the RAID array. If you do - not wish to use the built-in RAID functionality, this - field should be set to the same as - Dir0. You should not use the built-in - RAID if you have a hardware RAID solution or if you're - using another type of software RAID (like md on - Linux). - - - - - Dir2 - - - The third directory in the RAID array. The same - notes that apply to Dir2 also apply to - Dir3. - - - - - - - - - - Files - - /etc/box/raidfile.conf - - - - See Also - - - raidfile-config - - 8 - , - bbstored.conf - - 5 - - - - - Authors - - - Ben Summers - - - - Per Thomsen - - - - James O'Gorman - - - Copied: box/trunk/docs/tools/generate_except_xml.pl (from rev 2463, box/trunk/docs/generate_except_xml.pl) =================================================================== --- box/trunk/docs/tools/generate_except_xml.pl (rev 0) +++ box/trunk/docs/tools/generate_except_xml.pl 2009-03-28 12:25:05 UTC (rev 2474) @@ -0,0 +1,74 @@ +#!/usr/bin/perl -w +use strict; + +open (EXCEPT, "< $ARGV[0]") or die "Can't open $ARGV[0]: $!\n"; +open (DOCBOOK, "> $ARGV[1]") or die "Can't open $ARGV[1] for writing: $!\n"; + +print DOCBOOK < + + + Exception codes + +EOD +my $sectionName; +my $sectionNum; +my $sectionDesc; +my $exceptionCode; +my $exceptionShortDesc; +my $exceptionLongDesc; +while() +{ + next if(m/^#/); + chomp; + if(m/^EXCEPTION TYPE (\w+) (\d+)/) + { + $sectionName = ucfirst(lc($1)); + $sectionNum = $2; + if($sectionName ne "Common") + { + $sectionDesc = "the " . $sectionName; + } + else + { + $sectionDesc = "any"; + } + print DOCBOOK < + $sectionName Exceptions ($sectionNum) + + These are exceptions that can occur in $sectionDesc module + of the system. + + +EOD + } + + # The END TYPE line + if(m/^END TYPE$/) + { + print DOCBOOK " \n \n"; + } + + # The actual exceptions + if(m/(\(\d+\/\d+\)) - (\w+ \w+)(?: - )?(.*)$/) + { + $exceptionCode = $1; + $exceptionShortDesc = $2; + $exceptionLongDesc = $3; + + print DOCBOOK " \n "; + print DOCBOOK $exceptionCode . ": " . $exceptionShortDesc . ""; + if($exceptionLongDesc ne "") + { + print DOCBOOK " -- " . $exceptionLongDesc; + } + print DOCBOOK "\n \n"; + } +} + +print DOCBOOK "\n"; + +close EXCEPT; +close DOCBOOK; + From boxbackup-dev at boxbackup.org Sat Mar 28 14:01:40 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sat, 28 Mar 2009 14:01:40 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2475 - box/trunk/infrastructure Message-ID: <20090328140140.6F6EE326025@www.boxbackup.org> Author: chris Date: 2009-03-28 14:01:39 +0000 (Sat, 28 Mar 2009) New Revision: 2475 Modified: box/trunk/infrastructure/makeparcels.pl.in Log: Use the Makefile in docs/ to build and clean documentation. Don't rely on location of bb-man.xsl any more. Modified: box/trunk/infrastructure/makeparcels.pl.in =================================================================== --- box/trunk/infrastructure/makeparcels.pl.in 2009-03-28 12:25:05 UTC (rev 2474) +++ box/trunk/infrastructure/makeparcels.pl.in 2009-03-28 14:01:39 UTC (rev 2475) @@ -110,19 +110,21 @@ { print MAKE "\tfind release debug -type f -exec rm -f {} \\;\n"; } -print MAKE "\trm -rf docs/*.[58] docs/bb-man.xsl docs/man\n" if $product_version =~ /trunk_[0-9]+/; -print MAKE "\n"; +print MAKE <<__END_OF_FRAGMENT; + \$(MAKE) -C docs clean -print MAKE "test:\trelease/common/test\n\nrelease/common/test:\n\t./runtest.pl ALL release\n\n"; +test: release/common/test -print MAKE < Author: chris Date: 2009-03-28 14:04:47 +0000 (Sat, 28 Mar 2009) New Revision: 2476 Modified: box/trunk/distribution/COMMON-MANIFEST.txt box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt box/trunk/infrastructure/makedistribution.pl.in Log: Preserve timestamps when doing a binary copy to make a distribution, so that the built gzipped man pages are still newer than the XML source. Make NO-LICENSE also copy the file, to avoid redundancy in manifests. Fix location of API docs. Create docs/docbook directory to avoid error in making distribution. Modified: box/trunk/distribution/COMMON-MANIFEST.txt =================================================================== --- box/trunk/distribution/COMMON-MANIFEST.txt 2009-03-28 14:01:39 UTC (rev 2475) +++ box/trunk/distribution/COMMON-MANIFEST.txt 2009-03-28 14:04:47 UTC (rev 2476) @@ -12,10 +12,10 @@ test/crypto test/compress test/win32 -docs/api-docs/common docs/api-notes -docs/api-docs/common/lib_common docs/api-notes/lib_common -docs/api-docs/common/lib_crypto docs/api-notes/lib_crypto -docs/api-docs/common/lib_server docs/api-notes/lib_server +docs/api-notes +docs/api-notes/common/lib_common +docs/api-notes/common/lib_crypto +docs/api-notes/common/lib_server MKDIR infrastructure infrastructure/buildenv-testmain-template.cpp infrastructure/makebuildenv.pl.in @@ -26,13 +26,11 @@ infrastructure/BoxPlatform.pm.in infrastructure/mingw infrastructure/msvc -configure.ac NO-LICENSE config.sub -config.sub NO-LICENSE config.guess -config.guess bootstrap +configure.ac +configure parcels.txt runtest.pl.in NO-LICENSE-IN-DIR infrastructure/m4 -configure Modified: box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt =================================================================== --- box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt 2009-03-28 14:01:39 UTC (rev 2475) +++ box/trunk/distribution/boxbackup/DISTRIBUTION-MANIFEST.txt 2009-03-28 14:04:47 UTC (rev 2476) @@ -22,15 +22,14 @@ docs/Makefile docs/tools docs/api-notes -docs/api-notes/lib_raidfile +docs/api-notes/raidfile/lib_raidfile docs/htmlguide docs/htmlguide/adminguide docs/htmlguide/images docs/htmlguide/instguide docs/htmlguide/manpages docs/man -# RUN cd docs; sed -i"" -e '/^ExceptionCodes/,/^$/s/\(.*\)/# &/' Makefile -docs/docbook/Makefile +MKDIR docs/docbook docs/docbook/ExceptionCodes.xml docs/docbook/adminguide.xml docs/docbook/bb-book.xsl @@ -51,7 +50,6 @@ docs/docbook/raidfile.conf.xml docs/docbook/html docs/docbook/html/images -# RUN svn revert docs/Makefile TODO.txt BUGS.txt contrib Modified: box/trunk/infrastructure/makedistribution.pl.in =================================================================== --- box/trunk/infrastructure/makedistribution.pl.in 2009-03-28 14:01:39 UTC (rev 2475) +++ box/trunk/infrastructure/makedistribution.pl.in 2009-03-28 14:04:47 UTC (rev 2476) @@ -115,6 +115,13 @@ elsif($src eq 'NO-LICENSE') { $no_license{$dst} = 1; + + # actually copy it, to remove redundancy in manifests + $src = $dst; + $dst = $other; + $dst = $src if $dst eq ''; + + copy_file($src,$dst); } elsif($src eq 'RUN') { @@ -252,7 +259,8 @@ { # copy as binary # print "binary copy $fn to $base_name/$dst_fn\n"; - system 'cp',$fn,"$base_name/$dst_fn" + my $cmd = "cp -p $fn $base_name/$dst_fn"; + system($cmd) == 0 or die "copy failed: $cmd"; } # copy executable bit from src From boxbackup-dev at boxbackup.org Sat Mar 28 15:41:10 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sat, 28 Mar 2009 15:41:10 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2477 - box/trunk/docs Message-ID: <20090328154110.1E80432501F@www.boxbackup.org> Author: chris Date: 2009-03-28 15:41:09 +0000 (Sat, 28 Mar 2009) New Revision: 2477 Modified: box/trunk/docs/Makefile Log: Fix the docs Makefile to work on both linux and BSD again. Modified: box/trunk/docs/Makefile =================================================================== --- box/trunk/docs/Makefile 2009-03-28 14:04:47 UTC (rev 2476) +++ box/trunk/docs/Makefile 2009-03-28 15:41:09 UTC (rev 2477) @@ -12,8 +12,8 @@ NOCHUNKBOOKXSL = $(DOCBOOK_DIR)/bb-nochunk-book.xsl MANXSL = $(DOCBOOK_DIR)/bb-man.xsl -# VPATH= adminguide -# .SUFFIXES: .html .xml .1 .5 .8 +VPATH = $(DOCBOOK_DIR) +.SUFFIXES: .html .xml .gz .1 .5 .8 all: docs @@ -23,19 +23,21 @@ @cp $(DOCBOOK_DIR)/html/*.css $(HTML_DIR)/. @cp $(DOCBOOK_DIR)/html/*.ico $(HTML_DIR)/. -adminguide: $(HTML_DIR)/adminguide/index.html +adminguide: $(DOCBOOK_DIR)/ExceptionCodes.xml $(HTML_DIR)/adminguide/index.html -$(HTML_DIR)/adminguide/index.html: $(DOCBOOK_DIR)/adminguide.xml $(DOCBOOK_DIR)/ExceptionCodes.xml $(BOOKXSL) - # docname=`echo $@ | sed -e 's/\/index.html//'` - $(DBPROC) -o $(HTML_DIR)/adminguide/ $(BOOKXSL) $< +# all sources ($>) is exactly the right args for xsltproc +$(HTML_DIR)/adminguide/index.html: $(BOOKXSL) $(DOCBOOK_DIR)/adminguide.xml + $(DBPROC) -o $(HTML_DIR)/adminguide/ $> instguide: $(HTML_DIR)/instguide/index.html -$(HTML_DIR)/instguide/index.html: $(DOCBOOK_DIR)/instguide.xml $(BOOKXSL) - $(DBPROC) -o $(HTML_DIR)/instguide/ $(BOOKXSL) $< +$(HTML_DIR)/instguide/index.html: $(BOOKXSL) $(DOCBOOK_DIR)/instguide.xml + $(DBPROC) -o $(HTML_DIR)/instguide/ $> +# $< is empty on BSD make when making this rule, $> has all sources +# $< has the target on GNU make, $> is empty $(DOCBOOK_DIR)/ExceptionCodes.xml: ../ExceptionCodes.txt - perl tools/generate_except_xml.pl $< $@ + perl tools/generate_except_xml.pl $< $> $@ manpages: $(MANXSL) man-dirs man-nroff man-html @@ -58,7 +60,7 @@ touch $(HTML_DIR)/man-html/.there man/.there: - then mkdir -p man + mkdir -p man touch man/.there NROFF_PAGES = bbackupd.8 bbackupd-config.8 bbackupctl.8 bbackupquery.8 \ @@ -76,17 +78,32 @@ man-html: $(HTML_FILES) +# GNU make $(HTML_DIR)/man-html/%.html: $(DOCBOOK_DIR)/%.xml $(NOCHUNKBOOKXSL) $(DBPROC) -o $@ $(NOCHUNKBOOKXSL) $< +# GNU make $(MAN_DIR)/%.8.gz: $(DOCBOOK_DIR)/%.xml $(MANXSL) $(DBPROC) -o $(@:.gz=) $(MANXSL) $< gzip $(@:.gz=) +# GNU make $(MAN_DIR)/%.5.gz: $(DOCBOOK_DIR)/%.xml $(MANXSL) $(DBPROC) -o $(@:.gz=) $(MANXSL) $< gzip $(@:.gz=) +# BSD make: the final colon (:) is required to make this line valid syntax +# for a dummy rule in GNU make. It creates a dummy rule in BSD make too. +# Both dummy rules are harmless. +.for MAN_PAGE in $(NROFF_PAGES) : +$(MAN_DIR)/$(MAN_PAGE).gz: $(DOCBOOK_DIR)/$(MAN_PAGE:R).xml + $(DBPROC) -o $(.TARGET:.gz=) $(MANXSL) $> + gzip $(@:.gz=) + +$(HTML_DIR)/man-html/$(MAN_PAGE:R).html: $(DOCBOOK_DIR)/$(MAN_PAGE:R).xml + $(DBPROC) -o $@ $(NOCHUNKBOOKXSL) $> +.endfor : + dockit: clean docs tar zcf documentation-kit-0.10.tar.gz $(HTML_DIR)/ From boxbackup-dev at boxbackup.org Sat Mar 28 15:51:04 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sat, 28 Mar 2009 15:51:04 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2478 - in box/trunk/docs/api-notes: . backup raidfile Message-ID: <20090328155104.5BB4732501F@www.boxbackup.org> Author: chris Date: 2009-03-28 15:51:04 +0000 (Sat, 28 Mar 2009) New Revision: 2478 Added: box/trunk/docs/api-notes/INDEX.txt box/trunk/docs/api-notes/Win32_Clients.txt box/trunk/docs/api-notes/backup_encryption.txt box/trunk/docs/api-notes/bin_bbackupd.txt box/trunk/docs/api-notes/bin_bbstored.txt box/trunk/docs/api-notes/encrypt_rsync.txt box/trunk/docs/api-notes/lib_backupclient.txt box/trunk/docs/api-notes/lib_backupstore.txt box/trunk/docs/api-notes/raidfile/RaidFileRead.txt box/trunk/docs/api-notes/raidfile/RaidFileWrite.txt box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt box/trunk/docs/api-notes/win32_build_on_linux_using_mingw.txt box/trunk/docs/api-notes/windows_porting.txt Removed: box/trunk/docs/api-notes/backup/INDEX.txt box/trunk/docs/api-notes/backup/Win32_Clients.txt box/trunk/docs/api-notes/backup/backup_encryption.txt box/trunk/docs/api-notes/backup/bin_bbackupd.txt box/trunk/docs/api-notes/backup/bin_bbstored.txt box/trunk/docs/api-notes/backup/encrypt_rsync.txt box/trunk/docs/api-notes/backup/lib_backupclient.txt box/trunk/docs/api-notes/backup/lib_backupstore.txt box/trunk/docs/api-notes/backup/win32_build_on_cygwin_using_mingw.txt box/trunk/docs/api-notes/backup/win32_build_on_linux_using_mingw.txt box/trunk/docs/api-notes/backup/windows_porting.txt box/trunk/docs/api-notes/raidfile/lib_raidfile/ Log: Rearrangement of api-notes directory. Copied: box/trunk/docs/api-notes/INDEX.txt (from rev 2474, box/trunk/docs/api-notes/backup/INDEX.txt) =================================================================== --- box/trunk/docs/api-notes/INDEX.txt (rev 0) +++ box/trunk/docs/api-notes/INDEX.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,61 @@ +TITLE Programmers Notes for Box Backup + +This directory contains the programmers notes for the Box Backup system. They will be of interest only if you want to review or modify the code. + +These notes are intended to be run through a script to produce HTML at some later stage, hence the marks such as 'TITLE'. + + +SUBTITLE Organisation + +The project is split up into several modules. The modules within 'lib' are building blocks for creation of the actual executable programs within 'bin'. 'test' contains unit tests for lib and bin modules. + +The file modules.txt lists the modules which are to be built, and their dependencies. It also allows for platform differences. + + +SUBTITLE Documentation Organisation + +In this directory, the files correspond to modules or areas of interest. Sub directories of the same name contain files documenting specific classes within that module. + + +SUBTITLE Suggested reading order + +* common/lib_common.txt +* common/lib_server.txt +* bin_bbackupd.txt +* backup_encryption.txt +* bin_bbstored.txt +* raidfile/lib_raidfile.txt + +and refer to other sections as required. + + +SUBTITLE Building + +The makefiles are generated by makebuildenv.pl. (The top level makefile is generated by makeparcels.pl, but this is for the end user to use, not a programmer.) + +To build a module, cd to it and type make. If the -DRELEASE option is specified (RELEASE=1 with GNU make) the release version will be built. The object files and exes are placed in a directory structure under 'release' or 'debug'. + +It is intended that a test will be written for everything, so in general make commands will be issued only within the test/* modules. Once it has been built, cd to debug/test/ and run the test with ./t . + + +SUBTITLE Programming style + +The code is written to be easy to write. Ease of programming is the primary concern, as this should lead to fewer bugs. Efficiency improvements can be made later when the system as a whole works. + +Much use is made of the STL. + +There is no common base class. + +All errors are reported using exceptions. + +Some of the boring code is generated by perl scripts from description files. + +There are a lot of modules and classes which can easily be used to build other projects in the future -- there is a lot of "framework" code. + + +SUBTITLE Lots more documentation + +The files are extensively commented. Consider this notes as an overview, and then read the source files for detailed and definitive information. + +Each function and class has a very brief decsription of it's purpose in a standard header, and extensive efforts have been maed to comment the code itself. + Copied: box/trunk/docs/api-notes/Win32_Clients.txt (from rev 2474, box/trunk/docs/api-notes/backup/Win32_Clients.txt) =================================================================== --- box/trunk/docs/api-notes/Win32_Clients.txt (rev 0) +++ box/trunk/docs/api-notes/Win32_Clients.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,13 @@ +The basic client tools now run on Win32 natively. +The port was done by nick at omniis.com. + +* bbackupd +* bbackupquery +* bbackupctl + +Have been ported. bbackupd runs as a NT style service. + +Known limitations: + +* File attributes and permissions are not backed up. + Deleted: box/trunk/docs/api-notes/backup/INDEX.txt =================================================================== --- box/trunk/docs/api-notes/backup/INDEX.txt 2009-03-28 15:41:09 UTC (rev 2477) +++ box/trunk/docs/api-notes/backup/INDEX.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -1,61 +0,0 @@ -TITLE Programmers Notes for Box Backup - -This directory contains the programmers notes for the Box Backup system. They will be of interest only if you want to review or modify the code. - -These notes are intended to be run through a script to produce HTML at some later stage, hence the marks such as 'TITLE'. - - -SUBTITLE Organisation - -The project is split up into several modules. The modules within 'lib' are building blocks for creation of the actual executable programs within 'bin'. 'test' contains unit tests for lib and bin modules. - -The file modules.txt lists the modules which are to be built, and their dependencies. It also allows for platform differences. - - -SUBTITLE Documentation Organisation - -In this directory, the files correspond to modules or areas of interest. Sub directories of the same name contain files documenting specific classes within that module. - - -SUBTITLE Suggested reading order - -* lib_common -* lib_server -* bin_bbackupd -* backup_encryption.txt -* bin_bstored -* lib_raidfile - -and refer to other sections as required. - - -SUBTITLE Building - -The makefiles are generated by makebuildenv.pl. (The top level makefile is generated by makeparcels.pl, but this is for the end user to use, not a programmer.) - -To build a module, cd to it and type make. If the -DRELEASE option is specified (RELEASE=1 with GNU make) the release version will be built. The object files and exes are placed in a directory structure under 'release' or 'debug'. - -It is intended that a test will be written for everything, so in general make commands will be issued only within the test/* modules. Once it has been built, cd to debug/test/ and run the test with ./t . - - -SUBTITLE Programming style - -The code is written to be easy to write. Ease of programming is the primary concern, as this should lead to fewer bugs. Efficiency improvements can be made later when the system as a whole works. - -Much use is made of the STL. - -There is no common base class. - -All errors are reported using exceptions. - -Some of the boring code is generated by perl scripts from description files. - -There are a lot of modules and classes which can easily be used to build other projects in the future -- there is a lot of "framework" code. - - -SUBTITLE Lots more documentation - -The files are extensively commented. Consider this notes as an overview, and then read the source files for detailed and definitive information. - -Each function and class has a very brief decsription of it's purpose in a standard header, and extensive efforts have been maed to comment the code itself. - Deleted: box/trunk/docs/api-notes/backup/Win32_Clients.txt =================================================================== --- box/trunk/docs/api-notes/backup/Win32_Clients.txt 2009-03-28 15:41:09 UTC (rev 2477) +++ box/trunk/docs/api-notes/backup/Win32_Clients.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -1,13 +0,0 @@ -The basic client tools now run on Win32 natively. -The port was done by nick at omniis.com. - -* bbackupd -* bbackupquery -* bbackupctl - -Have been ported. bbackupd runs as a NT style service. - -Known limitations: - -* File attributes and permissions are not backed up. - Deleted: box/trunk/docs/api-notes/backup/backup_encryption.txt =================================================================== --- box/trunk/docs/api-notes/backup/backup_encryption.txt 2009-03-28 15:41:09 UTC (rev 2477) +++ box/trunk/docs/api-notes/backup/backup_encryption.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -1,109 +0,0 @@ -TITLE Encryption in the backup system - -This document explains how everything is encrypted in the backup system, and points to the various functions which need reviewing to ensure they do actually follow this scheme. - - -SUBTITLE Security objectives - -The crpyto system is designed to keep the following things secret from an attacker who has full access to the server. - -* The names of the files and directories -* The contents of files and directories -* The exact size of files - -Things which are not secret are - -* Directory heirarchy and number of files in each directory -* How the files change over time -* Approximate size of files - - -SUBTITLE Keys - -There are four separate keys used: - -* Filename -* File attributes -* File block index -* File data - -and an additional secret for file attribute hashes. - -The Cipher is Blowfish in CBC mode in most cases, except for the file data. All keys are maximum length 448 bit keys, since the key size only affects the setup time and this is done very infrequently. - -The file data is encrypted with AES in CBC mode, with a 256 bit key (max length). Blowfish is used elsewhere because the larger block size of AES, while more secure, would be terribly space inefficient. Note that Blowfish may also be used when older versions of OpenSSL are in use, and for backwards compatibility with older versions. - -The keys are generated using "openssl rand", and a 1k file of key material is stored in /etc/box/bbackupd. The configuration scripts make this readable only by root. - -Code for review: BackupClientCryptoKeys_Setup() -in lib/backupclient/BackupClientCryptoKeys.cpp - - -SUBTITLE Filenames - -Filenames need to be secret from the attacker, but they need to be compared on the server so it can determine whether or not is it a new version of an old file. - -So, the same Initialisation Vector is used for every single filename, so the same filename encrypted twice will have the same binary representation. - -Filenames use standard PKCS padding implemented by OpenSSL. They are proceeded by two bytes of header which describe the length, and the encoding. - -Code for review: BackupStoreFilenameClear::EncryptClear() -in lib/backupclient/BackupStoreFilenameClear.cpp - - -SUBTITLE File attributes - -These are kept secret as well, since they reveal information. Especially as they contain the target name of symbolic links. - -To encrypt, a random Initialisation Vector is choosen. This is stored first, followed by the attribute data encrypted with PKCS padding. - -Code for review: BackupClientFileAttributes::EncryptAttr() -in lib/backupclient/BackupClientFileAttributes.cpp - - -SUBTITLE File attribute hashes - -To detect and update file attributes efficiently, the file status change time is not used, as this would give suprious results and result in unnecessary updates to the server. Instead, a hash of user id, group id, and mode is used. - -To avoid revealing details about attributes - -1) The filename is added to the hash, so that an attacker cannot determine whether or not two files have identical attributes - -2) A secret is added to the hash, so that an attacker cannot compare attributes between accounts. - -The hash used is the first 64 bits of an MD5 hash. - - -SUBTITLE File block index - -Files are encoded in blocks, so that the rsync algorithm can be used on them. The data is compressed first before encryption. These small blocks don't give the best possible compression, but there is no alternative because the server can't see their contents. - -The file contains a number of blocks, which contain among other things - -* Size of the block when it's not compressed -* MD5 checksum of the block -* RollingChecksum of the block - -We don't want the attacker to know the size, so the first is bad. (Because of compression and padding, there's uncertainty on the size.) - -When the block is only a few bytes long, the latter two reveal it's contents with only a moderate amount of work. So these need to be encrypted. - -In the header of the index, a 64 bit number is chosen. The sensitive parts of the block are then encrypted, without padding, with an Initialisation Vector of this 64 bit number + the block index. - -If a block from an previous file is included in a new version of a file, the same checksum data will be encrypted again, but with a different IV. An eavesdropper will be able to easily find out which data has been re-encrypted, but the plaintext is not revealed. - -Code for review: BackupStoreFileEncodeStream::Read() (IV base choosen about half-way through) -BackupStoreFileEncodeStream::EncodeCurrentBlock() (encrypt index entry) -in lib/backupclient/BackupStoreFileEncodeStream.cpp - - -SUBTITLE File data - -As above, the first is split into chunks and compressed. - -Then, a random initialisation vector is chosen, stored first, followed by the compressed file data encrypted using PKCS padding. - -Code for review: BackupStoreFileEncodeStream::EncodeCurrentBlock() -in lib/backupclient/BackupStoreFileEncodeStream.cpp - - Deleted: box/trunk/docs/api-notes/backup/bin_bbackupd.txt =================================================================== --- box/trunk/docs/api-notes/backup/bin_bbackupd.txt 2009-03-28 15:41:09 UTC (rev 2477) +++ box/trunk/docs/api-notes/backup/bin_bbackupd.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -1,88 +0,0 @@ -TITLE bin/bbackupd - -The backup client daemon. - -This aims to maintain as little information as possible to record which files have been uploaded to the server, while minimising the amount of queries which have to be made to the server. - - -SUBTITLE Scanning - -The daemon is given a length of time, t, which files over this age should be uploaded to the server. This is to stop recently updated files being uploaded immediately to avoid uploading something repeatedly (on the assumption that if a file has been written, it is likely to be modified again shortly). - -It will scan the files at a configured interval, and connect to the server if it needs to upload files or make queries about files and directories. - -The scan interval is actually varied slightly between each run by adding a random number up to a 64th of the configured time. This is to reduce cyclic patterns of load on the backup servers -- otherwise if all the boxes are turned on at about 9am, every morning at 9am there will be a huge spike in load on the server. - -Each scan chooses a time interval, which ends at the current time - t. This will be from 0 to current time - t on the first run, then the next run takes the start time as the end time of the previous run. The scan is only performed if the difference between the start and end times is greater or equal to t. - -For each configured location, the client scans the directories on disc recursively. - -For each directory - -* If the directory has never been scanned before (in this invocation of the daemon) or the modified time on the directory is not that recorded, the listing on the server is downloaded. - -* For each file, if it's modified time is within the time period, it is uploaded. If the directory has been downloaded, it is compared against that, and only uploaded if it's changed. - -* Find all the new files, and upload them if they lie within the time interval. - -* Recurse to sub directories, creating them on the server if necessary. - -Hence, the first time it runs, it will download and compare the entries on the disc to those on the server, but in future runs it will use the file and directory modification times to work out if there is anything which needs uploading. - -If there aren't any changes, it won't even need to connect to the server. - -There are some extra details which allow this to work reliably, but they are documented in the source. - - -SUBTITLE File attributes - -The backup client will update the file attributes on files as soon as it notices they are changed. It records most of the details from stat(), but only a few can be restored. Attributes will only be considered changed if the user id, group id or mode is changed. Detection is by a 64 bit hash, so detection is strictly speaking probablistic. - - -SUBTITLE Encryption - -All the user data is encrypted. There is a separate file, backup_encryption.txt which describes this, and where in the code to look to verify it works as described. - - -SUBTITLE Tracking files and directories - -Renaming files is a difficult problem under this minimal data scanning scheme, because you don't really know whether a file has been renamed, or another file deleted and new one created. - -The solution is to keep (on disc) a map of inode numbers to server object IDs for all directories and files over a certain user configurable threshold. Then, when a new file is discovered, it is first checked to see if it's in this map. If so, a rename is considered, which will take place if the local object corresponding to the name of the tracked object doesn't exist any more. - -Because of the renaming requirement, deletions of objects from the server are recorded and delayed until the end of the scan. - - -SUBTITLE Running out of space - -If the store server indicates on login to the backup client, it will scan, but not upload anything nor adjust it's internal stored details of the local objects. However, deletions and renames happen. - -This is to allow deletions to still work and reduce the amount of storage space used on the server, in the hope that in the future there will be enough space. - -Just not doing anything would mean that one big file created and then deleted at the wrong time would stall the whole backup process. - - -SUBTITLE BackupDaemon - -This is the daemon class for the backup daemon. It handles setting up of all the objects, and implements calulcation of the time intervals for the scanning. - - -SUBTITLE BackupClientContext - -State information for the scans, including maintaining a connection to the store server if required. - - -SUBTITLE BackupClientDirectoryRecord - -A record of state of a directory on the local filesystem. Containing the recursive scanning function, which is long and entertaining, but very necessary. It contains lots of comments which explain the exact details of what's going on. - - -SUBTITLE BackupClientInodeToIDMap - -A implementation of a map of inode number to object ID on the server. If Berkeley DB is available on the platform, it is stored on disc, otherwise there is an in memory version which isn't so good. - - - - - - Deleted: box/trunk/docs/api-notes/backup/bin_bbstored.txt =================================================================== --- box/trunk/docs/api-notes/backup/bin_bbstored.txt 2009-03-28 15:41:09 UTC (rev 2477) +++ box/trunk/docs/api-notes/backup/bin_bbstored.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -1,54 +0,0 @@ -TITLE bin/bbstored - -The backup store daemon. - -Maintains a store of encrypted files, and every so often goes through deleting unnecessary data. - -Uses an implementation of Protocol to communicate with the backup client daemon. See bin/bbstored/backupprotocol.txt for details. - - -SUBTITLE Data storage - -The data is arranged as a set of objects within a RaidFile disc set. Each object has a 64 bit object ID, which is turned into a filename in a mildly complex manner which ensure that directories don't have too many objects in them, but there is a minimal number of nested directories. See StoreStructure::MakeObjectFilename in lib/backupstore/StoreStructure.cpp for more details. - -An object can be a directory or a file. Directories contain files and other directories. - -Files in directories are supersceded by new versions when uploaded, but the old versions are flagged as such. A new version has a different object ID to the old version. - -Every so often, a housekeeping process works out what can be deleted, and deletes unnecessary files to take them below the storage limits set in the store info file. - - -SUBTITLE Note about file storage and downloading - -There's one slight entertainment to file storage, in that the format of the file streamed depends on whether it's being downloaded or uploaded. - -The problem is that it contains an index of all the blocks. For efficiency in managing these blocks, they all need to be in the same place. - -Files are encoded and decoded as they are streamed to and from the server. With encoding, the index is only completely known at the end of the process, so it's sent last, and lives in the filesystem last. - -When it's downloaded, it can't be decoded without knowing the index. So the index is sent first, followed by the data. - - -SUBTITLE BackupContext - -The context of the current connection, and the object which modifies the store. - -Maintains a cache of directories, to avoid reading them continuously, and keeps a track of a BackupStoreInfo object which is written back periodiocally. - - -SUBTITLE BackupStoreDaemon - -A ServerTLS daemon which forks off a separate housekeeping process as it starts up. - -Handling connections is delegated to a Protocol implementation. - - -SUBTITLE BackupCommands.cpp - -Implementation of all the commands. Work which requires writing is handled in the context, read only commands mainly in this file. - - -SUBTITLE HousekeepStoreAccount - -A class which performs housekeeping on a single account. - Deleted: box/trunk/docs/api-notes/backup/encrypt_rsync.txt =================================================================== --- box/trunk/docs/api-notes/backup/encrypt_rsync.txt 2009-03-28 15:41:09 UTC (rev 2477) +++ box/trunk/docs/api-notes/backup/encrypt_rsync.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -1,66 +0,0 @@ -TITLE Encrypted rsync algorithm - -The backup system uses a modified version of the rsync algorithm. A description of the plain algorithm can be found here: - - http://samba.anu.edu.au/rsync/tech_report/ - -The algorithm is modified to allow the server side to be encrypted, yet still benefit from the reduced bandwidth usage. For a single file transfer, the result will be only slightly less efficient than plain rsync. For a backup of a large directory, the overall bandwidth may be less due to the way the backup client daemon detects changes. - -This document assumes you have read the rsync document. - -The code is in lib/backupclient/BackupStoreFile*.*. - - -SUBTITLE Blocks - -Each file is broken up into small blocks. These are individually compressed and encrypted, and have an entry in an index which contains, encrypted, it's weak and strong checksums and decoded plaintext size. This is all done on the client. - -Why not just encrypt the file, and use the standard rsync algorithm? - -1) Compression cannot be used, since encryption turns the file into essentially random data. This is not very compressible. - -2) Any modification to the file will result in all data after that in the file having different ciphertext (in any cipher mode we might want to use). Therefore the rsync algorithm will only be able to detect "same" blocks up until the first modification. This significantly reduces the effectiveness of the process. - -Note that blocks are not all the same size. The last block in the file is unlikely to be a full block, and if data is inserted which is not a integral multiple of the block size, odd sized blocks need to be created. This is because the server cannot reassemble the blocks, because the contents are opaque to the server. - - -SUBTITLE Modifed algorithm - -To produce a list of the changes to send the new version, the client requests the block index of the file. This is the same step as requesting the weak and strong checksums from the remote side with rsync. - -The client then decrypts the index, and builds a list of the 8 most used block sizes above a certain threshold size. - -The new version of the file is then scanned in exactly the same way as rsync for these 8 block sizes. If a block is found, then it is added to a list of found blocks, sorted by position in the file. If a block has already been found at that position, then the old entry is only replaced by the new entry if the new entry is a "better" (bigger) match. - -The block size covering the biggest file area is searched first, so that most of the file can be skipped over after the first pass without expensive checksumming. - -A "recipe" is then built from the found list, by trivially discarding overlapping blocks. Each entry consists of a number of bytes of "new" data, a block start number, and a number of blocks from the old file. The data is stored like this as a memory optimisation, assuming that files mostly stay the same rather than having all their blocks reordered. - -The file is then encoded, with new data being sent as blocks of data, and references to blocks in the old file. The new index is built completely, as the checksums and size need to be rencrypted to match their position in the index. - - -SUBTITLE Combination on server - -The "diff" which is sent from the client is assembled into a full file on the server, simply by adding in blocks from the old file where they are specified in the block index. - - -SUBTITLE Storage on server - -Given that the server will in general store several versions of a file, combining old and new files to form a new file is not terribly efficient on storage space. Particularly for large multi-Gb database files. - -An alternative scheme is outlined below, however, it is significantly more complex to implement, and so is not implemented in this version. - -1) In the block index of the files, store the file ID of the file which each block is source from. This allows a single file to reference blocks from many files. - -2) When the file is downloaded, the server combines the blocks from all the files into a new file as it is streamed to the client. (This is not particuarly complicated to do.) - -This all sounds fine, until housekeeping is considered. Old versions need to be deleted, without losing any blocks necessary for future versions. - -Instead of just deleting a file, the server works out which blocks are still required, and rebuilds the file omitting those blocks which aren't required. - -This complicates working out how much space a file will release when it is "deleted", and indeed, adds a whole new level of complexity to the housekeeping process. (And the tests!) - -The directory structure will need an additional flag, "Partial file", which specifies that the entry cannot be built as previous blocks are no longer available. Entries with this flag should never be sent to the client. - - - Deleted: box/trunk/docs/api-notes/backup/lib_backupclient.txt =================================================================== --- box/trunk/docs/api-notes/backup/lib_backupclient.txt 2009-03-28 15:41:09 UTC (rev 2477) +++ box/trunk/docs/api-notes/backup/lib_backupclient.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -1,46 +0,0 @@ -TITLE lib/backupclient - -Classes used on the store and on the server. - -See documentation in the files for more details. - - -SUBTITLE BackupStoreDirectory - -The directory listing class, containing a number of entries, representing files. - - -SUBTITLE BackupStoreFile - -Handles compressing and encrypting files, and decoding files downloaded from the server. - - -SUBTITLE BackupStoreFilename - -An encrypted filename. - - -SUBTITLE BackupStoreFilenameClear - -Derived from BackupStoreFilename, but with the ability to encrypt and decrypt filenames. Client side only. - - -SUBTITLE BackupClientFileAttributes - -Only used on the client -- the server treats attributes as blocks of opaque data. - -This reads attributes from files on discs, stores them, encrypts them, and applies them to new files. - -Also has a static function to generate filename attribute hashes given a struct stat and the filename. - - -SUBTITLE BackupClientRestore - -Routines to restore files from the server onto the client filesystem. - - -SUBTITLE BackupClientCryptoKeys - -This reads the key material from disc, and sets up the crypto for storing files, attributes and directories. - - Deleted: box/trunk/docs/api-notes/backup/lib_backupstore.txt =================================================================== --- box/trunk/docs/api-notes/backup/lib_backupstore.txt 2009-03-28 15:41:09 UTC (rev 2477) +++ box/trunk/docs/api-notes/backup/lib_backupstore.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -1,30 +0,0 @@ -TITLE lib/backupstore - -Classes which are shared amongst the server side store utilities, bbstored and bbstoreaccounts. Note also depends on lib/backupclient, as a lot of code is shared between the client and server. - - -SUBTITLE BackupStoreAccountDatabase - -A simple implementation of an account database. This will be replaced with a more suitable implementation. - - -SUBTITLE BackupStoreAccounts - -An interface to the account database, and knowledge of how to initialise an account on disc. - - -SUBTITLE BackupStoreConfigVerify - -The same configuration file is used by all the utilities. This is the Configuration verification structure for this file. - - -SUBTITLE BackupStoreInfo - -The "header" information about an account, specifying current disc usage, space limits, etc. - - -SUBTITLE StoreStructure - -Functions specifying how the files are laid out on disc in the store. - - Deleted: box/trunk/docs/api-notes/backup/win32_build_on_cygwin_using_mingw.txt =================================================================== --- box/trunk/docs/api-notes/backup/win32_build_on_cygwin_using_mingw.txt 2009-03-28 15:41:09 UTC (rev 2477) +++ box/trunk/docs/api-notes/backup/win32_build_on_cygwin_using_mingw.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -1,53 +0,0 @@ -How to build Box Backup on Win32 using Cygwin and MinGW -By Chris Wilson, 2007-05-26 - -(To read this document online with better formatting, browse to: -http://www.boxbackup.org/trac/wiki/CompileWithMinGW) - -Start by installing Cygwin on your Windows machine [http://www.cygwin.org]. -Make sure to select the following packages during installation: - -* Devel/gcc-mingw -* Devel/gcc-mingw-core -* Devel/gcc-mingw-g++ -* Mingw/mingw-zlib - -If you already have Cygwin installed, please re-run the installer and -ensure that those packages are installed. - -Download OpenSSL from -[http://www.openssl.org/source/openssl-0.9.7i.tar.gz] - -Open a Cygwin shell, and unpack OpenSSL: - - tar xzvf openssl-0.9.7i.tar.gz - -Configure OpenSSL for MinGW compilation, and build and install it: - - cd openssl-0.9.7i - ./Configure --prefix=/usr/i686-pc-mingw32/ mingw - make - make install - -Download PCRE from -[http://prdownloads.sourceforge.net/pcre/pcre-6.3.tar.bz2?download] - -Open a Cygwin shell, and unpack PCRE: - - tar xjvf pcre-6.3.tar.bz2 - -Configure PCRE for MinGW compilation, and build and install it: - - cd pcre-6.3 - export CFLAGS="-mno-cygwin" - ./configure - make winshared - cp .libs/libpcre.a .libs/libpcreposix.a /usr/lib/mingw - cp pcreposix.h /usr/include/mingw - -Now unpack the Box Backup sources, enter the source directory, -and configure like this: - - ./infrastructure/mingw/configure.sh - make - Deleted: box/trunk/docs/api-notes/backup/win32_build_on_linux_using_mingw.txt =================================================================== --- box/trunk/docs/api-notes/backup/win32_build_on_linux_using_mingw.txt 2009-03-28 15:41:09 UTC (rev 2477) +++ box/trunk/docs/api-notes/backup/win32_build_on_linux_using_mingw.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -1,108 +0,0 @@ -How to build Box Backup for Windows (Native) on Linux using MinGW -By Chris Wilson, 2005-12-07 - -Install the MinGW cross-compiler for Windows: - -- Debian and Ubuntu users can "apt-get install mingw32" -- Fedora and SuSE users can download RPM packages from - [http://mirzam.it.vu.nl/mingw/] - -You will need to know the prefix used by the cross-compiler executables. -It will usually be something like "ix86-mingw32*-". All the binaries in the -cross-compiler package will start with this prefix. The documentation below -assumes that it is "i386-mingw32-". Adjust to taste. - -You will also need to install Wine and the Linux kernel "binary formats" -(binfmt) support, so that you can run Windows executables on Linux, -otherwise the configure scripts will not work properly with a cross-compiler. -On Ubuntu, run: - - apt-get install wine binfmt-support - /etc/init.d/binfmt-support start - -Start by downloading Zlib from [http://www.zlib.net/], unpack and enter -source directory: - - export CC=i386-mingw32-gcc - export AR="i386-mingw32-ar rc" - export RANLIB="i386-mingw32-ranlib" - ./configure - make - make install prefix=/usr/local/i386-mingw32 - -Download OpenSSL 0.9.8b from -[http://www.openssl.org/source/openssl-0.9.8b.tar.gz] - -Unpack and configure: - - tar xzvf openssl-0.9.8b.tar.gz - cd openssl-0.9.8b - ./Configure --prefix=/usr/local/i386-mingw32 mingw - make makefile.one - wget http://www.boxbackup.org/svn/box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch - patch -p1 < openssl-0.9.8b-mingw-cross.patch - make -f makefile.one - make -f makefile.one install - -Download PCRE from -[http://prdownloads.sourceforge.net/pcre/pcre-6.3.tar.bz2?download] - -Unpack: - - tar xjvf pcre-6.3.tar.bz2 - cd pcre-6.3 - -Configure and make: - - export AR=i386-mingw32-ar - ./configure --host=i386-mingw32 --prefix=/usr/local/i386-mingw32/ - make winshared - -If you get this error: - - ./dftables.exe pcre_chartables.c - /bin/bash: ./dftables.exe: cannot execute binary file - make: *** [pcre_chartables.c] Error 126 - -then run: - - wine ./dftables.exe pcre_chartables.c - make winshared - -to complete the build. Finally: - - cp .libs/libpcre.a /usr/local/i386-pc-mingw32/lib - cp .libs/libpcreposix.a /usr/local/i386-pc-mingw32/lib - cp pcreposix.h /usr/local/i386-pc-mingw32/include - -You will need to find a copy of mingwm10.dll that matches your cross-compiler. -Most MinGW distributions should come with it. On Debian and Ubuntu, for some -bizarre reason, you'll find it compressed as -/usr/share/doc/mingw32-runtime/mingwm10.dll.gz, in which case you'll -have to un-gzip it with "gzip -d". Copy it to a known location, e.g. -/usr/local/i386-mingw32/bin. - -Download and extract Box Backup, and change into the base directory, -e.g. boxbackup-0.11rc2. Change the path to mingwm10.dll in parcels.txt to -match where you found or installed it. - -Now configure Box with: - - ./configure --host=i386-mingw32 \ - CXXFLAGS="-mthreads -I/usr/local/i386-mingw32/include" \ - LDFLAGS=" -mthreads -L/usr/local/i386-mingw32/lib" \ - LIBS="-lcrypto -lws2_32 -lgdi32" - make - -or, if that fails, try this: - - export CXX="i386-mingw32-g++" - export AR=i386-mingw32-ar - export RANLIB=i386-mingw32-ranlib - export CFLAGS="-mthreads" - export CXXFLAGS="-mthreads" - export LDFLAGS="-mthreads" - export LIBS="-lcrypto -lws2_32 -lgdi32" - (if you don't have a "configure" file, run "./bootstrap") - ./configure --target=i386-mingw32 - make CXX="$CXX" AR="$AR" RANLIB="$RANLIB" WINDRES="i386-mingw32-windres" Deleted: box/trunk/docs/api-notes/backup/windows_porting.txt =================================================================== --- box/trunk/docs/api-notes/backup/windows_porting.txt 2009-03-28 15:41:09 UTC (rev 2477) +++ box/trunk/docs/api-notes/backup/windows_porting.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -1,100 +0,0 @@ -TITLE Notes on porting the backup client to Windows - -It should be relatively easy to port the backup client (bbackupd) to Windows. However, the server relies on unlink() behaviour of the UNIX filesystem which is different on the Windows platform, so it will not be so easy to port. - -An installation of perl is required to build the system. The ActiveState port is the easiest to install. - - -SUBTITLE Build environment - -The build environment generation script, makebuildenv.pl, uses perl to scan all the files and generate makefiles. It's rather orientated towards UNIX systems with gcc. - -Probably the easiest way to handle this is to detect the Windows platform, set up a few variables appropriately (in BoxPlatform.pm) and then post-process the generated Makefiles to mould them into something more handy for the MS Windows compiler and toolchain. - -The script itself is a bit messy. It was fine at first, but then the multi-platform thing got in the way. I do intend to rewrite it at some point in the future. - -Make sure your new version defines PLATFORM_WIN32 on the compile line. - -All files #include "Box.h" as the first include file. Use this for pre-compiled headers. Edit BoxPlatform.h to include the Windows headers required, and include a PLATFORM_WIN32 section. The easiest start will be to leave this blank, apart from typedefs for the basic types and any "not supported" #defines you can find. - -Boring bits of the code, such as exceptions and protocol definitions, are autogenerated using perl scripts. This code should be portable without modification. - -I have tried to avoid the things I know won't work with the MS compiler, so hopefully the code will be fairly clean. However, it might be a little easier to use the MinGW compiler [ http://www.mingw.org/ ] just to be consistent with the UNIX version. But I hope this won't be necessary. - -You'll need the latest version of OpenSSL. This was slightly difficult to get to compile last time I tried -- especially if you're determined to use the optimised assembler version. The main difficulty was getting a version which would link properly with the options used in my project, the default libraries selected got in the way. - - -SUBTITLE Porting as UNIX emulation - -Since the daemon uses so few UNIX system calls and with a limited set of options, it seems to make sense to port it by writing emulations of these functions. It's probably nicest to create a lib/win32 directory, and populate this with header files corresponding to the UNIX header files used. These just contain inline functions which map the UNIX calls to Win32 calls. - -File/socket handles may have to be translated. -1 is used as a failure return value and by the code internally to mark an invalid socket handle. (0 is a valid socket handle) - -Of course, some bits of code aren't relevant, so will just be #ifdefed out, or replaced. But this should be minimal. (Only perhaps the small bit relating to filesystem structure -- there aren't really mount points as such.) - - -SUBTITLE File tracking - -The daemon uses the inode number of a file to keep track of files and directories, so when they're renamed they can be moved efficiently on the store. Some unique (per filesystem) number will have to be found and used instead. - -It uses the Berkeley DB to store these on disc. It's likely another storage system will need to be used. (It just has to map the file's unique number into to a 8 byte struct.) - -There is a in-memory implementation for platforms which don't support Berkeley DB, but this isn't so good when the daemon has to be restarted as all the tracking is lost. But it's an easy start. - - -SUBTITLE Expected filesystem behaviour - -File and directories have (at least) two modification times, for contents and attributes. - -For files, the contents modification time must change when the contents change, and the attributes time when the attributes change (and may change when the contents change too.) - -For directories, the contents modification time must change when files or directories are deleted or added. If it changes any more frequently than this, then the client will be slightly less efficient -- it will download the store's directory listing whenever this time changes. The attributes modification time is less important, as the actual attributes are compared and only uploaded if different. - - -SUBTITLE Attributes - -Attributes means file modification times, flags, and filesystem permissions. - -The BackupClientFileAttribute class will need to be extended. Allocate another "attribute type" for the Win32 attributes, and then serialise it in a compatible way -- put your new attribute type in the header, and then a serialised network byte order structure in the rest. The different size of block is handled for you, and the server never looks inside. - -Add code so that under UNIX, Win32 attributes are ignored, and UNIX attributes under Win32. - -It's probably not necessary to worry too much about these for the first version. Not many people seem to use these attributes anyway. - - -SUBTITLE Times - -The system uses it's own 64 bit time type -- see BoxTime.h. Everything is translated to this from the various different system time types, and calculated and stored internally in this form. - - -SUBTITLE Daemon as a Service - -The client is derived from the Daemon class, which implements a daemon. The interface is simple, and it shouldn't be hard to write a compatible class which implements a Windows Service instead. - -Or cheat and run it as a Win32 application. - -Note that the daemon expects to be able to read every file it wants, and will abort a scan and upload run if it gets an error. The daemon must therefore be run with sufficient privileges. It runs as root under UNIX. - - -SUBTITLE Command Socket - -The backup daemon accepts commands from bbackupctl through a UNIX domain socket. When a connection is made, the user ID of the connecting process is checked to see if it's the same user ID as the daemon is running under. - -This may not have any exact analogue under Win32, so another communications scheme may have to be devised. - -This is only actually necessary if the client is to be run in snapshot mode. It can be safely left unimplemented if snapshot mode is not required, or the prompts for it to sync with the server are implemented some other way. - - -SUBTITLE NTFS streams - -If you want to back up NTFS streams, then a generic solution should probably be defined, so that the Mac OS X resource forks can be backed up with the same mechanism. - - -SUBTITLE Source code - -I work on a slightly different version of the source files. A make distribution script adds the license header and removes private sections of code. This means submitted diffs need a slight bit of translation. - - - - - Copied: box/trunk/docs/api-notes/backup_encryption.txt (from rev 2474, box/trunk/docs/api-notes/backup/backup_encryption.txt) =================================================================== --- box/trunk/docs/api-notes/backup_encryption.txt (rev 0) +++ box/trunk/docs/api-notes/backup_encryption.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,109 @@ +TITLE Encryption in the backup system + +This document explains how everything is encrypted in the backup system, and points to the various functions which need reviewing to ensure they do actually follow this scheme. + + +SUBTITLE Security objectives + +The crpyto system is designed to keep the following things secret from an attacker who has full access to the server. + +* The names of the files and directories +* The contents of files and directories +* The exact size of files + +Things which are not secret are + +* Directory heirarchy and number of files in each directory +* How the files change over time +* Approximate size of files + + +SUBTITLE Keys + +There are four separate keys used: + +* Filename +* File attributes +* File block index +* File data + +and an additional secret for file attribute hashes. + +The Cipher is Blowfish in CBC mode in most cases, except for the file data. All keys are maximum length 448 bit keys, since the key size only affects the setup time and this is done very infrequently. + +The file data is encrypted with AES in CBC mode, with a 256 bit key (max length). Blowfish is used elsewhere because the larger block size of AES, while more secure, would be terribly space inefficient. Note that Blowfish may also be used when older versions of OpenSSL are in use, and for backwards compatibility with older versions. + +The keys are generated using "openssl rand", and a 1k file of key material is stored in /etc/box/bbackupd. The configuration scripts make this readable only by root. + +Code for review: BackupClientCryptoKeys_Setup() +in lib/backupclient/BackupClientCryptoKeys.cpp + + +SUBTITLE Filenames + +Filenames need to be secret from the attacker, but they need to be compared on the server so it can determine whether or not is it a new version of an old file. + +So, the same Initialisation Vector is used for every single filename, so the same filename encrypted twice will have the same binary representation. + +Filenames use standard PKCS padding implemented by OpenSSL. They are proceeded by two bytes of header which describe the length, and the encoding. + +Code for review: BackupStoreFilenameClear::EncryptClear() +in lib/backupclient/BackupStoreFilenameClear.cpp + + +SUBTITLE File attributes + +These are kept secret as well, since they reveal information. Especially as they contain the target name of symbolic links. + +To encrypt, a random Initialisation Vector is choosen. This is stored first, followed by the attribute data encrypted with PKCS padding. + +Code for review: BackupClientFileAttributes::EncryptAttr() +in lib/backupclient/BackupClientFileAttributes.cpp + + +SUBTITLE File attribute hashes + +To detect and update file attributes efficiently, the file status change time is not used, as this would give suprious results and result in unnecessary updates to the server. Instead, a hash of user id, group id, and mode is used. + +To avoid revealing details about attributes + +1) The filename is added to the hash, so that an attacker cannot determine whether or not two files have identical attributes + +2) A secret is added to the hash, so that an attacker cannot compare attributes between accounts. + +The hash used is the first 64 bits of an MD5 hash. + + +SUBTITLE File block index + +Files are encoded in blocks, so that the rsync algorithm can be used on them. The data is compressed first before encryption. These small blocks don't give the best possible compression, but there is no alternative because the server can't see their contents. + +The file contains a number of blocks, which contain among other things + +* Size of the block when it's not compressed +* MD5 checksum of the block +* RollingChecksum of the block + +We don't want the attacker to know the size, so the first is bad. (Because of compression and padding, there's uncertainty on the size.) + +When the block is only a few bytes long, the latter two reveal it's contents with only a moderate amount of work. So these need to be encrypted. + +In the header of the index, a 64 bit number is chosen. The sensitive parts of the block are then encrypted, without padding, with an Initialisation Vector of this 64 bit number + the block index. + +If a block from an previous file is included in a new version of a file, the same checksum data will be encrypted again, but with a different IV. An eavesdropper will be able to easily find out which data has been re-encrypted, but the plaintext is not revealed. + +Code for review: BackupStoreFileEncodeStream::Read() (IV base choosen about half-way through) +BackupStoreFileEncodeStream::EncodeCurrentBlock() (encrypt index entry) +in lib/backupclient/BackupStoreFileEncodeStream.cpp + + +SUBTITLE File data + +As above, the first is split into chunks and compressed. + +Then, a random initialisation vector is chosen, stored first, followed by the compressed file data encrypted using PKCS padding. + +Code for review: BackupStoreFileEncodeStream::EncodeCurrentBlock() +in lib/backupclient/BackupStoreFileEncodeStream.cpp + + Copied: box/trunk/docs/api-notes/bin_bbackupd.txt (from rev 2474, box/trunk/docs/api-notes/backup/bin_bbackupd.txt) =================================================================== --- box/trunk/docs/api-notes/bin_bbackupd.txt (rev 0) +++ box/trunk/docs/api-notes/bin_bbackupd.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,88 @@ +TITLE bin/bbackupd + +The backup client daemon. + +This aims to maintain as little information as possible to record which files have been uploaded to the server, while minimising the amount of queries which have to be made to the server. + + +SUBTITLE Scanning + +The daemon is given a length of time, t, which files over this age should be uploaded to the server. This is to stop recently updated files being uploaded immediately to avoid uploading something repeatedly (on the assumption that if a file has been written, it is likely to be modified again shortly). + +It will scan the files at a configured interval, and connect to the server if it needs to upload files or make queries about files and directories. + +The scan interval is actually varied slightly between each run by adding a random number up to a 64th of the configured time. This is to reduce cyclic patterns of load on the backup servers -- otherwise if all the boxes are turned on at about 9am, every morning at 9am there will be a huge spike in load on the server. + +Each scan chooses a time interval, which ends at the current time - t. This will be from 0 to current time - t on the first run, then the next run takes the start time as the end time of the previous run. The scan is only performed if the difference between the start and end times is greater or equal to t. + +For each configured location, the client scans the directories on disc recursively. + +For each directory + +* If the directory has never been scanned before (in this invocation of the daemon) or the modified time on the directory is not that recorded, the listing on the server is downloaded. + +* For each file, if it's modified time is within the time period, it is uploaded. If the directory has been downloaded, it is compared against that, and only uploaded if it's changed. + +* Find all the new files, and upload them if they lie within the time interval. + +* Recurse to sub directories, creating them on the server if necessary. + +Hence, the first time it runs, it will download and compare the entries on the disc to those on the server, but in future runs it will use the file and directory modification times to work out if there is anything which needs uploading. + +If there aren't any changes, it won't even need to connect to the server. + +There are some extra details which allow this to work reliably, but they are documented in the source. + + +SUBTITLE File attributes + +The backup client will update the file attributes on files as soon as it notices they are changed. It records most of the details from stat(), but only a few can be restored. Attributes will only be considered changed if the user id, group id or mode is changed. Detection is by a 64 bit hash, so detection is strictly speaking probablistic. + + +SUBTITLE Encryption + +All the user data is encrypted. There is a separate file, backup_encryption.txt which describes this, and where in the code to look to verify it works as described. + + +SUBTITLE Tracking files and directories + +Renaming files is a difficult problem under this minimal data scanning scheme, because you don't really know whether a file has been renamed, or another file deleted and new one created. + +The solution is to keep (on disc) a map of inode numbers to server object IDs for all directories and files over a certain user configurable threshold. Then, when a new file is discovered, it is first checked to see if it's in this map. If so, a rename is considered, which will take place if the local object corresponding to the name of the tracked object doesn't exist any more. + +Because of the renaming requirement, deletions of objects from the server are recorded and delayed until the end of the scan. + + +SUBTITLE Running out of space + +If the store server indicates on login to the backup client, it will scan, but not upload anything nor adjust it's internal stored details of the local objects. However, deletions and renames happen. + +This is to allow deletions to still work and reduce the amount of storage space used on the server, in the hope that in the future there will be enough space. + +Just not doing anything would mean that one big file created and then deleted at the wrong time would stall the whole backup process. + + +SUBTITLE BackupDaemon + +This is the daemon class for the backup daemon. It handles setting up of all the objects, and implements calulcation of the time intervals for the scanning. + + +SUBTITLE BackupClientContext + +State information for the scans, including maintaining a connection to the store server if required. + + +SUBTITLE BackupClientDirectoryRecord + +A record of state of a directory on the local filesystem. Containing the recursive scanning function, which is long and entertaining, but very necessary. It contains lots of comments which explain the exact details of what's going on. + + +SUBTITLE BackupClientInodeToIDMap + +A implementation of a map of inode number to object ID on the server. If Berkeley DB is available on the platform, it is stored on disc, otherwise there is an in memory version which isn't so good. + + + + + + Copied: box/trunk/docs/api-notes/bin_bbstored.txt (from rev 2474, box/trunk/docs/api-notes/backup/bin_bbstored.txt) =================================================================== --- box/trunk/docs/api-notes/bin_bbstored.txt (rev 0) +++ box/trunk/docs/api-notes/bin_bbstored.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,54 @@ +TITLE bin/bbstored + +The backup store daemon. + +Maintains a store of encrypted files, and every so often goes through deleting unnecessary data. + +Uses an implementation of Protocol to communicate with the backup client daemon. See bin/bbstored/backupprotocol.txt for details. + + +SUBTITLE Data storage + +The data is arranged as a set of objects within a RaidFile disc set. Each object has a 64 bit object ID, which is turned into a filename in a mildly complex manner which ensure that directories don't have too many objects in them, but there is a minimal number of nested directories. See StoreStructure::MakeObjectFilename in lib/backupstore/StoreStructure.cpp for more details. + +An object can be a directory or a file. Directories contain files and other directories. + +Files in directories are supersceded by new versions when uploaded, but the old versions are flagged as such. A new version has a different object ID to the old version. + +Every so often, a housekeeping process works out what can be deleted, and deletes unnecessary files to take them below the storage limits set in the store info file. + + +SUBTITLE Note about file storage and downloading + +There's one slight entertainment to file storage, in that the format of the file streamed depends on whether it's being downloaded or uploaded. + +The problem is that it contains an index of all the blocks. For efficiency in managing these blocks, they all need to be in the same place. + +Files are encoded and decoded as they are streamed to and from the server. With encoding, the index is only completely known at the end of the process, so it's sent last, and lives in the filesystem last. + +When it's downloaded, it can't be decoded without knowing the index. So the index is sent first, followed by the data. + + +SUBTITLE BackupContext + +The context of the current connection, and the object which modifies the store. + +Maintains a cache of directories, to avoid reading them continuously, and keeps a track of a BackupStoreInfo object which is written back periodiocally. + + +SUBTITLE BackupStoreDaemon + +A ServerTLS daemon which forks off a separate housekeeping process as it starts up. + +Handling connections is delegated to a Protocol implementation. + + +SUBTITLE BackupCommands.cpp + +Implementation of all the commands. Work which requires writing is handled in the context, read only commands mainly in this file. + + +SUBTITLE HousekeepStoreAccount + +A class which performs housekeeping on a single account. + Copied: box/trunk/docs/api-notes/encrypt_rsync.txt (from rev 2474, box/trunk/docs/api-notes/backup/encrypt_rsync.txt) =================================================================== --- box/trunk/docs/api-notes/encrypt_rsync.txt (rev 0) +++ box/trunk/docs/api-notes/encrypt_rsync.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,66 @@ +TITLE Encrypted rsync algorithm + +The backup system uses a modified version of the rsync algorithm. A description of the plain algorithm can be found here: + + http://samba.anu.edu.au/rsync/tech_report/ + +The algorithm is modified to allow the server side to be encrypted, yet still benefit from the reduced bandwidth usage. For a single file transfer, the result will be only slightly less efficient than plain rsync. For a backup of a large directory, the overall bandwidth may be less due to the way the backup client daemon detects changes. + +This document assumes you have read the rsync document. + +The code is in lib/backupclient/BackupStoreFile*.*. + + +SUBTITLE Blocks + +Each file is broken up into small blocks. These are individually compressed and encrypted, and have an entry in an index which contains, encrypted, it's weak and strong checksums and decoded plaintext size. This is all done on the client. + +Why not just encrypt the file, and use the standard rsync algorithm? + +1) Compression cannot be used, since encryption turns the file into essentially random data. This is not very compressible. + +2) Any modification to the file will result in all data after that in the file having different ciphertext (in any cipher mode we might want to use). Therefore the rsync algorithm will only be able to detect "same" blocks up until the first modification. This significantly reduces the effectiveness of the process. + +Note that blocks are not all the same size. The last block in the file is unlikely to be a full block, and if data is inserted which is not a integral multiple of the block size, odd sized blocks need to be created. This is because the server cannot reassemble the blocks, because the contents are opaque to the server. + + +SUBTITLE Modifed algorithm + +To produce a list of the changes to send the new version, the client requests the block index of the file. This is the same step as requesting the weak and strong checksums from the remote side with rsync. + +The client then decrypts the index, and builds a list of the 8 most used block sizes above a certain threshold size. + +The new version of the file is then scanned in exactly the same way as rsync for these 8 block sizes. If a block is found, then it is added to a list of found blocks, sorted by position in the file. If a block has already been found at that position, then the old entry is only replaced by the new entry if the new entry is a "better" (bigger) match. + +The block size covering the biggest file area is searched first, so that most of the file can be skipped over after the first pass without expensive checksumming. + +A "recipe" is then built from the found list, by trivially discarding overlapping blocks. Each entry consists of a number of bytes of "new" data, a block start number, and a number of blocks from the old file. The data is stored like this as a memory optimisation, assuming that files mostly stay the same rather than having all their blocks reordered. + +The file is then encoded, with new data being sent as blocks of data, and references to blocks in the old file. The new index is built completely, as the checksums and size need to be rencrypted to match their position in the index. + + +SUBTITLE Combination on server + +The "diff" which is sent from the client is assembled into a full file on the server, simply by adding in blocks from the old file where they are specified in the block index. + + +SUBTITLE Storage on server + +Given that the server will in general store several versions of a file, combining old and new files to form a new file is not terribly efficient on storage space. Particularly for large multi-Gb database files. + +An alternative scheme is outlined below, however, it is significantly more complex to implement, and so is not implemented in this version. + +1) In the block index of the files, store the file ID of the file which each block is source from. This allows a single file to reference blocks from many files. + +2) When the file is downloaded, the server combines the blocks from all the files into a new file as it is streamed to the client. (This is not particuarly complicated to do.) + +This all sounds fine, until housekeeping is considered. Old versions need to be deleted, without losing any blocks necessary for future versions. + +Instead of just deleting a file, the server works out which blocks are still required, and rebuilds the file omitting those blocks which aren't required. + +This complicates working out how much space a file will release when it is "deleted", and indeed, adds a whole new level of complexity to the housekeeping process. (And the tests!) + +The directory structure will need an additional flag, "Partial file", which specifies that the entry cannot be built as previous blocks are no longer available. Entries with this flag should never be sent to the client. + + + Copied: box/trunk/docs/api-notes/lib_backupclient.txt (from rev 2474, box/trunk/docs/api-notes/backup/lib_backupclient.txt) =================================================================== --- box/trunk/docs/api-notes/lib_backupclient.txt (rev 0) +++ box/trunk/docs/api-notes/lib_backupclient.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,46 @@ +TITLE lib/backupclient + +Classes used on the store and on the server. + +See documentation in the files for more details. + + +SUBTITLE BackupStoreDirectory + +The directory listing class, containing a number of entries, representing files. + + +SUBTITLE BackupStoreFile + +Handles compressing and encrypting files, and decoding files downloaded from the server. + + +SUBTITLE BackupStoreFilename + +An encrypted filename. + + +SUBTITLE BackupStoreFilenameClear + +Derived from BackupStoreFilename, but with the ability to encrypt and decrypt filenames. Client side only. + + +SUBTITLE BackupClientFileAttributes + +Only used on the client -- the server treats attributes as blocks of opaque data. + +This reads attributes from files on discs, stores them, encrypts them, and applies them to new files. + +Also has a static function to generate filename attribute hashes given a struct stat and the filename. + + +SUBTITLE BackupClientRestore + +Routines to restore files from the server onto the client filesystem. + + +SUBTITLE BackupClientCryptoKeys + +This reads the key material from disc, and sets up the crypto for storing files, attributes and directories. + + Copied: box/trunk/docs/api-notes/lib_backupstore.txt (from rev 2474, box/trunk/docs/api-notes/backup/lib_backupstore.txt) =================================================================== --- box/trunk/docs/api-notes/lib_backupstore.txt (rev 0) +++ box/trunk/docs/api-notes/lib_backupstore.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,30 @@ +TITLE lib/backupstore + +Classes which are shared amongst the server side store utilities, bbstored and bbstoreaccounts. Note also depends on lib/backupclient, as a lot of code is shared between the client and server. + + +SUBTITLE BackupStoreAccountDatabase + +A simple implementation of an account database. This will be replaced with a more suitable implementation. + + +SUBTITLE BackupStoreAccounts + +An interface to the account database, and knowledge of how to initialise an account on disc. + + +SUBTITLE BackupStoreConfigVerify + +The same configuration file is used by all the utilities. This is the Configuration verification structure for this file. + + +SUBTITLE BackupStoreInfo + +The "header" information about an account, specifying current disc usage, space limits, etc. + + +SUBTITLE StoreStructure + +Functions specifying how the files are laid out on disc in the store. + + Copied: box/trunk/docs/api-notes/raidfile/RaidFileRead.txt (from rev 2474, box/trunk/docs/api-notes/raidfile/lib_raidfile/RaidFileRead.txt) =================================================================== --- box/trunk/docs/api-notes/raidfile/RaidFileRead.txt (rev 0) +++ box/trunk/docs/api-notes/raidfile/RaidFileRead.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,14 @@ +CLASS RaidFileRead + +Read a raid file. + +IOStream interface, plus a few extras, including reading directories and checking that files exist. + + +FUNCTION RaidFileRead::Open + +Open a given raid file -- returns a pointer to a new RaidFileRead object. + +Note that one of two types could be returned, depending on the representation of the file. + + Copied: box/trunk/docs/api-notes/raidfile/RaidFileWrite.txt (from rev 2474, box/trunk/docs/api-notes/raidfile/lib_raidfile/RaidFileWrite.txt) =================================================================== --- box/trunk/docs/api-notes/raidfile/RaidFileWrite.txt (rev 0) +++ box/trunk/docs/api-notes/raidfile/RaidFileWrite.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,36 @@ +CLASS RaidFileWrite + +Interface to writing raidfiles. + +See IOStream interface. + +Some other useful functions are available, see h and cpp files. + + +FUNCTION RaidFileWrite::RaidFileWrite() + +The constructor takes the disc set number and filename of the file you're interested. + + +FUNCTION RaidFileWrite::Open() + +Open() opens the file for writing, and takes an "allow overwrite" flag. + + +FUNCTION RaidFileWrite::Commit() + +Commmit the file, and make it visible to RaidFileRead. If ConvertToRaidNow == true, it will be converted to raid file representation immediately. + +Setting it to false is not a good idea. Later on, it will tell a daemon to convert it in the background, but for now it simply won't be converted. + + +FUNCTION RaidFileWrite::Discard() + +Abort the creation/update. Equivalent to just deleting the object without calling Commit(). + + +FUNCTION RaidFileWrite::Delete() + +Delete a file -- don't need to Open() it first. + + Copied: box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt (from rev 2474, box/trunk/docs/api-notes/backup/win32_build_on_cygwin_using_mingw.txt) =================================================================== --- box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt (rev 0) +++ box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,53 @@ +How to build Box Backup on Win32 using Cygwin and MinGW +By Chris Wilson, 2007-05-26 + +(To read this document online with better formatting, browse to: +http://www.boxbackup.org/trac/wiki/CompileWithMinGW) + +Start by installing Cygwin on your Windows machine [http://www.cygwin.org]. +Make sure to select the following packages during installation: + +* Devel/gcc-mingw +* Devel/gcc-mingw-core +* Devel/gcc-mingw-g++ +* Mingw/mingw-zlib + +If you already have Cygwin installed, please re-run the installer and +ensure that those packages are installed. + +Download OpenSSL from +[http://www.openssl.org/source/openssl-0.9.7i.tar.gz] + +Open a Cygwin shell, and unpack OpenSSL: + + tar xzvf openssl-0.9.7i.tar.gz + +Configure OpenSSL for MinGW compilation, and build and install it: + + cd openssl-0.9.7i + ./Configure --prefix=/usr/i686-pc-mingw32/ mingw + make + make install + +Download PCRE from +[http://prdownloads.sourceforge.net/pcre/pcre-6.3.tar.bz2?download] + +Open a Cygwin shell, and unpack PCRE: + + tar xjvf pcre-6.3.tar.bz2 + +Configure PCRE for MinGW compilation, and build and install it: + + cd pcre-6.3 + export CFLAGS="-mno-cygwin" + ./configure + make winshared + cp .libs/libpcre.a .libs/libpcreposix.a /usr/lib/mingw + cp pcreposix.h /usr/include/mingw + +Now unpack the Box Backup sources, enter the source directory, +and configure like this: + + ./infrastructure/mingw/configure.sh + make + Copied: box/trunk/docs/api-notes/win32_build_on_linux_using_mingw.txt (from rev 2474, box/trunk/docs/api-notes/backup/win32_build_on_linux_using_mingw.txt) =================================================================== --- box/trunk/docs/api-notes/win32_build_on_linux_using_mingw.txt (rev 0) +++ box/trunk/docs/api-notes/win32_build_on_linux_using_mingw.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,108 @@ +How to build Box Backup for Windows (Native) on Linux using MinGW +By Chris Wilson, 2005-12-07 + +Install the MinGW cross-compiler for Windows: + +- Debian and Ubuntu users can "apt-get install mingw32" +- Fedora and SuSE users can download RPM packages from + [http://mirzam.it.vu.nl/mingw/] + +You will need to know the prefix used by the cross-compiler executables. +It will usually be something like "ix86-mingw32*-". All the binaries in the +cross-compiler package will start with this prefix. The documentation below +assumes that it is "i386-mingw32-". Adjust to taste. + +You will also need to install Wine and the Linux kernel "binary formats" +(binfmt) support, so that you can run Windows executables on Linux, +otherwise the configure scripts will not work properly with a cross-compiler. +On Ubuntu, run: + + apt-get install wine binfmt-support + /etc/init.d/binfmt-support start + +Start by downloading Zlib from [http://www.zlib.net/], unpack and enter +source directory: + + export CC=i386-mingw32-gcc + export AR="i386-mingw32-ar rc" + export RANLIB="i386-mingw32-ranlib" + ./configure + make + make install prefix=/usr/local/i386-mingw32 + +Download OpenSSL 0.9.8b from +[http://www.openssl.org/source/openssl-0.9.8b.tar.gz] + +Unpack and configure: + + tar xzvf openssl-0.9.8b.tar.gz + cd openssl-0.9.8b + ./Configure --prefix=/usr/local/i386-mingw32 mingw + make makefile.one + wget http://www.boxbackup.org/svn/box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch + patch -p1 < openssl-0.9.8b-mingw-cross.patch + make -f makefile.one + make -f makefile.one install + +Download PCRE from +[http://prdownloads.sourceforge.net/pcre/pcre-6.3.tar.bz2?download] + +Unpack: + + tar xjvf pcre-6.3.tar.bz2 + cd pcre-6.3 + +Configure and make: + + export AR=i386-mingw32-ar + ./configure --host=i386-mingw32 --prefix=/usr/local/i386-mingw32/ + make winshared + +If you get this error: + + ./dftables.exe pcre_chartables.c + /bin/bash: ./dftables.exe: cannot execute binary file + make: *** [pcre_chartables.c] Error 126 + +then run: + + wine ./dftables.exe pcre_chartables.c + make winshared + +to complete the build. Finally: + + cp .libs/libpcre.a /usr/local/i386-pc-mingw32/lib + cp .libs/libpcreposix.a /usr/local/i386-pc-mingw32/lib + cp pcreposix.h /usr/local/i386-pc-mingw32/include + +You will need to find a copy of mingwm10.dll that matches your cross-compiler. +Most MinGW distributions should come with it. On Debian and Ubuntu, for some +bizarre reason, you'll find it compressed as +/usr/share/doc/mingw32-runtime/mingwm10.dll.gz, in which case you'll +have to un-gzip it with "gzip -d". Copy it to a known location, e.g. +/usr/local/i386-mingw32/bin. + +Download and extract Box Backup, and change into the base directory, +e.g. boxbackup-0.11rc2. Change the path to mingwm10.dll in parcels.txt to +match where you found or installed it. + +Now configure Box with: + + ./configure --host=i386-mingw32 \ + CXXFLAGS="-mthreads -I/usr/local/i386-mingw32/include" \ + LDFLAGS=" -mthreads -L/usr/local/i386-mingw32/lib" \ + LIBS="-lcrypto -lws2_32 -lgdi32" + make + +or, if that fails, try this: + + export CXX="i386-mingw32-g++" + export AR=i386-mingw32-ar + export RANLIB=i386-mingw32-ranlib + export CFLAGS="-mthreads" + export CXXFLAGS="-mthreads" + export LDFLAGS="-mthreads" + export LIBS="-lcrypto -lws2_32 -lgdi32" + (if you don't have a "configure" file, run "./bootstrap") + ./configure --target=i386-mingw32 + make CXX="$CXX" AR="$AR" RANLIB="$RANLIB" WINDRES="i386-mingw32-windres" Copied: box/trunk/docs/api-notes/windows_porting.txt (from rev 2474, box/trunk/docs/api-notes/backup/windows_porting.txt) =================================================================== --- box/trunk/docs/api-notes/windows_porting.txt (rev 0) +++ box/trunk/docs/api-notes/windows_porting.txt 2009-03-28 15:51:04 UTC (rev 2478) @@ -0,0 +1,100 @@ +TITLE Notes on porting the backup client to Windows + +It should be relatively easy to port the backup client (bbackupd) to Windows. However, the server relies on unlink() behaviour of the UNIX filesystem which is different on the Windows platform, so it will not be so easy to port. + +An installation of perl is required to build the system. The ActiveState port is the easiest to install. + + +SUBTITLE Build environment + +The build environment generation script, makebuildenv.pl, uses perl to scan all the files and generate makefiles. It's rather orientated towards UNIX systems with gcc. + +Probably the easiest way to handle this is to detect the Windows platform, set up a few variables appropriately (in BoxPlatform.pm) and then post-process the generated Makefiles to mould them into something more handy for the MS Windows compiler and toolchain. + +The script itself is a bit messy. It was fine at first, but then the multi-platform thing got in the way. I do intend to rewrite it at some point in the future. + +Make sure your new version defines PLATFORM_WIN32 on the compile line. + +All files #include "Box.h" as the first include file. Use this for pre-compiled headers. Edit BoxPlatform.h to include the Windows headers required, and include a PLATFORM_WIN32 section. The easiest start will be to leave this blank, apart from typedefs for the basic types and any "not supported" #defines you can find. + +Boring bits of the code, such as exceptions and protocol definitions, are autogenerated using perl scripts. This code should be portable without modification. + +I have tried to avoid the things I know won't work with the MS compiler, so hopefully the code will be fairly clean. However, it might be a little easier to use the MinGW compiler [ http://www.mingw.org/ ] just to be consistent with the UNIX version. But I hope this won't be necessary. + +You'll need the latest version of OpenSSL. This was slightly difficult to get to compile last time I tried -- especially if you're determined to use the optimised assembler version. The main difficulty was getting a version which would link properly with the options used in my project, the default libraries selected got in the way. + + +SUBTITLE Porting as UNIX emulation + +Since the daemon uses so few UNIX system calls and with a limited set of options, it seems to make sense to port it by writing emulations of these functions. It's probably nicest to create a lib/win32 directory, and populate this with header files corresponding to the UNIX header files used. These just contain inline functions which map the UNIX calls to Win32 calls. + +File/socket handles may have to be translated. -1 is used as a failure return value and by the code internally to mark an invalid socket handle. (0 is a valid socket handle) + +Of course, some bits of code aren't relevant, so will just be #ifdefed out, or replaced. But this should be minimal. (Only perhaps the small bit relating to filesystem structure -- there aren't really mount points as such.) + + +SUBTITLE File tracking + +The daemon uses the inode number of a file to keep track of files and directories, so when they're renamed they can be moved efficiently on the store. Some unique (per filesystem) number will have to be found and used instead. + +It uses the Berkeley DB to store these on disc. It's likely another storage system will need to be used. (It just has to map the file's unique number into to a 8 byte struct.) + +There is a in-memory implementation for platforms which don't support Berkeley DB, but this isn't so good when the daemon has to be restarted as all the tracking is lost. But it's an easy start. + + +SUBTITLE Expected filesystem behaviour + +File and directories have (at least) two modification times, for contents and attributes. + +For files, the contents modification time must change when the contents change, and the attributes time when the attributes change (and may change when the contents change too.) + +For directories, the contents modification time must change when files or directories are deleted or added. If it changes any more frequently than this, then the client will be slightly less efficient -- it will download the store's directory listing whenever this time changes. The attributes modification time is less important, as the actual attributes are compared and only uploaded if different. + + +SUBTITLE Attributes + +Attributes means file modification times, flags, and filesystem permissions. + +The BackupClientFileAttribute class will need to be extended. Allocate another "attribute type" for the Win32 attributes, and then serialise it in a compatible way -- put your new attribute type in the header, and then a serialised network byte order structure in the rest. The different size of block is handled for you, and the server never looks inside. + +Add code so that under UNIX, Win32 attributes are ignored, and UNIX attributes under Win32. + +It's probably not necessary to worry too much about these for the first version. Not many people seem to use these attributes anyway. + + +SUBTITLE Times + +The system uses it's own 64 bit time type -- see BoxTime.h. Everything is translated to this from the various different system time types, and calculated and stored internally in this form. + + +SUBTITLE Daemon as a Service + +The client is derived from the Daemon class, which implements a daemon. The interface is simple, and it shouldn't be hard to write a compatible class which implements a Windows Service instead. + +Or cheat and run it as a Win32 application. + +Note that the daemon expects to be able to read every file it wants, and will abort a scan and upload run if it gets an error. The daemon must therefore be run with sufficient privileges. It runs as root under UNIX. + + +SUBTITLE Command Socket + +The backup daemon accepts commands from bbackupctl through a UNIX domain socket. When a connection is made, the user ID of the connecting process is checked to see if it's the same user ID as the daemon is running under. + +This may not have any exact analogue under Win32, so another communications scheme may have to be devised. + +This is only actually necessary if the client is to be run in snapshot mode. It can be safely left unimplemented if snapshot mode is not required, or the prompts for it to sync with the server are implemented some other way. + + +SUBTITLE NTFS streams + +If you want to back up NTFS streams, then a generic solution should probably be defined, so that the Mac OS X resource forks can be backed up with the same mechanism. + + +SUBTITLE Source code + +I work on a slightly different version of the source files. A make distribution script adds the license header and removes private sections of code. This means submitted diffs need a slight bit of translation. + + + + + From boxbackup-dev at boxbackup.org Sat Mar 28 15:52:28 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sat, 28 Mar 2009 15:52:28 +0000 (GMT) Subject: [Box Backup-commit] COMMIT r2479 - in box/trunk/docs: . docbook Message-ID: <20090328155228.4E7F432501F@www.boxbackup.org> Author: chris Date: 2009-03-28 15:52:28 +0000 (Sat, 28 Mar 2009) New Revision: 2479 Modified: box/trunk/docs/ box/trunk/docs/docbook/ Log: Update svn:ignore attributes for new docs layout. Property changes on: box/trunk/docs ___________________________________________________________________ Modified: svn:ignore - *.5 *.8 *.html bb-man.xsl box-html man ExceptionCodes.xml + man htmlguide Property changes on: box/trunk/docs/docbook ___________________________________________________________________ Added: svn:ignore + bb-man.xsl ExceptionCodes.xml From boxbackup-dev at boxbackup.org Sun Mar 29 14:31:15 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sun, 29 Mar 2009 14:31:15 +0100 (BST) Subject: [Box Backup-commit] COMMIT r2480 - box/trunk/lib/server Message-ID: <20090329133115.B331832501F@www.boxbackup.org> Author: chris Date: 2009-03-29 14:31:14 +0100 (Sun, 29 Mar 2009) New Revision: 2480 Modified: box/trunk/lib/server/Daemon.cpp Log: Hopefully fix compile error reported by Mirko on cross-compiler. Modified: box/trunk/lib/server/Daemon.cpp =================================================================== --- box/trunk/lib/server/Daemon.cpp 2009-03-28 15:52:28 UTC (rev 2479) +++ box/trunk/lib/server/Daemon.cpp 2009-03-29 13:31:14 UTC (rev 2480) @@ -689,10 +689,10 @@ BOX_NOTICE("Reloading configuration file: " << mConfigFileName); std::string errors; - std::auto_ptr pconfig = + std::auto_ptr pconfig( Configuration::LoadAndVerify( mConfigFileName.c_str(), - GetConfigVerify(), errors); + GetConfigVerify(), errors)); // Got errors? if(pconfig.get() == 0 || !errors.empty()) From boxbackup-dev at boxbackup.org Sun Mar 29 14:51:25 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sun, 29 Mar 2009 14:51:25 +0100 (BST) Subject: [Box Backup-commit] COMMIT r2481 - in box/trunk: bin/bbackupd bin/bbstored lib/backupclient lib/backupstore test/backupstore Message-ID: <20090329135125.61E0B32501F@www.boxbackup.org> Author: chris Date: 2009-03-29 14:51:24 +0100 (Sun, 29 Mar 2009) New Revision: 2481 Modified: box/trunk/bin/bbackupd/BackupClientDirectoryRecord.cpp box/trunk/bin/bbstored/HousekeepStoreAccount.cpp box/trunk/lib/backupclient/BackupStoreFilename.cpp box/trunk/lib/backupclient/BackupStoreFilename.h box/trunk/lib/backupclient/BackupStoreFilenameClear.cpp box/trunk/lib/backupclient/BackupStoreObjectDump.cpp box/trunk/lib/backupstore/BackupStoreCheck2.cpp box/trunk/test/backupstore/testbackupstore.cpp Log: Change type of BackupStoreFilename not to derive from std::string, so it can't accidentally be used as one. Fix use of encrypted filename in deleted file message, thanks to Kenny Millington for reporting. Modified: box/trunk/bin/bbackupd/BackupClientDirectoryRecord.cpp =================================================================== --- box/trunk/bin/bbackupd/BackupClientDirectoryRecord.cpp 2009-03-29 13:31:14 UTC (rev 2480) +++ box/trunk/bin/bbackupd/BackupClientDirectoryRecord.cpp 2009-03-29 13:51:24 UTC (rev 2481) @@ -1386,8 +1386,9 @@ // aren't actually deleted, as the whole state will be reset anyway. BackupClientDeleteList &rdel(rContext.GetDeleteList()); + BackupStoreFilenameClear clear(en->GetName()); std::string localName = MakeFullPath(rLocalPath, - en->GetName()); + clear.GetClearFilename()); // Delete this entry -- file or directory? if((en->GetFlags() & BackupStoreDirectory::Entry::Flags_File) != 0) Modified: box/trunk/bin/bbstored/HousekeepStoreAccount.cpp =================================================================== --- box/trunk/bin/bbstored/HousekeepStoreAccount.cpp 2009-03-29 13:31:14 UTC (rev 2480) +++ box/trunk/bin/bbstored/HousekeepStoreAccount.cpp 2009-03-29 13:51:24 UTC (rev 2481) @@ -349,7 +349,8 @@ // Add files to the list of potential deletions // map to count the distance from the mark - std::map, int32_t> markVersionAges; + typedef std::pair version_t; + std::map markVersionAges; // map of pair (filename, mark number) -> version age // NOTE: use a reverse iterator to allow the distance from mark stuff to work @@ -367,7 +368,10 @@ // Work out ages of this version from the last mark int32_t enVersionAge = 0; - std::map, int32_t>::iterator enVersionAgeI(markVersionAges.find(std::pair(en->GetName(), en->GetMarkNumber()))); + std::map::iterator enVersionAgeI( + markVersionAges.find( + version_t(en->GetName().GetEncodedFilename(), + en->GetMarkNumber()))); if(enVersionAgeI != markVersionAges.end()) { enVersionAge = enVersionAgeI->second + 1; @@ -375,7 +379,7 @@ } else { - markVersionAges[std::pair(en->GetName(), en->GetMarkNumber())] = enVersionAge; + markVersionAges[version_t(en->GetName().GetEncodedFilename(), en->GetMarkNumber())] = enVersionAge; } // enVersionAge is now the age of this version. Modified: box/trunk/lib/backupclient/BackupStoreFilename.cpp =================================================================== --- box/trunk/lib/backupclient/BackupStoreFilename.cpp 2009-03-29 13:31:14 UTC (rev 2480) +++ box/trunk/lib/backupclient/BackupStoreFilename.cpp 2009-03-29 13:51:24 UTC (rev 2481) @@ -37,7 +37,7 @@ // // -------------------------------------------------------------------------- BackupStoreFilename::BackupStoreFilename(const BackupStoreFilename &rToCopy) - : BackupStoreFilename_base(rToCopy) + : mEncryptedName(rToCopy.mEncryptedName) { } @@ -65,7 +65,7 @@ { bool ok = true; - if(size() < 2) + if(mEncryptedName.size() < 2) { // Isn't long enough to have a header ok = false; @@ -73,14 +73,14 @@ else { // Check size is consistent - unsigned int dsize = BACKUPSTOREFILENAME_GET_SIZE(*this); - if(dsize != size()) + unsigned int dsize = BACKUPSTOREFILENAME_GET_SIZE(this->mEncryptedName); + if(dsize != mEncryptedName.size()) { ok = false; } // And encoding is an accepted value - unsigned int encoding = BACKUPSTOREFILENAME_GET_ENCODING(*this); + unsigned int encoding = BACKUPSTOREFILENAME_GET_ENCODING(this->mEncryptedName); if(encoding < Encoding_Min || encoding > Encoding_Max) { ok = false; @@ -119,8 +119,8 @@ rProtocol.Read(data, dsize - 2); // assign to this string, storing the header and the extra data - assign(hdr, 2); - append(data.c_str(), data.size()); + mEncryptedName.assign(hdr, 2); + mEncryptedName.append(data.c_str(), data.size()); // Check it CheckValid(); @@ -141,7 +141,7 @@ { CheckValid(); - rProtocol.Write(c_str(), size()); + rProtocol.Write(mEncryptedName.c_str(), mEncryptedName.size()); } // -------------------------------------------------------------------------- @@ -177,7 +177,7 @@ buf[0] = hdr[0]; buf[1] = hdr[1]; // assign to this string, storing the header and the extra data - assign(buf, dsize); + mEncryptedName.assign(buf, dsize); } else { @@ -194,7 +194,7 @@ data[0] = hdr[0]; data[1] = hdr[1]; // assign to this string, storing the header and the extra data - assign(data, dsize); + mEncryptedName.assign(data, dsize); } // Check it @@ -216,7 +216,7 @@ { CheckValid(); - rStream.Write(c_str(), size()); + rStream.Write(mEncryptedName.c_str(), mEncryptedName.size()); } // -------------------------------------------------------------------------- @@ -242,7 +242,8 @@ // -------------------------------------------------------------------------- bool BackupStoreFilename::IsEncrypted() const { - return BACKUPSTOREFILENAME_GET_ENCODING(*this) != Encoding_Clear; + return BACKUPSTOREFILENAME_GET_ENCODING(this->mEncryptedName) != + Encoding_Clear; } @@ -250,8 +251,9 @@ // // Function // Name: BackupStoreFilename::SetAsClearFilename(const char *) -// Purpose: Sets this object to be a valid filename, but with a filename in the clear. -// Used on the server to create filenames when there's no way of encrypting it. +// Purpose: Sets this object to be a valid filename, but with a +// filename in the clear. Used on the server to create +// filenames when there's no way of encrypting it. // Created: 22/4/04 // // -------------------------------------------------------------------------- @@ -268,7 +270,7 @@ ASSERT(encoded.size() == toEncode.size() + 2); // Store the encoded string - assign(encoded); + mEncryptedName.assign(encoded); // Stuff which must be done EncodedFilenameChanged(); Modified: box/trunk/lib/backupclient/BackupStoreFilename.h =================================================================== --- box/trunk/lib/backupclient/BackupStoreFilename.h 2009-03-29 13:31:14 UTC (rev 2480) +++ box/trunk/lib/backupclient/BackupStoreFilename.h 2009-03-29 13:51:24 UTC (rev 2481) @@ -40,8 +40,11 @@ // Created: 2003/08/26 // // -------------------------------------------------------------------------- -class BackupStoreFilename : public BackupStoreFilename_base +class BackupStoreFilename /* : public BackupStoreFilename_base */ { +private: + std::string mEncryptedName; + public: BackupStoreFilename(); BackupStoreFilename(const BackupStoreFilename &rToCopy); @@ -71,8 +74,27 @@ Encoding_Max = 2 }; + const std::string& GetEncodedFilename() const + { + return mEncryptedName; + } + + bool operator==(const BackupStoreFilename& rOther) const + { + return mEncryptedName == rOther.mEncryptedName; + } + + bool operator!=(const BackupStoreFilename& rOther) const + { + return mEncryptedName != rOther.mEncryptedName; + } + protected: virtual void EncodedFilenameChanged(); + void SetEncodedFilename(const std::string &rEncoded) + { + mEncryptedName = rEncoded; + } }; // On the wire utilities for class and derived class Modified: box/trunk/lib/backupclient/BackupStoreFilenameClear.cpp =================================================================== --- box/trunk/lib/backupclient/BackupStoreFilenameClear.cpp 2009-03-29 13:31:14 UTC (rev 2480) +++ box/trunk/lib/backupclient/BackupStoreFilenameClear.cpp 2009-03-29 13:51:24 UTC (rev 2481) @@ -160,8 +160,8 @@ CheckValid(); // Decode the header - int size = BACKUPSTOREFILENAME_GET_SIZE(*this); - int encoding = BACKUPSTOREFILENAME_GET_ENCODING(*this); + int size = BACKUPSTOREFILENAME_GET_SIZE(GetEncodedFilename()); + int encoding = BACKUPSTOREFILENAME_GET_ENCODING(GetEncodedFilename()); // Decode based on encoding given in the header switch(encoding) @@ -169,7 +169,8 @@ case Encoding_Clear: BOX_TRACE("**** BackupStoreFilename encoded with " "Clear encoding ****"); - mClearFilename.assign(c_str() + 2, size - 2); + mClearFilename.assign(GetEncodedFilename().c_str() + 2, + size - 2); break; case Encoding_Blowfish: @@ -244,7 +245,7 @@ BACKUPSTOREFILENAME_MAKE_HDR(buffer, encSize, StoreAsEncoding); // Store the encoded string - assign((char*)buffer, encSize); + SetEncodedFilename(std::string((char*)buffer, encSize)); } @@ -258,8 +259,10 @@ // -------------------------------------------------------------------------- void BackupStoreFilenameClear::DecryptEncoded(CipherContext &rCipherContext) const { + const std::string& rEncoded = GetEncodedFilename(); + // Work out max size - int maxOutSize = rCipherContext.MaxOutSizeForInBufferSize(size()) + 4; + int maxOutSize = rCipherContext.MaxOutSizeForInBufferSize(rEncoded.size()) + 4; // Make sure encode/decode buffer has enough space EnsureEncDecBufferSize(maxOutSize); @@ -268,8 +271,8 @@ uint8_t *buffer = *spEncDecBuffer; // Decrypt - const char *str = c_str() + 2; - int sizeOut = rCipherContext.TransformBlock(buffer, sEncDecBufferSize, str, size() - 2); + const char *str = rEncoded.c_str() + 2; + int sizeOut = rCipherContext.TransformBlock(buffer, sEncDecBufferSize, str, rEncoded.size() - 2); // Assign to this mClearFilename.assign((char*)buffer, sizeOut); Modified: box/trunk/lib/backupclient/BackupStoreObjectDump.cpp =================================================================== --- box/trunk/lib/backupclient/BackupStoreObjectDump.cpp 2009-03-29 13:31:14 UTC (rev 2480) +++ box/trunk/lib/backupclient/BackupStoreObjectDump.cpp 2009-03-29 13:51:24 UTC (rev 2481) @@ -70,7 +70,7 @@ mAttributesModTime, mAttributes.GetSize()); // So repeated filenames can be illustrated, even though they can't be decoded - std::map nameNum; + std::map nameNum; int nameNumI = 0; // Dump items @@ -78,7 +78,7 @@ for(std::vector::const_iterator i(mEntries.begin()); i != mEntries.end(); ++i) { // Choose file name index number for this file - std::map::iterator nn(nameNum.find((*i)->GetName())); + std::map::iterator nn(nameNum.find((*i)->GetName().GetEncodedFilename())); int ni = nameNumI; if(nn != nameNum.end()) { @@ -86,7 +86,7 @@ } else { - nameNum[(*i)->GetName()] = nameNumI; + nameNum[(*i)->GetName().GetEncodedFilename()] = nameNumI; ++nameNumI; } @@ -124,7 +124,7 @@ (*i)->GetSizeInBlocks(), (*i)->GetAttributesHash(), (*i)->GetAttributes().GetSize(), - (*i)->GetName().size(), + (*i)->GetName().GetEncodedFilename().size(), ni, ((f & BackupStoreDirectory::Entry::Flags_File)?" file":""), ((f & BackupStoreDirectory::Entry::Flags_Dir)?" dir":""), @@ -173,7 +173,8 @@ // Read the next two objects BackupStoreFilename fn; fn.ReadFromStream(rFile, IOStream::TimeOutInfinite); - OutputLine(file, ToTrace, "Filename size: %d\n", fn.size()); + OutputLine(file, ToTrace, "Filename size: %d\n", + fn.GetEncodedFilename().size()); BackupClientFileAttributes attr; attr.ReadFromStream(rFile, IOStream::TimeOutInfinite); Modified: box/trunk/lib/backupstore/BackupStoreCheck2.cpp =================================================================== --- box/trunk/lib/backupstore/BackupStoreCheck2.cpp 2009-03-29 13:31:14 UTC (rev 2480) +++ box/trunk/lib/backupstore/BackupStoreCheck2.cpp 2009-03-29 13:51:24 UTC (rev 2481) @@ -750,7 +750,7 @@ // Records of things seen std::set idsEncountered; - std::set filenamesEncountered; + std::set filenamesEncountered; do { @@ -792,7 +792,7 @@ // Check to see if the name has already been encountered -- if not, then it // needs to have the old version flag set - if(filenamesEncountered.find((*i)->GetName()) != filenamesEncountered.end()) + if(filenamesEncountered.find((*i)->GetName().GetEncodedFilename()) != filenamesEncountered.end()) { // Seen before -- check old version flag set if(((*i)->GetFlags() & Entry::Flags_OldVersion) != Entry::Flags_OldVersion @@ -818,7 +818,7 @@ } // Remember filename - filenamesEncountered.insert((*i)->GetName()); + filenamesEncountered.insert((*i)->GetName().GetEncodedFilename()); } } } Modified: box/trunk/test/backupstore/testbackupstore.cpp =================================================================== --- box/trunk/test/backupstore/testbackupstore.cpp 2009-03-29 13:31:14 UTC (rev 2480) +++ box/trunk/test/backupstore/testbackupstore.cpp 2009-03-29 13:51:24 UTC (rev 2481) @@ -232,7 +232,8 @@ TEST_THAT(fn1 == fn3); // Check that it's been encrypted - TEST_THAT(fn2.find("name") == fn2.npos); + std::string name(fn2.GetEncodedFilename()); + TEST_THAT(name.find("name") == name.npos); // Bung it in a stream, get it out in a Clear filename { From boxbackup-dev at boxbackup.org Sun Mar 29 15:08:12 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Sun, 29 Mar 2009 15:08:12 +0100 (BST) Subject: [Box Backup-commit] COMMIT r2482 - box/trunk/lib/common Message-ID: <20090329140812.8E21332501F@www.boxbackup.org> Author: chris Date: 2009-03-29 15:08:12 +0100 (Sun, 29 Mar 2009) New Revision: 2482 Modified: box/trunk/lib/common/BoxTime.cpp box/trunk/lib/common/BoxTime.h box/trunk/lib/common/Logging.cpp box/trunk/lib/common/Timer.cpp Log: Add date to file logger, as requested by Kenny Millington. Modified: box/trunk/lib/common/BoxTime.cpp =================================================================== --- box/trunk/lib/common/BoxTime.cpp 2009-03-29 13:51:24 UTC (rev 2481) +++ box/trunk/lib/common/BoxTime.cpp 2009-03-29 14:08:12 UTC (rev 2482) @@ -53,7 +53,7 @@ return SecondsToBoxTime(time(0)); } -std::string FormatTime(box_time_t time, bool showMicros) +std::string FormatTime(box_time_t time, bool includeDate, bool showMicros) { std::ostringstream buf; @@ -68,8 +68,16 @@ if (localtime_r(&seconds, &tm_now) != NULL) #endif { - buf << std::setfill('0') << - std::setw(2) << tm_ptr->tm_hour << ":" << + buf << std::setfill('0'); + + if (includeDate) + { + buf << std::setw(4) << (tm_ptr->tm_year + 1900) << "-" << + std::setw(2) << (tm_ptr->tm_mon + 1) << "-" << + std::setw(2) << (tm_ptr->tm_mday) << " "; + } + + buf << std::setw(2) << tm_ptr->tm_hour << ":" << std::setw(2) << tm_ptr->tm_min << ":" << std::setw(2) << tm_ptr->tm_sec; Modified: box/trunk/lib/common/BoxTime.h =================================================================== --- box/trunk/lib/common/BoxTime.h 2009-03-29 13:51:24 UTC (rev 2481) +++ box/trunk/lib/common/BoxTime.h 2009-03-29 14:08:12 UTC (rev 2482) @@ -40,6 +40,7 @@ return Time; } -std::string FormatTime(box_time_t time, bool showMicros = false); +std::string FormatTime(box_time_t time, bool includeDate, + bool showMicros = false); #endif // BOXTIME__H Modified: box/trunk/lib/common/Logging.cpp =================================================================== --- box/trunk/lib/common/Logging.cpp 2009-03-29 13:51:24 UTC (rev 2481) +++ box/trunk/lib/common/Logging.cpp 2009-03-29 14:08:12 UTC (rev 2482) @@ -288,7 +288,7 @@ if (sShowTime) { - buf << FormatTime(GetCurrentBoxTime(), sShowTimeMicros); + buf << FormatTime(GetCurrentBoxTime(), false, sShowTimeMicros); buf << " "; } @@ -449,7 +449,7 @@ Logging::Remove(this); std::ostringstream buf; - buf << FormatTime(GetCurrentBoxTime(), false); + buf << FormatTime(GetCurrentBoxTime(), true, false); buf << " "; if (Level <= Log::FATAL) Modified: box/trunk/lib/common/Timer.cpp =================================================================== --- box/trunk/lib/common/Timer.cpp 2009-03-29 13:51:24 UTC (rev 2481) +++ box/trunk/lib/common/Timer.cpp 2009-03-29 14:08:12 UTC (rev 2482) @@ -357,7 +357,7 @@ else { BOX_TRACE(TIMER_ID "initialised for " << timeoutSecs << - " secs, to fire at " << FormatTime(mExpires, true)); + " secs, to fire at " << FormatTime(mExpires, false, true)); } #endif From boxbackup-dev at boxbackup.org Tue Mar 31 13:11:25 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 31 Mar 2009 13:11:25 +0100 (BST) Subject: [Box Backup-commit] COMMIT r2483 - box/trunk Message-ID: <20090331121125.83A9A32501F@www.boxbackup.org> Author: chris Date: 2009-03-31 13:11:24 +0100 (Tue, 31 Mar 2009) New Revision: 2483 Modified: box/trunk/configure.ac Log: Remove unused check for unaligned accesses, fails on cross-compiling. Disable large file support when cross-compiling, as the check cannot run in this environment. Modified: box/trunk/configure.ac =================================================================== --- box/trunk/configure.ac 2009-03-29 14:08:12 UTC (rev 2482) +++ box/trunk/configure.ac 2009-03-31 12:11:24 UTC (rev 2483) @@ -160,7 +160,6 @@ AC_HEADER_STDBOOL AC_C_CONST AC_C_BIGENDIAN -AX_CHECK_NONALIGNED_ACCESS AC_TYPE_UID_T AC_TYPE_MODE_T AC_TYPE_OFF_T @@ -243,10 +242,12 @@ ## Check for large file support active. AC_SYS_LARGEFILE has already worked ## out how to enable it if necessary, we just use this to report to the user AC_CACHE_CHECK([if we have large file support enabled], [box_cv_have_large_file_support], - [AC_RUN_IFELSE([AC_LANG_PROGRAM([[$ac_includes_default]], [[ + [AC_TRY_RUN([AC_LANG_PROGRAM([[$ac_includes_default]], [[ return sizeof(off_t)==4; ]])], - [box_cv_have_large_file_support=yes], [box_cv_have_large_file_support=no] + [box_cv_have_large_file_support=yes], + [box_cv_have_large_file_support=no], + [box_cv_have_large_file_support=no # safe for cross-compile] )]) if test "x$box_cv_have_large_file_support" = "xyes"; then From boxbackup-dev at boxbackup.org Tue Mar 31 13:12:35 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 31 Mar 2009 13:12:35 +0100 (BST) Subject: [Box Backup-commit] COMMIT r2484 - box/trunk/infrastructure/m4 Message-ID: <20090331121235.7E08A32503B@www.boxbackup.org> Author: chris Date: 2009-03-31 13:12:35 +0100 (Tue, 31 Mar 2009) New Revision: 2484 Modified: box/trunk/infrastructure/m4/ax_check_syscall_lseek.m4 Log: Assume that lseek does not take a "dummy parameter" when cross-compiling, as this is only relevent for intercept tests anyway, and was probably fixed by casting the offset parameter to off_t anyway. Modified: box/trunk/infrastructure/m4/ax_check_syscall_lseek.m4 =================================================================== --- box/trunk/infrastructure/m4/ax_check_syscall_lseek.m4 2009-03-31 12:11:24 UTC (rev 2483) +++ box/trunk/infrastructure/m4/ax_check_syscall_lseek.m4 2009-03-31 12:12:35 UTC (rev 2484) @@ -16,7 +16,7 @@ AC_REQUIRE([AX_FUNC_SYSCALL])dnl if test "x$ac_cv_header_sys_syscall_h" = "xyes"; then AC_CACHE_CHECK([[whether syscall lseek requires dummy parameter]], [box_cv_have_lseek_dummy_param], - [AC_RUN_IFELSE( + [AC_TRY_RUN( [AC_LANG_PROGRAM([[ $ac_includes_default #include @@ -50,7 +50,9 @@ unlink("lseektest"); return res!=-1; ]])], - [box_cv_have_lseek_dummy_param=yes], [box_cv_have_lseek_dummy_param=no] + [box_cv_have_lseek_dummy_param=yes], + [box_cv_have_lseek_dummy_param=no], + [box_cv_have_lseek_dummy_param=no # assume not for cross-compiling] )]) if test "x$box_cv_have_lseek_dummy_param" = "xyes"; then AC_DEFINE([HAVE_LSEEK_DUMMY_PARAM], 1, From boxbackup-dev at boxbackup.org Tue Mar 31 13:13:35 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 31 Mar 2009 13:13:35 +0100 (BST) Subject: [Box Backup-commit] COMMIT r2485 - box/trunk/infrastructure/m4 Message-ID: <20090331121335.23ACA32501F@www.boxbackup.org> Author: chris Date: 2009-03-31 13:13:34 +0100 (Tue, 31 Mar 2009) New Revision: 2485 Modified: box/trunk/infrastructure/m4/ax_check_dirent_d_type.m4 Log: Assume that we have no way to determine whether struct dirent has a valid d_type when cross-compiling, as this should be the safest option. Modified: box/trunk/infrastructure/m4/ax_check_dirent_d_type.m4 =================================================================== --- box/trunk/infrastructure/m4/ax_check_dirent_d_type.m4 2009-03-31 12:12:35 UTC (rev 2484) +++ box/trunk/infrastructure/m4/ax_check_dirent_d_type.m4 2009-03-31 12:13:34 UTC (rev 2485) @@ -16,7 +16,7 @@ AC_CHECK_MEMBERS([struct dirent.d_type],,, [[#include ]]) if test "x$ac_cv_member_struct_dirent_d_type" = "xyes"; then AC_CACHE_CHECK([[whether struct dirent.d_type is valid]], [box_cv_have_valid_dirent_d_type], - [AC_RUN_IFELSE( + [AC_TRY_RUN( [AC_LANG_PROGRAM([[ $ac_includes_default #include @@ -26,7 +26,9 @@ if(dir) res = readdir(dir); return res ? (res->d_type != DT_FILE && res->d_type != DT_DIR) : 1; ]])], - [box_cv_have_valid_dirent_d_type=yes], [box_cv_have_valid_dirent_d_type=no] + [box_cv_have_valid_dirent_d_type=yes], + [box_cv_have_valid_dirent_d_type=no], + [box_cv_have_valid_dirent_d_type=cross] )]) if test "x$box_cv_have_valid_dirent_d_type" = "xyes"; then AC_DEFINE([HAVE_VALID_DIRENT_D_TYPE], 1, [Define to 1 if struct dirent.d_type is valid]) From boxbackup-dev at boxbackup.org Tue Mar 31 14:36:32 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 31 Mar 2009 14:36:32 +0100 (BST) Subject: [Box Backup-commit] COMMIT r2486 - box/trunk/docs/api-notes Message-ID: <20090331133632.C0D5532501F@www.boxbackup.org> Author: chris Date: 2009-03-31 14:36:31 +0100 (Tue, 31 Mar 2009) New Revision: 2486 Modified: box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt Log: Win32 build instructions updated thanks to Achim. Modified: box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt =================================================================== --- box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt 2009-03-31 12:13:34 UTC (rev 2485) +++ box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt 2009-03-31 13:36:31 UTC (rev 2486) @@ -1,22 +1,29 @@ How to build Box Backup on Win32 using Cygwin and MinGW -By Chris Wilson, 2007-05-26 +By Chris Wilson, 2009-03-31 (To read this document online with better formatting, browse to: -http://www.boxbackup.org/trac/wiki/CompileWithMinGW) +[http://www.boxbackup.org/trac/wiki/CompileWithMinGW]) -Start by installing Cygwin on your Windows machine [http://www.cygwin.org]. +Start by installing Cygwin on your Windows machine from [http://www.cygwin.org/cygwin/]. + Make sure to select the following packages during installation: +* Devel/automake +* Devel/autoconf * Devel/gcc-mingw * Devel/gcc-mingw-core * Devel/gcc-mingw-g++ +* Devel/make +* Devel/mingw-runtime +* Lib/libxml2 +* Lib/libxslt * Mingw/mingw-zlib +* Perl/Perl If you already have Cygwin installed, please re-run the installer and ensure that those packages are installed. -Download OpenSSL from -[http://www.openssl.org/source/openssl-0.9.7i.tar.gz] +Download OpenSSL from [http://www.openssl.org/source/openssl-0.9.7i.tar.gz] Open a Cygwin shell, and unpack OpenSSL: @@ -42,12 +49,17 @@ export CFLAGS="-mno-cygwin" ./configure make winshared - cp .libs/libpcre.a .libs/libpcreposix.a /usr/lib/mingw + cp .libs/libpcre.a .libs/libpcreposix.a /lib/mingw cp pcreposix.h /usr/include/mingw -Now unpack the Box Backup sources, enter the source directory, -and configure like this: +Now get the latest Box Backup sources + svn co https://www.boxbackup.org/svn/box/trunk/ trunk + +Enter the source directory and configure like this: + + cd trunk ./infrastructure/mingw/configure.sh make +Note: In case you have any problems during the configure or make stage, please try to eliminate one potential source of problems by running "find -type f -not \( -wholename .*svn*. \) -exec dos2unix {} \;" in the trunk directory to fix potential line break problems. From boxbackup-dev at boxbackup.org Tue Mar 31 14:54:46 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 31 Mar 2009 14:54:46 +0100 (BST) Subject: [Box Backup-commit] COMMIT r2487 - box/trunk/docs/api-notes Message-ID: <20090331135446.3318032501F@www.boxbackup.org> Author: chris Date: 2009-03-31 14:54:45 +0100 (Tue, 31 Mar 2009) New Revision: 2487 Modified: box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt Log: Merge some wiki formatting from Trac page, and installation instructions. Modified: box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt =================================================================== --- box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt 2009-03-31 13:36:31 UTC (rev 2486) +++ box/trunk/docs/api-notes/win32_build_on_cygwin_using_mingw.txt 2009-03-31 13:54:45 UTC (rev 2487) @@ -4,28 +4,40 @@ (To read this document online with better formatting, browse to: [http://www.boxbackup.org/trac/wiki/CompileWithMinGW]) +== MinGW C++ == + Start by installing Cygwin on your Windows machine from [http://www.cygwin.org/cygwin/]. Make sure to select the following packages during installation: -* Devel/automake -* Devel/autoconf -* Devel/gcc-mingw -* Devel/gcc-mingw-core -* Devel/gcc-mingw-g++ -* Devel/make -* Devel/mingw-runtime -* Lib/libxml2 -* Lib/libxslt -* Mingw/mingw-zlib -* Perl/Perl + * Devel/automake + * Devel/autoconf + * Devel/gcc-mingw + * Devel/gcc-mingw-core + * Devel/gcc-mingw-g++ + * Devel/make + * Devel/mingw-runtime + * Lib/libxml2 + * Lib/libxslt + * Mingw/mingw-zlib + * Perl/Perl If you already have Cygwin installed, please re-run the installer and ensure that those packages are installed. +== Base Directory == + +Choose a directory where you will unpack and compile OpenSSL, Zlib and Box Backup. We will call this the ''base directory''. An example might be: + + C:\Cygwin\Home\Your Username + +Make sure you know the full path to this directory. + +== OpenSSL == + Download OpenSSL from [http://www.openssl.org/source/openssl-0.9.7i.tar.gz] -Open a Cygwin shell, and unpack OpenSSL: +Open a Cygwin shell, go to the base directory, and unpack OpenSSL: tar xzvf openssl-0.9.7i.tar.gz @@ -36,10 +48,14 @@ make make install +== PCRE == + +This step is only required to support regular expressions in including/excluding files from backups. However, this is a very useful feature. + Download PCRE from [http://prdownloads.sourceforge.net/pcre/pcre-6.3.tar.bz2?download] -Open a Cygwin shell, and unpack PCRE: +Open a Cygwin shell, go to the base directory, and unpack PCRE: tar xjvf pcre-6.3.tar.bz2 @@ -52,14 +68,40 @@ cp .libs/libpcre.a .libs/libpcreposix.a /lib/mingw cp pcreposix.h /usr/include/mingw -Now get the latest Box Backup sources +== Download Box Backup == +Go back to the base directory, and download the latest Box Backup sources: + svn co https://www.boxbackup.org/svn/box/trunk/ trunk +Note: If you have problems during the configure or make stage, please try to eliminate one potential source of problems by running the following command to convert all line endings to Unix format: + + find -type f -not \( -wholename .*svn*. \) -exec dos2unix {} \; + +== Compile Box Backup == + Enter the source directory and configure like this: cd trunk ./infrastructure/mingw/configure.sh make -Note: In case you have any problems during the configure or make stage, please try to eliminate one potential source of problems by running "find -type f -not \( -wholename .*svn*. \) -exec dos2unix {} \;" in the trunk directory to fix potential line break problems. +== Installation == + +Create the destination directory, ''C:\Program Files\Box Backup\bbackupd''. + +Write a configuration file, keys and certificate on a Unix machine, and copy them into the ''Box Backup'' directory, together with the following files from the ''base directory'': + + * boxbackup\release\bin\bbackupd\bbackupd.exe + * boxbackup\release\bin\bbackupquery\bbackupquery.exe + * boxbackup\release\bin\bbackupctl\bbackupctl.exe + * openssl\out32dll\libeay32.dll + * openssl\out32dll\ssleay32.dll + * zlib\zlib1.dll + +Ensure that the user running Box Backup can read from the ''Box Backup'' directory, and write to the ''bbackupd'' directory inside it. + +Run Box Backup by double-clicking on it, and check that it connects to the server. If the window opens and closes immediately, it's probably due to a problem with the configuration file - check the Windows Event Viewer for details. + +See also the service installation and upgrade instructions at +[https://www.boxbackup.org/trac/wiki/CompilationOnWindows]. From boxbackup-dev at boxbackup.org Tue Mar 31 20:25:22 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 31 Mar 2009 20:25:22 +0100 (BST) Subject: [Box Backup-commit] COMMIT r2488 - box/trunk/docs Message-ID: <20090331192522.D0F6A32501F@www.boxbackup.org> Author: chris Date: 2009-03-31 20:25:22 +0100 (Tue, 31 Mar 2009) New Revision: 2488 Modified: box/trunk/docs/Makefile Log: Remove generated bb-man.xsl when making clean. Modified: box/trunk/docs/Makefile =================================================================== --- box/trunk/docs/Makefile 2009-03-31 13:54:45 UTC (rev 2487) +++ box/trunk/docs/Makefile 2009-03-31 19:25:22 UTC (rev 2488) @@ -112,5 +112,5 @@ rm -f $(NROFF_FILES) rm -f $(DOCBOOK_DIR)/ExceptionCodes.xml rm -f documentation-kit-0.10.tar.gz + rm -f $(MANXSL) - From boxbackup-dev at boxbackup.org Tue Mar 31 20:56:09 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 31 Mar 2009 20:56:09 +0100 (BST) Subject: [Box Backup-commit] COMMIT r2489 - box/trunk/docs Message-ID: <20090331195609.E7BB132501F@www.boxbackup.org> Author: chris Date: 2009-03-31 20:56:09 +0100 (Tue, 31 Mar 2009) New Revision: 2489 Modified: box/trunk/docs/Makefile Log: Hopefully fix building and cleaning of bb-man.xsl. Modified: box/trunk/docs/Makefile =================================================================== --- box/trunk/docs/Makefile 2009-03-31 19:25:22 UTC (rev 2488) +++ box/trunk/docs/Makefile 2009-03-31 19:56:09 UTC (rev 2489) @@ -39,8 +39,10 @@ $(DOCBOOK_DIR)/ExceptionCodes.xml: ../ExceptionCodes.txt perl tools/generate_except_xml.pl $< $> $@ -manpages: $(MANXSL) man-dirs man-nroff man-html +manpages: man-dirs man-nroff man-html +xslt: $(MANXSL) + $(MANXSL): $(MANXSL).tmpl @if [ -f /usr/local/share/xsl/docbook/manpages/docbook.xsl ]; then \ DOCBOOK=file:///usr/local/share/xsl/docbook/manpages/docbook.xsl; \ @@ -82,21 +84,32 @@ $(HTML_DIR)/man-html/%.html: $(DOCBOOK_DIR)/%.xml $(NOCHUNKBOOKXSL) $(DBPROC) -o $@ $(NOCHUNKBOOKXSL) $< +# Before running xsltproc to generate manual pages, we need to check +# that $(MANXSL) has been built. We don't want to add it to dependencies, +# because that would cause # the man pages to try to be rebuilt even if +# they already exist if the date of the xslt file changes, and that +# requires xsltproc, which negates the point of precompiling them for +# distribution users. + # GNU make -$(MAN_DIR)/%.8.gz: $(DOCBOOK_DIR)/%.xml $(MANXSL) +$(MAN_DIR)/%.8.gz: $(DOCBOOK_DIR)/%.xml + $(MAKE) xslt $(DBPROC) -o $(@:.gz=) $(MANXSL) $< gzip $(@:.gz=) # GNU make -$(MAN_DIR)/%.5.gz: $(DOCBOOK_DIR)/%.xml $(MANXSL) +$(MAN_DIR)/%.5.gz: $(DOCBOOK_DIR)/%.xml + $(MAKE) xslt $(DBPROC) -o $(@:.gz=) $(MANXSL) $< gzip $(@:.gz=) -# BSD make: the final colon (:) is required to make this line valid syntax -# for a dummy rule in GNU make. It creates a dummy rule in BSD make too. -# Both dummy rules are harmless. +# BSD make: the final colon (:) is required to make the .for and .endfor +# lines valid in GNU make. It creates (different) dummy rules in GNU and +# BSD make. Both dummy rules are harmless. + .for MAN_PAGE in $(NROFF_PAGES) : $(MAN_DIR)/$(MAN_PAGE).gz: $(DOCBOOK_DIR)/$(MAN_PAGE:R).xml + $(MAKE) xslt $(DBPROC) -o $(.TARGET:.gz=) $(MANXSL) $> gzip $(@:.gz=) From boxbackup-dev at boxbackup.org Tue Mar 31 21:42:31 2009 From: boxbackup-dev at boxbackup.org (boxbackup-dev at boxbackup.org) Date: Tue, 31 Mar 2009 21:42:31 +0100 (BST) Subject: [Box Backup-commit] COMMIT r2490 - box/trunk/contrib/mac_osx Message-ID: <20090331204231.A213732501F@www.boxbackup.org> Author: chris Date: 2009-03-31 21:42:31 +0100 (Tue, 31 Mar 2009) New Revision: 2490 Modified: box/trunk/contrib/mac_osx/org.boxbackup.bbackupd.plist.in box/trunk/contrib/mac_osx/org.boxbackup.bbstored.plist.in Log: OSX Plist file updates, thanks to Achim. Modified: box/trunk/contrib/mac_osx/org.boxbackup.bbackupd.plist.in =================================================================== --- box/trunk/contrib/mac_osx/org.boxbackup.bbackupd.plist.in 2009-03-31 19:56:09 UTC (rev 2489) +++ box/trunk/contrib/mac_osx/org.boxbackup.bbackupd.plist.in 2009-03-31 20:42:31 UTC (rev 2490) @@ -9,6 +9,12 @@ ProgramArguments @prefix@/sbin/bbackupd + -F + @prefix@/etc/boxbackup/bbackupd.conf + LowPriorityIO + + Nice + 1 Modified: box/trunk/contrib/mac_osx/org.boxbackup.bbstored.plist.in =================================================================== --- box/trunk/contrib/mac_osx/org.boxbackup.bbstored.plist.in 2009-03-31 19:56:09 UTC (rev 2489) +++ box/trunk/contrib/mac_osx/org.boxbackup.bbstored.plist.in 2009-03-31 20:42:31 UTC (rev 2490) @@ -9,6 +9,13 @@ ProgramArguments @prefix@/sbin/bbstored + -F + @prefix@/etc/boxbackup/bbackupd.conf + + LowPriorityIO + + Nice + 1