From boxbackup-dev at fluffy.co.uk Thu Nov 2 22:20:19 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 02 Nov 2006 22:20:19 -0000 Subject: [Box Backup-commit] #9: SSL connection may time out when backing up large data sets Message-ID: <075.382a81ac9c31eb8a9dfae053eb0061b0@fluffy.co.uk> #9: SSL connection may time out when backing up large data sets ------------------------------------------------------------+--------------- Reporter: chris | Owner: chris Type: defect | Status: new Priority: normal | Milestone: 0.11 Component: bbackupctl | Version: 0.10 Keywords: ssl timeout client reload scanning directories | ------------------------------------------------------------+--------------- Tobias Balle-Petersen reports that when backing up 1 Tb of files, the client needs to be reloaded before each backup, otherwise the backup will fail with an SSL timeout error. He has KeepAliveTime enabled and set low (10 seconds). It appears that the client can spend large amounts of time scanning (locally) for changed files without contacting the server, and during these long delays, the connection can time out. Reloading the client causes it to discard cached information and request data from the store more often, which avoids the timeout. I suspect that the root of the problem is that KeepAliveTime is only honoured while diffing individual files, and not while scanning directories. In a large data set where few files change between backup runs, a timeout may occur while scanning directories. Sending KeepAlive messages regularly while scanning directories may fix this problem. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Mon Nov 6 20:40:46 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 06 Nov 2006 20:40:46 +0000 Subject: [Box Backup-commit] COMMIT r1140 - box/chris/merge/bin/bbstored Message-ID: Author: chris Date: 2006-11-06 20:40:46 +0000 (Mon, 06 Nov 2006) New Revision: 1140 Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp Log: Compile fix Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp =================================================================== --- box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2006-10-27 21:34:36 UTC (rev 1139) +++ box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2006-11-06 20:40:46 UTC (rev 1140) @@ -42,7 +42,6 @@ mExtendedLogging(false), mHaveForkedHousekeeping(false), mIsHousekeepingProcess(false), - mHousekeepingInited(false), mInterProcessComms(mInterProcessCommsSocket) { } From boxbackup-dev at fluffy.co.uk Mon Nov 6 20:43:12 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 06 Nov 2006 20:43:12 +0000 Subject: [Box Backup-commit] COMMIT r1141 - in box/chris/merge: lib/common test/common Message-ID: Author: chris Date: 2006-11-06 20:43:12 +0000 (Mon, 06 Nov 2006) New Revision: 1141 Added: box/chris/merge/lib/common/Timer.cpp box/chris/merge/lib/common/Timer.h Modified: box/chris/merge/lib/common/BoxTime.h box/chris/merge/test/common/testcommon.cpp Log: Added generic timer support class Modified: box/chris/merge/lib/common/BoxTime.h =================================================================== --- box/chris/merge/lib/common/BoxTime.h 2006-11-06 20:40:46 UTC (rev 1140) +++ box/chris/merge/lib/common/BoxTime.h 2006-11-06 20:43:12 UTC (rev 1141) @@ -35,6 +35,9 @@ { return Time / MILLI_SEC_IN_NANO_SEC_LL; } +inline uint64_t BoxTimeToMicroSeconds(box_time_t Time) +{ + return Time; +} #endif // BOXTIME__H - Copied: box/chris/merge/lib/common/Timer.cpp (from rev 966, box/chris/merge/lib/common/WaitForEvent.cpp) =================================================================== --- box/chris/merge/lib/common/Timer.cpp (rev 0) +++ box/chris/merge/lib/common/Timer.cpp 2006-11-06 20:43:12 UTC (rev 1141) @@ -0,0 +1,252 @@ +// -------------------------------------------------------------------------- +// +// File +// Name: Timer.cpp +// Purpose: Generic timers which execute arbitrary code when +// they expire. +// Created: 5/11/2006 +// +// -------------------------------------------------------------------------- + +#include "Box.h" + +#include + +#include "Timer.h" + +#include "MemLeakFindOn.h" + +std::vector Timers::sTimers; +bool Timers::sInitialised = false; + +// -------------------------------------------------------------------------- +// +// Function +// Name: static void TimerSigHandler(int) +// Purpose: Signal handler, notifies Timers class +// Created: 19/3/04 +// +// -------------------------------------------------------------------------- +static void TimerSigHandler(int iUnused) +{ + Timers::Signal(); +} + +// -------------------------------------------------------------------------- +// +// Function +// Name: static void Timers::Init() +// Purpose: Initialise timers, prepare signal handler +// Created: 5/11/2006 +// +// -------------------------------------------------------------------------- +void Timers::Init() +{ + ASSERT(!sInitialised); + + #ifdef PLATFORM_CYGWIN + ASSERT(::signal(SIGALRM, TimerSigHandler) == 0); + #elif defined WIN32 + // no support for signals at all + InitTimer(); + SetTimerHandler(TimerSigHandler); + #else + ASSERT(::signal(SIGALRM, TimerSigHandler) == 0); + #endif // PLATFORM_CYGWIN + + sInitialised = true; +} + +// -------------------------------------------------------------------------- +// +// Function +// Name: static void Timers::Cleanup() +// Purpose: Clean up timers, stop signal handler +// Created: 6/11/2006 +// +// -------------------------------------------------------------------------- +void Timers::Cleanup() +{ + ASSERT(sInitialised); + + #ifdef PLATFORM_CYGWIN + ASSERT(::signal(SIGALRM, NULL) == TimerSigHandler); + #elif defined WIN32 + // no support for signals at all + SetTimerHandler(NULL); + FiniTimer(); + #else + ASSERT(::signal(SIGALRM, NULL) == TimerSigHandler); + #endif // PLATFORM_CYGWIN + + sTimers.clear(); + + sInitialised = false; +} + +// -------------------------------------------------------------------------- +// +// Function +// Name: static void Timers::Add(Timer&) +// Purpose: Add a new timer to the set, and reschedule next wakeup +// Created: 5/11/2006 +// +// -------------------------------------------------------------------------- +void Timers::Add(Timer& rTimer) +{ + ASSERT(sInitialised); + sTimers.push_back(&rTimer); + Reschedule(); +} + +// -------------------------------------------------------------------------- +// +// Function +// Name: static void Timers::Remove(Timer&) +// Purpose: Removes the timer from the set (preventing it from +// being called) and reschedule next wakeup +// Created: 5/11/2006 +// +// -------------------------------------------------------------------------- +void Timers::Remove(Timer& rTimer) +{ + ASSERT(sInitialised); + + bool restart = true; + while (restart) + { + restart = false; + for (std::vector::iterator i = sTimers.begin(); + i != sTimers.end(); i++) + { + if (&rTimer == *i) + { + sTimers.erase(i); + restart = true; + break; + } + } + } + + Reschedule(); +} + +// -------------------------------------------------------------------------- +// +// Function +// Name: static void Timers::Reschedule() +// Purpose: Recalculate when the next wakeup is due +// Created: 5/11/2006 +// +// -------------------------------------------------------------------------- +void Timers::Reschedule() +{ + ASSERT(sInitialised); + + box_time_t timeNow = GetCurrentBoxTime(); + box_time_t timeToNextEvent = 0; + + for (std::vector::iterator i = sTimers.begin(); + i != sTimers.end(); i++) + { + Timer& rTimer = **i; + ASSERT(!rTimer.HasExpired()); + + box_time_t timeToExpiry = rTimer.GetExpiryTime() - timeNow; + + if (timeToNextEvent == 0 || timeToNextEvent > timeToExpiry) + { + timeToNextEvent = timeToExpiry; + } + } + + ASSERT(timeToNextEvent >= 0); + + struct itimerval timeout; + memset(&timeout, 0, sizeof(timeout)); + + timeout.it_value.tv_sec = BoxTimeToSeconds(timeToNextEvent); + timeout.it_value.tv_usec = (int) + (BoxTimeToMicroSeconds(timeToNextEvent) % MICRO_SEC_IN_SEC); + +#ifdef PLATFORM_CYGWIN + if(::setitimer(ITIMER_REAL, &timeout, NULL) != 0) +#else + if(::setitimer(ITIMER_REAL, &timeout, NULL) != 0) +#endif // PLATFORM_CYGWIN + { + TRACE0("WARNING: couldn't initialise timer\n"); + THROW_EXCEPTION(CommonException, Internal) + } +} + +// -------------------------------------------------------------------------- +// +// Function +// Name: static void Timers::Signal() +// Purpose: Called by signal handler. Signals any timers which +// are due or overdue, removes them from the set, +// and reschedules next wakeup. +// Created: 5/11/2006 +// +// -------------------------------------------------------------------------- +void Timers::Signal() +{ + ASSERT(sInitialised); + + box_time_t timeNow = GetCurrentBoxTime(); + + std::vector timersCopy = sTimers; + + for (std::vector::iterator i = timersCopy.begin(); + i != timersCopy.end(); i++) + { + Timer& rTimer = **i; + ASSERT(!rTimer.HasExpired()); + + box_time_t timeToExpiry = rTimer.GetExpiryTime() - timeNow; + + if (timeToExpiry <= 0) + { + rTimer.OnExpire(); + } + } + + Reschedule(); +} + +Timer::Timer(size_t timeoutSecs) +: mExpires(GetCurrentBoxTime() + SecondsToBoxTime(timeoutSecs)), + mExpired(false) +{ + Timers::Add(*this); +} + +Timer::~Timer() +{ + Timers::Remove(*this); +} + +Timer::Timer(const Timer& rToCopy) +: mExpires(rToCopy.mExpires), + mExpired(rToCopy.mExpired) +{ + Timers::Add(*this); +} + +Timer& Timer::operator=(const Timer& rToCopy) +{ + Timers::Remove(*this); + mExpires = rToCopy.mExpires; + mExpired = rToCopy.mExpired; + if (!mExpired) + { + Timers::Add(*this); + } +} + +void Timer::OnExpire() +{ + mExpired = true; + Timers::Remove(*this); +} Copied: box/chris/merge/lib/common/Timer.h (from rev 966, box/chris/merge/lib/common/WaitForEvent.h) =================================================================== --- box/chris/merge/lib/common/Timer.h (rev 0) +++ box/chris/merge/lib/common/Timer.h 2006-11-06 20:43:12 UTC (rev 1141) @@ -0,0 +1,65 @@ +// -------------------------------------------------------------------------- +// +// File +// Name: Timer.h +// Purpose: Generic timers which execute arbitrary code when +// they expire. +// Created: 5/11/2006 +// +// -------------------------------------------------------------------------- + +#ifndef TIMER__H +#define TIMER__H + +#include + +#include + +#include "MemLeakFindOn.h" +#include "BoxTime.h" + +class Timer +{ +public: + Timer(size_t timeoutSecs); + virtual ~Timer(); + Timer(const Timer &); + Timer &operator=(const Timer &); + +public: + box_time_t GetExpiryTime() { return mExpires; } + bool HasExpired () { return mExpired; } + virtual void OnExpire (); + +private: + box_time_t mExpires; + bool mExpired; +}; + +// -------------------------------------------------------------------------- +// +// Class +// Name: Timers +// Purpose: Static class to manage all timers and arrange +// efficient delivery of wakeup signals +// Created: 19/3/04 +// +// -------------------------------------------------------------------------- +class Timers +{ + private: + static std::vector sTimers; + static bool sInitialised; + static void Reschedule(); + + public: + static void Init(); + static void Cleanup(); + static void Add (Timer& rTimer); + static void Remove(Timer& rTimer); + static void Signal(); +}; + +#include "MemLeakFindOff.h" + +#endif // TIMER__H Modified: box/chris/merge/test/common/testcommon.cpp =================================================================== --- box/chris/merge/test/common/testcommon.cpp 2006-11-06 20:40:46 UTC (rev 1140) +++ box/chris/merge/test/common/testcommon.cpp 2006-11-06 20:43:12 UTC (rev 1141) @@ -27,6 +27,7 @@ #include "autogen_ConversionException.h" #include "CollectInBufferStream.h" #include "Archive.h" +#include "Timer.h" #include "MemLeakFindOn.h" @@ -204,7 +205,57 @@ } #endif // BOX_MEMORY_LEAK_TESTING + // Check that using timer methods without initialisation + // throws an exception + TEST_CHECK_THROWS(Timers::Add(*(Timer*)NULL), + CommonException, AssertFailed); + TEST_CHECK_THROWS(Timers::Remove(*(Timer*)NULL), + CommonException, AssertFailed); + TEST_CHECK_THROWS(Timers::Signal(), CommonException, AssertFailed); + TEST_CHECK_THROWS(Timers::Cleanup(), CommonException, AssertFailed); + + // Check that we can initialise the timers + Timers::Init(); + + // Check that double initialisation throws an exception + TEST_CHECK_THROWS(Timers::Init(), CommonException, AssertFailed); + // Check that we can clean up the timers + Timers::Cleanup(); + + // Check that double cleanup throws an exception + TEST_CHECK_THROWS(Timers::Cleanup(), CommonException, AssertFailed); + + Timers::Init(); + + Timer t1(1); + Timer t2(2); + Timer t3(3); + + TEST_THAT(!t1.HasExpired()); + TEST_THAT(!t2.HasExpired()); + TEST_THAT(!t3.HasExpired()); + + sleep(1); + TEST_THAT(t1.HasExpired()); + TEST_THAT(!t2.HasExpired()); + TEST_THAT(!t3.HasExpired()); + + sleep(1); + TEST_THAT(t1.HasExpired()); + TEST_THAT(t2.HasExpired()); + TEST_THAT(!t3.HasExpired()); + + t1 = Timer(1); + t2 = Timer(2); + TEST_THAT(!t1.HasExpired()); + TEST_THAT(!t2.HasExpired()); + + sleep(1); + TEST_THAT(t1.HasExpired()); + TEST_THAT(!t2.HasExpired()); + TEST_THAT(t3.HasExpired()); + static char *testfilelines[] = { "First line", From boxbackup-dev at fluffy.co.uk Mon Nov 6 20:45:32 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 06 Nov 2006 20:45:32 +0000 Subject: [Box Backup-commit] COMMIT r1142 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2006-11-06 20:45:32 +0000 (Mon, 06 Nov 2006) New Revision: 1142 Modified: box/chris/merge/lib/win32/emu.cpp Log: Protect against double initialisation of win32 timers Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2006-11-06 20:43:12 UTC (rev 1141) +++ box/chris/merge/lib/win32/emu.cpp 2006-11-06 20:45:32 UTC (rev 1142) @@ -28,6 +28,7 @@ // our implementation for a timer, based on a // simple thread which sleeps for a period of time +static bool gTimerInitialised = false; static bool gFinishTimer; static CRITICAL_SECTION gLock; @@ -43,6 +44,8 @@ int setitimer(int type, struct itimerval *timeout, void *arg) { + ASSERT(gTimerInitialised); + if (ITIMER_VIRTUAL == type) { EnterCriticalSection(&gLock); @@ -131,20 +134,25 @@ void InitTimer(void) { + ASSERT(!gTimerInitialised); InitializeCriticalSection(&gLock); - + // create our thread HANDLE ourThread = (HANDLE)_beginthreadex(NULL, 0, RunTimer, 0, CREATE_SUSPENDED, NULL); SetThreadPriority(ourThread, THREAD_PRIORITY_LOWEST); ResumeThread(ourThread); + + gTimerInitialised = true; } void FiniTimer(void) { + ASSERT(gTimerInitialised); gFinishTimer = true; EnterCriticalSection(&gLock); DeleteCriticalSection(&gLock); + gTimerInitialised = false; } //Our constants we need to keep track of From boxbackup-dev at fluffy.co.uk Mon Nov 6 20:47:15 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 06 Nov 2006 20:47:15 +0000 Subject: [Box Backup-commit] COMMIT r1143 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2006-11-06 20:47:15 +0000 (Mon, 06 Nov 2006) New Revision: 1143 Modified: box/chris/merge/lib/common/DebugMemLeakFinder.cpp Log: Fix compile warnings (refs #3) Modified: box/chris/merge/lib/common/DebugMemLeakFinder.cpp =================================================================== --- box/chris/merge/lib/common/DebugMemLeakFinder.cpp 2006-11-06 20:45:32 UTC (rev 1142) +++ box/chris/merge/lib/common/DebugMemLeakFinder.cpp 2006-11-06 20:47:15 UTC (rev 1143) @@ -55,7 +55,7 @@ static std::set sNotLeaks; void *sNotLeaksPre[1024]; - int sNotLeaksPreNum = 0; + size_t sNotLeaksPreNum = 0; } void memleakfinder_malloc_add_block(void *b, size_t size, const char *file, int line) @@ -161,7 +161,7 @@ void memleakfinder_notaleak_insert_pre() { if(!memleakfinder_global_enable) return; - for(int l = 0; l < sNotLeaksPreNum; l++) + for(size_t l = 0; l < sNotLeaksPreNum; l++) { sNotLeaks.insert(sNotLeaksPre[l]); } @@ -184,7 +184,7 @@ else { if ( sNotLeaksPreNum < - (unsigned)( sizeof(sNotLeaksPre)/sizeof(*sNotLeaksPre) ) ) + sizeof(sNotLeaksPre)/sizeof(*sNotLeaksPre) ) sNotLeaksPre[sNotLeaksPreNum++] = ptr; } /* { @@ -255,11 +255,11 @@ { for(std::map::const_iterator i(sMallocBlocks.begin()); i != sMallocBlocks.end(); ++i) { - if(is_leak(i->first)) ::fprintf(file, "Block 0x%08p size %d allocated at %s:%d\n", i->first, i->second.size, i->second.file, i->second.line); + if(is_leak(i->first)) ::fprintf(file, "Block 0x%p size %d allocated at %s:%d\n", i->first, i->second.size, i->second.file, i->second.line); } for(std::map::const_iterator i(sObjectBlocks.begin()); i != sObjectBlocks.end(); ++i) { - if(is_leak(i->first)) ::fprintf(file, "Object%s 0x%08p size %d allocated at %s:%d\n", i->second.array?" []":"", i->first, i->second.size, i->second.file, i->second.line); + if(is_leak(i->first)) ::fprintf(file, "Object%s 0x%p size %d allocated at %s:%d\n", i->second.array?" []":"", i->first, i->second.size, i->second.file, i->second.line); } } From boxbackup-dev at fluffy.co.uk Mon Nov 6 20:48:47 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 06 Nov 2006 20:48:47 -0000 Subject: [Box Backup-commit] Re: #9: SSL connection may time out when backing up large data sets In-Reply-To: <075.382a81ac9c31eb8a9dfae053eb0061b0@fluffy.co.uk> References: <075.382a81ac9c31eb8a9dfae053eb0061b0@fluffy.co.uk> Message-ID: <084.d0593e2a3f12a312c3d7482027974356@fluffy.co.uk> #9: SSL connection may time out when backing up large data sets -------------------------+-------------------------------------------------- Reporter: chris | Owner: chris Type: defect | Status: new Priority: normal | Milestone: 0.11 Component: bbackupctl | Version: 0.10 Resolution: | Keywords: ssl timeout client reload scanning directories -------------------------+-------------------------------------------------- Comment (by chris): Please review [1141] [1142]. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:13:59 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:13:59 +0000 Subject: [Box Backup-commit] COMMIT r1144 - box/chris/merge/test/common Message-ID: Author: chris Date: 2006-11-13 15:13:58 +0000 (Mon, 13 Nov 2006) New Revision: 1144 Modified: box/chris/merge/test/common/testcommon.cpp Log: Fix scoping to ensure that objects which allocate memory via the standard libraries, free it before the memleak tests, to avoid test failures (refs #3) Modified: box/chris/merge/test/common/testcommon.cpp =================================================================== --- box/chris/merge/test/common/testcommon.cpp 2006-11-06 20:47:15 UTC (rev 1143) +++ box/chris/merge/test/common/testcommon.cpp 2006-11-13 15:13:58 UTC (rev 1144) @@ -137,56 +137,67 @@ int test(int argc, const char *argv[]) { // Test self-deleting temporary file streams - std::string tempfile("testfiles/tempfile"); - TEST_CHECK_THROWS(InvisibleTempFileStream fs(tempfile.c_str()), - CommonException, OSFileOpenError); - InvisibleTempFileStream fs(tempfile.c_str(), O_CREAT); + { + std::string tempfile("testfiles/tempfile"); + TEST_CHECK_THROWS(InvisibleTempFileStream fs(tempfile.c_str()), + CommonException, OSFileOpenError); + InvisibleTempFileStream fs(tempfile.c_str(), O_CREAT); -#ifdef WIN32 - // file is still visible under Windows - TEST_THAT(TestFileExists(tempfile.c_str())); + #ifdef WIN32 + // file is still visible under Windows + TEST_THAT(TestFileExists(tempfile.c_str())); - // opening it again should work - InvisibleTempFileStream fs2(tempfile.c_str()); - TEST_THAT(TestFileExists(tempfile.c_str())); + // opening it again should work + InvisibleTempFileStream fs2(tempfile.c_str()); + TEST_THAT(TestFileExists(tempfile.c_str())); - // opening it to create should work - InvisibleTempFileStream fs3(tempfile.c_str(), O_CREAT); - TEST_THAT(TestFileExists(tempfile.c_str())); + // opening it to create should work + InvisibleTempFileStream fs3(tempfile.c_str(), O_CREAT); + TEST_THAT(TestFileExists(tempfile.c_str())); - // opening it to create exclusively should fail - TEST_CHECK_THROWS(InvisibleTempFileStream fs4(tempfile.c_str(), - O_CREAT | O_EXCL), CommonException, OSFileOpenError); + // opening it to create exclusively should fail + TEST_CHECK_THROWS(InvisibleTempFileStream fs4(tempfile.c_str(), + O_CREAT | O_EXCL), CommonException, OSFileOpenError); - fs2.Close(); -#else - // file is not visible under Unix - TEST_THAT(!TestFileExists(tempfile.c_str())); + fs2.Close(); + #else + // file is not visible under Unix + TEST_THAT(!TestFileExists(tempfile.c_str())); - // opening it again should fail - TEST_CHECK_THROWS(InvisibleTempFileStream fs2(tempfile.c_str()), - CommonException, OSFileOpenError); + // opening it again should fail + TEST_CHECK_THROWS(InvisibleTempFileStream fs2(tempfile.c_str()), + CommonException, OSFileOpenError); - // opening it to create should work - InvisibleTempFileStream fs3(tempfile.c_str(), O_CREAT); - TEST_THAT(!TestFileExists(tempfile.c_str())); + // opening it to create should work + InvisibleTempFileStream fs3(tempfile.c_str(), O_CREAT); + TEST_THAT(!TestFileExists(tempfile.c_str())); - // opening it to create exclusively should work - InvisibleTempFileStream fs4(tempfile.c_str(), O_CREAT | O_EXCL); - TEST_THAT(!TestFileExists(tempfile.c_str())); + // opening it to create exclusively should work + InvisibleTempFileStream fs4(tempfile.c_str(), O_CREAT | O_EXCL); + TEST_THAT(!TestFileExists(tempfile.c_str())); - fs4.Close(); -#endif + fs4.Close(); + #endif - fs.Close(); - fs3.Close(); + fs.Close(); + fs3.Close(); - // now that it's closed, it should be invisible on all platforms - TEST_THAT(!TestFileExists(tempfile.c_str())); + // now that it's closed, it should be invisible on all platforms + TEST_THAT(!TestFileExists(tempfile.c_str())); + } - // Test memory leak detection + // Test that memory leak detection doesn't crash + { + char *test = new char[1024]; + delete [] test; + MemBlockStream *s = new MemBlockStream(test,12); + delete s; + } + #ifdef BOX_MEMORY_LEAK_TESTING { + Timers::Cleanup(); + TEST_THAT(memleakfinder_numleaks() == 0); void *block = ::malloc(12); TEST_THAT(memleakfinder_numleaks() == 1); @@ -202,8 +213,13 @@ TEST_THAT(memleakfinder_numleaks() == 1); delete [] test; TEST_THAT(memleakfinder_numleaks() == 0); + + Timers::Init(); } #endif // BOX_MEMORY_LEAK_TESTING + + // test main() initialises timers for us, so uninitialise them + Timers::Cleanup(); // Check that using timer methods without initialisation // throws an exception From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:15:23 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:15:23 +0000 Subject: [Box Backup-commit] COMMIT r1145 - box/chris/merge/test/common Message-ID: Author: chris Date: 2006-11-13 15:15:23 +0000 (Mon, 13 Nov 2006) New Revision: 1145 Modified: box/chris/merge/test/common/testcommon.cpp Log: Added tests for timers with zero interval, which should never expire (refs #9) Modified: box/chris/merge/test/common/testcommon.cpp =================================================================== --- box/chris/merge/test/common/testcommon.cpp 2006-11-13 15:13:58 UTC (rev 1144) +++ box/chris/merge/test/common/testcommon.cpp 2006-11-13 15:15:23 UTC (rev 1145) @@ -244,34 +244,43 @@ Timers::Init(); + Timer t0(0); // should never expire Timer t1(1); Timer t2(2); Timer t3(3); + TEST_THAT(!t0.HasExpired()); TEST_THAT(!t1.HasExpired()); TEST_THAT(!t2.HasExpired()); TEST_THAT(!t3.HasExpired()); sleep(1); + TEST_THAT(!t0.HasExpired()); TEST_THAT(t1.HasExpired()); TEST_THAT(!t2.HasExpired()); TEST_THAT(!t3.HasExpired()); sleep(1); + TEST_THAT(!t0.HasExpired()); TEST_THAT(t1.HasExpired()); TEST_THAT(t2.HasExpired()); TEST_THAT(!t3.HasExpired()); t1 = Timer(1); t2 = Timer(2); + TEST_THAT(!t0.HasExpired()); TEST_THAT(!t1.HasExpired()); TEST_THAT(!t2.HasExpired()); sleep(1); + TEST_THAT(!t0.HasExpired()); TEST_THAT(t1.HasExpired()); TEST_THAT(!t2.HasExpired()); TEST_THAT(t3.HasExpired()); + // Leave timers initialised for rest of test. + // Test main() will cleanup after test finishes. + static char *testfilelines[] = { "First line", From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:20:56 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:20:56 +0000 Subject: [Box Backup-commit] COMMIT r1146 - box/chris/merge/infrastructure Message-ID: Author: chris Date: 2006-11-13 15:20:56 +0000 (Mon, 13 Nov 2006) New Revision: 1146 Modified: box/chris/merge/infrastructure/makebuildenv.pl.in Log: Force glibc to use new/delete to allocate memory and disable its internal pools for the unit tests, to make memory leak detection work (refs #3) Modified: box/chris/merge/infrastructure/makebuildenv.pl.in =================================================================== --- box/chris/merge/infrastructure/makebuildenv.pl.in 2006-11-13 15:15:23 UTC (rev 1145) +++ box/chris/merge/infrastructure/makebuildenv.pl.in 2006-11-13 15:20:56 UTC (rev 1146) @@ -422,9 +422,9 @@ close TESTFILE; } - writetestfile("$mod/_t", + writetestfile("$mod/_t", "GLIBCXX_FORCE_NEW=1 ". './test' . $platform_exe_ext . ' $1 $2 $3 $4 $5', $mod); - writetestfile("$mod/_t-gdb", + writetestfile("$mod/_t-gdb", "GLIBCXX_FORCE_NEW=1 ". 'gdb ./test' . $platform_exe_ext, $mod); } From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:46:08 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:46:08 +0000 Subject: [Box Backup-commit] COMMIT r1147 - box/chris/merge/lib/backupstore Message-ID: Author: chris Date: 2006-11-13 15:46:08 +0000 (Mon, 13 Nov 2006) New Revision: 1147 Modified: box/chris/merge/lib/backupstore/BackupStoreCheck2.cpp Log: Fixed typo. Modified: box/chris/merge/lib/backupstore/BackupStoreCheck2.cpp =================================================================== --- box/chris/merge/lib/backupstore/BackupStoreCheck2.cpp 2006-11-13 15:20:56 UTC (rev 1146) +++ box/chris/merge/lib/backupstore/BackupStoreCheck2.cpp 2006-11-13 15:46:08 UTC (rev 1147) @@ -653,7 +653,7 @@ if(dependsOlder != 0 && FindEntryByID(dependsOlder) == 0) { // Has an older version marked, but this doesn't exist. Remove this mark - TRACE2("Entry id %llx was marked that %llx depended on it, which doesn't exist, dependency info cleaered\n", (*i)->GetObjectID(), dependsOlder); + TRACE2("Entry id %llx was marked that %llx depended on it, which doesn't exist, dependency info cleared\n", (*i)->GetObjectID(), dependsOlder); (*i)->SetDependsOlder(0); From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:47:12 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:47:12 +0000 Subject: [Box Backup-commit] COMMIT r1148 - box/trunk/lib/backupstore Message-ID: Author: chris Date: 2006-11-13 15:47:12 +0000 (Mon, 13 Nov 2006) New Revision: 1148 Modified: box/trunk/lib/backupstore/BackupStoreCheck2.cpp Log: Fixed typo. Modified: box/trunk/lib/backupstore/BackupStoreCheck2.cpp =================================================================== --- box/trunk/lib/backupstore/BackupStoreCheck2.cpp 2006-11-13 15:46:08 UTC (rev 1147) +++ box/trunk/lib/backupstore/BackupStoreCheck2.cpp 2006-11-13 15:47:12 UTC (rev 1148) @@ -653,7 +653,7 @@ if(dependsOlder != 0 && FindEntryByID(dependsOlder) == 0) { // Has an older version marked, but this doesn't exist. Remove this mark - TRACE2("Entry id %llx was marked that %llx depended on it, which doesn't exist, dependency info cleaered\n", (*i)->GetObjectID(), dependsOlder); + TRACE2("Entry id %llx was marked that %llx depended on it, which doesn't exist, dependency info cleared\n", (*i)->GetObjectID(), dependsOlder); (*i)->SetDependsOlder(0); From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:48:03 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:48:03 +0000 Subject: [Box Backup-commit] COMMIT r1149 - box/chris/general/lib/backupstore Message-ID: Author: chris Date: 2006-11-13 15:48:02 +0000 (Mon, 13 Nov 2006) New Revision: 1149 Modified: box/chris/general/lib/backupstore/BackupStoreCheck2.cpp Log: Fixed typo. Modified: box/chris/general/lib/backupstore/BackupStoreCheck2.cpp =================================================================== --- box/chris/general/lib/backupstore/BackupStoreCheck2.cpp 2006-11-13 15:47:12 UTC (rev 1148) +++ box/chris/general/lib/backupstore/BackupStoreCheck2.cpp 2006-11-13 15:48:02 UTC (rev 1149) @@ -653,7 +653,7 @@ if(dependsOlder != 0 && FindEntryByID(dependsOlder) == 0) { // Has an older version marked, but this doesn't exist. Remove this mark - TRACE2("Entry id %llx was marked that %llx depended on it, which doesn't exist, dependency info cleaered\n", (*i)->GetObjectID(), dependsOlder); + TRACE2("Entry id %llx was marked that %llx depended on it, which doesn't exist, dependency info cleared\n", (*i)->GetObjectID(), dependsOlder); (*i)->SetDependsOlder(0); From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:49:48 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:49:48 +0000 Subject: [Box Backup-commit] COMMIT r1150 - box/chris/merge/lib/server Message-ID: Author: chris Date: 2006-11-13 15:49:48 +0000 (Mon, 13 Nov 2006) New Revision: 1150 Modified: box/chris/merge/lib/server/TLSContext.cpp Log: Fix memory leak when TLSContext is reinitialised (refs #3) Modified: box/chris/merge/lib/server/TLSContext.cpp =================================================================== --- box/chris/merge/lib/server/TLSContext.cpp 2006-11-13 15:48:02 UTC (rev 1149) +++ box/chris/merge/lib/server/TLSContext.cpp 2006-11-13 15:49:48 UTC (rev 1150) @@ -61,6 +61,11 @@ // -------------------------------------------------------------------------- void TLSContext::Initialise(bool AsServer, const char *CertificatesFile, const char *PrivateKeyFile, const char *TrustedCAsFile) { + if(mpContext != 0) + { + ::SSL_CTX_free(mpContext); + } + mpContext = ::SSL_CTX_new(AsServer?TLSv1_server_method():TLSv1_client_method()); if(mpContext == NULL) { From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:50:21 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:50:21 +0000 Subject: [Box Backup-commit] COMMIT r1151 - box/chris/merge/lib/server Message-ID: Author: chris Date: 2006-11-13 15:50:21 +0000 (Mon, 13 Nov 2006) New Revision: 1151 Modified: box/chris/merge/lib/server/ServerStream.h Log: Fixed typo (refs #3) Modified: box/chris/merge/lib/server/ServerStream.h =================================================================== --- box/chris/merge/lib/server/ServerStream.h 2006-11-13 15:49:48 UTC (rev 1150) +++ box/chris/merge/lib/server/ServerStream.h 2006-11-13 15:50:21 UTC (rev 1151) @@ -259,7 +259,7 @@ else { #endif // !WIN32 - // Just handle in this connection + // Just handle in this process SetProcessTitle("handling"); HandleConnection(*connection); SetProcessTitle("idle"); From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:53:29 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:53:29 +0000 Subject: [Box Backup-commit] COMMIT r1152 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2006-11-13 15:53:29 +0000 (Mon, 13 Nov 2006) New Revision: 1152 Modified: box/chris/merge/lib/common/Box.h Log: Declare MEMLEAKFINDER_INIT and MEMLEAKFINDER_NO_LEAKS macros which reference function and class in DebugMemLeakFinder only in debug mode (refs #3) Modified: box/chris/merge/lib/common/Box.h =================================================================== --- box/chris/merge/lib/common/Box.h 2006-11-13 15:50:21 UTC (rev 1151) +++ box/chris/merge/lib/common/Box.h 2006-11-13 15:53:29 UTC (rev 1152) @@ -95,16 +95,19 @@ // Memory leak testing #include "MemLeakFinder.h" #define MEMLEAKFINDER_NOT_A_LEAK(x) memleakfinder_notaleak(x); + #define MEMLEAKFINDER_NO_LEAKS MemLeakSuppressionGuard _guard; + #define MEMLEAKFINDER_INIT memleakfinder_init(); #define MEMLEAKFINDER_START {memleakfinder_global_enable = true;} - #define MEMLEAKFINDER_STOP {memleakfinder_global_enable = false;} + #define MEMLEAKFINDER_STOP {memleakfinder_global_enable = false;} #else #define DEBUG_NEW new #define MEMLEAKFINDER_NOT_A_LEAK(x) + #define MEMLEAKFINDER_NO_LEAKS + #define MEMLEAKFINDER_INIT #define MEMLEAKFINDER_START #define MEMLEAKFINDER_STOP #endif - #define THROW_EXCEPTION(type, subtype) \ { \ OPTIONAL_DO_BACKTRACE \ From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:54:49 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:54:49 +0000 Subject: [Box Backup-commit] COMMIT r1153 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2006-11-13 15:54:49 +0000 (Mon, 13 Nov 2006) New Revision: 1153 Modified: box/chris/merge/lib/common/Utils.cpp Log: Free backtrace strings even in debug mode by suppressing warnings from DebugMemLeakFinder, to avoid a memory leak (refs #3) Modified: box/chris/merge/lib/common/Utils.cpp =================================================================== --- box/chris/merge/lib/common/Utils.cpp 2006-11-13 15:53:29 UTC (rev 1152) +++ box/chris/merge/lib/common/Utils.cpp 2006-11-13 15:54:49 UTC (rev 1153) @@ -74,11 +74,13 @@ printf ("Obtained %zd stack frames.\n", size); for(i = 0; i < size; i++) + { printf("%s\n", strings[i]); + } -#ifndef MEMLEAKFINDER_MALLOC_MONITORING_DEFINED +#include "MemLeakFindOff.h" free (strings); -#endif +#include "MemLeakFindOn.h" } #endif From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:56:09 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:56:09 +0000 Subject: [Box Backup-commit] COMMIT r1154 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2006-11-13 15:56:09 +0000 (Mon, 13 Nov 2006) New Revision: 1154 Modified: box/chris/merge/lib/common/Timer.cpp box/chris/merge/lib/common/Timer.h Log: Use a static pointer rather than a static object, to allow it to be freed in Timers::Cleanup, removing a reported memory leak (refs #9) Modified: box/chris/merge/lib/common/Timer.cpp =================================================================== --- box/chris/merge/lib/common/Timer.cpp 2006-11-13 15:54:49 UTC (rev 1153) +++ box/chris/merge/lib/common/Timer.cpp 2006-11-13 15:56:09 UTC (rev 1154) @@ -16,8 +16,7 @@ #include "MemLeakFindOn.h" -std::vector Timers::sTimers; -bool Timers::sInitialised = false; +std::vector* Timers::spTimers = NULL; // -------------------------------------------------------------------------- // @@ -42,7 +41,7 @@ // -------------------------------------------------------------------------- void Timers::Init() { - ASSERT(!sInitialised); + ASSERT(!spTimers); #ifdef PLATFORM_CYGWIN ASSERT(::signal(SIGALRM, TimerSigHandler) == 0); @@ -54,7 +53,7 @@ ASSERT(::signal(SIGALRM, TimerSigHandler) == 0); #endif // PLATFORM_CYGWIN - sInitialised = true; + spTimers = new std::vector; } // -------------------------------------------------------------------------- @@ -67,7 +66,7 @@ // -------------------------------------------------------------------------- void Timers::Cleanup() { - ASSERT(sInitialised); + ASSERT(spTimers); #ifdef PLATFORM_CYGWIN ASSERT(::signal(SIGALRM, NULL) == TimerSigHandler); @@ -79,9 +78,9 @@ ASSERT(::signal(SIGALRM, NULL) == TimerSigHandler); #endif // PLATFORM_CYGWIN - sTimers.clear(); - - sInitialised = false; + spTimers->clear(); + delete spTimers; + spTimers = NULL; } // -------------------------------------------------------------------------- @@ -94,8 +93,8 @@ // -------------------------------------------------------------------------- void Timers::Add(Timer& rTimer) { - ASSERT(sInitialised); - sTimers.push_back(&rTimer); + ASSERT(spTimers); + spTimers->push_back(&rTimer); Reschedule(); } @@ -110,18 +109,18 @@ // -------------------------------------------------------------------------- void Timers::Remove(Timer& rTimer) { - ASSERT(sInitialised); + ASSERT(spTimers); bool restart = true; while (restart) { restart = false; - for (std::vector::iterator i = sTimers.begin(); - i != sTimers.end(); i++) + for (std::vector::iterator i = spTimers->begin(); + i != spTimers->end(); i++) { if (&rTimer == *i) { - sTimers.erase(i); + spTimers->erase(i); restart = true; break; } @@ -141,13 +140,13 @@ // -------------------------------------------------------------------------- void Timers::Reschedule() { - ASSERT(sInitialised); + ASSERT(spTimers); box_time_t timeNow = GetCurrentBoxTime(); box_time_t timeToNextEvent = 0; - for (std::vector::iterator i = sTimers.begin(); - i != sTimers.end(); i++) + for (std::vector::iterator i = spTimers->begin(); + i != spTimers->end(); i++) { Timer& rTimer = **i; ASSERT(!rTimer.HasExpired()); @@ -192,11 +191,11 @@ // -------------------------------------------------------------------------- void Timers::Signal() { - ASSERT(sInitialised); + ASSERT(spTimers); box_time_t timeNow = GetCurrentBoxTime(); - std::vector timersCopy = sTimers; + std::vector timersCopy = *spTimers; for (std::vector::iterator i = timersCopy.begin(); i != timersCopy.end(); i++) @@ -211,7 +210,7 @@ rTimer.OnExpire(); } } - + Reschedule(); } Modified: box/chris/merge/lib/common/Timer.h =================================================================== --- box/chris/merge/lib/common/Timer.h 2006-11-13 15:54:49 UTC (rev 1153) +++ box/chris/merge/lib/common/Timer.h 2006-11-13 15:56:09 UTC (rev 1154) @@ -48,8 +48,7 @@ class Timers { private: - static std::vector sTimers; - static bool sInitialised; + static std::vector* spTimers; static void Reschedule(); public: From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:57:21 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:57:21 +0000 Subject: [Box Backup-commit] COMMIT r1155 - box/chris/merge/bin/bbackupctl Message-ID: Author: chris Date: 2006-11-13 15:57:20 +0000 (Mon, 13 Nov 2006) New Revision: 1155 Modified: box/chris/merge/bin/bbackupctl/bbackupctl.cpp Log: openlog inside MAINHELPER block, to ensure that any memory leaks will be caught (refs #3) Modified: box/chris/merge/bin/bbackupctl/bbackupctl.cpp =================================================================== --- box/chris/merge/bin/bbackupctl/bbackupctl.cpp 2006-11-13 15:56:09 UTC (rev 1154) +++ box/chris/merge/bin/bbackupctl/bbackupctl.cpp 2006-11-13 15:57:20 UTC (rev 1155) @@ -56,15 +56,15 @@ { int returnCode = 0; -#if defined WIN32 && ! defined NDEBUG - ::openlog("Box Backup (bbackupctl)", 0, 0); -#endif - MAINHELPER_SETUP_MEMORY_LEAK_EXIT_REPORT("bbackupctl.memleaks", "bbackupctl") MAINHELPER_START +#if defined WIN32 && ! defined NDEBUG + ::openlog("Box Backup (bbackupctl)", 0, 0); +#endif + // Filename for configuraiton file? const char *configFilename = BOX_FILE_BBACKUPD_DEFAULT_CONFIG; From boxbackup-dev at fluffy.co.uk Mon Nov 13 15:58:53 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 15:58:53 +0000 Subject: [Box Backup-commit] COMMIT r1156 - box/chris/merge/bin/bbackupquery Message-ID: Author: chris Date: 2006-11-13 15:58:53 +0000 (Mon, 13 Nov 2006) New Revision: 1156 Modified: box/chris/merge/bin/bbackupquery/bbackupquery.cpp Log: Setup MAINHELPER as early as possible, and clear it as late as possible, for consistency (refs #3) Modified: box/chris/merge/bin/bbackupquery/bbackupquery.cpp =================================================================== --- box/chris/merge/bin/bbackupquery/bbackupquery.cpp 2006-11-13 15:57:20 UTC (rev 1155) +++ box/chris/merge/bin/bbackupquery/bbackupquery.cpp 2006-11-13 15:58:53 UTC (rev 1156) @@ -63,7 +63,10 @@ int main(int argc, const char *argv[]) { + int returnCode = 0; + MAINHELPER_SETUP_MEMORY_LEAK_EXIT_REPORT("bbackupquery.memleaks", "bbackupquery") + MAINHELPER_START #ifdef WIN32 WSADATA info; @@ -83,10 +86,6 @@ BoxDebugTraceOn = false; #endif - int returnCode = 0; - - MAINHELPER_START - FILE *logFile = 0; // Filename for configuraiton file? @@ -314,13 +313,13 @@ // Let everything be cleaned up on exit. - MAINHELPER_END - #ifdef WIN32 // Clean up our sockets WSACleanup(); #endif + MAINHELPER_END + return returnCode; } From boxbackup-dev at fluffy.co.uk Mon Nov 13 16:00:27 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 16:00:27 +0000 Subject: [Box Backup-commit] COMMIT r1157 - box/chris/merge/bin/bbackupquery Message-ID: Author: chris Date: 2006-11-13 16:00:27 +0000 (Mon, 13 Nov 2006) New Revision: 1157 Modified: box/chris/merge/bin/bbackupquery/bbackupquery.cpp Log: Fixed typo. Modified: box/chris/merge/bin/bbackupquery/bbackupquery.cpp =================================================================== --- box/chris/merge/bin/bbackupquery/bbackupquery.cpp 2006-11-13 15:58:53 UTC (rev 1156) +++ box/chris/merge/bin/bbackupquery/bbackupquery.cpp 2006-11-13 16:00:27 UTC (rev 1157) @@ -88,7 +88,7 @@ FILE *logFile = 0; - // Filename for configuraiton file? + // Filename for configuration file? const char *configFilename = BOX_FILE_BBACKUPD_DEFAULT_CONFIG; // Flags From boxbackup-dev at fluffy.co.uk Mon Nov 13 16:00:52 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 16:00:52 +0000 Subject: [Box Backup-commit] COMMIT r1158 - box/trunk/bin/bbackupquery Message-ID: Author: chris Date: 2006-11-13 16:00:52 +0000 (Mon, 13 Nov 2006) New Revision: 1158 Modified: box/trunk/bin/bbackupquery/bbackupquery.cpp Log: Fixed typo. Modified: box/trunk/bin/bbackupquery/bbackupquery.cpp =================================================================== --- box/trunk/bin/bbackupquery/bbackupquery.cpp 2006-11-13 16:00:27 UTC (rev 1157) +++ box/trunk/bin/bbackupquery/bbackupquery.cpp 2006-11-13 16:00:52 UTC (rev 1158) @@ -89,7 +89,7 @@ FILE *logFile = 0; - // Filename for configuraiton file? + // Filename for configuration file? const char *configFilename = BOX_FILE_BBACKUPD_DEFAULT_CONFIG; // Flags From boxbackup-dev at fluffy.co.uk Mon Nov 13 16:07:36 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 16:07:36 +0000 Subject: [Box Backup-commit] COMMIT r1159 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2006-11-13 16:07:36 +0000 (Mon, 13 Nov 2006) New Revision: 1159 Modified: box/chris/merge/lib/common/DebugMemLeakFinder.cpp box/chris/merge/lib/common/MemLeakFinder.h Log: * Track memory leaks in allocations via the standard libraries, and avoid malloc/delete mismatches, by overriding standard new operator. * Added another global enable flag to memleak finder, which is used to mark the end of static allocations and the start of dynamic code, since the memory leak detection is done before cleanup of static objects. * Added a public guard class, to allow safe scoped disabling of memory leak detection. * Added InternalAllocGuard to protect against recursive loops when allocating memory inside the memory leak checker. (refs #3) Modified: box/chris/merge/lib/common/DebugMemLeakFinder.cpp =================================================================== --- box/chris/merge/lib/common/DebugMemLeakFinder.cpp 2006-11-13 16:00:52 UTC (rev 1158) +++ box/chris/merge/lib/common/DebugMemLeakFinder.cpp 2006-11-13 16:07:36 UTC (rev 1159) @@ -25,6 +25,7 @@ #include #include +static bool memleakfinder_initialised = false; bool memleakfinder_global_enable = false; typedef struct @@ -58,8 +59,38 @@ size_t sNotLeaksPreNum = 0; } +void memleakfinder_init() +{ + ASSERT(!memleakfinder_initialised); + memleakfinder_initialised = true; +} + +MemLeakSuppressionGuard::MemLeakSuppressionGuard() +{ + ASSERT(memleakfinder_global_enable); + memleakfinder_global_enable = false; +} + +MemLeakSuppressionGuard::~MemLeakSuppressionGuard() +{ + ASSERT(!memleakfinder_global_enable); + memleakfinder_global_enable = true; +} + +// these functions may well allocate memory, which we don't want to track. +static int sInternalAllocDepth = 0; + +class InternalAllocGuard +{ + public: + InternalAllocGuard () { sInternalAllocDepth++; } + ~InternalAllocGuard() { sInternalAllocDepth--; } +}; + void memleakfinder_malloc_add_block(void *b, size_t size, const char *file, int line) { + InternalAllocGuard guard; + if(b != 0) { MallocBlockInfo i; @@ -75,11 +106,13 @@ } } - void *memleakfinder_malloc(size_t size, const char *file, int line) { + InternalAllocGuard guard; + void *b = ::malloc(size); if(!memleakfinder_global_enable) return b; + if(!memleakfinder_initialised) return b; memleakfinder_malloc_add_block(b, size, file, line); @@ -89,7 +122,9 @@ void *memleakfinder_realloc(void *ptr, size_t size) { - if(!memleakfinder_global_enable) + InternalAllocGuard guard; + + if(!memleakfinder_global_enable || !memleakfinder_initialised) { return ::realloc(ptr, size); } @@ -133,7 +168,9 @@ void memleakfinder_free(void *ptr) { - if(memleakfinder_global_enable) + InternalAllocGuard guard; + + if(memleakfinder_global_enable && memleakfinder_initialised) { // Check it's been allocated std::map::iterator i(sMallocBlocks.find(ptr)); @@ -143,7 +180,7 @@ } else { - TRACE1("Block %x freed, but not known. Error? Or allocated in startup static allocation?\n", ptr); + TRACE1("Block %p freed, but not known. Error? Or allocated in startup static allocation?\n", ptr); } if(sTrackMallocInSection) @@ -160,24 +197,34 @@ void memleakfinder_notaleak_insert_pre() { + InternalAllocGuard guard; + if(!memleakfinder_global_enable) return; + if(!memleakfinder_initialised) return; + for(size_t l = 0; l < sNotLeaksPreNum; l++) { sNotLeaks.insert(sNotLeaksPre[l]); } + sNotLeaksPreNum = 0; } bool is_leak(void *ptr) { + InternalAllocGuard guard; + + ASSERT(memleakfinder_initialised); memleakfinder_notaleak_insert_pre(); return sNotLeaks.find(ptr) == sNotLeaks.end(); } void memleakfinder_notaleak(void *ptr) { + InternalAllocGuard guard; + memleakfinder_notaleak_insert_pre(); - if(memleakfinder_global_enable) + if(memleakfinder_global_enable && memleakfinder_initialised) { sNotLeaks.insert(ptr); } @@ -206,6 +253,9 @@ // start monitoring a section of code void memleakfinder_startsectionmonitor() { + InternalAllocGuard guard; + + ASSERT(memleakfinder_initialised); sTrackMallocInSection = true; sSectionMallocBlocks.clear(); sTrackObjectsInSection = true; @@ -215,6 +265,10 @@ // trace all blocks allocated and still allocated since memleakfinder_startsectionmonitor() called void memleakfinder_traceblocksinsection() { + InternalAllocGuard guard; + + ASSERT(memleakfinder_initialised); + std::set::iterator s(sSectionMallocBlocks.begin()); for(; s != sSectionMallocBlocks.end(); ++s) { @@ -225,17 +279,21 @@ } else { - TRACE4("Block 0x%08p size %d allocated at %s:%d\n", i->first, i->second.size, i->second.file, i->second.line); + TRACE4("Block %p size %d allocated at %s:%d\n", i->first, i->second.size, i->second.file, i->second.line); } } for(std::map::const_iterator i(sSectionObjectBlocks.begin()); i != sSectionObjectBlocks.end(); ++i) { - TRACE5("Object%s 0x%08p size %d allocated at %s:%d\n", i->second.array?" []":"", i->first, i->second.size, i->second.file, i->second.line); + TRACE5("Object%s %p size %d allocated at %s:%d\n", i->second.array?" []":"", i->first, i->second.size, i->second.file, i->second.line); } } int memleakfinder_numleaks() { + InternalAllocGuard guard; + + ASSERT(memleakfinder_initialised); + int n = 0; for(std::map::const_iterator i(sMallocBlocks.begin()); i != sMallocBlocks.end(); ++i) @@ -245,6 +303,7 @@ for(std::map::const_iterator i(sObjectBlocks.begin()); i != sObjectBlocks.end(); ++i) { + const ObjectInfo& rInfo = i->second; if(is_leak(i->first)) ++n; } @@ -253,6 +312,8 @@ void memleakfinder_reportleaks_file(FILE *file) { + InternalAllocGuard guard; + for(std::map::const_iterator i(sMallocBlocks.begin()); i != sMallocBlocks.end(); ++i) { if(is_leak(i->first)) ::fprintf(file, "Block 0x%p size %d allocated at %s:%d\n", i->first, i->second.size, i->second.file, i->second.line); @@ -265,12 +326,16 @@ void memleakfinder_reportleaks() { + InternalAllocGuard guard; + // report to stdout memleakfinder_reportleaks_file(stdout); } void memleakfinder_reportleaks_appendfile(const char *filename, const char *markertext) { + InternalAllocGuard guard; + FILE *file = ::fopen(filename, "a"); if(file != 0) { @@ -317,7 +382,10 @@ void add_object_block(void *block, size_t size, const char *file, int line, bool array) { + InternalAllocGuard guard; + if(!memleakfinder_global_enable) return; + if(!memleakfinder_initialised) return; if(block != 0) { @@ -337,7 +405,10 @@ void remove_object_block(void *block) { + InternalAllocGuard guard; + if(!memleakfinder_global_enable) return; + if(!memleakfinder_initialised) return; std::map::iterator i(sObjectBlocks.find(block)); if(i != sObjectBlocks.end()) @@ -357,34 +428,64 @@ // If it's not in the list, just ignore it, as lots of stuff goes this way... } -void *operator new(size_t size, const char *file, int line) +static void *internal_new(size_t size, const char *file, int line) { - void *r = ::malloc(size); - add_object_block(r, size, file, line, false); - //TRACE4("new(), %d, %s, %d, %08x\n", size, file, line, r); + void *r; + + { + InternalAllocGuard guard; + r = ::malloc(size); + } + + if (sInternalAllocDepth == 0) + { + InternalAllocGuard guard; + add_object_block(r, size, file, line, false); + //TRACE4("new(), %d, %s, %d, %08x\n", size, file, line, r); + } + return r; } +void *operator new(size_t size, const char *file, int line) +{ + return internal_new(size, file, line); +} + void *operator new[](size_t size, const char *file, int line) { - void *r = ::malloc(size); - add_object_block(r, size, file, line, true); - //TRACE4("new[](), %d, %s, %d, %08x\n", size, file, line, r); - return r; + return internal_new(size, file, line); } -void operator delete[](void *ptr) throw () +// where there is no doctor... need to override standard new() too +// http://www.relisoft.com/book/tech/9new.html +void *operator new(size_t size) { + return internal_new(size, "standard libraries", 0); +} + +void *operator new[](size_t size) +{ + return internal_new(size, "standard libraries", 0); +} + +void internal_delete(void *ptr) +{ + InternalAllocGuard guard; + ::free(ptr); remove_object_block(ptr); //TRACE1("delete[]() called, %08x\n", ptr); } +void operator delete[](void *ptr) throw () +{ + internal_delete(ptr); +} + void operator delete(void *ptr) throw () { - ::free(ptr); - remove_object_block(ptr); - //TRACE1("delete() called, %08x\n", ptr); + internal_delete(ptr); } #endif // NDEBUG Modified: box/chris/merge/lib/common/MemLeakFinder.h =================================================================== --- box/chris/merge/lib/common/MemLeakFinder.h 2006-11-13 16:00:52 UTC (rev 1158) +++ box/chris/merge/lib/common/MemLeakFinder.h 2006-11-13 16:07:36 UTC (rev 1159) @@ -20,6 +20,13 @@ // global enable flag extern bool memleakfinder_global_enable; +class MemLeakSuppressionGuard +{ + public: + MemLeakSuppressionGuard(); + ~MemLeakSuppressionGuard(); +}; + extern "C" { void *memleakfinder_malloc(size_t size, const char *file, int line); @@ -27,6 +34,8 @@ void memleakfinder_free(void *ptr); } +void memleakfinder_init(); + int memleakfinder_numleaks(); void memleakfinder_reportleaks(); @@ -41,10 +50,12 @@ void memleakfinder_notaleak(void *ptr); -void *operator new(size_t size, const char *file, int line); +void *operator new (size_t size, const char *file, int line); void *operator new[](size_t size, const char *file, int line); +void *operator new (size_t size); +void *operator new[](size_t size); -void operator delete(void *ptr) throw (); +void operator delete (void *ptr) throw (); void operator delete[](void *ptr) throw (); // define the malloc functions now, if required @@ -55,6 +66,5 @@ #define MEMLEAKFINDER_MALLOC_MONITORING_DEFINED #endif - #endif // MEMLEAKFINDER__H From boxbackup-dev at fluffy.co.uk Mon Nov 13 16:10:00 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 16:10:00 +0000 Subject: [Box Backup-commit] COMMIT r1160 - in box/chris/merge/test: backupstore backupstorefix Message-ID: Author: chris Date: 2006-11-13 16:10:00 +0000 (Mon, 13 Nov 2006) New Revision: 1160 Modified: box/chris/merge/test/backupstore/testbackupstore.cpp box/chris/merge/test/backupstorefix/testbackupstorefix.cpp Log: Fix memory leak false alarms caused by modifying static objects (refs #3) Modified: box/chris/merge/test/backupstore/testbackupstore.cpp =================================================================== --- box/chris/merge/test/backupstore/testbackupstore.cpp 2006-11-13 16:07:36 UTC (rev 1159) +++ box/chris/merge/test/backupstore/testbackupstore.cpp 2006-11-13 16:10:00 UTC (rev 1160) @@ -1926,15 +1926,20 @@ // for seeing what's going on. BackupClientCryptoKeys_Setup("testfiles/bbackupd.keys"); - // encode in some filenames -- can't do static initialisation because the key won't be set up when these are initialised - for(unsigned int l = 0; l < sizeof(ens_filenames) / sizeof(ens_filenames[0]); ++l) + // encode in some filenames -- can't do static initialisation + // because the key won't be set up when these are initialised { - ens[l].fn = BackupStoreFilenameClear(ens_filenames[l]); + MEMLEAKFINDER_NO_LEAKS + + for(unsigned int l = 0; l < sizeof(ens_filenames) / sizeof(ens_filenames[0]); ++l) + { + ens[l].fn = BackupStoreFilenameClear(ens_filenames[l]); + } + for(unsigned int l = 0; l < sizeof(uploads_filenames) / sizeof(uploads_filenames[0]); ++l) + { + uploads[l].name = BackupStoreFilenameClear(uploads_filenames[l]); + } } - for(unsigned int l = 0; l < sizeof(uploads_filenames) / sizeof(uploads_filenames[0]); ++l) - { - uploads[l].name = BackupStoreFilenameClear(uploads_filenames[l]); - } // Trace errors out SET_DEBUG_SSLLIB_TRACE_ERRORS Modified: box/chris/merge/test/backupstorefix/testbackupstorefix.cpp =================================================================== --- box/chris/merge/test/backupstorefix/testbackupstorefix.cpp 2006-11-13 16:07:36 UTC (rev 1159) +++ box/chris/merge/test/backupstorefix/testbackupstorefix.cpp 2006-11-13 16:10:00 UTC (rev 1160) @@ -202,9 +202,12 @@ void test_dir_fixing() { - fnames[0].SetAsClearFilename("x1"); - fnames[1].SetAsClearFilename("x2"); - fnames[2].SetAsClearFilename("x3"); + { + MEMLEAKFINDER_NO_LEAKS; + fnames[0].SetAsClearFilename("x1"); + fnames[1].SetAsClearFilename("x2"); + fnames[2].SetAsClearFilename("x3"); + } { BackupStoreDirectory dir; @@ -344,6 +347,7 @@ TEST_THAT(::sscanf(line, "%x %s %s", &id, flags, name) == 3); bool isDir = (::strcmp(flags, "-d---") == 0); //TRACE3("%x,%d,%s\n", id, isDir, name); + MEMLEAKFINDER_NO_LEAKS; nameToID[std::string(name)] = id; objectIsDir[id] = isDir; } From boxbackup-dev at fluffy.co.uk Mon Nov 13 16:12:09 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 16:12:09 +0000 Subject: [Box Backup-commit] COMMIT r1161 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2006-11-13 16:12:09 +0000 (Mon, 13 Nov 2006) New Revision: 1161 Modified: box/chris/merge/lib/common/MainHelper.h Log: Initialise memleak finder at the start of every program that uses MAINHELPER (all except unit tests). (refs #3) Modified: box/chris/merge/lib/common/MainHelper.h =================================================================== --- box/chris/merge/lib/common/MainHelper.h 2006-11-13 16:10:00 UTC (rev 1160) +++ box/chris/merge/lib/common/MainHelper.h 2006-11-13 16:12:09 UTC (rev 1161) @@ -17,6 +17,7 @@ #define MAINHELPER_START \ if(argc == 2 && ::strcmp(argv[1], "--version") == 0) \ { printf(BOX_VERSION "\n"); return 0; } \ + MEMLEAKFINDER_INIT \ MEMLEAKFINDER_START \ try { #define MAINHELPER_END \ From boxbackup-dev at fluffy.co.uk Mon Nov 13 16:13:20 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 16:13:20 +0000 Subject: [Box Backup-commit] COMMIT r1162 - box/chris/merge/bin/bbackupd Message-ID: Author: chris Date: 2006-11-13 16:13:20 +0000 (Mon, 13 Nov 2006) New Revision: 1162 Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp Log: Initialise cross-platform timers on all platforms, remove win32-specific code (refs #9) Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2006-11-13 16:12:09 UTC (rev 1161) +++ box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2006-11-13 16:13:20 UTC (rev 1162) @@ -75,6 +75,7 @@ #include "IOStreamGetLine.h" #include "Conversion.h" #include "Archive.h" +#include "Timer.h" #include "MemLeakFindOn.h" @@ -429,21 +430,19 @@ // -------------------------------------------------------------------------- void BackupDaemon::Run() { + // initialise global timer mechanism + Timers::Init(); + #ifdef WIN32 - // init our own timer for file diff timeouts - InitTimer(); - try { Run2(); } catch(...) { - FiniTimer(); + Timers::Cleanup(); throw; } - - FiniTimer(); #else // ! WIN32 // Ignore SIGPIPE (so that if a command connection is broken, the daemon doesn't terminate) ::signal(SIGPIPE, SIG_IGN); @@ -487,6 +486,8 @@ mpCommandSocketInfo = 0; } + Timers::Cleanup(); + throw; } @@ -497,6 +498,8 @@ mpCommandSocketInfo = 0; } #endif + + Timers::Cleanup(); } // -------------------------------------------------------------------------- From boxbackup-dev at fluffy.co.uk Mon Nov 13 16:14:08 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 16:14:08 +0000 Subject: [Box Backup-commit] COMMIT r1163 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2006-11-13 16:14:08 +0000 (Mon, 13 Nov 2006) New Revision: 1163 Modified: box/chris/merge/lib/common/Test.h Log: Fixed control reaching end of non-void functions (refs #3) Modified: box/chris/merge/lib/common/Test.h =================================================================== --- box/chris/merge/lib/common/Test.h 2006-11-13 16:13:20 UTC (rev 1162) +++ box/chris/merge/lib/common/Test.h 2006-11-13 16:14:08 UTC (rev 1163) @@ -239,7 +239,9 @@ inline bool KillServerInternal(int pid) { - TEST_THAT(SendCommands("terminate")); + bool sent = SendCommands("terminate"); + TEST_THAT(sent); + return sent; } #else // !WIN32 @@ -259,14 +261,19 @@ inline bool KillServerInternal(int pid) { if(pid == 0 || pid == -1) return false; - TEST_THAT(::kill(pid, SIGTERM) != -1); + bool killed = (::kill(pid, SIGTERM) == 0); + TEST_THAT(killed); + return killed; } #endif // WIN32 inline bool KillServer(int pid) { - KillServerInternal(pid); + if (!KillServerInternal(pid)) + { + return false; + } for (int i = 0; i < 30; i++) { From boxbackup-dev at fluffy.co.uk Mon Nov 13 16:16:54 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 16:16:54 +0000 Subject: [Box Backup-commit] COMMIT r1164 - box/chris/merge/infrastructure Message-ID: Author: chris Date: 2006-11-13 16:16:54 +0000 (Mon, 13 Nov 2006) New Revision: 1164 Modified: box/chris/merge/infrastructure/buildenv-testmain-template.cpp Log: Initialise memory leak finder in all unit tests (refs #3) Modified: box/chris/merge/infrastructure/buildenv-testmain-template.cpp =================================================================== --- box/chris/merge/infrastructure/buildenv-testmain-template.cpp 2006-11-13 16:14:08 UTC (rev 1163) +++ box/chris/merge/infrastructure/buildenv-testmain-template.cpp 2006-11-13 16:16:54 UTC (rev 1164) @@ -114,6 +114,10 @@ } try { + #ifdef BOX_MEMORY_LEAK_TESTING + memleakfinder_init(); + #endif + int returncode = test(argc, argv); // check for memory leaks, if enabled From boxbackup-dev at fluffy.co.uk Mon Nov 13 16:18:03 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 16:18:03 +0000 Subject: [Box Backup-commit] COMMIT r1165 - box/chris/merge/infrastructure Message-ID: Author: chris Date: 2006-11-13 16:18:03 +0000 (Mon, 13 Nov 2006) New Revision: 1165 Modified: box/chris/merge/infrastructure/buildenv-testmain-template.cpp Log: Initialise timers in all unit tests (refs #9) Modified: box/chris/merge/infrastructure/buildenv-testmain-template.cpp =================================================================== --- box/chris/merge/infrastructure/buildenv-testmain-template.cpp 2006-11-13 16:16:54 UTC (rev 1164) +++ box/chris/merge/infrastructure/buildenv-testmain-template.cpp 2006-11-13 16:18:03 UTC (rev 1165) @@ -30,6 +30,8 @@ #include #endif +#include "Timer.h" + #include "MemLeakFindOn.h" int test(int argc, const char *argv[]); @@ -118,7 +120,9 @@ memleakfinder_init(); #endif + Timers::Init(); int returncode = test(argc, argv); + Timers::Cleanup(); // check for memory leaks, if enabled #ifdef BOX_MEMORY_LEAK_TESTING From boxbackup-dev at fluffy.co.uk Mon Nov 13 16:19:59 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 16:19:59 +0000 Subject: [Box Backup-commit] COMMIT r1166 - box/chris/merge/lib/raidfile Message-ID: Author: chris Date: 2006-11-13 16:19:59 +0000 (Mon, 13 Nov 2006) New Revision: 1166 Modified: box/chris/merge/lib/raidfile/RaidFileController.cpp Log: Fix memory leak false alarms caused by static allocations. Modified: box/chris/merge/lib/raidfile/RaidFileController.cpp =================================================================== --- box/chris/merge/lib/raidfile/RaidFileController.cpp 2006-11-13 16:18:03 UTC (rev 1165) +++ box/chris/merge/lib/raidfile/RaidFileController.cpp 2006-11-13 16:19:59 UTC (rev 1166) @@ -66,6 +66,8 @@ // -------------------------------------------------------------------------- void RaidFileController::Initialise(const char *ConfigFilename) { + MEMLEAKFINDER_NO_LEAKS; + static const ConfigurationVerifyKey verifykeys[] = { {"SetNumber", 0, ConfigTest_Exists | ConfigTest_IsInt, 0}, From boxbackup-dev at fluffy.co.uk Mon Nov 13 19:52:05 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 19:52:05 +0000 Subject: [Box Backup-commit] COMMIT r1167 - box/chris/merge/bin/bbstored Message-ID: Author: chris Date: 2006-11-13 19:52:05 +0000 (Mon, 13 Nov 2006) New Revision: 1167 Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp Log: Revert [1096] as it causes infinite loops if the listening socket can't be opened (refs #3) Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp =================================================================== --- box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2006-11-13 16:19:59 UTC (rev 1166) +++ box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2006-11-13 19:52:05 UTC (rev 1167) @@ -227,38 +227,13 @@ else { // In server process -- use the base class to do the magic - try - { - ServerTLS::Run(); - } - catch(BoxException &e) - { - ::syslog(LOG_ERR, "%s: disconnecting due to " - "exception %s (%d/%d)", DaemonName(), - e.what(), e.GetType(), e.GetSubType()); - } - catch(std::exception &e) - { - ::syslog(LOG_ERR, "%s: disconnecting due to " - "exception %s", DaemonName(), e.what()); - } - catch(...) - { - ::syslog(LOG_ERR, "%s: disconnecting due to " - "unknown exception", DaemonName()); - } - - if (!mInterProcessCommsSocket.IsOpened()) - { - return; - } - + ServerTLS::Run(); + // Why did it stop? Tell the housekeeping process to do the same if(IsReloadConfigWanted()) { mInterProcessCommsSocket.Write("h\n", 2); } - if(IsTerminateWanted()) { mInterProcessCommsSocket.Write("t\n", 2); @@ -266,17 +241,49 @@ } } - // -------------------------------------------------------------------------- // // Function // Name: BackupStoreDaemon::Connection(SocketStreamTLS &) -// Purpose: Handles a connection +// Purpose: Handles a connection, by catching exceptions and +// delegating to Connection2 // Created: 2003/08/20 // // -------------------------------------------------------------------------- void BackupStoreDaemon::Connection(SocketStreamTLS &rStream) { + try + { + Connection2(rStream); + } + catch(BoxException &e) + { + ::syslog(LOG_ERR, "%s: disconnecting due to " + "exception %s (%d/%d)", DaemonName(), + e.what(), e.GetType(), e.GetSubType()); + } + catch(std::exception &e) + { + ::syslog(LOG_ERR, "%s: disconnecting due to " + "exception %s", DaemonName(), e.what()); + } + catch(...) + { + ::syslog(LOG_ERR, "%s: disconnecting due to " + "unknown exception", DaemonName()); + } +} + +// -------------------------------------------------------------------------- +// +// Function +// Name: BackupStoreDaemon::Connection2(SocketStreamTLS &) +// Purpose: Handles a connection from bbackupd +// Created: 2006/11/12 +// +// -------------------------------------------------------------------------- +void BackupStoreDaemon::Connection2(SocketStreamTLS &rStream) +{ // Get the common name from the certificate std::string clientCommonName(rStream.GetPeerCommonName()); From boxbackup-dev at fluffy.co.uk Mon Nov 13 20:01:39 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 20:01:39 +0000 Subject: [Box Backup-commit] COMMIT r1168 - box/chris/merge/bin/bbstored Message-ID: Author: chris Date: 2006-11-13 20:01:39 +0000 (Mon, 13 Nov 2006) New Revision: 1168 Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp Log: Properly revert [1096] (refs #3) Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp =================================================================== --- box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2006-11-13 19:52:05 UTC (rev 1167) +++ box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2006-11-13 20:01:39 UTC (rev 1168) @@ -241,49 +241,17 @@ } } + // -------------------------------------------------------------------------- // // Function // Name: BackupStoreDaemon::Connection(SocketStreamTLS &) -// Purpose: Handles a connection, by catching exceptions and -// delegating to Connection2 +// Purpose: Handles a connection // Created: 2003/08/20 // // -------------------------------------------------------------------------- void BackupStoreDaemon::Connection(SocketStreamTLS &rStream) { - try - { - Connection2(rStream); - } - catch(BoxException &e) - { - ::syslog(LOG_ERR, "%s: disconnecting due to " - "exception %s (%d/%d)", DaemonName(), - e.what(), e.GetType(), e.GetSubType()); - } - catch(std::exception &e) - { - ::syslog(LOG_ERR, "%s: disconnecting due to " - "exception %s", DaemonName(), e.what()); - } - catch(...) - { - ::syslog(LOG_ERR, "%s: disconnecting due to " - "unknown exception", DaemonName()); - } -} - -// -------------------------------------------------------------------------- -// -// Function -// Name: BackupStoreDaemon::Connection2(SocketStreamTLS &) -// Purpose: Handles a connection from bbackupd -// Created: 2006/11/12 -// -// -------------------------------------------------------------------------- -void BackupStoreDaemon::Connection2(SocketStreamTLS &rStream) -{ // Get the common name from the certificate std::string clientCommonName(rStream.GetPeerCommonName()); From boxbackup-dev at fluffy.co.uk Mon Nov 13 20:04:01 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 13 Nov 2006 20:04:01 +0000 Subject: [Box Backup-commit] COMMIT r1169 - box/chris/merge/bin/bbstored Message-ID: Author: chris Date: 2006-11-13 20:04:01 +0000 (Mon, 13 Nov 2006) New Revision: 1169 Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp Log: Don't try to write to the interprocess socket if it's not open (refs #3) Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp =================================================================== --- box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2006-11-13 20:01:39 UTC (rev 1168) +++ box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2006-11-13 20:04:01 UTC (rev 1169) @@ -228,12 +228,18 @@ { // In server process -- use the base class to do the magic ServerTLS::Run(); - + + if (!mInterProcessCommsSocket.IsOpened()) + { + return; + } + // Why did it stop? Tell the housekeeping process to do the same if(IsReloadConfigWanted()) { mInterProcessCommsSocket.Write("h\n", 2); } + if(IsTerminateWanted()) { mInterProcessCommsSocket.Write("t\n", 2); From boxbackup-dev at fluffy.co.uk Tue Nov 14 05:12:03 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Tue, 14 Nov 2006 05:12:03 +0000 Subject: [Box Backup-commit] COMMIT r1170 - in box/chris/merge: bin/bbackupd lib/backupclient Message-ID: Author: chris Date: 2006-11-14 05:12:03 +0000 (Tue, 14 Nov 2006) New Revision: 1170 Modified: box/chris/merge/bin/bbackupd/BackupClientContext.cpp box/chris/merge/bin/bbackupd/BackupClientContext.h box/chris/merge/bin/bbackupd/BackupDaemon.cpp box/chris/merge/lib/backupclient/BackupDaemonConfigVerify.cpp Log: Add ExtendedLogFile option to bbackupd config (refs #9) Modified: box/chris/merge/bin/bbackupd/BackupClientContext.cpp =================================================================== --- box/chris/merge/bin/bbackupd/BackupClientContext.cpp 2006-11-13 20:04:01 UTC (rev 1169) +++ box/chris/merge/bin/bbackupd/BackupClientContext.cpp 2006-11-14 05:12:03 UTC (rev 1170) @@ -35,13 +35,21 @@ // -------------------------------------------------------------------------- // // Function -// Name: BackupClientContext::BackupClientContext(BackupDaemon &, TLSContext &, const std::string &, int32_t, bool) +// Name: BackupClientContext::BackupClientContext(BackupDaemon &, TLSContext &, const std::string &, int32_t, bool, bool, std::string) // Purpose: Constructor // Created: 2003/10/08 // // -------------------------------------------------------------------------- -BackupClientContext::BackupClientContext(BackupDaemon &rDaemon, TLSContext &rTLSContext, const std::string &rHostname, - int32_t AccountNumber, bool ExtendedLogging) +BackupClientContext::BackupClientContext +( + BackupDaemon &rDaemon, + TLSContext &rTLSContext, + const std::string &rHostname, + int32_t AccountNumber, + bool ExtendedLogging, + bool ExtendedLogToFile, + std::string ExtendedLogFile +) : mrDaemon(rDaemon), mrTLSContext(rTLSContext), mHostname(rHostname), @@ -49,6 +57,9 @@ mpSocket(0), mpConnection(0), mExtendedLogging(ExtendedLogging), + mExtendedLogToFile(ExtendedLogToFile), + mExtendedLogFile(ExtendedLogFile), + mpExtendedLogFileHandle(NULL), mClientStoreMarker(ClientStoreMarker_NotKnown), mpDeleteList(0), mpCurrentIDMap(0), @@ -126,6 +137,24 @@ // Set logging option mpConnection->SetLogToSysLog(mExtendedLogging); + if (mExtendedLogToFile) + { + ASSERT(mpExtendedLogFileHandle == NULL); + + mpExtendedLogFileHandle = fopen( + mExtendedLogFile.c_str(), "w"); + + if (!mpExtendedLogFileHandle) + { + ::syslog(LOG_ERR, "Failed to open extended " + "log file: %s", strerror(errno)); + } + else + { + mpConnection->SetLogToFile(mpExtendedLogFileHandle); + } + } + // Handshake mpConnection->Handshake(); @@ -256,6 +285,12 @@ delete mpDeleteList; mpDeleteList = 0; } + + if (mpExtendedLogFileHandle != NULL) + { + fclose(mpExtendedLogFileHandle); + mpExtendedLogFileHandle = NULL; + } } Modified: box/chris/merge/bin/bbackupd/BackupClientContext.h =================================================================== --- box/chris/merge/bin/bbackupd/BackupClientContext.h 2006-11-13 20:04:01 UTC (rev 1169) +++ box/chris/merge/bin/bbackupd/BackupClientContext.h 2006-11-14 05:12:03 UTC (rev 1170) @@ -35,8 +35,16 @@ class BackupClientContext : public DiffTimer { public: - BackupClientContext(BackupDaemon &rDaemon, TLSContext &rTLSContext, const std::string &rHostname, - int32_t AccountNumber, bool ExtendedLogging); + BackupClientContext + ( + BackupDaemon &rDaemon, + TLSContext &rTLSContext, + const std::string &rHostname, + int32_t AccountNumber, + bool ExtendedLogging, + bool ExtendedLogToFile, + std::string ExtendedLogFile + ); virtual ~BackupClientContext(); private: BackupClientContext(const BackupClientContext &); @@ -197,6 +205,9 @@ SocketStreamTLS *mpSocket; BackupProtocolClient *mpConnection; bool mExtendedLogging; + bool mExtendedLogToFile; + std::string mExtendedLogFile; + FILE* mpExtendedLogFileHandle; int64_t mClientStoreMarker; BackupClientDeleteList *mpDeleteList; const BackupClientInodeToIDMap *mpCurrentIDMap; Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2006-11-13 20:04:01 UTC (rev 1169) +++ box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2006-11-14 05:12:03 UTC (rev 1170) @@ -697,9 +697,25 @@ SetState(State_Connected); ::syslog(LOG_INFO, "Beginning scan of local files"); - // Then create a client context object (don't just connect, as this may be unnecessary) - BackupClientContext clientContext(*this, tlsContext, conf.GetKeyValue("StoreHostname"), - conf.GetKeyValueInt("AccountNumber"), conf.GetKeyValueBool("ExtendedLogging")); + std::string extendedLogFile; + if (conf.KeyExists("ExtendedLogFile")) + { + extendedLogFile = conf.GetKeyValue( + "ExtendedLogFile"); + } + + // Then create a client context object (don't + // just connect, as this may be unnecessary) + BackupClientContext clientContext + ( + *this, + tlsContext, + conf.GetKeyValue("StoreHostname"), + conf.GetKeyValueInt("AccountNumber"), + conf.GetKeyValueBool("ExtendedLogging"), + conf.KeyExists("ExtendedLogFile"), + extendedLogFile + ); // Set up the sync parameters BackupClientDirectoryRecord::SyncParams params(*this, clientContext); Modified: box/chris/merge/lib/backupclient/BackupDaemonConfigVerify.cpp =================================================================== --- box/chris/merge/lib/backupclient/BackupDaemonConfigVerify.cpp 2006-11-13 20:04:01 UTC (rev 1169) +++ box/chris/merge/lib/backupclient/BackupDaemonConfigVerify.cpp 2006-11-14 05:12:03 UTC (rev 1170) @@ -81,7 +81,8 @@ {"FileTrackingSizeThreshold", 0, ConfigTest_Exists | ConfigTest_IsInt, 0}, {"DiffingUploadSizeThreshold", 0, ConfigTest_Exists | ConfigTest_IsInt, 0}, {"StoreHostname", 0, ConfigTest_Exists, 0}, - {"ExtendedLogging", "no", ConfigTest_IsBool, 0}, // make value "yes" to enable in config file + {"ExtendedLogging", "no", ConfigTest_IsBool, 0}, // extended log to syslog + {"ExtendedLogFile", NULL, 0, 0}, // extended log to a file {"CommandSocket", 0, 0, 0}, // not compulsory to have this {"KeepAliveTime", 0, ConfigTest_IsInt, 0}, // optional From boxbackup-dev at fluffy.co.uk Wed Nov 15 22:35:22 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Wed, 15 Nov 2006 22:35:22 -0000 Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <075.1b7d6133054ee2a6460aa460f0db838a@fluffy.co.uk> References: <075.1b7d6133054ee2a6460aa460f0db838a@fluffy.co.uk> Message-ID: <084.9519066f46b74c04a49e79d48bebe7e7@fluffy.co.uk> #3: Merge Win32 branch -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: trunk Resolution: | Keywords: win32 merge branch -----------------------+---------------------------------------------------- Comment (by martin): * [840] ok * [995] ok * [1013] ok * [1014] incorrect. otherlibs is the 5th argument * [1015] ok * [1016] ok, but why would you ever not want O_CREAT on an invisible temp file? Similarly, what's the idea of it defaulting to O_RDONLY? * [1017] ok * [1022] ok -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Wed Nov 15 23:45:32 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Wed, 15 Nov 2006 23:45:32 -0000 Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <075.1b7d6133054ee2a6460aa460f0db838a@fluffy.co.uk> References: <075.1b7d6133054ee2a6460aa460f0db838a@fluffy.co.uk> Message-ID: <084.39f98f70591e018a7fc9b846a284177a@fluffy.co.uk> #3: Merge Win32 branch -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: trunk Resolution: | Keywords: win32 merge branch -----------------------+---------------------------------------------------- Comment (by martin): * [1027] ok * [1029] ok * [1030] ok * [1031] ok (should be part of same changeset as 1027) * [1032] ok * [1035] ok * [1036] ok * [1037] what does this change do? * [1049] ok, but error message at line 976 needs updating (or at least an improved explanation, I don't understand what it is trying to say) -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sat Nov 18 00:13:14 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 18 Nov 2006 00:13:14 -0000 Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <075.1b7d6133054ee2a6460aa460f0db838a@fluffy.co.uk> References: <075.1b7d6133054ee2a6460aa460f0db838a@fluffy.co.uk> Message-ID: <084.44e31082d6a82f31cf991598a39951af@fluffy.co.uk> #3: Merge Win32 branch -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: trunk Resolution: | Keywords: win32 merge branch -----------------------+---------------------------------------------------- Comment (by martin): * [1050] + [1051] ok * [1052] ok * [1053] ok * [1054] ok * [1055] ok * [1058] ok - this case insensitive stuff should probably be more generic. I believe HFS+ on Mac OS X is also case insensitive so this and other related changesets should work there too * [1059] ok * [1062] ok * [1064] ok * [1070] ok * [1071] ok * [1072] I'd like to know the underlying issue here, it's not clear to me why this is the right fix * [1073] ok * [1075] ok * [1076] why are you using TRUE instead of true? * [1077] ok * [1078] I presume it works, so ok to merge. style comments: in the regexp you could have used {{{ (....){5} }}} instead of having five of them. Not sure why you couldn't just {{{..\..\..\..\..}}} instead. Also why is require using \ but open using /? * [1079] macros are nasty, prefer inline fns. Why undefining fstat only in these files? (needs explanation I think) * [1080] ok -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sat Nov 18 10:14:10 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 18 Nov 2006 10:14:10 -0000 Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <075.1b7d6133054ee2a6460aa460f0db838a@fluffy.co.uk> References: <075.1b7d6133054ee2a6460aa460f0db838a@fluffy.co.uk> Message-ID: <084.2d1aac5c63067fd9b58601a3f6a3621c@fluffy.co.uk> #3: Merge Win32 branch -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: trunk Resolution: | Keywords: win32 merge branch -----------------------+---------------------------------------------------- Comment (by martin): * [1076] ok -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Thu Nov 23 19:54:51 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 23 Nov 2006 19:54:51 +0000 Subject: [Box Backup-commit] COMMIT r1171 - in box/chris/merge: . lib/common Message-ID: Author: chris Date: 2006-11-23 19:54:51 +0000 (Thu, 23 Nov 2006) New Revision: 1171 Modified: box/chris/merge/configure.ac box/chris/merge/lib/common/BoxTime.cpp Log: Use gettimeofday() to increase accuracy of GetCurrentBoxTime() on platforms which support it. Fixes busy waits for 1 second in backup client when time for next backup is not on a 1 second boundary (which it never is). (refs #3) Modified: box/chris/merge/configure.ac =================================================================== --- box/chris/merge/configure.ac 2006-11-14 05:12:03 UTC (rev 1170) +++ box/chris/merge/configure.ac 2006-11-23 19:54:51 UTC (rev 1171) @@ -158,7 +158,7 @@ AC_FUNC_ERROR_AT_LINE AC_TYPE_SIGNAL AC_FUNC_STAT -AC_CHECK_FUNCS([getpeereid lchown setproctitle getpid]) +AC_CHECK_FUNCS([getpeereid lchown setproctitle getpid gettimeofday]) # NetBSD implements kqueue too differently for us to get it fixed by 0.10 # TODO: Remove this when NetBSD kqueue implementation is working netbsd_hack=`echo $target_os | sed 's/netbsd.*/netbsd/'` Modified: box/chris/merge/lib/common/BoxTime.cpp =================================================================== --- box/chris/merge/lib/common/BoxTime.cpp 2006-11-14 05:12:03 UTC (rev 1170) +++ box/chris/merge/lib/common/BoxTime.cpp 2006-11-23 19:54:51 UTC (rev 1171) @@ -9,7 +9,17 @@ #include "Box.h" -#include +#ifdef HAVE_SYS_TIME_H + #include +#endif +#ifdef HAVE_TIME_H + #include +#endif +#ifdef HAVE_SYSLOG_H + #include +#endif +#include +#include #include "BoxTime.h" @@ -19,13 +29,27 @@ // // Function // Name: GetCurrentBoxTime() -// Purpose: Returns the current time as a box time. (1 sec precision) +// Purpose: Returns the current time as a box time. +// (1 sec precision, or better if supported by system) // Created: 2003/10/08 // // -------------------------------------------------------------------------- box_time_t GetCurrentBoxTime() { + #ifdef HAVE_GETTIMEOFDAY + struct timeval tv; + if (gettimeofday(&tv, NULL) != 0) + { + ::syslog(LOG_ERR, "gettimeofday() failed (%s), " + "dropping precision", strerror(errno)); + } + else + { + box_time_t timeNow = (tv.tv_sec * MICRO_SEC_IN_SEC_LL) + + tv.tv_usec; + return timeNow; + } + #endif + return SecondsToBoxTime(time(0)); } - - From boxbackup-dev at fluffy.co.uk Sat Nov 25 15:24:34 2006 From: boxbackup-dev at fluffy.co.uk (Box Backup) Date: Sat, 25 Nov 2006 15:24:34 -0000 Subject: [Box Backup-commit] Re: [Box Backup] #5: Debian start/stop scripts In-Reply-To: <075.80b63f28eedc010bb744c85de146251c@fluffy.co.uk> References: <075.80b63f28eedc010bb744c85de146251c@fluffy.co.uk> Message-ID: <084.a625294c552883b592c05c534dd5cb01@fluffy.co.uk> #5: Debian start/stop scripts --------------------------+------------------------------------------------- Reporter: martin | Owner: martin Type: enhancement | Status: new Priority: normal | Milestone: 0.11 Component: scripts | Version: 0.10 Resolution: | Keywords: --------------------------+------------------------------------------------- Comment (by hickinbottoms): How do these relate to the Debian packages available on the wiki (wiki:Installation#Debian)? Is that the correct route for their integration as they're part of the packaging for a particular release? Failing that, I think they could be added as a 'Debian recipe' on the wiki:HOWTO page with a reminder to that added to the wiki:Installation#Debian page. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sun Nov 26 16:18:17 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 16:18:17 -0000 Subject: [Box Backup-commit] #11: Document how to use unicode characters and native character sets Message-ID: <075.3a6bf91a9140a22c9f91912e91ce16b2@fluffy.co.uk> #11: Document how to use unicode characters and native character sets --------------------------------------------------------------------------+- Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupctl | Version: 0.10 Keywords: console encoding characters character set documentation docs | --------------------------------------------------------------------------+- The information in [wiki:KnownBugsWin32#ConsoleEncoding] should be turned into docs. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sun Nov 26 16:23:03 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 16:23:03 +0000 Subject: [Box Backup-commit] COMMIT r1172 - box/trunk/lib/server Message-ID: Author: chris Date: 2006-11-26 16:23:03 +0000 (Sun, 26 Nov 2006) New Revision: 1172 Modified: box/trunk/lib/server/makeprotocol.pl.in Log: Fix compile in trunk, trivial. Modified: box/trunk/lib/server/makeprotocol.pl.in =================================================================== --- box/trunk/lib/server/makeprotocol.pl.in 2006-11-23 19:54:51 UTC (rev 1171) +++ box/trunk/lib/server/makeprotocol.pl.in 2006-11-26 16:23:03 UTC (rev 1172) @@ -1,6 +1,10 @@ #!@PERL@ + use strict; +use lib "../../infrastructure"; +use BoxPlatform; + # Make protocol C++ classes from a protocol description file # built in type info (values are is basic type, C++ typename) From boxbackup-dev at fluffy.co.uk Sun Nov 26 16:25:00 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 16:25:00 -0000 Subject: [Box Backup-commit] Re: #10: debian etch trouble In-Reply-To: <075.e308a9b28b769cec56220259b7e16729@fluffy.co.uk> References: <075.e308a9b28b769cec56220259b7e16729@fluffy.co.uk> Message-ID: <084.5663ddc5da4490ce81e3bd9bd5fcc6fe@fluffy.co.uk> #10: debian etch trouble -------------------------+-------------------------------------------------- Reporter: mvanbaak | Owner: Type: defect | Status: new Priority: major | Milestone: Component: bbackupctl | Version: 0.10 Resolution: | Keywords: -------------------------+-------------------------------------------------- Comment (by chris): Hi mvanbaak, sorry about the problem with compiling trunk, it was my fault. I've just committed a fix in [1172], please could you update your trunk and try again? -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sun Nov 26 16:36:18 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 16:36:18 -0000 Subject: [Box Backup-commit] #12: Fix default config file paths on Windows Message-ID: <075.5c4996e1706fecc18296828ecfb1d1b0@fluffy.co.uk> #12: Fix default config file paths on Windows -----------------------------------------------------------+---------------- Reporter: chris | Owner: chris Type: defect | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: 0.10 Keywords: configuration file location directory windows | -----------------------------------------------------------+---------------- The [wiki:KnownBugsWin32#DefaultConfiguration Known Bugs] page says: "We should either always use the default hard-coded path (C:\Program Files\Box Backup\bbackupd.conf), or always assume that the configuration file is in the same directory as the executable, or always try both in the same order." -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sun Nov 26 16:37:46 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 16:37:46 -0000 Subject: [Box Backup-commit] Re: #11: Document how to use unicode characters and native character sets In-Reply-To: <075.3a6bf91a9140a22c9f91912e91ce16b2@fluffy.co.uk> References: <075.3a6bf91a9140a22c9f91912e91ce16b2@fluffy.co.uk> Message-ID: <084.f5839e491dedd009ebdd8255d308a4d4@fluffy.co.uk> #11: Document how to use unicode characters and native character sets ---------------------------+------------------------------------------------ Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupquery | Version: 0.10 Resolution: | Keywords: console encoding characters character set documentation docs ---------------------------+------------------------------------------------ Changes (by chris): * component: bbackupctl => bbackupquery -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sun Nov 26 16:39:08 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 16:39:08 -0000 Subject: [Box Backup-commit] Re: #11: Document how to use unicode characters and native character sets In-Reply-To: <075.3a6bf91a9140a22c9f91912e91ce16b2@fluffy.co.uk> References: <075.3a6bf91a9140a22c9f91912e91ce16b2@fluffy.co.uk> Message-ID: <084.35a24477c8ff21e9d4c6f29362508156@fluffy.co.uk> #11: Document how to use unicode characters and native character sets ---------------------------+------------------------------------------------ Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupquery | Version: 0.10 Resolution: | Keywords: console encoding characters character set documentation docs windows ---------------------------+------------------------------------------------ Changes (by chris): * keywords: console encoding characters character set documentation docs => console encoding characters character set documentation docs windows -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sun Nov 26 16:53:46 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 16:53:46 -0000 Subject: [Box Backup-commit] #13: Fix file locking on Windows Message-ID: <075.6a135d547e77b1d892fe19f2350ca547@fluffy.co.uk> #13: Fix file locking on Windows ------------------------------------------------------------------------+--- Reporter: chris | Owner: chris Type: defect | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: 0.10 Keywords: windows file locking locked volume shadow copy service vss | ------------------------------------------------------------------------+--- There are two locking issues on Windows: Box cannot back up open files, and while Box is backing up a file, other applications cannot open it for writing. Changing our access method to [http://technet2.microsoft.com/windowsserver/en/library//3CF204E6-709A- 4EB8-8CBC-AD9655DE91BA1033.mspx Volume Shadow Copy Service] is probably the right fix for both of these, but it only works on XP and above, and I cannot install the SDK it on my development machine. I would also like to make it work with MinGW without requiring a restrictively-licensed SDK download from Microsoft. Please could someone use a Windows machine (XP or above, Windows Activation compliant) to download the SDK and [mailto:chris+sdk at qwirx.com send it to me]? Another option (workaround) might be to use the Windows command-line tools to create a shadow copy and back that up instead of the original. However, I think that would be a kluge, difficult to install, and difficult to detect and handle errors with. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sun Nov 26 16:55:08 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 16:55:08 -0000 Subject: [Box Backup-commit] Re: #13: Fix file locking on Windows In-Reply-To: <075.6a135d547e77b1d892fe19f2350ca547@fluffy.co.uk> References: <075.6a135d547e77b1d892fe19f2350ca547@fluffy.co.uk> Message-ID: <084.ec181271d8da58aac81efd5d2c4e90f5@fluffy.co.uk> #13: Fix file locking on Windows -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: defect | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: 0.10 Resolution: | Keywords: windows file locking locked volume shadow copy service vss -----------------------+---------------------------------------------------- Comment (by chris): The SDK can be found [http://www.microsoft.com/downloads/details.aspx?FamilyID=0B4F56E4-0CCC-4626 -826A-ED2C4C95C871&displaylang=en here]. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sun Nov 26 16:56:50 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 16:56:50 -0000 Subject: [Box Backup-commit] #14: Fix large file issues on Windows Message-ID: <075.ee91e2e030f110455c060367fb7c9076@fluffy.co.uk> #14: Fix large file issues on Windows ------------------------------------------------+--------------------------- Reporter: chris | Owner: chris Type: defect | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: 0.10 Keywords: windows large files 2gb corruption | ------------------------------------------------+--------------------------- Paul Nash reports that files over 2 GB are not backed up correctly: * those files are silently truncated in the backup * total byte counter at the end of the backup run will be wrong (negative) -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sun Nov 26 17:11:01 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 17:11:01 -0000 Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <075.1b7d6133054ee2a6460aa460f0db838a@fluffy.co.uk> References: <075.1b7d6133054ee2a6460aa460f0db838a@fluffy.co.uk> Message-ID: <084.be7ce40be89eb0508e456aa5ba1336f1@fluffy.co.uk> #3: Merge Win32 branch -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: trunk Resolution: | Keywords: windows win32 merge branch -----------------------+---------------------------------------------------- Changes (by chris): * keywords: win32 merge branch => windows win32 merge branch -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sun Nov 26 19:38:07 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 19:38:07 +0000 Subject: [Box Backup-commit] COMMIT r1173 - box/chris/merge/lib/server Message-ID: Author: chris Date: 2006-11-26 19:38:07 +0000 (Sun, 26 Nov 2006) New Revision: 1173 Modified: box/chris/merge/lib/server/Daemon.cpp Log: * Allow Daemons to be created more than once per process * Don't initialise signal handler until after fork, in case the parent is actually a unit test or another complex application * Don't exit(0) in the parent, for the same reason (refs #9) Modified: box/chris/merge/lib/server/Daemon.cpp =================================================================== --- box/chris/merge/lib/server/Daemon.cpp 2006-11-26 16:23:03 UTC (rev 1172) +++ box/chris/merge/lib/server/Daemon.cpp 2006-11-26 19:38:07 UTC (rev 1173) @@ -48,11 +48,11 @@ // // -------------------------------------------------------------------------- Daemon::Daemon() - : mpConfiguration(0), + : mpConfiguration(NULL), mReloadConfigWanted(false), mTerminateWanted(false) { - if(spDaemon != 0) + if(spDaemon != NULL) { THROW_EXCEPTION(ServerException, AlreadyDaemonConstructed) } @@ -79,6 +79,9 @@ delete mpConfiguration; mpConfiguration = 0; } + + ASSERT(spDaemon == this); + spDaemon = NULL; } // -------------------------------------------------------------------------- @@ -183,18 +186,6 @@ // Let the derived class have a go at setting up stuff in the initial process SetupInInitialProcess(); -#ifndef WIN32 - // Set signal handler - struct sigaction sa; - sa.sa_handler = SignalHandler; - sa.sa_flags = 0; - sigemptyset(&sa.sa_mask); // macro - if(::sigaction(SIGHUP, &sa, NULL) != 0 || ::sigaction(SIGTERM, &sa, NULL) != 0) - { - THROW_EXCEPTION(ServerException, DaemoniseFailed) - } -#endif // !WIN32 - // Server configuration const Configuration &serverConfig( mpConfiguration->GetSubConfiguration("Server")); @@ -232,7 +223,7 @@ default: // parent - _exit(0); + // _exit(0); return 0; break; @@ -269,8 +260,21 @@ break; } } -#endif // ! WIN32 + // Set signal handler + // Don't do this in the parent, since it might be anything + // (e.g. test/bbackupd) + + struct sigaction sa; + sa.sa_handler = SignalHandler; + sa.sa_flags = 0; + sigemptyset(&sa.sa_mask); // macro + if(::sigaction(SIGHUP, &sa, NULL) != 0 || ::sigaction(SIGTERM, &sa, NULL) != 0) + { + THROW_EXCEPTION(ServerException, DaemoniseFailed) + } +#endif // !WIN32 + // open the log ::openlog(DaemonName(), LOG_PID, LOG_LOCAL6); From boxbackup-dev at fluffy.co.uk Sun Nov 26 19:38:46 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 19:38:46 +0000 Subject: [Box Backup-commit] COMMIT r1174 - box/chris/merge/lib/server Message-ID: Author: chris Date: 2006-11-26 19:38:46 +0000 (Sun, 26 Nov 2006) New Revision: 1174 Modified: box/chris/merge/lib/server/makeprotocol.pl.in Log: Add missing newlines to protocol logging to a file (refs #9) Modified: box/chris/merge/lib/server/makeprotocol.pl.in =================================================================== --- box/chris/merge/lib/server/makeprotocol.pl.in 2006-11-26 19:38:07 UTC (rev 1173) +++ box/chris/merge/lib/server/makeprotocol.pl.in 2006-11-26 19:38:46 UTC (rev 1174) @@ -829,8 +829,8 @@ } if($implement_filelog) { - $fR .= qq~\tif(mLogToFile) { ::fprintf(mLogToFile, (Size==Protocol::ProtocolStream_SizeUncertain)?"Receiving stream, size uncertain":"Receiving stream, size %d\\n", Size); ::fflush(mLogToFile); }\n~; - $fS .= qq~\tif(mLogToFile) { ::fprintf(mLogToFile, (Size==Protocol::ProtocolStream_SizeUncertain)?"Sending stream, size uncertain":"Sending stream, size %d\\n", Size); ::fflush(mLogToFile); }\n~; + $fR .= qq~\tif(mLogToFile) { ::fprintf(mLogToFile, (Size==Protocol::ProtocolStream_SizeUncertain)?"Receiving stream, size uncertain\\n":"Receiving stream, size %d\\n", Size); ::fflush(mLogToFile); }\n~; + $fS .= qq~\tif(mLogToFile) { ::fprintf(mLogToFile, (Size==Protocol::ProtocolStream_SizeUncertain)?"Sending stream, size uncertain\\n":"Sending stream, size %d\\n", Size); ::fflush(mLogToFile); }\n~; } print CPP <<__E; From boxbackup-dev at fluffy.co.uk Sun Nov 26 19:41:12 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 19:41:12 +0000 Subject: [Box Backup-commit] COMMIT r1175 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2006-11-26 19:41:12 +0000 (Sun, 26 Nov 2006) New Revision: 1175 Modified: box/chris/merge/lib/common/Test.h Log: Separate ReadPidFile() out from LaunchServer() in test code (refs #9) Modified: box/chris/merge/lib/common/Test.h =================================================================== --- box/chris/merge/lib/common/Test.h 2006-11-26 19:38:46 UTC (rev 1174) +++ box/chris/merge/lib/common/Test.h 2006-11-26 19:41:12 UTC (rev 1175) @@ -104,30 +104,19 @@ return ::system(command.c_str()); } -inline int LaunchServer(const char *pCommandLine, const char *pidFile) +inline int ReadPidFile(const char *pidFile) { - if(RunCommand(pCommandLine) != 0) - { - printf("Server: %s\n", pCommandLine); - TEST_FAIL_WITH_MESSAGE("Couldn't start server"); - return -1; - } - // time for it to start up - ::sleep(1); - - // read pid file if(!TestFileExists(pidFile)) { - printf("Server: %s\n", pCommandLine); TEST_FAIL_WITH_MESSAGE("Server didn't save PID file"); return -1; } + int pid = -1; + FILE *f = fopen(pidFile, "r"); - int pid = -1; if(f == NULL || fscanf(f, "%d", &pid) != 1) { - printf("Server: %s (pidfile %s)\n", pCommandLine, pidFile); TEST_FAIL_WITH_MESSAGE("Couldn't read PID file"); return -1; } @@ -136,6 +125,30 @@ return pid; } +inline int LaunchServer(const char *pCommandLine, const char *pidFile) +{ + if(RunCommand(pCommandLine) != 0) + { + printf("Server: %s\n", pCommandLine); + TEST_FAIL_WITH_MESSAGE("Couldn't start server"); + return -1; + } + + // give it time to start up + ::sleep(1); + + // read pid file + int pid = ReadPidFile(pidFile); + + if(pid == -1) + { + // helps with debugging: + printf("Server: %s (pidfile %s)\n", pCommandLine, pidFile); + } + + return pid; +} + #ifdef WIN32 #include "WinNamedPipeStream.h" @@ -335,4 +348,3 @@ } #endif // TEST__H - From boxbackup-dev at fluffy.co.uk Sun Nov 26 19:43:20 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 19:43:20 +0000 Subject: [Box Backup-commit] COMMIT r1176 - in box/chris/merge: . lib lib/intercept test/raidfile Message-ID: Author: chris Date: 2006-11-26 19:43:20 +0000 (Sun, 26 Nov 2006) New Revision: 1176 Added: box/chris/merge/lib/intercept/ box/chris/merge/lib/intercept/intercept.cpp Removed: box/chris/merge/test/raidfile/intercept.cpp Modified: box/chris/merge/modules.txt Log: Moved intercept code to a library module to allow it to be used by test/bbackupd as well (refs #3) Copied: box/chris/merge/lib/intercept/intercept.cpp (from rev 1139, box/chris/merge/test/raidfile/intercept.cpp) =================================================================== --- box/chris/merge/lib/intercept/intercept.cpp (rev 0) +++ box/chris/merge/lib/intercept/intercept.cpp 2006-11-26 19:43:20 UTC (rev 1176) @@ -0,0 +1,315 @@ +// -------------------------------------------------------------------------- +// +// File +// Name: intercept.cpp +// Purpose: Syscall interception code for the raidfile test +// Created: 2003/07/22 +// +// -------------------------------------------------------------------------- + +#include "Box.h" + +#ifdef HAVE_SYS_SYSCALL_H + #include +#endif +#include +#include + +#ifdef HAVE_SYS_UIO_H +#include +#endif + +#include + +#ifndef PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE + +#if !defined(HAVE_SYSCALL) && !defined(HAVE___SYSCALL) && !defined(HAVE___SYSCALL_NEED_DEFN) + #define PLATFORM_NO_SYSCALL +#endif + +#ifdef PLATFORM_NO_SYSCALL + // For some reason, syscall just doesn't work on Darwin + // so instead, we build functions using assembler in a varient + // of the technique used in the Darwin Libc + extern "C" int + TEST_open(const char *path, int flags, mode_t mode); + extern "C" int + TEST_close(int d); + extern "C" ssize_t + TEST_write(int d, const void *buf, size_t nbytes); + extern "C" ssize_t + TEST_read(int d, void *buf, size_t nbytes); + extern "C" ssize_t + TEST_readv(int d, const struct iovec *iov, int iovcnt); + extern "C" off_t + TEST_lseek(int fildes, off_t offset, int whence); +#else + #ifdef HAVE___SYSCALL_NEED_DEFN + // Need this, not declared in syscall.h nor unistd.h + extern "C" off_t __syscall(quad_t number, ...); + #endif + #ifndef HAVE_SYSCALL + #undef syscall + #define syscall __syscall + #endif +#endif + +#include +#include + +#include "MemLeakFindOn.h" + +bool intercept_enabled = false; +const char *intercept_filename = 0; +int intercept_filedes = -1; +off_t intercept_errorafter = 0; +int intercept_errno = 0; +int intercept_syscall = 0; +off_t intercept_filepos = 0; +int intercept_delay_ms = 0; + +#define SIZE_ALWAYS_ERROR -773 + +void intercept_clear_setup() +{ + intercept_enabled = false; + intercept_filename = 0; + intercept_filedes = -1; + intercept_errorafter = 0; + intercept_syscall = 0; + intercept_filepos = 0; + intercept_delay_ms = 0; +} + +bool intercept_triggered() +{ + return !intercept_enabled; +} + +void intercept_setup_error(const char *filename, unsigned int errorafter, int errortoreturn, int syscalltoerror) +{ + TRACE4("Setup for error: %s, after %d, err %d, syscall %d\n", filename, errorafter, errortoreturn, syscalltoerror); + intercept_enabled = true; + intercept_filename = filename; + intercept_filedes = -1; + intercept_errorafter = errorafter; + intercept_syscall = syscalltoerror; + intercept_errno = errortoreturn; + intercept_filepos = 0; + intercept_delay_ms = 0; +} + +void intercept_setup_delay(const char *filename, unsigned int delay_after, + int delay_ms, int syscall_to_delay) +{ + TRACE4("Setup for delay: %s, after %d, wait %d ms, syscall %d\n", + filename, delay_after, delay_ms, syscall_to_delay); + intercept_enabled = true; + intercept_filename = filename; + intercept_filedes = -1; + intercept_errorafter = delay_after; + intercept_syscall = syscall_to_delay; + intercept_errno = 0; + intercept_filepos = 0; + intercept_delay_ms = delay_ms; +} +bool intercept_errornow(int d, int size, int syscallnum) +{ + if(intercept_filedes != -1 && d == intercept_filedes && syscallnum == intercept_syscall) + { + //printf("Checking for err, %d, %d, %d\n", d, size, syscallnum); + if(size == SIZE_ALWAYS_ERROR) + { + // Looks good for an error! + TRACE2("Returning error %d for syscall %d\n", intercept_errno, syscallnum); + return true; + } + // where are we in the file? + if(intercept_filepos >= intercept_errorafter || intercept_filepos >= ((off_t)intercept_errorafter - size)) + { + if (intercept_errno != 0) + { + TRACE3("Returning error %d for syscall %d, " + "file pos %d\n", intercept_errno, + syscallnum, (int)intercept_filepos); + } + else if (intercept_delay_ms != 0) + { + TRACE3("Delaying %d ms for syscall %d, " + "file pos %d\n", intercept_delay_ms, + syscallnum, (int)intercept_filepos); + } + + return true; + } + } + return false; // no error please! +} + +int intercept_reterr() +{ + int err = intercept_errno; + intercept_clear_setup(); + return err; +} + +#define CHECK_FOR_FAKE_ERROR_COND(D, S, CALL, FAILRES) \ + if(intercept_enabled) \ + { \ + if(intercept_errornow(D, S, CALL)) \ + { \ + if(intercept_delay_ms > 0) \ + { \ + struct timespec tm; \ + tm.tv_sec = intercept_delay_ms / 1000; \ + tm.tv_nsec = (intercept_delay_ms % 1000) \ + * 1000000; \ + while (nanosleep(&tm, &tm) != 0 && \ + errno == EINTR) { } \ + intercept_clear_setup(); \ + } \ + else \ + { \ + errno = intercept_reterr(); \ + return FAILRES; \ + } \ + } \ + } + +extern "C" int +open(const char *path, int flags, mode_t mode) +{ + if(intercept_enabled) + { + if(intercept_syscall == SYS_open && strcmp(path, intercept_filename) == 0) + { + errno = intercept_reterr(); + return -1; + } + } +#ifdef PLATFORM_NO_SYSCALL + int r = TEST_open(path, flags, mode); +#else + int r = syscall(SYS_open, path, flags, mode); +#endif + if(intercept_enabled && intercept_filedes == -1) + { + // Right file? + if(strcmp(intercept_filename, path) == 0) + { + intercept_filedes = r; + //printf("Found file to intercept, h = %d\n", r); + } + } + return r; +} + +extern "C" int +open64(const char *path, int flags, mode_t mode) +{ + // With _FILE_OFFSET_BITS set to 64 this should really use (flags | + // O_LARGEFILE) here, but not actually necessary for the tests and not + // worth the trouble finding O_LARGEFILE + return open(path, flags, mode); +} + +extern "C" int +close(int d) +{ + CHECK_FOR_FAKE_ERROR_COND(d, SIZE_ALWAYS_ERROR, SYS_close, -1); +#ifdef PLATFORM_NO_SYSCALL + int r = TEST_close(d); +#else + int r = syscall(SYS_close, d); +#endif + if(r == 0) + { + if(d == intercept_filedes) + { + intercept_filedes = -1; + } + } + return r; +} + +extern "C" ssize_t +write(int d, const void *buf, size_t nbytes) +{ + CHECK_FOR_FAKE_ERROR_COND(d, nbytes, SYS_write, -1); +#ifdef PLATFORM_NO_SYSCALL + int r = TEST_write(d, buf, nbytes); +#else + int r = syscall(SYS_write, d, buf, nbytes); +#endif + if(r != -1) + { + intercept_filepos += r; + } + return r; +} + +extern "C" ssize_t +read(int d, void *buf, size_t nbytes) +{ + CHECK_FOR_FAKE_ERROR_COND(d, nbytes, SYS_read, -1); +#ifdef PLATFORM_NO_SYSCALL + int r = TEST_read(d, buf, nbytes); +#else + int r = syscall(SYS_read, d, buf, nbytes); +#endif + if(r != -1) + { + intercept_filepos += r; + } + return r; +} + +extern "C" ssize_t +readv(int d, const struct iovec *iov, int iovcnt) +{ + // how many bytes? + int nbytes = 0; + for(int b = 0; b < iovcnt; ++b) + { + nbytes += iov[b].iov_len; + } + + CHECK_FOR_FAKE_ERROR_COND(d, nbytes, SYS_readv, -1); +#ifdef PLATFORM_NO_SYSCALL + int r = TEST_readv(d, iov, iovcnt); +#else + int r = syscall(SYS_readv, d, iov, iovcnt); +#endif + if(r != -1) + { + intercept_filepos += r; + } + return r; +} + +extern "C" off_t +lseek(int fildes, off_t offset, int whence) +{ + // random magic for lseek syscall, see /usr/src/lib/libc/sys/lseek.c + CHECK_FOR_FAKE_ERROR_COND(fildes, 0, SYS_lseek, -1); +#ifdef PLATFORM_NO_SYSCALL + int r = TEST_lseek(fildes, offset, whence); +#else + #ifdef HAVE_LSEEK_DUMMY_PARAM + off_t r = syscall(SYS_lseek, fildes, 0 /* extra 0 required here! */, offset, whence); + #elif defined(_FILE_OFFSET_BITS) + // Don't bother trying to call SYS__llseek on 32 bit since it is + // fiddly and not needed for the tests + off_t r = syscall(SYS_lseek, fildes, (uint32_t)offset, whence); + #else + off_t r = syscall(SYS_lseek, fildes, offset, whence); + #endif +#endif + if(r != -1) + { + intercept_filepos = r; + } + return r; +} + +#endif // n PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE Modified: box/chris/merge/modules.txt =================================================================== --- box/chris/merge/modules.txt 2006-11-26 19:41:12 UTC (rev 1175) +++ box/chris/merge/modules.txt 2006-11-26 19:43:20 UTC (rev 1176) @@ -18,6 +18,7 @@ lib/server lib/win32 lib/server lib/compress +lib/intercept test/common lib/win32 test/crypto lib/crypto lib/win32 test/compress lib/compress lib/win32 @@ -25,7 +26,7 @@ OMIT:mingw32 test/basicserver lib/server lib/win32 OMIT:CYGWIN -test/raidfile lib/raidfile +test/raidfile lib/raidfile lib/intercept END-OMIT # IF_DISTRIBUTION(boxbackup) @@ -52,7 +53,7 @@ test/backupstorefix bin/bbstored bin/bbstoreaccounts lib/backupstore lib/raidfile bin/bbackupquery bin/bbackupd test/backupstorepatch bin/bbstored bin/bbstoreaccounts lib/backupstore lib/raidfile test/backupdiff lib/backupclient -test/bbackupd bin/bbackupd bin/bbstored bin/bbstoreaccounts bin/bbackupquery bin/bbackupctl lib/server lib/backupstore lib/backupclient +test/bbackupd bin/bbackupd bin/bbstored bin/bbstoreaccounts bin/bbackupquery bin/bbackupctl lib/server lib/backupstore lib/backupclient lib/intercept END-OMIT # END_IF_DISTRIBUTION Deleted: box/chris/merge/test/raidfile/intercept.cpp =================================================================== --- box/chris/merge/test/raidfile/intercept.cpp 2006-11-26 19:41:12 UTC (rev 1175) +++ box/chris/merge/test/raidfile/intercept.cpp 2006-11-26 19:43:20 UTC (rev 1176) @@ -1,276 +0,0 @@ -// -------------------------------------------------------------------------- -// -// File -// Name: intercept.cpp -// Purpose: Syscall interception code for the raidfile test -// Created: 2003/07/22 -// -// -------------------------------------------------------------------------- - -#include "Box.h" - -#ifdef HAVE_SYS_SYSCALL_H - #include -#endif -#include -#include - -#ifdef HAVE_SYS_UIO_H -#include -#endif - -#include - -#ifndef PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE - -#if !defined(HAVE_SYSCALL) && !defined(HAVE___SYSCALL) && !defined(HAVE___SYSCALL_NEED_DEFN) - #define PLATFORM_NO_SYSCALL -#endif - -#ifdef PLATFORM_NO_SYSCALL - // For some reason, syscall just doesn't work on Darwin - // so instead, we build functions using assembler in a varient - // of the technique used in the Darwin Libc - extern "C" int - TEST_open(const char *path, int flags, mode_t mode); - extern "C" int - TEST_close(int d); - extern "C" ssize_t - TEST_write(int d, const void *buf, size_t nbytes); - extern "C" ssize_t - TEST_read(int d, void *buf, size_t nbytes); - extern "C" ssize_t - TEST_readv(int d, const struct iovec *iov, int iovcnt); - extern "C" off_t - TEST_lseek(int fildes, off_t offset, int whence); -#else - #ifdef HAVE___SYSCALL_NEED_DEFN - // Need this, not declared in syscall.h nor unistd.h - extern "C" off_t __syscall(quad_t number, ...); - #endif - #ifndef HAVE_SYSCALL - #undef syscall - #define syscall __syscall - #endif -#endif - -#include -#include - -#include "MemLeakFindOn.h" - -bool intercept_enabled = false; -const char *intercept_filename = 0; -int intercept_filedes = -1; -off_t intercept_errorafter = 0; -int intercept_errno = 0; -int intercept_syscall = 0; -off_t intercept_filepos = 0; - -#define SIZE_ALWAYS_ERROR -773 - -void intercept_clear_setup() -{ - intercept_enabled = false; - intercept_filename = 0; - intercept_filedes = -1; - intercept_errorafter = 0; - intercept_syscall = 0; - intercept_filepos = 0; -} - -bool intercept_triggered() -{ - return !intercept_enabled; -} - -void intercept_setup_error(const char *filename, unsigned int errorafter, int errortoreturn, int syscalltoerror) -{ - TRACE4("Setup for error: %s, after %d, err %d, syscall %d\n", filename, errorafter, errortoreturn, syscalltoerror); - intercept_enabled = true; - intercept_filename = filename; - intercept_filedes = -1; - intercept_errorafter = errorafter; - intercept_syscall = syscalltoerror; - intercept_errno = errortoreturn; - intercept_filepos = 0; -} - -bool intercept_errornow(int d, int size, int syscallnum) -{ - if(intercept_filedes != -1 && d == intercept_filedes && syscallnum == intercept_syscall) - { - //printf("Checking for err, %d, %d, %d\n", d, size, syscallnum); - if(size == SIZE_ALWAYS_ERROR) - { - // Looks good for an error! - TRACE2("Returning error %d for syscall %d\n", intercept_errno, syscallnum); - return true; - } - // where are we in the file? - if(intercept_filepos >= intercept_errorafter || intercept_filepos >= ((off_t)intercept_errorafter - size)) - { - TRACE3("Returning error %d for syscall %d, file pos %d\n", intercept_errno, syscallnum, (int)intercept_filepos); - return true; - } - } - return false; // no error please! -} - -int intercept_reterr() -{ - intercept_enabled = false; - intercept_filename = 0; - intercept_filedes = -1; - intercept_errorafter = 0; - intercept_syscall = 0; - return intercept_errno; -} - -#define CHECK_FOR_FAKE_ERROR_COND(D, S, CALL, FAILRES) \ - if(intercept_enabled) \ - { \ - if(intercept_errornow(D, S, CALL)) \ - { \ - errno = intercept_reterr(); \ - return FAILRES; \ - } \ - } - -extern "C" int -open(const char *path, int flags, mode_t mode) -{ - if(intercept_enabled) - { - if(intercept_syscall == SYS_open && strcmp(path, intercept_filename) == 0) - { - errno = intercept_reterr(); - return -1; - } - } -#ifdef PLATFORM_NO_SYSCALL - int r = TEST_open(path, flags, mode); -#else - int r = syscall(SYS_open, path, flags, mode); -#endif - if(intercept_enabled && intercept_filedes == -1) - { - // Right file? - if(strcmp(intercept_filename, path) == 0) - { - intercept_filedes = r; - //printf("Found file to intercept, h = %d\n", r); - } - } - return r; -} - -extern "C" int -open64(const char *path, int flags, mode_t mode) -{ - // With _FILE_OFFSET_BITS set to 64 this should really use (flags | - // O_LARGEFILE) here, but not actually necessary for the tests and not - // worth the trouble finding O_LARGEFILE - return open(path, flags, mode); -} - -extern "C" int -close(int d) -{ - CHECK_FOR_FAKE_ERROR_COND(d, SIZE_ALWAYS_ERROR, SYS_close, -1); -#ifdef PLATFORM_NO_SYSCALL - int r = TEST_close(d); -#else - int r = syscall(SYS_close, d); -#endif - if(r == 0) - { - if(d == intercept_filedes) - { - intercept_filedes = -1; - } - } - return r; -} - -extern "C" ssize_t -write(int d, const void *buf, size_t nbytes) -{ - CHECK_FOR_FAKE_ERROR_COND(d, nbytes, SYS_write, -1); -#ifdef PLATFORM_NO_SYSCALL - int r = TEST_write(d, buf, nbytes); -#else - int r = syscall(SYS_write, d, buf, nbytes); -#endif - if(r != -1) - { - intercept_filepos += r; - } - return r; -} - -extern "C" ssize_t -read(int d, void *buf, size_t nbytes) -{ - CHECK_FOR_FAKE_ERROR_COND(d, nbytes, SYS_read, -1); -#ifdef PLATFORM_NO_SYSCALL - int r = TEST_read(d, buf, nbytes); -#else - int r = syscall(SYS_read, d, buf, nbytes); -#endif - if(r != -1) - { - intercept_filepos += r; - } - return r; -} - -extern "C" ssize_t -readv(int d, const struct iovec *iov, int iovcnt) -{ - // how many bytes? - int nbytes = 0; - for(int b = 0; b < iovcnt; ++b) - { - nbytes += iov[b].iov_len; - } - - CHECK_FOR_FAKE_ERROR_COND(d, nbytes, SYS_readv, -1); -#ifdef PLATFORM_NO_SYSCALL - int r = TEST_readv(d, iov, iovcnt); -#else - int r = syscall(SYS_readv, d, iov, iovcnt); -#endif - if(r != -1) - { - intercept_filepos += r; - } - return r; -} - -extern "C" off_t -lseek(int fildes, off_t offset, int whence) -{ - // random magic for lseek syscall, see /usr/src/lib/libc/sys/lseek.c - CHECK_FOR_FAKE_ERROR_COND(fildes, 0, SYS_lseek, -1); -#ifdef PLATFORM_NO_SYSCALL - int r = TEST_lseek(fildes, offset, whence); -#else - #ifdef HAVE_LSEEK_DUMMY_PARAM - off_t r = syscall(SYS_lseek, fildes, 0 /* extra 0 required here! */, offset, whence); - #elif defined(_FILE_OFFSET_BITS) - // Don't bother trying to call SYS__llseek on 32 bit since it is - // fiddly and not needed for the tests - off_t r = syscall(SYS_lseek, fildes, (uint32_t)offset, whence); - #else - off_t r = syscall(SYS_lseek, fildes, offset, whence); - #endif -#endif - if(r != -1) - { - intercept_filepos = r; - } - return r; -} - -#endif // n PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE From boxbackup-dev at fluffy.co.uk Sun Nov 26 19:45:45 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 19:45:45 +0000 Subject: [Box Backup-commit] COMMIT r1177 - box/chris/merge/infrastructure Message-ID: Author: chris Date: 2006-11-26 19:45:44 +0000 (Sun, 26 Nov 2006) New Revision: 1177 Modified: box/chris/merge/infrastructure/makebuildenv.pl.in Log: * Try to kill any daemons left over from previous tests before running new tests. * Try to kill any remaining daemons at the end of the test (refs #3) Modified: box/chris/merge/infrastructure/makebuildenv.pl.in =================================================================== --- box/chris/merge/infrastructure/makebuildenv.pl.in 2006-11-26 19:43:20 UTC (rev 1176) +++ box/chris/merge/infrastructure/makebuildenv.pl.in 2006-11-26 19:45:44 UTC (rev 1177) @@ -401,24 +401,46 @@ sub writetestfile { my ($filename,$runcmd,$module) = @_; - open TESTFILE,">$filename" or die "Can't open test script file for $module for writing\n"; + + open TESTFILE,">$filename" or die "Can't open " . + "test script file for $module for writing\n"; print TESTFILE "#!/bin/sh\necho TEST: $module\n"; + if(-d "$module/testfiles") { print TESTFILE <<__E; +echo Killing any running daemons... +test -r testfiles/bbackupd.pid && kill `cat testfiles/bbackupd.pid` +test -r testfiles/bbstored.pid && kill `cat testfiles/bbstored.pid` + echo Removing old test files... rm -rf testfiles + echo Copying new test files... cp -p -R ../../../$module/testfiles . + __E } + if(-e "$module/testextra") { - open FL,"$module/testextra" or die "Can't open $module/testextra"; + open FL,"$module/testextra" or die + "Can't open $module/testextra"; while() {print TESTFILE} close FL; } + print TESTFILE "$runcmd\n"; + + if(-d "$module/testfiles") + { + print TESTFILE <<__E; +# echo Killing any running daemons... +test -r testfiles/bbackupd.pid && kill `cat testfiles/bbackupd.pid` +test -r testfiles/bbstored.pid && kill `cat testfiles/bbstored.pid` +__E + } + close TESTFILE; } From boxbackup-dev at fluffy.co.uk Sun Nov 26 19:47:49 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 19:47:49 +0000 Subject: [Box Backup-commit] COMMIT r1178 - box/chris/merge/bin/bbstored Message-ID: Author: chris Date: 2006-11-26 19:47:49 +0000 (Sun, 26 Nov 2006) New Revision: 1178 Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp box/chris/merge/bin/bbstored/BackupStoreDaemon.h Log: Catch any exceptions while handling a connection and report to user rather than terminating. Useful for non-forking servers like bbstored on Windows. (refs #3) Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp =================================================================== --- box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2006-11-26 19:45:44 UTC (rev 1177) +++ box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2006-11-26 19:47:49 UTC (rev 1178) @@ -247,17 +247,49 @@ } } - // -------------------------------------------------------------------------- // // Function // Name: BackupStoreDaemon::Connection(SocketStreamTLS &) -// Purpose: Handles a connection +// Purpose: Handles a connection, by catching exceptions and +// delegating to Connection2 // Created: 2003/08/20 // // -------------------------------------------------------------------------- void BackupStoreDaemon::Connection(SocketStreamTLS &rStream) { + try + { + Connection2(rStream); + } + catch(BoxException &e) + { + ::syslog(LOG_ERR, "%s: disconnecting due to " + "exception %s (%d/%d)", DaemonName(), + e.what(), e.GetType(), e.GetSubType()); + } + catch(std::exception &e) + { + ::syslog(LOG_ERR, "%s: disconnecting due to " + "exception %s", DaemonName(), e.what()); + } + catch(...) + { + ::syslog(LOG_ERR, "%s: disconnecting due to " + "unknown exception", DaemonName()); + } +} + +// -------------------------------------------------------------------------- +// +// Function +// Name: BackupStoreDaemon::Connection2(SocketStreamTLS &) +// Purpose: Handles a connection from bbackupd +// Created: 2006/11/12 +// +// -------------------------------------------------------------------------- +void BackupStoreDaemon::Connection2(SocketStreamTLS &rStream) +{ // Get the common name from the certificate std::string clientCommonName(rStream.GetPeerCommonName()); Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.h =================================================================== --- box/chris/merge/bin/bbstored/BackupStoreDaemon.h 2006-11-26 19:45:44 UTC (rev 1177) +++ box/chris/merge/bin/bbstored/BackupStoreDaemon.h 2006-11-26 19:47:49 UTC (rev 1178) @@ -52,7 +52,8 @@ virtual void Run(); - void Connection(SocketStreamTLS &rStream); + virtual void Connection(SocketStreamTLS &rStream); + void Connection2(SocketStreamTLS &rStream); virtual const char *DaemonName() const; virtual const char *DaemonBanner() const; From boxbackup-dev at fluffy.co.uk Sun Nov 26 19:49:27 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 19:49:27 +0000 Subject: [Box Backup-commit] COMMIT r1179 - in box/chris/merge/test/bbackupd: . testfiles Message-ID: Author: chris Date: 2006-11-26 19:49:27 +0000 (Sun, 26 Nov 2006) New Revision: 1179 Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp box/chris/merge/test/bbackupd/testfiles/bbackupd.conf.in Log: Added test for keepalives being sent (refs #9) Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2006-11-26 19:47:49 UTC (rev 1178) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2006-11-26 19:49:27 UTC (rev 1179) @@ -22,6 +22,10 @@ #endif #include +#ifdef HAVE_SYSCALL +#include +#endif + #include "Test.h" #include "BackupClientFileAttributes.h" #include "CommonException.h" @@ -41,6 +45,10 @@ #include "Utils.h" #include "BoxTime.h" #include "BoxTimeToUnix.h" +#include "BackupDaemon.h" +#include "Timer.h" +#include "FileStream.h" +#include "IOStreamGetLine.h" #include "MemLeakFindOn.h" @@ -500,7 +508,75 @@ } } +void intercept_setup_delay(const char *filename, unsigned int delay_after, + int delay_ms, int syscall_to_delay); +bool intercept_triggered(); +int start_internal_daemon() +{ + // ensure that no child processes end up running tests! + int own_pid = getpid(); + + BackupDaemon daemon; + const char* fake_argv[] = { "bbackupd", "testfiles/bbackupd.conf" }; + + int result = daemon.Main(BOX_FILE_BBACKUPD_DEFAULT_CONFIG, 2, + fake_argv); + + TEST_THAT(result == 0); + if (result != 0) + { + printf("Daemon exited with code %d\n", result); + } + + // ensure that no child processes end up running tests! + TEST_THAT(getpid() == own_pid); + if (getpid() != own_pid) + { + // abort! + _exit(1); + } + + TEST_THAT(TestFileExists("testfiles/bbackupd.pid")); + + printf("Waiting for daemon to start"); + int pid = -1; + + for (int i = 0; i < 30; i++) + { + printf("."); + fflush(stdout); + sleep(1); + + pid = ReadPidFile("testfiles/bbackupd.pid"); + if (pid > 0) + { + break; + } + } + + printf("\n"); + + TEST_THAT(pid > 0); + return pid; +} + +void stop_internal_daemon(int pid) +{ + TEST_THAT(KillServer(pid)); + + /* + int status; + TEST_THAT(waitpid(pid, &status, 0) == pid); + TEST_THAT(WIFEXITED(status)); + + if (WIFEXITED(status)) + { + TEST_THAT(WEXITSTATUS(status) == 0); + } + */ +} + int test_bbackupd() { // // First, wait for a normal period to make sure the last changes attributes are within a normal backup timeframe. @@ -517,7 +593,91 @@ TEST_THAT(::system("rm -rf testfiles/TestDir1") == 0); TEST_THAT(::system("mkdir testfiles/TestDir1") == 0); TEST_THAT(::system("gzip -d < testfiles/spacetest1.tgz | ( cd testfiles/TestDir1 && tar xf - )") == 0); + +#ifdef PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE + printf("Skipping intercept-based KeepAlive tests on this platform.\n"); +#else + { + #ifdef WIN32 + #error TODO: implement threads on Win32, or this test \ + will not finish properly + #endif + // bbackupd daemon will try to initialise timers itself + Timers::Cleanup(); + + // something to diff against (empty file doesn't work) + int fd = open("testfiles/TestDir1/spacetest/f1", O_WRONLY); + TEST_THAT(fd > 0); + char buffer[1024]; + TEST_THAT(write(fd, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd) == 0); + + int pid = start_internal_daemon(); + wait_for_backup_operation(); + stop_internal_daemon(pid); + + intercept_setup_delay("testfiles/TestDir1/spacetest/f1", + 0, 2000, SYS_read); + TEST_THAT(unlink("testfiles/bbackupd.log") == 0); + + pid = start_internal_daemon(); + + fd = open("testfiles/TestDir1/spacetest/f1", O_WRONLY); + TEST_THAT(fd > 0); + // write again, to update the file's timestamp + TEST_THAT(write(fd, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd) == 0); + + wait_for_backup_operation(); + // can't test whether intercept was triggered, because + // it's in a different process. + // TEST_THAT(intercept_triggered()); + stop_internal_daemon(pid); + + // check that keepalive was written to logs, and + // diff was not aborted, i.e. upload was a diff + FileStream fs("testfiles/bbackupd.log", O_RDONLY); + IOStreamGetLine reader(fs); + bool found1 = false; + + while (!reader.IsEOF()) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + if (line == "Send GetBlockIndexByName(0x3,\"f1\")") + { + found1 = true; + break; + } + } + + TEST_THAT(found1); + if (found1) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive Success(0xe)"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receiving stream, size 60"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Send GetIsAlive()"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive IsAlive()"); + TEST_THAT(reader.GetLine(line)); + + std::string comp = "Send StoreFile(0x3,"; + TEST_THAT(line.substr(0, comp.size()) == comp); + comp = ",0xe,\"f1\")"; + TEST_THAT(line.substr(line.size() - comp.size()) + == comp); + } + + // restore timers for rest of tests + Timers::Init(); + } +#endif // PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE + int pid = LaunchServer("../../bin/bbackupd/bbackupd testfiles/bbackupd.conf", "testfiles/bbackupd.pid"); TEST_THAT(pid != -1 && pid != 0); if(pid > 0) @@ -1055,4 +1215,3 @@ return 0; } - Modified: box/chris/merge/test/bbackupd/testfiles/bbackupd.conf.in =================================================================== --- box/chris/merge/test/bbackupd/testfiles/bbackupd.conf.in 2006-11-26 19:47:49 UTC (rev 1178) +++ box/chris/merge/test/bbackupd/testfiles/bbackupd.conf.in 2006-11-26 19:49:27 UTC (rev 1179) @@ -17,9 +17,11 @@ FileTrackingSizeThreshold = 1024 DiffingUploadSizeThreshold = 1024 -MaximumDiffingTime = 8 +MaximumDiffingTime = 3 +KeepAliveTime = 1 -ExtendedLogging = yes +ExtendedLogging = no +ExtendedLogFile = testfiles/bbackupd.log CommandSocket = testfiles/bbackupd.sock From boxbackup-dev at fluffy.co.uk Sun Nov 26 19:54:00 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 26 Nov 2006 19:54:00 +0000 Subject: [Box Backup-commit] COMMIT r1180 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2006-11-26 19:54:00 +0000 (Sun, 26 Nov 2006) New Revision: 1180 Modified: box/chris/merge/lib/common/Timer.cpp Log: * Fix timer expiry calculation when timers expire in the past * Fix handling of timers which never expire (zero deadline) (refs #9) Modified: box/chris/merge/lib/common/Timer.cpp =================================================================== --- box/chris/merge/lib/common/Timer.cpp 2006-11-26 19:49:27 UTC (rev 1179) +++ box/chris/merge/lib/common/Timer.cpp 2006-11-26 19:54:00 UTC (rev 1180) @@ -143,7 +143,7 @@ ASSERT(spTimers); box_time_t timeNow = GetCurrentBoxTime(); - box_time_t timeToNextEvent = 0; + int64_t timeToNextEvent = 0; for (std::vector::iterator i = spTimers->begin(); i != spTimers->end(); i++) @@ -151,7 +151,8 @@ Timer& rTimer = **i; ASSERT(!rTimer.HasExpired()); - box_time_t timeToExpiry = rTimer.GetExpiryTime() - timeNow; + int64_t timeToExpiry = rTimer.GetExpiryTime() - timeNow; + if (timeToExpiry <= 0) timeToExpiry = 1; if (timeToNextEvent == 0 || timeToNextEvent > timeToExpiry) { @@ -203,7 +204,7 @@ Timer& rTimer = **i; ASSERT(!rTimer.HasExpired()); - box_time_t timeToExpiry = rTimer.GetExpiryTime() - timeNow; + int64_t timeToExpiry = rTimer.GetExpiryTime() - timeNow; if (timeToExpiry <= 0) { @@ -218,7 +219,14 @@ : mExpires(GetCurrentBoxTime() + SecondsToBoxTime(timeoutSecs)), mExpired(false) { - Timers::Add(*this); + if (timeoutSecs == 0) + { + mExpires = 0; + } + else + { + Timers::Add(*this); + } } Timer::~Timer() @@ -230,7 +238,10 @@ : mExpires(rToCopy.mExpires), mExpired(rToCopy.mExpired) { - Timers::Add(*this); + if (mExpires != 0) + { + Timers::Add(*this); + } } Timer& Timer::operator=(const Timer& rToCopy) @@ -238,10 +249,11 @@ Timers::Remove(*this); mExpires = rToCopy.mExpires; mExpired = rToCopy.mExpired; - if (!mExpired) + if (!mExpired && mExpires != 0) { Timers::Add(*this); } + return *this; } void Timer::OnExpire() From boxbackup-dev at fluffy.co.uk Mon Nov 27 07:40:41 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 27 Nov 2006 07:40:41 -0000 Subject: [Box Backup-commit] Re: #9: SSL connection may time out when backing up large data sets In-Reply-To: <075.382a81ac9c31eb8a9dfae053eb0061b0@fluffy.co.uk> References: <075.382a81ac9c31eb8a9dfae053eb0061b0@fluffy.co.uk> Message-ID: <084.92c167bb465e16ba6ac52e190a1b796a@fluffy.co.uk> #9: SSL connection may time out when backing up large data sets -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: defect | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: 0.10 Resolution: | Keywords: ssl timeout client reload scanning directories -----------------------+---------------------------------------------------- Changes (by chris): * component: bbackupctl => bbackupd Comment: (In [1171]) * Use gettimeofday() to increase accuracy of GetCurrentBoxTime?() on platforms which support it. Fixes busy waits for 1 second in backup client when time for next backup is not on a 1 second boundary (which it never is). (In [1176]) * Moved intercept code to a library module to allow it to be used by test/bbackupd as well. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Mon Nov 27 19:19:57 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 27 Nov 2006 19:19:57 +0000 Subject: [Box Backup-commit] COMMIT r1181 - box/chris/merge/test/bbackupd Message-ID: Author: chris Date: 2006-11-27 19:19:57 +0000 (Mon, 27 Nov 2006) New Revision: 1181 Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp Log: Added a test for diff termination if MaximumDiffingTime is exceeded (refs #3, refs #9) Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2006-11-26 19:54:00 UTC (rev 1180) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2006-11-27 19:19:57 UTC (rev 1181) @@ -673,6 +673,58 @@ == comp); } + intercept_setup_delay("testfiles/TestDir1/spacetest/f1", + 0, 4000, SYS_read); + pid = start_internal_daemon(); + + fd = open("testfiles/TestDir1/spacetest/f1", O_WRONLY); + TEST_THAT(fd > 0); + // write again, to update the file's timestamp + TEST_THAT(write(fd, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd) == 0); + + wait_for_backup_operation(); + // can't test whether intercept was triggered, because + // it's in a different process. + // TEST_THAT(intercept_triggered()); + stop_internal_daemon(pid); + + // check that the diff was aborted, i.e. upload was not a diff + found1 = false; + + while (!reader.IsEOF()) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + if (line == "Send GetBlockIndexByName(0x3,\"f1\")") + { + found1 = true; + break; + } + } + + TEST_THAT(found1); + if (found1) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive Success(0xf)"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receiving stream, size 60"); + + // delaying for 4 seconds in one step means that + // the diff timer and the keepalive timer will + // both expire, and the diff timer is honoured first, + // so there will be no keepalives. + + TEST_THAT(reader.GetLine(line)); + std::string comp = "Send StoreFile(0x3,"; + TEST_THAT(line.substr(0, comp.size()) == comp); + comp = ",0x0,\"f1\")"; + TEST_THAT(line.substr(line.size() - comp.size()) + == comp); + } + // restore timers for rest of tests Timers::Init(); } From boxbackup-dev at fluffy.co.uk Tue Nov 28 20:28:44 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Tue, 28 Nov 2006 20:28:44 +0000 Subject: [Box Backup-commit] COMMIT r1182 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2006-11-28 20:28:43 +0000 (Tue, 28 Nov 2006) New Revision: 1182 Modified: box/chris/merge/lib/common/Timer.cpp Log: Added debug tracing code for timers. Modified: box/chris/merge/lib/common/Timer.cpp =================================================================== --- box/chris/merge/lib/common/Timer.cpp 2006-11-27 19:19:57 UTC (rev 1181) +++ box/chris/merge/lib/common/Timer.cpp 2006-11-28 20:28:43 UTC (rev 1182) @@ -219,8 +219,26 @@ : mExpires(GetCurrentBoxTime() + SecondsToBoxTime(timeoutSecs)), mExpired(false) { + #ifndef NDEBUG + struct timeval tv; + gettimeofday(&tv, NULL); if (timeoutSecs == 0) { + TRACE4("%d.%d: timer %p initialised for %d secs, " + "will not fire\n", tv.tv_sec, tv.tv_usec, this, + timeoutSecs); + } + else + { + TRACE6("%d.%d: timer %p initialised for %d secs, " + "to fire at %d.%d\n", tv.tv_sec, tv.tv_usec, this, + timeoutSecs, (int)(mExpires / 1000000), + (int)(mExpires % 1000000)); + } + #endif + + if (timeoutSecs == 0) + { mExpires = 0; } else @@ -231,6 +249,13 @@ Timer::~Timer() { + #ifndef NDEBUG + struct timeval tv; + gettimeofday(&tv, NULL); + TRACE3("%d.%d: timer %p destroyed, will not fire\n", + tv.tv_sec, tv.tv_usec, this); + #endif + Timers::Remove(*this); } @@ -238,14 +263,62 @@ : mExpires(rToCopy.mExpires), mExpired(rToCopy.mExpired) { - if (mExpires != 0) + #ifndef NDEBUG + struct timeval tv; + gettimeofday(&tv, NULL); + if (mExpired) { + TRACE4("%d.%d: timer %p initialised from timer %p, " + "already expired, will not fire\n", tv.tv_sec, + tv.tv_usec, this, &rToCopy); + } + else if (mExpires == 0) + { + TRACE4("%d.%d: timer %p initialised from timer %p, " + "will not fire\n", tv.tv_sec, tv.tv_usec, this, + &rToCopy); + } + else + { + TRACE6("%d.%d: timer %p initialised from timer %p, " + "to fire at %d.%d\n", tv.tv_sec, tv.tv_usec, this, + &rToCopy, (int)(mExpires / 1000000), + (int)(mExpires % 1000000)); + } + #endif + + if (!mExpired && mExpires != 0) + { Timers::Add(*this); } } Timer& Timer::operator=(const Timer& rToCopy) { + #ifndef NDEBUG + struct timeval tv; + gettimeofday(&tv, NULL); + if (rToCopy.mExpired) + { + TRACE4("%d.%d: timer %p initialised from timer %p, " + "already expired, will not fire\n", tv.tv_sec, + tv.tv_usec, this, &rToCopy); + } + else if (rToCopy.mExpires == 0) + { + TRACE4("%d.%d: timer %p initialised from timer %p, " + "will not fire\n", tv.tv_sec, tv.tv_usec, this, + &rToCopy); + } + else + { + TRACE6("%d.%d: timer %p initialised from timer %p, " + "to fire at %d.%d\n", tv.tv_sec, tv.tv_usec, this, + &rToCopy, (int)(rToCopy.mExpires / 1000000), + (int)(rToCopy.mExpires % 1000000)); + } + #endif + Timers::Remove(*this); mExpires = rToCopy.mExpires; mExpired = rToCopy.mExpired; @@ -258,6 +331,12 @@ void Timer::OnExpire() { + #ifndef NDEBUG + struct timeval tv; + gettimeofday(&tv, NULL); + TRACE3("%d.%d: timer %p fired\n", tv.tv_sec, tv.tv_usec, this); + #endif + mExpired = true; Timers::Remove(*this); } From boxbackup-dev at fluffy.co.uk Tue Nov 28 20:30:36 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Tue, 28 Nov 2006 20:30:36 +0000 Subject: [Box Backup-commit] COMMIT r1183 - in box/chris/merge: lib/intercept test/bbackupd Message-ID: Author: chris Date: 2006-11-28 20:30:36 +0000 (Tue, 28 Nov 2006) New Revision: 1183 Modified: box/chris/merge/lib/intercept/intercept.cpp box/chris/merge/test/bbackupd/testbbackupd.cpp Log: Added ability for delay intercepts to fire multiple times. Added test for repeat keepalives to test/bbackupd. Modified: box/chris/merge/lib/intercept/intercept.cpp =================================================================== --- box/chris/merge/lib/intercept/intercept.cpp 2006-11-28 20:28:43 UTC (rev 1182) +++ box/chris/merge/lib/intercept/intercept.cpp 2006-11-28 20:30:36 UTC (rev 1183) @@ -59,7 +59,7 @@ #include "MemLeakFindOn.h" -bool intercept_enabled = false; +int intercept_count = 0; const char *intercept_filename = 0; int intercept_filedes = -1; off_t intercept_errorafter = 0; @@ -72,7 +72,7 @@ void intercept_clear_setup() { - intercept_enabled = false; + intercept_count = 0; intercept_filename = 0; intercept_filedes = -1; intercept_errorafter = 0; @@ -83,13 +83,13 @@ bool intercept_triggered() { - return !intercept_enabled; + return intercept_count == 0; } void intercept_setup_error(const char *filename, unsigned int errorafter, int errortoreturn, int syscalltoerror) { TRACE4("Setup for error: %s, after %d, err %d, syscall %d\n", filename, errorafter, errortoreturn, syscalltoerror); - intercept_enabled = true; + intercept_count = 1; intercept_filename = filename; intercept_filedes = -1; intercept_errorafter = errorafter; @@ -100,11 +100,12 @@ } void intercept_setup_delay(const char *filename, unsigned int delay_after, - int delay_ms, int syscall_to_delay) + int delay_ms, int syscall_to_delay, int num_delays) { - TRACE4("Setup for delay: %s, after %d, wait %d ms, syscall %d\n", - filename, delay_after, delay_ms, syscall_to_delay); - intercept_enabled = true; + TRACE5("Setup for delay: %s, after %d, wait %d ms, times %d, " + "syscall %d\n", filename, delay_after, delay_ms, + num_delays, syscall_to_delay); + intercept_count = num_delays; intercept_filename = filename; intercept_filedes = -1; intercept_errorafter = delay_after; @@ -154,7 +155,7 @@ } #define CHECK_FOR_FAKE_ERROR_COND(D, S, CALL, FAILRES) \ - if(intercept_enabled) \ + if(intercept_count > 0) \ { \ if(intercept_errornow(D, S, CALL)) \ { \ @@ -166,7 +167,11 @@ * 1000000; \ while (nanosleep(&tm, &tm) != 0 && \ errno == EINTR) { } \ - intercept_clear_setup(); \ + intercept_count --; \ + if (intercept_count == 0) \ + { \ + intercept_clear_setup(); \ + } \ } \ else \ { \ @@ -179,7 +184,7 @@ extern "C" int open(const char *path, int flags, mode_t mode) { - if(intercept_enabled) + if(intercept_count > 0) { if(intercept_syscall == SYS_open && strcmp(path, intercept_filename) == 0) { @@ -192,7 +197,7 @@ #else int r = syscall(SYS_open, path, flags, mode); #endif - if(intercept_enabled && intercept_filedes == -1) + if(intercept_count > 0 && intercept_filedes == -1) { // Right file? if(strcmp(intercept_filename, path) == 0) Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2006-11-28 20:28:43 UTC (rev 1182) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2006-11-28 20:30:36 UTC (rev 1183) @@ -508,8 +508,9 @@ } } -void intercept_setup_delay(const char *filename, unsigned int delay_after, - int delay_ms, int syscall_to_delay); +void intercept_setup_delay(const char *filename, unsigned int delay_after, + int delay_ms, int syscall_to_delay, int num_delays); + bool intercept_triggered(); int start_internal_daemon() @@ -609,7 +610,7 @@ // something to diff against (empty file doesn't work) int fd = open("testfiles/TestDir1/spacetest/f1", O_WRONLY); TEST_THAT(fd > 0); - char buffer[1024]; + char buffer[10000]; TEST_THAT(write(fd, buffer, sizeof(buffer)) == sizeof(buffer)); TEST_THAT(close(fd) == 0); @@ -618,7 +619,7 @@ stop_internal_daemon(pid); intercept_setup_delay("testfiles/TestDir1/spacetest/f1", - 0, 2000, SYS_read); + 0, 2000, SYS_read, 1); TEST_THAT(unlink("testfiles/bbackupd.log") == 0); pid = start_internal_daemon(); @@ -659,13 +660,13 @@ TEST_THAT(reader.GetLine(line)); TEST_THAT(line == "Receive Success(0xe)"); TEST_THAT(reader.GetLine(line)); - TEST_THAT(line == "Receiving stream, size 60"); + TEST_THAT(line == "Receiving stream, size 124"); TEST_THAT(reader.GetLine(line)); TEST_THAT(line == "Send GetIsAlive()"); TEST_THAT(reader.GetLine(line)); TEST_THAT(line == "Receive IsAlive()"); + TEST_THAT(reader.GetLine(line)); - std::string comp = "Send StoreFile(0x3,"; TEST_THAT(line.substr(0, comp.size()) == comp); comp = ",0xe,\"f1\")"; @@ -674,7 +675,7 @@ } intercept_setup_delay("testfiles/TestDir1/spacetest/f1", - 0, 4000, SYS_read); + 0, 4000, SYS_read, 1); pid = start_internal_daemon(); fd = open("testfiles/TestDir1/spacetest/f1", O_WRONLY); @@ -710,7 +711,7 @@ TEST_THAT(reader.GetLine(line)); TEST_THAT(line == "Receive Success(0xf)"); TEST_THAT(reader.GetLine(line)); - TEST_THAT(line == "Receiving stream, size 60"); + TEST_THAT(line == "Receiving stream, size 124"); // delaying for 4 seconds in one step means that // the diff timer and the keepalive timer will @@ -724,7 +725,69 @@ TEST_THAT(line.substr(line.size() - comp.size()) == comp); } - + + intercept_setup_delay("testfiles/TestDir1/spacetest/f1", + 0, 1000, SYS_read, 3); + pid = start_internal_daemon(); + + fd = open("testfiles/TestDir1/spacetest/f1", O_WRONLY); + TEST_THAT(fd > 0); + // write again, to update the file's timestamp + TEST_THAT(write(fd, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd) == 0); + + wait_for_backup_operation(); + // can't test whether intercept was triggered, because + // it's in a different process. + // TEST_THAT(intercept_triggered()); + stop_internal_daemon(pid); + + // check that the diff was aborted, i.e. upload was not a diff + found1 = false; + + while (!reader.IsEOF()) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + if (line == "Send GetBlockIndexByName(0x3,\"f1\")") + { + found1 = true; + break; + } + } + + TEST_THAT(found1); + if (found1) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive Success(0x10)"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receiving stream, size 124"); + + // delaying for 3 seconds in steps of 1 second + // means that the keepalive timer will expire 3 times, + // and on the 3rd time the diff timer will expire too. + // The diff timer is honoured first, so there will be + // only two keepalives. + + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Send GetIsAlive()"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive IsAlive()"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Send GetIsAlive()"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive IsAlive()"); + + TEST_THAT(reader.GetLine(line)); + std::string comp = "Send StoreFile(0x3,"; + TEST_THAT(line.substr(0, comp.size()) == comp); + comp = ",0x0,\"f1\")"; + TEST_THAT(line.substr(line.size() - comp.size()) + == comp); + } + // restore timers for rest of tests Timers::Init(); } From boxbackup-dev at fluffy.co.uk Tue Nov 28 20:34:58 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Tue, 28 Nov 2006 20:34:58 +0000 Subject: [Box Backup-commit] COMMIT r1184 - in box/chris/merge: bin/bbackupd lib/backupclient Message-ID: Author: chris Date: 2006-11-28 20:34:57 +0000 (Tue, 28 Nov 2006) New Revision: 1184 Modified: box/chris/merge/bin/bbackupd/BackupClientContext.cpp box/chris/merge/bin/bbackupd/BackupClientContext.h box/chris/merge/lib/backupclient/BackupStoreFile.h box/chris/merge/lib/backupclient/BackupStoreFileDiff.cpp Log: Replace old-style setitimers for KeepAliveTime and MaximumDiffingTime with new Timer objects. (refs #3, refs #9) Modified: box/chris/merge/bin/bbackupd/BackupClientContext.cpp =================================================================== --- box/chris/merge/bin/bbackupd/BackupClientContext.cpp 2006-11-28 20:30:36 UTC (rev 1183) +++ box/chris/merge/bin/bbackupd/BackupClientContext.cpp 2006-11-28 20:34:57 UTC (rev 1184) @@ -67,8 +67,7 @@ mStorageLimitExceeded(false), mpExcludeFiles(0), mpExcludeDirs(0), - mbIsManaged(false), - mTimeMgmtEpoch(0) + mbIsManaged(false) { } @@ -142,7 +141,7 @@ ASSERT(mpExtendedLogFileHandle == NULL); mpExtendedLogFileHandle = fopen( - mExtendedLogFile.c_str(), "w"); + mExtendedLogFile.c_str(), "a+"); if (!mpExtendedLogFileHandle) { @@ -338,8 +337,8 @@ // -------------------------------------------------------------------------- // // Function -// Name: -// Purpose: +// Name: BackupClientContext::PerformDeletions() +// Purpose: Perform any pending file deletions. // Created: 10/11/03 // // -------------------------------------------------------------------------- @@ -496,7 +495,6 @@ return true; } - // maximum time to spend diffing static int sMaximumDiffTime = 600; // maximum time of SSL inactivity (keep-alive interval) @@ -517,19 +515,6 @@ // -------------------------------------------------------------------------- // // Function -// Name: static TimerSigHandler(int) -// Purpose: Signal handler -// Created: 19/3/04 -// -// -------------------------------------------------------------------------- -static void TimerSigHandler(int iUnused) -{ - BackupStoreFile::DiffTimerExpired(); -} - -// -------------------------------------------------------------------------- -// -// Function // Name: BackupClientContext::ManageDiffProcess() // Purpose: Initiates a file diff control timer // Created: 04/19/2005 @@ -537,59 +522,8 @@ // -------------------------------------------------------------------------- void BackupClientContext::ManageDiffProcess() { - if (mbIsManaged || !mpConnection) - return; - - ASSERT(mTimeMgmtEpoch == 0); - -#ifdef PLATFORM_CYGWIN - ::signal(SIGALRM, TimerSigHandler); -#elif defined WIN32 - // no support for SIGVTALRM - SetTimerHandler(TimerSigHandler); -#else - ::signal(SIGVTALRM, TimerSigHandler); -#endif // PLATFORM_CYGWIN - - struct itimerval timeout; - memset(&timeout, 0, sizeof(timeout)); - - // - // - // - if (sMaximumDiffTime <= 0 && sKeepAliveTime <= 0) - { - TRACE0("Diff control not requested - letting things run wild\n"); - return; - } - else if (sMaximumDiffTime > 0 && sKeepAliveTime > 0) - { - timeout.it_value.tv_sec = sKeepAliveTime < sMaximumDiffTime ? sKeepAliveTime : sMaximumDiffTime; - timeout.it_interval.tv_sec = sKeepAliveTime < sMaximumDiffTime ? sKeepAliveTime : sMaximumDiffTime; - } - else - { - timeout.it_value.tv_sec = sKeepAliveTime > 0 ? sKeepAliveTime : sMaximumDiffTime; - timeout.it_interval.tv_sec = sKeepAliveTime > 0 ? sKeepAliveTime : sMaximumDiffTime; - } - - // avoid race - mTimeMgmtEpoch = time(NULL); - -#ifdef PLATFORM_CYGWIN - if(::setitimer(ITIMER_REAL, &timeout, NULL) != 0) -#else - if(::setitimer(ITIMER_VIRTUAL, &timeout, NULL) != 0) -#endif // PLATFORM_CYGWIN - { - mTimeMgmtEpoch = 0; - - TRACE0("WARNING: couldn't set file diff control timeout\n"); - THROW_EXCEPTION(BackupStoreException, Internal) - } - + ASSERT(!mbIsManaged); mbIsManaged = true; - TRACE0("Initiated timer for file diff control\n"); } // -------------------------------------------------------------------------- @@ -602,26 +536,8 @@ // -------------------------------------------------------------------------- void BackupClientContext::UnManageDiffProcess() { - if (!mbIsManaged /* don't test for active connection, just do it */) - return; - - struct itimerval timeout; - memset(&timeout, 0, sizeof(timeout)); - -#ifdef PLATFORM_CYGWIN - if(::setitimer(ITIMER_REAL, &timeout, NULL) != 0) -#else - if(::setitimer(ITIMER_VIRTUAL, &timeout, NULL) != 0) -#endif // PLATFORM_CYGWIN - { - TRACE0("WARNING: couldn't clear file diff control timeout\n"); - THROW_EXCEPTION(BackupStoreException, Internal) - } - + // ASSERT(mbIsManaged); mbIsManaged = false; - mTimeMgmtEpoch = 0; - - TRACE0("Suspended timer for file diff control\n"); } // -------------------------------------------------------------------------- @@ -643,26 +559,12 @@ mpConnection->QueryGetIsAlive(); } -// -------------------------------------------------------------------------- -// -// Function -// Name: BackupClientContext::GetTimeMgmtEpoch() -// Purpose: Returns the unix time when the diff was started, or zero -// if the diff process is unmanaged. -// Created: 04/19/2005 -// -// -------------------------------------------------------------------------- -time_t BackupClientContext::GetTimeMgmtEpoch() -{ - return mTimeMgmtEpoch; -} - int BackupClientContext::GetMaximumDiffingTime() { return sMaximumDiffTime; } -int BackupClientContext::GetKeepaliveTime() +int BackupClientContext::GetKeepAliveTime() { return sKeepAliveTime; } Modified: box/chris/merge/bin/bbackupd/BackupClientContext.h =================================================================== --- box/chris/merge/bin/bbackupd/BackupClientContext.h 2006-11-28 20:30:36 UTC (rev 1183) +++ box/chris/merge/bin/bbackupd/BackupClientContext.h 2006-11-28 20:34:57 UTC (rev 1184) @@ -193,9 +193,9 @@ // // -------------------------------------------------------------------------- virtual void DoKeepAlive(); - virtual time_t GetTimeMgmtEpoch(); virtual int GetMaximumDiffingTime(); - virtual int GetKeepaliveTime(); + virtual int GetKeepAliveTime(); + virtual bool IsManaged() { return mbIsManaged; } private: BackupDaemon &mrDaemon; @@ -217,9 +217,6 @@ ExcludeList *mpExcludeDirs; bool mbIsManaged; - // unix time when diff was started - time_t mTimeMgmtEpoch; }; - #endif // BACKUPCLIENTCONTEXT__H Modified: box/chris/merge/lib/backupclient/BackupStoreFile.h =================================================================== --- box/chris/merge/lib/backupclient/BackupStoreFile.h 2006-11-28 20:30:36 UTC (rev 1183) +++ box/chris/merge/lib/backupclient/BackupStoreFile.h 2006-11-28 20:34:57 UTC (rev 1184) @@ -51,16 +51,16 @@ virtual ~DiffTimer(); public: virtual void DoKeepAlive() = 0; - virtual time_t GetTimeMgmtEpoch() = 0; virtual int GetMaximumDiffingTime() = 0; - virtual int GetKeepaliveTime() = 0; + virtual int GetKeepAliveTime() = 0; + virtual bool IsManaged() = 0; }; // -------------------------------------------------------------------------- // // Class // Name: BackupStoreFile -// Purpose: Class to hold together utils for maniplating files. +// Purpose: Class to hold together utils for manipulating files. // Created: 2003/08/28 // // -------------------------------------------------------------------------- Modified: box/chris/merge/lib/backupclient/BackupStoreFileDiff.cpp =================================================================== --- box/chris/merge/lib/backupclient/BackupStoreFileDiff.cpp 2006-11-28 20:30:36 UTC (rev 1183) +++ box/chris/merge/lib/backupclient/BackupStoreFileDiff.cpp 2006-11-28 20:34:57 UTC (rev 1184) @@ -29,6 +29,7 @@ #include "RollingChecksum.h" #include "MD5Digest.h" #include "CommonException.h" +#include "Timer.h" #include "MemLeakFindOn.h" @@ -52,34 +53,9 @@ BlocksAvailableEntry *pIndex, std::map &rFoundBlocks); static void GenerateRecipe(BackupStoreFileEncodeStream::Recipe &rRecipe, BlocksAvailableEntry *pIndex, int64_t NumBlocks, std::map &rFoundBlocks, int64_t SizeOfInputFile); -// sDiffTimerExpired flags when the diff timer has expired. When true, the -// diff routine should check the wall clock as soon as possible, to determine -// whether it's time for a keepalive to be sent, or whether the diff has been -// running for too long and should be terminated. -static bool sDiffTimerExpired = false; - - // -------------------------------------------------------------------------- // // Function -// Name: BackupStoreFile::DiffTimerExpired() -// Purpose: Notifies BackupStoreFile object that the diff operation -// timer has expired, which may mean that a keepalive should -// be sent, or the diff should be terminated. Called from an -// external timer, so it should not do more than set a flag. -// -// Created: 19/1/06 -// -// -------------------------------------------------------------------------- -void BackupStoreFile::DiffTimerExpired() -{ - sDiffTimerExpired = true; -} - - -// -------------------------------------------------------------------------- -// -// Function // Name: BackupStoreFile::MoveStreamPositionToBlockIndex(IOStream &) // Purpose: Move the file pointer in this stream to just before the block index. // Assumes that the stream is at the beginning, seekable, and @@ -483,15 +459,13 @@ BlocksAvailableEntry *pIndex, int64_t NumBlocks, int32_t Sizes[BACKUP_FILE_DIFF_MAX_BLOCK_SIZES], DiffTimer *pDiffTimer) { - time_t TimeMgmtEpoch = 0; - int MaximumDiffingTime = 0; - int KeepAliveTime = 0; + Timer maximumDiffingTime(0); + Timer keepAliveTime(0); - if (pDiffTimer) + if (pDiffTimer && pDiffTimer->IsManaged()) { - TimeMgmtEpoch = pDiffTimer->GetTimeMgmtEpoch(); - MaximumDiffingTime = pDiffTimer->GetMaximumDiffingTime(); - KeepAliveTime = pDiffTimer->GetKeepaliveTime(); + maximumDiffingTime = Timer(pDiffTimer->GetMaximumDiffingTime()); + keepAliveTime = Timer(pDiffTimer->GetKeepAliveTime()); } std::map goodnessOfFit; @@ -577,30 +551,24 @@ int rollOverInitialBytes = 0; while(true) { - if(sDiffTimerExpired) + if(maximumDiffingTime.HasExpired()) { - ASSERT(TimeMgmtEpoch > 0); ASSERT(pDiffTimer != NULL); - - time_t tTotalRunIntvl = time(NULL) - TimeMgmtEpoch; - - if(MaximumDiffingTime > 0 && - tTotalRunIntvl >= MaximumDiffingTime) - { - TRACE0("MaximumDiffingTime reached - " - "suspending file diff\n"); - abortSearch = true; - break; - } - else if(KeepAliveTime > 0) - { - TRACE0("KeepAliveTime reached - " - "initiating keep-alive\n"); - pDiffTimer->DoKeepAlive(); - } - - sDiffTimerExpired = false; + TRACE0("MaximumDiffingTime reached - " + "suspending file diff\n"); + abortSearch = true; + break; } + + if(keepAliveTime.HasExpired()) + { + ASSERT(pDiffTimer != NULL); + TRACE0("KeepAliveTime reached - " + "initiating keep-alive\n"); + pDiffTimer->DoKeepAlive(); + keepAliveTime = Timer( + pDiffTimer->GetKeepAliveTime()); + } // Load in another block of data, and record how big it is int bytesInEndings = rFile.Read(endings, Sizes[s]); From boxbackup-dev at fluffy.co.uk Tue Nov 28 20:36:11 2006 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Tue, 28 Nov 2006 20:36:11 -0000 Subject: [Box Backup-commit] Re: #9: SSL connection may time out when backing up large data sets In-Reply-To: <075.382a81ac9c31eb8a9dfae053eb0061b0@fluffy.co.uk> References: <075.382a81ac9c31eb8a9dfae053eb0061b0@fluffy.co.uk> Message-ID: <084.274d59aaacc482c089cd75c5bf586377@fluffy.co.uk> #9: SSL connection may time out when backing up large data sets -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: defect | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: 0.10 Resolution: | Keywords: ssl timeout client reload scanning directories -----------------------+---------------------------------------------------- Comment (by chris): Also required: [1182], [1183]. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX.