From boxbackup-dev at fluffy.co.uk Sat Mar 3 19:47:00 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 19:47:00 +0000 Subject: [Box Backup-commit] COMMIT r1303 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-03 19:47:00 +0000 (Sat, 03 Mar 2007) New Revision: 1303 Modified: box/chris/general/lib/server/ServerControl.h Log: Stricter tests for kill() success in HUPServer and KillServerInternal on Unix (merge with merge branch r1302) Modified: box/chris/general/lib/server/ServerControl.h =================================================================== --- box/chris/general/lib/server/ServerControl.h 2007-02-25 15:33:17 UTC (rev 1302) +++ box/chris/general/lib/server/ServerControl.h 2007-03-03 19:47:00 UTC (rev 1303) @@ -125,13 +125,13 @@ inline bool HUPServer(int pid) { if(pid == 0) return false; - return ::kill(pid, SIGHUP) != -1; + return ::kill(pid, SIGHUP) == 0; } inline bool KillServerInternal(int pid) { if(pid == 0 || pid == -1) return false; - bool killed = (::kill(pid, SIGTERM) != -1); + bool killed = (::kill(pid, SIGTERM) == 0); TEST_THAT(killed); return killed; } @@ -140,7 +140,10 @@ inline bool KillServer(int pid) { - KillServerInternal(pid); + if (!KillServerInternal(pid)) + { + return false; + } for (int i = 0; i < 30; i++) { From boxbackup-dev at fluffy.co.uk Sat Mar 3 19:57:31 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 19:57:31 +0000 Subject: [Box Backup-commit] COMMIT r1304 - box/chris/general/lib/win32 Message-ID: Author: chris Date: 2007-03-03 19:57:31 +0000 (Sat, 03 Mar 2007) New Revision: 1304 Modified: box/chris/general/lib/win32/emu.cpp Log: Spacing Modified: box/chris/general/lib/win32/emu.cpp =================================================================== --- box/chris/general/lib/win32/emu.cpp 2007-03-03 19:47:00 UTC (rev 1303) +++ box/chris/general/lib/win32/emu.cpp 2007-03-03 19:57:31 UTC (rev 1304) @@ -138,6 +138,7 @@ void InitTimer(void) { assert(!gTimerInitialised); + InitializeCriticalSection(&gLock); // create our thread From boxbackup-dev at fluffy.co.uk Sat Mar 3 19:59:20 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 19:59:20 +0000 Subject: [Box Backup-commit] COMMIT r1305 - box/chris/general Message-ID: Author: chris Date: 2007-03-03 19:59:20 +0000 (Sat, 03 Mar 2007) New Revision: 1305 Modified: box/chris/general/modules.txt Log: Spacing Modified: box/chris/general/modules.txt =================================================================== --- box/chris/general/modules.txt 2007-03-03 19:57:31 UTC (rev 1304) +++ box/chris/general/modules.txt 2007-03-03 19:59:20 UTC (rev 1305) @@ -14,6 +14,7 @@ lib/server lib/compress lib/intercept + test/common test/crypto lib/crypto test/compress lib/compress From boxbackup-dev at fluffy.co.uk Sat Mar 3 20:22:34 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 20:22:34 +0000 Subject: [Box Backup-commit] COMMIT r1306 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 20:22:34 +0000 (Sat, 03 Mar 2007) New Revision: 1306 Modified: box/chris/general/lib/common/Logging.h Log: Win32 compile fix (refs #3) Modified: box/chris/general/lib/common/Logging.h =================================================================== --- box/chris/general/lib/common/Logging.h 2007-03-03 19:59:20 UTC (rev 1305) +++ box/chris/general/lib/common/Logging.h 2007-03-03 20:22:34 UTC (rev 1306) @@ -42,10 +42,21 @@ #define BOX_TRACE(stuff) BOX_LOG(Log::TRACE, stuff) #endif +#undef ERROR + namespace Log { - enum Level { NOTHING = 1, FATAL, ERROR, WARNING, NOTICE, INFO, TRACE, - EVERYTHING }; + enum Level + { + NOTHING = 1, + FATAL, + ERROR, + WARNING, + NOTICE, + INFO, + TRACE, + EVERYTHING + }; } // -------------------------------------------------------------------------- From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:26:04 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:26:04 +0000 Subject: [Box Backup-commit] COMMIT r1307 - box/chris/general/infrastructure Message-ID: Author: chris Date: 2007-03-03 21:26:04 +0000 (Sat, 03 Mar 2007) New Revision: 1307 Modified: box/chris/general/infrastructure/makebuildenv.pl.in Log: Try to kill any running daemons before and after each test. (merged back from chris/merge) Modified: box/chris/general/infrastructure/makebuildenv.pl.in =================================================================== --- box/chris/general/infrastructure/makebuildenv.pl.in 2007-03-03 20:22:34 UTC (rev 1306) +++ box/chris/general/infrastructure/makebuildenv.pl.in 2007-03-03 21:26:04 UTC (rev 1307) @@ -401,31 +401,53 @@ sub writetestfile { my ($filename,$runcmd,$module) = @_; - open TESTFILE,">$filename" or die "Can't open test script file for $module for writing\n"; + + open TESTFILE,">$filename" or die "Can't open " . + "test script file for $module for writing\n"; print TESTFILE "#!/bin/sh\necho TEST: $module\n"; + if(-d "$module/testfiles") { print TESTFILE <<__E; +echo Killing any running daemons... +test -r testfiles/bbackupd.pid && kill `cat testfiles/bbackupd.pid` +test -r testfiles/bbstored.pid && kill `cat testfiles/bbstored.pid` + echo Removing old test files... rm -rf testfiles + echo Copying new test files... cp -p -R ../../../$module/testfiles . + __E } + if(-e "$module/testextra") { - open FL,"$module/testextra" or die "Can't open $module/testextra"; + open FL,"$module/testextra" or die + "Can't open $module/testextra"; while() {print TESTFILE} close FL; } + print TESTFILE "$runcmd\n"; + + if(-d "$module/testfiles") + { + print TESTFILE <<__E; +# echo Killing any running daemons... +test -r testfiles/bbackupd.pid && kill `cat testfiles/bbackupd.pid` +test -r testfiles/bbstored.pid && kill `cat testfiles/bbstored.pid` +__E + } + close TESTFILE; } - writetestfile("$mod/_t", + writetestfile("$mod/_t", "GLIBCXX_FORCE_NEW=1 ". './test' . $platform_exe_ext . ' $1 $2 $3 $4 $5', $mod); - writetestfile("$mod/_t-gdb", - 'gdb ./test' . $platform_exe_ext, $mod); + writetestfile("$mod/_t-gdb", "GLIBCXX_FORCE_NEW=1 ". + 'gdb ./test' . $platform_exe_ext . ' $*', $mod); } From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:28:26 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:28:26 +0000 Subject: [Box Backup-commit] COMMIT r1308 - in box/chris/general: . infrastructure Message-ID: Author: chris Date: 2007-03-03 21:28:26 +0000 (Sat, 03 Mar 2007) New Revision: 1308 Added: box/chris/general/infrastructure/makedistribution.pl.in Removed: box/chris/general/infrastructure/makedistribution.pl Modified: box/chris/general/configure.ac Log: Check for dlfcn.h and dlsym (for intercepts) Check for gettimeofday() (for timers) Auto-generate makedistribution.pl (requires path to perl) Modified: box/chris/general/configure.ac =================================================================== --- box/chris/general/configure.ac 2007-03-03 21:26:04 UTC (rev 1307) +++ box/chris/general/configure.ac 2007-03-03 21:28:26 UTC (rev 1308) @@ -102,7 +102,7 @@ AC_HEADER_STDC AC_HEADER_SYS_WAIT -AC_CHECK_HEADERS([execinfo.h process.h pwd.h signal.h]) +AC_CHECK_HEADERS([dlfcn.h execinfo.h process.h pwd.h signal.h]) AC_CHECK_HEADERS([syslog.h time.h]) AC_CHECK_HEADERS([netinet/in.h]) AC_CHECK_HEADERS([sys/param.h sys/socket.h sys/time.h sys/types.h sys/wait.h]) @@ -115,6 +115,8 @@ AC_SEARCH_LIBS([regcomp], ["pcreposix -lpcre"]) fi +AC_SEARCH_LIBS([dlsym], ["dl"]) + ### Checks for typedefs, structures, and compiler characteristics. AC_CHECK_TYPES([u_int8_t, u_int16_t, u_int32_t, u_int64_t]) @@ -166,7 +168,7 @@ AC_FUNC_ERROR_AT_LINE AC_TYPE_SIGNAL AC_FUNC_STAT -AC_CHECK_FUNCS([getpeereid lchown setproctitle getpid]) +AC_CHECK_FUNCS([getpeereid lchown setproctitle getpid gettimeofday]) # NetBSD implements kqueue too differently for us to get it fixed by 0.10 # TODO: Remove this when NetBSD kqueue implementation is working netbsd_hack=`echo $target_os | sed 's/netbsd.*/netbsd/'` @@ -246,6 +248,7 @@ bin/bbstored/bbstored-config infrastructure/makebuildenv.pl infrastructure/makeparcels.pl + infrastructure/makedistribution.pl lib/common/makeexception.pl lib/raidfile/raidfile-config lib/server/makeprotocol.pl Deleted: box/chris/general/infrastructure/makedistribution.pl =================================================================== --- box/chris/general/infrastructure/makedistribution.pl 2007-03-03 21:26:04 UTC (rev 1307) +++ box/chris/general/infrastructure/makedistribution.pl 2007-03-03 21:28:26 UTC (rev 1308) @@ -1,327 +0,0 @@ -#!@PERL@ -use strict; -use Symbol; - -# comment string for various endings -my %comment_chars = ('cpp' => '// ', 'h' => '// ', 'pl' => '# ', 'pm' => '# ', '' => '# '); - -# other extensions which need text copying, just to remove the private stuff -my %text_files = ('txt' => 1); - -# files which don't get the license added -my %no_license = (); # 'filename' => 1 - -# ---------------------------------------------- - -# filled in from the manifest file -my %no_license_dir = (); - -# distribution name -my $distribution = $ARGV[0]; -die "No distribution name specified on the command line" if $distribution eq ''; -my $dist_root = "distribution/$distribution"; - -# check distribution exists -die "Distribution '$distribution' does not exist" unless -d $dist_root; - -# get version -open VERSION,"$dist_root/VERSION.txt" or die "Can't open $dist_root/VERSION.txt"; -my $version = ; -chomp $version; -my $archive_name = ; -chomp $archive_name; -close VERSION; - -# consistency check -die "Archive name '$archive_name' is not equal to the distribution name '$distribution'" - unless $archive_name eq $distribution; - -# make initial directory -my $base_name = "$archive_name-$version"; -system "rm -rf $base_name"; -system "rm $base_name.tgz"; -mkdir $base_name,0755; - -# get license file -open LICENSE,"$dist_root/LICENSE.txt" or die "Can't open $dist_root/LICENSE.txt"; -my $license_f; -read LICENSE,$license_f,100000; -close LICENSE; -my $svnversion = `svnversion .`; -chomp $svnversion; -my @license = ('distribution '.$base_name.' (svn version: '.$svnversion.')',split(/\n/,$license_f)); - -# copy files, make a note of all the modules included -my %modules_included; -my $private_sections_removed = 0; -my $non_distribution_sections_removed = 0; -sub copy_from_list -{ - my $list = $_[0]; - open LIST,$list or die "Can't open $list"; - - while() - { - next unless m/\S/; - chomp; - my ($src,$dst) = split /\s+/; - $dst = $src if $dst eq ''; - if($src eq 'MKDIR') - { - # actually we just need to make a directory here - mkdir "$base_name/$dst",0755; - } - elsif($src eq 'NO-LICENSE-IN-DIR') - { - # record that this directory shouldn't have the license added - $no_license_dir{$dst} = 1; - } - elsif($src eq 'REPLACE-VERSION-IN') - { - replace_version_in($dst); - } - elsif($src eq 'NO-LICENSE') - { - $no_license{$dst} = 1; - } - elsif($src eq 'RUN') - { - print "Running $dst...\n"; - if(system($dst) != 0) - { - print "Error running $dst. Aborting.\n"; - exit(1); - } - } - elsif(-d $src) - { - $modules_included{$_} = 1; - copy_dir($src,$dst); - } - else - { - copy_file($src,$dst); - } - } - - close LIST; -} -copy_from_list("distribution/COMMON-MANIFEST.txt"); -copy_from_list("$dist_root/DISTRIBUTION-MANIFEST.txt"); - -# Copy in the root directory and delete the DISTRIBUTION-MANIFEST file -(system("cp $dist_root/*.* $base_name/") == 0) - or die "Copy of root extra files failed"; -unlink "$base_name/DISTRIBUTION-MANIFEST.txt" - or die "Delete of DISTRIBUTION-MANIFEST.txt file failed"; - -# produce a new modules file -my $modules = gensym; -open $modules,"modules.txt" or die "Can't open modules.txt for reading"; -open MODULES_OUT,">$base_name/modules.txt"; - -while(<$modules>) -{ - # skip lines for modules which aren't included - next if m/\A(\w+\/\w+)\s/ && !exists $modules_included{$1}; - - # skip private sections - unless(skip_non_applicable_section($_, $modules, 'modules.txt')) - { - # copy line to out files - print MODULES_OUT - } -} - -close MODULES_OUT; -close $modules; - -# report on how many private sections were removed -print "Private sections removed: $private_sections_removed\nNon-distribution sections removed: $non_distribution_sections_removed\n"; - -# tar it up -system "tar cf - $base_name | gzip -9 - > $base_name.tgz"; - -sub copy_file -{ - my ($fn,$dst_fn) = @_; - - my $ext; - $ext = $1 if $fn =~ m/\.(\w+)\Z/; - - # licenses not used in this directory? - my $license_in_dir = 1; - $dst_fn =~ m~\A(.+)/[^/]+?\Z~; - $license_in_dir = 0 if exists $no_license_dir{$1}; - - # licensed or not? - if(exists $comment_chars{$ext} && !exists $no_license{$fn} && $license_in_dir) - { - my $b = $comment_chars{$ext}; - - # copy as text, inserting license - my $in = gensym; - open $in,$fn; - open OUT,">$base_name/$dst_fn"; - - my $first = <$in>; - if($first =~ m/\A#!/) - { - print OUT $first; - $first = ''; - } - - # write license - for(@license) - { - print OUT $b,$_,"\n" - } - - if($first ne '') - { - print OUT $first; - } - - while(<$in>) - { - unless(skip_non_applicable_section($_, $in, $fn)) - { - print OUT - } - } - - close OUT; - close $in; - } - else - { - if(exists $text_files{$ext}) - { - # copy this as text, to remove private stuff - my $in = gensym; - open $in,$fn; - open OUT,">$base_name/$dst_fn"; - - while(<$in>) - { - unless(skip_non_applicable_section($_, $in, $fn)) - { - print OUT - } - } - - close OUT; - close $in; - } - else - { - # copy as binary - system 'cp',$fn,"$base_name/$dst_fn" - } - } - - # copy executable bit from src - if(-x $fn) - { - system 'chmod','a+x',"$base_name/$dst_fn" - } - else - { - system 'chmod','a-x',"$base_name/$dst_fn" - } -} - -sub skip_non_applicable_section -{ - my ($l, $filehandle, $filename) = @_; - if($l =~ m/BOX_PRIVATE_BEGIN/) - { - # skip private section - print "Removing private section from $filename\n"; - $private_sections_removed++; - while(<$filehandle>) {last if m/BOX_PRIVATE_END/} - - # skipped something - return 1; - } - elsif($l =~ m/IF_DISTRIBUTION\((.+?)\)/) - { - # which distributions does this apply to? - my $applies = 0; - for(split /,/,$1) - { - $applies = 1 if $_ eq $distribution - } - unless($applies) - { - # skip section? - print "Removing distribution specific section from $filename\n"; - $non_distribution_sections_removed++; - while(<$filehandle>) {last if m/END_IF_DISTRIBUTION/} - } - # hide this line - return 1; - } - elsif($l =~ m/END_IF_DISTRIBUTION/) - { - # hide these lines - return 1; - } - else - { - # no skipping, return this line - return 0; - } -} - -sub copy_dir -{ - my ($dir,$dst_dir) = @_; - - # copy an entire directory... first make sure it exists - my @n = split /\//,$dst_dir; - my $d = $base_name; - for(@n) - { - $d .= '/'; - $d .= $_; - mkdir $d,0755; - } - - # then do each of the files within in - opendir DIR,$dir; - my @items = readdir DIR; - closedir DIR; - - for(@items) - { - next if m/\A\./; - next if m/\A_/; - next if m/\AMakefile\Z/; - next if m/\Aautogen/; - next if !-f "$dir/$_"; - - copy_file("$dir/$_","$dst_dir/$_"); - } -} - -sub replace_version_in -{ - my ($file) = @_; - - my $fn = $base_name . '/' . $file; - open IN,$fn or die "Can't open $fn"; - open OUT,'>'.$fn.'.new' or die "Can't open $fn.new for writing"; - - while() - { - s/###DISTRIBUTION-VERSION-NUMBER###/$version/g; - print OUT - } - - close OUT; - close IN; - - rename($fn.'.new', $fn) or die "Can't rename in place $fn"; -} - Added: box/chris/general/infrastructure/makedistribution.pl.in =================================================================== --- box/chris/general/infrastructure/makedistribution.pl.in (rev 0) +++ box/chris/general/infrastructure/makedistribution.pl.in 2007-03-03 21:28:26 UTC (rev 1308) @@ -0,0 +1,327 @@ +#!@PERL@ +use strict; +use Symbol; + +# comment string for various endings +my %comment_chars = ('cpp' => '// ', 'h' => '// ', 'pl' => '# ', 'pm' => '# ', '' => '# '); + +# other extensions which need text copying, just to remove the private stuff +my %text_files = ('txt' => 1, 'spec' => 1); + +# files which don't get the license added +my %no_license = (); # 'filename' => 1 + +# ---------------------------------------------- + +# filled in from the manifest file +my %no_license_dir = (); + +# distribution name +my $distribution = $ARGV[0]; +die "No distribution name specified on the command line" if $distribution eq ''; +my $dist_root = "distribution/$distribution"; + +# check distribution exists +die "Distribution '$distribution' does not exist" unless -d $dist_root; + +# get version +open VERSION,"$dist_root/VERSION.txt" or die "Can't open $dist_root/VERSION.txt"; +my $version = ; +chomp $version; +my $archive_name = ; +chomp $archive_name; +close VERSION; + +# consistency check +die "Archive name '$archive_name' is not equal to the distribution name '$distribution'" + unless $archive_name eq $distribution; + +# make initial directory +my $base_name = "$archive_name-$version"; +system "rm -rf $base_name"; +system "rm $base_name.tgz"; +mkdir $base_name,0755; + +# get license file +open LICENSE,"$dist_root/LICENSE.txt" or die "Can't open $dist_root/LICENSE.txt"; +my $license_f; +read LICENSE,$license_f,100000; +close LICENSE; +my $svnversion = `svnversion .`; +chomp $svnversion; +my @license = ('distribution '.$base_name.' (svn version: '.$svnversion.')',split(/\n/,$license_f)); + +# copy files, make a note of all the modules included +my %modules_included; +my $private_sections_removed = 0; +my $non_distribution_sections_removed = 0; +sub copy_from_list +{ + my $list = $_[0]; + open LIST,$list or die "Can't open $list"; + + while() + { + next unless m/\S/; + chomp; + my ($src,$dst) = split /\s+/; + $dst = $src if $dst eq ''; + if($src eq 'MKDIR') + { + # actually we just need to make a directory here + mkdir "$base_name/$dst",0755; + } + elsif($src eq 'NO-LICENSE-IN-DIR') + { + # record that this directory shouldn't have the license added + $no_license_dir{$dst} = 1; + } + elsif($src eq 'REPLACE-VERSION-IN') + { + replace_version_in($dst); + } + elsif($src eq 'NO-LICENSE') + { + $no_license{$dst} = 1; + } + elsif($src eq 'RUN') + { + print "Running $dst...\n"; + if(system($dst) != 0) + { + print "Error running $dst. Aborting.\n"; + exit(1); + } + } + elsif(-d $src) + { + $modules_included{$_} = 1; + copy_dir($src,$dst); + } + else + { + copy_file($src,$dst); + } + } + + close LIST; +} +copy_from_list("distribution/COMMON-MANIFEST.txt"); +copy_from_list("$dist_root/DISTRIBUTION-MANIFEST.txt"); + +# Copy in the root directory and delete the DISTRIBUTION-MANIFEST file +(system("cp $dist_root/*.* $base_name/") == 0) + or die "Copy of root extra files failed"; +unlink "$base_name/DISTRIBUTION-MANIFEST.txt" + or die "Delete of DISTRIBUTION-MANIFEST.txt file failed"; + +# produce a new modules file +my $modules = gensym; +open $modules,"modules.txt" or die "Can't open modules.txt for reading"; +open MODULES_OUT,">$base_name/modules.txt"; + +while(<$modules>) +{ + # skip lines for modules which aren't included + next if m/\A(\w+\/\w+)\s/ && !exists $modules_included{$1}; + + # skip private sections + unless(skip_non_applicable_section($_, $modules, 'modules.txt')) + { + # copy line to out files + print MODULES_OUT + } +} + +close MODULES_OUT; +close $modules; + +# report on how many private sections were removed +print "Private sections removed: $private_sections_removed\nNon-distribution sections removed: $non_distribution_sections_removed\n"; + +# tar it up +system "tar cf - $base_name | gzip -9 - > $base_name.tgz"; + +sub copy_file +{ + my ($fn,$dst_fn) = @_; + + my $ext; + $ext = $1 if $fn =~ m/\.(\w+)\Z/; + + # licenses not used in this directory? + my $license_in_dir = 1; + $dst_fn =~ m~\A(.+)/[^/]+?\Z~; + $license_in_dir = 0 if exists $no_license_dir{$1}; + + # licensed or not? + if(exists $comment_chars{$ext} && !exists $no_license{$fn} && $license_in_dir) + { + my $b = $comment_chars{$ext}; + + # copy as text, inserting license + my $in = gensym; + open $in,$fn; + open OUT,">$base_name/$dst_fn"; + + my $first = <$in>; + if($first =~ m/\A#!/) + { + print OUT $first; + $first = ''; + } + + # write license + for(@license) + { + print OUT $b,$_,"\n" + } + + if($first ne '') + { + print OUT $first; + } + + while(<$in>) + { + unless(skip_non_applicable_section($_, $in, $fn)) + { + print OUT + } + } + + close OUT; + close $in; + } + else + { + if(exists $text_files{$ext}) + { + # copy this as text, to remove private stuff + my $in = gensym; + open $in,$fn; + open OUT,">$base_name/$dst_fn"; + + while(<$in>) + { + unless(skip_non_applicable_section($_, $in, $fn)) + { + print OUT + } + } + + close OUT; + close $in; + } + else + { + # copy as binary + system 'cp',$fn,"$base_name/$dst_fn" + } + } + + # copy executable bit from src + if(-x $fn) + { + system 'chmod','a+x',"$base_name/$dst_fn" + } + else + { + system 'chmod','a-x',"$base_name/$dst_fn" + } +} + +sub skip_non_applicable_section +{ + my ($l, $filehandle, $filename) = @_; + if($l =~ m/BOX_PRIVATE_BEGIN/) + { + # skip private section + print "Removing private section from $filename\n"; + $private_sections_removed++; + while(<$filehandle>) {last if m/BOX_PRIVATE_END/} + + # skipped something + return 1; + } + elsif($l =~ m/IF_DISTRIBUTION\((.+?)\)/) + { + # which distributions does this apply to? + my $applies = 0; + for(split /,/,$1) + { + $applies = 1 if $_ eq $distribution + } + unless($applies) + { + # skip section? + print "Removing distribution specific section from $filename\n"; + $non_distribution_sections_removed++; + while(<$filehandle>) {last if m/END_IF_DISTRIBUTION/} + } + # hide this line + return 1; + } + elsif($l =~ m/END_IF_DISTRIBUTION/) + { + # hide these lines + return 1; + } + else + { + # no skipping, return this line + return 0; + } +} + +sub copy_dir +{ + my ($dir,$dst_dir) = @_; + + # copy an entire directory... first make sure it exists + my @n = split /\//,$dst_dir; + my $d = $base_name; + for(@n) + { + $d .= '/'; + $d .= $_; + mkdir $d,0755; + } + + # then do each of the files within in + opendir DIR,$dir; + my @items = readdir DIR; + closedir DIR; + + for(@items) + { + next if m/\A\./; + next if m/\A_/; + next if m/\AMakefile\Z/; + next if m/\Aautogen/; + next if !-f "$dir/$_"; + + copy_file("$dir/$_","$dst_dir/$_"); + } +} + +sub replace_version_in +{ + my ($file) = @_; + + my $fn = $base_name . '/' . $file; + open IN,$fn or die "Can't open $fn"; + open OUT,'>'.$fn.'.new' or die "Can't open $fn.new for writing"; + + while() + { + s/###DISTRIBUTION-VERSION-NUMBER###/$version/g; + print OUT + } + + close OUT; + close IN; + + rename($fn.'.new', $fn) or die "Can't rename in place $fn"; +} + Property changes on: box/chris/general/infrastructure/makedistribution.pl.in ___________________________________________________________________ Name: svn:executable + * From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:29:49 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:29:49 +0000 Subject: [Box Backup-commit] COMMIT r1309 - box/chris/general/lib/backupclient Message-ID: Author: chris Date: 2007-03-03 21:29:49 +0000 (Sat, 03 Mar 2007) New Revision: 1309 Modified: box/chris/general/lib/backupclient/BackupStoreFileDiff.cpp Log: Use timers for managing diffs (from chris/merge) Modified: box/chris/general/lib/backupclient/BackupStoreFileDiff.cpp =================================================================== --- box/chris/general/lib/backupclient/BackupStoreFileDiff.cpp 2007-03-03 21:28:26 UTC (rev 1308) +++ box/chris/general/lib/backupclient/BackupStoreFileDiff.cpp 2007-03-03 21:29:49 UTC (rev 1309) @@ -29,6 +29,7 @@ #include "RollingChecksum.h" #include "MD5Digest.h" #include "CommonException.h" +#include "Timer.h" #include "MemLeakFindOn.h" @@ -52,34 +53,9 @@ BlocksAvailableEntry *pIndex, std::map &rFoundBlocks); static void GenerateRecipe(BackupStoreFileEncodeStream::Recipe &rRecipe, BlocksAvailableEntry *pIndex, int64_t NumBlocks, std::map &rFoundBlocks, int64_t SizeOfInputFile); -// sDiffTimerExpired flags when the diff timer has expired. When true, the -// diff routine should check the wall clock as soon as possible, to determine -// whether it's time for a keepalive to be sent, or whether the diff has been -// running for too long and should be terminated. -static bool sDiffTimerExpired = false; - - // -------------------------------------------------------------------------- // // Function -// Name: BackupStoreFile::DiffTimerExpired() -// Purpose: Notifies BackupStoreFile object that the diff operation -// timer has expired, which may mean that a keepalive should -// be sent, or the diff should be terminated. Called from an -// external timer, so it should not do more than set a flag. -// -// Created: 19/1/06 -// -// -------------------------------------------------------------------------- -void BackupStoreFile::DiffTimerExpired() -{ - sDiffTimerExpired = true; -} - - -// -------------------------------------------------------------------------- -// -// Function // Name: BackupStoreFile::MoveStreamPositionToBlockIndex(IOStream &) // Purpose: Move the file pointer in this stream to just before the block index. // Assumes that the stream is at the beginning, seekable, and @@ -483,15 +459,11 @@ BlocksAvailableEntry *pIndex, int64_t NumBlocks, int32_t Sizes[BACKUP_FILE_DIFF_MAX_BLOCK_SIZES], DiffTimer *pDiffTimer) { - time_t TimeMgmtEpoch = 0; - int MaximumDiffingTime = 0; - int KeepAliveTime = 0; + Timer maximumDiffingTime(0); - if (pDiffTimer) + if(pDiffTimer && pDiffTimer->IsManaged()) { - TimeMgmtEpoch = pDiffTimer->GetTimeMgmtEpoch(); - MaximumDiffingTime = pDiffTimer->GetMaximumDiffingTime(); - KeepAliveTime = pDiffTimer->GetKeepaliveTime(); + maximumDiffingTime = Timer(pDiffTimer->GetMaximumDiffingTime()); } std::map goodnessOfFit; @@ -577,31 +549,20 @@ int rollOverInitialBytes = 0; while(true) { - if(sDiffTimerExpired) + if(maximumDiffingTime.HasExpired()) { - ASSERT(TimeMgmtEpoch > 0); ASSERT(pDiffTimer != NULL); - - time_t tTotalRunIntvl = time(NULL) - TimeMgmtEpoch; - - if(MaximumDiffingTime > 0 && - tTotalRunIntvl >= MaximumDiffingTime) - { - TRACE0("MaximumDiffingTime reached - " - "suspending file diff\n"); - abortSearch = true; - break; - } - else if(KeepAliveTime > 0) - { - TRACE0("KeepAliveTime reached - " - "initiating keep-alive\n"); - pDiffTimer->DoKeepAlive(); - } - - sDiffTimerExpired = false; + TRACE0("MaximumDiffingTime reached - " + "suspending file diff\n"); + abortSearch = true; + break; } - + + if(pDiffTimer) + { + pDiffTimer->DoKeepAlive(); + } + // Load in another block of data, and record how big it is int bytesInEndings = rFile.Read(endings, Sizes[s]); int tmp; From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:34:46 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:34:46 +0000 Subject: [Box Backup-commit] COMMIT r1310 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-03 21:34:46 +0000 (Sat, 03 Mar 2007) New Revision: 1310 Modified: box/chris/general/lib/server/Daemon.cpp box/chris/general/lib/server/Daemon.h Log: Initialise and use logging. Allow Daemons to be instantiated more than once in a program. Add new Daemon::Main that takes parsed options instead of argv. Stop Daemon::Main from exiting the parent after forking. Stop Daemon::Main from interfering with the parent's signal handlers. (copied from chris/merge) Modified: box/chris/general/lib/server/Daemon.cpp =================================================================== --- box/chris/general/lib/server/Daemon.cpp 2007-03-03 21:29:49 UTC (rev 1309) +++ box/chris/general/lib/server/Daemon.cpp 2007-03-03 21:34:46 UTC (rev 1310) @@ -33,6 +33,7 @@ #include "Guards.h" #include "UnixUser.h" #include "FileModificationTime.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -48,11 +49,11 @@ // // -------------------------------------------------------------------------- Daemon::Daemon() - : mpConfiguration(0), + : mpConfiguration(NULL), mReloadConfigWanted(false), mTerminateWanted(false) { - if(spDaemon != 0) + if(spDaemon != NULL) { THROW_EXCEPTION(ServerException, AlreadyDaemonConstructed) } @@ -79,56 +80,139 @@ delete mpConfiguration; mpConfiguration = 0; } + + ASSERT(spDaemon == this); + spDaemon = NULL; } // -------------------------------------------------------------------------- // // Function // Name: Daemon::Main(const char *, int, const char *[]) -// Purpose: Starts the daemon off -- equivalent of C main() function +// Purpose: Parses command-line options, and then calls +// Main(std::string& configFile, bool singleProcess) +// to start the daemon. // Created: 2003/07/29 // // -------------------------------------------------------------------------- int Daemon::Main(const char *DefaultConfigFile, int argc, const char *argv[]) { + // Find filename of config file + mConfigFileName = DefaultConfigFile; + bool haveConfigFile = false; + bool singleProcess = false; + int masterLevel = Log::NOTICE; // need an int to do math with + char c; + + while((c = getopt(argc, (char * const *)argv, "c:Dqv")) != -1) + { + switch(c) + { + case 'c': + { + mConfigFileName = optarg; + haveConfigFile = true; + } + break; + + case 'D': + { + singleProcess = true; + } + break; + + case 'q': + { + if(masterLevel == Log::NOTHING) + { + BOX_FATAL("Too many '-q': " + "Cannot reduce logging " + "level any more"); + return 2; + } + masterLevel--; + } + break; + + case 'v': + { + if(masterLevel == Log::EVERYTHING) + { + BOX_FATAL("Too many '-v': " + "Cannot increase logging " + "level any more"); + return 2; + } + masterLevel++; + } + break; + + case '?': + { + BOX_FATAL("Unknown option on command line: " + << "'" << optopt << "'"); + return 2; + } + break; + + default: + { + BOX_FATAL("Unknown error in getopt: returned " + << "'" << c << "'"); + return 1; + } + } + } + + if (argc > optind && !haveConfigFile) + { + mConfigFileName = argv[optind]; optind++; + } + + if (argc > optind && ::strcmp(argv[optind], "SINGLEPROCESS") == 0) + { + singleProcess = true; optind++; + } + + if (argc > optind) + { + BOX_FATAL("Unknown parameter on command line: " + << "'" << std::string(argv[optind]) << "'"); + return 2; + } + + Logging::SetGlobalLevel((Log::Level)masterLevel); + + return Main(mConfigFileName, singleProcess); +} + +// -------------------------------------------------------------------------- +// +// Function +// Name: Daemon::Main(const std::string& rConfigFileName, +// bool singleProcess) +// Purpose: Starts the daemon off -- equivalent of C main() function +// Created: 2003/07/29 +// +// -------------------------------------------------------------------------- +int Daemon::Main(const std::string &rConfigFileName, bool singleProcess) +{ // Banner (optional) { const char *banner = DaemonBanner(); if(banner != 0) { - printf("%s", banner); + BOX_NOTICE(banner); } } std::string pidFileName; + mConfigFileName = rConfigFileName; + bool asDaemon = !singleProcess; + try { - // Find filename of config file - mConfigFileName = DefaultConfigFile; - if(argc >= 2) - { - // First argument is config file, or it's -c and the next arg is the config file - if(::strcmp(argv[1], "-c") == 0 && argc >= 3) - { - mConfigFileName = argv[2]; - } - else - { - mConfigFileName = argv[1]; - } - } - - // Test mode with no daemonisation? - bool asDaemon = true; - if(argc >= 3) - { - if(::strcmp(argv[2], "SINGLEPROCESS") == 0) - { - asDaemon = false; - } - } - // Load the configuration file. std::string errors; std::auto_ptr pconfig; @@ -144,16 +228,9 @@ if(e.GetType() == CommonException::ExceptionType && e.GetSubType() == CommonException::OSFileOpenError) { - fprintf(stderr, "%s: failed to start: " - "failed to open configuration file: " - "%s\n", DaemonName(), - mConfigFileName.c_str()); -#ifdef WIN32 - ::syslog(LOG_ERR, "%s: failed to start: " - "failed to open configuration file: " - "%s", DaemonName(), - mConfigFileName.c_str()); -#endif + BOX_FATAL("Failed to start: failed to open " + "configuration file: " + << mConfigFileName); return 1; } @@ -164,14 +241,8 @@ if(pconfig.get() == 0 || !errors.empty()) { // Tell user about errors - fprintf(stderr, "%s: Errors in config file %s:\n%s", - DaemonName(), mConfigFileName.c_str(), - errors.c_str()); -#ifdef WIN32 - ::syslog(LOG_ERR, "%s: Errors in config file %s:\n%s", - DaemonName(), mConfigFileName.c_str(), - errors.c_str()); -#endif + BOX_FATAL("Failed to start: errors in configuration " + "file: " << mConfigFileName << ": " << errors); // And give up return 1; } @@ -183,18 +254,6 @@ // Let the derived class have a go at setting up stuff in the initial process SetupInInitialProcess(); -#ifndef WIN32 - // Set signal handler - struct sigaction sa; - sa.sa_handler = SignalHandler; - sa.sa_flags = 0; - sigemptyset(&sa.sa_mask); // macro - if(::sigaction(SIGHUP, &sa, NULL) != 0 || ::sigaction(SIGTERM, &sa, NULL) != 0) - { - THROW_EXCEPTION(ServerException, DaemoniseFailed) - } -#endif // !WIN32 - // Server configuration const Configuration &serverConfig( mpConfiguration->GetSubConfiguration("Server")); @@ -232,7 +291,7 @@ default: // parent - _exit(0); + // _exit(0); return 0; break; @@ -269,14 +328,24 @@ break; } } -#endif // ! WIN32 - // open the log - ::openlog(DaemonName(), LOG_PID, LOG_LOCAL6); + // Set signal handler + // Don't do this in the parent, since it might be anything + // (e.g. test/bbackupd) + + struct sigaction sa; + sa.sa_handler = SignalHandler; + sa.sa_flags = 0; + sigemptyset(&sa.sa_mask); // macro + if(::sigaction(SIGHUP, &sa, NULL) != 0 || ::sigaction(SIGTERM, &sa, NULL) != 0) + { + THROW_EXCEPTION(ServerException, DaemoniseFailed) + } +#endif // !WIN32 // Log the start message - ::syslog(LOG_INFO, "Starting daemon (config: %s) (version " - BOX_VERSION ")", mConfigFileName.c_str()); + BOX_NOTICE("Starting daemon, version " << BOX_VERSION + << ", config: " << mConfigFileName); // Write PID to file char pid[32]; @@ -289,7 +358,7 @@ if(::write(pidFile, pid, pidsize) != pidsize) { - ::syslog(LOG_ERR, "can't write pid file"); + BOX_FATAL("can't write pid file"); THROW_EXCEPTION(ServerException, DaemoniseFailed) } @@ -330,37 +399,24 @@ // And definitely don't try and send anything to those file descriptors // -- this has in the past sent text to something which isn't expecting it. TRACE_TO_STDOUT(false); + Logging::ToConsole(false); } } catch(BoxException &e) { - fprintf(stderr, "%s: failed to start: exception %s (%d/%d)\n", - DaemonName(), e.what(), e.GetType(), e.GetSubType()); -#ifdef WIN32 - ::syslog(LOG_ERR, "%s: failed to start: " - "exception %s (%d/%d)\n", DaemonName(), - e.what(), e.GetType(), e.GetSubType()); -#endif + BOX_FATAL("Failed to start: exception " << e.what() + << " (" << e.GetType() + << "/" << e.GetSubType() << ")"); return 1; } catch(std::exception &e) { - fprintf(stderr, "%s: failed to start: exception %s\n", - DaemonName(), e.what()); -#ifdef WIN32 - ::syslog(LOG_ERR, "%s: failed to start: exception %s\n", - DaemonName(), e.what()); -#endif + BOX_FATAL("Failed to start: exception " << e.what()); return 1; } catch(...) { - fprintf(stderr, "%s: failed to start: unknown exception\n", - DaemonName()); -#ifdef WIN32 - ::syslog(LOG_ERR, "%s: failed to start: unknown exception\n", - DaemonName()); -#endif + BOX_FATAL("Failed to start: unknown error"); return 1; } @@ -373,7 +429,7 @@ if (WSAStartup(0x0101, &info) == SOCKET_ERROR) { // will not run without sockets - ::syslog(LOG_ERR, "Failed to initialise Windows Sockets"); + BOX_FATAL("Failed to initialise Windows Sockets"); THROW_EXCEPTION(CommonException, Internal) } #endif @@ -390,9 +446,8 @@ if(mReloadConfigWanted && !mTerminateWanted) { // Need to reload that config file... - ::syslog(LOG_INFO, "Reloading configuration " - "(config: %s)", - mConfigFileName.c_str()); + BOX_NOTICE("Reloading configuration file: " + << mConfigFileName); std::string errors; std::auto_ptr pconfig = Configuration::LoadAndVerify( @@ -403,10 +458,9 @@ if(pconfig.get() == 0 || !errors.empty()) { // Tell user about errors - ::syslog(LOG_ERR, "Errors in config " - "file %s:\n%s", - mConfigFileName.c_str(), - errors.c_str()); + BOX_FATAL("Error in configuration " + << "file: " << mConfigFileName + << ": " << errors); // And give up retcode = 1; break; @@ -430,25 +484,23 @@ ::unlink(pidFileName.c_str()); // Log - ::syslog(LOG_INFO, "Terminating daemon"); + BOX_NOTICE("Terminating daemon"); } catch(BoxException &e) { - ::syslog(LOG_ERR, "%s: terminating due to exception %s " - "(%d/%d)", DaemonName(), e.what(), e.GetType(), - e.GetSubType()); + BOX_FATAL("Terminating due to exception " << e.what() + << " (" << e.GetType() + << "/" << e.GetSubType() << ")"); retcode = 1; } catch(std::exception &e) { - ::syslog(LOG_ERR, "%s: terminating due to exception %s", - DaemonName(), e.what()); + BOX_FATAL("Terminating due to exception " << e.what()); retcode = 1; } catch(...) { - ::syslog(LOG_ERR, "%s: terminating due to unknown exception", - DaemonName()); + BOX_FATAL("Terminating due to unknown exception"); retcode = 1; } Modified: box/chris/general/lib/server/Daemon.h =================================================================== --- box/chris/general/lib/server/Daemon.h 2007-03-03 21:29:49 UTC (rev 1309) +++ box/chris/general/lib/server/Daemon.h 2007-03-03 21:34:46 UTC (rev 1310) @@ -41,6 +41,7 @@ public: int Main(const char *DefaultConfigFile, int argc, const char *argv[]); + int Main(const std::string &rConfigFile, bool singleProcess); virtual void Run(); const Configuration &GetConfiguration() const; From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:35:10 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:35:10 +0000 Subject: [Box Backup-commit] COMMIT r1311 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-03 21:35:10 +0000 (Sat, 03 Mar 2007) New Revision: 1311 Modified: box/chris/general/lib/server/TLSContext.cpp Log: Fix memory leak. (from chris/merge) Modified: box/chris/general/lib/server/TLSContext.cpp =================================================================== --- box/chris/general/lib/server/TLSContext.cpp 2007-03-03 21:34:46 UTC (rev 1310) +++ box/chris/general/lib/server/TLSContext.cpp 2007-03-03 21:35:10 UTC (rev 1311) @@ -61,6 +61,11 @@ // -------------------------------------------------------------------------- void TLSContext::Initialise(bool AsServer, const char *CertificatesFile, const char *PrivateKeyFile, const char *TrustedCAsFile) { + if(mpContext != 0) + { + ::SSL_CTX_free(mpContext); + } + mpContext = ::SSL_CTX_new(AsServer?TLSv1_server_method():TLSv1_client_method()); if(mpContext == NULL) { From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:35:54 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:35:54 +0000 Subject: [Box Backup-commit] COMMIT r1312 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-03 21:35:54 +0000 (Sat, 03 Mar 2007) New Revision: 1312 Modified: box/chris/general/lib/server/makeprotocol.pl.in Log: Add missing newlines in debug output (from chris/merge) Modified: box/chris/general/lib/server/makeprotocol.pl.in =================================================================== --- box/chris/general/lib/server/makeprotocol.pl.in 2007-03-03 21:35:10 UTC (rev 1311) +++ box/chris/general/lib/server/makeprotocol.pl.in 2007-03-03 21:35:54 UTC (rev 1312) @@ -829,8 +829,8 @@ } if($implement_filelog) { - $fR .= qq~\tif(mLogToFile) { ::fprintf(mLogToFile, (Size==Protocol::ProtocolStream_SizeUncertain)?"Receiving stream, size uncertain":"Receiving stream, size %d\\n", Size); ::fflush(mLogToFile); }\n~; - $fS .= qq~\tif(mLogToFile) { ::fprintf(mLogToFile, (Size==Protocol::ProtocolStream_SizeUncertain)?"Sending stream, size uncertain":"Sending stream, size %d\\n", Size); ::fflush(mLogToFile); }\n~; + $fR .= qq~\tif(mLogToFile) { ::fprintf(mLogToFile, (Size==Protocol::ProtocolStream_SizeUncertain)?"Receiving stream, size uncertain\\n":"Receiving stream, size %d\\n", Size); ::fflush(mLogToFile); }\n~; + $fS .= qq~\tif(mLogToFile) { ::fprintf(mLogToFile, (Size==Protocol::ProtocolStream_SizeUncertain)?"Sending stream, size uncertain\\n":"Sending stream, size %d\\n", Size); ::fflush(mLogToFile); }\n~; } print CPP <<__E; From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:36:16 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:36:16 +0000 Subject: [Box Backup-commit] COMMIT r1313 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-03 21:36:16 +0000 (Sat, 03 Mar 2007) New Revision: 1313 Modified: box/chris/general/lib/server/ServerStream.h Log: Fix typo Modified: box/chris/general/lib/server/ServerStream.h =================================================================== --- box/chris/general/lib/server/ServerStream.h 2007-03-03 21:35:54 UTC (rev 1312) +++ box/chris/general/lib/server/ServerStream.h 2007-03-03 21:36:16 UTC (rev 1313) @@ -261,7 +261,7 @@ else { #endif // !WIN32 - // Just handle in this connection + // Just handle in this process SetProcessTitle("handling"); HandleConnection(*connection); SetProcessTitle("idle"); From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:40:15 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:40:15 +0000 Subject: [Box Backup-commit] COMMIT r1314 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 21:40:15 +0000 (Sat, 03 Mar 2007) New Revision: 1314 Modified: box/chris/general/lib/common/DebugMemLeakFinder.cpp box/chris/general/lib/common/MemLeakFinder.h Log: Improve leak finder to track some leaks in allocations by standard libs. Added memleakfinder_init() which must be called in main(). Added a MemLeakSuppressionGuard class to suppress leaks in test code. (from chris/merge) Modified: box/chris/general/lib/common/DebugMemLeakFinder.cpp =================================================================== --- box/chris/general/lib/common/DebugMemLeakFinder.cpp 2007-03-03 21:36:16 UTC (rev 1313) +++ box/chris/general/lib/common/DebugMemLeakFinder.cpp 2007-03-03 21:40:15 UTC (rev 1314) @@ -25,6 +25,9 @@ #include #include +#include "MemLeakFinder.h" + +static bool memleakfinder_initialised = false; bool memleakfinder_global_enable = false; typedef struct @@ -55,11 +58,41 @@ static std::set sNotLeaks; void *sNotLeaksPre[1024]; - int sNotLeaksPreNum = 0; + size_t sNotLeaksPreNum = 0; } +void memleakfinder_init() +{ + ASSERT(!memleakfinder_initialised); + memleakfinder_initialised = true; +} + +MemLeakSuppressionGuard::MemLeakSuppressionGuard() +{ + ASSERT(memleakfinder_global_enable); + memleakfinder_global_enable = false; +} + +MemLeakSuppressionGuard::~MemLeakSuppressionGuard() +{ + ASSERT(!memleakfinder_global_enable); + memleakfinder_global_enable = true; +} + +// these functions may well allocate memory, which we don't want to track. +static int sInternalAllocDepth = 0; + +class InternalAllocGuard +{ + public: + InternalAllocGuard () { sInternalAllocDepth++; } + ~InternalAllocGuard() { sInternalAllocDepth--; } +}; + void memleakfinder_malloc_add_block(void *b, size_t size, const char *file, int line) { + InternalAllocGuard guard; + if(b != 0) { MallocBlockInfo i; @@ -75,11 +108,13 @@ } } - void *memleakfinder_malloc(size_t size, const char *file, int line) { + InternalAllocGuard guard; + void *b = ::malloc(size); if(!memleakfinder_global_enable) return b; + if(!memleakfinder_initialised) return b; memleakfinder_malloc_add_block(b, size, file, line); @@ -89,7 +124,9 @@ void *memleakfinder_realloc(void *ptr, size_t size) { - if(!memleakfinder_global_enable) + InternalAllocGuard guard; + + if(!memleakfinder_global_enable || !memleakfinder_initialised) { return ::realloc(ptr, size); } @@ -133,7 +170,9 @@ void memleakfinder_free(void *ptr) { - if(memleakfinder_global_enable) + InternalAllocGuard guard; + + if(memleakfinder_global_enable && memleakfinder_initialised) { // Check it's been allocated std::map::iterator i(sMallocBlocks.find(ptr)); @@ -143,7 +182,7 @@ } else { - TRACE1("Block %x freed, but not known. Error? Or allocated in startup static allocation?\n", ptr); + TRACE1("Block %p freed, but not known. Error? Or allocated in startup static allocation?\n", ptr); } if(sTrackMallocInSection) @@ -160,31 +199,41 @@ void memleakfinder_notaleak_insert_pre() { + InternalAllocGuard guard; + if(!memleakfinder_global_enable) return; - for(int l = 0; l < sNotLeaksPreNum; l++) + if(!memleakfinder_initialised) return; + + for(size_t l = 0; l < sNotLeaksPreNum; l++) { sNotLeaks.insert(sNotLeaksPre[l]); } + sNotLeaksPreNum = 0; } bool is_leak(void *ptr) { + InternalAllocGuard guard; + + ASSERT(memleakfinder_initialised); memleakfinder_notaleak_insert_pre(); return sNotLeaks.find(ptr) == sNotLeaks.end(); } void memleakfinder_notaleak(void *ptr) { + InternalAllocGuard guard; + memleakfinder_notaleak_insert_pre(); - if(memleakfinder_global_enable) + if(memleakfinder_global_enable && memleakfinder_initialised) { sNotLeaks.insert(ptr); } else { if ( sNotLeaksPreNum < - (unsigned)( sizeof(sNotLeaksPre)/sizeof(*sNotLeaksPre) ) ) + sizeof(sNotLeaksPre)/sizeof(*sNotLeaksPre) ) sNotLeaksPre[sNotLeaksPreNum++] = ptr; } /* { @@ -206,6 +255,9 @@ // start monitoring a section of code void memleakfinder_startsectionmonitor() { + InternalAllocGuard guard; + + ASSERT(memleakfinder_initialised); sTrackMallocInSection = true; sSectionMallocBlocks.clear(); sTrackObjectsInSection = true; @@ -215,6 +267,10 @@ // trace all blocks allocated and still allocated since memleakfinder_startsectionmonitor() called void memleakfinder_traceblocksinsection() { + InternalAllocGuard guard; + + ASSERT(memleakfinder_initialised); + std::set::iterator s(sSectionMallocBlocks.begin()); for(; s != sSectionMallocBlocks.end(); ++s) { @@ -225,17 +281,21 @@ } else { - TRACE4("Block 0x%08p size %d allocated at %s:%d\n", i->first, i->second.size, i->second.file, i->second.line); + TRACE4("Block %p size %d allocated at %s:%d\n", i->first, i->second.size, i->second.file, i->second.line); } } for(std::map::const_iterator i(sSectionObjectBlocks.begin()); i != sSectionObjectBlocks.end(); ++i) { - TRACE5("Object%s 0x%08p size %d allocated at %s:%d\n", i->second.array?" []":"", i->first, i->second.size, i->second.file, i->second.line); + TRACE5("Object%s %p size %d allocated at %s:%d\n", i->second.array?" []":"", i->first, i->second.size, i->second.file, i->second.line); } } int memleakfinder_numleaks() { + InternalAllocGuard guard; + + ASSERT(memleakfinder_initialised); + int n = 0; for(std::map::const_iterator i(sMallocBlocks.begin()); i != sMallocBlocks.end(); ++i) @@ -245,6 +305,7 @@ for(std::map::const_iterator i(sObjectBlocks.begin()); i != sObjectBlocks.end(); ++i) { + const ObjectInfo& rInfo = i->second; if(is_leak(i->first)) ++n; } @@ -253,24 +314,30 @@ void memleakfinder_reportleaks_file(FILE *file) { + InternalAllocGuard guard; + for(std::map::const_iterator i(sMallocBlocks.begin()); i != sMallocBlocks.end(); ++i) { - if(is_leak(i->first)) ::fprintf(file, "Block 0x%08p size %d allocated at %s:%d\n", i->first, i->second.size, i->second.file, i->second.line); + if(is_leak(i->first)) ::fprintf(file, "Block 0x%p size %d allocated at %s:%d\n", i->first, i->second.size, i->second.file, i->second.line); } for(std::map::const_iterator i(sObjectBlocks.begin()); i != sObjectBlocks.end(); ++i) { - if(is_leak(i->first)) ::fprintf(file, "Object%s 0x%08p size %d allocated at %s:%d\n", i->second.array?" []":"", i->first, i->second.size, i->second.file, i->second.line); + if(is_leak(i->first)) ::fprintf(file, "Object%s 0x%p size %d allocated at %s:%d\n", i->second.array?" []":"", i->first, i->second.size, i->second.file, i->second.line); } } void memleakfinder_reportleaks() { + InternalAllocGuard guard; + // report to stdout memleakfinder_reportleaks_file(stdout); } void memleakfinder_reportleaks_appendfile(const char *filename, const char *markertext) { + InternalAllocGuard guard; + FILE *file = ::fopen(filename, "a"); if(file != 0) { @@ -317,7 +384,10 @@ void add_object_block(void *block, size_t size, const char *file, int line, bool array) { + InternalAllocGuard guard; + if(!memleakfinder_global_enable) return; + if(!memleakfinder_initialised) return; if(block != 0) { @@ -337,7 +407,10 @@ void remove_object_block(void *block) { + InternalAllocGuard guard; + if(!memleakfinder_global_enable) return; + if(!memleakfinder_initialised) return; std::map::iterator i(sObjectBlocks.find(block)); if(i != sObjectBlocks.end()) @@ -357,34 +430,68 @@ // If it's not in the list, just ignore it, as lots of stuff goes this way... } -void *operator new(size_t size, const char *file, int line) +static void *internal_new(size_t size, const char *file, int line) { - void *r = ::malloc(size); - add_object_block(r, size, file, line, false); - //TRACE4("new(), %d, %s, %d, %08x\n", size, file, line, r); + void *r; + + { + InternalAllocGuard guard; + r = ::malloc(size); + } + + if (sInternalAllocDepth == 0) + { + InternalAllocGuard guard; + add_object_block(r, size, file, line, false); + //TRACE4("new(), %d, %s, %d, %08x\n", size, file, line, r); + } + return r; } +void *operator new(size_t size, const char *file, int line) +{ + return internal_new(size, file, line); +} + void *operator new[](size_t size, const char *file, int line) { - void *r = ::malloc(size); - add_object_block(r, size, file, line, true); - //TRACE4("new[](), %d, %s, %d, %08x\n", size, file, line, r); - return r; + return internal_new(size, file, line); } -void operator delete[](void *ptr) throw () +// where there is no doctor... need to override standard new() too +// http://www.relisoft.com/book/tech/9new.html +// disabled because it causes hangs on FC2 in futex() in test/common +// while reading files. reason unknown. +/* +void *operator new(size_t size) { + return internal_new(size, "standard libraries", 0); +} +*/ + +void *operator new[](size_t size) +{ + return internal_new(size, "standard libraries", 0); +} + +void internal_delete(void *ptr) +{ + InternalAllocGuard guard; + ::free(ptr); remove_object_block(ptr); //TRACE1("delete[]() called, %08x\n", ptr); } +void operator delete[](void *ptr) throw () +{ + internal_delete(ptr); +} + void operator delete(void *ptr) throw () { - ::free(ptr); - remove_object_block(ptr); - //TRACE1("delete() called, %08x\n", ptr); + internal_delete(ptr); } #endif // NDEBUG Modified: box/chris/general/lib/common/MemLeakFinder.h =================================================================== --- box/chris/general/lib/common/MemLeakFinder.h 2007-03-03 21:36:16 UTC (rev 1313) +++ box/chris/general/lib/common/MemLeakFinder.h 2007-03-03 21:40:15 UTC (rev 1314) @@ -20,6 +20,13 @@ // global enable flag extern bool memleakfinder_global_enable; +class MemLeakSuppressionGuard +{ + public: + MemLeakSuppressionGuard(); + ~MemLeakSuppressionGuard(); +}; + extern "C" { void *memleakfinder_malloc(size_t size, const char *file, int line); @@ -27,6 +34,8 @@ void memleakfinder_free(void *ptr); } +void memleakfinder_init(); + int memleakfinder_numleaks(); void memleakfinder_reportleaks(); @@ -41,12 +50,9 @@ void memleakfinder_notaleak(void *ptr); -void *operator new(size_t size, const char *file, int line); +void *operator new (size_t size, const char *file, int line); void *operator new[](size_t size, const char *file, int line); -void operator delete(void *ptr) throw (); -void operator delete[](void *ptr) throw (); - // define the malloc functions now, if required #ifdef MEMLEAKFINDER_FULL_MALLOC_MONITORING #define malloc(X) memleakfinder_malloc(X, __FILE__, __LINE__) @@ -55,6 +61,5 @@ #define MEMLEAKFINDER_MALLOC_MONITORING_DEFINED #endif - #endif // MEMLEAKFINDER__H From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:41:00 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:41:00 +0000 Subject: [Box Backup-commit] COMMIT r1315 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 21:41:00 +0000 (Sat, 03 Mar 2007) New Revision: 1315 Modified: box/chris/general/lib/common/Utils.cpp Log: Free stack trace strings even in DEBUG mode to avoid memory leaks. (from chris/merge) Modified: box/chris/general/lib/common/Utils.cpp =================================================================== --- box/chris/general/lib/common/Utils.cpp 2007-03-03 21:40:15 UTC (rev 1314) +++ box/chris/general/lib/common/Utils.cpp 2007-03-03 21:41:00 UTC (rev 1315) @@ -74,11 +74,13 @@ printf ("Obtained %zd stack frames.\n", size); for(i = 0; i < size; i++) + { printf("%s\n", strings[i]); + } -#ifndef MEMLEAKFINDER_MALLOC_MONITORING_DEFINED +#include "MemLeakFindOff.h" free (strings); -#endif +#include "MemLeakFindOn.h" } #endif From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:41:32 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:41:32 +0000 Subject: [Box Backup-commit] COMMIT r1316 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 21:41:31 +0000 (Sat, 03 Mar 2007) New Revision: 1316 Modified: box/chris/general/lib/common/BoxTime.h Log: Added a function to get BoxTime in microseconds (from chris/merge) Modified: box/chris/general/lib/common/BoxTime.h =================================================================== --- box/chris/general/lib/common/BoxTime.h 2007-03-03 21:41:00 UTC (rev 1315) +++ box/chris/general/lib/common/BoxTime.h 2007-03-03 21:41:31 UTC (rev 1316) @@ -35,6 +35,9 @@ { return Time / MILLI_SEC_IN_NANO_SEC_LL; } +inline uint64_t BoxTimeToMicroSeconds(box_time_t Time) +{ + return Time; +} #endif // BOXTIME__H - From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:42:39 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:42:39 +0000 Subject: [Box Backup-commit] COMMIT r1317 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 21:42:39 +0000 (Sat, 03 Mar 2007) New Revision: 1317 Modified: box/chris/general/lib/common/FileStream.cpp Log: Throw Common AccessDenied instead of Common OSFileOpenError when access to a file is denied, to improve log messages (from chris/merge) Modified: box/chris/general/lib/common/FileStream.cpp =================================================================== --- box/chris/general/lib/common/FileStream.cpp 2007-03-03 21:41:31 UTC (rev 1316) +++ box/chris/general/lib/common/FileStream.cpp 2007-03-03 21:42:39 UTC (rev 1317) @@ -11,6 +11,8 @@ #include "FileStream.h" #include "CommonException.h" +#include + #include "MemLeakFindOn.h" // -------------------------------------------------------------------------- @@ -36,7 +38,15 @@ #endif { MEMLEAKFINDER_NOT_A_LEAK(this); - THROW_EXCEPTION(CommonException, OSFileOpenError) + + if(errno == EACCES) + { + THROW_EXCEPTION(CommonException, AccessDenied) + } + else + { + THROW_EXCEPTION(CommonException, OSFileOpenError) + } } #ifdef WIN32 this->fileName = Filename; @@ -211,7 +221,7 @@ NULL ); - if ( (res == 0) || (numBytesWritten != NBytes)) + if ((res == 0) || (numBytesWritten != (DWORD)NBytes)) { // DWORD err = GetLastError(); THROW_EXCEPTION(CommonException, OSFileWriteError) From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:43:51 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:43:51 +0000 Subject: [Box Backup-commit] COMMIT r1318 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 21:43:51 +0000 (Sat, 03 Mar 2007) New Revision: 1318 Modified: box/chris/general/lib/common/BoxTime.cpp Log: Use gettimeofday() if we have it, to return system time more accurately and avoid spinning (from chris/merge) Modified: box/chris/general/lib/common/BoxTime.cpp =================================================================== --- box/chris/general/lib/common/BoxTime.cpp 2007-03-03 21:42:39 UTC (rev 1317) +++ box/chris/general/lib/common/BoxTime.cpp 2007-03-03 21:43:51 UTC (rev 1318) @@ -9,7 +9,17 @@ #include "Box.h" -#include +#ifdef HAVE_SYS_TIME_H + #include +#endif +#ifdef HAVE_TIME_H + #include +#endif +#ifdef HAVE_SYSLOG_H + #include +#endif +#include +#include #include "BoxTime.h" @@ -19,13 +29,27 @@ // // Function // Name: GetCurrentBoxTime() -// Purpose: Returns the current time as a box time. (1 sec precision) +// Purpose: Returns the current time as a box time. +// (1 sec precision, or better if supported by system) // Created: 2003/10/08 // // -------------------------------------------------------------------------- box_time_t GetCurrentBoxTime() { + #ifdef HAVE_GETTIMEOFDAY + struct timeval tv; + if (gettimeofday(&tv, NULL) != 0) + { + ::syslog(LOG_ERR, "gettimeofday() failed (%s), " + "dropping precision", strerror(errno)); + } + else + { + box_time_t timeNow = (tv.tv_sec * MICRO_SEC_IN_SEC_LL) + + tv.tv_usec; + return timeNow; + } + #endif + return SecondsToBoxTime(time(0)); } - - From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:44:43 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:44:43 +0000 Subject: [Box Backup-commit] COMMIT r1319 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 21:44:43 +0000 (Sat, 03 Mar 2007) New Revision: 1319 Modified: box/chris/general/lib/common/ExcludeList.cpp Log: Log a warning for exclude entries which end in a path separator and will therefore never match (from chris/merge) Modified: box/chris/general/lib/common/ExcludeList.cpp =================================================================== --- box/chris/general/lib/common/ExcludeList.cpp 2007-03-03 21:43:51 UTC (rev 1318) +++ box/chris/general/lib/common/ExcludeList.cpp 2007-03-03 21:44:43 UTC (rev 1319) @@ -18,6 +18,7 @@ #include "Utils.h" #include "Configuration.h" #include "Archive.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -136,6 +137,14 @@ entry = ReplaceSlashesDefinite(entry); #endif + if (entry.size() > 0 && entry[entry.size() - 1] == + DIRECTORY_SEPARATOR_ASCHAR) + { + BOX_WARNING("Exclude entry ends in path " + "separator, will never match: " + << entry); + } + mDefinite.insert(entry); } } From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:45:34 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:45:34 +0000 Subject: [Box Backup-commit] COMMIT r1320 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 21:45:33 +0000 (Sat, 03 Mar 2007) New Revision: 1320 Added: box/chris/general/lib/common/BufferedStream.cpp box/chris/general/lib/common/BufferedStream.h Log: Add BufferedStream to improve performance on local file I/O (from chris/merge) Added: box/chris/general/lib/common/BufferedStream.cpp =================================================================== --- box/chris/general/lib/common/BufferedStream.cpp (rev 0) +++ box/chris/general/lib/common/BufferedStream.cpp 2007-03-03 21:45:33 UTC (rev 1320) @@ -0,0 +1,207 @@ +// -------------------------------------------------------------------------- +// +// File +// Name: BufferedStream.cpp +// Purpose: Buffering wrapper around IOStreams +// Created: 2007/01/16 +// +// -------------------------------------------------------------------------- + +#include "Box.h" +#include "BufferedStream.h" +#include "CommonException.h" + +#include + +#include "MemLeakFindOn.h" + +// -------------------------------------------------------------------------- +// +// Function +// Name: BufferedStream::BufferedStream(const char *, int, int) +// Purpose: Constructor, set up buffer +// Created: 2007/01/16 +// +// -------------------------------------------------------------------------- +BufferedStream::BufferedStream(IOStream& rSource) +: mrSource(rSource), mBufferSize(0), mBufferPosition(0) +{ } + + +// -------------------------------------------------------------------------- +// +// Function +// Name: BufferedStream::Read(void *, int) +// Purpose: Reads bytes from the file +// Created: 2007/01/16 +// +// -------------------------------------------------------------------------- +int BufferedStream::Read(void *pBuffer, int NBytes, int Timeout) +{ + if (mBufferSize == mBufferPosition) + { + // buffer is empty, fill it. + + int numBytesRead = mrSource.Read(mBuffer, sizeof(mBuffer), + Timeout); + + if (numBytesRead < 0) + { + return numBytesRead; + } + + mBufferSize = numBytesRead; + } + + int sizeToReturn = mBufferSize - mBufferPosition; + + if (sizeToReturn > NBytes) + { + sizeToReturn = NBytes; + } + + memcpy(pBuffer, mBuffer + mBufferPosition, sizeToReturn); + mBufferPosition += sizeToReturn; + + if (mBufferPosition == mBufferSize) + { + // clear out the buffer + mBufferSize = 0; + mBufferPosition = 0; + } + + return sizeToReturn; +} + + +// -------------------------------------------------------------------------- +// +// Function +// Name: BufferedStream::BytesLeftToRead() +// Purpose: Returns number of bytes to read (may not be most efficient function ever) +// Created: 2007/01/16 +// +// -------------------------------------------------------------------------- +IOStream::pos_type BufferedStream::BytesLeftToRead() +{ + return mrSource.BytesLeftToRead() + mBufferSize - mBufferPosition; +} + + +// -------------------------------------------------------------------------- +// +// Function +// Name: BufferedStream::Write(void *, int) +// Purpose: Writes bytes to the underlying stream (not supported) +// Created: 2003/07/31 +// +// -------------------------------------------------------------------------- +void BufferedStream::Write(const void *pBuffer, int NBytes) +{ + THROW_EXCEPTION(CommonException, NotSupported); +} + + +// -------------------------------------------------------------------------- +// +// Function +// Name: BufferedStream::GetPosition() +// Purpose: Get position in stream +// Created: 2003/08/21 +// +// -------------------------------------------------------------------------- +IOStream::pos_type BufferedStream::GetPosition() const +{ + return mrSource.GetPosition() - mBufferSize + mBufferPosition; +} + + +// -------------------------------------------------------------------------- +// +// Function +// Name: BufferedStream::Seek(pos_type, int) +// Purpose: Seeks within file, as lseek, invalidate buffer +// Created: 2003/07/31 +// +// -------------------------------------------------------------------------- +void BufferedStream::Seek(IOStream::pos_type Offset, int SeekType) +{ + switch (SeekType) + { + case SeekType_Absolute: + { + // just go there + mrSource.Seek(Offset, SeekType); + } + break; + + case SeekType_Relative: + { + // Actual underlying file position is + // (mBufferSize - mBufferPosition) ahead of us. + // Need to subtract that amount from the seek + // to seek forward that much less, putting the + // real pointer in the right place. + mrSource.Seek(Offset - mBufferSize + mBufferPosition, + SeekType); + } + break; + + case SeekType_End: + { + // Actual underlying file position is + // (mBufferSize - mBufferPosition) ahead of us. + // Need to add that amount to the seek + // to seek backwards that much more, putting the + // real pointer in the right place. + mrSource.Seek(Offset + mBufferSize - mBufferPosition, + SeekType); + } + } + + // always clear the buffer for now (may be slightly wasteful) + mBufferSize = 0; + mBufferPosition = 0; +} + + +// -------------------------------------------------------------------------- +// +// Function +// Name: BufferedStream::Close() +// Purpose: Closes the underlying stream (not needed) +// Created: 2003/07/31 +// +// -------------------------------------------------------------------------- +void BufferedStream::Close() +{ + THROW_EXCEPTION(CommonException, NotSupported); +} + + +// -------------------------------------------------------------------------- +// +// Function +// Name: BufferedStream::StreamDataLeft() +// Purpose: Any data left to write? +// Created: 2003/08/02 +// +// -------------------------------------------------------------------------- +bool BufferedStream::StreamDataLeft() +{ + return mrSource.StreamDataLeft(); +} + +// -------------------------------------------------------------------------- +// +// Function +// Name: BufferedStream::StreamClosed() +// Purpose: Is the stream closed? +// Created: 2003/08/02 +// +// -------------------------------------------------------------------------- +bool BufferedStream::StreamClosed() +{ + return mrSource.StreamClosed(); +} + Added: box/chris/general/lib/common/BufferedStream.h =================================================================== --- box/chris/general/lib/common/BufferedStream.h (rev 0) +++ box/chris/general/lib/common/BufferedStream.h 2007-03-03 21:45:33 UTC (rev 1320) @@ -0,0 +1,43 @@ +// -------------------------------------------------------------------------- +// +// File +// Name: BufferedStream.h +// Purpose: Buffering wrapper around IOStreams +// Created: 2007/01/16 +// +// -------------------------------------------------------------------------- + +#ifndef BUFFEREDSTREAM__H +#define BUFFEREDSTREAM__H + +#include "IOStream.h" + +class BufferedStream : public IOStream +{ +private: + IOStream& mrSource; + char mBuffer[4096]; + int mBufferSize; + int mBufferPosition; + +public: + BufferedStream(IOStream& rSource); + + virtual int Read(void *pBuffer, int NBytes, int Timeout = IOStream::TimeOutInfinite); + virtual pos_type BytesLeftToRead(); + virtual void Write(const void *pBuffer, int NBytes); + virtual pos_type GetPosition() const; + virtual void Seek(IOStream::pos_type Offset, int SeekType); + virtual void Close(); + + virtual bool StreamDataLeft(); + virtual bool StreamClosed(); + +private: + BufferedStream(const BufferedStream &rToCopy) + : mrSource(rToCopy.mrSource) { /* do not call */ } +}; + +#endif // BUFFEREDSTREAM__H + + From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:46:01 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:46:01 +0000 Subject: [Box Backup-commit] COMMIT r1321 - box/chris/general/bin/bbstored Message-ID: Author: chris Date: 2007-03-03 21:46:01 +0000 (Sat, 03 Mar 2007) New Revision: 1321 Modified: box/chris/general/bin/bbstored/HousekeepStoreAccount.cpp Log: Use buffered streams to reduce file I/O while reading directories (from chris/merge) Modified: box/chris/general/bin/bbstored/HousekeepStoreAccount.cpp =================================================================== --- box/chris/general/bin/bbstored/HousekeepStoreAccount.cpp 2007-03-03 21:45:33 UTC (rev 1320) +++ box/chris/general/bin/bbstored/HousekeepStoreAccount.cpp 2007-03-03 21:46:01 UTC (rev 1321) @@ -23,6 +23,7 @@ #include "NamedLock.h" #include "autogen_BackupStoreException.h" #include "BackupStoreFile.h" +#include "BufferedStream.h" #include "MemLeakFindOn.h" @@ -252,7 +253,8 @@ // Read the directory in BackupStoreDirectory dir; - dir.ReadFromStream(*dirStream, IOStream::TimeOutInfinite); + BufferedStream buf(*dirStream); + dir.ReadFromStream(buf, IOStream::TimeOutInfinite); dirStream->Close(); // Is it empty? From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:47:28 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:47:28 +0000 Subject: [Box Backup-commit] COMMIT r1322 - box/chris/general/bin/bbstored Message-ID: Author: chris Date: 2007-03-03 21:47:28 +0000 (Sat, 03 Mar 2007) New Revision: 1322 Modified: box/chris/general/bin/bbstored/BackupStoreDaemon.cpp box/chris/general/bin/bbstored/BackupStoreDaemon.h Log: Avoid infinite loops if we can't bind socket, or something else goes wrong and throws an exception before the server starts accepting connections (from chris/merge) Modified: box/chris/general/bin/bbstored/BackupStoreDaemon.cpp =================================================================== --- box/chris/general/bin/bbstored/BackupStoreDaemon.cpp 2007-03-03 21:46:01 UTC (rev 1321) +++ box/chris/general/bin/bbstored/BackupStoreDaemon.cpp 2007-03-03 21:47:28 UTC (rev 1322) @@ -228,26 +228,7 @@ else { // In server process -- use the base class to do the magic - try - { - ServerTLS::Run(); - } - catch(BoxException &e) - { - ::syslog(LOG_ERR, "%s: disconnecting due to " - "exception %s (%d/%d)", DaemonName(), - e.what(), e.GetType(), e.GetSubType()); - } - catch(std::exception &e) - { - ::syslog(LOG_ERR, "%s: disconnecting due to " - "exception %s", DaemonName(), e.what()); - } - catch(...) - { - ::syslog(LOG_ERR, "%s: disconnecting due to " - "unknown exception", DaemonName()); - } + ServerTLS::Run(); if (!mInterProcessCommsSocket.IsOpened()) { @@ -267,17 +248,49 @@ } } - // -------------------------------------------------------------------------- // // Function // Name: BackupStoreDaemon::Connection(SocketStreamTLS &) -// Purpose: Handles a connection +// Purpose: Handles a connection, by catching exceptions and +// delegating to Connection2 // Created: 2003/08/20 // // -------------------------------------------------------------------------- void BackupStoreDaemon::Connection(SocketStreamTLS &rStream) { + try + { + Connection2(rStream); + } + catch(BoxException &e) + { + ::syslog(LOG_ERR, "%s: disconnecting due to " + "exception %s (%d/%d)", DaemonName(), + e.what(), e.GetType(), e.GetSubType()); + } + catch(std::exception &e) + { + ::syslog(LOG_ERR, "%s: disconnecting due to " + "exception %s", DaemonName(), e.what()); + } + catch(...) + { + ::syslog(LOG_ERR, "%s: disconnecting due to " + "unknown exception", DaemonName()); + } +} + +// -------------------------------------------------------------------------- +// +// Function +// Name: BackupStoreDaemon::Connection2(SocketStreamTLS &) +// Purpose: Handles a connection from bbackupd +// Created: 2006/11/12 +// +// -------------------------------------------------------------------------- +void BackupStoreDaemon::Connection2(SocketStreamTLS &rStream) +{ // Get the common name from the certificate std::string clientCommonName(rStream.GetPeerCommonName()); Modified: box/chris/general/bin/bbstored/BackupStoreDaemon.h =================================================================== --- box/chris/general/bin/bbstored/BackupStoreDaemon.h 2007-03-03 21:46:01 UTC (rev 1321) +++ box/chris/general/bin/bbstored/BackupStoreDaemon.h 2007-03-03 21:47:28 UTC (rev 1322) @@ -52,7 +52,8 @@ virtual void Run(); - void Connection(SocketStreamTLS &rStream); + virtual void Connection(SocketStreamTLS &rStream); + void Connection2(SocketStreamTLS &rStream); virtual const char *DaemonName() const; virtual const char *DaemonBanner() const; From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:48:18 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:48:18 +0000 Subject: [Box Backup-commit] COMMIT r1323 - box/chris/general/bin/bbackupctl Message-ID: Author: chris Date: 2007-03-03 21:48:18 +0000 (Sat, 03 Mar 2007) New Revision: 1323 Modified: box/chris/general/bin/bbackupctl/bbackupctl.cpp Log: Move openlog() inside memory leak checks (from chris/merge) Modified: box/chris/general/bin/bbackupctl/bbackupctl.cpp =================================================================== --- box/chris/general/bin/bbackupctl/bbackupctl.cpp 2007-03-03 21:47:28 UTC (rev 1322) +++ box/chris/general/bin/bbackupctl/bbackupctl.cpp 2007-03-03 21:48:18 UTC (rev 1323) @@ -56,16 +56,16 @@ { int returnCode = 0; -#if defined WIN32 && ! defined NDEBUG - ::openlog("Box Backup (bbackupctl)", 0, 0); -#endif - MAINHELPER_SETUP_MEMORY_LEAK_EXIT_REPORT("bbackupctl.memleaks", "bbackupctl") MAINHELPER_START - // Filename for configuraiton file? +#if defined WIN32 && ! defined NDEBUG + ::openlog("Box Backup (bbackupctl)", 0, 0); +#endif + + // Filename for configuration file? const char *configFilename = BOX_FILE_BBACKUPD_DEFAULT_CONFIG; // Quiet? From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:53:51 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:53:51 +0000 Subject: [Box Backup-commit] COMMIT r1324 - box/chris/general/bin/bbackupd Message-ID: Author: chris Date: 2007-03-03 21:53:50 +0000 (Sat, 03 Mar 2007) New Revision: 1324 Modified: box/chris/general/bin/bbackupd/bbackupd.cpp Log: Port to new logging framework (from chris/merge) Modified: box/chris/general/bin/bbackupd/bbackupd.cpp =================================================================== --- box/chris/general/bin/bbackupd/bbackupd.cpp 2007-03-03 21:48:18 UTC (rev 1323) +++ box/chris/general/bin/bbackupd/bbackupd.cpp 2007-03-03 21:53:50 UTC (rev 1324) @@ -12,6 +12,7 @@ #include "MainHelper.h" #include "BoxPortsAndFiles.h" #include "BackupStoreException.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -26,10 +27,12 @@ { MAINHELPER_START + Logging::SetProgramName("Box Backup (bbackupd)"); + Logging::ToConsole(true); + Logging::ToSyslog (true); + #ifdef WIN32 - ::openlog("Box Backup (bbackupd)", LOG_PID, LOG_LOCAL6); - if(argc == 2 && (::strcmp(argv[1], "--help") == 0 || ::strcmp(argv[1], "-h") == 0)) @@ -66,7 +69,7 @@ if (runAsWin32Service) { - syslog(LOG_INFO, "Box Backup service starting"); + BOX_INFO("Box Backup service starting"); char* config = NULL; if (argc >= 3) @@ -81,7 +84,7 @@ free(config); } - syslog(LOG_INFO, "Box Backup service shut down"); + BOX_INFO("Box Backup service shut down"); } else { From boxbackup-dev at fluffy.co.uk Sat Mar 3 21:55:53 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 21:55:53 +0000 Subject: [Box Backup-commit] COMMIT r1325 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 21:55:53 +0000 (Sat, 03 Mar 2007) New Revision: 1325 Added: box/chris/general/lib/common/PathUtils.cpp box/chris/general/lib/common/PathUtils.h Log: Moved MakeFullPath out of BackupClientDirectoryRecord since other modules need it too (from chris/merge) Added: box/chris/general/lib/common/PathUtils.cpp =================================================================== --- box/chris/general/lib/common/PathUtils.cpp (rev 0) +++ box/chris/general/lib/common/PathUtils.cpp 2007-03-03 21:55:53 UTC (rev 1325) @@ -0,0 +1,34 @@ +// -------------------------------------------------------------------------- +// +// File +// Name: PathUtils.cpp +// Purpose: Platform-independent path manipulation +// Created: 2007/01/17 +// +// -------------------------------------------------------------------------- + +#include "Box.h" +#include + +// -------------------------------------------------------------------------- +// +// Function +// Name: MakeFullPath(const std::string& rDir, const std::string& rFile) +// Purpose: Combine directory and file name +// Created: 2006/08/10 +// +// -------------------------------------------------------------------------- +std::string MakeFullPath(const std::string& rDir, const std::string& rEntry) +{ + std::string result(rDir); + + if (result.size() > 0 && + result[result.size()-1] != DIRECTORY_SEPARATOR_ASCHAR) + { + result += DIRECTORY_SEPARATOR; + } + + result += rEntry; + + return result; +} Added: box/chris/general/lib/common/PathUtils.h =================================================================== --- box/chris/general/lib/common/PathUtils.h (rev 0) +++ box/chris/general/lib/common/PathUtils.h 2007-03-03 21:55:53 UTC (rev 1325) @@ -0,0 +1,26 @@ +// -------------------------------------------------------------------------- +// +// File +// Name: PathUtils.h +// Purpose: Platform-independent path manipulation +// Created: 2007/01/17 +// +// -------------------------------------------------------------------------- + +#ifndef PATHUTILS_H +#define PATHUTILS_H + +#include + +// -------------------------------------------------------------------------- +// +// Function +// Name: MakeFullPath(const std::string& rDir, const std::string& rFile) +// Purpose: Combine directory and file name +// Created: 2006/08/10 +// +// -------------------------------------------------------------------------- + +std::string MakeFullPath(const std::string& rDir, const std::string& rEntry); + +#endif // !PATHUTILS_H From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:01:25 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:01:25 +0000 Subject: [Box Backup-commit] COMMIT r1326 - box/chris/general/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-03 22:01:24 +0000 (Sat, 03 Mar 2007) New Revision: 1326 Modified: box/chris/general/bin/bbackupquery/BackupQueries.cpp Log: Use MakeFullPath to make local paths instead of doing it ourselves. Disable symlink attribute compare on Darwin (doesn't work). Log excluded directories still present on the server. Don't recurse into excluded directories during compare. (from chris/merge) Modified: box/chris/general/bin/bbackupquery/BackupQueries.cpp =================================================================== --- box/chris/general/bin/bbackupquery/BackupQueries.cpp 2007-03-03 21:55:53 UTC (rev 1325) +++ box/chris/general/bin/bbackupquery/BackupQueries.cpp 2007-03-03 22:01:24 UTC (rev 1326) @@ -46,6 +46,7 @@ #include "BackupStoreException.h" #include "ExcludeList.h" #include "BackupClientMakeExcludeList.h" +#include "PathUtils.h" #include "MemLeakFindOn.h" @@ -1420,9 +1421,8 @@ } #ifndef HAVE_VALID_DIRENT_D_TYPE - std::string fn(rLocalDir); - fn += DIRECTORY_SEPARATOR_ASCHAR; - fn += localDirEn->d_name; + std::string fn(MakeFullPath + (rLocalDir, localDirEn->d_name)); struct stat st; if(::lstat(fn.c_str(), &st) != 0) { @@ -1506,10 +1506,12 @@ const std::string& fileNameDisplay(i->first); #endif - std::string localPathDisplay = localDirDisplay + - DIRECTORY_SEPARATOR + fileNameDisplay; - std::string storePathDisplay = storeDirDisplay + - "/" + fileNameDisplay; + std::string localPath(MakeFullPath + (rLocalDir, fileName)); + std::string localPathDisplay(MakeFullPath + (localDirDisplay, fileNameDisplay)); + std::string storePathDisplay + (storeDirDisplay + "/" + fileNameDisplay); // Does the file exist locally? string_set_iter_t local(localFiles.find(fileName)); @@ -1526,9 +1528,6 @@ { try { - // make local name of file for comparison - std::string localPath(rLocalDir + DIRECTORY_SEPARATOR + fileName); - // Files the same flag? bool equal = true; @@ -1556,7 +1555,7 @@ // Decode it std::auto_ptr fileOnServerStream; - // Got additional attibutes? + // Got additional attributes? if(i->second->HasAttributes()) { // Use these attributes @@ -1590,6 +1589,9 @@ #endif if(!rParams.mIgnoreAttributes && + #ifdef PLATFORM_DISABLE_SYMLINK_ATTRIB_COMPARE + !fileOnServerStream->IsSymLink() && + #endif !localAttr.Compare(fileOnServerStream->GetAttributes(), ignoreAttrModTime, fileOnServerStream->IsSymLink() /* ignore modification time if it's a symlink */)) @@ -1711,12 +1713,12 @@ const std::string& fileNameDisplay(*i); #endif - std::string localPath(rLocalDir + - DIRECTORY_SEPARATOR + *i); - std::string localPathDisplay(localDirDisplay + - DIRECTORY_SEPARATOR + fileNameDisplay); - std::string storePathDisplay(storeDirDisplay + - "/" + fileNameDisplay); + std::string localPath(MakeFullPath + (rLocalDir, *i)); + std::string localPathDisplay(MakeFullPath + (localDirDisplay, fileNameDisplay)); + std::string storePathDisplay + (storeDirDisplay + "/" + fileNameDisplay); // Should this be ignored (ie is excluded)? if(rParams.mpExcludeFiles == 0 || @@ -1752,7 +1754,7 @@ localFiles.clear(); storeFiles.clear(); - // Now do the directories, recusively to check subdirectories + // Now do the directories, recursively to check subdirectories for(std::set >::const_iterator i = storeDirs.begin(); i != storeDirs.end(); ++i) { #ifdef WIN32 @@ -1766,26 +1768,44 @@ const std::string& subdirNameDisplay(i->first); #endif - std::string localPathDisplay = localDirDisplay + - DIRECTORY_SEPARATOR + subdirNameDisplay; - std::string storePathDisplay = storeDirDisplay + - "/" + subdirNameDisplay; + std::string localPath(MakeFullPath + (rLocalDir, i->first)); + std::string localPathDisplay(MakeFullPath + (localDirDisplay, subdirNameDisplay)); + std::string storePathDisplay + (storeDirDisplay + "/" + subdirNameDisplay); // Does the directory exist locally? string_set_iter_t local(localDirs.find(i->first)); - if(local == localDirs.end()) + if(local == localDirs.end() && + rParams.mpExcludeDirs != NULL && + rParams.mpExcludeDirs->IsExcluded(localPath)) { // Not found -- report + printf("Local directory '%s' is excluded, but " + "store directory '%s' still exists.\n", + localPathDisplay.c_str(), + storePathDisplay.c_str()); + rParams.mDifferences ++; + } + else if(local == localDirs.end()) + { + // Not found -- report printf("Local directory '%s' does not exist, " "but store directory '%s' does.\n", localPathDisplay.c_str(), storePathDisplay.c_str()); rParams.mDifferences ++; } + else if(rParams.mpExcludeDirs != NULL && + rParams.mpExcludeDirs->IsExcluded(localPath)) + { + // don't recurse into excluded directories + } else { // Compare directory - Compare(i->second->GetObjectID(), rStoreDir + "/" + i->first, rLocalDir + DIRECTORY_SEPARATOR + i->first, rParams); + Compare(i->second->GetObjectID(), rStoreDir + "/" + i->first, localPath, rParams); // Remove from set so that we know it's been compared localDirs.erase(local); @@ -1805,15 +1825,15 @@ const std::string& fileNameDisplay(*i); #endif - std::string localPath = rLocalDir + - DIRECTORY_SEPARATOR + *i; - std::string storePath = rStoreDir + - "/" + *i; + std::string localPath(MakeFullPath + (rLocalDir, *i)); + std::string localPathDisplay(MakeFullPath + (localDirDisplay, fileNameDisplay)); - std::string localPathDisplay = localDirDisplay + - DIRECTORY_SEPARATOR + fileNameDisplay; - std::string storePathDisplay = storeDirDisplay + - "/" + fileNameDisplay; + std::string storePath + (rStoreDir + "/" + *i); + std::string storePathDisplay + (storeDirDisplay + "/" + fileNameDisplay); // Should this be ignored (ie is excluded)? if(rParams.mpExcludeDirs == 0 || !(rParams.mpExcludeDirs->IsExcluded(localPath))) From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:02:07 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:02:07 +0000 Subject: [Box Backup-commit] COMMIT r1327 - box/chris/general/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-03 22:02:07 +0000 (Sat, 03 Mar 2007) New Revision: 1327 Modified: box/chris/general/bin/bbackupquery/documentation.txt Log: Document that restore -d restores deleted files inside a not-deleted directory (from chris/merge) Modified: box/chris/general/bin/bbackupquery/documentation.txt =================================================================== --- box/chris/general/bin/bbackupquery/documentation.txt 2007-03-03 22:01:24 UTC (rev 1326) +++ box/chris/general/bin/bbackupquery/documentation.txt 2007-03-03 22:02:07 UTC (rev 1327) @@ -123,7 +123,7 @@ The root cannot be restored -- restore locations individually. - -d -- restore a deleted directory. + -d -- restore a deleted directory or deleted files inside -r -- resume an interrupted restoration -i -- directory name is actually an ID From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:03:45 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:03:45 +0000 Subject: [Box Backup-commit] COMMIT r1328 - box/chris/general/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-03 22:03:45 +0000 (Sat, 03 Mar 2007) New Revision: 1328 Modified: box/chris/general/bin/bbackupquery/bbackupquery.cpp Log: Disable WSACleanup() since it causes an abort() suddenly Modified: box/chris/general/bin/bbackupquery/bbackupquery.cpp =================================================================== --- box/chris/general/bin/bbackupquery/bbackupquery.cpp 2007-03-03 22:02:07 UTC (rev 1327) +++ box/chris/general/bin/bbackupquery/bbackupquery.cpp 2007-03-03 22:03:45 UTC (rev 1328) @@ -63,7 +63,10 @@ int main(int argc, const char *argv[]) { + int returnCode = 0; + MAINHELPER_SETUP_MEMORY_LEAK_EXIT_REPORT("bbackupquery.memleaks", "bbackupquery") + MAINHELPER_START #ifdef WIN32 WSADATA info; @@ -83,13 +86,9 @@ BoxDebugTraceOn = false; #endif - int returnCode = 0; - - MAINHELPER_START - FILE *logFile = 0; - // Filename for configuraiton file? + // Filename for configuration file? const char *configFilename = BOX_FILE_BBACKUPD_DEFAULT_CONFIG; // Flags @@ -314,13 +313,14 @@ // Let everything be cleaned up on exit. - MAINHELPER_END - #ifdef WIN32 // Clean up our sockets - WSACleanup(); + // FIXME we should do this, but I get an abort() when I try + // WSACleanup(); #endif + MAINHELPER_END + return returnCode; } From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:05:06 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:05:06 +0000 Subject: [Box Backup-commit] COMMIT r1329 - box/chris/general/distribution Message-ID: Author: chris Date: 2007-03-03 22:05:06 +0000 (Sat, 03 Mar 2007) New Revision: 1329 Modified: box/chris/general/distribution/COMMON-MANIFEST.txt Log: Added missing files to distribution manifest (from chris/merge) Modified: box/chris/general/distribution/COMMON-MANIFEST.txt =================================================================== --- box/chris/general/distribution/COMMON-MANIFEST.txt 2007-03-03 22:03:45 UTC (rev 1328) +++ box/chris/general/distribution/COMMON-MANIFEST.txt 2007-03-03 22:05:06 UTC (rev 1329) @@ -17,9 +17,12 @@ docs/common/lib_server notes/lib_server MKDIR infrastructure infrastructure/buildenv-testmain-template.cpp -infrastructure/makebuildenv.pl +infrastructure/makebuildenv.pl.in +infrastructure/makedistribution.pl.in +infrastructure/makeparcels.pl.in infrastructure/BoxPlatform.pm.in -infrastructure/makeparcels.pl +infrastructure/mingw +infrastructure/msvc configure.ac NO-LICENSE config.sub config.sub @@ -27,7 +30,7 @@ config.guess bootstrap parcels.txt -runtest.pl +runtest.pl.in NO-LICENSE-IN-DIR infrastructure/m4 infrastructure/m4 configure From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:06:22 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:06:22 +0000 Subject: [Box Backup-commit] COMMIT r1330 - box/chris/general/distribution/boxbackup/contrib/rpm Message-ID: Author: chris Date: 2007-03-03 22:06:22 +0000 (Sat, 03 Mar 2007) New Revision: 1330 Modified: box/chris/general/distribution/boxbackup/contrib/rpm/boxbackup.spec Log: Fix spec file so that RPMs can be built from unofficial distribution tarballs (from chris/merge) Modified: box/chris/general/distribution/boxbackup/contrib/rpm/boxbackup.spec =================================================================== --- box/chris/general/distribution/boxbackup/contrib/rpm/boxbackup.spec 2007-03-03 22:05:06 UTC (rev 1329) +++ box/chris/general/distribution/boxbackup/contrib/rpm/boxbackup.spec 2007-03-03 22:06:22 UTC (rev 1330) @@ -1,6 +1,20 @@ %define bb_user_id 171 %define ident %{name}-%{version} +# In official distribution tarballs, distribution files are copied to +# the base directory (where configure is), so distribution_dir should be empty. +# This is the default, overridden by the following block in non-distribution +# builds. +%define distribution_dir '' + +# BOX_PRIVATE_BEGIN +# In unofficial tarballs, made from svn, distribution files are still in +# distribution/boxbackup, so the following line overrides the default above: +# (this section will be removed automatically from distribution tarballs +# by infrastructure/makedistribution.pl) +%define distribution_dir distribution/boxbackup/ +# BOX_PRIVATE_END + # Detect distribution. So far we only special-case SUSE. If you need to make # any distro specific changes to get the package building on your system # please email them to boxbackup-dev at fluffy.co.uk @@ -72,6 +86,7 @@ %setup -q %build +echo -e '%{version}\n%{name}' > VERSION.txt test -e configure || ./bootstrap %configure @@ -88,19 +103,28 @@ mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/box/bbstored mkdir -p $RPM_BUILD_ROOT%{_var}/lib/box -install -m 644 BUGS.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} -install -m 644 LINUX.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} -install -m 644 VERSION.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} -install -m 644 CONTACT.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} -install -m 644 DOCUMENTATION.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} -install -m 644 ExceptionCodes.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} -install -m 644 THANKS.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} -install -m 644 LICENSE.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} -install -m 644 TODO.txt $RPM_BUILD_ROOT%{_docdir}/%{ident} +install -m 644 BUGS.txt \ + $RPM_BUILD_ROOT%{_docdir}/%{ident} +install -m 644 VERSION.txt \ + $RPM_BUILD_ROOT%{_docdir}/%{ident} +install -m 644 ExceptionCodes.txt \ + $RPM_BUILD_ROOT%{_docdir}/%{ident} +install -m 644 LICENSE.txt \ + $RPM_BUILD_ROOT%{_docdir}/%{ident} +install -m 644 %{distribution_dir}CONTACT.txt \ + $RPM_BUILD_ROOT%{_docdir}/%{ident} +install -m 644 %{distribution_dir}DOCUMENTATION.txt \ + $RPM_BUILD_ROOT%{_docdir}/%{ident} +install -m 644 %{distribution_dir}LINUX.txt \ + $RPM_BUILD_ROOT%{_docdir}/%{ident} +install -m 644 %{distribution_dir}THANKS.txt \ + $RPM_BUILD_ROOT%{_docdir}/%{ident} + # Client touch $RPM_BUILD_ROOT%{_sysconfdir}/box/bbackupd.conf -install -m 755 contrib/%{dist}/bbackupd $RPM_BUILD_ROOT%{init_dir} +install -m 755 %{distribution_dir}contrib/%{dist}/bbackupd \ + $RPM_BUILD_ROOT%{init_dir} %if %{is_suse} ln -s ../../%{init_dir}/bbackupd $RPM_BUILD_ROOT%{_sbindir}/rcbbackupd %endif @@ -113,7 +137,8 @@ # Server touch $RPM_BUILD_ROOT%{_sysconfdir}/box/bbstored.conf touch $RPM_BUILD_ROOT%{_sysconfdir}/box/raidfile.conf -install -m 755 contrib/%{dist}/bbstored $RPM_BUILD_ROOT%{init_dir} +install -m 755 %{distribution_dir}contrib/%{dist}/bbstored \ + $RPM_BUILD_ROOT%{init_dir} %if %{is_suse} ln -s ../../%{init_dir}/bbstored $RPM_BUILD_ROOT%{_sbindir}/rcbbstored %endif @@ -195,6 +220,11 @@ %{_sbindir}/raidfile-config %changelog +* Sat Jan 13 2006 Chris Wilson +- Support building from an unofficial tarball (from svn) by changing + %{distribution_dir} at the top. +- Write our RPM version number into VERSION.txt and hence compile it in + * Wed Dec 28 2005 Martin Ebourne - Box now uses autoconf so use configure macro From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:07:34 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:07:34 +0000 Subject: [Box Backup-commit] COMMIT r1331 - box/chris/general/test/common Message-ID: Author: chris Date: 2007-03-03 22:07:34 +0000 (Sat, 03 Mar 2007) New Revision: 1331 Modified: box/chris/general/test/common/testcommon.cpp Log: Added logging tests. Fixed scope of objects that were interfering with memory leak test. (from chris/merge) Modified: box/chris/general/test/common/testcommon.cpp =================================================================== --- box/chris/general/test/common/testcommon.cpp 2007-03-03 22:06:22 UTC (rev 1330) +++ box/chris/general/test/common/testcommon.cpp 2007-03-03 22:07:34 UTC (rev 1331) @@ -9,7 +9,9 @@ #include "Box.h" +#include #include +#include #include "Test.h" #include "Configuration.h" @@ -27,6 +29,8 @@ #include "autogen_ConversionException.h" #include "CollectInBufferStream.h" #include "Archive.h" +#include "Timer.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -133,59 +137,117 @@ 0 }; +void safe_sleep(int seconds) +{ +#ifdef WIN32 + Sleep(seconds * 1000); +#else + struct timespec ts; + ts.tv_sec = seconds; + ts.tv_nsec = 0; + while (nanosleep(&ts, &ts) == -1 && errno == EINTR) + { /* sleep again */ } +#endif +} + +class TestLogger : public Logger +{ + private: + bool mTriggered; + Log::Level mTargetLevel; + + public: + TestLogger(Log::Level targetLevel) + : mTriggered(false), mTargetLevel(targetLevel) + { + Logging::Add(this); + } + ~TestLogger() + { + Logging::Remove(this); + } + + bool IsTriggered() { return mTriggered; } + void Reset() { mTriggered = false; } + + virtual bool Log(Log::Level level, const std::string& rFile, + int line, std::string& rMessage) + { + if (level == mTargetLevel) + { + mTriggered = true; + } + return true; + } + + virtual const char* GetType() { return "Test"; } + virtual void SetProgramName(const std::string& rProgramName) { } +}; + int test(int argc, const char *argv[]) { // Test self-deleting temporary file streams - std::string tempfile("testfiles/tempfile"); - TEST_CHECK_THROWS(InvisibleTempFileStream fs(tempfile.c_str()), - CommonException, OSFileOpenError); - InvisibleTempFileStream fs(tempfile.c_str(), O_CREAT); + { + std::string tempfile("testfiles/tempfile"); + TEST_CHECK_THROWS(InvisibleTempFileStream fs(tempfile.c_str()), + CommonException, OSFileOpenError); + InvisibleTempFileStream fs(tempfile.c_str(), O_CREAT); -#ifdef WIN32 - // file is still visible under Windows - TEST_THAT(TestFileExists(tempfile.c_str())); + #ifdef WIN32 + // file is still visible under Windows + TEST_THAT(TestFileExists(tempfile.c_str())); - // opening it again should work - InvisibleTempFileStream fs2(tempfile.c_str()); - TEST_THAT(TestFileExists(tempfile.c_str())); + // opening it again should work + InvisibleTempFileStream fs2(tempfile.c_str()); + TEST_THAT(TestFileExists(tempfile.c_str())); - // opening it to create should work - InvisibleTempFileStream fs3(tempfile.c_str(), O_CREAT); - TEST_THAT(TestFileExists(tempfile.c_str())); + // opening it to create should work + InvisibleTempFileStream fs3(tempfile.c_str(), O_CREAT); + TEST_THAT(TestFileExists(tempfile.c_str())); - // opening it to create exclusively should fail - TEST_CHECK_THROWS(InvisibleTempFileStream fs4(tempfile.c_str(), - O_CREAT | O_EXCL), CommonException, OSFileOpenError); + // opening it to create exclusively should fail + TEST_CHECK_THROWS(InvisibleTempFileStream fs4(tempfile.c_str(), + O_CREAT | O_EXCL), CommonException, OSFileOpenError); - fs2.Close(); -#else - // file is not visible under Unix - TEST_THAT(!TestFileExists(tempfile.c_str())); + fs2.Close(); + #else + // file is not visible under Unix + TEST_THAT(!TestFileExists(tempfile.c_str())); - // opening it again should fail - TEST_CHECK_THROWS(InvisibleTempFileStream fs2(tempfile.c_str()), - CommonException, OSFileOpenError); + // opening it again should fail + TEST_CHECK_THROWS(InvisibleTempFileStream fs2(tempfile.c_str()), + CommonException, OSFileOpenError); - // opening it to create should work - InvisibleTempFileStream fs3(tempfile.c_str(), O_CREAT); - TEST_THAT(!TestFileExists(tempfile.c_str())); + // opening it to create should work + InvisibleTempFileStream fs3(tempfile.c_str(), O_CREAT); + TEST_THAT(!TestFileExists(tempfile.c_str())); - // opening it to create exclusively should work - InvisibleTempFileStream fs4(tempfile.c_str(), O_CREAT | O_EXCL); - TEST_THAT(!TestFileExists(tempfile.c_str())); + // opening it to create exclusively should work + InvisibleTempFileStream fs4(tempfile.c_str(), O_CREAT | O_EXCL); + TEST_THAT(!TestFileExists(tempfile.c_str())); - fs4.Close(); -#endif + fs4.Close(); + #endif - fs.Close(); - fs3.Close(); + fs.Close(); + fs3.Close(); - // now that it's closed, it should be invisible on all platforms - TEST_THAT(!TestFileExists(tempfile.c_str())); + // now that it's closed, it should be invisible on all platforms + TEST_THAT(!TestFileExists(tempfile.c_str())); + } - // Test memory leak detection + // Test that memory leak detection doesn't crash + { + char *test = new char[1024]; + delete [] test; + MemBlockStream *s = new MemBlockStream(test,12); + delete s; + } + #ifdef BOX_MEMORY_LEAK_TESTING { + Timers::Cleanup(); + TEST_THAT(memleakfinder_numleaks() == 0); void *block = ::malloc(12); TEST_THAT(memleakfinder_numleaks() == 1); @@ -201,10 +263,74 @@ TEST_THAT(memleakfinder_numleaks() == 1); delete [] test; TEST_THAT(memleakfinder_numleaks() == 0); + + Timers::Init(); } #endif // BOX_MEMORY_LEAK_TESTING + + // test main() initialises timers for us, so uninitialise them + Timers::Cleanup(); + // Check that using timer methods without initialisation + // throws an exception + TEST_CHECK_THROWS(Timers::Add(*(Timer*)NULL), + CommonException, AssertFailed); + TEST_CHECK_THROWS(Timers::Remove(*(Timer*)NULL), + CommonException, AssertFailed); + // TEST_CHECK_THROWS(Timers::Signal(), CommonException, AssertFailed); + TEST_CHECK_THROWS(Timers::Cleanup(), CommonException, AssertFailed); + + // Check that we can initialise the timers + Timers::Init(); + + // Check that double initialisation throws an exception + TEST_CHECK_THROWS(Timers::Init(), CommonException, AssertFailed); + // Check that we can clean up the timers + Timers::Cleanup(); + + // Check that double cleanup throws an exception + TEST_CHECK_THROWS(Timers::Cleanup(), CommonException, AssertFailed); + + Timers::Init(); + + Timer t0(0); // should never expire + Timer t1(1); + Timer t2(2); + Timer t3(3); + + TEST_THAT(!t0.HasExpired()); + TEST_THAT(!t1.HasExpired()); + TEST_THAT(!t2.HasExpired()); + TEST_THAT(!t3.HasExpired()); + + safe_sleep(1); + TEST_THAT(!t0.HasExpired()); + TEST_THAT(t1.HasExpired()); + TEST_THAT(!t2.HasExpired()); + TEST_THAT(!t3.HasExpired()); + + safe_sleep(1); + TEST_THAT(!t0.HasExpired()); + TEST_THAT(t1.HasExpired()); + TEST_THAT(t2.HasExpired()); + TEST_THAT(!t3.HasExpired()); + + t1 = Timer(1); + t2 = Timer(2); + TEST_THAT(!t0.HasExpired()); + TEST_THAT(!t1.HasExpired()); + TEST_THAT(!t2.HasExpired()); + + safe_sleep(1); + TEST_THAT(!t0.HasExpired()); + TEST_THAT(t1.HasExpired()); + TEST_THAT(!t2.HasExpired()); + TEST_THAT(t3.HasExpired()); + + // Leave timers initialised for rest of test. + // Test main() will cleanup after test finishes. + static char *testfilelines[] = { "First line", @@ -577,6 +703,8 @@ // Test ExcludeList { + TestLogger logger(Log::WARNING); + ExcludeList elist; // Check assumption TEST_THAT(Configuration::MultiValueSeparator == '\x01'); @@ -634,6 +762,24 @@ #endif #undef CASE_SENSITIVE + + TEST_THAT(!logger.IsTriggered()); + elist.AddDefiniteEntries(std::string("/foo")); + TEST_THAT(!logger.IsTriggered()); + elist.AddDefiniteEntries(std::string("/foo/")); + TEST_THAT(logger.IsTriggered()); + logger.Reset(); + elist.AddDefiniteEntries(std::string("/foo" + DIRECTORY_SEPARATOR)); + TEST_THAT(logger.IsTriggered()); + logger.Reset(); + elist.AddDefiniteEntries(std::string("/foo" + DIRECTORY_SEPARATOR "bar\x01/foo")); + TEST_THAT(!logger.IsTriggered()); + elist.AddDefiniteEntries(std::string("/foo" + DIRECTORY_SEPARATOR "bar\x01/foo" + DIRECTORY_SEPARATOR)); + TEST_THAT(logger.IsTriggered()); } test_conversions(); From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:09:23 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:09:23 +0000 Subject: [Box Backup-commit] COMMIT r1332 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 22:09:23 +0000 (Sat, 03 Mar 2007) New Revision: 1332 Modified: box/chris/general/lib/common/Box.h Log: Add new MEMLEAKFINDER_INIT and MEMLEAKFINDER_NO_LEAKS convenience macros (from chris/merge) Modified: box/chris/general/lib/common/Box.h =================================================================== --- box/chris/general/lib/common/Box.h 2007-03-03 22:07:34 UTC (rev 1331) +++ box/chris/general/lib/common/Box.h 2007-03-03 22:09:23 UTC (rev 1332) @@ -95,16 +95,19 @@ // Memory leak testing #include "MemLeakFinder.h" #define MEMLEAKFINDER_NOT_A_LEAK(x) memleakfinder_notaleak(x); + #define MEMLEAKFINDER_NO_LEAKS MemLeakSuppressionGuard _guard; + #define MEMLEAKFINDER_INIT memleakfinder_init(); #define MEMLEAKFINDER_START {memleakfinder_global_enable = true;} - #define MEMLEAKFINDER_STOP {memleakfinder_global_enable = false;} + #define MEMLEAKFINDER_STOP {memleakfinder_global_enable = false;} #else #define DEBUG_NEW new #define MEMLEAKFINDER_NOT_A_LEAK(x) + #define MEMLEAKFINDER_NO_LEAKS + #define MEMLEAKFINDER_INIT #define MEMLEAKFINDER_START #define MEMLEAKFINDER_STOP #endif - #define THROW_EXCEPTION(type, subtype) \ { \ OPTIONAL_DO_BACKTRACE \ From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:10:03 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:10:03 +0000 Subject: [Box Backup-commit] COMMIT r1333 - box/chris/general/bin/bbstored Message-ID: Author: chris Date: 2007-03-03 22:10:03 +0000 (Sat, 03 Mar 2007) New Revision: 1333 Modified: box/chris/general/bin/bbstored/BackupCommands.cpp Log: Use buffered streams when verifying files to reduce I/O (from chris/merge) Modified: box/chris/general/bin/bbstored/BackupCommands.cpp =================================================================== --- box/chris/general/bin/bbstored/BackupCommands.cpp 2007-03-03 22:09:23 UTC (rev 1332) +++ box/chris/general/bin/bbstored/BackupCommands.cpp 2007-03-03 22:10:03 UTC (rev 1333) @@ -30,6 +30,7 @@ #include "RaidFileController.h" #include "FileStream.h" #include "InvisibleTempFileStream.h" +#include "BufferedStream.h" #include "MemLeakFindOn.h" @@ -388,9 +389,10 @@ // Open the object std::auto_ptr object(rContext.OpenObject(mObjectID)); + BufferedStream buf(*object); // Verify it - if(!BackupStoreFile::VerifyEncodedFileFormat(*object)) + if(!BackupStoreFile::VerifyEncodedFileFormat(buf)) { return std::auto_ptr(new BackupProtocolServerError( BackupProtocolServerError::ErrorType, BackupProtocolServerError::Err_FileDoesNotVerify)); From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:10:44 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:10:44 +0000 Subject: [Box Backup-commit] COMMIT r1334 - box/chris/general/bin/bbstored Message-ID: Author: chris Date: 2007-03-03 22:10:44 +0000 (Sat, 03 Mar 2007) New Revision: 1334 Modified: box/chris/general/bin/bbstored/BackupContext.cpp Log: Use buffered streams when reading directories to reduce I/O Modified: box/chris/general/bin/bbstored/BackupContext.cpp =================================================================== --- box/chris/general/bin/bbstored/BackupContext.cpp 2007-03-03 22:10:03 UTC (rev 1333) +++ box/chris/general/bin/bbstored/BackupContext.cpp 2007-03-03 22:10:44 UTC (rev 1334) @@ -25,6 +25,7 @@ #include "RaidFileController.h" #include "FileStream.h" #include "InvisibleTempFileStream.h" +#include "BufferedStream.h" #include "MemLeakFindOn.h" @@ -306,7 +307,8 @@ std::auto_ptr dir(new BackupStoreDirectory); // Read it from the stream, then set it's revision ID - dir->ReadFromStream(*objectFile, IOStream::TimeOutInfinite); + BufferedStream buf(*objectFile); + dir->ReadFromStream(buf, IOStream::TimeOutInfinite); dir->SetRevisionID(revID); // Make sure the size of the directory is available for writing the dir back From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:14:06 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:14:06 +0000 Subject: [Box Backup-commit] COMMIT r1335 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 22:14:05 +0000 (Sat, 03 Mar 2007) New Revision: 1335 Modified: box/chris/general/lib/common/MainHelper.h Log: Initialise memory leak finder in MAINHELPER_START (from chris/merge) Modified: box/chris/general/lib/common/MainHelper.h =================================================================== --- box/chris/general/lib/common/MainHelper.h 2007-03-03 22:10:44 UTC (rev 1334) +++ box/chris/general/lib/common/MainHelper.h 2007-03-03 22:14:05 UTC (rev 1335) @@ -17,6 +17,7 @@ #define MAINHELPER_START \ if(argc == 2 && ::strcmp(argv[1], "--version") == 0) \ { printf(BOX_VERSION "\n"); return 0; } \ + MEMLEAKFINDER_INIT \ MEMLEAKFINDER_START \ try { #define MAINHELPER_END \ From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:17:01 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:17:01 +0000 Subject: [Box Backup-commit] COMMIT r1336 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 22:17:01 +0000 (Sat, 03 Mar 2007) New Revision: 1336 Modified: box/chris/general/lib/common/Test.h Log: Moved code for converting Unix paths to Windows format into its own helper function, ConvertPaths(). RunCommand and LaunchServer call ConvertPaths. Moved code for reading PID files into its own helper function, ReadPidFile. (from chris/merge) Modified: box/chris/general/lib/common/Test.h =================================================================== --- box/chris/general/lib/common/Test.h 2007-03-03 22:14:05 UTC (rev 1335) +++ box/chris/general/lib/common/Test.h 2007-03-03 22:17:01 UTC (rev 1336) @@ -20,7 +20,8 @@ #endif #include - +#include + extern int failures; extern int first_fail_line; extern std::string first_fail_file; @@ -91,7 +92,37 @@ return -1; } +inline std::string ConvertPaths(const char *pCommandLine) +{ #ifdef WIN32 + // convert UNIX paths to native + + std::string command; + for (int i = 0; pCommandLine[i] != 0; i++) + { + if (pCommandLine[i] == '/') + { + command += '\\'; + } + else + { + command += pCommandLine[i]; + } + } + +#else // !WIN32 + std::string command = pCommandLine; +#endif + + return command; +} + +inline int RunCommand(const char *pCommandLine) +{ + return ::system(ConvertPaths(pCommandLine).c_str()); +} + +#ifdef WIN32 #include #endif @@ -116,8 +147,30 @@ #endif // WIN32 } -inline int LaunchServer(const char *CommandLine, const char *pidFile) +inline int ReadPidFile(const char *pidFile) { + if(!TestFileExists(pidFile)) + { + TEST_FAIL_WITH_MESSAGE("Server didn't save PID file " + "(perhaps one was already running?)"); + return -1; + } + + int pid = -1; + + FILE *f = fopen(pidFile, "r"); + if(f == NULL || fscanf(f, "%d", &pid) != 1) + { + TEST_FAIL_WITH_MESSAGE("Couldn't read PID file"); + return -1; + } + fclose(f); + + return pid; +} + +inline int LaunchServer(const char *pCommandLine, const char *pidFile) +{ #ifdef WIN32 PROCESS_INFORMATION procInfo; @@ -131,7 +184,8 @@ startInfo.cbReserved2 = 0; startInfo.lpReserved2 = NULL; - CHAR* tempCmd = strdup(CommandLine); + std::string cmd = ConvertPaths(pCommandLine); + CHAR* tempCmd = strdup(cmd.c_str()); DWORD result = CreateProcess ( @@ -152,7 +206,7 @@ if (result == 0) { DWORD err = GetLastError(); - printf("Launch failed: %s: error %d\n", CommandLine, (int)err); + printf("Launch failed: %s: error %d\n", pCommandLine, (int)err); return -1; } @@ -161,9 +215,9 @@ #else // !WIN32 - if(::system(CommandLine) != 0) + if(RunCommand(pCommandLine) != 0) { - printf("Server: %s\n", CommandLine); + printf("Server: %s\n", pCommandLine); TEST_FAIL_WITH_MESSAGE("Couldn't start server"); return -1; } @@ -183,7 +237,7 @@ #endif // time for it to start up - ::fprintf(stdout, "Starting server: %s\n", CommandLine); + ::fprintf(stdout, "Starting server: %s\n", pCommandLine); ::fprintf(stdout, "Waiting for server to start: "); for (int i = 0; i < 15; i++) @@ -230,14 +284,8 @@ ::fprintf(stdout, "done.\n"); } - FILE *f = fopen(pidFile, "r"); - if(f == NULL || fscanf(f, "%d", &pid) != 1) - { - printf("Server: %s (pidfile %s)\n", CommandLine, pidFile); - TEST_FAIL_WITH_MESSAGE("Couldn't read PID file"); - return -1; - } - fclose(f); + // read pid file + pid = ReadPidFile(pidFile); #ifdef WIN32 // On Win32 we can check whether the PID in the pidFile matches @@ -355,4 +403,3 @@ } #endif // TEST__H - From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:17:56 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:17:56 +0000 Subject: [Box Backup-commit] COMMIT r1337 - box/chris/general/infrastructure Message-ID: Author: chris Date: 2007-03-03 22:17:56 +0000 (Sat, 03 Mar 2007) New Revision: 1337 Modified: box/chris/general/infrastructure/buildenv-testmain-template.cpp Log: Initialise memory leak finder at start of tests. Initialise timers before tests and clean them up afterwards. (from chris/merge) Modified: box/chris/general/infrastructure/buildenv-testmain-template.cpp =================================================================== --- box/chris/general/infrastructure/buildenv-testmain-template.cpp 2007-03-03 22:17:01 UTC (rev 1336) +++ box/chris/general/infrastructure/buildenv-testmain-template.cpp 2007-03-03 22:17:56 UTC (rev 1337) @@ -2,6 +2,9 @@ // AUTOMATICALLY GENERATED FILE // do not edit // +// Note that infrastructure/buildenv-testmain-template.cpp is NOT +// auto-generated, but test/*/_main.cpp are generated from it. +// // -------------------------------------------------------------------------- @@ -23,6 +26,7 @@ #include #include #include +#include #ifdef WIN32 #include "emu.h" @@ -31,6 +35,7 @@ #endif #include "Test.h" +#include "Timer.h" #include "MemLeakFindOn.h" @@ -117,7 +122,13 @@ } try { + #ifdef BOX_MEMORY_LEAK_TESTING + memleakfinder_init(); + #endif + + Timers::Init(); int returncode = test(argc, argv); + Timers::Cleanup(); // check for memory leaks, if enabled #ifdef BOX_MEMORY_LEAK_TESTING From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:33:37 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:33:37 +0000 Subject: [Box Backup-commit] COMMIT r1338 - box/chris/general/test/bbackupd/testfiles Message-ID: Author: chris Date: 2007-03-03 22:33:37 +0000 (Sat, 03 Mar 2007) New Revision: 1338 Modified: box/chris/general/test/bbackupd/testfiles/bbackupd.conf.in Log: Reduce maximum diffing time to test timers better. Add keepalive time to be used by keepalive tests. Disable extended logging to console, log to a file instead. (from chris/merge) Modified: box/chris/general/test/bbackupd/testfiles/bbackupd.conf.in =================================================================== --- box/chris/general/test/bbackupd/testfiles/bbackupd.conf.in 2007-03-03 22:17:56 UTC (rev 1337) +++ box/chris/general/test/bbackupd/testfiles/bbackupd.conf.in 2007-03-03 22:33:37 UTC (rev 1338) @@ -17,9 +17,11 @@ FileTrackingSizeThreshold = 1024 DiffingUploadSizeThreshold = 1024 -MaximumDiffingTime = 8 +MaximumDiffingTime = 3 +KeepAliveTime = 1 -ExtendedLogging = yes +ExtendedLogging = no +ExtendedLogFile = testfiles/bbackupd.log CommandSocket = testfiles/bbackupd.sock From boxbackup-dev at fluffy.co.uk Sat Mar 3 22:34:12 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 22:34:12 +0000 Subject: [Box Backup-commit] COMMIT r1339 - box/chris/general/test/backupstorefix Message-ID: Author: chris Date: 2007-03-03 22:34:12 +0000 (Sat, 03 Mar 2007) New Revision: 1339 Modified: box/chris/general/test/backupstorefix/testbackupstorefix.cpp Log: Ignore a few memory "leaks" in static allocations in the test (from chris/merge) Modified: box/chris/general/test/backupstorefix/testbackupstorefix.cpp =================================================================== --- box/chris/general/test/backupstorefix/testbackupstorefix.cpp 2007-03-03 22:33:37 UTC (rev 1338) +++ box/chris/general/test/backupstorefix/testbackupstorefix.cpp 2007-03-03 22:34:12 UTC (rev 1339) @@ -196,9 +196,12 @@ void test_dir_fixing() { - fnames[0].SetAsClearFilename("x1"); - fnames[1].SetAsClearFilename("x2"); - fnames[2].SetAsClearFilename("x3"); + { + MEMLEAKFINDER_NO_LEAKS; + fnames[0].SetAsClearFilename("x1"); + fnames[1].SetAsClearFilename("x2"); + fnames[2].SetAsClearFilename("x3"); + } { BackupStoreDirectory dir; @@ -357,6 +360,7 @@ TEST_THAT(::sscanf(line, "%x %s %s", &id, flags, name) == 3); bool isDir = (::strcmp(flags, "-d---") == 0); //TRACE3("%x,%d,%s\n", id, isDir, name); + MEMLEAKFINDER_NO_LEAKS; nameToID[std::string(name)] = id; objectIsDir[id] = isDir; } From boxbackup-dev at fluffy.co.uk Sat Mar 3 23:56:12 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 23:56:12 +0000 Subject: [Box Backup-commit] COMMIT r1340 - box/chris/general/lib/win32 Message-ID: Author: chris Date: 2007-03-03 23:56:12 +0000 (Sat, 03 Mar 2007) New Revision: 1340 Modified: box/chris/general/lib/win32/emu.cpp Log: Fix two memory leaks and one buffer overflow in codepage conversion code. Modified: box/chris/general/lib/win32/emu.cpp =================================================================== --- box/chris/general/lib/win32/emu.cpp 2007-03-03 22:34:12 UTC (rev 1339) +++ box/chris/general/lib/win32/emu.cpp 2007-03-03 23:56:12 UTC (rev 1340) @@ -976,7 +976,7 @@ return -1; } - free(pBuffer); + delete [] pBuffer; return 0; } @@ -1619,7 +1619,7 @@ } size_t WideSize = BufferSize / 5; - WCHAR* pWideBuffer = new WCHAR [WideSize]; + WCHAR* pWideBuffer = new WCHAR [WideSize + 1]; if (!pWideBuffer) { @@ -1645,6 +1645,8 @@ pWideBuffer[numCharsRead] = 0; char* pUtf8 = ConvertFromWideString(pWideBuffer, GetConsoleCP()); + delete [] pWideBuffer; + strncpy(pBuffer, pUtf8, BufferSize); delete [] pUtf8; From boxbackup-dev at fluffy.co.uk Sat Mar 3 23:57:23 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 03 Mar 2007 23:57:23 +0000 Subject: [Box Backup-commit] COMMIT r1341 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-03 23:57:23 +0000 (Sat, 03 Mar 2007) New Revision: 1341 Modified: box/chris/general/lib/common/DebugMemLeakFinder.cpp Log: Watch out for our leak tracking data being destroyed and don't crash when subsequent objects are destroyed. Modified: box/chris/general/lib/common/DebugMemLeakFinder.cpp =================================================================== --- box/chris/general/lib/common/DebugMemLeakFinder.cpp 2007-03-03 23:56:12 UTC (rev 1340) +++ box/chris/general/lib/common/DebugMemLeakFinder.cpp 2007-03-03 23:57:23 UTC (rev 1341) @@ -49,6 +49,17 @@ { static std::map sMallocBlocks; static std::map sObjectBlocks; + static bool sTrackingDataDestroyed = false; + + static class DestructionWatchdog + { + public: + ~DestructionWatchdog() + { + sTrackingDataDestroyed = true; + } + } + sWatchdog; static bool sTrackMallocInSection = false; static std::set sSectionMallocBlocks; @@ -225,6 +236,8 @@ { InternalAllocGuard guard; + ASSERT(!sTrackingDataDestroyed); + memleakfinder_notaleak_insert_pre(); if(memleakfinder_global_enable && memleakfinder_initialised) { @@ -258,6 +271,8 @@ InternalAllocGuard guard; ASSERT(memleakfinder_initialised); + ASSERT(!sTrackingDataDestroyed); + sTrackMallocInSection = true; sSectionMallocBlocks.clear(); sTrackObjectsInSection = true; @@ -270,6 +285,7 @@ InternalAllocGuard guard; ASSERT(memleakfinder_initialised); + ASSERT(!sTrackingDataDestroyed); std::set::iterator s(sSectionMallocBlocks.begin()); for(; s != sSectionMallocBlocks.end(); ++s) @@ -295,6 +311,7 @@ InternalAllocGuard guard; ASSERT(memleakfinder_initialised); + ASSERT(!sTrackingDataDestroyed); int n = 0; @@ -316,6 +333,8 @@ { InternalAllocGuard guard; + ASSERT(!sTrackingDataDestroyed); + for(std::map::const_iterator i(sMallocBlocks.begin()); i != sMallocBlocks.end(); ++i) { if(is_leak(i->first)) ::fprintf(file, "Block 0x%p size %d allocated at %s:%d\n", i->first, i->second.size, i->second.file, i->second.line); @@ -388,6 +407,7 @@ if(!memleakfinder_global_enable) return; if(!memleakfinder_initialised) return; + ASSERT(!sTrackingDataDestroyed); if(block != 0) { @@ -411,6 +431,7 @@ if(!memleakfinder_global_enable) return; if(!memleakfinder_initialised) return; + if(sTrackingDataDestroyed) return; std::map::iterator i(sObjectBlocks.find(block)); if(i != sObjectBlocks.end()) From boxbackup-dev at fluffy.co.uk Sun Mar 4 00:18:30 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 00:18:30 +0000 Subject: [Box Backup-commit] COMMIT r1342 - in box/chris/general: bin/bbackupd lib/backupclient Message-ID: Author: chris Date: 2007-03-04 00:18:30 +0000 (Sun, 04 Mar 2007) New Revision: 1342 Modified: box/chris/general/bin/bbackupd/BackupClientContext.cpp box/chris/general/bin/bbackupd/BackupClientContext.h box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.cpp box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.h box/chris/general/bin/bbackupd/BackupDaemon.cpp box/chris/general/bin/bbackupd/BackupDaemon.h box/chris/general/lib/backupclient/BackupStoreFile.h Log: BIG patch: Use Timers to manage diffs and keepalives. Remove signal handler mess from BackupClientContext. Use ProgressNotifier to report upload status of each file. Implement ProgressNotifier in bbackupd, using logging framework. Add LogAllFileAccess option for verbose file access logging for debugging. Add support for ExtendedLogFile directive to write extended (protocol) log to a file instead of to the console/syslog. Move MakeFullPath from BackupClientDirectoryRecord.cpp to PathUtils.cpp. Send keepalives while scanning a directory, in case it's a very big or slow directory. Enable keepalives and diff time limits by default, at 60 and 600 seconds respectively. Don't delete remote directories just because we can't stat the local one. Catch failure to read attributes from a location directory and log it rather than aborting the backup run. Log errors trying to write state to the command socket (doesn't affect win32). (from chris/merge) Modified: box/chris/general/bin/bbackupd/BackupClientContext.cpp =================================================================== --- box/chris/general/bin/bbackupd/BackupClientContext.cpp 2007-03-03 23:57:23 UTC (rev 1341) +++ box/chris/general/bin/bbackupd/BackupClientContext.cpp 2007-03-04 00:18:30 UTC (rev 1342) @@ -35,13 +35,21 @@ // -------------------------------------------------------------------------- // // Function -// Name: BackupClientContext::BackupClientContext(BackupDaemon &, TLSContext &, const std::string &, int32_t, bool) +// Name: BackupClientContext::BackupClientContext(BackupDaemon &, TLSContext &, const std::string &, int32_t, bool, bool, std::string) // Purpose: Constructor // Created: 2003/10/08 // // -------------------------------------------------------------------------- -BackupClientContext::BackupClientContext(BackupDaemon &rDaemon, TLSContext &rTLSContext, const std::string &rHostname, - int32_t AccountNumber, bool ExtendedLogging) +BackupClientContext::BackupClientContext +( + BackupDaemon &rDaemon, + TLSContext &rTLSContext, + const std::string &rHostname, + int32_t AccountNumber, + bool ExtendedLogging, + bool ExtendedLogToFile, + std::string ExtendedLogFile +) : mrDaemon(rDaemon), mrTLSContext(rTLSContext), mHostname(rHostname), @@ -49,6 +57,9 @@ mpSocket(0), mpConnection(0), mExtendedLogging(ExtendedLogging), + mExtendedLogToFile(ExtendedLogToFile), + mExtendedLogFile(ExtendedLogFile), + mpExtendedLogFileHandle(NULL), mClientStoreMarker(ClientStoreMarker_NotKnown), mpDeleteList(0), mpCurrentIDMap(0), @@ -56,8 +67,8 @@ mStorageLimitExceeded(false), mpExcludeFiles(0), mpExcludeDirs(0), - mbIsManaged(false), - mTimeMgmtEpoch(0) + mKeepAliveTimer(0), + mbIsManaged(false) { } @@ -126,6 +137,24 @@ // Set logging option mpConnection->SetLogToSysLog(mExtendedLogging); + if (mExtendedLogToFile) + { + ASSERT(mpExtendedLogFileHandle == NULL); + + mpExtendedLogFileHandle = fopen( + mExtendedLogFile.c_str(), "a+"); + + if (!mpExtendedLogFileHandle) + { + ::syslog(LOG_ERR, "Failed to open extended " + "log file: %s", strerror(errno)); + } + else + { + mpConnection->SetLogToFile(mpExtendedLogFileHandle); + } + } + // Handshake mpConnection->Handshake(); @@ -256,6 +285,12 @@ delete mpDeleteList; mpDeleteList = 0; } + + if (mpExtendedLogFileHandle != NULL) + { + fclose(mpExtendedLogFileHandle); + mpExtendedLogFileHandle = NULL; + } } @@ -303,8 +338,8 @@ // -------------------------------------------------------------------------- // // Function -// Name: -// Purpose: +// Name: BackupClientContext::PerformDeletions() +// Purpose: Perform any pending file deletions. // Created: 10/11/03 // // -------------------------------------------------------------------------- @@ -461,40 +496,22 @@ return true; } - -// maximum time to spend diffing -static int sMaximumDiffTime = 600; -// maximum time of SSL inactivity (keep-alive interval) -static int sKeepAliveTime = 0; - void BackupClientContext::SetMaximumDiffingTime(int iSeconds) { - sMaximumDiffTime = iSeconds < 0 ? 0 : iSeconds; - TRACE1("Set maximum diffing time to %d seconds\n", sMaximumDiffTime); + mMaximumDiffingTime = iSeconds < 0 ? 0 : iSeconds; + TRACE1("Set maximum diffing time to %d seconds\n", mMaximumDiffingTime); } void BackupClientContext::SetKeepAliveTime(int iSeconds) { - sKeepAliveTime = iSeconds < 0 ? 0 : iSeconds; - TRACE1("Set keep-alive time to %d seconds\n", sKeepAliveTime); + mKeepAliveTime = iSeconds < 0 ? 0 : iSeconds; + TRACE1("Set keep-alive time to %d seconds\n", mKeepAliveTime); + mKeepAliveTimer = Timer(mKeepAliveTime); } // -------------------------------------------------------------------------- // // Function -// Name: static TimerSigHandler(int) -// Purpose: Signal handler -// Created: 19/3/04 -// -// -------------------------------------------------------------------------- -static void TimerSigHandler(int iUnused) -{ - BackupStoreFile::DiffTimerExpired(); -} - -// -------------------------------------------------------------------------- -// -// Function // Name: BackupClientContext::ManageDiffProcess() // Purpose: Initiates a file diff control timer // Created: 04/19/2005 @@ -502,59 +519,8 @@ // -------------------------------------------------------------------------- void BackupClientContext::ManageDiffProcess() { - if (mbIsManaged || !mpConnection) - return; - - ASSERT(mTimeMgmtEpoch == 0); - -#ifdef PLATFORM_CYGWIN - ::signal(SIGALRM, TimerSigHandler); -#elif defined WIN32 - // no support for SIGVTALRM - SetTimerHandler(TimerSigHandler); -#else - ::signal(SIGVTALRM, TimerSigHandler); -#endif // PLATFORM_CYGWIN - - struct itimerval timeout; - memset(&timeout, 0, sizeof(timeout)); - - // - // - // - if (sMaximumDiffTime <= 0 && sKeepAliveTime <= 0) - { - TRACE0("Diff control not requested - letting things run wild\n"); - return; - } - else if (sMaximumDiffTime > 0 && sKeepAliveTime > 0) - { - timeout.it_value.tv_sec = sKeepAliveTime < sMaximumDiffTime ? sKeepAliveTime : sMaximumDiffTime; - timeout.it_interval.tv_sec = sKeepAliveTime < sMaximumDiffTime ? sKeepAliveTime : sMaximumDiffTime; - } - else - { - timeout.it_value.tv_sec = sKeepAliveTime > 0 ? sKeepAliveTime : sMaximumDiffTime; - timeout.it_interval.tv_sec = sKeepAliveTime > 0 ? sKeepAliveTime : sMaximumDiffTime; - } - - // avoid race - mTimeMgmtEpoch = time(NULL); - -#ifdef PLATFORM_CYGWIN - if(::setitimer(ITIMER_REAL, &timeout, NULL) != 0) -#else - if(::setitimer(ITIMER_VIRTUAL, &timeout, NULL) != 0) -#endif // PLATFORM_CYGWIN - { - mTimeMgmtEpoch = 0; - - TRACE0("WARNING: couldn't set file diff control timeout\n"); - THROW_EXCEPTION(BackupStoreException, Internal) - } - + ASSERT(!mbIsManaged); mbIsManaged = true; - TRACE0("Initiated timer for file diff control\n"); } // -------------------------------------------------------------------------- @@ -567,33 +533,16 @@ // -------------------------------------------------------------------------- void BackupClientContext::UnManageDiffProcess() { - if (!mbIsManaged /* don't test for active connection, just do it */) - return; - - struct itimerval timeout; - memset(&timeout, 0, sizeof(timeout)); - -#ifdef PLATFORM_CYGWIN - if(::setitimer(ITIMER_REAL, &timeout, NULL) != 0) -#else - if(::setitimer(ITIMER_VIRTUAL, &timeout, NULL) != 0) -#endif // PLATFORM_CYGWIN - { - TRACE0("WARNING: couldn't clear file diff control timeout\n"); - THROW_EXCEPTION(BackupStoreException, Internal) - } - + // ASSERT(mbIsManaged); mbIsManaged = false; - mTimeMgmtEpoch = 0; - - TRACE0("Suspended timer for file diff control\n"); } // -------------------------------------------------------------------------- // // Function // Name: BackupClientContext::DoKeepAlive() -// Purpose: Does something inconsequential over the SSL link to keep it up +// Purpose: Check whether it's time to send a KeepAlive +// message over the SSL link, and if so, send it. // Created: 04/19/2005 // // -------------------------------------------------------------------------- @@ -601,33 +550,26 @@ { if (!mpConnection) { - ::syslog(LOG_ERR, "DoKeepAlive() called with no connection!"); return; } + + if (mKeepAliveTime == 0) + { + return; + } + if (!mKeepAliveTimer.HasExpired()) + { + return; + } + + TRACE0("KeepAliveTime reached, sending keep-alive message\n"); mpConnection->QueryGetIsAlive(); + + mKeepAliveTimer = Timer(mKeepAliveTime); } -// -------------------------------------------------------------------------- -// -// Function -// Name: BackupClientContext::GetTimeMgmtEpoch() -// Purpose: Returns the unix time when the diff was started, or zero -// if the diff process is unmanaged. -// Created: 04/19/2005 -// -// -------------------------------------------------------------------------- -time_t BackupClientContext::GetTimeMgmtEpoch() -{ - return mTimeMgmtEpoch; -} - int BackupClientContext::GetMaximumDiffingTime() { - return sMaximumDiffTime; + return mMaximumDiffingTime; } - -int BackupClientContext::GetKeepaliveTime() -{ - return sKeepAliveTime; -} Modified: box/chris/general/bin/bbackupd/BackupClientContext.h =================================================================== --- box/chris/general/bin/bbackupd/BackupClientContext.h 2007-03-03 23:57:23 UTC (rev 1341) +++ box/chris/general/bin/bbackupd/BackupClientContext.h 2007-03-04 00:18:30 UTC (rev 1342) @@ -14,6 +14,7 @@ #include "BackupClientDeleteList.h" #include "BackupStoreFile.h" #include "ExcludeList.h" +#include "Timer.h" class TLSContext; class BackupProtocolClient; @@ -35,8 +36,16 @@ class BackupClientContext : public DiffTimer { public: - BackupClientContext(BackupDaemon &rDaemon, TLSContext &rTLSContext, const std::string &rHostname, - int32_t AccountNumber, bool ExtendedLogging); + BackupClientContext + ( + BackupDaemon &rDaemon, + TLSContext &rTLSContext, + const std::string &rHostname, + int32_t AccountNumber, + bool ExtendedLogging, + bool ExtendedLogToFile, + std::string ExtendedLogFile + ); virtual ~BackupClientContext(); private: BackupClientContext(const BackupClientContext &); @@ -143,7 +152,7 @@ // Created: 04/19/2005 // // -------------------------------------------------------------------------- - static void SetMaximumDiffingTime(int iSeconds); + void SetMaximumDiffingTime(int iSeconds); // -------------------------------------------------------------------------- // @@ -153,7 +162,7 @@ // Created: 04/19/2005 // // -------------------------------------------------------------------------- - static void SetKeepAliveTime(int iSeconds); + void SetKeepAliveTime(int iSeconds); // -------------------------------------------------------------------------- // @@ -175,19 +184,18 @@ // -------------------------------------------------------------------------- void UnManageDiffProcess(); - // -------------------------------------------------------------------------- + // ------------------------------------------------------------------- // // Function // Name: BackupClientContext::DoKeepAlive() - // Purpose: Does something inconsequential over the SSL link to - // keep it up, implements DiffTimer interface + // Purpose: Check whether it's time to send a KeepAlive + // message over the SSL link, and if so, send it. // Created: 04/19/2005 // - // -------------------------------------------------------------------------- + // ------------------------------------------------------------------- virtual void DoKeepAlive(); - virtual time_t GetTimeMgmtEpoch(); virtual int GetMaximumDiffingTime(); - virtual int GetKeepaliveTime(); + virtual bool IsManaged() { return mbIsManaged; } private: BackupDaemon &mrDaemon; @@ -197,6 +205,9 @@ SocketStreamTLS *mpSocket; BackupProtocolClient *mpConnection; bool mExtendedLogging; + bool mExtendedLogToFile; + std::string mExtendedLogFile; + FILE* mpExtendedLogFileHandle; int64_t mClientStoreMarker; BackupClientDeleteList *mpDeleteList; const BackupClientInodeToIDMap *mpCurrentIDMap; @@ -204,11 +215,10 @@ bool mStorageLimitExceeded; ExcludeList *mpExcludeFiles; ExcludeList *mpExcludeDirs; - + Timer mKeepAliveTimer; bool mbIsManaged; - // unix time when diff was started - time_t mTimeMgmtEpoch; + int mKeepAliveTime; + int mMaximumDiffingTime; }; - #endif // BACKUPCLIENTCONTEXT__H Modified: box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.cpp =================================================================== --- box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.cpp 2007-03-03 23:57:23 UTC (rev 1341) +++ box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.cpp 2007-03-04 00:18:30 UTC (rev 1342) @@ -29,6 +29,7 @@ #include "BackupDaemon.h" #include "BackupStoreException.h" #include "Archive.h" +#include "PathUtils.h" #include "MemLeakFindOn.h" @@ -97,33 +98,6 @@ // -------------------------------------------------------------------------- // // Function -// Name: MakeFullPath(const std::string& rDir, const std::string& rFile) -// Purpose: Combine directory and file name -// Created: 2006/08/10 -// -// -------------------------------------------------------------------------- -static std::string MakeFullPath(const std::string& rDir, - const std::string& rFile) -{ - std::string result; - - if (rDir.size() > 0 && - rDir[rDir.size()-1] == DIRECTORY_SEPARATOR_ASCHAR) - { - result = rDir + rFile; - } - else - { - result = rDir + DIRECTORY_SEPARATOR + rFile; - } - - return result; -} - - -// -------------------------------------------------------------------------- -// -// Function // Name: BackupClientDirectoryRecord::SyncDirectory(BackupClientDirectoryRecord::SyncParams &, int64_t, const std::string &, bool) // Purpose: Syncronise, recusively, a local directory with the server. // Created: 2003/10/08 @@ -164,8 +138,8 @@ { // The directory has probably been deleted, so just ignore this error. // In a future scan, this deletion will be noticed, deleted from server, and this object deleted. - TRACE1("Stat failed for '%s' (directory)\n", - rLocalPath.c_str()); + rParams.GetProgressNotifier().NotifyDirStatFailed( + this, rLocalPath, strerror(errno)); return; } // Store inode number in map so directories are tracked in case they're renamed @@ -193,15 +167,48 @@ std::vector dirs; std::vector files; bool downloadDirectoryRecordBecauseOfFutureFiles = false; + + struct stat dir_st; + if(::lstat(rLocalPath.c_str(), &dir_st) != 0) + { + // Report the error (logs and + // eventual email to administrator) + rParams.GetProgressNotifier().NotifyFileStatFailed(this, + rLocalPath, strerror(errno)); + + // FIXME move to NotifyFileStatFailed() + SetErrorWhenReadingFilesystemObject(rParams, + rLocalPath.c_str()); + + // This shouldn't happen, so we'd better not continue + THROW_EXCEPTION(CommonException, OSFileError) + } + // BLOCK { // read the contents... DIR *dirHandle = 0; try { + rParams.GetProgressNotifier().NotifyScanDirectory( + this, rLocalPath); + dirHandle = ::opendir(rLocalPath.c_str()); if(dirHandle == 0) { + // Report the error (logs and + // eventual email to administrator) + if (errno == EACCES) + { + rParams.GetProgressNotifier().NotifyDirListFailed( + this, rLocalPath, "Access denied"); + } + else + { + rParams.GetProgressNotifier().NotifyDirListFailed(this, + rLocalPath, strerror(errno)); + } + // Report the error (logs and eventual email to administrator) SetErrorWhenReadingFilesystemObject(rParams, rLocalPath.c_str()); // Ignore this directory for now. @@ -223,6 +230,8 @@ std::string filename; while((en = ::readdir(dirHandle)) != 0) { + rParams.mrContext.DoKeepAlive(); + // Don't need to use LinuxWorkaround_FinishDirentStruct(en, rLocalPath.c_str()); // on Linux, as a stat is performed to get all this info @@ -252,6 +261,10 @@ { // Report the error (logs and // eventual email to administrator) + rParams.GetProgressNotifier().NotifyFileStatFailed(this, + filename, strerror(errno)); + + // FIXME move to NotifyFileStatFailed() SetErrorWhenReadingFilesystemObject( rParams, filename.c_str()); @@ -259,6 +272,14 @@ continue; } + if(st.st_dev != dir_st.st_dev) + { + rParams.GetProgressNotifier() + .NotifyMountPointSkipped(this, + filename); + continue; + } + int type = st.st_mode & S_IFMT; #endif @@ -269,6 +290,11 @@ // Exclude it? if(rParams.mrContext.ExcludeFile(filename)) { + rParams.GetProgressNotifier() + .NotifyFileExcluded( + this, + filename); + // Next item! continue; } @@ -283,6 +309,11 @@ // Exclude it? if(rParams.mrContext.ExcludeDir(filename)) { + rParams.GetProgressNotifier() + .NotifyDirExcluded( + this, + filename); + // Next item! continue; } @@ -292,13 +323,22 @@ } else { - #ifdef WIN32 - ::syslog(LOG_ERR, "Unknown file type: " - "%d (%s)", type, - filename.c_str()); - #endif - SetErrorWhenReadingFilesystemObject( - rParams, filename.c_str()); + if(rParams.mrContext.ExcludeFile(filename)) + { + rParams.GetProgressNotifier() + .NotifyFileExcluded( + this, + filename); + } + else + { + rParams.GetProgressNotifier() + .NotifyUnsupportedFileType( + this, filename); + SetErrorWhenReadingFilesystemObject( + rParams, filename.c_str()); + } + continue; } @@ -310,6 +350,11 @@ // but now we need the information. if(::lstat(filename.c_str(), &st) != 0) { + rParams.GetProgressNotifier() + .NotifyFileStatFailed(this, + filename, + strerror(errno)); + // Report the error (logs and // eventual email to administrator) SetErrorWhenReadingFilesystemObject( @@ -318,6 +363,14 @@ // Ignore this entry for now. continue; } + + if(st.st_dev != dir_st.st_dev) + { + rParams.GetProgressNotifier() + .NotifyMountPointSkipped(this, + filename); + continue; + } #endif checksum_info.mModificationTime = FileModificationTime(st); @@ -335,8 +388,8 @@ // Log that this has happened if(!rParams.mHaveLoggedWarningAboutFutureFileTimes) { - ::syslog(LOG_ERR, "Some files have modification times excessively in the future. Check clock syncronisation.\n"); - ::syslog(LOG_ERR, "Example file (only one shown) : %s\n", filename.c_str()); + rParams.GetProgressNotifier().NotifyFileModifiedInFuture( + this, filename); rParams.mHaveLoggedWarningAboutFutureFileTimes = true; } } @@ -574,6 +627,9 @@ for(std::vector::const_iterator f = rFiles.begin(); f != rFiles.end(); ++f) { + // Send keep-alive message if needed + rParams.mrContext.DoKeepAlive(); + // Filename of this file std::string filename(MakeFullPath(rLocalPath, *f)); @@ -589,7 +645,16 @@ struct stat st; if(::lstat(filename.c_str(), &st) != 0) { - THROW_EXCEPTION(CommonException, OSFileError) + rParams.GetProgressNotifier().NotifyFileStatFailed(this, + filename, strerror(errno)); + + // Report the error (logs and + // eventual email to administrator) + SetErrorWhenReadingFilesystemObject(rParams, + filename.c_str()); + + // Ignore this entry for now. + continue; } // Extract required data @@ -801,6 +866,9 @@ { // Connection errors should just be passed on to the main handler, retries // would probably just cause more problems. + rParams.GetProgressNotifier() + .NotifyFileUploadException( + this, filename, e); throw; } catch(BoxException &e) @@ -809,8 +877,9 @@ allUpdatedSuccessfully = false; // Log it. SetErrorWhenReadingFilesystemObject(rParams, filename.c_str()); - // Log error. - ::syslog(LOG_ERR, "Error code when uploading was (%d/%d), %s", e.GetType(), e.GetSubType(), e.what()); + rParams.GetProgressNotifier() + .NotifyFileUploadException( + this, filename, e); } // Update structures if the file was uploaded successfully. @@ -823,6 +892,11 @@ } } } + else + { + rParams.GetProgressNotifier().NotifyFileSkippedServerFull(this, + filename); + } } else if(en != 0 && en->GetAttributesHash() != attributesHash) { @@ -904,6 +978,9 @@ } } } + + rParams.GetProgressNotifier().NotifyFileSynchronised(this, + filename, fileSize); } // Erase contents of files to save space when recursing @@ -921,6 +998,9 @@ for(std::vector::const_iterator d = rDirs.begin(); d != rDirs.end(); ++d) { + // Send keep-alive message if needed + rParams.mrContext.DoKeepAlive(); + // Get the local filename std::string dirname(MakeFullPath(rLocalPath, *d)); @@ -1217,7 +1297,11 @@ if(diffFromID != 0) { - // Found an old version -- get the index + // Found an old version + rParams.GetProgressNotifier().NotifyFileUploadingPatch(this, + rFilename); + + // Get the index std::auto_ptr blockIndexStream(connection.ReceiveStream()); // @@ -1290,6 +1374,8 @@ throw; } + rParams.GetProgressNotifier().NotifyFileUploaded(this, rFilename, FileSize); + // Return the new object ID of this file return objID; } @@ -1309,8 +1395,11 @@ // Zero hash, so it gets synced properly next time round. ::memset(mStateChecksum, 0, sizeof(mStateChecksum)); - // Log the error - ::syslog(LOG_ERR, "Backup object failed, error when reading %s", Filename); + // Log the error - already done by caller + /* + rParams.GetProgressNotifier().NotifyFileReadFailed(this, + Filename, strerror(errno)); + */ // Mark that an error occured in the parameters object rParams.mReadErrorsOnFilesystemObjects = true; @@ -1326,8 +1415,10 @@ // Created: 8/3/04 // // -------------------------------------------------------------------------- -BackupClientDirectoryRecord::SyncParams::SyncParams(BackupDaemon &rDaemon, BackupClientContext &rContext) - : mSyncPeriodStart(0), +BackupClientDirectoryRecord::SyncParams::SyncParams(BackupDaemon &rDaemon, + ProgressNotifier &rProgressNotifier, BackupClientContext &rContext) + : mrProgressNotifier(rProgressNotifier), + mSyncPeriodStart(0), mSyncPeriodEnd(0), mMaxUploadWait(0), mMaxFileTimeInFuture(99999999999999999LL), Modified: box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.h =================================================================== --- box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.h 2007-03-03 23:57:23 UTC (rev 1341) +++ box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.h 2007-03-04 00:18:30 UTC (rev 1342) @@ -25,6 +25,78 @@ // -------------------------------------------------------------------------- // // Class +// Name: ProgressNotifier +// Purpose: Provides methods for the backup library to inform the user +// interface about its progress with the backup +// Created: 2005/11/20 +// +// -------------------------------------------------------------------------- +class BackupClientDirectoryRecord; + +class ProgressNotifier +{ + public: + virtual ~ProgressNotifier() { } + virtual void NotifyScanDirectory( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) = 0; + virtual void NotifyDirStatFailed( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + const std::string& rErrorMsg) = 0; + virtual void NotifyFileStatFailed( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + const std::string& rErrorMsg) = 0; + virtual void NotifyDirListFailed( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + const std::string& rErrorMsg) = 0; + virtual void NotifyMountPointSkipped( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) = 0; + virtual void NotifyFileExcluded( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) = 0; + virtual void NotifyDirExcluded( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) = 0; + virtual void NotifyUnsupportedFileType( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) = 0; + virtual void NotifyFileReadFailed( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + const std::string& rErrorMsg) = 0; + virtual void NotifyFileModifiedInFuture( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) = 0; + virtual void NotifyFileSkippedServerFull( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) = 0; + virtual void NotifyFileUploadException( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + const BoxException& rException) = 0; + virtual void NotifyFileUploading( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) = 0; + virtual void NotifyFileUploadingPatch( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) = 0; + virtual void NotifyFileUploaded( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + int64_t FileSize) = 0; + virtual void NotifyFileSynchronised( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + int64_t FileSize) = 0; +}; + +// -------------------------------------------------------------------------- +// +// Class // Name: BackupClientDirectoryRecord // Purpose: Implementation of record about directory for backup client // Created: 2003/10/08 @@ -59,14 +131,18 @@ class SyncParams { public: - SyncParams(BackupDaemon &rDaemon, BackupClientContext &rContext); + SyncParams( + BackupDaemon &rDaemon, + ProgressNotifier &rProgressNotifier, + BackupClientContext &rContext); ~SyncParams(); private: // No copying SyncParams(const SyncParams&); SyncParams &operator=(const SyncParams&); + ProgressNotifier &mrProgressNotifier; + public: - // Data members are public, as accessors are not justified here box_time_t mSyncPeriodStart; box_time_t mSyncPeriodEnd; @@ -81,6 +157,11 @@ // Member variables modified by syncing process box_time_t mUploadAfterThisTimeInTheFuture; bool mHaveLoggedWarningAboutFutureFileTimes; + + ProgressNotifier& GetProgressNotifier() const + { + return mrProgressNotifier; + } }; void SyncDirectory(SyncParams &rParams, int64_t ContainingDirectoryID, const std::string &rLocalPath, Modified: box/chris/general/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/chris/general/bin/bbackupd/BackupDaemon.cpp 2007-03-03 23:57:23 UTC (rev 1341) +++ box/chris/general/bin/bbackupd/BackupDaemon.cpp 2007-03-04 00:18:30 UTC (rev 1342) @@ -72,6 +72,7 @@ #include "IOStreamGetLine.h" #include "Conversion.h" #include "Archive.h" +#include "Timer.h" #include "Logging.h" #include "MemLeakFindOn.h" @@ -112,7 +113,8 @@ BackupDaemon::BackupDaemon() : mState(BackupDaemon::State_Initialising), mpCommandSocketInfo(0), - mDeleteUnusedRootDirEntriesAfter(0) + mDeleteUnusedRootDirEntriesAfter(0), + mLogAllFileAccess(false) { // Only ever one instance of a daemon SSLLib::Initialise(); @@ -355,6 +357,9 @@ HANDLE handles[2]; handles[0] = mhMessageToSendEvent; handles[1] = rSocket.GetReadableEvent(); + + BOX_TRACE("Received command '" << command + << "' over command socket"); DWORD result = WaitForMultipleObjects( sizeof(handles)/sizeof(*handles), @@ -503,21 +508,19 @@ // -------------------------------------------------------------------------- void BackupDaemon::Run() { + // initialise global timer mechanism + Timers::Init(); + #ifdef WIN32 - // init our own timer for file diff timeouts - InitTimer(); - try { Run2(); } catch(...) { - FiniTimer(); + Timers::Cleanup(); throw; } - - FiniTimer(); #else // ! WIN32 // Ignore SIGPIPE (so that if a command connection is broken, the daemon doesn't terminate) ::signal(SIGPIPE, SIG_IGN); @@ -560,6 +563,8 @@ mpCommandSocketInfo = 0; } + Timers::Cleanup(); + throw; } @@ -570,6 +575,8 @@ mpCommandSocketInfo = 0; } #endif + + Timers::Cleanup(); } // -------------------------------------------------------------------------- @@ -593,18 +600,20 @@ // Set up the keys for various things BackupClientCryptoKeys_Setup(conf.GetKeyValue("KeysFile").c_str()); + // Setup various timings + int maximumDiffingTime = 600; + int keepAliveTime = 60; + // max diffing time, keep-alive time if(conf.KeyExists("MaximumDiffingTime")) { - BackupClientContext::SetMaximumDiffingTime(conf.GetKeyValueInt("MaximumDiffingTime")); + maximumDiffingTime = conf.GetKeyValueInt("MaximumDiffingTime"); } if(conf.KeyExists("KeepAliveTime")) { - BackupClientContext::SetKeepAliveTime(conf.GetKeyValueInt("KeepAliveTime")); + keepAliveTime = conf.GetKeyValueInt("KeepAliveTime"); } - // Setup various timings - // How often to connect to the store (approximate) box_time_t updateStoreInterval = SecondsToBoxTime(conf.GetKeyValueInt("UpdateStoreInterval")); @@ -767,18 +776,44 @@ SetState(State_Connected); BOX_NOTICE("Beginning scan of local files"); - // Then create a client context object (don't just connect, as this may be unnecessary) - BackupClientContext clientContext(*this, tlsContext, conf.GetKeyValue("StoreHostname"), - conf.GetKeyValueInt("AccountNumber"), conf.GetKeyValueBool("ExtendedLogging")); + std::string extendedLogFile; + if (conf.KeyExists("ExtendedLogFile")) + { + extendedLogFile = conf.GetKeyValue( + "ExtendedLogFile"); + } + + if (conf.KeyExists("LogAllFileAccess")) + { + mLogAllFileAccess = + conf.GetKeyValueBool( + "LogAllFileAccess"); + } + + // Then create a client context object (don't + // just connect, as this may be unnecessary) + BackupClientContext clientContext + ( + *this, + tlsContext, + conf.GetKeyValue("StoreHostname"), + conf.GetKeyValueInt("AccountNumber"), + conf.GetKeyValueBool("ExtendedLogging"), + conf.KeyExists("ExtendedLogFile"), + extendedLogFile + ); // Set up the sync parameters - BackupClientDirectoryRecord::SyncParams params(*this, clientContext); + BackupClientDirectoryRecord::SyncParams params(*this, *this, clientContext); params.mSyncPeriodStart = syncPeriodStart; params.mSyncPeriodEnd = syncPeriodEndExtended; // use potentially extended end time params.mMaxUploadWait = maxUploadWait; params.mFileTrackingSizeThreshold = conf.GetKeyValueInt("FileTrackingSizeThreshold"); params.mDiffingUploadSizeThreshold = conf.GetKeyValueInt("DiffingUploadSizeThreshold"); params.mMaxFileTimeInFuture = SecondsToBoxTime(conf.GetKeyValueInt("MaxFileTimeInFuture")); + + clientContext.SetMaximumDiffingTime(maximumDiffingTime); + clientContext.SetKeepAliveTime(keepAliveTime); // Set store marker clientContext.SetClientStoreMarker(clientStoreMarker); @@ -1529,7 +1564,23 @@ // Read the exclude lists from the Configuration ploc->mpExcludeFiles = BackupClientMakeExcludeList_Files(i->second); ploc->mpExcludeDirs = BackupClientMakeExcludeList_Dirs(i->second); - + // Does this exist on the server? + // Remove from dir object early, so that if we fail + // to stat the local directory, we still don't + // consider to remote one for deletion. + BackupStoreDirectory::Iterator iter(dir); + BackupStoreFilenameClear dirname(ploc->mName); // generate the filename + BackupStoreDirectory::Entry *en = iter.FindMatchingClearName(dirname); + int64_t oid = 0; + if(en != 0) + { + oid = en->GetObjectID(); + + // Delete the entry from the directory, so we get a list of + // unused root directories at the end of this. + dir.DeleteEntry(oid); + } + // Do a fsstat on the pathname to find out which mount it's on { @@ -1607,26 +1658,26 @@ } // Does this exist on the server? - BackupStoreDirectory::Iterator iter(dir); - BackupStoreFilenameClear dirname(ploc->mName); // generate the filename - BackupStoreDirectory::Entry *en = iter.FindMatchingClearName(dirname); - int64_t oid = 0; - if(en != 0) + if(en == 0) { - oid = en->GetObjectID(); - - // Delete the entry from the directory, so we get a list of - // unused root directories at the end of this. - dir.DeleteEntry(oid); - } - else - { // Doesn't exist, so it has to be created on the server. Let's go! // First, get the directory's attributes and modification time box_time_t attrModTime = 0; BackupClientFileAttributes attr; - attr.ReadAttributes(ploc->mPath.c_str(), true /* directories have zero mod times */, - 0 /* not interested in mod time */, &attrModTime /* get the attribute modification time */); + try + { + attr.ReadAttributes(ploc->mPath.c_str(), + true /* directories have zero mod times */, + 0 /* not interested in mod time */, + &attrModTime /* get the attribute modification time */); + } + catch (BoxException &e) + { + BOX_ERROR("Failed to get attributes " + "for path '" << ploc->mPath + << "', skipping."); + continue; + } // Execute create directory command MemBlockStream attrStream(attr); @@ -1650,6 +1701,9 @@ { delete ploc; ploc = 0; + BOX_ERROR("Failed to setup location '" + << ploc->mName << "' path '" + << ploc->mPath << "'"); throw; } } @@ -1988,11 +2042,18 @@ // Something connected to the command socket, tell it about the new state try { - mpCommandSocketInfo->mpConnectedSocket->Write(message.c_str(), - message.length()); + mpCommandSocketInfo->mpConnectedSocket->Write(newState, newStateSize); } + catch(std::exception &e) + { + BOX_ERROR("Internal error while writing state " + "to command socket: " << e.what()); + CloseCommandConnection(); + } catch(...) { + BOX_ERROR("Internal error while writing state " + "to command socket: unknown error"); CloseCommandConnection(); } #endif Modified: box/chris/general/bin/bbackupd/BackupDaemon.h =================================================================== --- box/chris/general/bin/bbackupd/BackupDaemon.h 2007-03-03 23:57:23 UTC (rev 1341) +++ box/chris/general/bin/bbackupd/BackupDaemon.h 2007-03-04 00:18:30 UTC (rev 1342) @@ -16,9 +16,12 @@ #include "BoxTime.h" #include "Daemon.h" +#include "BackupClientDirectoryRecord.h" #include "Socket.h" #include "SocketListen.h" #include "SocketStream.h" +#include "Logging.h" + #ifdef WIN32 #include "WinNamedPipeStream.h" #endif @@ -39,7 +42,7 @@ // Created: 2003/10/08 // // -------------------------------------------------------------------------- -class BackupDaemon : public Daemon +class BackupDaemon : public Daemon, ProgressNotifier { public: BackupDaemon(); @@ -180,6 +183,163 @@ box_time_t mDeleteUnusedRootDirEntriesAfter; // time to delete them std::vector > mUnusedRootDirEntries; +public: + bool StopRun() { return this->Daemon::StopRun(); } + +private: + bool mLogAllFileAccess; + + /* ProgressNotifier implementation */ +public: + virtual void NotifyScanDirectory( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) + { + if (mLogAllFileAccess) + { + BOX_INFO("Scanning directory: " << rLocalPath); + } + } + virtual void NotifyDirStatFailed( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + const std::string& rErrorMsg) + { + BOX_WARNING("Failed to access directory: " << rLocalPath + << ": " << rErrorMsg); + } + virtual void NotifyFileStatFailed( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + const std::string& rErrorMsg) + { + BOX_WARNING("Failed to access file: " << rLocalPath + << ": " << rErrorMsg); + } + virtual void NotifyDirListFailed( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + const std::string& rErrorMsg) + { + BOX_WARNING("Failed to list directory: " << rLocalPath + << ": " << rErrorMsg); + } + virtual void NotifyMountPointSkipped( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) + { + BOX_WARNING("Ignored directory: " << rLocalPath << ": " + #ifdef WIN32 + "is an NTFS junction/reparse point; " + #else + "is a mount point; " + #endif + "create a new location if you want to back it up"); + } + virtual void NotifyFileExcluded( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) + { + if (mLogAllFileAccess) + { + BOX_INFO("Skipping excluded file: " << rLocalPath); + } + } + virtual void NotifyDirExcluded( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) + { + if (mLogAllFileAccess) + { + BOX_INFO("Skipping excluded directory: " << rLocalPath); + } + } + virtual void NotifyUnsupportedFileType( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) + { + BOX_WARNING("Ignoring file of unknown type: " << rLocalPath); + } + virtual void NotifyFileReadFailed( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + const std::string& rErrorMsg) + { + BOX_WARNING("Error reading file: " << rLocalPath + << ": " << rErrorMsg); + } + virtual void NotifyFileModifiedInFuture( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) + { + BOX_WARNING("Some files have modification times excessively " + "in the future. Check clock synchronisation. " + "Example file (only one shown): " << rLocalPath); + } + virtual void NotifyFileSkippedServerFull( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) + { + BOX_WARNING("Skipped file: server is full: " << rLocalPath); + } + virtual void NotifyFileUploadException( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + const BoxException& rException) + { + if (rException.GetType() == CommonException::ExceptionType && + rException.GetSubType() == CommonException::AccessDenied) + { + BOX_ERROR("Failed to upload file: " << rLocalPath + << ": Access denied"); + } + else + { + BOX_ERROR("Failed to upload file: " << rLocalPath + << ": caught exception: " << rException.what() + << " (" << rException.GetType() + << "/" << rException.GetSubType() << ")"); + } + } + virtual void NotifyFileUploading( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) + { + if (mLogAllFileAccess) + { + BOX_INFO("Uploading file: " << rLocalPath); + } + } + virtual void NotifyFileUploadingPatch( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath) + { + if (mLogAllFileAccess) + { + BOX_INFO("Uploading patch to file: " << rLocalPath); + } + } + virtual void NotifyFileUploaded( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + int64_t FileSize) + { + if (mLogAllFileAccess) + { + BOX_INFO("Uploaded file: " << rLocalPath); + } + } + virtual void NotifyFileSynchronised( + const BackupClientDirectoryRecord* pDirRecord, + const std::string& rLocalPath, + int64_t FileSize) + { + if (mLogAllFileAccess) + { + BOX_INFO("Synchronised file: " << rLocalPath); + } + } + #ifdef WIN32 public: void RunHelperThread(void); Modified: box/chris/general/lib/backupclient/BackupStoreFile.h =================================================================== --- box/chris/general/lib/backupclient/BackupStoreFile.h 2007-03-03 23:57:23 UTC (rev 1341) +++ box/chris/general/lib/backupclient/BackupStoreFile.h 2007-03-04 00:18:30 UTC (rev 1342) @@ -50,17 +50,16 @@ DiffTimer(); virtual ~DiffTimer(); public: - virtual void DoKeepAlive() = 0; - virtual time_t GetTimeMgmtEpoch() = 0; - virtual int GetMaximumDiffingTime() = 0; - virtual int GetKeepaliveTime() = 0; + virtual void DoKeepAlive() = 0; + virtual int GetMaximumDiffingTime() = 0; + virtual bool IsManaged() = 0; }; // -------------------------------------------------------------------------- // // Class // Name: BackupStoreFile -// Purpose: Class to hold together utils for maniplating files. +// Purpose: Class to hold together utils for manipulating files. // Created: 2003/08/28 // // -------------------------------------------------------------------------- From boxbackup-dev at fluffy.co.uk Sun Mar 4 00:22:51 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 00:22:51 +0000 Subject: [Box Backup-commit] COMMIT r1343 - box/chris/general/test/bbackupd Message-ID: Author: chris Date: 2007-03-04 00:22:51 +0000 (Sun, 04 Mar 2007) New Revision: 1343 Modified: box/chris/general/test/bbackupd/testbbackupd.cpp Log: Use predefined paths from Test.h to remove some #ifdefs. Merge keepalive tests, although they don't work on win32 yet due to lack of fork() and intercepts. Use predefined paths in Test.h to remove some #ifdefs. (from chris/merge) Modified: box/chris/general/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-04 00:18:30 UTC (rev 1342) +++ box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-04 00:22:51 UTC (rev 1343) @@ -9,6 +9,13 @@ #include "Box.h" +// do not include MinGW's dirent.h on Win32, +// as we override some of it in lib/win32. + +#ifndef WIN32 + #include +#endif + #include #include #include @@ -27,6 +34,10 @@ #include +#ifdef HAVE_SYSCALL + #include +#endif + #include "Test.h" #include "BackupClientFileAttributes.h" #include "CommonException.h" @@ -52,6 +63,9 @@ #include "IOStreamGetLine.h" #include "FileStream.h" #include "LocalProcessStream.h" +#include "BackupDaemon.h" +#include "Timer.h" +#include "intercept.h" #include "MemLeakFindOn.h" @@ -402,23 +416,19 @@ int test_setupaccount() { -#ifdef WIN32 - TEST_THAT_ABORTONFAIL(::system("..\\..\\bin\\bbstoreaccounts\\bbstoreaccounts -c testfiles/bbstored.conf create 01234567 0 1000B 2000B") == 0); -#else - TEST_THAT_ABORTONFAIL(::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf create 01234567 0 1000B 2000B") == 0); + TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS " -c " + "testfiles/bbstored.conf create 01234567 0 1000B 2000B") == 0); TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); -#endif return 0; } int test_run_bbstored() { -#ifdef WIN32 - bbstored_pid = LaunchServer("..\\..\\bin\\bbstored\\bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); -#else - bbstored_pid = LaunchServer("../../bin/bbstored/bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); -#endif + bbstored_pid = LaunchServer(BBSTORED " testfiles/bbstored.conf", + "testfiles/bbstored.pid"); + TEST_THAT(bbstored_pid != -1 && bbstored_pid != 0); + if(bbstored_pid > 0) { ::sleep(1); @@ -580,6 +590,10 @@ return success; } +void intercept_setup_delay(const char *filename, unsigned int delay_after, + int delay_ms, int syscall_to_delay); +bool intercept_triggered(); + int64_t SearchDir(BackupStoreDirectory& rDir, const std::string& rChildName) { @@ -593,10 +607,150 @@ return id; } +int start_internal_daemon() +{ + // ensure that no child processes end up running tests! + int own_pid = getpid(); + + BackupDaemon daemon; + const char* fake_argv[] = { "bbackupd", "testfiles/bbackupd.conf" }; + + int result = daemon.Main(BOX_FILE_BBACKUPD_DEFAULT_CONFIG, 2, + fake_argv); + + TEST_THAT(result == 0); + if (result != 0) + { + printf("Daemon exited with code %d\n", result); + } + + // ensure that no child processes end up running tests! + TEST_THAT(getpid() == own_pid); + if (getpid() != own_pid) + { + // abort! + _exit(1); + } + + TEST_THAT(TestFileExists("testfiles/bbackupd.pid")); + + printf("Waiting for daemon to start"); + int pid = -1; + + for (int i = 0; i < 30; i++) + { + printf("."); + fflush(stdout); + sleep(1); + + pid = ReadPidFile("testfiles/bbackupd.pid"); + if (pid > 0) + { + break; + } + } + + printf("\n"); + + TEST_THAT(pid > 0); + return pid; +} + +void stop_internal_daemon(int pid) +{ + TEST_THAT(KillServer(pid)); + + /* + int status; + TEST_THAT(waitpid(pid, &status, 0) == pid); + TEST_THAT(WIFEXITED(status)); + + if (WIFEXITED(status)) + { + TEST_THAT(WEXITSTATUS(status) == 0); + } + */ +} + +static struct dirent readdir_test_dirent; +static int readdir_test_counter = 0; +static int readdir_stop_time = 0; +static char stat_hook_filename[512]; + +// First test hook, during the directory scanning stage, returns empty. +// This will not match the directory on the store, so a sync will start. +// We set up the next intercept for the same directory by passing NULL. + +struct dirent *readdir_test_hook_2(DIR *dir); + +#ifdef LINUX_WEIRD_LSTAT +int lstat_test_hook(int ver, const char *file_name, struct stat *buf); +#else +int lstat_test_hook(const char *file_name, struct stat *buf); +#endif + +struct dirent *readdir_test_hook_1(DIR *dir) +{ +#ifndef PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE + intercept_setup_readdir_hook(NULL, readdir_test_hook_2); +#endif + return NULL; +} + +// Second test hook, during the directory sync stage, keeps returning +// new filenames until the timer expires, then disables the intercept. + +struct dirent *readdir_test_hook_2(DIR *dir) +{ + if (time(NULL) >= readdir_stop_time) + { +#ifndef PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE + intercept_setup_readdir_hook(NULL, NULL); + intercept_setup_lstat_hook (NULL, NULL); +#endif + // we will not be called again. + } + + // fill in the struct dirent appropriately + memset(&readdir_test_dirent, 0, sizeof(readdir_test_dirent)); + +#ifdef HAVE_STRUCT_DIRENT_D_INO + readdir_test_dirent.d_ino = ++readdir_test_counter; +#endif + + snprintf(readdir_test_dirent.d_name, + sizeof(readdir_test_dirent.d_name), + "test.%d", readdir_test_counter); + + // ensure that when bbackupd stats the file, it gets the + // right answer + snprintf(stat_hook_filename, sizeof(stat_hook_filename), + "testfiles/TestDir1/spacetest/d1/test.%d", + readdir_test_counter); +#ifndef PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE + intercept_setup_lstat_hook(stat_hook_filename, lstat_test_hook); +#endif + + return &readdir_test_dirent; +} + +#ifdef LINUX_WEIRD_LSTAT +int lstat_test_hook(int ver, const char *file_name, struct stat *buf) +#else +int lstat_test_hook(const char *file_name, struct stat *buf) +#endif +{ + // TRACE1("lstat hook triggered for %s", file_name); + memset(buf, 0, sizeof(*buf)); + buf->st_mode = S_IFREG; + return 0; +} + int test_bbackupd() { -// // First, wait for a normal period to make sure the last changes attributes are within a normal backup timeframe. -// wait_for_backup_operation(); + // First, wait for a normal period to make sure the last changes + // attributes are within a normal backup timeframe. + // wait_for_backup_operation(); // Connection gubbins TLSContext context; @@ -608,18 +762,290 @@ // unpack the files for the initial test TEST_THAT(::system("rm -rf testfiles/TestDir1") == 0); TEST_THAT(::mkdir("testfiles/TestDir1", 0) == 0); + #ifdef WIN32 TEST_THAT(::system("tar xzvf testfiles/spacetest1.tgz -C testfiles/TestDir1") == 0); #else TEST_THAT(::system("gzip -d < testfiles/spacetest1.tgz | ( cd testfiles/TestDir1 && tar xf - )") == 0); #endif + +#ifdef PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE + printf("Skipping intercept-based KeepAlive tests on this platform.\n"); +#else + { + #ifdef WIN32 + #error TODO: implement threads on Win32, or this test \ + will not finish properly + #endif -#ifdef WIN32 - int pid = LaunchServer("..\\..\\bin\\bbackupd\\bbackupd testfiles/bbackupd.conf", "testfiles/bbackupd.pid"); -#else - int pid = LaunchServer("../../bin/bbackupd/bbackupd testfiles/bbackupd.conf", "testfiles/bbackupd.pid"); -#endif + // bbackupd daemon will try to initialise timers itself + Timers::Cleanup(); + + // something to diff against (empty file doesn't work) + int fd = open("testfiles/TestDir1/spacetest/f1", O_WRONLY); + TEST_THAT(fd > 0); + char buffer[10000]; + TEST_THAT(write(fd, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd) == 0); + + int pid = start_internal_daemon(); + wait_for_backup_operation(); + stop_internal_daemon(pid); + + intercept_setup_delay("testfiles/TestDir1/spacetest/f1", + 0, 2000, SYS_read, 1); + TEST_THAT(unlink("testfiles/bbackupd.log") == 0); + + pid = start_internal_daemon(); + + fd = open("testfiles/TestDir1/spacetest/f1", O_WRONLY); + TEST_THAT(fd > 0); + // write again, to update the file's timestamp + TEST_THAT(write(fd, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd) == 0); + + wait_for_backup_operation(); + // can't test whether intercept was triggered, because + // it's in a different process. + // TEST_THAT(intercept_triggered()); + stop_internal_daemon(pid); + + // check that keepalive was written to logs, and + // diff was not aborted, i.e. upload was a diff + FileStream fs("testfiles/bbackupd.log", O_RDONLY); + IOStreamGetLine reader(fs); + bool found1 = false; + + while (!reader.IsEOF()) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + if (line == "Send GetBlockIndexByName(0x3,\"f1\")") + { + found1 = true; + break; + } + } + + TEST_THAT(found1); + if (found1) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive Success(0xe)"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receiving stream, size 124"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Send GetIsAlive()"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive IsAlive()"); + + TEST_THAT(reader.GetLine(line)); + std::string comp = "Send StoreFile(0x3,"; + TEST_THAT(line.substr(0, comp.size()) == comp); + comp = ",0xe,\"f1\")"; + TEST_THAT(line.substr(line.size() - comp.size()) + == comp); + } + + intercept_setup_delay("testfiles/TestDir1/spacetest/f1", + 0, 4000, SYS_read, 1); + pid = start_internal_daemon(); + + fd = open("testfiles/TestDir1/spacetest/f1", O_WRONLY); + TEST_THAT(fd > 0); + // write again, to update the file's timestamp + TEST_THAT(write(fd, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd) == 0); + + wait_for_backup_operation(); + // can't test whether intercept was triggered, because + // it's in a different process. + // TEST_THAT(intercept_triggered()); + stop_internal_daemon(pid); + + // check that the diff was aborted, i.e. upload was not a diff + found1 = false; + + while (!reader.IsEOF()) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + if (line == "Send GetBlockIndexByName(0x3,\"f1\")") + { + found1 = true; + break; + } + } + + TEST_THAT(found1); + if (found1) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive Success(0xf)"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receiving stream, size 124"); + + // delaying for 4 seconds in one step means that + // the diff timer and the keepalive timer will + // both expire, and the diff timer is honoured first, + // so there will be no keepalives. + + TEST_THAT(reader.GetLine(line)); + std::string comp = "Send StoreFile(0x3,"; + TEST_THAT(line.substr(0, comp.size()) == comp); + comp = ",0x0,\"f1\")"; + TEST_THAT(line.substr(line.size() - comp.size()) + == comp); + } + + intercept_setup_delay("testfiles/TestDir1/spacetest/f1", + 0, 1000, SYS_read, 3); + pid = start_internal_daemon(); + + fd = open("testfiles/TestDir1/spacetest/f1", O_WRONLY); + TEST_THAT(fd > 0); + // write again, to update the file's timestamp + TEST_THAT(write(fd, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd) == 0); + + wait_for_backup_operation(); + // can't test whether intercept was triggered, because + // it's in a different process. + // TEST_THAT(intercept_triggered()); + stop_internal_daemon(pid); + + // check that the diff was aborted, i.e. upload was not a diff + found1 = false; + + while (!reader.IsEOF()) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + if (line == "Send GetBlockIndexByName(0x3,\"f1\")") + { + found1 = true; + break; + } + } + + TEST_THAT(found1); + if (found1) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive Success(0x10)"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receiving stream, size 124"); + + // delaying for 3 seconds in steps of 1 second + // means that the keepalive timer will expire 3 times, + // and on the 3rd time the diff timer will expire too. + // The diff timer is honoured first, so there will be + // only two keepalives. + + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Send GetIsAlive()"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive IsAlive()"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Send GetIsAlive()"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive IsAlive()"); + + TEST_THAT(reader.GetLine(line)); + std::string comp = "Send StoreFile(0x3,"; + TEST_THAT(line.substr(0, comp.size()) == comp); + comp = ",0x0,\"f1\")"; + TEST_THAT(line.substr(line.size() - comp.size()) + == comp); + } + + intercept_setup_readdir_hook("testfiles/TestDir1/spacetest/d1", + readdir_test_hook_1); + + // time for at least two keepalives + readdir_stop_time = time(NULL) + 12 + 2; + + pid = start_internal_daemon(); + + std::string touchfile = + "testfiles/TestDir1/spacetest/d1/touch-me"; + + fd = open(touchfile.c_str(), O_CREAT | O_WRONLY); + TEST_THAT(fd > 0); + // write again, to update the file's timestamp + TEST_THAT(write(fd, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd) == 0); + + wait_for_backup_operation(); + // can't test whether intercept was triggered, because + // it's in a different process. + // TEST_THAT(intercept_triggered()); + stop_internal_daemon(pid); + + // check that keepalives were sent during the dir search + found1 = false; + + // skip to next login + while (!reader.IsEOF()) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + if (line == "Send ListDirectory(0x3,0xffffffff,0xc,true)") + { + found1 = true; + break; + } + } + + TEST_THAT(found1); + if (found1) + { + found1 = false; + + while (!reader.IsEOF()) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + if (line == "Send ListDirectory(0x3,0xffffffff,0xc,true)") + { + found1 = true; + break; + } + } + } + + if (found1) + { + std::string line; + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive Success(0x3)"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receiving stream, size 425"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Send GetIsAlive()"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive IsAlive()"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Send GetIsAlive()"); + TEST_THAT(reader.GetLine(line)); + TEST_THAT(line == "Receive IsAlive()"); + } + + TEST_THAT(unlink(touchfile.c_str()) == 0); + + // restore timers for rest of tests + Timers::Init(); + } +#endif // PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE + + int pid = LaunchServer(BBACKUPD " testfiles/bbackupd.conf", + "testfiles/bbackupd.pid"); + TEST_THAT(pid != -1 && pid != 0); + if(pid > 0) { ::sleep(1); @@ -627,16 +1053,14 @@ // First, check storage space handling -- wait for file to be uploaded wait_for_backup_operation(); - //TEST_THAT_ABORTONFAIL(::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf info 01234567") == 0); + // Set limit to something very small // About 28 blocks will be used at this point. bbackupd will only pause if the size used is // greater than soft limit + 1/3 of (hard - soft). Set small values for limits accordingly. -#ifdef WIN32 - TEST_THAT_ABORTONFAIL(::system("..\\..\\bin\\bbstoreaccounts\\bbstoreaccounts -c testfiles/bbstored.conf setlimit 01234567 10B 40B") == 0); -#else - TEST_THAT_ABORTONFAIL(::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf setlimit 01234567 10B 40B") == 0); + TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS " -c " + "testfiles/bbstored.conf setlimit 01234567 10B 40B") + == 0); TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); -#endif // Unpack some more files #ifdef WIN32 @@ -644,6 +1068,7 @@ #else TEST_THAT(::system("gzip -d < testfiles/spacetest2.tgz | ( cd testfiles/TestDir1 && tar xf - )") == 0); #endif + // Delete a file and a directory TEST_THAT(::unlink("testfiles/TestDir1/spacetest/d1/f3") == 0); TEST_THAT(::system("rm -rf testfiles/TestDir1/spacetest/d3/d4") == 0); @@ -655,12 +1080,10 @@ TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // Put the limit back -#ifdef WIN32 - TEST_THAT_ABORTONFAIL(::system("..\\..\\bin\\bbstoreaccounts\\bbstoreaccounts -c testfiles/bbstored.conf setlimit 01234567 1000B 2000B") == 0); -#else - TEST_THAT_ABORTONFAIL(::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf setlimit 01234567 1000B 2000B") == 0); - testRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); -#endif + TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS " -c " + "testfiles/bbstored.conf setlimit 01234567 " + "1000B 2000B") == 0); + TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); // Check that the notify script was run TEST_THAT(TestFileExists("testfiles/notifyran.store-full.1")); @@ -679,7 +1102,9 @@ // Check that the contents of the store are the same as the contents // of the disc (-a = all, -c = give result in return code) - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query1.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q -c " + "testfiles/bbackupd.conf -l testfiles/query1.log " + "\"compare -ac\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -708,7 +1133,7 @@ // // In ISO-8859-1 (Danish locale) they are three Danish // accented characters, which are supported in code page - // 850. + // 850. Depending on your locale, YYMV (your yak may vomit). std::string foreignCharsNative("\x91\x9b\x86"); std::string foreignCharsUnicode; @@ -1759,9 +2184,11 @@ terminate_bbackupd(pid); // Start it again - pid = LaunchServer("../../bin/bbackupd/bbackupd " - "testfiles/bbackupd.conf", "testfiles/bbackupd.pid"); + pid = LaunchServer(BBACKUPD " testfiles/bbackupd.conf", + "testfiles/bbackupd.pid"); + TEST_THAT(pid != -1 && pid != 0); + if(pid != -1 && pid != 0) { // Wait and compare From boxbackup-dev at fluffy.co.uk Sun Mar 4 00:23:44 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 00:23:44 +0000 Subject: [Box Backup-commit] COMMIT r1344 - box/chris/general/bin/bbackupd Message-ID: Author: chris Date: 2007-03-04 00:23:44 +0000 (Sun, 04 Mar 2007) New Revision: 1344 Modified: box/chris/general/bin/bbackupd/BackupDaemon.cpp Log: Undo bad change (use real length when writing to bbackupctl) Modified: box/chris/general/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/chris/general/bin/bbackupd/BackupDaemon.cpp 2007-03-04 00:22:51 UTC (rev 1343) +++ box/chris/general/bin/bbackupd/BackupDaemon.cpp 2007-03-04 00:23:44 UTC (rev 1344) @@ -2042,7 +2042,8 @@ // Something connected to the command socket, tell it about the new state try { - mpCommandSocketInfo->mpConnectedSocket->Write(newState, newStateSize); + mpCommandSocketInfo->mpConnectedSocket->Write(message.c_str(), + message.length()); } catch(std::exception &e) { From boxbackup-dev at fluffy.co.uk Sun Mar 4 20:39:50 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 20:39:50 +0000 Subject: [Box Backup-commit] COMMIT r1345 - box/chris/general/lib/raidfile Message-ID: Author: chris Date: 2007-03-04 20:39:50 +0000 (Sun, 04 Mar 2007) New Revision: 1345 Modified: box/chris/general/lib/raidfile/RaidFileRead.cpp Log: Code formatting (cosmetic) Modified: box/chris/general/lib/raidfile/RaidFileRead.cpp =================================================================== --- box/chris/general/lib/raidfile/RaidFileRead.cpp 2007-03-04 00:23:44 UTC (rev 1344) +++ box/chris/general/lib/raidfile/RaidFileRead.cpp 2007-03-04 20:39:50 UTC (rev 1345) @@ -9,24 +9,24 @@ #include "Box.h" -#include +#include +#include +#include #include -#include -#include + #include +#include #ifdef HAVE_SYS_UIO_H -#include + #include #endif #ifdef HAVE_SYSLOG_H -#include + #include #endif -#include - #ifdef HAVE_DIRENT_H -#include + #include #endif #include @@ -42,7 +42,7 @@ #include "MemLeakFindOn.h" #define READ_NUMBER_DISCS_REQUIRED 3 -#define READV_MAX_BLOCKS 64 +#define READV_MAX_BLOCKS 64 // We want to use POSIX fstat() for now, not the emulated one #undef fstat From boxbackup-dev at fluffy.co.uk Sun Mar 4 20:54:43 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 20:54:43 +0000 Subject: [Box Backup-commit] COMMIT r1346 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-04 20:54:42 +0000 (Sun, 04 Mar 2007) New Revision: 1346 Modified: box/chris/general/lib/common/Test.h Log: Don't use an imaginary PID to check whether the process exists on Unix Modified: box/chris/general/lib/common/Test.h =================================================================== --- box/chris/general/lib/common/Test.h 2007-03-04 20:39:50 UTC (rev 1345) +++ box/chris/general/lib/common/Test.h 2007-03-04 20:54:42 UTC (rev 1346) @@ -224,8 +224,6 @@ #endif // WIN32 - int pid = -1; - #ifdef WIN32 // on other platforms there is no other way to get // the PID, so a NULL pidFile doesn't make sense. @@ -249,12 +247,10 @@ #ifdef WIN32 if (!ServerIsAlive((int)procInfo.dwProcessId)) - #else - if (!ServerIsAlive(pid)) - #endif { break; } + #endif ::fprintf(stdout, "."); ::fflush(stdout); @@ -285,7 +281,7 @@ } // read pid file - pid = ReadPidFile(pidFile); + int pid = ReadPidFile(pidFile); #ifdef WIN32 // On Win32 we can check whether the PID in the pidFile matches From boxbackup-dev at fluffy.co.uk Sun Mar 4 20:57:33 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 20:57:33 +0000 Subject: [Box Backup-commit] COMMIT r1347 - box/chris/general/lib/win32 Message-ID: Author: chris Date: 2007-03-04 20:57:33 +0000 (Sun, 04 Mar 2007) New Revision: 1347 Modified: box/chris/general/lib/win32/emu.cpp Log: Use autoconf check for process.h instead of hardcoded platform define. Fix potential access beyond end of array in short filenames. Use platform separator instead of assuming that it's a backslash (cleaner). Modified: box/chris/general/lib/win32/emu.cpp =================================================================== --- box/chris/general/lib/win32/emu.cpp 2007-03-04 20:54:42 UTC (rev 1346) +++ box/chris/general/lib/win32/emu.cpp 2007-03-04 20:57:33 UTC (rev 1347) @@ -11,11 +11,14 @@ #include #include -#ifdef __MINGW32__ +#ifdef HAVE_UNISTD_H #include #endif -#include +#ifdef HAVE_PROCESS_H + #include +#endif + #include #include @@ -455,7 +458,7 @@ // Absolute paths on Windows are always a drive letter // followed by ':' - if (filename[1] != ':') + if (filename.length() >= 2 && filename[1] != ':') { // Must be relative. We need to get the // current directory to make it absolute. @@ -865,7 +868,7 @@ _ui64toa(fi.dwVolumeSerialNumber, s->f_mntonname + 1, 16); // pseudo unix mount point - s->f_mntonname[0] = '\\'; + s->f_mntonname[0] = DIRECTORY_SEPARATOR_ASCHAR; CloseHandle(handle); // close the handle From boxbackup-dev at fluffy.co.uk Sun Mar 4 20:58:23 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 20:58:23 +0000 Subject: [Box Backup-commit] COMMIT r1348 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-04 20:58:23 +0000 (Sun, 04 Mar 2007) New Revision: 1348 Modified: box/chris/general/lib/common/UnixUser.cpp Log: Minimise #ifdeffed out code on win32. Modified: box/chris/general/lib/common/UnixUser.cpp =================================================================== --- box/chris/general/lib/common/UnixUser.cpp 2007-03-04 20:57:33 UTC (rev 1347) +++ box/chris/general/lib/common/UnixUser.cpp 2007-03-04 20:58:23 UTC (rev 1348) @@ -75,17 +75,18 @@ // -------------------------------------------------------------------------- UnixUser::~UnixUser() { -#ifndef WIN32 if(mRevertOnDestruction) { // Revert to "real" user and group id of the process - if(::setegid(::getgid()) != 0 - || ::seteuid(::getuid()) != 0) + #ifdef WIN32 + if(0) + #else + if(::setegid(::getgid()) != 0 || ::seteuid(::getuid()) != 0) + #endif { THROW_EXCEPTION(CommonException, CouldNotRestoreProcessUser) } } -#endif } @@ -100,12 +101,14 @@ // -------------------------------------------------------------------------- void UnixUser::ChangeProcessUser(bool Temporary) { -#ifndef WIN32 if(Temporary) { // Change temporarily (change effective only) - if(::setegid(mGID) != 0 - || ::seteuid(mUID) != 0) + #ifdef WIN32 + if(0) + #else + if(::setegid(mGID) != 0 || ::seteuid(mUID) != 0) + #endif { THROW_EXCEPTION(CommonException, CouldNotChangeProcessUser) } @@ -115,14 +118,16 @@ } else { - // Change perminantely (change all UIDs and GIDs) - if(::setgid(mGID) != 0 - || ::setuid(mUID) != 0) + // Change permanently (change all UIDs and GIDs) + #ifdef WIN32 + if(0) + #else + if(::setgid(mGID) != 0 || ::setuid(mUID) != 0) + #endif { THROW_EXCEPTION(CommonException, CouldNotChangeProcessUser) } } -#endif } From boxbackup-dev at fluffy.co.uk Sun Mar 4 21:38:25 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 21:38:25 +0000 Subject: [Box Backup-commit] COMMIT r1349 - box/chris/merge/bin/bbackupctl Message-ID: Author: chris Date: 2007-03-04 21:38:25 +0000 (Sun, 04 Mar 2007) New Revision: 1349 Modified: box/chris/merge/bin/bbackupctl/bbackupctl.cpp Log: Typo fix Modified: box/chris/merge/bin/bbackupctl/bbackupctl.cpp =================================================================== --- box/chris/merge/bin/bbackupctl/bbackupctl.cpp 2007-03-04 20:58:23 UTC (rev 1348) +++ box/chris/merge/bin/bbackupctl/bbackupctl.cpp 2007-03-04 21:38:25 UTC (rev 1349) @@ -65,7 +65,7 @@ ::openlog("Box Backup (bbackupctl)", 0, 0); #endif - // Filename for configuraiton file? + // Filename for configuration file? const char *configFilename = BOX_FILE_BBACKUPD_DEFAULT_CONFIG; // Quiet? From boxbackup-dev at fluffy.co.uk Sun Mar 4 21:38:40 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 21:38:40 +0000 Subject: [Box Backup-commit] COMMIT r1350 - box/trunk/bin/bbackupctl Message-ID: Author: chris Date: 2007-03-04 21:38:40 +0000 (Sun, 04 Mar 2007) New Revision: 1350 Modified: box/trunk/bin/bbackupctl/bbackupctl.cpp Log: Typo fix Modified: box/trunk/bin/bbackupctl/bbackupctl.cpp =================================================================== --- box/trunk/bin/bbackupctl/bbackupctl.cpp 2007-03-04 21:38:25 UTC (rev 1349) +++ box/trunk/bin/bbackupctl/bbackupctl.cpp 2007-03-04 21:38:40 UTC (rev 1350) @@ -65,7 +65,7 @@ MAINHELPER_START - // Filename for configuraiton file? + // Filename for configuration file? const char *configFilename = BOX_FILE_BBACKUPD_DEFAULT_CONFIG; // Quiet? From boxbackup-dev at fluffy.co.uk Sun Mar 4 22:22:44 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 22:22:44 +0000 Subject: [Box Backup-commit] COMMIT r1351 - box/chris/general Message-ID: Author: chris Date: 2007-03-04 22:22:44 +0000 (Sun, 04 Mar 2007) New Revision: 1351 Modified: box/chris/general/configure.ac Log: Check whether struct dirent has a d_ino member (for intercepts) Modified: box/chris/general/configure.ac =================================================================== --- box/chris/general/configure.ac 2007-03-04 21:38:40 UTC (rev 1350) +++ box/chris/general/configure.ac 2007-03-04 22:22:44 UTC (rev 1351) @@ -156,6 +156,7 @@ AX_CHECK_MOUNT_POINT(,[ AC_MSG_ERROR([[cannot work out how to discover mount points on your platform]]) ]) + AC_CHECK_MEMBERS([struct dirent.d_ino],,, [[#include ]]) ;; esac From boxbackup-dev at fluffy.co.uk Sun Mar 4 22:47:59 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 22:47:59 +0000 Subject: [Box Backup-commit] COMMIT r1352 - box/chris/merge Message-ID: Author: chris Date: 2007-03-04 22:47:59 +0000 (Sun, 04 Mar 2007) New Revision: 1352 Modified: box/chris/merge/configure.ac Log: Simplify check for PERL Define substitution TARGET_PERL and preprocessor PERL_EXECUTABLE to a native Perl (not Cygwin) on Win32, since Cygwin perl will not run inside bbackupd for unit tests (e.g. SyncAllowScript). (refs #3) Modified: box/chris/merge/configure.ac =================================================================== --- box/chris/merge/configure.ac 2007-03-04 22:22:44 UTC (rev 1351) +++ box/chris/merge/configure.ac 2007-03-04 22:47:59 UTC (rev 1352) @@ -26,14 +26,22 @@ # Use -rdynamic if we have gcc. This is needed for backtrace AC_SUBST([LDADD_RDYNAMIC], ['-rdynamic']) fi -AC_PATH_PROG([PERL], [perl], [no]) -if test "x$PERL" != "xno"; then - AC_DEFINE_UNQUOTED([PERL_EXECUTABLE], ["$PERL"], [Location of the perl executable]) -else - AC_MSG_ERROR([[perl executable was not found]]) -fi +AC_PATH_PROG([PERL], [perl], [AC_MSG_ERROR([[perl executable was not found]])]) +case $target_os in +mingw*) + TARGET_PERL=perl + ;; +*) + TARGET_PERL=$PERL + ;; +esac + +AC_SUBST([TARGET_PERL]) +AC_DEFINE_UNQUOTED([PERL_EXECUTABLE], ["$TARGET_PERL"], + [Location of the perl executable]) + ### Checks for libraries. case $target_os in From boxbackup-dev at fluffy.co.uk Sun Mar 4 22:48:42 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 22:48:42 +0000 Subject: [Box Backup-commit] COMMIT r1353 - box/chris/merge/docs/backup Message-ID: Author: chris Date: 2007-03-04 22:48:42 +0000 (Sun, 04 Mar 2007) New Revision: 1353 Modified: box/chris/merge/docs/backup/win32_build_on_linux_using_mingw.txt Log: Update build instructions for Cygwin/Win32 (refs #3) Modified: box/chris/merge/docs/backup/win32_build_on_linux_using_mingw.txt =================================================================== --- box/chris/merge/docs/backup/win32_build_on_linux_using_mingw.txt 2007-03-04 22:47:59 UTC (rev 1352) +++ box/chris/merge/docs/backup/win32_build_on_linux_using_mingw.txt 2007-03-04 22:48:42 UTC (rev 1353) @@ -28,12 +28,28 @@ tar xzvf openssl-0.9.8b.tar.gz cd openssl-0.9.8b - ./Configure mingw + ./Configure --prefix=/usr/local/i386-mingw32 mingw make makefile.one wget http://bbdev.fluffy.co.uk/svn/box/chris/win32/support/openssl-0.9.8b-mingw-cross.patch patch -p1 < openssl-0.9.8b-mingw-cross.patch make -f makefile.one + make -f makefile.one install +Download PCRE from +[http://prdownloads.sourceforge.net/pcre/pcre-6.3.tar.bz2?download] + +Unpack: + + tar xjvf pcre-6.3.tar.bz2 + cd pcre-6.3 + +Configure and make: + + ./configure --host=i586-mingw32msvc --prefix=/usr/i386-mingw32/ + make winshared wininstall + cp .libs/libpcreposix.a /usr/i386-pc-mingw32/lib + cp pcreposix.h /usr/i386-pc-mingw32/include/regex.h + Configure Box with: export CXX="i386-mingw32-g++" From boxbackup-dev at fluffy.co.uk Sun Mar 4 22:50:26 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 22:50:26 +0000 Subject: [Box Backup-commit] COMMIT r1354 - box/chris/general/infrastructure Message-ID: Author: chris Date: 2007-03-04 22:50:26 +0000 (Sun, 04 Mar 2007) New Revision: 1354 Modified: box/chris/general/infrastructure/BoxPlatform.pm.in Log: Remove unused variable $build_cpu Modified: box/chris/general/infrastructure/BoxPlatform.pm.in =================================================================== --- box/chris/general/infrastructure/BoxPlatform.pm.in 2007-03-04 22:48:42 UTC (rev 1353) +++ box/chris/general/infrastructure/BoxPlatform.pm.in 2007-03-04 22:50:26 UTC (rev 1354) @@ -1,7 +1,7 @@ package BoxPlatform; use Exporter; @ISA = qw/Exporter/; - at EXPORT = qw/$build_os $build_cpu $target_os $make_command $bsd_make $platform_define $platform_cpu $gcc_v3 $product_version $product_name $install_into_dir $sub_make_options $platform_compile_line_extra $platform_link_line_extra $platform_lib_files $platform_exe_ext $target_windows/; + at EXPORT = qw/$build_os $target_os $make_command $bsd_make $platform_define $platform_cpu $gcc_v3 $product_version $product_name $install_into_dir $sub_make_options $platform_compile_line_extra $platform_link_line_extra $platform_lib_files $platform_exe_ext $target_windows/; BEGIN { @@ -15,19 +15,15 @@ if ($^O eq "MSWin32" and not -x "/usr/bin/uname") { $build_os = "winnt"; - $build_cpu = "ix86"; } else { $build_os = `uname`; chomp $build_os; - $build_cpu = `uname -m`; - chomp $build_cpu; } # Cygwin Builds usually something like CYGWIN_NT-5.0, CYGWIN_NT-5.1 # Box Backup tried on Win2000,XP only :) - $build_os = 'CYGWIN' if $build_os =~ m/CYGWIN/; $make_command = ($build_os eq 'Darwin') ? 'bsdmake' : ($build_os eq 'SunOS') ? 'gmake' : 'make'; From boxbackup-dev at fluffy.co.uk Sun Mar 4 22:57:15 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 22:57:15 +0000 Subject: [Box Backup-commit] COMMIT r1355 - box/chris/merge/infrastructure Message-ID: Author: chris Date: 2007-03-04 22:57:15 +0000 (Sun, 04 Mar 2007) New Revision: 1355 Modified: box/chris/merge/infrastructure/buildenv-testmain-template.cpp Log: #include Test.h to fix compile (needed for TEST_THAT) Improved error message Improved comments (refs #3, merges parts of [712], [1289] and [1337]) Modified: box/chris/merge/infrastructure/buildenv-testmain-template.cpp =================================================================== --- box/chris/merge/infrastructure/buildenv-testmain-template.cpp 2007-03-04 22:50:26 UTC (rev 1354) +++ box/chris/merge/infrastructure/buildenv-testmain-template.cpp 2007-03-04 22:57:15 UTC (rev 1355) @@ -2,6 +2,9 @@ // AUTOMATICALLY GENERATED FILE // do not edit // +// Note that infrastructure/buildenv-testmain-template.cpp is NOT +// auto-generated, but test/*/_main.cpp are generated from it. +// // -------------------------------------------------------------------------- @@ -31,6 +34,7 @@ #include #endif +#include "Test.h" #include "Timer.h" #include "MemLeakFindOn.h" @@ -78,7 +82,8 @@ { if(filedes_open_at_beginning == -1) { - // Not used correctly, pretend that there were things left open so this gets invesitgated + // Not used correctly, pretend that there were things + // left open so this gets investigated return true; } @@ -130,7 +135,7 @@ if(memleakfinder_numleaks() != 0) { failures++; - printf("FAILURE: Memory leaks detected\n"); + printf("FAILURE: Memory leaks detected in test code\n"); printf("==== MEMORY LEAKS =================================\n"); memleakfinder_reportleaks(); printf("===================================================\n"); From boxbackup-dev at fluffy.co.uk Sun Mar 4 22:59:22 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 22:59:22 +0000 Subject: [Box Backup-commit] COMMIT r1356 - box/chris/merge/infrastructure Message-ID: Author: chris Date: 2007-03-04 22:59:22 +0000 (Sun, 04 Mar 2007) New Revision: 1356 Modified: box/chris/merge/infrastructure/makeparcels.pl.in Log: Fix make error when optional file doesn't exist (refs #3, merges [1098]) Modified: box/chris/merge/infrastructure/makeparcels.pl.in =================================================================== --- box/chris/merge/infrastructure/makeparcels.pl.in 2007-03-04 22:57:15 UTC (rev 1355) +++ box/chris/merge/infrastructure/makeparcels.pl.in 2007-03-04 22:59:22 UTC (rev 1356) @@ -123,7 +123,8 @@ { if ($optional) { - print MAKE "\ttest -r $name && cp $name $dir\n"; + print MAKE "\ttest -r $name " . + "&& cp $name $dir || true\n"; } else { From boxbackup-dev at fluffy.co.uk Sun Mar 4 23:04:02 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 23:04:02 +0000 Subject: [Box Backup-commit] COMMIT r1357 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-04 23:04:02 +0000 (Sun, 04 Mar 2007) New Revision: 1357 Modified: box/chris/merge/lib/common/BoxConfig-MSVC.h Log: Declare that we have O_BINARY but not for MSVC (refs #3, merges [760] and [763]) Modified: box/chris/merge/lib/common/BoxConfig-MSVC.h =================================================================== --- box/chris/merge/lib/common/BoxConfig-MSVC.h 2007-03-04 22:59:22 UTC (rev 1356) +++ box/chris/merge/lib/common/BoxConfig-MSVC.h 2007-03-04 23:04:02 UTC (rev 1357) @@ -56,6 +56,10 @@ you don't. */ #define HAVE_DECL_XATTR_NOFOLLOW 0 +/* Define to 1 if you have the declaration of `O_BINARY', and to 0 if you + don't. */ +#define HAVE_DECL_O_BINARY 1 + /* Define to 1 if #define of pragmas works */ /* #undef HAVE_DEFINE_PRAGMA */ @@ -268,7 +272,7 @@ // #define HAVE_SYS_TIME_H 1 /* Define to 1 if you have the header file. */ -#define HAVE_SYS_TYPES_H 1 +// #define HAVE_SYS_TYPES_H 1 /* Define to 1 if you have the header file. */ /* #undef HAVE_SYS_WAIT_H */ @@ -318,6 +322,12 @@ /* Define to 1 if __syscall is available but needs a definition */ /* #undef HAVE___SYSCALL_NEED_DEFN */ +/* max value of long long calculated by configure */ +/* #undef LLONG_MAX */ + +/* min value of long long calculated by configure */ +/* #undef LLONG_MIN */ + /* Define to 1 if `lstat' dereferences a symlink specified with a trailing slash. */ /* #undef LSTAT_FOLLOWS_SLASHED_SYMLINK */ From boxbackup-dev at fluffy.co.uk Sun Mar 4 23:06:18 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 23:06:18 +0000 Subject: [Box Backup-commit] COMMIT r1358 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-04 23:06:18 +0000 (Sun, 04 Mar 2007) New Revision: 1358 Modified: box/chris/merge/lib/common/DebugMemLeakFinder.cpp Log: Watch out for our leak tracking data being destroyed and don't crash when subsequent objects are destroyed. (refs #3, merges [1341]) Modified: box/chris/merge/lib/common/DebugMemLeakFinder.cpp =================================================================== --- box/chris/merge/lib/common/DebugMemLeakFinder.cpp 2007-03-04 23:04:02 UTC (rev 1357) +++ box/chris/merge/lib/common/DebugMemLeakFinder.cpp 2007-03-04 23:06:18 UTC (rev 1358) @@ -49,6 +49,17 @@ { static std::map sMallocBlocks; static std::map sObjectBlocks; + static bool sTrackingDataDestroyed = false; + + static class DestructionWatchdog + { + public: + ~DestructionWatchdog() + { + sTrackingDataDestroyed = true; + } + } + sWatchdog; static bool sTrackMallocInSection = false; static std::set sSectionMallocBlocks; @@ -225,6 +236,8 @@ { InternalAllocGuard guard; + ASSERT(!sTrackingDataDestroyed); + memleakfinder_notaleak_insert_pre(); if(memleakfinder_global_enable && memleakfinder_initialised) { @@ -258,6 +271,8 @@ InternalAllocGuard guard; ASSERT(memleakfinder_initialised); + ASSERT(!sTrackingDataDestroyed); + sTrackMallocInSection = true; sSectionMallocBlocks.clear(); sTrackObjectsInSection = true; @@ -270,6 +285,7 @@ InternalAllocGuard guard; ASSERT(memleakfinder_initialised); + ASSERT(!sTrackingDataDestroyed); std::set::iterator s(sSectionMallocBlocks.begin()); for(; s != sSectionMallocBlocks.end(); ++s) @@ -295,6 +311,7 @@ InternalAllocGuard guard; ASSERT(memleakfinder_initialised); + ASSERT(!sTrackingDataDestroyed); int n = 0; @@ -316,6 +333,8 @@ { InternalAllocGuard guard; + ASSERT(!sTrackingDataDestroyed); + for(std::map::const_iterator i(sMallocBlocks.begin()); i != sMallocBlocks.end(); ++i) { if(is_leak(i->first)) ::fprintf(file, "Block 0x%p size %d allocated at %s:%d\n", i->first, i->second.size, i->second.file, i->second.line); @@ -388,6 +407,7 @@ if(!memleakfinder_global_enable) return; if(!memleakfinder_initialised) return; + ASSERT(!sTrackingDataDestroyed); if(block != 0) { @@ -411,6 +431,7 @@ if(!memleakfinder_global_enable) return; if(!memleakfinder_initialised) return; + if(sTrackingDataDestroyed) return; std::map::iterator i(sObjectBlocks.find(block)); if(i != sObjectBlocks.end()) From boxbackup-dev at fluffy.co.uk Sun Mar 4 23:36:14 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 23:36:14 +0000 Subject: [Box Backup-commit] COMMIT r1359 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-04 23:36:14 +0000 (Sun, 04 Mar 2007) New Revision: 1359 Modified: box/chris/general/lib/common/EventWatchFilesystemObject.cpp Log: Use logging framework Modified: box/chris/general/lib/common/EventWatchFilesystemObject.cpp =================================================================== --- box/chris/general/lib/common/EventWatchFilesystemObject.cpp 2007-03-04 23:06:18 UTC (rev 1358) +++ box/chris/general/lib/common/EventWatchFilesystemObject.cpp 2007-03-04 23:36:14 UTC (rev 1359) @@ -17,6 +17,7 @@ #include "EventWatchFilesystemObject.h" #include "autogen_CommonException.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -37,9 +38,9 @@ #ifdef HAVE_KQUEUE if(mDescriptor == -1) { - ::syslog(LOG_ERR, "EventWatchFilesystemObject: " - "Failed to open file '%s': %s", - Filename, strerror(errno)); + BOX_ERROR("EventWatchFilesystemObject: " + "Failed to open file '" << Filename << "': " << + strerror(errno)); THROW_EXCEPTION(CommonException, OSFileOpenError) } #else From boxbackup-dev at fluffy.co.uk Sun Mar 4 23:38:04 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 23:38:04 +0000 Subject: [Box Backup-commit] COMMIT r1360 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-04 23:38:03 +0000 (Sun, 04 Mar 2007) New Revision: 1360 Modified: box/chris/merge/lib/common/EventWatchFilesystemObject.cpp Log: Add logging in remaining places where Common OSFileOpenError can be thrown. (refs #3, merges part of [1099] and [1359]) Modified: box/chris/merge/lib/common/EventWatchFilesystemObject.cpp =================================================================== --- box/chris/merge/lib/common/EventWatchFilesystemObject.cpp 2007-03-04 23:36:14 UTC (rev 1359) +++ box/chris/merge/lib/common/EventWatchFilesystemObject.cpp 2007-03-04 23:38:03 UTC (rev 1360) @@ -17,6 +17,7 @@ #include "EventWatchFilesystemObject.h" #include "autogen_CommonException.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -37,6 +38,9 @@ #ifdef HAVE_KQUEUE if(mDescriptor == -1) { + BOX_ERROR("EventWatchFilesystemObject: " + "Failed to open file '" << Filename << "': " << + strerror(errno)); THROW_EXCEPTION(CommonException, OSFileOpenError) } #else From boxbackup-dev at fluffy.co.uk Sun Mar 4 23:41:22 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 23:41:22 +0000 Subject: [Box Backup-commit] COMMIT r1361 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-04 23:41:22 +0000 (Sun, 04 Mar 2007) New Revision: 1361 Modified: box/chris/merge/lib/common/ExcludeList.cpp Log: Update to match new recommended installation of pcreposix.h as regex.h. (refs #3, merges [1281], reverting [1233]) Modified: box/chris/merge/lib/common/ExcludeList.cpp =================================================================== --- box/chris/merge/lib/common/ExcludeList.cpp 2007-03-04 23:38:03 UTC (rev 1360) +++ box/chris/merge/lib/common/ExcludeList.cpp 2007-03-04 23:41:22 UTC (rev 1361) @@ -10,11 +10,7 @@ #include "Box.h" #ifdef HAVE_REGEX_H - #ifdef WIN32 - #include - #else - #include - #endif // WIN32 + #include #define EXCLUDELIST_IMPLEMENTATION_REGEX_T_DEFINED #endif From boxbackup-dev at fluffy.co.uk Sun Mar 4 23:55:42 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 23:55:42 +0000 Subject: [Box Backup-commit] COMMIT r1362 - box/chris/general/lib/win32 Message-ID: Author: chris Date: 2007-03-04 23:55:41 +0000 (Sun, 04 Mar 2007) New Revision: 1362 Modified: box/chris/general/lib/win32/emu.cpp Log: Revert bad parts of [1347] Modified: box/chris/general/lib/win32/emu.cpp =================================================================== --- box/chris/general/lib/win32/emu.cpp 2007-03-04 23:41:22 UTC (rev 1361) +++ box/chris/general/lib/win32/emu.cpp 2007-03-04 23:55:41 UTC (rev 1362) @@ -14,11 +14,8 @@ #ifdef HAVE_UNISTD_H #include #endif +#include -#ifdef HAVE_PROCESS_H - #include -#endif - #include #include @@ -868,7 +865,7 @@ _ui64toa(fi.dwVolumeSerialNumber, s->f_mntonname + 1, 16); // pseudo unix mount point - s->f_mntonname[0] = DIRECTORY_SEPARATOR_ASCHAR; + s->f_mntonname[0] = '\\'; CloseHandle(handle); // close the handle From boxbackup-dev at fluffy.co.uk Sun Mar 4 23:58:19 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 04 Mar 2007 23:58:19 +0000 Subject: [Box Backup-commit] COMMIT r1363 - box/chris/general/test/bbackupd Message-ID: Author: chris Date: 2007-03-04 23:58:19 +0000 (Sun, 04 Mar 2007) New Revision: 1363 Modified: box/chris/general/test/bbackupd/testbbackupd.cpp Log: Reorder headers for readability Modified: box/chris/general/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-04 23:55:41 UTC (rev 1362) +++ box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-04 23:58:19 UTC (rev 1363) @@ -38,33 +38,34 @@ #include #endif -#include "Test.h" +#include "BackupClientCryptoKeys.h" #include "BackupClientFileAttributes.h" -#include "CommonException.h" +#include "BackupClientRestore.h" +#include "BackupDaemon.h" +#include "BackupDaemonConfigVerify.h" +#include "BackupStoreConstants.h" +#include "BackupStoreDirectory.h" #include "BackupStoreException.h" -#include "FileModificationTime.h" -#include "autogen_BackupProtocolClient.h" -#include "SSLLib.h" -#include "TLSContext.h" -#include "SocketStreamTLS.h" #include "BoxPortsAndFiles.h" -#include "BackupStoreConstants.h" -#include "Socket.h" -#include "BackupClientRestore.h" -#include "BackupStoreDirectory.h" -#include "BackupClientCryptoKeys.h" -#include "CollectInBufferStream.h" -#include "Utils.h" #include "BoxTime.h" #include "BoxTimeToUnix.h" -#include "ServerControl.h" +#include "CollectInBufferStream.h" +#include "CommonException.h" #include "Configuration.h" -#include "BackupDaemonConfigVerify.h" +#include "FileModificationTime.h" +#include "FileStream.h" #include "IOStreamGetLine.h" -#include "FileStream.h" #include "LocalProcessStream.h" -#include "BackupDaemon.h" +#include "SSLLib.h" +#include "ServerControl.h" +#include "Socket.h" +#include "SocketStreamTLS.h" +#include "TLSContext.h" +#include "Test.h" #include "Timer.h" +#include "Utils.h" + +#include "autogen_BackupProtocolClient.h" #include "intercept.h" #include "MemLeakFindOn.h" From boxbackup-dev at fluffy.co.uk Mon Mar 5 00:00:42 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 05 Mar 2007 00:00:42 +0000 Subject: [Box Backup-commit] COMMIT r1364 - box/chris/general/lib/win32 Message-ID: Author: chris Date: 2007-03-05 00:00:42 +0000 (Mon, 05 Mar 2007) New Revision: 1364 Modified: box/chris/general/lib/win32/emu.cpp Log: Improve GetErrorMessage() by including the error number/code in the message (helps debugging on foreign langauge versions of Windows) Modified: box/chris/general/lib/win32/emu.cpp =================================================================== --- box/chris/general/lib/win32/emu.cpp 2007-03-04 23:58:19 UTC (rev 1363) +++ box/chris/general/lib/win32/emu.cpp 2007-03-05 00:00:42 UTC (rev 1364) @@ -9,15 +9,16 @@ #include #include +#include #include #ifdef HAVE_UNISTD_H #include #endif -#include #include #include +#include // message resource definitions for syslog() @@ -502,10 +503,11 @@ return std::string("failed to get error message"); } - std::string out(pMsgBuf); + std::ostringstream line; + line << pMsgBuf << " (" << errorCode << ")"; LocalFree(pMsgBuf); - return out; + return line.str(); } // -------------------------------------------------------------------------- From boxbackup-dev at fluffy.co.uk Mon Mar 5 00:01:38 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 05 Mar 2007 00:01:38 +0000 Subject: [Box Backup-commit] COMMIT r1366 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-05 00:01:38 +0000 (Mon, 05 Mar 2007) New Revision: 1366 Modified: box/chris/general/lib/common/Test.h Log: Sleep for a second after pid file is created to give the process time to write its pid into it. Modified: box/chris/general/lib/common/Test.h =================================================================== --- box/chris/general/lib/common/Test.h 2007-03-05 00:01:05 UTC (rev 1365) +++ box/chris/general/lib/common/Test.h 2007-03-05 00:01:38 UTC (rev 1366) @@ -280,6 +280,9 @@ ::fprintf(stdout, "done.\n"); } + // wait a second for the pid to be written to the file + ::sleep(1); + // read pid file int pid = ReadPidFile(pidFile); From boxbackup-dev at fluffy.co.uk Mon Mar 5 00:01:05 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 05 Mar 2007 00:01:05 +0000 Subject: [Box Backup-commit] COMMIT r1365 - box/chris/general/lib/win32 Message-ID: Author: chris Date: 2007-03-05 00:01:05 +0000 (Mon, 05 Mar 2007) New Revision: 1365 Modified: box/chris/general/lib/win32/emu.h Log: Reorder for clarity Expose GetErrorMessage() Improve comments Modified: box/chris/general/lib/win32/emu.h =================================================================== --- box/chris/general/lib/win32/emu.h 2007-03-05 00:00:42 UTC (rev 1364) +++ box/chris/general/lib/win32/emu.h 2007-03-05 00:01:05 UTC (rev 1365) @@ -291,7 +291,7 @@ time_t st_mtime; time_t st_ctime; }; -#endif +#endif // 0 // need this for conversions time_t ConvertFileTimeToTime_t(FILETIME *fileTime); @@ -319,6 +319,19 @@ int statfs(const char * name, struct statfs * s); int poll(struct pollfd *ufds, unsigned long nfds, int timeout); + +struct iovec { + void *iov_base; /* Starting address */ + size_t iov_len; /* Number of bytes */ +}; + +int readv (int filedes, const struct iovec *vector, size_t count); +int writev(int filedes, const struct iovec *vector, size_t count); + +// The following functions are not emulations, but utilities for other +// parts of the code where Windows API is used or windows-specific stuff +// is needed, like codepage conversion. + bool EnableBackupRights( void ); bool ConvertEncoding (const std::string& rSource, int sourceCodePage, @@ -330,15 +343,12 @@ bool ConvertUtf8ToConsole(const char* pString, std::string& rDest); bool ConvertConsoleToUtf8(const char* pString, std::string& rDest); -// replacement for _cgetws which requires a relatively recent C runtime lib +// GetErrorMessage() returns a system error message, like strerror() +// but for Windows error codes. +std::string GetErrorMessage(DWORD errorCode); + +// console_read() is a replacement for _cgetws which requires a +// relatively recent C runtime lib int console_read(char* pBuffer, size_t BufferSize); -struct iovec { - void *iov_base; /* Starting address */ - size_t iov_len; /* Number of bytes */ -}; - -int readv (int filedes, const struct iovec *vector, size_t count); -int writev(int filedes, const struct iovec *vector, size_t count); - #endif // !EMU_INCLUDE && WIN32 From boxbackup-dev at fluffy.co.uk Mon Mar 5 00:02:03 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 05 Mar 2007 00:02:03 +0000 Subject: [Box Backup-commit] COMMIT r1367 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-05 00:02:03 +0000 (Mon, 05 Mar 2007) New Revision: 1367 Modified: box/chris/general/lib/common/Timer.cpp Log: Throw an assertion error if a NULL timer is added Modified: box/chris/general/lib/common/Timer.cpp =================================================================== --- box/chris/general/lib/common/Timer.cpp 2007-03-05 00:01:38 UTC (rev 1366) +++ box/chris/general/lib/common/Timer.cpp 2007-03-05 00:02:03 UTC (rev 1367) @@ -90,6 +90,7 @@ void Timers::Add(Timer& rTimer) { ASSERT(spTimers); + ASSERT(&rTimer); spTimers->push_back(&rTimer); Reschedule(); } @@ -106,11 +107,13 @@ void Timers::Remove(Timer& rTimer) { ASSERT(spTimers); + ASSERT(&rTimer); bool restart = true; while (restart) { restart = false; + for (std::vector::iterator i = spTimers->begin(); i != spTimers->end(); i++) { From boxbackup-dev at fluffy.co.uk Mon Mar 5 00:02:37 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 05 Mar 2007 00:02:37 +0000 Subject: [Box Backup-commit] COMMIT r1368 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-05 00:02:37 +0000 (Mon, 05 Mar 2007) New Revision: 1368 Modified: box/chris/general/lib/common/FileStream.cpp Log: Use logging framework Modified: box/chris/general/lib/common/FileStream.cpp =================================================================== --- box/chris/general/lib/common/FileStream.cpp 2007-03-05 00:02:03 UTC (rev 1367) +++ box/chris/general/lib/common/FileStream.cpp 2007-03-05 00:02:37 UTC (rev 1368) @@ -10,6 +10,7 @@ #include "Box.h" #include "FileStream.h" #include "CommonException.h" +#include "Logging.h" #include @@ -73,8 +74,7 @@ #endif { MEMLEAKFINDER_NOT_A_LEAK(this); - ::syslog(LOG_ERR, "FileStream: called with invalid " - "file handle"); + BOX_ERROR("FileStream: called with invalid file handle"); THROW_EXCEPTION(CommonException, OSFileOpenError) } #ifdef WIN32 @@ -98,7 +98,7 @@ if(mOSFileHandle < 0) { MEMLEAKFINDER_NOT_A_LEAK(this); - ::syslog(LOG_ERR, "FileStream: copying unopened file"); + BOX_ERROR("FileStream: copying unopened file"); THROW_EXCEPTION(CommonException, OSFileOpenError) } } @@ -156,8 +156,8 @@ } else { - ::syslog(LOG_ERR, "Failed to read from file: error %d", - GetLastError()); + BOX_ERROR("Failed to read from file: " << + GetErrorMessage(GetLastError())); r = -1; } #else From boxbackup-dev at fluffy.co.uk Mon Mar 5 00:06:26 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 05 Mar 2007 00:06:26 +0000 Subject: [Box Backup-commit] COMMIT r1369 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-05 00:06:26 +0000 (Mon, 05 Mar 2007) New Revision: 1369 Modified: box/chris/merge/lib/common/FileStream.cpp Log: Add logging in remaining places where Common OSFileOpenError can be thrown. Fix compile warning (signed vs unsigned comparison) (refs #3, merges part of [1099], and [1368]) Modified: box/chris/merge/lib/common/FileStream.cpp =================================================================== --- box/chris/merge/lib/common/FileStream.cpp 2007-03-05 00:02:37 UTC (rev 1368) +++ box/chris/merge/lib/common/FileStream.cpp 2007-03-05 00:06:26 UTC (rev 1369) @@ -10,6 +10,7 @@ #include "Box.h" #include "FileStream.h" #include "CommonException.h" +#include "Logging.h" #include @@ -73,6 +74,7 @@ #endif { MEMLEAKFINDER_NOT_A_LEAK(this); + BOX_ERROR("FileStream: called with invalid file handle"); THROW_EXCEPTION(CommonException, OSFileOpenError) } #ifdef WIN32 @@ -100,6 +102,7 @@ #endif { MEMLEAKFINDER_NOT_A_LEAK(this); + BOX_ERROR("FileStream: copying unopened file"); THROW_EXCEPTION(CommonException, OSFileOpenError) } } @@ -157,8 +160,8 @@ } else { - ::syslog(LOG_ERR, "Failed to read from file: error %d", - GetLastError()); + BOX_ERROR("Failed to read from file: " << + GetErrorMessage(GetLastError())); r = -1; } #else @@ -222,7 +225,7 @@ NULL ); - if ( (res == 0) || (numBytesWritten != NBytes)) + if ((res == 0) || (numBytesWritten != (DWORD)NBytes)) { // DWORD err = GetLastError(); THROW_EXCEPTION(CommonException, OSFileWriteError) From boxbackup-dev at fluffy.co.uk Mon Mar 5 00:12:34 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 05 Mar 2007 00:12:34 +0000 Subject: [Box Backup-commit] COMMIT r1370 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-05 00:12:34 +0000 (Mon, 05 Mar 2007) New Revision: 1370 Modified: box/chris/general/lib/common/Guards.h Log: Use logging framework Modified: box/chris/general/lib/common/Guards.h =================================================================== --- box/chris/general/lib/common/Guards.h 2007-03-05 00:06:26 UTC (rev 1369) +++ box/chris/general/lib/common/Guards.h 2007-03-05 00:12:34 UTC (rev 1370) @@ -21,6 +21,7 @@ #include #include "CommonException.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -33,9 +34,8 @@ { if(mOSFileHandle < 0) { - ::syslog(LOG_ERR, "FileHandleGuard: " - "failed to open file '%s': error %s", - filename, strerror(errno)); + BOX_ERROR("FileHandleGuard: failed to open file '" << + filename << "': " << strerror(errno)); THROW_EXCEPTION(CommonException, OSFileOpenError) } } From boxbackup-dev at fluffy.co.uk Mon Mar 5 00:14:56 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 05 Mar 2007 00:14:56 +0000 Subject: [Box Backup-commit] COMMIT r1371 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-05 00:14:55 +0000 (Mon, 05 Mar 2007) New Revision: 1371 Modified: box/chris/merge/lib/common/Guards.h Log: Add logging in remaining places where Common OSFileOpenError can be thrown. (refs #3, merges part of [1099], and [1370]) Modified: box/chris/merge/lib/common/Guards.h =================================================================== --- box/chris/merge/lib/common/Guards.h 2007-03-05 00:12:34 UTC (rev 1370) +++ box/chris/merge/lib/common/Guards.h 2007-03-05 00:14:55 UTC (rev 1371) @@ -21,6 +21,7 @@ #include #include "CommonException.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -33,6 +34,8 @@ { if(mOSFileHandle < 0) { + BOX_ERROR("FileHandleGuard: failed to open file '" << + filename << "': " << strerror(errno)); THROW_EXCEPTION(CommonException, OSFileOpenError) } } From boxbackup-dev at fluffy.co.uk Mon Mar 5 00:16:29 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 05 Mar 2007 00:16:29 +0000 Subject: [Box Backup-commit] COMMIT r1372 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-05 00:16:29 +0000 (Mon, 05 Mar 2007) New Revision: 1372 Modified: box/chris/merge/lib/common/Logging.h Log: Win32 compile fix (refs #3, merges [1306]) Modified: box/chris/merge/lib/common/Logging.h =================================================================== --- box/chris/merge/lib/common/Logging.h 2007-03-05 00:14:55 UTC (rev 1371) +++ box/chris/merge/lib/common/Logging.h 2007-03-05 00:16:29 UTC (rev 1372) @@ -42,10 +42,21 @@ #define BOX_TRACE(stuff) BOX_LOG(Log::TRACE, stuff) #endif +#undef ERROR + namespace Log { - enum Level { NOTHING = 1, FATAL, ERROR, WARNING, NOTICE, INFO, TRACE, - EVERYTHING }; + enum Level + { + NOTHING = 1, + FATAL, + ERROR, + WARNING, + NOTICE, + INFO, + TRACE, + EVERYTHING + }; } // -------------------------------------------------------------------------- From boxbackup-dev at fluffy.co.uk Thu Mar 8 22:15:09 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 22:15:09 +0000 Subject: [Box Backup-commit] COMMIT r1373 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-08 22:15:08 +0000 (Thu, 08 Mar 2007) New Revision: 1373 Modified: box/chris/general/lib/common/FileStream.cpp Log: Add error handling to commented-out function in case it is reinstated Modified: box/chris/general/lib/common/FileStream.cpp =================================================================== --- box/chris/general/lib/common/FileStream.cpp 2007-03-05 00:16:29 UTC (rev 1372) +++ box/chris/general/lib/common/FileStream.cpp 2007-03-08 22:15:08 UTC (rev 1373) @@ -95,7 +95,11 @@ : mOSFileHandle(::dup(rToCopy.mOSFileHandle)), mIsEOF(rToCopy.mIsEOF) { +#ifdef WIN32 + if(mOSFileHandle == INVALID_HANDLE_VALUE) +#else if(mOSFileHandle < 0) +#endif { MEMLEAKFINDER_NOT_A_LEAK(this); BOX_ERROR("FileStream: copying unopened file"); From boxbackup-dev at fluffy.co.uk Thu Mar 8 22:18:34 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 22:18:34 +0000 Subject: [Box Backup-commit] COMMIT r1374 - box/chris/general/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-08 22:18:34 +0000 (Thu, 08 Mar 2007) New Revision: 1374 Modified: box/chris/general/bin/bbackupquery/BackupQueries.cpp Log: Move definition of fileModTime back to original place to reduce diff Modified: box/chris/general/bin/bbackupquery/BackupQueries.cpp =================================================================== --- box/chris/general/bin/bbackupquery/BackupQueries.cpp 2007-03-08 22:15:08 UTC (rev 1373) +++ box/chris/general/bin/bbackupquery/BackupQueries.cpp 2007-03-08 22:18:34 UTC (rev 1374) @@ -1576,8 +1576,8 @@ } // Compare attributes + BackupClientFileAttributes localAttr; box_time_t fileModTime = 0; - BackupClientFileAttributes localAttr; localAttr.ReadAttributes(localPath.c_str(), false /* don't zero mod times */, &fileModTime); modifiedAfterLastSync = (fileModTime > rParams.mLatestFileUploadTime); bool ignoreAttrModTime = true; From boxbackup-dev at fluffy.co.uk Thu Mar 8 22:20:31 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 22:20:31 +0000 Subject: [Box Backup-commit] COMMIT r1375 - box/chris/merge/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-08 22:20:31 +0000 (Thu, 08 Mar 2007) New Revision: 1375 Modified: box/chris/merge/bin/bbackupquery/BackupQueries.cpp Log: Count a difference when a local directory does not exist, but it does exist on the server (refs #3) Modified: box/chris/merge/bin/bbackupquery/BackupQueries.cpp =================================================================== --- box/chris/merge/bin/bbackupquery/BackupQueries.cpp 2007-03-08 22:18:34 UTC (rev 1374) +++ box/chris/merge/bin/bbackupquery/BackupQueries.cpp 2007-03-08 22:20:31 UTC (rev 1375) @@ -1317,6 +1317,7 @@ "(compared to server directory '%s')\n", localDirDisplay.c_str(), storeDirDisplay.c_str()); + rParams.mDifferences ++; } else { From boxbackup-dev at fluffy.co.uk Thu Mar 8 22:22:57 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 22:22:57 +0000 Subject: [Box Backup-commit] COMMIT r1376 - box/chris/merge/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-08 22:22:56 +0000 (Thu, 08 Mar 2007) New Revision: 1376 Modified: box/chris/merge/bin/bbackupquery/BackupQueries.cpp Log: Catch exceptions during restore and report them, rather than aborting bbackupquery (refs #3) Modified: box/chris/merge/bin/bbackupquery/BackupQueries.cpp =================================================================== --- box/chris/merge/bin/bbackupquery/BackupQueries.cpp 2007-03-08 22:20:31 UTC (rev 1375) +++ box/chris/merge/bin/bbackupquery/BackupQueries.cpp 2007-03-08 22:22:56 UTC (rev 1376) @@ -1908,11 +1908,34 @@ #endif // Go and restore... - switch(BackupClientRestore(mrConnection, dirID, localName.c_str(), - true /* print progress dots */, restoreDeleted, - false /* don't undelete after restore! */, - opts['r'] /* resume? */)) + int result; + + try { + result = BackupClientRestore(mrConnection, dirID, + localName.c_str(), + true /* print progress dots */, restoreDeleted, + false /* don't undelete after restore! */, + opts['r'] /* resume? */); + } + catch (BoxException &e) + { + ::syslog(LOG_ERR, "Failed to restore: %s", e.what()); + return; + } + catch(std::exception &e) + { + ::syslog(LOG_ERR, "Failed to restore: %s", e.what()); + return; + } + catch(...) + { + ::syslog(LOG_ERR, "Failed to restore: unknown error"); + return; + } + + switch(result) + { case Restore_Complete: printf("Restore complete\n"); break; From boxbackup-dev at fluffy.co.uk Thu Mar 8 22:27:39 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 22:27:39 +0000 Subject: [Box Backup-commit] COMMIT r1377 - box/chris/merge/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-08 22:27:39 +0000 (Thu, 08 Mar 2007) New Revision: 1377 Modified: box/chris/merge/bin/bbackupquery/BackupQueries.cpp Log: Report an appropriate error if the target path of the restore operation is not found (refs #3, merges [514]) Modified: box/chris/merge/bin/bbackupquery/BackupQueries.cpp =================================================================== --- box/chris/merge/bin/bbackupquery/BackupQueries.cpp 2007-03-08 22:22:56 UTC (rev 1376) +++ box/chris/merge/bin/bbackupquery/BackupQueries.cpp 2007-03-08 22:27:39 UTC (rev 1377) @@ -1948,6 +1948,14 @@ printf("The target directory exists. You cannot restore over an existing directory.\n"); break; + #ifdef WIN32 + case Restore_TargetPathNotFound: + printf("The target directory path does not exist.\n" + "To restore to a directory whose parent " + "does not exist, create the parent first.\n"); + break; + #endif + default: printf("ERROR: Unknown restore result.\n"); break; From boxbackup-dev at fluffy.co.uk Thu Mar 8 22:47:29 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 22:47:29 +0000 Subject: [Box Backup-commit] COMMIT r1378 - box/chris/merge/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-08 22:47:29 +0000 (Thu, 08 Mar 2007) New Revision: 1378 Modified: box/chris/merge/bin/bbackupquery/BackupQueries.cpp box/chris/merge/bin/bbackupquery/BackupQueries.h Log: Report number of files and directories which could not be compared separately from compare failures, and report the appropriate return code (refs #3, merges [651]) Modified: box/chris/merge/bin/bbackupquery/BackupQueries.cpp =================================================================== --- box/chris/merge/bin/bbackupquery/BackupQueries.cpp 2007-03-08 22:27:39 UTC (rev 1377) +++ box/chris/merge/bin/bbackupquery/BackupQueries.cpp 2007-03-08 22:47:29 UTC (rev 1378) @@ -1054,6 +1054,7 @@ mIgnoreAttributes(false), mDifferences(0), mDifferencesExplainedByModTime(0), + mUncheckedFiles(0), mExcludedDirs(0), mExcludedFiles(0), mpExcludeFiles(0), @@ -1177,13 +1178,29 @@ return; } - printf("\n[ %d (of %d) differences probably due to file modifications after the last upload ]\nDifferences: %d (%d dirs excluded, %d files excluded)\n", - params.mDifferencesExplainedByModTime, params.mDifferences, params.mDifferences, params.mExcludedDirs, params.mExcludedFiles); + printf("\n[ %d (of %d) differences probably due to file " + "modifications after the last upload ]\n" + "Differences: %d (%d dirs excluded, %d files excluded, " + "%d files not checked)\n", + params.mDifferencesExplainedByModTime, params.mDifferences, + params.mDifferences, params.mExcludedDirs, + params.mExcludedFiles, params.mUncheckedFiles); // Set return code? if(opts['c']) { - SetReturnCode((params.mDifferences == 0)?COMPARE_RETURN_SAME:COMPARE_RETURN_DIFFERENT); + if (params.mUncheckedFiles != 0) + { + SetReturnCode(COMPARE_RETURN_ERROR); + } + else if (params.mDifferences != 0) + { + SetReturnCode(COMPARE_RETURN_DIFFERENT); + } + else + { + SetReturnCode(COMPARE_RETURN_SAME); + } } } @@ -1323,6 +1340,7 @@ { printf("ERROR: stat on local dir '%s'\n", localDirDisplay.c_str()); + rParams.mUncheckedFiles ++; } return; } @@ -1372,6 +1390,7 @@ { printf("ERROR: opendir on local dir '%s'\n", localDirDisplay.c_str()); + rParams.mUncheckedFiles ++; return; } try @@ -1668,10 +1687,12 @@ e.GetType(), e.GetSubType(), storePathDisplay.c_str()); + rParams.mUncheckedFiles ++; } catch(...) - { + { printf("ERROR: (unknown) during file fetch and comparison for '%s'\n", storePathDisplay.c_str()); + rParams.mUncheckedFiles ++; } // Remove from set so that we know it's been compared Modified: box/chris/merge/bin/bbackupquery/BackupQueries.h =================================================================== --- box/chris/merge/bin/bbackupquery/BackupQueries.h 2007-03-08 22:27:39 UTC (rev 1377) +++ box/chris/merge/bin/bbackupquery/BackupQueries.h 2007-03-08 22:47:29 UTC (rev 1378) @@ -71,6 +71,7 @@ bool mIgnoreAttributes; int mDifferences; int mDifferencesExplainedByModTime; + int mUncheckedFiles; int mExcludedDirs; int mExcludedFiles; const ExcludeList *mpExcludeFiles; From boxbackup-dev at fluffy.co.uk Thu Mar 8 22:49:05 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 22:49:05 +0000 Subject: [Box Backup-commit] COMMIT r1379 - box/chris/general/bin/bbackupd Message-ID: Author: chris Date: 2007-03-08 22:49:04 +0000 (Thu, 08 Mar 2007) New Revision: 1379 Modified: box/chris/general/bin/bbackupd/bbackupd.cpp Log: Don't close the syslog forcibly, let logging framework do it Modified: box/chris/general/bin/bbackupd/bbackupd.cpp =================================================================== --- box/chris/general/bin/bbackupd/bbackupd.cpp 2007-03-08 22:47:29 UTC (rev 1378) +++ box/chris/general/bin/bbackupd/bbackupd.cpp 2007-03-08 22:49:04 UTC (rev 1379) @@ -94,8 +94,6 @@ delete gpDaemonService; - ::closelog(); - return ExitCode; #else // !WIN32 From boxbackup-dev at fluffy.co.uk Thu Mar 8 22:59:11 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 22:59:11 +0000 Subject: [Box Backup-commit] COMMIT r1380 - box/chris/merge/bin/bbstored Message-ID: Author: chris Date: 2007-03-08 22:59:11 +0000 (Thu, 08 Mar 2007) New Revision: 1380 Modified: box/chris/merge/bin/bbstored/BBStoreDHousekeeping.cpp box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp box/chris/merge/bin/bbstored/BackupStoreDaemon.h Log: Run housekeeping in idle time on Win32 (refs #3) Modified: box/chris/merge/bin/bbstored/BBStoreDHousekeeping.cpp =================================================================== --- box/chris/merge/bin/bbstored/BBStoreDHousekeeping.cpp 2007-03-08 22:49:04 UTC (rev 1379) +++ box/chris/merge/bin/bbstored/BBStoreDHousekeeping.cpp 2007-03-08 22:59:11 UTC (rev 1380) @@ -50,7 +50,8 @@ { RunHousekeepingIfNeeded(); - // Calculate how long should wait before doing the next housekeeping run + // Calculate how long should wait before doing the next + // housekeeping run int64_t timeNow = GetCurrentBoxTime(); time_t secondsToGo = BoxTimeToSeconds( (mLastHousekeepingRun + housekeepingInterval) - @@ -72,6 +73,7 @@ // Time now int64_t timeNow = GetCurrentBoxTime(); + // Do housekeeping if the time interval has elapsed since the last check if((timeNow - mLastHousekeepingRun) < housekeepingInterval) { @@ -148,6 +150,19 @@ SetProcessTitle("housekeeping, idle"); } +void BackupStoreDaemon::OnIdle() +{ + #ifdef WIN32 + if (!mHousekeepingInited) + { + HousekeepingInit(); + mHousekeepingInited = true; + } + + RunHousekeepingIfNeeded(); + #endif +} + // -------------------------------------------------------------------------- // // Function @@ -159,6 +174,11 @@ // -------------------------------------------------------------------------- bool BackupStoreDaemon::CheckForInterProcessMsg(int AccountNum, int MaximumWaitTime) { + if(!mInterProcessCommsSocket.IsOpened()) + { + return false; + } + // First, check to see if it's EOF -- this means something has gone wrong, and the housekeeping should terminate. if(mInterProcessComms.IsEOF()) { Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp =================================================================== --- box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2007-03-08 22:49:04 UTC (rev 1379) +++ box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2007-03-08 22:59:11 UTC (rev 1380) @@ -42,6 +42,7 @@ mExtendedLogging(false), mHaveForkedHousekeeping(false), mIsHousekeepingProcess(false), + mHousekeepingInited(false), mInterProcessComms(mInterProcessCommsSocket) { } Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.h =================================================================== --- box/chris/merge/bin/bbstored/BackupStoreDaemon.h 2007-03-08 22:49:04 UTC (rev 1379) +++ box/chris/merge/bin/bbstored/BackupStoreDaemon.h 2007-03-08 22:59:11 UTC (rev 1380) @@ -72,10 +72,12 @@ bool mExtendedLogging; bool mHaveForkedHousekeeping; bool mIsHousekeepingProcess; + bool mHousekeepingInited; SocketStream mInterProcessCommsSocket; IOStreamGetLine mInterProcessComms; + virtual void OnIdle(); void HousekeepingInit(); void RunHousekeepingIfNeeded(); int64_t mLastHousekeepingRun; From boxbackup-dev at fluffy.co.uk Thu Mar 8 22:59:42 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 22:59:42 +0000 Subject: [Box Backup-commit] COMMIT r1381 - box/chris/general/bin/bbstored Message-ID: Author: chris Date: 2007-03-08 22:59:42 +0000 (Thu, 08 Mar 2007) New Revision: 1381 Modified: box/chris/general/bin/bbstored/BackupStoreDaemon.h Log: Merge back cosmetic fixes Modified: box/chris/general/bin/bbstored/BackupStoreDaemon.h =================================================================== --- box/chris/general/bin/bbstored/BackupStoreDaemon.h 2007-03-08 22:59:11 UTC (rev 1380) +++ box/chris/general/bin/bbstored/BackupStoreDaemon.h 2007-03-08 22:59:42 UTC (rev 1381) @@ -38,7 +38,7 @@ BackupStoreDaemon(const BackupStoreDaemon &rToCopy); public: - // For BackupContext to comminicate with housekeeping process + // For BackupContext to communicate with housekeeping process void SendMessageToHousekeepingProcess(const void *Msg, int MsgLen) { #ifndef WIN32 @@ -73,7 +73,7 @@ bool mHaveForkedHousekeeping; bool mIsHousekeepingProcess; bool mHousekeepingInited; - + SocketStream mInterProcessCommsSocket; IOStreamGetLine mInterProcessComms; From boxbackup-dev at fluffy.co.uk Thu Mar 8 23:00:22 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 23:00:22 +0000 Subject: [Box Backup-commit] COMMIT r1382 - box/chris/general/bin/bbstored Message-ID: Author: chris Date: 2007-03-08 23:00:21 +0000 (Thu, 08 Mar 2007) New Revision: 1382 Modified: box/chris/general/bin/bbstored/BBStoreDHousekeeping.cpp Log: Only do housekeeping in idle time on Win32; other platforms fork a separate process to do it. Modified: box/chris/general/bin/bbstored/BBStoreDHousekeeping.cpp =================================================================== --- box/chris/general/bin/bbstored/BBStoreDHousekeeping.cpp 2007-03-08 22:59:42 UTC (rev 1381) +++ box/chris/general/bin/bbstored/BBStoreDHousekeeping.cpp 2007-03-08 23:00:21 UTC (rev 1382) @@ -152,6 +152,7 @@ void BackupStoreDaemon::OnIdle() { + #ifdef WIN32 if (!mHousekeepingInited) { HousekeepingInit(); @@ -159,6 +160,7 @@ } RunHousekeepingIfNeeded(); + #endif } // -------------------------------------------------------------------------- From boxbackup-dev at fluffy.co.uk Thu Mar 8 23:03:32 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 23:03:32 +0000 Subject: [Box Backup-commit] COMMIT r1383 - box/chris/merge Message-ID: Author: chris Date: 2007-03-08 23:03:32 +0000 (Thu, 08 Mar 2007) New Revision: 1383 Modified: box/chris/merge/configure.ac Log: Check for d_ino member in struct dirent (refs #3, merges [1351]) Modified: box/chris/merge/configure.ac =================================================================== --- box/chris/merge/configure.ac 2007-03-08 23:00:21 UTC (rev 1382) +++ box/chris/merge/configure.ac 2007-03-08 23:03:32 UTC (rev 1383) @@ -156,6 +156,7 @@ AX_CHECK_MOUNT_POINT(,[ AC_MSG_ERROR([[cannot work out how to discover mount points on your platform]]) ]) + AC_CHECK_MEMBERS([struct dirent.d_ino],,, [[#include ]]) ;; esac From boxbackup-dev at fluffy.co.uk Thu Mar 8 23:09:08 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 08 Mar 2007 23:09:08 +0000 Subject: [Box Backup-commit] COMMIT r1384 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-08 23:09:08 +0000 (Thu, 08 Mar 2007) New Revision: 1384 Modified: box/chris/merge/lib/common/Test.h Log: Record the file and line of first test failure (refs #3, merges [593]) Modified: box/chris/merge/lib/common/Test.h =================================================================== --- box/chris/merge/lib/common/Test.h 2007-03-08 23:03:32 UTC (rev 1383) +++ box/chris/merge/lib/common/Test.h 2007-03-08 23:09:08 UTC (rev 1384) @@ -23,14 +23,26 @@ #include extern int failures; +extern int first_fail_line; +extern std::string first_fail_file; -#define TEST_FAIL_WITH_MESSAGE(msg) {failures++; printf("FAILURE: " msg " at " __FILE__ "(%d)\n", __LINE__);} -#define TEST_ABORT_WITH_MESSAGE(msg) {failures++; printf("FAILURE: " msg " at " __FILE__ "(%d)\n", __LINE__); return 1;} +#define TEST_FAIL_WITH_MESSAGE(msg) \ +{ \ + if (failures == 0) \ + { \ + first_fail_file = __FILE__; \ + first_fail_line = __LINE__; \ + } \ + failures++; \ + printf("FAILURE: " msg " at " __FILE__ "(%d)\n", __LINE__); \ +} +#define TEST_ABORT_WITH_MESSAGE(msg) {TEST_FAIL_WITH_MESSAGE(msg); return 1;} + #define TEST_THAT(condition) {if(!(condition)) TEST_FAIL_WITH_MESSAGE("Condition [" #condition "] failed")} #define TEST_THAT_ABORTONFAIL(condition) {if(!(condition)) TEST_ABORT_WITH_MESSAGE("Condition [" #condition "] failed")} -// NOTE: The 0- bit it to allow this to work with stuff which has negative constants for flags (eg ConnectionException) +// NOTE: The 0- bit is to allow this to work with stuff which has negative constants for flags (eg ConnectionException) #define TEST_CHECK_THROWS(statement, excepttype, subtype) \ { \ bool didthrow = false; \ From boxbackup-dev at fluffy.co.uk Sat Mar 10 15:20:22 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 15:20:22 +0000 Subject: [Box Backup-commit] COMMIT r1385 - box/chris/general/test/bbackupd Message-ID: Author: chris Date: 2007-03-10 15:20:22 +0000 (Sat, 10 Mar 2007) New Revision: 1385 Modified: box/chris/general/test/bbackupd/testbbackupd.cpp Log: Check for return code 3 (unreadable files) instead of return code 2 Modified: box/chris/general/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-08 23:09:08 UTC (rev 1384) +++ box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-10 15:20:22 UTC (rev 1385) @@ -1720,7 +1720,7 @@ compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3e.log \"compare -ac\" quit"); // should find differences - TEST_RETURN(compareReturnValue, 2); + TEST_RETURN(compareReturnValue, 3); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // Check that it was reported correctly From boxbackup-dev at fluffy.co.uk Sat Mar 10 15:31:25 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 15:31:25 +0000 Subject: [Box Backup-commit] COMMIT r1386 - box/chris/merge/test/bbackupd Message-ID: Author: chris Date: 2007-03-10 15:31:25 +0000 (Sat, 10 Mar 2007) New Revision: 1386 Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp Log: Check for return code 3 (unreadable files) instead of return code 2 (refs #3, depends on [1378]) Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-10 15:20:22 UTC (rev 1385) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-10 15:31:25 UTC (rev 1386) @@ -1131,14 +1131,23 @@ TEST_THAT(fd != -1); ::close(fd); } + // Wait and test... wait_for_backup_operation(); compareReturnValue = ::system("../../bin/bbackupquery/bbackupquery -q -c testfiles/bbackupd.conf -l testfiles/query3e.log \"compare -ac\" quit"); - TEST_THAT(compareReturnValue == 2*256); // should find differences + + // Check that unreadable files were found + TEST_THAT(compareReturnValue == 3*256); + + // Check for memory leaks during compare TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + // Check that it was reported correctly TEST_THAT(TestFileExists("testfiles/notifyran.read-error.1")); + + // Check that the error was only reorted once TEST_THAT(!TestFileExists("testfiles/notifyran.read-error.2")); + // Set permissions on file and dir to stop errors in the future ::chmod("testfiles/TestDir1/sub23/read-fail-test-dir", 0770); ::chmod("testfiles/TestDir1/read-fail-test-file", 0770); From boxbackup-dev at fluffy.co.uk Sat Mar 10 16:12:02 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 16:12:02 +0000 Subject: [Box Backup-commit] COMMIT r1387 - in box/chris/merge: lib/common test/backupstore test/backupstorefix test/backupstorepatch test/basicserver test/bbackupd Message-ID: Author: chris Date: 2007-03-10 16:12:01 +0000 (Sat, 10 Mar 2007) New Revision: 1387 Added: box/chris/merge/lib/common/ServerControl.h Modified: box/chris/merge/lib/common/Test.h box/chris/merge/test/backupstore/testbackupstore.cpp box/chris/merge/test/backupstorefix/testbackupstorefix.cpp box/chris/merge/test/backupstorepatch/testbackupstorepatch.cpp box/chris/merge/test/basicserver/testbasicserver.cpp box/chris/merge/test/bbackupd/testbbackupd.cpp Log: Moved SendCommands(), HUPServer(), KillServer() to lib/server/ServerCommands.h. All of these use lib/server/WinNamedPipeStream on Win32, so they don't belong in lib/common. Made LaunchServer() work on Win32. Added constants for paths to executables, for use in tests, removing the need for #ifdefs and clumsy DIRECTORY_SEPARATORs in paths. Added terminate_bbackupd() and wait_for_operation() functions. Update unit tests to #include "ServerControl.h" if they need it. (refs #3) Copied: box/chris/merge/lib/common/ServerControl.h (from rev 1372, box/chris/general/lib/server/ServerControl.h) =================================================================== --- box/chris/merge/lib/common/ServerControl.h (rev 0) +++ box/chris/merge/lib/common/ServerControl.h 2007-03-10 16:12:01 UTC (rev 1387) @@ -0,0 +1,177 @@ +#ifndef SERVER_CONTROL_H +#define SERVER_CONTROL_H + +#ifdef WIN32 + +#include "WinNamedPipeStream.h" +#include "IOStreamGetLine.h" +#include "BoxPortsAndFiles.h" +#include "Test.h" + +static bool SendCommands(const std::string& rCmd) +{ + WinNamedPipeStream connection; + + try + { + connection.Connect(BOX_NAMED_PIPE_NAME); + } + catch(...) + { + printf("Failed to connect to daemon control socket.\n"); + return false; + } + + // For receiving data + IOStreamGetLine getLine(connection); + + // Wait for the configuration summary + std::string configSummary; + if(!getLine.GetLine(configSummary)) + { + printf("Failed to receive configuration summary from daemon\n"); + return false; + } + + // Was the connection rejected by the server? + if(getLine.IsEOF()) + { + printf("Server rejected the connection.\n"); + return false; + } + + // Decode it + int autoBackup, updateStoreInterval, minimumFileAge, maxUploadWait; + if(::sscanf(configSummary.c_str(), "bbackupd: %d %d %d %d", + &autoBackup, &updateStoreInterval, + &minimumFileAge, &maxUploadWait) != 4) + { + printf("Config summary didn't decode\n"); + return false; + } + + std::string cmds; + bool expectResponse; + + if (rCmd != "") + { + cmds = rCmd; + cmds += "\nquit\n"; + expectResponse = true; + } + else + { + cmds = "quit\n"; + expectResponse = false; + } + + connection.Write(cmds.c_str(), cmds.size()); + + // Read the response + std::string line; + bool statusOk = !expectResponse; + + while (expectResponse && !getLine.IsEOF() && getLine.GetLine(line)) + { + // Is this an OK or error line? + if (line == "ok") + { + statusOk = true; + } + else if (line == "error") + { + printf("ERROR (%s)\n", rCmd.c_str()); + break; + } + else + { + printf("WARNING: Unexpected response to command '%s': " + "%s", rCmd.c_str(), line.c_str()); + } + } + + return statusOk; +} + +inline bool HUPServer(int pid) +{ + return SendCommands("reload"); +} + +inline bool KillServerInternal(int pid) +{ + HANDLE hProcess = OpenProcess(PROCESS_TERMINATE, false, pid); + if (hProcess == NULL) + { + printf("Failed to open process %d: error %d\n", + pid, (int)GetLastError()); + return false; + } + + if (!TerminateProcess(hProcess, 1)) + { + printf("Failed to terminate process %d: error %d\n", + pid, (int)GetLastError()); + CloseHandle(hProcess); + return false; + } + + CloseHandle(hProcess); + return true; +} + +#else // !WIN32 + +inline bool HUPServer(int pid) +{ + if(pid == 0) return false; + return ::kill(pid, SIGHUP) == 0; +} + +inline bool KillServerInternal(int pid) +{ + if(pid == 0 || pid == -1) return false; + bool killed = (::kill(pid, SIGTERM) == 0); + TEST_THAT(killed); + return killed; +} + +#endif // WIN32 + +inline bool KillServer(int pid) +{ + if (!KillServerInternal(pid)) + { + return false; + } + + for (int i = 0; i < 30; i++) + { + if (!ServerIsAlive(pid)) break; + ::sleep(1); + if (!ServerIsAlive(pid)) break; + + if (i == 0) + { + printf("waiting for server to die"); + } + + printf("."); + fflush(stdout); + } + + if (!ServerIsAlive(pid)) + { + printf("done.\n"); + } + else + { + printf("failed!\n"); + } + + fflush(stdout); + + return !ServerIsAlive(pid); +} + +#endif // SERVER_CONTROL_H Modified: box/chris/merge/lib/common/Test.h =================================================================== --- box/chris/merge/lib/common/Test.h 2007-03-10 15:31:25 UTC (rev 1386) +++ box/chris/merge/lib/common/Test.h 2007-03-10 16:12:01 UTC (rev 1387) @@ -92,7 +92,7 @@ return -1; } -inline int RunCommand(const char *pCommandLine) +inline std::string ConvertPaths(const char *pCommandLine) { #ifdef WIN32 // convert UNIX paths to native @@ -114,9 +114,39 @@ std::string command = pCommandLine; #endif - return ::system(command.c_str()); + return command; } +inline int RunCommand(const char *pCommandLine) +{ + return ::system(ConvertPaths(pCommandLine).c_str()); +} + +#ifdef WIN32 +#include +#endif + +inline bool ServerIsAlive(int pid) +{ +#ifdef WIN32 + HANDLE hProcess = OpenProcess(PROCESS_QUERY_INFORMATION, false, pid); + if (hProcess == NULL) + { + if (GetLastError() != ERROR_INVALID_PARAMETER) + { + printf("Failed to open process %d: error %d\n", + pid, (int)GetLastError()); + } + return false; + } + CloseHandle(hProcess); + return true; +#else // !WIN32 + if(pid == 0) return false; + return ::kill(pid, 0) != -1; +#endif // WIN32 +} + inline int ReadPidFile(const char *pidFile) { if(!TestFileExists(pidFile)) @@ -141,192 +171,136 @@ inline int LaunchServer(const char *pCommandLine, const char *pidFile) { - if(RunCommand(pCommandLine) != 0) - { - printf("Server: %s\n", pCommandLine); - TEST_FAIL_WITH_MESSAGE("Couldn't start server"); - return -1; - } - - // give it time to start up - ::sleep(1); - - // read pid file - int pid = ReadPidFile(pidFile); +#ifdef WIN32 - if(pid == -1) - { - // helps with debugging: - printf("Server: %s (pidfile %s)\n", pCommandLine, pidFile); - } + PROCESS_INFORMATION procInfo; - return pid; -} + STARTUPINFO startInfo; + startInfo.cb = sizeof(startInfo); + startInfo.lpReserved = NULL; + startInfo.lpDesktop = NULL; + startInfo.lpTitle = NULL; + startInfo.dwFlags = 0; + startInfo.cbReserved2 = 0; + startInfo.lpReserved2 = NULL; -#ifdef WIN32 + std::string cmd = ConvertPaths(pCommandLine); + CHAR* tempCmd = strdup(cmd.c_str()); -#include "WinNamedPipeStream.h" -#include "IOStreamGetLine.h" -#include "BoxPortsAndFiles.h" + DWORD result = CreateProcess + ( + NULL, // lpApplicationName, naughty! + tempCmd, // lpCommandLine + NULL, // lpProcessAttributes + NULL, // lpThreadAttributes + false, // bInheritHandles + 0, // dwCreationFlags + NULL, // lpEnvironment + NULL, // lpCurrentDirectory + &startInfo, // lpStartupInfo + &procInfo // lpProcessInformation + ); -bool SendCommands(const std::string& rCmd) -{ - WinNamedPipeStream connection; + free(tempCmd); - try + if (result == 0) { - connection.Connect(BOX_NAMED_PIPE_NAME); + DWORD err = GetLastError(); + printf("Launch failed: %s: error %d\n", pCommandLine, (int)err); + return -1; } - catch(...) - { - printf("Failed to connect to daemon control socket.\n"); - return false; - } - // For receiving data - IOStreamGetLine getLine(connection); - - // Wait for the configuration summary - std::string configSummary; - if(!getLine.GetLine(configSummary)) - { - printf("Failed to receive configuration summary from daemon\n"); - return false; - } + CloseHandle(procInfo.hProcess); + CloseHandle(procInfo.hThread); - // Was the connection rejected by the server? - if(getLine.IsEOF()) - { - printf("Server rejected the connection.\n"); - return false; - } +#else // !WIN32 - // Decode it - int autoBackup, updateStoreInterval, minimumFileAge, maxUploadWait; - if(::sscanf(configSummary.c_str(), "bbackupd: %d %d %d %d", - &autoBackup, &updateStoreInterval, - &minimumFileAge, &maxUploadWait) != 4) + if(RunCommand(pCommandLine) != 0) { - printf("Config summary didn't decode\n"); - return false; + printf("Server: %s\n", pCommandLine); + TEST_FAIL_WITH_MESSAGE("Couldn't start server"); + return -1; } - std::string cmds; - bool expectResponse; +#endif // WIN32 - if (rCmd != "") + #ifdef WIN32 + // on other platforms there is no other way to get + // the PID, so a NULL pidFile doesn't make sense. + + if (pidFile == NULL) { - cmds = rCmd; - cmds += "\nquit\n"; - expectResponse = true; + return (int)procInfo.dwProcessId; } - else - { - cmds = "quit\n"; - expectResponse = false; - } - - connection.Write(cmds.c_str(), cmds.size()); - - // Read the response - std::string line; - bool statusOk = !expectResponse; + #endif - while (expectResponse && !getLine.IsEOF() && getLine.GetLine(line)) + // time for it to start up + ::fprintf(stdout, "Starting server: %s\n", pCommandLine); + ::fprintf(stdout, "Waiting for server to start: "); + + for (int i = 0; i < 15; i++) { - // Is this an OK or error line? - if (line == "ok") + if (TestFileExists(pidFile)) { - statusOk = true; + break; } - else if (line == "error") + + #ifdef WIN32 + if (!ServerIsAlive((int)procInfo.dwProcessId)) { - printf("ERROR (%s)\n", rCmd.c_str()); break; } - else - { - printf("WARNING: Unexpected response to command '%s': " - "%s", rCmd.c_str(), line.c_str()); - } + #endif + + ::fprintf(stdout, "."); + ::fflush(stdout); + ::sleep(1); } - - return statusOk; -} -inline bool ServerIsAlive(int pid) -{ - return SendCommands(""); -} + #ifdef WIN32 + // on Win32 we can check whether the process is alive + // without even checking the PID file -inline bool HUPServer(int pid) -{ - return SendCommands("reload"); -} - -inline bool KillServerInternal(int pid) -{ - bool sent = SendCommands("terminate"); - TEST_THAT(sent); - return sent; -} - -#else // !WIN32 - -inline bool ServerIsAlive(int pid) -{ - if(pid == 0) return false; - return ::kill(pid, 0) != -1; -} - -inline bool HUPServer(int pid) -{ - if(pid == 0) return false; - return ::kill(pid, SIGHUP) != -1; -} - -inline bool KillServerInternal(int pid) -{ - if(pid == 0 || pid == -1) return false; - bool killed = (::kill(pid, SIGTERM) == 0); - TEST_THAT(killed); - return killed; -} - -#endif // WIN32 - -inline bool KillServer(int pid) -{ - if (!KillServerInternal(pid)) + if (!ServerIsAlive((int)procInfo.dwProcessId)) { - return false; + ::fprintf(stdout, "server died!\n"); + TEST_FAIL_WITH_MESSAGE("Server died!"); + return -1; } + #endif - for (int i = 0; i < 30; i++) + if (!TestFileExists(pidFile)) { - if (!ServerIsAlive(pid)) break; - ::sleep(1); - if (!ServerIsAlive(pid)) break; - - if (i == 0) - { - printf("waiting for server to die"); - } - printf("."); - fflush(stdout); + ::fprintf(stdout, "timed out!\n"); + TEST_FAIL_WITH_MESSAGE("Server didn't save PID file"); + return -1; } - - if (!ServerIsAlive(pid)) + else { - printf("done.\n"); + ::fprintf(stdout, "done.\n"); } - else + + // wait a second for the pid to be written to the file + ::sleep(1); + + // read pid file + int pid = ReadPidFile(pidFile); + + #ifdef WIN32 + // On Win32 we can check whether the PID in the pidFile matches + // the one returned by the system, which it always should. + + if (pid != (int)procInfo.dwProcessId) { - printf("failed!\n"); + printf("Server wrote wrong pid to file (%s): expected %d " + "but found %d\n", pidFile, + (int)procInfo.dwProcessId, pid); + TEST_FAIL_WITH_MESSAGE("Server wrote wrong pid to file"); + return -1; } - fflush(stdout); + #endif - return !ServerIsAlive(pid); + return pid; } inline void TestRemoteProcessMemLeaks(const char *filename) @@ -361,4 +335,53 @@ #endif } +#ifdef WIN32 +#define BBACKUPCTL "..\\..\\bin\\bbackupctl\\bbackupctl.exe" +#define BBACKUPD "..\\..\\bin\\bbackupd\\bbackupd.exe" +#define BBSTORED "..\\..\\bin\\bbstored\\bbstored.exe" +#define BBACKUPQUERY "..\\..\\bin\\bbackupquery\\bbackupquery.exe" +#define BBSTOREACCOUNTS "..\\..\\bin\\bbstoreaccounts\\bbstoreaccounts.exe" +#define TEST_RETURN(actual, expected) TEST_THAT(actual == expected); +#else +#define BBACKUPCTL "../../bin/bbackupctl/bbackupctl" +#define BBACKUPD "../../bin/bbackupd/bbackupd" +#define BBSTORED "../../bin/bbackupd/bbstored" +#define BBACKUPQUERY "../../bin/bbackupquery/bbackupquery" +#define BBSTOREACCOUNTS "../../bin/bbstoreaccounts/bbstoreaccounts" +#define TEST_RETURN(actual, expected) TEST_THAT(actual == expected*256); +#endif + +inline void terminate_bbackupd(int pid) +{ + TEST_THAT(::system(BBACKUPCTL " -q -c testfiles/bbackupd.conf " + "terminate") == 0); + TestRemoteProcessMemLeaks("bbackupctl.memleaks"); + + for (int i = 0; i < 20; i++) + { + if (!ServerIsAlive(pid)) break; + fprintf(stdout, "."); + fflush(stdout); + sleep(1); + } + + TEST_THAT(!ServerIsAlive(pid)); + TestRemoteProcessMemLeaks("bbackupd.memleaks"); +} + + +// Wait a given number of seconds for something to complete +inline void wait_for_operation(int seconds) +{ + printf("waiting: "); + fflush(stdout); + for(int l = 0; l < seconds; ++l) + { + sleep(1); + printf("."); + fflush(stdout); + } + printf("\n"); +} + #endif // TEST__H Modified: box/chris/merge/test/backupstore/testbackupstore.cpp =================================================================== --- box/chris/merge/test/backupstore/testbackupstore.cpp 2007-03-10 15:31:25 UTC (rev 1386) +++ box/chris/merge/test/backupstore/testbackupstore.cpp 2007-03-10 16:12:01 UTC (rev 1387) @@ -33,6 +33,7 @@ #include "MemBlockStream.h" #include "BackupClientFileAttributes.h" #include "BackupClientCryptoKeys.h" +#include "ServerControl.h" #include "MemLeakFindOn.h" Modified: box/chris/merge/test/backupstorefix/testbackupstorefix.cpp =================================================================== --- box/chris/merge/test/backupstorefix/testbackupstorefix.cpp 2007-03-10 15:31:25 UTC (rev 1386) +++ box/chris/merge/test/backupstorefix/testbackupstorefix.cpp 2007-03-10 16:12:01 UTC (rev 1387) @@ -29,6 +29,7 @@ #include "RaidFileException.h" #include "StoreStructure.h" #include "BackupStoreFileWire.h" +#include "ServerControl.h" #include "MemLeakFindOn.h" @@ -68,20 +69,6 @@ ::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf check 01234567"); \ ::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf check 01234567 fix"); -// Wait a given number of seconds for something to complete -void wait_for_operation(int seconds) -{ - printf("waiting: "); - fflush(stdout); - for(int l = 0; l < seconds; ++l) - { - sleep(1); - printf("."); - fflush(stdout); - } - printf("\n"); -} - // Get ID of an object given a filename int32_t getID(const char *name) { Modified: box/chris/merge/test/backupstorepatch/testbackupstorepatch.cpp =================================================================== --- box/chris/merge/test/backupstorepatch/testbackupstorepatch.cpp 2007-03-10 15:31:25 UTC (rev 1386) +++ box/chris/merge/test/backupstorepatch/testbackupstorepatch.cpp 2007-03-10 16:12:01 UTC (rev 1387) @@ -35,6 +35,7 @@ #include "MemBlockStream.h" #include "BackupClientFileAttributes.h" #include "BackupClientCryptoKeys.h" +#include "ServerControl.h" #include "MemLeakFindOn.h" Modified: box/chris/merge/test/basicserver/testbasicserver.cpp =================================================================== --- box/chris/merge/test/basicserver/testbasicserver.cpp 2007-03-10 15:31:25 UTC (rev 1386) +++ box/chris/merge/test/basicserver/testbasicserver.cpp 2007-03-10 16:12:01 UTC (rev 1387) @@ -28,6 +28,7 @@ #include "TestContext.h" #include "autogen_TestProtocolClient.h" #include "autogen_TestProtocolServer.h" +#include "ServerControl.h" #include "MemLeakFindOn.h" Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-10 15:31:25 UTC (rev 1386) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-10 16:12:01 UTC (rev 1387) @@ -51,6 +51,7 @@ #include "FileStream.h" #include "IOStreamGetLine.h" #include "intercept.h" +#include "ServerControl.h" #include "MemLeakFindOn.h" From boxbackup-dev at fluffy.co.uk Sat Mar 10 16:23:14 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 16:23:14 +0000 Subject: [Box Backup-commit] COMMIT r1388 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-10 16:23:12 +0000 (Sat, 10 Mar 2007) New Revision: 1388 Modified: box/chris/merge/lib/common/Test.h Log: Report file and line of memory leak test failures (refs #3, merges [714]) Modified: box/chris/merge/lib/common/Test.h =================================================================== --- box/chris/merge/lib/common/Test.h 2007-03-10 16:12:01 UTC (rev 1387) +++ box/chris/merge/lib/common/Test.h 2007-03-10 16:23:12 UTC (rev 1388) @@ -303,12 +303,21 @@ return pid; } -inline void TestRemoteProcessMemLeaks(const char *filename) +#define TestRemoteProcessMemLeaks(filename) \ + TestRemoteProcessMemLeaksFunc(filename, __FILE__, __LINE__) + +inline void TestRemoteProcessMemLeaksFunc(const char *filename, + const char* file, int line) { #ifdef BOX_MEMORY_LEAK_TESTING // Does the file exist? if(!TestFileExists(filename)) { + if (failures == 0) + { + first_fail_file = file; + first_fail_line = line; + } ++failures; printf("FAILURE: MemLeak report not available (file %s)\n", filename); } @@ -317,8 +326,14 @@ // Is it empty? if(TestGetFileSize(filename) > 0) { + if (failures == 0) + { + first_fail_file = file; + first_fail_line = line; + } ++failures; - printf("FAILURE: Memory leaks found in other process (file %s)\n==========\n", filename); + printf("FAILURE: Memory leaks found in other process " + "(file %s)\n==========\n", filename); FILE *f = fopen(filename, "r"); char line[512]; while(::fgets(line, sizeof(line), f) != 0) From boxbackup-dev at fluffy.co.uk Sat Mar 10 16:49:34 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 16:49:34 +0000 Subject: [Box Backup-commit] COMMIT r1389 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-10 16:49:34 +0000 (Sat, 10 Mar 2007) New Revision: 1389 Modified: box/chris/merge/lib/common/Timer.cpp Log: Throw an assertion error if a NULL timer is added (refs #3, merges [1367]) Modified: box/chris/merge/lib/common/Timer.cpp =================================================================== --- box/chris/merge/lib/common/Timer.cpp 2007-03-10 16:23:12 UTC (rev 1388) +++ box/chris/merge/lib/common/Timer.cpp 2007-03-10 16:49:34 UTC (rev 1389) @@ -90,6 +90,7 @@ void Timers::Add(Timer& rTimer) { ASSERT(spTimers); + ASSERT(&rTimer); spTimers->push_back(&rTimer); Reschedule(); } @@ -106,11 +107,13 @@ void Timers::Remove(Timer& rTimer) { ASSERT(spTimers); + ASSERT(&rTimer); bool restart = true; while (restart) { restart = false; + for (std::vector::iterator i = spTimers->begin(); i != spTimers->end(); i++) { From boxbackup-dev at fluffy.co.uk Sat Mar 10 16:53:11 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 16:53:11 +0000 Subject: [Box Backup-commit] COMMIT r1390 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-10 16:53:11 +0000 (Sat, 10 Mar 2007) New Revision: 1390 Added: box/chris/general/lib/common/Timer.h Log: Added timer header file Copied: box/chris/general/lib/common/Timer.h (from rev 1385, box/chris/merge/lib/common/Timer.h) =================================================================== --- box/chris/general/lib/common/Timer.h (rev 0) +++ box/chris/general/lib/common/Timer.h 2007-03-10 16:53:11 UTC (rev 1390) @@ -0,0 +1,84 @@ +// -------------------------------------------------------------------------- +// +// File +// Name: Timer.h +// Purpose: Generic timers which execute arbitrary code when +// they expire. +// Created: 5/11/2006 +// +// -------------------------------------------------------------------------- + +#ifndef TIMER__H +#define TIMER__H + +#include + +#include + +#include "MemLeakFindOn.h" +#include "BoxTime.h" + +class Timer; + +// -------------------------------------------------------------------------- +// +// Class +// Name: Timers +// Purpose: Static class to manage all timers and arrange +// efficient delivery of wakeup signals +// Created: 19/3/04 +// +// -------------------------------------------------------------------------- +class Timers +{ + private: + static std::vector* spTimers; + static void Reschedule(); + + static bool sRescheduleNeeded; + static void SignalHandler(int iUnused); + + public: + static void Init(); + static void Cleanup(); + static void Add (Timer& rTimer); + static void Remove(Timer& rTimer); + static void RequestReschedule() + { + sRescheduleNeeded = true; + } + + static void RescheduleIfNeeded() + { + if (sRescheduleNeeded) + { + Reschedule(); + } + } +}; + +class Timer +{ +public: + Timer(size_t timeoutSecs); + virtual ~Timer(); + Timer(const Timer &); + Timer &operator=(const Timer &); + +public: + box_time_t GetExpiryTime() { return mExpires; } + virtual void OnExpire(); + bool HasExpired() + { + Timers::RescheduleIfNeeded(); + return mExpired; + } + +private: + box_time_t mExpires; + bool mExpired; +}; + +#include "MemLeakFindOff.h" + +#endif // TIMER__H From boxbackup-dev at fluffy.co.uk Sat Mar 10 16:53:56 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 16:53:56 +0000 Subject: [Box Backup-commit] COMMIT r1391 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-10 16:53:56 +0000 (Sat, 10 Mar 2007) New Revision: 1391 Modified: box/chris/merge/lib/common/Timer.cpp Log: Win32 compile fixes (no gettimeofday(), no signal()) (refs #3) Modified: box/chris/merge/lib/common/Timer.cpp =================================================================== --- box/chris/merge/lib/common/Timer.cpp 2007-03-10 16:53:11 UTC (rev 1390) +++ box/chris/merge/lib/common/Timer.cpp 2007-03-10 16:53:56 UTC (rev 1391) @@ -144,10 +144,13 @@ { THROW_EXCEPTION(CommonException, Internal) } + + #ifndef WIN32 if (::signal(SIGALRM, Timers::SignalHandler) != Timers::SignalHandler) { THROW_EXCEPTION(CommonException, Internal) } + #endif // Clear the reschedule-needed flag to false before we start. // If a timer event occurs while we are scheduling, then we @@ -255,7 +258,7 @@ : mExpires(GetCurrentBoxTime() + SecondsToBoxTime(timeoutSecs)), mExpired(false) { - #ifndef NDEBUG + #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); if (timeoutSecs == 0) @@ -285,7 +288,7 @@ Timer::~Timer() { - #ifndef NDEBUG + #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); TRACE3("%d.%d: timer %p destroyed, will not fire\n", @@ -299,7 +302,7 @@ : mExpires(rToCopy.mExpires), mExpired(rToCopy.mExpired) { - #ifndef NDEBUG + #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); if (mExpired) @@ -331,7 +334,7 @@ Timer& Timer::operator=(const Timer& rToCopy) { - #ifndef NDEBUG + #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); if (rToCopy.mExpired) @@ -367,7 +370,7 @@ void Timer::OnExpire() { - #ifndef NDEBUG + #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); TRACE3("%d.%d: timer %p fired\n", tv.tv_sec, tv.tv_usec, this); From boxbackup-dev at fluffy.co.uk Sat Mar 10 16:59:30 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 16:59:30 +0000 Subject: [Box Backup-commit] COMMIT r1392 - box/chris/merge/lib/raidfile Message-ID: Author: chris Date: 2007-03-10 16:59:30 +0000 (Sat, 10 Mar 2007) New Revision: 1392 Modified: box/chris/merge/lib/raidfile/RaidFileRead.cpp Log: Code formatting (cosmetic) (refs #3, merges [1345]) Modified: box/chris/merge/lib/raidfile/RaidFileRead.cpp =================================================================== --- box/chris/merge/lib/raidfile/RaidFileRead.cpp 2007-03-10 16:53:56 UTC (rev 1391) +++ box/chris/merge/lib/raidfile/RaidFileRead.cpp 2007-03-10 16:59:30 UTC (rev 1392) @@ -9,13 +9,14 @@ #include "Box.h" -#include -#include +#include #include -#include -#include #include +#include +#include +#include + #ifdef HAVE_SYS_UIO_H #include #endif @@ -41,7 +42,7 @@ #include "MemLeakFindOn.h" #define READ_NUMBER_DISCS_REQUIRED 3 -#define READV_MAX_BLOCKS 64 +#define READV_MAX_BLOCKS 64 // We want to use POSIX fstat() for now, not the emulated one #undef fstat From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:00:17 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:00:17 +0000 Subject: [Box Backup-commit] COMMIT r1393 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-10 17:00:17 +0000 (Sat, 10 Mar 2007) New Revision: 1393 Modified: box/chris/general/lib/server/ServerStream.h Log: Always implement OnIdle() Modified: box/chris/general/lib/server/ServerStream.h =================================================================== --- box/chris/general/lib/server/ServerStream.h 2007-03-10 16:59:30 UTC (rev 1392) +++ box/chris/general/lib/server/ServerStream.h 2007-03-10 17:00:17 UTC (rev 1393) @@ -56,9 +56,7 @@ return "generic-stream-server"; } - #ifdef WIN32 virtual void OnIdle() { } - #endif virtual void Run() { From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:01:42 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:01:42 +0000 Subject: [Box Backup-commit] COMMIT r1394 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-10 17:01:42 +0000 (Sat, 10 Mar 2007) New Revision: 1394 Modified: box/chris/general/lib/server/ServerStream.h Log: Merge back changes from merge tree (call OnIdle() when idle on all platforms, not just Win32) Modified: box/chris/general/lib/server/ServerStream.h =================================================================== --- box/chris/general/lib/server/ServerStream.h 2007-03-10 17:00:17 UTC (rev 1393) +++ box/chris/general/lib/server/ServerStream.h 2007-03-10 17:01:42 UTC (rev 1394) @@ -269,9 +269,9 @@ } } - #ifdef WIN32 OnIdle(); - #else // !WIN32 + + #ifndef WIN32 // Clean up child processes (if forking daemon) if(ForkToHandleRequests) { @@ -286,7 +286,7 @@ } } while(p > 0); } - #endif // WIN32 + #endif // !WIN32 } } catch(...) From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:03:13 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:03:13 +0000 Subject: [Box Backup-commit] COMMIT r1395 - box/chris/merge/lib/server Message-ID: Author: chris Date: 2007-03-10 17:03:13 +0000 (Sat, 10 Mar 2007) New Revision: 1395 Modified: box/chris/merge/lib/server/WinNamedPipeStream.cpp Log: Catch exceptions thrown by closing the WinNamedPipeStream during shutdown and suppress them. (refs #3, merges [1284]) Modified: box/chris/merge/lib/server/WinNamedPipeStream.cpp =================================================================== --- box/chris/merge/lib/server/WinNamedPipeStream.cpp 2007-03-10 17:01:42 UTC (rev 1394) +++ box/chris/merge/lib/server/WinNamedPipeStream.cpp 2007-03-10 17:03:13 UTC (rev 1395) @@ -57,7 +57,15 @@ { if (mSocketHandle != INVALID_HANDLE_VALUE) { - Close(); + try + { + Close(); + } + catch (std::exception &e) + { + ::syslog(LOG_ERR, "Caught exception while destroying " + "named pipe, ignored."); + } } } From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:06:13 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:06:13 +0000 Subject: [Box Backup-commit] COMMIT r1396 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 17:06:13 +0000 (Sat, 10 Mar 2007) New Revision: 1396 Modified: box/chris/merge/lib/win32/emu.cpp Log: Always include process.h, as we don't know whether it was detected or not (we don't have access to lib/common/BoxConfig.h in lib/win32) (refs #3) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:03:13 UTC (rev 1395) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:06:13 UTC (rev 1396) @@ -9,14 +9,12 @@ #include #include +#include #include #ifdef HAVE_UNISTD_H #include #endif -#ifdef HAVE_PROCESS_H - #include -#endif #include #include From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:12:01 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:12:01 +0000 Subject: [Box Backup-commit] COMMIT r1397 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 17:12:01 +0000 (Sat, 10 Mar 2007) New Revision: 1397 Modified: box/chris/merge/lib/win32/emu.cpp Log: Our timer code only supports ITIMER_REAL (refs #3) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:06:13 UTC (rev 1396) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:12:01 UTC (rev 1397) @@ -44,7 +44,14 @@ { ASSERT(gTimerInitialised); + if (ITIMER_REAL != type) + { + errno = ENOSYS; + return -1; + } + EnterCriticalSection(&gLock); + // we only need seconds for the mo! if (timeout->it_value.tv_sec == 0 && timeout->it_value.tv_usec == 0) @@ -58,6 +65,7 @@ ourTimer.interval = timeout->it_interval.tv_sec; gTimerList.push_back(ourTimer); } + LeaveCriticalSection(&gLock); // indicate success From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:15:04 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:15:04 +0000 Subject: [Box Backup-commit] COMMIT r1398 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 17:15:03 +0000 (Sat, 10 Mar 2007) New Revision: 1398 Modified: box/chris/merge/lib/win32/emu.cpp Log: We can't use lib/common here, so we don't have ASSERT() (refs #3) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:12:01 UTC (rev 1397) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:15:03 UTC (rev 1398) @@ -42,7 +42,7 @@ int setitimer(int type, struct itimerval *timeout, void *arg) { - ASSERT(gTimerInitialised); + assert(gTimerInitialised); if (ITIMER_REAL != type) { @@ -137,7 +137,8 @@ void InitTimer(void) { - ASSERT(!gTimerInitialised); + assert(!gTimerInitialised); + InitializeCriticalSection(&gLock); // create our thread @@ -151,7 +152,7 @@ void FiniTimer(void) { - ASSERT(gTimerInitialised); + assert(gTimerInitialised); gFinishTimer = true; EnterCriticalSection(&gLock); DeleteCriticalSection(&gLock); From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:20:40 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:20:40 +0000 Subject: [Box Backup-commit] COMMIT r1399 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 17:20:40 +0000 (Sat, 10 Mar 2007) New Revision: 1399 Modified: box/chris/merge/lib/win32/emu.cpp box/chris/merge/lib/win32/emu.h Log: Expanded character set conversion API to allow arbitrary conversions (needed to handle command lines with international encodings) (refs #3, merges [1038]) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:15:03 UTC (rev 1398) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:20:40 UTC (rev 1399) @@ -374,23 +374,25 @@ // -------------------------------------------------------------------------- // // Function -// Name: ConvertUtf8ToConsole -// Purpose: Converts a string from UTF-8 to the console -// code page. On success, replaces contents of rDest -// and returns true. In case of fire, logs the error -// and returns false. -// Created: 4th February 2006 +// Name: ConvertEncoding(const std::string&, int, +// std::string&, int) +// Purpose: Converts a string from one code page to another. +// On success, replaces contents of rDest and returns +// true. In case of fire, logs the error and returns +// false. +// Created: 15th October 2006 // // -------------------------------------------------------------------------- -bool ConvertUtf8ToConsole(const char* pString, std::string& rDest) +bool ConvertEncoding(const std::string& rSource, int sourceCodePage, + std::string& rDest, int destCodePage) { - WCHAR* pWide = ConvertUtf8ToWideString(pString); + WCHAR* pWide = ConvertToWideString(rSource.c_str(), sourceCodePage); if (pWide == NULL) { return false; } - char* pConsole = ConvertFromWideString(pWide, GetConsoleOutputCP()); + char* pConsole = ConvertFromWideString(pWide, destCodePage); delete [] pWide; if (!pConsole) @@ -404,40 +406,26 @@ return true; } -// -------------------------------------------------------------------------- -// -// Function -// Name: ConvertConsoleToUtf8 -// Purpose: Converts a string from the console code page -// to UTF-8. On success, replaces contents of rDest -// and returns true. In case of fire, logs the error -// and returns false. -// Created: 4th February 2006 -// -// -------------------------------------------------------------------------- -bool ConvertConsoleToUtf8(const char* pString, std::string& rDest) +bool ConvertToUtf8(const char* pString, std::string& rDest, int sourceCodePage) { - WCHAR* pWide = ConvertToWideString(pString, GetConsoleCP()); - if (pWide == NULL) - { - return false; - } + return ConvertEncoding(pString, sourceCodePage, rDest, CP_UTF8); +} - char* pConsole = ConvertFromWideString(pWide, CP_UTF8); - delete [] pWide; +bool ConvertFromUtf8(const char* pString, std::string& rDest, int destCodePage) +{ + return ConvertEncoding(pString, CP_UTF8, rDest, destCodePage); +} - if (!pConsole) - { - return false; - } +bool ConvertConsoleToUtf8(const char* pString, std::string& rDest) +{ + return ConvertEncoding(pString, GetConsoleCP(), rDest, CP_UTF8); +} - rDest = pConsole; - delete [] pConsole; - - return true; +bool ConvertUtf8ToConsole(const char* pString, std::string& rDest) +{ + return ConvertEncoding(pString, CP_UTF8, rDest, GetConsoleOutputCP()); } - // -------------------------------------------------------------------------- // // Function Modified: box/chris/merge/lib/win32/emu.h =================================================================== --- box/chris/merge/lib/win32/emu.h 2007-03-10 17:15:03 UTC (rev 1398) +++ box/chris/merge/lib/win32/emu.h 2007-03-10 17:20:40 UTC (rev 1399) @@ -254,7 +254,7 @@ int emu_mkdir(const char* pPathName); -inline int mkdir(const char *pPathName, mode_t mode) +inline int mkdir(const char *pPathName, mode_t mode = 0) { return emu_mkdir(pPathName); } @@ -398,6 +398,12 @@ int poll(struct pollfd *ufds, unsigned long nfds, int timeout); bool EnableBackupRights( void ); +bool ConvertEncoding (const std::string& rSource, int sourceCodePage, + std::string& rDest, int destCodePage); +bool ConvertToUtf8 (const std::string& rSource, std::string& rDest, + int sourceCodePage); +bool ConvertFromUtf8 (const std::string& rSource, std::string& rDest, + int destCodePage); bool ConvertUtf8ToConsole(const char* pString, std::string& rDest); bool ConvertConsoleToUtf8(const char* pString, std::string& rDest); From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:27:31 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:27:31 +0000 Subject: [Box Backup-commit] COMMIT r1400 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 17:27:31 +0000 (Sat, 10 Mar 2007) New Revision: 1400 Modified: box/chris/merge/lib/win32/emu.cpp Log: Print localised error messages rather than error codes for all errors (refs #3, merges [1046]) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:20:40 UTC (rev 1399) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:27:31 UTC (rev 1400) @@ -482,6 +482,32 @@ return tmpStr; } +std::string GetErrorMessage(DWORD errorCode) +{ + char* pMsgBuf = NULL; + + DWORD chars = FormatMessage + ( + FORMAT_MESSAGE_ALLOCATE_BUFFER | + FORMAT_MESSAGE_FROM_SYSTEM, + NULL, + errorCode, + MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), + (char *)(&pMsgBuf), + 0, NULL + ); + + if (chars == 0 || pMsgBuf == NULL) + { + return std::string("failed to get error message"); + } + + std::string out(pMsgBuf); + LocalFree(pMsgBuf); + + return out; +} + // -------------------------------------------------------------------------- // // Function @@ -560,8 +586,9 @@ if (hdir == INVALID_HANDLE_VALUE) { - ::syslog(LOG_WARNING, "Failed to open file %s: " - "error %i", pFileName, GetLastError()); + ::syslog(LOG_WARNING, "Failed to open file '%s': " + "%s", pFileName, + GetErrorMessage(GetLastError()).c_str()); return INVALID_HANDLE_VALUE; } @@ -589,7 +616,7 @@ if (!GetFileInformationByHandle(hdir, &fi)) { ::syslog(LOG_WARNING, "Failed to read file information: " - "error %d", GetLastError()); + "%s", GetErrorMessage(GetLastError()).c_str()); errno = EACCES; return -1; } @@ -597,7 +624,7 @@ if (INVALID_FILE_ATTRIBUTES == fi.dwFileAttributes) { ::syslog(LOG_WARNING, "Failed to get file attributes: " - "error %d", GetLastError()); + "%s", GetErrorMessage(GetLastError()).c_str()); errno = EACCES; return -1; } @@ -628,7 +655,7 @@ if (!GetFileSizeEx(hdir, &st_size)) { ::syslog(LOG_WARNING, "Failed to get file size: " - "error %d", GetLastError()); + "%s", GetErrorMessage(GetLastError()).c_str()); errno = EACCES; return -1; } @@ -751,8 +778,9 @@ } else { - ::syslog(LOG_WARNING, - "Failed to open '%s': error %d", pFileName, err); + ::syslog(LOG_WARNING, "Failed to open '%s': " + "%s", pFileName, + GetErrorMessage(err).c_str()); errno = EACCES; } @@ -820,7 +848,8 @@ if (!GetFileInformationByHandle(handle, &fi)) { ::syslog(LOG_WARNING, "Failed to get file information " - "for '%s': error %d", pName, GetLastError()); + "for '%s': %s", pName, + GetErrorMessage(GetLastError()).c_str()); CloseHandle(handle); errno = EACCES; return -1; @@ -873,8 +902,8 @@ if (!SetFileTime(handle, &creationTime, NULL, &modificationTime)) { - ::syslog(LOG_ERR, "Failed to set times on '%s': error %d", - pName, GetLastError()); + ::syslog(LOG_ERR, "Failed to set times on '%s': %s", pName, + GetErrorMessage(GetLastError()).c_str()); CloseHandle(handle); return 1; } @@ -916,8 +945,8 @@ DWORD attribs = GetFileAttributesW(pBuffer); if (attribs == INVALID_FILE_ATTRIBUTES) { - ::syslog(LOG_ERR, "Failed to get file attributes of '%s': " - "error %d", pName, GetLastError()); + ::syslog(LOG_ERR, "Failed to get file attributes of '%s': %s", + pName, GetErrorMessage(GetLastError()).c_str()); errno = EACCES; free(pBuffer); return -1; @@ -934,8 +963,8 @@ if (!SetFileAttributesW(pBuffer, attribs)) { - ::syslog(LOG_ERR, "Failed to set file attributes of '%s': " - "error %d", pName, GetLastError()); + ::syslog(LOG_ERR, "Failed to set file attributes of '%s': %s", + pName, GetErrorMessage(GetLastError()).c_str()); errno = EACCES; free(pBuffer); return -1; @@ -1204,8 +1233,8 @@ char cmd[MAX_PATH]; if (GetModuleFileName(NULL, cmd, sizeof(cmd)-1) == 0) { - ::syslog(LOG_ERR, "Failed to get the program file name: " - "error %d", GetLastError()); + ::syslog(LOG_ERR, "Failed to get the program file name: %s", + GetErrorMessage(GetLastError()).c_str()); return FALSE; } cmd[sizeof(cmd)-1] = 0; @@ -1224,8 +1253,8 @@ 0, NULL, REG_OPTION_NON_VOLATILE, KEY_WRITE, NULL, &hk, &dwDisp)) { - ::syslog(LOG_ERR, "Failed to create the registry key: " - "error %d", GetLastError()); + ::syslog(LOG_ERR, "Failed to create the registry key: %s", + GetErrorMessage(GetLastError()).c_str()); return FALSE; } @@ -1238,8 +1267,8 @@ (LPBYTE) exepath.c_str(), // pointer to value data (DWORD) (exepath.size()))) // data size { - ::syslog(LOG_ERR, "Failed to set the event message file: " - "error %d", GetLastError()); + ::syslog(LOG_ERR, "Failed to set the event message file: %s", + GetErrorMessage(GetLastError()).c_str()); RegCloseKey(hk); return FALSE; } @@ -1256,8 +1285,8 @@ (LPBYTE) &dwData, // pointer to value data sizeof(DWORD))) // length of value data { - ::syslog(LOG_ERR, "Failed to set the supported types: " - "error %d", GetLastError()); + ::syslog(LOG_ERR, "Failed to set the supported types: %s", + GetErrorMessage(GetLastError()).c_str()); RegCloseKey(hk); return FALSE; } @@ -1272,7 +1301,7 @@ (DWORD) (exepath.size()))) // data size { ::syslog(LOG_ERR, "Failed to set the category message file: " - "error %d", GetLastError()); + "%s", GetErrorMessage(GetLastError()).c_str()); RegCloseKey(hk); return FALSE; } @@ -1284,8 +1313,8 @@ (LPBYTE) &dwNum, // pointer to value data sizeof(DWORD))) // length of value data { - ::syslog(LOG_ERR, "Failed to set the category count: " - "error %d", GetLastError()); + ::syslog(LOG_ERR, "Failed to set the category count: %s", + GetErrorMessage(GetLastError()).c_str()); RegCloseKey(hk); return FALSE; } @@ -1318,7 +1347,7 @@ if (newSyslogH == NULL) { ::syslog(LOG_ERR, "Failed to register our own event source: " - "error %d", GetLastError()); + "%s", GetErrorMessage(GetLastError()).c_str()); return; } @@ -1417,8 +1446,8 @@ } else { - printf("Unable to send message to Event Log: " - "error %i:\r\n", (int)err); + printf("Unable to send message to Event Log: %s:\r\n", + GetErrorMessage(err).c_str()); fflush(stdout); } } @@ -1562,7 +1591,8 @@ else { ::syslog(LOG_WARNING, "Failed to delete file " - "'%s': error %d", pFileName, (int)err); + "'%s': %s", pFileName, + GetErrorMessage(err).c_str()); errno = ENOSYS; } return -1; @@ -1578,7 +1608,7 @@ if (hConsole == INVALID_HANDLE_VALUE) { ::fprintf(stderr, "Failed to get a handle on standard input: " - "error %d\n", GetLastError()); + "%s", GetErrorMessage(GetLastError()).c_str()); return -1; } @@ -1601,8 +1631,8 @@ NULL // reserved )) { - ::fprintf(stderr, "Failed to read from console: error %d\n", - GetLastError()); + ::fprintf(stderr, "Failed to read from console: %s\n", + GetErrorMessage(GetLastError()).c_str()); return -1; } @@ -1698,13 +1728,13 @@ // Convert the last-write time to local time. if (!SystemTimeToFileTime(&stUTC, pTo)) { - syslog(LOG_ERR, "Failed to convert between time formats: " - "error %d", GetLastError()); + syslog(LOG_ERR, "Failed to convert between time formats: %s", + GetErrorMessage(GetLastError()).c_str()); return false; } return true; } +#endif // WIN32 -#endif // WIN32 From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:29:31 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:29:31 +0000 Subject: [Box Backup-commit] COMMIT r1401 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 17:29:31 +0000 (Sat, 10 Mar 2007) New Revision: 1401 Modified: box/chris/merge/lib/win32/emu.cpp Log: Improve GetErrorMessage() by including the error number/code in the message (helps debugging on foreign langauge versions of Windows) (refs #3, merges [1364]) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:27:31 UTC (rev 1400) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:29:31 UTC (rev 1401) @@ -18,6 +18,7 @@ #include #include +#include // message resource definitions for syslog() @@ -502,10 +503,11 @@ return std::string("failed to get error message"); } - std::string out(pMsgBuf); + std::ostringstream line; + line << pMsgBuf << " (" << errorCode << ")"; LocalFree(pMsgBuf); - return out; + return line.str(); } // -------------------------------------------------------------------------- From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:32:44 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:32:44 +0000 Subject: [Box Backup-commit] COMMIT r1402 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 17:32:44 +0000 (Sat, 10 Mar 2007) New Revision: 1402 Modified: box/chris/merge/lib/win32/emu.cpp box/chris/merge/lib/win32/emu.h Log: Fix handling of O_EXCL to behave just like Unix, not abused to lock files. Add a new constant which specifies that files are to be locked open. (refs #3, merges [1288]) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:29:31 UTC (rev 1401) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:32:44 UTC (rev 1402) @@ -541,31 +541,38 @@ // flags could be O_WRONLY | O_CREAT | O_RDONLY DWORD createDisposition = OPEN_EXISTING; - DWORD shareMode = FILE_SHARE_READ; - DWORD accessRights = FILE_READ_ATTRIBUTES | FILE_LIST_DIRECTORY | FILE_READ_EA; + DWORD shareMode = FILE_SHARE_READ | FILE_SHARE_WRITE + | FILE_SHARE_DELETE; + DWORD accessRights = FILE_READ_ATTRIBUTES | FILE_LIST_DIRECTORY + | FILE_READ_EA; if (flags & O_WRONLY) { accessRights = FILE_WRITE_DATA; - shareMode = FILE_SHARE_WRITE; } else if (flags & O_RDWR) { accessRights |= FILE_WRITE_ATTRIBUTES | FILE_WRITE_DATA | FILE_WRITE_EA; - shareMode |= FILE_SHARE_WRITE; } if (flags & O_CREAT) { createDisposition = OPEN_ALWAYS; } + if (flags & O_TRUNC) { createDisposition = CREATE_ALWAYS; } - if (flags & O_EXCL) + + if ((flags & O_CREAT) && (flags & O_EXCL)) { + createDisposition = CREATE_NEW; + } + + if (flags & O_LOCK) + { shareMode = 0; } @@ -573,7 +580,6 @@ if (flags & O_TEMPORARY) { winFlags |= FILE_FLAG_DELETE_ON_CLOSE; - shareMode |= FILE_SHARE_DELETE; } HANDLE hdir = CreateFileW(pBuffer, Modified: box/chris/merge/lib/win32/emu.h =================================================================== --- box/chris/merge/lib/win32/emu.h 2007-03-10 17:29:31 UTC (rev 1401) +++ box/chris/merge/lib/win32/emu.h 2007-03-10 17:32:44 UTC (rev 1402) @@ -290,6 +290,9 @@ struct dirent *readdir(DIR *dp); int closedir(DIR *dp); +// local constant to open file exclusively without shared access +#define O_LOCK 0x10000 + HANDLE openfile(const char *filename, int flags, int mode); #define LOG_INFO 6 From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:36:54 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:36:54 +0000 Subject: [Box Backup-commit] COMMIT r1403 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 17:36:54 +0000 (Sat, 10 Mar 2007) New Revision: 1403 Modified: box/chris/merge/lib/win32/emu.cpp Log: We don't have access to DIRECTORY_SEPARATOR_ASCHAR in lib/win32 (refs #3, merges [1362]) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:32:44 UTC (rev 1402) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:36:54 UTC (rev 1403) @@ -867,7 +867,7 @@ _ui64toa(fi.dwVolumeSerialNumber, s->f_mntonname + 1, 16); // pseudo unix mount point - s->f_mntonname[0] = DIRECTORY_SEPARATOR_ASCHAR; + s->f_mntonname[0] = '\\'; CloseHandle(handle); // close the handle From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:37:30 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:37:30 +0000 Subject: [Box Backup-commit] COMMIT r1404 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 17:37:30 +0000 (Sat, 10 Mar 2007) New Revision: 1404 Modified: box/chris/merge/lib/win32/emu.cpp Log: Typo fix (refs #3) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:36:54 UTC (rev 1403) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:37:30 UTC (rev 1404) @@ -701,7 +701,7 @@ st->st_mode |= S_IWRITE; } - // st_dev is nroammly zero, regardless of the drive letter, + // st_dev is normally zero, regardless of the drive letter, // since backup locations can't normally span drives. However, // a reparse point does allow all kinds of weird stuff to happen. // We set st_dev to 1 for a reparse point, so that Box will detect From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:38:48 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:38:48 +0000 Subject: [Box Backup-commit] COMMIT r1405 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 17:38:48 +0000 (Sat, 10 Mar 2007) New Revision: 1405 Modified: box/chris/merge/lib/win32/emu.cpp Log: Fix two memory leaks and one buffer overflow in codepage conversion code. (refs #3, merges [1340]) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:37:30 UTC (rev 1404) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-10 17:38:48 UTC (rev 1405) @@ -978,7 +978,7 @@ return -1; } - free(pBuffer); + delete [] pBuffer; return 0; } @@ -1621,7 +1621,7 @@ } size_t WideSize = BufferSize / 5; - WCHAR* pWideBuffer = new WCHAR [WideSize]; + WCHAR* pWideBuffer = new WCHAR [WideSize + 1]; if (!pWideBuffer) { @@ -1647,6 +1647,8 @@ pWideBuffer[numCharsRead] = 0; char* pUtf8 = ConvertFromWideString(pWideBuffer, GetConsoleCP()); + delete [] pWideBuffer; + strncpy(pBuffer, pUtf8, BufferSize); delete [] pUtf8; From boxbackup-dev at fluffy.co.uk Sat Mar 10 17:50:33 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 17:50:33 +0000 Subject: [Box Backup-commit] COMMIT r1406 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 17:50:33 +0000 (Sat, 10 Mar 2007) New Revision: 1406 Modified: box/chris/merge/lib/win32/emu.h Log: First attempt to achieve a more logical order in this chaos: reordered all typedefs to be clearer and more readable (refs #3, merges [766]) Modified: box/chris/merge/lib/win32/emu.h =================================================================== --- box/chris/merge/lib/win32/emu.h 2007-03-10 17:38:48 UTC (rev 1405) +++ box/chris/merge/lib/win32/emu.h 2007-03-10 17:50:33 UTC (rev 1406) @@ -6,27 +6,24 @@ // basic types, may be required by other headers since we // don't include sys/types.h -#ifdef __MINGW32__ - #include - typedef uint32_t u_int32_t; -#else // MSVC - typedef __int64 int64_t; - typedef __int32 int32_t; - typedef __int16 int16_t; - typedef __int8 int8_t; - +#ifndef __MINGW32__ typedef unsigned __int64 u_int64_t; - typedef unsigned __int32 u_int32_t; - typedef unsigned __int64 uint64_t; + typedef __int64 int64_t; typedef unsigned __int32 uint32_t; + typedef unsigned __int32 u_int32_t; + typedef __int32 int32_t; typedef unsigned __int16 uint16_t; + typedef __int16 int16_t; typedef unsigned __int8 uint8_t; + typedef __int8 int8_t; #endif // emulated types, present on MinGW but not MSVC or vice versa -#ifdef _MSC_VER +#ifdef __MINGW32__ + typedef uint32_t u_int32_t; +#else typedef unsigned int mode_t; typedef unsigned int pid_t; From boxbackup-dev at fluffy.co.uk Sat Mar 10 18:01:07 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 18:01:07 +0000 Subject: [Box Backup-commit] COMMIT r1407 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 18:01:07 +0000 (Sat, 10 Mar 2007) New Revision: 1407 Modified: box/chris/merge/lib/win32/emu.h Log: Added d_type member to struct dirent, initialise with S_IFDIR or S_IFREG MinGW compile fix (refs #3, merges [775]) Modified: box/chris/merge/lib/win32/emu.h =================================================================== --- box/chris/merge/lib/win32/emu.h 2007-03-10 17:50:33 UTC (rev 1406) +++ box/chris/merge/lib/win32/emu.h 2007-03-10 18:01:07 UTC (rev 1407) @@ -6,7 +6,9 @@ // basic types, may be required by other headers since we // don't include sys/types.h -#ifndef __MINGW32__ +#ifdef __MINGW32__ + #include +#else // MSVC typedef unsigned __int64 u_int64_t; typedef unsigned __int64 uint64_t; typedef __int64 int64_t; From boxbackup-dev at fluffy.co.uk Sat Mar 10 18:52:15 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 18:52:15 +0000 Subject: [Box Backup-commit] COMMIT r1408 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 18:52:15 +0000 (Sat, 10 Mar 2007) New Revision: 1408 Modified: box/chris/merge/lib/win32/emu.h Log: Compile fix for [1397] (refs #3) Modified: box/chris/merge/lib/win32/emu.h =================================================================== --- box/chris/merge/lib/win32/emu.h 2007-03-10 18:01:07 UTC (rev 1407) +++ box/chris/merge/lib/win32/emu.h 2007-03-10 18:52:15 UTC (rev 1408) @@ -65,7 +65,7 @@ ( *(_result) = *gmtime( (_clock) ), \ (_result) ) -#define ITIMER_VIRTUAL 0 +#define ITIMER_REAL 0 #ifdef _MSC_VER // Microsoft decided to deprecate the standard POSIX functions. Great! From boxbackup-dev at fluffy.co.uk Sat Mar 10 18:57:00 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 18:57:00 +0000 Subject: [Box Backup-commit] COMMIT r1409 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 18:57:00 +0000 (Sat, 10 Mar 2007) New Revision: 1409 Modified: box/chris/merge/lib/win32/emu.h Log: Use #defines to replace POSIX functions with emulated ones on MinGW, like we do on MSVC. This allows us to #undef them when we really need to use the original platform function (if available). Disable emulated fstat() in raidfile (and use the platform one) by undefining fstat, since it doesn't use the other emulated file handling functions, or need Unicode support, and it can't take a filehandle returned by open() (only openfile()). (refs #3, merges [1045]) Modified: box/chris/merge/lib/win32/emu.h =================================================================== --- box/chris/merge/lib/win32/emu.h 2007-03-10 18:52:15 UTC (rev 1408) +++ box/chris/merge/lib/win32/emu.h 2007-03-10 18:57:00 UTC (rev 1409) @@ -147,41 +147,6 @@ return 0; } -int emu_chdir (const char* pDirName); -int emu_unlink(const char* pFileName); -char* emu_getcwd(char* pBuffer, int BufSize); -int emu_utimes(const char* pName, const struct timeval[]); -int emu_chmod (const char* pName, mode_t mode); - -#define utimes(buffer, times) emu_utimes(buffer, times) - -#ifdef _MSC_VER - #define chmod(file, mode) emu_chmod(file, mode) - #define chdir(directory) emu_chdir(directory) - #define unlink(file) emu_unlink(file) - #define getcwd(buffer, size) emu_getcwd(buffer, size) -#else - inline int chmod(const char * pName, mode_t mode) - { - return emu_chmod(pName, mode); - } - - inline int chdir(const char* pDirName) - { - return emu_chdir(pDirName); - } - - inline char* getcwd(char* pBuffer, int BufSize) - { - return emu_getcwd(pBuffer, BufSize); - } - - inline int unlink(const char* pFileName) - { - return emu_unlink(pFileName); - } -#endif - //I do not perceive a need to change the user or group on a backup client //at any rate the owner of a service can be set in the service settings inline int setegid(int) @@ -251,13 +216,6 @@ #define vsnprintf _vsnprintf -int emu_mkdir(const char* pPathName); - -inline int mkdir(const char *pPathName, mode_t mode = 0) -{ - return emu_mkdir(pPathName); -} - #ifndef __MINGW32__ inline int strcasecmp(const char *s1, const char *s2) { @@ -370,33 +328,31 @@ }; #endif -int emu_stat(const char * name, struct stat * st); -int emu_fstat(HANDLE file, struct stat * st); -int statfs(const char * name, struct statfs * s); - // need this for conversions time_t ConvertFileTimeToTime_t(FILETIME *fileTime); bool ConvertTime_tToFileTime(const time_t from, FILETIME *pTo); -#ifdef _MSC_VER - #define stat(filename, struct) emu_stat (filename, struct) - #define lstat(filename, struct) emu_stat (filename, struct) - #define fstat(handle, struct) emu_fstat(handle, struct) -#else - inline int stat(const char* filename, struct stat* stat) - { - return emu_stat(filename, stat); - } - inline int lstat(const char* filename, struct stat* stat) - { - return emu_stat(filename, stat); - } - inline int fstat(HANDLE handle, struct stat* stat) - { - return emu_fstat(handle, stat); - } -#endif +int emu_chdir (const char* pDirName); +int emu_mkdir (const char* pPathName); +int emu_unlink (const char* pFileName); +int emu_fstat (HANDLE file, struct stat* st); +int emu_stat (const char* pName, struct stat* st); +int emu_utimes (const char* pName, const struct timeval[]); +int emu_chmod (const char* pName, mode_t mode); +char* emu_getcwd (char* pBuffer, int BufSize); +#define chdir(directory) emu_chdir (directory) +#define mkdir(path, mode) emu_mkdir (path) +#define unlink(file) emu_unlink (file) +#define stat(filename, struct) emu_stat (filename, struct) +#define lstat(filename, struct) emu_stat (filename, struct) +#define fstat(handle, struct) emu_fstat (handle, struct) +#define utimes(buffer, times) emu_utimes (buffer, times) +#define chmod(file, mode) emu_chmod (file, mode) +#define getcwd(buffer, size) emu_getcwd (buffer, size) + +int statfs(const char * name, struct statfs * s); + int poll(struct pollfd *ufds, unsigned long nfds, int timeout); bool EnableBackupRights( void ); From boxbackup-dev at fluffy.co.uk Sat Mar 10 19:01:11 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 19:01:11 +0000 Subject: [Box Backup-commit] COMMIT r1410 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 19:01:11 +0000 (Sat, 10 Mar 2007) New Revision: 1410 Modified: box/chris/merge/lib/win32/emu.h Log: Remove #define BOX_VERSION since we now get it from BoxVersion.h via BoxPlatform.h when building with MSVC, and from the Makefiles when building with MinGW. (refs #3, merges part of [634]) Modified: box/chris/merge/lib/win32/emu.h =================================================================== --- box/chris/merge/lib/win32/emu.h 2007-03-10 18:57:00 UTC (rev 1409) +++ box/chris/merge/lib/win32/emu.h 2007-03-10 19:01:11 UTC (rev 1410) @@ -199,11 +199,6 @@ typedef int socklen_t; #endif -// I (re-)defined here for the moment; has to be removed later !!! -#ifndef BOX_VERSION -#define BOX_VERSION "0.09hWin32" -#endif - #define S_IRGRP S_IWRITE #define S_IWGRP S_IREAD #define S_IROTH S_IWRITE | S_IREAD From boxbackup-dev at fluffy.co.uk Sat Mar 10 19:03:37 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 19:03:37 +0000 Subject: [Box Backup-commit] COMMIT r1411 - box/chris/general/lib/win32 Message-ID: Author: chris Date: 2007-03-10 19:03:37 +0000 (Sat, 10 Mar 2007) New Revision: 1411 Modified: box/chris/general/lib/win32/emu.h Log: Compile fix (merges back [1054] from merge tree) Modified: box/chris/general/lib/win32/emu.h =================================================================== --- box/chris/general/lib/win32/emu.h 2007-03-10 19:01:11 UTC (rev 1410) +++ box/chris/general/lib/win32/emu.h 2007-03-10 19:03:37 UTC (rev 1411) @@ -186,7 +186,7 @@ #endif #ifdef _DIRENT_H_ -#error You must not include MinGW's dirent.h! +#error You must not include the MinGW dirent.h! #endif struct dirent From boxbackup-dev at fluffy.co.uk Sat Mar 10 19:05:52 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 19:05:52 +0000 Subject: [Box Backup-commit] COMMIT r1412 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 19:05:52 +0000 (Sat, 10 Mar 2007) New Revision: 1412 Modified: box/chris/merge/lib/win32/emu.h Log: Reorder for clarity Expose GetErrorMessage() Improve comments (refs #3, merges [1365]) Modified: box/chris/merge/lib/win32/emu.h =================================================================== --- box/chris/merge/lib/win32/emu.h 2007-03-10 19:03:37 UTC (rev 1411) +++ box/chris/merge/lib/win32/emu.h 2007-03-10 19:05:52 UTC (rev 1412) @@ -321,7 +321,7 @@ time_t st_mtime; time_t st_ctime; }; -#endif +#endif // 0 // need this for conversions time_t ConvertFileTimeToTime_t(FILETIME *fileTime); @@ -349,6 +349,19 @@ int statfs(const char * name, struct statfs * s); int poll(struct pollfd *ufds, unsigned long nfds, int timeout); + +struct iovec { + void *iov_base; /* Starting address */ + size_t iov_len; /* Number of bytes */ +}; + +int readv (int filedes, const struct iovec *vector, size_t count); +int writev(int filedes, const struct iovec *vector, size_t count); + +// The following functions are not emulations, but utilities for other +// parts of the code where Windows API is used or windows-specific stuff +// is needed, like codepage conversion. + bool EnableBackupRights( void ); bool ConvertEncoding (const std::string& rSource, int sourceCodePage, @@ -360,15 +373,12 @@ bool ConvertUtf8ToConsole(const char* pString, std::string& rDest); bool ConvertConsoleToUtf8(const char* pString, std::string& rDest); -// replacement for _cgetws which requires a relatively recent C runtime lib +// GetErrorMessage() returns a system error message, like strerror() +// but for Windows error codes. +std::string GetErrorMessage(DWORD errorCode); + +// console_read() is a replacement for _cgetws which requires a +// relatively recent C runtime lib int console_read(char* pBuffer, size_t BufferSize); -struct iovec { - void *iov_base; /* Starting address */ - size_t iov_len; /* Number of bytes */ -}; - -int readv (int filedes, const struct iovec *vector, size_t count); -int writev(int filedes, const struct iovec *vector, size_t count); - #endif // !EMU_INCLUDE && WIN32 From boxbackup-dev at fluffy.co.uk Sat Mar 10 19:09:09 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 19:09:09 +0000 Subject: [Box Backup-commit] COMMIT r1413 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 19:09:09 +0000 (Sat, 10 Mar 2007) New Revision: 1413 Modified: box/chris/merge/lib/win32/emu.h Log: Add new syslog level emulations (refs #3, merges remainder of [1299]) Modified: box/chris/merge/lib/win32/emu.h =================================================================== --- box/chris/merge/lib/win32/emu.h 2007-03-10 19:05:52 UTC (rev 1412) +++ box/chris/merge/lib/win32/emu.h 2007-03-10 19:09:09 UTC (rev 1413) @@ -247,9 +247,12 @@ HANDLE openfile(const char *filename, int flags, int mode); +#define LOG_DEBUG LOG_INFO #define LOG_INFO 6 +#define LOG_NOTICE LOG_INFO #define LOG_WARNING 4 #define LOG_ERR 3 +#define LOG_CRIT LOG_ERR #define LOG_PID 0 #define LOG_LOCAL5 0 #define LOG_LOCAL6 0 From boxbackup-dev at fluffy.co.uk Sat Mar 10 19:12:57 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 19:12:57 +0000 Subject: [Box Backup-commit] COMMIT r1414 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-10 19:12:57 +0000 (Sat, 10 Mar 2007) New Revision: 1414 Modified: box/chris/merge/lib/win32/emu.h Log: Group remaining set*id() and get*id() functions. Improve comments about why they are being retained. (refs #3, related to [634]) Modified: box/chris/merge/lib/win32/emu.h =================================================================== --- box/chris/merge/lib/win32/emu.h 2007-03-10 19:09:09 UTC (rev 1413) +++ box/chris/merge/lib/win32/emu.h 2007-03-10 19:12:57 UTC (rev 1414) @@ -83,12 +83,6 @@ void InitTimer(void); void FiniTimer(void); -inline int geteuid(void) -{ - //lets pretend to be root! - return 0; -} - struct passwd { char *pw_name; char *pw_passwd; @@ -147,8 +141,9 @@ return 0; } -//I do not perceive a need to change the user or group on a backup client -//at any rate the owner of a service can be set in the service settings +// Windows and Unix owners and groups are pretty fundamentally different. +// Ben prefers that we kludge here rather than litter the code with #ifdefs. +// Pretend to be root, and pretend that set...() operations succeed. inline int setegid(int) { return true; @@ -173,6 +168,10 @@ { return 0; } +inline int geteuid(void) +{ + return 0; +} #ifndef PATH_MAX #define PATH_MAX MAX_PATH From boxbackup-dev at fluffy.co.uk Sat Mar 10 19:13:36 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 19:13:36 +0000 Subject: [Box Backup-commit] COMMIT r1415 - box/chris/general/lib/win32 Message-ID: Author: chris Date: 2007-03-10 19:13:36 +0000 (Sat, 10 Mar 2007) New Revision: 1415 Modified: box/chris/general/lib/win32/emu.h Log: Revert part of [634] with better comments why this code should stay. Modified: box/chris/general/lib/win32/emu.h =================================================================== --- box/chris/general/lib/win32/emu.h 2007-03-10 19:12:57 UTC (rev 1414) +++ box/chris/general/lib/win32/emu.h 2007-03-10 19:13:36 UTC (rev 1415) @@ -141,6 +141,38 @@ return 0; } +// Windows and Unix owners and groups are pretty fundamentally different. +// Ben prefers that we kludge here rather than litter the code with #ifdefs. +// Pretend to be root, and pretend that set...() operations succeed. +inline int setegid(int) +{ + return true; +} +inline int seteuid(int) +{ + return true; +} +inline int setgid(int) +{ + return true; +} +inline int setuid(int) +{ + return true; +} +inline int getgid(void) +{ + return 0; +} +inline int getuid(void) +{ + return 0; +} +inline int geteuid(void) +{ + return 0; +} + #ifndef PATH_MAX #define PATH_MAX MAX_PATH #endif From boxbackup-dev at fluffy.co.uk Sat Mar 10 19:21:11 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 19:21:11 +0000 Subject: [Box Backup-commit] COMMIT r1416 - box/chris/general/lib/backupclient Message-ID: Author: chris Date: 2007-03-10 19:21:10 +0000 (Sat, 10 Mar 2007) New Revision: 1416 Modified: box/chris/general/lib/backupclient/BackupClientFileAttributes.cpp Log: Remove #ifdefs around geteuid() and chown() (no longer required, and Ben prefers it this way) Modified: box/chris/general/lib/backupclient/BackupClientFileAttributes.cpp =================================================================== --- box/chris/general/lib/backupclient/BackupClientFileAttributes.cpp 2007-03-10 19:13:36 UTC (rev 1415) +++ box/chris/general/lib/backupclient/BackupClientFileAttributes.cpp 2007-03-10 19:21:10 UTC (rev 1416) @@ -642,7 +642,6 @@ } // If working as root, set user IDs - #ifndef WIN32 if(::geteuid() == 0) { #ifndef HAVE_LCHOWN @@ -662,7 +661,6 @@ } #endif } - #endif if(static_cast(xattrOffset+sizeof(u_int32_t))<=mpClearAttributes->GetSize()) { From boxbackup-dev at fluffy.co.uk Sat Mar 10 19:23:37 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 19:23:37 +0000 Subject: [Box Backup-commit] COMMIT r1417 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-10 19:23:37 +0000 (Sat, 10 Mar 2007) New Revision: 1417 Modified: box/chris/general/lib/common/Test.h Log: Revert to original indentation Modified: box/chris/general/lib/common/Test.h =================================================================== --- box/chris/general/lib/common/Test.h 2007-03-10 19:21:10 UTC (rev 1416) +++ box/chris/general/lib/common/Test.h 2007-03-10 19:23:37 UTC (rev 1417) @@ -309,8 +309,7 @@ inline void TestRemoteProcessMemLeaksFunc(const char *filename, const char* file, int line) { - #ifdef BOX_MEMORY_LEAK_TESTING - +#ifdef BOX_MEMORY_LEAK_TESTING // Does the file exist? if(!TestFileExists(filename)) { @@ -348,8 +347,7 @@ // Delete it ::unlink(filename); } - - #endif +#endif } #ifdef WIN32 From boxbackup-dev at fluffy.co.uk Sat Mar 10 19:25:04 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 19:25:04 +0000 Subject: [Box Backup-commit] COMMIT r1418 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-10 19:25:03 +0000 (Sat, 10 Mar 2007) New Revision: 1418 Modified: box/chris/general/lib/common/UnixUser.cpp Log: Remove #ifdefs, no longer required Modified: box/chris/general/lib/common/UnixUser.cpp =================================================================== --- box/chris/general/lib/common/UnixUser.cpp 2007-03-10 19:23:37 UTC (rev 1417) +++ box/chris/general/lib/common/UnixUser.cpp 2007-03-10 19:25:03 UTC (rev 1418) @@ -78,11 +78,7 @@ if(mRevertOnDestruction) { // Revert to "real" user and group id of the process - #ifdef WIN32 - if(0) - #else if(::setegid(::getgid()) != 0 || ::seteuid(::getuid()) != 0) - #endif { THROW_EXCEPTION(CommonException, CouldNotRestoreProcessUser) } @@ -104,11 +100,7 @@ if(Temporary) { // Change temporarily (change effective only) - #ifdef WIN32 - if(0) - #else if(::setegid(mGID) != 0 || ::seteuid(mUID) != 0) - #endif { THROW_EXCEPTION(CommonException, CouldNotChangeProcessUser) } @@ -119,11 +111,7 @@ else { // Change permanently (change all UIDs and GIDs) - #ifdef WIN32 - if(0) - #else if(::setgid(mGID) != 0 || ::setuid(mUID) != 0) - #endif { THROW_EXCEPTION(CommonException, CouldNotChangeProcessUser) } From boxbackup-dev at fluffy.co.uk Sat Mar 10 19:26:02 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 19:26:02 +0000 Subject: [Box Backup-commit] COMMIT r1419 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-10 19:26:02 +0000 (Sat, 10 Mar 2007) New Revision: 1419 Modified: box/chris/merge/lib/common/UnixUser.cpp Log: Remove #ifdefs, no longer required (refs #3, merges [1418]) Modified: box/chris/merge/lib/common/UnixUser.cpp =================================================================== --- box/chris/merge/lib/common/UnixUser.cpp 2007-03-10 19:25:03 UTC (rev 1418) +++ box/chris/merge/lib/common/UnixUser.cpp 2007-03-10 19:26:02 UTC (rev 1419) @@ -78,12 +78,7 @@ if(mRevertOnDestruction) { // Revert to "real" user and group id of the process - #ifdef WIN32 - if(0) - #else - if(::setegid(::getgid()) != 0 - || ::seteuid(::getuid()) != 0) - #endif + if(::setegid(::getgid()) != 0 || ::seteuid(::getuid()) != 0) { THROW_EXCEPTION(CommonException, CouldNotRestoreProcessUser) } @@ -105,12 +100,7 @@ if(Temporary) { // Change temporarily (change effective only) - #ifdef WIN32 - if(0) - #else - if(::setegid(mGID) != 0 - || ::seteuid(mUID) != 0) - #endif + if(::setegid(mGID) != 0 || ::seteuid(mUID) != 0) { THROW_EXCEPTION(CommonException, CouldNotChangeProcessUser) } @@ -121,12 +111,7 @@ else { // Change permanently (change all UIDs and GIDs) - #ifdef WIN32 - if(0) - #else - if(::setgid(mGID) != 0 - || ::setuid(mUID) != 0) - #endif + if(::setgid(mGID) != 0 || ::setuid(mUID) != 0) { THROW_EXCEPTION(CommonException, CouldNotChangeProcessUser) } From boxbackup-dev at fluffy.co.uk Sat Mar 10 19:35:39 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 19:35:39 +0000 Subject: [Box Backup-commit] COMMIT r1420 - box/chris/merge Message-ID: Author: chris Date: 2007-03-10 19:35:38 +0000 (Sat, 10 Mar 2007) New Revision: 1420 Modified: box/chris/merge/modules.txt box/chris/merge/parcels.txt Log: Fixes for cross-compiling from Ubuntu Breezy (refs #3, merges [573]) Modified: box/chris/merge/modules.txt =================================================================== --- box/chris/merge/modules.txt 2007-03-10 19:26:02 UTC (rev 1419) +++ box/chris/merge/modules.txt 2007-03-10 19:35:38 UTC (rev 1420) @@ -10,6 +10,7 @@ # Generic support code and modules OMIT:mingw32 +OMIT:mingw32msvc OMIT:CYGWIN lib/raidfile END-OMIT @@ -24,6 +25,7 @@ test/compress lib/compress lib/win32 OMIT:mingw32 +OMIT:mingw32msvc test/basicserver lib/server lib/win32 OMIT:CYGWIN test/raidfile lib/raidfile lib/intercept @@ -36,6 +38,7 @@ lib/backupclient lib/server lib/crypto lib/compress OMIT:mingw32 +OMIT:mingw32msvc OMIT:CYGWIN lib/backupstore lib/server lib/raidfile lib/backupclient bin/bbstored lib/raidfile lib/server lib/backupstore lib/backupclient lib/win32 @@ -48,6 +51,7 @@ bin/bbackupctl lib/server lib/backupclient lib/win32 OMIT:mingw32 +OMIT:mingw32msvc OMIT:CYGWIN test/backupstore bin/bbstored bin/bbstoreaccounts lib/server lib/backupstore lib/backupclient lib/raidfile test/backupstorefix bin/bbstored bin/bbstoreaccounts lib/backupstore lib/raidfile bin/bbackupquery bin/bbackupd Modified: box/chris/merge/parcels.txt =================================================================== --- box/chris/merge/parcels.txt 2007-03-10 19:26:02 UTC (rev 1419) +++ box/chris/merge/parcels.txt 2007-03-10 19:35:38 UTC (rev 1420) @@ -9,16 +9,20 @@ bin bbackupctl script bin/bbackupd/bbackupd-config -ONLY:mingw32 +ONLY:mingw32,mingw32msvc script bin/bbackupd/win32/installer.iss script bin/bbackupd/win32/ReadMe.txt script bin/bbackupd/win32/bbackupd.conf +END-ONLY + +ONLY:mingw32 script /bin/mgwz.dll script /bin/mingwm10.dll optional script /bin/pcreposix.dll END-ONLY OMIT:mingw32 +OMIT:mingw32msvc OMIT:CYGWIN backup-server From boxbackup-dev at fluffy.co.uk Sat Mar 10 22:26:21 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 10 Mar 2007 22:26:21 -0000 Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> References: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> Message-ID: <060.2a28754d05bae4a671ab9f4c67ae0500@fluffy.co.uk> #3: Merge Win32 branch -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: trunk Resolution: | Keywords: windows win32 merge branch -----------------------+---------------------------------------------------- Comment (by invisnet): [1083] An OVERLAPPED struct needs to be cleared before every use when talking to pipes, not just the first time. The pipe is in message mode; why would there be more data in the buffer than is being asked for? Use stream mode if this is supposed to be a generic class, or provide some way to select the mode. memcpy() is not guaranteed to work with overlapping buffers (line 288). Use memmove() instead. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Mon Mar 19 14:23:43 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Mon, 19 Mar 2007 14:23:43 -0000 Subject: [Box Backup-commit] Re: #17: List files using wildcards In-Reply-To: <051.7063f5df64cfe771a605efaa49ea6c8f@fluffy.co.uk> References: <051.7063f5df64cfe771a605efaa49ea6c8f@fluffy.co.uk> Message-ID: <060.d39e20aa98bc43c017c73a78846203e3@fluffy.co.uk> #17: List files using wildcards ---------------------------+------------------------------------------------ Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupquery | Version: 0.10 Resolution: | Keywords: windows list files matching wildcard ---------------------------+------------------------------------------------ Comment (by petej): For what it's worth, in the interim, I use things like the following. If cygwin is not installed, then in a Windows cmd window: {{{ bbackupquery "list -odstr" quit | findstr /r "\.doc$" bbackupquery "list -odstr" quit | find ".doc" }}} Or if I'm lucky and cygwin is installed, in a cygwin window: {{{ bbackupquery "list -odstr" quit | grep "\.doc$" }}} I always have to flail around to escape or quote spaces in path names, but I eventually get it to work, and then promptly forget to document what I did. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:14:23 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:14:23 +0000 Subject: [Box Backup-commit] COMMIT r1421 - box/chris/merge/test/bbackupd Message-ID: Author: chris Date: 2007-03-22 23:14:23 +0000 (Thu, 22 Mar 2007) New Revision: 1421 Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp Log: Simplify wait code (refs #3) Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-10 19:35:38 UTC (rev 1420) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-22 23:14:23 UTC (rev 1421) @@ -65,15 +65,7 @@ void wait_for_backup_operation(int seconds = TIME_TO_WAIT_FOR_BACKUP_OPERATION) { - printf("waiting: "); - fflush(stdout); - for(int l = 0; l < seconds; ++l) - { - sleep(1); - printf("."); - fflush(stdout); - } - printf("\n"); + wait_for_operation(seconds); } int bbstored_pid = 0; From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:15:23 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:15:23 +0000 Subject: [Box Backup-commit] COMMIT r1422 - box/chris/merge/test/bbackupd Message-ID: Author: chris Date: 2007-03-22 23:15:22 +0000 (Thu, 22 Mar 2007) New Revision: 1422 Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp Log: Improve output messages when waiting for daemon to start (refs #3) Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-22 23:14:23 UTC (rev 1421) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-22 23:15:22 UTC (rev 1422) @@ -530,7 +530,7 @@ TEST_THAT(TestFileExists("testfiles/bbackupd.pid")); - printf("Waiting for daemon to start"); + printf("Waiting for backup daemon to start: "); int pid = -1; for (int i = 0; i < 30; i++) @@ -546,7 +546,8 @@ } } - printf("\n"); + printf(" done.\n"); + fflush(stdout); TEST_THAT(pid > 0); return pid; From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:19:20 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:19:20 +0000 Subject: [Box Backup-commit] COMMIT r1423 - box/chris/merge/test/bbackupd Message-ID: Author: chris Date: 2007-03-22 23:19:19 +0000 (Thu, 22 Mar 2007) New Revision: 1423 Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp Log: Test expected behaviour for modifying a file without changing its modtime, both tracked and untracked (refs #3) Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-22 23:15:22 UTC (rev 1422) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-22 23:19:19 UTC (rev 1423) @@ -1050,6 +1050,79 @@ TEST_THAT(compareReturnValue == 1*256); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + // rename an untracked file over an + // existing untracked file + printf("Rename over existing untracked file\n"); + int fd1 = open("testfiles/TestDir1/untracked-1", + O_CREAT | O_EXCL | O_WRONLY, 0700); + int fd2 = open("testfiles/TestDir1/untracked-2", + O_CREAT | O_EXCL | O_WRONLY, 0700); + TEST_THAT(fd1 > 0); + TEST_THAT(fd2 > 0); + TEST_THAT(write(fd1, "hello", 5) == 5); + TEST_THAT(close(fd1) == 0); + sleep(1); + TEST_THAT(write(fd2, "world", 5) == 5); + TEST_THAT(close(fd2) == 0); + TEST_THAT(TestFileExists("testfiles/TestDir1/untracked-1")); + TEST_THAT(TestFileExists("testfiles/TestDir1/untracked-2")); + wait_for_operation(5); + // back up both files + wait_for_backup_operation(); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf -l testfiles/query3t.log " + "\"compare -ac\" quit"); + TEST_THAT(compareReturnValue == 1*256); + TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(::rename("testfiles/TestDir1/untracked-1", + "testfiles/TestDir1/untracked-2") == 0); + TEST_THAT(!TestFileExists("testfiles/TestDir1/untracked-1")); + TEST_THAT( TestFileExists("testfiles/TestDir1/untracked-2")); + wait_for_backup_operation(); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf -l testfiles/query3t.log " + "\"compare -ac\" quit"); + TEST_THAT(compareReturnValue == 1*256); + TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + + // case which went wrong: rename a tracked file over an + // existing tracked file + printf("Rename over existing tracked file\n"); + fd1 = open("testfiles/TestDir1/tracked-1", + O_CREAT | O_EXCL | O_WRONLY, 0700); + fd2 = open("testfiles/TestDir1/tracked-2", + O_CREAT | O_EXCL | O_WRONLY, 0700); + TEST_THAT(fd1 > 0); + TEST_THAT(fd2 > 0); + char buffer[1024]; + TEST_THAT(write(fd1, "hello", 5) == 5); + TEST_THAT(write(fd1, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd1) == 0); + sleep(1); + TEST_THAT(write(fd2, "world", 5) == 5); + TEST_THAT(write(fd2, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd2) == 0); + TEST_THAT(TestFileExists("testfiles/TestDir1/tracked-1")); + TEST_THAT(TestFileExists("testfiles/TestDir1/tracked-2")); + wait_for_operation(5); + // back up both files + wait_for_backup_operation(); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf -l testfiles/query3u.log " + "\"compare -ac\" quit"); + TEST_THAT(compareReturnValue == 1*256); + TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(::rename("testfiles/TestDir1/tracked-1", + "testfiles/TestDir1/tracked-2") == 0); + TEST_THAT(!TestFileExists("testfiles/TestDir1/tracked-1")); + TEST_THAT( TestFileExists("testfiles/TestDir1/tracked-2")); + wait_for_backup_operation(); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf -l testfiles/query3v.log " + "\"compare -ac\" quit"); + TEST_THAT(compareReturnValue == 1*256); + TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + // case which went wrong: rename a tracked file over a deleted file printf("Rename an existing file over a deleted file\n"); TEST_THAT(::rename("testfiles/TestDir1/df9834.dsf", "testfiles/TestDir1/x1/dsfdsfs98.fd") == 0); From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:20:19 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:20:19 +0000 Subject: [Box Backup-commit] COMMIT r1424 - box/chris/merge/test/bbackupd Message-ID: Author: chris Date: 2007-03-22 23:20:19 +0000 (Thu, 22 Mar 2007) New Revision: 1424 Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp Log: Flush stdout after writing to it (refs #3) Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-22 23:19:19 UTC (rev 1423) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-22 23:20:19 UTC (rev 1424) @@ -1242,6 +1242,7 @@ ::fclose(f); } printf("\n"); + fflush(stdout); // Check there's a difference compareReturnValue = ::system("testfiles/extcheck1.pl"); @@ -1262,6 +1263,7 @@ ::fclose(f); } printf("\n"); + fflush(stdout); compareReturnValue = ::system("testfiles/extcheck2.pl"); TEST_THAT(compareReturnValue == 1*256); From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:20:57 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:20:57 +0000 Subject: [Box Backup-commit] COMMIT r1425 - box/chris/merge/test/bbackupd Message-ID: Author: chris Date: 2007-03-22 23:20:57 +0000 (Thu, 22 Mar 2007) New Revision: 1425 Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp Log: Use unique name for compare log (refs #3) Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-22 23:20:19 UTC (rev 1424) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-22 23:20:57 UTC (rev 1425) @@ -1127,7 +1127,7 @@ printf("Rename an existing file over a deleted file\n"); TEST_THAT(::rename("testfiles/TestDir1/df9834.dsf", "testfiles/TestDir1/x1/dsfdsfs98.fd") == 0); wait_for_backup_operation(); - compareReturnValue = ::system("../../bin/bbackupquery/bbackupquery -q -c testfiles/bbackupd.conf -l testfiles/query3s.log \"compare -ac\" quit"); + compareReturnValue = ::system("../../bin/bbackupquery/bbackupquery -q -c testfiles/bbackupd.conf -l testfiles/query3w.log \"compare -ac\" quit"); TEST_THAT(compareReturnValue == 1*256); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:21:38 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:21:38 +0000 Subject: [Box Backup-commit] COMMIT r1426 - box/chris/merge/test/bbackupd Message-ID: Author: chris Date: 2007-03-22 23:21:38 +0000 (Thu, 22 Mar 2007) New Revision: 1426 Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp Log: Test that bbackupd reports an error when the backup failed due to an exception (refs #3) Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-22 23:20:57 UTC (rev 1425) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-22 23:21:38 UTC (rev 1426) @@ -1007,6 +1007,39 @@ TEST_THAT(compareReturnValue == 1*256); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + // Check that store errors are reported neatly + printf("Create store error\n"); + TEST_THAT(::rename("testfiles/0_0/backup/01234567/info.rf", + "testfiles/0_0/backup/01234567/info.rf.bak") == 0); + TEST_THAT(::rename("testfiles/0_1/backup/01234567/info.rf", + "testfiles/0_1/backup/01234567/info.rf.bak") == 0); + TEST_THAT(::rename("testfiles/0_2/backup/01234567/info.rf", + "testfiles/0_2/backup/01234567/info.rf.bak") == 0); + // Create a file to trigger an upload + { + int fd1 = open("testfiles/TestDir1/force-upload", + O_CREAT | O_EXCL | O_WRONLY, 0700); + TEST_THAT(fd1 > 0); + TEST_THAT(write(fd1, "just do it", 10) == 10); + TEST_THAT(close(fd1) == 0); + wait_for_backup_operation(4); + } + // Wait and test... + wait_for_backup_operation(); + // Check that it was reported correctly + TEST_THAT(TestFileExists("testfiles/notifyran.backup-error.1")); + // Check that the error was only reorted once + TEST_THAT(!TestFileExists("testfiles/notifyran.backup-error.2")); + // Fix the store + TEST_THAT(::rename("testfiles/0_0/backup/01234567/info.rf.bak", + "testfiles/0_0/backup/01234567/info.rf") == 0); + TEST_THAT(::rename("testfiles/0_1/backup/01234567/info.rf.bak", + "testfiles/0_1/backup/01234567/info.rf") == 0); + TEST_THAT(::rename("testfiles/0_2/backup/01234567/info.rf.bak", + "testfiles/0_2/backup/01234567/info.rf") == 0); + // wait until bbackupd recovers from the exception + wait_for_backup_operation(100); + // Bad case: delete a file/symlink, replace it with a directory printf("Replace symlink with directory, add new directory\n"); TEST_THAT(::unlink("testfiles/TestDir1/symlink-to-dir") == 0); From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:22:30 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:22:30 +0000 Subject: [Box Backup-commit] COMMIT r1427 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-22 23:22:30 +0000 (Thu, 22 Mar 2007) New Revision: 1427 Modified: box/chris/merge/lib/common/Test.h Log: Flush stdout when writing to it (refs #3) Modified: box/chris/merge/lib/common/Test.h =================================================================== --- box/chris/merge/lib/common/Test.h 2007-03-22 23:21:38 UTC (rev 1426) +++ box/chris/merge/lib/common/Test.h 2007-03-22 23:22:30 UTC (rev 1427) @@ -397,6 +397,7 @@ fflush(stdout); } printf("\n"); + fflush(stdout); } #endif // TEST__H From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:26:37 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:26:37 +0000 Subject: [Box Backup-commit] COMMIT r1428 - box/chris/merge/bin/bbackupd Message-ID: Author: chris Date: 2007-03-22 23:26:37 +0000 (Thu, 22 Mar 2007) New Revision: 1428 Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp box/chris/merge/bin/bbackupd/BackupDaemon.h Log: Add a new notification constant, NotifyEvent_BackupError, for use when an exception occurs during the backup. Make bbackupd notify sysadmin when an exception occurs during the backup, using this error code, and the notification string "backup-error". Change NotifyEvent__MAX to be one greater than the highest notification constant, makes code more maintainable. (refs #3) Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2007-03-22 23:22:30 UTC (rev 1427) +++ box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2007-03-22 23:26:37 UTC (rev 1428) @@ -119,8 +119,8 @@ // Only ever one instance of a daemon SSLLib::Initialise(); - // Initialise notifcation sent status - for(int l = 0; l <= NotifyEvent__MAX; ++l) + // Initialise notification sent status + for(int l = 0; l < NotifyEvent__MAX; ++l) { mNotificationsSent[l] = false; } @@ -878,6 +878,8 @@ else { // Not restart/terminate, pause and retry + // Notify administrator + NotifySysadmin(NotifyEvent_BackupError); SetState(State_Error); BOX_ERROR("Exception caught (" << errorString @@ -2013,11 +2015,17 @@ // -------------------------------------------------------------------------- void BackupDaemon::NotifySysadmin(int Event) { - static const char *sEventNames[] = {"store-full", "read-error", 0}; + static const char *sEventNames[] = + { + "store-full", + "read-error", + "backup-error", + 0 + }; BOX_TRACE("BackupDaemon::NotifySysadmin() called, event = " << Event); - if(Event < 0 || Event > NotifyEvent__MAX) + if(Event < 0 || Event >= NotifyEvent__MAX) { THROW_EXCEPTION(BackupStoreException, BadNotifySysadminEventCode); } Modified: box/chris/merge/bin/bbackupd/BackupDaemon.h =================================================================== --- box/chris/merge/bin/bbackupd/BackupDaemon.h 2007-03-22 23:22:30 UTC (rev 1427) +++ box/chris/merge/bin/bbackupd/BackupDaemon.h 2007-03-22 23:26:37 UTC (rev 1428) @@ -82,8 +82,9 @@ enum { NotifyEvent_StoreFull = 0, - NotifyEvent_ReadError = 1, - NotifyEvent__MAX = 1 + NotifyEvent_ReadError, + NotifyEvent_BackupError, + NotifyEvent__MAX // When adding notifications, remember to add strings to NotifySysadmin() }; void NotifySysadmin(int Event); @@ -177,7 +178,7 @@ CommandSocketInfo *mpCommandSocketInfo; // Stop notifications being repeated. - bool mNotificationsSent[NotifyEvent__MAX + 1]; + bool mNotificationsSent[NotifyEvent__MAX]; // Unused entries in the root directory wait a while before being deleted box_time_t mDeleteUnusedRootDirEntriesAfter; // time to delete them From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:27:18 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:27:18 +0000 Subject: [Box Backup-commit] COMMIT r1429 - box/chris/merge/lib/server Message-ID: Author: chris Date: 2007-03-22 23:27:18 +0000 (Thu, 22 Mar 2007) New Revision: 1429 Modified: box/chris/merge/lib/server/Daemon.cpp Log: Log at trace level by default in debug builds (refs #3) Modified: box/chris/merge/lib/server/Daemon.cpp =================================================================== --- box/chris/merge/lib/server/Daemon.cpp 2007-03-22 23:26:37 UTC (rev 1428) +++ box/chris/merge/lib/server/Daemon.cpp 2007-03-22 23:27:18 UTC (rev 1429) @@ -101,7 +101,13 @@ mConfigFileName = DefaultConfigFile; bool haveConfigFile = false; bool singleProcess = false; + + #ifdef NDEBUG int masterLevel = Log::NOTICE; // need an int to do math with + #else + int masterLevel = Log::TRACE; // need an int to do math with + #endif + char c; while((c = getopt(argc, (char * const *)argv, "c:Dqv")) != -1) From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:28:07 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:28:07 +0000 Subject: [Box Backup-commit] COMMIT r1430 - box/chris/merge/bin/bbackupd Message-ID: Author: chris Date: 2007-03-22 23:28:07 +0000 (Thu, 22 Mar 2007) New Revision: 1430 Modified: box/chris/merge/bin/bbackupd/BackupClientDirectoryRecord.cpp Log: Trace reasons for uploading (or not) each file Modified: box/chris/merge/bin/bbackupd/BackupClientDirectoryRecord.cpp =================================================================== --- box/chris/merge/bin/bbackupd/BackupClientDirectoryRecord.cpp 2007-03-22 23:27:18 UTC (rev 1429) +++ box/chris/merge/bin/bbackupd/BackupClientDirectoryRecord.cpp 2007-03-22 23:28:07 UTC (rev 1430) @@ -30,6 +30,7 @@ #include "BackupStoreException.h" #include "Archive.h" #include "PathUtils.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -793,10 +794,14 @@ if (pDirOnStore != 0 && en == 0) { doUpload = true; + BOX_TRACE(filename << ": will upload " + "(not on server)"); } else if (modTime >= rParams.mSyncPeriodStart) { doUpload = true; + BOX_TRACE(filename << ": will upload " + "(modified since last sync)"); } } @@ -813,6 +818,8 @@ > rParams.mMaxUploadWait) { doUpload = true; + BOX_TRACE(filename << ": will upload " + "(continually modified)"); } // Then make sure that if files are added with a @@ -828,6 +835,8 @@ en->GetModificationTime() != modTime) { doUpload = true; + BOX_TRACE(filename << ": will upload " + "(mod time changed)"); } // And just to catch really badly off clocks in @@ -838,9 +847,20 @@ rParams.mUploadAfterThisTimeInTheFuture) { doUpload = true; + BOX_TRACE(filename << ": will upload " + "(mod time in the future)"); } } + if (!doUpload) + { + BOX_TRACE(filename << ": will not upload " + "(no reason to upload, mod time is " + << modTime << " versus sync period " + << rParams.mSyncPeriodStart << " to " + << rParams.mSyncPeriodEnd << ")"); + } + if (doUpload) { // Make sure we're connected -- must connect here so we know whether From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:36:12 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:36:12 +0000 Subject: [Box Backup-commit] COMMIT r1431 - box/chris/general/test/bbackupd Message-ID: Author: chris Date: 2007-03-22 23:36:12 +0000 (Thu, 22 Mar 2007) New Revision: 1431 Modified: box/chris/general/test/bbackupd/testbbackupd.cpp Log: Merge back changes from merge tree: Test for expected behaviour when modifying untracked and tracked files without changing their modtime. Test that bbackupd notifies the sysadmin when throwing an exception. Flush stdout when necessary. Typo fixes in comments. Modified: box/chris/general/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-22 23:28:07 UTC (rev 1430) +++ box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-22 23:36:12 UTC (rev 1431) @@ -67,6 +67,7 @@ #include "autogen_BackupProtocolClient.h" #include "intercept.h" +#include "ServerControl.h" #include "MemLeakFindOn.h" @@ -80,15 +81,7 @@ void wait_for_backup_operation(int seconds = TIME_TO_WAIT_FOR_BACKUP_OPERATION) { - printf("waiting: "); - fflush(stdout); - for(int l = 0; l < seconds; ++l) - { - sleep(1); - printf("."); - fflush(stdout); - } - printf("\n"); + wait_for_operation(seconds); } int bbstored_pid = 0; @@ -635,7 +628,7 @@ TEST_THAT(TestFileExists("testfiles/bbackupd.pid")); - printf("Waiting for daemon to start"); + printf("Waiting for backup daemon to start: "); int pid = -1; for (int i = 0; i < 30; i++) @@ -651,7 +644,8 @@ } } - printf("\n"); + printf(" done.\n"); + fflush(stdout); TEST_THAT(pid > 0); return pid; @@ -1463,6 +1457,39 @@ TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + // Check that store errors are reported neatly + printf("Create store error\n"); + TEST_THAT(::rename("testfiles/0_0/backup/01234567/info.rf", + "testfiles/0_0/backup/01234567/info.rf.bak") == 0); + TEST_THAT(::rename("testfiles/0_1/backup/01234567/info.rf", + "testfiles/0_1/backup/01234567/info.rf.bak") == 0); + TEST_THAT(::rename("testfiles/0_2/backup/01234567/info.rf", + "testfiles/0_2/backup/01234567/info.rf.bak") == 0); + // Create a file to trigger an upload + { + int fd1 = open("testfiles/TestDir1/force-upload", + O_CREAT | O_EXCL | O_WRONLY, 0700); + TEST_THAT(fd1 > 0); + TEST_THAT(write(fd1, "just do it", 10) == 10); + TEST_THAT(close(fd1) == 0); + wait_for_backup_operation(4); + } + // Wait and test... + wait_for_backup_operation(); + // Check that it was reported correctly + TEST_THAT(TestFileExists("testfiles/notifyran.backup-error.1")); + // Check that the error was only reported once + TEST_THAT(!TestFileExists("testfiles/notifyran.backup-error.2")); + // Fix the store + TEST_THAT(::rename("testfiles/0_0/backup/01234567/info.rf.bak", + "testfiles/0_0/backup/01234567/info.rf") == 0); + TEST_THAT(::rename("testfiles/0_1/backup/01234567/info.rf.bak", + "testfiles/0_1/backup/01234567/info.rf") == 0); + TEST_THAT(::rename("testfiles/0_2/backup/01234567/info.rf.bak", + "testfiles/0_2/backup/01234567/info.rf") == 0); + // wait until bbackupd recovers from the exception + wait_for_backup_operation(100); + // Bad case: delete a file/symlink, replace it with a directory printf("==== Replace symlink with directory, add new directory\n"); #ifndef WIN32 @@ -1523,6 +1550,79 @@ TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + // rename an untracked file over an + // existing untracked file + printf("Rename over existing untracked file\n"); + int fd1 = open("testfiles/TestDir1/untracked-1", + O_CREAT | O_EXCL | O_WRONLY, 0700); + int fd2 = open("testfiles/TestDir1/untracked-2", + O_CREAT | O_EXCL | O_WRONLY, 0700); + TEST_THAT(fd1 > 0); + TEST_THAT(fd2 > 0); + TEST_THAT(write(fd1, "hello", 5) == 5); + TEST_THAT(close(fd1) == 0); + sleep(1); + TEST_THAT(write(fd2, "world", 5) == 5); + TEST_THAT(close(fd2) == 0); + TEST_THAT(TestFileExists("testfiles/TestDir1/untracked-1")); + TEST_THAT(TestFileExists("testfiles/TestDir1/untracked-2")); + wait_for_operation(5); + // back up both files + wait_for_backup_operation(); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf -l testfiles/query3t.log " + "\"compare -ac\" quit"); + TEST_THAT(compareReturnValue == 1*256); + TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(::rename("testfiles/TestDir1/untracked-1", + "testfiles/TestDir1/untracked-2") == 0); + TEST_THAT(!TestFileExists("testfiles/TestDir1/untracked-1")); + TEST_THAT( TestFileExists("testfiles/TestDir1/untracked-2")); + wait_for_backup_operation(); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf -l testfiles/query3t.log " + "\"compare -ac\" quit"); + TEST_THAT(compareReturnValue == 1*256); + TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + + // case which went wrong: rename a tracked file over an + // existing tracked file + printf("Rename over existing tracked file\n"); + fd1 = open("testfiles/TestDir1/tracked-1", + O_CREAT | O_EXCL | O_WRONLY, 0700); + fd2 = open("testfiles/TestDir1/tracked-2", + O_CREAT | O_EXCL | O_WRONLY, 0700); + TEST_THAT(fd1 > 0); + TEST_THAT(fd2 > 0); + char buffer[1024]; + TEST_THAT(write(fd1, "hello", 5) == 5); + TEST_THAT(write(fd1, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd1) == 0); + sleep(1); + TEST_THAT(write(fd2, "world", 5) == 5); + TEST_THAT(write(fd2, buffer, sizeof(buffer)) == sizeof(buffer)); + TEST_THAT(close(fd2) == 0); + TEST_THAT(TestFileExists("testfiles/TestDir1/tracked-1")); + TEST_THAT(TestFileExists("testfiles/TestDir1/tracked-2")); + wait_for_operation(5); + // back up both files + wait_for_backup_operation(); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf -l testfiles/query3u.log " + "\"compare -ac\" quit"); + TEST_THAT(compareReturnValue == 1*256); + TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(::rename("testfiles/TestDir1/tracked-1", + "testfiles/TestDir1/tracked-2") == 0); + TEST_THAT(!TestFileExists("testfiles/TestDir1/tracked-1")); + TEST_THAT( TestFileExists("testfiles/TestDir1/tracked-2")); + wait_for_backup_operation(); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf -l testfiles/query3v.log " + "\"compare -ac\" quit"); + TEST_THAT(compareReturnValue == 1*256); + TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + // case which went wrong: rename a tracked file over a deleted file printf("Rename an existing file over a deleted file\n"); #ifdef WIN32 @@ -1530,7 +1630,7 @@ #endif TEST_THAT(::rename("testfiles/TestDir1/df9834.dsf", "testfiles/TestDir1/x1/dsfdsfs98.fd") == 0); wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3s.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3w.log \"compare -ac\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -1715,6 +1815,7 @@ TEST_THAT(fd != -1); ::close(fd); } + // Wait and test... wait_for_backup_operation(); compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3e.log \"compare -ac\" quit"); @@ -1725,7 +1826,10 @@ // Check that it was reported correctly TEST_THAT(TestFileExists("testfiles/notifyran.read-error.1")); + + // Check that the error was only reported once TEST_THAT(!TestFileExists("testfiles/notifyran.read-error.2")); + // Set permissions on file and dir to stop errors in the future ::chmod("testfiles/TestDir1/sub23/read-fail-test-dir", 0770); ::chmod("testfiles/TestDir1/read-fail-test-file", 0770); @@ -1756,6 +1860,7 @@ sleep(1); } printf("\n"); + fflush(stdout); // Check there's a difference #ifdef WIN32 @@ -1783,6 +1888,7 @@ sleep(1); } printf("\n"); + fflush(stdout); #ifdef WIN32 compareReturnValue = ::system("perl testfiles/extcheck2.pl A"); From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:38:47 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:38:47 +0000 Subject: [Box Backup-commit] COMMIT r1432 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-22 23:38:47 +0000 (Thu, 22 Mar 2007) New Revision: 1432 Modified: box/chris/general/lib/common/Test.h Log: Flush stdout when necessary. Modified: box/chris/general/lib/common/Test.h =================================================================== --- box/chris/general/lib/common/Test.h 2007-03-22 23:36:12 UTC (rev 1431) +++ box/chris/general/lib/common/Test.h 2007-03-22 23:38:47 UTC (rev 1432) @@ -397,6 +397,7 @@ fflush(stdout); } printf("\n"); + fflush(stdout); } #endif // TEST__H From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:45:24 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:45:24 +0000 Subject: [Box Backup-commit] COMMIT r1433 - in box/chris/general: bin/bbackupd lib/common lib/server Message-ID: Author: chris Date: 2007-03-22 23:45:24 +0000 (Thu, 22 Mar 2007) New Revision: 1433 Added: box/chris/general/lib/common/ServerControl.h Modified: box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.cpp box/chris/general/bin/bbackupd/BackupDaemon.cpp box/chris/general/bin/bbackupd/BackupDaemon.h box/chris/general/lib/server/Daemon.cpp Log: Merge changes from chris/merge tree: Trace reasons for uploading (or not) each file. Log at trace level by default in debug builds. Add a new notification constant, NotifyEvent_BackupError, for use when an exception occurs during the backup. Make bbackupd notify sysadmin when an exception occurs during the backup, using this error code, and the notification string "backup-error". Change NotifyEvent__MAX to be one greater than the highest notification constant, makes code more maintainable. Modified: box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.cpp =================================================================== --- box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.cpp 2007-03-22 23:38:47 UTC (rev 1432) +++ box/chris/general/bin/bbackupd/BackupClientDirectoryRecord.cpp 2007-03-22 23:45:24 UTC (rev 1433) @@ -30,6 +30,7 @@ #include "BackupStoreException.h" #include "Archive.h" #include "PathUtils.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -793,10 +794,14 @@ if (pDirOnStore != 0 && en == 0) { doUpload = true; + BOX_TRACE(filename << ": will upload " + "(not on server)"); } else if (modTime >= rParams.mSyncPeriodStart) { doUpload = true; + BOX_TRACE(filename << ": will upload " + "(modified since last sync)"); } } @@ -813,6 +818,8 @@ > rParams.mMaxUploadWait) { doUpload = true; + BOX_TRACE(filename << ": will upload " + "(continually modified)"); } // Then make sure that if files are added with a @@ -828,6 +835,8 @@ en->GetModificationTime() != modTime) { doUpload = true; + BOX_TRACE(filename << ": will upload " + "(mod time changed)"); } // And just to catch really badly off clocks in @@ -838,9 +847,20 @@ rParams.mUploadAfterThisTimeInTheFuture) { doUpload = true; + BOX_TRACE(filename << ": will upload " + "(mod time in the future)"); } } + if (!doUpload) + { + BOX_TRACE(filename << ": will not upload " + "(no reason to upload, mod time is " + << modTime << " versus sync period " + << rParams.mSyncPeriodStart << " to " + << rParams.mSyncPeriodEnd << ")"); + } + if (doUpload) { // Make sure we're connected -- must connect here so we know whether Modified: box/chris/general/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/chris/general/bin/bbackupd/BackupDaemon.cpp 2007-03-22 23:38:47 UTC (rev 1432) +++ box/chris/general/bin/bbackupd/BackupDaemon.cpp 2007-03-22 23:45:24 UTC (rev 1433) @@ -119,8 +119,8 @@ // Only ever one instance of a daemon SSLLib::Initialise(); - // Initialise notifcation sent status - for(int l = 0; l <= NotifyEvent__MAX; ++l) + // Initialise notification sent status + for(int l = 0; l < NotifyEvent__MAX; ++l) { mNotificationsSent[l] = false; } @@ -963,6 +963,8 @@ else { // Not restart/terminate, pause and retry + // Notify administrator + NotifySysadmin(NotifyEvent_BackupError); SetState(State_Error); BOX_ERROR("Exception caught (" << errorString @@ -2092,11 +2094,17 @@ // -------------------------------------------------------------------------- void BackupDaemon::NotifySysadmin(int Event) { - static const char *sEventNames[] = {"store-full", "read-error", 0}; + static const char *sEventNames[] = + { + "store-full", + "read-error", + "backup-error", + 0 + }; BOX_TRACE("BackupDaemon::NotifySysadmin() called, event = " << Event); - if(Event < 0 || Event > NotifyEvent__MAX) + if(Event < 0 || Event >= NotifyEvent__MAX) { THROW_EXCEPTION(BackupStoreException, BadNotifySysadminEventCode); } Modified: box/chris/general/bin/bbackupd/BackupDaemon.h =================================================================== --- box/chris/general/bin/bbackupd/BackupDaemon.h 2007-03-22 23:38:47 UTC (rev 1432) +++ box/chris/general/bin/bbackupd/BackupDaemon.h 2007-03-22 23:45:24 UTC (rev 1433) @@ -82,8 +82,9 @@ enum { NotifyEvent_StoreFull = 0, - NotifyEvent_ReadError = 1, - NotifyEvent__MAX = 1 + NotifyEvent_ReadError, + NotifyEvent_BackupError, + NotifyEvent__MAX // When adding notifications, remember to add strings to NotifySysadmin() }; void NotifySysadmin(int Event); @@ -177,7 +178,7 @@ CommandSocketInfo *mpCommandSocketInfo; // Stop notifications being repeated. - bool mNotificationsSent[NotifyEvent__MAX + 1]; + bool mNotificationsSent[NotifyEvent__MAX]; // Unused entries in the root directory wait a while before being deleted box_time_t mDeleteUnusedRootDirEntriesAfter; // time to delete them Copied: box/chris/general/lib/common/ServerControl.h (from rev 1430, box/chris/merge/lib/common/ServerControl.h) =================================================================== --- box/chris/general/lib/common/ServerControl.h (rev 0) +++ box/chris/general/lib/common/ServerControl.h 2007-03-22 23:45:24 UTC (rev 1433) @@ -0,0 +1,177 @@ +#ifndef SERVER_CONTROL_H +#define SERVER_CONTROL_H + +#ifdef WIN32 + +#include "WinNamedPipeStream.h" +#include "IOStreamGetLine.h" +#include "BoxPortsAndFiles.h" +#include "Test.h" + +static bool SendCommands(const std::string& rCmd) +{ + WinNamedPipeStream connection; + + try + { + connection.Connect(BOX_NAMED_PIPE_NAME); + } + catch(...) + { + printf("Failed to connect to daemon control socket.\n"); + return false; + } + + // For receiving data + IOStreamGetLine getLine(connection); + + // Wait for the configuration summary + std::string configSummary; + if(!getLine.GetLine(configSummary)) + { + printf("Failed to receive configuration summary from daemon\n"); + return false; + } + + // Was the connection rejected by the server? + if(getLine.IsEOF()) + { + printf("Server rejected the connection.\n"); + return false; + } + + // Decode it + int autoBackup, updateStoreInterval, minimumFileAge, maxUploadWait; + if(::sscanf(configSummary.c_str(), "bbackupd: %d %d %d %d", + &autoBackup, &updateStoreInterval, + &minimumFileAge, &maxUploadWait) != 4) + { + printf("Config summary didn't decode\n"); + return false; + } + + std::string cmds; + bool expectResponse; + + if (rCmd != "") + { + cmds = rCmd; + cmds += "\nquit\n"; + expectResponse = true; + } + else + { + cmds = "quit\n"; + expectResponse = false; + } + + connection.Write(cmds.c_str(), cmds.size()); + + // Read the response + std::string line; + bool statusOk = !expectResponse; + + while (expectResponse && !getLine.IsEOF() && getLine.GetLine(line)) + { + // Is this an OK or error line? + if (line == "ok") + { + statusOk = true; + } + else if (line == "error") + { + printf("ERROR (%s)\n", rCmd.c_str()); + break; + } + else + { + printf("WARNING: Unexpected response to command '%s': " + "%s", rCmd.c_str(), line.c_str()); + } + } + + return statusOk; +} + +inline bool HUPServer(int pid) +{ + return SendCommands("reload"); +} + +inline bool KillServerInternal(int pid) +{ + HANDLE hProcess = OpenProcess(PROCESS_TERMINATE, false, pid); + if (hProcess == NULL) + { + printf("Failed to open process %d: error %d\n", + pid, (int)GetLastError()); + return false; + } + + if (!TerminateProcess(hProcess, 1)) + { + printf("Failed to terminate process %d: error %d\n", + pid, (int)GetLastError()); + CloseHandle(hProcess); + return false; + } + + CloseHandle(hProcess); + return true; +} + +#else // !WIN32 + +inline bool HUPServer(int pid) +{ + if(pid == 0) return false; + return ::kill(pid, SIGHUP) == 0; +} + +inline bool KillServerInternal(int pid) +{ + if(pid == 0 || pid == -1) return false; + bool killed = (::kill(pid, SIGTERM) == 0); + TEST_THAT(killed); + return killed; +} + +#endif // WIN32 + +inline bool KillServer(int pid) +{ + if (!KillServerInternal(pid)) + { + return false; + } + + for (int i = 0; i < 30; i++) + { + if (!ServerIsAlive(pid)) break; + ::sleep(1); + if (!ServerIsAlive(pid)) break; + + if (i == 0) + { + printf("waiting for server to die"); + } + + printf("."); + fflush(stdout); + } + + if (!ServerIsAlive(pid)) + { + printf("done.\n"); + } + else + { + printf("failed!\n"); + } + + fflush(stdout); + + return !ServerIsAlive(pid); +} + +#endif // SERVER_CONTROL_H Modified: box/chris/general/lib/server/Daemon.cpp =================================================================== --- box/chris/general/lib/server/Daemon.cpp 2007-03-22 23:38:47 UTC (rev 1432) +++ box/chris/general/lib/server/Daemon.cpp 2007-03-22 23:45:24 UTC (rev 1433) @@ -101,7 +101,13 @@ mConfigFileName = DefaultConfigFile; bool haveConfigFile = false; bool singleProcess = false; + + #ifdef NDEBUG int masterLevel = Log::NOTICE; // need an int to do math with + #else + int masterLevel = Log::TRACE; // need an int to do math with + #endif + char c; while((c = getopt(argc, (char * const *)argv, "c:Dqv")) != -1) From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:46:55 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:46:55 +0000 Subject: [Box Backup-commit] COMMIT r1434 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-22 23:46:55 +0000 (Thu, 22 Mar 2007) New Revision: 1434 Removed: box/chris/general/lib/common/ServerControl.h Log: Remove ServerControl.h, now in lib/server Deleted: box/chris/general/lib/common/ServerControl.h =================================================================== --- box/chris/general/lib/common/ServerControl.h 2007-03-22 23:45:24 UTC (rev 1433) +++ box/chris/general/lib/common/ServerControl.h 2007-03-22 23:46:55 UTC (rev 1434) @@ -1,177 +0,0 @@ -#ifndef SERVER_CONTROL_H -#define SERVER_CONTROL_H - -#ifdef WIN32 - -#include "WinNamedPipeStream.h" -#include "IOStreamGetLine.h" -#include "BoxPortsAndFiles.h" -#include "Test.h" - -static bool SendCommands(const std::string& rCmd) -{ - WinNamedPipeStream connection; - - try - { - connection.Connect(BOX_NAMED_PIPE_NAME); - } - catch(...) - { - printf("Failed to connect to daemon control socket.\n"); - return false; - } - - // For receiving data - IOStreamGetLine getLine(connection); - - // Wait for the configuration summary - std::string configSummary; - if(!getLine.GetLine(configSummary)) - { - printf("Failed to receive configuration summary from daemon\n"); - return false; - } - - // Was the connection rejected by the server? - if(getLine.IsEOF()) - { - printf("Server rejected the connection.\n"); - return false; - } - - // Decode it - int autoBackup, updateStoreInterval, minimumFileAge, maxUploadWait; - if(::sscanf(configSummary.c_str(), "bbackupd: %d %d %d %d", - &autoBackup, &updateStoreInterval, - &minimumFileAge, &maxUploadWait) != 4) - { - printf("Config summary didn't decode\n"); - return false; - } - - std::string cmds; - bool expectResponse; - - if (rCmd != "") - { - cmds = rCmd; - cmds += "\nquit\n"; - expectResponse = true; - } - else - { - cmds = "quit\n"; - expectResponse = false; - } - - connection.Write(cmds.c_str(), cmds.size()); - - // Read the response - std::string line; - bool statusOk = !expectResponse; - - while (expectResponse && !getLine.IsEOF() && getLine.GetLine(line)) - { - // Is this an OK or error line? - if (line == "ok") - { - statusOk = true; - } - else if (line == "error") - { - printf("ERROR (%s)\n", rCmd.c_str()); - break; - } - else - { - printf("WARNING: Unexpected response to command '%s': " - "%s", rCmd.c_str(), line.c_str()); - } - } - - return statusOk; -} - -inline bool HUPServer(int pid) -{ - return SendCommands("reload"); -} - -inline bool KillServerInternal(int pid) -{ - HANDLE hProcess = OpenProcess(PROCESS_TERMINATE, false, pid); - if (hProcess == NULL) - { - printf("Failed to open process %d: error %d\n", - pid, (int)GetLastError()); - return false; - } - - if (!TerminateProcess(hProcess, 1)) - { - printf("Failed to terminate process %d: error %d\n", - pid, (int)GetLastError()); - CloseHandle(hProcess); - return false; - } - - CloseHandle(hProcess); - return true; -} - -#else // !WIN32 - -inline bool HUPServer(int pid) -{ - if(pid == 0) return false; - return ::kill(pid, SIGHUP) == 0; -} - -inline bool KillServerInternal(int pid) -{ - if(pid == 0 || pid == -1) return false; - bool killed = (::kill(pid, SIGTERM) == 0); - TEST_THAT(killed); - return killed; -} - -#endif // WIN32 - -inline bool KillServer(int pid) -{ - if (!KillServerInternal(pid)) - { - return false; - } - - for (int i = 0; i < 30; i++) - { - if (!ServerIsAlive(pid)) break; - ::sleep(1); - if (!ServerIsAlive(pid)) break; - - if (i == 0) - { - printf("waiting for server to die"); - } - - printf("."); - fflush(stdout); - } - - if (!ServerIsAlive(pid)) - { - printf("done.\n"); - } - else - { - printf("failed!\n"); - } - - fflush(stdout); - - return !ServerIsAlive(pid); -} - -#endif // SERVER_CONTROL_H From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:48:12 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:48:12 +0000 Subject: [Box Backup-commit] COMMIT r1435 - in box/chris/merge/lib: common server Message-ID: Author: chris Date: 2007-03-22 23:48:12 +0000 (Thu, 22 Mar 2007) New Revision: 1435 Added: box/chris/merge/lib/server/ServerControl.h Removed: box/chris/merge/lib/common/ServerControl.h Log: Move lib/common/ServerControl.h to lib/server where it belongs, since it uses server functions (WinNamedPipeStream on win32) (refs #3) Deleted: box/chris/merge/lib/common/ServerControl.h =================================================================== --- box/chris/merge/lib/common/ServerControl.h 2007-03-22 23:46:55 UTC (rev 1434) +++ box/chris/merge/lib/common/ServerControl.h 2007-03-22 23:48:12 UTC (rev 1435) @@ -1,177 +0,0 @@ -#ifndef SERVER_CONTROL_H -#define SERVER_CONTROL_H - -#ifdef WIN32 - -#include "WinNamedPipeStream.h" -#include "IOStreamGetLine.h" -#include "BoxPortsAndFiles.h" -#include "Test.h" - -static bool SendCommands(const std::string& rCmd) -{ - WinNamedPipeStream connection; - - try - { - connection.Connect(BOX_NAMED_PIPE_NAME); - } - catch(...) - { - printf("Failed to connect to daemon control socket.\n"); - return false; - } - - // For receiving data - IOStreamGetLine getLine(connection); - - // Wait for the configuration summary - std::string configSummary; - if(!getLine.GetLine(configSummary)) - { - printf("Failed to receive configuration summary from daemon\n"); - return false; - } - - // Was the connection rejected by the server? - if(getLine.IsEOF()) - { - printf("Server rejected the connection.\n"); - return false; - } - - // Decode it - int autoBackup, updateStoreInterval, minimumFileAge, maxUploadWait; - if(::sscanf(configSummary.c_str(), "bbackupd: %d %d %d %d", - &autoBackup, &updateStoreInterval, - &minimumFileAge, &maxUploadWait) != 4) - { - printf("Config summary didn't decode\n"); - return false; - } - - std::string cmds; - bool expectResponse; - - if (rCmd != "") - { - cmds = rCmd; - cmds += "\nquit\n"; - expectResponse = true; - } - else - { - cmds = "quit\n"; - expectResponse = false; - } - - connection.Write(cmds.c_str(), cmds.size()); - - // Read the response - std::string line; - bool statusOk = !expectResponse; - - while (expectResponse && !getLine.IsEOF() && getLine.GetLine(line)) - { - // Is this an OK or error line? - if (line == "ok") - { - statusOk = true; - } - else if (line == "error") - { - printf("ERROR (%s)\n", rCmd.c_str()); - break; - } - else - { - printf("WARNING: Unexpected response to command '%s': " - "%s", rCmd.c_str(), line.c_str()); - } - } - - return statusOk; -} - -inline bool HUPServer(int pid) -{ - return SendCommands("reload"); -} - -inline bool KillServerInternal(int pid) -{ - HANDLE hProcess = OpenProcess(PROCESS_TERMINATE, false, pid); - if (hProcess == NULL) - { - printf("Failed to open process %d: error %d\n", - pid, (int)GetLastError()); - return false; - } - - if (!TerminateProcess(hProcess, 1)) - { - printf("Failed to terminate process %d: error %d\n", - pid, (int)GetLastError()); - CloseHandle(hProcess); - return false; - } - - CloseHandle(hProcess); - return true; -} - -#else // !WIN32 - -inline bool HUPServer(int pid) -{ - if(pid == 0) return false; - return ::kill(pid, SIGHUP) == 0; -} - -inline bool KillServerInternal(int pid) -{ - if(pid == 0 || pid == -1) return false; - bool killed = (::kill(pid, SIGTERM) == 0); - TEST_THAT(killed); - return killed; -} - -#endif // WIN32 - -inline bool KillServer(int pid) -{ - if (!KillServerInternal(pid)) - { - return false; - } - - for (int i = 0; i < 30; i++) - { - if (!ServerIsAlive(pid)) break; - ::sleep(1); - if (!ServerIsAlive(pid)) break; - - if (i == 0) - { - printf("waiting for server to die"); - } - - printf("."); - fflush(stdout); - } - - if (!ServerIsAlive(pid)) - { - printf("done.\n"); - } - else - { - printf("failed!\n"); - } - - fflush(stdout); - - return !ServerIsAlive(pid); -} - -#endif // SERVER_CONTROL_H Copied: box/chris/merge/lib/server/ServerControl.h (from rev 1432, box/chris/merge/lib/common/ServerControl.h) =================================================================== --- box/chris/merge/lib/server/ServerControl.h (rev 0) +++ box/chris/merge/lib/server/ServerControl.h 2007-03-22 23:48:12 UTC (rev 1435) @@ -0,0 +1,177 @@ +#ifndef SERVER_CONTROL_H +#define SERVER_CONTROL_H + +#ifdef WIN32 + +#include "WinNamedPipeStream.h" +#include "IOStreamGetLine.h" +#include "BoxPortsAndFiles.h" +#include "Test.h" + +static bool SendCommands(const std::string& rCmd) +{ + WinNamedPipeStream connection; + + try + { + connection.Connect(BOX_NAMED_PIPE_NAME); + } + catch(...) + { + printf("Failed to connect to daemon control socket.\n"); + return false; + } + + // For receiving data + IOStreamGetLine getLine(connection); + + // Wait for the configuration summary + std::string configSummary; + if(!getLine.GetLine(configSummary)) + { + printf("Failed to receive configuration summary from daemon\n"); + return false; + } + + // Was the connection rejected by the server? + if(getLine.IsEOF()) + { + printf("Server rejected the connection.\n"); + return false; + } + + // Decode it + int autoBackup, updateStoreInterval, minimumFileAge, maxUploadWait; + if(::sscanf(configSummary.c_str(), "bbackupd: %d %d %d %d", + &autoBackup, &updateStoreInterval, + &minimumFileAge, &maxUploadWait) != 4) + { + printf("Config summary didn't decode\n"); + return false; + } + + std::string cmds; + bool expectResponse; + + if (rCmd != "") + { + cmds = rCmd; + cmds += "\nquit\n"; + expectResponse = true; + } + else + { + cmds = "quit\n"; + expectResponse = false; + } + + connection.Write(cmds.c_str(), cmds.size()); + + // Read the response + std::string line; + bool statusOk = !expectResponse; + + while (expectResponse && !getLine.IsEOF() && getLine.GetLine(line)) + { + // Is this an OK or error line? + if (line == "ok") + { + statusOk = true; + } + else if (line == "error") + { + printf("ERROR (%s)\n", rCmd.c_str()); + break; + } + else + { + printf("WARNING: Unexpected response to command '%s': " + "%s", rCmd.c_str(), line.c_str()); + } + } + + return statusOk; +} + +inline bool HUPServer(int pid) +{ + return SendCommands("reload"); +} + +inline bool KillServerInternal(int pid) +{ + HANDLE hProcess = OpenProcess(PROCESS_TERMINATE, false, pid); + if (hProcess == NULL) + { + printf("Failed to open process %d: error %d\n", + pid, (int)GetLastError()); + return false; + } + + if (!TerminateProcess(hProcess, 1)) + { + printf("Failed to terminate process %d: error %d\n", + pid, (int)GetLastError()); + CloseHandle(hProcess); + return false; + } + + CloseHandle(hProcess); + return true; +} + +#else // !WIN32 + +inline bool HUPServer(int pid) +{ + if(pid == 0) return false; + return ::kill(pid, SIGHUP) == 0; +} + +inline bool KillServerInternal(int pid) +{ + if(pid == 0 || pid == -1) return false; + bool killed = (::kill(pid, SIGTERM) == 0); + TEST_THAT(killed); + return killed; +} + +#endif // WIN32 + +inline bool KillServer(int pid) +{ + if (!KillServerInternal(pid)) + { + return false; + } + + for (int i = 0; i < 30; i++) + { + if (!ServerIsAlive(pid)) break; + ::sleep(1); + if (!ServerIsAlive(pid)) break; + + if (i == 0) + { + printf("waiting for server to die"); + } + + printf("."); + fflush(stdout); + } + + if (!ServerIsAlive(pid)) + { + printf("done.\n"); + } + else + { + printf("failed!\n"); + } + + fflush(stdout); + + return !ServerIsAlive(pid); +} + +#endif // SERVER_CONTROL_H From boxbackup-dev at fluffy.co.uk Thu Mar 22 23:57:14 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Thu, 22 Mar 2007 23:57:14 +0000 Subject: [Box Backup-commit] COMMIT r1436 - box/chris/general/lib/win32 Message-ID: Author: chris Date: 2007-03-22 23:57:13 +0000 (Thu, 22 Mar 2007) New Revision: 1436 Modified: box/chris/general/lib/win32/emu.cpp box/chris/general/lib/win32/emu.h Log: Add emulated rename() with path conversion Modified: box/chris/general/lib/win32/emu.cpp =================================================================== --- box/chris/general/lib/win32/emu.cpp 2007-03-22 23:48:12 UTC (rev 1435) +++ box/chris/general/lib/win32/emu.cpp 2007-03-22 23:57:13 UTC (rev 1436) @@ -1609,6 +1609,72 @@ return 0; } +int emu_rename(const char* pOldFileName, const char* pNewFileName) +{ + std::string OldFilePathWithUnicode = + ConvertPathToAbsoluteUnicode(pOldFileName); + + if (OldPathWithUnicode.size() == 0) + { + // error already logged by ConvertPathToAbsoluteUnicode() + return -1; + } + + WCHAR* pOldBuffer = ConvertUtf8ToWideString(OldPathWithUnicode.c_str()); + if (!pOldBuffer) + { + return -1; + } + + std::string NewFilePathWithUnicode = + ConvertPathToAbsoluteUnicode(pNewFileName); + + if (NewPathWithUnicode.size() == 0) + { + // error already logged by ConvertPathToAbsoluteUnicode() + delete [] pOldBuffer; + return -1; + } + + WCHAR* pNewBuffer = ConvertUtf8ToWideString(NewPathWithUnicode.c_str()); + if (!pNewBuffer) + { + delete [] pOldBuffer; + return -1; + } + + BOOL result = MoveFileW(pOldBuffer, pNewBuffer); + DWORD err = GetLastError(); + delete [] pOldBuffer; + delete [] pNewBuffer; + + if (!result) + { + if (err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND) + { + errno = ENOENT; + } + else if (err == ERROR_SHARING_VIOLATION) + { + errno = EBUSY; + } + else if (err == ERROR_ACCESS_DENIED) + { + errno = EACCES; + } + else + { + ::syslog(LOG_WARNING, "Failed to rename file " + "'%s' to '%s': %s", pOldFileName, pNewFileName, + GetErrorMessage(err).c_str()); + errno = ENOSYS; + } + return -1; + } + + return 0; +} + int console_read(char* pBuffer, size_t BufferSize) { HANDLE hConsole = GetStdHandle(STD_INPUT_HANDLE); Modified: box/chris/general/lib/win32/emu.h =================================================================== --- box/chris/general/lib/win32/emu.h 2007-03-22 23:48:12 UTC (rev 1435) +++ box/chris/general/lib/win32/emu.h 2007-03-22 23:57:13 UTC (rev 1436) @@ -337,6 +337,7 @@ int emu_utimes (const char* pName, const struct timeval[]); int emu_chmod (const char* pName, mode_t mode); char* emu_getcwd (char* pBuffer, int BufSize); +char* emu_rename (const char* pOldName, const char* pNewName); #define chdir(directory) emu_chdir (directory) #define mkdir(path, mode) emu_mkdir (path) @@ -347,6 +348,7 @@ #define utimes(buffer, times) emu_utimes (buffer, times) #define chmod(file, mode) emu_chmod (file, mode) #define getcwd(buffer, size) emu_getcwd (buffer, size) +#define rename(oldname, newname) emu_rename (oldname, newname) int statfs(const char * name, struct statfs * s); From boxbackup-dev at fluffy.co.uk Sat Mar 24 00:40:59 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 00:40:59 +0000 Subject: [Box Backup-commit] COMMIT r1437 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-24 00:40:59 +0000 (Sat, 24 Mar 2007) New Revision: 1437 Modified: box/chris/merge/lib/common/Guards.h Log: Fix compilation error reported by Torsten Boob (refs #3) Modified: box/chris/merge/lib/common/Guards.h =================================================================== --- box/chris/merge/lib/common/Guards.h 2007-03-22 23:57:13 UTC (rev 1436) +++ box/chris/merge/lib/common/Guards.h 2007-03-24 00:40:59 UTC (rev 1437) @@ -15,9 +15,12 @@ #include #endif +#include #include #include +#include #include + #include #include "CommonException.h" From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:36:17 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:36:17 +0000 Subject: [Box Backup-commit] COMMIT r1438 - box/chris/general/lib/win32 Message-ID: Author: chris Date: 2007-03-24 14:36:17 +0000 (Sat, 24 Mar 2007) New Revision: 1438 Modified: box/chris/general/lib/win32/emu.cpp box/chris/general/lib/win32/emu.h Log: Compile fix (wrong variable names in emu_rename) Compile fix (wrong return type of emu_rename in emu.h prototype) Modified: box/chris/general/lib/win32/emu.cpp =================================================================== --- box/chris/general/lib/win32/emu.cpp 2007-03-24 00:40:59 UTC (rev 1437) +++ box/chris/general/lib/win32/emu.cpp 2007-03-24 14:36:17 UTC (rev 1438) @@ -1611,7 +1611,7 @@ int emu_rename(const char* pOldFileName, const char* pNewFileName) { - std::string OldFilePathWithUnicode = + std::string OldPathWithUnicode = ConvertPathToAbsoluteUnicode(pOldFileName); if (OldPathWithUnicode.size() == 0) @@ -1626,7 +1626,7 @@ return -1; } - std::string NewFilePathWithUnicode = + std::string NewPathWithUnicode = ConvertPathToAbsoluteUnicode(pNewFileName); if (NewPathWithUnicode.size() == 0) Modified: box/chris/general/lib/win32/emu.h =================================================================== --- box/chris/general/lib/win32/emu.h 2007-03-24 00:40:59 UTC (rev 1437) +++ box/chris/general/lib/win32/emu.h 2007-03-24 14:36:17 UTC (rev 1438) @@ -337,7 +337,7 @@ int emu_utimes (const char* pName, const struct timeval[]); int emu_chmod (const char* pName, mode_t mode); char* emu_getcwd (char* pBuffer, int BufSize); -char* emu_rename (const char* pOldName, const char* pNewName); +int emu_rename (const char* pOldName, const char* pNewName); #define chdir(directory) emu_chdir (directory) #define mkdir(path, mode) emu_mkdir (path) From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:37:04 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:37:04 +0000 Subject: [Box Backup-commit] COMMIT r1439 - box/chris/general/lib/backupclient Message-ID: Author: chris Date: 2007-03-24 14:37:04 +0000 (Sat, 24 Mar 2007) New Revision: 1439 Modified: box/chris/general/lib/backupclient/BackupStoreFile.cpp Log: Use logging framework to reduce "Reallocating EncodingBuffer" noise for those who don't want it Modified: box/chris/general/lib/backupclient/BackupStoreFile.cpp =================================================================== --- box/chris/general/lib/backupclient/BackupStoreFile.cpp 2007-03-24 14:36:17 UTC (rev 1438) +++ box/chris/general/lib/backupclient/BackupStoreFile.cpp 2007-03-24 14:37:04 UTC (rev 1439) @@ -46,6 +46,7 @@ #include "ReadGatherStream.h" #include "Random.h" #include "BackupStoreFileEncodeStream.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -1485,9 +1486,8 @@ // -------------------------------------------------------------------------- void BackupStoreFile::EncodingBuffer::Reallocate(int NewSize) { -#ifndef WIN32 - TRACE2("Reallocating EncodingBuffer from %d to %d\n", mBufferSize, NewSize); -#endif + BOX_TRACE("Reallocating EncodingBuffer from " << mBufferSize << + " to " << NewSize); ASSERT(mpBuffer != 0); uint8_t *buffer = (uint8_t*)BackupStoreFile::CodingChunkAlloc(NewSize); if(buffer == 0) From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:37:44 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:37:44 +0000 Subject: [Box Backup-commit] COMMIT r1440 - box/chris/general/lib/backupclient Message-ID: Author: chris Date: 2007-03-24 14:37:44 +0000 (Sat, 24 Mar 2007) New Revision: 1440 Modified: box/chris/general/lib/backupclient/BackupStoreFilenameClear.cpp Log: Use logging framework to remove "Reallocating filename encoding/decoding buffer" noise for those who don't want it Modified: box/chris/general/lib/backupclient/BackupStoreFilenameClear.cpp =================================================================== --- box/chris/general/lib/backupclient/BackupStoreFilenameClear.cpp 2007-03-24 14:37:04 UTC (rev 1439) +++ box/chris/general/lib/backupclient/BackupStoreFilenameClear.cpp 2007-03-24 14:37:44 UTC (rev 1440) @@ -13,6 +13,7 @@ #include "CipherContext.h" #include "CipherBlowfish.h" #include "Guards.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -203,9 +204,9 @@ { if(sEncDecBufferSize < BufSize) { -#ifndef WIN32 - TRACE2("Reallocating filename encoding/decoding buffer from %d to %d\n", sEncDecBufferSize, BufSize); -#endif + BOX_TRACE("Reallocating filename encoding/decoding " + "buffer from " << sEncDecBufferSize << + " to " << BufSize); spEncDecBuffer->Resize(BufSize); sEncDecBufferSize = BufSize; MEMLEAKFINDER_NOT_A_LEAK(*spEncDecBuffer); From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:38:38 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:38:38 +0000 Subject: [Box Backup-commit] COMMIT r1441 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-24 14:38:37 +0000 (Sat, 24 Mar 2007) New Revision: 1441 Modified: box/chris/general/lib/server/Daemon.cpp Log: Reduce default logging level in debug builds from TRACE back down to INFO, to reduce noise in tests Modified: box/chris/general/lib/server/Daemon.cpp =================================================================== --- box/chris/general/lib/server/Daemon.cpp 2007-03-24 14:37:44 UTC (rev 1440) +++ box/chris/general/lib/server/Daemon.cpp 2007-03-24 14:38:37 UTC (rev 1441) @@ -105,7 +105,7 @@ #ifdef NDEBUG int masterLevel = Log::NOTICE; // need an int to do math with #else - int masterLevel = Log::TRACE; // need an int to do math with + int masterLevel = Log::INFO; // need an int to do math with #endif char c; From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:39:08 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:39:08 +0000 Subject: [Box Backup-commit] COMMIT r1442 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-24 14:39:08 +0000 (Sat, 24 Mar 2007) New Revision: 1442 Modified: box/chris/general/lib/server/WinNamedPipeStream.cpp Log: Use memmove() for overlapping buffers as pointed out by Charles (thanks!) Modified: box/chris/general/lib/server/WinNamedPipeStream.cpp =================================================================== --- box/chris/general/lib/server/WinNamedPipeStream.cpp 2007-03-24 14:38:37 UTC (rev 1441) +++ box/chris/general/lib/server/WinNamedPipeStream.cpp 2007-03-24 14:39:08 UTC (rev 1442) @@ -293,7 +293,7 @@ } memcpy(pBuffer, mReadBuffer, BytesToCopy); - memcpy(mReadBuffer, mReadBuffer + BytesToCopy, BytesRemaining); + memmove(mReadBuffer, mReadBuffer + BytesToCopy, BytesRemaining); mBytesInBuffer = BytesRemaining; NumBytesRead = BytesToCopy; From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:39:50 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:39:50 +0000 Subject: [Box Backup-commit] COMMIT r1443 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-24 14:39:50 +0000 (Sat, 24 Mar 2007) New Revision: 1443 Modified: box/chris/general/lib/server/Protocol.cpp Log: Use logging framework to remove "Send block allocation size" noise for those who don't want it Modified: box/chris/general/lib/server/Protocol.cpp =================================================================== --- box/chris/general/lib/server/Protocol.cpp 2007-03-24 14:39:08 UTC (rev 1442) +++ box/chris/general/lib/server/Protocol.cpp 2007-03-24 14:39:50 UTC (rev 1443) @@ -22,6 +22,7 @@ #include "ServerException.h" #include "PartialReadStream.h" #include "ProtocolUncertainStream.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -55,7 +56,8 @@ mLastErrorType(NoError), mLastErrorSubType(NoError) { - TRACE1("Send block allocation size is %d\n", PROTOCOL_ALLOCATE_SEND_BLOCK_CHUNK); + BOX_TRACE("Send block allocation size is " << + PROTOCOL_ALLOCATE_SEND_BLOCK_CHUNK); } // -------------------------------------------------------------------------- From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:40:12 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:40:12 +0000 Subject: [Box Backup-commit] COMMIT r1444 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-24 14:40:10 +0000 (Sat, 24 Mar 2007) New Revision: 1444 Modified: box/chris/general/lib/common/Test.h Log: Trivial code simplification Modified: box/chris/general/lib/common/Test.h =================================================================== --- box/chris/general/lib/common/Test.h 2007-03-24 14:39:50 UTC (rev 1443) +++ box/chris/general/lib/common/Test.h 2007-03-24 14:40:10 UTC (rev 1444) @@ -275,11 +275,9 @@ TEST_FAIL_WITH_MESSAGE("Server didn't save PID file"); return -1; } - else - { - ::fprintf(stdout, "done.\n"); - } + ::fprintf(stdout, "done.\n"); + // wait a second for the pid to be written to the file ::sleep(1); From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:40:39 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:40:39 +0000 Subject: [Box Backup-commit] COMMIT r1445 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-24 14:40:39 +0000 (Sat, 24 Mar 2007) New Revision: 1445 Modified: box/chris/general/lib/common/Timer.cpp Log: Use logging framework to remove timer noise for those who don't want it Modified: box/chris/general/lib/common/Timer.cpp =================================================================== --- box/chris/general/lib/common/Timer.cpp 2007-03-24 14:40:10 UTC (rev 1444) +++ box/chris/general/lib/common/Timer.cpp 2007-03-24 14:40:39 UTC (rev 1445) @@ -175,11 +175,10 @@ if (timeToExpiry <= 0) { - TRACE3("%d.%d: timer %p has expired, " - "triggering it\n", - (int)(timeNow / 1000000), - (int)(timeNow % 1000000), - *i); + BOX_TRACE((int)(timeNow / 1000000) << "." << + (int)(timeNow % 1000000) << + ": timer " << *i << " has expired, " + "triggering it"); rTimer.OnExpire(); spTimers->erase(i); restart = true; @@ -187,13 +186,13 @@ } else { - TRACE5("%d.%d: timer %p has not expired, " - "triggering in %d.%d seconds\n", - (int)(timeNow / 1000000), - (int)(timeNow % 1000000), - *i, - (int)(timeToExpiry / 1000000), - (int)(timeToExpiry % 1000000)); + BOX_TRACE((int)(timeNow / 1000000) << "." << + (int)(timeNow % 1000000) << + ": timer " << *i << " has not " + "expired, triggering in " << + (int)(timeToExpiry / 1000000) << "." << + (int)(timeToExpiry % 1000000) << + " seconds"); } } } @@ -231,7 +230,7 @@ if(::setitimer(ITIMER_REAL, &timeout, NULL) != 0) { - TRACE0("WARNING: couldn't initialise timer\n"); + BOX_ERROR("Failed to initialise timer\n"); THROW_EXCEPTION(CommonException, Internal) } } @@ -263,15 +262,16 @@ gettimeofday(&tv, NULL); if (timeoutSecs == 0) { - TRACE4("%d.%d: timer %p initialised for %d secs, " - "will not fire\n", tv.tv_sec, tv.tv_usec, this, - timeoutSecs); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised for " << + timeoutSecs << " secs, will not fire"); } else { - TRACE6("%d.%d: timer %p initialised for %d secs, " - "to fire at %d.%d\n", tv.tv_sec, tv.tv_usec, this, - timeoutSecs, (int)(mExpires / 1000000), + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised for " << + timeoutSecs << " secs, to fire at " << + (int)(mExpires / 1000000) << "." << (int)(mExpires % 1000000)); } #endif @@ -291,8 +291,8 @@ #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); - TRACE3("%d.%d: timer %p destroyed, will not fire\n", - tv.tv_sec, tv.tv_usec, this); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " destroyed"); #endif Timers::Remove(*this); @@ -307,21 +307,22 @@ gettimeofday(&tv, NULL); if (mExpired) { - TRACE4("%d.%d: timer %p initialised from timer %p, " - "already expired, will not fire\n", tv.tv_sec, - tv.tv_usec, this, &rToCopy); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << ", already expired, will not fire"); } else if (mExpires == 0) { - TRACE4("%d.%d: timer %p initialised from timer %p, " - "will not fire\n", tv.tv_sec, tv.tv_usec, this, - &rToCopy); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << ", no expiry, will not fire"); } else { - TRACE6("%d.%d: timer %p initialised from timer %p, " - "to fire at %d.%d\n", tv.tv_sec, tv.tv_usec, this, - &rToCopy, (int)(mExpires / 1000000), + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << " to fire at " << + (int)(mExpires / 1000000) << "." << (int)(mExpires % 1000000)); } #endif @@ -339,21 +340,22 @@ gettimeofday(&tv, NULL); if (rToCopy.mExpired) { - TRACE4("%d.%d: timer %p initialised from timer %p, " - "already expired, will not fire\n", tv.tv_sec, - tv.tv_usec, this, &rToCopy); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << ", already expired, will not fire"); } else if (rToCopy.mExpires == 0) { - TRACE4("%d.%d: timer %p initialised from timer %p, " - "will not fire\n", tv.tv_sec, tv.tv_usec, this, - &rToCopy); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << ", no expiry, will not fire"); } else { - TRACE6("%d.%d: timer %p initialised from timer %p, " - "to fire at %d.%d\n", tv.tv_sec, tv.tv_usec, this, - &rToCopy, (int)(rToCopy.mExpires / 1000000), + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << " to fire at " << + (int)(rToCopy.mExpires / 1000000) << "." << (int)(rToCopy.mExpires % 1000000)); } #endif @@ -373,7 +375,8 @@ #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); - TRACE3("%d.%d: timer %p fired\n", tv.tv_sec, tv.tv_usec, this); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " fired"); #endif mExpired = true; From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:41:14 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:41:14 +0000 Subject: [Box Backup-commit] COMMIT r1446 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-24 14:41:14 +0000 (Sat, 24 Mar 2007) New Revision: 1446 Modified: box/chris/general/lib/common/Timer.h Log: Fix header include order Modified: box/chris/general/lib/common/Timer.h =================================================================== --- box/chris/general/lib/common/Timer.h 2007-03-24 14:40:39 UTC (rev 1445) +++ box/chris/general/lib/common/Timer.h 2007-03-24 14:41:14 UTC (rev 1446) @@ -15,9 +15,10 @@ #include -#include "MemLeakFindOn.h" #include "BoxTime.h" +#include "MemLeakFindOn.h" + class Timer; // -------------------------------------------------------------------------- From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:41:36 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:41:36 +0000 Subject: [Box Backup-commit] COMMIT r1447 - box/chris/general/bin/bbstored Message-ID: Author: chris Date: 2007-03-24 14:41:36 +0000 (Sat, 24 Mar 2007) New Revision: 1447 Modified: box/chris/general/bin/bbstored/BackupStoreDaemon.cpp Log: Remove newlines from syslog() messages Modified: box/chris/general/bin/bbstored/BackupStoreDaemon.cpp =================================================================== --- box/chris/general/bin/bbstored/BackupStoreDaemon.cpp 2007-03-24 14:41:14 UTC (rev 1446) +++ box/chris/general/bin/bbstored/BackupStoreDaemon.cpp 2007-03-24 14:41:36 UTC (rev 1447) @@ -295,7 +295,7 @@ std::string clientCommonName(rStream.GetPeerCommonName()); // Log the name - ::syslog(LOG_INFO, "Certificate CN: %s\n", clientCommonName.c_str()); + ::syslog(LOG_INFO, "Certificate CN: %s", clientCommonName.c_str()); // Check it int32_t id; @@ -342,7 +342,7 @@ { // Log the amount of data transferred ::syslog(LOG_INFO, "Connection statistics for %s: " - "IN=%lld OUT=%lld TOTAL=%lld\n", commonName, + "IN=%lld OUT=%lld TOTAL=%lld", commonName, (long long)s.GetBytesRead(), (long long)s.GetBytesWritten(), (long long)s.GetBytesRead() + From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:42:08 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:42:08 +0000 Subject: [Box Backup-commit] COMMIT r1448 - box/chris/general/bin/bbackupd Message-ID: Author: chris Date: 2007-03-24 14:42:08 +0000 (Sat, 24 Mar 2007) New Revision: 1448 Modified: box/chris/general/bin/bbackupd/BackupClientContext.cpp Log: Use logging framework to remove timer noise for those who don't want it Modified: box/chris/general/bin/bbackupd/BackupClientContext.cpp =================================================================== --- box/chris/general/bin/bbackupd/BackupClientContext.cpp 2007-03-24 14:41:36 UTC (rev 1447) +++ box/chris/general/bin/bbackupd/BackupClientContext.cpp 2007-03-24 14:42:08 UTC (rev 1448) @@ -29,6 +29,7 @@ #include "BackupDaemon.h" #include "autogen_BackupProtocolClient.h" #include "BackupStoreFile.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -499,13 +500,14 @@ void BackupClientContext::SetMaximumDiffingTime(int iSeconds) { mMaximumDiffingTime = iSeconds < 0 ? 0 : iSeconds; - TRACE1("Set maximum diffing time to %d seconds\n", mMaximumDiffingTime); + BOX_TRACE("Set maximum diffing time to " << mMaximumDiffingTime << + " seconds"); } void BackupClientContext::SetKeepAliveTime(int iSeconds) { mKeepAliveTime = iSeconds < 0 ? 0 : iSeconds; - TRACE1("Set keep-alive time to %d seconds\n", mKeepAliveTime); + BOX_TRACE("Set keep-alive time to " << mKeepAliveTime << " seconds"); mKeepAliveTimer = Timer(mKeepAliveTime); } @@ -563,7 +565,7 @@ return; } - TRACE0("KeepAliveTime reached, sending keep-alive message\n"); + BOX_TRACE("KeepAliveTime reached, sending keep-alive message"); mpConnection->QueryGetIsAlive(); mKeepAliveTimer = Timer(mKeepAliveTime); From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:47:20 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:47:20 +0000 Subject: [Box Backup-commit] COMMIT r1449 - box/chris/general/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-24 14:47:20 +0000 (Sat, 24 Mar 2007) New Revision: 1449 Modified: box/chris/general/bin/bbackupquery/bbackupquery.cpp Log: Initialise logging framework and set sensible default verbosity levels Modified: box/chris/general/bin/bbackupquery/bbackupquery.cpp =================================================================== --- box/chris/general/bin/bbackupquery/bbackupquery.cpp 2007-03-24 14:42:08 UTC (rev 1448) +++ box/chris/general/bin/bbackupquery/bbackupquery.cpp 2007-03-24 14:47:20 UTC (rev 1449) @@ -45,6 +45,7 @@ #include "FdGetLine.h" #include "BackupClientCryptoKeys.h" #include "BannerText.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -95,11 +96,19 @@ bool quiet = false; bool readWrite = false; + Logging::SetProgramName("Box Backup (bbackupquery)"); + + #ifdef NDEBUG + int masterLevel = Log::NOTICE; // need an int to do math with + #else + int masterLevel = Log::INFO; // need an int to do math with + #endif + #ifdef WIN32 - const char* validOpts = "qwuc:l:"; + const char* validOpts = "qvwuc:l:"; bool unicodeConsole = false; #else - const char* validOpts = "qwc:l:"; + const char* validOpts = "qvwc:l:"; #endif // See if there's another entry on the command line @@ -108,11 +117,35 @@ { switch(c) { - case 'q': - // Quiet mode - quiet = true; + case 'q': + { + // Quiet mode + quiet = true; + + if(masterLevel == Log::NOTHING) + { + BOX_FATAL("Too many '-q': " + "Cannot reduce logging " + "level any more"); + return 2; + } + masterLevel--; + } break; - + + case 'v': + { + if(masterLevel == Log::EVERYTHING) + { + BOX_FATAL("Too many '-v': " + "Cannot increase logging " + "level any more"); + return 2; + } + masterLevel++; + } + break; + case 'w': // Read/write mode readWrite = true; @@ -147,6 +180,8 @@ argc -= optind; argv += optind; + Logging::SetGlobalLevel((Log::Level)masterLevel); + // Print banner? if(!quiet) { From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:49:12 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:49:12 +0000 Subject: [Box Backup-commit] COMMIT r1450 - box/chris/general/test/bbackupd Message-ID: Author: chris Date: 2007-03-24 14:49:11 +0000 (Sat, 24 Mar 2007) New Revision: 1450 Modified: box/chris/general/test/bbackupd/testbbackupd.cpp Log: Use "\n====" prefix on all test messages to make test output more readable. Modified: box/chris/general/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-24 14:47:20 UTC (rev 1449) +++ box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-24 14:49:11 UTC (rev 1450) @@ -1333,7 +1333,7 @@ #endif // WIN32 // Check that SyncAllowScript is executed and can pause backup - printf("==== Check that SyncAllowScript is executed and can " + printf("\n==== Check that SyncAllowScript is executed and can " "pause backup\n"); fflush(stdout); @@ -1424,7 +1424,7 @@ TestRemoteProcessMemLeaks("bbackupquery.memleaks"); } - printf("==== Delete file and update another, create symlink.\n"); + printf("\n==== Delete file and update another, create symlink.\n"); // Delete a file TEST_THAT(::unlink("testfiles/TestDir1/x1/dsfdsfs98.fd") == 0); @@ -1491,7 +1491,8 @@ wait_for_backup_operation(100); // Bad case: delete a file/symlink, replace it with a directory - printf("==== Replace symlink with directory, add new directory\n"); + printf("\n==== Replace symlink with directory, " + "add new directory\n"); #ifndef WIN32 TEST_THAT(::unlink("testfiles/TestDir1/symlink-to-dir") == 0); #endif @@ -1508,7 +1509,7 @@ TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // And the inverse, replace a directory with a file/symlink - printf("==== Replace directory with symlink\n"); + printf("\n==== Replace directory with symlink\n"); #ifndef WIN32 TEST_THAT(::unlink("testfiles/TestDir1/x1/dir-to-file/contents") == 0); #endif @@ -1522,7 +1523,8 @@ TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // And then, put it back to how it was before. - printf("==== Replace symlink with directory (which was a symlink)\n"); + printf("\n==== Replace symlink with directory " + "(which was a symlink)\n"); #ifndef WIN32 TEST_THAT(::unlink("testfiles/TestDir1/x1/dir-to-file") == 0); #endif @@ -1537,7 +1539,7 @@ // And finally, put it back to how it was before it was put back to how it was before // This gets lots of nasty things in the store with directories over other old directories. - printf("==== Put it all back to how it was\n"); + printf("\n==== Put it all back to how it was\n"); #ifndef WIN32 TEST_THAT(::unlink("testfiles/TestDir1/x1/dir-to-file/contents2") == 0); #endif @@ -1552,7 +1554,7 @@ // rename an untracked file over an // existing untracked file - printf("Rename over existing untracked file\n"); + printf("\n==== Rename over existing untracked file\n"); int fd1 = open("testfiles/TestDir1/untracked-1", O_CREAT | O_EXCL | O_WRONLY, 0700); int fd2 = open("testfiles/TestDir1/untracked-2", @@ -1572,8 +1574,11 @@ compareReturnValue = ::system(BBACKUPQUERY " -q " "-c testfiles/bbackupd.conf -l testfiles/query3t.log " "\"compare -ac\" quit"); - TEST_THAT(compareReturnValue == 1*256); + TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + #ifdef WIN32 + TEST_THAT(::unlink("testfiles/TestDir1/untracked-2") == 0); + #endif TEST_THAT(::rename("testfiles/TestDir1/untracked-1", "testfiles/TestDir1/untracked-2") == 0); TEST_THAT(!TestFileExists("testfiles/TestDir1/untracked-1")); @@ -1582,12 +1587,12 @@ compareReturnValue = ::system(BBACKUPQUERY " -q " "-c testfiles/bbackupd.conf -l testfiles/query3t.log " "\"compare -ac\" quit"); - TEST_THAT(compareReturnValue == 1*256); + TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // case which went wrong: rename a tracked file over an // existing tracked file - printf("Rename over existing tracked file\n"); + printf("\n==== Rename over existing tracked file\n"); fd1 = open("testfiles/TestDir1/tracked-1", O_CREAT | O_EXCL | O_WRONLY, 0700); fd2 = open("testfiles/TestDir1/tracked-2", @@ -1610,8 +1615,11 @@ compareReturnValue = ::system(BBACKUPQUERY " -q " "-c testfiles/bbackupd.conf -l testfiles/query3u.log " "\"compare -ac\" quit"); - TEST_THAT(compareReturnValue == 1*256); + TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + #ifdef WIN32 + TEST_THAT(::unlink("testfiles/TestDir1/tracked-2") == 0); + #endif TEST_THAT(::rename("testfiles/TestDir1/tracked-1", "testfiles/TestDir1/tracked-2") == 0); TEST_THAT(!TestFileExists("testfiles/TestDir1/tracked-1")); @@ -1620,11 +1628,11 @@ compareReturnValue = ::system(BBACKUPQUERY " -q " "-c testfiles/bbackupd.conf -l testfiles/query3v.log " "\"compare -ac\" quit"); - TEST_THAT(compareReturnValue == 1*256); + TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // case which went wrong: rename a tracked file over a deleted file - printf("Rename an existing file over a deleted file\n"); + printf("\n==== Rename an existing file over a deleted file\n"); #ifdef WIN32 TEST_THAT(::unlink("testfiles/TestDir1/x1/dsfdsfs98.fd")); #endif @@ -1634,7 +1642,8 @@ TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); - printf("==== Add files with old times, update attributes of one to latest time\n"); + printf("\n==== Add files with old times, update " + "attributes of one to latest time\n"); // Move that file back TEST_THAT(::rename("testfiles/TestDir1/x1/dsfdsfs98.fd", "testfiles/TestDir1/df9834.dsf") == 0); @@ -1656,7 +1665,7 @@ TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // Check that modifying files with old timestamps still get added - printf("==== Modify existing file, but change timestamp " + printf("\n==== Modify existing file, but change timestamp " "to rather old\n"); wait_for_sync_end(); @@ -1700,7 +1709,7 @@ TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // Add some files and directories which are marked as excluded - printf("==== Add files and dirs for exclusion test\n"); + printf("\n==== Add files and dirs for exclusion test\n"); #ifdef WIN32 TEST_THAT(::system("tar xzvf testfiles/testexclude.tgz -C testfiles") == 0); #else @@ -1727,7 +1736,7 @@ // check that the excluded files did not make it // into the store, and the included files did - printf("==== Check that exclude/alwaysinclude commands " + printf("\n==== Check that exclude/alwaysinclude commands " "actually work\n"); { @@ -1807,7 +1816,7 @@ if(::getuid() != 0) { // Check that read errors are reported neatly - printf("==== Add unreadable files\n"); + printf("\n==== Add unreadable files\n"); { // Dir and file which can't be read TEST_THAT(::mkdir("testfiles/TestDir1/sub23/read-fail-test-dir", 0000) == 0); @@ -1836,7 +1845,8 @@ } #endif - printf("==== Continuously update file, check isn't uploaded\n"); + printf("\n==== Continuously update file, " + "check isn't uploaded\n"); // Make sure everything happens at the same point in the // sync cycle: wait until exactly the start of a sync @@ -1872,7 +1882,7 @@ TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); - printf("==== Keep on continuously updating file, " + printf("\n==== Keep on continuously updating file, " "check it is uploaded eventually\n"); for(int l = 0; l < 28; ++l) @@ -1900,7 +1910,7 @@ TestRemoteProcessMemLeaks("bbackupquery.memleaks"); } - printf("==== Delete directory, change attributes\n"); + printf("\n==== Delete directory, change attributes\n"); // Delete a directory TEST_THAT(::system("rm -rf testfiles/TestDir1/x1") == 0); @@ -1913,7 +1923,7 @@ TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); - printf("==== Restore files and directories\n"); + printf("\n==== Restore files and directories\n"); int64_t deldirid = 0; int64_t restoredirid = 0; { @@ -2041,7 +2051,7 @@ TestRemoteProcessMemLeaks("bbackupquery.memleaks"); #endif - printf("==== Add files with current time\n"); + printf("\n==== Add files with current time\n"); // Add some more files and modify others // Use the m flag this time so they have a recent modification time @@ -2058,7 +2068,7 @@ TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // Rename directory - printf("==== Rename directory\n"); + printf("\n==== Rename directory\n"); TEST_THAT(rename("testfiles/TestDir1/sub23/dhsfdss", "testfiles/TestDir1/renamed-dir") == 0); wait_for_backup_operation(); compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query6.log \"compare -ac\" quit"); @@ -2070,7 +2080,7 @@ TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // Rename some files -- one under the threshold, others above - printf("==== Rename files\n"); + printf("\n==== Rename files\n"); TEST_THAT(rename("testfiles/TestDir1/continousupdate", "testfiles/TestDir1/continousupdate-ren") == 0); TEST_THAT(rename("testfiles/TestDir1/df324", "testfiles/TestDir1/df324-ren") == 0); TEST_THAT(rename("testfiles/TestDir1/sub23/find2perl", "testfiles/TestDir1/find2perl-ren") == 0); @@ -2080,7 +2090,7 @@ TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // Check that modifying files with madly in the future timestamps still get added - printf("==== Create a file with timestamp way ahead " + printf("\n==== Create a file with timestamp way ahead " "in the future\n"); // Time critical, so sync wait_for_sync_start(); @@ -2107,7 +2117,7 @@ TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); - printf("==== Change client store marker\n"); + printf("\n==== Change client store marker\n"); // Then... connect to the server, and change the client store marker. See what that does! { @@ -2142,7 +2152,7 @@ TEST_THAT(done); } - printf("==== Check change of store marker pauses daemon\n"); + printf("\n==== Check change of store marker pauses daemon\n"); // Make a change to a file, to detect whether or not // it's hanging around waiting to retry. @@ -2159,8 +2169,12 @@ TEST_RETURN(compareReturnValue, 2); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + printf("\n==== Waiting for bbackupd to recover\n"); + // 100 seconds - (12*3/2) + wait_for_operation(82); + #ifndef WIN32 - printf("==== Interrupted restore\n"); + printf("\n==== Interrupted restore\n"); { do_interrupted_restore(context, restoredirid); int64_t resumesize = 0; @@ -2194,7 +2208,7 @@ } #endif - printf("==== Check restore deleted files\n"); + printf("\n==== Check restore deleted files\n"); { SocketStreamTLS conn; conn.Open(context, Socket::TypeINET, "localhost", BOX_PORT_BBSTORED); @@ -2222,7 +2236,7 @@ TEST_THAT(!TestFileExists("testfiles/notifyran.read-error.2")); #ifdef WIN32 - printf("==== Testing locked file behaviour:\n"); + printf("\n==== Testing locked file behaviour:\n"); // Test that locked files cannot be backed up, // and the appropriate error is reported. @@ -2302,7 +2316,7 @@ wait_for_backup_operation((TIME_TO_WAIT_FOR_BACKUP_OPERATION*3) / 2); // little bit longer than usual compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf " - "-l testfiles/query4.log \"compare -ac\" quit"); + "-l testfiles/query4a.log \"compare -ac\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:52:00 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:52:00 +0000 Subject: [Box Backup-commit] COMMIT r1451 - in box/chris/general/lib: . intercept Message-ID: Author: chris Date: 2007-03-24 14:52:00 +0000 (Sat, 24 Mar 2007) New Revision: 1451 Added: box/chris/general/lib/intercept/ box/chris/general/lib/intercept/intercept.h Log: Check in new intercept library header Copied: box/chris/general/lib/intercept (from rev 1180, box/chris/merge/lib/intercept) Copied: box/chris/general/lib/intercept/intercept.h (from rev 1292, box/chris/merge/lib/intercept/intercept.h) =================================================================== --- box/chris/general/lib/intercept/intercept.h (rev 0) +++ box/chris/general/lib/intercept/intercept.h 2007-03-24 14:52:00 UTC (rev 1451) @@ -0,0 +1,47 @@ +// -------------------------------------------------------------------------- +// +// File +// Name: intercept.h +// Purpose: Syscall interception code for unit tests +// Created: 2006/11/29 +// +// -------------------------------------------------------------------------- + +#ifndef INTERCEPT_H +#define INTERCEPT_H +#ifndef PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE + +#include + +#include +#include + +extern "C" +{ + typedef DIR *(opendir_t) (const char *name); + typedef struct dirent *(readdir_t) (DIR *dir); + typedef struct dirent *(readdir_t) (DIR *dir); + typedef int (closedir_t)(DIR *dir); +#if defined __GNUC__ && __GNUC__ >= 2 +#define LINUX_WEIRD_LSTAT +#define STAT_STRUCT struct stat /* should be stat64 */ + typedef int (lstat_t) (int ver, const char *file_name, + STAT_STRUCT *buf); +#else +#define STAT_STRUCT struct stat + typedef int (lstat_t) (const char *file_name, + STAT_STRUCT *buf); +#endif +} + +void intercept_setup_error(const char *filename, unsigned int errorafter, + int errortoreturn, int syscalltoerror); +void intercept_setup_delay(const char *filename, unsigned int delay_after, + int delay_ms, int syscall_to_delay, int num_delays); +bool intercept_triggered(); + +void intercept_setup_readdir_hook(const char *dirname, readdir_t hookfn); +void intercept_setup_lstat_hook (const char *filename, lstat_t hookfn); + +#endif // !PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE +#endif // !INTERCEPT_H From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:54:24 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:54:24 +0000 Subject: [Box Backup-commit] COMMIT r1452 - box/chris/general/lib/intercept Message-ID: Author: chris Date: 2007-03-24 14:54:24 +0000 (Sat, 24 Mar 2007) New Revision: 1452 Modified: box/chris/general/lib/intercept/intercept.cpp Log: Intercept updates merged from chris/merge Modified: box/chris/general/lib/intercept/intercept.cpp =================================================================== --- box/chris/general/lib/intercept/intercept.cpp 2007-03-24 14:52:00 UTC (rev 1451) +++ box/chris/general/lib/intercept/intercept.cpp 2007-03-24 14:54:24 UTC (rev 1452) @@ -9,6 +9,8 @@ #include "Box.h" +#include "intercept.h" + #ifdef HAVE_SYS_SYSCALL_H #include #endif @@ -16,11 +18,15 @@ #include #ifdef HAVE_SYS_UIO_H -#include + #include #endif #include +#ifdef HAVE_DLFCN_H +#include +#endif + #ifndef PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE #if !defined(HAVE_SYSCALL) && !defined(HAVE___SYSCALL) && !defined(HAVE___SYSCALL_NEED_DEFN) @@ -59,7 +65,7 @@ #include "MemLeakFindOn.h" -bool intercept_enabled = false; +int intercept_count = 0; const char *intercept_filename = 0; int intercept_filedes = -1; off_t intercept_errorafter = 0; @@ -72,7 +78,7 @@ void intercept_clear_setup() { - intercept_enabled = false; + intercept_count = 0; intercept_filename = 0; intercept_filedes = -1; intercept_errorafter = 0; @@ -83,13 +89,13 @@ bool intercept_triggered() { - return !intercept_enabled; + return intercept_count == 0; } void intercept_setup_error(const char *filename, unsigned int errorafter, int errortoreturn, int syscalltoerror) { TRACE4("Setup for error: %s, after %d, err %d, syscall %d\n", filename, errorafter, errortoreturn, syscalltoerror); - intercept_enabled = true; + intercept_count = 1; intercept_filename = filename; intercept_filedes = -1; intercept_errorafter = errorafter; @@ -100,11 +106,12 @@ } void intercept_setup_delay(const char *filename, unsigned int delay_after, - int delay_ms, int syscall_to_delay) + int delay_ms, int syscall_to_delay, int num_delays) { - TRACE4("Setup for delay: %s, after %d, wait %d ms, syscall %d\n", - filename, delay_after, delay_ms, syscall_to_delay); - intercept_enabled = true; + TRACE5("Setup for delay: %s, after %d, wait %d ms, times %d, " + "syscall %d\n", filename, delay_after, delay_ms, + num_delays, syscall_to_delay); + intercept_count = num_delays; intercept_filename = filename; intercept_filedes = -1; intercept_errorafter = delay_after; @@ -154,7 +161,7 @@ } #define CHECK_FOR_FAKE_ERROR_COND(D, S, CALL, FAILRES) \ - if(intercept_enabled) \ + if(intercept_count > 0) \ { \ if(intercept_errornow(D, S, CALL)) \ { \ @@ -166,7 +173,11 @@ * 1000000; \ while (nanosleep(&tm, &tm) != 0 && \ errno == EINTR) { } \ - intercept_clear_setup(); \ + intercept_count --; \ + if (intercept_count == 0) \ + { \ + intercept_clear_setup(); \ + } \ } \ else \ { \ @@ -179,7 +190,7 @@ extern "C" int open(const char *path, int flags, mode_t mode) { - if(intercept_enabled) + if(intercept_count > 0) { if(intercept_syscall == SYS_open && strcmp(path, intercept_filename) == 0) { @@ -192,7 +203,7 @@ #else int r = syscall(SYS_open, path, flags, mode); #endif - if(intercept_enabled && intercept_filedes == -1) + if(intercept_count > 0 && intercept_filedes == -1) { // Right file? if(strcmp(intercept_filename, path) == 0) @@ -312,4 +323,173 @@ return r; } +static opendir_t* opendir_real = NULL; +static readdir_t* readdir_real = NULL; +static readdir_t* readdir_hook = NULL; +static closedir_t* closedir_real = NULL; +static lstat_t* lstat_real = NULL; +static lstat_t* lstat_hook = NULL; +static const char* lstat_file = NULL; + +void intercept_setup_readdir_hook(const char *dirname, readdir_t hookfn) +{ + if (hookfn != NULL && dirname == NULL) + { + dirname = intercept_filename; + } + + if (hookfn != NULL) + { + TRACE2("readdir hooked to %p for %s\n", hookfn, dirname); + } + else + { + TRACE2("readdir unhooked from %p for %s\n", readdir_hook, + intercept_filename); + } + + intercept_filename = dirname; + readdir_hook = hookfn; +} + +void intercept_setup_lstat_hook(const char *filename, lstat_t hookfn) +{ + /* + if (hookfn != NULL) + { + TRACE2("lstat hooked to %p for %s\n", hookfn, filename); + } + else + { + TRACE2("lstat unhooked from %p for %s\n", lstat_hook, + lstat_file); + } + */ + + lstat_file = filename; + lstat_hook = hookfn; +} + +extern "C" +DIR *opendir(const char *dirname) +{ + if (opendir_real == NULL) + { + opendir_real = (opendir_t*)(dlsym(RTLD_NEXT, "opendir")); + } + + if (opendir_real == NULL) + { + perror("cannot find real opendir"); + return NULL; + } + + DIR* r = opendir_real(dirname); + + if(readdir_hook != NULL && intercept_filedes == -1 && + strcmp(intercept_filename, dirname) == 0) + { + intercept_filedes = dirfd(r); + //printf("Found file to intercept, h = %d\n", r); + } + + return r; +} + +extern "C" +struct dirent *readdir(DIR *dir) +{ + if (readdir_hook != NULL && dirfd(dir) == intercept_filedes) + { + return readdir_hook(dir); + } + + if (readdir_real == NULL) + { + #if readdir == readdir64 + readdir_real = (readdir_t*)(dlsym(RTLD_NEXT, "readdir64")); + #else + readdir_real = (readdir_t*)(dlsym(RTLD_NEXT, "readdir")); + #endif + } + + if (readdir_real == NULL) + { + perror("cannot find real readdir"); + return NULL; + } + + return readdir_real(dir); +} + +extern "C" +int closedir(DIR *dir) +{ + if (dirfd(dir) == intercept_filedes) + { + intercept_filedes = -1; + } + + if (closedir_real == NULL) + { + closedir_real = (closedir_t*)(dlsym(RTLD_NEXT, "closedir")); + } + + if (closedir_real == NULL) + { + perror("cannot find real closedir"); + errno = ENOSYS; + return -1; + } + + return closedir_real(dir); +} + +extern "C" int +#ifdef LINUX_WEIRD_LSTAT +__lxstat(int ver, const char *file_name, STAT_STRUCT *buf) +#else +lstat(const char *file_name, STAT_STRUCT *buf) +#endif +{ + if (lstat_real == NULL) + { + #ifdef LINUX_WEIRD_LSTAT + #if __lxstat == __lxstat64 + lstat_real = (lstat_t*)(dlsym(RTLD_NEXT, "__lxstat64")); + #else + lstat_real = (lstat_t*)(dlsym(RTLD_NEXT, "__lxstat")); + #endif + #else + #if lstat == lstat64 + lstat_real = (lstat_t*)(dlsym(RTLD_NEXT, "lstat64")); + #else + lstat_real = (lstat_t*)(dlsym(RTLD_NEXT, "lstat")); + #endif + #endif + } + + if (lstat_real == NULL) + { + perror("cannot find real lstat"); + errno = ENOSYS; + return -1; + } + + if (lstat_hook == NULL || strcmp(file_name, lstat_file) != 0) + { + #ifdef LINUX_WEIRD_LSTAT + return lstat_real(ver, file_name, buf); + #else + return lstat_real(file_name, buf); + #endif + } + + #ifdef LINUX_WEIRD_LSTAT + return lstat_hook(ver, file_name, buf); + #else + return lstat_hook(file_name, buf); + #endif +} + #endif // n PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:55:46 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:55:46 +0000 Subject: [Box Backup-commit] COMMIT r1453 - box/chris/general/infrastructure/m4 Message-ID: Author: chris Date: 2007-03-24 14:55:46 +0000 (Sat, 24 Mar 2007) New Revision: 1453 Added: box/chris/general/infrastructure/m4/ax_check_llong_minmax.m4 Log: Merge new m4 from chris/merge Added: box/chris/general/infrastructure/m4/ax_check_llong_minmax.m4 =================================================================== --- box/chris/general/infrastructure/m4/ax_check_llong_minmax.m4 (rev 0) +++ box/chris/general/infrastructure/m4/ax_check_llong_minmax.m4 2007-03-24 14:55:46 UTC (rev 1453) @@ -0,0 +1,76 @@ +dnl @synopsis AX_CHECK_LLONG_MINMAX +dnl +dnl This macro will fix up LLONG_MIN and LLONG_MAX as appropriate. I'm finding +dnl it quite difficult to believe that so many hoops are necessary. The world +dnl seems to have gone quite mad. +dnl +dnl This gem is adapted from the OpenSSH configure script so here's +dnl the original copyright notice: +dnl +dnl Copyright (c) 1999-2004 Damien Miller +dnl +dnl Permission to use, copy, modify, and distribute this software for any +dnl purpose with or without fee is hereby granted, provided that the above +dnl copyright notice and this permission notice appear in all copies. +dnl +dnl @category C +dnl @author Martin Ebourne and Damien Miller +dnl @version 2005/07/07 + +AC_DEFUN([AX_CHECK_LLONG_MINMAX], [ + AC_CHECK_DECL([LLONG_MAX], [have_llong_max=1], , [[#include ]]) + if test -z "$have_llong_max"; then + AC_MSG_CHECKING([[for max value of long long]]) + AC_RUN_IFELSE([AC_LANG_SOURCE([[ + #include + /* Why is this so damn hard? */ + #undef __GNUC__ + #undef __USE_ISOC99 + #define __USE_ISOC99 + #include + #define DATA "conftest.llminmax" + int main(void) { + FILE *f; + long long i, llmin, llmax = 0; + + if((f = fopen(DATA,"w")) == NULL) + exit(1); + + #if defined(LLONG_MIN) && defined(LLONG_MAX) + fprintf(stderr, "Using system header for LLONG_MIN and LLONG_MAX\n"); + llmin = LLONG_MIN; + llmax = LLONG_MAX; + #else + fprintf(stderr, "Calculating LLONG_MIN and LLONG_MAX\n"); + /* This will work on one's complement and two's complement */ + for (i = 1; i > llmax; i <<= 1, i++) + llmax = i; + llmin = llmax + 1LL; /* wrap */ + #endif + + /* Sanity check */ + if (llmin + 1 < llmin || llmin - 1 < llmin || llmax + 1 > llmax || llmax - 1 > llmax) { + fprintf(f, "unknown unknown\n"); + exit(2); + } + + if (fprintf(f ,"%lld %lld", llmin, llmax) < 0) + exit(3); + + exit(0); + } + ]])], [ + read llong_min llong_max < conftest.llminmax + AC_MSG_RESULT([$llong_max]) + AC_DEFINE_UNQUOTED([LLONG_MAX], [${llong_max}LL], + [max value of long long calculated by configure]) + AC_MSG_CHECKING([[for min value of long long]]) + AC_MSG_RESULT([$llong_min]) + AC_DEFINE_UNQUOTED([LLONG_MIN], [${llong_min}LL], + [min value of long long calculated by configure]) + ], + [AC_MSG_RESULT(not found)], + [AC_MSG_WARN([[cross compiling: not checking]])] + ) + fi + ])dnl From boxbackup-dev at fluffy.co.uk Sat Mar 24 14:58:31 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 14:58:31 +0000 Subject: [Box Backup-commit] COMMIT r1454 - box/chris/general/test/bbackupd Message-ID: Author: chris Date: 2007-03-24 14:58:31 +0000 (Sat, 24 Mar 2007) New Revision: 1454 Added: box/chris/general/test/bbackupd/Makefile.extra Log: Merge test/bbackupd/Makefile.extra from chris/merge, to fix compilation Copied: box/chris/general/test/bbackupd/Makefile.extra (from rev 1292, box/chris/merge/test/bbackupd/Makefile.extra) =================================================================== --- box/chris/general/test/bbackupd/Makefile.extra (rev 0) +++ box/chris/general/test/bbackupd/Makefile.extra 2007-03-24 14:58:31 UTC (rev 1454) @@ -0,0 +1 @@ +link-extra: ../../bin/bbackupd/BackupClientContext.o ../../bin/bbackupd/BackupClientDeleteList.o ../../bin/bbackupd/BackupClientDirectoryRecord.o ../../bin/bbackupd/Win32BackupService.o ../../bin/bbackupd/BackupClientInodeToIDMap.o ../../bin/bbackupd/Win32ServiceFunctions.o ../../bin/bbackupd/BackupDaemon.o From boxbackup-dev at fluffy.co.uk Sat Mar 24 16:55:02 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 16:55:02 +0000 Subject: [Box Backup-commit] COMMIT r1455 - box/chris/general/bin/bbackupd Message-ID: Author: chris Date: 2007-03-24 16:55:01 +0000 (Sat, 24 Mar 2007) New Revision: 1455 Added: box/chris/general/bin/bbackupd/ClientException.txt box/chris/general/bin/bbackupd/Makefile.extra Log: Add new exception codes for use in bbackupd (and possibly other clients) Added: box/chris/general/bin/bbackupd/ClientException.txt =================================================================== --- box/chris/general/bin/bbackupd/ClientException.txt (rev 0) +++ box/chris/general/bin/bbackupd/ClientException.txt 2007-03-24 16:55:01 UTC (rev 1455) @@ -0,0 +1,11 @@ + +# NOTE: Exception descriptions are for public distributions of Box Backup only -- do not rely for other applications. + + +EXCEPTION Client 13 + +Internal 0 +AssertFailed 1 +ClockWentBackwards 2 Invalid (negative) sync period: perhaps your clock is going backwards? +FailedToDeleteStoreObjectInfoFile 3 Failed to delete the StoreObjectInfoFile, backup cannot continue safely. +CorruptStoreObjectInfoFile 4 The store object info file contained an invalid value and is probably corrupt. Try deleting it. Added: box/chris/general/bin/bbackupd/Makefile.extra =================================================================== --- box/chris/general/bin/bbackupd/Makefile.extra (rev 0) +++ box/chris/general/bin/bbackupd/Makefile.extra 2007-03-24 16:55:01 UTC (rev 1455) @@ -0,0 +1,7 @@ + +MAKEEXCEPTION = ../../lib/common/makeexception.pl + +# AUTOGEN SEEDING +autogen_ClientException.h autogen_ClientException.cpp: $(MAKEEXCEPTION) ClientException.txt + $(PERL) $(MAKEEXCEPTION) ClientException.txt + From boxbackup-dev at fluffy.co.uk Sat Mar 24 19:00:27 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 19:00:27 +0000 Subject: [Box Backup-commit] COMMIT r1456 - box/chris/general/test/bbackupd Message-ID: Author: chris Date: 2007-03-24 19:00:26 +0000 (Sat, 24 Mar 2007) New Revision: 1456 Modified: box/chris/general/test/bbackupd/Makefile.extra Log: Add autogen_ClientException.o to testbbackupd extra objects, because it links against bbackupd objects which need this. Modified: box/chris/general/test/bbackupd/Makefile.extra =================================================================== --- box/chris/general/test/bbackupd/Makefile.extra 2007-03-24 16:55:01 UTC (rev 1455) +++ box/chris/general/test/bbackupd/Makefile.extra 2007-03-24 19:00:26 UTC (rev 1456) @@ -1 +1 @@ -link-extra: ../../bin/bbackupd/BackupClientContext.o ../../bin/bbackupd/BackupClientDeleteList.o ../../bin/bbackupd/BackupClientDirectoryRecord.o ../../bin/bbackupd/Win32BackupService.o ../../bin/bbackupd/BackupClientInodeToIDMap.o ../../bin/bbackupd/Win32ServiceFunctions.o ../../bin/bbackupd/BackupDaemon.o +link-extra: ../../bin/bbackupd/autogen_ClientException.o ../../bin/bbackupd/BackupClientContext.o ../../bin/bbackupd/BackupClientDeleteList.o ../../bin/bbackupd/BackupClientDirectoryRecord.o ../../bin/bbackupd/Win32BackupService.o ../../bin/bbackupd/BackupClientInodeToIDMap.o ../../bin/bbackupd/Win32ServiceFunctions.o ../../bin/bbackupd/BackupDaemon.o From boxbackup-dev at fluffy.co.uk Sat Mar 24 19:23:16 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 19:23:16 +0000 Subject: [Box Backup-commit] COMMIT r1457 - box/chris/general/lib/win32 Message-ID: Author: chris Date: 2007-03-24 19:23:16 +0000 (Sat, 24 Mar 2007) New Revision: 1457 Modified: box/chris/general/lib/win32/emu.cpp Log: Register our event source with the name passed to us, so that it goes into the event logs. Modified: box/chris/general/lib/win32/emu.cpp =================================================================== --- box/chris/general/lib/win32/emu.cpp 2007-03-24 19:00:26 UTC (rev 1456) +++ box/chris/general/lib/win32/emu.cpp 2007-03-24 19:23:16 UTC (rev 1457) @@ -1345,13 +1345,17 @@ { } - if (!AddEventSource("Box Backup", 0)) + char* name = strdup(daemonName); + BOOL success = AddEventSource(name, 0); + free(name); + + if (!success) { ::syslog(LOG_ERR, "Failed to add our own event source"); return; } - HANDLE newSyslogH = RegisterEventSource(NULL, "Box Backup"); + HANDLE newSyslogH = RegisterEventSource(NULL, daemonName); if (newSyslogH == NULL) { ::syslog(LOG_ERR, "Failed to register our own event source: " From boxbackup-dev at fluffy.co.uk Sat Mar 24 19:26:25 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 19:26:25 +0000 Subject: [Box Backup-commit] COMMIT r1458 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-24 19:26:25 +0000 (Sat, 24 Mar 2007) New Revision: 1458 Modified: box/chris/general/lib/server/WinNamedPipeStream.cpp Log: Improve logging of pipe errors by including the error message. Reinitialise the OVERLAPPED structure each time we start a new overlapped read (thanks Charles!) Don't log errors or throw exceptions when we get ERROR_NO_DATA, which just means that the pipe is being closed. Treat it as a normal remote close (EOF) instead. Don't log an error if DisconnectNamedPipe tells us that the remote end already closed the pipe (ERROR_PIPE_NOT_CONNECTED) Modified: box/chris/general/lib/server/WinNamedPipeStream.cpp =================================================================== --- box/chris/general/lib/server/WinNamedPipeStream.cpp 2007-03-24 19:23:16 UTC (rev 1457) +++ box/chris/general/lib/server/WinNamedPipeStream.cpp 2007-03-24 19:26:25 UTC (rev 1458) @@ -100,8 +100,8 @@ if (mSocketHandle == INVALID_HANDLE_VALUE) { - ::syslog(LOG_ERR, "CreateNamedPipeW failed: %d", - GetLastError()); + ::syslog(LOG_ERR, "CreateNamedPipeW failed: %s", + GetErrorMessage(GetLastError()).c_str()); THROW_EXCEPTION(ServerException, SocketOpenError) } @@ -109,8 +109,8 @@ if (!connected) { - ::syslog(LOG_ERR, "ConnectNamedPipe failed: %d", - GetLastError()); + ::syslog(LOG_ERR, "ConnectNamedPipe failed: %s", + GetErrorMessage(GetLastError()).c_str()); Close(); THROW_EXCEPTION(ServerException, SocketOpenError) } @@ -126,8 +126,8 @@ if (mReadableEvent == INVALID_HANDLE_VALUE) { - ::syslog(LOG_ERR, "Failed to create the Readable event: " - "error %d", GetLastError()); + ::syslog(LOG_ERR, "Failed to create the Readable event: %s", + GetErrorMessage(GetLastError()).c_str()); Close(); THROW_EXCEPTION(CommonException, Internal) } @@ -145,7 +145,7 @@ if (err != ERROR_IO_PENDING) { ::syslog(LOG_ERR, "Failed to start overlapped read: " - "error %d", err); + "%s", GetErrorMessage(err).c_str()); Close(); THROW_EXCEPTION(ConnectionException, Conn_SocketReadError) @@ -189,7 +189,7 @@ else { ::syslog(LOG_ERR, "Failed to connect to backup " - "daemon: error %d", err); + "daemon: %s", GetErrorMessage(err).c_str()); } THROW_EXCEPTION(ServerException, SocketOpenError) } @@ -269,7 +269,8 @@ ::syslog(LOG_ERR, "Failed to wait for " "ReadFile to complete: " - "error %d", err); + "%s", + GetErrorMessage(err).c_str()); } Close(); @@ -298,6 +299,13 @@ mBytesInBuffer = BytesRemaining; NumBytesRead = BytesToCopy; + if (needAnotherRead) + { + // reinitialise the OVERLAPPED structure + memset(&mReadOverlap, 0, sizeof(mReadOverlap)); + mReadOverlap.hEvent = mReadableEvent; + } + // start the next overlapped read if (needAnotherRead && !ReadFile(mSocketHandle, mReadBuffer + mBytesInBuffer, @@ -325,7 +333,8 @@ else { ::syslog(LOG_ERR, "Failed to start " - "overlapped read: error %d", err); + "overlapped read: %s", + GetErrorMessage(err).c_str()); Close(); THROW_EXCEPTION(ConnectionException, Conn_SocketReadError) @@ -364,9 +373,25 @@ &NumBytesRead, // number of bytes read NULL)) // not overlapped { + DWORD err = GetLastError(); + Close(); - THROW_EXCEPTION(ConnectionException, - Conn_SocketReadError) + + // ERROR_NO_DATA is a strange name for + // "The pipe is being closed". No exception wanted. + + if (err == ERROR_NO_DATA) + { + NumBytesRead = 0; + } + else + { + ::syslog(LOG_ERR, "Failed to read from " + "control socket: %s", + GetErrorMessage(err).c_str()); + THROW_EXCEPTION(ConnectionException, + Conn_SocketReadError) + } } // Closed for reading at EOF? @@ -413,9 +438,23 @@ if (!Success) { + DWORD err = GetLastError(); + ::syslog(LOG_ERR, "Failed to write to control socket: " + "%s", GetErrorMessage(err).c_str()); Close(); - THROW_EXCEPTION(ConnectionException, - Conn_SocketWriteError) + + // ERROR_NO_DATA is a strange name for + // "The pipe is being closed". No exception wanted. + + if (err == ERROR_NO_DATA) + { + return; + } + else + { + THROW_EXCEPTION(ConnectionException, + Conn_SocketWriteError) + } } NumBytesWrittenTotal += NumBytesWrittenThisTime; @@ -449,7 +488,8 @@ if (!CancelIo(mSocketHandle)) { ::syslog(LOG_ERR, "Failed to cancel outstanding " - "I/O: error %d", GetLastError()); + "I/O: %s", + GetErrorMessage(GetLastError()).c_str()); } if (mReadableEvent == INVALID_HANDLE_VALUE) @@ -460,21 +500,27 @@ else if (!CloseHandle(mReadableEvent)) { ::syslog(LOG_ERR, "Failed to destroy Readable " - "event: error %d", GetLastError()); + "event: %s", + GetErrorMessage(GetLastError()).c_str()); } mReadableEvent = INVALID_HANDLE_VALUE; if (!FlushFileBuffers(mSocketHandle)) { - ::syslog(LOG_INFO, "FlushFileBuffers failed: %d", - GetLastError()); + ::syslog(LOG_INFO, "FlushFileBuffers failed: %s", + GetErrorMessage(GetLastError()).c_str()); } if (!DisconnectNamedPipe(mSocketHandle)) { - ::syslog(LOG_ERR, "DisconnectNamedPipe failed: %d", - GetLastError()); + DWORD err = GetLastError(); + if (err != ERROR_PIPE_NOT_CONNECTED) + { + ::syslog(LOG_ERR, "DisconnectNamedPipe " + "failed: %s", + GetErrorMessage(err).c_str()); + } } mIsServer = false; @@ -489,7 +535,8 @@ if (!result) { - ::syslog(LOG_ERR, "CloseHandle failed: %d", GetLastError()); + ::syslog(LOG_ERR, "CloseHandle failed: %s", + GetErrorMessage(GetLastError()).c_str()); THROW_EXCEPTION(ServerException, SocketCloseError) } } @@ -537,8 +584,8 @@ if (!FlushFileBuffers(mSocketHandle)) { - ::syslog(LOG_WARNING, "FlushFileBuffers failed: %d", - GetLastError()); + ::syslog(LOG_WARNING, "FlushFileBuffers failed: %s", + GetErrorMessage(GetLastError()).c_str()); } } From boxbackup-dev at fluffy.co.uk Sat Mar 24 19:43:31 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 19:43:31 +0000 Subject: [Box Backup-commit] COMMIT r1459 - box/chris/general/bin/bbackupd Message-ID: Author: chris Date: 2007-03-24 19:43:31 +0000 (Sat, 24 Mar 2007) New Revision: 1459 Modified: box/chris/general/bin/bbackupd/BackupDaemon.cpp Log: Use our new ClientException codes for clock skew and archive problems. Better handle a case where a force-sync command comes in immediately after (or during) a sync, i.e. less than MinimumFileAge seconds after the last one. In this case, just move back the syncPeriodStart by 1 second. Reformat long lines for readability. Modified: box/chris/general/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/chris/general/bin/bbackupd/BackupDaemon.cpp 2007-03-24 19:26:25 UTC (rev 1458) +++ box/chris/general/bin/bbackupd/BackupDaemon.cpp 2007-03-24 19:43:31 UTC (rev 1459) @@ -74,6 +74,7 @@ #include "Archive.h" #include "Timer.h" #include "Logging.h" +#include "autogen_ClientException.h" #include "MemLeakFindOn.h" @@ -663,38 +664,64 @@ box_time_t currentTime; do { - // Need to check the stop run thing here too, so this loop isn't run if we should be stopping + // Check whether we should be stopping, + // and don't run a sync if so. if(StopRun()) break; currentTime = GetCurrentBoxTime(); - // Pause a while, but no more than MAX_SLEEP_TIME seconds (use the conditional because times are unsigned) - box_time_t requiredDelay = (nextSyncTime < currentTime)?(0):(nextSyncTime - currentTime); - // If there isn't automatic backup happening, set a long delay. And limit delays at the same time. - if(!automaticBackup || requiredDelay > SecondsToBoxTime(MAX_SLEEP_TIME)) - requiredDelay = SecondsToBoxTime(MAX_SLEEP_TIME); + // Pause a while, but no more than + // MAX_SLEEP_TIME seconds (use the conditional + // because times are unsigned) + box_time_t requiredDelay = + (nextSyncTime < currentTime) + ? (0) + : (nextSyncTime - currentTime); - // Only do the delay if there is a delay required + // If there isn't automatic backup happening, + // set a long delay. And limit delays at the + // same time. + if(!automaticBackup || requiredDelay > + SecondsToBoxTime(MAX_SLEEP_TIME)) + { + requiredDelay = SecondsToBoxTime( + MAX_SLEEP_TIME); + } + + // Only delay if necessary if(requiredDelay > 0) { - // Sleep somehow. There are choices on how this should be done, depending on the state of the control connection + // Sleep somehow. There are choices + // on how this should be done, + // depending on the state of the + // control connection if(mpCommandSocketInfo != 0) { - // A command socket exists, so sleep by handling connections with it - WaitOnCommandSocket(requiredDelay, doSync, doSyncForcedByCommand); + // A command socket exists, + // so sleep by waiting on it + WaitOnCommandSocket( + requiredDelay, doSync, + doSyncForcedByCommand); } else { - // No command socket or connection, just do a normal sleep - time_t sleepSeconds = BoxTimeToSeconds(requiredDelay); - ::sleep((sleepSeconds <= 0)?1:sleepSeconds); + // No command socket or + // connection, just do a + // normal sleep + time_t sleepSeconds = + BoxTimeToSeconds( + requiredDelay); + ::sleep((sleepSeconds <= 0) + ? 1 + : sleepSeconds); } } } while((!automaticBackup || (currentTime < nextSyncTime)) && !doSync && !StopRun()); } - // Time of sync start, and if it's time for another sync (and we're doing automatic syncs), set the flag + // Time of sync start, and if it's time for another sync + // (and we're doing automatic syncs), set the flag box_time_t currentSyncStartTime = GetCurrentBoxTime(); if(automaticBackup && currentSyncStartTime >= nextSyncTime) { @@ -708,12 +735,14 @@ if(d > 0) { // Script has asked for a delay - nextSyncTime = GetCurrentBoxTime() + SecondsToBoxTime(d); + nextSyncTime = GetCurrentBoxTime() + + SecondsToBoxTime(d); doSync = false; } } - // Ready to sync? (but only if we're not supposed to be stopping) + // Ready to sync? (but only if we're not supposed + // to be stopping) if(doSync && !StopRun()) { // Touch a file to record times in filesystem @@ -727,22 +756,51 @@ // Calculate the sync period of files to examine box_time_t syncPeriodStart = lastSyncTime; - box_time_t syncPeriodEnd = currentSyncStartTime - minimumFileAge; + box_time_t syncPeriodEnd = currentSyncStartTime - + minimumFileAge; + + if(syncPeriodStart >= syncPeriodEnd && + syncPeriodStart - syncPeriodEnd < minimumFileAge) + { + // This can happen if we receive a force-sync + // command less than minimumFileAge after + // the last sync. Deal with it by moving back + // syncPeriodStart, which should not do any + // damage. + syncPeriodStart = syncPeriodEnd - + SecondsToBoxTime(1); + } + + if(syncPeriodStart >= syncPeriodEnd) + { + BOX_ERROR("Invalid (negative) sync period: " + "perhaps your clock is going " + "backwards (" << syncPeriodStart << + " to " << syncPeriodEnd << ")"); + THROW_EXCEPTION(ClientException, + ClockWentBackwards); + } + // Check logic ASSERT(syncPeriodEnd > syncPeriodStart); // Paranoid check on sync times if(syncPeriodStart >= syncPeriodEnd) continue; - // Adjust syncPeriodEnd to emulate snapshot behaviour properly + // Adjust syncPeriodEnd to emulate snapshot + // behaviour properly box_time_t syncPeriodEndExtended = syncPeriodEnd; // Using zero min file age? if(minimumFileAge == 0) { - // Add a year on to the end of the end time, to make sure we sync - // files which are modified after the scan run started. - // Of course, they may be eligable to be synced again the next time round, - // but this should be OK, because the changes only upload should upload no data. - syncPeriodEndExtended += SecondsToBoxTime((time_t)(356*24*3600)); + // Add a year on to the end of the end time, + // to make sure we sync files which are + // modified after the scan run started. + // Of course, they may be eligible to be + // synced again the next time round, + // but this should be OK, because the changes + // only upload should upload no data. + syncPeriodEndExtended += SecondsToBoxTime( + (time_t)(356*24*3600)); } // Delete the serialised store object file, @@ -754,10 +812,8 @@ BOX_ERROR("Failed to delete the " "StoreObjectInfoFile, backup cannot " "continue safely."); - // prevent runaway process where the logs fill up -- without this - // the log message will be emitted in a tight loop. - ::sleep(60); - continue; + THROW_EXCEPTION(ClientException, + FailedToDeleteStoreObjectInfoFile); } // In case the backup throws an exception, @@ -804,13 +860,22 @@ ); // Set up the sync parameters - BackupClientDirectoryRecord::SyncParams params(*this, *this, clientContext); + BackupClientDirectoryRecord::SyncParams params( + *this, *this, clientContext); params.mSyncPeriodStart = syncPeriodStart; - params.mSyncPeriodEnd = syncPeriodEndExtended; // use potentially extended end time + params.mSyncPeriodEnd = syncPeriodEndExtended; + // use potentially extended end time params.mMaxUploadWait = maxUploadWait; - params.mFileTrackingSizeThreshold = conf.GetKeyValueInt("FileTrackingSizeThreshold"); - params.mDiffingUploadSizeThreshold = conf.GetKeyValueInt("DiffingUploadSizeThreshold"); - params.mMaxFileTimeInFuture = SecondsToBoxTime(conf.GetKeyValueInt("MaxFileTimeInFuture")); + params.mFileTrackingSizeThreshold = + conf.GetKeyValueInt( + "FileTrackingSizeThreshold"); + params.mDiffingUploadSizeThreshold = + conf.GetKeyValueInt( + "DiffingUploadSizeThreshold"); + params.mMaxFileTimeInFuture = + SecondsToBoxTime( + conf.GetKeyValueInt( + "MaxFileTimeInFuture")); clientContext.SetMaximumDiffingTime(maximumDiffingTime); clientContext.SetKeepAliveTime(keepAliveTime); @@ -818,12 +883,17 @@ // Set store marker clientContext.SetClientStoreMarker(clientStoreMarker); - // Set up the locations, if necessary -- need to do it here so we have a (potential) connection to use + // Set up the locations, if necessary -- + // need to do it here so we have a + // (potential) connection to use if(mLocations.empty()) { - const Configuration &locations(conf.GetSubConfiguration("BackupLocations")); + const Configuration &locations( + conf.GetSubConfiguration( + "BackupLocations")); - // Make sure all the directory records are set up + // Make sure all the directory records + // are set up SetupLocations(clientContext, locations); } @@ -834,16 +904,25 @@ DeleteUnusedRootDirEntries(clientContext); // Go through the records, syncing them - for(std::vector::const_iterator i(mLocations.begin()); i != mLocations.end(); ++i) + for(std::vector::const_iterator + i(mLocations.begin()); + i != mLocations.end(); ++i) { - // Set current and new ID map pointers in the context + // Set current and new ID map pointers + // in the context clientContext.SetIDMaps(mCurrentIDMaps[(*i)->mIDMapIndex], mNewIDMaps[(*i)->mIDMapIndex]); - // Set exclude lists (context doesn't take ownership) - clientContext.SetExcludeLists((*i)->mpExcludeFiles, (*i)->mpExcludeDirs); + // Set exclude lists (context doesn't + // take ownership) + clientContext.SetExcludeLists( + (*i)->mpExcludeFiles, + (*i)->mpExcludeDirs); // Sync the directory - (*i)->mpDirectoryRecord->SyncDirectory(params, BackupProtocolClientListDirectory::RootDirectory, (*i)->mPath); + (*i)->mpDirectoryRecord->SyncDirectory( + params, + BackupProtocolClientListDirectory::RootDirectory, + (*i)->mPath); // Unset exclude lists (just in case) clientContext.SetExcludeLists(0, 0); @@ -857,13 +936,14 @@ } else { - // Unset the read error flag, so the error is - // reported again in the future + // Unset the read error flag, so the // error is reported again if it + // happens again mNotificationsSent[NotifyEvent_ReadError] = false; } - // Perform any deletions required -- these are delayed until the end - // to allow renaming to happen neatly. + // Perform any deletions required -- these are + // delayed until the end to allow renaming to + // happen neatly. clientContext.PerformDeletions(); // Close any open connection @@ -880,15 +960,25 @@ } else { - // The start time of the next run is the end time of this run - // This is only done if the storage limit wasn't exceeded (as things won't have been done properly if it was) + // The start time of the next run is + // the end time of this run. + // This is only done if the storage + // limit wasn't exceeded (as things + // won't have been done properly if + // it was) lastSyncTime = syncPeriodEnd; - // unflag the storage full notify flag so that next time the store is full, and alert will be sent + + // unflag the storage full notify flag + // so that next time the store is full, + // an alert will be sent mNotificationsSent[NotifyEvent_StoreFull] = false; } // Calculate when the next sync run should be - nextSyncTime = currentSyncStartTime + updateStoreInterval + Random::RandomInt(updateStoreInterval >> SYNC_PERIOD_RANDOM_EXTRA_TIME_SHIFT_BY); + nextSyncTime = currentSyncStartTime + + updateStoreInterval + + Random::RandomInt(updateStoreInterval >> + SYNC_PERIOD_RANDOM_EXTRA_TIME_SHIFT_BY); // Commit the ID Maps CommitIDMapsAfterSync(); @@ -921,18 +1011,26 @@ BOX_ERROR("Internal error during " "backup run: " << e.what()); errorOccurred = true; + errorString = e.what(); } catch(...) { - // TODO: better handling of exceptions here... need to be very careful + // TODO: better handling of exceptions here... + // need to be very careful errorOccurred = true; } if(errorOccurred) { // Is it a berkely db failure? - bool isBerkelyDbFailure = (errorCode == BackupStoreException::ExceptionType - && errorSubCode == BackupStoreException::BerkelyDBFailure); + bool isBerkelyDbFailure = false; + + if (errorCode == BackupStoreException::ExceptionType + && errorSubCode == BackupStoreException::BerkelyDBFailure) + { + isBerkelyDbFailure = true; + } + if(isBerkelyDbFailure) { // Delete corrupt files @@ -940,7 +1038,8 @@ } // Clear state data - syncPeriodStart = 0; // go back to beginning of time + syncPeriodStart = 0; + // go back to beginning of time clientStoreMarker = BackupClientContext::ClientStoreMarker_NotKnown; // no store marker, so download everything DeleteAllLocations(); DeleteAllIDMaps(); @@ -2294,7 +2393,7 @@ else { // there is something going on here - THROW_EXCEPTION(CommonException, Internal) + THROW_EXCEPTION(ClientException, CorruptStoreObjectInfoFile); } // @@ -2319,7 +2418,7 @@ else { // there is something going on here - THROW_EXCEPTION(CommonException, Internal) + THROW_EXCEPTION(ClientException, CorruptStoreObjectInfoFile); } // @@ -2344,7 +2443,7 @@ else { // there is something going on here - THROW_EXCEPTION(CommonException, Internal) + THROW_EXCEPTION(ClientException, CorruptStoreObjectInfoFile); } } From boxbackup-dev at fluffy.co.uk Sat Mar 24 19:44:32 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 19:44:32 +0000 Subject: [Box Backup-commit] COMMIT r1460 - box/chris/general/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-24 19:44:32 +0000 (Sat, 24 Mar 2007) New Revision: 1460 Modified: box/chris/general/bin/bbackupquery/BackupQueries.cpp box/chris/general/bin/bbackupquery/BackupQueries.h Log: Add a quiet mode to compare command, to make test output easier to read. Modified: box/chris/general/bin/bbackupquery/BackupQueries.cpp =================================================================== --- box/chris/general/bin/bbackupquery/BackupQueries.cpp 2007-03-24 19:43:31 UTC (rev 1459) +++ box/chris/general/bin/bbackupquery/BackupQueries.cpp 2007-03-24 19:44:32 UTC (rev 1460) @@ -207,7 +207,7 @@ { "sh", "" }, { "getobject", "" }, { "get", "i" }, - { "compare", "alcqAE" }, + { "compare", "alcqAEQ" }, { "restore", "dri" }, { "help", "" }, { "usage", "" }, @@ -1114,6 +1114,7 @@ // Parameters, including count of differences BackupQueries::CompareParams params; params.mQuickCompare = opts['q']; + params.mQuietCompare = opts['Q']; params.mIgnoreExcludes = opts['E']; params.mIgnoreAttributes = opts['A']; @@ -1177,12 +1178,17 @@ printf("Incorrect usage.\ncompare -a\n or compare -l \n or compare \n"); return; } - - printf("\n[ %d (of %d) differences probably due to file " - "modifications after the last upload ]\n" - "Differences: %d (%d dirs excluded, %d files excluded, " + + if (!params.mQuietCompare) + { + printf("\n[ %d (of %d) differences probably due to file " + "modifications after the last upload ]\n", + params.mDifferencesExplainedByModTime, + params.mDifferences); + } + + printf("Differences: %d (%d dirs excluded, %d files excluded, " "%d files not checked)\n", - params.mDifferencesExplainedByModTime, params.mDifferences, params.mDifferences, params.mExcludedDirs, params.mExcludedFiles, params.mUncheckedFiles); Modified: box/chris/general/bin/bbackupquery/BackupQueries.h =================================================================== --- box/chris/general/bin/bbackupquery/BackupQueries.h 2007-03-24 19:43:31 UTC (rev 1459) +++ box/chris/general/bin/bbackupquery/BackupQueries.h 2007-03-24 19:44:32 UTC (rev 1460) @@ -67,6 +67,7 @@ ~CompareParams(); void DeleteExcludeLists(); bool mQuickCompare; + bool mQuietCompare; bool mIgnoreExcludes; bool mIgnoreAttributes; int mDifferences; From boxbackup-dev at fluffy.co.uk Sat Mar 24 19:57:12 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 19:57:12 +0000 Subject: [Box Backup-commit] COMMIT r1461 - box/chris/general/test/bbackupd Message-ID: Author: chris Date: 2007-03-24 19:57:11 +0000 (Sat, 24 Mar 2007) New Revision: 1461 Modified: box/chris/general/test/bbackupd/testbbackupd.cpp Log: Keep bbackupd PID globally. Terminate bbackupd and bbstored if test_bbackupd() fails. Reformat #ifdef WIN32s for readability: short ones indent like if(), long ones (more than a screenful) outdent to make them more obvious. Add frequent checks that bbackupd and bbstored are still running during test_bbackupd(), and abort the test if either of them dies. Reformat long lines and bbackupquery lines for readability. Use bbackupquery compare -Q flag for quieter compares. Fix duplicate local variable that overrode pid on win32. Add more checks for correct behaviour of bbackupd around exception sleep/retry. Fix unlink of a file that shouldn't exist anyway, to check for existence instead. Test for success of chmod() commands. Remove Windows Sockets initialisation, which is now done by buildenv-testmain-template.cpp. Modified: box/chris/general/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-24 19:44:32 UTC (rev 1460) +++ box/chris/general/test/bbackupd/testbbackupd.cpp 2007-03-24 19:57:11 UTC (rev 1461) @@ -85,6 +85,7 @@ } int bbstored_pid = 0; +int bbackupd_pid = 0; #ifdef HAVE_SYS_XATTR_H bool readxattr_into_map(const char *filename, std::map &rOutput) @@ -758,11 +759,13 @@ TEST_THAT(::system("rm -rf testfiles/TestDir1") == 0); TEST_THAT(::mkdir("testfiles/TestDir1", 0) == 0); -#ifdef WIN32 - TEST_THAT(::system("tar xzvf testfiles/spacetest1.tgz -C testfiles/TestDir1") == 0); -#else - TEST_THAT(::system("gzip -d < testfiles/spacetest1.tgz | ( cd testfiles/TestDir1 && tar xf - )") == 0); -#endif + #ifdef WIN32 + TEST_THAT(::system("tar xzvf testfiles/spacetest1.tgz " + "-C testfiles/TestDir1") == 0); + #else + TEST_THAT(::system("gzip -d < testfiles/spacetest1.tgz " + "| ( cd testfiles/TestDir1 && tar xf - )") == 0); + #endif #ifdef PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE printf("Skipping intercept-based KeepAlive tests on this platform.\n"); @@ -1036,33 +1039,41 @@ } #endif // PLATFORM_CLIB_FNS_INTERCEPTION_IMPOSSIBLE - int pid = LaunchServer(BBACKUPD " testfiles/bbackupd.conf", + bbackupd_pid = LaunchServer(BBACKUPD " testfiles/bbackupd.conf", "testfiles/bbackupd.pid"); - TEST_THAT(pid != -1 && pid != 0); + TEST_THAT(bbackupd_pid != -1 && bbackupd_pid != 0); - if(pid > 0) + ::sleep(1); + + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + + if(bbackupd_pid > 0) { - ::sleep(1); - TEST_THAT(ServerIsAlive(pid)); - // First, check storage space handling -- wait for file to be uploaded wait_for_backup_operation(); // Set limit to something very small - // About 28 blocks will be used at this point. bbackupd will only pause if the size used is - // greater than soft limit + 1/3 of (hard - soft). Set small values for limits accordingly. + // About 28 blocks will be used at this point. bbackupd + // will only pause if the size used is greater than + // soft limit + 1/3 of (hard - soft). Set small values + // for limits accordingly. TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS " -c " "testfiles/bbstored.conf setlimit 01234567 10B 40B") == 0); TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); // Unpack some more files -#ifdef WIN32 - TEST_THAT(::system("tar xzvf testfiles/spacetest2.tgz -C testfiles/TestDir1") == 0); -#else - TEST_THAT(::system("gzip -d < testfiles/spacetest2.tgz | ( cd testfiles/TestDir1 && tar xf - )") == 0); -#endif + #ifdef WIN32 + TEST_THAT(::system("tar xzvf testfiles/spacetest2.tgz " + "-C testfiles/TestDir1") == 0); + #else + TEST_THAT(::system("gzip -d < testfiles/spacetest2.tgz " + "| ( cd testfiles/TestDir1 && tar xf - )") == 0); + #endif // Delete a file and a directory TEST_THAT(::unlink("testfiles/TestDir1/spacetest/d1/f3") == 0); @@ -1070,7 +1081,10 @@ wait_for_backup_operation(); // Make sure there are some differences - int compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query0a.log \"compare -ac\" quit"); + int compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query0a.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 2); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -1086,23 +1100,33 @@ TEST_THAT(!TestFileExists("testfiles/notifyran.store-full.2")); // unpack the initial files again -#ifdef WIN32 - TEST_THAT(::system("tar xzvf testfiles/test_base.tgz -C testfiles") == 0); -#else - TEST_THAT(::system("gzip -d < testfiles/test_base.tgz | ( cd testfiles && tar xf - )") == 0); -#endif + #ifdef WIN32 + TEST_THAT(::system("tar xzvf testfiles/test_base.tgz " + "-C testfiles") == 0); + #else + TEST_THAT(::system("gzip -d < testfiles/test_base.tgz " + "| ( cd testfiles && tar xf - )") == 0); + #endif // wait for it to do it's stuff wait_for_backup_operation(); - // Check that the contents of the store are the same as the contents - // of the disc (-a = all, -c = give result in return code) - compareReturnValue = ::system(BBACKUPQUERY " -q -c " - "testfiles/bbackupd.conf -l testfiles/query1.log " - "\"compare -ac\" quit"); + // Check that the contents of the store are the same + // as the contents of the disc + // (-a = all, -c = give result in return code) + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query1.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + #ifdef WIN32 printf("\n==== Check that filenames in UTF-8 " "can be backed up\n"); @@ -1155,7 +1179,7 @@ wait_for_backup_operation(); // Compare to check that the file was uploaded compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf \"compare -ac\" quit"); + "-c testfiles/bbackupd.conf \"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -1218,13 +1242,15 @@ consoleFileName)); // Check that bbackupquery shows the dir in console encoding - std::string command(BBACKUPQUERY " -c testfiles/bbackupd.conf " + std::string command(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " "-q \"list Test1\" quit"); - pid_t pid; + pid_t bbackupquery_pid; std::auto_ptr queryout; - queryout = LocalProcessStream(command.c_str(), pid); + queryout = LocalProcessStream(command.c_str(), + bbackupquery_pid); TEST_THAT(queryout.get() != NULL); - TEST_THAT(pid != -1); + TEST_THAT(bbackupquery_pid != -1); IOStreamGetLine reader(*queryout); std::string line; @@ -1248,9 +1274,10 @@ command = BBACKUPQUERY " -c testfiles/bbackupd.conf " "-q \"list Test1"; command += "/" + systemDirName + "\" quit"; - queryout = LocalProcessStream(command.c_str(), pid); + queryout = LocalProcessStream(command.c_str(), + bbackupquery_pid); TEST_THAT(queryout.get() != NULL); - TEST_THAT(pid != -1); + TEST_THAT(bbackupquery_pid != -1); IOStreamGetLine reader2(*queryout); found = false; @@ -1270,7 +1297,7 @@ // Check that bbackupquery can compare the dir when given // on the command line in system encoding. command = BBACKUPQUERY " -c testfiles/bbackupd.conf " - "-q \"compare -cE Test1/" + systemDirName + + "-q \"compare -cEQ Test1/" + systemDirName + " testfiles/TestDir1/" + systemDirName + "\" quit"; compareReturnValue = ::system(command.c_str()); @@ -1289,7 +1316,7 @@ // Compare to make sure it was restored properly. command = BBACKUPQUERY " -c testfiles/bbackupd.conf " - "-q \"compare -cE Test1/" + systemDirName + + "-q \"compare -cEQ Test1/" + systemDirName + " testfiles/restore-" + systemDirName + "\" quit"; compareReturnValue = ::system(command.c_str()); @@ -1315,7 +1342,7 @@ // The Get command does not restore attributes, so // we must compare without them (-A) to succeed. command = BBACKUPQUERY " -c testfiles/bbackupd.conf " - "-q \"compare -cAE Test1/" + systemDirName + + "-q \"compare -cAEQ Test1/" + systemDirName + " testfiles/restore-" + systemDirName + "\" quit"; compareReturnValue = ::system(command.c_str()); @@ -1324,7 +1351,7 @@ // Compare without attributes. This should fail. command = BBACKUPQUERY " -c testfiles/bbackupd.conf " - "-q \"compare -cE Test1/" + systemDirName + + "-q \"compare -cEQ Test1/" + systemDirName + " testfiles/restore-" + systemDirName + "\" quit"; compareReturnValue = ::system(command.c_str()); @@ -1332,6 +1359,11 @@ TEST_RETURN(compareReturnValue, 2); #endif // WIN32 + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + // Check that SyncAllowScript is executed and can pause backup printf("\n==== Check that SyncAllowScript is executed and can " "pause backup\n"); @@ -1400,7 +1432,8 @@ // check that no backup has run (compare fails) compareReturnValue = ::system(BBACKUPQUERY " -q " "-c testfiles/bbackupd.conf " - "-l testfiles/query3.log \"compare -ac\" quit"); + "-l testfiles/query3.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 2); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -1419,46 +1452,69 @@ // check that backup has run (compare succeeds) compareReturnValue = ::system(BBACKUPQUERY " -q " "-c testfiles/bbackupd.conf " - "-l testfiles/query3.log \"compare -ac\" quit"); + "-l testfiles/query3a.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); } - printf("\n==== Delete file and update another, create symlink.\n"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + + printf("\n==== Delete file and update another, " + "create symlink.\n"); // Delete a file TEST_THAT(::unlink("testfiles/TestDir1/x1/dsfdsfs98.fd") == 0); -#ifndef WIN32 - // New symlink - TEST_THAT(::symlink("does-not-exist", "testfiles/TestDir1/symlink-to-dir") == 0); -#endif + #ifndef WIN32 + // New symlink + TEST_THAT(::symlink("does-not-exist", + "testfiles/TestDir1/symlink-to-dir") == 0); + #endif + // Update a file (will be uploaded as a diff) { - // Check that the file is over the diffing threshold in the bbackupd.conf file - TEST_THAT(TestGetFileSize("testfiles/TestDir1/f45.df") > 1024); + // Check that the file is over the diffing + // threshold in the bbackupd.conf file + TEST_THAT(TestGetFileSize("testfiles/TestDir1/f45.df") + > 1024); // Add a bit to the end FILE *f = ::fopen("testfiles/TestDir1/f45.df", "a"); TEST_THAT(f != 0); ::fprintf(f, "EXTRA STUFF"); ::fclose(f); - TEST_THAT(TestGetFileSize("testfiles/TestDir1/f45.df") > 1024); + TEST_THAT(TestGetFileSize("testfiles/TestDir1/f45.df") + > 1024); } // wait for backup daemon to do it's stuff, and compare again wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query2.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query2.log " + "\"compare -acQ\" quit"); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); TEST_RETURN(compareReturnValue, 1); // Try a quick compare, just for fun - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query2q.log \"compare -acq\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query2q.log " + "\"compare -acqQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + // Check that store errors are reported neatly - printf("Create store error\n"); + printf("\n==== Create store error\n"); TEST_THAT(::rename("testfiles/0_0/backup/01234567/info.rf", "testfiles/0_0/backup/01234567/info.rf.bak") == 0); TEST_THAT(::rename("testfiles/0_1/backup/01234567/info.rf", @@ -1487,71 +1543,150 @@ "testfiles/0_1/backup/01234567/info.rf") == 0); TEST_THAT(::rename("testfiles/0_2/backup/01234567/info.rf.bak", "testfiles/0_2/backup/01234567/info.rf") == 0); - // wait until bbackupd recovers from the exception + + // Check that we DO get errors on compare + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query3b.log " + "\"compare -acQ\" quit"); + TEST_RETURN(compareReturnValue, 2); + TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + + // Wait until bbackupd recovers from the exception wait_for_backup_operation(100); + // Ensure that the force-upload file gets uploaded, + // meaning that bbackupd recovered + sync_and_wait(); + + // Check that it did get uploaded, and we have no more errors + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query3b.log " + "\"compare -acQ\" quit"); + TEST_RETURN(compareReturnValue, 1); + TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + // Bad case: delete a file/symlink, replace it with a directory printf("\n==== Replace symlink with directory, " "add new directory\n"); -#ifndef WIN32 - TEST_THAT(::unlink("testfiles/TestDir1/symlink-to-dir") == 0); -#endif - TEST_THAT(::mkdir("testfiles/TestDir1/symlink-to-dir", 0755) == 0); - TEST_THAT(::mkdir("testfiles/TestDir1/x1/dir-to-file", 0755) == 0); - // NOTE: create a file within the directory to avoid deletion by the housekeeping process later -#ifndef WIN32 - TEST_THAT(::symlink("does-not-exist", "testfiles/TestDir1/x1/dir-to-file/contents") == 0); -#endif + #ifndef WIN32 + TEST_THAT(::unlink("testfiles/TestDir1/symlink-to-dir") + == 0); + #endif + TEST_THAT(::mkdir("testfiles/TestDir1/symlink-to-dir", 0755) + == 0); + TEST_THAT(::mkdir("testfiles/TestDir1/x1/dir-to-file", 0755) + == 0); + // NOTE: create a file within the directory to + // avoid deletion by the housekeeping process later + #ifndef WIN32 + TEST_THAT(::symlink("does-not-exist", + "testfiles/TestDir1/x1/dir-to-file/contents") + == 0); + #endif wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3s.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query3c.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + // And the inverse, replace a directory with a file/symlink printf("\n==== Replace directory with symlink\n"); -#ifndef WIN32 - TEST_THAT(::unlink("testfiles/TestDir1/x1/dir-to-file/contents") == 0); -#endif + #ifndef WIN32 + TEST_THAT(::unlink("testfiles/TestDir1/x1/dir-to-file" + "/contents") == 0); + #endif TEST_THAT(::rmdir("testfiles/TestDir1/x1/dir-to-file") == 0); -#ifndef WIN32 - TEST_THAT(::symlink("does-not-exist", "testfiles/TestDir1/x1/dir-to-file") == 0); -#endif + #ifndef WIN32 + TEST_THAT(::symlink("does-not-exist", + "testfiles/TestDir1/x1/dir-to-file") == 0); + #endif wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3s.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query3d.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + // And then, put it back to how it was before. printf("\n==== Replace symlink with directory " "(which was a symlink)\n"); -#ifndef WIN32 - TEST_THAT(::unlink("testfiles/TestDir1/x1/dir-to-file") == 0); -#endif - TEST_THAT(::mkdir("testfiles/TestDir1/x1/dir-to-file", 0755) == 0); -#ifndef WIN32 - TEST_THAT(::symlink("does-not-exist", "testfiles/TestDir1/x1/dir-to-file/contents2") == 0); -#endif + #ifndef WIN32 + TEST_THAT(::unlink("testfiles/TestDir1/x1" + "/dir-to-file") == 0); + #endif + TEST_THAT(::mkdir("testfiles/TestDir1/x1/dir-to-file", + 0755) == 0); + #ifndef WIN32 + TEST_THAT(::symlink("does-not-exist", + "testfiles/TestDir1/x1/dir-to-file/contents2") + == 0); + #endif wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3s.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query3e.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); - // And finally, put it back to how it was before it was put back to how it was before - // This gets lots of nasty things in the store with directories over other old directories. + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + + // And finally, put it back to how it was before + // it was put back to how it was before + // This gets lots of nasty things in the store with + // directories over other old directories. printf("\n==== Put it all back to how it was\n"); -#ifndef WIN32 - TEST_THAT(::unlink("testfiles/TestDir1/x1/dir-to-file/contents2") == 0); -#endif + #ifndef WIN32 + TEST_THAT(::unlink("testfiles/TestDir1/x1/dir-to-file" + "/contents2") == 0); + #endif TEST_THAT(::rmdir("testfiles/TestDir1/x1/dir-to-file") == 0); -#ifndef WIN32 - TEST_THAT(::symlink("does-not-exist", "testfiles/TestDir1/x1/dir-to-file") == 0); -#endif + #ifndef WIN32 + TEST_THAT(::symlink("does-not-exist", + "testfiles/TestDir1/x1/dir-to-file") == 0); + #endif wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3s.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query3f.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + // rename an untracked file over an // existing untracked file printf("\n==== Rename over existing untracked file\n"); @@ -1572,8 +1707,9 @@ // back up both files wait_for_backup_operation(); compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query3t.log " - "\"compare -ac\" quit"); + "-c testfiles/bbackupd.conf " + "-l testfiles/query3g.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); #ifdef WIN32 @@ -1585,11 +1721,17 @@ TEST_THAT( TestFileExists("testfiles/TestDir1/untracked-2")); wait_for_backup_operation(); compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query3t.log " - "\"compare -ac\" quit"); + "-c testfiles/bbackupd.conf " + "-l testfiles/query3g.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + // case which went wrong: rename a tracked file over an // existing tracked file printf("\n==== Rename over existing tracked file\n"); @@ -1613,8 +1755,9 @@ // back up both files wait_for_backup_operation(); compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query3u.log " - "\"compare -ac\" quit"); + "-c testfiles/bbackupd.conf " + "-l testfiles/query3h.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); #ifdef WIN32 @@ -1626,45 +1769,72 @@ TEST_THAT( TestFileExists("testfiles/TestDir1/tracked-2")); wait_for_backup_operation(); compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query3v.log " - "\"compare -ac\" quit"); + "-c testfiles/bbackupd.conf " + "-l testfiles/query3i.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); - // case which went wrong: rename a tracked file over a deleted file + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + + // case which went wrong: rename a tracked file + // over a deleted file printf("\n==== Rename an existing file over a deleted file\n"); -#ifdef WIN32 - TEST_THAT(::unlink("testfiles/TestDir1/x1/dsfdsfs98.fd")); -#endif - TEST_THAT(::rename("testfiles/TestDir1/df9834.dsf", "testfiles/TestDir1/x1/dsfdsfs98.fd") == 0); + TEST_THAT(!TestFileExists("testfiles/TestDir1/x1/dsfdsfs98.fd")); + TEST_THAT(::rename("testfiles/TestDir1/df9834.dsf", + "testfiles/TestDir1/x1/dsfdsfs98.fd") == 0); wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3w.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query3j.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + printf("\n==== Add files with old times, update " "attributes of one to latest time\n"); // Move that file back - TEST_THAT(::rename("testfiles/TestDir1/x1/dsfdsfs98.fd", "testfiles/TestDir1/df9834.dsf") == 0); + TEST_THAT(::rename("testfiles/TestDir1/x1/dsfdsfs98.fd", + "testfiles/TestDir1/df9834.dsf") == 0); // Add some more files - // Because the 'm' option is not used, these files will look very old to the daemon. + // Because the 'm' option is not used, these files will + // look very old to the daemon. // Lucky it'll upload them then! -#ifdef WIN32 - TEST_THAT(::system("tar xzvf testfiles/test2.tgz -C testfiles") == 0); -#else - TEST_THAT(::system("gzip -d < testfiles/test2.tgz | ( cd testfiles && tar xf - )") == 0); - ::chmod("testfiles/TestDir1/sub23/dhsfdss/blf.h", 0415); -#endif + #ifdef WIN32 + TEST_THAT(::system("tar xzvf testfiles/test2.tgz " + "-C testfiles") == 0); + #else + TEST_THAT(::system("gzip -d < testfiles/test2.tgz " + "| ( cd testfiles && tar xf - )") == 0); + ::chmod("testfiles/TestDir1/sub23/dhsfdss/blf.h", 0415); + #endif // Wait and test wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query3k.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); - // Check that modifying files with old timestamps still get added + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + + // Check that modifying files with old timestamps + // still get added printf("\n==== Modify existing file, but change timestamp " "to rather old\n"); wait_for_sync_end(); @@ -1672,11 +1842,14 @@ // Then modify an existing file { // in the archive, it's read only -#ifdef WIN32 - TEST_THAT(::system("chmod 0777 testfiles/TestDir1/sub23/rand.h") == 0); -#else - TEST_THAT(chmod("testfiles/TestDir1/sub23/rand.h", 0777) == 0); -#endif + #ifdef WIN32 + TEST_THAT(::system("chmod 0777 testfiles" + "/TestDir1/sub23/rand.h") == 0); + #else + TEST_THAT(chmod("testfiles/TestDir1/sub23" + "/rand.h", 0777) == 0); + #endif + FILE *f = fopen("testfiles/TestDir1/sub23/rand.h", "w+"); @@ -1695,26 +1868,39 @@ // and then move the time backwards! struct timeval times[2]; - BoxTimeToTimeval(SecondsToBoxTime((time_t)(365*24*60*60)), times[1]); + BoxTimeToTimeval(SecondsToBoxTime( + (time_t)(365*24*60*60)), times[1]); times[0] = times[1]; - TEST_THAT(::utimes("testfiles/TestDir1/sub23/rand.h", times) == 0); + TEST_THAT(::utimes("testfiles/TestDir1/sub23/rand.h", + times) == 0); } // Wait and test wait_for_sync_end(); // files too new wait_for_sync_end(); // should (not) be backed up this time - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3e.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query3l.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + // Add some files and directories which are marked as excluded printf("\n==== Add files and dirs for exclusion test\n"); -#ifdef WIN32 - TEST_THAT(::system("tar xzvf testfiles/testexclude.tgz -C testfiles") == 0); -#else - TEST_THAT(::system("gzip -d < testfiles/testexclude.tgz | ( cd testfiles && tar xf - )") == 0); -#endif + #ifdef WIN32 + TEST_THAT(::system("tar xzvf testfiles/testexclude.tgz " + "-C testfiles") == 0); + #else + TEST_THAT(::system("gzip -d < " + "testfiles/testexclude.tgz " + "| ( cd testfiles && tar xf - )") == 0); + #endif // Wait and test wait_for_sync_end(); @@ -1722,18 +1908,25 @@ // compare with exclusions, should not find differences compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query3c.log " - "\"compare -ac\" quit"); + "-c testfiles/bbackupd.conf " + "-l testfiles/query3m.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // compare without exclusions, should find differences compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query3d.log " - "\"compare -acE\" quit"); + "-c testfiles/bbackupd.conf " + "-l testfiles/query3n.log " + "\"compare -acEQ\" quit"); TEST_RETURN(compareReturnValue, 2); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + // check that the excluded files did not make it // into the store, and the included files did printf("\n==== Check that exclude/alwaysinclude commands " @@ -1811,6 +2004,11 @@ connection.QueryFinished(); } + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + #ifndef WIN32 // These tests only work as non-root users. if(::getuid() != 0) @@ -1819,15 +2017,21 @@ printf("\n==== Add unreadable files\n"); { // Dir and file which can't be read - TEST_THAT(::mkdir("testfiles/TestDir1/sub23/read-fail-test-dir", 0000) == 0); - int fd = ::open("testfiles/TestDir1/read-fail-test-file", O_CREAT | O_WRONLY, 0000); + TEST_THAT(::mkdir("testfiles/TestDir1/sub23" + "/read-fail-test-dir", 0000) == 0); + int fd = ::open("testfiles/TestDir1" + "/read-fail-test-file", + O_CREAT | O_WRONLY, 0000); TEST_THAT(fd != -1); ::close(fd); } // Wait and test... wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3e.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query3o.log " + "\"compare -acQ\" quit"); // should find differences TEST_RETURN(compareReturnValue, 3); @@ -1839,12 +2043,20 @@ // Check that the error was only reported once TEST_THAT(!TestFileExists("testfiles/notifyran.read-error.2")); - // Set permissions on file and dir to stop errors in the future - ::chmod("testfiles/TestDir1/sub23/read-fail-test-dir", 0770); - ::chmod("testfiles/TestDir1/read-fail-test-file", 0770); + // Set permissions on file and dir to stop + // errors in the future + TEST_THAT(::chmod("testfiles/TestDir1/sub23" + "/read-fail-test-dir", 0770) == 0); + TEST_THAT(::chmod("testfiles/TestDir1" + "/read-fail-test-file", 0770) == 0); } #endif + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + printf("\n==== Continuously update file, " "check isn't uploaded\n"); @@ -1874,9 +2086,10 @@ // Check there's a difference #ifdef WIN32 - compareReturnValue = ::system("perl testfiles/extcheck1.pl A"); + compareReturnValue = ::system("perl testfiles/" "extcheck1.pl A"); #else - compareReturnValue = ::system("perl testfiles/extcheck1.pl"); + compareReturnValue = ::system("perl testfiles/" + "extcheck1.pl"); #endif TEST_RETURN(compareReturnValue, 1); @@ -1887,7 +2100,8 @@ for(int l = 0; l < 28; ++l) { - FILE *f = ::fopen("testfiles/TestDir1/continousupdate", "w+"); + FILE *f = ::fopen("testfiles/TestDir1/" + "continousupdate", "w+"); TEST_THAT(f != 0); fprintf(f, "Loop 2 iteration %d\n", l); fflush(f); @@ -1901,15 +2115,22 @@ fflush(stdout); #ifdef WIN32 - compareReturnValue = ::system("perl testfiles/extcheck2.pl A"); + compareReturnValue = ::system("perl testfiles/" + "extcheck2.pl A"); #else - compareReturnValue = ::system("perl testfiles/extcheck2.pl"); + compareReturnValue = ::system("perl testfiles/" + "extcheck2.pl"); #endif TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); } + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + printf("\n==== Delete directory, change attributes\n"); // Delete a directory @@ -1919,7 +2140,10 @@ // Wait and test wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query4.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query4.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -1929,20 +2153,30 @@ { // connect and log in SocketStreamTLS conn; - conn.Open(context, Socket::TypeINET, "localhost", BOX_PORT_BBSTORED); + conn.Open(context, Socket::TypeINET, "localhost", + BOX_PORT_BBSTORED); BackupProtocolClient protocol(conn); protocol.QueryVersion(BACKUP_STORE_SERVER_VERSION); - std::auto_ptr loginConf(protocol.QueryLogin(0x01234567, BackupProtocolClientLogin::Flags_ReadOnly)); + std::auto_ptr + loginConf(protocol.QueryLogin(0x01234567, + BackupProtocolClientLogin::Flags_ReadOnly)); // Find the ID of the Test1 directory - restoredirid = GetDirID(protocol, "Test1", BackupProtocolClientListDirectory::RootDirectory); + restoredirid = GetDirID(protocol, "Test1", + BackupProtocolClientListDirectory::RootDirectory); TEST_THAT(restoredirid != 0); // Test the restoration - TEST_THAT(BackupClientRestore(protocol, restoredirid, "testfiles/restore-Test1", true /* print progress dots */) == Restore_Complete); + TEST_THAT(BackupClientRestore(protocol, restoredirid, + "testfiles/restore-Test1", + true /* print progress dots */) + == Restore_Complete); // Make sure you can't restore a restored directory - TEST_THAT(BackupClientRestore(protocol, restoredirid, "testfiles/restore-Test1", true /* print progress dots */) == Restore_TargetExists); + TEST_THAT(BackupClientRestore(protocol, restoredirid, + "testfiles/restore-Test1", + true /* print progress dots */) + == Restore_TargetExists); // Make sure you can't restore to a nonexistant path printf("Try to restore to a path that doesn't exist\n"); @@ -1955,8 +2189,13 @@ deldirid = GetDirID(protocol, "x1", restoredirid); TEST_THAT(deldirid != 0); - // Just check it doesn't bomb out -- will check this properly later (when bbackupd is stopped) - TEST_THAT(BackupClientRestore(protocol, deldirid, "testfiles/restore-Test1-x1", true /* print progress dots */, true /* deleted files */) == Restore_Complete); + // Just check it doesn't bomb out -- will check this + // properly later (when bbackupd is stopped) + TEST_THAT(BackupClientRestore(protocol, deldirid, + "testfiles/restore-Test1-x1", + true /* print progress dots */, + true /* deleted files */) + == Restore_Complete); // Log out protocol.QueryFinished(); @@ -1964,21 +2203,28 @@ // Compare the restored files compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query10.log " - "\"compare -cE Test1 testfiles/restore-Test1\" " + "-c testfiles/bbackupd.conf " + "-l testfiles/query10.log " + "\"compare -cEQ Test1 testfiles/restore-Test1\" " "quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); - #ifdef WIN32 + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + +#ifdef WIN32 // make one of the files read-only, expect a compare failure compareReturnValue = ::system("attrib +r " "testfiles\\restore-Test1\\f1.dat"); TEST_RETURN(compareReturnValue, 0); compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query10a.log " - "\"compare -cE Test1 testfiles/restore-Test1\" " + "-c testfiles/bbackupd.conf " + "-l testfiles/query10a.log " + "\"compare -cEQ Test1 testfiles/restore-Test1\" " "quit"); TEST_RETURN(compareReturnValue, 2); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -1990,7 +2236,7 @@ compareReturnValue = ::system(BBACKUPQUERY " -q " "-c testfiles/bbackupd.conf -l testfiles/query10a.log " - "\"compare -cE Test1 testfiles/restore-Test1\" " + "\"compare -cEQ Test1 testfiles/restore-Test1\" " "quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -2013,8 +2259,9 @@ TEST_THAT(set_file_time(testfile, dummyTime, lastModTime, lastAccessTime)); compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query10a.log " - "\"compare -cE Test1 testfiles/restore-Test1\" " + "-c testfiles/bbackupd.conf " + "-l testfiles/query10a.log " + "\"compare -cEQ Test1 testfiles/restore-Test1\" " "quit"); TEST_RETURN(compareReturnValue, 2); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -2023,8 +2270,9 @@ TEST_THAT(set_file_time(testfile, creationTime, lastModTime, dummyTime)); compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query10a.log " - "\"compare -cE Test1 testfiles/restore-Test1\" " + "-c testfiles/bbackupd.conf " + "-l testfiles/query10a.log " + "\"compare -cEQ Test1 testfiles/restore-Test1\" " "quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -2034,8 +2282,9 @@ TEST_THAT(set_file_time(testfile, creationTime, dummyTime, lastAccessTime)); compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query10a.log " - "\"compare -cE Test1 testfiles/restore-Test1\" " + "-c testfiles/bbackupd.conf " + "-l testfiles/query10a.log " + "\"compare -cEQ Test1 testfiles/restore-Test1\" " "quit"); TEST_RETURN(compareReturnValue, 2); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -2044,52 +2293,88 @@ TEST_THAT(set_file_time(testfile, creationTime, lastModTime, lastAccessTime)); compareReturnValue = ::system(BBACKUPQUERY " -q " - "-c testfiles/bbackupd.conf -l testfiles/query10a.log " - "\"compare -cE Test1 testfiles/restore-Test1\" " + "-c testfiles/bbackupd.conf " + "-l testfiles/query10a.log " + "\"compare -cEQ Test1 testfiles/restore-Test1\" " "quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); - #endif +#endif + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + printf("\n==== Add files with current time\n"); // Add some more files and modify others // Use the m flag this time so they have a recent modification time -#ifdef WIN32 - TEST_THAT(::system("tar xzvmf testfiles/test3.tgz -C testfiles") == 0); -#else - TEST_THAT(::system("gzip -d < testfiles/test3.tgz | ( cd testfiles && tar xmf - )") == 0); -#endif + #ifdef WIN32 + TEST_THAT(::system("tar xzvmf testfiles/test3.tgz " + "-C testfiles") == 0); + #else + TEST_THAT(::system("gzip -d < testfiles/test3.tgz " + "| ( cd testfiles && tar xmf - )") == 0); + #endif // Wait and test wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query5.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query5.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + // Rename directory printf("\n==== Rename directory\n"); - TEST_THAT(rename("testfiles/TestDir1/sub23/dhsfdss", "testfiles/TestDir1/renamed-dir") == 0); + TEST_THAT(rename("testfiles/TestDir1/sub23/dhsfdss", + "testfiles/TestDir1/renamed-dir") == 0); wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query6.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query6.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + // and again, but with quick flag - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query6q.log \"compare -acq\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query6q.log " + "\"compare -acqQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // Rename some files -- one under the threshold, others above printf("\n==== Rename files\n"); - TEST_THAT(rename("testfiles/TestDir1/continousupdate", "testfiles/TestDir1/continousupdate-ren") == 0); - TEST_THAT(rename("testfiles/TestDir1/df324", "testfiles/TestDir1/df324-ren") == 0); - TEST_THAT(rename("testfiles/TestDir1/sub23/find2perl", "testfiles/TestDir1/find2perl-ren") == 0); + TEST_THAT(rename("testfiles/TestDir1/continousupdate", + "testfiles/TestDir1/continousupdate-ren") == 0); + TEST_THAT(rename("testfiles/TestDir1/df324", + "testfiles/TestDir1/df324-ren") == 0); + TEST_THAT(rename("testfiles/TestDir1/sub23/find2perl", + "testfiles/TestDir1/find2perl-ren") == 0); wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query6.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query6.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); - // Check that modifying files with madly in the future timestamps still get added + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + + // Check that modifying files with madly in the future + // timestamps still get added printf("\n==== Create a file with timestamp way ahead " "in the future\n"); // Time critical, so sync @@ -2100,26 +2385,39 @@ // Then modify an existing file { - FILE *f = fopen("testfiles/TestDir1/sub23/in-the-future", "w"); + FILE *f = fopen("testfiles/TestDir1/sub23/" + "in-the-future", "w"); TEST_THAT(f != 0); fprintf(f, "Back to the future!\n"); fclose(f); // and then move the time forwards! struct timeval times[2]; - BoxTimeToTimeval(GetCurrentBoxTime() + SecondsToBoxTime((time_t)(365*24*60*60)), times[1]); + BoxTimeToTimeval(GetCurrentBoxTime() + + SecondsToBoxTime((time_t)(365*24*60*60)), + times[1]); times[0] = times[1]; - TEST_THAT(::utimes("testfiles/TestDir1/sub23/in-the-future", times) == 0); + TEST_THAT(::utimes("testfiles/TestDir1/sub23/" + "in-the-future", times) == 0); } // Wait and test wait_for_backup_operation(); - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query3e.log \"compare -ac\" quit"); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query3e.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + printf("\n==== Change client store marker\n"); - // Then... connect to the server, and change the client store marker. See what that does! + // Then... connect to the server, and change the + // client store marker. See what that does! { bool done = false; int tries = 4; @@ -2128,7 +2426,8 @@ try { SocketStreamTLS conn; - conn.Open(context, Socket::TypeINET, "localhost", BOX_PORT_BBSTORED); + conn.Open(context, Socket::TypeINET, + "localhost", BOX_PORT_BBSTORED); BackupProtocolClient protocol(conn); protocol.QueryVersion(BACKUP_STORE_SERVER_VERSION); std::auto_ptr loginConf(protocol.QueryLogin(0x01234567, 0)); // read-write @@ -2152,6 +2451,11 @@ TEST_THAT(done); } + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + printf("\n==== Check change of store marker pauses daemon\n"); // Make a change to a file, to detect whether or not @@ -2164,60 +2468,103 @@ } // Wait and test that there *are* differences - wait_for_backup_operation((TIME_TO_WAIT_FOR_BACKUP_OPERATION * 3) / 2); // little bit longer than usual - compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf -l testfiles/query6.log \"compare -ac\" quit"); + wait_for_backup_operation((TIME_TO_WAIT_FOR_BACKUP_OPERATION * + 3) / 2); // little bit longer than usual + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query6.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 2); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + printf("\n==== Waiting for bbackupd to recover\n"); // 100 seconds - (12*3/2) wait_for_operation(82); + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + #ifndef WIN32 printf("\n==== Interrupted restore\n"); { do_interrupted_restore(context, restoredirid); int64_t resumesize = 0; - TEST_THAT(FileExists("testfiles/restore-interrupt.boxbackupresume", &resumesize)); - TEST_THAT(resumesize > 16); // make sure it has recorded something to resume + TEST_THAT(FileExists("testfiles/" + "restore-interrupt.boxbackupresume", + &resumesize)); + // make sure it has recorded something to resume + TEST_THAT(resumesize > 16); - printf("\nResume restore\n"); + printf("\n==== Resume restore\n"); SocketStreamTLS conn; - conn.Open(context, Socket::TypeINET, "localhost", BOX_PORT_BBSTORED); + conn.Open(context, Socket::TypeINET, "localhost", + BOX_PORT_BBSTORED); BackupProtocolClient protocol(conn); protocol.QueryVersion(BACKUP_STORE_SERVER_VERSION); - std::auto_ptr loginConf(protocol.QueryLogin(0x01234567, 0)); // read-write + std::auto_ptr + loginConf(protocol.QueryLogin(0x01234567, + 0 /* read-write */)); - // Check that the restore fn returns resume possible, rather than doing anything - TEST_THAT(BackupClientRestore(protocol, restoredirid, "testfiles/restore-interrupt", true /* print progress dots */) == Restore_ResumePossible); + // Check that the restore fn returns resume possible, + // rather than doing anything + TEST_THAT(BackupClientRestore(protocol, restoredirid, + "testfiles/restore-interrupt", + true /* print progress dots */) + == Restore_ResumePossible); // Then resume it - TEST_THAT(BackupClientRestore(protocol, restoredirid, "testfiles/restore-interrupt", true /* print progress dots */, false /* deleted files */, false /* undelete server */, true /* resume */) == Restore_Complete); + TEST_THAT(BackupClientRestore(protocol, restoredirid, + "testfiles/restore-interrupt", + true /* print progress dots */, + false /* deleted files */, + false /* undelete server */, + true /* resume */) + == Restore_Complete); protocol.QueryFinished(); // Then check it has restored the correct stuff - compareReturnValue = ::system(BBACKUPQUERY - " -q -c testfiles/bbackupd.conf " + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " "-l testfiles/query14.log " - "\"compare -cE Test1 " + "\"compare -cEQ Test1 " "testfiles/restore-interrupt\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); } #endif + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + printf("\n==== Check restore deleted files\n"); { SocketStreamTLS conn; - conn.Open(context, Socket::TypeINET, "localhost", BOX_PORT_BBSTORED); + conn.Open(context, Socket::TypeINET, "localhost", + BOX_PORT_BBSTORED); BackupProtocolClient protocol(conn); protocol.QueryVersion(BACKUP_STORE_SERVER_VERSION); - std::auto_ptr loginConf(protocol.QueryLogin(0x01234567, 0)); // read-write + std::auto_ptr + loginConf(protocol.QueryLogin(0x01234567, + 0 /* read-write */)); // Do restore and undelete - TEST_THAT(BackupClientRestore(protocol, deldirid, "testfiles/restore-Test1-x1-2", true /* print progress dots */, true /* deleted files */, true /* undelete on server */) == Restore_Complete); + TEST_THAT(BackupClientRestore(protocol, deldirid, + "testfiles/restore-Test1-x1-2", + true /* print progress dots */, + true /* deleted files */, + true /* undelete on server */) + == Restore_Complete); protocol.QueryFinished(); @@ -2225,7 +2572,7 @@ compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf " "-l testfiles/query11.log " - "\"compare -cE " + "\"compare -cEQ " "Test1/x1 testfiles/restore-Test1-x1-2\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -2235,7 +2582,12 @@ TEST_THAT(!TestFileExists("testfiles/notifyran.store-full.2")); TEST_THAT(!TestFileExists("testfiles/notifyran.read-error.2")); - #ifdef WIN32 + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + +#ifdef WIN32 printf("\n==== Testing locked file behaviour:\n"); // Test that locked files cannot be backed up, @@ -2252,26 +2604,30 @@ { // first sync will ignore the file, it's too new wait_for_sync_end(); - TEST_THAT(!TestFileExists("testfiles/notifyran.read-error.1")); + TEST_THAT(!TestFileExists("testfiles/" + "notifyran.read-error.1")); // this sync should try to back up the file, // and fail, because it's locked wait_for_sync_end(); - TEST_THAT(TestFileExists("testfiles/notifyran.read-error.1")); - TEST_THAT(!TestFileExists("testfiles/notifyran.read-error.2")); + TEST_THAT(TestFileExists("testfiles/" + "notifyran.read-error.1")); + TEST_THAT(!TestFileExists("testfiles/" + "notifyran.read-error.2")); // now close the file and check that it is // backed up on the next run. CloseHandle(handle); wait_for_sync_end(); - TEST_THAT(!TestFileExists("testfiles/notifyran.read-error.2")); + TEST_THAT(!TestFileExists("testfiles/" + "notifyran.read-error.2")); // compare, and check that it works // reports the correct error message (and finishes) - compareReturnValue = ::system(BBACKUPQUERY - " -q -c testfiles/bbackupd.conf " + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " "-l testfiles/query15a.log " - "\"compare -ac\" quit"); + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -2284,7 +2640,7 @@ compareReturnValue = ::system(BBACKUPQUERY " -q -c testfiles/bbackupd.conf " "-l testfiles/query15.log " - "\"compare -ac\" quit"); + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 3); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); @@ -2292,36 +2648,44 @@ // works again CloseHandle(handle); - compareReturnValue = ::system(BBACKUPQUERY - " -q -c testfiles/bbackupd.conf " + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " "-l testfiles/query15a.log " - "\"compare -ac\" quit"); + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); } - #endif +#endif // Kill the daemon - terminate_bbackupd(pid); + terminate_bbackupd(bbackupd_pid); // Start it again - pid = LaunchServer(BBACKUPD " testfiles/bbackupd.conf", + bbackupd_pid = LaunchServer(BBACKUPD + " testfiles/bbackupd.conf", "testfiles/bbackupd.pid"); - TEST_THAT(pid != -1 && pid != 0); + TEST_THAT(bbackupd_pid != -1 && bbackupd_pid != 0); - if(pid != -1 && pid != 0) + TEST_THAT(ServerIsAlive(bbackupd_pid)); + TEST_THAT(ServerIsAlive(bbstored_pid)); + if (!ServerIsAlive(bbackupd_pid)) return 1; + if (!ServerIsAlive(bbstored_pid)) return 1; + + if(bbackupd_pid != -1 && bbackupd_pid != 0) { - // Wait and compare - wait_for_backup_operation((TIME_TO_WAIT_FOR_BACKUP_OPERATION*3) / 2); // little bit longer than usual - compareReturnValue = ::system(BBACKUPQUERY - " -q -c testfiles/bbackupd.conf " - "-l testfiles/query4a.log \"compare -ac\" quit"); + // Wait and compare (a little bit longer than usual) + wait_for_backup_operation( + (TIME_TO_WAIT_FOR_BACKUP_OPERATION*3) / 2); + compareReturnValue = ::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf " + "-l testfiles/query4a.log " + "\"compare -acQ\" quit"); TEST_RETURN(compareReturnValue, 1); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); // Kill it again - terminate_bbackupd(pid); + terminate_bbackupd(bbackupd_pid); } } @@ -2330,26 +2694,19 @@ "-l testfiles/queryLIST.log \"list -rotdh\" quit"); TestRemoteProcessMemLeaks("bbackupquery.memleaks"); -#ifndef WIN32 - if(::getuid() == 0) - { - ::printf("WARNING: This test was run as root. Some tests have been omitted.\n"); - } -#endif + #ifndef WIN32 + if(::getuid() == 0) + { + ::printf("WARNING: This test was run as root. " + "Some tests have been omitted.\n"); + } + #endif return 0; } int test(int argc, const char *argv[]) { -#ifdef WIN32 - // Under win32 we must initialise the Winsock library - // before using sockets - - WSADATA info; - TEST_THAT(WSAStartup(0x0101, &info) != SOCKET_ERROR) -#endif - // SSL library SSLLib::Initialise(); @@ -2357,11 +2714,13 @@ BackupClientCryptoKeys_Setup("testfiles/bbackupd.keys"); // Initial files -#ifdef WIN32 - TEST_THAT(::system("tar xzvf testfiles/test_base.tgz -C testfiles") == 0); -#else - TEST_THAT(::system("gzip -d < testfiles/test_base.tgz | ( cd testfiles && tar xf - )") == 0); -#endif + #ifdef WIN32 + TEST_THAT(::system("tar xzvf testfiles/test_base.tgz " + "-C testfiles") == 0); + #else + TEST_THAT(::system("gzip -d < testfiles/test_base.tgz " + "| ( cd testfiles && tar xf - )") == 0); + #endif // Do the tests @@ -2375,7 +2734,12 @@ if(r != 0) return r; r = test_bbackupd(); - if(r != 0) return r; + if(r != 0) + { + KillServer(bbackupd_pid); + KillServer(bbstored_pid); + return r; + } test_kill_bbstored(); From boxbackup-dev at fluffy.co.uk Sat Mar 24 19:57:50 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 19:57:50 +0000 Subject: [Box Backup-commit] COMMIT r1462 - box/chris/general/bin/bbstored Message-ID: Author: chris Date: 2007-03-24 19:57:50 +0000 (Sat, 24 Mar 2007) New Revision: 1462 Modified: box/chris/general/bin/bbstored/bbstored.cpp Log: Initialise logging framework and set our program name to Box Backup (bbstored) Modified: box/chris/general/bin/bbstored/bbstored.cpp =================================================================== --- box/chris/general/bin/bbstored/bbstored.cpp 2007-03-24 19:57:11 UTC (rev 1461) +++ box/chris/general/bin/bbstored/bbstored.cpp 2007-03-24 19:57:50 UTC (rev 1462) @@ -10,6 +10,7 @@ #include "Box.h" #include "BackupStoreDaemon.h" #include "MainHelper.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -17,6 +18,10 @@ { MAINHELPER_START + Logging::SetProgramName("Box Backup (bbstored)"); + Logging::ToConsole(true); + Logging::ToSyslog (true); + BackupStoreDaemon daemon; return daemon.Main(BOX_FILE_BBSTORED_DEFAULT_CONFIG, argc, argv); From boxbackup-dev at fluffy.co.uk Sat Mar 24 20:23:16 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 20:23:16 +0000 Subject: [Box Backup-commit] COMMIT r1463 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-24 20:23:16 +0000 (Sat, 24 Mar 2007) New Revision: 1463 Modified: box/chris/general/lib/server/WinNamedPipeStream.cpp Log: Treat ERR_PIPE_NOT_CONNECTED during pipe reads as EOF as well Modified: box/chris/general/lib/server/WinNamedPipeStream.cpp =================================================================== --- box/chris/general/lib/server/WinNamedPipeStream.cpp 2007-03-24 19:57:50 UTC (rev 1462) +++ box/chris/general/lib/server/WinNamedPipeStream.cpp 2007-03-24 20:23:16 UTC (rev 1463) @@ -380,7 +380,8 @@ // ERROR_NO_DATA is a strange name for // "The pipe is being closed". No exception wanted. - if (err == ERROR_NO_DATA) + if (err == ERROR_NO_DATA || + err == ERROR_PIPE_NOT_CONNECTED) { NumBytesRead = 0; } From boxbackup-dev at fluffy.co.uk Sat Mar 24 21:42:00 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 21:42:00 +0000 Subject: [Box Backup-commit] COMMIT r1464 - box/chris/general/lib/server Message-ID: Author: chris Date: 2007-03-24 21:42:00 +0000 (Sat, 24 Mar 2007) New Revision: 1464 Modified: box/chris/general/lib/server/WinNamedPipeStream.cpp Log: Change named pipe from message to byte mode. Thanks to Charles for pointing this out. Modified: box/chris/general/lib/server/WinNamedPipeStream.cpp =================================================================== --- box/chris/general/lib/server/WinNamedPipeStream.cpp 2007-03-24 20:23:16 UTC (rev 1463) +++ box/chris/general/lib/server/WinNamedPipeStream.cpp 2007-03-24 21:42:00 UTC (rev 1464) @@ -89,8 +89,8 @@ pName, // pipe name PIPE_ACCESS_DUPLEX | // read/write access FILE_FLAG_OVERLAPPED, // enabled overlapped I/O - PIPE_TYPE_MESSAGE | // message type pipe - PIPE_READMODE_MESSAGE | // message-read mode + PIPE_TYPE_BYTE | // message type pipe + PIPE_READMODE_BYTE | // message-read mode PIPE_WAIT, // blocking mode 1, // max. instances 4096, // output buffer size From boxbackup-dev at fluffy.co.uk Sat Mar 24 21:50:37 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 21:50:37 +0000 Subject: [Box Backup-commit] COMMIT r1465 - box/chris/merge/lib/server Message-ID: Author: chris Date: 2007-03-24 21:50:37 +0000 (Sat, 24 Mar 2007) New Revision: 1465 Modified: box/chris/merge/lib/server/WinNamedPipeStream.cpp Log: Reinitialise the OVERLAPPED structure each time we start a new overlapped read. Thanks to Charles Lecklider for pointing this out. (refs #3, merges part of [1458]) Modified: box/chris/merge/lib/server/WinNamedPipeStream.cpp =================================================================== --- box/chris/merge/lib/server/WinNamedPipeStream.cpp 2007-03-24 21:42:00 UTC (rev 1464) +++ box/chris/merge/lib/server/WinNamedPipeStream.cpp 2007-03-24 21:50:37 UTC (rev 1465) @@ -298,6 +298,13 @@ mBytesInBuffer = BytesRemaining; NumBytesRead = BytesToCopy; + if (needAnotherRead) + { + // reinitialise the OVERLAPPED structure + memset(&mReadOverlap, 0, sizeof(mReadOverlap)); + mReadOverlap.hEvent = mReadableEvent; + } + // start the next overlapped read if (needAnotherRead && !ReadFile(mSocketHandle, mReadBuffer + mBytesInBuffer, From boxbackup-dev at fluffy.co.uk Sat Mar 24 21:51:58 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 21:51:58 +0000 Subject: [Box Backup-commit] COMMIT r1466 - box/chris/merge/lib/server Message-ID: Author: chris Date: 2007-03-24 21:51:58 +0000 (Sat, 24 Mar 2007) New Revision: 1466 Modified: box/chris/merge/lib/server/WinNamedPipeStream.cpp Log: Use memmove() for overlapping source and destination buffer. Thanks to Charles Lecklider for pointing this out. (refs #3, merges [1442]) Modified: box/chris/merge/lib/server/WinNamedPipeStream.cpp =================================================================== --- box/chris/merge/lib/server/WinNamedPipeStream.cpp 2007-03-24 21:50:37 UTC (rev 1465) +++ box/chris/merge/lib/server/WinNamedPipeStream.cpp 2007-03-24 21:51:58 UTC (rev 1466) @@ -293,7 +293,7 @@ } memcpy(pBuffer, mReadBuffer, BytesToCopy); - memcpy(mReadBuffer, mReadBuffer + BytesToCopy, BytesRemaining); + memmove(mReadBuffer, mReadBuffer + BytesToCopy, BytesRemaining); mBytesInBuffer = BytesRemaining; NumBytesRead = BytesToCopy; From boxbackup-dev at fluffy.co.uk Sat Mar 24 21:53:01 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 21:53:01 +0000 Subject: [Box Backup-commit] COMMIT r1467 - box/chris/merge/lib/server Message-ID: Author: chris Date: 2007-03-24 21:53:01 +0000 (Sat, 24 Mar 2007) New Revision: 1467 Modified: box/chris/merge/lib/server/WinNamedPipeStream.cpp Log: Change named pipe from message to byte mode. Thanks to Charles Lecklider for pointing this out. Modified: box/chris/merge/lib/server/WinNamedPipeStream.cpp =================================================================== --- box/chris/merge/lib/server/WinNamedPipeStream.cpp 2007-03-24 21:51:58 UTC (rev 1466) +++ box/chris/merge/lib/server/WinNamedPipeStream.cpp 2007-03-24 21:53:01 UTC (rev 1467) @@ -89,8 +89,8 @@ pName, // pipe name PIPE_ACCESS_DUPLEX | // read/write access FILE_FLAG_OVERLAPPED, // enabled overlapped I/O - PIPE_TYPE_MESSAGE | // message type pipe - PIPE_READMODE_MESSAGE | // message-read mode + PIPE_TYPE_BYTE | // message type pipe + PIPE_READMODE_BYTE | // message-read mode PIPE_WAIT, // blocking mode 1, // max. instances 4096, // output buffer size From boxbackup-dev at fluffy.co.uk Sat Mar 24 21:54:17 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 21:54:17 -0000 Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> References: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> Message-ID: <060.f1fa8982587ea4366fa51a8059da4e96@fluffy.co.uk> #3: Merge Win32 branch -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: trunk Resolution: | Keywords: windows win32 merge branch -----------------------+---------------------------------------------------- Comment (by chris): (In [1467]) Change named pipe from message to byte mode. Thanks to Charles Lecklider for pointing this out. (merges [1464]) -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sat Mar 24 21:55:49 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 21:55:49 -0000 Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> References: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> Message-ID: <060.51801bb4d0abe2bd124757593383770f@fluffy.co.uk> #3: Merge Win32 branch -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: trunk Resolution: | Keywords: windows win32 merge branch -----------------------+---------------------------------------------------- Comment (by chris): Hi Charles, please could you review again [1083]+[1465]+[1466]+[1467]? I hope this addresses all the issues that you rightly pointed out. Thanks again for reviewing! -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sat Mar 24 22:43:50 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 22:43:50 +0000 Subject: [Box Backup-commit] COMMIT r1468 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-24 22:43:50 +0000 (Sat, 24 Mar 2007) New Revision: 1468 Modified: box/chris/general/lib/common/Guards.h Log: Fix compilation error reported by Torsten Boob (merges [1437]) Modified: box/chris/general/lib/common/Guards.h =================================================================== --- box/chris/general/lib/common/Guards.h 2007-03-24 21:53:01 UTC (rev 1467) +++ box/chris/general/lib/common/Guards.h 2007-03-24 22:43:50 UTC (rev 1468) @@ -15,9 +15,12 @@ #include #endif +#include #include #include +#include #include + #include #include "CommonException.h" From boxbackup-dev at fluffy.co.uk Sat Mar 24 22:53:45 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 22:53:45 +0000 Subject: [Box Backup-commit] COMMIT r1469 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-24 22:53:45 +0000 (Sat, 24 Mar 2007) New Revision: 1469 Modified: box/chris/merge/lib/win32/emu.cpp box/chris/merge/lib/win32/emu.h Log: Add emulated rename() with path conversion. (refs #3, merges [1436] and [1438]) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-24 22:43:50 UTC (rev 1468) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-24 22:53:45 UTC (rev 1469) @@ -1609,6 +1609,72 @@ return 0; } +int emu_rename(const char* pOldFileName, const char* pNewFileName) +{ + std::string OldPathWithUnicode = + ConvertPathToAbsoluteUnicode(pOldFileName); + + if (OldPathWithUnicode.size() == 0) + { + // error already logged by ConvertPathToAbsoluteUnicode() + return -1; + } + + WCHAR* pOldBuffer = ConvertUtf8ToWideString(OldPathWithUnicode.c_str()); + if (!pOldBuffer) + { + return -1; + } + + std::string NewPathWithUnicode = + ConvertPathToAbsoluteUnicode(pNewFileName); + + if (NewPathWithUnicode.size() == 0) + { + // error already logged by ConvertPathToAbsoluteUnicode() + delete [] pOldBuffer; + return -1; + } + + WCHAR* pNewBuffer = ConvertUtf8ToWideString(NewPathWithUnicode.c_str()); + if (!pNewBuffer) + { + delete [] pOldBuffer; + return -1; + } + + BOOL result = MoveFileW(pOldBuffer, pNewBuffer); + DWORD err = GetLastError(); + delete [] pOldBuffer; + delete [] pNewBuffer; + + if (!result) + { + if (err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND) + { + errno = ENOENT; + } + else if (err == ERROR_SHARING_VIOLATION) + { + errno = EBUSY; + } + else if (err == ERROR_ACCESS_DENIED) + { + errno = EACCES; + } + else + { + ::syslog(LOG_WARNING, "Failed to rename file " + "'%s' to '%s': %s", pOldFileName, pNewFileName, + GetErrorMessage(err).c_str()); + errno = ENOSYS; + } + return -1; + } + + return 0; +} + int console_read(char* pBuffer, size_t BufferSize) { HANDLE hConsole = GetStdHandle(STD_INPUT_HANDLE); Modified: box/chris/merge/lib/win32/emu.h =================================================================== --- box/chris/merge/lib/win32/emu.h 2007-03-24 22:43:50 UTC (rev 1468) +++ box/chris/merge/lib/win32/emu.h 2007-03-24 22:53:45 UTC (rev 1469) @@ -337,6 +337,7 @@ int emu_utimes (const char* pName, const struct timeval[]); int emu_chmod (const char* pName, mode_t mode); char* emu_getcwd (char* pBuffer, int BufSize); +int emu_rename (const char* pOldName, const char* pNewName); #define chdir(directory) emu_chdir (directory) #define mkdir(path, mode) emu_mkdir (path) @@ -347,6 +348,7 @@ #define utimes(buffer, times) emu_utimes (buffer, times) #define chmod(file, mode) emu_chmod (file, mode) #define getcwd(buffer, size) emu_getcwd (buffer, size) +#define rename(oldname, newname) emu_rename (oldname, newname) int statfs(const char * name, struct statfs * s); From boxbackup-dev at fluffy.co.uk Sat Mar 24 22:54:30 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 22:54:30 +0000 Subject: [Box Backup-commit] COMMIT r1470 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-24 22:54:30 +0000 (Sat, 24 Mar 2007) New Revision: 1470 Modified: box/chris/merge/lib/win32/emu.cpp Log: Initialise logging framework and set our program name to Box Backup (bbstored) (refs #3, merges [1462]) Modified: box/chris/merge/lib/win32/emu.cpp =================================================================== --- box/chris/merge/lib/win32/emu.cpp 2007-03-24 22:53:45 UTC (rev 1469) +++ box/chris/merge/lib/win32/emu.cpp 2007-03-24 22:54:30 UTC (rev 1470) @@ -1345,13 +1345,17 @@ { } - if (!AddEventSource("Box Backup", 0)) + char* name = strdup(daemonName); + BOOL success = AddEventSource(name, 0); + free(name); + + if (!success) { ::syslog(LOG_ERR, "Failed to add our own event source"); return; } - HANDLE newSyslogH = RegisterEventSource(NULL, "Box Backup"); + HANDLE newSyslogH = RegisterEventSource(NULL, daemonName); if (newSyslogH == NULL) { ::syslog(LOG_ERR, "Failed to register our own event source: " From boxbackup-dev at fluffy.co.uk Sat Mar 24 22:56:13 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 22:56:13 +0000 Subject: [Box Backup-commit] COMMIT r1471 - in box/chris/merge/lib: backupclient server Message-ID: Author: chris Date: 2007-03-24 22:56:13 +0000 (Sat, 24 Mar 2007) New Revision: 1471 Modified: box/chris/merge/lib/backupclient/BackupStoreFile.cpp box/chris/merge/lib/backupclient/BackupStoreFilenameClear.cpp box/chris/merge/lib/server/Protocol.cpp Log: Use logging framework to reduce noise for those who don't want it (refs #3, merges [1439] [1440] [1443]) Modified: box/chris/merge/lib/backupclient/BackupStoreFile.cpp =================================================================== --- box/chris/merge/lib/backupclient/BackupStoreFile.cpp 2007-03-24 22:54:30 UTC (rev 1470) +++ box/chris/merge/lib/backupclient/BackupStoreFile.cpp 2007-03-24 22:56:13 UTC (rev 1471) @@ -46,6 +46,7 @@ #include "ReadGatherStream.h" #include "Random.h" #include "BackupStoreFileEncodeStream.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -1485,9 +1486,8 @@ // -------------------------------------------------------------------------- void BackupStoreFile::EncodingBuffer::Reallocate(int NewSize) { -#ifndef WIN32 - TRACE2("Reallocating EncodingBuffer from %d to %d\n", mBufferSize, NewSize); -#endif + BOX_TRACE("Reallocating EncodingBuffer from " << mBufferSize << + " to " << NewSize); ASSERT(mpBuffer != 0); uint8_t *buffer = (uint8_t*)BackupStoreFile::CodingChunkAlloc(NewSize); if(buffer == 0) Modified: box/chris/merge/lib/backupclient/BackupStoreFilenameClear.cpp =================================================================== --- box/chris/merge/lib/backupclient/BackupStoreFilenameClear.cpp 2007-03-24 22:54:30 UTC (rev 1470) +++ box/chris/merge/lib/backupclient/BackupStoreFilenameClear.cpp 2007-03-24 22:56:13 UTC (rev 1471) @@ -13,6 +13,7 @@ #include "CipherContext.h" #include "CipherBlowfish.h" #include "Guards.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -203,9 +204,9 @@ { if(sEncDecBufferSize < BufSize) { -#ifndef WIN32 - TRACE2("Reallocating filename encoding/decoding buffer from %d to %d\n", sEncDecBufferSize, BufSize); -#endif + BOX_TRACE("Reallocating filename encoding/decoding " + "buffer from " << sEncDecBufferSize << + " to " << BufSize); spEncDecBuffer->Resize(BufSize); sEncDecBufferSize = BufSize; MEMLEAKFINDER_NOT_A_LEAK(*spEncDecBuffer); Modified: box/chris/merge/lib/server/Protocol.cpp =================================================================== --- box/chris/merge/lib/server/Protocol.cpp 2007-03-24 22:54:30 UTC (rev 1470) +++ box/chris/merge/lib/server/Protocol.cpp 2007-03-24 22:56:13 UTC (rev 1471) @@ -22,6 +22,7 @@ #include "ServerException.h" #include "PartialReadStream.h" #include "ProtocolUncertainStream.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -55,7 +56,8 @@ mLastErrorType(NoError), mLastErrorSubType(NoError) { - TRACE1("Send block allocation size is %d\n", PROTOCOL_ALLOCATE_SEND_BLOCK_CHUNK); + BOX_TRACE("Send block allocation size is " << + PROTOCOL_ALLOCATE_SEND_BLOCK_CHUNK); } // -------------------------------------------------------------------------- From boxbackup-dev at fluffy.co.uk Sat Mar 24 22:58:16 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 22:58:16 +0000 Subject: [Box Backup-commit] COMMIT r1472 - box/chris/merge/lib/server Message-ID: Author: chris Date: 2007-03-24 22:58:16 +0000 (Sat, 24 Mar 2007) New Revision: 1472 Modified: box/chris/merge/lib/server/Daemon.cpp Log: Reduce default logging level in debug builds from TRACE back down to INFO, to reduce noise in tests. (refs #3, merges [1441]) Modified: box/chris/merge/lib/server/Daemon.cpp =================================================================== --- box/chris/merge/lib/server/Daemon.cpp 2007-03-24 22:56:13 UTC (rev 1471) +++ box/chris/merge/lib/server/Daemon.cpp 2007-03-24 22:58:16 UTC (rev 1472) @@ -105,7 +105,7 @@ #ifdef NDEBUG int masterLevel = Log::NOTICE; // need an int to do math with #else - int masterLevel = Log::TRACE; // need an int to do math with + int masterLevel = Log::INFO; // need an int to do math with #endif char c; From boxbackup-dev at fluffy.co.uk Sat Mar 24 22:59:26 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 22:59:26 +0000 Subject: [Box Backup-commit] COMMIT r1474 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-24 22:59:26 +0000 (Sat, 24 Mar 2007) New Revision: 1474 Modified: box/chris/merge/lib/common/Timer.cpp Log: Use logging framework to remove timer noise for those who don't want it. (refs #3, merges [1445]) Modified: box/chris/merge/lib/common/Timer.cpp =================================================================== --- box/chris/merge/lib/common/Timer.cpp 2007-03-24 22:58:55 UTC (rev 1473) +++ box/chris/merge/lib/common/Timer.cpp 2007-03-24 22:59:26 UTC (rev 1474) @@ -175,11 +175,10 @@ if (timeToExpiry <= 0) { - TRACE3("%d.%d: timer %p has expired, " - "triggering it\n", - (int)(timeNow / 1000000), - (int)(timeNow % 1000000), - *i); + BOX_TRACE((int)(timeNow / 1000000) << "." << + (int)(timeNow % 1000000) << + ": timer " << *i << " has expired, " + "triggering it"); rTimer.OnExpire(); spTimers->erase(i); restart = true; @@ -187,13 +186,13 @@ } else { - TRACE5("%d.%d: timer %p has not expired, " - "triggering in %d.%d seconds\n", - (int)(timeNow / 1000000), - (int)(timeNow % 1000000), - *i, - (int)(timeToExpiry / 1000000), - (int)(timeToExpiry % 1000000)); + BOX_TRACE((int)(timeNow / 1000000) << "." << + (int)(timeNow % 1000000) << + ": timer " << *i << " has not " + "expired, triggering in " << + (int)(timeToExpiry / 1000000) << "." << + (int)(timeToExpiry % 1000000) << + " seconds"); } } } @@ -231,7 +230,7 @@ if(::setitimer(ITIMER_REAL, &timeout, NULL) != 0) { - TRACE0("WARNING: couldn't initialise timer\n"); + BOX_ERROR("Failed to initialise timer\n"); THROW_EXCEPTION(CommonException, Internal) } } @@ -263,15 +262,16 @@ gettimeofday(&tv, NULL); if (timeoutSecs == 0) { - TRACE4("%d.%d: timer %p initialised for %d secs, " - "will not fire\n", tv.tv_sec, tv.tv_usec, this, - timeoutSecs); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised for " << + timeoutSecs << " secs, will not fire"); } else { - TRACE6("%d.%d: timer %p initialised for %d secs, " - "to fire at %d.%d\n", tv.tv_sec, tv.tv_usec, this, - timeoutSecs, (int)(mExpires / 1000000), + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised for " << + timeoutSecs << " secs, to fire at " << + (int)(mExpires / 1000000) << "." << (int)(mExpires % 1000000)); } #endif @@ -291,8 +291,8 @@ #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); - TRACE3("%d.%d: timer %p destroyed, will not fire\n", - tv.tv_sec, tv.tv_usec, this); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " destroyed"); #endif Timers::Remove(*this); @@ -307,21 +307,22 @@ gettimeofday(&tv, NULL); if (mExpired) { - TRACE4("%d.%d: timer %p initialised from timer %p, " - "already expired, will not fire\n", tv.tv_sec, - tv.tv_usec, this, &rToCopy); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << ", already expired, will not fire"); } else if (mExpires == 0) { - TRACE4("%d.%d: timer %p initialised from timer %p, " - "will not fire\n", tv.tv_sec, tv.tv_usec, this, - &rToCopy); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << ", no expiry, will not fire"); } else { - TRACE6("%d.%d: timer %p initialised from timer %p, " - "to fire at %d.%d\n", tv.tv_sec, tv.tv_usec, this, - &rToCopy, (int)(mExpires / 1000000), + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << " to fire at " << + (int)(mExpires / 1000000) << "." << (int)(mExpires % 1000000)); } #endif @@ -339,21 +340,22 @@ gettimeofday(&tv, NULL); if (rToCopy.mExpired) { - TRACE4("%d.%d: timer %p initialised from timer %p, " - "already expired, will not fire\n", tv.tv_sec, - tv.tv_usec, this, &rToCopy); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << ", already expired, will not fire"); } else if (rToCopy.mExpires == 0) { - TRACE4("%d.%d: timer %p initialised from timer %p, " - "will not fire\n", tv.tv_sec, tv.tv_usec, this, - &rToCopy); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << ", no expiry, will not fire"); } else { - TRACE6("%d.%d: timer %p initialised from timer %p, " - "to fire at %d.%d\n", tv.tv_sec, tv.tv_usec, this, - &rToCopy, (int)(rToCopy.mExpires / 1000000), + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " initialised from timer " << + &rToCopy << " to fire at " << + (int)(rToCopy.mExpires / 1000000) << "." << (int)(rToCopy.mExpires % 1000000)); } #endif @@ -373,7 +375,8 @@ #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); - TRACE3("%d.%d: timer %p fired\n", tv.tv_sec, tv.tv_usec, this); + BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + ": timer " << this << " fired"); #endif mExpired = true; From boxbackup-dev at fluffy.co.uk Sat Mar 24 22:58:55 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 22:58:55 +0000 Subject: [Box Backup-commit] COMMIT r1473 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-24 22:58:55 +0000 (Sat, 24 Mar 2007) New Revision: 1473 Modified: box/chris/merge/lib/common/Test.h Log: Trivial code simplification. (refs #3, merges [1444]) Modified: box/chris/merge/lib/common/Test.h =================================================================== --- box/chris/merge/lib/common/Test.h 2007-03-24 22:58:16 UTC (rev 1472) +++ box/chris/merge/lib/common/Test.h 2007-03-24 22:58:55 UTC (rev 1473) @@ -275,11 +275,9 @@ TEST_FAIL_WITH_MESSAGE("Server didn't save PID file"); return -1; } - else - { - ::fprintf(stdout, "done.\n"); - } + ::fprintf(stdout, "done.\n"); + // wait a second for the pid to be written to the file ::sleep(1); From boxbackup-dev at fluffy.co.uk Sat Mar 24 22:59:53 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 22:59:53 +0000 Subject: [Box Backup-commit] COMMIT r1475 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-24 22:59:53 +0000 (Sat, 24 Mar 2007) New Revision: 1475 Modified: box/chris/merge/lib/common/Timer.h Log: Fix header include order. (refs #3, merges [1446]) Modified: box/chris/merge/lib/common/Timer.h =================================================================== --- box/chris/merge/lib/common/Timer.h 2007-03-24 22:59:26 UTC (rev 1474) +++ box/chris/merge/lib/common/Timer.h 2007-03-24 22:59:53 UTC (rev 1475) @@ -15,9 +15,10 @@ #include -#include "MemLeakFindOn.h" #include "BoxTime.h" +#include "MemLeakFindOn.h" + class Timer; // -------------------------------------------------------------------------- From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:00:41 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:00:41 +0000 Subject: [Box Backup-commit] COMMIT r1476 - box/chris/merge/bin/bbstored Message-ID: Author: chris Date: 2007-03-24 23:00:41 +0000 (Sat, 24 Mar 2007) New Revision: 1476 Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp Log: Remove newlines from syslog() messages. (refs #3, merges [1447]) Modified: box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp =================================================================== --- box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2007-03-24 22:59:53 UTC (rev 1475) +++ box/chris/merge/bin/bbstored/BackupStoreDaemon.cpp 2007-03-24 23:00:41 UTC (rev 1476) @@ -295,7 +295,7 @@ std::string clientCommonName(rStream.GetPeerCommonName()); // Log the name - ::syslog(LOG_INFO, "Certificate CN: %s\n", clientCommonName.c_str()); + ::syslog(LOG_INFO, "Certificate CN: %s", clientCommonName.c_str()); // Check it int32_t id; @@ -342,7 +342,7 @@ { // Log the amount of data transferred ::syslog(LOG_INFO, "Connection statistics for %s: " - "IN=%lld OUT=%lld TOTAL=%lld\n", commonName, + "IN=%lld OUT=%lld TOTAL=%lld", commonName, (long long)s.GetBytesRead(), (long long)s.GetBytesWritten(), (long long)s.GetBytesRead() + From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:01:37 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:01:37 +0000 Subject: [Box Backup-commit] COMMIT r1477 - box/chris/merge/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-24 23:01:37 +0000 (Sat, 24 Mar 2007) New Revision: 1477 Modified: box/chris/merge/bin/bbackupquery/bbackupquery.cpp Log: Initialise logging framework and set sensible default verbosity levels in bbackupquery (refs #3, merges [1449]) Modified: box/chris/merge/bin/bbackupquery/bbackupquery.cpp =================================================================== --- box/chris/merge/bin/bbackupquery/bbackupquery.cpp 2007-03-24 23:00:41 UTC (rev 1476) +++ box/chris/merge/bin/bbackupquery/bbackupquery.cpp 2007-03-24 23:01:37 UTC (rev 1477) @@ -45,6 +45,7 @@ #include "FdGetLine.h" #include "BackupClientCryptoKeys.h" #include "BannerText.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -95,11 +96,19 @@ bool quiet = false; bool readWrite = false; + Logging::SetProgramName("Box Backup (bbackupquery)"); + + #ifdef NDEBUG + int masterLevel = Log::NOTICE; // need an int to do math with + #else + int masterLevel = Log::INFO; // need an int to do math with + #endif + #ifdef WIN32 - const char* validOpts = "qwuc:l:"; + const char* validOpts = "qvwuc:l:"; bool unicodeConsole = false; #else - const char* validOpts = "qwc:l:"; + const char* validOpts = "qvwc:l:"; #endif // See if there's another entry on the command line @@ -108,11 +117,35 @@ { switch(c) { - case 'q': - // Quiet mode - quiet = true; + case 'q': + { + // Quiet mode + quiet = true; + + if(masterLevel == Log::NOTHING) + { + BOX_FATAL("Too many '-q': " + "Cannot reduce logging " + "level any more"); + return 2; + } + masterLevel--; + } break; - + + case 'v': + { + if(masterLevel == Log::EVERYTHING) + { + BOX_FATAL("Too many '-v': " + "Cannot increase logging " + "level any more"); + return 2; + } + masterLevel++; + } + break; + case 'w': // Read/write mode readWrite = true; @@ -147,6 +180,8 @@ argc -= optind; argv += optind; + Logging::SetGlobalLevel((Log::Level)masterLevel); + // Print banner? if(!quiet) { From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:02:36 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:02:36 +0000 Subject: [Box Backup-commit] COMMIT r1478 - box/chris/merge/bin/bbackupd Message-ID: Author: chris Date: 2007-03-24 23:02:36 +0000 (Sat, 24 Mar 2007) New Revision: 1478 Added: box/chris/merge/bin/bbackupd/ClientException.txt box/chris/merge/bin/bbackupd/Makefile.extra Log: Add new exception codes for use in bbackupd, and possibly other clients. (refs #3, merges [1455]) Copied: box/chris/merge/bin/bbackupd/ClientException.txt (from rev 1455, box/chris/general/bin/bbackupd/ClientException.txt) =================================================================== --- box/chris/merge/bin/bbackupd/ClientException.txt (rev 0) +++ box/chris/merge/bin/bbackupd/ClientException.txt 2007-03-24 23:02:36 UTC (rev 1478) @@ -0,0 +1,11 @@ + +# NOTE: Exception descriptions are for public distributions of Box Backup only -- do not rely for other applications. + + +EXCEPTION Client 13 + +Internal 0 +AssertFailed 1 +ClockWentBackwards 2 Invalid (negative) sync period: perhaps your clock is going backwards? +FailedToDeleteStoreObjectInfoFile 3 Failed to delete the StoreObjectInfoFile, backup cannot continue safely. +CorruptStoreObjectInfoFile 4 The store object info file contained an invalid value and is probably corrupt. Try deleting it. Copied: box/chris/merge/bin/bbackupd/Makefile.extra (from rev 1455, box/chris/general/bin/bbackupd/Makefile.extra) =================================================================== --- box/chris/merge/bin/bbackupd/Makefile.extra (rev 0) +++ box/chris/merge/bin/bbackupd/Makefile.extra 2007-03-24 23:02:36 UTC (rev 1478) @@ -0,0 +1,7 @@ + +MAKEEXCEPTION = ../../lib/common/makeexception.pl + +# AUTOGEN SEEDING +autogen_ClientException.h autogen_ClientException.cpp: $(MAKEEXCEPTION) ClientException.txt + $(PERL) $(MAKEEXCEPTION) ClientException.txt + From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:03:11 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:03:11 +0000 Subject: [Box Backup-commit] COMMIT r1479 - box/chris/merge/test/bbackupd Message-ID: Author: chris Date: 2007-03-24 23:03:11 +0000 (Sat, 24 Mar 2007) New Revision: 1479 Modified: box/chris/merge/test/bbackupd/Makefile.extra Log: Add autogen_ClientException.o to testbbackupd extra objects, because it links against bbackupd objects which need this. (refs #3, merges [1456]) Modified: box/chris/merge/test/bbackupd/Makefile.extra =================================================================== --- box/chris/merge/test/bbackupd/Makefile.extra 2007-03-24 23:02:36 UTC (rev 1478) +++ box/chris/merge/test/bbackupd/Makefile.extra 2007-03-24 23:03:11 UTC (rev 1479) @@ -1 +1 @@ -link-extra: ../../bin/bbackupd/BackupClientContext.o ../../bin/bbackupd/BackupClientDeleteList.o ../../bin/bbackupd/BackupClientDirectoryRecord.o ../../bin/bbackupd/Win32BackupService.o ../../bin/bbackupd/BackupClientInodeToIDMap.o ../../bin/bbackupd/Win32ServiceFunctions.o ../../bin/bbackupd/BackupDaemon.o +link-extra: ../../bin/bbackupd/autogen_ClientException.o ../../bin/bbackupd/BackupClientContext.o ../../bin/bbackupd/BackupClientDeleteList.o ../../bin/bbackupd/BackupClientDirectoryRecord.o ../../bin/bbackupd/Win32BackupService.o ../../bin/bbackupd/BackupClientInodeToIDMap.o ../../bin/bbackupd/Win32ServiceFunctions.o ../../bin/bbackupd/BackupDaemon.o From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:07:13 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:07:13 +0000 Subject: [Box Backup-commit] COMMIT r1480 - box/chris/merge/lib/server Message-ID: Author: chris Date: 2007-03-24 23:07:13 +0000 (Sat, 24 Mar 2007) New Revision: 1480 Modified: box/chris/merge/lib/server/WinNamedPipeStream.cpp Log: Improve logging of pipe errors by including the error message. (refs #3, merges part of [1458]) Modified: box/chris/merge/lib/server/WinNamedPipeStream.cpp =================================================================== --- box/chris/merge/lib/server/WinNamedPipeStream.cpp 2007-03-24 23:03:11 UTC (rev 1479) +++ box/chris/merge/lib/server/WinNamedPipeStream.cpp 2007-03-24 23:07:13 UTC (rev 1480) @@ -100,8 +100,8 @@ if (mSocketHandle == INVALID_HANDLE_VALUE) { - ::syslog(LOG_ERR, "CreateNamedPipeW failed: %d", - GetLastError()); + ::syslog(LOG_ERR, "CreateNamedPipeW failed: %s", + GetErrorMessage(GetLastError()).c_str()); THROW_EXCEPTION(ServerException, SocketOpenError) } @@ -109,8 +109,8 @@ if (!connected) { - ::syslog(LOG_ERR, "ConnectNamedPipe failed: %d", - GetLastError()); + ::syslog(LOG_ERR, "ConnectNamedPipe failed: %s", + GetErrorMessage(GetLastError()).c_str()); Close(); THROW_EXCEPTION(ServerException, SocketOpenError) } @@ -126,8 +126,8 @@ if (mReadableEvent == INVALID_HANDLE_VALUE) { - ::syslog(LOG_ERR, "Failed to create the Readable event: " - "error %d", GetLastError()); + ::syslog(LOG_ERR, "Failed to create the Readable event: %s", + GetErrorMessage(GetLastError()).c_str()); Close(); THROW_EXCEPTION(CommonException, Internal) } @@ -145,7 +145,7 @@ if (err != ERROR_IO_PENDING) { ::syslog(LOG_ERR, "Failed to start overlapped read: " - "error %d", err); + "%s", GetErrorMessage(err).c_str()); Close(); THROW_EXCEPTION(ConnectionException, Conn_SocketReadError) @@ -189,7 +189,7 @@ else { ::syslog(LOG_ERR, "Failed to connect to backup " - "daemon: error %d", err); + "daemon: %s", GetErrorMessage(err).c_str()); } THROW_EXCEPTION(ServerException, SocketOpenError) } @@ -269,7 +269,8 @@ ::syslog(LOG_ERR, "Failed to wait for " "ReadFile to complete: " - "error %d", err); + "%s", + GetErrorMessage(err).c_str()); } Close(); @@ -332,7 +333,8 @@ else { ::syslog(LOG_ERR, "Failed to start " - "overlapped read: error %d", err); + "overlapped read: %s", + GetErrorMessage(err).c_str()); Close(); THROW_EXCEPTION(ConnectionException, Conn_SocketReadError) @@ -420,6 +422,9 @@ if (!Success) { + DWORD err = GetLastError(); + ::syslog(LOG_ERR, "Failed to write to control socket: " + "%s", GetErrorMessage(err).c_str()); Close(); THROW_EXCEPTION(ConnectionException, Conn_SocketWriteError) @@ -456,7 +461,8 @@ if (!CancelIo(mSocketHandle)) { ::syslog(LOG_ERR, "Failed to cancel outstanding " - "I/O: error %d", GetLastError()); + "I/O: %s", + GetErrorMessage(GetLastError()).c_str()); } if (mReadableEvent == INVALID_HANDLE_VALUE) @@ -467,21 +473,27 @@ else if (!CloseHandle(mReadableEvent)) { ::syslog(LOG_ERR, "Failed to destroy Readable " - "event: error %d", GetLastError()); + "event: %s", + GetErrorMessage(GetLastError()).c_str()); } mReadableEvent = INVALID_HANDLE_VALUE; if (!FlushFileBuffers(mSocketHandle)) { - ::syslog(LOG_INFO, "FlushFileBuffers failed: %d", - GetLastError()); + ::syslog(LOG_INFO, "FlushFileBuffers failed: %s", + GetErrorMessage(GetLastError()).c_str()); } if (!DisconnectNamedPipe(mSocketHandle)) { - ::syslog(LOG_ERR, "DisconnectNamedPipe failed: %d", - GetLastError()); + DWORD err = GetLastError(); + if (err != ERROR_PIPE_NOT_CONNECTED) + { + ::syslog(LOG_ERR, "DisconnectNamedPipe " + "failed: %s", + GetErrorMessage(err).c_str()); + } } mIsServer = false; @@ -496,7 +508,8 @@ if (!result) { - ::syslog(LOG_ERR, "CloseHandle failed: %d", GetLastError()); + ::syslog(LOG_ERR, "CloseHandle failed: %s", + GetErrorMessage(GetLastError()).c_str()); THROW_EXCEPTION(ServerException, SocketCloseError) } } @@ -544,8 +557,8 @@ if (!FlushFileBuffers(mSocketHandle)) { - ::syslog(LOG_WARNING, "FlushFileBuffers failed: %d", - GetLastError()); + ::syslog(LOG_WARNING, "FlushFileBuffers failed: %s", + GetErrorMessage(GetLastError()).c_str()); } } From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:12:20 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:12:20 +0000 Subject: [Box Backup-commit] COMMIT r1481 - box/chris/merge/bin/bbackupd Message-ID: Author: chris Date: 2007-03-24 23:12:20 +0000 (Sat, 24 Mar 2007) New Revision: 1481 Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp Log: iUse our new ClientException codes for clock skew and archive problems. (refs #3, merges part of [1459]) Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2007-03-24 23:07:13 UTC (rev 1480) +++ box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2007-03-24 23:12:20 UTC (rev 1481) @@ -74,6 +74,7 @@ #include "Archive.h" #include "Timer.h" #include "Logging.h" +#include "autogen_ClientException.h" #include "MemLeakFindOn.h" @@ -642,7 +643,19 @@ // Calculate the sync period of files to examine box_time_t syncPeriodStart = lastSyncTime; - box_time_t syncPeriodEnd = currentSyncStartTime - minimumFileAge; + box_time_t syncPeriodEnd = currentSyncStartTime - + minimumFileAge; + + if(syncPeriodStart >= syncPeriodEnd) + { + BOX_ERROR("Invalid (negative) sync period: " + "perhaps your clock is going " + "backwards (" << syncPeriodStart << + " to " << syncPeriodEnd << ")"); + THROW_EXCEPTION(ClientException, + ClockWentBackwards); + } + // Check logic ASSERT(syncPeriodEnd > syncPeriodStart); // Paranoid check on sync times @@ -669,10 +682,8 @@ BOX_ERROR("Failed to delete the " "StoreObjectInfoFile, backup cannot " "continue safely."); - // prevent runaway process where the logs fill up -- without this - // the log message will be emitted in a tight loop. - ::sleep(60); - continue; + THROW_EXCEPTION(ClientException, + FailedToDeleteStoreObjectInfoFile); } // In case the backup throws an exception, @@ -836,6 +847,7 @@ BOX_ERROR("Internal error during " "backup run: " << e.what()); errorOccurred = true; + errorString = e.what(); } catch(...) { @@ -2215,7 +2227,7 @@ else { // there is something going on here - THROW_EXCEPTION(CommonException, Internal) + THROW_EXCEPTION(ClientException, CorruptStoreObjectInfoFile); } // @@ -2240,7 +2252,7 @@ else { // there is something going on here - THROW_EXCEPTION(CommonException, Internal) + THROW_EXCEPTION(ClientException, CorruptStoreObjectInfoFile); } // @@ -2265,7 +2277,7 @@ else { // there is something going on here - THROW_EXCEPTION(CommonException, Internal) + THROW_EXCEPTION(ClientException, CorruptStoreObjectInfoFile); } } From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:14:27 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:14:27 +0000 Subject: [Box Backup-commit] COMMIT r1482 - box/chris/merge/bin/bbackupd Message-ID: Author: chris Date: 2007-03-24 23:14:27 +0000 (Sat, 24 Mar 2007) New Revision: 1482 Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp Log: Better handle a case where a force-sync command comes in immediately after (or during) a sync, i.e. less than MinimumFileAge seconds after the last one. In this case, just move back the syncPeriodStart by 1 second. (refs #3, merges part if [1459]) Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2007-03-24 23:12:20 UTC (rev 1481) +++ box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2007-03-24 23:14:27 UTC (rev 1482) @@ -646,6 +646,18 @@ box_time_t syncPeriodEnd = currentSyncStartTime - minimumFileAge; + if(syncPeriodStart >= syncPeriodEnd && + syncPeriodStart - syncPeriodEnd < minimumFileAge) + { + // This can happen if we receive a force-sync + // command less than minimumFileAge after + // the last sync. Deal with it by moving back + // syncPeriodStart, which should not do any + // damage. + syncPeriodStart = syncPeriodEnd - + SecondsToBoxTime(1); + } + if(syncPeriodStart >= syncPeriodEnd) { BOX_ERROR("Invalid (negative) sync period: " From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:16:58 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:16:58 +0000 Subject: [Box Backup-commit] COMMIT r1483 - box/chris/merge/bin/bbackupd Message-ID: Author: chris Date: 2007-03-24 23:16:58 +0000 (Sat, 24 Mar 2007) New Revision: 1483 Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp Log: Reformat long lines for readability. (refs #3, merges remainder of [1459]) Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2007-03-24 23:14:27 UTC (rev 1482) +++ box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2007-03-24 23:16:58 UTC (rev 1483) @@ -579,38 +579,64 @@ box_time_t currentTime; do { - // Need to check the stop run thing here too, so this loop isn't run if we should be stopping + // Check whether we should be stopping, + // and don't run a sync if so. if(StopRun()) break; currentTime = GetCurrentBoxTime(); - // Pause a while, but no more than MAX_SLEEP_TIME seconds (use the conditional because times are unsigned) - box_time_t requiredDelay = (nextSyncTime < currentTime)?(0):(nextSyncTime - currentTime); - // If there isn't automatic backup happening, set a long delay. And limit delays at the same time. - if(!automaticBackup || requiredDelay > SecondsToBoxTime(MAX_SLEEP_TIME)) - requiredDelay = SecondsToBoxTime(MAX_SLEEP_TIME); + // Pause a while, but no more than + // MAX_SLEEP_TIME seconds (use the conditional + // because times are unsigned) + box_time_t requiredDelay = + (nextSyncTime < currentTime) + ? (0) + : (nextSyncTime - currentTime); - // Only do the delay if there is a delay required + // If there isn't automatic backup happening, + // set a long delay. And limit delays at the + // same time. + if(!automaticBackup || requiredDelay > + SecondsToBoxTime(MAX_SLEEP_TIME)) + { + requiredDelay = SecondsToBoxTime( + MAX_SLEEP_TIME); + } + + // Only delay if necessary if(requiredDelay > 0) { - // Sleep somehow. There are choices on how this should be done, depending on the state of the control connection + // Sleep somehow. There are choices + // on how this should be done, + // depending on the state of the + // control connection if(mpCommandSocketInfo != 0) { - // A command socket exists, so sleep by handling connections with it - WaitOnCommandSocket(requiredDelay, doSync, doSyncForcedByCommand); + // A command socket exists, + // so sleep by waiting on it + WaitOnCommandSocket( + requiredDelay, doSync, + doSyncForcedByCommand); } else { - // No command socket or connection, just do a normal sleep - time_t sleepSeconds = BoxTimeToSeconds(requiredDelay); - ::sleep((sleepSeconds <= 0)?1:sleepSeconds); + // No command socket or + // connection, just do a + // normal sleep + time_t sleepSeconds = + BoxTimeToSeconds( + requiredDelay); + ::sleep((sleepSeconds <= 0) + ? 1 + : sleepSeconds); } } } while((!automaticBackup || (currentTime < nextSyncTime)) && !doSync && !StopRun()); } - // Time of sync start, and if it's time for another sync (and we're doing automatic syncs), set the flag + // Time of sync start, and if it's time for another sync + // (and we're doing automatic syncs), set the flag box_time_t currentSyncStartTime = GetCurrentBoxTime(); if(automaticBackup && currentSyncStartTime >= nextSyncTime) { @@ -624,12 +650,14 @@ if(d > 0) { // Script has asked for a delay - nextSyncTime = GetCurrentBoxTime() + SecondsToBoxTime(d); + nextSyncTime = GetCurrentBoxTime() + + SecondsToBoxTime(d); doSync = false; } } - // Ready to sync? (but only if we're not supposed to be stopping) + // Ready to sync? (but only if we're not supposed + // to be stopping) if(doSync && !StopRun()) { // Touch a file to record times in filesystem @@ -673,16 +701,21 @@ // Paranoid check on sync times if(syncPeriodStart >= syncPeriodEnd) continue; - // Adjust syncPeriodEnd to emulate snapshot behaviour properly + // Adjust syncPeriodEnd to emulate snapshot + // behaviour properly box_time_t syncPeriodEndExtended = syncPeriodEnd; // Using zero min file age? if(minimumFileAge == 0) { - // Add a year on to the end of the end time, to make sure we sync - // files which are modified after the scan run started. - // Of course, they may be eligable to be synced again the next time round, - // but this should be OK, because the changes only upload should upload no data. - syncPeriodEndExtended += SecondsToBoxTime((time_t)(356*24*3600)); + // Add a year on to the end of the end time, + // to make sure we sync files which are + // modified after the scan run started. + // Of course, they may be eligible to be + // synced again the next time round, + // but this should be OK, because the changes + // only upload should upload no data. + syncPeriodEndExtended += SecondsToBoxTime( + (time_t)(356*24*3600)); } // Delete the serialised store object file, @@ -742,13 +775,22 @@ ); // Set up the sync parameters - BackupClientDirectoryRecord::SyncParams params(*this, *this, clientContext); + BackupClientDirectoryRecord::SyncParams params( + *this, *this, clientContext); params.mSyncPeriodStart = syncPeriodStart; - params.mSyncPeriodEnd = syncPeriodEndExtended; // use potentially extended end time + params.mSyncPeriodEnd = syncPeriodEndExtended; + // use potentially extended end time params.mMaxUploadWait = maxUploadWait; - params.mFileTrackingSizeThreshold = conf.GetKeyValueInt("FileTrackingSizeThreshold"); - params.mDiffingUploadSizeThreshold = conf.GetKeyValueInt("DiffingUploadSizeThreshold"); - params.mMaxFileTimeInFuture = SecondsToBoxTime(conf.GetKeyValueInt("MaxFileTimeInFuture")); + params.mFileTrackingSizeThreshold = + conf.GetKeyValueInt( + "FileTrackingSizeThreshold"); + params.mDiffingUploadSizeThreshold = + conf.GetKeyValueInt( + "DiffingUploadSizeThreshold"); + params.mMaxFileTimeInFuture = + SecondsToBoxTime( + conf.GetKeyValueInt( + "MaxFileTimeInFuture")); clientContext.SetMaximumDiffingTime(maximumDiffingTime); clientContext.SetKeepAliveTime(keepAliveTime); @@ -756,12 +798,17 @@ // Set store marker clientContext.SetClientStoreMarker(clientStoreMarker); - // Set up the locations, if necessary -- need to do it here so we have a (potential) connection to use + // Set up the locations, if necessary -- + // need to do it here so we have a + // (potential) connection to use if(mLocations.empty()) { - const Configuration &locations(conf.GetSubConfiguration("BackupLocations")); + const Configuration &locations( + conf.GetSubConfiguration( + "BackupLocations")); - // Make sure all the directory records are set up + // Make sure all the directory records + // are set up SetupLocations(clientContext, locations); } @@ -772,16 +819,25 @@ DeleteUnusedRootDirEntries(clientContext); // Go through the records, syncing them - for(std::vector::const_iterator i(mLocations.begin()); i != mLocations.end(); ++i) + for(std::vector::const_iterator + i(mLocations.begin()); + i != mLocations.end(); ++i) { - // Set current and new ID map pointers in the context + // Set current and new ID map pointers + // in the context clientContext.SetIDMaps(mCurrentIDMaps[(*i)->mIDMapIndex], mNewIDMaps[(*i)->mIDMapIndex]); - // Set exclude lists (context doesn't take ownership) - clientContext.SetExcludeLists((*i)->mpExcludeFiles, (*i)->mpExcludeDirs); + // Set exclude lists (context doesn't + // take ownership) + clientContext.SetExcludeLists( + (*i)->mpExcludeFiles, + (*i)->mpExcludeDirs); // Sync the directory - (*i)->mpDirectoryRecord->SyncDirectory(params, BackupProtocolClientListDirectory::RootDirectory, (*i)->mPath); + (*i)->mpDirectoryRecord->SyncDirectory( + params, + BackupProtocolClientListDirectory::RootDirectory, + (*i)->mPath); // Unset exclude lists (just in case) clientContext.SetExcludeLists(0, 0); @@ -795,13 +851,14 @@ } else { - // Unset the read error flag, so the error is - // reported again in the future + // Unset the read error flag, so the // error is reported again if it + // happens again mNotificationsSent[NotifyEvent_ReadError] = false; } - // Perform any deletions required -- these are delayed until the end - // to allow renaming to happen neatly. + // Perform any deletions required -- these are + // delayed until the end to allow renaming to + // happen neatly. clientContext.PerformDeletions(); // Close any open connection @@ -818,15 +875,25 @@ } else { - // The start time of the next run is the end time of this run - // This is only done if the storage limit wasn't exceeded (as things won't have been done properly if it was) + // The start time of the next run is + // the end time of this run. + // This is only done if the storage + // limit wasn't exceeded (as things + // won't have been done properly if + // it was) lastSyncTime = syncPeriodEnd; - // unflag the storage full notify flag so that next time the store is full, and alert will be sent + + // unflag the storage full notify flag + // so that next time the store is full, + // an alert will be sent mNotificationsSent[NotifyEvent_StoreFull] = false; } // Calculate when the next sync run should be - nextSyncTime = currentSyncStartTime + updateStoreInterval + Random::RandomInt(updateStoreInterval >> SYNC_PERIOD_RANDOM_EXTRA_TIME_SHIFT_BY); + nextSyncTime = currentSyncStartTime + + updateStoreInterval + + Random::RandomInt(updateStoreInterval >> + SYNC_PERIOD_RANDOM_EXTRA_TIME_SHIFT_BY); // Commit the ID Maps CommitIDMapsAfterSync(); @@ -863,15 +930,22 @@ } catch(...) { - // TODO: better handling of exceptions here... need to be very careful + // TODO: better handling of exceptions here... + // need to be very careful errorOccurred = true; } if(errorOccurred) { // Is it a berkely db failure? - bool isBerkelyDbFailure = (errorCode == BackupStoreException::ExceptionType - && errorSubCode == BackupStoreException::BerkelyDBFailure); + bool isBerkelyDbFailure = false; + + if (errorCode == BackupStoreException::ExceptionType + && errorSubCode == BackupStoreException::BerkelyDBFailure) + { + isBerkelyDbFailure = true; + } + if(isBerkelyDbFailure) { // Delete corrupt files @@ -879,7 +953,8 @@ } // Clear state data - syncPeriodStart = 0; // go back to beginning of time + syncPeriodStart = 0; + // go back to beginning of time clientStoreMarker = BackupClientContext::ClientStoreMarker_NotKnown; // no store marker, so download everything DeleteAllLocations(); DeleteAllIDMaps(); From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:35:33 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:35:33 +0000 Subject: [Box Backup-commit] COMMIT r1484 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-24 23:35:33 +0000 (Sat, 24 Mar 2007) New Revision: 1484 Modified: box/chris/merge/lib/common/Timer.cpp Log: Compile fix to [1448] (refs #3) Modified: box/chris/merge/lib/common/Timer.cpp =================================================================== --- box/chris/merge/lib/common/Timer.cpp 2007-03-24 23:16:58 UTC (rev 1483) +++ box/chris/merge/lib/common/Timer.cpp 2007-03-24 23:35:33 UTC (rev 1484) @@ -262,13 +262,13 @@ gettimeofday(&tv, NULL); if (timeoutSecs == 0) { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised for " << timeoutSecs << " secs, will not fire"); } else { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised for " << timeoutSecs << " secs, to fire at " << (int)(mExpires / 1000000) << "." << @@ -291,7 +291,7 @@ #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " destroyed"); #endif @@ -307,19 +307,19 @@ gettimeofday(&tv, NULL); if (mExpired) { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << ", already expired, will not fire"); } else if (mExpires == 0) { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << ", no expiry, will not fire"); } else { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << " to fire at " << (int)(mExpires / 1000000) << "." << @@ -340,19 +340,19 @@ gettimeofday(&tv, NULL); if (rToCopy.mExpired) { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << ", already expired, will not fire"); } else if (rToCopy.mExpires == 0) { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << ", no expiry, will not fire"); } else { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << " to fire at " << (int)(rToCopy.mExpires / 1000000) << "." << @@ -375,7 +375,7 @@ #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " fired"); #endif From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:37:26 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:37:26 +0000 Subject: [Box Backup-commit] COMMIT r1485 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-24 23:37:26 +0000 (Sat, 24 Mar 2007) New Revision: 1485 Modified: box/chris/general/lib/common/Timer.cpp Log: Compile fix to [1448] (merges back [1484]) Modified: box/chris/general/lib/common/Timer.cpp =================================================================== --- box/chris/general/lib/common/Timer.cpp 2007-03-24 23:35:33 UTC (rev 1484) +++ box/chris/general/lib/common/Timer.cpp 2007-03-24 23:37:26 UTC (rev 1485) @@ -262,13 +262,13 @@ gettimeofday(&tv, NULL); if (timeoutSecs == 0) { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised for " << timeoutSecs << " secs, will not fire"); } else { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised for " << timeoutSecs << " secs, to fire at " << (int)(mExpires / 1000000) << "." << @@ -291,7 +291,7 @@ #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " destroyed"); #endif @@ -307,19 +307,19 @@ gettimeofday(&tv, NULL); if (mExpired) { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << ", already expired, will not fire"); } else if (mExpires == 0) { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << ", no expiry, will not fire"); } else { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << " to fire at " << (int)(mExpires / 1000000) << "." << @@ -340,19 +340,19 @@ gettimeofday(&tv, NULL); if (rToCopy.mExpired) { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << ", already expired, will not fire"); } else if (rToCopy.mExpires == 0) { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << ", no expiry, will not fire"); } else { - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " initialised from timer " << &rToCopy << " to fire at " << (int)(rToCopy.mExpires / 1000000) << "." << @@ -375,7 +375,7 @@ #if !defined NDEBUG && !defined WIN32 struct timeval tv; gettimeofday(&tv, NULL); - BOX_TRACE(tv.tv_secs << "." << tv.tv_usecs << + BOX_TRACE(tv.tv_sec << "." << tv.tv_usec << ": timer " << this << " fired"); #endif From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:40:19 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:40:19 +0000 Subject: [Box Backup-commit] COMMIT r1486 - box/chris/merge/bin/bbackupd Message-ID: Author: chris Date: 2007-03-24 23:40:19 +0000 (Sat, 24 Mar 2007) New Revision: 1486 Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp box/chris/merge/bin/bbackupd/BackupDaemon.h Log: Move all command socket communications to the worker thread, to avoid deadlocks. Use events, and a message list protected by a critical section, to pass messages between threads. (refs #3) Modified: box/chris/merge/bin/bbackupd/BackupDaemon.cpp =================================================================== --- box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2007-03-24 23:37:26 UTC (rev 1485) +++ box/chris/merge/bin/bbackupd/BackupDaemon.cpp 2007-03-24 23:40:19 UTC (rev 1486) @@ -127,6 +127,29 @@ } #ifdef WIN32 + // Create the event object to signal from main thread to worker + // when new messages are queued to be sent to the command socket. + mhMessageToSendEvent = CreateEvent(NULL, TRUE, FALSE, NULL); + if(mhMessageToSendEvent == INVALID_HANDLE_VALUE) + { + BOX_ERROR("Failed to create event object: error " << + GetLastError()); + exit(1); + } + + // Create the event object to signal from worker to main thread + // when a command has been received on the command socket. + mhCommandReceivedEvent = CreateEvent(NULL, TRUE, FALSE, NULL); + if(mhCommandReceivedEvent == INVALID_HANDLE_VALUE) + { + BOX_ERROR("Failed to create event object: error " << + GetLastError()); + exit(1); + } + + // Create the critical section to protect the message queue + InitializeCriticalSection(&mMessageQueueLock); + // Create a thread to handle the named pipe HANDLE hThread; unsigned int dwThreadId; @@ -264,7 +287,6 @@ #ifdef WIN32 void BackupDaemon::RunHelperThread(void) { - this->mReceivedCommandConn = false; mpCommandSocketInfo = new CommandSocketInfo; WinNamedPipeStream& rSocket(mpCommandSocketInfo->mListeningSocket); @@ -322,16 +344,74 @@ rSocket.Write(summary, summarySize); rSocket.Write("ping\n", 5); + // old queued messages are not useful + EnterCriticalSection(&mMessageQueueLock); + mMessageList.clear(); + ResetEvent(mhMessageToSendEvent); + LeaveCriticalSection(&mMessageQueueLock); + IOStreamGetLine readLine(rSocket); std::string command; - while (rSocket.IsConnected() && - readLine.GetLine(command) && - !IsTerminateWanted()) + while (rSocket.IsConnected() && !IsTerminateWanted()) { + HANDLE handles[2]; + handles[0] = mhMessageToSendEvent; + handles[1] = rSocket.GetReadableEvent(); + BOX_TRACE("Received command '" << command << "' over command socket"); + DWORD result = WaitForMultipleObjects( + sizeof(handles)/sizeof(*handles), + handles, FALSE, 1000); + + if(result == 0) + { + ResetEvent(mhMessageToSendEvent); + + EnterCriticalSection(&mMessageQueueLock); + try + { + while (mMessageList.size() > 0) + { + std::string message = *(mMessageList.begin()); + mMessageList.erase(mMessageList.begin()); + printf("Sending '%s' to waiting client... ", message.c_str()); + message += "\n"; + rSocket.Write(message.c_str(), + message.length()); + + printf("done.\n"); + } + } + catch (...) + { + LeaveCriticalSection(&mMessageQueueLock); + throw; + } + LeaveCriticalSection(&mMessageQueueLock); + continue; + } + else if(result == WAIT_TIMEOUT) + { + continue; + } + else if(result != 1) + { + BOX_ERROR("WaitForMultipleObjects returned invalid result " << result); + continue; + } + + if(!readLine.GetLine(command)) + { + BOX_ERROR("Failed to read line"); + continue; + } + + BOX_INFO("Received command " << command << + " from client"); + bool sendOK = false; bool sendResponse = true; bool disconnect = false; @@ -349,6 +429,7 @@ this->mDoSyncFlagOut = true; this->mSyncIsForcedOut = false; sendOK = true; + SetEvent(mhCommandReceivedEvent); } else if(command == "force-sync") { @@ -356,18 +437,21 @@ this->mDoSyncFlagOut = true; this->mSyncIsForcedOut = true; sendOK = true; + SetEvent(mhCommandReceivedEvent); } else if(command == "reload") { // Reload the configuration SetReloadConfigWanted(); sendOK = true; + SetEvent(mhCommandReceivedEvent); } else if(command == "terminate") { // Terminate the daemon cleanly SetTerminateWanted(); sendOK = true; + SetEvent(mhCommandReceivedEvent); } else { @@ -390,8 +474,6 @@ { break; } - - this->mReceivedCommandConn = true; } rSocket.Close(); @@ -411,6 +493,9 @@ BOX_ERROR("Communication error with control client"); } } + + CloseHandle(mhCommandReceivedEvent); + CloseHandle(mhMessageToSendEvent); } #endif @@ -1123,25 +1208,27 @@ void BackupDaemon::WaitOnCommandSocket(box_time_t RequiredDelay, bool &DoSyncFlagOut, bool &SyncIsForcedOut) { #ifdef WIN32 - // Really could use some interprocess protection, mutex etc - // any side effect should be too bad???? :) - DWORD timeout = (DWORD)BoxTimeToMilliSeconds(RequiredDelay); + DWORD requiredDelayMs = BoxTimeToMilliSeconds(RequiredDelay); - while ( this->mReceivedCommandConn == false ) + DWORD result = WaitForSingleObject(mhCommandReceivedEvent, + (DWORD)requiredDelayMs); + + if(result == WAIT_OBJECT_0) { - Sleep(1); - - if( timeout == 0 ) - { - DoSyncFlagOut = false; - SyncIsForcedOut = false; - return; - } - timeout--; + DoSyncFlagOut = this->mDoSyncFlagOut; + SyncIsForcedOut = this->mSyncIsForcedOut; + ResetEvent(mhCommandReceivedEvent); } - this->mReceivedCommandConn = false; - DoSyncFlagOut = this->mDoSyncFlagOut; - SyncIsForcedOut = this->mSyncIsForcedOut; + else if(result == WAIT_TIMEOUT) + { + DoSyncFlagOut = false; + SyncIsForcedOut = false; + } + else + { + BOX_ERROR("Unexpected result from WaitForSingleObject: " + "error " << GetLastError()); + } return; #else // ! WIN32 @@ -1383,10 +1470,6 @@ // The bbackupctl program can't rely on a state change, because it // may never change if the server doesn't need to be contacted. -#ifdef __MINGW32__ -#warning race condition: what happens if socket is closed? -#endif - if(mpCommandSocketInfo != NULL && #ifdef WIN32 mpCommandSocketInfo->mListeningSocket.IsConnected() @@ -1395,15 +1478,18 @@ #endif ) { - const char* message = SendStart ? "start-sync\n" : "finish-sync\n"; + std::string message = SendStart ? "start-sync" : "finish-sync"; try { #ifdef WIN32 - mpCommandSocketInfo->mListeningSocket.Write(message, - (int)strlen(message)); + EnterCriticalSection(&mMessageQueueLock); + mMessageList.push_back(message); + SetEvent(mhMessageToSendEvent); + LeaveCriticalSection(&mMessageQueueLock); #else - mpCommandSocketInfo->mpConnectedSocket->Write(message, - strlen(message)); + message += "\n"; + mpCommandSocketInfo->mpConnectedSocket->Write( + message.c_str(), message.size()); #endif } catch(std::exception &e) @@ -2033,52 +2119,45 @@ // command socket if there's an error char newState[64]; - char newStateSize = sprintf(newState, "state %d\n", State); + sprintf(newState, "state %d", State); + std::string message = newState; #ifdef WIN32 - #ifndef _MSC_VER - #warning FIX ME: race condition - #endif + EnterCriticalSection(&mMessageQueueLock); + mMessageList.push_back(newState); + SetEvent(mhMessageToSendEvent); + LeaveCriticalSection(&mMessageQueueLock); +#else + message += "\n"; - // what happens if the socket is closed by the other thread before - // we can write to it? Null pointer deref at best. - if(mpCommandSocketInfo && - mpCommandSocketInfo->mListeningSocket.IsConnected()) + if(mpCommandSocketInfo == 0) { - try - { - mpCommandSocketInfo->mListeningSocket.Write(newState, newStateSize); - } - catch(std::exception &e) - { - BOX_ERROR("Internal error while writing state " - "to command socket: " << e.what()); - CloseCommandConnection(); - } - catch(...) - { - CloseCommandConnection(); - } + return; } -#else - if(mpCommandSocketInfo != 0 && mpCommandSocketInfo->mpConnectedSocket.get() != 0) + + if(mpCommandSocketInfo->mpConnectedSocket.get() == 0) { - // Something connected to the command socket, tell it about the new state - try - { - mpCommandSocketInfo->mpConnectedSocket->Write(newState, newStateSize); - } - catch(std::exception &e) - { - BOX_ERROR("Internal error while writing state " - "to command socket: " << e.what()); - CloseCommandConnection(); - } - catch(...) - { - CloseCommandConnection(); - } + return; } + + // Something connected to the command socket, tell it about the new state + try + { + mpCommandSocketInfo->mpConnectedSocket->Write(message.c_str(), + message.length()); + } + catch(std::exception &e) + { + BOX_ERROR("Internal error while writing state " + "to command socket: " << e.what()); + CloseCommandConnection(); + } + catch(...) + { + BOX_ERROR("Internal error while writing state " + "to command socket: unknown error"); + CloseCommandConnection(); + } #endif } Modified: box/chris/merge/bin/bbackupd/BackupDaemon.h =================================================================== --- box/chris/merge/bin/bbackupd/BackupDaemon.h 2007-03-24 23:37:26 UTC (rev 1485) +++ box/chris/merge/bin/bbackupd/BackupDaemon.h 2007-03-24 23:40:19 UTC (rev 1486) @@ -346,7 +346,10 @@ void RunHelperThread(void); private: - bool mDoSyncFlagOut, mSyncIsForcedOut, mReceivedCommandConn; + bool mDoSyncFlagOut, mSyncIsForcedOut; + HANDLE mhMessageToSendEvent, mhCommandReceivedEvent; + CRITICAL_SECTION mMessageQueueLock; + std::vector mMessageList; #endif }; From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:41:06 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:41:06 +0000 Subject: [Box Backup-commit] COMMIT r1487 - box/chris/merge/bin/bbackupquery Message-ID: Author: chris Date: 2007-03-24 23:41:06 +0000 (Sat, 24 Mar 2007) New Revision: 1487 Modified: box/chris/merge/bin/bbackupquery/BackupQueries.cpp box/chris/merge/bin/bbackupquery/BackupQueries.h Log: Add a quiet mode to compare command, to make test output easier to read. (refs #3, merges [1460]) Modified: box/chris/merge/bin/bbackupquery/BackupQueries.cpp =================================================================== --- box/chris/merge/bin/bbackupquery/BackupQueries.cpp 2007-03-24 23:40:19 UTC (rev 1486) +++ box/chris/merge/bin/bbackupquery/BackupQueries.cpp 2007-03-24 23:41:06 UTC (rev 1487) @@ -207,7 +207,7 @@ { "sh", "" }, { "getobject", "" }, { "get", "i" }, - { "compare", "alcqAE" }, + { "compare", "alcqAEQ" }, { "restore", "dri" }, { "help", "" }, { "usage", "" }, @@ -1114,6 +1114,7 @@ // Parameters, including count of differences BackupQueries::CompareParams params; params.mQuickCompare = opts['q']; + params.mQuietCompare = opts['Q']; params.mIgnoreExcludes = opts['E']; params.mIgnoreAttributes = opts['A']; @@ -1177,12 +1178,17 @@ printf("Incorrect usage.\ncompare -a\n or compare -l \n or compare \n"); return; } - - printf("\n[ %d (of %d) differences probably due to file " - "modifications after the last upload ]\n" - "Differences: %d (%d dirs excluded, %d files excluded, " + + if (!params.mQuietCompare) + { + printf("\n[ %d (of %d) differences probably due to file " + "modifications after the last upload ]\n", + params.mDifferencesExplainedByModTime, + params.mDifferences); + } + + printf("Differences: %d (%d dirs excluded, %d files excluded, " "%d files not checked)\n", - params.mDifferencesExplainedByModTime, params.mDifferences, params.mDifferences, params.mExcludedDirs, params.mExcludedFiles, params.mUncheckedFiles); Modified: box/chris/merge/bin/bbackupquery/BackupQueries.h =================================================================== --- box/chris/merge/bin/bbackupquery/BackupQueries.h 2007-03-24 23:40:19 UTC (rev 1486) +++ box/chris/merge/bin/bbackupquery/BackupQueries.h 2007-03-24 23:41:06 UTC (rev 1487) @@ -67,6 +67,7 @@ ~CompareParams(); void DeleteExcludeLists(); bool mQuickCompare; + bool mQuietCompare; bool mIgnoreExcludes; bool mIgnoreAttributes; int mDifferences; From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:41:48 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:41:48 +0000 Subject: [Box Backup-commit] COMMIT r1488 - box/chris/merge/bin/bbstored Message-ID: Author: chris Date: 2007-03-24 23:41:48 +0000 (Sat, 24 Mar 2007) New Revision: 1488 Modified: box/chris/merge/bin/bbstored/bbstored.cpp Log: Initialise logging framework and set our program name to "Box Backup (bbstored)". (refs #3, merges [1462]) Modified: box/chris/merge/bin/bbstored/bbstored.cpp =================================================================== --- box/chris/merge/bin/bbstored/bbstored.cpp 2007-03-24 23:41:06 UTC (rev 1487) +++ box/chris/merge/bin/bbstored/bbstored.cpp 2007-03-24 23:41:48 UTC (rev 1488) @@ -10,6 +10,7 @@ #include "Box.h" #include "BackupStoreDaemon.h" #include "MainHelper.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -17,6 +18,10 @@ { MAINHELPER_START + Logging::SetProgramName("Box Backup (bbstored)"); + Logging::ToConsole(true); + Logging::ToSyslog (true); + BackupStoreDaemon daemon; return daemon.Main(BOX_FILE_BBSTORED_DEFAULT_CONFIG, argc, argv); From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:45:04 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:45:04 +0000 Subject: [Box Backup-commit] COMMIT r1489 - box/chris/merge/bin/bbackupd Message-ID: Author: chris Date: 2007-03-24 23:45:04 +0000 (Sat, 24 Mar 2007) New Revision: 1489 Modified: box/chris/merge/bin/bbackupd/BackupClientContext.cpp Log: Use logging framework to remove timer noise for those who don't want it. (refs #3, merges [1448]) Modified: box/chris/merge/bin/bbackupd/BackupClientContext.cpp =================================================================== --- box/chris/merge/bin/bbackupd/BackupClientContext.cpp 2007-03-24 23:41:48 UTC (rev 1488) +++ box/chris/merge/bin/bbackupd/BackupClientContext.cpp 2007-03-24 23:45:04 UTC (rev 1489) @@ -29,6 +29,7 @@ #include "BackupDaemon.h" #include "autogen_BackupProtocolClient.h" #include "BackupStoreFile.h" +#include "Logging.h" #include "MemLeakFindOn.h" @@ -499,13 +500,14 @@ void BackupClientContext::SetMaximumDiffingTime(int iSeconds) { mMaximumDiffingTime = iSeconds < 0 ? 0 : iSeconds; - TRACE1("Set maximum diffing time to %d seconds\n", mMaximumDiffingTime); + BOX_TRACE("Set maximum diffing time to " << mMaximumDiffingTime << + " seconds"); } void BackupClientContext::SetKeepAliveTime(int iSeconds) { mKeepAliveTime = iSeconds < 0 ? 0 : iSeconds; - TRACE1("Set keep-alive time to %d seconds\n", mKeepAliveTime); + BOX_TRACE("Set keep-alive time to " << mKeepAliveTime << " seconds"); mKeepAliveTimer = Timer(mKeepAliveTime); } @@ -563,7 +565,7 @@ return; } - TRACE0("KeepAliveTime reached, sending keep-alive message\n"); + BOX_TRACE("KeepAliveTime reached, sending keep-alive message"); mpConnection->QueryGetIsAlive(); mKeepAliveTimer = Timer(mKeepAliveTime); From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:48:17 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:48:17 +0000 Subject: [Box Backup-commit] COMMIT r1490 - box/chris/merge/infrastructure/m4 Message-ID: Author: chris Date: 2007-03-24 23:48:17 +0000 (Sat, 24 Mar 2007) New Revision: 1490 Added: box/chris/merge/infrastructure/m4/ax_check_llong_minmax.m4 Log: Merge new m4 from trunk. (refs #3, merges [1453]) Copied: box/chris/merge/infrastructure/m4/ax_check_llong_minmax.m4 (from rev 1453, box/chris/general/infrastructure/m4/ax_check_llong_minmax.m4) =================================================================== --- box/chris/merge/infrastructure/m4/ax_check_llong_minmax.m4 (rev 0) +++ box/chris/merge/infrastructure/m4/ax_check_llong_minmax.m4 2007-03-24 23:48:17 UTC (rev 1490) @@ -0,0 +1,76 @@ +dnl @synopsis AX_CHECK_LLONG_MINMAX +dnl +dnl This macro will fix up LLONG_MIN and LLONG_MAX as appropriate. I'm finding +dnl it quite difficult to believe that so many hoops are necessary. The world +dnl seems to have gone quite mad. +dnl +dnl This gem is adapted from the OpenSSH configure script so here's +dnl the original copyright notice: +dnl +dnl Copyright (c) 1999-2004 Damien Miller +dnl +dnl Permission to use, copy, modify, and distribute this software for any +dnl purpose with or without fee is hereby granted, provided that the above +dnl copyright notice and this permission notice appear in all copies. +dnl +dnl @category C +dnl @author Martin Ebourne and Damien Miller +dnl @version 2005/07/07 + +AC_DEFUN([AX_CHECK_LLONG_MINMAX], [ + AC_CHECK_DECL([LLONG_MAX], [have_llong_max=1], , [[#include ]]) + if test -z "$have_llong_max"; then + AC_MSG_CHECKING([[for max value of long long]]) + AC_RUN_IFELSE([AC_LANG_SOURCE([[ + #include + /* Why is this so damn hard? */ + #undef __GNUC__ + #undef __USE_ISOC99 + #define __USE_ISOC99 + #include + #define DATA "conftest.llminmax" + int main(void) { + FILE *f; + long long i, llmin, llmax = 0; + + if((f = fopen(DATA,"w")) == NULL) + exit(1); + + #if defined(LLONG_MIN) && defined(LLONG_MAX) + fprintf(stderr, "Using system header for LLONG_MIN and LLONG_MAX\n"); + llmin = LLONG_MIN; + llmax = LLONG_MAX; + #else + fprintf(stderr, "Calculating LLONG_MIN and LLONG_MAX\n"); + /* This will work on one's complement and two's complement */ + for (i = 1; i > llmax; i <<= 1, i++) + llmax = i; + llmin = llmax + 1LL; /* wrap */ + #endif + + /* Sanity check */ + if (llmin + 1 < llmin || llmin - 1 < llmin || llmax + 1 > llmax || llmax - 1 > llmax) { + fprintf(f, "unknown unknown\n"); + exit(2); + } + + if (fprintf(f ,"%lld %lld", llmin, llmax) < 0) + exit(3); + + exit(0); + } + ]])], [ + read llong_min llong_max < conftest.llminmax + AC_MSG_RESULT([$llong_max]) + AC_DEFINE_UNQUOTED([LLONG_MAX], [${llong_max}LL], + [max value of long long calculated by configure]) + AC_MSG_CHECKING([[for min value of long long]]) + AC_MSG_RESULT([$llong_min]) + AC_DEFINE_UNQUOTED([LLONG_MIN], [${llong_min}LL], + [min value of long long calculated by configure]) + ], + [AC_MSG_RESULT(not found)], + [AC_MSG_WARN([[cross compiling: not checking]])] + ) + fi + ])dnl From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:50:26 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:50:26 +0000 Subject: [Box Backup-commit] COMMIT r1491 - box/chris/merge/lib/intercept Message-ID: Author: chris Date: 2007-03-24 23:50:26 +0000 (Sat, 24 Mar 2007) New Revision: 1491 Modified: box/chris/merge/lib/intercept/intercept.cpp Log: Indent cleanup (refs #3) Modified: box/chris/merge/lib/intercept/intercept.cpp =================================================================== --- box/chris/merge/lib/intercept/intercept.cpp 2007-03-24 23:48:17 UTC (rev 1490) +++ box/chris/merge/lib/intercept/intercept.cpp 2007-03-24 23:50:26 UTC (rev 1491) @@ -18,7 +18,7 @@ #include #ifdef HAVE_SYS_UIO_H -#include + #include #endif #include From boxbackup-dev at fluffy.co.uk Sat Mar 24 23:52:52 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sat, 24 Mar 2007 23:52:52 +0000 Subject: [Box Backup-commit] COMMIT r1492 - box/chris/merge/lib/win32 Message-ID: Author: chris Date: 2007-03-24 23:52:52 +0000 (Sat, 24 Mar 2007) New Revision: 1492 Added: box/chris/merge/lib/win32/MSG00001.bin Log: Added precompiled output from message compiler, for those who don't have it, e.g. using MinGW instead of MSVC. (refs #3, merges [526]) Copied: box/chris/merge/lib/win32/MSG00001.bin (from rev 1468, box/chris/general/lib/win32/MSG00001.bin) =================================================================== (Binary files differ) From boxbackup-dev at fluffy.co.uk Sun Mar 25 00:12:01 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 00:12:01 +0000 Subject: [Box Backup-commit] COMMIT r1493 - box/chris/merge/lib/common Message-ID: Author: chris Date: 2007-03-25 00:12:01 +0000 (Sun, 25 Mar 2007) New Revision: 1493 Modified: box/chris/merge/lib/common/Test.h Log: Fix unix path to bbstored (refs #3) Modified: box/chris/merge/lib/common/Test.h =================================================================== --- box/chris/merge/lib/common/Test.h 2007-03-24 23:52:52 UTC (rev 1492) +++ box/chris/merge/lib/common/Test.h 2007-03-25 00:12:01 UTC (rev 1493) @@ -358,7 +358,7 @@ #else #define BBACKUPCTL "../../bin/bbackupctl/bbackupctl" #define BBACKUPD "../../bin/bbackupd/bbackupd" -#define BBSTORED "../../bin/bbackupd/bbstored" +#define BBSTORED "../../bin/bbstored/bbstored" #define BBACKUPQUERY "../../bin/bbackupquery/bbackupquery" #define BBSTOREACCOUNTS "../../bin/bbstoreaccounts/bbstoreaccounts" #define TEST_RETURN(actual, expected) TEST_THAT(actual == expected*256); From boxbackup-dev at fluffy.co.uk Sun Mar 25 00:15:46 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 00:15:46 +0000 Subject: [Box Backup-commit] COMMIT r1494 - box/chris/merge/test/backupstore Message-ID: Author: chris Date: 2007-03-25 00:15:46 +0000 (Sun, 25 Mar 2007) New Revision: 1494 Modified: box/chris/merge/test/backupstore/testbackupstore.cpp Log: Use #defined paths for applications to fix cross unix/win32 compatibility. Fix memory leak. Reformat comments. (refs #3) Modified: box/chris/merge/test/backupstore/testbackupstore.cpp =================================================================== --- box/chris/merge/test/backupstore/testbackupstore.cpp 2007-03-25 00:12:01 UTC (rev 1493) +++ box/chris/merge/test/backupstore/testbackupstore.cpp 2007-03-25 00:15:46 UTC (rev 1494) @@ -1684,7 +1684,10 @@ "testfiles/clientTrustedCAs.pem"); // First, try logging in without an account having been created... just make sure login fails. - int pid = LaunchServer("../../bin/bbstored/bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); + + int pid = LaunchServer(BBSTORED " testfiles/bbstored.conf", + "testfiles/bbstored.pid"); + TEST_THAT(pid != -1 && pid != 0); if(pid > 0) { @@ -1713,12 +1716,17 @@ } // Create an account for the test client - TEST_THAT_ABORTONFAIL(::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf create 01234567 0 10000B 20000B") == 0); + TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS + " -c testfiles/bbstored.conf create 01234567 0 " + "10000B 20000B") == 0); + TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); + TEST_THAT(TestDirExists("testfiles/0_0/backup/01234567")); TEST_THAT(TestDirExists("testfiles/0_1/backup/01234567")); TEST_THAT(TestDirExists("testfiles/0_2/backup/01234567")); - TEST_THAT(TestGetFileSize("testfiles/accounts.txt") > 8); // make sure something is written to it + TEST_THAT(TestGetFileSize("testfiles/accounts.txt") > 8); + // make sure something is written to it TEST_THAT(ServerIsAlive(pid)); @@ -1742,13 +1750,18 @@ TestRemoteProcessMemLeaks("bbstored.memleaks"); #endif - // Set a new limit on the account -- leave the hard limit high to make sure the target for - // freeing space is the soft limit. - TEST_THAT_ABORTONFAIL(::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf setlimit 01234567 10B 20000B") == 0); + // Set a new limit on the account -- leave the hard limit + // high to make sure the target for freeing space is the + // soft limit. + TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS + " -c testfiles/bbstored.conf setlimit 01234567 " + "10B 20000B") == 0); TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); // Start things up - pid = LaunchServer("../../bin/bbstored/bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); + pid = LaunchServer(BBSTORED " testfiles/bbstored.conf", + "testfiles/bbstored.pid"); + ::sleep(1); TEST_THAT(ServerIsAlive(pid)); @@ -1764,8 +1777,9 @@ // Count the objects again recursive_count_objects_results after = {0,0,0}; - recursive_count_objects("localhost", BackupProtocolClientListDirectory::RootDirectory, after); -printf("after.objectsNotDel=%i, deleted=%i, old=%i\n",after.objectsNotDel, after.deleted, after.old); + recursive_count_objects("localhost", + BackupProtocolClientListDirectory::RootDirectory, + after); // If these tests fail then try increasing the timeout above TEST_THAT(after.objectsNotDel == before.objectsNotDel); @@ -1773,7 +1787,9 @@ TEST_THAT(after.old == 0); // Set a really small hard limit - TEST_THAT_ABORTONFAIL(::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf setlimit 01234567 10B 20B") == 0); + TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS + " -c testfiles/bbstored.conf setlimit 01234567 " + "10B 20B") == 0); TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); // Try to upload a file and create a directory, and check an error is generated @@ -1837,11 +1853,16 @@ printf("Starting server for connection from remote machines...\n"); // Create an account for the test client - TEST_THAT_ABORTONFAIL(::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf create 01234567 0 30000B 40000B") == 0); + TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS + " -c testfiles/bbstored.conf create 01234567 0 " + "30000B 40000B") == 0); TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); // First, try logging in without an account having been created... just make sure login fails. - int pid = LaunchServer("../../bin/bbstored/bbstored testfiles/bbstored_multi.conf", "testfiles/bbstored.pid"); + + int pid = LaunchServer(BBSTORED " testfiles/bbstored_multi.conf", + "testfiles/bbstored.pid"); + TEST_THAT(pid != -1 && pid != 0); if(pid > 0) { @@ -1900,6 +1921,7 @@ CloseHandle(h2); CloseHandle(h1); + delete [] wfile; h1 = openfile("foo", O_CREAT | O_RDWR, 0); TEST_THAT(h1 != INVALID_HANDLE_VALUE); From boxbackup-dev at fluffy.co.uk Sun Mar 25 00:16:56 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 00:16:56 +0000 Subject: [Box Backup-commit] COMMIT r1495 - box/chris/general/lib/common Message-ID: Author: chris Date: 2007-03-25 00:16:56 +0000 (Sun, 25 Mar 2007) New Revision: 1495 Modified: box/chris/general/lib/common/Test.h Log: Fix unix path to bbstored (merges back [1493]) Modified: box/chris/general/lib/common/Test.h =================================================================== --- box/chris/general/lib/common/Test.h 2007-03-25 00:15:46 UTC (rev 1494) +++ box/chris/general/lib/common/Test.h 2007-03-25 00:16:56 UTC (rev 1495) @@ -358,7 +358,7 @@ #else #define BBACKUPCTL "../../bin/bbackupctl/bbackupctl" #define BBACKUPD "../../bin/bbackupd/bbackupd" -#define BBSTORED "../../bin/bbackupd/bbstored" +#define BBSTORED "../../bin/bbstored/bbstored" #define BBACKUPQUERY "../../bin/bbackupquery/bbackupquery" #define BBSTOREACCOUNTS "../../bin/bbstoreaccounts/bbstoreaccounts" #define TEST_RETURN(actual, expected) TEST_THAT(actual == expected*256); From boxbackup-dev at fluffy.co.uk Sun Mar 25 00:24:00 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 00:24:00 +0000 Subject: [Box Backup-commit] COMMIT r1496 - box/chris/general/lib/backupstore Message-ID: Author: chris Date: 2007-03-25 00:23:59 +0000 (Sun, 25 Mar 2007) New Revision: 1496 Modified: box/chris/general/lib/backupstore/BackupStoreInfo.cpp Log: Allow '/' as a path separator on all platforms, even Win32 Modified: box/chris/general/lib/backupstore/BackupStoreInfo.cpp =================================================================== --- box/chris/general/lib/backupstore/BackupStoreInfo.cpp 2007-03-25 00:16:56 UTC (rev 1495) +++ box/chris/general/lib/backupstore/BackupStoreInfo.cpp 2007-03-25 00:23:59 UTC (rev 1496) @@ -124,7 +124,8 @@ }; // Generate the filename - ASSERT(rRootDir[rRootDir.size() - 1] == DIRECTORY_SEPARATOR_ASCHAR); + ASSERT(rRootDir[rRootDir.size() - 1] == '/' || + rRootDir[rRootDir.size() - 1] == DIRECTORY_SEPARATOR_ASCHAR); std::string fn(rRootDir + INFO_FILENAME); // Open the file for writing From boxbackup-dev at fluffy.co.uk Sun Mar 25 00:24:40 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 00:24:40 +0000 Subject: [Box Backup-commit] COMMIT r1497 - box/chris/general/test/backupstore Message-ID: Author: chris Date: 2007-03-25 00:24:40 +0000 (Sun, 25 Mar 2007) New Revision: 1497 Modified: box/chris/general/test/backupstore/testbackupstore.cpp Log: Revert to unix path separators to reduce diff to chris/merge Modified: box/chris/general/test/backupstore/testbackupstore.cpp =================================================================== --- box/chris/general/test/backupstore/testbackupstore.cpp 2007-03-25 00:23:59 UTC (rev 1496) +++ box/chris/general/test/backupstore/testbackupstore.cpp 2007-03-25 00:24:40 UTC (rev 1497) @@ -973,6 +973,7 @@ #endif test_server_1(protocol, protocolReadOnly); + // Create and upload some test files int64_t maxID = 0; for(int t = 0; t < UPLOAD_NUM; ++t) @@ -1530,48 +1531,35 @@ // The test block to a file { - FileStream f("testfiles" DIRECTORY_SEPARATOR - "testenc1", O_WRONLY | O_CREAT | O_EXCL); + FileStream f("testfiles/testenc1", O_WRONLY | O_CREAT | O_EXCL); f.Write(encfile, sizeof(encfile)); } // Encode it { - FileStream out("testfiles" DIRECTORY_SEPARATOR - "testenc1_enc", O_WRONLY | O_CREAT | O_EXCL); - BackupStoreFilenameClear name("testfiles" - DIRECTORY_SEPARATOR "testenc1"); + FileStream out("testfiles/testenc1_enc", O_WRONLY | O_CREAT | O_EXCL); + BackupStoreFilenameClear name("testfiles/testenc1"); - std::auto_ptr encoded( - BackupStoreFile::EncodeFile( - "testfiles" DIRECTORY_SEPARATOR - "testenc1", 32, name)); + std::auto_ptr encoded(BackupStoreFile::EncodeFile("testfiles/testenc1", 32, name)); encoded->CopyStreamTo(out); } // Verify it { - FileStream enc("testfiles" DIRECTORY_SEPARATOR - "testenc1_enc"); + FileStream enc("testfiles/testenc1_enc"); TEST_THAT(BackupStoreFile::VerifyEncodedFileFormat(enc) == true); } // Decode it { - FileStream enc("testfiles" DIRECTORY_SEPARATOR - "testenc1_enc"); - BackupStoreFile::DecodeFile(enc, "testfiles" - DIRECTORY_SEPARATOR "testenc1_orig", - IOStream::TimeOutInfinite); + FileStream enc("testfiles/testenc1_enc"); + BackupStoreFile::DecodeFile(enc, "testfiles/testenc1_orig", IOStream::TimeOutInfinite); } // Read in rebuilt original, and compare contents { - TEST_THAT(TestGetFileSize("testfiles" - DIRECTORY_SEPARATOR "testenc1_orig") - == sizeof(encfile)); - FileStream in("testfiles" DIRECTORY_SEPARATOR - "testenc1_orig"); + TEST_THAT(TestGetFileSize("testfiles/testenc1_orig") == sizeof(encfile)); + FileStream in("testfiles/testenc1_orig"); int encfile_i[ENCFILE_SIZE]; in.Read(encfile_i, sizeof(encfile_i)); TEST_THAT(memcmp(encfile, encfile_i, sizeof(encfile)) == 0); @@ -1579,8 +1567,7 @@ // Check how many blocks it had, and test the stream based interface { - FileStream enc("testfiles" DIRECTORY_SEPARATOR - "testenc1_enc"); + FileStream enc("testfiles/testenc1_enc"); std::auto_ptr decoded(BackupStoreFile::DecodeFileStream(enc, IOStream::TimeOutInfinite)); CollectInBufferStream d; decoded->CopyStreamTo(d, IOStream::TimeOutInfinite, 971 /* buffer block size */); @@ -1594,14 +1581,10 @@ // Test that the last block in a file, if less than 256 bytes, gets put into the last block { #define FILE_SIZE_JUST_OVER ((4096*2)+58) - FileStream f("testfiles" DIRECTORY_SEPARATOR - "testenc2", O_WRONLY | O_CREAT | O_EXCL); + FileStream f("testfiles/testenc2", O_WRONLY | O_CREAT | O_EXCL); f.Write(encfile + 2, FILE_SIZE_JUST_OVER); BackupStoreFilenameClear name("testenc2"); - std::auto_ptr encoded( - BackupStoreFile::EncodeFile( - "testfiles" DIRECTORY_SEPARATOR - "testenc2", 32, name)); + std::auto_ptr encoded(BackupStoreFile::EncodeFile("testfiles/testenc2", 32, name)); CollectInBufferStream e; encoded->CopyStreamTo(e); e.SetForReading(); @@ -1617,8 +1600,7 @@ // Test that reordered streams work too { - FileStream enc("testfiles" DIRECTORY_SEPARATOR - "testenc1_enc"); + FileStream enc("testfiles/testenc1_enc"); std::auto_ptr reordered(BackupStoreFile::ReorderFileToStreamOrder(&enc, false)); std::auto_ptr decoded(BackupStoreFile::DecodeFileStream(*reordered, IOStream::TimeOutInfinite)); CollectInBufferStream d; @@ -1650,14 +1632,9 @@ // Store info { RaidFileWrite::CreateDirectory(0, "test-info"); - BackupStoreInfo::CreateNew(76, "test-info" DIRECTORY_SEPARATOR, - 0, 3461231233455433LL, 2934852487LL); - TEST_CHECK_THROWS(BackupStoreInfo::CreateNew(76, - "test-info" DIRECTORY_SEPARATOR, 0, 0, 0), - RaidFileException, CannotOverwriteExistingFile); - std::auto_ptr info( - BackupStoreInfo::Load(76, - "test-info" DIRECTORY_SEPARATOR, 0, true)); + BackupStoreInfo::CreateNew(76, "test-info/", 0, 3461231233455433LL, 2934852487LL); + TEST_CHECK_THROWS(BackupStoreInfo::CreateNew(76, "test-info/", 0, 0, 0), RaidFileException, CannotOverwriteExistingFile); + std::auto_ptr info(BackupStoreInfo::Load(76, "test-info/", 0, true)); TEST_CHECK_THROWS(info->Save(), BackupStoreException, StoreInfoIsReadOnly); TEST_CHECK_THROWS(info->ChangeBlocksUsed(1), BackupStoreException, StoreInfoIsReadOnly); TEST_CHECK_THROWS(info->ChangeBlocksInOldFiles(1), BackupStoreException, StoreInfoIsReadOnly); @@ -1666,8 +1643,7 @@ TEST_CHECK_THROWS(info->AddDeletedDirectory(2), BackupStoreException, StoreInfoIsReadOnly); } { - std::auto_ptr info(BackupStoreInfo::Load(76, - "test-info" DIRECTORY_SEPARATOR, 0, false)); + std::auto_ptr info(BackupStoreInfo::Load(76, "test-info/", 0, false)); info->ChangeBlocksUsed(8); info->ChangeBlocksInOldFiles(9); info->ChangeBlocksInDeletedFiles(10); @@ -1685,8 +1661,7 @@ info->Save(); } { - std::auto_ptr info(BackupStoreInfo::Load(76, - "test-info" DIRECTORY_SEPARATOR, 0, true)); + std::auto_ptr info(BackupStoreInfo::Load(76, "test-info/", 0, true)); TEST_THAT(info->GetBlocksUsed() == 7); TEST_THAT(info->GetBlocksInOldFiles() == 5); TEST_THAT(info->GetBlocksInDeletedFiles() == 1); @@ -1704,17 +1679,14 @@ // Context TLSContext context; context.Initialise(false /* client */, - "testfiles" DIRECTORY_SEPARATOR "clientCerts.pem", - "testfiles" DIRECTORY_SEPARATOR "clientPrivKey.pem", - "testfiles" DIRECTORY_SEPARATOR "clientTrustedCAs.pem"); + "testfiles/clientCerts.pem", + "testfiles/clientPrivKey.pem", + "testfiles/clientTrustedCAs.pem"); // First, try logging in without an account having been created... just make sure login fails. -#ifdef WIN32 - int pid = LaunchServer("..\\..\\bin\\bbstored\\bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); -#else - int pid = LaunchServer("../../bin/bbstored/bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); -#endif + int pid = LaunchServer(BBSTORED " testfiles/bbstored.conf", + "testfiles/bbstored.pid"); TEST_THAT(pid != -1 && pid != 0); if(pid > 0) @@ -1975,11 +1947,7 @@ // Use the setup crypto command to set up all these keys, so that the bbackupquery command can be used // for seeing what's going on. -#ifdef WIN32 - BackupClientCryptoKeys_Setup("testfiles\\bbackupd.keys"); -#else BackupClientCryptoKeys_Setup("testfiles/bbackupd.keys"); -#endif // encode in some filenames -- can't do static initialisation // because the key won't be set up when these are initialised From boxbackup-dev at fluffy.co.uk Sun Mar 25 00:37:54 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 00:37:54 +0000 Subject: [Box Backup-commit] COMMIT r1498 - box/chris/merge/test/backupstorefix Message-ID: Author: chris Date: 2007-03-25 00:37:54 +0000 (Sun, 25 Mar 2007) New Revision: 1498 Modified: box/chris/merge/test/backupstorefix/testbackupstorefix.cpp Log: Fix spurious/suprious typos. Use #defined paths for executables to remove win32/unix differences. Don't check for bbstored memory leaks on win32. Wrap long lines for readability. (refs #3) Modified: box/chris/merge/test/backupstorefix/testbackupstorefix.cpp =================================================================== --- box/chris/merge/test/backupstorefix/testbackupstorefix.cpp 2007-03-25 00:24:40 UTC (rev 1497) +++ box/chris/merge/test/backupstorefix/testbackupstorefix.cpp 2007-03-25 00:37:54 UTC (rev 1498) @@ -43,7 +43,7 @@ - all old flags delete store info -add suprious file +add spurious file delete directory (should appear again) change container ID of directory delete a file @@ -66,8 +66,8 @@ std::map objectIsDir; #define RUN_CHECK \ - ::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf check 01234567"); \ - ::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf check 01234567 fix"); + ::system(BBSTOREACCOUNTS " -c testfiles/bbstored.conf check 01234567"); \ + ::system(BBSTOREACCOUNTS " -c testfiles/bbstored.conf check 01234567 fix"); // Get ID of an object given a filename int32_t getID(const char *name) @@ -265,7 +265,7 @@ TEST_THAT(dir.CheckAndFix() == false); check_dir_dep(dir, c1); - // Check that a suprious depends older ref is undone + // Check that a spurious depends older ref is undone e2->SetDependsOlder(1); TEST_THAT(dir.CheckAndFix() == true); TEST_THAT(dir.CheckAndFix() == false); @@ -290,37 +290,52 @@ rcontroller.Initialise("testfiles/raidfile.conf"); // Create an account - TEST_THAT_ABORTONFAIL(::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf create 01234567 0 10000B 20000B") == 0); + TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS + " -c testfiles/bbstored.conf " + "create 01234567 0 10000B 20000B") == 0); TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); // Start the bbstored server - int pid = LaunchServer("../../bin/bbstored/bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); + int pid = LaunchServer(BBSTORED " testfiles/bbstored.conf", + "testfiles/bbstored.pid"); TEST_THAT(pid != -1 && pid != 0); + if(pid > 0) { ::sleep(1); TEST_THAT(ServerIsAlive(pid)); // Run the perl script to create the initial directories - TEST_THAT_ABORTONFAIL(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl init") == 0); + TEST_THAT_ABORTONFAIL(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl init") == 0); - int bbackupd_pid = LaunchServer("../../bin/bbackupd/bbackupd testfiles/bbackupd.conf", "testfiles/bbackupd.pid"); + int bbackupd_pid = LaunchServer(BBACKUPD + " testfiles/bbackupd.conf", "testfiles/bbackupd.pid"); TEST_THAT(bbackupd_pid != -1 && bbackupd_pid != 0); + if(bbackupd_pid > 0) { ::sleep(1); TEST_THAT(ServerIsAlive(bbackupd_pid)); // Create a nice store directory - wait_for_operation(30); + wait_for_operation(14); // That'll do nicely, stop the server - TEST_THAT(KillServer(bbackupd_pid)); - TestRemoteProcessMemLeaks("bbackupd.memleaks"); + #ifdef WIN32 + terminate_bbackupd(bbackupd_pid); + // implicit check for memory leaks + #else + TEST_THAT(KillServer(bbackupd_pid)); + TestRemoteProcessMemLeaks("bbackupd.memleaks"); + #endif } // Generate a list of all the object IDs - TEST_THAT_ABORTONFAIL(::system("../../bin/bbackupquery/bbackupquery -q -c testfiles/bbackupd.conf \"list -r\" quit > testfiles/initial-listing.txt") == 0); + TEST_THAT_ABORTONFAIL(::system(BBACKUPQUERY " -q " + "-c testfiles/bbackupd.conf \"list -r\" quit " + "> testfiles/initial-listing.txt") == 0); + // And load it in { FILE *f = ::fopen("testfiles/initial-listing.txt", "r"); @@ -331,7 +346,8 @@ char name[256]; while(::fgets(line, sizeof(line), f) != 0) { - TEST_THAT(::sscanf(line, "%x %s %s", &id, flags, name) == 3); + TEST_THAT(::sscanf(line, "%x %s %s", &id, + flags, name) == 3); bool isDir = (::strcmp(flags, "-d---") == 0); //TRACE3("%x,%d,%s\n", id, isDir, name); MEMLEAKFINDER_NO_LEAKS; @@ -349,19 +365,24 @@ del.Delete(); } { - // Add a suprious file - RaidFileWrite random(discSetNum, storeRoot + "randomfile"); + // Add a spurious file + RaidFileWrite random(discSetNum, + storeRoot + "randomfile"); random.Open(); random.Write("test", 4); random.Commit(true); } + // Fix it RUN_CHECK + // Check everything is as it was - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 0") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl check 0") == 0); // Check the random file doesn't exist { - TEST_THAT(!RaidFileRead::FileExists(discSetNum, storeRoot + "01/randomfile")); + TEST_THAT(!RaidFileRead::FileExists(discSetNum, + storeRoot + "01/randomfile")); } // ------------------------------------------------------------------------------------------------ @@ -401,6 +422,8 @@ file_BlockIndexEntry e[2]; } h; TEST_THAT(file->Read(&h, sizeof(h)) == sizeof(h)); + file->Close(); + // Modify TEST_THAT(box_ntoh64(h.hdr.mOtherFileID) == 0); TEST_THAT(box_ntoh64(h.hdr.mNumBlocks) >= 2); @@ -416,7 +439,9 @@ // Fix it RUN_CHECK // Check - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 1") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl check 1") + == 0); // Check the modified file doesn't exist TEST_THAT(!RaidFileRead::FileExists(discSetNum, fn)); @@ -431,7 +456,7 @@ SaveDirectory("Test1/foreomizes/stemptinevidate/ict", dir); } int64_t duplicatedID = 0; - int64_t notSupriousFileSize = 0; + int64_t notSpuriousFileSize = 0; { BackupStoreDirectory dir; LoadDirectory("Test1/cannes/ict/peep", dir); @@ -449,7 +474,7 @@ BackupStoreDirectory::Iterator i(dir); BackupStoreDirectory::Entry *en = i.Next(BackupStoreDirectory::Entry::Flags_File); TEST_THAT(en != 0); - notSupriousFileSize = en->GetSizeInBlocks(); + notSpuriousFileSize = en->GetSizeInBlocks(); en->SetSizeInBlocks(3473874); TEST_THAT(en->GetSizeInBlocks() == 3473874); } @@ -462,7 +487,8 @@ // Fix it RUN_CHECK // Check everything is as it should be - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 2") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl check 2") == 0); { BackupStoreDirectory dir; LoadDirectory("Test1/foreomizes/stemptinevidate/ict", dir); @@ -488,7 +514,7 @@ BackupStoreDirectory::Iterator i(dir); BackupStoreDirectory::Entry *en = i.Next(BackupStoreDirectory::Entry::Flags_File); TEST_THAT(en != 0); - TEST_THAT(en->GetSizeInBlocks() == notSupriousFileSize); + TEST_THAT(en->GetSizeInBlocks() == notSpuriousFileSize); } } @@ -518,7 +544,8 @@ // Fix it RUN_CHECK // Check everything is as it should be - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 3") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl check 3") == 0); { BackupStoreDirectory dir; LoadDirectory("Test1/foreomizes/stemptinevidate/ict", dir); @@ -532,18 +559,22 @@ // Fix it RUN_CHECK // Check everything is where it is predicted to be - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 4") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl check 4") == 0); // ------------------------------------------------------------------------------------------------ ::printf(" === Corrupt file and dir\n"); // File - CorruptObject("Test1/foreomizes/stemptinevidate/algoughtnerge", 33, "34i729834298349283479233472983sdfhasgs"); + CorruptObject("Test1/foreomizes/stemptinevidate/algoughtnerge", + 33, "34i729834298349283479233472983sdfhasgs"); // Dir - CorruptObject("Test1/cannes/imulatrougge/foreomizes", 23, "dsf32489sdnadf897fd2hjkesdfmnbsdfcsfoisufio2iofe2hdfkjhsf"); + CorruptObject("Test1/cannes/imulatrougge/foreomizes",23, + "dsf32489sdnadf897fd2hjkesdfmnbsdfcsfoisufio2iofe2hdfkjhsf"); // Fix it RUN_CHECK // Check everything is where it should be - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 5") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl check 5") == 0); // ------------------------------------------------------------------------------------------------ ::printf(" === Overwrite root with a file\n"); @@ -557,13 +588,16 @@ // Fix it RUN_CHECK // Check everything is where it should be - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl reroot 6") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl reroot 6") == 0); // ------------------------------------------------------------------------------------------------ // Stop server TEST_THAT(KillServer(pid)); - TestRemoteProcessMemLeaks("bbstored.memleaks"); + #ifndef WIN32 + TestRemoteProcessMemLeaks("bbstored.memleaks"); + #endif } return 0; From boxbackup-dev at fluffy.co.uk Sun Mar 25 00:42:41 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 00:42:41 +0000 Subject: [Box Backup-commit] COMMIT r1499 - box/chris/merge/test/backupstorepatch Message-ID: Author: chris Date: 2007-03-25 00:42:41 +0000 (Sun, 25 Mar 2007) New Revision: 1499 Modified: box/chris/merge/test/backupstorepatch/testbackupstorepatch.cpp Log: Use #defined paths for executables to remove win32/unix differences. (refs #3) Modified: box/chris/merge/test/backupstorepatch/testbackupstorepatch.cpp =================================================================== --- box/chris/merge/test/backupstorepatch/testbackupstorepatch.cpp 2007-03-25 00:37:54 UTC (rev 1498) +++ box/chris/merge/test/backupstorepatch/testbackupstorepatch.cpp 2007-03-25 00:42:41 UTC (rev 1499) @@ -310,9 +310,8 @@ "testfiles/clientTrustedCAs.pem"); // Create an account - TEST_THAT_ABORTONFAIL(RunCommand( - "../../bin/bbstoreaccounts/bbstoreaccounts " - "-c testfiles/bbstored.conf " + TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS + " -c testfiles/bbstored.conf " "create 01234567 0 30000B 40000B") == 0); TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); @@ -323,7 +322,8 @@ test_depends_in_dirs(); // First, try logging in without an account having been created... just make sure login fails. - int pid = LaunchServer("../../bin/bbstored/bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); + int pid = LaunchServer(BBSTORED " testfiles/bbstored.conf", + "testfiles/bbstored.pid"); TEST_THAT(pid != -1 && pid != 0); if(pid > 0) { From boxbackup-dev at fluffy.co.uk Sun Mar 25 00:43:59 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 00:43:59 +0000 Subject: [Box Backup-commit] COMMIT r1500 - box/chris/general/test/backupstorefix Message-ID: Author: chris Date: 2007-03-25 00:43:59 +0000 (Sun, 25 Mar 2007) New Revision: 1500 Modified: box/chris/general/test/backupstorefix/testbackupstorefix.cpp Log: Merge back changes from win32/merge: Wrap long lines. Fix suprious/spurious typos. Modified: box/chris/general/test/backupstorefix/testbackupstorefix.cpp =================================================================== --- box/chris/general/test/backupstorefix/testbackupstorefix.cpp 2007-03-25 00:42:41 UTC (rev 1499) +++ box/chris/general/test/backupstorefix/testbackupstorefix.cpp 2007-03-25 00:43:59 UTC (rev 1500) @@ -43,7 +43,7 @@ - all old flags delete store info -add suprious file +add spurious file delete directory (should appear again) change container ID of directory delete a file @@ -65,15 +65,9 @@ std::map nameToID; std::map objectIsDir; -#ifdef WIN32 #define RUN_CHECK \ - ::system("..\\..\\bin\\bbstoreaccounts\\bbstoreaccounts -c testfiles/bbstored.conf check 01234567"); \ - ::system("..\\..\\bin\\bbstoreaccounts\\bbstoreaccounts -c testfiles/bbstored.conf check 01234567 fix"); -#else -#define RUN_CHECK \ - ::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf check 01234567"); \ - ::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf check 01234567 fix"); -#endif + ::system(BBSTOREACCOUNTS " -c testfiles/bbstored.conf check 01234567"); \ + ::system(BBSTOREACCOUNTS " -c testfiles/bbstored.conf check 01234567 fix"); // Get ID of an object given a filename int32_t getID(const char *name) @@ -272,7 +266,7 @@ TEST_THAT(dir.CheckAndFix() == false); check_dir_dep(dir, c1); - // Check that a suprious depends older ref is undone + // Check that a spurious depends older ref is undone e2->SetDependsOlder(1); TEST_THAT(dir.CheckAndFix() == true); TEST_THAT(dir.CheckAndFix() == false); @@ -297,33 +291,29 @@ rcontroller.Initialise("testfiles/raidfile.conf"); // Create an account -#ifdef WIN32 - TEST_THAT_ABORTONFAIL(::system("..\\..\\bin\\bbstoreaccounts\\bbstoreaccounts -c testfiles/bbstored.conf create 01234567 0 10000B 20000B") == 0); -#else - TEST_THAT_ABORTONFAIL(::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf create 01234567 0 10000B 20000B") == 0); + TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS + " -c testfiles/bbstored.conf " + "create 01234567 0 10000B 20000B") == 0); TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); -#endif // Start the bbstored server -#ifdef WIN32 - int pid = LaunchServer("..\\..\\bin\\bbstored\\bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); -#else - int pid = LaunchServer("../../bin/bbstored/bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); -#endif + int pid = LaunchServer(BBSTORED " testfiles/bbstored.conf", + "testfiles/bbstored.pid"); + TEST_THAT(pid != -1 && pid != 0); - TEST_THAT(pid != -1 && pid != 0); if(pid > 0) { ::sleep(1); TEST_THAT(ServerIsAlive(pid)); // Run the perl script to create the initial directories - TEST_THAT_ABORTONFAIL(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl init") == 0); + TEST_THAT_ABORTONFAIL(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl init") == 0); - int bbackupd_pid = LaunchServer(BBACKUPD + int bbackupd_pid = LaunchServer(BBACKUPD " testfiles/bbackupd.conf", "testfiles/bbackupd.pid"); + TEST_THAT(bbackupd_pid != -1 && bbackupd_pid != 0); - TEST_THAT(bbackupd_pid != -1 && bbackupd_pid != 0); if(bbackupd_pid > 0) { ::sleep(1); @@ -334,11 +324,11 @@ // That'll do nicely, stop the server #ifdef WIN32 - terminate_bbackupd(bbackupd_pid); - // implicit check for memory leaks + terminate_bbackupd(bbackupd_pid); + // implicit check for memory leaks #else - TEST_THAT(KillServer(bbackupd_pid)); - TestRemoteProcessMemLeaks("bbackupd.memleaks"); + TEST_THAT(KillServer(bbackupd_pid)); + TestRemoteProcessMemLeaks("bbackupd.memleaks"); #endif } @@ -357,7 +347,8 @@ char name[256]; while(::fgets(line, sizeof(line), f) != 0) { - TEST_THAT(::sscanf(line, "%x %s %s", &id, flags, name) == 3); + TEST_THAT(::sscanf(line, "%x %s %s", &id, + flags, name) == 3); bool isDir = (::strcmp(flags, "-d---") == 0); //TRACE3("%x,%d,%s\n", id, isDir, name); MEMLEAKFINDER_NO_LEAKS; @@ -376,7 +367,8 @@ } { // Add a spurious file - RaidFileWrite random(discSetNum, storeRoot + "randomfile"); + RaidFileWrite random(discSetNum, + storeRoot + "randomfile"); random.Open(); random.Write("test", 4); random.Commit(true); @@ -388,7 +380,6 @@ // Check everything is as it was TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 0") == 0); - // Check the random file doesn't exist { TEST_THAT(!RaidFileRead::FileExists(discSetNum, @@ -449,7 +440,9 @@ // Fix it RUN_CHECK // Check - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 1") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl check 1") + == 0); // Check the modified file doesn't exist TEST_THAT(!RaidFileRead::FileExists(discSetNum, fn)); @@ -495,7 +488,8 @@ // Fix it RUN_CHECK // Check everything is as it should be - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 2") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl check 2") == 0); { BackupStoreDirectory dir; LoadDirectory("Test1/foreomizes/stemptinevidate/ict", dir); @@ -551,7 +545,8 @@ // Fix it RUN_CHECK // Check everything is as it should be - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 3") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl check 3") == 0); { BackupStoreDirectory dir; LoadDirectory("Test1/foreomizes/stemptinevidate/ict", dir); @@ -565,18 +560,22 @@ // Fix it RUN_CHECK // Check everything is where it is predicted to be - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 4") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl check 4") == 0); // ------------------------------------------------------------------------------------------------ ::printf(" === Corrupt file and dir\n"); // File - CorruptObject("Test1/foreomizes/stemptinevidate/algoughtnerge", 33, "34i729834298349283479233472983sdfhasgs"); + CorruptObject("Test1/foreomizes/stemptinevidate/algoughtnerge", + 33, "34i729834298349283479233472983sdfhasgs"); // Dir - CorruptObject("Test1/cannes/imulatrougge/foreomizes", 23, "dsf32489sdnadf897fd2hjkesdfmnbsdfcsfoisufio2iofe2hdfkjhsf"); + CorruptObject("Test1/cannes/imulatrougge/foreomizes",23, + "dsf32489sdnadf897fd2hjkesdfmnbsdfcsfoisufio2iofe2hdfkjhsf"); // Fix it RUN_CHECK // Check everything is where it should be - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl check 5") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl check 5") == 0); // ------------------------------------------------------------------------------------------------ ::printf(" === Overwrite root with a file\n"); @@ -590,15 +589,16 @@ // Fix it RUN_CHECK // Check everything is where it should be - TEST_THAT(::system(PERL_EXECUTABLE " testfiles/testbackupstorefix.pl reroot 6") == 0); + TEST_THAT(::system(PERL_EXECUTABLE + " testfiles/testbackupstorefix.pl reroot 6") == 0); // ------------------------------------------------------------------------------------------------ // Stop server TEST_THAT(KillServer(pid)); -#ifndef WIN32 - TestRemoteProcessMemLeaks("bbstored.memleaks"); -#endif + #ifndef WIN32 + TestRemoteProcessMemLeaks("bbstored.memleaks"); + #endif } return 0; From boxbackup-dev at fluffy.co.uk Sun Mar 25 00:49:16 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 00:49:16 +0000 Subject: [Box Backup-commit] COMMIT r1501 - box/chris/merge/test/backupstorefix Message-ID: Author: chris Date: 2007-03-25 00:49:16 +0000 (Sun, 25 Mar 2007) New Revision: 1501 Modified: box/chris/merge/test/backupstorefix/testbackupstorefix.cpp Log: Close RaidFile before committing, so that commit can work on win32. Typo fix. (refs #3) Modified: box/chris/merge/test/backupstorefix/testbackupstorefix.cpp =================================================================== --- box/chris/merge/test/backupstorefix/testbackupstorefix.cpp 2007-03-25 00:43:59 UTC (rev 1500) +++ box/chris/merge/test/backupstorefix/testbackupstorefix.cpp 2007-03-25 00:49:16 UTC (rev 1501) @@ -125,6 +125,7 @@ w.Write(rubbish, rubbish_len); // Copy rest of file r->CopyStreamTo(w); + r->Close(); // Commit w.Commit(true /* convert now */); } @@ -448,7 +449,7 @@ } // ------------------------------------------------------------------------------------------------ - ::printf(" === Delete directory, change container ID of another, duplicate entry in dir, supurious file size, delete file\n"); + ::printf(" === Delete directory, change container ID of another, duplicate entry in dir, spurious file size, delete file\n"); { BackupStoreDirectory dir; LoadDirectory("Test1/foreomizes/stemptinevidate/ict", dir); From boxbackup-dev at fluffy.co.uk Sun Mar 25 00:50:44 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 00:50:44 +0000 Subject: [Box Backup-commit] COMMIT r1502 - box/chris/merge/lib/backupstore Message-ID: Author: chris Date: 2007-03-25 00:50:44 +0000 (Sun, 25 Mar 2007) New Revision: 1502 Modified: box/chris/merge/lib/backupstore/BackupStoreInfo.cpp Log: Allow '/' as a path separator on all platforms, even Win32 (refs #3, merges [1496]) Modified: box/chris/merge/lib/backupstore/BackupStoreInfo.cpp =================================================================== --- box/chris/merge/lib/backupstore/BackupStoreInfo.cpp 2007-03-25 00:49:16 UTC (rev 1501) +++ box/chris/merge/lib/backupstore/BackupStoreInfo.cpp 2007-03-25 00:50:44 UTC (rev 1502) @@ -124,7 +124,8 @@ }; // Generate the filename - ASSERT(rRootDir[rRootDir.size() - 1] == DIRECTORY_SEPARATOR_ASCHAR); + ASSERT(rRootDir[rRootDir.size() - 1] == '/' || + rRootDir[rRootDir.size() - 1] == DIRECTORY_SEPARATOR_ASCHAR); std::string fn(rRootDir + INFO_FILENAME); // Open the file for writing From boxbackup-dev at fluffy.co.uk Sun Mar 25 02:05:45 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 02:05:45 +0100 Subject: [Box Backup-commit] COMMIT r1503 - box/chris/general/test/backupstorepatch Message-ID: Author: chris Date: 2007-03-25 02:05:45 +0100 (Sun, 25 Mar 2007) New Revision: 1503 Modified: box/chris/general/test/backupstorepatch/testbackupstorepatch.cpp Log: Merge back changes from chris/merge: Use #defined paths to executables to reduce unix/win32 differences. Reformat long lines for readability. Modified: box/chris/general/test/backupstorepatch/testbackupstorepatch.cpp =================================================================== --- box/chris/general/test/backupstorepatch/testbackupstorepatch.cpp 2007-03-25 00:50:44 UTC (rev 1502) +++ box/chris/general/test/backupstorepatch/testbackupstorepatch.cpp 2007-03-25 01:05:45 UTC (rev 1503) @@ -310,12 +310,10 @@ "testfiles/clientTrustedCAs.pem"); // Create an account -#ifdef WIN32 - TEST_THAT_ABORTONFAIL(::system("..\\..\\bin\\bbstoreaccounts\\bbstoreaccounts -c testfiles/bbstored.conf create 01234567 0 30000B 40000B") == 0); -#else - TEST_THAT_ABORTONFAIL(::system("../../bin/bbstoreaccounts/bbstoreaccounts -c testfiles/bbstored.conf create 01234567 0 30000B 40000B") == 0); + TEST_THAT_ABORTONFAIL(::system(BBSTOREACCOUNTS + " -c testfiles/bbstored.conf " + "create 01234567 0 30000B 40000B") == 0); TestRemoteProcessMemLeaks("bbstoreaccounts.memleaks"); -#endif // Create test files create_test_files(); @@ -324,12 +322,8 @@ test_depends_in_dirs(); // First, try logging in without an account having been created... just make sure login fails. -#ifdef WIN32 - int pid = LaunchServer("..\\..\\bin\\bbstored\\bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); -#else - int pid = LaunchServer("../../bin/bbstored/bbstored testfiles/bbstored.conf", "testfiles/bbstored.pid"); -#endif - + int pid = LaunchServer(BBSTORED " testfiles/bbstored.conf", + "testfiles/bbstored.pid"); TEST_THAT(pid != -1 && pid != 0); if(pid > 0) { @@ -582,10 +576,14 @@ } #ifdef WIN32 + // Cannot signal bbstored to do housekeeping now, + // so just wait until we're sure it's done wait_for_operation(12); #else - // Send the server a restart signal, so it does housekeeping immediately, and wait for it to happen - ::sleep(1); // wait for old connections to terminate + // Send the server a restart signal, so it does + // housekeeping immediately, and wait for it to happen + // Wait for old connections to terminate + ::sleep(1); ::kill(pid, SIGHUP); #endif From boxbackup-dev at fluffy.co.uk Sun Mar 25 02:16:53 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 02:16:53 +0100 Subject: [Box Backup-commit] COMMIT r1504 - box/chris/merge/test/basicserver Message-ID: Author: chris Date: 2007-03-25 02:16:53 +0100 (Sun, 25 Mar 2007) New Revision: 1504 Modified: box/chris/merge/test/basicserver/testbasicserver.cpp Log: Win32 fixes (paths to executables, and don't try things that don't work on Windows, like sending HUP signals) (refs #3) Modified: box/chris/merge/test/basicserver/testbasicserver.cpp =================================================================== --- box/chris/merge/test/basicserver/testbasicserver.cpp 2007-03-25 01:05:45 UTC (rev 1503) +++ box/chris/merge/test/basicserver/testbasicserver.cpp 2007-03-25 01:16:53 UTC (rev 1504) @@ -431,84 +431,149 @@ } } -//printf("SKIPPING TESTS------------------------\n"); -//goto protocolserver; + //printf("SKIPPING TESTS------------------------\n"); + //goto protocolserver; // Launch a basic server { - int pid = LaunchServer("./test srv1 testfiles/srv1.conf", "testfiles/srv1.pid"); + #ifdef WIN32 + int pid = LaunchServer("test srv1 testfiles\\srv1.conf", + "testfiles\\srv1.pid"); + #else + int pid = LaunchServer("./test srv1 testfiles/srv1.conf", + "testfiles/srv1.pid"); + #endif + TEST_THAT(pid != -1 && pid != 0); if(pid > 0) { // Check that it's written the expected file - TEST_THAT(TestFileExists("testfiles/srv1.test1")); + TEST_THAT(TestFileExists("testfiles" + DIRECTORY_SEPARATOR "srv1.test1")); TEST_THAT(ServerIsAlive(pid)); + // Move the config file over - TEST_THAT(::rename("testfiles/srv1b.conf", "testfiles/srv1.conf") != -1); - // Get it to reread the config file - TEST_THAT(HUPServer(pid)); - ::sleep(1); - TEST_THAT(ServerIsAlive(pid)); - // Check that new file exists - TEST_THAT(TestFileExists("testfiles/srv1.test2")); + #ifdef WIN32 + TEST_THAT(::unlink("testfiles/srv1.conf") + != -1); + #endif + TEST_THAT(::rename( + "testfiles" DIRECTORY_SEPARATOR "srv1b.conf", + "testfiles" DIRECTORY_SEPARATOR "srv1.conf") + != -1); + + #ifndef WIN32 + // Get it to reread the config file + TEST_THAT(HUPServer(pid)); + ::sleep(1); + TEST_THAT(ServerIsAlive(pid)); + // Check that new file exists + TEST_THAT(TestFileExists("testfiles" + DIRECTORY_SEPARATOR "srv1.test2")); + #endif // !WIN32 + // Kill it off TEST_THAT(KillServer(pid)); - TestRemoteProcessMemLeaks("generic-daemon.memleaks"); + #ifndef WIN32 + TestRemoteProcessMemLeaks( + "generic-daemon.memleaks"); + #endif // !WIN32 } } // Launch a test forking server { - int pid = LaunchServer("./test srv2 testfiles/srv2.conf", "testfiles/srv2.pid"); + #ifdef WIN32 + int pid = LaunchServer("test srv2 testfiles\\srv2.conf", + "testfiles\\srv2.pid"); + #else + int pid = LaunchServer("./test srv2 testfiles/srv2.conf", + "testfiles/srv2.pid"); + #endif + TEST_THAT(pid != -1 && pid != 0); if(pid > 0) { // Will it restart? TEST_THAT(ServerIsAlive(pid)); - TEST_THAT(HUPServer(pid)); - ::sleep(1); - TEST_THAT(ServerIsAlive(pid)); + + #ifndef WIN32 + TEST_THAT(HUPServer(pid)); + ::sleep(1); + TEST_THAT(ServerIsAlive(pid)); + #endif // !WIN32 + // Make some connections { SocketStream conn1; conn1.Open(Socket::TypeINET, "localhost", 2003); - SocketStream conn2; - conn2.Open(Socket::TypeUNIX, "testfiles/srv2.sock"); - SocketStream conn3; - conn3.Open(Socket::TypeINET, "localhost", 2003); + + #ifndef WIN32 + SocketStream conn2; + conn2.Open(Socket::TypeUNIX, + "testfiles/srv2.sock"); + SocketStream conn3; + conn3.Open(Socket::TypeINET, + "localhost", 2003); + #endif // !WIN32 + // Quick check that reconnections fail - TEST_CHECK_THROWS(conn1.Open(Socket::TypeUNIX, "testfiles/srv2.sock");, ServerException, SocketAlreadyOpen); + TEST_CHECK_THROWS(conn1.Open(Socket::TypeUNIX, + "testfiles/srv2.sock");, + ServerException, SocketAlreadyOpen); + // Stuff some data around std::vector conns; conns.push_back(&conn1); - conns.push_back(&conn2); - conns.push_back(&conn3); + + #ifndef WIN32 + conns.push_back(&conn2); + conns.push_back(&conn3); + #endif // !WIN32 Srv2TestConversations(conns); // Implicit close } - // HUP again - TEST_THAT(HUPServer(pid)); - ::sleep(1); - TEST_THAT(ServerIsAlive(pid)); + + #ifndef WIN32 + // HUP again + TEST_THAT(HUPServer(pid)); + ::sleep(1); + TEST_THAT(ServerIsAlive(pid)); + #endif // !WIN32 + // Kill it TEST_THAT(KillServer(pid)); ::sleep(1); TEST_THAT(!ServerIsAlive(pid)); - TestRemoteProcessMemLeaks("test-srv2.memleaks"); + + #ifndef WIN32 + TestRemoteProcessMemLeaks("test-srv2.memleaks"); + #endif // !WIN32 } } // Launch a test SSL server { - int pid = LaunchServer("./test srv3 testfiles/srv3.conf", "testfiles/srv3.pid"); + #ifdef WIN32 + int pid = LaunchServer("test srv3 testfiles\\srv3.conf", + "testfiles\\srv3.pid"); + #else + int pid = LaunchServer("./test srv3 testfiles/srv3.conf", + "testfiles/srv3.pid"); + #endif + TEST_THAT(pid != -1 && pid != 0); if(pid > 0) { // Will it restart? TEST_THAT(ServerIsAlive(pid)); - TEST_THAT(HUPServer(pid)); - ::sleep(1); - TEST_THAT(ServerIsAlive(pid)); + + #ifndef WIN32 + TEST_THAT(HUPServer(pid)); + ::sleep(1); + TEST_THAT(ServerIsAlive(pid)); + #endif + // Make some connections { // SSL library @@ -523,36 +588,63 @@ SocketStreamTLS conn1; conn1.Open(context, Socket::TypeINET, "localhost", 2003); - SocketStreamTLS conn2; - conn2.Open(context, Socket::TypeUNIX, "testfiles/srv3.sock"); - SocketStreamTLS conn3; - conn3.Open(context, Socket::TypeINET, "localhost", 2003); + #ifndef WIN32 + SocketStreamTLS conn2; + conn2.Open(context, Socket::TypeUNIX, + "testfiles/srv3.sock"); + SocketStreamTLS conn3; + conn3.Open(context, Socket::TypeINET, + "localhost", 2003); + #endif + // Quick check that reconnections fail - TEST_CHECK_THROWS(conn1.Open(context, Socket::TypeUNIX, "testfiles/srv3.sock");, ServerException, SocketAlreadyOpen); + TEST_CHECK_THROWS(conn1.Open(context, + Socket::TypeUNIX, + "testfiles/srv3.sock"), + ServerException, SocketAlreadyOpen); + // Stuff some data around std::vector conns; conns.push_back(&conn1); - conns.push_back(&conn2); - conns.push_back(&conn3); + + #ifndef WIN32 + conns.push_back(&conn2); + conns.push_back(&conn3); + #endif + Srv2TestConversations(conns); // Implicit close } - // HUP again - TEST_THAT(HUPServer(pid)); - ::sleep(1); - TEST_THAT(ServerIsAlive(pid)); + + #ifndef WIN32 + // HUP again + TEST_THAT(HUPServer(pid)); + ::sleep(1); + TEST_THAT(ServerIsAlive(pid)); + #endif + // Kill it TEST_THAT(KillServer(pid)); ::sleep(1); TEST_THAT(!ServerIsAlive(pid)); - TestRemoteProcessMemLeaks("test-srv3.memleaks"); + + #ifndef WIN32 + TestRemoteProcessMemLeaks("test-srv3.memleaks"); + #endif } } //protocolserver: // Launch a test protocol handling server { - int pid = LaunchServer("./test srv4 testfiles/srv4.conf", "testfiles/srv4.pid"); + #ifdef WIN32 + int pid = LaunchServer("test srv4 testfiles\\srv4.conf", + "testfiles\\srv4.pid"); + #else + int pid = LaunchServer("./test srv4 testfiles/srv4.conf", + "testfiles/srv4.pid"); + #endif + TEST_THAT(pid != -1 && pid != 0); if(pid > 0) { @@ -561,7 +653,12 @@ // Open a connection to it SocketStream conn; - conn.Open(Socket::TypeUNIX, "testfiles/srv4.sock"); + #ifdef WIN32 + conn.Open(Socket::TypeINET, "localhost", 2003); + #else + conn.Open(Socket::TypeUNIX, + "testfiles/srv4.sock"); + #endif // Create a protocol TestProtocolClient protocol(conn); @@ -624,7 +721,10 @@ TEST_THAT(KillServer(pid)); ::sleep(1); TEST_THAT(!ServerIsAlive(pid)); - TestRemoteProcessMemLeaks("test-srv4.memleaks"); + + #ifndef WIN32 + TestRemoteProcessMemLeaks("test-srv4.memleaks"); + #endif } } From boxbackup-dev at fluffy.co.uk Sun Mar 25 16:53:40 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 16:53:40 +0100 Subject: [Box Backup-commit] COMMIT r1505 - box/chris/merge/test/bbackupd Message-ID: Author: chris Date: 2007-03-25 16:53:39 +0100 (Sun, 25 Mar 2007) New Revision: 1505 Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp Log: Only include headers on systems which have them Modified: box/chris/merge/test/bbackupd/testbbackupd.cpp =================================================================== --- box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-25 01:16:53 UTC (rev 1504) +++ box/chris/merge/test/bbackupd/testbbackupd.cpp 2007-03-25 15:53:39 UTC (rev 1505) @@ -9,22 +9,33 @@ #include "Box.h" -#include +// do not include MinGW's dirent.h on Win32, +// as we override some of it in lib/win32. + +#ifndef WIN32 + #include +#endif + #include #include #include #include #include -#include #include + +#ifdef HAVE_SYS_WAIT_H + #include +#endif + #ifdef HAVE_SYS_XATTR_H -#include -#include + #include + #include #endif + #include #ifdef HAVE_SYSCALL -#include + #include #endif #include "Test.h" From boxbackup-dev at fluffy.co.uk Sun Mar 25 19:40:28 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 18:40:28 -0000 Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> References: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> Message-ID: <060.0aa2af72f8ac97b7448b75dcce34143d@fluffy.co.uk> #3: Merge Win32 branch -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: trunk Resolution: | Keywords: windows win32 merge branch -----------------------+---------------------------------------------------- Comment (by invisnet): [1486] I don't know if I can do this, but I'm going to try :-) I'd like to veto this change. You seem to have done a partial re- implementation of a completion port - see CreateIoCompletionPort and friends. -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From boxbackup-dev at fluffy.co.uk Sun Mar 25 23:02:34 2007 From: boxbackup-dev at fluffy.co.uk (boxbackup-dev at fluffy.co.uk) Date: Sun, 25 Mar 2007 22:02:34 -0000 Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> References: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> Message-ID: <060.568fea32f3f2e18039810af822f93a4c@fluffy.co.uk> #3: Merge Win32 branch -----------------------+---------------------------------------------------- Reporter: chris | Owner: chris Type: task | Status: new Priority: normal | Milestone: 0.11 Component: bbackupd | Version: trunk Resolution: | Keywords: windows win32 merge branch -----------------------+---------------------------------------------------- Comment (by chris): (In [1505]) Only include headers on systems which have them (refs #3) -- Ticket URL: Box Backup An open source, completely automatic on-line backup system for UNIX. From chris at qwirx.com Sun Mar 25 23:51:01 2007 From: chris at qwirx.com (Chris Wilson) Date: Sun, 25 Mar 2007 23:51:01 +0100 (BST) Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <060.0aa2af72f8ac97b7448b75dcce34143d@fluffy.co.uk> References: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> <060.0aa2af72f8ac97b7448b75dcce34143d@fluffy.co.uk> Message-ID: Hi Charles, On Sun, 25 Mar 2007, trac at fluffy.co.uk wrote: > [1486] I don't know if I can do this, but I'm going to try :-) > > I'd like to veto this change. You seem to have done a partial re- > implementation of a completion port - see CreateIoCompletionPort and > friends. Thanks for the suggestion, but I'm afraid that after reading the CreateIoCompletionPort docs I'm still not completely sure how I could use it to replace [1486]. So far, what I understand (I think) is that the command socket thread, instead of calling WaitForMultipleObjects, would call GetQueuedCompletionStatus, which blocks until either (i) the main thread posts a message to the port with PostQueuedCompletionStatus, or (ii) a packet is received over the command socket by the pending overlapped I/O. Is that correct? What bothers me is that MSDN says "After an instance of an open file is associated with an I/O completion port, it cannot be used in the ReadFileEx or WriteFileEx function." I assume that this means that I have to use overlapped I/O for both reading and writing from the pipe? I don't care about completion of writes, although I suppose I could dispatch the next waiting message (if any) to the pipe in this case. And how do I actually read or write on the pipe, if ReadFileEx and WriteFileEx are forbidden? Do good old ReadFile and WriteFile work properly? I have a feeling that implementing this will move the WinNamedPipeStream API so far away from POSIX that I will have to create a new class that doesn't extend IOStream for this purpose. While it might be a cleaner solution to the current overlapped/nonoverlapped read branches in the code, I'm worried about the duplication of code and possible maintenance headaches. Are there any samples of using CreateIoCompletionPort and PostQueuedCompletionStatus that I can freely copy and adapt for use in this case? -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at invis.net Mon Mar 26 00:44:22 2007 From: boxbackup at invis.net (Charles Lecklider) Date: Mon, 26 Mar 2007 00:44:22 +0100 Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: References: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> <060.0aa2af72f8ac97b7448b75dcce34143d@fluffy.co.uk> Message-ID: <46070956.3010206@invis.net> Chris Wilson wrote: > Hi Charles, > > On Sun, 25 Mar 2007, trac at fluffy.co.uk wrote: > >> [1486] I don't know if I can do this, but I'm going to try :-) >> >> I'd like to veto this change. You seem to have done a partial re- >> implementation of a completion port - see CreateIoCompletionPort and >> friends. > > Thanks for the suggestion, but I'm afraid that after reading the > CreateIoCompletionPort docs I'm still not completely sure how I could > use it to replace [1486]. > > So far, what I understand (I think) is that the command socket thread, > instead of calling WaitForMultipleObjects, would call > GetQueuedCompletionStatus, which blocks until either (i) the main thread > posts a message to the port with PostQueuedCompletionStatus, or (ii) a > packet is received over the command socket by the pending overlapped > I/O. Is that correct? Or until the pipe is closed, yes. > What bothers me is that MSDN says "After an instance of an open file is > associated with an I/O completion port, it cannot be used in the > ReadFileEx or WriteFileEx function." > > I assume that this means that I have to use overlapped I/O for both > reading and writing from the pipe? Erm, you have to do that anyway or you'll get some really strange behaviour. > I don't care about completion of > writes, although I suppose I could dispatch the next waiting message (if > any) to the pipe in this case. Trying to cut corners when using overlapped IO is not a good idea. > And how do I actually read or write on the pipe, if ReadFileEx and > WriteFileEx are forbidden? Do good old ReadFile and WriteFile work > properly? Yes. You can't use ReadFileEx because you can't use completion routines with completion ports (kinda makes sense when you think about it). > I have a feeling that implementing this will move the WinNamedPipeStream > API so far away from POSIX that I will have to create a new class that > doesn't extend IOStream for this purpose. While it might be a cleaner > solution to the current overlapped/nonoverlapped read branches in the > code, I'm worried about the duplication of code and possible maintenance > headaches. Well, I'm still trying to work out what you're hoping to achieve; the change talks about deadlock, but this is a pipe - that can't happen if you always check for errors. Obviously there needs to be a thread to deal with commands, but if there's only one thread there's no harm in using normal boring synchronous IO. The actual backup sockets ought to be using IOCP, but that's a completely different issue. However, it looks like you're using another worker thread as well in order to do the actual work. If there's only one connection to the command pipe, why is this thread needed? There's nothing to synchronise, so why not do it as part of the pipe handling? If you really need another thread, just create an IOCP, not bind anything to it, and PostQueuedCompletionStatus at it. Loop on GetQueuedCompletionStatus in the other thread and you're done. No critsec, events, or lists needed. IIRC the Win32 server part was just to allow the tests to run. This seems like a lot of work for that. Maybe I'm missing something? -C From chris at qwirx.com Mon Mar 26 21:03:14 2007 From: chris at qwirx.com (Chris Wilson) Date: Mon, 26 Mar 2007 21:03:14 +0100 (BST) Subject: [Box Backup-commit] Re: #3: Merge Win32 branch In-Reply-To: <46070956.3010206@invis.net> References: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> <060.0aa2af72f8ac97b7448b75dcce34143d@fluffy.co.uk> <46070956.3010206@invis.net> Message-ID: Hi Charles, On Mon, 26 Mar 2007, Charles Lecklider wrote: >>> [1486] I don't know if I can do this, but I'm going to try :-) >>> >>> I'd like to veto this change. You seem to have done a partial re- >>> implementation of a completion port - see CreateIoCompletionPort and >>> friends. >> >> I assume that this means that I have to use overlapped I/O for both >> reading and writing from the pipe? > > Erm, you have to do that anyway or you'll get some really strange > behaviour. So far I haven't had any problems using overlapped I/O for read only. What kind of strange behaviour might I get? >> I have a feeling that implementing this will move the WinNamedPipeStream >> API so far away from POSIX that I will have to create a new class that >> doesn't extend IOStream for this purpose. While it might be a cleaner >> solution to the current overlapped/nonoverlapped read branches in the >> code, I'm worried about the duplication of code and possible maintenance >> headaches. > > Well, I'm still trying to work out what you're hoping to achieve; the > change talks about deadlock, but this is a pipe - that can't happen if > you always check for errors. No, you get deadlocks if you perform simultaneous operations on the pipe in multiple threads, e.g. Write() in one thread while there is ablocked Read() happening in the other. This used to happen to me quite often until I moved all the reads and writes to happen in the same thread, and never since. > Obviously there needs to be a thread to deal with commands, but if > there's only one thread there's no harm in using normal boring > synchronous IO. The actual backup sockets ought to be using IOCP, but > that's a completely different issue. The pipe thread can't use blocking I/O because it also has to listen for events from the main threads which signal that it should write to the pipe. I couldn't find a way to wait on both a filehandle and an event, and I didn't want to create a separate pipe just to communicate between threads, so I changed the pipe thread to use only non-blocking I/O and events. > However, it looks like you're using another worker thread as well in > order to do the actual work. If there's only one connection to the > command pipe, why is this thread needed? There's nothing to synchronise, > so why not do it as part of the pipe handling? No, there are only two threads, the main one and the pipe handler. > If you really need another thread, just create an IOCP, not bind > anything to it, and PostQueuedCompletionStatus at it. Loop on > GetQueuedCompletionStatus in the other thread and you're done. No > critsec, events, or lists needed. Does that still apply after my explanation above? > IIRC the Win32 server part was just to allow the tests to run. This > seems like a lot of work for that. Maybe I'm missing something? Not at all, all of this is on the client side (bbackupd). Nothing at all to do with the Win32 server, which doesn't use WinNamedPipeStream at all. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |