From larry at geekmax.org Fri Dec 2 14:53:16 2011 From: larry at geekmax.org (Larry Wimble) Date: Fri, 02 Dec 2011 09:53:16 -0500 Subject: [Box Backup] Concerns about housekeeping / deletion of old files Message-ID: <4ED8E65C.5060406@geekmax.org> Hello folks... I've been doing backups for the longest time with rsync over stunnel. This solution basically works, however, I want to move my backups offsite which means encryption (both in transport and in storage) is now a requirement. Moving backups offsite will also allow me to backup my web and application servers as well. Consequently, I've been researching new backup solutions. Boxbackup is the only open source solution that fits my requirements. It's reasonably light weight, doesn't depend on things like Java, Perl or Python, supports encrypted transport and storage out of the box, keeps bandwidth to a minimum, has it's own transport protocol (doesn't rely on scp/sftp/ftp, etc), doesn't try to apply the archaic concept of tapes to disks, supports Window$ clients, and keeps restoration simple and instant. I am absolutely blown away at the thought and work that has gone into Boxbackup. Huge kudos to the developers! As I was installing Boxbackup, I came across issue #62, "Backups of deleted files may be removed very quickly by housekeeping". Is there a workaround or perhaps any progress on making snapshots? Any recommendations on how one might eliminate the possibility of this happening with current code? If disk space is not an issue, does this problem come into play? Could one use a versioned filesystem to do the storage on as a means to temporarily backing up the backups? How about periodic automated snapshots through bbackupctl and cron ? Other ideas? Kind Regards, Larry Wimble From chris at qwirx.com Fri Dec 2 22:49:36 2011 From: chris at qwirx.com (Chris Wilson) Date: Fri, 2 Dec 2011 22:49:36 +0000 (GMT) Subject: [Box Backup] Concerns about housekeeping / deletion of old files In-Reply-To: <4ED8E65C.5060406@geekmax.org> References: <4ED8E65C.5060406@geekmax.org> Message-ID: Hi Larry, On Fri, 2 Dec 2011, Larry Wimble wrote: > Boxbackup is the only open source solution that fits my requirements. > It's reasonably light weight, doesn't depend on things like Java, Perl > or Python, supports encrypted transport and storage out of the box, > keeps bandwidth to a minimum, has it's own transport protocol (doesn't > rely on scp/sftp/ftp, etc), doesn't try to apply the archaic concept of > tapes to disks, supports Window$ clients, and keeps restoration simple > and instant. I am absolutely blown away at the thought and work that > has gone into Boxbackup. Huge kudos to the developers! Thanks, that's also why I'm interested in Box Backup and keep working on it, and I'm glad that these features appeal to you too. > As I was installing Boxbackup, I came across issue #62, "Backups of deleted > files may be removed very quickly by housekeeping". Is there a workaround or > perhaps any progress on making snapshots? Unfortunately not yet. It's a major undertaking and one which I've been thinking about for the longest time, but simply not had the time to invest to implement such a major reworking. > Any recommendations on how one might eliminate the possibility of this > happening with current code? It's possible to set the soft and hard limits equal, which means that old and deleted files will never be removed. This means that when the store is full, backups will simply stop uploading new data. One should monitor the store and allocate more space to the account when this happens. > If disk space is not an issue, does this problem come into play? I don't believe so, if you do the above. > Could one use a versioned filesystem to do the storage on as a means to > temporarily backing up the backups? I think that is possible in theory. Users' attempts to run Box Backup against various versioned filesystems (notably S3 via fuse) have so far not been successful. I think this is because Box Backup relies on POSIX semantics in the filesystem, which versioned filesystems tend to fail to achieve in various ways. > How about periodic automated snapshots through bbackupctl and cron ? Unfortunately these are not real snapshots. The only difference between a bbackupctl snapshot and automatic backups is that with these "snapshots" one controls exactly when the backup takes place. It makes no difference to the store, and hence the maintenance of old versions, which is currently controlled by the store. > Other ideas? LVM snapshots of the destination filesystem? Unfortunately LVM snapshots have a high overhead in disk space and performance which precludes maintaining many of them simultaneously. ZFS/btrfs/BSD UFS snapshots are another possibility which might have lower overhead. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From maddog at mir.com Fri Dec 2 23:10:58 2011 From: maddog at mir.com (Matt Marjanovic) Date: Fri, 02 Dec 2011 15:10:58 -0800 Subject: [Box Backup] Concerns about housekeeping / deletion of old files In-Reply-To: References: <4ED8E65C.5060406@geekmax.org> Message-ID: <4ED95B02.1000501@mir.com> On 12/02/2011 02:49 PM, Chris Wilson wrote: >> As I was installing Boxbackup, I came across issue #62, "Backups of >> deleted files may be removed very quickly by housekeeping". Is there >> a workaround or perhaps any progress on making snapshots? > > Unfortunately not yet. It's a major undertaking and one which I've been > thinking about for the longest time, but simply not had the time to > invest to implement such a major reworking. For those of us (e.g. me) whose interest is now piqued, could you give a quick explanation of the current housekeeping algorithm and its problems? (Or point me to a write-up, if it's already written in something other than C++ --- I know where to find that. :) -m From chris at qwirx.com Sat Dec 3 15:55:02 2011 From: chris at qwirx.com (Chris Wilson) Date: Sat, 3 Dec 2011 15:55:02 +0000 (GMT) Subject: [Box Backup] Concerns about housekeeping / deletion of old files In-Reply-To: <4ED95B02.1000501@mir.com> References: <4ED8E65C.5060406@geekmax.org> <4ED95B02.1000501@mir.com> Message-ID: Hi Matt, On Fri, 2 Dec 2011, Matt Marjanovic wrote: > On 12/02/2011 02:49 PM, Chris Wilson wrote: >>> As I was installing Boxbackup, I came across issue #62, "Backups of >>> deleted files may be removed very quickly by housekeeping". Is there >>> a workaround or perhaps any progress on making snapshots? >> >> Unfortunately not yet. It's a major undertaking and one which I've been >> thinking about for the longest time, but simply not had the time to >> invest to implement such a major reworking. > > For those of us (e.g. me) whose interest is now piqued, could you > give a quick explanation of the current housekeeping algorithm and > its problems? (Or point me to a write-up, if it's already written > in something other than C++ --- I know where to find that. :) Ben's statement, the most definitive source of information I have, as he wrote the code: http://lists.boxbackup.org/pipermail/boxbackup/2009-August/005463.html The rest of that discussion is most informative, starting around here: http://lists.boxbackup.org/pipermail/boxbackup/2009-August/005434.html Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Sat Dec 3 22:49:40 2011 From: chris at qwirx.com (Chris Wilson) Date: Sat, 3 Dec 2011 22:49:40 +0000 (GMT) Subject: [Box Backup] NTFS junction/reparse point In-Reply-To: <4EC4FA6D.3030401@qustodium.net> References: <8b5baa553589ce9d8ac220c887da95d9@localhost> <4EC4FA6D.3030401@qustodium.net> Message-ID: Hi Achim, On Thu, 17 Nov 2011, Achim J. Latz wrote: > On 24/10/2011 22:42, Chris Wilson wrote: > >> On recent Windows versions that have C:\Users, C:\Documents and Settings >> is just a reparse point that points to that folder, for backwards >> compatibility. C:\Users should be backed up normally, although some >> folders in user home directories are also reparse points for the same >> reason. Do you see it in your backups? > > I can see that the data is indeed backed within the "original" directory. I > am coming from an end-user perspective: > > 1) The user saved something important in "C:\Documents and > Settings\username\BIGDEAL.doc". If they are a Windows Vista/7 user then they should not be saving anything in "C:\Documents and Settings". That directory is not visible to them. They should not know it's there. As far as they are and should be concerned, it's C:\Users. PEBCAK. > Through NTFS junctions, the data is physically stored in > "c:\Users\username\BIGDEAL.doc", but the user does not know this, and > can easily open the file from within "Documents and Settings" for the > time being. How? I can't see that folder on Windows 7. >> If we backed up C:\Documents and Settings as well as C:\Users, I think >> you'd have two copies of the same data. > > I understand that, and I agree that uploading the same data again makes no > sense (unless there would be an amazing new deduplication feature, which > would eliminate all additional space requirements for the additional backups > of the same file, just like in Cumulus or BackupPC). I'm afraid it's not going to happen in a hurry. > Basic question: How could we "hang in" the corresponding data from the > NTFS junction points at all "links/mount points", so that the data is > only stored once, but the user sees it in all the places he is familiar > with? This is quite a Windows-specific "feature" for Box Backup to have. I'd prefer not to add it if it's at all possible to avoid it. If it's really essential, after considering the caveats above, perhaps we can either special-case (hard code) these directories in whatever client the user is using, or we can back up junction points (as Charles suggested) as a special object type that points back to the target of the junction point. Under Windows, all of a user's data (the stuff that Box Backup is designed to back up) should be under c:\users anyway, so the only location you should need is this: Location { Name = Documents and Settings Path = c:\users } which solves the problem in a different way. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From chris at qwirx.com Sat Dec 3 23:12:24 2011 From: chris at qwirx.com (Chris Wilson) Date: Sat, 3 Dec 2011 23:12:24 +0000 (GMT) Subject: [Box Backup] Regex exclusions In-Reply-To: <4EC50F77.30006@qustodium.net> References: <4E846D7F.1070507@qustodium.net> <4EC50F77.30006@qustodium.net> Message-ID: Hi Achim, On Thu, 17 Nov 2011, Achim J. Latz wrote: > On 29/09/2011 18:50, Peter Jalajas, GigaLock Backup Services wrote: > >> A few planning or philosophical topics we should consider: 1. Windows >> Exclusions are not case-sensitive, so we no longer have to add both >> upper and lower case versions of each Exclusion--yeah! > > I would assume after reading Chris' comments that this is true for all > exclusions, not Windows only. No, it's only true on Windows. It was also a bit broken until now, because it would also convert all characters in the regular expression to lower case, including \S to \s for example. I've just fixed that. The case-insensitive comparison flag is now used when compiling the regexes, on Windows only. >> (I wish we could push some common Exclusion strings into variables of >> some sort that different BackupLocations could "import". Exclusions >> are fraught with typographic errors.) > > Chris, would this perhaps be possible by adding additional "keys" in > bbackupd.conf like this (not very elegant, I know): > > ExcludeTemplate_1, ExcludeTemplate_2, [..], ExcludeTemplate_n > IncludeTemplate_1, IncludeTemplate_2, [..], IncludeTemplate_n > > that can be used within the regular ExcludeFilesRegex etc. like this: > > ExcludeFilesRegex = IncludeTemplate_2|IncludeTemplate_42|[...] It's possible, but quite a bit of work. How about global Include and Exclude options instead? > Some other ideas can be found here > , > particularly the part of defining an exclusion from a given file pattern > would be great for Boxi (although not a priority, as >90% of the users > will never touch exclusions). What do you mean by a file pattern? Is that a regex applied to the file name? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From pj08 at gigalock.com Mon Dec 12 02:14:05 2011 From: pj08 at gigalock.com (Pete Jalajas) Date: Sun, 11 Dec 2011 21:14:05 -0500 Subject: [Box Backup] Notes on trunk_3047 upgrade today Message-ID: Hi Chris and all, A few things from this evening's work while I watched the family decorate the solstice tree. I received via email today from Amazon a "Degraded hardware" warning on my AWS EC2 instance. I took the opportunity to upgrade to trunk_3047. Following: http://www.boxbackup.org/wiki/SourceCodeRepository I did: cd /tmp svn co https://www.boxbackup.org/svn/box/trunk/ cd trunk ./bootstrap && ./configure && make && echo "Finished." I couldn't make it compile because of the doc Errors, so, being fairly ignorant of these things, I tweaked the Makefile to take out the doc-related dependencies from this line: parcels/boxbackup-trunk_3047-backup-server-linux-gnu.tgz: parcels/boxbackup-trunk_3047-backup-server-linux-gnu/bbstored parcels/boxbackup-trunk_3047-backup-server-linux-gnu/bbstoreaccounts parcels/boxbackup-trunk_3047-backup-server-linux-gnu/bbstored-certs parcels/boxbackup-trunk_3047-backup-server-linux-gnu/bbstored-config parcels/boxbackup-trunk_3047-backup-server-linux-gnu/raidfile-config parcels/boxbackup-trunk_3047-backup-server-linux-gnu/bbstored.8.gz parcels/boxbackup-trunk_3047-backup-server-linux-gnu/bbstoreaccounts.8.gz parcels/boxbackup-trunk_3047-backup-server-linux-gnu/bbstored-certs.8.gz parcels/boxbackup-trunk_3047-backup-server-linux-gnu/bbstored-config.8.gz parcels/boxbackup-trunk_3047-backup-server-linux-gnu/raidfile-config.8.gz parcels/boxbackup-trunk_3047-backup-server-linux-gnu/bbstored.conf.5.gz parcels/boxbackup-trunk_3047-backup-server-linux-gnu/raidfile.conf.5.gz (and maybe a couple of other lines as well). Then I ran only: make build-backup-server # not bootstrap or configure; I only needed the server and it compiled quickly and fine. I then: 1096 cd /usr/local/sbin 1104 sudo 7z a bbstored_trunk_2818.7z bb* raid* 1115 sudo killall bbstored 1116 sudo rm bb* raid* 1118 sudo cp /tmp/trunk/parcels/boxbackup-trunk_3047-backup-server-linux-gnu/bb* /tmp/trunk/parcels/boxbackup-trunk_3047-backup-server-linux-gnu/raid* . 1121 sudo bbstored That all seemed to work. Maybe there's a better way? Another item. Here's a typical server-side log sequence: Dec 11 19:39:34 gl8 bbstored client=0x10008027[19468]: NOTICE: Login from Client ID 0x10008027 Read/Write Dec 11 19:39:43 gl8 bbstored client=0x10008027[19468]: NOTICE: Session finished for Client ID 0x10008027 Dec 11 19:39:43 gl8 bbstored client=0x10008027[19468]: NOTICE: Connection statistics for BACKUP-10008027: IN=514 OUT=36910 NET_IN=-36396 TOTAL=37424 $ find /bu/backup/10008027/ -mmin -500 -ls # slightly redacted 62652417 12 drwxr-xr-x 72 root root 12288 Dec 11 20:12 /bu/backup/10008027/ \62783495 28 -rw-r--r-- 1 root root 28009 Dec 11 19:39 /bu/backup/10008027/21/o2b.rfw 62658132 32 -rw-r--r-- 1 root root 30377 Dec 11 15:43 /bu/backup/10008027/ocf.rfw 62652421 76 -rw-r--r-- 1 root root 72088 Dec 11 15:38 /bu/backup/10008027/refcount.db.rfw 62783542 28 -rw-r--r-- 1 root root 28593 Dec 11 19:39 /bu/backup/10008027/11/oc7.rfw 62663035 4 -rw-r--r-- 1 root root 124 Dec 11 20:12 /bu/backup/10008027/info.rfw 62658127 28 -rw-r--r-- 1 root root 24697 Dec 11 15:43 /bu/backup/10008027/01/of3.rfw 62652418 0 -rw------- 1 root root 0 Dec 11 20:12 /bu/backup/10008027/write.lock So, at 19:39, it looks like 2 files were backed up with very small changes only (<514 bytes total, per "IN=514" in "Connection statistics" log entry above). Correct? Not sure what the OUT=36910 bytes represents (file listings? checksums?) What else would make up part of the "IN=" value? If "IN=" is greater than zero, does that mean at least one file was backed up? If the client connected at all in Read/Write mode, perhaps that means by definition at least one file needed to be backed up? Second to last-ly, looking at the source code: $ grep -nRi -e "Login from Client" -e mClientID -e mAccountName lib/backupstore/BackupCommands.cpp 89: if(mClientID != rContext.GetClientID()) 91: BOX_WARNING("Failed login from client ID " << 92: BOX_FORMAT_ACCOUNT(mClientID) << 99: BOX_WARNING("Failed login from client ID " << 100: BOX_FORMAT_ACCOUNT(mClientID) << 112: BOX_FORMAT_ACCOUNT(mClientID)); 130: BOX_NOTICE("Login from Client ID " << 131: BOX_FORMAT_ACCOUNT(mClientID) << and $ grep -nRi -A1 -e "Session finished for Client" lib/backupstore/BackupCommands.cpp 154: BOX_NOTICE("Session finished for Client ID " << 155- BOX_FORMAT_ACCOUNT(rContext.GetClientID())); In lib/backupstore/BackupCommands.cpp, it would be very nice to have the "mAccountName" (or at least the first 8 characters or so) listed in the "Login from Client ID " and "Session finished for " log records, so I don't have to grep mClientID in my little local lookup table every time. Similarly, and lastly: $ grep -nRi -e "Connection statistics" bin/bbstored/* bin/bbstored/BackupStoreDaemon.cpp:367: BOX_NOTICE("Connection statistics for " << commonName << ":" Not sure why this log record is in a separate code tree source file from the "Login from Client" and "Session finished for " entries above. Maybe it could be tweaked to match the other log entries, like BOX_FORMAT_ACCOUNT(mClientID), perhaps with "mAccountName" as well? Thanks! Pete PS: For extra credit, it would be nice to have the mClientID and mAccountName in this log record: bbstored[19139]: NOTICE: Message from child process 20111: Incoming connection from 24.62.185.131 port 39402 $ grep -nRi -C1 -e "Incoming connection from" lib/* lib/server/Socket.cpp-130- case AF_UNIX: lib/server/Socket.cpp:131: BOX_INFO("Incoming connection from local (UNIX socket)"); lib/server/Socket.cpp-132- break; -- lib/server/Socket.cpp-136- sockaddr_in *a = (sockaddr_in*)addr; lib/server/Socket.cpp:137: BOX_INFO("Incoming connection from " << lib/server/Socket.cpp-138- inet_ntoa(a->sin_addr) << " port " << -- lib/server/Socket.cpp-163- case AF_UNIX: lib/server/Socket.cpp:164: return std::string("Incoming connection from local (UNIX socket)"); lib/server/Socket.cpp-165- break; -- lib/server/Socket.cpp-170- sockaddr_in *a = (sockaddr_in*)addr; lib/server/Socket.cpp:171: sprintf(msg, "Incoming connection from %s port %d", inet_ntoa(a->sin_addr), ntohs(a->sin_port)); lib/server/Socket.cpp-172- return std::string(msg); ubuntu at gl8:/tmp/trunk$ But I'm guessing that that's because the server doesn't yet know which client it is... From chris at qwirx.com Tue Dec 13 00:44:09 2011 From: chris at qwirx.com (Chris Wilson) Date: Tue, 13 Dec 2011 00:44:09 +0000 (GMT) Subject: [Box Backup] Notes on trunk_3047 upgrade today In-Reply-To: References: Message-ID: Hi Pete, On Sun, 11 Dec 2011, Pete Jalajas wrote: > I couldn't make it compile because of the doc Errors, so, being fairly > ignorant of these things, I tweaked the Makefile to take out the > doc-related dependencies from this line: I tried to make a Makefile target that doesn't build the docs, but it's quite complicated because the install script also references them, and I'd prefer not to have two ways to build backup-server-linux-gnu.tgz, one with docs and one without, because it could lead to more confusion in future. Perhaps if you tell me the errors that you experienced, I can help you to fix them? It's usually just a case of installing xsltproc. > Dec 11 19:39:43 gl8 bbstored client=0x10008027[19468]: NOTICE: > Connection statistics for BACKUP-10008027: IN=514 OUT=36910 > NET_IN=-36396 TOTAL=37424 [...] > So, at 19:39, it looks like 2 files were backed up with very small > changes only (<514 bytes total, per "IN=514" in "Connection > statistics" log entry above). Correct? Not sure what the OUT=36910 > bytes represents (file listings? checksums?) What else would make up > part of the "IN=" value? If "IN=" is greater than zero, does that > mean at least one file was backed up? It just measures the total number of bytes sent by the client to the server. That would include commands and their arguments, as well as file data uploaded (complete and patched). Without more detailed logs, it's only possible to say that less than 514 bytes of changes were uploaded. You can enable LogAllFileAccess if you want logging with that amount of detail. > If the client connected at all in Read/Write mode, perhaps that means by > definition at least one file needed to be backed up? I think so, yes, provided that the client had a working state already in memory (either bbackupd hadn't been restarted since the last backup, or StoreObjectInfoFile was enabled and working). > In lib/backupstore/BackupCommands.cpp, it would be very nice to have the > "mAccountName" (or at least the first 8 characters or so) listed in the > "Login from Client ID " and "Session finished for " log records, so I > don't have to grep mClientID in my little local lookup table every time. Added. > Similarly, and lastly: > $ grep -nRi -e "Connection statistics" bin/bbstored/* > bin/bbstored/BackupStoreDaemon.cpp:367: BOX_NOTICE("Connection > statistics for " << commonName << ":" > > Not sure why this log record is in a separate code tree source file > from the "Login from Client" and "Session finished for " entries > above. Because the protocol command handlers in BackupCommands.cpp don't have direct access to the raw stream, so they can't see the byte counters; and the logging happens at the end of the connection, even if it was aborted by a network error, in which case no protocol command is executed at all. > Maybe it could be tweaked to match the other log entries, > like BOX_FORMAT_ACCOUNT(mClientID), perhaps with "mAccountName" as > well? Done. > PS: For extra credit, it would be nice to have the mClientID and > mAccountName in this log record: > bbstored[19139]: NOTICE: Message from child process 20111: Incoming > connection from 24.62.185.131 port 39402 Unfortunately it's too early. We don't know that information when the message is logged, because the client hasn't sent us anything yet, not even an SSL client certificate. I've added them to the post-login message instead. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From pjalajas at gigalock.com Tue Dec 13 16:03:01 2011 From: pjalajas at gigalock.com (Peter Jalajas, GigaLock Backup Services) Date: Tue, 13 Dec 2011 11:03:01 -0500 Subject: [Box Backup] Notes on trunk_3047 upgrade today In-Reply-To: References: Message-ID: Awesome, Chris. Thanks so much for the compiling info and log changes. I'll check to make sure I have xsltproc installed on my next upgrade and report back with any errors. Thanks again, Pete From pjalajas at gigalock.com Tue Dec 13 18:26:19 2011 From: pjalajas at gigalock.com (Peter Jalajas, GigaLock Backup Services) Date: Tue, 13 Dec 2011 13:26:19 -0500 Subject: [Box Backup] Notes on trunk_3047 upgrade today In-Reply-To: References: Message-ID: Hi Chris, Installing xsltproc fixed my compiling problem, thanks: 1251 cd /tmp 1252 cd trunk/ 1256 svn up # to 3049 1257 sudo apt-get install xsltproc 1258 ./bootstrap && ./configure && make && echo "Finished." 1259 cd /usr/local/sbin 1261 sudo 7z a bbstored_trunk_3047.7z bb* raid* #backing up old version 1266 sudo iftop -P Now I just need to wait for my clients to disconnect. It would be real nice if there was a way to instruct bbstored to no long accept incoming clients and let me know when the last one has disconnected (so that I can perform the upgrade without killing clients mid-backup which I'm guessing can be quite "expensive"). Maybe I can kill only the 2 initial bbstored processes and write a little script that let's me know when no more bbstored processes are running--would that work? I should update http://www.boxbackup.org/wiki/GeneralInstall (at least) with the xsltproc dependency (and the others), but I can't seem to login--can you help me with that (again, sorry). Thanks again, Pete On Tue, Dec 13, 2011 at 11:03 AM, Peter Jalajas, GigaLock Backup Services wrote: > Awesome, Chris. ?Thanks so much for the compiling info and log > changes. ? I'll check to make sure I have xsltproc installed on my > next upgrade and report back with any errors. > > Thanks again, > Pete From james at netinertia.co.uk Tue Dec 13 19:58:00 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Tue, 13 Dec 2011 19:58:00 +0000 Subject: [Box Backup] Notes on trunk_3047 upgrade today In-Reply-To: References: Message-ID: <20111213195800.GT18215@netinertia.co.uk> On Tue, Dec 13, 2011 at 12:44:09AM +0000, Chris Wilson wrote: > On Sun, 11 Dec 2011, Pete Jalajas wrote: > > > I couldn't make it compile because of the doc Errors, so, being fairly > > ignorant of these things, I tweaked the Makefile to take out the > > doc-related dependencies from this line: > > I tried to make a Makefile target that doesn't build the docs, but it's > quite complicated because the install script also references them, and I'd > prefer not to have two ways to build backup-server-linux-gnu.tgz, one with > docs and one without, because it could lead to more confusion in future. > Perhaps if you tell me the errors that you experienced, I can help you to > fix them? It's usually just a case of installing xsltproc. Ideally I think we should leave the SVN version so that docs are always built. My intention was (still is, really) to have the build automation upload a distribution tarball to the web server so that people can compile from that. This (hopefully) will already have the xsltproc stuff done and the docs built so the end user only needs to compile the software. Otherwise I think we should just clarify somewhere that xsltproc and the docbook stylesheets are required to build, along with openssl headers, zlib headers, etc. Speaking of the build automation, Chris you've probably noticed that it's not working at the moment. It turns out bitten is quite sensitive to network issues and the slaves kept dying or hanging at the slightest blip and it was starting to get very tedious manually fixing them. Bitten is nice, but I think it's a little inflexible for what I'm trying to do with it so I'm hoping to redo things using buildbot. A quick test seemed like it will be good. Haven't had time to get a full environment up and running yet but with luck things will be quiet enough between Christmas and New Year that I'll be able to give it a go. James From pjalajas at gigalock.com Tue Dec 13 20:50:37 2011 From: pjalajas at gigalock.com (Peter Jalajas, GigaLock Backup Services) Date: Tue, 13 Dec 2011 15:50:37 -0500 Subject: [Box Backup] Notes on trunk_3047 upgrade today In-Reply-To: References: Message-ID: Hi Chris, Your new changes seem to be working fine (3049), thanks! Here are the lightly redacted log entries: Dec 13 15:30:24 srv bbstored[8547]: NOTICE: Handling incoming connection from 76.x.x.x. port 3153 Dec 13 15:30:24 srv bbstored client=0x10009999[8547]: NOTICE: Login from Client ID 0x10009999 (name=GLName1): Read/Write from 76.25.193.238 port 3153 Dec 13 15:30:51 srv bbstored client=0x10009999[8547]: NOTICE: Session finished for Client ID 0x10009999 (name=GLName1) Dec 13 15:30:51 srv bbstored client=0x10009999[8547]: NOTICE: Connection statistics for 0x10009999 (name=GLName1): IN=273 OUT=34657 NET_IN=-34384 TOTAL=34930 See how the Client ID hex number is listed twice for each record? Should the one on the left be removed? I noticed this: Dec 13 14:32:39 srv bbstored[7569]: NOTICE: Box Backup Store Server vtrunk_3049, (c) Ben Summers and contributors 2003-2011 Dec 13 14:32:39 srv bbstored[7571]: NOTICE: Starting daemon, version: trunk_3047 Did I screw up my manual upgrade, or does the trunk number need to change somewhere in the code? (see 3047 vs 3049 above). (Not sure what "vtrunk" vs "trunk" means.) While I'm at it, I noticed this, too: Dec 13 11:51:56 gl8 bbstored/hk[19140]: WARNING: BlocksInDirectories changed from 9605 to 9604 Dec 13 12:37:44 gl8 bbstored/hk[19140]: WARNING: Reference count of object 0x7457 changed from 1 to 0 Dec 13 12:37:44 gl8 bbstored/hk[19140]: WARNING: Reference count of object 0x7458 changed from 1 to 0 Dec 13 12:37:44 gl8 bbstored/hk[19140]: WARNING: BlocksUsed changed from 8843593 to 8843591 Dec 13 12:37:44 gl8 bbstored/hk[19140]: WARNING: BlocksInOldFiles changed from 2 to 0 Dec 13 14:09:45 gl8 bbstored/hk[19140]: WARNING: Reference count of object 0x17fe8 changed from 1 to 0 Dec 13 14:09:45 gl8 bbstored/hk[19140]: WARNING: BlocksUsed changed from 16177218 to 16177217 Dec 13 14:09:45 gl8 bbstored/hk[19140]: WARNING: BlocksInDirectories changed from 16200 to 16199 Dec 13 14:26:13 gl8 bbstored/hk[19140]: NOTICE: Terminating daemon Low priority: Does it make sense to add the Client IDs/Names to the hk records? Do I care about these records? Should they be NOTICEs rather than WARNINGs? Should I submit this as a trac ticket (I'll need help logging in again)? Thanks again, Chris. Sorry to bother you with all this minutiae. Pete From chris at qwirx.com Thu Dec 15 10:39:14 2011 From: chris at qwirx.com (Chris Wilson) Date: Thu, 15 Dec 2011 10:39:14 +0000 (GMT) Subject: [Box Backup] Notes on trunk_3047 upgrade today In-Reply-To: <20111213195800.GT18215@netinertia.co.uk> References: <20111213195800.GT18215@netinertia.co.uk> Message-ID: Hi James, On Tue, 13 Dec 2011, James O'Gorman wrote: > Ideally I think we should leave the SVN version so that docs are always > built. My intention was (still is, really) to have the build automation > upload a distribution tarball to the web server so that people can > compile from that. This (hopefully) will already have the xsltproc stuff > done and the docs built so the end user only needs to compile the > software. > > Otherwise I think we should just clarify somewhere that xsltproc and the > docbook stylesheets are required to build, along with openssl headers, > zlib headers, etc. I'm not sure if we still need the docbook stylesheets. I bundled a version of some stylesheets with the code. > Speaking of the build automation, Chris you've probably noticed that > it's not working at the moment. It turns out bitten is quite sensitive > to network issues and the slaves kept dying or hanging at the slightest > blip and it was starting to get very tedious manually fixing them. > > Bitten is nice, but I think it's a little inflexible for what I'm trying > to do with it so I'm hoping to redo things using buildbot. A quick test > seemed like it will be good. Haven't had time to get a full environment > up and running yet but with luck things will be quiet enough between > Christmas and New Year that I'll be able to give it a go. Thanks James! Now I just need to get time to install a recent slowaris and fix the build issues on it. Unless you can give me access to the build slave that you're using? Cheers, Chris. From pjalajas at gigalock.com Thu Dec 15 14:12:36 2011 From: pjalajas at gigalock.com (Peter Jalajas, GigaLock Backup Services) Date: Thu, 15 Dec 2011 09:12:36 -0500 Subject: [Box Backup] Notes on trunk_3047 upgrade today In-Reply-To: References: Message-ID: Hi Chris, On Tue, Dec 13, 2011 at 1:26 PM, Peter Jalajas, GigaLock Backup Services wrote: > Hi Chris, > > Installing xsltproc fixed my compiling problem, thanks: > I should update http://www.boxbackup.org/wiki/GeneralInstall (at > least) with the xsltproc dependency (and the others), but I can't seem > to login--can you help me with that (again, sorry). Thanks for the updated trac login credentials via private message. Here is my list of installed apparent dependencies that suffice to build boxbackup on Ubuntu (I can build boxbackup with these, but I'm sure many are not needed or relevant): $ dpkg -l | grep -i "\(zlib\|openssl\|libz\|berkeley\|libeditline\|libreadline\|xsltproc\)" xii libcurl3 7.21.0-1ubuntu1.1 Multi-protocol file transfer library (OpenSSL) xii libdb-ruby1.8 0.6.5-6 Interface to Berkeley DB for Ruby 1.8 xii libdb4.7 4.7.25-9 Berkeley v4.7 Database Libraries [runtime] ii libdb4.8 4.8.30-1 Berkeley v4.8 Database Libraries [runtime] ii libdb4.8++ 4.8.30-1 Berkeley v4.8 Database Libraries for C++ [runtime] ii libdb4.8++-dev 4.8.30-1 Berkeley v4.8 Database Libraries for C++ [development] ii libdb4.8-dev 4.8.30-1 Berkeley v4.8 Database Libraries [development] xii libeditline-dev 1.12-6 development files for libeditline xii libeditline0 1.12-6 line editing library similar to readline ii libreadline-dev 6.1-3 GNU readline and history libraries, development files xii libreadline5 5.2-7build1 GNU readline and history libraries, run-time libraries ii libreadline6 6.1-3 GNU readline and history libraries, run-time libraries ii libreadline6-dev 6.1-3 GNU readline and history libraries, development files ii openssl 0.9.8o-1ubuntu4.4 Secure Socket Layer (SSL) binary and related cryptographic tools xii python-openssl 0.10-1 Python wrapper around the OpenSSL library ii xsltproc 1.1.26-6 XSLT 1.0 command line processor ii zlib1g 1:1.2.3.4.dfsg-3ubuntu1 compression library - runtime ii zlib1g-dev 1:1.2.3.4.dfsg-3ubuntu1 compression library - development ii zlibc 0.9k-4.1 An on-fly auto-uncompressing C library To be complete, perhaps did I not include a real dependency in my grep? (As you can tell, I usually just flail around with "apt-get install" until I can get the dang thing to build). Can someone please help me trim this down? I put an "x" in the first column of those I think we probably don't need, but I'm surely wrong on some. After confirmation from others, I'll upate http://www.boxbackup.org/wiki/GeneralInstall with the final tightened-up list. Thanks, Pete From achim+box at qustodium.net Thu Dec 15 23:27:18 2011 From: achim+box at qustodium.net (Achim) Date: Fri, 16 Dec 2011 00:27:18 +0100 Subject: [Box Backup] bbackupquery Message-ID: Hello list: Is it possible that the output format of bbackupquery has slightly changed, or perhaps I never realised it? Have a look at the following output: ----- snip ----- $ bbackupquery NOTICE: Box Backup Query Tool vtrunk_3049, (c) Ben Summers and contributors 2003-2011 Login complete. Type "help" for a list of commands. query > list -t 0001203f -d---- 1970-01-01T01:00:00 Users ----- snap ----- Please note the two spaces in front of "Users", whereas all the other fields are separated by only one space. Is this a side-effect or has it always been that way? A couple of my scripts seemed to have relied on a single space, and now there are two. Could this be a micro regression? Thanks, Achim From chris at qwirx.com Sun Dec 18 11:02:48 2011 From: chris at qwirx.com (Chris Wilson) Date: Sun, 18 Dec 2011 11:02:48 +0000 (GMT) Subject: [Box Backup] bbackupquery In-Reply-To: References: Message-ID: Hi Achim, On Fri, 16 Dec 2011, Achim wrote: > Is it possible that the output format of bbackupquery has slightly > changed, or perhaps I never realised it? Yes, it did change, in revision 2636, with a new micro-feature: displaying whether the file has new attributes uploaded, in which case a "*" is shown after the filename instead of a space. The second space is there to keep the columns lined up and still separated if no "*" is displayed. > Please note the two spaces in front of "Users", whereas all the other > fields are separated by only one space. > > Is this a side-effect or has it always been that way? A couple of my > scripts seemed to have relied on a single space, and now there are two. > > Could this be a micro regression? Is it actually a problem? What version did you upgrade from? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From james at netinertia.co.uk Sun Dec 18 12:12:14 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Sun, 18 Dec 2011 12:12:14 +0000 Subject: [Box Backup] Notes on trunk_3047 upgrade today In-Reply-To: References: <20111213195800.GT18215@netinertia.co.uk> Message-ID: <20111218121214.GV18215@netinertia.co.uk> On Thu, Dec 15, 2011 at 10:39:14AM +0000, Chris Wilson wrote: > I'm not sure if we still need the docbook stylesheets. I bundled a version > of some stylesheets with the code. Oh yes, I forgot. Probably still need to add the requirement for xsltproc somewhere. Do we have a definitive list of build requirements anywhere? > Thanks James! Now I just need to get time to install a recent slowaris > and fix the build issues on it. Unless you can give me access to the build > slave that you're using? I can give you access to the _Solaris_ host, yes. I'll create an account for you on all the build slaves today and pop that over to you. I still haven't decided what I'm going to do with the build infrastructure yet so it might all get rebuilt sometime. At the moment they're all KVM virtual machines on a single box so if all the slaves kick off at once it can cause some I/O issues :-) James From james at netinertia.co.uk Sun Dec 18 12:19:54 2011 From: james at netinertia.co.uk (James O'Gorman) Date: Sun, 18 Dec 2011 12:19:54 +0000 Subject: [Box Backup] Notes on trunk_3047 upgrade today In-Reply-To: References: Message-ID: <20111218121953.GW18215@netinertia.co.uk> On Thu, Dec 15, 2011 at 09:12:36AM -0500, Peter Jalajas, GigaLock Backup Services wrote: > Here is my list of installed apparent dependencies that suffice to > build boxbackup on Ubuntu (I can build boxbackup with these, but I'm > sure many are not needed or relevant): [snip] > To be complete, perhaps did I not include a real dependency in my > grep? (As you can tell, I usually just flail around with "apt-get > install" until I can get the dang thing to build). Can someone please > help me trim this down? I put an "x" in the first column of those I > think we probably don't need, but I'm surely wrong on some. After > confirmation from others, I'll upate > http://www.boxbackup.org/wiki/GeneralInstall with the final > tightened-up list. If it helps, the build automation script checks for the following commands: svn autoconf automake gcc g++ xsltproc Also required are LWP::UserAgent (from Perl's libwww), OpenSSL headers and zlib headers. Somewhere I had a file which told me which packages to use on Debian and CentOS but I can't find it at the moment. Hopefully this list should help though. James