From achim+box at qustodium.net Mon Mar 1 13:20:14 2010 From: achim+box at qustodium.net (Achim) Date: Mon, 01 Mar 2010 14:20:14 +0100 Subject: [Box Backup] Data corruption bug in trunk > 2585 and 0.11rc6 In-Reply-To: <4B05AD1C.7030101@hp-factory.de> References: <44db1ae754eec516d1ab1c63fb49a3be@localhost> <4B05AD1C.7030101@hp-factory.de> Message-ID: Hello Chris: On Sun, 28 Feb 2010 18:04:48 +0100 (CET), Chris Wilson wrote: > I've just discovered and fixed a potentially serious data corruption bug > on the store server (bbstored) trunk revisions since 2585 (dated > 11/11/09, ironically enough) and the tagged but unannounced 0.11rc6. > The client is not affected. Thanks to you and James for catching this! With regard to SVN revisions, RCs etc, there is one observation I wanted to share with you/the list: I think that the fact that some people are willing to run SVN versions of BB server can be explained by (at least) two reasons: 1) BB is very stable and reliable, and developed with so much care that SVN is no risk at all 2) People need features that are introduced in SVN before they become available as a official release The combination of 1) and 2) would lead me to think that proper releases should happen a lot more frequently than at the moment. I am not trying to push for a "1.0" version, but right now the situation is for instance that * The last official "stable release" 0.10 was released more than FOUR years ago, on 23 February 2006 [0] * As a result, for instance the upcoming Ubuntu LTS release 10.4 "Lucid Lynx" will include the package "0.11~rc3~r2502-2" [1]. I believe it would make a lot of sense to increase the frequency of official releases. This should also get more adopters on board who might be comfortable downloading a binary but do not want to compile their own latest snapshots from SVN. I think the latter aspect is something that we should not underestimate, additional people and eyeballs will deliver good suggestions for functionality, stability and UI. Best regards, and thanks again for your watchful eyes! Achim [0] [1] From chris at qwirx.com Mon Mar 1 13:38:49 2010 From: chris at qwirx.com (Chris Wilson) Date: Mon, 1 Mar 2010 14:38:49 +0100 (CET) Subject: [Box Backup] Data corruption bug in trunk > 2585 and 0.11rc6 In-Reply-To: References: <44db1ae754eec516d1ab1c63fb49a3be@localhost> <4B05AD1C.7030101@hp-factory.de> Message-ID: Hi Achim, On Mon, 1 Mar 2010, Achim wrote: > 1) BB is very stable and reliable, and developed with so much care that > SVN is no risk at all > 2) People need features that are introduced in SVN before they become > available as a official release > > The combination of 1) and 2) would lead me to think that proper releases > should happen a lot more frequently than at the moment. I am not trying to > push for a "1.0" version, but right now the situation is for instance that I tend to agree, but I have reasons for not having released 0.11 yet: * I'm getting very little feedback about release candidates (and admit to not having announced 0.11rc6, which was probably a good thing in retrospect). * I haven't had time to test recent versions myself, on the platforms that I have virtual machines for. * My home machine has blown up, so until I get back to the UK I have no access to my solaris VM, for example. * Some tests are currently failing on Windows and I haven't had time to investigate and fix them. > This should also get more adopters on board who might be comfortable > downloading a binary but do not want to compile their own latest > snapshots from SVN. There are release candidates for 0.11, and running them is highly recommended, especially as 0.10 no longer compiles on modern platforms. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From achim+box at qustodium.net Wed Mar 10 16:05:13 2010 From: achim+box at qustodium.net (Achim) Date: Wed, 10 Mar 2010 17:05:13 +0100 Subject: [Box Backup] 0.11RC7, attributes, ACLs under Windows Message-ID: <9f0ca8c730246d143ad63c42709b4d60@localhost> Hello list: Chris just released 0.11RC7, and it compiles perfectly under MingW: thanks so much Chris[0]! Shout if anyone wants to testdrive RC7 and I will put up my binaries. On a related note, I wanted to come back to the topic of attributes and ACLs under Windows. Inspired by an article on how to use BackupPC for a "Bare Metal Restore" (BRM) of Windows XP [1], I think we should revisit the capability to backup and restore attributes and ACLs under Windows. If you have a brief look at the described process, you can see the role those two play if you want to have any chance of BRM. I filed two tickets [2, 3] with some additional information, and it would be great to hear the list's opinions on this, and perhaps give Chris a hand in this. Thanks again for a wonderful project, Achim [0] Apart from two small issues: During "make test" several times "Unable to send message to Event Log (Event Log is full):" shows up and bbackupquery.exe still has the old dates "2003-2008" NOTICE: Box Backup Query Tool v0.11rc7, (c) Ben Summers and contributors 2003-2008 [1] [2] Some file attributes under Windows are not backed up [3] Windows ACL support From chris at qwirx.com Wed Mar 10 16:40:36 2010 From: chris at qwirx.com (Chris Wilson) Date: Wed, 10 Mar 2010 17:40:36 +0100 (CET) Subject: [Box Backup] 0.11RC7, attributes, ACLs under Windows In-Reply-To: <9f0ca8c730246d143ad63c42709b4d60@localhost> References: <9f0ca8c730246d143ad63c42709b4d60@localhost> Message-ID: Hi Achim, On Wed, 10 Mar 2010, Achim wrote: > [0] Apart from two small issues: > During "make test" several times "Unable to send message to Event Log > (Event Log is full):" shows up This is not really Box Backup's fault except that it generates a lot of Event Log messages, and you may need to reconfigure your Event Log to handle them. I think users would be unpleasantly surprised if we did that for them. > and bbackupquery.exe still has the old dates "2003-2008" > NOTICE: Box Backup Query Tool v0.11rc7, (c) Ben Summers and contributors > 2003-2008 Thanks for reminding me, will fix. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From jwark at eastlink.ca Wed Mar 10 18:35:15 2010 From: jwark at eastlink.ca (Jack Warkentin) Date: Wed, 10 Mar 2010 14:35:15 -0400 Subject: [Box Backup] Permissions prevent bbstored starting during bootup, but not later from root command line Message-ID: <4B97E663.3060307@eastlink.ca> Hi Everybody Here is the problem in a nutshell. bbstored fails to start at bootup but can easily be started by root from a command line. Excerpt from syslog showing failure to start at bootup: Mar 10 13:09:59 JaxToyVA named[1576]: running Mar 10 13:10:01 JaxToyVA Box Backup (bbackupd)[1614]: NOTICE: Starting daemon, version 0.11rc2, config: /etc/boxbackup/bbackupd.conf Mar 10 13:10:01 JaxToyVA Box Backup (bbackupd)[1614]: NOTICE: Beginning scan of local files Mar 10 13:10:01 JaxToyVA Box Backup (bbackupd)[1614]: ERROR: Failed to connect to socket (type 1, name JaxToyVA, port 2201): Connection refused (111) Mar 10 13:10:01 JaxToyVA Box Backup (bbackupd)[1614]: WARNING: Exception thrown: ConnectionException(Conn_SocketConnectError) at SocketStream.cpp(223) Mar 10 13:10:01 JaxToyVA Box Backup (bbackupd)[1614]: NOTICE: About to notify administrator about event backup-error, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-error' Mar 10 13:10:01 JaxToyVA Box Backup (bbstored)[1627]: NOTICE: Starting daemon, version 0.11rc2, config: /etc/boxbackup/bbstored.conf Mar 10 13:10:01 JaxToyVA Box Backup (bbstored)[1627]: ERROR: SSL error during Load certificates: error:0200100D:system library:fopen:Permission denied Mar 10 13:10:01 JaxToyVA Box Backup (bbstored)[1627]: ERROR: SSL error during Load certificates: error:20074002:BIO routines:FILE_CTRL:system lib Mar 10 13:10:01 JaxToyVA Box Backup (bbstored)[1627]: ERROR: SSL error during Load certificates: error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib Mar 10 13:10:01 JaxToyVA Box Backup (bbstored)[1627]: WARNING: Exception thrown: ServerException(TLSLoadCertificatesFailed) at TLSContext.cpp(118) Mar 10 13:10:01 JaxToyVA Box Backup (bbstored)[1627]: FATAL: Terminating due to exception Server TLSLoadCertificatesFailed (3/25) Mar 10 13:10:01 JaxToyVA bbstored/hk[1629]: NOTICE: Terminating daemon Mar 10 13:10:02 JaxToyVA Box Backup (bbackupd)[1614]: ERROR: Exception caught (Connection SocketConnectError (Probably a network issue between client and server, bad hostname, or server not running.) 7/15), reset state and waiting to retry... Mar 10 13:10:03 JaxToyVA kernel: [ 32.652008] eth0: no IPv6 routers present Excerpt from CLI showing startup from command line as root: $ ps | grep -Fi -e stor root 286 2 [usb-storage] root 2029 1962 hald-addon-storage: polling /dev/hdc (every 2 sec) root 2030 1962 hald-addon-storage: polling /dev/hdd (every 2 sec) root 2046 1962 hald-addon-storage: no polling on /dev/fd0 because it is explicitly disabled jcw 2763 2711 grep -Fi -e stor $ su Password: # cd /etc/init.d # ls boxbackup-* boxbackup-client* boxbackup-server* # ./boxbackup-server start Starting boxbackup-server: bbstored. # NOTICE: Starting daemon, version 0.11rc2, config: /etc/boxbackup/bbstored.conf # ps | grep -Fi -e bbstored bbstored 2931 1 /usr/sbin/bbstored /etc/boxbackup/bbstored.conf bbstored 2932 2931 /usr/sbin/bbstored /etc/boxbackup/bbstored.conf root 2991 2826 grep -Fi -e bbstored # cd /var/log # ls -tr syslog* syslog.6.gz syslog.5.gz syslog.4.gz syslog.3.gz syslog.2.gz syslog.1.gz syslog.0 syslog # less syslog syslog.0 Later excerpt from syslog showing daemon now started successfully: Mar 10 13:21:52 JaxToyVA Box Backup (bbackupd)[1614]: WARNING: Suppressing duplicate notification about backup -error Mar 10 13:21:52 JaxToyVA Box Backup (bbackupd)[1614]: ERROR: Exception caught (Connection SocketConnectError ( Probably a network issue between client and server, bad hostname, or server not running.) 7/15), reset state a nd waiting to retry... Mar 10 13:22:02 JaxToyVA Box Backup (bbackupd)[1614]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 Mar 10 13:22:47 JaxToyVA Box Backup (bbstored)[2931]: NOTICE: Starting daemon, version 0.11rc2, config: /etc/boxbackup/bbstored.conf Mar 10 13:23:53 JaxToyVA Box Backup (bbackupd)[1614]: NOTICE: Beginning scan of local files Mar 10 13:23:53 JaxToyVA Box Backup (bbstored)[2931]: WARNING: Message from child process 2951: Incoming connection from 127.0.1.1 port 33715 Mar 10 13:23:53 JaxToyVA Box Backup (bbstored)[2951]: NOTICE: Login from Client ID 0x00000001 Read/Write Mar 10 13:23:53 JaxToyVA Box Backup (bbstored)[2951]: Sending stream, size 304 Mar 10 13:23:53 JaxToyVA Box Backup (bbackupd)[1614]: Receiving stream, size 304 My system is pretty simple. It is a home desktop running Debian Squeeze with the Box Backup server and client both running on the same machine. $ dpkg-query -l 'boxbackup*' Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Description +++-=====================-=====================-========================================================== ii boxbackup-client 0.11~rc2-6 client for the BoxBackup remote backup system ii boxbackup-server 0.11~rc2-6 server for the BoxBackup remote backup system I didn't realize until later that I should have installed the server first and got it working properly before installing the client, so it took a while to get everything installed properly. As a result I had to copy and edit (as root) several files in the /etc/boxbackup directory and subdirectories. Since my root umask is set to 027 these files have mostly -rw-r----- permissions, and root ownership. Another CLI excerpt: # cd /etc/boxbackup # find . -exec ls -adl \{\} \; drwxr-xr-x 4 root root 4096 2010-03-08 19:45 . drwxr-xr-x 2 root root 4096 2010-01-31 15:08 ./bbackupd -rwx------ 1 root root 1721 2010-01-31 15:08 ./bbackupd/NotifySysadmin.sh -rw-r----- 1 root root 1024 2010-01-31 14:02 ./bbackupd/1-FileEncKeys.raw -rwxr-xr-x 1 root root 1107 2009-05-09 17:14 ./bbackupd/notifyadmin -rw-r----- 1 root root 985 2010-01-31 14:27 ./bbackupd/1-cert.pem -rw-r----- 1 root root 1021 2010-01-28 14:59 ./bbackupd/serverCA.pem -rw-r----- 1 root root 891 2010-01-31 14:02 ./bbackupd/1-csr.pem -rw-r----- 1 root root 1675 2010-01-31 14:02 ./bbackupd/1-key.pem -rw------- 1 root root 1024 2009-05-09 17:14 ./bbackupd/boxbackup-client-encrypt-key.raw -rw-r--r-- 1 root root 16875 2010-03-08 19:45 ./bbackupd.conf -rw-r--r-- 1 root root 579 2010-01-31 15:26 ./bbstored.conf drwx------ 6 bbstored bbstored 4096 2010-01-28 15:51 ./bbstored -rw-r----- 1 root root 985 2010-01-28 15:30 ./bbstored/JaxToyVA-cert.pem -rw-r--r-- 1 root root 891 2009-05-09 17:14 ./bbstored/boxbackup-server-cert-req.pem -rw-r----- 1 root root 1021 2010-01-28 14:59 ./bbstored/clientCA.pem -rw-r----- 1 root root 4 2010-01-31 12:16 ./bbstored/accounts.txt -rw-r----- 1 root root 891 2010-01-28 14:28 ./bbstored/JaxToyVA-csr.pem -rw-r----- 1 root root 1675 2010-01-28 14:28 ./bbstored/JaxToyVA-key.pem -rw------- 1 bbstored bbstored 0 2009-05-09 17:14 ./bbstored/boxbackup-server-accounts.txt -rw-r----- 1 root root 891 2010-01-14 16:50 ./bbstored/localhost-csr.pem drwx------ 2 root root 4096 2010-01-28 14:59 ./bbstored/roots -rw-r----- 1 root root 1021 2010-01-28 14:59 ./bbstored/roots/clientCA.pem -rw-r----- 1 root root 3 2010-01-31 14:27 ./bbstored/roots/clientCA.srl -rw-r----- 1 root root 1021 2010-01-28 14:59 ./bbstored/roots/serverCA.pem -rw-r----- 1 root root 3 2010-01-28 15:30 ./bbstored/roots/serverCA.srl drwx------ 2 root root 4096 2010-01-28 15:30 ./bbstored/servers -rw-r----- 1 root root 985 2010-01-28 15:30 ./bbstored/servers/JaxToyVA-cert.pem drwx------ 2 root root 4096 2010-01-31 14:27 ./bbstored/clients -rw-r----- 1 root root 985 2010-01-31 14:27 ./bbstored/clients/1-cert.pem drwx------ 2 root root 4096 2010-01-28 14:59 ./bbstored/keys -rw-r----- 1 root root 911 2010-01-28 14:59 ./bbstored/keys/serverRootCSR.pem -rw-r----- 1 root root 1679 2010-01-28 14:59 ./bbstored/keys/serverRootKey.pem -rw-r----- 1 root root 1679 2010-01-28 14:59 ./bbstored/keys/clientRootKey.pem -rw-r----- 1 root root 911 2010-01-28 14:59 ./bbstored/keys/clientRootCSR.pem -rw-r----- 1 root root 1675 2010-01-14 16:50 ./bbstored/localhost-key.pem -rw------- 1 bbstored bbstored 1675 2009-05-09 17:14 ./bbstored/boxbackup-server-key.pem -rw-r--r-- 1 root root 262 2010-01-14 16:39 ./raidfile.conf I feel pretty sure that my problem is that of permissions/ownership of one of the key files. But I don't know which one and am reluctant to make changes willy-nilly until I get it correct. Any and all suggestions would be most appreciated. Regards Jack Jack Warkentin, phone 902-404-0457, email jwark at eastlink.ca 39 Inverness Avenue, Halifax, Nova Scotia, Canada, B3P 1X6 From james at netinertia.co.uk Wed Mar 10 22:13:09 2010 From: james at netinertia.co.uk (James O'Gorman) Date: Wed, 10 Mar 2010 22:13:09 +0000 Subject: [Box Backup] Bare Metal Recovery (was Re: [Box Backup-commit] #66: Windows ACL support) In-Reply-To: <042.d7d15c842e4274edcc96b15ea0cce237@boxbackup.org> References: <042.d7d15c842e4274edcc96b15ea0cce237@boxbackup.org> Message-ID: <92196DEF-15D5-46A4-896B-D862E5968438@netinertia.co.uk> On 10 Mar 2010, at 15:55, Box Backup wrote: > #66: Windows ACL support > -------------------------+-------------------------------------------------- > Reporter: achim | Owner: ben > Type: enhancement | Status: new > Priority: normal | Milestone: > Component: bbackupd | Version: 0.11rc2 > Keywords: Windows ACL | > -------------------------+-------------------------------------------------- > This would be great, especially if we want to start making BB "bare-metal > restore" capable. I'm really not sure about this. This has never been one of the aims of Box Backup. "Bare metal" implies that you could boot off a USB key and restore the *entire* system from Box Backup. This isn't possible currently, and I'm not sure it's even on the long-term plans (maybe Chris can give a definitive answer on that). A bare metal restore would either need to be some kind of disk image, or the backup system would need to know about partitions, bootloaders etc; at the moment it only knows about whichever directories you tell it about. I suppose it might be possible to create some kind of kickstart/jumpstart config that reinstalls the OS and the Box Backup client and automatically starts doing some kind of full restore (because you all have backups of your raw keys, don't you :-). I'm not sure that could be an "official feature" of Box Backup itself, but perhaps an FAQ. James From achim+box at qustodium.net Thu Mar 11 00:16:33 2010 From: achim+box at qustodium.net (Achim) Date: Thu, 11 Mar 2010 01:16:33 +0100 Subject: [Box Backup] Bare Metal Recovery (was Re: [Box Backup-commit] #66: Windows ACL support) In-Reply-To: <92196DEF-15D5-46A4-896B-D862E5968438@netinertia.co.uk> References: <042.d7d15c842e4274edcc96b15ea0cce237@boxbackup.org> <92196DEF-15D5-46A4-896B-D862E5968438@netinertia.co.uk> Message-ID: <4B983661.3030800@qustodium.net> Hello James: On 10/03/2010 23:13, James O'Gorman wrote: > On 10 Mar 2010, at 15:55, Box Backup wrote: > >> #66: Windows ACL support [..] This would be great, especially if we >> want to start making BB "bare-metal restore" capable. > > I'm really not sure about this. This has never been one of the aims > of Box Backup. "Bare metal" implies that you could boot off a USB key > and restore the *entire* system from Box Backup. This isn't possible > currently, and I'm not sure it's even on the long-term plans (maybe > Chris can give a definitive answer on that). The author of the original article I pointed out [1] earlier today uses "the term "bare metal" in the sense that when the process is started, there is no software or operating system installed. In other words, [he] is talking about the system being bare metal, not [BB] being a "bare metal restore" imaging program." [2] My potential strategy would be to use Clonezilla or Mondo Resuce to make a backup of the underlying OS at an early point, then do incremental backups with Box Backup. At the day of disaster, you restore your Clonezilla/Mondo Resuce backup (with all the correct device drives, registration info etc), and then restore over it the latest BB backup. Since BB at some point will be VSS and ACL and ATTRIB compatible, this should result in a very usable system that can be up an running in a day or less (-: > A bare metal restore would either need to be some kind of disk image, > or the backup system would need to know about partitions, bootloaders > etc; at the moment it only knows about whichever directories you tell > it about. Hence the combined Clone/Mondo/BB plan: BB is only a part of the recovery, but critical to be able to restore *all* changes after the initial OS backup. > I suppose it might be possible to create some kind of > kickstart/jumpstart config that reinstalls the OS and the Box Backup > client and automatically starts doing some kind of full restore > (because you all have backups of your raw keys, don't you :-). I'm > not sure that could be an "official feature" of Box Backup itself, > but perhaps an FAQ. Absolutely, this would always be a last resort for the fearless few, not an advertised feature. Again, [1] does a nice job explaining in how you would go about this, only that he reinstalls Windows and uses the Windows Recovery Console and Bacula, whereas I would suggest Clone/Mondo and a lightweight BB client to do the restore. This could all be packed into a neat bootable ISO for CD/DVD/USB. Backups of raw keys is another topic where some additional thinking could be warranted: how to automate the backup of the secret keys? Encrypt them with another passphrase and store them in a "side account" to every BB account, so that you only need to know the passphrase and always have access to at least one copy of your "secret" key? Best regards, Achim [1] [2] From stuart at bmsi.com Thu Mar 11 21:13:51 2010 From: stuart at bmsi.com (Stuart D. Gathman) Date: Thu, 11 Mar 2010 16:13:51 -0500 (EST) Subject: [Box Backup] Bare Metal Recovery (was Re: [Box Backup-commit] #66: Windows ACL support) In-Reply-To: <92196DEF-15D5-46A4-896B-D862E5968438@netinertia.co.uk> References: <042.d7d15c842e4274edcc96b15ea0cce237@boxbackup.org> <92196DEF-15D5-46A4-896B-D862E5968438@netinertia.co.uk> Message-ID: On Wed, 10 Mar 2010, James O'Gorman wrote: > I'm really not sure about this. This has never been one of the aims of Box > Backup. "Bare metal" implies that you could boot off a USB key and restore > the *entire* system from Box Backup. This isn't possible currently, and I'm > not sure it's even on the long-term plans (maybe Chris can give a definitive > answer on that). Bootable backup should be a separate project from the actual backup. For instance, 10 years ago I used an open source project called 'mindi', which created a bootable linux CD that ran an install script to partition the disk the same way as the original system, then load a backup (a tape backup at that time). This was a very good system - actually, it was *too* good. We had a customer that called to complain that all their files had reverted to last night. It turns out that they had left the bootable CD in the drive, and there was a power failure, and when the system came back up, it had the little menu asking for permission to restore the tape backup currently in the tape drive and overwrite the disk. Customers are like toddlers, they always answer "Yes" to any Yes/No prompt. (They also complained that the system took over an hour to "boot".) I have side stepped this need at present by having customers keep a live CD that lets them set a password and enable incoming SSH. Standard CentOS live CDs do that for you. Once I'm in on SSH, I can then partition, LVM, etc and restore backup. If I had to do a 'mindi' like system again, I would likely base it on Anaconda: basically, a customized install CD. -- Stuart D. Gathman Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial. From brendon at netcal.com Thu Mar 11 22:10:51 2010 From: brendon at netcal.com (Brendon Baumgartner) Date: Thu, 11 Mar 2010 14:10:51 -0800 Subject: [Box Backup] Win32 OS usage vs Box Backup usage discrepency Message-ID: <5DB1D2D8020D4449AF3111C784529E627185FE@jeep.netcal.l> Box Backup has a major discrepancy between what the OS reports and what it reports. The OS says 66 GB for the data that is specified to be backed up and the server reports 100 GB usage. Server version: 2587 Client version: 2520 Client disk usage according to the OS: 66.54 GB Client disk usage according to box backup: query > usage -m Used: 104846664 kB, 99% Old files: 0 kB, 0% Deleted files: 0 kB, 0% Directories: 54444 kB, 0% Soft limit: 96468992 kB, 92% Hard limit: 104857600 kB, 100% -Brendon From joseyluis at gmail.com Wed Mar 17 00:19:36 2010 From: joseyluis at gmail.com (Jose Luis Rodriguez Garcia) Date: Wed, 17 Mar 2010 01:19:36 +0100 Subject: [Box Backup] How to contribute to the project and two questions Message-ID: <26a0d8261003161719m26c722qf0988a20234dcfed@mail.gmail.com> I would like to contribute in me free time to the boxbackup project. I have a degree of science of computer. I am sysadmin on Solaris/Linux and other UNIX. I am also a Oracle DBA and have knowledge about C programming. What do you sugest for me to contribute to project: fix bus, new funcionalities?. I can dedicate little time: 4-6 hours per week. I would like to ask these questions: 1- It seems that the mailing lists have been lost. Is it possible to access a archive of the mailing list? If it isn't possible, is planned to restore this archive? 2- Does the trunk version of repository belong to the 11rc version, or to the future 12 version? Best regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at qwirx.com Wed Mar 17 08:06:55 2010 From: chris at qwirx.com (Chris Wilson) Date: Wed, 17 Mar 2010 09:06:55 +0100 (CET) Subject: [Box Backup] How to contribute to the project and two questions In-Reply-To: <26a0d8261003161719m26c722qf0988a20234dcfed@mail.gmail.com> References: <26a0d8261003161719m26c722qf0988a20234dcfed@mail.gmail.com> Message-ID: Hi Jose, On Wed, 17 Mar 2010, Jose Luis Rodriguez Garcia wrote: > I would like to contribute in me free time to the boxbackup project. That's excellent, thanks! > What do you sugest for me to contribute to project: fix bus, new > funcionalities?. I can dedicate little time: 4-6 hours per week. You could have a look at the bug tracker on our wiki as a way to get into the project. If you don't like the look of any of them, there are three major projects in progress: * Support for volume shadow copies (VSS) on Windows * Support for snapshot backups (in progress) * Support for backing up to Amazon S3 > 1- It seems that the mailing lists have been lost. Is it possible to > access a archive of the mailing list? If it isn't possible, is planned > to restore this archive? The mailing lists themselves are not lost, but the archives seem to have been mislaid somehow. I'm not involved in managing them, perhaps Alaric or James can help you? > 2- Does the trunk version of repository belong to the 11rc version, or to > the future 12 version? There's no specific distinction between the two at the moment. Trunk has some features which are not backported to the latest 0.11 rc, and which may or may not (probably not) make it into 0.11 final, such as using QDBM for renamed file tracking, and full reference storage (not just reference counts). Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From james at netinertia.co.uk Wed Mar 17 11:42:35 2010 From: james at netinertia.co.uk (James O'Gorman) Date: Wed, 17 Mar 2010 11:42:35 +0000 Subject: [Box Backup] How to contribute to the project and two questions In-Reply-To: References: <26a0d8261003161719m26c722qf0988a20234dcfed@mail.gmail.com> Message-ID: <16527699168110886507297ee6430413@netinertia.co.uk> On Wed, 17 Mar 2010 09:06:55 +0100 (CET), Chris Wilson wrote: > On Wed, 17 Mar 2010, Jose Luis Rodriguez Garcia wrote: >> 1- It seems that the mailing lists have been lost. Is it possible to >> access a archive of the mailing list? If it isn't possible, is planned >> to restore this archive? > > The mailing lists themselves are not lost, but the archives seem to have > been mislaid somehow. I'm not involved in managing them, perhaps Alaric or > James can help you? I do have copies of the old list archives now, I just haven't had time to integrate them into the new archives yet. I need to do it in a way that won't break things. :-) What I'll probably do is take the lists down for an hour or so one evening so that I can do it safely. I'll give another heads-up when I've found some time to do this. James From achim+box at qustodium.net Wed Mar 17 16:06:36 2010 From: achim+box at qustodium.net (Achim) Date: Wed, 17 Mar 2010 17:06:36 +0100 Subject: [Box Backup] 0.11RC7, attributes, ACLs under Windows In-Reply-To: References: <9f0ca8c730246d143ad63c42709b4d60@localhost> Message-ID: <71e1c68fd4212b89976691ff06b8754f@localhost> Hello Chris: On Wed, 10 Mar 2010 17:40:36 +0100 (CET), Chris Wilson wrote: > Hi Achim, > > On Wed, 10 Mar 2010, Achim wrote: > >> [0] Apart from two small issues: >> During "make test" several times "Unable to send message to Event Log >> (Event Log is full):" shows up > > This is not really Box Backup's fault except that it generates a lot of > Event Log messages, and you may need to reconfigure your Event Log to > handle them. I think users would be unpleasantly surprised if we did that > for them. Does it make sense for Box Backup to generate sufficient log messages to saturate a standard XP event log? Perhaps there is a way to either group messages or turn down the default verbosity? An unexperienced user might understand of the message "Unable to send message to Event Log (Event Log is full)" as an error rather than a warning, so this would be a source of confusion with a simple remedy: less verbose logging by default. What do you think? Best regards, Achim From mat+bbackup at matws.net Wed Mar 24 07:34:27 2010 From: mat+bbackup at matws.net (Matthieu Patou) Date: Wed, 24 Mar 2010 10:34:27 +0300 Subject: [Box Backup] How to contribute to the project and two questions In-Reply-To: References: <26a0d8261003161719m26c722qf0988a20234dcfed@mail.gmail.com> Message-ID: <4BA9C083.6020305@matws.net> Hi Chris, > You could have a look at the bug tracker on our wiki as a way to get > into the project. If you don't like the look of any of them, there are > three major projects in progress: > > * Support for volume shadow copies (VSS) on Windows What is the status on this is there any code for this already ? > * Support for snapshot backups (in progress) What is the definition of snapshot backup for you ? is the fact that you'll get all the file at a given time what ever the time you take to make the backup ? Matthieu. From mat+bbackup at matws.net Wed Mar 24 07:34:37 2010 From: mat+bbackup at matws.net (Matthieu Patou) Date: Wed, 24 Mar 2010 10:34:37 +0300 Subject: [Box Backup] 0.11RC7, attributes, ACLs under Windows In-Reply-To: <9f0ca8c730246d143ad63c42709b4d60@localhost> References: <9f0ca8c730246d143ad63c42709b4d60@localhost> Message-ID: <4BA9C08D.5010601@matws.net> On 10/03/2010 19:05, Achim wrote: > Hello list: > > Chris just released 0.11RC7, and it compiles perfectly under MingW: thanks > so much Chris[0]! > > Shout if anyone wants to testdrive RC7 and I will put up my binaries. > > On a related note, I wanted to come back to the topic of attributes and > ACLs under Windows. Inspired by an article on how to use BackupPC for a > "Bare Metal Restore" (BRM) of Windows XP [1], I think we should revisit the > capability to backup and restore attributes and ACLs under Windows. If you > have a brief look at the described process, you can see the role those two > play if you want to have any chance of BRM. > > Why ACL are not backuped up for the moment by boxbackup ? What about posix acls ? Also I have the same idea as Achim, it's maybe not very important to be able to have a system with a livecd that is able to reinstall everything but boxbackup should be able to restore a whole drive with correct acls and any others attributes and alternate data stream (http://en.wikipedia.org/wiki/Fork_%28filesystem%29#Microsoft) Matthieu. From chris at qwirx.com Wed Mar 24 19:40:22 2010 From: chris at qwirx.com (Chris Wilson) Date: Wed, 24 Mar 2010 20:40:22 +0100 (CET) Subject: [Box Backup] 0.11RC7, attributes, ACLs under Windows In-Reply-To: <71e1c68fd4212b89976691ff06b8754f@localhost> References: <9f0ca8c730246d143ad63c42709b4d60@localhost> <71e1c68fd4212b89976691ff06b8754f@localhost> Message-ID: Hi Achim, On Wed, 17 Mar 2010, Achim wrote: >>> [0] Apart from two small issues: >>> During "make test" several times "Unable to send message to Event Log >>> (Event Log is full):" shows up >> >> This is not really Box Backup's fault except that it generates a lot of >> Event Log messages, and you may need to reconfigure your Event Log to >> handle them. I think users would be unpleasantly surprised if we did >> that for them. > > Does it make sense for Box Backup to generate sufficient log messages to > saturate a standard XP event log? Perhaps there is a way to either group > messages or turn down the default verbosity? > > An unexperienced user might understand of the message "Unable to send > message to Event Log (Event Log is full)" as an error rather than a > warning, so this would be a source of confusion with a simple remedy: less > verbose logging by default. This isn't default verbosity, it's only while running tests. I don't expect inexperienced users to run the tests, I expect developers to do it for them. If the default verbosity level of the application in production is too high, please suggest which messages (that you see in the event log in normal use) you don't think are useful and let's see what we can do about it. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From chris at qwirx.com Wed Mar 24 19:44:27 2010 From: chris at qwirx.com (Chris Wilson) Date: Wed, 24 Mar 2010 20:44:27 +0100 (CET) Subject: [Box Backup] How to contribute to the project and two questions In-Reply-To: <4BA9C083.6020305@matws.net> References: <26a0d8261003161719m26c722qf0988a20234dcfed@mail.gmail.com> <4BA9C083.6020305@matws.net> Message-ID: Hi Matthieu, On Wed, 24 Mar 2010, Matthieu Patou wrote: >> You could have a look at the bug tracker on our wiki as a way to get >> into the project. If you don't like the look of any of them, there are >> three major projects in progress: >> >> * Support for volume shadow copies (VSS) on Windows > What is the status on this is there any code for this already ? There is no code already integrated into Box, but there is an open ticket on Trac where Achim has been collecting information: http://www.boxbackup.org/trac/ticket/13 And we are now allowed to use code from Bacula, which does have support for VSS already. >> * Support for snapshot backups (in progress) > > What is the definition of snapshot backup for you ? is the fact that > you'll get all the file at a given time what ever the time you take to > make the backup ? Yes, all those files, no other files, and their contents at as close a time as possible to the time of the snapshot (given that the snapshot is unlikely to be instantaneous). So that you could drag a slider back in time, like Time Machine, and see all files and directories as they were at that time, or as close as possible to that effect. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From chris at qwirx.com Wed Mar 24 19:48:01 2010 From: chris at qwirx.com (Chris Wilson) Date: Wed, 24 Mar 2010 20:48:01 +0100 (CET) Subject: [Box Backup] 0.11RC7, attributes, ACLs under Windows In-Reply-To: <4BA9C08D.5010601@matws.net> References: <9f0ca8c730246d143ad63c42709b4d60@localhost> <4BA9C08D.5010601@matws.net> Message-ID: Hi Matthieu, On Wed, 24 Mar 2010, Matthieu Patou wrote: >> On a related note, I wanted to come back to the topic of attributes and >> ACLs under Windows. Inspired by an article on how to use BackupPC for a >> "Bare Metal Restore" (BRM) of Windows XP [1], I think we should revisit >> the capability to backup and restore attributes and ACLs under Windows. >> If you have a brief look at the described process, you can see the role >> those two play if you want to have any chance of BRM. >> > Why ACL are not backuped up for the moment by boxbackup? Because nobody has written the code yet. Actually I made a start, which you can find here: http://www.boxbackup.org/trac/browser/box/chris/acl But it broke the tests and I never had enough time to fix it completely, so I stopped work on it at some point. It will also break backwards compatibility for Windows users. > What about posix acls ? They are supported on some platforms. > Also I have the same idea as Achim, it's maybe not very important to be > able to have a system with a livecd that is able to reinstall everything > but boxbackup should be able to restore a whole drive with correct acls > and any others attributes and alternate data stream > (http://en.wikipedia.org/wiki/Fork_%28filesystem%29#Microsoft) I think it would be nice, but it's not currently essential, and I have many other important features to implement and bugs to fix too. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From maillist at diffingo.com Sat Mar 27 17:28:50 2010 From: maillist at diffingo.com (Stewart Adam) Date: Sat, 27 Mar 2010 13:28:50 -0400 Subject: [Box Backup] Request for Trac/Wiki account Message-ID: <4BAE4052.60602@diffingo.com> Hi, I was wondering if I could have a trac account created for me with username "firewing1". I've written a small patch for this issue and wanted to post it: http://www.boxbackup.org/trac/ticket/47 Thanks! Stewart From achim+box at qustodium.net Sun Mar 28 00:11:03 2010 From: achim+box at qustodium.net (Achim) Date: Sun, 28 Mar 2010 01:11:03 +0100 Subject: [Box Backup] 0.11RC7, attributes, ACLs under Windows In-Reply-To: References: <9f0ca8c730246d143ad63c42709b4d60@localhost> <71e1c68fd4212b89976691ff06b8754f@localhost> Message-ID: <4BAE9E97.807@qustodium.net> Hello Chris: On 24/03/2010 20:40, Chris Wilson wrote: > This isn't default verbosity, it's only while running tests. I don't > expect inexperienced users to run the tests, I expect developers to do > it for them. > > If the default verbosity level of the application in production is too > high, please suggest which messages (that you see in the event log in > normal use) you don't think are useful and let's see what we can do > about it. Hm, could it be that Box Backup requires admin rights under Windows in order to write to the eventlog? I get the "event log full" error also when running bbackupd.exe from the command line as a standard (non-admin) user, not only when running "make test". Me and others [1] have also detected the following problem that shows up in the event log: > The description for Event ID(1) in Source (Box Backup (Box Backup)) cannot > be found.... Best regards, Achim [1] From achim+box at qustodium.net Mon Mar 29 12:36:29 2010 From: achim+box at qustodium.net (Achim) Date: Mon, 29 Mar 2010 13:36:29 +0200 Subject: [Box Backup] Regex questions Message-ID: Hello list: After reading through the related "documentation" [1] (at this moment more of a "works for me collection"), I am still banging my head against a couple of issues. Solving them and updating the Wiki accordingly might be beneficial for everybody. OS X likes to pollute directories with its meta data that has filenames such as ._.Trashes, ._WHATEVERFILE.EXT and directory names such as .Spotlight-V100. I am able to exclude the directories (they have fixed names), but I am struggling with the filenames that are dynamic. A specific example: BEFORE: Synchronised file: F:\Set_9-Excludes\._.Trashes Synchronised file: F:\Set_9-Excludes\.typeAttributes.dict Now I want to exclude all files that start with ._ (dot underscore), but not exclude those that only start with a dot (think: unix-y configuration dotfiles!): ExcludeFilesRegex = .+\._*|.*\\\._* Skipping excluded file: F:\Set_9-Excludes\._.Trashes Skipping excluded file: F:\Set_9-Excludes\.typeAttributes.dict I have not been able to find a regex that distinguihes between those two files, it appears that the underscore in the regex is ignored... or the more likely explanation: I have no clue how this is supposed to work (-: Thank you for any pointers, Achim PS: And while we are at it, what does ".+\\~.*" exclude? It is mentioned on [1] but has no explanation: my guess is "all files that start with tilde dot", but why is the dot not escaped with a backslash such as in the other examples? PPS: Perhaps we could even generate a "best practice" [file|dir]excluderegex? [1] From achim+box at qustodium.net Mon Mar 29 13:44:04 2010 From: achim+box at qustodium.net (Achim) Date: Mon, 29 Mar 2010 14:44:04 +0200 Subject: [Box Backup] Mass Deletion Protection Message-ID: <01b8d069eaf052417b941e26c9f33047@localhost> Hello list: In line with a discussion from about 2 years ago [1], I was wondering if there is anything to help prevent the following scenario: 1. Create backup location F_Photos that points to external drive f: 2. Make backup (takes a looong time, many photos) 3. Remove backup location F_Photos from bbackupd.conf and add a location G_Nothing instead 4. The next time Box Backup syncs, it deletes all data that were stored in F_Photos from the server! Now imagine the above with Step 3 changed as follows: accidentally overwrite your bbackupd.conf with an older version that did not have F_Photos in it. This would render all the effort and bandwidth that went in to backing up F_Photos useless: the location is deleted from the store, and even "list -otrs" does not bring it back. This seems like a rather dangerous "feature", is there a way to either turn it off or even change the default behaviour (e.g. "mark as deleted" instead of actually deleting the data?). Thanks, Achim [1] From chris at qwirx.com Mon Mar 29 13:59:46 2010 From: chris at qwirx.com (Chris Wilson) Date: Mon, 29 Mar 2010 14:59:46 +0200 (CEST) Subject: [Box Backup] Mass Deletion Protection In-Reply-To: <01b8d069eaf052417b941e26c9f33047@localhost> References: <01b8d069eaf052417b941e26c9f33047@localhost> Message-ID: On Mon, 29 Mar 2010, Achim wrote: > 1. Create backup location F_Photos that points to external drive f: > 2. Make backup (takes a looong time, many photos) > 3. Remove backup location F_Photos from bbackupd.conf and add a location > G_Nothing instead > 4. The next time Box Backup syncs, it deletes all data that were stored in > F_Photos from the server! Yes, the DeleteRedundantLocationsAfter option in bbackupd.conf. > Now imagine the above with Step 3 changed as follows: accidentally > overwrite your bbackupd.conf with an older version that did not have > F_Photos in it. Same. > This would render all the effort and bandwidth that went in to backing up > F_Photos useless: the location is deleted from the store, and even "list > -otrs" does not bring it back. The files are only actually removed when housekeeping needs to remove them. If you don't exceed your softlimit on the store, they will still be there until you do. > This seems like a rather dangerous "feature", is there a way to either > turn it off or even change the default behaviour (e.g. "mark as deleted" > instead of actually deleting the data?). We all agree snapshots would be safer, but nobody has actually implemented them yet. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From kenny.millington at 3ait.co.uk Mon Mar 29 13:32:54 2010 From: kenny.millington at 3ait.co.uk (Kenny Millington) Date: Mon, 29 Mar 2010 13:32:54 +0100 Subject: [Box Backup] Regex questions In-Reply-To: References: Message-ID: <1269865974.4161.4.camel@helios> Hi, > > ExcludeFilesRegex = .+\._*|.*\\\._* > > Skipping excluded file: F:\Set_9-Excludes\._.Trashes > Skipping excluded file: F:\Set_9-Excludes\.typeAttributes.dict > > I have not been able to find a regex that distinguihes between those two > files, it appears that the underscore in the regex is ignored... or the > more likely explanation: I have no clue how this is supposed to work (-: Could be the monday-blues but surely "_*" will match zero or more underscores and thus the case of no underscore at all... -- Kenny Millington Systems Developer kenny.millington at 3ait.co.uk 3aIT Limited - Official Corporate Sponsor of the British Bobsleigh Team 4-10 Barttelot Rd Horsham West Sussex RH12 1DQ CoReg: 3866698 VATReg: 771388600 T: +44 (0)870 881 5097 F: +44 (0)870 116 0793 Visit www.3aIT.co.uk for Design, Systems, Support Disclaimer: The information contained within this email is confidential and may be legally privileged. It is intended solely for the addressee. If you are not the intended recipient, any disclosure, copying or distribution of this email is prohibited and may be unlawful. The content of this email represents the views of the individual and not necessarily 3aIT Limited. 3aIT Limited reserves the right to monitor the content of all emails in accordance with lawful business practice. Whilst every effort is made to ensure that attachments are free from computer viruses before transmission, 3aIT Limited does not accept any liability in respect of any virus that is not detected. From pjalajas at gigalock.com Mon Mar 29 14:26:21 2010 From: pjalajas at gigalock.com (Peter Jalajas, GigaLock Backup Services) Date: Mon, 29 Mar 2010 09:26:21 -0400 Subject: [Box Backup] Regex questions In-Reply-To: References: Message-ID: <74d01c7a1003290626o4b2ff4fcw7836bf3f44a5094c@mail.gmail.com> Hi Achim, On Mon, Mar 29, 2010 at 7:36 AM, Achim wrote: > PS: And while we are at it, what does ".+\\~.*" exclude? It is mentioned > on [1] but has no explanation: my guess is "all files that start with tilde > dot", but why is the dot not escaped with a backslash such as in the other > examples? You are close, I think. The dot in this case is not escaped, so it is a wildcard. So, that regex matches any file (or directory, I think) that begins with tilde, or consists of a tilde by itself, I think. Answering a question with a question, I wonder how the dot-plus (".+") works in this case. I think it is intended to match any directory name (the + requiring at least one character, as opposed to the star which would match against an absence of anything before the path delimiter "\\"), but does that mean that it will not match a file in the root directory of the BackupLocation? Thanks, Pete From pjalajas at gigalock.com Mon Mar 29 14:29:23 2010 From: pjalajas at gigalock.com (Peter Jalajas, GigaLock Backup Services) Date: Mon, 29 Mar 2010 09:29:23 -0400 Subject: [Box Backup] Mass Deletion Protection In-Reply-To: References: <01b8d069eaf052417b941e26c9f33047@localhost> Message-ID: <74d01c7a1003290629l5720a5c4vde4d10de9f1dff28@mail.gmail.com> Hi Achim, On Mon, Mar 29, 2010 at 8:59 AM, Chris Wilson wrote: > On Mon, 29 Mar 2010, Achim wrote: >> This would render all the effort and bandwidth that went in to backing up >> F_Photos useless: the location is deleted from the store, and even "list >> -otrs" does not bring it back. The -d option lists files marked for deletion. So, would "-otrsd" help you find your marked-for-deletion files? Hope that helps. Pete From james at netinertia.co.uk Mon Mar 29 19:16:14 2010 From: james at netinertia.co.uk (James O'Gorman) Date: Mon, 29 Mar 2010 19:16:14 +0100 Subject: [Box Backup] Regex questions In-Reply-To: <1269865974.4161.4.camel@helios> References: <1269865974.4161.4.camel@helios> Message-ID: <5801A309-AEFA-4DAE-9146-EEB016252DC3@netinertia.co.uk> On 29 Mar 2010, at 13:32, Kenny Millington wrote: > Hi, > >> >> ExcludeFilesRegex = .+\._*|.*\\\._* >> >> Skipping excluded file: F:\Set_9-Excludes\._.Trashes >> Skipping excluded file: F:\Set_9-Excludes\.typeAttributes.dict >> >> I have not been able to find a regex that distinguihes between those two >> files, it appears that the underscore in the regex is ignored... or the >> more likely explanation: I have no clue how this is supposed to work (-: > > Could be the monday-blues but surely "_*" will match zero or more > underscores and thus the case of no underscore at all... Pretty much, yes. It should be more like this: .*\\\._.* i.e. OPTIONALLY begins with any character, then a \, then a dot, then an underscore, then any OPTIONALLY any character. I suppose it's made slightly tricker by the DOS-style paths. I can't say I've ever used Box Backup on a Windows machine, so the regex seems a little more confusing :-) This bit: >> ExcludeFilesRegex = .+\._* Means "one or more characters followed by a dot, OPTIONALLY followed by an underscore" so will catch any file starting with a dot whether or not it has an underscore. James From achim+box at qustodium.net Tue Mar 30 11:36:16 2010 From: achim+box at qustodium.net (Achim) Date: Tue, 30 Mar 2010 12:36:16 +0200 Subject: [Box Backup] Regex questions In-Reply-To: <1269865974.4161.4.camel@helios> References: <1269865974.4161.4.camel@helios> Message-ID: Hello Kenny: On Mon, 29 Mar 2010 13:32:54 +0100, Kenny Millington wrote: >> ExcludeFilesRegex = .+\._*|.*\\\._* >> >> Skipping excluded file: F:\Set_9-Excludes\._.Trashes >> Skipping excluded file: F:\Set_9-Excludes\.typeAttributes.dict >> >> I have not been able to find a regex that distinguihes between those two >> files, it appears that the underscore in the regex is ignored... or the >> more likely explanation: I have no clue how this is supposed to work (-: > > Could be the monday-blues but surely "_*" will match zero or more > underscores and thus the case of no underscore at all... Thanks for spotting this (-: My final excludes look as follows, perhaps somebody wants to use them or get inspired: ExcludeDirsRegex = .+\\Temporary Internet Files$|.*\\Cache$|.*\\Temp$|.+\\RECYCLER$|.+\\RECYCLED$|.+\\\.Trashes$|.+\\System Volume Information$|.+\\\.TemporaryItems$|.+\\\.Spotlight-V100$|.+\\\.fseventsd$|.+\\Microsoft\Search\Data$|.+\\Local Settings\\.*\\Cache$|.+\\\.svn$|.+\\\.cvs$|.+\\Google Desktop Search$|.+\\Windows\\Prefetch$|.+\\WINDOWS\\system32\\spool\\PRINTERS$' ExcludeFilesRegex = .+\.([aA][vV][iI]|[wW][mM][aAvV]|[dD][iI][vV]|[xX][sS][oO]|[mM][pP][eE]?[2345gG]|[tT][mM][pP]|[bB][aAcC][kK]|[dD][bB][kK]|[bB][kK][~!1-9]|[mMtT][bB][kK]|[oO][lL][dD]|[sS][aA][vV]|[sS][wW][pP]|[xX][lL][kK]|[cC][sS][mM]|[dD][sS][kK]|[oO][bB][jJ]|[pP][aA][rR]|[dD][bB][xX]|[dD][lL][lL])$|.+\\thumbs\.db$|.+\\Perflib.*|.+\\UsrClass\.dat\.LOG$|.+\\hyberfile\.sys$|.+\\pagefile\.sys$|.+\\\.DS_Store$|.+\\\.Spotlight-V100$|.+\\UsrClass\.dat\.LOG$|.+\\UsrClass\.dat$|.+\\~.*|.*\\\._.* Best regards, Achim From achim+box at qustodium.net Tue Mar 30 11:43:13 2010 From: achim+box at qustodium.net (Achim) Date: Tue, 30 Mar 2010 12:43:13 +0200 Subject: [Box Backup] Mass Deletion Protection In-Reply-To: References: <01b8d069eaf052417b941e26c9f33047@localhost> Message-ID: <7bb0a901cafd6f578852158e3e86b2c5@localhost> Hello Chris: On Mon, 29 Mar 2010 14:59:46 +0200 (CEST), Chris Wilson wrote: > On Mon, 29 Mar 2010, Achim wrote: > >> 1. Create backup location F_Photos that points to external drive f: >> 2. Make backup (takes a looong time, many photos) >> 3. Remove backup location F_Photos from bbackupd.conf and add a location >> G_Nothing instead >> 4. The next time Box Backup syncs, it deletes all data that were stored >> in >> F_Photos from the server! > > Yes, the DeleteRedundantLocationsAfter option in bbackupd.conf. OK, found the DeleteRedundantLocationsAfter and set it to 0 in order to avoid deletion. >> This would render all the effort and bandwidth that went in to backing up >> F_Photos useless: the location is deleted from the store, and even "list >> -otrs" does not bring it back. > > The files are only actually removed when housekeeping needs to remove > them. If you don't exceed your softlimit on the store, they will still be > there until you do. Pete's suggestion of using "list -otrsd" seems to do the trick: the data is still there, I just was not able to look for it in the right way. >> This seems like a rather dangerous "feature", is there a way to either >> turn it off or even change the default behaviour (e.g. "mark as deleted" >> instead of actually deleting the data?). > > We all agree snapshots would be safer, but nobody has actually implemented > them yet. With the DeleteRedundantLocationsAfter option and setting hard limit = soft limit, I think that this particular "issue" is resolved for me. How would snapshots add anything to this situation, apart from giving a much more natural "interface" (look for the copy of important.odt from 22 March at 13:15). Achim From micha at kovoks.nl Tue Mar 30 11:26:33 2010 From: micha at kovoks.nl (Micha Kersloot) Date: Tue, 30 Mar 2010 12:26:33 +0200 Subject: [Box Backup] Webmin module for boxbackup client. Message-ID: <201003301226.34086.micha@kovoks.nl> Hi, Just for your info, i've written a webmin module around bbackupquery. You can find it at: http://sf.own-it.nl/projects/show/webmin-boxbackup It's GPL, so have fun with it. -- Met vriendelijke groet, Micha Kersloot http://www.kovoks.nl/ KovoKs B.V. is ingeschreven onder KvK nummer: 11033334 From achim+box at qustodium.net Tue Mar 30 11:47:42 2010 From: achim+box at qustodium.net (Achim) Date: Tue, 30 Mar 2010 12:47:42 +0200 Subject: [Box Backup] Mass Deletion Protection In-Reply-To: <74d01c7a1003290629l5720a5c4vde4d10de9f1dff28@mail.gmail.com> References: <01b8d069eaf052417b941e26c9f33047@localhost> <74d01c7a1003290629l5720a5c4vde4d10de9f1dff28@mail.gmail.com> Message-ID: Hello Pete: On Mon, 29 Mar 2010 09:29:23 -0400, "Peter Jalajas, GigaLock Backup Services" wrote: > The -d option lists files marked for deletion. So, would "-otrsd" > help you find your marked-for-deletion files? Thanks for the hint, I had completely missed that. Related to this, I detected that CRTL+C closes the bbackupquery.exe console window under Windows and filed a bug for that [1]. Best regards, Achim [1] From chris at qwirx.com Tue Mar 30 12:51:09 2010 From: chris at qwirx.com (Chris Wilson) Date: Tue, 30 Mar 2010 13:51:09 +0200 (CEST) Subject: [Box Backup] Mass Deletion Protection In-Reply-To: <7bb0a901cafd6f578852158e3e86b2c5@localhost> References: <01b8d069eaf052417b941e26c9f33047@localhost> <7bb0a901cafd6f578852158e3e86b2c5@localhost> Message-ID: Hi Achim, On Tue, 30 Mar 2010, Achim wrote: > How would snapshots add anything to this situation, apart from giving a > much more natural "interface" (look for the copy of important.odt from 22 > March at 13:15). It would give you (on the client or the server) the ability to control for how long history was preserved, e.g. to keep the last 365 days worth of snapshots, giving you protection for a year if you misconfigured Box Backup in the mean time. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From flo at orbie.de Tue Mar 30 14:08:32 2010 From: flo at orbie.de (Florian Eyben) Date: Tue, 30 Mar 2010 15:08:32 +0200 Subject: [Box Backup] Feature request - for lousy connections Message-ID: <4BB1F7D0.2090609@orbie.de> Hello Boxbackup developers, I have noticed that a resume function exists for the restore command, but not for the compare or sync commands. I am on a DSL line which disconnects every 24 hours (or sometimes more often). Thus, I (re)start the sync after a disconnect, however it starts to sync from the beginning, which causes a few hours of overhead in my case (250 GB on a slow CPU to save power). The same is true for comparing (quick compare). During DSL disconnect the connection is down for approx. 1-2 minutes. So my solution would be: Instead of detecting a communication error TLSWriteError, or similar, and terminating the operation, wouldn't it be easy to wait for a connection resume for a configurable timeout and then resuming the connection? This might be transparently implemented on the TLS connection layer, thus the higher levels won't even notice it... Do you think this is possible, or are there some issues why it might not work as proposed? Best, Florian From pjalajas at gigalock.com Tue Mar 30 14:37:20 2010 From: pjalajas at gigalock.com (Peter Jalajas, GigaLock Backup Services) Date: Tue, 30 Mar 2010 09:37:20 -0400 Subject: [Box Backup] Mass Deletion Protection In-Reply-To: <7bb0a901cafd6f578852158e3e86b2c5@localhost> References: <01b8d069eaf052417b941e26c9f33047@localhost> <7bb0a901cafd6f578852158e3e86b2c5@localhost> Message-ID: <74d01c7a1003300637y415e415s89538eb8bfd22211@mail.gmail.com> Hi Achim and Chris, On Tue, Mar 30, 2010 at 6:43 AM, Achim wrote: > Hello Chris: > > On Mon, 29 Mar 2010 14:59:46 +0200 (CEST), Chris Wilson > wrote: >> On Mon, 29 Mar 2010, Achim wrote: > > Pete's suggestion of using "list -otrsd" seems to do the trick: the data > is still there, I just was not able to look for it in the right way. One thing I just noticed recently, and, Chris, maybe it's new behavior, is that: 1) the -o option shows old _and_ current versions (and no deleted versions), but 2) the -d option shows _only_ the deleted versions (and no old or current versions). I thought I recalled that I had previously routinely used -odstr to list all my files and then used grep to find the ones I was looking for. So, I believe now there is no way to use a single command to list _all_ versions of a file. Chris: what do you think? Thanks, Pete From chris at qwirx.com Tue Mar 30 14:39:03 2010 From: chris at qwirx.com (Chris Wilson) Date: Tue, 30 Mar 2010 15:39:03 +0200 (CEST) Subject: [Box Backup] Feature request - for lousy connections In-Reply-To: <4BB1F7D0.2090609@orbie.de> References: <4BB1F7D0.2090609@orbie.de> Message-ID: Hi Florian, On Tue, 30 Mar 2010, Florian Eyben wrote: > I have noticed that a resume function exists for the restore command, > but not for the compare or sync commands. I am on a DSL line which > disconnects every 24 hours (or sometimes more often). Thus, I (re)start > the sync after a disconnect, however it starts to sync from the > beginning, which causes a few hours of overhead in my case (250 GB on a > slow CPU to save power). The same is true for comparing (quick compare). Backup should automatically resume more or less from where it left off. It will need to download remote directory listings, but files already completely uploaded should not be uploaded again. > During DSL disconnect the connection is down for approx. 1-2 minutes. > So my solution would be: Instead of detecting a communication error > TLSWriteError, or similar, and terminating the operation, wouldn't it be easy > to wait for a connection resume for a configurable timeout and then resuming > the connection? This might be transparently implemented on the TLS connection > layer, thus the higher levels won't even notice it... Do you think this is > possible, or are there some issues why it might not work as proposed? I don't think OpenSSL supports this at the TLS layer (perhaps using DTLS it does?), especially as the IP address might change. It would also open you to compromise if an attacker (e.g. someone in the same office) could kick your computer offline and then connect to the server and resume the session with your login and keys. So this would have to use a session key to log into the server again, which opens up more vulnerabilities. The application also does not know, without checking on the server, how much data was successfully received and saved there, because it did not receive confirmation. So it has to check the server again anyway. I think this would be difficult to implement, and would add security vulnerabilities and increase risk of bugs with consequent risk of data loss through faulty backups. There is also a workaround in that you could switch to an ISP that doesn't suck. So I'm reluctant to implement it as proposed. If there is a shortcut, such as caching directories locally with checksums, or if there is a bug that causes more data to be uploaded than needs to be, then we could look at that as a change with less potential impact. I'd also like to see the actual bug demonstrated before tackling it, with bandwidth numbers and estimated savings. It would be reasonable for a compare operation to support resuming, I agree. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From pjalajas at gigalock.com Tue Mar 30 14:47:24 2010 From: pjalajas at gigalock.com (Peter Jalajas, GigaLock Backup Services) Date: Tue, 30 Mar 2010 09:47:24 -0400 Subject: [Box Backup] Regex questions In-Reply-To: References: <1269865974.4161.4.camel@helios> Message-ID: <74d01c7a1003300647g3f7c1de9sf31f01d538251c29@mail.gmail.com> Hi Achim, On Tue, Mar 30, 2010 at 6:36 AM, Achim wrote: > Hello Kenny: > > On Mon, 29 Mar 2010 13:32:54 +0100, Kenny Millington > wrote: > My final excludes look as follows, perhaps somebody wants to use them or > get inspired: > > ExcludeDirsRegex = .+\\Temporary Internet > Files$|.*\\Cache$|.*\\Temp$|.+\\RECYCLER$|.+\\RECYCLED$|.+\\\.Trashes$|.+\\System > Volume > Information$|.+\\\.TemporaryItems$|.+\\\.Spotlight-V100$|.+\\\.fseventsd$|.+\\Microsoft\Search\Data$|.+\\Local > Settings\\.*\\Cache$|.+\\\.svn$|.+\\\.cvs$|.+\\Google Desktop > Search$|.+\\Windows\\Prefetch$|.+\\WINDOWS\\system32\\spool\\PRINTERS$' > > ExcludeFilesRegex = > .+\.([aA][vV][iI]|[wW][mM][aAvV]|[dD][iI][vV]|[xX][sS][oO]|[mM][pP][eE]?[2345gG]|[tT][mM][pP]|[bB][aAcC][kK]|[dD][bB][kK]|[bB][kK][~!1-9]|[mMtT][bB][kK]|[oO][lL][dD]|[sS][aA][vV]|[sS][wW][pP]|[xX][lL][kK]|[cC][sS][mM]|[dD][sS][kK]|[oO][bB][jJ]|[pP][aA][rR]|[dD][bB][xX]|[dD][lL][lL])$|.+\\thumbs\.db$|.+\\Perflib.*|.+\\UsrClass\.dat\.LOG$|.+\\hyberfile\.sys$|.+\\pagefile\.sys$|.+\\\.DS_Store$|.+\\\.Spotlight-V100$|.+\\UsrClass\.dat\.LOG$|.+\\UsrClass\.dat$|.+\\~.*|.*\\\._.* I believe Excludes are now case-INsensitive. It was recommended to me to use multiple Regex lines for readability. Not a regex thing, but I routinely exclude .pst and .qbw to avoid locking the user out of those mission-critical and typically large and slow-to-backup files, and instead backup their application-generated backup files (.*backup\.pst and .*\.qbb). Thanks, Pete From flo at orbie.de Tue Mar 30 15:10:59 2010 From: flo at orbie.de (Florian Eyben) Date: Tue, 30 Mar 2010 16:10:59 +0200 Subject: [Box Backup] Feature request - for lousy connections In-Reply-To: References: <4BB1F7D0.2090609@orbie.de> Message-ID: <4BB20673.7080300@orbie.de> Hi Chris, thanks for your quick replay. I get your point there. So changing the TLS layer is tabu, I understand. Well, let me provide more details on the problem: Let's say, I have three directories I want to sync: /dir1 (1000 files) /dir2 (2000 files) /dir3 (100 files) These are listed in this order in the bbackupd.conf file. (There are more files actually, but this is only a quantitative example) I change a file in dir2 and add a file to dir3, and start the sync. Then, first all files in dir1 are scanned, no changes there. This scan, however takes a while, because server communication is required, I guess. I cannot give you exact numbers right now for my setting, but it's approx. 5 hours scan time, if there are no modifications, i.e. no data to transmit. I see an entry, transmitted, already on the server, etc. in the syslog at the end of the sync. Then I can also see how long the sync took. If that information is useful I can extract it from the next backup sync and post it. Let's say, dir2 gets synced, my changed file transferred, and then - be it the isp (all isps in germany drop the connection after 24h, except the expensive bussiness lines), or some other reason - the connection drops. "boxbackupctl sync-and-wait" will now exit with "sync finished" (maybe this should be changed to "sync incomplete...?". I know it's possible to use the notify-admin.sh here, so this issue is minor). When I restart the sync, the sync will start at /dir1 again. Ok, it doesn't transmit the changed file in /dir2 again, because it detects no change there - fair point. However, the scanning takes a while, and if the connection drops again before reaching /dir3, dir3 will never get synced! This is something I observed in my actual setup. The sync ran from 2-9am every day. stopping at 9 (not the isp..., but some connection failure). It did sync some files, because I added some files to (in my example) dir2. However, for the last 5 days it seemed to try to upload everything in dir2 without syncing my files in dir3. Now - in principle this is ok, because all data is equally valueable, however if indeed the scanning of dir1 and dir2 take a few hours, then this time is wasted, if I know the last sync left off somewhere in the middle of dir2, then I could start resuming there - ignoring changes in /dir1 of course, because the changed files in /dir3 are older and thus should be added to the backup with a higher priority. I am running the debian lenny 0.11rc2 package on both the server and the client. Thanks again for the help! Cheers, Florian Chris Wilson schrieb: > Hi Florian, > > On Tue, 30 Mar 2010, Florian Eyben wrote: > >> I have noticed that a resume function exists for the restore command, >> but not for the compare or sync commands. I am on a DSL line which >> disconnects every 24 hours (or sometimes more often). Thus, I >> (re)start the sync after a disconnect, however it starts to sync from >> the beginning, which causes a few hours of overhead in my case (250 >> GB on a slow CPU to save power). The same is true for comparing >> (quick compare). > > Backup should automatically resume more or less from where it left > off. It will need to download remote directory listings, but files > already completely uploaded should not be uploaded again. > >> During DSL disconnect the connection is down for approx. 1-2 minutes. >> So my solution would be: Instead of detecting a communication error >> TLSWriteError, or similar, and terminating the operation, wouldn't it >> be easy to wait for a connection resume for a configurable timeout >> and then resuming the connection? This might be transparently >> implemented on the TLS connection layer, thus the higher levels won't >> even notice it... Do you think this is possible, or are there some >> issues why it might not work as proposed? > > I don't think OpenSSL supports this at the TLS layer (perhaps using > DTLS it does?), especially as the IP address might change. It would > also open you to compromise if an attacker (e.g. someone in the same > office) could kick your computer offline and then connect to the > server and resume the session with your login and keys. So this would > have to use a session key to log into the server again, which opens up > more vulnerabilities. > > The application also does not know, without checking on the server, > how much data was successfully received and saved there, because it > did not receive confirmation. So it has to check the server again anyway. > > I think this would be difficult to implement, and would add security > vulnerabilities and increase risk of bugs with consequent risk of data > loss through faulty backups. There is also a workaround in that you > could switch to an ISP that doesn't suck. So I'm reluctant to > implement it as proposed. > > If there is a shortcut, such as caching directories locally with > checksums, or if there is a bug that causes more data to be uploaded > than needs to be, then we could look at that as a change with less > potential impact. > > I'd also like to see the actual bug demonstrated before tackling it, > with bandwidth numbers and estimated savings. > > It would be reasonable for a compare operation to support resuming, I > agree. > > Cheers, Chris. From chris at qwirx.com Tue Mar 30 15:20:05 2010 From: chris at qwirx.com (Chris Wilson) Date: Tue, 30 Mar 2010 16:20:05 +0200 (CEST) Subject: [Box Backup] Feature request - for lousy connections In-Reply-To: <4BB20673.7080300@orbie.de> References: <4BB1F7D0.2090609@orbie.de> <4BB20673.7080300@orbie.de> Message-ID: Hi Florian, On Tue, 30 Mar 2010, Florian Eyben wrote: > thanks for your quick replay. I get your point there. So changing the TLS > layer is tabu, I understand. Not entirely off limits, but certainly something that would require a massive amount of thought and care to achieve. > Well, let me provide more details on the problem: > > Let's say, I have three directories I want to sync: > /dir1 (1000 files) > /dir2 (2000 files) > /dir3 (100 files) > These are listed in this order in the bbackupd.conf file. (There are more > files actually, but this is only a quantitative example) > > I change a file in dir2 and add a file to dir3, and start the sync. Then, > first all files in dir1 are scanned, no changes there. This scan, however > takes a while, because server communication is required, I guess. I cannot > give you exact numbers right now for my setting, but it's approx. 5 hours > scan time, if there are no modifications, i.e. no data to transmit. I see an > entry, transmitted, already on the server, etc. in the syslog at the end of > the sync. Then I can also see how long the sync took. If that information is > useful I can extract it from the next backup sync and post it. > Let's say, dir2 gets synced, my changed file transferred, and then - be it > the isp (all isps in germany drop the connection after 24h, except the > expensive bussiness lines), or some other reason - the connection drops. > "boxbackupctl sync-and-wait" will now exit with "sync finished" (maybe this > should be changed to "sync incomplete...?". I know it's possible to use the > notify-admin.sh here, so this issue is minor). When I restart the sync, the > sync will start at /dir1 again. Ok, it doesn't transmit the changed file in > /dir2 again, because it detects no change there - fair point. However, the > scanning takes a while, and if the connection drops again before reaching > /dir3, dir3 will never get synced! Then the time taken to scan is the problem, not the downloading and updating of file or directory data. Scanning a few thousand files should only take a few seconds. Even a hundred thousand should take tens of minutes. I'm curious why it takes five hours. Could you run some software to graph the bandwidth use on your internet connection, e.g. mrtg or cacti or vnstat or ntop? It would also be interesting to see detailed logs of what bbackupd is doing when it takes five hours to sync. Perhaps you could run "bbackupd -kVT" from the command line (or set LogFile=/tmp/log.txt and run the daemon with -V) and send me the logs of a single sync run by private email? Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From achim+box at qustodium.net Tue Mar 30 15:55:25 2010 From: achim+box at qustodium.net (Achim) Date: Tue, 30 Mar 2010 16:55:25 +0200 Subject: [Box Backup] Feature request - for lousy connections In-Reply-To: References: <4BB1F7D0.2090609@orbie.de> <4BB20673.7080300@orbie.de> Message-ID: <4f5fd957e5aa50e2813ec254360a9b06@localhost> Hello Chris: On Tue, 30 Mar 2010 16:20:05 +0200 (CEST), Chris Wilson wrote: > Then the time taken to scan is the problem, not the downloading and > updating of file or directory data. > > Scanning a few thousand files should only take a few seconds. Even a > hundred thousand should take tens of minutes. I'm curious why it takes > five hours. I have been using the StoreObjectInfoFile setting in the past: # Uncomment the StoreObjectInfoFile to enable the experimental archiving # of the daemon's state (including client store marker and configuration) # between backup runs. This saves time and increases efficiency when # bbackupd is frequently stopped and started, since it removes the need # to rescan all directories on the remote server. However, it is new and # not yet heavily tested, so use with caution. StoreObjectInfoFile = $bbworkdir/bbackupd.state and was wondering how that fits in here. In addition, I remember having seen resumable uploads somewhere in connection with StoreObjectInfoFile for the following scenario: 1. bbackupd starts upload of 1,5GB HD video file to server (Windows is limited to 2GB [1]) 2. Backup gets interrupted somewhere during the process (user shuts down computer, moves into different WLAN network, ...) 3. Will bbackupd restart the upload from where it left of (for instance at 1GB, leaving only 0,5 GB left to transfer), or from the beginning? Thanks, Achim [1] From achim+box at qustodium.net Tue Mar 30 15:58:06 2010 From: achim+box at qustodium.net (Achim) Date: Tue, 30 Mar 2010 16:58:06 +0200 Subject: [Box Backup] Regex questions In-Reply-To: <74d01c7a1003300647g3f7c1de9sf31f01d538251c29@mail.gmail.com> References: <1269865974.4161.4.camel@helios> <74d01c7a1003300647g3f7c1de9sf31f01d538251c29@mail.gmail.com> Message-ID: <536f4b628131414ab7883774b4d65209@localhost> Hello Pete: On Tue, 30 Mar 2010 09:47:24 -0400, "Peter Jalajas, GigaLock Backup Services" wrote: > I believe Excludes are now case-INsensitive. OK, was not aware of this. > It was recommended to me to use multiple Regex lines for readability. I generate those lines automatically, there should be limited need for my users to play around with those lines. > Not a regex thing, but I routinely exclude .pst and .qbw to avoid > locking the user out of those mission-critical and typically large and > slow-to-backup files, and instead backup their application-generated > backup files (.*backup\.pst and .*\.qbb). Good point. I hope that when VSS [1] becomes available, we don't need to worry about that anymore. Achim [1] From chris at qwirx.com Tue Mar 30 16:01:45 2010 From: chris at qwirx.com (Chris Wilson) Date: Tue, 30 Mar 2010 17:01:45 +0200 (CEST) Subject: [Box Backup] Feature request - for lousy connections In-Reply-To: <4f5fd957e5aa50e2813ec254360a9b06@localhost> References: <4BB1F7D0.2090609@orbie.de> <4BB20673.7080300@orbie.de> <4f5fd957e5aa50e2813ec254360a9b06@localhost> Message-ID: Hi Achim, On Tue, 30 Mar 2010, Achim wrote: > I have been using the StoreObjectInfoFile setting in the past: > > # Uncomment the StoreObjectInfoFile to enable the experimental archiving > # of the daemon's state (including client store marker and configuration) > # between backup runs. This saves time and increases efficiency when > # bbackupd is frequently stopped and started, since it removes the need > # to rescan all directories on the remote server. However, it is new and > # not yet heavily tested, so use with caution. > StoreObjectInfoFile = $bbworkdir/bbackupd.state > > and was wondering how that fits in here. I think it only makes a difference if bbackupd is shut down and restarted, as it saves a copy on disk of state that is normally only kept in memory. If the connection goes down but bbackupd is not stopped, it should make no difference. > > In addition, I remember having seen resumable uploads somewhere > > connection with StoreObjectInfoFile for the following scenario: > > 1. bbackupd starts upload of 1,5GB HD video file to server (Windows is > limited to 2GB [1]) Have you tested this recently? Or are you assuming that the 2GB limit still applies because of the ticket? I was never able to reproduce the problem, so it may not exist. > 2. Backup gets interrupted somewhere during the process (user shuts down > computer, moves into different WLAN network, ...) > 3. Will bbackupd restart the upload from where it left of (for instance at > 1GB, leaving only 0,5 GB left to transfer), or from the beginning? I'm afraid resuming individual files is not currently supported, so it will start again from the beginning. My best advice for now would be to get a connection fast enough to upload the individual files that you need to back up in less than 24 hours. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From flo at orbie.de Tue Mar 30 16:05:52 2010 From: flo at orbie.de (Florian Eyben) Date: Tue, 30 Mar 2010 17:05:52 +0200 Subject: [Box Backup] Feature request - for lousy connections In-Reply-To: References: <4BB1F7D0.2090609@orbie.de> <4BB20673.7080300@orbie.de> Message-ID: <4BB21350.2090201@orbie.de> Hi Chris, Chris Wilson schrieb: > Hi Florian, > > On Tue, 30 Mar 2010, Florian Eyben wrote: > >> thanks for your quick replay. I get your point there. So changing the >> TLS layer is tabu, I understand. > > Not entirely off limits, but certainly something that would require a > massive amount of thought and care to achieve. > >> Well, let me provide more details on the problem: >> >> Let's say, I have three directories I want to sync: >> /dir1 (1000 files) >> /dir2 (2000 files) >> /dir3 (100 files) >> These are listed in this order in the bbackupd.conf file. (There are >> more files actually, but this is only a quantitative example) >> >> I change a file in dir2 and add a file to dir3, and start the sync. >> Then, first all files in dir1 are scanned, no changes there. This >> scan, however takes a while, because server communication is >> required, I guess. I cannot give you exact numbers right now for my >> setting, but it's approx. 5 hours scan time, if there are no >> modifications, i.e. no data to transmit. I see an entry, transmitted, >> already on the server, etc. in the syslog at the end of the sync. >> Then I can also see how long the sync took. If that information is >> useful I can extract it from the next backup sync and post it. >> Let's say, dir2 gets synced, my changed file transferred, and then - >> be it the isp (all isps in germany drop the connection after 24h, >> except the expensive bussiness lines), or some other reason - the >> connection drops. "boxbackupctl sync-and-wait" will now exit with >> "sync finished" (maybe this should be changed to "sync >> incomplete...?". I know it's possible to use the notify-admin.sh >> here, so this issue is minor). When I restart the sync, the sync will >> start at /dir1 again. Ok, it doesn't transmit the changed file in >> /dir2 again, because it detects no change there - fair point. >> However, the scanning takes a while, and if the connection drops >> again before reaching /dir3, dir3 will never get synced! > > Then the time taken to scan is the problem, not the downloading and > updating of file or directory data. > Yes. Uploading / Downloading times we cannot change. > Scanning a few thousand files should only take a few seconds. Even a > hundred thousand should take tens of minutes. I'm curious why it takes > five hours. > I am not sure how bb identifies changed files. For my understanding it has to retrieve some file lists from the server. But I would assume they are quite small, so downloading times should not be of concern. One explanation might be that my CPU is a 500Mhz VIA cpu, my internet bandwidth is 6 Mbit/s downstream and 0.6 Mbit/s upstream. The inital off-line sync of my 250 GB disc via USB too more than 3 days. > Could you run some software to graph the bandwidth use on your > internet connection, e.g. mrtg or cacti or vnstat or ntop? It would > also be interesting to see detailed logs of what bbackupd is doing > when it takes five hours to sync. Perhaps you could run "bbackupd > -kVT" from the command line (or set LogFile=/tmp/log.txt and run the > daemon with -V) and send me the logs of a single sync run by private > email? > I will try to get some logs as soon as possible (might take a few days though). Thanks for the recommandation of bandwidth usage monitoring tools, I'll try to work with them. Florian. > Cheers, Chris. From flo at orbie.de Tue Mar 30 16:08:32 2010 From: flo at orbie.de (Florian Eyben) Date: Tue, 30 Mar 2010 17:08:32 +0200 Subject: [Box Backup] Feature request - for lousy connections In-Reply-To: <4f5fd957e5aa50e2813ec254360a9b06@localhost> References: <4BB1F7D0.2090609@orbie.de> <4BB20673.7080300@orbie.de> <4f5fd957e5aa50e2813ec254360a9b06@localhost> Message-ID: <4BB213F0.6050305@orbie.de> Hi Achim, Achim schrieb: > Hello Chris: > > On Tue, 30 Mar 2010 16:20:05 +0200 (CEST), Chris Wilson > wrote: > >> Then the time taken to scan is the problem, not the downloading and >> updating of file or directory data. >> >> Scanning a few thousand files should only take a few seconds. Even a >> hundred thousand should take tens of minutes. I'm curious why it takes >> five hours. >> > > I have been using the StoreObjectInfoFile setting in the past: > > # Uncomment the StoreObjectInfoFile to enable the experimental archiving > # of the daemon's state (including client store marker and configuration) > # between backup runs. This saves time and increases efficiency when > # bbackupd is frequently stopped and started, since it removes the need > # to rescan all directories on the remote server. However, it is new and > # not yet heavily tested, so use with caution. > StoreObjectInfoFile = $bbworkdir/bbackupd.state > > and was wondering how that fits in here. > > This sound exactly like the feature I was looking for. Do you know from what version / svn rev. on this available? I guess it's not in 0.11rc2... > In addition, I remember having seen resumable uploads somewhere in > connection with StoreObjectInfoFile for the following scenario: > > 1. bbackupd starts upload of 1,5GB HD video file to server (Windows is > limited to 2GB [1]) > 2. Backup gets interrupted somewhere during the process (user shuts down > computer, moves into different WLAN network, ...) > 3. Will bbackupd restart the upload from where it left of (for instance at > 1GB, leaving only 0,5 GB left to transfer), or from the beginning? > This (3.) might also be useful, for exactly this scenario. Florian. > Thanks, Achim > > [1] > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup > From flo at orbie.de Tue Mar 30 16:15:48 2010 From: flo at orbie.de (Florian Eyben) Date: Tue, 30 Mar 2010 17:15:48 +0200 Subject: [Box Backup] Feature request - for lousy connections In-Reply-To: References: <4BB1F7D0.2090609@orbie.de> <4BB20673.7080300@orbie.de> <4f5fd957e5aa50e2813ec254360a9b06@localhost> Message-ID: <4BB215A4.4080905@orbie.de> Hi Chris, Chris Wilson schrieb: > Hi Achim, > > On Tue, 30 Mar 2010, Achim wrote: > >> I have been using the StoreObjectInfoFile setting in the past: >> >> # Uncomment the StoreObjectInfoFile to enable the experimental archiving >> # of the daemon's state (including client store marker and >> configuration) >> # between backup runs. This saves time and increases efficiency when >> # bbackupd is frequently stopped and started, since it removes the need >> # to rescan all directories on the remote server. However, it is new and >> # not yet heavily tested, so use with caution. >> StoreObjectInfoFile = $bbworkdir/bbackupd.state >> >> and was wondering how that fits in here. > > I think it only makes a difference if bbackupd is shut down and > restarted, as it saves a copy on disk of state that is normally only > kept in memory. If the connection goes down but bbackupd is not > stopped, it should make no difference. > Ah, ok. Now, this would be really nice to have to save the state, when the connection goes down or any other error occurs. What do you think? Is this hard to implement? >> > In addition, I remember having seen resumable uploads somewhere > >> connection with StoreObjectInfoFile for the following scenario: >> >> 1. bbackupd starts upload of 1,5GB HD video file to server (Windows is >> limited to 2GB [1]) > > Have you tested this recently? Or are you assuming that the 2GB limit > still applies because of the ticket? I was never able to reproduce the > problem, so it may not exist. > >> 2. Backup gets interrupted somewhere during the process (user shuts down >> computer, moves into different WLAN network, ...) >> 3. Will bbackupd restart the upload from where it left of (for >> instance at >> 1GB, leaving only 0,5 GB left to transfer), or from the beginning? > > I'm afraid resuming individual files is not currently supported, so it > will start again from the beginning. My best advice for now would be > to get a connection fast enough to upload the individual files that > you need to back up in less than 24 hours. > I think this an interesting feature, but currently - for me - it is not a priority. Large Media files are not that important and can be cloned to a disk stored somewhere else from time to time, since they don't change that often... I would assume that resuming a 1,5GB transfer would also require a lot of matching, since the server data is encrypted and you cannot directly diff. Either you encode, upload and compare, or you upload only hashes and compare them. Either way, it always involves encrypting 1,5GB... or am I wrong? Cheers, Florian > Cheers, Chris. From chris at qwirx.com Tue Mar 30 16:29:27 2010 From: chris at qwirx.com (Chris Wilson) Date: Tue, 30 Mar 2010 17:29:27 +0200 (CEST) Subject: [Box Backup] Feature request - for lousy connections In-Reply-To: <4BB215A4.4080905@orbie.de> References: <4BB1F7D0.2090609@orbie.de> <4BB20673.7080300@orbie.de> <4f5fd957e5aa50e2813ec254360a9b06@localhost> <4BB215A4.4080905@orbie.de> Message-ID: Hi Florian, On Tue, 30 Mar 2010, Florian Eyben wrote: >>> I have been using the StoreObjectInfoFile setting in the past [...] >>> and was wondering how that fits in here. >> >> I think it only makes a difference if bbackupd is shut down and >> restarted, as it saves a copy on disk of state that is normally only >> kept in memory. If the connection goes down but bbackupd is not >> stopped, it should make no difference. >> > Ah, ok. Now, this would be really nice to have to save the state, when the > connection goes down or any other error occurs. What do you think? Is this > hard to implement? I think there is a misunderstanding here. If you are not shutting down Box Backup (and I think you are not) then this feature makes no difference. In any case, it is implemented, so you can try it if you like. > I would assume that resuming a 1,5GB transfer would also require a lot of > matching, since the server data is encrypted and you cannot directly diff. Box Backup has a clever way of doing it that does work. > Either you encode, upload and compare, or you upload only hashes and > compare them. Either way, it always involves encrypting 1,5GB... or am I > wrong? Yes, you would have to encrypt 1.5 GB, and normally that's not a problem compared to the time taken to upload it if you didn't. Perhaps if you have a really slow CPU, that's not the case, but I'm not particularly interested in supporting such devices. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From pjalajas at gigalock.com Tue Mar 30 16:33:46 2010 From: pjalajas at gigalock.com (Peter Jalajas, GigaLock Backup Services) Date: Tue, 30 Mar 2010 11:33:46 -0400 Subject: [Box Backup] Feature request - for lousy connections In-Reply-To: <4BB215A4.4080905@orbie.de> References: <4BB1F7D0.2090609@orbie.de> <4BB20673.7080300@orbie.de> <4f5fd957e5aa50e2813ec254360a9b06@localhost> <4BB215A4.4080905@orbie.de> Message-ID: <74d01c7a1003300833wa67a759k2274e3d716614766@mail.gmail.com> Hi all, On Tue, Mar 30, 2010 at 11:15 AM, Florian Eyben wrote: > Hi Chris, > > Chris Wilson schrieb: >> >> Hi Achim, >> >> On Tue, 30 Mar 2010, Achim wrote: > I would assume that resuming a 1,5GB transfer would also require a lot of > matching, since the server data is encrypted and you cannot directly diff. > Either you encode, upload and compare, or you upload only hashes and compare > them. Either way, it always involves encrypting 1,5GB... or am I wrong? Would it be interesting to offer a bbackupd.conf option to _not_ encrypt certain files, by path/filename regex or size or other attribute, either within BackupLocations or system-wide? Maybe something like: DoNotEncryptFilesRegex = .*\\.*\.(avi|mpg)$ DoNotEncryptDirsRegex = .*\\public\\.* DoNotEncryptBySize = +100M From flo at orbie.de Tue Mar 30 16:44:24 2010 From: flo at orbie.de (Florian Eyben) Date: Tue, 30 Mar 2010 17:44:24 +0200 Subject: [Box Backup] Feature request - for lousy connections In-Reply-To: References: <4BB1F7D0.2090609@orbie.de> <4BB20673.7080300@orbie.de> <4f5fd957e5aa50e2813ec254360a9b06@localhost> <4BB215A4.4080905@orbie.de> Message-ID: <4BB21C58.1020009@orbie.de> Hi Chris, Chris Wilson schrieb: > Hi Florian, > > On Tue, 30 Mar 2010, Florian Eyben wrote: >>>> I have been using the StoreObjectInfoFile setting in the past [...] >>>> and was wondering how that fits in here. >>> >>> I think it only makes a difference if bbackupd is shut down and >>> restarted, as it saves a copy on disk of state that is normally only >>> kept in memory. If the connection goes down but bbackupd is not >>> stopped, it should make no difference. >>> >> Ah, ok. Now, this would be really nice to have to save the state, >> when the connection goes down or any other error occurs. What do you >> think? Is this hard to implement? > > I think there is a misunderstanding here. If you are not shutting down > Box Backup (and I think you are not) then this feature makes no > difference. In any case, it is implemented, so you can try it if you > like. No, I am not shutting down box backup. My question was, if you think it is possible to save the state when the connection drops (i.e. save the last state before the transfer of the file or the action where the connection dropped). I mean, more or less, call the function that saves the state (currently when stopping the daemon), when an error occurs. If this is not possible, because maybe the state is corrupt in case of an error, it may help to be able to periodically save the state, lets say every 10 or 15 minutes. So in case of a lost connection you only loose the last 10 minutes, and you would ensure that eventually you back up all of your data and not skip the last directories in case of repeated dropped connections. > >> I would assume that resuming a 1,5GB transfer would also require a >> lot of matching, since the server data is encrypted and you cannot >> directly diff. > > Box Backup has a clever way of doing it that does work. > >> Either you encode, upload and compare, or you upload only hashes and >> compare them. Either way, it always involves encrypting 1,5GB... or >> am I wrong? > > Yes, you would have to encrypt 1.5 GB, and normally that's not a > problem compared to the time taken to upload it if you didn't. Perhaps > if you have a really slow CPU, that's not the case, but I'm not > particularly interested in supporting such devices. > Yes, I understand. As I said, large files are also not my top priority. If it were, I'd have a faster CPU and more disc space anyway. Cheers, Florian From sm8ps-boxbackup1 at yahoo.com Wed Mar 31 07:25:04 2010 From: sm8ps-boxbackup1 at yahoo.com (sm8ps-boxbackup1 at yahoo.com) Date: Tue, 30 Mar 2010 23:25:04 -0700 (PDT) Subject: [Box Backup] Verbosity level: hardlinks in data In-Reply-To: References: Message-ID: <472106.1090.qm@web51602.mail.re2.yahoo.com> On 24/03/2010 20:40, Chris Wilson wrote: > This isn't default verbosity, it's only while running tests. I don't > expect inexperienced users to run the tests, I expect developers to do > it for them. > > If the default verbosity level of the application in production is too > high, please suggest which messages (that you see in the event log in > normal use) you don't think are useful and let's see what we can do > about it. Hello all! First off I would like to thank everybody working on Boxbackup for the excellent piece of software! It makes me sleep more quietly, knowing my data is securely backed up. The statement quoted above made me think about the following warnings I find during regular use (there is many of them): "WARNING: Found conflicting parent ID for file ID 877342 (): expected 105753 but found 105777 (same directory used in two different locations?)" Ticket #63 says these messages may be due to hard links on the data files which is the case here. Since I had foolishly introduced hardlinks in order to save disk space, I am now cursed with log files of 17MB each. I suggest introducing options to either suppress such warning messages alltogether (maybe by type if that is possible?) or to reduce the number of warnings of identical type to 1 (like "showing only one of ##"). Something similar must already be implemented for I also read: "c314-stefan bbackupd[1574]: WARNING: Suppressing duplicate notification about read-error" (BTW, would anybody please explain to me what exactly that warning is caused by? I have not found any specific read-error in the log files. I would be grateful for a pointer to locate the problem.) Cheers! = = = = = = = = = Stefan M?ller Wildi Unterhof 5 CH-6208 Oberkirch +41 41/920 3336 __________________________________________________ Do You Yahoo!? Sie sind Spam leid? Yahoo! Mail verf?gt ?ber einen herausragenden Schutz gegen Massenmails. http://mail.yahoo.com From chris at qwirx.com Wed Mar 31 08:33:09 2010 From: chris at qwirx.com (Chris Wilson) Date: Wed, 31 Mar 2010 09:33:09 +0200 (CEST) Subject: [Box Backup] Verbosity level: hardlinks in data In-Reply-To: <472106.1090.qm@web51602.mail.re2.yahoo.com> References: <472106.1090.qm@web51602.mail.re2.yahoo.com> Message-ID: Hi Stefan, On Tue, 30 Mar 2010, sm8ps-boxbackup1 at yahoo.com wrote: > The statement quoted above made me think about the following warnings I > find during regular use (there is many of them): "WARNING: Found > conflicting parent ID for file ID 877342 (): expected 105753 > but found 105777 (same directory used in two different locations?)" > > Ticket #63 says these messages may be due to hard links on the data > files which is the case here. Since I had foolishly introduced hardlinks > in order to save disk space, I am now cursed with log files of 17MB > each. > > I suggest introducing options to either suppress such warning messages > alltogether (maybe by type if that is possible?) or to reduce the number > of warnings of identical type to 1 (like "showing only one of ##"). > Something similar must already be implemented for I also read: > "c314-stefan bbackupd[1574]: WARNING: Suppressing duplicate notification > about read-error" There is such suppression for executing the notify script but not for the logging system (except in one case that I can think of). I would need to add it on a message-by-message basis at the moment. > (BTW, would anybody please explain to me what exactly that warning is > caused by? I have not found any specific read-error in the log files. I > would be grateful for a pointer to locate the problem.) The problem is that Box Backup isn't actually prepared to handle your situation. Hard links are not currently supported, because they conflict with single parent file ownership and renamed file tracking, and your backups may be incomplete. If you try to restore them, the multiply-linked files will probably only end up in one of the parent directories. This warning is (still) there to remind you that this problem exists, and to remind us to do something about it. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |