From boxbackup at boxbackup.org Wed Jun 3 08:46:15 2009 From: boxbackup at boxbackup.org (Dave Bamford) Date: Wed, 03 Jun 2009 08:46:15 +0100 Subject: [Box Backup] Virtual Machine backup Message-ID: <4A262A47.9090801@logical-progress.com> Has anyone on the list had experience using box to backup a virtual machine at the hypervisor level. There are some commercial products out there but all very expensive. I have a server running Vmware ESXi and am using a script ghettoVCB.sh to copy the vm to a NFS drive on another machine but ideally need to backup to offsite. The files in a VM are very big but I would guess they compress well. Ideally box should run at the hypervisor level but there are no tools to compile at this level. I can get ssh access and get to an ash shell. uname -a reports VMKernel. Dave Bamford From boxbackup at boxbackup.org Wed Jun 3 10:29:07 2009 From: boxbackup at boxbackup.org (boxbackup at boxbackup.org) Date: Wed, 3 Jun 2009 10:29:07 +0100 (BST) Subject: [Box Backup] Ports Message-ID: <1707.131.111.92.106.1244021347.squirrel@tjpcattwood.org.uk> What port/s does boxbackup communicate over please. Regards Tony From boxbackup at boxbackup.org Wed Jun 3 11:00:19 2009 From: boxbackup at boxbackup.org (Stuart Hickinbottom) Date: Wed, 03 Jun 2009 11:00:19 +0100 Subject: [Box Backup] Ports In-Reply-To: <1707.131.111.92.106.1244021347.squirrel@tjpcattwood.org.uk> References: <1707.131.111.92.106.1244021347.squirrel@tjpcattwood.org.uk> Message-ID: <4A2649B3.1020908@hickinbottom.com> The server (bbstored) listens on port 2201 (TCP). That's the only incoming port you need to open up on firewalls, I believe. Stuart On 03/06/2009 10:29 AM, boxbackup at attwood.me wrote: > What port/s does boxbackup communicate over please. > > Regards > > Tony > > _______________________________________________ > boxbackup mailing list > boxbackup at boxbackup.org > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > > From boxbackup at boxbackup.org Wed Jun 3 22:06:16 2009 From: boxbackup at boxbackup.org (Chris Wilson) Date: Thu, 4 Jun 2009 00:06:16 +0300 (EAT) Subject: [Box Backup] Virtual Machine backup In-Reply-To: <4A262A47.9090801@logical-progress.com> References: <4A262A47.9090801@logical-progress.com> Message-ID: Hi Dave, On Wed, 3 Jun 2009, Dave Bamford wrote: > Has anyone on the list had experience using box to backup a virtual > machine at the hypervisor level. There are some commercial products out > there but all very expensive. > > I have a server running Vmware ESXi and am using a script ghettoVCB.sh > to copy the vm to a NFS drive on another machine but ideally need to > backup to offsite. The files in a VM are very big but I would guess they > compress well. > > Ideally box should run at the hypervisor level but there are no tools to > compile at this level. I can get ssh access and get to an ash shell. > uname -a reports VMKernel. Box Backup is designed to compile in pretty minimal environments. You may be able to cross-compile for the hypervisor environment by setting "LDFLAGS=-static" at configure time when building on a machine with a build environment where you can compile Box Backup normally. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at boxbackup.org Wed Jun 3 22:30:00 2009 From: boxbackup at boxbackup.org (Dave Bamford) Date: Wed, 03 Jun 2009 22:30:00 +0100 Subject: [Box Backup] Virtual Machine backup In-Reply-To: References: <4A262A47.9090801@logical-progress.com> Message-ID: <4A26EB58.5090204@logical-progress.com> Chris Wilson wrote: > Hi Dave, > > On Wed, 3 Jun 2009, Dave Bamford wrote: > >> Has anyone on the list had experience using box to backup a virtual >> machine at the hypervisor level. There are some commercial products >> out there but all very expensive. >> >> I have a server running Vmware ESXi and am using a script >> ghettoVCB.sh to copy the vm to a NFS drive on another machine but >> ideally need to backup to offsite. The files in a VM are very big but >> I would guess they compress well. >> >> Ideally box should run at the hypervisor level but there are no tools >> to compile at this level. I can get ssh access and get to an ash >> shell. uname -a reports VMKernel. > > Box Backup is designed to compile in pretty minimal environments. You > may be able to cross-compile for the hypervisor environment by setting > "LDFLAGS=-static" at configure time when building on a machine with a > build environment where you can compile Box Backup normally. > > Cheers, Chris. Hi Chris Where would that go exactly, I am not an expert on building /compiling. I'll give it a try. I copied the files from a vmware esxi machine to another machine via NFS today 220Gb in total ( a virgin install of sbs 2003 ) and it tar gzipped to 2.3Gb, of course it only contained the OS and no user data. All "bare Metal" backups solutions seem to cost in excess of $500. The big issue to me is the delta blocking which box does well. But by taking a new snapshot each time box will see this as a new file and back the whole thing up. Even with an uplink of 2M which you can get with ADSL2+ it will take too long. Cheers Dave From boxbackup at boxbackup.org Thu Jun 4 19:08:30 2009 From: boxbackup at boxbackup.org (Chris Wilson) Date: Thu, 4 Jun 2009 21:08:30 +0300 (EAT) Subject: [Box Backup] Virtual Machine backup In-Reply-To: <4A26EB58.5090204@logical-progress.com> References: <4A262A47.9090801@logical-progress.com> <4A26EB58.5090204@logical-progress.com> Message-ID: Hi Dave, On Wed, 3 Jun 2009, Dave Bamford wrote: >>> Ideally box should run at the hypervisor level but there are no tools >>> to compile at this level. I can get ssh access and get to an ash >>> shell. uname -a reports VMKernel. >> >> Box Backup is designed to compile in pretty minimal environments. You >> may be able to cross-compile for the hypervisor environment by setting >> "LDFLAGS=-static" at configure time when building on a machine with a >> build environment where you can compile Box Backup normally. > > Where would that go exactly, I am not an expert on building /compiling. > I'll give it a try. You could try putting it on the configure command line, e.g. ./configure LDFLAGS=-static If that doesn't work, please post the results. You can use the ldd command to see whether the binary is linked to any dynamic libraries or not. If not then it should be good to run on the hypervisor. > I copied the files from a vmware esxi machine to another machine via NFS > today 220Gb in total ( a virgin install of sbs 2003 ) and it tar gzipped > to 2.3Gb, of course it only contained the OS and no user data. > > All "bare Metal" backups solutions seem to cost in excess of $500. The > big issue to me is the delta blocking which box does well. But by taking > a new snapshot each time box will see this as a new file and back the > whole thing up. Even with an uplink of 2M which you can get with ADSL2+ > it will take too long. If you replace the existing snapshot with a new one with the same file name, then Box Backup should do an incremental backup of it, in theory. If not, please let me know and I'll investigate. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at boxbackup.org Fri Jun 5 14:41:06 2009 From: boxbackup at boxbackup.org (Dave Bamford) Date: Fri, 05 Jun 2009 14:41:06 +0100 Subject: [Box Backup] Virtual Machine backup In-Reply-To: References: <4A262A47.9090801@logical-progress.com> <4A26EB58.5090204@logical-progress.com> Message-ID: <4A292072.1080907@logical-progress.com> Hi Chris I tried the LDFLAGS=static but I got this error configure:8360: checking for openssl/ssl.h configure:8368: result: yes configure:8384: checking for SSL_read in -lssl configure:8419: g++ -o conftest -g -O2 -static conftest.cpp -lssl -lcrypto -lz >&5 /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_func': (.text+0x2fb): undefined reference to `dlsym' /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_func': (.text+0x3ae): undefined reference to `dlerror' /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_var': (.text+0x42b): undefined reference to `dlsym' /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_var': (.text+0x4d6): undefined reference to `dlerror' /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_unload': (.text+0x52d): undefined reference to `dlclose' /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_load': (.text+0x5e2): undefined reference to `dlopen' /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_load': (.text+0x64b): undefined reference to `dlclose' /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_load': (.text+0x678): undefined reference to `dlerror' collect2: ld returned 1 exit status Using debian lenny. I think the best idea is to let the script which takes the machine snapshots replace the output file and see if box diffs it. It should work according to Ben. The only issue now is how to set up the box parameters so box can upload a 200G file. I think I will build a test vm with a small footprint first. Thanks Dave Chris Wilson wrote: > Hi Dave, > > On Wed, 3 Jun 2009, Dave Bamford wrote: > >>>> Ideally box should run at the hypervisor level but there are no >>>> tools to compile at this level. I can get ssh access and get to an >>>> ash shell. uname -a reports VMKernel. >>> >>> Box Backup is designed to compile in pretty minimal environments. >>> You may be able to cross-compile for the hypervisor environment by >>> setting "LDFLAGS=-static" at configure time when building on a >>> machine with a build environment where you can compile Box Backup >>> normally. >> >> Where would that go exactly, I am not an expert on building >> /compiling. I'll give it a try. > > You could try putting it on the configure command line, e.g. > > ./configure LDFLAGS=-static > > If that doesn't work, please post the results. You can use the ldd > command to see whether the binary is linked to any dynamic libraries > or not. If not then it should be good to run on the hypervisor. > >> I copied the files from a vmware esxi machine to another machine via >> NFS today 220Gb in total ( a virgin install of sbs 2003 ) and it tar >> gzipped to 2.3Gb, of course it only contained the OS and no user data. >> >> All "bare Metal" backups solutions seem to cost in excess of $500. >> The big issue to me is the delta blocking which box does well. But by >> taking a new snapshot each time box will see this as a new file and >> back the whole thing up. Even with an uplink of 2M which you can get >> with ADSL2+ it will take too long. > > If you replace the existing snapshot with a new one with the same file > name, then Box Backup should do an incremental backup of it, in > theory. If not, please let me know and I'll investigate. > > Cheers, Chris. From boxbackup at boxbackup.org Sat Jun 6 15:12:01 2009 From: boxbackup at boxbackup.org (Chris Wilson) Date: Sat, 6 Jun 2009 17:12:01 +0300 (EAT) Subject: [Box Backup] Virtual Machine backup In-Reply-To: <4A292072.1080907@logical-progress.com> References: <4A262A47.9090801@logical-progress.com> <4A26EB58.5090204@logical-progress.com> <4A292072.1080907@logical-progress.com> Message-ID: Hi Dave, On Fri, 5 Jun 2009, Dave Bamford wrote: > I tried the LDFLAGS=static but I got this error > > configure:8360: checking for openssl/ssl.h > configure:8368: result: yes > configure:8384: checking for SSL_read in -lssl > configure:8419: g++ -o conftest -g -O2 -static conftest.cpp -lssl -lcrypto > -lz >&5 > /usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/libcrypto.a(dso_dlfcn.o): > In function `dlfcn_bind_func': > (.text+0x2fb): undefined reference to `dlsym' It seems that there's already an option to use static linking, which might work better. If you want to try this again, please run ./configure with the --enable-static-bin option instead of LDFLAGS=-static. > I think the best idea is to let the script which takes the machine > snapshots replace the output file and see if box diffs it. It should > work according to Ben. The only issue now is how to set up the box > parameters so box can upload a 200G file. I think I will build a test vm > with a small footprint first. You'll need 200G free in the account before the first backup, and enough free space to store the patch after each snapshot. So you could try a difference between soft and hard limits of e.g. 20G. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at boxbackup.org Sat Jun 6 23:39:44 2009 From: boxbackup at boxbackup.org (Anton Bogdanovitch) Date: Sun, 07 Jun 2009 02:39:44 +0400 Subject: [Box Backup] boxbackup + pg_dump Message-ID: <4A2AF030.60407@gmail.com> Hi, What is the best way to prepare weekly postgres dump for bbackupd upload? Do I need to compress the dump? 1) pg_dumpall > dump.sql (then upload dump.sql) 2) pg_dumpall > dump.sql | pg_dumpall > dump.gz (then upload dump.gz) What should I choose in order to get minimum amount of network bandwith usage and to save backup storage space? Thanks. From boxbackup at boxbackup.org Sun Jun 7 09:15:28 2009 From: boxbackup at boxbackup.org (Anton Bogdanovitch) Date: Sun, 07 Jun 2009 12:15:28 +0400 Subject: [Box Backup] Re: boxbackup + pg_dump Message-ID: <4A2B7720.6020602@gmail.com> Sorry. There was a typo. 1) pg_dumpall > dump.sql 2) pg_dumpall > dump.sql | gzip > dump.gz From boxbackup at boxbackup.org Sun Jun 7 14:39:08 2009 From: boxbackup at boxbackup.org (Roy) Date: Sun, 07 Jun 2009 15:39:08 +0200 Subject: [Box Backup] Re: boxbackup + pg_dump In-Reply-To: <4A2B7720.6020602@gmail.com> References: <4A2B7720.6020602@gmail.com> Message-ID: <4A2BC2FC.2010301@hostingbrothers.nl> Anton Bogdanovitch wrote: > Sorry. There was a typo. > > 1) pg_dumpall > dump.sql > 2) pg_dumpall > dump.sql | gzip > dump.gz > _______________________________________________ > boxbackup mailing list > boxbackup at boxbackup.org > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > > !DSPAM:4a2b778b306301804284693! > Hi Anton, You shouldn't gzip it, because then Box Backup cannot efficiently backup the file. Because it then needs to diff a binary file instead of a text file. Greets, Roy From boxbackup at boxbackup.org Mon Jun 8 21:33:34 2009 From: boxbackup at boxbackup.org (Dave Bamford) Date: Mon, 08 Jun 2009 21:33:34 +0100 Subject: [Box Backup] Virtual Machine backup In-Reply-To: References: <4A262A47.9090801@logical-progress.com> <4A26EB58.5090204@logical-progress.com> <4A292072.1080907@logical-progress.com> Message-ID: <4A2D759E.7020301@logical-progress.com> The diffing test works, here is a snapshot of the logs. First I create a virtual machine (XP pro in this case) with a 2G hard drive. Then copy it via NFS to another machine running box client and let it back up over the net to one of my servers. > Jun 8 09:26:55 backup6 bbackupd[18765]: NOTICE: Beginning scan of > local files > Jun 8 11:00:45 backup6 bbackupd[18765]: NOTICE: Finished scan of > local files > Jun 8 11:00:45 backup6 bbackupd[18765]: NOTICE: File statistics: > total file size uploaded 2147486231, bytes already on server 0, > encoded size 423852639 > Then I make some changes to the vm (I add vmware tools support) replace the files on the nfs drive with the new ones of the same name Box back it up but only the differences if I have interpreted it correctly. > Jun 8 19:05:02 backup6 bbackupd[18765]: NOTICE: Beginning scan of > local files > Jun 8 19:33:15 backup6 bbackupd[18765]: NOTICE: Finished scan of > local files > Jun 8 19:33:15 backup6 bbackupd[18765]: NOTICE: File statistics: > total file size uploaded 2185862686, bytes already on server > 1720804357, encoded size 210338595 So now we have a means of "bare metal backup" with box. Now I need to test a restore. Dave Bamford From boxbackup at boxbackup.org Tue Jun 9 19:33:34 2009 From: boxbackup at boxbackup.org (James O'Gorman) Date: Tue, 9 Jun 2009 19:33:34 +0100 Subject: [Box Backup] Re: boxbackup + pg_dump In-Reply-To: <4A2BC2FC.2010301@hostingbrothers.nl> References: <4A2B7720.6020602@gmail.com> <4A2BC2FC.2010301@hostingbrothers.nl> Message-ID: <70A55826-EC89-42DF-93BE-61D5A021DBDD@netinertia.co.uk> On 7 Jun 2009, at 14:39, Roy wrote: > Anton Bogdanovitch wrote: >> Sorry. There was a typo. >> >> 1) pg_dumpall > dump.sql >> 2) pg_dumpall > dump.sql | gzip > dump.gz >> _______________________________________________ >> boxbackup mailing list >> boxbackup at boxbackup.org >> http://lists.warhead.org.uk/mailman/listinfo/boxbackup >> >> !DSPAM:4a2b778b306301804284693! >> > Hi Anton, > > You shouldn't gzip it, because then Box Backup cannot efficiently > backup the file. Because it then needs to diff a binary file instead > of a text file. It depends how you want to handle the dumps really. I have my PostgreSQL servers do a nightly dump with a timestamp in the dump name (e.g. pgdump_dbname_yyyy-mm-dd-HH:MM:SS). I actually do the dump in the postgres binary format anyway, which is compressed by default. I like having a nightly backup of each database, so I'm not bothered about having diffs of the same SQL dump. As Chris pointed out on the -dev list recently [1], too, another option is to use pg_start_backup() as a pre-backup command and just backup the actual database cluster rather than dumping out. James [1] http://lists.warhead.org.uk/pipermail/boxbackup-dev/2009-June/002457.html From boxbackup at boxbackup.org Tue Jun 9 23:19:44 2009 From: boxbackup at boxbackup.org (Anton Bogdanovitch) Date: Wed, 10 Jun 2009 02:19:44 +0400 Subject: [Box Backup] Re: Re: Re: boxbackup + pg_dump In-Reply-To: <70A55826-EC89-42DF-93BE-61D5A021DBDD@netinertia.co.uk> References: <4A2B7720.6020602@gmail.com> <4A2BC2FC.2010301@hostingbrothers.nl> <70A55826-EC89-42DF-93BE-61D5A021DBDD@netinertia.co.uk> Message-ID: <4A2EE000.60708@gmail.com> James O'Gorman wrote: > It depends how you want to handle the dumps really. I have my PostgreSQL > servers do a nightly dump with a timestamp in the dump name (e.g. > pgdump_dbname_yyyy-mm-dd-HH:MM:SS). I actually do the dump in the > postgres binary format anyway, which is compressed by default. > > I like having a nightly backup of each database, so I'm not bothered > about having diffs of the same SQL dump. I have 13Gb dump (3.5Gb gzipped) and growing. I was worried about huge file backup capability and diff operations cpu usage. I just tried plain dump and it seems it was transferred perfectly. I still have to check how boxbackup will handle updated dump. > As Chris pointed out on the -dev list recently [1], too, another option > is to use pg_start_backup() as a pre-backup command and just backup the > actual database cluster rather than dumping out. > > James > > [1] > http://lists.warhead.org.uk/pipermail/boxbackup-dev/2009-June/002457.html Has anyone tried to perform postgres backup this way? From boxbackup at boxbackup.org Mon Jun 22 03:43:04 2009 From: boxbackup at boxbackup.org (Ben Bennett) Date: Sun, 21 Jun 2009 22:43:04 -0400 Subject: [Box Backup] Hard Links & Ignoring directories Message-ID: <20090622024304.GA18203@ayup.limey.net> Thanks for the excellent software. I have two odd questions after upgrading to a recent trunk version of boxbackup. Now I see: Jun 21 22:23:42 ayup bbackupd[24222]: WARNING: Found conflicting parent ID for file ID 1579976 (/home/fiji/public_html/pictures/49/014 - Nursery.jpg): expected 34689 but found 69964 (same directory used in two different locations?) The warning is correct, the file is in two locations... > ls -li '/home/fiji/public_html/pictures/49/013 - Nursery.jpg' '/home/fiji/public_html/family/3/013 - Nursery.jpg' 1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/family/3/013 - Nursery.jpg 1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/pictures/49/013 - Nursery.jpg I see no reason why hard links should be discouraged... if box can not store that relationship, fine (not great though). But how can I at least supress the warning? Question the second. I keep getting: Jun 21 22:24:56 ayup bbackupd[24222]: WARNING: Failed to access file: /home/jb/.gvfs: Permission denied Which while true is irritating because I have: BackupLocations { home { Path = /home ExcludeFilesRegex = ^/home/[^/]+/(\.procmail/log|\.bash_history|dead\.letter|\.Xauthority|\.ICEauthority|\.xsession-errors) ExcludeFilesRegex = ^/home/[^/]+/[Mm]ail/(SPAM|Trash) ExcludeDirsRegex = ^/home/[^/]+/(\.razor|\.spamassassin|(mail/)?\.imap|\.emacs\.d/auto-save-list|\.mozilla|\.links|\.gconf|\.gnome|\.xmms|\.ssh|\.gvfs) ExcludeDirsRegex = ^/home/(mvm|harrison)/(log|archive) ExcludeDir = /home/sskoog/backups ExcludeDir = /home/fiji/dvdrip-data ExcludeFile = /home/mcmanus/named/ducksong.current ExcludeFilesRegex = ^/etc/hosts\.deny(\.purge\.bak)? } } And that should match: ^/home/[^/]+/(\.razor|\.spamassassin|(mail/)?\.imap|\.emacs\.d/auto-save-list|\.mozilla|\.links|\.gconf|\.gnome|\.xmms|\.ssh|\.gvfs) BTW, is there any advantage to combining it all into one regexp, or should I just have multiple lines? Thanks! -ben From boxbackup at boxbackup.org Mon Jun 22 06:34:25 2009 From: boxbackup at boxbackup.org (Chris Wilson) Date: Mon, 22 Jun 2009 08:34:25 +0300 (EAT) Subject: [Box Backup] Hard Links & Ignoring directories In-Reply-To: <20090622024304.GA18203@ayup.limey.net> References: <20090622024304.GA18203@ayup.limey.net> Message-ID: Hi Ben, On Sun, 21 Jun 2009, Ben Bennett wrote: > Thanks for the excellent software. > > I have two odd questions after upgrading to a recent trunk version of > boxbackup. Now I see: > > Jun 21 22:23:42 ayup bbackupd[24222]: WARNING: Found conflicting parent > ID for file ID 1579976 (/home/fiji/public_html/pictures/49/014 - > Nursery.jpg): expected 34689 but found 69964 (same directory used in two > different locations?) > > The warning is correct, the file is in two locations... > >> ls -li '/home/fiji/public_html/pictures/49/013 - Nursery.jpg' '/home/fiji/public_html/family/3/013 - Nursery.jpg' > 1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/family/3/013 - Nursery.jpg > 1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/pictures/49/013 - Nursery.jpg > > I see no reason why hard links should be discouraged... if box can not > store that relationship, fine (not great though). But how can I at > least supress the warning? I agree that hard links should be supported, and that this is a bug. Please could you get an account in the bug tracker and file a bug? The reasons for this warning are: 1. Almost nobody uses hard links on their data files, so nobody has reported this problem before. 2. Including the same files in two backup locations (not two directories) is an error and a waste of bandwidth. 3. There was a bug on Windows where file IDs were being duplicated all the time, which was only spotted because I added this warning. 4. Box Backup uses file IDs to track renaming of files. If you use hard links then this will not work correctly and will cause your files to be renamed between the two directories all the time. This is the real bug. > Question the second. I keep getting: Jun 21 22:24:56 ayup > bbackupd[24222]: WARNING: Failed to access file: /home/jb/.gvfs: > Permission denied > > Which while true is irritating because I have: Thanks for the report. Should be fixed now. Please could you try updating to the latest trunk version and see if that fixes this problem? > BTW, is there any advantage to combining it all into one regexp, or > should I just have multiple lines? It's very slightly more efficient, but much less readable, so unless you can see a strong CPU time advantage I'd advise you to use separate ExcludeDir entries instead. This also doesn't require you to use regular expressions to compare them, so it may even be faster. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at boxbackup.org Tue Jun 23 17:50:24 2009 From: boxbackup at boxbackup.org (Achim) Date: Tue, 23 Jun 2009 18:50:24 +0200 Subject: [Box Backup] Additional information for Volume Snapshot Service (Shadow Copy) Message-ID: Hello list: In my quest to contribute to solve one of the major tickets under Windows [0], I managed to locate some code that was originally developed for rsync 3.0 [1] and seems to be pretty straightforward. The code referenced above was developed by Leen Besselink who in turn used code from Elias Penttil??. I checked with both authors and both have given their OK to reuse their code under the Box Backup license (see messages below). Chris, is this something that you would want to look at in order to add VSS to Box Backup? The patch seems very simple to my untrained eye and could probably be used in Box without too much modification, as it only adds the function convert_to_vss_path() which creates a VSS copy for a given path and returns the actual VSS path that can then be used to access the shadow copy: all the magic is done within the VSS libraries. FYI, I tested the process outlined in [1] to compile a VSS-aware copy of rsync, and it work flawlessly. I just do not know where to include that capability in the Box code. Am I correct in assuming that the licensing mentioned below (with respect to the GPL'ed Box Backup) should not be an issue with the BSD-style Box license? I hope this gives VSS another important push, in addition to the recent XP license donated by Per Reedtz Thomsen. Best regards, Achim [0] http://www.boxbackup.org/trac/ticket/13 [1] http://www.consolejunky.net/cwrsync-vss/ -------- Original Message -------- Subject: Re: [Fwd: Re: rsync 3.0.x with Windows VSS-support] Date: Tue, 16 Jun 2009 13:51:37 +0300 From: Elias Penttil?? To: "Achim J. Latz" Achim J. Latz wrote: > Hello Elias: > > As you can see from the messages included below, we are trying to use > some of your code in certain backup projects in order to offer VSS > capability. > > Could you please indicate whether you are fine with re-licensing your > code under BSD-like and GPL licenses? According to Leen Besselink you > have not yet responded to his enquiries, so I would like to propose that > I take a non-answer in the next 14 days as a "green light", OK? > > Thank you very much in advance, Achim Sure, go ahead, I don't mind. Do note that there was some discussion on the mailing list back in 2006 about the compatibility of the rsync license and the VSS SDK license. I have no idea if that matters, but I myself took down the precompiled versions then. -------- Original Message -------- Subject: Re: rsync 3.0.x with Windows VSS-support Date: Sat, 06 Jun 2009 20:40:10 +0200 From: Leen Besselink To: Achim J. Latz Achim J. Latz wrote: > Hello Leen: > Hi Achim, > I just found your excellent post on VSS under rsync (included below) on > the BackupPC mailing list. > > This functionality and your very clear build instructions could also be > a very valuable additional for another excellent free backup solution > called Box Backup [0], as you can see from this open ticket [1]. The > license for Box Backup is [2] BSD-like in spirit. > > I wanted to ask you if you would be willing to accept that license in > order to include some of your code in Box Backup for the benefit of its > users? > Most of the code is still from Elias Penttil??. I've sent him an e-mail ones and got no reply. As I'm doing the same thing as he was doing with it (integrating it with rsync), I didn't think it could be a big problem for what I'm doing with it, without his approval. It could also be his code is partly from examples by Microsoft, because there is example code available. I personally wouldn't mind if you took some of the code, but I'm not really the main contributor to it at the moment. I guess I'm more of maintainer. > Thank you very much in advance for your consideration, Achim > > [0] http://www.boxbackup.org/ > [1] http://www.boxbackup.org/trac/ticket/13 > [2] http://www.boxbackup.org/trac/wiki/BoxLicense From boxbackup at boxbackup.org Tue Jun 23 18:16:25 2009 From: boxbackup at boxbackup.org (Achim) Date: Tue, 23 Jun 2009 19:16:25 +0200 Subject: [Box Backup] Additional information for Volume Snapshot Service (Shadow Copy) In-Reply-To: References: Message-ID: <44626ea4051ea5e8b3309024b941712d@localhost> Hello again: Some more details on the licensing issue: On Tue, 23 Jun 2009 18:50:24 +0200, Achim wrote: > Am I correct in assuming that the licensing mentioned below (with respect > to the GPL'ed Box Backup) should not be an issue with the BSD-style Box > license? That was of course meant to read "(with respect to the GPL'ed rsync)". Meanwhile, in a licensing discussion over at the BackupPC list, it turns out (according to Leen, one of the original authors) that perhaps the "SDK just includes the include and library-files the compiler needs to know how to build it" [1]. This would probably eliminate the issue anyway for GPL and BSD, as the libraries are then part of the OS and not a separate download. Best regards, Achim [0] From boxbackup at boxbackup.org Wed Jun 24 07:14:41 2009 From: boxbackup at boxbackup.org (Chris Wilson) Date: Wed, 24 Jun 2009 09:14:41 +0300 (EAT) Subject: [Box Backup] Additional information for Volume Snapshot Service (Shadow Copy) In-Reply-To: <44626ea4051ea5e8b3309024b941712d@localhost> References: <44626ea4051ea5e8b3309024b941712d@localhost> Message-ID: Hi Achim, On Tue, 23 Jun 2009, Achim wrote: > Meanwhile, in a licensing discussion over at the BackupPC list, it turns > out (according to Leen, one of the original authors) that perhaps the > "SDK just includes the include and library-files the compiler needs to > know how to build it" [1]. > > This would probably eliminate the issue anyway for GPL and BSD, as the > libraries are then part of the OS and not a separate download. It doesn't eliminate the issue for GPL, because Microsoft's license explicitly forbids linking any of their code against GPL code and distributing the resulting binaries. However, BSD code does not fall under that restriction. It's against the spirit of the GPL because it requires users to download and use non-free software to rebuild their free software, in addition to the non-free OS which they already have. Cheers, Chris. -- _ ___ __ _ / __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer | \ _/_/_/_//_/___/ | We are GNU-free your mind-and your software | From boxbackup at boxbackup.org Wed Jun 24 18:54:47 2009 From: boxbackup at boxbackup.org (Matthieu Patou) Date: Wed, 24 Jun 2009 21:54:47 +0400 Subject: [Box Backup] Additional information for Volume Snapshot Service (Shadow Copy) In-Reply-To: References: <44626ea4051ea5e8b3309024b941712d@localhost> Message-ID: <4A426867.20501@matws.net> On 06/24/2009 10:14 AM, Chris Wilson wrote: > Hi Achim, > > On Tue, 23 Jun 2009, Achim wrote: > >> Meanwhile, in a licensing discussion over at the BackupPC list, it >> turns out (according to Leen, one of the original authors) that >> perhaps the "SDK just includes the include and library-files the >> compiler needs to know how to build it" [1]. >> >> This would probably eliminate the issue anyway for GPL and BSD, as the >> libraries are then part of the OS and not a separate download. > > It doesn't eliminate the issue for GPL, because Microsoft's license > explicitly forbids linking any of their code against GPL code and > distributing the resulting binaries. However, BSD code does not fall > under that restriction. > > It's against the spirit of the GPL because it requires users to download > and use non-free software to rebuild their free software, in addition to > the non-free OS which they already have. > > Cheers, Chris. I'm wondering how bacula do it with this limitation ? Because they are compiling with this library and their code is GPL. I reread the licence for the VSS SDK and it's not really clear the licence for those two libes in the lib directory (vss_uuid.lib and vss_api.lib) Matthieu. From boxbackup at boxbackup.org Mon Jun 29 09:10:45 2009 From: boxbackup at boxbackup.org (cheryl.gutierrez14@yahoo.com) Date: Mon, 29 Jun 2009 08:10:45 +0000 Subject: [Box Backup] Re confirmation Message-ID: <20090629081046.701E1325020@www.boxbackup.org> Confirm 889379 ---------- Sent via Cricket Mobile Email From boxbackup at boxbackup.org Mon Jun 29 09:14:28 2009 From: boxbackup at boxbackup.org (Chris Wilson) Date: Mon, 29 Jun 2009 09:14:28 +0100 (BST) Subject: [Box Backup] Re confirmation In-Reply-To: <20090629081046.701E1325020@www.boxbackup.org> References: <20090629081046.701E1325020@www.boxbackup.org> Message-ID: On Mon, 29 Jun 2009, cheryl.gutierrez14 at yahoo.com wrote: > Confirm 889379 Confirmed. Wait, what? -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at boxbackup.org Mon Jun 29 12:02:48 2009 From: boxbackup at boxbackup.org (Achim) Date: Mon, 29 Jun 2009 13:02:48 +0200 Subject: [Box Backup] Additional information for Volume Snapshot Service (Shadow Copy) In-Reply-To: <4A426867.20501@matws.net> References: <44626ea4051ea5e8b3309024b941712d@localhost> <4A426867.20501@matws.net> Message-ID: <377703ed2a4e27d81d4bea2f0b8f7f18@localhost> Hello List: On Wed, 24 Jun 2009 21:54:47 +0400, Matthieu Patou wrote: > On 06/24/2009 10:14 AM, Chris Wilson wrote: >> Hi Achim, >> >> On Tue, 23 Jun 2009, Achim wrote: >> >>> Meanwhile, in a licensing discussion over at the BackupPC list, it >>> turns out (according to Leen, one of the original authors) that >>> perhaps the "SDK just includes the include and library-files the >>> compiler needs to know how to build it" [1]. >>> >>> This would probably eliminate the issue anyway for GPL and BSD, as the >>> libraries are then part of the OS and not a separate download. >> >> It doesn't eliminate the issue for GPL, because Microsoft's license >> explicitly forbids linking any of their code against GPL code and >> distributing the resulting binaries. However, BSD code does not fall >> under that restriction. >> >> It's against the spirit of the GPL because it requires users to download >> and use non-free software to rebuild their free software, in addition to >> the non-free OS which they already have. >> >> Cheers, Chris. > > I'm wondering how bacula do it with this limitation ? > Because they are compiling with this library and their code is GPL. > > I reread the licence for the VSS SDK and it's not really clear the > licence for those two libes in the lib directory (vss_uuid.lib and > vss_api.lib) As I mentioned in another posting back in March 2009 [1], the Bacula project gets around the GPL "restriction" (actually: liberations) by adding a special clause that allows linking their GPL code with proprietary libraries. A similar topic came up recently on the Bacula list again [2], and it appears that perhaps there will be some progress towards having Free, reverse-engineered header files. On the MingW list there is yet another discussion on how to go about such a reverse-engineering process, and I can see that Chris is actively participating [3]. It would be great if all the pieces would fall into place at some point, however I believe that a practical "in the meantime" approach would be a licensing exception, which, as Chris confirmed, would actually not even be required for Box Backup's BSD-style license. Best regards, Achim [1] [2] [3] From boxbackup at boxbackup.org Mon Jun 29 13:28:41 2009 From: boxbackup at boxbackup.org (Matthieu Patou) Date: Mon, 29 Jun 2009 16:28:41 +0400 Subject: [Box Backup] Additional information for Volume Snapshot Service (Shadow Copy) In-Reply-To: <377703ed2a4e27d81d4bea2f0b8f7f18@localhost> References: <44626ea4051ea5e8b3309024b941712d@localhost> <4A426867.20501@matws.net> <377703ed2a4e27d81d4bea2f0b8f7f18@localhost> Message-ID: <4A48B379.2010000@matws.net> >>> It doesn't eliminate the issue for GPL, because Microsoft's license >>> explicitly forbids linking any of their code against GPL code and >>> distributing the resulting binaries. However, BSD code does not fall >>> under that restriction. >>> >>> It's against the spirit of the GPL because it requires users to download >>> and use non-free software to rebuild their free software, in addition to >>> the non-free OS which they already have. >>> >>> Cheers, Chris. >> I'm wondering how bacula do it with this limitation ? >> Because they are compiling with this library and their code is GPL. >> >> I reread the licence for the VSS SDK and it's not really clear the >> licence for those two libes in the lib directory (vss_uuid.lib and >> vss_api.lib) > > As I mentioned in another posting back in March 2009 [1], the Bacula > project gets around the GPL "restriction" (actually: liberations) by adding > a special clause that allows linking their GPL code with proprietary > libraries. > > A similar topic came up recently on the Bacula list again [2], and it > appears that perhaps there will be some progress towards having Free, > reverse-engineered header files. On the MingW list there is yet another > discussion on how to go about such a reverse-engineering process, and I can > see that Chris is actively participating [3]. > > It would be great if all the pieces would fall into place at some point, > however I believe that a practical "in the meantime" approach would be a > licensing exception, which, as Chris confirmed, would actually not even be > required for Box Backup's BSD-style license. > Achim my question was not in the sense of problems of GPL with proprietary software but the opposite. Chris stated that the Microsoft's license explicitly forbids linking any of their codes against GPL. First I read the VSS licence and I didn't find this (but I am not a lawyer so I might have misunderstood some parts), and if it's true then I am wondering how Bacula manage to respect Microsoft license. Matthieu. From boxbackup at boxbackup.org Mon Jun 29 21:45:33 2009 From: boxbackup at boxbackup.org (Chris Wilson) Date: Mon, 29 Jun 2009 21:45:33 +0100 (BST) Subject: [Box Backup] Additional information for Volume Snapshot Service (Shadow Copy) In-Reply-To: <4A48B379.2010000@matws.net> References: <44626ea4051ea5e8b3309024b941712d@localhost> <4A426867.20501@matws.net> <377703ed2a4e27d81d4bea2f0b8f7f18@localhost> <4A48B379.2010000@matws.net> Message-ID: Hi Matthieu, On Mon, 29 Jun 2009, Matthieu Patou wrote: >>>> It doesn't eliminate the issue for GPL, because Microsoft's license >>>> explicitly forbids linking any of their code against GPL code and >>>> distributing the resulting binaries. However, BSD code does not fall >>>> under that restriction. >>> >>> I'm wondering how bacula do it with this limitation ? >>> Because they are compiling with this library and their code is GPL. >> >> As I mentioned in another posting back in March 2009 [1], the Bacula >> project gets around the GPL "restriction" (actually: liberations) by >> adding a special clause that allows linking their GPL code with >> proprietary libraries. >> > Achim my question was not in the sense of problems of GPL with > proprietary software but the opposite. Chris stated that the Microsoft's > license explicitly forbids linking any of their codes against GPL. First > I read the VSS licence and I didn't find this (but I am not a lawyer so > I might have misunderstood some parts), and if it's true then I am > wondering how Bacula manage to respect Microsoft license. Microsoft's license says that you may not distribute any of their code linked to any applications that would cause their code to fall under the GPL (or any "Excluded License"). Because Bacula has an exception for these proprietary libraries, that means that it does not force them to fall under the GPL, so Microsoft's own license still applies, and that's OK under their license. They don't have a problem with linking their stuff to LGPL code either, for the same reason. See the Silverlight SDK license at: http://msdn.microsoft.com/en-us/silverlight/bb743266.aspx . I think that essentially the same license applies to the VSS SDK. Linking non-free code to GPL code is actually forbidden under the usual GPL for the same reason, that it would force that code to fall under the GPL, and if you can't distribute the entire code under the GPL then you can't distribute it at all. However, copyright owners are entitled to create a new license based on the GPL, as Bacula have apparently done, and to release their code under that license if they wish. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at boxbackup.org Mon Jun 29 23:47:05 2009 From: boxbackup at boxbackup.org (Chris Wilson) Date: Mon, 29 Jun 2009 23:47:05 +0100 (BST) Subject: [Box Backup] Virtual Machine backup In-Reply-To: <4A2D759E.7020301@logical-progress.com> References: <4A262A47.9090801@logical-progress.com> <4A26EB58.5090204@logical-progress.com> <4A292072.1080907@logical-progress.com> <4A2D759E.7020301@logical-progress.com> Message-ID: Hi Dave, On Mon, 8 Jun 2009, Dave Bamford wrote: > First I create a virtual machine (XP pro in this case) with a 2G hard > drive. > > Then copy it via NFS to another machine running box client > and let it back up over the net to one of my servers. [...] > Then I make some changes to the vm (I add vmware tools support) > > replace the files on the nfs drive with the new ones of the same name > > Box back it up but only the differences if I have interpreted it correctly. > >> Jun 8 19:05:02 backup6 bbackupd[18765]: NOTICE: Beginning scan of >> local files >> Jun 8 19:33:15 backup6 bbackupd[18765]: NOTICE: Finished scan of local >> files >> Jun 8 19:33:15 backup6 bbackupd[18765]: NOTICE: File statistics: total >> file size uploaded 2185862686, bytes already on server 1720804357, >> encoded size 210338595 > > So now we have a means of "bare metal backup" with box. I believe that your understanding is correct. bbackupd "only" uploaded 210 MB of differences in this case. That's still quite a large diff, though (about 10% of the total disk image size). Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup at boxbackup.org Mon Jun 8 17:10:40 2009 From: boxbackup at boxbackup.org (Andres Fernandez Garcia) Date: Mon, 8 Jun 2009 13:10:40 -0300 Subject: [Box Backup] Multiple BoxBackup client on a PC In-Reply-To: <20090608110001.1327.94701.Mailman@fear.love.warhead.org.uk> References: <20090608110001.1327.94701.Mailman@fear.love.warhead.org.uk> Message-ID: <135B88EBD4D0412CB43699C22FC24B9E@PCFIJOADV1> Hi list! I have installed bb rc3.2502 client on a PC, this client install is working good, but I need install a new client for another Redundancy = Server. When I try to install the Windows service on the second path, say me = that: "it already exist" the service. The second path is logical diferent to = the first. Some idea? Thanks everyone. Andres -----Mensaje original----- De: boxbackup-admin at boxbackup.org [mailto:boxbackup-admin at boxbackup.org] = En nombre de boxbackup-request at boxbackup.org Enviado el: lunes, 08 de junio de 2009 8:00 Para: boxbackup at boxbackup.org Asunto: boxbackup digest, Vol 1 #1245 - 1 msg Send boxbackup mailing list submissions to boxbackup at boxbackup.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.warhead.org.uk/mailman/listinfo/boxbackup or, via email, send a message with subject or body 'help' to boxbackup-request at boxbackup.org You can reach the person managing the list at boxbackup-admin at boxbackup.org When replying, please edit your Subject line so it is more specific than "Re: Contents of boxbackup digest..." Today's Topics: 1. Re: Re: boxbackup + pg_dump (Roy) --__--__-- Message: 1 Date: Sun, 07 Jun 2009 15:39:08 +0200 From: Roy To: boxbackup at boxbackup.org Subject: Re: [Box Backup] Re: boxbackup + pg_dump Reply-To: boxbackup at boxbackup.org Anton Bogdanovitch wrote: > Sorry. There was a typo. > > 1) pg_dumpall > dump.sql > 2) pg_dumpall > dump.sql | gzip > dump.gz > _______________________________________________ > boxbackup mailing list > boxbackup at boxbackup.org > http://lists.warhead.org.uk/mailman/listinfo/boxbackup > > !DSPAM:4a2b778b306301804284693! > Hi Anton, You shouldn't gzip it, because then Box Backup cannot efficiently backup = the file. Because it then needs to diff a binary file instead of a text=20 file. Greets, Roy --__--__-- _______________________________________________ boxbackup mailing list boxbackup at boxbackup.org http://lists.warhead.org.uk/mailman/listinfo/boxbackup End of boxbackup Digest __________ Informaci=F3n de ESET NOD32 Antivirus, versi=F3n de la base = de firmas de virus 4137 (20090608) __________ ESET NOD32 Antivirus ha comprobado este mensaje. http://www.eset.com =20 __________ Informaci=F3n de ESET NOD32 Antivirus, versi=F3n de la base = de firmas de virus 4137 (20090608) __________ ESET NOD32 Antivirus ha comprobado este mensaje. http://www.eset.com =20