From boxbackup at fluffy.co.uk  Wed Aug  1 11:26:51 2007
From: boxbackup at fluffy.co.uk (Nuno Fernandes)
Date: Wed, 1 Aug 2007 11:26:51 +0100
Subject: [Box Backup] Question about certificates
In-Reply-To: <20070731151540.2c021ea2@pc-buna.former03.local>
References: <200707311050.15027.npf-mlists@eurotux.com> <20070731151540.2c021ea2@pc-buna.former03.local>
Message-ID: <200708011126.51675.npf-mlists@eurotux.com>

Hi,

On Tuesday 31 July 2007 14:15:40 Baltasar Cevc wrote:
> On Tue, 31 Jul 2007 10:50:14 +0100
>
> Nuno Fernandes <npf-mlists at eurotux.com> wrote:
> > Hi,
> >
> > I have a question about managing certificates. We have a company CA
> > and we've created a sub-ca just for boxbackup.
> >
> > How can we use it in bbstored-certs script?
>
> I assume that it should be enough to replace the files in the
> CA directory by your CAs (see below about the CAs), I'm not
> sure though.
>
> > Aparently
> > bbstored-certs /etc/box/bbstored/certs init
> > creates 2 root CAs (one for clients and the other for servers). Why
> > does it create 2 CAs?
>
> One is for validating servers, the other for validating clients.
> I think servers are just accepted as valid if they present a
> valid certificate signed by the server CA.
> For clients, the CN must match - I think it must
> BACKUP-<account number> (without zeros at the beginning), the
> certificate being signed by the client CA.
Can't i use the same CA to validate servers and clients?


Thanks
Nuno Fernandes

>
> Hope that helps,
> Baltasar
> _______________________________________________
> boxbackup mailing list
> boxbackup at fluffy.co.uk
> http://lists.warhead.org.uk/mailman/listinfo/boxbackup


From boxbackup at fluffy.co.uk  Wed Aug  1 19:25:00 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Wed, 1 Aug 2007 19:25:00 +0100 (BST)
Subject: [Box Backup] Question about certificates
In-Reply-To: <200708011126.51675.npf-mlists@eurotux.com>
References: <200707311050.15027.npf-mlists@eurotux.com>
 <20070731151540.2c021ea2@pc-buna.former03.local> <200708011126.51675.npf-mlists@eurotux.com>
Message-ID: <Pine.LNX.4.64.0708011923370.23507@top.qwarx.com>

Hi Nuno,

On Wed, 1 Aug 2007, Nuno Fernandes wrote:

>>> Aparently bbstored-certs /etc/box/bbstored/certs init creates 2 root 
>>> CAs (one for clients and the other for servers). Why does it create 2 
>>> CAs?
>>
>> One is for validating servers, the other for validating clients. I 
>> think servers are just accepted as valid if they present a valid 
>> certificate signed by the server CA. For clients, the CN must match - I 
>> think it must BACKUP-<account number> (without zeros at the beginning), 
>> the certificate being signed by the client CA.
>
> Can't i use the same CA to validate servers and clients?

You can, but it's not secure. It allows one of your clients to pretend to 
be a valid server for any other client.

If you really want to do that, just set the ServerCA (on the client) and 
ClientCA (on the server) to point to the same certificate.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Thu Aug  2 17:50:47 2007
From: boxbackup at fluffy.co.uk (Nuno Fernandes)
Date: Thu, 2 Aug 2007 17:50:47 +0100
Subject: [Box Backup] Question about certificates
In-Reply-To: <Pine.LNX.4.64.0708011923370.23507@top.qwarx.com>
References: <200707311050.15027.npf-mlists@eurotux.com> <200708011126.51675.npf-mlists@eurotux.com> <Pine.LNX.4.64.0708011923370.23507@top.qwarx.com>
Message-ID: <200708021750.47167.npf-mlists@eurotux.com>

Hi,

On Wednesday 01 August 2007 19:25:00 Chris Wilson wrote:
> Hi Nuno,
>
> On Wed, 1 Aug 2007, Nuno Fernandes wrote:
> >>> Aparently bbstored-certs /etc/box/bbstored/certs init creates 2 root
> >>> CAs (one for clients and the other for servers). Why does it create 2
> >>> CAs?
> >>
> >> One is for validating servers, the other for validating clients. I
> >> think servers are just accepted as valid if they present a valid
> >> certificate signed by the server CA. For clients, the CN must match - I
> >> think it must BACKUP-<account number> (without zeros at the beginning),
> >> the certificate being signed by the client CA.
> >
> > Can't i use the same CA to validate servers and clients?
>
> You can, but it's not secure. It allows one of your clients to pretend to
> be a valid server for any other client.
It's not secure? Why not? A client can only pretend to be a server with the 
name BACKUP-X where X is the client number. If another client would connect 
to server1.domain.com and a client would only have a certificate with the 
common name of BACKUP-X and not server1.domain.com.

Rgds
Nuno Fernandes


> If you really want to do that, just set the ServerCA (on the client) and
> ClientCA (on the server) to point to the same certificate.
>
> Cheers, Chris.


From boxbackup at fluffy.co.uk  Thu Aug  2 19:50:07 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Thu, 2 Aug 2007 19:50:07 +0100 (BST)
Subject: [Box Backup] Question about certificates
In-Reply-To: <200708021750.47167.npf-mlists@eurotux.com>
References: <200707311050.15027.npf-mlists@eurotux.com>
 <200708011126.51675.npf-mlists@eurotux.com> <Pine.LNX.4.64.0708011923370.23507@top.qwarx.com>
 <200708021750.47167.npf-mlists@eurotux.com>
Message-ID: <Pine.LNX.4.64.0708021948090.23507@top.qwarx.com>

Hi Nuno,

On Thu, 2 Aug 2007, Nuno Fernandes wrote:

>>> Can't i use the same CA to validate servers and clients?
>>
>> You can, but it's not secure. It allows one of your clients to pretend to
>> be a valid server for any other client.
>
> It's not secure? Why not? A client can only pretend to be a server with the
> name BACKUP-X where X is the client number. If another client would connect
> to server1.domain.com and a client would only have a certificate with the
> common name of BACKUP-X and not server1.domain.com.

I'm not 100% sure, but I don't think the client verifies the CN of the 
server certificate at all, except that it was signed by the expected CA. 
So it doesn't matter that the server has a "name" of BACKUP-1 or anything 
else, as long as it was signed by the ServerCA, which in your case would 
be the same as the ClientCA.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Thu Aug  2 19:51:56 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Thu, 2 Aug 2007 19:51:56 +0100 (BST)
Subject: [Box Backup] Holiday
Message-ID: <Pine.LNX.4.64.0708021950430.23507@top.qwarx.com>

Hi all,

I'm going on holiday (to Mexico) on the 6th August and won't be back until 
the 24th. So if you have any questions about BB that urgently need my 
help, or any feedback to give, please send them ASAP.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Fri Aug  3 08:50:02 2007
From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen)
Date: Fri, 03 Aug 2007 09:50:02 +0200
Subject: [Box Backup] Holiday
In-Reply-To: <Pine.LNX.4.64.0708021950430.23507@top.qwarx.com>
References: <Pine.LNX.4.64.0708021950430.23507@top.qwarx.com>
Message-ID: <46B2DE2A.3060308@kontrapunkt.com>

Hello Chris.

Enjoy the trip!

Tobias

Chris Wilson wrote:
> Hi all,
> 
> I'm going on holiday (to Mexico) on the 6th August and won't be back
> until the 24th. So if you have any questions about BB that urgently need
> my help, or any feedback to give, please send them ASAP.
> 
> Cheers, Chris.



From boxbackup at fluffy.co.uk  Fri Aug  3 13:27:20 2007
From: boxbackup at fluffy.co.uk (Baltasar Cevc)
Date: Fri, 3 Aug 2007 14:27:20 +0200
Subject: [Box Backup] Question about certificates
In-Reply-To: <Pine.LNX.4.64.0708021948090.23507@top.qwarx.com>
References: <200707311050.15027.npf-mlists@eurotux.com>
 <200708011126.51675.npf-mlists@eurotux.com>
 <Pine.LNX.4.64.0708011923370.23507@top.qwarx.com>
 <200708021750.47167.npf-mlists@eurotux.com>
 <Pine.LNX.4.64.0708021948090.23507@top.qwarx.com>
Message-ID: <20070803142720.0649be74@pc-buna.former03.local>

Hi Chris, hi Nuno,

On Thu, 2 Aug 2007 19:50:07 +0100 (BST)
Chris Wilson <chris at qwirx.com> wrote:

> > It's not secure? Why not? A client can only pretend to be a server
> > with the name BACKUP-X where X is the client number. If another
> > client would connect to server1.domain.com and a client would only
> > have a certificate with the common name of BACKUP-X and not
> > server1.domain.com.
> 
> I'm not 100% sure, but I don't think the client verifies the CN of
> the server certificate at all, except that it was signed by the
> expected CA. So it doesn't matter that the server has a "name" of
> BACKUP-1 or anything else, as long as it was signed by the ServerCA,
> which in your case would be the same as the ClientCA.

I'm positive it _does not_ verify the name (as you write). We've
often used the name of the NAT machine instead of the internal name
(which was the CN of the certificate) and it did work perfectly.

Baltasar
 


From boxbackup at fluffy.co.uk  Fri Aug  3 15:20:31 2007
From: boxbackup at fluffy.co.uk (Nuno Fernandes)
Date: Fri, 3 Aug 2007 15:20:31 +0100
Subject: [Box Backup] Question about certificates
In-Reply-To: <Pine.LNX.4.64.0708021948090.23507@top.qwarx.com>
References: <200707311050.15027.npf-mlists@eurotux.com> <200708021750.47167.npf-mlists@eurotux.com> <Pine.LNX.4.64.0708021948090.23507@top.qwarx.com>
Message-ID: <200708031520.32281.npf-mlists@eurotux.com>

Hi Chris,

On Thursday 02 August 2007 19:50:07 Chris Wilson wrote:
> Hi Nuno,
>
> On Thu, 2 Aug 2007, Nuno Fernandes wrote:
> >>> Can't i use the same CA to validate servers and clients?
> >>
> >> You can, but it's not secure. It allows one of your clients to pretend
> >> to be a valid server for any other client.
> >
> > It's not secure? Why not? A client can only pretend to be a server with
> > the name BACKUP-X where X is the client number. If another client would
> > connect to server1.domain.com and a client would only have a certificate
> > with the common name of BACKUP-X and not server1.domain.com.
>
> I'm not 100% sure, but I don't think the client verifies the CN of the
> server certificate at all, except that it was signed by the expected CA.
> So it doesn't matter that the server has a "name" of BACKUP-1 or anything
> else, as long as it was signed by the ServerCA, which in your case would
> be the same as the ClientCA.
=46rom http://www.fluffy.co.uk/boxbackup/server.html i can see in the serve=
r=20
configuration:
########
Server basic setup
=2E..
(set hostname to the address the clients will use to contact this server) A=
re=20
you using a NAT device or firewall? See the note below.
########

It creates a csr with the hostname as CN of the certificate. In
http://bbdev.fluffy.co.uk/trac/wiki/CertificatesAndAccountsManagement we ca=
n=20
read:

########
Sign a Server Certificate=20
 When you use the bbstored-config script to set up a config file for a serv=
er,=20
it will generate a certificate request (CSR) for you. Transfer it to the=20
machine with your CA, then do the following:=20
/usr/local/bin/bbstored-certs ca sign-server hostname-csr.pem

 which signs the certificate for the server. Follow the instructions in the=
=20
output on which files to install on the server. The CSR file is now no long=
er=20
needed. Make sure you run this command from the directory above the=20
directory 'ca'.
########

So it signs the server certificate with the valid CN as the server hostname=
=2E I=20
haven't read the source code, but apearently bbackupd validates CN when it=
=20
connects to bbstored.

Best rgds
Nuno Fernandes


From boxbackup at fluffy.co.uk  Fri Aug  3 17:10:42 2007
From: boxbackup at fluffy.co.uk (dave bamford)
Date: Fri, 03 Aug 2007 17:10:42 +0100
Subject: [Box Backup] Exchange Server and shadow copy stuff using box backup
Message-ID: <46B35382.6030108@logical-progress.com>

Hi,

I have a client who has a need for MS Exchange to be backed up via box 
backup.

He is rather insistent that we use box backup as it "has saved his bacon 
a few times" and doesn't want to move to anything else!

I've told his this is not a current feature and he has now responded and 
said what would it cost to make it happen.

Obviously I don't have the coding skills to do this.

Any suggestions as to potential costs of writing this assuming any new 
code remains under the existing "open source" licence for all to use?

I'm assuming it would be able to run in the background.

Dave


From boxbackup at fluffy.co.uk  Fri Aug  3 20:35:14 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Fri, 3 Aug 2007 20:35:14 +0100 (BST)
Subject: [Box Backup] Question about certificates
In-Reply-To: <200708031520.32281.npf-mlists@eurotux.com>
References: <200707311050.15027.npf-mlists@eurotux.com>
 <200708021750.47167.npf-mlists@eurotux.com> <Pine.LNX.4.64.0708021948090.23507@top.qwarx.com>
 <200708031520.32281.npf-mlists@eurotux.com>
Message-ID: <Pine.LNX.4.64.0708032031110.23507@top.qwarx.com>

Hi Nuno,

On Fri, 3 Aug 2007, Nuno Fernandes wrote:

>> I'm not 100% sure, but I don't think the client verifies the CN of the
>> server certificate at all, except that it was signed by the expected CA.
>> So it doesn't matter that the server has a "name" of BACKUP-1 or anything
>> else, as long as it was signed by the ServerCA, which in your case would
>> be the same as the ClientCA.
>
> From http://www.fluffy.co.uk/boxbackup/server.html i can see in the server
> configuration:
> ########
> Server basic setup
> ...
> (set hostname to the address the clients will use to contact this server) Are
> you using a NAT device or firewall? See the note below.
> ########

That note says:

########
The hostname specified is used for 1) the name in the server's certificate 
and 2) the address the server will listen on.

If the IP address of the machine isn't the same as the IP address it 
appears to have to the outside world (because the NAT device or firewall 
translates it), then this will fail. The server will look up the hostname, 
and then fail to bind to that address since it is not a local address.

To get around this, you have two options. Either specify the local IP 
address with the bbstored-config command (the name in the certificate 
won't match the real address, but this is not a problem at the moment), or 
specify the real address, but edit the bbstored.conf file and correct the 
ListenAddresses directive later to reflect the local address.
########

Note that:

1. The server normally listens on the address specified in the
    certificate. A bad server could choose to override that.

2. The client DOES NOT check the address in the certificate: "the name in
    the certificate won't match the real address, but this is not a problem
    at the moment"

> So it signs the server certificate with the valid CN as the server 
> hostname. I haven't read the source code, but apearently bbackupd 
> validates CN when it connects to bbstored.

No, I'm afraid it does not at the moment, and it would break NAT setups if 
it did.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Fri Aug  3 21:11:22 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Fri, 3 Aug 2007 21:11:22 +0100 (BST)
Subject: [Box Backup] Exchange Server and shadow copy stuff using box
 backup
In-Reply-To: <46B35382.6030108@logical-progress.com>
References: <46B35382.6030108@logical-progress.com>
Message-ID: <Pine.LNX.4.64.0708032106560.23507@top.qwarx.com>

Hi Dave,

> I have a client who has a need for MS Exchange to be backed up via box 
> backup.
>
> He is rather insistent that we use box backup as it "has saved his bacon 
> a few times" and doesn't want to move to anything else!
>
> I've told his this is not a current feature and he has now responded and 
> said what would it cost to make it happen.

It is possible to back up Exchange right now, but it is a bit convoluted. 
Basically you have to use MS Backup to create a Volume Shadow Service 
backup of Exchange, which is saved in a file, which Box Backup will back 
up.

Restore will then be a two-stage process: restore the backed-up Exchange 
backup file with Box, and then restore it over the Exchange installation 
with MS Backup.

There has been a discussion of implementing Volume Shadow Services backups 
within Box Backup itself, removing the need for the MS Backup step, but so 
far nobody has had time to do that.

> Any suggestions as to potential costs of writing this assuming any new 
> code remains under the existing "open source" licence for all to use?

I'm afraid I won't really have time to implement that this year. However, 
Charles Lecklider has expressed an interest and has the skills to do this. 
I believe that lack of time has prevented him from doing it so far, and a 
financial offer might well help to persuade him. Might I suggest that you 
contact him directly for a quote?

> I'm assuming it would be able to run in the background.

I guess that by "in the background" you mean "without shutting down 
Exchange", which is what I assumed above.

If you don't mind shutting down Exchange regularly then you can use Box 
Backup to back up its files without any problems while it's shut down.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Fri Aug  3 20:39:03 2007
From: boxbackup at fluffy.co.uk (Reinhard Tartler)
Date: Fri, 03 Aug 2007 21:39:03 +0200
Subject: [Box Backup] Re: Bug#435860: boxbackup-client: AlwaysInclude[File|Dir] is not working as expected
In-Reply-To: <20070803165959.GA9757@localhost> (Andreas Putzo's message of
 "Fri, 3 Aug 2007 18:59:59 +0200")
References: <20070803165959.GA9757@localhost>
Message-ID: <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>

forwarded 435860 boxbackup at fluffy.co.uk
tags 435860 upstream
stop

Hi boxbackup developers.

I tried to forwarded the following bug in the trac, but I keep on
getting the error: "Submission rejected as potential spam (Akismet says
content is spam)". I'm therefore forced to submit this bug via email.

This bug was originally received at the debian bug tracking
system. Please retain the CC in your answers.

Andreas Putzo <andreas at putzo.net> writes:

> Package: boxbackup-client
> Version: 0.10+really0.10-1
> Severity: important
>
> Hi,
>
> from the config file:
>
> # The exclude directives are of the form
> #
> #	[Exclude|AlwaysInclude][File|Dir][|sRegex] = regex or full pathname
> [..]
> # In general, Exclude excludes a file or directory, unless the directory is
> # explicitly mentioned in a AlwaysInclude directive.
>
> BackupLocations
> {
> 	home
> 		{
>             Path = /home/andreas
>             ExcludeDir = /home/andreas/chroot
>             AlwaysIncludeDir = /home/andreas/chroot/sid/home/andreas
> 		}
>
> }
>
> I expected that /home/andreas/chroot/sid/home/andreas would be
> included in the backup but this is not the case.
> boxbackup is running several days now so it should be there, even in
> lazy mode.
>
> Cheers, Andreas
>
>
> -- System Information:
> Debian Release: lenny/sid
>   APT prefers testing
>   APT policy: (500, 'testing'), (500, 'stable')
> Architecture: i386 (i686)
>
> Kernel: Linux 2.6.17.11+x40 (PREEMPT)
> Locale: LANG=de_DE at euro, LC_CTYPE=de_DE at euro (charmap=ISO-8859-15)
> Shell: /bin/sh linked to /bin/bash
>
> Versions of packages boxbackup-client depends on:
> ii  debconf [debconf-2.0] 1.5.13             Debian configuration management sy
> ii  libc6                 2.5-9+b1           GNU C Library: Shared libraries
> ii  libdb4.3              4.3.29-8           Berkeley v4.3 Database Libraries [
> ii  libedit2              2.9.cvs.20050518-3 BSD editline and history libraries
> ii  libgcc1               1:4.2-20070609-1   GCC support library
> ii  libssl0.9.8           0.9.8e-5           SSL shared libraries
> ii  libstdc++6            4.2-20070609-1     The GNU Standard C++ Library v3
> ii  openssl               0.9.8e-5           Secure Socket Layer (SSL) binary a
> ii  perl                  5.8.8-7            Larry Wall's Practical Extraction 
> ii  ucf                   3.001              Update Configuration File: preserv
> ii  zlib1g                1:1.2.3.3.dfsg-5   compression library - runtime
>
> boxbackup-client recommends no packages.
>
> -- debconf information:
>   boxbackup-client/MaxUploadWait: 86400
>   boxbackup-client/incorrectAccountNumber:
>   boxbackup-client/IncorrectNumber:
>   boxbackup-client/incorrectDirectories:
> * boxbackup-client/notifyMail: root
> * boxbackup-client/debconf: true
> * boxbackup-client/generateCertificate: true
>   boxbackup-client/UpdateStoreInterval: 3600
> * boxbackup-client/backupMode: lazy
> * boxbackup-client/backupServer: somewhere
> * boxbackup-client/backupDirs: somedirs
>   boxbackup-client/MinimumFileAge: 21600
> * boxbackup-client/accountNumber: 1

-- 
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4


From boxbackup at fluffy.co.uk  Fri Aug  3 21:44:11 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Fri, 3 Aug 2007 21:44:11 +0100 (BST)
Subject: [Box Backup] Re: Bug#435860: boxbackup-client: AlwaysInclude[File|Dir]
 is not working as expected
In-Reply-To: <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>
References: <20070803165959.GA9757@localhost> <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>
Message-ID: <Pine.LNX.4.64.0708032139090.23507@top.qwarx.com>

Hi Andreas,

On Fri, 3 Aug 2007, Reinhard Tartler wrote:

> Andreas Putzo <andreas at putzo.net> writes:
>
>> from the config file:
>>
>> BackupLocations
>> {
>> 	home
>> 		{
>>             Path = /home/andreas
>>             ExcludeDir = /home/andreas/chroot
>>             AlwaysIncludeDir = /home/andreas/chroot/sid/home/andreas
>> 		}
>>
>> }
>>
>> I expected that /home/andreas/chroot/sid/home/andreas would be
>> included in the backup but this is not the case.
>> boxbackup is running several days now so it should be there, even in
>> lazy mode.

Unfortunately not. If you Exclude a directory, then Box Backup will never 
scan it or its subdirectories, and will never make it down the tree to 
/home/andreas/chroot/sid/home/andreas which should be backed up.

At the moment, the workarounds are to either (1) create a new location, or 
(2) exclude all files and directories under the excluded directory, except 
the ones on the path to the AlwaysIncluded directory, like so:

ExcludeFilesRegex = /home/andreas/chroot/.*
ExcludeDirsRegex = /home/andreas/chroot/.*
AlwaysIncludeDir = /home/andreas/chroot/sid
AlwaysIncludeDir = /home/andreas/chroot/sid/home
AlwaysIncludeDir = /home/andreas/chroot/sid/home/andreas
AlwaysIncludeFilesRegex = /home/andreas/chroot/sid/home/andreas/.*
AlwaysIncludeDirsRegex = /home/andreas/chroot/sid/home/andreas/.*

I'm sorry that this is not very convenient. I would like to change the 
include/exclude logic in a subsequent release to make it easier to specify 
configurations like yours.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Fri Aug  3 21:45:29 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Fri, 3 Aug 2007 21:45:29 +0100 (BST)
Subject: [Box Backup] Re: Bug#435860: boxbackup-client: AlwaysInclude[File|Dir]
 is not working as expected
In-Reply-To: <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>
References: <20070803165959.GA9757@localhost> <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>
Message-ID: <Pine.LNX.4.64.0708032144160.23507@top.qwarx.com>

Hi all, especially James,

On Fri, 3 Aug 2007, Reinhard Tartler wrote:

> I tried to forwarded the following bug in the trac, but I keep on
> getting the error: "Submission rejected as potential spam (Akismet says
> content is spam)". I'm therefore forced to submit this bug via email.

Please could you look into this?

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Fri Aug  3 21:54:46 2007
From: boxbackup at fluffy.co.uk (James O'Gorman)
Date: Fri, 3 Aug 2007 21:54:46 +0100
Subject: [Box Backup] Re: Bug#435860: boxbackup-client:
 AlwaysInclude[File|Dir] is not working as expected
In-Reply-To: <Pine.LNX.4.64.0708032144160.23507@top.qwarx.com>
References: <20070803165959.GA9757@localhost> <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de> <Pine.LNX.4.64.0708032144160.23507@top.qwarx.com>
Message-ID: <20070803205445.GE51827@netinertia.co.uk>

On Fri, Aug 03, 2007 at 09:45:29PM +0100, Chris Wilson wrote:
>  Hi all, especially James,
> 
>  On Fri, 3 Aug 2007, Reinhard Tartler wrote:
> 
> > I tried to forwarded the following bug in the trac, but I keep on
> > getting the error: "Submission rejected as potential spam (Akismet says
> > content is spam)". I'm therefore forced to submit this bug via email.
> 
>  Please could you look into this?

Since we've disabled self-registration I suppose I could disable the
spam filter. Comments?

James


From boxbackup at fluffy.co.uk  Fri Aug  3 20:55:26 2007
From: boxbackup at fluffy.co.uk (E.W. Peter Jalajas)
Date: Fri, 3 Aug 2007 12:55:26 -0700 (PDT)
Subject: [Box Backup] Exchange Server and shadow copy stuff using box backup
In-Reply-To: <46B35382.6030108@logical-progress.com>
Message-ID: <122581.30870.qm@web60624.mail.yahoo.com>

Hi Dave,

In the interim, until any such code is developed, I'm not sure if this will help, but look at
ticket 13,
http://bbdev.fluffy.co.uk/trac/ticket/13

and specifically my recent entry at the bottom.
27/07/2007 03:10:43 changed by petej

In that hideous batch file attachment is some info on how I handle that.  I still need to test
that I am backing everything up that I need to.  

Option B (or A), is to use the native ntbackup tool.  

Both require using Scheduled Tasks.

Let me know if I can help clarify anything.

Thanks,
Pete

--- dave bamford <dave at logical-progress.com> wrote:

> Hi,
> 
> I have a client who has a need for MS Exchange to be backed up via box 
> backup.



From boxbackup at fluffy.co.uk  Fri Aug  3 22:20:48 2007
From: boxbackup at fluffy.co.uk (Martin Ebourne)
Date: Fri, 03 Aug 2007 22:20:48 +0100
Subject: [Box Backup] Re: Bug#435860: boxbackup-client:
 AlwaysInclude[File|Dir] is not working as expected
In-Reply-To: <20070803205445.GE51827@netinertia.co.uk>
References: <20070803165959.GA9757@localhost>
 <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>
 <Pine.LNX.4.64.0708032144160.23507@top.qwarx.com>
 <20070803205445.GE51827@netinertia.co.uk>
Message-ID: <1186176048.9933.0.camel@avenin.ebourne.me.uk>

On Fri, 2007-08-03 at 21:54 +0100, James O'Gorman wrote:
> On Fri, Aug 03, 2007 at 09:45:29PM +0100, Chris Wilson wrote:
> >  Hi all, especially James,
> > 
> >  On Fri, 3 Aug 2007, Reinhard Tartler wrote:
> > 
> > > I tried to forwarded the following bug in the trac, but I keep on
> > > getting the error: "Submission rejected as potential spam (Akismet says
> > > content is spam)". I'm therefore forced to submit this bug via email.
> > 
> >  Please could you look into this?
> 
> Since we've disabled self-registration I suppose I could disable the
> spam filter. Comments?

If its easy to switch off, try it and see. Can always put it back on
again if necessary.

Cheers,

Martin.



From boxbackup at fluffy.co.uk  Fri Aug  3 22:25:20 2007
From: boxbackup at fluffy.co.uk (James O'Gorman)
Date: Fri, 3 Aug 2007 22:25:20 +0100
Subject: [Box Backup] Re: Bug#435860: boxbackup-client:
 AlwaysInclude[File|Dir] is not working as expected
In-Reply-To: <1186176048.9933.0.camel@avenin.ebourne.me.uk>
References: <20070803165959.GA9757@localhost> <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de> <Pine.LNX.4.64.0708032144160.23507@top.qwarx.com> <20070803205445.GE51827@netinertia.co.uk> <1186176048.9933.0.camel@avenin.ebourne.me.uk>
Message-ID: <20070803212519.GF51827@netinertia.co.uk>

On Fri, Aug 03, 2007 at 10:20:48PM +0100, Martin Ebourne wrote:
> If its easy to switch off, try it and see. Can always put it back on
> again if necessary.

Done.

James


From boxbackup at fluffy.co.uk  Fri Aug  3 22:27:31 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Fri, 3 Aug 2007 22:27:31 +0100 (BST)
Subject: [Box Backup] Re: Bug#435860: boxbackup-client:	AlwaysInclude[File|Dir]
 is not working as expected
In-Reply-To: <20070803212519.GF51827@netinertia.co.uk>
References: <20070803165959.GA9757@localhost> <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>
 <Pine.LNX.4.64.0708032144160.23507@top.qwarx.com> <20070803205445.GE51827@netinertia.co.uk>
 <1186176048.9933.0.camel@avenin.ebourne.me.uk> <20070803212519.GF51827@netinertia.co.uk>
Message-ID: <Pine.LNX.4.64.0708032227090.23507@top.qwarx.com>

Hi James and Martin,

On Fri, 3 Aug 2007, James O'Gorman wrote:

> On Fri, Aug 03, 2007 at 10:20:48PM +0100, Martin Ebourne wrote:
>> If its easy to switch off, try it and see. Can always put it back on
>> again if necessary.
>
> Done.

Thanks for the quick response!

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Fri Aug  3 22:28:02 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Fri, 3 Aug 2007 22:28:02 +0100 (BST)
Subject: [Box Backup] Re: Bug#435860: boxbackup-client: AlwaysInclude[File|Dir]
 is not working as expected
In-Reply-To: <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>
References: <20070803165959.GA9757@localhost> <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>
Message-ID: <Pine.LNX.4.64.0708032227380.23507@top.qwarx.com>

Hi Reinhard,

On Fri, 3 Aug 2007, Reinhard Tartler wrote:

> Hi boxbackup developers.
>
> I tried to forwarded the following bug in the trac, but I keep on
> getting the error: "Submission rejected as potential spam (Akismet says
> content is spam)". I'm therefore forced to submit this bug via email.

Sorry about that, it should hopefully be fixed now.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Fri Aug  3 22:34:13 2007
From: boxbackup at fluffy.co.uk (Martin Ebourne)
Date: Fri, 03 Aug 2007 22:34:13 +0100
Subject: [Box Backup] Re: Bug#435860: boxbackup-client:
 AlwaysInclude[File|Dir] is not working as expected
In-Reply-To: <20070803212519.GF51827@netinertia.co.uk>
References: <20070803165959.GA9757@localhost>
 <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>
 <Pine.LNX.4.64.0708032144160.23507@top.qwarx.com>
 <20070803205445.GE51827@netinertia.co.uk>
 <1186176048.9933.0.camel@avenin.ebourne.me.uk>
 <20070803212519.GF51827@netinertia.co.uk>
Message-ID: <1186176853.9933.7.camel@avenin.ebourne.me.uk>

On Fri, 2007-08-03 at 22:25 +0100, James O'Gorman wrote:
> On Fri, Aug 03, 2007 at 10:20:48PM +0100, Martin Ebourne wrote:
> > If its easy to switch off, try it and see. Can always put it back on
> > again if necessary.
> 
> Done.

While you're at it can you get bbdev.fluffy.co.uk to redirect to trac?
It gets me every time.

Cheers,

Martin.



From boxbackup at fluffy.co.uk  Fri Aug  3 22:48:58 2007
From: boxbackup at fluffy.co.uk (James O'Gorman)
Date: Fri, 3 Aug 2007 22:48:58 +0100
Subject: [Box Backup] Re: Bug#435860: boxbackup-client:
 AlwaysInclude[File|Dir] is not working as expected
In-Reply-To: <1186176853.9933.7.camel@avenin.ebourne.me.uk>
References: <20070803165959.GA9757@localhost> <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de> <Pine.LNX.4.64.0708032144160.23507@top.qwarx.com> <20070803205445.GE51827@netinertia.co.uk> <1186176048.9933.0.camel@avenin.ebourne.me.uk> <20070803212519.GF51827@netinertia.co.uk> <1186176853.9933.7.camel@avenin.ebourne.me.uk>
Message-ID: <20070803214858.GG51827@netinertia.co.uk>

On Fri, Aug 03, 2007 at 10:34:13PM +0100, Martin Ebourne wrote:
> While you're at it can you get bbdev.fluffy.co.uk to redirect to trac?
> It gets me every time.

Done again. I was going to put a couple of links on there point to SVN
and Trac but this way seems better.

James


From boxbackup at fluffy.co.uk  Fri Aug  3 23:17:46 2007
From: boxbackup at fluffy.co.uk (Dave Bamford)
Date: Fri, 03 Aug 2007 23:17:46 +0100
Subject: [Box Backup] Exchange Server and shadow copy stuff using box
 backup
In-Reply-To: <Pine.LNX.4.64.0708032106560.23507@top.qwarx.com>
References: <46B35382.6030108@logical-progress.com> <Pine.LNX.4.64.0708032106560.23507@top.qwarx.com>
Message-ID: <46B3A98A.1050509@logical-progress.com>

Thanks Chris

I tried the MS NTbackup route this afternoon. It created a file of about 3Gb
from the exchange folder which would take over a day to backup at current
upload speeds. This means I can only run the scheduled task every other day.
I think shutting down exchange overnight is currently the best option, 
barring
integrating the code.

Anyway have a great time in Mexico, don't drink too much tequilla :-)

Dave

Chris Wilson wrote:
> Hi Dave,
>
>> I have a client who has a need for MS Exchange to be backed up via 
>> box backup.
>>
>> He is rather insistent that we use box backup as it "has saved his 
>> bacon a few times" and doesn't want to move to anything else!
>>
>> I've told his this is not a current feature and he has now responded 
>> and said what would it cost to make it happen.
>
> It is possible to back up Exchange right now, but it is a bit 
> convoluted. Basically you have to use MS Backup to create a Volume 
> Shadow Service backup of Exchange, which is saved in a file, which Box 
> Backup will back up.
>
> Restore will then be a two-stage process: restore the backed-up 
> Exchange backup file with Box, and then restore it over the Exchange 
> installation with MS Backup.
>
> There has been a discussion of implementing Volume Shadow Services 
> backups within Box Backup itself, removing the need for the MS Backup 
> step, but so far nobody has had time to do that.
>
>> Any suggestions as to potential costs of writing this assuming any 
>> new code remains under the existing "open source" licence for all to 
>> use?
>
> I'm afraid I won't really have time to implement that this year. 
> However, Charles Lecklider has expressed an interest and has the 
> skills to do this. I believe that lack of time has prevented him from 
> doing it so far, and a financial offer might well help to persuade 
> him. Might I suggest that you contact him directly for a quote?
>
>> I'm assuming it would be able to run in the background.
>
> I guess that by "in the background" you mean "without shutting down 
> Exchange", which is what I assumed above.
>
> If you don't mind shutting down Exchange regularly then you can use 
> Box Backup to back up its files without any problems while it's shut 
> down.
>
> Cheers, Chris.




From boxbackup at fluffy.co.uk  Fri Aug  3 23:22:45 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Fri, 3 Aug 2007 23:22:45 +0100 (BST)
Subject: [Box Backup] Exchange Server and shadow copy stuff using box
 backup
In-Reply-To: <46B3A98A.1050509@logical-progress.com>
References: <46B35382.6030108@logical-progress.com>
 <Pine.LNX.4.64.0708032106560.23507@top.qwarx.com> <46B3A98A.1050509@logical-progress.com>
Message-ID: <Pine.LNX.4.64.0708032321260.23507@top.qwarx.com>

Hi Dave,

> I tried the MS NTbackup route this afternoon. It created a file of about 
> 3Gb from the exchange folder which would take over a day to backup at 
> current upload speeds. This means I can only run the scheduled task 
> every other day.

Maybe the first time, but subsequent backups should be much quicker if you 
can turn off compression in NTBackup.

> I think shutting down exchange overnight is currently the best option, 
> barring integrating the code.

I hope that Charles can help you with this, it's something that we all 
want to see in Box Backup :-)

> Anyway have a great time in Mexico, don't drink too much tequilla :-)

Thanks, I'll try not to!

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Sun Aug  5 12:41:26 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Sun, 5 Aug 2007 12:41:26 +0100 (BST)
Subject: [Box Backup] Re: Bug#435860: boxbackup-client: AlwaysInclude[File|Dir]
 is not working as expected
In-Reply-To: <20070805111735.GA21477@localhost>
References: <20070803165959.GA9757@localhost> <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>
 <Pine.LNX.4.64.0708032139090.23507@top.qwarx.com> <20070805111735.GA21477@localhost>
Message-ID: <Pine.LNX.4.64.0708051238360.23507@top.qwarx.com>

Hi Andreas,

On Sun, 5 Aug 2007, Andreas Putzo wrote:

> Ah ok. I assumed something like this. Perhaps the comments in the 
> generated bbackupd.conf should be improved then to be more clear on 
> this. It can be terrible if one learns the hard way, that the backup 
> system is not backing up all the files you was thinking it would. :)

Yes, but that can happen to any backup system, that's why test restores 
are important (nothing else will really reassure you).

>> At the moment, the workarounds are to either (1) create a new location, or
>> (2) exclude all files and directories under the excluded directory, except
>> the ones on the path to the AlwaysIncluded directory, like so:
>
> I have several directives like this in my config. Since they are all
> subdirectories of 'home' i don't want to create different locations
> for each of them.

Why not?

> Using (2)  would render the config file more complicated and 
> error-prone.

Indeed.

> If the include/exclude logic can be improved to be aware of 
> AlwaysIncluded subdirectories i would appreciate this.

I wish it were so simple, but because AlwaysInclude*Regex can apply at any 
point in the tree, it would mean that we always have to scan all the way 
down the tree. So we'd need another directive like SkipDir(sRegex) to 
completely exclude descending into a directory and any possibility of 
files inside it being backed up.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Sun Aug  5 20:44:14 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Sun, 5 Aug 2007 20:44:14 +0100 (BST)
Subject: [Box Backup] Re: Bug#435860: boxbackup-client: AlwaysInclude[File|Dir]
 is not working as expected
In-Reply-To: <20070805125930.GB21477@localhost>
References: <20070803165959.GA9757@localhost> <87sl70s90o.fsf@faui44a.informatik.uni-erlangen.de>
 <Pine.LNX.4.64.0708032139090.23507@top.qwarx.com> <20070805111735.GA21477@localhost>
 <Pine.LNX.4.64.0708051238360.23507@top.qwarx.com> <20070805125930.GB21477@localhost>
Message-ID: <Pine.LNX.4.64.0708052040190.23507@top.qwarx.com>

Hi Andreas,

On Sun, 5 Aug 2007, Andreas Putzo wrote:

>>> If the include/exclude logic can be improved to be aware of
>>> AlwaysIncluded subdirectories i would appreciate this.
>>
>> I wish it were so simple, but because AlwaysInclude*Regex can apply at any
>> point in the tree, it would mean that we always have to scan all the way
>> down the tree. So we'd need another directive like SkipDir(sRegex) to
>> completely exclude descending into a directory and any possibility of
>> files inside it being backed up.
>
> Mmh, true. I wasn't thinking about that because i was using a simple
> ExcludeDir/AlwaysIncludeDir directive without any regex in it.
> After all, i think the current possibilities to define backup locations are
> already powerful enough.

Actually, I don't think they are. There is no way to exclude some files 
inside a location that is AlwaysIncluded. I think an 
Exclude/Include/Exclude/Include type of system would be much better, but I 
don't have time to implement it yet.

> It's just that i got fooled (dumb me :) by the comment
>
> # For example:
> #
> #   ExcludeDir = /home/guest-user
> #   ExcludeFilesRegex = \.(mp3|MP3)$
> #   AlwaysIncludeFile = /home/username/veryimportant.mp3
> #
>
> # In general, Exclude excludes a file or directory, unless the
> # directory is explicitly mentioned in a AlwaysInclude directive.
>
> Perhaps it would be sufficient to be a little more precise on this, eg. 
> that AlwaysIncludeFile = /home/guest-user/veryimportant.mp3 will not 
> work in the above example.

Thanks for pointing that out, I think I've fixed the script that generates 
the bbackupd.conf file to include more accurate comments.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Thu Aug  9 22:20:26 2007
From: boxbackup at fluffy.co.uk (=?ISO-8859-1?Q?N=E9her_M=E1rton?=)
Date: Thu, 09 Aug 2007 23:20:26 +0200
Subject: [Box Backup] boxbackup bug with openssl
Message-ID: <46BB851A.7080104@lanten.hu>

Hi!

I have a backup servers, and 5 other computers, to back them up.
4 of the 5 servers uses openssl-0.9.8, so as the backupserver, but 1 
uses openssl-0.9.7.
I was simply unable to sync with that client, but backup was really 
urgent, so i made some hack with nfs, but it is really not important.

Here are the extended logs of the client and server, hopefully you can 
use it to fix this problem. (### is commented out, that is fully 
qualified hostname and living ip address)

marton neher


Aug  9 21:30:20 veszely bbackupd[7734]: Incoming connection from local 
(UNIX socket)
Aug  9 21:30:20 veszely bbackupd[7734]: Connection from command socket
Aug  9 21:30:20 veszely bbackupd[7734]: Beginning scan of local files
Aug  9 21:30:20 veszely bbackupd[7734]: Opening connection to server 
#############...
Aug  9 21:30:20 veszely bbackupd[7734]: Send Version(0x1)
Aug  9 21:30:20 veszely bbackupd[7734]: Receive Version(0x1)
Aug  9 21:30:20 veszely bbackupd[7734]: Send Login(0xa002,0x0)
Aug  9 21:30:20 veszely bbackupd[7734]: Receive 
LoginConfirmed(0x4374948d68040,0x336ce,0x15e00000,0x19000000)
Aug  9 21:30:20 veszely bbackupd[7734]: Connection made, login successful
Aug  9 21:30:20 veszely bbackupd[7734]: Send 
ListDirectory(0x1,0x2,0xc,false)
Aug  9 21:30:20 veszely bbackupd[7734]: Receive Success(0x1)
Aug  9 21:30:20 veszely bbackupd[7734]: Receiving stream, size 96
Aug  9 21:30:20 veszely bbackupd[7734]: Send 
SetClientStoreMarker(0x4374949f86b00)
Aug  9 21:30:20 veszely bbackupd[7734]: SSL err during Read: 
error:06065064:digital envelope routines:EVP_DecryptFinal:bad decrypt
Aug  9 21:30:20 veszely bbackupd[7734]: SSL err during Read: 
error:06065064:digital envelope routines:EVP_DecryptFinal:bad decrypt
Aug  9 21:30:20 veszely bbackupd[7734]: Exception caught (Cipher 
EVPFinalFailure 5/6), reset state and waiting to retry...
Aug  9 21:30:30 veszely bbackupd[7734]: File statistics: total file size 
uploaded 0, bytes already on server 0, encoded size 0
Aug  9 21:31:55 veszely bbackupd[7734]: Beginning scan of local files
Aug  9 21:31:55 veszely bbackupd[7734]: Opening connection to server 
backup.lanten.hu...
Aug  9 21:31:55 veszely bbackupd[7734]: Send Version(0x1)
Aug  9 21:31:55 veszely bbackupd[7734]: Receive Version(0x1)
Aug  9 21:31:55 veszely bbackupd[7734]: Send Login(0xa002,0x0)
Aug  9 21:31:55 veszely bbackupd[7734]: Receive 
LoginConfirmed(0x4374949f86b00,0x336ce,0x15e00000,0x19000000)
Aug  9 21:31:55 veszely bbackupd[7734]: Connection made, login successful
Aug  9 21:31:55 veszely bbackupd[7734]: Send 
ListDirectory(0x1,0x2,0xc,false)
Aug  9 21:31:55 veszely bbackupd[7734]: Receive Success(0x1)
Aug  9 21:31:55 veszely bbackupd[7734]: Receiving stream, size 96
Aug  9 21:31:55 veszely bbackupd[7734]: Send 
SetClientStoreMarker(0x437494fa200c0)
Aug  9 21:31:55 veszely bbackupd[7734]: SSL err during Read: 
error:06065064:digital envelope routines:EVP_DecryptFinal:bad decrypt
Aug  9 21:31:55 veszely bbackupd[7734]: SSL err during Read: 
error:06065064:digital envelope routines:EVP_DecryptFinal:bad decrypt
Aug  9 21:31:55 veszely bbackupd[7734]: Exception caught (Cipher 
EVPFinalFailure 5/6), reset state and waiting to retry...
Aug  9 21:32:05 veszely bbackupd[7734]: File statistics: total file size 
uploaded 0, bytes already on server 0, encoded size 0



Aug  9 21:30:27 selenyi bbstored[13199]: Incoming connection from 
###.###.###.### port 47990 (handling in child 11480)
Aug  9 21:30:28 selenyi bbstored[11480]: Certificate CN: BACKUP-a002
Aug  9 21:30:28 selenyi bbstored[11480]: Receive Version(0x1)
Aug  9 21:30:28 selenyi bbstored[11480]: Receive Version(0x1)
Aug  9 21:30:28 selenyi bbstored[11480]: Send Version(0x1)
Aug  9 21:30:28 selenyi bbstored[11480]: Send Version(0x1)
Aug  9 21:30:28 selenyi bbstored[11480]: Receive Login(0xa002,0x0)
Aug  9 21:30:28 selenyi bbstored[11480]: Receive Login(0xa002,0x0)
Aug  9 21:30:28 selenyi bbstored[11480]: Login: Client ID 0000A002, 
Read/Write
Aug  9 21:30:28 selenyi bbstored[11480]: Send 
LoginConfirmed(0x4374948d68040,0x336ce,0x15e00000,0x19000000)
Aug  9 21:30:28 selenyi bbstored[11480]: Send 
LoginConfirmed(0x4374948d68040,0x336ce,0x15e00000,0x19000000)
Aug  9 21:30:28 selenyi bbstored[11480]: Receive 
ListDirectory(0x1,0x2,0xc,false)
Aug  9 21:30:28 selenyi bbstored[11480]: Receive 
ListDirectory(0x1,0x2,0xc,false)
Aug  9 21:30:28 selenyi bbstored[11480]: Send Success(0x1)
Aug  9 21:30:28 selenyi bbstored[11480]: Send Success(0x1)
Aug  9 21:30:28 selenyi bbstored[11480]: Sending stream, size 96
Aug  9 21:30:28 selenyi bbstored[11480]: Receive 
SetClientStoreMarker(0x4374949f86b00)
Aug  9 21:30:28 selenyi bbstored[11480]: Receive 
SetClientStoreMarker(0x4374949f86b00)
Aug  9 21:30:28 selenyi bbstored[11480]: Send Success(0x4374949f86b00)
Aug  9 21:30:28 selenyi bbstored[11480]: Send Success(0x4374949f86b00)
Aug  9 21:30:28 selenyi bbstored[11480]: Connection statistics for 
BACKUP-a002: IN=97 OUT=220 TOTAL=317
Aug  9 21:30:28 selenyi bbstored[11480]: in server child, exception 
Connection Protocol_Timeout (Probably a network issue between client and 
server.) (7/41) -- terminating child




From boxbackup at fluffy.co.uk  Sun Aug 12 03:37:51 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Sat, 11 Aug 2007 21:37:51 -0500 (CDT)
Subject: [Box Backup] boxbackup bug with openssl
In-Reply-To: <46BB851A.7080104@lanten.hu>
References: <46BB851A.7080104@lanten.hu>
Message-ID: <Pine.LNX.4.64.0708112135460.10354@rocio.int.aidworld.org>

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--8323328-1847886830-1186886271=:10354
Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: QUOTED-PRINTABLE

Hi Marton,

On Thu, 9 Aug 2007, N=E9her M=E1rton wrote:

> I have a backup servers, and 5 other computers, to back them up. 4 of=20
> the 5 servers uses openssl-0.9.8, so as the backupserver, but 1 uses=20
> openssl-0.9.7. I was simply unable to sync with that client, but backup=
=20
> was really urgent, so i made some hack with nfs, but it is really not=20
> important.
>
> Here are the extended logs of the client and server, hopefully you can=20
> use it to fix this problem. (### is commented out, that is fully=20
> qualified hostname and living ip address)
=2E..
> Aug  9 21:30:20 veszely bbackupd[7734]: Send=20
> SetClientStoreMarker(0x4374949f86b00)
> Aug  9 21:30:20 veszely bbackupd[7734]: SSL err during Read:=20
> error:06065064:digital envelope routines:EVP_DecryptFinal:bad decrypt
> Aug  9 21:30:20 veszely bbackupd[7734]: SSL err during Read:=20
> error:06065064:digital envelope routines:EVP_DecryptFinal:bad decrypt
> Aug  9 21:30:20 veszely bbackupd[7734]: Exception caught (Cipher=20
> EVPFinalFailure 5/6), reset state and waiting to retry...

This looks like an OpenSSL bug. Please try to use a newer version of=20
OpenSSL on that client.

Also, it's not actually trying to upload anything. It dies as the=20
client is logging out, without doing anything. Please check your=20
configuration.

Cheers, Chris.
--=20
_ ___ __     _
  / __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |
--8323328-1847886830-1186886271=:10354--


From boxbackup at fluffy.co.uk  Sun Aug 12 20:42:13 2007
From: boxbackup at fluffy.co.uk (=?ISO-8859-1?Q?N=E9her_M=E1rton?=)
Date: Sun, 12 Aug 2007 21:42:13 +0200
Subject: [Box Backup] boxbackup bug with openssl
In-Reply-To: <Pine.LNX.4.64.0708112135460.10354@rocio.int.aidworld.org>
References: <46BB851A.7080104@lanten.hu> <Pine.LNX.4.64.0708112135460.10354@rocio.int.aidworld.org>
Message-ID: <46BF6295.1010503@lanten.hu>

Chris Wilson wrote:
> This looks like an OpenSSL bug. Please try to use a newer version of 
> OpenSSL on that client.
>
> Also, it's not actually trying to upload anything. It dies as the 
> client is logging out, without doing anything. Please check your 
> configuration.
>
> Cheers, Chris.

Hi!

Actually, it is not possible for me to upgrade the openssl (that machine 
is integrated with hosting administration hacks so much, that I don't 
dare to change anything important right now), that's why I wrote that I 
had pumped the data through nfs to an another machine. Btw, it has 
worked with that version once, that's why i asked, i tought that there 
is a secret option somewhere, to make them compatible. It is not very 
important right now.

Best regards:

Marton Neher


From boxbackup at fluffy.co.uk  Tue Aug 14 12:30:35 2007
From: boxbackup at fluffy.co.uk (Johann Glaser)
Date: Tue, 14 Aug 2007 13:30:35 +0200
Subject: [Box Backup] BadBackupStoreFile
Message-ID: <1187091035.5344.4.camel@localhost.localdomain>

Hi!

On our backup the error message 
  Backup object failed, error when reading /backup/daily/svn/qm/db/strings
  Error code when uploading was (4/11), BackupStore BadBackupStoreFile
was sent to Syslog. I've switched on ExtendedLogging and now have
several megabytes of Syslog files. :-) Unfortunately I'm not able to
interpret the information which backup store file belong to the problem.

How can I find out the real reason of the problem and how can I correct
it? BTW, yesterday I did a "bbstoreaccounts check 1" and no errors were
found.

Thanks
  Hansi

-- 
Johann Glaser                          <glaser at ict.tuwien.ac.at>
             Institute of Computer Technology, E384
Vienna University of Technology, Gusshausstr. 27-29, A-1040 Wien
Phone: ++43/1/58801-38444                Fax: ++43/1/58801-38499



From boxbackup at fluffy.co.uk  Mon Aug 20 21:56:04 2007
From: boxbackup at fluffy.co.uk (Paul MacKenzie)
Date: Mon, 20 Aug 2007 16:56:04 -0400
Subject: [Box Backup] boxbackup bug with openssl
In-Reply-To: <Pine.LNX.4.64.0708112135460.10354@rocio.int.aidworld.org>
References: <46BB851A.7080104@lanten.hu>
 <Pine.LNX.4.64.0708112135460.10354@rocio.int.aidworld.org>
Message-ID: <200708202055.l7KKt77e048421@elephants.elehost.com>

Hello,

I am trying to create a one line command for restoring a text file

/usr/local/bin/bbackupquery 'cd /usr/home  get -i 004329f5 
/usr/home/somefile' quit

It needs to be run as one line.

Is there any way to do this from unix as I am running it from Perl?

Thanks,

Paul




From boxbackup at fluffy.co.uk  Tue Aug 21 02:07:27 2007
From: boxbackup at fluffy.co.uk (Ben Bennett)
Date: Mon, 20 Aug 2007 21:07:27 -0400
Subject: [Box Backup] boxbackup bug with openssl
In-Reply-To: <200708202055.l7KKt77e048421@elephants.elehost.com>
References: <46BB851A.7080104@lanten.hu> <Pine.LNX.4.64.0708112135460.10354@rocio.int.aidworld.org> <200708202055.l7KKt77e048421@elephants.elehost.com>
Message-ID: <20070821010727.GA21570@ayup.limey.net>

On Mon, Aug 20, 2007 at 04:56:04PM -0400, Paul MacKenzie wrote:
> Hello,
> 
> I am trying to create a one line command for restoring a text file
> 
> /usr/local/bin/bbackupquery 'cd /usr/home  get -i 004329f5 
> /usr/home/somefile' quit
> 
> It needs to be run as one line.

I suspect you want:
 /usr/local/bin/bbackupquery 'cd /usr/home' 'get -i 004329f5 /usr/home/somefile' quit

				 -ben


From boxbackup at fluffy.co.uk  Wed Aug 22 01:40:07 2007
From: boxbackup at fluffy.co.uk (Paul MacKenzie)
Date: Tue, 21 Aug 2007 20:40:07 -0400
Subject: [Box Backup] boxbackup bug with openssl
In-Reply-To: <20070821010727.GA21570@ayup.limey.net>
References: <46BB851A.7080104@lanten.hu>
 <Pine.LNX.4.64.0708112135460.10354@rocio.int.aidworld.org>
 <200708202055.l7KKt77e048421@elephants.elehost.com>
 <20070821010727.GA21570@ayup.limey.net>
Message-ID: <200708220039.l7M0dDbm082881@elephants.elehost.com>

At 09:07 PM 20/08/2007, you wrote:
>On Mon, Aug 20, 2007 at 04:56:04PM -0400, Paul MacKenzie wrote:
> > Hello,
> >
> > I am trying to create a one line command for restoring a text file
> >
> > /usr/local/bin/bbackupquery 'cd /usr/home  get -i 004329f5
> > /usr/home/somefile' quit
> >
> > It needs to be run as one line.
>
>I suspect you want:
>  /usr/local/bin/bbackupquery 'cd /usr/home' 'get -i 004329f5 
> /usr/home/somefile' quit
>
>                                 -ben

Dear Ben,

Thanks for your reply. I tried that initially but it never exits from 
boxbackup nor restores the file from what I can tell.

Here is what is shows when I use the quotes around the commands:

Box Backup Query Tool v0.10, (c) Ben Summers and contributors 2003-2006
Using configuration file /usr/local/etc/box/bbackupd.conf
Connecting to store...
Handshake with store...
Login to store...
Login complete.

Type "help" for a list of commands.

And then just gets stuck.

Any other ideas as I seem to have tried everything I can think of but 
must be overlooking the obvious.

In case anyone is interested I wrote a perl script to restore every 
historical copy of a specific file. It worked fine with the getobject 
command but I need to use the get command for my file types.

Now if I could just get it to work :)

Cheers,

Paul




From boxbackup at fluffy.co.uk  Fri Aug 24 23:38:07 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Fri, 24 Aug 2007 23:38:07 +0100 (BST)
Subject: [Box Backup] boxbackup bug with openssl
In-Reply-To: <46BF6295.1010503@lanten.hu>
References: <46BB851A.7080104@lanten.hu> <Pine.LNX.4.64.0708112135460.10354@rocio.int.aidworld.org>
 <46BF6295.1010503@lanten.hu>
Message-ID: <Pine.LNX.4.64.0708242328001.20516@top.qwarx.com>

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

---128931150-1836665581-1187995087=:20516
Content-Type: TEXT/PLAIN; charset=utf-8; format=flowed
Content-Transfer-Encoding: QUOTED-PRINTABLE

Hi Marton,

On Sun, 12 Aug 2007, N=E9her M=E1rton wrote:

> Chris Wilson wrote:
>>  This looks like an OpenSSL bug. Please try to use a newer version of
>>  OpenSSL on that client.
>
> Actually, it is not possible for me to upgrade the openssl (that machine=
=20
> is integrated with hosting administration hacks so much, that I don't=20
> dare to change anything important right now), that's why I wrote that I=
=20
> had pumped the data through nfs to an another machine. Btw, it has=20
> worked with that version once, that's why i asked, i tought that there=20
> is a secret option somewhere, to make them compatible. It is not very=20
> important right now.

Sorry for tthe delayed reply, I've been on holiday in Mexico without=20
access to email and still working through the backlog.

There is a "secret" option to enable compatibility with OpenSSL 0.9.6, but=
=20
this is definitely not recommended. In any case I doubt very much that it=
=20
will help you with this problem unless you are actually running 0.9.6 on=20
this client.

There is a serious bug with OpenSSL 0.9.8d/e (see=20
http://bbdev.fluffy.co.uk/trac/ticket/21 for details). What hardware, OS,=
=20
and OpenSSL version are you running on? And what version of Box Backup,=20
0.10 release?

If you are running OpenSSL 0.9.8d/e and refuse to upgrade or downgrade,=20
then I'm afraid I don't have the resoures to help you right now. If not,=20
then I would really appreciate the details of your client configuration.

When you say "=A3i has worked with that version once", which version do you=
=20
mean? And do you know what you did just before it stopped working?

Cheers, Chris.
--=20
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |
---128931150-1836665581-1187995087=:20516--


From boxbackup at fluffy.co.uk  Fri Aug 24 23:45:25 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Fri, 24 Aug 2007 23:45:25 +0100 (BST)
Subject: [Box Backup] boxbackup bug with openssl
In-Reply-To: <200708220039.l7M0dDbm082881@elephants.elehost.com>
References: <46BB851A.7080104@lanten.hu> <Pine.LNX.4.64.0708112135460.10354@rocio.int.aidworld.org>
 <200708202055.l7KKt77e048421@elephants.elehost.com> <20070821010727.GA21570@ayup.limey.net>
 <200708220039.l7M0dDbm082881@elephants.elehost.com>
Message-ID: <Pine.LNX.4.64.0708242341370.20516@top.qwarx.com>

Hi Paul,

On Tue, 21 Aug 2007, Paul MacKenzie wrote:

>> >  I am trying to create a one line command for restoring a text file
>> > 
>> >  /usr/local/bin/bbackupquery 'cd /usr/home  get -i 004329f5
>> >  /usr/home/somefile' quit
>> > 
>> >  It needs to be run as one line.
>> 
>> I suspect you want:
>>  /usr/local/bin/bbackupquery 'cd /usr/home' 'get -i 004329f5
>>  /usr/home/somefile' quit
>
> Thanks for your reply. I tried that initially but it never exits from 
> boxbackup nor restores the file from what I can tell.
>
> Here is what is shows when I use the quotes around the commands:
>
> Box Backup Query Tool v0.10, (c) Ben Summers and contributors 2003-2006
> Using configuration file /usr/local/etc/box/bbackupd.conf
> Connecting to store...
> Handshake with store...
> Login to store...
> Login complete.
>
> Type "help" for a list of commands.
>
> And then just gets stuck.

Sorry, that should definitely not happen. Is your server using version 
0.10 as well? Could you enable ExtendedLogging on the client and report 
the log messages from the syslog on the client? What happens if you do the 
restore using bbackupquery interactively? Could you show us the exact 
command line that you used?

> In case anyone is interested I wrote a perl script to restore every 
> historical copy of a specific file. It worked fine with the getobject 
> command but I need to use the get command for my file types.

I think that getobject downloads the encrypted copy of the file from the 
server, which will probably not help you much.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Sat Aug 25 18:11:16 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Sat, 25 Aug 2007 18:11:16 +0100 (BST)
Subject: [Box Backup] BadBackupStoreFile
In-Reply-To: <1187091035.5344.4.camel@localhost.localdomain>
References: <1187091035.5344.4.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.64.0708251705330.20516@top.qwarx.com>

Hi Johann,

> On our backup the error message
>  Backup object failed, error when reading /backup/daily/svn/qm/db/strings
>  Error code when uploading was (4/11), BackupStore BadBackupStoreFile
> was sent to Syslog. I've switched on ExtendedLogging and now have
> several megabytes of Syslog files. :-) Unfortunately I'm not able to
> interpret the information which backup store file belong to the problem.
>
> How can I find out the real reason of the problem and how can I correct
> it? BTW, yesterday I did a "bbstoreaccounts check 1" and no errors were
> found.

Probably that file is over 2 GB. Unfortunately there is a bug in 0.10 
which causes this. I've updated the Installation page on the wiki to 
describe the problem more clearly:

   http://bbdev.fluffy.co.uk/trac/wiki/Installation

Please could you try updating both server and client to the chris/merge 
branch, as described on that page? You may also need to delete the old 
version of the file from the store in order to ensure that it is backed up 
cleanly. (for example, exclude it from the backup set, run a backup, then 
remove the exclusion).

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Sun Aug 26 14:00:10 2007
From: boxbackup at fluffy.co.uk (Matt Brown)
Date: Sun, 26 Aug 2007 14:00:10 +0100
Subject: [Box Backup] BoxBackup (chris/merge) - Store Error
Message-ID: <1D4EFCF0-2501-4947-B261-B0F0DEE59532@mbrown.co.uk>

Hi Chris.

Hope the hols in mexico were good and your feeling refreshed :-)

Just a quickie ...

When I run the bbstoreaccounts check option I get ...

root at pegasus:~# /usr/local/bin/bbstoreaccounts check 1
Checking store account ID 0x000001...
Phase 1, check objects...
Phase 2, check directories...
Phase 3, check root...
WARNING: Root directory doesn't exist
Phase 4, fix unattached objects...
Phase 5, fix unrecovered inconsistencies...
Phase 6, regenerate store info...
WARNING: Finished checking store account ID 0x000001: 0x1 errors found
WARNING: No changes to the store account have been made.
WARNING: Run again with fix option to fix these errors

So I run with the fix option, then get ...

root at pegasus:~# /usr/local/bin/bbstoreaccounts check 1 fix
NOTICE: Will fix errors encountered during checking.
Checking store account ID 0x000001...
Phase 1, check objects...
Phase 2, check directories...
Phase 3, check root...
WARNING: Root directory doesn't exist
WARNING: Exception thrown: RaidFileException 
(CannotOverwriteExistingFile) at RaidFileWrite.cpp(100)
Exception: RaidFile CannotOverwriteExistingFile (2/4)

I have a Ubuntu Server with your merge branch 1794 (as I have been  
testing for you since the 2GB file bug as promised) in a Dell  
PowerEdge running a Raid 5 Reiser file system ...

Any ideas ?

Does not seem to be causing any harm .... that I can see..

-- 
Matt Brown
matt at mbrown.co.uk





From boxbackup at fluffy.co.uk  Mon Aug 27 09:54:39 2007
From: boxbackup at fluffy.co.uk (Johann Glaser)
Date: Mon, 27 Aug 2007 10:54:39 +0200
Subject: [Box Backup] BadBackupStoreFile
In-Reply-To: <Pine.LNX.4.64.0708251705330.20516@top.qwarx.com>
References: <1187091035.5344.4.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708251705330.20516@top.qwarx.com>
Message-ID: <1188204879.5254.12.camel@localhost.localdomain>

Hi!

> Probably that file is over 2 GB. Unfortunately there is a bug in 0.10 
> which causes this. I've updated the Installation page on the wiki to 
> describe the problem more clearly:
> 
>    http://bbdev.fluffy.co.uk/trac/wiki/Installation
> 
> Please could you try updating both server and client to the chris/merge 
> branch, as described on that page? You may also need to delete the old 
> version of the file from the store in order to ensure that it is backed up 
> cleanly. (for example, exclude it from the backup set, run a backup, then 
> remove the exclusion).

Yes, the original file is >2GB, but we have some more, even larger
files, which don't show this problem. 

Is it sure that the backup continues after the problem with all other
files? Or does the backup stop after the first error?

How can I find out which files in the backup store belong to which
original files?

Your second paragraph suggests that excluding files from the backup by
adding an exclusion-statement in bbackupd.conf will remove all backups
of this file. Is this true?

Unfortunately we don't want to try unstable or testing backup software
on our production server, therefore we will want to wait until a more
stable version is available.

Bye
  Hansi

-- 
Johann Glaser                          <glaser at ict.tuwien.ac.at>
             Institute of Computer Technology, E384
Vienna University of Technology, Gusshausstr. 27-29, A-1040 Wien
Phone: ++43/1/58801-38444                Fax: ++43/1/58801-38499



From boxbackup at fluffy.co.uk  Tue Aug 28 00:07:43 2007
From: boxbackup at fluffy.co.uk (Matt Brown)
Date: Tue, 28 Aug 2007 00:07:43 +0100
Subject: [Box Backup] Backup Stats ?
Message-ID: <C19D949A-5347-4727-810B-E6B035617A4F@mbrown.co.uk>

Hi,

Looking through some of the older posts to this list (around Sept-Nov  
04) I noticed a few threads about backup stats.

I am currently running Box Backup 0.10 (Chris Merge 1794) for a  
handful of clients in place of an Rsync script - clients however were  
used to the verbose output to show what files were backed up and the  
rsync summary at the end of the batch was also semi-informative...

Is there likely to be a similar feature for Box  ? I did grab the  
bbclientstatus.pl script along with the box::config perl mod which  
can email the last backup etc of a client which is great for me to  
see the status of the clients, and poke this data at Nagios, however  
I think what the clients would love to see is something along the  
lines of:

1. Did it backup last night ?
2. What size was the backup.
3. Any errors ?
4. Space left on store/used.

Any thoughts ?

Keep up the fantastic work on this project, I am a great fan and of  
course as always I am happy to test or assist if need be :-)

Matt Brown
matt at mbrown.co.uk





From boxbackup at fluffy.co.uk  Wed Aug 29 01:12:27 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Wed, 29 Aug 2007 01:12:27 +0100 (BST)
Subject: [Box Backup] BadBackupStoreFile
In-Reply-To: <1188204879.5254.12.camel@localhost.localdomain>
References: <1187091035.5344.4.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708251705330.20516@top.qwarx.com> <1188204879.5254.12.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.64.0708290108130.20516@top.qwarx.com>

Hi Johann,

On Mon, 27 Aug 2007, Johann Glaser wrote:

> Yes, the original file is >2GB, but we have some more, even larger
> files, which don't show this problem.

It has to be over 2GB _compressed_ to trigger the bug. Perhaps the other 
files compress down to under 2GB?

> Is it sure that the backup continues after the problem with all other 
> files? Or does the backup stop after the first error?

It should continue, but if an exception is being thrown then it might well 
not do so as that should be an extreme case. I haven't verified the code 
path in this case. Does it matter to you, i.e. would you be happy to 
continue to run this version knowing that a few large files are not being 
backed up?

You can tell pretty easily from the backup logs. If it says "Caught 
exception - reset state and waiting to retry" then it did not complete the 
backup run, but aborted because of the exception.

> How can I find out which files in the backup store belong to which 
> original files?

Sorry, I don't know exactly what you mean here, does bbackupquery's list 
command tell you what you want?

> Your second paragraph suggests that excluding files from the backup by 
> adding an exclusion-statement in bbackupd.conf will remove all backups 
> of this file. Is this true?

No, it's only that when bbackupd uploads a new version, it does not send a 
patch against deleted versions, so it will upload a fresh copy and avoid 
triggering the bug.

> Unfortunately we don't want to try unstable or testing backup software 
> on our production server, therefore we will want to wait until a more 
> stable version is available.

OK, your call.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Wed Aug 29 01:27:05 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Wed, 29 Aug 2007 01:27:05 +0100 (BST)
Subject: [Box Backup] Backup Stats ?
In-Reply-To: <C19D949A-5347-4727-810B-E6B035617A4F@mbrown.co.uk>
References: <C19D949A-5347-4727-810B-E6B035617A4F@mbrown.co.uk>
Message-ID: <Pine.LNX.4.64.0708290118140.20516@top.qwarx.com>

Hi Matt,

> I am currently running Box Backup 0.10 (Chris Merge 1794) for a handful 
> of clients in place of an Rsync script - clients however were used to 
> the verbose output to show what files were backed up and the rsync 
> summary at the end of the batch was also semi-informative...
>
> Is there likely to be a similar feature for Box ? I did grab the 
> bbclientstatus.pl script along with the box::config perl mod which can 
> email the last backup etc of a client which is great for me to see the 
> status of the clients, and poke this data at Nagios, however I think 
> what the clients would love to see is something along the lines of:
>
> 1. Did it backup last night ?
> 2. What size was the backup.
> 3. Any errors ?
> 4. Space left on store/used.

The Box Backup client can now log a lot of this information, if you turn 
on the LogAllFileAccess option and run it in verbose mode (sufficiently 
high -v level, or -V for maximum). You could grep it out of your syslogs 
(with timestamps for everything) and report it, or I can provide a way to 
write the logs to a file instead.

You will see it connect/disconnect (so you know that it backed up), a list 
of files processed, ignored due to not being modified, backed up 
successfully, or failed to back up. As far as I know, no existing scripts 
have been modified to process and report on this new verbose data.

You can get the space left on store at any time with bbackupquery.

> Keep up the fantastic work on this project, I am a great fan and of 
> course as always I am happy to test or assist if need be :-)

Thanks, it's highly appreciated!

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Wed Aug 29 08:45:49 2007
From: boxbackup at fluffy.co.uk (Johann Glaser)
Date: Wed, 29 Aug 2007 09:45:49 +0200
Subject: [Box Backup] BadBackupStoreFile
In-Reply-To: <Pine.LNX.4.64.0708290108130.20516@top.qwarx.com>
References: <1187091035.5344.4.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708251705330.20516@top.qwarx.com>
 <1188204879.5254.12.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708290108130.20516@top.qwarx.com>
Message-ID: <1188373549.5379.27.camel@localhost.localdomain>

Hi Chris!

> > Yes, the original file is >2GB, but we have some more, even larger
> > files, which don't show this problem.
> 
> It has to be over 2GB _compressed_ to trigger the bug. Perhaps the other 
> files compress down to under 2GB?

We have two >2GB files in the backup store. See the bottom of a sorted
listing:
[...]
748651521 ./cc/02/o1f.rfw
748679825 ./72/03/o8e.rfw
748705025 ./3c/03/oa7.rfw
759350104 ./20/04/o10.rfw
759356296 ./23/06/o12.rfw
759552073 ./9e/06/o8f.rfw
759633897 ./1a/07/obc.rfw
1529736909 ./ba/o55.rfw
1539937214 ./69/01/o05.rfw
2744679666 ./16/01/o5f.rfw
5133609317 ./f2/o44.rfw

> > Is it sure that the backup continues after the problem with all other 
> > files? Or does the backup stop after the first error?
> 
> It should continue, but if an exception is being thrown then it might well 
> not do so as that should be an extreme case. I haven't verified the code 
> path in this case. Does it matter to you, i.e. would you be happy to 
> continue to run this version knowing that a few large files are not being 
> backed up?
> 
> You can tell pretty easily from the backup logs. If it says "Caught 
> exception - reset state and waiting to retry" then it did not complete the 
> backup run, but aborted because of the exception.

I see. This error message is not in our logs.

I think that for all users of a backup tool it is vital that the backup
runs as reliable and complete as possible. Therefore it should continue
even if some "small" errors occur and only notify the user of them. It
should also be stated clearly (either in documentation or better in
syslog) whether an error is "small" enough and the backup continues or
if the backup run was stopped. With "clearly" I suggest that it is
explicitly written "Backup continues" or "Backup stopped" instead of the
above exception message where one needs to know the meaning and
implications.

> > How can I find out which files in the backup store belong to which 
> > original files?
> 
> Sorry, I don't know exactly what you mean here, does bbackupquery's list 
> command tell you what you want?

In our backup store we have files with hex numbers in their names, e.g.
"./3c/05/o95.rfw". How can I find out to which file on the backup client
that belongs? Or (the other way round): how can I find out which backup
store files belong to a particular file on the client?

> > Your second paragraph suggests that excluding files from the backup by 
> > adding an exclusion-statement in bbackupd.conf will remove all backups 
> > of this file. Is this true?
> 
> No, it's only that when bbackupd uploads a new version, it does not send a 
> patch against deleted versions, so it will upload a fresh copy and avoid 
> triggering the bug.

Ah, I see.

Another question: In our backup store we have lots of files with
identical size but different md5sums, e.g.
532869699 ./3d/05/o6a.rfw
532869699 ./42/04/of3.rfw
532869699 ./46/06/o24.rfw
532869699 ./8d/04/o89.rfw
532869699 ./94/03/o45.rfw
532869699 ./96/o81.rfw
532869699 ./a6/07/o00.rfw
532869699 ./c2/06/o38.rfw
532869699 ./ca/05/oa9.rfw
532869699 ./d8/04/o19.rfw
There are several such runs. I assume that these all belong to the same
original file on the client. So, my question is, if a large file changes
slightly (e.g. gets a bit longer, or a few (kilo)bytes are modified),
will the backup store then hold only the difference (like SVN) or the
whole new plus the whole old versions of the file?

To know this is important for us, because currently we intentionally
don't compress backups of database-dumps (MySQL, SVN) to allow the
diff-algorithm to find as small differences as possible. 

Thanks
  Hansi

-- 
Johann Glaser                          <glaser at ict.tuwien.ac.at>
             Institute of Computer Technology, E384
Vienna University of Technology, Gusshausstr. 27-29, A-1040 Wien
Phone: ++43/1/58801-38444                Fax: ++43/1/58801-38499



From boxbackup at fluffy.co.uk  Wed Aug 29 09:53:35 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Wed, 29 Aug 2007 09:53:35 +0100 (BST)
Subject: [Box Backup] BadBackupStoreFile
In-Reply-To: <1188373549.5379.27.camel@localhost.localdomain>
References: <1187091035.5344.4.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708251705330.20516@top.qwarx.com>
 <1188204879.5254.12.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708290108130.20516@top.qwarx.com> <1188373549.5379.27.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.64.0708290944160.20516@top.qwarx.com>

Hi Johann,

> We have two >2GB files in the backup store. See the bottom of a sorted
> listing:
> [...]
> 748651521 ./cc/02/o1f.rfw
> 748679825 ./72/03/o8e.rfw
> 748705025 ./3c/03/oa7.rfw
> 759350104 ./20/04/o10.rfw
> 759356296 ./23/06/o12.rfw
> 759552073 ./9e/06/o8f.rfw
> 759633897 ./1a/07/obc.rfw
> 1529736909 ./ba/o55.rfw
> 1539937214 ./69/01/o05.rfw
> 2744679666 ./16/01/o5f.rfw
> 5133609317 ./f2/o44.rfw

Do you have any errors restoring or comparing the other large file?

>>> Is it sure that the backup continues after the problem with all other
>>> files? Or does the backup stop after the first error?
...
> I think that for all users of a backup tool it is vital that the backup
> runs as reliable and complete as possible. Therefore it should continue
> even if some "small" errors occur and only notify the user of them.

Of course I agree, but an exception is an exception. It's not supposed to 
happen, usually no recovery strategy is planned, and it's not even sure 
that it's safe to continue execution or to continue to use the current 
connection to the server. In this case it indicates that an internal bug 
was found.

> It should also be stated clearly (either in documentation or better in 
> syslog) whether an error is "small" enough and the backup continues or 
> if the backup run was stopped. With "clearly" I suggest that it is 
> explicitly written "Backup continues" or "Backup stopped" instead of the 
> above exception message where one needs to know the meaning and 
> implications.

I agree partly, but I think that we shouldn't have to write "backup 
continues" after every error message. It should be safe to say that if you 
see a message saying that it stopped because of an exception, then it did, 
otherwise it didn't. Perhaps we should document that better.

> In our backup store we have files with hex numbers in their names, e.g.
> "./3c/05/o95.rfw". How can I find out to which file on the backup client
> that belongs? Or (the other way round): how can I find out which backup
> store files belong to a particular file on the client?

The path name is converted to ID by taking the two hex digits from each 
component, reversing the order (most significant byte is the last one, 
before ".rfw") and padding with zeroes on the left. So, for example, 
./cc/02/o1f.rfw is 001f02cc. (I think that's right anyway).

You can compare those IDs to the ones given in the remote directory 
listings in bbackupquery, but unfortunately there isn't a global reverse 
mapping so you need to manually hunt through directories to find them, 
sorry.

> Another question: In our backup store we have lots of files with
> identical size but different md5sums, e.g.
> 532869699 ./3d/05/o6a.rfw
> 532869699 ./42/04/of3.rfw
> 532869699 ./46/06/o24.rfw
> 532869699 ./8d/04/o89.rfw
> 532869699 ./94/03/o45.rfw
> 532869699 ./96/o81.rfw
> 532869699 ./a6/07/o00.rfw
> 532869699 ./c2/06/o38.rfw
> 532869699 ./ca/05/oa9.rfw
> 532869699 ./d8/04/o19.rfw
> There are several such runs. I assume that these all belong to the same
> original file on the client. So, my question is, if a large file changes
> slightly (e.g. gets a bit longer, or a few (kilo)bytes are modified),
> will the backup store then hold only the difference (like SVN) or the
> whole new plus the whole old versions of the file?

Normally it will only hold the difference. It would be interesting to know 
if these are really multiple copies of old files, and if so why that 
happened.

In order to calculate the difference, bbackupd will perform a diff of the 
file against a downloading stream. Since this could take a long time, 
there is a built-in timeout (MaximumDiffingTime) and once that time is 
exceeded, bbackupd will upload an entire new copy. If these are copies 
of the same file, you might like to increase that timeout.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Wed Aug 29 13:33:32 2007
From: boxbackup at fluffy.co.uk (Johann Glaser)
Date: Wed, 29 Aug 2007 14:33:32 +0200
Subject: [Box Backup] BadBackupStoreFile
In-Reply-To: <Pine.LNX.4.64.0708290944160.20516@top.qwarx.com>
References: <1187091035.5344.4.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708251705330.20516@top.qwarx.com>
 <1188204879.5254.12.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708290108130.20516@top.qwarx.com>
 <1188373549.5379.27.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708290944160.20516@top.qwarx.com>
Message-ID: <1188390812.10391.39.camel@localhost.localdomain>

Am Mittwoch, den 29.08.2007, 09:53 +0100 schrieb Chris Wilson:
> Hi Johann,
> 
> > We have two >2GB files in the backup store. See the bottom of a sorted
> > listing:
> > [...]
> > 748651521 ./cc/02/o1f.rfw
> > 748679825 ./72/03/o8e.rfw
> > 748705025 ./3c/03/oa7.rfw
> > 759350104 ./20/04/o10.rfw
> > 759356296 ./23/06/o12.rfw
> > 759552073 ./9e/06/o8f.rfw
> > 759633897 ./1a/07/obc.rfw
> > 1529736909 ./ba/o55.rfw
> > 1539937214 ./69/01/o05.rfw
> > 2744679666 ./16/01/o5f.rfw
> > 5133609317 ./f2/o44.rfw
> 
> Do you have any errors restoring or comparing the other large file?

Yes, there are errors:
query > compare -E . .
Local file './__db.002/__db.002' has different contents to store file './__db.002'.
Local file './__db.003/__db.003' has different contents to store file './__db.003'.
Local file './__db.004/__db.004' has different contents to store file './__db.004'.
Local file './__db.005/__db.005' has different contents to store file './__db.005'.
Local file './__db.006/__db.006' has different contents to store file './__db.006'.
ERROR: (4/48) during file fetch and comparsion for './strings'
ERROR: (7/41) during file fetch and comparsion for './transactions'
ERROR: (7/41) during file fetch and comparsion for './uuids'

[ 0 (of 5) differences probably due to file modifications after the last upload ]
Differences: 5 (0 dirs excluded, 0 files excluded)

In the directory there are some more files which haven't been mentioned
in the output above. "strings" is the only large file (7.4GB). All other
files in this directory are <55MB. In syslog I got the following errors:

Aug 29 11:59:58 clara bbstored[1703]: Incoming connection from 128.131.80.105 port 40500 (handling in child 25231)
Aug 29 11:59:58 clara bbstored[25231]: Certificate CN: BACKUP-1 
Aug 29 11:59:58 clara bbstored[25231]: Login: Client ID 00000001, Read-only
Aug 29 12:04:16 clara bbstored[25231]: Connection statistics for BACKUP-1: IN=1062 OUT=399083 TOTAL=400145 
Aug 29 12:04:16 clara bbstored[25231]: in server child, exception Connection TLSWriteFailed (Probably a network issue between client and server.) (7/33) -- terminating child
Aug 29 12:05:24 clara bbstored[1703]: Incoming connection from 128.131.80.105 port 39024 (handling in child 28114)
Aug 29 12:05:24 clara bbstored[28114]: Certificate CN: BACKUP-1 
Aug 29 12:05:24 clara bbstored[28114]: Login: Client ID 00000001, Read-only
Aug 29 12:12:44 clara bbstored[1703]: Incoming connection from 128.131.80.105 port 49901 (handling in child 29146)
Aug 29 12:12:44 clara bbstored[29146]: Certificate CN: BACKUP-1 
Aug 29 12:12:44 clara bbstored[29146]: Login: Client ID 00000001, Read-only
Aug 29 12:21:40 clara bbstored[29146]: Session finished
Aug 29 12:21:40 clara bbstored[29146]: Connection statistics for BACKUP-1: IN=992 OUT=1565396 TOTAL=1566388 
Aug 29 12:23:00 clara bbstored[1703]: Incoming connection from 128.131.80.105 port 59254 (handling in child 31071)
Aug 29 12:23:00 clara bbstored[31071]: Certificate CN: BACKUP-1 
Aug 29 12:23:00 clara bbstored[31071]: Login: Client ID 00000001, Read-only
Aug 29 12:39:51 clara bbstored[31071]: Connection statistics for BACKUP-1: IN=417 OUT=31964 TOTAL=32381 
Aug 29 12:39:51 clara bbstored[31071]: in server child, exception Connection Protocol_Timeout (Probably a network issue between client and server.) (7/41) -- terminating child

PID 28114 was still running with nearly 100% CPU when already at the
bbackupquery prompt. Typing "ls" just hang. I had to kill it, just
restarting the boxbackup-server didn't stop this task.

> Of course I agree, but an exception is an exception. It's not supposed to 
> happen, usually no recovery strategy is planned, and it's not even sure 
> that it's safe to continue execution or to continue to use the current 
> connection to the server. In this case it indicates that an internal bug 
> was found.

Yes, indeed. But I want to state that there are cases where an exception
is not an internal bug but another problem, e.g. that a (single) backup
store file was deleted or its permissions changed by somebody playing
around. Therefore there should be some fault tolerance or graceful error
recovery to not endanger the rest of the backup.

> I agree partly, but I think that we shouldn't have to write "backup 
> continues" after every error message. It should be safe to say that if you 
> see a message saying that it stopped because of an exception, then it did, 
> otherwise it didn't. Perhaps we should document that better.

Good idea.

> > In our backup store we have files with hex numbers in their names, e.g.
> > "./3c/05/o95.rfw". How can I find out to which file on the backup client
> > that belongs? Or (the other way round): how can I find out which backup
> > store files belong to a particular file on the client?
> 
> The path name is converted to ID by taking the two hex digits from each 
> component, reversing the order (most significant byte is the last one, 
> before ".rfw") and padding with zeroes on the left. So, for example, 
> ./cc/02/o1f.rfw is 001f02cc. (I think that's right anyway).

I found its a bit more complicated. For two levels above mentioned
file ./f2/o44.rfw belongs to the ID 0000f244. For three levels the
translation is ./xx/yy/ozz.rfw -> ID=00yyxxzz.

> You can compare those IDs to the ones given in the remote directory 
> listings in bbackupquery, but unfortunately there isn't a global reverse 
> mapping so you need to manually hunt through directories to find them, 
> sorry.

Hehe, thats a good point to mention a feature request. :-)

Another one: please add the feature to use "list" with filenames, e.g.
  list -dots *.txt
and with subdirectories
  list -dots db/
and combined
  list -dots */db/*.txt

> > Another question: In our backup store we have lots of files with
> > identical size but different md5sums, e.g.
> > 532869699 ./3d/05/o6a.rfw
> > 532869699 ./42/04/of3.rfw
> > 532869699 ./46/06/o24.rfw
> > 532869699 ./8d/04/o89.rfw
> > 532869699 ./94/03/o45.rfw
> > 532869699 ./96/o81.rfw
> > 532869699 ./a6/07/o00.rfw
> > 532869699 ./c2/06/o38.rfw
> > 532869699 ./ca/05/oa9.rfw
> > 532869699 ./d8/04/o19.rfw
> > There are several such runs. I assume that these all belong to the same
> > original file on the client. So, my question is, if a large file changes
> > slightly (e.g. gets a bit longer, or a few (kilo)bytes are modified),
> > will the backup store then hold only the difference (like SVN) or the
> > whole new plus the whole old versions of the file?
> 
> Normally it will only hold the difference. It would be interesting to know 
> if these are really multiple copies of old files, and if so why that 
> happened.

For some nearly-equally-sized files in the backup store I found that
they belong to the very same file on the client, so they represent
different versions. When looking with bbackupquery at old versions, all
of them have similar large size.

> In order to calculate the difference, bbackupd will perform a diff of the 
> file against a downloading stream. Since this could take a long time, 
> there is a built-in timeout (MaximumDiffingTime) and once that time is 
> exceeded, bbackupd will upload an entire new copy. If these are copies 
> of the same file, you might like to increase that timeout.

Unfortunately we backup on an external storage (Iomega StorCenter 150D)
connected with NFS over 100MBit/s which is _extremely_ slow, especially
for directory listings. So such timeouts might well be the problem. 

I found this option in bbackupd.conf. Which unit is used for the time?
Seconds? Milliseconds?

Another feature request: The backup server should (additionally) store
checksums across large blocks of files, e.g. 1MB blocks. Then only these
checksums need to be read from disk instead of the whole backup file.

Bye
  Hansi




From boxbackup at fluffy.co.uk  Wed Aug 29 20:51:08 2007
From: boxbackup at fluffy.co.uk (Matt Brown)
Date: Wed, 29 Aug 2007 20:51:08 +0100
Subject: [Box Backup] Backup Stats ?
In-Reply-To: <Pine.LNX.4.64.0708290118140.20516@top.qwarx.com>
References: <C19D949A-5347-4727-810B-E6B035617A4F@mbrown.co.uk> <Pine.LNX.4.64.0708290118140.20516@top.qwarx.com>
Message-ID: <44AFF5C3-C447-4757-B0C2-2CB6A50CBEE1@mbrown.co.uk>

> Hi Matt,
>

Hi Chris,

>> I am currently running Box Backup 0.10 (Chris Merge 1794) for a  
>> handful of clients in place of an Rsync script - clients however  
>> were used to the verbose output to show what files were backed up  
>> and the rsync summary at the end of the batch was also semi- 
>> informative...
>>
>> Is there likely to be a similar feature for Box ? I did grab the  
>> bbclientstatus.pl script along with the box::config perl mod which  
>> can email the last backup etc of a client which is great for me to  
>> see the status of the clients, and poke this data at Nagios,  
>> however I think what the clients would love to see is something  
>> along the lines of:
>>
>> 1. Did it backup last night ?
>> 2. What size was the backup.
>> 3. Any errors ?
>> 4. Space left on store/used.
>
> The Box Backup client can now log a lot of this information, if you  
> turn on the LogAllFileAccess option and run it in verbose mode  
> (sufficiently high -v level, or -V for maximum). You could grep it  
> out of your syslogs (with timestamps for everything) and report it,  
> or I can provide a way to write the logs to a file instead.
>

Is that a new option ? I did not know it existed ? - however a  
LogAllFileAccess = {filename_here}  or similar would be handy :-)

I presume if I put in LogAllFileAccess = yes  - it will poke the  
verbose detail into syslog or in my case /var/log/box ? or is it a  
parameter passed when starting the Daemon ?

> You will see it connect/disconnect (so you know that it backed up),  
> a list of files processed, ignored due to not being modified,  
> backed up successfully, or failed to back up. As far as I know, no  
> existing scripts have been modified to process and report on this  
> new verbose data.
>
> You can get the space left on store at any time with bbackupquery.
>> Keep up the fantastic work on this project, I am a great fan and  
>> of course as always I am happy to test or assist if need be :-)
>
> Thanks, it's highly appreciated!

Just out of interest, how long before 0.11 ? Is the merge stable  
enough yet ?

Regards

Matt Brown
matt at mbrown.co.uk



From boxbackup at fluffy.co.uk  Wed Aug 29 22:39:15 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Wed, 29 Aug 2007 22:39:15 +0100 (BST)
Subject: [Box Backup] Backup Stats ?
In-Reply-To: <44AFF5C3-C447-4757-B0C2-2CB6A50CBEE1@mbrown.co.uk>
References: <C19D949A-5347-4727-810B-E6B035617A4F@mbrown.co.uk>
 <Pine.LNX.4.64.0708290118140.20516@top.qwarx.com>
 <44AFF5C3-C447-4757-B0C2-2CB6A50CBEE1@mbrown.co.uk>
Message-ID: <Pine.LNX.4.64.0708292234070.20516@top.qwarx.com>

Hi Matt,

On Wed, 29 Aug 2007, Matt Brown wrote:

>> The Box Backup client can now log a lot of this information, if you 
>> turn on the LogAllFileAccess option and run it in verbose mode 
>> (sufficiently high -v level, or -V for maximum). You could grep it out 
>> of your syslogs (with timestamps for everything) and report it, or I 
>> can provide a way to write the logs to a file instead.
>
> Is that a new option ? I did not know it existed ? - however a 
> LogAllFileAccess = {filename_here} or similar would be handy :-)
>
> I presume if I put in LogAllFileAccess = yes  - it will poke the verbose 
> detail into syslog or in my case /var/log/box ? or is it a parameter passed 
> when starting the Daemon ?

It's a new config file option, added in chris/merge. Log messages will go 
to syslog (if you enable the "info" level in /etc/syslog.conf) and/or the 
console (if you run bbackupd in the foreground).

I know that having a logfile would be a useful option, but I haven't got 
around to implementing it yet.

> Just out of interest, how long before 0.11 ? Is the merge stable enough 
> yet ?

I don't know, some users (such as you) have reported good results, but 
others (Pete Jalajas and Tobias Balle-Petersen) have reported problems 
that I haven't solved yet.

I'd like to release 0.11 in the next two months or so, but not until those 
problems are resolved and not if any others appear.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Thu Aug 30 20:34:54 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Thu, 30 Aug 2007 20:34:54 +0100 (BST)
Subject: [Box Backup] BadBackupStoreFile
In-Reply-To: <1188390812.10391.39.camel@localhost.localdomain>
References: <1187091035.5344.4.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708251705330.20516@top.qwarx.com>
 <1188204879.5254.12.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708290108130.20516@top.qwarx.com>
 <1188373549.5379.27.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708290944160.20516@top.qwarx.com>
 <1188390812.10391.39.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.64.0708302016560.12581@top.qwarx.com>

Hi Johann,

On Wed, 29 Aug 2007, Johann Glaser wrote:

>>> We have two >2GB files in the backup store. See the bottom of a sorted
>>> listing:
>>> [...]
>>> 748651521 ./cc/02/o1f.rfw
>>> 748679825 ./72/03/o8e.rfw
>>> 748705025 ./3c/03/oa7.rfw
>>> 759350104 ./20/04/o10.rfw
>>> 759356296 ./23/06/o12.rfw
>>> 759552073 ./9e/06/o8f.rfw
>>> 759633897 ./1a/07/obc.rfw
>>> 1529736909 ./ba/o55.rfw
>>> 1539937214 ./69/01/o05.rfw
>>> 2744679666 ./16/01/o5f.rfw
>>> 5133609317 ./f2/o44.rfw
>>
>> Do you have any errors restoring or comparing the other large file?
>
> Yes, there are errors:
> query > compare -E . .
> Local file './__db.002/__db.002' has different contents to store file './__db.002'.
> Local file './__db.003/__db.003' has different contents to store file './__db.003'.
> Local file './__db.004/__db.004' has different contents to store file './__db.004'.
> Local file './__db.005/__db.005' has different contents to store file './__db.005'.
> Local file './__db.006/__db.006' has different contents to store file './__db.006'.
...
> [ 0 (of 5) differences probably due to file modifications after the last upload ]
> Differences: 5 (0 dirs excluded, 0 files excluded)

Are these errors expected, i.e. did those files change since the last 
backup? The message seems to indicate that they did not, and therefore 
another possible bug in 0.10. But it's also possible that Subversion or 
BDB manually changes the timestamps on these files, rendering Box Backups' 
timestamp comparison useless.

> ERROR: (4/48) during file fetch and comparsion for './strings'
> ERROR: (7/41) during file fetch and comparsion for './transactions'
> ERROR: (7/41) during file fetch and comparsion for './uuids'

The 7/41 errors are a symptom of a broken connection (loss of 
synchronisation) after the comparison for ./strings failed, which is 
expected (unfortunately). Please could you try to identify the other large 
file and to compare it separately, to see if you get a 4/48 error? (I'd 
expect so).

> In the directory there are some more files which haven't been mentioned
> in the output above. "strings" is the only large file (7.4GB). All other
> files in this directory are <55MB.

Any idea, then, what the other file over 2GB is? (./f2/o44.rfw)

> PID 28114 was still running with nearly 100% CPU when already at the 
> bbackupquery prompt. Typing "ls" just hang. I had to kill it, just 
> restarting the boxbackup-server didn't stop this task.

That's really bad, sorry. Can you reproduce this?

> Yes, indeed. But I want to state that there are cases where an exception 
> is not an internal bug but another problem, e.g. that a (single) backup 
> store file was deleted or its permissions changed by somebody playing 
> around. Therefore there should be some fault tolerance or graceful error 
> recovery to not endanger the rest of the backup.

The cases that you mention should not cause an exception to be thrown, but 
rather a recoverable error condition. If you think that they are aborting 
the backup, then I'd really appreciate your help to find out why.

>> I agree partly, but I think that we shouldn't have to write "backup 
>> continues" after every error message. It should be safe to say that if 
>> you see a message saying that it stopped because of an exception, then 
>> it did, otherwise it didn't. Perhaps we should document that better.
>
> Good idea.

What Box Backup documentation have you read so far? Do you have an idea 
where the best place to document this would be, so that you would have 
found it if it existed?

>> The path name is converted to ID by taking the two hex digits from each 
>> component, reversing the order (most significant byte is the last one, 
>> before ".rfw") and padding with zeroes on the left. So, for example, 
>> ./cc/02/o1f.rfw is 001f02cc. (I think that's right anyway).
>
> I found its a bit more complicated. For two levels above mentioned file 
> ./f2/o44.rfw belongs to the ID 0000f244. For three levels the 
> translation is ./xx/yy/ozz.rfw -> ID=00yyxxzz.

OK, sorry, you learn something every day :-)

>> You can compare those IDs to the ones given in the remote directory 
>> listings in bbackupquery, but unfortunately there isn't a global 
>> reverse mapping so you need to manually hunt through directories to 
>> find them, sorry.
>
> Hehe, thats a good point to mention a feature request. :-)

OK, added to http://bbdev.fluffy.co.uk/trac/wiki/FeatureRequests. Feel 
free to add your own feature requests there too.

> For some nearly-equally-sized files in the backup store I found that
> they belong to the very same file on the client, so they represent
> different versions. When looking with bbackupquery at old versions, all
> of them have similar large size.
...
> Unfortunately we backup on an external storage (Iomega StorCenter 150D)
> connected with NFS over 100MBit/s which is _extremely_ slow, especially
> for directory listings. So such timeouts might well be the problem.
>
> I found this option in bbackupd.conf. Which unit is used for the time?
> Seconds? Milliseconds?

Units are seconds. Where did you look for this information, i.e. where 
should we improve the documentation?

> Another feature request: The backup server should (additionally) store 
> checksums across large blocks of files, e.g. 1MB blocks. Then only these 
> checksums need to be read from disk instead of the whole backup file.

I have a feeling that we already do, but I'm not 100% sure. Ben, do you 
know?

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Thu Aug 30 20:43:42 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Thu, 30 Aug 2007 20:43:42 +0100 (BST)
Subject: [Box Backup] BoxBackup (chris/merge) - Store Error
In-Reply-To: <1D4EFCF0-2501-4947-B261-B0F0DEE59532@mbrown.co.uk>
References: <1D4EFCF0-2501-4947-B261-B0F0DEE59532@mbrown.co.uk>
Message-ID: <Pine.LNX.4.64.0708302040370.12581@top.qwarx.com>

Hi Matt,

> Hope the hols in mexico were good and your feeling refreshed :-)

Yes, they were great, thanks! It's really hot and humid in Yucatan, and we 
had a hurricane which spiced things up a bit :-)

> When I run the bbstoreaccounts check option I get ...
>
> root at pegasus:~# /usr/local/bin/bbstoreaccounts check 1
> Checking store account ID 0x000001...
> Phase 1, check objects...
> Phase 2, check directories...
> Phase 3, check root...
> WARNING: Root directory doesn't exist

That's very interesting. Since backups are working correctly, and 
bbstoreaccounts' attempt to recreate it fails, I believe that the root 
directory does actually exist, and is not found for some reason. 
bbstoreaccounts appears to be checking some kind of block index to find 
it. I don't claim to understand exactly what it's doing. Ben, do you have 
any ideas?

If nobody else can help, and you're willing to try a new version of Box 
Backup, I can add some debugging code to bbstoreaccounts to help track 
this down.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Thu Aug 30 20:45:40 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Thu, 30 Aug 2007 20:45:40 +0100 (BST)
Subject: [Box Backup] boxbackup bug with openssl
In-Reply-To: <Pine.LNX.4.64.0708242328001.20516@top.qwarx.com>
References: <46BB851A.7080104@lanten.hu> <Pine.LNX.4.64.0708112135460.10354@rocio.int.aidworld.org>
 <46BF6295.1010503@lanten.hu> <Pine.LNX.4.64.0708242328001.20516@top.qwarx.com>
Message-ID: <Pine.LNX.4.64.0708302044570.12581@top.qwarx.com>

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

---128931150-1894675568-1188503140=:12581
Content-Type: TEXT/PLAIN; charset=utf-8; format=flowed
Content-Transfer-Encoding: QUOTED-PRINTABLE

Hi Marton,

On Fri, 24 Aug 2007, Chris Wilson wrote:

> Sorry for tthe delayed reply, I've been on holiday in Mexico without acce=
ss=20
> to email and still working through the backlog.
>
> There is a "secret" option to enable compatibility with OpenSSL 0.9.6, bu=
t=20
> this is definitely not recommended. In any case I doubt very much that it=
=20
> will help you with this problem unless you are actually running 0.9.6 on =
this=20
> client.
>
> There is a serious bug with OpenSSL 0.9.8d/e (see=20
> http://bbdev.fluffy.co.uk/trac/ticket/21 for details). What hardware, OS,=
 and=20
> OpenSSL version are you running on? And what version of Box Backup, 0.10=
=20
> release?
>
> If you are running OpenSSL 0.9.8d/e and refuse to upgrade or downgrade, t=
hen=20
> I'm afraid I don't have the resoures to help you right now. If not, then =
I=20
> would really appreciate the details of your client configuration.
>
> When you say "=A3i has worked with that version once", which version do y=
ou=20
> mean? And do you know what you did just before it stopped working?

Do you have any more information for me? I'd really like to get to the=20
bottom of this before releasing 0.11, which is imminent.

Cheers, Chris.
--=20
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |
---128931150-1894675568-1188503140=:12581--


From boxbackup at fluffy.co.uk  Thu Aug 30 20:55:27 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Thu, 30 Aug 2007 20:55:27 +0100 (BST)
Subject: [Box Backup] Failed to setup location
In-Reply-To: <Pine.LNX.4.64.0707292322350.23507@top.qwarx.com>
References: <819133.80520.qm@web60618.mail.yahoo.com>
 <Pine.LNX.4.64.0707292322350.23507@top.qwarx.com>
Message-ID: <Pine.LNX.4.64.0708302054520.12581@top.qwarx.com>

Hi Pete,

On Sun, 29 Jul 2007, Chris Wilson wrote:

> Hi Pete,
>
> On Thu, 26 Jul 2007, E.W. Peter Jalajas wrote:
>
>>  Thanks again, Chris.
>> 
>> > >  OK, I've downloaded all three, 1569, 1516, and 1280.  (We tried 1662
>> > >  last night to no avail; see the bug report.)
>
> Please could you also try the latest release, 1781? It probably won't fix the 
> problem but it does include much more debugging information in this case, 
> which should help me to track down the problem.

Do you have any more feedback for me about this problem? I'd really like 
to get it fixed before releasing 0.11.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Thu Aug 30 20:57:50 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Thu, 30 Aug 2007 20:57:50 +0100 (BST)
Subject: [Box Backup] Redundant locations not removed?
In-Reply-To: <Pine.LNX.4.64.0707261320390.15754@top.qwarx.com>
References: <469CE4C2.90602@kontrapunkt.com> <Pine.LNX.4.64.0707180937160.28048@top.qwarx.com>
 <469E58AC.7000702@kontrapunkt.com> <Pine.LNX.4.64.0707182051330.28048@top.qwarx.com>
 <469E7AC1.2090208@kontrapunkt.com> <Pine.LNX.4.64.0707182212100.28048@top.qwarx.com>
 <Pine.LNX.4.64.0707182258150.28048@top.qwarx.com> <469F5D31.90107@kontrapunkt.com>
 <Pine.LNX.4.64.0707192033070.9380@top.qwarx.com> <46A5DF82.1060809@kontrapunkt.com>
 <Pine.LNX.4.64.0707242041260.23507@top.qwarx.com> <46A718BA.9070607@kontrapunkt.com>
 <46A8603E.20705@kontrapunkt.com> <Pine.LNX.4.64.0707261320390.15754@top.qwarx.com>
Message-ID: <Pine.LNX.4.64.0708302057200.12581@top.qwarx.com>

Hi Tobias,

On Thu, 26 Jul 2007, Chris Wilson wrote:

>>  When the client starts to upload to the store, the store is 96% full (soft
>>  limit). The upload continues until the store is full. Housekeeping on the
>>  store does not delete any "deleted" files while the client is logged in.
>>  The client gets disconnected when the store is full. The following
>>  housekeeping on the store will delete "deleted" files until the usage of
>>  the store is again 96% (soft limit). At this point I can initiate another
>>  sync. The 4 hours is the time it takes to move enough data to get from 96%
>>  to 99%(100%).
>>
>>  Is this the way it is supposed to work?
>>
>>  I guess I could lower the soft limit temporarely to get around this.
>
> That's very interesting, the errors in the logs put me off suspecting this.
>
> Jul 25 20:02:30 yoiko bbackupd[24426]: SSL err during Write:
>  error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry
> Jul 25 20:02:30 yoiko bbackupd[24426]: Exception caught (Connection
>   TLSWriteFailed (Probably a network issue between client and server.)
>   7/33), reset state and waiting to retry...
>
> Did you have anything in the server logs around this time, lime an exception?

Do you have any more feedback for me about this problem? I'd really like
to get it fixed before releasing 0.11.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Thu Aug 30 20:12:41 2007
From: boxbackup at fluffy.co.uk (E.W. Peter Jalajas)
Date: Thu, 30 Aug 2007 12:12:41 -0700 (PDT)
Subject: [Box Backup] Failed to setup location
In-Reply-To: <Pine.LNX.4.64.0707292322350.23507@top.qwarx.com>
Message-ID: <948614.11399.qm@web60613.mail.yahoo.com>

Hi Chris,

I hope you had a nice vacation in Mexico.  

We finally got a chance to test 1781 and below is what we see in eventvwr.msc.

(Again, J:\ is a mapped drive to a share on another box across the LAN.)

What do you think?

Thanks,
Pete

Failed to open file 'D:\Program Files\Box Backup\bbackupd\bbackupd.dat': The system cannot find
the file specified. (2)

Starting daemon, version chris_general_1781, config: D:\Program Files\Box Backup\bbackupd.conf

Failed to open file 'D:\Program Files\Box Backup\bbackupd\bbackupd.dat': The system cannot find
the file specified. (2)

Exception thrown: CommonException(OSFileOpenError) at FileStream.cpp(49)

Internal error reading store object info file: D:\Program Files\Box Backup\bbackupd\bbackupd.dat:
Common OSFileOpenError (Can't open a file -- attempted to load a non-existant config file or bad
file referenced within?)

Store object info file is missing, not accessible, or inconsistent. Will re-cache from store.
(D:\Program Files\Box Backup\bbackupd\bbackupd.dat)

Beginning scan of local files

... (login succeeds)

Failed to stat location: J:\04CommonFiles: No such file or directory

Exception thrown: CommonException(OSFileError) at BackupDaemon.cpp(1703)

I opened up Windows Explorer and pasted J:\04CommonFiles into the address bar and that directory
opened up fine.



Chris Wilson <chris at qwirx.com> wrote:

    Please could you also try the latest release, 1781? It probably won't fix
    the problem but it does include much more debugging information in this
    case, which should help me to track down the problem.



From boxbackup at fluffy.co.uk  Fri Aug 31 09:26:51 2007
From: boxbackup at fluffy.co.uk (Johann Glaser)
Date: Fri, 31 Aug 2007 10:26:51 +0200
Subject: [Box Backup] Wiki Account request
Message-ID: <1188548811.6494.17.camel@localhost.localdomain>

Hi!

Could you please create a Wiki account for me? I'd like to get the
username "hansi".

Thanks
  Hansi

-- 
Johann Glaser                          <glaser at ict.tuwien.ac.at>
             Institute of Computer Technology, E384
Vienna University of Technology, Gusshausstr. 27-29, A-1040 Wien
Phone: ++43/1/58801-38444                Fax: ++43/1/58801-38499



From boxbackup at fluffy.co.uk  Fri Aug 31 09:40:16 2007
From: boxbackup at fluffy.co.uk (Tobias Balle-Petersen)
Date: Fri, 31 Aug 2007 10:40:16 +0200
Subject: [Box Backup] Redundant locations not removed?
In-Reply-To: <Pine.LNX.4.64.0708302057200.12581@top.qwarx.com>
References: <469CE4C2.90602@kontrapunkt.com> <Pine.LNX.4.64.0707180937160.28048@top.qwarx.com> <469E58AC.7000702@kontrapunkt.com> <Pine.LNX.4.64.0707182051330.28048@top.qwarx.com> <469E7AC1.2090208@kontrapunkt.com> <Pine.LNX.4.64.0707182212100.28048@top.qwarx.com> <Pine.LNX.4.64.0707182258150.28048@top.qwarx.com> <469F5D31.90107@kontrapunkt.com> <Pine.LNX.4.64.0707192033070.9380@top.qwarx.com> <46A5DF82.1060809@kontrapunkt.com> <Pine.LNX.4.64.0707242041260.23507@top.qwarx.com> <46A718BA.9070607@kontrapunkt.com> <46A8603E.20705@kontrapunkt.com> <Pine.LNX.4.64.0707261320390.15754@top.qwarx.com> <Pine.LNX.4.64.0708302057200.12581@top.qwarx.com>
Message-ID: <46D7D3F0.4040700@kontrapunkt.com>

Hello Chris.

I ended up reverting to 0.10. I waited for the files marked for deletion (Entries in the conf.) to
be deleted. I then reverted to your merge, and things are working as expected.

2GB+ files are also beeing backed up correctly now. Stores id debian and client is OS X.

Thanks,
Tobias


Chris Wilson wrote:
> Hi Tobias,
> 
> On Thu, 26 Jul 2007, Chris Wilson wrote:
> 
>>>  When the client starts to upload to the store, the store is 96% full
>>> (soft
>>>  limit). The upload continues until the store is full. Housekeeping
>>> on the
>>>  store does not delete any "deleted" files while the client is logged
>>> in.
>>>  The client gets disconnected when the store is full. The following
>>>  housekeeping on the store will delete "deleted" files until the
>>> usage of
>>>  the store is again 96% (soft limit). At this point I can initiate
>>> another
>>>  sync. The 4 hours is the time it takes to move enough data to get
>>> from 96%
>>>  to 99%(100%).
>>>
>>>  Is this the way it is supposed to work?
>>>
>>>  I guess I could lower the soft limit temporarely to get around this.
>>
>> That's very interesting, the errors in the logs put me off suspecting
>> this.
>>
>> Jul 25 20:02:30 yoiko bbackupd[24426]: SSL err during Write:
>>  error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry
>> Jul 25 20:02:30 yoiko bbackupd[24426]: Exception caught (Connection
>>   TLSWriteFailed (Probably a network issue between client and server.)
>>   7/33), reset state and waiting to retry...
>>
>> Did you have anything in the server logs around this time, lime an
>> exception?
> 
> Do you have any more feedback for me about this problem? I'd really like
> to get it fixed before releasing 0.11.
> 
> Cheers, Chris.



From boxbackup at fluffy.co.uk  Fri Aug 31 09:48:06 2007
From: boxbackup at fluffy.co.uk (Johann Glaser)
Date: Fri, 31 Aug 2007 10:48:06 +0200
Subject: [Box Backup] BadBackupStoreFile
In-Reply-To: <Pine.LNX.4.64.0708302016560.12581@top.qwarx.com>
References: <1187091035.5344.4.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708251705330.20516@top.qwarx.com>
 <1188204879.5254.12.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708290108130.20516@top.qwarx.com>
 <1188373549.5379.27.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708290944160.20516@top.qwarx.com>
 <1188390812.10391.39.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708302016560.12581@top.qwarx.com>
Message-ID: <1188550086.6494.32.camel@localhost.localdomain>

Hi!

> > Yes, there are errors:
> > query > compare -E . .
> > Local file './__db.002/__db.002' has different contents to store file './__db.002'.
> > Local file './__db.003/__db.003' has different contents to store file './__db.003'.
> > Local file './__db.004/__db.004' has different contents to store file './__db.004'.
> > Local file './__db.005/__db.005' has different contents to store file './__db.005'.
> > Local file './__db.006/__db.006' has different contents to store file './__db.006'.
> ...
> > [ 0 (of 5) differences probably due to file modifications after the last upload ]
> > Differences: 5 (0 dirs excluded, 0 files excluded)
> 
> Are these errors expected, i.e. did those files change since the last 
> backup? The message seems to indicate that they did not, and therefore 
> another possible bug in 0.10. But it's also possible that Subversion or 
> BDB manually changes the timestamps on these files, rendering Box Backups' 
> timestamp comparison useless.

Before backing up SVN files I do a hotcopy and only backup these
"exported" databases. So these don't change any more because SVN doesn't
know of its copies.

bbackupquery says the date of the above mentioned files is the same as
the date in the file system.

BTW: bbackupquery gives the time in (i guess) UTC while in the
filesystem we have +2h (Austria, Vienna, DST). E.g. for __db.002
bbackupquery says "2007-08-10T01:12:19" while the filesystem (using "ls
-l" says "2007-08-10 03:12". The same holds for some other files I've
randomly checked.

> > ERROR: (4/48) during file fetch and comparsion for './strings'
> > ERROR: (7/41) during file fetch and comparsion for './transactions'
> > ERROR: (7/41) during file fetch and comparsion for './uuids'
> 
> The 7/41 errors are a symptom of a broken connection (loss of 
> synchronisation) after the comparison for ./strings failed, which is 
> expected (unfortunately). Please could you try to identify the other large 
> file and to compare it separately, to see if you get a 4/48 error? (I'd 
> expect so).

Please see below, I got the same error. I have to mention that all these
tests were accomplished while the daily backup run was active (it
usually lasts from 3:00-

> > In the directory there are some more files which haven't been mentioned
> > in the output above. "strings" is the only large file (7.4GB). All other
> > files in this directory are <55MB.
> 
> Any idea, then, what the other file over 2GB is? (./f2/o44.rfw)

./f2/o44.rfw    is .../linux/db/strings    (7.5GiB -> 5.1GiB)
./16/01/o5f.rfw is .../qm/db/strings       (3.1GiB -> 2.7GiB)
./69/01/o05.rfw is .../docstore/db/strings (2.1GiB -> 1.5GiB)

I started a "compare -E . ." in .../docstore/db/ which prints the
following:
query > compare -E . .
Local file './__db.001' does not exist, but store file './__db.001' does.
Local file './__db.002' does not exist, but store file './__db.002' does.
Local file './__db.003' does not exist, but store file './__db.003' does.
Local file './__db.004' does not exist, but store file './__db.004' does.
Local file './__db.005' does not exist, but store file './__db.005' does.
Local file './__db.006' does not exist, but store file './__db.006' does.
Local file './__db.register' does not exist, but store file './__db.register' does.
Local file './log.0000005287' does not exist, but store file './log.0000005287' does.
Local file './log.0000005288' does not exist, but store file './log.0000005288' does.
ERROR: (4/48) during file fetch and comparsion for './strings'
ERROR: (7/41) during file fetch and comparsion for './transactions'
ERROR: (7/41) during file fetch and comparsion for './uuids'

[ 0 (of 9) differences probably due to file modifications after the last
upload ]
Differences: 9 (0 dirs excluded, 0 files excluded)

The files which don't exist really don't exist (any more?, probably the
currently running backup didn't come across this directory yet). The
three Error lines were printed with several minutes delay between them.

> > PID 28114 was still running with nearly 100% CPU when already at the 
> > bbackupquery prompt. Typing "ls" just hang. I had to kill it, just 
> > restarting the boxbackup-server didn't stop this task.
> 
> That's really bad, sorry. Can you reproduce this?

Now (after the above error 4/48 not when starting bbackupquery it prints

Box Backup Query Tool v0.10, (c) Ben Summers and contributors 2003-2006
Using configuration file /etc/boxbackup/bbackupd.conf
Connecting to store...
Handshake with store...
Login to store...

and hangs.

> The cases that you mention should not cause an exception to be thrown, but 
> rather a recoverable error condition. If you think that they are aborting 
> the backup, then I'd really appreciate your help to find out why.

The problem is that I can't check wbether such conditions stop the
backup or not. That was why I've written my concerns a few emails
before.

> What Box Backup documentation have you read so far? Do you have an idea 
> where the best place to document this would be, so that you would have 
> found it if it existed?

The main documentation was an installation guide at a German Linux news
site (http://www.pro-linux.de/berichte/boxbackup.html). I've also
visited the BoxBackup main site and its Wiki. I'm not really sure but I
remember there was no explicit documentation about logging, so I'd
suggest to write such a section.

> >> The path name is converted to ID by taking the two hex digits from each 
> >> component, reversing the order (most significant byte is the last one, 
> >> before ".rfw") and padding with zeroes on the left. So, for example, 
> >> ./cc/02/o1f.rfw is 001f02cc. (I think that's right anyway).
> >
> > I found its a bit more complicated. For two levels above mentioned file 
> > ./f2/o44.rfw belongs to the ID 0000f244. For three levels the 
> > translation is ./xx/yy/ozz.rfw -> ID=00yyxxzz.
> 
> OK, sorry, you learn something every day :-)

Could you please provide a tool (e.g. as a new command for bbackupquery)
to do this translation in both directions? I wanted to write this to the
feature request Wiki but didn't have an account.

> > For some nearly-equally-sized files in the backup store I found that
> > they belong to the very same file on the client, so they represent
> > different versions. When looking with bbackupquery at old versions, all
> > of them have similar large size.
> ...
> > Unfortunately we backup on an external storage (Iomega StorCenter 150D)
> > connected with NFS over 100MBit/s which is _extremely_ slow, especially
> > for directory listings. So such timeouts might well be the problem.
> >
> > I found this option in bbackupd.conf. Which unit is used for the time?
> > Seconds? Milliseconds?
> 
> Units are seconds. Where did you look for this information, i.e. where 
> should we improve the documentation?

I'd suggest directly in the config file comments.

Currently the value is 20 sec. which is really small. I assume this time
is the wall clock time from starting the diff until giving up, including
all delays while the daemon waits for the transfer of the backup store
file, reading the real file, ...

I suggest to measure this time only as real CPU usage for the diff
algorithm, so that the data transfer delays, ... are excluded. Or even
better: make it configurable, e.g. by introducing a second config
variable called MaximumDiffingCpuTime which is checked as mentioned. If
one of the two variables is set to 0 no limit is applied, of both are
set to >0, the limit which is reached first is applied.

A note about the diff algorithm: The "diff" tool provided by my Linux
distribution wants to load both files into memory and then does the
diff. This is bad for large files due to memory consumption. Usually the
differences of files are either huge or tiny, so it should be enough to
have a several MB sliding window. I ask to use such an algorithm instead
of the whole-file-approach (if it is not already done) for Box Backup.

Bye
  Hansi




From boxbackup at fluffy.co.uk  Fri Aug 31 09:57:20 2007
From: boxbackup at fluffy.co.uk (Ben Summers)
Date: Fri, 31 Aug 2007 09:57:20 +0100
Subject: [Box Backup] BadBackupStoreFile
Message-ID: <934A2180-C674-4609-B824-7CEF58396C3B@fluffy.co.uk>

On Thu, 30 Aug 2007, Chris Wilson wrote:
>> Another feature request: The backup server should (additionally)  
>> store
>> checksums across large blocks of files, e.g. 1MB blocks. Then only  
>> these
>> checksums need to be read from disk instead of the whole backup file.
>>
>
> I have a feeling that we already do, but I'm not 100% sure. Ben, do  
> you
> know?

Checksums are per-block, encrypted. The block size depends on the  
size of the file. You can do a compare using the checksums only using  
the -q option, and only the checksums are read from disc and  
transferred over the network.

Ben







From boxbackup at fluffy.co.uk  Fri Aug 31 10:02:35 2007
From: boxbackup at fluffy.co.uk (Ben Summers)
Date: Fri, 31 Aug 2007 10:02:35 +0100
Subject: [Box Backup] BoxBackup (chris/merge) - Store Error
Message-ID: <885DC1C0-98BA-499A-87C9-83FF76902FE1@fluffy.co.uk>

On Thu, 30 Aug 2007, Chris Wilson wrote:
>
>
>> When I run the bbstoreaccounts check option I get ...
>>
>> root at pegasus:~# /usr/local/bin/bbstoreaccounts check 1
>> Checking store account ID 0x000001...
>> Phase 1, check objects...
>> Phase 2, check directories...
>> Phase 3, check root...
>> WARNING: Root directory doesn't exist
>>
>
> That's very interesting. Since backups are working correctly, and
> bbstoreaccounts' attempt to recreate it fails, I believe that the root
> directory does actually exist, and is not found for some reason.
> bbstoreaccounts appears to be checking some kind of block index to  
> find
> it. I don't claim to understand exactly what it's doing. Ben, do  
> you have
> any ideas?


It's possible that there are renmants of a previous attempt to over  
write the root directory, so there's enough there to stop it being  
rewritten and not enough to allow it to be read.

BackupStoreCheck::CreateBlankDirectory() in lib/backupstore/ 
BackupStoreCheck2.cpp could be modified to force overwriting of the  
root. If it couldn't be found in earlier checks, I reckon overwriting  
anything left is fine.

Change

   	obj.Open(false /* don't allow overwriting */);
to
	obj.Open(true /* allow overwriting */);

and it should be fine. Check that the other callers will be fine too.

Also, can we have an ls -la for the root directory of the store,  
containing the o01.* files? Then you can write a test which sets  
things up so it looks like your situation, and we can test to make  
sure it's recovered in future.

Ben







From boxbackup at fluffy.co.uk  Fri Aug 31 13:09:18 2007
From: boxbackup at fluffy.co.uk (Tobias Heinzmann)
Date: Fri, 31 Aug 2007 14:09:18 +0200
Subject: [Box Backup] NATed Server, static IP clients
Message-ID: <20070831120918.306710@gmx.net>

Hi list,

I`ve got a server thats behind a NAT and two clients that 
have their own static IPs, spreaded on the broad wide net.

To use the server for this clients, too, I thought of opening
a tunnel from the server to the clients so that the clients
simply can connect to themselves and get redirected to the server.
Because there is no way to get an additional forwarding rule
in the fw. :-/

Think thats better than running a server on each client-machine
and copy the backuped data via cron-scripting to the "real" server.

Now the problem is:

- how do I tell the clients to connect to themselves, while using the
  real server cert?

Thanks and best regards!
-toby

-- 
GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS.
Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail


From boxbackup at fluffy.co.uk  Fri Aug 31 14:13:06 2007
From: boxbackup at fluffy.co.uk (Matt Brown)
Date: Fri, 31 Aug 2007 14:13:06 +0100
Subject: [Box Backup] BoxBackup (chris/merge) - Store Error
In-Reply-To: <Pine.LNX.4.64.0708302040370.12581@top.qwarx.com>
References: <1D4EFCF0-2501-4947-B261-B0F0DEE59532@mbrown.co.uk> <Pine.LNX.4.64.0708302040370.12581@top.qwarx.com>
Message-ID: <F8A6145F-5559-4FEC-AD76-AA315DE6DA18@mbrown.co.uk>

> Hi Matt,
>

Hi Chris,

>> Hope the hols in mexico were good and your feeling refreshed :-)
>
> Yes, they were great, thanks! It's really hot and humid in Yucatan,  
> and we had a hurricane which spiced things up a bit :-)
>
>> When I run the bbstoreaccounts check option I get ...
>>
>> root at pegasus:~# /usr/local/bin/bbstoreaccounts check 1
>> Checking store account ID 0x000001...
>> Phase 1, check objects...
>> Phase 2, check directories...
>> Phase 3, check root...
>> WARNING: Root directory doesn't exist
>
> That's very interesting. Since backups are working correctly, and  
> bbstoreaccounts' attempt to recreate it fails, I believe that the  
> root directory does actually exist, and is not found for some  
> reason. bbstoreaccounts appears to be checking some kind of block  
> index to find it. I don't claim to understand exactly what it's  
> doing. Ben, do you have any ideas?
>

Well I have just implemented this server with 4 clients so far - so  
we are still in the honeymoon phase of getting the data.. upto approx  
200GB so far. I have managed to restore data from the store  
successfully and without issues.

> If nobody else can help, and you're willing to try a new version of  
> Box Backup, I can add some debugging code to bbstoreaccounts to  
> help track this down.
>

As always, happy to assist.

Matt



From boxbackup at fluffy.co.uk  Fri Aug 31 14:38:29 2007
From: boxbackup at fluffy.co.uk (Matt Brown)
Date: Fri, 31 Aug 2007 14:38:29 +0100
Subject: [Box Backup] BoxBackup (chris/merge) - Store Error
In-Reply-To: <885DC1C0-98BA-499A-87C9-83FF76902FE1@fluffy.co.uk>
References: <885DC1C0-98BA-499A-87C9-83FF76902FE1@fluffy.co.uk>
Message-ID: <F700F352-7F51-4570-9A4B-05DA460C3A44@mbrown.co.uk>

Hi Ben,

>>> When I run the bbstoreaccounts check option I get ...
>>>
>>> root at pegasus:~# /usr/local/bin/bbstoreaccounts check 1
>>> Checking store account ID 0x000001...
>>> Phase 1, check objects...
>>> Phase 2, check directories...
>>> Phase 3, check root...
>>> WARNING: Root directory doesn't exist
>>>
>>
>> That's very interesting. Since backups are working correctly, and
>> bbstoreaccounts' attempt to recreate it fails, I believe that the  
>> root
>> directory does actually exist, and is not found for some reason.
>> bbstoreaccounts appears to be checking some kind of block index to  
>> find
>> it. I don't claim to understand exactly what it's doing. Ben, do  
>> you have
>> any ideas?

> It's possible that there are renmants of a previous attempt to over  
> write the root directory, so there's enough there to stop it being  
> rewritten and not enough to allow it to be read.
>

Well the backup store is mounted as /data which is infact /dev/md0 as  
this is 3 x 500GB drives running in software raid to give 932G of  
usable space. The 3 drives are SATA 3GB/s connected to a 4 Port SATA  
PCI-X card.

I then formatted the /data (/dev/md0) as Reiser FS , ran the box  
backup raidconf and it wrote a single dir called backup which now all  
the client stores reside in i.e 000001 000002 etc.

The /data mount point should have been clean I would have thought ??  
or is this due to running software raid ?

> BackupStoreCheck::CreateBlankDirectory() in lib/backupstore/ 
> BackupStoreCheck2.cpp could be modified to force overwriting of the  
> root. If it couldn't be found in earlier checks, I reckon  
> overwriting anything left is fine.
>
> Change
>
>   	obj.Open(false /* don't allow overwriting */);
> to
> 	obj.Open(true /* allow overwriting */);
>
> and it should be fine. Check that the other callers will be fine too.
>

Ok, I can do this - however I have just setup four clients who are  
backing data to the store, is this likely to cause any issue with the  
data grabbed so far ?  If this needs be, we can re-sync - just need  
to know before hand.

> Also, can we have an ls -la for the root directory of the store,  
> containing the o01.* files? Then you can write a test which sets  
> things up so it looks like your situation, and we can test to make  
> sure it's recovered in future.
>

root at pegasus:/data/backup/00000001# ls -al
total 21239
drwxr-xr-x 188 _bbstored _bbstored   10688 Aug 31 14:28 .
drwxr-xr-x   8 _bbstored _bbstored     192 Aug 28 11:22 ..
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:28 01
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:27 02
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:13 03
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:13 04
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:13 05
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:13 06
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:14 07
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:14 08
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:14 09
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:14 0a
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:14 0b
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:15 0c
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:15 0d
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:15 0e
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:16 0f
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:16 10
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 24 10:49 11
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 22 18:32 12
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:17 13
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:17 14
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 22 18:32 15
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:17 16
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:18 17
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 22 18:32 18
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:18 19
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:18 1a
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:18 1b
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:18 1c
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 22 18:32 1d
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 22 18:32 1e
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:18 1f
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 23:28 20
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:19 21
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 23:28 22
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:19 23
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:19 24
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 23:28 25
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:20 26
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:20 27
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:20 28
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:21 29
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:21 2a
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:21 2b
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:21 2c
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 23:28 2d
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 24 10:49 2e
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 22 18:32 2f
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 22 18:32 30
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 24 10:49 31
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:22 32
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:23 33
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 22 18:32 34
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:23 35
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:23 36
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:23 37
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:23 38
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:24 39
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 22 18:33 3a
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:24 3b
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 24 10:49 3c
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:24 3d
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:25 3e
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 22 18:33 3f
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:25 40
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:25 41
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:25 42
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:25 43
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:26 44
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:26 45
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:26 46
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:27 47
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:28 48
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:28 49
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:28 4a
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:29 4b
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:29 4c
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:29 4d
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:29 4e
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:30 4f
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:30 50
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:31 51
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:31 52
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:31 53
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:32 54
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:32 55
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:33 56
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:33 57
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:33 58
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:34 59
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:34 5a
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:34 5b
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:35 5c
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:35 5d
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:36 5e
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:36 5f
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:37 60
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:38 61
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:39 62
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:39 63
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:40 64
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:40 65
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:41 66
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:42 67
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:43 68
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:43 69
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:27 6a
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:27 6b
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 17:03 6c
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:45 6d
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:46 6e
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:46 6f
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:46 70
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:46 71
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:46 72
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:46 73
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:47 74
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:47 75
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:47 76
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:47 77
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:47 78
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:47 79
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 29 23:28 7a
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:48 7b
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 21 18:48 7c
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 29 23:28 7d
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 29 23:28 7e
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 29 23:30 7f
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:28 80
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 01:10 81
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 01:10 82
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 01:10 83
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 29 23:29 84
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 19:05 85
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 28 10:04 86
drwxr-xr-x   2 _bbstored _bbstored    6144 Aug 30 23:43 87
drwxr-xr-x   2 _bbstored _bbstored    6144 Aug 25 17:16 88
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 23:28 89
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 01:09 8a
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 01:10 8b
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 24 16:53 8c
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 28 23:28 8d
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:02 8e
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:02 8f
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:02 90
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:03 91
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:03 92
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:03 93
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:03 94
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:28 95
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:28 96
drwxr-xr-x   2 _bbstored _bbstored    6168 Aug 30 23:28 97
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:07 98
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:28 99
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:28 9a
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:09 9b
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:09 9c
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 18:06 9d
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:10 9e
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:10 9f
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:10 a0
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:10 a1
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:10 a2
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 10:10 a3
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:28 a4
drwxr-xr-x   2 _bbstored _bbstored    6168 Aug 28 10:01 a5
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 01:08 a6
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 19:04 a7
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 01:12 a8
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 19:04 a9
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 19:04 aa
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 19:05 ab
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 19:05 ac
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 21:05 ad
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 01:10 ae
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 27 01:12 af
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 19:04 b0
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 25 19:04 b1
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:27 b2
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:27 b3
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 29 23:29 b4
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 28 09:58 b5
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 29 23:29 b6
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 28 23:28 b7
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 29 23:29 b8
drwxr-xr-x   2 _bbstored _bbstored    6192 Aug 30 23:28 b9
drwxr-xr-x   2 _bbstored _bbstored    5784 Aug 30 23:28 ba
-rw-r--r--   1 _bbstored _bbstored      88 Aug 31 14:28 info.rfw
-rw-r--r--   1 _bbstored _bbstored     232 Aug 30 23:27 o01.rfw
-rw-r--r--   1 _bbstored _bbstored     337 Aug 21 18:49 o02.rfw
-rw-r--r--   1 _bbstored _bbstored   10617 Aug 30 23:27 o03.rfw
-rw-r--r--   1 _bbstored _bbstored    3033 Aug 30 23:27 o04.rfw
-rw-r--r--   1 _bbstored _bbstored    1264 Aug 21 18:12 o05.rfw
-rw-r--r--   1 _bbstored _bbstored     456 Aug 21 18:12 o06.rfw
-rw-r--r--   1 _bbstored _bbstored     408 Aug 21 18:12 o07.rfw
-rw-r--r--   1 _bbstored _bbstored     151 Aug 21 18:12 o08.rfw
-rw-r--r--   1 _bbstored _bbstored    1913 Aug 21 18:12 o09.rfw
-rw-r--r--   1 _bbstored _bbstored     232 Aug 21 18:12 o0a.rfw
-rw-r--r--   1 _bbstored _bbstored     280 Aug 21 18:12 o0b.rfw
-rw-r--r--   1 _bbstored _bbstored 6102498 Aug 21 18:12 o0c.rfw
-rw-r--r--   1 _bbstored _bbstored     776 Aug 21 18:12 o0d.rfw
-rw-r--r--   1 _bbstored _bbstored     392 Aug 21 18:12 o0e.rfw
-rw-r--r--   1 _bbstored _bbstored     257 Aug 25 18:03 o0f.rfw
-rw-r--r--   1 _bbstored _bbstored    1064 Aug 21 18:12 o10.rfw
-rw-r--r--   1 _bbstored _bbstored    9385 Aug 30 23:27 o11.rfw
-rw-r--r--   1 _bbstored _bbstored     592 Aug 21 18:12 o12.rfw
-rw-r--r--   1 _bbstored _bbstored      89 Aug 21 18:12 o13.rfw
-rw-r--r--   1 _bbstored _bbstored     705 Aug 29 23:30 o14.rfw
-rw-r--r--   1 _bbstored _bbstored   69496 Aug 30 23:28 o15.rfw
-rw-r--r--   1 _bbstored _bbstored    8326 Aug 21 18:12 o16.rfw
-rw-r--r--   1 _bbstored _bbstored    2289 Aug 21 18:12 o17.rfw
-rw-r--r--   1 _bbstored _bbstored    4929 Aug 21 18:12 o18.rfw
-rw-r--r--   1 _bbstored _bbstored   64932 Aug 21 18:12 o19.rfw
-rw-r--r--   1 _bbstored _bbstored    2176 Aug 21 18:12 o1a.rfw
-rw-r--r--   1 _bbstored _bbstored    2193 Aug 21 18:12 o1b.rfw
-rw-r--r--   1 _bbstored _bbstored  116810 Aug 21 18:12 o1c.rfw
-rw-r--r--   1 _bbstored _bbstored    1040 Aug 21 18:12 o1d.rfw
-rw-r--r--   1 _bbstored _bbstored    8135 Aug 21 18:12 o1e.rfw
-rw-r--r--   1 _bbstored _bbstored    1088 Aug 21 18:12 o1f.rfw
-rw-r--r--   1 _bbstored _bbstored   10313 Aug 21 18:12 o20.rfw
-rw-r--r--   1 _bbstored _bbstored    2225 Aug 21 18:12 o21.rfw
-rw-r--r--   1 _bbstored _bbstored    7222 Aug 21 18:12 o22.rfw
-rw-r--r--   1 _bbstored _bbstored    5011 Aug 21 18:12 o23.rfw
-rw-r--r--   1 _bbstored _bbstored    5233 Aug 21 18:12 o24.rfw
-rw-r--r--   1 _bbstored _bbstored    2961 Aug 21 18:12 o25.rfw
-rw-r--r--   1 _bbstored _bbstored    3682 Aug 21 18:12 o26.rfw
-rw-r--r--   1 _bbstored _bbstored   30332 Aug 21 18:12 o27.rfw
-rw-r--r--   1 _bbstored _bbstored    8069 Aug 21 18:12 o28.rfw
-rw-r--r--   1 _bbstored _bbstored     976 Aug 21 18:12 o29.rfw
-rw-r--r--   1 _bbstored _bbstored    1616 Aug 21 18:12 o2a.rfw
-rw-r--r--   1 _bbstored _bbstored    1696 Aug 21 18:12 o2b.rfw
-rw-r--r--   1 _bbstored _bbstored  223062 Aug 21 18:12 o2c.rfw
-rw-r--r--   1 _bbstored _bbstored    1104 Aug 21 18:12 o2d.rfw
-rw-r--r--   1 _bbstored _bbstored    3025 Aug 21 18:12 o2e.rfw
-rw-r--r--   1 _bbstored _bbstored    3073 Aug 21 18:12 o2f.rfw
-rw-r--r--   1 _bbstored _bbstored   32077 Aug 21 18:12 o30.rfw
-rw-r--r--   1 _bbstored _bbstored    3377 Aug 21 18:12 o31.rfw
-rw-r--r--   1 _bbstored _bbstored    1376 Aug 21 18:12 o32.rfw
-rw-r--r--   1 _bbstored _bbstored   11530 Aug 21 18:12 o33.rfw
-rw-r--r--   1 _bbstored _bbstored    1680 Aug 21 18:12 o34.rfw
-rw-r--r--   1 _bbstored _bbstored    1680 Aug 21 18:12 o35.rfw
-rw-r--r--   1 _bbstored _bbstored    1904 Aug 21 18:12 o36.rfw
-rw-r--r--   1 _bbstored _bbstored    1520 Aug 21 18:12 o37.rfw
-rw-r--r--   1 _bbstored _bbstored    3121 Aug 21 18:12 o38.rfw
-rw-r--r--   1 _bbstored _bbstored    1184 Aug 21 18:12 o39.rfw
-rw-r--r--   1 _bbstored _bbstored    2513 Aug 21 18:12 o3a.rfw
-rw-r--r--   1 _bbstored _bbstored    2321 Aug 21 18:12 o3b.rfw
-rw-r--r--   1 _bbstored _bbstored    1136 Aug 21 18:12 o3c.rfw
-rw-r--r--   1 _bbstored _bbstored    2849 Aug 21 18:12 o3d.rfw
-rw-r--r--   1 _bbstored _bbstored    2945 Aug 21 18:12 o3e.rfw
-rw-r--r--   1 _bbstored _bbstored    1616 Aug 21 18:12 o3f.rfw
-rw-r--r--   1 _bbstored _bbstored    1696 Aug 21 18:12 o40.rfw
-rw-r--r--   1 _bbstored _bbstored  493939 Aug 21 18:12 o41.rfw
-rw-r--r--   1 _bbstored _bbstored    2561 Aug 21 18:12 o42.rfw
-rw-r--r--   1 _bbstored _bbstored    2561 Aug 21 18:12 o43.rfw
-rw-r--r--   1 _bbstored _bbstored    2561 Aug 21 18:12 o44.rfw
-rw-r--r--   1 _bbstored _bbstored    2561 Aug 21 18:12 o45.rfw
-rw-r--r--   1 _bbstored _bbstored    1504 Aug 21 18:12 o46.rfw
-rw-r--r--   1 _bbstored _bbstored    1472 Aug 21 18:12 o47.rfw
-rw-r--r--   1 _bbstored _bbstored  379025 Aug 21 18:12 o48.rfw
-rw-r--r--   1 _bbstored _bbstored    1536 Aug 21 18:12 o49.rfw
-rw-r--r--   1 _bbstored _bbstored    1872 Aug 21 18:12 o4a.rfw
-rw-r--r--   1 _bbstored _bbstored  361129 Aug 21 18:12 o4b.rfw
-rw-r--r--   1 _bbstored _bbstored    2225 Aug 21 18:12 o4c.rfw
-rw-r--r--   1 _bbstored _bbstored     784 Aug 21 18:12 o4d.rfw
-rw-r--r--   1 _bbstored _bbstored    8212 Aug 21 18:12 o4e.rfw
-rw-r--r--   1 _bbstored _bbstored    1809 Aug 21 18:12 o4f.rfw
-rw-r--r--   1 _bbstored _bbstored    1888 Aug 21 18:12 o50.rfw
-rw-r--r--   1 _bbstored _bbstored    3137 Aug 21 18:12 o51.rfw
-rw-r--r--   1 _bbstored _bbstored    1712 Aug 21 18:12 o52.rfw
-rw-r--r--   1 _bbstored _bbstored    1776 Aug 21 18:12 o53.rfw
-rw-r--r--   1 _bbstored _bbstored    2673 Aug 21 18:12 o54.rfw
-rw-r--r--   1 _bbstored _bbstored    3698 Aug 21 18:12 o55.rfw
-rw-r--r--   1 _bbstored _bbstored  310136 Aug 21 18:12 o56.rfw
-rw-r--r--   1 _bbstored _bbstored    2529 Aug 21 18:12 o57.rfw
-rw-r--r--   1 _bbstored _bbstored    7669 Aug 21 18:12 o58.rfw
-rw-r--r--   1 _bbstored _bbstored    1552 Aug 21 18:12 o59.rfw
-rw-r--r--   1 _bbstored _bbstored    1856 Aug 21 18:12 o5a.rfw
-rw-r--r--   1 _bbstored _bbstored   14733 Aug 21 18:12 o5b.rfw
-rw-r--r--   1 _bbstored _bbstored    1632 Aug 21 18:12 o5c.rfw
-rw-r--r--   1 _bbstored _bbstored    1600 Aug 21 18:12 o5d.rfw
-rw-r--r--   1 _bbstored _bbstored    2449 Aug 21 18:12 o5e.rfw
-rw-r--r--   1 _bbstored _bbstored  965400 Aug 21 18:12 o5f.rfw
-rw-r--r--   1 _bbstored _bbstored    3377 Aug 21 18:12 o60.rfw
-rw-r--r--   1 _bbstored _bbstored    4883 Aug 21 18:12 o61.rfw
-rw-r--r--   1 _bbstored _bbstored     912 Aug 21 18:12 o62.rfw
-rw-r--r--   1 _bbstored _bbstored     992 Aug 21 18:12 o63.rfw
-rw-r--r--   1 _bbstored _bbstored     960 Aug 21 18:12 o64.rfw
-rw-r--r--   1 _bbstored _bbstored    1040 Aug 21 18:12 o65.rfw
-rw-r--r--   1 _bbstored _bbstored    4450 Aug 21 18:12 o66.rfw
-rw-r--r--   1 _bbstored _bbstored    2497 Aug 21 18:12 o67.rfw
-rw-r--r--   1 _bbstored _bbstored    5924 Aug 21 18:12 o68.rfw
-rw-r--r--   1 _bbstored _bbstored    1792 Aug 21 18:12 o69.rfw
-rw-r--r--   1 _bbstored _bbstored    1984 Aug 21 18:12 o6a.rfw
-rw-r--r--   1 _bbstored _bbstored    6467 Aug 21 18:12 o6b.rfw
-rw-r--r--   1 _bbstored _bbstored   19213 Aug 21 18:12 o6c.rfw
-rw-r--r--   1 _bbstored _bbstored    2369 Aug 21 18:12 o6d.rfw
-rw-r--r--   1 _bbstored _bbstored  287500 Aug 21 18:12 o6e.rfw
-rw-r--r--   1 _bbstored _bbstored    5634 Aug 21 18:12 o6f.rfw
-rw-r--r--   1 _bbstored _bbstored     752 Aug 21 18:12 o70.rfw
-rw-r--r--   1 _bbstored _bbstored    5842 Aug 21 18:12 o71.rfw
-rw-r--r--   1 _bbstored _bbstored   10039 Aug 21 18:12 o72.rfw
-rw-r--r--   1 _bbstored _bbstored    5572 Aug 21 18:12 o73.rfw
-rw-r--r--   1 _bbstored _bbstored    1488 Aug 21 18:12 o74.rfw
-rw-r--r--   1 _bbstored _bbstored    5010 Aug 21 18:12 o75.rfw
-rw-r--r--   1 _bbstored _bbstored    3890 Aug 21 18:12 o76.rfw
-rw-r--r--   1 _bbstored _bbstored     736 Aug 21 18:12 o77.rfw
-rw-r--r--   1 _bbstored _bbstored    6885 Aug 21 18:12 o78.rfw
-rw-r--r--   1 _bbstored _bbstored    4755 Aug 21 18:12 o79.rfw
-rw-r--r--   1 _bbstored _bbstored     912 Aug 21 18:12 o7a.rfw
-rw-r--r--   1 _bbstored _bbstored    1024 Aug 21 18:12 o7b.rfw
-rw-r--r--   1 _bbstored _bbstored     976 Aug 21 18:12 o7c.rfw
-rw-r--r--   1 _bbstored _bbstored     976 Aug 21 18:12 o7d.rfw
-rw-r--r--   1 _bbstored _bbstored     800 Aug 21 18:12 o7e.rfw
-rw-r--r--   1 _bbstored _bbstored     752 Aug 21 18:12 o7f.rfw
-rw-r--r--   1 _bbstored _bbstored    4723 Aug 21 18:12 o80.rfw
-rw-r--r--   1 _bbstored _bbstored     816 Aug 21 18:12 o81.rfw
-rw-r--r--   1 _bbstored _bbstored    1841 Aug 21 18:12 o82.rfw
-rw-r--r--   1 _bbstored _bbstored    2641 Aug 21 18:12 o83.rfw
-rw-r--r--   1 _bbstored _bbstored    1408 Aug 21 18:12 o84.rfw
-rw-r--r--   1 _bbstored _bbstored     768 Aug 21 18:12 o85.rfw
-rw-r--r--   1 _bbstored _bbstored    4163 Aug 21 18:12 o86.rfw
-rw-r--r--   1 _bbstored _bbstored     624 Aug 21 18:12 o87.rfw
-rw-r--r--   1 _bbstored _bbstored    1456 Aug 21 18:12 o88.rfw
-rw-r--r--   1 _bbstored _bbstored    1696 Aug 21 18:12 o89.rfw
-rw-r--r--   1 _bbstored _bbstored    1600 Aug 21 18:12 o8a.rfw
-rw-r--r--   1 _bbstored _bbstored    1376 Aug 21 18:12 o8b.rfw
-rw-r--r--   1 _bbstored _bbstored    6068 Aug 21 18:12 o8c.rfw
-rw-r--r--   1 _bbstored _bbstored    3073 Aug 21 18:12 o8d.rfw
-rw-r--r--   1 _bbstored _bbstored    1440 Aug 21 18:12 o8e.rfw
-rw-r--r--   1 _bbstored _bbstored     800 Aug 21 18:12 o8f.rfw
-rw-r--r--   1 _bbstored _bbstored     736 Aug 21 18:12 o90.rfw
-rw-r--r--   1 _bbstored _bbstored    5347 Aug 21 18:12 o91.rfw
-rw-r--r--   1 _bbstored _bbstored    2225 Aug 21 18:12 o92.rfw
-rw-r--r--   1 _bbstored _bbstored   51762 Aug 21 18:12 o93.rfw
-rw-r--r--   1 _bbstored _bbstored    1472 Aug 21 18:12 o94.rfw
-rw-r--r--   1 _bbstored _bbstored    1921 Aug 21 18:12 o95.rfw
-rw-r--r--   1 _bbstored _bbstored    1184 Aug 21 18:12 o96.rfw
-rw-r--r--   1 _bbstored _bbstored    2177 Aug 21 18:12 o97.rfw
-rw-r--r--   1 _bbstored _bbstored     768 Aug 21 18:12 o98.rfw
-rw-r--r--   1 _bbstored _bbstored     688 Aug 21 18:12 o99.rfw
-rw-r--r--   1 _bbstored _bbstored    3265 Aug 21 18:12 o9a.rfw
-rw-r--r--   1 _bbstored _bbstored    2657 Aug 21 18:12 o9b.rfw
-rw-r--r--   1 _bbstored _bbstored    2096 Aug 21 18:12 o9c.rfw
-rw-r--r--   1 _bbstored _bbstored    1600 Aug 21 18:12 o9d.rfw
-rw-r--r--   1 _bbstored _bbstored    3361 Aug 21 18:12 o9e.rfw
-rw-r--r--   1 _bbstored _bbstored    4145 Aug 21 18:12 o9f.rfw
-rw-r--r--   1 _bbstored _bbstored    7061 Aug 21 18:12 oa0.rfw
-rw-r--r--   1 _bbstored _bbstored    2000 Aug 21 18:12 oa1.rfw
-rw-r--r--   1 _bbstored _bbstored    3089 Aug 21 18:12 oa2.rfw
-rw-r--r--   1 _bbstored _bbstored  163875 Aug 21 18:12 oa3.rfw
-rw-r--r--   1 _bbstored _bbstored    3009 Aug 21 18:12 oa4.rfw
-rw-r--r--   1 _bbstored _bbstored    5219 Aug 21 18:12 oa5.rfw
-rw-r--r--   1 _bbstored _bbstored    6532 Aug 21 18:12 oa6.rfw
-rw-r--r--   1 _bbstored _bbstored   11929 Aug 21 18:12 oa7.rfw
-rw-r--r--   1 _bbstored _bbstored    1856 Aug 21 18:12 oa8.rfw
-rw-r--r--   1 _bbstored _bbstored    2032 Aug 21 18:12 oa9.rfw
-rw-r--r--   1 _bbstored _bbstored   12938 Aug 21 18:12 oaa.rfw
-rw-r--r--   1 _bbstored _bbstored    1984 Aug 21 18:12 oab.rfw
-rw-r--r--   1 _bbstored _bbstored    1888 Aug 21 18:12 oac.rfw
-rw-r--r--   1 _bbstored _bbstored    3922 Aug 21 18:12 oad.rfw
-rw-r--r--   1 _bbstored _bbstored    4355 Aug 21 18:12 oae.rfw
-rw-r--r--   1 _bbstored _bbstored   10869 Aug 21 18:12 oaf.rfw
-rw-r--r--   1 _bbstored _bbstored    1408 Aug 21 18:12 ob0.rfw
-rw-r--r--   1 _bbstored _bbstored    1280 Aug 21 18:12 ob1.rfw
-rw-r--r--   1 _bbstored _bbstored  163723 Aug 21 18:12 ob2.rfw
-rw-r--r--   1 _bbstored _bbstored     736 Aug 21 18:12 ob3.rfw
-rw-r--r--   1 _bbstored _bbstored    2080 Aug 21 18:12 ob4.rfw
-rw-r--r--   1 _bbstored _bbstored    2048 Aug 21 18:12 ob5.rfw
-rw-r--r--   1 _bbstored _bbstored    1392 Aug 21 18:12 ob6.rfw
-rw-r--r--   1 _bbstored _bbstored    4738 Aug 21 18:12 ob7.rfw
-rw-r--r--   1 _bbstored _bbstored    4594 Aug 21 18:12 ob8.rfw
-rw-r--r--   1 _bbstored _bbstored    9839 Aug 21 18:12 ob9.rfw
-rw-r--r--   1 _bbstored _bbstored    1504 Aug 21 18:12 oba.rfw
-rw-r--r--   1 _bbstored _bbstored    3097 Aug 21 18:12 obb.rfw
-rw-r--r--   1 _bbstored _bbstored    6891 Aug 21 18:12 obc.rfw
-rw-r--r--   1 _bbstored _bbstored    1744 Aug 21 18:12 obd.rfw
-rw-r--r--   1 _bbstored _bbstored    1024 Aug 21 18:12 obe.rfw
-rw-r--r--   1 _bbstored _bbstored    1936 Aug 21 18:12 obf.rfw
-rw-r--r--   1 _bbstored _bbstored    3745 Aug 21 18:12 oc0.rfw
-rw-r--r--   1 _bbstored _bbstored    1184 Aug 21 18:12 oc1.rfw
-rw-r--r--   1 _bbstored _bbstored    1056 Aug 21 18:12 oc2.rfw
-rw-r--r--   1 _bbstored _bbstored    2561 Aug 21 18:12 oc3.rfw
-rw-r--r--   1 _bbstored _bbstored    1056 Aug 21 18:12 oc4.rfw
-rw-r--r--   1 _bbstored _bbstored    2240 Aug 21 18:12 oc5.rfw
-rw-r--r--   1 _bbstored _bbstored     976 Aug 21 18:12 oc6.rfw
-rw-r--r--   1 _bbstored _bbstored    6052 Aug 21 18:12 oc7.rfw
-rw-r--r--   1 _bbstored _bbstored    3121 Aug 21 18:12 oc8.rfw
-rw-r--r--   1 _bbstored _bbstored    1472 Aug 21 18:12 oc9.rfw
-rw-r--r--   1 _bbstored _bbstored  252279 Aug 21 18:12 oca.rfw
-rw-r--r--   1 _bbstored _bbstored 3332087 Aug 21 18:12 ocb.rfw
-rw-r--r--   1 _bbstored _bbstored    4354 Aug 21 18:12 occ.rfw
-rw-r--r--   1 _bbstored _bbstored    4834 Aug 21 18:12 ocd.rfw
-rw-r--r--   1 _bbstored _bbstored    1984 Aug 21 18:12 oce.rfw
-rw-r--r--   1 _bbstored _bbstored    1456 Aug 21 18:12 ocf.rfw
-rw-r--r--   1 _bbstored _bbstored    4434 Aug 21 18:12 od0.rfw
-rw-r--r--   1 _bbstored _bbstored    4770 Aug 21 18:12 od1.rfw
-rw-r--r--   1 _bbstored _bbstored    6052 Aug 21 18:12 od2.rfw
-rw-r--r--   1 _bbstored _bbstored    1632 Aug 21 18:12 od3.rfw
-rw-r--r--   1 _bbstored _bbstored    2577 Aug 21 18:12 od4.rfw
-rw-r--r--   1 _bbstored _bbstored    5346 Aug 21 18:12 od5.rfw
-rw-r--r--   1 _bbstored _bbstored    1728 Aug 21 18:12 od6.rfw
-rw-r--r--   1 _bbstored _bbstored    4738 Aug 21 18:12 od7.rfw
-rw-r--r--   1 _bbstored _bbstored   61735 Aug 21 18:12 od8.rfw
-rw-r--r--   1 _bbstored _bbstored    2929 Aug 21 18:12 od9.rfw
-rw-r--r--   1 _bbstored _bbstored    1856 Aug 21 18:12 oda.rfw
-rw-r--r--   1 _bbstored _bbstored    3217 Aug 21 18:12 odb.rfw
-rw-r--r--   1 _bbstored _bbstored  396489 Aug 21 18:12 odc.rfw
-rw-r--r--   1 _bbstored _bbstored    4642 Aug 21 18:12 odd.rfw
-rw-r--r--   1 _bbstored _bbstored    2144 Aug 21 18:12 ode.rfw
-rw-r--r--   1 _bbstored _bbstored     960 Aug 21 18:12 odf.rfw
-rw-r--r--   1 _bbstored _bbstored    1120 Aug 21 18:12 oe0.rfw
-rw-r--r--   1 _bbstored _bbstored    2416 Aug 21 18:12 oe1.rfw
-rw-r--r--   1 _bbstored _bbstored    1040 Aug 21 18:12 oe2.rfw
-rw-r--r--   1 _bbstored _bbstored     992 Aug 21 18:12 oe3.rfw
-rw-r--r--   1 _bbstored _bbstored    8502 Aug 21 18:12 oe4.rfw
-rw-r--r--   1 _bbstored _bbstored    2176 Aug 21 18:12 oe5.rfw
-rw-r--r--   1 _bbstored _bbstored    1008 Aug 21 18:12 oe6.rfw
-rw-r--r--   1 _bbstored _bbstored    3073 Aug 21 18:12 oe7.rfw
-rw-r--r--   1 _bbstored _bbstored    1008 Aug 21 18:12 oe8.rfw
-rw-r--r--   1 _bbstored _bbstored    1776 Aug 21 18:12 oe9.rfw
-rw-r--r--   1 _bbstored _bbstored    6900 Aug 21 18:12 oea.rfw
-rw-r--r--   1 _bbstored _bbstored 1335769 Aug 21 18:12 oeb.rfw
-rw-r--r--   1 _bbstored _bbstored    1488 Aug 21 18:12 oec.rfw
-rw-r--r--   1 _bbstored _bbstored    6931 Aug 21 18:12 oed.rfw
-rw-r--r--   1 _bbstored _bbstored    6373 Aug 21 18:12 oee.rfw
-rw-r--r--   1 _bbstored _bbstored 2638262 Aug 21 18:12 oef.rfw
-rw-r--r--   1 _bbstored _bbstored    2001 Aug 21 18:12 of0.rfw
-rw-r--r--   1 _bbstored _bbstored    3826 Aug 21 18:12 of1.rfw
-rw-r--r--   1 _bbstored _bbstored  262788 Aug 21 18:12 of2.rfw
-rw-r--r--   1 _bbstored _bbstored    2256 Aug 21 18:12 of3.rfw
-rw-r--r--   1 _bbstored _bbstored  203794 Aug 21 18:12 of4.rfw
-rw-r--r--   1 _bbstored _bbstored  785034 Aug 21 18:12 of5.rfw
-rw-r--r--   1 _bbstored _bbstored  173881 Aug 21 18:12 of6.rfw
-rw-r--r--   1 _bbstored _bbstored    1600 Aug 21 18:12 of7.rfw
-rw-r--r--   1 _bbstored _bbstored    2753 Aug 21 18:12 of8.rfw
-rw-r--r--   1 _bbstored _bbstored   14749 Aug 21 18:12 of9.rfw
-rw-r--r--   1 _bbstored _bbstored   10760 Aug 21 18:12 ofa.rfw
-rw-r--r--   1 _bbstored _bbstored   10119 Aug 21 18:12 ofb.rfw
-rw-r--r--   1 _bbstored _bbstored    1696 Aug 21 18:12 ofc.rfw
-rw-r--r--   1 _bbstored _bbstored    3201 Aug 21 18:12 ofd.rfw
-rw-r--r--   1 _bbstored _bbstored    4242 Aug 21 18:12 ofe.rfw
-rw-r--r--   1 _bbstored _bbstored    2609 Aug 21 18:12 off.rfw
-rw-------   1 _bbstored _bbstored       0 Aug 31 14:28 write.lock

Regards

Matt Brown




From boxbackup at fluffy.co.uk  Fri Aug 31 17:45:46 2007
From: boxbackup at fluffy.co.uk (E.W. Peter Jalajas)
Date: Fri, 31 Aug 2007 09:45:46 -0700 (PDT)
Subject: [Box Backup] Failed to setup location
In-Reply-To: <Pine.LNX.4.64.0708302054520.12581@top.qwarx.com>
Message-ID: <445492.54450.qm@web60623.mail.yahoo.com>

Hi Chris, 

Did you get this one:
http://lists.warhead.org.uk/pipermail/boxbackup/2007-August/003731.html

Thanks,
Pete

--- Chris Wilson <chris at qwirx.com> wrote:

> Do you have any more feedback for me about this problem? I'd really like 
> to get it fixed before releasing 0.11.



From boxbackup at fluffy.co.uk  Fri Aug 31 22:11:29 2007
From: boxbackup at fluffy.co.uk (E.W. Peter Jalajas)
Date: Fri, 31 Aug 2007 14:11:29 -0700 (PDT)
Subject: [Box Backup] Failed to setup location
Message-ID: <502929.41839.qm@web60618.mail.yahoo.com>

Hi Chris,

We just discovered the same problem on another client server (with Box Backup 1781 installed on
Windows Server 2003; we can't backup from a share, with the same error messages re "No such file
or directory").  As we flailed around on it, we got these messages in the Event Viewer (I may have
pasted the first two events in the wrong order below):

Event Type:	Warning
Event Source:	Box Backup (bbackupd)
Event Category:	None
Event ID:	1
Date:		8/31/2007
Time:		2:09:21 PM
User:		N/A
Computer:	SERVER2000
Description:
Failed to stat location: Z:\Admin: Permission denied 

Event Type:	Warning
Event Source:	Box Backup (bbackupd)
Event Category:	None
Event ID:	1
Date:		8/31/2007
Time:		2:09:21 PM
User:		N/A
Computer:	SERVER2000
Description:
Failed to open 'Z:\Admin': The network name cannot be found. (67) 

Event Type:	Warning
Event Source:	Box Backup (bbackupd)
Event Category:	None
Event ID:	1
Date:		8/31/2007
Time:		2:09:21 PM
User:		N/A
Computer:	SERVER2000
Description:
Exception thrown: CommonException(OSFileError) at BackupDaemon.cpp(1703) 

At another customer, we have a version 784 client backing up from a remote share no problem, so
something broke sometime after 784.   

We tried changing the remote share Path line in bbackupd.conf to lowercase, we started getting
some weird network errors (109, 67).  Are you handling case correctly when parsing the Path line? 


(We of course can write to the Z: drive and open it in Windows Explorer.  Box Backup is running as
a Domain Admin with Full Control on the share.)

(We tried various combinations of Box Backup on Windows 2000/Windows 2003 with the shared drive on
those same OSs, but didn't learn anything.  The one that is working happens to have version 784
running on Win2K backing up from a share on Win2K3.)

Maybe this is a DNS/NetBIOS problem?  We stumbled across this page:
http://support.microsoft.com/kb/875441
See the section on "Method 1: Increase the value of the LmhostsTimeout registry entry"
If you think it might help, let me know and we can try to experiment with that.

Sorry, but I think that this is a showstopper for 0.11 (if it's not simply operator error on my
part).

Thanks, Chris.
Pete

--- "E.W. Peter Jalajas" <pjalajas at tebucosafe.com> wrote:

> Hi Chris, 
> 
> Did you get this one:
> http://lists.warhead.org.uk/pipermail/boxbackup/2007-August/003731.html
> 
> Thanks,
> Pete



From boxbackup at fluffy.co.uk  Fri Aug 31 23:14:59 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Fri, 31 Aug 2007 23:14:59 +0100 (BST)
Subject: [Box Backup] BadBackupStoreFile
In-Reply-To: <1188390812.10391.39.camel@localhost.localdomain>
References: <1187091035.5344.4.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708251705330.20516@top.qwarx.com>
 <1188204879.5254.12.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708290108130.20516@top.qwarx.com>
 <1188373549.5379.27.camel@localhost.localdomain>
 <Pine.LNX.4.64.0708290944160.20516@top.qwarx.com>
 <1188390812.10391.39.camel@localhost.localdomain>
Message-ID: <Pine.LNX.4.64.0708312300570.12581@top.qwarx.com>

Hi Johann,

Many thanks for testing and reporting these problems!

On Wed, 29 Aug 2007, Johann Glaser wrote:

>>> 759356296 ./23/06/o12.rfw
>>> 759552073 ./9e/06/o8f.rfw
>>> 759633897 ./1a/07/obc.rfw
>>> 1529736909 ./ba/o55.rfw
>>> 1539937214 ./69/01/o05.rfw
>>> 2744679666 ./16/01/o5f.rfw
>>> 5133609317 ./f2/o44.rfw
>>
>> Do you have any errors restoring or comparing the other large file?
>
> Yes, there are errors:
> query > compare -E . .
> Local file './__db.002/__db.002' has different contents to store file './__db.002'.
> Local file './__db.003/__db.003' has different contents to store file './__db.003'.
> Local file './__db.004/__db.004' has different contents to store file './__db.004'.
> Local file './__db.005/__db.005' has different contents to store file './__db.005'.
> Local file './__db.006/__db.006' has different contents to store file './__db.006'.
> ERROR: (4/48) during file fetch and comparsion for './strings'
> ERROR: (7/41) during file fetch and comparsion for './transactions'
> ERROR: (7/41) during file fetch and comparsion for './uuids'

Can i just check this? You later said:

> Now (after the above error 4/48 not when starting bbackupquery it prints
> 
> Box Backup Query Tool v0.10, (c) Ben Summers and contributors 2003-2006
> Using configuration file /etc/boxbackup/bbackupd.conf
> Connecting to store...
> Handshake with store...
> Login to store...
> 
> and hangs.

It prints the banner in the MIDDLE of your bbackupquery session, AFTER you 
run the compare command? And then hangs? And the server is hung as well?
That's pretty messed up! I can't even see how the code could do that.

What platform are the server and client, what version of GCC do you use, 
where did you get the packages (or did you build them yourself) and do you 
have any reason to suspect hardward problems such as bad RAM on either 
server or client?

> PID 28114 was still running with nearly 100% CPU when already at the 
> bbackupquery prompt. Typing "ls" just hang. I had to kill it, just 
> restarting the boxbackup-server didn't stop this task.

Could you possibly attach a debugger to the hung bbstored process and get 
a backtrace for me? ("bt" command in gdb).

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup at fluffy.co.uk  Fri Aug 31 23:21:28 2007
From: boxbackup at fluffy.co.uk (Chris Wilson)
Date: Fri, 31 Aug 2007 23:21:28 +0100 (BST)
Subject: [Box Backup] BadBackupStoreFile
In-Reply-To: <934A2180-C674-4609-B824-7CEF58396C3B@fluffy.co.uk>
References: <934A2180-C674-4609-B824-7CEF58396C3B@fluffy.co.uk>
Message-ID: <Pine.LNX.4.64.0708312320560.12581@top.qwarx.com>

Hi Ben,

On Fri, 31 Aug 2007, Ben Summers wrote:

>> > Another feature request: The backup server should (additionally) store
>> > checksums across large blocks of files, e.g. 1MB blocks. Then only these
>> > checksums need to be read from disk instead of the whole backup file.
>> 
>> I have a feeling that we already do, but I'm not 100% sure. Ben, do you
>> know?
>
> Checksums are per-block, encrypted. The block size depends on the size of the 
> file. You can do a compare using the checksums only using the -q option, and 
> only the checksums are read from disc and transferred over the network.

Do you know if we do that for diffing as well, or only for compare?

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |