From cableninja at cableninja.net Sat Apr 6 13:03:56 2013 From: cableninja at cableninja.net (Chris Walker) Date: Sat, 06 Apr 2013 05:03:56 -0700 Subject: [Box Backup] RSA_padding_check_PKCS1_type_1:block type is not 01 Message-ID: <51600F2C.7070208@cableninja.net> Hi, I just setup boxbackup for the first time. After a few troubles here and there (debian wheezy's packages are not quite 'up to par' as far as pre-configuration goes, theres things missing, things get put in places that are referenced differently (bbstored/clients vs /clients). Anyway, I got the server up and running, I'm now having problems getting clients functioning. 1) I did bbackupd-config /etc/boxbackup lazy 0 backup01.cableninja.net 2) I got the csr it spit out and took it to the backup server 3) used bbstored-certs /etc/boxbackup sign /etc/boxbackup/bbstored/clients/0-csr.pem 4) I took the /etc/boxbackup/bbstored/roots/serverCA.pem and /etc/boxbackup/bbstored/clients/0-cert.pem to the server being backed up. 5) verified all paths were correct and attempted to start bbackupd. Initially I got no output until I started using -V -D (and specifying the config path). Any help would be greatly appreciated OpenVZ System on Centos 6.4 2.6.32-042stab075.2, Container Debian Wheezy - Kernel 3.2.0-4 - BoxBackup 0.11rc8+2837 root at db01:/# bbackupd -V -D /etc/boxbackup/bbackupd.conf NOTICE: Starting daemon, version: 0.11rc8+2837 NOTICE: Using configuration file: /etc/boxbackup/bbackupd.conf TRACE: BackupDaemon::NotifySysadmin() called, event = backup-start INFO: About to notify administrator about event backup-start, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-start "/etc/boxbackup/bbackupd.conf"' NOTICE: Beginning scan of local files TRACE: Set maximum diffing time to 120 seconds TRACE: Set keep-alive time to 60 seconds TRACE: timer: next event: KeepAliveTime expires in 59.999994 seconds TRACE: timer: next event: KeepAliveTime expires in 59.999968 seconds TRACE: timer: next event: KeepAliveTime expires in 59.999938 seconds TRACE: timer: next event: KeepAliveTime expires in 59.999910 seconds INFO: Opening connection to server 'backup01.cableninja.net'... ERROR: SSL error while connecting: error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 ERROR: SSL error while connecting: error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed ERROR: SSL error while connecting: error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib ERROR: SSL error while connecting: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed TRACE: Obtained 10 stack frames. TRACE: Stack frame 0: bbackupd(DumpStackBacktrace()+0x23) [0x4f7c83] TRACE: Stack frame 1: bbackupd(SocketStreamTLS::Handshake(TLSContext const&, bool)+0xd8c) [0x4b731c] TRACE: Stack frame 2: bbackupd(SocketStreamTLS::Open(TLSContext const&, Socket::Type, std::string const&, int)+0x2e) [0x4b7a8e] TRACE: Stack frame 3: bbackupd(BackupClientContext::GetConnection()+0x32b) [0x42927b] TRACE: Stack frame 4: bbackupd(BackupDaemon::SetupLocations(BackupClientContext&, Configuration const&)+0x89) [0x43deb9] TRACE: Stack frame 5: bbackupd(BackupDaemon::RunSyncNow()+0xe2d) [0x440a3d] TRACE: Stack frame 6: bbackupd(BackupDaemon::RunSyncNowWithExceptionHandling()+0x35) [0x440ea5] TRACE: Stack frame 7: bbackupd(BackupDaemon::Run2()+0x266) [0x441756] TRACE: Stack frame 8: bbackupd(BackupDaemon::Run()+0x108) [0x441948] TRACE: Stack frame 9: bbackupd(Daemon::Main(std::string const&)+0x2252) [0x48b2a2] WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(212) TRACE: timer: no more events, going to sleep. TRACE: BackupDaemon::NotifySysadmin() called, event = backup-error INFO: About to notify administrator about event backup-error, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-error "/etc/boxbackup/bbackupd.conf"' /etc/boxbackup/bbackupd/NotifySysadmin.sh: line 56: sendmail:: command not found WARNING: Notify script returned error code: 32512 (/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-error "/etc/boxbackup/bbackupd.conf") ERROR: Exception caught (Connection TLSHandshakeFailed 7/30), reset state and waiting to retry... From james at netinertia.co.uk Sat Apr 6 15:25:12 2013 From: james at netinertia.co.uk (James O'Gorman) Date: Sat, 6 Apr 2013 15:25:12 +0100 Subject: [Box Backup] RSA_padding_check_PKCS1_type_1:block type is not 01 In-Reply-To: <51600F2C.7070208@cableninja.net> References: <51600F2C.7070208@cableninja.net> Message-ID: <20130406142512.GO1672@netinertia.co.uk> Hi Chris, Just a quick sanity check - is the time correct on both machines? SSL can throw some weird errors if the time is off by more than about 2 minutes. On Sat, Apr 06, 2013 at 05:03:56AM -0700, Chris Walker wrote: > > TRACE: BackupDaemon::NotifySysadmin() called, event = backup-error > INFO: About to notify administrator about event backup-error, running > script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-error > "/etc/boxbackup/bbackupd.conf"' > /etc/boxbackup/bbackupd/NotifySysadmin.sh: line 56: sendmail:: command > not found Also looks like you don't have any mail daemon installed, so you won't get email notifications from the system. James From cwalker at cableninja.net Sat Apr 6 15:32:27 2013 From: cwalker at cableninja.net (Chris Walker) Date: Sat, 06 Apr 2013 07:32:27 -0700 Subject: [Box Backup] RSA_padding_check_PKCS1_type_1:block type is not 01 In-Reply-To: <20130406142512.GO1672@netinertia.co.uk> References: <51600F2C.7070208@cableninja.net> <20130406142512.GO1672@netinertia.co.uk> Message-ID: <516031FB.9010609@cableninja.net> Hi James, The time is indeed correct. I saw a thread regarding an SSL problem earlier in the mailing list, did a check for that. I just ran the date command through tmux at the same time on both servers: root at backup02:/# date Sat Apr 6 08:28:29 MDT 2013 root at db02:/# date Sat Apr 6 08:28:29 MDT 2013 I'm aware of the sendmail bit, these are fresh "very light" servers, boxbackup was the first thing to be installed. Thanks, On 04/06/2013 07:25 AM, James O'Gorman wrote: > Hi Chris, > > Just a quick sanity check - is the time correct on both machines? SSL > can throw some weird errors if the time is off by more than about 2 > minutes. > > On Sat, Apr 06, 2013 at 05:03:56AM -0700, Chris Walker wrote: >> TRACE: BackupDaemon::NotifySysadmin() called, event = backup-error >> INFO: About to notify administrator about event backup-error, running >> script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-error >> "/etc/boxbackup/bbackupd.conf"' >> /etc/boxbackup/bbackupd/NotifySysadmin.sh: line 56: sendmail:: command >> not found > Also looks like you don't have any mail daemon installed, so you won't > get email notifications from the system. > > James > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup From chris at qwirx.com Sun Apr 7 12:37:55 2013 From: chris at qwirx.com (Chris Wilson) Date: Sun, 7 Apr 2013 12:37:55 +0100 (BST) Subject: [Box Backup] RSA_padding_check_PKCS1_type_1:block type is not 01 In-Reply-To: <51600F2C.7070208@cableninja.net> References: <51600F2C.7070208@cableninja.net> Message-ID: Hi Chris, I'm sorry you had problems installing the Debian Wheezy packages. I wish I could help in getting them fixed up. On Sat, 6 Apr 2013, Chris Walker wrote: > Anyway, I got the server up and running, I'm now having problems getting > clients functioning. > > 1) I did bbackupd-config /etc/boxbackup lazy 0 backup01.cableninja.net > 2) I got the csr it spit out and took it to the backup server > 3) used bbstored-certs /etc/boxbackup sign > /etc/boxbackup/bbstored/clients/0-csr.pem > 4) I took the /etc/boxbackup/bbstored/roots/serverCA.pem and > /etc/boxbackup/bbstored/clients/0-cert.pem to the server being backed up. > 5) verified all paths were correct and attempted to start bbackupd. > > Initially I got no output until I started using -V -D (and specifying > the config path). > > Any help would be greatly appreciated > > OpenVZ System on Centos 6.4 2.6.32-042stab075.2, Container Debian Wheezy > - Kernel 3.2.0-4 - BoxBackup 0.11rc8+2837 > > root at db01:/# bbackupd -V -D /etc/boxbackup/bbackupd.conf > NOTICE: Starting daemon, version: 0.11rc8+2837 ... > INFO: Opening connection to server 'backup01.cableninja.net'... > ERROR: SSL error while connecting: error:0407006A:rsa > routines:RSA_padding_check_PKCS1_type_1:block type is not 01 > ERROR: SSL error while connecting: error:04067072:rsa > routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed > ERROR: SSL error while connecting: error:0D0C5006:asn1 encoding > routines:ASN1_item_verify:EVP lib > ERROR: SSL error while connecting: error:14090086:SSL > routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed I've never seen this error before, and I can't guess very accurately what's causing it. It looks almost like a failure to speak the (same) SSL protocol between the two sides. I was able to connect to your boxbackup server over the Internet using openssl s_client and it did negotiate SSL properly, so I think the server is working. I'm not sure I understand your setup correctly. Are both server and client OpenVZ containers running on Centos 6.4 hosts? Is the host 32-bit or 64-bit? Are the clients both Debian Wheezy? 32-bit or 64-bit? Did you get the Box Backup packages from Debian in both cases? Do you have any other Box Backup servers that you could try connecting your client to, that are known to be working? Could you try this on your client, and let me know what the output is: openssl s_client -connect backup01.cableninja.net:2201 You could try the same command with -cert, -key and -CAfile pointing to your bbackupd certificate, private key and trusted CAs PEM files, and it should stay connected to the server (and not be disconnected due to not providing a valid certificate to the server). Unless something obvious comes up, I think I'm going to have to reproduce the problem. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From cableninja at cableninja.net Mon Apr 8 06:04:27 2013 From: cableninja at cableninja.net (cableninja at cableninja.net) Date: Sun, 07 Apr 2013 22:04:27 -0700 Subject: [Box Backup] RSA_padding_check_PKCS1_type_1:block type is not 01 In-Reply-To: References: <51600F2C.7070208@cableninja.net> Message-ID: <51624FDB.6060504@cableninja.net> Hi Chris, > I'm sorry you had problems installing the Debian Wheezy packages. I wish I could help in getting them fixed up. I'll setup another debian wheezy server and document everything that I had to do in order to get it working so that you guys can work that out. > I'm not sure I understand your setup correctly. Are both server and client OpenVZ containers running on Centos 6.4 hosts? Is the host 32-bit or 64-bit? > Are the clients both Debian Wheezy? 32-bit or 64-bit? Did you get the Box Backup packages from Debian in both cases? I did indeed get the packages from the debian repositories (maybe this is the problem? I've seen broken deb packages but the original releases were fine). To better explain my setup: 2 CentOS 6.4 x64 OpenVZ Host nodes, Both servers hardware contains 2 network ports - one public (eth0), one private (eth1) - eth1 is directly connected to the other host node 2 BoxBackup-bbstored servers running as debian wheezy (x64) containers, one on each host node 2 BoxBackup-bbackupd servers running as debian wheezy (x64) containers, one on each host node The bbstored servers backup the other containers on their respective node. I setup both bbstored servers at the same time using tmux + synchronize-panes. I did the same for the bbackupd clients >Could you try this on your client, and let me know what the output is: > > openssl s_client -connect backup01.cableninja.net:2201 Below is the output of this command # openssl s_client -connect backup01.cableninja.net:2201 > test01.out depth=0 CN = backup02.cableninja.net verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 CN = backup02.cableninja.net verify error:num=27:certificate not trusted verify return:1 depth=0 CN = backup02.cableninja.net verify error:num=21:unable to verify the first certificate verify return:1 140255127504552:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1256:SSL alert number 40 140255127504552:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: # cat test01.out CONNECTED(00000003) --- Certificate chain 0 s:/CN=backup02.cableninja.net i:/CN=Backup system server root --- Server certificate -----BEGIN CERTIFICATE----- MIICujCCAaICAQIwDQYJKoZIhvcNAQEFBQAwJDEiMCAGA1UEAwwZQmFja3VwIHN5 c3RlbSBzZXJ2ZXIgcm9vdDAeFw0xMzA0MDYwOTAyNDRaFw0yNjEyMTQwOTAyNDRa MCIxIDAeBgNVBAMMF2JhY2t1cDAyLmNhYmxlbmluamEubmV0MIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEA873ol7cKT7KZ4lHqV2OOqjv/XTrEMIUO87Cf 3jx2G2ekFe0KSmhSLvCONCA/LqT21nHiHGotiBQcZq1r9KNM8Zv3vSBXKDMIr0EW vVAzba0/YQLTyXbYCG7f4+y7iEszGQsYLjm5mTkPXbEX7iPNHi5fgcnfiY7D5YtL wpdm3beirLR/l4PmOIJF8EeFxPUmLg/E9ZdbKLAAYEtPT4x2Y6FVUArkbZF+iag1 dIHDUsE3KqRSsWlL5N32GNYJmF1fW9S99/vFxeJMdIRqe//XHA4gzxqd5paf+uCG XV8JkKmgwPzpSGlAtlVIaS2X87Oqr63d7uMJ1mulRVp6OuO3SwIDAQABMA0GCSqGSIb3DQEBBQUAA4IBAQCmV2FOLkOCjv8yZ1Uj1aFu5hu9sX/LnEP0mSCZcbsCjUpv V9L1RQVPaH7K6kxWrF0rlWsktSx9f3AIfQYQyqRi/iuTe08MHYgUFfFbdZ2rGxmR 7YNsSBAijmnG+0kSa/AlgatPrAUMn71LyVacEJ3wCsKSu4fi8rgw8k5fFLf0JB47 ESxmKe2qH6pSlwvoyJO2vDd8o7mod6AtEsFN2s8tS1AlKGSli3RmLVYqyz/C5wDb XM0XlC9WSYUMnctpEktRZbDd6Hpb3dfBCZHXhZWBE7SaYqWPKCic8zuDLH235xCw irDywQDGUlZnjTqDUdmzuYPSsjkY01e7cTUJOBzm -----END CERTIFICATE----- subject=/CN=backup02.cableninja.net issuer=/CN=Backup system server root --- No client certificate CA names sent --- SSL handshake has read 806 bytes and written 338 bytes --- New, TLSv1/SSLv3, Cipher is AES256-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: zlib compression Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher : AES256-SHA Session-ID: Session-ID-ctx: Master-Key: 663E593F61624670450E96E0E04645E60945DACAB503EE99220B27FE50F0CAD36979EBAF96600D8DF164273160F703CC Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None Compression: 1 (zlib compression) Start Time: 1365390077 Timeout : 300 (sec) Verify return code: 21 (unable to verify the first certificate) Below is the output of the command with the cert/key/CAFile options # openssl s_client -cert /etc/boxbackup/bbackupd/0-cert.pem -key /etc/boxbackup/bbackupd/0-key.pem -CAfile /etc/boxbackup/bbackupd/serverCA.pem -connect backup01.cableninja.net:2201 > test01.out depth=1 CN = Backup system server root verify return:1 depth=0 CN = backup02.cableninja.net verify error:num=7:certificate signature failure verify return:1 depth=0 CN = backup02.cableninja.net verify return:1 140498802353832:error:1409441B:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error:s3_pkt.c:1256:SSL alert number 51 140498802353832:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: # cat test02.out CONNECTED(00000003) --- Certificate chain 0 s:/CN=backup02.cableninja.net i:/CN=Backup system server root --- Server certificate -----BEGIN CERTIFICATE----- MIICujCCAaICAQIwDQYJKoZIhvcNAQEFBQAwJDEiMCAGA1UEAwwZQmFja3VwIHN5 c3RlbSBzZXJ2ZXIgcm9vdDAeFw0xMzA0MDYwOTAyNDRaFw0yNjEyMTQwOTAyNDRa MCIxIDAeBgNVBAMMF2JhY2t1cDAyLmNhYmxlbmluamEubmV0MIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEA873ol7cKT7KZ4lHqV2OOqjv/XTrEMIUO87Cf 3jx2G2ekFe0KSmhSLvCONCA/LqT21nHiHGotiBQcZq1r9KNM8Zv3vSBXKDMIr0EW vVAzba0/YQLTyXbYCG7f4+y7iEszGQsYLjm5mTkPXbEX7iPNHi5fgcnfiY7D5YtL wpdm3beirLR/l4PmOIJF8EeFxPUmLg/E9ZdbKLAAYEtPT4x2Y6FVUArkbZF+iag1 dIHDUsE3KqRSsWlL5N32GNYJmF1fW9S99/vFxeJMdIRqe//XHA4gzxqd5paf+uCG XV8JkKmgwPzpSGlAtlVIaS2X87Oqr63d7uMJ1mulRVp6OuO3SwIDAQABMA0GCSqG SIb3DQEBBQUAA4IBAQCmV2FOLkOCjv8yZ1Uj1aFu5hu9sX/LnEP0mSCZcbsCjUpv V9L1RQVPaH7K6kxWrF0rlWsktSx9f3AIfQYQyqRi/iuTe08MHYgUFfFbdZ2rGxmR 7YNsSBAijmnG+0kSa/AlgatPrAUMn71LyVacEJ3wCsKSu4fi8rgw8k5fFLf0JB47 ESxmKe2qH6pSlwvoyJO2vDd8o7mod6AtEsFN2s8tS1AlKGSli3RmLVYqyz/C5wDb XM0XlC9WSYUMnctpEktRZbDd6Hpb3dfBCZHXhZWBE7SaYqWPKCic8zuDLH235xCw irDywQDGUlZnjTqDUdmzuYPSsjkY01e7cTUJOBzm -----END CERTIFICATE----- subject=/CN=backup02.cableninja.net issuer=/CN=Backup system server root --- No client certificate CA names sent --- SSL handshake has read 806 bytes and written 1295 bytes --- New, TLSv1/SSLv3, Cipher is AES256-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: zlib compression Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher : AES256-SHA Session-ID: Session-ID-ctx: Master-Key: 71FB504CCB830E4E584CC3DE1409D81F97CF7220972E6578EACDA900476EB33C0FFDE33D07475E845A2C8FB87EF5666C Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None Compression: 1 (zlib compression) Start Time: 1365390419 Timeout : 300 (sec) Verify return code: 7 (certificate signature failure) --- I'm not too familiar with openssl, but it appears to me that theres something wrong with my serverCA? or my certificates in general? I'm beginning to think that this stems from 'confusion' in paths and locations between the scripts. Example: bbstored-certs init makes folders at /etc/boxbackup, but bbstored-config assumes /etc/boxbackup/bbstored/ (or visa versa). I've just nuked all of my configs and certs, I'm currently trying again from scratch. I'll update when I have tried all possibilities and what we've got. > Unless something obvious comes up, I think I'm going to have to reproduce the problem. I can provide test containers for you, even give you access to these existing ones, if you dont have an existing setup similar to what I have, to test on. Let me know if you'd like me to do so. Thanks for your time/help. - Chris On 04/07/2013 04:37 AM, Chris Wilson wrote: > Hi Chris, > > I'm sorry you had problems installing the Debian Wheezy packages. I > wish I could help in getting them fixed up. > > On Sat, 6 Apr 2013, Chris Walker wrote: > >> Anyway, I got the server up and running, I'm now having problems getting >> clients functioning. >> >> 1) I did bbackupd-config /etc/boxbackup lazy 0 backup01.cableninja.net >> 2) I got the csr it spit out and took it to the backup server >> 3) used bbstored-certs /etc/boxbackup sign >> /etc/boxbackup/bbstored/clients/0-csr.pem >> 4) I took the /etc/boxbackup/bbstored/roots/serverCA.pem and >> /etc/boxbackup/bbstored/clients/0-cert.pem to the server being backed >> up. >> 5) verified all paths were correct and attempted to start bbackupd. >> >> Initially I got no output until I started using -V -D (and specifying >> the config path). >> >> Any help would be greatly appreciated >> >> OpenVZ System on Centos 6.4 2.6.32-042stab075.2, Container Debian Wheezy >> - Kernel 3.2.0-4 - BoxBackup 0.11rc8+2837 >> >> root at db01:/# bbackupd -V -D /etc/boxbackup/bbackupd.conf >> NOTICE: Starting daemon, version: 0.11rc8+2837 > ... >> INFO: Opening connection to server 'backup01.cableninja.net'... >> ERROR: SSL error while connecting: error:0407006A:rsa >> routines:RSA_padding_check_PKCS1_type_1:block type is not 01 >> ERROR: SSL error while connecting: error:04067072:rsa >> routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed >> ERROR: SSL error while connecting: error:0D0C5006:asn1 encoding >> routines:ASN1_item_verify:EVP lib >> ERROR: SSL error while connecting: error:14090086:SSL >> routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed > > I've never seen this error before, and I can't guess very accurately > what's causing it. It looks almost like a failure to speak the (same) > SSL protocol between the two sides. I was able to connect to your > boxbackup server over the Internet using openssl s_client and it did > negotiate SSL properly, so I think the server is working. > > I'm not sure I understand your setup correctly. Are both server and > client OpenVZ containers running on Centos 6.4 hosts? Is the host > 32-bit or 64-bit? > > Are the clients both Debian Wheezy? 32-bit or 64-bit? Did you get the > Box Backup packages from Debian in both cases? > > Do you have any other Box Backup servers that you could try connecting > your client to, that are known to be working? > > Could you try this on your client, and let me know what the output is: > > openssl s_client -connect backup01.cableninja.net:2201 > > You could try the same command with -cert, -key and -CAfile pointing > to your bbackupd certificate, private key and trusted CAs PEM files, > and it should stay connected to the server (and not be disconnected > due to not providing a valid certificate to the server). > > Unless something obvious comes up, I think I'm going to have to > reproduce the problem. > > Cheers, Chris. From chris at qwirx.com Mon Apr 8 10:35:48 2013 From: chris at qwirx.com (Chris Wilson) Date: Mon, 8 Apr 2013 10:35:48 +0100 (BST) Subject: [Box Backup] RSA_padding_check_PKCS1_type_1:block type is not 01 In-Reply-To: <51624FDB.6060504@cableninja.net> References: <51600F2C.7070208@cableninja.net> <51624FDB.6060504@cableninja.net> Message-ID: Hi Chris, On Sun, 7 Apr 2013, cableninja at cableninja.net wrote: >> Could you try this on your client, and let me know what the output is: >> >> openssl s_client -connect backup01.cableninja.net:2201 > > Below is the output of this command > > # openssl s_client -connect backup01.cableninja.net:2201 > test01.out > depth=0 CN = backup02.cableninja.net Note that you connected to backup01.cableninja.net, but the certificate that you received was for backup02.cableninja.net. Unless the CAs are the same, that might be a problem, and it might indicate that the hostname changed or that some network trickery is going on. Otherwise, it appears that bbstored is working properly on both systems, so the problem is likely to be in bbackupd or the network inbetween. I set up bbstored and bbackupd from Debian packages on a fresh 32-bit Wheezy server (both on the same server) and wasn't able to reproduce the problem. So either it's only in the 64-bit version of Wheezy, or it's in the network. Could you try making each machine back up to its local bbstored instead of the remote one, to eliminate the network? > Below is the output of the command with the cert/key/CAFile options > > # openssl s_client -cert /etc/boxbackup/bbackupd/0-cert.pem -key > /etc/boxbackup/bbackupd/0-key.pem -CAfile > /etc/boxbackup/bbackupd/serverCA.pem -connect backup01.cableninja.net:2201 > > test01.out > depth=1 CN = Backup system server root > verify return:1 > depth=0 CN = backup02.cableninja.net > verify error:num=7:certificate signature failure > verify return:1 > depth=0 CN = backup02.cableninja.net > verify return:1 > 140498802353832:error:1409441B:SSL routines:SSL3_READ_BYTES:tlsv1 alert > decrypt error:s3_pkt.c:1256:SSL alert number 51 > 140498802353832:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake > failure:s23_lib.c:177: > > # cat test02.out > CONNECTED(00000003) > --- > Certificate chain > 0 s:/CN=backup02.cableninja.net > i:/CN=Backup system server root > --- > Server certificate > -----BEGIN CERTIFICATE----- > MIICujCCAaICAQIwDQYJKoZIhvcNAQEFBQAwJDEiMCAGA1UEAwwZQmFja3VwIHN5 > c3RlbSBzZXJ2ZXIgcm9vdDAeFw0xMzA0MDYwOTAyNDRaFw0yNjEyMTQwOTAyNDRa > MCIxIDAeBgNVBAMMF2JhY2t1cDAyLmNhYmxlbmluamEubmV0MIIBIjANBgkqhkiG > 9w0BAQEFAAOCAQ8AMIIBCgKCAQEA873ol7cKT7KZ4lHqV2OOqjv/XTrEMIUO87Cf > 3jx2G2ekFe0KSmhSLvCONCA/LqT21nHiHGotiBQcZq1r9KNM8Zv3vSBXKDMIr0EW > vVAzba0/YQLTyXbYCG7f4+y7iEszGQsYLjm5mTkPXbEX7iPNHi5fgcnfiY7D5YtL > wpdm3beirLR/l4PmOIJF8EeFxPUmLg/E9ZdbKLAAYEtPT4x2Y6FVUArkbZF+iag1 > dIHDUsE3KqRSsWlL5N32GNYJmF1fW9S99/vFxeJMdIRqe//XHA4gzxqd5paf+uCG > XV8JkKmgwPzpSGlAtlVIaS2X87Oqr63d7uMJ1mulRVp6OuO3SwIDAQABMA0GCSqG > SIb3DQEBBQUAA4IBAQCmV2FOLkOCjv8yZ1Uj1aFu5hu9sX/LnEP0mSCZcbsCjUpv > V9L1RQVPaH7K6kxWrF0rlWsktSx9f3AIfQYQyqRi/iuTe08MHYgUFfFbdZ2rGxmR > 7YNsSBAijmnG+0kSa/AlgatPrAUMn71LyVacEJ3wCsKSu4fi8rgw8k5fFLf0JB47 > ESxmKe2qH6pSlwvoyJO2vDd8o7mod6AtEsFN2s8tS1AlKGSli3RmLVYqyz/C5wDb > XM0XlC9WSYUMnctpEktRZbDd6Hpb3dfBCZHXhZWBE7SaYqWPKCic8zuDLH235xCw > irDywQDGUlZnjTqDUdmzuYPSsjkY01e7cTUJOBzm > -----END CERTIFICATE----- > subject=/CN=backup02.cableninja.net > issuer=/CN=Backup system server root > --- > > I'm not too familiar with openssl, but it appears to me that theres something > wrong with my serverCA? or my certificates in general? I think so too (in addition to the padding error). Are both your server certificates signed by the same CA? > I've just nuked all of my configs and certs, I'm currently trying again from > scratch. I'll update when I have tried all possibilities and what we've got. > >> Unless something obvious comes up, I think I'm going to have to reproduce > the problem. > > I can provide test containers for you, even give you access to these existing > ones, if you dont have an existing setup similar to what I have, to test on. > Let me know if you'd like me to do so. > > Thanks for your time/help. Thanks for your patience and investigation :) Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From cableninja at cableninja.net Mon Apr 8 17:54:44 2013 From: cableninja at cableninja.net (cableninja at cableninja.net) Date: Mon, 08 Apr 2013 09:54:44 -0700 Subject: [Box Backup] RSA_padding_check_PKCS1_type_1:block type is not 01 In-Reply-To: References: <51600F2C.7070208@cableninja.net> <51624FDB.6060504@cableninja.net> Message-ID: <5162F654.4050208@cableninja.net> Hi Chris, >Note that you connected to backup01.cableninja.net, but the certificate that you received was for backup02.cableninja.net. Unless the CAs are the same, that might be a problem, and it might >indicate that the hostname changed or that some network trickery is going on. Hah, thank you for pointing this pointing this out to me, I apologize for all of the run around I've caused you. Since not all containers have a public IP, the private lan exists. As this is the case, hostnames resolve to an external IP, and that causes a whole mess of problems. I put the backup servers hostnames into the bbackupd's servers hosts file, turns out, I put the entries backwards. backup01 and backup02's last octet were backwards, backup01 should have been .101 where it was entered as .201 I just ran a test after adjusting the hosts file, I now get the following: @db01:/# openssl s_client -connect backup01.cableninja.net:2201 > test01.out depth=0 CN = backup01.cableninja.net verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 CN = backup01.cableninja.net verify error:num=27:certificate not trusted verify return:1 depth=0 CN = backup01.cableninja.net verify error:num=21:unable to verify the first certificate verify return:1 140158446786216:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1256:SSL alert number 40 140158446786216:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177: @db01:/# openssl s_client -cert /etc/boxbackup/bbackupd/0-cert.pem -key /etc/boxbackup/bbackupd/0-key.pem -CAfile /etc/boxbackup/bbackupd/serverCA.pem -connect backup01.cableninja.net:2201 > test01.out depth=1 CN = Backup system server root verify return:1 depth=0 CN = backup01.cableninja.net verify return:1 ^C Now it looks like everything is exactly as its supposed to be! However... When I start the client, I'm seeing the following on the backup server, it appears its still having problems. Apr 8 10:39:40 backup01 bbstored[2475]: NOTICE: Message from child process 5712: Incoming connection from 10.1.10.251 port 49503 Apr 8 10:39:40 backup01 bbstored[5712]: ERROR: SSL error while accepting connection: error:1409441B:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error Apr 8 10:39:40 backup01 bbstored[5712]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(207) Apr 8 10:39:40 backup01 bbstored[5712]: ERROR: Error in child process, terminating connection: exception Connection TLSHandshakeFailed(7/30) Apr 8 10:39:45 backup01 bbstored[2475]: NOTICE: Message from child process 5713: Incoming connection from 10.1.10.250 port 40327 Apr 8 10:39:46 backup01 bbstored client=0000000000[5713]: NOTICE: Login from Client ID 0000000000 Read/Write Apr 8 10:39:46 backup01 bbstored client=0000000000[5713]: NOTICE: Session finished for Client ID 0000000000 Apr 8 10:39:46 backup01 bbstored client=0000000000[5713]: NOTICE: Connection statistics for BACKUP-0: IN=105 OUT=172 TOTAL=277 I found the solution on the troubleshooting page here: http://www.boxbackup.org/wiki/Troubleshooting#TLSHandshakeFailed but I'm not quite clear on how to generate a new server CA. I know how to do the rest of the solution. If you could give me some info here, I'd appreciate it. Thank you again for your help. I'll submit another thread soon for what I believe are missing/incorrect post-install steps on the wheezy packages. - Chris On 04/08/2013 02:35 AM, Chris Wilson wrote: > Hi Chris, > > On Sun, 7 Apr 2013, cableninja at cableninja.net wrote: > >>> Could you try this on your client, and let me know what the output is: >>> >>> openssl s_client -connect backup01.cableninja.net:2201 >> >> Below is the output of this command >> >> # openssl s_client -connect backup01.cableninja.net:2201 > test01.out >> depth=0 CN = backup02.cableninja.net > > Note that you connected to backup01.cableninja.net, but the > certificate that you received was for backup02.cableninja.net. Unless > the CAs are the same, that might be a problem, and it might indicate > that the hostname changed or that some network trickery is going on. > > Otherwise, it appears that bbstored is working properly on both > systems, so the problem is likely to be in bbackupd or the network > inbetween. > > I set up bbstored and bbackupd from Debian packages on a fresh 32-bit > Wheezy server (both on the same server) and wasn't able to reproduce > the problem. So either it's only in the 64-bit version of Wheezy, or > it's in the network. > > Could you try making each machine back up to its local bbstored > instead of the remote one, to eliminate the network? > >> Below is the output of the command with the cert/key/CAFile options >> >> # openssl s_client -cert /etc/boxbackup/bbackupd/0-cert.pem -key >> /etc/boxbackup/bbackupd/0-key.pem -CAfile >> /etc/boxbackup/bbackupd/serverCA.pem -connect >> backup01.cableninja.net:2201 > test01.out >> depth=1 CN = Backup system server root >> verify return:1 >> depth=0 CN = backup02.cableninja.net >> verify error:num=7:certificate signature failure >> verify return:1 >> depth=0 CN = backup02.cableninja.net >> verify return:1 >> 140498802353832:error:1409441B:SSL routines:SSL3_READ_BYTES:tlsv1 >> alert decrypt error:s3_pkt.c:1256:SSL alert number 51 >> 140498802353832:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake >> failure:s23_lib.c:177: >> >> # cat test02.out >> CONNECTED(00000003) >> --- >> Certificate chain >> 0 s:/CN=backup02.cableninja.net >> i:/CN=Backup system server root >> --- >> Server certificate >> -----BEGIN CERTIFICATE----- >> MIICujCCAaICAQIwDQYJKoZIhvcNAQEFBQAwJDEiMCAGA1UEAwwZQmFja3VwIHN5 >> c3RlbSBzZXJ2ZXIgcm9vdDAeFw0xMzA0MDYwOTAyNDRaFw0yNjEyMTQwOTAyNDRa >> MCIxIDAeBgNVBAMMF2JhY2t1cDAyLmNhYmxlbmluamEubmV0MIIBIjANBgkqhkiG >> 9w0BAQEFAAOCAQ8AMIIBCgKCAQEA873ol7cKT7KZ4lHqV2OOqjv/XTrEMIUO87Cf >> 3jx2G2ekFe0KSmhSLvCONCA/LqT21nHiHGotiBQcZq1r9KNM8Zv3vSBXKDMIr0EW >> vVAzba0/YQLTyXbYCG7f4+y7iEszGQsYLjm5mTkPXbEX7iPNHi5fgcnfiY7D5YtL >> wpdm3beirLR/l4PmOIJF8EeFxPUmLg/E9ZdbKLAAYEtPT4x2Y6FVUArkbZF+iag1 >> dIHDUsE3KqRSsWlL5N32GNYJmF1fW9S99/vFxeJMdIRqe//XHA4gzxqd5paf+uCG >> XV8JkKmgwPzpSGlAtlVIaS2X87Oqr63d7uMJ1mulRVp6OuO3SwIDAQABMA0GCSqG >> SIb3DQEBBQUAA4IBAQCmV2FOLkOCjv8yZ1Uj1aFu5hu9sX/LnEP0mSCZcbsCjUpv >> V9L1RQVPaH7K6kxWrF0rlWsktSx9f3AIfQYQyqRi/iuTe08MHYgUFfFbdZ2rGxmR >> 7YNsSBAijmnG+0kSa/AlgatPrAUMn71LyVacEJ3wCsKSu4fi8rgw8k5fFLf0JB47 >> ESxmKe2qH6pSlwvoyJO2vDd8o7mod6AtEsFN2s8tS1AlKGSli3RmLVYqyz/C5wDb >> XM0XlC9WSYUMnctpEktRZbDd6Hpb3dfBCZHXhZWBE7SaYqWPKCic8zuDLH235xCw >> irDywQDGUlZnjTqDUdmzuYPSsjkY01e7cTUJOBzm >> -----END CERTIFICATE----- >> subject=/CN=backup02.cableninja.net >> issuer=/CN=Backup system server root >> --- >> >> I'm not too familiar with openssl, but it appears to me that theres >> something wrong with my serverCA? or my certificates in general? > > I think so too (in addition to the padding error). Are both your > server certificates signed by the same CA? > >> I've just nuked all of my configs and certs, I'm currently trying >> again from scratch. I'll update when I have tried all possibilities >> and what we've got. >> >>> Unless something obvious comes up, I think I'm going to have to >>> reproduce >> the problem. >> >> I can provide test containers for you, even give you access to these >> existing ones, if you dont have an existing setup similar to what I >> have, to test on. Let me know if you'd like me to do so. >> >> Thanks for your time/help. > > Thanks for your patience and investigation :) > > Cheers, Chris. From chris at qwirx.com Mon Apr 8 19:25:25 2013 From: chris at qwirx.com (Chris Wilson) Date: Mon, 8 Apr 2013 19:25:25 +0100 (BST) Subject: [Box Backup] RSA_padding_check_PKCS1_type_1:block type is not 01 In-Reply-To: <51630ACF.5010004@cableninja.net> References: <51600F2C.7070208@cableninja.net> <51624FDB.6060504@cableninja.net> <5162F654.4050208@cableninja.net> <51630ACF.5010004@cableninja.net> Message-ID: Hi Chris, On Mon, 8 Apr 2013, Chris Walker wrote: > This was because the backup locations of the config on bbackupd client did > not have any entries (and I initially wrote them incorrectly). This has also > been resolved. My first backup is running right now: I'm glad you got everything sorted, and thanks for providing detailed notes about what you did, to help others in future. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \__/_/_/_//_/___/ | We are GNU : free your mind & your software | From cwalker at cableninja.net Mon Apr 8 19:22:07 2013 From: cwalker at cableninja.net (Chris Walker) Date: Mon, 08 Apr 2013 11:22:07 -0700 Subject: [Box Backup] RSA_padding_check_PKCS1_type_1:block type is not 01 In-Reply-To: <5162F654.4050208@cableninja.net> References: <51600F2C.7070208@cableninja.net> <51624FDB.6060504@cableninja.net> <5162F654.4050208@cableninja.net> Message-ID: <51630ACF.5010004@cableninja.net> Hi all, I've resolved all SSL issues at this time. I moved all of my existing certs on the bbstored side to a backup folder, and then did the following: - Re-init all certs @backup01 # bbstored-certs /etc/boxbackup init - Sign the server cert @backup01 # bbstored-certs /etc/boxbackup sign-server /etc/boxbackup/bbstored/backup01.cableninja.net-csr.pem - Resign the client cert @backup01 # bbstored-certs /etc/boxbackup sign /etc/boxbackup/bbstored/clients/0-csr.pem - Move the necessary certs to the client (I'm not sure the backup01.cableninja-cert.pem is required, but I copied it during testing) @backup01 # scp /etc/boxbackup/bbstored/roots/serverCA.pem root at db01.cableninja.net:/etc/boxbackup/bbackupd/ @backup01 # scp /etc/boxbackup/bbstored/servers/backup01.cableninja.net-cert.pem root at db01.cableninja.net:/etc/boxbackup/bbackupd/ @backup01 # scp /etc/boxbackup/bbstored/clients/0-cert.pem root at db01.cableninja.net:/etc/boxbackup/bbackupd/ - Stop both server and client @backup01 # /etc/init.d/boxbackup-server stop @db01 # /etc/init.d/boxbackup-client stop - Start the server, then the client @backup01 # /etc/init.d/boxbackup-server start @db01 # /etc/init.d/boxbackup-client start I now see the following in the log files: @backup01 # tail /var/log/syslog Apr 8 11:56:34 backup01 bbstored[5981]: NOTICE: Box Backup Store Server v0.11rc8+2837, (c) Ben Summers and contributors 2003-2010 Apr 8 11:56:34 backup01 bbstored[5983]: NOTICE: Starting daemon, version: 0.11rc8+2837 Apr 8 11:56:34 backup01 bbstored[5983]: NOTICE: Using configuration file: /etc/boxbackup/bbstored.conf Apr 8 11:56:55 backup01 bbstored[5983]: NOTICE: Message from child process 5986: Incoming connection from 10.1.10.250 port 40337 Apr 8 11:56:55 backup01 bbstored client=0000000000[5986]: NOTICE: Login from Client ID 0000000000 Read/Write Apr 8 11:56:55 backup01 bbstored client=0000000000[5986]: NOTICE: Session finished for Client ID 0000000000 Apr 8 11:56:55 backup01 bbstored client=0000000000[5986]: NOTICE: Connection statistics for BACKUP-0: IN=105 OUT=172 TOTAL=277 @db01 # tail /var/log/syslog Apr 8 11:56:55 db01 bbackupd[6651]: NOTICE: Box Backup Client v0.11rc8+2837, (c) Ben Summers and contributors 2003-2010 Apr 8 11:56:55 db01 bbackupd[6653]: NOTICE: Starting daemon, version: 0.11rc8+2837 Apr 8 11:56:55 db01 bbackupd[6653]: NOTICE: Using configuration file: /etc/boxbackup/bbackupd.conf Apr 8 11:56:55 db01 bbackupd[6653]: NOTICE: Beginning scan of local files Apr 8 11:56:55 db01 bbackupd[6653]: WARNING: Notify script returned error code: 32512 (/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-ok "/etc/boxbackup/bbackupd.conf") Apr 8 11:56:55 db01 bbackupd[6653]: NOTICE: Finished scan of local files Apr 8 11:56:55 db01 bbackupd[6653]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0 So from what I'm reading, backups are connecting just fine, but its not pushing any files. This was because the backup locations of the config on bbackupd client did not have any entries (and I initially wrote them incorrectly). This has also been resolved. My first backup is running right now: @backup01 # Apr 8 12:13:01 backup01 bbstored[5983]: NOTICE: Message from child process 5996: Incoming connection from 10.1.10.250 port 40342 Apr 8 12:13:01 backup01 bbstored client=0000000000[5996]: NOTICE: Login from Client ID 0000000000 Read/Write @db01 # Apr 8 12:13:01 db01 bbackupd[6779]: NOTICE: Box Backup Client v0.11rc8+2837, (c) Ben Summers and contributors 2003-2010 Apr 8 12:13:01 db01 bbackupd[6781]: NOTICE: Starting daemon, version: 0.11rc8+2837 Apr 8 12:13:01 db01 bbackupd[6781]: NOTICE: Using configuration file: /etc/boxbackup/bbackupd.conf Apr 8 12:13:01 db01 bbackupd[6781]: NOTICE: Beginning scan of local files Apr 8 12:13:01 db01 bbackupd[6781]: WARNING: Ignored directory: /run: is a mount point; create a new location if you want to back it up Now that I've completed the head bashing process. I'm hoping that my next setups will be 500% easier! Thank you again for your help. On 04/08/2013 09:54 AM, cableninja at cableninja.net wrote: > Hi Chris, > >> Note that you connected to backup01.cableninja.net, but the > certificate that you received was for backup02.cableninja.net. Unless > the CAs are the same, that might be a problem, and it might >> indicate that the hostname changed or that some network trickery is > going on. > > Hah, thank you for pointing this pointing this out to me, I apologize > for all of the run around I've caused you. Since not all containers > have a public IP, the private lan exists. As this is the case, > hostnames resolve to an external IP, and that causes a whole mess of > problems. I put the backup servers hostnames into the bbackupd's > servers hosts file, turns out, I put the entries backwards. backup01 > and backup02's last octet were backwards, backup01 should have been > .101 where it was entered as .201 > > I just ran a test after adjusting the hosts file, I now get the > following: > > @db01:/# openssl s_client -connect backup01.cableninja.net:2201 > > test01.out > depth=0 CN = backup01.cableninja.net > verify error:num=20:unable to get local issuer certificate > verify return:1 > depth=0 CN = backup01.cableninja.net > verify error:num=27:certificate not trusted > verify return:1 > depth=0 CN = backup01.cableninja.net > verify error:num=21:unable to verify the first certificate > verify return:1 > 140158446786216:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 > alert handshake failure:s3_pkt.c:1256:SSL alert number 40 > 140158446786216:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake > failure:s23_lib.c:177: > > > @db01:/# openssl s_client -cert /etc/boxbackup/bbackupd/0-cert.pem > -key /etc/boxbackup/bbackupd/0-key.pem -CAfile > /etc/boxbackup/bbackupd/serverCA.pem -connect > backup01.cableninja.net:2201 > test01.out > depth=1 CN = Backup system server root > verify return:1 > depth=0 CN = backup01.cableninja.net > verify return:1 > ^C > > Now it looks like everything is exactly as its supposed to be! However... > > When I start the client, I'm seeing the following on the backup > server, it appears its still having problems. > > Apr 8 10:39:40 backup01 bbstored[2475]: NOTICE: Message from child > process 5712: Incoming connection from 10.1.10.251 port 49503 > Apr 8 10:39:40 backup01 bbstored[5712]: ERROR: SSL error while > accepting connection: error:1409441B:SSL > routines:SSL3_READ_BYTES:tlsv1 alert decrypt error > Apr 8 10:39:40 backup01 bbstored[5712]: WARNING: Exception thrown: > ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(207) > Apr 8 10:39:40 backup01 bbstored[5712]: ERROR: Error in child > process, terminating connection: exception Connection > TLSHandshakeFailed(7/30) > Apr 8 10:39:45 backup01 bbstored[2475]: NOTICE: Message from child > process 5713: Incoming connection from 10.1.10.250 port 40327 > Apr 8 10:39:46 backup01 bbstored client=0000000000[5713]: NOTICE: > Login from Client ID 0000000000 Read/Write > Apr 8 10:39:46 backup01 bbstored client=0000000000[5713]: NOTICE: > Session finished for Client ID 0000000000 > Apr 8 10:39:46 backup01 bbstored client=0000000000[5713]: NOTICE: > Connection statistics for BACKUP-0: IN=105 OUT=172 TOTAL=277 > > I found the solution on the troubleshooting page here: > http://www.boxbackup.org/wiki/Troubleshooting#TLSHandshakeFailed > but I'm not quite clear on how to generate a new server CA. I know how > to do the rest of the solution. If you could give me some info here, > I'd appreciate it. > > Thank you again for your help. I'll submit another thread soon for > what I believe are missing/incorrect post-install steps on the wheezy > packages. > > - Chris > > On 04/08/2013 02:35 AM, Chris Wilson wrote: >> Hi Chris, >> >> On Sun, 7 Apr 2013, cableninja at cableninja.net wrote: >> >>>> Could you try this on your client, and let me know what the output is: >>>> >>>> openssl s_client -connect backup01.cableninja.net:2201 >>> >>> Below is the output of this command >>> >>> # openssl s_client -connect backup01.cableninja.net:2201 > test01.out >>> depth=0 CN = backup02.cableninja.net >> >> Note that you connected to backup01.cableninja.net, but the >> certificate that you received was for backup02.cableninja.net. Unless >> the CAs are the same, that might be a problem, and it might indicate >> that the hostname changed or that some network trickery is going on. >> >> Otherwise, it appears that bbstored is working properly on both >> systems, so the problem is likely to be in bbackupd or the network >> inbetween. >> >> I set up bbstored and bbackupd from Debian packages on a fresh 32-bit >> Wheezy server (both on the same server) and wasn't able to reproduce >> the problem. So either it's only in the 64-bit version of Wheezy, or >> it's in the network. >> >> Could you try making each machine back up to its local bbstored >> instead of the remote one, to eliminate the network? >> >>> Below is the output of the command with the cert/key/CAFile options >>> >>> # openssl s_client -cert /etc/boxbackup/bbackupd/0-cert.pem -key >>> /etc/boxbackup/bbackupd/0-key.pem -CAfile >>> /etc/boxbackup/bbackupd/serverCA.pem -connect >>> backup01.cableninja.net:2201 > test01.out >>> depth=1 CN = Backup system server root >>> verify return:1 >>> depth=0 CN = backup02.cableninja.net >>> verify error:num=7:certificate signature failure >>> verify return:1 >>> depth=0 CN = backup02.cableninja.net >>> verify return:1 >>> 140498802353832:error:1409441B:SSL routines:SSL3_READ_BYTES:tlsv1 >>> alert decrypt error:s3_pkt.c:1256:SSL alert number 51 >>> 140498802353832:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake >>> failure:s23_lib.c:177: >>> >>> # cat test02.out >>> CONNECTED(00000003) >>> --- >>> Certificate chain >>> 0 s:/CN=backup02.cableninja.net >>> i:/CN=Backup system server root >>> --- >>> Server certificate >>> -----BEGIN CERTIFICATE----- >>> MIICujCCAaICAQIwDQYJKoZIhvcNAQEFBQAwJDEiMCAGA1UEAwwZQmFja3VwIHN5 >>> c3RlbSBzZXJ2ZXIgcm9vdDAeFw0xMzA0MDYwOTAyNDRaFw0yNjEyMTQwOTAyNDRa >>> MCIxIDAeBgNVBAMMF2JhY2t1cDAyLmNhYmxlbmluamEubmV0MIIBIjANBgkqhkiG >>> 9w0BAQEFAAOCAQ8AMIIBCgKCAQEA873ol7cKT7KZ4lHqV2OOqjv/XTrEMIUO87Cf >>> 3jx2G2ekFe0KSmhSLvCONCA/LqT21nHiHGotiBQcZq1r9KNM8Zv3vSBXKDMIr0EW >>> vVAzba0/YQLTyXbYCG7f4+y7iEszGQsYLjm5mTkPXbEX7iPNHi5fgcnfiY7D5YtL >>> wpdm3beirLR/l4PmOIJF8EeFxPUmLg/E9ZdbKLAAYEtPT4x2Y6FVUArkbZF+iag1 >>> dIHDUsE3KqRSsWlL5N32GNYJmF1fW9S99/vFxeJMdIRqe//XHA4gzxqd5paf+uCG >>> XV8JkKmgwPzpSGlAtlVIaS2X87Oqr63d7uMJ1mulRVp6OuO3SwIDAQABMA0GCSqG >>> SIb3DQEBBQUAA4IBAQCmV2FOLkOCjv8yZ1Uj1aFu5hu9sX/LnEP0mSCZcbsCjUpv >>> V9L1RQVPaH7K6kxWrF0rlWsktSx9f3AIfQYQyqRi/iuTe08MHYgUFfFbdZ2rGxmR >>> 7YNsSBAijmnG+0kSa/AlgatPrAUMn71LyVacEJ3wCsKSu4fi8rgw8k5fFLf0JB47 >>> ESxmKe2qH6pSlwvoyJO2vDd8o7mod6AtEsFN2s8tS1AlKGSli3RmLVYqyz/C5wDb >>> XM0XlC9WSYUMnctpEktRZbDd6Hpb3dfBCZHXhZWBE7SaYqWPKCic8zuDLH235xCw >>> irDywQDGUlZnjTqDUdmzuYPSsjkY01e7cTUJOBzm >>> -----END CERTIFICATE----- >>> subject=/CN=backup02.cableninja.net >>> issuer=/CN=Backup system server root >>> --- >>> >>> I'm not too familiar with openssl, but it appears to me that theres >>> something wrong with my serverCA? or my certificates in general? >> >> I think so too (in addition to the padding error). Are both your >> server certificates signed by the same CA? >> >>> I've just nuked all of my configs and certs, I'm currently trying >>> again from scratch. I'll update when I have tried all possibilities >>> and what we've got. >>> >>>> Unless something obvious comes up, I think I'm going to have to >>>> reproduce >>> the problem. >>> >>> I can provide test containers for you, even give you access to these >>> existing ones, if you dont have an existing setup similar to what I >>> have, to test on. Let me know if you'd like me to do so. >>> >>> Thanks for your time/help. >> >> Thanks for your patience and investigation :) >> >> Cheers, Chris. > > _______________________________________________ > Boxbackup mailing list > Boxbackup at boxbackup.org > http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup From cableninja at cableninja.net Wed Apr 10 13:34:34 2013 From: cableninja at cableninja.net (cableninja at cableninja.net) Date: Wed, 10 Apr 2013 05:34:34 -0700 Subject: [Box Backup] Debian Wheezy package/configuration confusion Message-ID: <51655C5A.5090203@cableninja.net> Hello all, Before I begin, I finally have functioning backups on multiple servers working (after many nights of head bashing) and, though the pain of setting it up, so far, I enjoy boxbackup! Thank you all for the help in getting it going in the other thread I had going. I've determined where most of the problems have been stemming from. It turns out a lot of my problems were due to path confusion using the bb* tools. I mentioned I would follow up regarding the issues I noticed, so here we are. Debian Wheezy with backports amd x64, kernel 3.2.0-4-amd64 root at test01:/etc/boxbackup# bbstored --version 0.11rc8+2837 root at test01:/etc/boxbackup# bbackupd --version 0.11rc8+2837 Problem 1: Problematic unless, die function in bbstored-certs /usr/bin/bbstored-certs @ line85: elsif($command eq 'sign-server') {&cmd_sign_server;} >else >{ > die "Unknown command $command" >} > >sub cmd_init >{ > # create directories ># unless( > mkdir($cert_dir,0700); > mkdir($cert_dir.'/roots',0700); > mkdir($cert_dir.'/keys',0700); > mkdir($cert_dir.'/servers',0700); > mkdir($cert_dir.'/clients',0700);#) ># { ># die "Failed to create directory structure" ># } > > # create root keys and certrs > cmd_init_create_root('client'); > cmd_init_create_root('server'); >} I was required to comment out/adjust the 'unless()' function because this would fail if the directories existed or not. Problem 2: bbstored-config/bbstored-certs confusion ># bbstored-config /etc/boxbackup test01.cableninja.net bbstored /etc/boxbackup/raidfile.conf [145/897] >Checking permissions on //backup >Checking permissions on //backup >Checking permissions on //backup > >Setup bbstored config utility. > >Configuration: > Writing configuration file: /etc/boxbackup/bbstored.conf > Writing empty accounts file: /etc/boxbackup/bbstored/accounts.txt > Server hostname: test01.cableninja.net > RaidFile config: /etc/boxbackup/raidfile.conf > >Creating blank accounts file >Generating private key... >Generating RSA private key, 2048 bit long modulus >.+++ >............................+++ >e is 65537 (0x10001) >You are about to be asked to enter information that will be incorporated >into your certificate request. >What you are about to enter is what is called a Distinguished Name or a DN. >There are quite a few fields but you can leave some blank >For some fields there will be a default value, >If you enter '.', the field will be left blank. >----- >Country Name (2 letter code) [AU]:State or Province Name (full name) [Some-State]:Locality Name (eg, city) []:Organization Na >me (eg, company) [Internet Widgits Pty Ltd]:Organizational Unit Name (eg, section) []:Common Name (e.g. server FQDN or YOUR n >ame) []:Email Address []: >Please enter the following 'extra' attributes >to be sent with your certificate request >A challenge password []:An optional company name []: > >Writing configuration file /etc/boxbackup/bbstored.conf > > >=================================================================== >bbstored basic configuration complete. > >What you need to do now... > >1) Sign /etc/boxbackup/bbstored/test01.cableninja.net-csr.pem > using the bbstored-certs utility. > >2) Install the server certificate and root CA certificate as > /etc/boxbackup/bbstored/test01.cableninja.net-cert.pem > /etc/boxbackup/bbstored/clientCA.pem > >3) You may wish to read the configuration file > /etc/boxbackup/bbstored.conf > and adjust as appropraite. > >4) Create accounts with bbstoreaccounts > >5) Start the backup store daemon with the command > /usr/local/sbin/bbstored > in /etc/rc.local, or your local equivalent. > >=================================================================== > >root at test01:/# cd /etc/boxbackup/ >root at test01:/etc/boxbackup# ls -alh >total 24K >drwxr-xr-x 4 root root 4.0K Apr 10 13:57 . >drwxr-xr-x 52 root root 4.0K Apr 10 13:49 .. >drwxr-xr-x 2 root root 4.0K Oct 28 2011 bbackupd >drwxr-xr-x 2 root root 4.0K Apr 10 13:57 bbstored >-rw-r--r-- 1 root root 620 Apr 10 13:57 bbstored.conf >-rw-r--r-- 1 root root 75 Apr 10 13:56 raidfile.conf >root at test01:/etc/boxbackup# cd bbstored/ >root at test01:/etc/boxbackup/bbstored# ls -alh >total 16K >drwxr-xr-x 2 root root 4.0K Apr 10 13:57 . >drwxr-xr-x 4 root root 4.0K Apr 10 13:57 .. >-rw-r--r-- 1 root root 0 Apr 10 13:57 accounts.txt >-rw-r--r-- 1 root root 907 Apr 10 13:57 test01.cableninja.net-csr.pem >-rw-r--r-- 1 root root 1.7K Apr 10 13:57 test01.cableninja.net-key.pem >root at test01:/etc/boxbackup/bbstored# bbstored-certs /etc/boxbackup init >Generating RSA private key, 2048 bit long modulus >.............+++ >.............................................................................................................+++ >e is 65537 (0x10001) >You are about to be asked to enter information that will be incorporated >into your certificate request. >What you are about to enter is what is called a Distinguished Name or a DN. >There are quite a few fields but you can leave some blank >For some fields there will be a default value, >If you enter '.', the field will be left blank. >----- >Country Name (2 letter code) [AU]:State or Province Name (full name) [Some-State]:Locality Name (eg, city) []:Organization Na >me (eg, company) [Internet Widgits Pty Ltd]:Organizational Unit Name (eg, section) []:Common Name (e.g. server FQDN or YOUR n >ame) []:Email Address []: >Please enter the following 'extra' attributes >to be sent with your certificate request >A challenge password []:An optional company name []: > >Signature ok >subject=/CN=Backup system client root >Getting Private key >Generating RSA private key, 2048 bit long modulus >............................................+++ >...................................................+++ >e is 65537 (0x10001) >You are about to be asked to enter information that will be incorporated >into your certificate request. >What you are about to enter is what is called a Distinguished Name or a DN. >There are quite a few fields but you can leave some blank >For some fields there will be a default value, >If you enter '.', the field will be left blank. >----- >Country Name (2 letter code) [AU]:State or Province Name (full name) [Some-State]:Locality Name (eg, city) []:Organization Na >me (eg, company) [Internet Widgits Pty Ltd]:Organizational Unit Name (eg, section) []:Common Name (e.g. server FQDN or YOUR n >ame) []:Email Address []: >Please enter the following 'extra' attributes >to be sent with your certificate request >A challenge password []:An optional company name []: > >Signature ok >subject=/CN=Backup system server root >Getting Private key >root at test01:/etc/boxbackup/bbstored# ls -alh >total 16K >drwxr-xr-x 2 root root 4.0K Apr 10 13:57 . >drwxr-xr-x 8 root root 4.0K Apr 10 13:58 .. >-rw-r--r-- 1 root root 0 Apr 10 13:57 accounts.txt >-rw-r--r-- 1 root root 907 Apr 10 13:57 test01.cableninja.net-csr.pem >-rw-r--r-- 1 root root 1.7K Apr 10 13:57 test01.cableninja.net-key.pem >root at test01:/etc/boxbackup/bbstored# cd ../ >root at test01:/etc/boxbackup# ls -alh >total 40K >drwxr-xr-x 8 root root 4.0K Apr 10 13:58 . >drwxr-xr-x 52 root root 4.0K Apr 10 13:49 .. >drwxr-xr-x 2 root root 4.0K Oct 28 2011 bbackupd >drwxr-xr-x 2 root root 4.0K Apr 10 13:57 bbstored >-rw-r--r-- 1 root root 620 Apr 10 13:57 bbstored.conf >drwx------ 2 root root 4.0K Apr 10 13:58 clients >drwx------ 2 root root 4.0K Apr 10 13:58 keys >-rw-r--r-- 1 root root 75 Apr 10 13:56 raidfile.conf >drwx------ 2 root root 4.0K Apr 10 13:58 roots >drwx------ 2 root root 4.0K Apr 10 13:58 servers >root at test01:/etc/boxbackup# ls -alh roots/ >total 24K >drwx------ 2 root root 4.0K Apr 10 13:58 . >drwxr-xr-x 8 root root 4.0K Apr 10 13:58 .. >-rw-r--r-- 1 root root 1021 Apr 10 13:58 clientCA.pem >-rw-r--r-- 1 root root 3 Apr 10 13:58 clientCA.srl >-rw-r--r-- 1 root root 1021 Apr 10 13:58 serverCA.pem >-rw-r--r-- 1 root root 3 Apr 10 13:58 serverCA.srl >root at test01:/etc/boxbackup# ls -alh bbstored/ >total 16K >drwxr-xr-x 2 root root 4.0K Apr 10 13:57 . >drwxr-xr-x 8 root root 4.0K Apr 10 13:58 .. >-rw-r--r-- 1 root root 0 Apr 10 13:57 accounts.txt >-rw-r--r-- 1 root root 907 Apr 10 13:57 test01.cableninja.net-csr.pem >-rw-r--r-- 1 root root 1.7K Apr 10 13:57 test01.cableninja.net-key.pem >root at test01:/etc/boxbackup# cat bbstored.conf | grep clientCA > TrustedCAsFile = /etc/boxbackup/bbstored/clientCA.pem Note above that the clientCA is actually in /etc/boxbackup/roots/clientCA.pem instead of what bbstored-config actually sets it to (bbstored/clientCA.pem). ># bbstored-certs /etc/boxbackup sign-server bbstored/test01.cableninja.net-csr.pem > >This certificate is for backup server > > test01.cableninja.net > >Signing the wrong certificate compromises the security of your backup system. > >Would you like to sign this certificate? (type 'yes' to confirm) >yes >Signature ok >subject=/CN=test01.cableninja.net >Getting CA Private Key > > >Certificate signed. > >Install the files > > /etc/boxbackup/servers/test01.cableninja.net-cert.pem > /etc/boxbackup/roots/clientCA.pem > >on the server. Again, here bbstored-certs references roots/clientCA.pem ># bbstored-certs /etc/boxbackup sign clients/0-csr.pem > >This certificate is for backup account > > 0 > >Ensure this matches the account number you are expecting. The filename is > > clients/0-csr.pem > >which should include this account number, and additionally, you should check >that you received it from the right person. > >Signing the wrong certificate compromises the security of your backup system. > >Would you like to sign this certificate? (type 'yes' to confirm) >yes >Signature ok >subject=/CN=BACKUP-0 >Getting CA Private Key > > >Certificate signed. > >Send the files > > /etc/boxbackup/clients/0-cert.pem > /etc/boxbackup/roots/serverCA.pem > >to the client. these commands also seem to be based off of where you run them. I ran them in /etc/boxbackup/bbstored/ on my past installations that are now working, it created the clients, keys, servers, roots, folders in bbstored instead of /etc/boxbackup/, like it did this time (this doesnt seem logical, some form of default standard should exist). Problem 3: Jumbled help in bbackupd-config ># bbackupd-config > >Setup bbackupd config utility. > >Bad command line parameters. >Usage: > bbackupd-config config-dir backup-mode account-num server-hostname > working-dir [backup directories] > >Parameters: > config-dir is usually /etc/boxbackup > backup-mode is lazy or snapshot: > lazy mode runs continously, uploading files over a specified age > snapshot mode uploads a snapshot of the filesystem when instructed > explicitly, using bbackupctl sync > account-num (hexdecimal) and server-hostname > are supplied by the server administrator > working-dir is usually /var/bbackupd > backup directories is list of directories to back up the line " account-num (hexdecimal) and server-hostname" - this should be on separate lines, it causes confusion (I missed it the first 3 times I tried to run the command). the line "working-dir is usually /var/bbackupd" - this directory isnt even created in the debian wheezy packages. ># bbackupd-config /etc/boxbackup snapshot 0 test01.cableninja.net / / >It is not recommended that you backup the root directory of your disc at /usr/sbin/bbackupd-config line 103. This warning should have an override, or should not complain at all. Problem 4: Bad directory entries in bbackupd.conf after bbackupd-config > PidFile = //bbackupd.pid My guess is this relates to the working directory/"DirectoryPath" config option. However, I've had no luck getting them to correlate properly and always end up with either nothing but dual slashes (as noted) or a path like /var/run//bbackupd.pid, even without a trailing slash. Lastly, I would like to propse that you guys begin making errors more descriptive, while 'exception X thrown @ y' are somewhat useful, they unfortunately dont really give insight into the real problem. Today I was getting the following error: >Apr 10 03:45:25 backup02 bbstored[3494]: NOTICE: Message from child process 7718: Incoming connection from 10.1.10.251 port 57633 >Apr 10 03:45:25 backup02 bbstored[7718]: ERROR: SSL error while accepting connection: error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 >Apr 10 03:45:25 backup02 bbstored[7718]: ERROR: SSL error while accepting connection: error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed >Apr 10 03:45:25 backup02 bbstored[7718]: ERROR: SSL error while accepting connection: error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib >Apr 10 03:45:25 backup02 bbstored[7718]: ERROR: SSL error while accepting connection: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned >Apr 10 03:45:25 backup02 bbstored[7718]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(207) >Apr 10 03:45:25 backup02 bbstored[7718]: ERROR: Error in child process, terminating connection: exception Connection TLSHandshakeFailed(7/30) >Apr 10 03:54:54 backup02 bbstored[3494]: NOTICE: Message from child process 7867: Incoming connection from 10.1.10.251 port 57641 >Apr 10 03:54:54 backup02 bbstored[7867]: ERROR: SSL error while accepting connection: error:1409441B:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error >Apr 10 03:54:54 backup02 bbstored[7867]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(207) >Apr 10 03:54:54 backup02 bbstored[7867]: ERROR: Error in child process, terminating connection: exception Connection TLSHandshakeFailed(7/30) After repeatedly re-signing the certificates to attempt to fix the first block of errors, I finally was only being given the second block. After slamming hammers in the dark, I finally determined that it was because I had copied the clientCA.pem instead of the serverCA.pem to the client system. Most of the resolution was in 99% guesswork. I also seem to run into permissions problems on the backup location. It seems like this is something that should be handled by raidfile-config or bbstoreaccounts. AAAAAANNNND Lastly, one final suggestion, (if it exists already, apologies), being able to restore modified files from X days ago (may X be configurable?) Anyway, keep up the good work, looking forward to advances in this software for sure! Thanks, - Chris From jan.haastrecht at gmail.com Wed Apr 10 15:00:54 2013 From: jan.haastrecht at gmail.com (Jan Haastrecht) Date: Wed, 10 Apr 2013 16:00:54 +0200 Subject: [Box Backup] Cache Directory Tagging Standard Message-ID: <51657096.4030609@gmail.com> Hello, I am sure that some people manually keep track of cache directories to make sure they are not included in the backups they make, but adding support for http://www.brynosaurus.com/cachedir/spec.html would save some of this work. Can you add support for this standard? I think it should be added here: while((en = ::readdir(dirHandle)) != 0) { rParams.mrContext.DoKeepAlive(); /* Insert code here*/ // Don't need to use // LinuxWorkaround_FinishDirentStruct(en, // rLocalPath.c_str()); // on Linux, as a stat is performed to // get all this info - Jan From jwark at bellaliant.net Wed Apr 17 16:47:42 2013 From: jwark at bellaliant.net (Jack Warkentin) Date: Wed, 17 Apr 2013 12:47:42 -0300 Subject: [Box Backup] Backing up small changes to large files Message-ID: <516EC41E.30602@bellaliant.net> Hi Everybody Box backup has "saved my bacon" several times during the 3 1/2 years or so that I have been using it, including a complete recovery after a motherboard failure. It suits me very well. I greatly appreciate all of the time and efforts that the creators and maintainers of the system have put in in order to come up with such a fine system. I run the server and the client on the same home machine. After a vacation trip I annotated my jpeg format photographs on my GNU/Linux system using exiv2. After annotation I reset the modified time of each annotated file to the Exif "create" time stamp. Subsequently the image files on the system differed by a few tens of bytes from the 1.3 megabytes or so of the files as originally backed up by Box Backup. A curious phenomenon occurred. Some weeks after I had made the above annotations I ran bbackupquery "compare -a" quit and discovered that the backed up versions of the image files were not the same as the versions on the system. I concluded that the small changes resulting from the annotations had not been backed up. A few weeks later, when I ran the above compare command again, I discovered that the backed up versions were now the same as those on the system. I had been meaning to post a query about how to force the small changes to be backed up. Now I would like to know why the small changes were not backed up shortly after they were made, and also, how to force those changes in a timely manner in the future. I run Debian Wheezy, and the 0.11.1~r2837-1 amd64 versions of the boxbackup-server and boxbackup-client Debian packages. The relevant entries from the bbackupd.conf file are UpdateStoreInterval = 3600 MinimumFileAge = 3600 MaxUploadWait = 7200 KeepAliveTime = 120 FileTrackingSizeThreshold = 65535 DiffingUploadSizeThreshold = 8192 MaximumDiffingTime = 300 Any help would be appreciated. Regards Jack Jack Warkentin, phone 902-404-0457, email jwark at bellaliant.net 39 Inverness Avenue, Halifax, Nova Scotia, Canada, B3P 1X6