[Box Backup] Object too big error on 32 bit client

Chris Wilson chris+google at qwirx.com
Mon Dec 31 14:57:25 GMT 2018


Hi Hans-Joachim,

If the file cannot be restored correctly on any client, then it seems that
the file data is corrupt. Since individual blocks are encrypted separately,
it seems that one block was probably corrupted before being encrypted,
perhaps due to bad memory or a cosmic ray bit flip on the client being
backed up. (It should not be possible that it was damaged after encryption,
including on the server, as a different error would occur on decryption).

Do you have any old versions of this file on the store, and can they be
restored successfully?

This should be detected by verifying (comparing) the backup on the original
client, which is something that you should really be doing regularly to
check the integrity of your backups. It would probably be useful to add a
command to bbackupquery that decrypts the blocks and checks that they can
be decompressed, without comparing to the current file data, as this would
be easier on the box (less file I/O) and could be done on a different box
(with the same key).

The Zlib manual says that it's possible to resync the compression and
partially restore the file in such cases, or at least we could drop the
encrypted block. The file would have errors, but in some cases some content
could be recovered from it. In this case, if this is really a MySQL ISAM
database file as it appears, it could probably be repaired by myisamchk
with a few lost records.

You can delete the file from the server with the bbackupquery delete
command, and try the restore again, and it should succeed this time
(hopefully!).

Thanks, Chris.

On Sun, 30 Dec 2018 at 23:09, Hans-Joachim Baader <hjb at pro-linux.de> wrote:

> Hi Chris,
>
> > Very sorry to hear that you're having problems. I'm not aware of any
> > difference or incompatibility between 32-bit and 64-bit clients and
> > servers. I want to fix this bug, but will need your help to reproduce it.
>
> thank you.
>
> > This particular error can happen when the client encounters an error in a
> > previous operation, and tries to move onto the next file, but the
> protocol
> > is out of sync (the server is still sending file data, but the client
> tries
> > to interpret it as a reply to its next request, and aborts). Therefore it
> > would be useful to see the last few lines of the output, including any
> > error that occurred immediately before, or while restoring the previous
> > file. The server logs may also have some useful information, such as
> > warnings.
> >
> > How large is the file that failed on the server, in blocks? Can you
> restore
> > it on its own (without the other files)? Can you restore the file
> > immediately before, and then perform another protocol operation such as
> > listing files in a directory?
>
> After some tests, I must correct some of my statements. I can reproduce
> the problem with a single file, 26 MB long (maybe the size doesn't have
> anything to do with the problem). Contrary to what I said the behavior
> is the same on all 32 and 64 bit clients.
>
> A simple get command is sufficient to reproduce the error. It also happens
> when the restore command is used. On the server side I see the following
> in the log, and nothing else of interest:
>
> Dec 30 23:38:54 server bbstored client=0x00000016[28613]: WARNING:
> Exception thrown: ConnectionException(TLSWriteFailed) at
> lib/server/SocketStreamTLS.cpp(372)
> Dec 30 23:38:54 server bbstored client=0x00000016[28613]: NOTICE:
> Connection statistics for 0x00000016 (name=): IN=189 OUT=628976
> NET_IN=-628787 TOTAL=629165
> Dec 30 23:38:54 server bbstored[28613]: ERROR: Error in child process,
> terminating connection: TLSWriteFailed (7/33)
>
> These entries only appear in the log after the client disconnects,
> this may be long after the error happens on the client.
>
> I straced the server and didn't see any error there. So the error
> seems to be produced on the client.
>
> The client says:
>
> query > get item.MYI
> WARNING: zlib error code = -3
> WARNING: Exception thrown: CompressException(TransformFailed) at
> ../../lib/compress/Compress.h(154)
> ERROR:   Failed to fetch file: Compress TransformFailed
>
> And after additional commands:
>
> query > ls
> WARNING: Exception thrown: ConnectionException(Conn_Protocol_ObjTooBig) at
> Protocol.cpp(219)
> ERROR:   Failed to list directory: Connection Protocol_ObjTooBig
> query > exit
> WARNING: Exception thrown: ConnectionException(Conn_Protocol_ObjTooBig) at
> Protocol.cpp(219)
> Exception: Connection Protocol_ObjTooBig (7/42)
>
> The same messages are in the syslog. I did a ltrace on the client and got:
> 19104 SSL_read(0x55a6a0643d40, 0x55a6a067079f, 2129, 0xdbba0)
>   = 2129
> 19104 SSL_get_error(0x55a6a0643d40, 2129, 0x55a6a0644410, 0)
>    = 0
> 19104 EVP_CIPHER_CTX_iv_length(0x55a69f0dcba0, 2129, 0x55a69f0dcd60,
> 8608)   = 16
> 19104 EVP_CipherInit(0x55a69f0dcba0, 0, 0, 0x55a6a06707a0)
>    = 1
> 19104 EVP_CipherInit(0x55a69f0dcba0, 0, 0, 0)
>   = 1
> 19104 EVP_CIPHER_CTX_block_size(0x55a69f0dcba0, 2048, 16, 0)
>    = 16
> 19104 inflateInit_(0x7fffa6d0ba60, 0x55a69eeb90be, 112, 0)
>    = 0
> 19104 EVP_CIPHER_CTX_block_size(0x55a69f0dcba0, 0x7fffa6d0bf60, 2048,
> 0x55a6a06707b0) = 16
> 19104 EVP_CipherUpdate(0x55a69f0dcba0, 0x7fffa6d0bf60, 0x7fffa6d0b55c,
> 0x55a6a06707b0) = 1
> 19104 inflate(0x7fffa6d0ba60, 0, 0, 0)
>    = 0xfffffffd
>
> So, zlib has indeed a problem. What next? The data store on the server
> seems Ok, I checked that. I could test again with more verbose output.
> Or delete the file in the store to have it backed up again.
>
> Zlib is version 1.2.8 or 1.2.6.
>
> Regards,
> Hans-Joachim
>
> Hans-Joachim Baader - Pro-Linux.de
> Poppelfeld 9, D-76646 Bruchsal
> Geschäftsführer/Herausgeber * CEO/Editor
> Fon +49 07257-930142 * Mobil 0170-6500228
> --
> Pro-Linux - Germany's largest volunteer Linux support site
> Support Pro-Linux: https://www.pro-linux.de/user/index50.html
> https://www.pro-linux.de/    Public Key ID 2DBC923E3DDBDDEA
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.boxbackup.org/pipermail/boxbackup/attachments/20181231/d7cd4a3e/attachment-0001.html>


More information about the Boxbackup mailing list