From chris at qwirx.com  Wed Feb  1 09:44:47 2012
From: chris at qwirx.com (Chris Wilson)
Date: Wed, 1 Feb 2012 09:44:47 +0000 (GMT)
Subject: [Box Backup] No housekeeping?
In-Reply-To: <CAPZkA7ep-p2G+wDGwLQwOgWtVCrQbRt2aLh2SotxPtR0u0pWGg@mail.gmail.com>
References: <CAPZkA7da-M8eXmKc2BoVnr7a5TuoV651fes7J_66b2Oo+=d2jw@mail.gmail.com>
	<1327339595.6225.44.camel@crusty.backed-up.net>
	<alpine.DEB.2.00.1201232109050.5062@chris-desktop.fen.aptivate.org>
	<CAPZkA7ep-p2G+wDGwLQwOgWtVCrQbRt2aLh2SotxPtR0u0pWGg@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1202010941520.5062@chris-desktop.fen.aptivate.org>

Hi Peter,

On Sun, 29 Jan 2012, Peter Hall wrote:

> I was unable to use -o, but I ran "bbstored -DV" and have attached the log of "grep bbstored /var/log/messages".
> 
> Strangely enough, you can see in the log that housekeeping seems to be alive and well, and have continously deleted items up until now, but I can't see any difference in used space on the store:
> # bbstoreaccounts info 1
> ????????? Account ID: 0x00000001
> ????? Last object ID: 0x1106496
> ??????????????? Used:? 243199227 blocks,? 927.73 GB,? 99% |*************** |
> ?????????? Old files:??? 3849197 blocks,?? 14.68 GB,?? 1% |??????????????? |
> ?????? Deleted files:? 127779108 blocks,? 487.44 GB,? 52% |********??????? |
> ???????? Directories:???? 451080 blocks,??? 1.72 GB,?? 0% |??????????????? |
> ????????? Soft limit:? 230400000 blocks,? 878.91 GB,? 94% |*************** |
> ????????? Hard limit:? 243200000 blocks,? 927.73 GB, 100% |****************|
> ?Client store marker: 17851542
> 
> (I have terminated the client while try to get to the bottom of this problem, so it't not filling up the store.)
> 
> I know that there is a very large number of small files in the store, 
> and some of those large collections are possibly deleted. Could bbstored 
> be ineffient in deleting large amount of files (rough estimate: 
> 50000-100000 in one directory)?

It might be. But I did notice that housekeeping was running for three days 
without finishing in your log. The block counts are not updated while 
housekeeping is running, only when it finishes. So is it possible that 
housekeeping did actually finish and remove enough files to bring the 
store back under the soft limit, and you didn't notice that the block 
counts have finally been reduced?

> ERROR:?? Housekeeping on account 0x00000001 found and fixed wrong block 
> counts: used (243199923,243199227), old (3849882,3849197), deleted 
> (127779749,127779108), dirs (451091,451080)

That's unusual, but without a complete and detailed log I wouldn't be able 
to diagnose how that happened. In any case the differences are small, so 
it's likely just not accounting for a deletion properly, which is a minor 
bug.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <chris+sig at qwirx.com> Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer |
\__/_/_/_//_/___/ | We are GNU : free your mind & your software |

From peter at preacher.se  Wed Feb  1 21:17:50 2012
From: peter at preacher.se (Peter Hall)
Date: Wed, 1 Feb 2012 22:17:50 +0100
Subject: [Box Backup] No housekeeping?
In-Reply-To: <alpine.DEB.2.00.1202010941520.5062@chris-desktop.fen.aptivate.org>
References: <CAPZkA7da-M8eXmKc2BoVnr7a5TuoV651fes7J_66b2Oo+=d2jw@mail.gmail.com>
	<1327339595.6225.44.camel@crusty.backed-up.net>
	<alpine.DEB.2.00.1201232109050.5062@chris-desktop.fen.aptivate.org>
	<CAPZkA7ep-p2G+wDGwLQwOgWtVCrQbRt2aLh2SotxPtR0u0pWGg@mail.gmail.com>
	<alpine.DEB.2.00.1202010941520.5062@chris-desktop.fen.aptivate.org>
Message-ID: <CAPZkA7c1n+vg+mf+mrNjHEd_hfTEp5F6LFYpyJCjt26Ck=bOhA@mail.gmail.com>

Thanks for the reply,

2012/2/1 Chris Wilson <chris at qwirx.com>

> Hi Peter,
>
>
> On Sun, 29 Jan 2012, Peter Hall wrote::    3849197 blocks,   14.68 GB,
> 1% |                |>z
>
>>        Deleted files:  127779108 blocks,  487.44 GB,  52%
>> |********        |
>>          Directories:     451080 blocks,    1.72 GB,   0%
>> |                |
>>           Soft limit:  230400000 blocks,  878.91 GB,  94%
>> |*************** |
>>           Hard limit:  243200000 blocks,  927.73 GB, 100%
>> |****************|
>>  Client store marker: 17851542
>>
>> (I have terminated the client while try to get to the bottom of this
>> problem, so it't not filling up the store.)
>>
>> I know that there is a very large number of small files in the store, and
>> some of those large collections are possibly deleted. Could bbstored be
>> ineffient in deleting large amount of files (rough estimate: 50000-100000
>> in one directory)?
>>
>
> It might be. But I did notice that housekeeping was running for three days
> without finishing in your log. The block counts are not updated while
> housekeeping is running, only when it finishes. So is it possible that
> housekeeping did actually finish and remove enough files to bring the store
> back under the soft limit, and you didn't notice that the block counts have
> finally been reduced?
>
>
>  ERROR:   Housekeeping on account 0x00000001 found and fixed wrong block
>> counts: used (243199923,243199227), old (3849882,3849197), deleted
>> (127779749,127779108), dirs (451091,451080)
>>
>
> That's unusual, but without a complete and detailed log I wouldn't be able
> to diagnose how that happened. In any case the differences are small, so
> it's likely just not accounting for a deletion properly, which is a minor
> bug.
>
> Cheers, Chris.
>
> --
> _____ __     _
> \  __/ / ,__(_)_  | Chris Wilson <chris+sig at qwirx.com> Cambs UK |
> / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/**SQL Developer |
> \__/_/_/_//_/___/ | We are GNU : free your mind & your software |
>
> _______________________________________________
> Boxbackup mailing list
> Boxbackup at boxbackup.org
> http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.boxbackup.org/pipermail/boxbackup/attachments/20120201/2e7c3d16/attachment.htm>

From peter at preacher.se  Wed Feb  1 21:25:41 2012
From: peter at preacher.se (Peter Hall)
Date: Wed, 1 Feb 2012 22:25:41 +0100
Subject: [Box Backup] No housekeeping?
In-Reply-To: <alpine.DEB.2.00.1202010941520.5062@chris-desktop.fen.aptivate.org>
References: <CAPZkA7da-M8eXmKc2BoVnr7a5TuoV651fes7J_66b2Oo+=d2jw@mail.gmail.com>
	<1327339595.6225.44.camel@crusty.backed-up.net>
	<alpine.DEB.2.00.1201232109050.5062@chris-desktop.fen.aptivate.org>
	<CAPZkA7ep-p2G+wDGwLQwOgWtVCrQbRt2aLh2SotxPtR0u0pWGg@mail.gmail.com>
	<alpine.DEB.2.00.1202010941520.5062@chris-desktop.fen.aptivate.org>
Message-ID: <CAPZkA7dJeHRiPakCj89FkNG8bZF8a08QnfzrWUWb2PZDEucYLg@mail.gmail.com>

Thanks for the reply,
(and sorry for the previous unfinished email, be wary of cats jumping in
close proximity to your keyboard...)

2012/2/1 Chris Wilson <chris at qwirx.com>

> Hi Peter,
>
>
> On Sun, 29 Jan 2012, Peter Hall wrote:
>
> It might be. But I did notice that housekeeping was running for three days
> without finishing in your log. The block counts are not updated while
> housekeeping is running, only when it finishes. So is it possible that
> housekeeping did actually finish and remove enough files to bring the store
> back under the soft limit, and you didn't notice that the block counts have
> finally been reduced?


That sounds plausible. I just checked the server and housekeeping is still
running. Same kind of messages in the log as the one i previously attached.
One file is removed every couple of minutes, which could take a very long
while if there are tens or even hundreds of thousands of files to be
deleted.

Unfortunately I am now stuck (admittedly self inflicted) with a full store
and unable to upload new backups until housekeeping finishes.

Would it be possible to change future versions to update the block count
while running housekeeping? I see very high memory usage as well which I
guess is due to boxbackup having to keep track of deleted files, or
something similar?

Best regards,
Peter


>
> Cheers, Chris.
>
> --
> _____ __     _
> \  __/ / ,__(_)_  | Chris Wilson <chris+sig at qwirx.com> Cambs UK |
> / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/**SQL Developer |
> \__/_/_/_//_/___/ | We are GNU : free your mind & your software |
>
> _______________________________________________
> Boxbackup mailing list
> Boxbackup at boxbackup.org
> http://lists.boxbackup.org/cgi-bin/mailman/listinfo/boxbackup
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.boxbackup.org/pipermail/boxbackup/attachments/20120201/856b262b/attachment.htm>

From achim+box at qustodium.net  Sat Feb 11 15:02:39 2012
From: achim+box at qustodium.net (Achim J. Latz)
Date: Sat, 11 Feb 2012 16:02:39 +0100
Subject: [Box Backup] Regex exclusions
In-Reply-To: <alpine.DEB.2.00.1112032307440.3053@chris-desktop.fen.aptivate.org>
References: <f5f03a8807ea5d068f6b36d690ab0de3@localhost>
	<4E846D7F.1070507@qustodium.net>
	<CAL5zbj_N5d2tQMyZMGo4gTiB=2T1_xzvc4JMwDq=pNYNXrvRLw@mail.gmail.com>
	<4EC50F77.30006@qustodium.net>
	<alpine.DEB.2.00.1112032307440.3053@chris-desktop.fen.aptivate.org>
Message-ID: <4F36830F.5030801@qustodium.net>

Hello Chris, and happy 2012 to the list (it *is* still January ;-)

On 04/12/2011 00:12, Chris Wilson wrote:
> On Thu, 17 Nov 2011, Achim J. Latz wrote:
>
>> On 29/09/2011 18:50, Peter Jalajas, GigaLock Backup Services wrote:
>>
>>> A few planning or philosophical topics we should consider: 1. Windows
>>> Exclusions are not case-sensitive, so we no longer have to add both
>>> upper and lower case versions of each Exclusion--yeah!
>>
>> I would assume after reading Chris' comments that this is true for all
>> exclusions, not Windows only.
>
> No, it's only true on Windows. It was also a bit broken until now,
> because it would also convert all characters in the regular expression
> to lower case, including \S to \s for example. I've just fixed that.
>
> The case-insensitive comparison flag is now used when compiling the
> regexes, on Windows only.

Thank you for fixing something that at least I didn't know was even 
broken (-:

>>> (I wish we could push some common Exclusion strings into variables of
>>> some sort that different BackupLocations could "import". Exclusions
>>> are fraught with typographic errors.)
>>
>> Chris, would this perhaps be possible by adding additional "keys" in
>> bbackupd.conf like this (not very elegant, I know):
>>
>> ExcludeTemplate_1, ExcludeTemplate_2, [..], ExcludeTemplate_n
>> IncludeTemplate_1, IncludeTemplate_2, [..], IncludeTemplate_n
>>
>> that can be used within the regular ExcludeFilesRegex etc. like this:
>>
>> ExcludeFilesRegex = IncludeTemplate_2|IncludeTemplate_42|[...]
>
> It's possible, but quite a bit of work. How about global Include and
> Exclude options instead?

That would make configuration a lot easier, I think. Global Include and 
Exclude options that may be overwritten by per-location ones. Now the 
interesting question is of course: what is the order of precedence etc. 
Do you have an idea how that could be implemented with minimal effort?

>> Some other ideas can be found here
>> <https://crashplan.zendesk.com/entries/302133-improve-ui-for-backup-exclusions>,
>> particularly the part of defining an exclusion from a given file
>> pattern would be great for Boxi (although not a priority, as >90% of
>> the users will never touch exclusions).
>
> What do you mean by a file pattern? Is that a regex applied to the file
> name?

It is basically a simpler way to define excludes, not requiring the user 
to know about how to generate a regex if only he wants to exclude "all 
MP3s" or "all PST files": automatically create a RegEx from a given file 
name in the background, the user only specifies "no mp3". Does that make 
sense?

Best regards, Achim

-- 
Achim J. Latz, Qustodium Internet Security
achim.latz at qustodium.net ? http://www.qustodium.net
Data Encryption ? Backup Automatisation ? E-Mail Protection

From achim+box at qustodium.net  Sat Feb 11 15:10:03 2012
From: achim+box at qustodium.net (Achim J. Latz)
Date: Sat, 11 Feb 2012 16:10:03 +0100
Subject: [Box Backup] Adaptive MaxUploadRate, Quality of Service
Message-ID: <4F3684CB.4070203@qustodium.net>

Hello list:

I have been wondering about MaxUploadRate and its implications:

At the moment, at "install" time, I time a simple wget POST request for 
a 500kb upload to determine the current "bandwidth" available for a 
particular machine, and set MaxUploadRate to 50% of that value.

Ideally, I would like to determine the existing bandwidth not only once 
at install time, but dynamically every time (at least) bbackupd starts a 
backup process. For instance, think about a laptop user that uses the 
same machine in the office (10MB+ SDSL or fibre), home (ADSL) and 3G: 
three rather different upload rate scenarios.

SyncAllowScript seems like the natural place to do this (via the same 
timed wget POST request), but unfortunately updating MaxUploadRate has 
no effect on the currently running, about-to-start-backing-up bbackupd 
instance.

Any ideas how to get around this? In fact, would it be possible to 
"internalise" that kind of "background intelligent transfer system" or 
use existing frameworks depending on the OS, such as Microsoft's [1]?

Have a great weekend, Achim

[1] 
<http://msdn.microsoft.com/en-us/library/windows/desktop/bb968799%28v=vs.85%29.aspx>

-- 
Achim J. Latz, Qustodium Internet Security
achim.latz at qustodium.net ? http://www.qustodium.net
Data Encryption ? Backup Automatisation ? E-Mail Protection

From achim+box at qustodium.net  Sat Feb 11 15:31:11 2012
From: achim+box at qustodium.net (Achim J. Latz)
Date: Sat, 11 Feb 2012 16:31:11 +0100
Subject: [Box Backup] VSS on x64 not working with x86 binaries
In-Reply-To: <4EA57951.5030903@invis.net>
References: <f4aa68c8783e6ad4b5bd0fc9860c91d7@localhost>	<alpine.DEB.2.00.1110182050240.3053@chris-desktop.fen.aptivate.org>	<0942d6dc17bcbd1fb38e88a987e7c3bc@localhost>	<alpine.DEB.2.00.1110231939000.3053@chris-desktop.fen.aptivate.org>	<e041181e03e7919104c954231622799f@localhost>
	<4EA57951.5030903@invis.net>
Message-ID: <4F3689BF.7060908@qustodium.net>

Hello Charles (repost from an older message):

On 24/10/2011 16:42, Charles Lecklider wrote:
> On 24/10/2011 11:38, Achim wrote:
>> In addition, there are quite a few warnings generated by compiling for
>> an x64 target, I am not sure which ones are important and which ones can
>> be ignored. I include the build log below for reference.
>
> There are lots of problems compiling trunk for Windows x64, many of
> which can cause data loss. The root cause of almost all of the problems
> is that sizeof(int) == sizeof(long) == 4 on both Windows x86 and x64,
> but BB assumes that sizeof(long) == 8 (and in places int too) for x64.
>
> I spent a lot of time sorting this out in my branch, and while in some
> cases it's still possible to get into trouble with really huge files, in
> practical terms it'll never happen as the diffing will take too long for
> BB to be useful.
>
> I strongly recommend that you do not use trunk compiled for Windows x64
> for anything but testing.

Any chance to merge your branch and trunk to get the "best of both 
worlds", i.e. VSS, 4GB patch, and at least a basic level of 64bit 
compatibility?

Or is there anything I (as a non-pro programmer) can do to identify the 
issues that BB-on-x64 could cause? As you mention:

 > The root cause of almost all of the problems
 > is that sizeof(int) == sizeof(long) == 4 on both Windows x86 and x64,
 > but BB assumes that sizeof(long) == 8 (and in places int too) for x64.

As x64 systems (and soon WOA, Windows-on-Arm) seem to become the norm in 
that universe, perhaps we should future-proof BB now?

Best regards, Achim

-- 
Achim J. Latz, Qustodium Internet Security
achim.latz at qustodium.net ? http://www.qustodium.net
Data Encryption ? Backup Automatisation ? E-Mail Protection

From chris at qwirx.com  Sat Feb 11 15:54:38 2012
From: chris at qwirx.com (Chris Wilson)
Date: Sat, 11 Feb 2012 15:54:38 +0000 (GMT)
Subject: [Box Backup] Adaptive MaxUploadRate, Quality of Service
In-Reply-To: <4F3684CB.4070203@qustodium.net>
References: <4F3684CB.4070203@qustodium.net>
Message-ID: <alpine.DEB.2.00.1202111544280.5062@chris-desktop.fen.aptivate.org>

Hi Achim,

On Sat, 11 Feb 2012, Achim J. Latz wrote:

> I have been wondering about MaxUploadRate and its implications:
>
> At the moment, at "install" time, I time a simple wget POST request for 
> a 500kb upload to determine the current "bandwidth" available for a 
> particular machine, and set MaxUploadRate to 50% of that value.

The problem is that the value you get will vary massively, between 0% and 
100% of your line capacity, depending on factors outside your control such 
as how many other people are watching youtube/news/audio streams/making 
backups at the same time.

I implemented this upload rate limit reluctantly, because I think it's 
best done on the router, but I'm aware that almost nobody has a router 
capable of doing real QoS or the network wizard skills needed to configure 
it.

> Ideally, I would like to determine the existing bandwidth not only once 
> at install time, but dynamically every time (at least) bbackupd starts a 
> backup process. For instance, think about a laptop user that uses the 
> same machine in the office (10MB+ SDSL or fibre), home (ADSL) and 3G: 
> three rather different upload rate scenarios.
>
> SyncAllowScript seems like the natural place to do this (via the same 
> timed wget POST request), but unfortunately updating MaxUploadRate has 
> no effect on the currently running, about-to-start-backing-up bbackupd 
> instance.

I could add an optional second parameter output from the SyncAllowScript 
that, if present, overrides the current MaxUploadRate.

> Any ideas how to get around this? In fact, would it be possible to 
> "internalise" that kind of "background intelligent transfer system" or 
> use existing frameworks depending on the OS, such as Microsoft's [1]?

I'm not going to implement BITS support in Box. Someone else might, but I 
really doubt it will work because it's designed for a completely different 
goal.

I might implement tcp-nice or something like it that's cross platform, but 
only if I can find a way to do it entirely in user space.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <chris+sig at qwirx.com> Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer |
\__/_/_/_//_/___/ | We are GNU : free your mind & your software |

From chris at qwirx.com  Sun Feb 12 14:39:08 2012
From: chris at qwirx.com (Chris Wilson)
Date: Sun, 12 Feb 2012 14:39:08 +0000 (GMT)
Subject: [Box Backup] Questions about the "Background Transfer Service" paper
Message-ID: <alpine.DEB.2.02.1202121437370.15771@lap-x201>

Dear sirs,

I read with great interest your paper on implementing background TCP 
transfers that do not affect foreground flows, entirely in user space 
<http://www.thlab.net/~lmassoul/p18-key.pdf>. Thank you for this excellent 
contribution, and for making the paper freely available.

I'm a developer of an open source online backup application for Windows 
and Unix, Box Backup <http://www.boxbackup.org> and one of our users 
recently requested a feature to allow the client to adapt automatically to 
the available bandwidth without affecting foreground flows 
<http://lists.boxbackup.org/pipermail/boxbackup/2012-February/006319.html>.

I had previously read about TCP-Nice and was waiting for it to land in 
mainstream kernels. Using your paper I was able to write a background flow 
manager 
<http://www.boxbackup.org/browser/box/trunk/lib/server/TcpNice.cpp>.

However it does not perform quite as I had hoped :) I'm hoping that you 
will be interested enough in a real-world implementation to give me some 
advice about making it work properly.

The main symptom that I'm seeing is that the window size quickly grows to 
about 3 MB and remains there, with positive and negative adjustments. You 
can find some example data attached. This is data logged by the algorithm 
at the end of each control period when calculating a new window size for 
the next control period. Let me explain the columns:

* b is the number of bytes sent during the previous control period
* T is the length (in us) of the previous control period
* rtt is the round trip time (in us) reported by the kernel on the socket
* e is epsilon, a parameter of the formula, calculated as alpha/rtt
* rb is the actual rate (goodput) over the previous period
* rbhat is the previous (last-but-one) EWMA rate estimate
* raw is the unscaled adjustment to the window size
* gamma is the scaled adjustment to the window size
* wb is the final window size

Initially in my experiments I found that the window size would grow 
without bound. I think this was because I was using the algorithm 
continuously, but it's only valid when the flow rate is limited by delays 
or packet drops. The backup client is often bursty, when comparing 
directory listings or computing diffs in files, so I switched to only 
using (enabling) the algorithm during file uploads. It has no effect on
uploads of less than 1 second. This explains the time gaps in the data.

In my implementation the control is applied by the sender (backup client), 
for two reasons: the sender has a better estimate of the RTT in the 
TCP engine, because it constantly receives acks from the receiver; and in 
the case that the OS does not allow retrieving the RTT, it can be set 
manually on the client, and varies from client to client, depending on 
distance from the server.

QUESTIONS:

Does controlling the sender side (with SO_SNDBUF) instead of the receiver 
side (with SO_RCVBUF -> TCP window size) make sense?

Does switching the algorithm on and off invalidate it? I can see that the 
window size has reached 2.6 MB before the first time that the flow rate 
(rb) drops below the average, resulting in the first negative correction 
to the window size. Perhaps the rapid changes in flow mode from bursty to 
bulk require more time to stabilise before making the first window size 
adjustment? Or the control interval T is too long at 1 second?

I'm using the TCP RTT as tau-B, the propagation delay. However it occurs 
to me that this includes queueing delays, and perhaps should not. Perhaps 
it's better to use the minimum RTT seen thus far, or a value supplied by 
the user for a connection not under load?

I wasn't sure how to set the alpha-star parameter. The paper says it 
should be less than 1, just after equation (10). If I set it to 0.1, and 
the RTT is also 0.1 seconds (typical) then epsilon is 0.1/0.1 = 1 byte, 
which is not a very large window size adjustment. When the RTT goes over 
200ms it becomes 0. At these values I don't see the point of including 
epsilon in the window size at all. What am I missing?

Am I right that the algorithm doesn't use the RTT value or tau-B anywhere 
else except in the calculation of epsilon?

Why am I seeing the window size increase to silly values? Should I cap it 
at 2*rbhat*minrtt (twice the value required for efficient use of the 
connection) to ensure fast adjustment to any change in conditions (sudden 
appearance of another flow)?

Thanks in advance for your advice,

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <chris+sig at qwirx.com> Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer |
\__/_/_/_//_/___/ | We are GNU : free your mind & your software |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tcpnice.data.txt.gz
Type: application/octet-stream
Size: 3041 bytes
Desc: 
URL: <http://lists.boxbackup.org/pipermail/boxbackup/attachments/20120212/85a38fa9/attachment.obj>