From boxbackup-dev at fluffy.co.uk  Thu Mar  8 23:39:29 2007
From: boxbackup-dev at fluffy.co.uk (Chris Wilson)
Date: Thu, 8 Mar 2007 23:39:29 +0000 (GMT)
Subject: [Box Backup-dev] Call for help
Message-ID: <Pine.LNX.4.62.0703082323530.1528@top.qwarx.com>

Hi all,

There are a number of win32-specific changes to the core Box Backup code 
which have been sitting in my General and Merge trees for quite some time.

I apologise for allowing the situation to degenerate this far. I am very 
keen to see these changes either merged into to trunk, or rejected and 
removed from the General branch (win32 compatibility). Ultimately, my aim 
is that the trunk should be the best source for clients that run well on 
Windows and all Unixes.

I felt, and still feel, that it is useful for me to have a branch of my 
own to maintain the Win32 port, since this makes its maintenance, bug 
fixes and introduction of new features much easier. However, I would 
really like to see these changes merged into the trunk as soon as 
possible.

Martin Ebourne has very kindly reviewed a great number of these changes 
and either approved them for merge, or asked me to go back and rewrite 
them in order to fix bugs and to avoid polluting the trunk with any bad 
code. However, I gather that he is now too busy to do so any more. I would 
like to thank Martin for the considerable time and effort he has expended, 
and all his comments which have been extremely valuable and helpful.

I appeal urgently to anyone who would like to see better compatibility 
with Win32 in the trunk to step forward and offer to review my patches. 
You can see them at [http://bbdev.fluffy.co.uk/trac/ticket/3#comment:209] 
and following comments on this bug ticket.

Please note that Ben Summers is the ultimate arbiter of what makes it into 
the trunk, and he will need to approve your candidacy before I can take 
your "ok" to mean that I can merge a change into the trunk. However, 
comments about any patch are very much appreciated and I am very happy to 
do anything I can to satisfy any concerns that you may have.

Thank you very much for your support!

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup-dev at fluffy.co.uk  Fri Mar  9 14:32:38 2007
From: boxbackup-dev at fluffy.co.uk (E.W. Peter Jalajas)
Date: Fri, 9 Mar 2007 06:32:38 -0800 (PST)
Subject: [Box Backup-dev] Call for help
In-Reply-To: <Pine.LNX.4.62.0703082323530.1528@top.qwarx.com>
Message-ID: <335460.63516.qm@web60618.mail.yahoo.com>

Hi Chris,

As you know, I'm no developer, but let me know how I can help.

Thanks for all your amazing hard work!
Pete



From boxbackup-dev at fluffy.co.uk  Fri Mar  9 16:02:55 2007
From: boxbackup-dev at fluffy.co.uk (Charles Lecklider)
Date: Fri, 09 Mar 2007 16:02:55 +0000
Subject: [Box Backup-dev] Call for help
In-Reply-To: <Pine.LNX.4.62.0703082323530.1528@top.qwarx.com>
References: <Pine.LNX.4.62.0703082323530.1528@top.qwarx.com>
Message-ID: <45F1852F.6000805@invis.net>

Chris Wilson wrote:
> I appeal urgently to anyone who would like to see better compatibility
> with Win32 in the trunk to step forward and offer to review my patches.
> You can see them at
> [http://bbdev.fluffy.co.uk/trac/ticket/3#comment:209] and following
> comments on this bug ticket.

OK, how do I add a comment to a changeset? Or shall I just deal with it
on-list?

-C



From boxbackup-dev at fluffy.co.uk  Fri Mar  9 16:26:11 2007
From: boxbackup-dev at fluffy.co.uk (Martin Ebourne)
Date: Fri, 09 Mar 2007 16:26:11 +0000
Subject: [Box Backup-dev] Call for help
In-Reply-To: <45F1852F.6000805@invis.net>
References: <Pine.LNX.4.62.0703082323530.1528@top.qwarx.com>
 <45F1852F.6000805@invis.net>
Message-ID: <20070309162611.55rwn4hg8cscc0os@ebourne.me.uk>

Charles Lecklider <boxbackup at invis.net> wrote:

> Chris Wilson wrote:
>> I appeal urgently to anyone who would like to see better compatibility
>> with Win32 in the trunk to step forward and offer to review my patches.
>> You can see them at
>> [http://bbdev.fluffy.co.uk/trac/ticket/3#comment:209] and following
>> comments on this bug ticket.
>
> OK, how do I add a comment to a changeset? Or shall I just deal with it
> on-list?

Just add comments at the end of the ticket (ticket #3) in a similar  
manner to how we've been doing before. They are automatically sent to  
the list so we'll all see them anyway.

Cheers,

Martin.


From boxbackup-dev at fluffy.co.uk  Sat Mar 10 22:33:36 2007
From: boxbackup-dev at fluffy.co.uk (Charles Lecklider)
Date: Sat, 10 Mar 2007 22:33:36 +0000
Subject: [Box Backup-dev] Call for help
In-Reply-To: <20070309162611.55rwn4hg8cscc0os@ebourne.me.uk>
References: <Pine.LNX.4.62.0703082323530.1528@top.qwarx.com>	<45F1852F.6000805@invis.net> <20070309162611.55rwn4hg8cscc0os@ebourne.me.uk>
Message-ID: <45F33240.1020203@invis.net>

Martin Ebourne wrote:
> Just add comments at the end of the ticket (ticket #3) in a similar
> manner to how we've been doing before. They are automatically sent to
> the list so we'll all see them anyway.

OK, done that for 1083. I'll work through the rest as I regain the will
to deal with trac. (Gripe: can't submit my change because someone had
updated the list before me - wtf?).

-C



From boxbackup-dev at fluffy.co.uk  Sun Mar 11 00:58:04 2007
From: boxbackup-dev at fluffy.co.uk (Chris Wilson)
Date: Sun, 11 Mar 2007 00:58:04 +0000 (GMT)
Subject: [Box Backup-dev] Call for help
In-Reply-To: <45F33240.1020203@invis.net>
References: <Pine.LNX.4.62.0703082323530.1528@top.qwarx.com> <45F1852F.6000805@invis.net>
 <20070309162611.55rwn4hg8cscc0os@ebourne.me.uk> <45F33240.1020203@invis.net>
Message-ID: <Pine.LNX.4.64.0703110056440.8514@top.qwarx.com>

Hi Charles,

On Sat, 10 Mar 2007, Charles Lecklider wrote:

> OK, done that for 1083. I'll work through the rest as I regain the will 
> to deal with trac. (Gripe: can't submit my change because someone had 
> updated the list before me - wtf?).

Sorry, that could happen if I submitted a patch with (refs #3) while you 
were writing your ticket. It annoys me as well. Sorry for any 
inconvenience, and thanks for reviewing!

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup-dev at fluffy.co.uk  Sun Mar 11 11:58:59 2007
From: boxbackup-dev at fluffy.co.uk (Martin Ebourne)
Date: Sun, 11 Mar 2007 11:58:59 +0000
Subject: [Box Backup-dev] Call for help
In-Reply-To: <Pine.LNX.4.64.0703110056440.8514@top.qwarx.com>
References: <Pine.LNX.4.62.0703082323530.1528@top.qwarx.com>
 <45F1852F.6000805@invis.net>
 <20070309162611.55rwn4hg8cscc0os@ebourne.me.uk>
 <45F33240.1020203@invis.net>
 <Pine.LNX.4.64.0703110056440.8514@top.qwarx.com>
Message-ID: <1173614340.3845.1.camel@avenin.ebourne.me.uk>

On Sun, 2007-03-11 at 00:58 +0000, Chris Wilson wrote:
> Hi Charles,
> 
> On Sat, 10 Mar 2007, Charles Lecklider wrote:
> 
> > OK, done that for 1083. I'll work through the rest as I regain the will 
> > to deal with trac. (Gripe: can't submit my change because someone had 
> > updated the list before me - wtf?).
> 
> Sorry, that could happen if I submitted a patch with (refs #3) while you 
> were writing your ticket. It annoys me as well. Sorry for any 
> inconvenience, and thanks for reviewing!

Have they still not fixed that yet? Sigh.

My strategies for dealing with that are:
1) Try not to review stuff while Chris is making changes
2) Copy the comment to clipboard before submit to make it easy to
resubmit if it clashes

Cheers,

Martin.



From boxbackup-dev at fluffy.co.uk  Tue Mar 13 15:40:18 2007
From: boxbackup-dev at fluffy.co.uk (G.)
Date: Tue, 13 Mar 2007 08:40:18 -0700 (PDT)
Subject: [Box Backup-dev] Call for help
Message-ID: <422903.1790.qm@web36713.mail.mud.yahoo.com>

Chris,=0A=0AI run your Win32 port (although built using Visual Studio inste=
ad of MinGW) as my primary backup platform, so I will be happy to help with=
 any build testing, review, etc. Drop me a line how I can help out.=0A=0AGa=
ry=0A=0A=0A=0A=0A =0A______________________________________________________=
______________________________=0ANo need to miss a message. Get email on-th=
e-go =0Awith Yahoo! Mail for Mobile. Get started.=0Ahttp://mobile.yahoo.com=
/mail 


From boxbackup-dev at fluffy.co.uk  Tue Mar 13 20:37:51 2007
From: boxbackup-dev at fluffy.co.uk (Chris Wilson)
Date: Tue, 13 Mar 2007 20:37:51 +0000 (GMT)
Subject: [Box Backup-dev] Call for help
In-Reply-To: <422903.1790.qm@web36713.mail.mud.yahoo.com>
References: <422903.1790.qm@web36713.mail.mud.yahoo.com>
Message-ID: <Pine.LNX.4.64.0703132036280.1286@top.qwarx.com>

Hi Gary,

> I run your Win32 port (although built using Visual Studio instead of 
> MinGW) as my primary backup platform, so I will be happy to help with 
> any build testing, review, etc. Drop me a line how I can help out.

If you could help out by reviewing my patches, it would be greatly 
appreciated. Basically just look at the patch link that I posted, see 
which ones are outstanding (not yet reviewed by Martin or Charles) and add 
your comments to the task tracker, saying if you think the patch is OK, or 
if not, what changes are required.

Thanks for your offer!

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup-dev at fluffy.co.uk  Wed Mar 14 21:50:28 2007
From: boxbackup-dev at fluffy.co.uk (Andrei)
Date: Wed, 14 Mar 2007 23:50:28 +0200
Subject: [Box Backup-dev] Idea for compare
Message-ID: <bab9f1d50703141450q5b95a6b2x7c5453a520512626@mail.gmail.com>

Is this a viable idea?

The server computes the hashes of the encrypted files and sends this
to the client. The client encrypts every file and computes the hashes
of the (locally) encrypted files. In this way the actual integrity of
the files on the server is checked and very little information is
transfered.

This may be useful if the internet connection is very slow and the
local computer has enough cpu to spare to be able to encrypt and
compute the checksum for every file.

Regards


From boxbackup-dev at fluffy.co.uk  Wed Mar 14 22:26:50 2007
From: boxbackup-dev at fluffy.co.uk (Chris Wilson)
Date: Wed, 14 Mar 2007 22:26:50 +0000 (GMT)
Subject: [Box Backup-dev] Idea for compare
In-Reply-To: <bab9f1d50703141450q5b95a6b2x7c5453a520512626@mail.gmail.com>
References: <bab9f1d50703141450q5b95a6b2x7c5453a520512626@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0703142226020.15294@top.qwarx.com>

Hi Andrei,

On Wed, 14 Mar 2007, Andrei wrote:

> Is this a viable idea?
>
> The server computes the hashes of the encrypted files and sends this
> to the client. The client encrypts every file and computes the hashes
> of the (locally) encrypted files. In this way the actual integrity of
> the files on the server is checked and very little information is
> transfered.
>
> This may be useful if the internet connection is very slow and the
> local computer has enough cpu to spare to be able to encrypt and
> compute the checksum for every file.

Yes it is viable (with some small modifications, because the encrypted 
data is not always the same, because of initialisation vectors). It is 
already on our feature requests list.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup-dev at fluffy.co.uk  Sun Mar 25 09:04:43 2007
From: boxbackup-dev at fluffy.co.uk (Martin Ebourne)
Date: Sun, 25 Mar 2007 09:04:43 +0100
Subject: [Box Backup-dev] Re: [Box Backup] Parse Logfiles
In-Reply-To: <Pine.LNX.4.64.0703241637531.8580@top.qwarx.com>
References: <20070322185332.5034fb91@matrix.tuxianer.homelinux.net>
 <Pine.LNX.4.64.0703222359310.8580@top.qwarx.com>
 <20070323224527.218e479a@apfeltasche.neo2k.homeip.net>
 <Pine.LNX.4.64.0703240040310.8580@top.qwarx.com>
 <20070324162556.5751e708@apfeltasche.neo2k.homeip.net>
 <Pine.LNX.4.64.0703241637531.8580@top.qwarx.com>
Message-ID: <1174809883.15103.5.camel@avenin.ebourne.me.uk>

On Sat, 2007-03-24 at 16:45 +0000, Chris Wilson wrote:
> Sorry, I have no idea, nobody has much time to review my changes any more, 
> which is needed before they can be merged.
> 
> In the mean time I'm encouraging people to try my merge tree and let me 
> know how it goes. The more people test it, the easier it will be to 
> persuade the other developers that my changes don't break anything and 
> should be accepted.

>From my point of view I think that's a good plan.

I'm not sure anyone is going to have the time to review the 100s of
changesets Chris has worked on and the worst possible result would be
for them to languish for ever and us never produce another box release.

I think the way forward is probably to merge and test extensively. We've
got plenty of people who will gladly help test and Chris's code quality
has been generally good.

Maybe this would allow us to get a release out in the summer, otherwise
I'm pretty sure we won't have on this year.

Cheers,

Martin.



From boxbackup-dev at fluffy.co.uk  Sun Mar 25 10:29:47 2007
From: boxbackup-dev at fluffy.co.uk (Stefan Norlin)
Date: Sun, 25 Mar 2007 11:29:47 +0200
Subject: [Box Backup-dev] Re: [Box Backup] Parse Logfiles
References: <20070322185332.5034fb91@matrix.tuxianer.homelinux.net> <Pine.LNX.4.64.0703222359310.8580@top.qwarx.com> <20070323224527.218e479a@apfeltasche.neo2k.homeip.net> <Pine.LNX.4.64.0703240040310.8580@top.qwarx.com> <20070324162556.5751e708@apfeltasche.neo2k.homeip.net> <Pine.LNX.4.64.0703241637531.8580@top.qwarx.com> <1174809883.15103.5.camel@avenin.ebourne.me.uk>
Message-ID: <000b01c76ec0$229fe970$e901050a@stenorvaio>

> I'm not sure anyone is going to have the time to review the 100s of
> changesets Chris has worked on and the worst possible result would be
> for them to languish for ever and us never produce another box release.
> 
> I think the way forward is probably to merge and test extensively. We've
> got plenty of people who will gladly help test and Chris's code quality
> has been generally good.
> 
> Maybe this would allow us to get a release out in the summer, otherwise
> I'm pretty sure we won't have on this year.

I totally agree on this one.

Merge, test and then possibly publish a "0.11-beta" that could be easily
downloaded and tested for those interested. Otherwise there is a risk the
whole project halts for an "unforeseeable" period of time.

Regards,
Stefan




From boxbackup-dev at fluffy.co.uk  Sun Mar 25 14:17:19 2007
From: boxbackup-dev at fluffy.co.uk (G.)
Date: Sun, 25 Mar 2007 06:17:19 -0700 (PDT)
Subject: [Box Backup-dev] Re: [Box Backup] Parse Logfiles
Message-ID: <61721.16564.qm@web36707.mail.mud.yahoo.com>

> I think the way forward is probably to merge and test extensively. We've
> got plenty of people who will gladly help test and Chris's code quality
> has been generally good.

Yeah, a "cook-up" release for major milestones and patch reviews only for minor fixes seem to work out well as a release process for many projects, including Linux distributions. I would be happy to run a "bleeding edge" release on real-world backup servers, since I have other redundant backup mechanisms in place in addition to Box.

Gary





 
____________________________________________________________________________________
It's here! Your new message!  
Get new email alerts with the free Yahoo! Toolbar.
http://tools.search.yahoo.com/toolbar/features/mail/


From boxbackup-dev at fluffy.co.uk  Sun Mar 25 23:51:01 2007
From: boxbackup-dev at fluffy.co.uk (Chris Wilson)
Date: Sun, 25 Mar 2007 23:51:01 +0100 (BST)
Subject: [Box Backup-dev] Re: [Box Backup-commit] Re: #3: Merge Win32 branch
In-Reply-To: <060.0aa2af72f8ac97b7448b75dcce34143d@fluffy.co.uk>
References: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk>
 <060.0aa2af72f8ac97b7448b75dcce34143d@fluffy.co.uk>
Message-ID: <Pine.LNX.4.64.0703252308500.8580@top.qwarx.com>

Hi Charles,

On Sun, 25 Mar 2007, trac at fluffy.co.uk wrote:

> [1486] I don't know if I can do this, but I'm going to try :-)
>
> I'd like to veto this change. You seem to have done a partial re- 
> implementation of a completion port - see CreateIoCompletionPort and 
> friends.

Thanks for the suggestion, but I'm afraid that after reading the 
CreateIoCompletionPort docs I'm still not completely sure how I could use 
it to replace [1486].

So far, what I understand (I think) is that the command socket thread, 
instead of calling WaitForMultipleObjects, would call 
GetQueuedCompletionStatus, which blocks until either (i) the main thread 
posts a message to the port with PostQueuedCompletionStatus, or (ii) a 
packet is received over the command socket by the pending overlapped I/O. 
Is that correct?

What bothers me is that MSDN says "After an instance of an open file is 
associated with an I/O completion port, it cannot be used in the 
ReadFileEx or WriteFileEx function."

I assume that this means that I have to use overlapped I/O for both 
reading and writing from the pipe? I don't care about completion of 
writes, although I suppose I could dispatch the next waiting message (if 
any) to the pipe in this case.

And how do I actually read or write on the pipe, if ReadFileEx and 
WriteFileEx are forbidden? Do good old ReadFile and WriteFile work 
properly?

I have a feeling that implementing this will move the WinNamedPipeStream 
API so far away from POSIX that I will have to create a new class that 
doesn't extend IOStream for this purpose. While it might be a cleaner 
solution to the current overlapped/nonoverlapped read branches in the 
code, I'm worried about the duplication of code and possible maintenance 
headaches.

Are there any samples of using CreateIoCompletionPort and 
PostQueuedCompletionStatus that I can freely copy and adapt for use in 
this case?


-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup-dev at fluffy.co.uk  Mon Mar 26 00:44:22 2007
From: boxbackup-dev at fluffy.co.uk (Charles Lecklider)
Date: Mon, 26 Mar 2007 00:44:22 +0100
Subject: [Box Backup-dev] Re: [Box Backup-commit] Re: #3: Merge Win32 branch
In-Reply-To: <Pine.LNX.4.64.0703252308500.8580@top.qwarx.com>
References: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk> <060.0aa2af72f8ac97b7448b75dcce34143d@fluffy.co.uk> <Pine.LNX.4.64.0703252308500.8580@top.qwarx.com>
Message-ID: <46070956.3010206@invis.net>

Chris Wilson wrote:
> Hi Charles,
> 
> On Sun, 25 Mar 2007, trac at fluffy.co.uk wrote:
> 
>> [1486] I don't know if I can do this, but I'm going to try :-)
>>
>> I'd like to veto this change. You seem to have done a partial re-
>> implementation of a completion port - see CreateIoCompletionPort and
>> friends.
> 
> Thanks for the suggestion, but I'm afraid that after reading the
> CreateIoCompletionPort docs I'm still not completely sure how I could
> use it to replace [1486].
> 
> So far, what I understand (I think) is that the command socket thread,
> instead of calling WaitForMultipleObjects, would call
> GetQueuedCompletionStatus, which blocks until either (i) the main thread
> posts a message to the port with PostQueuedCompletionStatus, or (ii) a
> packet is received over the command socket by the pending overlapped
> I/O. Is that correct?

Or until the pipe is closed, yes.

> What bothers me is that MSDN says "After an instance of an open file is
> associated with an I/O completion port, it cannot be used in the
> ReadFileEx or WriteFileEx function."
> 
> I assume that this means that I have to use overlapped I/O for both
> reading and writing from the pipe?

Erm, you have to do that anyway or you'll get some really strange behaviour.

> I don't care about completion of
> writes, although I suppose I could dispatch the next waiting message (if
> any) to the pipe in this case.

Trying to cut corners when using overlapped IO is not a good idea.

> And how do I actually read or write on the pipe, if ReadFileEx and
> WriteFileEx are forbidden? Do good old ReadFile and WriteFile work
> properly?

Yes. You can't use ReadFileEx because you can't use completion routines
with completion ports (kinda makes sense when you think about it).

> I have a feeling that implementing this will move the WinNamedPipeStream
> API so far away from POSIX that I will have to create a new class that
> doesn't extend IOStream for this purpose. While it might be a cleaner
> solution to the current overlapped/nonoverlapped read branches in the
> code, I'm worried about the duplication of code and possible maintenance
> headaches.

Well, I'm still trying to work out what you're hoping to achieve; the
change talks about deadlock, but this is a pipe - that can't happen if
you always check for errors.

Obviously there needs to be a thread to deal with commands, but if
there's only one thread there's no harm in using normal boring
synchronous IO. The actual backup sockets ought to be using IOCP, but
that's a completely different issue.

However, it looks like you're using another worker thread as well in
order to do the actual work. If there's only one connection to the
command pipe, why is this thread needed? There's nothing to synchronise,
so why not do it as part of the pipe handling?

If you really need another thread, just create an IOCP, not bind
anything to it, and PostQueuedCompletionStatus at it. Loop on
GetQueuedCompletionStatus in the other thread and you're done. No
critsec, events, or lists needed.


IIRC the Win32 server part was just to allow the tests to run. This
seems like a lot of work for that. Maybe I'm missing something?

-C



From boxbackup-dev at fluffy.co.uk  Mon Mar 26 21:03:14 2007
From: boxbackup-dev at fluffy.co.uk (Chris Wilson)
Date: Mon, 26 Mar 2007 21:03:14 +0100 (BST)
Subject: [Box Backup-dev] Re: [Box Backup-commit] Re: #3: Merge Win32 branch
In-Reply-To: <46070956.3010206@invis.net>
References: <051.40b7ac89331367bc827cfdd2095c441d@fluffy.co.uk>
 <060.0aa2af72f8ac97b7448b75dcce34143d@fluffy.co.uk>
 <Pine.LNX.4.64.0703252308500.8580@top.qwarx.com> <46070956.3010206@invis.net>
Message-ID: <Pine.LNX.4.64.0703262056140.8580@top.qwarx.com>

Hi Charles,

On Mon, 26 Mar 2007, Charles Lecklider wrote:

>>> [1486] I don't know if I can do this, but I'm going to try :-)
>>>
>>> I'd like to veto this change. You seem to have done a partial re-
>>> implementation of a completion port - see CreateIoCompletionPort and
>>> friends.
>>
>> I assume that this means that I have to use overlapped I/O for both
>> reading and writing from the pipe?
>
> Erm, you have to do that anyway or you'll get some really strange 
> behaviour.

So far I haven't had any problems using overlapped I/O for read only. What 
kind of strange behaviour might I get?

>> I have a feeling that implementing this will move the WinNamedPipeStream
>> API so far away from POSIX that I will have to create a new class that
>> doesn't extend IOStream for this purpose. While it might be a cleaner
>> solution to the current overlapped/nonoverlapped read branches in the
>> code, I'm worried about the duplication of code and possible maintenance
>> headaches.
>
> Well, I'm still trying to work out what you're hoping to achieve; the
> change talks about deadlock, but this is a pipe - that can't happen if
> you always check for errors.

No, you get deadlocks if you perform simultaneous operations on the pipe 
in multiple threads, e.g. Write() in one thread while there is ablocked 
Read() happening in the other. This used to happen to me quite often until 
I moved all the reads and writes to happen in the same thread, and never 
since.

> Obviously there needs to be a thread to deal with commands, but if 
> there's only one thread there's no harm in using normal boring 
> synchronous IO. The actual backup sockets ought to be using IOCP, but 
> that's a completely different issue.

The pipe thread can't use blocking I/O because it also has to listen for 
events from the main threads which signal that it should write to the 
pipe. I couldn't find a way to wait on both a filehandle and an event, and 
I didn't want to create a separate pipe just to communicate between 
threads, so I changed the pipe thread to use only non-blocking I/O and 
events.

> However, it looks like you're using another worker thread as well in 
> order to do the actual work. If there's only one connection to the 
> command pipe, why is this thread needed? There's nothing to synchronise, 
> so why not do it as part of the pipe handling?

No, there are only two threads, the main one and the pipe handler.

> If you really need another thread, just create an IOCP, not bind 
> anything to it, and PostQueuedCompletionStatus at it. Loop on 
> GetQueuedCompletionStatus in the other thread and you're done. No 
> critsec, events, or lists needed.

Does that still apply after my explanation above?

> IIRC the Win32 server part was just to allow the tests to run. This 
> seems like a lot of work for that. Maybe I'm missing something?

Not at all, all of this is on the client side (bbackupd). Nothing at all 
to do with the Win32 server, which doesn't use WinNamedPipeStream at all.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup-dev at fluffy.co.uk  Thu Mar 29 14:37:16 2007
From: boxbackup-dev at fluffy.co.uk (Ben Summers)
Date: Thu, 29 Mar 2007 14:37:16 +0100
Subject: [Box Backup-dev] Making a new release
Message-ID: <430861ED-2261-47A5-98E1-5E3321E5AA14@fluffy.co.uk>

We've stalled again. Everyone apart from Chris that is, who's  
storming ahead.

I propose:

* Chris to finish off whatever he's doing

* When he's happy it's 'complete', create 0.11 test branch from  
Chris' merge branch

* Ask people to test. (as has previously been said, Chris' code is  
decent enough!)

* Work towards release, including a code review.

* Release!

What's the current state of the test suite? I'm assuming it's still  
nice and strong.

I will try to find some time to do the review and some testing on  
OpenBSD and Solaris. Other eyes on the code would be good too,  
volunteers?

Ben







From boxbackup-dev at fluffy.co.uk  Thu Mar 29 15:31:07 2007
From: boxbackup-dev at fluffy.co.uk (Martin Ebourne)
Date: Thu, 29 Mar 2007 15:31:07 +0100
Subject: [Box Backup-dev] Making a new release
In-Reply-To: <430861ED-2261-47A5-98E1-5E3321E5AA14@fluffy.co.uk>
References: <430861ED-2261-47A5-98E1-5E3321E5AA14@fluffy.co.uk>
Message-ID: <20070329153107.ugqdvgg10kgkwo0o@ebourne.me.uk>

Ben Summers <ben at fluffy.co.uk> wrote:
> * Chris to finish off whatever he's doing
>
> * When he's happy it's 'complete', create 0.11 test branch from
> Chris' merge branch

I think there might be changes on trunk we wouldn't want to lose.

Either way it would be just as easy to merge from Chris's merge branch  
into trunk as branch directly, and more consistent too.

> I will try to find some time to do the review and some testing on
> OpenBSD and Solaris. Other eyes on the code would be good too,
> volunteers?

I'm here to help when I get any chance (2nd son due Monday!!).

Cheers,

Martin.


From boxbackup-dev at fluffy.co.uk  Thu Mar 29 20:43:16 2007
From: boxbackup-dev at fluffy.co.uk (Chris Wilson)
Date: Thu, 29 Mar 2007 20:43:16 +0100 (BST)
Subject: [Box Backup-dev] Making a new release
In-Reply-To: <430861ED-2261-47A5-98E1-5E3321E5AA14@fluffy.co.uk>
References: <430861ED-2261-47A5-98E1-5E3321E5AA14@fluffy.co.uk>
Message-ID: <Pine.LNX.4.64.0703292037500.8580@top.qwarx.com>

Hi Ben,

On Thu, 29 Mar 2007, Ben Summers wrote:

> We've stalled again. Everyone apart from Chris that is, who's storming 
> ahead.

Storming in a teacup, maybe :-)

> * Chris to finish off whatever he's doing

All I need to do now, as far as I know, is:

* Track down and fix the crash that Torsten Boob is seeing with my tree.

* Fix security on the named pipe on Win32 (add new options to specify a
   user and group that should own the pipe).

* Merge the last of my changes, which are all to test/bbackupd if I
   remember rightly.

Anyone else have any opinions on "must haves" for 0.11?

> * Ask people to test. (as has previously been said, Chris' code is
>   decent enough!)

Thanks, I appreciate the compliment!

> What's the current state of the test suite? I'm assuming it's still nice 
> and strong.

I hope so, I have been adding tests for almost every new feature and 
reported bug. I still have to merge the bbackupd tests. All tests pass on 
Win32, which is everything except symlinks and Unix sockets, I think.

> I will try to find some time to do the review and some testing on 
> OpenBSD and Solaris. Other eyes on the code would be good too, 
> volunteers?

I can test (i.e. run the unit tests) on several platforms courtesy of 
Sourceforge's compile farm (or the bits of it that happen to be up and 
running at the moment :-)

There is still a problem with running a server on MacOS X (and Darwin), 
which I believe was in 0.10 as well (i.e. not all tests pass on this 
platform, probably due to a bug in the server). Should we consider this a 
blocker for 0.11 release? If so, I can devote some more time to 
investigating and fixing it.

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup-dev at fluffy.co.uk  Thu Mar 29 20:44:05 2007
From: boxbackup-dev at fluffy.co.uk (Chris Wilson)
Date: Thu, 29 Mar 2007 20:44:05 +0100 (BST)
Subject: [Box Backup-dev] Making a new release
In-Reply-To: <20070329153107.ugqdvgg10kgkwo0o@ebourne.me.uk>
References: <430861ED-2261-47A5-98E1-5E3321E5AA14@fluffy.co.uk>
 <20070329153107.ugqdvgg10kgkwo0o@ebourne.me.uk>
Message-ID: <Pine.LNX.4.64.0703292043360.8580@top.qwarx.com>

Hi Martin,

On Thu, 29 Mar 2007, Martin Ebourne wrote:

> I'm here to help when I get any chance (2nd son due Monday!!).

Thanks and congrats, but I'm not sure I can get a new release done by then 
:-)

Cheers, Chris.
-- 
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Perl/SQL/HTML Developer |
\ _/_/_/_//_/___/ | We are GNU-free your mind-and your software |


From boxbackup-dev at fluffy.co.uk  Thu Mar 29 22:14:02 2007
From: boxbackup-dev at fluffy.co.uk (James O'Gorman)
Date: Thu, 29 Mar 2007 22:14:02 +0100
Subject: [Box Backup-dev] Making a new release
In-Reply-To: <Pine.LNX.4.64.0703292037500.8580@top.qwarx.com>
References: <430861ED-2261-47A5-98E1-5E3321E5AA14@fluffy.co.uk> <Pine.LNX.4.64.0703292037500.8580@top.qwarx.com>
Message-ID: <20070329211402.GA3004@netinertia.co.uk>

On Thu, Mar 29, 2007 at 08:43:16PM +0100, Chris Wilson wrote:
> On Thu, 29 Mar 2007, Ben Summers wrote:
> >I will try to find some time to do the review and some testing on 
> >OpenBSD and Solaris. Other eyes on the code would be good too, 
> >volunteers?
> 
> I can test (i.e. run the unit tests) on several platforms courtesy of 
> Sourceforge's compile farm (or the bits of it that happen to be up and 
> running at the moment :-)

I can run tests on FreeBSD again too, plus I have an account on HP's
Testdrive farm so can test on various platforms (once I get my password
reset, it seems...).

James


From boxbackup-dev at fluffy.co.uk  Fri Mar 30 10:22:27 2007
From: boxbackup-dev at fluffy.co.uk (Martin Ebourne)
Date: Fri, 30 Mar 2007 10:22:27 +0100
Subject: [Box Backup-dev] Making a new release
In-Reply-To: <Pine.LNX.4.64.0703292037500.8580@top.qwarx.com>
References: <430861ED-2261-47A5-98E1-5E3321E5AA14@fluffy.co.uk>
 <Pine.LNX.4.64.0703292037500.8580@top.qwarx.com>
Message-ID: <20070330102227.re3u434u74kw4c04@ebourne.me.uk>

Chris Wilson <chris at qwirx.com> wrote:
> There is still a problem with running a server on MacOS X (and
> Darwin), which I believe was in 0.10 as well (i.e. not all tests pass
> on this platform, probably due to a bug in the server). Should we
> consider this a blocker for 0.11 release? If so, I can devote some
> more time to investigating and fixing it.

I would argue that almost by definition any bug present in 0.10 cannot  
be a blocker for 0.11.

That doesn't mean we shouldn't do our best to fix it though! Is the  
bug in trac?

Cheers,

Martin.