From boxbackup-dev at fluffy.co.uk Wed Aug 6 16:31:54 2008 From: boxbackup-dev at fluffy.co.uk (Peter Jalajas, GigaLock Backup Services) Date: Wed, 6 Aug 2008 11:31:54 -0400 Subject: [Box Backup-dev] ProtocolUncertainStream error Message-ID: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> Hi all, I'm experimenting with boxbackup running on an amazon ec2 ubuntu image directly into s3 using jungledisk plus/fuse. It worked great for hours for my 40GB test client, but now I'm getting this every minute or two (/var/log/messages, ExtendedLogging=yes): Aug 6 13:06:28 (none) bbstored[24249]: NOTICE: Terminating daemon Aug 6 13:06:28 (none) bbstored/hk[24250]: NOTICE: Terminating daemon Aug 6 13:07:11 (none) bbstored[4035]: NOTICE: Box Backup Store Server vtrunk_2206, (c) Ben Summers and contributors 2003-2008 Aug 6 13:07:11 (none) bbstored[4037]: NOTICE: Starting daemon, version: trunk_2206 Aug 6 13:07:11 (none) bbstored[4037]: NOTICE: Using configuration file: /etc/box/bbstored.conf Aug 6 13:09:11 (none) bbstored[4037]: NOTICE: Message from child process 4040: Incoming connection from 72.xxxx port 47271 Aug 6 13:09:11 (none) bbstored[4040]: NOTICE: Login from Client ID 0x1000xxxx Read/Write Aug 6 13:10:04 (none) bbstored[4040]: WARNING: Exception thrown: CommonException(OSFileOpenError) at FileStream.cpp(50) Aug 6 13:10:04 (none) bbstored[4040]: WARNING: ProtocolUncertainStream destroyed before stream finished bbstoreaccounts check fix reports: root at domU~# bbstoreaccounts check 1000xxxx fix NOTICE: Will fix errors encountered during checking. INFO: Checking store account ID 0x1000xxxx... INFO: Phase 1, check objects... TRACE: Max dir starting ID is 0x24700 WARNING: RaidFile dir backup/1000xxxx/00/01 does not exist INFO: Phase 2, check directories... WARNING: File ID 0x1486e has different container ID, probably moved INFO: Phase 3, check root... INFO: Phase 4, fix unattached objects... INFO: Phase 5, fix unrecovered inconsistencies... INFO: Phase 6, regenerate store info... NOTICE: Finished checking store account ID 0x1000xxxx: no errors found Any help to make this work would be appreciated. Thanks, Pete From boxbackup-dev at fluffy.co.uk Thu Aug 7 14:05:41 2008 From: boxbackup-dev at fluffy.co.uk (Chris Wilson) Date: Thu, 7 Aug 2008 14:05:41 +0100 (BST) Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> Message-ID: Hi Pete, On Wed, 6 Aug 2008, Peter Jalajas, GigaLock Backup Services wrote: > I'm experimenting with boxbackup running on an amazon ec2 ubuntu image > directly into s3 using jungledisk plus/fuse. > > It worked great for hours for my 40GB test client, but > now I'm getting this every minute or two (/var/log/messages, > ExtendedLogging=yes): > > Aug 6 13:10:04 (none) bbstored[4040]: WARNING: Exception thrown: > CommonException(OSFileOpenError) at FileStream.cpp(50) This is the real error, the following is just a warning caused by it: > Aug 6 13:10:04 (none) bbstored[4040]: WARNING: > ProtocolUncertainStream destroyed before stream finished I've added some extra debugging in Subversion that should tell us which file failed and how. Please could you try building the latest code and see what it reports when this happens? I think S3 has some issues with newly uploaded files taking some time to appear on the store, particularly if the S3 server that you're using is not your home server, so that may be related (unless s3fs has a way to hide that from the user). Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup-dev at fluffy.co.uk Thu Aug 7 16:01:26 2008 From: boxbackup-dev at fluffy.co.uk (Peter Jalajas, GigaLock Backup Services) Date: Thu, 7 Aug 2008 11:01:26 -0400 Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> Message-ID: <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> Awesome, Chris. Aug 7 14:52:05 (none) bbstored[9997]: NOTICE: Box Backup Store Server vtrunk_2221, (c) Ben Summers and contributors 2003-2008 Aug 7 14:52:05 (none) bbstored[9999]: NOTICE: Starting daemon, version: trunk_2221 Aug 7 14:54:34 (none) bbstored[10004]: WARNING: Failed to open file: /mnt/bu/backup/10002xxxx/47/02/o2e.difftemp: No such file or directory (2) /mnt/bu is the jungledisk s3 fuse mount point. I took the liberty of updating https://www.boxbackup.org/trac/wiki/SourceCodeRepository re how to do svn up. You might check my work. Thanks again, Chris, Pete On Thu, Aug 7, 2008 at 9:05 AM, Chris Wilson wrote: > I've added some extra debugging in Subversion that should tell us which > file failed and how. Please could you try building the latest code and > see what it reports when this happens? From boxbackup-dev at fluffy.co.uk Thu Aug 7 16:47:22 2008 From: boxbackup-dev at fluffy.co.uk (Chris Wilson) Date: Thu, 7 Aug 2008 16:47:22 +0100 (BST) Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> Message-ID: Hi Pete, On Thu, 7 Aug 2008, Peter Jalajas, GigaLock Backup Services wrote: > Aug 7 14:54:34 (none) bbstored[10004]: WARNING: Failed to open file: > /mnt/bu/backup/10002xxxx/47/02/o2e.difftemp: No such file or directory > (2) > > /mnt/bu is the jungledisk s3 fuse mount point. This definitely looks like delayed propagation. We open the same file twice in close succession, the first open creates it and the second one requires it to already exist. Normal filesystem semantics guarantee that this will work, but S3 is not a normal filesystem. In particular: > Updates to a single key are atomic. For example, if you PUT to an > existing key, a subsequent read might return the old data or the updated > data, but it will never write corrupted or partial data... > > Amazon S3 achieves high availability by replicating data across multiple > servers within Amazon's data centers. After a "success" is returned, > your data is safely stored. However, information about the changes might not > immediately replicate across Amazon S3 and you might observe the > following behaviors: > > * A process writes a new object to Amazon S3 and immediately attempts to > read it. Until the change is fully propagated, Amazon S3 might report > "key does not exist." > > * A process writes a new object to Amazon S3 and immediately lists > keys within its bucket. Until the change is fully propagated, the > object might not appear in the list. [http://docs.amazonwebservices.com/AmazonS3/2006-03-01/ConsistencyModel.html] I believe that what is happening is related to this behaviour. What to do about it depends on whether the s3fs developers consider this to be a bug or a feature of s3fs :) In my view, s3fs should try hard to provide as close to normal filesystem semantics as possible, so after creating a file, if a subsequent access discovers that the file does not exist, it should keep retrying until the file does exist (or else silently pretend that the file does exist). This behaviour would increase application compatibility with s3fs. On the other hand, if they only want to provide a lightweight layer over S3, and don't want to implement complex workarounds for S3's behaviour, then each application would have to fix it itself. It is possible to do the same workarounds in Box Backup, but it's really ugly and I'd be reluctant to either work on it or to merge that code into the mainline, unless other important filesystems have similar properties (which I doubt). My feeling is that the real place for such code in Box Backup would be in an alternative storage backend that implements the RaidFile interface and stores data in S3. Unfortunately, as we've discussed before and I'm sure you know, quite a bit of work is involved in doing that, especially in unit-testing it. > I took the liberty of updating > https://www.boxbackup.org/trac/wiki/SourceCodeRepository re how to do > svn up. You might check my work. Thanks, looks fine to me! Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup-dev at fluffy.co.uk Thu Aug 7 19:00:50 2008 From: boxbackup-dev at fluffy.co.uk (James O'Gorman) Date: Thu, 07 Aug 2008 19:00:50 +0100 Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> Message-ID: <489B3852.6050607@netinertia.co.uk> Peter Jalajas, GigaLock Backup Services wrote: > Awesome, Chris. > > Aug 7 14:52:05 (none) bbstored[9997]: NOTICE: Box Backup Store Server > vtrunk_2221, (c) Ben Summers and contributors 2003-2008 > Aug 7 14:52:05 (none) bbstored[9999]: NOTICE: Starting daemon, > version: trunk_2221 > > Aug 7 14:54:34 (none) bbstored[10004]: WARNING: Failed to open file: > /mnt/bu/backup/10002xxxx/47/02/o2e.difftemp: No such file or directory > (2) That's really cool - I've been wondering what's been causing this on one of my boxes too (I'm just assuming it's a log file at the moment). Thanks, Chris. > I took the liberty of updating > https://www.boxbackup.org/trac/wiki/SourceCodeRepository re how to do > svn up. You might check my work. One comment - I assume: " If you are updating a previous svn installation, cd to that same working directory, and then use this command: svn up http://www.boxbackup.org/svn/box/trunk/ " refers to updating an existing working copy rather than an installation of Subversion? In which case you can simply run "svn up" (or svn update, if you prefer). If, however you're *switching* your working copy from the old bbdev.fluffy.co.uk repo then the command goes like so: svn switch http://bbdev.fluffy.co.uk/svn/box/trunk https://www.boxbackup.org/svn/box/trunk Let me know if that's what you meant or if I've lost the plot somewhere :-) James From boxbackup-dev at fluffy.co.uk Thu Aug 7 19:15:53 2008 From: boxbackup-dev at fluffy.co.uk (Peter Jalajas, GigaLock Backup Services) Date: Thu, 7 Aug 2008 14:15:53 -0400 Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: <489B3852.6050607@netinertia.co.uk> References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> <489B3852.6050607@netinertia.co.uk> Message-ID: <74d01c7a0808071115o1507a995j5ee2999ea613a68b@mail.gmail.com> You're exactly right, James, thanks. Sorry for the error. Fixed now (I hope): https://www.boxbackup.org/trac/wiki/SourceCodeRepository On Thu, Aug 7, 2008 at 2:00 PM, James O'Gorman wrote: > Peter Jalajas, GigaLock Backup Services wrote: >> >> I took the liberty of updating >> https://www.boxbackup.org/trac/wiki/SourceCodeRepository re how to do >> svn up. You might check my work. > > One comment - I assume: > > " > If you are updating a previous svn installation, cd to that same working > directory, and then use this command: > > svn up http://www.boxbackup.org/svn/box/trunk/ > " > > refers to updating an existing working copy rather than an installation of > Subversion? In which case you can simply run "svn up" (or svn update, if you > prefer). > > > If, however you're *switching* your working copy from the old > bbdev.fluffy.co.uk repo then the command goes like so: > > svn switch http://bbdev.fluffy.co.uk/svn/box/trunk > https://www.boxbackup.org/svn/box/trunk > > Let me know if that's what you meant or if I've lost the plot somewhere :-) > > James From boxbackup-dev at fluffy.co.uk Thu Aug 7 19:19:13 2008 From: boxbackup-dev at fluffy.co.uk (Peter Jalajas, GigaLock Backup Services) Date: Thu, 7 Aug 2008 14:19:13 -0400 Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> Message-ID: <74d01c7a0808071119s27f5ae8cr4f2b337ede124125@mail.gmail.com> Thanks, Chris. On Thu, Aug 7, 2008 at 11:47 AM, Chris Wilson wrote: > Hi Pete, > On Thu, 7 Aug 2008, Peter Jalajas, GigaLock Backup Services wrote: > >> * A process writes a new object to Amazon S3 and immediately attempts to >> read it. Until the change is fully propagated, Amazon S3 might report >> "key does not exist." > I believe that what is happening is related to this behaviour. What to do > about it depends on whether the s3fs developers consider this to be a bug > or a feature of s3fs :) Shucks. I tried s3fs first (cuz it is open-source), but it couldn't keep up at all. So I broke down and bought jungledisk plus. What's disappointing, and hopefully instructive, is that it worked perfectly for hours and hours with about 40 GB of data flying in via a FiOS line at 1.2Mbps. Do maybe I have 1 bad file that I need to manipulate or delete or something? To test that, I deleted the .rfwX file, and am running bbstoreaccountch check fix...it is taking quite a while...I'll report back when done and tested... I'll also submit a ticket to jungledisk pointing to the box backup code that's failing: .// -------------------------------------------------------------------------- // // Function // Name: FileStream::FileStream(const char *, int, int) // Purpose: Constructor, opens file // Created: 2003/07/31 // // -------------------------------------------------------------------------- FileStream::FileStream(const char *Filename, int flags, int mode) #ifdef WIN32 : mOSFileHandle(::openfile(Filename, flags, mode)), #else : mOSFileHandle(::open(Filename, flags, mode)), #endif mIsEOF(false), mFileName(Filename) { #ifdef WIN32 if(mOSFileHandle == INVALID_HANDLE_VALUE) #else if(mOSFileHandle < 0) #endif From boxbackup-dev at fluffy.co.uk Thu Aug 7 19:29:16 2008 From: boxbackup-dev at fluffy.co.uk (Peter Jalajas, GigaLock Backup Services) Date: Thu, 7 Aug 2008 14:29:16 -0400 Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: <74d01c7a0808071119s27f5ae8cr4f2b337ede124125@mail.gmail.com> References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> <74d01c7a0808071119s27f5ae8cr4f2b337ede124125@mail.gmail.com> Message-ID: <74d01c7a0808071129o3bd3b6f5wc8cdc4a08968e881@mail.gmail.com> P.S. Not sure if it helps, but here is the tail of the jungledisk cache log: 1,bu,/backup/10002xxx/47/02/o2e.rfwX,0,0,1218123168,0,OK, , 1,bu,/backup/10002xxx/write.lock,0,0,1218123173,0,OK, , 1,bu,/backup/10002xxx/write.lock,0,0,1218123232,0,OK, , 1,bu,/backup/10002xxx/47/02/o2e.difftemp,0,0,1218123288,0,OK, , 2,bu,/backup/10002xxx/47/02/o2e.difftemp,0,0,1218123288,0,OK, , 2,bu,/backup/10002xxx/47/02/o2e.rfwX,0,0,1218123288,0,OK, , 1,bu,/backup/10002xxx/47/02/o2e.rfwX,0,0,1218123293,0,OK, , 1,bu,/backup/10002xxx/write.lock,0,0,1218123298,0,OK, , 1,bu,/backup/10002xxx/write.lock,0,0,1218123343,0,OK, , 1,bu,/backup/10002xxx/47/02/o2e.difftemp,0,0,1218123390,0,OK, , 2,bu,/backup/10002xxx/47/02/o2e.difftemp,0,0,1218123390,0,OK, , 2,bu,/backup/10002xxx/47/02/o2e.rfwX,0,0,1218123390,0,OK, , 1,bu,/backup/10002xxx/47/02/o2e.rfwX,0,0,1218123395,0,OK, , 1,bu,/backup/10002xxx/write.lock,0,0,1218123400,0,OK, , 1,bu,/backup/10002xxx/write.lock,0,0,1218123431,1,OK, , 1,bu,/backup/10002xxx/info.rfw,88,88,1218123435,0,OK, , 1,bu,/backup/10002xxx/write.lock,0,0,1218123440,0,OK, , 1,bu,/backup/10002xxx/write.lock,0,0,1218123457,0,OK, , 1,bu,/backup/10008xxx/info.rfw,88,88,1218123500,0,OK, , 1,bu,/backup/10008xxx/write.lock,0,0,1218123500,0 On Thu, Aug 7, 2008 at 2:19 PM, Peter Jalajas, GigaLock Backup Services wrote: > Thanks, Chris. > > On Thu, Aug 7, 2008 at 11:47 AM, Chris Wilson wrote: >> Hi Pete, >> On Thu, 7 Aug 2008, Peter Jalajas, GigaLock Backup Services wrote: >> >>> * A process writes a new object to Amazon S3 and immediately attempts to >>> read it. Until the change is fully propagated, Amazon S3 might report >>> "key does not exist." > >> I believe that what is happening is related to this behaviour. What to do >> about it depends on whether the s3fs developers consider this to be a bug >> or a feature of s3fs :) > > Shucks. > > I tried s3fs first (cuz it is open-source), but it couldn't keep up at > all. So I broke down and bought jungledisk plus. > > What's disappointing, and hopefully instructive, is that it worked > perfectly for hours and hours with about 40 GB of data flying in via a > FiOS line at 1.2Mbps. Do maybe I have 1 bad file that I need to > manipulate or delete or something? To test that, I deleted the .rfwX > file, and am running bbstoreaccountch check fix...it is taking quite a > while...I'll report back when done and tested... > > I'll also submit a ticket to jungledisk pointing to the box backup > code that's failing: > .// -------------------------------------------------------------------------- > // > // Function > // Name: FileStream::FileStream(const char *, int, int) > // Purpose: Constructor, opens file > // Created: 2003/07/31 > // > // -------------------------------------------------------------------------- > FileStream::FileStream(const char *Filename, int flags, int mode) > #ifdef WIN32 > : mOSFileHandle(::openfile(Filename, flags, mode)), > #else > : mOSFileHandle(::open(Filename, flags, mode)), > #endif > mIsEOF(false), > mFileName(Filename) > { > #ifdef WIN32 > if(mOSFileHandle == INVALID_HANDLE_VALUE) > #else > if(mOSFileHandle < 0) > #endif > -- E.W. Peter Jalajas PJalajas at GigaLock.com | http://GigaLock.com/ 978-495-2062 | PO Box 1223, Newburyport, MA 01950-8223 Digital Fingerprint: 5D13 7C5C D003 D806 8BD1 438A B396 EC3D 1C23 B5F0 GigaLock is a secure off-site backup service for computers and servers. We are currently protecting more than 1.1 million files for clients nationally. Every day, your files are automatically and securely backed up off-site and across the country. Plans include $19/month per gigabyte, or up to 200 GB for a flat fee of $149/month! We are looking for additional Customers, Resellers, and Sales Associates. From boxbackup-dev at fluffy.co.uk Fri Aug 8 00:10:07 2008 From: boxbackup-dev at fluffy.co.uk (Peter Jalajas, GigaLock Backup Services) Date: Thu, 7 Aug 2008 19:10:07 -0400 Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: <74d01c7a0808071129o3bd3b6f5wc8cdc4a08968e881@mail.gmail.com> References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> <74d01c7a0808071119s27f5ae8cr4f2b337ede124125@mail.gmail.com> <74d01c7a0808071129o3bd3b6f5wc8cdc4a08968e881@mail.gmail.com> Message-ID: <74d01c7a0808071610g35e17f3di95734e574e021d3a@mail.gmail.com> Hi Chris and James, I ran bbstoreaccounts check fix successfully, stopped bbstored, umounted the fuse fs, remounted it, started bbstored, and got the same error on the same file. The jungledisk developer wrote back: "The caching system in Jungle Disk should ensure that delayed prorogation doesn't have any effect. So long as the file is closed before you attempt to re-open it, it should work fine. Note that if you attempt to open it before closing it won't find the file." Is there a line or two of code that we can add to ./lib/common/FileStream.cpp to address his point? Thanks, Pete From boxbackup-dev at fluffy.co.uk Fri Aug 8 17:28:52 2008 From: boxbackup-dev at fluffy.co.uk (Peter Jalajas, GigaLock Backup Services) Date: Fri, 8 Aug 2008 12:28:52 -0400 Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: <74d01c7a0808071610g35e17f3di95734e574e021d3a@mail.gmail.com> References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> <74d01c7a0808071119s27f5ae8cr4f2b337ede124125@mail.gmail.com> <74d01c7a0808071129o3bd3b6f5wc8cdc4a08968e881@mail.gmail.com> <74d01c7a0808071610g35e17f3di95734e574e021d3a@mail.gmail.com> Message-ID: <74d01c7a0808080928k75c69b60ndb658abaf2477981@mail.gmail.com> Hi again, Just thinking out loud, is there an easy way to, at least temporarily or off trunk, put in a little code just before line 50 in ./lib/common/FileStream.cpp to check the file, and if not ready yet, wait an increasing number of milliseconds and try again, and if not yet ready, loop again, for a maximum delay of, what, a few seconds? Does that make any sense? Separately, since it seems to always be hanging on this one file, can you think of a way to clear out that one file somehow? I tried deleting the file with .rfwX but I don't think that helped. Thanks again, Pete On Thu, Aug 7, 2008 at 7:10 PM, Peter Jalajas, GigaLock Backup Services wrote: > Hi Chris and James, > > I ran bbstoreaccounts check fix successfully, stopped bbstored, > umounted the fuse fs, remounted it, started bbstored, and got the same > error on the same file. > > The jungledisk developer wrote back: > > "The caching system in Jungle Disk should ensure that delayed > prorogation doesn't have any effect. So long as the file is closed > before you attempt to re-open it, it should work fine. > Note that if you attempt to open it before closing it won't find the file." > > Is there a line or two of code that we can add to > ./lib/common/FileStream.cpp to address his point? > > Thanks, > Pete > -- E.W. Peter Jalajas PJalajas at GigaLock.com | http://GigaLock.com/ 978-495-2062 | PO Box 1223, Newburyport, MA 01950-8223 Digital Fingerprint: 5D13 7C5C D003 D806 8BD1 438A B396 EC3D 1C23 B5F0 GigaLock is a secure off-site backup service for computers and servers. We are currently protecting more than 1.1 million files for clients nationally. Every day, your files are automatically and securely backed up off-site and across the country. Plans include $19/month per gigabyte, or up to 200 GB for a flat fee of $149/month! We are looking for additional Customers, Resellers, and Sales Associates. From boxbackup-dev at fluffy.co.uk Sat Aug 9 08:17:18 2008 From: boxbackup-dev at fluffy.co.uk (Chris Wilson) Date: Sat, 9 Aug 2008 08:17:18 +0100 (BST) Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: <74d01c7a0808071610g35e17f3di95734e574e021d3a@mail.gmail.com> References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> <74d01c7a0808071119s27f5ae8cr4f2b337ede124125@mail.gmail.com> <74d01c7a0808071129o3bd3b6f5wc8cdc4a08968e881@mail.gmail.com> <74d01c7a0808071610g35e17f3di95734e574e021d3a@mail.gmail.com> Message-ID: Hi Pete, On Thu, 7 Aug 2008, Peter Jalajas, GigaLock Backup Services wrote: > The jungledisk developer wrote back: > > "The caching system in Jungle Disk should ensure that delayed > prorogation doesn't have any effect. So long as the file is closed > before you attempt to re-open it, it should work fine. Note that if you > attempt to open it before closing it won't find the file." That explains it. We do open the file twice without closing it the first time. > Is there a line or two of code that we can add to > ./lib/common/FileStream.cpp to address his point? Unfortunately this is required by the way that the diffing algorithm works. I think it will require some major work to rewrite it not to do this. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup-dev at fluffy.co.uk Sat Aug 9 15:18:35 2008 From: boxbackup-dev at fluffy.co.uk (Chris Wilson) Date: Sat, 9 Aug 2008 15:18:35 +0100 (BST) Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: <74d01c7a0808071119s27f5ae8cr4f2b337ede124125@mail.gmail.com> References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> <74d01c7a0808071119s27f5ae8cr4f2b337ede124125@mail.gmail.com> Message-ID: Hi Pete, On Thu, 7 Aug 2008, Peter Jalajas, GigaLock Backup Services wrote: > I tried s3fs first (cuz it is open-source), but it couldn't keep up at > all. So I broke down and bought jungledisk plus. > > What's disappointing, and hopefully instructive, is that it worked > perfectly for hours and hours with about 40 GB of data flying in via a > FiOS line at 1.2Mbps. Do maybe I have 1 bad file that I need to > manipulate or delete or something? I think diffing against existing files is completely broken on Jungle Disks because of this behaviour. > I'll also submit a ticket to jungledisk pointing to the box backup > code that's failing: I don't think that will tell him anything. The relevant code, if it means anything at all to him, is at: http://www.boxbackup.org/trac/browser/box/trunk/bin/bbstored/BackupStoreContext.cpp around line 470, where it open the file twice, the first time creating it (and keeping it open) and the second time for reading. If he decides not to change this behaviour of Jungle Disk, then we'd have to rewrite this algorithm. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup-dev at boxbackup.org Sun Aug 10 20:39:09 2008 From: boxbackup-dev at boxbackup.org (Chris Wilson) Date: Sun, 10 Aug 2008 20:39:09 +0100 (BST) Subject: [Box Backup-dev] ProtocolUncertainStream error In-Reply-To: <74d01c7a0808080928k75c69b60ndb658abaf2477981@mail.gmail.com> References: <74d01c7a0808060831h2d042091s1ef0982f49d43b13@mail.gmail.com> <74d01c7a0808070801x19b5cc06sda8365f632b997d0@mail.gmail.com> <74d01c7a0808071119s27f5ae8cr4f2b337ede124125@mail.gmail.com> <74d01c7a0808071129o3bd3b6f5wc8cdc4a08968e881@mail.gmail.com> <74d01c7a0808071610g35e17f3di95734e574e021d3a@mail.gmail.com> <74d01c7a0808080928k75c69b60ndb658abaf2477981@mail.gmail.com> Message-ID: Hi Pete, On Fri, 8 Aug 2008, Peter Jalajas, GigaLock Backup Services wrote: > Just thinking out loud, is there an easy way to, at least temporarily or > off trunk, put in a little code just before line 50 in > ./lib/common/FileStream.cpp to check the file, and if not ready yet, > wait an increasing number of milliseconds and try again, and if not yet > ready, loop again, for a maximum delay of, what, a few seconds? Does > that make any sense? Amazon makes no guarantees about how long it might take for the change to propagate, so this would probably just be guesswork. The Jungle Disk author in his reply to you seemed to say that the file would never appear if reopened before being closed the first time. One thing we could do is to host this difftemp file on a completely different filesystem, as it is after all just a temporary file. > Separately, since it seems to always be hanging on this one file, can > you think of a way to clear out that one file somehow? I tried deleting > the file with .rfwX but I don't think that helped. Why does it need to be cleared out? Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup-dev at boxbackup.org Mon Aug 11 10:57:34 2008 From: boxbackup-dev at boxbackup.org (Roy) Date: Mon, 11 Aug 2008 11:57:34 +0200 Subject: [Box Backup-dev] dll's In-Reply-To: References: <47B3F50E.20307@hostingbrothers.nl> Message-ID: <48A00D0E.8090601@hostingbrothers.nl> Chris Wilson wrote: > Hi Roy, > > On Thu, 14 Feb 2008, Roy wrote: > >> Is it possible to put the logic of box backup in dll's for the >> windows platform? That way it can be used by other programs as well, >> especially a GUI for box backup. And with updates of box backup, you >> don't have to track those fixes to put back in the code for a GUI. We >> really need a GUI for easy restoring of files on the windows >> platform, but at the moment you would have to duplicate the code for >> that. > > Yes, it is possible. Boxi uses the code as a library, although it > doesn't actually have a "Box Backup DLL". > > Cheers, Chris. Hi Chris, I'm sorry, I only just received this mail in my box. So my reply is a bit late. Is it possible to build a DLL which can be used in other programs, instead of just using the source files? Because it then can be used in other languages as well (C# for example). Thanks in advance, Roy PS. building Windows executables didn't succeed with Visual Studio nor with Mingw on Debian. Has something changed? From boxbackup-dev at boxbackup.org Mon Aug 11 11:00:47 2008 From: boxbackup-dev at boxbackup.org (Chris Wilson) Date: Mon, 11 Aug 2008 11:00:47 +0100 (BST) Subject: [Box Backup-dev] dll's In-Reply-To: <48A00D0E.8090601@hostingbrothers.nl> References: <47B3F50E.20307@hostingbrothers.nl> <48A00D0E.8090601@hostingbrothers.nl> Message-ID: Hi Roy, On Mon, 11 Aug 2008, Roy wrote: > I'm sorry, I only just received this mail in my box. So my reply is a > bit late. > > Is it possible to build a DLL which can be used in other programs, > instead of just using the source files? Because it then can be used in > other languages as well (C# for example). It's possible to use the code as a library, but not to build a DLL specifically yet, as in, nobody has written the scripts to build a DLL from it. The easiest thing to do for now is just to nick the code modules that you want, provided that the license is compatible. The code isn't stable enough to produce an API that we could package into a DLL. > PS. building Windows executables didn't succeed with Visual Studio nor > with Mingw on Debian. Has something changed? No, it should work, please let us know what the errors are. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software | From boxbackup-dev at boxbackup.org Mon Aug 11 11:16:04 2008 From: boxbackup-dev at boxbackup.org (Roy) Date: Mon, 11 Aug 2008 12:16:04 +0200 Subject: [Box Backup-dev] dll's In-Reply-To: <74d01c7a0802140736j4d5eaf95td4c18cda64b6b5f@mail.gmail.com> References: <47B3F50E.20307@hostingbrothers.nl> <74d01c7a0802140736j4d5eaf95td4c18cda64b6b5f@mail.gmail.com> Message-ID: <48A01164.4050504@hostingbrothers.nl> Peter Jalajas, GigaLock Backup Services wrote: > For the record, for what's it's worth, I've built a simple installer > for Box Backup for Windows using InstallJammer. It installs the files > in the directory selected at install time, installs the service, and > prompts for backup locations and exclusions and creates a > bbackupd.conf. It also creates a stack of buttons under the Start, > All Programs menu system that launch various box backup common > functions, like opening notepad to edit the .conf file, starting the > service, stop the service, reload config, output query listings to > text files, etc. That stack of buttons is getting kind of large, so I > got to thinking about replacing all those buttons with a GUI. > > Knowing that Boxi is out there, but while having lots of potential, it > has a funky widget thing that I could never figure out, and requires > Chris' time which is being spent so well on core code development that > I hate to have him distracted. > > I thought a Java GUI might be fairly simple way to creat a GUI around > the box backup commands. (I tried playing around Eclipse and NetBeans > this week, and NetBeans looks far better.) > > What do you all think about building a Java GUI interface? > > Thanks, > Pete > Hi Pete, Sorry for the late reply, but I only just recieved this message. Did you already started on this Java GUI? Or did you abandon this idea? Because a gui would be very handy in restoring files from the backup. Gr, Roy > On 2/14/08, Roy wrote: > >> Hello guys, >> >> Is it possible to put the logic of box backup in dll's for the windows >> platform? That way it can be used by other programs as well, especially >> a GUI for box backup. And with updates of box backup, you don't have to >> track those fixes to put back in the code for a GUI. >> We really need a GUI for easy restoring of files on the windows >> platform, but at the moment you would have to duplicate the code for that. >> >> Thanks in advance, >> >> Roy >> >> >> >> _______________________________________________ >> Boxbackup-dev mailing list >> Boxbackup-dev at fluffy.co.uk >> http://lists.warhead.org.uk/mailman/listinfo/boxbackup-dev >> >> > > > From boxbackup-dev at boxbackup.org Mon Aug 11 11:25:40 2008 From: boxbackup-dev at boxbackup.org (Chris Wilson) Date: Mon, 11 Aug 2008 11:25:40 +0100 (BST) Subject: [Box Backup-dev] dll's In-Reply-To: <48A01164.4050504@hostingbrothers.nl> References: <47B3F50E.20307@hostingbrothers.nl> <74d01c7a0802140736j4d5eaf95td4c18cda64b6b5f@mail.gmail.com> <48A01164.4050504@hostingbrothers.nl> Message-ID: Hi all, On Mon, 11 Aug 2008, Roy wrote: > Peter Jalajas, GigaLock Backup Services wrote: > > > > Knowing that Boxi is out there, but while having lots of potential, it > > has a funky widget thing that I could never figure out What funky widget thing? The tree control with checkboxes to select files to back up? > > I thought a Java GUI might be fairly simple way to creat a GUI around > > the box backup commands. (I tried playing around Eclipse and NetBeans > > this week, and NetBeans looks far better.) > > > > What do you all think about building a Java GUI interface? > > Did you already started on this Java GUI? Or did you abandon this idea? > Because a gui would be very handy in restoring files from the backup. I considered writing the GUI in Java to start with, but I had some bad experiences with interfacing native code to Java, and decided that it was better to either rewrite/port the client code in Java, or code in C++. I dediced that coding in C++ was less work and resulted in better performance, for what it's worth. Some other things which are difficult to do in Java are: fully native-looking GUIs, access to OS constructs such as file timestamps and inode numbers, extended attributes, Windows ACLs and Windows security. You'd have to write about 30% of the code as native code anyway to add support for all of those, whether or not you port the whole existing Box Backup network/encryption code. What are the barriers to people helping out on Boxi? It is an open-source project on Sourceforge, anyone can join in and help. Cheers, Chris. -- _____ __ _ \ __/ / ,__(_)_ | Chris Wilson <0000 at qwirx.com> - Cambs UK | / (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer | \ _/_/_/_//_/___/ | We are GNU : free your mind & your software |