Nirvana 2.0: Sync question
First, just a comment: Kudos to Elbert for tackling what was likely the most difficult challenge from all the items in the "What's Next" list - OFFLINE mode (i.e. sync). I am impressed Elbert decided to attack this key element of the software and excited about what it offers. I'm sure the flood of requests for tweeks and GUI implementations that will flood the forums are peanuts compared to rebuilding the software from the ground up to allow off-line mode. Congrats.
So, now for the question: I see a SYNC button. Are we supposed to click that button every time we want the changes (local) to be pushed to server? Is there or will there be an automatic SYNC option available (I hope so)? What happens if -shudder- we try to access the data simultaneously on the laptop browser and on our mobile phone at the same time? I guess I could test these questions by trial and error but just wondering if there's a general outline of how it works (and what the final implementation is likely to be).
Thanks, Alex
Comments are currently closed for this discussion. You can start a new one.
2 Posted by Proximo on 13 Nov, 2010 05:42 PM
I was wondering the same thing
How does the sync work?
Can I just close my browser when I am done and not worry?
3 Posted by Kenny Grant on 14 Nov, 2010 09:17 PM
+1 for the explanation on this.
From what I can see you never have to press it either on main or mobile site if you don't lose internet connection - but should use it after any connection drop out.
But I'm guessing here so the definitive use would be useful
4 Posted by Elurven on 15 Nov, 2010 10:05 AM
I'm also very curious about this. I may be wrong but didn't Elbert at some point write that syncing was done periodically throughout the session? At the moment, I am pressing the "sync" button more than I need to, just because I am unsure as to how the syncing works. So please Elbert, let us know how the syncing is implemented and relieve my exhausted sync pressing finger.
Support Staff 5 Posted by Elbert McLaughlin on 15 Nov, 2010 11:03 AM
OK, I was saving this for a blog post, and I will still post one at some point, but here's what you probably need to know in a nut-shell, followed by all the gory technical reasons why a full on migration was necessary to make offline mode a reality. Pseudo FAQ style, here goes.
Do I need to press sync all the time?
Nope. As long as you have a connection, you should never need to hit the Sync button to post changes to the server. The only time you need to hit Sync is if you need to suck down any changes that have been made on another device that you are not seeing on the device you are on... This is analogous to "fetch new mail" in your email client. N2 polls the server every so often, but if you're impatient and want to force a fetch/post "right now" then you can invoke it to do so.
Caveat during the N2 preview release phase -- it's probably a good idea to hit sync at the beginning of each sit down with N2 if you know you've made changes via another device. Changes from that device have likely already been pushed to the server, and you might as well ensure that you've got them sucked down right away. I'm sure y'all do this with email too. :-)
OK, I think get it, but I'd still feel better knowing how this all really works.
So, behind the scenes your tasks, tags, prefs etc are now residing in completely separate data stores from the 1.0 data. Because... every field on every task now has its own last-modified timestamp. This is so that the Nirvana servers can "merge" changes to a task, let's say, that comes in from multiple devices where maybe notes were added via your desktop, a due date was added via your laptop, and it was checked off as complete via your phone. The most recent change per field is what is saved. We think this works as your account isn't really multi-user... it's single-user, multi-device. So as you are the only one making changes to your data, it's unlikely that you'll be on two devices making changes to the same field at the exact same time, right? Ok, so far so good.
As far as creating tasks offline, N2 uses UUIDs generated at the client (browser), rather than relying on the server to assign unique / incremental IDs. So now it no longer matters if you have a connection or not to create tasks. When N2 syncs up, it first saves any "stale" objects (ergo, items that have been touched on your device but not up-to-date with the cloud), then sucks down any changes that have occurred at the server since the last sync time on the device you are on. If the server is seeing a UUID for the first time, it creates a new record. If the UUID already exists, it does it's merge magic, inspecting each field and only updating that field if its timestamp is greater than the last update to that field.
Every time you touch an object (task, tag, whatever) N2 will attempt to save it to the server. But what it actually does is save a copy into localStorage (an html5 underpinnings thing) marked as needing to be saved, immediately returns control of the browser back to you, then goes back to the business of posting the changes to the cloud. If there is no connection, no worries, as the object is flagged as needing to be saved, N2 will continue to attempt saving every few seconds until it gets a "success" response. This "every few seconds" can repeat for hours or days, but it doesn't matter really. The next time a connection is found, N2 running on that device will automatically sync up.
OK, that's the theory. Now in practice. :-)
N2 does not currently poll for changes made on the server unless you manually hit Sync. This is purely for server performance reasons... if we had, say 10,000 devices logged in, and each one was polling every 5 seconds, well that's ~2000 requests per minute, before you add all of the mouse-click generated incremental data that comes flying in near-realtime. Google wouldn't flinch at this - they have bazillion dollar server farms with spare capacity on Mars and Saturn. We're not so deep in the pockets. So while we wait to see how much N2 impacts our servers, you'll want to hit sync if you're not seeing something you know should be. We've observed that people's natural web instincts tell them to reflexively hit refresh whenever something looks out of date, so we figured we'll be ok with this situation for now.
OK, did I confuse the hell out of everybody?
I gotta give props to my colleagues Mitch and Joël for all of the mental white-boarding that went into devising this approach to offline mode. Couldn't find many web apps that have tackled this problem, so it felt like Nirvana was wading into uncharted territory with this. But so far it looks like things are working pretty well.
6 Posted by Elurven on 15 Nov, 2010 11:44 AM
Wow Elbert! My fears are all gone. My itchy trigger finger will finally get some rest ;-).
7 Posted by Proximo on 15 Nov, 2010 12:14 PM
Wow,
There is a show coming on tonight called "Star Wars Tech". I am sure Elbert has a time slot in there somewhere.
:-)
8 Posted by Alex on 16 Nov, 2010 03:01 AM
Elbert,
Thanks for the detailed explanation. I think I understand how it works better.
To summarize. Right now N2 uploads changes from client to server as soon as it can. The opposite direction sync (server -> client) is less frequent (minutes? hours?). So, if we know we've worked on another device, we should hit SYNC as soon as we start a new session of N2 so that changes from the another device that reside on server get pulled to current session.
I might ask at some point some way of polling the GUI to know if sync is up to date. (i.e. something simple like "Last Sync: Date Time")
Curious to know how conflicts are handled - do you just take the changes with the newest time-stamp?
Thanks, -Alex
Support Staff 9 Posted by Elbert McLaughlin on 16 Nov, 2010 03:49 PM
@Alex - you summed it up nicely. and there i went yapping for 72 paragraphs. Your suggestion points are very good... we should show the last sync time... like google docs. as for conflicts, there are no conflicts. ;-) just kidding, sorta. last updated field wins every time. n2 is smart enough to ignore fields that haven't changed.
10 Posted by JamesT on 16 Nov, 2010 04:19 PM
Great explanation Elbert. Makes me even more impressed at the work that went into re-writing this from the ground up and STILL having 95% of the features from v1 ported too. Very nice work indeed.
James
David McLaughlin closed this discussion on 11 Feb, 2011 08:03 AM.