I'm working on a fairly ambitious project idea that all hinges on a library I will create that I am tentatively calling "LibASync" (sic), standing for "Asynchronous Synchronization". The apparent oxymoron is intended, and makes sense once the purpose of the library is clear. While I draft the overarching project's design, I wanted to see what people had to say about LibASync - about feasibility, about implementation, about application.
So, what is ASync? ASync is a library that will perform best-effort data synchronization across clients (via AceComm) using a "word of mouth" network. Clients will 'subscribe' to a data feed, and receive or post updates to that feed, which ASync will then attempt to communicate and synchronize across however many users exist in the subscription cloud. Let me give an example.
Alice, Bob, and Eve all subscribe to a cloud. The subscription cloud is identified by a printable string name, which is also used as the prefix in the AceComm library to send the serialized data. Alice updates her local variable and then pushes that update to the cloud. ASynch serializes the variable via AceSerialize, and also prepends a field to the serialized message representing a UNIX timestamp, and then sends it to the cloud. Bob and Eve get the message, and their copies of ASynch see that they have no previous record for this subscription, and so triggers a callback to Bob and Eve's client saying that a subscription with the given name was updated at the given time, sent by Alice.
Now, Bob logs off. Afterwards, Eve pushes a newer update to the cloud. Alice gets the update, records it, including that it was sent by Eve. Eve logs off, and Bob logs back on. His client, as soon as it has finished loading, queries every subscription pool he belongs to, saying "I have this version, is there an update?" Alice's client responds by pushing Eve's update, and Bob records the update. Importantly, Alice doesn't say that Eve was the originator of the message - even if Alice did say that, Bob would have no way of trusting Alice. If the originator of the message is important, it could be encoded in the serialized data, but that data would remain untrustworthy.
I expect that this will be a fairly simple library to write with some hefty overhead concerns. Certainly at the very least, any large subscriptions could cause chat throttling to hamper communication. There's also the comparatively smaller concern that every subscription's state will need to be tracked both in ASync as well as in any addon using that subscription.
It's important to note that ASync specifies no security or data injection prevention. Such concerns are strictly the domain of the clients. If true data protection is required, this scheme should be fully compatible with a PGP-like web of trust, though I would be very surprised if that was ever needed by anyone.
So, the project I'm planning will be doing a lot of cool things with this technology, but I'd love to hear what else you guys can come up with. This project will be focusing on a suite of guild tools, but any ideas are welcome.
"Shout box" - the first client for ASync will be a very simple proof of concept. The client will track (at first) a single string-type variable across the cloud. People can push to the cloud, creating a new "shout". The most recent shout gets displayed through a LibBroker interface. A more advanced version that may require enhanced functionality from Async (or may not, I haven't decided) will track the last N shouts.
"Notice" - a replacement to the extremely boring default Guild Information tab. By default, will only allow guild officers to push to the cloud. The first test of a 'security policy' (ie, ignore pushes from people less privileged than an officer).
"Bulletin board" - the logical next step from the shout box is a bulletin board system for guild communication. Post new topics, reply to posted topics, ignore and drop topics that haven't been updated in N days. Officers can sticky topics that shouldn't ever expire. The first real, heavyweight application.
"Yellowpages" - several addons already try and do this, but I think with ASync they can do it better. Synchronize guild crafting links across the guild, so that you can find out exactly who crafts what.
"DKP" - Track item purchasing points (DKP) so that everyone can see running DKP totals. This module would probably also have an item disbursement and officer management interface, but that's a whole 'nuther ball game.
Well, that's the gist of it. What do people think? Am I re-inventing something that already exists? Anyone want to help?
And, by the way, I hereby license this work, "AceComm-enabled data synchronization across subscription clouds", as copyright Erich Blume 2010, released to the public under the Creative Commons Attribution 3.0 Unported License. See http://creativecommons.org/licenses/by/3.0/ for license information. Share and enjoy.
also worth noting that the cataclysm guild system will include access to profession links for all guild members. however, i love the idea of a data sync library. i've been sort of hacking at one for gnomishyellowpages, but it's hard to test with a single account. while publishing guild links wouldn't be needed, being able to share all the links you've seen in chat would still be nice.
i'm guessing you would support timestamped records in your system, yeah? rather than just having a generic "database" that gets serialized and shared each update. i would imagine most big databases have some sort of timestamp associated with individual records. being able to leverage that to know what to send seems like a logical way to reduce traffic. if such time stamps don't apply the database, then you could fallback to just sending everything.
Thanks for the heads up sparky, I'd have missed that if you hadn't mentioned it. Thank goodness that that's finally going in to the core UI - it's very much needed. (Your add-ons have done a lot for me in that regard, so many thanks there!)
Today I've been going back and forth and back and forth one exactly how "smart" I want the library to be. On the smartest end, I want it to be a standalone mod with its own PersistentVars, tracking the latest X updates (X configurable per-feed?). There are a lot of advantages to this including reducing how much traffic needs to be generated, and how many callbacks need to be fired. Of course there are serious issues too - duplicated data, possible memory issues, a more confusing install process (well, I mean, relatively speaking.)
I think a lot of the work that I'll be doing for this library isn't directly related to time-stamping, but is actually related to how the cloud updates itself. I'm thinking that polls would be done in a two-round phase - first ask everyone what their current state is, and then only poll the states you're interested in. (ie. "Give me the values assosciated with timestamps X, Y, and Z"). I'm also thinking that I'll make clients wait a random amount of time before responding, so that they don't all respond at once, and so they can stop themselves from responding if someone else beat them to it. There's a name for that tactic, does anyone know the name so I can do research? I think it's used in some LAN topologies.
Another possibility is to have a simple master selection algorithm, so when several clients are online at the same time, one of them is chosen as the "master" and will respond of the others. Ideally, the selection mechanism should be predictive enough so all clients could determine which is the master with minimum communication.
Worked on a similar project a couple of months back with a few others, Some of the big issues are traffic overheads and data redundancy, once clouds reach around 50 ppl we were getting some clients dropped from server, we alternated and used a throttle which made updates rather long winded and made it vunerable to ppl either starting to lag or logoff during updates hence the need for redudancy in you sync which sadly also increases your traffic overheads.
At some point during every update you will end up sending some form of representation of every record held by oneside of the update.
How about a token-ring topology? Like Adirelle mentioned, it would give a "Master" responsible for all outbound communication, and then clients would pass their tokens around at specified intervals. It allows for maximum throughput with only one sender at a time, plus the small overhead involved in joining the topology.
I've also been looking into that idea for myself for some time and there are several problems that are hard to solve
- trust, meaning if the data isn't directly received from the author you can't check for fakes unless it's signed
- merging data, meaning what happens if at some point in time a branching happens because a change wasn't propagated before another change happens eg due to the person editing being the only one online and then logging out
an idea I had was the following
- every data piece has a owner and is signed via public/private key
- the data is timestamped whenever the owner shares it with someone and expires after some time
- you can't edit other people's data but only 'reply' to it
the problem that remains is syncing the data without too much overhead since in worst case in a group of a few tens of people every one could have some data pieces others are missing. One could broadcast a hash of its own data on which those not matching respond and a sync is then done via p2p in some way. Would take time but I don't think that matters much then.
OrionShock - are the examples I gave in the post not sufficient? Sure, some could be done with more specific logic (rather than a separate library), but perhaps there is performance optimization to be gained here.
no sadly nothing said so far, and nothing said before (you are not the first ofc) has ever been convincing enough to justify making this system. The only reason that has ever been viable to create something like this was an in game calender synch and system.
That problem was solved recently with blizzard doing that for us.
The rest of the reasons given sadly are not enough, they are quite useless to be honest. I figure that time be better spent on figuring out what it would be used for before it's made. The nitty-gritty implementation details are quite irrelevant.
I'm aware this thread is a few months old, but I wonder if there are any updates?
I've been looking around for an addon that could make use of such a library.
I have a fairly successful crafting empire that relies a great deal on relationships with "farmers". Not the gold-selling variety, but those that sell ore, herbs, etc. so I guess I will call them gatherers. I actually publish a price list that I distribute to these gatherers and they COD their product to me. Maintaining this price list is not fun, but more fun than farming ore myself.
So the addon I've been looking for would allow me to place orders, similar to the system in place for the stock market. I could place orders to buy or sell goods. These orders could be of varying degrees of complexity (see the Wikipedia article on Order (exchange) for some ideas). Other people could see my orders and respond. Functionality could exist to highlight items in bag with current orders, display information in tooltips, automatically COD items to buyers, accept items from sellers etc...
On the one hand, I could use an addon like this to facilitate communication just between myself and my gatherers, but I could also see it being used server-wide, or even for faction or server arbitrage.
The only real concern would be the amount of traffic generated, and keeping a sufficiently updated database so that you weren't constantly sending goods to people with orders that had already been filled.