Authors CurseForge

@Pastamancer concerning the sharing of data:

First I wanted to name my project Addon Exchange Manager. But actually you can exchange any kind of data with it, addons just happen to be executable. I've designed the header to be as extendable and independent of my implementation as possible. Who knows, if my idea works out then other developers might do things with it too and it makes sense to define some kind of standard format to avoid conversions later on. Even if you don't agree with my idea, some of my work may be useful to you too since it touches many different areas. Let me show you some of the problems concerning the header format and how I tried to solve them. It looks like this:

Id: "DEM1"
Hash: <40 character hex value>
Signature: <256 character hex value>
Fields: "NEVDACF"

Name: "Test Module"
Extension: "exe"
Version: "1.0"
Description: "This is a test module!"
Author: "Lenja"
Category: "Raid"
File: <string containing addon or data>

A valid module must contain the first four entries (id,hash,sig,fields). Fields specifies what and in which order additional fields appear in the header.

Fields are separated by an underline "_". So a valid header will look like this:

DEM1_637EF7048617C2E72D9D71755BCA35397C65C3D7_C0E7
C47EAE056A5410B677815D53991C68C28E2076553D5E8833A9
CFB77214E6DF04A003BD49F1A4A37983FAA01EDA10D9AA5586
8F9864F810F576E601C5BF491547A51ACE4C2AFEAFE097F3B7
FD2152FBDDC176C442B28D42E76482BA5F890181FE48C8FEBE
EFD708404C162F7EFA03F5C0735AC74F6B1CAA02D4D66D4B6E
4_NEVDACF_ (additional fields following...)

You see that the character "_" is used to separate fields, so naturally that character may not occur in any fields themselves. Zero characters may cause problems in C strings and won't work with SendAddonMessage() so those cannot appear as well (which is really bad if you want to include binary data). Any special characters above ASCII 127 (depending on language settings) cannot be reliably used to copy&paste from and to edit boxes inside WoW because of UTF8 conversion, which is inconvenient if you want the user to be able to import modules without having to write data to save files in the WTF folder.

For all fields except the first four, I've chosen the Base64 encoding standard which uses 64 unproblematic characters that are also used in email transmission. This will increase the size by one third but that can't be helped. We can shrink the fields again once they're loaded in memory.

We gain the following by this proposed format:

- A module can be identified by the first five characters "DEM1_". Future formats that differ from this specification will have a different ID so you can hold them apart and implement different handlers.

- The SHA1 hash value guarantees that the message has not been changed by a transmission error. (better than the Sig for error detection because it's faster)

- The RSA signature guarantees that the following data has not been tampered with and has been given an "OK" by someone in possession of the private RSA-1024 key. If I have implemented it correctly, there is no way how you can crack this unless you use 1mio PCs and let them work 1 year on it. If you find a way, tell me how. I'm sure many mathematicians and intelligence agencies around the world would also like to talk to you. Meaning: You won't crack this.

- The "fields" field allows us to omit or add more fields in the future in any order without breaking the basic format. Future modules should be downwards compatible.

- All other fields are encoded in a way that is compatible with Lua, Lua save files, WoW edit boxes and Internet message boards (which tend to truncate space and tab characters).

So I hope you see, I'm not just some guy who wanted to code something together that works. I've been thinking on this for weeks. I want to set standards and get a rock solid solution. I'm sure you guys with your wonderful Ace libraries can appreciate that.

I'm hoping you might think my idea is worthwhile and if anyone wants to participate...

The first implementation of this model is already done and working so far. I'm sure it contains a few bugs and inaccuracies so I'll have to rewrite some portions of the code. But the concept stands and it would be wonderful if this could be turned into a community effort if anyone's interested.

Since there weren't any crypto libraries for hashing and RSA available in plain Lua, I had to implement them myself which wasn't trivial. I've published them on LuaForge, so they're freely available here: http://luaforge.net/projects/sha1-rsa/

I think many of you fail to see how cool cryptography is and how it may help you in your projects, addons or otherwise. For instance, you all know the problem of bad code causing all sorts of problems. I've read somewhere that people should not worry about the quality of the Ace libraries but about the code that uses them. Anyone can take your libraries and do whatever evil things they want. Luckily the community is there to tell them just how crappy their code is.

But I won't even have that problem. I may simply choose not to sign badly coded modules. And that's the end of it. No chance to distribute your module if it's not signed even if I have no direct control over the distribution system. Signed data can take care of itself. I don't know if that's a good or bad thing depending on how picky the holder of the key is, but you can't deny it's really cool from a technical standpoint.

Coming back to your Grid idea: The addons that receive such data have to be aware it comes from a potentially unsafe source and sanitize all input. I'm sure someone will find a way to design a Grid config file in such a way that the recipient's WoW will crash. So in that respect, immutable executable addons are saver than changeable data that can't be signed.

@HunterZ:
You're right. Although signatures guarantee that bad people cannot send you evil stuff, you really want some kind of filtering to keep random people from flooding your inbox. It is another deterrent for abuse since you can configure it to only accept data from friends. So somebody would have to get on your friendslist first just to be allowed to send you anything at all. Once in your inbox, you'll still have to give your "OK" to install something.

No to this. All it takes is for one malicious addon to modify the code executing permission code (that asks the user whether to execute code or not) for that one compromised person to start spreading code that modifies the code executing permission code.

As with most internet worms/viruses, most of them are obtained voluntarily downloading stuff that is already compromised (cracks, etc) and many unaware users will simply just click YES to execute code just because it came from someone on your friends list.

Quote from Lenja »

But I won't even have that problem. I may simply choose not to sign badly coded modules. And that's the end of it. No chance to distribute your module if it's not signed even if I have no direct control over the distribution system. Signed data can take care of itself. I don't know if that's a good or bad thing depending on how picky the holder of the key is, but you can't deny it's really cool from a technical standpoint.

Technically it is very interesting, but I think you're trying to solve a problem that doesn't exist.

If the addon authors are required to submit code to you for review and signing, then this isn't any more convenient than uploading to an addon site.

@Pastamancer:
Modules should be convenient for users, not for developers. Developers can test their code and exchange it with friends even when it's unsigned, although users have to jump through hoops to accept unsigned modules. But that's the price of security.

I propose that a module has to be posted on a public site for review for one week before it's getting a signature. Several responsible groups could have their individual private key that works for signing. I will obviously have one and e.g. the Ace moderators could have one etc.


@Xinhuan: There's no worm effect here. It is possible to breach the security defenses of one person by modifying the code that is supposed to protect the user. But that doesn't mean the code is spreading now. Each user has their own defenses that have to be breached. As long as your very own defenses aren't breached, the signature checking code is in place and will keep you from getting anything dangerous.

Anyway, once bad code is in your system, all bets are off. Your system is compromised. It doesn't matter if the bad code changes my addon to allow further evil code to get in or if it comes with it's own copy of my routines (a general problem with open source) or even with its very own download stub (which isn't hard to code). All of this is possible with normal addons too.


@Kagaro: Now I think you're being unfair. :D The signing process makes sure addons have to be reviewed by the public before they can be spread which is exactly how it works with real addons that are uploaded to a site. Suggesting that the public might mistakenly sign a module is the same as the public missing bad code in a publicly posted addon.

One thing that is true is that it's easy to delete a bad addon even if you miss it for one week. Having already signed a bad addon means that you have to revoke a signature by spreading some kind of emergency update to put it on a blacklist (designing such a revocation system won't be hard having the rest already in place). It's more complicated but it's possible. You are right that we have to be extra careful to make sure no bad code is signed by mistake.

But even if it does happen, users still have to accept every incoming transmission. So there's still no worm effect. I have to admit that users may accept addons more quickly when they're signed because that seems to be a guarantee that they're safe. But the same applies to addon sites that only post addons that were reviewed.

Modules might indeed spread faster than normal addons because the whole system is designed to spread any addon very quickly. But you won't sell your car just because walking is safer, right?


I admit that this all seems very complicated. Maybe too complicated to implement properly. But most of it is already done. And these technologies weren't invented by me. They're being widely used in the industry and have been proven to work. I've been interested in this kind of stuff for a while so this seems easy to me.

I'm really glad about the many comments so far. Like I said, I will go ahead with this project no matter what, but if I could win the support of a few people here who think the project isn't all that bad, this would really be a big help. There's still a lot of work to do, too...

This just makes addons far more needlessly complex. If I make addon A that interacts with addon B (optionaldep) and it has a dependency on library X and Y, i would need to distribute A and X and Y via this method you are proposing, and make sure A loads AFTER B (if B exists) via code, and make sure X and Y gets loaded first before A, and so forth and so on.

There needlessly needs to be a whole addon internal system of accounting for all these dependencies and loading order because the entire system is to load addons via loadstring() and there exists all sorts of complications when it comes down to debugging these addons. Particularly, because the addon is loaded as a single string via loadstring(), there is only 1 line of code and all errors if traceable on the stack just traces itself to line 1 to an anonymous function.

Now lets say my addon A now requires a savedvariable, since we don't have a TOC file so to speak for my addon A as it is distributed via ingame transmission strings (bittorent if you will), the host security addon that does all this transmission/receiving has to handle savedvariables of all its subaddons distributed this way directly. Now lets go a step further, if my addon A wants to include some graphic, mp3, wav file or other form of media, this isn't even possible with Data Exchange Manager.

In fact distribution of addons in game in this manner may not be something Blizzard wants to encourage, if it eventually leads to data exchange via bittorent p2p style exchanges and causes network/bandwidth issues. Already, addons like Omen, transmit 25x24 messages every 2 seconds to update threat to everyone else in the same raid and these addons do take up a significant amount of net traffic having to handle 600 addon messages per 2 seconds per raid.

Lenja, while what you say is all possible, it introduces a whole new slew of issues and problems that are already handled by the default wow.exe and game UI (loading dependencies, optional/reqdeps, savedvariables). It also cannot handle any addons that wish to use its own media (sound, graphics), and it has security issues as mentioned in the posts above, and is not a better system that an addon reviewed manually on a website like wowinterface, other than via the ease of distribution. Again, debugging such addons from a end-user perspective without any useful stack trace (due to loadstring()) again hinders development of the addon to the addon developer when an error that occurs cannot be easily traced from a user report.

Please think carefully what you are proposing, and consider again if what you are suggesting is worth the effort at all. While DEM might be useful for exchanging and spreading small code snippets, it will never replace real addons that are more than just 10 lines of trivial code.

Edit: To add stuff.

Why don't we just try to find a common denominator? You don't like sending addons ingame, fine. I can put that on the shelve and try to slip it in later. If it catches on fine, if not then fine too.

So what should we (or I) then start working on? A general way to exchange addon data? I've seen in the API that TOC information is available at runtime. Maybe one could find out which save variables are used and pack them into a string? This sounds more like a problem that could be solved more easily with one of your libraries.

Just put some data exchange interface into your profile DB lib thingy and BAAAM, all Ace addons are able to export settings to other users ingame. It's probably more complicated than that but still... you don't need me for that.

Oh well, I see I'm the only one who believes in the idea and that there doesn't seem to be any interest to pursue this any further. No matter, I've already spent enough time on this so I might as well finish it properly. I'll just start converting addons into modules to get a good package and then let users decide whether they like it or not. That way it'll resolve itself.

SavedVariables aren't a problem. There are always ways to getting them saved even if modules lack TOC files because the base addon has one and can provide tables if necessary.

I don't know if error reporting can somehow be solved but I will certainly look into it.

I still think this idea could turn into a great solution for very small snippets and mini addons. But I guess I shouldn't have described it as an alternative solution to the current addon system in general. People here seem to feel threatened because they have a great system in place already (e.g. Ace Updater) and because my idea might cause some inconvenience later on even though there's not much dispute that it may work in principle.

So, better to squash the whole idea exactly because it could work. I'm sure there are much smaller addons that aren't all that useful that haven't been dissed in that way.

I've even signalled that I'd be willing to adapt the idea to something the community might feel more comfortable with without getting any positive responses. I just find it hard to accept that people think the idea is THAT bad even with all the technology in place to make sure things don't get out of hand.

Of course everyone is entitled to their own opinion. But I guess I'm just a bit disappointed. People are usually so inventive. For any other addon, they may spot possible problems and even give hints as to how they might be solved later on. Not so here.

IMO, very very very few addons could bennifit from this. First thing that comes to mind is Bossmods... bout it

Quote from OrionShock »

IMO, very very very few addons could bennifit from this. First thing that comes to mind is Bossmods... bout it

RDX tried to do this. It was considered successful from the viewpoint of distributing code digitally in-game for immediate execution, but it never really took off or was accepted into mainstream community because

A) RDX was a pay-to-use addon. Free alternatives were available - mainly bigwigs, deadly boss mods and ct-boss mods.

B) It allowed the raid leader/officers arbitrary and complete control of your wow.exe and almost every aspect of your character - this is important, all it takes is one malicious person to distribute malicious code and bam, you could very well mail away all your gold next visit to the mailbox automatically, or self-delete every piece of gear in your bags.

While (A) obviously won't be an issue, (B) will remain an issue for any code distributing mechanism without peer review. All it takes again is one compromised account by a hacker to distribute malicious code, and unsuspecting members (not knowing the account was compromised) accept the code and run it. While a website distributing addons like wowace/curse/wowui/curseforge/wowi can suffer the same issue, they have tighter reviews and security measures and need to maintain the confidence of users to visit their websites.

While the original poster suggests code signing and stuff via a website, and users then obtain signed code from said website which is considered "trustworthy" for ingame distribution so that others need not visit the addon website regularly for updates (to overcome (B)), I still see this as a cumbersome method for all the reasons I listed in my previous post. The one I feel most strongly about is the almost impossibility of code debugging a loadstring()ed piece of code.

@Ringleron:

SavedVariables aren't protected in any way from being read or written to by other addons on the same client. This is usually not a problem since nobody does it anyway (why would you) and it'll only cause problems later on if you did. But in my case, the base addon can register two variables in its own TOC file which it doesn't need itself but which it provides to modules for saving data. You might access data in this fashion: DEM_SaveVarPerChar["MyModule"]["x"] = 3. You could also use some load/restore handlers or mapping metatables to make these variables more accessible (shorter variable name).

You make it sound awfully childish. :D Of course I'm not asking everyone to agree with my idea, far from it. But any thread where one person is trying to advocate an idea and 2-3 people are strongly and relentlessly opposed is doomed. I just didn't expect that much resistance. Normally people will tell you if something is a bad idea or not useful but not actually fight you every step of the way. This just made me think that maybe not everyone is completely objective about this. And if that was the case I would be right to feel disappointed. But seeing as how the idea still gets lots of negative feedback even after I've given up, maybe people are objective and really do just think it's as bad as they say. I'm new here so I don't really know the community yet.

@lilsparky:

A friend of mine has about 200 addons which consume ~40MB of memory. He tells me that if he didn't use WoWAce it would very well use twice as much or more. I'd never want to completely replace the current addon system which works really well. And there's really no point in trying to load all 200 addons via my idea and double the memory requirements. This is intended for maybe 20-30 small addons, each about 10 to 20KB in size.

@Xinhuan:

I've done some research and it seems that any global code loaded by RunScript() somewhere that produces an error will result in an error message that includes the first line of the loaded code. So you could include a comment such as "-- Module:MyMod --" that points in the right direction. It would look like this:

Date: 2008-09-01 09:17:35
ID: 1
Error occured in: Global
Count: 1
Message: [string "-- Module:LFGVoyeur --..."] line 69:
 attempt to call global 'LFGVoyeur_Pull' (a nil value)
Debug:
 [C]: LFGVoyeur_Pull()
 [string "-- Module:LFGVoyeur --..."]:69: LFGVoyeur_Refresh()

Using loadstring() you can specify a second parameter naming the code block as described on WoWWiki. Thus there shouldn't be any difference whether code has been loaded from a Lua file or by loadstring(). Who knows, maybe the client itself uses loadstring() to load code from files specifying the filename as second parameter.

If an error occurs, the base addon could even process the error message directly and display the offending portion of code in an ingame editor/debugger. That's again something you can't do with normal addons.

Why am I always the one who has to point out these things? :D

I'd like to point out again - that this is not the community that will find such a thing acceptable. It goes against alot of things that this community dislikes.

That doesn't invalidate it, it just says that you shouldn't expect alot of support from the audiance here.