Authors CurseForge

As presented I would have to join the crowd that is saying no to the idea. That said however at the core I think the idea would be very good. That is where it would need to be implemented though, in the core code by Blizzard. We are seeing addons that read and write pretty large SavedVariable files and static lua structures. A lot of this information would be far better off being in a keyed dbm, isam or even perhaps an sqlite db.

Maybe suggest the idea of an isam or sqlite db data store to blizzard somehow?

They already have.... just it has you dumping a whole text file and reading a whole text file at a time.... there is no reason they could not sandbox the API for this to only read a SAVED/modname.db or STATIC/modname.db file... there is no reason for them not to other then not wanting to put the resource into it, all other issues can be worked around without too much effort .

Quote from twobits

Maybe suggest the idea of an isam or sqlite db data store to blizzard somehow?

How would you make the case to Blizzard?

"Hey guys, it would be really cool if you spent hundreds of developer hours on adding in-game SQLite support for the handful of addons that might actually use it."

The default interface drives the API design and I don't see Blizzard adding anything to the default interface that would require SQLite.

Hey guys, it would be really nice if you added features to allow the use of data files that are not required to be fully memory resident. This would let some of these addons not have load freezes like they sometimes do and run on more of your clients hardware. In addition it would allow more addons to effectively cache more information client side reducing your bandwidth and server loads. The cost of the programming time in the larger picture should be negligible given that the code for this is already done and in the public domain... so all they need to do is the removing of directory paths/filename specifications and such in a stripped down version of the lua glue available.


Given that the current implementation of their saved variables stuff is taking over 20 minutes to write out the files, they need to improve this area of the code anyway.

Quote from Mikk

Mmm you missed half the point I think. It's not just that we're not allowed access to random files (which is good to avoid trojans and stuff), it's that we're not allowed outside world communication at any other point in time than login / reload / logout.

Having said that, I wouldn't mind some sort of key:value storage that's more efficient than constructing and parsing Lua - the files could certainly be cached in RAM and read/written to at the usual times. SQL though? I think not.

(And Pasta's argument applies. Addons storing boatloads of data are kinda pushing the boundaries of what Blizzard intends for addons to be doing, so I wouldn't hold my breath for new storage mechanisms.)

Don't think I missed the point, as that was why I said other issues could be worked around with out much effort. A ram overlay with delayed flushes was pretty much what I had in mind when I was writing that as you yourself put. As for SQL... shrug... I used to hate it, but sqlite3 seems lean enough that it still would be more efficient then an lua solution without sql. Of course people would probably do dumb things like forget indecies and such... which is why also mention the isam option which is what I would lean towards myself. I would prefer the sqlite3 option to what we have now though. All in all I think we are in agreement that that a keyed storage system would be a nice alternative to the current SV system.

I am sure it is pushing what they envisioned... between auctioneer, lightheaded and other data driven addons though it does add up to a lot of memory/load time.

Quote from Wintrow2

I think you are using a TAD bit too much addons... Or your hardrive is REALLY slow :p.

Probably too many addons, since I am messing with some things lately... but the real issue is, not a slow harddrive, but a network share when one of my harddrives failed I am now using. It seems however they write out the saved variable files is horrid and really shows as such when used like this.

Like I said there are other solutions to the issue then that path.... better ones IMHO that do not take all that much work for the gains they could give. If they really need to prevent updates while the client is running (and I can't see why they make this a design goal, and yes I know the knee jerk reaction is going to say bots, but again... there are counters to this in other ways) they can take some hash signatures of the file and start to return all nils for reads if it is changed during a run for example.

I do think people are right that blizzard won't do this though, if the people in the addon community argue why blizzard can't do it, when it would help them, and they don't even have to do the work, getting blizzard to put the mental resources into it would be even harder.

A packed, binary format for the SavedVariables files would certainly speed-up load times but it would make it harder to see what's going on in them.

Back in alpha, you could load precompiled lua chunks similar to how we can use loadstring now. Someone took advantage of this and used it to hide some mischevious features in an addon. I don't think you're going to see non-human-readable SavedVariables for the same reason, transparency.

Most users aren't going to look too close if you tell them to download AddonName.lua and stick it in their SavedVariables directory, but it's important that they can easily see what's in that file. With a more efficient and almost certainly not human-readable SavedVariables format, you'd have the ability to hide code in SavedVariables files and then loadstring it later.

Quote from Mikk

... except we don't download to ...\WTF\Account\BLAH\SavedVariables ...

UI compilations often do.

hmm.... yeah, was not aware of an eval/loadstring like function in the scripting language yet. Still very new to lua. Now you would need to take it out or add something like taint tracking. If indeed an addon can do something malicious, I thought they are supposed to be sandboxed enough to not allow this already?... depends on how sandboxed they really are I guess and what would be wanted to be prevented.

Really though, it is not so much in my mind a feature for just speeding up load time, as it is to reduce the memory footprint of things like lightheaded or everyquest. So a text file format that supported isam and record structures could be designed... more work now then just plugging in existing isam engines though....

In regards to the .wdb files, that would more then likely give me a lot of what I desire if they gave it an api to read them, and created stand alone tools to create them. I am assuming they are at least keyed files though probably not isam.

Quote from twobits

Really though, it is not so much in my mind a feature for just speeding up load time, as it is to reduce the memory footprint of things like lightheaded or everyquest. So a text file format that supported isam and record structures could be designed... more work now then just plugging in existing isam engines though....

It really wouldn't reduce the memory footprint.

Addon data cannot be read or written except during load, reload, and exit. Therefore all addon data must be stored in memory while the game is being played.

ISAM and WDB are file formats and have no bearing on how the data contained in them is stored in memory. Lua's tables are about as efficient as an interpreted language can be. They even have separate 'array' and 'hash' sections so that sequential numerically-indexed data doesn't suffer from hash overhead (details).

With any data storage, in memory or on disk, we have to make trade-offs between access speed and storage size. RDBMS software has the luxury of large disks to store its information on and therefore uses lots of space to make access fast. WDB files have fixed-length records which makes the files small but has no indexes so access would be slow if they weren't loaded into memory when WoW starts up.

We can store data efficiently using lua tables if we use tables like arrays (sequential numeric indexes) but then we have to search the entire table to find the entry we want, and that's slow. We can use tables like hash-maps (key/value pairs) so that finding a particular entry is fast, but then we have to store the key and not just the value. Additionally hash-maps need extra space to make inserting new values fast. This means that hash-maps take up more space.

WoW is a video game, and the lua interpreter only runs inbetween frame redraws. This means that we are almost always interested in increasing the speed at which data is accessed. Because we need fast access, the data will take up more room in memory. Using lots of room in memory isn't bad until you run out of room.

When you run out of memory for WoW, you need to figure out what you don't need anymore and get rid of it. Just like when your hard drive runs out of room.

Quote from Wintrow2

I think you are using a TAD bit too much addons... Or your hardrive is REALLY slow :p.

WoW can be stored on paper tape, didn't you hear? :-)