• 0

    posted a message on What do you think of a portable offline WoWWiki?
    The Wiki XML dumps only use XML as the "storage" method, not for article markup. There's page tag for a page, title, etc.. but the actual "meat" of the articles is still in the Wiki markup.

    It appears I will have the rights to distribute the SQL dump, as long as I reference (and include a copy of) the GNU Free Document License. Compressed, the SQL dump is about 100mb (before importing) - I'll probably use gzip or bzip2 so it can be easily piped to mysql.

    Unfortunately there's no real way of distinguishing API related pages programmatically (most are labeled "API" but some important pages like Events aren't). The best I can do is try to remove other distinguishable non-API pages. Currently I have filters for Users, talk pages, quest pages, and image pages (no image files are included in the dump, so no need to keep the descriptions). With these four filters, it drops to about half the articles - but I didn't have time yet to see how it affects the size of the database.

    I will probably just supply 2 SQL dumps, the full version, and the "slim" version. And if it's popular enough, maybe I'll work on slimming it even moreso down the road.



    Update:
    I added more exclusion filters, currently they are: User Pages, Talk Pages, Image Pages, Quest Pages, NPC Pages and Item Pages - as most of the info will not be needed offline. The import from XML takes about 2.5 hours with all those filters, as there's about 75% less articles. However, I'll still provide the even faster SQL dump for the "slim" database - this is about a 1 minute import (instead of the 2.5 hours from XML).

    For the people who prefer the XML import (slower):
    I optimized the code of WoWWikiSplit (my C++ program for filtering it) for speed. I hard-coded the search and comparison keys, instead of using the POSIX strnicmp, strlwr, and strstr functions. Among other optimizations I reduced the CPU time (not including disk I/O) of WoWWikiSplit of a whole database with all filters from about 5 minutes to 2 seconds. When you actually do an import using WoWWikiSplit with all filters on a WoWWiki XML dump, it'll take about 2.5 hours - about 2-5 minutes of that will be CPU and disk I/O time due to WoWWikiSplit itself - the rest is MediaWiki's importdump script. I think that ratio should be acceptable given the benefit WoWWikiSplit gives.

    Additionally WoWWikiSplit get's the Split part of its name by splitting the import into sections. The default is about 16MB sections. If it needs 19 splits, you can have it do 1-7 for now, then do the rest another time. Between each split it gives you a 5 second "ctrl+c safe" window. Hitting ctrl+c at that point will gracefully close WoWWikiSplit at a point which it can be safely resumed later. That will all be explained in the procedure.

    The source code for WoWWikiSplit will be supplied along with the compiled binary when it's time.

    This slim database takes up about 300mb after all is set-up. I think things are looking better and better in the practicality aspect.
    Posted in: General Chat
  • 0

    posted a message on What do you think of a portable offline WoWWiki?
    Because it's easy for me to do, I know the procedure, and I prefer to maintain MediaWiki's search functionality.

    If you'd like to give some insight into how to convert the XML to HTML, in a way that maintains format (and supports the ParserFunctions extensions), that's fast and small that'd be great. I don't know if you've downloaded the XML dump already, it's about 300MB uncompressed, and about 164,000 articles. The static HTML dump will probably be larger than 300 MB (probably not by too much) because of the extra HTML markup.

    I'm not reall sure if CHM is a good approach for such a large database.

    Edit:
    BTW, if I do get permission to redistribute the SQL dump (I imagine I should, the Encylopodia guy [haven't kept up, dunno if he still does] used to distribute processed Wikipedia databases); then the import process is only about 15 minutes. That should be much faster than an HTML approach, as the static HTML dump will require parsing of each and every article. The dynamic parsing on MediaWiki only parses the pages as you view them (then optionally stores them in a cache).

    Furthermore on my tone. I really do apologize if I sound angry - I'm really not in the mood (not because of you or this) to be replying. I really am curious of if you have any insight to a fast HTML approach or anything else.
    Posted in: General Chat
  • 0

    posted a message on What do you think of a portable offline WoWWiki?
    In an effort to reply to most of the questions without having a mess of quotes, I'm just going to touch on key issues, each labeled with '--'

    --I didn't know WoWWiki had XML dumps?
    http://www.wikia.com/wiki/Wikia:Database_download

    --Size and performance issue
    I decided to try using MyISAM tables instead of the default InnoDB for the MySQL database. The result is a faster, smaller database. Now it is just about 900MB with search index done - about 700MB without search index. This is also with using my program to filter out the "talk" and "user" pages during import (about a 33% reduction).

    --But isn't InnoDB better than MyISAM for a wiki?
    Technically it is, however, the advantages are really for supporting many concurrent users doing reading and writing at the same time. For your offline wiki, you probably won't be doing any writing, and if you do, you won't be editing multiple pages at the same time (maybe consecutively, but that's different).

    --Import time
    Now that I'm using MyISAM, the import time from an XML is only about 12 hours - I know, only. However, if I do that work once, and make a SQL dump to distribute, the import is only about 15 MINUTES!! Lol, that was kind of exciting to see. The not so exciting part -- I don't know if I have the legal rights to distribute the SQL dump. I contacted a WoWWiki admin who referred me to contact the Wikia community. I am now awaiting a response from them.

    --Performance on a flash drive
    The initial set-up requires a lot of writes, and should therefore be done on a hard-drive for both speed and to prevent undue wear on the flash memory. However, once it is all set, it can easily be copied to a flash drive and the read performance is not bad at all. You do see a difference between HDD and Flash reads, but it is both understandable and not really appeciable (maybe a second or two longer on some page loads).

    --I don't want to locally install a Web Server!
    I am using a portable package for that, "MoWeS" by CH Software. It contains functioning apache, mysql, php5, and MediaWiki applications; however, they are "portable apps" as in there is no "permanent" install. They are designed to run in the folder they are extracted in without attaching in anyway to the host. You can even move the folder around at will (while the servers are down), and they'll still work next time. This is especially good for Flash memory where the volume letter may not always be the same.

    Additionally, the package is installed with "default" settings. There's very few changes needed to make it work for the WoWWiki dump. It's mostly fool proof - but I'm sure somebody will mess up.

    --User level restrictions
    None of the code requires administrative priveledges. You may get Firewall warnings, however all the "internet" access is done through the localhost loopback interface. So even if a firewall blocks it, the localhost loopback interface should continue to function. The reason the Firewall will come up is because Appache and MySQL will each create a socket for listening. Anytime a listening socket is opened, a good firewall will let you know about it - however, a good firewall shouldn't affect the localhost loopback interface (I'm not sure if it's even possible or not). I actually recommend blocking the programs from WAN access, since it's not needed.

    --I'm opening ports!?
    MoWeS comes with an option (default on) to only allow access through the localhost interface. Additionally, you can safely and easily block both the appache server and the mysql server with your firewall. The only inbound access will be through the local loopback interface.
    Posted in: General Chat
  • 0

    posted a message on What do you think of a portable offline WoWWiki?
    That's actually not a bad idea I might want to look into.

    I'm currently modifying my CLI program that splits the XML dump up, so that it has an option to exclude User and User Talk pages, as well as an option to exclude all Talk pages. That should basically cut the number of articles down. Which in turn should speed up the import time, and decrease the size.

    The user pages are nice, but are not really needed for an offline dump I think. The Talk pages sometimes have good problem/solutions; but the majority of that information is likely useless to the general reader (good to have online, not really needed off). However, since some might disagree with the necessity of that information, I'm leaving it as an option for both.
    Posted in: General Chat
  • 0

    posted a message on What do you think of a portable offline WoWWiki?
    AFAIK that's just a book. It's not really "searchable" and it lacks a certain kind of interactivity I crave... I hate books. It does come with the "companion site," but that requires internet access.

    Plus, if we're talking portability without internet access, we're probably talking laptop on the road. A software dump would already be on the laptop; wheras with the book you'd need to have both handy. That is unless people are still programming add-ons with punch cards.
    Posted in: General Chat
  • 0

    posted a message on free premium account for addon authros?
    I am just wondering on the duration of the complimentary premium upgrade. It appears to me that it's only for the first month, and I didn't see anywhere that explicitly stated how long it lasts.

    After this month does it reduce to the standard subsciption option? Or as long as we maintain an add-on with 1k+ (current limit) downloads, will we maintain our premium status?
    Posted in: General Chat
  • 0

    posted a message on What do you think of a portable offline WoWWiki?
    TL;DR version: If you decide to stop reading midway through, you're probably not interested in the idea anyway, so you won't be missing out. It starts out with the basics then gets more detailed as you read.

    People may think it's foolish, so let me explain. You may be familiar with the MSDN library.

    I want to get a laptop (need funds) which means I'll be add-on programming on the road (yes, I'm that much of a dork - no offense to others). I won't always have internet access; however, the wowwiki reference will still be a desirable resource.

    Currently for myself, I am setting up a portable offline MediaWiki to access a static dump of WoWWiki. It's portable in the sense that it doesn't require any "install" on the running system, but it requires Windows (not OS portable). Also, when all is set up and ready to go, it consumes about 2GB.

    Before I spend the time to make a "tutorial," I want to poll to see how many people would actually use it. I know it's an "off" idea, and kind of specific purpose, so I don't expect many people to vote for it.

    Current Constraints:
    Existing "reader" applications like WikiTaxi, BZReader and WikiFilter were not too friendly with WoWWiki's dump for some reason (most didn't even work at all with it). So I use a portable apache, php, mysql, and mediawiki solution.

    Importing the XML dump into the sql database can take up to 48 hours on a 2.6 ghz dual core. I have to look into if I can legally distribute a sql database backup that I make from the XML, which reduces the import to about 1 hour, and greatly simplifies initial set-up.

    In case I can't distribute the SQL dump, I wrote small C++ CLI program that reads the large XML from STDIN (allowing for programs like bzip/gzip to be used in a pipe), and breaks it up into smaller (default 16MB) XML files that are imported. This allows for "pausing" so you don't need 48 consecutive CPU hours to import a whole XML dump.

    I'm currently running my CPU at 3.2 ghz to see if it goes a little faster, but I'm likely only going to shave a few hours off since most of the time is spent accessing the hdd. And the last time I attached a Dremel to the platters to overclock my hard drive, it didn't go over so well for data integrity - jkjk.

    In the end, all of the text content and search features work - however it's missing images and the WoWWiki look. Functionally it's great, but fashionably it may lack - however you'd likely be reading it to make an add-on, not to take it to the prom.
    Posted in: General Chat
  • 0

    posted a message on Why use ace3?
    Ah, I didn't know that only one copy of the latest version in the lib stays resident. I know Ace3 is like 99% fat free, but my trimming was more in the sense of: "Why have a 99% fat free meal, when you just need some 99% fat free dessert."

    However, if Ace3 only keeps one running copy, me programming my own desserts would actually be adding more fat. Since Ace3 will likely be installed anyway from some other add-on.

    I was thinking of it more along the lines of (if you compare it to C++) that Ace3 worked more like a added statically linked object, as opposed to an externally linked library. And I thought that object got recompiled for each add-on that included it. Often I make a lot of assumptions, and they're usually wrong, haha.

    Still, trying to learn some of it on my own will be fun. And I think it's usually beneficial to have a decent - if not intimate - knowledge of the inner workings of the API. Even if you're going to abstract it with something like Ace3, knowing what Ace3 is actually wrapping sometimes helps you use the library more effectively.

    But usually in the end it just comes down to preference. The difference between using the library and writing your own functions - if you do it right - probably may only be a small difference in memory usage and CPU time.
    Posted in: Ace3
  • 0

    posted a message on Why use ace3?
    I started programming C about 10 years ago (I don't really remember when I was in the 5th grade - of course it was stupid stuff, but it was fun). So when I came into doing WoW addons, I already had programming experience (I'm currently in college for it). However, I was not at all familiar with Lua functions or the WoW API; so I hit the proverbial brick wall at first.

    I had an Addon idea that I wanted to hit the ground running with, and didn't want to take the time to familiarize myself with the WoW API and such first. That's where Ace3 made things a lot simpler. The documentation isn't really full and detailed, but it was enough. And the fact that most of the functionality that I would've needed to research on my own was in one lib package made things easier.

    When it comes to "reinventing the wheel." When I first got into add-on programming, I like to think of it as I just jumped on the bike with Ace3 as my training wheels. However, now that I'm getting more comfortable, I think I want to remove the training wheels. This is a matter of preference, as I don't feel I need the full functionality of the libs (even the modules I use), so I want to trim the fat in a sense. Plus, it'll be fun.
    Posted in: Ace3
  • 0

    posted a message on Lua generated frame: FontString tool-tip.
    Actually, using a button would probably be a much better idea - as the user will interact with it to do something with that item. If I use a button, it'll be much more obvious to the user "Click Me." As opposed to me having to add additional "Click Here..." instructions.
    Posted in: Lua Code Discussion
  • 0

    posted a message on Lua generated frame: FontString tool-tip.
    I finally decided to try to learn WoW GUI code. However, I really prefer NOT to do the XML way. The reason is a matter of preference, I can much better visualize what's happening and what's going where if I create it programmatically. Unfortunately, it appears most of the examples out there are XML.

    I successfully created the main frame, then inside of that frame I have a FontString object which I'd like to give an item tool-tip on hover. However, it doesn't appear that the FontString object can get scripts set.

    To accomplish this, I'm thinking about creating the FontString, then creating an empty, invisible Frame over the FontString with the same dimensions. Then defining Enter and Leave for that invisible frame to show the item tool-tip.

    Part of me says: "Oh good, that works." But I'm also cautious about the efficiency of using a dummy frame like that. Is there a better way?
    Posted in: Lua Code Discussion
  • 0

    posted a message on Dumping multiple returned values in a table
    Quote from egingell
    Because the client (or OS, I don't know exactly how RAM works) allocates RAM for a new throw-away table every time. Which is horribly inefficient.


    I know a little bit about this in terms of Windows - I'm not familiar with linux or Mac API.

    I'm a little disappointed, because I wrote a LOT of info on how memory allocation is handled, and why there's "Garbage" and "Garbage Collection" in WoW. But it really came out as a REALLY long rant.

    I'm not ALL that familiar with Lua (as you can probably see by my many questions). However, I am very familiar with C++, Windows API, and x86 Assembly programming. I am not a big fan of predefined data structures (i.e. Tables in Lua). ANY C++ program that I write, I ALWAYS write custom code for any advanced data structure. I never use templates which may be bulkier than what I need (always Tables in Lua, although since I can't write custom code, I'm stuck when it comes to NEEDing an array:().

    Basically, using Tables the way [OP] wanted to is horribly inefficient. It will induce RAM fragmentation inside of WoW's memory manager, which can (and will) eventually lead to RAM fragmentation inside of the OS. Which will eventually lead to slow downs as things need to be organized.

    Does the code look neat, and does the code "look" efficient? Yes. Does using tables to retrieve multi-return values like that look attractive to the programmer? Initially, yes. Are you trying to swat a fly with a Cannon Ball? YES. Is that Cannon Ball going to mess a bunch of things up in it's path before (and after) it gets to where you want it? YES!

    I strongly recommend you use the function that has already been suggested by others in the thread. After all, the inefficiency won't just affect YOUR addon. It'll affect the user's entire WoW, and eventually entire computer, experience.
    Posted in: Lua Code Discussion
  • 0

    posted a message on button to do /random
    Also, you might want to look at the following GlobalString: (taken from enUS GlobalStrings.lua)
    RANDOM_ROLL_RESULT = "%s rolls %d (%d-%d)"; -- Example: Jeremy rolls 4 (1-10)


    As this is the format that the roll will be displayed in the chat window.
    Posted in: Lua Code Discussion
  • 0

    posted a message on AceLocale-3.0: Can you force a locale?
    I know my title may not have been the best, but let me try to explain.

    AFAIK AceLocale-3.0 just uses the language that the user has their client set to (if that Locale is defined for the add-on).

    But say for example, a user is using the enUS client and therefore doesn't have any language options, but would enjoy the add-on better if it was in German. Is there anyway I can add an option to my add-on to change the language to German?

    I currently have an idea for this, where the code is normally:
    local L = LibStub("AceLocale-3.0"):NewLocale("myaddon", "deDE", false);
    if L then
        L["Text"] = GermanText;
    end


    I'm thinking maybe if I do something like:
    local L = LibStub("AceLocale-3.0"):NewLocale("myaddon", "deDE", false);
    myaddon:UsedeDE(L ~= nil);
    
    function myaddon:UsedeDE(force)
        if (force) then
            L["Text"] = GermanText;
        end
    end


    Then later in myaddon.lua I could have a command for German that runs:
    myaddon:UsedeDE(true);


    Would that work, or is that grossly inefficient and may cause problems? Is there a better way to do it?

    Edit:
    For the 'myaddon:UsedeDE(L ~= nil);' I assumed L would be nil and not false. I'm not actually sure. I might be better off leaving that as just (L) instead of (L ~= nil), but I'd rather pass a bool than a table. Although, (I don't know the inner workings of Lua) I'm not sure if the table would be passed by reference or another copy made (this is what I'd want to avoid for garbage).
    Posted in: Lua Code Discussion
  • 0

    posted a message on Version history (old files)
    Lol, ya, I always thought that box was .. semi-annoying. I can see a point to it if multiple people manage a project, and someone deletes a file and everyone else wonders why. Or even if you tend to forget why you did things, it may be a good reference. But I think it should be optional - or have default text in it, "Because I said so." Lol, or something similar.

    Thanks for the info though.
    Posted in: General Chat
  • To post a comment, please or register a new account.