Me as a "sheep" is using this script Jerry posted, but i wont consider it tested or anything. I like to tryout things so ive taken a look at the script and used it.
If you arent doing Backups before using a script posted somewhere in a forum its your own fault anyway...
If i get Errors i would first restore my Backup and see if the script caused it. But until now the script worked good for me and reduced my loading time so why shouldnt i use it :P
Why shouldn't we change the .zip script so that it removes embeds.xml if it exists?
It seems to me that authors could then choose to use that convention or not at their discretion. Addons that didn't use that convention would still work the way they do today. It also seems like a far less controversial solution than editing the .toc file.
I fail to see how this is a bad thing or an unreasonable request...
Concatenating files won't give any meaningful performance increase (in addition to just being a bad idea). Using the following script to generate addons with varying numbers of files (between 10 and 10000):
#!/bin/env python
import os
path = "c:\\Games\\World of Warcraft\\Interface\\Addons\\TestAddon"
tf = open(os.path.join(path, "TestAddon.toc"), "a")
for i in range(10):
f = open(os.path.join(path, str(i) + ".lua"), "w")
f.write("local a = 1")
f.close()
tf.write("%d.lua\r\n" % i)
tf.close()
With my laptop's rather slow and low quality harddrive, each additional file makes the addon load take an additional .0015 seconds. I have 1085 .lua files in my addons folder. Reducing this to 171 (one per addon) would save 1.371 seconds. (Less actually, as parsing the "local a = 1" takes a non-zero amount of time)
I'm not quite sure why I just spent the time to test that.
Concatenating files won't give any meaningful performance increase (in addition to just being a bad idea). Using the following script to generate addons with varying numbers of files (between 10 and 10000):
#!/bin/env python
import os
path = "c:\\Games\\World of Warcraft\\Interface\\Addons\\TestAddon"
tf = open(os.path.join(path, "TestAddon.toc"), "a")
for i in range(10):
f = open(os.path.join(path, str(i) + ".lua"), "w")
f.write("local a = 1")
f.close()
tf.write("%d.lua\r\n" % i)
tf.close()
With my laptop's rather slow and low quality harddrive, each additional file makes the addon load take an additional .0015 seconds. I have 1085 .lua files in my addons folder. Reducing this to 171 (one per addon) would save 1.371 seconds. (Less actually, as parsing the "local a = 1" takes a non-zero amount of time)
I'm not quite sure why I just spent the time to test that.
How much time does it take to load if all the lua files are missing?
How much time does it take to load if all the lua files are missing?
Average of .0013 seconds per file over a total of 40 logins/UI reloads with varying ToC sizes. Unless it's only happening on initial logins and warmup is not recording the time spent in error routines (entirly possible), I can't reproduce any speedup from empty files existing. I'd consider having a useful FrameXML.log to justify removing the dead ToC entries, though.
How much time does it take to load if all the lua files are missing?
Average of .0013 seconds per file over a total of 40 logins/UI reloads with varying ToC sizes. Unless it's only happening on initial logins and warmup is not recording the time spent in error routines (entirly possible), I can't reproduce any speedup from empty files existing. I'd consider having a useful FrameXML.log to justify removing the dead ToC entries, though.
So you have a toc which references many lua files which dont exist, and one that references blank files?
So you have a toc which references many lua files which dont exist, and one that references blank files?
Yes.
More testing:
There appears to be a 140 second time limit on logging in.
Warmup's login time is consistantly ~10 seconds shorter than my manually timed amount, regardless of addon stuff. With no addons it takes me about 10 seconds to log in, so this seems accurate.
Putting actual code into the files has no mesuarable effect -- the additional time for each file is consistantly .0013 - .0015 seconds, regardless of whether or not the file exists as long as the total amount of lua is kept consistant (and lua parse time is incredibly trivial - 35k lines (8.8 MB) of non-executing code split into 100 takes ~2 seconds to load)
Regardless of what I do, I can't reproduce a performance problem from a large number of files. Even 50k files only takes 60-75 seconds (and produced a 8 megabyte FrameXML.log)
So you have a toc which references many lua files which dont exist, and one that references blank files?
Yes.
More testing:
There appears to be a 140 second time limit on logging in.
Warmup's login time is consistantly ~10 seconds shorter than my manually timed amount, regardless of addon stuff. With no addons it takes me about 10 seconds to log in, so this seems accurate.
Putting actual code into the files has no mesuarable effect -- the additional time for each file is consistantly .0013 - .0015 seconds, regardless of whether or not the file exists as long as the total amount of lua is kept consistant (and lua parse time is incredibly trivial - 35k lines (8.8 MB) of non-executing code split into 100 takes ~2 seconds to load)
Regardless of what I do, I can't reproduce a performance problem from a large number of files. Even 50k files only takes 60-75 seconds (and produced a 8 megabyte FrameXML.log)
Use ProcMon (or FileMon) instead if you really want to see what is happening. I warn you though, its addictive.
W:\World of Warcraft\WoWTest\Interface\AddOns\Prat_Modules\modules\PopupMessage.lua
W:\World of Warcraft\WoWTest\Interface\AddOns\Prat\Prat_Modules\modules\PopupMessage.lua
Unless embeds.xml is replaced with an empty one when libraries are disembedded, I don't see any gain from having libraries this way, as FrameXML.log is still created.
I've managed to make a Python script to comment out externals in embeds.xml files, similar to the cleantoc script posted a few pages ago.
#!/bin/env python
import os, msvcrt
ADDON_PATH = "C:\\Games\\World of Warcraft\\Interface\\AddOns"
UTF8_MARKER = "\xEF\xBB\xBF"
for addon in os.listdir(ADDON_PATH):
path = os.path.join(ADDON_PATH, addon)
xml = os.path.join(path, "libs", "embeds.xml")
if os.path.isfile(xml):
print "embeds.xml found in %s" % os.path.join(path, "libs")
f = open(xml, "r")
lines = f.readlines()
f.close()
f = open(xml, "w")
for line in lines:
if line[-1:] == '\n':
line = line[:-1]
if line[1:7] == "Script":
#print line[1:7]
m = line.split('\"')
#print m[1]
if not os.path.isfile(os.path.join(path, "libs", m[1])):
print("Unable to find file \"%s\"" % os.path.join(addon, "libs", m[1]))
f.write("<!-- %s -->\n" % line)
else:
f.write("%s\n" % line)
else:
f.write("%s\n" % line)
f.close()
char = 0
print "\nPress any key to continue..."
while not char:
char = msvcrt.getch()
And slightly off-topic I've made a very very messy script to clean out old changelog files for those who don't like to delete beforehand and re-extract due to custom/3rd party addon files.
#!/bin/env python
import re, os, msvcrt
ADDON_PATH = "C:\\Games\\World of Warcraft\\Interface\\AddOns"
currentAddons = [f for f in os.listdir(ADDON_PATH) if os.path.isdir(os.path.join(ADDON_PATH, f))]
c = 0
for addon in currentAddons:
addonDir = os.listdir(os.path.join(ADDON_PATH, addon))
addonDir.sort()
addonDir.reverse()
tmpold = 0
for f in addonDir:
r = re.search(r"Changelog-(.+)-r([0-9]+).xml", f)
if r:
if r.group(2) < tmpold:
print "Deleting %s..." % f
os.remove(os.path.join(ADDON_PATH, addon, f))
tmpf = os.path.join(ADDON_PATH, addon, f[:-3] + "txt")
tmpf2 = os.path.join(ADDON_PATH, addon, "changelog-r" + r.group(2) + ".txt")
if os.path.isfile(tmpf):
print "Deleting %s..." % os.path.basename(tmpf)
os.remove(tmpf)
elif os.path.isfile(tmpf2):
print "Deleting %s..." % os.path.basename(tmpf2)
os.remove(tmpf2)
tmpold = r.group(2)
if c == 0:
print "Nothing to delete!\n"
char = 0
print "Press any key to continue..."
while not char:
char = msvcrt.getch()
These are my first ever attempts at coding in Python but it wasn't at all hard, I'm not bothered about optimising them as they only should be run after each addon update.
Combined that .toc-cleaning and embeds.xml-commenting and FrameXML.log went down from 73kB to 3kB (MultiTips and simpleMinimap being the only "Ace" addons mentioned there).
Also, loading times improved a lot. Did not measure it, but it really is noticeable.
WTB "scriptable post processing" for WAU :D
Notice that the library does not need to embed other libraries as long as you can assume the library will be in the same directory which is true of all existing libraries now.
Combined that .toc-cleaning and embeds.xml-commenting and FrameXML.log went down from 73kB to 3kB (MultiTips and simpleMinimap being the only "Ace" addons mentioned there).
Also, loading times improved a lot. Did not measure it, but it really is noticeable.
WTB "scriptable post processing" for WAU :D
WAU has a command line interface, and a log file (there's your scriptable post processing)
1) embeds.xml (Refers to your embedded libs)
2) includes.xml (Uses relative paths, to include lua files in subfolders. includes.xml usually in the subfolder)
3) modules.xml (Refers to your optional LoD modules)
Wau deletes embeds.xml if you install an addon without externals. It deletes modules.xml if you unpack an addon into its LoD subaddons. You dont always need modules.xml to be deleted for you, because you can place it into one of the LoD addon's folders, and when its unpacked, it goes with that folder.
embeds.xml, should be in addondir/embeds.xml
modules.xml, should be in addondir/modules.xml
includes.xml should be kept in the same directory as the files it refers to.
As you can see embeds.xml is the link to the embedded library files.
It doesnt stop there:
Addon
Addon.toc (refers to Addon_LoD1/includes.xml)
Addon_LoD1/
includes.xml
Addon_LoD1.toc (refers to includes.xml)
This is a LoD module example. Both the Addon.toc and the Addon.LoD1.toc reference the same includes.xml.
If you move Addon_LoD1 up one level, then Addon.toc no longer points to a valid file, effectively removing the lua file references. You also gain centralization of your addons's file structur, instead of 2 toc's with duplicate data, you now have 2 tocs, pointing to a single file.
The general pattern is use includes.xml to reduce a directory down to a single file which contains all the lua references.
Another trick, if you have many modules with single lua files in them, pick a module, and place a 'modules.xml' there, then use relative (..\foo\foo.lua, or ..\foo\includes.xml) references to include all the other modules. Inlcude this modules.xml in your addon TOC.
If the modules are made LoD, when they are moved up into the addon directory, the reference to modules.xml is no longer valid.
WAU goes through and removes addondir/embeds.xml. Since you defined it, you control what references are removed.
its not always possible to use includes.xml, especially in addons designed to be flexible so the user can delete the sub folders they don't want, which will result in addon breakage if the xml file was there. So could i request WAU to delete includes.xml if unpack ran succesfully?
its not always possible to use includes.xml, especially in addons designed to be flexible so the user can delete the sub folders they don't want, which will result in addon breakage if the xml file was there. So could i request WAU to delete includes.xml if unpack ran succesfully?
Give me an example. I cant see a scenerio that couldn't be handled via the xml's but, if you can give me one. sure
its not always possible to use includes.xml, especially in addons designed to be flexible so the user can delete the sub folders they don't want, which will result in addon breakage if the xml file was there. So could i request WAU to delete includes.xml if unpack ran succesfully?
Give me an example. I cant see a scenerio that couldn't be handled via the xml's but, if you can give me one. sure
LittleWigs
Rollback Post to RevisionRollBack
To post a comment, please login or register a new account.
If you arent doing Backups before using a script posted somewhere in a forum its your own fault anyway...
If i get Errors i would first restore my Backup and see if the script caused it. But until now the script worked good for me and reduced my loading time so why shouldnt i use it :P
Why shouldn't we change the .zip script so that it removes embeds.xml if it exists?
It seems to me that authors could then choose to use that convention or not at their discretion. Addons that didn't use that convention would still work the way they do today. It also seems like a far less controversial solution than editing the .toc file.
I fail to see how this is a bad thing or an unreasonable request...
#!/bin/env python import os path = "c:\\Games\\World of Warcraft\\Interface\\Addons\\TestAddon" tf = open(os.path.join(path, "TestAddon.toc"), "a") for i in range(10): f = open(os.path.join(path, str(i) + ".lua"), "w") f.write("local a = 1") f.close() tf.write("%d.lua\r\n" % i) tf.close()With my laptop's rather slow and low quality harddrive, each additional file makes the addon load take an additional .0015 seconds. I have 1085 .lua files in my addons folder. Reducing this to 171 (one per addon) would save 1.371 seconds. (Less actually, as parsing the "local a = 1" takes a non-zero amount of time)
I'm not quite sure why I just spent the time to test that.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumHow much time does it take to load if all the lua files are missing?
Average of .0013 seconds per file over a total of 40 logins/UI reloads with varying ToC sizes. Unless it's only happening on initial logins and warmup is not recording the time spent in error routines (entirly possible), I can't reproduce any speedup from empty files existing. I'd consider having a useful FrameXML.log to justify removing the dead ToC entries, though.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumSo you have a toc which references many lua files which dont exist, and one that references blank files?
Try introducing dependancies.
Yes.
More testing:
There appears to be a 140 second time limit on logging in.
Warmup's login time is consistantly ~10 seconds shorter than my manually timed amount, regardless of addon stuff. With no addons it takes me about 10 seconds to log in, so this seems accurate.
Putting actual code into the files has no mesuarable effect -- the additional time for each file is consistantly .0013 - .0015 seconds, regardless of whether or not the file exists as long as the total amount of lua is kept consistant (and lua parse time is incredibly trivial - 35k lines (8.8 MB) of non-executing code split into 100 takes ~2 seconds to load)
Regardless of what I do, I can't reproduce a performance problem from a large number of files. Even 50k files only takes 60-75 seconds (and produced a 8 megabyte FrameXML.log)
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumUse ProcMon (or FileMon) instead if you really want to see what is happening. I warn you though, its addictive.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumAddon.toc:
Embeds.xml:
Embeds.xml refers to things using relative paths, so libs\embeds.xml gets to leave off "libs".
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumPrat.toc:
Prat_Modules.toc:
Note that both toc's refer to the same file.
Prat_Modules\embeds.xml:
So the module is valid from either path:
-
View User Profile
-
View Posts
-
Send Message
Curse Premium#!/bin/env python import os, msvcrt ADDON_PATH = "C:\\Games\\World of Warcraft\\Interface\\AddOns" UTF8_MARKER = "\xEF\xBB\xBF" for addon in os.listdir(ADDON_PATH): path = os.path.join(ADDON_PATH, addon) xml = os.path.join(path, "libs", "embeds.xml") if os.path.isfile(xml): print "embeds.xml found in %s" % os.path.join(path, "libs") f = open(xml, "r") lines = f.readlines() f.close() f = open(xml, "w") for line in lines: if line[-1:] == '\n': line = line[:-1] if line[1:7] == "Script": #print line[1:7] m = line.split('\"') #print m[1] if not os.path.isfile(os.path.join(path, "libs", m[1])): print("Unable to find file \"%s\"" % os.path.join(addon, "libs", m[1])) f.write("<!-- %s -->\n" % line) else: f.write("%s\n" % line) else: f.write("%s\n" % line) f.close() char = 0 print "\nPress any key to continue..." while not char: char = msvcrt.getch()And slightly off-topic I've made a very very messy script to clean out old changelog files for those who don't like to delete beforehand and re-extract due to custom/3rd party addon files.
#!/bin/env python import re, os, msvcrt ADDON_PATH = "C:\\Games\\World of Warcraft\\Interface\\AddOns" currentAddons = [f for f in os.listdir(ADDON_PATH) if os.path.isdir(os.path.join(ADDON_PATH, f))] c = 0 for addon in currentAddons: addonDir = os.listdir(os.path.join(ADDON_PATH, addon)) addonDir.sort() addonDir.reverse() tmpold = 0 for f in addonDir: r = re.search(r"Changelog-(.+)-r([0-9]+).xml", f) if r: if r.group(2) < tmpold: print "Deleting %s..." % f os.remove(os.path.join(ADDON_PATH, addon, f)) tmpf = os.path.join(ADDON_PATH, addon, f[:-3] + "txt") tmpf2 = os.path.join(ADDON_PATH, addon, "changelog-r" + r.group(2) + ".txt") if os.path.isfile(tmpf): print "Deleting %s..." % os.path.basename(tmpf) os.remove(tmpf) elif os.path.isfile(tmpf2): print "Deleting %s..." % os.path.basename(tmpf2) os.remove(tmpf2) tmpold = r.group(2) if c == 0: print "Nothing to delete!\n" char = 0 print "Press any key to continue..." while not char: char = msvcrt.getch()These are my first ever attempts at coding in Python but it wasn't at all hard, I'm not bothered about optimising them as they only should be run after each addon update.
Also, loading times improved a lot. Did not measure it, but it really is noticeable.
WTB "scriptable post processing" for WAU :D
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumIf you apply it to the current library system you get this:
MyAddon/ libs embeds.xml Ace2/ includes.xml Ace2.toc ... DewdropLib/ includes.xml DewdropLib.toc dewdrop.lua ... MyAddon.tocMyAddon.toc:
Ace2.toc:
DewdropLib.toc:
Heres the part that is different:
DewdropLib/includes.xml
Notice that the library does not need to embed other libraries as long as you can assume the library will be in the same directory which is true of all existing libraries now.
libs/embeds.xml
To disembed:
move all directories mentioned in libs/embeds.xml to the addon folder
then delete libs/embeds.xml
You would need to be able to determine the version of both the
existing addon and the embedded one via some file contained in each.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumWAU has a command line interface, and a log file (there's your scriptable post processing)
-
View User Profile
-
View Posts
-
Send Message
Curse Premium1) embeds.xml (Refers to your embedded libs)
2) includes.xml (Uses relative paths, to include lua files in subfolders. includes.xml usually in the subfolder)
3) modules.xml (Refers to your optional LoD modules)
Wau deletes embeds.xml if you install an addon without externals. It deletes modules.xml if you unpack an addon into its LoD subaddons. You dont always need modules.xml to be deleted for you, because you can place it into one of the LoD addon's folders, and when its unpacked, it goes with that folder.
embeds.xml, should be in addondir/embeds.xml
modules.xml, should be in addondir/modules.xml
includes.xml should be kept in the same directory as the files it refers to.
e.g. libs
addondir/
addon.toc
embeds.xml
libs/
includes.xml
addon.toc -> embeds.xml
embeds.xml -> libs/includes.xml
As you can see embeds.xml is the link to the embedded library files.
It doesnt stop there:
Addon
Addon.toc (refers to Addon_LoD1/includes.xml)
Addon_LoD1/
includes.xml
Addon_LoD1.toc (refers to includes.xml)
This is a LoD module example. Both the Addon.toc and the Addon.LoD1.toc reference the same includes.xml.
If you move Addon_LoD1 up one level, then Addon.toc no longer points to a valid file, effectively removing the lua file references. You also gain centralization of your addons's file structur, instead of 2 toc's with duplicate data, you now have 2 tocs, pointing to a single file.
The general pattern is use includes.xml to reduce a directory down to a single file which contains all the lua references.
Another trick, if you have many modules with single lua files in them, pick a module, and place a 'modules.xml' there, then use relative (..\foo\foo.lua, or ..\foo\includes.xml) references to include all the other modules. Inlcude this modules.xml in your addon TOC.
If the modules are made LoD, when they are moved up into the addon directory, the reference to modules.xml is no longer valid.
WAU goes through and removes addondir/embeds.xml. Since you defined it, you control what references are removed.
Also, there is no chance of TOC corruption.
-
View User Profile
-
View Posts
-
Send Message
Forum Moderator-
View User Profile
-
View Posts
-
Send Message
Curse PremiumGive me an example. I cant see a scenerio that couldn't be handled via the xml's but, if you can give me one. sure
-
View User Profile
-
View Posts
-
Send Message
Forum ModeratorLittleWigs