Just a (perhaps) crazy idea. Forget about embedding completely. Chuck all libraries into a zip-file and ask users to download and install that.
I don't like embedding either, but I think it's here to stay. For whatever reason, many people appear to be more prone to adopt an addon if they don't have to download a separate addon to make it work.
That's true. But the individual author is of course free to embed the required libraries as AceLibrary's embedding code still works. He will then have some users complain that he is embedding probably. Can't satisfy everybody. :P
But the individual author is of course free to embed the required libraries as AceLibrary's embedding code still works.
All the more reason to just make the zip script adjustment and allow the removal of just an embeds.xml file instead. If some authors are going to continue to do it one way and not another, then a universal convention makes the most sense. I would go so far as to change the SVN to only allow authors with externals to update if they have an embeds.xml file in the directory.
Just a (perhaps) crazy idea. Forget about embedding completely. Chuck all libraries into a zip-file and ask users to download and install that.
I don't like embedding either, but I think it's here to stay. For whatever reason, many people appear to be more prone to adopt an addon if they don't have to download a separate addon to make it work.
I agree that changing the zip script seems to be the way to go. That way, it could be setup to simply not include any of the embeds or the embed.xml file.
How would this affect authors who didn't update? It would still be pretty much the way it is now, wouldn't it?
If the zip script was running something similar to that python script, splitting the embeds into an xml file is redundant.
Anyway, I know that for my own purposes, I'm not gonna use a seperate file just for embedded library loading. Much easier for me to just look at the toc to see what I'm loading.
Why change the whole damn repo around for something that can be done just as effectively with a tiny script?
Okay, I played with the idea of concatenating addon files. The included script here will scan the TOC and replace consecutive (existing) files into a single one. In order to be able to easily concatenate XML content, this script requires 4Suite for python.
I have not tested this fully, and not with the game. I think it's more a funny test than anything serious.
As the previous script, this one will modify your toc file, so backup before using it.
#!/bin/env python
import os
from Ft.Xml.Domlette import Print, NonvalidatingReader, implementation
ADDON_PATH = "/home/jerry/wow/Interface/AddOns"
UTF8_MARKER = "\xEF\xBB\xBF"
def merge_lua_files(output, files):
f = open(output, "w")
for file in files:
f.write("-- %s\n" % file)
i = open(file, "r")
for line in i.readlines():
f.write(line)
i.close()
f.close()
def merge_xml_files(output, files):
out = implementation.createDocument("[url]http://www.blizzard.com/wow/ui/"[/url], "Ui", None)
root = out.firstChild
root.setAttributeNS("[url]http://www.w3.org/2001/XMLSchema-instance"[/url], "xsi:schemaLocation", "[url]http://www.blizzard.com/wow/ui/../FrameXML/UI.xsd"[/url])
for file in files:
i = open(file, "r")
doc = NonvalidatingReader.parseStream(i)
i.close()
root.appendChild(out.createComment(file))
child = doc.firstChild.firstChild
while child:
root.appendChild(child.cloneNode(True))
child = child.nextSibling
o = open(output, "w")
Print(out, o)
o.close()
for addon in os.listdir(ADDON_PATH):
path = os.path.join(ADDON_PATH, addon)
if os.path.isdir(path):
toc = os.path.join(path, addon + ".toc")
files = []
f = open(toc, "r")
lines = f.readlines()
f.close()
f = open(toc, "w")
for line in lines:
if line[:3] == UTF8_MARKER:
line = line[3:]
if line[-2:] == '\r\n':
line = line[:-2]
elif line[-1:] == '\n':
line = line[:-1]
line = line.strip()
if line[:1] == "#":
f.write("%s\n" % line)
elif len(line) > 0:
n = os.path.join(path, line.replace("\\", os.sep))
if os.path.isfile(n):
files.append(n)
id = 1
while len(files):
file = files[0]
ext = os.path.splitext(file)[1]
i = 1
while i < len(files) and os.path.splitext(files[i])[1] == ext:
i += 1
if i == 1:
f.write("%s\n" % files[0])
else:
o = "content%d%s" % (id, ext)
if ext == ".lua":
merge_lua_files(os.path.join(path, o), files[:i])
elif ext == ".xml":
merge_xml_files(os.path.join(path, o), files[:i])
else:
raise "Invalid extension"
f.write("# Merged "+", ".join(files[:i])+"\n")
f.write("%s\n" % o)
id += 1
files = files[i:]
f.close()
Hmm, if you start concatenating Lua files, then error messages are going to be very hard to interpret because line numbers will no longer match the original file. That is a huge problem in my opinion.
Why does _anyone_ think that's a good idea. I'll take over-optimisation for 100 Alex!
I don't know, although up until this very thread, I never had though that removing non existing files from a TOC would have such an impact on loading time.
Maybe creating the "environment" to process a lua chunck is time consuming, so reducing the number of time such "environments" would need to be created would be a good idea. On a pure lua implementation, this does not seem very likely, but because wow's security model can't have been implemented without changes to lua_Object and lua_State, who knows ?
We've made the assumption that the presence of a file in the TOC was not really serious, and the absence of this file to have only the impact of a filesystem check. We were wrong. It doesn't hurt to try to check if other assumptions could be verified.
Why does _anyone_ think that's a good idea. I'll take over-optimisation for 100 Alex!
I don't know, although up until this very thread, I never had though that removing non existing files from a TOC would have such an impact on loading time.
Maybe creating the "environment" to process a lua chunck is time consuming, so reducing the number of time such "environments" would need to be created would be a good idea. On a pure lua implementation, this does not seem very likely, but because wow's security model can't have been implemented without changes to lua_Object and lua_State, who knows ?
We've made the assumption that the presence of a file in the TOC was not really serious, and the absence of this file to have only the impact of a filesystem check. We were wrong. It doesn't hurt to try to check if other assumptions could be verified.
def merge_lua_files(output, files):
f = open(output, "w")
for file in files:
f.write("-- %s\n" % file)
f.write("loadstring([=====[\n")
i = open(file, "r")
for line in i.readlines():
if line[:3] == UTF8_MARKER:
line = line[3:]
if line[-2:] == "\r\n":
line = line[:-2]
elif line[-1:] == "\n":
line = line[:-1]
f.write(line+"\n")
f.write("]=====], \"%s\")\n" % file)
i.close()
f.close()
should take care of both Bam and Ammo's concerns, by the way :-)
Yes, it's hideous. and assumes that there is no particular cost associated with loadstring().
loadstring requires an interning of the string, the creation of the closure, garbage collection of the string, running of the function, garbage collection of the closure, etc.
Perhaps you should do an ounce of testing before posting and distributing a script.
loadstring requires an interning of the string, the creation of the closure, garbage collection of the string, running of the function, garbage collection of the closure, etc.
Perhaps you should do an ounce of testing before posting and distributing a script.
All this being done once per file, as compared to an unknown processing that is done once per file too. I can't test, but I can still have fun. I never thought that posting on this thread should be a serious business that would require "testing before posting and distributing".
In my opinion it is. People read these threads, and there have been a number of low level users who have already been pointed to this thread and are running arbitrary python scripts to accomplish one thing.
If you'd like to explore another alternative, make another thread, and show that your scripts are "experimental" and "untested" rather than just posting them here, where sheep are already blindly running things. Its called responsibility, and if you're posting things, without running them and without fully understand what and why, then I call that unresponsible.
You should not be posting that stuff in this thread, because people are coming here to get resolution to one issue and having multiple scripts for multiple purposes here just plain confuses the issue.
You should not be posting that stuff in this thread, because people are coming here to get resolution to one issue and having multiple scripts for multiple purposes here just plain confuses the issue.
Well, I disagree. People that blindly execute scripts are still responsible for what they do, they should know what they are doing, and are still to blame if they don't.
As long as I don't purposefully do something bad, which I honestly don't think any of my scripts has done, then that's enough. I'm exploring ideas, I'm "thinking out loud". I'm also semi-joking. If someone don't get the joke, then it may be sad, but I'm not to blame because of it.
What is, in you opinion ? The fact that I have posted a script about concatenating addon files, the idea of concatenating, or simply to use loadstring ?
This is the developer's corner. Threads like this are what this section of the forum is for. Its OUR area to talk about this stuff. People should not be linking to it.
We should just limit access to it so we can discuss things freely then, if these discussions are a problem.
Not all the ideas expressed in this thread were meant as anything other than "lets see what happens if". This is one of the most valuable parts of the creative process, and under no circumstances should it ever be made into a negative thing for someone to post an idea. Otherwise fewer people will, or they will think twice before they do - which ruins the colaborative process.
Rollback Post to RevisionRollBack
To post a comment, please login or register a new account.
That's true. But the individual author is of course free to embed the required libraries as AceLibrary's embedding code still works. He will then have some users complain that he is embedding probably. Can't satisfy everybody. :P
Load time reduced by about 5sec here. FrameXML.log down to 4kb from 92kb
Anyway, I know that for my own purposes, I'm not gonna use a seperate file just for embedded library loading. Much easier for me to just look at the toc to see what I'm loading.
Why change the whole damn repo around for something that can be done just as effectively with a tiny script?
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumI have not tested this fully, and not with the game. I think it's more a funny test than anything serious.
As the previous script, this one will modify your toc file, so backup before using it.
#!/bin/env python import os from Ft.Xml.Domlette import Print, NonvalidatingReader, implementation ADDON_PATH = "/home/jerry/wow/Interface/AddOns" UTF8_MARKER = "\xEF\xBB\xBF" def merge_lua_files(output, files): f = open(output, "w") for file in files: f.write("-- %s\n" % file) i = open(file, "r") for line in i.readlines(): f.write(line) i.close() f.close() def merge_xml_files(output, files): out = implementation.createDocument("[url]http://www.blizzard.com/wow/ui/"[/url], "Ui", None) root = out.firstChild root.setAttributeNS("[url]http://www.w3.org/2001/XMLSchema-instance"[/url], "xsi:schemaLocation", "[url]http://www.blizzard.com/wow/ui/../FrameXML/UI.xsd"[/url]) for file in files: i = open(file, "r") doc = NonvalidatingReader.parseStream(i) i.close() root.appendChild(out.createComment(file)) child = doc.firstChild.firstChild while child: root.appendChild(child.cloneNode(True)) child = child.nextSibling o = open(output, "w") Print(out, o) o.close() for addon in os.listdir(ADDON_PATH): path = os.path.join(ADDON_PATH, addon) if os.path.isdir(path): toc = os.path.join(path, addon + ".toc") files = [] f = open(toc, "r") lines = f.readlines() f.close() f = open(toc, "w") for line in lines: if line[:3] == UTF8_MARKER: line = line[3:] if line[-2:] == '\r\n': line = line[:-2] elif line[-1:] == '\n': line = line[:-1] line = line.strip() if line[:1] == "#": f.write("%s\n" % line) elif len(line) > 0: n = os.path.join(path, line.replace("\\", os.sep)) if os.path.isfile(n): files.append(n) id = 1 while len(files): file = files[0] ext = os.path.splitext(file)[1] i = 1 while i < len(files) and os.path.splitext(files[i])[1] == ext: i += 1 if i == 1: f.write("%s\n" % files[0]) else: o = "content%d%s" % (id, ext) if ext == ".lua": merge_lua_files(os.path.join(path, o), files[:i]) elif ext == ".xml": merge_xml_files(os.path.join(path, o), files[:i]) else: raise "Invalid extension" f.write("# Merged "+", ".join(files[:i])+"\n") f.write("%s\n" % o) id += 1 files = files[i:] f.close()-
View User Profile
-
View Posts
-
Send Message
Curse Premium-Ammo
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumThis can be easily been taken care of by enclosing each file content within a "do end" scope.
Other things that could be done is to generate an XML file containing just a list of <Script file="foo.lua" /> and <Include file="bar.xml" />.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumI don't know, although up until this very thread, I never had though that removing non existing files from a TOC would have such an impact on loading time.
Maybe creating the "environment" to process a lua chunck is time consuming, so reducing the number of time such "environments" would need to be created would be a good idea. On a pure lua implementation, this does not seem very likely, but because wow's security model can't have been implemented without changes to lua_Object and lua_State, who knows ?
We've made the assumption that the presence of a file in the TOC was not really serious, and the absence of this file to have only the impact of a filesystem check. We were wrong. It doesn't hurt to try to check if other assumptions could be verified.
Have fun... =/.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumIsn't it the whole point ? :-)
def merge_lua_files(output, files): f = open(output, "w") for file in files: f.write("-- %s\n" % file) f.write("loadstring([=====[\n") i = open(file, "r") for line in i.readlines(): if line[:3] == UTF8_MARKER: line = line[3:] if line[-2:] == "\r\n": line = line[:-2] elif line[-1:] == "\n": line = line[:-1] f.write(line+"\n") f.write("]=====], \"%s\")\n" % file) i.close() f.close()should take care of both Bam and Ammo's concerns, by the way :-)
Yes, it's hideous. and assumes that there is no particular cost associated with loadstring().
Perhaps you should do an ounce of testing before posting and distributing a script.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumAll this being done once per file, as compared to an unknown processing that is done once per file too. I can't test, but I can still have fun. I never thought that posting on this thread should be a serious business that would require "testing before posting and distributing".
If you'd like to explore another alternative, make another thread, and show that your scripts are "experimental" and "untested" rather than just posting them here, where sheep are already blindly running things. Its called responsibility, and if you're posting things, without running them and without fully understand what and why, then I call that unresponsible.
You should not be posting that stuff in this thread, because people are coming here to get resolution to one issue and having multiple scripts for multiple purposes here just plain confuses the issue.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumWell, I disagree. People that blindly execute scripts are still responsible for what they do, they should know what they are doing, and are still to blame if they don't.
As long as I don't purposefully do something bad, which I honestly don't think any of my scripts has done, then that's enough. I'm exploring ideas, I'm "thinking out loud". I'm also semi-joking. If someone don't get the joke, then it may be sad, but I'm not to blame because of it.
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumWhat is, in you opinion ? The fact that I have posted a script about concatenating addon files, the idea of concatenating, or simply to use loadstring ?
-
View User Profile
-
View Posts
-
Send Message
Curse PremiumWe should just limit access to it so we can discuss things freely then, if these discussions are a problem.
Not all the ideas expressed in this thread were meant as anything other than "lets see what happens if". This is one of the most valuable parts of the creative process, and under no circumstances should it ever be made into a negative thing for someone to post an idea. Otherwise fewer people will, or they will think twice before they do - which ruins the colaborative process.