Authors CurseForge

Quote from Yssaril »

for chat messages the itemlinks have to be valid for addon messages it just has not contain illegal characters i think i wouldn't worry about chat messages since all the data should always be sent via the addon channels

Well, for example a cross-guild tool, communication is forced to happen on chat channels... GuildAds is one such tool (but has its own encoding/decoding) and GuildEventManager as well.

Ideally, Blizzard should provide addon-channels on the same principle as normal chat channels. Maybe even just enable addon messages to happen on the chat channels you are on (except perhaps the general/trade/etc channels). But until this is possible, normal chat is the only way.

Anyway, addon channel allows every but \000. Chat channels is something like:

-- Following byte values are not allowed/unwanted:
-- \000, s, S, \010, \124, %
-- s and S triggers the drunken "corruption"
-- \010 = newline and \124 = | (pipe)
-- Because SendChatMessage will error if an UTF8 multibyte character is incomplete,
-- all character values above 127 have to be encoded to avoid this. This costs quite a bit of bandwidth (about 13-14%)
-- Also, because drunken status is unknown for the received, strings used with SendChatMessage should be terminated with
-- an identifying byte value, after which the server MAY add "...hic!" or as much as it can fit(!).
-- Pass the identifying byte as a reserved character to this function to ensure the encoding doesn't contain that value.

This can be done. Requires more than 1 escape character or deeper nesting. But doable.

But I have no information about what characters are safe/unsafe when data is stored in the saved variables files.

as long as its unicode i think anything goes in saved vars

Quote from OrionShock »

Check out AceComm-2.0 's code wrt chat channel comms, it covers just about everything iirc.

We did look at it to use in GuildAds, but AceComm-2.0 didn't cover all problems as far as I can remember. Also, AceComm-2.0 is huge and does much more than is required for LibCompress... (I know the space is only used once for all addons using AceComm-2.0).

The major problem with AceComm-2.0 is that when it is encoding values above 128, it may risk using the bytes "s" and "S" which will lead to corruption if the player is drunk. This is fixed in GuildAds (based on AceComm-2.0).

I'll probably include the GuildAds encoding in LibCompress (if I do not make this a separate library, LibEncode), but seeing as encoding and decoding functions aren't that big and probably will get used with LibCompress anyway, I'll probably add them to LibCompress.

Quote from Allara »

If AceComm-2.0 handles it's own encoding, I see no reason to also handle it here.

Look at the AceComm-2.0 encoding. It is buggy and will not provide safe encoding (we tested it).

I would think anyone using this library would also be using Ace to some degree.

I do not think so, but that is not really the issue as AceComm-2.0 is buggy and large.

I am a fan of AceComm-3.0, which is a lot more minimal (but doesn't support channels and such IIRC, so it wouldn't work for GuildAds). I personally see 2 uses for compression -- either direct addon-to-addon communication, or stored in saved variables. Since AceComm-3.0 doesn't encode \000 characters (I don't believe), it needs to happen somewhere else.

AceComm-3.0 cannot be used as it only supports SendAddonMessage and doesn't encode \000. External encoding is needed.

If you're really going to get into other types of encoding, then maybe this should be made into a separate library. If you're only going to encode for \000, I think it should just be built in, since that character is disallowed everywhere (or could be if it's not).

\000 is not allowed for SendAddonMessage nor SendChatMessage. I do not know if \000 is allowed in storage.

However, the argument for inclusion of all encoding methods in LibCompress is this: LibCompress will only be used in WoW. WoW requires some kind of encoding for all uses of strings, period. Therefore, in order to provide an end-to-end solution, LibCompress should handle encoding. Otherwise it will always be useless without some other library included.

True. And this was my thought when I wrote that I probably would end up including the encoding in LibCompress directly. This leaves it up to the user to use the encoding after the compressed, e.g. like this:

-- encode
c = LibStub:GetLibrary("LibCompress")
chat_data = c:Encode(c:Compress(raw_data))

-- decode
raw_data = c:Decode(c:Decompress(chat_data))

The Encode and Decode functions may take names after which channel to encode for and will also also allow custom addition bytes to be encoded (used in e.g. a transmission/encapsulating protocol).

Or:

encode_func = c:GetChatEncodeFunction(optional_escape_byte_to_use, list_of_optional_reserved_byte_values)
chat_data = encode_func(c:Compress(raw_data))

Quote from Allara »

I like it. Alternatively (or in addition), Compress could take one or more additional arguments which specify encoding type (or no encoding if none specified). Decompress could decode transparently.

That would mean the encoding type must be included in the header (additional header byte). This hardly seems necessary as the receiver probably always will know how the data was encoded, but can't always know how it was compressed.

Quote from galmok »

e.g. like this:

-- encode
c = LibStub:GetLibrary("LibCompress")
chat_data = c:Encode(c:Compress(raw_data))

-- decode
raw_data = c:Decode(c:Decompress(chat_data))

You meant this, right?

c = LibStub:GetLibrary("LibCompress")
chat_data = c:Encode(c:Compress(raw_data))

-- decode
raw_data = c:Decompress(c:Decode(chat_data))

Of course. :-P

But it hasn't been implemented yet. My wow account has expired and I may not return. Blizzard's arena elitism and warlock changes are just not fun for me.

For the time being I encourage you to do your own encoding/decoding. If you need to encode for the chat channel, I can recommend having a look at GuildAds that has implemented a working chat channel encoding/decoding algorithm (pretty much a bug fixed Ace2 encode/decode).

My optimised version of the huffman compressor/decompressor had two bugs. Both are fixed in the r32-release of LibCompress.

Future plans for LibCompress include:

- Encode/Decode (with optional list of reserved characters and quote-bytes). This will provide the possibility to encode for any channel with any number of reserved bytes.
Possible syntax:

codec = LibCompress:GetEncoder(reservedchars, quotebytes)

encodedData = codec:Encode(compressedData)

compressedData = codec:Decode(encodedData)

The reservedchars contains a string with characters that may not occur in th encoded data. Quotebytes contains the characters(s) necessary to perform the encoding. OF course, the bytes in quotebytes may not appear in reservedchars.

- Encode/Decode to/from 7-bit characters to 8-bit bytes. This is motivated by the fact that encoding raw huffman data for the chat channel will make the size increase a lot, neglecting any compressing gained with huffman. Reducing from 8 bit to 7 bit will only cause a slight increase in size, requiring much less encoding. Huffman compression will still save bandwidth in this case.

Suggestion API:
Encoding:
sevenbitencoded = LibCompress:Encode7bit(rawdata)

Decoding:
rawdata = LibCompress:Decode7bit(sevenbitencoded)

This could have been built into huffman compression/decompression with 0 speed penalty, but LibCompress has to offer this for all compressors/decompressors which is why I think a separate function should be offered.

- Support for predefined symbol statistics for the huffman compressor. For short strings to be compressed, the symbol map is a large part of the compressed data and can neglect any compression gained. Even suboptimal symbol map statistics could provide a gain in compression. This change will not be backwards compatible though.

I also want to add checksum algorithms to LibCompress so you can make a more secure check if the compressed data is ok or not (before uncompressing it).

Currently I have FCS (16 and 32 bit versions).

Quote from galmok

My optimised version of the huffman compressor/decompressor had two bugs. Both are fixed in the r32-release of LibCompress.

Future plans for LibCompress include:

- Encode/Decode (with optional list of reserved characters and quote-bytes). This will provide the possibility to encode for any channel with any number of reserved bytes.
Possible syntax:

codec = LibCompress:GetEncoder(reservedchars, quotebytes)

encodedData = codec:Encode(compressedData)

compressedData = codec:Decode(encodedData)

The reservedchars contains a string with characters that may not occur in th encoded data. Quotebytes contains the characters(s) necessary to perform the encoding. OF course, the bytes in quotebytes may not appear in reservedchars.

This has been added (not tagged as release yet).

    table, msg = LibCompress:GetEncodeTable(reservedChars, escapeChars,  mapChars)

reservedChars: The characters in this string will not appear in the encoded data.
escapeChars: A string of characters used as escape-characters (don't supply more than needed). #escapeChars >= 1
mapChars: First characters in reservedChars maps to first characters in mapChars. (#mapChars <= #reservedChars)
return value:
table
if nil then msg holds an error message, otherwise use like this:

            encoded_message = table:Encode(message)
            message = table:Decode(encoded_message)

- Encode/Decode to/from 7-bit characters to 8-bit bytes. This is motivated by the fact that encoding raw huffman data for the chat channel will make the size increase a lot, neglecting any compressing gained with huffman. Reducing from 8 bit to 7 bit will only cause a slight increase in size, requiring much less encoding. Huffman compression will still save bandwidth in this case.

Suggestion API:
Encoding:
sevenbitencoded = LibCompress:Encode7bit(rawdata)

Decoding:
rawdata = LibCompress:Decode7bit(sevenbitencoded)

This could have been built into huffman compression/decompression with 0 speed penalty, but LibCompress has to offer this for all compressors/decompressors which is why I think a separate function should be offered.

This has also been added and works as described in the quote.

- Support for predefined symbol statistics for the huffman compressor. For short strings to be compressed, the symbol map is a large part of the compressed data and can neglect any compression gained. Even suboptimal symbol map statistics could provide a gain in compression. This change will not be backwards compatible though.

This is on the to-do list still.

16/32 bit FCS checksum/hash algorithm has been added and r36 is tagged as release.

is there a page/document somewhere that has all the API info in one organized place?

Not yet. :-/ But the place will be on curse.com:

http://wow.curse.com/downloads/wow-addons/details/libcompress.aspx

Quote from galmok

Not yet. :-/ But the place will be on curse.com:

http://wow.curse.com/downloads/wow-addons/details/libcompress.aspx

I can't navigate curse.com to find the place to edit the page. This will have to wait until it becomes obvious. Until then, use this thread or ask questions at curse.com.