When I made Masque support for my addon that had a small action bar I just looked it up how Dominos made it work. All that worked for me was basicly this:
local Masque = LibStub("Masque", true)
if Masque ~= nil then
group = Masque:Group("Choose Mount", "Buttons")
Put this kind of code in OnIntialize fuction if you use Ace API or your AddonLoaded event handler. It tries to load masque and then checks if it worked. If it did then it adds a group "Buttons" under my addon's name "Choose Mount". So you just put there your addon's name and whatever name you want for your group.