I figured out that the numbers over the buffs are not actually from ag_UnitFrames, but from CoolDownCount. I tried changing the opacity, but that did not work. Changing the Font size does work, but it makes it small for everything, not just the buffs. :/
I just wanted to clarify that it isn't a fault of ag_UF, it's CoolDownCount which is the problem.