In trying to set up a modifier self cast in new versions of Bartender3, the actual modifier used cannot be selected. You can select to use a modifier self cast, but then cannot select alt, ctrl, or shift afterwards. Bars that were set up and unchanged from a long time ago (like on my main) work perfectly and like they always have. However, when I try to set up a new profile (like on a new alt) the mod self cast does not work. I attempted to copy over my main's profile to my alt, but it does not copy over the self cast functionality.
Correct, the self-cast key selection was removed from the Blizzard Interface Options. You can set the modifier with /script commands, but I don't know the exact function names off-hand. Or if you're lazy like me, you can use CVarEditor to change it (and many other things, although be careful as you can get into trouble changing things in there :P).