You can't use the Flyout system Blizzard has to create your own flyouts. There are 2 reasons for this:
1. Blizzard's flyout buttons are created on demand. This includes being created while in combat. This also means any addon that wants to modify their behavior needs to setup checks so that when combat has ended, to hook the necessary stuff on any new buttons that got created while in combat.
2. The flyout button's spellID to be casted is assigned on the flyouts being shown on screen, CastSpellByID(self.spellID) is then performed on it in a future button click. Any addon that tries to set a spellID would just cause taint.