This function will determine whether two units are friendly to each other (i.e. able to help each other in combat). But when I request a duel with a member of my party/raid, the reaction should change, because the player becomes aggressive towards me. Instead, it still reports as a friendly unit.
However, it works correctly if I request a duel with player in my faction that is not part of my party/raid.
Is there a way to determine if a player is part of my party/raid?
Depending on what you wanted to do, you could use UnitCanAssist or UnitCanAttack. The former tells if the first unit could heal or buff the second one. The latter tells if the first unit can attack the second one. This allows to handle some corner cases, like really neutral NPCs or players of the other faction in sanctuaries, namely Dalaran, and Shattrath City.