Strange, it use to be easy to restrict custom NPCs to only creative players (which is usually restricted to admin / op on survival servers). I used it in my 1.6.4 modpack without trouble, and on the server I used them specifically as bankers (a trader as a money changer, plus a vault keeper). I never used them for quests.
Well, Minecraft Comes Alive makes the villagers way better. more like people you might meet in an RPG.
I would consider some of the dungeons mods: Doomlike Dungeon (got to have that one ;-) ) and Roguelike Dungeons work well.
If you want to it to possible to make your own RPG plots / questlines or have special NPC for a server there's also Custom NPCs; just be warned, it doesn't really come with content or make content, it provide a way for you to make content.