IMO a library using built-in settings (e.g. voice activation level and push-to-talk button) and mimicking built-in voice-chat methods and events should be quite straightforward to write. Obviously, the library should wrap built-in events and statuses so the addon would only use one entry point.
I don't think so. The purpose of the addon is simply to show in game who is talking using push-to-talk button in an external voicechat (e.g. TeamSpeak or any other).
The trick is that if you map the push-to-talk key in WoW, it is received both by the voice-chat application and the WoW client. So you can catch the key events in WoW to emulate the built-in voice icon feature.
Given that your library should only send simple "starts talking", "stops talking" events, I think you shouldn't embed AceComm. Your library should deal directly with addon communication.
Wouldn't hooking UnitIsTalking cause taint issues ? Moreover, why making it a library if your addon already hooks UnitIsTalking ? In other words, the purpose of turning it into a library would be to make it less intrusive. If your addon already pushes the functionality into other addons by hooking UnitIsTalking, turning it into a library is pointless (unless there are some drawbacks you didn't tell us about.)