Whoa there! None of my addons use libraries (8-10 i don't count). They aren't localized either.I don't think there's a single addon left out there that doesn't use a library.
I don't think loading times is an issue at all. You put more load on the system, expect it to take longer.
Duplicate library loading can't be 'solved' unless you put the libraries in the addons folder as /AddOns/libMyShinyLibrary-1.6/ (note the version, I think this is totally doable) as separate addons and put dependancies of them in your addons. I don't think it's a hassle at all to manage and for all the one click update done people (lazy) it would be no hassle either because an updater can be made to spot unused dependancies and remove them if you would so desire.