When I need to reload a lot, I disable the heaviest addons, those that increase my load time. Namely Cart gets disabled, KCI, probably some others that don't require positioning etc. Then when all is set (which lol it never is), I re-enable, play some, want a change - repeat process. AddonLoader does help. To me load time does matter, but it's just a personal opinion. It's give and take, more or less. You can't have AND mods that require a good load AND short load time, it's all about compromises and the likes.
But from memory:
Pitbull(Several Modules Disabled)
Grid(And a few of the Grid Addons)
Align(I don't think this even uses a noteable amount of memory)
Buffalo Dream Layout
Cowtip(Not sure, but I *think* this one might not be exactly "lightweight")
As a side note/update: I switched to standalone libs which did show a slight reduction in loading time, cheers to Seerah for that.
Also switched from SWSTats to the much more memory friendly(and up to date) Assessment.
Side question: One of the mods listed above is playing a bell sound when my character gets a debuff, I've searched pretty extensively through the options of the various mods I have that may have *anything* to do with my character getting debuffed, specifically the sound is something similar to the sound of the bell rung to begin a boxing match. Any ideas would be much appreciated.
EDIT - Also thanks to tekkub for warmup, been working with this as well to work towards minimizing even more.