Okay, I see what you mean. Yes, it's the same in all configuration modes (disabled, party, raid, and solo). Regardless of the mode, I see the checkboxes for coloring by hostility in all other frames. The only frame where this is missing is Mouse-over. Could it be related to attempting to copy a layout between frames?
I'm not an experienced LUA coder, but I am a programmer, if there's something I can tweak in the source, I'd be happy to give it a shot if someone could direct me to the file where this is defined.
Saved the file, logged back in, and the mouseover frame works as expected. However, the PitBull configuration window still does not show the options to toggle these values (all other frames continue to show all toggles).
Hope this helps someone else out if they want to use the mouse-over frame.