Cisco :: 5508 Separate RF Group For Every Location

I'm looking for some input on RRM.  I personally have NOT used it in a LONG TIME, since probably the 4.0 days and then very shortly due to massive issues it was causing and admittedly, in part due to my ignorance at the time.  So, every since that point, I have always set all my channels and power manually but now feel I am getting to some points where RRM may be required / beneficial.  So, I've invested some time and have begun researching and trying to get the ends and outs on it but I'm forseeing a potential issue in myworld anyways and am hoping for some clarification.  Lets take the below example:

-WLC5508a and b - (2 100ap license controllers) - these hold the majority of the AP's for the main hospital.Lets say, 140AP's.

-WLC5508c and d - (1 100ap and 1 50ap licensed controllers) -  These tend to hold our smaller sites and and buildings, not all connected and some a few miles from each other

-WLC4402a and b - (failover ready)

So, with RRM, I can set setting it up on the 5508A/B with out issue as this is one big large building. However,what about C and D?  I suppose I can make them a separate RF Group, but how would RRM respond when it has16 AP's in Building X and then 3 AP's in Building Y 30 AP's in Building Z and sporadic buildings with 1's and 2's?  Everything I've read so far, leads me to believe if these devices are separated it probably won't be an issue, however, I just don't want something causing a change in Building Z and Building X be affected because RRM decided it would try to fix it. My point is, I can't afford to have a separate RF Group (meaning separate controllers) for every location.