Cisco WAN :: Fast Ethernet Interface Being Take Down With Burst Of Traffic 2800

It is a single router with dual ISPs. It is a 2800 and there is failover configured. I have implemented object tracking and the feature works great except that lately, whenever there is a lot of traffic coming perhaps from the internal users, we start getting intermittent outages.
 
I have gone deep into looking into this problem and have determined that our ISP#1 does not have any problems. What I think is happening is that whenever the router receives a lot of packets (30-40 users on the internal network) destined to the outside, the router CPU maybe gets too busy and the router then believes that the objects are no longer reachable and it triggers a failover which causes the router to re-direct traffic to the ISP#2. Then, because these are just quick burst of traffic, in the next 30 to 45 seconds after the router re-directed the traffic to ISP#2, the router object tracking engine detects that the objects are now again reachable and this then causes the router to re-direct all traffic back to ISP#1.
 
This cycle then continues all day on how to prevent this.

Is there a way to perhaps tell the router to completely shut down the interface facing ISP#1 whenever there is a hicup and to keep it shutdown for at least 8 hours? This way I can prevent the router from going crazy.
 
Or is there a way to perhaps prevent this at the internal (LAN)  interface? Are there some metrics like QoS that I can implement on the internal Fast Ethernet Interface to prevent the burst of traffic from eating up the CPU?