Cisco VPN :: 2800 - Remote Access To VRRP VIP?

One of our customers had 2 x 2800 series routers that they needed to reconfigure to support new services, The existing public subnet has 2 free ip addresses to use for the routers, unfortunetely even though the ISP can reconfigure this to create more addresses the customer cannot have any downtime (there are other routers in this subnet that are live) and therefore i had my hand forced into using VRRP as opposed to HSRP (3 addresses).
 
I used VRRP to share the master address as the VIP, so one address on the master, one on the backup, using the master address as the VIP.
 
This all works fine, outbound and inbound traffic failover as expected and preemption works just fine.
 
My problem is, he asked me to configure a remote access VPN. So i have configured the VPN to connect to the VRRP VIP. When the master is active, the VPN connects, traffic passes, all fine. When the master is switched off the VPN traffic hits the backup, as its now assumed the VIP, and completes phase 1, xauth works, but phase 2 will not come up and the client displayed "not connected".
 
So doing some debugs, the phase 2 policies are accepted, but i get the message
 
*Mar  1 01:36:21.451: IPSEC(validate_transform_proposal): invalid local address x.x.x.164
 
where x.x.x.164 is the VRRP VIP address, the physical address which has the crypto map applied is .165
 
So here lies the problem, the client is connecting to .164, the crypto map is applied to the interface the is configured with .165. Hence the "invalid local address"
 
I have found some documentation online that suggests that VPN redundancy is possible with HSRP, but not on the 2800 series router. I cant use HSRP as i have only 2 addresses, and i cant use that feature as my routers dont support it.