Can not connect VNCServer using a load balancer device at home.

 

I have a question for you. 

 

My home network has a load balancer with two different providers WAN1 (provider1) and WAN2 (provider2), so when the main connection WAN1 is down, all the traffic in the network goes through WAN2 and the users are still online. However, when I use the RealVNC viewer for my remote access it can not connect to the local host where I have installed RealVNC Server. Now, when the WAN1 is up again, I'm able to connect again with the RealVNC server. 

My logic said that the WAN1 (IP) is taken to make an available connection between my VNCViewer and RealVNC Server. So your cloud connection is not smart enough to detect that I have two IPs for redundancy purposes. How can I fix this issue?

Thank you.