I had similar frustrations, but with different switches.
The real breakthrough I had on my shorting issues is when I upgraded transformers. When the locomotives shorted out they left little burn marks on the rails with the more powerful transformer.
When the locomotive shorts out, get a flashlight, get down on your belly, and look underneath. Carefully roll the locomotive up on to one rail, and LOOK as you do it to see where the rollers and wheels are contacting. Up, down, up, down, looking all the time. Look for things sticking
In my case, the track was O-27 tubular. The problem was the wide pickup rollers were bridging between the center rail and the swing rails on the switch. The solution was to cover the tip of the center rail "diamond" with a piece of black duct tape to insulate it.
Your M4000 should be leaving some pretty obvious burn marks on the track.
99% of the time, switch derailments are because the switch isn't closed all the way in one position or the other.
Isolating a siding, all you really have to do is insulate the center rail. It's easy with tubular track, you simply replace the steel pin with a plastic pin. On realtrax, I expect there's probably a special isolation piece you can buy. Run a power feed to the isolated siding through a toggle switch so you can turn it on and off.
Don't forget you need one siding for each locomotive you want to run like this. You can't run two locomotives with only one siding. Where do you park the locomotive on the main line when you want to run the one that's parked on the siding?