Thanks for responding, I was just about to post what I've found. This is really long but it kind of needs to be.
Layout Configuration
The GarGraves track is 20 years old. The wiring is 2 years old as we just downsized to a new house (Naperville). 14 gauge twisted pair home run to 4 quadrants so that I can use crossing gates etc. Everything to the track is soldered. DCS and iphone app. I don't use banana plugs at the Z, instead I run the wire through the hole in the post and tighten the red/black knobs.
The layout controlled by the Z is track 1 mainline (left handle) and track 2 mainline (right handle). Each mainline has passing siding by using left-right 072 turnouts. All sidings can be turned on/off by a switch to the center rail.
Layout Test
I took the Z off the table and replaced it with 2 Atlas O 80 watt transformers. Siding 1 of both mainlines is a passenger train, siding 2 is freight. Every engine is Proto 2. I used different sidings, and ran each train for 20 minutes. Running this way was flawless.
Bench Test the Z
On my workbench I had 2 Gargraves straights wired for each handle. I put left handle to 17 volts, I added the 2 passenger cars. It dropped to 14 volts. I kicked the voltage back to 17 volts. I took off the cars, it went to 20 volts. I did the same test with the right handle, same result. I put both handles at 17 volts, added cars to left handle, left handles drops to 14, right stays at 17. Kicked left back to 17. Put 2 cars on the right handle, it drops to 14 while left stays at 17. For all of this I confirmed that the voltage at the track matched the voltage displayed.
Wire Z back to the Layout
Left handle I ran the passenger train at 17 volts. No problem. I stopped the passenger via DCS on siding 1, switched the turnouts to curved and ran the freight train. No problem. Then I toggled off siding 1 with the passenger train, while the freight train was still running, boom went to 20 volts.
When I run trains I'm switching the sidings on/off all the time. Part of how I enjoy the layout is having different trains going different directions, replacing the trains I have on the layout etc.
Conclusion
1) Being an IT guy for 40 years this feels like a software bug. On my old layout I just had 2 loops without sidings so it's possible it's always been this way and I never had a reason to notice.
2) In running trains on the new layout for the last 2 years I only recently noticed this. So maybe there is a connection inside the transformer coming loose? I haven't had time yet to open it up.
Thanks for hanging in there with this. I'm thinking of keeping the Z now knowing this quirk is there. If it gets worse I'll replace it. The wife is fine with me spending near 1K on a new transformer. I'm not TBH.
I'm open to all thoughts on this.