Does "takes a long time" mean several seconds? If you stop the TT but then restart it in the SAME direction, does it also take the same "long time" to re-engage? And during this "long time" can you hear the motor whirring? In other words are you sure it's a mechanical issue and not some kind of electronic artifact. That is, DC train controllers (e.g., HO) can have "momentum control" where the motor drive voltage slowly ramps down to zero, reverses, then slowly ramps up in the opposite direction. Since this is apparently a custom TT, did your builder simply install a DC train controller to drive the TT motor?
It's hard to imagine a scenario where there's so much "slop". mis-alignment, or slippage in a gear or transmission mechanism that it takes a "long time" to reverse direction (or continue on in the same direction if stopping).
Again, if you could post as many photos as possible (a video would be nice) I'm sure we could help you out.