Right. This is a hands-on hobby after all so the boredom factor may be the dominant factor in how time gets scaled (or not). Perhaps the real goal is to make time stand-still or even go backwards in time to capture nostalgia or a different (better?) period of our life!
To the issue of how to scale time, I see 3 factors:
Separate from scaled timing for fast-clock operations is the scaled (or not) timing of individual mechanisms. Our scaled models can accelerate 0 to 50 sMPH in a couple seconds, and decelerate just as fast. Electronic momentum techniques can slow this down but I think the boredom factor comes into play...who wants to wait several loops past the next train station to get up to mainline speed! Or I can startup a "cold" steam engine and having it moving down the track in a few seconds; that's one ginormous fast-clock multiplier!
Likewise, mechanisms like turnouts activated in a fraction of a second using original solenoid technology. Better components like micro geared motors and suitable electronics can now slow this down to tortoise-like speeds. So here it's an issue of better modeling vs. boredom factor.
I think whether scaled down in size or not, the brain expects some things to operate with a certain timing. Crossing flashers or alternating ditch lights flashing other than once per second (or whatever) would simply look funny.