There's a lot you can do in 8x12, depending on what kind of scenery you want, if any. This is the size I'm currently running, and I've moved the layout twice.
I started with an 8x12 Mianne benchwork setup. First layout was with 1/2" cheap plywood with Homasote on top for noise, second time is with a better quality 3/4 ply and the Homasote. The whole thing feels more stable, because with 8x12 you have to be able to get up on it at times. That middle is hard to reach even with a stool and still do effective work. it is 40" high, for ease of getting underneath it and wiring, and loads of storage space afterwards.
I recommend you download SCARM or purchase RR Track, and start dabbling with layouts. You'd be amazed at what you can come up with. This forum has MANY examples of what can be done in limited space. The software themselves come with samples if you want them. I myself bought RR Track with every library for O scale and all the samples that were available at that time. When I started I didn't know what track I wanted, so wanted to try different things. But that's me. SCARM has "come of age" and is still free, so I'd give that a whirl if you don't have anything but some basic tubular track. And there's nothing wrong with that either. Gosh the world is full of it, and it can be had cheaply.
I'm running FasTrack right now. You're right, it is reliable. The switches are awesome. And, it's quite noisy. When I've got 4 conventional trains high-balling, it's like standing next to Niagara Falls. Sometimes that's OK, but if you want to have a conversation, forget it. I've got my FasTrack stuffed with foam underneath to minimize the noise, but it's still quite the cacophony when all trains run. I run mostly legacy now, so I keep my speeds way down around 25-35 SMPH, and it's very quiet then. Be ready to spend some money if you don't already have some of this track system. They don't give away that track, or the switches that go with them. I've got all Command Control switches, so I can run them from my Legacy 990 controller setup without having separate wiring. You can obvious go with mechanical in O-36, but I think all the newer larger sizes available today are all CC.
My current setup, which is easy to change around with FasTrack, has two outer loops of O-72, and inner loop of O-48, and I've got a single second layer 4x8 in size with a O-36 loop on it and some bridges. All three bottom loops connect together with switches, but the layout is much simpler than my past ones. Depends on your running style, quiet looping or lots of switching. Seems I like running my articulated engines more than others, so I've simplified it.
Below is the RR track plan, and a recent photo of how far it's progressed over the last couple of months since our new home arrival. It's very simple, but it's been fun.
Good luck on your endeavor!
.