In O gauge, a helix with a 2% grade will be a room filling structure. Using O-48 track the circumference is 150". With a 2% grade the track only climbs 3", that sure will not work. If the helix uses 1/2" plywood, no roadbed, and the track is 1/4" from bottom of ties to top of rail, that consumes 3/4" of the rise. Using O-96 track the circumference is 300", 2% grade is 6" of climb, less the 3/4" is a 5 1/4" clearance. That should be enough for most O gauge cars.
I recommend you consider a 2.5% grade and not use a helix. My layout has track from -6" to +11", a 17" climb. I wanted all 2% grades but to make it fit in the room we had to go to 2.5%. The layout is built around the walls, the six 6"climb connectors, with one exception, are hidden behind scenery adjacent to the room walls. There are six because there is an up and a down for continuous running.
So I basically have what AlanRail suggested, with the "helix" spread out to the room walls. There is a mainline at the 0, +6 and +12 elevations, the mainlines provide the flat part of the climb. It is a big climb but worth it. It is interesting to run a train that comes out of a tunnel down at river level, then with three laps around the layout is up on the high bridge.