Roughly O scale coaches come in 60 or 72 foot models these range from 15 to 16 inches for the 60 foot cars and 17-18 inches for the 72 footers, O-27 coaches run around 12 inches in length.
The first thing is to choose your locomotives these come in O scale, Semi-scale, and O27. You want to proportion the coaches to the locomotives. O-27 coaches look out of proportion to O scale locomotives (too short) while O scale coaches look out of proportion to O-27 locomotives these coaches being too long for the locomotive.
O-36 track is a small step up from O-27 so being somewhat conservative in size is the safest bet. That would mean either a semi-scale locomotive with 60 foot coaches or a smaller O scale locomotive like a set of F3 or F7 models with 60 foot coaches. Or if desired an O-27 set with O-27 or semi-scale locomotives with 12 inch O-27 size coaches. These make the run over the track look more distant if the overall layout doesn't allow long lengths of track.
My layout is around the room 14 by 12 so it isn't really big nor is it small. The mainline curves are O-72 so that can accommodate O scale pretty easily. I have locomotives that range from O-scale to O27 and passenger cars that are 60 foot scale and O27 sized. The 72 footers look too big on the layout, the 60 footers are a little more proportional to the layout, these only run with the post war Lionel F3's, GP9s, Trainmasters, and the Williams O scale Alcos. I have a set of 72 foot Madison's that run with the old Lionel 773 Hudson I keep this train to 6 cars as it takes up a lot of rail. I have several O-27 size streamliner car sets; one runs with the repainted to Brunswick green Lionel 681 in Pennsylvania fleet of modernism scheme that I painted onto Lionel's O-27 streamline cars, an original 1950 yellow Union Pacific anniversary set with the add on pair of cars, a 1990's Northern Pacific 6 car set set pulled by an A-A pair of K-Line Alcos, and a couple sets of generic silver Lionel 12 inchers for hooking on other, mostly steam, locomotives of O-27 size as I feel like making up those type trains. The look of O-27 mixed with O scale looks jarring to my eye so operating sessions are either only O-27 or O scale. The only place where O-27 and O scale cars get mixed is with O scale freight operations as I model the transition period so a freight car size mix is common to that time thus looks as what I'd expect to see but these get O scale cabooses. When running O-27 locomotives for freight operations only O-27 cars and cabooses are used.
Bogie