What radius is ok.

Although I use nothing but Kadee "scale" couplers, 072 works fine. I do think that 060 might give you problems, especial with 6-axle diesels.

Even though the six-axle diesels are rated at 42" radius (O-84,) they seem to negotiate 36" radius (O-72) just fine with cars or other locomotives coupled to them with the scale wheels. I use Kadee 805's with the backs of the boxes shaved off about 1/8" to clear the three-rail coupler mount on the power trucks.

My experience matches Matt's; all run fine with scale wheels on 072. You are switching the wheels to hi-rail. The limiting factor then is the extent to which the fixed pilot/coupler combination swings toward the outside rail. Here again 072 is generally fine, though bear in mind the rule that shorter equipment coupled to long equipment is more likely to be derailed (especially with body-mounted couplers). You might squeeze through the occasional 060 curve, but I don't know.

Another limiting factor is "S" curves. The traditional solution of a 10-inch straight between two opposite curves won't prevent derailments with body-mounted couplers on equipment this size. Either eliminate any S curves or at least follow the rule of thumb of having a straight track as long as your longest piece of equipment between the curves. In my view, operating these engines is worth any adjustments in track work you have to make. While everyone has to make concessions to space-limits, it is always better if the track work serves the trains rather than the other way around.

On couplers, the 805 is indeed a good choice, and I use them or the plastic equivalent (just as reliable). But as a safeguard against short circuits reaching the engine, I avoid combining metal couplers and metal draft gear boxes on engines. If I use an 805 coupler, which is metal, I usually substitute a plastic draft gear box for the die cast one that comes with the Kadee. The new Kadee 745s actually come with plastic draft gear boxes and have a better appearance and that's what I've taken to using. In all events, be sure the Kadee trip pin doesn't contact the center rail. It can catch on switches, and again with metal couplers it may be a potential source of a short. Either bend it up as needed, or even cut it off (as some folks do). 

I generally avoid the 806 Kadee coupler (the short shank model). It has more limited swing and while the engine will negotiate 072 curves, the cut levers on the GE models (Dash 8 and 9, ES44 and AC4400) will hit together on those curves. The 805 and plastic variants and the 745s avoid this problem. Just do as Matt says, and trim a little off the back of the box for clearance.

RM

If you're using curve-replacement turnouts -- i.e., O-72, don't use scale wheels. The arc of the curve extends through the frog and scale wheels will want to pick the frog and derail. "Straight" turnouts like #4's, #5's, MTH #6's and Ross 11-degree (also roughly  #5) seem to work well, but you'll sometimes get a "bobble" when the scale wheels pass through the frog, but the frog being straight, they don't derail. We have a curved Ross turnout on the club layout which we modified with a "frog point" to turn it into a closed-frog turnout because we had the occasional hi-rail car or locomotive derailing because the frog was so big and some of us started running scale-wheeled rolling stock.

One idea I had been bouncing around was replacing the axles on the pilot ends of the locomotives with the hi-rail wheels, which would make them a little more forgiving, but I haven't had a chance to caliper the hi-rail wheels to verify they're the same diameter at the treads as the scale wheels.

I generally run high wheels and swing pilots as a accommodation to my curves. I try to stick with 4 axle engines because they are nimble compared to the 6 axle units.

My radius is somewhat in the air right now as I draw and redraw the availible space around Matt's wye. I like 072 and must come down to 054 or even 048 ( I think that is correct for real trax...) I don't know yet.