Hi KC, welcome to the Forum! You have some great ideas, and I really hope that you're able to follow through and implement them! I hope you share your progress by posting videos every now and then.
Personally, I wouldn't use the insulated rail method for block detection. The redundancy of two ground rails is one of the best things about 3-rail O gauge. Lionel makes an infra-red detector, and I imagine that there are optical sensors available and used in other scales.
Regarding routing, train priority, etc.: along with 3-rail O gauge I've messed around with V-scale (virtual simulations.) Specifically, the Auran Trainz series. Using this software, you can build a virtual "layout" with passing sidings, signals to control junctions, etc. You can also assign AI drivers and create "rules" allowing trains to traverse the network in an autonomous scenario. However, at least in early versions of the software, when you start creating loops the AI gets confused and usually required some degree of human intervention. You will definitely need to designate train priorites, decide how many blocks "ahead" a train will reserve at any one time, and perhaps designate some arbitrary point on the loop as your world origin (think of it like the international time-date line for your railroad!) I think you'll find that in a complex network with loops, alternate routes, etc., the logic gets pretty mind-boggling. But that's what makes it worth doing!!
I encourage you to purchase a copy of Trainz Railroad Simulator to experiment with rules and AI in a software simulation before you create your own code for the 3-D world. Can't wait to hear and see more about your project!!
