A real signal system depends on the two rails being insulated from each other, but not using them to carry opposite polarity traction power. Three rail would be an excellent way to build a railroad type signal system if you have track with the two outside rails insulated from each other. Then making the red and green light signal system work is easy. The problem will be if you expect the system to also stop the trains using insulated center rail dead spots. This becomes difficult if you expect to operate in both directions. Also, cars with two rollers, like some passenger cars, can carry a train through a dead spot.
To operate color lights you can connect power to the outside rails in series with a relay. When a train is present the wheels and axles will complete the circuit between the outside rails and operate the relay. A real railroad uses a scheme, in its simplest form, where power is applied to both rails at one end of a block through a series resistor. At the other end of the block there is a relay connected to the two rails which is held energized by this power. This energized relay provides a green signal at each end of the block indicating it is clear. When a train comes along the wheels and axles short out the power, the voltage drops to zero, and the relay changes state causing the signals to go red indicating the block is occupied.
