Just starting out can be really hard on the budget as you say. I had the same problems and here is what I have done over about a five-six year period so far. It took me a while to figure all of this out and to decide what I really wanted. I knew very little about real or model railroads when I got back into the hobby, not that I know that much now, but I have mostly decided what I like and gotten a few things figured out.
I decided on command control only, no conventional and no plans for any later on, but I could run conventional using the TIU's variable channels if I ever need to. I used Lionel PowerHouse 180s (PH-180) for power to my TIU channels. These are a constant 18 volts at 10 amps and have excellent breakers, arguably the best breakers of any O gauge power source. The Z4000 and ZW-L are very nice transformers, but the PH-180s are the best bang for the buck and the very fast breakers help protect your train's electronics. When I got mine they were about $80-$90 each, brand new at street prices. However, they may go up a little when the next batch comes out, but only Lionel knows for sure?
After I got my first starter set I added one more. Next I got the DCS system and 1 PH-180. I then got that going with my temp layout which was now 2 separate, small loops of track. I used a Z500 from a set to power my TIU through the Aux port, recommended when using more than one TIU channel, and powered the 2 fixed channels with the PH-180. Next I studied all the track systems available and decided on Atlas O track. I started out liking Fastrack, but it was too noisy so I ended up going with Atlas, which I really like better anyway. I then started buying track and switches. I had a nice layout all planned for our existing basement and was ready to start building it when our old house started having more major problems. We had already fixed quite a few over the years. We ended up moving to a new house so I had to start over with layout plans.
After moving I got a 6'x16' Mianne benchwork kit and put up a fairly nice layout on that, which is what I have now. I went with Mianne because it's easy to reconfigure and my plan was to add on when the train budget recovered from the initial purchases I had been making. This stuff adds up ($$$) very quickly, as you said in your original post! That's about where I am now, planning the expansion and getting ready to start adding on. I just need to stop buying trains and concentrate on routing all train funds to the expansion.
I also decided to stay with diesels only, mostly modern, as that is what I see around here all the time and they also don't require the larger curves like the steamers do. The diesels are also a lot less expensive than the steamers. I have also gone to the scale sized items as opposed to the semi-scale like Railking and Lionel traditional items, as I have now decided I liked the scale size better. They are usually a lot nicer, more detail, etc. I also decided to try and stick with mostly one road name, BNSF, to help keep my purchases somewhat in check. Lionel doesn't offer much in BNSF, but MTH usually has more in each catalog than I can afford. Sticking with one road name does reduce the selection and help the budget though.
Not that this is the right way to do things, but it might give you some more ideas to help you along. Everyone does things differently so the right way is the way that works best for you and your layout. I found the OGR forum fairly early on, read many posts here and that helped me a lot in selecting the path I chose. It was a combination of many ideas learned from the others here and I am still learning.