Well, here's the difference, guys.
You're forgetting that the three commercial locomotive builders were in business to MAKE MONEY designing, building and SELLING steam locomotives. They had a couple or three things to sell - engineering, workmanship and quick delivery dates.
In the 1920s, these builders (and if you've forgotten, they're Alco, Baldwin and Lima, in alphabetical order) were trying to develop SALES TOOLS that would keep them going. They were not, as some of you seem to believe, kindly, benevolent souls intent only on purveying to the customer the most profitable or efficient machines possible; they were trying to SELL LOCOMOTIVES, hopefully at a profit.
In this era, Baldwin did little innovating, relying on high production capability and the export market. Alco thought that the three-cylinder path was the way to go, obtaining the US license to use the British Gresley valve gear to operate the middle cylinder, and effectively forcing Baldwin to use another method for its few three-cylinder locomotives.
Lima went with the big firebox/big boiler concept, and called it "SuperPower".
A large part of the success of superpower over the three-cylinder setup was the fact that it represented the smallest deviation from traditional locomotive engineering, promising the least increase in maintenance costs.
It must be taken into account that the builders were going to insert their own ideas between the needs of the customer and the finished product. After all, that's what they employed high-powered engineering departments for. And they sometimes, ten years later, helped EMD's Dick Dilworth sell his FT diesels.
For instance, Alco's Alfred W. Bruce in 1927 recommended the 69" driving wheel for "general freight work". There was a good reason for this; the 63" driver common then did not have enough room to adequately counterbalance the heavy rods necessary to transmit the power of the new big engines (disc drivers and lightweight rods were in the future). The higher driver helped the problem in two ways - it had more room for counterbalance weight, and it reduced the RPM necessary to run at a satisfactory speed. But many of the locomotives built to this standard found themselves in heavy service on grades, and made them easy meat for a diesel-electric. They had horsepower to burn at high speeds, but were bought by many railroads which couldn't use all that speed, but needed more slow speed power.
Baldwin seemed to have more counterbalance problems than the other builders; even an 80"-drivered Northern like the ACL's R-1 (now THERE's a handsome locomotive for you) had those troubles.
And all the builders discarded the Mallet compound in favor of the faster simple articulated; these saved the builders in two ways - they were usually bigger and heavier and therefore more profitable to build than comparable Mallets, and the builders didn't have to go to the trouble of finding out what made the Mallet slow, and doing something about it. There was a reason why N&W's Y-6 lasted the longest; they were fast enough for N&W's conditions, had the economy of compounding, and had a boiler the size of a big 4-8-4 - they wouldn't have been as profitable for a builder wanting to sell simple 2-8-8-4s and 4-8-8-4s.
There were two railroads that built a significant proportion of their own power; the Pennsy in its Juniata Shops, and N&W at Roanoke.
Pennsy built a lot of locomotives to basic 1910s design standards; they had about 1500 K4 Pacifics, L1 Mikados and I1 Decapods. They ignored everybody else's progress until producing their best effort, the M1 4-8-2 in the 1920s. Even this, the best of their homegrown designs, was designed to be hand fired; it took some work on the test plant at Altoona to convince them that only a stoker would get the best out of PRR's best design. But when the World War II crunch came, Pennsy didn't have a modern design and had to get one from somewhere else.
N&W was a special case. First of all, its top managers over the years had at least some operating experience; in other words, they knew what made the wheels go around. So N&W wanted "purpose-driven" locomotive designs. Post WWI saw it concentrating on USRA's heavy Mountains for passenger service and the USRA 2-8-8-2 Mallet for freight. After its regrettable misstep in 1926 with the 63"-drivered K-3 4-8-2, it came up with the Class A 2-6-6-4 in 1936. N&W's president at the time was a fellow named A. C. Needles who, from all reports, was as prickly as his name - but he knew what made the wheels go around.
N&W was identifying the problems that kept the compound 2-8-8-2 from being faster, and was doing something about them; this resulted in the Y-6 referred to above.
N&W's 4-8-4 was an extension of all its thinking up to that time.
N&W did its own mechanical thinking. It knew what was going on elsewhere in the locomotive industry, and selected those features which would further its own goals - maximum gross ton miles per train hour PER DOLLAR. It didn't have a commercial builder inserting non-desirable ideas into its designs, or trying to make money on a low-bid contract. And, it didn't care what others did.
If N&W's engineers had taken a design for a locomotive like Lima's 2-6-6-6 and laid it on Needles' desk, I'm convinced that he'd have kicked their a**es back to the Motive Power Building and told them to start over - a locomotive weighing that much that was no more powerful at low speeds than the class Y3 USRA 2-8-8-2 . . .
Well, of course all this long-winded discourse is just my opinion, but . . .
EdKing