Down in Baltimore they have a railroad museum full of steam locomotives. It goes right back to the first steamer the B&O ever built, Tom Thumb. Walking around the museum you can see the machines evolving from the first experimental model to the humonguous machines built in the 1940's.
The first model, Tom Thumb, of the famous horse race, is an obvious design. Take a four wheel chassis and plunk a steam boiler down in the middle of it. Firebox on the bottom, fire tube boiler above the fire, and a stack, for draft, atop the boiler. Vertical cylinder driving the wheels thru gears. The number 2 and number 3 engines, dubbed "grasshoppers", are bigger and heavier versions of Tom Thumb. Design is moving forward, number 2 grasshopper still has the expensive-to-make gear drive. Number 3 grasshopper drops the gear and replaces it with cheaper and simpler drive rods.
Grasshopper design reached a dead end. The grasshoppers were not very big or very powerful and they could not be made larger. A larger vertical boiler would be taller and would not fit under the bridges. A look around the museum shows the stacks on all the locomotives rise to exactly the same height, namely the height of the lowest bridge on the mainline.
Follow on design, which lasted until the end of steam, was the horizontal boiler. Technical difficulty of horizontal boilers is how to get the heat of the fire to flow sideways thru the boiler. Heat rises as we all know, and sidewise is a long way from up. The solution was to put the stack at one end of the boiler and the fire at the other. The draft of the stack sucked the flames thru the firetubes of the boiler. The final trick was to vent the waste steam from the cylinders up the stack. As the steam rushed up the stack it created a strong draft that kept things burning merrily.
In the museum you can see all these design improvements coming on very rapidly. It took less than 20 years to go from Tom Thumb to the American standard locomotive design that stayed in service until the end of steam.