It does make sense particularly when you consider that a colony needs to collect a surprising amount of nectar and pollen simply to maintain and grow the colony itself for much of the season, before it can then start to accumulate and store a significant surplus for later.
In my area there are relatively short and defined periods within the foraging season when there is enough forage for bees to collect nectar over and above what they need to maintain themselves. My bees are probably able to find something to forage between say April and early October (if they can find Himalayan Balsam), say roughly 6 months or less active flying weather. In that period they only have probably two distinct windows of maybe three weeks each when if the weather and soil moisture are favourable and they can really pile in a surplus over colony maintenance. So in that scenario only larger colonies can really make hay by putting lots of workers on foraging duties, smaller colonies simply cannot capitalise on such short periods to the same extent and may respond by rearing more brood which by the time they are foraging age the boom times may have ended.
Having said that, building larger colonies and keeping them that way for a prolonged period is not without its challenges either and pointless if your hives are static and the area you are in simply cannot sustain them over the whole season - its all horses for courses and working with what forage is available to you in your area, not just following an abstract theory.
I was fishing Chesil beach this evening and conditions were far from ideal. I could have stuck out with big baits for fish that might have been present but probably were not, or do as I actually did, which was to fish with small baits for fish that were actually there. As a consequence I had a good evening of catching (and returning) plenty of small stuff rather than watching a motionless rod top in the dark for the small likelihood something big might just turn up.
Again, working with what you have actually got around you rather than what you might wish you had. Very Zen, me..