Five. That is the number to build toward and the number to stop at, and it is worth structuring your testing around rather than treating as a loose guideline.
The reasoning is about what happens after the test rather than during it. It is rare for all five ads to survive. You will usually turn one or two off, which still leaves three or four running. If you had started with one or two ads, that same attrition leaves you with a single ad, and a single ad limits how far the adset can scale, because facebook needs several ads competing to win more auctions than one ad can win alone.
Going the other direction has its own cost. Above roughly seven ads in one adset, budget gets spread so thin that facebook cannot spend enough on any individual ad to learn anything useful, so you end up with more creative in market and less information about which part of it works.
A single-ad adset is not always wrong. Cutting a losing adset down to one proven winner is a different situation from starting a test with one untested ad, and there is no reason to add ads back in purely to reach a count.