Question

how did Madison argue that self interests and factionalism could actually lead to good governor

Answer 1

He didn't. He just argued that self-interest and factionalism would prevent any ONE faction from gaining permanent power over all the others and turning a republic into a dictatorship. Keep in mind this was the Founder's #1 fear, inasmuch as all previous republics had, sooner or later, ended up as dictatorships.

The way it works is this: one faction becomes temporarily larger, and for some curious fluke of fate -- they won the war, cured the economic depression, have an unusually charismatic leader who gives great speech -- they acquire a lot of power. They then use this power to favor their followers and punish those who oppose them. Then people join the faction, to avoid being punished and gather the goodies. That gives the faction (or rather its leaders) still more power, and to squeeze still more people into the coalition, and avoid losing power, they need to start ruling undemocratically. The process is self-reinforcing, and always ends up in tyranny.

Madison argued that this was less likely, not more, in a large republic with many competing factions, because Step 1, one faction gaining power over all the others long enough to start reinforcing its position, couldn't occur. There would be too many factions competing for power, and too eager to tear down quickly any that temporarily achieved ascendance.

He may have been right, or it may just be that it takes longer for a larger republic to decay into tyranny than a smaller. The jury is still out on that. So far, the American Republic has lasted about half as long as the Roman Republic. If the United States does not decay into tyranny within the next 400 years, then at least Madison's work will be the world record holder.