This is the problem. Once a lord loses a single battle, he needs money to recruit more troops, but the upkeep cost for his remaining troops means that he can't afford it. The lord who wins his first battle gains money in the form of loot, and can afford to replace losses. The winning lord then goes on to beat another weak lord who has lost a previous battle, taking whatever little bit that lord has left, and the situation just keeps getting more one-sided. Like a snowball rolling downhill, it can grow to massive size.
The solutions are to provide enough of a passive steady income for defeated lords to regain their original size after a couple of weeks, and NOT recruit past the point where they can afford to pay and feed their armies on that income. As it is, they just sit there until besieged, or are easy prey for routine bandits if they leave. A lord should be able to draw a FEW troops from his castle garrison, but should prioritize replacing them as soon as new recruits are found, rather than leaving the garrison depleted for more than a couple of days.
It would also help if nearby lords would come to the assistance of nearby castles under siege, provided that relations between the besieged and his neighbors is decent, rather than having the winner of a field battle then able to besiege the defeated lord, and nobody intervene. Once a faction reaches a significant size, the other factions should prioritize them as an enemy, making it increasingly difficult to "blob" too far beyond a faction's initial boundaries. Newly taken castles should require a decent garrison or risk being re-taken, slowing any snowballing when a garrison is created from the besieger's army and while replacements are recruited for that army. Villages and towns recently taken should have a risk of rebellion, where you have to go in with troops and suppress it by force, or have it revert back to its previous owners if left in rebellion for too long. Basically, if you take it, you should have to work to hold it until it accepts its new owner.
In any kind of "balanced" strategy game, you need "economies of scale", where it pays to grow because you simply have "more" than the next guy. The big tend to get bigger, at least until they don't. There are also "diseconomies of scale", where costs increase exponentially with size, as communications take longer, your troops have far more perimeter to cover, responsibilities in distant locations need to be delegated, leading to inefficiency and corruption, and you have competing centers of power and wealth competing and backstabbing within your realm. Push to hard and grow too large, and you're facing multiple hostile opponents ganging up to stop you, revolts from within, and reduced income due to inefficiencies. The trick is to make it beneficial to grow, but difficult to grow BIG, so expanding slightly isn't difficult, but it takes a tremendous effort and skill to conquer the entire map in spite of the increasing difficulty.
