Yeah I see that as an issue as well, I think the player should be the major balance changer, once he decides to do it. Otherwise it gets very dull, when you live the trader life and start something else after a few years just to have only one major faction really left.
It can be all dynamic with as mentioned, garrison bonuses, hidden stat bonuses, or the losing factions getting alliances and help, while the aggressor might have to deal with a second front opening up, or being more susceptible to peace treaties after a successful push.