I'm not sure if multiple tiny armies are really the big offender here. It might just be that the there are too few defenders, too slowly genereted. When I played one campain with the bannerlord tweaks mods which massivle boost the defensive side and garrions and all that stuff a 900+ men army won't take barely anything. A settlement had around 1.5k defenders ~400 garrison and ~1100 milita there was also a crazy settlement with a whopping 1.8k defenders, took only a 3k army to take it
Anyway castles also had always over 700 defenders so it might be enough to boost militias, garrions, food storage and food production to severely slowing down the snowballing.
Not to mention that leading an army of 2k+ soldiers requires lots of parties, influence, cohesion, provision and so on. Getting all of that and replenshing it after a siege is not quickly done.