But imho there should be no hard limit on how many troops you bring into a battle, it should be more like a softcap, similar to party speed:
There is a base chance for Ambush Success to happen, that decreases, the larger the party is from a certain party size (imho somewhere between 50 and 100 men should be the threshold, where success chances start to drop).
This can be counteracted by the Roguery Skill (Usually I would attribute Ambushes to "Tactics", but since TW associates with Tactics purely the auto-resolved battles, put it into Roguery), either by certain perks increasing Ambush Chance by flat percentage or, more dynamic, by every point in Roguery Skill giving a bonus to Ambush Success Chance (e.g. for every point in Roguery you get +0.167% Ambush Success Chance, making it ~+50% Ambush Success Chance at 300 Roguery Skill).
Terrain should have influence on that too, with flat, open terrain giving a straight -50%, so only true experts could ambush there.
We would see Blocs Unit Marching for the Ambushed happening in a straight line, with the new battle map system the map being generated in a way, that this path is open, while the sides are forests, high grasses, a plateau cliff or something similar, where attackers could hide crouched.
The Attackers would have their spawn rooms to the sides of the enemy, where they could set up their troop formations, close enough so they can reach the enemy before they reform their troops.
The Ambushed should only act on the base of which formations they can see, so after the oppening you could still surprise the enemy with formations you keep in hiding to strike them in their back.
AI should be able to do this, for this to work, Warbands/Armies in Ambush Position must have a drastically reduced detection range on the campaign map, with only a high detection range beating that, counteracting the Ambush Success Chance.
So basically, from the players perspective, an ambushing party would be undiscoverable (bordering invisible) right until they stand next to them, too late to escape.
This would cause players being in a war being rightfully wary about approaching bridges or narrow paths or forests and really being surprised.
If the player is ambushed or part of the ambushed army, he and his army won't get the pre-battle set up screen, his troops will spawn in a premade column and he gets instantly attacked by the AI.
I think this is the best way to simulate a surprise attack on the player, because the player can't really be surprised, if he suddenly gets an "You're being ambushed by X!"-window on the campaign map.
In this way the player would find him-/herself at the start of a very chaotic battle, being forced to reorganize the formations from his mixed marching troop column in the middle of combat to effectively counter the ambush attack.
Now one thing is missing: Usually, outnumbered AI parties don't do high risk attacks and attrition warfare, they run away, until they find enough allies in their vicinity to attack you together and beat you in an uneven open field battle.
So how to get the AI to engage in attritional battles?
Enemy Lords too far away from allied armies and of cause bandits should be more inclined to engage the enemy in ambush battles.
Obviously they wouldn't fight until the last man, at some point enemies start running.
This should include the nobles too.
In Warband, nobles could escape after defeat, but it wasn't implemented for the player (player always got captured) and for the nobles imho it was poorely implemented, that guy could lay unconscious on the field of battle, surrounded by your troops and yet miracelously they escaped after battle.
Now we got the opposite, they get captured, always.
Imho this should only be true, if they're striken unconscious in battle, then there should be a 100% chance of them being captured.
When they're dehorsed but still alive, when they rout, they should attempt to run out of the map like any other foot trooper, giving you the chance to still find them and strike them down, to capture them.
If they make it out of the map, they should be gone, only counteractable by survivng allied cavalry, giving you a low chance to capture them after battle (maybe something like up to 25%, depending on how much cavalry is still alive and combat worthy to also perform "lord hunting").
However, if they rout while on horse, there should be a near 0% chance to catch them, with up to 5%, depending on how much cavalry you got.
Now, tieing that into an Ambush mechanic:
To simulate Lords being a somewhat intelligent person and not a kamikazeing cavalry guy, they should automatically be more inclined to more watch from the distance their troops engaging the ambushed enemy, only attacking themselves, when the ambushed enemy is beaten and fleeing, accompanied by a small number of cavalry men acting as body guards. Maybe, but this is optional, this could be tied to new traits, marking characters as either brave or cowards, having an influence if the lord attacks at all and maybe runs earlier or if they engage in battle like in regular battles and don't run away or run away later.
The same should be true for a "bandit boss", trying to escape after a failed bandit ambush.
Once a certain percentage of his troops are dead or unconscious, equaling a failure of the ambush, the lord/bandit boss routes and tries to escape to the border of the map.
If he escapes, the rules written above apply to determine the outcome of the escape attempt.
If the lord/bandit boss escapes succesfully, both parties should be in an disorganized state, as usual influenced by perks, how long that stat state lasts for the individual parties.
This way, a high leadership character could try to pursue and catch the still disorganized enemy after the ambush, making this the "high risk" part that we strive for.
If the character (player or AI) discovers an ambush on the campaign map thanks to sufficient scouting skill and engages the ambushing party (or they engage you from invisible state but don't get granted a succesful ambush), a regular field battle occurs.
An Ambush should take time to set up, if a lord is too far away from an allied army or the kingdom as a whole is too weak (total number of field troops) to beat the enemy in open field battles, they should choose a friendly city close to the enemy and on the path way from an enemy city to the friendly city set up an ambush.
This should not work while they're seen by an enemy party or army on the campaign map, then they should run away as it is now.
Set up time should be reduced by roguery skill and/or perks in the roguery tree.
Cultural traits that gain terrain advantages (so Aserai, Battanian and Sturgian Cultures) should also grant a bonus to Ambush Success Chance, if the Ambush is setup in such a location.
Imho also Battanian, Aserai and Sturgian Troops should get Skill bonuses in their favored terrain to make fighting them in their accustomed terrain/climate tougher, but that is another suggestion.