The fact you were able to do it with RBM 'keep at' distance is a game-changer. Don't know the complexities of modding but the fact this can be done as a mod must've meant TW was always able to do it 'relatively' easily too but rather opted to have them in hugging distance at all times.Tweaking the AI of individual soldiers and entire formations is definitelly doable. What is really hard to mod is making AI properly switch their weapons (spear from distance vs sword in close quarters, etc).
With switching weapons, guessing it has to do with them 'picking' their targets and the ensuing logic; especially with enemies coming in/out of their 'range'.