No you're right to micro them, it's very important IMO in bannerlord.
I would do a as you, but use even more HA and Archers! In fact I would leave Infantry as garrison filler once I had more ranged units, or just the low tier units to use as a disposable shield for foot archers.
Using Jav Cav on fallow me is Pro! I would position archer and HA slightly apart at the rear of the map so they slightly converge fire on the center where enemy approaches and kite all mounted units into their fire range and kill with player and Jav cav. I the Cav has shields you can put them in front of stationary HA in SW formation too, it's pretty good defense, basically enemy archer will shoot at them while you HA/archers shoot the enemy archers and obviously you come out ahead in that exchange. Of course the noble Jav cav may be too good to subject to this...
As the enemy get closer I would move the HA formation more to their side, then to the rear to **** over any shield wall units remaining in enemies, putting archers on advance should make them back up, but be careful if the enemy is too close already (they may melee instead

)