The higher you get in level the larger outlaw parties get. You need to get a core group of trained soldiers together before you can train soldiers by fighting bandits at this stage. Try mercenaries if you have the money or training them yourself at the training grounds. The training grounds takes a lot of time but you will get cheaper better troops in the long run. Also Khergit troops train too cavalry in one step making the the most effective quick army, just be mindful of the stepee bandits they tend to stay too the east so villages in the west are fair game.
If you don't have money to upgrade your troops try tournaments or tax quests from lords. Try to get enterprises in each city of your planned faction so you will not have to worry about money in the future.
Finally be warry of how many small groups of bandits are around you. A group of ten is easy pickings but 5 groups of ten can join and give you problems. Only pick bandits to fight that you can easily crush if even a few of your soldiers die it might not have been worth your time.
