I think a handy addition would be something called a recruitment keep that can be built in a castle or town, which triples the rate a recruit is taken, perhaps 2 per day upto a maximum of 100 recruited soldiers in a town and perhaps 50-70 in a castle. To balance this out it takes 6 engineering to build (either from a companions skill or your own) and 12 thousand denaari. I for one find it very hard to defend my castle once i strike out on my own because they enemy just arrives with 2k men leaving me barely alive and basically unable to go and recruit a hundred or so recruits because my castle will be sieged/villagers burned. So if this was implemented id target cities with this addition in place to help out.
A second thing is if you own a castle you can hire 1 patrol to guard the sorrounding area and villages that you own allowing you to more freely roam about. (Not sure if towns can do this while you are a vassal of a kingdom, i always only accepted castles)