There is a lot to answer here and i don't have the time to give a long answer. but this is are my suggestions. well guess i did anyway...
1) You can limit the wage you want to be used for a specific garrison, i usually don't but it looks like in your case it may help.
2) You can also limit the wages for your other parties too
3) Funny enough you don't really need good units in your garrisons, recruits will do, especially if the town/castle is far from a war zone, still security is a thing so let's say don't leave many tier 4 & 5 units in garrisons.
4) Garrison have an auto recruit feature: when i get a new Settlement i leave 50 "bad" units in there and let the garrison recruit the rest. has worked real well in my last campaign.
5) Good Governors can help with Towns, like a lot. too bad good governors are kinda rare but that's what the wife is for. (don't kill me pls, it's a joke)
1)Set them on Defensive in the War screen under Kingdom stuff, you know what i mean. This will make it so that they will prioritize defense first and then, when the enemy lost his few good armies go and attack. This doesn't mean you are not going to lose any army but it's less likely and my go to strategy.
2) Use "Balanced" & "Offensive" only when you have AT LEAST the entire strength of all the faction you are at war combined.
3) Take 1000 man or more and siege, the AI is for Defense.
4) Don't KILL your vassals: If a party in your army has less then 50 units let them go and let them recruit, it's not worth losing a party leader for an handful of units.
Kingdom management and Policies.
1) Since you are low on money and sounds like your kingdom owns a few share of towns try implementing this two "State Monopolies" & "War Tax"
2) Party size could be bigger and not just for you: Noble Retinues will give a boost to all Clan Leaders and Royal Guard is essential for you as it gives the ruler party an extra 60 units.
3) Any rebellions? just vote "Forgivness Of Debts"
4) Sounds like you need help with vassals, i have made a tutorial video a few months back on how to get your hands on vassals easier, and yes you need around 25 Clans if you want to take over the entire map, or a good axe.
5) Smithing is fine. i don't like it or use it but if you are struggling don't feel like you are cheating for exploiting the smithy.
1) This may sound stupid but like any good kingdom it must be very well defended and the best way to defend a kingdom is a mountain or a body of water. look for the factions that aren't bounded to anything and push for their territories, as you are in the S. Empire and N.Empire i would suggest going for the Khuzait, form there you get an hold on Sturgia, maybe losing a few empire towns in the process. and then you kinda know what to do from there.
2) Leave the Aserai for last, i think they have always been the last i took out in any of my campaigns.
If you are interest on the video on vassals just look for my Nickname on the Tube, it should be there.
Hope this helped you to keep your Bannerlording going.