How do I increase garrison and militia men size for towns?

[talior3]

I keep conquering towns through siege but the moment I leave for a bit they get attacked because of low number of men.

Any tip on how to avoid that? I can't give away my own soldiers I still haven't maxed out mine and my companion's parties.

 

[Pejot]

I can't give away my own soldiers I still haven't maxed out mine and my companion's parties.

This is your answer. You're not ready to capture cities for yourself.
Militia is growing back in numbers in time but You need to be able to put a decent garrison cause militia is mostly useless.

 

[lostsanctuary]

go to the town then select Manage Town, upgrade militia barracks and put daily defaults to train militia but as the post says before me even then it will take time for it to grow. The easiest way to protect your town after a siege is by recruiting from villages and towns or putt your own infantry/archers into the garrison.