Did you start a new kingdom?
Few thoughts:
1) Gather a bigger army in your refuge. Move them to your new holding once you conquered it. You could start with 1000 garrison.
2) Train better troops. If you have a elite army they will die trying to climb your walls.
3) Choose your defensive position wisely. Find a area in your fortress where you can position your archers to attack the enemy from the flank/behind and your melee troops to hold the line.
Bebbanburg is a good example. I dont defend the wall. Instead I always go up the stairs. I hold my archers to the side and my melee to the end of the stairs. Invasion force will come in small groups (3-5 at time) and my archers will eat them alive.
Remember to use commands to control your archers. Dont let them fire at will or you will waste your arrows.
4) That comes with 3. Dont just conquer any place in the map. Choose one that is easy to defend.
