I started playing recently, so probably the main problem should be my lack of experience... but anyway it is supposed that being rewarded with a new fief (castles in this case) should be a great experience for the player but it turned into game breaking for me. That's my history.
I started with a small war band (about 20 warriors) killing some bandits to earn some money, then I bought a brewery to get some regular income (about 160 per day). with this income I decided to join Sturgians to get married, grow my troops and accompany kingdom armies to get better loot, it looked a nice plan but... the king rewarded me with a castle, intermediately my economy sunk. I had problems to pay the small garrison wages so I couldn't increase my party size neither train more people to properly defending the fief.
The pace was agreed so I decided to kill bandits to be able to open more enterprises but then my income for the brewery get down to a 60 denars per day, I bought a second one but it had more or less the same result... Both of them were in tows - villages that weren't raided for long time and with cheap grain prices (around 10). I avoided caravans because I read that they are currently bugged.
So I totally failed to build up my economy to be able to pay a good garrison for the castle, mastly due the expenses of the castle. After few time war came again and the enemy toke back the castle loosing my few influence because that.
It was a sad experience, but my king wanted revenge and conquered a new Castle and .... again he gifted it to me, even worse, a second castle was conquered and also gifted to me!!!!
So now the situation was double terrible, two fief to defend with a ruined economy so I was unable to build up a minimum defense. They would be taken by the enemy in short time.
I stopped playing and started again.