estevesbk - sounds like you should abandon ship, or that castle at least. After starting a few games, i realized you have prepare from day 1 (figuratively speaking) to build your kingdom, it's hard to do so too early on. I'm at day 200, with 3 castles and 1 town (all given by the King). Dhirim and 1 castle are close together, so I've stocked about 500 well trained troops between the town and the 1 castle, in addition to the 120 men I bring with me. I have about 90k Denars stocked up and in peace time build up the villages around Dhirim so I can recruit large numbers quickly.
I don't plan on rebelling against the Nords for another 50-100 days, I want to add at least another 300-400 men to my garrison, because i know the Nords or someone else will come to my doorstep with 1000+ troops, I plan to weather the storm and build up a few Lord/Companions then slowly expand against a weaker faction if possible.
Best advice tho i got from another post, be ready to lose castles....fight enemies in the open field where you smash them with superior troop quality over and over again until they can't muster an army against you, offence, offence!
Anyway, i hope this helps in some small way, i know how you feel!