Hello CktheFat
I use to be in the same boat as you. King Harlaus gave me Ehlerdah, which was raided almost as soon as it was restored from the previous raid on it. The lords of enemy factions go about raiding the villages of other lords whom they want to provoke and anger, so if you are in every battle defeating lords, they will target your village. I have enemy lords who are "friendly" and "gracious" towards me (55+ relation) who attack my village constantly because I repeatedly introduce them to my sword. Anyways... I used to have to fight sea raiders constantly just to keep a balance of 3,000 daenars, so that when payday came I would have about 1,000 daenars left, despite having a village. I had a brewery too, but that brought in about 200-300 daenars a week, so I was still running a very tight budget. My big break came when Swadia took Shariz from the Sarranids, the Sarranids promptly took it back, because of it's small garrison, and then I followed right after that and took Shariz. I requested King Harlaus to give me Shariz, and he complied. I immediately ran out and recruited all of the villagers I could get my hands on and shoved them in the garrison. I still get about 7,000 daenars a week from Shariz, which helped propel me towards a nice healthy budget of 200,000 daenars. I used this same tactic for Weyyah castle, sieging it immediately after the enemy faction takes it back. With the large sums of money coming your way, you'll be able to invest in expensive, but more lucrative businesses in the towns.
As a side note, I find I can take a Khergit or Vaegir (especially Vaegir) castle by myself if it has about 150-160 men when I have about 100 well trained (read: knights, sargents, and sharphsooters) men.
Hope this helps! In sum: resiege a castle immediately after it has been taken, and you will only have to go up against 60 or less men of mid grade quality.