A strong garrison in your castle is a great idea, since they're the first line of defense against an enemy that chooses to attack. I've found after you start getting castles, especially after the second one, it becomes a fine balancing act between spending too much money and losing your castles...