Castles are pretty disappointing and feel like Warband.
To keep it simple, I think castles should have Notables, who, instead of being landowners, merchants and the like are Captains, Sergeants who will have access to more elite troops. AFAIK the noble troop line is random for which Notables get access to, so instead, maybe they should be linked to castles.
Castle 'Notables' would have quests could be more military-based, such as troop patrols, or garrison training, and in turn, these Notables would have access to higher-tier troops upon the get-go.
Of course, balancing needs to be kept in mind, but I think making castles more than just the fortresses they are could do the game good.