Castles are pretty disappointing and feel like Warband.
To keep it simple, I think castles should have Notables, who, instead of being landowners, merchants and the like are Captains, Sergeants who will have access to more elite troops. AFAIK the noble troop line is random for which Notables...