I'm not sure about Nords, but I'm playing with Swadians and I can quite easily capture castles with about 40 high-level troops. Things like Hired Blades, Ironclad Knights of Swadia and so on. Well those and another 50 other units.
You should try focusing on Castles that other kingdoms just captured recently, they will have less troops and less of them. I would recommend attacking castles with the least amount of top troops.
So a castle without Scourges, Ironclad Knights, Goliaths, ... are the best bets. Or with as few of them as possible.