I don't think it should restrict you from going past the castle, but there should be consequences for doing so.. Like if we could build barricades on the map near the castle that wouldn't "stop" an army, but it slows them down greatly, and at the same time the castles having long range catapults that fire at the army trying to pass by (but too long range to fire on an army sieging the castle). Any regular parties (not in an army) can pass by the barricades to raid villages... So basically, an army could get by, but they would lose a ton of troops in doing so and would be better off taking the castle first...
Or something to this effect