There's a big problem with that. During the siege I can see that in the castle or town there are no food stocks at all, so they should like starve, desert from the settlement or die from starvation. But nothing of these happens, it seems that the only way to reduce the garrison without storming the settlement.
But, to be honest, the whole settlement stat system needs lots of tweaking.