It is a pain. Slightly less painful workaround: dismiss your clan members from the army just fractionally before you enter the settlement. They always pause for a few seconds, so you have time to enter, do what you want and leave again before re-inviting them (at no influence cost). The downside is that they can't buy any food (but if you stop at villages, they can).