That is a very exciting new discovery for me to hear. Do you need to have a affiliation to that village, like a merc contract of belong to the faction? Or can you just be a rando?
That's actually a good question. I've only done it to my own settlements or the ones for my faction. And you have to be a vassal or king because you have to be leading the army, but it can just be an army with you and one of your clan parties so you don't need to spend any influence to maintain it. This makes it much more possible to maintain whatever army comp you want late game instead of having 800 different unit types because you're forced to fight with only prisoners.
I like to use prisoners to keep me going while I'm out on campaign, then when I take breaks between sieges get my army comp back to what I like by sitting in a town like this with one of my clan parties in my army with no men (they can't join an army with zero men but you can bring them into your army and then talk to them and take back all the guys) and they'll just keep buying up all the recruits the moment they spawn while I get up and do something else for 10 minutes.