So after slaughtering all the Brittons my men banded together to form a circle and dedicated their afternoon to taming pokemons (a.k.a ladybugs)
As you can see it says "can't retreat, there are enemies nearby!" the enemy for some reason is beneath the floor, that must be why there staring at the floor or else they lookin for ladybugs but I doubt 103 veteran axeman spend their free time catching ladybugs.
When I move away from the group lets say about 20 ft. now I am able to retreat so there definitely is a enemy stuck under the floor, I don't if this is a design flaw or a bug, but it really sucks. You can also see my men is the background of that picture so you can see the distance at which I moved. I replayed the siege about 4 more times and each to that gathered around that one little patch of dirt, so the guy is in only 1 location, and there is no enemy left on the map besides the guy in the floor believe me I checked.