Yeah, I currently have a town I lost to rebels (Hubyar). My only town. When rebels took it I besieged it planning to starve out the defenders. Then I got 2 other armies to help, but they also took control. We weren't at war, but then the Aserai declared war on us. They came and attacked our besiegers, but we won that battle with about 25 troops left in fighting condition. We recovered and now have over 300 troops to their 75 militia. The walls are down, they have no siege engines, we have battering ram and siege towers, but of course we don't attack.
I lost the town because the stupid game instituted Debasement of currency and I am empire culture. So loyalty is -3 because of my culture and another -1 due to Debasement of Currency. I do have an Aserai culture governor there (my wife). When I took over the town there was no fairgrounds upgrade in place so thats my top priority. Unfortunately the highest I can get loyalty is 20%, which gives 0 build. So I can't build anything and the town is constantly under risk of rebellion. At first 3 villages were flaming from having been raided. When they recovered they immediately had tasks for me. When I was over 15 loyalty I risked going out to do the tasks. I did 5 of them before it eventually rebelled. I spent probably about 2 seasons at the town just trying to get up to 25 loyalty so I could build stuff. It simply couldn't be done. Although this is off-topic for this thread, here's some questions for you. What use is a governor of the town with 300 troops if they don't do village tasks or fight off rebellions? They also don't take care of hideouts in the area. Nice that they can give you a bonus to loyalty, but they don't actually seem to perform any actions as governor. Why would any lord besides the ruling clan vote for Debasement of currency? Yet it receives 100% approval and can't be overridden unless you are the ruler. Why is the penalty for ruling a non-culture settlement so stiff that it basically destroys any chance of owning it unless you put all resources into getting fairgrounds in place (if able). I understand having a penalty that can affect your income from the settlement and force you to have a higher military there. But its ridiculously too high of a penalty.
Edit: Here's an update from my game play tonight...Rhagea came to Hubyar with an army of over 900. I thought, "Oh good. She'll gain control because she is the kingdom leader and she'll attack with her fresh troops that have them outnumbered about 6 to 1. Nothing happened even though the walls are still down, there's no ballistas/catapults in the town, we have trebuchets, and we still have the ram & siege towers. I decided maybe if I joined the siege and then left it would trigger it. Nope, didn't work. As an additional note, the rebels joined the Aserai, so now its even part of a kingdom we are specifically at war with.