I have similar issue. I captured a town and I was able to perform all tasks I wanted and leave the town. If however I would like to enter the town the game crashes. I'm able to play normally if i do not go near the town. If I do even without selecting the town it self the game crashes. In the same play-through, before this town I captured other one which gave me the same issues. So I was unable to defend the town and it got captured and now I can go in and out of it without issues. I uploaded my save to
upload.taleworlds.com and added the link of the topic in the description.