Your comment made me revisit the mods I had and think bout if any of them modify the save game data. Turns out, yup, Improved Garrisons does and even includes steps in the mod's portal to describe the uninstall process. Went through that, "delete all Improved Garrisons Parties" under mod settings, saved the game, disabled the mod, and was able to siege a town without crashing.
So this crash is related to that mod, so feel free to close this thread since it's not related to the base game.