I also get horrible framerate and studdering during sieges, only for the game to crash when or soon after the enemies reach the walls. Has happened to me three times while defending in a siege so far. When attacking, the AI seems very inclined to rush out to fight you, avoiding the actual siege battle. I feel the developers may have set it that way on purpose from knowing that sieges are still rather buggy.
I have been playing the game in borderless windowed mode, like mentioned by another poster, so not sure if that has something to do with it yet though.