Do you remember which castle/town? Right now there are certain ones that have terrible performance due to poor AI pathing, which will drop fps regardless of your system. This seems to impact the majority of battanian castles/towns and a few others, but there are some castles/towns with good performance so if you have ****ty performance on all of the maps then it might be your 2700 bottlenecking you (if this is the case then limiting battle size should help)
Oooh that makes a whole lot of sense, I get decent performance (everything runs fine on high settings) on normal battle maps, plains hills rivers whatever, but some of the sieges have small lags/FPS dips, and some of them have crazy FPS drops, whole thing just like a slideshow.
The town I just had a bad experience with was Diathma, now that I think about it, I started this playthrough on the Khuzait side, already took several Khuzait towns/castles, most of those didn't really have bad performance issues, only tiny lags here and there.
But yeah, 2700X could use an upgrade, might just build a new rig next year. But even assuming the CPU is bottlenecking, I can't see how it would cause the FPS go from 60ish to 15ish, likely the pathing issue you mentioned, thanks for the heads up.