Not sure if you observed what your hardware does in the time when FPS drops, so gonna drop my observation.
In moment when Ladders hit a walls, Ive noticed that CPU usage on my Ryzen 5 3600 goes to nearly 100% on ALL 12 threads, andGPU usage goes down, so in here Im very happy to agree on Pathfinding/queue and it eat up on CPU like mad. It also does appear with Siege Towers time by time, and Ive noticed soldiers getting stuck on siege tower ladders and ladders too.They also often dont know what to do, they crack a gate with Ram, and proceed to clog a single ladder. rofl.