I'd say just about any place becomes twice as nasty in the late game with all the high level defenders they have. I'm just going to list 1 place in each faction that I think is the hardest to siege in the late game stages.
Swadia: Reindi Castle because of the side tower super far away that has insanely easy potshots on a siege ladder. Also, AI seems to have troubles climbing said ladder so my troops get stuck halfway up for some reason. Anyone who does make it up gets stabbed by great-swords or smashed by morning-stars. The sharpshooters tend to coalesce into a large clump behind the little building in the courtyard and as soon as they see you about 40 bolts come flying into your face.
Vaegirs Rivacheg because its a siege tower spawn, and there is a tower to the far side close to spawn where archers shoot unsuspecting AI, and on the other side, there is a covered ramparts section where marksmen blend into the wood like the trees in Nam. The bardiches of the guards and knights gut anyone trying to push in and the archers pummel you from the sides relentlessly.
Khergits Tulbuk Castle because it is a long flat wall where horse archers can shoot pretty much all down the line. Khergits are the least difficutl to siege but this one always takes a while. One is stuck with javelins from seemingly invisible throwers as the ramparts are perfect size for hiding behind.
Nords Hrus Castle in my humble opinion is the most difficult siege late game. Again it is a siege tower spawn, and everyone spawns below a ledge so you constantly have to order archers to move so they can actually shoot. Nord Veteran Archers have easy potshots from many sides on the tower which makes the moving process take forever. And when you finally climb up to the castle wall, the Huscarls decide to yeet all there axes at you while your blocking you sides from arrows and then you watch helplessly as your footmen get chopped apart by great axes. Rough times to say the least.
Rhodoks Grunwalder Castle is the obvious choice as it has the most difficult approach of any spawn. The sheer height of the towers makes shooting the crossbowman a very arduous task. The sergeants give up no room once you approach the top of the ladder and usually I get shot by crossbowmen as I try to fight my way in. Also, that annoying lower ledge to the right always gets me as I forget enemies spawn on that darned thing.
Sarranids Durrin Castle just makes me angry. You literally have to walk around the whole freaking castle just to get to a ladder on the other side while master archers pour arrows upon you and your mates the entire trip. If you can beat the giant mace spam at the mouth of the attack, you will likely find yourself shot at from at least 3 different locations by archers either in the left tower overlooking the long walk, the high walled area to the right, or the giant mosh pit that develops in the middle.
Now if I am defending, pretty much any place that has a ladder that leads to a small opening where you can spam a two-hander is ideal.