It's depending, if catapults, rames, it's take a mouth due they build it, unless they had gunpower, cannon, they can bring it and can do matter of in day.
I read somewhere, larger city, longer it's will be, need huge attacker to do that last many weeks to do so, even so they might take over, but hold city is other matter, unless they trying to kill all people, then no point to hold as it's dead city, but people in living city won't be loyal to conquering unless their ruler is bad and one who conquering do care of people in city and improve their life, not seek to take from them but to gave them better life (that is very rare, most conquering would be jerk and selffish even cruel, they just take and take, not seek improve life of town for most part and cause suffering)
It's all depend on size of city/castle design, and how good defender and attacker are, number, equipment, morale, weather at least.
But in all, I had not try solo myself as attacker and defender yet, but I do join army that are attacker and petty easier, so I'm little worried that defender might be harder to do so, so I don't know. Defender should be easier than attackers unless attacker had huge number while defender had small number.
In normal battle, if small party see huge number of attacker, small party simple got away easy, fast, hide, so large number unable to get it, but of course town/castle/land can't move or got out of way or run away, end up got stuck and get attacked.
I'm not sure what high successful and failed sieges castle in real life history? I assume that castle is way useful till 16th century 1501 ad? True cannons come up in late 15 Century? I guess everyone see that castle is useless at that time, it's was useful, now it's not useful, but oddly american civil war, even lot of cannon, they still use "fort" in American land during civil war, I guess it's might be useful, Castle vs Fort?
Ah, I see, Castle vs Fort similar yet difference. Someone post this "Castles are fortified, full time residences for lords or monarchs. It's also an administrative center. Forts are simply garrisons of troops placed in strategic locations."
I guess Fort is still useful even vs cannon, it's just no longer ruler live in fort, I guess it's costy and better off flee then hide, from there control from shadows, more or less.
So castle might not useless, but it's not very cost efficient and draw unwanted attention from enemies, I mean too many castle here and there, but enemies take to fight in full power when they know where ruler live rather than fighting many castle here and there.
Once castle no longer called, but fort, meaning ruler hide somewhere, then enemies unable to take full power to fight one single place, and too many fort here and there that very costly going on.
just theory
I guess it's should be harder to take over castle unless huge number or full force of number or something like that, common army shouldn't take castle that easy more or less while it's should be easier to guard castle from attacker unless attacker is united many lords, and huge number.