Have insane surgery, best skill in the game.
Train a lot of Sharpshooters - Rhodoks or Swadian, Rhodoks better. No archers. Sharpshooters are better foot soldiers when they are left with no arrows.
Put them on the top of the queue in your party.
Train Huscarls and put them after the sharpshooters.
Start the siege! Ladders are the best, because they take the least time, and you'll need several attempts.
When the siege starts, tell the sharpshooters not to shoot - they raise their shields. Position them so that they have the best view towards the enemy archers/crossbowmen. Give the sharpshooters green light to fire. Sharpshooters are the best weapon against castles, they will kill any enemy archers very quickly. Enemy castles that have crossbowmen/sharpshooters will have worse ratio of trained sharpshooters compared to you, so you should be able to win against them as well. If they have 100 sharpshooters - then you are f**ked. You'll need several battles/retreats/retrains of your troops until you soften the castle.
So when the enemy archers are dispatched, you'll want to tell your archers to stop shooting (they will only run out of arrows shooting on shields). you can order an assault. Your sharpshooters are no match for huscarls, or really good infantry, but they ARE very good sword&shield infantry, and they'll do damage and then drop unconscious. If the next enemy army batch arrives, you'll want to tell your sharpshooters to get back in front of the walls and fire arrows against the new enemy archers (and if they have arrows). If not, let them attack until your next batch of sharpshooters arrives and THEN tell them to hold position in front of the walls and fire arrows.
When the castle is left only with infantry, you'll want to retrieve and put the huscarls in front mixed with archers/sharpshooters. When the huscarls attack, the enemy will drop the shields and raise the weapons - leaving them vulnerable for the archers, which should ALWAYS stay in front of the wall.
Several attempts of those can bring any castle to its knees, mostly because of the surgery exploit - you'll get back your troops on their feet, but they won't. So even if the first attempts fail and you lose some troops, they'd have lost more. Most importantly - eliminate their sharpshooters. When order the assault, remember - you can always order them to go back to their positions in front of the walls (where is your place). You can also shoot yourself, I've made lots of killings with a crossbow, but it is a risk, and not a necessary one. If you die, you lose a lot usually.
Make some of the companions additional surgeons, as a backup if you die.
That makes taking a castle a slow, but easy venture.