I keep a completely separate army of footmen, don't waste mounted troops on sieges. (2 armies are very expensive though
)
I have an army of 75, half Swadian sergeants and half sharpshooters for sieges.
Castles that require a siege tower are a pain and will almost always mean many casualties. I avoid them unless there are other lords siege with me. (Which is no problem because I became marshal recently)
Taking castles with ladders is normally very doable alone with few deaths.
At the start of the battle I let my sergeants rip apart any enemies on the ground outside the castle and have them hold their ground below the ladder. They do a very good job at attracting all the arrows from the enemy, and very few if any get killed/knocked out.
Tell your archers to follow you and have them hold position at the best angle for lobbing arrows at the visible enemies on the castle. I use a crossbow here too.
If your archers run out of arrows and there are still too many enemy troops simply retreat and siege it again this way. Normally I would lose around 5 troops and they will lose 20-40 men. Once the enemy numbers are low simply tell your infantry to charge.