In terms of historical accuracy, yes, sieges are obviously way too quick and far too easy.
In terms of gameplay, I don't know if easy is quite the right word, but they don't make a ton of sense.
I've been finding that siege towers actually increase my casualties, which is strange. They should be helpful, but they take so long to reach the walls that I lose a lot of people to archers and ballistae.
The best approach I've found starts with waiting for enemy lords to clear out, which they usually do pretty quickly. Once it's down to me vs the militia I start the siege, build one catapult (and nothing else), and then attack as soon as it's complete. I put a small group near the catapult, spread everyone else out in loose formations, and rush the walls. My troops near the catapult use it to suppress the enemy siege engines, everybody else hits the ladders, and I use my bow to pick defending archers off the walls. As soon as my guys have a bridgehead (so to speak) I head up the ladder and do my best to flank the defenders.
If I'm careful and don't get myself taken out I can usually take a town with single digit fatalities, even if I'm outnumbered to start. That doesn't seem right, but I'm not sure that gameplay would be improved by making me lose more troops, since refilling companion parties is such a clicky pain in the ass.
So yeah, I dunno. Maybe.