Yes, you should have to capture castles all by yourself, because vassals are dumb. You want something done right, do it yourself
In all honesty, whether they take a castle is a bit random. The most important thing is to send more than one lord for every siege. If there's just one guy with a 100 blokes vs. a garrison of 150+, naturally he won't stay there. The other thing is to patrol (yourself) around the castle and make sure there's no random targets or big threats that could throw the besiegers off of their current job. It takes a bit of managing, but I would still suggest that you take fortresses by yourself. Building an army capable of that is not hard, and while your lords patrol the area, you shouldn't be bothered too much by annoying enemies.