No way cavalry should get arbitary penalties for siege battles. They just should be way more expensive to maintain, both in gold and food. Cavalry in history was always more or less elite force. Less elite for nomads, but even nomads had heavy cavalry which was elite. And heavily armored cavalry were well trained in melee even if they were primarily using bows. They were formidable opponets even when dismounted - and dismounting cavalry was far more common than people realize. European knights are known for their couched lance charges, yet it was common for them to dismount and instead assault or defend on foot. Depending on era and place, many of them also had bows.
Heavily armored cavalry were generally from richer families and/or nobility. They had more servants and underlings to do work for them, which meant they could spend more time training. And figuratively and literally, even best of horsemen are born with 2 legs and 2 hands.
You should avoid using cavalry in siege battles because they consume more supplies and money and are more rare. But if you do use them, they should be just as good as they are currently.