i never break down walls so i guess we found the reason. although because of force of habit i can't tell for sure if my guys do attack the second gate by themselves.
i generally form 1 or 2 group of infantrymen and let them wall for my archers then i manually tell them to attack the gate, after the Ram has done it's job. but if i have to guess, the broken walls may be the culprit