That happens when you're outnumbered and don't have many points in the tactics skill. That "battle advantage" isn't some arbitrary number tacked on to show the force balance, it affects how many troops spawn on either side. Which means that the outnumbered side spawns with fewer troops.
Also, I'm guessing you've set battle size pretty low. You could try increasing that if your system can handle it, and you'd get more men to the fight. Of course this would mean more enemies spawn as well, but if they are a mostly-rabble army and you've got high-end units it would be in your favor.
Also, blindly charging when outnumbered might seem a very glorious thing to do, but you'll only get yourself killed. If you've got mostly footmen and archers, just set up on a hill, as those are easy to deffend. If you've got horsemen, have them follow you and circle around your enemy, picking off anyone foolish enough to break formation.