Before you get a Town or Castle, there really isn't much reason to have non-cavalry, you are right. When you start taking part in faction warfare and sieges those cavalry are a liability, but as a newer player your army will probably be an afterthought to most of the sieges you will take part in so it is mostly irrelevant. I just recently abandoned foot armies in the early part of the game in my playstyle, which is unfortunate 'cause I liked all Nord armies. One of the reasons I look forward to the first Town or Castle in my recent game is so I can diversify my army.
Cav armies are not a pancea, but in the early game they are infinitely superior.
Between escorting caravans across the map, and rescuing princesses on the other side of the map, I threw in the towel. Cav Whore 4eva.
Of course when you can store an army you can have whatever you need either way.