My brother also didn't like Native's banners. He used to think like you: the lion was the general symbol for Swadian Lords as well as the horses (due to their heavy knights); while the two-headed eagles and some stags belonged to the Vaegirs because said faction was based off of the Rus'; and that castle-like, bolts were for Rhodoks only; etc. He once spent an entire day re-arranging the banners -according to what each of them represented- instead of actually playing - it was hilarious.
He once convinced me too that certain banners had to belong to certain Lords for displaying certain objects. But when I began playing and I read about the Old Calradian Empire and its story through the several hints given in the game, I reached the conclusion that it didn't really matter what a Lord's banner displayed, if either an eagle or a lion, because (from a roleplaying point-of-view), maybe said Lord was given said banner for a reason.
As example, perhaps the lion was the Calradian Empire's most iconic symbol, with several Lords of different lands (Swadians, Vegirs) having them in their banner because it was granted due to a remarkble noble action or what-not. These banners would be a legacy through generations among that certain family, where every member had a "variation" of it - i.g, Lord Montewar(?)'s father was given a golden lion on azure because of a noble deed, and his son inherited said symbol, though now on argent, because it's something valuable for them.
And then when the Empire fell and split into five (six, in WB) different factions some Lords left or changed factions, which lead to their banners to "mix" as time went on and generations inherited a huge variation of different banners with different symbols. And thus, when the game begins, we see a Swadian and Vaegir (and perhaps a Rhodok, too) Lords displaying lions or knights on their banners because of that.
