Which is the 'etc...' portion, unrelated to the higher tier armor tangent. There's a ton of opportunities/avenues besides just the building army/stronger character aspect - whether that's better tournaments, more varied bandits, challenging quests, remove party# requirement for quests, and so on.
The small scale bandit fights are there already (if you mean in the ~20 or less battles), bandit hideouts offer some challenge the first one or two go arounds; though maybe a more varied mix of bandits would be nice. Tournaments, I honestly don't think they can ever make it challenging (the practice training is harder than tournaments imo).
End of the day, selling point of the game are the battles, all else is the 'build-up' to said battles (large or small). As it is, the 'build-up' is seriously lacking any impact which is why the battles all feel very grindy.