You can, and do, level up faster if you decide not to fill those slots. But you can, and will, lose effectiveness in your party trying to level up everything.
Sending companions out on mission will require them to have a myriad of different skills. From what I’ve seen, each skill is covered somewhere as a requirement to having a suitable companion to do the job. And you can start another party with companions who will then need things like leadership, tactics, or trade or what have you.
Smithing is another area where a companion can actually be the one that does all those tasks versus your main character. If you invest in stats other then endurance, this may be a good option.