Probably the ones marked as MP only, like most of the heavy armors, crossbows and other high tier stuff
There is a thread saying that the wanderers are not companions like in warband it a new mechanic and that companions, the ones were you can customize from low Lvl, will probably be added later
Even if you have a bigger party it can increase, my party of 20 was fighting with 11 robbers and increased the same way