yea... at full damage, early in the game with low shield, armor and life, you probably wont last 1 wave of javalins.
having a big party early on just slows you down, eats away your money and serves no real purpose. later in the game you can raise a large army in very short periods anyways, so what would be the point.
i imagine its possible to get your party size to 200. 300 might be possible (though i dont think so) but would probably involve you running around spending all your time (for many many many months) getting renown. and again, theres no real point in it. slow movement, expensive and time-consuming.
AI lords on the other hand, have no party size cap. if i need a super big army, i get 1 of my lords who has only villages (hence unable to garrison) and spam him with troops then get him to follow. he will have serious problems keeping up though, but is useful for sieges.
i would suggest 50-60 for non-war roaming parties. 80-100 for a war party. 20-30 when you want speed and to attract bandits.