I find that the more territory your have the more unwieldy your army gets. You might want to attack a faction at your far flung border but some noble will call a feast on the other side of the map. Makes holding ground harder as your empire gets picked off from all sides by opportunistic factions.
Its not like warband is a race. You don't get anything from winning in the shortest space of time.I like to take my time, I find that the first 100-200 days i'm just rping my character, getting into the groove with the way he thinks and stuff. Then I do a few merc contracts, develop my companions.