The main problem is you start out as a relative nobody that's basically broke. You also have other resources that you have to essentially farm to gain stature. Having to cater to the acquisition of these resources, combined with the fact that the rest of the factions 'play themselves and can end the game' create a very ridiculous scenario of you trying to get to a level where you can actually have an impact, but the world has left you in the dust. Sure, in some scenarios, the main factions seem to exist in a relative stalemate, but the overwhelming amount of playthroughs have a snowballing/winning contender. This forces the player to either skill up or do something else while they watch the game end before their eyes or rush renown/influence in order to participate.
As such, due to these circumstances, I don't believe in the 'player should have little agency' argument. If you started out on relative equal footing then perhaps that could hold some weight, but because you don't, you, in contrast to others, have to play an early game while you watch factions eat each other, and by the time you pick a side, you then have to grind up influence or go it on your own, putting you, again, at a severe disadvantage.
This coupled with the relative non-existence of peace time, further makes the argument of 'the world should be able to exist without you and play on without you' also ridiculous.
All in all, it just removes fun from a game that's supposed to be about having fun. Sure, there are certain games where I don't want to be the center of the universe and they market themselves around that fact, and design the game around that fact, but in this game, it should be in a relative stalemate until you either a) begin to influence the world in a noticeable way, or b) progress the story to a sufficient level to unlock a competition to recreate or make your own empire.