...yeah I actually agree with you on this one. I liked the versatility of being able to drop in and out of 1-2 hands during a fight.
On the other hand, I do support the playability approach TW went with here. I miss the functionality as you do but I think it was the right call for the game to take it out, it had a tendency to confuse more than it solved.z
's right about the controls being the ultimate problem, Bannerlord does not remember player inputs that were not valid during an action. (Same problem with the weird weapon-switching delay that was put in a few updates ago).
Many games "store" an input like this and perform it when the character is able to (after completing the current state).
If we got that, I think we would all fully support putting the X-toggle back in for swords.