...yeah I actually agree with you on this one. I liked the versatility of being able to drop in and out of 1-2 hands during a fight.
On the other hand, I do support the playability approach TW went with here. I miss the functionality as you do but I think it was the right call for the game to take it out, it had a tendency to confuse more than it solved.z
@flashn00b 's right about the controls being the ultimate problem, Bannerlord does not remember player inputs that were not valid during an action. (Same problem with the weird weapon-switching delay that was put in a few updates ago).
Many games "store" an input like this and perform it when the character is able to (after completing the current state).
If we got that, I think we would all fully support putting the X-toggle back in for swords.