Great swordsmen are just like all those 'undefeated' martial artists. If you are undefeated, ever, it means that you are not fighting enough people. In order to really improve, you need to challenge yourself.
And as someone who used to kick-boxing semi-professionally, I can tell you one thing: there are people who are really, really good, but they are not super-humanly good. I've never been anything but OK. I made it to the hand-to-hand finals of the Bulgarian Army in the 80s, I have sparred with the (then) kick-boxing champion of Europe, and with two of the US Taekwondo team in Seoul (years later). I could hit them, I knocked one the latter down once, and on a bad day of theirs, I might have been able to win a match. Once again, I was never anything special - I had a 11-1-3 record when I stopped, but I was semi-pro, and was very careful to pick my fights so as not to endanger my health too much. I once won a match against someone who was damn better than me, and who had me scared in the ring because I miscalculated, and got myself really disoriented - I spun around while desperately keeping my elbow up... he stepped into it, and got up at the count of twelve, i.e. too late. And in fencing, it is even easier to lose a point than it is to get knocked down or out in martial arts, because of a stupid mistake.
No one is as good as the legends make them sound. There are people who have made real advances in fighting techniques, and there are people who have organized what was intuitively practiced into very good 'systems'. But if they were undefeated, it means that they did not fight all that much, not against everyone who showed up, and that they did not allow people to observe them, let alone teach people. In the 80s, in the ЦСКА (Central Sport Club of the Army) kick-boxing gym in Bulgaria, we were visited by one of the first students of the founder of Jiujitsu. He was amazing - he handled most of us trivially. One of us, not I, and not the best of us, knocked him down in his match, though. I wish I could remember the name.
It is very, very easy to close the distance to whoever is training you if you are healthy and dedicated. You may never reach him, but that would be because he is not standing in place, either. Musashi might have been a genius swordsman, I can believe that. He might have been able to defeat anyone unfamiliar with his skills the first time. But he would not have been invincible, and there is no way he was accepting all challengers if he never lost a fight. I've read his books in different translations, and he sounds very human - sometimes he is insightful, sometimes he is wrong, and often he has the brains to realize it, and the guts to admit it later.
But if I had to bet on a fight between him, with his weapon of choice, against an Olympic level fencer with a rapier, my money goes to the guy with the rapier.