I mean, magic is itself kinda overpowered, chances are if you got some healthy HP, magic defense and magic power, you can take small armies yourself and larger armies with only a few similarly equipped companions, so it seems like something of a futile effort if the objective is to limit player power, even with the negative magic mods from armor.
It doesn't lose me too much time, to restart now, but I feel like this was an inconvenience that didn't really need to happen, and didn't used to happen. Perhaps a function to allow players to change their secondary class post-creation via some sort of guild could be good, if this sort of magic limitation is going to be maintained, so that people can try out different playstyles without having to lose hours of progression.