I ignored it but it's actually pretty immersion-breaking to know that your siblings have been kidnapped by a brigand and are nobody-knows-where and you're just like... Meh, I'll just do my adventures and not think about that anymore, especially if you're playing a honorable and brave knight who would litterally travel the entire world and face an army by himself to save his family if he had the option to.

I can ignore the quest but I just can't forget it magically, unfortunately

Wouldn't it be pretty easy to just give the choice to follow this main quest or not right at the end of the character creation, leaving you play a "blank state" character if you want to ?