You cannot so far. No banner - no claim, nobody sees you as a pretender. Somebody else is the bannerlord now
You can load the aforementioned mod, wait for the contract to run out (max 30 days) then stop including the mod. Very convenient, used it myself
My list of wanderers in the encyclopedia is currently polluted to the brim with those damsels... and some family feud guys (they stay in the villages too). Useless, and most probably a bug