I think i figured out the problem, i haven't tested it out yet due to other modding concerns, but i believe the problem lies with the various towns and castles scattered around the map, but disabled. I think my villages are taking a disabled castle thats within their radius and making i their parent castle and thus acquiring is faction.