Yes and they won't come just once but they keep coming with infinite number of troops, and you should know you won't be able to get any good loots out of them. Your fiefs will be constantly attacked to the point that you won't want to defend them anymore. The villagers will be angry and spread rumors about you treating them badly although you have 30 relation points with them, thus getting even more factions declare on you. Finally, you are running out of denars and its so troublesome defending your castle you will abandon all troops left the castle for whoever want it
So the best way to make your own kingdom?
First, stay neutral, get some cavalries and recruit all companions. Farming sea raiders until you are about lvl 30 with 10 leadership, rack up some hundreds thousand denars, get best gears.
Second, make friends with some lords (over 30 relation points at least), you would need to recruit them later on.
Third, helping villagers whenever you can, it is necessary now, cause you won't have time for that later when you are at war.
4th, look at the map, finding a castle that is constantly get attacked by a lot of factions, or get taking over every now and then, and take over it. You know it is a good one with high income fief because the AI is coded so that they would try taking over wherever have high income. If you get lucky you would get a village that give you over 10k denars a week (I got once give me 16k denars a week). You would need to get good relations with people around your starting point so its easy to recruit troops.
5th, with stable income now, you just have to keep stepping your way up.