I started with 12-15 RTR. I had no lords at all.
I took a castle from the Rhoduks, then a city. It was originally a Swadian city, so they ended up going to war with me. After a few beatdowns the Rhoduks asked for peace. After another two cities and a bunch of beatdowns the Swadians asked for peace too.
Several Lords joined me out of nowhere. Now I have about 10, but I'm only "in" with about 5. I've heard that's plenty...
The Rhoduks declared war a little while later. I took a couple castles from them and they asked for peace. (peace usually comes after you decimate their armies on the field or in a castle/city defense)