As I understand it:
1st Son after Father's Father
2nd Son after Mother's Father
3rd Son after Father
4th Son after Fathers eldest Brother
5th Son after Mothers eldest Brother
etc
Daughters similarly
The first three are pretty reliable, sometimes the grandparents were brought in as well.
You can see the pattern in our family
Things get confused when Father, Grandfather and Father in Law have the same name as happened in our family.
My GGrandfather Joseph was a ring-in and I don't know where his name came from. There were only three boys but the other two were named Matthew and John and there were already a surfeit of John and Matthew ancestors. I can only presume that he was named after a brother of his Mother. There were no previous Josephs in our family but there were 13 children and I was told by my Mother that there were seven Josephs alive at one time. My Grandfather was the youngest of the 13 and named Thomas even thought there had been no Thomas' for generations. He had a cousin Tom who was appropriately named after John's Father in Law Thomas Smith. I presume that is where the name came from. The third son bacame Tom also and two of the grandchildren. Another eldest son named Tom died in infancy. So we only made 4 Tom's at once.
Things also tended to get mixed up a bit if there were early childhood deaths. Sometimes the child died and the name was lost. Sometimes a later child was given the name to preserve it and it became "out of order". I believe that occasionally another child was given the same name as an earlier sickly child expected to die. The sickly child survived and nicknames had to be used.
I came to a new land and our family has not been named traditionally much to my Mother's disgust although, to her credit, she never showed it.

The fourth child in my Grandfthers family was named Matthew and this was the name of his Parternal GGrandFather but also (naturally) the name of his Father's eldest Brother - so the pattern fits.

I don't know how it works with small families or families with a large number of children of the same sex.