From what I can make of this census, it seems as if each of the oldest sons lived in houses on either side of their mother and most likely helped her with the farm and the other younger children.  This would have been a couple years before Henry, James, Elijah, and Winfield Taylor joined the Confederate Army.

July 1860 Murray County, GA Census, Holly Creek, 984 District

Series M653, Roll 132, Page 82

Dwelling # 571 Williams, Henry M 30 Farming GA

Louisa F 18 SC

Thomas M 2 GA

Sarah J F 1 GA

Dwelling # 572 Williams, Sarah F 46 Farmer VA

William M 26 Farming GA

Thomas M 23 " GA

Stephen M 18 " GA

Margrett R F 19 " GA

Elijah M 16 " GA

Winfield T M 14 " GA

George W M 12 " GA

Charles I M 10 GA

Alexander M 8 GA

Joseph M 4 GA

Dwelling # 573 Williams, James M 28 Carpenter GA

Cornelia F F 17 SC