Rufus Lenoir Patterson was born in Caldwell County, North Carolina on June 22, 1830. Patterson was a wealthy planter, politician, and former enslaver who served as a director of the North Carolina Railroad Company. He died in Salem (Forsyth County) on July 15, 1879.