Brown was raised on Lake Lanier by his mother, Bettye, and stepfather Dr. Jody Moses, a dentist in Cumming, Georgia. The eleventh of twelve children, Brown attended Mashburn Elementary School in Cumming; Lakeview Academy in Gainesville, Georgia; and South Forsyth High School in Cumming. He later moved to Dahlonega, Georgia, at age 17, where he graduated from Lumpkin County High School. Brown was given his mother's guitar at age 8, and one of his stepfather's patients was enlisted to teach him classical guitar. He completed two years of lessons, but soon after developed a love for bluegrass music while playing with his father and brother on weekend visits. In his mid-teens, Brown spent almost a year being tutored by a vocal coach from his church
While still in high school, with James Taylor as his inspiration, Brown began playing solo gigs in local venues, performing country and pop cover songs. Brown attended the University of West Georgia, where he became a member of the Zeta Kappa chapter of the Kappa Alpha Order fraternity. He was also a camp counselor at Camp Mikell in Toccoa, Georgia, and Camp Glisson, a United Methodist summer camp and retreat center located in Dahlonega. Brown later founded Camp Southern Ground, a 501c3 non-profit camp in Fayetteville, Georgia.
In 2017, Brown made an investment into the Stillhouse Creek Distillery in Lumpkin County, Georgia, which was later rebranded the Z. Brown Distillery. The distillery closed down on November 18, 2018.