Wagner was born in Detroit, to Thelma Hazel Alvera (née Boe; 1898–1993), a former telephone operator, and Robert John Wagner Sr. (1890–1964), a traveling salesman who worked for the Ford Motor Company. He had one older sister, Mary Scott (1926–2023).
Wagner's mother came from La Crosse, Wisconsin. Her parents were both immigrants from Norway, who married in La Crosse in 1887. Wagner's father was a native of Kalamazoo, Michigan. His parents were from Germany.
The family moved to Bel-Air, an upscale area of Los Angeles, in 1937.
