Duryea was born and raised in White Plains, New York. He graduated from White Plains High School in 1924 and Cornell University in 1928. While at Cornell, Duryea was elected into the Sphinx Head Society, Cornell's oldest senior honor society. He majored in English, and in his senior year succeeded Franchot Tone as president of the college drama society.
As his parents did not approve of his choice to pursue an acting career, Duryea became an advertising executive. After six stress-filled years, he had a heart attack that sidelined him for a year.
