Hirsch was born in the Bronx, New York, to Sally (née Kitzis) (1903–1999) and Joseph Sidney Hirsch (1904–1989), an electrician. Joseph was born in New York, to a German Jewish father, Benjamin Hirsch, and an English-born mother, Rosa Hirsch Benjamin, whose family were Dutch Jews. Sally Hirsch was born in Russia, also to a Jewish family. Judd Hirsch has a brother named Roland.
He grew up in both Brooklyn and the Bronx and graduated from DeWitt Clinton High School (in the Bronx) in 1952. He earned a degree in physics from City College of New York.
After graduating from college, Hirsch served in the United States Army Reserve in 1958 at Fort Leonard Wood for six months as a surveyor. Next Hirsch worked as an engineer for Westinghouse before he found work in the theater. He studied acting at HB Studio in New York City. He graduated in 1962 from the American Academy of Dramatic Arts in New York City.
