Baker was born in Ferndale, Glamorgan, Wales, the youngest of three children. His father was a coal miner who lost a leg in a pit accident but continued working as a lift operator at the mine until his death. Baker grew up a self-proclaimed "wild kid", interested in only "football and boxing". He thought he would most likely be a miner or maybe a boxer.
His artistic ability was spotted at an early age by a local teacher, Glynne Morse, who encouraged Baker to act. When he was 14 he was performing in a school play when seen by a casting director from Ealing Studios, who recommended him for a role in Undercover (1943), a war film about the Yugoslav guerrillas in Serbia. He was paid £20 a week, caught the acting bug, and pursued a professional acting career. Six months later Baker appeared with Emlyn Williams in a play in the West End called The Druid's Rest, appearing alongside Richard Burton.
Baker worked for a time as an apprentice electrician, then through Morse's influence, he managed to secure a position with the Birmingham Repertory Theatre in 1944. He was there for three years when he had to do his national service. He served in the Royal Army Service Corps from 1946 until 1948, attaining the rank of sergeant. Following his demobilisation Baker returned to London determined to resume his acting career. He was recommended by Richard Burton for casting in a small role in Terence Rattigan's West End play, Adventure Story (1949).
