Woo was born as Wu Yu-seng (Ng Yu-sum in Cantonese) on 1 May 1946, in Guangzhou, China, amidst the chaotic Chinese Civil War. Due to school age restrictions, his mother changed his birth date to 22 September 1948, which is what remains on his passport. The Woo family, who were Protestant Christians, faced persecution during Mao Zedong's early anti-bourgeois purges after the communist revolution in China, and fled to Hong Kong when he was five.
Impoverished, the Woo family lived in the slums at Shek Kip Mei. His father was a teacher, though rendered unable to work by tuberculosis, and his mother was a manual laborer on construction sites. The family was rendered homeless by the Shek Kip Mei Fire of 1953. Charitable donations from disaster relief efforts enabled the family to relocate; however, violent crime had by then become commonplace in Hong Kong housing projects. At age three he was diagnosed with a serious medical condition. Following surgery on his spine, he was unable to walk correctly until eight years old, and as a result his right leg is shorter than his left leg.
His Christian upbringing shows influences in his films. As a young boy, Woo had wanted to be a Christian minister. He later found a passion for movies influenced by the French New Wave especially Jean-Pierre Melville. Woo has said he was shy and had difficulty speaking, but found making movies a way to explore his feelings and thinking and would "use movies as a language".
Woo found respite in Bob Dylan and in American Westerns. He has stated the final scene of Butch Cassidy and the Sundance Kid made a particular impression on him in his youth: the device of two comrades, each of whom fire pistols from each hand, is a recurrent spectacle later found in his own work.