Olmos was born and raised in East Los Angeles, California, the son of [Mary] Eleanor (née Huizar Flores Magán) and Pedro Olmos Escamilla, who was a welder and mail carrier. His mother was born in Los Angeles of Tejano ancestry, who gave birth to Olmos' older brother in Mexico City out of wedlock at the age of 17 in 1944. There she married Olmos' father, Pedro, 2 months later. In 1945, Eleanor returned with Olmos' brother to Los Angeles. Pedro, who was Mexican-born, followed her by train to America, the day WWII ended. Pedro was naturalized as a US citizen in 1953.
Olmos was raised by his American Baptist maternal great-grandparents as his parents worked. He describes his mother as "feisty," and throughout her life performed as a taxi-driver, sheriff, electrical engineer, and nurse (the first woman to walk into the LA General AIDS ward). His great-grandfather was a church custodian. Around the age of seven, his parents divorced. Following the divorce, he primarily lived with his mother and great-grandparents, seeing his father every other Sunday for 8 hours and during the summer for 2 weeks. His father attempted to raise him and his siblings under the Catholic faith and introduced him to cinema at the Egyptian, Chinese Theatre, El Capitan, and Paramount after church services. During the summer, his father would introduce him to dance and television around the age of 9 or 10.
Olmos describes his community growing up as a "salad" of many separate ethnicities, rather than a melting pot. In the midst of early LA gangs, he focused on learning to play baseball as an exercise in discipline. Having won the California state batting championship 2 years in a row, he was collected by the Los Angeles Dodgers' farm system, as a catcher, at age 13. He left baseball at age 15 to join a rock and roll band, which caused a rift with his father, who was hurt by the decision.
He graduated from Montebello High School in 1964. While there, he lost a race for Student Body President to future California Democratic Party Chair Art Torres. In his teen years, he was the lead singer for a psychedelic/hard rock band he named Pacific Ocean, so called because it was to be "the biggest thing on the West Coast". At the same time, he attended classes at East Los Angeles College, including courses in acting, though says he had difficulty due to undiagnosed dyslexia.
For several years, Pacific Ocean (later renamed Eddie James and The Pacific Ocean) performed at various clubs in and around Los Angeles, and released the album Purgatory via VMC Records in late 1968. The album was promoted with two singles, "I Can't Stand It" / "I Wanna Testify", and "My Shrink"/"16 Tons," and was followed by a nation-wide tour in early 1969. While setting up for a Pacific Ocean show one night, Olmos slipped on stage and landed on a nail that went through his knee. Another time during a concert, Olmos jumped from the top of an organ across the stage into the drum set, getting knocked unconscious and dislocating the drummer's shoulder.
