Castor started as a doo-wop singer in New York. He wrote and recorded "I Promise to Remember" in 1956. Castor then replaced Frankie Lymon in The Teenagers in 1957 before switching to the saxophone in 1960. He had a solo hit with "Hey Leroy, Your Mama's Callin' You" on Smash Records in 1966. Castor also played sax on Dave "Baby" Cortez's hit "Rinky Dink." He formed the Jimmy Castor Bunch in 1972 and signed with RCA. As leader of The Jimmy Castor Bunch in the 1970s, and also as a solo artist, he has released several successful albumsand singles. The group reached the peak of their commercial success in 1972 with the release of their album, It's Just Begun, which featured two hit singles: the title track and "Troglodyte (Cave Man)," which was a large hit in the U.S., peaking at #6 in the Billboard Hot 100. The track stayed in the chart for 14 weeks and was a million seller by 30 June 1972, and received agold disc award from the R.I.A.A.
Castor continued the trend in 1975 with "The Bertha Butt Boogie" and later recorded "E-Man Boogie," "King Kong," "Bom Bom," and "Potential." The Castor band includedkeyboardist/trumpeter Gerry Thomas, bassist Doug Gibson, guitarist Harry Jensen, conga player Lenny Fridle, Jr., and drummer Bobby Manigault. Thomas, who simultaneously recorded with the Fatback Band, left in the '80s to exclusively record with them. Castor recorded as a solo performer from 1976 until 1988. He had one of his bigger hits in many years with a 1988 revival of "Love Makes a Woman," which paired him with disco diva Joyce Sims. Castor had his own record label, Long Distance, in the 1980s.
Many of the group's tunes have been heavily sampled in films and in hip-hop. In particular, the saxophone hook and groove from "It's Just Begun" and the spoken word intro and groove from "Troglodyte" (namely, "What we're gonna do right here is go back...") have been sampled extensively.
He died in 2012 from heart failure.