Carlen died in Columbia, according to Dale Morton at Dunbar Funeral Home. He did not know the cause of death.
Carlen was 107-69-6 in his 16-year coaching career and had just three losing seasons. He led his teams to eight bowl games.
Carlen was a punter and linebacker for Georgia Tech and was an assistant for the Yellow Jackets before he got his first head coaching job in West Virginia in 1966. He's credited with bringing West Virginia football to the big stage, convincing the school's leaders to leave the Southern Conference and become an independent.
The Mountaineers went 25-13-3 in Carlen's four years, including their second 10-win season in the program's history in 1969 that ended with a 14-3 win over South Carolina in the Peach Bowl.
"Jim was very instrumental in the overall growth of our football program, and even after he left, he still remained interested in the Mountaineers. Our thoughts and prayers are with the Carlen family and friends," West Virginia athletic director Oliver Luck said in a prepared statement.
Carlen went on to Texas Tech, where he went 37-20-2 in five seasons, including the second ever 11-win season for the Red Raiders in 1973. He was named Southwest Conference Coach of the Year twice and led Texas Tech to four bowls.
South Carolina hired Carlen to be both football coach and athletic director in 1975. He went 46-36-1 in seven seasons and ranked second in Gamecocks history in coaching wins until he was passed by Steve Spurrier last season.
Former running back George Rogers said Carlen was like a father to him and said his coach deserved much of the credit for his 1980 Heisman Trophy.
"If we didn't have him, there probably wouldn't have been a Heisman Trophy," Rogers told The State newspaper of Columbia on Sunday.
The Gamecocks had been to just two bowl games since the program started before Carlen arrived. He took them to three bowls. His 1981 team beat No. 3 North Carolina, the highest ranked team ever defeated by South Carolina before the Gamecocks knocked off No. 1 Alabama in 2010.
Carlen was born in Cookeville, Tenn., and served for four years in the U.S. Air Force, rising to lieutenant.
A memorial service for Carlen will be at 4 p.m. Friday at Trenholm Road United Methodist Church in Columbia.