"I didn't have any of my good stuff," said Kelly, who threw 109 pitches. "From pitch one to the last pitch nothing was easy."
Kelly (5-3) allowed two runs and seven hits while improving to 5-0 with a 1.79 ERA in his last seven starts. St. Louis improved to 7-1 in Kelly's eight starts since he rejoined the rotation on July 6.
"We needed a solid outing from him and he was very good," St. Louis manager Mike Matheny said. "He did a nice job for us."
The NL East-leading Braves played without Jason Heyward, who is out with a broken right jaw after he was beaned during Wednesday's 4-1 win at the New York Mets. The outfielder had surgery on Thursday in Atlanta and could miss the rest of the regular season.
The Braves had at least one batter reach in each of the first five innings, but couldn't come up with a big blow against Kelly, who struck out three and walked two.
"He did a good job of keeping us off balance," Atlanta outfielder Justin Upton said. "He threw his fastball in the zone and got the job done."
Kelly may not have been pleased with his effort, but his teammates certainly were impressed.
"He's like having a Ferrari in your garage," second baseman Matt Carpenter said. "He's settling in as a starter and he's running with it. I love it when he goes out there and competes. He's aggressive and he's pitching with confidence."
Upton hit his 24th homer, but the Braves lost for the second time in three games at the start of a six-game trip. Atlanta had won eight of its last nine regular-season games against St. Louis.
St. Louis slugger Matt Holliday went 2 for 3 and drove in a run, matching the amount of hits he had during a 2-for-24 slide over his previous six games. He doubled and scored in the fourth, then capped the Cardinals' three-run fifth with an RBI double that made it 4-2.
David Freese had a run-scoring groundout in the fourth for St. Louis, and then singled in a run in the seventh.
Paul Maholm (9-10) came off the disabled list and allowed five runs, four earned, and five hits in 5 1-3 innings for Atlanta. It was the first game for the left-hander since he left a July 20 start at the Chicago White Sox with a bruised left wrist.
"It was good to get back out there and compete against a tough lineup," Maholm said. "Hopefully in five days I'll be able to make some adjustments and get back out there."
Maholm helped himself with a leadoff single in the fifth. Upton then hit a one-out drive to left-center for his eighth homer this month, giving Atlanta a 2-1 lead.
"Getting hits like that in a row, it doesn't happen a lot," Beltran said. "We're concentrating more with guys in scoring position."
The Cardinals began a string of 17 consecutive games against opponents with winning records. ... St. Louis RHP Jake Westbrook is questionable for his next start Monday with back stiffness. ... INF Elliot Johnson made his first start for the Braves and broke out of an 0-for-31 slump with a second-inning single. He was claimed off waivers from Kansas City on Wednesday. ... Atlanta's Kris Medlen (10-11, 3.71 ERA) will face Adam Wainwright (14-7, 2.66 ERA) in the second game of the four-game set on Friday. ... Holliday grounded into his major league-leading 28th double play in the seventh.