New York scored twice off Sean Marshall in the ninth, but Baker pulled his regular closer in favor of Arredondo and he got the job done.
"I could have done without the drama, but we did win the game," Baker said.
Arredondo entered with two on and one out, and Derek Jeter grounded his first pitch to third. Cincinnati nearly turned a game-ending double play, but Jeter just beat the relay. Even Votto thought the Reds had it won - he clenched his fist when he caught the ball at first base, hoping the game was over.
Curtis Granderson then fouled off a 3-0 pitch before topping a slow roller to Votto for the final out.
|More on Reds-Yankees|
|More MLB coverage|
"It wasn't fun the way it ended," Votto said. "Well, I take that back. It was fun the way it ended. Just a little bit nerve-racking at the end, but that's part of wins and losses and it was nice for Jose to come in and pick us up."
Brandon Phillips got Cincinnati started with a run-scoring single and the Reds overcame Ivan Nova's career-high 12 strikeouts to improve to 2-2 during their five-game visit to both New York ballparks this week.
Homer Bailey (2-3) pitched effectively into the seventh inning for the second consecutive outing. Cincinnati then turned it over to its normally lights-out bullpen and held on to hand New York its fourth loss in five games.
"We just didn't get the last hit that we needed," Yankees manager Joe Girardi said. "It's good that we're able to come back, but it's frustrating that you get so close and lose."
Logan Ondrusek retired Jeter on a shallow fly with two on to end the seventh. Cuban flame-thrower Aroldis Chapman, hitting 99 mph on the stadium scoreboard, whiffed Granderson and Robinson Cano in a perfect eighth.
Raul Ibanez doubled to start the ninth and scored on Nick Swisher's single, snapping New York's 0-for-22 skid with runners in scoring position. Marshall threw a called third strike past Russell Martin, but Andruw Jones singled to put runners at the corners and Jayson Nix singled to make it 6-5.
That's when Baker went to Arredondo.
"I didn't really like the matchup there at the end with all those righties coming up there against Marshall," said Baker, who hinted that he might consider moving Chapman into the closer role soon.
Martin and Nix homered for the Yankees on a beautiful, 74-degree day when three balls barely cleared the short porch in right field.
Bailey, drafted seventh overall in 2004, was 0-1 in his previous four starts since beating the Cubs on April 20 for the franchise's 10,000th victory. The 26-year-old right-hander struck out seven and walked one in his first outing at Yankee Stadium.
"Overall, pitched a good game," he said. "I'd be lying if I said the anticipation wasn't there. The first couple pitches it looked like the plate was about 100 feet away."
With the score tied 2-all, No. 9 batter Wilson Valdez singled leading off the Cincinnati fifth and Chris Heisey followed with a bunt single. One out later, Votto got just enough of an inner-half slider from Nova (4-2) to drive it over the auxiliary scoreboard in right-center.
Five of Votto's seven homers this season have come in the last seven games, including a three-homer performance Sunday against Washington that he capped with a game-ending grand slam.
"I'm starting to make progress," the 2010 NL MVP said.
Nova had no problem with the bruised and sprained right foot that knocked him out of Monday night's start at Baltimore in the sixth inning. Flashing a sharp curve, he struck out eight in the first three innings and had 11 strikeouts after five, surpassing his previous career high of 10.
The right-hander got himself in trouble with consecutive walks in the first and Phillips put the Reds ahead with an RBI single. Devin Mesoraco and Heisey both doubled in the second to make it 2-0.
It was an early breakthrough for the Reds, silenced by 39-year-old Andy Pettitte in their first shutout loss of the season Friday night.
Martin homered on the first pitch of the third and New York tied it on Ibanez's two-out RBI double in the fourth. Nova allowed seven hits and two walks over six innings in his second loss since a 15-game winning streak.
"I don't care about strikeouts," Nova said. "I don't like the way I'm pitching right now. I'm making a lot of mistakes."
- The series finale Sunday features an excellent pitching matchup. Reds RHP Johnny Cueto (4-1, 1.89 ERA) faces Yankees ace CC Sabathia (5-1, 3.77). Cueto's 2.20 ERA over the past two seasons is the best in the majors.
- Cincinnati rookie SS Zack Cozart, with one hit in his last 26 at-bats, was not in the lineup. Valdez started in his place and moved to third when Cozart entered for defense in the eighth.
- Costanzo got his first major-league start (at DH) and singled in the sixth for his first career hit.