"He's some kind of pitcher," Orioles manager Buck Showalter said. "We were fortunate to catch him on a day when he wasn't as sharp as he normally is."
Jonathan Schoop homered off Tanaka for the second time and fellow rookie Caleb Joseph capped the scoring with his first career homer. J.J. Hardy hit a three-run double for the Orioles, who spoiled Old-Timers' Day at Yankee Stadium and took two of three from their AL East rivals.
After squandering a ninth-inning lead in Friday night's loss, Baltimore outhomered the Yankees 6-1 and outscored them 14-1 in the final two games of the series.
"What does it say about this team? That we have a short memory," Hardy said.
New York slugger Mark Teixeira left in the eighth after getting hit on the toes of his left foot by an 85 mph slider from T.J. McFarland. Teixeira hobbled toward first, then slammed his helmet to the ground in anger before heading into the dugout.
X-rays were negative, manager Joe Girardi said.
"I thought it was broken again. I took it out on my helmet," Teixeira said. "Just breathing a sigh of relief right now."
Tillman (6-4) snapped a two-start skid, though he pitched well in those outings, and improved to 6-0 on the road -- though he entered with a 6.33 ERA away from home.
"I made some adjustments during my side sessions and I was able to carry them out to the mound with me today," Tillman said.
Tanaka (11-2) allowed three runs in seven innings, giving him a quality start all 15 times out this season. But the Yankees provided no support for the Japanese rookie, who entered with the most wins in the majors and the best ERA (1.99) in the American League.
His only other defeat came on May 20 against the Chicago Cubs at Wrigley Field. Then he went 5-0 with a 1.26 ERA in his next five starts before Sunday.
Hardy hit a bases-loaded double in a four-run eighth against Adam Warren after a disputed call went against the Yankees.
As he tried for a double play, third baseman Kelly Johnson was taken out by Steve Pearce on what appeared to be an illegal slide too far from the bag. Girardi called it a "pretty malicious" and "dangerous" play, but Pearce said he wasn't trying to hurt Johnson.
Manny Machado chopped an RBI single, and Joseph homered in the ninth.
Joseph said he was excited that his first home run came at Yankee Stadium, and he ran up the tunnel and hid for a while to turn the tables on his teammates who were planning the silent treatment.
The 28-year-old catcher got the souvenir ball back, and said he was told it was Derek Jeter who went to the ball boy and made sure of it.
"He might have been the only one on the field that knew it was my first major league home run," Joseph said. "I'm really thankful that he did that."
Schoop homered to left on a 2-2 pitch in the second, making the 22-year-old infielder from Curacao the only big league player to go deep twice against Tanaka. Schoop also connected for a three-run shot at Yankee Stadium on April 9, when Tanaka ended up with a no-decision in Baltimore's 5-4 victory.
"I saw the ball pretty well. It was up," Schoop said. "I think he missed a little bit."
Hardy singled to start the seventh and went to third on Machado's double. Schoop had an RBI groundout, and Joseph hit a long sacrifice fly after fouling off a suicide squeeze attempt.
"I feel like that I kind of let the game go there," Tanaka said through a translator.
Before the game, Hall of Fame reliever Goose Gossage was honored with a plaque to be put in Monument Park -- part of festivities for the 68th annual Old-Timers' Day at Yankee Stadium. Yogi Berra and Whitey Ford, both in their 80s, waved from a golf cart and received standing ovations. ... The recorded voice of late public address announcer Bob Sheppard welcomed fans to Old-Timers' Day. Showalter, who managed the Yankees from 1992-95, was introduced to a warm ovation. He popped out of the dugout in full Orioles gear and waved to the crowd before hugging former GM Gene Michael and other ex-Yankees along the third base line. ... Baltimore signed veteran LHP Randy Wolf to a minor-league contract. Wolf was cut by Miami last week.