"It was beautiful," Tampa Bay manager Joe Maddon said. "None of it surprises me. I just love his demeanor right now on the mound."
Kazmir (3-2) gave up two runs and five hits with one walk, beating Boston for the second time this season. He is 4-1 with a 2.45 ERA overall against the Red Sox.
Toby Hall and Carl Crawford hit two-run homers for Tampa Bay. Chad Orvella worked the eighth and allowed back-to-back homers to Mike Lowell and Wily Mo Pena in the ninth before he was pulled for Tyler Walker, who walked pinch-hitter Trot Nixon.
"I still have not exhaled," Maddon said.
The Devil Rays took two of three from Boston despite the bullpen allowing four runs in the ninth inning of a 9-6 loss Saturday.
"We've got to make it interesting," Kazmir said with a smile. "It was a great win for us to pull out the series after a tough loss like that."
Schilling (4-1) was looking to become the third Boston pitcher -- joining Pedro Martinez (2000) and Babe Ruth (1917) -- to win five games in April. He allowed three runs, six hits, one walk and struck out nine in six innings. The right-hander started the 2001 season with Arizona by winning his first five decisions.
The Red Sox have lost seven of 10 heading into a two-game series against the New York Yankees that starts Monday at Fenway Park. Boston has dropped three consecutive series for the first time since June 2004.
"The way we're going right now, the starting pitching has to step up and it just didn't happen," Schilling said.
Hall put Tampa Bay ahead 2-0 with a two-run homer in the second.
"That was the game," Schilling said. "I went 0-2 and I rushed the pitch. I knew what I wanted to do. I threw the pitch thinking about how the inning was going to progress instead of being focused on making the pitch. Just a horrific mistake."
The Devil Rays got the run back in the bottom of the fourth when Sean Burroughs snapped a season-opening 0-for-10 slump with a two-out RBI single.
Pena had a sacrifice fly in the seventh for Boston, but Crawford hit a two-run drive off Rudy Seanez in bottom half. It was his first homer of the season.
The Red Sox and Devil Rays have traded beanballs and been involved in several dustups in recent years, but there were no significant incidents Sunday.
The teams were warned Saturday. In that game, Tampa Bay's Doug Waechter hit Alex Cora near the hip on a 2-2 pitch with two outs in the third. Crawford was the first Devil Rays batter in the bottom half and was hit in the leg by a 1-1 curveball from Boston starter Lenny DiNardo.
- Hall has homered in consecutive games and has three this season. He had one homer in his previous 103 games before Saturday.
- Ramirez went 0-for-3 with two strikeouts, and has two hits in 22 at-bats overall against Kazmir.
- Red Sox C Jason Varitek doubled in the second, extending his hitting streak to seven games. It's the longest streak by a Boston player this season.
- Tampa Bay SS Tomas Perez had his bat slip of his hands and wind up between first and second three times when he struck out in the fourth. His consecutive strikeout streak reached eight when struck out after fouling off three bunt attempts in the seventh.
- The major league record for consecutive strikeouts by a position player is nine, which has been done five times. Kansas City's Bo Jackson was the last to do it in September 1988.