Cory Stillman assisted on Horton's goal that made it 2-0 in the first, then scored his third of the season midway through the second to restore the two-goal lead.
"We wanted to have a quick start, and the start won us the game," coach Peter DeBoer said.
Anderson made 41 saves -- including 38 over the final 40 minutes -- in his first start of the season.
"It's good to get the first one out of the way -- first win, first start -- I just wanted to go out there and battle for the guys and make sure that I gave them a chance to win," Anderson said. "We came out flying in the first and took it to them, I think, and we were able to get a good lead. After that, it was just holding down the fort."
Anderson, Tomas Vokoun's backup, stopped 26 shots in the third period alone, lunging to his left to make a glove save on Antoine Vermette's shot as Ottawa enjoyed a power-play for the final 3:05 of the third after David Booth was assessed a double minor for high-sticking.
"I knew I slid way too far and I was just trying to get back, and luckily I was back in time," Anderson said.
Florida, coming off a 3-1 loss in Montreal on Monday night, has alternated losses and victories in getting off to a 3-3 start.
Daniel Alfredsson scored a power-play goal early in the second for the Senators, who lost their second straight.
Outshot 11-3 in the first period, Ottawa held a 13-7 margin in shots in the second before dominating the third 26-3.
"It's frustrating," Alfredsson said. "We really wanted to get off to a good start, and they got us on our heels. We struggled to get out of our own end and obviously they got some breaks right away. We were a much better team in the second and the third, but it's tough when you give up two goals early."
Senators defenseman Filip Kuba helped set up Alfredsson's goal to extend his assists and points streaks to six games.
Martin Gerber stopped 18 shots. The Senators are 1-3 on a five-game homestand that ends Friday night against Anaheim.
Olesz scored what was initially announced as an unassisted goal when he jumped on an apparent turnover in the slot by Ottawa defenseman Alexandre Picard and beat Gerber 4:45 in.
An assist was subsequently awarded to Panthers forward Gregory Campbell, whose pass initiated the play before Picard fanned on his pass attempt while he was being forechecked by Radek Dvorak, with the puck going directly to Olesz in the slot.
"The puck was bouncing and I heard my partner yell for a pass," Picard said. "I went to do it and (Dvorak) slashed my stick and the pass didn't go as hard as it should have."
Horton made it 2-0 at 5:33 when he put a shot over Gerber's glove from the left faceoff dot for his second of the season.
Alfredsson had a goal disallowed in the final seconds of the first before he scored a second-effort goal early in the second to draw Ottawa within one.
A video review confirmed that Alfredsson's deflection from the slot with 3.5 seconds remaining in the first made contact with the puck above the level of the crossbar.
Alfredsson made his next scoring chance count for his second goal of the season 2:23 into the second when he poked his own rebound past Anderson.
- Panthers D Keith Ballard saw his four-game points streak end. Ballard, acquired from Phoenix as part of the draft day trade that sent Olli Jokinen to the Coyotes, had a goal and four assists over that stretch.
- Kuba, who has no goals and eight assists, was acquired from Tampa Bay along with Picard and a first-round draft pick in an Aug. 29 trade for Andrej Meszaros.
- Senators LW Christoph Schubert returned after he was left out of the lineup for two games. RW Shean Donovan was a healthy scratch and missed his first game of the season.