James, the league's leading scorer, was out of the Cavaliers' lineup with a sprained left index finger, while Toronto's Bosh sat out with a strained groin.
Cleveland lost for just the second time in eight games without James since the start of the 2005-06 season.
"We know we can play without him but its easier said than done," Cavaliers forward Drew Gooden said. "But if we don't have him, we can't feel sorry for ourselves and be down and wait for him to get ready. We've got to do whatever it takes to win."
Daniel Gibson led Cleveland with 24 points and seven assists. Gooden added 15 points and 14 rebounds.
Sasha Pavlovic had 15 for the Cavaliers, who have lost two straight following a four-game winning streak.
James sat on Cleveland's sideline wearing a black jacket and gray turtleneck. Bosh remained in the locker room getting treatment during the first half but returned after the intermission.
Cleveland coach Mike Brown expects James to practice Saturday but would not say whether he will play Sunday when Cleveland visits Boston.
"We want him to play, but I'm not going to force anything or overdo anything with him or anyone else on our team," Brown said.
Brown said he failed to get enough shots for center Zydrunas Ilgauskas, who went 3-for-7 and scored six points.
"I didn't do a good job of trying to force-feed Z," Brown said. "He had seven shots, which is nowhere near enough. I went away from trying to get him the ball and I can't do that, especially when LeBron is out."
Brown also said defensive miscues created too many open looks for the Raptors.
"We did some things defensively where we put ourselves in a bind," Brown said. "At times we switched when we weren't supposed to, at times we didn't switch when we were supposed to."
Those breakdowns may have been the result of Cleveland's lengthy list of injured players. Besides James, the Cavaliers were without guard Larry Hughes (bone bruise, left leg), forward Donyell Marshall (sprained right wrist) and forward Cedric Simmons (sprained left ankle).
"We're missing half of the team," Ilgauskas said. "Guys are trying to step up and we're playing all different kinds of rotations. We just came up a little short today."
Delfino made his first six field goals. He didn't miss until he was blocked by Drew Gooden at 1:04 of the second quarter.
"One of the thing we talked about after he hit his first couple was, 'Let's get up into his chest and take away his air space, make him put it on the floor,"' Brown said. "We didn't start doing that until late in the game."
Toronto's Jose Calderon had 11 points and 10 assists and has posted double-digits in assists in four consecutive games.
T.J. Ford returned from a four games absence after injuring his left arm in a collision Nov. 20 with Dallas' Josh Howard. Ford had two points and six assists in 16 minutes.
"We told him to try to get everybody going and just see how he feels," Toronto coach Sam Mitchell said. "You could tell he was rusty. He's only had one practice since he's been out."
Cleveland came in having won eight of its past nine against Toronto, including four in a row.
The Cavaliers were ahead 28-27 after the first but fell behind early in the second on a jump shot by Jamario Moon and never led again. Moon finished with nine points and nine rebounds.
Toronto led 54-49 at the half.
- Toronto placed guard Darrick Martin on the inactive list and activated forward Maceo Baston.
- Cleveland is 5-5 on the road this season.
- Gibson's seven assists were a career-high.