Monta Ellis scored 26 points for the Bucks and Mike Dunleavy had 21. Milwaukee has lost four straight, all by fewer than 10 points.
Brandon Jennings had 10 assists but made just three of 15 shot attempts and scored six points.
Ellis cut it to 94-92 on a jumper with 1:50 left, but Toronto's Kyle Lowry replied with a 3-pointer. Lowry scored 15 and was one of six Raptors to reach double figures.
Bucks forward Larry Sanders stayed home for what coach Scott Skiles called disciplinary reasons. Rookie guard Terrence Ross (flulike symptoms) was Toronto's only absentee.