He won't be alone. Patriots safety Devin McCourty won't attend either.
"I'm not going to the White House," McCourty told TIME on Monday in a text message. "Basic reason for me is I don't feel accepted in the White House. With the president having so many strong opinions and prejudices I believe certain people might feel accepted there while others won't."
McCourty added: "I can't imagine a way I go there."
Meanwhile, president Donald Trump considers Patriots owner Robert Kraft, coach Bill Belichick and quarterback Tom Brady friends. Kraft, a lifelong Democrat, explained last week why he's close with Trump.
"When [Kraft's wife] Myra died [in 2011], Melania [Trump] and Donald came up to the funeral in our synagogue, then they came for memorial week to visit with me," Kraft told Gary Myers of the New York Daily News. "Then he called me once a week for the whole year, the most depressing year of my life when I was down and out. He called me every week to see how I was doing, invited me to things, tried to lift my spirits. He was one of five or six people that were like that. I remember that."
Brady's wife, Gisele, has barred him from talking publicly about politics but in December 2015 the Patriots quarterback spoke about Trump's presidential run.
"I support all my friends in everything they do. I think it's pretty remarkable what he's achieved in his life," he said at the time. "You're going from business, kind of an incredible business man and then a TV star, and then getting into politics. It's a pretty different career path. I think that is pretty remarkable."
Fast-forward to election-day eve, November 2016 -- Trump told supporters in Manchester, New Hampshire that Brady called him earlier in the day to say he voted for him, and then he read a letter of support he received from Belichick.
Not surprisingly, Trump predicted the Patriots would beat the Falcons.
McCourty and Bennett wouldn't be the first Patriots to skip out on the White House. Brady did just that back in 2015, when Barack Obama was still in office, choosing instead to work out at Gillette Stadium before he was later spotted at the Apple Store in New York City.