NEW ORLEANS -- Baylor coach Matt Rhule, expected to be a leading candidate for head-coaching vacancies with the New York Giants and Carolina Panthers, said Tuesday he expects to be back with the Bears next season.

"I plan on that," Rhule said. "I certainly think I will be."

Sources told ESPN's Adam Schefter earlier Tuesday that Rhule had declined the chance to interview for the Cleveland Browns' head-coaching vacancy.

Rhule, who signed a contract extension in September that runs through the 2027 season, told reporters he has not been in contact with any NFL teams.

The Giants were making Rhule a focus of their coaching search, ESPN's Chris Mortensen reported Monday.

Rhule on Tuesday lamented the NFL speculation surrounding him, calling it a distraction for his players. He said he is happy at Baylor and believes the program could become a perennial College Football Playoff contender but acknowledged that any move he makes has to be in the best interest of his family.

"One thing people don't realize is coaches, we pick up our families," Rhule said. "We rip them out of their homes. We rip them out of the places that they are. Sometimes you do that until you get to a point where you find happy. You shouldn't mess with happy.

"There's a lot to accomplish at Baylor. And most importantly, it's just each and every year, I want to put together a championship-caliber team. And I think we have a chance to be even better next year than we are this year. ... More than money, it's about the situation for my family."

Baylor, who went from 1-11 two years ago to 11-2 this season, will play Georgia in the Allstate Sugar Bowl on Wednesday.

"I'm always looking three, four, five years down the line," Rhule said. "I think we have a chance to be a perennial national contender, to be in the top 10 and be in the top five and go to College Football Playoff and play in New Year's Six bowls."