Graves served as Yorkshire chair between 2012 and 2015, a spell that coincided with Rafiq's initial period at the club. And his comments come after Yorkshire admitted to four amended charges relating to their handling of Rafiq's allegations of racism and bullying, with sanctions expected to be handed down on June 27.
"When I was chairman of the club, when some of these allegations were made, I can tell you now, nothing was brought up within the club," Graves told Sky Sports News.
"And I ran the club on an open-door basis as chairman, players could come and see me if they had issues and they did so on a regular basis ... nothing was ever brought up on the table about any racism allegations whatsoever.
"I think there have been odd occasions where words have been said that people may regret afterwards," he added. "I don't think it was done on a racist, savage basis. I think there was a lot of - I know people don't like the word banter - but I think there could have been a lot of banter in there about it, and I know people don't like that."
Graves' comments drew swift condemnation from the ECB, for whom he served as chair between 2015 and 2020.
"We are disappointed to see the comments made by Colin Graves on the historical racist issues at Yorkshire County Cricket Club," the ECB said in a statement.
"We must never again hear of accounts like Azeem Rafiq's, where racist slurs are used as part of normalised everyday language.
"These events, along with many issues experienced by Azeem and others during their time at Yorkshire have been upheld more than once, including during proceedings overseen by the Cricket Discipline Committee.
"We vehemently disagree that this is 'just banter' and believe that any debate in that regard should stop immediately. Racism isn't banter."
In 2002, Graves bailed the club out with a multimillion-pound loan, of which some £14.9 million is still owed to the Graves Family Trust. The club is due to repay £500,000 in October, with the rest due for repayment next year, and Graves had offered to help restructure that debt in return for a second stint as chair.
Last week, however, he withdrew his candidacy after criticising the length of time the board had taken to nominate a successor to Lord Patel, who stepped down in March.
"As far as I know [Yorkshire] are going to run out of liquid cash in September time," Graves said. "If [a buyer or investor] doesn't [come in], presumably the only way forward for them is to go into administration.
"And when that happens - and I hope it doesn't happen - if they go into administration, then the administrator will have to talk to the trustees of the trust, because the trust has got first legal charge over the ground."
In response, Yorkshire's board insisted that the club was conducting "many positive conversations" around investment - including, it is believed, with the former owner of Newcastle United, Mike Ashley, and the Saudi Arabia national investment fund - but insisted that "the club is not for sale".
The ECB added: "Yorkshire is of huge importance to cricket in England and Wales with a deep history and an ability to inspire and connect all cricket fans. They are currently working hard to resolve their financial and cultural challenges and the club, and its new management, have our full support.''