Roberts claims Kick It Out have not been hardline in their response to the John Terry and Luis Suarez abuse incidents.
The week of action started on Thursday but Roberts told BBC Sport: "I find it hard to wear a T-shirt after what has happened in the last year. I won't wear one."
He went on: "I'm totally committed to kicking racism out of football but when there's a movement I feel represents the issue in the way that speaks for me and my colleagues, then I will happily support it.
"I think people feel let down by what used to be called 'Let's Kick Racism Out of Football'. People don't feel like they have been strong enough."
Kick It Out chairman Lord Ouseley said he understood Roberts' feelings.
Ouseley said: "He's not been slow in coming forward about this and I respect his views.
"I can understand his frustrations. If Kick It Out had the power to sort this out then we would have done. Sorry Jason, we don't have that kind of power."
Roberts added that the four-match ban handed to Terry for racially abusing QPR's Anton Ferdinand was too lenient.
He added: "The four-match ban was, for me, not a heavy enough sanction for what happened. I'm not happy. Certainly they should have given him a longer ban. The sanction is nowhere near harsh enough."