Equality group Kick It Out's annual week of action to highlight the evil of racism in football starts on Saturday, but Roberts claims the organisation has not done enough in their response to the John Terry and Luis Suarez incidents.
Roberts, who has also played for Blackburn, Wigan and West Bromwich Albion during his well-travelled career, believes Kick It Out should have pushed for a more severe punishment for Chelsea defender Terry, who was eventually handed a four-match ban by the Football Association for racially abusing QPR's Anton Ferdinand despite being cleared of the charges in a court case.
"The four-match ban was, for me, not a heavy enough sanction for what happened," Roberts told BBC Sport.
"I'm not happy. Certainly they should have given him a longer ban. The sanction is nowhere near harsh enough."
The Terry controversy came after Liverpool striker Suarez was banned for eight matches for repeatedly hurling racist insults at Manchester United defender Patrice Evra last season.
The issue of racism came up again on Tuesday when several England players, including Sunderland defender Danny Rose, were racially abused during an Under-21 match in Serbia and Roberts has had enough.
"I find it hard to wear a T-shirt after what has happened in the last year. I won't wear one," Roberts added.
"I'm totally committed to kicking racism out of football but when there's a movement I feel represents the issue in the way that speaks for me and my colleagues, then I will happily support it.
"I think people feel let down by what used to be called 'Let's Kick Racism Out of Football'. People don't feel like they have been strong enough."
Kick It Out chairman Lord Ouseley admitted he understood Roberts' feelings, but conceded there was little his organisation could do about the punishment for racist abuse.
"He's not been slow in coming forward about this and I respect his views," Ouseley said.
"I can understand his frustrations. If Kick It Out had the power to sort this out then we would have done. Sorry Jason, we don't have that kind of power."