The Equality and Human Rights Commission (EHRC) has written to the UK's political parties telling them to tone down their Brexit rhetoric.
The letter suggests there needs to be a discussion about British values as the Government looks to formalise the nation's divorce from Brussels following the leave vote on June 23.
The national equality body and human rights institution states in the letter that racist, anti-Semitic and homophobic attacks which have taken place in the aftermath of the referendum result are "all stains on our society".
The letter reads: "We are concerned that attacks on supporters of both sides of the Brexit debate have polarised many parts of the country.
"There are those who used, and continue to use, public concern about immigration policy and the economy to legitimise hate.
"The vast majority of people who voted to leave the European Union did so because they believe it is best for Britain and not because they are intolerant of others."
The letter calls on the Government to do more to combat hate peddled by a "small minority" as it also suggests there should be a review of the effectiveness of sentencing for hate crimes in England and Wales, including the ability to increase sentencing for crimes motivated by hate.
It states that "politicians of all sides should be aware of the effect on national mood of their words and policies" even when those policies are not acted upon – like the Government's now-ditched proposal for companies to list foreign workers.
"Your offices bring with them a responsibility to ensure that policy debate is conducted in a way that brings the country together and moves it forward," the letter states.
"Robust discussion is a central pillar of our democracy and nothing should be done to undermine freedom of expression.
"The right to free and fair elections supported by accurate information and respectful debate is also essential to our democratic process.
"Our elected representatives and the media should reflect and foster the best values in our society and engage people on contentious issues in a responsible and considered way."
Meanwhile, the chairman of the EHRC, David Isaac has also expressed concerns about the way in which businesses approach religious belief in the work place.
He said: "There are a lot of myths out there when it comes to dealing with religion at work. I want to put the record straight.
"It is about taking a common sense approach. You can send Christmas cards and have a Christmas party and you might also decide to provide facilities for special religious diets, but that is your choice."