Britons do not have racist feelings toward Romanians and Bulgarians, but blame the government for a lack of real information on how further immigration will impact Britain, a study has indicated.
A total of 80% of respondents to the survey by independent immigration think-tank British Future backed tougher measures.
However, 77% said they would welcome migrants from Romania and Bulgaria who work hard and pay taxes, learn the language and become part of the community.
A young Romanian couple walk with suitcases outside the main bus station
The British Future survey found that 38% of people think the number of people coming from Romania and Bulgaria will be up to 50,000 in 2014, while 18% believe it will be between 50,000 and 100,000. A further 13% think the number could be up to 500,000.
The Romanian government has said it does not believe the figure will be anywhere near that many.
Nearly three-quarters of people think the Government has failed to provide enough information on what will happen when restrictions to the UK Labour market are lifted for Romanians and Bulgarians.
British Future director Sunder Katwala said: "The timing of new EU migration from Romania and Bulgaria isn't popular and people have worries about the pressures it may cause - but they're not prejudiced.
"People want issues to be sorted out, not stirred up. If new arrivals make the effort to work hard, contribute and integrate into Britain, most people are broadly accepting."
Temporary curbs imposed on Romanians and Bulgarians in 2005 to protect the British labour market will be lifted on January 1.
The Government has refused repeated demands to publish or commission estimates of the numbers expected to enter Britain in the face of unofficial research predicting as many as 50,000 people arriving from the eastern European countries each year.
Leading politicians from Bulgaria and Romania have dismissed fears that the change in access restrictions will trigger a wave of immigration to the UK.
Katwala went on: "The Government has said it's too unpredictable to put a figure on how many people may come in 2014. Our report examines how significant the public thinks migration from these countries might be.
"They don't think millions from Romania and Bulgaria are about to up sticks, but they do think the numbers could be high enough to mean sensible plans should be in place."
Fewer than half of respondents - 43% - said they have heard Prime Minister David Cameron talking about changes to UK working rights for Romanians and Bulgarians and nearly a third - 32% - had not heard from any mainstream politicians.
"People want a sensible conversation about immigration from Romania and Bulgaria and how we're going to manage the pressures that may arise - on public services, on jobs and on the 'Britishness' of their communities," Katwala added.
"Where there is prejudice from a minority, as we have seen towards the Roma community, this should be tackled head-on. But politicians need to understand that this is different from the genuine anxieties of ordinary people."
The survey comes after the Prime Minister set out radical plans to overhaul welfare rules, including stopping new arrivals from the EU receiving out-of-work benefits for their first three months in the country.
Cameron and Home Secretary Theresa May have both signalled strong support for shaking up free movement rules in the European Union.
May has said reform of the right to free movement should form part of any negotiations on new arrangements for the UK's membership of the European Union.
A leaked Home Office paper suggests the Home Secretary wants to introduce a cap on migration from the EU, possibly at 75,000 a year - around 30,000 lower than the current level.
Despite the heated debate about EU free movement and Britain's relationship with Europe, the British Future survey found respondents to be undecided on the issue.
Some 24% said the Government should reluctantly accept free movement within the EU and focus on practical measures to manage its impacts, while 30% think the UK should leave the European Union to stop free movement.