There is a "genuine concern" that leaving the EU could see thousands of asylum seekers flocking to the UK overnight, Downing Street said as it battled claims of "desperate scaremongering" by David Cameron.
Tory Eurosceptics and other "no" campaigners reacted with fury to suggestions that a referendum vote in favour of Brexit could see France scrap border control agreements - leaving migrants in the so-called Jungle camp in Calais free to cross the Channel.
Former defence secretary Liam Fox said he was "sad and disappointed to see our Prime Minister stoop to this level of scaremongering" and fellow MP Sarah Wollaston said he was "taking voters for fools".
And senior Tory David Davis said: "As the argument slips away from the Remain campaign they are forced to rely on desperate scaremongering."
But a Number 10 spokesman said it was a "perfectly feasible scenario" that quitting the EU could result in Paris ending cooperation and "thousands of asylum seekers pitching up in south-east England effectively overnight".
The row broke out after the Daily Telegraph reported that Mr Cameron was preparing to say that France would pull out of the 2003 Le Touquet treaty which means that checks for migrants trying to stow away on lorries or trains heading for Britain are carried out by UK officials in Calais.
"The French would love to pull out of the arrangement," a senior source told the Telegraph. "We will be telling people - look, if we leave the EU the Jungle camp in Calais will move to Folkestone. That is not something people want."
The spokesman declined to say whether France had made any specific threat to axe the right of British border staff to carry out checks on the other side of the Channel, saying only there was "routine and regular dialogue" on many issues, including immigration.
But he said there was "no guarantee" it would continue as it was founded on a friendly working relations with Paris and there was a risk that a vote to leave the EU would "throw that whole relationship into question".
He said he could not rule out the suggestion that the influx would be so great and sudden that it would result in similar camps being created in port towns such as Folkestone.
Downing Street's position was backed by Rob Whiteman, a former chief executive of the UK Border Agency, who said Mr Cameron was entitled to argue that France would "almost certainly" end the treaty, which has been the subject of controversy with Paris.
"There has been lots of upsides for the UK since the treaty was negotiated in 2003, not much upside for the French," he told the BBC Radio 4 Today programme.
"If you're found in a concealed vehicle, you're a clandestine, once you set foot on British soil you can claim asylum whereas if you're found in a vehicle on French soil you can't claim asylum in Britain.
"Before that treaty was put in place asylum claims were running at 80,000 a year in the UK. They are now running at about 30,000 a year so we would probably see, let's say, another 50,000 asylum claims a year which we used to get before the treaty came in."
But UKIP's migration spokesman Steven Woolfe said the warning was "based on fear, negativity and a falsehood" that the deal was connected to Brussels rather than a bi-lateral agreement.