Michael Fallon has admitted he was "a bit careless" to say that some towns risk being "swamped" by migrant workers from Europe.
But the Defence Secretary maintained that the "large number" of immigrants coming from the continent was putting pressure on housing and essential services in the UK.
David Cameron will face questions from MPs later about his Government's increasingly fraught relationship with Europe following Brussels' shock demand for an extra £1.7 billion from the British taxpayer.
The Prime Minister is also likely to be asked about his approach to tackling immigration within the EU after Fallon was effectively slapped down by Number 10 yesterday.
Under pressure from Ukip and with a crunch by-election looming in the Rochester and Strood constituency, Fallon had significantly ramped up Tory rhetoric on the sensitive issue with his warning that some areas felt "under siege" and action was needed "to prevent whole towns and communities being swamped by huge numbers of migrant workers".
A Government source said the Defence Secretary should have "chosen his words better" as Downing Street sought to calm the situation following Fallon's comments.
But the minister stood by the substance of his remarks as he acknowledged he had been careless in his choice of language.
Fallon told Sky News: "I was a bit careless with my words, I accept that. But, yes, there is pressure now, there are a large number of people coming here from the rest of Europe - this is one of the more successful economies in Europe and there is pressure as a result of that migration on social services, on housing, on school places for example.
"That's what the Prime Minister will be addressing when he puts forward his proposals for some kind of control."
Labour's Keith Vaz, chairman of the influential Commons Home Affairs Select Committee, said Fallon's original comments had been "nasty, inappropriate and wrong".