Benefits are wrongly being paid to more than 5,000 immigrants with no right to claim state help, research carried out by the government suggests.
They are among some 371,000 arrivals to this country on welfare identified by an exercise to match benefit, border control and tax records for the first time.
Employment minister Chris Grayling told the BBC's Today programme that Labour had made a "scandalous omission" by not routinely noting the nationality of benefits claimants.
The government plans to begin doing so when Universal Credit is introduced from 2013.
The data-matching project found 371,000 working-age benefit claimants were non-UK nationals when they first applied for a National Insurance number, 258,000 of them from outside the European Economic Area.
So far, detailed work has been done only on a sample of 9,000 of the latter group - three quarters of whose records were able to be matched. It found 54% were now British citizens and most others had an immigration status that allowed welfare claims.
The statistics indicated that around 2% of all foreign-born benefits claimants did not have the correct immigration status. Around 125 cases are now under investigation.
The rate of illegal claimants is roughly equivalent to the most recently released statistics on fraud and error in the benefits system as a whole, which the Department of Work and Pensions (DWP) has put at around 2.1%.
While that statistic accounts for the raw amount of illegal benefits paid out and not the number of claimants, it does indicate the fraud rate may not be much higher among immigrants than the general population.
Grayling said: "We will root out those claimants who can not prove their immigration status and in turn they will be stripped of their benefits. I was shocked to discover that the current system does not record the nationality of benefit claimants and we are urgently taking steps to make sure we know exactly how many non-UK nationals are claiming UK benefits."
Immigration minister Damian Green said: "These findings uncover a worrying issue we have inherited, which is why we've ordered urgent work to pursue claimants suspected of abuse and to withdraw their benefits if they cannot prove they are entitled to claim."
Those found to be abusing the system will also have their details sent to other agencies including local councils to ensure other benefits were also stopped, the Treasury said.
Labour claimed that the government was using "rhetoric" to hide their failures on immigration.
Chris Bryant, shadow immigration minister, said: "Far from strengthening immigration control the government is returning fewer illegal migrants and net migration has continued to increase.
"And at the same time increasing numbers of people born here and abroad are on out-of-work benefits because the government's failure on the economy has pushed unemployment up."
Robert Oxley, campaign manager of the TaxPayers' Alliance, said: "These figures reveal that a substantial amount of welfare payments are ending up in the pockets of non-uk nationals.
"With billions at stake the proper controls need to be in place to prevent benefit tourism from swallowing up taxpayers' money. Taxpayers will rightly worry that the rules designed to prevent benefit tourists are steadily being eroded by a meddlesome EU, leaving Britain to pick up a bigger welfare bill than it needs to."