Labour has warned most people would find it unacceptable if British workers under the age of 22 are prevented from receiving tax credits and housing benefits.
Shadow work and pensions minister Stephen Timms said the Government should consider adopting the Opposition's plan to restrict access to benefits for new migrants for two years rather than four years.
He said this would affect fewer people if a lack of treaty change during the renegotiation of the UK's European Union (EU) membership meant the restrictions also had to apply to Britons.
The BBC has reported British workers under the age of 22 may be prevented from receiving tax credits and housing benefits.
A four-year ban on access to benefits for new migrants is part of the UK's aim to stem EU migration to the country.
According to the BBC, without treaty change - which British officials have previously confirmed may not be in place by the time of the in/out referendum promised by the end of 2017 - the Government has had to consider plans which would extend the ban to Britons to prevent "direct discrimination".
The broadcaster added the residency test plan would apply from the age of 18 and not take into consideration if someone had lived in the UK all their life.
Speaking on BBC Radio 4's Today programme, Mr Timms said: "It certainly sounds as though these negotiations are not going well and ministers are waking up to the fact they won't be able to deliver the renegotiation they've promised.
"Our view has always been that there is a good case for restricting benefits to new migrants from elsewhere in the European Union.
"Our manifesto argued for a two-year restriction. The Government has been aiming a four-year restriction."
He added: "I think most people would take the view it would not be acceptable for ordinary UK citizens to be badly hit because the Government's renegotiation efforts have not succeeded."
Asked if Labour would have dropped the restrictions if they had secured power, Mr Timms replied: "I think the Government should - they haven't been able to deliver, it appears, the four years - have a go at two years. They might be able to get further with that.
"But of course if they didn't, the fallback, which they now seem to be looking at, would affect many less people if it was a two-year restriction than the four years they're discussing."
Mr Timms said Labour's plan would have a smaller impact should Britons be affected by the changes as it would apply to 18 and 19-year-olds rather than 18 to 21-year-olds.
A Government spokeswoman said: "We've already taken action to protect the benefits system and ensure that EU migrants come to this country for the right reasons and to contribute to the economy.
"Now we're focused on renegotiating our relationship with Europe and getting a better deal for Britons, and we won't speculate on other options."