Unregulated free movement of labour after Brexit would "not keep faith" with the EU referendum result, International Trade Secretary Liam Fox has said.
Dr Fox insisted the Cabinet has not agreed a deal on immigration after withdrawal from the EU.
The remarks came as Chancellor Philip Hammond signalled there was "broad acceptance" in Cabinet of a wide-ranging post-Brexit transitional period lasting up to three years.
Dr Fox told The Sunday Times: "We made it clear that control of our own borders was one of the elements we wanted in the referendum, and unregulated free movement would seem to me not to keep faith with that decision."
The International Trade Secretary said he had not been involved in any Cabinet talks on extending free movement for up to three years after Brexit, stating: "If there have been discussions on that I have not been party to them. I have not been involved in any discussions on that, nor have I signified my agreement to anything like that."
In remarks that are likely to be seen as directed at the Chancellor, Dr Fox said: "I am very happy to discuss whatever transitional arrangements and whatever implementation agreement we might want, but that has to be an agreement by the Cabinet.
"It can't just be made by an individual or any group within the Cabinet."
Mr Hammond said on Friday the transitional period up to 2022 would mean "many arrangements remaining very similar to how they were the day before we exited the European Union".
The Chancellor said there would be a registration system in place for people coming to work in the UK after Brexit, during the transitional period.
"If they come here to work after we leave the European Union, during that transitional period, the sensible approach will be to seek to register people so that we know who's coming and who's going," he said.
A Government source told the Press Association that freedom of movement will end when Brexit occurs in March 2019, and a time-limited implementation period would then follow.
The source said that the idea of mass, uncontrolled migration is "not a vision of Brexit that we will undertake".
However, the EU has made it clear continued access to the single market is dependent on free movement of labour.
Tory tensions were also on show as former Brexit minister David Jones branded Mr Hammond's transition initiative "deeply dangerous" as he accused the Chancellor of "going on manoeuvres" while Prime Minister Theresa May was abroad on holiday.
"All this agitation by the Chancellor and his allies is hugely discourteous to her and undermines her authority," Mr Jones said in The Mail On Sunday.
Pro-EU Tory MP Sir Nicholas Soames praised the Chancellor, telling the newspaper: "He has restored discipline to the political infant class who want Brexit at any cost.
"Instead of mindlessly criticising him, they should thank him for putting a stop to what was in danger of becoming a pub brawl."
The war of words came as an ally of Foreign Secretary Boris Johnson also criticised Mr Hammond's plans.
Gerard Lyons, who was an economic adviser to Mr Johnson when he was London mayor, said a two-year transition period would work better.
Writing in The Sunday Telegraph, he said: "Many of the 'risks' being highlighted about Brexit are perceived risks, not real risks. And a two-year transition would alleviate many concerns."
Mr Johnson has not yet commented publicly on Mr Hammond's transition plans.
Liberal Democrat leader Sir Vince Cable said Dr Fox's remarks point to a Cabinet "civil war".
He said: "Dr Fox's comments, directly contradicting the Chancellor over a transitional agreement on Brexit, show all the signs of a Cabinet in a state of civil war.
"The latest clash over freedom of movement in the proper transitional period reveals a deep, unbridgeable chasm between the Brexit fundamentalists and the pragmatists.
"Businesses which might have hoped that Phillip Hammond had pulled the Government back from a commitment to a catastrophic cliff edge, crashing out of the EU, have been misled.
"There is no Cabinet consensus for moderation. And the rumours of Boris Johnson being about to resign fuel the uncertainty."
Pro-Brexit Tory MP Nadhim Zahawi told Channel Four News: "This idea that there is a divide amongst the Cabinet is for the birds."
Mr Zahawi said he would be happy to see freedom of movement, with registration, for three years after Brexit, as part of an implementation process.
Dr Fox's remarks were backed by ex-Ukip leader Nigel Farage, who said: "At least someone is fighting to honour the referendum result."