The taxman is chasing nearly £2 billion in taxes that have potentially gone unpaid by the super-rich, according to the public spending watchdog.
More than half of the cash is linked to controversial aggressive tax avoidance schemes, the National Audit Office (NAO) found.
It assessed the work of a team dedicated to collecting taxes from high net worth individuals, with around 6,500 people worth more than £20 million falling into the bracket last year.
HM Revenue and Customs (HMRC) is dealing with disputes over how much tax should be paid with one third of the group.
On average, each of the taxpayers has an average of four issues to be dealt with and 4,000 of the inquiries have been open for more than three years.
In the last five years, HMRC has also looked into and closed 72 cases into suspected of tax fraud by high net worth individuals. Only two were criminally investigated and just one case led to a conviction.
Investigations into 10 of the country's richest over suspected fraud are on-going.
NAO head Amyas Morse said: "The tax affairs of the wealthiest in society are complex, making it harder for HMRC to ensure that they are paying the right amount of tax.
"HMRC's specialist team gives it a better understanding of the tax affairs and behaviours of these taxpayers. While the yields from HMRC's work in this area have increased it needs to evaluate what approaches are the most effective and to understand the outcomes it achieves."
HMRC's specialist unit, which assigns high net worth individuals a "customer relationship manager" to identify problems, raised £416m on top of the tax voluntarily declared by the country's richest in 2015/16.
But it is investigating a potential £1.9 billion in unpaid taxes dating back a number of years and estimates around £1.1 billion is linked to tax avoidance schemes, with 15% of the wealthiest having used at least one.
In 2014-15, the super-rich, around 0.02% of all taxpayers, paid more than £4.3bn in tax, according to the NAO's report.
It also found that 137 of the country's richest who had undisclosed assets in Liechtenstein used an agreement with the tax haven in 2009 to admit liabilities of £141m in return for less harsh penalties. The average settlement was £1m per person.
The unit has is also investigating 40 wealthy individuals named in the Panama Papers following the leak at law firm Mossack Fonseca.
Meg Hillier, who chairs the public accounts committee, said: "For the ordinary taxpayer, HMRC's use of 'customer relationship managers' will sound rather a cosy way for HMRC to engage with the richest people in the country; people worth over £20 million each.
"However, I am pleased to see HMRC is beginning to link these individuals to the businesses and trusts they are also involved in, to help tackle the £1.9 billion of tax that is potentially at risk."
An HMRC spokesman said: "Everyone must pay the tax that is due and we do not accept less.
"HMRC enforces the rules impartially and last year we tracked down an additional £416 million in tax from the wealthiest, which would have otherwise gone unpaid.
"We will continue to evaluate our results so that we carry on getting what is due to the country.
"The NAO's report recognises that taxing the wealthy is complex but also commends HMRC's approach as both sensible and in line with OECD best practice advice."