The taxman has been accused of "going after the minnows" after the number of super-rich grew but the amount they paid in tax fell.
MPs told HM Revenue and Customs (HMRC) that ordinary taxpayers felt there was a "two-tier" tax system that left them squeezed while the wealthiest failed to pay up.
The system for dealing with the very richest gives the impression of cosy chats on leather backed chairs over cappuccinos, tax bosses were told when they appeared before the Public Accounts Committee.
But new tax chief Jon Thompson insisted that HMRC was now taking a tougher stance on "compliance" with wealthy individuals and it focused its efforts on reaping back cash.
Conservative Charlie Elphicke said: "The number of high net worth taxpayers has risen from 5,900 to about 6,500, so there's more rich people and yet over the time the tax take seems to have fallen by £1 billion.
"My constituents would say to me that it doesn't look like this unit has been very successful. What do I say to them?"
Mr Thompson replied: "Well, the answer to that is that if it wasn't for spending £14.5 million on this unit you would not have received £416 million worth of compliance yield last year.
"That's the answer to that."
It comes after a report by the public spending watchdog last month found the taxman was chasing nearly £2 billion in taxes that have potentially gone unpaid by the wealthiest people in the country.
More than half of the cash was linked to controversial aggressive tax avoidance schemes, the National Audit Office (NAO) found.
It assessed the work of a team dedicated to collecting taxes from high net worth individuals, with around 6,500 people worth more than £20 million falling into the bracket last year.
HMRC is dealing with disputes over how much tax should be paid with one third of the group.
Since the report, which recorded that only one case led to a conviction, a further arrest has been made, MPs were told.
Mr Elphicke said it appeared that HMRC was "going after the minnows while the big fish swim away".
But Mr Thompson insisted that was "not the case", insisting 10 criminal cases were "in the pipeline".
"I don't think we are complacent about it," he said.
The committee also heard that across HMRC staff around 300 staff had been sacked for poor performance and a group "in the low tens" were kicked out because they stepped "over the line".
But the no-one has been removed from the high net worth 380 strong task force for such behaviour, they heard.