Immigration
But Robert Jenrick said removing the mural was the right decision.
It's yet another blow for Rishi Sunak's "small boats week".
The prime minister's hopes of getting on the front foot were scuppered by the Tories' foul-mouthed deputy chairman.
Justice secretary Alex Chalk said the comment was "not bigotry at all".
The move comes after weeks of delays amid safety concerns.
The immigration minister repeatedly refused to name a lawyer he claimed was working with Keir Starmer to block migrant deportations.
"Are you trying to blame the Labour Party?"
Even Tories were appalled by the prime minister's accusation.
The Freedom from Torture group put a blue plaque up at the PM's constituency office accusing him of being on the "wrong side of history".
The work and pensions secretary made the embarrassing gaffe as he insisted the prime minister's plan to "stop the boats" was working.