Kevin Hollinrake said there was "no doubt" it is now more difficult to trade with the EU.
New poll shows the result of the 2016 Brexit referendum has been reversed.
The PM also answered questions on Liz Truss's long-term impact on the Conservative Party.
The findings follow a separate survey which showed just a quarter of Brits believe the UK should be outside the European Union.