Boris Johnson and his fiancee Carrie Symonds have cut short their holiday in Scotland after pictures of their cottage were published.
The PM, his partner and their baby son Wilfred – along with dog Dilyn – were reported to be staying in a remote three-bedroom house on the north-west coast.
But The Sun said the family were forced to abandon their break because it was no longer considered safe for them to stay.
A government source confirmed to the PA news agency that the PM has returned from Scotland.
Pictures published in the Daily Mail on Friday showed a canvas bell tent erected near the cottage, which was said to boast “spectacular views” towards the islands of Raasay, Rona and Skye.
However, the owner of the field where the tent appeared to be erected told the MailOnline Johnson had not asked for permission to use his land.
Sheep farmer Kenny Cameron also claimed the PM climbed over a fence, using chairs from the cottage, rather than a gate.
He told the news website: “Mr Johnson is meant to be leading the country and yet he is not setting a great example.
“Usually if people want to go inside a fenced area, they ask for permission first, but I was not asked at all. It is only polite to ask.
“He could have put up his tent in the garden of the cottage and there would have been no problem – but he didn’t do that.”
Cameron said the PM could “easily have damaged the fence by climbing over it as a short cut”, and also criticised him for apparently lighting a bonfire after a period of dry weather.
The Sun, quoting a senior Tory, said blame on the location of the PM’s holiday being revealed is being pointed at the SNP.
But the party’s Westminster leader Ian Blackford said on Twitter: “To blame @theSNP for this is classic deflection from a prime minister who is under attack for his shambolic government.
“It was not @theSNP who told the Daily Mail and for what it’s worth so many folk in the Highlands told me they knew where the PM was.”
Johnson had been expected back at Westminster next week, after he defied calls to return amid the A-level results fiasco.