Staffing levels in many jails are too low to maintain order, the prisons watchdog has warned.
In a scathing critique, Peter Clarke insisted reform ambitions will be in vain unless violence and drugs behind bars are addressed.
The Chief Inspector of Prisons in England and Wales said the "seeming inability" to keep drugs out has been a major factor in declining safety standards.
Mr Clarke's annual report for 2016/17 said debt, bullying and self-segregation by prisoners are commonplace.
"This has all been compounded by staffing levels in many jails that are simply too low to keep order and at the same time run a decent regime that allows prisoners to be let out of their cells to get to training and education, and have access to basic facilities," he said.
"When a person is sent to prison, the state accepts responsibility for their well-being, including their physical and mental health, safety and education. There is clear evidence that for too many prisoners the state is failing in its duty."
Staffing levels have repeatedly been highlighted as violence and self-harm surged to record levels across the estate, which has also been hit by a number of disturbances.
Ministers have launched a recruitment drive to add 2,500 frontline officers as part of prison reform measures.
At the end of March, there were 18,403 "full-time equivalent" staff in frontline categories, which was an increase of 75 on the last year and 515 on the previous quarter.
The chief inspector said additional officers "can only be of help" but he stressed that staffing numbers are "not the only issue".
Of the 29 local and training prisons inspected during the year, inspectors judged 21 of them to be poor or "not sufficiently good" in the area of safety.
Mr Clarke criticised "squalid, dirty and disgraceful" conditions seen in some establishments, saying prisoners have pointed out insect and vermin infestations on several occasions.
Far too often, men are sharing a cell in which they are locked up for up to 23 hours a day, his report said.
The chief inspector said that in many cases, the response to his inspectorate's previous recommendations has been "unforgivably poor" - saying there is a "huge gap" between accepting and achieving them.
This "leads to the inevitable suspicion that there is a degree of lip service being paid to our recommendations", he added.
His report also raised the alarm over the "staggering" speed of decline in the youth custody estate, which has seen rising levels of self-harm and assaults.
Mr Clarke disclosed that earlier this year, he had felt compelled to raise his serious concerns with ministers.
"By February 2017, we concluded that there was not a single establishment that we inspected in England and Wales in which it was safe to hold children and young people," he said.
Mr Clarke gave a more positive assessment of women's prisons, adding that open prisons and high-security establishments also "inspect well".
Elsewhere, the report warned that new psychoactive substances, previously known as "legal highs", are beginning to have an impact in immigration detention.
The Ministry of Justice (MoJ) said it was forming a new unit to track the implementation of recommendations issued by the prisons inspectorate and the Prisons and Probation Ombudsman.
An MoJ spokeswoman said: "We know that our prisons have faced a number of long-standing challenges, with annual increases in violence and self-inflicted deaths.
"That is why we have taken immediate action to boost prison officer numbers and put in place new measures to tackle drugs and mobile phones.
"The Justice Secretary has been clear that we need to create calm and ordered prison environments to help ensure effective rehabilitation, and that achieving this is his priority."