Fewer than a fifth of children get the recommended level of exercise by the time they leave primary school, data suggests.
Between the ages of five to seven and 11 to 12, there is a 39% drop in the proportion being moderately active for at least 60 minutes every day.
Aged five to seven, 28% are hitting the target, but this drops to 17% by ages 11 and 12 all across England.
The figures were released by Public Health England (PHE) to launch a new Change4Life campaign with Disney and Sport England.
It aims to get youngsters following their favourite Disney character in a 10-minute "shake up" exercise challenge.
A poll of more than 1,000 parents and 1,000 children to support the campaign found that 79% of parents thought exercise made their children happier.
They also said it boosted confidence and made their children more sociable.
The main motivation among children to be more active was having friends to join in (53%) and having more activities they liked to choose from (48%).
But more than a fifth (22%) of all children worried about not being very good at some sports or activities, rising to 29% of 11-year-olds.
The poll found that children's overall happiness declines with age - 64% of five and six-year-olds said they always felt happy, compared with 48% of 11-year-olds.
Eustace de Sousa, from PHE, said: "Children's physical activity levels in England are alarmingly low, and the drop in activity from the ages of five to 12 is concerning.
"Children who get enough physical activity are mentally and physically healthier, and have all-round better development into adulthood - getting into the habit of doing short bursts of activity early can deliver lifelong benefits."
The shake up games are at www.nhs.uk/10-minute-shake-up/shake-ups.