Defence Secretary Sir Michael Fallon said he wanted to see more recruits from ethnic minority backgrounds join the armed forces.
He said the numbers were increasing but he wanted the military to be more representative of the community it defends.
The armed forces have a target of a 10% of recruits coming from black and ethnic minority backgrounds by 2020.
But in the 12 months to September 30 2016 just 5.8% of the total intake into the combined regular and reserve forces were from black and minority ethnic (BAME) backgrounds.
Sir Michael, speaking at the unveiling of a memorial to troops from Africa and the Caribbean who served in the two World Wars, said: “We are working towards that, we are working hard with the minority communities to encourage people to join up.
“The numbers are increasing and we are determined to get there.
"The armed forces defend us all and they should represent us all.”
He said he wanted to see “more women and more from the ethnic communities as well”.
The memorial, unveiled in Brixton, south London, followed a long campaign.
Addressing a crowd at the event, Sir Michael said: “This recognition today is long overdue.”
He added: “I hope this memorial will remind us of the ongoing contribution of our African and Caribbean communities to our country and to the defence of our country.”
In a reference to the recent terror attacks that had hit Britain, he said: “In recent weeks it is easy, of course, to focus attention on those things that divide our society.
“But it is exactly our shared history, commemorated in this memorial, that reminds us that there is so much more that connects us - our shared humanity and that duty of remembrance to those who gave their lives.”
London Mayor Sadiq Khan told the event: “We are famous here in London for our diversity.
"We not only tolerate our differences, we celebrate, respect and embrace them.
“What’s also clear is that the freedoms we hold dear, the very freedoms that allow us to be open and inclusive, are hard-won by people from different races, nationalities, religions and backgrounds.
“And this includes courageous African and Caribbean men and women who fought with the British army during the First and Second World War.”
He added: “As a society we must never forget the sacrifices they made.
"It’s vital that we pass this on to future generations, connecting children with their history.”