Two years ago we began the long overdue work of modernising our electoral rolls, the authoritative lists of who is eligible to vote.
The old system, which required the 'head of a household' to register on your behalf, belonged in a bygone era. In fact it dates back to 1832.
It was also open to fraud.
Under the old system you could register anyone you liked, without having to produce any ID. This meant fraudsters could bag votes for people who didn't actually exist. We saw serious abuse of this flaw in the Tower Hamlets Mayoral elections last year, and the courts have since overturned the election result.
In 21st Century Britain ballot stuffing is not acceptable. That's why we've moved to a system of Individual Electoral Registration (IER). Electors now have a responsibility to make sure they register to vote. For the first time they must also prove their identity before they appear on the rolls. This brings us into line with every other serious democracy in the world.
The new system cleans things up hugely. Since it was introduced last year, IER has confirmed 96 out of every hundred electors as genuine.
But what about the remaining four out of every hundred? How do we know if they are real people or 'ghosts'?
These entries have now been contacted five or six times by election officials writing to them and knocking on doors. So far there's been no answer. By the end of this year those five or six attempts will have gone up to nine, 10 or 11. The chances of these entries being real people rather than ghosts is vanishingly small.
Yet the impact of these ghosts on democratic elections is scary. In May the Electoral Commission found that there could still be up to 1.9million ghost entries on the electoral register. Take Hackney. At the last count nearly a quarter of all the entries on the register in Hackney could have been ineligible to vote. This leaves elections wide open to fraud and, as we saw in Tower Hamlets, can potentially even swing the result.
Those who oppose this change argue it will leave disengaged voters off the rolls. They have a point that some communities like students, young black men, or British citizens living abroad have much lower rates of registration. But the answer to this is to have a strong and effective registration drive, not to stuff the electoral rolls and potentially even the ballot boxes with the names of people who don't exist.
I am already working on ways to improve the way we do registration drives in future. But in the meantime, these ghost voters have got to go. They make fraud more likely and distort the accuracy of our electoral rolls. Pretending they exist also means some constituencies appear bigger or smaller than they ought to be, stopping votes in one part of the country having the same weight as others. This is particularly unfair to Northern Ireland, which switched to the new system years ago.
Cleaner, more accurate registers will make for fairer and cleaner elections. We can't let a sensible bit of democratic plumbing get clogged up by a nasty party-political blockage.
John Penrose is the minister for constitutional reform, and the Conservative MP for Weston-super-Mare