The concept of people being linked by something they see as being inextricably negative is ridiculous to them. A typical response goes like this one from a commenter on an article I wrote for the BBC in August: "Saying 'the deaf community' is like saying 'the left handed community', it's utter rubbish."
My reply (in real life - I ignored the online commenter) is usually to start explaining the world I grew up in, with Deaf parents, watching amazingly visual stories being told in a blur of signing hands at Nottingham Deaf club, feeling utterly at home in this community.
I tell them about BSL's own unique structure, grammar and syntax, and I also tell them about Deaf films, theatre and art, and mention our TV programmes, such asSee Hear and the BSL Zone.
Sometimes, people start to get it. Most times, I can see in their facial expression, and their eyes, that they don't.
It's not just random commenters, either. The words of an Observer TV review of BBC1's The Silence, the last high profile drama I can remember which starred a Deaf actress, have been burned on my eyes for years. In it, Euan Ferguson wrote: "I'm not allowed by my hands to write the "deaf community" except in those inverted commas."
The problem is that the Deaf community (note I'm using a big D - referring to Deaf people who use sign language and take pride in their identity) is largely invisible to the wider world, because of the language divide between spoken language and sign language.
This invisibility is reflected in the mainstream media. In TV dramas, how many recurring characters happen to use sign language or lipread, or seem in any way Deaf? Meanwhile, if a Deaf story pops up in the news, you'll nearly always see non-Deaf people speaking on our behalf - rather than Deaf people representing themselves.
That's why it's so refreshing to see a hearing person who knows nothing about Deaf people at all actually meeting some Deaf people and understanding their world. Step forward, Grayson Perry.
In tonight's episode of his identity series Who Are You?, he meets Paula Garfield and Tomato Lichy, a Deaf couple who are also campaigners, from North London.
Perry's well on his way to being a national treasure, but he's going to be very popular with Deaf people after tonight's episode. The reason for this is, he actually gets us.
The couple, whose daughter became the first child in Britain to be legally registered with a sign name, tell him that Deaf culture is based on a shared language and lifestyle, and crucially, a shared experience of oppression by hearing people. As Paula tells him, "people think Deaf culture doesn't exist."
Perry soon homes in on the conflict between the medical profession's focus on encouraging Deaf people to "function in a hearing world," and Deaf culture, with its positive idea of Deaf identity, with Deaf people being connected by language and shared values.
He attends a party full of Deaf people signing, and says, quite perfectly, that this is "a rich rounded culture that is based on sign language, which is their own culture and is a beautiful thing."
Perry soon becomes inspired to create an artwork featuring the Deaf people he has met while holding Lichy's punk-style hearing aid covers, which, to Perry, say: "I am deaf and I am proud."
The programme shows how strongly Deaf identity is felt, even to the exclusion of other cultural identities, when Garfield, who directs plays in sign language, explains to her Jewish mother that her Deaf identity outweighs her sense of Jewish identity. "Deaf culture is my religion," she tells her. Earlier in the programme she has said "without sign language, I would be dead."
So to all those doubters out there, people who think just because they can't see something, just because they've never come into contact with it, that it doesn't exist - Deaf culture, and the Deaf community are very much alive.
Tonight, on Channel 4, you've got a chance to access our world for just a few minutes. Why not tune in? As a famous movie quote says, this could be the beginning of a beautiful friendship.
Who Are You? is on Channel 4 tonight at 10pm.
Charlie Swinbourne is the editor of the world's most popular deaf blog, the Limping Chicken, as well as being a journalist and an award-winning scriptwriter. Find out more about him on his personal site.