I think you're conflating two different issues. You are worried about social epistemology -- i.e. how can a society arrive at collective knowledge -- which is fine but it's not the same thing as asking whether the words are violent.
I wonder if you'd feel differently if you were on the receiving end. I've heard enough spoken and written testimonials from black people along the lines of, "I've been in fights and had my ass whipped, but nothing hurt me more than when they called me a n*" or "told me I couldn't do it because I was black" or "told I was a criminal because I was black." I've heard the same thing from gay people.
If Alice calls Bob a f*, knowing that it will deeply injure Bob, then how is that not as violent as hitting him in the face?
If a woman is raped, and someone tells her that she had it coming because she was dressing too sexy, how is that not as violent as slapping her for being a silly woman? It's perhaps not as bad as being raped, but I've heard and seen plenty of accounts of women talking about how being dismissed can be so hurtful. This is one reason, of course, that raped women often don't go to the police, or don't press charges. Not the only reason, but it's in the mix.
Trans people commit suicide with abnormally high frequency. Knowing that, if someone with a microphone and an audience demeans trans people, they will have knowingly contributed to their deaths. Why is that not violence?