if you term someone telling you to "fuck off" or "fuck you" violence, you're cheapening the meaning of that term and harming those who have suffered from actual violence
you're also handing a convenient tool to our enemies who will point to instances of some rando "bernie bros" telling some neoliberal ghoul to fuck off and say "look! that's violence!"
dehumanizing language like calling someone the n-word is a form of psychological violence: but again, if you add saying "fuck you" to that category you're cheapening it, and harming those victimized by such violent rhetoric
another rather bad effect is that it helps inoculate republicans who bear the brunt of similar language and can draw out the "that's leftist violence!" card. there are plenty of alternative adjectives one can use that doesn't have all these negative effects, like uncomradely
some more alternatives one can use: needlessly abrasive, toxic, meanspirited, uncharitable, malicious, callous, vile, etc. they are all perfectly adequate to convey your point without weakening and undermining the normative force that (rightly) resides behind "violence"
on the one hand the overly expansive use of "violence" is bad, but also an overly narrow one, as is the norm in the dominant discourse to obfuscate the systemic violence endemic to capitalism. I discuss that in more detail in this thread: https://twitter.com/zei_squirrel/status/1274830396870602752