I never thought of it that way, but I can see how people would use it as a "I'm ending the conversation here because I want to cut you off and don't consider you important enough to continue the conversation with."

It bothers me because it's mangling of regular English.

But you're right. If you insult someone it is going to generate a response to the insult rather than whatever point was trying to be made inside the insult.