First of all, this is not a phrase, it's an idiom with SOLID basis.
Secondly, why would this even have the power to annoy anyone? It's just a very useful idiom, why would this cause anything bad? What about things like
"Kys"? Wouldn't that be way worse? How about "You are a failed abortion"?
Come on, this pudding-thing is a piece of cake, easy as pie.