People take a lot of pride in how hard they work. And so telling someone they have privilege largely amounts to chipping at their sense of pride.
Even if you don't mean it that way, it is bound to be perceived that way much of the time. And, conversely, claiming that you have less privilege than others is a way of bolstering your own pride.
Ultimately, it is good for people to be aware of their privileges. I think the real problem is that we don't do it thoroughly enough. Because if we really go down the rabbit hole (which we should), we'll realize everybody's success in life is dictated EXACTLY by their privilege.
We don't control where we come from, how smart we are, who our parents are, what opportunities we are given, or what traumas happen to us. We don't even really control how much grit or drive we have. Because, after all, the fuel for both grit and drive is hope. And because whether we respond to problems with hope or despair depends on what our life has taught us is the more appropriate response. And at an even deeper level, grit and drive are still just results of our brain and DNA, which we don't control. So in the end, everything we have, everything we are, and everything we've achieved, are all the result of things we don't control. They are all the result of our relative privilege.
So what's the solution? I think the solution is to stop chasing pride. We don't really deserve to feel proud anyway, because we don't control our lives. And we really don't need it. All we need is to do our best, try to get along, and be as happy as we can with what we have.