Like many truisms in our culture, the "truth" of this one depends on the circumstance. We have to understand, first, that we live in what scholar and cultural critic Henry Giroux calls a "culture of cruelty" in the United States. In such a culture, people are continually blamed for their own problems when, in fact, many of the problems people are face are systemic.
I would say that, while a "correct" interpretation of this quote is probably "you are responsible for your own self-esteem and your own experience," that answer, in fact, is not entirely truthful. It depends on the power differential between you and the person or group trying to make you feel inferior. A parent, for example, can very easily make a young child feel terrible about himself without his consent, simply because the child is too young to fight back against the adult's idea of his worth. The child is learning to define his self-worth against the approval of the parent—and who is he to say the parent is wrong?
Likewise, if a powerful group that controls resources tells you that you are worthless and, worse, denies you access to the resources that define worth in your culture, you are likely to have a hard time not feeling inferior, especially if the message is repeated over and over in small ways (this is called micro-aggression).
On the other hand, if you are among people who are truly your equals or over whom you have power, then, yes, they cannot make you feel inferior without your consent.
It's important to always put statements in the context of power. No society is completely flat: there are always some people with more power than others. Therefore, it is best simply to acknowledge feelings of inferiority that might arise in some contexts. Don't feel guilty or bad about yourself for having these feelings, or waste energy denying them. You can then try to move on with your life and find ways to counterattack the negativity.