If we do not use cultural relativism in such a circumstance, we run the risk of becoming prejudiced against the people of that culture. We will think that there is something wrong with them instead of understanding that their ways are simply different from ours.
For example, I grew up outside the United States in a very different culture. Many people who worked at my father's office would not show up for work consistently. They might stay home on any particular day because they were needed to help (for example) harvest a certain sort of plant for a ceremonial feast the next day. If we do not use cultural relativism, we denounce such people for being lazy and lacking a work ethic. If we use cultural relativism, we realize that they have different cultural values that are not inherently worse than ours. We then try to work with them rather than condemning them.
If we fail to be understanding, we end up seeing others in a bad light. This makes us very much more likely to have bad relations with them since we look down on them.