What we have to fear from failing to conform is being ostracized by our peers.
Anyone who has ever been in high school should understand this idea. In high school (as in other small societies where people generally all know one another to some extent), there is intense pressure to conform. People who act different are often picked on and harassed. Their lives can become very miserable because people condemn them for being different than everyone else.
In our broader society, we do not have to fear this as much. However, it is still possible to be harmed if we fail to conform. We can, for example, lose our job if we are unable to conform to the norms of the places in which we work.