Charlie is warned from the beginning (or at least the beginning of his coherent thought processes) that his intellectual growth will "outstrip" his emotional growth. This makes sense because emotional growth is built of experiences of interacting with other people. For neurotypical adults, their emotional growth and intellectual growth are generally around the same socially accepted pace. However, Charlie is undergoing such face-paced intellectual growth that he will be experiencing confusion and tension in many social situations.
Both Alice and Dr. Strauss believe that Charlie, as an adult man, begins experiencing sexual and romantic feelings as an adolescent, placing his emotional intelligence disconcertingly lower than his intellectual rate of change. Eventually, this manner of thinking becomes something of a menacing, alienated figure to Charlie, who refers to his emotional wiring as "old Charlie". He slowly learns that emotional problems are incredibly complex and that he still struggles to become a part of human culture, despite his intellectual advances.
Overall, Charlie avoids emotions because they cloud his ability to think clearly. Although he can access the world at an objective, fact-based level, emotions are incredibly confusing to him.