When Charlie is mentally handicapped, he calls Frank and Joe his best friends. He doesn't realize that when they laugh, they are laughing at him, not with him. He doesn't understand that they are making him the cruel butt of their jokes. For example, once they have Charlie show some girls at a bar after work how he uses a mop to clean out the toilets. They also get him drunk and abandon him to get home on his own. Nevertheless, he still believes they like him.
As he gains intelligence, Charlie realizes his co-workers, including Joe and Frank, seem afraid of him and are not so friendly anymore. Finally, Charlie recognizes that Frank and Joe used to laugh at him and never were his real friends. He feels foolish for being deceived by them. He writes:
It's a funny thing I never knew that Joe and Frank and the others liked to have me around all the time to make fun of me. Now I know what it means when they say "to pull a Charlie Gordon."