The experiment to whom you are referring is a breakthrough surgical procedure done in the brains of mentally disabled people, that is supposed to make them increase their intelligence and thus make them smarter.
A mouse named Algernon was the first subject used for this experiment, with success. Charlie would be the first human subject to be put under it.
If you are asking what was it that happened to make the experiment fail is that Nemur and Strauss, the project designers had a problem with their theory of intelligence-enhancing and it was Charlie himself who found that out after reaching more than 185 in his IQ. Apparently the two did not think of the long or short lasting effect of the procedure and, as Algernon showed, the span of intelligence is limited and reverses itself to the beginning.