A lot of the "evidence" is really just inferences, or guesses that Pip makes, using what little knowledge he does have. Miss Havisham is the only rich person that he knows, so, his mind logically jumped right to her when he received his money. He didn't know anyone else that knew him that had money--she was the only one. So, by a mere process of elimination, based on his social circles at the time, she was the only logical choice. He didn't know any other rich people.
Another bit of evidence that Pip chalks up in Miss Havisham's favor is that Jaggers is the man in charge of his accounts, and in being the representative for his benefactor. This is significant, because at an earlier occasion, Pip had seen Jaggers at Miss Havisham's house, and was led to believe that Jaggers was employed by Miss Havisham. So naturally, when Pip discovers that Jaggers is the man representing his benefactor, he connects that to Miss Havisham, considering that is the only other place he's seen Jaggers. His lodgings and teachings are also to occur with her distant relations, the Pockets, so he assumes that it is through Miss Havisham that happens also.
A last bit of circumstance that Pip uses to draw conclusions from is that Miss Havisham, from the beginning, wanted him to love Estella. She made that very clear in her rather demented way. He assumed, naturally, that she was giving him money to make him worthy for her. Since they were children, she put the two of them together, and seemed determined to make them a pair.
These bits of information lead Pip to conclude that Miss Havisham is his benfactor. The story will reveal whether he was correct or not. Good luck!