I think that Pip's reasons for acting in this way can be summed up by the following sentence from the start of Chapter 47. Here, Pip says that
I had quite determined that it would be a heartless fraud to take more money from my patron in the existing state of my uncertain thoughts and plans.
This is why he sends the pocketbook back without opening it.
So why would it be a "heartless fraud?" I think that the reason is that Pip does not like the fact that his benefactor is a convict and a felon. He had thought all this time that Miss Havisham was the one who was helping him out. But now he comes to find that his money comes from a criminal. This does not fit with Pip's vision of himself as a gentleman.
To me, then, Pip is giving the money back because he does not want to be hypocritical. He does not want to take money from someone that he looks down on. Now that he knows the money comes from a criminal, he does not want it because he is too good for that (this is like how he does not want to acknowledge Joe -- he thinks he is above his former status).
But I had quite determined that it would be a heartless fraud to take more money from my patron in the existing state of my uncertain thoughts and plans. Therefore, I had sent him the unopened pocket-book by Herbert, to hold in his own keeping, and I felt a kind of satisfaction - whether it was a false kind or a true, I hardly know - in not having profited by his generosity since his revelation of himself. Ch.47.
These lines clearly tell us that Pip returned Magwitch's "unopened pocket book" because he did not want to live off Magwitch's money after he had learnt that it was the criminal Magwitch who had been supporting him in London.
All along Pip had imagined that Miss Havisham was his benefactress and that she had spent a lot of money to have him transformed into a gentleman so that he would become a suitable life partner for Estella. So when he realises that it is Magwitch a criminal and not Miss Havisham who is his benefactor his moral scruples prevent him from accepting money from Magwitch and so he returns Magwitch's "unopened pocket book" which contains his money back to him through his friend Herbert.