In Chapter 34, Pip starts to realize the difference between the reality of having wealth versus the "great expectations" of those who assume that having wealth means living lavishly and beyond their means.
So now, as an infallible way of making little ease great ease, I began to contract a quantity of debt. I could hardly begin but Herbert must begin too, so he soon followed.
By now, in trying to keep up with his peers, impress others, and act as a benefactor, Pip has started to waste money away and fall into debts. Herbert does the same, and now that debts are creeping in, Pip realizes that, as far as expectations go, there will never be enough money to abide by the habits of the very rich.
My lavish habits led [Herbert's] easy nature into expenses that he could not afford, corrupted the simplicity of his life, and disturbed his peace with anxieties and regrets.
Still immature to his reality, Pip tries other ways to remain in the social hodgepodge that he is expected to be a part of, and this leads to his living the life of an idle dandy.
we put ourselves down for election into a club called The Finches of the Grove: [...] if it were not that the members should dine expensively once a fortnight, to quarrel among themselves as much as possible after dinner, and to cause six waiters to get drunk on the stairs.
That, however, was indeed the mindset of the time of the dandies--the young bachelors who idled away at their respective clubs during the afternoons and then partied away the night. These men inevitably hovered around one another. Now, keep in mind, the behavior that Pip and his cronies display is not unique to this novel. Plenty of Regency and Victorian novels, and the likes of Oscar Wilde, H.H. Munro (Saki), P.G. Wodehouse, G.B. Shaw and a plethora of other authors of the period, would describe the behaviors of the rich London bachelors exactly as Dickens does in Great Expectations.
All this being said, it is evident that Chapter 34 discloses a lot of information about Pip and his new lifestyle, which is why he says the quote in question:
I used to think, with a weariness on my spirits, that I should have been happier and better if I had never seen Miss Havisham's face, and had risen to manhood content to be partners with Joe in the honest old forge.