This part of The Time Machine seems to be a satire on the utopian ideas of the time it was written, which concentrated on the elimination of all work and effort from human society, "the machines, our new race of slaves," and the idea that enormous surplus value was being unfairly appropriated from the system by capitalism that could be made available for the benefit of all given some social engineering.
If you read George Orwell's journalism in the period before he wrote Nineteen Eighty-Four, you can see that the unreality of this type of dream was almost an obsession with him. In another of his books, I think it was The Road to Wigan Pier, he characterized the "progressive" thought of his time as a mad dash toward a goal that people hoped never would be reached, because it would be too boring. It is a little bit unfair, given this criticism in The Time Machine, but most of Orwell's criticism takes H.G. Wells as the chief author of "rush to corruption" utopian thought.
It was in reaction to this type of thought, and the feeling that it could not motivate a ruling class, that inspired Orwell to define the aim of the Party in Nineteen Eighty-Four as a desire for power and a pleasure in causing pain to others to affirm that power. He made O'Brien, the Party spokeman, denouce utopianism of the Wells kind and admit that the Party only felt alive when crushing its adversaries -- and thus, the struggle would never end.
A consequence of a civilization that experiences no hardships or wants results in a society of lazy, inept people who don't know how to take care of themselves.
They have little or no intellect, no curiosity and no ambition to learn anything. The Time Traveler is frustrated by this lack of intellectual curiosity as well as disappointed.
I agree that a life of total leisure would make my brain turn to mush. Whenever I have time off from teaching, I find that I get really lazy, I get more done when I'm working. That is why I like teaching Summer School. Too much time off dulls the senses.
The Time Traveler, who is an educated man hoping to find a society of advanced intellectuals ends up in a society where people no longer understand the need for learning. A tragic turn of events in his mind.
The Time Traveller is at first marvelling at what he thinks is an almost Utopian Society. The Eloi are so peaceful and live a simple life without strife. He also notices, though, that the simple life has led to the ruination of the surrounding buildings as well as a total loss of creativity and industry. In this way, he becomes discouraged and even impatient with the Eloi because they seem apathetic, even towards one another. When Weena was drowning in the river, none of the Eloi even seemed to care. The Time Traveller was incredulous that they would not even pay attention to one who was obviously in distress.
I would have to agree that a life of leisure might seem like Heaven for a short time, but it's only a matter of time before ennui and apathy set in. Personally, I would go nuts without something to do! I'm a high school English teacher, and I remember how, before I started teaching summer school, I would go crazy with ten weeks of not going to work!