The Invisible Man decides to live underground because he feels betrayed by the Brotherhood and does not think he will be safe anywhere above ground.

Chapter 25 finds The Invisible Man in a difficult situation.  He is drunk, and the Brotherhood meeting did not go well.  His neighborhood is rioting, and he fears the Brotherhood is to blame.   He feels he has been misled. He decides to cut ties with them, and just tries to survive. 

Not knowing where to go, the Invisible Man decides to run to his landlady Mary's.  He has her antique bank, because he dropped it and it broke.  He does not want to disappoint her, so he brings it with him.

While he is running away, he falls into an open manhole.  The manhole cover is replaced, and he finally has peace and quiet.  As he sleeps, he has nightmares about the riots and his supposed friends.

In the epilogue, the Invisible Man explains how he felt.

“I was never more hated than when I tried to be honest. Or when, even as just now I've tried to articulate exactly what I felt to be the truth. No one was satisfied.” (Epilogue)

Eventually, the Invisible Man decides that he does not owe anyone his allegiance.  People are all using him, and no one trusts him.  He will stay underground, literally or at least figuratively, until he figures out what he can really do with his life.