The biggest thing that seems to bother Brian is "the secret." At the beginning of chapter four it is described as something that is terrifically painful to him. The memory seems to come to him as he passes out after miraculously escaping from the wreck of the plane.
"The memory was like a knife." He had been out riding bikes with his friend when he saw his mom meeting her lover and he had kept the knowledge a secret. His mother didn't know that he knew and his father had no idea. Brian hated the man and the memory because it was (at least in his mind) the reason for his parents splitting up and his life turning upside down. It led to all the pain he remembers, "the breaking and shattering of all the solid things" in his life.
Even at the end of the story, he keeps the secret to himself. He considers it so painful and so terrible that he won't open up about it even after his terrible ordeal.