Throughout the novel , Brian is haunted by the memory of his mother's "secret". The secret that continues to invade Brian's memory is the affair that his mother had while still married to his father. Readers are never given the specifics of this affair, but we do know that this is probably the reason Brian's parents are now divorced.
It is possible that the reason the affair has affected Brian so much is because he was witness to it. Brian accidentally stumbles upon his mother kissing another man in a parked car at the mall. The memory of this creeps into Brian's thoughts and dreams unexpectedly.
Yes, it is true that he is haunted by the memory of his mother kissing that man however, he also is hauted by the image of the dead pilot in the plane.