This is a very interesting question and, at the root of it, is the difference between the two parts of this novel. In short, the first part of the novel precisely recounts Tony’s memories and the second part of the novel explores Tony’s present. Because you are primarily interested in the memory of Tony, let us take a look at this part of the novel in full.
Tony’s memories begin (for the reader) when he is finally retired and living on his own. The memories begin in the decade of the 1960s when Tony was in school and friends with three students who, like himself, are quite self-absorbed and arrogant and intelligent. Probably the most important two of the four are Tony and Adrian (who happens to be the one with the most smarts). The memory gets more specific because the end of a particular school day is recounted: one of their fellow classmates commits suicide (by hanging). The rumor is that the young man impregnated a female student. The memory continues as Tony, Adrian, and the other friends talk about this suicide at the school. They are disturbed in not knowing the actual facts of the situation.
A bit later in the story, the four friends go to college. Specifically, Tony goes to Bristol and Adrian goes to Cambridge. The memories now switch to Tony’s college life where he acquires his own girlfriend named Veronica and has many awkward moments with her (especially when Tony visits her family). Tony and Veronica eventually drift apart and, when Tony is a senior, he gets a cryptic letter from Adrian that now Adrian is dating Tony’s past girlfriend.
To further the disturbing aspects of the love triangle, Tony replies to the letter only to find out a few months in the future that Adrian has also killed himself (similar to the schoolmate discussed when they were young). Adrian also leaves a note saying that people should consider it a duty to figure out the meaning of life and, if needed, renounce that life. Tony is fairly impressed with these philosophical musings of Adrian before his death.
Tony, himself, explores the importance of memory in what I feel to be the most imperative quotation in the book:
How often do we tell our own life story? How often do we adjust, embellish, make sly cuts? And the longer life goes on, the fewer are those around to challenge our account, to remind us that our life is not our life, merely the story we have told about our life. Told to others, but—mainly—to ourselves.
Now, what does this quotation have to do with the reality revealed in the second part of the book? Let us explore it. First of all, we find out that Adrian, at one time, impregnated Veronica’s mom leading to the birth of Veronica’s mentally ill brother. We are never sure, as readers, whether this had something to do with Adrian’s suicide; however, it does relate to the earlier childhood story. Next, we find out that Veronica’s mother has betrayed many (both in the birth of an illegitimate child and in gossiping about her own daughter). Further, Tony thinks that Veronica is a victim of abuse by some male member of the family (her father or brother, perhaps?); however, the reader knows that it is most likely Veronica’s mother that did the abusing. This presents a definite depiction of dramatic irony. This is where our actual knowledge of reality ends.
From here on, the reader is unsure of the reality. We are left wondering if Veronica has provided her mother with lovers over the years (all of which were her former beaus). We are left wondering if this is why Tony is left alone with Veronica’s mother on a mysterious “walk” taken by Veronica and dad. We are left wondering why Veronica’s mother leaves a huge sum of money to Tony and, worse, left wondering why Veronica calls it “blood money.” There are no answers to these questions. It is the beauty of a mystery and/or tragedy similar to the works of Henry James (and specifically his Turn of the Screw).
In conclusion, even though your question implies that you are only interested in Tony’s memories, something more should be said about Tony’s present. Tony has basically spent forty uneventful years after the memories of the first part of his life. It is only when Tony hears from Veronica’s mother, given two important letters and given two thousand British pounds that Tony begins to re-evaluate his memories from the past. It is the revisiting of these memories that comprise the second two thirds of the book. Further, it is the comparison between the memories and the reality that make Barnes’ work reminiscent of other writers such as Henry James.