The Longest Memory tells the story of what slavery was like on a Virginia plantation at the end of the 18th century and beginning of the 19th century. The story opens with Whitechapel discussing his life. Whitechapel is the oldest and most respected slave on the plantation, and he is willing to do anything for his family members who comprise the majority of the plantation workers. He is married to Cook, who was previously raped by the then-overseer, Mr. Sanders Senior. The rape resulted in a pregnancy from which Chapel was born, but Whitechapel raises Chapel as his own son.

As Chapel gets older, he starts to fall in love with the plantation owner’s daughter, Lydia, after she helps him learn how to read and write—both of which are forbidden for slaves. The two of them dream of running away to the North together, and once his mother dies, he decides to put this plan into action. Whitechapel learns about his son’s escape, and after deliberating about it, he decides to tell the master where his son is. Whitechapel knew that if his son was found by a search party, he would be killed, but if he was returned to the plantation, he thought he would be able to spare his son’s life.

Upon his capture, Chapel is returned to the plantation, but the plantation owner, Mr. Whitechapel, is away for the day. He orders Chapel to be locked up and held for his return with plans to punish him—in such a violent manner that it would deter him or any other slave from attempting escape again—but not kill him. However, the overseer, Mr. Sanders Junior, decides to whip him two hundred lashes without Mr. Whitechapel's knowledge. Shortly after, Chapel dies from his injuries, and Whitechapel is blamed for his death because he had been the one to divulge his location.

The rest of the story travels back in time and tells of the events through diary entries and memories. It then includes newspaper articles from 1810. Through each character’s thoughts and memories, the story shows the injustices as well as the political, economic, and social issues that slaves suffered on a regular basis.