Axel and Mary Jordache's lives are marred by their relentless and fruitless attempts to gain social respectability and financial independence. Mary marries Axel to escape the poverty of being an orphan, but she is repulsed by Axel's intimate desires. She responds by withdrawing emotionally and threatening suicide. In turn, Axel reacts to her coldness with physical violence. Their attempts to run a business are as unsuccessful as their marriage. After two decades of long hours and minimal pay, their bakery remains unstable. When an urban renewal project threatens their building, Axel loses hope and takes his own life. Mary becomes a recluse, only approachable by her youngest son.

Tom, Gretchen, and Rudolph Jordache grapple with the burdens left by their parents. Each seems destined for a life as troubled as their parents'. Tom inherits Axel's temper, desire, and inclination toward violence. He frequently gets into trouble until a boxing career offers an outlet for his intense passions and a way to make a living.

Gretchen inherits her mother's yearning for respectability and her father's intense nature. She uses these traits to advance using her appearance and allure, first as the mistress of a wealthy man and later as an actress.

Rudolph suppresses his violent and sexual instincts, channeling his energy into gaining social acceptance. He attends college and becomes a manager at a local department store, avoiding romantic entanglements to focus on his career.

If the Jordaches are the "poor men" of the title, Theodore Boylan represents the "rich man." Boylan's wealth plays a pivotal role in the lives of Gretchen, Tom, and Rudolph. Gretchen's fascination with Boylan's lifestyle leads her to become his mistress. Tom's resentment of Boylan's enticing wealth drives him to a violent act. Rudolph's ambition to emulate Boylan (thus securing financial stability for himself and Mary) makes him the rich man's protégé.

The novel follows the Jordache children's lives from 1945 to 1968, filled with minor characters they love, hate, compete with, or desire. Despite hardships, the Jordache children build lives filled with affection and success their parents could not have imagined. Tom, Gretchen, and Rudolph all discover true love amidst the heartbreak of failed relationships. Tom finds his calling as a sloop captain, fighting the sea rather than other men. Gretchen thrives as a film editor in Hollywood. Rudolph becomes wealthy and is elected mayor of their hometown. However, success comes with losses: Gretchen's husband dies in an accident, Rudolph's political career ends after a mishandled student riot at the local college, and Tom dies protecting Rudolph's wife.

For Axel and Mary Jordache, life was a constant struggle of setbacks. For their children, life progresses forward.