According to one definition in the Webster dictionary, a slave is "a person who is strongly influenced and controlled by something."
In the story, Brother is remembering his brother, Doodle, and how he dies. Brother mentions his pride two times in the story. The first instance of pride is when Brother says, "I did not know then that pride is a wonderful, terrible thing, a seed that bears two vines, life and death." In this quote, Brother understands that pride can be a positive or negative thing. Like a seed, it grows and grows and can have two different outcomes--one that respects life, and one that causes death. As he reminisces about the past, he realizes the role his pride played in the death of Doodle.
The second instance of when Brother mentions pride is when he cries after Doodle shows his parents that he can walk. It is here that he mentions that he is a slave to pride for pushing Doodle to be normal, "They did not know that I did it for myself, that pride, whose slave I was, spoke to me louder than all their voices, and that Doodle walked only because I was ashamed of having a crippled brother."
When Brother says, "their voices," he may be talking about the seven deadly sins. Pride is one of them. He is feeling guilty as an older man who is facing his own vices and flaws.
The fact that the story is a flashback and Brother has had time to analyze and reflect upon his actions tells us that he was a slave to pride, but he didn't know it at the time. Pride is a concept a six year old child wouldn’t have any experience with nor be able to articulate into words. Looking back, however, Brother realizes how his own ego and shame for having a crippled brother caused him to do the things he did. Brother has grown and matured, but his pride will continue to haunt him throughout his life as he realizes the control pride had over him.