Mifflin Wistar Gibbs was born in Philadelphia, Pennsylvania. He was the eldest of four children, and as a result had to go to work very early on to support his family after his father died. He was a carpenter by trade, and learned this skill through an apprenticeship.
He had no formal schooling of note, but did read voraciously in his spare time, and could be considered self-educated. He learned about oratory and debate while attending lectures at the Philadelphia Colored People's Library. There, he met Frederick Douglass, who took the youngster under his wing and got him involved in the abolition movement. This association exposed him to some of the brightest African-American thinkers of the age, and did much to help him develope the skills that would later benefit him in San Francisco, Ohio and Arkansas. He also had dealings with the Underground Railroad, though the scope of them are limited.