It is the "masters and lords" of the world who are responsible for the wretched condition of the man with the hoe and millions of others like him. It is their insatiable greed which has taken this thing that God made to exercise dominion over land and sea and turned him into little more than a slave worn down by years of endless, back-breaking toil.

The latter part of the poem represents an extended plea by the speaker to these masters and lords to return the working man back to his original godly state. Otherwise they will be guilty in the eyes of history of ruining God's most remarkable creation and will give back to the Almighty nothing more than a "monstrous thing distorted and soul-quenched".

In Markham's "The Man with a Hoe" it is wealthy and privileged people, the "masters" and "rulers" of the world, who are responsible for the degraded plight of the working man. The narrator addresses this privileged group directly with a series of questions, such as:

O masters, lords and rulers in all lands,
Is this the handiwork you give to God,
This monstrous thing distorted and soul-quenched?

The narrator implies that the wealthy have distorted God's creation by cruelly overworking the poor. The speaker also asks the wealthy what they will do to straighten the bent backs of such men so that they can be rehumanized and gaze up at the stars as humans are supposed to, and what they will do when such oppressed people rise up in rebellion against their lot.

This poem, written in 1898, when the wealth divide was at a high point, condemns those who live well at the expense of those who toil endlessly, saying the wealthy rob the humanity of the poor, turning them into little more than animals.

"The Man with the Hoe" by Edwin Markham was triggered by a painting of the same name, by Jean-Francois Millet, and I have included a link to the painting, so you can see what a vivid representation of the painting this poem is. However, the poem has far deeper meaning than as a description of the painting.  

The man with the hoe represents all the working men who have been burdened, abused, and misused by the power and greed of the wealthy, over thousands of years, from serfs working for lords, to tenant farmers, to migrant workers, to factory workers today.  There are several references to show us who is responsible in the poem.  The poem refers to "the world's blind greed" (line 19), the working man a  protest "to the Powers that made the world" (line 31), the "masters, lords and rulers in all lands" (33).  

The poem is not a representation of the nobility in work, which the wealthy and powerful would have us believe should be sufficient, but rather, shows us that work such as men like this have endured has stolen their souls, taken all the joy out of life, and...