The speaker of the poem associates with the Chaplinesque people he describes: childlike outcasts who love the world despite the scorn they receive from society. He says,

We make our meek adjustments,
Contented with such random consolations
As the wind deposits . . .

He claims that he and people like him make minor changes to themselves so that they can get by in a world where they are alienated, and they content themselves with small moments of happiness whenever and wherever they find them. The speaker continues,

For we can still love the world, who find
A famished kitten on the step, and know
Recesses for it from the fury of the street.

Further, even though the world does not love them, they still find a way to love the world and to take care of the small, defenseless, and starving kitten who has no home—even though they perhaps have no home of their own and no one who shows them the same kindness. They are able to exercise a measure of humanity that others who are better off seem unable to muster.

These Chaplinesque individuals, society's vagrants, "can evade" those people who would judge or condemn them—those people who are likely better off financially and socially.

. . . and all else but the heart:
What blame to us if the heart live on.

Individuals like the speaker can avoid the materialistic "others" who value things and people based only on material benefit. These individuals can evade, the speaker says, just about anything, except for the feelings of their own hearts. They cannot help themselves from feeling, and they must follow their hearts where they lead, including saving lost kittens "in the wilderness." They can also find laughter and joy in places that the numberless horde that judges them cannot. They can "make / A grail of laughter of an empty ash can . . ." They can locate something holy—happiness—in the smallest and most mundane of objects.