The speaker of the poem represents humankind and contemplates man's destiny and death. The "rude forefathers" also symbolize humanity as well. All human beings are going to die no matter how rich they are or how famous they are while living. The speaker then divides humans into categories based on their characteristics and attitudes. He says "Ambition" and "Grandeur" shouldn't think less of poor people because they didn't accomplish great things. "Pride" and "Memory" have no right to ignore or forget them, and "Honor" and "Flattery" will no longer be useful to the rich once they are dead. The speaker, who is educated, doesn't even consider "Knowledge" to be a factor in making someone a good person.
The speaker also speculates what many of the poor could have accomplished if they had been given the opportunity to do so. They too could have been a great leader or a great poet if given the chance. But then the speaker also says that even though the poor accomplished nothing of greatness, they are probably morally superior to the wealthy and the famous.