The speaker says that fidelity or faithfulness towards the dead fades. People don't stay faithful to the dead. They move on.

The poem opens with the speaker describing the rocks and flowers and animals hiding his beloved's grave. He says that when she was first buried, those who loved her thought they would never be happy again and would mourn her passing forever:

They thought their hearts could ne'er recall
The light of joy again.

Yet, soon enough their hearts begin to heal and life continues:

But where is all their anguish now,
And where are all their tears?

But, the poem also says, those who are dead and in the grave are not faithful either. It is a two-way street of unfaithfulness:

The dweller in the land of death
Is changed and careless too.

The dead are asleep and won't respond to grieving, so it doesn't much matter if one shows sadness or not. Death and living separate people and make fidelity almost impossible—which makes Catherine and Heathcliff's faithfulness to each other after Catherine's death in Bronte's classic novel Wuthering Heights all the more remarkable.