This means, "You can't take it with you."  It does not matter how wealthy, successful, or famous one has been on earth.  All the money and prestige in the world will be useless on Judgment Day (if indeed, the speaker suggests, such a fate awaits us.  The only certainty is death.) Or, think of the mummified Pharoahs of Egypt, so sure their goods would follow them beyond the grave. 

As for Earth, whatever accumulations we have managed during our brief lives is not permananent.  Eventually, all will be dispersed and our fleeting time here will be, if we are lucky, unconvered in shards by zealous archeaologists.  But for most of us, the "remains of the day" will be shortlived, indeed.