a job interview, you should imagine that your interviewers are naked. This is supposed to make the whole process seem much less scary, as it drags those rather intimidating-looking people sitting on the other side of the desk down to your level. And Forster appears to be making much the same point, except that he's urgingus to climb down from the exalted level to which we, in our arrogance, so often raise ourselves.

Without doing much research on this quote as far as its context, it seems that Forster is making a comment about his place in the world and a reminder to remain a humble being.  We are born naked...nothing under which to hide but vulnerable, cold, and perhaps a bit scared.  When we die, although our family's will undoubtedly clothe us for the funeral, those clothes remain on earth.  This is very much like the old adage, "You can't take it with you," in regards to anything earthly.  So, in essence, we leave the world the same way we came in...naked.  "And a very good thing, too, for it reminds me that I'm naked under my shirt whatever its color" makes me think he is reminding himself that the wearing of clothes does not hide who he really is and how he should be for others.  Remain humble, courteous, and treat others as he would have them treat him.  So, ultimately, his nakedness at birth and death reminds him to be the same during life...not to hide behind clothing, wealth, position, success...just be a humble creature on God's earth.