In this poem, the speaker starts out by looking into his mirror and seeing how thin his body has become.  He is clearly getting older and his muscles are getting smaller.  What he is saying in the line you cite is that he wishes his heart would get thin too -- that his heart would wither like his muscles and he would not care about things anymore.

In the rest of the poem, we see that he is sad about people who no longer care about him -- the "hearts grown cold to me."  He wishes he did not care what they think, but he still does.