As I understand it, what the speaker means is that the object of the poem ("thou") will come to be one with time because these lines (the poem) are eternal. So what he is saying is that the object of the poem will never die any more than time will die.
This goes with the prediction in the lines before. This is because the eternal life that the object will have (because of this poem) will ensure that she does not die and that her beauty will never grow old. Because of being in this poem, the object will live forever, just as lines 9 through 11 predict.