If you think of a summer's day as beautiful and perfect, then the comparison seems like it will be very positive, especially when you consider that the speaker of the poem states in the second line that the person is "more lovely and more temperate." That seems like a very high compliment. What is odd about the poem then is that next six lines make mention of all of the negative things about summer. Sometimes it is cloudy, hot, windy, and the season doesn't last long. With that established, then the speaker explains that while all pretty things eventually fade and die, the person he is speaking of will not fade and die because his or her beauty will be captured forever within the lines of the poem. I don't know if the comparison is inappropriate, but the poem does have a surprising intention as revealed in the last six lines of the poem.