I am not sure what you mean by "not appropriate," but I can explain how the poem establishes the comparison and what the intent of the speaker is.
The poem opens with the question, "shall I compare thee to a summer's day," but the rest of the thought in the next line then states that the person is MORE lovely and MORE temperate. This means that the person is BETTER than a summer's day. That is probably not what the reader first expects -- after all, a typical summer's day is beautiful and lovely.
The speaker then goes on for several lines talking about all of the problems with summer. For example, he talks about the rough winds of May, the short time of a the summer season, the fact that sometimes the sun is too hot, that sometimes clouds cover the sun, and that, in general, all good things eventually fade and end. This seems like a rather bleak picture of summer! But it sets up the point that the speaker is ultimately making the final part of the poem. He next says, "But thy eternal summer shall not fade." That is where the speaker makes the comparison really work in favor of the the person he is speaking to. This person is better than summer because he or she will not fade or be blemished -- he or she will live forever by the fact that this poem is written for him/her and about him/her, and this poem will live forever. It is true when you consider that the person in the poem is still being discussed as better than a summer's day more than 400 years later!