The lines in the poem 'What lips my lips have kissed' which refer to unanswered requests seem to transcend Time, and may have to do with the poet's treatment of it. Perhaps the reason the voices will never get a reply is that they belong to people long dead - the poet cannot reply because she exists in different time-scale - that of the present. Groups of people that may fit into this category could be soldiers that died, losing their youth in War, or loves who died prematurely through illness. The voice would also fit the persona of a lady who has lived to a ripe old age and has long outlived her peers. She cannot speak to them when they visit her from beyond the grave because she has not joined them there yet - although the voices don't know that.
Lines 3, 4, and 5 bring with them the idea that something is calling out for an answer that is never to be heard. If we examine the opening lines, there is a feeling of forlornness that is present because the experiences of love and companionship which used to be felt are no longer present. The construction of one's memory is all that remains, and these memories are craving out for some type of physical recreation. The employment of the rain "tapping" against the window seems to being with it a connotation that someone or something is striving to receive an answer. Perhaps, this could be the memories of the speaker that yearn to bring back what was once lost. The line "listen for a reply" indicates that something is asked or stated without an audience. The "ghosts" might also be those memories of companionship that might be yearning for some type of absolution or resolution, and elements that strike pain at the heart of the speaker.
In my opinion, this poem is about loss, specifically, the loss of possiblities that occurs in all people as we age. It talks of the memories of times and people and feelings past that can never come again. I believe the lines you cite speak to this idea.
In these lines, the ghosts of these past things are in the rain. They are tapping at her window, asking her to remember them, waiting for a reply.
But she can not completely reply to them because they are part of the past and that "summer" that they were part of sings in her no more.
So, your three lines are specifically referring to the ghosts and memories of past feelings and people and events. They are asking to be with her again, but she can not be with them because the part of her that knew them is gone forever.