The primary symbol in the poem is the night, which comes to represent or contain (or possibly cause) all the action in the poem: the poet's memory of his lover, their love, his imperfect understanding of her, his sense of loss, and, as referenced in the title, his ability to write. Neruda conflates the "night" with the "night sky" or the "endless sky"—these can be understood as roughly equivalent in meaning. His example of the "saddest line" combines these elements (the night, the night sky, distance, coldness):
The night is starry
and the stars are blue and shiver in the distance.
These ideas become a reference point for understanding his feelings about his lover.
Take, for instance, the juxtaposition of the "sad line" "The night wind revolves in the sky and sings" with "I loved her, and sometimes she loved me too." The connection is emotive and suggestive, rather than explicit. The hidden nature of the night wind, unseen but heard, can be seen as a kind of natural counterpart to the inconsistent and ephemeral nature of his love—she loved him "sometimes," suggesting perhaps her affection varied like the wind, or that the incompleteness of their love was like hidden and forlorn, like the wind.
There is a similar pairing a few lines later, when Neruda juxtaposes these lines:
I kissed her again and again under the endless sky
She loved me, sometimes I loved her too.
The endless sky, implicitly a night sky, is contrasted with the inconsistency of their love, this time on his part: his kisses (perhaps endless, like the sky?) are nevertheless only an indication that "sometimes" he loved her.
This kind of comparison is more fully realizes in the line "To hear the immense night, still more immense without her." Neruda connects the night, "immense" because it covers the world in darkness, with the absence of his lover. His longing for her, which includes his memory of their incompleteness, only adds to the darkness of the night.