Death and the eternity that follows is a common theme for Emily Dickinson. The idea of finding comfort and having someone with whom to connect in the afterlife is certainly suggested in Dickinson's poem "I died for beauty but was scarce," as well as the longstanding poetic tradition that Truth and Beauty are connected. (Think Keats and his "Ode on a Grecian Urn.")
The plot (what happens in the poem) is fairly simple. The speaker of the poem "died for beauty" and has been placed in a tomb; before the speaker is able to fully adjust to her new home, someone else is placed in a tomb next to hers, what Dickinson calls "an adjoining room." He, we discover, "died for truth." They see themselves as being rather the same, and they talk until enough time passes that their mouths are covered by moss and they can no longer speak.
The man who died for truth is the first to describe their relationship; he says that, since beauty and truth
We brethren are....
The speaker, who died for beauty, agrees, saying they are "kinsmen."
These two beings are now connected by death, and they talk quietly to one another between the walls of their tombs until they can no longer speak. In fact, their eternal connection is solidified by the fact that their names are, eventually, forever hidden together behind the moss which grows and covers their names. While they are able, then, they share a camaraderie which lasts for a figurative eternity.
Neither seems to have any regret about dying, any fear of being entombed, or any apparent dread of the prospect of an eternity spent buried and living in the ground. These things do, as your question suggests, "support the idea that contact and connections with other people is integral to a person's sense of belonging." The two of them are like-minded, according to Dickinson, because truth and beauty are one in the same; therefore they have plenty to talk about as they molder in the ground.
And so, as kinsmen met a night,
We talked between the rooms,
Until the moss had reached our lips,
And covered up our names.
They talk until they no longer have the ability to speak.
Of course, the connection between their lack of regret, fear, and their sharing the experience is not explicit; there is no evidence to suggest that they would not have adjusted to their underground, eternal homes if they had been alone, if they had not had a neighbor. Despite that lack of precise causation, Dickinson does seem to be making the point that death (and the eternity that follows) will not be terrifying if we find like-minded people with whom we can share it. Is this connection to another person, and the sense of belonging that creates, integral to eternal contentment? Perhaps. In this poem that contact exists, and neither corpse exhibits regret, fear, or dread.
This optimistic view is, however, tempered by the fact that this comfort (the contact and connections to which you refer) can only be had indirectly, through the walls of a tomb, and only until Truth and Beauty are forced into silence by the encroachment of time, represented by the moss.