Thomas closed his eyes and this story came to him: "We are all given one thing by which our lives are measured, one determination. Mine are the stories which can change or not change the world. It doesn't matter which as long as I continue to tell the stories.
The text makes it fairly clear why they cannot be friends, though one always feels there must be some additional reason. Thomas says to Victor that he understands things between them will continue as the were before, that Victor's friends would cause him too much grief if Victor and Thomas were seen to be friends. So the surface reason is that since Thomas is ostracized in the community because of his stories, which are actually prophesies, Victor will not opt to suffer the same ostracism by befriending him.
Victor knew that Thomas would remain the crazy storyteller who talked to dogs and cars, who listened to the wind and pine trees. Victor knew that he couldn't really be friends with Thomas, ....
"I know how it is," Thomas said. "I know you ain't going to treat me any better than you did before. I know your friends would give you too much [grief] about it."
A deeper reason has to do with why Thomas's stories are all old meaningless repeats of stories. When they were young, Thomas told the story to Victor of Victor's father leaving home and never coming back. When it came true, Victor's eventual reaction was to beat Thomas up.
It was on that night that Thomas's stories dried up leaving him with only old dead stories. One reason that Victor and Thomas can't be friends after returning is that the breach the truth caused between then--caused by Victor's reaction--was too great to overcome. Nonetheless, Victor and Thomas are reconciled by the promise exchanged as they part because when Thomas goes in his house and closes his eyes, he has a new story given to him and released in his mind.
"[When] I'm telling a story somewhere, why don't you stop and listen?"
Thomas went into his house, closed the door behind him, and heard a new story come to him in the silence afterwards.