This is a good question. We need to first establish that Elie and Moishe had a good relationship at first. In fact, it is clear that there was a tight bond between the two. However, when Moishe came back, things changed. To be more specific, Moishe changed. Here is what Elie writes:
Moishe was not the same. The joy in his eyes was gone. He no longer sang. He no longer mentioned either God or Kabbalah. He spoke only of what he had seen. But people not only refused to believe his tales, they refused to listen. Some even insinuated that he only wanted their pity, that he was imagining things. Others flatly said that he had gone mad.
We can infer from this statement that most people did not really believe in the horror stories that he told. We can also infer that they did not want to believe in him. So they politely dismissed him or thought him to be mad.
Since Elie was young, he was in a hard situation. On the one hand, he saw how people treated Moishe, Elie must have question whether Moishe was really mad. On the other hand, he know that Moishe was not one to make things up. So he wanted to believe in what he said and still respect him. In the end, he just pitied him. These are the words of Elie:
Even I did not believe him. I often sat with him, after ser- vices, and listened to his tales, trying to understand his grief. But all I felt was pity.
"They think I'm mad," he whispered, and tears, like drops of wax, flowed from his eyes.