In chapter 3, Simon comes to visit Daniel on the mountain to let him know that Amalek, his former master, has died and that it is safe for Daniel to come to the village to visit his grandmother and sister. There are several reasons he is hesitant to go back. Although he realizes he should want to see his family, he doesn't, for multiple reasons. First, he knows his family lives in severe poverty, and it would not be pleasant to see that. Second, his sister, Leah, is mentally disabled. She is always cowering in fear and never leaves their house. In addition, Daniel probably feels guilt and regret that he has been unable to help them and that he ran away and has not seen them for five years.
Another reason he does not want to go back to the village is that he believes there is nothing there for him "but the old troubles that had ceased to bother him" while he lived on the mountain. Those troubles include not only the condition of his family members, but also the presence of Roman soldiers in the streets and the constant images of Roman oppression that stirred his memories of his parents' deaths.
The other reason he doesn't want to visit the village is that he has found a purpose on the mountain. His whole life, since his parents' death, has been devoted to throwing off the Roman rule of Israel. He resents anything that will take him away from that goal. While he is at his grandmother's house, watching Leah, he realizes that "everything he cared about and worked for was threatened by that small helpless figure."
Daniel hesitates to visit the village because of his emotions toward his family, because of the bad memories associated with his time there, and because it distracts him from his goal of defeating the Romans.