It is not possible to know the answer to this question with certainty. We can only speculate about the causes of phenomena such as anti-Semitism or racism. I would argue that there are at least two main reasons why the Jews have been singled out for persecution.
One of these reasons is specifically religious. Both Christians and the pagan Romans have been opposed to the Jews at times for religious reasons. With the Christians, the religious reasons start with the fact that the Jews (in a certain time and place) specifically rejected Jesus Christ and his message. It was felt that they caused his death. The Romans disliked the Jews because the Jews refused to even pay lip service to the Roman gods. It was felt that all people needed to perform the proper rituals to honor these gods, regardless of their beliefs. But the Jewish law did not allow this. Therefore, the Romans felt the Jews were endangering the relationship between Roman society and the gods.
The other reason is not religious. Instead, it has to do with society’s love of scapegoating groups of people when problems arise. Societies seem to want to blame some identifiable group when problems arise. In many times and places, the Jews have been available to blame. Since they have often lived apart from the rest of society and have behaved in different ways, they have been an obvious target for this sort of scapegoating. In this view, the Jews have been persecuted because they have been a distinctive and identifiable group that has often lived apart from the rest of society.
While we cannot know for sure, we can generally say that it is likely that these two reasons have caused Jews to be singled out for persecution.