The main reason Silas moved to Raveloe has to do with his "friend" Wiliam Dane. In the town where he lived before, he and Dane were religious buddies and they attended the same church. Dane made it seem that they were great friends, but at one point he decided that he had had enough of pretending.
So he betrayed Silas on every level possible, he had Silas blamed for stealing money from the church (something that Dane arranged) and because of the social fall out over this, Silas lost his fiancee and Dane ended up swooping in to rescue her and married her himself.
Because of this betrayal, Silas was far too angry and bitter to live in the same town so he moved to Raveloe to escape the memories and the social stigma he faces.