The easy answer to this question is: because he was. Also, because he acted like one. He looked the part and walked the walk, too.
The more complex answer is at the heart of author George Elliot (nee Mary Ann Evans) who has been at times accused of being anti Semitic in her portrayal of Silas. Did she intend for Silas to be different in order to show the dynamic change that her character undergoes? Did she purposely characterize him in a way that would make him look ridiculous? We can only speculate.
For how was it possible to believe that those large brown protuberant eyes in Silas Marner's pale face really saw nothing very distinctly that was not close to them, and not rather that their dreadful stare could dart cramp, or rickets, or a wry mouth at any boy who happened to be in the rear?
Regardless of his well-established enigmatic and exotic looks , Silas Marner is a man who intended to be alienated from others since he first set foot on Raveloe, after being banished from his hometown of Lantern Yard.
Even so, he was already looked upon strangely by the Lantern Yard crowd that he was a part of. When doing a close reading of the novel, it is easy to conclude that Silas was "marked" by a number of gifts (and a number of curses, as well) that set him aside as almost otherworldly.
...he invited no comer to step across his door-sill, and he never strolled into the village to drink a pint at the Rainbow, or to gossip at the wheelwright's: he sought no man or woman, save for the purposes of his calling, or in order to supply himself with necessaries; and it was soon clear to the Raveloe lasses that he would never urge one of them to accept him against her will...
Silas alienated himself from everyone.
Also: The fact that he had an amazing knowledge for herbal cures, combined with the strange cataleptic fits he would undergo while worshiping back in Lantern Yard certainly put Silas in a category of his own, that is, in comparison to the common folk who lead boring and predictable lives.
All this being said, Silas is simply a complex, dynamic and enigmatic character in every way it is characterized. Eventually all of this will change when his gold is stolen and he had to completely shift his paradigms of life. Plus, becoming a father to the abandoned baby Eppie was also a huge change that infused humanity into him. Silas is special in many ways.