Silas Marner is looked upon as an alien by the people of Raveloe, basically, because that is precisely what he is.
He is an alien first and foremost because he is not from Raveloe. Silas is from the town of Lantern Yard, from which he left with a shady reputation caused by the betrayal of his former best friend.
Second, he is an alien precisely because, like the word entails, Silas's anger, frustration and resentment from the incidents in Lantern Yard prompted him to live alienated from the people of Raveloe. Silas used his loom to get away from his reality, and this also kept him from socializing more than what he is interested in.
Third, Silas Marner is meant to be also ethnically alienated. Although George Eliot does not directly propose that Silas is a Jewish man, it is the conclusion of plenty of scholars in literature, such as the Modern Language Review. In fact, this journal goes as far as citing unintended racism as the rationale that George Eliot used to depict Silas as stingy, weird-looking, not very well-liked, and antisocial. Read "George Eliot and racism: how should one read 'The Modern Hep! Hep! Hep!'" This is a critical essay published in The Modern Language Review.
Finally, Silas's appearance was strange. He had unnaturally big eyes and eyebrows that would scare people. He had penchants and eccentricities. His behaviors, tendencies, and that strange knowledge of alternative medicine which actually cured people from ailments give him an even eerier sense of alienation. These are some of many reasons why he would have been considered an "alien" among the people of Raveloe.