Before the Industrial Revolution, being a weaver was a solitary and isolated occupation. Furthermore, without human contact, people often develop odd ways to compensate for their loss of socialization.
In the time of the setting of Eliot's novel, also, people were anything but peripatetic; instead, they stayed close to where their parents lived.
To the peasants of old times, the world outside their own direct experience was a region of vagueness and mystery. . . a settler, if he came from distant parts, hardly ever ceased to be viewed with a remnant of distrust (Chapter 1).
Weavers were sometimes not considered members of the community. At any rate, they often lived outside a village and among superstitious people, and the "questionable sound of the loom" held a "half-fearful fascination" for boys of the village, who would peep into the window of the weaver and taunt him (Chapter 1).
Because of the repetitious nature of this indoor occupation, the weaver seemed odd compared to those who lived an agrarian life. His life reduced itself to "the mere function of weaving" (Chapter 1), and he appeared pale and strange in comparison to the men who worked outdoors.
Knowing, then, that they were viewed with some suspicion, weavers may have tended to become more introverted, talking little to those to whom they made deliveries of their linen and being mistrustful of other people.