Chabin is treated differently because he's an albino. Albinism is a congenital disorder characterized by a complete or partial absence of pigmentation in the skin, hair, and eyes.
In some traditional cultures, such as the Haiti culture depicted in Breath, Eyes, Memory, albinos are held to have magical powers. In Sophie's village, Chabin is widely believed to have the spirits on his side. It's said that if someone's chasing him, he can turn them into a snake with one flip of his tongue. He's also believed to be able to see someone's future simply by looking into their eyes.
The villagers are thus quite wary of Chabin. Although, as he's the bélét man, the biggest lottery agent in the village, they regularly approach him in the hopes of transforming their fortunes; they believe he has the power to bring them good luck as well as misfortune.