The boys are enrolled in an exclusive school. Many of them cite their families as proof of their belonging. Piggy only references his mother which gives the impression of a "mama's boy". In addition, he is easily rattled by all events. The final reason is his lack of health. The other boys are relatively rugged and can handle the rigor of the island, whereas Piggy has glasses, asthma and a weight issue. He has no peer who shares any of the problems he has therefore making him an outsider.
Besides Piggy's lack of physical prowess, his upbringing also affects his reputation. While the other boys discuss their fathers, Piggy reflects on the female members of his clan. He appears to respect feminine values which harms his reputation on the island.
Another way of framing the same response is by considering the centrality of masculinity to Golding's novel. Those boys most masculine are most admired, while those more timid are not. The aggressive children become leaders, the more passive children become followers. Piggy is not aggressive but has other qualities of leadership. Unable to categorize him in their very narrow concept of masculinity, the boys ostracize Piggy for his "powers" exceed--are outside--those they understand.
Piggy is an outsider for many reasons. His health is one. Boys like Ralph are strong and healthy. They are confident in their ability to act. By contrast, Piggy is fat and has asthma. He wears glasses, and his mom has been very protective of him, making him hesitant to act. However, he's also an outsider because he is fundamentally an intellectual. He lives in his mind, while the other boys in their bodies or the community. That makes him seem weird to them.