The name "third" on Earth is automatically negative.  Even if a third makes it into battle school (the purpose of the government allowing a 3rd child to be born into one family), he is still considered an outcast among his peers, as well as an outcast in his family.  He was permitted to be born, which is a reminder to his older siblings that they didn't make it.  And in Ender's case, he is a constant reminder of the religious lives they left behind when they chose to comply with IF orders governing the number of children they could have.  Both came from deeply religious families which praised having several children.

Consider, in chapters 1-3, Ender is still on Earth.  He has not yet been selected for Battle School.  His monitor comes out and there is no immediate reaction from the IF officials to come get him.  Therefore, Ender's status as a "3rd" (which essentially means he was permitted by the government to be born with the hope that he would meet the criteria his brother and sister did not meet to allow him into Battle School), is a negative thing.  For the few short hours (what, a day and a half?) that he goes without the monitor, he fears as well as hears from Peter that he is worthless.  The idea of living the rest of his life, on Earth, as a 3rd, would be the constant reminder that he is a failure.