The literal meaning of this quote is evident on its face value. Gregory is talking about how he was perceived by his teacher at a certain point in his grade school days. He is saying that the teacher simply thought that his lack of success in school was due to his lack of mental ability.
To truly understand what Gregory is saying, however, you need to understand why he is telling us this. He is telling us that he was really doing so badly in school because he was hungry. He was too poor to be able to eat well and so he was constantly distracted by his hunger.
The point that Gregory is trying to make is that children from disadvantaged backgrounds are often not given a fair chance by the school system. They run up against people who write them off as stupid when their problems are really connected to their poverty, not to their intellectual ability.