Walter is reluctant to follow Scout and Jem home to dinner as a matter of pride.  His family does not believe in taking anything they can't pay back, and while they might sometimes pay people back in ways other than money, as does Mr. Cunningham when it comes to paying Atticus for his legal work with firewood and hickory nuts, their pride keeps them from taking charity.  Walter knows he will not be able to return the favor to Jem and Scout.

The issue first comes up in the classroom when Miss Caroline notices that Walter did not bring a lunch.  When she also finds that he does not have money to buy a lunch, she tries to give him a quarter, which he refuses to take.  Too shy to explain himself, Scout steps in, saying:

"You're shamin' him, Miss Caroline.  Walter hasn't got a quarter at home to bring you, and you can't use any stovewood" (Lee 34).

Miss Caroline ends up punishing Scout for her remarks, so once outside the classroom, Scout starts a fight with Walter, upset that he made her "start off on the wrong foot" (36).  Jem breaks up the fight, then issues an invitation to Walter:

"Come on home to dinner with us, Walter.... We'd be glad to have you" (36).

Walter's face initially brightens at this offer, but soon darkens again as he realizes he cannot possibly accept.  That is, until Jem says, "Our daddy's a friend of you daddy's" (36).  This makes Walter's acceptance of the invitation okay: it's not charity, it's a friendly get-together.  Hungry, and excited at the opportunity to eat a delicious dinner, Walter eventually accepts, much to Scout's chagrin.