One of the best indicators that shows Lyddie's family is poor from those first chapters is that the family carries debt. If Lyddie's family was wealthy, they would not have the massive debt. Lyddie's family is poor enough that they can not pay the debt with cash either. They have to work off the debt, and that is why Lyddie is hired out to Cutler's Tavern. The debt is being paid back through her employment.
A good piece of specific textual evidence can be found in chapter one. Lyddie's mother decides to leave the house with the kids, but Lyddie points out that they don't have enough money to pay the coach fare for the entire family. Also in chapter one, Lyddie and Charles are awarded with a newborn calf, and there is an abundance of milk and cream. We are told that the two characters felt as rich as "townsfolk." That means they are poorer than normal, middle class townsfolk. A bit later Lyddie gets some coins out of the "almost empty cash box."
Chapter 3 has Lyddie arriving at Cutler's Tavern, and the owner tells Lyddie to go back to the township poor farm because of how ragged Lyddie looks. Later in the chapter we are told that Lyddie hasn't had a new dress in four years because the family can't afford it, and her current clothes are described as "ragged." Mistress Cutler provides Lyddie with servant's clothing. It isn't especially nice, but it is a major upgrade for Lyddie.