This is an interesting idea. However, I must be honest. The fact that the dolls are only mentioned once in the entire short story and that they are given a very cursory description suggests to me that they are not really important, and that we need to be careful when reading in to them any symbolic meaning. Far more attention is given to the lamp, for example, which clearly can be argued to be symbolic. Let us consider the description of the dolls that we are given:
The father and mother dolls, who sprawled very stiff as though they had fainted in the drawing room, and their two little children asleep upstairs, were really too big for the doll's house. They didn't look as though they belonged.
Now, the phrase "They didn't look as if they belonged," could be used to suggest that the dolls could be symbolic of the Kelvey's, as you suggest, because they definitely do not belong to their surroundings either. However, this is much more likely to be a rather humorous description of the doll's house as being rather cheap and not well-made. In spite of all the detail, let us remember that there were "big lumps of congealed paint hanging along the edge." Therefore, I think it is more likely that this is not a symbolic comment as such, and just relates to the shoddy presentation of this doll's house, which nevertheless holds such importance for the Burnell girls.