What “erroneous conclusion” reminded Holmes of the dangers of reasoning from “insufficient data?”
The only thing I can think of is that Sherlock Holmes had concluded that the gypsies had something to do with the death of the first sister (Julia). I say this because Holmes says (after talking about his erroneous conclusion) that he came to that conclusion because of the presence of the gypsies and the use of the word "band."
I am not completely happy with this answer, though. Earlier, when Helen Stoner comes to Holmes's home, she comes up with that same theory. When she does, Holmes shakes his head "like a man who is far from being satisfied." If he does that, it doesn't make sense to me that he would come to the same conclusion.
However, he never tells us what conclusion he did come to. And it seems that his conclusion must have been connected to the gypsies, given what he says.