First of all, please note that Broca was not unable to take age and height into account. Instead, he simply chose not to do so.
Gould points this out because this is one of the mistakes that he says cause Broca's results to be invalid. He points out that the women Broca studied were "considerably older" than the men in the study. He also notes that the men were, on average, about 6 inches taller than the women. Brains shrink with age and increase with the overall body size of the person. This means that the results in Broca's test were skewed. The differences in brain size were not due mainly (as Broca claimed) to sex. Instead, Gould argues, they were due mainly to the age and overall size of the people whose brains were studied. This, Gould says, means that Broca was wrong when he claimed that a person's sex influenced the size of their brain.