I believe that this line was spoken by Ellie's father in the novel when the Jews were first forced to wear the star on their clothing. At this point in the novel, his father didn't think it was "too bad" to put the star of their faith on their clothing. Wearing this star on their clothing labeled them as Jews and led them to their death. As you know from the rest of the novel, Jews were taken from their homes and brought to the concentration camps where they were eventually executed.
Most likely, this particular line is a reference to the Holocaust. My guess is that the father and child are Jews. In this case, the father is trying to downplay the significance of the "yellow star": an identifying badge all Jews had to wear. This line, then, is ironic because almost all Jews who were unfortunate enough to be labeled with this star were eventually executed because of their faith. The star is a symbol--one that does indeed lead to the father's death.