Let us remember that the central theme of this brilliant play is the way that society literally encroaches into our personal lives. This is of course most stridently shown through the character of Mrs. Alving, who is so focussed on maintaining appearances and trying to protect the reputation of her dead husband that she in turn destroys the lives of Oswald and Regina and devotes all of her energies into preserving a lie.
Of course, she is not the only character to be dominated by the "ghosts" of public standards and good opinion. Pastor Manders shows himself to be completely dominated by these aspects which results in a number of seriously foolish acts on his part. Note too the way that his principals and the fear of what others would think lead him to ignore his own personal feelings that would have made him and Mrs. Alving happy.
By far the most important quote in the play which supports this theme is what Mrs. Alving says about ghosts in Act II:
I am half inclined to think we are all ghosts, Mr. Manders. It is not only what we have inherited from our fathers and mothers that exists again in us, but all sorts of old dead ideas and all kinds of old dead beliefs and things of that kind. They are not actually alive in us; but there they are dormant, all the same, and we can never be rid of them. Whenever I take up a newspaper and read it, I fancy I see ghosts creeping between the lines. There must be ghosts all over the world. They must be as countless as the grains of the sands, it seems to me. And we are so miserably afraid of the light, all of us.
This quote explains how duty and public opinion are viewed as "ghosts" in a brilliant metaphor, and how they can completely destroy whole generations of lives. It is not only what we inherit from our family members that can have this impact on us, but also the general customs of society that we need to watch out for. Filial piety is certainly one such "ghost" that comes to have a negative impact on the characters in this play and saps the joy of life.