What does Hamlet mean when he says "Wormwood, wormwood" in Act III, scene ii?
Let's first get some context as to when and why this moment happens in the play. As you have mentioned in your question, Hamlet speaks these words in Act III, scene ii. By this time, Hamlet has met with the ghost of his father, the deceased King Hamlet, and learned that his uncle, the new King Claudius, had murdered King Hamlet in order to assume the throne and marry Hamlet's mother, Queen Gertrude. Hamlet devises a way to test if what the Ghost says is true: he will have a play staged that mimics these events and invite Claudius and Gertrude to sit in the audience. Hamlet believes that Claudius' reaction to the play will provide him with the emotional evidence necessary to assess his guilt.
The plan is carried out as Hamlet intended. As the Player King draws his dying breath, he suggests that the Player Queen might find a new husband, to which the Player Queen responds:
Oh, confound the rest!
Such love must needs be treason in my breast.
In second husband let me be accursed!
None wed the second but who killed the first.
What the Player Queen is suggesting here is that the only way a woman could stand to marry a new man after the death of her beloved was if she had killed her beloved in the first place; no woman who was truly in love would be able to emotionally move on from the death of her husband.
Uttering "Wormwood, wormwood" at this moment in the play-within-a-play was Hamlet's way of saying that the speech of the Player Queen is particularly harsh to hear, knowing that this is exactly what Gertrude did: allow her first husband to be killed while willingly re-marrying the man who did the killing. The word "wormwood" itself is a reference to a bitter extract of a plant, so Hamlet's expression is to be taken quite literally: this is a bitter moment for the family.
Hamlet says "Wormwood!" in response to the player queen's speech. Her husband the king has just told her that he believes he is about to die, and she tells him that she will not remarry, believing that to find a second love is to destroy or demean the first. Wormwood is an exceedingly bitter plant, and Hamlet means to say that the sentiments expressed in the speech are bitter and heart-rending. Of course, the Queen views it differently, claiming that "the lady protests too much." Clearly, the speech has hit close to home with her, which is, of course, precisely what Hamlet had wished.