Hamlet is upset with the gravedigger because the gravedigger does not seem to treat his task with the proper gravity. Hamlet asks Horatio, of the gravedigger, "Has this fellow no feeling of his business? He sings in grave-making" (5.1.67-68). Having recently lost his father, Hamlet is especially aware of the real-life tragedy of dealing with the death of someone one loves. When the gravedigger digs up a skull, Hamlet says, "That skull had a tongue in it and could sing once" (5.1.77-78). He considers the lives of the people whose bones the gravedigger digs up, thinking about what they might have been like during their lives, humanizing them in a way that the gravedigger does not. He becomes a bit philosophical as a result.
Once Hamlet begins to converse with the gravedigger, he becomes a little irritated at how literal the man is in his speech. The gravedigger says that the grave is not for a man, and is not for a woman, but it is for a person who used to be a woman during her life. "How absolute the knave is! We must speak by the card, or equivocation will undo us" Hamlet exclaims (5.1.140). The gravedigger's speech is so literal that it is difficult to actually have a conversation with him.