In the play Hamlet, incest is a display of dysfunction and brokenness, and it is used as a way to exaggerate how drastically messed up this family is. There is overt incest in the play, which shows the twisted nature of royal families. (Incest was, in fact, fairly common among the ruling members of society in that day, so this is a commentary on the political elite as much as anything.)

The continual obsession with the sexuality of relatives is a form of control and a source of anger. Hamlet becomes enraged at Claudius, who is his mother's new husband. This shows that Hamlet believes he is entitled to decide with whom his mother engages in sexual activity and that he must approve of her partners. The incest, both implied and actual, is a portrayal of control and dominance, as well as the dysfunction apparent in this family. It is indicative of the true nature of royalty in that era.

Although I'm not sure that Shakespeare intended this result, all of the incestuous undertones -- as well as the actual incest between Claudius and Gertrude, a brother and sister-in-law -- help to show the way women are viewed as sex objects in Hamlet's society.  Laertes, for example, talks pretty explicitly to his sister, Ophelia, about sexual intercourse and female genitalia (via the worm/flower metaphor), sexually objectifying her at the same time that he advises her of the importance of remaining chaste.  

Further, Hamlet's obsession with his mother's sex life, especially the sex that he assumes she is having with Claudius, her new husband, is particularly disturbing. Hamlet dwells on his mother's sexuality to an extreme. Laertes seems not to be able to separate his sister from general female sexual objectification, and Hamlet has the same problem.  The fact that they view their female relatives in a sexual way shows how deeply the sexual objectification of women is engendered in males in a society that values women mainly for their sexuality.