Is Hamlet the protagonist or the antagonist of the play or both?
Hamlet is most clearly the protagonist of the play, which follows him as he seeks revenge on the murderer of his father (the late King Hamlet). After learning that his father was poisoned by his father's own brother, Claudius--who has since assumed the throne and married his dead brother's wife, Gertrude--Hamlet is determined (with a little motivational help from the ghost of his father) to kill Claudius before he can repent his sins, thus sending his traitorous uncle to hell. This positions Claudius as the clear antagonist of the play, and the man follows suit, attempting to send Hamlet away or have him deemed crazy in order to prevent his dark secret from being revealed.
However, your question does hint at an important point, which is just how often Hamlet's behavior is antagonistic--at least towards his own self. Hamlet's internal struggle throughout the play almost reads as a battle between a protagonist and antagonist, and it is these rivaling, negative qualities (his doubt, his fear, his cowardice) rather than any external human intervention that serve as the greatest impositions to Hamlet carrying out his revenge.
Hamlet is the protagonist of the play and Claudius would be considered the antagonist of the play. The entire play deals with Hamlet's quandary over how to deal with his father's death at the hand of Claudius. Once Claudius sees Hamlet as a threat, especially after seeing the play "The Mousetrap", Claudius becomes a real enemy of Hamlet and seeks to kill Hamlet. First, Claudius tries to send Hamlet to England and have the English king kill his nephew and when that fails, he plots to poison Hamlet. That certainly qualifies him as the antagonist. It's true Hamlet does wrestle with his conscience a great deal during the play, but that does not absolve Claudius from opposing Hamlet to the degree of wanting to take his life.