The answer to whether Claudius or Hamlet is the protagonist of Hamlet resides in the definition of tragic hero, assuming, of course, that when referring to a Shakespearean tragedy "protagonist" is equated with "tragic hero." Though it may be said that Claudius has a flaw in personal moral character, Claudius's actions remove him from consideration under the Aristotelian definition of tragic hero. Claudius's actions are horrible instead of tragic.
The Aristotelian definition of a tragic hero requires that the tragic hero commit a tragic mistake that arises from a tragic error in judgment or from a tragic character flaw. The horrible events that the tragic hero precipitates are not out of intended deliberate manipulative plotting but out of error or flawed character. What about Macbeth? He is represented as being loyal to Duncan and tending toward graciously accepting his reward, though less than expected. He is swayed from his inclinations of loyalty by ambition, by the supernatural powers that come to thwart him, and by the overbearing influence of his intentionally and deliberately cruel wife. The difference between a tragic hero and a wicked character is intention and deliberation.
Shakespeare doesn't seem to deviate significantly on this point of definition except in his preference for tragic character flaw over tragic error in judgment. Shakespeare's tragic heroes were all men who were superior humans who were fallible to the point of tragedy. Claudius's character traits depart from this definition in that his fallability displays itself as deliberate choice rather than mistakes stemming from error or unperceived character flaws. In this, Claudius is more akin to Lady Macbeth than to, for example, Othello. In other words, Claudius's actions are derived from intention and deliberation; they are not mistakes that take him by surprise with their consequences.
Therefore, even though Claudius's character is flawed--by villainous levels of ambition and greed--and even though Claudius tries to pray for repentance and even though Claudius's affection for Gertrude may give a glimmer of redemptive power, his actions--which are manipulated and derived from intention and deliberation with nothing mistaken or accidental about them and which he himself calls murder--are horrible, not tragic. Thus Hamlet is the protagonist/tragic hero of Hamlet. His mistakes and tragic end can be traced to errors in judgment and to one or more tragic flaws in his nature.