This is clearly a very difficult question to answer, because there seems to be a fine line between madness and the way that some people act as a response to the death of a loved one, for example. This means that it is problematic to point the finger at any one behaviour and label it as a definite sign of madness.
However, you might want to consider Hamlet's actions as reported by Ophelia to her father in Act II scene 1. Consider what Ophelia tells her father about how Hamlet appeared to her:
My lord, as I was sewing in my closet,
Lord Hamlet, with his doublet all unbraced;
No hat upon his head; his stockings foul'd,
Ungarter'd, and down-gyved to his ancle;
Pale as his shirt; his knees knocking each other;
And with a look so piteous in purport
As if he had been loosed out of hell
To speak of horrors,--he comes before me.
The way that his clothing is unkempt and he is not dressed properly combined with his facial expressions and how they move Ophelia and make her scared, so much so that she runs to report it to her father straight away does seem to indicate that Hamlet is mad, or at the very least an incredibly convincing actor.
Other scenes you might want to consider would be his famous rant at Ophelia in Act III scene 1 when he tells her to "Get thee to a nunnery." At each and every stage, however, his actions can be explained through extreme grief or the response to betrayal.