Othello's threat may seem sudden, but Iago's cruel torments have been working away at his master's soul to the point that he can take no more.
Initially, Iago raises Othello's suspicions of Cassio by simply returning Othello's words, implying he feels his comments would be too indelicate-
Think, my lord? By heaven, he echoes me,
As if there were some monster in his thought
Too hideous to be shown.
Iago then directs Othello to consider whether he trusts his wife, having warned him of the evils of jealousy-
I speak not yet of proof.
Look to your wife; observe her well with Cassio
Wear your eye thus, not jealous nor secure.
I would not have your free and noble nature
Out of selfbounty be abused. Look to't.
Iago is leading Othello, as he indicated he would , to his own downfall. Iago has sworn to bring down Othello and Desdemona.
Othello’s anger builds and he threatens Iago-
Villain, be sure thou prove my love a whore;
Be sure of it. Give me the ocular proof;
Or, by the worth of man's eternal soul,
Thou hadst been better have been born a dog
Than answer my waked wrath!
Othello’s passionate temper is tragic in that the audience already know that Iago has Desdemona’s handkerchief, and has long conceived of a plan to use it in his cruel deception. The audience realises that Iago will do as Othello asks, and is likely to succeed in his devilish plan.