The wedding guest is on his way to a wedding when the Mariner stops him. The third stanza lets readers know that the Mariner must have stopped the wedding guest by physical force.
He holds him with his skinny hand,
'There was a ship,' quoth he.
'Hold off! unhand me, grey-beard loon!'
I think that would automatically make most people nervous. If someone that you didn't know grabbed you and forced you to stop, that would likely put you on edge. The wedding guest responds in a manner that I think most nervous people would respond. He tells the Mariner to let go, and the wedding guest does so with an insult attached to it. The Mariner lets go, but the wedding guest discovers that he isn't free to go.
He holds him with his glittering eye—
The Wedding-Guest stood still,
And listens like a three years' child:
The Mariner hath his will.
The Wedding-Guest sat on a stone:
He cannot choose but hear;
Somehow the Mariner is able to influence the mental decision making process of the wedding guest. That's got to be terrifying. To know that you should be doing something, but you are unable to even choose that course of action has to be nerve wracking. Then the Mariner proceeds to tell a horrific story of starving and undead sailors, and the wedding guest flat out states that he is afraid.
'I fear thee, ancient Mariner!'