When people suddenly stop giving me eye contact and start looking all around in different directions, as well as start saying fewer words, it leaves me to believe they're just not interested in the conversation anymore. I also would wonder if I said something to make this person lose interest in talking with me.

Gestures that imply speech by catching my attention can make me feel uncomfortable when they are not followed by speech. A non-verbal sound or a raised hand can make me stop doing what I am doing and attend to the person making the gesture. This is fine when the gesture is genuine, but off-putting when it's not.

This is as much an annoyance as a discomfort, I suppose, but I am certainly nonplussed when people ask for attention non-verbally and I give it to them and then they don't say anything to warrant that attention. 

The deep sigh which says, 'I am bored with you wasting my time' gets me every time. As does sucking the teeth. I have to say I have got over the looking into the eyes thing as in Maori culture to look directly into someone's eyes can be a sign of confrontation and disrespect.

I absolutely abhor when a person will not look me in the eyes or they roll their eyes. I believe in the age old adage that "the eyes are the window to the soul." I want to see more than the side of a person's face or eyelids.

As for the rolling eyes, I look at it as a sign of disrespect.

When I am talking to someone and they stand in front of me with their arms folded I feel like they are putting up a barrier. It's as if they are saying, "I'm not really listening to you," or "I'm not interested in what you have to say."

Or when someone starts shaking their head at something I'm saying before I'm done saying it. That seems to say, "I've already decided that you're wrong" even though I'm not done yet.

For me, one of the nonverbal things that really bugs me is people saying "um" a lot.  When people talk, I like them to be more ready to actually say what they have to say rather than being all hesitant about it.  It just sounds really bad when people do this.