The short answer to your question is that everything related to his visit to the bank rattles the protagonist in this short story. I would argue that he is probably a nervous person in general, and his nerves certainly get the better of him when he goes into the bank.

The clerks make him uncomfortable and it is quickly apparent that he does not trust them, because he asks to speak to the manager with regard to the deposit of the relatively small amount of $56. He is also intimidated by the wickets (service windows) and money itself. He sums it up well in the following three words:

Everything rattles me.

We can tell that he is nervous based on the way he "shambled" in, which implies a certain unwillingness. He looks "timidly" at the clerks, which speaks of shyness and awkwardness. The fact that he asks to see the manager "alone" and does not even know why he made this request for privacy is also indicative of the fact that this man is "rattled" and not thinking clearly.

Humiliation is then added to the mix when the manager realizes that he is neither a detective nor a man investing a large fortune, and asks one of the accountants to assist him. This "rattles" him further.

The ultimate proof that everything about the experience of visiting the bank has "rattled" or disturbed our protagonist is that immediately after making his deposit, he accidentally draws out all his money, having intended to draw out just $6.