Switzerland prided itself on its neutrality in world affairs at that time(and still does). It also did not participate in World War I or any other wars for centuries prior, so there was a longstanding tradition of neutrality that the Swiss wished very deeply to maintain.
A second reason is geography. Switzerland is surrounded by the Alps mountain range, with limited passes and access points for armor and infantry to invade. This makes it both discouraging for attacking armies and easy to defend. Hitler's air force, the Luftwaffe, was engaged in small skirmishes with the Swiss Air Force (who, ironically, had bought German Me-109's for their defense) but Hitler was never that interested in actually invading the country. The one time he brought it up, his generals quickly talked him out of it. He also did not feel the least bit threatened by them.
Lastly, Switzerland was no pushover militarily. They maintained a large standing army and were reasonably well equipped, in the belief that the best way to avoid war was through a strong defensive force (armed neutrality) that would make an invasion too costly to attempt.